2024-04-12T05:11:53.685 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-12T05:11:53.685 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2024-04-11_20:32:36-orch-reef-release-distro-default-smithi --verbose --description orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2024-04-11_20:32:36-orch-reef-release-distro-default-smithi/7652637 -- /home/teuthworker/archive/yuriw-2024-04-11_20:32:36-orch-reef-release-distro-default-smithi/7652637/orig.config.yaml 2024-04-12T05:11:53.737 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-12T05:11:53.827 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-11_20:32:36-orch-reef-release-distro-default-smithi/7652637 branch: reef-release description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: null first_in_suite: false job_id: '7652637' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: yuriw-2024-04-11_20:32:36-orch-reef-release-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: reef-release ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: reef-release sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 2064 sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 sleep_before_teardown: 0 subset: 111/120000 suite: orch suite_branch: reef-release suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 targets: smithi032.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGwLR3pYMLw2TyxNMHAAobb3adjJnjw42qJCHwFMzm8aE4E0dQ2GbQUtEsEKQqZDklHhErCNA+7oG7UyJm7OX6E= smithi067.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCm5sT0zJNfvCAmY690EA0xz0PW/cSYnhpKCrJCL1Lg+BAHvqx6BIRr6D1bIVyb7LtWm42y47x55+MBu0dYE0wE= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - ceph-fuse: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-11_20:32:36 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-04-12T05:11:53.827 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa; will attempt to use it 2024-04-12T05:11:53.827 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks 2024-04-12T05:11:53.828 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-12T05:11:53.829 INFO:teuthology.task.internal:Checking packages... 2024-04-12T05:11:53.851 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd540ebaca6b131a1dd560e7f69e024b133bbaa42' 2024-04-12T05:11:53.851 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-12T05:11:53.851 INFO:teuthology.packaging:ref: None 2024-04-12T05:11:53.851 INFO:teuthology.packaging:tag: None 2024-04-12T05:11:53.851 INFO:teuthology.packaging:branch: reef-release 2024-04-12T05:11:53.851 INFO:teuthology.packaging:sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:11:53.851 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=reef-release 2024-04-12T05:11:53.998 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1187.gd540ebac 2024-04-12T05:11:54.000 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-12T05:11:54.006 INFO:teuthology.task.internal:no buildpackages task found 2024-04-12T05:11:54.006 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-12T05:11:54.019 INFO:teuthology.task.internal:Saving configuration 2024-04-12T05:11:54.039 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-12T05:11:54.049 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-12T05:11:54.072 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-11_20:32:36-orch-reef-release-distro-default-smithi/7652637', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-12 05:06:36.170556', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGwLR3pYMLw2TyxNMHAAobb3adjJnjw42qJCHwFMzm8aE4E0dQ2GbQUtEsEKQqZDklHhErCNA+7oG7UyJm7OX6E='} 2024-04-12T05:11:54.090 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-11_20:32:36-orch-reef-release-distro-default-smithi/7652637', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-12 05:06:36.168617', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCm5sT0zJNfvCAmY690EA0xz0PW/cSYnhpKCrJCL1Lg+BAHvqx6BIRr6D1bIVyb7LtWm42y47x55+MBu0dYE0wE='} 2024-04-12T05:11:54.090 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-12T05:11:54.101 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-04-12T05:11:54.101 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-04-12T05:11:54.101 INFO:teuthology.run_tasks:Running task console_log... 2024-04-12T05:11:54.174 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fdbb57ac0d0>, signals=[15]) 2024-04-12T05:11:54.175 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-12T05:11:54.186 INFO:teuthology.task.internal:Opening connections... 2024-04-12T05:11:54.186 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi032.front.sepia.ceph.com 2024-04-12T05:11:54.188 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T05:11:54.266 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi067.front.sepia.ceph.com 2024-04-12T05:11:54.267 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T05:11:54.337 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-12T05:11:54.346 DEBUG:teuthology.orchestra.run.smithi032:> uname -m 2024-04-12T05:11:54.370 INFO:teuthology.orchestra.run.smithi032.stdout:x86_64 2024-04-12T05:11:54.370 DEBUG:teuthology.orchestra.run.smithi032:> cat /etc/os-release 2024-04-12T05:11:54.431 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="CentOS Stream" 2024-04-12T05:11:54.431 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="8" 2024-04-12T05:11:54.432 INFO:teuthology.orchestra.run.smithi032.stdout:ID="centos" 2024-04-12T05:11:54.432 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE="rhel fedora" 2024-04-12T05:11:54.432 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="8" 2024-04-12T05:11:54.432 INFO:teuthology.orchestra.run.smithi032.stdout:PLATFORM_ID="platform:el8" 2024-04-12T05:11:54.432 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-12T05:11:54.432 INFO:teuthology.orchestra.run.smithi032.stdout:ANSI_COLOR="0;31" 2024-04-12T05:11:54.432 INFO:teuthology.orchestra.run.smithi032.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-12T05:11:54.432 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://centos.org/" 2024-04-12T05:11:54.432 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-12T05:11:54.432 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-12T05:11:54.432 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-12T05:11:54.433 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server 2024-04-12T05:11:54.455 DEBUG:teuthology.orchestra.run.smithi067:> uname -m 2024-04-12T05:11:54.477 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2024-04-12T05:11:54.477 DEBUG:teuthology.orchestra.run.smithi067:> cat /etc/os-release 2024-04-12T05:11:54.539 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="CentOS Stream" 2024-04-12T05:11:54.539 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="8" 2024-04-12T05:11:54.539 INFO:teuthology.orchestra.run.smithi067.stdout:ID="centos" 2024-04-12T05:11:54.539 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE="rhel fedora" 2024-04-12T05:11:54.539 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="8" 2024-04-12T05:11:54.539 INFO:teuthology.orchestra.run.smithi067.stdout:PLATFORM_ID="platform:el8" 2024-04-12T05:11:54.539 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-12T05:11:54.539 INFO:teuthology.orchestra.run.smithi067.stdout:ANSI_COLOR="0;31" 2024-04-12T05:11:54.539 INFO:teuthology.orchestra.run.smithi067.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-12T05:11:54.539 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://centos.org/" 2024-04-12T05:11:54.539 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-12T05:11:54.539 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-12T05:11:54.539 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-12T05:11:54.540 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2024-04-12T05:11:54.563 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-12T05:11:54.574 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-12T05:11:54.586 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-12T05:11:54.586 DEBUG:teuthology.orchestra.run.smithi032:> test '!' -e /home/ubuntu/cephtest 2024-04-12T05:11:54.589 DEBUG:teuthology.orchestra.run.smithi067:> test '!' -e /home/ubuntu/cephtest 2024-04-12T05:11:54.609 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-12T05:11:54.618 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-12T05:11:54.618 DEBUG:teuthology.orchestra.run.smithi032:> test -z $(ls -A /var/lib/ceph) 2024-04-12T05:11:54.651 DEBUG:teuthology.orchestra.run.smithi067:> test -z $(ls -A /var/lib/ceph) 2024-04-12T05:11:54.683 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-12T05:11:54.740 INFO:teuthology.run_tasks:Running task kernel... 2024-04-12T05:11:54.757 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-12T05:11:54.757 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-12T05:11:54.757 DEBUG:teuthology.orchestra.run.smithi032:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-12T05:11:54.758 DEBUG:teuthology.orchestra.run.smithi067:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-12T05:11:54.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T05:11:54.778 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-04-12T05:11:54.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T05:11:54.779 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-04-12T05:11:54.838 INFO:teuthology.orchestra.run.smithi067.stdout:4.18.0-489.el8.x86_64 2024-04-12T05:11:54.838 INFO:teuthology.task.kernel:Running kernel on smithi067: 4.18.0-489.el8.x86_64 2024-04-12T05:11:54.839 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum install -y kernel 2024-04-12T05:11:54.840 INFO:teuthology.orchestra.run.smithi032.stdout:4.18.0-489.el8.x86_64 2024-04-12T05:11:54.840 INFO:teuthology.task.kernel:Running kernel on smithi032: 4.18.0-489.el8.x86_64 2024-04-12T05:11:54.840 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum install -y kernel 2024-04-12T05:11:58.918 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 8 - AppStream 19 MB/s | 28 MB 00:01 2024-04-12T05:11:58.966 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 8 - AppStream 19 MB/s | 28 MB 00:01 2024-04-12T05:12:04.983 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 8 - BaseOS 14 MB/s | 10 MB 00:00 2024-04-12T05:12:05.018 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 8 - BaseOS 14 MB/s | 10 MB 00:00 2024-04-12T05:12:07.284 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 8 - Extras 31 kB/s | 18 kB 00:00 2024-04-12T05:12:07.302 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 8 - Extras 31 kB/s | 18 kB 00:00 2024-04-12T05:12:07.778 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 8 - Extras common packages 21 kB/s | 7.7 kB 00:00 2024-04-12T05:12:07.784 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 8 - Extras common packages 20 kB/s | 7.7 kB 00:00 2024-04-12T05:12:08.004 INFO:teuthology.orchestra.run.smithi032.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-04-12T05:12:08.056 INFO:teuthology.orchestra.run.smithi067.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2024-04-12T05:12:09.038 INFO:teuthology.orchestra.run.smithi032.stdout:Extra Packages for Enterprise Linux 17 MB/s | 16 MB 00:00 2024-04-12T05:12:10.478 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 7.0 MB/s | 16 MB 00:02 2024-04-12T05:12:13.063 INFO:teuthology.orchestra.run.smithi032.stdout:lab-extras 437 kB/s | 24 kB 00:00 2024-04-12T05:12:14.512 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 463 kB/s | 24 kB 00:00 2024-04-12T05:12:16.294 INFO:teuthology.orchestra.run.smithi032.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-12T05:12:16.294 INFO:teuthology.orchestra.run.smithi032.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-12T05:12:16.392 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-12T05:12:16.393 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:12:16.393 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-04-12T05:12:16.393 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:12:16.393 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-12T05:12:16.393 INFO:teuthology.orchestra.run.smithi032.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-12T05:12:16.393 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-12T05:12:16.393 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-12T05:12:16.393 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-12T05:12:16.393 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:12:16.393 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-12T05:12:16.394 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:12:16.394 INFO:teuthology.orchestra.run.smithi032.stdout:Install 3 Packages 2024-04-12T05:12:16.394 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:12:16.394 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 90 M 2024-04-12T05:12:16.394 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 96 M 2024-04-12T05:12:16.394 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-12T05:12:17.348 INFO:teuthology.orchestra.run.smithi032.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 13 MB/s | 10 MB 00:00 2024-04-12T05:12:17.670 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-12T05:12:17.671 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-12T05:12:17.791 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-12T05:12:17.792 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:12:17.792 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-04-12T05:12:17.792 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:12:17.792 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-12T05:12:17.792 INFO:teuthology.orchestra.run.smithi067.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-12T05:12:17.792 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-04-12T05:12:17.792 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-12T05:12:17.792 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-12T05:12:17.792 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:12:17.792 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-12T05:12:17.792 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:12:17.793 INFO:teuthology.orchestra.run.smithi067.stdout:Install 3 Packages 2024-04-12T05:12:17.793 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:12:17.793 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 90 M 2024-04-12T05:12:17.793 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 96 M 2024-04-12T05:12:17.793 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-12T05:12:18.282 INFO:teuthology.orchestra.run.smithi032.stdout:(2/3): kernel-core-4.18.0-552.el8.x86_64.rpm 26 MB/s | 44 MB 00:01 2024-04-12T05:12:18.657 INFO:teuthology.orchestra.run.smithi032.stdout:(3/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 18 MB/s | 36 MB 00:02 2024-04-12T05:12:18.658 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:12:18.658 INFO:teuthology.orchestra.run.smithi032.stdout:Total 40 MB/s | 90 MB 00:02 2024-04-12T05:12:18.746 INFO:teuthology.orchestra.run.smithi067.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 14 MB/s | 10 MB 00:00 2024-04-12T05:12:19.382 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-12T05:12:19.613 INFO:teuthology.orchestra.run.smithi067.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 22 MB/s | 36 MB 00:01 2024-04-12T05:12:19.835 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-12T05:12:19.835 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-12T05:12:20.105 INFO:teuthology.orchestra.run.smithi067.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 21 MB/s | 44 MB 00:02 2024-04-12T05:12:20.106 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:12:20.106 INFO:teuthology.orchestra.run.smithi067.stdout:Total 39 MB/s | 90 MB 00:02 2024-04-12T05:12:20.824 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-12T05:12:21.156 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-12T05:12:21.175 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-12T05:12:21.262 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-12T05:12:21.262 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-12T05:12:22.576 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-12T05:12:22.594 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-12T05:12:24.127 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-12T05:12:24.653 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-12T05:12:25.554 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-12T05:12:26.072 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-12T05:12:28.374 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-12T05:12:28.728 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-12T05:12:29.804 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-12T05:12:30.213 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-12T05:12:32.409 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-12T05:12:32.698 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-12T05:12:33.895 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-12T05:12:34.201 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-12T05:12:56.288 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-12T05:12:56.314 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-12T05:12:56.671 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-12T05:12:56.671 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-12T05:12:56.671 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-12T05:12:57.176 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-12T05:12:57.177 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:12:57.177 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-12T05:12:57.177 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-12T05:12:57.177 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-12T05:12:57.177 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:12:57.177 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-12T05:12:57.322 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q kernel | sort -rV | head -n 1 2024-04-12T05:12:57.550 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-12T05:12:57.551 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-12T05:12:57.551 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-12T05:12:57.551 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-12T05:12:57.551 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-12T05:12:57.551 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-04-12T05:12:57.551 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi032.front.sepia.ceph.com, path=None, version=distro) 2024-04-12T05:12:57.552 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q kernel | sort -rV | head -n 1 2024-04-12T05:12:57.552 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-12T05:12:57.577 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-12T05:12:57.773 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-12T05:12:57.774 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-12T05:12:57.774 DEBUG:teuthology.orchestra.run.smithi032:> sudo rpm -qi grub2-tools 2024-04-12T05:12:57.808 INFO:teuthology.orchestra.run.smithi032.stdout:Name : grub2-tools 2024-04-12T05:12:57.808 INFO:teuthology.orchestra.run.smithi032.stdout:Epoch : 1 2024-04-12T05:12:57.808 INFO:teuthology.orchestra.run.smithi032.stdout:Version : 2.02 2024-04-12T05:12:57.808 INFO:teuthology.orchestra.run.smithi032.stdout:Release : 129.el8 2024-04-12T05:12:57.809 INFO:teuthology.orchestra.run.smithi032.stdout:Architecture: x86_64 2024-04-12T05:12:57.809 INFO:teuthology.orchestra.run.smithi032.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-12T05:12:57.809 INFO:teuthology.orchestra.run.smithi032.stdout:Group : System Environment/Base 2024-04-12T05:12:57.809 INFO:teuthology.orchestra.run.smithi032.stdout:Size : 9265586 2024-04-12T05:12:57.809 INFO:teuthology.orchestra.run.smithi032.stdout:License : GPLv3+ 2024-04-12T05:12:57.809 INFO:teuthology.orchestra.run.smithi032.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-12T05:12:57.809 INFO:teuthology.orchestra.run.smithi032.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-12T05:12:57.809 INFO:teuthology.orchestra.run.smithi032.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-12T05:12:57.809 INFO:teuthology.orchestra.run.smithi032.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-12T05:12:57.809 INFO:teuthology.orchestra.run.smithi032.stdout:Relocations : (not relocatable) 2024-04-12T05:12:57.809 INFO:teuthology.orchestra.run.smithi032.stdout:Packager : CentOS BuildSystem 2024-04-12T05:12:57.809 INFO:teuthology.orchestra.run.smithi032.stdout:Vendor : CentOS 2024-04-12T05:12:57.809 INFO:teuthology.orchestra.run.smithi032.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-12T05:12:57.809 INFO:teuthology.orchestra.run.smithi032.stdout:Summary : Support tools for GRUB. 2024-04-12T05:12:57.809 INFO:teuthology.orchestra.run.smithi032.stdout:Description : 2024-04-12T05:12:57.810 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:12:57.810 INFO:teuthology.orchestra.run.smithi032.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-12T05:12:57.810 INFO:teuthology.orchestra.run.smithi032.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-12T05:12:57.810 INFO:teuthology.orchestra.run.smithi032.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-12T05:12:57.810 INFO:teuthology.orchestra.run.smithi032.stdout:hardware devices. 2024-04-12T05:12:57.810 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:12:57.810 INFO:teuthology.orchestra.run.smithi032.stdout:This subpackage provides tools for support of all platforms. 2024-04-12T05:12:57.811 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-12T05:12:57.811 INFO:teuthology.task.kernel:Updating grub on smithi032 to boot 4.18.0-552.el8.x86_64 2024-04-12T05:12:57.811 DEBUG:teuthology.orchestra.run.smithi032:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-12T05:12:57.946 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-12T05:12:57.946 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-12T05:12:57.946 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-12T05:12:58.450 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-12T05:12:58.521 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:12:58.521 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-12T05:12:58.521 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-12T05:12:58.521 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-12T05:12:58.521 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:12:58.521 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-12T05:12:58.585 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q kernel | sort -rV | head -n 1 2024-04-12T05:12:58.812 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-12T05:12:58.812 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-12T05:12:58.812 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-12T05:12:58.812 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-12T05:12:58.812 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-12T05:12:58.813 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-04-12T05:12:58.813 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi067.front.sepia.ceph.com, path=None, version=distro) 2024-04-12T05:12:58.813 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q kernel | sort -rV | head -n 1 2024-04-12T05:12:59.034 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-12T05:12:59.035 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-12T05:12:59.035 DEBUG:teuthology.orchestra.run.smithi067:> sudo rpm -qi grub2-tools 2024-04-12T05:12:59.069 INFO:teuthology.orchestra.run.smithi067.stdout:Name : grub2-tools 2024-04-12T05:12:59.069 INFO:teuthology.orchestra.run.smithi067.stdout:Epoch : 1 2024-04-12T05:12:59.069 INFO:teuthology.orchestra.run.smithi067.stdout:Version : 2.02 2024-04-12T05:12:59.069 INFO:teuthology.orchestra.run.smithi067.stdout:Release : 129.el8 2024-04-12T05:12:59.069 INFO:teuthology.orchestra.run.smithi067.stdout:Architecture: x86_64 2024-04-12T05:12:59.069 INFO:teuthology.orchestra.run.smithi067.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-12T05:12:59.069 INFO:teuthology.orchestra.run.smithi067.stdout:Group : System Environment/Base 2024-04-12T05:12:59.069 INFO:teuthology.orchestra.run.smithi067.stdout:Size : 9265586 2024-04-12T05:12:59.070 INFO:teuthology.orchestra.run.smithi067.stdout:License : GPLv3+ 2024-04-12T05:12:59.070 INFO:teuthology.orchestra.run.smithi067.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-12T05:12:59.070 INFO:teuthology.orchestra.run.smithi067.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-12T05:12:59.070 INFO:teuthology.orchestra.run.smithi067.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-12T05:12:59.070 INFO:teuthology.orchestra.run.smithi067.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-12T05:12:59.070 INFO:teuthology.orchestra.run.smithi067.stdout:Relocations : (not relocatable) 2024-04-12T05:12:59.070 INFO:teuthology.orchestra.run.smithi067.stdout:Packager : CentOS BuildSystem 2024-04-12T05:12:59.070 INFO:teuthology.orchestra.run.smithi067.stdout:Vendor : CentOS 2024-04-12T05:12:59.070 INFO:teuthology.orchestra.run.smithi067.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-12T05:12:59.070 INFO:teuthology.orchestra.run.smithi067.stdout:Summary : Support tools for GRUB. 2024-04-12T05:12:59.070 INFO:teuthology.orchestra.run.smithi067.stdout:Description : 2024-04-12T05:12:59.070 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:12:59.070 INFO:teuthology.orchestra.run.smithi067.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-12T05:12:59.070 INFO:teuthology.orchestra.run.smithi067.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-12T05:12:59.071 INFO:teuthology.orchestra.run.smithi067.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-12T05:12:59.071 INFO:teuthology.orchestra.run.smithi067.stdout:hardware devices. 2024-04-12T05:12:59.071 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:12:59.071 INFO:teuthology.orchestra.run.smithi067.stdout:This subpackage provides tools for support of all platforms. 2024-04-12T05:12:59.072 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-12T05:12:59.072 INFO:teuthology.task.kernel:Updating grub on smithi067 to boot 4.18.0-552.el8.x86_64 2024-04-12T05:12:59.072 DEBUG:teuthology.orchestra.run.smithi067:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-12T05:12:59.271 INFO:teuthology.orchestra.run.smithi032.stderr:Generating grub configuration file ... 2024-04-12T05:13:00.414 INFO:teuthology.orchestra.run.smithi067.stderr:Generating grub configuration file ... 2024-04-12T05:13:00.966 INFO:teuthology.orchestra.run.smithi032.stderr:done 2024-04-12T05:13:01.299 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2024-04-12T05:13:01.339 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.cHMz1r0FRS 2024-04-12T05:13:01.339 DEBUG:teuthology.orchestra.run.smithi032:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.cHMz1r0FRS 2024-04-12T05:13:01.413 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.cHMz1r0FRS 2024-04-12T05:13:01.565 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.cHMz1r0FRS is 6KB 2024-04-12T05:13:01.590 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.cHMz1r0FRS 2024-04-12T05:13:01.609 DEBUG:teuthology.orchestra.run.smithi032:> sudo /bin/ls /boot/loader/entries || true 2024-04-12T05:13:01.691 INFO:teuthology.orchestra.run.smithi032.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-12T05:13:01.691 INFO:teuthology.orchestra.run.smithi032.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-12T05:13:01.691 INFO:teuthology.orchestra.run.smithi032.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-12T05:13:01.691 INFO:teuthology.orchestra.run.smithi032.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-12T05:13:01.692 DEBUG:teuthology.orchestra.run.smithi032:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-12T05:13:01.867 DEBUG:teuthology.orchestra.run.smithi032:> sudo shutdown -r now 2024-04-12T05:13:02.263 INFO:teuthology.orchestra.run.smithi067.stderr:done 2024-04-12T05:13:02.264 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-04-12T05:13:02.286 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.zHgCpwqMYd 2024-04-12T05:13:02.287 DEBUG:teuthology.orchestra.run.smithi067:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.zHgCpwqMYd 2024-04-12T05:13:02.361 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.zHgCpwqMYd 2024-04-12T05:13:02.645 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.zHgCpwqMYd is 6KB 2024-04-12T05:13:02.699 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.zHgCpwqMYd 2024-04-12T05:13:02.717 DEBUG:teuthology.orchestra.run.smithi067:> sudo /bin/ls /boot/loader/entries || true 2024-04-12T05:13:02.902 INFO:teuthology.orchestra.run.smithi067.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-12T05:13:02.902 INFO:teuthology.orchestra.run.smithi067.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-12T05:13:02.902 INFO:teuthology.orchestra.run.smithi067.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-12T05:13:02.902 INFO:teuthology.orchestra.run.smithi067.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-12T05:13:02.903 DEBUG:teuthology.orchestra.run.smithi067:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-12T05:13:03.058 DEBUG:teuthology.orchestra.run.smithi067:> sudo shutdown -r now 2024-04-12T05:13:31.898 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-04-12T05:13:31.899 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-12T05:13:31.900 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T05:13:33.061 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-04-12T05:13:33.062 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-12T05:13:33.062 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T05:14:07.518 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-04-12T05:14:16.527 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-12T05:14:16.528 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T05:14:31.910 DEBUG:teuthology.orchestra.remote:timed out 2024-04-12T05:14:40.921 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-12T05:14:40.922 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T05:14:41.253 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-04-12T05:14:41.539 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-12T05:14:41.539 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-552.el8.x86_64"... 2024-04-12T05:14:41.539 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-04-12T05:14:41.560 INFO:teuthology.orchestra.run.smithi032.stdout:4.18.0-552.el8.x86_64 2024-04-12T05:14:41.561 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-12T05:14:41.561 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-12T05:14:41.561 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-12T05:14:42.562 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-04-12T05:14:42.562 DEBUG:teuthology.orchestra.run.smithi032:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-12T05:14:42.634 INFO:teuthology.orchestra.run.smithi032.stdout:ttyS1 2024-04-12T05:14:42.647 DEBUG:teuthology.parallel:result is None 2024-04-12T05:14:48.261 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-04-12T05:14:48.578 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-12T05:14:48.578 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-552.el8.x86_64"... 2024-04-12T05:14:48.578 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-04-12T05:14:48.595 INFO:teuthology.orchestra.run.smithi067.stdout:4.18.0-552.el8.x86_64 2024-04-12T05:14:48.596 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-12T05:14:48.596 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-12T05:14:48.596 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-12T05:14:49.596 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-04-12T05:14:49.597 DEBUG:teuthology.orchestra.run.smithi067:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-12T05:14:49.696 INFO:teuthology.orchestra.run.smithi067.stdout:ttyS1 2024-04-12T05:14:49.710 DEBUG:teuthology.parallel:result is None 2024-04-12T05:14:49.710 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-12T05:14:49.721 INFO:teuthology.task.internal:Creating test directory... 2024-04-12T05:14:49.721 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-12T05:14:49.724 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-12T05:14:49.744 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-12T05:14:49.753 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-12T05:14:49.761 INFO:teuthology.task.internal:Creating archive directory... 2024-04-12T05:14:49.762 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-12T05:14:49.787 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-12T05:14:49.832 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-12T05:14:49.838 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-12T05:14:49.839 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-12T05:14:49.870 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-12T05:14:49.906 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T05:14:49.910 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T05:14:49.915 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T05:14:49.923 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T05:14:49.924 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-12T05:14:49.931 INFO:teuthology.task.internal:Configuring sudo... 2024-04-12T05:14:49.931 DEBUG:teuthology.orchestra.run.smithi032:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-12T05:14:49.968 DEBUG:teuthology.orchestra.run.smithi067:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-12T05:14:49.998 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-12T05:14:50.007 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-12T05:14:50.007 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-12T05:14:50.042 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-12T05:14:50.063 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-12T05:14:50.146 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-12T05:14:50.210 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-12T05:14:50.255 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-12T05:14:50.299 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-12T05:14:50.299 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-12T05:14:50.369 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-12T05:14:50.414 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-12T05:14:50.473 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-12T05:14:50.559 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-12T05:14:50.604 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-12T05:14:50.604 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-12T05:14:50.673 DEBUG:teuthology.orchestra.run.smithi032:> sudo service rsyslog restart 2024-04-12T05:14:50.676 DEBUG:teuthology.orchestra.run.smithi067:> sudo service rsyslog restart 2024-04-12T05:14:50.737 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-12T05:14:50.771 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-12T05:14:51.108 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-12T05:14:51.118 INFO:teuthology.task.internal:Starting timer... 2024-04-12T05:14:51.118 INFO:teuthology.run_tasks:Running task pcp... 2024-04-12T05:14:51.131 INFO:teuthology.run_tasks:Running task selinux... 2024-04-12T05:14:51.158 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-12T05:14:51.158 DEBUG:teuthology.orchestra.run.smithi032:> sudo service auditd rotate 2024-04-12T05:14:51.257 INFO:teuthology.orchestra.run.smithi032.stdout:Rotating logs: 2024-04-12T05:14:51.259 DEBUG:teuthology.orchestra.run.smithi067:> sudo service auditd rotate 2024-04-12T05:14:51.341 INFO:teuthology.orchestra.run.smithi067.stdout:Rotating logs: 2024-04-12T05:14:51.342 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-12T05:14:51.342 DEBUG:teuthology.orchestra.run.smithi032:> /usr/sbin/getenforce 2024-04-12T05:14:51.392 INFO:teuthology.orchestra.run.smithi032.stdout:Permissive 2024-04-12T05:14:51.395 DEBUG:teuthology.orchestra.run.smithi067:> /usr/sbin/getenforce 2024-04-12T05:14:51.435 INFO:teuthology.orchestra.run.smithi067.stdout:Permissive 2024-04-12T05:14:51.435 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi032.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi067.front.sepia.ceph.com': 'permissive'} 2024-04-12T05:14:51.435 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-12T05:14:51.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T05:14:51.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-12T05:14:51.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T05:14:51.506 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-12T05:14:51.506 DEBUG:teuthology.orchestra.run.smithi032:> sudo /usr/sbin/setenforce permissive 2024-04-12T05:14:51.541 DEBUG:teuthology.orchestra.run.smithi067:> sudo /usr/sbin/setenforce permissive 2024-04-12T05:14:51.584 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-12T05:14:51.594 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-12T05:14:51.671 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-12T05:14:51.672 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi032.front.sepia.ceph.com,smithi067.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-12T05:21:31.676 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi032.front.sepia.ceph.com'), Remote(name='ubuntu@smithi067.front.sepia.ceph.com')] 2024-04-12T05:21:31.677 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-12T05:21:31.678 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T05:21:31.753 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-04-12T05:21:31.821 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-12T05:21:31.821 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-12T05:21:31.822 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T05:21:31.894 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-04-12T05:21:31.964 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-12T05:21:31.965 INFO:teuthology.run_tasks:Running task clock... 2024-04-12T05:21:31.976 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-12T05:21:31.976 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-12T05:21:31.976 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-12T05:21:31.979 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-12T05:21:31.979 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-12T05:21:32.012 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-12T05:21:32.027 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-12T05:21:32.038 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-12T05:21:32.052 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-12T05:21:32.097 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: ntpd: command not found 2024-04-12T05:21:32.109 INFO:teuthology.orchestra.run.smithi032.stdout:506 Cannot talk to daemon 2024-04-12T05:21:32.123 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-12T05:21:32.133 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: ntpd: command not found 2024-04-12T05:21:32.137 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-12T05:21:32.145 INFO:teuthology.orchestra.run.smithi067.stdout:506 Cannot talk to daemon 2024-04-12T05:21:32.160 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-12T05:21:32.173 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-12T05:21:32.196 INFO:teuthology.orchestra.run.smithi032.stderr:bash: ntpq: command not found 2024-04-12T05:21:32.201 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-12T05:21:32.201 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2024-04-12T05:21:32.201 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T05:21:32.201 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T05:21:32.201 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T05:21:32.201 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T05:21:32.240 INFO:teuthology.orchestra.run.smithi067.stderr:bash: ntpq: command not found 2024-04-12T05:21:32.245 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-12T05:21:32.245 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-04-12T05:21:32.245 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T05:21:32.245 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T05:21:32.245 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T05:21:32.245 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T05:21:32.246 INFO:teuthology.run_tasks:Running task pexec... 2024-04-12T05:21:32.255 INFO:teuthology.task.pexec:Executing custom commands... 2024-04-12T05:21:32.255 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi032.front.sepia.ceph.com 2024-04-12T05:21:32.256 DEBUG:teuthology.orchestra.run.smithi032:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-04-12T05:21:32.256 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi067.front.sepia.ceph.com 2024-04-12T05:21:32.256 DEBUG:teuthology.orchestra.run.smithi067:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-04-12T05:21:33.045 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:55 ago on Fri 12 Apr 2024 05:19:37 AM UTC. 2024-04-12T05:21:33.074 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:05 ago on Fri 12 Apr 2024 05:19:27 AM UTC. 2024-04-12T05:21:35.335 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-12T05:21:35.337 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:21:35.337 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-04-12T05:21:35.337 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:21:35.337 INFO:teuthology.orchestra.run.smithi032.stdout:Resetting modules: 2024-04-12T05:21:35.337 INFO:teuthology.orchestra.run.smithi032.stdout: container-tools 2024-04-12T05:21:35.337 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:21:35.337 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-12T05:21:35.337 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:21:35.337 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:21:35.360 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-12T05:21:35.362 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:21:35.362 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-04-12T05:21:35.362 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:21:35.362 INFO:teuthology.orchestra.run.smithi067.stdout:Resetting modules: 2024-04-12T05:21:35.362 INFO:teuthology.orchestra.run.smithi067.stdout: container-tools 2024-04-12T05:21:35.362 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:21:35.362 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-12T05:21:35.362 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:21:35.362 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:21:35.594 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-12T05:21:35.623 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-12T05:21:36.440 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:59 ago on Fri 12 Apr 2024 05:19:37 AM UTC. 2024-04-12T05:21:36.481 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:09 ago on Fri 12 Apr 2024 05:19:27 AM UTC. 2024-04-12T05:21:38.750 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-12T05:21:38.791 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:21:38.792 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-12T05:21:38.792 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:21:38.792 INFO:teuthology.orchestra.run.smithi032.stdout:Installing group/module packages: 2024-04-12T05:21:38.792 INFO:teuthology.orchestra.run.smithi032.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-04-12T05:21:38.792 INFO:teuthology.orchestra.run.smithi032.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-04-12T05:21:38.792 INFO:teuthology.orchestra.run.smithi032.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-04-12T05:21:38.792 INFO:teuthology.orchestra.run.smithi032.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-04-12T05:21:38.792 INFO:teuthology.orchestra.run.smithi032.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-04-12T05:21:38.792 INFO:teuthology.orchestra.run.smithi032.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-04-12T05:21:38.792 INFO:teuthology.orchestra.run.smithi032.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-04-12T05:21:38.792 INFO:teuthology.orchestra.run.smithi032.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-04-12T05:21:38.792 INFO:teuthology.orchestra.run.smithi032.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-04-12T05:21:38.792 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-12T05:21:38.793 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-04-12T05:21:38.793 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-04-12T05:21:38.793 INFO:teuthology.orchestra.run.smithi032.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-04-12T05:21:38.793 INFO:teuthology.orchestra.run.smithi032.stdout:Installing weak dependencies: 2024-04-12T05:21:38.793 INFO:teuthology.orchestra.run.smithi032.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-04-12T05:21:38.793 INFO:teuthology.orchestra.run.smithi032.stdout:Installing module profiles: 2024-04-12T05:21:38.793 INFO:teuthology.orchestra.run.smithi032.stdout: container-tools/common 2024-04-12T05:21:38.793 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:21:38.793 INFO:teuthology.orchestra.run.smithi032.stdout:Enabling module streams: 2024-04-12T05:21:38.793 INFO:teuthology.orchestra.run.smithi032.stdout: container-tools rhel8 2024-04-12T05:21:38.793 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:21:38.793 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-12T05:21:38.793 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:21:38.793 INFO:teuthology.orchestra.run.smithi032.stdout:Install 13 Packages 2024-04-12T05:21:38.793 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:21:38.794 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 27 M 2024-04-12T05:21:38.794 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 91 M 2024-04-12T05:21:38.794 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-12T05:21:38.807 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-12T05:21:38.848 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:21:38.848 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-04-12T05:21:38.848 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:21:38.848 INFO:teuthology.orchestra.run.smithi067.stdout:Installing group/module packages: 2024-04-12T05:21:38.848 INFO:teuthology.orchestra.run.smithi067.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-04-12T05:21:38.848 INFO:teuthology.orchestra.run.smithi067.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-04-12T05:21:38.848 INFO:teuthology.orchestra.run.smithi067.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-04-12T05:21:38.848 INFO:teuthology.orchestra.run.smithi067.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-04-12T05:21:38.848 INFO:teuthology.orchestra.run.smithi067.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-04-12T05:21:38.848 INFO:teuthology.orchestra.run.smithi067.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-04-12T05:21:38.848 INFO:teuthology.orchestra.run.smithi067.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-04-12T05:21:38.848 INFO:teuthology.orchestra.run.smithi067.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-04-12T05:21:38.848 INFO:teuthology.orchestra.run.smithi067.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-04-12T05:21:38.849 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-04-12T05:21:38.849 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-04-12T05:21:38.849 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-04-12T05:21:38.849 INFO:teuthology.orchestra.run.smithi067.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-04-12T05:21:38.849 INFO:teuthology.orchestra.run.smithi067.stdout:Installing weak dependencies: 2024-04-12T05:21:38.849 INFO:teuthology.orchestra.run.smithi067.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-04-12T05:21:38.849 INFO:teuthology.orchestra.run.smithi067.stdout:Installing module profiles: 2024-04-12T05:21:38.849 INFO:teuthology.orchestra.run.smithi067.stdout: container-tools/common 2024-04-12T05:21:38.849 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:21:38.849 INFO:teuthology.orchestra.run.smithi067.stdout:Enabling module streams: 2024-04-12T05:21:38.849 INFO:teuthology.orchestra.run.smithi067.stdout: container-tools rhel8 2024-04-12T05:21:38.849 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:21:38.849 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-12T05:21:38.849 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:21:38.849 INFO:teuthology.orchestra.run.smithi067.stdout:Install 13 Packages 2024-04-12T05:21:38.850 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:21:38.850 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 27 M 2024-04-12T05:21:38.850 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 91 M 2024-04-12T05:21:38.850 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-12T05:21:40.166 INFO:teuthology.orchestra.run.smithi032.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 524 kB/s | 689 kB 00:01 2024-04-12T05:21:40.234 INFO:teuthology.orchestra.run.smithi067.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 520 kB/s | 689 kB 00:01 2024-04-12T05:21:40.483 INFO:teuthology.orchestra.run.smithi032.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 120 kB/s | 38 kB 00:00 2024-04-12T05:21:40.551 INFO:teuthology.orchestra.run.smithi067.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 120 kB/s | 38 kB 00:00 2024-04-12T05:21:40.726 INFO:teuthology.orchestra.run.smithi067.stdout:(3/13): aardvark-dns-1.10.0-1.module_el8+871+55 567 kB/s | 1.0 MB 00:01 2024-04-12T05:21:40.733 INFO:teuthology.orchestra.run.smithi032.stdout:(3/13): aardvark-dns-1.10.0-1.module_el8+871+55 548 kB/s | 1.0 MB 00:01 2024-04-12T05:21:41.134 INFO:teuthology.orchestra.run.smithi032.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 396 kB/s | 258 kB 00:00 2024-04-12T05:21:41.310 INFO:teuthology.orchestra.run.smithi067.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 340 kB/s | 258 kB 00:00 2024-04-12T05:21:41.684 INFO:teuthology.orchestra.run.smithi032.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 328 kB/s | 180 kB 00:00 2024-04-12T05:21:41.894 INFO:teuthology.orchestra.run.smithi067.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 309 kB/s | 180 kB 00:00 2024-04-12T05:21:41.992 INFO:teuthology.orchestra.run.smithi032.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 80 kB/s | 25 kB 00:00 2024-04-12T05:21:42.186 INFO:teuthology.orchestra.run.smithi067.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 85 kB/s | 25 kB 00:00 2024-04-12T05:21:42.434 INFO:teuthology.orchestra.run.smithi032.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 212 kB/s | 94 kB 00:00 2024-04-12T05:21:42.644 INFO:teuthology.orchestra.run.smithi067.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 205 kB/s | 94 kB 00:00 2024-04-12T05:21:46.651 INFO:teuthology.orchestra.run.smithi032.stdout:(8/13): netavark-1.10.3-1.module_el8+951+32019c 710 kB/s | 4.1 MB 00:05 2024-04-12T05:21:46.703 INFO:teuthology.orchestra.run.smithi067.stdout:(8/13): netavark-1.10.3-1.module_el8+951+32019c 703 kB/s | 4.1 MB 00:05 2024-04-12T05:21:50.676 INFO:teuthology.orchestra.run.smithi032.stdout:(9/13): toolbox-0.0.99.5-2.module_el8+971+3d3df 643 kB/s | 2.5 MB 00:04 2024-04-12T05:21:50.778 INFO:teuthology.orchestra.run.smithi067.stdout:(9/13): toolbox-0.0.99.5-2.module_el8+971+3d3df 635 kB/s | 2.5 MB 00:04 2024-04-12T05:21:51.060 INFO:teuthology.orchestra.run.smithi032.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 137 kB/s | 53 kB 00:00 2024-04-12T05:21:51.220 INFO:teuthology.orchestra.run.smithi067.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 119 kB/s | 53 kB 00:00 2024-04-12T05:21:51.518 INFO:teuthology.orchestra.run.smithi032.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 89 kB/s | 41 kB 00:00 2024-04-12T05:21:51.662 INFO:teuthology.orchestra.run.smithi067.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 93 kB/s | 41 kB 00:00 2024-04-12T05:21:52.568 INFO:teuthology.orchestra.run.smithi032.stdout:(12/13): buildah-1.33.6-2.module_el8+976+113607 720 kB/s | 9.6 MB 00:13 2024-04-12T05:21:54.012 INFO:teuthology.orchestra.run.smithi067.stdout:(12/13): buildah-1.33.6-2.module_el8+976+113607 654 kB/s | 9.6 MB 00:15 2024-04-12T05:21:55.444 INFO:teuthology.orchestra.run.smithi032.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 693 kB/s | 8.8 MB 00:13 2024-04-12T05:21:55.445 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:21:55.445 INFO:teuthology.orchestra.run.smithi032.stdout:Total 1.6 MB/s | 27 MB 00:16 2024-04-12T05:21:55.554 INFO:teuthology.orchestra.run.smithi067.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 699 kB/s | 8.8 MB 00:12 2024-04-12T05:21:55.555 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:21:55.555 INFO:teuthology.orchestra.run.smithi067.stdout:Total 1.6 MB/s | 27 MB 00:16 2024-04-12T05:21:55.725 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-12T05:21:55.747 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-12T05:21:55.748 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-12T05:21:55.839 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-12T05:21:55.861 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-12T05:21:55.861 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-12T05:21:56.027 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-12T05:21:56.030 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-12T05:21:56.138 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-12T05:21:56.142 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-12T05:21:56.729 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-12T05:21:56.841 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-12T05:21:57.714 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-04-12T05:21:57.821 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-04-12T05:21:57.909 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-04-12T05:21:58.069 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-04-12T05:21:58.081 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-04-12T05:21:58.233 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-04-12T05:21:58.267 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-04-12T05:21:58.438 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-04-12T05:21:58.485 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-04-12T05:21:58.698 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-04-12T05:21:59.061 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-04-12T05:21:59.267 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-04-12T05:22:00.112 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-04-12T05:22:00.299 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-04-12T05:22:00.331 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-04-12T05:22:00.532 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-04-12T05:22:00.571 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-04-12T05:22:00.756 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-04-12T05:22:01.154 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-04-12T05:22:01.230 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-04-12T05:22:01.347 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-12T05:22:01.440 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-12T05:22:01.584 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-12T05:22:01.777 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-12T05:22:01.814 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-12T05:22:02.007 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-12T05:22:02.244 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-12T05:22:02.244 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-04-12T05:22:02.244 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-04-12T05:22:02.244 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-04-12T05:22:02.244 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-04-12T05:22:02.244 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-04-12T05:22:02.244 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-04-12T05:22:02.244 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-04-12T05:22:02.245 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-04-12T05:22:02.245 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-04-12T05:22:02.245 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-04-12T05:22:02.245 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-12T05:22:02.245 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-12T05:22:02.498 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-12T05:22:02.498 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-04-12T05:22:02.498 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-04-12T05:22:02.498 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-04-12T05:22:02.498 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-04-12T05:22:02.498 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-04-12T05:22:02.498 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-04-12T05:22:02.499 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-04-12T05:22:02.499 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-04-12T05:22:02.499 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-04-12T05:22:02.499 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-04-12T05:22:02.499 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-12T05:22:02.499 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-12T05:22:02.737 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-04-12T05:22:02.738 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:22:02.738 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-12T05:22:02.738 INFO:teuthology.orchestra.run.smithi032.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-04-12T05:22:02.738 INFO:teuthology.orchestra.run.smithi032.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-04-12T05:22:02.738 INFO:teuthology.orchestra.run.smithi032.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-04-12T05:22:02.738 INFO:teuthology.orchestra.run.smithi032.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-04-12T05:22:02.738 INFO:teuthology.orchestra.run.smithi032.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-04-12T05:22:02.738 INFO:teuthology.orchestra.run.smithi032.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-04-12T05:22:02.738 INFO:teuthology.orchestra.run.smithi032.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-04-12T05:22:02.738 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-04-12T05:22:02.738 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-04-12T05:22:02.738 INFO:teuthology.orchestra.run.smithi032.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-04-12T05:22:02.738 INFO:teuthology.orchestra.run.smithi032.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-04-12T05:22:02.738 INFO:teuthology.orchestra.run.smithi032.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-04-12T05:22:02.739 INFO:teuthology.orchestra.run.smithi032.stdout: yajl-2.1.0-12.el8.x86_64 2024-04-12T05:22:02.739 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:22:02.739 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-12T05:22:03.166 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-04-12T05:22:03.167 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:22:03.167 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-12T05:22:03.167 INFO:teuthology.orchestra.run.smithi067.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-04-12T05:22:03.167 INFO:teuthology.orchestra.run.smithi067.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-04-12T05:22:03.167 INFO:teuthology.orchestra.run.smithi067.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-04-12T05:22:03.167 INFO:teuthology.orchestra.run.smithi067.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-04-12T05:22:03.167 INFO:teuthology.orchestra.run.smithi067.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-04-12T05:22:03.167 INFO:teuthology.orchestra.run.smithi067.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-04-12T05:22:03.167 INFO:teuthology.orchestra.run.smithi067.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-04-12T05:22:03.167 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-04-12T05:22:03.167 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-04-12T05:22:03.168 INFO:teuthology.orchestra.run.smithi067.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-04-12T05:22:03.168 INFO:teuthology.orchestra.run.smithi067.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-04-12T05:22:03.168 INFO:teuthology.orchestra.run.smithi067.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-04-12T05:22:03.168 INFO:teuthology.orchestra.run.smithi067.stdout: yajl-2.1.0-12.el8.x86_64 2024-04-12T05:22:03.168 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:22:03.168 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-12T05:22:03.288 DEBUG:teuthology.parallel:result is None 2024-04-12T05:22:03.490 DEBUG:teuthology.parallel:result is None 2024-04-12T05:22:03.490 INFO:teuthology.run_tasks:Running task install... 2024-04-12T05:22:03.500 DEBUG:teuthology.task.install:project ceph 2024-04-12T05:22:03.500 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'}} 2024-04-12T05:22:03.500 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'} 2024-04-12T05:22:03.500 INFO:teuthology.task.install:Using flavor: default 2024-04-12T05:22:03.509 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-12T05:22:03.509 INFO:teuthology.task.install:extra packages: [] 2024-04-12T05:22:03.509 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'tag': None, 'wait_for_package': False} 2024-04-12T05:22:03.509 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-12T05:22:03.509 INFO:teuthology.packaging:ref: None 2024-04-12T05:22:03.509 INFO:teuthology.packaging:tag: None 2024-04-12T05:22:03.510 INFO:teuthology.packaging:branch: quincy 2024-04-12T05:22:03.510 INFO:teuthology.packaging:sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:22:03.510 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-04-12T05:22:03.512 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'tag': None, 'wait_for_package': False} 2024-04-12T05:22:03.512 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-12T05:22:03.512 INFO:teuthology.packaging:ref: None 2024-04-12T05:22:03.512 INFO:teuthology.packaging:tag: None 2024-04-12T05:22:03.512 INFO:teuthology.packaging:branch: quincy 2024-04-12T05:22:03.512 INFO:teuthology.packaging:sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:22:03.512 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-04-12T05:22:03.771 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/quincy/d3b5060a17449edfae9385cdf792afec109fca5d/centos/8/flavors/default/ 2024-04-12T05:22:03.772 INFO:teuthology.task.install.rpm:Package version is 17.2.7-827.gd3b5060a 2024-04-12T05:22:03.831 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/quincy/d3b5060a17449edfae9385cdf792afec109fca5d/centos/8/flavors/default/ 2024-04-12T05:22:03.831 INFO:teuthology.task.install.rpm:Package version is 17.2.7-827.gd3b5060a 2024-04-12T05:22:03.941 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/quincy/d3b5060a17449edfae9385cdf792afec109fca5d/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/d3b5060a17449edfae9385cdf792afec109fca5d/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/d3b5060a17449edfae9385cdf792afec109fca5d/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-12T05:22:03.941 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-12T05:22:03.941 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-12T05:22:03.945 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/quincy/d3b5060a17449edfae9385cdf792afec109fca5d/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/d3b5060a17449edfae9385cdf792afec109fca5d/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/d3b5060a17449edfae9385cdf792afec109fca5d/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-12T05:22:03.945 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-12T05:22:03.945 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-12T05:22:03.970 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-12T05:22:03.970 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-12T05:22:03.970 INFO:teuthology.packaging:ref: None 2024-04-12T05:22:03.971 INFO:teuthology.packaging:tag: None 2024-04-12T05:22:03.971 INFO:teuthology.packaging:branch: quincy 2024-04-12T05:22:03.971 INFO:teuthology.packaging:sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:22:03.971 DEBUG:teuthology.orchestra.run.smithi067:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-12T05:22:03.980 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-12T05:22:03.981 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-12T05:22:03.981 INFO:teuthology.packaging:ref: None 2024-04-12T05:22:03.981 INFO:teuthology.packaging:tag: None 2024-04-12T05:22:03.981 INFO:teuthology.packaging:branch: quincy 2024-04-12T05:22:03.981 INFO:teuthology.packaging:sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:22:03.981 DEBUG:teuthology.orchestra.run.smithi032:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-12T05:22:04.039 DEBUG:teuthology.orchestra.run.smithi067:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-12T05:22:04.054 DEBUG:teuthology.orchestra.run.smithi032:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-12T05:22:04.119 DEBUG:teuthology.orchestra.run.smithi067:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-12T05:22:04.134 DEBUG:teuthology.orchestra.run.smithi032:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-12T05:22:04.186 INFO:teuthology.orchestra.run.smithi067.stdout:check_obsoletes = 1 2024-04-12T05:22:04.187 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum clean all 2024-04-12T05:22:04.204 INFO:teuthology.orchestra.run.smithi032.stdout:check_obsoletes = 1 2024-04-12T05:22:04.277 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum clean all 2024-04-12T05:22:04.554 INFO:teuthology.orchestra.run.smithi032.stdout:76 files removed 2024-04-12T05:22:04.568 INFO:teuthology.orchestra.run.smithi067.stdout:76 files removed 2024-04-12T05:22:04.583 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-radosgw 2024-04-12T05:22:04.596 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-radosgw 2024-04-12T05:22:05.445 INFO:teuthology.orchestra.run.smithi067.stdout:ceph packages for x86_64 187 kB/s | 79 kB 00:00 2024-04-12T05:22:05.461 INFO:teuthology.orchestra.run.smithi032.stdout:ceph packages for x86_64 174 kB/s | 79 kB 00:00 2024-04-12T05:22:05.761 INFO:teuthology.orchestra.run.smithi032.stdout:ceph noarch packages 35 kB/s | 9.1 kB 00:00 2024-04-12T05:22:05.771 INFO:teuthology.orchestra.run.smithi067.stdout:ceph noarch packages 32 kB/s | 9.1 kB 00:00 2024-04-12T05:22:06.112 INFO:teuthology.orchestra.run.smithi032.stdout:ceph source packages 4.5 kB/s | 1.5 kB 00:00 2024-04-12T05:22:06.155 INFO:teuthology.orchestra.run.smithi067.stdout:ceph source packages 4.1 kB/s | 1.5 kB 00:00 2024-04-12T05:22:17.263 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 8 - AppStream 2.5 MB/s | 28 MB 00:11 2024-04-12T05:22:24.313 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 8 - BaseOS 5.9 MB/s | 10 MB 00:01 2024-04-12T05:22:26.397 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 8 - Extras 45 kB/s | 18 kB 00:00 2024-04-12T05:22:26.906 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 8 - Extras common packages 20 kB/s | 7.7 kB 00:00 2024-04-12T05:22:29.741 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 8 - PowerTools 1.8 MB/s | 5.0 MB 00:02 2024-04-12T05:22:30.851 INFO:teuthology.orchestra.run.smithi067.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2024-04-12T05:22:32.604 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 9.9 MB/s | 16 MB 00:01 2024-04-12T05:22:36.677 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 312 kB/s | 24 kB 00:00 2024-04-12T05:22:40.563 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-12T05:22:40.565 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:22:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-04-12T05:22:40.566 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:22:40.566 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-12T05:22:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 17 M 2024-04-12T05:22:40.566 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrading: 2024-04-12T05:22:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: librados2 x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 3.5 M 2024-04-12T05:22:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1 x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 3.6 M 2024-04-12T05:22:40.566 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-04-12T05:22:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 5.3 M 2024-04-12T05:22:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 24 M 2024-04-12T05:22:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 24 k 2024-04-12T05:22:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-12T05:22:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2 x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 798 k 2024-04-12T05:22:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-12T05:22:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-04-12T05:22:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1 x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 505 k 2024-04-12T05:22:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-04-12T05:22:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2 x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 5.6 M 2024-04-12T05:22:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-12T05:22:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-04-12T05:22:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 45 k 2024-04-12T05:22:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 117 k 2024-04-12T05:22:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 190 k 2024-04-12T05:22:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-04-12T05:22:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 370 k 2024-04-12T05:22:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 359 k 2024-04-12T05:22:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 107 k 2024-04-12T05:22:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-12T05:22:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:22:40.568 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-12T05:22:40.568 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:22:40.568 INFO:teuthology.orchestra.run.smithi067.stdout:Install 21 Packages 2024-04-12T05:22:40.568 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrade 2 Packages 2024-04-12T05:22:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:22:40.568 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 64 M 2024-04-12T05:22:40.568 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-12T05:22:41.727 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 8 - AppStream 802 kB/s | 28 MB 00:35 2024-04-12T05:22:41.845 INFO:teuthology.orchestra.run.smithi067.stdout:(1/23): ceph-base-17.2.7-827.gd3b5060a.el8.x86_ 4.8 MB/s | 5.3 MB 00:01 2024-04-12T05:22:41.945 INFO:teuthology.orchestra.run.smithi067.stdout:(2/23): ceph-selinux-17.2.7-827.gd3b5060a.el8.x 243 kB/s | 24 kB 00:00 2024-04-12T05:22:42.113 INFO:teuthology.orchestra.run.smithi067.stdout:(3/23): ceph-radosgw-17.2.7-827.gd3b5060a.el8.x 12 MB/s | 17 MB 00:01 2024-04-12T05:22:42.146 INFO:teuthology.orchestra.run.smithi067.stdout:(4/23): libcephfs2-17.2.7-827.gd3b5060a.el8.x86 3.9 MB/s | 798 kB 00:00 2024-04-12T05:22:42.180 INFO:teuthology.orchestra.run.smithi067.stdout:(5/23): libradosstriper1-17.2.7-827.gd3b5060a.e 7.4 MB/s | 505 kB 00:00 2024-04-12T05:22:42.222 INFO:teuthology.orchestra.run.smithi067.stdout:(6/23): python3-ceph-argparse-17.2.7-827.gd3b50 1.0 MB/s | 45 kB 00:00 2024-04-12T05:22:42.280 INFO:teuthology.orchestra.run.smithi067.stdout:(7/23): python3-ceph-common-17.2.7-827.gd3b5060 2.0 MB/s | 117 kB 00:00 2024-04-12T05:22:42.348 INFO:teuthology.orchestra.run.smithi067.stdout:(8/23): python3-cephfs-17.2.7-827.gd3b5060a.el8 2.8 MB/s | 190 kB 00:00 2024-04-12T05:22:42.423 INFO:teuthology.orchestra.run.smithi067.stdout:(9/23): python3-rados-17.2.7-827.gd3b5060a.el8. 4.8 MB/s | 370 kB 00:00 2024-04-12T05:22:42.532 INFO:teuthology.orchestra.run.smithi067.stdout:(10/23): python3-rbd-17.2.7-827.gd3b5060a.el8.x 3.2 MB/s | 359 kB 00:00 2024-04-12T05:22:42.607 INFO:teuthology.orchestra.run.smithi067.stdout:(11/23): python3-rgw-17.2.7-827.gd3b5060a.el8.x 1.4 MB/s | 107 kB 00:00 2024-04-12T05:22:42.732 INFO:teuthology.orchestra.run.smithi067.stdout:(12/23): librgw2-17.2.7-827.gd3b5060a.el8.x86_6 9.5 MB/s | 5.6 MB 00:00 2024-04-12T05:22:43.174 INFO:teuthology.orchestra.run.smithi067.stdout:(13/23): ceph-common-17.2.7-827.gd3b5060a.el8.x 9.7 MB/s | 24 MB 00:02 2024-04-12T05:22:43.374 INFO:teuthology.orchestra.run.smithi067.stdout:(14/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 404 kB/s | 259 kB 00:00 2024-04-12T05:22:43.525 INFO:teuthology.orchestra.run.smithi067.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 125 kB/s | 44 kB 00:00 2024-04-12T05:22:43.733 INFO:teuthology.orchestra.run.smithi067.stdout:(16/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 131 kB/s | 47 kB 00:00 2024-04-12T05:22:43.917 INFO:teuthology.orchestra.run.smithi067.stdout:(17/23): librdkafka-1.6.1-1.el8.x86_64.rpm 517 kB/s | 677 kB 00:01 2024-04-12T05:22:44.076 INFO:teuthology.orchestra.run.smithi067.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 172 kB/s | 59 kB 00:00 2024-04-12T05:22:44.101 INFO:teuthology.orchestra.run.smithi067.stdout:(19/23): gperftools-libs-2.7-9.el8.x86_64.rpm 532 kB/s | 306 kB 00:00 2024-04-12T05:22:44.308 INFO:teuthology.orchestra.run.smithi067.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 192 kB/s | 75 kB 00:00 2024-04-12T05:22:44.418 INFO:teuthology.orchestra.run.smithi067.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 5.1 MB/s | 1.7 MB 00:00 2024-04-12T05:22:44.585 INFO:teuthology.orchestra.run.smithi067.stdout:(22/23): librados2-17.2.7-827.gd3b5060a.el8.x86 7.2 MB/s | 3.5 MB 00:00 2024-04-12T05:22:44.960 INFO:teuthology.orchestra.run.smithi067.stdout:(23/23): librbd1-17.2.7-827.gd3b5060a.el8.x86_6 5.5 MB/s | 3.6 MB 00:00 2024-04-12T05:22:44.963 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:22:44.963 INFO:teuthology.orchestra.run.smithi067.stdout:Total 15 MB/s | 64 MB 00:04 2024-04-12T05:22:45.007 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-12T05:22:45.057 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-12T05:22:45.057 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-12T05:22:45.449 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-12T05:22:45.452 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-12T05:22:46.538 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-12T05:22:46.772 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-04-12T05:22:46.952 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-12T05:22:47.339 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-12T05:22:47.510 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librados2-2:17.2.7-827.gd3b5060a.el8.x86_64 3/25 2024-04-12T05:22:47.603 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:17.2.7-827.gd3b5060a.el8.x86_64 3/25 2024-04-12T05:22:47.856 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rados-2:17.2.7-827.gd3b5060a.el8.x86_64 4/25 2024-04-12T05:22:48.027 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs2-2:17.2.7-827.gd3b5060a.el8.x86_64 5/25 2024-04-12T05:22:48.443 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs2-2:17.2.7-827.gd3b5060a.el8.x86_64 5/25 2024-04-12T05:22:48.610 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librbd1-2:17.2.7-827.gd3b5060a.el8.x86_64 6/25 2024-04-12T05:22:48.836 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:17.2.7-827.gd3b5060a.el8.x86_64 6/25 2024-04-12T05:22:49.032 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2024-04-12T05:22:49.240 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/25 2024-04-12T05:22:49.919 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2024-04-12T05:22:50.053 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 8 - BaseOS 3.4 MB/s | 10 MB 00:03 2024-04-12T05:22:50.077 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librgw2-2:17.2.7-827.gd3b5060a.el8.x86_64 10/25 2024-04-12T05:22:50.170 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librgw2-2:17.2.7-827.gd3b5060a.el8.x86_64 10/25 2024-04-12T05:22:50.453 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-argparse-2:17.2.7-827.gd3b5060a.el8.x 11/25 2024-04-12T05:22:50.672 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cephfs-2:17.2.7-827.gd3b5060a.el8.x86_64 12/25 2024-04-12T05:22:50.914 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rgw-2:17.2.7-827.gd3b5060a.el8.x86_64 13/25 2024-04-12T05:22:51.125 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rbd-2:17.2.7-827.gd3b5060a.el8.x86_64 14/25 2024-04-12T05:22:51.252 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libradosstriper1-2:17.2.7-827.gd3b5060a.el8.x86_64 15/25 2024-04-12T05:22:51.346 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libradosstriper1-2:17.2.7-827.gd3b5060a.el8.x86_64 15/25 2024-04-12T05:22:51.556 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-04-12T05:22:51.719 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-04-12T05:22:51.934 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-04-12T05:22:52.135 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-common-2:17.2.7-827.gd3b5060a.el8.x86 19/25 2024-04-12T05:22:52.445 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 8 - Extras 28 kB/s | 18 kB 00:00 2024-04-12T05:22:52.904 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 8 - Extras common packages 23 kB/s | 7.7 kB 00:00 2024-04-12T05:22:54.981 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:17.2.7-827.gd3b5060a.el8.x86_64 20/25 2024-04-12T05:22:55.196 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 8 - PowerTools 2.3 MB/s | 5.0 MB 00:02 2024-04-12T05:22:55.502 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-common-2:17.2.7-827.gd3b5060a.el8.x86_64 20/25 2024-04-12T05:22:56.128 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:17.2.7-827.gd3b5060a.el8.x86_64 20/25 2024-04-12T05:22:56.379 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-base-2:17.2.7-827.gd3b5060a.el8.x86_64 21/25 2024-04-12T05:22:56.476 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-base-2:17.2.7-827.gd3b5060a.el8.x86_64 21/25 2024-04-12T05:22:56.623 INFO:teuthology.orchestra.run.smithi032.stdout:Copr repo for python3-asyncssh owned by ceph 7.9 kB/s | 3.5 kB 00:00 2024-04-12T05:22:56.651 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-selinux-2:17.2.7-827.gd3b5060a.el8.x86_64 22/25 2024-04-12T05:22:58.859 INFO:teuthology.orchestra.run.smithi032.stdout:Extra Packages for Enterprise Linux 7.6 MB/s | 16 MB 00:02 2024-04-12T05:23:02.907 INFO:teuthology.orchestra.run.smithi032.stdout:lab-extras 404 kB/s | 24 kB 00:00 2024-04-12T05:23:06.718 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-12T05:23:06.720 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:23:06.720 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-12T05:23:06.720 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:23:06.720 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-12T05:23:06.721 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 17 M 2024-04-12T05:23:06.721 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrading: 2024-04-12T05:23:06.721 INFO:teuthology.orchestra.run.smithi032.stdout: librados2 x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 3.5 M 2024-04-12T05:23:06.721 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1 x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 3.6 M 2024-04-12T05:23:06.721 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-12T05:23:06.721 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 5.3 M 2024-04-12T05:23:06.721 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 24 M 2024-04-12T05:23:06.721 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-selinux x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 24 k 2024-04-12T05:23:06.721 INFO:teuthology.orchestra.run.smithi032.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-12T05:23:06.721 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2 x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 798 k 2024-04-12T05:23:06.721 INFO:teuthology.orchestra.run.smithi032.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-12T05:23:06.721 INFO:teuthology.orchestra.run.smithi032.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-04-12T05:23:06.721 INFO:teuthology.orchestra.run.smithi032.stdout: libradosstriper1 x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 505 k 2024-04-12T05:23:06.722 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-04-12T05:23:06.722 INFO:teuthology.orchestra.run.smithi032.stdout: librgw2 x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 5.6 M 2024-04-12T05:23:06.722 INFO:teuthology.orchestra.run.smithi032.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-12T05:23:06.722 INFO:teuthology.orchestra.run.smithi032.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-04-12T05:23:06.722 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 45 k 2024-04-12T05:23:06.722 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-common x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 117 k 2024-04-12T05:23:06.722 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 190 k 2024-04-12T05:23:06.722 INFO:teuthology.orchestra.run.smithi032.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-04-12T05:23:06.722 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 370 k 2024-04-12T05:23:06.722 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 359 k 2024-04-12T05:23:06.722 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 107 k 2024-04-12T05:23:06.722 INFO:teuthology.orchestra.run.smithi032.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-12T05:23:06.722 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:23:06.722 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-12T05:23:06.723 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:23:06.723 INFO:teuthology.orchestra.run.smithi032.stdout:Install 21 Packages 2024-04-12T05:23:06.723 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrade 2 Packages 2024-04-12T05:23:06.723 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:23:06.723 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 64 M 2024-04-12T05:23:06.724 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-12T05:23:07.592 INFO:teuthology.orchestra.run.smithi032.stdout:(1/23): ceph-base-17.2.7-827.gd3b5060a.el8.x86_ 8.9 MB/s | 5.3 MB 00:00 2024-04-12T05:23:07.633 INFO:teuthology.orchestra.run.smithi032.stdout:(2/23): ceph-selinux-17.2.7-827.gd3b5060a.el8.x 586 kB/s | 24 kB 00:00 2024-04-12T05:23:07.701 INFO:teuthology.orchestra.run.smithi032.stdout:(3/23): libcephfs2-17.2.7-827.gd3b5060a.el8.x86 12 MB/s | 798 kB 00:00 2024-04-12T05:23:07.776 INFO:teuthology.orchestra.run.smithi032.stdout:(4/23): libradosstriper1-17.2.7-827.gd3b5060a.e 6.6 MB/s | 505 kB 00:00 2024-04-12T05:23:08.085 INFO:teuthology.orchestra.run.smithi032.stdout:(5/23): librgw2-17.2.7-827.gd3b5060a.el8.x86_64 18 MB/s | 5.6 MB 00:00 2024-04-12T05:23:08.134 INFO:teuthology.orchestra.run.smithi032.stdout:(6/23): python3-ceph-argparse-17.2.7-827.gd3b50 901 kB/s | 45 kB 00:00 2024-04-12T05:23:08.176 INFO:teuthology.orchestra.run.smithi032.stdout:(7/23): python3-ceph-common-17.2.7-827.gd3b5060 2.7 MB/s | 117 kB 00:00 2024-04-12T05:23:08.235 INFO:teuthology.orchestra.run.smithi032.stdout:(8/23): python3-cephfs-17.2.7-827.gd3b5060a.el8 3.2 MB/s | 190 kB 00:00 2024-04-12T05:23:08.285 INFO:teuthology.orchestra.run.smithi032.stdout:(9/23): python3-rados-17.2.7-827.gd3b5060a.el8. 7.2 MB/s | 370 kB 00:00 2024-04-12T05:23:08.336 INFO:teuthology.orchestra.run.smithi032.stdout:(10/23): python3-rbd-17.2.7-827.gd3b5060a.el8.x 7.0 MB/s | 359 kB 00:00 2024-04-12T05:23:08.386 INFO:teuthology.orchestra.run.smithi032.stdout:(11/23): python3-rgw-17.2.7-827.gd3b5060a.el8.x 2.1 MB/s | 107 kB 00:00 2024-04-12T05:23:09.628 INFO:teuthology.orchestra.run.smithi032.stdout:(12/23): librdkafka-1.6.1-1.el8.x86_64.rpm 545 kB/s | 677 kB 00:01 2024-04-12T05:23:10.120 INFO:teuthology.orchestra.run.smithi032.stdout:(13/23): ceph-common-17.2.7-827.gd3b5060a.el8.x 7.6 MB/s | 24 MB 00:03 2024-04-12T05:23:10.404 INFO:teuthology.orchestra.run.smithi032.stdout:(14/23): ceph-radosgw-17.2.7-827.gd3b5060a.el8. 4.9 MB/s | 17 MB 00:03 2024-04-12T05:23:10.429 INFO:teuthology.orchestra.run.smithi032.stdout:(15/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 324 kB/s | 259 kB 00:00 2024-04-12T05:23:10.462 INFO:teuthology.orchestra.run.smithi032.stdout:(16/23): python3-prettytable-0.7.2-14.el8.noarc 128 kB/s | 44 kB 00:00 2024-04-12T05:23:10.763 INFO:teuthology.orchestra.run.smithi032.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 196 kB/s | 59 kB 00:00 2024-04-12T05:23:10.780 INFO:teuthology.orchestra.run.smithi032.stdout:(18/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 125 kB/s | 47 kB 00:00 2024-04-12T05:23:10.880 INFO:teuthology.orchestra.run.smithi032.stdout:(19/23): gperftools-libs-2.7-9.el8.x86_64.rpm 679 kB/s | 306 kB 00:00 2024-04-12T05:23:10.997 INFO:teuthology.orchestra.run.smithi032.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 321 kB/s | 75 kB 00:00 2024-04-12T05:23:11.231 INFO:teuthology.orchestra.run.smithi032.stdout:(21/23): librados2-17.2.7-827.gd3b5060a.el8.x86 9.9 MB/s | 3.5 MB 00:00 2024-04-12T05:23:11.356 INFO:teuthology.orchestra.run.smithi032.stdout:(22/23): librbd1-17.2.7-827.gd3b5060a.el8.x86_6 10 MB/s | 3.6 MB 00:00 2024-04-12T05:23:11.464 INFO:teuthology.orchestra.run.smithi032.stdout:(23/23): thrift-0.13.0-2.el8.x86_64.rpm 2.5 MB/s | 1.7 MB 00:00 2024-04-12T05:23:11.466 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:23:11.467 INFO:teuthology.orchestra.run.smithi032.stdout:Total 13 MB/s | 64 MB 00:04 2024-04-12T05:23:11.513 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-12T05:23:11.563 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-12T05:23:11.563 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-12T05:23:11.963 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-12T05:23:11.966 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-12T05:23:13.039 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-12T05:23:13.242 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-04-12T05:23:13.468 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-12T05:23:13.860 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-12T05:23:14.068 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : librados2-2:17.2.7-827.gd3b5060a.el8.x86_64 3/25 2024-04-12T05:23:14.166 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados2-2:17.2.7-827.gd3b5060a.el8.x86_64 3/25 2024-04-12T05:23:14.397 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rados-2:17.2.7-827.gd3b5060a.el8.x86_64 4/25 2024-04-12T05:23:14.585 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephfs2-2:17.2.7-827.gd3b5060a.el8.x86_64 5/25 2024-04-12T05:23:15.005 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephfs2-2:17.2.7-827.gd3b5060a.el8.x86_64 5/25 2024-04-12T05:23:15.227 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : librbd1-2:17.2.7-827.gd3b5060a.el8.x86_64 6/25 2024-04-12T05:23:15.462 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librbd1-2:17.2.7-827.gd3b5060a.el8.x86_64 6/25 2024-04-12T05:23:15.673 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2024-04-12T05:23:15.890 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/25 2024-04-12T05:23:16.845 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2024-04-12T05:23:17.009 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librgw2-2:17.2.7-827.gd3b5060a.el8.x86_64 10/25 2024-04-12T05:23:17.074 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librgw2-2:17.2.7-827.gd3b5060a.el8.x86_64 10/25 2024-04-12T05:23:17.237 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ceph-argparse-2:17.2.7-827.gd3b5060a.el8.x 11/25 2024-04-12T05:23:17.301 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-selinux-2:17.2.7-827.gd3b5060a.el8.x86_64 22/25 2024-04-12T05:23:17.301 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /sys 2024-04-12T05:23:17.301 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /proc 2024-04-12T05:23:17.301 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /dev 2024-04-12T05:23:17.301 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /run 2024-04-12T05:23:17.302 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /mnt 2024-04-12T05:23:17.302 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /var/tmp 2024-04-12T05:23:17.302 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /home 2024-04-12T05:23:17.302 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /tmp 2024-04-12T05:23:17.302 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /dev 2024-04-12T05:23:17.302 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:23:17.438 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cephfs-2:17.2.7-827.gd3b5060a.el8.x86_64 12/25 2024-04-12T05:23:17.638 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rgw-2:17.2.7-827.gd3b5060a.el8.x86_64 13/25 2024-04-12T05:23:17.836 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rbd-2:17.2.7-827.gd3b5060a.el8.x86_64 14/25 2024-04-12T05:23:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libradosstriper1-2:17.2.7-827.gd3b5060a.el8.x86_64 15/25 2024-04-12T05:23:18.064 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libradosstriper1-2:17.2.7-827.gd3b5060a.el8.x86_64 15/25 2024-04-12T05:23:18.288 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-04-12T05:23:18.428 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-04-12T05:23:18.633 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-04-12T05:23:18.802 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ceph-common-2:17.2.7-827.gd3b5060a.el8.x86 19/25 2024-04-12T05:23:19.259 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-radosgw-2:17.2.7-827.gd3b5060a.el8.x86_64 23/25 2024-04-12T05:23:19.325 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-827.gd3b5060a.el8.x86_64 23/25 2024-04-12T05:23:19.325 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-12T05:23:19.534 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-12T05:23:19.534 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-12T05:23:20.261 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-12T05:23:20.261 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-base-2:17.2.7-827.gd3b5060a.el8.x86_64 1/25 2024-04-12T05:23:20.261 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-common-2:17.2.7-827.gd3b5060a.el8.x86_64 2/25 2024-04-12T05:23:20.262 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-radosgw-2:17.2.7-827.gd3b5060a.el8.x86_64 3/25 2024-04-12T05:23:20.262 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-selinux-2:17.2.7-827.gd3b5060a.el8.x86_64 4/25 2024-04-12T05:23:20.262 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs2-2:17.2.7-827.gd3b5060a.el8.x86_64 5/25 2024-04-12T05:23:20.262 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libradosstriper1-2:17.2.7-827.gd3b5060a.el8.x86_64 6/25 2024-04-12T05:23:20.262 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librgw2-2:17.2.7-827.gd3b5060a.el8.x86_64 7/25 2024-04-12T05:23:20.262 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-argparse-2:17.2.7-827.gd3b5060a.el8.x 8/25 2024-04-12T05:23:20.262 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-common-2:17.2.7-827.gd3b5060a.el8.x86 9/25 2024-04-12T05:23:20.262 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cephfs-2:17.2.7-827.gd3b5060a.el8.x86_64 10/25 2024-04-12T05:23:20.262 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rados-2:17.2.7-827.gd3b5060a.el8.x86_64 11/25 2024-04-12T05:23:20.262 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rbd-2:17.2.7-827.gd3b5060a.el8.x86_64 12/25 2024-04-12T05:23:20.262 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rgw-2:17.2.7-827.gd3b5060a.el8.x86_64 13/25 2024-04-12T05:23:20.262 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-04-12T05:23:20.262 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-04-12T05:23:20.263 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-04-12T05:23:20.263 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-04-12T05:23:20.263 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-04-12T05:23:20.263 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2024-04-12T05:23:20.263 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2024-04-12T05:23:20.263 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2024-04-12T05:23:20.263 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:17.2.7-827.gd3b5060a.el8.x86_64 22/25 2024-04-12T05:23:20.263 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-04-12T05:23:20.263 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:17.2.7-827.gd3b5060a.el8.x86_64 24/25 2024-04-12T05:23:21.133 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-04-12T05:23:21.133 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:23:21.133 INFO:teuthology.orchestra.run.smithi067.stdout:Upgraded: 2024-04-12T05:23:21.133 INFO:teuthology.orchestra.run.smithi067.stdout: librados2-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:21.133 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:21.133 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-12T05:23:21.133 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:21.133 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:21.133 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:21.134 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:21.134 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-12T05:23:21.134 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:21.134 INFO:teuthology.orchestra.run.smithi067.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-12T05:23:21.134 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-04-12T05:23:21.134 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:21.134 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-04-12T05:23:21.134 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:21.134 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-12T05:23:21.134 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-12T05:23:21.134 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:21.134 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:21.135 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:21.135 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-12T05:23:21.135 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:21.135 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:21.135 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:21.135 INFO:teuthology.orchestra.run.smithi067.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-12T05:23:21.135 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:23:21.135 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-12T05:23:21.433 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-test 2024-04-12T05:23:21.656 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-common-2:17.2.7-827.gd3b5060a.el8.x86_64 20/25 2024-04-12T05:23:22.001 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-common-2:17.2.7-827.gd3b5060a.el8.x86_64 20/25 2024-04-12T05:23:22.195 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:46 ago on Fri 12 Apr 2024 05:22:36 AM UTC. 2024-04-12T05:23:22.618 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-common-2:17.2.7-827.gd3b5060a.el8.x86_64 20/25 2024-04-12T05:23:22.916 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-base-2:17.2.7-827.gd3b5060a.el8.x86_64 21/25 2024-04-12T05:23:23.033 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-base-2:17.2.7-827.gd3b5060a.el8.x86_64 21/25 2024-04-12T05:23:23.217 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-selinux-2:17.2.7-827.gd3b5060a.el8.x86_64 22/25 2024-04-12T05:23:23.929 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-12T05:23:23.930 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:23:23.930 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-04-12T05:23:23.930 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:23:23.930 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-12T05:23:23.930 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 53 M 2024-04-12T05:23:23.931 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-04-12T05:23:23.931 INFO:teuthology.orchestra.run.smithi067.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-04-12T05:23:23.931 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 179 k 2024-04-12T05:23:23.931 INFO:teuthology.orchestra.run.smithi067.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-04-12T05:23:23.931 INFO:teuthology.orchestra.run.smithi067.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-04-12T05:23:23.931 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-04-12T05:23:23.931 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:23:23.931 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-12T05:23:23.931 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:23:23.931 INFO:teuthology.orchestra.run.smithi067.stdout:Install 6 Packages 2024-04-12T05:23:23.931 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:23:23.932 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 54 M 2024-04-12T05:23:23.932 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 236 M 2024-04-12T05:23:23.932 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-12T05:23:24.176 INFO:teuthology.orchestra.run.smithi067.stdout:(1/6): libcephsqlite-17.2.7-827.gd3b5060a.el8.x 962 kB/s | 179 kB 00:00 2024-04-12T05:23:24.601 INFO:teuthology.orchestra.run.smithi067.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 332 kB/s | 203 kB 00:00 2024-04-12T05:23:24.776 INFO:teuthology.orchestra.run.smithi067.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 313 kB/s | 188 kB 00:00 2024-04-12T05:23:25.177 INFO:teuthology.orchestra.run.smithi067.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 176 kB/s | 71 kB 00:00 2024-04-12T05:23:25.394 INFO:teuthology.orchestra.run.smithi067.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 408 kB/s | 323 kB 00:00 2024-04-12T05:23:26.469 INFO:teuthology.orchestra.run.smithi067.stdout:(6/6): ceph-test-17.2.7-827.gd3b5060a.el8.x86_6 21 MB/s | 53 MB 00:02 2024-04-12T05:23:26.471 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:23:26.471 INFO:teuthology.orchestra.run.smithi067.stdout:Total 21 MB/s | 54 MB 00:02 2024-04-12T05:23:26.515 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-12T05:23:26.537 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-12T05:23:26.538 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-12T05:23:26.845 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-12T05:23:26.848 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-12T05:23:27.355 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-12T05:23:27.613 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-04-12T05:23:27.785 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-04-12T05:23:27.889 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-04-12T05:23:27.999 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-04-12T05:23:28.205 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-04-12T05:23:28.372 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephsqlite-2:17.2.7-827.gd3b5060a.el8.x86_64 5/6 2024-04-12T05:23:33.763 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephsqlite-2:17.2.7-827.gd3b5060a.el8.x86_64 5/6 2024-04-12T05:23:34.042 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-test-2:17.2.7-827.gd3b5060a.el8.x86_64 6/6 2024-04-12T05:23:34.324 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-test-2:17.2.7-827.gd3b5060a.el8.x86_64 6/6 2024-04-12T05:23:34.324 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-test-2:17.2.7-827.gd3b5060a.el8.x86_64 1/6 2024-04-12T05:23:34.325 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephsqlite-2:17.2.7-827.gd3b5060a.el8.x86_64 2/6 2024-04-12T05:23:34.325 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-04-12T05:23:34.325 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-04-12T05:23:34.325 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-04-12T05:23:34.845 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-04-12T05:23:34.845 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:23:34.845 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-12T05:23:34.845 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:34.845 INFO:teuthology.orchestra.run.smithi067.stdout: jq-1.6-8.el8.x86_64 2024-04-12T05:23:34.845 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:34.845 INFO:teuthology.orchestra.run.smithi067.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-04-12T05:23:34.845 INFO:teuthology.orchestra.run.smithi067.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-12T05:23:34.845 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-12T05:23:34.845 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:23:34.846 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-12T05:23:35.101 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph 2024-04-12T05:23:35.875 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:59 ago on Fri 12 Apr 2024 05:22:36 AM UTC. 2024-04-12T05:23:37.640 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-12T05:23:37.643 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:23:37.644 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-04-12T05:23:37.644 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:23:37.644 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-12T05:23:37.644 INFO:teuthology.orchestra.run.smithi067.stdout: ceph x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 6.4 k 2024-04-12T05:23:37.644 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-04-12T05:23:37.644 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 2.3 M 2024-04-12T05:23:37.644 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 1.6 M 2024-04-12T05:23:37.644 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core noarch 2:17.2.7-827.gd3b5060a.el8 ceph-noarch 240 k 2024-04-12T05:23:37.644 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 4.0 M 2024-04-12T05:23:37.644 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 17 M 2024-04-12T05:23:37.644 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-12T05:23:37.644 INFO:teuthology.orchestra.run.smithi067.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-12T05:23:37.644 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-04-12T05:23:37.644 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-12T05:23:37.645 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-12T05:23:37.645 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-04-12T05:23:37.645 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-12T05:23:37.645 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-12T05:23:37.645 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-12T05:23:37.645 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-04-12T05:23:37.645 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-04-12T05:23:37.645 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-12T05:23:37.645 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-12T05:23:37.645 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-12T05:23:37.645 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-04-12T05:23:37.645 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-04-12T05:23:37.645 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-04-12T05:23:37.646 INFO:teuthology.orchestra.run.smithi067.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-12T05:23:37.646 INFO:teuthology.orchestra.run.smithi067.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-12T05:23:37.646 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-12T05:23:37.646 INFO:teuthology.orchestra.run.smithi067.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-12T05:23:37.646 INFO:teuthology.orchestra.run.smithi067.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-12T05:23:37.646 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-12T05:23:37.646 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-12T05:23:37.646 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-04-12T05:23:37.646 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-12T05:23:37.646 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:23:37.646 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-12T05:23:37.646 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:23:37.646 INFO:teuthology.orchestra.run.smithi067.stdout:Install 32 Packages 2024-04-12T05:23:37.646 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:23:37.647 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 29 M 2024-04-12T05:23:37.647 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 113 M 2024-04-12T05:23:37.647 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-12T05:23:38.271 INFO:teuthology.orchestra.run.smithi067.stdout:(1/32): ceph-17.2.7-827.gd3b5060a.el8.x86_64.rp 32 kB/s | 6.4 kB 00:00 2024-04-12T05:23:38.462 INFO:teuthology.orchestra.run.smithi067.stdout:(2/32): ceph-mds-17.2.7-827.gd3b5060a.el8.x86_6 5.9 MB/s | 2.3 MB 00:00 2024-04-12T05:23:38.504 INFO:teuthology.orchestra.run.smithi067.stdout:(3/32): ceph-mgr-17.2.7-827.gd3b5060a.el8.x86_6 3.6 MB/s | 1.6 MB 00:00 2024-04-12T05:23:38.613 INFO:teuthology.orchestra.run.smithi067.stdout:(4/32): ceph-mgr-modules-core-17.2.7-827.gd3b50 2.2 MB/s | 240 kB 00:00 2024-04-12T05:23:39.272 INFO:teuthology.orchestra.run.smithi067.stdout:(5/32): ceph-osd-17.2.7-827.gd3b5060a.el8.x86_6 21 MB/s | 17 MB 00:00 2024-04-12T05:23:39.297 INFO:teuthology.orchestra.run.smithi067.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 230 kB/s | 157 kB 00:00 2024-04-12T05:23:39.619 INFO:teuthology.orchestra.run.smithi067.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 112 kB/s | 39 kB 00:00 2024-04-12T05:23:39.747 INFO:teuthology.orchestra.run.smithi067.stdout:(8/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 230 kB/s | 103 kB 00:00 2024-04-12T05:23:39.998 INFO:teuthology.orchestra.run.smithi067.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 151 kB/s | 57 kB 00:00 2024-04-12T05:23:40.140 INFO:teuthology.orchestra.run.smithi067.stdout:(10/32): ceph-mon-17.2.7-827.gd3b5060a.el8.x86_ 2.2 MB/s | 4.0 MB 00:01 2024-04-12T05:23:40.574 INFO:teuthology.orchestra.run.smithi067.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 432 kB/s | 249 kB 00:00 2024-04-12T05:23:40.674 INFO:teuthology.orchestra.run.smithi067.stdout:(12/32): python3-werkzeug-0.12.2-4.el8.noarch.r 494 kB/s | 457 kB 00:00 2024-04-12T05:23:40.824 INFO:teuthology.orchestra.run.smithi067.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 291 kB/s | 44 kB 00:00 2024-04-12T05:23:40.916 INFO:teuthology.orchestra.run.smithi067.stdout:(14/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 2.0 MB/s | 185 kB 00:00 2024-04-12T05:23:40.966 INFO:teuthology.orchestra.run.smithi067.stdout:(15/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 3.4 MB/s | 173 kB 00:00 2024-04-12T05:23:41.033 INFO:teuthology.orchestra.run.smithi067.stdout:(16/32): python3-cherrypy-18.4.0-1.el8.noarch.r 5.6 MB/s | 384 kB 00:00 2024-04-12T05:23:41.058 INFO:teuthology.orchestra.run.smithi067.stdout:(17/32): python3-pycparser-2.14-14.el8.noarch.r 225 kB/s | 109 kB 00:00 2024-04-12T05:23:41.084 INFO:teuthology.orchestra.run.smithi067.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2024-04-12T05:23:41.134 INFO:teuthology.orchestra.run.smithi067.stdout:(19/32): python3-logutils-0.3.5-11.el8.noarch.r 1.1 MB/s | 49 kB 00:00 2024-04-12T05:23:41.184 INFO:teuthology.orchestra.run.smithi067.stdout:(20/32): python3-more-itertools-7.2.0-3.el8.noa 1.2 MB/s | 59 kB 00:00 2024-04-12T05:23:41.209 INFO:teuthology.orchestra.run.smithi067.stdout:(21/32): python3-jaraco-functools-2.0-4.el8.noa 117 kB/s | 18 kB 00:00 2024-04-12T05:23:41.235 INFO:teuthology.orchestra.run.smithi067.stdout:(22/32): python3-cryptography-3.2.1-7.el8.x86_6 586 kB/s | 641 kB 00:01 2024-04-12T05:23:41.260 INFO:teuthology.orchestra.run.smithi067.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 309 kB/s | 16 kB 00:00 2024-04-12T05:23:41.294 INFO:teuthology.orchestra.run.smithi067.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.5 MB/s | 283 kB 00:00 2024-04-12T05:23:41.336 INFO:teuthology.orchestra.run.smithi067.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 343 kB/s | 24 kB 00:00 2024-04-12T05:23:41.353 INFO:teuthology.orchestra.run.smithi067.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 505 kB/s | 29 kB 00:00 2024-04-12T05:23:41.369 INFO:teuthology.orchestra.run.smithi067.stdout:(27/32): python3-simplegeneric-0.8.1-17.el8.noa 140 kB/s | 19 kB 00:00 2024-04-12T05:23:41.386 INFO:teuthology.orchestra.run.smithi067.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 547 kB/s | 27 kB 00:00 2024-04-12T05:23:41.412 INFO:teuthology.orchestra.run.smithi067.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 4.2 MB/s | 255 kB 00:00 2024-04-12T05:23:41.462 INFO:teuthology.orchestra.run.smithi067.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.1 MB/s | 85 kB 00:00 2024-04-12T05:23:41.479 INFO:teuthology.orchestra.run.smithi067.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 345 kB/s | 23 kB 00:00 2024-04-12T05:23:41.512 INFO:teuthology.orchestra.run.smithi067.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.7 MB/s | 251 kB 00:00 2024-04-12T05:23:41.518 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:23:41.518 INFO:teuthology.orchestra.run.smithi067.stdout:Total 7.5 MB/s | 29 MB 00:03 2024-04-12T05:23:41.602 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-12T05:23:41.643 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-12T05:23:41.643 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-12T05:23:41.975 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-12T05:23:41.978 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-12T05:23:43.082 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-12T05:23:43.322 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-12T05:23:43.462 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-selinux-2:17.2.7-827.gd3b5060a.el8.x86_64 22/25 2024-04-12T05:23:43.462 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /sys 2024-04-12T05:23:43.462 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /proc 2024-04-12T05:23:43.462 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /dev 2024-04-12T05:23:43.462 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /run 2024-04-12T05:23:43.462 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /mnt 2024-04-12T05:23:43.463 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /var/tmp 2024-04-12T05:23:43.463 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /home 2024-04-12T05:23:43.463 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /tmp 2024-04-12T05:23:43.463 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /dev 2024-04-12T05:23:43.463 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:23:43.546 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-12T05:23:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-04-12T05:23:44.061 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-04-12T05:23:44.291 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-04-12T05:23:44.491 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-04-12T05:23:44.657 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-04-12T05:23:44.860 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-04-12T05:23:45.063 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-04-12T05:23:45.259 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-04-12T05:23:45.288 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-radosgw-2:17.2.7-827.gd3b5060a.el8.x86_64 23/25 2024-04-12T05:23:45.354 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-827.gd3b5060a.el8.x86_64 23/25 2024-04-12T05:23:45.354 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-12T05:23:45.539 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-04-12T05:23:45.583 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-12T05:23:45.584 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-12T05:23:45.770 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-04-12T05:23:46.188 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-04-12T05:23:46.246 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-12T05:23:46.246 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-base-2:17.2.7-827.gd3b5060a.el8.x86_64 1/25 2024-04-12T05:23:46.246 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-common-2:17.2.7-827.gd3b5060a.el8.x86_64 2/25 2024-04-12T05:23:46.246 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-radosgw-2:17.2.7-827.gd3b5060a.el8.x86_64 3/25 2024-04-12T05:23:46.246 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-selinux-2:17.2.7-827.gd3b5060a.el8.x86_64 4/25 2024-04-12T05:23:46.246 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephfs2-2:17.2.7-827.gd3b5060a.el8.x86_64 5/25 2024-04-12T05:23:46.246 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libradosstriper1-2:17.2.7-827.gd3b5060a.el8.x86_64 6/25 2024-04-12T05:23:46.246 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librgw2-2:17.2.7-827.gd3b5060a.el8.x86_64 7/25 2024-04-12T05:23:46.246 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ceph-argparse-2:17.2.7-827.gd3b5060a.el8.x 8/25 2024-04-12T05:23:46.247 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ceph-common-2:17.2.7-827.gd3b5060a.el8.x86 9/25 2024-04-12T05:23:46.247 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cephfs-2:17.2.7-827.gd3b5060a.el8.x86_64 10/25 2024-04-12T05:23:46.247 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rados-2:17.2.7-827.gd3b5060a.el8.x86_64 11/25 2024-04-12T05:23:46.247 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rbd-2:17.2.7-827.gd3b5060a.el8.x86_64 12/25 2024-04-12T05:23:46.247 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rgw-2:17.2.7-827.gd3b5060a.el8.x86_64 13/25 2024-04-12T05:23:46.247 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-04-12T05:23:46.247 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-04-12T05:23:46.247 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-04-12T05:23:46.247 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-04-12T05:23:46.247 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-04-12T05:23:46.247 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2024-04-12T05:23:46.247 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2024-04-12T05:23:46.247 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2024-04-12T05:23:46.247 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados2-2:17.2.7-827.gd3b5060a.el8.x86_64 22/25 2024-04-12T05:23:46.248 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-04-12T05:23:46.248 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librbd1-2:17.2.7-827.gd3b5060a.el8.x86_64 24/25 2024-04-12T05:23:46.426 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-04-12T05:23:46.644 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-04-12T05:23:46.924 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-04-12T05:23:47.065 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-04-12T05:23:47.065 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:23:47.065 INFO:teuthology.orchestra.run.smithi032.stdout:Upgraded: 2024-04-12T05:23:47.065 INFO:teuthology.orchestra.run.smithi032.stdout: librados2-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:47.065 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:47.065 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-12T05:23:47.066 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:47.066 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:47.066 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:47.066 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-selinux-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:47.066 INFO:teuthology.orchestra.run.smithi032.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-12T05:23:47.066 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:47.066 INFO:teuthology.orchestra.run.smithi032.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-12T05:23:47.066 INFO:teuthology.orchestra.run.smithi032.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-04-12T05:23:47.066 INFO:teuthology.orchestra.run.smithi032.stdout: libradosstriper1-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:47.066 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-04-12T05:23:47.066 INFO:teuthology.orchestra.run.smithi032.stdout: librgw2-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:47.066 INFO:teuthology.orchestra.run.smithi032.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-12T05:23:47.066 INFO:teuthology.orchestra.run.smithi032.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-12T05:23:47.067 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:47.067 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-common-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:47.067 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:47.067 INFO:teuthology.orchestra.run.smithi032.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-12T05:23:47.067 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:47.067 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:47.067 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:47.067 INFO:teuthology.orchestra.run.smithi032.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-12T05:23:47.067 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:23:47.067 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-12T05:23:47.122 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-12T05:23:47.377 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-test 2024-04-12T05:23:47.487 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-04-12T05:23:47.727 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-04-12T05:23:47.916 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-04-12T05:23:48.093 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-12T05:23:48.139 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:46 ago on Fri 12 Apr 2024 05:23:02 AM UTC. 2024-04-12T05:23:48.332 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-12T05:23:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-12T05:23:48.824 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-04-12T05:23:49.157 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-04-12T05:23:49.458 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-12T05:23:49.862 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-827.gd3b5060a.el8.n 27/32 2024-04-12T05:23:49.880 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-12T05:23:49.881 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:23:49.882 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-12T05:23:49.882 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:23:49.882 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-12T05:23:49.882 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 53 M 2024-04-12T05:23:49.882 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-12T05:23:49.882 INFO:teuthology.orchestra.run.smithi032.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-04-12T05:23:49.882 INFO:teuthology.orchestra.run.smithi032.stdout: libcephsqlite x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 179 k 2024-04-12T05:23:49.882 INFO:teuthology.orchestra.run.smithi032.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-04-12T05:23:49.882 INFO:teuthology.orchestra.run.smithi032.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-04-12T05:23:49.882 INFO:teuthology.orchestra.run.smithi032.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-04-12T05:23:49.882 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:23:49.882 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-12T05:23:49.882 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:23:49.883 INFO:teuthology.orchestra.run.smithi032.stdout:Install 6 Packages 2024-04-12T05:23:49.883 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:23:49.883 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 54 M 2024-04-12T05:23:49.883 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 236 M 2024-04-12T05:23:49.883 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-12T05:23:50.114 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-2:17.2.7-827.gd3b5060a.el8.x86_64 28/32 2024-04-12T05:23:50.452 INFO:teuthology.orchestra.run.smithi032.stdout:(1/6): libcephsqlite-17.2.7-827.gd3b5060a.el8.x 349 kB/s | 179 kB 00:00 2024-04-12T05:23:50.569 INFO:teuthology.orchestra.run.smithi032.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 323 kB/s | 203 kB 00:00 2024-04-12T05:23:51.003 INFO:teuthology.orchestra.run.smithi032.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 342 kB/s | 188 kB 00:00 2024-04-12T05:23:51.320 INFO:teuthology.orchestra.run.smithi032.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 430 kB/s | 323 kB 00:00 2024-04-12T05:23:51.412 INFO:teuthology.orchestra.run.smithi032.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 173 kB/s | 71 kB 00:00 2024-04-12T05:23:51.727 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-2:17.2.7-827.gd3b5060a.el8.x86_64 28/32 2024-04-12T05:23:52.582 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-osd-2:17.2.7-827.gd3b5060a.el8.x86_64 29/32 2024-04-12T05:23:52.679 INFO:teuthology.orchestra.run.smithi032.stdout:(6/6): ceph-test-17.2.7-827.gd3b5060a.el8.x86_6 19 MB/s | 53 MB 00:02 2024-04-12T05:23:52.681 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:23:52.681 INFO:teuthology.orchestra.run.smithi032.stdout:Total 19 MB/s | 54 MB 00:02 2024-04-12T05:23:52.725 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-12T05:23:52.747 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-12T05:23:52.748 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-12T05:23:53.012 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-osd-2:17.2.7-827.gd3b5060a.el8.x86_64 29/32 2024-04-12T05:23:53.052 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-12T05:23:53.056 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-12T05:23:53.219 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mon-2:17.2.7-827.gd3b5060a.el8.x86_64 30/32 2024-04-12T05:23:53.448 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mon-2:17.2.7-827.gd3b5060a.el8.x86_64 30/32 2024-04-12T05:23:53.576 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-12T05:23:53.653 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mds-2:17.2.7-827.gd3b5060a.el8.x86_64 31/32 2024-04-12T05:23:53.671 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mds-2:17.2.7-827.gd3b5060a.el8.x86_64 31/32 2024-04-12T05:23:53.803 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-2:17.2.7-827.gd3b5060a.el8.x86_64 32/32 2024-04-12T05:23:53.826 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-04-12T05:23:54.018 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-04-12T05:23:54.155 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-04-12T05:23:54.228 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-2:17.2.7-827.gd3b5060a.el8.x86_64 32/32 2024-04-12T05:23:54.228 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-2:17.2.7-827.gd3b5060a.el8.x86_64 1/32 2024-04-12T05:23:54.228 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mds-2:17.2.7-827.gd3b5060a.el8.x86_64 2/32 2024-04-12T05:23:54.228 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-2:17.2.7-827.gd3b5060a.el8.x86_64 3/32 2024-04-12T05:23:54.228 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mon-2:17.2.7-827.gd3b5060a.el8.x86_64 4/32 2024-04-12T05:23:54.228 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-osd-2:17.2.7-827.gd3b5060a.el8.x86_64 5/32 2024-04-12T05:23:54.228 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-827.gd3b5060a.el8.n 6/32 2024-04-12T05:23:54.229 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-04-12T05:23:54.229 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-04-12T05:23:54.229 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-04-12T05:23:54.229 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-04-12T05:23:54.229 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-04-12T05:23:54.229 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-04-12T05:23:54.229 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-04-12T05:23:54.229 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-04-12T05:23:54.229 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-04-12T05:23:54.229 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-04-12T05:23:54.229 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-12T05:23:54.229 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-04-12T05:23:54.229 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-04-12T05:23:54.230 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-04-12T05:23:54.230 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-04-12T05:23:54.230 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-04-12T05:23:54.230 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-04-12T05:23:54.230 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-04-12T05:23:54.230 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-04-12T05:23:54.230 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-04-12T05:23:54.230 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-04-12T05:23:54.230 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-04-12T05:23:54.230 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-04-12T05:23:54.230 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-04-12T05:23:54.231 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-04-12T05:23:54.234 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-04-12T05:23:54.430 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-04-12T05:23:54.588 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephsqlite-2:17.2.7-827.gd3b5060a.el8.x86_64 5/6 2024-04-12T05:23:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-04-12T05:23:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:23:55.057 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-12T05:23:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core-2:17.2.7-827.gd3b5060a.el8.noarch 2024-04-12T05:23:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:23:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-12T05:23:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-12T05:23:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-12T05:23:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-12T05:23:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-12T05:23:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-04-12T05:23:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-12T05:23:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-12T05:23:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-12T05:23:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-12T05:23:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-12T05:23:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-12T05:23:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-12T05:23:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend-2.6-1.el8.noarch 2024-04-12T05:23:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-12T05:23:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-12T05:23:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-12T05:23:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-12T05:23:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-12T05:23:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-12T05:23:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-12T05:23:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-12T05:23:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-12T05:23:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-12T05:23:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-12T05:23:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-12T05:23:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:23:55.060 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-12T05:23:55.340 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-base 2024-04-12T05:23:56.102 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:20 ago on Fri 12 Apr 2024 05:22:36 AM UTC. 2024-04-12T05:23:57.806 INFO:teuthology.orchestra.run.smithi067.stdout:Package ceph-base-2:17.2.7-827.gd3b5060a.el8.x86_64 is already installed. 2024-04-12T05:23:57.858 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-12T05:23:57.859 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-04-12T05:23:57.860 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-12T05:23:57.960 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install cephadm 2024-04-12T05:23:58.722 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:22 ago on Fri 12 Apr 2024 05:22:36 AM UTC. 2024-04-12T05:23:59.996 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephsqlite-2:17.2.7-827.gd3b5060a.el8.x86_64 5/6 2024-04-12T05:24:00.253 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-test-2:17.2.7-827.gd3b5060a.el8.x86_64 6/6 2024-04-12T05:24:00.472 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-12T05:24:00.472 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:24:00.472 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-04-12T05:24:00.472 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:24:00.472 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-12T05:24:00.472 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm noarch 2:17.2.7-827.gd3b5060a.el8 ceph-noarch 82 k 2024-04-12T05:24:00.472 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:24:00.473 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-12T05:24:00.473 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:24:00.473 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-04-12T05:24:00.473 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:24:00.473 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 82 k 2024-04-12T05:24:00.473 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 372 k 2024-04-12T05:24:00.473 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-12T05:24:00.549 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-test-2:17.2.7-827.gd3b5060a.el8.x86_64 6/6 2024-04-12T05:24:00.549 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-test-2:17.2.7-827.gd3b5060a.el8.x86_64 1/6 2024-04-12T05:24:00.549 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephsqlite-2:17.2.7-827.gd3b5060a.el8.x86_64 2/6 2024-04-12T05:24:00.549 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-04-12T05:24:00.550 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-04-12T05:24:00.550 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-04-12T05:24:00.736 INFO:teuthology.orchestra.run.smithi067.stdout:cephadm-17.2.7-827.gd3b5060a.el8.noarch.rpm 312 kB/s | 82 kB 00:00 2024-04-12T05:24:00.737 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:24:00.737 INFO:teuthology.orchestra.run.smithi067.stdout:Total 311 kB/s | 82 kB 00:00 2024-04-12T05:24:00.737 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-12T05:24:00.746 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-12T05:24:00.746 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-12T05:24:00.747 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-12T05:24:00.751 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-12T05:24:00.800 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-12T05:24:01.015 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-04-12T05:24:01.016 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:24:01.016 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-12T05:24:01.016 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:24:01.016 INFO:teuthology.orchestra.run.smithi032.stdout: jq-1.6-8.el8.x86_64 2024-04-12T05:24:01.016 INFO:teuthology.orchestra.run.smithi032.stdout: libcephsqlite-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:24:01.016 INFO:teuthology.orchestra.run.smithi032.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-04-12T05:24:01.016 INFO:teuthology.orchestra.run.smithi032.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-12T05:24:01.016 INFO:teuthology.orchestra.run.smithi032.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-12T05:24:01.016 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:24:01.016 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-12T05:24:01.221 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph 2024-04-12T05:24:01.285 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephadm-2:17.2.7-827.gd3b5060a.el8.noarch 1/1 2024-04-12T05:24:01.459 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephadm-2:17.2.7-827.gd3b5060a.el8.noarch 1/1 2024-04-12T05:24:01.694 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephadm-2:17.2.7-827.gd3b5060a.el8.noarch 1/1 2024-04-12T05:24:01.986 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:59 ago on Fri 12 Apr 2024 05:23:02 AM UTC. 2024-04-12T05:24:02.079 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephadm-2:17.2.7-827.gd3b5060a.el8.noarch 1/1 2024-04-12T05:24:02.079 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:24:02.079 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-12T05:24:02.079 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm-2:17.2.7-827.gd3b5060a.el8.noarch 2024-04-12T05:24:02.079 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:24:02.079 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-12T05:24:02.299 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-immutable-object-cache 2024-04-12T05:24:03.058 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:27 ago on Fri 12 Apr 2024 05:22:36 AM UTC. 2024-04-12T05:24:03.727 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-12T05:24:03.731 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:24:03.731 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-12T05:24:03.731 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:24:03.731 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-12T05:24:03.731 INFO:teuthology.orchestra.run.smithi032.stdout: ceph x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 6.4 k 2024-04-12T05:24:03.731 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-12T05:24:03.731 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 2.3 M 2024-04-12T05:24:03.731 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 1.6 M 2024-04-12T05:24:03.732 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-modules-core noarch 2:17.2.7-827.gd3b5060a.el8 ceph-noarch 240 k 2024-04-12T05:24:03.732 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 4.0 M 2024-04-12T05:24:03.732 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 17 M 2024-04-12T05:24:03.732 INFO:teuthology.orchestra.run.smithi032.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-12T05:24:03.732 INFO:teuthology.orchestra.run.smithi032.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-12T05:24:03.732 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-04-12T05:24:03.732 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-12T05:24:03.732 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-12T05:24:03.732 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-04-12T05:24:03.732 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-12T05:24:03.732 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-12T05:24:03.732 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-12T05:24:03.732 INFO:teuthology.orchestra.run.smithi032.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-04-12T05:24:03.732 INFO:teuthology.orchestra.run.smithi032.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-04-12T05:24:03.733 INFO:teuthology.orchestra.run.smithi032.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-12T05:24:03.733 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-12T05:24:03.733 INFO:teuthology.orchestra.run.smithi032.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-12T05:24:03.733 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-04-12T05:24:03.733 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-04-12T05:24:03.733 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-04-12T05:24:03.733 INFO:teuthology.orchestra.run.smithi032.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-12T05:24:03.733 INFO:teuthology.orchestra.run.smithi032.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-12T05:24:03.733 INFO:teuthology.orchestra.run.smithi032.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-12T05:24:03.733 INFO:teuthology.orchestra.run.smithi032.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-12T05:24:03.733 INFO:teuthology.orchestra.run.smithi032.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-12T05:24:03.733 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-12T05:24:03.733 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-12T05:24:03.733 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-04-12T05:24:03.733 INFO:teuthology.orchestra.run.smithi032.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-12T05:24:03.734 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:24:03.734 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-12T05:24:03.734 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:24:03.734 INFO:teuthology.orchestra.run.smithi032.stdout:Install 32 Packages 2024-04-12T05:24:03.734 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:24:03.734 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 29 M 2024-04-12T05:24:03.735 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 113 M 2024-04-12T05:24:03.735 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-12T05:24:04.298 INFO:teuthology.orchestra.run.smithi032.stdout:(1/32): ceph-17.2.7-827.gd3b5060a.el8.x86_64.rp 33 kB/s | 6.4 kB 00:00 2024-04-12T05:24:04.565 INFO:teuthology.orchestra.run.smithi032.stdout:(2/32): ceph-mgr-17.2.7-827.gd3b5060a.el8.x86_6 3.4 MB/s | 1.6 MB 00:00 2024-04-12T05:24:04.615 INFO:teuthology.orchestra.run.smithi032.stdout:(3/32): ceph-mds-17.2.7-827.gd3b5060a.el8.x86_6 4.5 MB/s | 2.3 MB 00:00 2024-04-12T05:24:04.707 INFO:teuthology.orchestra.run.smithi032.stdout:(4/32): ceph-mgr-modules-core-17.2.7-827.gd3b50 2.6 MB/s | 240 kB 00:00 2024-04-12T05:24:04.816 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-12T05:24:04.816 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:24:04.817 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-04-12T05:24:04.817 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:24:04.817 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-12T05:24:04.817 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 162 k 2024-04-12T05:24:04.817 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:24:04.817 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-12T05:24:04.817 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:24:04.817 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-04-12T05:24:04.817 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:24:04.818 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 162 k 2024-04-12T05:24:04.818 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 494 k 2024-04-12T05:24:04.818 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-12T05:24:05.166 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-immutable-object-cache-17.2.7-827.gd3b5060 466 kB/s | 162 kB 00:00 2024-04-12T05:24:05.166 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:24:05.166 INFO:teuthology.orchestra.run.smithi067.stdout:Total 464 kB/s | 162 kB 00:00 2024-04-12T05:24:05.167 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-12T05:24:05.178 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-12T05:24:05.178 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-12T05:24:05.232 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-12T05:24:05.234 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-12T05:24:05.265 INFO:teuthology.orchestra.run.smithi032.stdout:(5/32): ceph-mon-17.2.7-827.gd3b5060a.el8.x86_6 4.2 MB/s | 4.0 MB 00:00 2024-04-12T05:24:05.291 INFO:teuthology.orchestra.run.smithi032.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 269 kB/s | 157 kB 00:00 2024-04-12T05:24:05.376 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-12T05:24:05.541 INFO:teuthology.orchestra.run.smithi032.stdout:(7/32): ceph-osd-17.2.7-827.gd3b5060a.el8.x86_6 18 MB/s | 17 MB 00:00 2024-04-12T05:24:05.543 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-827.gd3b5060a.e 1/1 2024-04-12T05:24:05.791 INFO:teuthology.orchestra.run.smithi032.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 74 kB/s | 39 kB 00:00 2024-04-12T05:24:05.916 INFO:teuthology.orchestra.run.smithi032.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 152 kB/s | 57 kB 00:00 2024-04-12T05:24:05.933 INFO:teuthology.orchestra.run.smithi032.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 161 kB/s | 103 kB 00:00 2024-04-12T05:24:05.972 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-827.gd3b5060a.e 1/1 2024-04-12T05:24:06.370 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-827.gd3b5060a.e 1/1 2024-04-12T05:24:06.371 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:24:06.371 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-12T05:24:06.371 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:24:06.371 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:24:06.371 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-12T05:24:06.434 INFO:teuthology.orchestra.run.smithi032.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 481 kB/s | 249 kB 00:00 2024-04-12T05:24:06.605 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr 2024-04-12T05:24:06.793 INFO:teuthology.orchestra.run.smithi032.stdout:(12/32): python3-werkzeug-0.12.2-4.el8.noarch.r 457 kB/s | 457 kB 00:01 2024-04-12T05:24:06.851 INFO:teuthology.orchestra.run.smithi032.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 262 kB/s | 109 kB 00:00 2024-04-12T05:24:07.010 INFO:teuthology.orchestra.run.smithi032.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 596 kB/s | 641 kB 00:01 2024-04-12T05:24:07.133 INFO:teuthology.orchestra.run.smithi032.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 128 kB/s | 44 kB 00:00 2024-04-12T05:24:07.312 INFO:teuthology.orchestra.run.smithi032.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 402 kB/s | 185 kB 00:00 2024-04-12T05:24:07.369 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:31 ago on Fri 12 Apr 2024 05:22:36 AM UTC. 2024-04-12T05:24:07.413 INFO:teuthology.orchestra.run.smithi032.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 111 kB/s | 11 kB 00:00 2024-04-12T05:24:07.455 INFO:teuthology.orchestra.run.smithi032.stdout:(18/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 388 kB/s | 173 kB 00:00 2024-04-12T05:24:07.488 INFO:teuthology.orchestra.run.smithi032.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.1 MB/s | 384 kB 00:00 2024-04-12T05:24:07.522 INFO:teuthology.orchestra.run.smithi032.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 161 kB/s | 18 kB 00:00 2024-04-12T05:24:07.564 INFO:teuthology.orchestra.run.smithi032.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 448 kB/s | 49 kB 00:00 2024-04-12T05:24:07.589 INFO:teuthology.orchestra.run.smithi032.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 583 kB/s | 59 kB 00:00 2024-04-12T05:24:07.702 INFO:teuthology.orchestra.run.smithi032.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 113 kB/s | 16 kB 00:00 2024-04-12T05:24:07.731 INFO:teuthology.orchestra.run.smithi032.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 132 kB/s | 19 kB 00:00 2024-04-12T05:24:07.756 INFO:teuthology.orchestra.run.smithi032.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.2 MB/s | 283 kB 00:00 2024-04-12T05:24:07.798 INFO:teuthology.orchestra.run.smithi032.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 249 kB/s | 24 kB 00:00 2024-04-12T05:24:07.874 INFO:teuthology.orchestra.run.smithi032.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 235 kB/s | 27 kB 00:00 2024-04-12T05:24:07.899 INFO:teuthology.orchestra.run.smithi032.stdout:(28/32): python3-tempora-1.14.1-5.el8.noarch.rp 174 kB/s | 29 kB 00:00 2024-04-12T05:24:07.966 INFO:teuthology.orchestra.run.smithi032.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.5 MB/s | 255 kB 00:00 2024-04-12T05:24:08.008 INFO:teuthology.orchestra.run.smithi032.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.8 MB/s | 251 kB 00:00 2024-04-12T05:24:08.033 INFO:teuthology.orchestra.run.smithi032.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 638 kB/s | 85 kB 00:00 2024-04-12T05:24:08.058 INFO:teuthology.orchestra.run.smithi032.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 250 kB/s | 23 kB 00:00 2024-04-12T05:24:08.061 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:24:08.061 INFO:teuthology.orchestra.run.smithi032.stdout:Total 6.7 MB/s | 29 MB 00:04 2024-04-12T05:24:08.147 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-12T05:24:08.184 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-12T05:24:08.184 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-12T05:24:08.501 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-12T05:24:08.504 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-12T05:24:09.057 INFO:teuthology.orchestra.run.smithi067.stdout:Package ceph-mgr-2:17.2.7-827.gd3b5060a.el8.x86_64 is already installed. 2024-04-12T05:24:09.110 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-12T05:24:09.111 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-04-12T05:24:09.111 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-12T05:24:09.207 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-dashboard 2024-04-12T05:24:09.644 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-12T05:24:09.912 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-12T05:24:09.969 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:33 ago on Fri 12 Apr 2024 05:22:36 AM UTC. 2024-04-12T05:24:10.123 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-12T05:24:10.425 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-04-12T05:24:10.668 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-04-12T05:24:10.866 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-04-12T05:24:11.083 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-04-12T05:24:11.265 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-04-12T05:24:11.459 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-04-12T05:24:11.671 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-04-12T05:24:11.713 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-12T05:24:11.714 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:24:11.715 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-04-12T05:24:11.715 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:24:11.715 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-12T05:24:11.715 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard noarch 2:17.2.7-827.gd3b5060a.el8 ceph-noarch 1.7 M 2024-04-12T05:24:11.715 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-04-12T05:24:11.715 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards noarch 2:17.2.7-827.gd3b5060a.el8 ceph-noarch 24 k 2024-04-12T05:24:11.715 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts noarch 2:17.2.7-827.gd3b5060a.el8 ceph-noarch 14 k 2024-04-12T05:24:11.715 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-04-12T05:24:11.715 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-12T05:24:11.715 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-12T05:24:11.715 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:24:11.715 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-12T05:24:11.715 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:24:11.715 INFO:teuthology.orchestra.run.smithi067.stdout:Install 6 Packages 2024-04-12T05:24:11.716 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:24:11.716 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 2.0 M 2024-04-12T05:24:11.716 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 8.1 M 2024-04-12T05:24:11.716 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-12T05:24:11.874 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-04-12T05:24:12.145 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-04-12T05:24:12.396 INFO:teuthology.orchestra.run.smithi067.stdout:(1/6): ceph-grafana-dashboards-17.2.7-827.gd3b5 97 kB/s | 24 kB 00:00 2024-04-12T05:24:12.397 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-04-12T05:24:12.419 INFO:teuthology.orchestra.run.smithi067.stdout:(2/6): ceph-prometheus-alerts-17.2.7-827.gd3b50 53 kB/s | 14 kB 00:00 2024-04-12T05:24:12.454 INFO:teuthology.orchestra.run.smithi067.stdout:(3/6): ceph-mgr-dashboard-17.2.7-827.gd3b5060a. 5.5 MB/s | 1.7 MB 00:00 2024-04-12T05:24:12.703 INFO:teuthology.orchestra.run.smithi067.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 118 kB/s | 34 kB 00:00 2024-04-12T05:24:12.787 INFO:teuthology.orchestra.run.smithi067.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 109 kB/s | 43 kB 00:00 2024-04-12T05:24:12.813 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-04-12T05:24:12.979 INFO:teuthology.orchestra.run.smithi067.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 374 kB/s | 196 kB 00:00 2024-04-12T05:24:12.980 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:24:12.981 INFO:teuthology.orchestra.run.smithi067.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-04-12T05:24:12.993 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-12T05:24:13.003 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-12T05:24:13.003 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-12T05:24:13.048 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-04-12T05:24:13.099 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-12T05:24:13.102 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-12T05:24:13.295 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-04-12T05:24:13.371 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-12T05:24:13.552 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-04-12T05:24:13.600 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-04-12T05:24:13.780 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-12T05:24:13.852 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-04-12T05:24:14.053 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-04-12T05:24:14.127 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-04-12T05:24:14.157 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-827.gd3b5060a.el8.no 4/6 2024-04-12T05:24:14.377 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-04-12T05:24:14.502 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-827.gd3b5060a.el8.n 5/6 2024-04-12T05:24:14.532 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-04-12T05:24:14.719 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-827.gd3b5060a.el8.noarch 6/6 2024-04-12T05:24:14.725 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-12T05:24:15.003 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-12T05:24:15.016 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-827.gd3b5060a.el8.noarch 6/6 2024-04-12T05:24:15.016 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-827.gd3b5060a.el8.n 1/6 2024-04-12T05:24:15.016 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-827.gd3b5060a.el8.noarch 2/6 2024-04-12T05:24:15.016 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-827.gd3b5060a.el8.no 3/6 2024-04-12T05:24:15.016 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-04-12T05:24:15.016 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-04-12T05:24:15.244 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-12T05:24:15.436 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-04-12T05:24:15.436 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:24:15.436 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-12T05:24:15.436 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards-2:17.2.7-827.gd3b5060a.el8.noarch 2024-04-12T05:24:15.436 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard-2:17.2.7-827.gd3b5060a.el8.noarch 2024-04-12T05:24:15.436 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts-2:17.2.7-827.gd3b5060a.el8.noarch 2024-04-12T05:24:15.436 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-12T05:24:15.436 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-12T05:24:15.436 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-12T05:24:15.436 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:24:15.436 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-12T05:24:15.470 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-04-12T05:24:15.621 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-12T05:24:15.787 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-04-12T05:24:16.131 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-12T05:24:16.380 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:40 ago on Fri 12 Apr 2024 05:22:36 AM UTC. 2024-04-12T05:24:16.506 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-827.gd3b5060a.el8.n 27/32 2024-04-12T05:24:16.759 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-2:17.2.7-827.gd3b5060a.el8.x86_64 28/32 2024-04-12T05:24:18.124 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-12T05:24:18.126 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================================ 2024-04-12T05:24:18.126 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-04-12T05:24:18.126 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================================ 2024-04-12T05:24:18.126 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-12T05:24:18.126 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-827.gd3b5060a.el8 ceph-noarch 7.4 M 2024-04-12T05:24:18.126 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-04-12T05:24:18.126 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-04-12T05:24:18.126 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-04-12T05:24:18.126 INFO:teuthology.orchestra.run.smithi067.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-04-12T05:24:18.126 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-04-12T05:24:18.126 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-04-12T05:24:18.126 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-04-12T05:24:18.126 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-04-12T05:24:18.127 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:24:18.127 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-12T05:24:18.127 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================================ 2024-04-12T05:24:18.127 INFO:teuthology.orchestra.run.smithi067.stdout:Install 8 Packages 2024-04-12T05:24:18.127 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:24:18.127 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 36 M 2024-04-12T05:24:18.127 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 210 M 2024-04-12T05:24:18.127 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-12T05:24:18.377 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-2:17.2.7-827.gd3b5060a.el8.x86_64 28/32 2024-04-12T05:24:19.044 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-osd-2:17.2.7-827.gd3b5060a.el8.x86_64 29/32 2024-04-12T05:24:19.475 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-osd-2:17.2.7-827.gd3b5060a.el8.x86_64 29/32 2024-04-12T05:24:19.668 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mon-2:17.2.7-827.gd3b5060a.el8.x86_64 30/32 2024-04-12T05:24:19.897 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mon-2:17.2.7-827.gd3b5060a.el8.x86_64 30/32 2024-04-12T05:24:20.069 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mds-2:17.2.7-827.gd3b5060a.el8.x86_64 31/32 2024-04-12T05:24:20.087 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mds-2:17.2.7-827.gd3b5060a.el8.x86_64 31/32 2024-04-12T05:24:20.102 INFO:teuthology.orchestra.run.smithi067.stdout:(1/8): ceph-mgr-diskprediction-local-17.2.7-827 4.6 MB/s | 7.4 MB 00:01 2024-04-12T05:24:20.194 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-2:17.2.7-827.gd3b5060a.el8.x86_64 32/32 2024-04-12T05:24:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-2:17.2.7-827.gd3b5060a.el8.x86_64 32/32 2024-04-12T05:24:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-2:17.2.7-827.gd3b5060a.el8.x86_64 1/32 2024-04-12T05:24:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mds-2:17.2.7-827.gd3b5060a.el8.x86_64 2/32 2024-04-12T05:24:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-2:17.2.7-827.gd3b5060a.el8.x86_64 3/32 2024-04-12T05:24:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mon-2:17.2.7-827.gd3b5060a.el8.x86_64 4/32 2024-04-12T05:24:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-osd-2:17.2.7-827.gd3b5060a.el8.x86_64 5/32 2024-04-12T05:24:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-827.gd3b5060a.el8.n 6/32 2024-04-12T05:24:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-04-12T05:24:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-04-12T05:24:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-04-12T05:24:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-04-12T05:24:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-04-12T05:24:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-04-12T05:24:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-04-12T05:24:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-04-12T05:24:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-04-12T05:24:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-04-12T05:24:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-12T05:24:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-04-12T05:24:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-04-12T05:24:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-04-12T05:24:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-04-12T05:24:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-04-12T05:24:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-04-12T05:24:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-04-12T05:24:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-04-12T05:24:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-04-12T05:24:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-04-12T05:24:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-04-12T05:24:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-04-12T05:24:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-04-12T05:24:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-04-12T05:24:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-04-12T05:24:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:24:21.428 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-12T05:24:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:24:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:24:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:24:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-modules-core-2:17.2.7-827.gd3b5060a.el8.noarch 2024-04-12T05:24:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:24:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:24:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-12T05:24:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-12T05:24:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-12T05:24:21.428 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-12T05:24:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-12T05:24:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-04-12T05:24:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-12T05:24:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-12T05:24:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-12T05:24:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-12T05:24:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-12T05:24:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-12T05:24:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-12T05:24:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: python3-portend-2.6-1.el8.noarch 2024-04-12T05:24:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-12T05:24:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-12T05:24:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-12T05:24:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-12T05:24:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-12T05:24:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-12T05:24:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-12T05:24:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-12T05:24:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-12T05:24:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-12T05:24:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-12T05:24:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-12T05:24:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:24:21.430 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-12T05:24:21.664 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-base 2024-04-12T05:24:22.428 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:20 ago on Fri 12 Apr 2024 05:23:02 AM UTC. 2024-04-12T05:24:24.125 INFO:teuthology.orchestra.run.smithi032.stdout:Package ceph-base-2:17.2.7-827.gd3b5060a.el8.x86_64 is already installed. 2024-04-12T05:24:24.176 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-12T05:24:24.177 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-12T05:24:24.177 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-12T05:24:24.275 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install cephadm 2024-04-12T05:24:25.044 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:23 ago on Fri 12 Apr 2024 05:23:02 AM UTC. 2024-04-12T05:24:25.479 INFO:teuthology.orchestra.run.smithi067.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 694 kB/s | 4.7 MB 00:06 2024-04-12T05:24:25.912 INFO:teuthology.orchestra.run.smithi067.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 658 kB/s | 3.7 MB 00:05 2024-04-12T05:24:25.973 INFO:teuthology.orchestra.run.smithi067.stdout:(4/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 666 kB/s | 4.9 MB 00:07 2024-04-12T05:24:26.154 INFO:teuthology.orchestra.run.smithi067.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 3.4 MB/s | 645 kB 00:00 2024-04-12T05:24:26.230 INFO:teuthology.orchestra.run.smithi067.stdout:(6/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 300 kB/s | 225 kB 00:00 2024-04-12T05:24:26.538 INFO:teuthology.orchestra.run.smithi067.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 449 kB/s | 172 kB 00:00 2024-04-12T05:24:26.802 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-12T05:24:26.803 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:24:26.803 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-12T05:24:26.803 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:24:26.803 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-12T05:24:26.803 INFO:teuthology.orchestra.run.smithi032.stdout: cephadm noarch 2:17.2.7-827.gd3b5060a.el8 ceph-noarch 82 k 2024-04-12T05:24:26.803 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:24:26.803 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-12T05:24:26.803 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:24:26.803 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-04-12T05:24:26.803 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:24:26.804 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 82 k 2024-04-12T05:24:26.804 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 372 k 2024-04-12T05:24:26.804 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-12T05:24:26.973 INFO:teuthology.orchestra.run.smithi032.stdout:cephadm-17.2.7-827.gd3b5060a.el8.noarch.rpm 485 kB/s | 82 kB 00:00 2024-04-12T05:24:26.974 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:24:26.974 INFO:teuthology.orchestra.run.smithi032.stdout:Total 481 kB/s | 82 kB 00:00 2024-04-12T05:24:26.974 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-12T05:24:26.983 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-12T05:24:26.983 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-12T05:24:26.985 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-12T05:24:26.988 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-12T05:24:27.035 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-12T05:24:27.562 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cephadm-2:17.2.7-827.gd3b5060a.el8.noarch 1/1 2024-04-12T05:24:27.745 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephadm-2:17.2.7-827.gd3b5060a.el8.noarch 1/1 2024-04-12T05:24:27.984 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cephadm-2:17.2.7-827.gd3b5060a.el8.noarch 1/1 2024-04-12T05:24:28.391 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephadm-2:17.2.7-827.gd3b5060a.el8.noarch 1/1 2024-04-12T05:24:28.391 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:24:28.392 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-12T05:24:28.392 INFO:teuthology.orchestra.run.smithi032.stdout: cephadm-2:17.2.7-827.gd3b5060a.el8.noarch 2024-04-12T05:24:28.392 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:24:28.392 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-12T05:24:28.654 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-immutable-object-cache 2024-04-12T05:24:29.414 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:27 ago on Fri 12 Apr 2024 05:23:02 AM UTC. 2024-04-12T05:24:31.177 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-12T05:24:31.178 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:24:31.178 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-04-12T05:24:31.179 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:24:31.179 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-12T05:24:31.179 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 162 k 2024-04-12T05:24:31.179 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:24:31.179 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-12T05:24:31.179 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:24:31.179 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-04-12T05:24:31.179 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:24:31.179 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 162 k 2024-04-12T05:24:31.179 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 494 k 2024-04-12T05:24:31.179 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-12T05:24:31.702 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-immutable-object-cache-17.2.7-827.gd3b5060 310 kB/s | 162 kB 00:00 2024-04-12T05:24:31.703 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:24:31.703 INFO:teuthology.orchestra.run.smithi032.stdout:Total 309 kB/s | 162 kB 00:00 2024-04-12T05:24:31.703 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-12T05:24:31.716 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-12T05:24:31.716 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-12T05:24:31.770 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-12T05:24:31.774 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-12T05:24:31.900 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-12T05:24:32.100 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-827.gd3b5060a.e 1/1 2024-04-12T05:24:32.529 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-827.gd3b5060a.e 1/1 2024-04-12T05:24:32.925 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-827.gd3b5060a.e 1/1 2024-04-12T05:24:32.925 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:24:32.925 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-12T05:24:32.925 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:24:32.925 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:24:32.926 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-12T05:24:33.123 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr 2024-04-12T05:24:33.890 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:31 ago on Fri 12 Apr 2024 05:23:02 AM UTC. 2024-04-12T05:24:35.599 INFO:teuthology.orchestra.run.smithi032.stdout:Package ceph-mgr-2:17.2.7-827.gd3b5060a.el8.x86_64 is already installed. 2024-04-12T05:24:35.653 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-12T05:24:35.654 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-12T05:24:35.654 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-12T05:24:35.755 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr-dashboard 2024-04-12T05:24:36.514 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:34 ago on Fri 12 Apr 2024 05:23:02 AM UTC. 2024-04-12T05:24:38.266 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-12T05:24:38.267 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:24:38.267 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-12T05:24:38.267 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:24:38.267 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-12T05:24:38.267 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-dashboard noarch 2:17.2.7-827.gd3b5060a.el8 ceph-noarch 1.7 M 2024-04-12T05:24:38.267 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-12T05:24:38.267 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-grafana-dashboards noarch 2:17.2.7-827.gd3b5060a.el8 ceph-noarch 24 k 2024-04-12T05:24:38.267 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-prometheus-alerts noarch 2:17.2.7-827.gd3b5060a.el8 ceph-noarch 14 k 2024-04-12T05:24:38.267 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-04-12T05:24:38.267 INFO:teuthology.orchestra.run.smithi032.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-12T05:24:38.267 INFO:teuthology.orchestra.run.smithi032.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-12T05:24:38.267 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:24:38.267 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-12T05:24:38.268 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:24:38.268 INFO:teuthology.orchestra.run.smithi032.stdout:Install 6 Packages 2024-04-12T05:24:38.268 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:24:38.268 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 2.0 M 2024-04-12T05:24:38.268 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 8.1 M 2024-04-12T05:24:38.268 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-12T05:24:38.621 INFO:teuthology.orchestra.run.smithi032.stdout:(1/6): ceph-prometheus-alerts-17.2.7-827.gd3b50 89 kB/s | 14 kB 00:00 2024-04-12T05:24:38.646 INFO:teuthology.orchestra.run.smithi032.stdout:(2/6): ceph-grafana-dashboards-17.2.7-827.gd3b5 129 kB/s | 24 kB 00:00 2024-04-12T05:24:38.869 INFO:teuthology.orchestra.run.smithi032.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 173 kB/s | 43 kB 00:00 2024-04-12T05:24:39.064 INFO:teuthology.orchestra.run.smithi032.stdout:(4/6): ceph-mgr-dashboard-17.2.7-827.gd3b5060a. 2.8 MB/s | 1.7 MB 00:00 2024-04-12T05:24:39.181 INFO:teuthology.orchestra.run.smithi032.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 63 kB/s | 34 kB 00:00 2024-04-12T05:24:39.490 INFO:teuthology.orchestra.run.smithi032.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 319 kB/s | 196 kB 00:00 2024-04-12T05:24:39.493 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:24:39.493 INFO:teuthology.orchestra.run.smithi032.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-04-12T05:24:39.505 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-12T05:24:39.515 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-12T05:24:39.515 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-12T05:24:39.602 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-12T05:24:39.606 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-12T05:24:39.876 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-12T05:24:40.133 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-04-12T05:24:40.436 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-04-12T05:24:40.635 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-04-12T05:24:40.749 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-827.gd3b5060a.el8.no 4/6 2024-04-12T05:24:41.120 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-827.gd3b5060a.el8.n 5/6 2024-04-12T05:24:41.313 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-827.gd3b5060a.el8.noarch 6/6 2024-04-12T05:24:41.596 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-827.gd3b5060a.el8.noarch 6/6 2024-04-12T05:24:41.597 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-827.gd3b5060a.el8.n 1/6 2024-04-12T05:24:41.597 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-827.gd3b5060a.el8.noarch 2/6 2024-04-12T05:24:41.597 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-827.gd3b5060a.el8.no 3/6 2024-04-12T05:24:41.597 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-04-12T05:24:41.597 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-04-12T05:24:42.025 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-04-12T05:24:42.025 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:24:42.026 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-12T05:24:42.026 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-grafana-dashboards-2:17.2.7-827.gd3b5060a.el8.noarch 2024-04-12T05:24:42.026 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-dashboard-2:17.2.7-827.gd3b5060a.el8.noarch 2024-04-12T05:24:42.026 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-prometheus-alerts-2:17.2.7-827.gd3b5060a.el8.noarch 2024-04-12T05:24:42.026 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-12T05:24:42.026 INFO:teuthology.orchestra.run.smithi032.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-12T05:24:42.026 INFO:teuthology.orchestra.run.smithi032.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-12T05:24:42.026 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:24:42.026 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-12T05:24:42.185 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-12T05:24:42.948 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:40 ago on Fri 12 Apr 2024 05:23:02 AM UTC. 2024-04-12T05:24:44.694 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-12T05:24:44.695 INFO:teuthology.orchestra.run.smithi032.stdout:============================================================================================ 2024-04-12T05:24:44.695 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-12T05:24:44.695 INFO:teuthology.orchestra.run.smithi032.stdout:============================================================================================ 2024-04-12T05:24:44.695 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-12T05:24:44.695 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-827.gd3b5060a.el8 ceph-noarch 7.4 M 2024-04-12T05:24:44.695 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-12T05:24:44.695 INFO:teuthology.orchestra.run.smithi032.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-04-12T05:24:44.695 INFO:teuthology.orchestra.run.smithi032.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-04-12T05:24:44.696 INFO:teuthology.orchestra.run.smithi032.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-04-12T05:24:44.696 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-04-12T05:24:44.696 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-04-12T05:24:44.696 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-04-12T05:24:44.696 INFO:teuthology.orchestra.run.smithi032.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-04-12T05:24:44.696 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:24:44.696 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-12T05:24:44.696 INFO:teuthology.orchestra.run.smithi032.stdout:============================================================================================ 2024-04-12T05:24:44.696 INFO:teuthology.orchestra.run.smithi032.stdout:Install 8 Packages 2024-04-12T05:24:44.696 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:24:44.697 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 36 M 2024-04-12T05:24:44.697 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 210 M 2024-04-12T05:24:44.697 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-12T05:24:45.579 INFO:teuthology.orchestra.run.smithi032.stdout:(1/8): ceph-mgr-diskprediction-local-17.2.7-827 12 MB/s | 7.4 MB 00:00 2024-04-12T05:24:46.521 INFO:teuthology.orchestra.run.smithi067.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 704 kB/s | 14 MB 00:20 2024-04-12T05:24:46.522 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:24:46.522 INFO:teuthology.orchestra.run.smithi067.stdout:Total 1.3 MB/s | 36 MB 00:28 2024-04-12T05:24:46.759 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-12T05:24:46.778 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-12T05:24:46.778 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-12T05:24:47.183 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-12T05:24:47.186 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-12T05:24:47.952 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-12T05:24:48.129 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-12T05:24:48.282 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-12T05:24:48.421 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-12T05:24:49.055 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-12T05:24:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-04-12T05:24:50.023 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-04-12T05:24:50.154 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-12T05:24:51.004 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-12T05:24:51.031 INFO:teuthology.orchestra.run.smithi032.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 701 kB/s | 3.7 MB 00:05 2024-04-12T05:24:51.338 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-12T05:24:51.612 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-12T05:24:51.698 INFO:teuthology.orchestra.run.smithi032.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 717 kB/s | 4.7 MB 00:06 2024-04-12T05:24:51.723 INFO:teuthology.orchestra.run.smithi032.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 325 kB/s | 225 kB 00:00 2024-04-12T05:24:52.067 INFO:teuthology.orchestra.run.smithi032.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.8 MB/s | 645 kB 00:00 2024-04-12T05:24:52.305 INFO:teuthology.orchestra.run.smithi032.stdout:(6/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 675 kB/s | 4.9 MB 00:07 2024-04-12T05:24:52.724 INFO:teuthology.orchestra.run.smithi032.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 263 kB/s | 172 kB 00:00 2024-04-12T05:24:53.510 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-12T05:24:54.783 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-04-12T05:24:54.970 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-827.gd3b5060a 8/8 2024-04-12T05:24:55.325 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-827.gd3b5060a 8/8 2024-04-12T05:24:55.326 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-827.gd3b5060a 1/8 2024-04-12T05:24:55.326 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-04-12T05:24:55.326 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-04-12T05:24:55.326 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-12T05:24:55.326 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-12T05:24:55.326 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-04-12T05:24:55.326 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-04-12T05:24:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-04-12T05:24:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:24:55.768 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-12T05:24:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local-2:17.2.7-827.gd3b5060a.el8.noarch 2024-04-12T05:24:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-04-12T05:24:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-04-12T05:24:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-0.3.15-4.el8.x86_64 2024-04-12T05:24:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-04-12T05:24:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-12T05:24:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-12T05:24:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-04-12T05:24:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:24:55.769 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-12T05:24:55.890 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-rook 2024-04-12T05:24:56.649 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:20 ago on Fri 12 Apr 2024 05:22:36 AM UTC. 2024-04-12T05:24:58.395 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-12T05:24:58.397 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:24:58.397 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-04-12T05:24:58.397 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:24:58.397 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-12T05:24:58.397 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook noarch 2:17.2.7-827.gd3b5060a.el8 ceph-noarch 47 k 2024-04-12T05:24:58.397 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-04-12T05:24:58.397 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-12T05:24:58.397 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-12T05:24:58.397 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-12T05:24:58.397 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-04-12T05:24:58.397 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-04-12T05:24:58.397 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-12T05:24:58.397 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-04-12T05:24:58.398 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-04-12T05:24:58.398 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-04-12T05:24:58.398 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-04-12T05:24:58.398 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-12T05:24:58.398 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-12T05:24:58.398 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:24:58.398 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-12T05:24:58.398 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:24:58.398 INFO:teuthology.orchestra.run.smithi067.stdout:Install 13 Packages 2024-04-12T05:24:58.398 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:24:58.399 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 1.9 M 2024-04-12T05:24:58.399 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 25 M 2024-04-12T05:24:58.399 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-12T05:24:59.143 INFO:teuthology.orchestra.run.smithi067.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 67 kB/s | 20 kB 00:00 2024-04-12T05:24:59.176 INFO:teuthology.orchestra.run.smithi067.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 80 kB/s | 27 kB 00:00 2024-04-12T05:24:59.360 INFO:teuthology.orchestra.run.smithi067.stdout:(3/13): ceph-mgr-rook-17.2.7-827.gd3b5060a.el8. 92 kB/s | 47 kB 00:00 2024-04-12T05:24:59.451 INFO:teuthology.orchestra.run.smithi067.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.7 MB/s | 155 kB 00:00 2024-04-12T05:24:59.518 INFO:teuthology.orchestra.run.smithi067.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 640 kB/s | 43 kB 00:00 2024-04-12T05:24:59.636 INFO:teuthology.orchestra.run.smithi067.stdout:(6/13): python3-pyasn1-modules-0.3.7-6.el8.noar 240 kB/s | 110 kB 00:00 2024-04-12T05:24:59.686 INFO:teuthology.orchestra.run.smithi067.stdout:(7/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 233 kB/s | 126 kB 00:00 2024-04-12T05:24:59.786 INFO:teuthology.orchestra.run.smithi067.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 123 kB/s | 33 kB 00:00 2024-04-12T05:24:59.903 INFO:teuthology.orchestra.run.smithi067.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 61 kB/s | 16 kB 00:00 2024-04-12T05:25:00.053 INFO:teuthology.orchestra.run.smithi067.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 221 kB/s | 81 kB 00:00 2024-04-12T05:25:00.262 INFO:teuthology.orchestra.run.smithi067.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 179 kB/s | 64 kB 00:00 2024-04-12T05:25:00.404 INFO:teuthology.orchestra.run.smithi067.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 175 kB/s | 61 kB 00:00 2024-04-12T05:25:00.470 INFO:teuthology.orchestra.run.smithi067.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 1.7 MB/s | 1.2 MB 00:00 2024-04-12T05:25:00.471 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:25:00.472 INFO:teuthology.orchestra.run.smithi067.stdout:Total 954 kB/s | 1.9 MB 00:02 2024-04-12T05:25:00.531 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-12T05:25:00.548 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-12T05:25:00.548 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-12T05:25:00.644 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-12T05:25:00.647 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-12T05:25:01.208 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-12T05:25:01.458 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-04-12T05:25:01.747 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-04-12T05:25:01.997 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-04-12T05:25:02.218 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-04-12T05:25:02.402 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-04-12T05:25:02.654 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-04-12T05:25:02.907 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-04-12T05:25:03.160 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-04-12T05:25:04.246 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-04-12T05:25:04.518 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-04-12T05:25:04.710 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-04-12T05:25:04.912 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-04-12T05:25:05.069 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-rook-2:17.2.7-827.gd3b5060a.el8.noarch 13/13 2024-04-12T05:25:05.378 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-827.gd3b5060a.el8.noarch 13/13 2024-04-12T05:25:05.378 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-rook-2:17.2.7-827.gd3b5060a.el8.noarch 1/13 2024-04-12T05:25:05.378 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-04-12T05:25:05.378 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-04-12T05:25:05.378 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-04-12T05:25:05.378 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-04-12T05:25:05.378 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-04-12T05:25:05.378 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-04-12T05:25:05.378 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-04-12T05:25:05.379 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-04-12T05:25:05.379 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-04-12T05:25:05.379 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-04-12T05:25:05.379 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-04-12T05:25:05.873 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-04-12T05:25:05.873 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:25:05.873 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-12T05:25:05.873 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook-2:17.2.7-827.gd3b5060a.el8.noarch 2024-04-12T05:25:05.873 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-12T05:25:05.873 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-12T05:25:05.873 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-12T05:25:05.873 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-12T05:25:05.873 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-12T05:25:05.874 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-12T05:25:05.874 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-12T05:25:05.874 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-12T05:25:05.874 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-12T05:25:05.874 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-12T05:25:05.874 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-12T05:25:05.874 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-12T05:25:05.874 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:25:05.874 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-12T05:25:05.989 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-cephadm 2024-04-12T05:25:06.754 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:30 ago on Fri 12 Apr 2024 05:22:36 AM UTC. 2024-04-12T05:25:08.497 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-12T05:25:08.498 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:25:08.498 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-04-12T05:25:08.498 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:25:08.498 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-12T05:25:08.498 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm noarch 2:17.2.7-827.gd3b5060a.el8 ceph-noarch 123 k 2024-04-12T05:25:08.498 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-04-12T05:25:08.498 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-12T05:25:08.498 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-04-12T05:25:08.498 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-04-12T05:25:08.499 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-12T05:25:08.499 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:25:08.499 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-12T05:25:08.499 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:25:08.499 INFO:teuthology.orchestra.run.smithi067.stdout:Install 5 Packages 2024-04-12T05:25:08.499 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:25:08.499 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 6.0 M 2024-04-12T05:25:08.499 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 26 M 2024-04-12T05:25:08.500 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-12T05:25:09.319 INFO:teuthology.orchestra.run.smithi067.stdout:(1/5): ceph-mgr-cephadm-17.2.7-827.gd3b5060a.el 205 kB/s | 123 kB 00:00 2024-04-12T05:25:09.828 INFO:teuthology.orchestra.run.smithi067.stdout:(2/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 936 kB/s | 476 kB 00:00 2024-04-12T05:25:09.894 INFO:teuthology.orchestra.run.smithi067.stdout:(3/5): python3-jinja2-2.10.1-4.el8.noarch.rpm 492 kB/s | 578 kB 00:01 2024-04-12T05:25:09.936 INFO:teuthology.orchestra.run.smithi067.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 555 kB/s | 60 kB 00:00 2024-04-12T05:25:11.664 INFO:teuthology.orchestra.run.smithi032.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 727 kB/s | 14 MB 00:19 2024-04-12T05:25:11.665 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:25:11.666 INFO:teuthology.orchestra.run.smithi032.stdout:Total 1.3 MB/s | 36 MB 00:26 2024-04-12T05:25:11.904 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-12T05:25:11.923 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-12T05:25:11.923 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-12T05:25:12.319 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-12T05:25:12.322 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-12T05:25:12.908 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-12T05:25:13.078 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-12T05:25:13.200 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-12T05:25:13.482 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-12T05:25:14.101 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-12T05:25:14.278 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-04-12T05:25:14.934 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-04-12T05:25:15.057 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-12T05:25:15.736 INFO:teuthology.orchestra.run.smithi067.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 699 kB/s | 4.8 MB 00:07 2024-04-12T05:25:15.737 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:25:15.737 INFO:teuthology.orchestra.run.smithi067.stdout:Total 848 kB/s | 6.0 MB 00:07 2024-04-12T05:25:15.791 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-12T05:25:15.792 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-12T05:25:15.805 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-12T05:25:15.805 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-12T05:25:15.920 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-12T05:25:15.924 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-12T05:25:16.112 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-12T05:25:16.197 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-12T05:25:16.365 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-12T05:25:16.532 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2024-04-12T05:25:17.531 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-04-12T05:25:17.854 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2024-04-12T05:25:18.180 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 4/5 2024-04-12T05:25:18.257 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-12T05:25:18.376 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-827.gd3b5060a.el8.noarch 5/5 2024-04-12T05:25:18.681 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-827.gd3b5060a.el8.noarch 5/5 2024-04-12T05:25:18.681 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-827.gd3b5060a.el8.noarch 1/5 2024-04-12T05:25:18.681 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2024-04-12T05:25:18.681 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/5 2024-04-12T05:25:18.681 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-04-12T05:25:19.088 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2024-04-12T05:25:19.088 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:25:19.088 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-12T05:25:19.088 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm-2:17.2.7-827.gd3b5060a.el8.noarch 2024-04-12T05:25:19.088 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-12T05:25:19.088 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-12T05:25:19.089 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-04-12T05:25:19.089 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-12T05:25:19.089 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:25:19.089 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-12T05:25:19.206 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-fuse 2024-04-12T05:25:19.602 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-04-12T05:25:19.802 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-827.gd3b5060a 8/8 2024-04-12T05:25:19.975 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:43 ago on Fri 12 Apr 2024 05:22:36 AM UTC. 2024-04-12T05:25:20.139 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-827.gd3b5060a 8/8 2024-04-12T05:25:20.139 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-827.gd3b5060a 1/8 2024-04-12T05:25:20.139 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-04-12T05:25:20.140 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-04-12T05:25:20.140 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-12T05:25:20.140 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-12T05:25:20.140 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-04-12T05:25:20.140 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-04-12T05:25:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-04-12T05:25:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:25:20.605 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-12T05:25:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-diskprediction-local-2:17.2.7-827.gd3b5060a.el8.noarch 2024-04-12T05:25:20.605 INFO:teuthology.orchestra.run.smithi032.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-04-12T05:25:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-04-12T05:25:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-0.3.15-4.el8.x86_64 2024-04-12T05:25:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-04-12T05:25:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-12T05:25:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-12T05:25:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-04-12T05:25:20.606 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:25:20.606 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-12T05:25:20.746 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr-rook 2024-04-12T05:25:21.513 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:02:19 ago on Fri 12 Apr 2024 05:23:02 AM UTC. 2024-04-12T05:25:21.733 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-12T05:25:21.734 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:25:21.734 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-04-12T05:25:21.734 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:25:21.734 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-12T05:25:21.734 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 876 k 2024-04-12T05:25:21.734 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-04-12T05:25:21.734 INFO:teuthology.orchestra.run.smithi067.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-04-12T05:25:21.734 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:25:21.734 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-12T05:25:21.735 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:25:21.735 INFO:teuthology.orchestra.run.smithi067.stdout:Install 2 Packages 2024-04-12T05:25:21.735 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:25:21.735 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 959 k 2024-04-12T05:25:21.735 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 3.1 M 2024-04-12T05:25:21.735 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-12T05:25:22.027 INFO:teuthology.orchestra.run.smithi067.stdout:(1/2): fuse-2.9.7-19.el8.x86_64.rpm 753 kB/s | 83 kB 00:00 2024-04-12T05:25:22.160 INFO:teuthology.orchestra.run.smithi067.stdout:(2/2): ceph-fuse-17.2.7-827.gd3b5060a.el8.x86_6 3.5 MB/s | 876 kB 00:00 2024-04-12T05:25:22.161 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:25:22.161 INFO:teuthology.orchestra.run.smithi067.stdout:Total 2.2 MB/s | 959 kB 00:00 2024-04-12T05:25:22.172 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-12T05:25:22.193 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-12T05:25:22.194 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-12T05:25:22.285 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-12T05:25:22.289 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-12T05:25:22.463 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-12T05:25:22.747 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-04-12T05:25:22.942 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-fuse-2:17.2.7-827.gd3b5060a.el8.x86_64 2/2 2024-04-12T05:25:23.281 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-12T05:25:23.283 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:25:23.283 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-12T05:25:23.283 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:25:23.283 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-12T05:25:23.283 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-rook noarch 2:17.2.7-827.gd3b5060a.el8 ceph-noarch 47 k 2024-04-12T05:25:23.283 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-12T05:25:23.284 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-12T05:25:23.284 INFO:teuthology.orchestra.run.smithi032.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-12T05:25:23.284 INFO:teuthology.orchestra.run.smithi032.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-12T05:25:23.284 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-04-12T05:25:23.284 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-04-12T05:25:23.284 INFO:teuthology.orchestra.run.smithi032.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-12T05:25:23.284 INFO:teuthology.orchestra.run.smithi032.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-04-12T05:25:23.284 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-04-12T05:25:23.284 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-04-12T05:25:23.284 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-04-12T05:25:23.284 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-12T05:25:23.284 INFO:teuthology.orchestra.run.smithi032.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-12T05:25:23.284 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:25:23.284 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-12T05:25:23.285 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:25:23.285 INFO:teuthology.orchestra.run.smithi032.stdout:Install 13 Packages 2024-04-12T05:25:23.285 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:25:23.285 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 1.9 M 2024-04-12T05:25:23.285 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 25 M 2024-04-12T05:25:23.285 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-12T05:25:23.345 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-fuse-2:17.2.7-827.gd3b5060a.el8.x86_64 2/2 2024-04-12T05:25:23.345 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-fuse-2:17.2.7-827.gd3b5060a.el8.x86_64 1/2 2024-04-12T05:25:24.154 INFO:teuthology.orchestra.run.smithi032.stdout:(1/13): ceph-mgr-rook-17.2.7-827.gd3b5060a.el8. 177 kB/s | 47 kB 00:00 2024-04-12T05:25:24.178 INFO:teuthology.orchestra.run.smithi032.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 91 kB/s | 27 kB 00:00 2024-04-12T05:25:24.204 INFO:teuthology.orchestra.run.smithi032.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 63 kB/s | 20 kB 00:00 2024-04-12T05:25:24.279 INFO:teuthology.orchestra.run.smithi032.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 2.0 MB/s | 155 kB 00:00 2024-04-12T05:25:24.338 INFO:teuthology.orchestra.run.smithi032.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 733 kB/s | 43 kB 00:00 2024-04-12T05:25:24.515 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-04-12T05:25:24.516 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:25:24.516 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-12T05:25:24.516 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse-2:17.2.7-827.gd3b5060a.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-04-12T05:25:24.516 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:25:24.516 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-12T05:25:24.646 INFO:teuthology.orchestra.run.smithi032.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 107 kB/s | 33 kB 00:00 2024-04-12T05:25:24.680 INFO:teuthology.orchestra.run.smithi032.stdout:(7/13): python3-pyasn1-modules-0.3.7-6.el8.noar 219 kB/s | 110 kB 00:00 2024-04-12T05:25:24.705 INFO:teuthology.orchestra.run.smithi032.stdout:(8/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 229 kB/s | 126 kB 00:00 2024-04-12T05:25:24.764 INFO:teuthology.orchestra.run.smithi032.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 139 kB/s | 16 kB 00:00 2024-04-12T05:25:24.981 INFO:teuthology.orchestra.run.smithi032.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 296 kB/s | 64 kB 00:00 2024-04-12T05:25:25.082 INFO:teuthology.orchestra.run.smithi032.stdout:(11/13): python3-google-auth-1.1.1-10.el8.noarc 203 kB/s | 81 kB 00:00 2024-04-12T05:25:25.106 INFO:teuthology.orchestra.run.smithi032.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 490 kB/s | 61 kB 00:00 2024-04-12T05:25:25.398 INFO:teuthology.orchestra.run.smithi032.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 1.7 MB/s | 1.2 MB 00:00 2024-04-12T05:25:25.399 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:25:25.399 INFO:teuthology.orchestra.run.smithi032.stdout:Total 935 kB/s | 1.9 MB 00:02 2024-04-12T05:25:25.460 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-12T05:25:25.477 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-12T05:25:25.477 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-12T05:25:25.571 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-12T05:25:25.575 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-12T05:25:25.745 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librados-devel 2024-04-12T05:25:26.118 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-12T05:25:26.374 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-04-12T05:25:26.519 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:50 ago on Fri 12 Apr 2024 05:22:36 AM UTC. 2024-04-12T05:25:26.553 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-04-12T05:25:26.789 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-04-12T05:25:27.042 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-04-12T05:25:27.234 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-04-12T05:25:27.485 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-04-12T05:25:27.762 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-04-12T05:25:28.016 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-04-12T05:25:28.264 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-12T05:25:28.265 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:25:28.265 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-04-12T05:25:28.265 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:25:28.265 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-12T05:25:28.265 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 130 k 2024-04-12T05:25:28.265 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:25:28.265 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-12T05:25:28.265 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:25:28.265 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-04-12T05:25:28.266 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:25:28.266 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 130 k 2024-04-12T05:25:28.266 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 464 k 2024-04-12T05:25:28.266 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-12T05:25:28.443 INFO:teuthology.orchestra.run.smithi067.stdout:librados-devel-17.2.7-827.gd3b5060a.el8.x86_64. 734 kB/s | 130 kB 00:00 2024-04-12T05:25:28.444 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:25:28.444 INFO:teuthology.orchestra.run.smithi067.stdout:Total 729 kB/s | 130 kB 00:00 2024-04-12T05:25:28.444 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-12T05:25:28.450 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-12T05:25:28.450 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-12T05:25:28.503 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-12T05:25:28.506 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-12T05:25:28.621 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-12T05:25:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librados-devel-2:17.2.7-827.gd3b5060a.el8.x86_64 1/1 2024-04-12T05:25:29.125 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-04-12T05:25:29.174 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados-devel-2:17.2.7-827.gd3b5060a.el8.x86_64 1/1 2024-04-12T05:25:29.392 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-04-12T05:25:29.622 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados-devel-2:17.2.7-827.gd3b5060a.el8.x86_64 1/1 2024-04-12T05:25:29.622 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:25:29.622 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-12T05:25:29.622 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:25:29.622 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:25:29.622 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-12T05:25:29.654 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-04-12T05:25:29.836 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install libcephfs2 2024-04-12T05:25:29.854 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-04-12T05:25:30.001 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-rook-2:17.2.7-827.gd3b5060a.el8.noarch 13/13 2024-04-12T05:25:30.309 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-827.gd3b5060a.el8.noarch 13/13 2024-04-12T05:25:30.310 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-rook-2:17.2.7-827.gd3b5060a.el8.noarch 1/13 2024-04-12T05:25:30.310 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-04-12T05:25:30.310 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-04-12T05:25:30.310 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-04-12T05:25:30.310 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-04-12T05:25:30.310 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-04-12T05:25:30.310 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-04-12T05:25:30.310 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-04-12T05:25:30.310 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-04-12T05:25:30.310 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-04-12T05:25:30.310 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-04-12T05:25:30.310 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-04-12T05:25:30.605 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:54 ago on Fri 12 Apr 2024 05:22:36 AM UTC. 2024-04-12T05:25:30.779 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-04-12T05:25:30.779 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:25:30.780 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-12T05:25:30.780 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-rook-2:17.2.7-827.gd3b5060a.el8.noarch 2024-04-12T05:25:30.780 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-12T05:25:30.780 INFO:teuthology.orchestra.run.smithi032.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-12T05:25:30.780 INFO:teuthology.orchestra.run.smithi032.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-12T05:25:30.780 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-12T05:25:30.780 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-12T05:25:30.780 INFO:teuthology.orchestra.run.smithi032.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-12T05:25:30.781 INFO:teuthology.orchestra.run.smithi032.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-12T05:25:30.781 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-12T05:25:30.781 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-12T05:25:30.781 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-12T05:25:30.781 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-12T05:25:30.781 INFO:teuthology.orchestra.run.smithi032.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-12T05:25:30.781 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:25:30.781 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-12T05:25:30.898 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr-cephadm 2024-04-12T05:25:31.680 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:02:29 ago on Fri 12 Apr 2024 05:23:02 AM UTC. 2024-04-12T05:25:32.299 INFO:teuthology.orchestra.run.smithi067.stdout:Package libcephfs2-2:17.2.7-827.gd3b5060a.el8.x86_64 is already installed. 2024-04-12T05:25:32.353 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-12T05:25:32.354 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-04-12T05:25:32.354 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-12T05:25:32.470 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install libcephfs-devel 2024-04-12T05:25:33.232 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:57 ago on Fri 12 Apr 2024 05:22:36 AM UTC. 2024-04-12T05:25:33.561 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-12T05:25:33.562 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:25:33.562 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-12T05:25:33.562 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:25:33.562 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-12T05:25:33.562 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-cephadm noarch 2:17.2.7-827.gd3b5060a.el8 ceph-noarch 123 k 2024-04-12T05:25:33.562 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-12T05:25:33.562 INFO:teuthology.orchestra.run.smithi032.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-12T05:25:33.562 INFO:teuthology.orchestra.run.smithi032.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-04-12T05:25:33.562 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-04-12T05:25:33.563 INFO:teuthology.orchestra.run.smithi032.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-12T05:25:33.563 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:25:33.563 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-12T05:25:33.563 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:25:33.563 INFO:teuthology.orchestra.run.smithi032.stdout:Install 5 Packages 2024-04-12T05:25:33.563 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:25:33.563 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 6.0 M 2024-04-12T05:25:33.564 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 26 M 2024-04-12T05:25:33.564 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-12T05:25:34.105 INFO:teuthology.orchestra.run.smithi032.stdout:(1/5): ceph-mgr-cephadm-17.2.7-827.gd3b5060a.el 348 kB/s | 123 kB 00:00 2024-04-12T05:25:34.213 INFO:teuthology.orchestra.run.smithi032.stdout:(2/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 4.3 MB/s | 476 kB 00:00 2024-04-12T05:25:34.247 INFO:teuthology.orchestra.run.smithi032.stdout:(3/5): python3-natsort-7.1.1-2.el8.noarch.rpm 1.8 MB/s | 60 kB 00:00 2024-04-12T05:25:34.855 INFO:teuthology.orchestra.run.smithi032.stdout:(4/5): python3-jinja2-2.10.1-4.el8.noarch.rpm 523 kB/s | 578 kB 00:01 2024-04-12T05:25:34.993 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-12T05:25:34.994 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:25:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-04-12T05:25:34.994 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:25:34.994 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-12T05:25:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 25 k 2024-04-12T05:25:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:25:34.995 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-12T05:25:34.995 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:25:34.995 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-04-12T05:25:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:25:34.995 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 25 k 2024-04-12T05:25:34.995 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 109 k 2024-04-12T05:25:34.995 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-12T05:25:35.337 INFO:teuthology.orchestra.run.smithi067.stdout:libcephfs-devel-17.2.7-827.gd3b5060a.el8.x86_64 72 kB/s | 25 kB 00:00 2024-04-12T05:25:35.338 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:25:35.338 INFO:teuthology.orchestra.run.smithi067.stdout:Total 72 kB/s | 25 kB 00:00 2024-04-12T05:25:35.339 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-12T05:25:35.342 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-12T05:25:35.343 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-12T05:25:35.345 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-12T05:25:35.348 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-12T05:25:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-12T05:25:35.573 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs-devel-2:17.2.7-827.gd3b5060a.el8.x86_64 1/1 2024-04-12T05:25:35.841 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-827.gd3b5060a.el8.x86_64 1/1 2024-04-12T05:25:36.342 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs-devel-2:17.2.7-827.gd3b5060a.el8.x86_64 1/1 2024-04-12T05:25:36.342 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:25:36.342 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-12T05:25:36.343 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:25:36.343 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:25:36.343 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-12T05:25:36.489 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librados2 2024-04-12T05:25:37.252 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:03:01 ago on Fri 12 Apr 2024 05:22:36 AM UTC. 2024-04-12T05:25:38.931 INFO:teuthology.orchestra.run.smithi067.stdout:Package librados2-2:17.2.7-827.gd3b5060a.el8.x86_64 is already installed. 2024-04-12T05:25:38.983 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-12T05:25:38.984 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-04-12T05:25:38.984 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-12T05:25:39.089 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librbd1 2024-04-12T05:25:39.852 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:03:03 ago on Fri 12 Apr 2024 05:22:36 AM UTC. 2024-04-12T05:25:40.589 INFO:teuthology.orchestra.run.smithi032.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 717 kB/s | 4.8 MB 00:06 2024-04-12T05:25:40.589 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:25:40.590 INFO:teuthology.orchestra.run.smithi032.stdout:Total 874 kB/s | 6.0 MB 00:07 2024-04-12T05:25:40.645 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-12T05:25:40.657 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-12T05:25:40.658 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-12T05:25:40.765 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-12T05:25:40.769 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-12T05:25:41.040 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-12T05:25:41.391 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2024-04-12T05:25:41.531 INFO:teuthology.orchestra.run.smithi067.stdout:Package librbd1-2:17.2.7-827.gd3b5060a.el8.x86_64 is already installed. 2024-04-12T05:25:41.584 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-12T05:25:41.585 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-04-12T05:25:41.585 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-12T05:25:41.691 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rados 2024-04-12T05:25:42.451 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:03:06 ago on Fri 12 Apr 2024 05:22:36 AM UTC. 2024-04-12T05:25:42.466 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-04-12T05:25:42.854 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2024-04-12T05:25:43.116 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 4/5 2024-04-12T05:25:43.307 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-827.gd3b5060a.el8.noarch 5/5 2024-04-12T05:25:43.592 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-827.gd3b5060a.el8.noarch 5/5 2024-04-12T05:25:43.592 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-827.gd3b5060a.el8.noarch 1/5 2024-04-12T05:25:43.592 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2024-04-12T05:25:43.592 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/5 2024-04-12T05:25:43.592 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-04-12T05:25:44.011 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2024-04-12T05:25:44.011 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:25:44.011 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-12T05:25:44.011 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-cephadm-2:17.2.7-827.gd3b5060a.el8.noarch 2024-04-12T05:25:44.011 INFO:teuthology.orchestra.run.smithi032.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-12T05:25:44.011 INFO:teuthology.orchestra.run.smithi032.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-12T05:25:44.011 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-04-12T05:25:44.011 INFO:teuthology.orchestra.run.smithi032.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-12T05:25:44.011 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:25:44.011 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-12T05:25:44.155 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-rados-2:17.2.7-827.gd3b5060a.el8.x86_64 is already installed. 2024-04-12T05:25:44.207 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-12T05:25:44.208 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-04-12T05:25:44.208 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-12T05:25:44.310 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rgw 2024-04-12T05:25:44.515 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-fuse 2024-04-12T05:25:45.073 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:03:09 ago on Fri 12 Apr 2024 05:22:36 AM UTC. 2024-04-12T05:25:45.281 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:02:43 ago on Fri 12 Apr 2024 05:23:02 AM UTC. 2024-04-12T05:25:46.767 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-rgw-2:17.2.7-827.gd3b5060a.el8.x86_64 is already installed. 2024-04-12T05:25:46.819 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-12T05:25:46.821 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-04-12T05:25:46.821 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-12T05:25:46.918 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-cephfs 2024-04-12T05:25:47.047 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-12T05:25:47.047 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:25:47.047 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-12T05:25:47.047 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:25:47.047 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-12T05:25:47.047 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 876 k 2024-04-12T05:25:47.048 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-12T05:25:47.048 INFO:teuthology.orchestra.run.smithi032.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-04-12T05:25:47.048 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:25:47.048 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-12T05:25:47.048 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:25:47.048 INFO:teuthology.orchestra.run.smithi032.stdout:Install 2 Packages 2024-04-12T05:25:47.048 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:25:47.048 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 959 k 2024-04-12T05:25:47.048 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 3.1 M 2024-04-12T05:25:47.049 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-12T05:25:47.313 INFO:teuthology.orchestra.run.smithi032.stdout:(1/2): fuse-2.9.7-19.el8.x86_64.rpm 1.1 MB/s | 83 kB 00:00 2024-04-12T05:25:47.504 INFO:teuthology.orchestra.run.smithi032.stdout:(2/2): ceph-fuse-17.2.7-827.gd3b5060a.el8.x86_6 3.2 MB/s | 876 kB 00:00 2024-04-12T05:25:47.505 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:25:47.505 INFO:teuthology.orchestra.run.smithi032.stdout:Total 2.0 MB/s | 959 kB 00:00 2024-04-12T05:25:47.516 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-12T05:25:47.538 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-12T05:25:47.538 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-12T05:25:47.634 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-12T05:25:47.637 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-12T05:25:47.682 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:03:11 ago on Fri 12 Apr 2024 05:22:36 AM UTC. 2024-04-12T05:25:47.816 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-12T05:25:48.088 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-04-12T05:25:48.307 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-fuse-2:17.2.7-827.gd3b5060a.el8.x86_64 2/2 2024-04-12T05:25:48.728 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-fuse-2:17.2.7-827.gd3b5060a.el8.x86_64 2/2 2024-04-12T05:25:48.728 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-fuse-2:17.2.7-827.gd3b5060a.el8.x86_64 1/2 2024-04-12T05:25:49.144 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-04-12T05:25:49.144 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:25:49.144 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-12T05:25:49.144 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse-2:17.2.7-827.gd3b5060a.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-04-12T05:25:49.144 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:25:49.144 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-12T05:25:49.328 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install librados-devel 2024-04-12T05:25:49.369 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-cephfs-2:17.2.7-827.gd3b5060a.el8.x86_64 is already installed. 2024-04-12T05:25:49.421 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-12T05:25:49.422 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-04-12T05:25:49.422 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-12T05:25:49.512 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rbd 2024-04-12T05:25:50.101 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:02:48 ago on Fri 12 Apr 2024 05:23:02 AM UTC. 2024-04-12T05:25:50.271 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:03:14 ago on Fri 12 Apr 2024 05:22:36 AM UTC. 2024-04-12T05:25:51.838 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-12T05:25:51.838 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:25:51.839 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-04-12T05:25:51.839 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:25:51.839 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-12T05:25:51.839 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 130 k 2024-04-12T05:25:51.839 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:25:51.839 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-12T05:25:51.839 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:25:51.839 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-04-12T05:25:51.839 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:25:51.839 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 130 k 2024-04-12T05:25:51.839 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 464 k 2024-04-12T05:25:51.839 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-12T05:25:51.953 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-rbd-2:17.2.7-827.gd3b5060a.el8.x86_64 is already installed. 2024-04-12T05:25:52.005 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-12T05:25:52.006 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-04-12T05:25:52.006 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-12T05:25:52.037 INFO:teuthology.orchestra.run.smithi032.stdout:librados-devel-17.2.7-827.gd3b5060a.el8.x86_64. 658 kB/s | 130 kB 00:00 2024-04-12T05:25:52.037 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:25:52.038 INFO:teuthology.orchestra.run.smithi032.stdout:Total 653 kB/s | 130 kB 00:00 2024-04-12T05:25:52.038 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-12T05:25:52.044 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-12T05:25:52.044 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-12T05:25:52.097 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-12T05:25:52.100 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-12T05:25:52.211 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-fuse 2024-04-12T05:25:52.214 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-12T05:25:52.448 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librados-devel-2:17.2.7-827.gd3b5060a.el8.x86_64 1/1 2024-04-12T05:25:52.753 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados-devel-2:17.2.7-827.gd3b5060a.el8.x86_64 1/1 2024-04-12T05:25:52.978 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:03:16 ago on Fri 12 Apr 2024 05:22:36 AM UTC. 2024-04-12T05:25:53.172 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados-devel-2:17.2.7-827.gd3b5060a.el8.x86_64 1/1 2024-04-12T05:25:53.172 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:25:53.172 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-12T05:25:53.172 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:25:53.172 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:25:53.172 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-12T05:25:53.395 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install libcephfs2 2024-04-12T05:25:54.162 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:02:52 ago on Fri 12 Apr 2024 05:23:02 AM UTC. 2024-04-12T05:25:54.733 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-12T05:25:54.733 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:25:54.733 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-04-12T05:25:54.733 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:25:54.734 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-12T05:25:54.734 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 91 k 2024-04-12T05:25:54.734 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:25:54.734 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-12T05:25:54.734 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:25:54.734 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-04-12T05:25:54.734 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:25:54.735 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 91 k 2024-04-12T05:25:54.735 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 268 k 2024-04-12T05:25:54.735 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-12T05:25:55.169 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-fuse-17.2.7-827.gd3b5060a.el8.x86_64.rpm 209 kB/s | 91 kB 00:00 2024-04-12T05:25:55.170 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:25:55.170 INFO:teuthology.orchestra.run.smithi067.stdout:Total 208 kB/s | 91 kB 00:00 2024-04-12T05:25:55.170 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-12T05:25:55.181 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-12T05:25:55.181 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-12T05:25:55.235 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-12T05:25:55.238 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-12T05:25:55.358 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-12T05:25:55.596 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-fuse-2:17.2.7-827.gd3b5060a.el8.x86_64 1/1 2024-04-12T05:25:55.853 INFO:teuthology.orchestra.run.smithi032.stdout:Package libcephfs2-2:17.2.7-827.gd3b5060a.el8.x86_64 is already installed. 2024-04-12T05:25:55.892 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-fuse-2:17.2.7-827.gd3b5060a.el8.x86_64 1/1 2024-04-12T05:25:55.905 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-12T05:25:55.906 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-12T05:25:55.907 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-12T05:25:56.021 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install libcephfs-devel 2024-04-12T05:25:56.267 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-fuse-2:17.2.7-827.gd3b5060a.el8.x86_64 1/1 2024-04-12T05:25:56.267 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:25:56.267 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-12T05:25:56.267 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:25:56.267 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:25:56.267 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-12T05:25:56.614 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-mirror 2024-04-12T05:25:56.792 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:02:54 ago on Fri 12 Apr 2024 05:23:02 AM UTC. 2024-04-12T05:25:57.384 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:03:21 ago on Fri 12 Apr 2024 05:22:36 AM UTC. 2024-04-12T05:25:58.541 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-12T05:25:58.542 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:25:58.543 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-04-12T05:25:58.543 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:25:58.543 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-12T05:25:58.543 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-devel x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 25 k 2024-04-12T05:25:58.543 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:25:58.543 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-12T05:25:58.543 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:25:58.543 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-04-12T05:25:58.543 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:25:58.543 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 25 k 2024-04-12T05:25:58.543 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 109 k 2024-04-12T05:25:58.543 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-12T05:25:58.773 INFO:teuthology.orchestra.run.smithi032.stdout:libcephfs-devel-17.2.7-827.gd3b5060a.el8.x86_64 108 kB/s | 25 kB 00:00 2024-04-12T05:25:58.774 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:25:58.774 INFO:teuthology.orchestra.run.smithi032.stdout:Total 107 kB/s | 25 kB 00:00 2024-04-12T05:25:58.774 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-12T05:25:58.778 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-12T05:25:58.778 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-12T05:25:58.780 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-12T05:25:58.784 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-12T05:25:58.835 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-12T05:25:59.030 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephfs-devel-2:17.2.7-827.gd3b5060a.el8.x86_64 1/1 2024-04-12T05:25:59.125 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-12T05:25:59.126 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:25:59.126 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-04-12T05:25:59.126 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:25:59.127 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-12T05:25:59.127 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 3.5 M 2024-04-12T05:25:59.127 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:25:59.127 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-12T05:25:59.127 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:25:59.127 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-04-12T05:25:59.127 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:25:59.127 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 3.5 M 2024-04-12T05:25:59.127 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 16 M 2024-04-12T05:25:59.127 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-12T05:25:59.323 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-827.gd3b5060a.el8.x86_64 1/1 2024-04-12T05:25:59.638 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-mirror-17.2.7-827.gd3b5060a.el8.x86_64.rpm 6.8 MB/s | 3.5 MB 00:00 2024-04-12T05:25:59.639 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:25:59.639 INFO:teuthology.orchestra.run.smithi067.stdout:Total 6.7 MB/s | 3.5 MB 00:00 2024-04-12T05:25:59.640 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-12T05:25:59.652 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-12T05:25:59.652 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-12T05:25:59.691 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephfs-devel-2:17.2.7-827.gd3b5060a.el8.x86_64 1/1 2024-04-12T05:25:59.691 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:25:59.691 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-12T05:25:59.691 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-devel-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:25:59.691 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:25:59.691 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-12T05:25:59.717 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-12T05:25:59.720 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-12T05:25:59.808 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install librados2 2024-04-12T05:26:00.180 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-12T05:26:00.363 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-mirror-2:17.2.7-827.gd3b5060a.el8.x86_64 1/1 2024-04-12T05:26:00.576 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:02:58 ago on Fri 12 Apr 2024 05:23:02 AM UTC. 2024-04-12T05:26:00.793 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-mirror-2:17.2.7-827.gd3b5060a.el8.x86_64 1/1 2024-04-12T05:26:01.187 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-mirror-2:17.2.7-827.gd3b5060a.el8.x86_64 1/1 2024-04-12T05:26:01.187 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:26:01.188 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-12T05:26:01.188 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:26:01.188 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:26:01.188 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-12T05:26:01.384 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-nbd 2024-04-12T05:26:02.158 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:03:26 ago on Fri 12 Apr 2024 05:22:36 AM UTC. 2024-04-12T05:26:02.271 INFO:teuthology.orchestra.run.smithi032.stdout:Package librados2-2:17.2.7-827.gd3b5060a.el8.x86_64 is already installed. 2024-04-12T05:26:02.323 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-12T05:26:02.324 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-12T05:26:02.324 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-12T05:26:02.413 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install librbd1 2024-04-12T05:26:03.192 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:03:01 ago on Fri 12 Apr 2024 05:23:02 AM UTC. 2024-04-12T05:26:03.908 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-12T05:26:03.908 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:26:03.909 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-04-12T05:26:03.909 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:26:03.909 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-12T05:26:03.909 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 175 k 2024-04-12T05:26:03.909 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:26:03.909 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-12T05:26:03.909 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-12T05:26:03.909 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-04-12T05:26:03.909 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:26:03.910 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 175 k 2024-04-12T05:26:03.910 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 556 k 2024-04-12T05:26:03.910 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-12T05:26:04.532 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-nbd-17.2.7-827.gd3b5060a.el8.x86_64.rpm 281 kB/s | 175 kB 00:00 2024-04-12T05:26:04.532 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:26:04.533 INFO:teuthology.orchestra.run.smithi067.stdout:Total 280 kB/s | 175 kB 00:00 2024-04-12T05:26:04.533 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-12T05:26:04.546 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-12T05:26:04.546 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-12T05:26:04.602 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-12T05:26:04.605 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-12T05:26:04.738 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-12T05:26:04.891 INFO:teuthology.orchestra.run.smithi032.stdout:Package librbd1-2:17.2.7-827.gd3b5060a.el8.x86_64 is already installed. 2024-04-12T05:26:04.943 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-12T05:26:04.944 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-12T05:26:04.944 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-12T05:26:05.033 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-rados 2024-04-12T05:26:05.069 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-nbd-2:17.2.7-827.gd3b5060a.el8.x86_64 1/1 2024-04-12T05:26:05.381 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-nbd-2:17.2.7-827.gd3b5060a.el8.x86_64 1/1 2024-04-12T05:26:05.763 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-nbd-2:17.2.7-827.gd3b5060a.el8.x86_64 1/1 2024-04-12T05:26:05.763 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:26:05.763 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-12T05:26:05.764 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:26:05.764 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:26:05.764 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-12T05:26:05.792 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:03:03 ago on Fri 12 Apr 2024 05:23:02 AM UTC. 2024-04-12T05:26:05.954 DEBUG:teuthology.parallel:result is None 2024-04-12T05:26:07.487 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-rados-2:17.2.7-827.gd3b5060a.el8.x86_64 is already installed. 2024-04-12T05:26:07.539 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-12T05:26:07.540 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-12T05:26:07.540 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-12T05:26:07.628 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-rgw 2024-04-12T05:26:08.384 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:03:06 ago on Fri 12 Apr 2024 05:23:02 AM UTC. 2024-04-12T05:26:10.088 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-rgw-2:17.2.7-827.gd3b5060a.el8.x86_64 is already installed. 2024-04-12T05:26:10.139 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-12T05:26:10.140 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-12T05:26:10.141 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-12T05:26:10.237 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-cephfs 2024-04-12T05:26:10.996 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:03:08 ago on Fri 12 Apr 2024 05:23:02 AM UTC. 2024-04-12T05:26:12.681 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-cephfs-2:17.2.7-827.gd3b5060a.el8.x86_64 is already installed. 2024-04-12T05:26:12.734 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-12T05:26:12.735 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-12T05:26:12.735 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-12T05:26:12.825 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-rbd 2024-04-12T05:26:13.582 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:03:11 ago on Fri 12 Apr 2024 05:23:02 AM UTC. 2024-04-12T05:26:15.266 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-rbd-2:17.2.7-827.gd3b5060a.el8.x86_64 is already installed. 2024-04-12T05:26:15.318 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-12T05:26:15.319 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-12T05:26:15.319 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-12T05:26:15.547 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install rbd-fuse 2024-04-12T05:26:16.309 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:03:14 ago on Fri 12 Apr 2024 05:23:02 AM UTC. 2024-04-12T05:26:18.051 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-12T05:26:18.051 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:26:18.051 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-04-12T05:26:18.051 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:26:18.051 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-12T05:26:18.051 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 91 k 2024-04-12T05:26:18.052 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:26:18.052 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-12T05:26:18.052 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:26:18.052 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-04-12T05:26:18.052 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:26:18.052 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 91 k 2024-04-12T05:26:18.052 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 268 k 2024-04-12T05:26:18.053 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-12T05:26:18.321 INFO:teuthology.orchestra.run.smithi032.stdout:rbd-fuse-17.2.7-827.gd3b5060a.el8.x86_64.rpm 338 kB/s | 91 kB 00:00 2024-04-12T05:26:18.321 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:26:18.321 INFO:teuthology.orchestra.run.smithi032.stdout:Total 336 kB/s | 91 kB 00:00 2024-04-12T05:26:18.322 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-12T05:26:18.332 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-12T05:26:18.332 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-12T05:26:18.385 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-12T05:26:18.388 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-12T05:26:18.508 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-12T05:26:18.727 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-fuse-2:17.2.7-827.gd3b5060a.el8.x86_64 1/1 2024-04-12T05:26:19.029 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-fuse-2:17.2.7-827.gd3b5060a.el8.x86_64 1/1 2024-04-12T05:26:19.452 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-fuse-2:17.2.7-827.gd3b5060a.el8.x86_64 1/1 2024-04-12T05:26:19.452 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:26:19.452 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-12T05:26:19.452 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:26:19.452 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:26:19.453 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-12T05:26:19.655 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install rbd-mirror 2024-04-12T05:26:20.430 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:03:18 ago on Fri 12 Apr 2024 05:23:02 AM UTC. 2024-04-12T05:26:22.172 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-12T05:26:22.173 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:26:22.173 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-04-12T05:26:22.173 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:26:22.174 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-12T05:26:22.174 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 3.5 M 2024-04-12T05:26:22.174 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:26:22.174 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-12T05:26:22.174 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:26:22.174 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-04-12T05:26:22.174 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:26:22.174 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 3.5 M 2024-04-12T05:26:22.174 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 16 M 2024-04-12T05:26:22.174 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-12T05:26:22.707 INFO:teuthology.orchestra.run.smithi032.stdout:rbd-mirror-17.2.7-827.gd3b5060a.el8.x86_64.rpm 6.5 MB/s | 3.5 MB 00:00 2024-04-12T05:26:22.707 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:26:22.708 INFO:teuthology.orchestra.run.smithi032.stdout:Total 6.5 MB/s | 3.5 MB 00:00 2024-04-12T05:26:22.708 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-12T05:26:22.721 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-12T05:26:22.721 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-12T05:26:22.790 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-12T05:26:22.793 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-12T05:26:23.267 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-12T05:26:23.469 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-mirror-2:17.2.7-827.gd3b5060a.el8.x86_64 1/1 2024-04-12T05:26:23.891 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-mirror-2:17.2.7-827.gd3b5060a.el8.x86_64 1/1 2024-04-12T05:26:24.293 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-mirror-2:17.2.7-827.gd3b5060a.el8.x86_64 1/1 2024-04-12T05:26:24.293 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:26:24.293 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-12T05:26:24.293 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:26:24.293 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:26:24.293 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-12T05:26:24.515 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install rbd-nbd 2024-04-12T05:26:25.290 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:03:23 ago on Fri 12 Apr 2024 05:23:02 AM UTC. 2024-04-12T05:26:27.063 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-12T05:26:27.064 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:26:27.064 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-04-12T05:26:27.064 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:26:27.065 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-12T05:26:27.065 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd x86_64 2:17.2.7-827.gd3b5060a.el8 ceph 175 k 2024-04-12T05:26:27.065 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:26:27.065 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-12T05:26:27.065 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-04-12T05:26:27.065 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-04-12T05:26:27.065 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:26:27.065 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 175 k 2024-04-12T05:26:27.065 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 556 k 2024-04-12T05:26:27.065 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-12T05:26:27.709 INFO:teuthology.orchestra.run.smithi032.stdout:rbd-nbd-17.2.7-827.gd3b5060a.el8.x86_64.rpm 271 kB/s | 175 kB 00:00 2024-04-12T05:26:27.710 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:26:27.710 INFO:teuthology.orchestra.run.smithi032.stdout:Total 271 kB/s | 175 kB 00:00 2024-04-12T05:26:27.710 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-12T05:26:27.724 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-12T05:26:27.724 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-12T05:26:27.783 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-12T05:26:27.786 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-12T05:26:27.909 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-12T05:26:28.134 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-nbd-2:17.2.7-827.gd3b5060a.el8.x86_64 1/1 2024-04-12T05:26:28.426 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-nbd-2:17.2.7-827.gd3b5060a.el8.x86_64 1/1 2024-04-12T05:26:28.833 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-nbd-2:17.2.7-827.gd3b5060a.el8.x86_64 1/1 2024-04-12T05:26:28.833 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:26:28.833 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-12T05:26:28.833 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd-2:17.2.7-827.gd3b5060a.el8.x86_64 2024-04-12T05:26:28.833 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:26:28.833 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-12T05:26:29.044 DEBUG:teuthology.parallel:result is None 2024-04-12T05:26:29.045 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-12T05:26:29.045 INFO:teuthology.packaging:ref: None 2024-04-12T05:26:29.045 INFO:teuthology.packaging:tag: None 2024-04-12T05:26:29.045 INFO:teuthology.packaging:branch: quincy 2024-04-12T05:26:29.045 INFO:teuthology.packaging:sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:26:29.045 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-04-12T05:26:29.270 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-12T05:26:29.294 INFO:teuthology.orchestra.run.smithi032.stdout:17.2.7-827.gd3b5060a.el8 2024-04-12T05:26:29.295 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-827.gd3b5060a.el8 2024-04-12T05:26:29.295 INFO:teuthology.task.install:The correct ceph version 17.2.7-827.gd3b5060a is installed. 2024-04-12T05:26:29.297 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-12T05:26:29.297 INFO:teuthology.packaging:ref: None 2024-04-12T05:26:29.297 INFO:teuthology.packaging:tag: None 2024-04-12T05:26:29.297 INFO:teuthology.packaging:branch: quincy 2024-04-12T05:26:29.297 INFO:teuthology.packaging:sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:26:29.297 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-04-12T05:26:29.553 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-12T05:26:29.581 INFO:teuthology.orchestra.run.smithi067.stdout:17.2.7-827.gd3b5060a.el8 2024-04-12T05:26:29.582 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-827.gd3b5060a.el8 2024-04-12T05:26:29.582 INFO:teuthology.task.install:The correct ceph version 17.2.7-827.gd3b5060a is installed. 2024-04-12T05:26:29.584 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-12T05:26:29.584 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-12T05:26:29.584 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-12T05:26:29.613 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-12T05:26:29.613 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-12T05:26:29.657 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-12T05:26:29.658 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-12T05:26:29.658 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/daemon-helper 2024-04-12T05:26:29.685 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-12T05:26:29.753 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-12T05:26:29.753 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/daemon-helper 2024-04-12T05:26:29.786 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-12T05:26:29.858 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-12T05:26:29.859 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-12T05:26:29.859 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-12T05:26:29.886 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-12T05:26:29.954 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-12T05:26:29.954 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-12T05:26:29.986 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-12T05:26:30.058 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-12T05:26:30.062 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-12T05:26:30.062 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/stdin-killer 2024-04-12T05:26:30.090 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-12T05:26:30.158 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-12T05:26:30.158 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/stdin-killer 2024-04-12T05:26:30.189 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-12T05:26:30.263 INFO:teuthology.run_tasks:Running task print... 2024-04-12T05:26:30.272 INFO:teuthology.task.print:**** done install task... 2024-04-12T05:26:30.272 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-12T05:26:30.377 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'} 2024-04-12T05:26:30.378 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-04-12T05:26:30.378 INFO:tasks.cephadm:Cluster fsid is 37b58ce2-f88d-11ee-bc8c-c7b262605968 2024-04-12T05:26:30.378 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-12T05:26:30.378 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-04-12T05:26:30.379 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi032': '172.21.15.32', 'mon.smithi067': '172.21.15.67'} 2024-04-12T05:26:30.379 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-12T05:26:30.379 DEBUG:teuthology.orchestra.run.smithi032:> sudo hostname $(hostname -s) 2024-04-12T05:26:30.407 DEBUG:teuthology.orchestra.run.smithi067:> sudo hostname $(hostname -s) 2024-04-12T05:26:30.439 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-04-12T05:26:30.439 DEBUG:teuthology.orchestra.run.smithi032:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-12T05:26:30.651 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-rw-r--. 1 ubuntu ubuntu 375153 Apr 12 05:26 /home/ubuntu/cephtest/cephadm 2024-04-12T05:26:30.651 DEBUG:teuthology.orchestra.run.smithi067:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-12T05:26:30.791 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-rw-r--. 1 ubuntu ubuntu 375153 Apr 12 05:26 /home/ubuntu/cephtest/cephadm 2024-04-12T05:26:30.792 DEBUG:teuthology.orchestra.run.smithi032:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-12T05:26:30.812 DEBUG:teuthology.orchestra.run.smithi067:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-12T05:26:30.846 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-04-12T05:26:30.846 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-04-12T05:26:30.855 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-04-12T05:26:31.963 INFO:teuthology.orchestra.run.smithi067.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-04-12T05:26:32.083 INFO:teuthology.orchestra.run.smithi032.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-04-12T05:26:57.669 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-12T05:26:57.669 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_version": "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)", 2024-04-12T05:26:57.669 INFO:teuthology.orchestra.run.smithi067.stdout: "image_id": "21de9b7369ad8457bc44c6aca16b9dfec25bf172fb7b6ebd80f97d4df2005241", 2024-04-12T05:26:57.669 INFO:teuthology.orchestra.run.smithi067.stdout: "repo_digests": [ 2024-04-12T05:26:57.669 INFO:teuthology.orchestra.run.smithi067.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:5e334df698f432180b78bc9c46936bfdfbc92bc9a8983463e523f780d23765fe" 2024-04-12T05:26:57.669 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-12T05:26:57.670 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-12T05:26:57.768 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:26:57.769 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)", 2024-04-12T05:26:57.769 INFO:teuthology.orchestra.run.smithi032.stdout: "image_id": "21de9b7369ad8457bc44c6aca16b9dfec25bf172fb7b6ebd80f97d4df2005241", 2024-04-12T05:26:57.769 INFO:teuthology.orchestra.run.smithi032.stdout: "repo_digests": [ 2024-04-12T05:26:57.769 INFO:teuthology.orchestra.run.smithi032.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:5e334df698f432180b78bc9c46936bfdfbc92bc9a8983463e523f780d23765fe" 2024-04-12T05:26:57.769 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-12T05:26:57.769 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:26:57.792 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /etc/ceph 2024-04-12T05:26:57.823 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph 2024-04-12T05:26:57.854 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 777 /etc/ceph 2024-04-12T05:26:57.895 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 777 /etc/ceph 2024-04-12T05:26:57.922 INFO:tasks.cephadm:Writing seed config... 2024-04-12T05:26:57.924 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-04-12T05:26:57.924 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-04-12T05:26:57.924 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-12T05:26:57.924 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-12T05:26:57.924 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-12T05:26:57.924 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-12T05:26:57.924 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-12T05:26:57.924 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-12T05:26:57.924 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-12T05:26:57.924 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-12T05:26:57.924 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-12T05:26:57.924 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-12T05:26:57.925 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-12T05:26:57.925 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-12T05:26:57.925 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-12T05:26:57.925 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-12T05:26:57.925 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-12T05:26:57.925 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-12T05:26:57.925 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-12T05:26:57.925 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-12T05:26:57.925 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-12T05:26:57.925 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-12T05:26:57.925 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-12T05:26:57.925 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-04-12T05:26:57.925 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-12T05:26:57.925 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-12T05:26:57.925 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-12T05:26:57.926 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-12T05:26:57.926 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-12T05:26:57.926 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-12T05:26:57.926 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-12T05:26:57.926 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-12T05:26:57.926 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-12T05:26:57.926 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-12T05:26:57.926 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-12T05:26:57.926 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-12T05:26:57.926 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-12T05:26:57.926 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-12T05:26:57.926 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-12T05:26:57.926 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-12T05:26:57.927 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-12T05:26:57.927 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-12T05:26:57.954 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 37b58ce2-f88d-11ee-bc8c-c7b262605968 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-12T05:26:57.954 DEBUG:teuthology.orchestra.run.smithi032:mon.smithi032> sudo journalctl -f -n 0 -u ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@mon.smithi032.service 2024-04-12T05:26:57.997 INFO:tasks.cephadm:Bootstrapping... 2024-04-12T05:26:57.997 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.32 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-12T05:26:58.025 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:-- Logs begin at Fri 2024-04-12 05:14:13 UTC. -- 2024-04-12T05:26:58.209 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-12T05:26:58.209 INFO:teuthology.orchestra.run.smithi032.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', '37b58ce2-f88d-11ee-bc8c-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.32', '--skip-admin-label'] 2024-04-12T05:26:58.242 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-12T05:26:58.243 INFO:teuthology.orchestra.run.smithi032.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-12T05:26:58.243 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying podman|docker is present... 2024-04-12T05:26:58.277 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-12T05:26:58.277 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying lvm2 is present... 2024-04-12T05:26:58.277 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying time synchronization is in place... 2024-04-12T05:26:58.285 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-12T05:26:58.285 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-12T05:26:58.291 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-12T05:26:58.291 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-12T05:26:58.297 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout enabled 2024-04-12T05:26:58.303 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout active 2024-04-12T05:26:58.303 INFO:teuthology.orchestra.run.smithi032.stdout:Unit chronyd.service is enabled and running 2024-04-12T05:26:58.303 INFO:teuthology.orchestra.run.smithi032.stdout:Repeating the final host check... 2024-04-12T05:26:58.334 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-12T05:26:58.335 INFO:teuthology.orchestra.run.smithi032.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-12T05:26:58.335 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl is present 2024-04-12T05:26:58.335 INFO:teuthology.orchestra.run.smithi032.stdout:lvcreate is present 2024-04-12T05:26:58.341 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-12T05:26:58.341 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-12T05:26:58.347 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-12T05:26:58.347 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-12T05:26:58.353 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout enabled 2024-04-12T05:26:58.359 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout active 2024-04-12T05:26:58.359 INFO:teuthology.orchestra.run.smithi032.stdout:Unit chronyd.service is enabled and running 2024-04-12T05:26:58.359 INFO:teuthology.orchestra.run.smithi032.stdout:Host looks OK 2024-04-12T05:26:58.359 INFO:teuthology.orchestra.run.smithi032.stdout:Cluster fsid: 37b58ce2-f88d-11ee-bc8c-c7b262605968 2024-04-12T05:26:58.359 INFO:teuthology.orchestra.run.smithi032.stdout:Acquiring lock 140057905849232 on /run/cephadm/37b58ce2-f88d-11ee-bc8c-c7b262605968.lock 2024-04-12T05:26:58.359 INFO:teuthology.orchestra.run.smithi032.stdout:Lock 140057905849232 acquired on /run/cephadm/37b58ce2-f88d-11ee-bc8c-c7b262605968.lock 2024-04-12T05:26:58.360 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying IP 172.21.15.32 port 3300 ... 2024-04-12T05:26:58.360 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying IP 172.21.15.32 port 6789 ... 2024-04-12T05:26:58.360 INFO:teuthology.orchestra.run.smithi032.stdout:Base mon IP(s) is [172.21.15.32:3300, 172.21.15.32:6789], mon addrv is [v2:172.21.15.32:3300,v1:172.21.15.32:6789] 2024-04-12T05:26:58.364 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.32 metric 100 2024-04-12T05:26:58.364 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.32 metric 100 2024-04-12T05:26:58.367 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-12T05:26:58.367 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-04-12T05:26:58.367 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1784sec hoplimit 64 pref medium 2024-04-12T05:26:58.369 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-12T05:26:58.370 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-12T05:26:58.370 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-12T05:26:58.370 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-12T05:26:58.370 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fef8:2602/64 scope link 2024-04-12T05:26:58.370 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-12T05:26:58.370 INFO:teuthology.orchestra.run.smithi032.stdout:Mon IP `172.21.15.32` is in CIDR network `172.21.0.0/20` 2024-04-12T05:26:58.371 INFO:teuthology.orchestra.run.smithi032.stdout:Mon IP `172.21.15.32` is in CIDR network `172.21.0.0/20` 2024-04-12T05:26:58.371 INFO:teuthology.orchestra.run.smithi032.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-12T05:26:58.371 INFO:teuthology.orchestra.run.smithi032.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-12T05:26:58.372 INFO:teuthology.orchestra.run.smithi032.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-04-12T05:26:58.882 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stdout 21de9b7369ad8457bc44c6aca16b9dfec25bf172fb7b6ebd80f97d4df2005241 2024-04-12T05:26:58.882 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-04-12T05:26:58.882 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Getting image source signatures 2024-04-12T05:26:58.882 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Copying blob sha256:6a38c9a31ea4c6c1e3d079c8edc2f3b418b00f3ce1d3a48ffe92dcb2ef7b4e51 2024-04-12T05:26:58.882 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Copying blob sha256:9793b5b5582798bde394f4383d142e9f248a25e4ebfd72ee05ea9ec6251a877d 2024-04-12T05:26:58.882 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Copying config sha256:21de9b7369ad8457bc44c6aca16b9dfec25bf172fb7b6ebd80f97d4df2005241 2024-04-12T05:26:58.882 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-12T05:26:59.937 INFO:teuthology.orchestra.run.smithi032.stdout:ceph: stdout ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable) 2024-04-12T05:26:59.937 INFO:teuthology.orchestra.run.smithi032.stdout:Ceph version: ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable) 2024-04-12T05:26:59.937 INFO:teuthology.orchestra.run.smithi032.stdout:Extracting ceph user uid/gid from container image... 2024-04-12T05:27:00.998 INFO:teuthology.orchestra.run.smithi032.stdout:stat: stdout 167 167 2024-04-12T05:27:00.998 INFO:teuthology.orchestra.run.smithi032.stdout:Creating initial keys... 2024-04-12T05:27:02.155 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-authtool: stdout AQAlxhhmpbWkFxAA3EJXdXkX0g9csAN7Y0FCjg== 2024-04-12T05:27:03.262 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-authtool: stdout AQAmxhhmzm/XIxAAGM/IfrFrl7KyddhMdtEk/Q== 2024-04-12T05:27:04.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-authtool: stdout AQAnxhhmIkwbLBAAPXxn/f5K1jKaSut28TuMfQ== 2024-04-12T05:27:04.377 INFO:teuthology.orchestra.run.smithi032.stdout:Creating initial monmap... 2024-04-12T05:27:05.522 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-12T05:27:05.523 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-04-12T05:27:05.523 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 37b58ce2-f88d-11ee-bc8c-c7b262605968 2024-04-12T05:27:05.523 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-12T05:27:05.523 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool for smithi032 [v2:172.21.15.32:3300,v1:172.21.15.32:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-12T05:27:05.523 INFO:teuthology.orchestra.run.smithi032.stdout:setting min_mon_release = octopus 2024-04-12T05:27:05.523 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: set fsid to 37b58ce2-f88d-11ee-bc8c-c7b262605968 2024-04-12T05:27:05.523 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-12T05:27:05.523 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:27:05.523 INFO:teuthology.orchestra.run.smithi032.stdout:Creating mon... 2024-04-12T05:27:06.668 INFO:teuthology.orchestra.run.smithi032.stdout:create mon.smithi032 on 2024-04-12T05:27:06.777 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-04-12T05:27:06.873 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-12T05:27:06.973 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968.target → /etc/systemd/system/ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968.target. 2024-04-12T05:27:06.973 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968.target → /etc/systemd/system/ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968.target. 2024-04-12T05:27:07.115 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@mon.smithi032 2024-04-12T05:27:07.115 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to reset failed state of unit ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@mon.smithi032.service: Unit ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@mon.smithi032.service not loaded. 2024-04-12T05:27:07.213 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968.target.wants/ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@mon.smithi032.service → /etc/systemd/system/ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@.service. 2024-04-12T05:27:07.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:07 smithi032 systemd[1]: Starting Ceph mon.smithi032 for 37b58ce2-f88d-11ee-bc8c-c7b262605968... 2024-04-12T05:27:07.802 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-12T05:27:07.802 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout disabled 2024-04-12T05:27:07.810 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-12T05:27:07.811 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-12T05:27:07.811 INFO:teuthology.orchestra.run.smithi032.stdout:firewalld.service is not enabled 2024-04-12T05:27:07.811 INFO:teuthology.orchestra.run.smithi032.stdout:Not possible to enable service . firewalld.service is not available 2024-04-12T05:27:07.811 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mon to start... 2024-04-12T05:27:07.811 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mon... 2024-04-12T05:27:08.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:07 smithi032 ceph-mon[39248]: mkfs 37b58ce2-f88d-11ee-bc8c-c7b262605968 2024-04-12T05:27:08.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:07 smithi032 ceph-mon[39248]: mon.smithi032 is new leader, mons smithi032 in quorum (ranks 0) 2024-04-12T05:27:08.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:07 smithi032 bash[39211]: f1cbb6dfbc39ffe3b331aa78889c0e8db5944b351e9889b0175c1f7f5ebbb0a5 2024-04-12T05:27:08.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:07 smithi032 systemd[1]: Started Ceph mon.smithi032 for 37b58ce2-f88d-11ee-bc8c-c7b262605968. 2024-04-12T05:27:09.094 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:08 smithi032 ceph-mon[39248]: mon.smithi032 is new leader, mons smithi032 in quorum (ranks 0) 2024-04-12T05:27:09.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:08 smithi032 ceph-mon[39248]: monmap e1: 1 mons at {smithi032=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-12T05:27:09.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:08 smithi032 ceph-mon[39248]: fsmap 2024-04-12T05:27:09.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:08 smithi032 ceph-mon[39248]: osdmap e1: 0 total, 0 up, 0 in 2024-04-12T05:27:09.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:08 smithi032 ceph-mon[39248]: mgrmap e1: no daemons active 2024-04-12T05:27:10.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:09 smithi032 ceph-mon[39248]: from='client.? 172.21.15.32:0/3430629501' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-12T05:27:11.511 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout cluster: 2024-04-12T05:27:11.517 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout id: 37b58ce2-f88d-11ee-bc8c-c7b262605968 2024-04-12T05:27:11.517 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-12T05:27:11.517 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-12T05:27:11.517 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout services: 2024-04-12T05:27:11.517 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi032 (age 1.63011s) 2024-04-12T05:27:11.518 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-12T05:27:11.518 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-12T05:27:11.518 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-12T05:27:11.518 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout data: 2024-04-12T05:27:11.518 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-12T05:27:11.518 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-12T05:27:11.518 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-12T05:27:11.518 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout pgs: 2024-04-12T05:27:11.518 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-12T05:27:11.518 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.325+0000 7f55c4a18700 1 Processor -- start 2024-04-12T05:27:11.518 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.325+0000 7f55c4a18700 1 -- start start 2024-04-12T05:27:11.518 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.325+0000 7f55c4a18700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55c0101a90 0x7f55c0101ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:11.518 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.325+0000 7f55c4a18700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c01023e0 con 0x7f55c0101a90 2024-04-12T05:27:11.519 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.326+0000 7f55be59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55c0101a90 0x7f55c0101ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:11.519 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.326+0000 7f55be59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55c0101a90 0x7f55c0101ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36740/0 (socket says 172.21.15.32:36740) 2024-04-12T05:27:11.519 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.326+0000 7f55be59c700 1 -- 172.21.15.32:0/3146475898 learned_addr learned my addr 172.21.15.32:0/3146475898 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:11.519 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.326+0000 7f55be59c700 1 -- 172.21.15.32:0/3146475898 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c0102520 con 0x7f55c0101a90 2024-04-12T05:27:11.519 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.326+0000 7f55be59c700 1 --2- 172.21.15.32:0/3146475898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55c0101a90 0x7f55c0101ea0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f55b00098e0 tx=0x7f55b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66dd568bff465a3a server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:11.519 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.326+0000 7f55bd59a700 1 -- 172.21.15.32:0/3146475898 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55b000b860 con 0x7f55c0101a90 2024-04-12T05:27:11.519 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.326+0000 7f55bd59a700 1 -- 172.21.15.32:0/3146475898 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55b000b9c0 con 0x7f55c0101a90 2024-04-12T05:27:11.519 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.326+0000 7f55bd59a700 1 -- 172.21.15.32:0/3146475898 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55b000bc90 con 0x7f55c0101a90 2024-04-12T05:27:11.519 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.327+0000 7f55c4a18700 1 -- 172.21.15.32:0/3146475898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55c0101a90 msgr2=0x7f55c0101ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:11.519 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.327+0000 7f55c4a18700 1 --2- 172.21.15.32:0/3146475898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55c0101a90 0x7f55c0101ea0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f55b00098e0 tx=0x7f55b0009250 comp rx=0 tx=0).stop 2024-04-12T05:27:11.519 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.327+0000 7f55c4a18700 1 -- 172.21.15.32:0/3146475898 shutdown_connections 2024-04-12T05:27:11.519 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.327+0000 7f55c4a18700 1 --2- 172.21.15.32:0/3146475898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55c0101a90 0x7f55c0101ea0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:11.520 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.327+0000 7f55c4a18700 1 -- 172.21.15.32:0/3146475898 >> 172.21.15.32:0/3146475898 conn(0x7f55c00fc400 msgr2=0x7f55c00fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:11.520 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.327+0000 7f55c4a18700 1 -- 172.21.15.32:0/3146475898 shutdown_connections 2024-04-12T05:27:11.520 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.327+0000 7f55c4a18700 1 -- 172.21.15.32:0/3146475898 wait complete. 2024-04-12T05:27:11.520 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.328+0000 7f55c4a18700 1 Processor -- start 2024-04-12T05:27:11.520 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.328+0000 7f55c4a18700 1 -- start start 2024-04-12T05:27:11.520 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.328+0000 7f55c4a18700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55c0101a90 0x7f55c010fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:11.520 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.328+0000 7f55c4a18700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c010ffd0 con 0x7f55c0101a90 2024-04-12T05:27:11.520 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.328+0000 7f55be59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55c0101a90 0x7f55c010fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:11.520 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.328+0000 7f55be59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55c0101a90 0x7f55c010fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36754/0 (socket says 172.21.15.32:36754) 2024-04-12T05:27:11.520 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.328+0000 7f55be59c700 1 -- 172.21.15.32:0/3430629501 learned_addr learned my addr 172.21.15.32:0/3430629501 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:11.520 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.329+0000 7f55be59c700 1 -- 172.21.15.32:0/3430629501 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b00094f0 con 0x7f55c0101a90 2024-04-12T05:27:11.520 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.329+0000 7f55be59c700 1 --2- 172.21.15.32:0/3430629501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55c0101a90 0x7f55c010fa90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f55b0003080 tx=0x7f55b0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:11.520 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.329+0000 7f55b6ffd700 1 -- 172.21.15.32:0/3430629501 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55b001c5c0 con 0x7f55c0101a90 2024-04-12T05:27:11.520 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.329+0000 7f55b6ffd700 1 -- 172.21.15.32:0/3430629501 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55b001c720 con 0x7f55c0101a90 2024-04-12T05:27:11.520 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.329+0000 7f55c4a18700 1 -- 172.21.15.32:0/3430629501 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55c01101d0 con 0x7f55c0101a90 2024-04-12T05:27:11.521 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.329+0000 7f55b6ffd700 1 -- 172.21.15.32:0/3430629501 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55b001ca30 con 0x7f55c0101a90 2024-04-12T05:27:11.521 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.329+0000 7f55c4a18700 1 -- 172.21.15.32:0/3430629501 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55c0110d40 con 0x7f55c0101a90 2024-04-12T05:27:11.521 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.329+0000 7f55b6ffd700 1 -- 172.21.15.32:0/3430629501 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f55b001a450 con 0x7f55c0101a90 2024-04-12T05:27:11.521 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.329+0000 7f55b6ffd700 1 -- 172.21.15.32:0/3430629501 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f55b002e790 con 0x7f55c0101a90 2024-04-12T05:27:11.521 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.330+0000 7f55c4a18700 1 -- 172.21.15.32:0/3430629501 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55c005f5f0 con 0x7f55c0101a90 2024-04-12T05:27:11.521 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.332+0000 7f55b6ffd700 1 -- 172.21.15.32:0/3430629501 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f55b001a890 con 0x7f55c0101a90 2024-04-12T05:27:11.521 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.407+0000 7f55c4a18700 1 -- 172.21.15.32:0/3430629501 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f55c0111750 con 0x7f55c0101a90 2024-04-12T05:27:11.521 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.407+0000 7f55b6ffd700 1 -- 172.21.15.32:0/3430629501 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+324 (secure 0 0 0) 0x7f55b001aa30 con 0x7f55c0101a90 2024-04-12T05:27:11.521 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.408+0000 7f55c4a18700 1 -- 172.21.15.32:0/3430629501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55c0101a90 msgr2=0x7f55c010fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:11.521 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.408+0000 7f55c4a18700 1 --2- 172.21.15.32:0/3430629501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55c0101a90 0x7f55c010fa90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f55b0003080 tx=0x7f55b0016000 comp rx=0 tx=0).stop 2024-04-12T05:27:11.521 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.408+0000 7f55c4a18700 1 -- 172.21.15.32:0/3430629501 shutdown_connections 2024-04-12T05:27:11.521 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.408+0000 7f55c4a18700 1 --2- 172.21.15.32:0/3430629501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55c0101a90 0x7f55c010fa90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:11.521 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.408+0000 7f55c4a18700 1 -- 172.21.15.32:0/3430629501 >> 172.21.15.32:0/3430629501 conn(0x7f55c00fc400 msgr2=0x7f55c0103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:11.521 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.408+0000 7f55c4a18700 1 -- 172.21.15.32:0/3430629501 shutdown_connections 2024-04-12T05:27:11.522 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:09.408+0000 7f55c4a18700 1 -- 172.21.15.32:0/3430629501 wait complete. 2024-04-12T05:27:11.522 INFO:teuthology.orchestra.run.smithi032.stdout:mon is available 2024-04-12T05:27:11.522 INFO:teuthology.orchestra.run.smithi032.stdout:Assimilating anything we can from ceph.conf... 2024-04-12T05:27:12.497 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:12 smithi032 ceph-mon[39248]: from='client.? 172.21.15.32:0/4129701211' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-12T05:27:12.497 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:12 smithi032 ceph-mon[39248]: from='client.? 172.21.15.32:0/4129701211' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-12T05:27:12.708 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-12T05:27:12.708 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [global] 2024-04-12T05:27:12.708 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout fsid = 37b58ce2-f88d-11ee-bc8c-c7b262605968 2024-04-12T05:27:12.708 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.32:3300,v1:172.21.15.32:6789] 2024-04-12T05:27:12.708 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-12T05:27:12.708 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-12T05:27:12.709 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-12T05:27:12.709 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-12T05:27:12.709 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-12T05:27:12.709 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-12T05:27:12.709 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-12T05:27:12.709 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-12T05:27:12.709 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [osd] 2024-04-12T05:27:12.709 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-12T05:27:12.709 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-12T05:27:12.709 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-12T05:27:12.709 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-12T05:27:12.709 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-12T05:27:12.709 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.016+0000 7f77d3d8d700 1 Processor -- start 2024-04-12T05:27:12.709 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.016+0000 7f77d3d8d700 1 -- start start 2024-04-12T05:27:12.710 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.016+0000 7f77d3d8d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77cc076c40 0x7f77cc1051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:12.710 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.016+0000 7f77d3d8d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77cc105730 con 0x7f77cc076c40 2024-04-12T05:27:12.710 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.016+0000 7f77d1b29700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77cc076c40 0x7f77cc1051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:12.710 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.016+0000 7f77d1b29700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77cc076c40 0x7f77cc1051f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44248/0 (socket says 172.21.15.32:44248) 2024-04-12T05:27:12.710 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.016+0000 7f77d1b29700 1 -- 172.21.15.32:0/102251692 learned_addr learned my addr 172.21.15.32:0/102251692 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:12.710 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.017+0000 7f77d1b29700 1 -- 172.21.15.32:0/102251692 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77cc105870 con 0x7f77cc076c40 2024-04-12T05:27:12.710 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.017+0000 7f77d1b29700 1 --2- 172.21.15.32:0/102251692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77cc076c40 0x7f77cc1051f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f77c80098e0 tx=0x7f77c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b2f8cc0d0c6e755 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:12.710 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.017+0000 7f77d0b27700 1 -- 172.21.15.32:0/102251692 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77c800b860 con 0x7f77cc076c40 2024-04-12T05:27:12.710 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.017+0000 7f77d0b27700 1 -- 172.21.15.32:0/102251692 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77c800b9c0 con 0x7f77cc076c40 2024-04-12T05:27:12.710 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.017+0000 7f77d0b27700 1 -- 172.21.15.32:0/102251692 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77c800bc90 con 0x7f77cc076c40 2024-04-12T05:27:12.710 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.017+0000 7f77d3d8d700 1 -- 172.21.15.32:0/102251692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77cc076c40 msgr2=0x7f77cc1051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:12.710 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.017+0000 7f77d3d8d700 1 --2- 172.21.15.32:0/102251692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77cc076c40 0x7f77cc1051f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f77c80098e0 tx=0x7f77c8009250 comp rx=0 tx=0).stop 2024-04-12T05:27:12.710 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.018+0000 7f77d3d8d700 1 -- 172.21.15.32:0/102251692 shutdown_connections 2024-04-12T05:27:12.711 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.018+0000 7f77d3d8d700 1 --2- 172.21.15.32:0/102251692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77cc076c40 0x7f77cc1051f0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:12.711 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.018+0000 7f77d3d8d700 1 -- 172.21.15.32:0/102251692 >> 172.21.15.32:0/102251692 conn(0x7f77cc0fcc60 msgr2=0x7f77cc0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:12.711 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.018+0000 7f77d3d8d700 1 -- 172.21.15.32:0/102251692 shutdown_connections 2024-04-12T05:27:12.711 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.018+0000 7f77d3d8d700 1 -- 172.21.15.32:0/102251692 wait complete. 2024-04-12T05:27:12.711 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.018+0000 7f77d3d8d700 1 Processor -- start 2024-04-12T05:27:12.711 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.018+0000 7f77d3d8d700 1 -- start start 2024-04-12T05:27:12.711 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.018+0000 7f77d3d8d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77cc076c40 0x7f77cc112e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:12.711 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.018+0000 7f77d3d8d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77cc1133d0 con 0x7f77cc076c40 2024-04-12T05:27:12.711 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.018+0000 7f77d1b29700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77cc076c40 0x7f77cc112e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:12.711 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.018+0000 7f77d1b29700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77cc076c40 0x7f77cc112e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44256/0 (socket says 172.21.15.32:44256) 2024-04-12T05:27:12.711 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.018+0000 7f77d1b29700 1 -- 172.21.15.32:0/4129701211 learned_addr learned my addr 172.21.15.32:0/4129701211 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:12.711 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.019+0000 7f77d1b29700 1 -- 172.21.15.32:0/4129701211 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c80094f0 con 0x7f77cc076c40 2024-04-12T05:27:12.711 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.019+0000 7f77d1b29700 1 --2- 172.21.15.32:0/4129701211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77cc076c40 0x7f77cc112e90 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f77c8009290 tx=0x7f77c8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:12.711 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.019+0000 7f77c27fc700 1 -- 172.21.15.32:0/4129701211 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77c801c5c0 con 0x7f77cc076c40 2024-04-12T05:27:12.711 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.019+0000 7f77c27fc700 1 -- 172.21.15.32:0/4129701211 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77c801c740 con 0x7f77cc076c40 2024-04-12T05:27:12.712 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.019+0000 7f77d3d8d700 1 -- 172.21.15.32:0/4129701211 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77cc113630 con 0x7f77cc076c40 2024-04-12T05:27:12.712 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.019+0000 7f77c27fc700 1 -- 172.21.15.32:0/4129701211 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77c801ca50 con 0x7f77cc076c40 2024-04-12T05:27:12.712 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.019+0000 7f77d3d8d700 1 -- 172.21.15.32:0/4129701211 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77cc113fb0 con 0x7f77cc076c40 2024-04-12T05:27:12.712 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.019+0000 7f77c27fc700 1 -- 172.21.15.32:0/4129701211 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f77c801a450 con 0x7f77cc076c40 2024-04-12T05:27:12.712 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.019+0000 7f77c27fc700 1 -- 172.21.15.32:0/4129701211 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f77c802e790 con 0x7f77cc076c40 2024-04-12T05:27:12.712 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.020+0000 7f77d3d8d700 1 -- 172.21.15.32:0/4129701211 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77cc04bed0 con 0x7f77cc076c40 2024-04-12T05:27:12.712 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.022+0000 7f77c27fc700 1 -- 172.21.15.32:0/4129701211 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f77c801ce30 con 0x7f77cc076c40 2024-04-12T05:27:12.712 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.091+0000 7f77d3d8d700 1 -- 172.21.15.32:0/4129701211 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f77cc105b20 con 0x7f77cc076c40 2024-04-12T05:27:12.712 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.094+0000 7f77c27fc700 1 -- 172.21.15.32:0/4129701211 <== mon.0 v2:172.21.15.32:3300/0 7 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f77c801b3e0 con 0x7f77cc076c40 2024-04-12T05:27:12.712 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.094+0000 7f77c27fc700 1 -- 172.21.15.32:0/4129701211 <== mon.0 v2:172.21.15.32:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f77cc105b20 con 0x7f77cc076c40 2024-04-12T05:27:12.712 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.095+0000 7f77d3d8d700 1 -- 172.21.15.32:0/4129701211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77cc076c40 msgr2=0x7f77cc112e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:12.712 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.095+0000 7f77d3d8d700 1 --2- 172.21.15.32:0/4129701211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77cc076c40 0x7f77cc112e90 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f77c8009290 tx=0x7f77c8016000 comp rx=0 tx=0).stop 2024-04-12T05:27:12.712 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.095+0000 7f77d3d8d700 1 -- 172.21.15.32:0/4129701211 shutdown_connections 2024-04-12T05:27:12.713 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.095+0000 7f77d3d8d700 1 --2- 172.21.15.32:0/4129701211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77cc076c40 0x7f77cc112e90 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:12.713 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.095+0000 7f77d3d8d700 1 -- 172.21.15.32:0/4129701211 >> 172.21.15.32:0/4129701211 conn(0x7f77cc0fcc60 msgr2=0x7f77cc0fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:12.713 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.095+0000 7f77d3d8d700 1 -- 172.21.15.32:0/4129701211 shutdown_connections 2024-04-12T05:27:12.713 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:12.095+0000 7f77d3d8d700 1 -- 172.21.15.32:0/4129701211 wait complete. 2024-04-12T05:27:12.713 INFO:teuthology.orchestra.run.smithi032.stdout:Generating new minimal ceph.conf... 2024-04-12T05:27:13.753 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:13 smithi032 ceph-mon[39248]: from='client.? 172.21.15.32:0/2688516367' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:27:14.044 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.276+0000 7fe3cae30700 1 Processor -- start 2024-04-12T05:27:14.044 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.276+0000 7fe3cae30700 1 -- start start 2024-04-12T05:27:14.044 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.276+0000 7fe3cae30700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3c4076c20 0x7fe3c4105210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:14.044 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.276+0000 7fe3cae30700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c4105750 con 0x7fe3c4076c20 2024-04-12T05:27:14.044 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.276+0000 7fe3c8bcc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3c4076c20 0x7fe3c4105210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:14.044 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.276+0000 7fe3c8bcc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3c4076c20 0x7fe3c4105210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44272/0 (socket says 172.21.15.32:44272) 2024-04-12T05:27:14.044 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.276+0000 7fe3c8bcc700 1 -- 172.21.15.32:0/2601748844 learned_addr learned my addr 172.21.15.32:0/2601748844 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:14.045 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.277+0000 7fe3c8bcc700 1 -- 172.21.15.32:0/2601748844 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c4105890 con 0x7fe3c4076c20 2024-04-12T05:27:14.045 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.277+0000 7fe3c8bcc700 1 --2- 172.21.15.32:0/2601748844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3c4076c20 0x7fe3c4105210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe3b80098e0 tx=0x7fe3b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0fe628a55f81e47 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:14.045 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.277+0000 7fe3c37fe700 1 -- 172.21.15.32:0/2601748844 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3b800b860 con 0x7fe3c4076c20 2024-04-12T05:27:14.045 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.277+0000 7fe3c37fe700 1 -- 172.21.15.32:0/2601748844 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe3b800be30 con 0x7fe3c4076c20 2024-04-12T05:27:14.045 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.277+0000 7fe3c37fe700 1 -- 172.21.15.32:0/2601748844 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3b8010380 con 0x7fe3c4076c20 2024-04-12T05:27:14.045 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.278+0000 7fe3cae30700 1 -- 172.21.15.32:0/2601748844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3c4076c20 msgr2=0x7fe3c4105210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:14.045 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.278+0000 7fe3cae30700 1 --2- 172.21.15.32:0/2601748844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3c4076c20 0x7fe3c4105210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe3b80098e0 tx=0x7fe3b8009250 comp rx=0 tx=0).stop 2024-04-12T05:27:14.045 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.278+0000 7fe3cae30700 1 -- 172.21.15.32:0/2601748844 shutdown_connections 2024-04-12T05:27:14.045 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.278+0000 7fe3cae30700 1 --2- 172.21.15.32:0/2601748844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3c4076c20 0x7fe3c4105210 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:14.045 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.278+0000 7fe3cae30700 1 -- 172.21.15.32:0/2601748844 >> 172.21.15.32:0/2601748844 conn(0x7fe3c40fcc60 msgr2=0x7fe3c40ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:14.045 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.278+0000 7fe3cae30700 1 -- 172.21.15.32:0/2601748844 shutdown_connections 2024-04-12T05:27:14.045 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.278+0000 7fe3cae30700 1 -- 172.21.15.32:0/2601748844 wait complete. 2024-04-12T05:27:14.045 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.278+0000 7fe3cae30700 1 Processor -- start 2024-04-12T05:27:14.045 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.278+0000 7fe3cae30700 1 -- start start 2024-04-12T05:27:14.046 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.279+0000 7fe3cae30700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3c4076c20 0x7fe3c4111380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:14.046 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.279+0000 7fe3cae30700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c41118c0 con 0x7fe3c4076c20 2024-04-12T05:27:14.046 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.279+0000 7fe3c8bcc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3c4076c20 0x7fe3c4111380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:14.046 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.279+0000 7fe3c8bcc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3c4076c20 0x7fe3c4111380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44274/0 (socket says 172.21.15.32:44274) 2024-04-12T05:27:14.046 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.279+0000 7fe3c8bcc700 1 -- 172.21.15.32:0/2688516367 learned_addr learned my addr 172.21.15.32:0/2688516367 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:14.046 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.279+0000 7fe3c8bcc700 1 -- 172.21.15.32:0/2688516367 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b80094f0 con 0x7fe3c4076c20 2024-04-12T05:27:14.046 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.279+0000 7fe3c8bcc700 1 --2- 172.21.15.32:0/2688516367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3c4076c20 0x7fe3c4111380 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8006d40 tx=0x7fe3b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:14.046 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.279+0000 7fe3c17fa700 1 -- 172.21.15.32:0/2688516367 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3b80109b0 con 0x7fe3c4076c20 2024-04-12T05:27:14.046 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.279+0000 7fe3c17fa700 1 -- 172.21.15.32:0/2688516367 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe3b8024440 con 0x7fe3c4076c20 2024-04-12T05:27:14.047 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.279+0000 7fe3cae30700 1 -- 172.21.15.32:0/2688516367 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3c4111ac0 con 0x7fe3c4076c20 2024-04-12T05:27:14.047 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.279+0000 7fe3c17fa700 1 -- 172.21.15.32:0/2688516367 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3b801a5f0 con 0x7fe3c4076c20 2024-04-12T05:27:14.047 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.279+0000 7fe3cae30700 1 -- 172.21.15.32:0/2688516367 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3c4112630 con 0x7fe3c4076c20 2024-04-12T05:27:14.047 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.280+0000 7fe3c17fa700 1 -- 172.21.15.32:0/2688516367 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fe3b801aaa0 con 0x7fe3c4076c20 2024-04-12T05:27:14.047 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.280+0000 7fe3c17fa700 1 -- 172.21.15.32:0/2688516367 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fe3b802da00 con 0x7fe3c4076c20 2024-04-12T05:27:14.047 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.280+0000 7fe3cae30700 1 -- 172.21.15.32:0/2688516367 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3c405f5f0 con 0x7fe3c4076c20 2024-04-12T05:27:14.047 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.284+0000 7fe3c17fa700 1 -- 172.21.15.32:0/2688516367 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fe3b8024970 con 0x7fe3c4076c20 2024-04-12T05:27:14.047 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.352+0000 7fe3cae30700 1 -- 172.21.15.32:0/2688516367 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fe3c404bed0 con 0x7fe3c4076c20 2024-04-12T05:27:14.047 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.352+0000 7fe3c17fa700 1 -- 172.21.15.32:0/2688516367 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fe3b8006d70 con 0x7fe3c4076c20 2024-04-12T05:27:14.047 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.353+0000 7fe3cae30700 1 -- 172.21.15.32:0/2688516367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3c4076c20 msgr2=0x7fe3c4111380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:14.047 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.353+0000 7fe3cae30700 1 --2- 172.21.15.32:0/2688516367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3c4076c20 0x7fe3c4111380 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8006d40 tx=0x7fe3b8009250 comp rx=0 tx=0).stop 2024-04-12T05:27:14.047 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.353+0000 7fe3cae30700 1 -- 172.21.15.32:0/2688516367 shutdown_connections 2024-04-12T05:27:14.047 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.353+0000 7fe3cae30700 1 --2- 172.21.15.32:0/2688516367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3c4076c20 0x7fe3c4111380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:14.048 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.353+0000 7fe3cae30700 1 -- 172.21.15.32:0/2688516367 >> 172.21.15.32:0/2688516367 conn(0x7fe3c40fcc60 msgr2=0x7fe3c40fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:14.048 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.353+0000 7fe3cae30700 1 -- 172.21.15.32:0/2688516367 shutdown_connections 2024-04-12T05:27:14.048 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:13.353+0000 7fe3cae30700 1 -- 172.21.15.32:0/2688516367 wait complete. 2024-04-12T05:27:14.048 INFO:teuthology.orchestra.run.smithi032.stdout:Restarting the monitor... 2024-04-12T05:27:14.305 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:14 smithi032 systemd[1]: Stopping Ceph mon.smithi032 for 37b58ce2-f88d-11ee-bc8c-c7b262605968... 2024-04-12T05:27:14.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:14 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-mon-smithi032[39225]: 2024-04-12T05:27:14.302+0000 7f7152495700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi032 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T05:27:14.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:14 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-mon-smithi032[39225]: 2024-04-12T05:27:14.302+0000 7f7152495700 -1 mon.smithi032@0(leader) e1 *** Got Signal Terminated *** 2024-04-12T05:27:15.094 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:14 smithi032 bash[39574]: ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-mon-smithi032 2024-04-12T05:27:15.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 systemd[1]: ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@mon.smithi032.service: Succeeded. 2024-04-12T05:27:15.345 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 systemd[1]: Stopped Ceph mon.smithi032 for 37b58ce2-f88d-11ee-bc8c-c7b262605968. 2024-04-12T05:27:15.345 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 systemd[1]: Starting Ceph mon.smithi032 for 37b58ce2-f88d-11ee-bc8c-c7b262605968... 2024-04-12T05:27:15.789 INFO:teuthology.orchestra.run.smithi032.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-12T05:27:15.989 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: set uid:gid to 167:167 (ceph:ceph) 2024-04-12T05:27:15.989 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable), process ceph-mon, pid 7 2024-04-12T05:27:15.989 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: pidfile_write: ignore empty --pid-file 2024-04-12T05:27:15.989 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: load: jerasure load: lrc 2024-04-12T05:27:15.989 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: RocksDB version: 6.15.5 2024-04-12T05:27:15.990 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-12T05:27:15.990 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Compile date Apr 9 2024 2024-04-12T05:27:15.990 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: DB SUMMARY 2024-04-12T05:27:15.990 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: DB Session ID: ZM0W0MFFDAUHT5YD199R 2024-04-12T05:27:15.990 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: CURRENT file: CURRENT 2024-04-12T05:27:15.990 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: IDENTITY file: IDENTITY 2024-04-12T05:27:15.990 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-04-12T05:27:15.990 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi032/store.db dir, Total Num: 1, files: 000008.sst 2024-04-12T05:27:15.990 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi032/store.db: 000010.log size: 97025 ; 2024-04-12T05:27:15.990 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.error_if_exists: 0 2024-04-12T05:27:15.990 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.create_if_missing: 0 2024-04-12T05:27:15.990 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.paranoid_checks: 1 2024-04-12T05:27:15.990 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-12T05:27:15.990 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.env: 0x5612274e2a80 2024-04-12T05:27:15.991 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.fs: Posix File System 2024-04-12T05:27:15.991 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.info_log: 0x561228277c00 2024-04-12T05:27:15.991 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.max_file_opening_threads: 16 2024-04-12T05:27:15.991 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.statistics: (nil) 2024-04-12T05:27:15.991 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.use_fsync: 0 2024-04-12T05:27:15.991 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.max_log_file_size: 0 2024-04-12T05:27:15.991 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-12T05:27:15.991 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-12T05:27:15.991 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.keep_log_file_num: 1000 2024-04-12T05:27:15.991 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.recycle_log_file_num: 0 2024-04-12T05:27:15.991 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.allow_fallocate: 1 2024-04-12T05:27:15.991 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.allow_mmap_reads: 0 2024-04-12T05:27:15.991 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.allow_mmap_writes: 0 2024-04-12T05:27:15.991 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.use_direct_reads: 0 2024-04-12T05:27:15.992 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-12T05:27:15.992 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.create_missing_column_families: 0 2024-04-12T05:27:15.992 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.db_log_dir: 2024-04-12T05:27:15.992 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi032/store.db 2024-04-12T05:27:15.992 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-12T05:27:15.992 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-12T05:27:15.992 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-12T05:27:15.992 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-12T05:27:15.992 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-12T05:27:15.992 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-12T05:27:15.992 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.advise_random_on_open: 1 2024-04-12T05:27:15.992 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.db_write_buffer_size: 0 2024-04-12T05:27:15.992 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.write_buffer_manager: 0x561228fec9f0 2024-04-12T05:27:15.992 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-12T05:27:15.992 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-12T05:27:15.993 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-12T05:27:15.993 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-12T05:27:15.993 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.rate_limiter: (nil) 2024-04-12T05:27:15.993 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-12T05:27:15.993 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.wal_recovery_mode: 2 2024-04-12T05:27:15.993 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.enable_thread_tracking: 0 2024-04-12T05:27:15.993 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.enable_pipelined_write: 0 2024-04-12T05:27:15.993 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.unordered_write: 0 2024-04-12T05:27:15.993 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-12T05:27:15.993 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-12T05:27:15.993 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-12T05:27:15.993 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-12T05:27:15.993 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.row_cache: None 2024-04-12T05:27:15.993 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.wal_filter: None 2024-04-12T05:27:15.993 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-12T05:27:15.994 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.allow_ingest_behind: 0 2024-04-12T05:27:15.994 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.preserve_deletes: 0 2024-04-12T05:27:15.994 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.two_write_queues: 0 2024-04-12T05:27:15.994 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.manual_wal_flush: 0 2024-04-12T05:27:15.994 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.atomic_flush: 0 2024-04-12T05:27:15.994 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-12T05:27:15.994 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-12T05:27:15.994 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-12T05:27:15.994 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.log_readahead_size: 0 2024-04-12T05:27:15.994 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-12T05:27:15.994 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.best_efforts_recovery: 0 2024-04-12T05:27:15.994 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-12T05:27:15.994 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-12T05:27:15.994 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.allow_data_in_errors: 0 2024-04-12T05:27:15.994 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.db_host_id: __hostname__ 2024-04-12T05:27:15.995 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.max_background_jobs: 2 2024-04-12T05:27:15.996 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.max_background_compactions: -1 2024-04-12T05:27:15.996 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.max_subcompactions: 1 2024-04-12T05:27:15.996 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-12T05:27:15.996 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-12T05:27:15.996 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-12T05:27:15.996 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.max_total_wal_size: 0 2024-04-12T05:27:15.996 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-12T05:27:15.996 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-12T05:27:15.996 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-12T05:27:15.996 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-12T05:27:15.996 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.max_open_files: -1 2024-04-12T05:27:15.996 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.bytes_per_sync: 0 2024-04-12T05:27:15.997 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-12T05:27:15.997 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-12T05:27:15.997 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.compaction_readahead_size: 0 2024-04-12T05:27:15.997 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.max_background_flushes: -1 2024-04-12T05:27:15.997 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Compression algorithms supported: 2024-04-12T05:27:15.997 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-12T05:27:15.997 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: kZSTD supported: 0 2024-04-12T05:27:15.997 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: kXpressCompression supported: 0 2024-04-12T05:27:15.997 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: kLZ4HCCompression supported: 1 2024-04-12T05:27:15.997 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: kZlibCompression supported: 1 2024-04-12T05:27:15.997 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: kSnappyCompression supported: 1 2024-04-12T05:27:15.997 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: kLZ4Compression supported: 1 2024-04-12T05:27:15.997 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: kBZip2Compression supported: 0 2024-04-12T05:27:15.997 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-12T05:27:15.997 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi032/store.db/MANIFEST-000009 2024-04-12T05:27:15.998 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-04-12T05:27:15.998 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-12T05:27:15.998 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.merge_operator: 2024-04-12T05:27:15.998 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.compaction_filter: None 2024-04-12T05:27:15.998 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.compaction_filter_factory: None 2024-04-12T05:27:15.998 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.sst_partitioner_factory: None 2024-04-12T05:27:15.998 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-12T05:27:15.998 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-12T05:27:15.998 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5612282ccf58) 2024-04-12T05:27:15.998 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: cache_index_and_filter_blocks: 1 2024-04-12T05:27:15.998 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-12T05:27:15.998 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-12T05:27:15.998 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: pin_top_level_index_and_filter: 1 2024-04-12T05:27:15.998 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: index_type: 0 2024-04-12T05:27:15.998 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: data_block_index_type: 0 2024-04-12T05:27:15.999 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: index_shortening: 1 2024-04-12T05:27:15.999 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-12T05:27:15.999 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: hash_index_allow_collision: 1 2024-04-12T05:27:15.999 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: checksum: 1 2024-04-12T05:27:15.999 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: no_block_cache: 0 2024-04-12T05:27:15.999 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_cache: 0x5612282ff090 2024-04-12T05:27:15.999 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_cache_name: BinnedLRUCache 2024-04-12T05:27:15.999 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_cache_options: 2024-04-12T05:27:15.999 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: capacity : 536870912 2024-04-12T05:27:15.999 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: num_shard_bits : 4 2024-04-12T05:27:15.999 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: strict_capacity_limit : 0 2024-04-12T05:27:15.999 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: high_pri_pool_ratio: 0.000 2024-04-12T05:27:15.999 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_cache_compressed: (nil) 2024-04-12T05:27:15.999 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: persistent_cache: (nil) 2024-04-12T05:27:16.000 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_size: 4096 2024-04-12T05:27:16.000 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_size_deviation: 10 2024-04-12T05:27:16.000 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_restart_interval: 16 2024-04-12T05:27:16.000 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: index_block_restart_interval: 1 2024-04-12T05:27:16.000 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: metadata_block_size: 4096 2024-04-12T05:27:16.000 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: partition_filters: 0 2024-04-12T05:27:16.000 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: use_delta_encoding: 1 2024-04-12T05:27:16.000 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-12T05:27:16.000 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: whole_key_filtering: 1 2024-04-12T05:27:16.000 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: verify_compression: 0 2024-04-12T05:27:16.000 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: read_amp_bytes_per_bit: 0 2024-04-12T05:27:16.000 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: format_version: 4 2024-04-12T05:27:16.000 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: enable_index_compression: 1 2024-04-12T05:27:16.000 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_align: 0 2024-04-12T05:27:16.000 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.write_buffer_size: 33554432 2024-04-12T05:27:16.001 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.max_write_buffer_number: 2 2024-04-12T05:27:16.001 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.compression: NoCompression 2024-04-12T05:27:16.001 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.bottommost_compression: Disabled 2024-04-12T05:27:16.001 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.prefix_extractor: nullptr 2024-04-12T05:27:16.001 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-12T05:27:16.001 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.num_levels: 7 2024-04-12T05:27:16.001 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-12T05:27:16.001 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-12T05:27:16.001 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-12T05:27:16.001 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-12T05:27:16.001 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-12T05:27:16.001 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-12T05:27:16.001 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-12T05:27:16.001 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-12T05:27:16.001 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-12T05:27:16.002 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-12T05:27:16.002 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-12T05:27:16.002 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.compression_opts.level: 32767 2024-04-12T05:27:16.002 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.compression_opts.strategy: 0 2024-04-12T05:27:16.002 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-12T05:27:16.002 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-12T05:27:16.002 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-12T05:27:16.002 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.compression_opts.enabled: false 2024-04-12T05:27:16.002 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-12T05:27:16.002 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-12T05:27:16.002 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-12T05:27:16.002 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.target_file_size_base: 67108864 2024-04-12T05:27:16.002 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-12T05:27:16.002 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-12T05:27:16.002 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-12T05:27:16.003 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-12T05:27:16.003 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-12T05:27:16.003 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-12T05:27:16.003 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-12T05:27:16.003 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-12T05:27:16.003 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-12T05:27:16.003 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-12T05:27:16.003 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-12T05:27:16.003 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-12T05:27:16.003 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-12T05:27:16.003 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.arena_block_size: 4194304 2024-04-12T05:27:16.003 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-12T05:27:16.003 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-12T05:27:16.003 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-12T05:27:16.004 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.disable_auto_compactions: 0 2024-04-12T05:27:16.004 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-12T05:27:16.004 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-12T05:27:16.004 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-12T05:27:16.004 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-12T05:27:16.004 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-12T05:27:16.004 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-12T05:27:16.004 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-12T05:27:16.004 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-12T05:27:16.004 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-12T05:27:16.004 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-12T05:27:16.004 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.table_properties_collectors: 2024-04-12T05:27:16.004 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.inplace_update_support: 0 2024-04-12T05:27:16.004 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-12T05:27:16.004 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-12T05:27:16.004 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-12T05:27:16.005 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-12T05:27:16.005 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.bloom_locality: 0 2024-04-12T05:27:16.005 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.max_successive_merges: 0 2024-04-12T05:27:16.005 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-12T05:27:16.005 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.paranoid_file_checks: 0 2024-04-12T05:27:16.005 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.force_consistency_checks: 1 2024-04-12T05:27:16.005 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.report_bg_io_stats: 0 2024-04-12T05:27:16.005 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.ttl: 2592000 2024-04-12T05:27:16.005 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-12T05:27:16.005 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.enable_blob_files: false 2024-04-12T05:27:16.005 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.min_blob_size: 0 2024-04-12T05:27:16.005 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.blob_file_size: 268435456 2024-04-12T05:27:16.005 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-12T05:27:16.005 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-12T05:27:16.005 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-12T05:27:16.006 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi032/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-12T05:27:16.006 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-04-12T05:27:16.006 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-04-12T05:27:16.006 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712899635736619, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-04-12T05:27:16.006 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-04-12T05:27:16.006 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-12T05:27:16.006 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712899635737419, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 92247, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 90342, "index_size": 265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13255, "raw_average_key_size": 49, "raw_value_size": 83054, "raw_average_value_size": 308, "num_data_blocks": 11, "num_entries": 269, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1712899635, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "379414ef-b39f-4476-bfa9-d577bdbe733d", "db_session_id": "ZM0W0MFFDAUHT5YD199R"}} 2024-04-12T05:27:16.006 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-04-12T05:27:16.006 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712899635737801, "job": 1, "event": "recovery_finished"} 2024-04-12T05:27:16.006 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi032/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-12T05:27:16.006 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x561228293180 2024-04-12T05:27:16.006 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: DB pointer 0x561228368000 2024-04-12T05:27:16.006 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-04-12T05:27:16.006 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-12T05:27:16.007 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** DB Stats ** 2024-04-12T05:27:16.007 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T05:27:16.007 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-12T05:27:16.007 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-12T05:27:16.007 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-12T05:27:16.007 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-12T05:27:16.007 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-12T05:27:16.007 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-12T05:27:16.007 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-12T05:27:16.007 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** Compaction Stats [default] ** 2024-04-12T05:27:16.007 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-12T05:27:16.007 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T05:27:16.007 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: L0 2/0 91.88 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 172.2 0.00 0.00 1 0.001 0 0 2024-04-12T05:27:16.007 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Sum 2/0 91.88 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 172.2 0.00 0.00 1 0.001 0 0 2024-04-12T05:27:16.008 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 172.2 0.00 0.00 1 0.001 0 0 2024-04-12T05:27:16.008 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-12T05:27:16.008 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** Compaction Stats [default] ** 2024-04-12T05:27:16.008 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-12T05:27:16.008 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T05:27:16.008 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 172.2 0.00 0.00 1 0.001 0 0 2024-04-12T05:27:16.008 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T05:27:16.008 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-12T05:27:16.008 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-12T05:27:16.008 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-12T05:27:16.008 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-12T05:27:16.008 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-12T05:27:16.008 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Cumulative compaction: 0.00 GB write, 27.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T05:27:16.008 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Interval compaction: 0.00 GB write, 27.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T05:27:16.009 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-12T05:27:16.009 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-12T05:27:16.009 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-12T05:27:16.009 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-12T05:27:16.009 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** Compaction Stats [default] ** 2024-04-12T05:27:16.009 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-12T05:27:16.009 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T05:27:16.009 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: L0 2/0 91.88 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 172.2 0.00 0.00 1 0.001 0 0 2024-04-12T05:27:16.009 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Sum 2/0 91.88 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 172.2 0.00 0.00 1 0.001 0 0 2024-04-12T05:27:16.009 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-12T05:27:16.009 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-12T05:27:16.009 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** Compaction Stats [default] ** 2024-04-12T05:27:16.009 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-12T05:27:16.009 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T05:27:16.009 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 172.2 0.00 0.00 1 0.001 0 0 2024-04-12T05:27:16.010 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T05:27:16.010 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-12T05:27:16.010 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-12T05:27:16.010 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-12T05:27:16.010 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-12T05:27:16.010 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-12T05:27:16.010 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Cumulative compaction: 0.00 GB write, 27.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T05:27:16.010 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T05:27:16.010 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-12T05:27:16.010 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-12T05:27:16.010 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-12T05:27:16.010 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: starting mon.smithi032 rank 0 at public addrs [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] at bind addrs [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi032 fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 2024-04-12T05:27:16.010 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: mon.smithi032@-1(???) e1 preinit fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 2024-04-12T05:27:16.011 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: mon.smithi032@-1(???).mds e1 new map 2024-04-12T05:27:16.011 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: mon.smithi032@-1(???).mds e1 print_map 2024-04-12T05:27:16.011 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: e1 2024-04-12T05:27:16.011 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:27:16.011 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:27:16.011 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: legacy client fscid: -1 2024-04-12T05:27:16.011 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-12T05:27:16.011 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: No filesystems configured 2024-04-12T05:27:16.011 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: mon.smithi032@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-12T05:27:16.011 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: mon.smithi032@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-12T05:27:16.011 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: mon.smithi032@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-12T05:27:16.011 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: mon.smithi032@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-12T05:27:16.011 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: mon.smithi032@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-12T05:27:16.011 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 bash[39693]: 75aad95b984d4b850138dd118585d21b030b0339e25453cac625c2ae0776e96f 2024-04-12T05:27:16.011 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 systemd[1]: Started Ceph mon.smithi032 for 37b58ce2-f88d-11ee-bc8c-c7b262605968. 2024-04-12T05:27:16.012 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: mon.smithi032 is new leader, mons smithi032 in quorum (ranks 0) 2024-04-12T05:27:16.012 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: monmap e1: 1 mons at {smithi032=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-12T05:27:16.012 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: fsmap 2024-04-12T05:27:16.012 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: osdmap e1: 0 total, 0 up, 0 in 2024-04-12T05:27:16.012 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:15 smithi032 ceph-mon[39731]: mgrmap e1: no daemons active 2024-04-12T05:27:16.948 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.304+0000 7f7cfee0d700 1 Processor -- start 2024-04-12T05:27:16.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.304+0000 7f7cfee0d700 1 -- start start 2024-04-12T05:27:16.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.304+0000 7f7cfee0d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cf81030c0 0x7f7cf81054a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:16.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.304+0000 7f7cfee0d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf8076d20 con 0x7f7cf81030c0 2024-04-12T05:27:16.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.304+0000 7f7cfcba9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cf81030c0 0x7f7cf81054a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:16.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.304+0000 7f7cfcba9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cf81030c0 0x7f7cf81054a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44288/0 (socket says 172.21.15.32:44288) 2024-04-12T05:27:16.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.304+0000 7f7cfcba9700 1 -- 172.21.15.32:0/1500716733 learned_addr learned my addr 172.21.15.32:0/1500716733 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:16.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.305+0000 7f7cfcba9700 1 -- 172.21.15.32:0/1500716733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf8076e60 con 0x7f7cf81030c0 2024-04-12T05:27:16.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.305+0000 7f7cfcba9700 1 --2- 172.21.15.32:0/1500716733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cf81030c0 0x7f7cf81054a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7cec0098e0 tx=0x7f7cec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=982e5798b43b04df server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:16.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.305+0000 7f7cf77fe700 1 -- 172.21.15.32:0/1500716733 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cec00b860 con 0x7f7cf81030c0 2024-04-12T05:27:16.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.305+0000 7f7cf77fe700 1 -- 172.21.15.32:0/1500716733 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f7cec00be30 con 0x7f7cf81030c0 2024-04-12T05:27:16.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.305+0000 7f7cf77fe700 1 -- 172.21.15.32:0/1500716733 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cec010380 con 0x7f7cf81030c0 2024-04-12T05:27:16.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.305+0000 7f7cfee0d700 1 -- 172.21.15.32:0/1500716733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cf81030c0 msgr2=0x7f7cf81054a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:16.950 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.305+0000 7f7cfee0d700 1 --2- 172.21.15.32:0/1500716733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cf81030c0 0x7f7cf81054a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7cec0098e0 tx=0x7f7cec009250 comp rx=0 tx=0).stop 2024-04-12T05:27:16.950 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.306+0000 7f7cfee0d700 1 -- 172.21.15.32:0/1500716733 shutdown_connections 2024-04-12T05:27:16.950 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.306+0000 7f7cfee0d700 1 --2- 172.21.15.32:0/1500716733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cf81030c0 0x7f7cf81054a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:16.950 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.306+0000 7f7cfee0d700 1 -- 172.21.15.32:0/1500716733 >> 172.21.15.32:0/1500716733 conn(0x7f7cf80fcb20 msgr2=0x7f7cf80fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:16.950 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.306+0000 7f7cfee0d700 1 -- 172.21.15.32:0/1500716733 shutdown_connections 2024-04-12T05:27:16.950 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.306+0000 7f7cfee0d700 1 -- 172.21.15.32:0/1500716733 wait complete. 2024-04-12T05:27:16.950 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.306+0000 7f7cfee0d700 1 Processor -- start 2024-04-12T05:27:16.950 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.306+0000 7f7cfee0d700 1 -- start start 2024-04-12T05:27:16.950 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.306+0000 7f7cfee0d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cf81030c0 0x7f7cf8111390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:16.950 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.306+0000 7f7cfee0d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf81118d0 con 0x7f7cf81030c0 2024-04-12T05:27:16.950 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.307+0000 7f7cfcba9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cf81030c0 0x7f7cf8111390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:16.950 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.307+0000 7f7cfcba9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cf81030c0 0x7f7cf8111390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44298/0 (socket says 172.21.15.32:44298) 2024-04-12T05:27:16.950 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.307+0000 7f7cfcba9700 1 -- 172.21.15.32:0/3104957981 learned_addr learned my addr 172.21.15.32:0/3104957981 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:16.951 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.307+0000 7f7cfcba9700 1 -- 172.21.15.32:0/3104957981 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cec0094f0 con 0x7f7cf81030c0 2024-04-12T05:27:16.951 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.307+0000 7f7cfcba9700 1 --2- 172.21.15.32:0/3104957981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cf81030c0 0x7f7cf8111390 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7cec006d40 tx=0x7f7cec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:16.951 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.307+0000 7f7cf57fa700 1 -- 172.21.15.32:0/3104957981 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cec0109b0 con 0x7f7cf81030c0 2024-04-12T05:27:16.951 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.307+0000 7f7cf57fa700 1 -- 172.21.15.32:0/3104957981 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f7cec024440 con 0x7f7cf81030c0 2024-04-12T05:27:16.951 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.307+0000 7f7cfee0d700 1 -- 172.21.15.32:0/3104957981 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cf8111ad0 con 0x7f7cf81030c0 2024-04-12T05:27:16.951 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.307+0000 7f7cf57fa700 1 -- 172.21.15.32:0/3104957981 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cec01a5f0 con 0x7f7cf81030c0 2024-04-12T05:27:16.951 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.307+0000 7f7cfee0d700 1 -- 172.21.15.32:0/3104957981 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf8112640 con 0x7f7cf81030c0 2024-04-12T05:27:16.951 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.307+0000 7f7cf57fa700 1 -- 172.21.15.32:0/3104957981 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f7cec01aaa0 con 0x7f7cf81030c0 2024-04-12T05:27:16.951 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.307+0000 7f7cf57fa700 1 -- 172.21.15.32:0/3104957981 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f7cec02da40 con 0x7f7cf81030c0 2024-04-12T05:27:16.951 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.308+0000 7f7cfee0d700 1 -- 172.21.15.32:0/3104957981 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cf805f5f0 con 0x7f7cf81030c0 2024-04-12T05:27:16.951 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.310+0000 7f7cf57fa700 1 -- 172.21.15.32:0/3104957981 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f7cec024970 con 0x7f7cf81030c0 2024-04-12T05:27:16.951 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.379+0000 7f7cfee0d700 1 -- 172.21.15.32:0/3104957981 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f7cf8106570 con 0x7f7cf81030c0 2024-04-12T05:27:16.951 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.381+0000 7f7cf57fa700 1 -- 172.21.15.32:0/3104957981 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f7cec0245b0 con 0x7f7cf81030c0 2024-04-12T05:27:16.951 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.382+0000 7f7cfee0d700 1 -- 172.21.15.32:0/3104957981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cf81030c0 msgr2=0x7f7cf8111390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:16.952 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.382+0000 7f7cfee0d700 1 --2- 172.21.15.32:0/3104957981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cf81030c0 0x7f7cf8111390 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7cec006d40 tx=0x7f7cec009250 comp rx=0 tx=0).stop 2024-04-12T05:27:16.952 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.382+0000 7f7cfee0d700 1 -- 172.21.15.32:0/3104957981 shutdown_connections 2024-04-12T05:27:16.952 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.382+0000 7f7cfee0d700 1 --2- 172.21.15.32:0/3104957981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cf81030c0 0x7f7cf8111390 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:16.952 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.382+0000 7f7cfee0d700 1 -- 172.21.15.32:0/3104957981 >> 172.21.15.32:0/3104957981 conn(0x7f7cf80fcb20 msgr2=0x7f7cf80fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:16.952 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.382+0000 7f7cfee0d700 1 -- 172.21.15.32:0/3104957981 shutdown_connections 2024-04-12T05:27:16.952 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:16.382+0000 7f7cfee0d700 1 -- 172.21.15.32:0/3104957981 wait complete. 2024-04-12T05:27:16.952 INFO:teuthology.orchestra.run.smithi032.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-12T05:27:16.952 INFO:teuthology.orchestra.run.smithi032.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-12T05:27:16.952 INFO:teuthology.orchestra.run.smithi032.stdout:Creating mgr... 2024-04-12T05:27:16.952 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying port 9283 ... 2024-04-12T05:27:17.085 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@mgr.smithi032.hrwxrh 2024-04-12T05:27:17.085 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to reset failed state of unit ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@mgr.smithi032.hrwxrh.service: Unit ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@mgr.smithi032.hrwxrh.service not loaded. 2024-04-12T05:27:17.183 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968.target.wants/ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@mgr.smithi032.hrwxrh.service → /etc/systemd/system/ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@.service. 2024-04-12T05:27:17.845 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:17 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/3104957981' entity='client.admin' 2024-04-12T05:27:17.859 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-12T05:27:17.859 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout disabled 2024-04-12T05:27:17.868 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-12T05:27:17.869 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-12T05:27:17.869 INFO:teuthology.orchestra.run.smithi032.stdout:firewalld.service is not enabled 2024-04-12T05:27:17.869 INFO:teuthology.orchestra.run.smithi032.stdout:Not possible to enable service . firewalld.service is not available 2024-04-12T05:27:17.879 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-12T05:27:17.879 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout disabled 2024-04-12T05:27:17.887 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-12T05:27:17.887 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-12T05:27:17.887 INFO:teuthology.orchestra.run.smithi032.stdout:firewalld.service is not enabled 2024-04-12T05:27:17.887 INFO:teuthology.orchestra.run.smithi032.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-04-12T05:27:17.887 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mgr to start... 2024-04-12T05:27:17.887 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mgr... 2024-04-12T05:27:18.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:18 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/3738743734' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-12T05:27:19.172 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-12T05:27:19.173 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-12T05:27:19.173 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsid": "37b58ce2-f88d-11ee-bc8c-c7b262605968", 2024-04-12T05:27:19.173 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "health": { 2024-04-12T05:27:19.173 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-12T05:27:19.173 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-12T05:27:19.173 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-12T05:27:19.173 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-12T05:27:19.173 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-12T05:27:19.173 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-12T05:27:19.173 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 0 2024-04-12T05:27:19.173 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-12T05:27:19.173 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-12T05:27:19.173 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "smithi032" 2024-04-12T05:27:19.173 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-12T05:27:19.174 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-12T05:27:19.174 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-12T05:27:19.174 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-12T05:27:19.174 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-04-12T05:27:19.174 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-12T05:27:19.174 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-12T05:27:19.174 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-12T05:27:19.174 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-12T05:27:19.174 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-12T05:27:19.174 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-12T05:27:19.174 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-12T05:27:19.182 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-12T05:27:19.182 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-12T05:27:19.182 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-12T05:27:19.182 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-12T05:27:19.182 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-12T05:27:19.182 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-12T05:27:19.182 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-12T05:27:19.182 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-12T05:27:19.182 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-12T05:27:19.182 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-12T05:27:19.183 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-12T05:27:19.183 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-12T05:27:19.183 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-12T05:27:19.183 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-12T05:27:19.183 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-12T05:27:19.183 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-12T05:27:19.183 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-12T05:27:19.183 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-12T05:27:19.183 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-12T05:27:19.183 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-12T05:27:19.183 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-12T05:27:19.183 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-12T05:27:19.183 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-12T05:27:19.183 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-12T05:27:19.183 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-12T05:27:19.183 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "restful" 2024-04-12T05:27:19.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-12T05:27:19.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-12T05:27:19.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-12T05:27:19.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-12T05:27:19.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-12T05:27:19.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modified": "2024-04-12T05:27:07.779861+0000", 2024-04-12T05:27:19.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-12T05:27:19.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-12T05:27:19.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-12T05:27:19.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-12T05:27:19.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.428+0000 7f3be41c8700 1 Processor -- start 2024-04-12T05:27:19.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.428+0000 7f3be41c8700 1 -- start start 2024-04-12T05:27:19.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.428+0000 7f3be41c8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bdc102580 0x7f3bdc102990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:19.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.428+0000 7f3be41c8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bdc102ed0 con 0x7f3bdc102580 2024-04-12T05:27:19.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.429+0000 7f3be1f64700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bdc102580 0x7f3bdc102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:19.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.429+0000 7f3be1f64700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bdc102580 0x7f3bdc102990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44328/0 (socket says 172.21.15.32:44328) 2024-04-12T05:27:19.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.429+0000 7f3be1f64700 1 -- 172.21.15.32:0/3039631933 learned_addr learned my addr 172.21.15.32:0/3039631933 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:19.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.429+0000 7f3be1f64700 1 -- 172.21.15.32:0/3039631933 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bdc103010 con 0x7f3bdc102580 2024-04-12T05:27:19.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.429+0000 7f3be1f64700 1 --2- 172.21.15.32:0/3039631933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bdc102580 0x7f3bdc102990 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3bd80098e0 tx=0x7f3bd8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab5f42aec8ab4eea server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:19.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.430+0000 7f3be0f62700 1 -- 172.21.15.32:0/3039631933 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bd800b860 con 0x7f3bdc102580 2024-04-12T05:27:19.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.430+0000 7f3be0f62700 1 -- 172.21.15.32:0/3039631933 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f3bd800be30 con 0x7f3bdc102580 2024-04-12T05:27:19.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.430+0000 7f3be0f62700 1 -- 172.21.15.32:0/3039631933 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bd8010380 con 0x7f3bdc102580 2024-04-12T05:27:19.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.430+0000 7f3be41c8700 1 -- 172.21.15.32:0/3039631933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bdc102580 msgr2=0x7f3bdc102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:19.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.430+0000 7f3be41c8700 1 --2- 172.21.15.32:0/3039631933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bdc102580 0x7f3bdc102990 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3bd80098e0 tx=0x7f3bd8009250 comp rx=0 tx=0).stop 2024-04-12T05:27:19.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.430+0000 7f3be41c8700 1 -- 172.21.15.32:0/3039631933 shutdown_connections 2024-04-12T05:27:19.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.430+0000 7f3be41c8700 1 --2- 172.21.15.32:0/3039631933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bdc102580 0x7f3bdc102990 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:19.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.430+0000 7f3be41c8700 1 -- 172.21.15.32:0/3039631933 >> 172.21.15.32:0/3039631933 conn(0x7f3bdc0fdd00 msgr2=0x7f3bdc100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:19.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.430+0000 7f3be41c8700 1 -- 172.21.15.32:0/3039631933 shutdown_connections 2024-04-12T05:27:19.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.430+0000 7f3be41c8700 1 -- 172.21.15.32:0/3039631933 wait complete. 2024-04-12T05:27:19.186 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.431+0000 7f3be41c8700 1 Processor -- start 2024-04-12T05:27:19.186 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.431+0000 7f3be41c8700 1 -- start start 2024-04-12T05:27:19.186 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.431+0000 7f3be41c8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bdc102580 0x7f3bdc078c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:19.186 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.431+0000 7f3be41c8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bdc079190 con 0x7f3bdc102580 2024-04-12T05:27:19.186 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.431+0000 7f3be1f64700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bdc102580 0x7f3bdc078c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:19.186 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.431+0000 7f3be1f64700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bdc102580 0x7f3bdc078c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44344/0 (socket says 172.21.15.32:44344) 2024-04-12T05:27:19.186 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.431+0000 7f3be1f64700 1 -- 172.21.15.32:0/3738743734 learned_addr learned my addr 172.21.15.32:0/3738743734 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:19.186 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.431+0000 7f3be1f64700 1 -- 172.21.15.32:0/3738743734 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd80094f0 con 0x7f3bdc102580 2024-04-12T05:27:19.186 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.432+0000 7f3be1f64700 1 --2- 172.21.15.32:0/3738743734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bdc102580 0x7f3bdc078c50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3bd8006b00 tx=0x7f3bd8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:19.186 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.432+0000 7f3bd27fc700 1 -- 172.21.15.32:0/3738743734 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bd80109b0 con 0x7f3bdc102580 2024-04-12T05:27:19.186 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.432+0000 7f3bd27fc700 1 -- 172.21.15.32:0/3738743734 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f3bd8024440 con 0x7f3bdc102580 2024-04-12T05:27:19.186 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.432+0000 7f3be41c8700 1 -- 172.21.15.32:0/3738743734 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bdc077200 con 0x7f3bdc102580 2024-04-12T05:27:19.186 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.432+0000 7f3bd27fc700 1 -- 172.21.15.32:0/3738743734 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bd801a5f0 con 0x7f3bdc102580 2024-04-12T05:27:19.186 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.432+0000 7f3be41c8700 1 -- 172.21.15.32:0/3738743734 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bdc0776f0 con 0x7f3bdc102580 2024-04-12T05:27:19.187 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.432+0000 7f3bd27fc700 1 -- 172.21.15.32:0/3738743734 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f3bd801aae0 con 0x7f3bdc102580 2024-04-12T05:27:19.187 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.432+0000 7f3bd27fc700 1 -- 172.21.15.32:0/3738743734 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f3bd802db40 con 0x7f3bdc102580 2024-04-12T05:27:19.187 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.433+0000 7f3be41c8700 1 -- 172.21.15.32:0/3738743734 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bdc04bed0 con 0x7f3bdc102580 2024-04-12T05:27:19.187 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.435+0000 7f3bd27fc700 1 -- 172.21.15.32:0/3738743734 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f3bd8006d70 con 0x7f3bdc102580 2024-04-12T05:27:19.187 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.508+0000 7f3be41c8700 1 -- 172.21.15.32:0/3738743734 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f3bdc077930 con 0x7f3bdc102580 2024-04-12T05:27:19.187 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.508+0000 7f3bd27fc700 1 -- 172.21.15.32:0/3738743734 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f3bd8032030 con 0x7f3bdc102580 2024-04-12T05:27:19.187 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.509+0000 7f3be41c8700 1 -- 172.21.15.32:0/3738743734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bdc102580 msgr2=0x7f3bdc078c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:19.187 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.509+0000 7f3be41c8700 1 --2- 172.21.15.32:0/3738743734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bdc102580 0x7f3bdc078c50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3bd8006b00 tx=0x7f3bd8009250 comp rx=0 tx=0).stop 2024-04-12T05:27:19.187 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.509+0000 7f3be41c8700 1 -- 172.21.15.32:0/3738743734 shutdown_connections 2024-04-12T05:27:19.187 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.509+0000 7f3be41c8700 1 --2- 172.21.15.32:0/3738743734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bdc102580 0x7f3bdc078c50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:19.187 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.509+0000 7f3be41c8700 1 -- 172.21.15.32:0/3738743734 >> 172.21.15.32:0/3738743734 conn(0x7f3bdc0fdd00 msgr2=0x7f3bdc0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:19.187 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.509+0000 7f3be41c8700 1 -- 172.21.15.32:0/3738743734 shutdown_connections 2024-04-12T05:27:19.187 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:18.509+0000 7f3be41c8700 1 -- 172.21.15.32:0/3738743734 wait complete. 2024-04-12T05:27:19.187 INFO:teuthology.orchestra.run.smithi032.stdout:mgr not available, waiting (1/15)... 2024-04-12T05:27:23.981 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:23 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/3010639972' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-12T05:27:25.094 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:24 smithi032 ceph-mon[39731]: Activating manager daemon smithi032.hrwxrh 2024-04-12T05:27:25.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:24 smithi032 ceph-mon[39731]: mgrmap e2: smithi032.hrwxrh(active, starting, since 0.00281311s) 2024-04-12T05:27:25.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:24 smithi032 ceph-mon[39731]: from='mgr.14100 172.21.15.32:0/2050488731' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T05:27:25.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:24 smithi032 ceph-mon[39731]: from='mgr.14100 172.21.15.32:0/2050488731' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T05:27:25.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:24 smithi032 ceph-mon[39731]: from='mgr.14100 172.21.15.32:0/2050488731' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T05:27:25.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:24 smithi032 ceph-mon[39731]: from='mgr.14100 172.21.15.32:0/2050488731' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-12T05:27:25.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:24 smithi032 ceph-mon[39731]: from='mgr.14100 172.21.15.32:0/2050488731' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr metadata", "who": "smithi032.hrwxrh", "id": "smithi032.hrwxrh"}]: dispatch 2024-04-12T05:27:25.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:24 smithi032 ceph-mon[39731]: Manager daemon smithi032.hrwxrh is now available 2024-04-12T05:27:25.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:24 smithi032 ceph-mon[39731]: from='mgr.14100 172.21.15.32:0/2050488731' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.hrwxrh/mirror_snapshot_schedule"}]: dispatch 2024-04-12T05:27:25.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:24 smithi032 ceph-mon[39731]: from='mgr.14100 172.21.15.32:0/2050488731' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.hrwxrh/trash_purge_schedule"}]: dispatch 2024-04-12T05:27:25.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:24 smithi032 ceph-mon[39731]: from='mgr.14100 172.21.15.32:0/2050488731' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:27:25.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:24 smithi032 ceph-mon[39731]: from='mgr.14100 172.21.15.32:0/2050488731' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:27:25.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:24 smithi032 ceph-mon[39731]: from='mgr.14100 172.21.15.32:0/2050488731' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:27:26.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:26 smithi032 ceph-mon[39731]: mgrmap e3: smithi032.hrwxrh(active, since 1.00653s) 2024-04-12T05:27:28.593 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-12T05:27:28.593 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-12T05:27:28.594 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsid": "37b58ce2-f88d-11ee-bc8c-c7b262605968", 2024-04-12T05:27:28.594 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "health": { 2024-04-12T05:27:28.594 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-12T05:27:28.594 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-12T05:27:28.594 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-12T05:27:28.594 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-12T05:27:28.594 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-12T05:27:28.594 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-12T05:27:28.594 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 0 2024-04-12T05:27:28.594 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-12T05:27:28.594 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-12T05:27:28.594 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "smithi032" 2024-04-12T05:27:28.594 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-12T05:27:28.594 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-04-12T05:27:28.595 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-12T05:27:28.595 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-12T05:27:28.595 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-04-12T05:27:28.595 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-12T05:27:28.595 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-12T05:27:28.595 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-12T05:27:28.595 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-12T05:27:28.595 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-12T05:27:28.595 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-12T05:27:28.595 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-12T05:27:28.602 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-12T05:27:28.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-12T05:27:28.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-12T05:27:28.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-12T05:27:28.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-12T05:27:28.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-12T05:27:28.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-12T05:27:28.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-12T05:27:28.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-12T05:27:28.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-12T05:27:28.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-12T05:27:28.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-12T05:27:28.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-12T05:27:28.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-12T05:27:28.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-12T05:27:28.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-12T05:27:28.604 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-12T05:27:28.604 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-12T05:27:28.604 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-12T05:27:28.604 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-12T05:27:28.604 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-12T05:27:28.604 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-12T05:27:28.604 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-12T05:27:28.604 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-12T05:27:28.604 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-12T05:27:28.604 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "restful" 2024-04-12T05:27:28.604 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-12T05:27:28.604 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-12T05:27:28.604 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-12T05:27:28.604 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-12T05:27:28.604 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-12T05:27:28.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modified": "2024-04-12T05:27:07.779861+0000", 2024-04-12T05:27:28.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-12T05:27:28.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-12T05:27:28.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-12T05:27:28.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-12T05:27:28.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.541+0000 7fe7d14f3700 1 Processor -- start 2024-04-12T05:27:28.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.541+0000 7fe7d14f3700 1 -- start start 2024-04-12T05:27:28.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.542+0000 7fe7d14f3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7cc076c80 0x7fe7cc1051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:28.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.542+0000 7fe7d14f3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7cc1056f0 con 0x7fe7cc076c80 2024-04-12T05:27:28.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.542+0000 7fe7caffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7cc076c80 0x7fe7cc1051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:28.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.542+0000 7fe7caffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7cc076c80 0x7fe7cc1051b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54054/0 (socket says 172.21.15.32:54054) 2024-04-12T05:27:28.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.542+0000 7fe7caffd700 1 -- 172.21.15.32:0/2638367198 learned_addr learned my addr 172.21.15.32:0/2638367198 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:28.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.542+0000 7fe7caffd700 1 -- 172.21.15.32:0/2638367198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7cc105830 con 0x7fe7cc076c80 2024-04-12T05:27:28.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.542+0000 7fe7caffd700 1 --2- 172.21.15.32:0/2638367198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7cc076c80 0x7fe7cc1051b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc0098e0 tx=0x7fe7bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14472fee4a1bc04 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:28.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.542+0000 7fe7c9ffb700 1 -- 172.21.15.32:0/2638367198 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7bc00b860 con 0x7fe7cc076c80 2024-04-12T05:27:28.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.543+0000 7fe7c9ffb700 1 -- 172.21.15.32:0/2638367198 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe7bc00be30 con 0x7fe7cc076c80 2024-04-12T05:27:28.606 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.543+0000 7fe7c9ffb700 1 -- 172.21.15.32:0/2638367198 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7bc010380 con 0x7fe7cc076c80 2024-04-12T05:27:28.606 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.543+0000 7fe7d14f3700 1 -- 172.21.15.32:0/2638367198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7cc076c80 msgr2=0x7fe7cc1051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:28.606 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.543+0000 7fe7d14f3700 1 --2- 172.21.15.32:0/2638367198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7cc076c80 0x7fe7cc1051b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc0098e0 tx=0x7fe7bc009250 comp rx=0 tx=0).stop 2024-04-12T05:27:28.606 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.543+0000 7fe7d14f3700 1 -- 172.21.15.32:0/2638367198 shutdown_connections 2024-04-12T05:27:28.606 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.543+0000 7fe7d14f3700 1 --2- 172.21.15.32:0/2638367198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7cc076c80 0x7fe7cc1051b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:28.606 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.543+0000 7fe7d14f3700 1 -- 172.21.15.32:0/2638367198 >> 172.21.15.32:0/2638367198 conn(0x7fe7cc0fcc60 msgr2=0x7fe7cc0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:28.606 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.543+0000 7fe7d14f3700 1 -- 172.21.15.32:0/2638367198 shutdown_connections 2024-04-12T05:27:28.606 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.543+0000 7fe7d14f3700 1 -- 172.21.15.32:0/2638367198 wait complete. 2024-04-12T05:27:28.606 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.544+0000 7fe7d14f3700 1 Processor -- start 2024-04-12T05:27:28.606 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.544+0000 7fe7d14f3700 1 -- start start 2024-04-12T05:27:28.606 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.544+0000 7fe7d14f3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7cc076c80 0x7fe7cc111370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:28.606 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.544+0000 7fe7d14f3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7cc1118b0 con 0x7fe7cc076c80 2024-04-12T05:27:28.606 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.544+0000 7fe7caffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7cc076c80 0x7fe7cc111370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:28.606 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.544+0000 7fe7caffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7cc076c80 0x7fe7cc111370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54070/0 (socket says 172.21.15.32:54070) 2024-04-12T05:27:28.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.544+0000 7fe7caffd700 1 -- 172.21.15.32:0/3010639972 learned_addr learned my addr 172.21.15.32:0/3010639972 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:28.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.544+0000 7fe7caffd700 1 -- 172.21.15.32:0/3010639972 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7bc0094f0 con 0x7fe7cc076c80 2024-04-12T05:27:28.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.545+0000 7fe7caffd700 1 --2- 172.21.15.32:0/3010639972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7cc076c80 0x7fe7cc111370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc00ba90 tx=0x7fe7bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:28.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.545+0000 7fe7b37fe700 1 -- 172.21.15.32:0/3010639972 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7bc010ad0 con 0x7fe7cc076c80 2024-04-12T05:27:28.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.545+0000 7fe7d14f3700 1 -- 172.21.15.32:0/3010639972 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7cc111ab0 con 0x7fe7cc076c80 2024-04-12T05:27:28.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.545+0000 7fe7b37fe700 1 -- 172.21.15.32:0/3010639972 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe7bc010c30 con 0x7fe7cc076c80 2024-04-12T05:27:28.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.545+0000 7fe7b37fe700 1 -- 172.21.15.32:0/3010639972 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7bc01a430 con 0x7fe7cc076c80 2024-04-12T05:27:28.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.545+0000 7fe7d14f3700 1 -- 172.21.15.32:0/3010639972 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7cc112620 con 0x7fe7cc076c80 2024-04-12T05:27:28.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.545+0000 7fe7b37fe700 1 -- 172.21.15.32:0/3010639972 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fe7bc01a970 con 0x7fe7cc076c80 2024-04-12T05:27:28.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.545+0000 7fe7b37fe700 1 -- 172.21.15.32:0/3010639972 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fe7bc02db80 con 0x7fe7cc076c80 2024-04-12T05:27:28.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.546+0000 7fe7d14f3700 1 -- 172.21.15.32:0/3010639972 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7cc05f5f0 con 0x7fe7cc076c80 2024-04-12T05:27:28.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.548+0000 7fe7b37fe700 1 -- 172.21.15.32:0/3010639972 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fe7bc029070 con 0x7fe7cc076c80 2024-04-12T05:27:28.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.621+0000 7fe7d14f3700 1 -- 172.21.15.32:0/3010639972 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe7cc112860 con 0x7fe7cc076c80 2024-04-12T05:27:28.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.621+0000 7fe7b37fe700 1 -- 172.21.15.32:0/3010639972 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fe7bc010da0 con 0x7fe7cc076c80 2024-04-12T05:27:28.608 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.622+0000 7fe7d14f3700 1 -- 172.21.15.32:0/3010639972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7cc076c80 msgr2=0x7fe7cc111370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:28.608 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.622+0000 7fe7d14f3700 1 --2- 172.21.15.32:0/3010639972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7cc076c80 0x7fe7cc111370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc00ba90 tx=0x7fe7bc009250 comp rx=0 tx=0).stop 2024-04-12T05:27:28.608 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.622+0000 7fe7d14f3700 1 -- 172.21.15.32:0/3010639972 shutdown_connections 2024-04-12T05:27:28.608 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.622+0000 7fe7d14f3700 1 --2- 172.21.15.32:0/3010639972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7cc076c80 0x7fe7cc111370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:28.608 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.622+0000 7fe7d14f3700 1 -- 172.21.15.32:0/3010639972 >> 172.21.15.32:0/3010639972 conn(0x7fe7cc0fcc60 msgr2=0x7fe7cc0fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:28.608 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.622+0000 7fe7d14f3700 1 -- 172.21.15.32:0/3010639972 shutdown_connections 2024-04-12T05:27:28.608 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:23.622+0000 7fe7d14f3700 1 -- 172.21.15.32:0/3010639972 wait complete. 2024-04-12T05:27:28.608 INFO:teuthology.orchestra.run.smithi032.stdout:mgr not available, waiting (2/15)... 2024-04-12T05:27:28.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:28 smithi032 ceph-mon[39731]: mgrmap e4: smithi032.hrwxrh(active, since 3s) 2024-04-12T05:27:31.824 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:31 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/3096428892' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-12T05:27:32.088 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-12T05:27:32.088 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-12T05:27:32.088 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsid": "37b58ce2-f88d-11ee-bc8c-c7b262605968", 2024-04-12T05:27:32.088 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "health": { 2024-04-12T05:27:32.088 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-12T05:27:32.089 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-12T05:27:32.089 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-12T05:27:32.089 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-12T05:27:32.089 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-12T05:27:32.089 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-12T05:27:32.089 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 0 2024-04-12T05:27:32.089 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-12T05:27:32.089 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-12T05:27:32.089 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "smithi032" 2024-04-12T05:27:32.089 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-12T05:27:32.089 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_age": 15, 2024-04-12T05:27:32.089 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-12T05:27:32.089 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-12T05:27:32.089 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-04-12T05:27:32.090 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-12T05:27:32.090 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-12T05:27:32.090 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-12T05:27:32.090 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-12T05:27:32.090 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-12T05:27:32.090 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-12T05:27:32.090 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-12T05:27:32.090 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-12T05:27:32.090 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-12T05:27:32.090 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-12T05:27:32.090 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-12T05:27:32.090 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-12T05:27:32.090 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-12T05:27:32.090 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-12T05:27:32.091 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-12T05:27:32.091 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-12T05:27:32.091 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-12T05:27:32.091 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-12T05:27:32.091 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-12T05:27:32.091 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-12T05:27:32.091 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-12T05:27:32.091 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-12T05:27:32.091 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-12T05:27:32.091 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-12T05:27:32.091 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-12T05:27:32.091 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-12T05:27:32.091 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-12T05:27:32.091 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-12T05:27:32.091 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-12T05:27:32.092 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-12T05:27:32.092 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-12T05:27:32.092 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-12T05:27:32.092 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "restful" 2024-04-12T05:27:32.092 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-12T05:27:32.092 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-12T05:27:32.092 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-12T05:27:32.092 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-12T05:27:32.092 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-12T05:27:32.092 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modified": "2024-04-12T05:27:07.779861+0000", 2024-04-12T05:27:32.093 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-12T05:27:32.093 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-12T05:27:32.093 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-12T05:27:32.093 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-12T05:27:32.093 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.220+0000 7f86d4e74700 1 Processor -- start 2024-04-12T05:27:32.093 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.220+0000 7f86d4e74700 1 -- start start 2024-04-12T05:27:32.093 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.221+0000 7f86d4e74700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d01009b0 0x7f86d0100da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:32.093 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.221+0000 7f86d4e74700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d01012e0 con 0x7f86d01009b0 2024-04-12T05:27:32.093 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.221+0000 7f86ce59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d01009b0 0x7f86d0100da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:32.093 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.221+0000 7f86ce59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d01009b0 0x7f86d0100da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54138/0 (socket says 172.21.15.32:54138) 2024-04-12T05:27:32.093 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.221+0000 7f86ce59c700 1 -- 172.21.15.32:0/4104139151 learned_addr learned my addr 172.21.15.32:0/4104139151 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:32.093 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.221+0000 7f86ce59c700 1 -- 172.21.15.32:0/4104139151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d0101420 con 0x7f86d01009b0 2024-04-12T05:27:32.093 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.221+0000 7f86ce59c700 1 --2- 172.21.15.32:0/4104139151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d01009b0 0x7f86d0100da0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f86c00098e0 tx=0x7f86c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f2f6fb249e10fed server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:32.093 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.222+0000 7f86cd59a700 1 -- 172.21.15.32:0/4104139151 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86c000b860 con 0x7f86d01009b0 2024-04-12T05:27:32.094 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.222+0000 7f86cd59a700 1 -- 172.21.15.32:0/4104139151 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f86c000be30 con 0x7f86d01009b0 2024-04-12T05:27:32.094 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.222+0000 7f86cd59a700 1 -- 172.21.15.32:0/4104139151 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86c0010380 con 0x7f86d01009b0 2024-04-12T05:27:32.094 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.222+0000 7f86d4e74700 1 -- 172.21.15.32:0/4104139151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d01009b0 msgr2=0x7f86d0100da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:32.094 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.222+0000 7f86d4e74700 1 --2- 172.21.15.32:0/4104139151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d01009b0 0x7f86d0100da0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f86c00098e0 tx=0x7f86c0009250 comp rx=0 tx=0).stop 2024-04-12T05:27:32.094 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.222+0000 7f86d4e74700 1 -- 172.21.15.32:0/4104139151 shutdown_connections 2024-04-12T05:27:32.094 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.222+0000 7f86d4e74700 1 --2- 172.21.15.32:0/4104139151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d01009b0 0x7f86d0100da0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:32.094 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.222+0000 7f86d4e74700 1 -- 172.21.15.32:0/4104139151 >> 172.21.15.32:0/4104139151 conn(0x7f86d00fc3d0 msgr2=0x7f86d00fe800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:32.094 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.222+0000 7f86d4e74700 1 -- 172.21.15.32:0/4104139151 shutdown_connections 2024-04-12T05:27:32.094 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.222+0000 7f86d4e74700 1 -- 172.21.15.32:0/4104139151 wait complete. 2024-04-12T05:27:32.094 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.223+0000 7f86d4e74700 1 Processor -- start 2024-04-12T05:27:32.094 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.223+0000 7f86d4e74700 1 -- start start 2024-04-12T05:27:32.094 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.223+0000 7f86d4e74700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d01009b0 0x7f86d01113c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:32.094 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.223+0000 7f86d4e74700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d0111900 con 0x7f86d01009b0 2024-04-12T05:27:32.095 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.223+0000 7f86ce59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d01009b0 0x7f86d01113c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:32.095 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.223+0000 7f86ce59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d01009b0 0x7f86d01113c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54144/0 (socket says 172.21.15.32:54144) 2024-04-12T05:27:32.095 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.223+0000 7f86ce59c700 1 -- 172.21.15.32:0/3096428892 learned_addr learned my addr 172.21.15.32:0/3096428892 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:32.095 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.224+0000 7f86ce59c700 1 -- 172.21.15.32:0/3096428892 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c00094f0 con 0x7f86d01009b0 2024-04-12T05:27:32.095 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.224+0000 7f86ce59c700 1 --2- 172.21.15.32:0/3096428892 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d01009b0 0x7f86d01113c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f86c0006b00 tx=0x7f86c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:32.095 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.224+0000 7f86beffd700 1 -- 172.21.15.32:0/3096428892 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86c00109b0 con 0x7f86d01009b0 2024-04-12T05:27:32.095 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.224+0000 7f86beffd700 1 -- 172.21.15.32:0/3096428892 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f86c001a430 con 0x7f86d01009b0 2024-04-12T05:27:32.095 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.224+0000 7f86d4e74700 1 -- 172.21.15.32:0/3096428892 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86d0111b00 con 0x7f86d01009b0 2024-04-12T05:27:32.095 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.224+0000 7f86beffd700 1 -- 172.21.15.32:0/3096428892 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86c00245b0 con 0x7f86d01009b0 2024-04-12T05:27:32.095 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.224+0000 7f86d4e74700 1 -- 172.21.15.32:0/3096428892 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d0112670 con 0x7f86d01009b0 2024-04-12T05:27:32.095 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.225+0000 7f86beffd700 1 -- 172.21.15.32:0/3096428892 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f86c00247b0 con 0x7f86d01009b0 2024-04-12T05:27:32.095 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.225+0000 7f86beffd700 1 --2- 172.21.15.32:0/3096428892 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7f86b8036740 0x7f86b8038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:32.095 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.225+0000 7f86bcff9700 1 -- 172.21.15.32:0/3096428892 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86b40052b0 con 0x7f86d01009b0 2024-04-12T05:27:32.095 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.225+0000 7f86beffd700 1 -- 172.21.15.32:0/3096428892 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f86c004a910 con 0x7f86d01009b0 2024-04-12T05:27:32.096 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.225+0000 7f86cdd9b700 1 --2- 172.21.15.32:0/3096428892 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7f86b8036740 0x7f86b8038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:32.096 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.226+0000 7f86cdd9b700 1 --2- 172.21.15.32:0/3096428892 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7f86b8036740 0x7f86b8038bf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f86c4009990 tx=0x7f86c4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:32.096 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.230+0000 7f86beffd700 1 -- 172.21.15.32:0/3096428892 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f86c002e440 con 0x7f86d01009b0 2024-04-12T05:27:32.096 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.503+0000 7f86bcff9700 1 -- 172.21.15.32:0/3096428892 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f86b4005560 con 0x7f86d01009b0 2024-04-12T05:27:32.096 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.504+0000 7f86beffd700 1 -- 172.21.15.32:0/3096428892 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f86c00260c0 con 0x7f86d01009b0 2024-04-12T05:27:32.096 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.506+0000 7f86bcff9700 1 -- 172.21.15.32:0/3096428892 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7f86b8036740 msgr2=0x7f86b8038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:32.096 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.506+0000 7f86bcff9700 1 --2- 172.21.15.32:0/3096428892 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7f86b8036740 0x7f86b8038bf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f86c4009990 tx=0x7f86c4006e40 comp rx=0 tx=0).stop 2024-04-12T05:27:32.096 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.506+0000 7f86bcff9700 1 -- 172.21.15.32:0/3096428892 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d01009b0 msgr2=0x7f86d01113c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:32.096 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.506+0000 7f86bcff9700 1 --2- 172.21.15.32:0/3096428892 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d01009b0 0x7f86d01113c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f86c0006b00 tx=0x7f86c0009250 comp rx=0 tx=0).stop 2024-04-12T05:27:32.096 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.506+0000 7f86bcff9700 1 -- 172.21.15.32:0/3096428892 shutdown_connections 2024-04-12T05:27:32.096 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.506+0000 7f86bcff9700 1 --2- 172.21.15.32:0/3096428892 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7f86b8036740 0x7f86b8038bf0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:32.096 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.506+0000 7f86bcff9700 1 --2- 172.21.15.32:0/3096428892 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d01009b0 0x7f86d01113c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:32.096 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.506+0000 7f86bcff9700 1 -- 172.21.15.32:0/3096428892 >> 172.21.15.32:0/3096428892 conn(0x7f86d00fc3d0 msgr2=0x7f86d00fcfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:32.096 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.506+0000 7f86bcff9700 1 -- 172.21.15.32:0/3096428892 shutdown_connections 2024-04-12T05:27:32.096 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:31.506+0000 7f86bcff9700 1 -- 172.21.15.32:0/3096428892 wait complete. 2024-04-12T05:27:32.097 INFO:teuthology.orchestra.run.smithi032.stdout:mgr is available 2024-04-12T05:27:33.124 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:32 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/2565139650' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-12T05:27:33.375 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-12T05:27:33.375 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [global] 2024-04-12T05:27:33.375 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout fsid = 37b58ce2-f88d-11ee-bc8c-c7b262605968 2024-04-12T05:27:33.375 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-12T05:27:33.375 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-12T05:27:33.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-12T05:27:33.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-12T05:27:33.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-12T05:27:33.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-12T05:27:33.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-12T05:27:33.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-12T05:27:33.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [osd] 2024-04-12T05:27:33.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-12T05:27:33.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-12T05:27:33.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-12T05:27:33.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-12T05:27:33.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-12T05:27:33.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.610+0000 7fe804277700 1 Processor -- start 2024-04-12T05:27:33.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.610+0000 7fe804277700 1 -- start start 2024-04-12T05:27:33.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.610+0000 7fe804277700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7fc076c80 0x7fe7fc1051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:33.377 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.610+0000 7fe804277700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc1056f0 con 0x7fe7fc076c80 2024-04-12T05:27:33.377 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.610+0000 7fe802013700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7fc076c80 0x7fe7fc1051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:33.377 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.610+0000 7fe802013700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7fc076c80 0x7fe7fc1051b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35250/0 (socket says 172.21.15.32:35250) 2024-04-12T05:27:33.377 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.610+0000 7fe802013700 1 -- 172.21.15.32:0/4021413574 learned_addr learned my addr 172.21.15.32:0/4021413574 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:33.377 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.610+0000 7fe802013700 1 -- 172.21.15.32:0/4021413574 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7fc105830 con 0x7fe7fc076c80 2024-04-12T05:27:33.377 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.611+0000 7fe802013700 1 --2- 172.21.15.32:0/4021413574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7fc076c80 0x7fe7fc1051b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe7f80098e0 tx=0x7fe7f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ef53fd2728d8bc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:33.377 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.611+0000 7fe801011700 1 -- 172.21.15.32:0/4021413574 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7f800b860 con 0x7fe7fc076c80 2024-04-12T05:27:33.377 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.611+0000 7fe801011700 1 -- 172.21.15.32:0/4021413574 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe7f800be30 con 0x7fe7fc076c80 2024-04-12T05:27:33.377 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.611+0000 7fe801011700 1 -- 172.21.15.32:0/4021413574 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7f8010380 con 0x7fe7fc076c80 2024-04-12T05:27:33.377 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.611+0000 7fe804277700 1 -- 172.21.15.32:0/4021413574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7fc076c80 msgr2=0x7fe7fc1051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:33.377 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.611+0000 7fe804277700 1 --2- 172.21.15.32:0/4021413574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7fc076c80 0x7fe7fc1051b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe7f80098e0 tx=0x7fe7f8009250 comp rx=0 tx=0).stop 2024-04-12T05:27:33.378 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.611+0000 7fe804277700 1 -- 172.21.15.32:0/4021413574 shutdown_connections 2024-04-12T05:27:33.378 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.611+0000 7fe804277700 1 --2- 172.21.15.32:0/4021413574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7fc076c80 0x7fe7fc1051b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:33.378 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.611+0000 7fe804277700 1 -- 172.21.15.32:0/4021413574 >> 172.21.15.32:0/4021413574 conn(0x7fe7fc0fcc60 msgr2=0x7fe7fc0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:33.378 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.611+0000 7fe804277700 1 -- 172.21.15.32:0/4021413574 shutdown_connections 2024-04-12T05:27:33.378 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.611+0000 7fe804277700 1 -- 172.21.15.32:0/4021413574 wait complete. 2024-04-12T05:27:33.378 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.612+0000 7fe804277700 1 Processor -- start 2024-04-12T05:27:33.378 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.612+0000 7fe804277700 1 -- start start 2024-04-12T05:27:33.378 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.612+0000 7fe804277700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7fc076c80 0x7fe7fc102b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:33.378 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.612+0000 7fe804277700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc103040 con 0x7fe7fc076c80 2024-04-12T05:27:33.378 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.612+0000 7fe802013700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7fc076c80 0x7fe7fc102b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:33.378 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.612+0000 7fe802013700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7fc076c80 0x7fe7fc102b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35258/0 (socket says 172.21.15.32:35258) 2024-04-12T05:27:33.378 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.612+0000 7fe802013700 1 -- 172.21.15.32:0/2565139650 learned_addr learned my addr 172.21.15.32:0/2565139650 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:33.378 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.612+0000 7fe802013700 1 -- 172.21.15.32:0/2565139650 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f80094f0 con 0x7fe7fc076c80 2024-04-12T05:27:33.378 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.613+0000 7fe802013700 1 --2- 172.21.15.32:0/2565139650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7fc076c80 0x7fe7fc102b00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe7f8006b00 tx=0x7fe7f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:33.379 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.613+0000 7fe7f2ffd700 1 -- 172.21.15.32:0/2565139650 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7f8010a80 con 0x7fe7fc076c80 2024-04-12T05:27:33.379 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.613+0000 7fe7f2ffd700 1 -- 172.21.15.32:0/2565139650 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe7f801a430 con 0x7fe7fc076c80 2024-04-12T05:27:33.379 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.613+0000 7fe804277700 1 -- 172.21.15.32:0/2565139650 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7fc1010b0 con 0x7fe7fc076c80 2024-04-12T05:27:33.379 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.613+0000 7fe7f2ffd700 1 -- 172.21.15.32:0/2565139650 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7f80245b0 con 0x7fe7fc076c80 2024-04-12T05:27:33.379 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.613+0000 7fe804277700 1 -- 172.21.15.32:0/2565139650 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7fc1015a0 con 0x7fe7fc076c80 2024-04-12T05:27:33.379 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.614+0000 7fe7f2ffd700 1 -- 172.21.15.32:0/2565139650 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7fe7f8010be0 con 0x7fe7fc076c80 2024-04-12T05:27:33.379 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.614+0000 7fe7f2ffd700 1 --2- 172.21.15.32:0/2565139650 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7fe7e8036790 0x7fe7e8038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:33.379 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.614+0000 7fe804277700 1 -- 172.21.15.32:0/2565139650 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7fc04bed0 con 0x7fe7fc076c80 2024-04-12T05:27:33.379 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.614+0000 7fe7f2ffd700 1 -- 172.21.15.32:0/2565139650 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fe7f804ac10 con 0x7fe7fc076c80 2024-04-12T05:27:33.379 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.614+0000 7fe801812700 1 --2- 172.21.15.32:0/2565139650 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7fe7e8036790 0x7fe7e8038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:33.379 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.614+0000 7fe801812700 1 --2- 172.21.15.32:0/2565139650 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7fe7e8036790 0x7fe7e8038c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec009990 tx=0x7fe7ec006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:33.379 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.619+0000 7fe7f2ffd700 1 -- 172.21.15.32:0/2565139650 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fe7f80239e0 con 0x7fe7fc076c80 2024-04-12T05:27:33.379 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.812+0000 7fe804277700 1 -- 172.21.15.32:0/2565139650 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fe7fc101a20 con 0x7fe7fc076c80 2024-04-12T05:27:33.379 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.813+0000 7fe7f2ffd700 1 -- 172.21.15.32:0/2565139650 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fe7f802c390 con 0x7fe7fc076c80 2024-04-12T05:27:33.379 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.815+0000 7fe804277700 1 -- 172.21.15.32:0/2565139650 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7fe7e8036790 msgr2=0x7fe7e8038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:33.380 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.815+0000 7fe804277700 1 --2- 172.21.15.32:0/2565139650 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7fe7e8036790 0x7fe7e8038c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec009990 tx=0x7fe7ec006e40 comp rx=0 tx=0).stop 2024-04-12T05:27:33.380 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.815+0000 7fe804277700 1 -- 172.21.15.32:0/2565139650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7fc076c80 msgr2=0x7fe7fc102b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:33.380 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.815+0000 7fe804277700 1 --2- 172.21.15.32:0/2565139650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7fc076c80 0x7fe7fc102b00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe7f8006b00 tx=0x7fe7f8009250 comp rx=0 tx=0).stop 2024-04-12T05:27:33.380 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.815+0000 7fe804277700 1 -- 172.21.15.32:0/2565139650 shutdown_connections 2024-04-12T05:27:33.380 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.815+0000 7fe804277700 1 --2- 172.21.15.32:0/2565139650 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7fe7e8036790 0x7fe7e8038c40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:33.380 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.815+0000 7fe804277700 1 --2- 172.21.15.32:0/2565139650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7fc076c80 0x7fe7fc102b00 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:33.380 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.815+0000 7fe804277700 1 -- 172.21.15.32:0/2565139650 >> 172.21.15.32:0/2565139650 conn(0x7fe7fc0fcc60 msgr2=0x7fe7fc0fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:33.380 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.815+0000 7fe804277700 1 -- 172.21.15.32:0/2565139650 shutdown_connections 2024-04-12T05:27:33.380 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:32.815+0000 7fe804277700 1 -- 172.21.15.32:0/2565139650 wait complete. 2024-04-12T05:27:33.380 INFO:teuthology.orchestra.run.smithi032.stdout:Enabling cephadm module... 2024-04-12T05:27:34.562 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:34 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/1649215575' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-12T05:27:34.805 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.903+0000 7f62ec9a0700 1 Processor -- start 2024-04-12T05:27:34.805 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.903+0000 7f62ec9a0700 1 -- start start 2024-04-12T05:27:34.805 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.903+0000 7f62ec9a0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62e8076cd0 0x7f62e8104fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:34.805 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.903+0000 7f62ec9a0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e81054f0 con 0x7f62e8076cd0 2024-04-12T05:27:34.805 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.904+0000 7f62e77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62e8076cd0 0x7f62e8104fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:34.805 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.904+0000 7f62e77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62e8076cd0 0x7f62e8104fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35264/0 (socket says 172.21.15.32:35264) 2024-04-12T05:27:34.805 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.904+0000 7f62e77fe700 1 -- 172.21.15.32:0/3628809224 learned_addr learned my addr 172.21.15.32:0/3628809224 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:34.805 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.904+0000 7f62e77fe700 1 -- 172.21.15.32:0/3628809224 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e8105630 con 0x7f62e8076cd0 2024-04-12T05:27:34.805 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.904+0000 7f62e77fe700 1 --2- 172.21.15.32:0/3628809224 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62e8076cd0 0x7f62e8104fb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f62d80098e0 tx=0x7f62d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ca063e0b97b1c2f server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:34.805 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.904+0000 7f62e67fc700 1 -- 172.21.15.32:0/3628809224 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62d800b860 con 0x7f62e8076cd0 2024-04-12T05:27:34.805 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.904+0000 7f62e67fc700 1 -- 172.21.15.32:0/3628809224 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f62d800be30 con 0x7f62e8076cd0 2024-04-12T05:27:34.806 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.904+0000 7f62e67fc700 1 -- 172.21.15.32:0/3628809224 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62d8010380 con 0x7f62e8076cd0 2024-04-12T05:27:34.806 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.905+0000 7f62ec9a0700 1 -- 172.21.15.32:0/3628809224 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62e8076cd0 msgr2=0x7f62e8104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:34.806 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.905+0000 7f62ec9a0700 1 --2- 172.21.15.32:0/3628809224 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62e8076cd0 0x7f62e8104fb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f62d80098e0 tx=0x7f62d8009250 comp rx=0 tx=0).stop 2024-04-12T05:27:34.806 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.905+0000 7f62ec9a0700 1 -- 172.21.15.32:0/3628809224 shutdown_connections 2024-04-12T05:27:34.806 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.905+0000 7f62ec9a0700 1 --2- 172.21.15.32:0/3628809224 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62e8076cd0 0x7f62e8104fb0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:34.806 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.905+0000 7f62ec9a0700 1 -- 172.21.15.32:0/3628809224 >> 172.21.15.32:0/3628809224 conn(0x7f62e80fca60 msgr2=0x7f62e80fee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:34.806 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.905+0000 7f62ec9a0700 1 -- 172.21.15.32:0/3628809224 shutdown_connections 2024-04-12T05:27:34.806 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.905+0000 7f62ec9a0700 1 -- 172.21.15.32:0/3628809224 wait complete. 2024-04-12T05:27:34.806 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.905+0000 7f62ec9a0700 1 Processor -- start 2024-04-12T05:27:34.806 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.906+0000 7f62ec9a0700 1 -- start start 2024-04-12T05:27:34.806 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.906+0000 7f62ec9a0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62e8076cd0 0x7f62e81111e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:34.807 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.906+0000 7f62ec9a0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e8111720 con 0x7f62e8076cd0 2024-04-12T05:27:34.807 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.906+0000 7f62e77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62e8076cd0 0x7f62e81111e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:34.807 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.906+0000 7f62e77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62e8076cd0 0x7f62e81111e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35266/0 (socket says 172.21.15.32:35266) 2024-04-12T05:27:34.807 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.906+0000 7f62e77fe700 1 -- 172.21.15.32:0/1649215575 learned_addr learned my addr 172.21.15.32:0/1649215575 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:34.807 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.906+0000 7f62e77fe700 1 -- 172.21.15.32:0/1649215575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d80094f0 con 0x7f62e8076cd0 2024-04-12T05:27:34.807 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.906+0000 7f62e77fe700 1 --2- 172.21.15.32:0/1649215575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62e8076cd0 0x7f62e81111e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f62d800b440 tx=0x7f62d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:34.807 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.906+0000 7f62dffff700 1 -- 172.21.15.32:0/1649215575 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62d8010960 con 0x7f62e8076cd0 2024-04-12T05:27:34.807 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.906+0000 7f62ec9a0700 1 -- 172.21.15.32:0/1649215575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62e8111920 con 0x7f62e8076cd0 2024-04-12T05:27:34.807 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.906+0000 7f62dffff700 1 -- 172.21.15.32:0/1649215575 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f62d801a430 con 0x7f62e8076cd0 2024-04-12T05:27:34.807 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.906+0000 7f62dffff700 1 -- 172.21.15.32:0/1649215575 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62d80245b0 con 0x7f62e8076cd0 2024-04-12T05:27:34.807 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.907+0000 7f62ec9a0700 1 -- 172.21.15.32:0/1649215575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62e8112490 con 0x7f62e8076cd0 2024-04-12T05:27:34.807 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.907+0000 7f62dffff700 1 -- 172.21.15.32:0/1649215575 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f62d80247a0 con 0x7f62e8076cd0 2024-04-12T05:27:34.807 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.907+0000 7f62dffff700 1 --2- 172.21.15.32:0/1649215575 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7f62d4036790 0x7f62d4038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:34.807 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.907+0000 7f62dffff700 1 -- 172.21.15.32:0/1649215575 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f62d804a660 con 0x7f62e8076cd0 2024-04-12T05:27:34.808 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.908+0000 7f62e6ffd700 1 --2- 172.21.15.32:0/1649215575 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7f62d4036790 0x7f62d4038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:34.808 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.908+0000 7f62ec9a0700 1 -- 172.21.15.32:0/1649215575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62e805f5f0 con 0x7f62e8076cd0 2024-04-12T05:27:34.808 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.908+0000 7f62e6ffd700 1 --2- 172.21.15.32:0/1649215575 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7f62d4036790 0x7f62d4038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f62d0009990 tx=0x7f62d0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:34.808 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:33.912+0000 7f62dffff700 1 -- 172.21.15.32:0/1649215575 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f62d802e440 con 0x7f62e8076cd0 2024-04-12T05:27:34.808 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:34.167+0000 7f62ec9a0700 1 -- 172.21.15.32:0/1649215575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f62e8112700 con 0x7f62e8076cd0 2024-04-12T05:27:34.808 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:34.219+0000 7f62dffff700 1 -- 172.21.15.32:0/1649215575 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f62d8021d20 con 0x7f62e8076cd0 2024-04-12T05:27:34.808 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:34.221+0000 7f62dffff700 1 -- 172.21.15.32:0/1649215575 <== mon.0 v2:172.21.15.32:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f62d8018350 con 0x7f62e8076cd0 2024-04-12T05:27:34.808 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:34.223+0000 7f62ec9a0700 1 -- 172.21.15.32:0/1649215575 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7f62d4036790 msgr2=0x7f62d4038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:34.808 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:34.223+0000 7f62ec9a0700 1 --2- 172.21.15.32:0/1649215575 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7f62d4036790 0x7f62d4038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f62d0009990 tx=0x7f62d0006e40 comp rx=0 tx=0).stop 2024-04-12T05:27:34.808 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:34.223+0000 7f62ec9a0700 1 -- 172.21.15.32:0/1649215575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62e8076cd0 msgr2=0x7f62e81111e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:34.808 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:34.223+0000 7f62ec9a0700 1 --2- 172.21.15.32:0/1649215575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62e8076cd0 0x7f62e81111e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f62d800b440 tx=0x7f62d8009250 comp rx=0 tx=0).stop 2024-04-12T05:27:34.808 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:34.223+0000 7f62ec9a0700 1 -- 172.21.15.32:0/1649215575 shutdown_connections 2024-04-12T05:27:34.808 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:34.223+0000 7f62ec9a0700 1 --2- 172.21.15.32:0/1649215575 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7f62d4036790 0x7f62d4038c40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:34.808 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:34.223+0000 7f62ec9a0700 1 --2- 172.21.15.32:0/1649215575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62e8076cd0 0x7f62e81111e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:34.809 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:34.223+0000 7f62ec9a0700 1 -- 172.21.15.32:0/1649215575 >> 172.21.15.32:0/1649215575 conn(0x7f62e80fca60 msgr2=0x7f62e80fd5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:34.809 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:34.223+0000 7f62ec9a0700 1 -- 172.21.15.32:0/1649215575 shutdown_connections 2024-04-12T05:27:34.809 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:34.223+0000 7f62ec9a0700 1 -- 172.21.15.32:0/1649215575 wait complete. 2024-04-12T05:27:35.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:35 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/1649215575' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-12T05:27:35.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:35 smithi032 ceph-mon[39731]: mgrmap e5: smithi032.hrwxrh(active, since 9s) 2024-04-12T05:27:36.205 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-12T05:27:36.205 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-12T05:27:36.205 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-12T05:27:36.205 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "active_name": "smithi032.hrwxrh", 2024-04-12T05:27:36.206 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-12T05:27:36.206 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-12T05:27:36.206 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.376+0000 7f5af1589700 1 Processor -- start 2024-04-12T05:27:36.206 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.376+0000 7f5af1589700 1 -- start start 2024-04-12T05:27:36.206 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.377+0000 7f5af1589700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5aec102580 0x7f5aec102990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:36.206 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.377+0000 7f5af1589700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec102ed0 con 0x7f5aec102580 2024-04-12T05:27:36.206 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.377+0000 7f5aeaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5aec102580 0x7f5aec102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:36.206 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.377+0000 7f5aeaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5aec102580 0x7f5aec102990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35288/0 (socket says 172.21.15.32:35288) 2024-04-12T05:27:36.206 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.377+0000 7f5aeaffd700 1 -- 172.21.15.32:0/2297138039 learned_addr learned my addr 172.21.15.32:0/2297138039 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:36.206 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.377+0000 7f5aeaffd700 1 -- 172.21.15.32:0/2297138039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aec103010 con 0x7f5aec102580 2024-04-12T05:27:36.206 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.377+0000 7f5aeaffd700 1 --2- 172.21.15.32:0/2297138039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5aec102580 0x7f5aec102990 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5adc0098e0 tx=0x7f5adc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22b3a4ad7533098c server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:36.207 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.377+0000 7f5ae9ffb700 1 -- 172.21.15.32:0/2297138039 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5adc00b860 con 0x7f5aec102580 2024-04-12T05:27:36.207 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.377+0000 7f5ae9ffb700 1 -- 172.21.15.32:0/2297138039 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f5adc00be30 con 0x7f5aec102580 2024-04-12T05:27:36.207 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.377+0000 7f5ae9ffb700 1 -- 172.21.15.32:0/2297138039 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5adc010380 con 0x7f5aec102580 2024-04-12T05:27:36.207 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.378+0000 7f5af1589700 1 -- 172.21.15.32:0/2297138039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5aec102580 msgr2=0x7f5aec102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:36.207 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.378+0000 7f5af1589700 1 --2- 172.21.15.32:0/2297138039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5aec102580 0x7f5aec102990 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5adc0098e0 tx=0x7f5adc009250 comp rx=0 tx=0).stop 2024-04-12T05:27:36.207 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.378+0000 7f5af1589700 1 -- 172.21.15.32:0/2297138039 shutdown_connections 2024-04-12T05:27:36.207 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.378+0000 7f5af1589700 1 --2- 172.21.15.32:0/2297138039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5aec102580 0x7f5aec102990 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:36.207 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.378+0000 7f5af1589700 1 -- 172.21.15.32:0/2297138039 >> 172.21.15.32:0/2297138039 conn(0x7f5aec0fdd00 msgr2=0x7f5aec100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:36.207 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.378+0000 7f5af1589700 1 -- 172.21.15.32:0/2297138039 shutdown_connections 2024-04-12T05:27:36.207 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.378+0000 7f5af1589700 1 -- 172.21.15.32:0/2297138039 wait complete. 2024-04-12T05:27:36.207 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.379+0000 7f5af1589700 1 Processor -- start 2024-04-12T05:27:36.207 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.379+0000 7f5af1589700 1 -- start start 2024-04-12T05:27:36.207 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.379+0000 7f5af1589700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5aec102580 0x7f5aec078c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:36.208 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.379+0000 7f5af1589700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec079190 con 0x7f5aec102580 2024-04-12T05:27:36.208 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.379+0000 7f5aeaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5aec102580 0x7f5aec078c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:36.208 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.379+0000 7f5aeaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5aec102580 0x7f5aec078c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35302/0 (socket says 172.21.15.32:35302) 2024-04-12T05:27:36.208 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.379+0000 7f5aeaffd700 1 -- 172.21.15.32:0/3760205158 learned_addr learned my addr 172.21.15.32:0/3760205158 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:36.208 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.379+0000 7f5aeaffd700 1 -- 172.21.15.32:0/3760205158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5adc0094f0 con 0x7f5aec102580 2024-04-12T05:27:36.208 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.380+0000 7f5aeaffd700 1 --2- 172.21.15.32:0/3760205158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5aec102580 0x7f5aec078c50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5adc000c00 tx=0x7f5adc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:36.208 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.380+0000 7f5ad37fe700 1 -- 172.21.15.32:0/3760205158 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5adc010a80 con 0x7f5aec102580 2024-04-12T05:27:36.208 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.380+0000 7f5af1589700 1 -- 172.21.15.32:0/3760205158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aec077200 con 0x7f5aec102580 2024-04-12T05:27:36.208 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.380+0000 7f5ad37fe700 1 -- 172.21.15.32:0/3760205158 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f5adc01a430 con 0x7f5aec102580 2024-04-12T05:27:36.208 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.380+0000 7f5ad37fe700 1 -- 172.21.15.32:0/3760205158 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5adc0245b0 con 0x7f5aec102580 2024-04-12T05:27:36.208 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.380+0000 7f5af1589700 1 -- 172.21.15.32:0/3760205158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aec0776f0 con 0x7f5aec102580 2024-04-12T05:27:36.208 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.381+0000 7f5ad37fe700 1 -- 172.21.15.32:0/3760205158 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f5adc0247f0 con 0x7f5aec102580 2024-04-12T05:27:36.209 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.381+0000 7f5ad37fe700 1 --2- 172.21.15.32:0/3760205158 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7f5ad4036790 0x7f5ad4038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:36.209 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.381+0000 7f5af1589700 1 -- 172.21.15.32:0/3760205158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aec04bed0 con 0x7f5aec102580 2024-04-12T05:27:36.209 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.381+0000 7f5ad37fe700 1 -- 172.21.15.32:0/3760205158 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f5adc04afd0 con 0x7f5aec102580 2024-04-12T05:27:36.209 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.381+0000 7f5aea7fc700 1 -- 172.21.15.32:0/3760205158 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7f5ad4036790 msgr2=0x7f5ad4038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/656201914 2024-04-12T05:27:36.209 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.381+0000 7f5aea7fc700 1 --2- 172.21.15.32:0/3760205158 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7f5ad4036790 0x7f5ad4038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-12T05:27:36.209 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.390+0000 7f5ad37fe700 1 -- 172.21.15.32:0/3760205158 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5adc018350 con 0x7f5aec102580 2024-04-12T05:27:36.209 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.582+0000 7f5aea7fc700 1 -- 172.21.15.32:0/3760205158 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7f5ad4036790 msgr2=0x7f5ad4038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/656201914 2024-04-12T05:27:36.209 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.582+0000 7f5aea7fc700 1 --2- 172.21.15.32:0/3760205158 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7f5ad4036790 0x7f5ad4038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-12T05:27:36.209 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.638+0000 7f5af1589700 1 -- 172.21.15.32:0/3760205158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f5aec077d90 con 0x7f5aec102580 2024-04-12T05:27:36.209 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.638+0000 7f5ad37fe700 1 -- 172.21.15.32:0/3760205158 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f5adc023b80 con 0x7f5aec102580 2024-04-12T05:27:36.209 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.640+0000 7f5af1589700 1 -- 172.21.15.32:0/3760205158 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7f5ad4036790 msgr2=0x7f5ad4038c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T05:27:36.209 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.640+0000 7f5af1589700 1 --2- 172.21.15.32:0/3760205158 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7f5ad4036790 0x7f5ad4038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:36.210 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.640+0000 7f5af1589700 1 -- 172.21.15.32:0/3760205158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5aec102580 msgr2=0x7f5aec078c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:36.210 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.640+0000 7f5af1589700 1 --2- 172.21.15.32:0/3760205158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5aec102580 0x7f5aec078c50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5adc000c00 tx=0x7f5adc009250 comp rx=0 tx=0).stop 2024-04-12T05:27:36.210 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.640+0000 7f5af1589700 1 -- 172.21.15.32:0/3760205158 shutdown_connections 2024-04-12T05:27:36.210 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.640+0000 7f5af1589700 1 --2- 172.21.15.32:0/3760205158 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7f5ad4036790 0x7f5ad4038c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:36.210 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.640+0000 7f5af1589700 1 --2- 172.21.15.32:0/3760205158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5aec102580 0x7f5aec078c50 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:36.210 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.640+0000 7f5af1589700 1 -- 172.21.15.32:0/3760205158 >> 172.21.15.32:0/3760205158 conn(0x7f5aec0fdd00 msgr2=0x7f5aec0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:36.210 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.640+0000 7f5af1589700 1 -- 172.21.15.32:0/3760205158 shutdown_connections 2024-04-12T05:27:36.210 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:35.640+0000 7f5af1589700 1 -- 172.21.15.32:0/3760205158 wait complete. 2024-04-12T05:27:36.210 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for the mgr to restart... 2024-04-12T05:27:36.210 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mgr epoch 5... 2024-04-12T05:27:36.456 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:36 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/3760205158' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-12T05:27:41.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:41 smithi032 ceph-mon[39731]: Active manager daemon smithi032.hrwxrh restarted 2024-04-12T05:27:41.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:41 smithi032 ceph-mon[39731]: Activating manager daemon smithi032.hrwxrh 2024-04-12T05:27:41.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:41 smithi032 ceph-mon[39731]: osdmap e2: 0 total, 0 up, 0 in 2024-04-12T05:27:41.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:41 smithi032 ceph-mon[39731]: mgrmap e6: smithi032.hrwxrh(active, starting, since 0.00322518s) 2024-04-12T05:27:41.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:41 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-12T05:27:41.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:41 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr metadata", "who": "smithi032.hrwxrh", "id": "smithi032.hrwxrh"}]: dispatch 2024-04-12T05:27:41.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:41 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T05:27:41.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:41 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T05:27:41.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:41 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T05:27:41.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:41 smithi032 ceph-mon[39731]: Manager daemon smithi032.hrwxrh is now available 2024-04-12T05:27:41.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:41 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:27:41.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:41 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:27:41.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:41 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:27:41.096 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:41 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:27:41.096 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:41 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:27:42.310 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:42 smithi032 ceph-mon[39731]: Found migration_current of "None". Setting to last migration. 2024-04-12T05:27:42.311 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:42 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.hrwxrh/mirror_snapshot_schedule"}]: dispatch 2024-04-12T05:27:42.311 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:42 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.hrwxrh/trash_purge_schedule"}]: dispatch 2024-04-12T05:27:42.311 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:42 smithi032 ceph-mon[39731]: mgrmap e7: smithi032.hrwxrh(active, since 1.005s) 2024-04-12T05:27:42.580 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-12T05:27:42.580 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-12T05:27:42.580 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-12T05:27:42.580 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-12T05:27:42.580 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.743+0000 7fb60023c700 1 Processor -- start 2024-04-12T05:27:42.580 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.743+0000 7fb60023c700 1 -- start start 2024-04-12T05:27:42.580 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.743+0000 7fb60023c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5f8078d70 0x7fb5f8077280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:42.580 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.743+0000 7fb60023c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f8079180 con 0x7fb5f8078d70 2024-04-12T05:27:42.581 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.743+0000 7fb5fdfd8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5f8078d70 0x7fb5f8077280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:42.581 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.743+0000 7fb5fdfd8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5f8078d70 0x7fb5f8077280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35304/0 (socket says 172.21.15.32:35304) 2024-04-12T05:27:42.581 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.743+0000 7fb5fdfd8700 1 -- 172.21.15.32:0/2961587755 learned_addr learned my addr 172.21.15.32:0/2961587755 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:42.581 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.744+0000 7fb5fdfd8700 1 -- 172.21.15.32:0/2961587755 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f80777c0 con 0x7fb5f8078d70 2024-04-12T05:27:42.581 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.744+0000 7fb5fdfd8700 1 --2- 172.21.15.32:0/2961587755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5f8078d70 0x7fb5f8077280 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb5f40098e0 tx=0x7fb5f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7089f1ed1eadf130 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:42.581 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.744+0000 7fb5fcfd6700 1 -- 172.21.15.32:0/2961587755 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5f400b860 con 0x7fb5f8078d70 2024-04-12T05:27:42.581 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.744+0000 7fb5fcfd6700 1 -- 172.21.15.32:0/2961587755 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fb5f400be30 con 0x7fb5f8078d70 2024-04-12T05:27:42.581 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.744+0000 7fb5fcfd6700 1 -- 172.21.15.32:0/2961587755 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5f4010380 con 0x7fb5f8078d70 2024-04-12T05:27:42.581 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.745+0000 7fb60023c700 1 -- 172.21.15.32:0/2961587755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5f8078d70 msgr2=0x7fb5f8077280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:42.582 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.745+0000 7fb60023c700 1 --2- 172.21.15.32:0/2961587755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5f8078d70 0x7fb5f8077280 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb5f40098e0 tx=0x7fb5f4009250 comp rx=0 tx=0).stop 2024-04-12T05:27:42.582 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.745+0000 7fb60023c700 1 -- 172.21.15.32:0/2961587755 shutdown_connections 2024-04-12T05:27:42.582 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.745+0000 7fb60023c700 1 --2- 172.21.15.32:0/2961587755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5f8078d70 0x7fb5f8077280 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:42.582 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.745+0000 7fb60023c700 1 -- 172.21.15.32:0/2961587755 >> 172.21.15.32:0/2961587755 conn(0x7fb5f80fdd00 msgr2=0x7fb5f8100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:42.582 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.745+0000 7fb60023c700 1 -- 172.21.15.32:0/2961587755 shutdown_connections 2024-04-12T05:27:42.582 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.745+0000 7fb60023c700 1 -- 172.21.15.32:0/2961587755 wait complete. 2024-04-12T05:27:42.582 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.746+0000 7fb60023c700 1 Processor -- start 2024-04-12T05:27:42.582 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.746+0000 7fb60023c700 1 -- start start 2024-04-12T05:27:42.582 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.746+0000 7fb60023c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5f8078d70 0x7fb5f8113580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:42.583 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.746+0000 7fb60023c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f8113ac0 con 0x7fb5f8078d70 2024-04-12T05:27:42.583 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.746+0000 7fb5fdfd8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5f8078d70 0x7fb5f8113580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:42.583 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.746+0000 7fb5fdfd8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5f8078d70 0x7fb5f8113580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35312/0 (socket says 172.21.15.32:35312) 2024-04-12T05:27:42.583 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.746+0000 7fb5fdfd8700 1 -- 172.21.15.32:0/2278012974 learned_addr learned my addr 172.21.15.32:0/2278012974 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:42.583 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.746+0000 7fb5fdfd8700 1 -- 172.21.15.32:0/2278012974 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f40094f0 con 0x7fb5f8078d70 2024-04-12T05:27:42.583 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.747+0000 7fb5fdfd8700 1 --2- 172.21.15.32:0/2278012974 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5f8078d70 0x7fb5f8113580 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4006b00 tx=0x7fb5f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:42.583 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.747+0000 7fb5ee7fc700 1 -- 172.21.15.32:0/2278012974 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5f4010a80 con 0x7fb5f8078d70 2024-04-12T05:27:42.583 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.747+0000 7fb5ee7fc700 1 -- 172.21.15.32:0/2278012974 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fb5f401a430 con 0x7fb5f8078d70 2024-04-12T05:27:42.583 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.747+0000 7fb60023c700 1 -- 172.21.15.32:0/2278012974 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5f8113cc0 con 0x7fb5f8078d70 2024-04-12T05:27:42.584 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.747+0000 7fb5ee7fc700 1 -- 172.21.15.32:0/2278012974 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5f40245b0 con 0x7fb5f8078d70 2024-04-12T05:27:42.584 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.747+0000 7fb60023c700 1 -- 172.21.15.32:0/2278012974 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5f8114830 con 0x7fb5f8078d70 2024-04-12T05:27:42.584 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.748+0000 7fb5ee7fc700 1 -- 172.21.15.32:0/2278012974 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fb5f40247e0 con 0x7fb5f8078d70 2024-04-12T05:27:42.584 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.748+0000 7fb5ee7fc700 1 --2- 172.21.15.32:0/2278012974 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7fb5e40367e0 0x7fb5e4038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:42.584 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.748+0000 7fb5ee7fc700 1 -- 172.21.15.32:0/2278012974 --> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb5e40393a0 con 0x7fb5e40367e0 2024-04-12T05:27:42.584 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.748+0000 7fb5ee7fc700 1 -- 172.21.15.32:0/2278012974 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fb5f404ace0 con 0x7fb5f8078d70 2024-04-12T05:27:42.584 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.748+0000 7fb5fd7d7700 1 -- 172.21.15.32:0/2278012974 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7fb5e40367e0 msgr2=0x7fb5e4038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/656201914 2024-04-12T05:27:42.584 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.748+0000 7fb5fd7d7700 1 --2- 172.21.15.32:0/2278012974 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7fb5e40367e0 0x7fb5e4038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-12T05:27:42.584 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.949+0000 7fb5fd7d7700 1 -- 172.21.15.32:0/2278012974 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7fb5e40367e0 msgr2=0x7fb5e4038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/656201914 2024-04-12T05:27:42.584 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:36.949+0000 7fb5fd7d7700 1 --2- 172.21.15.32:0/2278012974 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7fb5e40367e0 0x7fb5e4038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-12T05:27:42.584 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:37.350+0000 7fb5fd7d7700 1 -- 172.21.15.32:0/2278012974 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7fb5e40367e0 msgr2=0x7fb5e4038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/656201914 2024-04-12T05:27:42.585 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:37.350+0000 7fb5fd7d7700 1 --2- 172.21.15.32:0/2278012974 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7fb5e40367e0 0x7fb5e4038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-12T05:27:42.585 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:38.151+0000 7fb5fd7d7700 1 -- 172.21.15.32:0/2278012974 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7fb5e40367e0 msgr2=0x7fb5e4038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/656201914 2024-04-12T05:27:42.585 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:38.151+0000 7fb5fd7d7700 1 --2- 172.21.15.32:0/2278012974 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7fb5e40367e0 0x7fb5e4038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-12T05:27:42.585 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:39.752+0000 7fb5fd7d7700 1 -- 172.21.15.32:0/2278012974 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7fb5e40367e0 msgr2=0x7fb5e4038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/656201914 2024-04-12T05:27:42.585 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:39.752+0000 7fb5fd7d7700 1 --2- 172.21.15.32:0/2278012974 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7fb5e40367e0 0x7fb5e4038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-12T05:27:42.585 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:40.953+0000 7fb5ee7fc700 1 -- 172.21.15.32:0/2278012974 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mgrmap(e 6) v1 ==== 43135+0+0 (secure 0 0 0) 0x7fb5f402e960 con 0x7fb5f8078d70 2024-04-12T05:27:42.585 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:40.953+0000 7fb5ee7fc700 1 -- 172.21.15.32:0/2278012974 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7fb5e40367e0 msgr2=0x7fb5e4038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T05:27:42.585 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:40.953+0000 7fb5ee7fc700 1 --2- 172.21.15.32:0/2278012974 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7fb5e40367e0 0x7fb5e4038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:42.586 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:41.954+0000 7fb5ee7fc700 1 -- 172.21.15.32:0/2278012974 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7fb5f402c390 con 0x7fb5f8078d70 2024-04-12T05:27:42.586 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:41.954+0000 7fb5ee7fc700 1 --2- 172.21.15.32:0/2278012974 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fb5e4039fc0 0x7fb5e403c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:42.586 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:41.954+0000 7fb5ee7fc700 1 -- 172.21.15.32:0/2278012974 --> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb5e40393a0 con 0x7fb5e4039fc0 2024-04-12T05:27:42.586 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:41.954+0000 7fb5fd7d7700 1 --2- 172.21.15.32:0/2278012974 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fb5e4039fc0 0x7fb5e403c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:42.586 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:41.955+0000 7fb5fd7d7700 1 --2- 172.21.15.32:0/2278012974 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fb5e4039fc0 0x7fb5e403c3a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb5e80039d0 tx=0x7fb5e8009350 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:42.586 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:41.956+0000 7fb5ee7fc700 1 -- 172.21.15.32:0/2278012974 <== mgr.14118 v2:172.21.15.32:6800/1962278696 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7fb5e40393a0 con 0x7fb5e4039fc0 2024-04-12T05:27:42.586 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:41.966+0000 7fb5e3fff700 1 -- 172.21.15.32:0/2278012974 --> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fb5f8114150 con 0x7fb5e4039fc0 2024-04-12T05:27:42.586 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:41.966+0000 7fb5ee7fc700 1 -- 172.21.15.32:0/2278012974 <== mgr.14118 v2:172.21.15.32:6800/1962278696 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fb5f8114150 con 0x7fb5e4039fc0 2024-04-12T05:27:42.586 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:41.967+0000 7fb5e3fff700 1 -- 172.21.15.32:0/2278012974 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fb5e4039fc0 msgr2=0x7fb5e403c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:42.586 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:41.967+0000 7fb5e3fff700 1 --2- 172.21.15.32:0/2278012974 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fb5e4039fc0 0x7fb5e403c3a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb5e80039d0 tx=0x7fb5e8009350 comp rx=0 tx=0).stop 2024-04-12T05:27:42.587 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:41.967+0000 7fb5e3fff700 1 -- 172.21.15.32:0/2278012974 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5f8078d70 msgr2=0x7fb5f8113580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:42.587 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:41.967+0000 7fb5e3fff700 1 --2- 172.21.15.32:0/2278012974 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5f8078d70 0x7fb5f8113580 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4006b00 tx=0x7fb5f4009250 comp rx=0 tx=0).stop 2024-04-12T05:27:42.587 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:41.967+0000 7fb5e3fff700 1 -- 172.21.15.32:0/2278012974 shutdown_connections 2024-04-12T05:27:42.587 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:41.967+0000 7fb5e3fff700 1 --2- 172.21.15.32:0/2278012974 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fb5e4039fc0 0x7fb5e403c3a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:42.587 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:41.967+0000 7fb5e3fff700 1 --2- 172.21.15.32:0/2278012974 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5f8078d70 0x7fb5f8113580 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:42.587 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:41.967+0000 7fb5e3fff700 1 --2- 172.21.15.32:0/2278012974 >> [v2:172.21.15.32:6800/656201914,v1:172.21.15.32:6801/656201914] conn(0x7fb5e40367e0 0x7fb5e4038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:42.587 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:41.967+0000 7fb5e3fff700 1 -- 172.21.15.32:0/2278012974 >> 172.21.15.32:0/2278012974 conn(0x7fb5f80fdd00 msgr2=0x7fb5f80fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:42.587 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:41.967+0000 7fb5e3fff700 1 -- 172.21.15.32:0/2278012974 shutdown_connections 2024-04-12T05:27:42.587 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:41.967+0000 7fb5e3fff700 1 -- 172.21.15.32:0/2278012974 wait complete. 2024-04-12T05:27:42.587 INFO:teuthology.orchestra.run.smithi032.stdout:mgr epoch 5 is available 2024-04-12T05:27:42.587 INFO:teuthology.orchestra.run.smithi032.stdout:Setting orchestrator backend to cephadm... 2024-04-12T05:27:43.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:43 smithi032 ceph-mon[39731]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-12T05:27:43.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:43 smithi032 ceph-mon[39731]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-12T05:27:43.925 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.082+0000 7fa722961700 1 Processor -- start 2024-04-12T05:27:43.925 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.082+0000 7fa722961700 1 -- start start 2024-04-12T05:27:43.926 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.082+0000 7fa722961700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71c076cc0 0x7fa71c105130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:43.926 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.082+0000 7fa722961700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa71c105670 con 0x7fa71c076cc0 2024-04-12T05:27:43.926 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.082+0000 7fa71bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71c076cc0 0x7fa71c105130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:43.926 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.082+0000 7fa71bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71c076cc0 0x7fa71c105130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55018/0 (socket says 172.21.15.32:55018) 2024-04-12T05:27:43.926 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.082+0000 7fa71bfff700 1 -- 172.21.15.32:0/1505686678 learned_addr learned my addr 172.21.15.32:0/1505686678 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:43.926 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.083+0000 7fa71bfff700 1 -- 172.21.15.32:0/1505686678 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa71c1057b0 con 0x7fa71c076cc0 2024-04-12T05:27:43.926 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.083+0000 7fa71bfff700 1 --2- 172.21.15.32:0/1505686678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71c076cc0 0x7fa71c105130 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa70c0098e0 tx=0x7fa70c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b17b0a0543b7ced server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:43.926 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.083+0000 7fa71affd700 1 -- 172.21.15.32:0/1505686678 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa70c00b860 con 0x7fa71c076cc0 2024-04-12T05:27:43.926 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.083+0000 7fa71affd700 1 -- 172.21.15.32:0/1505686678 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fa70c00be30 con 0x7fa71c076cc0 2024-04-12T05:27:43.926 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.083+0000 7fa71affd700 1 -- 172.21.15.32:0/1505686678 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa70c010380 con 0x7fa71c076cc0 2024-04-12T05:27:43.926 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.083+0000 7fa722961700 1 -- 172.21.15.32:0/1505686678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71c076cc0 msgr2=0x7fa71c105130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:43.926 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.083+0000 7fa722961700 1 --2- 172.21.15.32:0/1505686678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71c076cc0 0x7fa71c105130 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa70c0098e0 tx=0x7fa70c009250 comp rx=0 tx=0).stop 2024-04-12T05:27:43.926 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.083+0000 7fa722961700 1 -- 172.21.15.32:0/1505686678 shutdown_connections 2024-04-12T05:27:43.927 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.083+0000 7fa722961700 1 --2- 172.21.15.32:0/1505686678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71c076cc0 0x7fa71c105130 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:43.927 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.083+0000 7fa722961700 1 -- 172.21.15.32:0/1505686678 >> 172.21.15.32:0/1505686678 conn(0x7fa71c0fcb20 msgr2=0x7fa71c0fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:43.927 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.083+0000 7fa722961700 1 -- 172.21.15.32:0/1505686678 shutdown_connections 2024-04-12T05:27:43.927 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.084+0000 7fa722961700 1 -- 172.21.15.32:0/1505686678 wait complete. 2024-04-12T05:27:43.927 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.084+0000 7fa722961700 1 Processor -- start 2024-04-12T05:27:43.927 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.084+0000 7fa722961700 1 -- start start 2024-04-12T05:27:43.927 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.084+0000 7fa722961700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71c076cc0 0x7fa71c102a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:43.927 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.084+0000 7fa722961700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa71c102fc0 con 0x7fa71c076cc0 2024-04-12T05:27:43.927 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.085+0000 7fa71bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71c076cc0 0x7fa71c102a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:43.927 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.085+0000 7fa71bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71c076cc0 0x7fa71c102a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55028/0 (socket says 172.21.15.32:55028) 2024-04-12T05:27:43.927 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.085+0000 7fa71bfff700 1 -- 172.21.15.32:0/4077869650 learned_addr learned my addr 172.21.15.32:0/4077869650 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:43.927 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.085+0000 7fa71bfff700 1 -- 172.21.15.32:0/4077869650 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa70c0094f0 con 0x7fa71c076cc0 2024-04-12T05:27:43.927 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.085+0000 7fa71bfff700 1 --2- 172.21.15.32:0/4077869650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71c076cc0 0x7fa71c102a80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa70c0090b0 tx=0x7fa70c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:43.927 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.085+0000 7fa718ff9700 1 -- 172.21.15.32:0/4077869650 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa70c010960 con 0x7fa71c076cc0 2024-04-12T05:27:43.928 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.085+0000 7fa718ff9700 1 -- 172.21.15.32:0/4077869650 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fa70c01a430 con 0x7fa71c076cc0 2024-04-12T05:27:43.928 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.085+0000 7fa722961700 1 -- 172.21.15.32:0/4077869650 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa71c101030 con 0x7fa71c076cc0 2024-04-12T05:27:43.928 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.085+0000 7fa718ff9700 1 -- 172.21.15.32:0/4077869650 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa70c0245b0 con 0x7fa71c076cc0 2024-04-12T05:27:43.928 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.085+0000 7fa722961700 1 -- 172.21.15.32:0/4077869650 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa71c101520 con 0x7fa71c076cc0 2024-04-12T05:27:43.928 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.086+0000 7fa718ff9700 1 -- 172.21.15.32:0/4077869650 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fa70c024830 con 0x7fa71c076cc0 2024-04-12T05:27:43.928 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.086+0000 7fa722961700 1 -- 172.21.15.32:0/4077869650 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa71c04bed0 con 0x7fa71c076cc0 2024-04-12T05:27:43.928 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.086+0000 7fa718ff9700 1 --2- 172.21.15.32:0/4077869650 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fa704036790 0x7fa704038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:43.928 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.086+0000 7fa718ff9700 1 -- 172.21.15.32:0/4077869650 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa70c04a760 con 0x7fa71c076cc0 2024-04-12T05:27:43.928 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.086+0000 7fa71b7fe700 1 --2- 172.21.15.32:0/4077869650 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fa704036790 0x7fa704038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:43.928 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.087+0000 7fa71b7fe700 1 --2- 172.21.15.32:0/4077869650 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fa704036790 0x7fa704038c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa710009990 tx=0x7fa710006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:43.928 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.095+0000 7fa718ff9700 1 -- 172.21.15.32:0/4077869650 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fa70c02c340 con 0x7fa71c076cc0 2024-04-12T05:27:43.928 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.302+0000 7fa722961700 1 -- 172.21.15.32:0/4077869650 --> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fa71c101c30 con 0x7fa704036790 2024-04-12T05:27:43.928 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.310+0000 7fa718ff9700 1 -- 172.21.15.32:0/4077869650 <== mgr.14118 v2:172.21.15.32:6800/1962278696 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa71c101c30 con 0x7fa704036790 2024-04-12T05:27:43.928 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.312+0000 7fa722961700 1 -- 172.21.15.32:0/4077869650 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fa704036790 msgr2=0x7fa704038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:43.929 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.312+0000 7fa722961700 1 --2- 172.21.15.32:0/4077869650 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fa704036790 0x7fa704038c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa710009990 tx=0x7fa710006e40 comp rx=0 tx=0).stop 2024-04-12T05:27:43.929 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.312+0000 7fa722961700 1 -- 172.21.15.32:0/4077869650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71c076cc0 msgr2=0x7fa71c102a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:43.929 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.312+0000 7fa722961700 1 --2- 172.21.15.32:0/4077869650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71c076cc0 0x7fa71c102a80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa70c0090b0 tx=0x7fa70c009250 comp rx=0 tx=0).stop 2024-04-12T05:27:43.929 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.313+0000 7fa722961700 1 -- 172.21.15.32:0/4077869650 shutdown_connections 2024-04-12T05:27:43.929 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.313+0000 7fa722961700 1 --2- 172.21.15.32:0/4077869650 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fa704036790 0x7fa704038c40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:43.929 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.313+0000 7fa722961700 1 --2- 172.21.15.32:0/4077869650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71c076cc0 0x7fa71c102a80 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:43.929 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.313+0000 7fa722961700 1 -- 172.21.15.32:0/4077869650 >> 172.21.15.32:0/4077869650 conn(0x7fa71c0fcb20 msgr2=0x7fa71c0fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:43.929 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.313+0000 7fa722961700 1 -- 172.21.15.32:0/4077869650 shutdown_connections 2024-04-12T05:27:43.929 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:43.313+0000 7fa722961700 1 -- 172.21.15.32:0/4077869650 wait complete. 2024-04-12T05:27:44.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:44 smithi032 ceph-mon[39731]: mgrmap e8: smithi032.hrwxrh(active, since 2s) 2024-04-12T05:27:44.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:44 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:27:44.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:44 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:27:44.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:44 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:27:44.345 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:44 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:27:45.257 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-12T05:27:45.257 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.416+0000 7f11ed02a700 1 Processor -- start 2024-04-12T05:27:45.257 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.416+0000 7f11ed02a700 1 -- start start 2024-04-12T05:27:45.257 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.416+0000 7f11ed02a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e81014e0 0x7f11e81018f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:45.257 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.416+0000 7f11ed02a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e8101e30 con 0x7f11e81014e0 2024-04-12T05:27:45.257 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.416+0000 7f11e6d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e81014e0 0x7f11e81018f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:45.257 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.416+0000 7f11e6d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e81014e0 0x7f11e81018f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55038/0 (socket says 172.21.15.32:55038) 2024-04-12T05:27:45.258 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.416+0000 7f11e6d9d700 1 -- 172.21.15.32:0/1182446174 learned_addr learned my addr 172.21.15.32:0/1182446174 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:45.258 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.417+0000 7f11e6d9d700 1 -- 172.21.15.32:0/1182446174 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e8101f70 con 0x7f11e81014e0 2024-04-12T05:27:45.258 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.417+0000 7f11e6d9d700 1 --2- 172.21.15.32:0/1182446174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e81014e0 0x7f11e81018f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f11d80098e0 tx=0x7f11d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=990d8b662c4227ab server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:45.258 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.417+0000 7f11e5d9b700 1 -- 172.21.15.32:0/1182446174 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11d800b860 con 0x7f11e81014e0 2024-04-12T05:27:45.258 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.417+0000 7f11e5d9b700 1 -- 172.21.15.32:0/1182446174 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f11d800be30 con 0x7f11e81014e0 2024-04-12T05:27:45.258 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.417+0000 7f11e5d9b700 1 -- 172.21.15.32:0/1182446174 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11d8010380 con 0x7f11e81014e0 2024-04-12T05:27:45.258 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.417+0000 7f11ed02a700 1 -- 172.21.15.32:0/1182446174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e81014e0 msgr2=0x7f11e81018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:45.258 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.417+0000 7f11ed02a700 1 --2- 172.21.15.32:0/1182446174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e81014e0 0x7f11e81018f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f11d80098e0 tx=0x7f11d8009250 comp rx=0 tx=0).stop 2024-04-12T05:27:45.258 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.417+0000 7f11ed02a700 1 -- 172.21.15.32:0/1182446174 shutdown_connections 2024-04-12T05:27:45.258 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.417+0000 7f11ed02a700 1 --2- 172.21.15.32:0/1182446174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e81014e0 0x7f11e81018f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:45.258 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.417+0000 7f11ed02a700 1 -- 172.21.15.32:0/1182446174 >> 172.21.15.32:0/1182446174 conn(0x7f11e80fcc60 msgr2=0x7f11e80ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:45.258 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.417+0000 7f11ed02a700 1 -- 172.21.15.32:0/1182446174 shutdown_connections 2024-04-12T05:27:45.258 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.417+0000 7f11ed02a700 1 -- 172.21.15.32:0/1182446174 wait complete. 2024-04-12T05:27:45.258 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.418+0000 7f11ed02a700 1 Processor -- start 2024-04-12T05:27:45.258 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.418+0000 7f11ed02a700 1 -- start start 2024-04-12T05:27:45.259 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.418+0000 7f11ed02a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e81014e0 0x7f11e8076b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:45.259 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.418+0000 7f11ed02a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e8114040 con 0x7f11e81014e0 2024-04-12T05:27:45.259 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.418+0000 7f11e6d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e81014e0 0x7f11e8076b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:45.259 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.418+0000 7f11e6d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e81014e0 0x7f11e8076b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55044/0 (socket says 172.21.15.32:55044) 2024-04-12T05:27:45.259 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.418+0000 7f11e6d9d700 1 -- 172.21.15.32:0/2038980944 learned_addr learned my addr 172.21.15.32:0/2038980944 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:45.259 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.419+0000 7f11e6d9d700 1 -- 172.21.15.32:0/2038980944 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d80094f0 con 0x7f11e81014e0 2024-04-12T05:27:45.259 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.419+0000 7f11e6d9d700 1 --2- 172.21.15.32:0/2038980944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e81014e0 0x7f11e8076b90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f11d8000c00 tx=0x7f11d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:45.259 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.419+0000 7f11cf7fe700 1 -- 172.21.15.32:0/2038980944 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11d8010a80 con 0x7f11e81014e0 2024-04-12T05:27:45.259 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.419+0000 7f11cf7fe700 1 -- 172.21.15.32:0/2038980944 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f11d801a430 con 0x7f11e81014e0 2024-04-12T05:27:45.259 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.419+0000 7f11cf7fe700 1 -- 172.21.15.32:0/2038980944 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11d80245b0 con 0x7f11e81014e0 2024-04-12T05:27:45.259 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:45 smithi032 ceph-mon[39731]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:27:45.260 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:45 smithi032 ceph-mon[39731]: [12/Apr/2024:05:27:43] ENGINE Bus STARTING 2024-04-12T05:27:45.260 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:45 smithi032 ceph-mon[39731]: [12/Apr/2024:05:27:43] ENGINE Serving on https://172.21.15.32:7150 2024-04-12T05:27:45.260 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:45 smithi032 ceph-mon[39731]: [12/Apr/2024:05:27:43] ENGINE Bus STARTED 2024-04-12T05:27:45.262 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.419+0000 7f11ed02a700 1 -- 172.21.15.32:0/2038980944 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11e8114180 con 0x7f11e81014e0 2024-04-12T05:27:45.262 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.419+0000 7f11ed02a700 1 -- 172.21.15.32:0/2038980944 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11e8114670 con 0x7f11e81014e0 2024-04-12T05:27:45.262 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.420+0000 7f11cf7fe700 1 -- 172.21.15.32:0/2038980944 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f11d801a5a0 con 0x7f11e81014e0 2024-04-12T05:27:45.262 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.420+0000 7f11cf7fe700 1 --2- 172.21.15.32:0/2038980944 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f11d0036790 0x7f11d0038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:45.262 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.420+0000 7f11cd7fa700 1 -- 172.21.15.32:0/2038980944 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11c80052b0 con 0x7f11e81014e0 2024-04-12T05:27:45.263 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.420+0000 7f11cf7fe700 1 -- 172.21.15.32:0/2038980944 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f11d804af00 con 0x7f11e81014e0 2024-04-12T05:27:45.263 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.420+0000 7f11e659c700 1 --2- 172.21.15.32:0/2038980944 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f11d0036790 0x7f11d0038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:45.263 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.421+0000 7f11e659c700 1 --2- 172.21.15.32:0/2038980944 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f11d0036790 0x7f11d0038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f11dc009990 tx=0x7f11dc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:45.263 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.425+0000 7f11cf7fe700 1 -- 172.21.15.32:0/2038980944 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f11d8023df0 con 0x7f11e81014e0 2024-04-12T05:27:45.263 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.619+0000 7f11cd7fa700 1 -- 172.21.15.32:0/2038980944 --> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f11c8000bc0 con 0x7f11d0036790 2024-04-12T05:27:45.263 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.620+0000 7f11cf7fe700 1 -- 172.21.15.32:0/2038980944 <== mgr.14118 v2:172.21.15.32:6800/1962278696 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f11c8000bc0 con 0x7f11d0036790 2024-04-12T05:27:45.263 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.622+0000 7f11cd7fa700 1 -- 172.21.15.32:0/2038980944 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f11d0036790 msgr2=0x7f11d0038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:45.263 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.622+0000 7f11cd7fa700 1 --2- 172.21.15.32:0/2038980944 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f11d0036790 0x7f11d0038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f11dc009990 tx=0x7f11dc006e40 comp rx=0 tx=0).stop 2024-04-12T05:27:45.263 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.622+0000 7f11cd7fa700 1 -- 172.21.15.32:0/2038980944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e81014e0 msgr2=0x7f11e8076b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:45.263 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.622+0000 7f11cd7fa700 1 --2- 172.21.15.32:0/2038980944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e81014e0 0x7f11e8076b90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f11d8000c00 tx=0x7f11d8009250 comp rx=0 tx=0).stop 2024-04-12T05:27:45.263 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.622+0000 7f11cd7fa700 1 -- 172.21.15.32:0/2038980944 shutdown_connections 2024-04-12T05:27:45.263 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.622+0000 7f11cd7fa700 1 --2- 172.21.15.32:0/2038980944 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f11d0036790 0x7f11d0038c40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:45.263 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.622+0000 7f11cd7fa700 1 --2- 172.21.15.32:0/2038980944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e81014e0 0x7f11e8076b90 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:45.263 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.622+0000 7f11cd7fa700 1 -- 172.21.15.32:0/2038980944 >> 172.21.15.32:0/2038980944 conn(0x7f11e80fcc60 msgr2=0x7f11e80fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:45.264 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.622+0000 7f11cd7fa700 1 -- 172.21.15.32:0/2038980944 shutdown_connections 2024-04-12T05:27:45.264 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:44.622+0000 7f11cd7fa700 1 -- 172.21.15.32:0/2038980944 wait complete. 2024-04-12T05:27:45.264 INFO:teuthology.orchestra.run.smithi032.stdout:Generating ssh key... 2024-04-12T05:27:46.262 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:46 smithi032 ceph-mon[39731]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:27:46.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.793+0000 7f897fe79700 1 Processor -- start 2024-04-12T05:27:46.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.793+0000 7f897fe79700 1 -- start start 2024-04-12T05:27:46.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.793+0000 7f897fe79700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8978102580 0x7f8978102990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:46.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.793+0000 7f897fe79700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8978102ed0 con 0x7f8978102580 2024-04-12T05:27:46.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.794+0000 7f897dc15700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8978102580 0x7f8978102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:46.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.794+0000 7f897dc15700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8978102580 0x7f8978102990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55060/0 (socket says 172.21.15.32:55060) 2024-04-12T05:27:46.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.794+0000 7f897dc15700 1 -- 172.21.15.32:0/1006574956 learned_addr learned my addr 172.21.15.32:0/1006574956 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:46.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.794+0000 7f897dc15700 1 -- 172.21.15.32:0/1006574956 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8978103010 con 0x7f8978102580 2024-04-12T05:27:46.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.794+0000 7f897dc15700 1 --2- 172.21.15.32:0/1006574956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8978102580 0x7f8978102990 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f896c0098e0 tx=0x7f896c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0b35c68dc263a63 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:46.781 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.794+0000 7f897cc13700 1 -- 172.21.15.32:0/1006574956 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f896c00b860 con 0x7f8978102580 2024-04-12T05:27:46.781 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.794+0000 7f897cc13700 1 -- 172.21.15.32:0/1006574956 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f896c00be30 con 0x7f8978102580 2024-04-12T05:27:46.781 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.794+0000 7f897cc13700 1 -- 172.21.15.32:0/1006574956 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f896c010380 con 0x7f8978102580 2024-04-12T05:27:46.781 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.795+0000 7f897fe79700 1 -- 172.21.15.32:0/1006574956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8978102580 msgr2=0x7f8978102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:46.781 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.795+0000 7f897fe79700 1 --2- 172.21.15.32:0/1006574956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8978102580 0x7f8978102990 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f896c0098e0 tx=0x7f896c009250 comp rx=0 tx=0).stop 2024-04-12T05:27:46.781 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.795+0000 7f897fe79700 1 -- 172.21.15.32:0/1006574956 shutdown_connections 2024-04-12T05:27:46.781 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.795+0000 7f897fe79700 1 --2- 172.21.15.32:0/1006574956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8978102580 0x7f8978102990 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:46.781 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.795+0000 7f897fe79700 1 -- 172.21.15.32:0/1006574956 >> 172.21.15.32:0/1006574956 conn(0x7f89780fdd00 msgr2=0x7f8978100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:46.781 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.795+0000 7f897fe79700 1 -- 172.21.15.32:0/1006574956 shutdown_connections 2024-04-12T05:27:46.781 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.795+0000 7f897fe79700 1 -- 172.21.15.32:0/1006574956 wait complete. 2024-04-12T05:27:46.781 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.796+0000 7f897fe79700 1 Processor -- start 2024-04-12T05:27:46.781 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.796+0000 7f897fe79700 1 -- start start 2024-04-12T05:27:46.781 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.796+0000 7f897fe79700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8978102580 0x7f8978078c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:46.782 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.796+0000 7f897fe79700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8978079190 con 0x7f8978102580 2024-04-12T05:27:46.782 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.796+0000 7f897dc15700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8978102580 0x7f8978078c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:46.782 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.796+0000 7f897dc15700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8978102580 0x7f8978078c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55068/0 (socket says 172.21.15.32:55068) 2024-04-12T05:27:46.782 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.796+0000 7f897dc15700 1 -- 172.21.15.32:0/255469396 learned_addr learned my addr 172.21.15.32:0/255469396 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:46.782 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.796+0000 7f897dc15700 1 -- 172.21.15.32:0/255469396 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f896c0094f0 con 0x7f8978102580 2024-04-12T05:27:46.782 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.796+0000 7f897dc15700 1 --2- 172.21.15.32:0/255469396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8978102580 0x7f8978078c50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f896c000c00 tx=0x7f896c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:46.782 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.796+0000 7f896a7fc700 1 -- 172.21.15.32:0/255469396 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f896c010a80 con 0x7f8978102580 2024-04-12T05:27:46.782 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.797+0000 7f896a7fc700 1 -- 172.21.15.32:0/255469396 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f896c01a430 con 0x7f8978102580 2024-04-12T05:27:46.782 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.797+0000 7f896a7fc700 1 -- 172.21.15.32:0/255469396 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f896c0245b0 con 0x7f8978102580 2024-04-12T05:27:46.782 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.797+0000 7f897fe79700 1 -- 172.21.15.32:0/255469396 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8978077200 con 0x7f8978102580 2024-04-12T05:27:46.783 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.797+0000 7f897fe79700 1 -- 172.21.15.32:0/255469396 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89780776f0 con 0x7f8978102580 2024-04-12T05:27:46.783 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.797+0000 7f896a7fc700 1 -- 172.21.15.32:0/255469396 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f896c0247a0 con 0x7f8978102580 2024-04-12T05:27:46.783 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.798+0000 7f896a7fc700 1 --2- 172.21.15.32:0/255469396 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f8964036830 0x7f8964038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:46.783 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.798+0000 7f896a7fc700 1 -- 172.21.15.32:0/255469396 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f896c04ade0 con 0x7f8978102580 2024-04-12T05:27:46.783 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.798+0000 7f897fe79700 1 -- 172.21.15.32:0/255469396 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f897804bed0 con 0x7f8978102580 2024-04-12T05:27:46.783 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.798+0000 7f897d414700 1 --2- 172.21.15.32:0/255469396 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f8964036830 0x7f8964038ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:46.783 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.798+0000 7f897d414700 1 --2- 172.21.15.32:0/255469396 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f8964036830 0x7f8964038ce0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8974009990 tx=0x7f8974006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:46.783 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.803+0000 7f896a7fc700 1 -- 172.21.15.32:0/255469396 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f896c018350 con 0x7f8978102580 2024-04-12T05:27:46.783 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:45.998+0000 7f897fe79700 1 -- 172.21.15.32:0/255469396 --> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f8978077d80 con 0x7f8964036830 2024-04-12T05:27:46.783 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:46.230+0000 7f896a7fc700 1 -- 172.21.15.32:0/255469396 <== mgr.14118 v2:172.21.15.32:6800/1962278696 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8978077d80 con 0x7f8964036830 2024-04-12T05:27:46.784 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:46.232+0000 7f897fe79700 1 -- 172.21.15.32:0/255469396 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f8964036830 msgr2=0x7f8964038ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:46.784 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:46.232+0000 7f897fe79700 1 --2- 172.21.15.32:0/255469396 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f8964036830 0x7f8964038ce0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8974009990 tx=0x7f8974006e40 comp rx=0 tx=0).stop 2024-04-12T05:27:46.784 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:46.232+0000 7f897fe79700 1 -- 172.21.15.32:0/255469396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8978102580 msgr2=0x7f8978078c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:46.784 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:46.232+0000 7f897fe79700 1 --2- 172.21.15.32:0/255469396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8978102580 0x7f8978078c50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f896c000c00 tx=0x7f896c009250 comp rx=0 tx=0).stop 2024-04-12T05:27:46.784 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:46.232+0000 7f897fe79700 1 -- 172.21.15.32:0/255469396 shutdown_connections 2024-04-12T05:27:46.784 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:46.232+0000 7f897fe79700 1 --2- 172.21.15.32:0/255469396 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f8964036830 0x7f8964038ce0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:46.784 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:46.232+0000 7f897fe79700 1 --2- 172.21.15.32:0/255469396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8978102580 0x7f8978078c50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:46.784 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:46.232+0000 7f897fe79700 1 -- 172.21.15.32:0/255469396 >> 172.21.15.32:0/255469396 conn(0x7f89780fdd00 msgr2=0x7f89780fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:46.784 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:46.232+0000 7f897fe79700 1 -- 172.21.15.32:0/255469396 shutdown_connections 2024-04-12T05:27:46.784 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:46.232+0000 7f897fe79700 1 -- 172.21.15.32:0/255469396 wait complete. 2024-04-12T05:27:47.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:47 smithi032 ceph-mon[39731]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:27:47.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:47 smithi032 ceph-mon[39731]: Generating ssh key... 2024-04-12T05:27:47.345 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:47 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:27:47.345 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:47 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:27:48.195 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDi3bXu0Cy8xhr5T9PkF+z5FhEu/yuN+ZOK946TKkeBAiP61LERkkWB5r2v+pXNfxy/84+rZuxipOcTA9AgK3p4yfhJa4LWOTJGJmRoIDFa4V54+xsoAiPMLI40V+GLfUwJk9OaQB+JHYE2o0tvQrwNckCQKwODQ121HRpZngzXjtru4cTYAXwmRNikEFPShLp2zzJRF/0O7PxvBDO4pvHFV1zh1b0AMFtJ62ukPyBSZnJXzwfvvh7a2MdHpr7cavkhcCBe8VK4o/QpuXi6mE/jZhI7yww6RN+7hb6g296KA0agDnHAu5eoDdF1PgrowtSBg3y4ODNUySXfjo0v1ISG2Apo8HdXb3b2NoaW3BGDjWYosSsj26wVhFCa+9aMyuTre9WPjMfewPL8ShlkraGNTy3kawZUCVuoIPRN9Fl2MYKekSO8+59aPgQi4WCYj5sqT1s7bUqMSMIr17Xa1hoR23Y3SoeDzbGx92BSTm+tLUxYH5JByKGEHBoSxuVt2hM= ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968 2024-04-12T05:27:48.195 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.395+0000 7f233007f700 1 Processor -- start 2024-04-12T05:27:48.195 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.395+0000 7f233007f700 1 -- start start 2024-04-12T05:27:48.195 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.395+0000 7f233007f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2328076c30 0x7f23281051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:48.196 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.395+0000 7f233007f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2328105700 con 0x7f2328076c30 2024-04-12T05:27:48.196 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.396+0000 7f232de1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2328076c30 0x7f23281051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:48.196 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.396+0000 7f232de1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2328076c30 0x7f23281051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55082/0 (socket says 172.21.15.32:55082) 2024-04-12T05:27:48.196 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.396+0000 7f232de1b700 1 -- 172.21.15.32:0/102982285 learned_addr learned my addr 172.21.15.32:0/102982285 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:48.196 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.396+0000 7f232de1b700 1 -- 172.21.15.32:0/102982285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2328105840 con 0x7f2328076c30 2024-04-12T05:27:48.196 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.396+0000 7f232de1b700 1 --2- 172.21.15.32:0/102982285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2328076c30 0x7f23281051c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f231c0098e0 tx=0x7f231c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3660a5902ca98859 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:48.196 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.396+0000 7f232ce19700 1 -- 172.21.15.32:0/102982285 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f231c00b860 con 0x7f2328076c30 2024-04-12T05:27:48.196 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.396+0000 7f232ce19700 1 -- 172.21.15.32:0/102982285 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f231c00be30 con 0x7f2328076c30 2024-04-12T05:27:48.196 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.396+0000 7f232ce19700 1 -- 172.21.15.32:0/102982285 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f231c010380 con 0x7f2328076c30 2024-04-12T05:27:48.196 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.397+0000 7f233007f700 1 -- 172.21.15.32:0/102982285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2328076c30 msgr2=0x7f23281051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:48.196 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.397+0000 7f233007f700 1 --2- 172.21.15.32:0/102982285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2328076c30 0x7f23281051c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f231c0098e0 tx=0x7f231c009250 comp rx=0 tx=0).stop 2024-04-12T05:27:48.196 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.397+0000 7f233007f700 1 -- 172.21.15.32:0/102982285 shutdown_connections 2024-04-12T05:27:48.196 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.397+0000 7f233007f700 1 --2- 172.21.15.32:0/102982285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2328076c30 0x7f23281051c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:48.196 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.397+0000 7f233007f700 1 -- 172.21.15.32:0/102982285 >> 172.21.15.32:0/102982285 conn(0x7f23280fcc70 msgr2=0x7f23280ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:48.196 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.397+0000 7f233007f700 1 -- 172.21.15.32:0/102982285 shutdown_connections 2024-04-12T05:27:48.197 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.397+0000 7f233007f700 1 -- 172.21.15.32:0/102982285 wait complete. 2024-04-12T05:27:48.197 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.397+0000 7f233007f700 1 Processor -- start 2024-04-12T05:27:48.197 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.397+0000 7f233007f700 1 -- start start 2024-04-12T05:27:48.197 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.397+0000 7f233007f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2328076c30 0x7f2328111350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:48.197 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.397+0000 7f233007f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2328111890 con 0x7f2328076c30 2024-04-12T05:27:48.197 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.398+0000 7f232de1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2328076c30 0x7f2328111350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:48.197 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.398+0000 7f232de1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2328076c30 0x7f2328111350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55092/0 (socket says 172.21.15.32:55092) 2024-04-12T05:27:48.197 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.398+0000 7f232de1b700 1 -- 172.21.15.32:0/3897546570 learned_addr learned my addr 172.21.15.32:0/3897546570 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:48.197 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.398+0000 7f232de1b700 1 -- 172.21.15.32:0/3897546570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231c0094f0 con 0x7f2328076c30 2024-04-12T05:27:48.197 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.398+0000 7f232de1b700 1 --2- 172.21.15.32:0/3897546570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2328076c30 0x7f2328111350 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f231c006b00 tx=0x7f231c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:48.197 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.398+0000 7f231a7fc700 1 -- 172.21.15.32:0/3897546570 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f231c010a80 con 0x7f2328076c30 2024-04-12T05:27:48.197 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.398+0000 7f231a7fc700 1 -- 172.21.15.32:0/3897546570 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f231c01a430 con 0x7f2328076c30 2024-04-12T05:27:48.197 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.398+0000 7f233007f700 1 -- 172.21.15.32:0/3897546570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2328111a90 con 0x7f2328076c30 2024-04-12T05:27:48.197 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.398+0000 7f231a7fc700 1 -- 172.21.15.32:0/3897546570 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f231c0245b0 con 0x7f2328076c30 2024-04-12T05:27:48.197 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.398+0000 7f233007f700 1 -- 172.21.15.32:0/3897546570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2328112600 con 0x7f2328076c30 2024-04-12T05:27:48.198 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.399+0000 7f231a7fc700 1 -- 172.21.15.32:0/3897546570 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f231c0247e0 con 0x7f2328076c30 2024-04-12T05:27:48.198 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.399+0000 7f231a7fc700 1 --2- 172.21.15.32:0/3897546570 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f23140367e0 0x7f2314038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:48.198 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.399+0000 7f231a7fc700 1 -- 172.21.15.32:0/3897546570 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f231c04ad30 con 0x7f2328076c30 2024-04-12T05:27:48.198 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.399+0000 7f232d61a700 1 --2- 172.21.15.32:0/3897546570 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f23140367e0 0x7f2314038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:48.198 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.399+0000 7f233007f700 1 -- 172.21.15.32:0/3897546570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f232805f5f0 con 0x7f2328076c30 2024-04-12T05:27:48.198 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.399+0000 7f232d61a700 1 --2- 172.21.15.32:0/3897546570 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f23140367e0 0x7f2314038c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2324009990 tx=0x7f2324006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:48.198 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.404+0000 7f231a7fc700 1 -- 172.21.15.32:0/3897546570 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f231c024b30 con 0x7f2328076c30 2024-04-12T05:27:48.198 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.599+0000 7f233007f700 1 -- 172.21.15.32:0/3897546570 --> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f23280fdee0 con 0x7f23140367e0 2024-04-12T05:27:48.198 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.600+0000 7f231a7fc700 1 -- 172.21.15.32:0/3897546570 <== mgr.14118 v2:172.21.15.32:6800/1962278696 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f23280fdee0 con 0x7f23140367e0 2024-04-12T05:27:48.198 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.602+0000 7f233007f700 1 -- 172.21.15.32:0/3897546570 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f23140367e0 msgr2=0x7f2314038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:48.198 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.602+0000 7f233007f700 1 --2- 172.21.15.32:0/3897546570 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f23140367e0 0x7f2314038c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2324009990 tx=0x7f2324006e40 comp rx=0 tx=0).stop 2024-04-12T05:27:48.198 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.602+0000 7f233007f700 1 -- 172.21.15.32:0/3897546570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2328076c30 msgr2=0x7f2328111350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:48.198 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.602+0000 7f233007f700 1 --2- 172.21.15.32:0/3897546570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2328076c30 0x7f2328111350 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f231c006b00 tx=0x7f231c009250 comp rx=0 tx=0).stop 2024-04-12T05:27:48.198 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.602+0000 7f233007f700 1 -- 172.21.15.32:0/3897546570 shutdown_connections 2024-04-12T05:27:48.199 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.602+0000 7f233007f700 1 --2- 172.21.15.32:0/3897546570 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f23140367e0 0x7f2314038c90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:48.199 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.602+0000 7f233007f700 1 --2- 172.21.15.32:0/3897546570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2328076c30 0x7f2328111350 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:48.199 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.602+0000 7f233007f700 1 -- 172.21.15.32:0/3897546570 >> 172.21.15.32:0/3897546570 conn(0x7f23280fcc70 msgr2=0x7f23280fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:48.199 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.602+0000 7f233007f700 1 -- 172.21.15.32:0/3897546570 shutdown_connections 2024-04-12T05:27:48.199 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:47.602+0000 7f233007f700 1 -- 172.21.15.32:0/3897546570 wait complete. 2024-04-12T05:27:48.199 INFO:teuthology.orchestra.run.smithi032.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-12T05:27:48.199 INFO:teuthology.orchestra.run.smithi032.stdout:Adding key to root@localhost authorized_keys... 2024-04-12T05:27:48.199 INFO:teuthology.orchestra.run.smithi032.stdout:Adding host smithi032... 2024-04-12T05:27:48.472 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:48 smithi032 ceph-mon[39731]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:27:49.494 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:49 smithi032 ceph-mon[39731]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi032", "addr": "172.21.15.32", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:27:50.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:50 smithi032 ceph-mon[39731]: Deploying cephadm binary to smithi032 2024-04-12T05:27:51.371 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Added host 'smithi032' with addr '172.21.15.32' 2024-04-12T05:27:51.371 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.738+0000 7f1ec8564700 1 Processor -- start 2024-04-12T05:27:51.372 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.738+0000 7f1ec8564700 1 -- start start 2024-04-12T05:27:51.372 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.738+0000 7f1ec8564700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ec0102b00 0x7f1ec0102f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:51.372 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.738+0000 7f1ec8564700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec0103450 con 0x7f1ec0102b00 2024-04-12T05:27:51.372 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.739+0000 7f1ec6300700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ec0102b00 0x7f1ec0102f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:51.372 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.739+0000 7f1ec6300700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ec0102b00 0x7f1ec0102f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55096/0 (socket says 172.21.15.32:55096) 2024-04-12T05:27:51.372 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.739+0000 7f1ec6300700 1 -- 172.21.15.32:0/2103882417 learned_addr learned my addr 172.21.15.32:0/2103882417 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:51.372 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.739+0000 7f1ec6300700 1 -- 172.21.15.32:0/2103882417 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec0103590 con 0x7f1ec0102b00 2024-04-12T05:27:51.372 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.739+0000 7f1ec6300700 1 --2- 172.21.15.32:0/2103882417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ec0102b00 0x7f1ec0102f10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1eb00098e0 tx=0x7f1eb0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=197ff35358806c72 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:51.372 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.739+0000 7f1ec52fe700 1 -- 172.21.15.32:0/2103882417 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1eb000b860 con 0x7f1ec0102b00 2024-04-12T05:27:51.372 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.739+0000 7f1ec52fe700 1 -- 172.21.15.32:0/2103882417 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f1eb000be30 con 0x7f1ec0102b00 2024-04-12T05:27:51.372 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.739+0000 7f1ec52fe700 1 -- 172.21.15.32:0/2103882417 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1eb0010380 con 0x7f1ec0102b00 2024-04-12T05:27:51.373 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.740+0000 7f1ec8564700 1 -- 172.21.15.32:0/2103882417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ec0102b00 msgr2=0x7f1ec0102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:51.373 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.740+0000 7f1ec8564700 1 --2- 172.21.15.32:0/2103882417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ec0102b00 0x7f1ec0102f10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1eb00098e0 tx=0x7f1eb0009250 comp rx=0 tx=0).stop 2024-04-12T05:27:51.373 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.740+0000 7f1ec8564700 1 -- 172.21.15.32:0/2103882417 shutdown_connections 2024-04-12T05:27:51.373 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.740+0000 7f1ec8564700 1 --2- 172.21.15.32:0/2103882417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ec0102b00 0x7f1ec0102f10 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:51.373 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.740+0000 7f1ec8564700 1 -- 172.21.15.32:0/2103882417 >> 172.21.15.32:0/2103882417 conn(0x7f1ec00fc2b0 msgr2=0x7f1ec00fe6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:51.373 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.740+0000 7f1ec8564700 1 -- 172.21.15.32:0/2103882417 shutdown_connections 2024-04-12T05:27:51.373 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.740+0000 7f1ec8564700 1 -- 172.21.15.32:0/2103882417 wait complete. 2024-04-12T05:27:51.373 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.740+0000 7f1ec8564700 1 Processor -- start 2024-04-12T05:27:51.373 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.740+0000 7f1ec8564700 1 -- start start 2024-04-12T05:27:51.373 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.741+0000 7f1ec8564700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ec0102b00 0x7f1ec0113530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:51.374 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.741+0000 7f1ec8564700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec0113a70 con 0x7f1ec0102b00 2024-04-12T05:27:51.374 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.741+0000 7f1ec6300700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ec0102b00 0x7f1ec0113530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:51.374 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.741+0000 7f1ec6300700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ec0102b00 0x7f1ec0113530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55100/0 (socket says 172.21.15.32:55100) 2024-04-12T05:27:51.374 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.741+0000 7f1ec6300700 1 -- 172.21.15.32:0/624776633 learned_addr learned my addr 172.21.15.32:0/624776633 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:51.374 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.741+0000 7f1ec6300700 1 -- 172.21.15.32:0/624776633 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb00094f0 con 0x7f1ec0102b00 2024-04-12T05:27:51.374 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.741+0000 7f1ec6300700 1 --2- 172.21.15.32:0/624776633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ec0102b00 0x7f1ec0113530 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0000c00 tx=0x7f1eb0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:51.374 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.741+0000 7f1eb6ffd700 1 -- 172.21.15.32:0/624776633 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1eb0010a80 con 0x7f1ec0102b00 2024-04-12T05:27:51.374 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.741+0000 7f1eb6ffd700 1 -- 172.21.15.32:0/624776633 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f1eb001a430 con 0x7f1ec0102b00 2024-04-12T05:27:51.375 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.741+0000 7f1ec8564700 1 -- 172.21.15.32:0/624776633 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ec0113c70 con 0x7f1ec0102b00 2024-04-12T05:27:51.375 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.741+0000 7f1eb6ffd700 1 -- 172.21.15.32:0/624776633 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1eb00245b0 con 0x7f1ec0102b00 2024-04-12T05:27:51.375 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.741+0000 7f1ec8564700 1 -- 172.21.15.32:0/624776633 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ec01147e0 con 0x7f1ec0102b00 2024-04-12T05:27:51.375 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.742+0000 7f1eb6ffd700 1 -- 172.21.15.32:0/624776633 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f1eb00247a0 con 0x7f1ec0102b00 2024-04-12T05:27:51.375 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.742+0000 7f1ec8564700 1 -- 172.21.15.32:0/624776633 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ec005f5f0 con 0x7f1ec0102b00 2024-04-12T05:27:51.375 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.742+0000 7f1eb6ffd700 1 --2- 172.21.15.32:0/624776633 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f1eac0367e0 0x7f1eac038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:51.375 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.743+0000 7f1eb6ffd700 1 -- 172.21.15.32:0/624776633 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f1eb004af90 con 0x7f1ec0102b00 2024-04-12T05:27:51.375 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.743+0000 7f1ec5aff700 1 --2- 172.21.15.32:0/624776633 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f1eac0367e0 0x7f1eac038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:51.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.743+0000 7f1ec5aff700 1 --2- 172.21.15.32:0/624776633 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f1eac0367e0 0x7f1eac038c90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc009990 tx=0x7f1ebc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:51.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.747+0000 7f1eb6ffd700 1 -- 172.21.15.32:0/624776633 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f1eb0023d60 con 0x7f1ec0102b00 2024-04-12T05:27:51.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:48.945+0000 7f1ec8564700 1 -- 172.21.15.32:0/624776633 --> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi032", "addr": "172.21.15.32", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ec00fd560 con 0x7f1eac0367e0 2024-04-12T05:27:51.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:50.679+0000 7f1eb6ffd700 1 -- 172.21.15.32:0/624776633 <== mgr.14118 v2:172.21.15.32:6800/1962278696 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f1ec00fd560 con 0x7f1eac0367e0 2024-04-12T05:27:51.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:50.681+0000 7f1ec8564700 1 -- 172.21.15.32:0/624776633 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f1eac0367e0 msgr2=0x7f1eac038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:51.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:50.681+0000 7f1ec8564700 1 --2- 172.21.15.32:0/624776633 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f1eac0367e0 0x7f1eac038c90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc009990 tx=0x7f1ebc006e40 comp rx=0 tx=0).stop 2024-04-12T05:27:51.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:50.681+0000 7f1ec8564700 1 -- 172.21.15.32:0/624776633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ec0102b00 msgr2=0x7f1ec0113530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:51.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:50.681+0000 7f1ec8564700 1 --2- 172.21.15.32:0/624776633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ec0102b00 0x7f1ec0113530 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0000c00 tx=0x7f1eb0009250 comp rx=0 tx=0).stop 2024-04-12T05:27:51.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:50.681+0000 7f1ec8564700 1 -- 172.21.15.32:0/624776633 shutdown_connections 2024-04-12T05:27:51.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:50.681+0000 7f1ec8564700 1 --2- 172.21.15.32:0/624776633 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f1eac0367e0 0x7f1eac038c90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:51.376 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:50.681+0000 7f1ec8564700 1 --2- 172.21.15.32:0/624776633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ec0102b00 0x7f1ec0113530 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:51.377 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:50.681+0000 7f1ec8564700 1 -- 172.21.15.32:0/624776633 >> 172.21.15.32:0/624776633 conn(0x7f1ec00fc2b0 msgr2=0x7f1ec00fce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:51.377 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:50.681+0000 7f1ec8564700 1 -- 172.21.15.32:0/624776633 shutdown_connections 2024-04-12T05:27:51.377 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:50.681+0000 7f1ec8564700 1 -- 172.21.15.32:0/624776633 wait complete. 2024-04-12T05:27:51.377 INFO:teuthology.orchestra.run.smithi032.stdout:Deploying mon service with default placement... 2024-04-12T05:27:52.094 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:51 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:27:52.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:51 smithi032 ceph-mon[39731]: Added host smithi032 2024-04-12T05:27:52.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:51 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:27:52.705 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-12T05:27:52.705 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.920+0000 7f53e7195700 1 Processor -- start 2024-04-12T05:27:52.705 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.920+0000 7f53e7195700 1 -- start start 2024-04-12T05:27:52.706 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.920+0000 7f53e7195700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53e0078d30 0x7f53e0077240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:52.706 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.920+0000 7f53e7195700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e0079140 con 0x7f53e0078d30 2024-04-12T05:27:52.706 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.920+0000 7f53e4f31700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53e0078d30 0x7f53e0077240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:52.706 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.920+0000 7f53e4f31700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53e0078d30 0x7f53e0077240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47018/0 (socket says 172.21.15.32:47018) 2024-04-12T05:27:52.706 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.920+0000 7f53e4f31700 1 -- 172.21.15.32:0/3972624262 learned_addr learned my addr 172.21.15.32:0/3972624262 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:52.706 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.921+0000 7f53e4f31700 1 -- 172.21.15.32:0/3972624262 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e0077780 con 0x7f53e0078d30 2024-04-12T05:27:52.706 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.921+0000 7f53e4f31700 1 --2- 172.21.15.32:0/3972624262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53e0078d30 0x7f53e0077240 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f53d40098e0 tx=0x7f53d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f667040ff1024d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:52.706 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.921+0000 7f53df7fe700 1 -- 172.21.15.32:0/3972624262 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53d400b860 con 0x7f53e0078d30 2024-04-12T05:27:52.706 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.921+0000 7f53df7fe700 1 -- 172.21.15.32:0/3972624262 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f53d400be30 con 0x7f53e0078d30 2024-04-12T05:27:52.706 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.921+0000 7f53df7fe700 1 -- 172.21.15.32:0/3972624262 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53d4010380 con 0x7f53e0078d30 2024-04-12T05:27:52.706 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.921+0000 7f53e7195700 1 -- 172.21.15.32:0/3972624262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53e0078d30 msgr2=0x7f53e0077240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:52.706 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.922+0000 7f53e7195700 1 --2- 172.21.15.32:0/3972624262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53e0078d30 0x7f53e0077240 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f53d40098e0 tx=0x7f53d4009250 comp rx=0 tx=0).stop 2024-04-12T05:27:52.706 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.922+0000 7f53e7195700 1 -- 172.21.15.32:0/3972624262 shutdown_connections 2024-04-12T05:27:52.706 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.922+0000 7f53e7195700 1 --2- 172.21.15.32:0/3972624262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53e0078d30 0x7f53e0077240 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:52.707 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.922+0000 7f53e7195700 1 -- 172.21.15.32:0/3972624262 >> 172.21.15.32:0/3972624262 conn(0x7f53e00fdd00 msgr2=0x7f53e0100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:52.707 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.922+0000 7f53e7195700 1 -- 172.21.15.32:0/3972624262 shutdown_connections 2024-04-12T05:27:52.707 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.922+0000 7f53e7195700 1 -- 172.21.15.32:0/3972624262 wait complete. 2024-04-12T05:27:52.707 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.922+0000 7f53e7195700 1 Processor -- start 2024-04-12T05:27:52.707 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.922+0000 7f53e7195700 1 -- start start 2024-04-12T05:27:52.707 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.923+0000 7f53e7195700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53e0078d30 0x7f53e01135b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:52.707 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.923+0000 7f53e7195700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e0113af0 con 0x7f53e0078d30 2024-04-12T05:27:52.707 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.923+0000 7f53e4f31700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53e0078d30 0x7f53e01135b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:52.707 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.923+0000 7f53e4f31700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53e0078d30 0x7f53e01135b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47028/0 (socket says 172.21.15.32:47028) 2024-04-12T05:27:52.707 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.923+0000 7f53e4f31700 1 -- 172.21.15.32:0/2392804156 learned_addr learned my addr 172.21.15.32:0/2392804156 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:52.707 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.923+0000 7f53e4f31700 1 -- 172.21.15.32:0/2392804156 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d40094f0 con 0x7f53e0078d30 2024-04-12T05:27:52.707 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.923+0000 7f53e4f31700 1 --2- 172.21.15.32:0/2392804156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53e0078d30 0x7f53e01135b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f53d4006fd0 tx=0x7f53d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:52.707 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.923+0000 7f53dd7fa700 1 -- 172.21.15.32:0/2392804156 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53d4010750 con 0x7f53e0078d30 2024-04-12T05:27:52.707 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.923+0000 7f53dd7fa700 1 -- 172.21.15.32:0/2392804156 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f53d4010d20 con 0x7f53e0078d30 2024-04-12T05:27:52.707 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.923+0000 7f53e7195700 1 -- 172.21.15.32:0/2392804156 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53e0113cf0 con 0x7f53e0078d30 2024-04-12T05:27:52.708 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.923+0000 7f53dd7fa700 1 -- 172.21.15.32:0/2392804156 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53d401ada0 con 0x7f53e0078d30 2024-04-12T05:27:52.708 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.923+0000 7f53e7195700 1 -- 172.21.15.32:0/2392804156 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53e0114860 con 0x7f53e0078d30 2024-04-12T05:27:52.708 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.924+0000 7f53dd7fa700 1 -- 172.21.15.32:0/2392804156 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f53d40108b0 con 0x7f53e0078d30 2024-04-12T05:27:52.708 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.924+0000 7f53dd7fa700 1 --2- 172.21.15.32:0/2392804156 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f53d0036790 0x7f53d0038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:52.708 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.924+0000 7f53e7195700 1 -- 172.21.15.32:0/2392804156 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53e005f5f0 con 0x7f53e0078d30 2024-04-12T05:27:52.708 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.924+0000 7f53dd7fa700 1 -- 172.21.15.32:0/2392804156 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f53d4049d80 con 0x7f53e0078d30 2024-04-12T05:27:52.708 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.925+0000 7f53dffff700 1 --2- 172.21.15.32:0/2392804156 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f53d0036790 0x7f53d0038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:52.708 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.925+0000 7f53dffff700 1 --2- 172.21.15.32:0/2392804156 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f53d0036790 0x7f53d0038c40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f53cc009990 tx=0x7f53cc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:52.708 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:51.929+0000 7f53dd7fa700 1 -- 172.21.15.32:0/2392804156 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f53d4020070 con 0x7f53e0078d30 2024-04-12T05:27:52.708 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:52.137+0000 7f53e7195700 1 -- 172.21.15.32:0/2392804156 --> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f53e00fef50 con 0x7f53d0036790 2024-04-12T05:27:52.708 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:52.142+0000 7f53dd7fa700 1 -- 172.21.15.32:0/2392804156 <== mgr.14118 v2:172.21.15.32:6800/1962278696 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f53e00fef50 con 0x7f53d0036790 2024-04-12T05:27:52.708 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:52.144+0000 7f53e7195700 1 -- 172.21.15.32:0/2392804156 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f53d0036790 msgr2=0x7f53d0038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:52.708 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:52.144+0000 7f53e7195700 1 --2- 172.21.15.32:0/2392804156 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f53d0036790 0x7f53d0038c40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f53cc009990 tx=0x7f53cc006e40 comp rx=0 tx=0).stop 2024-04-12T05:27:52.708 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:52.144+0000 7f53e7195700 1 -- 172.21.15.32:0/2392804156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53e0078d30 msgr2=0x7f53e01135b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:52.708 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:52.144+0000 7f53e7195700 1 --2- 172.21.15.32:0/2392804156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53e0078d30 0x7f53e01135b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f53d4006fd0 tx=0x7f53d4009250 comp rx=0 tx=0).stop 2024-04-12T05:27:52.709 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:52.144+0000 7f53e7195700 1 -- 172.21.15.32:0/2392804156 shutdown_connections 2024-04-12T05:27:52.709 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:52.144+0000 7f53e7195700 1 --2- 172.21.15.32:0/2392804156 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f53d0036790 0x7f53d0038c40 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:52.709 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:52.144+0000 7f53e7195700 1 --2- 172.21.15.32:0/2392804156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53e0078d30 0x7f53e01135b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:52.709 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:52.144+0000 7f53e7195700 1 -- 172.21.15.32:0/2392804156 >> 172.21.15.32:0/2392804156 conn(0x7f53e00fdd00 msgr2=0x7f53e00fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:52.709 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:52.144+0000 7f53e7195700 1 -- 172.21.15.32:0/2392804156 shutdown_connections 2024-04-12T05:27:52.709 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:52.144+0000 7f53e7195700 1 -- 172.21.15.32:0/2392804156 wait complete. 2024-04-12T05:27:52.709 INFO:teuthology.orchestra.run.smithi032.stdout:Deploying mgr service with default placement... 2024-04-12T05:27:53.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:53 smithi032 ceph-mon[39731]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:27:53.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:53 smithi032 ceph-mon[39731]: Saving service mon spec with placement count:5 2024-04-12T05:27:53.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:53 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:27:54.592 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-12T05:27:54.592 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.513+0000 7f128322c700 1 Processor -- start 2024-04-12T05:27:54.593 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.513+0000 7f128322c700 1 -- start start 2024-04-12T05:27:54.593 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.513+0000 7f128322c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f127c076be0 0x7f127c107650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:54.593 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.513+0000 7f128322c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127c103040 con 0x7f127c076be0 2024-04-12T05:27:54.593 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.514+0000 7f1280fc8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f127c076be0 0x7f127c107650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:54.593 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.514+0000 7f1280fc8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f127c076be0 0x7f127c107650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47044/0 (socket says 172.21.15.32:47044) 2024-04-12T05:27:54.593 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.514+0000 7f1280fc8700 1 -- 172.21.15.32:0/345541561 learned_addr learned my addr 172.21.15.32:0/345541561 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:54.593 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.514+0000 7f1280fc8700 1 -- 172.21.15.32:0/345541561 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f127c107b90 con 0x7f127c076be0 2024-04-12T05:27:54.593 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.514+0000 7f1280fc8700 1 --2- 172.21.15.32:0/345541561 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f127c076be0 0x7f127c107650 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f12700098e0 tx=0x7f1270009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=213d4868f7c50d19 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:54.593 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.514+0000 7f127b7fe700 1 -- 172.21.15.32:0/345541561 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f127000b860 con 0x7f127c076be0 2024-04-12T05:27:54.593 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.514+0000 7f127b7fe700 1 -- 172.21.15.32:0/345541561 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f127000be30 con 0x7f127c076be0 2024-04-12T05:27:54.593 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.514+0000 7f127b7fe700 1 -- 172.21.15.32:0/345541561 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1270010380 con 0x7f127c076be0 2024-04-12T05:27:54.593 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.515+0000 7f128322c700 1 -- 172.21.15.32:0/345541561 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f127c076be0 msgr2=0x7f127c107650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:54.594 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.515+0000 7f128322c700 1 --2- 172.21.15.32:0/345541561 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f127c076be0 0x7f127c107650 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f12700098e0 tx=0x7f1270009250 comp rx=0 tx=0).stop 2024-04-12T05:27:54.594 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.515+0000 7f128322c700 1 -- 172.21.15.32:0/345541561 shutdown_connections 2024-04-12T05:27:54.594 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.515+0000 7f128322c700 1 --2- 172.21.15.32:0/345541561 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f127c076be0 0x7f127c107650 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:54.594 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.515+0000 7f128322c700 1 -- 172.21.15.32:0/345541561 >> 172.21.15.32:0/345541561 conn(0x7f127c0fcc60 msgr2=0x7f127c0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:54.594 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.515+0000 7f128322c700 1 -- 172.21.15.32:0/345541561 shutdown_connections 2024-04-12T05:27:54.594 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.515+0000 7f128322c700 1 -- 172.21.15.32:0/345541561 wait complete. 2024-04-12T05:27:54.594 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.515+0000 7f128322c700 1 Processor -- start 2024-04-12T05:27:54.594 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.516+0000 7f128322c700 1 -- start start 2024-04-12T05:27:54.594 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.516+0000 7f128322c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f127c076be0 0x7f127c113560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:54.594 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.516+0000 7f128322c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127c113aa0 con 0x7f127c076be0 2024-04-12T05:27:54.594 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.516+0000 7f1280fc8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f127c076be0 0x7f127c113560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:54.594 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.516+0000 7f1280fc8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f127c076be0 0x7f127c113560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47050/0 (socket says 172.21.15.32:47050) 2024-04-12T05:27:54.594 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.516+0000 7f1280fc8700 1 -- 172.21.15.32:0/2190366937 learned_addr learned my addr 172.21.15.32:0/2190366937 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:54.594 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.516+0000 7f1280fc8700 1 -- 172.21.15.32:0/2190366937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12700094f0 con 0x7f127c076be0 2024-04-12T05:27:54.595 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.516+0000 7f1280fc8700 1 --2- 172.21.15.32:0/2190366937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f127c076be0 0x7f127c113560 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1270006d40 tx=0x7f1270009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:54.595 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.517+0000 7f12797fa700 1 -- 172.21.15.32:0/2190366937 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12700109b0 con 0x7f127c076be0 2024-04-12T05:27:54.595 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.517+0000 7f128322c700 1 -- 172.21.15.32:0/2190366937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f127c113ca0 con 0x7f127c076be0 2024-04-12T05:27:54.595 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.517+0000 7f12797fa700 1 -- 172.21.15.32:0/2190366937 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f1270024440 con 0x7f127c076be0 2024-04-12T05:27:54.595 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.517+0000 7f12797fa700 1 -- 172.21.15.32:0/2190366937 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f127001a5f0 con 0x7f127c076be0 2024-04-12T05:27:54.595 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.517+0000 7f128322c700 1 -- 172.21.15.32:0/2190366937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f127c114810 con 0x7f127c076be0 2024-04-12T05:27:54.595 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.517+0000 7f12797fa700 1 -- 172.21.15.32:0/2190366937 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f127001a870 con 0x7f127c076be0 2024-04-12T05:27:54.595 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.518+0000 7f128322c700 1 -- 172.21.15.32:0/2190366937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f127c05f5f0 con 0x7f127c076be0 2024-04-12T05:27:54.595 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.518+0000 7f12797fa700 1 --2- 172.21.15.32:0/2190366937 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f126c036790 0x7f126c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:54.595 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.518+0000 7f12797fa700 1 -- 172.21.15.32:0/2190366937 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f127004a6e0 con 0x7f127c076be0 2024-04-12T05:27:54.595 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.518+0000 7f127bfff700 1 --2- 172.21.15.32:0/2190366937 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f126c036790 0x7f126c038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:54.596 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.518+0000 7f127bfff700 1 --2- 172.21.15.32:0/2190366937 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f126c036790 0x7f126c038c40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1268009990 tx=0x7f1268006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:54.596 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.522+0000 7f12797fa700 1 -- 172.21.15.32:0/2190366937 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f1270018b80 con 0x7f127c076be0 2024-04-12T05:27:54.596 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.732+0000 7f128322c700 1 -- 172.21.15.32:0/2190366937 --> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f127c0fde50 con 0x7f126c036790 2024-04-12T05:27:54.596 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.736+0000 7f12797fa700 1 -- 172.21.15.32:0/2190366937 <== mgr.14118 v2:172.21.15.32:6800/1962278696 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f127c0fde50 con 0x7f126c036790 2024-04-12T05:27:54.596 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.738+0000 7f128322c700 1 -- 172.21.15.32:0/2190366937 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f126c036790 msgr2=0x7f126c038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:54.596 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.738+0000 7f128322c700 1 --2- 172.21.15.32:0/2190366937 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f126c036790 0x7f126c038c40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1268009990 tx=0x7f1268006e40 comp rx=0 tx=0).stop 2024-04-12T05:27:54.596 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.738+0000 7f128322c700 1 -- 172.21.15.32:0/2190366937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f127c076be0 msgr2=0x7f127c113560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:54.596 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.738+0000 7f128322c700 1 --2- 172.21.15.32:0/2190366937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f127c076be0 0x7f127c113560 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1270006d40 tx=0x7f1270009250 comp rx=0 tx=0).stop 2024-04-12T05:27:54.596 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.738+0000 7f128322c700 1 -- 172.21.15.32:0/2190366937 shutdown_connections 2024-04-12T05:27:54.596 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.738+0000 7f128322c700 1 --2- 172.21.15.32:0/2190366937 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f126c036790 0x7f126c038c40 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:54.596 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.738+0000 7f128322c700 1 --2- 172.21.15.32:0/2190366937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f127c076be0 0x7f127c113560 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:54.596 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.738+0000 7f128322c700 1 -- 172.21.15.32:0/2190366937 >> 172.21.15.32:0/2190366937 conn(0x7f127c0fcc60 msgr2=0x7f127c0fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:54.596 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.738+0000 7f128322c700 1 -- 172.21.15.32:0/2190366937 shutdown_connections 2024-04-12T05:27:54.597 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:53.738+0000 7f128322c700 1 -- 172.21.15.32:0/2190366937 wait complete. 2024-04-12T05:27:54.597 INFO:teuthology.orchestra.run.smithi032.stdout:Deploying crash service with default placement... 2024-04-12T05:27:54.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:54 smithi032 ceph-mon[39731]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:27:54.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:54 smithi032 ceph-mon[39731]: Saving service mgr spec with placement count:2 2024-04-12T05:27:54.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:54 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:27:54.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:54 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:27:54.845 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:54 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:27:56.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:56 smithi032 ceph-mon[39731]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:27:56.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:56 smithi032 ceph-mon[39731]: Saving service crash spec with placement * 2024-04-12T05:27:56.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:56 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:27:56.595 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:56 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:27:57.488 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-04-12T05:27:57.488 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.097+0000 7f46a9129700 1 Processor -- start 2024-04-12T05:27:57.488 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.097+0000 7f46a9129700 1 -- start start 2024-04-12T05:27:57.488 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.097+0000 7f46a9129700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a4076c80 0x7f46a41051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:57.489 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.097+0000 7f46a9129700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a41056f0 con 0x7f46a4076c80 2024-04-12T05:27:57.489 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.097+0000 7f46a2d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a4076c80 0x7f46a41051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:57.489 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.097+0000 7f46a2d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a4076c80 0x7f46a41051b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47054/0 (socket says 172.21.15.32:47054) 2024-04-12T05:27:57.489 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.097+0000 7f46a2d9d700 1 -- 172.21.15.32:0/4011105618 learned_addr learned my addr 172.21.15.32:0/4011105618 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:57.489 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.097+0000 7f46a2d9d700 1 -- 172.21.15.32:0/4011105618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a4105830 con 0x7f46a4076c80 2024-04-12T05:27:57.489 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.097+0000 7f46a2d9d700 1 --2- 172.21.15.32:0/4011105618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a4076c80 0x7f46a41051b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f46940098e0 tx=0x7f4694009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a6da8bb410b1576 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:57.489 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.098+0000 7f46a1d9b700 1 -- 172.21.15.32:0/4011105618 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f469400b860 con 0x7f46a4076c80 2024-04-12T05:27:57.489 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.098+0000 7f46a1d9b700 1 -- 172.21.15.32:0/4011105618 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f469400be70 con 0x7f46a4076c80 2024-04-12T05:27:57.489 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.098+0000 7f46a1d9b700 1 -- 172.21.15.32:0/4011105618 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4694010380 con 0x7f46a4076c80 2024-04-12T05:27:57.489 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.098+0000 7f46a9129700 1 -- 172.21.15.32:0/4011105618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a4076c80 msgr2=0x7f46a41051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:57.489 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.098+0000 7f46a9129700 1 --2- 172.21.15.32:0/4011105618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a4076c80 0x7f46a41051b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f46940098e0 tx=0x7f4694009250 comp rx=0 tx=0).stop 2024-04-12T05:27:57.489 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.098+0000 7f46a9129700 1 -- 172.21.15.32:0/4011105618 shutdown_connections 2024-04-12T05:27:57.489 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.098+0000 7f46a9129700 1 --2- 172.21.15.32:0/4011105618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a4076c80 0x7f46a41051b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:57.489 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.098+0000 7f46a9129700 1 -- 172.21.15.32:0/4011105618 >> 172.21.15.32:0/4011105618 conn(0x7f46a40fcc60 msgr2=0x7f46a40ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:57.490 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.098+0000 7f46a9129700 1 -- 172.21.15.32:0/4011105618 shutdown_connections 2024-04-12T05:27:57.490 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.098+0000 7f46a9129700 1 -- 172.21.15.32:0/4011105618 wait complete. 2024-04-12T05:27:57.490 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.099+0000 7f46a9129700 1 Processor -- start 2024-04-12T05:27:57.490 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.099+0000 7f46a9129700 1 -- start start 2024-04-12T05:27:57.490 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.099+0000 7f46a9129700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a4076c80 0x7f46a41113b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:57.490 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.099+0000 7f46a9129700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a41118f0 con 0x7f46a4076c80 2024-04-12T05:27:57.492 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.099+0000 7f46a2d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a4076c80 0x7f46a41113b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:57.492 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.099+0000 7f46a2d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a4076c80 0x7f46a41113b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47066/0 (socket says 172.21.15.32:47066) 2024-04-12T05:27:57.492 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.099+0000 7f46a2d9d700 1 -- 172.21.15.32:0/732122451 learned_addr learned my addr 172.21.15.32:0/732122451 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:57.492 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.099+0000 7f46a2d9d700 1 -- 172.21.15.32:0/732122451 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46940094f0 con 0x7f46a4076c80 2024-04-12T05:27:57.492 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.100+0000 7f46a2d9d700 1 --2- 172.21.15.32:0/732122451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a4076c80 0x7f46a41113b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f46940094c0 tx=0x7f4694009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:57.493 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.100+0000 7f468b7fe700 1 -- 172.21.15.32:0/732122451 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4694010850 con 0x7f46a4076c80 2024-04-12T05:27:57.493 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.100+0000 7f468b7fe700 1 -- 172.21.15.32:0/732122451 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4694010e60 con 0x7f46a4076c80 2024-04-12T05:27:57.493 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.100+0000 7f46a9129700 1 -- 172.21.15.32:0/732122451 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46a4111af0 con 0x7f46a4076c80 2024-04-12T05:27:57.493 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.100+0000 7f468b7fe700 1 -- 172.21.15.32:0/732122451 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f469401a430 con 0x7f46a4076c80 2024-04-12T05:27:57.493 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.100+0000 7f46a9129700 1 -- 172.21.15.32:0/732122451 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46a4112660 con 0x7f46a4076c80 2024-04-12T05:27:57.493 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.101+0000 7f468b7fe700 1 -- 172.21.15.32:0/732122451 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f469401a6b0 con 0x7f46a4076c80 2024-04-12T05:27:57.493 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.101+0000 7f46a9129700 1 -- 172.21.15.32:0/732122451 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46a405f5f0 con 0x7f46a4076c80 2024-04-12T05:27:57.493 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.101+0000 7f468b7fe700 1 --2- 172.21.15.32:0/732122451 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f468c036790 0x7f468c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:57.493 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.101+0000 7f468b7fe700 1 -- 172.21.15.32:0/732122451 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f469404a740 con 0x7f46a4076c80 2024-04-12T05:27:57.493 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.101+0000 7f46a259c700 1 --2- 172.21.15.32:0/732122451 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f468c036790 0x7f468c038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:57.493 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.101+0000 7f46a259c700 1 --2- 172.21.15.32:0/732122451 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f468c036790 0x7f468c038c40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4698009990 tx=0x7f4698006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:57.493 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.106+0000 7f468b7fe700 1 -- 172.21.15.32:0/732122451 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4694018710 con 0x7f46a4076c80 2024-04-12T05:27:57.493 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.321+0000 7f46a9129700 1 -- 172.21.15.32:0/732122451 --> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f46a40fde50 con 0x7f468c036790 2024-04-12T05:27:57.493 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.325+0000 7f468b7fe700 1 -- 172.21.15.32:0/732122451 <== mgr.14118 v2:172.21.15.32:6800/1962278696 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f46a40fde50 con 0x7f468c036790 2024-04-12T05:27:57.493 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.327+0000 7f46a9129700 1 -- 172.21.15.32:0/732122451 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f468c036790 msgr2=0x7f468c038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:57.494 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.327+0000 7f46a9129700 1 --2- 172.21.15.32:0/732122451 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f468c036790 0x7f468c038c40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4698009990 tx=0x7f4698006e40 comp rx=0 tx=0).stop 2024-04-12T05:27:57.494 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.327+0000 7f46a9129700 1 -- 172.21.15.32:0/732122451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a4076c80 msgr2=0x7f46a41113b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:57.494 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.327+0000 7f46a9129700 1 --2- 172.21.15.32:0/732122451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a4076c80 0x7f46a41113b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f46940094c0 tx=0x7f4694009250 comp rx=0 tx=0).stop 2024-04-12T05:27:57.494 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.327+0000 7f46a9129700 1 -- 172.21.15.32:0/732122451 shutdown_connections 2024-04-12T05:27:57.494 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.327+0000 7f46a9129700 1 --2- 172.21.15.32:0/732122451 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f468c036790 0x7f468c038c40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:57.494 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.327+0000 7f46a9129700 1 --2- 172.21.15.32:0/732122451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46a4076c80 0x7f46a41113b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:57.494 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.327+0000 7f46a9129700 1 -- 172.21.15.32:0/732122451 >> 172.21.15.32:0/732122451 conn(0x7f46a40fcc60 msgr2=0x7f46a40fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:57.494 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.327+0000 7f46a9129700 1 -- 172.21.15.32:0/732122451 shutdown_connections 2024-04-12T05:27:57.494 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:55.327+0000 7f46a9129700 1 -- 172.21.15.32:0/732122451 wait complete. 2024-04-12T05:27:57.494 INFO:teuthology.orchestra.run.smithi032.stdout:Deploying prometheus service with default placement... 2024-04-12T05:27:58.827 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-04-12T05:27:58.827 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.036+0000 7f91fe084700 1 Processor -- start 2024-04-12T05:27:58.827 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.036+0000 7f91fe084700 1 -- start start 2024-04-12T05:27:58.827 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.036+0000 7f91fe084700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f8076c80 0x7f91f81051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:58.827 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.036+0000 7f91fe084700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f81056f0 con 0x7f91f8076c80 2024-04-12T05:27:58.827 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.036+0000 7f91f77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f8076c80 0x7f91f81051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:58.827 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.036+0000 7f91f77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f8076c80 0x7f91f81051b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47080/0 (socket says 172.21.15.32:47080) 2024-04-12T05:27:58.827 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.036+0000 7f91f77fe700 1 -- 172.21.15.32:0/1290501115 learned_addr learned my addr 172.21.15.32:0/1290501115 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:58.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.037+0000 7f91f77fe700 1 -- 172.21.15.32:0/1290501115 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f8105830 con 0x7f91f8076c80 2024-04-12T05:27:58.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.037+0000 7f91f77fe700 1 --2- 172.21.15.32:0/1290501115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f8076c80 0x7f91f81051b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f91e80098e0 tx=0x7f91e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff4f6ab8f0955c7f server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:58.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.037+0000 7f91f67fc700 1 -- 172.21.15.32:0/1290501115 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91e800b860 con 0x7f91f8076c80 2024-04-12T05:27:58.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.037+0000 7f91f67fc700 1 -- 172.21.15.32:0/1290501115 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f91e800be70 con 0x7f91f8076c80 2024-04-12T05:27:58.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.037+0000 7f91f67fc700 1 -- 172.21.15.32:0/1290501115 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91e8010380 con 0x7f91f8076c80 2024-04-12T05:27:58.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.038+0000 7f91fe084700 1 -- 172.21.15.32:0/1290501115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f8076c80 msgr2=0x7f91f81051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:58.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.038+0000 7f91fe084700 1 --2- 172.21.15.32:0/1290501115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f8076c80 0x7f91f81051b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f91e80098e0 tx=0x7f91e8009250 comp rx=0 tx=0).stop 2024-04-12T05:27:58.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.038+0000 7f91fe084700 1 -- 172.21.15.32:0/1290501115 shutdown_connections 2024-04-12T05:27:58.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.038+0000 7f91fe084700 1 --2- 172.21.15.32:0/1290501115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f8076c80 0x7f91f81051b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:58.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.038+0000 7f91fe084700 1 -- 172.21.15.32:0/1290501115 >> 172.21.15.32:0/1290501115 conn(0x7f91f80fcc60 msgr2=0x7f91f80ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:58.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.038+0000 7f91fe084700 1 -- 172.21.15.32:0/1290501115 shutdown_connections 2024-04-12T05:27:58.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.038+0000 7f91fe084700 1 -- 172.21.15.32:0/1290501115 wait complete. 2024-04-12T05:27:58.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.038+0000 7f91fe084700 1 Processor -- start 2024-04-12T05:27:58.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.038+0000 7f91fe084700 1 -- start start 2024-04-12T05:27:58.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.039+0000 7f91fe084700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f8076c80 0x7f91f8102b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:58.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.039+0000 7f91fe084700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f8103040 con 0x7f91f8076c80 2024-04-12T05:27:58.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.039+0000 7f91f77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f8076c80 0x7f91f8102b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:58.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.039+0000 7f91f77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f8076c80 0x7f91f8102b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47092/0 (socket says 172.21.15.32:47092) 2024-04-12T05:27:58.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.039+0000 7f91f77fe700 1 -- 172.21.15.32:0/556182584 learned_addr learned my addr 172.21.15.32:0/556182584 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:27:58.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.039+0000 7f91f77fe700 1 -- 172.21.15.32:0/556182584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e80094f0 con 0x7f91f8076c80 2024-04-12T05:27:58.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.039+0000 7f91f77fe700 1 --2- 172.21.15.32:0/556182584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f8076c80 0x7f91f8102b00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f91e8006d40 tx=0x7f91e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:58.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.039+0000 7f91dffff700 1 -- 172.21.15.32:0/556182584 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91e8010960 con 0x7f91f8076c80 2024-04-12T05:27:58.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.039+0000 7f91fe084700 1 -- 172.21.15.32:0/556182584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f81010b0 con 0x7f91f8076c80 2024-04-12T05:27:58.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.039+0000 7f91dffff700 1 -- 172.21.15.32:0/556182584 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f91e801a430 con 0x7f91f8076c80 2024-04-12T05:27:58.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.040+0000 7f91dffff700 1 -- 172.21.15.32:0/556182584 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91e80245b0 con 0x7f91f8076c80 2024-04-12T05:27:58.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.040+0000 7f91fe084700 1 -- 172.21.15.32:0/556182584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91f81015a0 con 0x7f91f8076c80 2024-04-12T05:27:58.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.040+0000 7f91dffff700 1 -- 172.21.15.32:0/556182584 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f91e802d3f0 con 0x7f91f8076c80 2024-04-12T05:27:58.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.041+0000 7f91fe084700 1 -- 172.21.15.32:0/556182584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91f804bed0 con 0x7f91f8076c80 2024-04-12T05:27:58.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.041+0000 7f91dffff700 1 --2- 172.21.15.32:0/556182584 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f91e0036790 0x7f91e0038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:27:58.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.041+0000 7f91dffff700 1 -- 172.21.15.32:0/556182584 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f91e804a590 con 0x7f91f8076c80 2024-04-12T05:27:58.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.041+0000 7f91f6ffd700 1 --2- 172.21.15.32:0/556182584 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f91e0036790 0x7f91e0038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:58.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.041+0000 7f91f6ffd700 1 --2- 172.21.15.32:0/556182584 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f91e0036790 0x7f91e0038c40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f91ec009990 tx=0x7f91ec006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:58.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.046+0000 7f91dffff700 1 -- 172.21.15.32:0/556182584 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f91e802b8e0 con 0x7f91f8076c80 2024-04-12T05:27:58.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.255+0000 7f91fe084700 1 -- 172.21.15.32:0/556182584 --> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f91f8101b80 con 0x7f91e0036790 2024-04-12T05:27:58.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.259+0000 7f91dffff700 1 -- 172.21.15.32:0/556182584 <== mgr.14118 v2:172.21.15.32:6800/1962278696 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f91f8101b80 con 0x7f91e0036790 2024-04-12T05:27:58.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.261+0000 7f91fe084700 1 -- 172.21.15.32:0/556182584 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f91e0036790 msgr2=0x7f91e0038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:58.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.261+0000 7f91fe084700 1 --2- 172.21.15.32:0/556182584 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f91e0036790 0x7f91e0038c40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f91ec009990 tx=0x7f91ec006e40 comp rx=0 tx=0).stop 2024-04-12T05:27:58.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.261+0000 7f91fe084700 1 -- 172.21.15.32:0/556182584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f8076c80 msgr2=0x7f91f8102b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:58.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.261+0000 7f91fe084700 1 --2- 172.21.15.32:0/556182584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f8076c80 0x7f91f8102b00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f91e8006d40 tx=0x7f91e8009250 comp rx=0 tx=0).stop 2024-04-12T05:27:58.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.261+0000 7f91fe084700 1 -- 172.21.15.32:0/556182584 shutdown_connections 2024-04-12T05:27:58.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.261+0000 7f91fe084700 1 --2- 172.21.15.32:0/556182584 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f91e0036790 0x7f91e0038c40 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:58.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.261+0000 7f91fe084700 1 --2- 172.21.15.32:0/556182584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f8076c80 0x7f91f8102b00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:27:58.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.261+0000 7f91fe084700 1 -- 172.21.15.32:0/556182584 >> 172.21.15.32:0/556182584 conn(0x7f91f80fcc60 msgr2=0x7f91f80fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:58.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.261+0000 7f91fe084700 1 -- 172.21.15.32:0/556182584 shutdown_connections 2024-04-12T05:27:58.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:58.261+0000 7f91fe084700 1 -- 172.21.15.32:0/556182584 wait complete. 2024-04-12T05:27:58.831 INFO:teuthology.orchestra.run.smithi032.stdout:Deploying grafana service with default placement... 2024-04-12T05:27:59.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:59 smithi032 ceph-mon[39731]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:27:59.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:59 smithi032 ceph-mon[39731]: Saving service prometheus spec with placement count:1 2024-04-12T05:27:59.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:59 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:27:59.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:27:59 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:00.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:00 smithi032 ceph-mon[39731]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:28:00.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:00 smithi032 ceph-mon[39731]: Saving service grafana spec with placement count:1 2024-04-12T05:28:00.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:00 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:00.957 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-04-12T05:28:00.957 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.350+0000 7f036523d700 1 Processor -- start 2024-04-12T05:28:00.957 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.350+0000 7f036523d700 1 -- start start 2024-04-12T05:28:00.957 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.350+0000 7f036523d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03601014a0 0x7f03601018b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:00.957 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.350+0000 7f036523d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0360101df0 con 0x7f03601014a0 2024-04-12T05:28:00.957 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.350+0000 7f035ed9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03601014a0 0x7f03601018b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:00.957 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.350+0000 7f035ed9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03601014a0 0x7f03601018b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47098/0 (socket says 172.21.15.32:47098) 2024-04-12T05:28:00.957 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.350+0000 7f035ed9d700 1 -- 172.21.15.32:0/885199748 learned_addr learned my addr 172.21.15.32:0/885199748 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:00.957 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.351+0000 7f035ed9d700 1 -- 172.21.15.32:0/885199748 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0360101f30 con 0x7f03601014a0 2024-04-12T05:28:00.957 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.351+0000 7f035ed9d700 1 --2- 172.21.15.32:0/885199748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03601014a0 0x7f03601018b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f03500098e0 tx=0x7f0350009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d075ce8a8ae82f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:00.957 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.351+0000 7f035dd9b700 1 -- 172.21.15.32:0/885199748 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f035000b860 con 0x7f03601014a0 2024-04-12T05:28:00.957 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.351+0000 7f035dd9b700 1 -- 172.21.15.32:0/885199748 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f035000be70 con 0x7f03601014a0 2024-04-12T05:28:00.958 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.351+0000 7f035dd9b700 1 -- 172.21.15.32:0/885199748 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0350010380 con 0x7f03601014a0 2024-04-12T05:28:00.958 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.351+0000 7f036523d700 1 -- 172.21.15.32:0/885199748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03601014a0 msgr2=0x7f03601018b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:00.958 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.351+0000 7f036523d700 1 --2- 172.21.15.32:0/885199748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03601014a0 0x7f03601018b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f03500098e0 tx=0x7f0350009250 comp rx=0 tx=0).stop 2024-04-12T05:28:00.958 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.352+0000 7f036523d700 1 -- 172.21.15.32:0/885199748 shutdown_connections 2024-04-12T05:28:00.958 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.352+0000 7f036523d700 1 --2- 172.21.15.32:0/885199748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03601014a0 0x7f03601018b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:00.958 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.352+0000 7f036523d700 1 -- 172.21.15.32:0/885199748 >> 172.21.15.32:0/885199748 conn(0x7f03600fcc60 msgr2=0x7f03600ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:00.958 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.352+0000 7f036523d700 1 -- 172.21.15.32:0/885199748 shutdown_connections 2024-04-12T05:28:00.958 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.352+0000 7f036523d700 1 -- 172.21.15.32:0/885199748 wait complete. 2024-04-12T05:28:00.958 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.353+0000 7f036523d700 1 Processor -- start 2024-04-12T05:28:00.958 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.353+0000 7f036523d700 1 -- start start 2024-04-12T05:28:00.958 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.353+0000 7f036523d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03601014a0 0x7f0360076b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:00.958 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.353+0000 7f036523d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0360114180 con 0x7f03601014a0 2024-04-12T05:28:00.958 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.353+0000 7f035ed9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03601014a0 0x7f0360076b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:00.958 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.353+0000 7f035ed9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03601014a0 0x7f0360076b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47112/0 (socket says 172.21.15.32:47112) 2024-04-12T05:28:00.959 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.353+0000 7f035ed9d700 1 -- 172.21.15.32:0/988749559 learned_addr learned my addr 172.21.15.32:0/988749559 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:00.959 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.353+0000 7f035ed9d700 1 -- 172.21.15.32:0/988749559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03500094f0 con 0x7f03601014a0 2024-04-12T05:28:00.959 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.354+0000 7f035ed9d700 1 --2- 172.21.15.32:0/988749559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03601014a0 0x7f0360076b30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0350000c00 tx=0x7f0350009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:00.959 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.354+0000 7f03477fe700 1 -- 172.21.15.32:0/988749559 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03500105f0 con 0x7f03601014a0 2024-04-12T05:28:00.959 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.354+0000 7f03477fe700 1 -- 172.21.15.32:0/988749559 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0350010c00 con 0x7f03601014a0 2024-04-12T05:28:00.959 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.354+0000 7f036523d700 1 -- 172.21.15.32:0/988749559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03601142c0 con 0x7f03601014a0 2024-04-12T05:28:00.959 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.354+0000 7f03477fe700 1 -- 172.21.15.32:0/988749559 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f035001ac90 con 0x7f03601014a0 2024-04-12T05:28:00.959 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.354+0000 7f036523d700 1 -- 172.21.15.32:0/988749559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0360114750 con 0x7f03601014a0 2024-04-12T05:28:00.959 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.355+0000 7f03477fe700 1 -- 172.21.15.32:0/988749559 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f0350025400 con 0x7f03601014a0 2024-04-12T05:28:00.959 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.355+0000 7f03477fe700 1 --2- 172.21.15.32:0/988749559 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f03480367e0 0x7f0348038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:00.959 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.355+0000 7f03477fe700 1 -- 172.21.15.32:0/988749559 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f035004a370 con 0x7f03601014a0 2024-04-12T05:28:00.959 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.355+0000 7f035e59c700 1 --2- 172.21.15.32:0/988749559 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f03480367e0 0x7f0348038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:00.959 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.355+0000 7f036523d700 1 -- 172.21.15.32:0/988749559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f036004bed0 con 0x7f03601014a0 2024-04-12T05:28:00.959 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.356+0000 7f035e59c700 1 --2- 172.21.15.32:0/988749559 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f03480367e0 0x7f0348038c90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0354009990 tx=0x7f0354006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:00.960 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.360+0000 7f03477fe700 1 -- 172.21.15.32:0/988749559 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f0350020030 con 0x7f03601014a0 2024-04-12T05:28:00.960 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.587+0000 7f036523d700 1 -- 172.21.15.32:0/988749559 --> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f03600fdf40 con 0x7f03480367e0 2024-04-12T05:28:00.960 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.593+0000 7f03477fe700 1 -- 172.21.15.32:0/988749559 <== mgr.14118 v2:172.21.15.32:6800/1962278696 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f03600fdf40 con 0x7f03480367e0 2024-04-12T05:28:00.960 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.595+0000 7f036523d700 1 -- 172.21.15.32:0/988749559 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f03480367e0 msgr2=0x7f0348038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:00.960 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.595+0000 7f036523d700 1 --2- 172.21.15.32:0/988749559 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f03480367e0 0x7f0348038c90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0354009990 tx=0x7f0354006e40 comp rx=0 tx=0).stop 2024-04-12T05:28:00.960 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.595+0000 7f036523d700 1 -- 172.21.15.32:0/988749559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03601014a0 msgr2=0x7f0360076b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:00.960 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.595+0000 7f036523d700 1 --2- 172.21.15.32:0/988749559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03601014a0 0x7f0360076b30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0350000c00 tx=0x7f0350009250 comp rx=0 tx=0).stop 2024-04-12T05:28:00.960 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.595+0000 7f036523d700 1 -- 172.21.15.32:0/988749559 shutdown_connections 2024-04-12T05:28:00.960 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.595+0000 7f036523d700 1 --2- 172.21.15.32:0/988749559 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f03480367e0 0x7f0348038c90 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:00.960 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.595+0000 7f036523d700 1 --2- 172.21.15.32:0/988749559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03601014a0 0x7f0360076b30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:00.960 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.595+0000 7f036523d700 1 -- 172.21.15.32:0/988749559 >> 172.21.15.32:0/988749559 conn(0x7f03600fcc60 msgr2=0x7f03600fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:00.960 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.595+0000 7f036523d700 1 -- 172.21.15.32:0/988749559 shutdown_connections 2024-04-12T05:28:00.960 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:27:59.595+0000 7f036523d700 1 -- 172.21.15.32:0/988749559 wait complete. 2024-04-12T05:28:00.960 INFO:teuthology.orchestra.run.smithi032.stdout:Deploying node-exporter service with default placement... 2024-04-12T05:28:02.094 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:01 smithi032 ceph-mon[39731]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:28:02.094 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:01 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:02.387 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-04-12T05:28:02.387 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.529+0000 7fdf156ca700 1 Processor -- start 2024-04-12T05:28:02.387 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.529+0000 7fdf156ca700 1 -- start start 2024-04-12T05:28:02.387 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.529+0000 7fdf156ca700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf10102a50 0x7fdf10102e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:02.387 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.529+0000 7fdf156ca700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf101033a0 con 0x7fdf10102a50 2024-04-12T05:28:02.387 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.529+0000 7fdf0effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf10102a50 0x7fdf10102e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:02.387 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.529+0000 7fdf0effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf10102a50 0x7fdf10102e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56990/0 (socket says 172.21.15.32:56990) 2024-04-12T05:28:02.387 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.529+0000 7fdf0effd700 1 -- 172.21.15.32:0/267357228 learned_addr learned my addr 172.21.15.32:0/267357228 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:02.387 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.529+0000 7fdf0effd700 1 -- 172.21.15.32:0/267357228 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf101034e0 con 0x7fdf10102a50 2024-04-12T05:28:02.388 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.529+0000 7fdf0effd700 1 --2- 172.21.15.32:0/267357228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf10102a50 0x7fdf10102e60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdf000098e0 tx=0x7fdf00009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f30abe11c061f2a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:02.388 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.530+0000 7fdf0dffb700 1 -- 172.21.15.32:0/267357228 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf0000b860 con 0x7fdf10102a50 2024-04-12T05:28:02.388 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.530+0000 7fdf0dffb700 1 -- 172.21.15.32:0/267357228 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf0000be70 con 0x7fdf10102a50 2024-04-12T05:28:02.388 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.530+0000 7fdf0dffb700 1 -- 172.21.15.32:0/267357228 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf00010380 con 0x7fdf10102a50 2024-04-12T05:28:02.388 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.530+0000 7fdf156ca700 1 -- 172.21.15.32:0/267357228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf10102a50 msgr2=0x7fdf10102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:02.388 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.530+0000 7fdf156ca700 1 --2- 172.21.15.32:0/267357228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf10102a50 0x7fdf10102e60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdf000098e0 tx=0x7fdf00009250 comp rx=0 tx=0).stop 2024-04-12T05:28:02.388 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.530+0000 7fdf156ca700 1 -- 172.21.15.32:0/267357228 shutdown_connections 2024-04-12T05:28:02.388 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.530+0000 7fdf156ca700 1 --2- 172.21.15.32:0/267357228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf10102a50 0x7fdf10102e60 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:02.388 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.530+0000 7fdf156ca700 1 -- 172.21.15.32:0/267357228 >> 172.21.15.32:0/267357228 conn(0x7fdf100fc400 msgr2=0x7fdf100fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:02.388 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.531+0000 7fdf156ca700 1 -- 172.21.15.32:0/267357228 shutdown_connections 2024-04-12T05:28:02.388 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.531+0000 7fdf156ca700 1 -- 172.21.15.32:0/267357228 wait complete. 2024-04-12T05:28:02.388 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.531+0000 7fdf156ca700 1 Processor -- start 2024-04-12T05:28:02.388 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.531+0000 7fdf156ca700 1 -- start start 2024-04-12T05:28:02.389 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.532+0000 7fdf156ca700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf10102a50 0x7fdf1010f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:02.389 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.532+0000 7fdf156ca700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1010f720 con 0x7fdf10102a50 2024-04-12T05:28:02.389 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.532+0000 7fdf0effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf10102a50 0x7fdf1010f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:02.389 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.532+0000 7fdf0effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf10102a50 0x7fdf1010f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57000/0 (socket says 172.21.15.32:57000) 2024-04-12T05:28:02.389 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.532+0000 7fdf0effd700 1 -- 172.21.15.32:0/1917488826 learned_addr learned my addr 172.21.15.32:0/1917488826 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:02.389 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.532+0000 7fdf0effd700 1 -- 172.21.15.32:0/1917488826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf000094f0 con 0x7fdf10102a50 2024-04-12T05:28:02.389 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.532+0000 7fdf0effd700 1 --2- 172.21.15.32:0/1917488826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf10102a50 0x7fdf1010f1e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdf000090b0 tx=0x7fdf00009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:02.389 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.532+0000 7fdef77fe700 1 -- 172.21.15.32:0/1917488826 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf00010800 con 0x7fdf10102a50 2024-04-12T05:28:02.389 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.532+0000 7fdef77fe700 1 -- 172.21.15.32:0/1917488826 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf00010e10 con 0x7fdf10102a50 2024-04-12T05:28:02.389 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.532+0000 7fdf156ca700 1 -- 172.21.15.32:0/1917488826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf1010f920 con 0x7fdf10102a50 2024-04-12T05:28:02.389 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.533+0000 7fdef77fe700 1 -- 172.21.15.32:0/1917488826 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf0001ae60 con 0x7fdf10102a50 2024-04-12T05:28:02.389 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.533+0000 7fdf156ca700 1 -- 172.21.15.32:0/1917488826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf10110490 con 0x7fdf10102a50 2024-04-12T05:28:02.389 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.533+0000 7fdef77fe700 1 -- 172.21.15.32:0/1917488826 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fdf00010960 con 0x7fdf10102a50 2024-04-12T05:28:02.389 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.534+0000 7fdef77fe700 1 --2- 172.21.15.32:0/1917488826 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fdef80389c0 0x7fdef803ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:02.390 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.533+0000 7fdf156ca700 1 -- 172.21.15.32:0/1917488826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf1005f5f0 con 0x7fdf10102a50 2024-04-12T05:28:02.390 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.534+0000 7fdef77fe700 1 -- 172.21.15.32:0/1917488826 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fdf0004a780 con 0x7fdf10102a50 2024-04-12T05:28:02.390 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.534+0000 7fdf0e7fc700 1 --2- 172.21.15.32:0/1917488826 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fdef80389c0 0x7fdef803ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:02.390 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.534+0000 7fdf0e7fc700 1 --2- 172.21.15.32:0/1917488826 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fdef80389c0 0x7fdef803ae70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fdf04009990 tx=0x7fdf04006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:02.390 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.538+0000 7fdef77fe700 1 -- 172.21.15.32:0/1917488826 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fdf00018530 con 0x7fdf10102a50 2024-04-12T05:28:02.390 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.747+0000 7fdf156ca700 1 -- 172.21.15.32:0/1917488826 --> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf10110840 con 0x7fdef80389c0 2024-04-12T05:28:02.390 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.751+0000 7fdef77fe700 1 -- 172.21.15.32:0/1917488826 <== mgr.14118 v2:172.21.15.32:6800/1962278696 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fdf10110840 con 0x7fdef80389c0 2024-04-12T05:28:02.390 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.753+0000 7fdf156ca700 1 -- 172.21.15.32:0/1917488826 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fdef80389c0 msgr2=0x7fdef803ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:02.390 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.753+0000 7fdf156ca700 1 --2- 172.21.15.32:0/1917488826 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fdef80389c0 0x7fdef803ae70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fdf04009990 tx=0x7fdf04006e40 comp rx=0 tx=0).stop 2024-04-12T05:28:02.390 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.753+0000 7fdf156ca700 1 -- 172.21.15.32:0/1917488826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf10102a50 msgr2=0x7fdf1010f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:02.390 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.753+0000 7fdf156ca700 1 --2- 172.21.15.32:0/1917488826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf10102a50 0x7fdf1010f1e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdf000090b0 tx=0x7fdf00009250 comp rx=0 tx=0).stop 2024-04-12T05:28:02.390 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.753+0000 7fdf156ca700 1 -- 172.21.15.32:0/1917488826 shutdown_connections 2024-04-12T05:28:02.390 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.753+0000 7fdf156ca700 1 --2- 172.21.15.32:0/1917488826 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fdef80389c0 0x7fdef803ae70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:02.390 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.753+0000 7fdf156ca700 1 --2- 172.21.15.32:0/1917488826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf10102a50 0x7fdf1010f1e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:02.391 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.753+0000 7fdf156ca700 1 -- 172.21.15.32:0/1917488826 >> 172.21.15.32:0/1917488826 conn(0x7fdf100fc400 msgr2=0x7fdf10105a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:02.391 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.753+0000 7fdf156ca700 1 -- 172.21.15.32:0/1917488826 shutdown_connections 2024-04-12T05:28:02.391 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:01.753+0000 7fdf156ca700 1 -- 172.21.15.32:0/1917488826 wait complete. 2024-04-12T05:28:02.391 INFO:teuthology.orchestra.run.smithi032.stdout:Deploying alertmanager service with default placement... 2024-04-12T05:28:03.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:02 smithi032 ceph-mon[39731]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:28:03.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:02 smithi032 ceph-mon[39731]: Saving service node-exporter spec with placement * 2024-04-12T05:28:03.643 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-04-12T05:28:03.643 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.922+0000 7f638d9e6700 1 Processor -- start 2024-04-12T05:28:03.643 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.922+0000 7f638d9e6700 1 -- start start 2024-04-12T05:28:03.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.922+0000 7f638d9e6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6388076c00 0x7f6388107630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:03.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.922+0000 7f638d9e6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6388103020 con 0x7f6388076c00 2024-04-12T05:28:03.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.922+0000 7f6386ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6388076c00 0x7f6388107630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:03.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.922+0000 7f6386ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6388076c00 0x7f6388107630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57010/0 (socket says 172.21.15.32:57010) 2024-04-12T05:28:03.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.922+0000 7f6386ffd700 1 -- 172.21.15.32:0/3868806426 learned_addr learned my addr 172.21.15.32:0/3868806426 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:03.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.923+0000 7f6386ffd700 1 -- 172.21.15.32:0/3868806426 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6388107b70 con 0x7f6388076c00 2024-04-12T05:28:03.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.923+0000 7f6386ffd700 1 --2- 172.21.15.32:0/3868806426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6388076c00 0x7f6388107630 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f637c0098e0 tx=0x7f637c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f0a25ebca58745d server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:03.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.923+0000 7f6385ffb700 1 -- 172.21.15.32:0/3868806426 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f637c00b860 con 0x7f6388076c00 2024-04-12T05:28:03.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.923+0000 7f6385ffb700 1 -- 172.21.15.32:0/3868806426 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f637c00be70 con 0x7f6388076c00 2024-04-12T05:28:03.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.923+0000 7f6385ffb700 1 -- 172.21.15.32:0/3868806426 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f637c010380 con 0x7f6388076c00 2024-04-12T05:28:03.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.923+0000 7f638d9e6700 1 -- 172.21.15.32:0/3868806426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6388076c00 msgr2=0x7f6388107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:03.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.923+0000 7f638d9e6700 1 --2- 172.21.15.32:0/3868806426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6388076c00 0x7f6388107630 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f637c0098e0 tx=0x7f637c009250 comp rx=0 tx=0).stop 2024-04-12T05:28:03.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.924+0000 7f638d9e6700 1 -- 172.21.15.32:0/3868806426 shutdown_connections 2024-04-12T05:28:03.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.924+0000 7f638d9e6700 1 --2- 172.21.15.32:0/3868806426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6388076c00 0x7f6388107630 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:03.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.924+0000 7f638d9e6700 1 -- 172.21.15.32:0/3868806426 >> 172.21.15.32:0/3868806426 conn(0x7f63880fcc60 msgr2=0x7f63880ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:03.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.924+0000 7f638d9e6700 1 -- 172.21.15.32:0/3868806426 shutdown_connections 2024-04-12T05:28:03.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.924+0000 7f638d9e6700 1 -- 172.21.15.32:0/3868806426 wait complete. 2024-04-12T05:28:03.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.924+0000 7f638d9e6700 1 Processor -- start 2024-04-12T05:28:03.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.924+0000 7f638d9e6700 1 -- start start 2024-04-12T05:28:03.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.925+0000 7f638d9e6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6388076c00 0x7f63881135b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:03.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.925+0000 7f638d9e6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6388113af0 con 0x7f6388076c00 2024-04-12T05:28:03.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.925+0000 7f6386ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6388076c00 0x7f63881135b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:03.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.925+0000 7f6386ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6388076c00 0x7f63881135b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57018/0 (socket says 172.21.15.32:57018) 2024-04-12T05:28:03.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.925+0000 7f6386ffd700 1 -- 172.21.15.32:0/2111196335 learned_addr learned my addr 172.21.15.32:0/2111196335 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:03.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.925+0000 7f6386ffd700 1 -- 172.21.15.32:0/2111196335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f637c0094f0 con 0x7f6388076c00 2024-04-12T05:28:03.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.925+0000 7f6386ffd700 1 --2- 172.21.15.32:0/2111196335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6388076c00 0x7f63881135b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f637c006d40 tx=0x7f637c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:03.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.925+0000 7f636f7fe700 1 -- 172.21.15.32:0/2111196335 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f637c010960 con 0x7f6388076c00 2024-04-12T05:28:03.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.925+0000 7f636f7fe700 1 -- 172.21.15.32:0/2111196335 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f637c01a430 con 0x7f6388076c00 2024-04-12T05:28:03.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.925+0000 7f638d9e6700 1 -- 172.21.15.32:0/2111196335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6388113cf0 con 0x7f6388076c00 2024-04-12T05:28:03.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.925+0000 7f636f7fe700 1 -- 172.21.15.32:0/2111196335 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f637c0245b0 con 0x7f6388076c00 2024-04-12T05:28:03.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.925+0000 7f638d9e6700 1 -- 172.21.15.32:0/2111196335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6388114860 con 0x7f6388076c00 2024-04-12T05:28:03.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.926+0000 7f636f7fe700 1 -- 172.21.15.32:0/2111196335 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f637c010ac0 con 0x7f6388076c00 2024-04-12T05:28:03.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.926+0000 7f638d9e6700 1 -- 172.21.15.32:0/2111196335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f638805f5f0 con 0x7f6388076c00 2024-04-12T05:28:03.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.926+0000 7f636f7fe700 1 --2- 172.21.15.32:0/2111196335 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f63700367e0 0x7f6370038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:03.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.926+0000 7f636f7fe700 1 -- 172.21.15.32:0/2111196335 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f637c04a350 con 0x7f6388076c00 2024-04-12T05:28:03.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.927+0000 7f63867fc700 1 --2- 172.21.15.32:0/2111196335 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f63700367e0 0x7f6370038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:03.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.927+0000 7f63867fc700 1 --2- 172.21.15.32:0/2111196335 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f63700367e0 0x7f6370038c90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6378009990 tx=0x7f6378006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:03.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:02.932+0000 7f636f7fe700 1 -- 172.21.15.32:0/2111196335 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f637c02b4d0 con 0x7f6388076c00 2024-04-12T05:28:03.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:03.140+0000 7f638d9e6700 1 -- 172.21.15.32:0/2111196335 --> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f63880fde70 con 0x7f63700367e0 2024-04-12T05:28:03.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:03.144+0000 7f636f7fe700 1 -- 172.21.15.32:0/2111196335 <== mgr.14118 v2:172.21.15.32:6800/1962278696 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f63880fde70 con 0x7f63700367e0 2024-04-12T05:28:03.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:03.147+0000 7f638d9e6700 1 -- 172.21.15.32:0/2111196335 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f63700367e0 msgr2=0x7f6370038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:03.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:03.147+0000 7f638d9e6700 1 --2- 172.21.15.32:0/2111196335 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f63700367e0 0x7f6370038c90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6378009990 tx=0x7f6378006e40 comp rx=0 tx=0).stop 2024-04-12T05:28:03.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:03.147+0000 7f638d9e6700 1 -- 172.21.15.32:0/2111196335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6388076c00 msgr2=0x7f63881135b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:03.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:03.147+0000 7f638d9e6700 1 --2- 172.21.15.32:0/2111196335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6388076c00 0x7f63881135b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f637c006d40 tx=0x7f637c009250 comp rx=0 tx=0).stop 2024-04-12T05:28:03.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:03.147+0000 7f638d9e6700 1 -- 172.21.15.32:0/2111196335 shutdown_connections 2024-04-12T05:28:03.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:03.147+0000 7f638d9e6700 1 --2- 172.21.15.32:0/2111196335 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f63700367e0 0x7f6370038c90 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:03.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:03.147+0000 7f638d9e6700 1 --2- 172.21.15.32:0/2111196335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6388076c00 0x7f63881135b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:03.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:03.147+0000 7f638d9e6700 1 -- 172.21.15.32:0/2111196335 >> 172.21.15.32:0/2111196335 conn(0x7f63880fcc60 msgr2=0x7f63880fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:03.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:03.147+0000 7f638d9e6700 1 -- 172.21.15.32:0/2111196335 shutdown_connections 2024-04-12T05:28:03.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:03.147+0000 7f638d9e6700 1 -- 172.21.15.32:0/2111196335 wait complete. 2024-04-12T05:28:04.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:04 smithi032 ceph-mon[39731]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:28:04.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:04 smithi032 ceph-mon[39731]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:28:04.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:04 smithi032 ceph-mon[39731]: Saving service alertmanager spec with placement count:1 2024-04-12T05:28:04.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:04 smithi032 ceph-mon[39731]: from='mgr.14118 172.21.15.32:0/1740327569' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:04.991 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.120+0000 7f4c0de48700 1 Processor -- start 2024-04-12T05:28:04.992 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.121+0000 7f4c0de48700 1 -- start start 2024-04-12T05:28:04.992 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.121+0000 7f4c0de48700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c08078d50 0x7f4c08077260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:04.992 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.121+0000 7f4c0de48700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c08079160 con 0x7f4c08078d50 2024-04-12T05:28:04.992 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.121+0000 7f4c077fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c08078d50 0x7f4c08077260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:04.992 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.121+0000 7f4c077fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c08078d50 0x7f4c08077260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57024/0 (socket says 172.21.15.32:57024) 2024-04-12T05:28:04.992 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.121+0000 7f4c077fe700 1 -- 172.21.15.32:0/3527044520 learned_addr learned my addr 172.21.15.32:0/3527044520 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:04.992 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.121+0000 7f4c077fe700 1 -- 172.21.15.32:0/3527044520 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c080777a0 con 0x7f4c08078d50 2024-04-12T05:28:04.992 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.121+0000 7f4c077fe700 1 --2- 172.21.15.32:0/3527044520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c08078d50 0x7f4c08077260 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0098e0 tx=0x7f4bfc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a1fad7f02f0e2b server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:04.992 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.121+0000 7f4c067fc700 1 -- 172.21.15.32:0/3527044520 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bfc00b860 con 0x7f4c08078d50 2024-04-12T05:28:04.992 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.122+0000 7f4c067fc700 1 -- 172.21.15.32:0/3527044520 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4bfc00be70 con 0x7f4c08078d50 2024-04-12T05:28:04.992 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.122+0000 7f4c067fc700 1 -- 172.21.15.32:0/3527044520 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bfc010380 con 0x7f4c08078d50 2024-04-12T05:28:04.992 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.122+0000 7f4c0de48700 1 -- 172.21.15.32:0/3527044520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c08078d50 msgr2=0x7f4c08077260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:04.992 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.122+0000 7f4c0de48700 1 --2- 172.21.15.32:0/3527044520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c08078d50 0x7f4c08077260 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0098e0 tx=0x7f4bfc009250 comp rx=0 tx=0).stop 2024-04-12T05:28:04.992 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.122+0000 7f4c0de48700 1 -- 172.21.15.32:0/3527044520 shutdown_connections 2024-04-12T05:28:04.993 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.122+0000 7f4c0de48700 1 --2- 172.21.15.32:0/3527044520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c08078d50 0x7f4c08077260 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:04.993 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.122+0000 7f4c0de48700 1 -- 172.21.15.32:0/3527044520 >> 172.21.15.32:0/3527044520 conn(0x7f4c080fdd00 msgr2=0x7f4c08100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:04.993 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.122+0000 7f4c0de48700 1 -- 172.21.15.32:0/3527044520 shutdown_connections 2024-04-12T05:28:04.993 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.122+0000 7f4c0de48700 1 -- 172.21.15.32:0/3527044520 wait complete. 2024-04-12T05:28:04.993 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.123+0000 7f4c0de48700 1 Processor -- start 2024-04-12T05:28:04.993 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.123+0000 7f4c0de48700 1 -- start start 2024-04-12T05:28:04.993 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.123+0000 7f4c0de48700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c08078d50 0x7f4c081135b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:04.993 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.123+0000 7f4c0de48700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c08113af0 con 0x7f4c08078d50 2024-04-12T05:28:04.994 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.123+0000 7f4c077fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c08078d50 0x7f4c081135b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:04.994 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.123+0000 7f4c077fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c08078d50 0x7f4c081135b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57026/0 (socket says 172.21.15.32:57026) 2024-04-12T05:28:04.994 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.123+0000 7f4c077fe700 1 -- 172.21.15.32:0/3953820537 learned_addr learned my addr 172.21.15.32:0/3953820537 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:04.994 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.124+0000 7f4c077fe700 1 -- 172.21.15.32:0/3953820537 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bfc0094f0 con 0x7f4c08078d50 2024-04-12T05:28:04.994 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.124+0000 7f4c077fe700 1 --2- 172.21.15.32:0/3953820537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c08078d50 0x7f4c081135b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0094c0 tx=0x7f4bfc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:04.994 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.124+0000 7f4beffff700 1 -- 172.21.15.32:0/3953820537 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bfc010850 con 0x7f4c08078d50 2024-04-12T05:28:04.994 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.124+0000 7f4beffff700 1 -- 172.21.15.32:0/3953820537 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4bfc010e60 con 0x7f4c08078d50 2024-04-12T05:28:04.994 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.124+0000 7f4c0de48700 1 -- 172.21.15.32:0/3953820537 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c08113cf0 con 0x7f4c08078d50 2024-04-12T05:28:04.994 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.124+0000 7f4beffff700 1 -- 172.21.15.32:0/3953820537 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bfc01a430 con 0x7f4c08078d50 2024-04-12T05:28:04.994 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.124+0000 7f4c0de48700 1 -- 172.21.15.32:0/3953820537 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c08114860 con 0x7f4c08078d50 2024-04-12T05:28:04.994 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.125+0000 7f4beffff700 1 -- 172.21.15.32:0/3953820537 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f4bfc01a6b0 con 0x7f4c08078d50 2024-04-12T05:28:04.995 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.125+0000 7f4beffff700 1 --2- 172.21.15.32:0/3953820537 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f4bf0036790 0x7f4bf0038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:04.995 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.125+0000 7f4beffff700 1 -- 172.21.15.32:0/3953820537 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f4bfc04a680 con 0x7f4c08078d50 2024-04-12T05:28:04.995 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.125+0000 7f4c06ffd700 1 --2- 172.21.15.32:0/3953820537 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f4bf0036790 0x7f4bf0038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:04.995 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.125+0000 7f4c0de48700 1 -- 172.21.15.32:0/3953820537 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c0805f5f0 con 0x7f4c08078d50 2024-04-12T05:28:04.995 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.125+0000 7f4c06ffd700 1 --2- 172.21.15.32:0/3953820537 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f4bf0036790 0x7f4bf0038c40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8009990 tx=0x7f4bf8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:04.995 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.130+0000 7f4beffff700 1 -- 172.21.15.32:0/3953820537 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4bfc0189a0 con 0x7f4c08078d50 2024-04-12T05:28:04.995 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.324+0000 7f4c0de48700 1 -- 172.21.15.32:0/3953820537 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f4c0804bed0 con 0x7f4c08078d50 2024-04-12T05:28:04.995 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.328+0000 7f4beffff700 1 -- 172.21.15.32:0/3953820537 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f4bfc023ed0 con 0x7f4c08078d50 2024-04-12T05:28:04.995 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.330+0000 7f4c0de48700 1 -- 172.21.15.32:0/3953820537 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f4bf0036790 msgr2=0x7f4bf0038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:04.995 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.330+0000 7f4c0de48700 1 --2- 172.21.15.32:0/3953820537 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f4bf0036790 0x7f4bf0038c40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8009990 tx=0x7f4bf8006e40 comp rx=0 tx=0).stop 2024-04-12T05:28:04.996 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.330+0000 7f4c0de48700 1 -- 172.21.15.32:0/3953820537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c08078d50 msgr2=0x7f4c081135b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:04.996 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.330+0000 7f4c0de48700 1 --2- 172.21.15.32:0/3953820537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c08078d50 0x7f4c081135b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0094c0 tx=0x7f4bfc009250 comp rx=0 tx=0).stop 2024-04-12T05:28:04.996 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.330+0000 7f4c0de48700 1 -- 172.21.15.32:0/3953820537 shutdown_connections 2024-04-12T05:28:04.996 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.330+0000 7f4c0de48700 1 --2- 172.21.15.32:0/3953820537 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f4bf0036790 0x7f4bf0038c40 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:04.996 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.330+0000 7f4c0de48700 1 --2- 172.21.15.32:0/3953820537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c08078d50 0x7f4c081135b0 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0094c0 tx=0x7f4bfc009250 comp rx=0 tx=0).stop 2024-04-12T05:28:04.996 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.330+0000 7f4c0de48700 1 -- 172.21.15.32:0/3953820537 >> 172.21.15.32:0/3953820537 conn(0x7f4c080fdd00 msgr2=0x7f4c080fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:04.996 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.331+0000 7f4c0de48700 1 -- 172.21.15.32:0/3953820537 shutdown_connections 2024-04-12T05:28:04.996 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:04.331+0000 7f4c0de48700 1 -- 172.21.15.32:0/3953820537 wait complete. 2024-04-12T05:28:05.597 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:05 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/3953820537' entity='client.admin' 2024-04-12T05:28:05.636 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:05 smithi032 ceph-mon[39731]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:28:06.356 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.519+0000 7f900a465700 1 Processor -- start 2024-04-12T05:28:06.356 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.519+0000 7f900a465700 1 -- start start 2024-04-12T05:28:06.357 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.519+0000 7f900a465700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9004103b00 0x7f9004103f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:06.357 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.519+0000 7f900a465700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9004104450 con 0x7f9004103b00 2024-04-12T05:28:06.357 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.519+0000 7f9003fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9004103b00 0x7f9004103f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:06.357 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.519+0000 7f9003fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9004103b00 0x7f9004103f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57040/0 (socket says 172.21.15.32:57040) 2024-04-12T05:28:06.357 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.519+0000 7f9003fff700 1 -- 172.21.15.32:0/1458845879 learned_addr learned my addr 172.21.15.32:0/1458845879 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:06.357 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.520+0000 7f9003fff700 1 -- 172.21.15.32:0/1458845879 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9004104590 con 0x7f9004103b00 2024-04-12T05:28:06.357 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.520+0000 7f9003fff700 1 --2- 172.21.15.32:0/1458845879 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9004103b00 0x7f9004103f10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8ff40098e0 tx=0x7f8ff4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47818b19f5a1d376 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:06.357 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.520+0000 7f9002ffd700 1 -- 172.21.15.32:0/1458845879 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ff400b860 con 0x7f9004103b00 2024-04-12T05:28:06.357 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.520+0000 7f9002ffd700 1 -- 172.21.15.32:0/1458845879 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ff400be70 con 0x7f9004103b00 2024-04-12T05:28:06.357 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.520+0000 7f9002ffd700 1 -- 172.21.15.32:0/1458845879 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ff4010380 con 0x7f9004103b00 2024-04-12T05:28:06.357 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.521+0000 7f900a465700 1 -- 172.21.15.32:0/1458845879 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9004103b00 msgr2=0x7f9004103f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:06.357 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.521+0000 7f900a465700 1 --2- 172.21.15.32:0/1458845879 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9004103b00 0x7f9004103f10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8ff40098e0 tx=0x7f8ff4009250 comp rx=0 tx=0).stop 2024-04-12T05:28:06.357 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.521+0000 7f900a465700 1 -- 172.21.15.32:0/1458845879 shutdown_connections 2024-04-12T05:28:06.358 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.521+0000 7f900a465700 1 --2- 172.21.15.32:0/1458845879 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9004103b00 0x7f9004103f10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:06.358 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.521+0000 7f900a465700 1 -- 172.21.15.32:0/1458845879 >> 172.21.15.32:0/1458845879 conn(0x7f90040fc2b0 msgr2=0x7f90040fe6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:06.358 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.521+0000 7f900a465700 1 -- 172.21.15.32:0/1458845879 shutdown_connections 2024-04-12T05:28:06.358 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.521+0000 7f900a465700 1 -- 172.21.15.32:0/1458845879 wait complete. 2024-04-12T05:28:06.358 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.521+0000 7f900a465700 1 Processor -- start 2024-04-12T05:28:06.358 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.521+0000 7f900a465700 1 -- start start 2024-04-12T05:28:06.358 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.522+0000 7f900a465700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9004103b00 0x7f90041032a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:06.358 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.522+0000 7f900a465700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90041037e0 con 0x7f9004103b00 2024-04-12T05:28:06.358 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.522+0000 7f9003fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9004103b00 0x7f90041032a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:06.358 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.522+0000 7f9003fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9004103b00 0x7f90041032a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57050/0 (socket says 172.21.15.32:57050) 2024-04-12T05:28:06.358 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.522+0000 7f9003fff700 1 -- 172.21.15.32:0/1886861245 learned_addr learned my addr 172.21.15.32:0/1886861245 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:06.358 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.522+0000 7f9003fff700 1 -- 172.21.15.32:0/1886861245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff40094f0 con 0x7f9004103b00 2024-04-12T05:28:06.358 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.522+0000 7f9003fff700 1 --2- 172.21.15.32:0/1886861245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9004103b00 0x7f90041032a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8ff400c010 tx=0x7f8ff4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:06.358 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.522+0000 7f9000ff9700 1 -- 172.21.15.32:0/1886861245 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ff4010960 con 0x7f9004103b00 2024-04-12T05:28:06.359 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.522+0000 7f9000ff9700 1 -- 172.21.15.32:0/1886861245 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ff401a430 con 0x7f9004103b00 2024-04-12T05:28:06.359 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.522+0000 7f9000ff9700 1 -- 172.21.15.32:0/1886861245 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ff4019580 con 0x7f9004103b00 2024-04-12T05:28:06.359 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.522+0000 7f900a465700 1 -- 172.21.15.32:0/1886861245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9004101850 con 0x7f9004103b00 2024-04-12T05:28:06.359 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.522+0000 7f900a465700 1 -- 172.21.15.32:0/1886861245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9004101d40 con 0x7f9004103b00 2024-04-12T05:28:06.359 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.523+0000 7f9000ff9700 1 -- 172.21.15.32:0/1886861245 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f8ff4019800 con 0x7f9004103b00 2024-04-12T05:28:06.359 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.523+0000 7f900a465700 1 -- 172.21.15.32:0/1886861245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f900404bed0 con 0x7f9004103b00 2024-04-12T05:28:06.359 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.523+0000 7f9000ff9700 1 --2- 172.21.15.32:0/1886861245 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f8ff0036790 0x7f8ff0038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:06.359 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.524+0000 7f9000ff9700 1 -- 172.21.15.32:0/1886861245 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f8ff404a5f0 con 0x7f9004103b00 2024-04-12T05:28:06.359 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.524+0000 7f90037fe700 1 --2- 172.21.15.32:0/1886861245 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f8ff0036790 0x7f8ff0038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:06.359 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.524+0000 7f90037fe700 1 --2- 172.21.15.32:0/1886861245 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f8ff0036790 0x7f8ff0038c40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8fec009990 tx=0x7f8fec006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:06.359 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.528+0000 7f9000ff9700 1 -- 172.21.15.32:0/1886861245 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f8ff402be60 con 0x7f9004103b00 2024-04-12T05:28:06.359 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.721+0000 7f900a465700 1 -- 172.21.15.32:0/1886861245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f90041021c0 con 0x7f9004103b00 2024-04-12T05:28:06.359 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.724+0000 7f9000ff9700 1 -- 172.21.15.32:0/1886861245 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f8ff402be60 con 0x7f9004103b00 2024-04-12T05:28:06.359 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.726+0000 7f900a465700 1 -- 172.21.15.32:0/1886861245 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f8ff0036790 msgr2=0x7f8ff0038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:06.359 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.726+0000 7f900a465700 1 --2- 172.21.15.32:0/1886861245 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f8ff0036790 0x7f8ff0038c40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8fec009990 tx=0x7f8fec006e40 comp rx=0 tx=0).stop 2024-04-12T05:28:06.360 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.726+0000 7f900a465700 1 -- 172.21.15.32:0/1886861245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9004103b00 msgr2=0x7f90041032a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:06.360 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.726+0000 7f900a465700 1 --2- 172.21.15.32:0/1886861245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9004103b00 0x7f90041032a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8ff400c010 tx=0x7f8ff4009250 comp rx=0 tx=0).stop 2024-04-12T05:28:06.360 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.726+0000 7f900a465700 1 -- 172.21.15.32:0/1886861245 shutdown_connections 2024-04-12T05:28:06.360 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.726+0000 7f900a465700 1 --2- 172.21.15.32:0/1886861245 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f8ff0036790 0x7f8ff0038c40 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:06.360 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.726+0000 7f900a465700 1 --2- 172.21.15.32:0/1886861245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9004103b00 0x7f90041032a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:06.360 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.726+0000 7f900a465700 1 -- 172.21.15.32:0/1886861245 >> 172.21.15.32:0/1886861245 conn(0x7f90040fc2b0 msgr2=0x7f90040fce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:06.360 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.726+0000 7f900a465700 1 -- 172.21.15.32:0/1886861245 shutdown_connections 2024-04-12T05:28:06.360 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:05.726+0000 7f900a465700 1 -- 172.21.15.32:0/1886861245 wait complete. 2024-04-12T05:28:06.360 INFO:teuthology.orchestra.run.smithi032.stdout:Enabling the dashboard module... 2024-04-12T05:28:07.094 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:06 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/1886861245' entity='client.admin' 2024-04-12T05:28:08.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:07 smithi032 ceph-mon[39731]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:28:08.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:07 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/1517112467' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-12T05:28:08.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.919+0000 7f00bd2d7700 1 Processor -- start 2024-04-12T05:28:08.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.919+0000 7f00bd2d7700 1 -- start start 2024-04-12T05:28:08.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.919+0000 7f00bd2d7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00b8076c80 0x7f00b8105180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:08.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.919+0000 7f00bd2d7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b81056c0 con 0x7f00b8076c80 2024-04-12T05:28:08.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.919+0000 7f00b6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00b8076c80 0x7f00b8105180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:08.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.919+0000 7f00b6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00b8076c80 0x7f00b8105180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57062/0 (socket says 172.21.15.32:57062) 2024-04-12T05:28:08.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.919+0000 7f00b6ffd700 1 -- 172.21.15.32:0/222801995 learned_addr learned my addr 172.21.15.32:0/222801995 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:08.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.920+0000 7f00b6ffd700 1 -- 172.21.15.32:0/222801995 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b8105800 con 0x7f00b8076c80 2024-04-12T05:28:08.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.920+0000 7f00b6ffd700 1 --2- 172.21.15.32:0/222801995 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00b8076c80 0x7f00b8105180 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f00ac0098e0 tx=0x7f00ac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb31c9a91699db4c server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:08.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.920+0000 7f00b5ffb700 1 -- 172.21.15.32:0/222801995 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00ac00b860 con 0x7f00b8076c80 2024-04-12T05:28:08.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.920+0000 7f00b5ffb700 1 -- 172.21.15.32:0/222801995 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f00ac00be70 con 0x7f00b8076c80 2024-04-12T05:28:08.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.920+0000 7f00b5ffb700 1 -- 172.21.15.32:0/222801995 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00ac010380 con 0x7f00b8076c80 2024-04-12T05:28:08.424 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.920+0000 7f00bd2d7700 1 -- 172.21.15.32:0/222801995 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00b8076c80 msgr2=0x7f00b8105180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:08.425 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.920+0000 7f00bd2d7700 1 --2- 172.21.15.32:0/222801995 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00b8076c80 0x7f00b8105180 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f00ac0098e0 tx=0x7f00ac009250 comp rx=0 tx=0).stop 2024-04-12T05:28:08.425 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.921+0000 7f00bd2d7700 1 -- 172.21.15.32:0/222801995 shutdown_connections 2024-04-12T05:28:08.425 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.921+0000 7f00bd2d7700 1 --2- 172.21.15.32:0/222801995 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00b8076c80 0x7f00b8105180 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:08.425 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.921+0000 7f00bd2d7700 1 -- 172.21.15.32:0/222801995 >> 172.21.15.32:0/222801995 conn(0x7f00b80fcc30 msgr2=0x7f00b80ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:08.425 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.921+0000 7f00bd2d7700 1 -- 172.21.15.32:0/222801995 shutdown_connections 2024-04-12T05:28:08.425 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.921+0000 7f00bd2d7700 1 -- 172.21.15.32:0/222801995 wait complete. 2024-04-12T05:28:08.425 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.921+0000 7f00bd2d7700 1 Processor -- start 2024-04-12T05:28:08.425 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.922+0000 7f00bd2d7700 1 -- start start 2024-04-12T05:28:08.425 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.922+0000 7f00bd2d7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00b8076c80 0x7f00b81113c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:08.425 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.922+0000 7f00bd2d7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b8111900 con 0x7f00b8076c80 2024-04-12T05:28:08.425 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.922+0000 7f00b6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00b8076c80 0x7f00b81113c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:08.425 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.922+0000 7f00b6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00b8076c80 0x7f00b81113c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57066/0 (socket says 172.21.15.32:57066) 2024-04-12T05:28:08.426 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.922+0000 7f00b6ffd700 1 -- 172.21.15.32:0/1517112467 learned_addr learned my addr 172.21.15.32:0/1517112467 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:08.426 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.922+0000 7f00b6ffd700 1 -- 172.21.15.32:0/1517112467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ac0094f0 con 0x7f00b8076c80 2024-04-12T05:28:08.426 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.922+0000 7f00b6ffd700 1 --2- 172.21.15.32:0/1517112467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00b8076c80 0x7f00b81113c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f00ac006d40 tx=0x7f00ac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:08.426 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.922+0000 7f009f7fe700 1 -- 172.21.15.32:0/1517112467 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00ac010960 con 0x7f00b8076c80 2024-04-12T05:28:08.426 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.923+0000 7f009f7fe700 1 -- 172.21.15.32:0/1517112467 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f00ac01a430 con 0x7f00b8076c80 2024-04-12T05:28:08.426 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.923+0000 7f00bd2d7700 1 -- 172.21.15.32:0/1517112467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00b8111b00 con 0x7f00b8076c80 2024-04-12T05:28:08.426 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.923+0000 7f009f7fe700 1 -- 172.21.15.32:0/1517112467 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00ac0245b0 con 0x7f00b8076c80 2024-04-12T05:28:08.426 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.923+0000 7f00bd2d7700 1 -- 172.21.15.32:0/1517112467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00b8112670 con 0x7f00b8076c80 2024-04-12T05:28:08.426 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.924+0000 7f009f7fe700 1 -- 172.21.15.32:0/1517112467 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f00ac02d3f0 con 0x7f00b8076c80 2024-04-12T05:28:08.426 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.924+0000 7f009f7fe700 1 --2- 172.21.15.32:0/1517112467 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f00a0036790 0x7f00a0038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:08.426 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.924+0000 7f00bd2d7700 1 -- 172.21.15.32:0/1517112467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00b805f5f0 con 0x7f00b8076c80 2024-04-12T05:28:08.426 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.924+0000 7f009f7fe700 1 -- 172.21.15.32:0/1517112467 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f00ac04a460 con 0x7f00b8076c80 2024-04-12T05:28:08.426 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.924+0000 7f00b67fc700 1 --2- 172.21.15.32:0/1517112467 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f00a0036790 0x7f00a0038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:08.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.924+0000 7f00b67fc700 1 --2- 172.21.15.32:0/1517112467 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f00a0036790 0x7f00a0038c40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f00a8009990 tx=0x7f00a8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:08.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:06.929+0000 7f009f7fe700 1 -- 172.21.15.32:0/1517112467 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f00ac010ac0 con 0x7f00b8076c80 2024-04-12T05:28:08.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:07.178+0000 7f00bd2d7700 1 -- 172.21.15.32:0/1517112467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f00b804bed0 con 0x7f00b8076c80 2024-04-12T05:28:08.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:07.728+0000 7f009f7fe700 1 -- 172.21.15.32:0/1517112467 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f00ac01a9f0 con 0x7f00b8076c80 2024-04-12T05:28:08.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:07.729+0000 7f009f7fe700 1 -- 172.21.15.32:0/1517112467 <== mon.0 v2:172.21.15.32:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f00ac0353a0 con 0x7f00b8076c80 2024-04-12T05:28:08.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:07.733+0000 7f00bd2d7700 1 -- 172.21.15.32:0/1517112467 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f00a0036790 msgr2=0x7f00a0038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:08.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:07.733+0000 7f00bd2d7700 1 --2- 172.21.15.32:0/1517112467 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f00a0036790 0x7f00a0038c40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f00a8009990 tx=0x7f00a8006e40 comp rx=0 tx=0).stop 2024-04-12T05:28:08.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:07.733+0000 7f00bd2d7700 1 -- 172.21.15.32:0/1517112467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00b8076c80 msgr2=0x7f00b81113c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:08.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:07.733+0000 7f00bd2d7700 1 --2- 172.21.15.32:0/1517112467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00b8076c80 0x7f00b81113c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f00ac006d40 tx=0x7f00ac009250 comp rx=0 tx=0).stop 2024-04-12T05:28:08.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:07.733+0000 7f00bd2d7700 1 -- 172.21.15.32:0/1517112467 shutdown_connections 2024-04-12T05:28:08.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:07.733+0000 7f00bd2d7700 1 --2- 172.21.15.32:0/1517112467 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7f00a0036790 0x7f00a0038c40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:08.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:07.733+0000 7f00bd2d7700 1 --2- 172.21.15.32:0/1517112467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00b8076c80 0x7f00b81113c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:08.428 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:07.733+0000 7f00bd2d7700 1 -- 172.21.15.32:0/1517112467 >> 172.21.15.32:0/1517112467 conn(0x7f00b80fcc30 msgr2=0x7f00b80fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:08.428 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:07.733+0000 7f00bd2d7700 1 -- 172.21.15.32:0/1517112467 shutdown_connections 2024-04-12T05:28:08.428 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:07.733+0000 7f00bd2d7700 1 -- 172.21.15.32:0/1517112467 wait complete. 2024-04-12T05:28:08.948 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:08 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/1517112467' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-12T05:28:08.948 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:08 smithi032 ceph-mon[39731]: mgrmap e9: smithi032.hrwxrh(active, since 26s) 2024-04-12T05:28:09.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-12T05:28:09.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-12T05:28:09.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-12T05:28:09.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "active_name": "smithi032.hrwxrh", 2024-04-12T05:28:09.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-12T05:28:09.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-12T05:28:09.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.936+0000 7feae0608700 1 Processor -- start 2024-04-12T05:28:09.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.937+0000 7feae0608700 1 -- start start 2024-04-12T05:28:09.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.937+0000 7feae0608700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fead81014e0 0x7fead81018f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:09.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.937+0000 7feae0608700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead8101e30 con 0x7fead81014e0 2024-04-12T05:28:09.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.937+0000 7feade3a4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fead81014e0 0x7fead81018f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:09.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.937+0000 7feade3a4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fead81014e0 0x7fead81018f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57106/0 (socket says 172.21.15.32:57106) 2024-04-12T05:28:09.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.937+0000 7feade3a4700 1 -- 172.21.15.32:0/3100516089 learned_addr learned my addr 172.21.15.32:0/3100516089 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:09.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.937+0000 7feade3a4700 1 -- 172.21.15.32:0/3100516089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead8101f70 con 0x7fead81014e0 2024-04-12T05:28:09.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.938+0000 7feade3a4700 1 --2- 172.21.15.32:0/3100516089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fead81014e0 0x7fead81018f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fead40098e0 tx=0x7fead4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb22be7fdf8a64bd server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:09.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.938+0000 7feadd3a2700 1 -- 172.21.15.32:0/3100516089 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fead400b860 con 0x7fead81014e0 2024-04-12T05:28:09.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.938+0000 7feadd3a2700 1 -- 172.21.15.32:0/3100516089 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fead400be70 con 0x7fead81014e0 2024-04-12T05:28:09.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.938+0000 7feadd3a2700 1 -- 172.21.15.32:0/3100516089 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fead4010380 con 0x7fead81014e0 2024-04-12T05:28:09.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.938+0000 7feae0608700 1 -- 172.21.15.32:0/3100516089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fead81014e0 msgr2=0x7fead81018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:09.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.938+0000 7feae0608700 1 --2- 172.21.15.32:0/3100516089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fead81014e0 0x7fead81018f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fead40098e0 tx=0x7fead4009250 comp rx=0 tx=0).stop 2024-04-12T05:28:09.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.938+0000 7feae0608700 1 -- 172.21.15.32:0/3100516089 shutdown_connections 2024-04-12T05:28:09.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.938+0000 7feae0608700 1 --2- 172.21.15.32:0/3100516089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fead81014e0 0x7fead81018f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:09.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.938+0000 7feae0608700 1 -- 172.21.15.32:0/3100516089 >> 172.21.15.32:0/3100516089 conn(0x7fead80fcc60 msgr2=0x7fead80ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:09.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.938+0000 7feae0608700 1 -- 172.21.15.32:0/3100516089 shutdown_connections 2024-04-12T05:28:09.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.939+0000 7feae0608700 1 -- 172.21.15.32:0/3100516089 wait complete. 2024-04-12T05:28:09.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.939+0000 7feae0608700 1 Processor -- start 2024-04-12T05:28:09.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.939+0000 7feae0608700 1 -- start start 2024-04-12T05:28:09.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.939+0000 7feae0608700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fead81014e0 0x7fead8113570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:09.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.939+0000 7feae0608700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead8113ab0 con 0x7fead81014e0 2024-04-12T05:28:09.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.939+0000 7feade3a4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fead81014e0 0x7fead8113570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:09.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.939+0000 7feade3a4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fead81014e0 0x7fead8113570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57110/0 (socket says 172.21.15.32:57110) 2024-04-12T05:28:09.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.939+0000 7feade3a4700 1 -- 172.21.15.32:0/2336614205 learned_addr learned my addr 172.21.15.32:0/2336614205 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:09.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.940+0000 7feade3a4700 1 -- 172.21.15.32:0/2336614205 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead40094f0 con 0x7fead81014e0 2024-04-12T05:28:09.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.940+0000 7feade3a4700 1 --2- 172.21.15.32:0/2336614205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fead81014e0 0x7fead8113570 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fead4011ca0 tx=0x7fead4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:09.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.940+0000 7feaceffd700 1 -- 172.21.15.32:0/2336614205 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fead4010850 con 0x7fead81014e0 2024-04-12T05:28:09.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.940+0000 7feaceffd700 1 -- 172.21.15.32:0/2336614205 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fead4010e60 con 0x7fead81014e0 2024-04-12T05:28:09.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.940+0000 7feae0608700 1 -- 172.21.15.32:0/2336614205 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fead8113cb0 con 0x7fead81014e0 2024-04-12T05:28:09.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.940+0000 7feae0608700 1 -- 172.21.15.32:0/2336614205 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fead8114820 con 0x7fead81014e0 2024-04-12T05:28:09.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.940+0000 7feaceffd700 1 -- 172.21.15.32:0/2336614205 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fead401ae20 con 0x7fead81014e0 2024-04-12T05:28:09.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.941+0000 7feaccff9700 1 -- 172.21.15.32:0/2336614205 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feac00052b0 con 0x7fead81014e0 2024-04-12T05:28:09.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.941+0000 7feaceffd700 1 -- 172.21.15.32:0/2336614205 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fead402d3f0 con 0x7fead81014e0 2024-04-12T05:28:09.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.941+0000 7feaceffd700 1 --2- 172.21.15.32:0/2336614205 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7feac4036810 0x7feac4038cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:09.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.941+0000 7feaceffd700 1 -- 172.21.15.32:0/2336614205 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fead404a440 con 0x7fead81014e0 2024-04-12T05:28:09.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.941+0000 7feaddba3700 1 -- 172.21.15.32:0/2336614205 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7feac4036810 msgr2=0x7feac4038cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/1962278696 2024-04-12T05:28:09.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.941+0000 7feaddba3700 1 --2- 172.21.15.32:0/2336614205 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7feac4036810 0x7feac4038cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-12T05:28:09.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:08.946+0000 7feaceffd700 1 -- 172.21.15.32:0/2336614205 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fead402b4d0 con 0x7fead81014e0 2024-04-12T05:28:09.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:09.142+0000 7feaddba3700 1 -- 172.21.15.32:0/2336614205 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7feac4036810 msgr2=0x7feac4038cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/1962278696 2024-04-12T05:28:09.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:09.142+0000 7feaddba3700 1 --2- 172.21.15.32:0/2336614205 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7feac4036810 0x7feac4038cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-12T05:28:09.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:09.197+0000 7feaccff9700 1 -- 172.21.15.32:0/2336614205 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7feac0005d70 con 0x7fead81014e0 2024-04-12T05:28:09.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:09.197+0000 7feaceffd700 1 -- 172.21.15.32:0/2336614205 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7fead402b2f0 con 0x7fead81014e0 2024-04-12T05:28:09.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:09.199+0000 7feaccff9700 1 -- 172.21.15.32:0/2336614205 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7feac4036810 msgr2=0x7feac4038cc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T05:28:09.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:09.199+0000 7feaccff9700 1 --2- 172.21.15.32:0/2336614205 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7feac4036810 0x7feac4038cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:09.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:09.199+0000 7feaccff9700 1 -- 172.21.15.32:0/2336614205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fead81014e0 msgr2=0x7fead8113570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:09.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:09.199+0000 7feaccff9700 1 --2- 172.21.15.32:0/2336614205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fead81014e0 0x7fead8113570 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fead4011ca0 tx=0x7fead4009250 comp rx=0 tx=0).stop 2024-04-12T05:28:09.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:09.199+0000 7feaccff9700 1 -- 172.21.15.32:0/2336614205 shutdown_connections 2024-04-12T05:28:09.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:09.199+0000 7feaccff9700 1 --2- 172.21.15.32:0/2336614205 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7feac4036810 0x7feac4038cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:09.833 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:09 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/2336614205' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-12T05:28:09.834 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:09.199+0000 7feaccff9700 1 --2- 172.21.15.32:0/2336614205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fead81014e0 0x7fead8113570 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:09.834 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:09.199+0000 7feaccff9700 1 -- 172.21.15.32:0/2336614205 >> 172.21.15.32:0/2336614205 conn(0x7fead80fcc60 msgr2=0x7fead8105970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:09.834 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:09.200+0000 7feaccff9700 1 -- 172.21.15.32:0/2336614205 shutdown_connections 2024-04-12T05:28:09.834 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:09.200+0000 7feaccff9700 1 -- 172.21.15.32:0/2336614205 wait complete. 2024-04-12T05:28:09.834 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for the mgr to restart... 2024-04-12T05:28:09.834 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mgr epoch 9... 2024-04-12T05:28:14.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:14 smithi032 ceph-mon[39731]: Active manager daemon smithi032.hrwxrh restarted 2024-04-12T05:28:14.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:14 smithi032 ceph-mon[39731]: Activating manager daemon smithi032.hrwxrh 2024-04-12T05:28:14.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:14 smithi032 ceph-mon[39731]: osdmap e3: 0 total, 0 up, 0 in 2024-04-12T05:28:14.595 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:14 smithi032 ceph-mon[39731]: mgrmap e10: smithi032.hrwxrh(active, starting, since 0.00313412s) 2024-04-12T05:28:14.595 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:14 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-12T05:28:14.595 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:14 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr metadata", "who": "smithi032.hrwxrh", "id": "smithi032.hrwxrh"}]: dispatch 2024-04-12T05:28:14.595 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:14 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T05:28:14.595 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:14 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T05:28:14.595 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:14 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T05:28:14.595 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:14 smithi032 ceph-mon[39731]: Manager daemon smithi032.hrwxrh is now available 2024-04-12T05:28:14.595 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:14 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:28:14.595 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:14 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:28:15.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:15 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.hrwxrh/mirror_snapshot_schedule"}]: dispatch 2024-04-12T05:28:15.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:15 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.hrwxrh/trash_purge_schedule"}]: dispatch 2024-04-12T05:28:15.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:15 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:15.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:15 smithi032 ceph-mon[39731]: mgrmap e11: smithi032.hrwxrh(active, since 1.00531s) 2024-04-12T05:28:16.207 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-12T05:28:16.207 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-12T05:28:16.207 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-12T05:28:16.207 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-12T05:28:16.207 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.373+0000 7fe2e1de4700 1 Processor -- start 2024-04-12T05:28:16.207 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.373+0000 7fe2e1de4700 1 -- start start 2024-04-12T05:28:16.207 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.373+0000 7fe2e1de4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2dc102580 0x7fe2dc102990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:16.207 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.373+0000 7fe2e1de4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc102ed0 con 0x7fe2dc102580 2024-04-12T05:28:16.207 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.373+0000 7fe2db7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2dc102580 0x7fe2dc102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:16.207 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.373+0000 7fe2db7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2dc102580 0x7fe2dc102990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57118/0 (socket says 172.21.15.32:57118) 2024-04-12T05:28:16.207 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.373+0000 7fe2db7fe700 1 -- 172.21.15.32:0/464900451 learned_addr learned my addr 172.21.15.32:0/464900451 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:16.208 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.374+0000 7fe2db7fe700 1 -- 172.21.15.32:0/464900451 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2dc103010 con 0x7fe2dc102580 2024-04-12T05:28:16.208 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.374+0000 7fe2db7fe700 1 --2- 172.21.15.32:0/464900451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2dc102580 0x7fe2dc102990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc0098e0 tx=0x7fe2cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb172f81f2071545 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:16.208 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.374+0000 7fe2da7fc700 1 -- 172.21.15.32:0/464900451 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2cc00b860 con 0x7fe2dc102580 2024-04-12T05:28:16.208 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.374+0000 7fe2da7fc700 1 -- 172.21.15.32:0/464900451 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe2cc00be70 con 0x7fe2dc102580 2024-04-12T05:28:16.208 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.374+0000 7fe2da7fc700 1 -- 172.21.15.32:0/464900451 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2cc010380 con 0x7fe2dc102580 2024-04-12T05:28:16.208 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.374+0000 7fe2e1de4700 1 -- 172.21.15.32:0/464900451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2dc102580 msgr2=0x7fe2dc102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:16.208 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.374+0000 7fe2e1de4700 1 --2- 172.21.15.32:0/464900451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2dc102580 0x7fe2dc102990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc0098e0 tx=0x7fe2cc009250 comp rx=0 tx=0).stop 2024-04-12T05:28:16.208 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.374+0000 7fe2e1de4700 1 -- 172.21.15.32:0/464900451 shutdown_connections 2024-04-12T05:28:16.208 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.374+0000 7fe2e1de4700 1 --2- 172.21.15.32:0/464900451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2dc102580 0x7fe2dc102990 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:16.208 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.374+0000 7fe2e1de4700 1 -- 172.21.15.32:0/464900451 >> 172.21.15.32:0/464900451 conn(0x7fe2dc0fdd00 msgr2=0x7fe2dc100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:16.208 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.375+0000 7fe2e1de4700 1 -- 172.21.15.32:0/464900451 shutdown_connections 2024-04-12T05:28:16.208 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.375+0000 7fe2e1de4700 1 -- 172.21.15.32:0/464900451 wait complete. 2024-04-12T05:28:16.208 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.375+0000 7fe2e1de4700 1 Processor -- start 2024-04-12T05:28:16.208 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.376+0000 7fe2e1de4700 1 -- start start 2024-04-12T05:28:16.208 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.376+0000 7fe2e1de4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2dc102580 0x7fe2dc111380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:16.209 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.376+0000 7fe2e1de4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc1118c0 con 0x7fe2dc102580 2024-04-12T05:28:16.209 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.376+0000 7fe2db7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2dc102580 0x7fe2dc111380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:16.209 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.376+0000 7fe2db7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2dc102580 0x7fe2dc111380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57120/0 (socket says 172.21.15.32:57120) 2024-04-12T05:28:16.209 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.376+0000 7fe2db7fe700 1 -- 172.21.15.32:0/616554069 learned_addr learned my addr 172.21.15.32:0/616554069 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:16.209 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.376+0000 7fe2db7fe700 1 -- 172.21.15.32:0/616554069 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2cc0094f0 con 0x7fe2dc102580 2024-04-12T05:28:16.209 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.376+0000 7fe2db7fe700 1 --2- 172.21.15.32:0/616554069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2dc102580 0x7fe2dc111380 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc011ca0 tx=0x7fe2cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:16.209 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.376+0000 7fe2c3fff700 1 -- 172.21.15.32:0/616554069 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2cc010850 con 0x7fe2dc102580 2024-04-12T05:28:16.209 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.377+0000 7fe2c3fff700 1 -- 172.21.15.32:0/616554069 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe2cc010e60 con 0x7fe2dc102580 2024-04-12T05:28:16.209 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.377+0000 7fe2e1de4700 1 -- 172.21.15.32:0/616554069 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2dc111ac0 con 0x7fe2dc102580 2024-04-12T05:28:16.209 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.377+0000 7fe2c3fff700 1 -- 172.21.15.32:0/616554069 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2cc01ae20 con 0x7fe2dc102580 2024-04-12T05:28:16.209 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.377+0000 7fe2e1de4700 1 -- 172.21.15.32:0/616554069 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2dc112630 con 0x7fe2dc102580 2024-04-12T05:28:16.209 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.377+0000 7fe2c3fff700 1 -- 172.21.15.32:0/616554069 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fe2cc02d3f0 con 0x7fe2dc102580 2024-04-12T05:28:16.209 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.378+0000 7fe2c3fff700 1 --2- 172.21.15.32:0/616554069 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fe2c40367e0 0x7fe2c4038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:16.209 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.378+0000 7fe2e1de4700 1 -- 172.21.15.32:0/616554069 --> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2dc05f5f0 con 0x7fe2c40367e0 2024-04-12T05:28:16.210 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.378+0000 7fe2daffd700 1 -- 172.21.15.32:0/616554069 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fe2c40367e0 msgr2=0x7fe2c4038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/1962278696 2024-04-12T05:28:16.210 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.378+0000 7fe2c3fff700 1 -- 172.21.15.32:0/616554069 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe2cc04b450 con 0x7fe2dc102580 2024-04-12T05:28:16.210 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.378+0000 7fe2daffd700 1 --2- 172.21.15.32:0/616554069 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fe2c40367e0 0x7fe2c4038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-12T05:28:16.210 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.579+0000 7fe2daffd700 1 -- 172.21.15.32:0/616554069 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fe2c40367e0 msgr2=0x7fe2c4038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/1962278696 2024-04-12T05:28:16.210 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.579+0000 7fe2daffd700 1 --2- 172.21.15.32:0/616554069 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fe2c40367e0 0x7fe2c4038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-12T05:28:16.210 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.980+0000 7fe2daffd700 1 -- 172.21.15.32:0/616554069 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fe2c40367e0 msgr2=0x7fe2c4038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/1962278696 2024-04-12T05:28:16.210 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:10.980+0000 7fe2daffd700 1 --2- 172.21.15.32:0/616554069 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fe2c40367e0 0x7fe2c4038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-12T05:28:16.210 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:11.780+0000 7fe2daffd700 1 -- 172.21.15.32:0/616554069 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fe2c40367e0 msgr2=0x7fe2c4038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/1962278696 2024-04-12T05:28:16.210 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:11.780+0000 7fe2daffd700 1 --2- 172.21.15.32:0/616554069 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fe2c40367e0 0x7fe2c4038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-12T05:28:16.210 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:13.381+0000 7fe2daffd700 1 -- 172.21.15.32:0/616554069 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fe2c40367e0 msgr2=0x7fe2c4038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/1962278696 2024-04-12T05:28:16.210 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:13.381+0000 7fe2daffd700 1 --2- 172.21.15.32:0/616554069 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fe2c40367e0 0x7fe2c4038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-12T05:28:16.210 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:14.471+0000 7fe2c3fff700 1 -- 172.21.15.32:0/616554069 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mgrmap(e 10) v1 ==== 43148+0+0 (secure 0 0 0) 0x7fe2cc02d740 con 0x7fe2dc102580 2024-04-12T05:28:16.210 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:14.471+0000 7fe2c3fff700 1 -- 172.21.15.32:0/616554069 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fe2c40367e0 msgr2=0x7fe2c4038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T05:28:16.210 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:14.471+0000 7fe2c3fff700 1 --2- 172.21.15.32:0/616554069 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fe2c40367e0 0x7fe2c4038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:16.210 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:15.472+0000 7fe2c3fff700 1 -- 172.21.15.32:0/616554069 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7fe2cc02dac0 con 0x7fe2dc102580 2024-04-12T05:28:16.211 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:15.473+0000 7fe2c3fff700 1 --2- 172.21.15.32:0/616554069 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fe2c4039ff0 0x7fe2c403c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:16.211 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:15.473+0000 7fe2c3fff700 1 -- 172.21.15.32:0/616554069 --> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2dc05f5f0 con 0x7fe2c4039ff0 2024-04-12T05:28:16.211 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:15.473+0000 7fe2daffd700 1 --2- 172.21.15.32:0/616554069 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fe2c4039ff0 0x7fe2c403c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:16.211 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:15.474+0000 7fe2daffd700 1 --2- 172.21.15.32:0/616554069 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fe2c4039ff0 0x7fe2c403c3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe2d00039d0 tx=0x7fe2d00092f0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:16.211 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:15.474+0000 7fe2c3fff700 1 -- 172.21.15.32:0/616554069 <== mgr.14160 v2:172.21.15.32:6800/3971961003 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7fe2dc05f5f0 con 0x7fe2c4039ff0 2024-04-12T05:28:16.211 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:15.480+0000 7fe2e1de4700 1 -- 172.21.15.32:0/616554069 --> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fe2dc111f50 con 0x7fe2c4039ff0 2024-04-12T05:28:16.211 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:15.481+0000 7fe2c3fff700 1 -- 172.21.15.32:0/616554069 <== mgr.14160 v2:172.21.15.32:6800/3971961003 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fe2dc111f50 con 0x7fe2c4039ff0 2024-04-12T05:28:16.211 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:15.481+0000 7fe2e1de4700 1 -- 172.21.15.32:0/616554069 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fe2c4039ff0 msgr2=0x7fe2c403c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:16.211 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:15.481+0000 7fe2e1de4700 1 --2- 172.21.15.32:0/616554069 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fe2c4039ff0 0x7fe2c403c3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe2d00039d0 tx=0x7fe2d00092f0 comp rx=0 tx=0).stop 2024-04-12T05:28:16.211 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:15.481+0000 7fe2e1de4700 1 -- 172.21.15.32:0/616554069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2dc102580 msgr2=0x7fe2dc111380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:16.211 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:15.481+0000 7fe2e1de4700 1 --2- 172.21.15.32:0/616554069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2dc102580 0x7fe2dc111380 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc011ca0 tx=0x7fe2cc009250 comp rx=0 tx=0).stop 2024-04-12T05:28:16.211 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:15.481+0000 7fe2e1de4700 1 -- 172.21.15.32:0/616554069 shutdown_connections 2024-04-12T05:28:16.211 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:15.481+0000 7fe2e1de4700 1 --2- 172.21.15.32:0/616554069 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fe2c4039ff0 0x7fe2c403c3d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:16.211 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:15.481+0000 7fe2e1de4700 1 --2- 172.21.15.32:0/616554069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2dc102580 0x7fe2dc111380 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:16.211 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:15.481+0000 7fe2e1de4700 1 --2- 172.21.15.32:0/616554069 >> [v2:172.21.15.32:6800/1962278696,v1:172.21.15.32:6801/1962278696] conn(0x7fe2c40367e0 0x7fe2c4038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:16.212 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:15.481+0000 7fe2e1de4700 1 -- 172.21.15.32:0/616554069 >> 172.21.15.32:0/616554069 conn(0x7fe2dc0fdd00 msgr2=0x7fe2dc0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:16.212 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:15.481+0000 7fe2e1de4700 1 -- 172.21.15.32:0/616554069 shutdown_connections 2024-04-12T05:28:16.212 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:15.481+0000 7fe2e1de4700 1 -- 172.21.15.32:0/616554069 wait complete. 2024-04-12T05:28:16.212 INFO:teuthology.orchestra.run.smithi032.stdout:mgr epoch 9 is available 2024-04-12T05:28:16.212 INFO:teuthology.orchestra.run.smithi032.stdout:Generating a dashboard self-signed certificate... 2024-04-12T05:28:16.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:16 smithi032 ceph-mon[39731]: [12/Apr/2024:05:28:15] ENGINE Bus STARTING 2024-04-12T05:28:16.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:16 smithi032 ceph-mon[39731]: [12/Apr/2024:05:28:15] ENGINE Serving on https://172.21.15.32:7150 2024-04-12T05:28:16.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:16 smithi032 ceph-mon[39731]: [12/Apr/2024:05:28:15] ENGINE Bus STARTED 2024-04-12T05:28:16.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:16 smithi032 ceph-mon[39731]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-12T05:28:16.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:16 smithi032 ceph-mon[39731]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-12T05:28:16.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:16 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:18.094 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:17 smithi032 ceph-mon[39731]: mgrmap e12: smithi032.hrwxrh(active, since 2s) 2024-04-12T05:28:18.094 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:17 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:18.094 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:17 smithi032 ceph-mon[39731]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:28:18.094 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:17 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:18.094 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:17 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:19.111 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-12T05:28:19.111 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.824+0000 7fb9ae374700 1 Processor -- start 2024-04-12T05:28:19.111 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.824+0000 7fb9ae374700 1 -- start start 2024-04-12T05:28:19.111 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.825+0000 7fb9ae374700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9a8102580 0x7fb9a8102990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:19.111 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.825+0000 7fb9ae374700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a8102ed0 con 0x7fb9a8102580 2024-04-12T05:28:19.111 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.825+0000 7fb9a7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9a8102580 0x7fb9a8102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:19.111 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.825+0000 7fb9a7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9a8102580 0x7fb9a8102990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40848/0 (socket says 172.21.15.32:40848) 2024-04-12T05:28:19.111 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.825+0000 7fb9a7fff700 1 -- 172.21.15.32:0/3678870252 learned_addr learned my addr 172.21.15.32:0/3678870252 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:19.111 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.825+0000 7fb9a7fff700 1 -- 172.21.15.32:0/3678870252 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a8103010 con 0x7fb9a8102580 2024-04-12T05:28:19.111 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.825+0000 7fb9a7fff700 1 --2- 172.21.15.32:0/3678870252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9a8102580 0x7fb9a8102990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb9980098e0 tx=0x7fb998009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38eed39f2b54884e server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:19.111 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.826+0000 7fb9a6ffd700 1 -- 172.21.15.32:0/3678870252 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb99800b860 con 0x7fb9a8102580 2024-04-12T05:28:19.112 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.826+0000 7fb9a6ffd700 1 -- 172.21.15.32:0/3678870252 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb99800be70 con 0x7fb9a8102580 2024-04-12T05:28:19.112 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.826+0000 7fb9a6ffd700 1 -- 172.21.15.32:0/3678870252 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb998010380 con 0x7fb9a8102580 2024-04-12T05:28:19.112 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.826+0000 7fb9ae374700 1 -- 172.21.15.32:0/3678870252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9a8102580 msgr2=0x7fb9a8102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:19.112 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.826+0000 7fb9ae374700 1 --2- 172.21.15.32:0/3678870252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9a8102580 0x7fb9a8102990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb9980098e0 tx=0x7fb998009250 comp rx=0 tx=0).stop 2024-04-12T05:28:19.112 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.826+0000 7fb9ae374700 1 -- 172.21.15.32:0/3678870252 shutdown_connections 2024-04-12T05:28:19.112 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.826+0000 7fb9ae374700 1 --2- 172.21.15.32:0/3678870252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9a8102580 0x7fb9a8102990 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:19.112 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.826+0000 7fb9ae374700 1 -- 172.21.15.32:0/3678870252 >> 172.21.15.32:0/3678870252 conn(0x7fb9a80fdd00 msgr2=0x7fb9a8100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:19.112 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.826+0000 7fb9ae374700 1 -- 172.21.15.32:0/3678870252 shutdown_connections 2024-04-12T05:28:19.112 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.826+0000 7fb9ae374700 1 -- 172.21.15.32:0/3678870252 wait complete. 2024-04-12T05:28:19.112 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.827+0000 7fb9ae374700 1 Processor -- start 2024-04-12T05:28:19.112 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.827+0000 7fb9ae374700 1 -- start start 2024-04-12T05:28:19.112 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.827+0000 7fb9ae374700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9a8102580 0x7fb9a8111380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:19.112 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.827+0000 7fb9ae374700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a81118c0 con 0x7fb9a8102580 2024-04-12T05:28:19.113 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.827+0000 7fb9a7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9a8102580 0x7fb9a8111380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:19.113 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.827+0000 7fb9a7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9a8102580 0x7fb9a8111380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40862/0 (socket says 172.21.15.32:40862) 2024-04-12T05:28:19.113 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.827+0000 7fb9a7fff700 1 -- 172.21.15.32:0/1996875572 learned_addr learned my addr 172.21.15.32:0/1996875572 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:19.113 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.828+0000 7fb9a7fff700 1 -- 172.21.15.32:0/1996875572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9980094f0 con 0x7fb9a8102580 2024-04-12T05:28:19.113 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.828+0000 7fb9a7fff700 1 --2- 172.21.15.32:0/1996875572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9a8102580 0x7fb9a8111380 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb99800c010 tx=0x7fb998009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:19.113 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.828+0000 7fb9a4ff9700 1 -- 172.21.15.32:0/1996875572 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb998010960 con 0x7fb9a8102580 2024-04-12T05:28:19.113 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.828+0000 7fb9a4ff9700 1 -- 172.21.15.32:0/1996875572 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb99801a430 con 0x7fb9a8102580 2024-04-12T05:28:19.113 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.828+0000 7fb9a4ff9700 1 -- 172.21.15.32:0/1996875572 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb998019580 con 0x7fb9a8102580 2024-04-12T05:28:19.113 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.828+0000 7fb9ae374700 1 -- 172.21.15.32:0/1996875572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9a8111ac0 con 0x7fb9a8102580 2024-04-12T05:28:19.113 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.828+0000 7fb9ae374700 1 -- 172.21.15.32:0/1996875572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9a8112630 con 0x7fb9a8102580 2024-04-12T05:28:19.113 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.829+0000 7fb9ae374700 1 -- 172.21.15.32:0/1996875572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9a805f5f0 con 0x7fb9a8102580 2024-04-12T05:28:19.114 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.829+0000 7fb9a4ff9700 1 -- 172.21.15.32:0/1996875572 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7fb99801a980 con 0x7fb9a8102580 2024-04-12T05:28:19.114 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.830+0000 7fb9a4ff9700 1 --2- 172.21.15.32:0/1996875572 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fb9900366b0 0x7fb990038b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:19.114 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.830+0000 7fb9a4ff9700 1 -- 172.21.15.32:0/1996875572 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb99804a630 con 0x7fb9a8102580 2024-04-12T05:28:19.114 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.830+0000 7fb9a77fe700 1 --2- 172.21.15.32:0/1996875572 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fb9900366b0 0x7fb990038b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:19.114 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.830+0000 7fb9a77fe700 1 --2- 172.21.15.32:0/1996875572 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fb9900366b0 0x7fb990038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb99c009990 tx=0x7fb99c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:19.114 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.834+0000 7fb9a4ff9700 1 -- 172.21.15.32:0/1996875572 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fb998018d20 con 0x7fb9a8102580 2024-04-12T05:28:19.114 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:16.836+0000 7fb9a4ff9700 1 -- 172.21.15.32:0/1996875572 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fb998049860 con 0x7fb9a8102580 2024-04-12T05:28:19.114 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:17.047+0000 7fb9ae374700 1 -- 172.21.15.32:0/1996875572 --> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9a80fef40 con 0x7fb9900366b0 2024-04-12T05:28:19.114 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:17.210+0000 7fb9a4ff9700 1 -- 172.21.15.32:0/1996875572 <== mgr.14160 v2:172.21.15.32:6800/3971961003 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fb9a80fef40 con 0x7fb9900366b0 2024-04-12T05:28:19.115 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:17.212+0000 7fb9ae374700 1 -- 172.21.15.32:0/1996875572 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fb9900366b0 msgr2=0x7fb990038b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:19.115 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:17.212+0000 7fb9ae374700 1 --2- 172.21.15.32:0/1996875572 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fb9900366b0 0x7fb990038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb99c009990 tx=0x7fb99c006e40 comp rx=0 tx=0).stop 2024-04-12T05:28:19.115 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:17.212+0000 7fb9ae374700 1 -- 172.21.15.32:0/1996875572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9a8102580 msgr2=0x7fb9a8111380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:19.115 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:17.212+0000 7fb9ae374700 1 --2- 172.21.15.32:0/1996875572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9a8102580 0x7fb9a8111380 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb99800c010 tx=0x7fb998009250 comp rx=0 tx=0).stop 2024-04-12T05:28:19.115 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:17.212+0000 7fb9ae374700 1 -- 172.21.15.32:0/1996875572 shutdown_connections 2024-04-12T05:28:19.115 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:17.212+0000 7fb9ae374700 1 --2- 172.21.15.32:0/1996875572 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fb9900366b0 0x7fb990038b60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:19.115 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:17.212+0000 7fb9ae374700 1 --2- 172.21.15.32:0/1996875572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9a8102580 0x7fb9a8111380 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:19.115 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:17.212+0000 7fb9ae374700 1 -- 172.21.15.32:0/1996875572 >> 172.21.15.32:0/1996875572 conn(0x7fb9a80fdd00 msgr2=0x7fb9a80fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:19.115 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:17.212+0000 7fb9ae374700 1 -- 172.21.15.32:0/1996875572 shutdown_connections 2024-04-12T05:28:19.115 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:17.213+0000 7fb9ae374700 1 -- 172.21.15.32:0/1996875572 wait complete. 2024-04-12T05:28:19.115 INFO:teuthology.orchestra.run.smithi032.stdout:Creating initial admin user... 2024-04-12T05:28:20.733 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$6knaW7x97tEy4Aaa.lb7f.VBSxoKbIQNn.cCRL6xlKANYBWa/byEG", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1712899700, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-12T05:28:20.734 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.617+0000 7f85f12ad700 1 Processor -- start 2024-04-12T05:28:20.734 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.617+0000 7f85f12ad700 1 -- start start 2024-04-12T05:28:20.734 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.617+0000 7f85f12ad700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ec102660 0x7f85ec102a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:20.734 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.617+0000 7f85f12ad700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec102fb0 con 0x7f85ec102660 2024-04-12T05:28:20.734 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.617+0000 7f85eaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ec102660 0x7f85ec102a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:20.734 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.617+0000 7f85eaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ec102660 0x7f85ec102a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40876/0 (socket says 172.21.15.32:40876) 2024-04-12T05:28:20.734 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.617+0000 7f85eaffd700 1 -- 172.21.15.32:0/1420120149 learned_addr learned my addr 172.21.15.32:0/1420120149 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:20.734 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.617+0000 7f85eaffd700 1 -- 172.21.15.32:0/1420120149 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ec1030f0 con 0x7f85ec102660 2024-04-12T05:28:20.734 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.617+0000 7f85eaffd700 1 --2- 172.21.15.32:0/1420120149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ec102660 0x7f85ec102a70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f85dc0098e0 tx=0x7f85dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63a77eed045fa5f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:20.734 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.618+0000 7f85e9ffb700 1 -- 172.21.15.32:0/1420120149 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85dc00b860 con 0x7f85ec102660 2024-04-12T05:28:20.734 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.618+0000 7f85e9ffb700 1 -- 172.21.15.32:0/1420120149 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f85dc00be70 con 0x7f85ec102660 2024-04-12T05:28:20.734 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.618+0000 7f85e9ffb700 1 -- 172.21.15.32:0/1420120149 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85dc010380 con 0x7f85ec102660 2024-04-12T05:28:20.734 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.618+0000 7f85f12ad700 1 -- 172.21.15.32:0/1420120149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ec102660 msgr2=0x7f85ec102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:20.734 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.618+0000 7f85f12ad700 1 --2- 172.21.15.32:0/1420120149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ec102660 0x7f85ec102a70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f85dc0098e0 tx=0x7f85dc009250 comp rx=0 tx=0).stop 2024-04-12T05:28:20.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.618+0000 7f85f12ad700 1 -- 172.21.15.32:0/1420120149 shutdown_connections 2024-04-12T05:28:20.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.618+0000 7f85f12ad700 1 --2- 172.21.15.32:0/1420120149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ec102660 0x7f85ec102a70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:20.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.618+0000 7f85f12ad700 1 -- 172.21.15.32:0/1420120149 >> 172.21.15.32:0/1420120149 conn(0x7f85ec0fdde0 msgr2=0x7f85ec1001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:20.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.618+0000 7f85f12ad700 1 -- 172.21.15.32:0/1420120149 shutdown_connections 2024-04-12T05:28:20.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.618+0000 7f85f12ad700 1 -- 172.21.15.32:0/1420120149 wait complete. 2024-04-12T05:28:20.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.619+0000 7f85f12ad700 1 Processor -- start 2024-04-12T05:28:20.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.619+0000 7f85f12ad700 1 -- start start 2024-04-12T05:28:20.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.620+0000 7f85f12ad700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ec102660 0x7f85ec1113b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:20.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.620+0000 7f85f12ad700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec1118f0 con 0x7f85ec102660 2024-04-12T05:28:20.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.620+0000 7f85eaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ec102660 0x7f85ec1113b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:20.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.620+0000 7f85eaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ec102660 0x7f85ec1113b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40884/0 (socket says 172.21.15.32:40884) 2024-04-12T05:28:20.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.620+0000 7f85eaffd700 1 -- 172.21.15.32:0/444897763 learned_addr learned my addr 172.21.15.32:0/444897763 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:20.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.620+0000 7f85eaffd700 1 -- 172.21.15.32:0/444897763 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85dc0094f0 con 0x7f85ec102660 2024-04-12T05:28:20.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.620+0000 7f85eaffd700 1 --2- 172.21.15.32:0/444897763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ec102660 0x7f85ec1113b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f85dc011ca0 tx=0x7f85dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:20.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.620+0000 7f85d37fe700 1 -- 172.21.15.32:0/444897763 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85dc010850 con 0x7f85ec102660 2024-04-12T05:28:20.736 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.620+0000 7f85d37fe700 1 -- 172.21.15.32:0/444897763 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f85dc010e60 con 0x7f85ec102660 2024-04-12T05:28:20.736 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.620+0000 7f85f12ad700 1 -- 172.21.15.32:0/444897763 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85ec111af0 con 0x7f85ec102660 2024-04-12T05:28:20.736 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.620+0000 7f85d37fe700 1 -- 172.21.15.32:0/444897763 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85dc01ae20 con 0x7f85ec102660 2024-04-12T05:28:20.736 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.620+0000 7f85f12ad700 1 -- 172.21.15.32:0/444897763 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85ec112660 con 0x7f85ec102660 2024-04-12T05:28:20.736 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.621+0000 7f85d37fe700 1 -- 172.21.15.32:0/444897763 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f85dc02d3f0 con 0x7f85ec102660 2024-04-12T05:28:20.736 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.621+0000 7f85f12ad700 1 -- 172.21.15.32:0/444897763 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85ec05f5f0 con 0x7f85ec102660 2024-04-12T05:28:20.736 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.621+0000 7f85d37fe700 1 --2- 172.21.15.32:0/444897763 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f85d40367e0 0x7f85d4038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:20.736 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.621+0000 7f85d37fe700 1 -- 172.21.15.32:0/444897763 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f85dc04a610 con 0x7f85ec102660 2024-04-12T05:28:20.736 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.622+0000 7f85ea7fc700 1 --2- 172.21.15.32:0/444897763 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f85d40367e0 0x7f85d4038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:20.736 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.622+0000 7f85ea7fc700 1 --2- 172.21.15.32:0/444897763 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f85d40367e0 0x7f85d4038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f85e0009990 tx=0x7f85e0006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:20.736 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.626+0000 7f85d37fe700 1 -- 172.21.15.32:0/444897763 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f85dc02b8e0 con 0x7f85ec102660 2024-04-12T05:28:20.736 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:19.840+0000 7f85f12ad700 1 -- 172.21.15.32:0/444897763 --> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f85ec05f410 con 0x7f85d40367e0 2024-04-12T05:28:20.736 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:20.085+0000 7f85d37fe700 1 -- 172.21.15.32:0/444897763 <== mgr.14160 v2:172.21.15.32:6800/3971961003 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f85ec05f410 con 0x7f85d40367e0 2024-04-12T05:28:20.736 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:20.088+0000 7f85f12ad700 1 -- 172.21.15.32:0/444897763 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f85d40367e0 msgr2=0x7f85d4038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:20.737 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:20.088+0000 7f85f12ad700 1 --2- 172.21.15.32:0/444897763 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f85d40367e0 0x7f85d4038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f85e0009990 tx=0x7f85e0006e40 comp rx=0 tx=0).stop 2024-04-12T05:28:20.737 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:20.088+0000 7f85f12ad700 1 -- 172.21.15.32:0/444897763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ec102660 msgr2=0x7f85ec1113b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:20.737 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:20.088+0000 7f85f12ad700 1 --2- 172.21.15.32:0/444897763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ec102660 0x7f85ec1113b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f85dc011ca0 tx=0x7f85dc009250 comp rx=0 tx=0).stop 2024-04-12T05:28:20.737 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:20.088+0000 7f85f12ad700 1 -- 172.21.15.32:0/444897763 shutdown_connections 2024-04-12T05:28:20.737 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:20.088+0000 7f85f12ad700 1 --2- 172.21.15.32:0/444897763 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f85d40367e0 0x7f85d4038c90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:20.737 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:20.088+0000 7f85f12ad700 1 --2- 172.21.15.32:0/444897763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ec102660 0x7f85ec1113b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:20.737 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:20.088+0000 7f85f12ad700 1 -- 172.21.15.32:0/444897763 >> 172.21.15.32:0/444897763 conn(0x7f85ec0fdde0 msgr2=0x7f85ec0fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:20.737 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:20.088+0000 7f85f12ad700 1 -- 172.21.15.32:0/444897763 shutdown_connections 2024-04-12T05:28:20.737 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:20.088+0000 7f85f12ad700 1 -- 172.21.15.32:0/444897763 wait complete. 2024-04-12T05:28:20.737 INFO:teuthology.orchestra.run.smithi032.stdout:Fetching dashboard port number... 2024-04-12T05:28:21.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:21 smithi032 ceph-mon[39731]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:28:21.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:21 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:22.098 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 8443 2024-04-12T05:28:22.098 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.304+0000 7f04880c5700 1 Processor -- start 2024-04-12T05:28:22.098 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.304+0000 7f04880c5700 1 -- start start 2024-04-12T05:28:22.098 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.304+0000 7f04880c5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0480102a00 0x7f0480102e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:22.098 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.304+0000 7f04880c5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0480103350 con 0x7f0480102a00 2024-04-12T05:28:22.098 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.304+0000 7f0485e61700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0480102a00 0x7f0480102e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:22.098 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.304+0000 7f0485e61700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0480102a00 0x7f0480102e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40890/0 (socket says 172.21.15.32:40890) 2024-04-12T05:28:22.099 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.304+0000 7f0485e61700 1 -- 172.21.15.32:0/3710750557 learned_addr learned my addr 172.21.15.32:0/3710750557 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:22.099 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.305+0000 7f0485e61700 1 -- 172.21.15.32:0/3710750557 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0480103490 con 0x7f0480102a00 2024-04-12T05:28:22.099 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.305+0000 7f0485e61700 1 --2- 172.21.15.32:0/3710750557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0480102a00 0x7f0480102e10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f047c00bb80 tx=0x7f047c00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abcf2f6f2f2c4082 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:22.099 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.305+0000 7f0484e5f700 1 -- 172.21.15.32:0/3710750557 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f047c00dd00 con 0x7f0480102a00 2024-04-12T05:28:22.099 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.305+0000 7f0484e5f700 1 -- 172.21.15.32:0/3710750557 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f047c00de60 con 0x7f0480102a00 2024-04-12T05:28:22.099 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.305+0000 7f0484e5f700 1 -- 172.21.15.32:0/3710750557 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f047c0188d0 con 0x7f0480102a00 2024-04-12T05:28:22.099 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.306+0000 7f04880c5700 1 -- 172.21.15.32:0/3710750557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0480102a00 msgr2=0x7f0480102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:22.099 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.306+0000 7f04880c5700 1 --2- 172.21.15.32:0/3710750557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0480102a00 0x7f0480102e10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f047c00bb80 tx=0x7f047c00b4f0 comp rx=0 tx=0).stop 2024-04-12T05:28:22.099 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.306+0000 7f04880c5700 1 -- 172.21.15.32:0/3710750557 shutdown_connections 2024-04-12T05:28:22.099 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.306+0000 7f04880c5700 1 --2- 172.21.15.32:0/3710750557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0480102a00 0x7f0480102e10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:22.099 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.306+0000 7f04880c5700 1 -- 172.21.15.32:0/3710750557 >> 172.21.15.32:0/3710750557 conn(0x7f04800fc2b0 msgr2=0x7f04800fe6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:22.099 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.306+0000 7f04880c5700 1 -- 172.21.15.32:0/3710750557 shutdown_connections 2024-04-12T05:28:22.099 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.306+0000 7f04880c5700 1 -- 172.21.15.32:0/3710750557 wait complete. 2024-04-12T05:28:22.099 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.306+0000 7f04880c5700 1 Processor -- start 2024-04-12T05:28:22.099 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.306+0000 7f04880c5700 1 -- start start 2024-04-12T05:28:22.099 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.306+0000 7f04880c5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0480102a00 0x7f0480113550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:22.100 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.307+0000 7f04880c5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0480113a90 con 0x7f0480102a00 2024-04-12T05:28:22.100 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.307+0000 7f0485e61700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0480102a00 0x7f0480113550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:22.100 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.307+0000 7f0485e61700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0480102a00 0x7f0480113550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40892/0 (socket says 172.21.15.32:40892) 2024-04-12T05:28:22.100 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.307+0000 7f0485e61700 1 -- 172.21.15.32:0/2544688074 learned_addr learned my addr 172.21.15.32:0/2544688074 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:22.100 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.307+0000 7f0485e61700 1 -- 172.21.15.32:0/2544688074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f047c00b790 con 0x7f0480102a00 2024-04-12T05:28:22.100 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.307+0000 7f0485e61700 1 --2- 172.21.15.32:0/2544688074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0480102a00 0x7f0480113550 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f047c018460 tx=0x7f047c00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:22.100 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.307+0000 7f04767fc700 1 -- 172.21.15.32:0/2544688074 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f047c010c00 con 0x7f0480102a00 2024-04-12T05:28:22.100 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.307+0000 7f04767fc700 1 -- 172.21.15.32:0/2544688074 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f047c010d60 con 0x7f0480102a00 2024-04-12T05:28:22.100 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.307+0000 7f04880c5700 1 -- 172.21.15.32:0/2544688074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0480113c90 con 0x7f0480102a00 2024-04-12T05:28:22.100 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.307+0000 7f04767fc700 1 -- 172.21.15.32:0/2544688074 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f047c021550 con 0x7f0480102a00 2024-04-12T05:28:22.100 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.307+0000 7f04880c5700 1 -- 172.21.15.32:0/2544688074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0480114800 con 0x7f0480102a00 2024-04-12T05:28:22.100 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.308+0000 7f04767fc700 1 -- 172.21.15.32:0/2544688074 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f047c0216b0 con 0x7f0480102a00 2024-04-12T05:28:22.100 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.308+0000 7f04767fc700 1 --2- 172.21.15.32:0/2544688074 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f046c0368d0 0x7f046c038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:22.100 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.308+0000 7f04767fc700 1 -- 172.21.15.32:0/2544688074 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f047c014070 con 0x7f0480102a00 2024-04-12T05:28:22.100 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.308+0000 7f04880c5700 1 -- 172.21.15.32:0/2544688074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f048005f5f0 con 0x7f0480102a00 2024-04-12T05:28:22.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.308+0000 7f0485660700 1 --2- 172.21.15.32:0/2544688074 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f046c0368d0 0x7f046c038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:22.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.309+0000 7f0485660700 1 --2- 172.21.15.32:0/2544688074 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f046c0368d0 0x7f046c038d80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0470009990 tx=0x7f0470006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:22.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.313+0000 7f04767fc700 1 -- 172.21.15.32:0/2544688074 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f047c026070 con 0x7f0480102a00 2024-04-12T05:28:22.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.506+0000 7f04880c5700 1 -- 172.21.15.32:0/2544688074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f048004bed0 con 0x7f0480102a00 2024-04-12T05:28:22.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.506+0000 7f04767fc700 1 -- 172.21.15.32:0/2544688074 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f047c00ecf0 con 0x7f0480102a00 2024-04-12T05:28:22.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.508+0000 7f04880c5700 1 -- 172.21.15.32:0/2544688074 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f046c0368d0 msgr2=0x7f046c038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:22.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.508+0000 7f04880c5700 1 --2- 172.21.15.32:0/2544688074 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f046c0368d0 0x7f046c038d80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0470009990 tx=0x7f0470006e40 comp rx=0 tx=0).stop 2024-04-12T05:28:22.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.508+0000 7f04880c5700 1 -- 172.21.15.32:0/2544688074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0480102a00 msgr2=0x7f0480113550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:22.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.508+0000 7f04880c5700 1 --2- 172.21.15.32:0/2544688074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0480102a00 0x7f0480113550 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f047c018460 tx=0x7f047c00b4f0 comp rx=0 tx=0).stop 2024-04-12T05:28:22.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.508+0000 7f04880c5700 1 -- 172.21.15.32:0/2544688074 shutdown_connections 2024-04-12T05:28:22.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.508+0000 7f04880c5700 1 --2- 172.21.15.32:0/2544688074 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f046c0368d0 0x7f046c038d80 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:22.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.508+0000 7f04880c5700 1 --2- 172.21.15.32:0/2544688074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0480102a00 0x7f0480113550 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:22.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.508+0000 7f04880c5700 1 -- 172.21.15.32:0/2544688074 >> 172.21.15.32:0/2544688074 conn(0x7f04800fc2b0 msgr2=0x7f0480105950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:22.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.508+0000 7f04880c5700 1 -- 172.21.15.32:0/2544688074 shutdown_connections 2024-04-12T05:28:22.101 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:21.508+0000 7f04880c5700 1 -- 172.21.15.32:0/2544688074 wait complete. 2024-04-12T05:28:22.106 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-12T05:28:22.106 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout disabled 2024-04-12T05:28:22.114 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-12T05:28:22.115 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-12T05:28:22.115 INFO:teuthology.orchestra.run.smithi032.stdout:firewalld.service is not enabled 2024-04-12T05:28:22.115 INFO:teuthology.orchestra.run.smithi032.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-12T05:28:22.117 INFO:teuthology.orchestra.run.smithi032.stdout:Ceph Dashboard is now available at: 2024-04-12T05:28:22.117 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:28:22.117 INFO:teuthology.orchestra.run.smithi032.stdout: URL: https://smithi032.front.sepia.ceph.com:8443/ 2024-04-12T05:28:22.117 INFO:teuthology.orchestra.run.smithi032.stdout: User: admin 2024-04-12T05:28:22.117 INFO:teuthology.orchestra.run.smithi032.stdout: Password: vrkc6af9nx 2024-04-12T05:28:22.117 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:28:22.117 INFO:teuthology.orchestra.run.smithi032.stdout:Saving cluster configuration to /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config directory 2024-04-12T05:28:22.118 INFO:teuthology.orchestra.run.smithi032.stdout:Enabling autotune for osd_memory_target 2024-04-12T05:28:22.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:22 smithi032 ceph-mon[39731]: mgrmap e13: smithi032.hrwxrh(active, since 6s) 2024-04-12T05:28:22.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:22 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/2544688074' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-12T05:28:23.538 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.732+0000 7f4158f29700 1 Processor -- start 2024-04-12T05:28:23.538 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.733+0000 7f4158f29700 1 -- start start 2024-04-12T05:28:23.538 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.733+0000 7f4158f29700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4154103bb0 0x7f4154103fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:23.538 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.733+0000 7f4158f29700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4154104500 con 0x7f4154103bb0 2024-04-12T05:28:23.538 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.733+0000 7f415259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4154103bb0 0x7f4154103fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:23.538 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.733+0000 7f415259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4154103bb0 0x7f4154103fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53864/0 (socket says 172.21.15.32:53864) 2024-04-12T05:28:23.538 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.733+0000 7f415259c700 1 -- 172.21.15.32:0/2201861204 learned_addr learned my addr 172.21.15.32:0/2201861204 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:23.538 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.733+0000 7f415259c700 1 -- 172.21.15.32:0/2201861204 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4154104640 con 0x7f4154103bb0 2024-04-12T05:28:23.538 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.733+0000 7f415259c700 1 --2- 172.21.15.32:0/2201861204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4154103bb0 0x7f4154103fc0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f41440098e0 tx=0x7f4144009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e82dc578094b3cfb server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:23.539 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.734+0000 7f415159a700 1 -- 172.21.15.32:0/2201861204 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f414400b860 con 0x7f4154103bb0 2024-04-12T05:28:23.539 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.734+0000 7f415159a700 1 -- 172.21.15.32:0/2201861204 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f414400be70 con 0x7f4154103bb0 2024-04-12T05:28:23.539 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.734+0000 7f415159a700 1 -- 172.21.15.32:0/2201861204 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4144010380 con 0x7f4154103bb0 2024-04-12T05:28:23.539 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.734+0000 7f4158f29700 1 -- 172.21.15.32:0/2201861204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4154103bb0 msgr2=0x7f4154103fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:23.539 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.734+0000 7f4158f29700 1 --2- 172.21.15.32:0/2201861204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4154103bb0 0x7f4154103fc0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f41440098e0 tx=0x7f4144009250 comp rx=0 tx=0).stop 2024-04-12T05:28:23.539 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.734+0000 7f4158f29700 1 -- 172.21.15.32:0/2201861204 shutdown_connections 2024-04-12T05:28:23.539 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.734+0000 7f4158f29700 1 --2- 172.21.15.32:0/2201861204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4154103bb0 0x7f4154103fc0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:23.539 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.734+0000 7f4158f29700 1 -- 172.21.15.32:0/2201861204 >> 172.21.15.32:0/2201861204 conn(0x7f41540fc2b0 msgr2=0x7f41540fe6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:23.539 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.734+0000 7f4158f29700 1 -- 172.21.15.32:0/2201861204 shutdown_connections 2024-04-12T05:28:23.539 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.734+0000 7f4158f29700 1 -- 172.21.15.32:0/2201861204 wait complete. 2024-04-12T05:28:23.539 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.735+0000 7f4158f29700 1 Processor -- start 2024-04-12T05:28:23.539 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.735+0000 7f4158f29700 1 -- start start 2024-04-12T05:28:23.539 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.735+0000 7f4158f29700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4154103bb0 0x7f41541032c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:23.539 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.735+0000 7f4158f29700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4154103800 con 0x7f4154103bb0 2024-04-12T05:28:23.539 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.735+0000 7f415259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4154103bb0 0x7f41541032c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:23.540 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.735+0000 7f415259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4154103bb0 0x7f41541032c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53876/0 (socket says 172.21.15.32:53876) 2024-04-12T05:28:23.540 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.735+0000 7f415259c700 1 -- 172.21.15.32:0/4173702880 learned_addr learned my addr 172.21.15.32:0/4173702880 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:23.540 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.736+0000 7f415259c700 1 -- 172.21.15.32:0/4173702880 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41440094f0 con 0x7f4154103bb0 2024-04-12T05:28:23.540 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.736+0000 7f415259c700 1 --2- 172.21.15.32:0/4173702880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4154103bb0 0x7f41541032c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f414400c010 tx=0x7f4144009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:23.540 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.736+0000 7f414affd700 1 -- 172.21.15.32:0/4173702880 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4144010960 con 0x7f4154103bb0 2024-04-12T05:28:23.540 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.736+0000 7f414affd700 1 -- 172.21.15.32:0/4173702880 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f414401a430 con 0x7f4154103bb0 2024-04-12T05:28:23.540 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.736+0000 7f414affd700 1 -- 172.21.15.32:0/4173702880 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4144019580 con 0x7f4154103bb0 2024-04-12T05:28:23.540 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.736+0000 7f4158f29700 1 -- 172.21.15.32:0/4173702880 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4154101870 con 0x7f4154103bb0 2024-04-12T05:28:23.540 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.736+0000 7f4158f29700 1 -- 172.21.15.32:0/4173702880 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4154101d60 con 0x7f4154103bb0 2024-04-12T05:28:23.540 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.737+0000 7f414affd700 1 -- 172.21.15.32:0/4173702880 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f41440197b0 con 0x7f4154103bb0 2024-04-12T05:28:23.540 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.737+0000 7f414affd700 1 --2- 172.21.15.32:0/4173702880 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f41400368d0 0x7f4140038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:23.540 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.737+0000 7f414affd700 1 -- 172.21.15.32:0/4173702880 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f414404a920 con 0x7f4154103bb0 2024-04-12T05:28:23.540 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.737+0000 7f4151d9b700 1 --2- 172.21.15.32:0/4173702880 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f41400368d0 0x7f4140038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:23.540 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.737+0000 7f4158f29700 1 -- 172.21.15.32:0/4173702880 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f415404bed0 con 0x7f4154103bb0 2024-04-12T05:28:23.540 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.737+0000 7f4151d9b700 1 --2- 172.21.15.32:0/4173702880 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f41400368d0 0x7f4140038d80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f413c009990 tx=0x7f413c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:23.541 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.742+0000 7f414affd700 1 -- 172.21.15.32:0/4173702880 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4144019b50 con 0x7f4154103bb0 2024-04-12T05:28:23.541 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.935+0000 7f4158f29700 1 -- 172.21.15.32:0/4173702880 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f4154102240 con 0x7f4154103bb0 2024-04-12T05:28:23.541 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.935+0000 7f414affd700 1 -- 172.21.15.32:0/4173702880 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f414402b300 con 0x7f4154103bb0 2024-04-12T05:28:23.541 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.937+0000 7f4158f29700 1 -- 172.21.15.32:0/4173702880 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f41400368d0 msgr2=0x7f4140038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:23.541 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.937+0000 7f4158f29700 1 --2- 172.21.15.32:0/4173702880 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f41400368d0 0x7f4140038d80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f413c009990 tx=0x7f413c006e40 comp rx=0 tx=0).stop 2024-04-12T05:28:23.541 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.937+0000 7f4158f29700 1 -- 172.21.15.32:0/4173702880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4154103bb0 msgr2=0x7f41541032c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:23.541 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.937+0000 7f4158f29700 1 --2- 172.21.15.32:0/4173702880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4154103bb0 0x7f41541032c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f414400c010 tx=0x7f4144009250 comp rx=0 tx=0).stop 2024-04-12T05:28:23.541 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.937+0000 7f4158f29700 1 -- 172.21.15.32:0/4173702880 shutdown_connections 2024-04-12T05:28:23.541 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.937+0000 7f4158f29700 1 --2- 172.21.15.32:0/4173702880 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f41400368d0 0x7f4140038d80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:23.541 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.937+0000 7f4158f29700 1 --2- 172.21.15.32:0/4173702880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4154103bb0 0x7f41541032c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:23.541 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.937+0000 7f4158f29700 1 -- 172.21.15.32:0/4173702880 >> 172.21.15.32:0/4173702880 conn(0x7f41540fc2b0 msgr2=0x7f41540fcdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:23.541 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.937+0000 7f4158f29700 1 -- 172.21.15.32:0/4173702880 shutdown_connections 2024-04-12T05:28:23.541 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:22.937+0000 7f4158f29700 1 -- 172.21.15.32:0/4173702880 wait complete. 2024-04-12T05:28:24.960 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.075+0000 7f036e6cf700 1 Processor -- start 2024-04-12T05:28:24.960 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.075+0000 7f036e6cf700 1 -- start start 2024-04-12T05:28:24.960 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.075+0000 7f036e6cf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0368076c30 0x7f03681051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:24.960 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.075+0000 7f036e6cf700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03681056e0 con 0x7f0368076c30 2024-04-12T05:28:24.961 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.075+0000 7f0367fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0368076c30 0x7f03681051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:24.961 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.075+0000 7f0367fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0368076c30 0x7f03681051a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53884/0 (socket says 172.21.15.32:53884) 2024-04-12T05:28:24.961 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.075+0000 7f0367fff700 1 -- 172.21.15.32:0/242985373 learned_addr learned my addr 172.21.15.32:0/242985373 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:24.961 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.076+0000 7f0367fff700 1 -- 172.21.15.32:0/242985373 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0368105820 con 0x7f0368076c30 2024-04-12T05:28:24.961 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.076+0000 7f0367fff700 1 --2- 172.21.15.32:0/242985373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0368076c30 0x7f03681051a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f03580098e0 tx=0x7f0358009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=351939cb41de9117 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:24.961 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.076+0000 7f0366ffd700 1 -- 172.21.15.32:0/242985373 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f035800b860 con 0x7f0368076c30 2024-04-12T05:28:24.961 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.076+0000 7f0366ffd700 1 -- 172.21.15.32:0/242985373 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f035800be70 con 0x7f0368076c30 2024-04-12T05:28:24.961 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.076+0000 7f0366ffd700 1 -- 172.21.15.32:0/242985373 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0358010380 con 0x7f0368076c30 2024-04-12T05:28:24.961 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.076+0000 7f036e6cf700 1 -- 172.21.15.32:0/242985373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0368076c30 msgr2=0x7f03681051a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:24.961 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.076+0000 7f036e6cf700 1 --2- 172.21.15.32:0/242985373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0368076c30 0x7f03681051a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f03580098e0 tx=0x7f0358009250 comp rx=0 tx=0).stop 2024-04-12T05:28:24.961 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.076+0000 7f036e6cf700 1 -- 172.21.15.32:0/242985373 shutdown_connections 2024-04-12T05:28:24.961 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.076+0000 7f036e6cf700 1 --2- 172.21.15.32:0/242985373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0368076c30 0x7f03681051a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:24.961 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.076+0000 7f036e6cf700 1 -- 172.21.15.32:0/242985373 >> 172.21.15.32:0/242985373 conn(0x7f03680fcc30 msgr2=0x7f03680ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:24.961 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.077+0000 7f036e6cf700 1 -- 172.21.15.32:0/242985373 shutdown_connections 2024-04-12T05:28:24.961 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.077+0000 7f036e6cf700 1 -- 172.21.15.32:0/242985373 wait complete. 2024-04-12T05:28:24.962 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.077+0000 7f036e6cf700 1 Processor -- start 2024-04-12T05:28:24.962 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.078+0000 7f036e6cf700 1 -- start start 2024-04-12T05:28:24.962 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.078+0000 7f036e6cf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0368076c30 0x7f03681113b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:24.962 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.078+0000 7f036e6cf700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03681118f0 con 0x7f0368076c30 2024-04-12T05:28:24.962 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.078+0000 7f0367fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0368076c30 0x7f03681113b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:24.962 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.078+0000 7f0367fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0368076c30 0x7f03681113b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53900/0 (socket says 172.21.15.32:53900) 2024-04-12T05:28:24.962 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.078+0000 7f0367fff700 1 -- 172.21.15.32:0/1070074157 learned_addr learned my addr 172.21.15.32:0/1070074157 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:24.962 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.078+0000 7f0367fff700 1 -- 172.21.15.32:0/1070074157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03580094f0 con 0x7f0368076c30 2024-04-12T05:28:24.962 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.078+0000 7f0367fff700 1 --2- 172.21.15.32:0/1070074157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0368076c30 0x7f03681113b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f03580094c0 tx=0x7f0358009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:24.962 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.078+0000 7f0364ff9700 1 -- 172.21.15.32:0/1070074157 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0358010850 con 0x7f0368076c30 2024-04-12T05:28:24.962 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.079+0000 7f036e6cf700 1 -- 172.21.15.32:0/1070074157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0368111af0 con 0x7f0368076c30 2024-04-12T05:28:24.962 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.079+0000 7f0364ff9700 1 -- 172.21.15.32:0/1070074157 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0358010e60 con 0x7f0368076c30 2024-04-12T05:28:24.962 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.079+0000 7f0364ff9700 1 -- 172.21.15.32:0/1070074157 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f035801a430 con 0x7f0368076c30 2024-04-12T05:28:24.963 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.079+0000 7f036e6cf700 1 -- 172.21.15.32:0/1070074157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0368112660 con 0x7f0368076c30 2024-04-12T05:28:24.963 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.079+0000 7f0364ff9700 1 -- 172.21.15.32:0/1070074157 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f035801a6b0 con 0x7f0368076c30 2024-04-12T05:28:24.963 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.080+0000 7f036e6cf700 1 -- 172.21.15.32:0/1070074157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f036805f5f0 con 0x7f0368076c30 2024-04-12T05:28:24.963 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.080+0000 7f0364ff9700 1 --2- 172.21.15.32:0/1070074157 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f0350036880 0x7f0350038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:24.963 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.080+0000 7f0364ff9700 1 -- 172.21.15.32:0/1070074157 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f035804a840 con 0x7f0368076c30 2024-04-12T05:28:24.963 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.080+0000 7f03677fe700 1 --2- 172.21.15.32:0/1070074157 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f0350036880 0x7f0350038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:24.963 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.080+0000 7f03677fe700 1 --2- 172.21.15.32:0/1070074157 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f0350036880 0x7f0350038d30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f035c009990 tx=0x7f035c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:24.963 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.084+0000 7f0364ff9700 1 -- 172.21.15.32:0/1070074157 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f03580189a0 con 0x7f0368076c30 2024-04-12T05:28:24.963 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.387+0000 7f036e6cf700 1 -- 172.21.15.32:0/1070074157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f036804bed0 con 0x7f0368076c30 2024-04-12T05:28:24.963 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.389+0000 7f0364ff9700 1 -- 172.21.15.32:0/1070074157 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7f0358020e90 con 0x7f0368076c30 2024-04-12T05:28:24.963 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.391+0000 7f036e6cf700 1 -- 172.21.15.32:0/1070074157 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f0350036880 msgr2=0x7f0350038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:24.963 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.391+0000 7f036e6cf700 1 --2- 172.21.15.32:0/1070074157 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f0350036880 0x7f0350038d30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f035c009990 tx=0x7f035c006e40 comp rx=0 tx=0).stop 2024-04-12T05:28:24.963 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.391+0000 7f036e6cf700 1 -- 172.21.15.32:0/1070074157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0368076c30 msgr2=0x7f03681113b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:24.963 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.391+0000 7f036e6cf700 1 --2- 172.21.15.32:0/1070074157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0368076c30 0x7f03681113b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f03580094c0 tx=0x7f0358009250 comp rx=0 tx=0).stop 2024-04-12T05:28:24.963 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.391+0000 7f036e6cf700 1 -- 172.21.15.32:0/1070074157 shutdown_connections 2024-04-12T05:28:24.964 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.391+0000 7f036e6cf700 1 --2- 172.21.15.32:0/1070074157 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f0350036880 0x7f0350038d30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:24.964 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.391+0000 7f036e6cf700 1 --2- 172.21.15.32:0/1070074157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0368076c30 0x7f03681113b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:24.964 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.391+0000 7f036e6cf700 1 -- 172.21.15.32:0/1070074157 >> 172.21.15.32:0/1070074157 conn(0x7f03680fcc30 msgr2=0x7f03680fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:24.964 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.391+0000 7f036e6cf700 1 -- 172.21.15.32:0/1070074157 shutdown_connections 2024-04-12T05:28:24.964 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-12T05:28:24.391+0000 7f036e6cf700 1 -- 172.21.15.32:0/1070074157 wait complete. 2024-04-12T05:28:24.964 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-12T05:28:24.964 INFO:teuthology.orchestra.run.smithi032.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-12T05:28:24.964 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:28:24.964 INFO:teuthology.orchestra.run.smithi032.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-12T05:28:24.964 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:28:24.964 INFO:teuthology.orchestra.run.smithi032.stdout:Or, if you are only running a single cluster on this host: 2024-04-12T05:28:24.964 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:28:24.964 INFO:teuthology.orchestra.run.smithi032.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-12T05:28:24.964 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:28:24.964 INFO:teuthology.orchestra.run.smithi032.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-12T05:28:24.965 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:28:24.965 INFO:teuthology.orchestra.run.smithi032.stdout: ceph telemetry on 2024-04-12T05:28:24.965 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:28:24.965 INFO:teuthology.orchestra.run.smithi032.stdout:For more information see: 2024-04-12T05:28:24.965 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:28:24.965 INFO:teuthology.orchestra.run.smithi032.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-04-12T05:28:24.965 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:28:24.965 INFO:teuthology.orchestra.run.smithi032.stdout:Bootstrap complete. 2024-04-12T05:28:24.994 INFO:tasks.cephadm:Fetching config... 2024-04-12T05:28:24.994 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-12T05:28:24.995 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-12T05:28:25.015 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-12T05:28:25.015 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-12T05:28:25.015 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-12T05:28:25.075 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-12T05:28:25.075 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-12T05:28:25.075 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/keyring of=/dev/stdout 2024-04-12T05:28:25.143 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-12T05:28:25.143 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-12T05:28:25.143 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-12T05:28:25.203 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-12T05:28:25.203 DEBUG:teuthology.orchestra.run.smithi032:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDi3bXu0Cy8xhr5T9PkF+z5FhEu/yuN+ZOK946TKkeBAiP61LERkkWB5r2v+pXNfxy/84+rZuxipOcTA9AgK3p4yfhJa4LWOTJGJmRoIDFa4V54+xsoAiPMLI40V+GLfUwJk9OaQB+JHYE2o0tvQrwNckCQKwODQ121HRpZngzXjtru4cTYAXwmRNikEFPShLp2zzJRF/0O7PxvBDO4pvHFV1zh1b0AMFtJ62ukPyBSZnJXzwfvvh7a2MdHpr7cavkhcCBe8VK4o/QpuXi6mE/jZhI7yww6RN+7hb6g296KA0agDnHAu5eoDdF1PgrowtSBg3y4ODNUySXfjo0v1ISG2Apo8HdXb3b2NoaW3BGDjWYosSsj26wVhFCa+9aMyuTre9WPjMfewPL8ShlkraGNTy3kawZUCVuoIPRN9Fl2MYKekSO8+59aPgQi4WCYj5sqT1s7bUqMSMIr17Xa1hoR23Y3SoeDzbGx92BSTm+tLUxYH5JByKGEHBoSxuVt2hM= ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-12T05:28:25.280 INFO:teuthology.orchestra.run.smithi032.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDi3bXu0Cy8xhr5T9PkF+z5FhEu/yuN+ZOK946TKkeBAiP61LERkkWB5r2v+pXNfxy/84+rZuxipOcTA9AgK3p4yfhJa4LWOTJGJmRoIDFa4V54+xsoAiPMLI40V+GLfUwJk9OaQB+JHYE2o0tvQrwNckCQKwODQ121HRpZngzXjtru4cTYAXwmRNikEFPShLp2zzJRF/0O7PxvBDO4pvHFV1zh1b0AMFtJ62ukPyBSZnJXzwfvvh7a2MdHpr7cavkhcCBe8VK4o/QpuXi6mE/jZhI7yww6RN+7hb6g296KA0agDnHAu5eoDdF1PgrowtSBg3y4ODNUySXfjo0v1ISG2Apo8HdXb3b2NoaW3BGDjWYosSsj26wVhFCa+9aMyuTre9WPjMfewPL8ShlkraGNTy3kawZUCVuoIPRN9Fl2MYKekSO8+59aPgQi4WCYj5sqT1s7bUqMSMIr17Xa1hoR23Y3SoeDzbGx92BSTm+tLUxYH5JByKGEHBoSxuVt2hM= ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968 2024-04-12T05:28:25.290 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDi3bXu0Cy8xhr5T9PkF+z5FhEu/yuN+ZOK946TKkeBAiP61LERkkWB5r2v+pXNfxy/84+rZuxipOcTA9AgK3p4yfhJa4LWOTJGJmRoIDFa4V54+xsoAiPMLI40V+GLfUwJk9OaQB+JHYE2o0tvQrwNckCQKwODQ121HRpZngzXjtru4cTYAXwmRNikEFPShLp2zzJRF/0O7PxvBDO4pvHFV1zh1b0AMFtJ62ukPyBSZnJXzwfvvh7a2MdHpr7cavkhcCBe8VK4o/QpuXi6mE/jZhI7yww6RN+7hb6g296KA0agDnHAu5eoDdF1PgrowtSBg3y4ODNUySXfjo0v1ISG2Apo8HdXb3b2NoaW3BGDjWYosSsj26wVhFCa+9aMyuTre9WPjMfewPL8ShlkraGNTy3kawZUCVuoIPRN9Fl2MYKekSO8+59aPgQi4WCYj5sqT1s7bUqMSMIr17Xa1hoR23Y3SoeDzbGx92BSTm+tLUxYH5JByKGEHBoSxuVt2hM= ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-12T05:28:25.337 INFO:teuthology.orchestra.run.smithi067.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDi3bXu0Cy8xhr5T9PkF+z5FhEu/yuN+ZOK946TKkeBAiP61LERkkWB5r2v+pXNfxy/84+rZuxipOcTA9AgK3p4yfhJa4LWOTJGJmRoIDFa4V54+xsoAiPMLI40V+GLfUwJk9OaQB+JHYE2o0tvQrwNckCQKwODQ121HRpZngzXjtru4cTYAXwmRNikEFPShLp2zzJRF/0O7PxvBDO4pvHFV1zh1b0AMFtJ62ukPyBSZnJXzwfvvh7a2MdHpr7cavkhcCBe8VK4o/QpuXi6mE/jZhI7yww6RN+7hb6g296KA0agDnHAu5eoDdF1PgrowtSBg3y4ODNUySXfjo0v1ISG2Apo8HdXb3b2NoaW3BGDjWYosSsj26wVhFCa+9aMyuTre9WPjMfewPL8ShlkraGNTy3kawZUCVuoIPRN9Fl2MYKekSO8+59aPgQi4WCYj5sqT1s7bUqMSMIr17Xa1hoR23Y3SoeDzbGx92BSTm+tLUxYH5JByKGEHBoSxuVt2hM= ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968 2024-04-12T05:28:25.350 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-12T05:28:25.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:25 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/1070074157' entity='client.admin' 2024-04-12T05:28:25.677 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:28:27.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.532+0000 7f0ec7bf2700 1 -- 172.21.15.32:0/3402642822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ec00fd050 msgr2=0x7f0ec00fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:27.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.532+0000 7f0ec7bf2700 1 --2- 172.21.15.32:0/3402642822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ec00fd050 0x7f0ec00fd460 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc0048c0 tx=0x7f0ebc009e30 comp rx=0 tx=0).stop 2024-04-12T05:28:27.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.532+0000 7f0ec7bf2700 1 -- 172.21.15.32:0/3402642822 shutdown_connections 2024-04-12T05:28:27.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.532+0000 7f0ec7bf2700 1 --2- 172.21.15.32:0/3402642822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ec00fd050 0x7f0ec00fd460 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:27.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.532+0000 7f0ec7bf2700 1 -- 172.21.15.32:0/3402642822 >> 172.21.15.32:0/3402642822 conn(0x7f0ec00f70d0 msgr2=0x7f0ec00f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:27.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.532+0000 7f0ec7bf2700 1 -- 172.21.15.32:0/3402642822 shutdown_connections 2024-04-12T05:28:27.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.532+0000 7f0ec7bf2700 1 -- 172.21.15.32:0/3402642822 wait complete. 2024-04-12T05:28:27.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.533+0000 7f0ec7bf2700 1 Processor -- start 2024-04-12T05:28:27.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.533+0000 7f0ec7bf2700 1 -- start start 2024-04-12T05:28:27.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.533+0000 7f0ec7bf2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ec00fd050 0x7f0ec010bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:27.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.533+0000 7f0ec7bf2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec010c460 con 0x7f0ec00fd050 2024-04-12T05:28:27.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.533+0000 7f0ec598e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ec00fd050 0x7f0ec010bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:27.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.533+0000 7f0ec598e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ec00fd050 0x7f0ec010bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53906/0 (socket says 172.21.15.32:53906) 2024-04-12T05:28:27.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.533+0000 7f0ec598e700 1 -- 172.21.15.32:0/3216602516 learned_addr learned my addr 172.21.15.32:0/3216602516 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:27.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.533+0000 7f0ec598e700 1 -- 172.21.15.32:0/3216602516 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ebc005040 con 0x7f0ec00fd050 2024-04-12T05:28:27.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.533+0000 7f0ec598e700 1 --2- 172.21.15.32:0/3216602516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ec00fd050 0x7f0ec010bf20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc009300 tx=0x7f0ebc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:27.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.534+0000 7f0eb67fc700 1 -- 172.21.15.32:0/3216602516 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ebc009810 con 0x7f0ec00fd050 2024-04-12T05:28:27.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.534+0000 7f0eb67fc700 1 -- 172.21.15.32:0/3216602516 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ebc01e8d0 con 0x7f0ec00fd050 2024-04-12T05:28:27.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.534+0000 7f0ec7bf2700 1 -- 172.21.15.32:0/3216602516 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ec010c660 con 0x7f0ec00fd050 2024-04-12T05:28:27.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.534+0000 7f0eb67fc700 1 -- 172.21.15.32:0/3216602516 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ebc013740 con 0x7f0ec00fd050 2024-04-12T05:28:27.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.534+0000 7f0ec7bf2700 1 -- 172.21.15.32:0/3216602516 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ec010d0f0 con 0x7f0ec00fd050 2024-04-12T05:28:27.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.535+0000 7f0eb67fc700 1 -- 172.21.15.32:0/3216602516 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f0ebc0139b0 con 0x7f0ec00fd050 2024-04-12T05:28:27.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.535+0000 7f0ec7bf2700 1 -- 172.21.15.32:0/3216602516 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ec0101d30 con 0x7f0ec00fd050 2024-04-12T05:28:27.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.535+0000 7f0eb67fc700 1 --2- 172.21.15.32:0/3216602516 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f0eac0368d0 0x7f0eac038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:27.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.535+0000 7f0eb67fc700 1 -- 172.21.15.32:0/3216602516 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0ebc04c630 con 0x7f0ec00fd050 2024-04-12T05:28:27.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.535+0000 7f0ec518d700 1 --2- 172.21.15.32:0/3216602516 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f0eac0368d0 0x7f0eac038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:27.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.535+0000 7f0ec518d700 1 --2- 172.21.15.32:0/3216602516 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f0eac0368d0 0x7f0eac038d80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0009990 tx=0x7f0eb0006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:27.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.539+0000 7f0eb67fc700 1 -- 172.21.15.32:0/3216602516 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f0ebc011780 con 0x7f0ec00fd050 2024-04-12T05:28:27.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.756+0000 7f0ec7bf2700 1 -- 172.21.15.32:0/3216602516 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f0ec010dbc0 con 0x7f0ec00fd050 2024-04-12T05:28:27.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.759+0000 7f0eb67fc700 1 -- 172.21.15.32:0/3216602516 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f0ebc026360 con 0x7f0ec00fd050 2024-04-12T05:28:27.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.762+0000 7f0ec7bf2700 1 -- 172.21.15.32:0/3216602516 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f0eac0368d0 msgr2=0x7f0eac038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:27.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.762+0000 7f0ec7bf2700 1 --2- 172.21.15.32:0/3216602516 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f0eac0368d0 0x7f0eac038d80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0009990 tx=0x7f0eb0006e40 comp rx=0 tx=0).stop 2024-04-12T05:28:27.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.762+0000 7f0ec7bf2700 1 -- 172.21.15.32:0/3216602516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ec00fd050 msgr2=0x7f0ec010bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:27.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.762+0000 7f0ec7bf2700 1 --2- 172.21.15.32:0/3216602516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ec00fd050 0x7f0ec010bf20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc009300 tx=0x7f0ebc009e30 comp rx=0 tx=0).stop 2024-04-12T05:28:27.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.762+0000 7f0ec7bf2700 1 -- 172.21.15.32:0/3216602516 shutdown_connections 2024-04-12T05:28:27.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.762+0000 7f0ec7bf2700 1 --2- 172.21.15.32:0/3216602516 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f0eac0368d0 0x7f0eac038d80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:27.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.762+0000 7f0ec7bf2700 1 --2- 172.21.15.32:0/3216602516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ec00fd050 0x7f0ec010bf20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:27.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.762+0000 7f0ec7bf2700 1 -- 172.21.15.32:0/3216602516 >> 172.21.15.32:0/3216602516 conn(0x7f0ec00f70d0 msgr2=0x7f0ec00f7990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:27.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.762+0000 7f0ec7bf2700 1 -- 172.21.15.32:0/3216602516 shutdown_connections 2024-04-12T05:28:27.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:27.762+0000 7f0ec7bf2700 1 -- 172.21.15.32:0/3216602516 wait complete. 2024-04-12T05:28:28.802 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:28 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:28.802 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:28 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:28.802 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:28 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "who": "osd/host:smithi032", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:28:28.803 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:28 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:28.803 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:28 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi032", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-12T05:28:28.803 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:28 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi032", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-12T05:28:28.803 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:28 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:28:28.803 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:28 smithi032 ceph-mon[39731]: Deploying daemon crash.smithi032 on smithi032 2024-04-12T05:28:28.803 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:28 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/3216602516' entity='client.admin' 2024-04-12T05:28:28.820 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-12T05:28:28.820 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-12T05:28:29.128 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:28:32.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.280+0000 7f96c4379700 1 -- 172.21.15.32:0/25500575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96bc0fca60 msgr2=0x7f96bc0fce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:32.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.280+0000 7f96c4379700 1 --2- 172.21.15.32:0/25500575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96bc0fca60 0x7f96bc0fce70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f96b80048c0 tx=0x7f96b8009e30 comp rx=0 tx=0).stop 2024-04-12T05:28:32.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.280+0000 7f96c4379700 1 -- 172.21.15.32:0/25500575 shutdown_connections 2024-04-12T05:28:32.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.280+0000 7f96c4379700 1 --2- 172.21.15.32:0/25500575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96bc0fca60 0x7f96bc0fce70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:32.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.280+0000 7f96c4379700 1 -- 172.21.15.32:0/25500575 >> 172.21.15.32:0/25500575 conn(0x7f96bc0f81c0 msgr2=0x7f96bc0fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:32.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.280+0000 7f96c4379700 1 -- 172.21.15.32:0/25500575 shutdown_connections 2024-04-12T05:28:32.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.280+0000 7f96c4379700 1 -- 172.21.15.32:0/25500575 wait complete. 2024-04-12T05:28:32.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.281+0000 7f96c4379700 1 Processor -- start 2024-04-12T05:28:32.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.281+0000 7f96c4379700 1 -- start start 2024-04-12T05:28:32.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.281+0000 7f96c4379700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96bc0fca60 0x7f96bc071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:32.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.281+0000 7f96c4379700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96bc0722a0 con 0x7f96bc0fca60 2024-04-12T05:28:32.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.281+0000 7f96c2115700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96bc0fca60 0x7f96bc071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:32.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.281+0000 7f96c2115700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96bc0fca60 0x7f96bc071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44474/0 (socket says 172.21.15.32:44474) 2024-04-12T05:28:32.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.281+0000 7f96c2115700 1 -- 172.21.15.32:0/3458649335 learned_addr learned my addr 172.21.15.32:0/3458649335 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:32.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.281+0000 7f96c2115700 1 -- 172.21.15.32:0/3458649335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b8005040 con 0x7f96bc0fca60 2024-04-12T05:28:32.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.282+0000 7f96c2115700 1 --2- 172.21.15.32:0/3458649335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96bc0fca60 0x7f96bc071d60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f96b8009300 tx=0x7f96b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:32.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.282+0000 7f96b2ffd700 1 -- 172.21.15.32:0/3458649335 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96b8009600 con 0x7f96bc0fca60 2024-04-12T05:28:32.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.282+0000 7f96b2ffd700 1 -- 172.21.15.32:0/3458649335 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f96b8009760 con 0x7f96bc0fca60 2024-04-12T05:28:32.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.282+0000 7f96c4379700 1 -- 172.21.15.32:0/3458649335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96bc070370 con 0x7f96bc0fca60 2024-04-12T05:28:32.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.282+0000 7f96b2ffd700 1 -- 172.21.15.32:0/3458649335 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96b801c5b0 con 0x7f96bc0fca60 2024-04-12T05:28:32.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.282+0000 7f96c4379700 1 -- 172.21.15.32:0/3458649335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96bc070860 con 0x7f96bc0fca60 2024-04-12T05:28:32.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.283+0000 7f96c4379700 1 -- 172.21.15.32:0/3458649335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96bc05f5f0 con 0x7f96bc0fca60 2024-04-12T05:28:32.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.283+0000 7f96b2ffd700 1 -- 172.21.15.32:0/3458649335 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f96b8013960 con 0x7f96bc0fca60 2024-04-12T05:28:32.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.283+0000 7f96b2ffd700 1 --2- 172.21.15.32:0/3458649335 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f96a8036880 0x7f96a8038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:32.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.283+0000 7f96b2ffd700 1 -- 172.21.15.32:0/3458649335 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f96b8049a70 con 0x7f96bc0fca60 2024-04-12T05:28:32.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.285+0000 7f96c1914700 1 --2- 172.21.15.32:0/3458649335 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f96a8036880 0x7f96a8038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:32.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.285+0000 7f96c1914700 1 --2- 172.21.15.32:0/3458649335 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f96a8036880 0x7f96a8038d30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f96ac009990 tx=0x7f96ac006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:32.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.288+0000 7f96b2ffd700 1 -- 172.21.15.32:0/3458649335 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f96b80098d0 con 0x7f96bc0fca60 2024-04-12T05:28:32.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.497+0000 7f96c4379700 1 -- 172.21.15.32:0/3458649335 --> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f96bc106bf0 con 0x7f96a8036880 2024-04-12T05:28:32.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.500+0000 7f96b2ffd700 1 -- 172.21.15.32:0/3458649335 <== mgr.14160 v2:172.21.15.32:6800/3971961003 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f96bc106bf0 con 0x7f96a8036880 2024-04-12T05:28:32.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.502+0000 7f96c4379700 1 -- 172.21.15.32:0/3458649335 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f96a8036880 msgr2=0x7f96a8038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:32.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.502+0000 7f96c4379700 1 --2- 172.21.15.32:0/3458649335 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f96a8036880 0x7f96a8038d30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f96ac009990 tx=0x7f96ac006e40 comp rx=0 tx=0).stop 2024-04-12T05:28:32.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.502+0000 7f96c4379700 1 -- 172.21.15.32:0/3458649335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96bc0fca60 msgr2=0x7f96bc071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:32.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.502+0000 7f96c4379700 1 --2- 172.21.15.32:0/3458649335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96bc0fca60 0x7f96bc071d60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f96b8009300 tx=0x7f96b8009e30 comp rx=0 tx=0).stop 2024-04-12T05:28:32.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.502+0000 7f96c4379700 1 -- 172.21.15.32:0/3458649335 shutdown_connections 2024-04-12T05:28:32.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.502+0000 7f96c4379700 1 --2- 172.21.15.32:0/3458649335 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f96a8036880 0x7f96a8038d30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:32.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.502+0000 7f96c4379700 1 --2- 172.21.15.32:0/3458649335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96bc0fca60 0x7f96bc071d60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:32.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.502+0000 7f96c4379700 1 -- 172.21.15.32:0/3458649335 >> 172.21.15.32:0/3458649335 conn(0x7f96bc0f81c0 msgr2=0x7f96bc0f8a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:32.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.502+0000 7f96c4379700 1 -- 172.21.15.32:0/3458649335 shutdown_connections 2024-04-12T05:28:32.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:32.502+0000 7f96c4379700 1 -- 172.21.15.32:0/3458649335 wait complete. 2024-04-12T05:28:33.236 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi067 2024-04-12T05:28:33.237 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-12T05:28:33.237 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/etc/ceph/ceph.conf 2024-04-12T05:28:33.259 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-12T05:28:33.260 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:28:33.322 INFO:tasks.cephadm:Adding host smithi067 to orchestrator... 2024-04-12T05:28:33.322 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph orch host add smithi067 2024-04-12T05:28:33.672 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:28:33.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:33 smithi032 ceph-mon[39731]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:28:33.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:33 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:33.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:33 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:33.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:33 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:33.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:33 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:33.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:33 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:34.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:34 smithi032 ceph-mon[39731]: Deploying daemon node-exporter.smithi032 on smithi032 2024-04-12T05:28:35.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:35 smithi032 ceph-mon[39731]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:28:35.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:35 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:36.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:36.412+0000 7f23486d8700 1 -- 172.21.15.32:0/1809067762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23400fca50 msgr2=0x7f23400fce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:36.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:36.412+0000 7f23486d8700 1 --2- 172.21.15.32:0/1809067762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23400fca50 0x7f23400fce60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f233c0048c0 tx=0x7f233c009e30 comp rx=0 tx=0).stop 2024-04-12T05:28:36.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:36.412+0000 7f23486d8700 1 -- 172.21.15.32:0/1809067762 shutdown_connections 2024-04-12T05:28:36.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:36.412+0000 7f23486d8700 1 --2- 172.21.15.32:0/1809067762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23400fca50 0x7f23400fce60 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:36.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:36.412+0000 7f23486d8700 1 -- 172.21.15.32:0/1809067762 >> 172.21.15.32:0/1809067762 conn(0x7f23400f8190 msgr2=0x7f23400fa5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:36.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:36.412+0000 7f23486d8700 1 -- 172.21.15.32:0/1809067762 shutdown_connections 2024-04-12T05:28:36.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:36.412+0000 7f23486d8700 1 -- 172.21.15.32:0/1809067762 wait complete. 2024-04-12T05:28:36.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:36.412+0000 7f23486d8700 1 Processor -- start 2024-04-12T05:28:36.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:36.412+0000 7f23486d8700 1 -- start start 2024-04-12T05:28:36.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:36.412+0000 7f23486d8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23400fca50 0x7f2340071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:36.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:36.412+0000 7f23486d8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23400722a0 con 0x7f23400fca50 2024-04-12T05:28:36.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:36.413+0000 7f2346474700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23400fca50 0x7f2340071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:36.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:36.413+0000 7f2346474700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23400fca50 0x7f2340071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44516/0 (socket says 172.21.15.32:44516) 2024-04-12T05:28:36.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:36.413+0000 7f2346474700 1 -- 172.21.15.32:0/1337032914 learned_addr learned my addr 172.21.15.32:0/1337032914 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:36.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:36.413+0000 7f2346474700 1 -- 172.21.15.32:0/1337032914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f233c005040 con 0x7f23400fca50 2024-04-12T05:28:36.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:36.413+0000 7f2346474700 1 --2- 172.21.15.32:0/1337032914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23400fca50 0x7f2340071d60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f233c000c00 tx=0x7f233c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:36.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:36.413+0000 7f2336ffd700 1 -- 172.21.15.32:0/1337032914 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f233c009620 con 0x7f23400fca50 2024-04-12T05:28:36.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:36.413+0000 7f2336ffd700 1 -- 172.21.15.32:0/1337032914 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f233c009780 con 0x7f23400fca50 2024-04-12T05:28:36.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:36.413+0000 7f23486d8700 1 -- 172.21.15.32:0/1337032914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2340070370 con 0x7f23400fca50 2024-04-12T05:28:36.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:36.413+0000 7f2336ffd700 1 -- 172.21.15.32:0/1337032914 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f233c01c5b0 con 0x7f23400fca50 2024-04-12T05:28:36.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:36.413+0000 7f23486d8700 1 -- 172.21.15.32:0/1337032914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2340070860 con 0x7f23400fca50 2024-04-12T05:28:36.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:36.414+0000 7f2336ffd700 1 -- 172.21.15.32:0/1337032914 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f233c01c7a0 con 0x7f23400fca50 2024-04-12T05:28:36.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:36.414+0000 7f23486d8700 1 -- 172.21.15.32:0/1337032914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f234005f5f0 con 0x7f23400fca50 2024-04-12T05:28:36.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:36.414+0000 7f2336ffd700 1 --2- 172.21.15.32:0/1337032914 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f232c0368d0 0x7f232c038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:36.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:36.414+0000 7f2336ffd700 1 -- 172.21.15.32:0/1337032914 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f233c049b70 con 0x7f23400fca50 2024-04-12T05:28:36.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:36.414+0000 7f2345c73700 1 --2- 172.21.15.32:0/1337032914 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f232c0368d0 0x7f232c038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:36.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:36.415+0000 7f2345c73700 1 --2- 172.21.15.32:0/1337032914 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f232c0368d0 0x7f232c038d80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2330009990 tx=0x7f2330006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:36.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:36.419+0000 7f2336ffd700 1 -- 172.21.15.32:0/1337032914 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f233c011db0 con 0x7f23400fca50 2024-04-12T05:28:36.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:36.619+0000 7f23486d8700 1 -- 172.21.15.32:0/1337032914 --> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi067", "target": ["mon-mgr", ""]}) v1 -- 0x7f2340106be0 con 0x7f232c0368d0 2024-04-12T05:28:37.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:37 smithi032 ceph-mon[39731]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:28:37.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:37 smithi032 ceph-mon[39731]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi067", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:28:37.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:37 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:37.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:37 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:37.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:37 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:37.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:37 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:38.503 INFO:teuthology.orchestra.run.smithi032.stdout:Added host 'smithi067' with addr '172.21.15.67' 2024-04-12T05:28:38.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:38.500+0000 7f2336ffd700 1 -- 172.21.15.32:0/1337032914 <== mgr.14160 v2:172.21.15.32:6800/3971961003 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f2340106be0 con 0x7f232c0368d0 2024-04-12T05:28:38.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:38.502+0000 7f23486d8700 1 -- 172.21.15.32:0/1337032914 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f232c0368d0 msgr2=0x7f232c038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:38.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:38.502+0000 7f23486d8700 1 --2- 172.21.15.32:0/1337032914 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f232c0368d0 0x7f232c038d80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2330009990 tx=0x7f2330006e40 comp rx=0 tx=0).stop 2024-04-12T05:28:38.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:38.502+0000 7f23486d8700 1 -- 172.21.15.32:0/1337032914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23400fca50 msgr2=0x7f2340071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:38.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:38.502+0000 7f23486d8700 1 --2- 172.21.15.32:0/1337032914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23400fca50 0x7f2340071d60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f233c000c00 tx=0x7f233c009e30 comp rx=0 tx=0).stop 2024-04-12T05:28:38.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:38.503+0000 7f23486d8700 1 -- 172.21.15.32:0/1337032914 shutdown_connections 2024-04-12T05:28:38.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:38.503+0000 7f23486d8700 1 --2- 172.21.15.32:0/1337032914 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f232c0368d0 0x7f232c038d80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:38.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:38.503+0000 7f23486d8700 1 --2- 172.21.15.32:0/1337032914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23400fca50 0x7f2340071d60 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:38.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:38.503+0000 7f23486d8700 1 -- 172.21.15.32:0/1337032914 >> 172.21.15.32:0/1337032914 conn(0x7f23400f8190 msgr2=0x7f23400f8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:38.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:38.503+0000 7f23486d8700 1 -- 172.21.15.32:0/1337032914 shutdown_connections 2024-04-12T05:28:38.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:38.503+0000 7f23486d8700 1 -- 172.21.15.32:0/1337032914 wait complete. 2024-04-12T05:28:38.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:38 smithi032 ceph-mon[39731]: Deploying daemon alertmanager.smithi032 on smithi032 2024-04-12T05:28:38.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:38 smithi032 ceph-mon[39731]: Deploying cephadm binary to smithi067 2024-04-12T05:28:38.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:38 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:39.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:39 smithi032 ceph-mon[39731]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:28:39.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:39 smithi032 ceph-mon[39731]: Added host smithi067 2024-04-12T05:28:39.749 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph orch host ls --format=json 2024-04-12T05:28:40.038 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:28:40.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:40 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:41.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:41 smithi032 ceph-mon[39731]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:28:43.763 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:43 smithi032 ceph-mon[39731]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:28:44.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.203+0000 7f607cda2700 1 -- 172.21.15.32:0/4156254309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60780fd050 msgr2=0x7f60780fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:44.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.203+0000 7f607cda2700 1 --2- 172.21.15.32:0/4156254309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60780fd050 0x7f60780fd460 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f60680048c0 tx=0x7f6068009e30 comp rx=0 tx=0).stop 2024-04-12T05:28:44.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.203+0000 7f607cda2700 1 -- 172.21.15.32:0/4156254309 shutdown_connections 2024-04-12T05:28:44.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.203+0000 7f607cda2700 1 --2- 172.21.15.32:0/4156254309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60780fd050 0x7f60780fd460 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:44.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.203+0000 7f607cda2700 1 -- 172.21.15.32:0/4156254309 >> 172.21.15.32:0/4156254309 conn(0x7f60780f70d0 msgr2=0x7f60780f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:44.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.203+0000 7f607cda2700 1 -- 172.21.15.32:0/4156254309 shutdown_connections 2024-04-12T05:28:44.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.203+0000 7f607cda2700 1 -- 172.21.15.32:0/4156254309 wait complete. 2024-04-12T05:28:44.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.204+0000 7f607cda2700 1 Processor -- start 2024-04-12T05:28:44.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.204+0000 7f607cda2700 1 -- start start 2024-04-12T05:28:44.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.204+0000 7f607cda2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60780fd050 0x7f607810f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:44.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.204+0000 7f607cda2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607810f6f0 con 0x7f60780fd050 2024-04-12T05:28:44.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.204+0000 7f607659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60780fd050 0x7f607810f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:44.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.204+0000 7f607659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60780fd050 0x7f607810f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50074/0 (socket says 172.21.15.32:50074) 2024-04-12T05:28:44.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.204+0000 7f607659c700 1 -- 172.21.15.32:0/497352088 learned_addr learned my addr 172.21.15.32:0/497352088 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:44.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.204+0000 7f607659c700 1 -- 172.21.15.32:0/497352088 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6068005040 con 0x7f60780fd050 2024-04-12T05:28:44.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.204+0000 7f607659c700 1 --2- 172.21.15.32:0/497352088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60780fd050 0x7f607810f1b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6068009300 tx=0x7f6068009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:44.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.205+0000 7f6066ffd700 1 -- 172.21.15.32:0/497352088 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6068009600 con 0x7f60780fd050 2024-04-12T05:28:44.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.205+0000 7f6066ffd700 1 -- 172.21.15.32:0/497352088 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6068009760 con 0x7f60780fd050 2024-04-12T05:28:44.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.205+0000 7f6066ffd700 1 -- 172.21.15.32:0/497352088 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f606801c5b0 con 0x7f60780fd050 2024-04-12T05:28:44.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.205+0000 7f607cda2700 1 -- 172.21.15.32:0/497352088 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f607810f8f0 con 0x7f60780fd050 2024-04-12T05:28:44.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.205+0000 7f607cda2700 1 -- 172.21.15.32:0/497352088 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6078110460 con 0x7f60780fd050 2024-04-12T05:28:44.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.206+0000 7f6066ffd700 1 -- 172.21.15.32:0/497352088 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f6068013420 con 0x7f60780fd050 2024-04-12T05:28:44.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.206+0000 7f607cda2700 1 -- 172.21.15.32:0/497352088 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6078101d30 con 0x7f60780fd050 2024-04-12T05:28:44.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.206+0000 7f6066ffd700 1 --2- 172.21.15.32:0/497352088 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f60600368d0 0x7f6060038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:44.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.206+0000 7f6066ffd700 1 -- 172.21.15.32:0/497352088 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6068049a10 con 0x7f60780fd050 2024-04-12T05:28:44.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.206+0000 7f6075d9b700 1 --2- 172.21.15.32:0/497352088 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f60600368d0 0x7f6060038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:44.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.206+0000 7f6075d9b700 1 --2- 172.21.15.32:0/497352088 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f60600368d0 0x7f6060038d80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f606c009990 tx=0x7f606c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:44.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.211+0000 7f6066ffd700 1 -- 172.21.15.32:0/497352088 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6068013960 con 0x7f60780fd050 2024-04-12T05:28:44.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.413+0000 7f607cda2700 1 -- 172.21.15.32:0/497352088 --> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f607805f690 con 0x7f60600368d0 2024-04-12T05:28:44.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.414+0000 7f6066ffd700 1 -- 172.21.15.32:0/497352088 <== mgr.14160 v2:172.21.15.32:6800/3971961003 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7f607805f690 con 0x7f60600368d0 2024-04-12T05:28:44.416 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:28:44.416 INFO:teuthology.orchestra.run.smithi032.stdout:[{"addr": "172.21.15.32", "hostname": "smithi032", "labels": [], "status": ""}, {"addr": "172.21.15.67", "hostname": "smithi067", "labels": [], "status": ""}] 2024-04-12T05:28:44.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.416+0000 7f607cda2700 1 -- 172.21.15.32:0/497352088 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f60600368d0 msgr2=0x7f6060038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:44.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.416+0000 7f607cda2700 1 --2- 172.21.15.32:0/497352088 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f60600368d0 0x7f6060038d80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f606c009990 tx=0x7f606c006e40 comp rx=0 tx=0).stop 2024-04-12T05:28:44.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.416+0000 7f607cda2700 1 -- 172.21.15.32:0/497352088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60780fd050 msgr2=0x7f607810f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:44.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.416+0000 7f607cda2700 1 --2- 172.21.15.32:0/497352088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60780fd050 0x7f607810f1b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6068009300 tx=0x7f6068009e30 comp rx=0 tx=0).stop 2024-04-12T05:28:44.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.416+0000 7f607cda2700 1 -- 172.21.15.32:0/497352088 shutdown_connections 2024-04-12T05:28:44.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.416+0000 7f607cda2700 1 --2- 172.21.15.32:0/497352088 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f60600368d0 0x7f6060038d80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:44.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.416+0000 7f607cda2700 1 --2- 172.21.15.32:0/497352088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60780fd050 0x7f607810f1b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:44.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.416+0000 7f607cda2700 1 -- 172.21.15.32:0/497352088 >> 172.21.15.32:0/497352088 conn(0x7f60780f70d0 msgr2=0x7f60780f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:44.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.416+0000 7f607cda2700 1 -- 172.21.15.32:0/497352088 shutdown_connections 2024-04-12T05:28:44.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:44.416+0000 7f607cda2700 1 -- 172.21.15.32:0/497352088 wait complete. 2024-04-12T05:28:45.462 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-12T05:28:45.463 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph osd crush tunables default 2024-04-12T05:28:45.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:45 smithi032 ceph-mon[39731]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T05:28:45.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:45 smithi032 ceph-mon[39731]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:28:46.286 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:28:47.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:47 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:47.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:47 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:47.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:47 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:47.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:47 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:47.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:47 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:47.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:47 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:47.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:47 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-12T05:28:47.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:47 smithi032 ceph-mon[39731]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-12T05:28:47.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:47 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:47.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:47 smithi032 ceph-mon[39731]: Deploying daemon grafana.smithi032 on smithi032 2024-04-12T05:28:47.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:47 smithi032 ceph-mon[39731]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:28:47.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:47.991+0000 7f34be77e700 1 -- 172.21.15.32:0/304694182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34b80fd050 msgr2=0x7f34b80fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:47.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:47.991+0000 7f34be77e700 1 --2- 172.21.15.32:0/304694182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34b80fd050 0x7f34b80fd460 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f34a80048c0 tx=0x7f34a8009e30 comp rx=0 tx=0).stop 2024-04-12T05:28:47.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:47.991+0000 7f34be77e700 1 -- 172.21.15.32:0/304694182 shutdown_connections 2024-04-12T05:28:47.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:47.991+0000 7f34be77e700 1 --2- 172.21.15.32:0/304694182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34b80fd050 0x7f34b80fd460 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:47.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:47.991+0000 7f34be77e700 1 -- 172.21.15.32:0/304694182 >> 172.21.15.32:0/304694182 conn(0x7f34b80f70d0 msgr2=0x7f34b80f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:47.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:47.991+0000 7f34be77e700 1 -- 172.21.15.32:0/304694182 shutdown_connections 2024-04-12T05:28:47.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:47.991+0000 7f34be77e700 1 -- 172.21.15.32:0/304694182 wait complete. 2024-04-12T05:28:47.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:47.992+0000 7f34be77e700 1 Processor -- start 2024-04-12T05:28:47.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:47.992+0000 7f34be77e700 1 -- start start 2024-04-12T05:28:47.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:47.992+0000 7f34be77e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34b80fd050 0x7f34b810bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:47.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:47.992+0000 7f34be77e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b810c460 con 0x7f34b80fd050 2024-04-12T05:28:47.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:47.992+0000 7f34b7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34b80fd050 0x7f34b810bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:47.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:47.992+0000 7f34b7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34b80fd050 0x7f34b810bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50080/0 (socket says 172.21.15.32:50080) 2024-04-12T05:28:47.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:47.992+0000 7f34b7fff700 1 -- 172.21.15.32:0/2110090777 learned_addr learned my addr 172.21.15.32:0/2110090777 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:28:47.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:47.993+0000 7f34b7fff700 1 -- 172.21.15.32:0/2110090777 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a8005040 con 0x7f34b80fd050 2024-04-12T05:28:48.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:47.993+0000 7f34b7fff700 1 --2- 172.21.15.32:0/2110090777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34b80fd050 0x7f34b810bf20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f34a8000c00 tx=0x7f34a8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:48.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:47.993+0000 7f34b4ff9700 1 -- 172.21.15.32:0/2110090777 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34a8009620 con 0x7f34b80fd050 2024-04-12T05:28:48.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:47.993+0000 7f34b4ff9700 1 -- 172.21.15.32:0/2110090777 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34a8009780 con 0x7f34b80fd050 2024-04-12T05:28:48.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:47.993+0000 7f34be77e700 1 -- 172.21.15.32:0/2110090777 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34b810c660 con 0x7f34b80fd050 2024-04-12T05:28:48.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:47.993+0000 7f34b4ff9700 1 -- 172.21.15.32:0/2110090777 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34a801c5b0 con 0x7f34b80fd050 2024-04-12T05:28:48.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:47.993+0000 7f34be77e700 1 -- 172.21.15.32:0/2110090777 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34b810d0f0 con 0x7f34b80fd050 2024-04-12T05:28:48.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:47.994+0000 7f34b4ff9700 1 -- 172.21.15.32:0/2110090777 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f34a801c7a0 con 0x7f34b80fd050 2024-04-12T05:28:48.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:47.994+0000 7f34b4ff9700 1 --2- 172.21.15.32:0/2110090777 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f34a00368d0 0x7f34a0038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:48.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:47.994+0000 7f34be77e700 1 -- 172.21.15.32:0/2110090777 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34b8101d30 con 0x7f34b80fd050 2024-04-12T05:28:48.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:47.994+0000 7f34b4ff9700 1 -- 172.21.15.32:0/2110090777 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f34a8049bb0 con 0x7f34b80fd050 2024-04-12T05:28:48.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:47.994+0000 7f34b77fe700 1 --2- 172.21.15.32:0/2110090777 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f34a00368d0 0x7f34a0038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:48.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:47.995+0000 7f34b77fe700 1 --2- 172.21.15.32:0/2110090777 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f34a00368d0 0x7f34a0038d80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f34ac009990 tx=0x7f34ac006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:48.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:47.999+0000 7f34b4ff9700 1 -- 172.21.15.32:0/2110090777 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f34a8011db0 con 0x7f34b80fd050 2024-04-12T05:28:48.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:48.215+0000 7f34be77e700 1 -- 172.21.15.32:0/2110090777 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f34b805f5f0 con 0x7f34b80fd050 2024-04-12T05:28:48.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:48.404+0000 7f34b4ff9700 1 -- 172.21.15.32:0/2110090777 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f34a8023990 con 0x7f34b80fd050 2024-04-12T05:28:48.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:48.406+0000 7f34be77e700 1 -- 172.21.15.32:0/2110090777 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f34a00368d0 msgr2=0x7f34a0038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:48.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:48.406+0000 7f34be77e700 1 --2- 172.21.15.32:0/2110090777 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f34a00368d0 0x7f34a0038d80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f34ac009990 tx=0x7f34ac006e40 comp rx=0 tx=0).stop 2024-04-12T05:28:48.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:48.406+0000 7f34be77e700 1 -- 172.21.15.32:0/2110090777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34b80fd050 msgr2=0x7f34b810bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:48.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:48.406+0000 7f34be77e700 1 --2- 172.21.15.32:0/2110090777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34b80fd050 0x7f34b810bf20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f34a8000c00 tx=0x7f34a8009e30 comp rx=0 tx=0).stop 2024-04-12T05:28:48.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:48.406+0000 7f34be77e700 1 -- 172.21.15.32:0/2110090777 shutdown_connections 2024-04-12T05:28:48.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:48.406+0000 7f34be77e700 1 --2- 172.21.15.32:0/2110090777 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f34a00368d0 0x7f34a0038d80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:48.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:48.406+0000 7f34be77e700 1 --2- 172.21.15.32:0/2110090777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34b80fd050 0x7f34b810bf20 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:48.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:48.406+0000 7f34be77e700 1 -- 172.21.15.32:0/2110090777 >> 172.21.15.32:0/2110090777 conn(0x7f34b80f70d0 msgr2=0x7f34b80f7990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:48.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:48.406+0000 7f34be77e700 1 -- 172.21.15.32:0/2110090777 shutdown_connections 2024-04-12T05:28:48.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:28:48.406+0000 7f34be77e700 1 -- 172.21.15.32:0/2110090777 wait complete. 2024-04-12T05:28:48.409 INFO:teuthology.orchestra.run.smithi032.stderr:adjusted tunables profile to default 2024-04-12T05:28:48.438 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:48 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/2110090777' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-12T05:28:49.218 INFO:tasks.cephadm:Adding mon.smithi032 on smithi032 2024-04-12T05:28:49.218 INFO:tasks.cephadm:Adding mon.smithi067 on smithi067 2024-04-12T05:28:49.218 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph orch apply mon '2;smithi032:172.21.15.32=smithi032;smithi067:172.21.15.67=smithi067' 2024-04-12T05:28:49.513 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-12T05:28:49.583 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-12T05:28:49.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:49 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/2110090777' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-12T05:28:49.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:49 smithi032 ceph-mon[39731]: osdmap e4: 0 total, 0 up, 0 in 2024-04-12T05:28:49.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:49 smithi032 ceph-mon[39731]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:28:50.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:50 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:51.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.160+0000 7f4a7ed83700 1 -- 172.21.15.67:0/1712775005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a780fd050 msgr2=0x7f4a780fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:51.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.160+0000 7f4a7ed83700 1 --2- 172.21.15.67:0/1712775005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a780fd050 0x7f4a780fd460 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c0048c0 tx=0x7f4a6c009e30 comp rx=0 tx=0).stop 2024-04-12T05:28:51.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.160+0000 7f4a7ed83700 1 -- 172.21.15.67:0/1712775005 shutdown_connections 2024-04-12T05:28:51.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.160+0000 7f4a7ed83700 1 --2- 172.21.15.67:0/1712775005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a780fd050 0x7f4a780fd460 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:51.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.160+0000 7f4a7ed83700 1 -- 172.21.15.67:0/1712775005 >> 172.21.15.67:0/1712775005 conn(0x7f4a780f70d0 msgr2=0x7f4a780f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:51.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.160+0000 7f4a7ed83700 1 -- 172.21.15.67:0/1712775005 shutdown_connections 2024-04-12T05:28:51.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.160+0000 7f4a7ed83700 1 -- 172.21.15.67:0/1712775005 wait complete. 2024-04-12T05:28:51.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.161+0000 7f4a7ed83700 1 Processor -- start 2024-04-12T05:28:51.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.161+0000 7f4a7ed83700 1 -- start start 2024-04-12T05:28:51.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.161+0000 7f4a7ed83700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a780fd050 0x7f4a7810d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:51.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.161+0000 7f4a7ed83700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7810d540 con 0x7f4a780fd050 2024-04-12T05:28:51.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.161+0000 7f4a7cb1f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a780fd050 0x7f4a7810d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:51.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.161+0000 7f4a7cb1f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a780fd050 0x7f4a7810d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.67:58532/0 (socket says 172.21.15.67:58532) 2024-04-12T05:28:51.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.161+0000 7f4a7cb1f700 1 -- 172.21.15.67:0/405801617 learned_addr learned my addr 172.21.15.67:0/405801617 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-12T05:28:51.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.162+0000 7f4a7cb1f700 1 -- 172.21.15.67:0/405801617 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a6c005040 con 0x7f4a780fd050 2024-04-12T05:28:51.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.162+0000 7f4a7cb1f700 1 --2- 172.21.15.67:0/405801617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a780fd050 0x7f4a7810d000 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c009300 tx=0x7f4a6c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:51.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.162+0000 7f4a757fa700 1 -- 172.21.15.67:0/405801617 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a6c009600 con 0x7f4a780fd050 2024-04-12T05:28:51.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.162+0000 7f4a7ed83700 1 -- 172.21.15.67:0/405801617 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a7810d740 con 0x7f4a780fd050 2024-04-12T05:28:51.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.162+0000 7f4a757fa700 1 -- 172.21.15.67:0/405801617 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4a6c009760 con 0x7f4a780fd050 2024-04-12T05:28:51.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.162+0000 7f4a757fa700 1 -- 172.21.15.67:0/405801617 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a6c01c5b0 con 0x7f4a780fd050 2024-04-12T05:28:51.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.162+0000 7f4a7ed83700 1 -- 172.21.15.67:0/405801617 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a7810e260 con 0x7f4a780fd050 2024-04-12T05:28:51.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.163+0000 7f4a757fa700 1 -- 172.21.15.67:0/405801617 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f4a6c013420 con 0x7f4a780fd050 2024-04-12T05:28:51.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.163+0000 7f4a7ed83700 1 -- 172.21.15.67:0/405801617 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a78101d30 con 0x7f4a780fd050 2024-04-12T05:28:51.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.163+0000 7f4a757fa700 1 --2- 172.21.15.67:0/405801617 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f4a68036880 0x7f4a68038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:51.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.163+0000 7f4a757fa700 1 -- 172.21.15.67:0/405801617 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4a6c049ae0 con 0x7f4a780fd050 2024-04-12T05:28:51.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.163+0000 7f4a77fff700 1 --2- 172.21.15.67:0/405801617 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f4a68036880 0x7f4a68038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:51.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.164+0000 7f4a77fff700 1 --2- 172.21.15.67:0/405801617 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f4a68036880 0x7f4a68038d30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4a64009990 tx=0x7f4a64006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:51.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.172+0000 7f4a757fa700 1 -- 172.21.15.67:0/405801617 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4a6c013960 con 0x7f4a780fd050 2024-04-12T05:28:51.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.375+0000 7f4a7ed83700 1 -- 172.21.15.67:0/405801617 --> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi032:172.21.15.32=smithi032;smithi067:172.21.15.67=smithi067", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a78065780 con 0x7f4a68036880 2024-04-12T05:28:51.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.380+0000 7f4a757fa700 1 -- 172.21.15.67:0/405801617 <== mgr.14160 v2:172.21.15.32:6800/3971961003 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f4a78065780 con 0x7f4a68036880 2024-04-12T05:28:51.382 INFO:teuthology.orchestra.run.smithi067.stdout:Scheduled mon update... 2024-04-12T05:28:51.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.382+0000 7f4a7ed83700 1 -- 172.21.15.67:0/405801617 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f4a68036880 msgr2=0x7f4a68038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:51.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.382+0000 7f4a7ed83700 1 --2- 172.21.15.67:0/405801617 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f4a68036880 0x7f4a68038d30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4a64009990 tx=0x7f4a64006e40 comp rx=0 tx=0).stop 2024-04-12T05:28:51.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.382+0000 7f4a7ed83700 1 -- 172.21.15.67:0/405801617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a780fd050 msgr2=0x7f4a7810d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:51.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.382+0000 7f4a7ed83700 1 --2- 172.21.15.67:0/405801617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a780fd050 0x7f4a7810d000 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c009300 tx=0x7f4a6c009e30 comp rx=0 tx=0).stop 2024-04-12T05:28:51.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.383+0000 7f4a7ed83700 1 -- 172.21.15.67:0/405801617 shutdown_connections 2024-04-12T05:28:51.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.383+0000 7f4a7ed83700 1 --2- 172.21.15.67:0/405801617 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f4a68036880 0x7f4a68038d30 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:51.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.383+0000 7f4a7ed83700 1 --2- 172.21.15.67:0/405801617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a780fd050 0x7f4a7810d000 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:51.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.383+0000 7f4a7ed83700 1 -- 172.21.15.67:0/405801617 >> 172.21.15.67:0/405801617 conn(0x7f4a780f70d0 msgr2=0x7f4a78065070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:51.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.383+0000 7f4a7ed83700 1 -- 172.21.15.67:0/405801617 shutdown_connections 2024-04-12T05:28:51.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:51.383+0000 7f4a7ed83700 1 -- 172.21.15.67:0/405801617 wait complete. 2024-04-12T05:28:51.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:51 smithi032 ceph-mon[39731]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:28:51.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:51 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:28:51.971 DEBUG:teuthology.orchestra.run.smithi067:mon.smithi067> sudo journalctl -f -n 0 -u ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@mon.smithi067.service 2024-04-12T05:28:51.973 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-12T05:28:51.973 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph mon dump -f json 2024-04-12T05:28:52.004 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:-- Logs begin at Fri 2024-04-12 05:14:32 UTC. -- 2024-04-12T05:28:52.289 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-12T05:28:52.351 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-12T05:28:52.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:52 smithi032 ceph-mon[39731]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi032:172.21.15.32=smithi032;smithi067:172.21.15.67=smithi067", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:28:52.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:52 smithi032 ceph-mon[39731]: Saving service mon spec with placement smithi032:172.21.15.32=smithi032;smithi067:172.21.15.67=smithi067;count:2 2024-04-12T05:28:53.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:53 smithi032 ceph-mon[39731]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:28:53.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:53.912+0000 7efd8370d700 1 -- 172.21.15.67:0/1526225119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd7c0fd050 msgr2=0x7efd7c0fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:53.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:53.912+0000 7efd8370d700 1 --2- 172.21.15.67:0/1526225119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd7c0fd050 0x7efd7c0fd460 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7efd780048c0 tx=0x7efd78009e30 comp rx=0 tx=0).stop 2024-04-12T05:28:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:53.912+0000 7efd8370d700 1 -- 172.21.15.67:0/1526225119 shutdown_connections 2024-04-12T05:28:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:53.912+0000 7efd8370d700 1 --2- 172.21.15.67:0/1526225119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd7c0fd050 0x7efd7c0fd460 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:53.912+0000 7efd8370d700 1 -- 172.21.15.67:0/1526225119 >> 172.21.15.67:0/1526225119 conn(0x7efd7c0f70d0 msgr2=0x7efd7c0f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:53.913+0000 7efd8370d700 1 -- 172.21.15.67:0/1526225119 shutdown_connections 2024-04-12T05:28:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:53.913+0000 7efd8370d700 1 -- 172.21.15.67:0/1526225119 wait complete. 2024-04-12T05:28:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:53.913+0000 7efd8370d700 1 Processor -- start 2024-04-12T05:28:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:53.913+0000 7efd8370d700 1 -- start start 2024-04-12T05:28:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:53.913+0000 7efd8370d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd7c0fd050 0x7efd7c10bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:53.913+0000 7efd8370d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c10c460 con 0x7efd7c0fd050 2024-04-12T05:28:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:53.913+0000 7efd814a9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd7c0fd050 0x7efd7c10bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:53.913+0000 7efd814a9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd7c0fd050 0x7efd7c10bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.67:57874/0 (socket says 172.21.15.67:57874) 2024-04-12T05:28:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:53.913+0000 7efd814a9700 1 -- 172.21.15.67:0/2637313301 learned_addr learned my addr 172.21.15.67:0/2637313301 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-12T05:28:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:53.914+0000 7efd814a9700 1 -- 172.21.15.67:0/2637313301 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd78005040 con 0x7efd7c0fd050 2024-04-12T05:28:53.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:53.914+0000 7efd814a9700 1 --2- 172.21.15.67:0/2637313301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd7c0fd050 0x7efd7c10bf20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7efd78009300 tx=0x7efd78009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:53.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:53.914+0000 7efd71ffb700 1 -- 172.21.15.67:0/2637313301 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd78009600 con 0x7efd7c0fd050 2024-04-12T05:28:53.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:53.914+0000 7efd71ffb700 1 -- 172.21.15.67:0/2637313301 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efd78009760 con 0x7efd7c0fd050 2024-04-12T05:28:53.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:53.914+0000 7efd71ffb700 1 -- 172.21.15.67:0/2637313301 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd7801c5b0 con 0x7efd7c0fd050 2024-04-12T05:28:53.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:53.914+0000 7efd8370d700 1 -- 172.21.15.67:0/2637313301 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd7c10c660 con 0x7efd7c0fd050 2024-04-12T05:28:53.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:53.914+0000 7efd8370d700 1 -- 172.21.15.67:0/2637313301 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd7c10d0f0 con 0x7efd7c0fd050 2024-04-12T05:28:53.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:53.915+0000 7efd71ffb700 1 -- 172.21.15.67:0/2637313301 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7efd78013960 con 0x7efd7c0fd050 2024-04-12T05:28:53.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:53.915+0000 7efd8370d700 1 -- 172.21.15.67:0/2637313301 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd7c101d30 con 0x7efd7c0fd050 2024-04-12T05:28:53.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:53.915+0000 7efd71ffb700 1 --2- 172.21.15.67:0/2637313301 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7efd680368d0 0x7efd68038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:53.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:53.915+0000 7efd71ffb700 1 -- 172.21.15.67:0/2637313301 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7efd78049b00 con 0x7efd7c0fd050 2024-04-12T05:28:53.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:53.915+0000 7efd80ca8700 1 --2- 172.21.15.67:0/2637313301 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7efd680368d0 0x7efd68038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:53.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:53.916+0000 7efd80ca8700 1 --2- 172.21.15.67:0/2637313301 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7efd680368d0 0x7efd68038d80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7efd6c009990 tx=0x7efd6c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:53.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:53.921+0000 7efd71ffb700 1 -- 172.21.15.67:0/2637313301 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7efd780098d0 con 0x7efd7c0fd050 2024-04-12T05:28:54.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:54.196+0000 7efd8370d700 1 -- 172.21.15.67:0/2637313301 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efd7c05f5f0 con 0x7efd7c0fd050 2024-04-12T05:28:54.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:54.196+0000 7efd71ffb700 1 -- 172.21.15.67:0/2637313301 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7efd78011370 con 0x7efd7c0fd050 2024-04-12T05:28:54.198 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:28:54.199 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":1,"fsid":"37b58ce2-f88d-11ee-bc8c-c7b262605968","modified":"2024-04-12T05:27:04.825332Z","created":"2024-04-12T05:27:04.825332Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-12T05:28:54.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:54.198+0000 7efd8370d700 1 -- 172.21.15.67:0/2637313301 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7efd680368d0 msgr2=0x7efd68038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:54.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:54.198+0000 7efd8370d700 1 --2- 172.21.15.67:0/2637313301 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7efd680368d0 0x7efd68038d80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7efd6c009990 tx=0x7efd6c006e40 comp rx=0 tx=0).stop 2024-04-12T05:28:54.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:54.198+0000 7efd8370d700 1 -- 172.21.15.67:0/2637313301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd7c0fd050 msgr2=0x7efd7c10bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:54.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:54.198+0000 7efd8370d700 1 --2- 172.21.15.67:0/2637313301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd7c0fd050 0x7efd7c10bf20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7efd78009300 tx=0x7efd78009e30 comp rx=0 tx=0).stop 2024-04-12T05:28:54.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:54.198+0000 7efd8370d700 1 -- 172.21.15.67:0/2637313301 shutdown_connections 2024-04-12T05:28:54.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:54.198+0000 7efd8370d700 1 --2- 172.21.15.67:0/2637313301 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7efd680368d0 0x7efd68038d80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:54.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:54.198+0000 7efd8370d700 1 --2- 172.21.15.67:0/2637313301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd7c0fd050 0x7efd7c10bf20 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:54.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:54.198+0000 7efd8370d700 1 -- 172.21.15.67:0/2637313301 >> 172.21.15.67:0/2637313301 conn(0x7efd7c0f70d0 msgr2=0x7efd7c0f7990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:54.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:54.198+0000 7efd8370d700 1 -- 172.21.15.67:0/2637313301 shutdown_connections 2024-04-12T05:28:54.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:54.198+0000 7efd8370d700 1 -- 172.21.15.67:0/2637313301 wait complete. 2024-04-12T05:28:54.201 INFO:teuthology.orchestra.run.smithi067.stderr:dumped monmap epoch 1 2024-04-12T05:28:54.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:54 smithi032 ceph-mon[39731]: from='client.? 172.21.15.67:0/2637313301' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:28:55.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:55 smithi032 ceph-mon[39731]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:28:55.891 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-12T05:28:55.891 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph mon dump -f json 2024-04-12T05:28:56.185 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-12T05:28:56.246 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-12T05:28:57.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:57 smithi032 ceph-mon[39731]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:28:57.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:57.864+0000 7f9ebcb3f700 1 -- 172.21.15.67:0/617842733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eb80fca60 msgr2=0x7f9eb80fce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:57.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:57.864+0000 7f9ebcb3f700 1 --2- 172.21.15.67:0/617842733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eb80fca60 0x7f9eb80fce70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9ea80048c0 tx=0x7f9ea8009e30 comp rx=0 tx=0).stop 2024-04-12T05:28:57.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:57.864+0000 7f9ebcb3f700 1 -- 172.21.15.67:0/617842733 shutdown_connections 2024-04-12T05:28:57.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:57.864+0000 7f9ebcb3f700 1 --2- 172.21.15.67:0/617842733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eb80fca60 0x7f9eb80fce70 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:57.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:57.864+0000 7f9ebcb3f700 1 -- 172.21.15.67:0/617842733 >> 172.21.15.67:0/617842733 conn(0x7f9eb80f81c0 msgr2=0x7f9eb80fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:57.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:57.864+0000 7f9ebcb3f700 1 -- 172.21.15.67:0/617842733 shutdown_connections 2024-04-12T05:28:57.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:57.864+0000 7f9ebcb3f700 1 -- 172.21.15.67:0/617842733 wait complete. 2024-04-12T05:28:57.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:57.865+0000 7f9ebcb3f700 1 Processor -- start 2024-04-12T05:28:57.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:57.865+0000 7f9ebcb3f700 1 -- start start 2024-04-12T05:28:57.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:57.865+0000 7f9ebcb3f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eb80fca60 0x7f9eb810cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:57.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:57.865+0000 7f9ebcb3f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb810d530 con 0x7f9eb80fca60 2024-04-12T05:28:57.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:57.865+0000 7f9eb659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eb80fca60 0x7f9eb810cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:57.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:57.865+0000 7f9eb659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eb80fca60 0x7f9eb810cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.67:57896/0 (socket says 172.21.15.67:57896) 2024-04-12T05:28:57.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:57.865+0000 7f9eb659c700 1 -- 172.21.15.67:0/1583561843 learned_addr learned my addr 172.21.15.67:0/1583561843 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-12T05:28:57.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:57.865+0000 7f9eb659c700 1 -- 172.21.15.67:0/1583561843 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea8005040 con 0x7f9eb80fca60 2024-04-12T05:28:57.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:57.866+0000 7f9eb659c700 1 --2- 172.21.15.67:0/1583561843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eb80fca60 0x7f9eb810cff0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9ea8009300 tx=0x7f9ea8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:57.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:57.866+0000 7f9ea6ffd700 1 -- 172.21.15.67:0/1583561843 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ea8009600 con 0x7f9eb80fca60 2024-04-12T05:28:57.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:57.866+0000 7f9ebcb3f700 1 -- 172.21.15.67:0/1583561843 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eb810d730 con 0x7f9eb80fca60 2024-04-12T05:28:57.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:57.866+0000 7f9ea6ffd700 1 -- 172.21.15.67:0/1583561843 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ea8009760 con 0x7f9eb80fca60 2024-04-12T05:28:57.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:57.866+0000 7f9ea6ffd700 1 -- 172.21.15.67:0/1583561843 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ea801c5b0 con 0x7f9eb80fca60 2024-04-12T05:28:57.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:57.866+0000 7f9ebcb3f700 1 -- 172.21.15.67:0/1583561843 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eb810e2a0 con 0x7f9eb80fca60 2024-04-12T05:28:57.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:57.867+0000 7f9ebcb3f700 1 -- 172.21.15.67:0/1583561843 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eb8102e30 con 0x7f9eb80fca60 2024-04-12T05:28:57.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:57.867+0000 7f9ea6ffd700 1 -- 172.21.15.67:0/1583561843 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f9ea8013960 con 0x7f9eb80fca60 2024-04-12T05:28:57.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:57.867+0000 7f9ea6ffd700 1 --2- 172.21.15.67:0/1583561843 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f9ea0036880 0x7f9ea0038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:28:57.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:57.867+0000 7f9ea6ffd700 1 -- 172.21.15.67:0/1583561843 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f9ea8049a80 con 0x7f9eb80fca60 2024-04-12T05:28:57.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:57.867+0000 7f9eb5d9b700 1 --2- 172.21.15.67:0/1583561843 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f9ea0036880 0x7f9ea0038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:57.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:57.868+0000 7f9eb5d9b700 1 --2- 172.21.15.67:0/1583561843 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f9ea0036880 0x7f9ea0038d30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9eac009990 tx=0x7f9eac006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:57.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:57.873+0000 7f9ea6ffd700 1 -- 172.21.15.67:0/1583561843 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9ea80098d0 con 0x7f9eb80fca60 2024-04-12T05:28:58.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:58.146+0000 7f9ebcb3f700 1 -- 172.21.15.67:0/1583561843 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9eb810ecb0 con 0x7f9eb80fca60 2024-04-12T05:28:58.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:58.147+0000 7f9ea6ffd700 1 -- 172.21.15.67:0/1583561843 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f9ea8011370 con 0x7f9eb80fca60 2024-04-12T05:28:58.149 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:28:58.149 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":1,"fsid":"37b58ce2-f88d-11ee-bc8c-c7b262605968","modified":"2024-04-12T05:27:04.825332Z","created":"2024-04-12T05:27:04.825332Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-12T05:28:58.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:58.149+0000 7f9ebcb3f700 1 -- 172.21.15.67:0/1583561843 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f9ea0036880 msgr2=0x7f9ea0038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:58.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:58.149+0000 7f9ebcb3f700 1 --2- 172.21.15.67:0/1583561843 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f9ea0036880 0x7f9ea0038d30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9eac009990 tx=0x7f9eac006e40 comp rx=0 tx=0).stop 2024-04-12T05:28:58.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:58.149+0000 7f9ebcb3f700 1 -- 172.21.15.67:0/1583561843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eb80fca60 msgr2=0x7f9eb810cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:58.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:58.149+0000 7f9ebcb3f700 1 --2- 172.21.15.67:0/1583561843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eb80fca60 0x7f9eb810cff0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9ea8009300 tx=0x7f9ea8009e30 comp rx=0 tx=0).stop 2024-04-12T05:28:58.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:58.149+0000 7f9ebcb3f700 1 -- 172.21.15.67:0/1583561843 shutdown_connections 2024-04-12T05:28:58.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:58.149+0000 7f9ebcb3f700 1 --2- 172.21.15.67:0/1583561843 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f9ea0036880 0x7f9ea0038d30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:58.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:58.149+0000 7f9ebcb3f700 1 --2- 172.21.15.67:0/1583561843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eb80fca60 0x7f9eb810cff0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:28:58.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:58.149+0000 7f9ebcb3f700 1 -- 172.21.15.67:0/1583561843 >> 172.21.15.67:0/1583561843 conn(0x7f9eb80f81c0 msgr2=0x7f9eb80f8a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:58.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:58.149+0000 7f9ebcb3f700 1 -- 172.21.15.67:0/1583561843 shutdown_connections 2024-04-12T05:28:58.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:28:58.149+0000 7f9ebcb3f700 1 -- 172.21.15.67:0/1583561843 wait complete. 2024-04-12T05:28:58.151 INFO:teuthology.orchestra.run.smithi067.stderr:dumped monmap epoch 1 2024-04-12T05:28:58.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:58 smithi032 ceph-mon[39731]: from='client.? 172.21.15.67:0/1583561843' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:28:59.695 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-12T05:28:59.696 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph mon dump -f json 2024-04-12T05:28:59.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:28:59 smithi032 ceph-mon[39731]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:00.007 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-12T05:29:00.074 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-12T05:29:01.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:01 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:01.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:01 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:01.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:01 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:01.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:01 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:01.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:01 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:01.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:01 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:01.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:01 smithi032 ceph-mon[39731]: Deploying daemon prometheus.smithi032 on smithi032 2024-04-12T05:29:01.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:01 smithi032 ceph-mon[39731]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:01.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.592+0000 7f2fddf32700 1 -- 172.21.15.67:0/246002703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fd80fd050 msgr2=0x7f2fd80fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:01.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.592+0000 7f2fddf32700 1 --2- 172.21.15.67:0/246002703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fd80fd050 0x7f2fd80fd460 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2fc80048c0 tx=0x7f2fc8009e30 comp rx=0 tx=0).stop 2024-04-12T05:29:01.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.592+0000 7f2fddf32700 1 -- 172.21.15.67:0/246002703 shutdown_connections 2024-04-12T05:29:01.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.592+0000 7f2fddf32700 1 --2- 172.21.15.67:0/246002703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fd80fd050 0x7f2fd80fd460 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:01.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.592+0000 7f2fddf32700 1 -- 172.21.15.67:0/246002703 >> 172.21.15.67:0/246002703 conn(0x7f2fd80f70d0 msgr2=0x7f2fd80f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:01.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.592+0000 7f2fddf32700 1 -- 172.21.15.67:0/246002703 shutdown_connections 2024-04-12T05:29:01.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.592+0000 7f2fddf32700 1 -- 172.21.15.67:0/246002703 wait complete. 2024-04-12T05:29:01.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.592+0000 7f2fddf32700 1 Processor -- start 2024-04-12T05:29:01.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.593+0000 7f2fddf32700 1 -- start start 2024-04-12T05:29:01.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.593+0000 7f2fddf32700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fd80fd050 0x7f2fd80f9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:29:01.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.593+0000 7f2fddf32700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd80f95a0 con 0x7f2fd80fd050 2024-04-12T05:29:01.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.593+0000 7f2fd77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fd80fd050 0x7f2fd80f9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:01.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.593+0000 7f2fd77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fd80fd050 0x7f2fd80f9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.67:43246/0 (socket says 172.21.15.67:43246) 2024-04-12T05:29:01.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.593+0000 7f2fd77fe700 1 -- 172.21.15.67:0/2469506172 learned_addr learned my addr 172.21.15.67:0/2469506172 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-12T05:29:01.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.593+0000 7f2fd77fe700 1 -- 172.21.15.67:0/2469506172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc8005040 con 0x7f2fd80fd050 2024-04-12T05:29:01.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.593+0000 7f2fd77fe700 1 --2- 172.21.15.67:0/2469506172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fd80fd050 0x7f2fd80f9060 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8000c00 tx=0x7f2fc8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:01.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.594+0000 7f2fbffff700 1 -- 172.21.15.67:0/2469506172 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fc8009620 con 0x7f2fd80fd050 2024-04-12T05:29:01.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.594+0000 7f2fbffff700 1 -- 172.21.15.67:0/2469506172 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2fc8009780 con 0x7f2fd80fd050 2024-04-12T05:29:01.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.594+0000 7f2fddf32700 1 -- 172.21.15.67:0/2469506172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd80f7670 con 0x7f2fd80fd050 2024-04-12T05:29:01.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.594+0000 7f2fbffff700 1 -- 172.21.15.67:0/2469506172 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fc801c5b0 con 0x7f2fd80fd050 2024-04-12T05:29:01.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.594+0000 7f2fddf32700 1 -- 172.21.15.67:0/2469506172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd80f7b60 con 0x7f2fd80fd050 2024-04-12T05:29:01.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.594+0000 7f2fbffff700 1 -- 172.21.15.67:0/2469506172 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f2fc801c7f0 con 0x7f2fd80fd050 2024-04-12T05:29:01.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.595+0000 7f2fbffff700 1 --2- 172.21.15.67:0/2469506172 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f2fc0036880 0x7f2fc0038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:29:01.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.595+0000 7f2fbffff700 1 -- 172.21.15.67:0/2469506172 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2fc8049b50 con 0x7f2fd80fd050 2024-04-12T05:29:01.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.595+0000 7f2fddf32700 1 -- 172.21.15.67:0/2469506172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fd805f5f0 con 0x7f2fd80fd050 2024-04-12T05:29:01.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.595+0000 7f2fd6ffd700 1 --2- 172.21.15.67:0/2469506172 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f2fc0036880 0x7f2fc0038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:01.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.595+0000 7f2fd6ffd700 1 --2- 172.21.15.67:0/2469506172 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f2fc0036880 0x7f2fc0038d30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc009990 tx=0x7f2fcc006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:01.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.600+0000 7f2fbffff700 1 -- 172.21.15.67:0/2469506172 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f2fc8011db0 con 0x7f2fd80fd050 2024-04-12T05:29:01.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.875+0000 7f2fddf32700 1 -- 172.21.15.67:0/2469506172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2fd8101670 con 0x7f2fd80fd050 2024-04-12T05:29:01.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.876+0000 7f2fbffff700 1 -- 172.21.15.67:0/2469506172 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f2fc8023990 con 0x7f2fd80fd050 2024-04-12T05:29:01.878 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:29:01.879 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":1,"fsid":"37b58ce2-f88d-11ee-bc8c-c7b262605968","modified":"2024-04-12T05:27:04.825332Z","created":"2024-04-12T05:27:04.825332Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-12T05:29:01.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.878+0000 7f2fddf32700 1 -- 172.21.15.67:0/2469506172 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f2fc0036880 msgr2=0x7f2fc0038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:01.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.878+0000 7f2fddf32700 1 --2- 172.21.15.67:0/2469506172 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f2fc0036880 0x7f2fc0038d30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc009990 tx=0x7f2fcc006e40 comp rx=0 tx=0).stop 2024-04-12T05:29:01.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.878+0000 7f2fddf32700 1 -- 172.21.15.67:0/2469506172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fd80fd050 msgr2=0x7f2fd80f9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:01.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.878+0000 7f2fddf32700 1 --2- 172.21.15.67:0/2469506172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fd80fd050 0x7f2fd80f9060 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8000c00 tx=0x7f2fc8009e30 comp rx=0 tx=0).stop 2024-04-12T05:29:01.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.878+0000 7f2fddf32700 1 -- 172.21.15.67:0/2469506172 shutdown_connections 2024-04-12T05:29:01.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.878+0000 7f2fddf32700 1 --2- 172.21.15.67:0/2469506172 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f2fc0036880 0x7f2fc0038d30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:01.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.878+0000 7f2fddf32700 1 --2- 172.21.15.67:0/2469506172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fd80fd050 0x7f2fd80f9060 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:01.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.878+0000 7f2fddf32700 1 -- 172.21.15.67:0/2469506172 >> 172.21.15.67:0/2469506172 conn(0x7f2fd80f70d0 msgr2=0x7f2fd8065070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:01.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.878+0000 7f2fddf32700 1 -- 172.21.15.67:0/2469506172 shutdown_connections 2024-04-12T05:29:01.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:01.878+0000 7f2fddf32700 1 -- 172.21.15.67:0/2469506172 wait complete. 2024-04-12T05:29:01.881 INFO:teuthology.orchestra.run.smithi067.stderr:dumped monmap epoch 1 2024-04-12T05:29:02.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:02 smithi032 ceph-mon[39731]: from='client.? 172.21.15.67:0/2469506172' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:29:03.538 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-12T05:29:03.538 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph mon dump -f json 2024-04-12T05:29:03.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:03 smithi032 ceph-mon[39731]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:03.840 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-12T05:29:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-12T05:29:05.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.434+0000 7fe068638700 1 -- 172.21.15.67:0/2251680972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0600fe0b0 msgr2=0x7fe0600fe4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:05.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.434+0000 7fe068638700 1 --2- 172.21.15.67:0/2251680972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0600fe0b0 0x7fe0600fe4c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe0540048c0 tx=0x7fe054009e30 comp rx=0 tx=0).stop 2024-04-12T05:29:05.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.434+0000 7fe068638700 1 -- 172.21.15.67:0/2251680972 shutdown_connections 2024-04-12T05:29:05.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.434+0000 7fe068638700 1 --2- 172.21.15.67:0/2251680972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0600fe0b0 0x7fe0600fe4c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:05.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.434+0000 7fe068638700 1 -- 172.21.15.67:0/2251680972 >> 172.21.15.67:0/2251680972 conn(0x7fe0600f6850 msgr2=0x7fe0600f8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:05.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.434+0000 7fe068638700 1 -- 172.21.15.67:0/2251680972 shutdown_connections 2024-04-12T05:29:05.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.434+0000 7fe068638700 1 -- 172.21.15.67:0/2251680972 wait complete. 2024-04-12T05:29:05.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.435+0000 7fe068638700 1 Processor -- start 2024-04-12T05:29:05.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.435+0000 7fe068638700 1 -- start start 2024-04-12T05:29:05.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.435+0000 7fe068638700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0600fe0b0 0x7fe06010cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:29:05.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.435+0000 7fe068638700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06010d510 con 0x7fe0600fe0b0 2024-04-12T05:29:05.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.435+0000 7fe0663d4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0600fe0b0 0x7fe06010cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:05.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.436+0000 7fe0663d4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0600fe0b0 0x7fe06010cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.67:43264/0 (socket says 172.21.15.67:43264) 2024-04-12T05:29:05.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.436+0000 7fe0663d4700 1 -- 172.21.15.67:0/1610636540 learned_addr learned my addr 172.21.15.67:0/1610636540 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-12T05:29:05.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.436+0000 7fe0663d4700 1 -- 172.21.15.67:0/1610636540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe054005040 con 0x7fe0600fe0b0 2024-04-12T05:29:05.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.436+0000 7fe0663d4700 1 --2- 172.21.15.67:0/1610636540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0600fe0b0 0x7fe06010cfd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe054000c00 tx=0x7fe054009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:05.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.436+0000 7fe052ffd700 1 -- 172.21.15.67:0/1610636540 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe054009620 con 0x7fe0600fe0b0 2024-04-12T05:29:05.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.436+0000 7fe052ffd700 1 -- 172.21.15.67:0/1610636540 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe054009780 con 0x7fe0600fe0b0 2024-04-12T05:29:05.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.436+0000 7fe068638700 1 -- 172.21.15.67:0/1610636540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe06010d710 con 0x7fe0600fe0b0 2024-04-12T05:29:05.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.436+0000 7fe052ffd700 1 -- 172.21.15.67:0/1610636540 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe05401c5b0 con 0x7fe0600fe0b0 2024-04-12T05:29:05.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.436+0000 7fe068638700 1 -- 172.21.15.67:0/1610636540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe06010e280 con 0x7fe0600fe0b0 2024-04-12T05:29:05.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.437+0000 7fe052ffd700 1 -- 172.21.15.67:0/1610636540 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fe05401c7a0 con 0x7fe0600fe0b0 2024-04-12T05:29:05.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.437+0000 7fe052ffd700 1 --2- 172.21.15.67:0/1610636540 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fe04c0368d0 0x7fe04c038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:29:05.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.437+0000 7fe052ffd700 1 -- 172.21.15.67:0/1610636540 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe054049c10 con 0x7fe0600fe0b0 2024-04-12T05:29:05.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.437+0000 7fe068638700 1 -- 172.21.15.67:0/1610636540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe060102e40 con 0x7fe0600fe0b0 2024-04-12T05:29:05.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.437+0000 7fe065bd3700 1 --2- 172.21.15.67:0/1610636540 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fe04c0368d0 0x7fe04c038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:05.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.438+0000 7fe065bd3700 1 --2- 172.21.15.67:0/1610636540 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fe04c0368d0 0x7fe04c038d80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe05c009990 tx=0x7fe05c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:05.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.442+0000 7fe052ffd700 1 -- 172.21.15.67:0/1610636540 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fe054011ba0 con 0x7fe0600fe0b0 2024-04-12T05:29:05.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.720+0000 7fe068638700 1 -- 172.21.15.67:0/1610636540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe06005f5f0 con 0x7fe0600fe0b0 2024-04-12T05:29:05.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.720+0000 7fe052ffd700 1 -- 172.21.15.67:0/1610636540 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fe06005f5f0 con 0x7fe0600fe0b0 2024-04-12T05:29:05.722 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:29:05.723 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":1,"fsid":"37b58ce2-f88d-11ee-bc8c-c7b262605968","modified":"2024-04-12T05:27:04.825332Z","created":"2024-04-12T05:27:04.825332Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-12T05:29:05.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.722+0000 7fe068638700 1 -- 172.21.15.67:0/1610636540 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fe04c0368d0 msgr2=0x7fe04c038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:05.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.722+0000 7fe068638700 1 --2- 172.21.15.67:0/1610636540 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fe04c0368d0 0x7fe04c038d80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe05c009990 tx=0x7fe05c006e40 comp rx=0 tx=0).stop 2024-04-12T05:29:05.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.722+0000 7fe068638700 1 -- 172.21.15.67:0/1610636540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0600fe0b0 msgr2=0x7fe06010cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:05.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.722+0000 7fe068638700 1 --2- 172.21.15.67:0/1610636540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0600fe0b0 0x7fe06010cfd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe054000c00 tx=0x7fe054009e30 comp rx=0 tx=0).stop 2024-04-12T05:29:05.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.722+0000 7fe068638700 1 -- 172.21.15.67:0/1610636540 shutdown_connections 2024-04-12T05:29:05.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.722+0000 7fe068638700 1 --2- 172.21.15.67:0/1610636540 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fe04c0368d0 0x7fe04c038d80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:05.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.722+0000 7fe068638700 1 --2- 172.21.15.67:0/1610636540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0600fe0b0 0x7fe06010cfd0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:05.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.722+0000 7fe068638700 1 -- 172.21.15.67:0/1610636540 >> 172.21.15.67:0/1610636540 conn(0x7fe0600f6850 msgr2=0x7fe0600f70b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:05.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.723+0000 7fe068638700 1 -- 172.21.15.67:0/1610636540 shutdown_connections 2024-04-12T05:29:05.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:05.723+0000 7fe068638700 1 -- 172.21.15.67:0/1610636540 wait complete. 2024-04-12T05:29:05.725 INFO:teuthology.orchestra.run.smithi067.stderr:dumped monmap epoch 1 2024-04-12T05:29:05.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:05 smithi032 ceph-mon[39731]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:05.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:05 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:06.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:06 smithi032 ceph-mon[39731]: from='client.? 172.21.15.67:0/1610636540' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:29:07.268 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-12T05:29:07.268 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph mon dump -f json 2024-04-12T05:29:07.590 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-12T05:29:07.650 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-12T05:29:07.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:07 smithi032 ceph-mon[39731]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:09.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.185+0000 7f73a19ed700 1 -- 172.21.15.67:0/1920829985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739c0fe0a0 msgr2=0x7f739c0fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:09.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.185+0000 7f73a19ed700 1 --2- 172.21.15.67:0/1920829985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739c0fe0a0 0x7f739c0fe4b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f738c0048c0 tx=0x7f738c009e30 comp rx=0 tx=0).stop 2024-04-12T05:29:09.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.185+0000 7f73a19ed700 1 -- 172.21.15.67:0/1920829985 shutdown_connections 2024-04-12T05:29:09.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.185+0000 7f73a19ed700 1 --2- 172.21.15.67:0/1920829985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739c0fe0a0 0x7f739c0fe4b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:09.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.185+0000 7f73a19ed700 1 -- 172.21.15.67:0/1920829985 >> 172.21.15.67:0/1920829985 conn(0x7f739c0f6820 msgr2=0x7f739c0f8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:09.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.185+0000 7f73a19ed700 1 -- 172.21.15.67:0/1920829985 shutdown_connections 2024-04-12T05:29:09.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.185+0000 7f73a19ed700 1 -- 172.21.15.67:0/1920829985 wait complete. 2024-04-12T05:29:09.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.185+0000 7f73a19ed700 1 Processor -- start 2024-04-12T05:29:09.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.185+0000 7f73a19ed700 1 -- start start 2024-04-12T05:29:09.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.186+0000 7f73a19ed700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739c0fe0a0 0x7f739c0fd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:29:09.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.186+0000 7f73a19ed700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739c0fdcf0 con 0x7f739c0fe0a0 2024-04-12T05:29:09.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.186+0000 7f739affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739c0fe0a0 0x7f739c0fd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:09.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.186+0000 7f739affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739c0fe0a0 0x7f739c0fd7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.67:43286/0 (socket says 172.21.15.67:43286) 2024-04-12T05:29:09.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.186+0000 7f739affd700 1 -- 172.21.15.67:0/2500044431 learned_addr learned my addr 172.21.15.67:0/2500044431 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-12T05:29:09.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.186+0000 7f739affd700 1 -- 172.21.15.67:0/2500044431 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f738c005040 con 0x7f739c0fe0a0 2024-04-12T05:29:09.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.186+0000 7f739affd700 1 --2- 172.21.15.67:0/2500044431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739c0fe0a0 0x7f739c0fd7b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f738c009300 tx=0x7f738c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:09.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.187+0000 7f73837fe700 1 -- 172.21.15.67:0/2500044431 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f738c009600 con 0x7f739c0fe0a0 2024-04-12T05:29:09.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.187+0000 7f73837fe700 1 -- 172.21.15.67:0/2500044431 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f738c009760 con 0x7f739c0fe0a0 2024-04-12T05:29:09.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.187+0000 7f73a19ed700 1 -- 172.21.15.67:0/2500044431 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f739c0fbdc0 con 0x7f739c0fe0a0 2024-04-12T05:29:09.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.187+0000 7f73837fe700 1 -- 172.21.15.67:0/2500044431 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f738c01c5b0 con 0x7f739c0fe0a0 2024-04-12T05:29:09.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.187+0000 7f73a19ed700 1 -- 172.21.15.67:0/2500044431 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f739c0fc2b0 con 0x7f739c0fe0a0 2024-04-12T05:29:09.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.187+0000 7f73837fe700 1 -- 172.21.15.67:0/2500044431 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f738c01c870 con 0x7f739c0fe0a0 2024-04-12T05:29:09.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.188+0000 7f73837fe700 1 --2- 172.21.15.67:0/2500044431 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f73840368d0 0x7f7384038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:29:09.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.188+0000 7f73a19ed700 1 -- 172.21.15.67:0/2500044431 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f739c05f5f0 con 0x7f739c0fe0a0 2024-04-12T05:29:09.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.188+0000 7f73837fe700 1 -- 172.21.15.67:0/2500044431 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f738c049b20 con 0x7f739c0fe0a0 2024-04-12T05:29:09.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.188+0000 7f739a7fc700 1 --2- 172.21.15.67:0/2500044431 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f73840368d0 0x7f7384038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:09.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.188+0000 7f739a7fc700 1 --2- 172.21.15.67:0/2500044431 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f73840368d0 0x7f7384038d80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7390009990 tx=0x7f7390006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:09.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.194+0000 7f73837fe700 1 -- 172.21.15.67:0/2500044431 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f738c011db0 con 0x7f739c0fe0a0 2024-04-12T05:29:09.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.469+0000 7f73a19ed700 1 -- 172.21.15.67:0/2500044431 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f739c04bed0 con 0x7f739c0fe0a0 2024-04-12T05:29:09.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.470+0000 7f73837fe700 1 -- 172.21.15.67:0/2500044431 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f738c023990 con 0x7f739c0fe0a0 2024-04-12T05:29:09.472 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:29:09.472 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":1,"fsid":"37b58ce2-f88d-11ee-bc8c-c7b262605968","modified":"2024-04-12T05:27:04.825332Z","created":"2024-04-12T05:27:04.825332Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-12T05:29:09.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.472+0000 7f73a19ed700 1 -- 172.21.15.67:0/2500044431 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f73840368d0 msgr2=0x7f7384038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:09.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.472+0000 7f73a19ed700 1 --2- 172.21.15.67:0/2500044431 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f73840368d0 0x7f7384038d80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7390009990 tx=0x7f7390006e40 comp rx=0 tx=0).stop 2024-04-12T05:29:09.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.472+0000 7f73a19ed700 1 -- 172.21.15.67:0/2500044431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739c0fe0a0 msgr2=0x7f739c0fd7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:09.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.472+0000 7f73a19ed700 1 --2- 172.21.15.67:0/2500044431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739c0fe0a0 0x7f739c0fd7b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f738c009300 tx=0x7f738c009e30 comp rx=0 tx=0).stop 2024-04-12T05:29:09.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.472+0000 7f73a19ed700 1 -- 172.21.15.67:0/2500044431 shutdown_connections 2024-04-12T05:29:09.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.472+0000 7f73a19ed700 1 --2- 172.21.15.67:0/2500044431 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f73840368d0 0x7f7384038d80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:09.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.472+0000 7f73a19ed700 1 --2- 172.21.15.67:0/2500044431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739c0fe0a0 0x7f739c0fd7b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:09.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.472+0000 7f73a19ed700 1 -- 172.21.15.67:0/2500044431 >> 172.21.15.67:0/2500044431 conn(0x7f739c0f6820 msgr2=0x7f739c0f70a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:09.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.472+0000 7f73a19ed700 1 -- 172.21.15.67:0/2500044431 shutdown_connections 2024-04-12T05:29:09.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:09.472+0000 7f73a19ed700 1 -- 172.21.15.67:0/2500044431 wait complete. 2024-04-12T05:29:09.475 INFO:teuthology.orchestra.run.smithi067.stderr:dumped monmap epoch 1 2024-04-12T05:29:09.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:09 smithi032 ceph-mon[39731]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:09.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:09 smithi032 ceph-mon[39731]: from='client.? 172.21.15.67:0/2500044431' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:29:11.101 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-12T05:29:11.102 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph mon dump -f json 2024-04-12T05:29:11.403 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-12T05:29:11.470 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-12T05:29:11.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:11 smithi032 ceph-mon[39731]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:13.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.016+0000 7fc77b9d9700 1 -- 172.21.15.67:0/2861296234 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7740fae90 msgr2=0x7fc7740fb2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:13.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.016+0000 7fc77b9d9700 1 --2- 172.21.15.67:0/2861296234 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7740fae90 0x7fc7740fb2c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc7680048c0 tx=0x7fc768009e30 comp rx=0 tx=0).stop 2024-04-12T05:29:13.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.016+0000 7fc77b9d9700 1 -- 172.21.15.67:0/2861296234 shutdown_connections 2024-04-12T05:29:13.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.016+0000 7fc77b9d9700 1 --2- 172.21.15.67:0/2861296234 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7740fae90 0x7fc7740fb2c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:13.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.016+0000 7fc77b9d9700 1 -- 172.21.15.67:0/2861296234 >> 172.21.15.67:0/2861296234 conn(0x7fc7740f6850 msgr2=0x7fc7740f8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:13.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.016+0000 7fc77b9d9700 1 -- 172.21.15.67:0/2861296234 shutdown_connections 2024-04-12T05:29:13.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.016+0000 7fc77b9d9700 1 -- 172.21.15.67:0/2861296234 wait complete. 2024-04-12T05:29:13.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.017+0000 7fc77b9d9700 1 Processor -- start 2024-04-12T05:29:13.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.017+0000 7fc77b9d9700 1 -- start start 2024-04-12T05:29:13.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.017+0000 7fc77b9d9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7740fae90 0x7fc77410cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:29:13.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.017+0000 7fc77b9d9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77410d510 con 0x7fc7740fae90 2024-04-12T05:29:13.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.017+0000 7fc779775700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7740fae90 0x7fc77410cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:13.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.017+0000 7fc779775700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7740fae90 0x7fc77410cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.67:40832/0 (socket says 172.21.15.67:40832) 2024-04-12T05:29:13.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.017+0000 7fc779775700 1 -- 172.21.15.67:0/1117097848 learned_addr learned my addr 172.21.15.67:0/1117097848 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-12T05:29:13.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.017+0000 7fc779775700 1 -- 172.21.15.67:0/1117097848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc768005040 con 0x7fc7740fae90 2024-04-12T05:29:13.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.018+0000 7fc779775700 1 --2- 172.21.15.67:0/1117097848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7740fae90 0x7fc77410cfd0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc76800b3b0 tx=0x7fc768009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:13.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.018+0000 7fc765ffb700 1 -- 172.21.15.67:0/1117097848 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7680095d0 con 0x7fc7740fae90 2024-04-12T05:29:13.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.018+0000 7fc77b9d9700 1 -- 172.21.15.67:0/1117097848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc77410d710 con 0x7fc7740fae90 2024-04-12T05:29:13.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.018+0000 7fc765ffb700 1 -- 172.21.15.67:0/1117097848 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc768009730 con 0x7fc7740fae90 2024-04-12T05:29:13.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.018+0000 7fc765ffb700 1 -- 172.21.15.67:0/1117097848 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc76801c5b0 con 0x7fc7740fae90 2024-04-12T05:29:13.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.018+0000 7fc77b9d9700 1 -- 172.21.15.67:0/1117097848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc77410e280 con 0x7fc7740fae90 2024-04-12T05:29:13.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.019+0000 7fc765ffb700 1 -- 172.21.15.67:0/1117097848 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fc76801c830 con 0x7fc7740fae90 2024-04-12T05:29:13.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.019+0000 7fc765ffb700 1 --2- 172.21.15.67:0/1117097848 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fc760036880 0x7fc760038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:29:13.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.019+0000 7fc765ffb700 1 -- 172.21.15.67:0/1117097848 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc768049a60 con 0x7fc7740fae90 2024-04-12T05:29:13.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.019+0000 7fc77b9d9700 1 -- 172.21.15.67:0/1117097848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc774102e40 con 0x7fc7740fae90 2024-04-12T05:29:13.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.019+0000 7fc778f74700 1 --2- 172.21.15.67:0/1117097848 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fc760036880 0x7fc760038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:13.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.020+0000 7fc778f74700 1 --2- 172.21.15.67:0/1117097848 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fc760036880 0x7fc760038d30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc770009990 tx=0x7fc770006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:13.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.028+0000 7fc765ffb700 1 -- 172.21.15.67:0/1117097848 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc768011dd0 con 0x7fc7740fae90 2024-04-12T05:29:13.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.303+0000 7fc77b9d9700 1 -- 172.21.15.67:0/1117097848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc77405f5f0 con 0x7fc7740fae90 2024-04-12T05:29:13.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.304+0000 7fc765ffb700 1 -- 172.21.15.67:0/1117097848 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fc77405f5f0 con 0x7fc7740fae90 2024-04-12T05:29:13.305 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:29:13.305 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":1,"fsid":"37b58ce2-f88d-11ee-bc8c-c7b262605968","modified":"2024-04-12T05:27:04.825332Z","created":"2024-04-12T05:27:04.825332Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-12T05:29:13.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.306+0000 7fc77b9d9700 1 -- 172.21.15.67:0/1117097848 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fc760036880 msgr2=0x7fc760038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:13.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.306+0000 7fc77b9d9700 1 --2- 172.21.15.67:0/1117097848 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fc760036880 0x7fc760038d30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc770009990 tx=0x7fc770006e40 comp rx=0 tx=0).stop 2024-04-12T05:29:13.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.306+0000 7fc77b9d9700 1 -- 172.21.15.67:0/1117097848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7740fae90 msgr2=0x7fc77410cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:13.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.306+0000 7fc77b9d9700 1 --2- 172.21.15.67:0/1117097848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7740fae90 0x7fc77410cfd0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc76800b3b0 tx=0x7fc768009e30 comp rx=0 tx=0).stop 2024-04-12T05:29:13.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.306+0000 7fc77b9d9700 1 -- 172.21.15.67:0/1117097848 shutdown_connections 2024-04-12T05:29:13.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.306+0000 7fc77b9d9700 1 --2- 172.21.15.67:0/1117097848 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fc760036880 0x7fc760038d30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:13.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.306+0000 7fc77b9d9700 1 --2- 172.21.15.67:0/1117097848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7740fae90 0x7fc77410cfd0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:13.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.306+0000 7fc77b9d9700 1 -- 172.21.15.67:0/1117097848 >> 172.21.15.67:0/1117097848 conn(0x7fc7740f6850 msgr2=0x7fc7740f70b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:13.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.306+0000 7fc77b9d9700 1 -- 172.21.15.67:0/1117097848 shutdown_connections 2024-04-12T05:29:13.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:13.306+0000 7fc77b9d9700 1 -- 172.21.15.67:0/1117097848 wait complete. 2024-04-12T05:29:13.308 INFO:teuthology.orchestra.run.smithi067.stderr:dumped monmap epoch 1 2024-04-12T05:29:13.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:13 smithi032 ceph-mon[39731]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:13.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:13 smithi032 ceph-mon[39731]: from='client.? 172.21.15.67:0/1117097848' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:29:14.901 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-12T05:29:14.901 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph mon dump -f json 2024-04-12T05:29:15.195 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-12T05:29:15.256 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-12T05:29:15.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:15 smithi032 ceph-mon[39731]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:16.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:16.842+0000 7f5b53a58700 1 -- 172.21.15.67:0/2836452758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b4c0fd040 msgr2=0x7f5b4c0fd450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:16.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:16.842+0000 7f5b53a58700 1 --2- 172.21.15.67:0/2836452758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b4c0fd040 0x7f5b4c0fd450 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5b480048c0 tx=0x7f5b48009e30 comp rx=0 tx=0).stop 2024-04-12T05:29:16.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:16.842+0000 7f5b53a58700 1 -- 172.21.15.67:0/2836452758 shutdown_connections 2024-04-12T05:29:16.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:16.842+0000 7f5b53a58700 1 --2- 172.21.15.67:0/2836452758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b4c0fd040 0x7f5b4c0fd450 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:16.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:16.842+0000 7f5b53a58700 1 -- 172.21.15.67:0/2836452758 >> 172.21.15.67:0/2836452758 conn(0x7f5b4c0f70c0 msgr2=0x7f5b4c0f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:16.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:16.842+0000 7f5b53a58700 1 -- 172.21.15.67:0/2836452758 shutdown_connections 2024-04-12T05:29:16.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:16.842+0000 7f5b53a58700 1 -- 172.21.15.67:0/2836452758 wait complete. 2024-04-12T05:29:16.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:16.842+0000 7f5b53a58700 1 Processor -- start 2024-04-12T05:29:16.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:16.842+0000 7f5b53a58700 1 -- start start 2024-04-12T05:29:16.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:16.843+0000 7f5b53a58700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b4c0fd040 0x7f5b4c066480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:29:16.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:16.843+0000 7f5b53a58700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4c10fc90 con 0x7f5b4c0fd040 2024-04-12T05:29:16.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:16.843+0000 7f5b517f4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b4c0fd040 0x7f5b4c066480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:16.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:16.843+0000 7f5b517f4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b4c0fd040 0x7f5b4c066480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.67:40860/0 (socket says 172.21.15.67:40860) 2024-04-12T05:29:16.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:16.843+0000 7f5b517f4700 1 -- 172.21.15.67:0/1215365541 learned_addr learned my addr 172.21.15.67:0/1215365541 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-12T05:29:16.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:16.843+0000 7f5b517f4700 1 -- 172.21.15.67:0/1215365541 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b48005040 con 0x7f5b4c0fd040 2024-04-12T05:29:16.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:16.843+0000 7f5b517f4700 1 --2- 172.21.15.67:0/1215365541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b4c0fd040 0x7f5b4c066480 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5b48000c00 tx=0x7f5b48009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:16.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:16.844+0000 7f5b41ffb700 1 -- 172.21.15.67:0/1215365541 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b48009620 con 0x7f5b4c0fd040 2024-04-12T05:29:16.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:16.844+0000 7f5b41ffb700 1 -- 172.21.15.67:0/1215365541 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b48009780 con 0x7f5b4c0fd040 2024-04-12T05:29:16.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:16.844+0000 7f5b41ffb700 1 -- 172.21.15.67:0/1215365541 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b4801c5b0 con 0x7f5b4c0fd040 2024-04-12T05:29:16.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:16.844+0000 7f5b53a58700 1 -- 172.21.15.67:0/1215365541 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b4c10fdd0 con 0x7f5b4c0fd040 2024-04-12T05:29:16.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:16.844+0000 7f5b53a58700 1 -- 172.21.15.67:0/1215365541 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b4c1102c0 con 0x7f5b4c0fd040 2024-04-12T05:29:16.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:16.845+0000 7f5b41ffb700 1 -- 172.21.15.67:0/1215365541 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f5b48013a60 con 0x7f5b4c0fd040 2024-04-12T05:29:16.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:16.845+0000 7f5b53a58700 1 -- 172.21.15.67:0/1215365541 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b4c106050 con 0x7f5b4c0fd040 2024-04-12T05:29:16.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:16.845+0000 7f5b41ffb700 1 --2- 172.21.15.67:0/1215365541 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f5b380368d0 0x7f5b38038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:29:16.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:16.845+0000 7f5b41ffb700 1 -- 172.21.15.67:0/1215365541 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5b48049ae0 con 0x7f5b4c0fd040 2024-04-12T05:29:16.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:16.845+0000 7f5b50ff3700 1 --2- 172.21.15.67:0/1215365541 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f5b380368d0 0x7f5b38038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:16.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:16.846+0000 7f5b50ff3700 1 --2- 172.21.15.67:0/1215365541 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f5b380368d0 0x7f5b38038d80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c009990 tx=0x7f5b3c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:16.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:16.850+0000 7f5b41ffb700 1 -- 172.21.15.67:0/1215365541 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5b48025400 con 0x7f5b4c0fd040 2024-04-12T05:29:17.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:17.127+0000 7f5b53a58700 1 -- 172.21.15.67:0/1215365541 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5b4c05f5f0 con 0x7f5b4c0fd040 2024-04-12T05:29:17.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:17.128+0000 7f5b41ffb700 1 -- 172.21.15.67:0/1215365541 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f5b48011b60 con 0x7f5b4c0fd040 2024-04-12T05:29:17.128 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:29:17.128 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":1,"fsid":"37b58ce2-f88d-11ee-bc8c-c7b262605968","modified":"2024-04-12T05:27:04.825332Z","created":"2024-04-12T05:27:04.825332Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-12T05:29:17.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:17.129+0000 7f5b53a58700 1 -- 172.21.15.67:0/1215365541 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f5b380368d0 msgr2=0x7f5b38038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:17.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:17.130+0000 7f5b53a58700 1 --2- 172.21.15.67:0/1215365541 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f5b380368d0 0x7f5b38038d80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c009990 tx=0x7f5b3c006e40 comp rx=0 tx=0).stop 2024-04-12T05:29:17.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:17.130+0000 7f5b53a58700 1 -- 172.21.15.67:0/1215365541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b4c0fd040 msgr2=0x7f5b4c066480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:17.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:17.130+0000 7f5b53a58700 1 --2- 172.21.15.67:0/1215365541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b4c0fd040 0x7f5b4c066480 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5b48000c00 tx=0x7f5b48009e30 comp rx=0 tx=0).stop 2024-04-12T05:29:17.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:17.130+0000 7f5b53a58700 1 -- 172.21.15.67:0/1215365541 shutdown_connections 2024-04-12T05:29:17.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:17.130+0000 7f5b53a58700 1 --2- 172.21.15.67:0/1215365541 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7f5b380368d0 0x7f5b38038d80 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:17.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:17.130+0000 7f5b53a58700 1 --2- 172.21.15.67:0/1215365541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b4c0fd040 0x7f5b4c066480 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:17.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:17.130+0000 7f5b53a58700 1 -- 172.21.15.67:0/1215365541 >> 172.21.15.67:0/1215365541 conn(0x7f5b4c0f70c0 msgr2=0x7f5b4c0f7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:17.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:17.130+0000 7f5b53a58700 1 -- 172.21.15.67:0/1215365541 shutdown_connections 2024-04-12T05:29:17.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:17.130+0000 7f5b53a58700 1 -- 172.21.15.67:0/1215365541 wait complete. 2024-04-12T05:29:17.131 INFO:teuthology.orchestra.run.smithi067.stderr:dumped monmap epoch 1 2024-04-12T05:29:17.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:17 smithi032 ceph-mon[39731]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:17.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:17 smithi032 ceph-mon[39731]: from='client.? 172.21.15.67:0/1215365541' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:29:17.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:17 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:17.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:17 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:17.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:17 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:17.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:17 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-12T05:29:18.748 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-12T05:29:18.748 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph mon dump -f json 2024-04-12T05:29:19.044 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-12T05:29:19.104 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-12T05:29:19.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:19 smithi032 ceph-mon[39731]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-12T05:29:19.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:19 smithi032 ceph-mon[39731]: mgrmap e14: smithi032.hrwxrh(active, since 63s) 2024-04-12T05:29:20.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.591+0000 7fd20fd91700 1 -- 172.21.15.67:0/3363445297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2080fca50 msgr2=0x7fd2080fce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:20.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.591+0000 7fd20fd91700 1 --2- 172.21.15.67:0/3363445297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2080fca50 0x7fd2080fce60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd2040048c0 tx=0x7fd204009e30 comp rx=0 tx=0).stop 2024-04-12T05:29:20.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.591+0000 7fd20fd91700 1 -- 172.21.15.67:0/3363445297 shutdown_connections 2024-04-12T05:29:20.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.591+0000 7fd20fd91700 1 --2- 172.21.15.67:0/3363445297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2080fca50 0x7fd2080fce60 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:20.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.591+0000 7fd20fd91700 1 -- 172.21.15.67:0/3363445297 >> 172.21.15.67:0/3363445297 conn(0x7fd2080f81d0 msgr2=0x7fd2080fa5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:20.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.591+0000 7fd20fd91700 1 -- 172.21.15.67:0/3363445297 shutdown_connections 2024-04-12T05:29:20.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.591+0000 7fd20fd91700 1 -- 172.21.15.67:0/3363445297 wait complete. 2024-04-12T05:29:20.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.592+0000 7fd20fd91700 1 Processor -- start 2024-04-12T05:29:20.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.592+0000 7fd20fd91700 1 -- start start 2024-04-12T05:29:20.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.592+0000 7fd20fd91700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2080fca50 0x7fd208071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:29:20.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.592+0000 7fd20fd91700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2080722a0 con 0x7fd2080fca50 2024-04-12T05:29:20.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.592+0000 7fd20db2d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2080fca50 0x7fd208071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:20.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.592+0000 7fd20db2d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2080fca50 0x7fd208071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.67:40874/0 (socket says 172.21.15.67:40874) 2024-04-12T05:29:20.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.592+0000 7fd20db2d700 1 -- 172.21.15.67:0/76916362 learned_addr learned my addr 172.21.15.67:0/76916362 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-12T05:29:20.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.592+0000 7fd20db2d700 1 -- 172.21.15.67:0/76916362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd204005040 con 0x7fd2080fca50 2024-04-12T05:29:20.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.593+0000 7fd20db2d700 1 --2- 172.21.15.67:0/76916362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2080fca50 0x7fd208071d60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd204009300 tx=0x7fd204009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:20.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.593+0000 7fd1fe7fc700 1 -- 172.21.15.67:0/76916362 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd204009600 con 0x7fd2080fca50 2024-04-12T05:29:20.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.593+0000 7fd1fe7fc700 1 -- 172.21.15.67:0/76916362 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd204009760 con 0x7fd2080fca50 2024-04-12T05:29:20.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.593+0000 7fd20fd91700 1 -- 172.21.15.67:0/76916362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd208070370 con 0x7fd2080fca50 2024-04-12T05:29:20.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.593+0000 7fd1fe7fc700 1 -- 172.21.15.67:0/76916362 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd20401c5b0 con 0x7fd2080fca50 2024-04-12T05:29:20.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.593+0000 7fd20fd91700 1 -- 172.21.15.67:0/76916362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd208070860 con 0x7fd2080fca50 2024-04-12T05:29:20.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.594+0000 7fd1fe7fc700 1 -- 172.21.15.67:0/76916362 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 14) v1 ==== 43438+0+0 (secure 0 0 0) 0x7fd20401c820 con 0x7fd2080fca50 2024-04-12T05:29:20.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.594+0000 7fd1fe7fc700 1 --2- 172.21.15.67:0/76916362 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fd1f4036920 0x7fd1f4038dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:29:20.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.594+0000 7fd1fe7fc700 1 -- 172.21.15.67:0/76916362 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd204049bb0 con 0x7fd2080fca50 2024-04-12T05:29:20.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.594+0000 7fd20d32c700 1 -- 172.21.15.67:0/76916362 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fd1f4036920 msgr2=0x7fd1f4038dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/3971961003 2024-04-12T05:29:20.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.594+0000 7fd20fd91700 1 -- 172.21.15.67:0/76916362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd20805f5f0 con 0x7fd2080fca50 2024-04-12T05:29:20.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.594+0000 7fd20d32c700 1 --2- 172.21.15.67:0/76916362 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fd1f4036920 0x7fd1f4038dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-12T05:29:20.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.599+0000 7fd1fe7fc700 1 -- 172.21.15.67:0/76916362 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fd204023300 con 0x7fd2080fca50 2024-04-12T05:29:20.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.795+0000 7fd20d32c700 1 -- 172.21.15.67:0/76916362 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fd1f4036920 msgr2=0x7fd1f4038dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/3971961003 2024-04-12T05:29:20.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.795+0000 7fd20d32c700 1 --2- 172.21.15.67:0/76916362 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fd1f4036920 0x7fd1f4038dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-12T05:29:20.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.871+0000 7fd20fd91700 1 -- 172.21.15.67:0/76916362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd208106b70 con 0x7fd2080fca50 2024-04-12T05:29:20.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.871+0000 7fd1fe7fc700 1 -- 172.21.15.67:0/76916362 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fd204011b60 con 0x7fd2080fca50 2024-04-12T05:29:20.872 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:29:20.872 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":1,"fsid":"37b58ce2-f88d-11ee-bc8c-c7b262605968","modified":"2024-04-12T05:27:04.825332Z","created":"2024-04-12T05:27:04.825332Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-12T05:29:20.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.873+0000 7fd20fd91700 1 -- 172.21.15.67:0/76916362 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fd1f4036920 msgr2=0x7fd1f4038dd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T05:29:20.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.873+0000 7fd20fd91700 1 --2- 172.21.15.67:0/76916362 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fd1f4036920 0x7fd1f4038dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:20.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.873+0000 7fd20fd91700 1 -- 172.21.15.67:0/76916362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2080fca50 msgr2=0x7fd208071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:20.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.873+0000 7fd20fd91700 1 --2- 172.21.15.67:0/76916362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2080fca50 0x7fd208071d60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd204009300 tx=0x7fd204009e30 comp rx=0 tx=0).stop 2024-04-12T05:29:20.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.873+0000 7fd20fd91700 1 -- 172.21.15.67:0/76916362 shutdown_connections 2024-04-12T05:29:20.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.873+0000 7fd20fd91700 1 --2- 172.21.15.67:0/76916362 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7fd1f4036920 0x7fd1f4038dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:20.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.873+0000 7fd20fd91700 1 --2- 172.21.15.67:0/76916362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2080fca50 0x7fd208071d60 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:20.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.873+0000 7fd20fd91700 1 -- 172.21.15.67:0/76916362 >> 172.21.15.67:0/76916362 conn(0x7fd2080f81d0 msgr2=0x7fd2080f8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:20.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.873+0000 7fd20fd91700 1 -- 172.21.15.67:0/76916362 shutdown_connections 2024-04-12T05:29:20.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:20.873+0000 7fd20fd91700 1 -- 172.21.15.67:0/76916362 wait complete. 2024-04-12T05:29:20.875 INFO:teuthology.orchestra.run.smithi067.stderr:dumped monmap epoch 1 2024-04-12T05:29:21.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:20 smithi032 ceph-mon[39731]: from='client.? 172.21.15.67:0/76916362' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:29:22.490 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-12T05:29:22.490 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph mon dump -f json 2024-04-12T05:29:22.791 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-12T05:29:22.855 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-12T05:29:24.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.482+0000 7eff4dd7d700 1 -- 172.21.15.67:0/3108612879 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff480fd050 msgr2=0x7eff480fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:24.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.482+0000 7eff4dd7d700 1 --2- 172.21.15.67:0/3108612879 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff480fd050 0x7eff480fd460 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7eff380048c0 tx=0x7eff38009e30 comp rx=0 tx=0).stop 2024-04-12T05:29:24.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.482+0000 7eff4dd7d700 1 -- 172.21.15.67:0/3108612879 shutdown_connections 2024-04-12T05:29:24.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.482+0000 7eff4dd7d700 1 --2- 172.21.15.67:0/3108612879 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff480fd050 0x7eff480fd460 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:24.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.482+0000 7eff4dd7d700 1 -- 172.21.15.67:0/3108612879 >> 172.21.15.67:0/3108612879 conn(0x7eff480f70d0 msgr2=0x7eff480f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:24.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.482+0000 7eff4dd7d700 1 -- 172.21.15.67:0/3108612879 shutdown_connections 2024-04-12T05:29:24.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.482+0000 7eff4dd7d700 1 -- 172.21.15.67:0/3108612879 wait complete. 2024-04-12T05:29:24.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.482+0000 7eff4dd7d700 1 Processor -- start 2024-04-12T05:29:24.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.482+0000 7eff4dd7d700 1 -- start start 2024-04-12T05:29:24.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.483+0000 7eff4dd7d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff480fd050 0x7eff4810d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:29:24.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.483+0000 7eff4dd7d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4810d540 con 0x7eff480fd050 2024-04-12T05:29:24.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.483+0000 7eff477fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff480fd050 0x7eff4810d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:24.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.483+0000 7eff477fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff480fd050 0x7eff4810d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.67:54554/0 (socket says 172.21.15.67:54554) 2024-04-12T05:29:24.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.483+0000 7eff477fe700 1 -- 172.21.15.67:0/1424179801 learned_addr learned my addr 172.21.15.67:0/1424179801 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-12T05:29:24.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.483+0000 7eff477fe700 1 -- 172.21.15.67:0/1424179801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff38005040 con 0x7eff480fd050 2024-04-12T05:29:24.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.483+0000 7eff477fe700 1 --2- 172.21.15.67:0/1424179801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff480fd050 0x7eff4810d000 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7eff38009300 tx=0x7eff38009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:24.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.483+0000 7eff3ffff700 1 -- 172.21.15.67:0/1424179801 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff38009600 con 0x7eff480fd050 2024-04-12T05:29:24.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.483+0000 7eff3ffff700 1 -- 172.21.15.67:0/1424179801 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff38009760 con 0x7eff480fd050 2024-04-12T05:29:24.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.483+0000 7eff4dd7d700 1 -- 172.21.15.67:0/1424179801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff4810d740 con 0x7eff480fd050 2024-04-12T05:29:24.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.483+0000 7eff3ffff700 1 -- 172.21.15.67:0/1424179801 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff3801c5b0 con 0x7eff480fd050 2024-04-12T05:29:24.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.483+0000 7eff4dd7d700 1 -- 172.21.15.67:0/1424179801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff4810e260 con 0x7eff480fd050 2024-04-12T05:29:24.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.484+0000 7eff3ffff700 1 -- 172.21.15.67:0/1424179801 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 14) v1 ==== 43438+0+0 (secure 0 0 0) 0x7eff38013420 con 0x7eff480fd050 2024-04-12T05:29:24.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.484+0000 7eff4dd7d700 1 -- 172.21.15.67:0/1424179801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff48101d30 con 0x7eff480fd050 2024-04-12T05:29:24.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.484+0000 7eff3ffff700 1 --2- 172.21.15.67:0/1424179801 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7eff340368d0 0x7eff34038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:29:24.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.484+0000 7eff3ffff700 1 -- 172.21.15.67:0/1424179801 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7eff38049c00 con 0x7eff480fd050 2024-04-12T05:29:24.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.484+0000 7eff46ffd700 1 -- 172.21.15.67:0/1424179801 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7eff340368d0 msgr2=0x7eff34038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/3971961003 2024-04-12T05:29:24.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.484+0000 7eff46ffd700 1 --2- 172.21.15.67:0/1424179801 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7eff340368d0 0x7eff34038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-12T05:29:24.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.489+0000 7eff3ffff700 1 -- 172.21.15.67:0/1424179801 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7eff38025400 con 0x7eff480fd050 2024-04-12T05:29:24.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.685+0000 7eff46ffd700 1 -- 172.21.15.67:0/1424179801 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7eff340368d0 msgr2=0x7eff34038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/3971961003 2024-04-12T05:29:24.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.685+0000 7eff46ffd700 1 --2- 172.21.15.67:0/1424179801 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7eff340368d0 0x7eff34038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-12T05:29:24.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.765+0000 7eff4dd7d700 1 -- 172.21.15.67:0/1424179801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7eff4810e4a0 con 0x7eff480fd050 2024-04-12T05:29:24.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.765+0000 7eff3ffff700 1 -- 172.21.15.67:0/1424179801 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7eff38049e60 con 0x7eff480fd050 2024-04-12T05:29:24.767 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:29:24.767 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":1,"fsid":"37b58ce2-f88d-11ee-bc8c-c7b262605968","modified":"2024-04-12T05:27:04.825332Z","created":"2024-04-12T05:27:04.825332Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-12T05:29:24.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.767+0000 7eff4dd7d700 1 -- 172.21.15.67:0/1424179801 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7eff340368d0 msgr2=0x7eff34038d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T05:29:24.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.767+0000 7eff4dd7d700 1 --2- 172.21.15.67:0/1424179801 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7eff340368d0 0x7eff34038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:24.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.767+0000 7eff4dd7d700 1 -- 172.21.15.67:0/1424179801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff480fd050 msgr2=0x7eff4810d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:24.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.767+0000 7eff4dd7d700 1 --2- 172.21.15.67:0/1424179801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff480fd050 0x7eff4810d000 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7eff38009300 tx=0x7eff38009e30 comp rx=0 tx=0).stop 2024-04-12T05:29:24.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.767+0000 7eff4dd7d700 1 -- 172.21.15.67:0/1424179801 shutdown_connections 2024-04-12T05:29:24.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.767+0000 7eff4dd7d700 1 --2- 172.21.15.67:0/1424179801 >> [v2:172.21.15.32:6800/3971961003,v1:172.21.15.32:6801/3971961003] conn(0x7eff340368d0 0x7eff34038d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:24.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.767+0000 7eff4dd7d700 1 --2- 172.21.15.67:0/1424179801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff480fd050 0x7eff4810d000 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:24.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.767+0000 7eff4dd7d700 1 -- 172.21.15.67:0/1424179801 >> 172.21.15.67:0/1424179801 conn(0x7eff480f70d0 msgr2=0x7eff48065070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:24.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.767+0000 7eff4dd7d700 1 -- 172.21.15.67:0/1424179801 shutdown_connections 2024-04-12T05:29:24.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:24.767+0000 7eff4dd7d700 1 -- 172.21.15.67:0/1424179801 wait complete. 2024-04-12T05:29:24.770 INFO:teuthology.orchestra.run.smithi067.stderr:dumped monmap epoch 1 2024-04-12T05:29:25.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:24 smithi032 ceph-mon[39731]: from='client.? 172.21.15.67:0/1424179801' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:29:26.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:25 smithi032 ceph-mon[39731]: Active manager daemon smithi032.hrwxrh restarted 2024-04-12T05:29:26.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:25 smithi032 ceph-mon[39731]: Activating manager daemon smithi032.hrwxrh 2024-04-12T05:29:26.094 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:25 smithi032 ceph-mon[39731]: osdmap e5: 0 total, 0 up, 0 in 2024-04-12T05:29:26.094 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:25 smithi032 ceph-mon[39731]: mgrmap e15: smithi032.hrwxrh(active, starting, since 0.0029503s) 2024-04-12T05:29:26.094 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:25 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-12T05:29:26.094 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:25 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr metadata", "who": "smithi032.hrwxrh", "id": "smithi032.hrwxrh"}]: dispatch 2024-04-12T05:29:26.094 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:25 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T05:29:26.094 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:25 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T05:29:26.094 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:25 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T05:29:26.094 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:25 smithi032 ceph-mon[39731]: Manager daemon smithi032.hrwxrh is now available 2024-04-12T05:29:26.094 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:25 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:29:26.094 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:25 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:29:26.094 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:25 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:26.094 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:25 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:29:26.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:25 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.hrwxrh/mirror_snapshot_schedule"}]: dispatch 2024-04-12T05:29:26.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:25 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.hrwxrh/trash_purge_schedule"}]: dispatch 2024-04-12T05:29:26.095 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:25 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:26.447 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-12T05:29:26.447 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph mon dump -f json 2024-04-12T05:29:26.748 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-12T05:29:26.807 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-12T05:29:27.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:27 smithi032 ceph-mon[39731]: mgrmap e16: smithi032.hrwxrh(active, since 1.00578s) 2024-04-12T05:29:27.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:27 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:27.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:27 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:28.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:28 smithi032 ceph-mon[39731]: [12/Apr/2024:05:29:27] ENGINE Bus STARTING 2024-04-12T05:29:28.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:28 smithi032 ceph-mon[39731]: [12/Apr/2024:05:29:27] ENGINE Serving on https://172.21.15.32:7150 2024-04-12T05:29:28.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:28 smithi032 ceph-mon[39731]: [12/Apr/2024:05:29:27] ENGINE Bus STARTED 2024-04-12T05:29:28.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:28 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:28.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:28 smithi032 ceph-mon[39731]: mgrmap e17: smithi032.hrwxrh(active, since 3s) 2024-04-12T05:29:29.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.504+0000 7fa8c0020700 1 -- 172.21.15.67:0/995069222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8b80fd050 msgr2=0x7fa8b80fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:29.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.504+0000 7fa8c0020700 1 --2- 172.21.15.67:0/995069222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8b80fd050 0x7fa8b80fd460 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa8b40048c0 tx=0x7fa8b4009e30 comp rx=0 tx=0).stop 2024-04-12T05:29:29.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.504+0000 7fa8c0020700 1 -- 172.21.15.67:0/995069222 shutdown_connections 2024-04-12T05:29:29.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.504+0000 7fa8c0020700 1 --2- 172.21.15.67:0/995069222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8b80fd050 0x7fa8b80fd460 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:29.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.504+0000 7fa8c0020700 1 -- 172.21.15.67:0/995069222 >> 172.21.15.67:0/995069222 conn(0x7fa8b80f70d0 msgr2=0x7fa8b80f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:29.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.504+0000 7fa8c0020700 1 -- 172.21.15.67:0/995069222 shutdown_connections 2024-04-12T05:29:29.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.504+0000 7fa8c0020700 1 -- 172.21.15.67:0/995069222 wait complete. 2024-04-12T05:29:29.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.504+0000 7fa8c0020700 1 Processor -- start 2024-04-12T05:29:29.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.505+0000 7fa8c0020700 1 -- start start 2024-04-12T05:29:29.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.505+0000 7fa8c0020700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8b80fd050 0x7fa8b8065bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:29:29.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.505+0000 7fa8c0020700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b810fcd0 con 0x7fa8b80fd050 2024-04-12T05:29:29.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.505+0000 7fa8bddbc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8b80fd050 0x7fa8b8065bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:29.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.505+0000 7fa8bddbc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8b80fd050 0x7fa8b8065bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.67:54580/0 (socket says 172.21.15.67:54580) 2024-04-12T05:29:29.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.505+0000 7fa8bddbc700 1 -- 172.21.15.67:0/3879287035 learned_addr learned my addr 172.21.15.67:0/3879287035 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-12T05:29:29.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.505+0000 7fa8bddbc700 1 -- 172.21.15.67:0/3879287035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b4005040 con 0x7fa8b80fd050 2024-04-12T05:29:29.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.505+0000 7fa8bddbc700 1 --2- 172.21.15.67:0/3879287035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8b80fd050 0x7fa8b8065bd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009300 tx=0x7fa8b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:29.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.505+0000 7fa8ae7fc700 1 -- 172.21.15.67:0/3879287035 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8b4009600 con 0x7fa8b80fd050 2024-04-12T05:29:29.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.505+0000 7fa8ae7fc700 1 -- 172.21.15.67:0/3879287035 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa8b4009760 con 0x7fa8b80fd050 2024-04-12T05:29:29.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.506+0000 7fa8c0020700 1 -- 172.21.15.67:0/3879287035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b810fe10 con 0x7fa8b80fd050 2024-04-12T05:29:29.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.506+0000 7fa8ae7fc700 1 -- 172.21.15.67:0/3879287035 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8b401c5b0 con 0x7fa8b80fd050 2024-04-12T05:29:29.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.506+0000 7fa8c0020700 1 -- 172.21.15.67:0/3879287035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b8110300 con 0x7fa8b80fd050 2024-04-12T05:29:29.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.506+0000 7fa8ae7fc700 1 -- 172.21.15.67:0/3879287035 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7fa8b4013420 con 0x7fa8b80fd050 2024-04-12T05:29:29.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.506+0000 7fa8c0020700 1 -- 172.21.15.67:0/3879287035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b8106060 con 0x7fa8b80fd050 2024-04-12T05:29:29.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.507+0000 7fa8ae7fc700 1 --2- 172.21.15.67:0/3879287035 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fa8a4036970 0x7fa8a4038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:29:29.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.507+0000 7fa8ae7fc700 1 -- 172.21.15.67:0/3879287035 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fa8b4049d30 con 0x7fa8b80fd050 2024-04-12T05:29:29.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.507+0000 7fa8bd5bb700 1 --2- 172.21.15.67:0/3879287035 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fa8a4036970 0x7fa8a4038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:29.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.507+0000 7fa8bd5bb700 1 --2- 172.21.15.67:0/3879287035 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fa8a4036970 0x7fa8a4038e20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa8a8009990 tx=0x7fa8a8006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:29.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.512+0000 7fa8ae7fc700 1 -- 172.21.15.67:0/3879287035 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fa8b4011550 con 0x7fa8b80fd050 2024-04-12T05:29:29.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.791+0000 7fa8c0020700 1 -- 172.21.15.67:0/3879287035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa8b8106060 con 0x7fa8b80fd050 2024-04-12T05:29:29.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.792+0000 7fa8ae7fc700 1 -- 172.21.15.67:0/3879287035 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fa8b4011370 con 0x7fa8b80fd050 2024-04-12T05:29:29.794 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:29:29.794 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":1,"fsid":"37b58ce2-f88d-11ee-bc8c-c7b262605968","modified":"2024-04-12T05:27:04.825332Z","created":"2024-04-12T05:27:04.825332Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-12T05:29:29.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.794+0000 7fa8c0020700 1 -- 172.21.15.67:0/3879287035 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fa8a4036970 msgr2=0x7fa8a4038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:29.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.794+0000 7fa8c0020700 1 --2- 172.21.15.67:0/3879287035 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fa8a4036970 0x7fa8a4038e20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa8a8009990 tx=0x7fa8a8006e40 comp rx=0 tx=0).stop 2024-04-12T05:29:29.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.794+0000 7fa8c0020700 1 -- 172.21.15.67:0/3879287035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8b80fd050 msgr2=0x7fa8b8065bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:29.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.794+0000 7fa8c0020700 1 --2- 172.21.15.67:0/3879287035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8b80fd050 0x7fa8b8065bd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009300 tx=0x7fa8b4009e30 comp rx=0 tx=0).stop 2024-04-12T05:29:29.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.794+0000 7fa8c0020700 1 -- 172.21.15.67:0/3879287035 shutdown_connections 2024-04-12T05:29:29.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.794+0000 7fa8c0020700 1 --2- 172.21.15.67:0/3879287035 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fa8a4036970 0x7fa8a4038e20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:29.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.794+0000 7fa8c0020700 1 --2- 172.21.15.67:0/3879287035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8b80fd050 0x7fa8b8065bd0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:29.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.794+0000 7fa8c0020700 1 -- 172.21.15.67:0/3879287035 >> 172.21.15.67:0/3879287035 conn(0x7fa8b80f70d0 msgr2=0x7fa8b80f7990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:29.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.794+0000 7fa8c0020700 1 -- 172.21.15.67:0/3879287035 shutdown_connections 2024-04-12T05:29:29.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:29.794+0000 7fa8c0020700 1 -- 172.21.15.67:0/3879287035 wait complete. 2024-04-12T05:29:29.796 INFO:teuthology.orchestra.run.smithi067.stderr:dumped monmap epoch 1 2024-04-12T05:29:30.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:30 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:30.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:30 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:30.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:30 smithi032 ceph-mon[39731]: from='client.? 172.21.15.67:0/3879287035' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:29:30.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:30 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:30.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:30 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:30.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:30 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "who": "osd/host:smithi032", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:29:31.559 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-12T05:29:31.559 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph mon dump -f json 2024-04-12T05:29:31.863 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-12T05:29:31.943 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-12T05:29:32.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:32 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:32.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:32 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:32.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:32 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:32.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:32 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:32.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:32 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "who": "osd/host:smithi067", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:29:32.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:32 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:29:32.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:32 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:29:32.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:32 smithi032 ceph-mon[39731]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-12T05:29:32.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:32 smithi032 ceph-mon[39731]: Updating smithi067:/etc/ceph/ceph.conf 2024-04-12T05:29:32.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:32 smithi032 ceph-mon[39731]: Updating smithi032:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:29:32.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:32 smithi032 ceph-mon[39731]: Updating smithi067:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:29:32.845 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:32 smithi032 ceph-mon[39731]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:29:32.845 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:32 smithi032 ceph-mon[39731]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:29:32.845 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:32 smithi032 ceph-mon[39731]: Updating smithi032:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:29:32.845 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:32 smithi032 ceph-mon[39731]: Updating smithi067:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:29:33.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:33.747+0000 7f1ef442f700 1 -- 172.21.15.67:0/3451181440 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1eec0702e0 msgr2=0x7f1eec0706b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:33.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:33.747+0000 7f1ef442f700 1 --2- 172.21.15.67:0/3451181440 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1eec0702e0 0x7f1eec0706b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1ee80048c0 tx=0x7f1ee8009e30 comp rx=0 tx=0).stop 2024-04-12T05:29:33.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:33.747+0000 7f1ef442f700 1 -- 172.21.15.67:0/3451181440 shutdown_connections 2024-04-12T05:29:33.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:33.747+0000 7f1ef442f700 1 --2- 172.21.15.67:0/3451181440 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1eec0702e0 0x7f1eec0706b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:33.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:33.747+0000 7f1ef442f700 1 -- 172.21.15.67:0/3451181440 >> 172.21.15.67:0/3451181440 conn(0x7f1eec0f85e0 msgr2=0x7f1eec0fa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:33.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:33.747+0000 7f1ef442f700 1 -- 172.21.15.67:0/3451181440 shutdown_connections 2024-04-12T05:29:33.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:33.747+0000 7f1ef442f700 1 -- 172.21.15.67:0/3451181440 wait complete. 2024-04-12T05:29:33.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:33.748+0000 7f1ef442f700 1 Processor -- start 2024-04-12T05:29:33.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:33.748+0000 7f1ef442f700 1 -- start start 2024-04-12T05:29:33.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:33.748+0000 7f1ef442f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1eec0702e0 0x7f1eec1153d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:29:33.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:33.748+0000 7f1ef442f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eec115910 con 0x7f1eec0702e0 2024-04-12T05:29:33.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:33.749+0000 7f1ef21cb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1eec0702e0 0x7f1eec1153d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:33.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:33.749+0000 7f1ef21cb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1eec0702e0 0x7f1eec1153d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.67:40674/0 (socket says 172.21.15.67:40674) 2024-04-12T05:29:33.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:33.749+0000 7f1ef21cb700 1 -- 172.21.15.67:0/1364686093 learned_addr learned my addr 172.21.15.67:0/1364686093 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-12T05:29:33.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:33.749+0000 7f1ef21cb700 1 -- 172.21.15.67:0/1364686093 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee8005040 con 0x7f1eec0702e0 2024-04-12T05:29:33.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:33.749+0000 7f1ef21cb700 1 --2- 172.21.15.67:0/1364686093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1eec0702e0 0x7f1eec1153d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8007e80 tx=0x7f1ee8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:33.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:33.749+0000 7f1ee2ffd700 1 -- 172.21.15.67:0/1364686093 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ee80093c0 con 0x7f1eec0702e0 2024-04-12T05:29:33.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:33.749+0000 7f1ee2ffd700 1 -- 172.21.15.67:0/1364686093 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ee8013420 con 0x7f1eec0702e0 2024-04-12T05:29:33.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:33.749+0000 7f1ef442f700 1 -- 172.21.15.67:0/1364686093 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eec115b10 con 0x7f1eec0702e0 2024-04-12T05:29:33.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:33.749+0000 7f1ee2ffd700 1 -- 172.21.15.67:0/1364686093 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ee801c5b0 con 0x7f1eec0702e0 2024-04-12T05:29:33.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:33.749+0000 7f1ef442f700 1 -- 172.21.15.67:0/1364686093 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eec115f80 con 0x7f1eec0702e0 2024-04-12T05:29:33.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:33.750+0000 7f1ee2ffd700 1 -- 172.21.15.67:0/1364686093 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f1ee801c7e0 con 0x7f1eec0702e0 2024-04-12T05:29:33.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:33.751+0000 7f1ef442f700 1 -- 172.21.15.67:0/1364686093 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eec05f5f0 con 0x7f1eec0702e0 2024-04-12T05:29:33.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:33.751+0000 7f1ee2ffd700 1 --2- 172.21.15.67:0/1364686093 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f1ed80369c0 0x7f1ed8038e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:29:33.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:33.751+0000 7f1ee2ffd700 1 -- 172.21.15.67:0/1364686093 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f1ee8049a30 con 0x7f1eec0702e0 2024-04-12T05:29:33.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:33.751+0000 7f1ef19ca700 1 --2- 172.21.15.67:0/1364686093 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f1ed80369c0 0x7f1ed8038e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:33.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:33.752+0000 7f1ef19ca700 1 --2- 172.21.15.67:0/1364686093 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f1ed80369c0 0x7f1ed8038e70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1edc009990 tx=0x7f1edc006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:33.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:33.756+0000 7f1ee2ffd700 1 -- 172.21.15.67:0/1364686093 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f1ee8023300 con 0x7f1eec0702e0 2024-04-12T05:29:33.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:33 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:33.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:33 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:33.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:33 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:33.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:33 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:33.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:33 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:33.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:33 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi067", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-12T05:29:33.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:33 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi067", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-12T05:29:33.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:33 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:29:33.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:33 smithi032 ceph-mon[39731]: Deploying daemon crash.smithi067 on smithi067 2024-04-12T05:29:34.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:34.031+0000 7f1ef442f700 1 -- 172.21.15.67:0/1364686093 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1eec110a20 con 0x7f1eec0702e0 2024-04-12T05:29:34.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:34.032+0000 7f1ee2ffd700 1 -- 172.21.15.67:0/1364686093 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f1ee8011b60 con 0x7f1eec0702e0 2024-04-12T05:29:34.034 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:29:34.034 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":1,"fsid":"37b58ce2-f88d-11ee-bc8c-c7b262605968","modified":"2024-04-12T05:27:04.825332Z","created":"2024-04-12T05:27:04.825332Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-12T05:29:34.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:34.034+0000 7f1ef442f700 1 -- 172.21.15.67:0/1364686093 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f1ed80369c0 msgr2=0x7f1ed8038e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:34.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:34.034+0000 7f1ef442f700 1 --2- 172.21.15.67:0/1364686093 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f1ed80369c0 0x7f1ed8038e70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1edc009990 tx=0x7f1edc006e40 comp rx=0 tx=0).stop 2024-04-12T05:29:34.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:34.034+0000 7f1ef442f700 1 -- 172.21.15.67:0/1364686093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1eec0702e0 msgr2=0x7f1eec1153d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:34.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:34.034+0000 7f1ef442f700 1 --2- 172.21.15.67:0/1364686093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1eec0702e0 0x7f1eec1153d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8007e80 tx=0x7f1ee8009e30 comp rx=0 tx=0).stop 2024-04-12T05:29:34.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:34.034+0000 7f1ef442f700 1 -- 172.21.15.67:0/1364686093 shutdown_connections 2024-04-12T05:29:34.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:34.034+0000 7f1ef442f700 1 --2- 172.21.15.67:0/1364686093 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f1ed80369c0 0x7f1ed8038e70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:34.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:34.034+0000 7f1ef442f700 1 --2- 172.21.15.67:0/1364686093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1eec0702e0 0x7f1eec1153d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:34.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:34.034+0000 7f1ef442f700 1 -- 172.21.15.67:0/1364686093 >> 172.21.15.67:0/1364686093 conn(0x7f1eec0f85e0 msgr2=0x7f1eec105b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:34.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:34.034+0000 7f1ef442f700 1 -- 172.21.15.67:0/1364686093 shutdown_connections 2024-04-12T05:29:34.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:34.034+0000 7f1ef442f700 1 -- 172.21.15.67:0/1364686093 wait complete. 2024-04-12T05:29:34.036 INFO:teuthology.orchestra.run.smithi067.stderr:dumped monmap epoch 1 2024-04-12T05:29:34.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:34 smithi032 ceph-mon[39731]: from='client.? 172.21.15.67:0/1364686093' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:29:35.847 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-12T05:29:35.847 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph mon dump -f json 2024-04-12T05:29:36.687 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:29:38.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:37 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:38.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:37 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:38.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:37 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:38.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:37 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:38.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:37 smithi032 ceph-mon[39731]: Deploying daemon node-exporter.smithi067 on smithi067 2024-04-12T05:29:38.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.605+0000 7f56cb9e5700 1 -- 172.21.15.67:0/2074691631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56c41019c0 msgr2=0x7f56c4101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:38.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.605+0000 7f56cb9e5700 1 --2- 172.21.15.67:0/2074691631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56c41019c0 0x7f56c4101d90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f56b4009c60 tx=0x7f56b4004d10 comp rx=0 tx=0).stop 2024-04-12T05:29:38.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.605+0000 7f56cb9e5700 1 -- 172.21.15.67:0/2074691631 shutdown_connections 2024-04-12T05:29:38.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.605+0000 7f56cb9e5700 1 --2- 172.21.15.67:0/2074691631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56c41019c0 0x7f56c4101d90 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:38.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.605+0000 7f56cb9e5700 1 -- 172.21.15.67:0/2074691631 >> 172.21.15.67:0/2074691631 conn(0x7f56c40f74e0 msgr2=0x7f56c40f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:38.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.606+0000 7f56cb9e5700 1 -- 172.21.15.67:0/2074691631 shutdown_connections 2024-04-12T05:29:38.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.606+0000 7f56cb9e5700 1 -- 172.21.15.67:0/2074691631 wait complete. 2024-04-12T05:29:38.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.606+0000 7f56cb9e5700 1 Processor -- start 2024-04-12T05:29:38.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.606+0000 7f56cb9e5700 1 -- start start 2024-04-12T05:29:38.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.606+0000 7f56cb9e5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56c41019c0 0x7f56c4112060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:29:38.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.606+0000 7f56cb9e5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c41125a0 con 0x7f56c41019c0 2024-04-12T05:29:38.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.607+0000 7f56c9781700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56c41019c0 0x7f56c4112060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:38.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.607+0000 7f56c9781700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56c41019c0 0x7f56c4112060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.67:40694/0 (socket says 172.21.15.67:40694) 2024-04-12T05:29:38.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.607+0000 7f56c9781700 1 -- 172.21.15.67:0/3787005929 learned_addr learned my addr 172.21.15.67:0/3787005929 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-12T05:29:38.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.607+0000 7f56c9781700 1 -- 172.21.15.67:0/3787005929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b4009940 con 0x7f56c41019c0 2024-04-12T05:29:38.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.607+0000 7f56c9781700 1 --2- 172.21.15.67:0/3787005929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56c41019c0 0x7f56c4112060 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f56b4007da0 tx=0x7f56b4004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:38.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.607+0000 7f56c1ffb700 1 -- 172.21.15.67:0/3787005929 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56b4009640 con 0x7f56c41019c0 2024-04-12T05:29:38.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.607+0000 7f56cb9e5700 1 -- 172.21.15.67:0/3787005929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56c41127a0 con 0x7f56c41019c0 2024-04-12T05:29:38.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.607+0000 7f56c1ffb700 1 -- 172.21.15.67:0/3787005929 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f56b40097a0 con 0x7f56c41019c0 2024-04-12T05:29:38.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.607+0000 7f56c1ffb700 1 -- 172.21.15.67:0/3787005929 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56b4012790 con 0x7f56c41019c0 2024-04-12T05:29:38.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.607+0000 7f56cb9e5700 1 -- 172.21.15.67:0/3787005929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56c4112c90 con 0x7f56c41019c0 2024-04-12T05:29:38.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.608+0000 7f56c1ffb700 1 -- 172.21.15.67:0/3787005929 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f56b4019040 con 0x7f56c41019c0 2024-04-12T05:29:38.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.608+0000 7f56cb9e5700 1 -- 172.21.15.67:0/3787005929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56c40fd0c0 con 0x7f56c41019c0 2024-04-12T05:29:38.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.608+0000 7f56c1ffb700 1 --2- 172.21.15.67:0/3787005929 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f56ac0369c0 0x7f56ac038e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:29:38.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.609+0000 7f56c1ffb700 1 -- 172.21.15.67:0/3787005929 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f56b404a3e0 con 0x7f56c41019c0 2024-04-12T05:29:38.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.609+0000 7f56c8f80700 1 --2- 172.21.15.67:0/3787005929 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f56ac0369c0 0x7f56ac038e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:38.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.609+0000 7f56c8f80700 1 --2- 172.21.15.67:0/3787005929 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f56ac0369c0 0x7f56ac038e70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f56b8009990 tx=0x7f56b8006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:38.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.614+0000 7f56c1ffb700 1 -- 172.21.15.67:0/3787005929 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f56b401dc10 con 0x7f56c41019c0 2024-04-12T05:29:38.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.891+0000 7f56cb9e5700 1 -- 172.21.15.67:0/3787005929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f56c405f5f0 con 0x7f56c41019c0 2024-04-12T05:29:38.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.891+0000 7f56c1ffb700 1 -- 172.21.15.67:0/3787005929 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f56b4021020 con 0x7f56c41019c0 2024-04-12T05:29:38.893 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:29:38.893 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":1,"fsid":"37b58ce2-f88d-11ee-bc8c-c7b262605968","modified":"2024-04-12T05:27:04.825332Z","created":"2024-04-12T05:27:04.825332Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-12T05:29:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.893+0000 7f56cb9e5700 1 -- 172.21.15.67:0/3787005929 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f56ac0369c0 msgr2=0x7f56ac038e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.893+0000 7f56cb9e5700 1 --2- 172.21.15.67:0/3787005929 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f56ac0369c0 0x7f56ac038e70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f56b8009990 tx=0x7f56b8006e40 comp rx=0 tx=0).stop 2024-04-12T05:29:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.893+0000 7f56cb9e5700 1 -- 172.21.15.67:0/3787005929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56c41019c0 msgr2=0x7f56c4112060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.893+0000 7f56cb9e5700 1 --2- 172.21.15.67:0/3787005929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56c41019c0 0x7f56c4112060 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f56b4007da0 tx=0x7f56b4004d10 comp rx=0 tx=0).stop 2024-04-12T05:29:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.893+0000 7f56cb9e5700 1 -- 172.21.15.67:0/3787005929 shutdown_connections 2024-04-12T05:29:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.893+0000 7f56cb9e5700 1 --2- 172.21.15.67:0/3787005929 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f56ac0369c0 0x7f56ac038e70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.893+0000 7f56cb9e5700 1 --2- 172.21.15.67:0/3787005929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56c41019c0 0x7f56c4112060 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.893+0000 7f56cb9e5700 1 -- 172.21.15.67:0/3787005929 >> 172.21.15.67:0/3787005929 conn(0x7f56c40f74e0 msgr2=0x7f56c40f8ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.893+0000 7f56cb9e5700 1 -- 172.21.15.67:0/3787005929 shutdown_connections 2024-04-12T05:29:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:38.893+0000 7f56cb9e5700 1 -- 172.21.15.67:0/3787005929 wait complete. 2024-04-12T05:29:38.896 INFO:teuthology.orchestra.run.smithi067.stderr:dumped monmap epoch 1 2024-04-12T05:29:39.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:38 smithi032 ceph-mon[39731]: from='client.? 172.21.15.67:0/3787005929' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:29:41.120 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-12T05:29:41.121 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph mon dump -f json 2024-04-12T05:29:41.402 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:29:41.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:41 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:41.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:41 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:29:42.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:42 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:42.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:42 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:42.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:42 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:42.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:42 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:42.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:42 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi067.dwichg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:29:42.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:42 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi067.dwichg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-12T05:29:42.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:42 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:29:42.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:42 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:29:42.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:42 smithi032 ceph-mon[39731]: Deploying daemon mgr.smithi067.dwichg on smithi067 2024-04-12T05:29:44.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.350+0000 7fba9fc5f700 1 -- 172.21.15.67:0/2690753889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba98102ae0 msgr2=0x7fba98102eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:44.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.350+0000 7fba9fc5f700 1 --2- 172.21.15.67:0/2690753889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba98102ae0 0x7fba98102eb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fba880048c0 tx=0x7fba88009e30 comp rx=0 tx=0).stop 2024-04-12T05:29:44.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.350+0000 7fba9fc5f700 1 -- 172.21.15.67:0/2690753889 shutdown_connections 2024-04-12T05:29:44.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.350+0000 7fba9fc5f700 1 --2- 172.21.15.67:0/2690753889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba98102ae0 0x7fba98102eb0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:44.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.350+0000 7fba9fc5f700 1 -- 172.21.15.67:0/2690753889 >> 172.21.15.67:0/2690753889 conn(0x7fba980f85e0 msgr2=0x7fba980fa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:44.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.350+0000 7fba9fc5f700 1 -- 172.21.15.67:0/2690753889 shutdown_connections 2024-04-12T05:29:44.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.350+0000 7fba9fc5f700 1 -- 172.21.15.67:0/2690753889 wait complete. 2024-04-12T05:29:44.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.350+0000 7fba9fc5f700 1 Processor -- start 2024-04-12T05:29:44.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.350+0000 7fba9fc5f700 1 -- start start 2024-04-12T05:29:44.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.350+0000 7fba9fc5f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba98102ae0 0x7fba981130b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:29:44.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.350+0000 7fba9fc5f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba981135f0 con 0x7fba98102ae0 2024-04-12T05:29:44.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.351+0000 7fba9d9fb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba98102ae0 0x7fba981130b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:44.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.351+0000 7fba9d9fb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba98102ae0 0x7fba981130b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.67:54164/0 (socket says 172.21.15.67:54164) 2024-04-12T05:29:44.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.351+0000 7fba9d9fb700 1 -- 172.21.15.67:0/1415801765 learned_addr learned my addr 172.21.15.67:0/1415801765 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-12T05:29:44.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.351+0000 7fba9d9fb700 1 -- 172.21.15.67:0/1415801765 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba88005040 con 0x7fba98102ae0 2024-04-12T05:29:44.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.351+0000 7fba9d9fb700 1 --2- 172.21.15.67:0/1415801765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba98102ae0 0x7fba981130b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fba88007e80 tx=0x7fba88009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:44.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.351+0000 7fba8e7fc700 1 -- 172.21.15.67:0/1415801765 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba880093c0 con 0x7fba98102ae0 2024-04-12T05:29:44.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.351+0000 7fba9fc5f700 1 -- 172.21.15.67:0/1415801765 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba981137f0 con 0x7fba98102ae0 2024-04-12T05:29:44.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.351+0000 7fba8e7fc700 1 -- 172.21.15.67:0/1415801765 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba88013420 con 0x7fba98102ae0 2024-04-12T05:29:44.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.351+0000 7fba8e7fc700 1 -- 172.21.15.67:0/1415801765 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba8801c5b0 con 0x7fba98102ae0 2024-04-12T05:29:44.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.351+0000 7fba9fc5f700 1 -- 172.21.15.67:0/1415801765 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba98113ce0 con 0x7fba98102ae0 2024-04-12T05:29:44.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.352+0000 7fba8e7fc700 1 -- 172.21.15.67:0/1415801765 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7fba88013a00 con 0x7fba98102ae0 2024-04-12T05:29:44.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.352+0000 7fba9fc5f700 1 -- 172.21.15.67:0/1415801765 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba980fe1e0 con 0x7fba98102ae0 2024-04-12T05:29:44.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.352+0000 7fba8e7fc700 1 --2- 172.21.15.67:0/1415801765 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fba84036970 0x7fba84038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:29:44.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.352+0000 7fba8e7fc700 1 -- 172.21.15.67:0/1415801765 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fba88049aa0 con 0x7fba98102ae0 2024-04-12T05:29:44.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.353+0000 7fba9d1fa700 1 --2- 172.21.15.67:0/1415801765 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fba84036970 0x7fba84038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:44.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.353+0000 7fba9d1fa700 1 --2- 172.21.15.67:0/1415801765 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fba84036970 0x7fba84038e20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fba94009990 tx=0x7fba94006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:44.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.358+0000 7fba8e7fc700 1 -- 172.21.15.67:0/1415801765 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fba88011940 con 0x7fba98102ae0 2024-04-12T05:29:44.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.650+0000 7fba9fc5f700 1 -- 172.21.15.67:0/1415801765 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fba9805f5f0 con 0x7fba98102ae0 2024-04-12T05:29:44.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.651+0000 7fba8e7fc700 1 -- 172.21.15.67:0/1415801765 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fba88011370 con 0x7fba98102ae0 2024-04-12T05:29:44.653 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:29:44.653 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":1,"fsid":"37b58ce2-f88d-11ee-bc8c-c7b262605968","modified":"2024-04-12T05:27:04.825332Z","created":"2024-04-12T05:27:04.825332Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-12T05:29:44.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.655+0000 7fba9fc5f700 1 -- 172.21.15.67:0/1415801765 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fba84036970 msgr2=0x7fba84038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:44.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.655+0000 7fba9fc5f700 1 --2- 172.21.15.67:0/1415801765 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fba84036970 0x7fba84038e20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fba94009990 tx=0x7fba94006e40 comp rx=0 tx=0).stop 2024-04-12T05:29:44.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.655+0000 7fba9fc5f700 1 -- 172.21.15.67:0/1415801765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba98102ae0 msgr2=0x7fba981130b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:44.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.655+0000 7fba9fc5f700 1 --2- 172.21.15.67:0/1415801765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba98102ae0 0x7fba981130b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fba88007e80 tx=0x7fba88009e30 comp rx=0 tx=0).stop 2024-04-12T05:29:44.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.655+0000 7fba9fc5f700 1 -- 172.21.15.67:0/1415801765 shutdown_connections 2024-04-12T05:29:44.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.655+0000 7fba9fc5f700 1 --2- 172.21.15.67:0/1415801765 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fba84036970 0x7fba84038e20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:44.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.655+0000 7fba9fc5f700 1 --2- 172.21.15.67:0/1415801765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba98102ae0 0x7fba981130b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:44.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.655+0000 7fba9fc5f700 1 -- 172.21.15.67:0/1415801765 >> 172.21.15.67:0/1415801765 conn(0x7fba980f85e0 msgr2=0x7fba980f9dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:44.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.655+0000 7fba9fc5f700 1 -- 172.21.15.67:0/1415801765 shutdown_connections 2024-04-12T05:29:44.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:44.655+0000 7fba9fc5f700 1 -- 172.21.15.67:0/1415801765 wait complete. 2024-04-12T05:29:44.657 INFO:teuthology.orchestra.run.smithi067.stderr:dumped monmap epoch 1 2024-04-12T05:29:46.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:45 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:46.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:45 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:46.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:45 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:46.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:45 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:46.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:45 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T05:29:46.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:45 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:29:46.094 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:45 smithi032 ceph-mon[39731]: Deploying daemon mon.smithi067 on smithi067 2024-04-12T05:29:46.094 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:45 smithi032 ceph-mon[39731]: from='client.? 172.21.15.67:0/1415801765' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:29:46.094 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:45 smithi032 ceph-mon[39731]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:46.094 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:45 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:46.341 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-12T05:29:46.341 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph mon dump -f json 2024-04-12T05:29:47.514 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi067/config 2024-04-12T05:29:48.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:48 smithi032 ceph-mon[39731]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:48.988 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: mon.smithi067@-1(synchronizing).mds e1 new map 2024-04-12T05:29:48.988 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: mon.smithi067@-1(synchronizing).mds e1 print_map 2024-04-12T05:29:48.988 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: e1 2024-04-12T05:29:48.988 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:29:48.989 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:29:48.989 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: legacy client fscid: -1 2024-04-12T05:29:48.989 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: 2024-04-12T05:29:48.989 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: No filesystems configured 2024-04-12T05:29:48.989 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: mon.smithi067@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-12T05:29:48.989 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: mon.smithi067@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-12T05:29:48.989 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: mon.smithi067@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-12T05:29:48.989 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: mon.smithi067@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-12T05:29:48.989 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: mon.smithi067@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-12T05:29:48.989 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: mon.smithi067@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-12T05:29:48.989 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: mon.smithi067@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-04-12T05:29:48.989 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: mon.smithi067@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-12T05:29:48.989 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: mon.smithi067@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-12T05:29:48.989 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: mon.smithi067@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-12T05:29:48.989 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: mon.smithi067@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-12T05:29:48.989 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: Deploying daemon node-exporter.smithi032 on smithi032 2024-04-12T05:29:48.989 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:48.990 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:48.990 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:48.990 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi067", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:29:48.990 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:48.990 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:48.990 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:48.990 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:48.990 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: Deploying daemon alertmanager.smithi032 on smithi032 2024-04-12T05:29:48.990 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: Deploying cephadm binary to smithi067 2024-04-12T05:29:48.990 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:48.990 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:48.990 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: Added host smithi067 2024-04-12T05:29:48.990 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:49.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:49.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T05:29:49.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:49.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.274 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.274 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.274 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-12T05:29:49.274 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-12T05:29:49.274 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.274 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: Deploying daemon grafana.smithi032 on smithi032 2024-04-12T05:29:49.274 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:49.274 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/2110090777' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-12T05:29:49.274 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/2110090777' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-12T05:29:49.274 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: osdmap e4: 0 total, 0 up, 0 in 2024-04-12T05:29:49.274 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:49.274 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.274 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:49.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi032:172.21.15.32=smithi032;smithi067:172.21.15.67=smithi067", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:29:49.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: Saving service mon spec with placement smithi032:172.21.15.32=smithi032;smithi067:172.21.15.67=smithi067;count:2 2024-04-12T05:29:49.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:49.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='client.? 172.21.15.67:0/2637313301' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:29:49.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:49.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:49.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='client.? 172.21.15.67:0/1583561843' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:29:49.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:49.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.276 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.276 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: Deploying daemon prometheus.smithi032 on smithi032 2024-04-12T05:29:49.276 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:49.276 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='client.? 172.21.15.67:0/2469506172' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:29:49.276 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:49.276 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:49.276 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.276 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='client.? 172.21.15.67:0/1610636540' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:29:49.276 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:49.276 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:49.276 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='client.? 172.21.15.67:0/2500044431' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:29:49.276 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:49.276 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:49.276 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='client.? 172.21.15.67:0/1117097848' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:29:49.276 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:49.276 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:49.277 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='client.? 172.21.15.67:0/1215365541' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:29:49.277 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.277 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.277 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.277 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-12T05:29:49.277 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14160 172.21.15.32:0/2322820913' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-12T05:29:49.277 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: mgrmap e14: smithi032.hrwxrh(active, since 63s) 2024-04-12T05:29:49.277 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='client.? 172.21.15.67:0/76916362' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:29:49.277 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='client.? 172.21.15.67:0/1424179801' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:29:49.277 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: Active manager daemon smithi032.hrwxrh restarted 2024-04-12T05:29:49.277 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: Activating manager daemon smithi032.hrwxrh 2024-04-12T05:29:49.277 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: osdmap e5: 0 total, 0 up, 0 in 2024-04-12T05:29:49.277 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: mgrmap e15: smithi032.hrwxrh(active, starting, since 0.0029503s) 2024-04-12T05:29:49.277 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-12T05:29:49.278 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr metadata", "who": "smithi032.hrwxrh", "id": "smithi032.hrwxrh"}]: dispatch 2024-04-12T05:29:49.278 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T05:29:49.278 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T05:29:49.278 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T05:29:49.278 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: Manager daemon smithi032.hrwxrh is now available 2024-04-12T05:29:49.278 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:29:49.278 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:29:49.278 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.278 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:29:49.278 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.hrwxrh/mirror_snapshot_schedule"}]: dispatch 2024-04-12T05:29:49.278 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.hrwxrh/trash_purge_schedule"}]: dispatch 2024-04-12T05:29:49.278 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.278 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: mgrmap e16: smithi032.hrwxrh(active, since 1.00578s) 2024-04-12T05:29:49.278 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.278 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.279 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: [12/Apr/2024:05:29:27] ENGINE Bus STARTING 2024-04-12T05:29:49.279 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: [12/Apr/2024:05:29:27] ENGINE Serving on https://172.21.15.32:7150 2024-04-12T05:29:49.279 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: [12/Apr/2024:05:29:27] ENGINE Bus STARTED 2024-04-12T05:29:49.279 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.279 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: mgrmap e17: smithi032.hrwxrh(active, since 3s) 2024-04-12T05:29:49.279 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.279 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.279 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='client.? 172.21.15.67:0/3879287035' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:29:49.279 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.279 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.279 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "who": "osd/host:smithi032", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:29:49.279 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.279 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.279 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "who": "osd/host:smithi067", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:29:49.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:29:49.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:29:49.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-12T05:29:49.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: Updating smithi067:/etc/ceph/ceph.conf 2024-04-12T05:29:49.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: Updating smithi032:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:29:49.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: Updating smithi067:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:29:49.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:29:49.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:29:49.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: Updating smithi032:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:29:49.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: Updating smithi067:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:29:49.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi067", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-12T05:29:49.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi067", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-12T05:29:49.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:29:49.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: Deploying daemon crash.smithi067 on smithi067 2024-04-12T05:29:49.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='client.? 172.21.15.67:0/1364686093' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:29:49.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: Deploying daemon node-exporter.smithi067 on smithi067 2024-04-12T05:29:49.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='client.? 172.21.15.67:0/3787005929' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:29:49.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:29:49.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi067.dwichg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:29:49.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi067.dwichg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-12T05:29:49.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:29:49.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:29:49.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: Deploying daemon mgr.smithi067.dwichg on smithi067 2024-04-12T05:29:49.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T05:29:49.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:29:49.283 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: Deploying daemon mon.smithi067 on smithi067 2024-04-12T05:29:49.283 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='client.? 172.21.15.67:0/1415801765' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:29:49.283 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:49.283 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:49.283 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:49.283 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:48 smithi067 ceph-mon[49010]: mon.smithi067@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-04-12T05:29:49.283 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:49 smithi067 bash[48917]: 157be8a8c6d8e4791568d92e481af4b3a889fb5277aeee77222474ef090b86e9 2024-04-12T05:29:49.283 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:49 smithi067 systemd[1]: Started Ceph mon.smithi067 for 37b58ce2-f88d-11ee-bc8c-c7b262605968. 2024-04-12T05:29:54.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:54 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-12T05:29:54.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:54 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-12T05:29:54.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:54 smithi032 ceph-mon[39731]: mon.smithi032 calling monitor election 2024-04-12T05:29:54.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:54 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-12T05:29:54.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:54 smithi032 ceph-mon[39731]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:54.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:54 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-12T05:29:54.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:54 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-12T05:29:54.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:54 smithi032 ceph-mon[39731]: mon.smithi067 calling monitor election 2024-04-12T05:29:54.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:54 smithi032 ceph-mon[39731]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:54.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:54 smithi032 ceph-mon[39731]: from='mgr.? 172.21.15.67:0/1142203720' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi067.dwichg/crt"}]: dispatch 2024-04-12T05:29:54.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:54 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-12T05:29:54.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:54 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-12T05:29:54.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:54 smithi032 ceph-mon[39731]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:54.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:54 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-12T05:29:54.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:54 smithi032 ceph-mon[39731]: mon.smithi032 is new leader, mons smithi032,smithi067 in quorum (ranks 0,1) 2024-04-12T05:29:54.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:54 smithi032 ceph-mon[39731]: monmap e2: 2 mons at {smithi032=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0],smithi067=[v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-12T05:29:54.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:54 smithi032 ceph-mon[39731]: fsmap 2024-04-12T05:29:54.345 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:54 smithi032 ceph-mon[39731]: osdmap e5: 0 total, 0 up, 0 in 2024-04-12T05:29:54.345 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:54 smithi032 ceph-mon[39731]: mgrmap e17: smithi032.hrwxrh(active, since 28s) 2024-04-12T05:29:54.345 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:54 smithi032 ceph-mon[39731]: overall HEALTH_OK 2024-04-12T05:29:54.345 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:54 smithi032 ceph-mon[39731]: Standby manager daemon smithi067.dwichg started 2024-04-12T05:29:54.345 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:54 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:54.345 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:54 smithi032 ceph-mon[39731]: from='mgr.? 172.21.15.67:0/1142203720' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T05:29:54.345 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:54 smithi032 ceph-mon[39731]: from='mgr.? 172.21.15.67:0/1142203720' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi067.dwichg/key"}]: dispatch 2024-04-12T05:29:54.345 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:54 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:54.345 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:54 smithi032 ceph-mon[39731]: from='mgr.? 172.21.15.67:0/1142203720' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T05:29:54.345 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:54 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:54.345 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:54 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:54.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:54 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-12T05:29:54.523 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:54 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-12T05:29:54.523 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:54 smithi067 ceph-mon[49010]: mon.smithi032 calling monitor election 2024-04-12T05:29:54.523 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:54 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-12T05:29:54.523 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:54 smithi067 ceph-mon[49010]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:54.523 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:54 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-12T05:29:54.523 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:54 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-12T05:29:54.523 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:54 smithi067 ceph-mon[49010]: mon.smithi067 calling monitor election 2024-04-12T05:29:54.523 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:54 smithi067 ceph-mon[49010]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:54.523 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:54 smithi067 ceph-mon[49010]: from='mgr.? 172.21.15.67:0/1142203720' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi067.dwichg/crt"}]: dispatch 2024-04-12T05:29:54.523 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:54 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-12T05:29:54.523 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:54 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-12T05:29:54.524 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:54 smithi067 ceph-mon[49010]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:54.524 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:54 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-12T05:29:54.524 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:54 smithi067 ceph-mon[49010]: mon.smithi032 is new leader, mons smithi032,smithi067 in quorum (ranks 0,1) 2024-04-12T05:29:54.524 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:54 smithi067 ceph-mon[49010]: monmap e2: 2 mons at {smithi032=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0],smithi067=[v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-12T05:29:54.524 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:54 smithi067 ceph-mon[49010]: fsmap 2024-04-12T05:29:54.524 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:54 smithi067 ceph-mon[49010]: osdmap e5: 0 total, 0 up, 0 in 2024-04-12T05:29:54.524 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:54 smithi067 ceph-mon[49010]: mgrmap e17: smithi032.hrwxrh(active, since 28s) 2024-04-12T05:29:54.524 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:54 smithi067 ceph-mon[49010]: overall HEALTH_OK 2024-04-12T05:29:54.524 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:54 smithi067 ceph-mon[49010]: Standby manager daemon smithi067.dwichg started 2024-04-12T05:29:54.524 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:54 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:54.524 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:54 smithi067 ceph-mon[49010]: from='mgr.? 172.21.15.67:0/1142203720' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T05:29:54.524 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:54 smithi067 ceph-mon[49010]: from='mgr.? 172.21.15.67:0/1142203720' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi067.dwichg/key"}]: dispatch 2024-04-12T05:29:54.524 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:54 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:54.525 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:54 smithi067 ceph-mon[49010]: from='mgr.? 172.21.15.67:0/1142203720' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T05:29:54.525 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:54 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:54.525 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:54 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:55.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:55 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:55.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:55 smithi067 ceph-mon[49010]: mgrmap e18: smithi032.hrwxrh(active, since 29s), standbys: smithi067.dwichg 2024-04-12T05:29:55.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:55 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr metadata", "who": "smithi067.dwichg", "id": "smithi067.dwichg"}]: dispatch 2024-04-12T05:29:55.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:55 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:29:55.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:55 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-12T05:29:55.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.578+0000 7f0d1c966700 1 -- 172.21.15.67:0/1225872255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d000061f0 msgr2=0x7f0d00008650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:55.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.578+0000 7f0d1c966700 1 --2- 172.21.15.67:0/1225872255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d000061f0 0x7f0d00008650 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0d18065770 tx=0x7f0d08002040 comp rx=0 tx=0).stop 2024-04-12T05:29:55.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.578+0000 7f0d1c966700 1 -- 172.21.15.67:0/1225872255 shutdown_connections 2024-04-12T05:29:55.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.578+0000 7f0d1c966700 1 --2- 172.21.15.67:0/1225872255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d000061f0 0x7f0d00008650 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:55.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.578+0000 7f0d1c966700 1 -- 172.21.15.67:0/1225872255 >> 172.21.15.67:0/1225872255 conn(0x7f0d18071960 msgr2=0x7f0d18071d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:55.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.578+0000 7f0d1c966700 1 -- 172.21.15.67:0/1225872255 shutdown_connections 2024-04-12T05:29:55.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.578+0000 7f0d1c966700 1 -- 172.21.15.67:0/1225872255 wait complete. 2024-04-12T05:29:55.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.579+0000 7f0d1c966700 1 Processor -- start 2024-04-12T05:29:55.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.579+0000 7f0d1c966700 1 -- start start 2024-04-12T05:29:55.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.579+0000 7f0d1c966700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d000061f0 0x7f0d18113390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:29:55.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.579+0000 7f0d1c966700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d181138d0 0x7f0d1810d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:29:55.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.579+0000 7f0d1c966700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d18113ee0 con 0x7f0d000061f0 2024-04-12T05:29:55.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.579+0000 7f0d1c966700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1810d830 con 0x7f0d181138d0 2024-04-12T05:29:55.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.579+0000 7f0d15d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d181138d0 0x7f0d1810d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:55.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.579+0000 7f0d15d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d181138d0 0x7f0d1810d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39504/0 (socket says 172.21.15.67:39504) 2024-04-12T05:29:55.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.579+0000 7f0d15d9b700 1 -- 172.21.15.67:0/1979273647 learned_addr learned my addr 172.21.15.67:0/1979273647 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-12T05:29:55.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.579+0000 7f0d1659c700 1 --2- 172.21.15.67:0/1979273647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d000061f0 0x7f0d18113390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:55.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.580+0000 7f0d15d9b700 1 -- 172.21.15.67:0/1979273647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d181138d0 msgr2=0x7f0d1810d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-12T05:29:55.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.580+0000 7f0d15d9b700 1 -- 172.21.15.67:0/1979273647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d181138d0 msgr2=0x7f0d1810d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-12T05:29:55.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.580+0000 7f0d15d9b700 1 --2- 172.21.15.67:0/1979273647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d181138d0 0x7f0d1810d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-12T05:29:55.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.580+0000 7f0d15d9b700 1 --2- 172.21.15.67:0/1979273647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d181138d0 0x7f0d1810d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-12T05:29:55.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.580+0000 7f0d1659c700 1 -- 172.21.15.67:0/1979273647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d181138d0 msgr2=0x7f0d1810d2f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T05:29:55.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.580+0000 7f0d1659c700 1 --2- 172.21.15.67:0/1979273647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d181138d0 0x7f0d1810d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:55.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.580+0000 7f0d1659c700 1 -- 172.21.15.67:0/1979273647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d08005040 con 0x7f0d000061f0 2024-04-12T05:29:55.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.580+0000 7f0d1659c700 1 --2- 172.21.15.67:0/1979273647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d000061f0 0x7f0d18113390 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0d18065770 tx=0x7f0d08002040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:55.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.580+0000 7f0d06ffd700 1 -- 172.21.15.67:0/1979273647 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d08004db0 con 0x7f0d000061f0 2024-04-12T05:29:55.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.580+0000 7f0d1c966700 1 -- 172.21.15.67:0/1979273647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d1810dab0 con 0x7f0d000061f0 2024-04-12T05:29:55.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.580+0000 7f0d06ffd700 1 -- 172.21.15.67:0/1979273647 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d08017410 con 0x7f0d000061f0 2024-04-12T05:29:55.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.580+0000 7f0d06ffd700 1 -- 172.21.15.67:0/1979273647 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d08022b30 con 0x7f0d000061f0 2024-04-12T05:29:55.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.580+0000 7f0d1c966700 1 -- 172.21.15.67:0/1979273647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d1810df90 con 0x7f0d000061f0 2024-04-12T05:29:55.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.581+0000 7f0d1c966700 1 -- 172.21.15.67:0/1979273647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d18107220 con 0x7f0d000061f0 2024-04-12T05:29:55.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.582+0000 7f0d06ffd700 1 -- 172.21.15.67:0/1979273647 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f0d08022c90 con 0x7f0d000061f0 2024-04-12T05:29:55.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.582+0000 7f0d06ffd700 1 --2- 172.21.15.67:0/1979273647 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f0cf8069320 0x7f0cf806b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:29:55.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.582+0000 7f0d06ffd700 1 -- 172.21.15.67:0/1979273647 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f0d08088120 con 0x7f0d000061f0 2024-04-12T05:29:55.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.582+0000 7f0d15d9b700 1 --2- 172.21.15.67:0/1979273647 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f0cf8069320 0x7f0cf806b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:55.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.583+0000 7f0d15d9b700 1 --2- 172.21.15.67:0/1979273647 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f0cf8069320 0x7f0cf806b7d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c006fd0 tx=0x7f0d0c008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:55.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.588+0000 7f0d06ffd700 1 -- 172.21.15.67:0/1979273647 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f0d08058ea0 con 0x7f0d000061f0 2024-04-12T05:29:55.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:55 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:55.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:55 smithi032 ceph-mon[39731]: mgrmap e18: smithi032.hrwxrh(active, since 29s), standbys: smithi067.dwichg 2024-04-12T05:29:55.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:55 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr metadata", "who": "smithi067.dwichg", "id": "smithi067.dwichg"}]: dispatch 2024-04-12T05:29:55.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:55 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:29:55.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:55 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-12T05:29:55.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.866+0000 7f0d1c966700 1 -- 172.21.15.67:0/1979273647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0d1805f5f0 con 0x7f0d000061f0 2024-04-12T05:29:55.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.867+0000 7f0d06ffd700 1 -- 172.21.15.67:0/1979273647 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1013 (secure 0 0 0) 0x7f0d08009a10 con 0x7f0d000061f0 2024-04-12T05:29:55.869 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-12T05:29:55.869 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":2,"fsid":"37b58ce2-f88d-11ee-bc8c-c7b262605968","modified":"2024-04-12T05:29:48.992958Z","created":"2024-04-12T05:27:04.825332Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi032","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-12T05:29:55.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.869+0000 7f0d1c966700 1 -- 172.21.15.67:0/1979273647 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f0cf8069320 msgr2=0x7f0cf806b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:55.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.869+0000 7f0d1c966700 1 --2- 172.21.15.67:0/1979273647 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f0cf8069320 0x7f0cf806b7d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c006fd0 tx=0x7f0d0c008040 comp rx=0 tx=0).stop 2024-04-12T05:29:55.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.869+0000 7f0d1c966700 1 -- 172.21.15.67:0/1979273647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d000061f0 msgr2=0x7f0d18113390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:55.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.869+0000 7f0d1c966700 1 --2- 172.21.15.67:0/1979273647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d000061f0 0x7f0d18113390 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0d18065770 tx=0x7f0d08002040 comp rx=0 tx=0).stop 2024-04-12T05:29:55.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.869+0000 7f0d1c966700 1 -- 172.21.15.67:0/1979273647 shutdown_connections 2024-04-12T05:29:55.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.869+0000 7f0d1c966700 1 --2- 172.21.15.67:0/1979273647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d000061f0 0x7f0d18113390 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:55.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.869+0000 7f0d1c966700 1 --2- 172.21.15.67:0/1979273647 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f0cf8069320 0x7f0cf806b7d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:55.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.869+0000 7f0d1c966700 1 --2- 172.21.15.67:0/1979273647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d181138d0 0x7f0d1810d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:55.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.869+0000 7f0d1c966700 1 -- 172.21.15.67:0/1979273647 >> 172.21.15.67:0/1979273647 conn(0x7f0d18071960 msgr2=0x7f0d181023b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:55.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.869+0000 7f0d1c966700 1 -- 172.21.15.67:0/1979273647 shutdown_connections 2024-04-12T05:29:55.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:29:55.869+0000 7f0d1c966700 1 -- 172.21.15.67:0/1979273647 wait complete. 2024-04-12T05:29:55.871 INFO:teuthology.orchestra.run.smithi067.stderr:dumped monmap epoch 2 2024-04-12T05:29:56.463 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:56 smithi067 ceph-mon[49010]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:56.463 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:56 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:29:56.463 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:56 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:56.463 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:56 smithi067 ceph-mon[49010]: from='client.? 172.21.15.67:0/1979273647' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:29:56.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:56 smithi032 ceph-mon[39731]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:56.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:56 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:29:56.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:56 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:56.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:56 smithi032 ceph-mon[39731]: from='client.? 172.21.15.67:0/1979273647' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:29:56.788 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-12T05:29:56.788 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph config generate-minimal-conf 2024-04-12T05:29:57.103 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:29:58.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:57 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:58.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:57 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:58.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:57 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:29:58.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:57 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:29:58.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:57 smithi067 ceph-mon[49010]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-12T05:29:58.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:57 smithi067 ceph-mon[49010]: Updating smithi067:/etc/ceph/ceph.conf 2024-04-12T05:29:58.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:57 smithi067 ceph-mon[49010]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:58.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:57 smithi067 ceph-mon[49010]: Updating smithi032:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:29:58.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:57 smithi067 ceph-mon[49010]: Updating smithi067:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:29:58.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:57 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:58.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:57 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:58.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:57 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:58.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:57 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:58.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:57 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:58.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:57 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T05:29:58.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:57 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T05:29:58.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:57 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:29:58.318 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:57 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:58.356 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:57 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:58.356 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:57 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:29:58.356 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:57 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:29:58.356 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:57 smithi032 ceph-mon[39731]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-12T05:29:58.356 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:57 smithi032 ceph-mon[39731]: Updating smithi067:/etc/ceph/ceph.conf 2024-04-12T05:29:58.356 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:57 smithi032 ceph-mon[39731]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:29:58.356 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:57 smithi032 ceph-mon[39731]: Updating smithi032:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:29:58.357 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:57 smithi032 ceph-mon[39731]: Updating smithi067:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:29:58.357 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:57 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:58.357 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:57 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:58.357 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:57 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:58.357 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:57 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:58.357 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:57 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:29:58.357 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:57 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T05:29:58.357 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:57 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T05:29:58.357 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:57 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:29:59.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.052+0000 7f79ae586700 1 -- 172.21.15.32:0/1867252158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79a81018f0 msgr2=0x7f79a8101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:59.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.052+0000 7f79ae586700 1 --2- 172.21.15.32:0/1867252158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79a81018f0 0x7f79a8101cc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f79980048c0 tx=0x7f7998009e30 comp rx=0 tx=0).stop 2024-04-12T05:29:59.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.053+0000 7f79ae586700 1 -- 172.21.15.32:0/1867252158 shutdown_connections 2024-04-12T05:29:59.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.053+0000 7f79ae586700 1 --2- 172.21.15.32:0/1867252158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79a81018f0 0x7f79a8101cc0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:59.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.053+0000 7f79ae586700 1 -- 172.21.15.32:0/1867252158 >> 172.21.15.32:0/1867252158 conn(0x7f79a80f6c30 msgr2=0x7f79a80f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:59.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.053+0000 7f79ae586700 1 -- 172.21.15.32:0/1867252158 shutdown_connections 2024-04-12T05:29:59.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.053+0000 7f79ae586700 1 -- 172.21.15.32:0/1867252158 wait complete. 2024-04-12T05:29:59.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.053+0000 7f79ae586700 1 Processor -- start 2024-04-12T05:29:59.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.053+0000 7f79ae586700 1 -- start start 2024-04-12T05:29:59.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.053+0000 7f79ae586700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79a81018f0 0x7f79a810e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:29:59.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.053+0000 7f79ae586700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79a810e640 0x7f79a810eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:29:59.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.053+0000 7f79ae586700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a8117200 con 0x7f79a81018f0 2024-04-12T05:29:59.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.053+0000 7f79ae586700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a8117340 con 0x7f79a810e640 2024-04-12T05:29:59.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.054+0000 7f79a7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79a81018f0 0x7f79a810e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:59.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.054+0000 7f79a7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79a81018f0 0x7f79a810e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35178/0 (socket says 172.21.15.32:35178) 2024-04-12T05:29:59.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.054+0000 7f79a7fff700 1 -- 172.21.15.32:0/4195647144 learned_addr learned my addr 172.21.15.32:0/4195647144 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:29:59.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.054+0000 7f79a7fff700 1 -- 172.21.15.32:0/4195647144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79a810e640 msgr2=0x7f79a810eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:59.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.054+0000 7f79a7fff700 1 --2- 172.21.15.32:0/4195647144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79a810e640 0x7f79a810eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:59.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.054+0000 7f79a7fff700 1 -- 172.21.15.32:0/4195647144 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7998005040 con 0x7f79a81018f0 2024-04-12T05:29:59.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.054+0000 7f79a7fff700 1 --2- 172.21.15.32:0/4195647144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79a81018f0 0x7f79a810e100 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7998000c00 tx=0x7f7998009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:59.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.054+0000 7f79a57fa700 1 -- 172.21.15.32:0/4195647144 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f799800e040 con 0x7f79a81018f0 2024-04-12T05:29:59.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.054+0000 7f79a57fa700 1 -- 172.21.15.32:0/4195647144 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7998013420 con 0x7f79a81018f0 2024-04-12T05:29:59.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.054+0000 7f79ae586700 1 -- 172.21.15.32:0/4195647144 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79a8117540 con 0x7f79a81018f0 2024-04-12T05:29:59.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.054+0000 7f79a57fa700 1 -- 172.21.15.32:0/4195647144 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f799801c630 con 0x7f79a81018f0 2024-04-12T05:29:59.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.054+0000 7f79ae586700 1 -- 172.21.15.32:0/4195647144 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79a8117b00 con 0x7f79a81018f0 2024-04-12T05:29:59.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.055+0000 7f79ae586700 1 -- 172.21.15.32:0/4195647144 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79a8100ca0 con 0x7f79a81018f0 2024-04-12T05:29:59.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.056+0000 7f79a57fa700 1 -- 172.21.15.32:0/4195647144 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f7998013590 con 0x7f79a81018f0 2024-04-12T05:29:59.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.056+0000 7f79a57fa700 1 --2- 172.21.15.32:0/4195647144 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f7994069370 0x7f799406b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:29:59.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.056+0000 7f79a57fa700 1 -- 172.21.15.32:0/4195647144 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f7998087300 con 0x7f79a81018f0 2024-04-12T05:29:59.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.056+0000 7f799ffff700 1 --2- 172.21.15.32:0/4195647144 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f7994069370 0x7f799406b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:59.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.057+0000 7f799ffff700 1 --2- 172.21.15.32:0/4195647144 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f7994069370 0x7f799406b820 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7990009930 tx=0x7f7990006de0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:59.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.061+0000 7f79a57fa700 1 -- 172.21.15.32:0/4195647144 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f7998054860 con 0x7f79a81018f0 2024-04-12T05:29:59.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.258+0000 7f79ae586700 1 -- 172.21.15.32:0/4195647144 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f79a80fd210 con 0x7f79a81018f0 2024-04-12T05:29:59.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.258+0000 7f79a57fa700 1 -- 172.21.15.32:0/4195647144 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+223 (secure 0 0 0) 0x7f7998057e80 con 0x7f79a81018f0 2024-04-12T05:29:59.260 INFO:teuthology.orchestra.run.smithi032.stdout:# minimal ceph.conf for 37b58ce2-f88d-11ee-bc8c-c7b262605968 2024-04-12T05:29:59.260 INFO:teuthology.orchestra.run.smithi032.stdout:[global] 2024-04-12T05:29:59.260 INFO:teuthology.orchestra.run.smithi032.stdout: fsid = 37b58ce2-f88d-11ee-bc8c-c7b262605968 2024-04-12T05:29:59.260 INFO:teuthology.orchestra.run.smithi032.stdout: mon_host = [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-04-12T05:29:59.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.260+0000 7f79ae586700 1 -- 172.21.15.32:0/4195647144 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f7994069370 msgr2=0x7f799406b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:59.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.260+0000 7f79ae586700 1 --2- 172.21.15.32:0/4195647144 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f7994069370 0x7f799406b820 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7990009930 tx=0x7f7990006de0 comp rx=0 tx=0).stop 2024-04-12T05:29:59.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.260+0000 7f79ae586700 1 -- 172.21.15.32:0/4195647144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79a81018f0 msgr2=0x7f79a810e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:59.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.260+0000 7f79ae586700 1 --2- 172.21.15.32:0/4195647144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79a81018f0 0x7f79a810e100 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7998000c00 tx=0x7f7998009e30 comp rx=0 tx=0).stop 2024-04-12T05:29:59.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.261+0000 7f79ae586700 1 -- 172.21.15.32:0/4195647144 shutdown_connections 2024-04-12T05:29:59.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.261+0000 7f79ae586700 1 --2- 172.21.15.32:0/4195647144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79a81018f0 0x7f79a810e100 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:59.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.261+0000 7f79ae586700 1 --2- 172.21.15.32:0/4195647144 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f7994069370 0x7f799406b820 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:59.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.261+0000 7f79ae586700 1 --2- 172.21.15.32:0/4195647144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79a810e640 0x7f79a810eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:29:59.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.261+0000 7f79ae586700 1 -- 172.21.15.32:0/4195647144 >> 172.21.15.32:0/4195647144 conn(0x7f79a80f6c30 msgr2=0x7f79a80f8880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:59.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.261+0000 7f79ae586700 1 -- 172.21.15.32:0/4195647144 shutdown_connections 2024-04-12T05:29:59.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:29:59.261+0000 7f79ae586700 1 -- 172.21.15.32:0/4195647144 wait complete. 2024-04-12T05:29:59.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:58 smithi067 ceph-mon[49010]: Reconfiguring mon.smithi032 (unknown last config time)... 2024-04-12T05:29:59.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:58 smithi067 ceph-mon[49010]: Reconfiguring daemon mon.smithi032 on smithi032 2024-04-12T05:29:59.283 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:58 smithi032 ceph-mon[39731]: Reconfiguring mon.smithi032 (unknown last config time)... 2024-04-12T05:29:59.283 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:58 smithi032 ceph-mon[39731]: Reconfiguring daemon mon.smithi032 on smithi032 2024-04-12T05:30:00.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:59 smithi032 ceph-mon[39731]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:00.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:29:59 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/4195647144' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:30:00.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:59 smithi067 ceph-mon[49010]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:00.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:29:59 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/4195647144' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:30:00.395 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-12T05:30:00.395 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-12T05:30:00.395 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.conf 2024-04-12T05:30:00.428 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-12T05:30:00.428 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:30:00.496 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-12T05:30:00.496 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.conf 2024-04-12T05:30:00.537 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-12T05:30:00.537 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:30:00.611 INFO:tasks.cephadm:Deploying OSDs... 2024-04-12T05:30:00.611 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-12T05:30:00.611 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/scratch_devs of=/dev/stdout 2024-04-12T05:30:00.630 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-12T05:30:00.630 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_1 2024-04-12T05:30:00.689 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-12T05:30:00.689 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T05:30:00.690 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 6h/6d Inode: 160692 Links: 1 2024-04-12T05:30:00.690 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T05:30:00.690 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T05:30:00.690 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-04-12 05:29:13.038763210 +0000 2024-04-12T05:30:00.690 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-12 05:20:34.978404802 +0000 2024-04-12T05:30:00.690 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-12 05:20:34.978404802 +0000 2024-04-12T05:30:00.690 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2024-04-12T05:30:00.690 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-12T05:30:00.758 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-04-12T05:30:00.758 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-04-12T05:30:00.758 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 9.7625e-05 s, 5.2 MB/s 2024-04-12T05:30:00.759 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-12T05:30:00.820 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_2 2024-04-12T05:30:00.878 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-12T05:30:00.878 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T05:30:00.878 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 6h/6d Inode: 166474 Links: 1 2024-04-12T05:30:00.878 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T05:30:00.878 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T05:30:00.878 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-04-12 05:29:13.038763210 +0000 2024-04-12T05:30:00.878 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-12 05:20:35.312398074 +0000 2024-04-12T05:30:00.878 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-12 05:20:35.312398074 +0000 2024-04-12T05:30:00.878 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2024-04-12T05:30:00.879 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-12T05:30:00.945 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-04-12T05:30:00.945 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-04-12T05:30:00.945 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 9.4325e-05 s, 5.4 MB/s 2024-04-12T05:30:00.946 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-12T05:30:00.981 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:00 smithi067 ceph-mon[49010]: overall HEALTH_OK 2024-04-12T05:30:01.006 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_3 2024-04-12T05:30:01.064 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-12T05:30:01.064 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T05:30:01.064 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 6h/6d Inode: 167549 Links: 1 2024-04-12T05:30:01.064 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T05:30:01.064 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T05:30:01.064 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-04-12 05:29:13.038763210 +0000 2024-04-12T05:30:01.064 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-12 05:20:35.647391325 +0000 2024-04-12T05:30:01.064 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-12 05:20:35.647391325 +0000 2024-04-12T05:30:01.064 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2024-04-12T05:30:01.065 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-12T05:30:01.130 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-04-12T05:30:01.130 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-04-12T05:30:01.130 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 8.245e-05 s, 6.2 MB/s 2024-04-12T05:30:01.131 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-12T05:30:01.190 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_4 2024-04-12T05:30:01.248 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-12T05:30:01.248 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T05:30:01.248 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 6h/6d Inode: 168441 Links: 1 2024-04-12T05:30:01.248 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T05:30:01.248 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T05:30:01.248 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-04-12 05:29:13.038763210 +0000 2024-04-12T05:30:01.248 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-12 05:20:36.002384174 +0000 2024-04-12T05:30:01.249 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-12 05:20:36.002384174 +0000 2024-04-12T05:30:01.249 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2024-04-12T05:30:01.249 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-12T05:30:01.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:00 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:01.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:00 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:01.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:00 smithi067 ceph-mon[49010]: Reconfiguring mgr.smithi032.hrwxrh (unknown last config time)... 2024-04-12T05:30:01.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:00 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi032.hrwxrh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:30:01.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:00 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:30:01.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:00 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:30:01.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:00 smithi067 ceph-mon[49010]: Reconfiguring daemon mgr.smithi032.hrwxrh on smithi032 2024-04-12T05:30:01.314 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:00 smithi032 ceph-mon[39731]: overall HEALTH_OK 2024-04-12T05:30:01.314 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:00 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:01.314 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:00 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:01.314 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:00 smithi032 ceph-mon[39731]: Reconfiguring mgr.smithi032.hrwxrh (unknown last config time)... 2024-04-12T05:30:01.314 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:00 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi032.hrwxrh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:30:01.314 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:00 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:30:01.314 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:00 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:30:01.314 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:00 smithi032 ceph-mon[39731]: Reconfiguring daemon mgr.smithi032.hrwxrh on smithi032 2024-04-12T05:30:01.316 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-04-12T05:30:01.316 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-04-12T05:30:01.316 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 9.7679e-05 s, 5.2 MB/s 2024-04-12T05:30:01.317 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-12T05:30:01.376 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-12T05:30:01.376 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/scratch_devs of=/dev/stdout 2024-04-12T05:30:01.398 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-12T05:30:01.399 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_1 2024-04-12T05:30:01.461 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-12T05:30:01.461 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T05:30:01.461 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 6h/6d Inode: 166417 Links: 1 2024-04-12T05:30:01.461 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T05:30:01.461 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T05:30:01.461 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-04-12 05:29:47.746483683 +0000 2024-04-12T05:30:01.461 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-12 05:20:26.268353129 +0000 2024-04-12T05:30:01.461 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-12 05:20:26.268353129 +0000 2024-04-12T05:30:01.462 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2024-04-12T05:30:01.462 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-12T05:30:01.531 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-12T05:30:01.531 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-12T05:30:01.531 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.00024956 s, 2.1 MB/s 2024-04-12T05:30:01.532 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-12T05:30:01.593 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_2 2024-04-12T05:30:01.653 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-12T05:30:01.653 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T05:30:01.653 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 6h/6d Inode: 166500 Links: 1 2024-04-12T05:30:01.653 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T05:30:01.653 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T05:30:01.653 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-04-12 05:29:47.746483683 +0000 2024-04-12T05:30:01.653 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-12 05:20:26.613345686 +0000 2024-04-12T05:30:01.653 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-12 05:20:26.613345686 +0000 2024-04-12T05:30:01.653 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2024-04-12T05:30:01.653 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-12T05:30:01.721 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-12T05:30:01.721 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-12T05:30:01.721 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000211077 s, 2.4 MB/s 2024-04-12T05:30:01.722 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-12T05:30:01.783 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_3 2024-04-12T05:30:01.843 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-12T05:30:01.843 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T05:30:01.843 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 6h/6d Inode: 170339 Links: 1 2024-04-12T05:30:01.843 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T05:30:01.843 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T05:30:01.843 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-04-12 05:29:47.746483683 +0000 2024-04-12T05:30:01.843 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-12 05:20:26.955338308 +0000 2024-04-12T05:30:01.843 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-12 05:20:26.955338308 +0000 2024-04-12T05:30:01.843 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2024-04-12T05:30:01.844 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-12T05:30:01.910 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-12T05:30:01.910 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-12T05:30:01.910 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000176231 s, 2.9 MB/s 2024-04-12T05:30:01.911 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-12T05:30:01.971 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_4 2024-04-12T05:30:02.030 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-12T05:30:02.030 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T05:30:02.030 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 6h/6d Inode: 167386 Links: 1 2024-04-12T05:30:02.030 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T05:30:02.030 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T05:30:02.031 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-04-12 05:29:47.746483683 +0000 2024-04-12T05:30:02.031 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-12 05:20:27.286331166 +0000 2024-04-12T05:30:02.031 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-12 05:20:27.286331166 +0000 2024-04-12T05:30:02.031 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2024-04-12T05:30:02.031 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-12T05:30:02.096 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-12T05:30:02.097 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-12T05:30:02.097 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000230054 s, 2.2 MB/s 2024-04-12T05:30:02.098 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-12T05:30:02.157 INFO:tasks.cephadm:Deploying osd.0 on smithi032 with /dev/vg_nvme/lv_4... 2024-04-12T05:30:02.157 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-12T05:30:02.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:01 smithi067 ceph-mon[49010]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:02.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:01 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:02.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:01 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:02.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:01 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi032", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-12T05:30:02.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:01 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:30:02.273 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:01 smithi032 ceph-mon[39731]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:02.273 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:01 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:02.273 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:01 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:02.273 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:01 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi032", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-12T05:30:02.273 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:01 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:30:02.828 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:30:03.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:02 smithi067 ceph-mon[49010]: Reconfiguring crash.smithi032 (monmap changed)... 2024-04-12T05:30:03.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:02 smithi067 ceph-mon[49010]: Reconfiguring daemon crash.smithi032 on smithi032 2024-04-12T05:30:03.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:02 smithi032 ceph-mon[39731]: Reconfiguring crash.smithi032 (monmap changed)... 2024-04-12T05:30:03.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:02 smithi032 ceph-mon[39731]: Reconfiguring daemon crash.smithi032 on smithi032 2024-04-12T05:30:04.222 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:03 smithi032 ceph-mon[39731]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:04.222 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:03 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:04.222 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:03 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:04.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:03 smithi067 ceph-mon[49010]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:04.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:03 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:04.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:03 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:05.274 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:04 smithi067 ceph-mon[49010]: Reconfiguring alertmanager.smithi032 (dependencies changed)... 2024-04-12T05:30:05.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:04 smithi067 ceph-mon[49010]: Reconfiguring daemon alertmanager.smithi032 on smithi032 2024-04-12T05:30:05.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:04 smithi032 ceph-mon[39731]: Reconfiguring alertmanager.smithi032 (dependencies changed)... 2024-04-12T05:30:05.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:04 smithi032 ceph-mon[39731]: Reconfiguring daemon alertmanager.smithi032 on smithi032 2024-04-12T05:30:05.618 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-12T05:30:05.618 INFO:teuthology.orchestra.run.smithi032.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-12T05:30:05.618 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10+0 records in 2024-04-12T05:30:05.618 INFO:teuthology.orchestra.run.smithi032.stderr:10+0 records out 2024-04-12T05:30:05.618 INFO:teuthology.orchestra.run.smithi032.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00804537 s, 1.3 GB/s 2024-04-12T05:30:05.618 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping successful for: 2024-04-12T05:30:06.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:05 smithi067 ceph-mon[49010]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:06.328 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:05 smithi032 ceph-mon[39731]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:06.351 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph orch daemon add osd smithi032:vg_nvme/lv_4 2024-04-12T05:30:07.651 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:30:08.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:08 smithi032 ceph-mon[39731]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:08.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:08 smithi067 ceph-mon[49010]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:10.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.303+0000 7f68bbded700 1 -- 172.21.15.32:0/557382137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68b40fcf70 msgr2=0x7f68b40fd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.303+0000 7f68bbded700 1 --2- 172.21.15.32:0/557382137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68b40fcf70 0x7f68b40fd3e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f68b0004940 tx=0x7f68b0009e30 comp rx=0 tx=0).stop 2024-04-12T05:30:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.303+0000 7f68bbded700 1 -- 172.21.15.32:0/557382137 shutdown_connections 2024-04-12T05:30:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.303+0000 7f68bbded700 1 --2- 172.21.15.32:0/557382137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68b40fcf70 0x7f68b40fd3e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:30:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.303+0000 7f68bbded700 1 --2- 172.21.15.32:0/557382137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68b40fbe00 0x7f68b40fc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:30:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.303+0000 7f68bbded700 1 -- 172.21.15.32:0/557382137 >> 172.21.15.32:0/557382137 conn(0x7f68b40f74e0 msgr2=0x7f68b40f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.303+0000 7f68bbded700 1 -- 172.21.15.32:0/557382137 shutdown_connections 2024-04-12T05:30:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.303+0000 7f68bbded700 1 -- 172.21.15.32:0/557382137 wait complete. 2024-04-12T05:30:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.303+0000 7f68bbded700 1 Processor -- start 2024-04-12T05:30:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.304+0000 7f68bbded700 1 -- start start 2024-04-12T05:30:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.304+0000 7f68bbded700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68b40fbe00 0x7f68b410fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:30:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.304+0000 7f68bbded700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68b40fcf70 0x7f68b41100a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:30:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.304+0000 7f68bbded700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b41106a0 con 0x7f68b40fcf70 2024-04-12T05:30:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.304+0000 7f68bbded700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b4110ef0 con 0x7f68b40fbe00 2024-04-12T05:30:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.304+0000 7f68b9388700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68b40fcf70 0x7f68b41100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.304+0000 7f68b9388700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68b40fcf70 0x7f68b41100a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50462/0 (socket says 172.21.15.32:50462) 2024-04-12T05:30:10.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.304+0000 7f68b9388700 1 -- 172.21.15.32:0/3511493624 learned_addr learned my addr 172.21.15.32:0/3511493624 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:30:10.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.304+0000 7f68b9b89700 1 --2- 172.21.15.32:0/3511493624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68b40fbe00 0x7f68b410fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:10.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.304+0000 7f68b9388700 1 -- 172.21.15.32:0/3511493624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68b40fbe00 msgr2=0x7f68b410fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:10.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.304+0000 7f68b9388700 1 --2- 172.21.15.32:0/3511493624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68b40fbe00 0x7f68b410fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:30:10.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.304+0000 7f68b9388700 1 -- 172.21.15.32:0/3511493624 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b0005040 con 0x7f68b40fcf70 2024-04-12T05:30:10.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.304+0000 7f68b9388700 1 --2- 172.21.15.32:0/3511493624 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68b40fcf70 0x7f68b41100a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f68b000b3b0 tx=0x7f68b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:10.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.304+0000 7f68aa7fc700 1 -- 172.21.15.32:0/3511493624 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68b000e040 con 0x7f68b40fcf70 2024-04-12T05:30:10.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.304+0000 7f68aa7fc700 1 -- 172.21.15.32:0/3511493624 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f68b000bc50 con 0x7f68b40fcf70 2024-04-12T05:30:10.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.304+0000 7f68bbded700 1 -- 172.21.15.32:0/3511493624 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68b4111170 con 0x7f68b40fcf70 2024-04-12T05:30:10.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.305+0000 7f68aa7fc700 1 -- 172.21.15.32:0/3511493624 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68b001c630 con 0x7f68b40fcf70 2024-04-12T05:30:10.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.305+0000 7f68bbded700 1 -- 172.21.15.32:0/3511493624 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b41116b0 con 0x7f68b40fcf70 2024-04-12T05:30:10.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.305+0000 7f68bbded700 1 -- 172.21.15.32:0/3511493624 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68b4101900 con 0x7f68b40fcf70 2024-04-12T05:30:10.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.306+0000 7f68aa7fc700 1 -- 172.21.15.32:0/3511493624 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f68b0013420 con 0x7f68b40fcf70 2024-04-12T05:30:10.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.306+0000 7f68aa7fc700 1 --2- 172.21.15.32:0/3511493624 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f68a0069210 0x7f68a006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:30:10.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.306+0000 7f68aa7fc700 1 -- 172.21.15.32:0/3511493624 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f68b0086cb0 con 0x7f68b40fcf70 2024-04-12T05:30:10.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.306+0000 7f68b9b89700 1 --2- 172.21.15.32:0/3511493624 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f68a0069210 0x7f68a006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:10.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.307+0000 7f68b9b89700 1 --2- 172.21.15.32:0/3511493624 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f68a0069210 0x7f68a006b6c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f68a4004f40 tx=0x7f68a400b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:10.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.311+0000 7f68aa7fc700 1 -- 172.21.15.32:0/3511493624 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f68b0057bf0 con 0x7f68b40fcf70 2024-04-12T05:30:10.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:10 smithi032 ceph-mon[39731]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:10.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:10.516+0000 7f68bbded700 1 -- 172.21.15.32:0/3511493624 --> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f68b4111920 con 0x7f68a0069210 2024-04-12T05:30:10.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:10 smithi067 ceph-mon[49010]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:11.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:11 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:30:11.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:11 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T05:30:11.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:11 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T05:30:11.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:11 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:30:11.583 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:11 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:30:11.583 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:11 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T05:30:11.584 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:11 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T05:30:11.584 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:11 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:30:12.438 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:12 smithi032 ceph-mon[39731]: from='client.14240 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:30:12.439 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:12 smithi032 ceph-mon[39731]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:12.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:12 smithi067 ceph-mon[49010]: from='client.14240 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:30:12.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:12 smithi067 ceph-mon[49010]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:14.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:13 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:14.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:13 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:14.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:13 smithi067 ceph-mon[49010]: Reconfiguring grafana.smithi032 (dependencies changed)... 2024-04-12T05:30:14.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:13 smithi067 ceph-mon[49010]: Reconfiguring daemon grafana.smithi032 on smithi032 2024-04-12T05:30:14.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:13 smithi067 ceph-mon[49010]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:14.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:13 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:14.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:13 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:14.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:13 smithi032 ceph-mon[39731]: Reconfiguring grafana.smithi032 (dependencies changed)... 2024-04-12T05:30:14.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:13 smithi032 ceph-mon[39731]: Reconfiguring daemon grafana.smithi032 on smithi032 2024-04-12T05:30:14.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:13 smithi032 ceph-mon[39731]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:15.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:14 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/756726679' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1f69d2a-5e37-47b8-b68f-0568af894b91"}]: dispatch 2024-04-12T05:30:15.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:14 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/756726679' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1f69d2a-5e37-47b8-b68f-0568af894b91"}]': finished 2024-04-12T05:30:15.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:14 smithi067 ceph-mon[49010]: osdmap e6: 1 total, 0 up, 1 in 2024-04-12T05:30:15.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:14 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:30:15.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:14 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/171410137' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T05:30:15.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:14 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/756726679' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1f69d2a-5e37-47b8-b68f-0568af894b91"}]: dispatch 2024-04-12T05:30:15.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:14 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/756726679' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1f69d2a-5e37-47b8-b68f-0568af894b91"}]': finished 2024-04-12T05:30:15.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:14 smithi032 ceph-mon[39731]: osdmap e6: 1 total, 0 up, 1 in 2024-04-12T05:30:15.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:14 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:30:15.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:14 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/171410137' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T05:30:16.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:15 smithi067 ceph-mon[49010]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:16.092 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:15 smithi032 ceph-mon[39731]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:18.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:18 smithi032 ceph-mon[39731]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:18.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:18 smithi067 ceph-mon[49010]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:20.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:20 smithi067 ceph-mon[49010]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:20.557 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:20 smithi032 ceph-mon[39731]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:22.381 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:22 smithi032 ceph-mon[39731]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:22.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:22 smithi067 ceph-mon[49010]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:24.031 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:23 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:24.031 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:23 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:24.032 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:23 smithi032 ceph-mon[39731]: Reconfiguring prometheus.smithi032 (dependencies changed)... 2024-04-12T05:30:24.032 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:23 smithi032 ceph-mon[39731]: Reconfiguring daemon prometheus.smithi032 on smithi032 2024-04-12T05:30:24.032 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:23 smithi032 ceph-mon[39731]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:24.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:23 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:24.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:23 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:24.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:23 smithi067 ceph-mon[49010]: Reconfiguring prometheus.smithi032 (dependencies changed)... 2024-04-12T05:30:24.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:23 smithi067 ceph-mon[49010]: Reconfiguring daemon prometheus.smithi032 on smithi032 2024-04-12T05:30:24.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:23 smithi067 ceph-mon[49010]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:25.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:24 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-12T05:30:25.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:24 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:30:25.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:24 smithi032 ceph-mon[39731]: Deploying daemon osd.0 on smithi032 2024-04-12T05:30:25.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:24 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-12T05:30:25.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:24 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:30:25.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:24 smithi067 ceph-mon[49010]: Deploying daemon osd.0 on smithi032 2024-04-12T05:30:26.221 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:25 smithi032 ceph-mon[39731]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:26.221 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:25 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:30:26.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:25 smithi067 ceph-mon[49010]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:26.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:25 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:30:28.489 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:28 smithi032 ceph-mon[39731]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:28.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:28 smithi067 ceph-mon[49010]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:29.960 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:29 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:29.960 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:29 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:29.960 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:29 smithi032 ceph-mon[39731]: Reconfiguring crash.smithi067 (monmap changed)... 2024-04-12T05:30:29.960 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:29 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi067", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-12T05:30:29.960 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:29 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:30:29.960 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:29 smithi032 ceph-mon[39731]: Reconfiguring daemon crash.smithi067 on smithi067 2024-04-12T05:30:29.960 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:29 smithi032 ceph-mon[39731]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:30.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:29 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:30.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:29 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:30.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:29 smithi067 ceph-mon[49010]: Reconfiguring crash.smithi067 (monmap changed)... 2024-04-12T05:30:30.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:29 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi067", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-12T05:30:30.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:29 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:30:30.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:29 smithi067 ceph-mon[49010]: Reconfiguring daemon crash.smithi067 on smithi067 2024-04-12T05:30:30.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:29 smithi067 ceph-mon[49010]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:31.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:31 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:31.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:31 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:31.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:31 smithi032 ceph-mon[39731]: Reconfiguring mgr.smithi067.dwichg (monmap changed)... 2024-04-12T05:30:31.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:31 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi067.dwichg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:30:31.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:31 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:30:31.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:31 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:30:31.344 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:31 smithi032 ceph-mon[39731]: Reconfiguring daemon mgr.smithi067.dwichg on smithi067 2024-04-12T05:30:31.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:31 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:31.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:31 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:31.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:31 smithi067 ceph-mon[49010]: Reconfiguring mgr.smithi067.dwichg (monmap changed)... 2024-04-12T05:30:31.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:31 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi067.dwichg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:30:31.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:31 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:30:31.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:31 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:30:31.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:31 smithi067 ceph-mon[49010]: Reconfiguring daemon mgr.smithi067.dwichg on smithi067 2024-04-12T05:30:32.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:32 smithi032 ceph-mon[39731]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:32.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:32 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:32.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:32 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:32.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:32 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T05:30:32.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:32 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T05:30:32.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:32 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:30:32.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:32 smithi067 ceph-mon[49010]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:32.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:32 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:32.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:32 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:32.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:32 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T05:30:32.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:32 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T05:30:32.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:32 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:30:33.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:33 smithi032 ceph-mon[39731]: Reconfiguring mon.smithi067 (monmap changed)... 2024-04-12T05:30:33.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:33 smithi032 ceph-mon[39731]: Reconfiguring daemon mon.smithi067 on smithi067 2024-04-12T05:30:33.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:33 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:33.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:33 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:33.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:33 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T05:30:33.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:33 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi032.front.sepia.ceph.com:9093"}]: dispatch 2024-04-12T05:30:33.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:33 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:33.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:33 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T05:30:33.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:33 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi032.front.sepia.ceph.com:3000"}]: dispatch 2024-04-12T05:30:33.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:33 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:33.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:33 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:30:33.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:33 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi032.front.sepia.ceph.com:9095"}]: dispatch 2024-04-12T05:30:33.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:33 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:33.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:33 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:30:33.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:33 smithi067 ceph-mon[49010]: Reconfiguring mon.smithi067 (monmap changed)... 2024-04-12T05:30:33.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:33 smithi067 ceph-mon[49010]: Reconfiguring daemon mon.smithi067 on smithi067 2024-04-12T05:30:33.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:33 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:33.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:33 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:33.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:33 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T05:30:33.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:33 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi032.front.sepia.ceph.com:9093"}]: dispatch 2024-04-12T05:30:33.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:33 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:33.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:33 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T05:30:33.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:33 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi032.front.sepia.ceph.com:3000"}]: dispatch 2024-04-12T05:30:33.523 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:33 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:33.523 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:33 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:30:33.523 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:33 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi032.front.sepia.ceph.com:9095"}]: dispatch 2024-04-12T05:30:33.523 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:33 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:33.523 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:33 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:30:34.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:34 smithi067 ceph-mon[49010]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:34.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:34 smithi067 ceph-mon[49010]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T05:30:34.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:34 smithi067 ceph-mon[49010]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi032.front.sepia.ceph.com:9093"}]: dispatch 2024-04-12T05:30:34.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:34 smithi067 ceph-mon[49010]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T05:30:34.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:34 smithi067 ceph-mon[49010]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi032.front.sepia.ceph.com:3000"}]: dispatch 2024-04-12T05:30:34.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:34 smithi067 ceph-mon[49010]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:30:34.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:34 smithi067 ceph-mon[49010]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi032.front.sepia.ceph.com:9095"}]: dispatch 2024-04-12T05:30:34.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:34 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:34.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:34 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:34.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:34 smithi032 ceph-mon[39731]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:34.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:34 smithi032 ceph-mon[39731]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T05:30:34.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:34 smithi032 ceph-mon[39731]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi032.front.sepia.ceph.com:9093"}]: dispatch 2024-04-12T05:30:34.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:34 smithi032 ceph-mon[39731]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T05:30:34.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:34 smithi032 ceph-mon[39731]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi032.front.sepia.ceph.com:3000"}]: dispatch 2024-04-12T05:30:34.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:34 smithi032 ceph-mon[39731]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:30:34.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:34 smithi032 ceph-mon[39731]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi032.front.sepia.ceph.com:9095"}]: dispatch 2024-04-12T05:30:34.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:34 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:34.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:34 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:36.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:36 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:36.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:36 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:36.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:36 smithi032 ceph-mon[39731]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:36.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:36 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:36.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:36 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:36.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:36 smithi067 ceph-mon[49010]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:38.263 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:38 smithi032 ceph-mon[39731]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:38.263 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:38 smithi032 ceph-mon[39731]: from='osd.0 [v2:172.21.15.32:6802/3061707028,v1:172.21.15.32:6803/3061707028]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-12T05:30:38.295 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 0 on host 'smithi032' 2024-04-12T05:30:38.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:38.295+0000 7f68aa7fc700 1 -- 172.21.15.32:0/3511493624 <== mgr.14208 v2:172.21.15.32:6800/595990262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f68b4111920 con 0x7f68a0069210 2024-04-12T05:30:38.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:38.297+0000 7f68bbded700 1 -- 172.21.15.32:0/3511493624 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f68a0069210 msgr2=0x7f68a006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:38.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:38.297+0000 7f68bbded700 1 --2- 172.21.15.32:0/3511493624 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f68a0069210 0x7f68a006b6c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f68a4004f40 tx=0x7f68a400b040 comp rx=0 tx=0).stop 2024-04-12T05:30:38.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:38.297+0000 7f68bbded700 1 -- 172.21.15.32:0/3511493624 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68b40fcf70 msgr2=0x7f68b41100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:38.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:38.297+0000 7f68bbded700 1 --2- 172.21.15.32:0/3511493624 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68b40fcf70 0x7f68b41100a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f68b000b3b0 tx=0x7f68b0009e30 comp rx=0 tx=0).stop 2024-04-12T05:30:38.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:38.297+0000 7f68bbded700 1 -- 172.21.15.32:0/3511493624 shutdown_connections 2024-04-12T05:30:38.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:38.297+0000 7f68bbded700 1 --2- 172.21.15.32:0/3511493624 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f68a0069210 0x7f68a006b6c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:30:38.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:38.297+0000 7f68bbded700 1 --2- 172.21.15.32:0/3511493624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68b40fbe00 0x7f68b410fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:30:38.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:38.297+0000 7f68bbded700 1 --2- 172.21.15.32:0/3511493624 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68b40fcf70 0x7f68b41100a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:30:38.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:38.297+0000 7f68bbded700 1 -- 172.21.15.32:0/3511493624 >> 172.21.15.32:0/3511493624 conn(0x7f68b40f74e0 msgr2=0x7f68b4066500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:38.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:38.297+0000 7f68bbded700 1 -- 172.21.15.32:0/3511493624 shutdown_connections 2024-04-12T05:30:38.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:38.297+0000 7f68bbded700 1 -- 172.21.15.32:0/3511493624 wait complete. 2024-04-12T05:30:38.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:38 smithi067 ceph-mon[49010]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:38.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:38 smithi067 ceph-mon[49010]: from='osd.0 [v2:172.21.15.32:6802/3061707028,v1:172.21.15.32:6803/3061707028]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-12T05:30:39.418 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:39 smithi032 ceph-mon[39731]: from='osd.0 [v2:172.21.15.32:6802/3061707028,v1:172.21.15.32:6803/3061707028]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-12T05:30:39.418 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:39 smithi032 ceph-mon[39731]: osdmap e7: 1 total, 0 up, 1 in 2024-04-12T05:30:39.418 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:39 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:30:39.419 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:39 smithi032 ceph-mon[39731]: from='osd.0 [v2:172.21.15.32:6802/3061707028,v1:172.21.15.32:6803/3061707028]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-12T05:30:39.419 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:39 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:39.419 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:39 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:39.440 DEBUG:teuthology.orchestra.run.smithi032:osd.0> sudo journalctl -f -n 0 -u ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@osd.0.service 2024-04-12T05:30:39.443 INFO:tasks.cephadm:Deploying osd.1 on smithi032 with /dev/vg_nvme/lv_3... 2024-04-12T05:30:39.443 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-12T05:30:39.475 INFO:journalctl@ceph.osd.0.smithi032.stdout:-- Logs begin at Fri 2024-04-12 05:14:13 UTC. -- 2024-04-12T05:30:39.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:39 smithi067 ceph-mon[49010]: from='osd.0 [v2:172.21.15.32:6802/3061707028,v1:172.21.15.32:6803/3061707028]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-12T05:30:39.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:39 smithi067 ceph-mon[49010]: osdmap e7: 1 total, 0 up, 1 in 2024-04-12T05:30:39.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:39 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:30:39.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:39 smithi067 ceph-mon[49010]: from='osd.0 [v2:172.21.15.32:6802/3061707028,v1:172.21.15.32:6803/3061707028]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-12T05:30:39.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:39 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:39.523 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:39 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:40.241 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:30:40.504 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:40 smithi032 ceph-mon[39731]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:40.505 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:40 smithi032 ceph-mon[39731]: from='osd.0 [v2:172.21.15.32:6802/3061707028,v1:172.21.15.32:6803/3061707028]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-12T05:30:40.505 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:40 smithi032 ceph-mon[39731]: osdmap e8: 1 total, 0 up, 1 in 2024-04-12T05:30:40.505 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:40 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:30:40.505 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:40 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:30:40.505 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:40 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:30:40.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:40 smithi067 ceph-mon[49010]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:40.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:40 smithi067 ceph-mon[49010]: from='osd.0 [v2:172.21.15.32:6802/3061707028,v1:172.21.15.32:6803/3061707028]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-12T05:30:40.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:40 smithi067 ceph-mon[49010]: osdmap e8: 1 total, 0 up, 1 in 2024-04-12T05:30:40.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:40 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:30:40.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:40 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:30:40.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:40 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:30:41.472 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:41 smithi032 ceph-mon[39731]: purged_snaps scrub starts 2024-04-12T05:30:41.473 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:41 smithi032 ceph-mon[39731]: purged_snaps scrub ok 2024-04-12T05:30:41.473 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:41 smithi032 ceph-mon[39731]: osd.0 [v2:172.21.15.32:6802/3061707028,v1:172.21.15.32:6803/3061707028] boot 2024-04-12T05:30:41.473 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:41 smithi032 ceph-mon[39731]: osdmap e9: 1 total, 1 up, 1 in 2024-04-12T05:30:41.473 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:41 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:30:41.473 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:41 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:41.473 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:41 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:41.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:41 smithi067 ceph-mon[49010]: purged_snaps scrub starts 2024-04-12T05:30:41.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:41 smithi067 ceph-mon[49010]: purged_snaps scrub ok 2024-04-12T05:30:41.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:41 smithi067 ceph-mon[49010]: osd.0 [v2:172.21.15.32:6802/3061707028,v1:172.21.15.32:6803/3061707028] boot 2024-04-12T05:30:41.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:41 smithi067 ceph-mon[49010]: osdmap e9: 1 total, 1 up, 1 in 2024-04-12T05:30:41.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:41 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:30:41.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:41 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:41.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:41 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:42.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:42 smithi067 ceph-mon[49010]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:42.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:42 smithi067 ceph-mon[49010]: osdmap e10: 1 total, 1 up, 1 in 2024-04-12T05:30:42.553 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:42 smithi032 ceph-mon[39731]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:30:42.553 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:42 smithi032 ceph-mon[39731]: osdmap e10: 1 total, 1 up, 1 in 2024-04-12T05:30:43.010 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-12T05:30:43.010 INFO:teuthology.orchestra.run.smithi032.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-12T05:30:43.010 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10+0 records in 2024-04-12T05:30:43.010 INFO:teuthology.orchestra.run.smithi032.stderr:10+0 records out 2024-04-12T05:30:43.010 INFO:teuthology.orchestra.run.smithi032.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00816183 s, 1.3 GB/s 2024-04-12T05:30:43.010 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 2024-04-12T05:30:43.011 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping successful for: 2024-04-12T05:30:43.678 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph orch daemon add osd smithi032:vg_nvme/lv_3 2024-04-12T05:30:44.186 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:30:44.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:44 smithi032 ceph-mon[39731]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:30:44.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:44 smithi067 ceph-mon[49010]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:30:46.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:46 smithi032 ceph-mon[39731]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:30:46.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:46 smithi067 ceph-mon[49010]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:30:46.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.651+0000 7f52adb08700 1 -- 172.21.15.32:0/1349611946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52a80fcfa0 msgr2=0x7f52a80fd410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:46.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.651+0000 7f52adb08700 1 --2- 172.21.15.32:0/1349611946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52a80fcfa0 0x7f52a80fd410 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f529c004940 tx=0x7f529c009e30 comp rx=0 tx=0).stop 2024-04-12T05:30:46.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.652+0000 7f52adb08700 1 -- 172.21.15.32:0/1349611946 shutdown_connections 2024-04-12T05:30:46.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.652+0000 7f52adb08700 1 --2- 172.21.15.32:0/1349611946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52a80fcfa0 0x7f52a80fd410 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:30:46.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.652+0000 7f52adb08700 1 --2- 172.21.15.32:0/1349611946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52a80fbe30 0x7f52a80fc240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:30:46.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.652+0000 7f52adb08700 1 -- 172.21.15.32:0/1349611946 >> 172.21.15.32:0/1349611946 conn(0x7f52a80f7480 msgr2=0x7f52a80f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:46.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.652+0000 7f52adb08700 1 -- 172.21.15.32:0/1349611946 shutdown_connections 2024-04-12T05:30:46.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.652+0000 7f52adb08700 1 -- 172.21.15.32:0/1349611946 wait complete. 2024-04-12T05:30:46.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.652+0000 7f52adb08700 1 Processor -- start 2024-04-12T05:30:46.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.652+0000 7f52adb08700 1 -- start start 2024-04-12T05:30:46.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.653+0000 7f52adb08700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52a80fbe30 0x7f52a81105d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:30:46.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.653+0000 7f52adb08700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52a80fcfa0 0x7f52a8112b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:30:46.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.653+0000 7f52adb08700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a8066be0 con 0x7f52a80fbe30 2024-04-12T05:30:46.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.653+0000 7f52adb08700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a8066d20 con 0x7f52a80fcfa0 2024-04-12T05:30:46.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.653+0000 7f52a77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52a80fbe30 0x7f52a81105d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:46.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.653+0000 7f52a77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52a80fbe30 0x7f52a81105d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50660/0 (socket says 172.21.15.32:50660) 2024-04-12T05:30:46.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.653+0000 7f52a77fe700 1 -- 172.21.15.32:0/545410530 learned_addr learned my addr 172.21.15.32:0/545410530 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:30:46.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.653+0000 7f52a6ffd700 1 --2- 172.21.15.32:0/545410530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52a80fcfa0 0x7f52a8112b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:46.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.653+0000 7f52a77fe700 1 -- 172.21.15.32:0/545410530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52a80fcfa0 msgr2=0x7f52a8112b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:46.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.653+0000 7f52a77fe700 1 --2- 172.21.15.32:0/545410530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52a80fcfa0 0x7f52a8112b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:30:46.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.653+0000 7f52a77fe700 1 -- 172.21.15.32:0/545410530 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f529c005040 con 0x7f52a80fbe30 2024-04-12T05:30:46.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.653+0000 7f52a77fe700 1 --2- 172.21.15.32:0/545410530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52a80fbe30 0x7f52a81105d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f52a000eb40 tx=0x7f52a000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:46.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.653+0000 7f528ffff700 1 -- 172.21.15.32:0/545410530 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52a0007cc0 con 0x7f52a80fbe30 2024-04-12T05:30:46.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.653+0000 7f52adb08700 1 -- 172.21.15.32:0/545410530 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a8113060 con 0x7f52a80fbe30 2024-04-12T05:30:46.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.653+0000 7f528ffff700 1 -- 172.21.15.32:0/545410530 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f52a000f040 con 0x7f52a80fbe30 2024-04-12T05:30:46.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.653+0000 7f528ffff700 1 -- 172.21.15.32:0/545410530 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52a000c990 con 0x7f52a80fbe30 2024-04-12T05:30:46.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.653+0000 7f52adb08700 1 -- 172.21.15.32:0/545410530 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a8113620 con 0x7f52a80fbe30 2024-04-12T05:30:46.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.655+0000 7f52adb08700 1 -- 172.21.15.32:0/545410530 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52a805f5f0 con 0x7f52a80fbe30 2024-04-12T05:30:46.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.655+0000 7f528ffff700 1 -- 172.21.15.32:0/545410530 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f52a000cc00 con 0x7f52a80fbe30 2024-04-12T05:30:46.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.655+0000 7f528ffff700 1 --2- 172.21.15.32:0/545410530 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f5290069210 0x7f529006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:30:46.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.656+0000 7f528ffff700 1 -- 172.21.15.32:0/545410530 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7f52a0085ea0 con 0x7f52a80fbe30 2024-04-12T05:30:46.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.656+0000 7f52a6ffd700 1 --2- 172.21.15.32:0/545410530 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f5290069210 0x7f529006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:46.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.656+0000 7f52a6ffd700 1 --2- 172.21.15.32:0/545410530 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f5290069210 0x7f529006b6c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f529c013680 tx=0x7f529c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:46.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.660+0000 7f528ffff700 1 -- 172.21.15.32:0/545410530 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f52a004e190 con 0x7f52a80fbe30 2024-04-12T05:30:46.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:30:46.869+0000 7f52adb08700 1 -- 172.21.15.32:0/545410530 --> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f52a8101310 con 0x7f5290069210 2024-04-12T05:30:47.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:47 smithi067 ceph-mon[49010]: Detected new or changed devices on smithi032 2024-04-12T05:30:47.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:47 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:47.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:47 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:47.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:47 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:30:47.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:47 smithi067 ceph-mon[49010]: Adjusting osd_memory_target on smithi032 to 12902M 2024-04-12T05:30:47.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:47 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:47.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:47 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:30:47.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:47 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:30:47.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:47 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:47.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:47 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:30:47.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:47 smithi067 ceph-mon[49010]: from='client.14258 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:30:47.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:47 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T05:30:47.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:47 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T05:30:47.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:47 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:30:47.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:47 smithi067 ceph-mon[49010]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:30:47.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:47 smithi032 ceph-mon[39731]: Detected new or changed devices on smithi032 2024-04-12T05:30:47.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:47 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:47.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:47 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:47.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:47 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:30:47.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:47 smithi032 ceph-mon[39731]: Adjusting osd_memory_target on smithi032 to 12902M 2024-04-12T05:30:47.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:47 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:47.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:47 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:30:47.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:47 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:30:47.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:47 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:47.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:47 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:30:47.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:47 smithi032 ceph-mon[39731]: from='client.14258 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:30:47.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:47 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T05:30:47.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:47 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T05:30:47.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:47 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:30:47.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:47 smithi032 ceph-mon[39731]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:30:49.992 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:49 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:49.992 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:49 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:49.992 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:49 smithi032 ceph-mon[39731]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:30:50.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:49 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:50.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:49 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:50.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:49 smithi067 ceph-mon[49010]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:30:51.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:50 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/3880572760' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86b93557-6e6f-4b84-a5f5-5bce4aa65a32"}]: dispatch 2024-04-12T05:30:51.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:50 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/3880572760' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "86b93557-6e6f-4b84-a5f5-5bce4aa65a32"}]': finished 2024-04-12T05:30:51.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:50 smithi032 ceph-mon[39731]: osdmap e11: 2 total, 1 up, 2 in 2024-04-12T05:30:51.093 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:50 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:30:51.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:50 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/3880572760' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86b93557-6e6f-4b84-a5f5-5bce4aa65a32"}]: dispatch 2024-04-12T05:30:51.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:50 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/3880572760' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "86b93557-6e6f-4b84-a5f5-5bce4aa65a32"}]': finished 2024-04-12T05:30:51.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:50 smithi067 ceph-mon[49010]: osdmap e11: 2 total, 1 up, 2 in 2024-04-12T05:30:51.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:50 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:30:52.092 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:51 smithi032 ceph-mon[39731]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:30:52.092 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:51 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/4106173284' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T05:30:52.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:51 smithi067 ceph-mon[49010]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:30:52.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:51 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/4106173284' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T05:30:53.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:53 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:53.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:53 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:53.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:53 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:30:53.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:53 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:30:53.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:53 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:53.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:53 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:53.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:53 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:53.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:53 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:30:53.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:53 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:30:53.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:53 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:30:54.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:54 smithi067 ceph-mon[49010]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:30:54.523 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:54 smithi032 ceph-mon[39731]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:30:55.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:55 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:30:55.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:55 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:30:56.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:56 smithi067 ceph-mon[49010]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:30:56.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:56 smithi032 ceph-mon[39731]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:30:58.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:58 smithi067 ceph-mon[49010]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:30:58.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:58 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-12T05:30:58.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:58 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:30:58.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:58 smithi032 ceph-mon[39731]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:30:58.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:58 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-12T05:30:58.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:58 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:30:59.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:30:59 smithi067 ceph-mon[49010]: Deploying daemon osd.1 on smithi032 2024-04-12T05:30:59.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:30:59 smithi032 ceph-mon[39731]: Deploying daemon osd.1 on smithi032 2024-04-12T05:31:00.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:00 smithi032 ceph-mon[39731]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:31:00.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:00 smithi067 ceph-mon[49010]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:31:02.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:02 smithi067 ceph-mon[49010]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:31:02.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:02 smithi032 ceph-mon[39731]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:31:04.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:04 smithi067 ceph-mon[49010]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:31:04.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:04 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:04.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:04 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:04.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:04 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:31:04.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:04 smithi032 ceph-mon[39731]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:31:04.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:04 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:04.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:04 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:04.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:04 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:31:06.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:06 smithi067 ceph-mon[49010]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:31:06.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:06 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:06.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:06 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:06.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:06 smithi032 ceph-mon[39731]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:31:06.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:06 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:06.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:06 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:07.582 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 1 on host 'smithi032' 2024-04-12T05:31:07.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:07.581+0000 7f528ffff700 1 -- 172.21.15.32:0/545410530 <== mgr.14208 v2:172.21.15.32:6800/595990262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f52a8101310 con 0x7f5290069210 2024-04-12T05:31:07.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:07.583+0000 7f52adb08700 1 -- 172.21.15.32:0/545410530 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f5290069210 msgr2=0x7f529006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:07.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:07.583+0000 7f52adb08700 1 --2- 172.21.15.32:0/545410530 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f5290069210 0x7f529006b6c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f529c013680 tx=0x7f529c009e30 comp rx=0 tx=0).stop 2024-04-12T05:31:07.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:07.584+0000 7f52adb08700 1 -- 172.21.15.32:0/545410530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52a80fbe30 msgr2=0x7f52a81105d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:07.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:07.584+0000 7f52adb08700 1 --2- 172.21.15.32:0/545410530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52a80fbe30 0x7f52a81105d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f52a000eb40 tx=0x7f52a000b040 comp rx=0 tx=0).stop 2024-04-12T05:31:07.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:07.584+0000 7f52adb08700 1 -- 172.21.15.32:0/545410530 shutdown_connections 2024-04-12T05:31:07.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:07.584+0000 7f52adb08700 1 --2- 172.21.15.32:0/545410530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52a80fbe30 0x7f52a81105d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:31:07.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:07.584+0000 7f52adb08700 1 --2- 172.21.15.32:0/545410530 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f5290069210 0x7f529006b6c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:31:07.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:07.584+0000 7f52adb08700 1 --2- 172.21.15.32:0/545410530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52a80fcfa0 0x7f52a8112b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:31:07.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:07.584+0000 7f52adb08700 1 -- 172.21.15.32:0/545410530 >> 172.21.15.32:0/545410530 conn(0x7f52a80f7480 msgr2=0x7f52a80f8bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:31:07.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:07.584+0000 7f52adb08700 1 -- 172.21.15.32:0/545410530 shutdown_connections 2024-04-12T05:31:07.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:07.584+0000 7f52adb08700 1 -- 172.21.15.32:0/545410530 wait complete. 2024-04-12T05:31:08.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:08 smithi032 ceph-mon[39731]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:31:08.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:08 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:08.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:08 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:08.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:08 smithi032 ceph-mon[39731]: from='osd.1 [v2:172.21.15.32:6810/164843120,v1:172.21.15.32:6811/164843120]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-12T05:31:08.436 DEBUG:teuthology.orchestra.run.smithi032:osd.1> sudo journalctl -f -n 0 -u ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@osd.1.service 2024-04-12T05:31:08.438 INFO:tasks.cephadm:Deploying osd.2 on smithi032 with /dev/vg_nvme/lv_2... 2024-04-12T05:31:08.438 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-12T05:31:08.470 INFO:journalctl@ceph.osd.1.smithi032.stdout:-- Logs begin at Fri 2024-04-12 05:14:13 UTC. -- 2024-04-12T05:31:08.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:08 smithi067 ceph-mon[49010]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:31:08.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:08 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:08.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:08 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:08.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:08 smithi067 ceph-mon[49010]: from='osd.1 [v2:172.21.15.32:6810/164843120,v1:172.21.15.32:6811/164843120]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-12T05:31:09.843 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:31:09 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-1[73003]: 2024-04-12T05:31:09.569+0000 7f48a3ada700 -1 osd.1 0 waiting for initial osdmap 2024-04-12T05:31:09.843 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:31:09 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-1[73003]: 2024-04-12T05:31:09.575+0000 7f489e0d4700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T05:31:09.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:09 smithi032 ceph-mon[39731]: from='osd.1 [v2:172.21.15.32:6810/164843120,v1:172.21.15.32:6811/164843120]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-12T05:31:09.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:09 smithi032 ceph-mon[39731]: osdmap e12: 2 total, 1 up, 2 in 2024-04-12T05:31:09.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:09 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:31:09.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:09 smithi032 ceph-mon[39731]: from='osd.1 [v2:172.21.15.32:6810/164843120,v1:172.21.15.32:6811/164843120]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-12T05:31:09.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:09 smithi032 ceph-mon[39731]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:31:09.917 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:31:10.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:09 smithi067 ceph-mon[49010]: from='osd.1 [v2:172.21.15.32:6810/164843120,v1:172.21.15.32:6811/164843120]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-12T05:31:10.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:09 smithi067 ceph-mon[49010]: osdmap e12: 2 total, 1 up, 2 in 2024-04-12T05:31:10.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:09 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:31:10.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:09 smithi067 ceph-mon[49010]: from='osd.1 [v2:172.21.15.32:6810/164843120,v1:172.21.15.32:6811/164843120]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-12T05:31:10.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:09 smithi067 ceph-mon[49010]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:31:10.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:10 smithi032 ceph-mon[39731]: from='osd.1 [v2:172.21.15.32:6810/164843120,v1:172.21.15.32:6811/164843120]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-12T05:31:10.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:10 smithi032 ceph-mon[39731]: osdmap e13: 2 total, 1 up, 2 in 2024-04-12T05:31:10.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:10 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:31:10.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:10 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:31:10.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:10 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:31:11.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:10 smithi067 ceph-mon[49010]: from='osd.1 [v2:172.21.15.32:6810/164843120,v1:172.21.15.32:6811/164843120]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-12T05:31:11.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:10 smithi067 ceph-mon[49010]: osdmap e13: 2 total, 1 up, 2 in 2024-04-12T05:31:11.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:10 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:31:11.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:10 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:31:11.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:10 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:31:11.799 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:11 smithi032 ceph-mon[39731]: purged_snaps scrub starts 2024-04-12T05:31:11.799 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:11 smithi032 ceph-mon[39731]: purged_snaps scrub ok 2024-04-12T05:31:11.799 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:11 smithi032 ceph-mon[39731]: osd.1 [v2:172.21.15.32:6810/164843120,v1:172.21.15.32:6811/164843120] boot 2024-04-12T05:31:11.799 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:11 smithi032 ceph-mon[39731]: osdmap e14: 2 total, 2 up, 2 in 2024-04-12T05:31:11.800 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:11 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:31:11.800 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:11 smithi032 ceph-mon[39731]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:31:12.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:11 smithi067 ceph-mon[49010]: purged_snaps scrub starts 2024-04-12T05:31:12.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:11 smithi067 ceph-mon[49010]: purged_snaps scrub ok 2024-04-12T05:31:12.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:11 smithi067 ceph-mon[49010]: osd.1 [v2:172.21.15.32:6810/164843120,v1:172.21.15.32:6811/164843120] boot 2024-04-12T05:31:12.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:11 smithi067 ceph-mon[49010]: osdmap e14: 2 total, 2 up, 2 in 2024-04-12T05:31:12.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:11 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:31:12.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:11 smithi067 ceph-mon[49010]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:31:12.435 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-12T05:31:12.435 INFO:teuthology.orchestra.run.smithi032.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-12T05:31:12.435 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10+0 records in 2024-04-12T05:31:12.435 INFO:teuthology.orchestra.run.smithi032.stderr:10+0 records out 2024-04-12T05:31:12.435 INFO:teuthology.orchestra.run.smithi032.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00815824 s, 1.3 GB/s 2024-04-12T05:31:12.435 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 2024-04-12T05:31:12.436 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping successful for: 2024-04-12T05:31:12.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:12 smithi032 ceph-mon[39731]: osdmap e15: 2 total, 2 up, 2 in 2024-04-12T05:31:12.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:12 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:12.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:12 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:13.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:12 smithi067 ceph-mon[49010]: osdmap e15: 2 total, 2 up, 2 in 2024-04-12T05:31:13.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:12 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:13.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:12 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:13.138 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph orch daemon add osd smithi032:vg_nvme/lv_2 2024-04-12T05:31:13.806 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:13 smithi032 ceph-mon[39731]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:31:13.840 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:31:14.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:13 smithi067 ceph-mon[49010]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:31:15.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.881+0000 7f5d45ca8700 1 -- 172.21.15.32:0/1383933542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d40071e40 msgr2=0x7f5d40070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:15.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.881+0000 7f5d45ca8700 1 --2- 172.21.15.32:0/1383933542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d40071e40 0x7f5d40070350 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5d30004940 tx=0x7f5d30009e30 comp rx=0 tx=0).stop 2024-04-12T05:31:15.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.881+0000 7f5d45ca8700 1 -- 172.21.15.32:0/1383933542 shutdown_connections 2024-04-12T05:31:15.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.881+0000 7f5d45ca8700 1 --2- 172.21.15.32:0/1383933542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d40070890 0x7f5d40070d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:31:15.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.881+0000 7f5d45ca8700 1 --2- 172.21.15.32:0/1383933542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d40071e40 0x7f5d40070350 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:31:15.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.881+0000 7f5d45ca8700 1 -- 172.21.15.32:0/1383933542 >> 172.21.15.32:0/1383933542 conn(0x7f5d400f8540 msgr2=0x7f5d400fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:31:15.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.881+0000 7f5d45ca8700 1 -- 172.21.15.32:0/1383933542 shutdown_connections 2024-04-12T05:31:15.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.882+0000 7f5d45ca8700 1 -- 172.21.15.32:0/1383933542 wait complete. 2024-04-12T05:31:15.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.882+0000 7f5d45ca8700 1 Processor -- start 2024-04-12T05:31:15.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.882+0000 7f5d45ca8700 1 -- start start 2024-04-12T05:31:15.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.883+0000 7f5d45ca8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d40070890 0x7f5d4010fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:31:15.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.883+0000 7f5d45ca8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d40071e40 0x7f5d401100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:31:15.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.883+0000 7f5d45ca8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d401106b0 con 0x7f5d40071e40 2024-04-12T05:31:15.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.883+0000 7f5d45ca8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d40110f00 con 0x7f5d40070890 2024-04-12T05:31:15.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.883+0000 7f5d3effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d40071e40 0x7f5d401100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:15.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.883+0000 7f5d3effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d40071e40 0x7f5d401100b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33986/0 (socket says 172.21.15.32:33986) 2024-04-12T05:31:15.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.883+0000 7f5d3effd700 1 -- 172.21.15.32:0/2408181596 learned_addr learned my addr 172.21.15.32:0/2408181596 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:31:15.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.883+0000 7f5d3f7fe700 1 --2- 172.21.15.32:0/2408181596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d40070890 0x7f5d4010fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:15.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.883+0000 7f5d3effd700 1 -- 172.21.15.32:0/2408181596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d40070890 msgr2=0x7f5d4010fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:15.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.883+0000 7f5d3effd700 1 --2- 172.21.15.32:0/2408181596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d40070890 0x7f5d4010fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:31:15.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.883+0000 7f5d3effd700 1 -- 172.21.15.32:0/2408181596 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d30005040 con 0x7f5d40071e40 2024-04-12T05:31:15.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.883+0000 7f5d3f7fe700 1 --2- 172.21.15.32:0/2408181596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d40070890 0x7f5d4010fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:31:15.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.883+0000 7f5d3effd700 1 --2- 172.21.15.32:0/2408181596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d40071e40 0x7f5d401100b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5d3400c830 tx=0x7f5d34004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:31:15.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.884+0000 7f5d27fff700 1 -- 172.21.15.32:0/2408181596 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d3400a760 con 0x7f5d40071e40 2024-04-12T05:31:15.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.884+0000 7f5d45ca8700 1 -- 172.21.15.32:0/2408181596 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d401111e0 con 0x7f5d40071e40 2024-04-12T05:31:15.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.884+0000 7f5d27fff700 1 -- 172.21.15.32:0/2408181596 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d3400ad70 con 0x7f5d40071e40 2024-04-12T05:31:15.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.884+0000 7f5d27fff700 1 -- 172.21.15.32:0/2408181596 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d340113a0 con 0x7f5d40071e40 2024-04-12T05:31:15.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.884+0000 7f5d45ca8700 1 -- 172.21.15.32:0/2408181596 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d40111720 con 0x7f5d40071e40 2024-04-12T05:31:15.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.884+0000 7f5d45ca8700 1 -- 172.21.15.32:0/2408181596 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d4005f5f0 con 0x7f5d40071e40 2024-04-12T05:31:15.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.886+0000 7f5d27fff700 1 -- 172.21.15.32:0/2408181596 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f5d34012400 con 0x7f5d40071e40 2024-04-12T05:31:15.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.886+0000 7f5d27fff700 1 --2- 172.21.15.32:0/2408181596 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f5d28069260 0x7f5d2806b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:31:15.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.887+0000 7f5d27fff700 1 -- 172.21.15.32:0/2408181596 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7f5d34086ff0 con 0x7f5d40071e40 2024-04-12T05:31:15.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.887+0000 7f5d3f7fe700 1 --2- 172.21.15.32:0/2408181596 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f5d28069260 0x7f5d2806b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:15.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.887+0000 7f5d3f7fe700 1 --2- 172.21.15.32:0/2408181596 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f5d28069260 0x7f5d2806b710 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5d30000c00 tx=0x7f5d30009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:31:15.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:15.890+0000 7f5d27fff700 1 -- 172.21.15.32:0/2408181596 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5d34057a60 con 0x7f5d40071e40 2024-04-12T05:31:16.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:16.132+0000 7f5d45ca8700 1 -- 172.21.15.32:0/2408181596 --> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d40111bc0 con 0x7f5d28069260 2024-04-12T05:31:16.458 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:16 smithi032 ceph-mon[39731]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:31:16.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:16 smithi067 ceph-mon[49010]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:31:17.306 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:17 smithi032 ceph-mon[39731]: from='client.14276 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:31:17.306 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:17 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T05:31:17.306 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:17 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T05:31:17.306 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:17 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:31:17.306 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:17 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:17.306 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:17 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:17.307 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:17 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:31:17.307 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:17 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:31:17.307 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:17 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:17.307 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:17 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:31:17.307 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:17 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:31:17.307 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:17 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:17.307 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:17 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:31:17.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:17 smithi067 ceph-mon[49010]: from='client.14276 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:31:17.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:17 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T05:31:17.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:17 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T05:31:17.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:17 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:31:17.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:17 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:17.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:17 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:17.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:17 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:31:17.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:17 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:31:17.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:17 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:17.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:17 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:31:17.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:17 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:31:17.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:17 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:17.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:17 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:31:18.227 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:18 smithi032 ceph-mon[39731]: Detected new or changed devices on smithi032 2024-04-12T05:31:18.227 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:18 smithi032 ceph-mon[39731]: Adjusting osd_memory_target on smithi032 to 6451M 2024-04-12T05:31:18.227 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:18 smithi032 ceph-mon[39731]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:31:18.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:18 smithi067 ceph-mon[49010]: Detected new or changed devices on smithi032 2024-04-12T05:31:18.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:18 smithi067 ceph-mon[49010]: Adjusting osd_memory_target on smithi032 to 6451M 2024-04-12T05:31:18.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:18 smithi067 ceph-mon[49010]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:31:19.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:19 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/2366411650' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1556613-e0de-4b0e-bb7d-e90361ec9007"}]: dispatch 2024-04-12T05:31:19.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:19 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/2366411650' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d1556613-e0de-4b0e-bb7d-e90361ec9007"}]': finished 2024-04-12T05:31:19.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:19 smithi032 ceph-mon[39731]: osdmap e16: 3 total, 2 up, 3 in 2024-04-12T05:31:19.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:19 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:31:19.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:19 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/2366411650' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1556613-e0de-4b0e-bb7d-e90361ec9007"}]: dispatch 2024-04-12T05:31:19.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:19 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/2366411650' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d1556613-e0de-4b0e-bb7d-e90361ec9007"}]': finished 2024-04-12T05:31:19.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:19 smithi067 ceph-mon[49010]: osdmap e16: 3 total, 2 up, 3 in 2024-04-12T05:31:19.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:19 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:31:20.496 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:20 smithi032 ceph-mon[39731]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:31:20.497 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:20 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/764188881' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T05:31:20.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:20 smithi067 ceph-mon[49010]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:31:20.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:20 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/764188881' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T05:31:22.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:22 smithi032 ceph-mon[39731]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:31:22.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:22 smithi067 ceph-mon[49010]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:31:23.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:23 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:23.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:23 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:23.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:23 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:31:23.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:23 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:31:23.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:23 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:23.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:23 smithi067 ceph-mon[49010]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:31:23.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:23 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:23.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:23 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:23.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:23 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:31:23.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:23 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:31:23.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:23 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:23.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:23 smithi032 ceph-mon[39731]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:31:25.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:25 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:31:25.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:25 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:31:26.428 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:26 smithi032 ceph-mon[39731]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:31:26.539 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:26 smithi067 ceph-mon[49010]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:31:27.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:27 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-12T05:31:27.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:27 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:31:27.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:27 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-12T05:31:27.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:27 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:31:28.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:28 smithi032 ceph-mon[39731]: Deploying daemon osd.2 on smithi032 2024-04-12T05:31:28.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:28 smithi032 ceph-mon[39731]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:31:28.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:28 smithi067 ceph-mon[49010]: Deploying daemon osd.2 on smithi032 2024-04-12T05:31:28.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:28 smithi067 ceph-mon[49010]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:31:30.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:30 smithi067 ceph-mon[49010]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:31:30.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:30 smithi032 ceph-mon[39731]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:31:32.440 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:32 smithi032 ceph-mon[39731]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:31:32.440 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:32 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:32.440 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:32 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:32.440 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:32 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:31:32.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:32 smithi067 ceph-mon[49010]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:31:32.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:32 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:32.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:32 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:32.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:32 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:31:34.443 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:34 smithi032 ceph-mon[39731]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:31:34.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:34 smithi067 ceph-mon[49010]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:31:36.040 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 2 on host 'smithi032' 2024-04-12T05:31:36.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:36.040+0000 7f5d27fff700 1 -- 172.21.15.32:0/2408181596 <== mgr.14208 v2:172.21.15.32:6800/595990262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5d40111bc0 con 0x7f5d28069260 2024-04-12T05:31:36.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:36.042+0000 7f5d45ca8700 1 -- 172.21.15.32:0/2408181596 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f5d28069260 msgr2=0x7f5d2806b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:36.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:36.042+0000 7f5d45ca8700 1 --2- 172.21.15.32:0/2408181596 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f5d28069260 0x7f5d2806b710 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5d30000c00 tx=0x7f5d30009e30 comp rx=0 tx=0).stop 2024-04-12T05:31:36.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:36.042+0000 7f5d45ca8700 1 -- 172.21.15.32:0/2408181596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d40071e40 msgr2=0x7f5d401100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:36.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:36.042+0000 7f5d45ca8700 1 --2- 172.21.15.32:0/2408181596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d40071e40 0x7f5d401100b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5d3400c830 tx=0x7f5d34004470 comp rx=0 tx=0).stop 2024-04-12T05:31:36.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:36.042+0000 7f5d45ca8700 1 -- 172.21.15.32:0/2408181596 shutdown_connections 2024-04-12T05:31:36.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:36.042+0000 7f5d45ca8700 1 --2- 172.21.15.32:0/2408181596 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f5d28069260 0x7f5d2806b710 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:31:36.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:36.042+0000 7f5d45ca8700 1 --2- 172.21.15.32:0/2408181596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d40070890 0x7f5d4010fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:31:36.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:36.042+0000 7f5d45ca8700 1 --2- 172.21.15.32:0/2408181596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d40071e40 0x7f5d401100b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:31:36.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:36.042+0000 7f5d45ca8700 1 -- 172.21.15.32:0/2408181596 >> 172.21.15.32:0/2408181596 conn(0x7f5d400f8540 msgr2=0x7f5d400f9d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:31:36.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:36.042+0000 7f5d45ca8700 1 -- 172.21.15.32:0/2408181596 shutdown_connections 2024-04-12T05:31:36.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:31:36.042+0000 7f5d45ca8700 1 -- 172.21.15.32:0/2408181596 wait complete. 2024-04-12T05:31:36.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:36 smithi032 ceph-mon[39731]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:31:36.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:36 smithi032 ceph-mon[39731]: from='osd.2 [v2:172.21.15.32:6818/2698911336,v1:172.21.15.32:6819/2698911336]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-12T05:31:36.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:36 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:36.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:36 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:36.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:36 smithi067 ceph-mon[49010]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:31:36.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:36 smithi067 ceph-mon[49010]: from='osd.2 [v2:172.21.15.32:6818/2698911336,v1:172.21.15.32:6819/2698911336]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-12T05:31:36.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:36 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:36.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:36 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:36.762 DEBUG:teuthology.orchestra.run.smithi032:osd.2> sudo journalctl -f -n 0 -u ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@osd.2.service 2024-04-12T05:31:36.764 INFO:tasks.cephadm:Deploying osd.3 on smithi067 with /dev/vg_nvme/lv_4... 2024-04-12T05:31:36.764 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-12T05:31:36.799 INFO:journalctl@ceph.osd.2.smithi032.stdout:-- Logs begin at Fri 2024-04-12 05:14:13 UTC. -- 2024-04-12T05:31:37.045 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi067/config 2024-04-12T05:31:37.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:37 smithi032 ceph-mon[39731]: from='osd.2 [v2:172.21.15.32:6818/2698911336,v1:172.21.15.32:6819/2698911336]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-12T05:31:37.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:37 smithi032 ceph-mon[39731]: osdmap e17: 3 total, 2 up, 3 in 2024-04-12T05:31:37.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:37 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:31:37.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:37 smithi032 ceph-mon[39731]: from='osd.2 [v2:172.21.15.32:6818/2698911336,v1:172.21.15.32:6819/2698911336]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-12T05:31:37.343 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:31:37 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-2[82754]: 2024-04-12T05:31:37.194+0000 7fe634ad9700 -1 osd.2 0 waiting for initial osdmap 2024-04-12T05:31:37.343 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:31:37 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-2[82754]: 2024-04-12T05:31:37.200+0000 7fe6318d8700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T05:31:37.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:37 smithi067 ceph-mon[49010]: from='osd.2 [v2:172.21.15.32:6818/2698911336,v1:172.21.15.32:6819/2698911336]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-12T05:31:37.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:37 smithi067 ceph-mon[49010]: osdmap e17: 3 total, 2 up, 3 in 2024-04-12T05:31:37.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:37 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:31:37.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:37 smithi067 ceph-mon[49010]: from='osd.2 [v2:172.21.15.32:6818/2698911336,v1:172.21.15.32:6819/2698911336]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-12T05:31:38.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:38 smithi032 ceph-mon[39731]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:31:38.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:38 smithi032 ceph-mon[39731]: from='osd.2 [v2:172.21.15.32:6818/2698911336,v1:172.21.15.32:6819/2698911336]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-12T05:31:38.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:38 smithi032 ceph-mon[39731]: osdmap e18: 3 total, 2 up, 3 in 2024-04-12T05:31:38.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:38 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:31:38.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:38 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:31:38.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:38 smithi067 ceph-mon[49010]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:31:38.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:38 smithi067 ceph-mon[49010]: from='osd.2 [v2:172.21.15.32:6818/2698911336,v1:172.21.15.32:6819/2698911336]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-12T05:31:38.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:38 smithi067 ceph-mon[49010]: osdmap e18: 3 total, 2 up, 3 in 2024-04-12T05:31:38.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:38 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:31:38.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:38 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:31:39.219 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-12T05:31:39.219 INFO:teuthology.orchestra.run.smithi067.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-12T05:31:39.220 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10+0 records in 2024-04-12T05:31:39.220 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records out 2024-04-12T05:31:39.220 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109906 s, 954 MB/s 2024-04-12T05:31:39.220 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping successful for: 2024-04-12T05:31:39.818 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:39 smithi067 ceph-mon[49010]: purged_snaps scrub starts 2024-04-12T05:31:39.818 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:39 smithi067 ceph-mon[49010]: purged_snaps scrub ok 2024-04-12T05:31:39.818 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:39 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:31:39.818 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:39 smithi067 ceph-mon[49010]: osd.2 [v2:172.21.15.32:6818/2698911336,v1:172.21.15.32:6819/2698911336] boot 2024-04-12T05:31:39.818 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:39 smithi067 ceph-mon[49010]: osdmap e19: 3 total, 3 up, 3 in 2024-04-12T05:31:39.818 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:39 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:31:39.839 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph orch daemon add osd smithi067:vg_nvme/lv_4 2024-04-12T05:31:39.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:39 smithi032 ceph-mon[39731]: purged_snaps scrub starts 2024-04-12T05:31:39.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:39 smithi032 ceph-mon[39731]: purged_snaps scrub ok 2024-04-12T05:31:39.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:39 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:31:39.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:39 smithi032 ceph-mon[39731]: osd.2 [v2:172.21.15.32:6818/2698911336,v1:172.21.15.32:6819/2698911336] boot 2024-04-12T05:31:39.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:39 smithi032 ceph-mon[39731]: osdmap e19: 3 total, 3 up, 3 in 2024-04-12T05:31:39.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:39 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:31:40.111 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi067/config 2024-04-12T05:31:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:40 smithi067 ceph-mon[49010]: pgmap v74: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:31:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:40 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-12T05:31:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:40 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-12T05:31:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:40 smithi067 ceph-mon[49010]: osdmap e20: 3 total, 3 up, 3 in 2024-04-12T05:31:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:40 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:40 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-12T05:31:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:40 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:40 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:31:40.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:40 smithi032 ceph-mon[39731]: pgmap v74: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:31:40.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:40 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-12T05:31:40.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:40 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-12T05:31:40.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:40 smithi032 ceph-mon[39731]: osdmap e20: 3 total, 3 up, 3 in 2024-04-12T05:31:40.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:40 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:40.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:40 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-12T05:31:40.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:40 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:40.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:40 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:31:41.811 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:41 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-12T05:31:41.811 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:41 smithi032 ceph-mon[39731]: osdmap e21: 3 total, 3 up, 3 in 2024-04-12T05:31:41.811 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:41 smithi032 ceph-mon[39731]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:31:41.811 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 05:31:41 smithi032 sudo[84797]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-12T05:31:41.812 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 05:31:41 smithi032 sudo[84797]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-12T05:31:41.812 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 05:31:41 smithi032 sudo[84797]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-12T05:31:41.812 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 05:31:41 smithi032 sudo[84797]: pam_unix(sudo:session): session closed for user root 2024-04-12T05:31:41.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.892+0000 7f3e08f18700 1 -- 172.21.15.67:0/752165592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e040fcf90 msgr2=0x7f3e040fd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:41.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.892+0000 7f3e08f18700 1 --2- 172.21.15.67:0/752165592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e040fcf90 0x7f3e040fd400 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f3df8004940 tx=0x7f3df8009e30 comp rx=0 tx=0).stop 2024-04-12T05:31:41.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.892+0000 7f3e08f18700 1 -- 172.21.15.67:0/752165592 shutdown_connections 2024-04-12T05:31:41.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.892+0000 7f3e08f18700 1 --2- 172.21.15.67:0/752165592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e040fcf90 0x7f3e040fd400 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:31:41.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.892+0000 7f3e08f18700 1 --2- 172.21.15.67:0/752165592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e040fbce0 0x7f3e040fc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:31:41.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.892+0000 7f3e08f18700 1 -- 172.21.15.67:0/752165592 >> 172.21.15.67:0/752165592 conn(0x7f3e040f7440 msgr2=0x7f3e040f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:31:41.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.892+0000 7f3e08f18700 1 -- 172.21.15.67:0/752165592 shutdown_connections 2024-04-12T05:31:41.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.892+0000 7f3e08f18700 1 -- 172.21.15.67:0/752165592 wait complete. 2024-04-12T05:31:41.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.892+0000 7f3e08f18700 1 Processor -- start 2024-04-12T05:31:41.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.892+0000 7f3e08f18700 1 -- start start 2024-04-12T05:31:41.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.893+0000 7f3e08f18700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e040fbce0 0x7f3e0410c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:31:41.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.893+0000 7f3e08f18700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e040fcf90 0x7f3e0410cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:31:41.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.893+0000 7f3e08f18700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0410d390 con 0x7f3e040fcf90 2024-04-12T05:31:41.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.893+0000 7f3e08f18700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0410dbe0 con 0x7f3e040fbce0 2024-04-12T05:31:41.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.893+0000 7f3e0259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e040fbce0 0x7f3e0410c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:41.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.893+0000 7f3e0259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e040fbce0 0x7f3e0410c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33360/0 (socket says 172.21.15.67:33360) 2024-04-12T05:31:41.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.893+0000 7f3e0259c700 1 -- 172.21.15.67:0/1474741466 learned_addr learned my addr 172.21.15.67:0/1474741466 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-12T05:31:41.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.893+0000 7f3e01d9b700 1 --2- 172.21.15.67:0/1474741466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e040fcf90 0x7f3e0410cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:41.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.893+0000 7f3e0259c700 1 -- 172.21.15.67:0/1474741466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e040fcf90 msgr2=0x7f3e0410cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:41.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.893+0000 7f3e0259c700 1 --2- 172.21.15.67:0/1474741466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e040fcf90 0x7f3e0410cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:31:41.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.893+0000 7f3e0259c700 1 -- 172.21.15.67:0/1474741466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df8005040 con 0x7f3e040fbce0 2024-04-12T05:31:41.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.893+0000 7f3e0259c700 1 --2- 172.21.15.67:0/1474741466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e040fbce0 0x7f3e0410c850 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f3df4002a50 tx=0x7f3df40046b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:31:41.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.893+0000 7f3df2ffd700 1 -- 172.21.15.67:0/1474741466 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3df4007cc0 con 0x7f3e040fbce0 2024-04-12T05:31:41.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.893+0000 7f3e08f18700 1 -- 172.21.15.67:0/1474741466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e0410dec0 con 0x7f3e040fbce0 2024-04-12T05:31:41.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.893+0000 7f3df2ffd700 1 -- 172.21.15.67:0/1474741466 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3df400f040 con 0x7f3e040fbce0 2024-04-12T05:31:41.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.893+0000 7f3df2ffd700 1 -- 172.21.15.67:0/1474741466 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3df400d9c0 con 0x7f3e040fbce0 2024-04-12T05:31:41.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.893+0000 7f3e08f18700 1 -- 172.21.15.67:0/1474741466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e0410e400 con 0x7f3e040fbce0 2024-04-12T05:31:41.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.894+0000 7f3e08f18700 1 -- 172.21.15.67:0/1474741466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e0405f5f0 con 0x7f3e040fbce0 2024-04-12T05:31:41.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.895+0000 7f3df2ffd700 1 -- 172.21.15.67:0/1474741466 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f3df400db20 con 0x7f3e040fbce0 2024-04-12T05:31:41.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.895+0000 7f3df2ffd700 1 --2- 172.21.15.67:0/1474741466 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f3dec069210 0x7f3dec06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:31:41.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.895+0000 7f3df2ffd700 1 -- 172.21.15.67:0/1474741466 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 3228+0+0 (secure 0 0 0) 0x7f3df40864e0 con 0x7f3e040fbce0 2024-04-12T05:31:41.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.895+0000 7f3e01d9b700 1 --2- 172.21.15.67:0/1474741466 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f3dec069210 0x7f3dec06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:41.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.896+0000 7f3e01d9b700 1 --2- 172.21.15.67:0/1474741466 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f3dec069210 0x7f3dec06b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3df800b900 tx=0x7f3df8009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:31:41.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:41.900+0000 7f3df2ffd700 1 -- 172.21.15.67:0/1474741466 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f3df4056bc0 con 0x7f3e040fbce0 2024-04-12T05:31:42.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:41 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-12T05:31:42.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:41 smithi067 ceph-mon[49010]: osdmap e21: 3 total, 3 up, 3 in 2024-04-12T05:31:42.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:41 smithi067 ceph-mon[49010]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:31:42.092 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 05:31:41 smithi032 sudo[84831]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-12T05:31:42.092 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 05:31:41 smithi032 sudo[84831]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-12T05:31:42.092 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 05:31:41 smithi032 sudo[84831]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-12T05:31:42.092 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 05:31:41 smithi032 sudo[84831]: pam_unix(sudo:session): session closed for user root 2024-04-12T05:31:42.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:42.104+0000 7f3e08f18700 1 -- 172.21.15.67:0/1474741466 --> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e0410e780 con 0x7f3dec069210 2024-04-12T05:31:42.579 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:31:42 smithi032 sudo[86939]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-12T05:31:42.579 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:31:42 smithi032 sudo[86939]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-12T05:31:42.579 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:31:42 smithi032 sudo[86939]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-12T05:31:42.579 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:31:42 smithi032 sudo[86939]: pam_unix(sudo:session): session closed for user root 2024-04-12T05:31:42.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:31:42.584+0000 7f3df2ffd700 1 -- 172.21.15.67:0/1474741466 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f3df4053500 con 0x7f3e040fbce0 2024-04-12T05:31:42.842 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:31:42 smithi032 sudo[88757]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-12T05:31:42.842 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:31:42 smithi032 sudo[88757]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-12T05:31:42.842 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:31:42 smithi032 sudo[88757]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-12T05:31:42.842 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:31:42 smithi032 sudo[88757]: pam_unix(sudo:session): session closed for user root 2024-04-12T05:31:42.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:42 smithi032 ceph-mon[39731]: osdmap e22: 3 total, 3 up, 3 in 2024-04-12T05:31:42.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:42 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:42.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:42 smithi032 ceph-mon[39731]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:31:42.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:42 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T05:31:42.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:42 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T05:31:42.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:42 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:31:42.868 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:42 smithi067 ceph-mon[49010]: osdmap e22: 3 total, 3 up, 3 in 2024-04-12T05:31:42.868 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:42 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:42.868 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:42 smithi067 ceph-mon[49010]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:31:42.868 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:42 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T05:31:42.868 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:42 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T05:31:42.868 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:42 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:31:43.586 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:31:43 smithi032 sudo[88768]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-12T05:31:43.586 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:31:43 smithi032 sudo[88768]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-12T05:31:43.586 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:31:43 smithi032 sudo[88768]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-12T05:31:43.586 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:31:43 smithi032 sudo[88768]: pam_unix(sudo:session): session closed for user root 2024-04-12T05:31:43.842 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:31:43 smithi032 sudo[88894]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-12T05:31:43.842 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:31:43 smithi032 sudo[88894]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-12T05:31:43.842 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:31:43 smithi032 sudo[88894]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-12T05:31:43.842 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:31:43 smithi032 sudo[88894]: pam_unix(sudo:session): session closed for user root 2024-04-12T05:31:43.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:43 smithi032 ceph-mon[39731]: mgrmap e19: smithi032.hrwxrh(active, since 2m), standbys: smithi067.dwichg 2024-04-12T05:31:43.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:43 smithi032 ceph-mon[39731]: pgmap v79: 1 pgs: 1 active+clean; 449 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:31:43.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:43 smithi032 ceph-mon[39731]: Detected new or changed devices on smithi032 2024-04-12T05:31:43.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:43 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:43.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:43 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:43.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:43 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:31:43.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:43 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:31:43.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:43 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:31:43.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:43 smithi032 ceph-mon[39731]: Adjusting osd_memory_target on smithi032 to 4300M 2024-04-12T05:31:43.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:43 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:43.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:43 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:31:43.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:43 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:31:43.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:43 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:43.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:43 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:31:43.844 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:43 smithi067 ceph-mon[49010]: mgrmap e19: smithi032.hrwxrh(active, since 2m), standbys: smithi067.dwichg 2024-04-12T05:31:43.844 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:43 smithi067 ceph-mon[49010]: pgmap v79: 1 pgs: 1 active+clean; 449 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:31:43.844 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:43 smithi067 ceph-mon[49010]: Detected new or changed devices on smithi032 2024-04-12T05:31:43.844 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:43 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:43.844 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:43 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:43.844 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:43 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:31:43.844 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:43 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:31:43.844 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:43 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:31:43.844 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:43 smithi067 ceph-mon[49010]: Adjusting osd_memory_target on smithi032 to 4300M 2024-04-12T05:31:43.844 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:43 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:43.844 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:43 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:31:43.844 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:43 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:31:43.845 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:43 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:43.845 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:43 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:31:44.569 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:44 smithi032 sudo[88988]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-12T05:31:44.569 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:44 smithi032 sudo[88988]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-12T05:31:44.569 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:44 smithi032 sudo[88988]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-12T05:31:44.570 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:44 smithi032 sudo[88988]: pam_unix(sudo:session): session closed for user root 2024-04-12T05:31:44.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:44 smithi032 sudo[89027]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-12T05:31:44.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:44 smithi032 sudo[89027]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-12T05:31:44.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:44 smithi032 sudo[89027]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-12T05:31:44.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:44 smithi032 sudo[89027]: pam_unix(sudo:session): session closed for user root 2024-04-12T05:31:45.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:45 smithi067 sudo[53035]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-12T05:31:45.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:45 smithi067 sudo[53035]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-12T05:31:45.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:45 smithi067 sudo[53035]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-12T05:31:45.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:45 smithi067 sudo[53035]: pam_unix(sudo:session): session closed for user root 2024-04-12T05:31:45.763 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:45 smithi032 ceph-mon[39731]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-12T05:31:45.763 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:45 smithi032 ceph-mon[39731]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-12T05:31:45.763 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:45 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-12T05:31:45.763 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:45 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-12T05:31:45.763 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:45 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-12T05:31:45.763 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:45 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-12T05:31:45.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:45 smithi067 sudo[53038]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-12T05:31:45.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:45 smithi067 sudo[53038]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-12T05:31:45.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:45 smithi067 sudo[53038]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-12T05:31:45.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:45 smithi067 sudo[53038]: pam_unix(sudo:session): session closed for user root 2024-04-12T05:31:45.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:45 smithi067 ceph-mon[49010]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-12T05:31:45.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:45 smithi067 ceph-mon[49010]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-12T05:31:45.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:45 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-12T05:31:45.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:45 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-12T05:31:45.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:45 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-12T05:31:45.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:45 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-12T05:31:46.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:46 smithi067 ceph-mon[49010]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-12T05:31:46.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:46 smithi067 ceph-mon[49010]: pgmap v80: 1 pgs: 1 active+clean; 449 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:31:46.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:46 smithi067 ceph-mon[49010]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-12T05:31:46.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:46 smithi067 ceph-mon[49010]: from='client.? 172.21.15.67:0/3680403965' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "213dc62e-a894-4bc7-a2dd-ca98d1313d6c"}]: dispatch 2024-04-12T05:31:46.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:46 smithi067 ceph-mon[49010]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "213dc62e-a894-4bc7-a2dd-ca98d1313d6c"}]: dispatch 2024-04-12T05:31:46.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:46 smithi067 ceph-mon[49010]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "213dc62e-a894-4bc7-a2dd-ca98d1313d6c"}]': finished 2024-04-12T05:31:46.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:46 smithi067 ceph-mon[49010]: osdmap e23: 4 total, 3 up, 4 in 2024-04-12T05:31:46.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:46 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:31:46.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:46 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:46.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:46 smithi067 ceph-mon[49010]: from='client.? 172.21.15.67:0/155063167' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T05:31:46.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:46 smithi032 ceph-mon[39731]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-12T05:31:46.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:46 smithi032 ceph-mon[39731]: pgmap v80: 1 pgs: 1 active+clean; 449 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:31:46.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:46 smithi032 ceph-mon[39731]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-12T05:31:46.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:46 smithi032 ceph-mon[39731]: from='client.? 172.21.15.67:0/3680403965' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "213dc62e-a894-4bc7-a2dd-ca98d1313d6c"}]: dispatch 2024-04-12T05:31:46.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:46 smithi032 ceph-mon[39731]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "213dc62e-a894-4bc7-a2dd-ca98d1313d6c"}]: dispatch 2024-04-12T05:31:46.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:46 smithi032 ceph-mon[39731]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "213dc62e-a894-4bc7-a2dd-ca98d1313d6c"}]': finished 2024-04-12T05:31:46.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:46 smithi032 ceph-mon[39731]: osdmap e23: 4 total, 3 up, 4 in 2024-04-12T05:31:46.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:46 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:31:46.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:46 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:46.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:46 smithi032 ceph-mon[39731]: from='client.? 172.21.15.67:0/155063167' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T05:31:47.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:47 smithi067 ceph-mon[49010]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.9 KiB/s rd, 237 KiB/s wr, 16 op/s 2024-04-12T05:31:47.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:47 smithi032 ceph-mon[39731]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.9 KiB/s rd, 237 KiB/s wr, 16 op/s 2024-04-12T05:31:49.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:49.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:49.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:31:49.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:31:49.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:49.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:48 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:49.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:48 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:49.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:48 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:31:49.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:48 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:31:49.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:48 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:50.003 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:49 smithi067 ceph-mon[49010]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.6 KiB/s rd, 174 KiB/s wr, 12 op/s 2024-04-12T05:31:50.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:49 smithi032 ceph-mon[39731]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.6 KiB/s rd, 174 KiB/s wr, 12 op/s 2024-04-12T05:31:52.468 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:52 smithi067 ceph-mon[49010]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.8 KiB/s rd, 137 KiB/s wr, 9 op/s 2024-04-12T05:31:52.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:52 smithi032 ceph-mon[39731]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.8 KiB/s rd, 137 KiB/s wr, 9 op/s 2024-04-12T05:31:53.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:53 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-12T05:31:53.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:53 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:31:53.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:53 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-12T05:31:53.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:53 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:31:54.162 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:54 smithi067 ceph-mon[49010]: Deploying daemon osd.3 on smithi067 2024-04-12T05:31:54.163 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:54 smithi067 ceph-mon[49010]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-04-12T05:31:54.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:54 smithi032 ceph-mon[39731]: Deploying daemon osd.3 on smithi067 2024-04-12T05:31:54.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:54 smithi032 ceph-mon[39731]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-04-12T05:31:55.361 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:55 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:31:55.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:55 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:31:56.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:56 smithi067 ceph-mon[49010]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-04-12T05:31:56.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:56 smithi032 ceph-mon[39731]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-04-12T05:31:58.420 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:58 smithi067 ceph-mon[49010]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.3 KiB/s rd, 112 KiB/s wr, 7 op/s 2024-04-12T05:31:58.420 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:58 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:58.420 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:58 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:58.420 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:31:58 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:31:58.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:58 smithi032 ceph-mon[39731]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.3 KiB/s rd, 112 KiB/s wr, 7 op/s 2024-04-12T05:31:58.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:58 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:58.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:58 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:31:58.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:31:58 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:32:00.503 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:00 smithi067 ceph-mon[49010]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:32:00.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:00 smithi032 ceph-mon[39731]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:32:02.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:02.072+0000 7f3df2ffd700 1 -- 172.21.15.67:0/1474741466 <== mgr.14208 v2:172.21.15.32:6800/595990262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3e0410e780 con 0x7f3dec069210 2024-04-12T05:32:02.074 INFO:teuthology.orchestra.run.smithi067.stdout:Created osd(s) 3 on host 'smithi067' 2024-04-12T05:32:02.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:02.075+0000 7f3e08f18700 1 -- 172.21.15.67:0/1474741466 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f3dec069210 msgr2=0x7f3dec06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:02.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:02.075+0000 7f3e08f18700 1 --2- 172.21.15.67:0/1474741466 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f3dec069210 0x7f3dec06b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3df800b900 tx=0x7f3df8009e30 comp rx=0 tx=0).stop 2024-04-12T05:32:02.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:02.075+0000 7f3e08f18700 1 -- 172.21.15.67:0/1474741466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e040fbce0 msgr2=0x7f3e0410c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:02.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:02.075+0000 7f3e08f18700 1 --2- 172.21.15.67:0/1474741466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e040fbce0 0x7f3e0410c850 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f3df4002a50 tx=0x7f3df40046b0 comp rx=0 tx=0).stop 2024-04-12T05:32:02.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:02.075+0000 7f3e08f18700 1 -- 172.21.15.67:0/1474741466 shutdown_connections 2024-04-12T05:32:02.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:02.075+0000 7f3e08f18700 1 --2- 172.21.15.67:0/1474741466 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f3dec069210 0x7f3dec06b6c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:32:02.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:02.075+0000 7f3e08f18700 1 --2- 172.21.15.67:0/1474741466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e040fbce0 0x7f3e0410c850 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:32:02.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:02.075+0000 7f3e08f18700 1 --2- 172.21.15.67:0/1474741466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e040fcf90 0x7f3e0410cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:32:02.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:02.075+0000 7f3e08f18700 1 -- 172.21.15.67:0/1474741466 >> 172.21.15.67:0/1474741466 conn(0x7f3e040f7440 msgr2=0x7f3e040f8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:32:02.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:02.075+0000 7f3e08f18700 1 -- 172.21.15.67:0/1474741466 shutdown_connections 2024-04-12T05:32:02.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:02.075+0000 7f3e08f18700 1 -- 172.21.15.67:0/1474741466 wait complete. 2024-04-12T05:32:02.471 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:02 smithi067 ceph-mon[49010]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:32:02.472 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:02 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:02.472 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:02 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:02.472 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:02 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:32:02.472 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:02 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:32:02.472 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:02 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:02.472 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:02 smithi067 ceph-mon[49010]: from='osd.3 [v2:172.21.15.67:6800/1205580917,v1:172.21.15.67:6801/1205580917]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T05:32:02.472 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:02 smithi067 ceph-mon[49010]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T05:32:02.472 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:02 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:02.472 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:02 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:02.472 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:02 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:32:02.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:02 smithi032 ceph-mon[39731]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:32:02.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:02 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:02.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:02 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:02.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:02 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:32:02.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:02 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:32:02.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:02 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:02.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:02 smithi032 ceph-mon[39731]: from='osd.3 [v2:172.21.15.67:6800/1205580917,v1:172.21.15.67:6801/1205580917]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T05:32:02.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:02 smithi032 ceph-mon[39731]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T05:32:02.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:02 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:02.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:02 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:02.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:02 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:32:02.821 DEBUG:teuthology.orchestra.run.smithi067:osd.3> sudo journalctl -f -n 0 -u ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@osd.3.service 2024-04-12T05:32:02.823 INFO:tasks.cephadm:Deploying osd.4 on smithi067 with /dev/vg_nvme/lv_3... 2024-04-12T05:32:02.823 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-12T05:32:02.854 INFO:journalctl@ceph.osd.3.smithi067.stdout:-- Logs begin at Fri 2024-04-12 05:14:32 UTC. -- 2024-04-12T05:32:03.577 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi067/config 2024-04-12T05:32:03.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:03 smithi067 ceph-mon[49010]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-12T05:32:03.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:03 smithi067 ceph-mon[49010]: osdmap e24: 4 total, 3 up, 4 in 2024-04-12T05:32:03.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:03 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:32:03.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:03 smithi067 ceph-mon[49010]: from='osd.3 [v2:172.21.15.67:6800/1205580917,v1:172.21.15.67:6801/1205580917]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-12T05:32:03.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:03 smithi067 ceph-mon[49010]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-12T05:32:03.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:03 smithi067 ceph-mon[49010]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:32:03.771 INFO:journalctl@ceph.osd.3.smithi067.stdout:Apr 12 05:32:03 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-3[55347]: 2024-04-12T05:32:03.431+0000 7f4f92de1700 -1 osd.3 0 waiting for initial osdmap 2024-04-12T05:32:03.772 INFO:journalctl@ceph.osd.3.smithi067.stdout:Apr 12 05:32:03 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-3[55347]: 2024-04-12T05:32:03.439+0000 7f4f8dbdc700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T05:32:03.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:03 smithi032 ceph-mon[39731]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-12T05:32:03.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:03 smithi032 ceph-mon[39731]: osdmap e24: 4 total, 3 up, 4 in 2024-04-12T05:32:03.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:03 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:32:03.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:03 smithi032 ceph-mon[39731]: from='osd.3 [v2:172.21.15.67:6800/1205580917,v1:172.21.15.67:6801/1205580917]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-12T05:32:03.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:03 smithi032 ceph-mon[39731]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-12T05:32:03.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:03 smithi032 ceph-mon[39731]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:32:04.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:04 smithi067 ceph-mon[49010]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-04-12T05:32:04.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:04 smithi067 ceph-mon[49010]: osdmap e25: 4 total, 3 up, 4 in 2024-04-12T05:32:04.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:04 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:32:04.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:04 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:32:04.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:04 smithi032 ceph-mon[39731]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-04-12T05:32:04.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:04 smithi032 ceph-mon[39731]: osdmap e25: 4 total, 3 up, 4 in 2024-04-12T05:32:04.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:04 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:32:04.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:04 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:32:05.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:05 smithi067 ceph-mon[49010]: purged_snaps scrub starts 2024-04-12T05:32:05.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:05 smithi067 ceph-mon[49010]: purged_snaps scrub ok 2024-04-12T05:32:05.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:05 smithi067 ceph-mon[49010]: osd.3 [v2:172.21.15.67:6800/1205580917,v1:172.21.15.67:6801/1205580917] boot 2024-04-12T05:32:05.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:05 smithi067 ceph-mon[49010]: osdmap e26: 4 total, 4 up, 4 in 2024-04-12T05:32:05.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:05 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:32:05.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:05 smithi067 ceph-mon[49010]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:32:05.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:05 smithi032 ceph-mon[39731]: purged_snaps scrub starts 2024-04-12T05:32:05.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:05 smithi032 ceph-mon[39731]: purged_snaps scrub ok 2024-04-12T05:32:05.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:05 smithi032 ceph-mon[39731]: osd.3 [v2:172.21.15.67:6800/1205580917,v1:172.21.15.67:6801/1205580917] boot 2024-04-12T05:32:05.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:05 smithi032 ceph-mon[39731]: osdmap e26: 4 total, 4 up, 4 in 2024-04-12T05:32:05.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:05 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:32:05.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:05 smithi032 ceph-mon[39731]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:32:06.320 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-12T05:32:06.320 INFO:teuthology.orchestra.run.smithi067.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-12T05:32:06.320 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10+0 records in 2024-04-12T05:32:06.320 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records out 2024-04-12T05:32:06.320 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0108767 s, 964 MB/s 2024-04-12T05:32:06.320 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping successful for: 2024-04-12T05:32:06.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:06 smithi067 ceph-mon[49010]: osdmap e27: 4 total, 4 up, 4 in 2024-04-12T05:32:06.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:06 smithi032 ceph-mon[39731]: osdmap e27: 4 total, 4 up, 4 in 2024-04-12T05:32:07.473 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph orch daemon add osd smithi067:vg_nvme/lv_3 2024-04-12T05:32:07.750 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi067/config 2024-04-12T05:32:07.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:07 smithi067 ceph-mon[49010]: osdmap e28: 4 total, 4 up, 4 in 2024-04-12T05:32:07.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:07 smithi067 ceph-mon[49010]: Detected new or changed devices on smithi067 2024-04-12T05:32:07.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:07 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:07.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:07 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:07.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:07 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:32:07.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:07 smithi067 ceph-mon[49010]: Adjusting osd_memory_target on smithi067 to 15970M 2024-04-12T05:32:07.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:07 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:07.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:07 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:32:07.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:07 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:32:07.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:07 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:07.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:07 smithi067 ceph-mon[49010]: pgmap v97: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-12T05:32:07.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:07 smithi032 ceph-mon[39731]: osdmap e28: 4 total, 4 up, 4 in 2024-04-12T05:32:07.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:07 smithi032 ceph-mon[39731]: Detected new or changed devices on smithi067 2024-04-12T05:32:07.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:07 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:07.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:07 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:07.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:07 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:32:07.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:07 smithi032 ceph-mon[39731]: Adjusting osd_memory_target on smithi067 to 15970M 2024-04-12T05:32:07.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:07 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:07.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:07 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:32:07.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:07 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:32:07.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:07 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:07.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:07 smithi032 ceph-mon[39731]: pgmap v97: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-12T05:32:08.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:08 smithi067 ceph-mon[49010]: osdmap e29: 4 total, 4 up, 4 in 2024-04-12T05:32:08.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:08 smithi032 ceph-mon[39731]: osdmap e29: 4 total, 4 up, 4 in 2024-04-12T05:32:09.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.570+0000 7fbae27cb700 1 -- 172.21.15.67:0/4121889387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbadc0fcf80 msgr2=0x7fbadc0fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:09.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.570+0000 7fbae27cb700 1 --2- 172.21.15.67:0/4121889387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbadc0fcf80 0x7fbadc0fd3f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbad0004940 tx=0x7fbad0009e30 comp rx=0 tx=0).stop 2024-04-12T05:32:09.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.570+0000 7fbae27cb700 1 -- 172.21.15.67:0/4121889387 shutdown_connections 2024-04-12T05:32:09.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.570+0000 7fbae27cb700 1 --2- 172.21.15.67:0/4121889387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbadc0fcf80 0x7fbadc0fd3f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:32:09.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.570+0000 7fbae27cb700 1 --2- 172.21.15.67:0/4121889387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbadc0fbcd0 0x7fbadc0fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:32:09.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.570+0000 7fbae27cb700 1 -- 172.21.15.67:0/4121889387 >> 172.21.15.67:0/4121889387 conn(0x7fbadc0f7430 msgr2=0x7fbadc0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:32:09.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.570+0000 7fbae27cb700 1 -- 172.21.15.67:0/4121889387 shutdown_connections 2024-04-12T05:32:09.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.570+0000 7fbae27cb700 1 -- 172.21.15.67:0/4121889387 wait complete. 2024-04-12T05:32:09.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.570+0000 7fbae27cb700 1 Processor -- start 2024-04-12T05:32:09.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.571+0000 7fbae27cb700 1 -- start start 2024-04-12T05:32:09.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.571+0000 7fbae27cb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbadc0fbcd0 0x7fbadc10c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:32:09.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.571+0000 7fbae27cb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbadc0fcf80 0x7fbadc10ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:32:09.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.571+0000 7fbae27cb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbadc10d2c0 con 0x7fbadc0fbcd0 2024-04-12T05:32:09.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.571+0000 7fbae27cb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbadc10db10 con 0x7fbadc0fcf80 2024-04-12T05:32:09.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.571+0000 7fbadb7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbadc0fcf80 0x7fbadc10ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:09.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.571+0000 7fbadb7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbadc0fcf80 0x7fbadc10ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57132/0 (socket says 172.21.15.67:57132) 2024-04-12T05:32:09.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.571+0000 7fbadb7fe700 1 -- 172.21.15.67:0/2778847524 learned_addr learned my addr 172.21.15.67:0/2778847524 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-12T05:32:09.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.571+0000 7fbadbfff700 1 --2- 172.21.15.67:0/2778847524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbadc0fbcd0 0x7fbadc10c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:09.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.571+0000 7fbadb7fe700 1 -- 172.21.15.67:0/2778847524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbadc0fbcd0 msgr2=0x7fbadc10c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:09.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.571+0000 7fbadb7fe700 1 --2- 172.21.15.67:0/2778847524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbadc0fbcd0 0x7fbadc10c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:32:09.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.571+0000 7fbadb7fe700 1 -- 172.21.15.67:0/2778847524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad0005040 con 0x7fbadc0fcf80 2024-04-12T05:32:09.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.571+0000 7fbadb7fe700 1 --2- 172.21.15.67:0/2778847524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbadc0fcf80 0x7fbadc10ccc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fbad000b0d0 tx=0x7fbad0009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:32:09.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.572+0000 7fbad8ff9700 1 -- 172.21.15.67:0/2778847524 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbad000e070 con 0x7fbadc0fcf80 2024-04-12T05:32:09.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.572+0000 7fbad8ff9700 1 -- 172.21.15.67:0/2778847524 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbad000b410 con 0x7fbadc0fcf80 2024-04-12T05:32:09.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.572+0000 7fbae27cb700 1 -- 172.21.15.67:0/2778847524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbadc10dd90 con 0x7fbadc0fcf80 2024-04-12T05:32:09.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.572+0000 7fbad8ff9700 1 -- 172.21.15.67:0/2778847524 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbad0012600 con 0x7fbadc0fcf80 2024-04-12T05:32:09.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.572+0000 7fbae27cb700 1 -- 172.21.15.67:0/2778847524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbadc10e2d0 con 0x7fbadc0fcf80 2024-04-12T05:32:09.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.573+0000 7fbae27cb700 1 -- 172.21.15.67:0/2778847524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbadc05f5f0 con 0x7fbadc0fcf80 2024-04-12T05:32:09.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.573+0000 7fbad8ff9700 1 -- 172.21.15.67:0/2778847524 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fbad000b9f0 con 0x7fbadc0fcf80 2024-04-12T05:32:09.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.574+0000 7fbad8ff9700 1 --2- 172.21.15.67:0/2778847524 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fbac40692b0 0x7fbac406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:32:09.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.574+0000 7fbad8ff9700 1 -- 172.21.15.67:0/2778847524 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3770+0+0 (secure 0 0 0) 0x7fbad00877c0 con 0x7fbadc0fcf80 2024-04-12T05:32:09.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.574+0000 7fbadbfff700 1 --2- 172.21.15.67:0/2778847524 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fbac40692b0 0x7fbac406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:09.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.574+0000 7fbadbfff700 1 --2- 172.21.15.67:0/2778847524 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fbac40692b0 0x7fbac406b760 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbacc009b80 tx=0x7fbacc00b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:32:09.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.578+0000 7fbad8ff9700 1 -- 172.21.15.67:0/2778847524 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fbad0057cd0 con 0x7fbadc0fcf80 2024-04-12T05:32:09.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:09 smithi067 ceph-mon[49010]: osdmap e30: 4 total, 4 up, 4 in 2024-04-12T05:32:09.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:09 smithi067 ceph-mon[49010]: pgmap v100: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-12T05:32:09.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:09.781+0000 7fbae27cb700 1 -- 172.21.15.67:0/2778847524 --> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fbadc10e680 con 0x7fbac40692b0 2024-04-12T05:32:09.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:09 smithi032 ceph-mon[39731]: osdmap e30: 4 total, 4 up, 4 in 2024-04-12T05:32:09.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:09 smithi032 ceph-mon[39731]: pgmap v100: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-12T05:32:10.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:10 smithi067 ceph-mon[49010]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:32:10.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:10 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T05:32:10.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:10 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T05:32:10.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:10 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:32:10.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:10 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:32:10.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:10 smithi032 ceph-mon[39731]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:32:10.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:10 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T05:32:10.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:10 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T05:32:10.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:10 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:32:10.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:10 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:32:11.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:11 smithi067 ceph-mon[49010]: pgmap v101: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-12T05:32:11.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:11 smithi032 ceph-mon[39731]: pgmap v101: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-12T05:32:12.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:12 smithi067 ceph-mon[49010]: from='client.? 172.21.15.67:0/1524832349' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a0ff860-fff7-434a-9e78-f2020bc84966"}]: dispatch 2024-04-12T05:32:12.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:12 smithi067 ceph-mon[49010]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a0ff860-fff7-434a-9e78-f2020bc84966"}]: dispatch 2024-04-12T05:32:12.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:12 smithi067 ceph-mon[49010]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8a0ff860-fff7-434a-9e78-f2020bc84966"}]': finished 2024-04-12T05:32:12.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:12 smithi067 ceph-mon[49010]: osdmap e31: 5 total, 4 up, 5 in 2024-04-12T05:32:12.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:12 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:32:12.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:12 smithi032 ceph-mon[39731]: from='client.? 172.21.15.67:0/1524832349' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a0ff860-fff7-434a-9e78-f2020bc84966"}]: dispatch 2024-04-12T05:32:12.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:12 smithi032 ceph-mon[39731]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a0ff860-fff7-434a-9e78-f2020bc84966"}]: dispatch 2024-04-12T05:32:12.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:12 smithi032 ceph-mon[39731]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8a0ff860-fff7-434a-9e78-f2020bc84966"}]': finished 2024-04-12T05:32:12.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:12 smithi032 ceph-mon[39731]: osdmap e31: 5 total, 4 up, 5 in 2024-04-12T05:32:12.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:12 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:32:13.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:13 smithi067 ceph-mon[49010]: from='client.? 172.21.15.67:0/1579950949' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T05:32:13.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:13 smithi067 ceph-mon[49010]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:32:13.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:13 smithi032 ceph-mon[39731]: from='client.? 172.21.15.67:0/1579950949' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T05:32:13.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:13 smithi032 ceph-mon[39731]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:32:16.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:16 smithi067 ceph-mon[49010]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:32:16.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:16 smithi032 ceph-mon[39731]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:32:18.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:18 smithi067 ceph-mon[49010]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:32:18.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:18 smithi032 ceph-mon[39731]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:32:20.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:20 smithi067 ceph-mon[49010]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:32:20.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:20 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-12T05:32:20.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:20 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:32:20.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:20 smithi032 ceph-mon[39731]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:32:20.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:20 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-12T05:32:20.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:20 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:32:21.405 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:21 smithi067 ceph-mon[49010]: Deploying daemon osd.4 on smithi067 2024-04-12T05:32:21.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:21 smithi032 ceph-mon[39731]: Deploying daemon osd.4 on smithi067 2024-04-12T05:32:22.464 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:22 smithi067 ceph-mon[49010]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:32:22.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:22 smithi032 ceph-mon[39731]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:32:24.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:24 smithi067 ceph-mon[49010]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:32:24.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:24 smithi032 ceph-mon[39731]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:32:25.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:25 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:32:25.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:25 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:32:26.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:26 smithi067 ceph-mon[49010]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:32:26.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:26 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:26.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:26 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:26.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:26 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:32:26.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:26 smithi032 ceph-mon[39731]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:32:26.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:26 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:26.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:26 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:26.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:26 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:32:28.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:28 smithi067 ceph-mon[49010]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:32:28.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:28 smithi032 ceph-mon[39731]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:32:29.307 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:29 smithi067 ceph-mon[49010]: from='osd.4 [v2:172.21.15.67:6808/286946785,v1:172.21.15.67:6809/286946785]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T05:32:29.307 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:29 smithi067 ceph-mon[49010]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T05:32:29.352 INFO:teuthology.orchestra.run.smithi067.stdout:Created osd(s) 4 on host 'smithi067' 2024-04-12T05:32:29.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:29.352+0000 7fbad8ff9700 1 -- 172.21.15.67:0/2778847524 <== mgr.14208 v2:172.21.15.32:6800/595990262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fbadc10e680 con 0x7fbac40692b0 2024-04-12T05:32:29.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:29.354+0000 7fbae27cb700 1 -- 172.21.15.67:0/2778847524 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fbac40692b0 msgr2=0x7fbac406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:29.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:29.354+0000 7fbae27cb700 1 --2- 172.21.15.67:0/2778847524 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fbac40692b0 0x7fbac406b760 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbacc009b80 tx=0x7fbacc00b040 comp rx=0 tx=0).stop 2024-04-12T05:32:29.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:29.354+0000 7fbae27cb700 1 -- 172.21.15.67:0/2778847524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbadc0fcf80 msgr2=0x7fbadc10ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:29.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:29.354+0000 7fbae27cb700 1 --2- 172.21.15.67:0/2778847524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbadc0fcf80 0x7fbadc10ccc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fbad000b0d0 tx=0x7fbad0009e30 comp rx=0 tx=0).stop 2024-04-12T05:32:29.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:29.354+0000 7fbae27cb700 1 -- 172.21.15.67:0/2778847524 shutdown_connections 2024-04-12T05:32:29.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:29.354+0000 7fbae27cb700 1 --2- 172.21.15.67:0/2778847524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbadc0fbcd0 0x7fbadc10c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:32:29.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:29.354+0000 7fbae27cb700 1 --2- 172.21.15.67:0/2778847524 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fbac40692b0 0x7fbac406b760 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:32:29.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:29.354+0000 7fbae27cb700 1 --2- 172.21.15.67:0/2778847524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbadc0fcf80 0x7fbadc10ccc0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:32:29.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:29.354+0000 7fbae27cb700 1 -- 172.21.15.67:0/2778847524 >> 172.21.15.67:0/2778847524 conn(0x7fbadc0f7430 msgr2=0x7fbadc0f8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:32:29.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:29.354+0000 7fbae27cb700 1 -- 172.21.15.67:0/2778847524 shutdown_connections 2024-04-12T05:32:29.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:29.354+0000 7fbae27cb700 1 -- 172.21.15.67:0/2778847524 wait complete. 2024-04-12T05:32:29.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:29 smithi032 ceph-mon[39731]: from='osd.4 [v2:172.21.15.67:6808/286946785,v1:172.21.15.67:6809/286946785]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T05:32:29.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:29 smithi032 ceph-mon[39731]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T05:32:30.021 DEBUG:teuthology.orchestra.run.smithi067:osd.4> sudo journalctl -f -n 0 -u ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@osd.4.service 2024-04-12T05:32:30.024 INFO:tasks.cephadm:Deploying osd.5 on smithi067 with /dev/vg_nvme/lv_2... 2024-04-12T05:32:30.024 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-12T05:32:30.054 INFO:journalctl@ceph.osd.4.smithi067.stdout:-- Logs begin at Fri 2024-04-12 05:14:32 UTC. -- 2024-04-12T05:32:30.256 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 05:32:30 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-4[63297]: 2024-04-12T05:32:30.247+0000 7f8b1c2ef700 -1 osd.4 0 waiting for initial osdmap 2024-04-12T05:32:30.256 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:30 smithi067 ceph-mon[49010]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:32:30.256 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:30 smithi067 ceph-mon[49010]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-12T05:32:30.256 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:30 smithi067 ceph-mon[49010]: osdmap e32: 5 total, 4 up, 5 in 2024-04-12T05:32:30.256 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:30 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:32:30.256 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:30 smithi067 ceph-mon[49010]: from='osd.4 [v2:172.21.15.67:6808/286946785,v1:172.21.15.67:6809/286946785]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-12T05:32:30.256 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:30 smithi067 ceph-mon[49010]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-12T05:32:30.256 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:30 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:30.256 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:30 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:30.256 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:30 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:30.257 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:30 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:30.521 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 05:32:30 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-4[63297]: 2024-04-12T05:32:30.255+0000 7f8b170ea700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T05:32:30.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:30 smithi032 ceph-mon[39731]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:32:30.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:30 smithi032 ceph-mon[39731]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-12T05:32:30.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:30 smithi032 ceph-mon[39731]: osdmap e32: 5 total, 4 up, 5 in 2024-04-12T05:32:30.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:30 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:32:30.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:30 smithi032 ceph-mon[39731]: from='osd.4 [v2:172.21.15.67:6808/286946785,v1:172.21.15.67:6809/286946785]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-12T05:32:30.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:30 smithi032 ceph-mon[39731]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-12T05:32:30.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:30 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:30.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:30 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:30.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:30 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:30.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:30 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:30.889 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi067/config 2024-04-12T05:32:31.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:31 smithi067 ceph-mon[49010]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-04-12T05:32:31.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:31 smithi067 ceph-mon[49010]: osdmap e33: 5 total, 4 up, 5 in 2024-04-12T05:32:31.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:31 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:32:31.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:31 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:32:31.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:31 smithi067 ceph-mon[49010]: osd.4 [v2:172.21.15.67:6808/286946785,v1:172.21.15.67:6809/286946785] boot 2024-04-12T05:32:31.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:31 smithi067 ceph-mon[49010]: osdmap e34: 5 total, 5 up, 5 in 2024-04-12T05:32:31.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:31 smithi032 ceph-mon[39731]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-04-12T05:32:31.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:31 smithi032 ceph-mon[39731]: osdmap e33: 5 total, 4 up, 5 in 2024-04-12T05:32:31.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:31 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:32:31.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:31 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:32:31.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:31 smithi032 ceph-mon[39731]: osd.4 [v2:172.21.15.67:6808/286946785,v1:172.21.15.67:6809/286946785] boot 2024-04-12T05:32:31.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:31 smithi032 ceph-mon[39731]: osdmap e34: 5 total, 5 up, 5 in 2024-04-12T05:32:32.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:32 smithi067 ceph-mon[49010]: purged_snaps scrub starts 2024-04-12T05:32:32.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:32 smithi067 ceph-mon[49010]: purged_snaps scrub ok 2024-04-12T05:32:32.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:32 smithi067 ceph-mon[49010]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:32:32.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:32 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:32:32.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:32 smithi032 ceph-mon[39731]: purged_snaps scrub starts 2024-04-12T05:32:32.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:32 smithi032 ceph-mon[39731]: purged_snaps scrub ok 2024-04-12T05:32:32.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:32 smithi032 ceph-mon[39731]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:32:32.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:32 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:32:33.523 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:33 smithi067 ceph-mon[49010]: osdmap e35: 5 total, 5 up, 5 in 2024-04-12T05:32:33.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:33 smithi032 ceph-mon[39731]: osdmap e35: 5 total, 5 up, 5 in 2024-04-12T05:32:34.429 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-12T05:32:34.430 INFO:teuthology.orchestra.run.smithi067.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-12T05:32:34.430 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10+0 records in 2024-04-12T05:32:34.430 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records out 2024-04-12T05:32:34.430 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0108652 s, 965 MB/s 2024-04-12T05:32:34.430 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping successful for: 2024-04-12T05:32:34.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:34 smithi067 ceph-mon[49010]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:32:34.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:34 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:34.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:34 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:34.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:34 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:32:34.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:34 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:32:34.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:34 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:34.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:34 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:32:34.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:34 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:32:34.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:34 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:34.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:34 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:32:34.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:34 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:32:34.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:34 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:32:34.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:34 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:34.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:34 smithi032 ceph-mon[39731]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:32:34.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:34 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:34.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:34 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:34.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:34 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:32:34.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:34 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:32:34.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:34 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:34.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:34 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:32:34.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:34 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:32:34.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:34 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:34.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:34 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:32:34.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:34 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:32:34.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:34 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:32:34.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:34 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:35.059 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph orch daemon add osd smithi067:vg_nvme/lv_2 2024-04-12T05:32:35.336 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi067/config 2024-04-12T05:32:35.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:35 smithi067 ceph-mon[49010]: Detected new or changed devices on smithi067 2024-04-12T05:32:35.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:35 smithi067 ceph-mon[49010]: Adjusting osd_memory_target on smithi067 to 7985M 2024-04-12T05:32:35.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:35 smithi032 ceph-mon[39731]: Detected new or changed devices on smithi067 2024-04-12T05:32:35.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:35 smithi032 ceph-mon[39731]: Adjusting osd_memory_target on smithi067 to 7985M 2024-04-12T05:32:36.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:36 smithi067 ceph-mon[49010]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:32:36.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:36 smithi032 ceph-mon[39731]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:32:37.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.018+0000 7f7148813700 1 -- 172.21.15.67:0/944438040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71440662e0 msgr2=0x7f71440ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:37.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.018+0000 7f7148813700 1 --2- 172.21.15.67:0/944438040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71440662e0 0x7f71440ff9c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f712c004940 tx=0x7f712c009e30 comp rx=0 tx=0).stop 2024-04-12T05:32:37.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.018+0000 7f7148813700 1 -- 172.21.15.67:0/944438040 shutdown_connections 2024-04-12T05:32:37.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.018+0000 7f7148813700 1 --2- 172.21.15.67:0/944438040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71440fff00 0x7f7144102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:32:37.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.018+0000 7f7148813700 1 --2- 172.21.15.67:0/944438040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71440662e0 0x7f71440ff9c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:32:37.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.018+0000 7f7148813700 1 -- 172.21.15.67:0/944438040 >> 172.21.15.67:0/944438040 conn(0x7f71440f7430 msgr2=0x7f71440f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:32:37.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.018+0000 7f7148813700 1 -- 172.21.15.67:0/944438040 shutdown_connections 2024-04-12T05:32:37.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.018+0000 7f7148813700 1 -- 172.21.15.67:0/944438040 wait complete. 2024-04-12T05:32:37.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.019+0000 7f7148813700 1 Processor -- start 2024-04-12T05:32:37.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.019+0000 7f7148813700 1 -- start start 2024-04-12T05:32:37.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.019+0000 7f7148813700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71440662e0 0x7f714410d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:32:37.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.019+0000 7f7148813700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71440fff00 0x7f714410de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:32:37.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.019+0000 7f7148813700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714410e4f0 con 0x7f71440662e0 2024-04-12T05:32:37.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.019+0000 7f7148813700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714410ece0 con 0x7f71440fff00 2024-04-12T05:32:37.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.019+0000 7f7141d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71440fff00 0x7f714410de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:37.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.019+0000 7f7141d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71440fff00 0x7f714410de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37560/0 (socket says 172.21.15.67:37560) 2024-04-12T05:32:37.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.019+0000 7f7141d9b700 1 -- 172.21.15.67:0/4286882096 learned_addr learned my addr 172.21.15.67:0/4286882096 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-12T05:32:37.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.019+0000 7f714259c700 1 --2- 172.21.15.67:0/4286882096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71440662e0 0x7f714410d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:37.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.019+0000 7f7141d9b700 1 -- 172.21.15.67:0/4286882096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71440662e0 msgr2=0x7f714410d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:37.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.019+0000 7f7141d9b700 1 --2- 172.21.15.67:0/4286882096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71440662e0 0x7f714410d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:32:37.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.019+0000 7f7141d9b700 1 -- 172.21.15.67:0/4286882096 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f712c005040 con 0x7f71440fff00 2024-04-12T05:32:37.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.019+0000 7f714259c700 1 --2- 172.21.15.67:0/4286882096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71440662e0 0x7f714410d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:32:37.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.019+0000 7f7141d9b700 1 --2- 172.21.15.67:0/4286882096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71440fff00 0x7f714410de60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f713400b790 tx=0x7f7134009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:32:37.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.020+0000 7f713affd700 1 -- 172.21.15.67:0/4286882096 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f713400f770 con 0x7f71440fff00 2024-04-12T05:32:37.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.020+0000 7f7148813700 1 -- 172.21.15.67:0/4286882096 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f714410efc0 con 0x7f71440fff00 2024-04-12T05:32:37.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.020+0000 7f713affd700 1 -- 172.21.15.67:0/4286882096 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f713400fd80 con 0x7f71440fff00 2024-04-12T05:32:37.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.020+0000 7f713affd700 1 -- 172.21.15.67:0/4286882096 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f713400d310 con 0x7f71440fff00 2024-04-12T05:32:37.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.020+0000 7f7148813700 1 -- 172.21.15.67:0/4286882096 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f714410f580 con 0x7f71440fff00 2024-04-12T05:32:37.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.020+0000 7f7148813700 1 -- 172.21.15.67:0/4286882096 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f714405f5f0 con 0x7f71440fff00 2024-04-12T05:32:37.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.021+0000 7f713affd700 1 -- 172.21.15.67:0/4286882096 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f7134010420 con 0x7f71440fff00 2024-04-12T05:32:37.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.021+0000 7f713affd700 1 --2- 172.21.15.67:0/4286882096 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f71300692a0 0x7f713006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:32:37.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.021+0000 7f713affd700 1 -- 172.21.15.67:0/4286882096 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4202+0+0 (secure 0 0 0) 0x7f7134086c40 con 0x7f71440fff00 2024-04-12T05:32:37.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.022+0000 7f714259c700 1 --2- 172.21.15.67:0/4286882096 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f71300692a0 0x7f713006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:37.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.022+0000 7f714259c700 1 --2- 172.21.15.67:0/4286882096 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f71300692a0 0x7f713006b750 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f712c00b3b0 tx=0x7f712c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:32:37.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.026+0000 7f713affd700 1 -- 172.21.15.67:0/4286882096 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f7134056ef0 con 0x7f71440fff00 2024-04-12T05:32:37.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:37.229+0000 7f7148813700 1 -- 172.21.15.67:0/4286882096 --> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f714410f870 con 0x7f71300692a0 2024-04-12T05:32:37.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:37 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T05:32:37.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:37 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T05:32:37.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:37 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:32:37.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:37 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T05:32:37.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:37 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T05:32:37.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:37 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:32:38.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:38 smithi067 ceph-mon[49010]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:32:38.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:38 smithi067 ceph-mon[49010]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:32:38.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:38 smithi032 ceph-mon[39731]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:32:38.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:38 smithi032 ceph-mon[39731]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:32:40.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:40 smithi032 ceph-mon[39731]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:32:40.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:40 smithi032 ceph-mon[39731]: from='client.? 172.21.15.67:0/488104439' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3486e5c9-9f44-4f32-a186-fc6198f09549"}]: dispatch 2024-04-12T05:32:40.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:40 smithi032 ceph-mon[39731]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3486e5c9-9f44-4f32-a186-fc6198f09549"}]: dispatch 2024-04-12T05:32:40.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:40 smithi032 ceph-mon[39731]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3486e5c9-9f44-4f32-a186-fc6198f09549"}]': finished 2024-04-12T05:32:40.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:40 smithi032 ceph-mon[39731]: osdmap e36: 6 total, 5 up, 6 in 2024-04-12T05:32:40.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:40 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:32:40.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:40 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:32:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:40 smithi067 ceph-mon[49010]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:32:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:40 smithi067 ceph-mon[49010]: from='client.? 172.21.15.67:0/488104439' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3486e5c9-9f44-4f32-a186-fc6198f09549"}]: dispatch 2024-04-12T05:32:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:40 smithi067 ceph-mon[49010]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3486e5c9-9f44-4f32-a186-fc6198f09549"}]: dispatch 2024-04-12T05:32:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:40 smithi067 ceph-mon[49010]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3486e5c9-9f44-4f32-a186-fc6198f09549"}]': finished 2024-04-12T05:32:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:40 smithi067 ceph-mon[49010]: osdmap e36: 6 total, 5 up, 6 in 2024-04-12T05:32:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:40 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:32:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:40 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:32:41.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:41 smithi032 ceph-mon[39731]: from='client.? 172.21.15.67:0/2343845684' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T05:32:41.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:41 smithi067 ceph-mon[49010]: from='client.? 172.21.15.67:0/2343845684' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T05:32:42.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:42 smithi032 ceph-mon[39731]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:32:42.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:42 smithi067 ceph-mon[49010]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:32:44.347 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:44 smithi067 ceph-mon[49010]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:32:44.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:44 smithi032 ceph-mon[39731]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:32:46.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:46 smithi067 ceph-mon[49010]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:32:46.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:46 smithi032 ceph-mon[39731]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:32:48.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:48 smithi032 ceph-mon[39731]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:32:48.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:48 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-12T05:32:48.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:48 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:32:48.594 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:48 smithi067 ceph-mon[49010]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:32:48.594 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-12T05:32:48.594 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:48 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:32:49.524 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:49 smithi067 ceph-mon[49010]: Deploying daemon osd.5 on smithi067 2024-04-12T05:32:49.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:49 smithi032 ceph-mon[39731]: Deploying daemon osd.5 on smithi067 2024-04-12T05:32:50.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:50 smithi067 ceph-mon[49010]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:32:50.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:50 smithi032 ceph-mon[39731]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:32:52.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:52 smithi032 ceph-mon[39731]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:32:52.614 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:52 smithi067 ceph-mon[49010]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:32:54.020 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:53 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:54.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:53 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:54.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:53 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:32:54.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:53 smithi067 ceph-mon[49010]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:32:54.092 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:53 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:54.092 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:53 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:54.092 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:53 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:32:54.092 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:53 smithi032 ceph-mon[39731]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:32:55.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:55 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:32:55.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:55 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:32:56.471 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:56 smithi067 ceph-mon[49010]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:32:56.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:56.503+0000 7f713affd700 1 -- 172.21.15.67:0/4286882096 <== mgr.14208 v2:172.21.15.32:6800/595990262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f714410f870 con 0x7f71300692a0 2024-04-12T05:32:56.502 INFO:teuthology.orchestra.run.smithi067.stdout:Created osd(s) 5 on host 'smithi067' 2024-04-12T05:32:56.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:56.505+0000 7f7148813700 1 -- 172.21.15.67:0/4286882096 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f71300692a0 msgr2=0x7f713006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:56.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:56.505+0000 7f7148813700 1 --2- 172.21.15.67:0/4286882096 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f71300692a0 0x7f713006b750 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f712c00b3b0 tx=0x7f712c009e30 comp rx=0 tx=0).stop 2024-04-12T05:32:56.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:56.505+0000 7f7148813700 1 -- 172.21.15.67:0/4286882096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71440fff00 msgr2=0x7f714410de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:56.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:56.505+0000 7f7148813700 1 --2- 172.21.15.67:0/4286882096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71440fff00 0x7f714410de60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f713400b790 tx=0x7f7134009250 comp rx=0 tx=0).stop 2024-04-12T05:32:56.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:56.505+0000 7f7148813700 1 -- 172.21.15.67:0/4286882096 shutdown_connections 2024-04-12T05:32:56.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:56.505+0000 7f7148813700 1 --2- 172.21.15.67:0/4286882096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71440662e0 0x7f714410d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:32:56.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:56.505+0000 7f7148813700 1 --2- 172.21.15.67:0/4286882096 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f71300692a0 0x7f713006b750 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:32:56.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:56.505+0000 7f7148813700 1 --2- 172.21.15.67:0/4286882096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71440fff00 0x7f714410de60 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:32:56.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:56.505+0000 7f7148813700 1 -- 172.21.15.67:0/4286882096 >> 172.21.15.67:0/4286882096 conn(0x7f71440f7430 msgr2=0x7f71440f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:32:56.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:56.505+0000 7f7148813700 1 -- 172.21.15.67:0/4286882096 shutdown_connections 2024-04-12T05:32:56.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:32:56.505+0000 7f7148813700 1 -- 172.21.15.67:0/4286882096 wait complete. 2024-04-12T05:32:56.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:56 smithi032 ceph-mon[39731]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:32:57.269 DEBUG:teuthology.orchestra.run.smithi067:osd.5> sudo journalctl -f -n 0 -u ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@osd.5.service 2024-04-12T05:32:57.271 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-04-12T05:32:57.272 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph osd stat -f json 2024-04-12T05:32:57.303 INFO:journalctl@ceph.osd.5.smithi067.stdout:-- Logs begin at Fri 2024-04-12 05:14:32 UTC. -- 2024-04-12T05:32:57.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:57 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:57.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:57 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:57.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:57 smithi067 ceph-mon[49010]: from='osd.5 [v2:172.21.15.67:6816/1361296892,v1:172.21.15.67:6817/1361296892]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T05:32:57.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:57 smithi067 ceph-mon[49010]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T05:32:57.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:57 smithi067 ceph-mon[49010]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:32:57.563 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:32:57.778 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:57 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:57.779 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:57 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:57.779 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:57 smithi032 ceph-mon[39731]: from='osd.5 [v2:172.21.15.67:6816/1361296892,v1:172.21.15.67:6817/1361296892]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T05:32:57.779 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:57 smithi032 ceph-mon[39731]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T05:32:57.779 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:57 smithi032 ceph-mon[39731]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:32:58.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:58 smithi032 ceph-mon[39731]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-12T05:32:58.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:58 smithi032 ceph-mon[39731]: osdmap e37: 6 total, 5 up, 6 in 2024-04-12T05:32:58.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:58 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:32:58.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:58 smithi032 ceph-mon[39731]: from='osd.5 [v2:172.21.15.67:6816/1361296892,v1:172.21.15.67:6817/1361296892]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-12T05:32:58.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:58 smithi032 ceph-mon[39731]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-12T05:32:58.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:58 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:58.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:58 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:58.748 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 05:32:58 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[71935]: 2024-04-12T05:32:58.511+0000 7f52b2aff700 -1 osd.5 0 waiting for initial osdmap 2024-04-12T05:32:58.748 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 05:32:58 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[71935]: 2024-04-12T05:32:58.519+0000 7f52af8fe700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T05:32:58.748 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:58 smithi067 ceph-mon[49010]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-12T05:32:58.748 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:58 smithi067 ceph-mon[49010]: osdmap e37: 6 total, 5 up, 6 in 2024-04-12T05:32:58.748 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:58 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:32:58.749 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:58 smithi067 ceph-mon[49010]: from='osd.5 [v2:172.21.15.67:6816/1361296892,v1:172.21.15.67:6817/1361296892]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-12T05:32:58.749 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:58 smithi067 ceph-mon[49010]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-12T05:32:58.749 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:58 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:58.749 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:58 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:32:59.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.223+0000 7faa3d621700 1 -- 172.21.15.32:0/1413561722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa380fe430 msgr2=0x7faa380fe840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:59.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.223+0000 7faa3d621700 1 --2- 172.21.15.32:0/1413561722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa380fe430 0x7faa380fe840 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7faa20004940 tx=0x7faa20009e30 comp rx=0 tx=0).stop 2024-04-12T05:32:59.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.223+0000 7faa3d621700 1 -- 172.21.15.32:0/1413561722 shutdown_connections 2024-04-12T05:32:59.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.223+0000 7faa3d621700 1 --2- 172.21.15.32:0/1413561722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa380fed80 0x7faa380faec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:32:59.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.223+0000 7faa3d621700 1 --2- 172.21.15.32:0/1413561722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa380fe430 0x7faa380fe840 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:32:59.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.223+0000 7faa3d621700 1 -- 172.21.15.32:0/1413561722 >> 172.21.15.32:0/1413561722 conn(0x7faa380f6bd0 msgr2=0x7faa380f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:32:59.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.223+0000 7faa3d621700 1 -- 172.21.15.32:0/1413561722 shutdown_connections 2024-04-12T05:32:59.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.223+0000 7faa3d621700 1 -- 172.21.15.32:0/1413561722 wait complete. 2024-04-12T05:32:59.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.224+0000 7faa3d621700 1 Processor -- start 2024-04-12T05:32:59.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.224+0000 7faa3d621700 1 -- start start 2024-04-12T05:32:59.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.224+0000 7faa3d621700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa380fe430 0x7faa3810d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:32:59.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.224+0000 7faa3d621700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa380fed80 0x7faa3810dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:32:59.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.224+0000 7faa3d621700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3810e370 con 0x7faa380fed80 2024-04-12T05:32:59.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.224+0000 7faa3d621700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3810e4e0 con 0x7faa380fe430 2024-04-12T05:32:59.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.224+0000 7faa367fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa380fed80 0x7faa3810dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:59.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.224+0000 7faa367fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa380fed80 0x7faa3810dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54146/0 (socket says 172.21.15.32:54146) 2024-04-12T05:32:59.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.224+0000 7faa367fc700 1 -- 172.21.15.32:0/1800284443 learned_addr learned my addr 172.21.15.32:0/1800284443 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:32:59.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.224+0000 7faa36ffd700 1 --2- 172.21.15.32:0/1800284443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa380fe430 0x7faa3810d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:59.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.224+0000 7faa367fc700 1 -- 172.21.15.32:0/1800284443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa380fe430 msgr2=0x7faa3810d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:59.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.224+0000 7faa367fc700 1 --2- 172.21.15.32:0/1800284443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa380fe430 0x7faa3810d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:32:59.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.224+0000 7faa367fc700 1 -- 172.21.15.32:0/1800284443 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa20005040 con 0x7faa380fed80 2024-04-12T05:32:59.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.225+0000 7faa367fc700 1 --2- 172.21.15.32:0/1800284443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa380fed80 0x7faa3810dd70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7faa2800c830 tx=0x7faa28004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:32:59.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.225+0000 7faa2f7fe700 1 -- 172.21.15.32:0/1800284443 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa2800a760 con 0x7faa380fed80 2024-04-12T05:32:59.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.225+0000 7faa2f7fe700 1 -- 172.21.15.32:0/1800284443 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa2800ad70 con 0x7faa380fed80 2024-04-12T05:32:59.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.225+0000 7faa2f7fe700 1 -- 172.21.15.32:0/1800284443 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa280113a0 con 0x7faa380fed80 2024-04-12T05:32:59.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.225+0000 7faa3d621700 1 -- 172.21.15.32:0/1800284443 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa38112f40 con 0x7faa380fed80 2024-04-12T05:32:59.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.225+0000 7faa3d621700 1 -- 172.21.15.32:0/1800284443 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa381134d0 con 0x7faa380fed80 2024-04-12T05:32:59.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.226+0000 7faa3d621700 1 -- 172.21.15.32:0/1800284443 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa3804c710 con 0x7faa380fed80 2024-04-12T05:32:59.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.226+0000 7faa2f7fe700 1 -- 172.21.15.32:0/1800284443 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7faa28012400 con 0x7faa380fed80 2024-04-12T05:32:59.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.227+0000 7faa2f7fe700 1 --2- 172.21.15.32:0/1800284443 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7faa240692a0 0x7faa2406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:32:59.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.227+0000 7faa2f7fe700 1 -- 172.21.15.32:0/1800284443 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4354+0+0 (secure 0 0 0) 0x7faa28086cd0 con 0x7faa380fed80 2024-04-12T05:32:59.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.227+0000 7faa36ffd700 1 --2- 172.21.15.32:0/1800284443 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7faa240692a0 0x7faa2406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:59.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.227+0000 7faa36ffd700 1 --2- 172.21.15.32:0/1800284443 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7faa240692a0 0x7faa2406b750 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7faa2000b900 tx=0x7faa20009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:32:59.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.232+0000 7faa2f7fe700 1 -- 172.21.15.32:0/1800284443 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7faa28056f80 con 0x7faa380fed80 2024-04-12T05:32:59.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.430+0000 7faa3d621700 1 -- 172.21.15.32:0/1800284443 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7faa381057a0 con 0x7faa380fed80 2024-04-12T05:32:59.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.430+0000 7faa2f7fe700 1 -- 172.21.15.32:0/1800284443 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v38) v1 ==== 74+0+130 (secure 0 0 0) 0x7faa28056b10 con 0x7faa380fed80 2024-04-12T05:32:59.430 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:32:59.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.432+0000 7faa3d621700 1 -- 172.21.15.32:0/1800284443 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7faa240692a0 msgr2=0x7faa2406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:59.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.432+0000 7faa3d621700 1 --2- 172.21.15.32:0/1800284443 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7faa240692a0 0x7faa2406b750 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7faa2000b900 tx=0x7faa20009e30 comp rx=0 tx=0).stop 2024-04-12T05:32:59.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.432+0000 7faa3d621700 1 -- 172.21.15.32:0/1800284443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa380fed80 msgr2=0x7faa3810dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:59.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.432+0000 7faa3d621700 1 --2- 172.21.15.32:0/1800284443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa380fed80 0x7faa3810dd70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7faa2800c830 tx=0x7faa28004470 comp rx=0 tx=0).stop 2024-04-12T05:32:59.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.432+0000 7faa3d621700 1 -- 172.21.15.32:0/1800284443 shutdown_connections 2024-04-12T05:32:59.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.432+0000 7faa3d621700 1 --2- 172.21.15.32:0/1800284443 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7faa240692a0 0x7faa2406b750 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:32:59.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.432+0000 7faa3d621700 1 --2- 172.21.15.32:0/1800284443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa380fe430 0x7faa3810d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:32:59.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.432+0000 7faa3d621700 1 --2- 172.21.15.32:0/1800284443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa380fed80 0x7faa3810dd70 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:32:59.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.432+0000 7faa3d621700 1 -- 172.21.15.32:0/1800284443 >> 172.21.15.32:0/1800284443 conn(0x7faa380f6bd0 msgr2=0x7faa380f8330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:32:59.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.432+0000 7faa3d621700 1 -- 172.21.15.32:0/1800284443 shutdown_connections 2024-04-12T05:32:59.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:32:59.432+0000 7faa3d621700 1 -- 172.21.15.32:0/1800284443 wait complete. 2024-04-12T05:32:59.731 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:59 smithi067 ceph-mon[49010]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-04-12T05:32:59.731 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:59 smithi067 ceph-mon[49010]: osdmap e38: 6 total, 5 up, 6 in 2024-04-12T05:32:59.731 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:59 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:32:59.732 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:59 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:32:59.732 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:59 smithi067 ceph-mon[49010]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:32:59.732 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:32:59 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/1800284443' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-12T05:32:59.746 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:59 smithi032 ceph-mon[39731]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-04-12T05:32:59.746 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:59 smithi032 ceph-mon[39731]: osdmap e38: 6 total, 5 up, 6 in 2024-04-12T05:32:59.746 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:59 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:32:59.747 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:59 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:32:59.747 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:59 smithi032 ceph-mon[39731]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:32:59.747 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:32:59 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/1800284443' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-12T05:33:00.107 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":38,"num_osds":6,"num_up_osds":5,"osd_up_since":1712899951,"num_in_osds":6,"osd_in_since":1712899959,"num_remapped_pgs":0} 2024-04-12T05:33:00.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:00 smithi067 ceph-mon[49010]: purged_snaps scrub starts 2024-04-12T05:33:00.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:00 smithi067 ceph-mon[49010]: purged_snaps scrub ok 2024-04-12T05:33:00.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:00 smithi067 ceph-mon[49010]: osd.5 [v2:172.21.15.67:6816/1361296892,v1:172.21.15.67:6817/1361296892] boot 2024-04-12T05:33:00.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:00 smithi067 ceph-mon[49010]: osdmap e39: 6 total, 6 up, 6 in 2024-04-12T05:33:00.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:00 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:33:00.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:00 smithi032 ceph-mon[39731]: purged_snaps scrub starts 2024-04-12T05:33:00.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:00 smithi032 ceph-mon[39731]: purged_snaps scrub ok 2024-04-12T05:33:00.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:00 smithi032 ceph-mon[39731]: osd.5 [v2:172.21.15.67:6816/1361296892,v1:172.21.15.67:6817/1361296892] boot 2024-04-12T05:33:00.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:00 smithi032 ceph-mon[39731]: osdmap e39: 6 total, 6 up, 6 in 2024-04-12T05:33:00.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:00 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:33:01.109 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph osd stat -f json 2024-04-12T05:33:01.399 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:33:01.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:01 smithi067 ceph-mon[49010]: osdmap e40: 6 total, 6 up, 6 in 2024-04-12T05:33:01.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:01 smithi067 ceph-mon[49010]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:01.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:01 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:33:01.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:01 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:33:01.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:01 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:33:01.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:01 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:33:01.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:01 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:33:01.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:01 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:33:01.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:01 smithi032 ceph-mon[39731]: osdmap e40: 6 total, 6 up, 6 in 2024-04-12T05:33:01.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:01 smithi032 ceph-mon[39731]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:01.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:01 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:33:01.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:01 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:33:01.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:01 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:33:01.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:01 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:33:01.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:01 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:33:01.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:01 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:33:02.670 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:02 smithi032 ceph-mon[39731]: Detected new or changed devices on smithi067 2024-04-12T05:33:02.670 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:02 smithi032 ceph-mon[39731]: Adjusting osd_memory_target on smithi067 to 5323M 2024-04-12T05:33:02.670 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:02 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:33:02.670 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:02 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:33:02.670 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:02 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:33:02.670 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:02 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:33:02.670 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:02 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:33:02.670 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:02 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:33:02.671 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:02 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:33:03.020 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:02 smithi067 ceph-mon[49010]: Detected new or changed devices on smithi067 2024-04-12T05:33:03.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:02 smithi067 ceph-mon[49010]: Adjusting osd_memory_target on smithi067 to 5323M 2024-04-12T05:33:03.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:02 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:33:03.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:02 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:33:03.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:02 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:33:03.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:02 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:33:03.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:02 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:33:03.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:02 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:33:03.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:02 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:33:03.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.236+0000 7fcf55450700 1 -- 172.21.15.32:0/3619894866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf500fcf80 msgr2=0x7fcf500fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:03.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.236+0000 7fcf55450700 1 --2- 172.21.15.32:0/3619894866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf500fcf80 0x7fcf500fd3f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcf38004940 tx=0x7fcf38009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:03.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.236+0000 7fcf55450700 1 -- 172.21.15.32:0/3619894866 shutdown_connections 2024-04-12T05:33:03.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.236+0000 7fcf55450700 1 --2- 172.21.15.32:0/3619894866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf500fcf80 0x7fcf500fd3f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:03.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.236+0000 7fcf55450700 1 --2- 172.21.15.32:0/3619894866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf500fbcd0 0x7fcf500fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:03.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.236+0000 7fcf55450700 1 -- 172.21.15.32:0/3619894866 >> 172.21.15.32:0/3619894866 conn(0x7fcf500f7430 msgr2=0x7fcf500f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:03.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.236+0000 7fcf55450700 1 -- 172.21.15.32:0/3619894866 shutdown_connections 2024-04-12T05:33:03.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.236+0000 7fcf55450700 1 -- 172.21.15.32:0/3619894866 wait complete. 2024-04-12T05:33:03.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.237+0000 7fcf55450700 1 Processor -- start 2024-04-12T05:33:03.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.237+0000 7fcf55450700 1 -- start start 2024-04-12T05:33:03.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.237+0000 7fcf55450700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf500fbcd0 0x7fcf5010d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:03.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.237+0000 7fcf55450700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf500fcf80 0x7fcf5010dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:03.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.237+0000 7fcf55450700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5010e4e0 con 0x7fcf500fcf80 2024-04-12T05:33:03.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.237+0000 7fcf55450700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5010ece0 con 0x7fcf500fbcd0 2024-04-12T05:33:03.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.237+0000 7fcf4e7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf500fcf80 0x7fcf5010dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:03.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.237+0000 7fcf4e7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf500fcf80 0x7fcf5010dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59404/0 (socket says 172.21.15.32:59404) 2024-04-12T05:33:03.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.237+0000 7fcf4e7fc700 1 -- 172.21.15.32:0/530083529 learned_addr learned my addr 172.21.15.32:0/530083529 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:33:03.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.237+0000 7fcf4effd700 1 --2- 172.21.15.32:0/530083529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf500fbcd0 0x7fcf5010d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:03.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.237+0000 7fcf4e7fc700 1 -- 172.21.15.32:0/530083529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf500fbcd0 msgr2=0x7fcf5010d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:03.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.238+0000 7fcf4e7fc700 1 --2- 172.21.15.32:0/530083529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf500fbcd0 0x7fcf5010d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:03.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.238+0000 7fcf4e7fc700 1 -- 172.21.15.32:0/530083529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf38005040 con 0x7fcf500fcf80 2024-04-12T05:33:03.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.238+0000 7fcf4e7fc700 1 --2- 172.21.15.32:0/530083529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf500fcf80 0x7fcf5010dee0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcf3800b3b0 tx=0x7fcf38009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:03.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.238+0000 7fcf477fe700 1 -- 172.21.15.32:0/530083529 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf3800e040 con 0x7fcf500fcf80 2024-04-12T05:33:03.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.238+0000 7fcf477fe700 1 -- 172.21.15.32:0/530083529 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf3800bc50 con 0x7fcf500fcf80 2024-04-12T05:33:03.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.238+0000 7fcf55450700 1 -- 172.21.15.32:0/530083529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf5010ef60 con 0x7fcf500fcf80 2024-04-12T05:33:03.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.238+0000 7fcf477fe700 1 -- 172.21.15.32:0/530083529 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf3801c630 con 0x7fcf500fcf80 2024-04-12T05:33:03.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.238+0000 7fcf55450700 1 -- 172.21.15.32:0/530083529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf5010f4a0 con 0x7fcf500fcf80 2024-04-12T05:33:03.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.239+0000 7fcf55450700 1 -- 172.21.15.32:0/530083529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf50101b10 con 0x7fcf500fcf80 2024-04-12T05:33:03.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.239+0000 7fcf477fe700 1 -- 172.21.15.32:0/530083529 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fcf38013420 con 0x7fcf500fcf80 2024-04-12T05:33:03.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.240+0000 7fcf477fe700 1 --2- 172.21.15.32:0/530083529 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fcf3c0692a0 0x7fcf3c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:03.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.240+0000 7fcf477fe700 1 -- 172.21.15.32:0/530083529 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fcf38087b20 con 0x7fcf500fcf80 2024-04-12T05:33:03.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.240+0000 7fcf4effd700 1 --2- 172.21.15.32:0/530083529 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fcf3c0692a0 0x7fcf3c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:03.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.240+0000 7fcf4effd700 1 --2- 172.21.15.32:0/530083529 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fcf3c0692a0 0x7fcf3c06b750 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcf40004f40 tx=0x7fcf4000b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:03.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.244+0000 7fcf477fe700 1 -- 172.21.15.32:0/530083529 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fcf38057c20 con 0x7fcf500fcf80 2024-04-12T05:33:03.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.438+0000 7fcf55450700 1 -- 172.21.15.32:0/530083529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fcf5005f5f0 con 0x7fcf500fcf80 2024-04-12T05:33:03.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.439+0000 7fcf477fe700 1 -- 172.21.15.32:0/530083529 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7fcf380577b0 con 0x7fcf500fcf80 2024-04-12T05:33:03.437 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:33:03.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.440+0000 7fcf55450700 1 -- 172.21.15.32:0/530083529 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fcf3c0692a0 msgr2=0x7fcf3c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:03.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.440+0000 7fcf55450700 1 --2- 172.21.15.32:0/530083529 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fcf3c0692a0 0x7fcf3c06b750 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcf40004f40 tx=0x7fcf4000b040 comp rx=0 tx=0).stop 2024-04-12T05:33:03.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.440+0000 7fcf55450700 1 -- 172.21.15.32:0/530083529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf500fcf80 msgr2=0x7fcf5010dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:03.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.440+0000 7fcf55450700 1 --2- 172.21.15.32:0/530083529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf500fcf80 0x7fcf5010dee0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcf3800b3b0 tx=0x7fcf38009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:03.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.440+0000 7fcf55450700 1 -- 172.21.15.32:0/530083529 shutdown_connections 2024-04-12T05:33:03.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.440+0000 7fcf55450700 1 --2- 172.21.15.32:0/530083529 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fcf3c0692a0 0x7fcf3c06b750 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:03.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.440+0000 7fcf55450700 1 --2- 172.21.15.32:0/530083529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf500fbcd0 0x7fcf5010d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:03.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.440+0000 7fcf55450700 1 --2- 172.21.15.32:0/530083529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf500fcf80 0x7fcf5010dee0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:03.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.440+0000 7fcf55450700 1 -- 172.21.15.32:0/530083529 >> 172.21.15.32:0/530083529 conn(0x7fcf500f7430 msgr2=0x7fcf50065d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:03.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.441+0000 7fcf55450700 1 -- 172.21.15.32:0/530083529 shutdown_connections 2024-04-12T05:33:03.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:03.441+0000 7fcf55450700 1 -- 172.21.15.32:0/530083529 wait complete. 2024-04-12T05:33:03.769 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:03 smithi032 ceph-mon[39731]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:03.769 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:03 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/530083529' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-12T05:33:04.020 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:03 smithi067 ceph-mon[49010]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:04.020 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:03 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/530083529' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-12T05:33:04.147 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1712899979,"num_in_osds":6,"osd_in_since":1712899959,"num_remapped_pgs":0} 2024-04-12T05:33:04.148 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph osd dump --format=json 2024-04-12T05:33:04.429 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:33:06.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.261+0000 7f47d5d77700 1 -- 172.21.15.32:0/2036756916 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47d0102e70 msgr2=0x7f47d0103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:06.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.261+0000 7f47d5d77700 1 --2- 172.21.15.32:0/2036756916 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47d0102e70 0x7f47d0103240 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f47c0004940 tx=0x7f47c0009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:06.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.262+0000 7f47d5d77700 1 -- 172.21.15.32:0/2036756916 shutdown_connections 2024-04-12T05:33:06.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.262+0000 7f47d5d77700 1 --2- 172.21.15.32:0/2036756916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47d00fb4a0 0x7f47d00fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:06.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.262+0000 7f47d5d77700 1 --2- 172.21.15.32:0/2036756916 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47d0102e70 0x7f47d0103240 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:06.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.262+0000 7f47d5d77700 1 -- 172.21.15.32:0/2036756916 >> 172.21.15.32:0/2036756916 conn(0x7f47d00714c0 msgr2=0x7f47d00718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:06.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.262+0000 7f47d5d77700 1 -- 172.21.15.32:0/2036756916 shutdown_connections 2024-04-12T05:33:06.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.262+0000 7f47d5d77700 1 -- 172.21.15.32:0/2036756916 wait complete. 2024-04-12T05:33:06.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.262+0000 7f47d5d77700 1 Processor -- start 2024-04-12T05:33:06.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.262+0000 7f47d5d77700 1 -- start start 2024-04-12T05:33:06.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.262+0000 7f47d5d77700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47d00fb4a0 0x7f47d0113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:06.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.262+0000 7f47d5d77700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47d0102e70 0x7f47d0113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:06.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.263+0000 7f47d5d77700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d0114600 con 0x7f47d0102e70 2024-04-12T05:33:06.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.263+0000 7f47d5d77700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d010d9f0 con 0x7f47d00fb4a0 2024-04-12T05:33:06.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.263+0000 7f47ceffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47d0102e70 0x7f47d0113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:06.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.263+0000 7f47ceffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47d0102e70 0x7f47d0113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59434/0 (socket says 172.21.15.32:59434) 2024-04-12T05:33:06.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.263+0000 7f47ceffd700 1 -- 172.21.15.32:0/3034528393 learned_addr learned my addr 172.21.15.32:0/3034528393 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:33:06.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.263+0000 7f47cf7fe700 1 --2- 172.21.15.32:0/3034528393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47d00fb4a0 0x7f47d0113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:06.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.263+0000 7f47ceffd700 1 -- 172.21.15.32:0/3034528393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47d00fb4a0 msgr2=0x7f47d0113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:06.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.263+0000 7f47ceffd700 1 --2- 172.21.15.32:0/3034528393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47d00fb4a0 0x7f47d0113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:06.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.263+0000 7f47ceffd700 1 -- 172.21.15.32:0/3034528393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c0005040 con 0x7f47d0102e70 2024-04-12T05:33:06.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.263+0000 7f47ceffd700 1 --2- 172.21.15.32:0/3034528393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47d0102e70 0x7f47d0113fb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f47c4002a50 tx=0x7f47c40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:06.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.263+0000 7f47b7fff700 1 -- 172.21.15.32:0/3034528393 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47c4007cc0 con 0x7f47d0102e70 2024-04-12T05:33:06.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.263+0000 7f47b7fff700 1 -- 172.21.15.32:0/3034528393 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f47c400f040 con 0x7f47d0102e70 2024-04-12T05:33:06.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.263+0000 7f47d5d77700 1 -- 172.21.15.32:0/3034528393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47d010dcd0 con 0x7f47d0102e70 2024-04-12T05:33:06.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.263+0000 7f47b7fff700 1 -- 172.21.15.32:0/3034528393 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47c4009540 con 0x7f47d0102e70 2024-04-12T05:33:06.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.263+0000 7f47d5d77700 1 -- 172.21.15.32:0/3034528393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47d010e290 con 0x7f47d0102e70 2024-04-12T05:33:06.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.264+0000 7f47d5d77700 1 -- 172.21.15.32:0/3034528393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47d0100110 con 0x7f47d0102e70 2024-04-12T05:33:06.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.265+0000 7f47b7fff700 1 -- 172.21.15.32:0/3034528393 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f47c40096a0 con 0x7f47d0102e70 2024-04-12T05:33:06.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.265+0000 7f47b7fff700 1 --2- 172.21.15.32:0/3034528393 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f47b80692b0 0x7f47b806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:06.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.265+0000 7f47b7fff700 1 -- 172.21.15.32:0/3034528393 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f47c4086b90 con 0x7f47d0102e70 2024-04-12T05:33:06.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.265+0000 7f47cf7fe700 1 --2- 172.21.15.32:0/3034528393 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f47b80692b0 0x7f47b806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:06.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.266+0000 7f47cf7fe700 1 --2- 172.21.15.32:0/3034528393 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f47b80692b0 0x7f47b806b760 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f47c000b3b0 tx=0x7f47c0009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:06.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.270+0000 7f47b7fff700 1 -- 172.21.15.32:0/3034528393 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f47c4056c90 con 0x7f47d0102e70 2024-04-12T05:33:06.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:06 smithi032 ceph-mon[39731]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:06.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.464+0000 7f47d5d77700 1 -- 172.21.15.32:0/3034528393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47d010f0c0 con 0x7f47d0102e70 2024-04-12T05:33:06.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.465+0000 7f47b7fff700 1 -- 172.21.15.32:0/3034528393 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11031 (secure 0 0 0) 0x7f47c4056820 con 0x7f47d0102e70 2024-04-12T05:33:06.464 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:33:06.464 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":40,"fsid":"37b58ce2-f88d-11ee-bc8c-c7b262605968","created":"2024-04-12T05:27:07.779407+0000","modified":"2024-04-12T05:33:00.514069+0000","last_up_change":"2024-04-12T05:32:59.508815+0000","last_in_change":"2024-04-12T05:32:39.814568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T05:31:39.568876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"c1f69d2a-5e37-47b8-b68f-0568af894b91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":3061707028},{"type":"v1","addr":"172.21.15.32:6803","nonce":3061707028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":3061707028},{"type":"v1","addr":"172.21.15.32:6805","nonce":3061707028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3061707028},{"type":"v1","addr":"172.21.15.32:6809","nonce":3061707028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":3061707028},{"type":"v1","addr":"172.21.15.32:6807","nonce":3061707028}]},"public_addr":"172.21.15.32:6803/3061707028","cluster_addr":"172.21.15.32:6805/3061707028","heartbeat_back_addr":"172.21.15.32:6809/3061707028","heartbeat_front_addr":"172.21.15.32:6807/3061707028","state":["exists","up"]},{"osd":1,"uuid":"86b93557-6e6f-4b84-a5f5-5bce4aa65a32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":164843120},{"type":"v1","addr":"172.21.15.32:6811","nonce":164843120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":164843120},{"type":"v1","addr":"172.21.15.32:6813","nonce":164843120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":164843120},{"type":"v1","addr":"172.21.15.32:6817","nonce":164843120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":164843120},{"type":"v1","addr":"172.21.15.32:6815","nonce":164843120}]},"public_addr":"172.21.15.32:6811/164843120","cluster_addr":"172.21.15.32:6813/164843120","heartbeat_back_addr":"172.21.15.32:6817/164843120","heartbeat_front_addr":"172.21.15.32:6815/164843120","state":["exists","up"]},{"osd":2,"uuid":"d1556613-e0de-4b0e-bb7d-e90361ec9007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":2698911336},{"type":"v1","addr":"172.21.15.32:6819","nonce":2698911336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":2698911336},{"type":"v1","addr":"172.21.15.32:6821","nonce":2698911336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":2698911336},{"type":"v1","addr":"172.21.15.32:6825","nonce":2698911336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":2698911336},{"type":"v1","addr":"172.21.15.32:6823","nonce":2698911336}]},"public_addr":"172.21.15.32:6819/2698911336","cluster_addr":"172.21.15.32:6821/2698911336","heartbeat_back_addr":"172.21.15.32:6825/2698911336","heartbeat_front_addr":"172.21.15.32:6823/2698911336","state":["exists","up"]},{"osd":3,"uuid":"213dc62e-a894-4bc7-a2dd-ca98d1313d6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1205580917},{"type":"v1","addr":"172.21.15.67:6801","nonce":1205580917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1205580917},{"type":"v1","addr":"172.21.15.67:6803","nonce":1205580917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1205580917},{"type":"v1","addr":"172.21.15.67:6807","nonce":1205580917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1205580917},{"type":"v1","addr":"172.21.15.67:6805","nonce":1205580917}]},"public_addr":"172.21.15.67:6801/1205580917","cluster_addr":"172.21.15.67:6803/1205580917","heartbeat_back_addr":"172.21.15.67:6807/1205580917","heartbeat_front_addr":"172.21.15.67:6805/1205580917","state":["exists","up"]},{"osd":4,"uuid":"8a0ff860-fff7-434a-9e78-f2020bc84966","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":286946785},{"type":"v1","addr":"172.21.15.67:6809","nonce":286946785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":286946785},{"type":"v1","addr":"172.21.15.67:6811","nonce":286946785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":286946785},{"type":"v1","addr":"172.21.15.67:6815","nonce":286946785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":286946785},{"type":"v1","addr":"172.21.15.67:6813","nonce":286946785}]},"public_addr":"172.21.15.67:6809/286946785","cluster_addr":"172.21.15.67:6811/286946785","heartbeat_back_addr":"172.21.15.67:6815/286946785","heartbeat_front_addr":"172.21.15.67:6813/286946785","state":["exists","up"]},{"osd":5,"uuid":"3486e5c9-9f44-4f32-a186-fc6198f09549","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1361296892},{"type":"v1","addr":"172.21.15.67:6817","nonce":1361296892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":1361296892},{"type":"v1","addr":"172.21.15.67:6819","nonce":1361296892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":1361296892},{"type":"v1","addr":"172.21.15.67:6823","nonce":1361296892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1361296892},{"type":"v1","addr":"172.21.15.67:6821","nonce":1361296892}]},"public_addr":"172.21.15.67:6817/1361296892","cluster_addr":"172.21.15.67:6819/1361296892","heartbeat_back_addr":"172.21.15.67:6823/1361296892","heartbeat_front_addr":"172.21.15.67:6821/1361296892","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:30:38.748993+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:31:08.677208+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:31:36.979481+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:32:02.668398+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:32:30.143788+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:32:57.613087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6801/3971961003":"2024-04-13T05:29:25.054120+0000","172.21.15.32:0/4112976740":"2024-04-13T05:29:25.054120+0000","172.21.15.32:0/146872823":"2024-04-13T05:28:14.470413+0000","172.21.15.32:0/3082743475":"2024-04-13T05:28:14.470413+0000","172.21.15.32:0/2162993076":"2024-04-13T05:28:14.470413+0000","172.21.15.32:6801/1962278696":"2024-04-13T05:28:14.470413+0000","172.21.15.32:0/795906028":"2024-04-13T05:29:25.054120+0000","172.21.15.32:0/138657961":"2024-04-13T05:27:40.952118+0000","172.21.15.32:0/1643708423":"2024-04-13T05:27:40.952118+0000","172.21.15.32:0/674029871":"2024-04-13T05:29:25.054120+0000","172.21.15.32:6800/656201914":"2024-04-13T05:27:40.952118+0000","172.21.15.32:6800/3971961003":"2024-04-13T05:29:25.054120+0000","172.21.15.32:6801/656201914":"2024-04-13T05:27:40.952118+0000","172.21.15.32:6800/1962278696":"2024-04-13T05:28:14.470413+0000","172.21.15.32:0/4031244002":"2024-04-13T05:27:40.952118+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T05:33:06.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.467+0000 7f47d5d77700 1 -- 172.21.15.32:0/3034528393 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f47b80692b0 msgr2=0x7f47b806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:06.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.467+0000 7f47d5d77700 1 --2- 172.21.15.32:0/3034528393 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f47b80692b0 0x7f47b806b760 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f47c000b3b0 tx=0x7f47c0009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:06.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.467+0000 7f47d5d77700 1 -- 172.21.15.32:0/3034528393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47d0102e70 msgr2=0x7f47d0113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:06.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.467+0000 7f47d5d77700 1 --2- 172.21.15.32:0/3034528393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47d0102e70 0x7f47d0113fb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f47c4002a50 tx=0x7f47c40046b0 comp rx=0 tx=0).stop 2024-04-12T05:33:06.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.467+0000 7f47d5d77700 1 -- 172.21.15.32:0/3034528393 shutdown_connections 2024-04-12T05:33:06.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.467+0000 7f47d5d77700 1 --2- 172.21.15.32:0/3034528393 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f47b80692b0 0x7f47b806b760 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:06.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.467+0000 7f47d5d77700 1 --2- 172.21.15.32:0/3034528393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47d00fb4a0 0x7f47d0113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:06.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.467+0000 7f47d5d77700 1 --2- 172.21.15.32:0/3034528393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47d0102e70 0x7f47d0113fb0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:06.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.467+0000 7f47d5d77700 1 -- 172.21.15.32:0/3034528393 >> 172.21.15.32:0/3034528393 conn(0x7f47d00714c0 msgr2=0x7f47d00fa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:06.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.467+0000 7f47d5d77700 1 -- 172.21.15.32:0/3034528393 shutdown_connections 2024-04-12T05:33:06.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:06.467+0000 7f47d5d77700 1 -- 172.21.15.32:0/3034528393 wait complete. 2024-04-12T05:33:06.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:06 smithi067 ceph-mon[49010]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:07.110 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-12T05:31:39.568876+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-04-12T05:33:07.110 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-12T05:33:07.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:07 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/3034528393' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T05:33:07.389 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:33:07.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:07 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/3034528393' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T05:33:08.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:08 smithi067 ceph-mon[49010]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:08.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:08 smithi032 ceph-mon[39731]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:09.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.218+0000 7f586d194700 1 -- 172.21.15.32:0/565899702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58680fbc80 msgr2=0x7f58680fc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:09.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.218+0000 7f586d194700 1 --2- 172.21.15.32:0/565899702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58680fbc80 0x7f58680fc050 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f585c004940 tx=0x7f585c009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:09.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.218+0000 7f586d194700 1 -- 172.21.15.32:0/565899702 shutdown_connections 2024-04-12T05:33:09.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.218+0000 7f586d194700 1 --2- 172.21.15.32:0/565899702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58680fc590 0x7f58681092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:09.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.218+0000 7f586d194700 1 --2- 172.21.15.32:0/565899702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58680fbc80 0x7f58680fc050 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:09.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.218+0000 7f586d194700 1 -- 172.21.15.32:0/565899702 >> 172.21.15.32:0/565899702 conn(0x7f5868072bc0 msgr2=0x7f5868072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:09.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.218+0000 7f586d194700 1 -- 172.21.15.32:0/565899702 shutdown_connections 2024-04-12T05:33:09.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.218+0000 7f586d194700 1 -- 172.21.15.32:0/565899702 wait complete. 2024-04-12T05:33:09.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.219+0000 7f586d194700 1 Processor -- start 2024-04-12T05:33:09.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.219+0000 7f586d194700 1 -- start start 2024-04-12T05:33:09.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.219+0000 7f586d194700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58680fbc80 0x7f5868115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:09.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.219+0000 7f586d194700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58680fc590 0x7f5868116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:09.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.219+0000 7f586d194700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5868116930 con 0x7f58680fc590 2024-04-12T05:33:09.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.219+0000 7f586d194700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f586810fc70 con 0x7f58680fbc80 2024-04-12T05:33:09.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.219+0000 7f586659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58680fc590 0x7f5868116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:09.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.219+0000 7f586659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58680fc590 0x7f5868116250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59456/0 (socket says 172.21.15.32:59456) 2024-04-12T05:33:09.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.219+0000 7f586659c700 1 -- 172.21.15.32:0/2418391623 learned_addr learned my addr 172.21.15.32:0/2418391623 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:33:09.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.220+0000 7f5866d9d700 1 --2- 172.21.15.32:0/2418391623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58680fbc80 0x7f5868115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:09.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.220+0000 7f586659c700 1 -- 172.21.15.32:0/2418391623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58680fbc80 msgr2=0x7f5868115d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:09.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.220+0000 7f586659c700 1 --2- 172.21.15.32:0/2418391623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58680fbc80 0x7f5868115d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:09.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.220+0000 7f586659c700 1 -- 172.21.15.32:0/2418391623 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f585c005040 con 0x7f58680fc590 2024-04-12T05:33:09.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.220+0000 7f586659c700 1 --2- 172.21.15.32:0/2418391623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58680fc590 0x7f5868116250 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f585800b790 tx=0x7f5858009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:09.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.220+0000 7f584f7fe700 1 -- 172.21.15.32:0/2418391623 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f585800f770 con 0x7f58680fc590 2024-04-12T05:33:09.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.220+0000 7f584f7fe700 1 -- 172.21.15.32:0/2418391623 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f585800fd80 con 0x7f58680fc590 2024-04-12T05:33:09.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.220+0000 7f586d194700 1 -- 172.21.15.32:0/2418391623 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f586810fef0 con 0x7f58680fc590 2024-04-12T05:33:09.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.220+0000 7f584f7fe700 1 -- 172.21.15.32:0/2418391623 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f585800d310 con 0x7f58680fc590 2024-04-12T05:33:09.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.220+0000 7f586d194700 1 -- 172.21.15.32:0/2418391623 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5868110430 con 0x7f58680fc590 2024-04-12T05:33:09.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.221+0000 7f586d194700 1 -- 172.21.15.32:0/2418391623 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5868103c90 con 0x7f58680fc590 2024-04-12T05:33:09.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.222+0000 7f584f7fe700 1 -- 172.21.15.32:0/2418391623 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f5858010420 con 0x7f58680fc590 2024-04-12T05:33:09.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.222+0000 7f584f7fe700 1 --2- 172.21.15.32:0/2418391623 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f58500692a0 0x7f585006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:09.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.222+0000 7f584f7fe700 1 -- 172.21.15.32:0/2418391623 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f58580874e0 con 0x7f58680fc590 2024-04-12T05:33:09.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.222+0000 7f5866d9d700 1 --2- 172.21.15.32:0/2418391623 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f58500692a0 0x7f585006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:09.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.223+0000 7f5866d9d700 1 --2- 172.21.15.32:0/2418391623 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f58500692a0 0x7f585006b750 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f585c004f90 tx=0x7f585c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:09.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.226+0000 7f584f7fe700 1 -- 172.21.15.32:0/2418391623 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5858057680 con 0x7f58680fc590 2024-04-12T05:33:09.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.418+0000 7f586d194700 1 -- 172.21.15.32:0/2418391623 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f586810c0a0 con 0x7f58680fc590 2024-04-12T05:33:09.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.418+0000 7f584f7fe700 1 -- 172.21.15.32:0/2418391623 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5858057210 con 0x7f58680fc590 2024-04-12T05:33:09.419 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-12T05:33:09.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.420+0000 7f586d194700 1 -- 172.21.15.32:0/2418391623 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f58500692a0 msgr2=0x7f585006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:09.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.420+0000 7f586d194700 1 --2- 172.21.15.32:0/2418391623 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f58500692a0 0x7f585006b750 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f585c004f90 tx=0x7f585c009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:09.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.420+0000 7f586d194700 1 -- 172.21.15.32:0/2418391623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58680fc590 msgr2=0x7f5868116250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:09.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.420+0000 7f586d194700 1 --2- 172.21.15.32:0/2418391623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58680fc590 0x7f5868116250 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f585800b790 tx=0x7f5858009250 comp rx=0 tx=0).stop 2024-04-12T05:33:09.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.420+0000 7f586d194700 1 -- 172.21.15.32:0/2418391623 shutdown_connections 2024-04-12T05:33:09.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.420+0000 7f586d194700 1 --2- 172.21.15.32:0/2418391623 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f58500692a0 0x7f585006b750 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:09.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.420+0000 7f586d194700 1 --2- 172.21.15.32:0/2418391623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58680fbc80 0x7f5868115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:09.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.420+0000 7f586d194700 1 --2- 172.21.15.32:0/2418391623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58680fc590 0x7f5868116250 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:09.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.420+0000 7f586d194700 1 -- 172.21.15.32:0/2418391623 >> 172.21.15.32:0/2418391623 conn(0x7f5868072bc0 msgr2=0x7f58680fa4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:09.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.420+0000 7f586d194700 1 -- 172.21.15.32:0/2418391623 shutdown_connections 2024-04-12T05:33:09.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:09.420+0000 7f586d194700 1 -- 172.21.15.32:0/2418391623 wait complete. 2024-04-12T05:33:10.127 INFO:tasks.cephadm:Setting up client nodes... 2024-04-12T05:33:10.127 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-12T05:33:10.408 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:33:10.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:10 smithi067 ceph-mon[49010]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:10.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:10 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/2418391623' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T05:33:10.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:10 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:33:10.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:10 smithi032 ceph-mon[39731]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:10.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:10 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/2418391623' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T05:33:10.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:10 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:33:12.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.245+0000 7f9138d51700 1 -- 172.21.15.32:0/1770020550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91340fffb0 msgr2=0x7f9134102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:12.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.245+0000 7f9138d51700 1 --2- 172.21.15.32:0/1770020550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91340fffb0 0x7f9134102390 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9124004940 tx=0x7f9124009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:12.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.245+0000 7f9138d51700 1 -- 172.21.15.32:0/1770020550 shutdown_connections 2024-04-12T05:33:12.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.245+0000 7f9138d51700 1 --2- 172.21.15.32:0/1770020550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91340fffb0 0x7f9134102390 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:12.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.245+0000 7f9138d51700 1 --2- 172.21.15.32:0/1770020550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9134066220 0x7f91340ffa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:12.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.245+0000 7f9138d51700 1 -- 172.21.15.32:0/1770020550 >> 172.21.15.32:0/1770020550 conn(0x7f91340f7460 msgr2=0x7f91340f98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:12.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.245+0000 7f9138d51700 1 -- 172.21.15.32:0/1770020550 shutdown_connections 2024-04-12T05:33:12.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.245+0000 7f9138d51700 1 -- 172.21.15.32:0/1770020550 wait complete. 2024-04-12T05:33:12.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.246+0000 7f9138d51700 1 Processor -- start 2024-04-12T05:33:12.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.246+0000 7f9138d51700 1 -- start start 2024-04-12T05:33:12.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.246+0000 7f9138d51700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9134066220 0x7f913410d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:12.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.246+0000 7f9138d51700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91340fffb0 0x7f913410ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:12.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.246+0000 7f9138d51700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913410e460 con 0x7f91340fffb0 2024-04-12T05:33:12.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.246+0000 7f9138d51700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9134112e00 con 0x7f9134066220 2024-04-12T05:33:12.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.246+0000 7f9131d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91340fffb0 0x7f913410ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:12.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.246+0000 7f9131d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91340fffb0 0x7f913410ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59106/0 (socket says 172.21.15.32:59106) 2024-04-12T05:33:12.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.246+0000 7f9131d9b700 1 -- 172.21.15.32:0/555330618 learned_addr learned my addr 172.21.15.32:0/555330618 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:33:12.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.246+0000 7f913259c700 1 --2- 172.21.15.32:0/555330618 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9134066220 0x7f913410d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:12.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.246+0000 7f9131d9b700 1 -- 172.21.15.32:0/555330618 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9134066220 msgr2=0x7f913410d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:12.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.246+0000 7f9131d9b700 1 --2- 172.21.15.32:0/555330618 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9134066220 0x7f913410d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:12.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.246+0000 7f9131d9b700 1 -- 172.21.15.32:0/555330618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9124005040 con 0x7f91340fffb0 2024-04-12T05:33:12.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.246+0000 7f9131d9b700 1 --2- 172.21.15.32:0/555330618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91340fffb0 0x7f913410ddd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f912400b3b0 tx=0x7f9124009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:12.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.247+0000 7f912affd700 1 -- 172.21.15.32:0/555330618 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f912400e040 con 0x7f91340fffb0 2024-04-12T05:33:12.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.247+0000 7f912affd700 1 -- 172.21.15.32:0/555330618 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f912400bc50 con 0x7f91340fffb0 2024-04-12T05:33:12.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.247+0000 7f912affd700 1 -- 172.21.15.32:0/555330618 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f912401c630 con 0x7f91340fffb0 2024-04-12T05:33:12.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.247+0000 7f9138d51700 1 -- 172.21.15.32:0/555330618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9134113000 con 0x7f91340fffb0 2024-04-12T05:33:12.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.247+0000 7f9138d51700 1 -- 172.21.15.32:0/555330618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9134113570 con 0x7f91340fffb0 2024-04-12T05:33:12.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.248+0000 7f9138d51700 1 -- 172.21.15.32:0/555330618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f913404c710 con 0x7f91340fffb0 2024-04-12T05:33:12.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.248+0000 7f912affd700 1 -- 172.21.15.32:0/555330618 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f9124013420 con 0x7f91340fffb0 2024-04-12T05:33:12.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.249+0000 7f912affd700 1 --2- 172.21.15.32:0/555330618 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f91200692a0 0x7f912006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:12.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.249+0000 7f912affd700 1 -- 172.21.15.32:0/555330618 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9124087f10 con 0x7f91340fffb0 2024-04-12T05:33:12.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.249+0000 7f913259c700 1 --2- 172.21.15.32:0/555330618 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f91200692a0 0x7f912006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:12.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.249+0000 7f913259c700 1 --2- 172.21.15.32:0/555330618 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f91200692a0 0x7f912006b750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9134065f60 tx=0x7f911c0091a0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:12.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.253+0000 7f912affd700 1 -- 172.21.15.32:0/555330618 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f91240580b0 con 0x7f91340fffb0 2024-04-12T05:33:12.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:12 smithi032 ceph-mon[39731]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:12.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:12 smithi067 ceph-mon[49010]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:12.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.528+0000 7f9138d51700 1 -- 172.21.15.32:0/555330618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f9134105860 con 0x7f91340fffb0 2024-04-12T05:33:12.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.532+0000 7f912affd700 1 -- 172.21.15.32:0/555330618 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7f9124057c40 con 0x7f91340fffb0 2024-04-12T05:33:12.531 INFO:teuthology.orchestra.run.smithi032.stdout:[client.0] 2024-04-12T05:33:12.532 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQCYxxhmkjaVHxAAFREi5mhZYTbVaQF8fY/cfA== 2024-04-12T05:33:12.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.534+0000 7f9138d51700 1 -- 172.21.15.32:0/555330618 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f91200692a0 msgr2=0x7f912006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:12.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.534+0000 7f9138d51700 1 --2- 172.21.15.32:0/555330618 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f91200692a0 0x7f912006b750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9134065f60 tx=0x7f911c0091a0 comp rx=0 tx=0).stop 2024-04-12T05:33:12.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.534+0000 7f9138d51700 1 -- 172.21.15.32:0/555330618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91340fffb0 msgr2=0x7f913410ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:12.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.534+0000 7f9138d51700 1 --2- 172.21.15.32:0/555330618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91340fffb0 0x7f913410ddd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f912400b3b0 tx=0x7f9124009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:12.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.534+0000 7f9138d51700 1 -- 172.21.15.32:0/555330618 shutdown_connections 2024-04-12T05:33:12.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.534+0000 7f9138d51700 1 --2- 172.21.15.32:0/555330618 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f91200692a0 0x7f912006b750 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:12.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.534+0000 7f9138d51700 1 --2- 172.21.15.32:0/555330618 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9134066220 0x7f913410d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:12.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.534+0000 7f9138d51700 1 --2- 172.21.15.32:0/555330618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91340fffb0 0x7f913410ddd0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:12.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.534+0000 7f9138d51700 1 -- 172.21.15.32:0/555330618 >> 172.21.15.32:0/555330618 conn(0x7f91340f7460 msgr2=0x7f91340f78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:12.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.534+0000 7f9138d51700 1 -- 172.21.15.32:0/555330618 shutdown_connections 2024-04-12T05:33:12.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:12.534+0000 7f9138d51700 1 -- 172.21.15.32:0/555330618 wait complete. 2024-04-12T05:33:13.171 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-12T05:33:13.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-12T05:33:13.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-12T05:33:13.215 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-12T05:33:13.503 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi067/config 2024-04-12T05:33:13.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:13 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/555330618' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-12T05:33:13.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:13 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/555330618' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-12T05:33:13.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:13 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/555330618' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-12T05:33:13.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:13 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/555330618' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-12T05:33:14.422 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:14 smithi067 ceph-mon[49010]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:14.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:14 smithi032 ceph-mon[39731]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:15.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.240+0000 7f21997bc700 1 -- 172.21.15.67:0/2324506543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21940fce80 msgr2=0x7f21940fd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:15.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.240+0000 7f21997bc700 1 --2- 172.21.15.67:0/2324506543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21940fce80 0x7f21940fd290 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2188004940 tx=0x7f2188009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:15.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.240+0000 7f21997bc700 1 -- 172.21.15.67:0/2324506543 shutdown_connections 2024-04-12T05:33:15.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.240+0000 7f21997bc700 1 --2- 172.21.15.67:0/2324506543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21940fdff0 0x7f21940fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:15.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.240+0000 7f21997bc700 1 --2- 172.21.15.67:0/2324506543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21940fce80 0x7f21940fd290 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:15.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.240+0000 7f21997bc700 1 -- 172.21.15.67:0/2324506543 >> 172.21.15.67:0/2324506543 conn(0x7f21940f8580 msgr2=0x7f21940fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:15.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.240+0000 7f21997bc700 1 -- 172.21.15.67:0/2324506543 shutdown_connections 2024-04-12T05:33:15.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.240+0000 7f21997bc700 1 -- 172.21.15.67:0/2324506543 wait complete. 2024-04-12T05:33:15.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.240+0000 7f21997bc700 1 Processor -- start 2024-04-12T05:33:15.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.241+0000 7f21997bc700 1 -- start start 2024-04-12T05:33:15.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.241+0000 7f21997bc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21940fce80 0x7f2194071c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:15.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.241+0000 7f21997bc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21940fdff0 0x7f2194070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:15.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.241+0000 7f21997bc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2194072260 con 0x7f21940fce80 2024-04-12T05:33:15.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.241+0000 7f21997bc700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2194070850 con 0x7f21940fdff0 2024-04-12T05:33:15.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.241+0000 7f21927fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21940fdff0 0x7f2194070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:15.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.241+0000 7f21927fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21940fdff0 0x7f2194070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53192/0 (socket says 172.21.15.67:53192) 2024-04-12T05:33:15.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.241+0000 7f21927fc700 1 -- 172.21.15.67:0/3939227894 learned_addr learned my addr 172.21.15.67:0/3939227894 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-12T05:33:15.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.241+0000 7f2192ffd700 1 --2- 172.21.15.67:0/3939227894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21940fce80 0x7f2194071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:15.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.241+0000 7f21927fc700 1 -- 172.21.15.67:0/3939227894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21940fce80 msgr2=0x7f2194071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:15.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.241+0000 7f21927fc700 1 --2- 172.21.15.67:0/3939227894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21940fce80 0x7f2194071c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:15.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.241+0000 7f21927fc700 1 -- 172.21.15.67:0/3939227894 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2188005040 con 0x7f21940fdff0 2024-04-12T05:33:15.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.241+0000 7f21927fc700 1 --2- 172.21.15.67:0/3939227894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21940fdff0 0x7f2194070310 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2184002a80 tx=0x7f21840046b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:15.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.241+0000 7f217b7fe700 1 -- 172.21.15.67:0/3939227894 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2184007cc0 con 0x7f21940fdff0 2024-04-12T05:33:15.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.242+0000 7f217b7fe700 1 -- 172.21.15.67:0/3939227894 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f218400f040 con 0x7f21940fdff0 2024-04-12T05:33:15.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.242+0000 7f21997bc700 1 -- 172.21.15.67:0/3939227894 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2194070b00 con 0x7f21940fdff0 2024-04-12T05:33:15.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.242+0000 7f217b7fe700 1 -- 172.21.15.67:0/3939227894 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f218400da60 con 0x7f21940fdff0 2024-04-12T05:33:15.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.242+0000 7f21997bc700 1 -- 172.21.15.67:0/3939227894 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21940710c0 con 0x7f21940fdff0 2024-04-12T05:33:15.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.243+0000 7f21997bc700 1 -- 172.21.15.67:0/3939227894 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f219404c710 con 0x7f21940fdff0 2024-04-12T05:33:15.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.243+0000 7f217b7fe700 1 -- 172.21.15.67:0/3939227894 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f218400dbc0 con 0x7f21940fdff0 2024-04-12T05:33:15.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.243+0000 7f217b7fe700 1 --2- 172.21.15.67:0/3939227894 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f217c0692a0 0x7f217c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:15.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.243+0000 7f217b7fe700 1 -- 172.21.15.67:0/3939227894 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2184086b00 con 0x7f21940fdff0 2024-04-12T05:33:15.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.244+0000 7f2192ffd700 1 --2- 172.21.15.67:0/3939227894 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f217c0692a0 0x7f217c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:15.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.244+0000 7f2192ffd700 1 --2- 172.21.15.67:0/3939227894 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f217c0692a0 0x7f217c06b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f218800b3b0 tx=0x7f2188009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:15.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.248+0000 7f217b7fe700 1 -- 172.21.15.67:0/3939227894 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f2184056b80 con 0x7f21940fdff0 2024-04-12T05:33:15.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.521+0000 7f21997bc700 1 -- 172.21.15.67:0/3939227894 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f21941057a0 con 0x7f21940fdff0 2024-04-12T05:33:15.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.526+0000 7f217b7fe700 1 -- 172.21.15.67:0/3939227894 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7f2184056710 con 0x7f21940fdff0 2024-04-12T05:33:15.526 INFO:teuthology.orchestra.run.smithi067.stdout:[client.1] 2024-04-12T05:33:15.526 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCbxxhm8F82HxAADRtGsdLiGRTT66UAOWSSxQ== 2024-04-12T05:33:15.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.528+0000 7f21997bc700 1 -- 172.21.15.67:0/3939227894 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f217c0692a0 msgr2=0x7f217c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:15.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.528+0000 7f21997bc700 1 --2- 172.21.15.67:0/3939227894 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f217c0692a0 0x7f217c06b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f218800b3b0 tx=0x7f2188009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:15.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.528+0000 7f21997bc700 1 -- 172.21.15.67:0/3939227894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21940fdff0 msgr2=0x7f2194070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:15.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.528+0000 7f21997bc700 1 --2- 172.21.15.67:0/3939227894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21940fdff0 0x7f2194070310 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2184002a80 tx=0x7f21840046b0 comp rx=0 tx=0).stop 2024-04-12T05:33:15.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.528+0000 7f21997bc700 1 -- 172.21.15.67:0/3939227894 shutdown_connections 2024-04-12T05:33:15.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.528+0000 7f21997bc700 1 --2- 172.21.15.67:0/3939227894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21940fce80 0x7f2194071c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:15.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.528+0000 7f21997bc700 1 --2- 172.21.15.67:0/3939227894 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f217c0692a0 0x7f217c06b750 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:15.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.528+0000 7f21997bc700 1 --2- 172.21.15.67:0/3939227894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21940fdff0 0x7f2194070310 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:15.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.528+0000 7f21997bc700 1 -- 172.21.15.67:0/3939227894 >> 172.21.15.67:0/3939227894 conn(0x7f21940f8580 msgr2=0x7f21940f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:15.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.528+0000 7f21997bc700 1 -- 172.21.15.67:0/3939227894 shutdown_connections 2024-04-12T05:33:15.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-12T05:33:15.528+0000 7f21997bc700 1 -- 172.21.15.67:0/3939227894 wait complete. 2024-04-12T05:33:16.177 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-12T05:33:16.177 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-12T05:33:16.177 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-12T05:33:16.222 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-12T05:33:16.222 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-12T05:33:16.222 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph mgr dump --format=json 2024-04-12T05:33:16.509 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:33:16.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:16 smithi067 ceph-mon[49010]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:16.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:16 smithi067 ceph-mon[49010]: from='client.? 172.21.15.67:0/3939227894' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-12T05:33:16.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:16 smithi067 ceph-mon[49010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-12T05:33:16.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:16 smithi067 ceph-mon[49010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-12T05:33:16.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:16 smithi032 ceph-mon[39731]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:16.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:16 smithi032 ceph-mon[39731]: from='client.? 172.21.15.67:0/3939227894' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-12T05:33:16.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:16 smithi032 ceph-mon[39731]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-12T05:33:16.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:16 smithi032 ceph-mon[39731]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-12T05:33:18.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.234+0000 7f16e0492700 1 -- 172.21.15.32:0/3890761436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16d8102e10 msgr2=0x7f16d81031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:18.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.234+0000 7f16e0492700 1 --2- 172.21.15.32:0/3890761436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16d8102e10 0x7f16d81031e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f16cc004940 tx=0x7f16cc009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:18.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.234+0000 7f16e0492700 1 -- 172.21.15.32:0/3890761436 shutdown_connections 2024-04-12T05:33:18.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.234+0000 7f16e0492700 1 --2- 172.21.15.32:0/3890761436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16d81037b0 0x7f16d80fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:18.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.234+0000 7f16e0492700 1 --2- 172.21.15.32:0/3890761436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16d8102e10 0x7f16d81031e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:18.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.234+0000 7f16e0492700 1 -- 172.21.15.32:0/3890761436 >> 172.21.15.32:0/3890761436 conn(0x7f16d80f89a0 msgr2=0x7f16d80fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:18.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.234+0000 7f16e0492700 1 -- 172.21.15.32:0/3890761436 shutdown_connections 2024-04-12T05:33:18.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.234+0000 7f16e0492700 1 -- 172.21.15.32:0/3890761436 wait complete. 2024-04-12T05:33:18.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.235+0000 7f16e0492700 1 Processor -- start 2024-04-12T05:33:18.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.235+0000 7f16e0492700 1 -- start start 2024-04-12T05:33:18.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.235+0000 7f16e0492700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16d8102e10 0x7f16d8113ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:18.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.235+0000 7f16e0492700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16d81037b0 0x7f16d8113ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:18.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.235+0000 7f16e0492700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d81146d0 con 0x7f16d8102e10 2024-04-12T05:33:18.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.235+0000 7f16e0492700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d810da30 con 0x7f16d81037b0 2024-04-12T05:33:18.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.235+0000 7f16de22e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16d8102e10 0x7f16d8113ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:18.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.235+0000 7f16de22e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16d8102e10 0x7f16d8113ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59124/0 (socket says 172.21.15.32:59124) 2024-04-12T05:33:18.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.235+0000 7f16de22e700 1 -- 172.21.15.32:0/2359462456 learned_addr learned my addr 172.21.15.32:0/2359462456 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:33:18.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.235+0000 7f16dda2d700 1 --2- 172.21.15.32:0/2359462456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16d81037b0 0x7f16d8113ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:18.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.235+0000 7f16de22e700 1 -- 172.21.15.32:0/2359462456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16d81037b0 msgr2=0x7f16d8113ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:18.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.235+0000 7f16de22e700 1 --2- 172.21.15.32:0/2359462456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16d81037b0 0x7f16d8113ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:18.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.235+0000 7f16de22e700 1 -- 172.21.15.32:0/2359462456 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16cc005040 con 0x7f16d8102e10 2024-04-12T05:33:18.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.235+0000 7f16dda2d700 1 --2- 172.21.15.32:0/2359462456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16d81037b0 0x7f16d8113ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:33:18.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.235+0000 7f16de22e700 1 --2- 172.21.15.32:0/2359462456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16d8102e10 0x7f16d8113ab0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f16cc00b3b0 tx=0x7f16cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:18.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.236+0000 7f16caffd700 1 -- 172.21.15.32:0/2359462456 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16cc00e040 con 0x7f16d8102e10 2024-04-12T05:33:18.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.236+0000 7f16caffd700 1 -- 172.21.15.32:0/2359462456 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f16cc013420 con 0x7f16d8102e10 2024-04-12T05:33:18.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.236+0000 7f16e0492700 1 -- 172.21.15.32:0/2359462456 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16d810dcb0 con 0x7f16d8102e10 2024-04-12T05:33:18.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.236+0000 7f16caffd700 1 -- 172.21.15.32:0/2359462456 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16cc01c630 con 0x7f16d8102e10 2024-04-12T05:33:18.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.236+0000 7f16e0492700 1 -- 172.21.15.32:0/2359462456 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16d810e270 con 0x7f16d8102e10 2024-04-12T05:33:18.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.237+0000 7f16e0492700 1 -- 172.21.15.32:0/2359462456 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16d80fe560 con 0x7f16d8102e10 2024-04-12T05:33:18.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.237+0000 7f16caffd700 1 -- 172.21.15.32:0/2359462456 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f16cc013970 con 0x7f16d8102e10 2024-04-12T05:33:18.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.237+0000 7f16caffd700 1 --2- 172.21.15.32:0/2359462456 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f16c40692a0 0x7f16c406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:18.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.238+0000 7f16caffd700 1 -- 172.21.15.32:0/2359462456 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f16cc087fc0 con 0x7f16d8102e10 2024-04-12T05:33:18.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.238+0000 7f16dda2d700 1 --2- 172.21.15.32:0/2359462456 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f16c40692a0 0x7f16c406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:18.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.238+0000 7f16dda2d700 1 --2- 172.21.15.32:0/2359462456 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f16c40692a0 0x7f16c406b750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006fd0 tx=0x7f16d4008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:18.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.242+0000 7f16caffd700 1 -- 172.21.15.32:0/2359462456 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f16cc058160 con 0x7f16d8102e10 2024-04-12T05:33:18.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:18 smithi032 ceph-mon[39731]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:18.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.491+0000 7f16e0492700 1 -- 172.21.15.32:0/2359462456 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f16d805f5f0 con 0x7f16d8102e10 2024-04-12T05:33:18.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.494+0000 7f16caffd700 1 -- 172.21.15.32:0/2359462456 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+167164 (secure 0 0 0) 0x7f16cc057cf0 con 0x7f16d8102e10 2024-04-12T05:33:18.496 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:33:18.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.496+0000 7f16e0492700 1 -- 172.21.15.32:0/2359462456 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f16c40692a0 msgr2=0x7f16c406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:18.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.496+0000 7f16e0492700 1 --2- 172.21.15.32:0/2359462456 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f16c40692a0 0x7f16c406b750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006fd0 tx=0x7f16d4008040 comp rx=0 tx=0).stop 2024-04-12T05:33:18.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.496+0000 7f16e0492700 1 -- 172.21.15.32:0/2359462456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16d8102e10 msgr2=0x7f16d8113ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:18.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.496+0000 7f16e0492700 1 --2- 172.21.15.32:0/2359462456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16d8102e10 0x7f16d8113ab0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f16cc00b3b0 tx=0x7f16cc009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:18.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.496+0000 7f16e0492700 1 -- 172.21.15.32:0/2359462456 shutdown_connections 2024-04-12T05:33:18.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.496+0000 7f16e0492700 1 --2- 172.21.15.32:0/2359462456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16d8102e10 0x7f16d8113ab0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:18.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.496+0000 7f16e0492700 1 --2- 172.21.15.32:0/2359462456 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f16c40692a0 0x7f16c406b750 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:18.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.496+0000 7f16e0492700 1 --2- 172.21.15.32:0/2359462456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16d81037b0 0x7f16d8113ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.497+0000 7f16e0492700 1 -- 172.21.15.32:0/2359462456 >> 172.21.15.32:0/2359462456 conn(0x7f16d80f89a0 msgr2=0x7f16d80f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.497+0000 7f16e0492700 1 -- 172.21.15.32:0/2359462456 shutdown_connections 2024-04-12T05:33:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:18.497+0000 7f16e0492700 1 -- 172.21.15.32:0/2359462456 wait complete. 2024-04-12T05:33:18.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:18 smithi067 ceph-mon[49010]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:19.224 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":19,"active_gid":14208,"active_name":"smithi032.hrwxrh","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":595990262},{"type":"v1","addr":"172.21.15.32:6801","nonce":595990262}]},"active_addr":"172.21.15.32:6801/595990262","active_change":"2024-04-12T05:29:25.054245+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14228,"name":"smithi067.dwichg","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.32:8443/","prometheus":"http://172.21.15.32:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":2247439109}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":2105642006}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":260214199}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":368340628}]}]} 2024-04-12T05:33:19.228 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-12T05:33:19.228 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-12T05:33:19.229 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph osd dump --format=json 2024-04-12T05:33:19.508 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:33:19.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:19 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/2359462456' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-12T05:33:19.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:19 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/2359462456' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-12T05:33:20.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:20 smithi067 ceph-mon[49010]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:20.600 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:20 smithi032 ceph-mon[39731]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:21.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.364+0000 7ff8f6216700 1 -- 172.21.15.32:0/751950690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8f0102710 msgr2=0x7ff8f00fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:21.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.364+0000 7ff8f6216700 1 --2- 172.21.15.32:0/751950690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8f0102710 0x7ff8f00fbd80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8004940 tx=0x7ff8d8009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:21.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.364+0000 7ff8f6216700 1 -- 172.21.15.32:0/751950690 shutdown_connections 2024-04-12T05:33:21.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.364+0000 7ff8f6216700 1 --2- 172.21.15.32:0/751950690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8f0102710 0x7ff8f00fbd80 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:21.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.364+0000 7ff8f6216700 1 --2- 172.21.15.32:0/751950690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8f0101d70 0x7ff8f0102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:21.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.364+0000 7ff8f6216700 1 -- 172.21.15.32:0/751950690 >> 172.21.15.32:0/751950690 conn(0x7ff8f0072bc0 msgr2=0x7ff8f0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:21.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.364+0000 7ff8f6216700 1 -- 172.21.15.32:0/751950690 shutdown_connections 2024-04-12T05:33:21.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.364+0000 7ff8f6216700 1 -- 172.21.15.32:0/751950690 wait complete. 2024-04-12T05:33:21.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.364+0000 7ff8f6216700 1 Processor -- start 2024-04-12T05:33:21.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.365+0000 7ff8f6216700 1 -- start start 2024-04-12T05:33:21.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.365+0000 7ff8f6216700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8f0101d70 0x7ff8f0102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:21.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.365+0000 7ff8f6216700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8f0102710 0x7ff8f0103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:21.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.365+0000 7ff8f6216700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f01036a0 con 0x7ff8f0101d70 2024-04-12T05:33:21.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.365+0000 7ff8f6216700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f01037e0 con 0x7ff8f0102710 2024-04-12T05:33:21.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.365+0000 7ff8ef7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8f0101d70 0x7ff8f0102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:21.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.365+0000 7ff8ef7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8f0101d70 0x7ff8f0102bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59140/0 (socket says 172.21.15.32:59140) 2024-04-12T05:33:21.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.365+0000 7ff8ef7fe700 1 -- 172.21.15.32:0/1843820048 learned_addr learned my addr 172.21.15.32:0/1843820048 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:33:21.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.365+0000 7ff8eeffd700 1 --2- 172.21.15.32:0/1843820048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8f0102710 0x7ff8f0103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:21.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.365+0000 7ff8ef7fe700 1 -- 172.21.15.32:0/1843820048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8f0102710 msgr2=0x7ff8f0103110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:21.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.365+0000 7ff8ef7fe700 1 --2- 172.21.15.32:0/1843820048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8f0102710 0x7ff8f0103110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:21.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.365+0000 7ff8ef7fe700 1 -- 172.21.15.32:0/1843820048 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d8005040 con 0x7ff8f0101d70 2024-04-12T05:33:21.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.365+0000 7ff8eeffd700 1 --2- 172.21.15.32:0/1843820048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8f0102710 0x7ff8f0103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:33:21.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.365+0000 7ff8ef7fe700 1 --2- 172.21.15.32:0/1843820048 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8f0101d70 0x7ff8f0102bd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff8e000cac0 tx=0x7ff8e0007ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:21.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.366+0000 7ff8e7fff700 1 -- 172.21.15.32:0/1843820048 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8e000aa50 con 0x7ff8f0101d70 2024-04-12T05:33:21.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.366+0000 7ff8e7fff700 1 -- 172.21.15.32:0/1843820048 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff8e000abb0 con 0x7ff8f0101d70 2024-04-12T05:33:21.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.366+0000 7ff8f6216700 1 -- 172.21.15.32:0/1843820048 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f0119dc0 con 0x7ff8f0101d70 2024-04-12T05:33:21.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.366+0000 7ff8e7fff700 1 -- 172.21.15.32:0/1843820048 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8e00115e0 con 0x7ff8f0101d70 2024-04-12T05:33:21.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.366+0000 7ff8f6216700 1 -- 172.21.15.32:0/1843820048 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f011a240 con 0x7ff8f0101d70 2024-04-12T05:33:21.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.367+0000 7ff8f6216700 1 -- 172.21.15.32:0/1843820048 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f00fd4c0 con 0x7ff8f0101d70 2024-04-12T05:33:21.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.367+0000 7ff8e7fff700 1 -- 172.21.15.32:0/1843820048 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ff8e0012440 con 0x7ff8f0101d70 2024-04-12T05:33:21.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.367+0000 7ff8e7fff700 1 --2- 172.21.15.32:0/1843820048 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7ff8dc0692a0 0x7ff8dc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:21.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.368+0000 7ff8e7fff700 1 -- 172.21.15.32:0/1843820048 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff8e00875a0 con 0x7ff8f0101d70 2024-04-12T05:33:21.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.368+0000 7ff8eeffd700 1 --2- 172.21.15.32:0/1843820048 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7ff8dc0692a0 0x7ff8dc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:21.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.368+0000 7ff8eeffd700 1 --2- 172.21.15.32:0/1843820048 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7ff8dc0692a0 0x7ff8dc06b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8000c00 tx=0x7ff8d8009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:21.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.372+0000 7ff8e7fff700 1 -- 172.21.15.32:0/1843820048 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ff8e0053b90 con 0x7ff8f0101d70 2024-04-12T05:33:21.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.571+0000 7ff8f6216700 1 -- 172.21.15.32:0/1843820048 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8f004bf60 con 0x7ff8f0101d70 2024-04-12T05:33:21.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.572+0000 7ff8e7fff700 1 -- 172.21.15.32:0/1843820048 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11031 (secure 0 0 0) 0x7ff8e00571b0 con 0x7ff8f0101d70 2024-04-12T05:33:21.572 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:33:21.572 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":40,"fsid":"37b58ce2-f88d-11ee-bc8c-c7b262605968","created":"2024-04-12T05:27:07.779407+0000","modified":"2024-04-12T05:33:00.514069+0000","last_up_change":"2024-04-12T05:32:59.508815+0000","last_in_change":"2024-04-12T05:32:39.814568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T05:31:39.568876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"c1f69d2a-5e37-47b8-b68f-0568af894b91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":3061707028},{"type":"v1","addr":"172.21.15.32:6803","nonce":3061707028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":3061707028},{"type":"v1","addr":"172.21.15.32:6805","nonce":3061707028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3061707028},{"type":"v1","addr":"172.21.15.32:6809","nonce":3061707028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":3061707028},{"type":"v1","addr":"172.21.15.32:6807","nonce":3061707028}]},"public_addr":"172.21.15.32:6803/3061707028","cluster_addr":"172.21.15.32:6805/3061707028","heartbeat_back_addr":"172.21.15.32:6809/3061707028","heartbeat_front_addr":"172.21.15.32:6807/3061707028","state":["exists","up"]},{"osd":1,"uuid":"86b93557-6e6f-4b84-a5f5-5bce4aa65a32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":164843120},{"type":"v1","addr":"172.21.15.32:6811","nonce":164843120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":164843120},{"type":"v1","addr":"172.21.15.32:6813","nonce":164843120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":164843120},{"type":"v1","addr":"172.21.15.32:6817","nonce":164843120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":164843120},{"type":"v1","addr":"172.21.15.32:6815","nonce":164843120}]},"public_addr":"172.21.15.32:6811/164843120","cluster_addr":"172.21.15.32:6813/164843120","heartbeat_back_addr":"172.21.15.32:6817/164843120","heartbeat_front_addr":"172.21.15.32:6815/164843120","state":["exists","up"]},{"osd":2,"uuid":"d1556613-e0de-4b0e-bb7d-e90361ec9007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":2698911336},{"type":"v1","addr":"172.21.15.32:6819","nonce":2698911336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":2698911336},{"type":"v1","addr":"172.21.15.32:6821","nonce":2698911336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":2698911336},{"type":"v1","addr":"172.21.15.32:6825","nonce":2698911336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":2698911336},{"type":"v1","addr":"172.21.15.32:6823","nonce":2698911336}]},"public_addr":"172.21.15.32:6819/2698911336","cluster_addr":"172.21.15.32:6821/2698911336","heartbeat_back_addr":"172.21.15.32:6825/2698911336","heartbeat_front_addr":"172.21.15.32:6823/2698911336","state":["exists","up"]},{"osd":3,"uuid":"213dc62e-a894-4bc7-a2dd-ca98d1313d6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1205580917},{"type":"v1","addr":"172.21.15.67:6801","nonce":1205580917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1205580917},{"type":"v1","addr":"172.21.15.67:6803","nonce":1205580917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1205580917},{"type":"v1","addr":"172.21.15.67:6807","nonce":1205580917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1205580917},{"type":"v1","addr":"172.21.15.67:6805","nonce":1205580917}]},"public_addr":"172.21.15.67:6801/1205580917","cluster_addr":"172.21.15.67:6803/1205580917","heartbeat_back_addr":"172.21.15.67:6807/1205580917","heartbeat_front_addr":"172.21.15.67:6805/1205580917","state":["exists","up"]},{"osd":4,"uuid":"8a0ff860-fff7-434a-9e78-f2020bc84966","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":286946785},{"type":"v1","addr":"172.21.15.67:6809","nonce":286946785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":286946785},{"type":"v1","addr":"172.21.15.67:6811","nonce":286946785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":286946785},{"type":"v1","addr":"172.21.15.67:6815","nonce":286946785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":286946785},{"type":"v1","addr":"172.21.15.67:6813","nonce":286946785}]},"public_addr":"172.21.15.67:6809/286946785","cluster_addr":"172.21.15.67:6811/286946785","heartbeat_back_addr":"172.21.15.67:6815/286946785","heartbeat_front_addr":"172.21.15.67:6813/286946785","state":["exists","up"]},{"osd":5,"uuid":"3486e5c9-9f44-4f32-a186-fc6198f09549","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1361296892},{"type":"v1","addr":"172.21.15.67:6817","nonce":1361296892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":1361296892},{"type":"v1","addr":"172.21.15.67:6819","nonce":1361296892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":1361296892},{"type":"v1","addr":"172.21.15.67:6823","nonce":1361296892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1361296892},{"type":"v1","addr":"172.21.15.67:6821","nonce":1361296892}]},"public_addr":"172.21.15.67:6817/1361296892","cluster_addr":"172.21.15.67:6819/1361296892","heartbeat_back_addr":"172.21.15.67:6823/1361296892","heartbeat_front_addr":"172.21.15.67:6821/1361296892","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:30:38.748993+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:31:08.677208+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:31:36.979481+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:32:02.668398+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:32:30.143788+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:32:57.613087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6801/3971961003":"2024-04-13T05:29:25.054120+0000","172.21.15.32:0/4112976740":"2024-04-13T05:29:25.054120+0000","172.21.15.32:0/146872823":"2024-04-13T05:28:14.470413+0000","172.21.15.32:0/3082743475":"2024-04-13T05:28:14.470413+0000","172.21.15.32:0/2162993076":"2024-04-13T05:28:14.470413+0000","172.21.15.32:6801/1962278696":"2024-04-13T05:28:14.470413+0000","172.21.15.32:0/795906028":"2024-04-13T05:29:25.054120+0000","172.21.15.32:0/138657961":"2024-04-13T05:27:40.952118+0000","172.21.15.32:0/1643708423":"2024-04-13T05:27:40.952118+0000","172.21.15.32:0/674029871":"2024-04-13T05:29:25.054120+0000","172.21.15.32:6800/656201914":"2024-04-13T05:27:40.952118+0000","172.21.15.32:6800/3971961003":"2024-04-13T05:29:25.054120+0000","172.21.15.32:6801/656201914":"2024-04-13T05:27:40.952118+0000","172.21.15.32:6800/1962278696":"2024-04-13T05:28:14.470413+0000","172.21.15.32:0/4031244002":"2024-04-13T05:27:40.952118+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T05:33:21.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.574+0000 7ff8f6216700 1 -- 172.21.15.32:0/1843820048 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7ff8dc0692a0 msgr2=0x7ff8dc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:21.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.574+0000 7ff8f6216700 1 --2- 172.21.15.32:0/1843820048 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7ff8dc0692a0 0x7ff8dc06b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8000c00 tx=0x7ff8d8009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:21.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.574+0000 7ff8f6216700 1 -- 172.21.15.32:0/1843820048 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8f0101d70 msgr2=0x7ff8f0102bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:21.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.574+0000 7ff8f6216700 1 --2- 172.21.15.32:0/1843820048 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8f0101d70 0x7ff8f0102bd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff8e000cac0 tx=0x7ff8e0007ee0 comp rx=0 tx=0).stop 2024-04-12T05:33:21.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.574+0000 7ff8f6216700 1 -- 172.21.15.32:0/1843820048 shutdown_connections 2024-04-12T05:33:21.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.574+0000 7ff8f6216700 1 --2- 172.21.15.32:0/1843820048 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8f0101d70 0x7ff8f0102bd0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:21.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.574+0000 7ff8f6216700 1 --2- 172.21.15.32:0/1843820048 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7ff8dc0692a0 0x7ff8dc06b750 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:21.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.574+0000 7ff8f6216700 1 --2- 172.21.15.32:0/1843820048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8f0102710 0x7ff8f0103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:21.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.574+0000 7ff8f6216700 1 -- 172.21.15.32:0/1843820048 >> 172.21.15.32:0/1843820048 conn(0x7ff8f0072bc0 msgr2=0x7ff8f0065a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:21.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.574+0000 7ff8f6216700 1 -- 172.21.15.32:0/1843820048 shutdown_connections 2024-04-12T05:33:21.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:21.574+0000 7ff8f6216700 1 -- 172.21.15.32:0/1843820048 wait complete. 2024-04-12T05:33:22.240 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-12T05:33:22.240 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph osd dump --format=json 2024-04-12T05:33:22.519 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:33:22.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:22 smithi067 ceph-mon[49010]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:22.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:22 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/1843820048' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T05:33:22.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:22 smithi032 ceph-mon[39731]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:22.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:22 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/1843820048' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T05:33:24.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.183+0000 7f99f7be6700 1 -- 172.21.15.32:0/43900589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f00fde90 msgr2=0x7f99f00fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:24.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.183+0000 7f99f7be6700 1 --2- 172.21.15.32:0/43900589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f00fde90 0x7f99f00fe260 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f99e0004940 tx=0x7f99e0009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:24.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.183+0000 7f99f7be6700 1 -- 172.21.15.32:0/43900589 shutdown_connections 2024-04-12T05:33:24.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.183+0000 7f99f7be6700 1 --2- 172.21.15.32:0/43900589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99f00fe830 0x7f99f0106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:24.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.183+0000 7f99f7be6700 1 --2- 172.21.15.32:0/43900589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f00fde90 0x7f99f00fe260 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:24.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.183+0000 7f99f7be6700 1 -- 172.21.15.32:0/43900589 >> 172.21.15.32:0/43900589 conn(0x7f99f0072bc0 msgr2=0x7f99f0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:24.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.183+0000 7f99f7be6700 1 -- 172.21.15.32:0/43900589 shutdown_connections 2024-04-12T05:33:24.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.183+0000 7f99f7be6700 1 -- 172.21.15.32:0/43900589 wait complete. 2024-04-12T05:33:24.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.184+0000 7f99f7be6700 1 Processor -- start 2024-04-12T05:33:24.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.184+0000 7f99f7be6700 1 -- start start 2024-04-12T05:33:24.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.184+0000 7f99f7be6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99f00fde90 0x7f99f0113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:24.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.184+0000 7f99f7be6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f00fe830 0x7f99f0114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:24.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.184+0000 7f99f7be6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f0114710 con 0x7f99f00fe830 2024-04-12T05:33:24.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.184+0000 7f99f7be6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f010da70 con 0x7f99f00fde90 2024-04-12T05:33:24.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.184+0000 7f99f5181700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f00fe830 0x7f99f0114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:24.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.184+0000 7f99f5181700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f00fe830 0x7f99f0114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42986/0 (socket says 172.21.15.32:42986) 2024-04-12T05:33:24.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.184+0000 7f99f5181700 1 -- 172.21.15.32:0/980890198 learned_addr learned my addr 172.21.15.32:0/980890198 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:33:24.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.184+0000 7f99f5982700 1 --2- 172.21.15.32:0/980890198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99f00fde90 0x7f99f0113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:24.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.185+0000 7f99f5181700 1 -- 172.21.15.32:0/980890198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99f00fde90 msgr2=0x7f99f0113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:24.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.185+0000 7f99f5181700 1 --2- 172.21.15.32:0/980890198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99f00fde90 0x7f99f0113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:24.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.185+0000 7f99f5181700 1 -- 172.21.15.32:0/980890198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e0005040 con 0x7f99f00fe830 2024-04-12T05:33:24.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.185+0000 7f99f5181700 1 --2- 172.21.15.32:0/980890198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f00fe830 0x7f99f0114030 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f99e400b790 tx=0x7f99e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:24.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.185+0000 7f99ee7fc700 1 -- 172.21.15.32:0/980890198 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99e400f6f0 con 0x7f99f00fe830 2024-04-12T05:33:24.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.185+0000 7f99f7be6700 1 -- 172.21.15.32:0/980890198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99f010dd50 con 0x7f99f00fe830 2024-04-12T05:33:24.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.185+0000 7f99ee7fc700 1 -- 172.21.15.32:0/980890198 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f99e400fd00 con 0x7f99f00fe830 2024-04-12T05:33:24.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.185+0000 7f99ee7fc700 1 -- 172.21.15.32:0/980890198 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99e40103a0 con 0x7f99f00fe830 2024-04-12T05:33:24.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.185+0000 7f99f7be6700 1 -- 172.21.15.32:0/980890198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f010e290 con 0x7f99f00fe830 2024-04-12T05:33:24.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.186+0000 7f99f7be6700 1 -- 172.21.15.32:0/980890198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99f0065e30 con 0x7f99f00fe830 2024-04-12T05:33:24.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.186+0000 7f99ee7fc700 1 -- 172.21.15.32:0/980890198 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f99e400f850 con 0x7f99f00fe830 2024-04-12T05:33:24.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.187+0000 7f99ee7fc700 1 --2- 172.21.15.32:0/980890198 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f99d80692a0 0x7f99d806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:24.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.187+0000 7f99ee7fc700 1 -- 172.21.15.32:0/980890198 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f99e4086c60 con 0x7f99f00fe830 2024-04-12T05:33:24.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.187+0000 7f99f5982700 1 --2- 172.21.15.32:0/980890198 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f99d80692a0 0x7f99d806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:24.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.187+0000 7f99f5982700 1 --2- 172.21.15.32:0/980890198 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f99d80692a0 0x7f99d806b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f99e000b7e0 tx=0x7f99e0009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:24.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.191+0000 7f99ee7fc700 1 -- 172.21.15.32:0/980890198 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f99e4056d60 con 0x7f99f00fe830 2024-04-12T05:33:24.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:24 smithi032 ceph-mon[39731]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:24.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.395+0000 7f99f7be6700 1 -- 172.21.15.32:0/980890198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99f0109e40 con 0x7f99f00fe830 2024-04-12T05:33:24.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.396+0000 7f99ee7fc700 1 -- 172.21.15.32:0/980890198 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11031 (secure 0 0 0) 0x7f99e40568f0 con 0x7f99f00fe830 2024-04-12T05:33:24.395 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:33:24.396 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":40,"fsid":"37b58ce2-f88d-11ee-bc8c-c7b262605968","created":"2024-04-12T05:27:07.779407+0000","modified":"2024-04-12T05:33:00.514069+0000","last_up_change":"2024-04-12T05:32:59.508815+0000","last_in_change":"2024-04-12T05:32:39.814568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T05:31:39.568876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"c1f69d2a-5e37-47b8-b68f-0568af894b91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":3061707028},{"type":"v1","addr":"172.21.15.32:6803","nonce":3061707028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":3061707028},{"type":"v1","addr":"172.21.15.32:6805","nonce":3061707028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3061707028},{"type":"v1","addr":"172.21.15.32:6809","nonce":3061707028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":3061707028},{"type":"v1","addr":"172.21.15.32:6807","nonce":3061707028}]},"public_addr":"172.21.15.32:6803/3061707028","cluster_addr":"172.21.15.32:6805/3061707028","heartbeat_back_addr":"172.21.15.32:6809/3061707028","heartbeat_front_addr":"172.21.15.32:6807/3061707028","state":["exists","up"]},{"osd":1,"uuid":"86b93557-6e6f-4b84-a5f5-5bce4aa65a32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":164843120},{"type":"v1","addr":"172.21.15.32:6811","nonce":164843120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":164843120},{"type":"v1","addr":"172.21.15.32:6813","nonce":164843120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":164843120},{"type":"v1","addr":"172.21.15.32:6817","nonce":164843120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":164843120},{"type":"v1","addr":"172.21.15.32:6815","nonce":164843120}]},"public_addr":"172.21.15.32:6811/164843120","cluster_addr":"172.21.15.32:6813/164843120","heartbeat_back_addr":"172.21.15.32:6817/164843120","heartbeat_front_addr":"172.21.15.32:6815/164843120","state":["exists","up"]},{"osd":2,"uuid":"d1556613-e0de-4b0e-bb7d-e90361ec9007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":2698911336},{"type":"v1","addr":"172.21.15.32:6819","nonce":2698911336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":2698911336},{"type":"v1","addr":"172.21.15.32:6821","nonce":2698911336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":2698911336},{"type":"v1","addr":"172.21.15.32:6825","nonce":2698911336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":2698911336},{"type":"v1","addr":"172.21.15.32:6823","nonce":2698911336}]},"public_addr":"172.21.15.32:6819/2698911336","cluster_addr":"172.21.15.32:6821/2698911336","heartbeat_back_addr":"172.21.15.32:6825/2698911336","heartbeat_front_addr":"172.21.15.32:6823/2698911336","state":["exists","up"]},{"osd":3,"uuid":"213dc62e-a894-4bc7-a2dd-ca98d1313d6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1205580917},{"type":"v1","addr":"172.21.15.67:6801","nonce":1205580917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1205580917},{"type":"v1","addr":"172.21.15.67:6803","nonce":1205580917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1205580917},{"type":"v1","addr":"172.21.15.67:6807","nonce":1205580917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1205580917},{"type":"v1","addr":"172.21.15.67:6805","nonce":1205580917}]},"public_addr":"172.21.15.67:6801/1205580917","cluster_addr":"172.21.15.67:6803/1205580917","heartbeat_back_addr":"172.21.15.67:6807/1205580917","heartbeat_front_addr":"172.21.15.67:6805/1205580917","state":["exists","up"]},{"osd":4,"uuid":"8a0ff860-fff7-434a-9e78-f2020bc84966","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":286946785},{"type":"v1","addr":"172.21.15.67:6809","nonce":286946785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":286946785},{"type":"v1","addr":"172.21.15.67:6811","nonce":286946785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":286946785},{"type":"v1","addr":"172.21.15.67:6815","nonce":286946785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":286946785},{"type":"v1","addr":"172.21.15.67:6813","nonce":286946785}]},"public_addr":"172.21.15.67:6809/286946785","cluster_addr":"172.21.15.67:6811/286946785","heartbeat_back_addr":"172.21.15.67:6815/286946785","heartbeat_front_addr":"172.21.15.67:6813/286946785","state":["exists","up"]},{"osd":5,"uuid":"3486e5c9-9f44-4f32-a186-fc6198f09549","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1361296892},{"type":"v1","addr":"172.21.15.67:6817","nonce":1361296892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":1361296892},{"type":"v1","addr":"172.21.15.67:6819","nonce":1361296892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":1361296892},{"type":"v1","addr":"172.21.15.67:6823","nonce":1361296892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1361296892},{"type":"v1","addr":"172.21.15.67:6821","nonce":1361296892}]},"public_addr":"172.21.15.67:6817/1361296892","cluster_addr":"172.21.15.67:6819/1361296892","heartbeat_back_addr":"172.21.15.67:6823/1361296892","heartbeat_front_addr":"172.21.15.67:6821/1361296892","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:30:38.748993+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:31:08.677208+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:31:36.979481+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:32:02.668398+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:32:30.143788+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:32:57.613087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6801/3971961003":"2024-04-13T05:29:25.054120+0000","172.21.15.32:0/4112976740":"2024-04-13T05:29:25.054120+0000","172.21.15.32:0/146872823":"2024-04-13T05:28:14.470413+0000","172.21.15.32:0/3082743475":"2024-04-13T05:28:14.470413+0000","172.21.15.32:0/2162993076":"2024-04-13T05:28:14.470413+0000","172.21.15.32:6801/1962278696":"2024-04-13T05:28:14.470413+0000","172.21.15.32:0/795906028":"2024-04-13T05:29:25.054120+0000","172.21.15.32:0/138657961":"2024-04-13T05:27:40.952118+0000","172.21.15.32:0/1643708423":"2024-04-13T05:27:40.952118+0000","172.21.15.32:0/674029871":"2024-04-13T05:29:25.054120+0000","172.21.15.32:6800/656201914":"2024-04-13T05:27:40.952118+0000","172.21.15.32:6800/3971961003":"2024-04-13T05:29:25.054120+0000","172.21.15.32:6801/656201914":"2024-04-13T05:27:40.952118+0000","172.21.15.32:6800/1962278696":"2024-04-13T05:28:14.470413+0000","172.21.15.32:0/4031244002":"2024-04-13T05:27:40.952118+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T05:33:24.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.398+0000 7f99f7be6700 1 -- 172.21.15.32:0/980890198 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f99d80692a0 msgr2=0x7f99d806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:24.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.398+0000 7f99f7be6700 1 --2- 172.21.15.32:0/980890198 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f99d80692a0 0x7f99d806b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f99e000b7e0 tx=0x7f99e0009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:24.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.398+0000 7f99f7be6700 1 -- 172.21.15.32:0/980890198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f00fe830 msgr2=0x7f99f0114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:24.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.398+0000 7f99f7be6700 1 --2- 172.21.15.32:0/980890198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f00fe830 0x7f99f0114030 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f99e400b790 tx=0x7f99e4009250 comp rx=0 tx=0).stop 2024-04-12T05:33:24.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.398+0000 7f99f7be6700 1 -- 172.21.15.32:0/980890198 shutdown_connections 2024-04-12T05:33:24.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.398+0000 7f99f7be6700 1 --2- 172.21.15.32:0/980890198 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f99d80692a0 0x7f99d806b750 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:24.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.398+0000 7f99f7be6700 1 --2- 172.21.15.32:0/980890198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99f00fde90 0x7f99f0113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:24.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.398+0000 7f99f7be6700 1 --2- 172.21.15.32:0/980890198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f00fe830 0x7f99f0114030 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:24.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.398+0000 7f99f7be6700 1 -- 172.21.15.32:0/980890198 >> 172.21.15.32:0/980890198 conn(0x7f99f0072bc0 msgr2=0x7f99f01037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:24.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.398+0000 7f99f7be6700 1 -- 172.21.15.32:0/980890198 shutdown_connections 2024-04-12T05:33:24.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:24.398+0000 7f99f7be6700 1 -- 172.21.15.32:0/980890198 wait complete. 2024-04-12T05:33:24.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:24 smithi067 ceph-mon[49010]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:25.133 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-12T05:33:25.134 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-12T05:33:25.134 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-12T05:33:25.135 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-12T05:33:25.135 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-12T05:33:25.135 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-12T05:33:25.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:25 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/980890198' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T05:33:25.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:25 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:33:25.527 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:33:25.539 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:33:25.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:25 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/980890198' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T05:33:25.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:25 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:33:26.306 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:33:26.308 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:33:26.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:26 smithi032 ceph-mon[39731]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:26.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:26 smithi067 ceph-mon[49010]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:28.233 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:33:28.234 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:33:28.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:28 smithi067 ceph-mon[49010]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:28.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:28 smithi032 ceph-mon[39731]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:30.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.293+0000 7f89edd06700 1 -- 172.21.15.32:0/2314181218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89e8102e10 msgr2=0x7f89e81031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:30.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.293+0000 7f89edd06700 1 --2- 172.21.15.32:0/2314181218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89e8102e10 0x7f89e81031e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f89dc004940 tx=0x7f89dc009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:30.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.293+0000 7f89edd06700 1 -- 172.21.15.32:0/2314181218 shutdown_connections 2024-04-12T05:33:30.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.293+0000 7f89edd06700 1 --2- 172.21.15.32:0/2314181218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89e81037b0 0x7f89e80fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:30.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.293+0000 7f89edd06700 1 --2- 172.21.15.32:0/2314181218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89e8102e10 0x7f89e81031e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:30.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.293+0000 7f89edd06700 1 -- 172.21.15.32:0/2314181218 >> 172.21.15.32:0/2314181218 conn(0x7f89e80f89a0 msgr2=0x7f89e80fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:30.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.293+0000 7f89edd06700 1 -- 172.21.15.32:0/2314181218 shutdown_connections 2024-04-12T05:33:30.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.293+0000 7f89edd06700 1 -- 172.21.15.32:0/2314181218 wait complete. 2024-04-12T05:33:30.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.294+0000 7f89edd06700 1 Processor -- start 2024-04-12T05:33:30.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.294+0000 7f89edd06700 1 -- start start 2024-04-12T05:33:30.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.294+0000 7f89edd06700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89e8102e10 0x7f89e80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:30.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.294+0000 7f89edd06700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89e81037b0 0x7f89e8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:30.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.294+0000 7f89edd06700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e8070e20 con 0x7f89e81037b0 2024-04-12T05:33:30.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.294+0000 7f89edd06700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e8070f60 con 0x7f89e8102e10 2024-04-12T05:33:30.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.294+0000 7f89e6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89e81037b0 0x7f89e8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:30.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.294+0000 7f89e6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89e81037b0 0x7f89e8070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43014/0 (socket says 172.21.15.32:43014) 2024-04-12T05:33:30.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.294+0000 7f89e6ffd700 1 -- 172.21.15.32:0/2767099702 learned_addr learned my addr 172.21.15.32:0/2767099702 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:33:30.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.294+0000 7f89e77fe700 1 --2- 172.21.15.32:0/2767099702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89e8102e10 0x7f89e80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:30.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.294+0000 7f89e6ffd700 1 -- 172.21.15.32:0/2767099702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89e8102e10 msgr2=0x7f89e80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:30.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.294+0000 7f89e6ffd700 1 --2- 172.21.15.32:0/2767099702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89e8102e10 0x7f89e80702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:30.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.294+0000 7f89e6ffd700 1 -- 172.21.15.32:0/2767099702 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89dc005040 con 0x7f89e81037b0 2024-04-12T05:33:30.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.295+0000 7f89e6ffd700 1 --2- 172.21.15.32:0/2767099702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89e81037b0 0x7f89e8070800 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f89d800b790 tx=0x7f89d8009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:30.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.295+0000 7f89cffff700 1 -- 172.21.15.32:0/2767099702 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f89d800f6f0 con 0x7f89e81037b0 2024-04-12T05:33:30.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.295+0000 7f89edd06700 1 -- 172.21.15.32:0/2767099702 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89e8119db0 con 0x7f89e81037b0 2024-04-12T05:33:30.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.295+0000 7f89cffff700 1 -- 172.21.15.32:0/2767099702 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f89d800fd00 con 0x7f89e81037b0 2024-04-12T05:33:30.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.295+0000 7f89cffff700 1 -- 172.21.15.32:0/2767099702 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f89d80103a0 con 0x7f89e81037b0 2024-04-12T05:33:30.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.295+0000 7f89edd06700 1 -- 172.21.15.32:0/2767099702 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e811a2c0 con 0x7f89e81037b0 2024-04-12T05:33:30.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.295+0000 7f89edd06700 1 -- 172.21.15.32:0/2767099702 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f89e80fa450 con 0x7f89e81037b0 2024-04-12T05:33:30.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.296+0000 7f89cffff700 1 -- 172.21.15.32:0/2767099702 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f89d800f850 con 0x7f89e81037b0 2024-04-12T05:33:30.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.296+0000 7f89cffff700 1 --2- 172.21.15.32:0/2767099702 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f89d00692a0 0x7f89d006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:30.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.297+0000 7f89cffff700 1 -- 172.21.15.32:0/2767099702 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f89d8087820 con 0x7f89e81037b0 2024-04-12T05:33:30.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.297+0000 7f89e77fe700 1 --2- 172.21.15.32:0/2767099702 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f89d00692a0 0x7f89d006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:30.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.297+0000 7f89cffff700 1 --2- 172.21.15.32:0/2767099702 >> [v2:172.21.15.32:6810/164843120,v1:172.21.15.32:6811/164843120] conn(0x7f89d006ef70 0x7f89d0071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:30.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.297+0000 7f89cffff700 1 -- 172.21.15.32:0/2767099702 --> [v2:172.21.15.32:6810/164843120,v1:172.21.15.32:6811/164843120] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f89d0071a30 con 0x7f89d006ef70 2024-04-12T05:33:30.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.297+0000 7f89cffff700 1 -- 172.21.15.32:0/2767099702 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f89d8087b60 con 0x7f89e81037b0 2024-04-12T05:33:30.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.297+0000 7f89e7fff700 1 --2- 172.21.15.32:0/2767099702 >> [v2:172.21.15.32:6810/164843120,v1:172.21.15.32:6811/164843120] conn(0x7f89d006ef70 0x7f89d0071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:30.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.297+0000 7f89e77fe700 1 --2- 172.21.15.32:0/2767099702 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f89d00692a0 0x7f89d006b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f89dc004560 tx=0x7f89dc009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:30.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.297+0000 7f89e7fff700 1 --2- 172.21.15.32:0/2767099702 >> [v2:172.21.15.32:6810/164843120,v1:172.21.15.32:6811/164843120] conn(0x7f89d006ef70 0x7f89d0071380 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:30.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.298+0000 7f89cffff700 1 -- 172.21.15.32:0/2767099702 <== osd.1 v2:172.21.15.32:6810/164843120 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f89d0071a30 con 0x7f89d006ef70 2024-04-12T05:33:30.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.313+0000 7f89edd06700 1 -- 172.21.15.32:0/2767099702 --> [v2:172.21.15.32:6810/164843120,v1:172.21.15.32:6811/164843120] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f89e80fe560 con 0x7f89d006ef70 2024-04-12T05:33:30.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.313+0000 7f89cffff700 1 -- 172.21.15.32:0/2767099702 <== osd.1 v2:172.21.15.32:6810/164843120 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f89e80fe560 con 0x7f89d006ef70 2024-04-12T05:33:30.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.313+0000 7f89edd06700 1 -- 172.21.15.32:0/2767099702 >> [v2:172.21.15.32:6810/164843120,v1:172.21.15.32:6811/164843120] conn(0x7f89d006ef70 msgr2=0x7f89d0071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:30.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.313+0000 7f89edd06700 1 --2- 172.21.15.32:0/2767099702 >> [v2:172.21.15.32:6810/164843120,v1:172.21.15.32:6811/164843120] conn(0x7f89d006ef70 0x7f89d0071380 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:30.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.313+0000 7f89edd06700 1 -- 172.21.15.32:0/2767099702 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f89d00692a0 msgr2=0x7f89d006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:30.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.313+0000 7f89edd06700 1 --2- 172.21.15.32:0/2767099702 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f89d00692a0 0x7f89d006b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f89dc004560 tx=0x7f89dc009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:30.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.313+0000 7f89edd06700 1 -- 172.21.15.32:0/2767099702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89e81037b0 msgr2=0x7f89e8070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:30.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.313+0000 7f89edd06700 1 --2- 172.21.15.32:0/2767099702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89e81037b0 0x7f89e8070800 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f89d800b790 tx=0x7f89d8009220 comp rx=0 tx=0).stop 2024-04-12T05:33:30.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.313+0000 7f89edd06700 1 -- 172.21.15.32:0/2767099702 shutdown_connections 2024-04-12T05:33:30.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.313+0000 7f89edd06700 1 --2- 172.21.15.32:0/2767099702 >> [v2:172.21.15.32:6810/164843120,v1:172.21.15.32:6811/164843120] conn(0x7f89d006ef70 0x7f89d0071380 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:30.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.313+0000 7f89edd06700 1 --2- 172.21.15.32:0/2767099702 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f89d00692a0 0x7f89d006b750 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:30.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.313+0000 7f89edd06700 1 --2- 172.21.15.32:0/2767099702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89e8102e10 0x7f89e80702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:30.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.313+0000 7f89edd06700 1 --2- 172.21.15.32:0/2767099702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89e81037b0 0x7f89e8070800 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:30.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.313+0000 7f89edd06700 1 -- 172.21.15.32:0/2767099702 >> 172.21.15.32:0/2767099702 conn(0x7f89e80f89a0 msgr2=0x7f89e80f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:30.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.313+0000 7f89edd06700 1 -- 172.21.15.32:0/2767099702 shutdown_connections 2024-04-12T05:33:30.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.313+0000 7f89edd06700 1 -- 172.21.15.32:0/2767099702 wait complete. 2024-04-12T05:33:30.476 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:30 smithi032 ceph-mon[39731]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:30.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:30 smithi067 ceph-mon[49010]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:30.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.437+0000 7f4e338f4700 1 -- 172.21.15.32:0/295220040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e2c070bf0 msgr2=0x7f4e2c109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:30.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.437+0000 7f4e338f4700 1 --2- 172.21.15.32:0/295220040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e2c070bf0 0x7f4e2c109190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4e20004940 tx=0x7f4e20009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:30.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.437+0000 7f4e338f4700 1 -- 172.21.15.32:0/295220040 shutdown_connections 2024-04-12T05:33:30.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.437+0000 7f4e338f4700 1 --2- 172.21.15.32:0/295220040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e2c070bf0 0x7f4e2c109190 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:30.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.437+0000 7f4e338f4700 1 --2- 172.21.15.32:0/295220040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e2c0702e0 0x7f4e2c0706b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:30.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.437+0000 7f4e338f4700 1 -- 172.21.15.32:0/295220040 >> 172.21.15.32:0/295220040 conn(0x7f4e2c0f89b0 msgr2=0x7f4e2c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:30.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.437+0000 7f4e338f4700 1 -- 172.21.15.32:0/295220040 shutdown_connections 2024-04-12T05:33:30.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.437+0000 7f4e338f4700 1 -- 172.21.15.32:0/295220040 wait complete. 2024-04-12T05:33:30.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.437+0000 7f4e338f4700 1 Processor -- start 2024-04-12T05:33:30.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.437+0000 7f4e338f4700 1 -- start start 2024-04-12T05:33:30.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.437+0000 7f4e338f4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e2c0702e0 0x7f4e2c115ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:30.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.437+0000 7f4e338f4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e2c070bf0 0x7f4e2c116000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:30.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.437+0000 7f4e338f4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c116650 con 0x7f4e2c0702e0 2024-04-12T05:33:30.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.437+0000 7f4e338f4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c10fc30 con 0x7f4e2c070bf0 2024-04-12T05:33:30.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.438+0000 7f4e31690700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e2c0702e0 0x7f4e2c115ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:30.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.438+0000 7f4e31690700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e2c0702e0 0x7f4e2c115ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43032/0 (socket says 172.21.15.32:43032) 2024-04-12T05:33:30.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.438+0000 7f4e31690700 1 -- 172.21.15.32:0/2428744003 learned_addr learned my addr 172.21.15.32:0/2428744003 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:33:30.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.438+0000 7f4e30e8f700 1 --2- 172.21.15.32:0/2428744003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e2c070bf0 0x7f4e2c116000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:30.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.438+0000 7f4e31690700 1 -- 172.21.15.32:0/2428744003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e2c070bf0 msgr2=0x7f4e2c116000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:30.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.438+0000 7f4e31690700 1 --2- 172.21.15.32:0/2428744003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e2c070bf0 0x7f4e2c116000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:30.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.438+0000 7f4e31690700 1 -- 172.21.15.32:0/2428744003 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e20005040 con 0x7f4e2c0702e0 2024-04-12T05:33:30.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.438+0000 7f4e31690700 1 --2- 172.21.15.32:0/2428744003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e2c0702e0 0x7f4e2c115ac0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c001d80 tx=0x7f4e1c009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:30.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.438+0000 7f4e29ffb700 1 -- 172.21.15.32:0/2428744003 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e1c007d10 con 0x7f4e2c0702e0 2024-04-12T05:33:30.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.438+0000 7f4e338f4700 1 -- 172.21.15.32:0/2428744003 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e2c10ff10 con 0x7f4e2c0702e0 2024-04-12T05:33:30.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.438+0000 7f4e29ffb700 1 -- 172.21.15.32:0/2428744003 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4e1c007e70 con 0x7f4e2c0702e0 2024-04-12T05:33:30.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.438+0000 7f4e29ffb700 1 -- 172.21.15.32:0/2428744003 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e1c014640 con 0x7f4e2c0702e0 2024-04-12T05:33:30.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.438+0000 7f4e338f4700 1 -- 172.21.15.32:0/2428744003 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e2c110450 con 0x7f4e2c0702e0 2024-04-12T05:33:30.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.439+0000 7f4e338f4700 1 -- 172.21.15.32:0/2428744003 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4e2c110990 con 0x7f4e2c0702e0 2024-04-12T05:33:30.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.440+0000 7f4e29ffb700 1 -- 172.21.15.32:0/2428744003 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f4e1c00e960 con 0x7f4e2c0702e0 2024-04-12T05:33:30.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.440+0000 7f4e29ffb700 1 --2- 172.21.15.32:0/2428744003 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4e140692a0 0x7f4e1406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:30.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.440+0000 7f4e29ffb700 1 -- 172.21.15.32:0/2428744003 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4e1c086c80 con 0x7f4e2c0702e0 2024-04-12T05:33:30.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.440+0000 7f4e30e8f700 1 --2- 172.21.15.32:0/2428744003 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4e140692a0 0x7f4e1406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:30.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.440+0000 7f4e29ffb700 1 --2- 172.21.15.32:0/2428744003 >> [v2:172.21.15.32:6802/3061707028,v1:172.21.15.32:6803/3061707028] conn(0x7f4e1406ef70 0x7f4e14071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:30.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.440+0000 7f4e29ffb700 1 -- 172.21.15.32:0/2428744003 --> [v2:172.21.15.32:6802/3061707028,v1:172.21.15.32:6803/3061707028] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e14071a30 con 0x7f4e1406ef70 2024-04-12T05:33:30.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.440+0000 7f4e29ffb700 1 -- 172.21.15.32:0/2428744003 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e1c08c2e0 con 0x7f4e2c0702e0 2024-04-12T05:33:30.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.440+0000 7f4e31e91700 1 --2- 172.21.15.32:0/2428744003 >> [v2:172.21.15.32:6802/3061707028,v1:172.21.15.32:6803/3061707028] conn(0x7f4e1406ef70 0x7f4e14071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:30.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.440+0000 7f4e30e8f700 1 --2- 172.21.15.32:0/2428744003 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4e140692a0 0x7f4e1406b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4e20000c00 tx=0x7f4e20009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:30.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.441+0000 7f4e31e91700 1 --2- 172.21.15.32:0/2428744003 >> [v2:172.21.15.32:6802/3061707028,v1:172.21.15.32:6803/3061707028] conn(0x7f4e1406ef70 0x7f4e14071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:30.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.441+0000 7f4e29ffb700 1 -- 172.21.15.32:0/2428744003 <== osd.0 v2:172.21.15.32:6802/3061707028 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f4e14071a30 con 0x7f4e1406ef70 2024-04-12T05:33:30.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.456+0000 7f4e338f4700 1 -- 172.21.15.32:0/2428744003 --> [v2:172.21.15.32:6802/3061707028,v1:172.21.15.32:6803/3061707028] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4e2c071a20 con 0x7f4e1406ef70 2024-04-12T05:33:30.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.457+0000 7f4e29ffb700 1 -- 172.21.15.32:0/2428744003 <== osd.0 v2:172.21.15.32:6802/3061707028 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4e2c071a20 con 0x7f4e1406ef70 2024-04-12T05:33:30.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.457+0000 7f4e338f4700 1 -- 172.21.15.32:0/2428744003 >> [v2:172.21.15.32:6802/3061707028,v1:172.21.15.32:6803/3061707028] conn(0x7f4e1406ef70 msgr2=0x7f4e14071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:30.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.457+0000 7f4e338f4700 1 --2- 172.21.15.32:0/2428744003 >> [v2:172.21.15.32:6802/3061707028,v1:172.21.15.32:6803/3061707028] conn(0x7f4e1406ef70 0x7f4e14071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:30.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.457+0000 7f4e338f4700 1 -- 172.21.15.32:0/2428744003 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4e140692a0 msgr2=0x7f4e1406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:30.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.457+0000 7f4e338f4700 1 --2- 172.21.15.32:0/2428744003 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4e140692a0 0x7f4e1406b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4e20000c00 tx=0x7f4e20009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:30.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.457+0000 7f4e338f4700 1 -- 172.21.15.32:0/2428744003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e2c0702e0 msgr2=0x7f4e2c115ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:30.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.457+0000 7f4e338f4700 1 --2- 172.21.15.32:0/2428744003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e2c0702e0 0x7f4e2c115ac0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c001d80 tx=0x7f4e1c009ec0 comp rx=0 tx=0).stop 2024-04-12T05:33:30.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.457+0000 7f4e338f4700 1 -- 172.21.15.32:0/2428744003 shutdown_connections 2024-04-12T05:33:30.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.457+0000 7f4e338f4700 1 --2- 172.21.15.32:0/2428744003 >> [v2:172.21.15.32:6802/3061707028,v1:172.21.15.32:6803/3061707028] conn(0x7f4e1406ef70 0x7f4e14071380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:30.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.457+0000 7f4e338f4700 1 --2- 172.21.15.32:0/2428744003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e2c0702e0 0x7f4e2c115ac0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:30.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.457+0000 7f4e338f4700 1 --2- 172.21.15.32:0/2428744003 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4e140692a0 0x7f4e1406b750 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:30.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.457+0000 7f4e338f4700 1 --2- 172.21.15.32:0/2428744003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e2c070bf0 0x7f4e2c116000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:30.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.457+0000 7f4e338f4700 1 -- 172.21.15.32:0/2428744003 >> 172.21.15.32:0/2428744003 conn(0x7f4e2c0f89b0 msgr2=0x7f4e2c105af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:30.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.457+0000 7f4e338f4700 1 -- 172.21.15.32:0/2428744003 shutdown_connections 2024-04-12T05:33:30.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:30.457+0000 7f4e338f4700 1 -- 172.21.15.32:0/2428744003 wait complete. 2024-04-12T05:33:31.561 INFO:teuthology.orchestra.run.smithi032.stdout:38654705700 2024-04-12T05:33:31.561 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-12T05:33:32.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:32 smithi032 ceph-mon[39731]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:32.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:32 smithi067 ceph-mon[49010]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:33.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.419+0000 7f0c4a592700 1 -- 172.21.15.32:0/1303747170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c440fde90 msgr2=0x7f0c440fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:33.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.419+0000 7f0c4a592700 1 --2- 172.21.15.32:0/1303747170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c440fde90 0x7f0c440fe260 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0c34004940 tx=0x7f0c34009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:33.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.419+0000 7f0c4a592700 1 -- 172.21.15.32:0/1303747170 shutdown_connections 2024-04-12T05:33:33.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.419+0000 7f0c4a592700 1 --2- 172.21.15.32:0/1303747170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c440fe830 0x7f0c44106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:33.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.419+0000 7f0c4a592700 1 --2- 172.21.15.32:0/1303747170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c440fde90 0x7f0c440fe260 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:33.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.419+0000 7f0c4a592700 1 -- 172.21.15.32:0/1303747170 >> 172.21.15.32:0/1303747170 conn(0x7f0c44072bc0 msgr2=0x7f0c44072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:33.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.419+0000 7f0c4a592700 1 -- 172.21.15.32:0/1303747170 shutdown_connections 2024-04-12T05:33:33.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.419+0000 7f0c4a592700 1 -- 172.21.15.32:0/1303747170 wait complete. 2024-04-12T05:33:33.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.419+0000 7f0c4a592700 1 Processor -- start 2024-04-12T05:33:33.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.420+0000 7f0c4a592700 1 -- start start 2024-04-12T05:33:33.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.420+0000 7f0c4a592700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c440fde90 0x7f0c44113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:33.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.420+0000 7f0c4a592700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c440fe830 0x7f0c44114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:33.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.420+0000 7f0c4a592700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c44114690 con 0x7f0c440fde90 2024-04-12T05:33:33.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.420+0000 7f0c4a592700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4410da80 con 0x7f0c440fe830 2024-04-12T05:33:33.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.420+0000 7f0c43fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c440fde90 0x7f0c44113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:33.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.420+0000 7f0c43fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c440fde90 0x7f0c44113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54692/0 (socket says 172.21.15.32:54692) 2024-04-12T05:33:33.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.420+0000 7f0c43fff700 1 -- 172.21.15.32:0/342471048 learned_addr learned my addr 172.21.15.32:0/342471048 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:33:33.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.420+0000 7f0c437fe700 1 --2- 172.21.15.32:0/342471048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c440fe830 0x7f0c44114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:33.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.420+0000 7f0c43fff700 1 -- 172.21.15.32:0/342471048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c440fe830 msgr2=0x7f0c44114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:33.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.420+0000 7f0c43fff700 1 --2- 172.21.15.32:0/342471048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c440fe830 0x7f0c44114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:33.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.420+0000 7f0c43fff700 1 -- 172.21.15.32:0/342471048 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c34005040 con 0x7f0c440fde90 2024-04-12T05:33:33.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.420+0000 7f0c43fff700 1 --2- 172.21.15.32:0/342471048 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c440fde90 0x7f0c44113b00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400b3b0 tx=0x7f0c34009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:33.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.420+0000 7f0c40ff9700 1 -- 172.21.15.32:0/342471048 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c3400e040 con 0x7f0c440fde90 2024-04-12T05:33:33.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.420+0000 7f0c4a592700 1 -- 172.21.15.32:0/342471048 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c4410dd00 con 0x7f0c440fde90 2024-04-12T05:33:33.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.420+0000 7f0c40ff9700 1 -- 172.21.15.32:0/342471048 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c34013420 con 0x7f0c440fde90 2024-04-12T05:33:33.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.420+0000 7f0c40ff9700 1 -- 172.21.15.32:0/342471048 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c3401c5f0 con 0x7f0c440fde90 2024-04-12T05:33:33.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.421+0000 7f0c4a592700 1 -- 172.21.15.32:0/342471048 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c4410e240 con 0x7f0c440fde90 2024-04-12T05:33:33.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.421+0000 7f0c4a592700 1 -- 172.21.15.32:0/342471048 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0c4410e720 con 0x7f0c440fde90 2024-04-12T05:33:33.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.422+0000 7f0c40ff9700 1 -- 172.21.15.32:0/342471048 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f0c34013590 con 0x7f0c440fde90 2024-04-12T05:33:33.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.422+0000 7f0c40ff9700 1 --2- 172.21.15.32:0/342471048 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f0c2c0692a0 0x7f0c2c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:33.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.422+0000 7f0c40ff9700 1 -- 172.21.15.32:0/342471048 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0c34088140 con 0x7f0c440fde90 2024-04-12T05:33:33.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.422+0000 7f0c437fe700 1 --2- 172.21.15.32:0/342471048 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f0c2c0692a0 0x7f0c2c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:33.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.422+0000 7f0c40ff9700 1 --2- 172.21.15.32:0/342471048 >> [v2:172.21.15.67:6816/1361296892,v1:172.21.15.67:6817/1361296892] conn(0x7f0c2c06ef70 0x7f0c2c071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:33.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.422+0000 7f0c40ff9700 1 -- 172.21.15.32:0/342471048 --> [v2:172.21.15.67:6816/1361296892,v1:172.21.15.67:6817/1361296892] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0c2c071a30 con 0x7f0c2c06ef70 2024-04-12T05:33:33.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.422+0000 7f0c40ff9700 1 -- 172.21.15.32:0/342471048 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c3408a730 con 0x7f0c440fde90 2024-04-12T05:33:33.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.423+0000 7f0c437fe700 1 --2- 172.21.15.32:0/342471048 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f0c2c0692a0 0x7f0c2c06b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0c4410ec40 tx=0x7f0c3800b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:33.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.423+0000 7f0c48b2f700 1 --2- 172.21.15.32:0/342471048 >> [v2:172.21.15.67:6816/1361296892,v1:172.21.15.67:6817/1361296892] conn(0x7f0c2c06ef70 0x7f0c2c071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:33.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.423+0000 7f0c48b2f700 1 --2- 172.21.15.32:0/342471048 >> [v2:172.21.15.67:6816/1361296892,v1:172.21.15.67:6817/1361296892] conn(0x7f0c2c06ef70 0x7f0c2c071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:33.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.424+0000 7f0c40ff9700 1 -- 172.21.15.32:0/342471048 <== osd.5 v2:172.21.15.67:6816/1361296892 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f0c2c071a30 con 0x7f0c2c06ef70 2024-04-12T05:33:33.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.439+0000 7f0c4a592700 1 -- 172.21.15.32:0/342471048 --> [v2:172.21.15.67:6816/1361296892,v1:172.21.15.67:6817/1361296892] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0c44106690 con 0x7f0c2c06ef70 2024-04-12T05:33:33.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.440+0000 7f0c40ff9700 1 -- 172.21.15.32:0/342471048 <== osd.5 v2:172.21.15.67:6816/1361296892 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0c44106690 con 0x7f0c2c06ef70 2024-04-12T05:33:33.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.440+0000 7f0c4a592700 1 -- 172.21.15.32:0/342471048 >> [v2:172.21.15.67:6816/1361296892,v1:172.21.15.67:6817/1361296892] conn(0x7f0c2c06ef70 msgr2=0x7f0c2c071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:33.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.440+0000 7f0c4a592700 1 --2- 172.21.15.32:0/342471048 >> [v2:172.21.15.67:6816/1361296892,v1:172.21.15.67:6817/1361296892] conn(0x7f0c2c06ef70 0x7f0c2c071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:33.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.440+0000 7f0c4a592700 1 -- 172.21.15.32:0/342471048 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f0c2c0692a0 msgr2=0x7f0c2c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:33.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.440+0000 7f0c4a592700 1 --2- 172.21.15.32:0/342471048 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f0c2c0692a0 0x7f0c2c06b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0c4410ec40 tx=0x7f0c3800b040 comp rx=0 tx=0).stop 2024-04-12T05:33:33.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.440+0000 7f0c4a592700 1 -- 172.21.15.32:0/342471048 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c440fde90 msgr2=0x7f0c44113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:33.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.440+0000 7f0c4a592700 1 --2- 172.21.15.32:0/342471048 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c440fde90 0x7f0c44113b00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400b3b0 tx=0x7f0c34009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:33.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.440+0000 7f0c4a592700 1 -- 172.21.15.32:0/342471048 shutdown_connections 2024-04-12T05:33:33.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.440+0000 7f0c4a592700 1 --2- 172.21.15.32:0/342471048 >> [v2:172.21.15.67:6816/1361296892,v1:172.21.15.67:6817/1361296892] conn(0x7f0c2c06ef70 0x7f0c2c071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:33.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.440+0000 7f0c4a592700 1 --2- 172.21.15.32:0/342471048 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c440fde90 0x7f0c44113b00 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:33.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.440+0000 7f0c4a592700 1 --2- 172.21.15.32:0/342471048 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f0c2c0692a0 0x7f0c2c06b750 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:33.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.440+0000 7f0c4a592700 1 --2- 172.21.15.32:0/342471048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c440fe830 0x7f0c44114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:33.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.440+0000 7f0c4a592700 1 -- 172.21.15.32:0/342471048 >> 172.21.15.32:0/342471048 conn(0x7f0c44072bc0 msgr2=0x7f0c44103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:33.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.440+0000 7f0c4a592700 1 -- 172.21.15.32:0/342471048 shutdown_connections 2024-04-12T05:33:33.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:33.441+0000 7f0c4a592700 1 -- 172.21.15.32:0/342471048 wait complete. 2024-04-12T05:33:34.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:34 smithi032 ceph-mon[39731]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:34.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:34 smithi067 ceph-mon[49010]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:34.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.683+0000 7fb8e24eb700 1 -- 172.21.15.32:0/3208293002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8dc101d70 msgr2=0x7fb8dc102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:34.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.683+0000 7fb8e24eb700 1 --2- 172.21.15.32:0/3208293002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8dc101d70 0x7fb8dc102140 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc004940 tx=0x7fb8cc009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:34.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.683+0000 7fb8e24eb700 1 -- 172.21.15.32:0/3208293002 shutdown_connections 2024-04-12T05:33:34.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.683+0000 7fb8e24eb700 1 --2- 172.21.15.32:0/3208293002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8dc102710 0x7fb8dc0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:34.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.683+0000 7fb8e24eb700 1 --2- 172.21.15.32:0/3208293002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8dc101d70 0x7fb8dc102140 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:34.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.683+0000 7fb8e24eb700 1 -- 172.21.15.32:0/3208293002 >> 172.21.15.32:0/3208293002 conn(0x7fb8dc072bc0 msgr2=0x7fb8dc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:34.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.683+0000 7fb8e24eb700 1 -- 172.21.15.32:0/3208293002 shutdown_connections 2024-04-12T05:33:34.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.683+0000 7fb8e24eb700 1 -- 172.21.15.32:0/3208293002 wait complete. 2024-04-12T05:33:34.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.683+0000 7fb8e24eb700 1 Processor -- start 2024-04-12T05:33:34.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.684+0000 7fb8e24eb700 1 -- start start 2024-04-12T05:33:34.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.684+0000 7fb8e24eb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8dc101d70 0x7fb8dc112920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:34.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.684+0000 7fb8e24eb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8dc102710 0x7fb8dc112e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:34.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.684+0000 7fb8e24eb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8dc1134b0 con 0x7fb8dc102710 2024-04-12T05:33:34.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.684+0000 7fb8e24eb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8dc10c8a0 con 0x7fb8dc101d70 2024-04-12T05:33:34.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.684+0000 7fb8db7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8dc102710 0x7fb8dc112e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:34.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.684+0000 7fb8db7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8dc102710 0x7fb8dc112e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54672/0 (socket says 172.21.15.32:54672) 2024-04-12T05:33:34.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.684+0000 7fb8db7fe700 1 -- 172.21.15.32:0/821617000 learned_addr learned my addr 172.21.15.32:0/821617000 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:33:34.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.684+0000 7fb8dbfff700 1 --2- 172.21.15.32:0/821617000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8dc101d70 0x7fb8dc112920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:34.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.684+0000 7fb8db7fe700 1 -- 172.21.15.32:0/821617000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8dc101d70 msgr2=0x7fb8dc112920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:34.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.684+0000 7fb8db7fe700 1 --2- 172.21.15.32:0/821617000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8dc101d70 0x7fb8dc112920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:34.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.684+0000 7fb8db7fe700 1 -- 172.21.15.32:0/821617000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8cc005040 con 0x7fb8dc102710 2024-04-12T05:33:34.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.684+0000 7fb8db7fe700 1 --2- 172.21.15.32:0/821617000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8dc102710 0x7fb8dc112e60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb8d000eb40 tx=0x7fb8d000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:34.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.685+0000 7fb8d8ff9700 1 -- 172.21.15.32:0/821617000 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8d0007cc0 con 0x7fb8dc102710 2024-04-12T05:33:34.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.685+0000 7fb8e24eb700 1 -- 172.21.15.32:0/821617000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8dc10cb20 con 0x7fb8dc102710 2024-04-12T05:33:34.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.685+0000 7fb8d8ff9700 1 -- 172.21.15.32:0/821617000 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb8d000f040 con 0x7fb8dc102710 2024-04-12T05:33:34.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.685+0000 7fb8d8ff9700 1 -- 172.21.15.32:0/821617000 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8d0009570 con 0x7fb8dc102710 2024-04-12T05:33:34.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.685+0000 7fb8e24eb700 1 -- 172.21.15.32:0/821617000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8dc10d0e0 con 0x7fb8dc102710 2024-04-12T05:33:34.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.685+0000 7fb8e24eb700 1 -- 172.21.15.32:0/821617000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb8dc10d5c0 con 0x7fb8dc102710 2024-04-12T05:33:34.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.686+0000 7fb8d8ff9700 1 -- 172.21.15.32:0/821617000 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fb8d00096d0 con 0x7fb8dc102710 2024-04-12T05:33:34.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.686+0000 7fb8d8ff9700 1 --2- 172.21.15.32:0/821617000 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fb8c40692a0 0x7fb8c406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:34.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.687+0000 7fb8d8ff9700 1 -- 172.21.15.32:0/821617000 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb8d0086970 con 0x7fb8dc102710 2024-04-12T05:33:34.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.687+0000 7fb8dbfff700 1 --2- 172.21.15.32:0/821617000 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fb8c40692a0 0x7fb8c406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:34.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.687+0000 7fb8d8ff9700 1 --2- 172.21.15.32:0/821617000 >> [v2:172.21.15.67:6808/286946785,v1:172.21.15.67:6809/286946785] conn(0x7fb8c406ef70 0x7fb8c4071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:34.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.687+0000 7fb8d8ff9700 1 -- 172.21.15.32:0/821617000 --> [v2:172.21.15.67:6808/286946785,v1:172.21.15.67:6809/286946785] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb8c4071a30 con 0x7fb8c406ef70 2024-04-12T05:33:34.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.687+0000 7fb8d8ff9700 1 -- 172.21.15.32:0/821617000 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb8d008b220 con 0x7fb8dc102710 2024-04-12T05:33:34.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.687+0000 7fb8dbfff700 1 --2- 172.21.15.32:0/821617000 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fb8c40692a0 0x7fb8c406b750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc004540 tx=0x7fb8cc009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:34.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.687+0000 7fb8e0a88700 1 --2- 172.21.15.32:0/821617000 >> [v2:172.21.15.67:6808/286946785,v1:172.21.15.67:6809/286946785] conn(0x7fb8c406ef70 0x7fb8c4071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:34.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.687+0000 7fb8e0a88700 1 --2- 172.21.15.32:0/821617000 >> [v2:172.21.15.67:6808/286946785,v1:172.21.15.67:6809/286946785] conn(0x7fb8c406ef70 0x7fb8c4071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:34.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.688+0000 7fb8d8ff9700 1 -- 172.21.15.32:0/821617000 <== osd.4 v2:172.21.15.67:6808/286946785 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fb8c4071a30 con 0x7fb8c406ef70 2024-04-12T05:33:34.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.703+0000 7fb8e24eb700 1 -- 172.21.15.32:0/821617000 --> [v2:172.21.15.67:6808/286946785,v1:172.21.15.67:6809/286946785] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb8dc05e6c0 con 0x7fb8c406ef70 2024-04-12T05:33:34.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.704+0000 7fb8d8ff9700 1 -- 172.21.15.32:0/821617000 <== osd.4 v2:172.21.15.67:6808/286946785 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb8dc05e6c0 con 0x7fb8c406ef70 2024-04-12T05:33:34.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.704+0000 7fb8e24eb700 1 -- 172.21.15.32:0/821617000 >> [v2:172.21.15.67:6808/286946785,v1:172.21.15.67:6809/286946785] conn(0x7fb8c406ef70 msgr2=0x7fb8c4071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:34.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.704+0000 7fb8e24eb700 1 --2- 172.21.15.32:0/821617000 >> [v2:172.21.15.67:6808/286946785,v1:172.21.15.67:6809/286946785] conn(0x7fb8c406ef70 0x7fb8c4071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:34.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.704+0000 7fb8e24eb700 1 -- 172.21.15.32:0/821617000 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fb8c40692a0 msgr2=0x7fb8c406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:34.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.704+0000 7fb8e24eb700 1 --2- 172.21.15.32:0/821617000 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fb8c40692a0 0x7fb8c406b750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc004540 tx=0x7fb8cc009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:34.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.704+0000 7fb8e24eb700 1 -- 172.21.15.32:0/821617000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8dc102710 msgr2=0x7fb8dc112e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:34.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.704+0000 7fb8e24eb700 1 --2- 172.21.15.32:0/821617000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8dc102710 0x7fb8dc112e60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb8d000eb40 tx=0x7fb8d000b040 comp rx=0 tx=0).stop 2024-04-12T05:33:34.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.704+0000 7fb8e24eb700 1 -- 172.21.15.32:0/821617000 shutdown_connections 2024-04-12T05:33:34.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.704+0000 7fb8e24eb700 1 --2- 172.21.15.32:0/821617000 >> [v2:172.21.15.67:6808/286946785,v1:172.21.15.67:6809/286946785] conn(0x7fb8c406ef70 0x7fb8c4071380 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:34.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.704+0000 7fb8e24eb700 1 --2- 172.21.15.32:0/821617000 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fb8c40692a0 0x7fb8c406b750 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:34.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.704+0000 7fb8e24eb700 1 --2- 172.21.15.32:0/821617000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8dc101d70 0x7fb8dc112920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:34.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.704+0000 7fb8e24eb700 1 --2- 172.21.15.32:0/821617000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8dc102710 0x7fb8dc112e60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:34.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.704+0000 7fb8e24eb700 1 -- 172.21.15.32:0/821617000 >> 172.21.15.32:0/821617000 conn(0x7fb8dc072bc0 msgr2=0x7fb8dc0faf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:34.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.704+0000 7fb8e24eb700 1 -- 172.21.15.32:0/821617000 shutdown_connections 2024-04-12T05:33:34.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:32.704+0000 7fb8e24eb700 1 -- 172.21.15.32:0/821617000 wait complete. 2024-04-12T05:33:35.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.229+0000 7f26b20d5700 1 -- 172.21.15.32:0/591796273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26ac0fc590 msgr2=0x7f26ac1092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:35.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.229+0000 7f26b20d5700 1 --2- 172.21.15.32:0/591796273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26ac0fc590 0x7f26ac1092f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f26a0004940 tx=0x7f26a0009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:35.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.229+0000 7f26b20d5700 1 -- 172.21.15.32:0/591796273 shutdown_connections 2024-04-12T05:33:35.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.229+0000 7f26b20d5700 1 --2- 172.21.15.32:0/591796273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26ac0fc590 0x7f26ac1092f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:35.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.229+0000 7f26b20d5700 1 --2- 172.21.15.32:0/591796273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26ac0fbc80 0x7f26ac0fc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:35.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.229+0000 7f26b20d5700 1 -- 172.21.15.32:0/591796273 >> 172.21.15.32:0/591796273 conn(0x7f26ac072bc0 msgr2=0x7f26ac072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:35.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.229+0000 7f26b20d5700 1 -- 172.21.15.32:0/591796273 shutdown_connections 2024-04-12T05:33:35.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.230+0000 7f26b20d5700 1 -- 172.21.15.32:0/591796273 wait complete. 2024-04-12T05:33:35.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.230+0000 7f26b20d5700 1 Processor -- start 2024-04-12T05:33:35.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.230+0000 7f26b20d5700 1 -- start start 2024-04-12T05:33:35.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.231+0000 7f26b20d5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26ac0fbc80 0x7f26ac115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:35.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.231+0000 7f26b20d5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26ac0fc590 0x7f26ac1161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:35.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.231+0000 7f26b20d5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26ac116800 con 0x7f26ac0fc590 2024-04-12T05:33:35.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.231+0000 7f26b20d5700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26ac10fbd0 con 0x7f26ac0fbc80 2024-04-12T05:33:35.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.231+0000 7f26aaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26ac0fc590 0x7f26ac1161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:35.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.231+0000 7f26aaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26ac0fc590 0x7f26ac1161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54708/0 (socket says 172.21.15.32:54708) 2024-04-12T05:33:35.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.231+0000 7f26aaffd700 1 -- 172.21.15.32:0/602207002 learned_addr learned my addr 172.21.15.32:0/602207002 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:33:35.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.231+0000 7f26ab7fe700 1 --2- 172.21.15.32:0/602207002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26ac0fbc80 0x7f26ac115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:35.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.231+0000 7f26aaffd700 1 -- 172.21.15.32:0/602207002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26ac0fbc80 msgr2=0x7f26ac115c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:35.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.231+0000 7f26aaffd700 1 --2- 172.21.15.32:0/602207002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26ac0fbc80 0x7f26ac115c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:35.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.231+0000 7f26aaffd700 1 -- 172.21.15.32:0/602207002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a0005040 con 0x7f26ac0fc590 2024-04-12T05:33:35.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.231+0000 7f26ab7fe700 1 --2- 172.21.15.32:0/602207002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26ac0fbc80 0x7f26ac115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:33:35.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.231+0000 7f26aaffd700 1 --2- 172.21.15.32:0/602207002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26ac0fc590 0x7f26ac1161b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f26a0009360 tx=0x7f26a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:35.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.232+0000 7f2693fff700 1 -- 172.21.15.32:0/602207002 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26a000e040 con 0x7f26ac0fc590 2024-04-12T05:33:35.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.232+0000 7f26b20d5700 1 -- 172.21.15.32:0/602207002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26ac10fe50 con 0x7f26ac0fc590 2024-04-12T05:33:35.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.232+0000 7f2693fff700 1 -- 172.21.15.32:0/602207002 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f26a0013420 con 0x7f26ac0fc590 2024-04-12T05:33:35.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.232+0000 7f2693fff700 1 -- 172.21.15.32:0/602207002 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26a001c5f0 con 0x7f26ac0fc590 2024-04-12T05:33:35.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.232+0000 7f26b20d5700 1 -- 172.21.15.32:0/602207002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26ac110410 con 0x7f26ac0fc590 2024-04-12T05:33:35.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.233+0000 7f26b20d5700 1 -- 172.21.15.32:0/602207002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f26ac1108f0 con 0x7f26ac0fc590 2024-04-12T05:33:35.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.233+0000 7f2693fff700 1 -- 172.21.15.32:0/602207002 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f26a0013590 con 0x7f26ac0fc590 2024-04-12T05:33:35.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.233+0000 7f2693fff700 1 --2- 172.21.15.32:0/602207002 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f26940692b0 0x7f269406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:35.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.234+0000 7f26ab7fe700 1 --2- 172.21.15.32:0/602207002 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f26940692b0 0x7f269406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:35.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.234+0000 7f2693fff700 1 -- 172.21.15.32:0/602207002 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f26a0088090 con 0x7f26ac0fc590 2024-04-12T05:33:35.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.234+0000 7f2693fff700 1 --2- 172.21.15.32:0/602207002 >> [v2:172.21.15.67:6800/1205580917,v1:172.21.15.67:6801/1205580917] conn(0x7f269406ef80 0x7f2694071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:35.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.234+0000 7f2693fff700 1 -- 172.21.15.32:0/602207002 --> [v2:172.21.15.67:6800/1205580917,v1:172.21.15.67:6801/1205580917] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2694071a40 con 0x7f269406ef80 2024-04-12T05:33:35.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.234+0000 7f2693fff700 1 -- 172.21.15.32:0/602207002 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f26a008a6e0 con 0x7f26ac0fc590 2024-04-12T05:33:35.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.234+0000 7f26ab7fe700 1 --2- 172.21.15.32:0/602207002 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f26940692b0 0x7f269406b760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f269c009fd0 tx=0x7f269c00b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:35.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.234+0000 7f26abfff700 1 --2- 172.21.15.32:0/602207002 >> [v2:172.21.15.67:6800/1205580917,v1:172.21.15.67:6801/1205580917] conn(0x7f269406ef80 0x7f2694071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:35.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.234+0000 7f26abfff700 1 --2- 172.21.15.32:0/602207002 >> [v2:172.21.15.67:6800/1205580917,v1:172.21.15.67:6801/1205580917] conn(0x7f269406ef80 0x7f2694071390 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:35.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.235+0000 7f2693fff700 1 -- 172.21.15.32:0/602207002 <== osd.3 v2:172.21.15.67:6800/1205580917 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f2694071a40 con 0x7f269406ef80 2024-04-12T05:33:35.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.250+0000 7f26b20d5700 1 -- 172.21.15.32:0/602207002 --> [v2:172.21.15.67:6800/1205580917,v1:172.21.15.67:6801/1205580917] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f26ac104880 con 0x7f269406ef80 2024-04-12T05:33:35.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.251+0000 7f2693fff700 1 -- 172.21.15.32:0/602207002 <== osd.3 v2:172.21.15.67:6800/1205580917 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f26ac104880 con 0x7f269406ef80 2024-04-12T05:33:35.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.251+0000 7f26b20d5700 1 -- 172.21.15.32:0/602207002 >> [v2:172.21.15.67:6800/1205580917,v1:172.21.15.67:6801/1205580917] conn(0x7f269406ef80 msgr2=0x7f2694071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:35.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.251+0000 7f26b20d5700 1 --2- 172.21.15.32:0/602207002 >> [v2:172.21.15.67:6800/1205580917,v1:172.21.15.67:6801/1205580917] conn(0x7f269406ef80 0x7f2694071390 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:35.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.251+0000 7f26b20d5700 1 -- 172.21.15.32:0/602207002 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f26940692b0 msgr2=0x7f269406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:35.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.251+0000 7f26b20d5700 1 --2- 172.21.15.32:0/602207002 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f26940692b0 0x7f269406b760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f269c009fd0 tx=0x7f269c00b040 comp rx=0 tx=0).stop 2024-04-12T05:33:35.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.251+0000 7f26b20d5700 1 -- 172.21.15.32:0/602207002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26ac0fc590 msgr2=0x7f26ac1161b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:35.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.251+0000 7f26b20d5700 1 --2- 172.21.15.32:0/602207002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26ac0fc590 0x7f26ac1161b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f26a0009360 tx=0x7f26a0009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:35.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.252+0000 7f26b20d5700 1 -- 172.21.15.32:0/602207002 shutdown_connections 2024-04-12T05:33:35.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.252+0000 7f26b20d5700 1 --2- 172.21.15.32:0/602207002 >> [v2:172.21.15.67:6800/1205580917,v1:172.21.15.67:6801/1205580917] conn(0x7f269406ef80 0x7f2694071390 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:35.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.252+0000 7f26b20d5700 1 --2- 172.21.15.32:0/602207002 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f26940692b0 0x7f269406b760 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:35.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.252+0000 7f26b20d5700 1 --2- 172.21.15.32:0/602207002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26ac0fbc80 0x7f26ac115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:35.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.252+0000 7f26b20d5700 1 --2- 172.21.15.32:0/602207002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26ac0fc590 0x7f26ac1161b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:35.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.252+0000 7f26b20d5700 1 -- 172.21.15.32:0/602207002 >> 172.21.15.32:0/602207002 conn(0x7f26ac072bc0 msgr2=0x7f26ac105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:35.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.252+0000 7f26b20d5700 1 -- 172.21.15.32:0/602207002 shutdown_connections 2024-04-12T05:33:35.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:35.252+0000 7f26b20d5700 1 -- 172.21.15.32:0/602207002 wait complete. 2024-04-12T05:33:35.993 INFO:teuthology.orchestra.run.smithi032.stdout:111669149716 2024-04-12T05:33:35.993 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-12T05:33:36.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:36 smithi032 ceph-mon[39731]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:36.357 INFO:teuthology.orchestra.run.smithi032.stdout:146028888079 2024-04-12T05:33:36.358 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-12T05:33:36.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:36 smithi067 ceph-mon[49010]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:37.407 INFO:teuthology.orchestra.run.smithi032.stdout:167503724554 2024-04-12T05:33:37.408 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-12T05:33:37.812 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:33:37.840 INFO:teuthology.orchestra.run.smithi032.stdout:60129542175 2024-04-12T05:33:37.841 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-12T05:33:37.865 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:33:38.376 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:38 smithi032 ceph-mon[39731]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:38.523 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:38 smithi067 ceph-mon[49010]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:38.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.563+0000 7f581fd1c700 1 -- 172.21.15.32:0/200205558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5818102f50 msgr2=0x7f5818103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:38.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.563+0000 7f581fd1c700 1 --2- 172.21.15.32:0/200205558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5818102f50 0x7f5818103320 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5814004940 tx=0x7f5814009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:38.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.563+0000 7f581fd1c700 1 -- 172.21.15.32:0/200205558 shutdown_connections 2024-04-12T05:33:38.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.563+0000 7f581fd1c700 1 --2- 172.21.15.32:0/200205558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58180fb4d0 0x7f58180fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:38.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.563+0000 7f581fd1c700 1 --2- 172.21.15.32:0/200205558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5818102f50 0x7f5818103320 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:38.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.563+0000 7f581fd1c700 1 -- 172.21.15.32:0/200205558 >> 172.21.15.32:0/200205558 conn(0x7f58180714c0 msgr2=0x7f58180718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:38.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.564+0000 7f581fd1c700 1 -- 172.21.15.32:0/200205558 shutdown_connections 2024-04-12T05:33:38.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.564+0000 7f581fd1c700 1 -- 172.21.15.32:0/200205558 wait complete. 2024-04-12T05:33:38.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.564+0000 7f581fd1c700 1 Processor -- start 2024-04-12T05:33:38.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.564+0000 7f581fd1c700 1 -- start start 2024-04-12T05:33:38.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.564+0000 7f581fd1c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58180fb4d0 0x7f58180fc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:38.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.564+0000 7f581fd1c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5818102f50 0x7f58180fcbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:38.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.564+0000 7f581fd1c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58180fe690 con 0x7f5818102f50 2024-04-12T05:33:38.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.564+0000 7f581fd1c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58180fd130 con 0x7f58180fb4d0 2024-04-12T05:33:38.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.564+0000 7f581d2b7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5818102f50 0x7f58180fcbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:38.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.565+0000 7f581d2b7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5818102f50 0x7f58180fcbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54736/0 (socket says 172.21.15.32:54736) 2024-04-12T05:33:38.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.565+0000 7f581d2b7700 1 -- 172.21.15.32:0/3916465958 learned_addr learned my addr 172.21.15.32:0/3916465958 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:33:38.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.565+0000 7f581dab8700 1 --2- 172.21.15.32:0/3916465958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58180fb4d0 0x7f58180fc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:38.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.565+0000 7f581d2b7700 1 -- 172.21.15.32:0/3916465958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58180fb4d0 msgr2=0x7f58180fc6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:38.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.565+0000 7f581d2b7700 1 --2- 172.21.15.32:0/3916465958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58180fb4d0 0x7f58180fc6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:38.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.565+0000 7f581d2b7700 1 -- 172.21.15.32:0/3916465958 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5814005040 con 0x7f5818102f50 2024-04-12T05:33:38.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.565+0000 7f581d2b7700 1 --2- 172.21.15.32:0/3916465958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5818102f50 0x7f58180fcbf0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f580c00b790 tx=0x7f580c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:38.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.565+0000 7f580a7fc700 1 -- 172.21.15.32:0/3916465958 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f580c00f770 con 0x7f5818102f50 2024-04-12T05:33:38.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.565+0000 7f580a7fc700 1 -- 172.21.15.32:0/3916465958 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f580c00fd80 con 0x7f5818102f50 2024-04-12T05:33:38.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.565+0000 7f580a7fc700 1 -- 172.21.15.32:0/3916465958 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f580c00d310 con 0x7f5818102f50 2024-04-12T05:33:38.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.565+0000 7f581fd1c700 1 -- 172.21.15.32:0/3916465958 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58180fd410 con 0x7f5818102f50 2024-04-12T05:33:38.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.565+0000 7f581fd1c700 1 -- 172.21.15.32:0/3916465958 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f581811a1c0 con 0x7f5818102f50 2024-04-12T05:33:38.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.566+0000 7f581fd1c700 1 -- 172.21.15.32:0/3916465958 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f581805e640 con 0x7f5818102f50 2024-04-12T05:33:38.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.567+0000 7f580a7fc700 1 -- 172.21.15.32:0/3916465958 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f580c010420 con 0x7f5818102f50 2024-04-12T05:33:38.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.567+0000 7f580a7fc700 1 --2- 172.21.15.32:0/3916465958 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f58040692a0 0x7f580406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:38.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.567+0000 7f580a7fc700 1 -- 172.21.15.32:0/3916465958 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f580c086e00 con 0x7f5818102f50 2024-04-12T05:33:38.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.567+0000 7f581dab8700 1 --2- 172.21.15.32:0/3916465958 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f58040692a0 0x7f580406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:38.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.567+0000 7f580a7fc700 1 --2- 172.21.15.32:0/3916465958 >> [v2:172.21.15.32:6818/2698911336,v1:172.21.15.32:6819/2698911336] conn(0x7f580406ef70 0x7f5804071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:38.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.567+0000 7f580a7fc700 1 -- 172.21.15.32:0/3916465958 --> [v2:172.21.15.32:6818/2698911336,v1:172.21.15.32:6819/2698911336] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5804071a30 con 0x7f580406ef70 2024-04-12T05:33:38.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.567+0000 7f580a7fc700 1 -- 172.21.15.32:0/3916465958 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f580c089350 con 0x7f5818102f50 2024-04-12T05:33:38.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.567+0000 7f581e2b9700 1 --2- 172.21.15.32:0/3916465958 >> [v2:172.21.15.32:6818/2698911336,v1:172.21.15.32:6819/2698911336] conn(0x7f580406ef70 0x7f5804071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:38.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.567+0000 7f581dab8700 1 --2- 172.21.15.32:0/3916465958 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f58040692a0 0x7f580406b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5814004f90 tx=0x7f5814009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:38.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.568+0000 7f581e2b9700 1 --2- 172.21.15.32:0/3916465958 >> [v2:172.21.15.32:6818/2698911336,v1:172.21.15.32:6819/2698911336] conn(0x7f580406ef70 0x7f5804071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:38.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.568+0000 7f580a7fc700 1 -- 172.21.15.32:0/3916465958 <== osd.2 v2:172.21.15.32:6818/2698911336 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f5804071a30 con 0x7f580406ef70 2024-04-12T05:33:38.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.584+0000 7f581fd1c700 1 -- 172.21.15.32:0/3916465958 --> [v2:172.21.15.32:6818/2698911336,v1:172.21.15.32:6819/2698911336] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f581805eac0 con 0x7f580406ef70 2024-04-12T05:33:38.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.584+0000 7f580a7fc700 1 -- 172.21.15.32:0/3916465958 <== osd.2 v2:172.21.15.32:6818/2698911336 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f581805eac0 con 0x7f580406ef70 2024-04-12T05:33:38.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.584+0000 7f581fd1c700 1 -- 172.21.15.32:0/3916465958 >> [v2:172.21.15.32:6818/2698911336,v1:172.21.15.32:6819/2698911336] conn(0x7f580406ef70 msgr2=0x7f5804071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:38.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.584+0000 7f581fd1c700 1 --2- 172.21.15.32:0/3916465958 >> [v2:172.21.15.32:6818/2698911336,v1:172.21.15.32:6819/2698911336] conn(0x7f580406ef70 0x7f5804071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:38.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.584+0000 7f581fd1c700 1 -- 172.21.15.32:0/3916465958 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f58040692a0 msgr2=0x7f580406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:38.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.584+0000 7f581fd1c700 1 --2- 172.21.15.32:0/3916465958 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f58040692a0 0x7f580406b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5814004f90 tx=0x7f5814009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:38.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.584+0000 7f581fd1c700 1 -- 172.21.15.32:0/3916465958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5818102f50 msgr2=0x7f58180fcbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:38.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.584+0000 7f581fd1c700 1 --2- 172.21.15.32:0/3916465958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5818102f50 0x7f58180fcbf0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f580c00b790 tx=0x7f580c009250 comp rx=0 tx=0).stop 2024-04-12T05:33:38.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.584+0000 7f581fd1c700 1 -- 172.21.15.32:0/3916465958 shutdown_connections 2024-04-12T05:33:38.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.584+0000 7f581fd1c700 1 --2- 172.21.15.32:0/3916465958 >> [v2:172.21.15.32:6818/2698911336,v1:172.21.15.32:6819/2698911336] conn(0x7f580406ef70 0x7f5804071380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:38.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.584+0000 7f581fd1c700 1 --2- 172.21.15.32:0/3916465958 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f58040692a0 0x7f580406b750 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:38.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.584+0000 7f581fd1c700 1 --2- 172.21.15.32:0/3916465958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58180fb4d0 0x7f58180fc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:38.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.584+0000 7f581fd1c700 1 --2- 172.21.15.32:0/3916465958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5818102f50 0x7f58180fcbf0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:38.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.584+0000 7f581fd1c700 1 -- 172.21.15.32:0/3916465958 >> 172.21.15.32:0/3916465958 conn(0x7f58180714c0 msgr2=0x7f58180fa7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:38.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.584+0000 7f581fd1c700 1 -- 172.21.15.32:0/3916465958 shutdown_connections 2024-04-12T05:33:38.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:38.585+0000 7f581fd1c700 1 -- 172.21.15.32:0/3916465958 wait complete. 2024-04-12T05:33:39.092 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:33:39.657 INFO:teuthology.orchestra.run.smithi032.stdout:81604378650 2024-04-12T05:33:39.657 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-12T05:33:40.324 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:40 smithi032 ceph-mon[39731]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:40.325 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:40 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:33:40.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:40 smithi067 ceph-mon[49010]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:40.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:40 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:33:41.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:40.999+0000 7f1f8a14d700 1 -- 172.21.15.32:0/3979287452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f841037c0 msgr2=0x7f1f840fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:41.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:40.999+0000 7f1f8a14d700 1 --2- 172.21.15.32:0/3979287452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f841037c0 0x7f1f840fce30 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1f78004940 tx=0x7f1f78009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:41.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:40.999+0000 7f1f8a14d700 1 -- 172.21.15.32:0/3979287452 shutdown_connections 2024-04-12T05:33:41.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:40.999+0000 7f1f8a14d700 1 --2- 172.21.15.32:0/3979287452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f841037c0 0x7f1f840fce30 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:41.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:40.999+0000 7f1f8a14d700 1 --2- 172.21.15.32:0/3979287452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f84102e20 0x7f1f841031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:41.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:40.999+0000 7f1f8a14d700 1 -- 172.21.15.32:0/3979287452 >> 172.21.15.32:0/3979287452 conn(0x7f1f840f89d0 msgr2=0x7f1f840fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:41.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:40.999+0000 7f1f8a14d700 1 -- 172.21.15.32:0/3979287452 shutdown_connections 2024-04-12T05:33:41.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:40.999+0000 7f1f8a14d700 1 -- 172.21.15.32:0/3979287452 wait complete. 2024-04-12T05:33:41.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.000+0000 7f1f8a14d700 1 Processor -- start 2024-04-12T05:33:41.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.000+0000 7f1f8a14d700 1 -- start start 2024-04-12T05:33:41.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.000+0000 7f1f8a14d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f84102e20 0x7f1f84113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:41.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.000+0000 7f1f8a14d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f841037c0 0x7f1f84114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:41.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.000+0000 7f1f8a14d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f84114660 con 0x7f1f84102e20 2024-04-12T05:33:41.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.000+0000 7f1f8a14d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8410da50 con 0x7f1f841037c0 2024-04-12T05:33:41.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.000+0000 7f1f837fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f84102e20 0x7f1f84113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:41.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.000+0000 7f1f837fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f84102e20 0x7f1f84113ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54746/0 (socket says 172.21.15.32:54746) 2024-04-12T05:33:41.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.000+0000 7f1f837fe700 1 -- 172.21.15.32:0/2069002303 learned_addr learned my addr 172.21.15.32:0/2069002303 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:33:41.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.000+0000 7f1f82ffd700 1 --2- 172.21.15.32:0/2069002303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f841037c0 0x7f1f84114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:41.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.001+0000 7f1f837fe700 1 -- 172.21.15.32:0/2069002303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f841037c0 msgr2=0x7f1f84114010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:41.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.001+0000 7f1f837fe700 1 --2- 172.21.15.32:0/2069002303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f841037c0 0x7f1f84114010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:41.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.001+0000 7f1f837fe700 1 -- 172.21.15.32:0/2069002303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f78005040 con 0x7f1f84102e20 2024-04-12T05:33:41.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.001+0000 7f1f837fe700 1 --2- 172.21.15.32:0/2069002303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f84102e20 0x7f1f84113ad0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1f7400c8c0 tx=0x7f1f740044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:41.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.001+0000 7f1f6bfff700 1 -- 172.21.15.32:0/2069002303 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f7400a980 con 0x7f1f84102e20 2024-04-12T05:33:41.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.001+0000 7f1f6bfff700 1 -- 172.21.15.32:0/2069002303 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f7400ce60 con 0x7f1f84102e20 2024-04-12T05:33:41.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.001+0000 7f1f6bfff700 1 -- 172.21.15.32:0/2069002303 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f740113a0 con 0x7f1f84102e20 2024-04-12T05:33:41.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.001+0000 7f1f8a14d700 1 -- 172.21.15.32:0/2069002303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f8410dd30 con 0x7f1f84102e20 2024-04-12T05:33:41.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.001+0000 7f1f8a14d700 1 -- 172.21.15.32:0/2069002303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f8410e2f0 con 0x7f1f84102e20 2024-04-12T05:33:41.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.002+0000 7f1f8a14d700 1 -- 172.21.15.32:0/2069002303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f840fe570 con 0x7f1f84102e20 2024-04-12T05:33:41.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.003+0000 7f1f6bfff700 1 -- 172.21.15.32:0/2069002303 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f1f74012400 con 0x7f1f84102e20 2024-04-12T05:33:41.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.003+0000 7f1f6bfff700 1 --2- 172.21.15.32:0/2069002303 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f1f6c0692b0 0x7f1f6c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:41.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.003+0000 7f1f6bfff700 1 -- 172.21.15.32:0/2069002303 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1f74086f60 con 0x7f1f84102e20 2024-04-12T05:33:41.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.003+0000 7f1f82ffd700 1 --2- 172.21.15.32:0/2069002303 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f1f6c0692b0 0x7f1f6c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:41.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.003+0000 7f1f82ffd700 1 --2- 172.21.15.32:0/2069002303 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f1f6c0692b0 0x7f1f6c06b760 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1f8410ec10 tx=0x7f1f78009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:41.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.008+0000 7f1f6bfff700 1 -- 172.21.15.32:0/2069002303 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f1f74057060 con 0x7f1f84102e20 2024-04-12T05:33:41.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.205+0000 7f1f8a14d700 1 -- 172.21.15.32:0/2069002303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f1f84109dc0 con 0x7f1f84102e20 2024-04-12T05:33:41.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.205+0000 7f1f6bfff700 1 -- 172.21.15.32:0/2069002303 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f1f74056bf0 con 0x7f1f84102e20 2024-04-12T05:33:41.205 INFO:teuthology.orchestra.run.smithi032.stdout:111669149718 2024-04-12T05:33:41.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.207+0000 7f1f8a14d700 1 -- 172.21.15.32:0/2069002303 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f1f6c0692b0 msgr2=0x7f1f6c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:41.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.207+0000 7f1f8a14d700 1 --2- 172.21.15.32:0/2069002303 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f1f6c0692b0 0x7f1f6c06b760 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1f8410ec10 tx=0x7f1f78009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:41.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.207+0000 7f1f8a14d700 1 -- 172.21.15.32:0/2069002303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f84102e20 msgr2=0x7f1f84113ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:41.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.207+0000 7f1f8a14d700 1 --2- 172.21.15.32:0/2069002303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f84102e20 0x7f1f84113ad0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1f7400c8c0 tx=0x7f1f740044c0 comp rx=0 tx=0).stop 2024-04-12T05:33:41.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.207+0000 7f1f8a14d700 1 -- 172.21.15.32:0/2069002303 shutdown_connections 2024-04-12T05:33:41.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.207+0000 7f1f8a14d700 1 --2- 172.21.15.32:0/2069002303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f84102e20 0x7f1f84113ad0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:41.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.207+0000 7f1f8a14d700 1 --2- 172.21.15.32:0/2069002303 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f1f6c0692b0 0x7f1f6c06b760 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:41.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.207+0000 7f1f8a14d700 1 --2- 172.21.15.32:0/2069002303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f841037c0 0x7f1f84114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:41.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.207+0000 7f1f8a14d700 1 -- 172.21.15.32:0/2069002303 >> 172.21.15.32:0/2069002303 conn(0x7f1f840f89d0 msgr2=0x7f1f840f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:41.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.207+0000 7f1f8a14d700 1 -- 172.21.15.32:0/2069002303 shutdown_connections 2024-04-12T05:33:41.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:41.207+0000 7f1f8a14d700 1 -- 172.21.15.32:0/2069002303 wait complete. 2024-04-12T05:33:42.074 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149716 got 111669149718 for osd.3 2024-04-12T05:33:42.074 DEBUG:teuthology.parallel:result is None 2024-04-12T05:33:42.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:42 smithi032 ceph-mon[39731]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:42.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:42 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/2069002303' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-12T05:33:42.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:42 smithi067 ceph-mon[49010]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:42.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:42 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/2069002303' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-12T05:33:43.003 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:33:43.003 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:33:43.051 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:33:44.268 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:44 smithi032 ceph-mon[39731]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:44.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:44 smithi067 ceph-mon[49010]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:45.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:45 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/2685275403' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-12T05:33:45.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:45 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/2685275403' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-12T05:33:46.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:46 smithi032 ceph-mon[39731]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:46.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.855+0000 7fad4ead3700 1 -- 172.21.15.32:0/2487938529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad48102e00 msgr2=0x7fad481031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:46.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.855+0000 7fad4ead3700 1 --2- 172.21.15.32:0/2487938529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad48102e00 0x7fad481031d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fad3c004940 tx=0x7fad3c009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:46.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.855+0000 7fad4ead3700 1 -- 172.21.15.32:0/2487938529 shutdown_connections 2024-04-12T05:33:46.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.855+0000 7fad4ead3700 1 --2- 172.21.15.32:0/2487938529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad481037a0 0x7fad480fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:46.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.855+0000 7fad4ead3700 1 --2- 172.21.15.32:0/2487938529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad48102e00 0x7fad481031d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:46.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.855+0000 7fad4ead3700 1 -- 172.21.15.32:0/2487938529 >> 172.21.15.32:0/2487938529 conn(0x7fad480f89b0 msgr2=0x7fad480fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:46.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.855+0000 7fad4ead3700 1 -- 172.21.15.32:0/2487938529 shutdown_connections 2024-04-12T05:33:46.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.855+0000 7fad4ead3700 1 -- 172.21.15.32:0/2487938529 wait complete. 2024-04-12T05:33:46.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.856+0000 7fad4ead3700 1 Processor -- start 2024-04-12T05:33:46.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.856+0000 7fad4ead3700 1 -- start start 2024-04-12T05:33:46.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.856+0000 7fad4ead3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad48102e00 0x7fad48113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:46.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.856+0000 7fad4ead3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad481037a0 0x7fad48114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:46.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.856+0000 7fad4ead3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad481146b0 con 0x7fad48102e00 2024-04-12T05:33:46.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.856+0000 7fad4ead3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4810daa0 con 0x7fad481037a0 2024-04-12T05:33:46.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.856+0000 7fad4c86f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad48102e00 0x7fad48113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:46.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.856+0000 7fad4c86f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad48102e00 0x7fad48113b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58112/0 (socket says 172.21.15.32:58112) 2024-04-12T05:33:46.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.856+0000 7fad4c86f700 1 -- 172.21.15.32:0/2685275403 learned_addr learned my addr 172.21.15.32:0/2685275403 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:33:46.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.856+0000 7fad4c86f700 1 -- 172.21.15.32:0/2685275403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad481037a0 msgr2=0x7fad48114060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:46.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.856+0000 7fad4c86f700 1 --2- 172.21.15.32:0/2685275403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad481037a0 0x7fad48114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:46.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.856+0000 7fad4c86f700 1 -- 172.21.15.32:0/2685275403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad3c005040 con 0x7fad48102e00 2024-04-12T05:33:46.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.857+0000 7fad4c86f700 1 --2- 172.21.15.32:0/2685275403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad48102e00 0x7fad48113b20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fad3c009180 tx=0x7fad3c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:46.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.857+0000 7fad457fa700 1 -- 172.21.15.32:0/2685275403 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad3c00e040 con 0x7fad48102e00 2024-04-12T05:33:46.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.857+0000 7fad457fa700 1 -- 172.21.15.32:0/2685275403 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad3c004290 con 0x7fad48102e00 2024-04-12T05:33:46.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.857+0000 7fad4ead3700 1 -- 172.21.15.32:0/2685275403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad4810dd20 con 0x7fad48102e00 2024-04-12T05:33:46.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.857+0000 7fad457fa700 1 -- 172.21.15.32:0/2685275403 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad3c01c3f0 con 0x7fad48102e00 2024-04-12T05:33:46.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.857+0000 7fad4ead3700 1 -- 172.21.15.32:0/2685275403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad4810e2e0 con 0x7fad48102e00 2024-04-12T05:33:46.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.858+0000 7fad4ead3700 1 -- 172.21.15.32:0/2685275403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad480fe550 con 0x7fad48102e00 2024-04-12T05:33:46.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.859+0000 7fad457fa700 1 -- 172.21.15.32:0/2685275403 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fad3c013790 con 0x7fad48102e00 2024-04-12T05:33:46.508 INFO:teuthology.orchestra.run.smithi032.stdout:146028888081 2024-04-12T05:33:46.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.859+0000 7fad457fa700 1 --2- 172.21.15.32:0/2685275403 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fad380692a0 0x7fad3806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:46.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.859+0000 7fad457fa700 1 -- 172.21.15.32:0/2685275403 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fad3c088b60 con 0x7fad48102e00 2024-04-12T05:33:46.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.859+0000 7fad47fff700 1 --2- 172.21.15.32:0/2685275403 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fad380692a0 0x7fad3806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:46.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.859+0000 7fad47fff700 1 --2- 172.21.15.32:0/2685275403 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fad380692a0 0x7fad3806b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fad4810ec60 tx=0x7fad34007cc0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:46.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:44.863+0000 7fad457fa700 1 -- 172.21.15.32:0/2685275403 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fad3c058d00 con 0x7fad48102e00 2024-04-12T05:33:46.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:45.064+0000 7fad4ead3700 1 -- 172.21.15.32:0/2685275403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fad48109df0 con 0x7fad48102e00 2024-04-12T05:33:46.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:45.064+0000 7fad457fa700 1 -- 172.21.15.32:0/2685275403 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fad3c013420 con 0x7fad48102e00 2024-04-12T05:33:46.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:45.066+0000 7fad4ead3700 1 -- 172.21.15.32:0/2685275403 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fad380692a0 msgr2=0x7fad3806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:46.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:45.066+0000 7fad4ead3700 1 --2- 172.21.15.32:0/2685275403 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fad380692a0 0x7fad3806b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fad4810ec60 tx=0x7fad34007cc0 comp rx=0 tx=0).stop 2024-04-12T05:33:46.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:45.066+0000 7fad4ead3700 1 -- 172.21.15.32:0/2685275403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad48102e00 msgr2=0x7fad48113b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:46.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:45.066+0000 7fad4ead3700 1 --2- 172.21.15.32:0/2685275403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad48102e00 0x7fad48113b20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fad3c009180 tx=0x7fad3c009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:46.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:45.066+0000 7fad4ead3700 1 -- 172.21.15.32:0/2685275403 shutdown_connections 2024-04-12T05:33:46.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:45.066+0000 7fad4ead3700 1 --2- 172.21.15.32:0/2685275403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad48102e00 0x7fad48113b20 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:46.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:45.066+0000 7fad4ead3700 1 --2- 172.21.15.32:0/2685275403 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fad380692a0 0x7fad3806b750 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:46.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:45.066+0000 7fad4ead3700 1 --2- 172.21.15.32:0/2685275403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad481037a0 0x7fad48114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:46.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:45.066+0000 7fad4ead3700 1 -- 172.21.15.32:0/2685275403 >> 172.21.15.32:0/2685275403 conn(0x7fad480f89b0 msgr2=0x7fad480f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:46.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:45.066+0000 7fad4ead3700 1 -- 172.21.15.32:0/2685275403 shutdown_connections 2024-04-12T05:33:46.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:45.066+0000 7fad4ead3700 1 -- 172.21.15.32:0/2685275403 wait complete. 2024-04-12T05:33:46.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:46 smithi067 ceph-mon[49010]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:47.265 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888079 got 146028888081 for osd.4 2024-04-12T05:33:47.265 DEBUG:teuthology.parallel:result is None 2024-04-12T05:33:47.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.460+0000 7f0f6ccc9700 1 -- 172.21.15.32:0/1134973449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f681037a0 msgr2=0x7f0f680fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:47.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.460+0000 7f0f6ccc9700 1 --2- 172.21.15.32:0/1134973449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f681037a0 0x7f0f680fce10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c004940 tx=0x7f0f5c009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:47.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.461+0000 7f0f6ccc9700 1 -- 172.21.15.32:0/1134973449 shutdown_connections 2024-04-12T05:33:47.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.461+0000 7f0f6ccc9700 1 --2- 172.21.15.32:0/1134973449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f681037a0 0x7f0f680fce10 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:47.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.461+0000 7f0f6ccc9700 1 --2- 172.21.15.32:0/1134973449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f68102e00 0x7f0f681031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:47.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.461+0000 7f0f6ccc9700 1 -- 172.21.15.32:0/1134973449 >> 172.21.15.32:0/1134973449 conn(0x7f0f680f89b0 msgr2=0x7f0f680fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:47.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.461+0000 7f0f6ccc9700 1 -- 172.21.15.32:0/1134973449 shutdown_connections 2024-04-12T05:33:47.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.461+0000 7f0f6ccc9700 1 -- 172.21.15.32:0/1134973449 wait complete. 2024-04-12T05:33:47.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.461+0000 7f0f6ccc9700 1 Processor -- start 2024-04-12T05:33:47.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.461+0000 7f0f6ccc9700 1 -- start start 2024-04-12T05:33:47.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.461+0000 7f0f6ccc9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f68102e00 0x7f0f680702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:47.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.461+0000 7f0f6ccc9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f681037a0 0x7f0f68070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:47.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.461+0000 7f0f6ccc9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f68072210 con 0x7f0f68102e00 2024-04-12T05:33:47.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.461+0000 7f0f6ccc9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f68070d40 con 0x7f0f681037a0 2024-04-12T05:33:47.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.462+0000 7f0f6659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f68102e00 0x7f0f680702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:47.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.462+0000 7f0f6659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f68102e00 0x7f0f680702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58126/0 (socket says 172.21.15.32:58126) 2024-04-12T05:33:47.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.462+0000 7f0f6659c700 1 -- 172.21.15.32:0/1699913542 learned_addr learned my addr 172.21.15.32:0/1699913542 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:33:47.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.462+0000 7f0f65d9b700 1 --2- 172.21.15.32:0/1699913542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f681037a0 0x7f0f68070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:47.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.462+0000 7f0f6659c700 1 -- 172.21.15.32:0/1699913542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f681037a0 msgr2=0x7f0f68070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:47.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.462+0000 7f0f6659c700 1 --2- 172.21.15.32:0/1699913542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f681037a0 0x7f0f68070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:47.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.462+0000 7f0f6659c700 1 -- 172.21.15.32:0/1699913542 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f5c005040 con 0x7f0f68102e00 2024-04-12T05:33:47.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.462+0000 7f0f6659c700 1 --2- 172.21.15.32:0/1699913542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f68102e00 0x7f0f680702c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0f58002a50 tx=0x7f0f580046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:47.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.462+0000 7f0f56ffd700 1 -- 172.21.15.32:0/1699913542 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f58007cc0 con 0x7f0f68102e00 2024-04-12T05:33:47.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.462+0000 7f0f56ffd700 1 -- 172.21.15.32:0/1699913542 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f5800f040 con 0x7f0f68102e00 2024-04-12T05:33:47.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.462+0000 7f0f6ccc9700 1 -- 172.21.15.32:0/1699913542 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f68071020 con 0x7f0f68102e00 2024-04-12T05:33:47.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.462+0000 7f0f56ffd700 1 -- 172.21.15.32:0/1699913542 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f5800d9c0 con 0x7f0f68102e00 2024-04-12T05:33:47.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.462+0000 7f0f6ccc9700 1 -- 172.21.15.32:0/1699913542 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f6811a070 con 0x7f0f68102e00 2024-04-12T05:33:47.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.464+0000 7f0f56ffd700 1 -- 172.21.15.32:0/1699913542 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f0f5800db20 con 0x7f0f68102e00 2024-04-12T05:33:47.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.464+0000 7f0f6ccc9700 1 -- 172.21.15.32:0/1699913542 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f680fe550 con 0x7f0f68102e00 2024-04-12T05:33:47.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.465+0000 7f0f56ffd700 1 --2- 172.21.15.32:0/1699913542 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f0f500692a0 0x7f0f5006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:47.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.465+0000 7f0f56ffd700 1 -- 172.21.15.32:0/1699913542 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0f58086be0 con 0x7f0f68102e00 2024-04-12T05:33:47.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.465+0000 7f0f65d9b700 1 --2- 172.21.15.32:0/1699913542 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f0f500692a0 0x7f0f5006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:47.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.465+0000 7f0f65d9b700 1 --2- 172.21.15.32:0/1699913542 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f0f500692a0 0x7f0f5006b750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0f680719f0 tx=0x7f0f5c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:47.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.469+0000 7f0f56ffd700 1 -- 172.21.15.32:0/1699913542 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f0f58053130 con 0x7f0f68102e00 2024-04-12T05:33:47.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.667+0000 7f0f6ccc9700 1 -- 172.21.15.32:0/1699913542 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f0f6804c710 con 0x7f0f68102e00 2024-04-12T05:33:47.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.667+0000 7f0f56ffd700 1 -- 172.21.15.32:0/1699913542 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0f58056750 con 0x7f0f68102e00 2024-04-12T05:33:47.667 INFO:teuthology.orchestra.run.smithi032.stdout:38654705704 2024-04-12T05:33:47.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.669+0000 7f0f6ccc9700 1 -- 172.21.15.32:0/1699913542 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f0f500692a0 msgr2=0x7f0f5006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:47.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.669+0000 7f0f6ccc9700 1 --2- 172.21.15.32:0/1699913542 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f0f500692a0 0x7f0f5006b750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0f680719f0 tx=0x7f0f5c009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:47.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.669+0000 7f0f6ccc9700 1 -- 172.21.15.32:0/1699913542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f68102e00 msgr2=0x7f0f680702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:47.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.669+0000 7f0f6ccc9700 1 --2- 172.21.15.32:0/1699913542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f68102e00 0x7f0f680702c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0f58002a50 tx=0x7f0f580046b0 comp rx=0 tx=0).stop 2024-04-12T05:33:47.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.669+0000 7f0f6ccc9700 1 -- 172.21.15.32:0/1699913542 shutdown_connections 2024-04-12T05:33:47.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.669+0000 7f0f6ccc9700 1 --2- 172.21.15.32:0/1699913542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f68102e00 0x7f0f680702c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:47.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.669+0000 7f0f6ccc9700 1 --2- 172.21.15.32:0/1699913542 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f0f500692a0 0x7f0f5006b750 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:47.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.669+0000 7f0f6ccc9700 1 --2- 172.21.15.32:0/1699913542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f681037a0 0x7f0f68070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:47.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.669+0000 7f0f6ccc9700 1 -- 172.21.15.32:0/1699913542 >> 172.21.15.32:0/1699913542 conn(0x7f0f680f89b0 msgr2=0x7f0f680f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:47.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.669+0000 7f0f6ccc9700 1 -- 172.21.15.32:0/1699913542 shutdown_connections 2024-04-12T05:33:47.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:47.669+0000 7f0f6ccc9700 1 -- 172.21.15.32:0/1699913542 wait complete. 2024-04-12T05:33:48.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:48 smithi067 ceph-mon[49010]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:48.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:48 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/1699913542' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-12T05:33:48.548 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:48 smithi032 ceph-mon[39731]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:48.548 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:48 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/1699913542' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-12T05:33:48.568 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705700 got 38654705704 for osd.0 2024-04-12T05:33:48.568 DEBUG:teuthology.parallel:result is None 2024-04-12T05:33:49.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.467+0000 7fd7b37fa700 1 -- 172.21.15.32:0/3617272157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7ac102e00 msgr2=0x7fd7ac1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:49.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.467+0000 7fd7b37fa700 1 --2- 172.21.15.32:0/3617272157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7ac102e00 0x7fd7ac1031d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8004940 tx=0x7fd7a8009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:49.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.467+0000 7fd7b37fa700 1 -- 172.21.15.32:0/3617272157 shutdown_connections 2024-04-12T05:33:49.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.467+0000 7fd7b37fa700 1 --2- 172.21.15.32:0/3617272157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7ac1037a0 0x7fd7ac0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:49.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.467+0000 7fd7b37fa700 1 --2- 172.21.15.32:0/3617272157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7ac102e00 0x7fd7ac1031d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:49.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.467+0000 7fd7b37fa700 1 -- 172.21.15.32:0/3617272157 >> 172.21.15.32:0/3617272157 conn(0x7fd7ac0f89b0 msgr2=0x7fd7ac0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:49.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.467+0000 7fd7b37fa700 1 -- 172.21.15.32:0/3617272157 shutdown_connections 2024-04-12T05:33:49.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.467+0000 7fd7b37fa700 1 -- 172.21.15.32:0/3617272157 wait complete. 2024-04-12T05:33:49.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.468+0000 7fd7b37fa700 1 Processor -- start 2024-04-12T05:33:49.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.468+0000 7fd7b37fa700 1 -- start start 2024-04-12T05:33:49.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.468+0000 7fd7b37fa700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7ac102e00 0x7fd7ac0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:49.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.468+0000 7fd7b37fa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7ac1037a0 0x7fd7ac070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:49.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.468+0000 7fd7b37fa700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ac070d90 con 0x7fd7ac1037a0 2024-04-12T05:33:49.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.468+0000 7fd7b37fa700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ac070ed0 con 0x7fd7ac102e00 2024-04-12T05:33:49.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.468+0000 7fd7b0d95700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7ac1037a0 0x7fd7ac070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:49.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.468+0000 7fd7b0d95700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7ac1037a0 0x7fd7ac070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58148/0 (socket says 172.21.15.32:58148) 2024-04-12T05:33:49.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.468+0000 7fd7b0d95700 1 -- 172.21.15.32:0/2841180973 learned_addr learned my addr 172.21.15.32:0/2841180973 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:33:49.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.468+0000 7fd7b1596700 1 --2- 172.21.15.32:0/2841180973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7ac102e00 0x7fd7ac0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:49.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.468+0000 7fd7b0d95700 1 -- 172.21.15.32:0/2841180973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7ac102e00 msgr2=0x7fd7ac0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:49.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.468+0000 7fd7b0d95700 1 --2- 172.21.15.32:0/2841180973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7ac102e00 0x7fd7ac0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:49.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.468+0000 7fd7b0d95700 1 -- 172.21.15.32:0/2841180973 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a8005040 con 0x7fd7ac1037a0 2024-04-12T05:33:49.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.468+0000 7fd7b1596700 1 --2- 172.21.15.32:0/2841180973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7ac102e00 0x7fd7ac0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:33:49.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.468+0000 7fd7b0d95700 1 --2- 172.21.15.32:0/2841180973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7ac1037a0 0x7fd7ac070800 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0002a50 tx=0x7fd7a0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:49.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.469+0000 7fd79dffb700 1 -- 172.21.15.32:0/2841180973 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7a0007cc0 con 0x7fd7ac1037a0 2024-04-12T05:33:49.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.469+0000 7fd79dffb700 1 -- 172.21.15.32:0/2841180973 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd7a000f040 con 0x7fd7ac1037a0 2024-04-12T05:33:49.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.469+0000 7fd7b37fa700 1 -- 172.21.15.32:0/2841180973 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7ac119e20 con 0x7fd7ac1037a0 2024-04-12T05:33:49.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.469+0000 7fd79dffb700 1 -- 172.21.15.32:0/2841180973 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7a000d990 con 0x7fd7ac1037a0 2024-04-12T05:33:49.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.469+0000 7fd7b37fa700 1 -- 172.21.15.32:0/2841180973 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7ac11a2a0 con 0x7fd7ac1037a0 2024-04-12T05:33:49.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.470+0000 7fd7b37fa700 1 -- 172.21.15.32:0/2841180973 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7ac0fe550 con 0x7fd7ac1037a0 2024-04-12T05:33:49.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.470+0000 7fd79dffb700 1 -- 172.21.15.32:0/2841180973 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fd7a0009710 con 0x7fd7ac1037a0 2024-04-12T05:33:49.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.471+0000 7fd79dffb700 1 --2- 172.21.15.32:0/2841180973 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fd7980692b0 0x7fd79806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:49.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.471+0000 7fd79dffb700 1 -- 172.21.15.32:0/2841180973 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd7a0086c20 con 0x7fd7ac1037a0 2024-04-12T05:33:49.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.471+0000 7fd7b1596700 1 --2- 172.21.15.32:0/2841180973 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fd7980692b0 0x7fd79806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:49.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.471+0000 7fd7b1596700 1 --2- 172.21.15.32:0/2841180973 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fd7980692b0 0x7fd79806b760 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd7a800b3b0 tx=0x7fd7a8009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:49.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.475+0000 7fd79dffb700 1 -- 172.21.15.32:0/2841180973 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fd7a00531f0 con 0x7fd7ac1037a0 2024-04-12T05:33:49.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.672+0000 7fd7b37fa700 1 -- 172.21.15.32:0/2841180973 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd7ac05f5f0 con 0x7fd7ac1037a0 2024-04-12T05:33:49.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.672+0000 7fd79dffb700 1 -- 172.21.15.32:0/2841180973 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd7a0056810 con 0x7fd7ac1037a0 2024-04-12T05:33:49.670 INFO:teuthology.orchestra.run.smithi032.stdout:60129542179 2024-04-12T05:33:49.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.675+0000 7fd7b37fa700 1 -- 172.21.15.32:0/2841180973 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fd7980692b0 msgr2=0x7fd79806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:49.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.675+0000 7fd7b37fa700 1 --2- 172.21.15.32:0/2841180973 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fd7980692b0 0x7fd79806b760 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd7a800b3b0 tx=0x7fd7a8009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:49.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.675+0000 7fd7b37fa700 1 -- 172.21.15.32:0/2841180973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7ac1037a0 msgr2=0x7fd7ac070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:49.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.675+0000 7fd7b37fa700 1 --2- 172.21.15.32:0/2841180973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7ac1037a0 0x7fd7ac070800 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0002a50 tx=0x7fd7a0004600 comp rx=0 tx=0).stop 2024-04-12T05:33:49.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.675+0000 7fd7b37fa700 1 -- 172.21.15.32:0/2841180973 shutdown_connections 2024-04-12T05:33:49.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.675+0000 7fd7b37fa700 1 --2- 172.21.15.32:0/2841180973 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fd7980692b0 0x7fd79806b760 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:49.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.675+0000 7fd7b37fa700 1 --2- 172.21.15.32:0/2841180973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7ac102e00 0x7fd7ac0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:49.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.675+0000 7fd7b37fa700 1 --2- 172.21.15.32:0/2841180973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7ac1037a0 0x7fd7ac070800 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:49.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.676+0000 7fd7b37fa700 1 -- 172.21.15.32:0/2841180973 >> 172.21.15.32:0/2841180973 conn(0x7fd7ac0f89b0 msgr2=0x7fd7ac0f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:49.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.676+0000 7fd7b37fa700 1 -- 172.21.15.32:0/2841180973 shutdown_connections 2024-04-12T05:33:49.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:49.676+0000 7fd7b37fa700 1 -- 172.21.15.32:0/2841180973 wait complete. 2024-04-12T05:33:50.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.402+0000 7f1eec659700 1 -- 172.21.15.32:0/2330351664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ee4102710 msgr2=0x7f1ee40fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:50.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.402+0000 7f1eec659700 1 --2- 172.21.15.32:0/2330351664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ee4102710 0x7f1ee40fbd80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1ed8004940 tx=0x7f1ed8009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:50.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.402+0000 7f1eec659700 1 -- 172.21.15.32:0/2330351664 shutdown_connections 2024-04-12T05:33:50.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.402+0000 7f1eec659700 1 --2- 172.21.15.32:0/2330351664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ee4102710 0x7f1ee40fbd80 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:50.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.402+0000 7f1eec659700 1 --2- 172.21.15.32:0/2330351664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ee4101d70 0x7f1ee4102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:50.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.402+0000 7f1eec659700 1 -- 172.21.15.32:0/2330351664 >> 172.21.15.32:0/2330351664 conn(0x7f1ee4072bc0 msgr2=0x7f1ee4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:50.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.402+0000 7f1eec659700 1 -- 172.21.15.32:0/2330351664 shutdown_connections 2024-04-12T05:33:50.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.402+0000 7f1eec659700 1 -- 172.21.15.32:0/2330351664 wait complete. 2024-04-12T05:33:50.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.403+0000 7f1eec659700 1 Processor -- start 2024-04-12T05:33:50.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.403+0000 7f1eec659700 1 -- start start 2024-04-12T05:33:50.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.403+0000 7f1eec659700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ee4101d70 0x7f1ee410ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:50.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.403+0000 7f1eec659700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ee4102710 0x7f1ee410efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:50.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.403+0000 7f1eec659700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee410f5f0 con 0x7f1ee4102710 2024-04-12T05:33:50.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.403+0000 7f1eec659700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee410f730 con 0x7f1ee4101d70 2024-04-12T05:33:50.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.403+0000 7f1ee9bf4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ee4102710 0x7f1ee410efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:50.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.403+0000 7f1ee9bf4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ee4102710 0x7f1ee410efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58164/0 (socket says 172.21.15.32:58164) 2024-04-12T05:33:50.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.403+0000 7f1ee9bf4700 1 -- 172.21.15.32:0/2937096183 learned_addr learned my addr 172.21.15.32:0/2937096183 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:33:50.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.403+0000 7f1ee9bf4700 1 -- 172.21.15.32:0/2937096183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ee4101d70 msgr2=0x7f1ee410ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:50.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.403+0000 7f1ee9bf4700 1 --2- 172.21.15.32:0/2937096183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ee4101d70 0x7f1ee410ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:50.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.403+0000 7f1ee9bf4700 1 -- 172.21.15.32:0/2937096183 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed8005040 con 0x7f1ee4102710 2024-04-12T05:33:50.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.404+0000 7f1ee9bf4700 1 --2- 172.21.15.32:0/2937096183 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ee4102710 0x7f1ee410efa0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1ed8009380 tx=0x7f1ed8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:50.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.404+0000 7f1ed6ffd700 1 -- 172.21.15.32:0/2937096183 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ed800e040 con 0x7f1ee4102710 2024-04-12T05:33:50.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.404+0000 7f1eec659700 1 -- 172.21.15.32:0/2937096183 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ee4117ad0 con 0x7f1ee4102710 2024-04-12T05:33:50.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.404+0000 7f1ed6ffd700 1 -- 172.21.15.32:0/2937096183 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ed8013420 con 0x7f1ee4102710 2024-04-12T05:33:50.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.404+0000 7f1ed6ffd700 1 -- 172.21.15.32:0/2937096183 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ed801c630 con 0x7f1ee4102710 2024-04-12T05:33:50.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.404+0000 7f1eec659700 1 -- 172.21.15.32:0/2937096183 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee4117fe0 con 0x7f1ee4102710 2024-04-12T05:33:50.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.405+0000 7f1eec659700 1 -- 172.21.15.32:0/2937096183 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ee40ff080 con 0x7f1ee4102710 2024-04-12T05:33:50.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.405+0000 7f1ed6ffd700 1 -- 172.21.15.32:0/2937096183 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f1ed80139e0 con 0x7f1ee4102710 2024-04-12T05:33:50.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.406+0000 7f1ed6ffd700 1 --2- 172.21.15.32:0/2937096183 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f1ed00692a0 0x7f1ed006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:50.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.406+0000 7f1ed6ffd700 1 -- 172.21.15.32:0/2937096183 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1ed8088090 con 0x7f1ee4102710 2024-04-12T05:33:50.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.406+0000 7f1eea3f5700 1 --2- 172.21.15.32:0/2937096183 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f1ed00692a0 0x7f1ed006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:50.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.406+0000 7f1eea3f5700 1 --2- 172.21.15.32:0/2937096183 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f1ed00692a0 0x7f1ed006b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1ee0004d00 tx=0x7f1ee0009ce0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:50.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.410+0000 7f1ed6ffd700 1 -- 172.21.15.32:0/2937096183 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f1ed8054660 con 0x7f1ee4102710 2024-04-12T05:33:50.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:50 smithi067 ceph-mon[49010]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:50.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:50 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/2841180973' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-12T05:33:50.569 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:50 smithi032 ceph-mon[39731]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:50.569 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:50 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/2841180973' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-12T05:33:50.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.606+0000 7f1eec659700 1 -- 172.21.15.32:0/2937096183 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f1ee40fd4c0 con 0x7f1ee4102710 2024-04-12T05:33:50.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.606+0000 7f1ed6ffd700 1 -- 172.21.15.32:0/2937096183 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1ed8057c80 con 0x7f1ee4102710 2024-04-12T05:33:50.604 INFO:teuthology.orchestra.run.smithi032.stdout:81604378652 2024-04-12T05:33:50.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.608+0000 7f1eec659700 1 -- 172.21.15.32:0/2937096183 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f1ed00692a0 msgr2=0x7f1ed006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:50.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.608+0000 7f1eec659700 1 --2- 172.21.15.32:0/2937096183 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f1ed00692a0 0x7f1ed006b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1ee0004d00 tx=0x7f1ee0009ce0 comp rx=0 tx=0).stop 2024-04-12T05:33:50.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.608+0000 7f1eec659700 1 -- 172.21.15.32:0/2937096183 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ee4102710 msgr2=0x7f1ee410efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:50.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.608+0000 7f1eec659700 1 --2- 172.21.15.32:0/2937096183 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ee4102710 0x7f1ee410efa0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1ed8009380 tx=0x7f1ed8009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:50.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.608+0000 7f1eec659700 1 -- 172.21.15.32:0/2937096183 shutdown_connections 2024-04-12T05:33:50.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.608+0000 7f1eec659700 1 --2- 172.21.15.32:0/2937096183 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f1ed00692a0 0x7f1ed006b750 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:50.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.608+0000 7f1eec659700 1 --2- 172.21.15.32:0/2937096183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ee4101d70 0x7f1ee410ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:50.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.608+0000 7f1eec659700 1 --2- 172.21.15.32:0/2937096183 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ee4102710 0x7f1ee410efa0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:50.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.608+0000 7f1eec659700 1 -- 172.21.15.32:0/2937096183 >> 172.21.15.32:0/2937096183 conn(0x7f1ee4072bc0 msgr2=0x7f1ee40fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:50.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.608+0000 7f1eec659700 1 -- 172.21.15.32:0/2937096183 shutdown_connections 2024-04-12T05:33:50.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:50.608+0000 7f1eec659700 1 -- 172.21.15.32:0/2937096183 wait complete. 2024-04-12T05:33:50.840 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542175 got 60129542179 for osd.1 2024-04-12T05:33:50.840 DEBUG:teuthology.parallel:result is None 2024-04-12T05:33:51.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.030+0000 7f0c9c9ae700 1 -- 172.21.15.32:0/1400316116 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c98102e10 msgr2=0x7f0c981031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:51.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.030+0000 7f0c9c9ae700 1 --2- 172.21.15.32:0/1400316116 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c98102e10 0x7f0c981031e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c004940 tx=0x7f0c8c009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:51.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.030+0000 7f0c9c9ae700 1 -- 172.21.15.32:0/1400316116 shutdown_connections 2024-04-12T05:33:51.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.030+0000 7f0c9c9ae700 1 --2- 172.21.15.32:0/1400316116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c981037b0 0x7f0c980fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:51.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.030+0000 7f0c9c9ae700 1 --2- 172.21.15.32:0/1400316116 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c98102e10 0x7f0c981031e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:51.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.030+0000 7f0c9c9ae700 1 -- 172.21.15.32:0/1400316116 >> 172.21.15.32:0/1400316116 conn(0x7f0c980f89a0 msgr2=0x7f0c980fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:51.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.030+0000 7f0c9c9ae700 1 -- 172.21.15.32:0/1400316116 shutdown_connections 2024-04-12T05:33:51.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.030+0000 7f0c9c9ae700 1 -- 172.21.15.32:0/1400316116 wait complete. 2024-04-12T05:33:51.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.030+0000 7f0c9c9ae700 1 Processor -- start 2024-04-12T05:33:51.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.030+0000 7f0c9c9ae700 1 -- start start 2024-04-12T05:33:51.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.031+0000 7f0c9c9ae700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c98102e10 0x7f0c98113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:51.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.031+0000 7f0c9c9ae700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c981037b0 0x7f0c98114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:51.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.031+0000 7f0c9c9ae700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c98114680 con 0x7f0c981037b0 2024-04-12T05:33:51.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.031+0000 7f0c9c9ae700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9810da70 con 0x7f0c98102e10 2024-04-12T05:33:51.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.031+0000 7f0c95d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c981037b0 0x7f0c98114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:51.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.031+0000 7f0c95d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c981037b0 0x7f0c98114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58188/0 (socket says 172.21.15.32:58188) 2024-04-12T05:33:51.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.031+0000 7f0c95d9b700 1 -- 172.21.15.32:0/1489105920 learned_addr learned my addr 172.21.15.32:0/1489105920 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:33:51.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.031+0000 7f0c9659c700 1 --2- 172.21.15.32:0/1489105920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c98102e10 0x7f0c98113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:51.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.031+0000 7f0c95d9b700 1 -- 172.21.15.32:0/1489105920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c98102e10 msgr2=0x7f0c98113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:51.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.031+0000 7f0c95d9b700 1 --2- 172.21.15.32:0/1489105920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c98102e10 0x7f0c98113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:51.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.031+0000 7f0c95d9b700 1 -- 172.21.15.32:0/1489105920 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c8c005040 con 0x7f0c981037b0 2024-04-12T05:33:51.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.031+0000 7f0c95d9b700 1 --2- 172.21.15.32:0/1489105920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c981037b0 0x7f0c98114030 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0c88002a50 tx=0x7f0c88004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:51.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.031+0000 7f0c86ffd700 1 -- 172.21.15.32:0/1489105920 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c88007cc0 con 0x7f0c981037b0 2024-04-12T05:33:51.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.031+0000 7f0c86ffd700 1 -- 172.21.15.32:0/1489105920 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c8800f040 con 0x7f0c981037b0 2024-04-12T05:33:51.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.031+0000 7f0c9c9ae700 1 -- 172.21.15.32:0/1489105920 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c9810dd50 con 0x7f0c981037b0 2024-04-12T05:33:51.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.031+0000 7f0c86ffd700 1 -- 172.21.15.32:0/1489105920 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c8800d920 con 0x7f0c981037b0 2024-04-12T05:33:51.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.031+0000 7f0c9c9ae700 1 -- 172.21.15.32:0/1489105920 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c9810e310 con 0x7f0c981037b0 2024-04-12T05:33:51.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.032+0000 7f0c9c9ae700 1 -- 172.21.15.32:0/1489105920 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c980fe560 con 0x7f0c981037b0 2024-04-12T05:33:51.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.033+0000 7f0c86ffd700 1 -- 172.21.15.32:0/1489105920 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f0c88009680 con 0x7f0c981037b0 2024-04-12T05:33:51.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.034+0000 7f0c86ffd700 1 --2- 172.21.15.32:0/1489105920 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f0c800692a0 0x7f0c8006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:51.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.034+0000 7f0c86ffd700 1 -- 172.21.15.32:0/1489105920 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0c88087110 con 0x7f0c981037b0 2024-04-12T05:33:51.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.034+0000 7f0c9659c700 1 --2- 172.21.15.32:0/1489105920 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f0c800692a0 0x7f0c8006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:51.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.034+0000 7f0c9659c700 1 --2- 172.21.15.32:0/1489105920 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f0c800692a0 0x7f0c8006b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c004f90 tx=0x7f0c8c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:51.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.038+0000 7f0c86ffd700 1 -- 172.21.15.32:0/1489105920 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f0c880572b0 con 0x7f0c981037b0 2024-04-12T05:33:51.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.238+0000 7f0c9c9ae700 1 -- 172.21.15.32:0/1489105920 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f0c9804bed0 con 0x7f0c981037b0 2024-04-12T05:33:51.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.238+0000 7f0c86ffd700 1 -- 172.21.15.32:0/1489105920 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f0c88056e40 con 0x7f0c981037b0 2024-04-12T05:33:51.237 INFO:teuthology.orchestra.run.smithi032.stdout:167503724558 2024-04-12T05:33:51.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.240+0000 7f0c9c9ae700 1 -- 172.21.15.32:0/1489105920 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f0c800692a0 msgr2=0x7f0c8006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:51.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.240+0000 7f0c9c9ae700 1 --2- 172.21.15.32:0/1489105920 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f0c800692a0 0x7f0c8006b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c004f90 tx=0x7f0c8c009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:51.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.240+0000 7f0c9c9ae700 1 -- 172.21.15.32:0/1489105920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c981037b0 msgr2=0x7f0c98114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:51.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.240+0000 7f0c9c9ae700 1 --2- 172.21.15.32:0/1489105920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c981037b0 0x7f0c98114030 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0c88002a50 tx=0x7f0c88004600 comp rx=0 tx=0).stop 2024-04-12T05:33:51.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.240+0000 7f0c9c9ae700 1 -- 172.21.15.32:0/1489105920 shutdown_connections 2024-04-12T05:33:51.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.240+0000 7f0c9c9ae700 1 --2- 172.21.15.32:0/1489105920 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f0c800692a0 0x7f0c8006b750 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:51.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.240+0000 7f0c9c9ae700 1 --2- 172.21.15.32:0/1489105920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c98102e10 0x7f0c98113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:51.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.240+0000 7f0c9c9ae700 1 --2- 172.21.15.32:0/1489105920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c981037b0 0x7f0c98114030 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:51.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.240+0000 7f0c9c9ae700 1 -- 172.21.15.32:0/1489105920 >> 172.21.15.32:0/1489105920 conn(0x7f0c980f89a0 msgr2=0x7f0c980f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:51.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.240+0000 7f0c9c9ae700 1 -- 172.21.15.32:0/1489105920 shutdown_connections 2024-04-12T05:33:51.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:51.240+0000 7f0c9c9ae700 1 -- 172.21.15.32:0/1489105920 wait complete. 2024-04-12T05:33:51.523 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:51 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/2937096183' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-12T05:33:51.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:51 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/2937096183' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-12T05:33:51.904 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724554 got 167503724558 for osd.5 2024-04-12T05:33:51.904 DEBUG:teuthology.parallel:result is None 2024-04-12T05:33:52.370 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378650 got 81604378652 for osd.2 2024-04-12T05:33:52.371 DEBUG:teuthology.parallel:result is None 2024-04-12T05:33:52.371 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-12T05:33:52.371 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph pg dump --format=json 2024-04-12T05:33:52.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:52 smithi067 ceph-mon[49010]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:52.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:52 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/1489105920' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-12T05:33:52.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:52 smithi032 ceph-mon[39731]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:52.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:52 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/1489105920' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-12T05:33:52.653 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:33:54.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:54 smithi032 ceph-mon[39731]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:54.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.346+0000 7fd4bd07d700 1 -- 172.21.15.32:0/1609268437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4b80fde90 msgr2=0x7fd4b80fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:54.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.346+0000 7fd4bd07d700 1 --2- 172.21.15.32:0/1609268437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4b80fde90 0x7fd4b80fe260 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8004940 tx=0x7fd4a8009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:54.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.346+0000 7fd4bd07d700 1 -- 172.21.15.32:0/1609268437 shutdown_connections 2024-04-12T05:33:54.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.346+0000 7fd4bd07d700 1 --2- 172.21.15.32:0/1609268437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4b80fe830 0x7fd4b8106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:54.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.346+0000 7fd4bd07d700 1 --2- 172.21.15.32:0/1609268437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4b80fde90 0x7fd4b80fe260 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:54.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.346+0000 7fd4bd07d700 1 -- 172.21.15.32:0/1609268437 >> 172.21.15.32:0/1609268437 conn(0x7fd4b8072bc0 msgr2=0x7fd4b8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:54.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.346+0000 7fd4bd07d700 1 -- 172.21.15.32:0/1609268437 shutdown_connections 2024-04-12T05:33:54.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.346+0000 7fd4bd07d700 1 -- 172.21.15.32:0/1609268437 wait complete. 2024-04-12T05:33:54.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.346+0000 7fd4bd07d700 1 Processor -- start 2024-04-12T05:33:54.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.347+0000 7fd4bd07d700 1 -- start start 2024-04-12T05:33:54.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.347+0000 7fd4bd07d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4b80fde90 0x7fd4b8113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:54.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.347+0000 7fd4bd07d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4b80fe830 0x7fd4b8114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:54.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.347+0000 7fd4bd07d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b8114710 con 0x7fd4b80fe830 2024-04-12T05:33:54.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.347+0000 7fd4bd07d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b810da70 con 0x7fd4b80fde90 2024-04-12T05:33:54.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.347+0000 7fd4b659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4b80fe830 0x7fd4b8114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:54.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.347+0000 7fd4b659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4b80fe830 0x7fd4b8114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40256/0 (socket says 172.21.15.32:40256) 2024-04-12T05:33:54.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.347+0000 7fd4b659c700 1 -- 172.21.15.32:0/176259468 learned_addr learned my addr 172.21.15.32:0/176259468 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:33:54.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.347+0000 7fd4b6d9d700 1 --2- 172.21.15.32:0/176259468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4b80fde90 0x7fd4b8113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:54.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.347+0000 7fd4b659c700 1 -- 172.21.15.32:0/176259468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4b80fde90 msgr2=0x7fd4b8113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:54.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.347+0000 7fd4b659c700 1 --2- 172.21.15.32:0/176259468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4b80fde90 0x7fd4b8113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:54.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.347+0000 7fd4b659c700 1 -- 172.21.15.32:0/176259468 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a8005040 con 0x7fd4b80fe830 2024-04-12T05:33:54.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.347+0000 7fd4b659c700 1 --2- 172.21.15.32:0/176259468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4b80fe830 0x7fd4b8114030 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd4a000b790 tx=0x7fd4a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:54.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.348+0000 7fd4af7fe700 1 -- 172.21.15.32:0/176259468 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4a000f6f0 con 0x7fd4b80fe830 2024-04-12T05:33:54.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.348+0000 7fd4bd07d700 1 -- 172.21.15.32:0/176259468 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4b810dd50 con 0x7fd4b80fe830 2024-04-12T05:33:54.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.348+0000 7fd4af7fe700 1 -- 172.21.15.32:0/176259468 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd4a000fd00 con 0x7fd4b80fe830 2024-04-12T05:33:54.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.348+0000 7fd4af7fe700 1 -- 172.21.15.32:0/176259468 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4a00103a0 con 0x7fd4b80fe830 2024-04-12T05:33:54.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.348+0000 7fd4bd07d700 1 -- 172.21.15.32:0/176259468 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4b810e290 con 0x7fd4b80fe830 2024-04-12T05:33:54.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.349+0000 7fd4bd07d700 1 -- 172.21.15.32:0/176259468 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4b8065e30 con 0x7fd4b80fe830 2024-04-12T05:33:54.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.349+0000 7fd4af7fe700 1 -- 172.21.15.32:0/176259468 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fd4a000f850 con 0x7fd4b80fe830 2024-04-12T05:33:54.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.350+0000 7fd4af7fe700 1 --2- 172.21.15.32:0/176259468 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fd4a40692a0 0x7fd4a406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:54.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.350+0000 7fd4af7fe700 1 -- 172.21.15.32:0/176259468 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd4a0087880 con 0x7fd4b80fe830 2024-04-12T05:33:54.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.350+0000 7fd4b6d9d700 1 --2- 172.21.15.32:0/176259468 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fd4a40692a0 0x7fd4a406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:54.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.350+0000 7fd4b6d9d700 1 --2- 172.21.15.32:0/176259468 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fd4a40692a0 0x7fd4a406b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd4a800b7e0 tx=0x7fd4a8009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:54.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.353+0000 7fd4af7fe700 1 -- 172.21.15.32:0/176259468 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fd4a0057960 con 0x7fd4b80fe830 2024-04-12T05:33:54.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:54 smithi067 ceph-mon[49010]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:54.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.543+0000 7fd4bd07d700 1 -- 172.21.15.32:0/176259468 --> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd4b81137c0 con 0x7fd4a40692a0 2024-04-12T05:33:54.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.544+0000 7fd4af7fe700 1 -- 172.21.15.32:0/176259468 <== mgr.14208 v2:172.21.15.32:6800/595990262 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28145 (secure 0 0 0) 0x7fd4b81137c0 con 0x7fd4a40692a0 2024-04-12T05:33:54.543 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:33:54.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.546+0000 7fd4bd07d700 1 -- 172.21.15.32:0/176259468 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fd4a40692a0 msgr2=0x7fd4a406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:54.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.546+0000 7fd4bd07d700 1 --2- 172.21.15.32:0/176259468 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fd4a40692a0 0x7fd4a406b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd4a800b7e0 tx=0x7fd4a8009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:54.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.546+0000 7fd4bd07d700 1 -- 172.21.15.32:0/176259468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4b80fe830 msgr2=0x7fd4b8114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:54.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.546+0000 7fd4bd07d700 1 --2- 172.21.15.32:0/176259468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4b80fe830 0x7fd4b8114030 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd4a000b790 tx=0x7fd4a0009250 comp rx=0 tx=0).stop 2024-04-12T05:33:54.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.546+0000 7fd4bd07d700 1 -- 172.21.15.32:0/176259468 shutdown_connections 2024-04-12T05:33:54.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.546+0000 7fd4bd07d700 1 --2- 172.21.15.32:0/176259468 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fd4a40692a0 0x7fd4a406b750 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:54.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.546+0000 7fd4bd07d700 1 --2- 172.21.15.32:0/176259468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4b80fde90 0x7fd4b8113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:54.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.546+0000 7fd4bd07d700 1 --2- 172.21.15.32:0/176259468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4b80fe830 0x7fd4b8114030 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:54.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.546+0000 7fd4bd07d700 1 -- 172.21.15.32:0/176259468 >> 172.21.15.32:0/176259468 conn(0x7fd4b8072bc0 msgr2=0x7fd4b81037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:54.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.546+0000 7fd4bd07d700 1 -- 172.21.15.32:0/176259468 shutdown_connections 2024-04-12T05:33:54.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:54.546+0000 7fd4bd07d700 1 -- 172.21.15.32:0/176259468 wait complete. 2024-04-12T05:33:54.546 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-12T05:33:55.233 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2024-04-12T05:33:53.102382+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1788016,"kb_used_data":3872,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":560682896,"statfs":{"total":575970213888,"available":574139285504,"internally_reserved":0,"allocated":3964928,"data_stored":2605939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002182"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":190,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-12T05:32:59.513736+0000","last_change":"2024-04-12T05:32:31.247806+0000","last_active":"2024-04-12T05:32:59.513736+0000","last_peered":"2024-04-12T05:32:59.513736+0000","last_clean":"2024-04-12T05:32:59.513736+0000","last_became_active":"2024-04-12T05:32:31.247513+0000","last_became_peered":"2024-04-12T05:32:31.247513+0000","last_unstale":"2024-04-12T05:32:59.513736+0000","last_undegraded":"2024-04-12T05:32:59.513736+0000","last_fullsized":"2024-04-12T05:32:59.513736+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T05:31:39.569317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T05:31:39.569317+0000","last_clean_scrub_stamp":"2024-04-12T05:31:39.569317+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T10:30:03.456241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724558,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]}]},{"osd":4,"up_from":34,"seq":146028888083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 12 05:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":4.3789999999999996,"5min":4.3789999999999996,"15min":4.3789999999999996},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":4.3620000000000001,"5min":4.3620000000000001,"15min":4.3620000000000001},"last":0.46100000000000002}]},{"osd":1,"last update":"Fri Apr 12 05:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":5.3079999999999998,"5min":5.3079999999999998,"15min":5.3079999999999998},"last":0.43099999999999999}]},{"osd":2,"last update":"Fri Apr 12 05:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":4.5800000000000001,"5min":4.5800000000000001,"15min":4.5800000000000001},"last":0.53000000000000003}]},{"osd":3,"last update":"Fri Apr 12 05:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.39000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31}]}]},{"osd":3,"up_from":26,"seq":111669149720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298296,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446856,"statfs":{"total":95995035648,"available":95689580544,"internally_reserved":0,"allocated":954368,"data_stored":729238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 12 05:33:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.41099999999999998}]},{"osd":1,"last update":"Fri Apr 12 05:33:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.46400000000000002}]},{"osd":2,"last update":"Fri Apr 12 05:33:05 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.52600000000000002}]},{"osd":4,"last update":"Fri Apr 12 05:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.377},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.438}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 12 05:33:42 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.35799999999999998,"15min":0.35899999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.495,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.38400000000000001,"15min":0.39000000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.34899999999999998}]},{"osd":1,"last update":"Fri Apr 12 05:33:42 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.38800000000000001,"15min":0.39400000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.55400000000000005,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.39500000000000002,"15min":0.40100000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.36499999999999999}]},{"osd":3,"last update":"Fri Apr 12 05:33:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.58399999999999996}]},{"osd":4,"last update":"Fri Apr 12 05:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.63},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.54300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705705,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298292,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446860,"statfs":{"total":95995035648,"available":95689584640,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Apr 12 05:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.33700000000000002,"15min":0.33200000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.371,"5min":0.34599999999999997,"15min":0.34200000000000003},"min":{"1min":0.23799999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.56799999999999995}]},{"osd":2,"last update":"Fri Apr 12 05:33:42 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.37,"15min":0.377},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.443,"5min":0.622,"15min":0.622},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.39700000000000002,"15min":0.40500000000000003},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.438,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.63200000000000001}]},{"osd":3,"last update":"Fri Apr 12 05:33:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.82199999999999995}]},{"osd":4,"last update":"Fri Apr 12 05:33:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.78400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542180,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298292,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446860,"statfs":{"total":95995035648,"available":95689584640,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 12 05:33:12 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.34399999999999997,"15min":0.33800000000000002},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.56000000000000005,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.36499999999999999,"15min":0.35999999999999999},"min":{"1min":0.23899999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.53300000000000003,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.33400000000000002}]},{"osd":2,"last update":"Fri Apr 12 05:33:45 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.36799999999999999,"15min":0.36299999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.317},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.38300000000000001,"15min":0.379},"min":{"1min":0.27900000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.307}]},{"osd":3,"last update":"Fri Apr 12 05:33:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.47199999999999998}]},{"osd":4,"last update":"Fri Apr 12 05:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.51900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-12T05:33:55.234 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph pg dump --format=json 2024-04-12T05:33:55.512 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:33:55.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:55 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:33:55.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:55 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:33:56.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:56 smithi067 ceph-mon[49010]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T05:33:56.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:56 smithi067 ceph-mon[49010]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:56.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:56 smithi032 ceph-mon[39731]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T05:33:56.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:56 smithi032 ceph-mon[39731]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:57.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.290+0000 7ff36f181700 1 -- 172.21.15.32:0/535784194 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3681037b0 msgr2=0x7ff3680fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:57.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.290+0000 7ff36f181700 1 --2- 172.21.15.32:0/535784194 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3681037b0 0x7ff3680fce20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff354004940 tx=0x7ff354009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:57.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.290+0000 7ff36f181700 1 -- 172.21.15.32:0/535784194 shutdown_connections 2024-04-12T05:33:57.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.290+0000 7ff36f181700 1 --2- 172.21.15.32:0/535784194 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3681037b0 0x7ff3680fce20 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:57.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.290+0000 7ff36f181700 1 --2- 172.21.15.32:0/535784194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff368102e10 0x7ff3681031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:57.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.290+0000 7ff36f181700 1 -- 172.21.15.32:0/535784194 >> 172.21.15.32:0/535784194 conn(0x7ff3680f89a0 msgr2=0x7ff3680fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:57.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.290+0000 7ff36f181700 1 -- 172.21.15.32:0/535784194 shutdown_connections 2024-04-12T05:33:57.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.290+0000 7ff36f181700 1 -- 172.21.15.32:0/535784194 wait complete. 2024-04-12T05:33:57.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.291+0000 7ff36f181700 1 Processor -- start 2024-04-12T05:33:57.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.291+0000 7ff36f181700 1 -- start start 2024-04-12T05:33:57.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.291+0000 7ff36f181700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff368102e10 0x7ff3680702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:57.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.291+0000 7ff36f181700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3681037b0 0x7ff368070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:57.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.291+0000 7ff36f181700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff368070e20 con 0x7ff368102e10 2024-04-12T05:33:57.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.291+0000 7ff36f181700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff368070f60 con 0x7ff3681037b0 2024-04-12T05:33:57.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.291+0000 7ff36cf1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff368102e10 0x7ff3680702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:57.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.291+0000 7ff36cf1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff368102e10 0x7ff3680702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40274/0 (socket says 172.21.15.32:40274) 2024-04-12T05:33:57.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.291+0000 7ff36cf1d700 1 -- 172.21.15.32:0/1598277 learned_addr learned my addr 172.21.15.32:0/1598277 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:33:57.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.292+0000 7ff367fff700 1 --2- 172.21.15.32:0/1598277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3681037b0 0x7ff368070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:57.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.292+0000 7ff36cf1d700 1 -- 172.21.15.32:0/1598277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3681037b0 msgr2=0x7ff368070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:57.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.292+0000 7ff36cf1d700 1 --2- 172.21.15.32:0/1598277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3681037b0 0x7ff368070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:57.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.292+0000 7ff36cf1d700 1 -- 172.21.15.32:0/1598277 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff354005040 con 0x7ff368102e10 2024-04-12T05:33:57.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.292+0000 7ff367fff700 1 --2- 172.21.15.32:0/1598277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3681037b0 0x7ff368070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:33:57.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.292+0000 7ff36cf1d700 1 --2- 172.21.15.32:0/1598277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff368102e10 0x7ff3680702c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff35c00b790 tx=0x7ff35c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:57.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.292+0000 7ff3657fa700 1 -- 172.21.15.32:0/1598277 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff35c00f770 con 0x7ff368102e10 2024-04-12T05:33:57.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.292+0000 7ff3657fa700 1 -- 172.21.15.32:0/1598277 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff35c00fd80 con 0x7ff368102e10 2024-04-12T05:33:57.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.292+0000 7ff36f181700 1 -- 172.21.15.32:0/1598277 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff368119df0 con 0x7ff368102e10 2024-04-12T05:33:57.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.292+0000 7ff3657fa700 1 -- 172.21.15.32:0/1598277 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff35c00d310 con 0x7ff368102e10 2024-04-12T05:33:57.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.292+0000 7ff36f181700 1 -- 172.21.15.32:0/1598277 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff36811a2a0 con 0x7ff368102e10 2024-04-12T05:33:57.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.293+0000 7ff36f181700 1 -- 172.21.15.32:0/1598277 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3680fe560 con 0x7ff368102e10 2024-04-12T05:33:57.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.293+0000 7ff3657fa700 1 -- 172.21.15.32:0/1598277 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ff35c010420 con 0x7ff368102e10 2024-04-12T05:33:57.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.294+0000 7ff3657fa700 1 --2- 172.21.15.32:0/1598277 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7ff3580692a0 0x7ff35806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:33:57.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.294+0000 7ff3657fa700 1 -- 172.21.15.32:0/1598277 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff35c086e10 con 0x7ff368102e10 2024-04-12T05:33:57.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.294+0000 7ff367fff700 1 --2- 172.21.15.32:0/1598277 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7ff3580692a0 0x7ff35806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:57.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.294+0000 7ff367fff700 1 --2- 172.21.15.32:0/1598277 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7ff3580692a0 0x7ff35806b750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff354009fd0 tx=0x7ff354009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:57.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.298+0000 7ff3657fa700 1 -- 172.21.15.32:0/1598277 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ff35c053480 con 0x7ff368102e10 2024-04-12T05:33:57.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.490+0000 7ff36f181700 1 -- 172.21.15.32:0/1598277 --> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff3680719b0 con 0x7ff3580692a0 2024-04-12T05:33:57.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.491+0000 7ff3657fa700 1 -- 172.21.15.32:0/1598277 <== mgr.14208 v2:172.21.15.32:6800/595990262 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28115 (secure 0 0 0) 0x7ff3680719b0 con 0x7ff3580692a0 2024-04-12T05:33:57.491 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:33:57.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.493+0000 7ff36f181700 1 -- 172.21.15.32:0/1598277 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7ff3580692a0 msgr2=0x7ff35806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:57.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.493+0000 7ff36f181700 1 --2- 172.21.15.32:0/1598277 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7ff3580692a0 0x7ff35806b750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff354009fd0 tx=0x7ff354009e30 comp rx=0 tx=0).stop 2024-04-12T05:33:57.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.493+0000 7ff36f181700 1 -- 172.21.15.32:0/1598277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff368102e10 msgr2=0x7ff3680702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:57.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.493+0000 7ff36f181700 1 --2- 172.21.15.32:0/1598277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff368102e10 0x7ff3680702c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff35c00b790 tx=0x7ff35c009250 comp rx=0 tx=0).stop 2024-04-12T05:33:57.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.493+0000 7ff36f181700 1 -- 172.21.15.32:0/1598277 shutdown_connections 2024-04-12T05:33:57.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.493+0000 7ff36f181700 1 --2- 172.21.15.32:0/1598277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff368102e10 0x7ff3680702c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:57.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.493+0000 7ff36f181700 1 --2- 172.21.15.32:0/1598277 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7ff3580692a0 0x7ff35806b750 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:57.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.493+0000 7ff36f181700 1 --2- 172.21.15.32:0/1598277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3681037b0 0x7ff368070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:33:57.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.493+0000 7ff36f181700 1 -- 172.21.15.32:0/1598277 >> 172.21.15.32:0/1598277 conn(0x7ff3680f89a0 msgr2=0x7ff3680f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:57.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.493+0000 7ff36f181700 1 -- 172.21.15.32:0/1598277 shutdown_connections 2024-04-12T05:33:57.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:33:57.493+0000 7ff36f181700 1 -- 172.21.15.32:0/1598277 wait complete. 2024-04-12T05:33:57.494 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-12T05:33:58.231 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-04-12T05:33:57.103155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1788016,"kb_used_data":3872,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":560682896,"statfs":{"total":575970213888,"available":574139285504,"internally_reserved":0,"allocated":3964928,"data_stored":2605939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002226"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":190,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-12T05:32:59.513736+0000","last_change":"2024-04-12T05:32:31.247806+0000","last_active":"2024-04-12T05:32:59.513736+0000","last_peered":"2024-04-12T05:32:59.513736+0000","last_clean":"2024-04-12T05:32:59.513736+0000","last_became_active":"2024-04-12T05:32:31.247513+0000","last_became_peered":"2024-04-12T05:32:31.247513+0000","last_unstale":"2024-04-12T05:32:59.513736+0000","last_undegraded":"2024-04-12T05:32:59.513736+0000","last_fullsized":"2024-04-12T05:32:59.513736+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T05:31:39.569317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T05:31:39.569317+0000","last_clean_scrub_stamp":"2024-04-12T05:31:39.569317+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T10:30:03.456241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724559,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]}]},{"osd":4,"up_from":34,"seq":146028888084,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 12 05:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":4.3789999999999996,"5min":4.3789999999999996,"15min":4.3789999999999996},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":4.3620000000000001,"5min":4.3620000000000001,"15min":4.3620000000000001},"last":0.69599999999999995}]},{"osd":1,"last update":"Fri Apr 12 05:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":5.3079999999999998,"5min":5.3079999999999998,"15min":5.3079999999999998},"last":0.64100000000000001}]},{"osd":2,"last update":"Fri Apr 12 05:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.62},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":4.5800000000000001,"5min":4.5800000000000001,"15min":4.5800000000000001},"last":0.66200000000000003}]},{"osd":3,"last update":"Fri Apr 12 05:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.38},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.36199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]}]},{"osd":3,"up_from":26,"seq":111669149721,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298296,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446856,"statfs":{"total":95995035648,"available":95689580544,"internally_reserved":0,"allocated":954368,"data_stored":729238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 12 05:33:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.52700000000000002}]},{"osd":1,"last update":"Fri Apr 12 05:33:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.55100000000000005}]},{"osd":2,"last update":"Fri Apr 12 05:33:05 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.5}]},{"osd":4,"last update":"Fri Apr 12 05:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.625}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378654,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 12 05:33:42 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.35799999999999998,"15min":0.35899999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.495,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.38400000000000001,"15min":0.39000000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.26600000000000001}]},{"osd":1,"last update":"Fri Apr 12 05:33:42 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.38800000000000001,"15min":0.39400000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.55400000000000005,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.39500000000000002,"15min":0.40100000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.28100000000000003}]},{"osd":3,"last update":"Fri Apr 12 05:33:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.622},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.61199999999999999}]},{"osd":4,"last update":"Fri Apr 12 05:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.46400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705706,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298292,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446860,"statfs":{"total":95995035648,"available":95689584640,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Apr 12 05:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.33700000000000002,"15min":0.33200000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.253},{"interface":"front","average":{"1min":0.371,"5min":0.34599999999999997,"15min":0.34200000000000003},"min":{"1min":0.23799999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.314}]},{"osd":2,"last update":"Fri Apr 12 05:33:42 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.37,"15min":0.377},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.443,"5min":0.622,"15min":0.622},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.39700000000000002,"15min":0.40500000000000003},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.438,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.30099999999999999}]},{"osd":3,"last update":"Fri Apr 12 05:33:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.66000000000000003}]},{"osd":4,"last update":"Fri Apr 12 05:33:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.63300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542181,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298292,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446860,"statfs":{"total":95995035648,"available":95689584640,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 12 05:33:12 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.34399999999999997,"15min":0.33800000000000002},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.56000000000000005,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.36499999999999999,"15min":0.35999999999999999},"min":{"1min":0.23899999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.53300000000000003,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.45300000000000001}]},{"osd":2,"last update":"Fri Apr 12 05:33:45 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.36799999999999999,"15min":0.36299999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.38300000000000001,"15min":0.379},"min":{"1min":0.27900000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.40300000000000002}]},{"osd":3,"last update":"Fri Apr 12 05:33:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.71299999999999997}]},{"osd":4,"last update":"Fri Apr 12 05:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.73499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-12T05:33:58.232 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-12T05:33:58.232 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-12T05:33:58.232 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-12T05:33:58.232 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph health --format=json 2024-04-12T05:33:58.514 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:33:58.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:58 smithi067 ceph-mon[49010]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:58.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:58 smithi032 ceph-mon[39731]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:59.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:33:59 smithi067 ceph-mon[49010]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T05:33:59.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:33:59 smithi032 ceph-mon[39731]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T05:34:00.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.337+0000 7f4b90f8e700 1 -- 172.21.15.32:0/852717880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b8c1037a0 msgr2=0x7f4b8c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:00.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.337+0000 7f4b90f8e700 1 --2- 172.21.15.32:0/852717880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b8c1037a0 0x7f4b8c0fce10 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4b74004940 tx=0x7f4b74009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:00.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.337+0000 7f4b90f8e700 1 -- 172.21.15.32:0/852717880 shutdown_connections 2024-04-12T05:34:00.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.337+0000 7f4b90f8e700 1 --2- 172.21.15.32:0/852717880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b8c1037a0 0x7f4b8c0fce10 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:00.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.337+0000 7f4b90f8e700 1 --2- 172.21.15.32:0/852717880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b8c102e00 0x7f4b8c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:00.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.337+0000 7f4b90f8e700 1 -- 172.21.15.32:0/852717880 >> 172.21.15.32:0/852717880 conn(0x7f4b8c0f89b0 msgr2=0x7f4b8c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:00.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.337+0000 7f4b90f8e700 1 -- 172.21.15.32:0/852717880 shutdown_connections 2024-04-12T05:34:00.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.337+0000 7f4b90f8e700 1 -- 172.21.15.32:0/852717880 wait complete. 2024-04-12T05:34:00.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.337+0000 7f4b90f8e700 1 Processor -- start 2024-04-12T05:34:00.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.338+0000 7f4b90f8e700 1 -- start start 2024-04-12T05:34:00.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.338+0000 7f4b90f8e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b8c102e00 0x7f4b8c113890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:00.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.338+0000 7f4b90f8e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b8c1037a0 0x7f4b8c113dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:00.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.338+0000 7f4b90f8e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c114420 con 0x7f4b8c1037a0 2024-04-12T05:34:00.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.338+0000 7f4b90f8e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c10da20 con 0x7f4b8c102e00 2024-04-12T05:34:00.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.338+0000 7f4b89d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b8c1037a0 0x7f4b8c113dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:00.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.338+0000 7f4b89d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b8c1037a0 0x7f4b8c113dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40286/0 (socket says 172.21.15.32:40286) 2024-04-12T05:34:00.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.338+0000 7f4b89d9b700 1 -- 172.21.15.32:0/2511757917 learned_addr learned my addr 172.21.15.32:0/2511757917 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:34:00.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.338+0000 7f4b8a59c700 1 --2- 172.21.15.32:0/2511757917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b8c102e00 0x7f4b8c113890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:00.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.338+0000 7f4b89d9b700 1 -- 172.21.15.32:0/2511757917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b8c102e00 msgr2=0x7f4b8c113890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.338+0000 7f4b89d9b700 1 --2- 172.21.15.32:0/2511757917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b8c102e00 0x7f4b8c113890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.338+0000 7f4b89d9b700 1 -- 172.21.15.32:0/2511757917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b74005040 con 0x7f4b8c1037a0 2024-04-12T05:34:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.338+0000 7f4b89d9b700 1 --2- 172.21.15.32:0/2511757917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b8c1037a0 0x7f4b8c113dd0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4b74009380 tx=0x7f4b74009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.339+0000 7f4b82ffd700 1 -- 172.21.15.32:0/2511757917 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b7400e040 con 0x7f4b8c1037a0 2024-04-12T05:34:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.339+0000 7f4b82ffd700 1 -- 172.21.15.32:0/2511757917 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b74013420 con 0x7f4b8c1037a0 2024-04-12T05:34:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.339+0000 7f4b82ffd700 1 -- 172.21.15.32:0/2511757917 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b7401c630 con 0x7f4b8c1037a0 2024-04-12T05:34:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.339+0000 7f4b90f8e700 1 -- 172.21.15.32:0/2511757917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b8c10dca0 con 0x7f4b8c1037a0 2024-04-12T05:34:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.339+0000 7f4b90f8e700 1 -- 172.21.15.32:0/2511757917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b8c10e260 con 0x7f4b8c1037a0 2024-04-12T05:34:00.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.339+0000 7f4b90f8e700 1 -- 172.21.15.32:0/2511757917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b8c0fe550 con 0x7f4b8c1037a0 2024-04-12T05:34:00.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.340+0000 7f4b82ffd700 1 -- 172.21.15.32:0/2511757917 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f4b740139b0 con 0x7f4b8c1037a0 2024-04-12T05:34:00.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.341+0000 7f4b82ffd700 1 --2- 172.21.15.32:0/2511757917 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4b780692a0 0x7f4b7806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:00.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.341+0000 7f4b82ffd700 1 -- 172.21.15.32:0/2511757917 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4b74087fe0 con 0x7f4b8c1037a0 2024-04-12T05:34:00.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.341+0000 7f4b8a59c700 1 --2- 172.21.15.32:0/2511757917 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4b780692a0 0x7f4b7806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:00.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.341+0000 7f4b8a59c700 1 --2- 172.21.15.32:0/2511757917 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4b780692a0 0x7f4b7806b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c004f40 tx=0x7f4b7c00b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:00.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:00 smithi032 ceph-mon[39731]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:00.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.345+0000 7f4b82ffd700 1 -- 172.21.15.32:0/2511757917 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4b740580e0 con 0x7f4b8c1037a0 2024-04-12T05:34:00.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:00 smithi067 ceph-mon[49010]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:00.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.581+0000 7f4b90f8e700 1 -- 172.21.15.32:0/2511757917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4b8c10de30 con 0x7f4b8c1037a0 2024-04-12T05:34:00.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.581+0000 7f4b82ffd700 1 -- 172.21.15.32:0/2511757917 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f4b8c10de30 con 0x7f4b8c1037a0 2024-04-12T05:34:00.581 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:34:00.581 INFO:teuthology.orchestra.run.smithi032.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-12T05:34:00.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.583+0000 7f4b90f8e700 1 -- 172.21.15.32:0/2511757917 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4b780692a0 msgr2=0x7f4b7806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:00.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.583+0000 7f4b90f8e700 1 --2- 172.21.15.32:0/2511757917 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4b780692a0 0x7f4b7806b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c004f40 tx=0x7f4b7c00b040 comp rx=0 tx=0).stop 2024-04-12T05:34:00.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.583+0000 7f4b90f8e700 1 -- 172.21.15.32:0/2511757917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b8c1037a0 msgr2=0x7f4b8c113dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:00.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.583+0000 7f4b90f8e700 1 --2- 172.21.15.32:0/2511757917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b8c1037a0 0x7f4b8c113dd0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4b74009380 tx=0x7f4b74009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:00.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.584+0000 7f4b90f8e700 1 -- 172.21.15.32:0/2511757917 shutdown_connections 2024-04-12T05:34:00.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.584+0000 7f4b90f8e700 1 --2- 172.21.15.32:0/2511757917 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4b780692a0 0x7f4b7806b750 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:00.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.584+0000 7f4b90f8e700 1 --2- 172.21.15.32:0/2511757917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b8c102e00 0x7f4b8c113890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:00.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.584+0000 7f4b90f8e700 1 --2- 172.21.15.32:0/2511757917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b8c1037a0 0x7f4b8c113dd0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:00.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.584+0000 7f4b90f8e700 1 -- 172.21.15.32:0/2511757917 >> 172.21.15.32:0/2511757917 conn(0x7f4b8c0f89b0 msgr2=0x7f4b8c0f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:00.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.584+0000 7f4b90f8e700 1 -- 172.21.15.32:0/2511757917 shutdown_connections 2024-04-12T05:34:00.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:00.584+0000 7f4b90f8e700 1 -- 172.21.15.32:0/2511757917 wait complete. 2024-04-12T05:34:01.411 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-12T05:34:01.411 INFO:tasks.cephadm:Setup complete, yielding 2024-04-12T05:34:01.411 INFO:teuthology.run_tasks:Running task print... 2024-04-12T05:34:01.420 INFO:teuthology.task.print:**** done end installing quincy cephadm ... 2024-04-12T05:34:01.420 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-12T05:34:01.445 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-12T05:34:01.446 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-04-12T05:34:01.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:01 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/2511757917' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-12T05:34:01.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:01 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/2511757917' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-12T05:34:01.727 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:34:02.296 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:02 smithi032 ceph-mon[39731]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:02.296 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:02 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:34:02.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:02 smithi067 ceph-mon[49010]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:02.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:02 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:34:03.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.471+0000 7f753a9b6700 1 -- 172.21.15.32:0/1365635609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7534070890 msgr2=0x7f7534070d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:03.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.471+0000 7f753a9b6700 1 --2- 172.21.15.32:0/1365635609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7534070890 0x7f7534070d00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7528004940 tx=0x7f7528009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:03.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.471+0000 7f753a9b6700 1 -- 172.21.15.32:0/1365635609 shutdown_connections 2024-04-12T05:34:03.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.471+0000 7f753a9b6700 1 --2- 172.21.15.32:0/1365635609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7534070890 0x7f7534070d00 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:03.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.471+0000 7f753a9b6700 1 --2- 172.21.15.32:0/1365635609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7534071e40 0x7f7534070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:03.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.471+0000 7f753a9b6700 1 -- 172.21.15.32:0/1365635609 >> 172.21.15.32:0/1365635609 conn(0x7f75340f8550 msgr2=0x7f75340fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:03.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.471+0000 7f753a9b6700 1 -- 172.21.15.32:0/1365635609 shutdown_connections 2024-04-12T05:34:03.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.471+0000 7f753a9b6700 1 -- 172.21.15.32:0/1365635609 wait complete. 2024-04-12T05:34:03.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.471+0000 7f753a9b6700 1 Processor -- start 2024-04-12T05:34:03.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.472+0000 7f753a9b6700 1 -- start start 2024-04-12T05:34:03.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.472+0000 7f753a9b6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7534070890 0x7f753410fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:03.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.472+0000 7f753a9b6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7534071e40 0x7f75341100d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:03.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.472+0000 7f753a9b6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75341106d0 con 0x7f7534071e40 2024-04-12T05:34:03.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.472+0000 7f753a9b6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7534110f20 con 0x7f7534070890 2024-04-12T05:34:03.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.472+0000 7f75337fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7534071e40 0x7f75341100d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:03.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.472+0000 7f75337fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7534071e40 0x7f75341100d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46768/0 (socket says 172.21.15.32:46768) 2024-04-12T05:34:03.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.472+0000 7f75337fe700 1 -- 172.21.15.32:0/2159027631 learned_addr learned my addr 172.21.15.32:0/2159027631 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:34:03.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.472+0000 7f7533fff700 1 --2- 172.21.15.32:0/2159027631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7534070890 0x7f753410fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:03.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.472+0000 7f75337fe700 1 -- 172.21.15.32:0/2159027631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7534070890 msgr2=0x7f753410fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:03.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.472+0000 7f75337fe700 1 --2- 172.21.15.32:0/2159027631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7534070890 0x7f753410fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:03.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.472+0000 7f75337fe700 1 -- 172.21.15.32:0/2159027631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7528005040 con 0x7f7534071e40 2024-04-12T05:34:03.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.472+0000 7f75337fe700 1 --2- 172.21.15.32:0/2159027631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7534071e40 0x7f75341100d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f752800b3b0 tx=0x7f7528009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:03.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.473+0000 7f7530ff9700 1 -- 172.21.15.32:0/2159027631 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f752800e040 con 0x7f7534071e40 2024-04-12T05:34:03.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.473+0000 7f7530ff9700 1 -- 172.21.15.32:0/2159027631 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7528013420 con 0x7f7534071e40 2024-04-12T05:34:03.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.473+0000 7f753a9b6700 1 -- 172.21.15.32:0/2159027631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75341111a0 con 0x7f7534071e40 2024-04-12T05:34:03.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.473+0000 7f7530ff9700 1 -- 172.21.15.32:0/2159027631 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f752801c630 con 0x7f7534071e40 2024-04-12T05:34:03.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.473+0000 7f753a9b6700 1 -- 172.21.15.32:0/2159027631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75341116e0 con 0x7f7534071e40 2024-04-12T05:34:03.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.474+0000 7f753a9b6700 1 -- 172.21.15.32:0/2159027631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f753405f5f0 con 0x7f7534071e40 2024-04-12T05:34:03.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.474+0000 7f7530ff9700 1 -- 172.21.15.32:0/2159027631 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f75280139e0 con 0x7f7534071e40 2024-04-12T05:34:03.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.474+0000 7f7530ff9700 1 --2- 172.21.15.32:0/2159027631 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f751c0692a0 0x7f751c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:03.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.475+0000 7f7530ff9700 1 -- 172.21.15.32:0/2159027631 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7528087b10 con 0x7f7534071e40 2024-04-12T05:34:03.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.475+0000 7f7533fff700 1 --2- 172.21.15.32:0/2159027631 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f751c0692a0 0x7f751c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:03.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.475+0000 7f7533fff700 1 --2- 172.21.15.32:0/2159027631 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f751c0692a0 0x7f751c06b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7534071b80 tx=0x7f752400b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:03.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.479+0000 7f7530ff9700 1 -- 172.21.15.32:0/2159027631 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f7528057c10 con 0x7f7534071e40 2024-04-12T05:34:03.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:03 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:34:03.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:03 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:34:03.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:03 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:03.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:03 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:34:03.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:03 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:34:03.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:03 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:03.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.676+0000 7f753a9b6700 1 -- 172.21.15.32:0/2159027631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f753404bed0 con 0x7f7534071e40 2024-04-12T05:34:03.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.680+0000 7f7530ff9700 1 -- 172.21.15.32:0/2159027631 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f75280577a0 con 0x7f7534071e40 2024-04-12T05:34:03.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.682+0000 7f753a9b6700 1 -- 172.21.15.32:0/2159027631 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f751c0692a0 msgr2=0x7f751c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:03.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.682+0000 7f753a9b6700 1 --2- 172.21.15.32:0/2159027631 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f751c0692a0 0x7f751c06b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7534071b80 tx=0x7f752400b040 comp rx=0 tx=0).stop 2024-04-12T05:34:03.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.682+0000 7f753a9b6700 1 -- 172.21.15.32:0/2159027631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7534071e40 msgr2=0x7f75341100d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:03.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.682+0000 7f753a9b6700 1 --2- 172.21.15.32:0/2159027631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7534071e40 0x7f75341100d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f752800b3b0 tx=0x7f7528009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:03.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.683+0000 7f753a9b6700 1 -- 172.21.15.32:0/2159027631 shutdown_connections 2024-04-12T05:34:03.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.683+0000 7f753a9b6700 1 --2- 172.21.15.32:0/2159027631 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f751c0692a0 0x7f751c06b750 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:03.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.683+0000 7f753a9b6700 1 --2- 172.21.15.32:0/2159027631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7534070890 0x7f753410fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:03.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.683+0000 7f753a9b6700 1 --2- 172.21.15.32:0/2159027631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7534071e40 0x7f75341100d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:03.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.683+0000 7f753a9b6700 1 -- 172.21.15.32:0/2159027631 >> 172.21.15.32:0/2159027631 conn(0x7f75340f8550 msgr2=0x7f75340f9d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:03.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.683+0000 7f753a9b6700 1 -- 172.21.15.32:0/2159027631 shutdown_connections 2024-04-12T05:34:03.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:03.683+0000 7f753a9b6700 1 -- 172.21.15.32:0/2159027631 wait complete. 2024-04-12T05:34:04.312 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:04 smithi032 ceph-mon[39731]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:04.312 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:04 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/2159027631' entity='client.admin' 2024-04-12T05:34:04.312 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:04 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:34:04.312 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:04 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:34:04.312 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:04 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:34:04.312 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:04 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:04.313 INFO:teuthology.run_tasks:Running task print... 2024-04-12T05:34:04.322 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-04-12T05:34:04.323 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-12T05:34:04.332 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-12T05:34:04.332 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- bash -c 'ceph orch status' 2024-04-12T05:34:04.584 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:04 smithi067 ceph-mon[49010]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:04.584 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:04 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/2159027631' entity='client.admin' 2024-04-12T05:34:04.585 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:04 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:34:04.585 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:04 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:34:04.585 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:04 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:34:04.585 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:04 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:04.613 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:34:06.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:06 smithi032 ceph-mon[39731]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:06.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.326+0000 7f9666318700 1 -- 172.21.15.32:0/2249009420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96600fce80 msgr2=0x7f96600fd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:06.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.326+0000 7f9666318700 1 --2- 172.21.15.32:0/2249009420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96600fce80 0x7f96600fd290 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9650004940 tx=0x7f9650009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:06.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.326+0000 7f9666318700 1 -- 172.21.15.32:0/2249009420 shutdown_connections 2024-04-12T05:34:06.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.326+0000 7f9666318700 1 --2- 172.21.15.32:0/2249009420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96600fdff0 0x7f96600fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:06.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.326+0000 7f9666318700 1 --2- 172.21.15.32:0/2249009420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96600fce80 0x7f96600fd290 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:06.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.326+0000 7f9666318700 1 -- 172.21.15.32:0/2249009420 >> 172.21.15.32:0/2249009420 conn(0x7f96600f8580 msgr2=0x7f96600fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:06.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.326+0000 7f9666318700 1 -- 172.21.15.32:0/2249009420 shutdown_connections 2024-04-12T05:34:06.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.327+0000 7f9666318700 1 -- 172.21.15.32:0/2249009420 wait complete. 2024-04-12T05:34:06.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.327+0000 7f9666318700 1 Processor -- start 2024-04-12T05:34:06.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.327+0000 7f9666318700 1 -- start start 2024-04-12T05:34:06.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.327+0000 7f9666318700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96600fce80 0x7f966010d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:06.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.327+0000 7f9666318700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96600fdff0 0x7f966010deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:06.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.327+0000 7f9666318700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966010e540 con 0x7f96600fdff0 2024-04-12T05:34:06.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.327+0000 7f9666318700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966010ed30 con 0x7f96600fce80 2024-04-12T05:34:06.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.327+0000 7f965f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96600fdff0 0x7f966010deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:06.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.327+0000 7f965f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96600fdff0 0x7f966010deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46792/0 (socket says 172.21.15.32:46792) 2024-04-12T05:34:06.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.327+0000 7f965f7fe700 1 -- 172.21.15.32:0/4048292030 learned_addr learned my addr 172.21.15.32:0/4048292030 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:34:06.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.328+0000 7f965ffff700 1 --2- 172.21.15.32:0/4048292030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96600fce80 0x7f966010d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:06.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.328+0000 7f965f7fe700 1 -- 172.21.15.32:0/4048292030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96600fce80 msgr2=0x7f966010d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:06.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.328+0000 7f965f7fe700 1 --2- 172.21.15.32:0/4048292030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96600fce80 0x7f966010d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.328+0000 7f965f7fe700 1 -- 172.21.15.32:0/4048292030 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9650005040 con 0x7f96600fdff0 2024-04-12T05:34:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.328+0000 7f965f7fe700 1 --2- 172.21.15.32:0/4048292030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96600fdff0 0x7f966010deb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f965400b790 tx=0x7f9654009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.328+0000 7f965cff9700 1 -- 172.21.15.32:0/4048292030 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f965400f6f0 con 0x7f96600fdff0 2024-04-12T05:34:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.328+0000 7f9666318700 1 -- 172.21.15.32:0/4048292030 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f966010f010 con 0x7f96600fdff0 2024-04-12T05:34:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.328+0000 7f965cff9700 1 -- 172.21.15.32:0/4048292030 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f965400fd00 con 0x7f96600fdff0 2024-04-12T05:34:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.328+0000 7f965cff9700 1 -- 172.21.15.32:0/4048292030 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96540103a0 con 0x7f96600fdff0 2024-04-12T05:34:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.328+0000 7f9666318700 1 -- 172.21.15.32:0/4048292030 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f966010f5d0 con 0x7f96600fdff0 2024-04-12T05:34:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.329+0000 7f9666318700 1 -- 172.21.15.32:0/4048292030 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f966005f5f0 con 0x7f96600fdff0 2024-04-12T05:34:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.330+0000 7f965cff9700 1 -- 172.21.15.32:0/4048292030 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f965400f850 con 0x7f96600fdff0 2024-04-12T05:34:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.330+0000 7f965cff9700 1 --2- 172.21.15.32:0/4048292030 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f96480692b0 0x7f964806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.330+0000 7f965cff9700 1 -- 172.21.15.32:0/4048292030 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9654086c70 con 0x7f96600fdff0 2024-04-12T05:34:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.330+0000 7f965ffff700 1 --2- 172.21.15.32:0/4048292030 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f96480692b0 0x7f964806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.330+0000 7f965ffff700 1 --2- 172.21.15.32:0/4048292030 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f96480692b0 0x7f964806b760 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9650004f90 tx=0x7f9650009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:06.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.334+0000 7f965cff9700 1 -- 172.21.15.32:0/4048292030 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9654056d70 con 0x7f96600fdff0 2024-04-12T05:34:06.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.542+0000 7f9666318700 1 -- 172.21.15.32:0/4048292030 --> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f966010f8c0 con 0x7f96480692b0 2024-04-12T05:34:06.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.543+0000 7f965cff9700 1 -- 172.21.15.32:0/4048292030 <== mgr.14208 v2:172.21.15.32:6800/595990262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f966010f8c0 con 0x7f96480692b0 2024-04-12T05:34:06.542 INFO:teuthology.orchestra.run.smithi032.stdout:Backend: cephadm 2024-04-12T05:34:06.542 INFO:teuthology.orchestra.run.smithi032.stdout:Available: Yes 2024-04-12T05:34:06.542 INFO:teuthology.orchestra.run.smithi032.stdout:Paused: No 2024-04-12T05:34:06.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.544+0000 7f9666318700 1 -- 172.21.15.32:0/4048292030 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f96480692b0 msgr2=0x7f964806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:06.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.544+0000 7f9666318700 1 --2- 172.21.15.32:0/4048292030 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f96480692b0 0x7f964806b760 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9650004f90 tx=0x7f9650009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:06.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.545+0000 7f9666318700 1 -- 172.21.15.32:0/4048292030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96600fdff0 msgr2=0x7f966010deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:06.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.545+0000 7f9666318700 1 --2- 172.21.15.32:0/4048292030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96600fdff0 0x7f966010deb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f965400b790 tx=0x7f9654009250 comp rx=0 tx=0).stop 2024-04-12T05:34:06.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.545+0000 7f9666318700 1 -- 172.21.15.32:0/4048292030 shutdown_connections 2024-04-12T05:34:06.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.545+0000 7f9666318700 1 --2- 172.21.15.32:0/4048292030 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f96480692b0 0x7f964806b760 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:06.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.545+0000 7f9666318700 1 --2- 172.21.15.32:0/4048292030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96600fce80 0x7f966010d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:06.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.545+0000 7f9666318700 1 --2- 172.21.15.32:0/4048292030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96600fdff0 0x7f966010deb0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:06.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.545+0000 7f9666318700 1 -- 172.21.15.32:0/4048292030 >> 172.21.15.32:0/4048292030 conn(0x7f96600f8580 msgr2=0x7f96600f9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:06.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.545+0000 7f9666318700 1 -- 172.21.15.32:0/4048292030 shutdown_connections 2024-04-12T05:34:06.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:06.545+0000 7f9666318700 1 -- 172.21.15.32:0/4048292030 wait complete. 2024-04-12T05:34:06.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:06 smithi067 ceph-mon[49010]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:07.188 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-12T05:34:07.471 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:34:08.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:08 smithi032 ceph-mon[39731]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:34:08.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:08 smithi032 ceph-mon[39731]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:08.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:08 smithi067 ceph-mon[49010]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:34:08.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:08 smithi067 ceph-mon[49010]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:09.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.257+0000 7f3759025700 1 -- 172.21.15.32:0/205310056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3754066af0 msgr2=0x7f37540ffa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:09.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.257+0000 7f3759025700 1 --2- 172.21.15.32:0/205310056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3754066af0 0x7f37540ffa50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3744004940 tx=0x7f3744009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:09.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.257+0000 7f3759025700 1 -- 172.21.15.32:0/205310056 shutdown_connections 2024-04-12T05:34:09.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.257+0000 7f3759025700 1 --2- 172.21.15.32:0/205310056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37540fff90 0x7f3754102370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:09.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.257+0000 7f3759025700 1 --2- 172.21.15.32:0/205310056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3754066af0 0x7f37540ffa50 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:09.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.257+0000 7f3759025700 1 -- 172.21.15.32:0/205310056 >> 172.21.15.32:0/205310056 conn(0x7f37540f7480 msgr2=0x7f37540f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:09.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.257+0000 7f3759025700 1 -- 172.21.15.32:0/205310056 shutdown_connections 2024-04-12T05:34:09.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.257+0000 7f3759025700 1 -- 172.21.15.32:0/205310056 wait complete. 2024-04-12T05:34:09.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.258+0000 7f3759025700 1 Processor -- start 2024-04-12T05:34:09.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.258+0000 7f3759025700 1 -- start start 2024-04-12T05:34:09.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.258+0000 7f3759025700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3754066af0 0x7f375410d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:09.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.258+0000 7f3759025700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37540fff90 0x7f375410de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:09.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.258+0000 7f3759025700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375410e430 con 0x7f37540fff90 2024-04-12T05:34:09.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.258+0000 7f3759025700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375410ec80 con 0x7f3754066af0 2024-04-12T05:34:09.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.258+0000 7f375259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37540fff90 0x7f375410de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:09.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.258+0000 7f375259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37540fff90 0x7f375410de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46802/0 (socket says 172.21.15.32:46802) 2024-04-12T05:34:09.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.258+0000 7f375259c700 1 -- 172.21.15.32:0/3226700857 learned_addr learned my addr 172.21.15.32:0/3226700857 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:34:09.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.258+0000 7f3752d9d700 1 --2- 172.21.15.32:0/3226700857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3754066af0 0x7f375410d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:09.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.259+0000 7f375259c700 1 -- 172.21.15.32:0/3226700857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3754066af0 msgr2=0x7f375410d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:09.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.259+0000 7f375259c700 1 --2- 172.21.15.32:0/3226700857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3754066af0 0x7f375410d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:09.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.259+0000 7f375259c700 1 -- 172.21.15.32:0/3226700857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3744005040 con 0x7f37540fff90 2024-04-12T05:34:09.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.259+0000 7f375259c700 1 --2- 172.21.15.32:0/3226700857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37540fff90 0x7f375410de30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f373c00eb40 tx=0x7f373c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:09.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.259+0000 7f374b7fe700 1 -- 172.21.15.32:0/3226700857 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f373c007cc0 con 0x7f37540fff90 2024-04-12T05:34:09.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.259+0000 7f374b7fe700 1 -- 172.21.15.32:0/3226700857 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f373c00f040 con 0x7f37540fff90 2024-04-12T05:34:09.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.259+0000 7f3759025700 1 -- 172.21.15.32:0/3226700857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f375410ef00 con 0x7f37540fff90 2024-04-12T05:34:09.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.259+0000 7f374b7fe700 1 -- 172.21.15.32:0/3226700857 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f373c00c990 con 0x7f37540fff90 2024-04-12T05:34:09.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.259+0000 7f3759025700 1 -- 172.21.15.32:0/3226700857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f375410f440 con 0x7f37540fff90 2024-04-12T05:34:09.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.260+0000 7f3759025700 1 -- 172.21.15.32:0/3226700857 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f375405f5f0 con 0x7f37540fff90 2024-04-12T05:34:09.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.261+0000 7f374b7fe700 1 -- 172.21.15.32:0/3226700857 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f373c00cc00 con 0x7f37540fff90 2024-04-12T05:34:09.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.261+0000 7f374b7fe700 1 --2- 172.21.15.32:0/3226700857 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f37400692b0 0x7f374006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:09.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.261+0000 7f374b7fe700 1 -- 172.21.15.32:0/3226700857 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f373c086a10 con 0x7f37540fff90 2024-04-12T05:34:09.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.261+0000 7f3752d9d700 1 --2- 172.21.15.32:0/3226700857 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f37400692b0 0x7f374006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:09.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.261+0000 7f3752d9d700 1 --2- 172.21.15.32:0/3226700857 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f37400692b0 0x7f374006b760 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f374400b0d0 tx=0x7f3744009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:09.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.265+0000 7f374b7fe700 1 -- 172.21.15.32:0/3226700857 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f373c04e190 con 0x7f37540fff90 2024-04-12T05:34:09.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.469+0000 7f3759025700 1 -- 172.21.15.32:0/3226700857 --> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f375410f770 con 0x7f37400692b0 2024-04-12T05:34:09.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.478+0000 7f374b7fe700 1 -- 172.21.15.32:0/3226700857 <== mgr.14208 v2:172.21.15.32:6800/595990262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2718 (secure 0 0 0) 0x7f375410f770 con 0x7f37400692b0 2024-04-12T05:34:09.477 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:34:09.477 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (3m) 2m ago 5m 27.0M - 0.25.0 c8568f914cd2 a5c616791a7e 2024-04-12T05:34:09.477 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (5m) 2m ago 5m 6656k - 17.2.7-827-gd3b5060a 21de9b7369ad a9cfc00c311b 2024-04-12T05:34:09.477 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (4m) 71s ago 4m 6651k - 17.2.7-827-gd3b5060a 21de9b7369ad 1f2eea4542d9 2024-04-12T05:34:09.477 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (3m) 2m ago 5m 78.4M - 9.4.7 954c08fa6188 8e5a7ebd77b5 2024-04-12T05:34:09.477 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:9283 running (6m) 2m ago 6m 475M - 17.2.7-827-gd3b5060a 21de9b7369ad fdf859c0313c 2024-04-12T05:34:09.477 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283 running (4m) 71s ago 4m 434M - 17.2.7-827-gd3b5060a 21de9b7369ad 117ab3f1dbe0 2024-04-12T05:34:09.477 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (6m) 2m ago 7m 42.5M 2048M 17.2.7-827-gd3b5060a 21de9b7369ad 75aad95b984d 2024-04-12T05:34:09.477 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (4m) 71s ago 4m 37.9M 2048M 17.2.7-827-gd3b5060a 21de9b7369ad 157be8a8c6d8 2024-04-12T05:34:09.478 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (5m) 2m ago 5m 14.9M - 1.5.0 0da6a335fe13 ca3585dd7b65 2024-04-12T05:34:09.478 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (4m) 71s ago 4m 15.9M - 1.5.0 0da6a335fe13 c324d9436cca 2024-04-12T05:34:09.478 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (3m) 2m ago 3m 40.4M 4300M 17.2.7-827-gd3b5060a 21de9b7369ad 97446e42ceba 2024-04-12T05:34:09.478 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (3m) 2m ago 3m 39.4M 4300M 17.2.7-827-gd3b5060a 21de9b7369ad bf9f9a2db2d5 2024-04-12T05:34:09.478 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (2m) 2m ago 2m 38.6M 4300M 17.2.7-827-gd3b5060a 21de9b7369ad f055cf1ca665 2024-04-12T05:34:09.478 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (2m) 71s ago 2m 39.5M 5323M 17.2.7-827-gd3b5060a 21de9b7369ad 61b851235351 2024-04-12T05:34:09.478 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (104s) 71s ago 104s 37.3M 5323M 17.2.7-827-gd3b5060a 21de9b7369ad 2c84d9047e9c 2024-04-12T05:34:09.478 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (77s) 71s ago 76s 13.3M 5323M 17.2.7-827-gd3b5060a 21de9b7369ad e54839bb6055 2024-04-12T05:34:09.478 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (3m) 2m ago 4m 39.4M - 2.43.0 a07b618ecd1d 996307aec3d2 2024-04-12T05:34:09.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.480+0000 7f3759025700 1 -- 172.21.15.32:0/3226700857 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f37400692b0 msgr2=0x7f374006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:09.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.480+0000 7f3759025700 1 --2- 172.21.15.32:0/3226700857 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f37400692b0 0x7f374006b760 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f374400b0d0 tx=0x7f3744009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:09.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.480+0000 7f3759025700 1 -- 172.21.15.32:0/3226700857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37540fff90 msgr2=0x7f375410de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:09.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.480+0000 7f3759025700 1 --2- 172.21.15.32:0/3226700857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37540fff90 0x7f375410de30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f373c00eb40 tx=0x7f373c00b040 comp rx=0 tx=0).stop 2024-04-12T05:34:09.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.480+0000 7f3759025700 1 -- 172.21.15.32:0/3226700857 shutdown_connections 2024-04-12T05:34:09.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.480+0000 7f3759025700 1 --2- 172.21.15.32:0/3226700857 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f37400692b0 0x7f374006b760 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:09.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.480+0000 7f3759025700 1 --2- 172.21.15.32:0/3226700857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3754066af0 0x7f375410d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:09.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.480+0000 7f3759025700 1 --2- 172.21.15.32:0/3226700857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37540fff90 0x7f375410de30 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:09.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.480+0000 7f3759025700 1 -- 172.21.15.32:0/3226700857 >> 172.21.15.32:0/3226700857 conn(0x7f37540f7480 msgr2=0x7f37540f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:09.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.480+0000 7f3759025700 1 -- 172.21.15.32:0/3226700857 shutdown_connections 2024-04-12T05:34:09.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:09.480+0000 7f3759025700 1 -- 172.21.15.32:0/3226700857 wait complete. 2024-04-12T05:34:10.235 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-12T05:34:10.517 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:34:10.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:10 smithi032 ceph-mon[39731]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:10.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:10 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:34:10.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:10 smithi067 ceph-mon[49010]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:10.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:10 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:34:11.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:11 smithi032 ceph-mon[39731]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:34:11.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:11 smithi067 ceph-mon[49010]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:34:12.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.340+0000 7f75c6537700 1 -- 172.21.15.32:0/1567996056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75c00fe400 msgr2=0x7f75c00fe810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:12.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.340+0000 7f75c6537700 1 --2- 172.21.15.32:0/1567996056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75c00fe400 0x7f75c00fe810 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f75b0004940 tx=0x7f75b0009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:12.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.340+0000 7f75c6537700 1 -- 172.21.15.32:0/1567996056 shutdown_connections 2024-04-12T05:34:12.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.340+0000 7f75c6537700 1 --2- 172.21.15.32:0/1567996056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c00fed50 0x7f75c00fae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:12.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.340+0000 7f75c6537700 1 --2- 172.21.15.32:0/1567996056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75c00fe400 0x7f75c00fe810 secure :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f75b0004940 tx=0x7f75b0009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:12.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.340+0000 7f75c6537700 1 -- 172.21.15.32:0/1567996056 >> 172.21.15.32:0/1567996056 conn(0x7f75c00f6ba0 msgr2=0x7f75c00f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:12.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.341+0000 7f75c6537700 1 -- 172.21.15.32:0/1567996056 shutdown_connections 2024-04-12T05:34:12.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.341+0000 7f75c6537700 1 -- 172.21.15.32:0/1567996056 wait complete. 2024-04-12T05:34:12.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.341+0000 7f75c6537700 1 Processor -- start 2024-04-12T05:34:12.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.342+0000 7f75c6537700 1 -- start start 2024-04-12T05:34:12.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.342+0000 7f75c6537700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75c00fed50 0x7f75c00fdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:12.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.342+0000 7f75c6537700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c00fc0c0 0x7f75c00fc530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:12.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.342+0000 7f75c6537700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c00fca70 con 0x7f75c00fed50 2024-04-12T05:34:12.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.342+0000 7f75c6537700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c00fcbb0 con 0x7f75c00fc0c0 2024-04-12T05:34:12.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.342+0000 7f75bffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75c00fed50 0x7f75c00fdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:12.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.342+0000 7f75bffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75c00fed50 0x7f75c00fdbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48728/0 (socket says 172.21.15.32:48728) 2024-04-12T05:34:12.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.342+0000 7f75bffff700 1 -- 172.21.15.32:0/727601920 learned_addr learned my addr 172.21.15.32:0/727601920 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:34:12.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.342+0000 7f75bf7fe700 1 --2- 172.21.15.32:0/727601920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c00fc0c0 0x7f75c00fc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:12.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.342+0000 7f75bffff700 1 -- 172.21.15.32:0/727601920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c00fc0c0 msgr2=0x7f75c00fc530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:12.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.342+0000 7f75bffff700 1 --2- 172.21.15.32:0/727601920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c00fc0c0 0x7f75c00fc530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:12.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.342+0000 7f75bffff700 1 -- 172.21.15.32:0/727601920 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b0005040 con 0x7f75c00fed50 2024-04-12T05:34:12.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.342+0000 7f75bffff700 1 --2- 172.21.15.32:0/727601920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75c00fed50 0x7f75c00fdbf0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f75b0000c00 tx=0x7f75b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:12.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.342+0000 7f75bcff9700 1 -- 172.21.15.32:0/727601920 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75b000e040 con 0x7f75c00fed50 2024-04-12T05:34:12.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.343+0000 7f75bcff9700 1 -- 172.21.15.32:0/727601920 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f75b0004290 con 0x7f75c00fed50 2024-04-12T05:34:12.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.343+0000 7f75c6537700 1 -- 172.21.15.32:0/727601920 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c00fce30 con 0x7f75c00fed50 2024-04-12T05:34:12.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.343+0000 7f75bcff9700 1 -- 172.21.15.32:0/727601920 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75b001c3f0 con 0x7f75c00fed50 2024-04-12T05:34:12.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.343+0000 7f75c6537700 1 -- 172.21.15.32:0/727601920 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c0115ea0 con 0x7f75c00fed50 2024-04-12T05:34:12.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.344+0000 7f75c6537700 1 -- 172.21.15.32:0/727601920 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c005f5f0 con 0x7f75c00fed50 2024-04-12T05:34:12.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.345+0000 7f75bcff9700 1 -- 172.21.15.32:0/727601920 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f75b0013790 con 0x7f75c00fed50 2024-04-12T05:34:12.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.345+0000 7f75bcff9700 1 --2- 172.21.15.32:0/727601920 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f75a80692a0 0x7f75a806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:12.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.345+0000 7f75bcff9700 1 -- 172.21.15.32:0/727601920 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f75b0088ab0 con 0x7f75c00fed50 2024-04-12T05:34:12.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.345+0000 7f75bf7fe700 1 --2- 172.21.15.32:0/727601920 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f75a80692a0 0x7f75a806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:12.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.346+0000 7f75bf7fe700 1 --2- 172.21.15.32:0/727601920 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f75a80692a0 0x7f75a806b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f75b4007fd0 tx=0x7f75b4009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:12.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.349+0000 7f75bcff9700 1 -- 172.21.15.32:0/727601920 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f75b00551c0 con 0x7f75c00fed50 2024-04-12T05:34:12.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.550+0000 7f75c6537700 1 -- 172.21.15.32:0/727601920 --> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f75c0102b00 con 0x7f75a80692a0 2024-04-12T05:34:12.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.554+0000 7f75bcff9700 1 -- 172.21.15.32:0/727601920 <== mgr.14208 v2:172.21.15.32:6800/595990262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1161 (secure 0 0 0) 0x7f75c0102b00 con 0x7f75a80692a0 2024-04-12T05:34:12.553 INFO:teuthology.orchestra.run.smithi032.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-12T05:34:12.553 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-04-12T05:34:12.553 INFO:teuthology.orchestra.run.smithi032.stdout:crash 2/2 2m ago 6m * 2024-04-12T05:34:12.553 INFO:teuthology.orchestra.run.smithi032.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-04-12T05:34:12.553 INFO:teuthology.orchestra.run.smithi032.stdout:mgr 2/2 2m ago 6m count:2 2024-04-12T05:34:12.553 INFO:teuthology.orchestra.run.smithi032.stdout:mon 2/2 2m ago 5m smithi032:172.21.15.32=smithi032;smithi067:172.21.15.67=smithi067;count:2 2024-04-12T05:34:12.554 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-04-12T05:34:12.554 INFO:teuthology.orchestra.run.smithi032.stdout:osd 6 2m ago - 2024-04-12T05:34:12.554 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-04-12T05:34:12.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.556+0000 7f75c6537700 1 -- 172.21.15.32:0/727601920 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f75a80692a0 msgr2=0x7f75a806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:12.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.556+0000 7f75c6537700 1 --2- 172.21.15.32:0/727601920 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f75a80692a0 0x7f75a806b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f75b4007fd0 tx=0x7f75b4009040 comp rx=0 tx=0).stop 2024-04-12T05:34:12.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.556+0000 7f75c6537700 1 -- 172.21.15.32:0/727601920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75c00fed50 msgr2=0x7f75c00fdbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:12.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.556+0000 7f75c6537700 1 --2- 172.21.15.32:0/727601920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75c00fed50 0x7f75c00fdbf0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f75b0000c00 tx=0x7f75b0009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:12.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.556+0000 7f75c6537700 1 -- 172.21.15.32:0/727601920 shutdown_connections 2024-04-12T05:34:12.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.556+0000 7f75c6537700 1 --2- 172.21.15.32:0/727601920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75c00fed50 0x7f75c00fdbf0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:12.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.556+0000 7f75c6537700 1 --2- 172.21.15.32:0/727601920 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f75a80692a0 0x7f75a806b750 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:12.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.556+0000 7f75c6537700 1 --2- 172.21.15.32:0/727601920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c00fc0c0 0x7f75c00fc530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:12.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.556+0000 7f75c6537700 1 -- 172.21.15.32:0/727601920 >> 172.21.15.32:0/727601920 conn(0x7f75c00f6ba0 msgr2=0x7f75c01013e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:12.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.556+0000 7f75c6537700 1 -- 172.21.15.32:0/727601920 shutdown_connections 2024-04-12T05:34:12.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:12.556+0000 7f75c6537700 1 -- 172.21.15.32:0/727601920 wait complete. 2024-04-12T05:34:12.578 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:12 smithi032 ceph-mon[39731]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:12.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:12 smithi067 ceph-mon[49010]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:13.126 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-12T05:34:13.402 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:34:14.583 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:14 smithi067 ceph-mon[49010]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:34:14.583 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:14 smithi067 ceph-mon[49010]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:14.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:14 smithi032 ceph-mon[39731]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:34:14.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:14 smithi032 ceph-mon[39731]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:15.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.257+0000 7fa943c1d700 1 -- 172.21.15.32:0/3813533999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa93c0fff00 msgr2=0x7fa93c102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:15.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.257+0000 7fa943c1d700 1 --2- 172.21.15.32:0/3813533999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa93c0fff00 0x7fa93c102320 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa92c004940 tx=0x7fa92c009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:15.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.257+0000 7fa943c1d700 1 -- 172.21.15.32:0/3813533999 shutdown_connections 2024-04-12T05:34:15.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.257+0000 7fa943c1d700 1 --2- 172.21.15.32:0/3813533999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa93c0fff00 0x7fa93c102320 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:15.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.257+0000 7fa943c1d700 1 --2- 172.21.15.32:0/3813533999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa93c0662e0 0x7fa93c0ff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:15.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.257+0000 7fa943c1d700 1 -- 172.21.15.32:0/3813533999 >> 172.21.15.32:0/3813533999 conn(0x7fa93c0f7430 msgr2=0x7fa93c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:15.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.257+0000 7fa943c1d700 1 -- 172.21.15.32:0/3813533999 shutdown_connections 2024-04-12T05:34:15.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.257+0000 7fa943c1d700 1 -- 172.21.15.32:0/3813533999 wait complete. 2024-04-12T05:34:15.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.258+0000 7fa943c1d700 1 Processor -- start 2024-04-12T05:34:15.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.258+0000 7fa943c1d700 1 -- start start 2024-04-12T05:34:15.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.258+0000 7fa943c1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa93c0662e0 0x7fa93c10d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:15.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.258+0000 7fa943c1d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa93c0fff00 0x7fa93c10de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:15.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.258+0000 7fa943c1d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93c10e460 con 0x7fa93c0662e0 2024-04-12T05:34:15.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.258+0000 7fa943c1d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93c10ec60 con 0x7fa93c0fff00 2024-04-12T05:34:15.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.258+0000 7fa9419b9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa93c0662e0 0x7fa93c10d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:15.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.258+0000 7fa9419b9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa93c0662e0 0x7fa93c10d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48750/0 (socket says 172.21.15.32:48750) 2024-04-12T05:34:15.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.258+0000 7fa9419b9700 1 -- 172.21.15.32:0/2037628390 learned_addr learned my addr 172.21.15.32:0/2037628390 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:34:15.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.258+0000 7fa9411b8700 1 --2- 172.21.15.32:0/2037628390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa93c0fff00 0x7fa93c10de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:15.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.258+0000 7fa9419b9700 1 -- 172.21.15.32:0/2037628390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa93c0fff00 msgr2=0x7fa93c10de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:15.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.258+0000 7fa9419b9700 1 --2- 172.21.15.32:0/2037628390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa93c0fff00 0x7fa93c10de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:15.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.258+0000 7fa9419b9700 1 -- 172.21.15.32:0/2037628390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa92c005040 con 0x7fa93c0662e0 2024-04-12T05:34:15.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.259+0000 7fa9419b9700 1 --2- 172.21.15.32:0/2037628390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa93c0662e0 0x7fa93c10d920 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa93800eb40 tx=0x7fa93800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:15.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.259+0000 7fa9327fc700 1 -- 172.21.15.32:0/2037628390 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa938007cc0 con 0x7fa93c0662e0 2024-04-12T05:34:15.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.259+0000 7fa943c1d700 1 -- 172.21.15.32:0/2037628390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa93c10eee0 con 0x7fa93c0662e0 2024-04-12T05:34:15.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.259+0000 7fa943c1d700 1 -- 172.21.15.32:0/2037628390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa93c10f420 con 0x7fa93c0662e0 2024-04-12T05:34:15.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.259+0000 7fa9327fc700 1 -- 172.21.15.32:0/2037628390 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa93800f040 con 0x7fa93c0662e0 2024-04-12T05:34:15.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.259+0000 7fa9327fc700 1 -- 172.21.15.32:0/2037628390 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa93800c910 con 0x7fa93c0662e0 2024-04-12T05:34:15.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.260+0000 7fa943c1d700 1 -- 172.21.15.32:0/2037628390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9200052e0 con 0x7fa93c0662e0 2024-04-12T05:34:15.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.260+0000 7fa9327fc700 1 -- 172.21.15.32:0/2037628390 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fa938009610 con 0x7fa93c0662e0 2024-04-12T05:34:15.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.261+0000 7fa9327fc700 1 --2- 172.21.15.32:0/2037628390 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fa9280692b0 0x7fa92806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:15.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.261+0000 7fa9327fc700 1 -- 172.21.15.32:0/2037628390 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa9380868a0 con 0x7fa93c0662e0 2024-04-12T05:34:15.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.261+0000 7fa9411b8700 1 --2- 172.21.15.32:0/2037628390 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fa9280692b0 0x7fa92806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:15.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.261+0000 7fa9411b8700 1 --2- 172.21.15.32:0/2037628390 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fa9280692b0 0x7fa92806b760 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa92c013680 tx=0x7fa92c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:15.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.265+0000 7fa9327fc700 1 -- 172.21.15.32:0/2037628390 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fa93804e190 con 0x7fa93c0662e0 2024-04-12T05:34:15.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.469+0000 7fa943c1d700 1 -- 172.21.15.32:0/2037628390 --> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa920000bf0 con 0x7fa9280692b0 2024-04-12T05:34:15.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.471+0000 7fa9327fc700 1 -- 172.21.15.32:0/2037628390 <== mgr.14208 v2:172.21.15.32:6800/595990262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+145 (secure 0 0 0) 0x7fa920000bf0 con 0x7fa9280692b0 2024-04-12T05:34:15.470 INFO:teuthology.orchestra.run.smithi032.stdout:HOST ADDR LABELS STATUS 2024-04-12T05:34:15.470 INFO:teuthology.orchestra.run.smithi032.stdout:smithi032 172.21.15.32 2024-04-12T05:34:15.470 INFO:teuthology.orchestra.run.smithi032.stdout:smithi067 172.21.15.67 2024-04-12T05:34:15.470 INFO:teuthology.orchestra.run.smithi032.stdout:2 hosts in cluster 2024-04-12T05:34:15.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.472+0000 7fa943c1d700 1 -- 172.21.15.32:0/2037628390 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fa9280692b0 msgr2=0x7fa92806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:15.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.472+0000 7fa943c1d700 1 --2- 172.21.15.32:0/2037628390 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fa9280692b0 0x7fa92806b760 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa92c013680 tx=0x7fa92c009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:15.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.472+0000 7fa943c1d700 1 -- 172.21.15.32:0/2037628390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa93c0662e0 msgr2=0x7fa93c10d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:15.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.472+0000 7fa943c1d700 1 --2- 172.21.15.32:0/2037628390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa93c0662e0 0x7fa93c10d920 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa93800eb40 tx=0x7fa93800b040 comp rx=0 tx=0).stop 2024-04-12T05:34:15.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.473+0000 7fa943c1d700 1 -- 172.21.15.32:0/2037628390 shutdown_connections 2024-04-12T05:34:15.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.473+0000 7fa943c1d700 1 --2- 172.21.15.32:0/2037628390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa93c0662e0 0x7fa93c10d920 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:15.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.473+0000 7fa943c1d700 1 --2- 172.21.15.32:0/2037628390 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fa9280692b0 0x7fa92806b760 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:15.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.473+0000 7fa943c1d700 1 --2- 172.21.15.32:0/2037628390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa93c0fff00 0x7fa93c10de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:15.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.473+0000 7fa943c1d700 1 -- 172.21.15.32:0/2037628390 >> 172.21.15.32:0/2037628390 conn(0x7fa93c0f7430 msgr2=0x7fa93c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:15.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.473+0000 7fa943c1d700 1 -- 172.21.15.32:0/2037628390 shutdown_connections 2024-04-12T05:34:15.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:15.473+0000 7fa943c1d700 1 -- 172.21.15.32:0/2037628390 wait complete. 2024-04-12T05:34:16.132 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-12T05:34:16.410 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:34:16.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:16 smithi032 ceph-mon[39731]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:16.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:16 smithi067 ceph-mon[49010]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:17.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:17 smithi032 ceph-mon[39731]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:34:17.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:17 smithi067 ceph-mon[49010]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:34:18.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.124+0000 7f4cbe96b700 1 -- 172.21.15.32:0/1604875011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cb80fcf80 msgr2=0x7f4cb80fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:18.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.124+0000 7f4cbe96b700 1 --2- 172.21.15.32:0/1604875011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cb80fcf80 0x7f4cb80fd3f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0004940 tx=0x7f4ca0009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:18.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.124+0000 7f4cbe96b700 1 -- 172.21.15.32:0/1604875011 shutdown_connections 2024-04-12T05:34:18.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.124+0000 7f4cbe96b700 1 --2- 172.21.15.32:0/1604875011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cb80fcf80 0x7f4cb80fd3f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:18.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.124+0000 7f4cbe96b700 1 --2- 172.21.15.32:0/1604875011 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb80fbcd0 0x7f4cb80fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:18.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.124+0000 7f4cbe96b700 1 -- 172.21.15.32:0/1604875011 >> 172.21.15.32:0/1604875011 conn(0x7f4cb80f7430 msgr2=0x7f4cb80f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:18.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.124+0000 7f4cbe96b700 1 -- 172.21.15.32:0/1604875011 shutdown_connections 2024-04-12T05:34:18.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.124+0000 7f4cbe96b700 1 -- 172.21.15.32:0/1604875011 wait complete. 2024-04-12T05:34:18.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.125+0000 7f4cbe96b700 1 Processor -- start 2024-04-12T05:34:18.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.125+0000 7f4cbe96b700 1 -- start start 2024-04-12T05:34:18.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.125+0000 7f4cbe96b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cb80fbcd0 0x7f4cb8110650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:18.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.125+0000 7f4cbe96b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb80fcf80 0x7f4cb8112ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:18.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.125+0000 7f4cbe96b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb8066370 con 0x7f4cb80fbcd0 2024-04-12T05:34:18.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.125+0000 7f4cbe96b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb80664b0 con 0x7f4cb80fcf80 2024-04-12T05:34:18.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.125+0000 7f4cb7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cb80fbcd0 0x7f4cb8110650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:18.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.125+0000 7f4cb7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cb80fbcd0 0x7f4cb8110650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48776/0 (socket says 172.21.15.32:48776) 2024-04-12T05:34:18.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.125+0000 7f4cb7fff700 1 -- 172.21.15.32:0/2605669207 learned_addr learned my addr 172.21.15.32:0/2605669207 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:34:18.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.125+0000 7f4cb77fe700 1 --2- 172.21.15.32:0/2605669207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb80fcf80 0x7f4cb8112ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:18.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.125+0000 7f4cb7fff700 1 -- 172.21.15.32:0/2605669207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb80fcf80 msgr2=0x7f4cb8112ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:18.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.125+0000 7f4cb7fff700 1 --2- 172.21.15.32:0/2605669207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb80fcf80 0x7f4cb8112ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:18.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.125+0000 7f4cb7fff700 1 -- 172.21.15.32:0/2605669207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca0005040 con 0x7f4cb80fbcd0 2024-04-12T05:34:18.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.125+0000 7f4cb77fe700 1 --2- 172.21.15.32:0/2605669207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb80fcf80 0x7f4cb8112ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:34:18.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.126+0000 7f4cb7fff700 1 --2- 172.21.15.32:0/2605669207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cb80fbcd0 0x7f4cb8110650 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4ca8002a50 tx=0x7f4ca80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:18.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.126+0000 7f4cb4ff9700 1 -- 172.21.15.32:0/2605669207 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ca8007cc0 con 0x7f4cb80fbcd0 2024-04-12T05:34:18.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.126+0000 7f4cb4ff9700 1 -- 172.21.15.32:0/2605669207 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ca800f040 con 0x7f4cb80fbcd0 2024-04-12T05:34:18.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.126+0000 7f4cbe96b700 1 -- 172.21.15.32:0/2605669207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cb81131a0 con 0x7f4cb80fbcd0 2024-04-12T05:34:18.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.126+0000 7f4cb4ff9700 1 -- 172.21.15.32:0/2605669207 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ca8009540 con 0x7f4cb80fbcd0 2024-04-12T05:34:18.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.126+0000 7f4cbe96b700 1 -- 172.21.15.32:0/2605669207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cb8113730 con 0x7f4cb80fbcd0 2024-04-12T05:34:18.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.127+0000 7f4cbe96b700 1 -- 172.21.15.32:0/2605669207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cb805f5f0 con 0x7f4cb80fbcd0 2024-04-12T05:34:18.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.127+0000 7f4cb4ff9700 1 -- 172.21.15.32:0/2605669207 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f4ca80096a0 con 0x7f4cb80fbcd0 2024-04-12T05:34:18.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.128+0000 7f4cb4ff9700 1 --2- 172.21.15.32:0/2605669207 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4ca40692a0 0x7f4ca406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:18.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.128+0000 7f4cb4ff9700 1 -- 172.21.15.32:0/2605669207 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4ca8086c80 con 0x7f4cb80fbcd0 2024-04-12T05:34:18.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.128+0000 7f4cb77fe700 1 --2- 172.21.15.32:0/2605669207 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4ca40692a0 0x7f4ca406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:18.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.128+0000 7f4cb77fe700 1 --2- 172.21.15.32:0/2605669207 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4ca40692a0 0x7f4ca406b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4ca000b3b0 tx=0x7f4ca0009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:18.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.132+0000 7f4cb4ff9700 1 -- 172.21.15.32:0/2605669207 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4ca8053270 con 0x7f4cb80fbcd0 2024-04-12T05:34:18.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.335+0000 7f4cbe96b700 1 -- 172.21.15.32:0/2605669207 --> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f4cb8101380 con 0x7f4ca40692a0 2024-04-12T05:34:18.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.337+0000 7f4cb4ff9700 1 -- 172.21.15.32:0/2605669207 <== mgr.14208 v2:172.21.15.32:6800/595990262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f4cb8101380 con 0x7f4ca40692a0 2024-04-12T05:34:18.336 INFO:teuthology.orchestra.run.smithi032.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-12T05:34:18.336 INFO:teuthology.orchestra.run.smithi032.stdout:smithi032 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB747402DN480DGN 447G No 2m ago Has a FileSystem, LVM detected 2024-04-12T05:34:18.337 INFO:teuthology.orchestra.run.smithi032.stdout:smithi067 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT53440042400BGN 372G No 76s ago Has a FileSystem, LVM detected 2024-04-12T05:34:18.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.339+0000 7f4cbe96b700 1 -- 172.21.15.32:0/2605669207 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4ca40692a0 msgr2=0x7f4ca406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:18.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.339+0000 7f4cbe96b700 1 --2- 172.21.15.32:0/2605669207 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4ca40692a0 0x7f4ca406b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4ca000b3b0 tx=0x7f4ca0009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:18.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.339+0000 7f4cbe96b700 1 -- 172.21.15.32:0/2605669207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cb80fbcd0 msgr2=0x7f4cb8110650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:18.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.339+0000 7f4cbe96b700 1 --2- 172.21.15.32:0/2605669207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cb80fbcd0 0x7f4cb8110650 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4ca8002a50 tx=0x7f4ca80046b0 comp rx=0 tx=0).stop 2024-04-12T05:34:18.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.339+0000 7f4cbe96b700 1 -- 172.21.15.32:0/2605669207 shutdown_connections 2024-04-12T05:34:18.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.339+0000 7f4cbe96b700 1 --2- 172.21.15.32:0/2605669207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4cb80fbcd0 0x7f4cb8110650 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:18.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.339+0000 7f4cbe96b700 1 --2- 172.21.15.32:0/2605669207 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4ca40692a0 0x7f4ca406b750 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:18.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.339+0000 7f4cbe96b700 1 --2- 172.21.15.32:0/2605669207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb80fcf80 0x7f4cb8112ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:18.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.339+0000 7f4cbe96b700 1 -- 172.21.15.32:0/2605669207 >> 172.21.15.32:0/2605669207 conn(0x7f4cb80f7430 msgr2=0x7f4cb80f8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:18.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.339+0000 7f4cbe96b700 1 -- 172.21.15.32:0/2605669207 shutdown_connections 2024-04-12T05:34:18.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:18.339+0000 7f4cbe96b700 1 -- 172.21.15.32:0/2605669207 wait complete. 2024-04-12T05:34:18.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:18 smithi032 ceph-mon[39731]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:18.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:18 smithi067 ceph-mon[49010]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:18.906 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-12T05:34:18.916 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-12T05:34:18.916 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-04-12T05:34:19.197 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:34:19.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:19 smithi032 ceph-mon[39731]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:34:19.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:19 smithi067 ceph-mon[49010]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:34:20.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:20 smithi032 ceph-mon[39731]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:20.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:20 smithi067 ceph-mon[49010]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:20.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.916+0000 7f19dba5e700 1 -- 172.21.15.32:0/2262447081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d40fce40 msgr2=0x7f19d40fd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:20.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.916+0000 7f19dba5e700 1 --2- 172.21.15.32:0/2262447081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d40fce40 0x7f19d40fd250 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f19d0004940 tx=0x7f19d0009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:20.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.916+0000 7f19dba5e700 1 -- 172.21.15.32:0/2262447081 shutdown_connections 2024-04-12T05:34:20.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.916+0000 7f19dba5e700 1 --2- 172.21.15.32:0/2262447081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d40fe040 0x7f19d40fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:20.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.916+0000 7f19dba5e700 1 --2- 172.21.15.32:0/2262447081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d40fce40 0x7f19d40fd250 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:20.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.916+0000 7f19dba5e700 1 -- 172.21.15.32:0/2262447081 >> 172.21.15.32:0/2262447081 conn(0x7f19d40f8510 msgr2=0x7f19d40fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:20.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.916+0000 7f19dba5e700 1 -- 172.21.15.32:0/2262447081 shutdown_connections 2024-04-12T05:34:20.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.916+0000 7f19dba5e700 1 -- 172.21.15.32:0/2262447081 wait complete. 2024-04-12T05:34:20.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.917+0000 7f19dba5e700 1 Processor -- start 2024-04-12T05:34:20.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.917+0000 7f19dba5e700 1 -- start start 2024-04-12T05:34:20.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.917+0000 7f19dba5e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d40fce40 0x7f19d410d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:20.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.917+0000 7f19dba5e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d40fe040 0x7f19d410dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:20.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.917+0000 7f19dba5e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d410e4a0 con 0x7f19d40fce40 2024-04-12T05:34:20.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.917+0000 7f19dba5e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d410eca0 con 0x7f19d40fe040 2024-04-12T05:34:20.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.917+0000 7f19d97fa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d40fce40 0x7f19d410d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:20.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.917+0000 7f19d97fa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d40fce40 0x7f19d410d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48796/0 (socket says 172.21.15.32:48796) 2024-04-12T05:34:20.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.917+0000 7f19d97fa700 1 -- 172.21.15.32:0/117221479 learned_addr learned my addr 172.21.15.32:0/117221479 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:34:20.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.917+0000 7f19d8ff9700 1 --2- 172.21.15.32:0/117221479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d40fe040 0x7f19d410dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:20.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.917+0000 7f19d97fa700 1 -- 172.21.15.32:0/117221479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d40fe040 msgr2=0x7f19d410dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:20.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.917+0000 7f19d97fa700 1 --2- 172.21.15.32:0/117221479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d40fe040 0x7f19d410dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:20.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.917+0000 7f19d97fa700 1 -- 172.21.15.32:0/117221479 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d0005040 con 0x7f19d40fce40 2024-04-12T05:34:20.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.917+0000 7f19d97fa700 1 --2- 172.21.15.32:0/117221479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d40fce40 0x7f19d410d960 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f19d000b3b0 tx=0x7f19d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:20.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.918+0000 7f19c9ffb700 1 -- 172.21.15.32:0/117221479 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19d000e040 con 0x7f19d40fce40 2024-04-12T05:34:20.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.918+0000 7f19c9ffb700 1 -- 172.21.15.32:0/117221479 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f19d000bd20 con 0x7f19d40fce40 2024-04-12T05:34:20.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.918+0000 7f19dba5e700 1 -- 172.21.15.32:0/117221479 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19d410ef20 con 0x7f19d40fce40 2024-04-12T05:34:20.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.918+0000 7f19c9ffb700 1 -- 172.21.15.32:0/117221479 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19d001c630 con 0x7f19d40fce40 2024-04-12T05:34:20.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.918+0000 7f19dba5e700 1 -- 172.21.15.32:0/117221479 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19d410f460 con 0x7f19d40fce40 2024-04-12T05:34:20.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.919+0000 7f19dba5e700 1 -- 172.21.15.32:0/117221479 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d405f5f0 con 0x7f19d40fce40 2024-04-12T05:34:20.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.919+0000 7f19c9ffb700 1 -- 172.21.15.32:0/117221479 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f19d0013960 con 0x7f19d40fce40 2024-04-12T05:34:20.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.920+0000 7f19c9ffb700 1 --2- 172.21.15.32:0/117221479 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f19c00692a0 0x7f19c006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:20.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.920+0000 7f19c9ffb700 1 -- 172.21.15.32:0/117221479 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f19d0087bc0 con 0x7f19d40fce40 2024-04-12T05:34:20.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.920+0000 7f19d8ff9700 1 --2- 172.21.15.32:0/117221479 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f19c00692a0 0x7f19c006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:20.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.920+0000 7f19d8ff9700 1 --2- 172.21.15.32:0/117221479 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f19c00692a0 0x7f19c006b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f19c4004990 tx=0x7f19c4009ee0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:20.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:20.924+0000 7f19c9ffb700 1 -- 172.21.15.32:0/117221479 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f19d0057cc0 con 0x7f19d40fce40 2024-04-12T05:34:21.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:21.150+0000 7f19dba5e700 1 -- 172.21.15.32:0/117221479 --> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f19d410f7e0 con 0x7f19c00692a0 2024-04-12T05:34:22.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:22 smithi032 ceph-mon[39731]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:22.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:22 smithi032 ceph-mon[39731]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:34:22.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:22 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-12T05:34:22.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:22 smithi067 ceph-mon[49010]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:22.602 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:22 smithi067 ceph-mon[49010]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:34:22.602 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:22 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-12T05:34:22.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:22.668+0000 7f19c9ffb700 1 -- 172.21.15.32:0/117221479 <== mgr.14208 v2:172.21.15.32:6800/595990262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f19d410f7e0 con 0x7f19c00692a0 2024-04-12T05:34:22.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:22.672+0000 7f19dba5e700 1 -- 172.21.15.32:0/117221479 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f19c00692a0 msgr2=0x7f19c006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:22.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:22.672+0000 7f19dba5e700 1 --2- 172.21.15.32:0/117221479 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f19c00692a0 0x7f19c006b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f19c4004990 tx=0x7f19c4009ee0 comp rx=0 tx=0).stop 2024-04-12T05:34:22.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:22.672+0000 7f19dba5e700 1 -- 172.21.15.32:0/117221479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d40fce40 msgr2=0x7f19d410d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:22.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:22.672+0000 7f19dba5e700 1 --2- 172.21.15.32:0/117221479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d40fce40 0x7f19d410d960 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f19d000b3b0 tx=0x7f19d0009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:22.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:22.673+0000 7f19dba5e700 1 -- 172.21.15.32:0/117221479 shutdown_connections 2024-04-12T05:34:22.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:22.673+0000 7f19dba5e700 1 --2- 172.21.15.32:0/117221479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d40fce40 0x7f19d410d960 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:22.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:22.673+0000 7f19dba5e700 1 --2- 172.21.15.32:0/117221479 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f19c00692a0 0x7f19c006b750 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:22.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:22.673+0000 7f19dba5e700 1 --2- 172.21.15.32:0/117221479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d40fe040 0x7f19d410dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:22.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:22.673+0000 7f19dba5e700 1 -- 172.21.15.32:0/117221479 >> 172.21.15.32:0/117221479 conn(0x7f19d40f8510 msgr2=0x7f19d40f9cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:22.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:22.673+0000 7f19dba5e700 1 -- 172.21.15.32:0/117221479 shutdown_connections 2024-04-12T05:34:22.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:22.673+0000 7f19dba5e700 1 -- 172.21.15.32:0/117221479 wait complete. 2024-04-12T05:34:23.057 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:22 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-mon-smithi032[39708]: 2024-04-12T05:34:22.640+0000 7f36d4e3d700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T05:34:23.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:23 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-12T05:34:23.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:23 smithi032 ceph-mon[39731]: osdmap e41: 6 total, 6 up, 6 in 2024-04-12T05:34:23.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:23 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-12T05:34:23.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:23 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-12T05:34:23.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:23 smithi032 ceph-mon[39731]: osdmap e42: 6 total, 6 up, 6 in 2024-04-12T05:34:23.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:23 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-12T05:34:23.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:23 smithi032 ceph-mon[39731]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T05:34:23.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:23 smithi032 ceph-mon[39731]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-12T05:34:23.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:23 smithi032 ceph-mon[39731]: osdmap e43: 6 total, 6 up, 6 in 2024-04-12T05:34:23.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:23 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-12T05:34:23.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:23 smithi032 ceph-mon[39731]: fsmap cephfs:0 2024-04-12T05:34:23.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:23 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:23.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:23 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:34:23.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:23 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:34:23.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:23 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:34:23.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:23 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:23.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:23 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.jehaaj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:34:23.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:23 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.jehaaj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T05:34:23.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:23 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:34:23.391 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-12T05:34:23.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:23 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-12T05:34:23.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:23 smithi067 ceph-mon[49010]: osdmap e41: 6 total, 6 up, 6 in 2024-04-12T05:34:23.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:23 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-12T05:34:23.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:23 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-12T05:34:23.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:23 smithi067 ceph-mon[49010]: osdmap e42: 6 total, 6 up, 6 in 2024-04-12T05:34:23.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:23 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-12T05:34:23.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:23 smithi067 ceph-mon[49010]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T05:34:23.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:23 smithi067 ceph-mon[49010]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-12T05:34:23.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:23 smithi067 ceph-mon[49010]: osdmap e43: 6 total, 6 up, 6 in 2024-04-12T05:34:23.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:23 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-12T05:34:23.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:23 smithi067 ceph-mon[49010]: fsmap cephfs:0 2024-04-12T05:34:23.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:23 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:23.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:23 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:34:23.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:23 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:34:23.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:23 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:34:23.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:23 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:23.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:23 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.jehaaj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:34:23.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:23 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.jehaaj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T05:34:23.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:23 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:34:24.294 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:34:24.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:24 smithi032 ceph-mon[39731]: Saving service mds.cephfs spec with placement count:4 2024-04-12T05:34:24.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:24 smithi032 ceph-mon[39731]: Deploying daemon mds.cephfs.smithi032.jehaaj on smithi032 2024-04-12T05:34:24.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:24 smithi032 ceph-mon[39731]: pgmap v180: 65 pgs: 64 unknown, 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:24.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:24 smithi032 ceph-mon[39731]: osdmap e44: 6 total, 6 up, 6 in 2024-04-12T05:34:24.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:24 smithi067 ceph-mon[49010]: Saving service mds.cephfs spec with placement count:4 2024-04-12T05:34:24.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:24 smithi067 ceph-mon[49010]: Deploying daemon mds.cephfs.smithi032.jehaaj on smithi032 2024-04-12T05:34:24.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:24 smithi067 ceph-mon[49010]: pgmap v180: 65 pgs: 64 unknown, 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:24.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:24 smithi067 ceph-mon[49010]: osdmap e44: 6 total, 6 up, 6 in 2024-04-12T05:34:26.194 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:26 smithi032 ceph-mon[39731]: osdmap e45: 6 total, 6 up, 6 in 2024-04-12T05:34:26.194 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:26 smithi032 ceph-mon[39731]: pgmap v183: 65 pgs: 55 unknown, 10 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:26.194 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:26 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:26.194 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:26 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:34:26.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:26 smithi067 ceph-mon[49010]: osdmap e45: 6 total, 6 up, 6 in 2024-04-12T05:34:26.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:26 smithi067 ceph-mon[49010]: pgmap v183: 65 pgs: 55 unknown, 10 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:26.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:26 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:26.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:26 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:34:26.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.696+0000 7f530bcb8700 1 -- 172.21.15.32:0/2102657027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5304066b70 msgr2=0x7f53040ffa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:26.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.696+0000 7f530bcb8700 1 --2- 172.21.15.32:0/2102657027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5304066b70 0x7f53040ffa50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5300004940 tx=0x7f5300009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:26.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.696+0000 7f530bcb8700 1 -- 172.21.15.32:0/2102657027 shutdown_connections 2024-04-12T05:34:26.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.696+0000 7f530bcb8700 1 --2- 172.21.15.32:0/2102657027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53040fff90 0x7f53041023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.696+0000 7f530bcb8700 1 --2- 172.21.15.32:0/2102657027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5304066b70 0x7f53040ffa50 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.696+0000 7f530bcb8700 1 -- 172.21.15.32:0/2102657027 >> 172.21.15.32:0/2102657027 conn(0x7f53040f74e0 msgr2=0x7f53040f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.696+0000 7f530bcb8700 1 -- 172.21.15.32:0/2102657027 shutdown_connections 2024-04-12T05:34:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.696+0000 7f530bcb8700 1 -- 172.21.15.32:0/2102657027 wait complete. 2024-04-12T05:34:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.697+0000 7f530bcb8700 1 Processor -- start 2024-04-12T05:34:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.697+0000 7f530bcb8700 1 -- start start 2024-04-12T05:34:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.697+0000 7f530bcb8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5304066b70 0x7f530410da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.697+0000 7f530bcb8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53040fff90 0x7f530410df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.697+0000 7f530bcb8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530410e540 con 0x7f53040fff90 2024-04-12T05:34:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.697+0000 7f530bcb8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530410ed40 con 0x7f5304066b70 2024-04-12T05:34:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.697+0000 7f5309253700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53040fff90 0x7f530410df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.697+0000 7f5309253700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53040fff90 0x7f530410df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49744/0 (socket says 172.21.15.32:49744) 2024-04-12T05:34:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.697+0000 7f5309253700 1 -- 172.21.15.32:0/2234085522 learned_addr learned my addr 172.21.15.32:0/2234085522 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:34:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.697+0000 7f5309a54700 1 --2- 172.21.15.32:0/2234085522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5304066b70 0x7f530410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.697+0000 7f5309253700 1 -- 172.21.15.32:0/2234085522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5304066b70 msgr2=0x7f530410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.697+0000 7f5309253700 1 --2- 172.21.15.32:0/2234085522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5304066b70 0x7f530410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.697+0000 7f5309253700 1 -- 172.21.15.32:0/2234085522 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5300005040 con 0x7f53040fff90 2024-04-12T05:34:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.698+0000 7f5309253700 1 --2- 172.21.15.32:0/2234085522 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53040fff90 0x7f530410df40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f52f4002a50 tx=0x7f52f40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.698+0000 7f52fa7fc700 1 -- 172.21.15.32:0/2234085522 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52f4007cc0 con 0x7f53040fff90 2024-04-12T05:34:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.698+0000 7f52fa7fc700 1 -- 172.21.15.32:0/2234085522 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f52f400f040 con 0x7f53040fff90 2024-04-12T05:34:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.698+0000 7f530bcb8700 1 -- 172.21.15.32:0/2234085522 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f530410f020 con 0x7f53040fff90 2024-04-12T05:34:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.698+0000 7f52fa7fc700 1 -- 172.21.15.32:0/2234085522 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52f400d9c0 con 0x7f53040fff90 2024-04-12T05:34:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.698+0000 7f530bcb8700 1 -- 172.21.15.32:0/2234085522 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f530410f560 con 0x7f53040fff90 2024-04-12T05:34:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.699+0000 7f530bcb8700 1 -- 172.21.15.32:0/2234085522 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f530405f5f0 con 0x7f53040fff90 2024-04-12T05:34:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.699+0000 7f52fa7fc700 1 -- 172.21.15.32:0/2234085522 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f52f400db20 con 0x7f53040fff90 2024-04-12T05:34:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.700+0000 7f52fa7fc700 1 --2- 172.21.15.32:0/2234085522 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f52f00692a0 0x7f52f006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.700+0000 7f52fa7fc700 1 -- 172.21.15.32:0/2234085522 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f52f4086eb0 con 0x7f53040fff90 2024-04-12T05:34:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.700+0000 7f5309a54700 1 --2- 172.21.15.32:0/2234085522 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f52f00692a0 0x7f52f006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.700+0000 7f5309a54700 1 --2- 172.21.15.32:0/2234085522 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f52f00692a0 0x7f52f006b750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5300004890 tx=0x7f5300009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:26.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.704+0000 7f52fa7fc700 1 -- 172.21.15.32:0/2234085522 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f52f4056bf0 con 0x7f53040fff90 2024-04-12T05:34:26.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.944+0000 7f530bcb8700 1 -- 172.21.15.32:0/2234085522 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f530410f8e0 con 0x7f53040fff90 2024-04-12T05:34:26.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.945+0000 7f52fa7fc700 1 -- 172.21.15.32:0/2234085522 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1076 (secure 0 0 0) 0x7f52f4056780 con 0x7f53040fff90 2024-04-12T05:34:26.944 INFO:teuthology.orchestra.run.smithi032.stdout:e2 2024-04-12T05:34:26.945 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:34:26.945 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:34:26.945 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:34:26.945 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:34:26.945 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:34:26.945 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:34:26.945 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 2 2024-04-12T05:34:26.945 INFO:teuthology.orchestra.run.smithi032.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-12T05:34:26.945 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:34:26.946 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:34:22.641808+0000 2024-04-12T05:34:26.946 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:34:26.946 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:34:26.946 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:34:26.946 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:34:26.946 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:34:26.946 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:34:26.946 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:34:26.946 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 0 2024-04-12T05:34:26.946 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:34:26.947 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 1 2024-04-12T05:34:26.947 INFO:teuthology.orchestra.run.smithi032.stdout:in 2024-04-12T05:34:26.947 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T05:34:26.947 INFO:teuthology.orchestra.run.smithi032.stdout:failed 2024-04-12T05:34:26.947 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:34:26.947 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:34:26.947 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:34:26.947 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:34:26.947 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:34:26.947 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:34:26.947 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 0 2024-04-12T05:34:26.948 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:34:26.948 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:34:26.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.947+0000 7f530bcb8700 1 -- 172.21.15.32:0/2234085522 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f52f00692a0 msgr2=0x7f52f006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:26.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.947+0000 7f530bcb8700 1 --2- 172.21.15.32:0/2234085522 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f52f00692a0 0x7f52f006b750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5300004890 tx=0x7f5300009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:26.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.947+0000 7f530bcb8700 1 -- 172.21.15.32:0/2234085522 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53040fff90 msgr2=0x7f530410df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:26.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.947+0000 7f530bcb8700 1 --2- 172.21.15.32:0/2234085522 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53040fff90 0x7f530410df40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f52f4002a50 tx=0x7f52f40046b0 comp rx=0 tx=0).stop 2024-04-12T05:34:26.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.947+0000 7f530bcb8700 1 -- 172.21.15.32:0/2234085522 shutdown_connections 2024-04-12T05:34:26.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.947+0000 7f530bcb8700 1 --2- 172.21.15.32:0/2234085522 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f52f00692a0 0x7f52f006b750 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:26.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.947+0000 7f530bcb8700 1 --2- 172.21.15.32:0/2234085522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5304066b70 0x7f530410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:26.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.947+0000 7f530bcb8700 1 --2- 172.21.15.32:0/2234085522 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53040fff90 0x7f530410df40 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:26.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.947+0000 7f530bcb8700 1 -- 172.21.15.32:0/2234085522 >> 172.21.15.32:0/2234085522 conn(0x7f53040f74e0 msgr2=0x7f53040f78b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:26.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.947+0000 7f530bcb8700 1 -- 172.21.15.32:0/2234085522 shutdown_connections 2024-04-12T05:34:26.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:26.947+0000 7f530bcb8700 1 -- 172.21.15.32:0/2234085522 wait complete. 2024-04-12T05:34:26.950 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 2 2024-04-12T05:34:27.314 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:27 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/2234085522' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:34:27.315 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:27 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:27.315 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:27 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:27.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:27 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/2234085522' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:34:27.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:27 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:27.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:27 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:27.581 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-12T05:34:27.591 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-12T05:34:27.592 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- bash -c 'ceph fs set cephfs max_mds 2' 2024-04-12T05:34:27.871 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:34:28.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:28 smithi032 ceph-mon[39731]: mds.? [v2:172.21.15.32:6826/2962187015,v1:172.21.15.32:6827/2962187015] up:boot 2024-04-12T05:34:28.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:28 smithi032 ceph-mon[39731]: daemon mds.cephfs.smithi032.jehaaj assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-12T05:34:28.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:28 smithi032 ceph-mon[39731]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T05:34:28.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:28 smithi032 ceph-mon[39731]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-12T05:34:28.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:28 smithi032 ceph-mon[39731]: Cluster is now healthy 2024-04-12T05:34:28.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:28 smithi032 ceph-mon[39731]: fsmap cephfs:0 1 up:standby 2024-04-12T05:34:28.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:28 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.jehaaj"}]: dispatch 2024-04-12T05:34:28.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:28 smithi032 ceph-mon[39731]: fsmap cephfs:1 {0=cephfs.smithi032.jehaaj=up:creating} 2024-04-12T05:34:28.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:28 smithi032 ceph-mon[39731]: daemon mds.cephfs.smithi032.jehaaj is now active in filesystem cephfs as rank 0 2024-04-12T05:34:28.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:28 smithi032 ceph-mon[39731]: pgmap v184: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:28.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:28 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:28.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:28 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.zesmgy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:34:28.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:28 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.zesmgy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T05:34:28.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:28 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:34:28.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:28 smithi032 ceph-mon[39731]: Deploying daemon mds.cephfs.smithi067.zesmgy on smithi067 2024-04-12T05:34:28.431 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:28 smithi067 ceph-mon[49010]: mds.? [v2:172.21.15.32:6826/2962187015,v1:172.21.15.32:6827/2962187015] up:boot 2024-04-12T05:34:28.431 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:28 smithi067 ceph-mon[49010]: daemon mds.cephfs.smithi032.jehaaj assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-12T05:34:28.431 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:28 smithi067 ceph-mon[49010]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T05:34:28.431 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:28 smithi067 ceph-mon[49010]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-12T05:34:28.431 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:28 smithi067 ceph-mon[49010]: Cluster is now healthy 2024-04-12T05:34:28.431 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:28 smithi067 ceph-mon[49010]: fsmap cephfs:0 1 up:standby 2024-04-12T05:34:28.432 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:28 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.jehaaj"}]: dispatch 2024-04-12T05:34:28.432 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:28 smithi067 ceph-mon[49010]: fsmap cephfs:1 {0=cephfs.smithi032.jehaaj=up:creating} 2024-04-12T05:34:28.432 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:28 smithi067 ceph-mon[49010]: daemon mds.cephfs.smithi032.jehaaj is now active in filesystem cephfs as rank 0 2024-04-12T05:34:28.432 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:28 smithi067 ceph-mon[49010]: pgmap v184: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:28.432 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:28 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:28.432 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:28 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.zesmgy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:34:28.432 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:28 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.zesmgy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T05:34:28.432 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:28 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:34:28.432 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:28 smithi067 ceph-mon[49010]: Deploying daemon mds.cephfs.smithi067.zesmgy on smithi067 2024-04-12T05:34:29.099 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:29 smithi067 ceph-mon[49010]: mds.? [v2:172.21.15.32:6826/2962187015,v1:172.21.15.32:6827/2962187015] up:active 2024-04-12T05:34:29.099 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:29 smithi067 ceph-mon[49010]: fsmap cephfs:1 {0=cephfs.smithi032.jehaaj=up:active} 2024-04-12T05:34:29.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:29 smithi032 ceph-mon[39731]: mds.? [v2:172.21.15.32:6826/2962187015,v1:172.21.15.32:6827/2962187015] up:active 2024-04-12T05:34:29.202 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:29 smithi032 ceph-mon[39731]: fsmap cephfs:1 {0=cephfs.smithi032.jehaaj=up:active} 2024-04-12T05:34:29.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.754+0000 7f5b53140700 1 -- 172.21.15.32:0/1512407498 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b4c0fff10 msgr2=0x7f5b4c102330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:29.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.754+0000 7f5b53140700 1 --2- 172.21.15.32:0/1512407498 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b4c0fff10 0x7f5b4c102330 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5b38004940 tx=0x7f5b38009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:29.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.755+0000 7f5b53140700 1 -- 172.21.15.32:0/1512407498 shutdown_connections 2024-04-12T05:34:29.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.755+0000 7f5b53140700 1 --2- 172.21.15.32:0/1512407498 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b4c0fff10 0x7f5b4c102330 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:29.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.755+0000 7f5b53140700 1 --2- 172.21.15.32:0/1512407498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b4c066b70 0x7f5b4c0ff9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:29.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.755+0000 7f5b53140700 1 -- 172.21.15.32:0/1512407498 >> 172.21.15.32:0/1512407498 conn(0x7f5b4c0f7440 msgr2=0x7f5b4c0f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:29.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.755+0000 7f5b53140700 1 -- 172.21.15.32:0/1512407498 shutdown_connections 2024-04-12T05:34:29.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.755+0000 7f5b53140700 1 -- 172.21.15.32:0/1512407498 wait complete. 2024-04-12T05:34:29.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.755+0000 7f5b53140700 1 Processor -- start 2024-04-12T05:34:29.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.755+0000 7f5b53140700 1 -- start start 2024-04-12T05:34:29.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.756+0000 7f5b53140700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b4c066b70 0x7f5b4c0fd400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:29.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.756+0000 7f5b53140700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b4c0fff10 0x7f5b4c0fb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:29.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.756+0000 7f5b53140700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4c0fbe80 con 0x7f5b4c066b70 2024-04-12T05:34:29.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.756+0000 7f5b53140700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4c0fbff0 con 0x7f5b4c0fff10 2024-04-12T05:34:29.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.756+0000 7f5b50edc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b4c066b70 0x7f5b4c0fd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:29.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.756+0000 7f5b50edc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b4c066b70 0x7f5b4c0fd400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49780/0 (socket says 172.21.15.32:49780) 2024-04-12T05:34:29.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.756+0000 7f5b50edc700 1 -- 172.21.15.32:0/2706733168 learned_addr learned my addr 172.21.15.32:0/2706733168 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:34:29.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.756+0000 7f5b4bfff700 1 --2- 172.21.15.32:0/2706733168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b4c0fff10 0x7f5b4c0fb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:29.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.756+0000 7f5b50edc700 1 -- 172.21.15.32:0/2706733168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b4c0fff10 msgr2=0x7f5b4c0fb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:29.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.756+0000 7f5b50edc700 1 --2- 172.21.15.32:0/2706733168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b4c0fff10 0x7f5b4c0fb8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:29.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.756+0000 7f5b50edc700 1 -- 172.21.15.32:0/2706733168 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b38005040 con 0x7f5b4c066b70 2024-04-12T05:34:29.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.756+0000 7f5b50edc700 1 --2- 172.21.15.32:0/2706733168 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b4c066b70 0x7f5b4c0fd400 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5b4000eb40 tx=0x7f5b4000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:29.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.757+0000 7f5b497fa700 1 -- 172.21.15.32:0/2706733168 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b40007cc0 con 0x7f5b4c066b70 2024-04-12T05:34:29.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.757+0000 7f5b497fa700 1 -- 172.21.15.32:0/2706733168 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b4000f040 con 0x7f5b4c066b70 2024-04-12T05:34:29.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.757+0000 7f5b53140700 1 -- 172.21.15.32:0/2706733168 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b4c0fc270 con 0x7f5b4c066b70 2024-04-12T05:34:29.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.757+0000 7f5b497fa700 1 -- 172.21.15.32:0/2706733168 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b4000c8f0 con 0x7f5b4c066b70 2024-04-12T05:34:29.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.757+0000 7f5b53140700 1 -- 172.21.15.32:0/2706733168 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b4c115970 con 0x7f5b4c066b70 2024-04-12T05:34:29.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.758+0000 7f5b53140700 1 -- 172.21.15.32:0/2706733168 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b4c05f5f0 con 0x7f5b4c066b70 2024-04-12T05:34:29.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.759+0000 7f5b497fa700 1 -- 172.21.15.32:0/2706733168 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f5b4000cb60 con 0x7f5b4c066b70 2024-04-12T05:34:29.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.759+0000 7f5b497fa700 1 --2- 172.21.15.32:0/2706733168 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f5b3c0692a0 0x7f5b3c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:29.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.759+0000 7f5b497fa700 1 -- 172.21.15.32:0/2706733168 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f5b40086ea0 con 0x7f5b4c066b70 2024-04-12T05:34:29.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.759+0000 7f5b4bfff700 1 --2- 172.21.15.32:0/2706733168 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f5b3c0692a0 0x7f5b3c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:29.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.760+0000 7f5b4bfff700 1 --2- 172.21.15.32:0/2706733168 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f5b3c0692a0 0x7f5b3c06b750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5b38004890 tx=0x7f5b38009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:29.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:29.763+0000 7f5b497fa700 1 -- 172.21.15.32:0/2706733168 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5b400531e0 con 0x7f5b4c066b70 2024-04-12T05:34:30.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:30.009+0000 7f5b53140700 1 -- 172.21.15.32:0/2706733168 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f5b4c1057a0 con 0x7f5b4c066b70 2024-04-12T05:34:30.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:30.087+0000 7f5b497fa700 1 -- 172.21.15.32:0/2706733168 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7f5b40056800 con 0x7f5b4c066b70 2024-04-12T05:34:30.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:30.089+0000 7f5b53140700 1 -- 172.21.15.32:0/2706733168 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f5b3c0692a0 msgr2=0x7f5b3c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:30.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:30.089+0000 7f5b53140700 1 --2- 172.21.15.32:0/2706733168 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f5b3c0692a0 0x7f5b3c06b750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5b38004890 tx=0x7f5b38009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:30.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:30.090+0000 7f5b53140700 1 -- 172.21.15.32:0/2706733168 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b4c066b70 msgr2=0x7f5b4c0fd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:30.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:30.090+0000 7f5b53140700 1 --2- 172.21.15.32:0/2706733168 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b4c066b70 0x7f5b4c0fd400 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5b4000eb40 tx=0x7f5b4000b040 comp rx=0 tx=0).stop 2024-04-12T05:34:30.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:30.090+0000 7f5b53140700 1 -- 172.21.15.32:0/2706733168 shutdown_connections 2024-04-12T05:34:30.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:30.090+0000 7f5b53140700 1 --2- 172.21.15.32:0/2706733168 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b4c066b70 0x7f5b4c0fd400 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:30.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:30.090+0000 7f5b53140700 1 --2- 172.21.15.32:0/2706733168 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f5b3c0692a0 0x7f5b3c06b750 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:30.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:30.090+0000 7f5b53140700 1 --2- 172.21.15.32:0/2706733168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b4c0fff10 0x7f5b4c0fb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:30.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:30.090+0000 7f5b53140700 1 -- 172.21.15.32:0/2706733168 >> 172.21.15.32:0/2706733168 conn(0x7f5b4c0f7440 msgr2=0x7f5b4c0f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:30.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:30.090+0000 7f5b53140700 1 -- 172.21.15.32:0/2706733168 shutdown_connections 2024-04-12T05:34:30.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:30.090+0000 7f5b53140700 1 -- 172.21.15.32:0/2706733168 wait complete. 2024-04-12T05:34:30.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:30 smithi032 ceph-mon[39731]: pgmap v185: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:30.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:30 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:30.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:30 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:30.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:30 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:30.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:30 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.pndtry", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:34:30.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:30 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.pndtry", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T05:34:30.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:30 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:34:30.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:30 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/2706733168' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-04-12T05:34:30.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:30 smithi067 ceph-mon[49010]: pgmap v185: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:30.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:30 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:30.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:30 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:30.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:30 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:30.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:30 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.pndtry", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:34:30.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:30 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.pndtry", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T05:34:30.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:30 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:34:30.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:30 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/2706733168' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-04-12T05:34:30.799 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-12T05:34:30.809 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-12T05:34:30.810 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- bash -c 'ceph fs set cephfs allow_standby_replay true' 2024-04-12T05:34:31.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:31 smithi032 ceph-mon[39731]: Deploying daemon mds.cephfs.smithi032.pndtry on smithi032 2024-04-12T05:34:31.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:31 smithi032 ceph-mon[39731]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-12T05:34:31.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:31 smithi032 ceph-mon[39731]: mds.? [v2:172.21.15.67:6824/1291914599,v1:172.21.15.67:6825/1291914599] up:boot 2024-04-12T05:34:31.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:31 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/2706733168' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-04-12T05:34:31.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:31 smithi032 ceph-mon[39731]: daemon mds.cephfs.smithi067.zesmgy assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-12T05:34:31.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:31 smithi032 ceph-mon[39731]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-12T05:34:31.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:31 smithi032 ceph-mon[39731]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-12T05:34:31.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:31 smithi032 ceph-mon[39731]: fsmap cephfs:1 {0=cephfs.smithi032.jehaaj=up:active} 1 up:standby 2024-04-12T05:34:31.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:31 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.zesmgy"}]: dispatch 2024-04-12T05:34:31.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:31 smithi032 ceph-mon[39731]: fsmap cephfs:2 {0=cephfs.smithi032.jehaaj=up:active,1=cephfs.smithi067.zesmgy=up:creating} 2024-04-12T05:34:31.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:31 smithi032 ceph-mon[39731]: daemon mds.cephfs.smithi067.zesmgy is now active in filesystem cephfs as rank 1 2024-04-12T05:34:31.503 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:34:31.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:31 smithi067 ceph-mon[49010]: Deploying daemon mds.cephfs.smithi032.pndtry on smithi032 2024-04-12T05:34:31.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:31 smithi067 ceph-mon[49010]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-12T05:34:31.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:31 smithi067 ceph-mon[49010]: mds.? [v2:172.21.15.67:6824/1291914599,v1:172.21.15.67:6825/1291914599] up:boot 2024-04-12T05:34:31.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:31 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/2706733168' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-04-12T05:34:31.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:31 smithi067 ceph-mon[49010]: daemon mds.cephfs.smithi067.zesmgy assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-12T05:34:31.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:31 smithi067 ceph-mon[49010]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-12T05:34:31.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:31 smithi067 ceph-mon[49010]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-12T05:34:31.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:31 smithi067 ceph-mon[49010]: fsmap cephfs:1 {0=cephfs.smithi032.jehaaj=up:active} 1 up:standby 2024-04-12T05:34:31.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:31 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.zesmgy"}]: dispatch 2024-04-12T05:34:31.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:31 smithi067 ceph-mon[49010]: fsmap cephfs:2 {0=cephfs.smithi032.jehaaj=up:active,1=cephfs.smithi067.zesmgy=up:creating} 2024-04-12T05:34:31.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:31 smithi067 ceph-mon[49010]: daemon mds.cephfs.smithi067.zesmgy is now active in filesystem cephfs as rank 1 2024-04-12T05:34:32.333 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:32 smithi032 ceph-mon[39731]: mds.? [v2:172.21.15.67:6824/1291914599,v1:172.21.15.67:6825/1291914599] up:active 2024-04-12T05:34:32.333 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:32 smithi032 ceph-mon[39731]: fsmap cephfs:2 {0=cephfs.smithi032.jehaaj=up:active,1=cephfs.smithi067.zesmgy=up:active} 2024-04-12T05:34:32.333 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:32 smithi032 ceph-mon[39731]: pgmap v186: 65 pgs: 65 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s wr, 6 op/s 2024-04-12T05:34:32.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:32 smithi067 ceph-mon[49010]: mds.? [v2:172.21.15.67:6824/1291914599,v1:172.21.15.67:6825/1291914599] up:active 2024-04-12T05:34:32.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:32 smithi067 ceph-mon[49010]: fsmap cephfs:2 {0=cephfs.smithi032.jehaaj=up:active,1=cephfs.smithi067.zesmgy=up:active} 2024-04-12T05:34:32.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:32 smithi067 ceph-mon[49010]: pgmap v186: 65 pgs: 65 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s wr, 6 op/s 2024-04-12T05:34:34.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.218+0000 7ff8c61fc700 1 -- 172.21.15.32:0/4205225965 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8c00fcf80 msgr2=0x7ff8c00fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:34.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.218+0000 7ff8c61fc700 1 --2- 172.21.15.32:0/4205225965 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8c00fcf80 0x7ff8c00fd3f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4004940 tx=0x7ff8b4009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:34.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.218+0000 7ff8c61fc700 1 -- 172.21.15.32:0/4205225965 shutdown_connections 2024-04-12T05:34:34.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.218+0000 7ff8c61fc700 1 --2- 172.21.15.32:0/4205225965 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8c00fcf80 0x7ff8c00fd3f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:34.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.218+0000 7ff8c61fc700 1 --2- 172.21.15.32:0/4205225965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c00fbcd0 0x7ff8c00fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:34.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.218+0000 7ff8c61fc700 1 -- 172.21.15.32:0/4205225965 >> 172.21.15.32:0/4205225965 conn(0x7ff8c00f7430 msgr2=0x7ff8c00f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:34.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.218+0000 7ff8c61fc700 1 -- 172.21.15.32:0/4205225965 shutdown_connections 2024-04-12T05:34:34.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.218+0000 7ff8c61fc700 1 -- 172.21.15.32:0/4205225965 wait complete. 2024-04-12T05:34:34.218 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.219+0000 7ff8c61fc700 1 Processor -- start 2024-04-12T05:34:34.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.219+0000 7ff8c61fc700 1 -- start start 2024-04-12T05:34:34.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.219+0000 7ff8c61fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c00fbcd0 0x7ff8c010c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:34.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.219+0000 7ff8c61fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8c00fcf80 0x7ff8c010cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:34.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.219+0000 7ff8c61fc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c010d420 con 0x7ff8c00fcf80 2024-04-12T05:34:34.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.219+0000 7ff8c61fc700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c010dc10 con 0x7ff8c00fbcd0 2024-04-12T05:34:34.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.219+0000 7ff8beffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8c00fcf80 0x7ff8c010cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:34.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.219+0000 7ff8beffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8c00fcf80 0x7ff8c010cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37142/0 (socket says 172.21.15.32:37142) 2024-04-12T05:34:34.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.219+0000 7ff8beffd700 1 -- 172.21.15.32:0/1879232839 learned_addr learned my addr 172.21.15.32:0/1879232839 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:34:34.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.220+0000 7ff8bf7fe700 1 --2- 172.21.15.32:0/1879232839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c00fbcd0 0x7ff8c010c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:34.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.220+0000 7ff8beffd700 1 -- 172.21.15.32:0/1879232839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c00fbcd0 msgr2=0x7ff8c010c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:34.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.220+0000 7ff8beffd700 1 --2- 172.21.15.32:0/1879232839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c00fbcd0 0x7ff8c010c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:34.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.220+0000 7ff8beffd700 1 -- 172.21.15.32:0/1879232839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b4005040 con 0x7ff8c00fcf80 2024-04-12T05:34:34.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.220+0000 7ff8bf7fe700 1 --2- 172.21.15.32:0/1879232839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c00fbcd0 0x7ff8c010c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:34:34.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.220+0000 7ff8beffd700 1 --2- 172.21.15.32:0/1879232839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8c00fcf80 0x7ff8c010cd90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff8b400b3b0 tx=0x7ff8b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:34.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.220+0000 7ff8a7fff700 1 -- 172.21.15.32:0/1879232839 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8b400e040 con 0x7ff8c00fcf80 2024-04-12T05:34:34.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.220+0000 7ff8a7fff700 1 -- 172.21.15.32:0/1879232839 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff8b400bd20 con 0x7ff8c00fcf80 2024-04-12T05:34:34.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.220+0000 7ff8c61fc700 1 -- 172.21.15.32:0/1879232839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8c010de90 con 0x7ff8c00fcf80 2024-04-12T05:34:34.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.220+0000 7ff8a7fff700 1 -- 172.21.15.32:0/1879232839 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8b401c630 con 0x7ff8c00fcf80 2024-04-12T05:34:34.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.220+0000 7ff8c61fc700 1 -- 172.21.15.32:0/1879232839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8c010e450 con 0x7ff8c00fcf80 2024-04-12T05:34:34.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.221+0000 7ff8c61fc700 1 -- 172.21.15.32:0/1879232839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8c005f5f0 con 0x7ff8c00fcf80 2024-04-12T05:34:34.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.222+0000 7ff8a7fff700 1 -- 172.21.15.32:0/1879232839 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ff8b4013420 con 0x7ff8c00fcf80 2024-04-12T05:34:34.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.222+0000 7ff8a7fff700 1 --2- 172.21.15.32:0/1879232839 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7ff8a80692a0 0x7ff8a806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:34.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.222+0000 7ff8a7fff700 1 -- 172.21.15.32:0/1879232839 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7ff8b4087f70 con 0x7ff8c00fcf80 2024-04-12T05:34:34.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.222+0000 7ff8bf7fe700 1 --2- 172.21.15.32:0/1879232839 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7ff8a80692a0 0x7ff8a806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:34.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.223+0000 7ff8bf7fe700 1 --2- 172.21.15.32:0/1879232839 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7ff8a80692a0 0x7ff8a806b750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0009b10 tx=0x7ff8b0008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:34.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.227+0000 7ff8a7fff700 1 -- 172.21.15.32:0/1879232839 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ff8b40ba190 con 0x7ff8c00fcf80 2024-04-12T05:34:34.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:34 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:34.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:34 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:34.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:34 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:34.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:34 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.ymvqih", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:34:34.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:34 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.ymvqih", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T05:34:34.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:34 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:34:34.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:34 smithi032 ceph-mon[39731]: Deploying daemon mds.cephfs.smithi067.ymvqih on smithi067 2024-04-12T05:34:34.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:34 smithi032 ceph-mon[39731]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-12T05:34:34.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:34 smithi032 ceph-mon[39731]: Cluster is now healthy 2024-04-12T05:34:34.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:34 smithi032 ceph-mon[39731]: mds.? [v2:172.21.15.32:6828/387754149,v1:172.21.15.32:6829/387754149] up:boot 2024-04-12T05:34:34.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:34 smithi032 ceph-mon[39731]: fsmap cephfs:2 {0=cephfs.smithi032.jehaaj=up:active,1=cephfs.smithi067.zesmgy=up:active} 1 up:standby 2024-04-12T05:34:34.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:34 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.pndtry"}]: dispatch 2024-04-12T05:34:34.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:34 smithi032 ceph-mon[39731]: pgmap v187: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s wr, 8 op/s 2024-04-12T05:34:34.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:34.468+0000 7ff8c61fc700 1 -- 172.21.15.32:0/1879232839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7ff8c004bed0 con 0x7ff8c00fcf80 2024-04-12T05:34:34.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:34 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:34.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:34 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:34.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:34 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:34.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:34 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.ymvqih", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:34:34.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:34 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.ymvqih", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T05:34:34.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:34 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:34:34.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:34 smithi067 ceph-mon[49010]: Deploying daemon mds.cephfs.smithi067.ymvqih on smithi067 2024-04-12T05:34:34.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:34 smithi067 ceph-mon[49010]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-12T05:34:34.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:34 smithi067 ceph-mon[49010]: Cluster is now healthy 2024-04-12T05:34:34.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:34 smithi067 ceph-mon[49010]: mds.? [v2:172.21.15.32:6828/387754149,v1:172.21.15.32:6829/387754149] up:boot 2024-04-12T05:34:34.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:34 smithi067 ceph-mon[49010]: fsmap cephfs:2 {0=cephfs.smithi032.jehaaj=up:active,1=cephfs.smithi067.zesmgy=up:active} 1 up:standby 2024-04-12T05:34:34.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:34 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.pndtry"}]: dispatch 2024-04-12T05:34:34.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:34 smithi067 ceph-mon[49010]: pgmap v187: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s wr, 8 op/s 2024-04-12T05:34:35.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:35.104+0000 7ff8a7fff700 1 -- 172.21.15.32:0/1879232839 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v10) v1 ==== 121+0+0 (secure 0 0 0) 0x7ff8c004bed0 con 0x7ff8c00fcf80 2024-04-12T05:34:35.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:35.107+0000 7ff8c61fc700 1 -- 172.21.15.32:0/1879232839 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7ff8a80692a0 msgr2=0x7ff8a806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:35.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:35.107+0000 7ff8c61fc700 1 --2- 172.21.15.32:0/1879232839 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7ff8a80692a0 0x7ff8a806b750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0009b10 tx=0x7ff8b0008040 comp rx=0 tx=0).stop 2024-04-12T05:34:35.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:35.107+0000 7ff8c61fc700 1 -- 172.21.15.32:0/1879232839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8c00fcf80 msgr2=0x7ff8c010cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:35.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:35.107+0000 7ff8c61fc700 1 --2- 172.21.15.32:0/1879232839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8c00fcf80 0x7ff8c010cd90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff8b400b3b0 tx=0x7ff8b4009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:35.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:35.107+0000 7ff8c61fc700 1 -- 172.21.15.32:0/1879232839 shutdown_connections 2024-04-12T05:34:35.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:35.107+0000 7ff8c61fc700 1 --2- 172.21.15.32:0/1879232839 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7ff8a80692a0 0x7ff8a806b750 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:35.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:35.107+0000 7ff8c61fc700 1 --2- 172.21.15.32:0/1879232839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c00fbcd0 0x7ff8c010c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:35.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:35.107+0000 7ff8c61fc700 1 --2- 172.21.15.32:0/1879232839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8c00fcf80 0x7ff8c010cd90 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:35.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:35.107+0000 7ff8c61fc700 1 -- 172.21.15.32:0/1879232839 >> 172.21.15.32:0/1879232839 conn(0x7ff8c00f7430 msgr2=0x7ff8c00f8b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:35.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:35.107+0000 7ff8c61fc700 1 -- 172.21.15.32:0/1879232839 shutdown_connections 2024-04-12T05:34:35.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:35.107+0000 7ff8c61fc700 1 -- 172.21.15.32:0/1879232839 wait complete. 2024-04-12T05:34:35.208 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:35 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/1879232839' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-12T05:34:35.521 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:35 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/1879232839' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-12T05:34:35.837 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-12T05:34:35.848 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-12T05:34:35.848 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- bash -c 'ceph fs set cephfs inline_data false' 2024-04-12T05:34:36.150 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:34:36.358 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:36 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/1879232839' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-12T05:34:36.358 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:36 smithi032 ceph-mon[39731]: fsmap cephfs:2 {0=cephfs.smithi032.jehaaj=up:active,1=cephfs.smithi067.zesmgy=up:active} 1 up:standby 2024-04-12T05:34:36.358 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:36 smithi032 ceph-mon[39731]: fsmap cephfs:2 {0=cephfs.smithi032.jehaaj=up:active,1=cephfs.smithi067.zesmgy=up:active} 1 up:standby-replay 2024-04-12T05:34:36.358 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:36 smithi032 ceph-mon[39731]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s wr, 8 op/s 2024-04-12T05:34:36.358 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:36 smithi032 ceph-mon[39731]: mds.? [v2:172.21.15.67:6826/873395842,v1:172.21.15.67:6827/873395842] up:boot 2024-04-12T05:34:36.358 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:36 smithi032 ceph-mon[39731]: fsmap cephfs:2 {0=cephfs.smithi032.jehaaj=up:active,1=cephfs.smithi067.zesmgy=up:active} 2 up:standby-replay 2024-04-12T05:34:36.358 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:36 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.ymvqih"}]: dispatch 2024-04-12T05:34:36.358 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:36 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.ymvqih"}]: dispatch 2024-04-12T05:34:36.358 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:36 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:36.359 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:36 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:36.359 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:36 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:36.359 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:36 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:36.359 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:36 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:36.359 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:36 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:34:36.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:36 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/1879232839' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-12T05:34:36.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:36 smithi067 ceph-mon[49010]: fsmap cephfs:2 {0=cephfs.smithi032.jehaaj=up:active,1=cephfs.smithi067.zesmgy=up:active} 1 up:standby 2024-04-12T05:34:36.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:36 smithi067 ceph-mon[49010]: fsmap cephfs:2 {0=cephfs.smithi032.jehaaj=up:active,1=cephfs.smithi067.zesmgy=up:active} 1 up:standby-replay 2024-04-12T05:34:36.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:36 smithi067 ceph-mon[49010]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s wr, 8 op/s 2024-04-12T05:34:36.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:36 smithi067 ceph-mon[49010]: mds.? [v2:172.21.15.67:6826/873395842,v1:172.21.15.67:6827/873395842] up:boot 2024-04-12T05:34:36.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:36 smithi067 ceph-mon[49010]: fsmap cephfs:2 {0=cephfs.smithi032.jehaaj=up:active,1=cephfs.smithi067.zesmgy=up:active} 2 up:standby-replay 2024-04-12T05:34:36.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:36 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.ymvqih"}]: dispatch 2024-04-12T05:34:36.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:36 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.ymvqih"}]: dispatch 2024-04-12T05:34:36.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:36 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:36.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:36 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:36.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:36 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:36.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:36 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:36.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:36 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:36.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:36 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:34:38.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.204+0000 7f5ca32c1700 1 -- 172.21.15.32:0/1163605360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c9c102360 msgr2=0x7f5c9c104740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:38.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.204+0000 7f5ca32c1700 1 --2- 172.21.15.32:0/1163605360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c9c102360 0x7f5c9c104740 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5c90004940 tx=0x7f5c90009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:38.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.204+0000 7f5ca32c1700 1 -- 172.21.15.32:0/1163605360 shutdown_connections 2024-04-12T05:34:38.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.204+0000 7f5ca32c1700 1 --2- 172.21.15.32:0/1163605360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c9c102360 0x7f5c9c104740 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:38.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.204+0000 7f5ca32c1700 1 --2- 172.21.15.32:0/1163605360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c9c066280 0x7f5c9c101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:38.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.204+0000 7f5ca32c1700 1 -- 172.21.15.32:0/1163605360 >> 172.21.15.32:0/1163605360 conn(0x7f5c9c0f7430 msgr2=0x7f5c9c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:38.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.204+0000 7f5ca32c1700 1 -- 172.21.15.32:0/1163605360 shutdown_connections 2024-04-12T05:34:38.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.204+0000 7f5ca32c1700 1 -- 172.21.15.32:0/1163605360 wait complete. 2024-04-12T05:34:38.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.205+0000 7f5ca32c1700 1 Processor -- start 2024-04-12T05:34:38.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.205+0000 7f5ca32c1700 1 -- start start 2024-04-12T05:34:38.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.205+0000 7f5ca32c1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c9c066280 0x7f5c9c10fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:38.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.205+0000 7f5ca32c1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c9c102360 0x7f5c9c1100a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:38.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.205+0000 7f5ca32c1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9c1106a0 con 0x7f5c9c066280 2024-04-12T05:34:38.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.205+0000 7f5ca32c1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9c110ef0 con 0x7f5c9c102360 2024-04-12T05:34:38.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.205+0000 7f5ca105d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c9c066280 0x7f5c9c10fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:38.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.205+0000 7f5ca105d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c9c066280 0x7f5c9c10fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37174/0 (socket says 172.21.15.32:37174) 2024-04-12T05:34:38.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.205+0000 7f5ca105d700 1 -- 172.21.15.32:0/3043060994 learned_addr learned my addr 172.21.15.32:0/3043060994 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:34:38.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.205+0000 7f5ca085c700 1 --2- 172.21.15.32:0/3043060994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c9c102360 0x7f5c9c1100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:38.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.205+0000 7f5ca105d700 1 -- 172.21.15.32:0/3043060994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c9c102360 msgr2=0x7f5c9c1100a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:38.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.205+0000 7f5ca105d700 1 --2- 172.21.15.32:0/3043060994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c9c102360 0x7f5c9c1100a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:38.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.205+0000 7f5ca105d700 1 -- 172.21.15.32:0/3043060994 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c90005040 con 0x7f5c9c066280 2024-04-12T05:34:38.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.206+0000 7f5ca105d700 1 --2- 172.21.15.32:0/3043060994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c9c066280 0x7f5c9c10fb60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c002a80 tx=0x7f5c8c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:38.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.206+0000 7f5c99ffb700 1 -- 172.21.15.32:0/3043060994 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c8c007cc0 con 0x7f5c9c066280 2024-04-12T05:34:38.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.206+0000 7f5c99ffb700 1 -- 172.21.15.32:0/3043060994 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c8c00f040 con 0x7f5c9c066280 2024-04-12T05:34:38.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.206+0000 7f5c99ffb700 1 -- 172.21.15.32:0/3043060994 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c8c00d9c0 con 0x7f5c9c066280 2024-04-12T05:34:38.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.206+0000 7f5ca32c1700 1 -- 172.21.15.32:0/3043060994 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c9c1111d0 con 0x7f5c9c066280 2024-04-12T05:34:38.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.206+0000 7f5ca32c1700 1 -- 172.21.15.32:0/3043060994 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c9c1116c0 con 0x7f5c9c066280 2024-04-12T05:34:38.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.207+0000 7f5ca32c1700 1 -- 172.21.15.32:0/3043060994 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c9c05f5f0 con 0x7f5c9c066280 2024-04-12T05:34:38.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.208+0000 7f5c99ffb700 1 -- 172.21.15.32:0/3043060994 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f5c8c00db20 con 0x7f5c9c066280 2024-04-12T05:34:38.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.208+0000 7f5c99ffb700 1 --2- 172.21.15.32:0/3043060994 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f5c840692a0 0x7f5c8406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:38.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.208+0000 7f5ca085c700 1 --2- 172.21.15.32:0/3043060994 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f5c840692a0 0x7f5c8406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:38.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.208+0000 7f5c99ffb700 1 -- 172.21.15.32:0/3043060994 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f5c8c086f10 con 0x7f5c9c066280 2024-04-12T05:34:38.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.208+0000 7f5ca085c700 1 --2- 172.21.15.32:0/3043060994 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f5c840692a0 0x7f5c8406b750 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5c9000b3b0 tx=0x7f5c90009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:38.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.213+0000 7f5c99ffb700 1 -- 172.21.15.32:0/3043060994 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5c8c056c00 con 0x7f5c9c066280 2024-04-12T05:34:38.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:38.454+0000 7f5ca32c1700 1 -- 172.21.15.32:0/3043060994 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"} v 0) v1 -- 0x7f5c9c04bed0 con 0x7f5c9c066280 2024-04-12T05:34:38.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:38 smithi067 ceph-mon[49010]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 2.0 KiB/s wr, 8 op/s 2024-04-12T05:34:38.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:38 smithi067 ceph-mon[49010]: mds.? [v2:172.21.15.32:6828/387754149,v1:172.21.15.32:6829/387754149] up:standby-replay 2024-04-12T05:34:38.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:38 smithi067 ceph-mon[49010]: Dropping low affinity active daemon mds.cephfs.smithi032.jehaaj in favor of higher affinity standby. 2024-04-12T05:34:38.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:38 smithi067 ceph-mon[49010]: Replacing daemon mds.cephfs.smithi032.jehaaj as rank 0 with standby daemon mds.cephfs.smithi032.pndtry 2024-04-12T05:34:38.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:38 smithi067 ceph-mon[49010]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T05:34:38.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:38 smithi067 ceph-mon[49010]: fsmap cephfs:2 {0=cephfs.smithi032.jehaaj=up:active,1=cephfs.smithi067.zesmgy=up:active} 2 up:standby-replay 2024-04-12T05:34:38.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:38 smithi067 ceph-mon[49010]: osdmap e46: 6 total, 6 up, 6 in 2024-04-12T05:34:38.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:38 smithi067 ceph-mon[49010]: fsmap cephfs:2/2 {0=cephfs.smithi032.pndtry=up:replay,1=cephfs.smithi067.zesmgy=up:active} 1 up:standby-replay 2024-04-12T05:34:38.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:38 smithi032 ceph-mon[39731]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 2.0 KiB/s wr, 8 op/s 2024-04-12T05:34:38.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:38 smithi032 ceph-mon[39731]: mds.? [v2:172.21.15.32:6828/387754149,v1:172.21.15.32:6829/387754149] up:standby-replay 2024-04-12T05:34:38.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:38 smithi032 ceph-mon[39731]: Dropping low affinity active daemon mds.cephfs.smithi032.jehaaj in favor of higher affinity standby. 2024-04-12T05:34:38.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:38 smithi032 ceph-mon[39731]: Replacing daemon mds.cephfs.smithi032.jehaaj as rank 0 with standby daemon mds.cephfs.smithi032.pndtry 2024-04-12T05:34:38.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:38 smithi032 ceph-mon[39731]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T05:34:38.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:38 smithi032 ceph-mon[39731]: fsmap cephfs:2 {0=cephfs.smithi032.jehaaj=up:active,1=cephfs.smithi067.zesmgy=up:active} 2 up:standby-replay 2024-04-12T05:34:38.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:38 smithi032 ceph-mon[39731]: osdmap e46: 6 total, 6 up, 6 in 2024-04-12T05:34:38.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:38 smithi032 ceph-mon[39731]: fsmap cephfs:2/2 {0=cephfs.smithi032.pndtry=up:replay,1=cephfs.smithi067.zesmgy=up:active} 1 up:standby-replay 2024-04-12T05:34:39.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:39.122+0000 7f5c99ffb700 1 -- 172.21.15.32:0/3043060994 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]=0 inline data disabled v16) v1 ==== 133+0+0 (secure 0 0 0) 0x7f5c8c056790 con 0x7f5c9c066280 2024-04-12T05:34:39.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:39.125+0000 7f5ca32c1700 1 -- 172.21.15.32:0/3043060994 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f5c840692a0 msgr2=0x7f5c8406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:39.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:39.125+0000 7f5ca32c1700 1 --2- 172.21.15.32:0/3043060994 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f5c840692a0 0x7f5c8406b750 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5c9000b3b0 tx=0x7f5c90009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:39.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:39.125+0000 7f5ca32c1700 1 -- 172.21.15.32:0/3043060994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c9c066280 msgr2=0x7f5c9c10fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:39.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:39.125+0000 7f5ca32c1700 1 --2- 172.21.15.32:0/3043060994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c9c066280 0x7f5c9c10fb60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c002a80 tx=0x7f5c8c004600 comp rx=0 tx=0).stop 2024-04-12T05:34:39.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:39.125+0000 7f5ca32c1700 1 -- 172.21.15.32:0/3043060994 shutdown_connections 2024-04-12T05:34:39.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:39.125+0000 7f5ca32c1700 1 --2- 172.21.15.32:0/3043060994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c9c066280 0x7f5c9c10fb60 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:39.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:39.125+0000 7f5ca32c1700 1 --2- 172.21.15.32:0/3043060994 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f5c840692a0 0x7f5c8406b750 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:39.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:39.125+0000 7f5ca32c1700 1 --2- 172.21.15.32:0/3043060994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c9c102360 0x7f5c9c1100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:39.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:39.125+0000 7f5ca32c1700 1 -- 172.21.15.32:0/3043060994 >> 172.21.15.32:0/3043060994 conn(0x7f5c9c0f7430 msgr2=0x7f5c9c0f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:39.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:39.125+0000 7f5ca32c1700 1 -- 172.21.15.32:0/3043060994 shutdown_connections 2024-04-12T05:34:39.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:39.125+0000 7f5ca32c1700 1 -- 172.21.15.32:0/3043060994 wait complete. 2024-04-12T05:34:39.124 INFO:teuthology.orchestra.run.smithi032.stderr:inline data disabled 2024-04-12T05:34:39.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:39 smithi032 ceph-mon[39731]: mds.? [v2:172.21.15.32:6828/387754149,v1:172.21.15.32:6829/387754149] up:resolve 2024-04-12T05:34:39.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:39 smithi032 ceph-mon[39731]: mds.? [v2:172.21.15.32:6826/3678031004,v1:172.21.15.32:6827/3678031004] up:boot 2024-04-12T05:34:39.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:39 smithi032 ceph-mon[39731]: mds.? [v2:172.21.15.67:6824/1291914599,v1:172.21.15.67:6825/1291914599] up:active 2024-04-12T05:34:39.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:39 smithi032 ceph-mon[39731]: fsmap cephfs:2/2 {0=cephfs.smithi032.pndtry=up:resolve,1=cephfs.smithi067.zesmgy=up:active} 1 up:standby-replay 1 up:standby 2024-04-12T05:34:39.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:39 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.jehaaj"}]: dispatch 2024-04-12T05:34:39.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:39 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:39.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:39 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:39.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:39 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/3043060994' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]: dispatch 2024-04-12T05:34:39.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:39 smithi067 ceph-mon[49010]: mds.? [v2:172.21.15.32:6828/387754149,v1:172.21.15.32:6829/387754149] up:resolve 2024-04-12T05:34:39.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:39 smithi067 ceph-mon[49010]: mds.? [v2:172.21.15.32:6826/3678031004,v1:172.21.15.32:6827/3678031004] up:boot 2024-04-12T05:34:39.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:39 smithi067 ceph-mon[49010]: mds.? [v2:172.21.15.67:6824/1291914599,v1:172.21.15.67:6825/1291914599] up:active 2024-04-12T05:34:39.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:39 smithi067 ceph-mon[49010]: fsmap cephfs:2/2 {0=cephfs.smithi032.pndtry=up:resolve,1=cephfs.smithi067.zesmgy=up:active} 1 up:standby-replay 1 up:standby 2024-04-12T05:34:39.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:39 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.jehaaj"}]: dispatch 2024-04-12T05:34:39.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:39 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:39.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:39 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:39.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:39 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/3043060994' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]: dispatch 2024-04-12T05:34:40.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:40 smithi032 ceph-mon[39731]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-04-12T05:34:40.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:40 smithi032 ceph-mon[39731]: mds.? [v2:172.21.15.32:6828/387754149,v1:172.21.15.32:6829/387754149] up:reconnect 2024-04-12T05:34:40.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:40 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/3043060994' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]': finished 2024-04-12T05:34:40.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:40 smithi032 ceph-mon[39731]: fsmap cephfs:2/2 {0=cephfs.smithi032.pndtry=up:reconnect,1=cephfs.smithi067.zesmgy=up:active} 1 up:standby-replay 1 up:standby 2024-04-12T05:34:40.438 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-12T05:34:40.448 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-12T05:34:40.448 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-12T05:34:40.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:40 smithi067 ceph-mon[49010]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-04-12T05:34:40.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:40 smithi067 ceph-mon[49010]: mds.? [v2:172.21.15.32:6828/387754149,v1:172.21.15.32:6829/387754149] up:reconnect 2024-04-12T05:34:40.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:40 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/3043060994' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]': finished 2024-04-12T05:34:40.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:40 smithi067 ceph-mon[49010]: fsmap cephfs:2/2 {0=cephfs.smithi032.pndtry=up:reconnect,1=cephfs.smithi067.zesmgy=up:active} 1 up:standby-replay 1 up:standby 2024-04-12T05:34:40.817 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:34:41.518 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:41 smithi032 ceph-mon[39731]: mds.? [v2:172.21.15.32:6828/387754149,v1:172.21.15.32:6829/387754149] up:rejoin 2024-04-12T05:34:41.518 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:41 smithi032 ceph-mon[39731]: mds.? [v2:172.21.15.67:6826/873395842,v1:172.21.15.67:6827/873395842] up:standby-replay 2024-04-12T05:34:41.518 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:41 smithi032 ceph-mon[39731]: fsmap cephfs:2/2 {0=cephfs.smithi032.pndtry=up:rejoin,1=cephfs.smithi067.zesmgy=up:active} 1 up:standby-replay 1 up:standby 2024-04-12T05:34:41.518 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:41 smithi032 ceph-mon[39731]: daemon mds.cephfs.smithi032.pndtry is now active in filesystem cephfs as rank 0 2024-04-12T05:34:41.518 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:41 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:41.519 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:41 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:34:41.519 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:41 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:41.519 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:41 smithi032 ceph-mon[39731]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T05:34:41.519 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:41 smithi032 ceph-mon[39731]: Cluster is now healthy 2024-04-12T05:34:41.519 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:41 smithi032 ceph-mon[39731]: mds.? [v2:172.21.15.32:6828/387754149,v1:172.21.15.32:6829/387754149] up:active 2024-04-12T05:34:41.519 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:41 smithi032 ceph-mon[39731]: fsmap cephfs:2 {0=cephfs.smithi032.pndtry=up:active,1=cephfs.smithi067.zesmgy=up:active} 2 up:standby-replay 2024-04-12T05:34:41.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:41 smithi067 ceph-mon[49010]: mds.? [v2:172.21.15.32:6828/387754149,v1:172.21.15.32:6829/387754149] up:rejoin 2024-04-12T05:34:41.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:41 smithi067 ceph-mon[49010]: mds.? [v2:172.21.15.67:6826/873395842,v1:172.21.15.67:6827/873395842] up:standby-replay 2024-04-12T05:34:41.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:41 smithi067 ceph-mon[49010]: fsmap cephfs:2/2 {0=cephfs.smithi032.pndtry=up:rejoin,1=cephfs.smithi067.zesmgy=up:active} 1 up:standby-replay 1 up:standby 2024-04-12T05:34:41.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:41 smithi067 ceph-mon[49010]: daemon mds.cephfs.smithi032.pndtry is now active in filesystem cephfs as rank 0 2024-04-12T05:34:41.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:41 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:41.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:41 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:34:41.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:41 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:41.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:41 smithi067 ceph-mon[49010]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T05:34:41.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:41 smithi067 ceph-mon[49010]: Cluster is now healthy 2024-04-12T05:34:41.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:41 smithi067 ceph-mon[49010]: mds.? [v2:172.21.15.32:6828/387754149,v1:172.21.15.32:6829/387754149] up:active 2024-04-12T05:34:41.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:41 smithi067 ceph-mon[49010]: fsmap cephfs:2 {0=cephfs.smithi032.pndtry=up:active,1=cephfs.smithi067.zesmgy=up:active} 2 up:standby-replay 2024-04-12T05:34:42.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:42 smithi032 ceph-mon[39731]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-04-12T05:34:42.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:42 smithi067 ceph-mon[49010]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-04-12T05:34:43.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.200+0000 7f9d4da2d700 1 -- 172.21.15.32:0/2753711184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d480fff80 msgr2=0x7f9d48102360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:43.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.200+0000 7f9d4da2d700 1 --2- 172.21.15.32:0/2753711184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d480fff80 0x7f9d48102360 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c004940 tx=0x7f9d3c009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:43.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.200+0000 7f9d4da2d700 1 -- 172.21.15.32:0/2753711184 shutdown_connections 2024-04-12T05:34:43.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.200+0000 7f9d4da2d700 1 --2- 172.21.15.32:0/2753711184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d480fff80 0x7f9d48102360 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:43.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.200+0000 7f9d4da2d700 1 --2- 172.21.15.32:0/2753711184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d48066260 0x7f9d480ffa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:43.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.200+0000 7f9d4da2d700 1 -- 172.21.15.32:0/2753711184 >> 172.21.15.32:0/2753711184 conn(0x7f9d480f7430 msgr2=0x7f9d480f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:43.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.200+0000 7f9d4da2d700 1 -- 172.21.15.32:0/2753711184 shutdown_connections 2024-04-12T05:34:43.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.200+0000 7f9d4da2d700 1 -- 172.21.15.32:0/2753711184 wait complete. 2024-04-12T05:34:43.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.201+0000 7f9d4da2d700 1 Processor -- start 2024-04-12T05:34:43.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.201+0000 7f9d4da2d700 1 -- start start 2024-04-12T05:34:43.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.202+0000 7f9d4da2d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d48066260 0x7f9d4810d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:43.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.202+0000 7f9d4da2d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d480fff80 0x7f9d4810dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:43.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.202+0000 7f9d4da2d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4810e4a0 con 0x7f9d48066260 2024-04-12T05:34:43.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.202+0000 7f9d4da2d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4810ecf0 con 0x7f9d480fff80 2024-04-12T05:34:43.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.202+0000 7f9d46ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d48066260 0x7f9d4810d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:43.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.202+0000 7f9d46ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d48066260 0x7f9d4810d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37138/0 (socket says 172.21.15.32:37138) 2024-04-12T05:34:43.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.202+0000 7f9d46ffd700 1 -- 172.21.15.32:0/277345749 learned_addr learned my addr 172.21.15.32:0/277345749 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:34:43.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.202+0000 7f9d467fc700 1 --2- 172.21.15.32:0/277345749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d480fff80 0x7f9d4810dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:43.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.202+0000 7f9d46ffd700 1 -- 172.21.15.32:0/277345749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d480fff80 msgr2=0x7f9d4810dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:43.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.202+0000 7f9d46ffd700 1 --2- 172.21.15.32:0/277345749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d480fff80 0x7f9d4810dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:43.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.202+0000 7f9d46ffd700 1 -- 172.21.15.32:0/277345749 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d3c005040 con 0x7f9d48066260 2024-04-12T05:34:43.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.202+0000 7f9d46ffd700 1 --2- 172.21.15.32:0/277345749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d48066260 0x7f9d4810d960 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9d38002a80 tx=0x7f9d38004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:43.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.202+0000 7f9d2f7fe700 1 -- 172.21.15.32:0/277345749 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d38007cc0 con 0x7f9d48066260 2024-04-12T05:34:43.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.202+0000 7f9d2f7fe700 1 -- 172.21.15.32:0/277345749 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d3800f040 con 0x7f9d48066260 2024-04-12T05:34:43.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.202+0000 7f9d4da2d700 1 -- 172.21.15.32:0/277345749 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d4810efd0 con 0x7f9d48066260 2024-04-12T05:34:43.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.202+0000 7f9d2f7fe700 1 -- 172.21.15.32:0/277345749 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d3800da20 con 0x7f9d48066260 2024-04-12T05:34:43.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.202+0000 7f9d4da2d700 1 -- 172.21.15.32:0/277345749 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d4810f510 con 0x7f9d48066260 2024-04-12T05:34:43.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.203+0000 7f9d4da2d700 1 -- 172.21.15.32:0/277345749 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d4805f5f0 con 0x7f9d48066260 2024-04-12T05:34:43.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.204+0000 7f9d2f7fe700 1 -- 172.21.15.32:0/277345749 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f9d3800db80 con 0x7f9d48066260 2024-04-12T05:34:43.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.204+0000 7f9d2f7fe700 1 --2- 172.21.15.32:0/277345749 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f9d300692a0 0x7f9d3006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:43.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.204+0000 7f9d2f7fe700 1 -- 172.21.15.32:0/277345749 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f9d38086f20 con 0x7f9d48066260 2024-04-12T05:34:43.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.204+0000 7f9d467fc700 1 --2- 172.21.15.32:0/277345749 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f9d300692a0 0x7f9d3006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:43.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.205+0000 7f9d467fc700 1 --2- 172.21.15.32:0/277345749 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f9d300692a0 0x7f9d3006b750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c00b3b0 tx=0x7f9d3c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:43.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.209+0000 7f9d2f7fe700 1 -- 172.21.15.32:0/277345749 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9d38056c90 con 0x7f9d48066260 2024-04-12T05:34:43.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.451+0000 7f9d4da2d700 1 -- 172.21.15.32:0/277345749 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9d4810f880 con 0x7f9d48066260 2024-04-12T05:34:43.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.452+0000 7f9d2f7fe700 1 -- 172.21.15.32:0/277345749 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1826 (secure 0 0 0) 0x7f9d38056820 con 0x7f9d48066260 2024-04-12T05:34:43.450 INFO:teuthology.orchestra.run.smithi032.stdout:e18 2024-04-12T05:34:43.451 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:34:43.451 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:34:43.451 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:34:43.451 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:34:43.451 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:34:43.451 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:34:43.451 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 18 2024-04-12T05:34:43.451 INFO:teuthology.orchestra.run.smithi032.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-12T05:34:43.452 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:34:43.452 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:34:40.809969+0000 2024-04-12T05:34:43.452 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:34:43.452 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:34:43.452 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:34:43.452 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:34:43.452 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:34:43.452 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:34:43.453 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:34:43.453 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 46 2024-04-12T05:34:43.453 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:34:43.453 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:34:43.453 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:34:43.453 INFO:teuthology.orchestra.run.smithi032.stdout:up {0=14480,1=24281} 2024-04-12T05:34:43.453 INFO:teuthology.orchestra.run.smithi032.stdout:failed 2024-04-12T05:34:43.453 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:34:43.453 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:34:43.453 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:34:43.454 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:34:43.454 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:34:43.454 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:34:43.455 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:34:43.455 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{0:14480} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.32:6828/387754149,v1:172.21.15.32:6829/387754149] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:34:43.455 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{0:14490} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3678031004,v1:172.21.15.32:6827/3678031004] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:34:43.455 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{1:24281} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.67:6824/1291914599,v1:172.21.15.67:6825/1291914599] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:34:43.455 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{1:24295} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.67:6826/873395842,v1:172.21.15.67:6827/873395842] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:34:43.456 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:34:43.456 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:34:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.454+0000 7f9d4da2d700 1 -- 172.21.15.32:0/277345749 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f9d300692a0 msgr2=0x7f9d3006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.454+0000 7f9d4da2d700 1 --2- 172.21.15.32:0/277345749 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f9d300692a0 0x7f9d3006b750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c00b3b0 tx=0x7f9d3c009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.454+0000 7f9d4da2d700 1 -- 172.21.15.32:0/277345749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d48066260 msgr2=0x7f9d4810d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.454+0000 7f9d4da2d700 1 --2- 172.21.15.32:0/277345749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d48066260 0x7f9d4810d960 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9d38002a80 tx=0x7f9d38004600 comp rx=0 tx=0).stop 2024-04-12T05:34:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.454+0000 7f9d4da2d700 1 -- 172.21.15.32:0/277345749 shutdown_connections 2024-04-12T05:34:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.454+0000 7f9d4da2d700 1 --2- 172.21.15.32:0/277345749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d48066260 0x7f9d4810d960 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.454+0000 7f9d4da2d700 1 --2- 172.21.15.32:0/277345749 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f9d300692a0 0x7f9d3006b750 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.454+0000 7f9d4da2d700 1 --2- 172.21.15.32:0/277345749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d480fff80 0x7f9d4810dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.454+0000 7f9d4da2d700 1 -- 172.21.15.32:0/277345749 >> 172.21.15.32:0/277345749 conn(0x7f9d480f7430 msgr2=0x7f9d480f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.454+0000 7f9d4da2d700 1 -- 172.21.15.32:0/277345749 shutdown_connections 2024-04-12T05:34:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:43.454+0000 7f9d4da2d700 1 -- 172.21.15.32:0/277345749 wait complete. 2024-04-12T05:34:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 18 2024-04-12T05:34:44.103 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-04-12T05:34:44.422 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:34:44.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:44 smithi067 ceph-mon[49010]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-12T05:34:44.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:44 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/277345749' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:34:44.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:44 smithi032 ceph-mon[39731]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-12T05:34:44.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:44 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/277345749' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:34:45.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:45 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:45.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:45 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:45.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:45 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:34:45.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:45 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:34:45.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:45 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:45.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:45 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:34:45.831 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:45 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:45.831 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:45 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:45.831 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:45 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:34:45.831 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:45 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:34:45.831 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:45 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:45.831 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:45 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:34:46.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:46 smithi032 ceph-mon[39731]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.0 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-12T05:34:46.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:46 smithi067 ceph-mon[49010]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.0 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-12T05:34:46.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.598+0000 7f10ae06c700 1 -- 172.21.15.32:0/2989568977 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10a80fbcd0 msgr2=0x7f10a80fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:46.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.598+0000 7f10ae06c700 1 --2- 172.21.15.32:0/2989568977 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10a80fbcd0 0x7f10a80fc0e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f10a0009ce0 tx=0x7f10a0004d10 comp rx=0 tx=0).stop 2024-04-12T05:34:46.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.598+0000 7f10ae06c700 1 -- 172.21.15.32:0/2989568977 shutdown_connections 2024-04-12T05:34:46.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.598+0000 7f10ae06c700 1 --2- 172.21.15.32:0/2989568977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10a80fcf80 0x7f10a80fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:46.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.598+0000 7f10ae06c700 1 --2- 172.21.15.32:0/2989568977 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10a80fbcd0 0x7f10a80fc0e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:46.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.598+0000 7f10ae06c700 1 -- 172.21.15.32:0/2989568977 >> 172.21.15.32:0/2989568977 conn(0x7f10a80f7430 msgr2=0x7f10a80f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:46.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.598+0000 7f10ae06c700 1 -- 172.21.15.32:0/2989568977 shutdown_connections 2024-04-12T05:34:46.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.598+0000 7f10ae06c700 1 -- 172.21.15.32:0/2989568977 wait complete. 2024-04-12T05:34:46.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.598+0000 7f10ae06c700 1 Processor -- start 2024-04-12T05:34:46.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.599+0000 7f10ae06c700 1 -- start start 2024-04-12T05:34:46.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.599+0000 7f10ae06c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10a80fbcd0 0x7f10a80ffda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:46.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.599+0000 7f10ae06c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10a80fcf80 0x7f10a80fe2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:46.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.599+0000 7f10ae06c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a80fe850 con 0x7f10a80fcf80 2024-04-12T05:34:46.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.599+0000 7f10ae06c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a80fe990 con 0x7f10a80fbcd0 2024-04-12T05:34:46.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.599+0000 7f10a6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10a80fcf80 0x7f10a80fe2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:46.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.599+0000 7f10a6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10a80fcf80 0x7f10a80fe2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37160/0 (socket says 172.21.15.32:37160) 2024-04-12T05:34:46.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.599+0000 7f10a6ffd700 1 -- 172.21.15.32:0/2014138459 learned_addr learned my addr 172.21.15.32:0/2014138459 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:34:46.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.599+0000 7f10a77fe700 1 --2- 172.21.15.32:0/2014138459 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10a80fbcd0 0x7f10a80ffda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:46.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.599+0000 7f10a6ffd700 1 -- 172.21.15.32:0/2014138459 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10a80fbcd0 msgr2=0x7f10a80ffda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:46.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.599+0000 7f10a6ffd700 1 --2- 172.21.15.32:0/2014138459 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10a80fbcd0 0x7f10a80ffda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:46.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.599+0000 7f10a6ffd700 1 -- 172.21.15.32:0/2014138459 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a0009940 con 0x7f10a80fcf80 2024-04-12T05:34:46.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.599+0000 7f10a6ffd700 1 --2- 172.21.15.32:0/2014138459 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10a80fcf80 0x7f10a80fe2c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f109000eb40 tx=0x7f109000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:46.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.600+0000 7f109ffff700 1 -- 172.21.15.32:0/2014138459 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1090007cc0 con 0x7f10a80fcf80 2024-04-12T05:34:46.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.600+0000 7f10ae06c700 1 -- 172.21.15.32:0/2014138459 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10a80fec10 con 0x7f10a80fcf80 2024-04-12T05:34:46.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.600+0000 7f109ffff700 1 -- 172.21.15.32:0/2014138459 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f109000f040 con 0x7f10a80fcf80 2024-04-12T05:34:46.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.600+0000 7f109ffff700 1 -- 172.21.15.32:0/2014138459 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f109000c990 con 0x7f10a80fcf80 2024-04-12T05:34:46.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.600+0000 7f10ae06c700 1 -- 172.21.15.32:0/2014138459 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10a81159b0 con 0x7f10a80fcf80 2024-04-12T05:34:46.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.600+0000 7f10ae06c700 1 -- 172.21.15.32:0/2014138459 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10a8101b10 con 0x7f10a80fcf80 2024-04-12T05:34:46.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.602+0000 7f109ffff700 1 -- 172.21.15.32:0/2014138459 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f109000cc00 con 0x7f10a80fcf80 2024-04-12T05:34:46.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.602+0000 7f109ffff700 1 --2- 172.21.15.32:0/2014138459 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f10940692a0 0x7f109406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:46.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.602+0000 7f109ffff700 1 -- 172.21.15.32:0/2014138459 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f1090086ff0 con 0x7f10a80fcf80 2024-04-12T05:34:46.977 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:34:46.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.602+0000 7f10a77fe700 1 --2- 172.21.15.32:0/2014138459 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f10940692a0 0x7f109406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:46.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.602+0000 7f10a77fe700 1 --2- 172.21.15.32:0/2014138459 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f10940692a0 0x7f109406b750 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f10a00093c0 tx=0x7f10a0004d10 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:46.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.605+0000 7f109ffff700 1 -- 172.21.15.32:0/2014138459 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f1090053380 con 0x7f10a80fcf80 2024-04-12T05:34:46.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.858+0000 7f10ae06c700 1 -- 172.21.15.32:0/2014138459 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10a805f5f0 con 0x7f10a80fcf80 2024-04-12T05:34:46.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.859+0000 7f109ffff700 1 -- 172.21.15.32:0/2014138459 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4713 (secure 0 0 0) 0x7f10900569a0 con 0x7f10a80fcf80 2024-04-12T05:34:46.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.861+0000 7f10ae06c700 1 -- 172.21.15.32:0/2014138459 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f10940692a0 msgr2=0x7f109406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:46.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.861+0000 7f10ae06c700 1 --2- 172.21.15.32:0/2014138459 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f10940692a0 0x7f109406b750 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f10a00093c0 tx=0x7f10a0004d10 comp rx=0 tx=0).stop 2024-04-12T05:34:46.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.861+0000 7f10ae06c700 1 -- 172.21.15.32:0/2014138459 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10a80fcf80 msgr2=0x7f10a80fe2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:46.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.861+0000 7f10ae06c700 1 --2- 172.21.15.32:0/2014138459 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10a80fcf80 0x7f10a80fe2c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f109000eb40 tx=0x7f109000b040 comp rx=0 tx=0).stop 2024-04-12T05:34:46.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.861+0000 7f10ae06c700 1 -- 172.21.15.32:0/2014138459 shutdown_connections 2024-04-12T05:34:46.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.861+0000 7f10ae06c700 1 --2- 172.21.15.32:0/2014138459 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f10940692a0 0x7f109406b750 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:46.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.861+0000 7f10ae06c700 1 --2- 172.21.15.32:0/2014138459 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10a80fbcd0 0x7f10a80ffda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:46.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.861+0000 7f10ae06c700 1 --2- 172.21.15.32:0/2014138459 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10a80fcf80 0x7f10a80fe2c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:46.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.861+0000 7f10ae06c700 1 -- 172.21.15.32:0/2014138459 >> 172.21.15.32:0/2014138459 conn(0x7f10a80f7430 msgr2=0x7f10a8065c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:46.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.861+0000 7f10ae06c700 1 -- 172.21.15.32:0/2014138459 shutdown_connections 2024-04-12T05:34:46.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:46.861+0000 7f10ae06c700 1 -- 172.21.15.32:0/2014138459 wait complete. 2024-04-12T05:34:46.979 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 18 2024-04-12T05:34:47.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:47 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/2014138459' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T05:34:47.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:47 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:47.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:47 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:47.662 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-04-12T05:34:47.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:47 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/2014138459' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T05:34:47.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:47 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:47.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:47 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:48.475 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:34:48.725 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:48 smithi032 ceph-mon[39731]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 7.1 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-12T05:34:48.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:48 smithi067 ceph-mon[49010]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 7.1 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-12T05:34:49.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:49 smithi067 ceph-mon[49010]: pgmap v196: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.9 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-12T05:34:49.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:49 smithi032 ceph-mon[39731]: pgmap v196: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.9 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-12T05:34:51.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.506+0000 7fca9983c700 1 -- 172.21.15.32:0/901450962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca940fe0d0 msgr2=0x7fca940fe540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:51.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.506+0000 7fca9983c700 1 --2- 172.21.15.32:0/901450962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca940fe0d0 0x7fca940fe540 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fca88004940 tx=0x7fca88009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:51.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.506+0000 7fca9983c700 1 -- 172.21.15.32:0/901450962 shutdown_connections 2024-04-12T05:34:51.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.506+0000 7fca9983c700 1 --2- 172.21.15.32:0/901450962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca940fe0d0 0x7fca940fe540 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:51.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.506+0000 7fca9983c700 1 --2- 172.21.15.32:0/901450962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca940fcee0 0x7fca940fd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:51.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.506+0000 7fca9983c700 1 -- 172.21.15.32:0/901450962 >> 172.21.15.32:0/901450962 conn(0x7fca940f8560 msgr2=0x7fca940fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:51.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.506+0000 7fca9983c700 1 -- 172.21.15.32:0/901450962 shutdown_connections 2024-04-12T05:34:51.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.506+0000 7fca9983c700 1 -- 172.21.15.32:0/901450962 wait complete. 2024-04-12T05:34:51.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.507+0000 7fca9983c700 1 Processor -- start 2024-04-12T05:34:51.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.507+0000 7fca9983c700 1 -- start start 2024-04-12T05:34:51.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.507+0000 7fca9983c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca940fcee0 0x7fca9410fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:51.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.507+0000 7fca9983c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca940fe0d0 0x7fca941100a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:51.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.507+0000 7fca9983c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca941106a0 con 0x7fca940fe0d0 2024-04-12T05:34:51.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.507+0000 7fca9983c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca94110e80 con 0x7fca940fcee0 2024-04-12T05:34:51.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.507+0000 7fca927fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca940fe0d0 0x7fca941100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:51.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.507+0000 7fca927fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca940fe0d0 0x7fca941100a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43598/0 (socket says 172.21.15.32:43598) 2024-04-12T05:34:51.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.507+0000 7fca927fc700 1 -- 172.21.15.32:0/3784217233 learned_addr learned my addr 172.21.15.32:0/3784217233 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:34:51.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.507+0000 7fca92ffd700 1 --2- 172.21.15.32:0/3784217233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca940fcee0 0x7fca9410fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:51.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.507+0000 7fca927fc700 1 -- 172.21.15.32:0/3784217233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca940fcee0 msgr2=0x7fca9410fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:51.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.507+0000 7fca927fc700 1 --2- 172.21.15.32:0/3784217233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca940fcee0 0x7fca9410fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:51.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.507+0000 7fca927fc700 1 -- 172.21.15.32:0/3784217233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca88005040 con 0x7fca940fe0d0 2024-04-12T05:34:51.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.507+0000 7fca927fc700 1 --2- 172.21.15.32:0/3784217233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca940fe0d0 0x7fca941100a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fca88009380 tx=0x7fca88009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:51.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.507+0000 7fca7b7fe700 1 -- 172.21.15.32:0/3784217233 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca8800e040 con 0x7fca940fe0d0 2024-04-12T05:34:51.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.508+0000 7fca9983c700 1 -- 172.21.15.32:0/3784217233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca94111100 con 0x7fca940fe0d0 2024-04-12T05:34:51.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.508+0000 7fca7b7fe700 1 -- 172.21.15.32:0/3784217233 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca8800bc50 con 0x7fca940fe0d0 2024-04-12T05:34:51.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.508+0000 7fca7b7fe700 1 -- 172.21.15.32:0/3784217233 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca8801c630 con 0x7fca940fe0d0 2024-04-12T05:34:51.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.508+0000 7fca9983c700 1 -- 172.21.15.32:0/3784217233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca94111640 con 0x7fca940fe0d0 2024-04-12T05:34:51.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.509+0000 7fca9983c700 1 -- 172.21.15.32:0/3784217233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca94101a60 con 0x7fca940fe0d0 2024-04-12T05:34:51.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.509+0000 7fca7b7fe700 1 -- 172.21.15.32:0/3784217233 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fca88013420 con 0x7fca940fe0d0 2024-04-12T05:34:51.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.509+0000 7fca7b7fe700 1 --2- 172.21.15.32:0/3784217233 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fca7c0692a0 0x7fca7c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:51.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.509+0000 7fca7b7fe700 1 -- 172.21.15.32:0/3784217233 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fca88087f30 con 0x7fca940fe0d0 2024-04-12T05:34:51.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.509+0000 7fca92ffd700 1 --2- 172.21.15.32:0/3784217233 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fca7c0692a0 0x7fca7c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:51.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.510+0000 7fca92ffd700 1 --2- 172.21.15.32:0/3784217233 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fca7c0692a0 0x7fca7c06b750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fca84009fd0 tx=0x7fca8400b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:51.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.514+0000 7fca7b7fe700 1 -- 172.21.15.32:0/3784217233 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fca880ba190 con 0x7fca940fe0d0 2024-04-12T05:34:51.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.794+0000 7fca9983c700 1 -- 172.21.15.32:0/3784217233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7fca9405f5f0 con 0x7fca940fe0d0 2024-04-12T05:34:51.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.794+0000 7fca7b7fe700 1 -- 172.21.15.32:0/3784217233 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v18) v1 ==== 78+0+99 (secure 0 0 0) 0x7fca8805be60 con 0x7fca940fe0d0 2024-04-12T05:34:51.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.796+0000 7fca9983c700 1 -- 172.21.15.32:0/3784217233 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fca7c0692a0 msgr2=0x7fca7c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:51.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.796+0000 7fca9983c700 1 --2- 172.21.15.32:0/3784217233 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fca7c0692a0 0x7fca7c06b750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fca84009fd0 tx=0x7fca8400b040 comp rx=0 tx=0).stop 2024-04-12T05:34:51.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.796+0000 7fca9983c700 1 -- 172.21.15.32:0/3784217233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca940fe0d0 msgr2=0x7fca941100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:51.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.796+0000 7fca9983c700 1 --2- 172.21.15.32:0/3784217233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca940fe0d0 0x7fca941100a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fca88009380 tx=0x7fca88009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:51.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.796+0000 7fca9983c700 1 -- 172.21.15.32:0/3784217233 shutdown_connections 2024-04-12T05:34:51.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.796+0000 7fca9983c700 1 --2- 172.21.15.32:0/3784217233 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fca7c0692a0 0x7fca7c06b750 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:51.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.796+0000 7fca9983c700 1 --2- 172.21.15.32:0/3784217233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca940fcee0 0x7fca9410fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:51.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.796+0000 7fca9983c700 1 --2- 172.21.15.32:0/3784217233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca940fe0d0 0x7fca941100a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:51.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.796+0000 7fca9983c700 1 -- 172.21.15.32:0/3784217233 >> 172.21.15.32:0/3784217233 conn(0x7fca940f8560 msgr2=0x7fca940f9d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:51.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.797+0000 7fca9983c700 1 -- 172.21.15.32:0/3784217233 shutdown_connections 2024-04-12T05:34:51.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:51.797+0000 7fca9983c700 1 -- 172.21.15.32:0/3784217233 wait complete. 2024-04-12T05:34:51.806 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:34:52.497 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:52 smithi032 ceph-mon[39731]: pgmap v197: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.7 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-12T05:34:52.497 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:52 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/3784217233' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-12T05:34:52.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:52 smithi067 ceph-mon[49010]: pgmap v197: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.7 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-12T05:34:52.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:52 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/3784217233' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-12T05:34:52.741 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-04-12T05:34:52.754 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-12T05:34:52.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.902+0000 7fb177811700 1 -- 172.21.15.32:0/52191656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb170071e40 msgr2=0x7fb170070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:52.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.902+0000 7fb177811700 1 --2- 172.21.15.32:0/52191656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb170071e40 0x7fb170070350 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb16c004940 tx=0x7fb16c009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:52.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.902+0000 7fb177811700 1 -- 172.21.15.32:0/52191656 shutdown_connections 2024-04-12T05:34:52.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.902+0000 7fb177811700 1 --2- 172.21.15.32:0/52191656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb170070890 0x7fb170070d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:52.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.902+0000 7fb177811700 1 --2- 172.21.15.32:0/52191656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb170071e40 0x7fb170070350 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:52.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.902+0000 7fb177811700 1 -- 172.21.15.32:0/52191656 >> 172.21.15.32:0/52191656 conn(0x7fb1700f8470 msgr2=0x7fb1700fa880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:52.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.902+0000 7fb177811700 1 -- 172.21.15.32:0/52191656 shutdown_connections 2024-04-12T05:34:52.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.902+0000 7fb177811700 1 -- 172.21.15.32:0/52191656 wait complete. 2024-04-12T05:34:52.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.902+0000 7fb177811700 1 Processor -- start 2024-04-12T05:34:52.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.903+0000 7fb177811700 1 -- start start 2024-04-12T05:34:52.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.903+0000 7fb177811700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb170070890 0x7fb17010f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:52.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.903+0000 7fb177811700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb170071e40 0x7fb17010feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:52.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.903+0000 7fb177811700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1701104b0 con 0x7fb170071e40 2024-04-12T05:34:52.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.903+0000 7fb177811700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb170110620 con 0x7fb170070890 2024-04-12T05:34:52.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.903+0000 7fb174dac700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb170071e40 0x7fb17010feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:52.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.903+0000 7fb174dac700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb170071e40 0x7fb17010feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43608/0 (socket says 172.21.15.32:43608) 2024-04-12T05:34:52.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.903+0000 7fb174dac700 1 -- 172.21.15.32:0/3251591769 learned_addr learned my addr 172.21.15.32:0/3251591769 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:34:52.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.903+0000 7fb1755ad700 1 --2- 172.21.15.32:0/3251591769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb170070890 0x7fb17010f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:52.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.903+0000 7fb174dac700 1 -- 172.21.15.32:0/3251591769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb170070890 msgr2=0x7fb17010f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:52.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.903+0000 7fb174dac700 1 --2- 172.21.15.32:0/3251591769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb170070890 0x7fb17010f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:52.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.903+0000 7fb174dac700 1 -- 172.21.15.32:0/3251591769 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb16c005040 con 0x7fb170071e40 2024-04-12T05:34:52.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.903+0000 7fb174dac700 1 --2- 172.21.15.32:0/3251591769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb170071e40 0x7fb17010feb0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb160002a50 tx=0x7fb1600046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:52.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.903+0000 7fb165ffb700 1 -- 172.21.15.32:0/3251591769 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb160007cc0 con 0x7fb170071e40 2024-04-12T05:34:52.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.903+0000 7fb177811700 1 -- 172.21.15.32:0/3251591769 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb170115080 con 0x7fb170071e40 2024-04-12T05:34:52.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.903+0000 7fb165ffb700 1 -- 172.21.15.32:0/3251591769 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb16000f040 con 0x7fb170071e40 2024-04-12T05:34:52.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.903+0000 7fb165ffb700 1 -- 172.21.15.32:0/3251591769 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb16000d9c0 con 0x7fb170071e40 2024-04-12T05:34:52.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.904+0000 7fb177811700 1 -- 172.21.15.32:0/3251591769 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb170115640 con 0x7fb170071e40 2024-04-12T05:34:52.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.905+0000 7fb177811700 1 -- 172.21.15.32:0/3251591769 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb170064970 con 0x7fb170071e40 2024-04-12T05:34:52.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.905+0000 7fb165ffb700 1 -- 172.21.15.32:0/3251591769 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fb16000db20 con 0x7fb170071e40 2024-04-12T05:34:52.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.905+0000 7fb165ffb700 1 --2- 172.21.15.32:0/3251591769 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fb15c0692a0 0x7fb15c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:52.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.906+0000 7fb165ffb700 1 -- 172.21.15.32:0/3251591769 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fb160086e30 con 0x7fb170071e40 2024-04-12T05:34:52.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.906+0000 7fb1755ad700 1 --2- 172.21.15.32:0/3251591769 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fb15c0692a0 0x7fb15c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:52.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.906+0000 7fb1755ad700 1 --2- 172.21.15.32:0/3251591769 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fb15c0692a0 0x7fb15c06b750 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb16c000c00 tx=0x7fb16c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:52.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:52.910+0000 7fb165ffb700 1 -- 172.21.15.32:0/3251591769 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fb160056ba0 con 0x7fb170071e40 2024-04-12T05:34:53.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.119+0000 7fb177811700 1 -- 172.21.15.32:0/3251591769 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb170115ac0 con 0x7fb170071e40 2024-04-12T05:34:53.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.120+0000 7fb165ffb700 1 -- 172.21.15.32:0/3251591769 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+14346 (secure 0 0 0) 0x7fb160056730 con 0x7fb170071e40 2024-04-12T05:34:53.119 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:34:53.119 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":46,"fsid":"37b58ce2-f88d-11ee-bc8c-c7b262605968","created":"2024-04-12T05:27:07.779407+0000","modified":"2024-04-12T05:34:37.113446+0000","last_up_change":"2024-04-12T05:32:59.508815+0000","last_in_change":"2024-04-12T05:32:39.814568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T05:31:39.568876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-12T05:34:21.574210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-12T05:34:22.056623+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c1f69d2a-5e37-47b8-b68f-0568af894b91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":3061707028},{"type":"v1","addr":"172.21.15.32:6803","nonce":3061707028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":3061707028},{"type":"v1","addr":"172.21.15.32:6805","nonce":3061707028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3061707028},{"type":"v1","addr":"172.21.15.32:6809","nonce":3061707028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":3061707028},{"type":"v1","addr":"172.21.15.32:6807","nonce":3061707028}]},"public_addr":"172.21.15.32:6803/3061707028","cluster_addr":"172.21.15.32:6805/3061707028","heartbeat_back_addr":"172.21.15.32:6809/3061707028","heartbeat_front_addr":"172.21.15.32:6807/3061707028","state":["exists","up"]},{"osd":1,"uuid":"86b93557-6e6f-4b84-a5f5-5bce4aa65a32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":164843120},{"type":"v1","addr":"172.21.15.32:6811","nonce":164843120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":164843120},{"type":"v1","addr":"172.21.15.32:6813","nonce":164843120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":164843120},{"type":"v1","addr":"172.21.15.32:6817","nonce":164843120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":164843120},{"type":"v1","addr":"172.21.15.32:6815","nonce":164843120}]},"public_addr":"172.21.15.32:6811/164843120","cluster_addr":"172.21.15.32:6813/164843120","heartbeat_back_addr":"172.21.15.32:6817/164843120","heartbeat_front_addr":"172.21.15.32:6815/164843120","state":["exists","up"]},{"osd":2,"uuid":"d1556613-e0de-4b0e-bb7d-e90361ec9007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":2698911336},{"type":"v1","addr":"172.21.15.32:6819","nonce":2698911336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":2698911336},{"type":"v1","addr":"172.21.15.32:6821","nonce":2698911336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":2698911336},{"type":"v1","addr":"172.21.15.32:6825","nonce":2698911336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":2698911336},{"type":"v1","addr":"172.21.15.32:6823","nonce":2698911336}]},"public_addr":"172.21.15.32:6819/2698911336","cluster_addr":"172.21.15.32:6821/2698911336","heartbeat_back_addr":"172.21.15.32:6825/2698911336","heartbeat_front_addr":"172.21.15.32:6823/2698911336","state":["exists","up"]},{"osd":3,"uuid":"213dc62e-a894-4bc7-a2dd-ca98d1313d6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1205580917},{"type":"v1","addr":"172.21.15.67:6801","nonce":1205580917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1205580917},{"type":"v1","addr":"172.21.15.67:6803","nonce":1205580917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1205580917},{"type":"v1","addr":"172.21.15.67:6807","nonce":1205580917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1205580917},{"type":"v1","addr":"172.21.15.67:6805","nonce":1205580917}]},"public_addr":"172.21.15.67:6801/1205580917","cluster_addr":"172.21.15.67:6803/1205580917","heartbeat_back_addr":"172.21.15.67:6807/1205580917","heartbeat_front_addr":"172.21.15.67:6805/1205580917","state":["exists","up"]},{"osd":4,"uuid":"8a0ff860-fff7-434a-9e78-f2020bc84966","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":286946785},{"type":"v1","addr":"172.21.15.67:6809","nonce":286946785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":286946785},{"type":"v1","addr":"172.21.15.67:6811","nonce":286946785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":286946785},{"type":"v1","addr":"172.21.15.67:6815","nonce":286946785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":286946785},{"type":"v1","addr":"172.21.15.67:6813","nonce":286946785}]},"public_addr":"172.21.15.67:6809/286946785","cluster_addr":"172.21.15.67:6811/286946785","heartbeat_back_addr":"172.21.15.67:6815/286946785","heartbeat_front_addr":"172.21.15.67:6813/286946785","state":["exists","up"]},{"osd":5,"uuid":"3486e5c9-9f44-4f32-a186-fc6198f09549","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1361296892},{"type":"v1","addr":"172.21.15.67:6817","nonce":1361296892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":1361296892},{"type":"v1","addr":"172.21.15.67:6819","nonce":1361296892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":1361296892},{"type":"v1","addr":"172.21.15.67:6823","nonce":1361296892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1361296892},{"type":"v1","addr":"172.21.15.67:6821","nonce":1361296892}]},"public_addr":"172.21.15.67:6817/1361296892","cluster_addr":"172.21.15.67:6819/1361296892","heartbeat_back_addr":"172.21.15.67:6823/1361296892","heartbeat_front_addr":"172.21.15.67:6821/1361296892","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:30:38.748993+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:31:08.677208+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:31:36.979481+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:32:02.668398+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:32:30.143788+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:32:57.613087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/4031244002":"2024-04-13T05:27:40.952118+0000","172.21.15.32:6800/3971961003":"2024-04-13T05:29:25.054120+0000","172.21.15.32:6800/656201914":"2024-04-13T05:27:40.952118+0000","172.21.15.32:0/1643708423":"2024-04-13T05:27:40.952118+0000","172.21.15.32:0/138657961":"2024-04-13T05:27:40.952118+0000","172.21.15.32:0/795906028":"2024-04-13T05:29:25.054120+0000","172.21.15.32:6827/2962187015":"2024-04-13T05:34:37.113413+0000","172.21.15.32:6801/1962278696":"2024-04-13T05:28:14.470413+0000","172.21.15.32:6800/1962278696":"2024-04-13T05:28:14.470413+0000","172.21.15.32:0/2162993076":"2024-04-13T05:28:14.470413+0000","172.21.15.32:6826/2962187015":"2024-04-13T05:34:37.113413+0000","172.21.15.32:6801/656201914":"2024-04-13T05:27:40.952118+0000","172.21.15.32:0/3082743475":"2024-04-13T05:28:14.470413+0000","172.21.15.32:0/146872823":"2024-04-13T05:28:14.470413+0000","172.21.15.32:0/674029871":"2024-04-13T05:29:25.054120+0000","172.21.15.32:0/4112976740":"2024-04-13T05:29:25.054120+0000","172.21.15.32:6801/3971961003":"2024-04-13T05:29:25.054120+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T05:34:53.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.122+0000 7fb177811700 1 -- 172.21.15.32:0/3251591769 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fb15c0692a0 msgr2=0x7fb15c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:53.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.122+0000 7fb177811700 1 --2- 172.21.15.32:0/3251591769 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fb15c0692a0 0x7fb15c06b750 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb16c000c00 tx=0x7fb16c009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:53.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.122+0000 7fb177811700 1 -- 172.21.15.32:0/3251591769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb170071e40 msgr2=0x7fb17010feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:53.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.122+0000 7fb177811700 1 --2- 172.21.15.32:0/3251591769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb170071e40 0x7fb17010feb0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb160002a50 tx=0x7fb1600046b0 comp rx=0 tx=0).stop 2024-04-12T05:34:53.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.122+0000 7fb177811700 1 -- 172.21.15.32:0/3251591769 shutdown_connections 2024-04-12T05:34:53.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.122+0000 7fb177811700 1 --2- 172.21.15.32:0/3251591769 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fb15c0692a0 0x7fb15c06b750 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:53.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.122+0000 7fb177811700 1 --2- 172.21.15.32:0/3251591769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb170070890 0x7fb17010f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:53.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.122+0000 7fb177811700 1 --2- 172.21.15.32:0/3251591769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb170071e40 0x7fb17010feb0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:53.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.122+0000 7fb177811700 1 -- 172.21.15.32:0/3251591769 >> 172.21.15.32:0/3251591769 conn(0x7fb1700f8470 msgr2=0x7fb1700f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:53.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.122+0000 7fb177811700 1 -- 172.21.15.32:0/3251591769 shutdown_connections 2024-04-12T05:34:53.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.122+0000 7fb177811700 1 -- 172.21.15.32:0/3251591769 wait complete. 2024-04-12T05:34:53.134 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-12T05:31:39.568876+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-12T05:34:21.574210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-12T05:34:22.056623+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-12T05:34:53.134 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-12T05:34:53.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.261+0000 7f4447442700 1 -- 172.21.15.32:0/505065474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44400fbba0 msgr2=0x7f44400fbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:53.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.261+0000 7f4447442700 1 --2- 172.21.15.32:0/505065474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44400fbba0 0x7f44400fbfb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f443c004940 tx=0x7f443c009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:53.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.261+0000 7f4447442700 1 -- 172.21.15.32:0/505065474 shutdown_connections 2024-04-12T05:34:53.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.261+0000 7f4447442700 1 --2- 172.21.15.32:0/505065474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44400fce50 0x7f44400fd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:53.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.261+0000 7f4447442700 1 --2- 172.21.15.32:0/505065474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44400fbba0 0x7f44400fbfb0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:53.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.261+0000 7f4447442700 1 -- 172.21.15.32:0/505065474 >> 172.21.15.32:0/505065474 conn(0x7f44400f7300 msgr2=0x7f44400f9730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:53.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.261+0000 7f4447442700 1 -- 172.21.15.32:0/505065474 shutdown_connections 2024-04-12T05:34:53.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.261+0000 7f4447442700 1 -- 172.21.15.32:0/505065474 wait complete. 2024-04-12T05:34:53.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.262+0000 7f4447442700 1 Processor -- start 2024-04-12T05:34:53.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.262+0000 7f4447442700 1 -- start start 2024-04-12T05:34:53.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.262+0000 7f4447442700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44400fbba0 0x7f4440065840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:53.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.262+0000 7f4447442700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44400fce50 0x7f4440112530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:53.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.262+0000 7f4447442700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4440112a70 con 0x7f44400fbba0 2024-04-12T05:34:53.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.262+0000 7f4447442700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4440112bb0 con 0x7f44400fce50 2024-04-12T05:34:53.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.262+0000 7f44451de700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44400fbba0 0x7f4440065840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:53.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.262+0000 7f44451de700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44400fbba0 0x7f4440065840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43624/0 (socket says 172.21.15.32:43624) 2024-04-12T05:34:53.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.262+0000 7f44451de700 1 -- 172.21.15.32:0/1455403894 learned_addr learned my addr 172.21.15.32:0/1455403894 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:34:53.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.262+0000 7f44449dd700 1 --2- 172.21.15.32:0/1455403894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44400fce50 0x7f4440112530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:53.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.262+0000 7f44451de700 1 -- 172.21.15.32:0/1455403894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44400fce50 msgr2=0x7f4440112530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:53.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.262+0000 7f44451de700 1 --2- 172.21.15.32:0/1455403894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44400fce50 0x7f4440112530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:53.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.262+0000 7f44451de700 1 -- 172.21.15.32:0/1455403894 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f443c005040 con 0x7f44400fbba0 2024-04-12T05:34:53.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.262+0000 7f44449dd700 1 --2- 172.21.15.32:0/1455403894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44400fce50 0x7f4440112530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:34:53.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.263+0000 7f44451de700 1 --2- 172.21.15.32:0/1455403894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44400fbba0 0x7f4440065840 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f443c00b3b0 tx=0x7f443c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:53.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.263+0000 7f4431ffb700 1 -- 172.21.15.32:0/1455403894 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f443c00e040 con 0x7f44400fbba0 2024-04-12T05:34:53.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.263+0000 7f4431ffb700 1 -- 172.21.15.32:0/1455403894 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f443c00bd20 con 0x7f44400fbba0 2024-04-12T05:34:53.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.263+0000 7f4447442700 1 -- 172.21.15.32:0/1455403894 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4440112de0 con 0x7f44400fbba0 2024-04-12T05:34:53.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.263+0000 7f4431ffb700 1 -- 172.21.15.32:0/1455403894 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f443c01c630 con 0x7f44400fbba0 2024-04-12T05:34:53.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.263+0000 7f4447442700 1 -- 172.21.15.32:0/1455403894 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44401132e0 con 0x7f44400fbba0 2024-04-12T05:34:53.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.264+0000 7f4447442700 1 -- 172.21.15.32:0/1455403894 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4440064970 con 0x7f44400fbba0 2024-04-12T05:34:53.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.264+0000 7f4431ffb700 1 -- 172.21.15.32:0/1455403894 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f443c013420 con 0x7f44400fbba0 2024-04-12T05:34:53.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.265+0000 7f4431ffb700 1 --2- 172.21.15.32:0/1455403894 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f442c0692a0 0x7f442c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:53.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.265+0000 7f44449dd700 1 --2- 172.21.15.32:0/1455403894 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f442c0692a0 0x7f442c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:53.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.265+0000 7f4431ffb700 1 -- 172.21.15.32:0/1455403894 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f443c088040 con 0x7f44400fbba0 2024-04-12T05:34:53.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.265+0000 7f44449dd700 1 --2- 172.21.15.32:0/1455403894 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f442c0692a0 0x7f442c06b750 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4434007fd0 tx=0x7f4434009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:53.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.269+0000 7f4431ffb700 1 -- 172.21.15.32:0/1455403894 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f443c057d10 con 0x7f44400fbba0 2024-04-12T05:34:53.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.466+0000 7f4447442700 1 -- 172.21.15.32:0/1455403894 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4440113680 con 0x7f44400fbba0 2024-04-12T05:34:53.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.466+0000 7f4431ffb700 1 -- 172.21.15.32:0/1455403894 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7f443c0578a0 con 0x7f44400fbba0 2024-04-12T05:34:53.466 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-12T05:34:53.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.469+0000 7f4447442700 1 -- 172.21.15.32:0/1455403894 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f442c0692a0 msgr2=0x7f442c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:53.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.469+0000 7f4447442700 1 --2- 172.21.15.32:0/1455403894 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f442c0692a0 0x7f442c06b750 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4434007fd0 tx=0x7f4434009040 comp rx=0 tx=0).stop 2024-04-12T05:34:53.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.469+0000 7f4447442700 1 -- 172.21.15.32:0/1455403894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44400fbba0 msgr2=0x7f4440065840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:53.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.469+0000 7f4447442700 1 --2- 172.21.15.32:0/1455403894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44400fbba0 0x7f4440065840 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f443c00b3b0 tx=0x7f443c009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:53.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.469+0000 7f4447442700 1 -- 172.21.15.32:0/1455403894 shutdown_connections 2024-04-12T05:34:53.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.469+0000 7f4447442700 1 --2- 172.21.15.32:0/1455403894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44400fbba0 0x7f4440065840 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:53.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.469+0000 7f4447442700 1 --2- 172.21.15.32:0/1455403894 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f442c0692a0 0x7f442c06b750 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:53.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.469+0000 7f4447442700 1 --2- 172.21.15.32:0/1455403894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44400fce50 0x7f4440112530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:53.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.469+0000 7f4447442700 1 -- 172.21.15.32:0/1455403894 >> 172.21.15.32:0/1455403894 conn(0x7f44400f7300 msgr2=0x7f44400f8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:53.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.469+0000 7f4447442700 1 -- 172.21.15.32:0/1455403894 shutdown_connections 2024-04-12T05:34:53.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.469+0000 7f4447442700 1 -- 172.21.15.32:0/1455403894 wait complete. 2024-04-12T05:34:53.480 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-12T05:34:53.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.606+0000 7f4c6683b700 1 -- 172.21.15.32:0/1197827671 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c600668b0 msgr2=0x7f4c60101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:53.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.606+0000 7f4c6683b700 1 --2- 172.21.15.32:0/1197827671 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c600668b0 0x7f4c60101d20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4c50004940 tx=0x7f4c50009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:53.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.606+0000 7f4c6683b700 1 -- 172.21.15.32:0/1197827671 shutdown_connections 2024-04-12T05:34:53.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.606+0000 7f4c6683b700 1 --2- 172.21.15.32:0/1197827671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c60102260 0x7f4c60104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:53.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.606+0000 7f4c6683b700 1 --2- 172.21.15.32:0/1197827671 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c600668b0 0x7f4c60101d20 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:53.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.606+0000 7f4c6683b700 1 -- 172.21.15.32:0/1197827671 >> 172.21.15.32:0/1197827671 conn(0x7f4c600f7330 msgr2=0x7f4c600f9760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:53.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.606+0000 7f4c6683b700 1 -- 172.21.15.32:0/1197827671 shutdown_connections 2024-04-12T05:34:53.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.606+0000 7f4c6683b700 1 -- 172.21.15.32:0/1197827671 wait complete. 2024-04-12T05:34:53.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.607+0000 7f4c6683b700 1 Processor -- start 2024-04-12T05:34:53.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.607+0000 7f4c6683b700 1 -- start start 2024-04-12T05:34:53.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.607+0000 7f4c6683b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c600668b0 0x7f4c6010f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:53.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.607+0000 7f4c6683b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c60102260 0x7f4c6010fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:53.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.607+0000 7f4c6683b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c601104a0 con 0x7f4c60102260 2024-04-12T05:34:53.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.607+0000 7f4c6683b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c60110610 con 0x7f4c600668b0 2024-04-12T05:34:53.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.607+0000 7f4c5f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c60102260 0x7f4c6010fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:53.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.607+0000 7f4c5f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c60102260 0x7f4c6010fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43640/0 (socket says 172.21.15.32:43640) 2024-04-12T05:34:53.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.608+0000 7f4c5f7fe700 1 -- 172.21.15.32:0/1506689020 learned_addr learned my addr 172.21.15.32:0/1506689020 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:34:53.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.608+0000 7f4c5ffff700 1 --2- 172.21.15.32:0/1506689020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c600668b0 0x7f4c6010f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:53.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.608+0000 7f4c5f7fe700 1 -- 172.21.15.32:0/1506689020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c600668b0 msgr2=0x7f4c6010f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:53.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.608+0000 7f4c5f7fe700 1 --2- 172.21.15.32:0/1506689020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c600668b0 0x7f4c6010f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:53.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.608+0000 7f4c5f7fe700 1 -- 172.21.15.32:0/1506689020 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c50005040 con 0x7f4c60102260 2024-04-12T05:34:53.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.608+0000 7f4c5f7fe700 1 --2- 172.21.15.32:0/1506689020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c60102260 0x7f4c6010fea0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4c54002a50 tx=0x7f4c540046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:53.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.608+0000 7f4c5cff9700 1 -- 172.21.15.32:0/1506689020 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c54007cc0 con 0x7f4c60102260 2024-04-12T05:34:53.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.608+0000 7f4c5cff9700 1 -- 172.21.15.32:0/1506689020 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c5400f040 con 0x7f4c60102260 2024-04-12T05:34:53.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.608+0000 7f4c6683b700 1 -- 172.21.15.32:0/1506689020 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c60115070 con 0x7f4c60102260 2024-04-12T05:34:53.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.608+0000 7f4c5cff9700 1 -- 172.21.15.32:0/1506689020 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c5400d9c0 con 0x7f4c60102260 2024-04-12T05:34:53.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.608+0000 7f4c6683b700 1 -- 172.21.15.32:0/1506689020 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c60115630 con 0x7f4c60102260 2024-04-12T05:34:53.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.609+0000 7f4c6683b700 1 -- 172.21.15.32:0/1506689020 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c600640e0 con 0x7f4c60102260 2024-04-12T05:34:53.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.610+0000 7f4c5cff9700 1 -- 172.21.15.32:0/1506689020 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f4c5400db20 con 0x7f4c60102260 2024-04-12T05:34:53.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.610+0000 7f4c5cff9700 1 --2- 172.21.15.32:0/1506689020 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4c480692a0 0x7f4c4806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:53.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.610+0000 7f4c5cff9700 1 -- 172.21.15.32:0/1506689020 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f4c54086f00 con 0x7f4c60102260 2024-04-12T05:34:53.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.610+0000 7f4c5ffff700 1 --2- 172.21.15.32:0/1506689020 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4c480692a0 0x7f4c4806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:53.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.610+0000 7f4c5ffff700 1 --2- 172.21.15.32:0/1506689020 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4c480692a0 0x7f4c4806b750 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4c50000c00 tx=0x7f4c50009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:53.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.614+0000 7f4c5cff9700 1 -- 172.21.15.32:0/1506689020 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4c54056bf0 con 0x7f4c60102260 2024-04-12T05:34:53.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.811+0000 7f4c6683b700 1 -- 172.21.15.32:0/1506689020 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f4c60107870 con 0x7f4c60102260 2024-04-12T05:34:53.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.811+0000 7f4c5cff9700 1 -- 172.21.15.32:0/1506689020 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f4c54056780 con 0x7f4c60102260 2024-04-12T05:34:53.820 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-12T05:34:53.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.813+0000 7f4c6683b700 1 -- 172.21.15.32:0/1506689020 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4c480692a0 msgr2=0x7f4c4806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:53.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.813+0000 7f4c6683b700 1 --2- 172.21.15.32:0/1506689020 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4c480692a0 0x7f4c4806b750 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4c50000c00 tx=0x7f4c50009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:53.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.813+0000 7f4c6683b700 1 -- 172.21.15.32:0/1506689020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c60102260 msgr2=0x7f4c6010fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:53.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.813+0000 7f4c6683b700 1 --2- 172.21.15.32:0/1506689020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c60102260 0x7f4c6010fea0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4c54002a50 tx=0x7f4c540046b0 comp rx=0 tx=0).stop 2024-04-12T05:34:53.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.813+0000 7f4c6683b700 1 -- 172.21.15.32:0/1506689020 shutdown_connections 2024-04-12T05:34:53.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.813+0000 7f4c6683b700 1 --2- 172.21.15.32:0/1506689020 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4c480692a0 0x7f4c4806b750 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:53.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.813+0000 7f4c6683b700 1 --2- 172.21.15.32:0/1506689020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c600668b0 0x7f4c6010f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:53.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.813+0000 7f4c6683b700 1 --2- 172.21.15.32:0/1506689020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c60102260 0x7f4c6010fea0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:53.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.813+0000 7f4c6683b700 1 -- 172.21.15.32:0/1506689020 >> 172.21.15.32:0/1506689020 conn(0x7f4c600f7330 msgr2=0x7f4c600f7750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:53.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.814+0000 7f4c6683b700 1 -- 172.21.15.32:0/1506689020 shutdown_connections 2024-04-12T05:34:53.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.814+0000 7f4c6683b700 1 -- 172.21.15.32:0/1506689020 wait complete. 2024-04-12T05:34:53.825 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-12T05:34:53.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.960+0000 7f49de070700 1 -- 172.21.15.32:0/3994526426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d8066df0 msgr2=0x7f49d80ff920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:53.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.960+0000 7f49de070700 1 --2- 172.21.15.32:0/3994526426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d8066df0 0x7f49d80ff920 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f49c8004940 tx=0x7f49c8009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:53.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.961+0000 7f49de070700 1 -- 172.21.15.32:0/3994526426 shutdown_connections 2024-04-12T05:34:53.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.961+0000 7f49de070700 1 --2- 172.21.15.32:0/3994526426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49d80ffe60 0x7f49d8102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:53.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.961+0000 7f49de070700 1 --2- 172.21.15.32:0/3994526426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d8066df0 0x7f49d80ff920 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:53.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.961+0000 7f49de070700 1 -- 172.21.15.32:0/3994526426 >> 172.21.15.32:0/3994526426 conn(0x7f49d80f7310 msgr2=0x7f49d80f9740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:53.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.961+0000 7f49de070700 1 -- 172.21.15.32:0/3994526426 shutdown_connections 2024-04-12T05:34:53.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.961+0000 7f49de070700 1 -- 172.21.15.32:0/3994526426 wait complete. 2024-04-12T05:34:53.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.961+0000 7f49de070700 1 Processor -- start 2024-04-12T05:34:53.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.961+0000 7f49de070700 1 -- start start 2024-04-12T05:34:53.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.961+0000 7f49de070700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49d8066df0 0x7f49d810d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:53.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.961+0000 7f49de070700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d80ffe60 0x7f49d810dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:53.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.961+0000 7f49de070700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d810e290 con 0x7f49d80ffe60 2024-04-12T05:34:53.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.961+0000 7f49de070700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d810e400 con 0x7f49d8066df0 2024-04-12T05:34:53.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.962+0000 7f49d6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d80ffe60 0x7f49d810dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:53.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.962+0000 7f49d6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d80ffe60 0x7f49d810dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43658/0 (socket says 172.21.15.32:43658) 2024-04-12T05:34:53.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.962+0000 7f49d6ffd700 1 -- 172.21.15.32:0/1766618864 learned_addr learned my addr 172.21.15.32:0/1766618864 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:34:53.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.962+0000 7f49d77fe700 1 --2- 172.21.15.32:0/1766618864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49d8066df0 0x7f49d810d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:53.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.962+0000 7f49d6ffd700 1 -- 172.21.15.32:0/1766618864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49d8066df0 msgr2=0x7f49d810d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:53.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.962+0000 7f49d6ffd700 1 --2- 172.21.15.32:0/1766618864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49d8066df0 0x7f49d810d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:53.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.962+0000 7f49d6ffd700 1 -- 172.21.15.32:0/1766618864 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c8005040 con 0x7f49d80ffe60 2024-04-12T05:34:53.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.962+0000 7f49d6ffd700 1 --2- 172.21.15.32:0/1766618864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d80ffe60 0x7f49d810dc90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f49cc002a80 tx=0x7f49cc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:53.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.962+0000 7f49bffff700 1 -- 172.21.15.32:0/1766618864 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49cc007cc0 con 0x7f49d80ffe60 2024-04-12T05:34:53.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.962+0000 7f49bffff700 1 -- 172.21.15.32:0/1766618864 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f49cc00f040 con 0x7f49d80ffe60 2024-04-12T05:34:53.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.962+0000 7f49de070700 1 -- 172.21.15.32:0/1766618864 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d8112e10 con 0x7f49d80ffe60 2024-04-12T05:34:53.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.962+0000 7f49bffff700 1 -- 172.21.15.32:0/1766618864 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49cc00da60 con 0x7f49d80ffe60 2024-04-12T05:34:53.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.962+0000 7f49de070700 1 -- 172.21.15.32:0/1766618864 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d81133a0 con 0x7f49d80ffe60 2024-04-12T05:34:53.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.963+0000 7f49de070700 1 -- 172.21.15.32:0/1766618864 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49d8064dc0 con 0x7f49d80ffe60 2024-04-12T05:34:53.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.964+0000 7f49bffff700 1 -- 172.21.15.32:0/1766618864 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f49cc00dbc0 con 0x7f49d80ffe60 2024-04-12T05:34:53.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.964+0000 7f49bffff700 1 --2- 172.21.15.32:0/1766618864 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f49c00692a0 0x7f49c006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:53.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.964+0000 7f49bffff700 1 -- 172.21.15.32:0/1766618864 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f49cc086e70 con 0x7f49d80ffe60 2024-04-12T05:34:53.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.964+0000 7f49d77fe700 1 --2- 172.21.15.32:0/1766618864 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f49c00692a0 0x7f49c006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:53.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.964+0000 7f49d77fe700 1 --2- 172.21.15.32:0/1766618864 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f49c00692a0 0x7f49c006b750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f49c8000c00 tx=0x7f49c8009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:53.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:53.968+0000 7f49bffff700 1 -- 172.21.15.32:0/1766618864 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f49cc056b60 con 0x7f49d80ffe60 2024-04-12T05:34:54.092 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:53 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:54.092 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:53 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:54.092 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:53 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:34:54.092 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:53 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:34:54.092 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:53 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:54.092 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:53 smithi032 ceph-mon[39731]: pgmap v198: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-12T05:34:54.092 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:53 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/3251591769' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T05:34:54.092 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:53 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/1455403894' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T05:34:54.092 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:53 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/1506689020' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-12T05:34:54.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.167+0000 7f49de070700 1 -- 172.21.15.32:0/1766618864 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f49d81135c0 con 0x7f49d80ffe60 2024-04-12T05:34:54.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.168+0000 7f49bffff700 1 -- 172.21.15.32:0/1766618864 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f49cc0566f0 con 0x7f49d80ffe60 2024-04-12T05:34:54.167 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-12T05:34:54.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.170+0000 7f49de070700 1 -- 172.21.15.32:0/1766618864 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f49c00692a0 msgr2=0x7f49c006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:54.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.170+0000 7f49de070700 1 --2- 172.21.15.32:0/1766618864 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f49c00692a0 0x7f49c006b750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f49c8000c00 tx=0x7f49c8009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:54.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.170+0000 7f49de070700 1 -- 172.21.15.32:0/1766618864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d80ffe60 msgr2=0x7f49d810dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:54.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.170+0000 7f49de070700 1 --2- 172.21.15.32:0/1766618864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d80ffe60 0x7f49d810dc90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f49cc002a80 tx=0x7f49cc0046b0 comp rx=0 tx=0).stop 2024-04-12T05:34:54.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.170+0000 7f49de070700 1 -- 172.21.15.32:0/1766618864 shutdown_connections 2024-04-12T05:34:54.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.170+0000 7f49de070700 1 --2- 172.21.15.32:0/1766618864 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f49c00692a0 0x7f49c006b750 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:54.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.170+0000 7f49de070700 1 --2- 172.21.15.32:0/1766618864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49d8066df0 0x7f49d810d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:54.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.170+0000 7f49de070700 1 --2- 172.21.15.32:0/1766618864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d80ffe60 0x7f49d810dc90 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:54.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.170+0000 7f49de070700 1 -- 172.21.15.32:0/1766618864 >> 172.21.15.32:0/1766618864 conn(0x7f49d80f7310 msgr2=0x7f49d80f9740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:54.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.170+0000 7f49de070700 1 -- 172.21.15.32:0/1766618864 shutdown_connections 2024-04-12T05:34:54.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.170+0000 7f49de070700 1 -- 172.21.15.32:0/1766618864 wait complete. 2024-04-12T05:34:54.181 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-12T05:34:54.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:53 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:54.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:53 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:54.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:53 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:34:54.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:53 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:34:54.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:53 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:54.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:53 smithi067 ceph-mon[49010]: pgmap v198: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-12T05:34:54.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:53 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/3251591769' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T05:34:54.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:53 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/1455403894' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T05:34:54.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:53 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/1506689020' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-12T05:34:54.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.305+0000 7fc9b64d3700 1 -- 172.21.15.32:0/3777308550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9b0065aa0 msgr2=0x7fc9b00ff8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:54.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.305+0000 7fc9b64d3700 1 --2- 172.21.15.32:0/3777308550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9b0065aa0 0x7fc9b00ff8c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0004940 tx=0x7fc9a0009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:54.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.305+0000 7fc9b64d3700 1 -- 172.21.15.32:0/3777308550 shutdown_connections 2024-04-12T05:34:54.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.305+0000 7fc9b64d3700 1 --2- 172.21.15.32:0/3777308550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9b00ffe00 0x7fc9b0102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:54.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.305+0000 7fc9b64d3700 1 --2- 172.21.15.32:0/3777308550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9b0065aa0 0x7fc9b00ff8c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:54.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.305+0000 7fc9b64d3700 1 -- 172.21.15.32:0/3777308550 >> 172.21.15.32:0/3777308550 conn(0x7fc9b00f7310 msgr2=0x7fc9b00f9760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:54.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.305+0000 7fc9b64d3700 1 -- 172.21.15.32:0/3777308550 shutdown_connections 2024-04-12T05:34:54.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.305+0000 7fc9b64d3700 1 -- 172.21.15.32:0/3777308550 wait complete. 2024-04-12T05:34:54.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.306+0000 7fc9b64d3700 1 Processor -- start 2024-04-12T05:34:54.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.306+0000 7fc9b64d3700 1 -- start start 2024-04-12T05:34:54.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.306+0000 7fc9b64d3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9b0065aa0 0x7fc9b010d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:54.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.306+0000 7fc9b64d3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9b00ffe00 0x7fc9b010dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:54.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.306+0000 7fc9b64d3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b010e2b0 con 0x7fc9b00ffe00 2024-04-12T05:34:54.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.306+0000 7fc9b64d3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b010e420 con 0x7fc9b0065aa0 2024-04-12T05:34:54.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.306+0000 7fc9af7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9b00ffe00 0x7fc9b010dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:54.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.306+0000 7fc9af7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9b00ffe00 0x7fc9b010dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43678/0 (socket says 172.21.15.32:43678) 2024-04-12T05:34:54.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.306+0000 7fc9af7fe700 1 -- 172.21.15.32:0/4092669448 learned_addr learned my addr 172.21.15.32:0/4092669448 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:34:54.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.306+0000 7fc9affff700 1 --2- 172.21.15.32:0/4092669448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9b0065aa0 0x7fc9b010d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:54.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.307+0000 7fc9af7fe700 1 -- 172.21.15.32:0/4092669448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9b0065aa0 msgr2=0x7fc9b010d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:54.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.307+0000 7fc9af7fe700 1 --2- 172.21.15.32:0/4092669448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9b0065aa0 0x7fc9b010d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:54.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.307+0000 7fc9af7fe700 1 -- 172.21.15.32:0/4092669448 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a0005040 con 0x7fc9b00ffe00 2024-04-12T05:34:54.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.307+0000 7fc9af7fe700 1 --2- 172.21.15.32:0/4092669448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9b00ffe00 0x7fc9b010dcb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4002a50 tx=0x7fc9a40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:54.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.307+0000 7fc9acff9700 1 -- 172.21.15.32:0/4092669448 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9a4007cc0 con 0x7fc9b00ffe00 2024-04-12T05:34:54.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.307+0000 7fc9acff9700 1 -- 172.21.15.32:0/4092669448 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc9a400f040 con 0x7fc9b00ffe00 2024-04-12T05:34:54.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.307+0000 7fc9b64d3700 1 -- 172.21.15.32:0/4092669448 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9b0112e80 con 0x7fc9b00ffe00 2024-04-12T05:34:54.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.307+0000 7fc9acff9700 1 -- 172.21.15.32:0/4092669448 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9a400d9c0 con 0x7fc9b00ffe00 2024-04-12T05:34:54.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.307+0000 7fc9b64d3700 1 -- 172.21.15.32:0/4092669448 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9b0113410 con 0x7fc9b00ffe00 2024-04-12T05:34:54.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.308+0000 7fc9b64d3700 1 -- 172.21.15.32:0/4092669448 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9b0064970 con 0x7fc9b00ffe00 2024-04-12T05:34:54.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.308+0000 7fc9acff9700 1 -- 172.21.15.32:0/4092669448 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fc9a400db20 con 0x7fc9b00ffe00 2024-04-12T05:34:54.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.309+0000 7fc9acff9700 1 --2- 172.21.15.32:0/4092669448 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fc9980692a0 0x7fc99806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:54.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.309+0000 7fc9acff9700 1 -- 172.21.15.32:0/4092669448 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fc9a4086ed0 con 0x7fc9b00ffe00 2024-04-12T05:34:54.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.309+0000 7fc9affff700 1 --2- 172.21.15.32:0/4092669448 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fc9980692a0 0x7fc99806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:54.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.309+0000 7fc9affff700 1 --2- 172.21.15.32:0/4092669448 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fc9980692a0 0x7fc99806b750 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0004890 tx=0x7fc9a0009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:54.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.313+0000 7fc9acff9700 1 -- 172.21.15.32:0/4092669448 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc9a4056bc0 con 0x7fc9b00ffe00 2024-04-12T05:34:54.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.556+0000 7fc9b64d3700 1 -- 172.21.15.32:0/4092669448 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9b0113630 con 0x7fc9b00ffe00 2024-04-12T05:34:54.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.556+0000 7fc9acff9700 1 -- 172.21.15.32:0/4092669448 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4713 (secure 0 0 0) 0x7fc9a4056750 con 0x7fc9b00ffe00 2024-04-12T05:34:54.555 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:34:54.555 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":18,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-12T05:34:22.641718+0000","modified":"2024-04-12T05:34:40.809969+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14480,"mds_1":24281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14480":{"gid":14480,"name":"cephfs.smithi032.pndtry","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6829/387754149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":387754149},{"type":"v1","addr":"172.21.15.32:6829","nonce":387754149}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14490":{"gid":14490,"name":"cephfs.smithi032.jehaaj","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6827/3678031004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3678031004},{"type":"v1","addr":"172.21.15.32:6827","nonce":3678031004}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24281":{"gid":24281,"name":"cephfs.smithi067.zesmgy","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6825/1291914599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1291914599},{"type":"v1","addr":"172.21.15.67:6825","nonce":1291914599}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24295":{"gid":24295,"name":"cephfs.smithi067.ymvqih","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.67:6827/873395842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873395842},{"type":"v1","addr":"172.21.15.67:6827","nonce":873395842}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2024-04-12T05:34:54.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.558+0000 7fc9b64d3700 1 -- 172.21.15.32:0/4092669448 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fc9980692a0 msgr2=0x7fc99806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:54.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.558+0000 7fc9b64d3700 1 --2- 172.21.15.32:0/4092669448 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fc9980692a0 0x7fc99806b750 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0004890 tx=0x7fc9a0009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:54.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.558+0000 7fc9b64d3700 1 -- 172.21.15.32:0/4092669448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9b00ffe00 msgr2=0x7fc9b010dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:54.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.558+0000 7fc9b64d3700 1 --2- 172.21.15.32:0/4092669448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9b00ffe00 0x7fc9b010dcb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4002a50 tx=0x7fc9a40046b0 comp rx=0 tx=0).stop 2024-04-12T05:34:54.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.558+0000 7fc9b64d3700 1 -- 172.21.15.32:0/4092669448 shutdown_connections 2024-04-12T05:34:54.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.558+0000 7fc9b64d3700 1 --2- 172.21.15.32:0/4092669448 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fc9980692a0 0x7fc99806b750 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:54.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.558+0000 7fc9b64d3700 1 --2- 172.21.15.32:0/4092669448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9b0065aa0 0x7fc9b010d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:54.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.558+0000 7fc9b64d3700 1 --2- 172.21.15.32:0/4092669448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9b00ffe00 0x7fc9b010dcb0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:54.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.558+0000 7fc9b64d3700 1 -- 172.21.15.32:0/4092669448 >> 172.21.15.32:0/4092669448 conn(0x7fc9b00f7310 msgr2=0x7fc9b00f7750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:54.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.559+0000 7fc9b64d3700 1 -- 172.21.15.32:0/4092669448 shutdown_connections 2024-04-12T05:34:54.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:54.559+0000 7fc9b64d3700 1 -- 172.21.15.32:0/4092669448 wait complete. 2024-04-12T05:34:54.559 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 18 2024-04-12T05:34:54.570 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 18, 'max_mds': 2, 'flags': 50} 2024-04-12T05:34:54.570 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-04-12T05:34:54.598 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-04-12T05:34:54.598 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.1': {}} 2024-04-12T05:34:54.598 INFO:tasks.ceph_fuse:client.0 config is {} 2024-04-12T05:34:54.598 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-12T05:34:54.599 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-12T05:34:54.599 INFO:tasks.ceph_fuse:client.1 config is {} 2024-04-12T05:34:54.599 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-12T05:34:54.599 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-12T05:34:54.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:34:54.599 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-04-12T05:34:54.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:34:54.619 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link delete ceph-brx 2024-04-12T05:34:54.689 INFO:teuthology.orchestra.run.smithi032.stderr:Cannot find device "ceph-brx" 2024-04-12T05:34:54.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T05:34:54.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:34:54.691 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-12T05:34:54.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:34:54.714 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link delete ceph-brx 2024-04-12T05:34:54.792 INFO:teuthology.orchestra.run.smithi067.stderr:Cannot find device "ceph-brx" 2024-04-12T05:34:54.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T05:34:54.794 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-04-12T05:34:54.794 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph fs ls 2024-04-12T05:34:55.070 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:34:55.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:54 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/1766618864' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-12T05:34:55.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:54 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/4092669448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T05:34:55.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:54 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/1766618864' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-12T05:34:55.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:54 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/4092669448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T05:34:56.310 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:56 smithi032 ceph-mon[39731]: pgmap v199: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-12T05:34:56.310 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:56 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:56.310 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:56 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:34:56.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:56 smithi067 ceph-mon[49010]: pgmap v199: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-12T05:34:56.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:56 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:34:56.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:56 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:34:56.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.859+0000 7f43e3d48700 1 -- 172.21.15.32:0/3152363919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43dc1026a0 msgr2=0x7f43dc0fbd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:56.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.859+0000 7f43e3d48700 1 --2- 172.21.15.32:0/3152363919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43dc1026a0 0x7f43dc0fbd10 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f43d8004940 tx=0x7f43d8009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:56.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.859+0000 7f43e3d48700 1 -- 172.21.15.32:0/3152363919 shutdown_connections 2024-04-12T05:34:56.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.859+0000 7f43e3d48700 1 --2- 172.21.15.32:0/3152363919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43dc1026a0 0x7f43dc0fbd10 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:56.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.859+0000 7f43e3d48700 1 --2- 172.21.15.32:0/3152363919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43dc101d00 0x7f43dc1020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:56.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.859+0000 7f43e3d48700 1 -- 172.21.15.32:0/3152363919 >> 172.21.15.32:0/3152363919 conn(0x7f43dc0f78d0 msgr2=0x7f43dc0f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:56.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.859+0000 7f43e3d48700 1 -- 172.21.15.32:0/3152363919 shutdown_connections 2024-04-12T05:34:56.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.859+0000 7f43e3d48700 1 -- 172.21.15.32:0/3152363919 wait complete. 2024-04-12T05:34:56.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.860+0000 7f43e3d48700 1 Processor -- start 2024-04-12T05:34:56.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.860+0000 7f43e3d48700 1 -- start start 2024-04-12T05:34:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.860+0000 7f43e3d48700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43dc101d00 0x7f43dc102b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.860+0000 7f43e3d48700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43dc1026a0 0x7f43dc1030a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.860+0000 7f43e3d48700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43dc103630 con 0x7f43dc1026a0 2024-04-12T05:34:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.860+0000 7f43e3d48700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43dc103770 con 0x7f43dc101d00 2024-04-12T05:34:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.860+0000 7f43e12e3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43dc1026a0 0x7f43dc1030a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.860+0000 7f43e12e3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43dc1026a0 0x7f43dc1030a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43706/0 (socket says 172.21.15.32:43706) 2024-04-12T05:34:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.860+0000 7f43e12e3700 1 -- 172.21.15.32:0/3489078450 learned_addr learned my addr 172.21.15.32:0/3489078450 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:34:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.860+0000 7f43e1ae4700 1 --2- 172.21.15.32:0/3489078450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43dc101d00 0x7f43dc102b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:56.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.860+0000 7f43e12e3700 1 -- 172.21.15.32:0/3489078450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43dc101d00 msgr2=0x7f43dc102b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:56.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.860+0000 7f43e12e3700 1 --2- 172.21.15.32:0/3489078450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43dc101d00 0x7f43dc102b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:56.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.860+0000 7f43e12e3700 1 -- 172.21.15.32:0/3489078450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d8005040 con 0x7f43dc1026a0 2024-04-12T05:34:56.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.860+0000 7f43e1ae4700 1 --2- 172.21.15.32:0/3489078450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43dc101d00 0x7f43dc102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:34:56.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.861+0000 7f43e12e3700 1 --2- 172.21.15.32:0/3489078450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43dc1026a0 0x7f43dc1030a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f43d8009380 tx=0x7f43d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:56.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.861+0000 7f43d27fc700 1 -- 172.21.15.32:0/3489078450 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43d800e040 con 0x7f43dc1026a0 2024-04-12T05:34:56.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.861+0000 7f43e3d48700 1 -- 172.21.15.32:0/3489078450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43dc1039c0 con 0x7f43dc1026a0 2024-04-12T05:34:56.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.861+0000 7f43d27fc700 1 -- 172.21.15.32:0/3489078450 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f43d8013420 con 0x7f43dc1026a0 2024-04-12T05:34:56.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.861+0000 7f43d27fc700 1 -- 172.21.15.32:0/3489078450 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43d801c630 con 0x7f43dc1026a0 2024-04-12T05:34:56.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.861+0000 7f43e3d48700 1 -- 172.21.15.32:0/3489078450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43dc11a230 con 0x7f43dc1026a0 2024-04-12T05:34:56.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.862+0000 7f43e3d48700 1 -- 172.21.15.32:0/3489078450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43dc0fd450 con 0x7f43dc1026a0 2024-04-12T05:34:56.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.862+0000 7f43d27fc700 1 -- 172.21.15.32:0/3489078450 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f43d80139e0 con 0x7f43dc1026a0 2024-04-12T05:34:56.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.863+0000 7f43d27fc700 1 --2- 172.21.15.32:0/3489078450 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f43c80692a0 0x7f43c806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:34:56.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.863+0000 7f43d27fc700 1 -- 172.21.15.32:0/3489078450 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f43d80883d0 con 0x7f43dc1026a0 2024-04-12T05:34:56.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.863+0000 7f43e1ae4700 1 --2- 172.21.15.32:0/3489078450 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f43c80692a0 0x7f43c806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:56.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.863+0000 7f43e1ae4700 1 --2- 172.21.15.32:0/3489078450 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f43c80692a0 0x7f43c806b750 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f43cc007fd0 tx=0x7f43cc009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:56.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:56.867+0000 7f43d27fc700 1 -- 172.21.15.32:0/3489078450 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f43d8054650 con 0x7f43dc1026a0 2024-04-12T05:34:57.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:57.112+0000 7f43e3d48700 1 -- 172.21.15.32:0/3489078450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f43dc108cd0 con 0x7f43dc1026a0 2024-04-12T05:34:57.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:57.113+0000 7f43d27fc700 1 -- 172.21.15.32:0/3489078450 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) v1 ==== 53+0+83 (secure 0 0 0) 0x7f43d8057c70 con 0x7f43dc1026a0 2024-04-12T05:34:57.112 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-12T05:34:57.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:57.115+0000 7f43e3d48700 1 -- 172.21.15.32:0/3489078450 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f43c80692a0 msgr2=0x7f43c806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:57.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:57.115+0000 7f43e3d48700 1 --2- 172.21.15.32:0/3489078450 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f43c80692a0 0x7f43c806b750 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f43cc007fd0 tx=0x7f43cc009040 comp rx=0 tx=0).stop 2024-04-12T05:34:57.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:57.115+0000 7f43e3d48700 1 -- 172.21.15.32:0/3489078450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43dc1026a0 msgr2=0x7f43dc1030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:57.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:57.115+0000 7f43e3d48700 1 --2- 172.21.15.32:0/3489078450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43dc1026a0 0x7f43dc1030a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f43d8009380 tx=0x7f43d8009e30 comp rx=0 tx=0).stop 2024-04-12T05:34:57.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:57.115+0000 7f43e3d48700 1 -- 172.21.15.32:0/3489078450 shutdown_connections 2024-04-12T05:34:57.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:57.115+0000 7f43e3d48700 1 --2- 172.21.15.32:0/3489078450 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f43c80692a0 0x7f43c806b750 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:57.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:57.115+0000 7f43e3d48700 1 --2- 172.21.15.32:0/3489078450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43dc101d00 0x7f43dc102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:57.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:57.115+0000 7f43e3d48700 1 --2- 172.21.15.32:0/3489078450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43dc1026a0 0x7f43dc1030a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:34:57.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:57.115+0000 7f43e3d48700 1 -- 172.21.15.32:0/3489078450 >> 172.21.15.32:0/3489078450 conn(0x7f43dc0f78d0 msgr2=0x7f43dc0f89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:57.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:57.115+0000 7f43e3d48700 1 -- 172.21.15.32:0/3489078450 shutdown_connections 2024-04-12T05:34:57.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:34:57.115+0000 7f43e3d48700 1 -- 172.21.15.32:0/3489078450 wait complete. 2024-04-12T05:34:57.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:57 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/3489078450' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-12T05:34:57.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:57 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/3489078450' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-12T05:34:58.060 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-12T05:34:58.060 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-12T05:34:58.060 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-04-12T05:34:58.060 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-12T05:34:58.060 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-12T05:34:58.060 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-12T05:34:58.060 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-12T05:34:58.060 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-12T05:34:58.060 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-12T05:34:58.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:34:58.061 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-04-12T05:34:58.083 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-12T05:34:58.083 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-12T05:34:58.083 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-04-12T05:34:58.084 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-04-12T05:34:58.084 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-04-12T05:34:58.084 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-04-12T05:34:58.084 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-12T05:34:58.084 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-04-12T05:34:58.084 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-04-12T05:34:58.084 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-12T05:34:58.084 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-04-12T05:34:58.084 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-04-12T05:34:58.084 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-12T05:34:58.084 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-04-12T05:34:58.084 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-04-12T05:34:58.084 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-12T05:34:58.084 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 41968sec preferred_lft 41968sec 2024-04-12T05:34:58.084 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link 2024-04-12T05:34:58.085 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-04-12T05:34:58.085 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-12T05:34:58.085 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-04-12T05:34:58.085 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-04-12T05:34:58.085 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-12T05:34:58.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:34:58.085 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-04-12T05:34:58.085 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-04-12T05:34:58.086 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link add name ceph-brx type bridge 2024-04-12T05:34:58.086 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip addr flush dev ceph-brx 2024-04-12T05:34:58.086 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set ceph-brx up 2024-04-12T05:34:58.086 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-12T05:34:58.086 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-04-12T05:34:58.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:34:58.261 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-12T05:34:58.287 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:34:58 smithi032 ceph-mon[39731]: pgmap v200: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-12T05:34:58.290 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-04-12T05:34:58.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:34:58.292 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-04-12T05:34:58.363 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-04-12T05:34:58.363 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-12T05:34:58.363 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-12T05:34:58.363 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-12T05:34:58.363 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-12T05:34:58.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:34:58.364 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-04-12T05:34:58.364 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-04-12T05:34:58.364 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-12T05:34:58.364 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-12T05:34:58.364 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-12T05:34:58.364 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-04-12T05:34:58.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:34:58 smithi067 ceph-mon[49010]: pgmap v200: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-12T05:34:58.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:34:58.649 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-04-12T05:34:58.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:34:58.668 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list-id 2024-04-12T05:34:58.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:34:58.726 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-04-12T05:34:58.726 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-04-12T05:34:58.726 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-12T05:34:58.726 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-12T05:34:58.726 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-04-12T05:34:58.822 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-12T05:34:58.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:34:58.822 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-04-12T05:34:58.822 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-04-12T05:34:58.822 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-12T05:34:58.822 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-12T05:34:58.822 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-12T05:34:58.822 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-12T05:34:58.823 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-12T05:34:58.823 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-04-12T05:34:59.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:34:59.015 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec bash -c ' 2024-04-12T05:34:59.015 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-04-12T05:34:59.015 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-04-12T05:34:59.015 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set dev brx.0 master ceph-brx 2024-04-12T05:34:59.016 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-04-12T05:34:59.114 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-12T05:34:59.114 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-12T05:34:59.114 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-12T05:34:59.132 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-12T05:34:59.133 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-12T05:34:59.133 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-12T05:34:59.191 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2024-04-12T05:34:59.259 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-12T05:34:59.259 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-04-12T05:34:59.328 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-04-12T05:34:59.329 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-04-12T05:34:59.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:34:59.329 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2024-04-12T05:34:59.388 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-12T05:34:59.388 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-12T05:34:59.431 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2024-04-12T05:34:59.459 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-12T05:34:59.459 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-04-12T05:34:59.527 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-04-12T05:34:59.528 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-04-12T05:34:59.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:34:59.528 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2024-04-12T05:34:59.546 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:2024-04-12T05:34:59.546+0000 7f1e34119540 -1 init, newargv = 0x564d96719dc0 newargc=15 2024-04-12T05:34:59.546 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[101577]: starting ceph client 2024-04-12T05:34:59.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[101577]: starting fuse 2024-04-12T05:34:59.565 INFO:teuthology.orchestra.run.smithi032.stdout:93 2024-04-12T05:34:59.565 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [93] 2024-04-12T05:34:59.565 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-04-12T05:34:59.566 DEBUG:teuthology.orchestra.run.smithi032:> import glob 2024-04-12T05:34:59.566 DEBUG:teuthology.orchestra.run.smithi032:> import re 2024-04-12T05:34:59.566 DEBUG:teuthology.orchestra.run.smithi032:> import os 2024-04-12T05:34:59.566 DEBUG:teuthology.orchestra.run.smithi032:> import subprocess 2024-04-12T05:34:59.566 DEBUG:teuthology.orchestra.run.smithi032:> 2024-04-12T05:34:59.566 DEBUG:teuthology.orchestra.run.smithi032:> def _find_admin_socket(client_name): 2024-04-12T05:34:59.566 DEBUG:teuthology.orchestra.run.smithi032:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-12T05:34:59.566 DEBUG:teuthology.orchestra.run.smithi032:> files = glob.glob(asok_path) 2024-04-12T05:34:59.566 DEBUG:teuthology.orchestra.run.smithi032:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-12T05:34:59.566 DEBUG:teuthology.orchestra.run.smithi032:> 2024-04-12T05:34:59.566 DEBUG:teuthology.orchestra.run.smithi032:> # Given a non-glob path, it better be there 2024-04-12T05:34:59.566 DEBUG:teuthology.orchestra.run.smithi032:> if "*" not in asok_path: 2024-04-12T05:34:59.566 DEBUG:teuthology.orchestra.run.smithi032:> assert(len(files) == 1) 2024-04-12T05:34:59.566 DEBUG:teuthology.orchestra.run.smithi032:> return files[0] 2024-04-12T05:34:59.566 DEBUG:teuthology.orchestra.run.smithi032:> 2024-04-12T05:34:59.567 DEBUG:teuthology.orchestra.run.smithi032:> for f in files: 2024-04-12T05:34:59.567 DEBUG:teuthology.orchestra.run.smithi032:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-12T05:34:59.567 DEBUG:teuthology.orchestra.run.smithi032:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-12T05:34:59.567 DEBUG:teuthology.orchestra.run.smithi032:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-12T05:34:59.567 DEBUG:teuthology.orchestra.run.smithi032:> contents = proc_f.read() 2024-04-12T05:34:59.567 DEBUG:teuthology.orchestra.run.smithi032:> if mountpoint in contents: 2024-04-12T05:34:59.567 DEBUG:teuthology.orchestra.run.smithi032:> return f 2024-04-12T05:34:59.567 DEBUG:teuthology.orchestra.run.smithi032:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-12T05:34:59.567 DEBUG:teuthology.orchestra.run.smithi032:> 2024-04-12T05:34:59.567 DEBUG:teuthology.orchestra.run.smithi032:> print(_find_admin_socket("client.0")) 2024-04-12T05:34:59.567 DEBUG:teuthology.orchestra.run.smithi032:> ' 2024-04-12T05:34:59.691 INFO:teuthology.orchestra.run.smithi032.stdout:/var/run/ceph/ceph-client.0.101577.asok 2024-04-12T05:34:59.873 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.101577.asok 2024-04-12T05:34:59.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:34:59.873 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.101577.asok status 2024-04-12T05:35:00.031 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:35:00.031 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": { 2024-04-12T05:35:00.031 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_sha1": "d3b5060a17449edfae9385cdf792afec109fca5d", 2024-04-12T05:35:00.031 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)", 2024-04-12T05:35:00.032 INFO:teuthology.orchestra.run.smithi032.stdout: "entity_id": "0", 2024-04-12T05:35:00.032 INFO:teuthology.orchestra.run.smithi032.stdout: "hostname": "smithi032", 2024-04-12T05:35:00.032 INFO:teuthology.orchestra.run.smithi032.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-12T05:35:00.032 INFO:teuthology.orchestra.run.smithi032.stdout: "pid": "101577", 2024-04-12T05:35:00.032 INFO:teuthology.orchestra.run.smithi032.stdout: "root": "/" 2024-04-12T05:35:00.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:35:00.032 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_count": 0, 2024-04-12T05:35:00.032 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_pinned_count": 0, 2024-04-12T05:35:00.032 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 14534, 2024-04-12T05:35:00.032 INFO:teuthology.orchestra.run.smithi032.stdout: "inst": { 2024-04-12T05:35:00.032 INFO:teuthology.orchestra.run.smithi032.stdout: "name": { 2024-04-12T05:35:00.032 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "client", 2024-04-12T05:35:00.032 INFO:teuthology.orchestra.run.smithi032.stdout: "num": 14534 2024-04-12T05:35:00.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:35:00.033 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2024-04-12T05:35:00.033 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-12T05:35:00.033 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "192.168.0.1:0", 2024-04-12T05:35:00.033 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2053061720 2024-04-12T05:35:00.033 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:35:00.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:35:00.033 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2024-04-12T05:35:00.033 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-12T05:35:00.033 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "192.168.0.1:0", 2024-04-12T05:35:00.033 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2053061720 2024-04-12T05:35:00.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:35:00.033 INFO:teuthology.orchestra.run.smithi032.stdout: "inst_str": "client.14534 192.168.0.1:0/2053061720", 2024-04-12T05:35:00.033 INFO:teuthology.orchestra.run.smithi032.stdout: "addr_str": "192.168.0.1:0/2053061720", 2024-04-12T05:35:00.033 INFO:teuthology.orchestra.run.smithi032.stdout: "inode_count": 1, 2024-04-12T05:35:00.033 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_epoch": 18, 2024-04-12T05:35:00.034 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch": 46, 2024-04-12T05:35:00.034 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch_barrier": 0, 2024-04-12T05:35:00.034 INFO:teuthology.orchestra.run.smithi032.stdout: "blocklisted": false, 2024-04-12T05:35:00.034 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs" 2024-04-12T05:35:00.034 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:35:00.041 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- ceph fs ls 2024-04-12T05:35:00.319 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:35:00.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:00 smithi032 ceph-mon[39731]: pgmap v201: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:35:00.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:00 smithi067 ceph-mon[49010]: pgmap v201: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:35:02.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.425+0000 7f079e681700 1 -- 172.21.15.32:0/3449670428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0798102710 msgr2=0x7f07980fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:02.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.425+0000 7f079e681700 1 --2- 172.21.15.32:0/3449670428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0798102710 0x7f07980fbd80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0788004940 tx=0x7f0788009e30 comp rx=0 tx=0).stop 2024-04-12T05:35:02.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.425+0000 7f079e681700 1 -- 172.21.15.32:0/3449670428 shutdown_connections 2024-04-12T05:35:02.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.425+0000 7f079e681700 1 --2- 172.21.15.32:0/3449670428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0798102710 0x7f07980fbd80 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:35:02.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.425+0000 7f079e681700 1 --2- 172.21.15.32:0/3449670428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0798101d70 0x7f0798102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:35:02.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.425+0000 7f079e681700 1 -- 172.21.15.32:0/3449670428 >> 172.21.15.32:0/3449670428 conn(0x7f0798072bc0 msgr2=0x7f0798072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:02.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.425+0000 7f079e681700 1 -- 172.21.15.32:0/3449670428 shutdown_connections 2024-04-12T05:35:02.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.425+0000 7f079e681700 1 -- 172.21.15.32:0/3449670428 wait complete. 2024-04-12T05:35:02.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.425+0000 7f079e681700 1 Processor -- start 2024-04-12T05:35:02.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.426+0000 7f079e681700 1 -- start start 2024-04-12T05:35:02.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.426+0000 7f079e681700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0798101d70 0x7f0798102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:35:02.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.426+0000 7f079e681700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0798102710 0x7f0798103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:35:02.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.426+0000 7f079e681700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07981036a0 con 0x7f0798101d70 2024-04-12T05:35:02.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.426+0000 7f079e681700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07981037e0 con 0x7f0798102710 2024-04-12T05:35:02.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.426+0000 7f0797fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0798101d70 0x7f0798102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:02.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.426+0000 7f0797fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0798101d70 0x7f0798102bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53234/0 (socket says 172.21.15.32:53234) 2024-04-12T05:35:02.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.426+0000 7f0797fff700 1 -- 172.21.15.32:0/1941238525 learned_addr learned my addr 172.21.15.32:0/1941238525 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:35:02.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.426+0000 7f07977fe700 1 --2- 172.21.15.32:0/1941238525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0798102710 0x7f0798103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:02.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.426+0000 7f0797fff700 1 -- 172.21.15.32:0/1941238525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0798102710 msgr2=0x7f0798103110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:02.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.426+0000 7f0797fff700 1 --2- 172.21.15.32:0/1941238525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0798102710 0x7f0798103110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:35:02.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.426+0000 7f0797fff700 1 -- 172.21.15.32:0/1941238525 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0788005040 con 0x7f0798101d70 2024-04-12T05:35:02.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.427+0000 7f0797fff700 1 --2- 172.21.15.32:0/1941238525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0798101d70 0x7f0798102bd0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f078000c830 tx=0x7f0780004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:02.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.427+0000 7f0794ff9700 1 -- 172.21.15.32:0/1941238525 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f078000a760 con 0x7f0798101d70 2024-04-12T05:35:02.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.427+0000 7f0794ff9700 1 -- 172.21.15.32:0/1941238525 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f078000ad70 con 0x7f0798101d70 2024-04-12T05:35:02.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.427+0000 7f079e681700 1 -- 172.21.15.32:0/1941238525 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0798119dc0 con 0x7f0798101d70 2024-04-12T05:35:02.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.427+0000 7f0794ff9700 1 -- 172.21.15.32:0/1941238525 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07800113a0 con 0x7f0798101d70 2024-04-12T05:35:02.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.427+0000 7f079e681700 1 -- 172.21.15.32:0/1941238525 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f079811a240 con 0x7f0798101d70 2024-04-12T05:35:02.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.428+0000 7f079e681700 1 -- 172.21.15.32:0/1941238525 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07980fd4c0 con 0x7f0798101d70 2024-04-12T05:35:02.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.428+0000 7f0794ff9700 1 -- 172.21.15.32:0/1941238525 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f0780012400 con 0x7f0798101d70 2024-04-12T05:35:02.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.429+0000 7f0794ff9700 1 --2- 172.21.15.32:0/1941238525 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f07840692a0 0x7f078406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:35:02.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.429+0000 7f0794ff9700 1 -- 172.21.15.32:0/1941238525 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0780087310 con 0x7f0798101d70 2024-04-12T05:35:02.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.429+0000 7f07977fe700 1 --2- 172.21.15.32:0/1941238525 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f07840692a0 0x7f078406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:02.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.429+0000 7f07977fe700 1 --2- 172.21.15.32:0/1941238525 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f07840692a0 0x7f078406b750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f078800b3b0 tx=0x7f0788009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:02.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.433+0000 7f0794ff9700 1 -- 172.21.15.32:0/1941238525 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f0780053550 con 0x7f0798101d70 2024-04-12T05:35:02.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:02 smithi067 ceph-mon[49010]: pgmap v202: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.6 KiB/s rd, 255 B/s wr, 4 op/s 2024-04-12T05:35:02.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:02 smithi032 ceph-mon[39731]: pgmap v202: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.6 KiB/s rd, 255 B/s wr, 4 op/s 2024-04-12T05:35:02.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.678+0000 7f079e681700 1 -- 172.21.15.32:0/1941238525 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f07981042c0 con 0x7f0798101d70 2024-04-12T05:35:02.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.678+0000 7f0794ff9700 1 -- 172.21.15.32:0/1941238525 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) v1 ==== 53+0+83 (secure 0 0 0) 0x7f0780056b70 con 0x7f0798101d70 2024-04-12T05:35:02.678 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-12T05:35:02.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.680+0000 7f079e681700 1 -- 172.21.15.32:0/1941238525 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f07840692a0 msgr2=0x7f078406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:02.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.680+0000 7f079e681700 1 --2- 172.21.15.32:0/1941238525 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f07840692a0 0x7f078406b750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f078800b3b0 tx=0x7f0788009e30 comp rx=0 tx=0).stop 2024-04-12T05:35:02.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.680+0000 7f079e681700 1 -- 172.21.15.32:0/1941238525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0798101d70 msgr2=0x7f0798102bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:02.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.680+0000 7f079e681700 1 --2- 172.21.15.32:0/1941238525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0798101d70 0x7f0798102bd0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f078000c830 tx=0x7f0780004470 comp rx=0 tx=0).stop 2024-04-12T05:35:02.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.680+0000 7f079e681700 1 -- 172.21.15.32:0/1941238525 shutdown_connections 2024-04-12T05:35:02.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.680+0000 7f079e681700 1 --2- 172.21.15.32:0/1941238525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0798101d70 0x7f0798102bd0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:35:02.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.680+0000 7f079e681700 1 --2- 172.21.15.32:0/1941238525 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f07840692a0 0x7f078406b750 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:35:02.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.680+0000 7f079e681700 1 --2- 172.21.15.32:0/1941238525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0798102710 0x7f0798103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:35:02.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.680+0000 7f079e681700 1 -- 172.21.15.32:0/1941238525 >> 172.21.15.32:0/1941238525 conn(0x7f0798072bc0 msgr2=0x7f0798065a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:02.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.681+0000 7f079e681700 1 -- 172.21.15.32:0/1941238525 shutdown_connections 2024-04-12T05:35:02.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:02.681+0000 7f079e681700 1 -- 172.21.15.32:0/1941238525 wait complete. 2024-04-12T05:35:03.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:03 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/1941238525' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-12T05:35:03.577 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:03 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/1941238525' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-12T05:35:03.600 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-12T05:35:03.600 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-12T05:35:03.601 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-12T05:35:03.601 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-12T05:35:03.601 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-12T05:35:03.601 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-12T05:35:03.601 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-12T05:35:03.601 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-12T05:35:03.601 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-12T05:35:03.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:35:03.601 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-12T05:35:03.625 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-12T05:35:03.626 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-12T05:35:03.626 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-12T05:35:03.626 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-12T05:35:03.626 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-12T05:35:03.626 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-12T05:35:03.626 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-12T05:35:03.626 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-12T05:35:03.626 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-12T05:35:03.626 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-12T05:35:03.626 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-12T05:35:03.626 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-12T05:35:03.626 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-12T05:35:03.627 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-12T05:35:03.627 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-12T05:35:03.627 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-12T05:35:03.627 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-12T05:35:03.627 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 41982sec preferred_lft 41982sec 2024-04-12T05:35:03.627 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-12T05:35:03.627 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-12T05:35:03.627 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-12T05:35:03.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:35:03.627 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-04-12T05:35:03.628 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-12T05:35:03.628 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add name ceph-brx type bridge 2024-04-12T05:35:03.628 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip addr flush dev ceph-brx 2024-04-12T05:35:03.628 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set ceph-brx up 2024-04-12T05:35:03.628 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-12T05:35:03.628 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-12T05:35:03.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:35:03.797 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-12T05:35:03.829 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-12T05:35:03.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:35:03.830 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-12T05:35:03.909 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-12T05:35:03.909 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-12T05:35:03.909 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-12T05:35:03.909 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-12T05:35:03.910 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-12T05:35:03.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:35:03.910 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-04-12T05:35:03.910 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-12T05:35:03.910 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-12T05:35:03.910 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-12T05:35:03.910 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-12T05:35:03.910 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-12T05:35:04.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:35:04.211 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-12T05:35:04.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:35:04.232 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list-id 2024-04-12T05:35:04.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:04 smithi067 ceph-mon[49010]: pgmap v203: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 341 B/s wr, 4 op/s 2024-04-12T05:35:04.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:35:04.289 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-04-12T05:35:04.289 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-12T05:35:04.290 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-12T05:35:04.290 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-12T05:35:04.290 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-12T05:35:04.389 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-12T05:35:04.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:35:04.389 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-04-12T05:35:04.389 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-12T05:35:04.389 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-12T05:35:04.389 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-12T05:35:04.389 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-12T05:35:04.389 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-12T05:35:04.389 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-12T05:35:04.389 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-12T05:35:04.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:04 smithi032 ceph-mon[39731]: pgmap v203: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 341 B/s wr, 4 op/s 2024-04-12T05:35:04.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:35:04.636 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-04-12T05:35:04.638 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-12T05:35:04.639 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-12T05:35:04.639 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set dev brx.0 master ceph-brx 2024-04-12T05:35:04.639 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-12T05:35:04.738 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-12T05:35:04.739 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-12T05:35:04.739 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-12T05:35:04.757 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-12T05:35:04.757 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-12T05:35:04.757 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-12T05:35:04.816 DEBUG:teuthology.orchestra.run.smithi067:> sudo modprobe fuse 2024-04-12T05:35:04.886 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-12T05:35:04.886 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-04-12T05:35:04.955 INFO:teuthology.orchestra.run.smithi067.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-04-12T05:35:04.956 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-04-12T05:35:04.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:35:04.957 DEBUG:teuthology.orchestra.run.smithi067:> ls /sys/fs/fuse/connections 2024-04-12T05:35:05.015 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-12T05:35:05.016 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-12T05:35:05.058 DEBUG:teuthology.orchestra.run.smithi067:> sudo modprobe fuse 2024-04-12T05:35:05.086 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-12T05:35:05.086 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-04-12T05:35:05.156 INFO:teuthology.orchestra.run.smithi067.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-04-12T05:35:05.156 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-04-12T05:35:05.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:35:05.157 DEBUG:teuthology.orchestra.run.smithi067:> ls /sys/fs/fuse/connections 2024-04-12T05:35:05.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi067.stderr:2024-04-12T05:35:05.191+0000 7ff8dcf5d540 -1 init, newargv = 0x555816885c80 newargc=15 2024-04-12T05:35:05.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi067.stderr:ceph-fuse[80410]: starting ceph client 2024-04-12T05:35:05.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi067.stderr:ceph-fuse[80410]: starting fuse 2024-04-12T05:35:05.211 INFO:teuthology.orchestra.run.smithi067.stdout:125 2024-04-12T05:35:05.211 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [125] 2024-04-12T05:35:05.211 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-04-12T05:35:05.211 DEBUG:teuthology.orchestra.run.smithi067:> import glob 2024-04-12T05:35:05.211 DEBUG:teuthology.orchestra.run.smithi067:> import re 2024-04-12T05:35:05.211 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-04-12T05:35:05.212 DEBUG:teuthology.orchestra.run.smithi067:> import subprocess 2024-04-12T05:35:05.212 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-12T05:35:05.212 DEBUG:teuthology.orchestra.run.smithi067:> def _find_admin_socket(client_name): 2024-04-12T05:35:05.212 DEBUG:teuthology.orchestra.run.smithi067:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-12T05:35:05.212 DEBUG:teuthology.orchestra.run.smithi067:> files = glob.glob(asok_path) 2024-04-12T05:35:05.212 DEBUG:teuthology.orchestra.run.smithi067:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-12T05:35:05.212 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-12T05:35:05.212 DEBUG:teuthology.orchestra.run.smithi067:> # Given a non-glob path, it better be there 2024-04-12T05:35:05.212 DEBUG:teuthology.orchestra.run.smithi067:> if "*" not in asok_path: 2024-04-12T05:35:05.212 DEBUG:teuthology.orchestra.run.smithi067:> assert(len(files) == 1) 2024-04-12T05:35:05.212 DEBUG:teuthology.orchestra.run.smithi067:> return files[0] 2024-04-12T05:35:05.212 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-12T05:35:05.212 DEBUG:teuthology.orchestra.run.smithi067:> for f in files: 2024-04-12T05:35:05.212 DEBUG:teuthology.orchestra.run.smithi067:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-12T05:35:05.212 DEBUG:teuthology.orchestra.run.smithi067:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-12T05:35:05.212 DEBUG:teuthology.orchestra.run.smithi067:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-12T05:35:05.212 DEBUG:teuthology.orchestra.run.smithi067:> contents = proc_f.read() 2024-04-12T05:35:05.213 DEBUG:teuthology.orchestra.run.smithi067:> if mountpoint in contents: 2024-04-12T05:35:05.213 DEBUG:teuthology.orchestra.run.smithi067:> return f 2024-04-12T05:35:05.213 DEBUG:teuthology.orchestra.run.smithi067:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-12T05:35:05.213 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-12T05:35:05.213 DEBUG:teuthology.orchestra.run.smithi067:> print(_find_admin_socket("client.1")) 2024-04-12T05:35:05.213 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-04-12T05:35:05.336 INFO:teuthology.orchestra.run.smithi067.stdout:/var/run/ceph/ceph-client.1.80410.asok 2024-04-12T05:35:05.517 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.80410.asok 2024-04-12T05:35:05.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:35:05.518 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.80410.asok status 2024-04-12T05:35:05.675 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-12T05:35:05.676 INFO:teuthology.orchestra.run.smithi067.stdout: "metadata": { 2024-04-12T05:35:05.676 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_sha1": "d3b5060a17449edfae9385cdf792afec109fca5d", 2024-04-12T05:35:05.676 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_version": "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)", 2024-04-12T05:35:05.676 INFO:teuthology.orchestra.run.smithi067.stdout: "entity_id": "1", 2024-04-12T05:35:05.676 INFO:teuthology.orchestra.run.smithi067.stdout: "hostname": "smithi067", 2024-04-12T05:35:05.676 INFO:teuthology.orchestra.run.smithi067.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-12T05:35:05.676 INFO:teuthology.orchestra.run.smithi067.stdout: "pid": "80410", 2024-04-12T05:35:05.676 INFO:teuthology.orchestra.run.smithi067.stdout: "root": "/" 2024-04-12T05:35:05.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-12T05:35:05.676 INFO:teuthology.orchestra.run.smithi067.stdout: "dentry_count": 0, 2024-04-12T05:35:05.676 INFO:teuthology.orchestra.run.smithi067.stdout: "dentry_pinned_count": 0, 2024-04-12T05:35:05.676 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 24339, 2024-04-12T05:35:05.677 INFO:teuthology.orchestra.run.smithi067.stdout: "inst": { 2024-04-12T05:35:05.677 INFO:teuthology.orchestra.run.smithi067.stdout: "name": { 2024-04-12T05:35:05.677 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "client", 2024-04-12T05:35:05.677 INFO:teuthology.orchestra.run.smithi067.stdout: "num": 24339 2024-04-12T05:35:05.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-12T05:35:05.677 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": { 2024-04-12T05:35:05.677 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-12T05:35:05.677 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "192.168.0.1:0", 2024-04-12T05:35:05.677 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 304126001 2024-04-12T05:35:05.677 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-12T05:35:05.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-12T05:35:05.677 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": { 2024-04-12T05:35:05.677 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-12T05:35:05.678 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "192.168.0.1:0", 2024-04-12T05:35:05.678 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 304126001 2024-04-12T05:35:05.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-12T05:35:05.678 INFO:teuthology.orchestra.run.smithi067.stdout: "inst_str": "client.24339 192.168.0.1:0/304126001", 2024-04-12T05:35:05.678 INFO:teuthology.orchestra.run.smithi067.stdout: "addr_str": "192.168.0.1:0/304126001", 2024-04-12T05:35:05.678 INFO:teuthology.orchestra.run.smithi067.stdout: "inode_count": 1, 2024-04-12T05:35:05.678 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_epoch": 18, 2024-04-12T05:35:05.678 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_epoch": 46, 2024-04-12T05:35:05.678 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_epoch_barrier": 0, 2024-04-12T05:35:05.679 INFO:teuthology.orchestra.run.smithi067.stdout: "blocklisted": false, 2024-04-12T05:35:05.679 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs" 2024-04-12T05:35:05.679 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-12T05:35:05.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:35:05.686 DEBUG:teuthology.orchestra.run.smithi032:> stat --file-system '--printf=%T 2024-04-12T05:35:05.686 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-12T05:35:05.712 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2024-04-12T05:35:05.712 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-12T05:35:05.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:35:05.712 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-12T05:35:05.790 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:35:05.790 DEBUG:teuthology.orchestra.run.smithi067:> stat --file-system '--printf=%T 2024-04-12T05:35:05.790 DEBUG:teuthology.orchestra.run.smithi067:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-12T05:35:05.809 INFO:teuthology.orchestra.run.smithi067.stdout:fuseblk 2024-04-12T05:35:05.809 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-12T05:35:05.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:35:05.809 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-12T05:35:06.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:06 smithi067 ceph-mon[49010]: pgmap v204: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-04-12T05:35:06.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:06 smithi032 ceph-mon[39731]: pgmap v204: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-04-12T05:35:07.948 INFO:teuthology.run_tasks:Running task print... 2024-04-12T05:35:07.959 INFO:teuthology.task.print:**** done client 2024-04-12T05:35:07.959 INFO:teuthology.run_tasks:Running task parallel... 2024-04-12T05:35:07.979 INFO:teuthology.task.parallel:starting parallel... 2024-04-12T05:35:07.980 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-12T05:35:07.980 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-12T05:35:07.980 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-12T05:35:07.981 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-12T05:35:07.981 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-12T05:35:07.981 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-04-12T05:35:07.985 INFO:tasks.workunit:Pulling workunits from ref d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:35:07.985 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-12T05:35:07.985 INFO:tasks.workunit:timeout=3h 2024-04-12T05:35:07.985 INFO:tasks.workunit:cleanup=True 2024-04-12T05:35:07.985 DEBUG:teuthology.orchestra.run.smithi032:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-12T05:35:08.010 INFO:teuthology.orchestra.run.smithi032.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-12T05:35:08.010 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-04-12T05:35:08.010 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5dh/93d Inode: 1 Links: 2 2024-04-12T05:35:08.010 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T05:35:08.010 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:fusefs_t:s0 2024-04-12T05:35:08.010 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-12T05:35:08.010 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-12 05:34:27.076834747 +0000 2024-04-12T05:35:08.010 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-12 05:35:05.789540074 +0000 2024-04-12T05:35:08.011 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2024-04-12T05:35:08.011 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-12T05:35:08.011 DEBUG:teuthology.orchestra.run.smithi032:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-12T05:35:08.088 DEBUG:teuthology.orchestra.run.smithi067:> stat -- /home/ubuntu/cephtest/mnt.1 2024-04-12T05:35:08.112 INFO:teuthology.orchestra.run.smithi067.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-04-12T05:35:08.113 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-04-12T05:35:08.113 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 7dh/125d Inode: 1 Links: 3 2024-04-12T05:35:08.113 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T05:35:08.113 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:fusefs_t:s0 2024-04-12T05:35:08.113 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-12T05:35:08.113 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-12 05:35:08.083756575 +0000 2024-04-12T05:35:08.113 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-12 05:35:08.083756575 +0000 2024-04-12T05:35:08.113 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2024-04-12T05:35:08.113 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-04-12T05:35:08.113 DEBUG:teuthology.orchestra.run.smithi067:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-04-12T05:35:08.192 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:35:08.193 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:35:08.214 INFO:tasks.workunit.client.0.smithi032.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-12T05:35:08.253 INFO:tasks.workunit.client.1.smithi067.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-04-12T05:35:08.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:08 smithi067 ceph-mon[49010]: pgmap v205: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-12T05:35:08.285 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:35:08.299 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:08 smithi032 ceph-mon[39731]: pgmap v205: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-12T05:35:10.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.244+0000 7f401d331700 1 -- 172.21.15.32:0/2570351043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40180fdfe0 msgr2=0x7f40180fe450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:10.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.244+0000 7f401d331700 1 --2- 172.21.15.32:0/2570351043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40180fdfe0 0x7f40180fe450 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4008004940 tx=0x7f4008009e30 comp rx=0 tx=0).stop 2024-04-12T05:35:10.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.244+0000 7f401d331700 1 -- 172.21.15.32:0/2570351043 shutdown_connections 2024-04-12T05:35:10.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.244+0000 7f401d331700 1 --2- 172.21.15.32:0/2570351043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40180fdfe0 0x7f40180fe450 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:35:10.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.244+0000 7f401d331700 1 --2- 172.21.15.32:0/2570351043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40180fce70 0x7f40180fd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:35:10.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.244+0000 7f401d331700 1 -- 172.21.15.32:0/2570351043 >> 172.21.15.32:0/2570351043 conn(0x7f40180f8550 msgr2=0x7f40180fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:10.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.244+0000 7f401d331700 1 -- 172.21.15.32:0/2570351043 shutdown_connections 2024-04-12T05:35:10.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.244+0000 7f401d331700 1 -- 172.21.15.32:0/2570351043 wait complete. 2024-04-12T05:35:10.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.245+0000 7f401d331700 1 Processor -- start 2024-04-12T05:35:10.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.245+0000 7f401d331700 1 -- start start 2024-04-12T05:35:10.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.245+0000 7f401d331700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40180fce70 0x7f401810d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:35:10.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.245+0000 7f401d331700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40180fdfe0 0x7f401810ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:35:10.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.245+0000 7f401d331700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401810e3c0 con 0x7f40180fce70 2024-04-12T05:35:10.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.245+0000 7f401d331700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401810e530 con 0x7f40180fdfe0 2024-04-12T05:35:10.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.245+0000 7f4016ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40180fce70 0x7f401810d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:10.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.245+0000 7f4016ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40180fce70 0x7f401810d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53250/0 (socket says 172.21.15.32:53250) 2024-04-12T05:35:10.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.245+0000 7f4016ffd700 1 -- 172.21.15.32:0/1050428874 learned_addr learned my addr 172.21.15.32:0/1050428874 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:35:10.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.245+0000 7f40167fc700 1 --2- 172.21.15.32:0/1050428874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40180fdfe0 0x7f401810ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:10.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.245+0000 7f4016ffd700 1 -- 172.21.15.32:0/1050428874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40180fdfe0 msgr2=0x7f401810ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:10.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.245+0000 7f4016ffd700 1 --2- 172.21.15.32:0/1050428874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40180fdfe0 0x7f401810ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:35:10.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.245+0000 7f4016ffd700 1 -- 172.21.15.32:0/1050428874 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4008005040 con 0x7f40180fce70 2024-04-12T05:35:10.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.245+0000 7f4016ffd700 1 --2- 172.21.15.32:0/1050428874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40180fce70 0x7f401810d880 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f400c00c8c0 tx=0x7f400c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:10.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.245+0000 7f3fff7fe700 1 -- 172.21.15.32:0/1050428874 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f400c00a760 con 0x7f40180fce70 2024-04-12T05:35:10.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.246+0000 7f3fff7fe700 1 -- 172.21.15.32:0/1050428874 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f400c00ad70 con 0x7f40180fce70 2024-04-12T05:35:10.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.246+0000 7f401d331700 1 -- 172.21.15.32:0/1050428874 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4018112f40 con 0x7f40180fce70 2024-04-12T05:35:10.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.246+0000 7f3fff7fe700 1 -- 172.21.15.32:0/1050428874 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f400c0113a0 con 0x7f40180fce70 2024-04-12T05:35:10.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.246+0000 7f401d331700 1 -- 172.21.15.32:0/1050428874 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4018113470 con 0x7f40180fce70 2024-04-12T05:35:10.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.247+0000 7f401d331700 1 -- 172.21.15.32:0/1050428874 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f401804c710 con 0x7f40180fce70 2024-04-12T05:35:10.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.247+0000 7f3fff7fe700 1 -- 172.21.15.32:0/1050428874 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f400c012400 con 0x7f40180fce70 2024-04-12T05:35:10.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.248+0000 7f3fff7fe700 1 --2- 172.21.15.32:0/1050428874 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f40000692a0 0x7f400006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:35:10.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.248+0000 7f40167fc700 1 --2- 172.21.15.32:0/1050428874 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f40000692a0 0x7f400006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:10.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.248+0000 7f3fff7fe700 1 -- 172.21.15.32:0/1050428874 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f400c087230 con 0x7f40180fce70 2024-04-12T05:35:10.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.248+0000 7f40167fc700 1 --2- 172.21.15.32:0/1050428874 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f40000692a0 0x7f400006b750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f400800b3b0 tx=0x7f4008009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:10.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.252+0000 7f3fff7fe700 1 -- 172.21.15.32:0/1050428874 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f400c056fa0 con 0x7f40180fce70 2024-04-12T05:35:10.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:10 smithi032 ceph-mon[39731]: pgmap v206: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-12T05:35:10.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:10 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:35:10.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.447+0000 7f401d331700 1 -- 172.21.15.32:0/1050428874 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f40181136c0 con 0x7f40180fce70 2024-04-12T05:35:10.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.447+0000 7f3fff7fe700 1 -- 172.21.15.32:0/1050428874 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7f400c056b30 con 0x7f40180fce70 2024-04-12T05:35:10.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.449+0000 7f401d331700 1 -- 172.21.15.32:0/1050428874 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f40000692a0 msgr2=0x7f400006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:10.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.449+0000 7f401d331700 1 --2- 172.21.15.32:0/1050428874 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f40000692a0 0x7f400006b750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f400800b3b0 tx=0x7f4008009e30 comp rx=0 tx=0).stop 2024-04-12T05:35:10.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.449+0000 7f401d331700 1 -- 172.21.15.32:0/1050428874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40180fce70 msgr2=0x7f401810d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:10.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.449+0000 7f401d331700 1 --2- 172.21.15.32:0/1050428874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40180fce70 0x7f401810d880 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f400c00c8c0 tx=0x7f400c0044c0 comp rx=0 tx=0).stop 2024-04-12T05:35:10.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.449+0000 7f401d331700 1 -- 172.21.15.32:0/1050428874 shutdown_connections 2024-04-12T05:35:10.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.449+0000 7f401d331700 1 --2- 172.21.15.32:0/1050428874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40180fce70 0x7f401810d880 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:35:10.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.449+0000 7f401d331700 1 --2- 172.21.15.32:0/1050428874 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f40000692a0 0x7f400006b750 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:35:10.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.449+0000 7f401d331700 1 --2- 172.21.15.32:0/1050428874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40180fdfe0 0x7f401810ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:35:10.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.449+0000 7f401d331700 1 -- 172.21.15.32:0/1050428874 >> 172.21.15.32:0/1050428874 conn(0x7f40180f8550 msgr2=0x7f40180f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:10.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.449+0000 7f401d331700 1 -- 172.21.15.32:0/1050428874 shutdown_connections 2024-04-12T05:35:10.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:10.449+0000 7f401d331700 1 -- 172.21.15.32:0/1050428874 wait complete. 2024-04-12T05:35:10.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:10 smithi067 ceph-mon[49010]: pgmap v206: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-12T05:35:10.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:10 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:35:11.153 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-12T05:35:11.494 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:35:12.065 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:11 smithi032 ceph-mon[39731]: pgmap v207: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s rd, 938 B/s wr, 5 op/s 2024-04-12T05:35:12.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:11 smithi067 ceph-mon[49010]: pgmap v207: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s rd, 938 B/s wr, 5 op/s 2024-04-12T05:35:13.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.539+0000 7f4dbfcd1700 1 -- 172.21.15.32:0/216028327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db80fefd0 msgr2=0x7f4db80ff3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:13.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.539+0000 7f4dbfcd1700 1 --2- 172.21.15.32:0/216028327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db80fefd0 0x7f4db80ff3e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f4db4004940 tx=0x7f4db4009e30 comp rx=0 tx=0).stop 2024-04-12T05:35:13.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.539+0000 7f4dbfcd1700 1 -- 172.21.15.32:0/216028327 shutdown_connections 2024-04-12T05:35:13.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.539+0000 7f4dbfcd1700 1 --2- 172.21.15.32:0/216028327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4db8100280 0x7f4db81006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:35:13.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.539+0000 7f4dbfcd1700 1 --2- 172.21.15.32:0/216028327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db80fefd0 0x7f4db80ff3e0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:35:13.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.539+0000 7f4dbfcd1700 1 -- 172.21.15.32:0/216028327 >> 172.21.15.32:0/216028327 conn(0x7f4db80fa950 msgr2=0x7f4db80fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:13.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.539+0000 7f4dbfcd1700 1 -- 172.21.15.32:0/216028327 shutdown_connections 2024-04-12T05:35:13.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.539+0000 7f4dbfcd1700 1 -- 172.21.15.32:0/216028327 wait complete. 2024-04-12T05:35:13.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.539+0000 7f4dbfcd1700 1 Processor -- start 2024-04-12T05:35:13.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.539+0000 7f4dbfcd1700 1 -- start start 2024-04-12T05:35:13.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.539+0000 7f4dbfcd1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db80fefd0 0x7f4db810fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:35:13.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.539+0000 7f4dbfcd1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4db8100280 0x7f4db810fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:35:13.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.539+0000 7f4dbfcd1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db81105f0 con 0x7f4db80fefd0 2024-04-12T05:35:13.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.540+0000 7f4dbfcd1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db8110e40 con 0x7f4db8100280 2024-04-12T05:35:13.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.540+0000 7f4dbda6d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db80fefd0 0x7f4db810fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:13.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.540+0000 7f4dbda6d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db80fefd0 0x7f4db810fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40154/0 (socket says 172.21.15.32:40154) 2024-04-12T05:35:13.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.540+0000 7f4dbda6d700 1 -- 172.21.15.32:0/1773050763 learned_addr learned my addr 172.21.15.32:0/1773050763 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:35:13.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.540+0000 7f4dbd26c700 1 --2- 172.21.15.32:0/1773050763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4db8100280 0x7f4db810fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:13.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.540+0000 7f4dbda6d700 1 -- 172.21.15.32:0/1773050763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4db8100280 msgr2=0x7f4db810fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:13.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.540+0000 7f4dbda6d700 1 --2- 172.21.15.32:0/1773050763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4db8100280 0x7f4db810fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:35:13.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.540+0000 7f4dbda6d700 1 -- 172.21.15.32:0/1773050763 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db4005040 con 0x7f4db80fefd0 2024-04-12T05:35:13.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.540+0000 7f4dbd26c700 1 --2- 172.21.15.32:0/1773050763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4db8100280 0x7f4db810fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T05:35:13.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.540+0000 7f4dbda6d700 1 --2- 172.21.15.32:0/1773050763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db80fefd0 0x7f4db810fab0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4db4007d40 tx=0x7f4db4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:13.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.540+0000 7f4dae7fc700 1 -- 172.21.15.32:0/1773050763 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4db400e030 con 0x7f4db80fefd0 2024-04-12T05:35:13.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.540+0000 7f4dbfcd1700 1 -- 172.21.15.32:0/1773050763 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db81110c0 con 0x7f4db80fefd0 2024-04-12T05:35:13.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.540+0000 7f4dae7fc700 1 -- 172.21.15.32:0/1773050763 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4db401c400 con 0x7f4db80fefd0 2024-04-12T05:35:13.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.540+0000 7f4dbfcd1700 1 -- 172.21.15.32:0/1773050763 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db8111600 con 0x7f4db80fefd0 2024-04-12T05:35:13.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.540+0000 7f4dae7fc700 1 -- 172.21.15.32:0/1773050763 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4db400b650 con 0x7f4db80fefd0 2024-04-12T05:35:13.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.541+0000 7f4dbfcd1700 1 -- 172.21.15.32:0/1773050763 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d9c0052e0 con 0x7f4db80fefd0 2024-04-12T05:35:13.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.542+0000 7f4dae7fc700 1 -- 172.21.15.32:0/1773050763 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f4db401c970 con 0x7f4db80fefd0 2024-04-12T05:35:13.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.542+0000 7f4dae7fc700 1 --2- 172.21.15.32:0/1773050763 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4da40692a0 0x7f4da406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:35:13.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.543+0000 7f4dae7fc700 1 -- 172.21.15.32:0/1773050763 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f4db40888f0 con 0x7f4db80fefd0 2024-04-12T05:35:13.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.543+0000 7f4dbd26c700 1 --2- 172.21.15.32:0/1773050763 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4da40692a0 0x7f4da406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:13.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.543+0000 7f4dbd26c700 1 --2- 172.21.15.32:0/1773050763 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4da40692a0 0x7f4da406b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4da8009b80 tx=0x7f4da800b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:13.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.551+0000 7f4dae7fc700 1 -- 172.21.15.32:0/1773050763 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4db4058660 con 0x7f4db80fefd0 2024-04-12T05:35:13.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.759+0000 7f4dbfcd1700 1 -- 172.21.15.32:0/1773050763 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f4d9c005590 con 0x7f4db80fefd0 2024-04-12T05:35:13.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.759+0000 7f4dae7fc700 1 -- 172.21.15.32:0/1773050763 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f4db40581f0 con 0x7f4db80fefd0 2024-04-12T05:35:13.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.761+0000 7f4dbfcd1700 1 -- 172.21.15.32:0/1773050763 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4da40692a0 msgr2=0x7f4da406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:13.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.761+0000 7f4dbfcd1700 1 --2- 172.21.15.32:0/1773050763 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4da40692a0 0x7f4da406b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4da8009b80 tx=0x7f4da800b040 comp rx=0 tx=0).stop 2024-04-12T05:35:13.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.761+0000 7f4dbfcd1700 1 -- 172.21.15.32:0/1773050763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db80fefd0 msgr2=0x7f4db810fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:13.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.761+0000 7f4dbfcd1700 1 --2- 172.21.15.32:0/1773050763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db80fefd0 0x7f4db810fab0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4db4007d40 tx=0x7f4db4009e30 comp rx=0 tx=0).stop 2024-04-12T05:35:13.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.761+0000 7f4dbfcd1700 1 -- 172.21.15.32:0/1773050763 shutdown_connections 2024-04-12T05:35:13.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.761+0000 7f4dbfcd1700 1 --2- 172.21.15.32:0/1773050763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db80fefd0 0x7f4db810fab0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:35:13.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.761+0000 7f4dbfcd1700 1 --2- 172.21.15.32:0/1773050763 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4da40692a0 0x7f4da406b750 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:35:13.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.761+0000 7f4dbfcd1700 1 --2- 172.21.15.32:0/1773050763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4db8100280 0x7f4db810fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:35:13.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.761+0000 7f4dbfcd1700 1 -- 172.21.15.32:0/1773050763 >> 172.21.15.32:0/1773050763 conn(0x7f4db80fa950 msgr2=0x7f4db80fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:13.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.761+0000 7f4dbfcd1700 1 -- 172.21.15.32:0/1773050763 shutdown_connections 2024-04-12T05:35:13.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:13.761+0000 7f4dbfcd1700 1 -- 172.21.15.32:0/1773050763 wait complete. 2024-04-12T05:35:14.474 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:14 smithi032 ceph-mon[39731]: pgmap v208: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 767 B/s wr, 4 op/s 2024-04-12T05:35:14.496 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-12T05:35:14.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:14 smithi067 ceph-mon[49010]: pgmap v208: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 767 B/s wr, 4 op/s 2024-04-12T05:35:14.794 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:35:16.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:16 smithi067 ceph-mon[49010]: pgmap v209: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 682 B/s wr, 4 op/s 2024-04-12T05:35:16.584 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:16 smithi032 ceph-mon[39731]: pgmap v209: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 682 B/s wr, 4 op/s 2024-04-12T05:35:16.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.862+0000 7f8d46c12700 1 -- 172.21.15.32:0/1636233968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d400fcee0 msgr2=0x7f8d400fd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:16.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.862+0000 7f8d46c12700 1 --2- 172.21.15.32:0/1636233968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d400fcee0 0x7f8d400fd310 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8d34004940 tx=0x7f8d34009e30 comp rx=0 tx=0).stop 2024-04-12T05:35:16.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.862+0000 7f8d46c12700 1 -- 172.21.15.32:0/1636233968 shutdown_connections 2024-04-12T05:35:16.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.862+0000 7f8d46c12700 1 --2- 172.21.15.32:0/1636233968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d400fe0d0 0x7f8d400fe540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:35:16.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.862+0000 7f8d46c12700 1 --2- 172.21.15.32:0/1636233968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d400fcee0 0x7f8d400fd310 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:35:16.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.862+0000 7f8d46c12700 1 -- 172.21.15.32:0/1636233968 >> 172.21.15.32:0/1636233968 conn(0x7f8d400f8560 msgr2=0x7f8d400fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:16.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.862+0000 7f8d46c12700 1 -- 172.21.15.32:0/1636233968 shutdown_connections 2024-04-12T05:35:16.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.862+0000 7f8d46c12700 1 -- 172.21.15.32:0/1636233968 wait complete. 2024-04-12T05:35:16.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.863+0000 7f8d46c12700 1 Processor -- start 2024-04-12T05:35:16.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.863+0000 7f8d46c12700 1 -- start start 2024-04-12T05:35:16.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.863+0000 7f8d46c12700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d400fcee0 0x7f8d4010d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:35:16.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.863+0000 7f8d46c12700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d400fe0d0 0x7f8d4010de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:35:16.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.863+0000 7f8d46c12700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4010e430 con 0x7f8d400fe0d0 2024-04-12T05:35:16.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.863+0000 7f8d46c12700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4010ec80 con 0x7f8d400fcee0 2024-04-12T05:35:16.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.863+0000 7f8d3ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d400fe0d0 0x7f8d4010de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:16.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.863+0000 7f8d449ae700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d400fcee0 0x7f8d4010d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:16.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.863+0000 7f8d3ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d400fe0d0 0x7f8d4010de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40162/0 (socket says 172.21.15.32:40162) 2024-04-12T05:35:16.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.863+0000 7f8d3ffff700 1 -- 172.21.15.32:0/2765659224 learned_addr learned my addr 172.21.15.32:0/2765659224 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:35:16.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.864+0000 7f8d3ffff700 1 -- 172.21.15.32:0/2765659224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d400fcee0 msgr2=0x7f8d4010d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:16.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.864+0000 7f8d3ffff700 1 --2- 172.21.15.32:0/2765659224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d400fcee0 0x7f8d4010d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:35:16.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.864+0000 7f8d3ffff700 1 -- 172.21.15.32:0/2765659224 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d34005040 con 0x7f8d400fe0d0 2024-04-12T05:35:16.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.864+0000 7f8d449ae700 1 --2- 172.21.15.32:0/2765659224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d400fcee0 0x7f8d4010d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T05:35:16.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.864+0000 7f8d3ffff700 1 --2- 172.21.15.32:0/2765659224 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d400fe0d0 0x7f8d4010de30 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c00eb40 tx=0x7f8d2c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:16.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.864+0000 7f8d3d7fa700 1 -- 172.21.15.32:0/2765659224 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d2c007cc0 con 0x7f8d400fe0d0 2024-04-12T05:35:16.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.864+0000 7f8d3d7fa700 1 -- 172.21.15.32:0/2765659224 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d2c00f040 con 0x7f8d400fe0d0 2024-04-12T05:35:16.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.864+0000 7f8d46c12700 1 -- 172.21.15.32:0/2765659224 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d4010ef00 con 0x7f8d400fe0d0 2024-04-12T05:35:16.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.864+0000 7f8d3d7fa700 1 -- 172.21.15.32:0/2765659224 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d2c00c990 con 0x7f8d400fe0d0 2024-04-12T05:35:16.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.864+0000 7f8d46c12700 1 -- 172.21.15.32:0/2765659224 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d4010f440 con 0x7f8d400fe0d0 2024-04-12T05:35:16.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.865+0000 7f8d46c12700 1 -- 172.21.15.32:0/2765659224 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d4005f5f0 con 0x7f8d400fe0d0 2024-04-12T05:35:16.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.866+0000 7f8d3d7fa700 1 -- 172.21.15.32:0/2765659224 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f8d2c00caf0 con 0x7f8d400fe0d0 2024-04-12T05:35:16.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.867+0000 7f8d3d7fa700 1 --2- 172.21.15.32:0/2765659224 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f8d300692a0 0x7f8d3006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:35:16.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.867+0000 7f8d3d7fa700 1 -- 172.21.15.32:0/2765659224 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f8d2c086ec0 con 0x7f8d400fe0d0 2024-04-12T05:35:16.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.867+0000 7f8d449ae700 1 --2- 172.21.15.32:0/2765659224 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f8d300692a0 0x7f8d3006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:16.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.867+0000 7f8d449ae700 1 --2- 172.21.15.32:0/2765659224 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f8d300692a0 0x7f8d3006b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8d34013680 tx=0x7f8d34009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:16.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:16.870+0000 7f8d3d7fa700 1 -- 172.21.15.32:0/2765659224 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f8d2c056d80 con 0x7f8d400fe0d0 2024-04-12T05:35:17.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:17.085+0000 7f8d46c12700 1 -- 172.21.15.32:0/2765659224 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f8d4004bed0 con 0x7f8d400fe0d0 2024-04-12T05:35:17.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:17.085+0000 7f8d3d7fa700 1 -- 172.21.15.32:0/2765659224 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7f8d2c056910 con 0x7f8d400fe0d0 2024-04-12T05:35:17.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:17.087+0000 7f8d46c12700 1 -- 172.21.15.32:0/2765659224 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f8d300692a0 msgr2=0x7f8d3006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:17.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:17.087+0000 7f8d46c12700 1 --2- 172.21.15.32:0/2765659224 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f8d300692a0 0x7f8d3006b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8d34013680 tx=0x7f8d34009e30 comp rx=0 tx=0).stop 2024-04-12T05:35:17.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:17.087+0000 7f8d46c12700 1 -- 172.21.15.32:0/2765659224 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d400fe0d0 msgr2=0x7f8d4010de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:17.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:17.087+0000 7f8d46c12700 1 --2- 172.21.15.32:0/2765659224 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d400fe0d0 0x7f8d4010de30 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c00eb40 tx=0x7f8d2c00b040 comp rx=0 tx=0).stop 2024-04-12T05:35:17.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:17.087+0000 7f8d46c12700 1 -- 172.21.15.32:0/2765659224 shutdown_connections 2024-04-12T05:35:17.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:17.087+0000 7f8d46c12700 1 --2- 172.21.15.32:0/2765659224 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f8d300692a0 0x7f8d3006b750 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:35:17.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:17.087+0000 7f8d46c12700 1 --2- 172.21.15.32:0/2765659224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d400fcee0 0x7f8d4010d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:35:17.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:17.087+0000 7f8d46c12700 1 --2- 172.21.15.32:0/2765659224 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d400fe0d0 0x7f8d4010de30 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:35:17.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:17.087+0000 7f8d46c12700 1 -- 172.21.15.32:0/2765659224 >> 172.21.15.32:0/2765659224 conn(0x7f8d400f8560 msgr2=0x7f8d400f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:17.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:17.087+0000 7f8d46c12700 1 -- 172.21.15.32:0/2765659224 shutdown_connections 2024-04-12T05:35:17.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:17.087+0000 7f8d46c12700 1 -- 172.21.15.32:0/2765659224 wait complete. 2024-04-12T05:35:17.879 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-04-12T05:35:18.165 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:35:18.440 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:18 smithi032 ceph-mon[39731]: pgmap v210: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.1 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-12T05:35:18.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:18 smithi067 ceph-mon[49010]: pgmap v210: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.1 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-12T05:35:20.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.180+0000 7fc5f3656700 1 -- 172.21.15.32:0/2684888526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5ec0fce80 msgr2=0x7fc5ec0fd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:20.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.180+0000 7fc5f3656700 1 --2- 172.21.15.32:0/2684888526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5ec0fce80 0x7fc5ec0fd290 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8004940 tx=0x7fc5e8009e30 comp rx=0 tx=0).stop 2024-04-12T05:35:20.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.180+0000 7fc5f3656700 1 -- 172.21.15.32:0/2684888526 shutdown_connections 2024-04-12T05:35:20.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.180+0000 7fc5f3656700 1 --2- 172.21.15.32:0/2684888526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5ec0fdff0 0x7fc5ec0fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:35:20.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.180+0000 7fc5f3656700 1 --2- 172.21.15.32:0/2684888526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5ec0fce80 0x7fc5ec0fd290 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:35:20.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.180+0000 7fc5f3656700 1 -- 172.21.15.32:0/2684888526 >> 172.21.15.32:0/2684888526 conn(0x7fc5ec0f8580 msgr2=0x7fc5ec0fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:20.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.181+0000 7fc5f3656700 1 -- 172.21.15.32:0/2684888526 shutdown_connections 2024-04-12T05:35:20.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.181+0000 7fc5f3656700 1 -- 172.21.15.32:0/2684888526 wait complete. 2024-04-12T05:35:20.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.181+0000 7fc5f3656700 1 Processor -- start 2024-04-12T05:35:20.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.181+0000 7fc5f3656700 1 -- start start 2024-04-12T05:35:20.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.181+0000 7fc5f3656700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5ec0fce80 0x7fc5ec10d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:35:20.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.181+0000 7fc5f3656700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5ec0fdff0 0x7fc5ec10de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:35:20.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.181+0000 7fc5f3656700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec10e430 con 0x7fc5ec0fdff0 2024-04-12T05:35:20.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.181+0000 7fc5f3656700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec10ec80 con 0x7fc5ec0fce80 2024-04-12T05:35:20.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.181+0000 7fc5f0bf1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5ec0fdff0 0x7fc5ec10de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:20.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.181+0000 7fc5f0bf1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5ec0fdff0 0x7fc5ec10de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40182/0 (socket says 172.21.15.32:40182) 2024-04-12T05:35:20.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.181+0000 7fc5f0bf1700 1 -- 172.21.15.32:0/2507194335 learned_addr learned my addr 172.21.15.32:0/2507194335 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:35:20.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.181+0000 7fc5f13f2700 1 --2- 172.21.15.32:0/2507194335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5ec0fce80 0x7fc5ec10d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:20.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.182+0000 7fc5f0bf1700 1 -- 172.21.15.32:0/2507194335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5ec0fce80 msgr2=0x7fc5ec10d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:20.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.182+0000 7fc5f0bf1700 1 --2- 172.21.15.32:0/2507194335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5ec0fce80 0x7fc5ec10d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:35:20.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.182+0000 7fc5f0bf1700 1 -- 172.21.15.32:0/2507194335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e8005040 con 0x7fc5ec0fdff0 2024-04-12T05:35:20.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.182+0000 7fc5f13f2700 1 --2- 172.21.15.32:0/2507194335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5ec0fce80 0x7fc5ec10d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T05:35:20.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.182+0000 7fc5f0bf1700 1 --2- 172.21.15.32:0/2507194335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5ec0fdff0 0x7fc5ec10de30 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc00eb40 tx=0x7fc5dc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:20.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.182+0000 7fc5e1ffb700 1 -- 172.21.15.32:0/2507194335 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5dc007cc0 con 0x7fc5ec0fdff0 2024-04-12T05:35:20.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.182+0000 7fc5e1ffb700 1 -- 172.21.15.32:0/2507194335 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc5dc00f040 con 0x7fc5ec0fdff0 2024-04-12T05:35:20.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.182+0000 7fc5f3656700 1 -- 172.21.15.32:0/2507194335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5ec10ef00 con 0x7fc5ec0fdff0 2024-04-12T05:35:20.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.182+0000 7fc5e1ffb700 1 -- 172.21.15.32:0/2507194335 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5dc00c990 con 0x7fc5ec0fdff0 2024-04-12T05:35:20.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.182+0000 7fc5f3656700 1 -- 172.21.15.32:0/2507194335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5ec10f440 con 0x7fc5ec0fdff0 2024-04-12T05:35:20.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.183+0000 7fc5f3656700 1 -- 172.21.15.32:0/2507194335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5ec05f5f0 con 0x7fc5ec0fdff0 2024-04-12T05:35:20.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.184+0000 7fc5e1ffb700 1 -- 172.21.15.32:0/2507194335 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fc5dc00caf0 con 0x7fc5ec0fdff0 2024-04-12T05:35:20.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.184+0000 7fc5e1ffb700 1 --2- 172.21.15.32:0/2507194335 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fc5d80692b0 0x7fc5d806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:35:20.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.184+0000 7fc5e1ffb700 1 -- 172.21.15.32:0/2507194335 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fc5dc086cf0 con 0x7fc5ec0fdff0 2024-04-12T05:35:20.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.184+0000 7fc5f13f2700 1 --2- 172.21.15.32:0/2507194335 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fc5d80692b0 0x7fc5d806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:20.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.185+0000 7fc5f13f2700 1 --2- 172.21.15.32:0/2507194335 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fc5d80692b0 0x7fc5d806b760 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8013680 tx=0x7fc5e8009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:20.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.189+0000 7fc5e1ffb700 1 -- 172.21.15.32:0/2507194335 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc5dc04e190 con 0x7fc5ec0fdff0 2024-04-12T05:35:20.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:20.421+0000 7fc5f3656700 1 -- 172.21.15.32:0/2507194335 --> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5ec10f920 con 0x7fc5d80692b0 2024-04-12T05:35:20.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:20 smithi067 ceph-mon[49010]: pgmap v211: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 597 B/s wr, 4 op/s 2024-04-12T05:35:20.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:20 smithi032 ceph-mon[39731]: pgmap v211: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 597 B/s wr, 4 op/s 2024-04-12T05:35:22.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:22 smithi067 ceph-mon[49010]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:35:22.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:22 smithi067 ceph-mon[49010]: pgmap v212: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.3 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-12T05:35:22.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:22 smithi032 ceph-mon[39731]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:35:22.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:22 smithi032 ceph-mon[39731]: pgmap v212: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.3 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-12T05:35:24.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:24 smithi067 ceph-mon[49010]: pgmap v213: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 682 B/s wr, 4 op/s 2024-04-12T05:35:24.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:24 smithi032 ceph-mon[39731]: pgmap v213: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 682 B/s wr, 4 op/s 2024-04-12T05:35:25.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:25 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:35:25.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:25 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:35:26.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:26 smithi067 ceph-mon[49010]: pgmap v214: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 597 B/s wr, 3 op/s 2024-04-12T05:35:26.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:26 smithi032 ceph-mon[39731]: pgmap v214: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 597 B/s wr, 3 op/s 2024-04-12T05:35:28.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:28 smithi067 ceph-mon[49010]: pgmap v215: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-12T05:35:28.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:28 smithi032 ceph-mon[39731]: pgmap v215: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-12T05:35:30.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:30 smithi067 ceph-mon[49010]: pgmap v216: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-12T05:35:30.598 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:30 smithi032 ceph-mon[39731]: pgmap v216: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-12T05:35:32.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:32 smithi067 ceph-mon[49010]: pgmap v217: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-12T05:35:32.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:32 smithi032 ceph-mon[39731]: pgmap v217: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-12T05:35:34.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:34 smithi067 ceph-mon[49010]: pgmap v218: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 85 B/s wr, 3 op/s 2024-04-12T05:35:34.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:34 smithi032 ceph-mon[39731]: pgmap v218: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 85 B/s wr, 3 op/s 2024-04-12T05:35:36.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:36 smithi067 ceph-mon[49010]: pgmap v219: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:35:36.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:36 smithi032 ceph-mon[39731]: pgmap v219: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:35:38.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:38 smithi067 ceph-mon[49010]: pgmap v220: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:35:38.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:38 smithi032 ceph-mon[39731]: pgmap v220: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:35:40.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:40 smithi067 ceph-mon[49010]: pgmap v221: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:35:40.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:40 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:35:40.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:40 smithi032 ceph-mon[39731]: pgmap v221: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:35:40.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:40 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:35:42.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:42 smithi067 ceph-mon[49010]: pgmap v222: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:35:42.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:42 smithi032 ceph-mon[39731]: pgmap v222: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:35:44.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:44 smithi067 ceph-mon[49010]: pgmap v223: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:35:44.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:44 smithi032 ceph-mon[39731]: pgmap v223: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:35:46.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:46 smithi067 ceph-mon[49010]: pgmap v224: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:35:46.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:46 smithi032 ceph-mon[39731]: pgmap v224: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:35:48.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:48 smithi067 ceph-mon[49010]: pgmap v225: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:35:48.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:48 smithi032 ceph-mon[39731]: pgmap v225: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:35:50.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:50 smithi067 ceph-mon[49010]: pgmap v226: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:35:50.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:50 smithi032 ceph-mon[39731]: pgmap v226: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:35:52.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:52 smithi067 ceph-mon[49010]: pgmap v227: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:35:52.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:52 smithi032 ceph-mon[39731]: pgmap v227: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:35:53.065 INFO:tasks.workunit.client.1.smithi067.stderr:Updating files: 95% (11585/12088) Updating files: 96% (11605/12088) Updating files: 97% (11726/12088) Updating files: 98% (11847/12088) Updating files: 99% (11968/12088) Updating files: 100% (12088/12088) Updating files: 100% (12088/12088), done. 2024-04-12T05:35:53.448 INFO:tasks.workunit.client.1.smithi067.stderr:Note: switching to 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'. 2024-04-12T05:35:53.449 INFO:tasks.workunit.client.1.smithi067.stderr: 2024-04-12T05:35:53.449 INFO:tasks.workunit.client.1.smithi067.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-12T05:35:53.449 INFO:tasks.workunit.client.1.smithi067.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-12T05:35:53.449 INFO:tasks.workunit.client.1.smithi067.stderr:state without impacting any branches by switching back to a branch. 2024-04-12T05:35:53.449 INFO:tasks.workunit.client.1.smithi067.stderr: 2024-04-12T05:35:53.449 INFO:tasks.workunit.client.1.smithi067.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-12T05:35:53.449 INFO:tasks.workunit.client.1.smithi067.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-12T05:35:53.449 INFO:tasks.workunit.client.1.smithi067.stderr: 2024-04-12T05:35:53.449 INFO:tasks.workunit.client.1.smithi067.stderr: git switch -c 2024-04-12T05:35:53.449 INFO:tasks.workunit.client.1.smithi067.stderr: 2024-04-12T05:35:53.449 INFO:tasks.workunit.client.1.smithi067.stderr:Or undo this operation with: 2024-04-12T05:35:53.449 INFO:tasks.workunit.client.1.smithi067.stderr: 2024-04-12T05:35:53.449 INFO:tasks.workunit.client.1.smithi067.stderr: git switch - 2024-04-12T05:35:53.449 INFO:tasks.workunit.client.1.smithi067.stderr: 2024-04-12T05:35:53.450 INFO:tasks.workunit.client.1.smithi067.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-12T05:35:53.450 INFO:tasks.workunit.client.1.smithi067.stderr: 2024-04-12T05:35:53.450 INFO:tasks.workunit.client.1.smithi067.stderr:HEAD is now at d540ebaca6b Merge pull request #56672 from batrick/wip-65294-reef 2024-04-12T05:35:53.454 DEBUG:teuthology.orchestra.run.smithi067:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-04-12T05:35:53.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:53 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:35:53.570 INFO:tasks.workunit.client.1.smithi067.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-12T05:35:53.572 INFO:tasks.workunit.client.1.smithi067.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-12T05:35:53.572 INFO:tasks.workunit.client.1.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-12T05:35:53.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:53 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:35:54.081 INFO:tasks.workunit.client.1.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-12T05:35:54.139 INFO:tasks.workunit.client.1.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-12T05:35:54.178 INFO:tasks.workunit.client.1.smithi067.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-12T05:35:54.179 INFO:tasks.workunit.client.1.smithi067.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-12T05:35:54.180 INFO:tasks.workunit.client.1.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-12T05:35:54.217 INFO:tasks.workunit.client.1.smithi067.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-12T05:35:54.221 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-12T05:35:54.221 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-04-12T05:35:54.279 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-04-12T05:35:54.282 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-04-12T05:35:54.282 DEBUG:teuthology.orchestra.run.smithi067:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d540ebaca6b131a1dd560e7f69e024b133bbaa42 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-04-12T05:35:54.350 INFO:tasks.workunit.client.1.smithi067.stderr:+ mkdir -p fsstress 2024-04-12T05:35:54.352 INFO:tasks.workunit.client.1.smithi067.stderr:+ pushd fsstress 2024-04-12T05:35:54.354 INFO:tasks.workunit.client.1.smithi067.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-12T05:35:54.354 INFO:tasks.workunit.client.1.smithi067.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-04-12T05:35:54.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:54 smithi067 ceph-mon[49010]: pgmap v228: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:35:54.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:54 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:35:54.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:54 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:35:54.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:54 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:35:54.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:54 smithi032 ceph-mon[39731]: pgmap v228: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:35:54.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:54 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:35:54.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:54 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:35:54.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:54 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:35:55.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:55 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:35:55.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:55 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:35:55.916 INFO:tasks.workunit.client.1.smithi067.stderr:+ tar xzf ltp-full.tgz 2024-04-12T05:35:56.261 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:56 smithi032 ceph-mon[39731]: pgmap v229: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:35:56.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:56 smithi067 ceph-mon[49010]: pgmap v229: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:35:57.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:57.193+0000 7fc5e1ffb700 1 -- 172.21.15.32:0/2507194335 <== mgr.14208 v2:172.21.15.32:6800/595990262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7fc5ec10f920 con 0x7fc5d80692b0 2024-04-12T05:35:57.192 INFO:teuthology.orchestra.run.smithi032.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:35:57.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:57.195+0000 7fc5f3656700 1 -- 172.21.15.32:0/2507194335 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fc5d80692b0 msgr2=0x7fc5d806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:57.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:57.195+0000 7fc5f3656700 1 --2- 172.21.15.32:0/2507194335 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fc5d80692b0 0x7fc5d806b760 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8013680 tx=0x7fc5e8009e30 comp rx=0 tx=0).stop 2024-04-12T05:35:57.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:57.195+0000 7fc5f3656700 1 -- 172.21.15.32:0/2507194335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5ec0fdff0 msgr2=0x7fc5ec10de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:57.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:57.195+0000 7fc5f3656700 1 --2- 172.21.15.32:0/2507194335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5ec0fdff0 0x7fc5ec10de30 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc00eb40 tx=0x7fc5dc00b040 comp rx=0 tx=0).stop 2024-04-12T05:35:57.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:57.198+0000 7fc5f3656700 1 -- 172.21.15.32:0/2507194335 shutdown_connections 2024-04-12T05:35:57.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:57.198+0000 7fc5f3656700 1 --2- 172.21.15.32:0/2507194335 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fc5d80692b0 0x7fc5d806b760 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:35:57.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:57.198+0000 7fc5f3656700 1 --2- 172.21.15.32:0/2507194335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5ec0fce80 0x7fc5ec10d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:35:57.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:57.198+0000 7fc5f3656700 1 --2- 172.21.15.32:0/2507194335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5ec0fdff0 0x7fc5ec10de30 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:35:57.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:57.198+0000 7fc5f3656700 1 -- 172.21.15.32:0/2507194335 >> 172.21.15.32:0/2507194335 conn(0x7fc5ec0f8580 msgr2=0x7fc5ec0f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:57.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:57.198+0000 7fc5f3656700 1 -- 172.21.15.32:0/2507194335 shutdown_connections 2024-04-12T05:35:57.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:35:57.198+0000 7fc5f3656700 1 -- 172.21.15.32:0/2507194335 wait complete. 2024-04-12T05:35:58.100 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-04-12T05:35:58.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:58 smithi032 ceph-mon[39731]: pgmap v230: 65 pgs: 65 active+clean; 7.1 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 564 KiB/s wr, 10 op/s 2024-04-12T05:35:58.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:58 smithi032 ceph-mon[39731]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:35:58.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:58 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:35:58.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:58 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:35:58.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:58 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:35:58.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:58 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:35:58.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:58 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:35:58.492 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:35:58.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:58 smithi067 ceph-mon[49010]: pgmap v230: 65 pgs: 65 active+clean; 7.1 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 564 KiB/s wr, 10 op/s 2024-04-12T05:35:58.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:58 smithi067 ceph-mon[49010]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:35:58.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:58 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:35:58.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:58 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:35:58.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:58 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:35:58.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:58 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:35:58.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:58 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:35:59.225 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:35:59 smithi032 ceph-mon[39731]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:35:59.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:35:59 smithi067 ceph-mon[49010]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:36:00.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:00 smithi067 ceph-mon[49010]: pgmap v231: 65 pgs: 65 active+clean; 7.1 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 564 KiB/s wr, 9 op/s 2024-04-12T05:36:00.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:00 smithi032 ceph-mon[39731]: pgmap v231: 65 pgs: 65 active+clean; 7.1 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 564 KiB/s wr, 9 op/s 2024-04-12T05:36:02.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:02 smithi067 ceph-mon[49010]: pgmap v232: 65 pgs: 65 active+clean; 15 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 114 KiB/s rd, 1.2 MiB/s wr, 65 op/s 2024-04-12T05:36:02.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:02 smithi032 ceph-mon[39731]: pgmap v232: 65 pgs: 65 active+clean; 15 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 114 KiB/s rd, 1.2 MiB/s wr, 65 op/s 2024-04-12T05:36:04.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:04 smithi067 ceph-mon[49010]: pgmap v233: 65 pgs: 65 active+clean; 16 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 113 KiB/s rd, 1.4 MiB/s wr, 111 op/s 2024-04-12T05:36:04.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:04 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:36:04.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:04 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:36:04.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:04 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:36:04.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.542+0000 7fb61d838700 1 -- 172.21.15.32:0/2693351358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6180fcf70 msgr2=0x7fb6180fd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:04.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.542+0000 7fb61d838700 1 --2- 172.21.15.32:0/2693351358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6180fcf70 0x7fb6180fd3e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb608004940 tx=0x7fb608009e30 comp rx=0 tx=0).stop 2024-04-12T05:36:04.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.542+0000 7fb61d838700 1 -- 172.21.15.32:0/2693351358 shutdown_connections 2024-04-12T05:36:04.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.542+0000 7fb61d838700 1 --2- 172.21.15.32:0/2693351358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6180fcf70 0x7fb6180fd3e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:04.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.542+0000 7fb61d838700 1 --2- 172.21.15.32:0/2693351358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6180fbe00 0x7fb6180fc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:04.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.542+0000 7fb61d838700 1 -- 172.21.15.32:0/2693351358 >> 172.21.15.32:0/2693351358 conn(0x7fb6180f74e0 msgr2=0x7fb6180f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:04.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.542+0000 7fb61d838700 1 -- 172.21.15.32:0/2693351358 shutdown_connections 2024-04-12T05:36:04.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.542+0000 7fb61d838700 1 -- 172.21.15.32:0/2693351358 wait complete. 2024-04-12T05:36:04.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.543+0000 7fb61d838700 1 Processor -- start 2024-04-12T05:36:04.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.543+0000 7fb61d838700 1 -- start start 2024-04-12T05:36:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.544+0000 7fb61d838700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6180fbe00 0x7fb61810fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:36:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.544+0000 7fb61d838700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6180fcf70 0x7fb61810ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:36:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.544+0000 7fb61d838700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6181105d0 con 0x7fb6180fbe00 2024-04-12T05:36:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.544+0000 7fb61d838700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb618110e20 con 0x7fb6180fcf70 2024-04-12T05:36:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.544+0000 7fb616ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6180fbe00 0x7fb61810fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.544+0000 7fb616ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6180fbe00 0x7fb61810fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39034/0 (socket says 172.21.15.32:39034) 2024-04-12T05:36:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.544+0000 7fb616ffd700 1 -- 172.21.15.32:0/145053444 learned_addr learned my addr 172.21.15.32:0/145053444 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:36:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.544+0000 7fb6167fc700 1 --2- 172.21.15.32:0/145053444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6180fcf70 0x7fb61810ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.544+0000 7fb616ffd700 1 -- 172.21.15.32:0/145053444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6180fcf70 msgr2=0x7fb61810ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.544+0000 7fb616ffd700 1 --2- 172.21.15.32:0/145053444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6180fcf70 0x7fb61810ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.544+0000 7fb616ffd700 1 -- 172.21.15.32:0/145053444 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb608005040 con 0x7fb6180fbe00 2024-04-12T05:36:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.544+0000 7fb616ffd700 1 --2- 172.21.15.32:0/145053444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6180fbe00 0x7fb61810fa90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb60c00eb40 tx=0x7fb60c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.544+0000 7fb5ff7fe700 1 -- 172.21.15.32:0/145053444 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb60c007cc0 con 0x7fb6180fbe00 2024-04-12T05:36:04.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.545+0000 7fb61d838700 1 -- 172.21.15.32:0/145053444 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6181110a0 con 0x7fb6180fbe00 2024-04-12T05:36:04.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.545+0000 7fb5ff7fe700 1 -- 172.21.15.32:0/145053444 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb60c00f040 con 0x7fb6180fbe00 2024-04-12T05:36:04.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.545+0000 7fb5ff7fe700 1 -- 172.21.15.32:0/145053444 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb60c00c990 con 0x7fb6180fbe00 2024-04-12T05:36:04.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.545+0000 7fb61d838700 1 -- 172.21.15.32:0/145053444 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6181115e0 con 0x7fb6180fbe00 2024-04-12T05:36:04.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.546+0000 7fb61d838700 1 -- 172.21.15.32:0/145053444 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb618101900 con 0x7fb6180fbe00 2024-04-12T05:36:04.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.546+0000 7fb5ff7fe700 1 -- 172.21.15.32:0/145053444 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fb60c00cc00 con 0x7fb6180fbe00 2024-04-12T05:36:04.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.547+0000 7fb5ff7fe700 1 --2- 172.21.15.32:0/145053444 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fb6000692a0 0x7fb60006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:36:04.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.547+0000 7fb5ff7fe700 1 -- 172.21.15.32:0/145053444 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fb60c086da0 con 0x7fb6180fbe00 2024-04-12T05:36:04.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.547+0000 7fb6167fc700 1 --2- 172.21.15.32:0/145053444 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fb6000692a0 0x7fb60006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:04.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.547+0000 7fb6167fc700 1 --2- 172.21.15.32:0/145053444 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fb6000692a0 0x7fb60006b750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb608004890 tx=0x7fb608009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:04.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.552+0000 7fb5ff7fe700 1 -- 172.21.15.32:0/145053444 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fb60c04e190 con 0x7fb6180fbe00 2024-04-12T05:36:04.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:04 smithi032 ceph-mon[39731]: pgmap v233: 65 pgs: 65 active+clean; 16 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 113 KiB/s rd, 1.4 MiB/s wr, 111 op/s 2024-04-12T05:36:04.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:04 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:36:04.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:04 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:36:04.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:04 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:36:04.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.802+0000 7fb61d838700 1 -- 172.21.15.32:0/145053444 --> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb618111800 con 0x7fb6000692a0 2024-04-12T05:36:04.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.803+0000 7fb5ff7fe700 1 -- 172.21.15.32:0/145053444 <== mgr.14208 v2:172.21.15.32:6800/595990262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7fb618111800 con 0x7fb6000692a0 2024-04-12T05:36:04.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.805+0000 7fb61d838700 1 -- 172.21.15.32:0/145053444 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fb6000692a0 msgr2=0x7fb60006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:04.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.805+0000 7fb61d838700 1 --2- 172.21.15.32:0/145053444 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fb6000692a0 0x7fb60006b750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb608004890 tx=0x7fb608009e30 comp rx=0 tx=0).stop 2024-04-12T05:36:04.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.805+0000 7fb61d838700 1 -- 172.21.15.32:0/145053444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6180fbe00 msgr2=0x7fb61810fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:04.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.805+0000 7fb61d838700 1 --2- 172.21.15.32:0/145053444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6180fbe00 0x7fb61810fa90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb60c00eb40 tx=0x7fb60c00b040 comp rx=0 tx=0).stop 2024-04-12T05:36:04.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.805+0000 7fb61d838700 1 -- 172.21.15.32:0/145053444 shutdown_connections 2024-04-12T05:36:04.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.805+0000 7fb61d838700 1 --2- 172.21.15.32:0/145053444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6180fbe00 0x7fb61810fa90 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:04.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.805+0000 7fb61d838700 1 --2- 172.21.15.32:0/145053444 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fb6000692a0 0x7fb60006b750 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:04.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.805+0000 7fb61d838700 1 --2- 172.21.15.32:0/145053444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6180fcf70 0x7fb61810ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:04.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.805+0000 7fb61d838700 1 -- 172.21.15.32:0/145053444 >> 172.21.15.32:0/145053444 conn(0x7fb6180f74e0 msgr2=0x7fb618066420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:04.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.805+0000 7fb61d838700 1 -- 172.21.15.32:0/145053444 shutdown_connections 2024-04-12T05:36:04.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.805+0000 7fb61d838700 1 -- 172.21.15.32:0/145053444 wait complete. 2024-04-12T05:36:04.816 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:36:04.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.924+0000 7f43675eb700 1 -- 172.21.15.32:0/998476089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43600662e0 msgr2=0x7f43600ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:04.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.924+0000 7f43675eb700 1 --2- 172.21.15.32:0/998476089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43600662e0 0x7f43600ff9c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f435c004940 tx=0x7f435c009e30 comp rx=0 tx=0).stop 2024-04-12T05:36:04.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.925+0000 7f43675eb700 1 -- 172.21.15.32:0/998476089 shutdown_connections 2024-04-12T05:36:04.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.925+0000 7f43675eb700 1 --2- 172.21.15.32:0/998476089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43600fff00 0x7f4360102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:04.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.925+0000 7f43675eb700 1 --2- 172.21.15.32:0/998476089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43600662e0 0x7f43600ff9c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:04.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.925+0000 7f43675eb700 1 -- 172.21.15.32:0/998476089 >> 172.21.15.32:0/998476089 conn(0x7f43600f7430 msgr2=0x7f43600f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:04.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.925+0000 7f43675eb700 1 -- 172.21.15.32:0/998476089 shutdown_connections 2024-04-12T05:36:04.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.925+0000 7f43675eb700 1 -- 172.21.15.32:0/998476089 wait complete. 2024-04-12T05:36:04.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.925+0000 7f43675eb700 1 Processor -- start 2024-04-12T05:36:04.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.925+0000 7f43675eb700 1 -- start start 2024-04-12T05:36:04.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.926+0000 7f43675eb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43600662e0 0x7f436010d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:36:04.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.926+0000 7f43675eb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43600fff00 0x7f436010de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:36:04.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.926+0000 7f43675eb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436010e430 con 0x7f43600662e0 2024-04-12T05:36:04.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.926+0000 7f43675eb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436010ec80 con 0x7f43600fff00 2024-04-12T05:36:04.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.926+0000 7f4365387700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43600662e0 0x7f436010d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:04.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.926+0000 7f4365387700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43600662e0 0x7f436010d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39054/0 (socket says 172.21.15.32:39054) 2024-04-12T05:36:04.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.926+0000 7f4365387700 1 -- 172.21.15.32:0/1430365213 learned_addr learned my addr 172.21.15.32:0/1430365213 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:36:04.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.926+0000 7f4364b86700 1 --2- 172.21.15.32:0/1430365213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43600fff00 0x7f436010de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:04.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.926+0000 7f4365387700 1 -- 172.21.15.32:0/1430365213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43600fff00 msgr2=0x7f436010de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:04.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.926+0000 7f4365387700 1 --2- 172.21.15.32:0/1430365213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43600fff00 0x7f436010de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:04.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.926+0000 7f4365387700 1 -- 172.21.15.32:0/1430365213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435c005040 con 0x7f43600662e0 2024-04-12T05:36:04.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.926+0000 7f4364b86700 1 --2- 172.21.15.32:0/1430365213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43600fff00 0x7f436010de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T05:36:04.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.926+0000 7f4365387700 1 --2- 172.21.15.32:0/1430365213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43600662e0 0x7f436010d8f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f435c0137b0 tx=0x7f435c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:04.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.927+0000 7f4355ffb700 1 -- 172.21.15.32:0/1430365213 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f435c021070 con 0x7f43600662e0 2024-04-12T05:36:04.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.927+0000 7f43675eb700 1 -- 172.21.15.32:0/1430365213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f436010ef00 con 0x7f43600662e0 2024-04-12T05:36:04.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.927+0000 7f4355ffb700 1 -- 172.21.15.32:0/1430365213 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f435c01c400 con 0x7f43600662e0 2024-04-12T05:36:04.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.927+0000 7f4355ffb700 1 -- 172.21.15.32:0/1430365213 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f435c00e070 con 0x7f43600662e0 2024-04-12T05:36:04.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.927+0000 7f43675eb700 1 -- 172.21.15.32:0/1430365213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f436010f440 con 0x7f43600662e0 2024-04-12T05:36:04.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.928+0000 7f43675eb700 1 -- 172.21.15.32:0/1430365213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f436005f5f0 con 0x7f43600662e0 2024-04-12T05:36:04.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.929+0000 7f4355ffb700 1 -- 172.21.15.32:0/1430365213 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f435c00b760 con 0x7f43600662e0 2024-04-12T05:36:04.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.929+0000 7f4355ffb700 1 --2- 172.21.15.32:0/1430365213 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f434c0692a0 0x7f434c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:36:04.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.929+0000 7f4355ffb700 1 -- 172.21.15.32:0/1430365213 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f435c088450 con 0x7f43600662e0 2024-04-12T05:36:04.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.929+0000 7f4364b86700 1 --2- 172.21.15.32:0/1430365213 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f434c0692a0 0x7f434c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:04.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.930+0000 7f4364b86700 1 --2- 172.21.15.32:0/1430365213 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f434c0692a0 0x7f434c06b750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4350009b80 tx=0x7f435000b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:04.937+0000 7f4355ffb700 1 -- 172.21.15.32:0/1430365213 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f435c0581c0 con 0x7f43600662e0 2024-04-12T05:36:05.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.188+0000 7f43675eb700 1 -- 172.21.15.32:0/1430365213 --> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f436010f7c0 con 0x7f434c0692a0 2024-04-12T05:36:05.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.189+0000 7f4355ffb700 1 -- 172.21.15.32:0/1430365213 <== mgr.14208 v2:172.21.15.32:6800/595990262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f436010f7c0 con 0x7f434c0692a0 2024-04-12T05:36:05.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.191+0000 7f43675eb700 1 -- 172.21.15.32:0/1430365213 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f434c0692a0 msgr2=0x7f434c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:05.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.191+0000 7f43675eb700 1 --2- 172.21.15.32:0/1430365213 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f434c0692a0 0x7f434c06b750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4350009b80 tx=0x7f435000b040 comp rx=0 tx=0).stop 2024-04-12T05:36:05.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.192+0000 7f43675eb700 1 -- 172.21.15.32:0/1430365213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43600662e0 msgr2=0x7f436010d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:05.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.192+0000 7f43675eb700 1 --2- 172.21.15.32:0/1430365213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43600662e0 0x7f436010d8f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f435c0137b0 tx=0x7f435c009e30 comp rx=0 tx=0).stop 2024-04-12T05:36:05.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.192+0000 7f43675eb700 1 -- 172.21.15.32:0/1430365213 shutdown_connections 2024-04-12T05:36:05.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.192+0000 7f43675eb700 1 --2- 172.21.15.32:0/1430365213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43600662e0 0x7f436010d8f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:05.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.192+0000 7f43675eb700 1 --2- 172.21.15.32:0/1430365213 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f434c0692a0 0x7f434c06b750 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:05.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.192+0000 7f43675eb700 1 --2- 172.21.15.32:0/1430365213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43600fff00 0x7f436010de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:05.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.192+0000 7f43675eb700 1 -- 172.21.15.32:0/1430365213 >> 172.21.15.32:0/1430365213 conn(0x7f43600f7430 msgr2=0x7f43600f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:05.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.192+0000 7f43675eb700 1 -- 172.21.15.32:0/1430365213 shutdown_connections 2024-04-12T05:36:05.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.192+0000 7f43675eb700 1 -- 172.21.15.32:0/1430365213 wait complete. 2024-04-12T05:36:05.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.311+0000 7f4b95f10700 1 -- 172.21.15.32:0/1226295965 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b900fbcd0 msgr2=0x7f4b900fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:05.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.311+0000 7f4b95f10700 1 --2- 172.21.15.32:0/1226295965 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b900fbcd0 0x7f4b900fc0e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4b80004940 tx=0x7f4b80009e30 comp rx=0 tx=0).stop 2024-04-12T05:36:05.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.311+0000 7f4b95f10700 1 -- 172.21.15.32:0/1226295965 shutdown_connections 2024-04-12T05:36:05.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.311+0000 7f4b95f10700 1 --2- 172.21.15.32:0/1226295965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b900fcf80 0x7f4b900fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:05.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.311+0000 7f4b95f10700 1 --2- 172.21.15.32:0/1226295965 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b900fbcd0 0x7f4b900fc0e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:05.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.311+0000 7f4b95f10700 1 -- 172.21.15.32:0/1226295965 >> 172.21.15.32:0/1226295965 conn(0x7f4b900f7430 msgr2=0x7f4b900f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:05.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.311+0000 7f4b95f10700 1 -- 172.21.15.32:0/1226295965 shutdown_connections 2024-04-12T05:36:05.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.311+0000 7f4b95f10700 1 -- 172.21.15.32:0/1226295965 wait complete. 2024-04-12T05:36:05.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.312+0000 7f4b95f10700 1 Processor -- start 2024-04-12T05:36:05.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.312+0000 7f4b95f10700 1 -- start start 2024-04-12T05:36:05.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.313+0000 7f4b95f10700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b900fbcd0 0x7f4b9010c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:36:05.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.313+0000 7f4b95f10700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b900fcf80 0x7f4b9010cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:36:05.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.313+0000 7f4b95f10700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9010d370 con 0x7f4b900fcf80 2024-04-12T05:36:05.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.313+0000 7f4b95f10700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9010dbc0 con 0x7f4b900fbcd0 2024-04-12T05:36:05.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.313+0000 7f4b8effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b900fcf80 0x7f4b9010cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:05.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.313+0000 7f4b8effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b900fcf80 0x7f4b9010cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39074/0 (socket says 172.21.15.32:39074) 2024-04-12T05:36:05.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.313+0000 7f4b8f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b900fbcd0 0x7f4b9010c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:05.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.313+0000 7f4b8effd700 1 -- 172.21.15.32:0/395654467 learned_addr learned my addr 172.21.15.32:0/395654467 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:36:05.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.313+0000 7f4b8effd700 1 -- 172.21.15.32:0/395654467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b900fbcd0 msgr2=0x7f4b9010c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:05.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.313+0000 7f4b8effd700 1 --2- 172.21.15.32:0/395654467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b900fbcd0 0x7f4b9010c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:05.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.313+0000 7f4b8effd700 1 -- 172.21.15.32:0/395654467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b80005040 con 0x7f4b900fcf80 2024-04-12T05:36:05.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.313+0000 7f4b8effd700 1 --2- 172.21.15.32:0/395654467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b900fcf80 0x7f4b9010cd70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4b8400eaa0 tx=0x7f4b8400c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:05.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.313+0000 7f4b77fff700 1 -- 172.21.15.32:0/395654467 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b84007ea0 con 0x7f4b900fcf80 2024-04-12T05:36:05.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.314+0000 7f4b95f10700 1 -- 172.21.15.32:0/395654467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b9010de40 con 0x7f4b900fcf80 2024-04-12T05:36:05.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.314+0000 7f4b77fff700 1 -- 172.21.15.32:0/395654467 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b8400dba0 con 0x7f4b900fcf80 2024-04-12T05:36:05.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.314+0000 7f4b77fff700 1 -- 172.21.15.32:0/395654467 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b84009990 con 0x7f4b900fcf80 2024-04-12T05:36:05.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.314+0000 7f4b95f10700 1 -- 172.21.15.32:0/395654467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b9010e400 con 0x7f4b900fcf80 2024-04-12T05:36:05.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.315+0000 7f4b95f10700 1 -- 172.21.15.32:0/395654467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b9005f5f0 con 0x7f4b900fcf80 2024-04-12T05:36:05.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.315+0000 7f4b77fff700 1 -- 172.21.15.32:0/395654467 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f4b8400dd10 con 0x7f4b900fcf80 2024-04-12T05:36:05.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.316+0000 7f4b77fff700 1 --2- 172.21.15.32:0/395654467 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4b780692a0 0x7f4b7806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:36:05.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.316+0000 7f4b77fff700 1 -- 172.21.15.32:0/395654467 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f4b84087830 con 0x7f4b900fcf80 2024-04-12T05:36:05.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.316+0000 7f4b8f7fe700 1 --2- 172.21.15.32:0/395654467 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4b780692a0 0x7f4b7806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:05.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.316+0000 7f4b8f7fe700 1 --2- 172.21.15.32:0/395654467 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4b780692a0 0x7f4b7806b750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4b8000b3b0 tx=0x7f4b80009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:05.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.324+0000 7f4b77fff700 1 -- 172.21.15.32:0/395654467 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4b84057520 con 0x7f4b900fcf80 2024-04-12T05:36:05.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:05 smithi067 ceph-mon[49010]: Upgrade: Target is version 18.2.2-1187-gd540ebac (unknown) 2024-04-12T05:36:05.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:05 smithi067 ceph-mon[49010]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3, digests ['quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3'] 2024-04-12T05:36:05.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:05 smithi067 ceph-mon[49010]: Upgrade: Need to upgrade myself (mgr.smithi032.hrwxrh) 2024-04-12T05:36:05.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:05 smithi067 ceph-mon[49010]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 on smithi067 2024-04-12T05:36:05.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.540+0000 7f4b95f10700 1 -- 172.21.15.32:0/395654467 --> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b9010e7c0 con 0x7f4b780692a0 2024-04-12T05:36:05.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.558+0000 7f4b77fff700 1 -- 172.21.15.32:0/395654467 <== mgr.14208 v2:172.21.15.32:6800/595990262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7f4b9010e7c0 con 0x7f4b780692a0 2024-04-12T05:36:05.557 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:36:05.557 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (5m) 72s ago 7m 27.0M - 0.25.0 c8568f914cd2 a5c616791a7e 2024-04-12T05:36:05.557 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (7m) 72s ago 7m 6656k - 17.2.7-827-gd3b5060a 21de9b7369ad a9cfc00c311b 2024-04-12T05:36:05.558 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (6m) 78s ago 6m 6651k - 17.2.7-827-gd3b5060a 21de9b7369ad 1f2eea4542d9 2024-04-12T05:36:05.558 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (5m) 72s ago 7m 87.0M - 9.4.7 954c08fa6188 8e5a7ebd77b5 2024-04-12T05:36:05.558 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (99s) 72s ago 98s 11.4M - 17.2.7-827-gd3b5060a 21de9b7369ad e7ada8e5259c 2024-04-12T05:36:05.558 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (93s) 72s ago 92s 14.0M - 17.2.7-827-gd3b5060a 21de9b7369ad 55124da9afdb 2024-04-12T05:36:05.558 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (90s) 78s ago 89s 13.3M - 17.2.7-827-gd3b5060a 21de9b7369ad 0c295d31a4a6 2024-04-12T05:36:05.558 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (96s) 78s ago 95s 16.8M - 17.2.7-827-gd3b5060a 21de9b7369ad 6eecd7deaac2 2024-04-12T05:36:05.558 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:9283 running (8m) 72s ago 8m 483M - 17.2.7-827-gd3b5060a 21de9b7369ad fdf859c0313c 2024-04-12T05:36:05.558 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283 running (6m) 78s ago 6m 435M - 17.2.7-827-gd3b5060a 21de9b7369ad 117ab3f1dbe0 2024-04-12T05:36:05.558 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (8m) 72s ago 8m 42.4M 2048M 17.2.7-827-gd3b5060a 21de9b7369ad 75aad95b984d 2024-04-12T05:36:05.558 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (6m) 78s ago 6m 31.6M 2048M 17.2.7-827-gd3b5060a 21de9b7369ad 157be8a8c6d8 2024-04-12T05:36:05.558 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (7m) 72s ago 7m 15.8M - 1.5.0 0da6a335fe13 ca3585dd7b65 2024-04-12T05:36:05.558 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (6m) 78s ago 6m 15.6M - 1.5.0 0da6a335fe13 c324d9436cca 2024-04-12T05:36:05.558 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (5m) 72s ago 5m 46.9M 4300M 17.2.7-827-gd3b5060a 21de9b7369ad 97446e42ceba 2024-04-12T05:36:05.558 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (5m) 72s ago 5m 47.6M 4300M 17.2.7-827-gd3b5060a 21de9b7369ad bf9f9a2db2d5 2024-04-12T05:36:05.559 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (4m) 72s ago 4m 44.1M 4300M 17.2.7-827-gd3b5060a 21de9b7369ad f055cf1ca665 2024-04-12T05:36:05.559 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (4m) 78s ago 4m 44.8M 5323M 17.2.7-827-gd3b5060a 21de9b7369ad 61b851235351 2024-04-12T05:36:05.559 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (3m) 78s ago 3m 42.6M 5323M 17.2.7-827-gd3b5060a 21de9b7369ad 2c84d9047e9c 2024-04-12T05:36:05.559 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (3m) 78s ago 3m 44.4M 5323M 17.2.7-827-gd3b5060a 21de9b7369ad e54839bb6055 2024-04-12T05:36:05.559 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (5m) 72s ago 6m 56.5M - 2.43.0 a07b618ecd1d 996307aec3d2 2024-04-12T05:36:05.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.561+0000 7f4b95f10700 1 -- 172.21.15.32:0/395654467 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4b780692a0 msgr2=0x7f4b7806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:05.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.561+0000 7f4b95f10700 1 --2- 172.21.15.32:0/395654467 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4b780692a0 0x7f4b7806b750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4b8000b3b0 tx=0x7f4b80009e30 comp rx=0 tx=0).stop 2024-04-12T05:36:05.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.561+0000 7f4b95f10700 1 -- 172.21.15.32:0/395654467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b900fcf80 msgr2=0x7f4b9010cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:05.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.561+0000 7f4b95f10700 1 --2- 172.21.15.32:0/395654467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b900fcf80 0x7f4b9010cd70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4b8400eaa0 tx=0x7f4b8400c0f0 comp rx=0 tx=0).stop 2024-04-12T05:36:05.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.561+0000 7f4b95f10700 1 -- 172.21.15.32:0/395654467 shutdown_connections 2024-04-12T05:36:05.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.561+0000 7f4b95f10700 1 --2- 172.21.15.32:0/395654467 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f4b780692a0 0x7f4b7806b750 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:05.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.561+0000 7f4b95f10700 1 --2- 172.21.15.32:0/395654467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b900fbcd0 0x7f4b9010c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:05.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.561+0000 7f4b95f10700 1 --2- 172.21.15.32:0/395654467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b900fcf80 0x7f4b9010cd70 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:05.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.561+0000 7f4b95f10700 1 -- 172.21.15.32:0/395654467 >> 172.21.15.32:0/395654467 conn(0x7f4b900f7430 msgr2=0x7f4b900f8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:05.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.562+0000 7f4b95f10700 1 -- 172.21.15.32:0/395654467 shutdown_connections 2024-04-12T05:36:05.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.562+0000 7f4b95f10700 1 -- 172.21.15.32:0/395654467 wait complete. 2024-04-12T05:36:05.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:05 smithi032 ceph-mon[39731]: Upgrade: Target is version 18.2.2-1187-gd540ebac (unknown) 2024-04-12T05:36:05.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:05 smithi032 ceph-mon[39731]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3, digests ['quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3'] 2024-04-12T05:36:05.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:05 smithi032 ceph-mon[39731]: Upgrade: Need to upgrade myself (mgr.smithi032.hrwxrh) 2024-04-12T05:36:05.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:05 smithi032 ceph-mon[39731]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 on smithi067 2024-04-12T05:36:05.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.708+0000 7fe68e728700 1 -- 172.21.15.32:0/2641058387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6880feec0 msgr2=0x7fe6881012a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:05.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.708+0000 7fe68e728700 1 --2- 172.21.15.32:0/2641058387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6880feec0 0x7fe6881012a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe67c004940 tx=0x7fe67c009e30 comp rx=0 tx=0).stop 2024-04-12T05:36:05.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.709+0000 7fe68e728700 1 -- 172.21.15.32:0/2641058387 shutdown_connections 2024-04-12T05:36:05.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.709+0000 7fe68e728700 1 --2- 172.21.15.32:0/2641058387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6880feec0 0x7fe6881012a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:05.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.709+0000 7fe68e728700 1 --2- 172.21.15.32:0/2641058387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6880fc5a0 0x7fe6880fe980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:05.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.709+0000 7fe68e728700 1 -- 172.21.15.32:0/2641058387 >> 172.21.15.32:0/2641058387 conn(0x7fe6880f6770 msgr2=0x7fe6880f8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:05.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.709+0000 7fe68e728700 1 -- 172.21.15.32:0/2641058387 shutdown_connections 2024-04-12T05:36:05.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.709+0000 7fe68e728700 1 -- 172.21.15.32:0/2641058387 wait complete. 2024-04-12T05:36:05.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.710+0000 7fe68e728700 1 Processor -- start 2024-04-12T05:36:05.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.710+0000 7fe68e728700 1 -- start start 2024-04-12T05:36:05.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.710+0000 7fe68e728700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6880fc5a0 0x7fe6880fb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:36:05.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.710+0000 7fe68e728700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6880feec0 0x7fe6880fba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:36:05.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.710+0000 7fe68e728700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6880fc0a0 con 0x7fe6880feec0 2024-04-12T05:36:05.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.710+0000 7fe68e728700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe688119290 con 0x7fe6880fc5a0 2024-04-12T05:36:05.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.710+0000 7fe687fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6880fc5a0 0x7fe6880fb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:05.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.710+0000 7fe687fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6880fc5a0 0x7fe6880fb4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.32:58686/0 (socket says 172.21.15.32:58686) 2024-04-12T05:36:05.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.710+0000 7fe687fff700 1 -- 172.21.15.32:0/1860913221 learned_addr learned my addr 172.21.15.32:0/1860913221 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:36:05.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.710+0000 7fe6877fe700 1 --2- 172.21.15.32:0/1860913221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6880feec0 0x7fe6880fba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:05.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.711+0000 7fe687fff700 1 -- 172.21.15.32:0/1860913221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6880feec0 msgr2=0x7fe6880fba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:05.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.711+0000 7fe687fff700 1 --2- 172.21.15.32:0/1860913221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6880feec0 0x7fe6880fba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:05.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.711+0000 7fe687fff700 1 -- 172.21.15.32:0/1860913221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67c005040 con 0x7fe6880fc5a0 2024-04-12T05:36:05.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.711+0000 7fe687fff700 1 --2- 172.21.15.32:0/1860913221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6880fc5a0 0x7fe6880fb4d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe67800eb40 tx=0x7fe67800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:05.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.711+0000 7fe684ff9700 1 -- 172.21.15.32:0/1860913221 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe678007cc0 con 0x7fe6880fc5a0 2024-04-12T05:36:05.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.711+0000 7fe684ff9700 1 -- 172.21.15.32:0/1860913221 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe67800f040 con 0x7fe6880fc5a0 2024-04-12T05:36:05.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.711+0000 7fe684ff9700 1 -- 172.21.15.32:0/1860913221 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe67800c9f0 con 0x7fe6880fc5a0 2024-04-12T05:36:05.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.711+0000 7fe68e728700 1 -- 172.21.15.32:0/1860913221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe688119510 con 0x7fe6880fc5a0 2024-04-12T05:36:05.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.712+0000 7fe68e728700 1 -- 172.21.15.32:0/1860913221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe688119ad0 con 0x7fe6880fc5a0 2024-04-12T05:36:05.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.713+0000 7fe68e728700 1 -- 172.21.15.32:0/1860913221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe68804c710 con 0x7fe6880fc5a0 2024-04-12T05:36:05.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.714+0000 7fe684ff9700 1 -- 172.21.15.32:0/1860913221 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fe6780090d0 con 0x7fe6880fc5a0 2024-04-12T05:36:05.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.714+0000 7fe684ff9700 1 --2- 172.21.15.32:0/1860913221 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fe6700692a0 0x7fe67006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:36:05.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.714+0000 7fe684ff9700 1 -- 172.21.15.32:0/1860913221 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fe678086c20 con 0x7fe6880fc5a0 2024-04-12T05:36:05.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.714+0000 7fe6877fe700 1 --2- 172.21.15.32:0/1860913221 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fe6700692a0 0x7fe67006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:05.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.715+0000 7fe6877fe700 1 --2- 172.21.15.32:0/1860913221 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fe6700692a0 0x7fe67006b750 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe67c00b0d0 tx=0x7fe67c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:05.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:05.722+0000 7fe684ff9700 1 -- 172.21.15.32:0/1860913221 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fe67804e190 con 0x7fe6880fc5a0 2024-04-12T05:36:06.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.098+0000 7fe68e728700 1 -- 172.21.15.32:0/1860913221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe688119ed0 con 0x7fe6880fc5a0 2024-04-12T05:36:06.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.098+0000 7fe684ff9700 1 -- 172.21.15.32:0/1860913221 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7fe678052fb0 con 0x7fe6880fc5a0 2024-04-12T05:36:06.098 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:36:06.098 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:36:06.098 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 2 2024-04-12T05:36:06.098 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:36:06.098 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:36:06.098 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 2 2024-04-12T05:36:06.098 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:36:06.098 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:36:06.099 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 6 2024-04-12T05:36:06.099 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:36:06.099 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:36:06.099 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 4 2024-04-12T05:36:06.099 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:36:06.099 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:36:06.099 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 14 2024-04-12T05:36:06.099 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:36:06.099 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:36:06.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.101+0000 7fe68e728700 1 -- 172.21.15.32:0/1860913221 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fe6700692a0 msgr2=0x7fe67006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:06.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.101+0000 7fe68e728700 1 --2- 172.21.15.32:0/1860913221 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fe6700692a0 0x7fe67006b750 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe67c00b0d0 tx=0x7fe67c009e30 comp rx=0 tx=0).stop 2024-04-12T05:36:06.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.101+0000 7fe68e728700 1 -- 172.21.15.32:0/1860913221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6880fc5a0 msgr2=0x7fe6880fb4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:06.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.101+0000 7fe68e728700 1 --2- 172.21.15.32:0/1860913221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6880fc5a0 0x7fe6880fb4d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe67800eb40 tx=0x7fe67800b040 comp rx=0 tx=0).stop 2024-04-12T05:36:06.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.101+0000 7fe68e728700 1 -- 172.21.15.32:0/1860913221 shutdown_connections 2024-04-12T05:36:06.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.101+0000 7fe68e728700 1 --2- 172.21.15.32:0/1860913221 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fe6700692a0 0x7fe67006b750 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:06.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.101+0000 7fe68e728700 1 --2- 172.21.15.32:0/1860913221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6880fc5a0 0x7fe6880fb4d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:06.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.101+0000 7fe68e728700 1 --2- 172.21.15.32:0/1860913221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6880feec0 0x7fe6880fba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:06.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.101+0000 7fe68e728700 1 -- 172.21.15.32:0/1860913221 >> 172.21.15.32:0/1860913221 conn(0x7fe6880f6770 msgr2=0x7fe6880f6b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:06.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.101+0000 7fe68e728700 1 -- 172.21.15.32:0/1860913221 shutdown_connections 2024-04-12T05:36:06.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.101+0000 7fe68e728700 1 -- 172.21.15.32:0/1860913221 wait complete. 2024-04-12T05:36:06.162 INFO:tasks.workunit.client.0.smithi032.stderr:Updating files: 82% (9949/12088) Updating files: 83% (10034/12088) Updating files: 84% (10154/12088) Updating files: 85% (10275/12088) Updating files: 86% (10396/12088) Updating files: 87% (10517/12088) Updating files: 88% (10638/12088) Updating files: 89% (10759/12088) Updating files: 90% (10880/12088) Updating files: 91% (11001/12088) Updating files: 92% (11121/12088) Updating files: 93% (11242/12088) Updating files: 94% (11363/12088) Updating files: 95% (11484/12088) Updating files: 96% (11605/12088) Updating files: 97% (11726/12088) Updating files: 98% (11847/12088) Updating files: 99% (11968/12088) Updating files: 100% (12088/12088) Updating files: 100% (12088/12088), done. 2024-04-12T05:36:06.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.229+0000 7f991fe35700 1 -- 172.21.15.32:0/3709616286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99180fcf80 msgr2=0x7f99180fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:06.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.229+0000 7f991fe35700 1 --2- 172.21.15.32:0/3709616286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99180fcf80 0x7f99180fd3f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9908004940 tx=0x7f9908009e30 comp rx=0 tx=0).stop 2024-04-12T05:36:06.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.229+0000 7f991fe35700 1 -- 172.21.15.32:0/3709616286 shutdown_connections 2024-04-12T05:36:06.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.229+0000 7f991fe35700 1 --2- 172.21.15.32:0/3709616286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99180fcf80 0x7f99180fd3f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:06.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.229+0000 7f991fe35700 1 --2- 172.21.15.32:0/3709616286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99180fbcd0 0x7f99180fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:06.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.229+0000 7f991fe35700 1 -- 172.21.15.32:0/3709616286 >> 172.21.15.32:0/3709616286 conn(0x7f99180f7430 msgr2=0x7f99180f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:06.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.229+0000 7f991fe35700 1 -- 172.21.15.32:0/3709616286 shutdown_connections 2024-04-12T05:36:06.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.229+0000 7f991fe35700 1 -- 172.21.15.32:0/3709616286 wait complete. 2024-04-12T05:36:06.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.230+0000 7f991fe35700 1 Processor -- start 2024-04-12T05:36:06.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.230+0000 7f991fe35700 1 -- start start 2024-04-12T05:36:06.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.230+0000 7f991fe35700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99180fbcd0 0x7f991810d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:36:06.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.230+0000 7f991fe35700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99180fcf80 0x7f991810dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:36:06.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.230+0000 7f991fe35700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991810e310 con 0x7f99180fbcd0 2024-04-12T05:36:06.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.230+0000 7f991fe35700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991810e480 con 0x7f99180fcf80 2024-04-12T05:36:06.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.230+0000 7f991dbd1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99180fbcd0 0x7f991810d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:06.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.230+0000 7f991dbd1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99180fbcd0 0x7f991810d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39126/0 (socket says 172.21.15.32:39126) 2024-04-12T05:36:06.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.230+0000 7f991dbd1700 1 -- 172.21.15.32:0/3583919773 learned_addr learned my addr 172.21.15.32:0/3583919773 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:36:06.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.230+0000 7f991d3d0700 1 --2- 172.21.15.32:0/3583919773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99180fcf80 0x7f991810dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:06.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.231+0000 7f991dbd1700 1 -- 172.21.15.32:0/3583919773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99180fcf80 msgr2=0x7f991810dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:06.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.231+0000 7f991dbd1700 1 --2- 172.21.15.32:0/3583919773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99180fcf80 0x7f991810dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:06.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.231+0000 7f991dbd1700 1 -- 172.21.15.32:0/3583919773 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9908005040 con 0x7f99180fbcd0 2024-04-12T05:36:06.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.231+0000 7f991dbd1700 1 --2- 172.21.15.32:0/3583919773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99180fbcd0 0x7f991810d7d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f991400eb40 tx=0x7f991400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:06.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.231+0000 7f990e7fc700 1 -- 172.21.15.32:0/3583919773 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9914007cc0 con 0x7f99180fbcd0 2024-04-12T05:36:06.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.231+0000 7f990e7fc700 1 -- 172.21.15.32:0/3583919773 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f991400f040 con 0x7f99180fbcd0 2024-04-12T05:36:06.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.231+0000 7f991fe35700 1 -- 172.21.15.32:0/3583919773 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9918112e80 con 0x7f99180fbcd0 2024-04-12T05:36:06.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.231+0000 7f990e7fc700 1 -- 172.21.15.32:0/3583919773 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f991400c990 con 0x7f99180fbcd0 2024-04-12T05:36:06.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.231+0000 7f991fe35700 1 -- 172.21.15.32:0/3583919773 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99181133b0 con 0x7f99180fbcd0 2024-04-12T05:36:06.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.233+0000 7f991fe35700 1 -- 172.21.15.32:0/3583919773 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f991804c710 con 0x7f99180fbcd0 2024-04-12T05:36:06.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.233+0000 7f990e7fc700 1 -- 172.21.15.32:0/3583919773 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f991400cc00 con 0x7f99180fbcd0 2024-04-12T05:36:06.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.233+0000 7f990e7fc700 1 --2- 172.21.15.32:0/3583919773 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f99040692a0 0x7f990406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:36:06.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.233+0000 7f990e7fc700 1 -- 172.21.15.32:0/3583919773 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f9914086da0 con 0x7f99180fbcd0 2024-04-12T05:36:06.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.233+0000 7f991d3d0700 1 --2- 172.21.15.32:0/3583919773 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f99040692a0 0x7f990406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:06.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.234+0000 7f991d3d0700 1 --2- 172.21.15.32:0/3583919773 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f99040692a0 0x7f990406b750 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9908013680 tx=0x7f9908009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:06.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.240+0000 7f990e7fc700 1 -- 172.21.15.32:0/3583919773 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f991404e190 con 0x7f99180fbcd0 2024-04-12T05:36:06.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:06 smithi067 ceph-mon[49010]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:06.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:06 smithi067 ceph-mon[49010]: pgmap v234: 65 pgs: 65 active+clean; 17 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 113 KiB/s rd, 1.4 MiB/s wr, 123 op/s 2024-04-12T05:36:06.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:06 smithi067 ceph-mon[49010]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:06.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:06 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/1860913221' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:36:06.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.539+0000 7f991fe35700 1 -- 172.21.15.32:0/3583919773 --> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f99181135a0 con 0x7f99040692a0 2024-04-12T05:36:06.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.540+0000 7f990e7fc700 1 -- 172.21.15.32:0/3583919773 <== mgr.14208 v2:172.21.15.32:6800/595990262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f99181135a0 con 0x7f99040692a0 2024-04-12T05:36:06.539 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:36:06.539 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:36:06.539 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:36:06.539 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-12T05:36:06.539 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [], 2024-04-12T05:36:06.539 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "0/2 daemons upgraded", 2024-04-12T05:36:06.539 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 image on host smithi067", 2024-04-12T05:36:06.539 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:36:06.540 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:36:06.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.543+0000 7f991fe35700 1 -- 172.21.15.32:0/3583919773 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f99040692a0 msgr2=0x7f990406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:06.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.543+0000 7f991fe35700 1 --2- 172.21.15.32:0/3583919773 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f99040692a0 0x7f990406b750 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9908013680 tx=0x7f9908009e30 comp rx=0 tx=0).stop 2024-04-12T05:36:06.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.543+0000 7f991fe35700 1 -- 172.21.15.32:0/3583919773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99180fbcd0 msgr2=0x7f991810d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:06.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.543+0000 7f991fe35700 1 --2- 172.21.15.32:0/3583919773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99180fbcd0 0x7f991810d7d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f991400eb40 tx=0x7f991400b040 comp rx=0 tx=0).stop 2024-04-12T05:36:06.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.543+0000 7f991fe35700 1 -- 172.21.15.32:0/3583919773 shutdown_connections 2024-04-12T05:36:06.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.543+0000 7f991fe35700 1 --2- 172.21.15.32:0/3583919773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99180fbcd0 0x7f991810d7d0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:06.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.543+0000 7f991fe35700 1 --2- 172.21.15.32:0/3583919773 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f99040692a0 0x7f990406b750 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:06.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.543+0000 7f991fe35700 1 --2- 172.21.15.32:0/3583919773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99180fcf80 0x7f991810dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:06.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.543+0000 7f991fe35700 1 -- 172.21.15.32:0/3583919773 >> 172.21.15.32:0/3583919773 conn(0x7f99180f7430 msgr2=0x7f99180f8b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:06.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.544+0000 7f991fe35700 1 -- 172.21.15.32:0/3583919773 shutdown_connections 2024-04-12T05:36:06.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:06.544+0000 7f991fe35700 1 -- 172.21.15.32:0/3583919773 wait complete. 2024-04-12T05:36:06.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:06 smithi032 ceph-mon[39731]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:06.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:06 smithi032 ceph-mon[39731]: pgmap v234: 65 pgs: 65 active+clean; 17 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 113 KiB/s rd, 1.4 MiB/s wr, 123 op/s 2024-04-12T05:36:06.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:06 smithi032 ceph-mon[39731]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:06.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:06 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/1860913221' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:36:06.658 INFO:tasks.workunit.client.0.smithi032.stderr:Note: switching to 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'. 2024-04-12T05:36:06.658 INFO:tasks.workunit.client.0.smithi032.stderr: 2024-04-12T05:36:06.658 INFO:tasks.workunit.client.0.smithi032.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-12T05:36:06.658 INFO:tasks.workunit.client.0.smithi032.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-12T05:36:06.658 INFO:tasks.workunit.client.0.smithi032.stderr:state without impacting any branches by switching back to a branch. 2024-04-12T05:36:06.658 INFO:tasks.workunit.client.0.smithi032.stderr: 2024-04-12T05:36:06.658 INFO:tasks.workunit.client.0.smithi032.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-12T05:36:06.658 INFO:tasks.workunit.client.0.smithi032.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-12T05:36:06.658 INFO:tasks.workunit.client.0.smithi032.stderr: 2024-04-12T05:36:06.659 INFO:tasks.workunit.client.0.smithi032.stderr: git switch -c 2024-04-12T05:36:06.659 INFO:tasks.workunit.client.0.smithi032.stderr: 2024-04-12T05:36:06.659 INFO:tasks.workunit.client.0.smithi032.stderr:Or undo this operation with: 2024-04-12T05:36:06.659 INFO:tasks.workunit.client.0.smithi032.stderr: 2024-04-12T05:36:06.659 INFO:tasks.workunit.client.0.smithi032.stderr: git switch - 2024-04-12T05:36:06.659 INFO:tasks.workunit.client.0.smithi032.stderr: 2024-04-12T05:36:06.659 INFO:tasks.workunit.client.0.smithi032.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-12T05:36:06.659 INFO:tasks.workunit.client.0.smithi032.stderr: 2024-04-12T05:36:06.659 INFO:tasks.workunit.client.0.smithi032.stderr:HEAD is now at d540ebaca6b Merge pull request #56672 from batrick/wip-65294-reef 2024-04-12T05:36:06.664 DEBUG:teuthology.orchestra.run.smithi032:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-12T05:36:06.797 INFO:tasks.workunit.client.0.smithi032.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-12T05:36:06.799 INFO:tasks.workunit.client.0.smithi032.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-12T05:36:06.799 INFO:tasks.workunit.client.0.smithi032.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-12T05:36:07.313 INFO:tasks.workunit.client.0.smithi032.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-12T05:36:07.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:07 smithi032 ceph-mon[39731]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:07.377 INFO:tasks.workunit.client.0.smithi032.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-12T05:36:07.419 INFO:tasks.workunit.client.0.smithi032.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-12T05:36:07.421 INFO:tasks.workunit.client.0.smithi032.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-12T05:36:07.421 INFO:tasks.workunit.client.0.smithi032.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-12T05:36:07.470 INFO:tasks.workunit.client.0.smithi032.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-12T05:36:07.473 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-12T05:36:07.473 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-12T05:36:07.492 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-04-12T05:36:07.493 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-04-12T05:36:07.493 DEBUG:teuthology.orchestra.run.smithi032:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d540ebaca6b131a1dd560e7f69e024b133bbaa42 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-04-12T05:36:07.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:07 smithi067 ceph-mon[49010]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:07.562 INFO:tasks.workunit.client.0.smithi032.stderr:+ mkdir -p fsstress 2024-04-12T05:36:07.564 INFO:tasks.workunit.client.0.smithi032.stderr:+ pushd fsstress 2024-04-12T05:36:07.565 INFO:tasks.workunit.client.0.smithi032.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-12T05:36:07.566 INFO:tasks.workunit.client.0.smithi032.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-04-12T05:36:08.403 INFO:tasks.workunit.client.0.smithi032.stderr:+ tar xzf ltp-full.tgz 2024-04-12T05:36:08.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:08 smithi067 ceph-mon[49010]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:08.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:08 smithi067 ceph-mon[49010]: pgmap v235: 65 pgs: 65 active+clean; 28 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 685 KiB/s rd, 2.4 MiB/s wr, 221 op/s 2024-04-12T05:36:08.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:08 smithi032 ceph-mon[39731]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:08.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:08 smithi032 ceph-mon[39731]: pgmap v235: 65 pgs: 65 active+clean; 28 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 685 KiB/s rd, 2.4 MiB/s wr, 221 op/s 2024-04-12T05:36:09.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:09 smithi067 ceph-mon[49010]: fsmap cephfs:2 {0=cephfs.smithi032.pndtry=up:active,1=cephfs.smithi067.zesmgy=up:active} 2 up:standby-replay 2024-04-12T05:36:09.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:09 smithi032 ceph-mon[39731]: fsmap cephfs:2 {0=cephfs.smithi032.pndtry=up:active,1=cephfs.smithi067.zesmgy=up:active} 2 up:standby-replay 2024-04-12T05:36:10.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:10 smithi067 ceph-mon[49010]: pgmap v236: 65 pgs: 65 active+clean; 28 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 684 KiB/s rd, 1.9 MiB/s wr, 214 op/s 2024-04-12T05:36:10.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:10 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:36:10.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:10 smithi032 ceph-mon[39731]: pgmap v236: 65 pgs: 65 active+clean; 28 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 684 KiB/s rd, 1.9 MiB/s wr, 214 op/s 2024-04-12T05:36:10.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:10 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:36:12.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:12 smithi067 ceph-mon[49010]: pgmap v237: 65 pgs: 65 active+clean; 45 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 1.0 MiB/s rd, 3.4 MiB/s wr, 299 op/s 2024-04-12T05:36:12.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:12 smithi032 ceph-mon[39731]: pgmap v237: 65 pgs: 65 active+clean; 45 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 1.0 MiB/s rd, 3.4 MiB/s wr, 299 op/s 2024-04-12T05:36:14.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:14 smithi067 ceph-mon[49010]: pgmap v238: 65 pgs: 65 active+clean; 57 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 999 KiB/s rd, 3.7 MiB/s wr, 320 op/s 2024-04-12T05:36:14.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:14 smithi032 ceph-mon[39731]: pgmap v238: 65 pgs: 65 active+clean; 57 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 999 KiB/s rd, 3.7 MiB/s wr, 320 op/s 2024-04-12T05:36:16.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:16 smithi067 ceph-mon[49010]: pgmap v239: 65 pgs: 65 active+clean; 58 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 999 KiB/s rd, 3.6 MiB/s wr, 291 op/s 2024-04-12T05:36:16.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:16 smithi032 ceph-mon[39731]: pgmap v239: 65 pgs: 65 active+clean; 58 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 999 KiB/s rd, 3.6 MiB/s wr, 291 op/s 2024-04-12T05:36:18.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:18 smithi067 ceph-mon[49010]: pgmap v240: 65 pgs: 65 active+clean; 71 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 4.8 MiB/s wr, 409 op/s 2024-04-12T05:36:18.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:18 smithi032 ceph-mon[39731]: pgmap v240: 65 pgs: 65 active+clean; 71 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 4.8 MiB/s wr, 409 op/s 2024-04-12T05:36:20.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:20 smithi067 ceph-mon[49010]: pgmap v241: 65 pgs: 65 active+clean; 71 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 3.8 MiB/s wr, 312 op/s 2024-04-12T05:36:20.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:20 smithi032 ceph-mon[39731]: pgmap v241: 65 pgs: 65 active+clean; 71 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 3.8 MiB/s wr, 312 op/s 2024-04-12T05:36:22.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:22 smithi067 ceph-mon[49010]: pgmap v242: 65 pgs: 65 active+clean; 80 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 428 op/s 2024-04-12T05:36:22.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:22 smithi032 ceph-mon[39731]: pgmap v242: 65 pgs: 65 active+clean; 80 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 428 op/s 2024-04-12T05:36:24.586 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:24 smithi067 ceph-mon[49010]: pgmap v243: 65 pgs: 65 active+clean; 87 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 438 op/s 2024-04-12T05:36:24.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:24 smithi032 ceph-mon[39731]: pgmap v243: 65 pgs: 65 active+clean; 87 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 438 op/s 2024-04-12T05:36:25.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:25 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:36:25.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:25 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:36:26.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:26 smithi032 ceph-mon[39731]: pgmap v244: 65 pgs: 65 active+clean; 90 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 3.0 MiB/s wr, 387 op/s 2024-04-12T05:36:26.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:26 smithi067 ceph-mon[49010]: pgmap v244: 65 pgs: 65 active+clean; 90 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 3.0 MiB/s wr, 387 op/s 2024-04-12T05:36:28.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:28 smithi032 ceph-mon[39731]: pgmap v245: 65 pgs: 65 active+clean; 111 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 2.2 MiB/s rd, 4.7 MiB/s wr, 537 op/s 2024-04-12T05:36:28.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:28 smithi067 ceph-mon[49010]: pgmap v245: 65 pgs: 65 active+clean; 111 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 2.2 MiB/s rd, 4.7 MiB/s wr, 537 op/s 2024-04-12T05:36:30.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:30 smithi032 ceph-mon[39731]: pgmap v246: 65 pgs: 65 active+clean; 111 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 3.5 MiB/s wr, 406 op/s 2024-04-12T05:36:30.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:30 smithi067 ceph-mon[49010]: pgmap v246: 65 pgs: 65 active+clean; 111 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 3.5 MiB/s wr, 406 op/s 2024-04-12T05:36:32.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:32 smithi032 ceph-mon[39731]: pgmap v247: 65 pgs: 65 active+clean; 128 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.4 MiB/s rd, 5.0 MiB/s wr, 518 op/s 2024-04-12T05:36:32.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:32 smithi067 ceph-mon[49010]: pgmap v247: 65 pgs: 65 active+clean; 128 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.4 MiB/s rd, 5.0 MiB/s wr, 518 op/s 2024-04-12T05:36:34.584 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:34 smithi067 ceph-mon[49010]: pgmap v248: 65 pgs: 65 active+clean; 132 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 4.6 MiB/s wr, 475 op/s 2024-04-12T05:36:34.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:34 smithi032 ceph-mon[39731]: pgmap v248: 65 pgs: 65 active+clean; 132 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 4.6 MiB/s wr, 475 op/s 2024-04-12T05:36:36.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:36 smithi032 ceph-mon[39731]: pgmap v249: 65 pgs: 65 active+clean; 133 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 404 op/s 2024-04-12T05:36:36.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.674+0000 7f5efaede700 1 -- 172.21.15.32:0/1266066919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ef40fbcd0 msgr2=0x7f5ef40fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:36.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.674+0000 7f5efaede700 1 --2- 172.21.15.32:0/1266066919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ef40fbcd0 0x7f5ef40fc0e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8004940 tx=0x7f5ee8009e30 comp rx=0 tx=0).stop 2024-04-12T05:36:36.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.675+0000 7f5efaede700 1 -- 172.21.15.32:0/1266066919 shutdown_connections 2024-04-12T05:36:36.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.675+0000 7f5efaede700 1 --2- 172.21.15.32:0/1266066919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ef40fcf80 0x7f5ef40fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:36.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.675+0000 7f5efaede700 1 --2- 172.21.15.32:0/1266066919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ef40fbcd0 0x7f5ef40fc0e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:36.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.675+0000 7f5efaede700 1 -- 172.21.15.32:0/1266066919 >> 172.21.15.32:0/1266066919 conn(0x7f5ef40f7430 msgr2=0x7f5ef40f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:36.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.675+0000 7f5efaede700 1 -- 172.21.15.32:0/1266066919 shutdown_connections 2024-04-12T05:36:36.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.675+0000 7f5efaede700 1 -- 172.21.15.32:0/1266066919 wait complete. 2024-04-12T05:36:36.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.676+0000 7f5efaede700 1 Processor -- start 2024-04-12T05:36:36.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.676+0000 7f5efaede700 1 -- start start 2024-04-12T05:36:36.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.676+0000 7f5efaede700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ef40fbcd0 0x7f5ef410d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:36:36.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.676+0000 7f5efaede700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ef40fcf80 0x7f5ef410de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:36:36.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.676+0000 7f5efaede700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef410e460 con 0x7f5ef40fcf80 2024-04-12T05:36:36.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.676+0000 7f5efaede700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef410ecb0 con 0x7f5ef40fbcd0 2024-04-12T05:36:36.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.677+0000 7f5ef3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ef40fcf80 0x7f5ef410de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:36.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.677+0000 7f5ef3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ef40fcf80 0x7f5ef410de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42058/0 (socket says 172.21.15.32:42058) 2024-04-12T05:36:36.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.677+0000 7f5ef3fff700 1 -- 172.21.15.32:0/4262007877 learned_addr learned my addr 172.21.15.32:0/4262007877 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:36:36.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.677+0000 7f5ef8c7a700 1 --2- 172.21.15.32:0/4262007877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ef40fbcd0 0x7f5ef410d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:36.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.677+0000 7f5ef3fff700 1 -- 172.21.15.32:0/4262007877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ef40fbcd0 msgr2=0x7f5ef410d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:36.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.677+0000 7f5ef3fff700 1 --2- 172.21.15.32:0/4262007877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ef40fbcd0 0x7f5ef410d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:36.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.677+0000 7f5ef3fff700 1 -- 172.21.15.32:0/4262007877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee8005040 con 0x7f5ef40fcf80 2024-04-12T05:36:36.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.677+0000 7f5ef8c7a700 1 --2- 172.21.15.32:0/4262007877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ef40fbcd0 0x7f5ef410d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:36:36.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.677+0000 7f5ef3fff700 1 --2- 172.21.15.32:0/4262007877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ef40fcf80 0x7f5ef410de60 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0002a50 tx=0x7f5ee00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:36.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.677+0000 7f5ef17fa700 1 -- 172.21.15.32:0/4262007877 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ee0007cc0 con 0x7f5ef40fcf80 2024-04-12T05:36:36.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.677+0000 7f5ef17fa700 1 -- 172.21.15.32:0/4262007877 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ee000f040 con 0x7f5ef40fcf80 2024-04-12T05:36:36.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.677+0000 7f5efaede700 1 -- 172.21.15.32:0/4262007877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef410ef90 con 0x7f5ef40fcf80 2024-04-12T05:36:36.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.678+0000 7f5ef17fa700 1 -- 172.21.15.32:0/4262007877 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ee0009540 con 0x7f5ef40fcf80 2024-04-12T05:36:36.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.678+0000 7f5efaede700 1 -- 172.21.15.32:0/4262007877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef410f4d0 con 0x7f5ef40fcf80 2024-04-12T05:36:36.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.679+0000 7f5efaede700 1 -- 172.21.15.32:0/4262007877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef4101b10 con 0x7f5ef40fcf80 2024-04-12T05:36:36.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.679+0000 7f5ef17fa700 1 -- 172.21.15.32:0/4262007877 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f5ee00096c0 con 0x7f5ef40fcf80 2024-04-12T05:36:36.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.680+0000 7f5ef17fa700 1 --2- 172.21.15.32:0/4262007877 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f5ee40692b0 0x7f5ee406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:36:36.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.680+0000 7f5ef17fa700 1 -- 172.21.15.32:0/4262007877 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f5ee0086f20 con 0x7f5ef40fcf80 2024-04-12T05:36:36.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.683+0000 7f5ef8c7a700 1 --2- 172.21.15.32:0/4262007877 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f5ee40692b0 0x7f5ee406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:36.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.683+0000 7f5ef8c7a700 1 --2- 172.21.15.32:0/4262007877 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f5ee40692b0 0x7f5ee406b760 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5ee800b3b0 tx=0x7f5ee8009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:36.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:36.685+0000 7f5ef17fa700 1 -- 172.21.15.32:0/4262007877 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5ee0056c90 con 0x7f5ef40fcf80 2024-04-12T05:36:36.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:36 smithi067 ceph-mon[49010]: pgmap v249: 65 pgs: 65 active+clean; 133 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 404 op/s 2024-04-12T05:36:37.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.004+0000 7f5efaede700 1 -- 172.21.15.32:0/4262007877 --> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ef4104830 con 0x7f5ee40692b0 2024-04-12T05:36:37.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.006+0000 7f5ef17fa700 1 -- 172.21.15.32:0/4262007877 <== mgr.14208 v2:172.21.15.32:6800/595990262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f5ee8008e30 con 0x7f5ee40692b0 2024-04-12T05:36:37.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.010+0000 7f5efaede700 1 -- 172.21.15.32:0/4262007877 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f5ee40692b0 msgr2=0x7f5ee406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:37.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.010+0000 7f5efaede700 1 --2- 172.21.15.32:0/4262007877 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f5ee40692b0 0x7f5ee406b760 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5ee800b3b0 tx=0x7f5ee8009e30 comp rx=0 tx=0).stop 2024-04-12T05:36:37.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.010+0000 7f5efaede700 1 -- 172.21.15.32:0/4262007877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ef40fcf80 msgr2=0x7f5ef410de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:37.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.010+0000 7f5efaede700 1 --2- 172.21.15.32:0/4262007877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ef40fcf80 0x7f5ef410de60 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0002a50 tx=0x7f5ee00046b0 comp rx=0 tx=0).stop 2024-04-12T05:36:37.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.010+0000 7f5efaede700 1 -- 172.21.15.32:0/4262007877 shutdown_connections 2024-04-12T05:36:37.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.010+0000 7f5efaede700 1 --2- 172.21.15.32:0/4262007877 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f5ee40692b0 0x7f5ee406b760 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:37.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.010+0000 7f5efaede700 1 --2- 172.21.15.32:0/4262007877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ef40fbcd0 0x7f5ef410d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:37.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.010+0000 7f5efaede700 1 --2- 172.21.15.32:0/4262007877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ef40fcf80 0x7f5ef410de60 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:37.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.010+0000 7f5efaede700 1 -- 172.21.15.32:0/4262007877 >> 172.21.15.32:0/4262007877 conn(0x7f5ef40f7430 msgr2=0x7f5ef4065c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:37.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.010+0000 7f5efaede700 1 -- 172.21.15.32:0/4262007877 shutdown_connections 2024-04-12T05:36:37.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.010+0000 7f5efaede700 1 -- 172.21.15.32:0/4262007877 wait complete. 2024-04-12T05:36:37.019 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:36:37.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.123+0000 7f0bcf86b700 1 -- 172.21.15.32:0/3106543589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc80fe440 msgr2=0x7f0bc80fe850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:37.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.123+0000 7f0bcf86b700 1 --2- 172.21.15.32:0/3106543589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc80fe440 0x7f0bc80fe850 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4004940 tx=0x7f0bc4009e30 comp rx=0 tx=0).stop 2024-04-12T05:36:37.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.123+0000 7f0bcf86b700 1 -- 172.21.15.32:0/3106543589 shutdown_connections 2024-04-12T05:36:37.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.123+0000 7f0bcf86b700 1 --2- 172.21.15.32:0/3106543589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc80fed90 0x7f0bc80faed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:37.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.123+0000 7f0bcf86b700 1 --2- 172.21.15.32:0/3106543589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc80fe440 0x7f0bc80fe850 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:37.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.123+0000 7f0bcf86b700 1 -- 172.21.15.32:0/3106543589 >> 172.21.15.32:0/3106543589 conn(0x7f0bc80f6be0 msgr2=0x7f0bc80f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:37.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.123+0000 7f0bcf86b700 1 -- 172.21.15.32:0/3106543589 shutdown_connections 2024-04-12T05:36:37.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.123+0000 7f0bcf86b700 1 -- 172.21.15.32:0/3106543589 wait complete. 2024-04-12T05:36:37.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.124+0000 7f0bcf86b700 1 Processor -- start 2024-04-12T05:36:37.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.124+0000 7f0bcf86b700 1 -- start start 2024-04-12T05:36:37.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.124+0000 7f0bcf86b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc80fe440 0x7f0bc810d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:36:37.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.124+0000 7f0bcf86b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc80fed90 0x7f0bc810dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:36:37.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.124+0000 7f0bcf86b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc810e360 con 0x7f0bc80fe440 2024-04-12T05:36:37.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.124+0000 7f0bcf86b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc810e4d0 con 0x7f0bc80fed90 2024-04-12T05:36:37.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.124+0000 7f0bcd607700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc80fe440 0x7f0bc810d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:37.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.124+0000 7f0bcd607700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc80fe440 0x7f0bc810d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42076/0 (socket says 172.21.15.32:42076) 2024-04-12T05:36:37.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.124+0000 7f0bcd607700 1 -- 172.21.15.32:0/1762024690 learned_addr learned my addr 172.21.15.32:0/1762024690 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:36:37.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.124+0000 7f0bcce06700 1 --2- 172.21.15.32:0/1762024690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc80fed90 0x7f0bc810dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:37.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.124+0000 7f0bcd607700 1 -- 172.21.15.32:0/1762024690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc80fed90 msgr2=0x7f0bc810dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:37.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.124+0000 7f0bcd607700 1 --2- 172.21.15.32:0/1762024690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc80fed90 0x7f0bc810dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:37.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.124+0000 7f0bcd607700 1 -- 172.21.15.32:0/1762024690 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc4005040 con 0x7f0bc80fe440 2024-04-12T05:36:37.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.125+0000 7f0bcd607700 1 --2- 172.21.15.32:0/1762024690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc80fe440 0x7f0bc810d820 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0bc40137b0 tx=0x7f0bc4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:37.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.125+0000 7f0bbdffb700 1 -- 172.21.15.32:0/1762024690 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bc400e070 con 0x7f0bc80fe440 2024-04-12T05:36:37.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.125+0000 7f0bcf86b700 1 -- 172.21.15.32:0/1762024690 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc8112ed0 con 0x7f0bc80fe440 2024-04-12T05:36:37.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.125+0000 7f0bbdffb700 1 -- 172.21.15.32:0/1762024690 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0bc401c400 con 0x7f0bc80fe440 2024-04-12T05:36:37.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.125+0000 7f0bbdffb700 1 -- 172.21.15.32:0/1762024690 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bc400b650 con 0x7f0bc80fe440 2024-04-12T05:36:37.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.125+0000 7f0bcf86b700 1 -- 172.21.15.32:0/1762024690 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc81133b0 con 0x7f0bc80fe440 2024-04-12T05:36:37.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.126+0000 7f0bcf86b700 1 -- 172.21.15.32:0/1762024690 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc804c710 con 0x7f0bc80fe440 2024-04-12T05:36:37.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.126+0000 7f0bbdffb700 1 -- 172.21.15.32:0/1762024690 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f0bc401c9e0 con 0x7f0bc80fe440 2024-04-12T05:36:37.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.127+0000 7f0bbdffb700 1 --2- 172.21.15.32:0/1762024690 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f0bb40692a0 0x7f0bb406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:36:37.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.127+0000 7f0bbdffb700 1 -- 172.21.15.32:0/1762024690 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0bc40887e0 con 0x7f0bc80fe440 2024-04-12T05:36:37.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.127+0000 7f0bcce06700 1 --2- 172.21.15.32:0/1762024690 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f0bb40692a0 0x7f0bb406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:37.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.128+0000 7f0bcce06700 1 --2- 172.21.15.32:0/1762024690 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f0bb40692a0 0x7f0bb406b750 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8009b80 tx=0x7f0bb800b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:37.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.131+0000 7f0bbdffb700 1 -- 172.21.15.32:0/1762024690 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f0bc4058550 con 0x7f0bc80fe440 2024-04-12T05:36:37.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.458+0000 7f0bcf86b700 1 -- 172.21.15.32:0/1762024690 --> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0bc81025b0 con 0x7f0bb40692a0 2024-04-12T05:36:37.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.459+0000 7f0bbdffb700 1 -- 172.21.15.32:0/1762024690 <== mgr.14208 v2:172.21.15.32:6800/595990262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f0bc81025b0 con 0x7f0bb40692a0 2024-04-12T05:36:37.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.462+0000 7f0bcf86b700 1 -- 172.21.15.32:0/1762024690 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f0bb40692a0 msgr2=0x7f0bb406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:37.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.462+0000 7f0bcf86b700 1 --2- 172.21.15.32:0/1762024690 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f0bb40692a0 0x7f0bb406b750 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8009b80 tx=0x7f0bb800b040 comp rx=0 tx=0).stop 2024-04-12T05:36:37.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.463+0000 7f0bcf86b700 1 -- 172.21.15.32:0/1762024690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc80fe440 msgr2=0x7f0bc810d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:37.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.463+0000 7f0bcf86b700 1 --2- 172.21.15.32:0/1762024690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc80fe440 0x7f0bc810d820 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0bc40137b0 tx=0x7f0bc4009e30 comp rx=0 tx=0).stop 2024-04-12T05:36:37.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.463+0000 7f0bcf86b700 1 -- 172.21.15.32:0/1762024690 shutdown_connections 2024-04-12T05:36:37.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.463+0000 7f0bcf86b700 1 --2- 172.21.15.32:0/1762024690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc80fe440 0x7f0bc810d820 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:37.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.463+0000 7f0bcf86b700 1 --2- 172.21.15.32:0/1762024690 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f0bb40692a0 0x7f0bb406b750 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:37.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.463+0000 7f0bcf86b700 1 --2- 172.21.15.32:0/1762024690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc80fed90 0x7f0bc810dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:37.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.463+0000 7f0bcf86b700 1 -- 172.21.15.32:0/1762024690 >> 172.21.15.32:0/1762024690 conn(0x7f0bc80f6be0 msgr2=0x7f0bc80f8340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:37.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.463+0000 7f0bcf86b700 1 -- 172.21.15.32:0/1762024690 shutdown_connections 2024-04-12T05:36:37.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.463+0000 7f0bcf86b700 1 -- 172.21.15.32:0/1762024690 wait complete. 2024-04-12T05:36:37.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.619+0000 7f59f2fa0700 1 -- 172.21.15.32:0/2155353311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59ec0fbcd0 msgr2=0x7f59ec0fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:37.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.619+0000 7f59f2fa0700 1 --2- 172.21.15.32:0/2155353311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59ec0fbcd0 0x7f59ec0fc0e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f59e0004940 tx=0x7f59e0009e30 comp rx=0 tx=0).stop 2024-04-12T05:36:37.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.619+0000 7f59f2fa0700 1 -- 172.21.15.32:0/2155353311 shutdown_connections 2024-04-12T05:36:37.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.619+0000 7f59f2fa0700 1 --2- 172.21.15.32:0/2155353311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59ec0fcf80 0x7f59ec0fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:37.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.619+0000 7f59f2fa0700 1 --2- 172.21.15.32:0/2155353311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59ec0fbcd0 0x7f59ec0fc0e0 secure :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f59e0004940 tx=0x7f59e0009e30 comp rx=0 tx=0).stop 2024-04-12T05:36:37.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.620+0000 7f59f2fa0700 1 -- 172.21.15.32:0/2155353311 >> 172.21.15.32:0/2155353311 conn(0x7f59ec0f7430 msgr2=0x7f59ec0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:37.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.620+0000 7f59f2fa0700 1 -- 172.21.15.32:0/2155353311 shutdown_connections 2024-04-12T05:36:37.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.621+0000 7f59f2fa0700 1 -- 172.21.15.32:0/2155353311 wait complete. 2024-04-12T05:36:37.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.621+0000 7f59f2fa0700 1 Processor -- start 2024-04-12T05:36:37.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.621+0000 7f59f2fa0700 1 -- start start 2024-04-12T05:36:37.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.622+0000 7f59f2fa0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59ec0fcf80 0x7f59ec1140a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:36:37.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.622+0000 7f59f2fa0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59ec1145e0 0x7f59ec115430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:36:37.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.622+0000 7f59f2fa0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59ec114aa0 con 0x7f59ec1145e0 2024-04-12T05:36:37.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.622+0000 7f59f2fa0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59ec114be0 con 0x7f59ec0fcf80 2024-04-12T05:36:37.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.622+0000 7f59f0d3c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59ec0fcf80 0x7f59ec1140a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:37.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.622+0000 7f59f0d3c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59ec0fcf80 0x7f59ec1140a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.32:44166/0 (socket says 172.21.15.32:44166) 2024-04-12T05:36:37.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.622+0000 7f59f0d3c700 1 -- 172.21.15.32:0/1434963235 learned_addr learned my addr 172.21.15.32:0/1434963235 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:36:37.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.622+0000 7f59ebfff700 1 --2- 172.21.15.32:0/1434963235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59ec1145e0 0x7f59ec115430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:37.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.623+0000 7f59f0d3c700 1 -- 172.21.15.32:0/1434963235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59ec1145e0 msgr2=0x7f59ec115430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:37.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.623+0000 7f59f0d3c700 1 --2- 172.21.15.32:0/1434963235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59ec1145e0 0x7f59ec115430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:37.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.623+0000 7f59f0d3c700 1 -- 172.21.15.32:0/1434963235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e0005040 con 0x7f59ec0fcf80 2024-04-12T05:36:37.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.623+0000 7f59f0d3c700 1 --2- 172.21.15.32:0/1434963235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59ec0fcf80 0x7f59ec1140a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f59e000b0d0 tx=0x7f59e0009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:37.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.623+0000 7f59e97fa700 1 -- 172.21.15.32:0/1434963235 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59e000e070 con 0x7f59ec0fcf80 2024-04-12T05:36:37.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.623+0000 7f59e97fa700 1 -- 172.21.15.32:0/1434963235 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f59e000b410 con 0x7f59ec0fcf80 2024-04-12T05:36:37.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.623+0000 7f59f2fa0700 1 -- 172.21.15.32:0/1434963235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59ec115a30 con 0x7f59ec0fcf80 2024-04-12T05:36:37.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.623+0000 7f59e97fa700 1 -- 172.21.15.32:0/1434963235 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59e0012620 con 0x7f59ec0fcf80 2024-04-12T05:36:37.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.624+0000 7f59f2fa0700 1 -- 172.21.15.32:0/1434963235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59ec115fc0 con 0x7f59ec0fcf80 2024-04-12T05:36:37.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.625+0000 7f59f2fa0700 1 -- 172.21.15.32:0/1434963235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59ec05f5f0 con 0x7f59ec0fcf80 2024-04-12T05:36:37.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.626+0000 7f59e97fa700 1 -- 172.21.15.32:0/1434963235 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f59e0012780 con 0x7f59ec0fcf80 2024-04-12T05:36:37.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.626+0000 7f59e97fa700 1 --2- 172.21.15.32:0/1434963235 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f59dc0692f0 0x7f59dc06b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:36:37.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.626+0000 7f59e97fa700 1 -- 172.21.15.32:0/1434963235 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f59e0087ba0 con 0x7f59ec0fcf80 2024-04-12T05:36:37.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.627+0000 7f59ebfff700 1 --2- 172.21.15.32:0/1434963235 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f59dc0692f0 0x7f59dc06b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:37.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.627+0000 7f59ebfff700 1 --2- 172.21.15.32:0/1434963235 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f59dc0692f0 0x7f59dc06b7a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f59d8009b80 tx=0x7f59d800b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:37.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.632+0000 7f59e97fa700 1 -- 172.21.15.32:0/1434963235 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f59e0057960 con 0x7f59ec0fcf80 2024-04-12T05:36:37.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.923+0000 7f59f2fa0700 1 -- 172.21.15.32:0/1434963235 --> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f59ec066240 con 0x7f59dc0692f0 2024-04-12T05:36:37.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.937+0000 7f59e97fa700 1 -- 172.21.15.32:0/1434963235 <== mgr.14208 v2:172.21.15.32:6800/595990262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3344 (secure 0 0 0) 0x7f59ec066240 con 0x7f59dc0692f0 2024-04-12T05:36:37.935 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:36:37.936 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (6m) 105s ago 7m 27.0M - 0.25.0 c8568f914cd2 a5c616791a7e 2024-04-12T05:36:37.936 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (8m) 105s ago 8m 6656k - 17.2.7-827-gd3b5060a 21de9b7369ad a9cfc00c311b 2024-04-12T05:36:37.936 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (7m) 110s ago 7m 6651k - 17.2.7-827-gd3b5060a 21de9b7369ad 1f2eea4542d9 2024-04-12T05:36:37.936 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (6m) 105s ago 7m 87.0M - 9.4.7 954c08fa6188 8e5a7ebd77b5 2024-04-12T05:36:37.936 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (2m) 105s ago 2m 11.4M - 17.2.7-827-gd3b5060a 21de9b7369ad e7ada8e5259c 2024-04-12T05:36:37.936 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (2m) 105s ago 2m 14.0M - 17.2.7-827-gd3b5060a 21de9b7369ad 55124da9afdb 2024-04-12T05:36:37.936 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (2m) 110s ago 2m 13.3M - 17.2.7-827-gd3b5060a 21de9b7369ad 0c295d31a4a6 2024-04-12T05:36:37.936 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (2m) 110s ago 2m 16.8M - 17.2.7-827-gd3b5060a 21de9b7369ad 6eecd7deaac2 2024-04-12T05:36:37.936 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:9283 running (9m) 105s ago 9m 483M - 17.2.7-827-gd3b5060a 21de9b7369ad fdf859c0313c 2024-04-12T05:36:37.936 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283 running (6m) 110s ago 6m 435M - 17.2.7-827-gd3b5060a 21de9b7369ad 117ab3f1dbe0 2024-04-12T05:36:37.936 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (9m) 105s ago 9m 42.4M 2048M 17.2.7-827-gd3b5060a 21de9b7369ad 75aad95b984d 2024-04-12T05:36:37.936 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (6m) 110s ago 6m 31.6M 2048M 17.2.7-827-gd3b5060a 21de9b7369ad 157be8a8c6d8 2024-04-12T05:36:37.936 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (8m) 105s ago 8m 15.8M - 1.5.0 0da6a335fe13 ca3585dd7b65 2024-04-12T05:36:37.936 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (6m) 110s ago 6m 15.6M - 1.5.0 0da6a335fe13 c324d9436cca 2024-04-12T05:36:37.937 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (6m) 105s ago 6m 46.9M 4300M 17.2.7-827-gd3b5060a 21de9b7369ad 97446e42ceba 2024-04-12T05:36:37.937 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (5m) 105s ago 5m 47.6M 4300M 17.2.7-827-gd3b5060a 21de9b7369ad bf9f9a2db2d5 2024-04-12T05:36:37.937 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (5m) 105s ago 5m 44.1M 4300M 17.2.7-827-gd3b5060a 21de9b7369ad f055cf1ca665 2024-04-12T05:36:37.937 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (4m) 110s ago 4m 44.8M 5323M 17.2.7-827-gd3b5060a 21de9b7369ad 61b851235351 2024-04-12T05:36:37.937 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (4m) 110s ago 4m 42.6M 5323M 17.2.7-827-gd3b5060a 21de9b7369ad 2c84d9047e9c 2024-04-12T05:36:37.937 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (3m) 110s ago 3m 44.4M 5323M 17.2.7-827-gd3b5060a 21de9b7369ad e54839bb6055 2024-04-12T05:36:37.937 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (6m) 105s ago 7m 56.5M - 2.43.0 a07b618ecd1d 996307aec3d2 2024-04-12T05:36:37.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.939+0000 7f59f2fa0700 1 -- 172.21.15.32:0/1434963235 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f59dc0692f0 msgr2=0x7f59dc06b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:37.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.939+0000 7f59f2fa0700 1 --2- 172.21.15.32:0/1434963235 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f59dc0692f0 0x7f59dc06b7a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f59d8009b80 tx=0x7f59d800b040 comp rx=0 tx=0).stop 2024-04-12T05:36:37.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.939+0000 7f59f2fa0700 1 -- 172.21.15.32:0/1434963235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59ec0fcf80 msgr2=0x7f59ec1140a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:37.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.939+0000 7f59f2fa0700 1 --2- 172.21.15.32:0/1434963235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59ec0fcf80 0x7f59ec1140a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f59e000b0d0 tx=0x7f59e0009e30 comp rx=0 tx=0).stop 2024-04-12T05:36:37.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.939+0000 7f59f2fa0700 1 -- 172.21.15.32:0/1434963235 shutdown_connections 2024-04-12T05:36:37.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.939+0000 7f59f2fa0700 1 --2- 172.21.15.32:0/1434963235 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f59dc0692f0 0x7f59dc06b7a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:37.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.939+0000 7f59f2fa0700 1 --2- 172.21.15.32:0/1434963235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59ec0fcf80 0x7f59ec1140a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:37.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.939+0000 7f59f2fa0700 1 --2- 172.21.15.32:0/1434963235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59ec1145e0 0x7f59ec115430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:37.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.939+0000 7f59f2fa0700 1 -- 172.21.15.32:0/1434963235 >> 172.21.15.32:0/1434963235 conn(0x7f59ec0f7430 msgr2=0x7f59ec0f8ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:37.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.940+0000 7f59f2fa0700 1 -- 172.21.15.32:0/1434963235 shutdown_connections 2024-04-12T05:36:37.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:37.940+0000 7f59f2fa0700 1 -- 172.21.15.32:0/1434963235 wait complete. 2024-04-12T05:36:38.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.083+0000 7f40e83b0700 1 -- 172.21.15.32:0/3904204684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40e0066280 msgr2=0x7f40e00ffa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:38.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.083+0000 7f40e83b0700 1 --2- 172.21.15.32:0/3904204684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40e0066280 0x7f40e00ffa20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f40d8009ce0 tx=0x7f40d8004d10 comp rx=0 tx=0).stop 2024-04-12T05:36:38.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.083+0000 7f40e83b0700 1 -- 172.21.15.32:0/3904204684 shutdown_connections 2024-04-12T05:36:38.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.083+0000 7f40e83b0700 1 --2- 172.21.15.32:0/3904204684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40e00fff60 0x7f40e0102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:38.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.083+0000 7f40e83b0700 1 --2- 172.21.15.32:0/3904204684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40e0066280 0x7f40e00ffa20 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:38.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.083+0000 7f40e83b0700 1 -- 172.21.15.32:0/3904204684 >> 172.21.15.32:0/3904204684 conn(0x7f40e00f7430 msgr2=0x7f40e00f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:38.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.083+0000 7f40e83b0700 1 -- 172.21.15.32:0/3904204684 shutdown_connections 2024-04-12T05:36:38.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.083+0000 7f40e83b0700 1 -- 172.21.15.32:0/3904204684 wait complete. 2024-04-12T05:36:38.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.084+0000 7f40e83b0700 1 Processor -- start 2024-04-12T05:36:38.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.084+0000 7f40e83b0700 1 -- start start 2024-04-12T05:36:38.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.084+0000 7f40e83b0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40e0066280 0x7f40e010d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:36:38.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.084+0000 7f40e83b0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40e00fff60 0x7f40e010df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:36:38.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.084+0000 7f40e83b0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e010e520 con 0x7f40e0066280 2024-04-12T05:36:38.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.084+0000 7f40e83b0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e010ed20 con 0x7f40e00fff60 2024-04-12T05:36:38.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.084+0000 7f40e594b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40e00fff60 0x7f40e010df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:38.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.084+0000 7f40e594b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40e00fff60 0x7f40e010df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.32:44186/0 (socket says 172.21.15.32:44186) 2024-04-12T05:36:38.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.084+0000 7f40e594b700 1 -- 172.21.15.32:0/2366342347 learned_addr learned my addr 172.21.15.32:0/2366342347 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:36:38.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.084+0000 7f40e614c700 1 --2- 172.21.15.32:0/2366342347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40e0066280 0x7f40e010d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:38.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.085+0000 7f40e594b700 1 -- 172.21.15.32:0/2366342347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40e0066280 msgr2=0x7f40e010d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:38.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.085+0000 7f40e594b700 1 --2- 172.21.15.32:0/2366342347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40e0066280 0x7f40e010d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:38.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.085+0000 7f40e594b700 1 -- 172.21.15.32:0/2366342347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d8009940 con 0x7f40e00fff60 2024-04-12T05:36:38.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.085+0000 7f40e594b700 1 --2- 172.21.15.32:0/2366342347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40e00fff60 0x7f40e010df20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f40dc00eb40 tx=0x7f40dc00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:38.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.085+0000 7f40d2ffd700 1 -- 172.21.15.32:0/2366342347 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40dc007cc0 con 0x7f40e00fff60 2024-04-12T05:36:38.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.085+0000 7f40d2ffd700 1 -- 172.21.15.32:0/2366342347 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f40dc00f040 con 0x7f40e00fff60 2024-04-12T05:36:38.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.085+0000 7f40d2ffd700 1 -- 172.21.15.32:0/2366342347 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40dc00c990 con 0x7f40e00fff60 2024-04-12T05:36:38.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.085+0000 7f40e83b0700 1 -- 172.21.15.32:0/2366342347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40e010efa0 con 0x7f40e00fff60 2024-04-12T05:36:38.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.085+0000 7f40e83b0700 1 -- 172.21.15.32:0/2366342347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40e010f490 con 0x7f40e00fff60 2024-04-12T05:36:38.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.086+0000 7f40e83b0700 1 -- 172.21.15.32:0/2366342347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40e005f5f0 con 0x7f40e00fff60 2024-04-12T05:36:38.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.087+0000 7f40d2ffd700 1 -- 172.21.15.32:0/2366342347 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f40dc00cc00 con 0x7f40e00fff60 2024-04-12T05:36:38.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.087+0000 7f40d2ffd700 1 --2- 172.21.15.32:0/2366342347 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f40cc0692a0 0x7f40cc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:36:38.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.087+0000 7f40d2ffd700 1 -- 172.21.15.32:0/2366342347 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f40dc086e10 con 0x7f40e00fff60 2024-04-12T05:36:38.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.087+0000 7f40e614c700 1 --2- 172.21.15.32:0/2366342347 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f40cc0692a0 0x7f40cc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:38.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.088+0000 7f40e614c700 1 --2- 172.21.15.32:0/2366342347 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f40cc0692a0 0x7f40cc06b750 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f40d80093c0 tx=0x7f40d8004d10 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:38.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.092+0000 7f40d2ffd700 1 -- 172.21.15.32:0/2366342347 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f40dc04e190 con 0x7f40e00fff60 2024-04-12T05:36:38.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.451+0000 7f40e83b0700 1 -- 172.21.15.32:0/2366342347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f40e010f7a0 con 0x7f40e00fff60 2024-04-12T05:36:38.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.452+0000 7f40d2ffd700 1 -- 172.21.15.32:0/2366342347 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f40dc056b50 con 0x7f40e00fff60 2024-04-12T05:36:38.450 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:36:38.450 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:36:38.450 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 2 2024-04-12T05:36:38.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:36:38.451 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:36:38.451 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 2 2024-04-12T05:36:38.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:36:38.451 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:36:38.451 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 6 2024-04-12T05:36:38.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:36:38.451 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:36:38.451 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 4 2024-04-12T05:36:38.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:36:38.451 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:36:38.451 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 14 2024-04-12T05:36:38.451 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:36:38.451 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:36:38.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.454+0000 7f40e83b0700 1 -- 172.21.15.32:0/2366342347 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f40cc0692a0 msgr2=0x7f40cc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:38.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.454+0000 7f40e83b0700 1 --2- 172.21.15.32:0/2366342347 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f40cc0692a0 0x7f40cc06b750 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f40d80093c0 tx=0x7f40d8004d10 comp rx=0 tx=0).stop 2024-04-12T05:36:38.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.454+0000 7f40e83b0700 1 -- 172.21.15.32:0/2366342347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40e00fff60 msgr2=0x7f40e010df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:38.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.454+0000 7f40e83b0700 1 --2- 172.21.15.32:0/2366342347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40e00fff60 0x7f40e010df20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f40dc00eb40 tx=0x7f40dc00b040 comp rx=0 tx=0).stop 2024-04-12T05:36:38.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.454+0000 7f40e83b0700 1 -- 172.21.15.32:0/2366342347 shutdown_connections 2024-04-12T05:36:38.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.454+0000 7f40e83b0700 1 --2- 172.21.15.32:0/2366342347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40e0066280 0x7f40e010d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:38.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.454+0000 7f40e83b0700 1 --2- 172.21.15.32:0/2366342347 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f40cc0692a0 0x7f40cc06b750 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:38.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.454+0000 7f40e83b0700 1 --2- 172.21.15.32:0/2366342347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40e00fff60 0x7f40e010df20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:38.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.454+0000 7f40e83b0700 1 -- 172.21.15.32:0/2366342347 >> 172.21.15.32:0/2366342347 conn(0x7f40e00f7430 msgr2=0x7f40e00f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:38.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.454+0000 7f40e83b0700 1 -- 172.21.15.32:0/2366342347 shutdown_connections 2024-04-12T05:36:38.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.454+0000 7f40e83b0700 1 -- 172.21.15.32:0/2366342347 wait complete. 2024-04-12T05:36:38.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:38 smithi032 ceph-mon[39731]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:38.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:38 smithi032 ceph-mon[39731]: pgmap v250: 65 pgs: 65 active+clean; 149 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.3 MiB/s rd, 5.2 MiB/s wr, 549 op/s 2024-04-12T05:36:38.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.601+0000 7fd2437b5700 1 -- 172.21.15.32:0/47494959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd23c064d70 msgr2=0x7fd23c0fb3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:38.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.601+0000 7fd2437b5700 1 --2- 172.21.15.32:0/47494959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd23c064d70 0x7fd23c0fb3b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd230004940 tx=0x7fd230009e30 comp rx=0 tx=0).stop 2024-04-12T05:36:38.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.602+0000 7fd2437b5700 1 -- 172.21.15.32:0/47494959 shutdown_connections 2024-04-12T05:36:38.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.602+0000 7fd2437b5700 1 --2- 172.21.15.32:0/47494959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd23c065210 0x7fd23c0fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:38.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.602+0000 7fd2437b5700 1 --2- 172.21.15.32:0/47494959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd23c064d70 0x7fd23c0fb3b0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:38.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.602+0000 7fd2437b5700 1 -- 172.21.15.32:0/47494959 >> 172.21.15.32:0/47494959 conn(0x7fd23c0f6ba0 msgr2=0x7fd23c0f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:38.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.602+0000 7fd2437b5700 1 -- 172.21.15.32:0/47494959 shutdown_connections 2024-04-12T05:36:38.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.602+0000 7fd2437b5700 1 -- 172.21.15.32:0/47494959 wait complete. 2024-04-12T05:36:38.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.602+0000 7fd2437b5700 1 Processor -- start 2024-04-12T05:36:38.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.602+0000 7fd2437b5700 1 -- start start 2024-04-12T05:36:38.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.603+0000 7fd2437b5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd23c064d70 0x7fd23c10b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:36:38.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.603+0000 7fd2437b5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd23c065210 0x7fd23c10bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:36:38.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.603+0000 7fd2437b5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23c10c200 con 0x7fd23c064d70 2024-04-12T05:36:38.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.603+0000 7fd2437b5700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23c10ca50 con 0x7fd23c065210 2024-04-12T05:36:38.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.603+0000 7fd241551700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd23c064d70 0x7fd23c10b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:38.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.603+0000 7fd241551700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd23c064d70 0x7fd23c10b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42124/0 (socket says 172.21.15.32:42124) 2024-04-12T05:36:38.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.603+0000 7fd241551700 1 -- 172.21.15.32:0/2714855198 learned_addr learned my addr 172.21.15.32:0/2714855198 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:36:38.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.603+0000 7fd241551700 1 -- 172.21.15.32:0/2714855198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd23c065210 msgr2=0x7fd23c10bc00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T05:36:38.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.603+0000 7fd241551700 1 --2- 172.21.15.32:0/2714855198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd23c065210 0x7fd23c10bc00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:38.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.603+0000 7fd241551700 1 -- 172.21.15.32:0/2714855198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd230005040 con 0x7fd23c064d70 2024-04-12T05:36:38.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.604+0000 7fd241551700 1 --2- 172.21.15.32:0/2714855198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd23c064d70 0x7fd23c10b6c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd2300043d0 tx=0x7fd230009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:38.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.604+0000 7fd22dffb700 1 -- 172.21.15.32:0/2714855198 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd23000e070 con 0x7fd23c064d70 2024-04-12T05:36:38.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.604+0000 7fd22dffb700 1 -- 172.21.15.32:0/2714855198 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd23001c400 con 0x7fd23c064d70 2024-04-12T05:36:38.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.604+0000 7fd22dffb700 1 -- 172.21.15.32:0/2714855198 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd23000b650 con 0x7fd23c064d70 2024-04-12T05:36:38.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.605+0000 7fd2437b5700 1 -- 172.21.15.32:0/2714855198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd23c10ccd0 con 0x7fd23c064d70 2024-04-12T05:36:38.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.605+0000 7fd2437b5700 1 -- 172.21.15.32:0/2714855198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd23c10d180 con 0x7fd23c064d70 2024-04-12T05:36:38.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.606+0000 7fd2437b5700 1 -- 172.21.15.32:0/2714855198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd23c1051f0 con 0x7fd23c064d70 2024-04-12T05:36:38.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.608+0000 7fd22dffb700 1 -- 172.21.15.32:0/2714855198 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fd23001c9c0 con 0x7fd23c064d70 2024-04-12T05:36:38.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.609+0000 7fd22dffb700 1 --2- 172.21.15.32:0/2714855198 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fd2280693b0 0x7fd22806b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:36:38.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.609+0000 7fd22dffb700 1 -- 172.21.15.32:0/2714855198 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fd230087c90 con 0x7fd23c064d70 2024-04-12T05:36:38.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.610+0000 7fd240d50700 1 --2- 172.21.15.32:0/2714855198 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fd2280693b0 0x7fd22806b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:38.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.610+0000 7fd240d50700 1 --2- 172.21.15.32:0/2714855198 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fd2280693b0 0x7fd22806b860 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd238017750 tx=0x7fd2380048d0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:38.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.615+0000 7fd22dffb700 1 -- 172.21.15.32:0/2714855198 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fd230057a20 con 0x7fd23c064d70 2024-04-12T05:36:38.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:38 smithi067 ceph-mon[49010]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:38.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:38 smithi067 ceph-mon[49010]: pgmap v250: 65 pgs: 65 active+clean; 149 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.3 MiB/s rd, 5.2 MiB/s wr, 549 op/s 2024-04-12T05:36:38.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.919+0000 7fd2437b5700 1 -- 172.21.15.32:0/2714855198 --> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd23c10d4b0 con 0x7fd2280693b0 2024-04-12T05:36:38.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.920+0000 7fd22dffb700 1 -- 172.21.15.32:0/2714855198 <== mgr.14208 v2:172.21.15.32:6800/595990262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7fd23c10d4b0 con 0x7fd2280693b0 2024-04-12T05:36:38.919 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:36:38.919 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:36:38.919 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:36:38.919 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-12T05:36:38.919 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [], 2024-04-12T05:36:38.919 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "0/2 daemons upgraded", 2024-04-12T05:36:38.919 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 image on host smithi067", 2024-04-12T05:36:38.920 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:36:38.920 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:36:38.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.923+0000 7fd2437b5700 1 -- 172.21.15.32:0/2714855198 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fd2280693b0 msgr2=0x7fd22806b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:38.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.923+0000 7fd2437b5700 1 --2- 172.21.15.32:0/2714855198 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fd2280693b0 0x7fd22806b860 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd238017750 tx=0x7fd2380048d0 comp rx=0 tx=0).stop 2024-04-12T05:36:38.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.923+0000 7fd2437b5700 1 -- 172.21.15.32:0/2714855198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd23c064d70 msgr2=0x7fd23c10b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:38.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.923+0000 7fd2437b5700 1 --2- 172.21.15.32:0/2714855198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd23c064d70 0x7fd23c10b6c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd2300043d0 tx=0x7fd230009e30 comp rx=0 tx=0).stop 2024-04-12T05:36:38.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.923+0000 7fd2437b5700 1 -- 172.21.15.32:0/2714855198 shutdown_connections 2024-04-12T05:36:38.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.923+0000 7fd2437b5700 1 --2- 172.21.15.32:0/2714855198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd23c064d70 0x7fd23c10b6c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:38.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.923+0000 7fd2437b5700 1 --2- 172.21.15.32:0/2714855198 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fd2280693b0 0x7fd22806b860 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:38.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.923+0000 7fd2437b5700 1 --2- 172.21.15.32:0/2714855198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd23c065210 0x7fd23c10bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:36:38.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.923+0000 7fd2437b5700 1 -- 172.21.15.32:0/2714855198 >> 172.21.15.32:0/2714855198 conn(0x7fd23c0f6ba0 msgr2=0x7fd23c0f8320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:38.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.923+0000 7fd2437b5700 1 -- 172.21.15.32:0/2714855198 shutdown_connections 2024-04-12T05:36:38.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:36:38.923+0000 7fd2437b5700 1 -- 172.21.15.32:0/2714855198 wait complete. 2024-04-12T05:36:39.577 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:39 smithi067 ceph-mon[49010]: from='client.14584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:39.577 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:39 smithi067 ceph-mon[49010]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:39.577 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:39 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/2366342347' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:36:39.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:39 smithi032 ceph-mon[39731]: from='client.14584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:39.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:39 smithi032 ceph-mon[39731]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:39.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:39 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/2366342347' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:36:40.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:40 smithi032 ceph-mon[39731]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:40.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:40 smithi032 ceph-mon[39731]: pgmap v251: 65 pgs: 65 active+clean; 149 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 380 op/s 2024-04-12T05:36:40.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:40 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:36:40.740 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:40 smithi067 ceph-mon[49010]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:40.740 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:40 smithi067 ceph-mon[49010]: pgmap v251: 65 pgs: 65 active+clean; 149 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 380 op/s 2024-04-12T05:36:40.740 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:40 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:36:42.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:41 smithi067 ceph-mon[49010]: Upgrade: Updating mgr.smithi067.dwichg 2024-04-12T05:36:42.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:41 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:36:42.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:41 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi067.dwichg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:36:42.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:41 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:36:42.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:41 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:36:42.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:41 smithi067 ceph-mon[49010]: Deploying daemon mgr.smithi067.dwichg on smithi067 2024-04-12T05:36:42.020 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:41 smithi067 ceph-mon[49010]: pgmap v252: 65 pgs: 65 active+clean; 157 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 4.1 MiB/s wr, 486 op/s 2024-04-12T05:36:42.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:41 smithi032 ceph-mon[39731]: Upgrade: Updating mgr.smithi067.dwichg 2024-04-12T05:36:42.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:41 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:36:42.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:41 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi067.dwichg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:36:42.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:41 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:36:42.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:41 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:36:42.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:41 smithi032 ceph-mon[39731]: Deploying daemon mgr.smithi067.dwichg on smithi067 2024-04-12T05:36:42.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:41 smithi032 ceph-mon[39731]: pgmap v252: 65 pgs: 65 active+clean; 157 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 4.1 MiB/s wr, 486 op/s 2024-04-12T05:36:43.394 INFO:tasks.workunit.client.1.smithi067.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-04-12T05:36:43.401 INFO:tasks.workunit.client.1.smithi067.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-12T05:36:43.401 INFO:tasks.workunit.client.1.smithi067.stderr:+ make 2024-04-12T05:36:43.433 INFO:tasks.workunit.client.1.smithi067.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-04-12T05:36:44.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:44 smithi067 ceph-mon[49010]: pgmap v253: 65 pgs: 65 active+clean; 172 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 457 op/s 2024-04-12T05:36:44.307 INFO:tasks.workunit.client.1.smithi067.stderr:++ readlink -f fsstress 2024-04-12T05:36:44.310 INFO:tasks.workunit.client.1.smithi067.stderr:+ BIN=/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-04-12T05:36:44.310 INFO:tasks.workunit.client.1.smithi067.stderr:+ popd 2024-04-12T05:36:44.312 INFO:tasks.workunit.client.1.smithi067.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-12T05:36:44.312 INFO:tasks.workunit.client.1.smithi067.stderr:+ popd 2024-04-12T05:36:44.313 INFO:tasks.workunit.client.1.smithi067.stdout:~/cephtest/mnt.1/client.1/tmp 2024-04-12T05:36:44.313 INFO:tasks.workunit.client.1.smithi067.stderr:++ mktemp -d -p . 2024-04-12T05:36:44.317 INFO:tasks.workunit.client.1.smithi067.stderr:+ T=./tmp.JyyLSyE8rf 2024-04-12T05:36:44.317 INFO:tasks.workunit.client.1.smithi067.stderr:+ /home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.JyyLSyE8rf -l 1 -n 1000 -p 10 -v 2024-04-12T05:36:44.322 INFO:tasks.workunit.client.1.smithi067.stdout:seed = 1713123392 2024-04-12T05:36:44.326 INFO:tasks.workunit.client.1.smithi067.stdout:0/0: read - no filename 2024-04-12T05:36:44.328 INFO:tasks.workunit.client.1.smithi067.stdout:1/0: write - no filename 2024-04-12T05:36:44.328 INFO:tasks.workunit.client.1.smithi067.stdout:1/1: dread - no filename 2024-04-12T05:36:44.328 INFO:tasks.workunit.client.1.smithi067.stdout:1/2: fdatasync - no filename 2024-04-12T05:36:44.328 INFO:tasks.workunit.client.1.smithi067.stdout:1/3: link - no file 2024-04-12T05:36:44.328 INFO:tasks.workunit.client.1.smithi067.stdout:1/4: rename - no filename 2024-04-12T05:36:44.328 INFO:tasks.workunit.client.1.smithi067.stdout:1/5: write - no filename 2024-04-12T05:36:44.328 INFO:tasks.workunit.client.1.smithi067.stdout:1/6: write - no filename 2024-04-12T05:36:44.328 INFO:tasks.workunit.client.1.smithi067.stdout:1/7: unlink - no file 2024-04-12T05:36:44.328 INFO:tasks.workunit.client.1.smithi067.stdout:1/8: fdatasync - no filename 2024-04-12T05:36:44.329 INFO:tasks.workunit.client.1.smithi067.stdout:0/1: creat f0 x:0 0 0 2024-04-12T05:36:44.330 INFO:tasks.workunit.client.1.smithi067.stdout:0/2: write f0 [826072,17482] 0 2024-04-12T05:36:44.330 INFO:tasks.workunit.client.1.smithi067.stdout:2/0: write - no filename 2024-04-12T05:36:44.330 INFO:tasks.workunit.client.1.smithi067.stdout:2/1: dread - no filename 2024-04-12T05:36:44.330 INFO:tasks.workunit.client.1.smithi067.stdout:2/2: write - no filename 2024-04-12T05:36:44.333 INFO:tasks.workunit.client.1.smithi067.stdout:1/9: creat f0 x:0 0 0 2024-04-12T05:36:44.334 INFO:tasks.workunit.client.1.smithi067.stdout:0/3: rename f0 to f1 0 2024-04-12T05:36:44.334 INFO:tasks.workunit.client.1.smithi067.stdout:1/10: fdatasync f0 0 2024-04-12T05:36:44.335 INFO:tasks.workunit.client.1.smithi067.stdout:4/0: dwrite - no filename 2024-04-12T05:36:44.335 INFO:tasks.workunit.client.1.smithi067.stdout:4/1: rmdir - no directory 2024-04-12T05:36:44.336 INFO:tasks.workunit.client.1.smithi067.stdout:2/3: mknod c0 0 2024-04-12T05:36:44.336 INFO:tasks.workunit.client.1.smithi067.stdout:0/4: creat f2 x:0 0 0 2024-04-12T05:36:44.337 INFO:tasks.workunit.client.1.smithi067.stdout:1/11: mknod c1 0 2024-04-12T05:36:44.337 INFO:tasks.workunit.client.1.smithi067.stdout:1/12: fdatasync f0 0 2024-04-12T05:36:44.339 INFO:tasks.workunit.client.1.smithi067.stdout:2/4: mknod c1 0 2024-04-12T05:36:44.339 INFO:tasks.workunit.client.1.smithi067.stdout:3/0: mkdir d0 0 2024-04-12T05:36:44.341 INFO:tasks.workunit.client.1.smithi067.stdout:6/0: rename - no filename 2024-04-12T05:36:44.341 INFO:tasks.workunit.client.1.smithi067.stdout:6/1: rename - no filename 2024-04-12T05:36:44.342 INFO:tasks.workunit.client.1.smithi067.stdout:2/5: mknod c2 0 2024-04-12T05:36:44.342 INFO:tasks.workunit.client.1.smithi067.stdout:2/6: dwrite - no filename 2024-04-12T05:36:44.342 INFO:tasks.workunit.client.1.smithi067.stdout:2/7: dread - no filename 2024-04-12T05:36:44.342 INFO:tasks.workunit.client.1.smithi067.stdout:2/8: write - no filename 2024-04-12T05:36:44.342 INFO:tasks.workunit.client.1.smithi067.stdout:2/9: truncate - no filename 2024-04-12T05:36:44.342 INFO:tasks.workunit.client.1.smithi067.stdout:1/13: rename f0 to f2 0 2024-04-12T05:36:44.342 INFO:tasks.workunit.client.1.smithi067.stdout:1/14: stat f2 0 2024-04-12T05:36:44.345 INFO:tasks.workunit.client.1.smithi067.stdout:2/10: creat f3 x:0 0 0 2024-04-12T05:36:44.345 INFO:tasks.workunit.client.1.smithi067.stdout:5/0: mkdir d0 0 2024-04-12T05:36:44.346 INFO:tasks.workunit.client.1.smithi067.stdout:1/15: write f2 [453569,118097] 0 2024-04-12T05:36:44.349 INFO:tasks.workunit.client.1.smithi067.stdout:3/1: rmdir d0 0 2024-04-12T05:36:44.350 INFO:tasks.workunit.client.1.smithi067.stdout:2/11: creat f4 x:0 0 0 2024-04-12T05:36:44.350 INFO:tasks.workunit.client.1.smithi067.stdout:2/12: dread - f4 zero size 2024-04-12T05:36:44.351 INFO:tasks.workunit.client.1.smithi067.stdout:7/0: getdents . 0 2024-04-12T05:36:44.351 INFO:tasks.workunit.client.1.smithi067.stdout:7/1: fsync - no filename 2024-04-12T05:36:44.352 INFO:tasks.workunit.client.1.smithi067.stdout:7/2: dwrite - no filename 2024-04-12T05:36:44.352 INFO:tasks.workunit.client.1.smithi067.stdout:7/3: link - no file 2024-04-12T05:36:44.352 INFO:tasks.workunit.client.1.smithi067.stdout:7/4: link - no file 2024-04-12T05:36:44.353 INFO:tasks.workunit.client.1.smithi067.stdout:6/2: creat f0 x:0 0 0 2024-04-12T05:36:44.353 INFO:tasks.workunit.client.1.smithi067.stdout:3/2: creat f1 x:0 0 0 2024-04-12T05:36:44.354 INFO:tasks.workunit.client.1.smithi067.stdout:9/0: write - no filename 2024-04-12T05:36:44.354 INFO:tasks.workunit.client.1.smithi067.stdout:9/1: dread - no filename 2024-04-12T05:36:44.354 INFO:tasks.workunit.client.1.smithi067.stdout:9/2: fdatasync - no filename 2024-04-12T05:36:44.354 INFO:tasks.workunit.client.1.smithi067.stdout:9/3: write - no filename 2024-04-12T05:36:44.355 INFO:tasks.workunit.client.1.smithi067.stdout:3/3: fdatasync f1 0 2024-04-12T05:36:44.355 INFO:tasks.workunit.client.1.smithi067.stdout:5/1: creat d0/f1 x:0 0 0 2024-04-12T05:36:44.360 INFO:tasks.workunit.client.1.smithi067.stdout:3/4: write f1 [553224,20584] 0 2024-04-12T05:36:44.360 INFO:tasks.workunit.client.1.smithi067.stdout:7/5: mkdir d0 0 2024-04-12T05:36:44.360 INFO:tasks.workunit.client.1.smithi067.stdout:7/6: dread - no filename 2024-04-12T05:36:44.360 INFO:tasks.workunit.client.1.smithi067.stdout:8/0: creat f0 x:0 0 0 2024-04-12T05:36:44.360 INFO:tasks.workunit.client.1.smithi067.stdout:7/7: chown d0 405462 1 2024-04-12T05:36:44.361 INFO:tasks.workunit.client.1.smithi067.stdout:7/8: readlink - no filename 2024-04-12T05:36:44.362 INFO:tasks.workunit.client.1.smithi067.stdout:5/2: creat d0/f2 x:0 0 0 2024-04-12T05:36:44.363 INFO:tasks.workunit.client.1.smithi067.stdout:8/1: creat f1 x:0 0 0 2024-04-12T05:36:44.363 INFO:tasks.workunit.client.1.smithi067.stdout:5/3: dread - d0/f1 zero size 2024-04-12T05:36:44.365 INFO:tasks.workunit.client.1.smithi067.stdout:9/4: mknod c0 0 2024-04-12T05:36:44.365 INFO:tasks.workunit.client.1.smithi067.stdout:9/5: write - no filename 2024-04-12T05:36:44.365 INFO:tasks.workunit.client.1.smithi067.stdout:9/6: write - no filename 2024-04-12T05:36:44.365 INFO:tasks.workunit.client.1.smithi067.stdout:8/2: fdatasync f0 0 2024-04-12T05:36:44.367 INFO:tasks.workunit.client.1.smithi067.stdout:2/13: dwrite f4 [0,4194304] 0 2024-04-12T05:36:44.367 INFO:tasks.workunit.client.1.smithi067.stdout:9/7: chown c0 25297 1 2024-04-12T05:36:44.368 INFO:tasks.workunit.client.1.smithi067.stdout:9/8: write - no filename 2024-04-12T05:36:44.368 INFO:tasks.workunit.client.1.smithi067.stdout:8/3: dread - f0 zero size 2024-04-12T05:36:44.368 INFO:tasks.workunit.client.1.smithi067.stdout:2/14: rmdir - no directory 2024-04-12T05:36:44.368 INFO:tasks.workunit.client.1.smithi067.stdout:5/4: mkdir d0/d3 0 2024-04-12T05:36:44.368 INFO:tasks.workunit.client.1.smithi067.stdout:3/5: fdatasync f1 0 2024-04-12T05:36:44.368 INFO:tasks.workunit.client.1.smithi067.stdout:7/9: creat d0/f1 x:0 0 0 2024-04-12T05:36:44.370 INFO:tasks.workunit.client.1.smithi067.stdout:8/4: fdatasync f1 0 2024-04-12T05:36:44.370 INFO:tasks.workunit.client.1.smithi067.stdout:2/15: mkdir d5 0 2024-04-12T05:36:44.370 INFO:tasks.workunit.client.1.smithi067.stdout:2/16: stat c1 0 2024-04-12T05:36:44.372 INFO:tasks.workunit.client.1.smithi067.stdout:2/17: read - f3 zero size 2024-04-12T05:36:44.372 INFO:tasks.workunit.client.1.smithi067.stdout:7/10: dread - d0/f1 zero size 2024-04-12T05:36:44.372 INFO:tasks.workunit.client.1.smithi067.stdout:8/5: dread - f1 zero size 2024-04-12T05:36:44.377 INFO:tasks.workunit.client.1.smithi067.stdout:3/6: chown f1 14292860 1 2024-04-12T05:36:44.378 INFO:tasks.workunit.client.1.smithi067.stdout:3/7: chown f1 1381075655 1 2024-04-12T05:36:44.379 INFO:tasks.workunit.client.1.smithi067.stdout:8/6: rename f0 to f2 0 2024-04-12T05:36:44.379 INFO:tasks.workunit.client.1.smithi067.stdout:7/11: mkdir d0/d2 0 2024-04-12T05:36:44.380 INFO:tasks.workunit.client.1.smithi067.stdout:8/7: chown f2 41750208 1 2024-04-12T05:36:44.380 INFO:tasks.workunit.client.1.smithi067.stdout:2/18: symlink d5/l6 0 2024-04-12T05:36:44.380 INFO:tasks.workunit.client.1.smithi067.stdout:3/8: write f1 [1392994,51716] 0 2024-04-12T05:36:44.383 INFO:tasks.workunit.client.1.smithi067.stdout:2/19: mknod d5/c7 0 2024-04-12T05:36:44.385 INFO:tasks.workunit.client.1.smithi067.stdout:7/12: creat d0/d2/f3 x:0 0 0 2024-04-12T05:36:44.386 INFO:tasks.workunit.client.1.smithi067.stdout:2/20: write f4 [411791,119294] 0 2024-04-12T05:36:44.386 INFO:tasks.workunit.client.1.smithi067.stdout:3/9: creat f2 x:0 0 0 2024-04-12T05:36:44.391 INFO:tasks.workunit.client.1.smithi067.stdout:3/10: rename f2 to f3 0 2024-04-12T05:36:44.392 INFO:tasks.workunit.client.1.smithi067.stdout:2/21: rename d5/c7 to d5/c8 0 2024-04-12T05:36:44.394 INFO:tasks.workunit.client.1.smithi067.stdout:3/11: chown f1 103940255 1 2024-04-12T05:36:44.398 INFO:tasks.workunit.client.1.smithi067.stdout:8/8: dwrite f2 [0,4194304] 0 2024-04-12T05:36:44.398 INFO:tasks.workunit.client.1.smithi067.stdout:3/12: symlink l4 0 2024-04-12T05:36:44.401 INFO:tasks.workunit.client.1.smithi067.stdout:8/9: truncate f1 466457 0 2024-04-12T05:36:44.403 INFO:tasks.workunit.client.1.smithi067.stdout:3/13: fdatasync f3 0 2024-04-12T05:36:44.403 INFO:tasks.workunit.client.1.smithi067.stdout:7/13: dwrite d0/f1 [0,4194304] 0 2024-04-12T05:36:44.405 INFO:tasks.workunit.client.1.smithi067.stdout:8/10: symlink l3 0 2024-04-12T05:36:44.406 INFO:tasks.workunit.client.1.smithi067.stdout:8/11: write f2 [1317599,22851] 0 2024-04-12T05:36:44.414 INFO:tasks.workunit.client.1.smithi067.stdout:2/22: dread f4 [0,4194304] 0 2024-04-12T05:36:44.423 INFO:tasks.workunit.client.1.smithi067.stdout:3/14: dwrite f3 [0,4194304] 0 2024-04-12T05:36:44.438 INFO:tasks.workunit.client.1.smithi067.stdout:3/15: fdatasync f3 0 2024-04-12T05:36:44.456 INFO:tasks.workunit.client.1.smithi067.stdout:0/5: getdents . 0 2024-04-12T05:36:44.457 INFO:tasks.workunit.client.1.smithi067.stdout:0/6: read - f2 zero size 2024-04-12T05:36:44.459 INFO:tasks.workunit.client.1.smithi067.stdout:0/7: getdents . 0 2024-04-12T05:36:44.462 INFO:tasks.workunit.client.1.smithi067.stdout:1/16: rename f2 to f3 0 2024-04-12T05:36:44.463 INFO:tasks.workunit.client.1.smithi067.stdout:0/8: dread f1 [0,4194304] 0 2024-04-12T05:36:44.468 INFO:tasks.workunit.client.1.smithi067.stdout:5/5: fsync d0/f1 0 2024-04-12T05:36:44.470 INFO:tasks.workunit.client.1.smithi067.stdout:5/6: dread - d0/f1 zero size 2024-04-12T05:36:44.474 INFO:tasks.workunit.client.1.smithi067.stdout:5/7: creat d0/f4 x:0 0 0 2024-04-12T05:36:44.475 INFO:tasks.workunit.client.1.smithi067.stdout:0/9: dwrite f1 [0,4194304] 0 2024-04-12T05:36:44.476 INFO:tasks.workunit.client.1.smithi067.stdout:0/10: truncate f2 388339 0 2024-04-12T05:36:44.476 INFO:tasks.workunit.client.1.smithi067.stdout:5/8: truncate d0/f4 349699 0 2024-04-12T05:36:44.477 INFO:tasks.workunit.client.1.smithi067.stdout:5/9: truncate d0/f1 930468 0 2024-04-12T05:36:44.478 INFO:tasks.workunit.client.1.smithi067.stdout:0/11: creat f3 x:0 0 0 2024-04-12T05:36:44.478 INFO:tasks.workunit.client.1.smithi067.stdout:7/14: fsync d0/d2/f3 0 2024-04-12T05:36:44.479 INFO:tasks.workunit.client.1.smithi067.stdout:7/15: chown d0/f1 7 1 2024-04-12T05:36:44.481 INFO:tasks.workunit.client.1.smithi067.stdout:5/10: symlink d0/d3/l5 0 2024-04-12T05:36:44.482 INFO:tasks.workunit.client.1.smithi067.stdout:5/11: fsync d0/f1 0 2024-04-12T05:36:44.483 INFO:tasks.workunit.client.1.smithi067.stdout:7/16: creat d0/f4 x:0 0 0 2024-04-12T05:36:44.485 INFO:tasks.workunit.client.1.smithi067.stdout:7/17: dread - d0/f4 zero size 2024-04-12T05:36:44.487 INFO:tasks.workunit.client.1.smithi067.stdout:5/12: mkdir d0/d6 0 2024-04-12T05:36:44.488 INFO:tasks.workunit.client.1.smithi067.stdout:0/12: dwrite f2 [0,4194304] 0 2024-04-12T05:36:44.489 INFO:tasks.workunit.client.1.smithi067.stdout:5/13: truncate d0/f4 1142362 0 2024-04-12T05:36:44.490 INFO:tasks.workunit.client.1.smithi067.stdout:7/18: mkdir d0/d2/d5 0 2024-04-12T05:36:44.493 INFO:tasks.workunit.client.1.smithi067.stdout:8/12: truncate f1 41120 0 2024-04-12T05:36:44.494 INFO:tasks.workunit.client.1.smithi067.stdout:7/19: symlink d0/l6 0 2024-04-12T05:36:44.494 INFO:tasks.workunit.client.1.smithi067.stdout:2/23: truncate f4 3767117 0 2024-04-12T05:36:44.500 INFO:tasks.workunit.client.1.smithi067.stdout:2/24: symlink d5/l9 0 2024-04-12T05:36:44.502 INFO:tasks.workunit.client.1.smithi067.stdout:7/20: creat d0/d2/f7 x:0 0 0 2024-04-12T05:36:44.504 INFO:tasks.workunit.client.1.smithi067.stdout:0/13: dread f1 [0,4194304] 0 2024-04-12T05:36:44.505 INFO:tasks.workunit.client.1.smithi067.stdout:7/21: chown d0/d2 15 1 2024-04-12T05:36:44.507 INFO:tasks.workunit.client.1.smithi067.stdout:5/14: dwrite d0/f2 [0,4194304] 0 2024-04-12T05:36:44.511 INFO:tasks.workunit.client.1.smithi067.stdout:2/25: chown f4 26218 1 2024-04-12T05:36:44.511 INFO:tasks.workunit.client.1.smithi067.stdout:5/15: fdatasync d0/f1 0 2024-04-12T05:36:44.513 INFO:tasks.workunit.client.1.smithi067.stdout:1/17: dwrite f3 [0,4194304] 0 2024-04-12T05:36:44.515 INFO:tasks.workunit.client.1.smithi067.stdout:7/22: link d0/f4 d0/f8 0 2024-04-12T05:36:44.517 INFO:tasks.workunit.client.1.smithi067.stdout:8/13: truncate f1 732071 0 2024-04-12T05:36:44.518 INFO:tasks.workunit.client.1.smithi067.stdout:1/18: mkdir d4 0 2024-04-12T05:36:44.525 INFO:tasks.workunit.client.1.smithi067.stdout:8/14: creat f4 x:0 0 0 2024-04-12T05:36:44.525 INFO:tasks.workunit.client.1.smithi067.stdout:0/14: dwrite f1 [0,4194304] 0 2024-04-12T05:36:44.525 INFO:tasks.workunit.client.1.smithi067.stdout:1/19: chown f3 4305096 1 2024-04-12T05:36:44.527 INFO:tasks.workunit.client.1.smithi067.stdout:1/20: chown f3 92 1 2024-04-12T05:36:44.530 INFO:tasks.workunit.client.1.smithi067.stdout:1/21: stat f3 0 2024-04-12T05:36:44.531 INFO:tasks.workunit.client.1.smithi067.stdout:1/22: chown d4 2785 1 2024-04-12T05:36:44.532 INFO:tasks.workunit.client.1.smithi067.stdout:2/26: write f4 [4574658,48635] 0 2024-04-12T05:36:44.533 INFO:tasks.workunit.client.1.smithi067.stdout:8/15: chown f1 40907 1 2024-04-12T05:36:44.538 INFO:tasks.workunit.client.1.smithi067.stdout:1/23: creat d4/f5 x:0 0 0 2024-04-12T05:36:44.539 INFO:tasks.workunit.client.1.smithi067.stdout:2/27: mkdir d5/da 0 2024-04-12T05:36:44.539 INFO:tasks.workunit.client.1.smithi067.stdout:5/16: dwrite d0/f1 [0,4194304] 0 2024-04-12T05:36:44.542 INFO:tasks.workunit.client.1.smithi067.stdout:1/24: fdatasync f3 0 2024-04-12T05:36:44.543 INFO:tasks.workunit.client.1.smithi067.stdout:8/16: read f1 [544351,82974] 0 2024-04-12T05:36:44.543 INFO:tasks.workunit.client.1.smithi067.stdout:5/17: creat d0/f7 x:0 0 0 2024-04-12T05:36:44.544 INFO:tasks.workunit.client.1.smithi067.stdout:0/15: dread f1 [0,4194304] 0 2024-04-12T05:36:44.546 INFO:tasks.workunit.client.1.smithi067.stdout:7/23: dwrite d0/f4 [0,4194304] 0 2024-04-12T05:36:44.546 INFO:tasks.workunit.client.1.smithi067.stdout:8/17: truncate f4 678395 0 2024-04-12T05:36:44.547 INFO:tasks.workunit.client.1.smithi067.stdout:5/18: write d0/f4 [1489750,86641] 0 2024-04-12T05:36:44.554 INFO:tasks.workunit.client.1.smithi067.stdout:1/25: creat d4/f6 x:0 0 0 2024-04-12T05:36:44.554 INFO:tasks.workunit.client.1.smithi067.stdout:5/19: rename d0/d3 to d0/d6/d8 0 2024-04-12T05:36:44.554 INFO:tasks.workunit.client.1.smithi067.stdout:5/20: read d0/f1 [917390,22265] 0 2024-04-12T05:36:44.557 INFO:tasks.workunit.client.1.smithi067.stdout:8/18: getdents . 0 2024-04-12T05:36:44.561 INFO:tasks.workunit.client.1.smithi067.stdout:8/19: write f4 [1107384,30408] 0 2024-04-12T05:36:44.563 INFO:tasks.workunit.client.1.smithi067.stdout:1/26: rename f3 to d4/f7 0 2024-04-12T05:36:44.564 INFO:tasks.workunit.client.1.smithi067.stdout:8/20: fdatasync f2 0 2024-04-12T05:36:44.567 INFO:tasks.workunit.client.1.smithi067.stdout:0/16: dwrite f3 [0,4194304] 0 2024-04-12T05:36:44.574 INFO:tasks.workunit.client.1.smithi067.stdout:1/27: rmdir d4 39 2024-04-12T05:36:44.574 INFO:tasks.workunit.client.1.smithi067.stdout:8/21: creat f5 x:0 0 0 2024-04-12T05:36:44.574 INFO:tasks.workunit.client.1.smithi067.stdout:5/21: dwrite d0/f2 [4194304,4194304] 0 2024-04-12T05:36:44.578 INFO:tasks.workunit.client.1.smithi067.stdout:8/22: mkdir d6 0 2024-04-12T05:36:44.581 INFO:tasks.workunit.client.1.smithi067.stdout:1/28: dread - d4/f6 zero size 2024-04-12T05:36:44.581 INFO:tasks.workunit.client.1.smithi067.stdout:5/22: symlink d0/d6/d8/l9 0 2024-04-12T05:36:44.581 INFO:tasks.workunit.client.1.smithi067.stdout:8/23: chown f2 1 1 2024-04-12T05:36:44.583 INFO:tasks.workunit.client.1.smithi067.stdout:1/29: dread - d4/f5 zero size 2024-04-12T05:36:44.583 INFO:tasks.workunit.client.1.smithi067.stdout:0/17: dwrite f2 [4194304,4194304] 0 2024-04-12T05:36:44.583 INFO:tasks.workunit.client.1.smithi067.stdout:5/23: mkdir d0/da 0 2024-04-12T05:36:44.584 INFO:tasks.workunit.client.1.smithi067.stdout:5/24: stat d0/f1 0 2024-04-12T05:36:44.585 INFO:tasks.workunit.client.1.smithi067.stdout:8/24: write f4 [1822408,92590] 0 2024-04-12T05:36:44.585 INFO:tasks.workunit.client.1.smithi067.stdout:0/18: chown f2 98 1 2024-04-12T05:36:44.586 INFO:tasks.workunit.client.1.smithi067.stdout:1/30: dread - d4/f5 zero size 2024-04-12T05:36:44.587 INFO:tasks.workunit.client.1.smithi067.stdout:5/25: read d0/f1 [2939856,122961] 0 2024-04-12T05:36:44.587 INFO:tasks.workunit.client.1.smithi067.stdout:0/19: symlink l4 0 2024-04-12T05:36:44.588 INFO:tasks.workunit.client.1.smithi067.stdout:5/26: readlink d0/d6/d8/l9 0 2024-04-12T05:36:44.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:44 smithi032 ceph-mon[39731]: pgmap v253: 65 pgs: 65 active+clean; 172 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 457 op/s 2024-04-12T05:36:44.591 INFO:tasks.workunit.client.1.smithi067.stdout:0/20: fdatasync f1 0 2024-04-12T05:36:44.591 INFO:tasks.workunit.client.1.smithi067.stdout:0/21: read f2 [1764397,122754] 0 2024-04-12T05:36:44.591 INFO:tasks.workunit.client.1.smithi067.stdout:0/22: write f1 [4796086,64925] 0 2024-04-12T05:36:44.591 INFO:tasks.workunit.client.1.smithi067.stdout:5/27: unlink d0/f4 0 2024-04-12T05:36:44.592 INFO:tasks.workunit.client.1.smithi067.stdout:5/28: read d0/f2 [3193246,66229] 0 2024-04-12T05:36:44.594 INFO:tasks.workunit.client.1.smithi067.stdout:1/31: link d4/f5 d4/f8 0 2024-04-12T05:36:44.598 INFO:tasks.workunit.client.1.smithi067.stdout:0/23: mknod c5 0 2024-04-12T05:36:44.602 INFO:tasks.workunit.client.1.smithi067.stdout:0/24: mknod c6 0 2024-04-12T05:36:44.602 INFO:tasks.workunit.client.1.smithi067.stdout:5/29: dwrite d0/f7 [0,4194304] 0 2024-04-12T05:36:44.604 INFO:tasks.workunit.client.1.smithi067.stdout:0/25: readlink l4 0 2024-04-12T05:36:44.612 INFO:tasks.workunit.client.1.smithi067.stdout:1/32: dwrite d4/f7 [4194304,4194304] 0 2024-04-12T05:36:44.619 INFO:tasks.workunit.client.1.smithi067.stdout:5/30: dwrite d0/f7 [0,4194304] 0 2024-04-12T05:36:44.620 INFO:tasks.workunit.client.1.smithi067.stdout:0/26: truncate f1 4726623 0 2024-04-12T05:36:44.621 INFO:tasks.workunit.client.1.smithi067.stdout:1/33: dread d4/f7 [4194304,4194304] 0 2024-04-12T05:36:44.622 INFO:tasks.workunit.client.1.smithi067.stdout:1/34: dread - d4/f8 zero size 2024-04-12T05:36:44.623 INFO:tasks.workunit.client.1.smithi067.stdout:5/31: write d0/f2 [6079886,50225] 0 2024-04-12T05:36:44.623 INFO:tasks.workunit.client.1.smithi067.stdout:0/27: mknod c7 0 2024-04-12T05:36:44.624 INFO:tasks.workunit.client.1.smithi067.stdout:5/32: chown d0/d6/d8/l5 0 1 2024-04-12T05:36:44.625 INFO:tasks.workunit.client.1.smithi067.stdout:0/28: mknod c8 0 2024-04-12T05:36:44.627 INFO:tasks.workunit.client.1.smithi067.stdout:5/33: creat d0/fb x:0 0 0 2024-04-12T05:36:44.629 INFO:tasks.workunit.client.1.smithi067.stdout:0/29: rename f3 to f9 0 2024-04-12T05:36:44.630 INFO:tasks.workunit.client.1.smithi067.stdout:0/30: fdatasync f2 0 2024-04-12T05:36:44.631 INFO:tasks.workunit.client.1.smithi067.stdout:0/31: fdatasync f2 0 2024-04-12T05:36:44.635 INFO:tasks.workunit.client.1.smithi067.stdout:0/32: symlink la 0 2024-04-12T05:36:44.636 INFO:tasks.workunit.client.1.smithi067.stdout:1/35: truncate d4/f8 18500 0 2024-04-12T05:36:44.637 INFO:tasks.workunit.client.1.smithi067.stdout:0/33: creat fb x:0 0 0 2024-04-12T05:36:44.637 INFO:tasks.workunit.client.1.smithi067.stdout:1/36: mkdir d4/d9 0 2024-04-12T05:36:44.640 INFO:tasks.workunit.client.1.smithi067.stdout:5/34: fsync d0/fb 0 2024-04-12T05:36:44.642 INFO:tasks.workunit.client.1.smithi067.stdout:5/35: creat d0/fc x:0 0 0 2024-04-12T05:36:44.647 INFO:tasks.workunit.client.1.smithi067.stdout:1/37: getdents d4 0 2024-04-12T05:36:44.651 INFO:tasks.workunit.client.1.smithi067.stdout:1/38: chown d4/f8 29958409 1 2024-04-12T05:36:44.652 INFO:tasks.workunit.client.1.smithi067.stdout:5/36: truncate d0/f7 889170 0 2024-04-12T05:36:44.653 INFO:tasks.workunit.client.1.smithi067.stdout:5/37: chown d0/d6 180 1 2024-04-12T05:36:44.653 INFO:tasks.workunit.client.1.smithi067.stdout:1/39: write d4/f7 [4207549,46312] 0 2024-04-12T05:36:44.654 INFO:tasks.workunit.client.1.smithi067.stdout:5/38: dread - d0/fb zero size 2024-04-12T05:36:44.656 INFO:tasks.workunit.client.1.smithi067.stdout:1/40: dread d4/f8 [0,4194304] 0 2024-04-12T05:36:44.657 INFO:tasks.workunit.client.1.smithi067.stdout:5/39: creat d0/fd x:0 0 0 2024-04-12T05:36:44.658 INFO:tasks.workunit.client.1.smithi067.stdout:1/41: fsync d4/f7 0 2024-04-12T05:36:44.659 INFO:tasks.workunit.client.1.smithi067.stdout:1/42: chown d4/f7 1 1 2024-04-12T05:36:44.661 INFO:tasks.workunit.client.1.smithi067.stdout:0/34: dwrite f1 [0,4194304] 0 2024-04-12T05:36:44.669 INFO:tasks.workunit.client.1.smithi067.stdout:5/40: dread d0/f7 [0,4194304] 0 2024-04-12T05:36:44.669 INFO:tasks.workunit.client.1.smithi067.stdout:0/35: dread f1 [0,4194304] 0 2024-04-12T05:36:44.670 INFO:tasks.workunit.client.1.smithi067.stdout:1/43: truncate d4/f7 7678781 0 2024-04-12T05:36:44.675 INFO:tasks.workunit.client.1.smithi067.stdout:0/36: creat fc x:0 0 0 2024-04-12T05:36:44.676 INFO:tasks.workunit.client.1.smithi067.stdout:0/37: chown f1 28557 1 2024-04-12T05:36:44.677 INFO:tasks.workunit.client.1.smithi067.stdout:5/41: mkdir d0/da/de 0 2024-04-12T05:36:44.678 INFO:tasks.workunit.client.1.smithi067.stdout:0/38: mkdir dd 0 2024-04-12T05:36:44.685 INFO:tasks.workunit.client.1.smithi067.stdout:1/44: write d4/f7 [8551799,72596] 0 2024-04-12T05:36:44.686 INFO:tasks.workunit.client.1.smithi067.stdout:5/42: truncate d0/f2 4085610 0 2024-04-12T05:36:44.686 INFO:tasks.workunit.client.1.smithi067.stdout:0/39: truncate f9 3583190 0 2024-04-12T05:36:44.688 INFO:tasks.workunit.client.1.smithi067.stdout:5/43: chown d0/d6/d8 16 1 2024-04-12T05:36:44.691 INFO:tasks.workunit.client.1.smithi067.stdout:1/45: link d4/f5 d4/d9/fa 0 2024-04-12T05:36:44.695 INFO:tasks.workunit.client.1.smithi067.stdout:0/40: dwrite fc [0,4194304] 0 2024-04-12T05:36:44.701 INFO:tasks.workunit.client.1.smithi067.stdout:1/46: dread d4/f7 [0,4194304] 0 2024-04-12T05:36:44.703 INFO:tasks.workunit.client.1.smithi067.stdout:0/41: dread f1 [0,4194304] 0 2024-04-12T05:36:44.706 INFO:tasks.workunit.client.1.smithi067.stdout:1/47: link d4/f6 d4/d9/fb 0 2024-04-12T05:36:44.714 INFO:tasks.workunit.client.1.smithi067.stdout:0/42: truncate f1 1310933 0 2024-04-12T05:36:44.718 INFO:tasks.workunit.client.1.smithi067.stdout:0/43: chown f1 103557763 1 2024-04-12T05:36:44.718 INFO:tasks.workunit.client.1.smithi067.stdout:0/44: stat f2 0 2024-04-12T05:36:44.721 INFO:tasks.workunit.client.1.smithi067.stdout:1/48: dwrite d4/f5 [0,4194304] 0 2024-04-12T05:36:44.722 INFO:tasks.workunit.client.1.smithi067.stdout:1/49: stat d4 0 2024-04-12T05:36:44.723 INFO:tasks.workunit.client.1.smithi067.stdout:0/45: mknod dd/ce 0 2024-04-12T05:36:44.728 INFO:tasks.workunit.client.1.smithi067.stdout:1/50: creat d4/fc x:0 0 0 2024-04-12T05:36:44.729 INFO:tasks.workunit.client.1.smithi067.stdout:1/51: dread - d4/f6 zero size 2024-04-12T05:36:44.736 INFO:tasks.workunit.client.1.smithi067.stdout:0/46: getdents dd 0 2024-04-12T05:36:44.738 INFO:tasks.workunit.client.1.smithi067.stdout:1/52: dwrite d4/f5 [0,4194304] 0 2024-04-12T05:36:44.739 INFO:tasks.workunit.client.1.smithi067.stdout:0/47: symlink dd/lf 0 2024-04-12T05:36:44.746 INFO:tasks.workunit.client.1.smithi067.stdout:1/53: creat d4/d9/fd x:0 0 0 2024-04-12T05:36:44.746 INFO:tasks.workunit.client.1.smithi067.stdout:1/54: write d4/f7 [5782011,38914] 0 2024-04-12T05:36:44.749 INFO:tasks.workunit.client.1.smithi067.stdout:1/55: creat d4/fe x:0 0 0 2024-04-12T05:36:44.750 INFO:tasks.workunit.client.1.smithi067.stdout:0/48: dwrite fc [0,4194304] 0 2024-04-12T05:36:44.752 INFO:tasks.workunit.client.1.smithi067.stdout:1/56: symlink d4/lf 0 2024-04-12T05:36:44.754 INFO:tasks.workunit.client.1.smithi067.stdout:0/49: symlink dd/l10 0 2024-04-12T05:36:44.754 INFO:tasks.workunit.client.1.smithi067.stdout:0/50: chown c8 2489 1 2024-04-12T05:36:44.755 INFO:tasks.workunit.client.1.smithi067.stdout:0/51: dread - fb zero size 2024-04-12T05:36:44.758 INFO:tasks.workunit.client.1.smithi067.stdout:0/52: symlink dd/l11 0 2024-04-12T05:36:44.760 INFO:tasks.workunit.client.1.smithi067.stdout:0/53: mkdir dd/d12 0 2024-04-12T05:36:44.762 INFO:tasks.workunit.client.1.smithi067.stdout:0/54: creat dd/f13 x:0 0 0 2024-04-12T05:36:44.765 INFO:tasks.workunit.client.1.smithi067.stdout:1/57: write d4/f6 [1041918,51328] 0 2024-04-12T05:36:44.767 INFO:tasks.workunit.client.1.smithi067.stdout:0/55: write fc [3663250,104850] 0 2024-04-12T05:36:44.769 INFO:tasks.workunit.client.1.smithi067.stdout:0/56: mkdir dd/d12/d14 0 2024-04-12T05:36:44.772 INFO:tasks.workunit.client.1.smithi067.stdout:0/57: truncate f9 4179169 0 2024-04-12T05:36:44.772 INFO:tasks.workunit.client.1.smithi067.stdout:0/58: write f2 [5820349,20700] 0 2024-04-12T05:36:44.775 INFO:tasks.workunit.client.1.smithi067.stdout:0/59: unlink dd/l11 0 2024-04-12T05:36:44.776 INFO:tasks.workunit.client.1.smithi067.stdout:0/60: dread - dd/f13 zero size 2024-04-12T05:36:44.784 INFO:tasks.workunit.client.1.smithi067.stdout:0/61: dwrite f2 [8388608,4194304] 0 2024-04-12T05:36:44.786 INFO:tasks.workunit.client.1.smithi067.stdout:0/62: write dd/f13 [920651,2967] 0 2024-04-12T05:36:44.797 INFO:tasks.workunit.client.1.smithi067.stdout:0/63: getdents dd 0 2024-04-12T05:36:44.799 INFO:tasks.workunit.client.1.smithi067.stdout:0/64: rename f2 to dd/d12/d14/f15 0 2024-04-12T05:36:44.801 INFO:tasks.workunit.client.1.smithi067.stdout:0/65: rmdir dd/d12/d14 39 2024-04-12T05:36:44.802 INFO:tasks.workunit.client.1.smithi067.stdout:0/66: chown la 188995 1 2024-04-12T05:36:44.805 INFO:tasks.workunit.client.1.smithi067.stdout:0/67: creat dd/d12/f16 x:0 0 0 2024-04-12T05:36:44.809 INFO:tasks.workunit.client.1.smithi067.stdout:0/68: rename dd/lf to dd/d12/d14/l17 0 2024-04-12T05:36:44.812 INFO:tasks.workunit.client.1.smithi067.stdout:0/69: creat dd/f18 x:0 0 0 2024-04-12T05:36:44.813 INFO:tasks.workunit.client.1.smithi067.stdout:0/70: dread - dd/d12/f16 zero size 2024-04-12T05:36:44.814 INFO:tasks.workunit.client.1.smithi067.stdout:0/71: symlink dd/l19 0 2024-04-12T05:36:44.816 INFO:tasks.workunit.client.1.smithi067.stdout:0/72: rmdir dd/d12/d14 39 2024-04-12T05:36:44.824 INFO:tasks.workunit.client.1.smithi067.stdout:0/73: dwrite dd/f18 [0,4194304] 0 2024-04-12T05:36:44.824 INFO:tasks.workunit.client.1.smithi067.stdout:0/74: dread - fb zero size 2024-04-12T05:36:44.827 INFO:tasks.workunit.client.1.smithi067.stdout:0/75: mkdir dd/d12/d14/d1a 0 2024-04-12T05:36:44.837 INFO:tasks.workunit.client.1.smithi067.stdout:0/76: dwrite dd/d12/f16 [0,4194304] 0 2024-04-12T05:36:44.843 INFO:tasks.workunit.client.1.smithi067.stdout:0/77: mknod dd/d12/d14/d1a/c1b 0 2024-04-12T05:36:44.844 INFO:tasks.workunit.client.1.smithi067.stdout:0/78: truncate dd/f13 1795250 0 2024-04-12T05:36:44.846 INFO:tasks.workunit.client.1.smithi067.stdout:0/79: read dd/f13 [42761,60296] 0 2024-04-12T05:36:44.858 INFO:tasks.workunit.client.1.smithi067.stdout:0/80: dwrite dd/f13 [0,4194304] 0 2024-04-12T05:36:44.870 INFO:tasks.workunit.client.1.smithi067.stdout:0/81: dwrite dd/d12/f16 [4194304,4194304] 0 2024-04-12T05:36:44.873 INFO:tasks.workunit.client.1.smithi067.stdout:0/82: chown c8 831609 1 2024-04-12T05:36:44.875 INFO:tasks.workunit.client.1.smithi067.stdout:0/83: creat dd/f1c x:0 0 0 2024-04-12T05:36:44.877 INFO:tasks.workunit.client.1.smithi067.stdout:0/84: chown dd/d12/d14/d1a 0 1 2024-04-12T05:36:44.880 INFO:tasks.workunit.client.1.smithi067.stdout:0/85: symlink dd/d12/d14/d1a/l1d 0 2024-04-12T05:36:44.887 INFO:tasks.workunit.client.1.smithi067.stdout:0/86: dread f9 [0,4194304] 0 2024-04-12T05:36:44.891 INFO:tasks.workunit.client.1.smithi067.stdout:0/87: rmdir dd 39 2024-04-12T05:36:44.892 INFO:tasks.workunit.client.1.smithi067.stdout:0/88: stat l4 0 2024-04-12T05:36:44.892 INFO:tasks.workunit.client.1.smithi067.stdout:0/89: fsync fb 0 2024-04-12T05:36:44.894 INFO:tasks.workunit.client.1.smithi067.stdout:0/90: unlink fc 0 2024-04-12T05:36:44.898 INFO:tasks.workunit.client.1.smithi067.stdout:0/91: creat dd/d12/d14/d1a/f1e x:0 0 0 2024-04-12T05:36:44.900 INFO:tasks.workunit.client.1.smithi067.stdout:0/92: mknod dd/d12/d14/c1f 0 2024-04-12T05:36:44.904 INFO:tasks.workunit.client.1.smithi067.stdout:0/93: dread dd/d12/d14/f15 [8388608,4194304] 0 2024-04-12T05:36:44.907 INFO:tasks.workunit.client.1.smithi067.stdout:0/94: truncate f1 1103993 0 2024-04-12T05:36:44.925 INFO:tasks.workunit.client.1.smithi067.stdout:0/95: dwrite f1 [0,4194304] 0 2024-04-12T05:36:44.929 INFO:tasks.workunit.client.1.smithi067.stdout:0/96: write f9 [1523861,93838] 0 2024-04-12T05:36:44.930 INFO:tasks.workunit.client.1.smithi067.stdout:0/97: stat dd/d12/d14 0 2024-04-12T05:36:44.934 INFO:tasks.workunit.client.1.smithi067.stdout:0/98: mknod dd/d12/d14/d1a/c20 0 2024-04-12T05:36:44.934 INFO:tasks.workunit.client.1.smithi067.stdout:0/99: dread - fb zero size 2024-04-12T05:36:44.945 INFO:tasks.workunit.client.1.smithi067.stdout:0/100: dwrite fb [0,4194304] 0 2024-04-12T05:36:44.946 INFO:tasks.workunit.client.1.smithi067.stdout:0/101: read - dd/f1c zero size 2024-04-12T05:36:44.946 INFO:tasks.workunit.client.1.smithi067.stdout:0/102: fdatasync f9 0 2024-04-12T05:36:44.947 INFO:tasks.workunit.client.1.smithi067.stdout:0/103: write f1 [2932402,72849] 0 2024-04-12T05:36:44.954 INFO:tasks.workunit.client.1.smithi067.stdout:0/104: creat dd/d12/f21 x:0 0 0 2024-04-12T05:36:44.961 INFO:tasks.workunit.client.1.smithi067.stdout:0/105: write dd/d12/d14/f15 [11484952,47487] 0 2024-04-12T05:36:44.962 INFO:tasks.workunit.client.1.smithi067.stdout:0/106: write fb [1195187,71655] 0 2024-04-12T05:36:44.967 INFO:tasks.workunit.client.1.smithi067.stdout:0/107: dread fb [0,4194304] 0 2024-04-12T05:36:44.970 INFO:tasks.workunit.client.1.smithi067.stdout:0/108: mknod dd/d12/c22 0 2024-04-12T05:36:44.971 INFO:tasks.workunit.client.1.smithi067.stdout:0/109: mknod dd/d12/d14/c23 0 2024-04-12T05:36:44.974 INFO:tasks.workunit.client.1.smithi067.stdout:0/110: symlink dd/d12/l24 0 2024-04-12T05:36:44.979 INFO:tasks.workunit.client.1.smithi067.stdout:0/111: dread dd/f13 [0,4194304] 0 2024-04-12T05:36:44.981 INFO:tasks.workunit.client.1.smithi067.stdout:0/112: write dd/f18 [1517723,78509] 0 2024-04-12T05:36:44.987 INFO:tasks.workunit.client.1.smithi067.stdout:0/113: dwrite dd/d12/f16 [0,4194304] 0 2024-04-12T05:36:44.988 INFO:tasks.workunit.client.1.smithi067.stdout:0/114: stat c5 0 2024-04-12T05:36:44.988 INFO:tasks.workunit.client.1.smithi067.stdout:0/115: read f9 [3865803,14165] 0 2024-04-12T05:36:44.996 INFO:tasks.workunit.client.1.smithi067.stdout:0/116: creat dd/d12/f25 x:0 0 0 2024-04-12T05:36:45.008 INFO:tasks.workunit.client.1.smithi067.stdout:0/117: truncate dd/d12/f16 2344469 0 2024-04-12T05:36:45.011 INFO:tasks.workunit.client.1.smithi067.stdout:0/118: mknod dd/d12/d14/d1a/c26 0 2024-04-12T05:36:45.021 INFO:tasks.workunit.client.1.smithi067.stdout:0/119: dwrite dd/d12/f16 [0,4194304] 0 2024-04-12T05:36:45.022 INFO:tasks.workunit.client.1.smithi067.stdout:0/120: read - dd/d12/f25 zero size 2024-04-12T05:36:45.030 INFO:tasks.workunit.client.1.smithi067.stdout:0/121: dwrite dd/f1c [0,4194304] 0 2024-04-12T05:36:46.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:46 smithi067 ceph-mon[49010]: pgmap v254: 65 pgs: 65 active+clean; 174 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 405 op/s 2024-04-12T05:36:46.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:46 smithi032 ceph-mon[39731]: pgmap v254: 65 pgs: 65 active+clean; 174 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 405 op/s 2024-04-12T05:36:48.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:48 smithi067 ceph-mon[49010]: pgmap v255: 65 pgs: 65 active+clean; 275 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.4 MiB/s rd, 15 MiB/s wr, 528 op/s 2024-04-12T05:36:48.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:48 smithi032 ceph-mon[39731]: pgmap v255: 65 pgs: 65 active+clean; 275 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.4 MiB/s rd, 15 MiB/s wr, 528 op/s 2024-04-12T05:36:50.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:50 smithi067 ceph-mon[49010]: pgmap v256: 65 pgs: 65 active+clean; 275 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 14 MiB/s wr, 358 op/s 2024-04-12T05:36:50.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:50 smithi032 ceph-mon[39731]: pgmap v256: 65 pgs: 65 active+clean; 275 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 14 MiB/s wr, 358 op/s 2024-04-12T05:36:52.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:52 smithi067 ceph-mon[49010]: pgmap v257: 65 pgs: 65 active+clean; 286 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 15 MiB/s wr, 415 op/s 2024-04-12T05:36:52.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:52 smithi032 ceph-mon[39731]: pgmap v257: 65 pgs: 65 active+clean; 286 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 15 MiB/s wr, 415 op/s 2024-04-12T05:36:54.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:54 smithi067 ceph-mon[49010]: pgmap v258: 65 pgs: 65 active+clean; 291 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 15 MiB/s wr, 365 op/s 2024-04-12T05:36:54.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:54 smithi032 ceph-mon[39731]: pgmap v258: 65 pgs: 65 active+clean; 291 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 15 MiB/s wr, 365 op/s 2024-04-12T05:36:55.276 INFO:tasks.workunit.client.0.smithi032.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-04-12T05:36:55.285 INFO:tasks.workunit.client.0.smithi032.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-12T05:36:55.285 INFO:tasks.workunit.client.0.smithi032.stderr:+ make 2024-04-12T05:36:55.323 INFO:tasks.workunit.client.0.smithi032.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-04-12T05:36:55.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:55 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:36:55.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:55 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:36:55.690 INFO:tasks.workunit.client.0.smithi032.stderr:++ readlink -f fsstress 2024-04-12T05:36:55.691 INFO:tasks.workunit.client.0.smithi032.stderr:+ BIN=/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-04-12T05:36:55.691 INFO:tasks.workunit.client.0.smithi032.stderr:+ popd 2024-04-12T05:36:55.692 INFO:tasks.workunit.client.0.smithi032.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-12T05:36:55.693 INFO:tasks.workunit.client.0.smithi032.stderr:+ popd 2024-04-12T05:36:55.693 INFO:tasks.workunit.client.0.smithi032.stdout:~/cephtest/mnt.0/client.0/tmp 2024-04-12T05:36:55.694 INFO:tasks.workunit.client.0.smithi032.stderr:++ mktemp -d -p . 2024-04-12T05:36:55.696 INFO:tasks.workunit.client.0.smithi032.stderr:+ T=./tmp.GNFw4y39uT 2024-04-12T05:36:55.696 INFO:tasks.workunit.client.0.smithi032.stderr:+ /home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.GNFw4y39uT -l 1 -n 1000 -p 10 -v 2024-04-12T05:36:55.699 INFO:tasks.workunit.client.0.smithi032.stdout:seed = 1712485114 2024-04-12T05:36:55.704 INFO:tasks.workunit.client.0.smithi032.stdout:1/0: read - no filename 2024-04-12T05:36:55.705 INFO:tasks.workunit.client.0.smithi032.stdout:0/0: creat f0 x:0 0 0 2024-04-12T05:36:55.706 INFO:tasks.workunit.client.0.smithi032.stdout:2/0: fsync - no filename 2024-04-12T05:36:55.706 INFO:tasks.workunit.client.0.smithi032.stdout:2/1: stat - no entries 2024-04-12T05:36:55.706 INFO:tasks.workunit.client.0.smithi032.stdout:2/2: fsync - no filename 2024-04-12T05:36:55.706 INFO:tasks.workunit.client.0.smithi032.stdout:2/3: dwrite - no filename 2024-04-12T05:36:55.706 INFO:tasks.workunit.client.0.smithi032.stdout:2/4: write - no filename 2024-04-12T05:36:55.706 INFO:tasks.workunit.client.0.smithi032.stdout:1/1: creat f0 x:0 0 0 2024-04-12T05:36:55.707 INFO:tasks.workunit.client.0.smithi032.stdout:3/0: dread - no filename 2024-04-12T05:36:55.707 INFO:tasks.workunit.client.0.smithi032.stdout:3/1: write - no filename 2024-04-12T05:36:55.708 INFO:tasks.workunit.client.0.smithi032.stdout:3/2: read - no filename 2024-04-12T05:36:55.708 INFO:tasks.workunit.client.0.smithi032.stdout:1/2: symlink l1 0 2024-04-12T05:36:55.709 INFO:tasks.workunit.client.0.smithi032.stdout:2/5: creat f0 x:0 0 0 2024-04-12T05:36:55.712 INFO:tasks.workunit.client.0.smithi032.stdout:3/3: creat f0 x:0 0 0 2024-04-12T05:36:55.713 INFO:tasks.workunit.client.0.smithi032.stdout:5/0: write - no filename 2024-04-12T05:36:55.713 INFO:tasks.workunit.client.0.smithi032.stdout:5/1: dread - no filename 2024-04-12T05:36:55.713 INFO:tasks.workunit.client.0.smithi032.stdout:5/2: write - no filename 2024-04-12T05:36:55.713 INFO:tasks.workunit.client.0.smithi032.stdout:1/3: link f0 f2 0 2024-04-12T05:36:55.716 INFO:tasks.workunit.client.0.smithi032.stdout:4/0: symlink l0 0 2024-04-12T05:36:55.716 INFO:tasks.workunit.client.0.smithi032.stdout:6/0: rmdir - no directory 2024-04-12T05:36:55.716 INFO:tasks.workunit.client.0.smithi032.stdout:6/1: chown . 439684641 1 2024-04-12T05:36:55.718 INFO:tasks.workunit.client.0.smithi032.stdout:1/4: rename f2 to f3 0 2024-04-12T05:36:55.720 INFO:tasks.workunit.client.0.smithi032.stdout:7/0: rename - no filename 2024-04-12T05:36:55.720 INFO:tasks.workunit.client.0.smithi032.stdout:5/3: creat f0 x:0 0 0 2024-04-12T05:36:55.721 INFO:tasks.workunit.client.0.smithi032.stdout:7/1: chown . 11404913 1 2024-04-12T05:36:55.721 INFO:tasks.workunit.client.0.smithi032.stdout:7/2: chown . 9699 1 2024-04-12T05:36:55.721 INFO:tasks.workunit.client.0.smithi032.stdout:7/3: write - no filename 2024-04-12T05:36:55.721 INFO:tasks.workunit.client.0.smithi032.stdout:7/4: fsync - no filename 2024-04-12T05:36:55.722 INFO:tasks.workunit.client.0.smithi032.stdout:3/4: dwrite f0 [0,4194304] 0 2024-04-12T05:36:55.722 INFO:tasks.workunit.client.0.smithi032.stdout:5/4: write f0 [801780,108889] 0 2024-04-12T05:36:55.723 INFO:tasks.workunit.client.0.smithi032.stdout:3/5: write f0 [2920930,12236] 0 2024-04-12T05:36:55.723 INFO:tasks.workunit.client.0.smithi032.stdout:1/5: unlink f0 0 2024-04-12T05:36:55.723 INFO:tasks.workunit.client.0.smithi032.stdout:5/5: chown f0 22 1 2024-04-12T05:36:55.724 INFO:tasks.workunit.client.0.smithi032.stdout:5/6: truncate f0 1431277 0 2024-04-12T05:36:55.725 INFO:tasks.workunit.client.0.smithi032.stdout:5/7: chown f0 256820683 1 2024-04-12T05:36:55.731 INFO:tasks.workunit.client.0.smithi032.stdout:9/0: chown . 39688166 1 2024-04-12T05:36:55.731 INFO:tasks.workunit.client.0.smithi032.stdout:9/1: link - no file 2024-04-12T05:36:55.731 INFO:tasks.workunit.client.0.smithi032.stdout:9/2: write - no filename 2024-04-12T05:36:55.731 INFO:tasks.workunit.client.0.smithi032.stdout:9/3: write - no filename 2024-04-12T05:36:55.731 INFO:tasks.workunit.client.0.smithi032.stdout:9/4: unlink - no file 2024-04-12T05:36:55.731 INFO:tasks.workunit.client.0.smithi032.stdout:5/8: creat f1 x:0 0 0 2024-04-12T05:36:55.732 INFO:tasks.workunit.client.0.smithi032.stdout:7/5: mkdir d0 0 2024-04-12T05:36:55.732 INFO:tasks.workunit.client.0.smithi032.stdout:7/6: truncate - no filename 2024-04-12T05:36:55.732 INFO:tasks.workunit.client.0.smithi032.stdout:7/7: unlink - no file 2024-04-12T05:36:55.734 INFO:tasks.workunit.client.0.smithi032.stdout:5/9: mkdir d2 0 2024-04-12T05:36:55.734 INFO:tasks.workunit.client.0.smithi032.stdout:9/5: mknod c0 0 2024-04-12T05:36:55.735 INFO:tasks.workunit.client.0.smithi032.stdout:3/6: dwrite f0 [0,4194304] 0 2024-04-12T05:36:55.735 INFO:tasks.workunit.client.0.smithi032.stdout:5/10: read - f1 zero size 2024-04-12T05:36:55.736 INFO:tasks.workunit.client.0.smithi032.stdout:7/8: symlink d0/l1 0 2024-04-12T05:36:55.736 INFO:tasks.workunit.client.0.smithi032.stdout:5/11: dread - f1 zero size 2024-04-12T05:36:55.737 INFO:tasks.workunit.client.0.smithi032.stdout:9/6: creat f1 x:0 0 0 2024-04-12T05:36:55.738 INFO:tasks.workunit.client.0.smithi032.stdout:3/7: creat f1 x:0 0 0 2024-04-12T05:36:55.738 INFO:tasks.workunit.client.0.smithi032.stdout:3/8: read - f1 zero size 2024-04-12T05:36:55.739 INFO:tasks.workunit.client.0.smithi032.stdout:9/7: mkdir d2 0 2024-04-12T05:36:55.740 INFO:tasks.workunit.client.0.smithi032.stdout:5/12: creat d2/f3 x:0 0 0 2024-04-12T05:36:55.741 INFO:tasks.workunit.client.0.smithi032.stdout:3/9: creat f2 x:0 0 0 2024-04-12T05:36:55.742 INFO:tasks.workunit.client.0.smithi032.stdout:5/13: creat d2/f4 x:0 0 0 2024-04-12T05:36:55.743 INFO:tasks.workunit.client.0.smithi032.stdout:5/14: truncate d2/f3 306070 0 2024-04-12T05:36:55.744 INFO:tasks.workunit.client.0.smithi032.stdout:3/10: rename f1 to f3 0 2024-04-12T05:36:55.744 INFO:tasks.workunit.client.0.smithi032.stdout:5/15: write f1 [912505,75996] 0 2024-04-12T05:36:55.744 INFO:tasks.workunit.client.0.smithi032.stdout:9/8: rmdir d2 0 2024-04-12T05:36:55.745 INFO:tasks.workunit.client.0.smithi032.stdout:5/16: fdatasync d2/f4 0 2024-04-12T05:36:55.745 INFO:tasks.workunit.client.0.smithi032.stdout:5/17: fdatasync f0 0 2024-04-12T05:36:55.746 INFO:tasks.workunit.client.0.smithi032.stdout:9/9: write f1 [290482,124144] 0 2024-04-12T05:36:55.748 INFO:tasks.workunit.client.0.smithi032.stdout:5/18: rename f0 to d2/f5 0 2024-04-12T05:36:55.749 INFO:tasks.workunit.client.0.smithi032.stdout:5/19: chown d2 114911 1 2024-04-12T05:36:55.750 INFO:tasks.workunit.client.0.smithi032.stdout:9/10: rename c0 to c3 0 2024-04-12T05:36:55.753 INFO:tasks.workunit.client.0.smithi032.stdout:9/11: rename f1 to f4 0 2024-04-12T05:36:55.757 INFO:tasks.workunit.client.0.smithi032.stdout:5/20: dwrite d2/f5 [0,4194304] 0 2024-04-12T05:36:55.771 INFO:tasks.workunit.client.0.smithi032.stdout:5/21: dread f1 [0,4194304] 0 2024-04-12T05:36:55.820 INFO:tasks.workunit.client.0.smithi032.stdout:0/1: fsync f0 0 2024-04-12T05:36:55.820 INFO:tasks.workunit.client.0.smithi032.stdout:0/2: rmdir - no directory 2024-04-12T05:36:55.821 INFO:tasks.workunit.client.0.smithi032.stdout:0/3: rename f0 to f1 0 2024-04-12T05:36:55.823 INFO:tasks.workunit.client.0.smithi032.stdout:0/4: write f1 [781054,51267] 0 2024-04-12T05:36:55.823 INFO:tasks.workunit.client.0.smithi032.stdout:2/6: getdents . 0 2024-04-12T05:36:55.824 INFO:tasks.workunit.client.0.smithi032.stdout:2/7: dread - f0 zero size 2024-04-12T05:36:55.825 INFO:tasks.workunit.client.0.smithi032.stdout:0/5: mkdir d2 0 2024-04-12T05:36:55.826 INFO:tasks.workunit.client.0.smithi032.stdout:2/8: creat f1 x:0 0 0 2024-04-12T05:36:55.827 INFO:tasks.workunit.client.0.smithi032.stdout:0/6: symlink d2/l3 0 2024-04-12T05:36:55.828 INFO:tasks.workunit.client.0.smithi032.stdout:4/1: getdents . 0 2024-04-12T05:36:55.828 INFO:tasks.workunit.client.0.smithi032.stdout:4/2: dread - no filename 2024-04-12T05:36:55.828 INFO:tasks.workunit.client.0.smithi032.stdout:4/3: dread - no filename 2024-04-12T05:36:55.832 INFO:tasks.workunit.client.0.smithi032.stdout:4/4: symlink l1 0 2024-04-12T05:36:55.836 INFO:tasks.workunit.client.0.smithi032.stdout:2/9: link f0 f2 0 2024-04-12T05:36:55.836 INFO:tasks.workunit.client.0.smithi032.stdout:2/10: stat f0 0 2024-04-12T05:36:55.839 INFO:tasks.workunit.client.0.smithi032.stdout:4/5: mknod c2 0 2024-04-12T05:36:55.839 INFO:tasks.workunit.client.0.smithi032.stdout:4/6: fsync - no filename 2024-04-12T05:36:55.839 INFO:tasks.workunit.client.0.smithi032.stdout:4/7: fsync - no filename 2024-04-12T05:36:55.843 INFO:tasks.workunit.client.0.smithi032.stdout:2/11: mknod c3 0 2024-04-12T05:36:55.845 INFO:tasks.workunit.client.0.smithi032.stdout:2/12: creat f4 x:0 0 0 2024-04-12T05:36:55.848 INFO:tasks.workunit.client.0.smithi032.stdout:2/13: creat f5 x:0 0 0 2024-04-12T05:36:55.848 INFO:tasks.workunit.client.0.smithi032.stdout:3/11: fsync f2 0 2024-04-12T05:36:55.848 INFO:tasks.workunit.client.0.smithi032.stdout:2/14: stat f5 0 2024-04-12T05:36:55.849 INFO:tasks.workunit.client.0.smithi032.stdout:2/15: chown c3 24 1 2024-04-12T05:36:55.850 INFO:tasks.workunit.client.0.smithi032.stdout:2/16: creat f6 x:0 0 0 2024-04-12T05:36:55.852 INFO:tasks.workunit.client.0.smithi032.stdout:5/22: getdents d2 0 2024-04-12T05:36:55.852 INFO:tasks.workunit.client.0.smithi032.stdout:5/23: readlink - no filename 2024-04-12T05:36:55.852 INFO:tasks.workunit.client.0.smithi032.stdout:2/17: truncate f1 336884 0 2024-04-12T05:36:55.852 INFO:tasks.workunit.client.0.smithi032.stdout:2/18: readlink - no filename 2024-04-12T05:36:55.854 INFO:tasks.workunit.client.0.smithi032.stdout:9/12: rename f4 to f5 0 2024-04-12T05:36:55.856 INFO:tasks.workunit.client.0.smithi032.stdout:3/12: dwrite f2 [0,4194304] 0 2024-04-12T05:36:55.856 INFO:tasks.workunit.client.0.smithi032.stdout:5/24: creat d2/f6 x:0 0 0 2024-04-12T05:36:55.856 INFO:tasks.workunit.client.0.smithi032.stdout:0/7: fsync f1 0 2024-04-12T05:36:55.858 INFO:tasks.workunit.client.0.smithi032.stdout:9/13: symlink l6 0 2024-04-12T05:36:55.858 INFO:tasks.workunit.client.0.smithi032.stdout:2/19: fsync f1 0 2024-04-12T05:36:55.859 INFO:tasks.workunit.client.0.smithi032.stdout:2/20: chown c3 3546 1 2024-04-12T05:36:55.859 INFO:tasks.workunit.client.0.smithi032.stdout:2/21: chown f1 2540 1 2024-04-12T05:36:55.859 INFO:tasks.workunit.client.0.smithi032.stdout:3/13: unlink f2 0 2024-04-12T05:36:55.860 INFO:tasks.workunit.client.0.smithi032.stdout:2/22: dread - f6 zero size 2024-04-12T05:36:55.860 INFO:tasks.workunit.client.0.smithi032.stdout:5/25: mkdir d2/d7 0 2024-04-12T05:36:55.861 INFO:tasks.workunit.client.0.smithi032.stdout:0/8: dread f1 [0,4194304] 0 2024-04-12T05:36:55.861 INFO:tasks.workunit.client.0.smithi032.stdout:2/23: write f5 [473684,33259] 0 2024-04-12T05:36:55.865 INFO:tasks.workunit.client.0.smithi032.stdout:0/9: write f1 [1693801,104496] 0 2024-04-12T05:36:55.865 INFO:tasks.workunit.client.0.smithi032.stdout:5/26: write d2/f6 [939093,115198] 0 2024-04-12T05:36:55.865 INFO:tasks.workunit.client.0.smithi032.stdout:2/24: mknod c7 0 2024-04-12T05:36:55.867 INFO:tasks.workunit.client.0.smithi032.stdout:4/8: getdents . 0 2024-04-12T05:36:55.867 INFO:tasks.workunit.client.0.smithi032.stdout:4/9: dread - no filename 2024-04-12T05:36:55.867 INFO:tasks.workunit.client.0.smithi032.stdout:4/10: write - no filename 2024-04-12T05:36:55.867 INFO:tasks.workunit.client.0.smithi032.stdout:0/10: stat d2 0 2024-04-12T05:36:55.869 INFO:tasks.workunit.client.0.smithi032.stdout:1/6: rename f3 to f4 0 2024-04-12T05:36:55.872 INFO:tasks.workunit.client.0.smithi032.stdout:4/11: creat f3 x:0 0 0 2024-04-12T05:36:55.873 INFO:tasks.workunit.client.0.smithi032.stdout:3/14: dwrite f0 [0,4194304] 0 2024-04-12T05:36:55.875 INFO:tasks.workunit.client.0.smithi032.stdout:4/12: write f3 [611515,77403] 0 2024-04-12T05:36:55.877 INFO:tasks.workunit.client.0.smithi032.stdout:5/27: creat d2/d7/f8 x:0 0 0 2024-04-12T05:36:55.877 INFO:tasks.workunit.client.0.smithi032.stdout:3/15: write f0 [213629,84415] 0 2024-04-12T05:36:55.878 INFO:tasks.workunit.client.0.smithi032.stdout:9/14: getdents . 0 2024-04-12T05:36:55.879 INFO:tasks.workunit.client.0.smithi032.stdout:3/16: stat f3 0 2024-04-12T05:36:55.881 INFO:tasks.workunit.client.0.smithi032.stdout:3/17: truncate f3 480087 0 2024-04-12T05:36:55.882 INFO:tasks.workunit.client.0.smithi032.stdout:2/25: rename f5 to f8 0 2024-04-12T05:36:55.882 INFO:tasks.workunit.client.0.smithi032.stdout:0/11: dwrite f1 [0,4194304] 0 2024-04-12T05:36:55.883 INFO:tasks.workunit.client.0.smithi032.stdout:4/13: mkdir d4 0 2024-04-12T05:36:55.885 INFO:tasks.workunit.client.0.smithi032.stdout:9/15: unlink c3 0 2024-04-12T05:36:55.891 INFO:tasks.workunit.client.0.smithi032.stdout:5/28: dwrite d2/f3 [0,4194304] 0 2024-04-12T05:36:55.892 INFO:tasks.workunit.client.0.smithi032.stdout:2/26: fsync f2 0 2024-04-12T05:36:55.894 INFO:tasks.workunit.client.0.smithi032.stdout:3/18: rename f0 to f4 0 2024-04-12T05:36:55.894 INFO:tasks.workunit.client.0.smithi032.stdout:3/19: chown f3 2064374 1 2024-04-12T05:36:55.895 INFO:tasks.workunit.client.0.smithi032.stdout:9/16: creat f7 x:0 0 0 2024-04-12T05:36:55.895 INFO:tasks.workunit.client.0.smithi032.stdout:9/17: readlink l6 0 2024-04-12T05:36:55.898 INFO:tasks.workunit.client.0.smithi032.stdout:3/20: dread f4 [0,4194304] 0 2024-04-12T05:36:55.902 INFO:tasks.workunit.client.0.smithi032.stdout:2/27: creat f9 x:0 0 0 2024-04-12T05:36:55.907 INFO:tasks.workunit.client.0.smithi032.stdout:4/14: mknod d4/c5 0 2024-04-12T05:36:55.908 INFO:tasks.workunit.client.0.smithi032.stdout:9/18: symlink l8 0 2024-04-12T05:36:55.909 INFO:tasks.workunit.client.0.smithi032.stdout:4/15: write f3 [1203166,61202] 0 2024-04-12T05:36:55.911 INFO:tasks.workunit.client.0.smithi032.stdout:3/21: unlink f4 0 2024-04-12T05:36:55.912 INFO:tasks.workunit.client.0.smithi032.stdout:4/16: write f3 [1158632,92720] 0 2024-04-12T05:36:55.912 INFO:tasks.workunit.client.0.smithi032.stdout:2/28: creat fa x:0 0 0 2024-04-12T05:36:55.912 INFO:tasks.workunit.client.0.smithi032.stdout:3/22: stat f3 0 2024-04-12T05:36:55.912 INFO:tasks.workunit.client.0.smithi032.stdout:1/7: write f4 [667089,95610] 0 2024-04-12T05:36:55.913 INFO:tasks.workunit.client.0.smithi032.stdout:9/19: fdatasync f5 0 2024-04-12T05:36:55.915 INFO:tasks.workunit.client.0.smithi032.stdout:2/29: dread - f0 zero size 2024-04-12T05:36:55.915 INFO:tasks.workunit.client.0.smithi032.stdout:0/12: dwrite f1 [4194304,4194304] 0 2024-04-12T05:36:55.917 INFO:tasks.workunit.client.0.smithi032.stdout:4/17: creat d4/f6 x:0 0 0 2024-04-12T05:36:55.920 INFO:tasks.workunit.client.0.smithi032.stdout:9/20: mkdir d9 0 2024-04-12T05:36:55.920 INFO:tasks.workunit.client.0.smithi032.stdout:5/29: truncate d2/f6 158348 0 2024-04-12T05:36:55.922 INFO:tasks.workunit.client.0.smithi032.stdout:9/21: write f7 [900964,104962] 0 2024-04-12T05:36:55.923 INFO:tasks.workunit.client.0.smithi032.stdout:9/22: readlink l8 0 2024-04-12T05:36:55.924 INFO:tasks.workunit.client.0.smithi032.stdout:3/23: rename f3 to f5 0 2024-04-12T05:36:55.925 INFO:tasks.workunit.client.0.smithi032.stdout:1/8: dwrite f4 [0,4194304] 0 2024-04-12T05:36:55.927 INFO:tasks.workunit.client.0.smithi032.stdout:4/18: symlink d4/l7 0 2024-04-12T05:36:55.934 INFO:tasks.workunit.client.0.smithi032.stdout:9/23: creat d9/fa x:0 0 0 2024-04-12T05:36:55.935 INFO:tasks.workunit.client.0.smithi032.stdout:2/30: write f2 [28018,105644] 0 2024-04-12T05:36:55.939 INFO:tasks.workunit.client.0.smithi032.stdout:0/13: truncate f1 3683928 0 2024-04-12T05:36:55.939 INFO:tasks.workunit.client.0.smithi032.stdout:1/9: fdatasync f4 0 2024-04-12T05:36:55.942 INFO:tasks.workunit.client.0.smithi032.stdout:9/24: dread f5 [0,4194304] 0 2024-04-12T05:36:55.944 INFO:tasks.workunit.client.0.smithi032.stdout:1/10: stat l1 0 2024-04-12T05:36:55.948 INFO:tasks.workunit.client.0.smithi032.stdout:9/25: dread - d9/fa zero size 2024-04-12T05:36:55.948 INFO:tasks.workunit.client.0.smithi032.stdout:3/24: dwrite f5 [0,4194304] 0 2024-04-12T05:36:55.950 INFO:tasks.workunit.client.0.smithi032.stdout:1/11: write f4 [1501505,99997] 0 2024-04-12T05:36:55.951 INFO:tasks.workunit.client.0.smithi032.stdout:4/19: dwrite d4/f6 [0,4194304] 0 2024-04-12T05:36:55.951 INFO:tasks.workunit.client.0.smithi032.stdout:0/14: rename f1 to d2/f4 0 2024-04-12T05:36:55.955 INFO:tasks.workunit.client.0.smithi032.stdout:1/12: chown l1 521998 1 2024-04-12T05:36:55.958 INFO:tasks.workunit.client.0.smithi032.stdout:4/20: write d4/f6 [4608071,24548] 0 2024-04-12T05:36:55.959 INFO:tasks.workunit.client.0.smithi032.stdout:0/15: creat d2/f5 x:0 0 0 2024-04-12T05:36:55.960 INFO:tasks.workunit.client.0.smithi032.stdout:9/26: link l8 d9/lb 0 2024-04-12T05:36:55.961 INFO:tasks.workunit.client.0.smithi032.stdout:2/31: dwrite fa [0,4194304] 0 2024-04-12T05:36:55.962 INFO:tasks.workunit.client.0.smithi032.stdout:5/30: dwrite d2/f6 [0,4194304] 0 2024-04-12T05:36:55.962 INFO:tasks.workunit.client.0.smithi032.stdout:1/13: mkdir d5 0 2024-04-12T05:36:55.962 INFO:tasks.workunit.client.0.smithi032.stdout:3/25: fsync f5 0 2024-04-12T05:36:55.963 INFO:tasks.workunit.client.0.smithi032.stdout:4/21: rmdir d4 39 2024-04-12T05:36:55.964 INFO:tasks.workunit.client.0.smithi032.stdout:0/16: write d2/f5 [757392,71266] 0 2024-04-12T05:36:55.966 INFO:tasks.workunit.client.0.smithi032.stdout:9/27: mkdir d9/dc 0 2024-04-12T05:36:55.969 INFO:tasks.workunit.client.0.smithi032.stdout:3/26: chown f5 58544023 1 2024-04-12T05:36:55.971 INFO:tasks.workunit.client.0.smithi032.stdout:2/32: creat fb x:0 0 0 2024-04-12T05:36:55.971 INFO:tasks.workunit.client.0.smithi032.stdout:5/31: creat d2/f9 x:0 0 0 2024-04-12T05:36:55.975 INFO:tasks.workunit.client.0.smithi032.stdout:5/32: read d2/f6 [133540,107679] 0 2024-04-12T05:36:55.977 INFO:tasks.workunit.client.0.smithi032.stdout:2/33: creat fc x:0 0 0 2024-04-12T05:36:55.977 INFO:tasks.workunit.client.0.smithi032.stdout:0/17: rename d2/f5 to d2/f6 0 2024-04-12T05:36:55.978 INFO:tasks.workunit.client.0.smithi032.stdout:4/22: rename l1 to d4/l8 0 2024-04-12T05:36:55.981 INFO:tasks.workunit.client.0.smithi032.stdout:5/33: rmdir d2/d7 39 2024-04-12T05:36:55.982 INFO:tasks.workunit.client.0.smithi032.stdout:2/34: dread f0 [0,4194304] 0 2024-04-12T05:36:55.983 INFO:tasks.workunit.client.0.smithi032.stdout:1/14: dread f4 [0,4194304] 0 2024-04-12T05:36:55.985 INFO:tasks.workunit.client.0.smithi032.stdout:4/23: mknod d4/c9 0 2024-04-12T05:36:55.985 INFO:tasks.workunit.client.0.smithi032.stdout:3/27: dread f5 [0,4194304] 0 2024-04-12T05:36:55.985 INFO:tasks.workunit.client.0.smithi032.stdout:0/18: dread d2/f6 [0,4194304] 0 2024-04-12T05:36:55.987 INFO:tasks.workunit.client.0.smithi032.stdout:9/28: dwrite d9/fa [0,4194304] 0 2024-04-12T05:36:55.988 INFO:tasks.workunit.client.0.smithi032.stdout:3/28: chown f5 100939133 1 2024-04-12T05:36:55.988 INFO:tasks.workunit.client.0.smithi032.stdout:1/15: creat d5/f6 x:0 0 0 2024-04-12T05:36:55.989 INFO:tasks.workunit.client.0.smithi032.stdout:5/34: mknod d2/d7/ca 0 2024-04-12T05:36:55.990 INFO:tasks.workunit.client.0.smithi032.stdout:0/19: fdatasync d2/f6 0 2024-04-12T05:36:55.994 INFO:tasks.workunit.client.0.smithi032.stdout:5/35: symlink d2/d7/lb 0 2024-04-12T05:36:55.998 INFO:tasks.workunit.client.0.smithi032.stdout:3/29: getdents . 0 2024-04-12T05:36:56.002 INFO:tasks.workunit.client.0.smithi032.stdout:2/35: dwrite f6 [0,4194304] 0 2024-04-12T05:36:56.007 INFO:tasks.workunit.client.0.smithi032.stdout:5/36: dread f1 [0,4194304] 0 2024-04-12T05:36:56.007 INFO:tasks.workunit.client.0.smithi032.stdout:3/30: rename f5 to f6 0 2024-04-12T05:36:56.007 INFO:tasks.workunit.client.0.smithi032.stdout:4/24: dwrite d4/f6 [0,4194304] 0 2024-04-12T05:36:56.007 INFO:tasks.workunit.client.0.smithi032.stdout:2/36: mkdir dd 0 2024-04-12T05:36:56.007 INFO:tasks.workunit.client.0.smithi032.stdout:9/29: dwrite f7 [0,4194304] 0 2024-04-12T05:36:56.011 INFO:tasks.workunit.client.0.smithi032.stdout:4/25: mkdir d4/da 0 2024-04-12T05:36:56.011 INFO:tasks.workunit.client.0.smithi032.stdout:0/20: write d2/f4 [1259347,117688] 0 2024-04-12T05:36:56.011 INFO:tasks.workunit.client.0.smithi032.stdout:2/37: truncate f8 1043715 0 2024-04-12T05:36:56.012 INFO:tasks.workunit.client.0.smithi032.stdout:4/26: fsync f3 0 2024-04-12T05:36:56.014 INFO:tasks.workunit.client.0.smithi032.stdout:3/31: rename f6 to f7 0 2024-04-12T05:36:56.015 INFO:tasks.workunit.client.0.smithi032.stdout:0/21: dread d2/f6 [0,4194304] 0 2024-04-12T05:36:56.015 INFO:tasks.workunit.client.0.smithi032.stdout:3/32: chown f7 25645123 1 2024-04-12T05:36:56.016 INFO:tasks.workunit.client.0.smithi032.stdout:5/37: dread d2/f5 [0,4194304] 0 2024-04-12T05:36:56.017 INFO:tasks.workunit.client.0.smithi032.stdout:9/30: mknod d9/dc/cd 0 2024-04-12T05:36:56.018 INFO:tasks.workunit.client.0.smithi032.stdout:4/27: dread f3 [0,4194304] 0 2024-04-12T05:36:56.020 INFO:tasks.workunit.client.0.smithi032.stdout:0/22: creat d2/f7 x:0 0 0 2024-04-12T05:36:56.021 INFO:tasks.workunit.client.0.smithi032.stdout:3/33: mknod c8 0 2024-04-12T05:36:56.021 INFO:tasks.workunit.client.0.smithi032.stdout:0/23: write d2/f7 [250376,91101] 0 2024-04-12T05:36:56.025 INFO:tasks.workunit.client.0.smithi032.stdout:4/28: rename c2 to d4/cb 0 2024-04-12T05:36:56.027 INFO:tasks.workunit.client.0.smithi032.stdout:9/31: rename d9/lb to d9/le 0 2024-04-12T05:36:56.028 INFO:tasks.workunit.client.0.smithi032.stdout:0/24: mkdir d2/d8 0 2024-04-12T05:36:56.030 INFO:tasks.workunit.client.0.smithi032.stdout:4/29: rename d4/f6 to d4/fc 0 2024-04-12T05:36:56.031 INFO:tasks.workunit.client.0.smithi032.stdout:0/25: fdatasync d2/f7 0 2024-04-12T05:36:56.031 INFO:tasks.workunit.client.0.smithi032.stdout:5/38: getdents d2 0 2024-04-12T05:36:56.032 INFO:tasks.workunit.client.0.smithi032.stdout:5/39: truncate f1 1486305 0 2024-04-12T05:36:56.033 INFO:tasks.workunit.client.0.smithi032.stdout:5/40: chown d2/f5 1621711602 1 2024-04-12T05:36:56.035 INFO:tasks.workunit.client.0.smithi032.stdout:9/32: chown l8 2 1 2024-04-12T05:36:56.036 INFO:tasks.workunit.client.0.smithi032.stdout:0/26: creat d2/f9 x:0 0 0 2024-04-12T05:36:56.036 INFO:tasks.workunit.client.0.smithi032.stdout:0/27: chown d2/f4 222836848 1 2024-04-12T05:36:56.037 INFO:tasks.workunit.client.0.smithi032.stdout:2/38: truncate f8 27408 0 2024-04-12T05:36:56.037 INFO:tasks.workunit.client.0.smithi032.stdout:3/34: getdents . 0 2024-04-12T05:36:56.037 INFO:tasks.workunit.client.0.smithi032.stdout:2/39: chown f4 93103 1 2024-04-12T05:36:56.038 INFO:tasks.workunit.client.0.smithi032.stdout:3/35: truncate f7 4259622 0 2024-04-12T05:36:56.039 INFO:tasks.workunit.client.0.smithi032.stdout:5/41: rmdir d2/d7 39 2024-04-12T05:36:56.041 INFO:tasks.workunit.client.0.smithi032.stdout:9/33: creat d9/dc/ff x:0 0 0 2024-04-12T05:36:56.041 INFO:tasks.workunit.client.0.smithi032.stdout:9/34: fdatasync d9/fa 0 2024-04-12T05:36:56.042 INFO:tasks.workunit.client.0.smithi032.stdout:9/35: dread - d9/dc/ff zero size 2024-04-12T05:36:56.042 INFO:tasks.workunit.client.0.smithi032.stdout:9/36: chown f5 8327802 1 2024-04-12T05:36:56.044 INFO:tasks.workunit.client.0.smithi032.stdout:0/28: creat d2/d8/fa x:0 0 0 2024-04-12T05:36:56.046 INFO:tasks.workunit.client.0.smithi032.stdout:2/40: link fc dd/fe 0 2024-04-12T05:36:56.046 INFO:tasks.workunit.client.0.smithi032.stdout:2/41: readlink - no filename 2024-04-12T05:36:56.046 INFO:tasks.workunit.client.0.smithi032.stdout:9/37: dread d9/fa [0,4194304] 0 2024-04-12T05:36:56.047 INFO:tasks.workunit.client.0.smithi032.stdout:0/29: dread d2/f7 [0,4194304] 0 2024-04-12T05:36:56.048 INFO:tasks.workunit.client.0.smithi032.stdout:0/30: fsync d2/f6 0 2024-04-12T05:36:56.050 INFO:tasks.workunit.client.0.smithi032.stdout:0/31: rename d2/d8 to d2/d8/db 22 2024-04-12T05:36:56.051 INFO:tasks.workunit.client.0.smithi032.stdout:9/38: creat d9/dc/f10 x:0 0 0 2024-04-12T05:36:56.054 INFO:tasks.workunit.client.0.smithi032.stdout:2/42: dwrite f6 [0,4194304] 0 2024-04-12T05:36:56.054 INFO:tasks.workunit.client.0.smithi032.stdout:9/39: creat d9/f11 x:0 0 0 2024-04-12T05:36:56.056 INFO:tasks.workunit.client.0.smithi032.stdout:0/32: mkdir d2/dc 0 2024-04-12T05:36:56.058 INFO:tasks.workunit.client.0.smithi032.stdout:3/36: truncate f7 2432972 0 2024-04-12T05:36:56.060 INFO:tasks.workunit.client.0.smithi032.stdout:5/42: truncate d2/f5 2848885 0 2024-04-12T05:36:56.061 INFO:tasks.workunit.client.0.smithi032.stdout:9/40: chown d9/le 5213 1 2024-04-12T05:36:56.061 INFO:tasks.workunit.client.0.smithi032.stdout:3/37: mknod c9 0 2024-04-12T05:36:56.063 INFO:tasks.workunit.client.0.smithi032.stdout:0/33: creat d2/dc/fd x:0 0 0 2024-04-12T05:36:56.064 INFO:tasks.workunit.client.0.smithi032.stdout:2/43: dwrite fa [0,4194304] 0 2024-04-12T05:36:56.066 INFO:tasks.workunit.client.0.smithi032.stdout:0/34: chown d2/l3 61 1 2024-04-12T05:36:56.067 INFO:tasks.workunit.client.0.smithi032.stdout:9/41: rmdir d9/dc 39 2024-04-12T05:36:56.067 INFO:tasks.workunit.client.0.smithi032.stdout:0/35: stat d2/d8 0 2024-04-12T05:36:56.068 INFO:tasks.workunit.client.0.smithi032.stdout:3/38: unlink c8 0 2024-04-12T05:36:56.068 INFO:tasks.workunit.client.0.smithi032.stdout:0/36: write d2/f6 [161381,115600] 0 2024-04-12T05:36:56.069 INFO:tasks.workunit.client.0.smithi032.stdout:0/37: read d2/f6 [746197,58503] 0 2024-04-12T05:36:56.071 INFO:tasks.workunit.client.0.smithi032.stdout:0/38: dread d2/f7 [0,4194304] 0 2024-04-12T05:36:56.074 INFO:tasks.workunit.client.0.smithi032.stdout:2/44: creat dd/ff x:0 0 0 2024-04-12T05:36:56.079 INFO:tasks.workunit.client.0.smithi032.stdout:0/39: rename d2/f4 to d2/fe 0 2024-04-12T05:36:56.079 INFO:tasks.workunit.client.0.smithi032.stdout:0/40: chown d2/f7 7678 1 2024-04-12T05:36:56.082 INFO:tasks.workunit.client.0.smithi032.stdout:5/43: getdents d2/d7 0 2024-04-12T05:36:56.084 INFO:tasks.workunit.client.0.smithi032.stdout:2/45: rmdir dd 39 2024-04-12T05:36:56.088 INFO:tasks.workunit.client.0.smithi032.stdout:2/46: dread - dd/ff zero size 2024-04-12T05:36:56.089 INFO:tasks.workunit.client.0.smithi032.stdout:9/42: getdents d9/dc 0 2024-04-12T05:36:56.090 INFO:tasks.workunit.client.0.smithi032.stdout:9/43: chown d9/dc/cd 58183167 1 2024-04-12T05:36:56.091 INFO:tasks.workunit.client.0.smithi032.stdout:3/39: truncate f7 756176 0 2024-04-12T05:36:56.091 INFO:tasks.workunit.client.0.smithi032.stdout:0/41: getdents d2/d8 0 2024-04-12T05:36:56.093 INFO:tasks.workunit.client.0.smithi032.stdout:5/44: dwrite d2/f3 [0,4194304] 0 2024-04-12T05:36:56.094 INFO:tasks.workunit.client.0.smithi032.stdout:0/42: mkdir d2/d8/df 0 2024-04-12T05:36:56.095 INFO:tasks.workunit.client.0.smithi032.stdout:0/43: chown d2/d8/fa 199 1 2024-04-12T05:36:56.096 INFO:tasks.workunit.client.0.smithi032.stdout:9/44: truncate d9/fa 328869 0 2024-04-12T05:36:56.097 INFO:tasks.workunit.client.0.smithi032.stdout:3/40: link c9 ca 0 2024-04-12T05:36:56.097 INFO:tasks.workunit.client.0.smithi032.stdout:9/45: fdatasync d9/dc/ff 0 2024-04-12T05:36:56.102 INFO:tasks.workunit.client.0.smithi032.stdout:3/41: creat fb x:0 0 0 2024-04-12T05:36:56.103 INFO:tasks.workunit.client.0.smithi032.stdout:0/44: rmdir d2 39 2024-04-12T05:36:56.103 INFO:tasks.workunit.client.0.smithi032.stdout:5/45: getdents d2/d7 0 2024-04-12T05:36:56.105 INFO:tasks.workunit.client.0.smithi032.stdout:3/42: symlink lc 0 2024-04-12T05:36:56.108 INFO:tasks.workunit.client.0.smithi032.stdout:5/46: symlink d2/d7/lc 0 2024-04-12T05:36:56.109 INFO:tasks.workunit.client.0.smithi032.stdout:9/46: mknod d9/c12 0 2024-04-12T05:36:56.110 INFO:tasks.workunit.client.0.smithi032.stdout:0/45: mknod d2/c10 0 2024-04-12T05:36:56.111 INFO:tasks.workunit.client.0.smithi032.stdout:5/47: fdatasync d2/f3 0 2024-04-12T05:36:56.112 INFO:tasks.workunit.client.0.smithi032.stdout:0/46: mknod d2/d8/c11 0 2024-04-12T05:36:56.114 INFO:tasks.workunit.client.0.smithi032.stdout:0/47: mknod d2/c12 0 2024-04-12T05:36:56.115 INFO:tasks.workunit.client.0.smithi032.stdout:5/48: write d2/f6 [1454162,95146] 0 2024-04-12T05:36:56.117 INFO:tasks.workunit.client.0.smithi032.stdout:9/47: link d9/dc/f10 d9/f13 0 2024-04-12T05:36:56.118 INFO:tasks.workunit.client.0.smithi032.stdout:3/43: getdents . 0 2024-04-12T05:36:56.121 INFO:tasks.workunit.client.0.smithi032.stdout:3/44: creat fd x:0 0 0 2024-04-12T05:36:56.123 INFO:tasks.workunit.client.0.smithi032.stdout:0/48: dwrite d2/dc/fd [0,4194304] 0 2024-04-12T05:36:56.123 INFO:tasks.workunit.client.0.smithi032.stdout:9/48: truncate d9/fa 490653 0 2024-04-12T05:36:56.125 INFO:tasks.workunit.client.0.smithi032.stdout:5/49: write d2/f5 [3130499,33515] 0 2024-04-12T05:36:56.126 INFO:tasks.workunit.client.0.smithi032.stdout:9/49: creat d9/dc/f14 x:0 0 0 2024-04-12T05:36:56.128 INFO:tasks.workunit.client.0.smithi032.stdout:0/49: symlink d2/d8/df/l13 0 2024-04-12T05:36:56.131 INFO:tasks.workunit.client.0.smithi032.stdout:3/45: dwrite fd [0,4194304] 0 2024-04-12T05:36:56.131 INFO:tasks.workunit.client.0.smithi032.stdout:5/50: mkdir d2/dd 0 2024-04-12T05:36:56.134 INFO:tasks.workunit.client.0.smithi032.stdout:9/50: mkdir d9/d15 0 2024-04-12T05:36:56.134 INFO:tasks.workunit.client.0.smithi032.stdout:9/51: readlink l8 0 2024-04-12T05:36:56.134 INFO:tasks.workunit.client.0.smithi032.stdout:5/51: mknod d2/d7/ce 0 2024-04-12T05:36:56.136 INFO:tasks.workunit.client.0.smithi032.stdout:3/46: fdatasync f7 0 2024-04-12T05:36:56.137 INFO:tasks.workunit.client.0.smithi032.stdout:3/47: read f7 [651218,56877] 0 2024-04-12T05:36:56.137 INFO:tasks.workunit.client.0.smithi032.stdout:9/52: creat d9/dc/f16 x:0 0 0 2024-04-12T05:36:56.140 INFO:tasks.workunit.client.0.smithi032.stdout:3/48: mknod ce 0 2024-04-12T05:36:56.141 INFO:tasks.workunit.client.0.smithi032.stdout:5/52: mkdir d2/df 0 2024-04-12T05:36:56.141 INFO:tasks.workunit.client.0.smithi032.stdout:0/50: fdatasync d2/dc/fd 0 2024-04-12T05:36:56.142 INFO:tasks.workunit.client.0.smithi032.stdout:9/53: rename d9/dc/cd to d9/dc/c17 0 2024-04-12T05:36:56.143 INFO:tasks.workunit.client.0.smithi032.stdout:3/49: symlink lf 0 2024-04-12T05:36:56.144 INFO:tasks.workunit.client.0.smithi032.stdout:3/50: write fd [3971849,99778] 0 2024-04-12T05:36:56.145 INFO:tasks.workunit.client.0.smithi032.stdout:3/51: chown lf 0 1 2024-04-12T05:36:56.148 INFO:tasks.workunit.client.0.smithi032.stdout:5/53: mknod d2/df/c10 0 2024-04-12T05:36:56.148 INFO:tasks.workunit.client.0.smithi032.stdout:5/54: stat d2 0 2024-04-12T05:36:56.149 INFO:tasks.workunit.client.0.smithi032.stdout:5/55: mknod d2/c11 0 2024-04-12T05:36:56.151 INFO:tasks.workunit.client.0.smithi032.stdout:5/56: symlink d2/d7/l12 0 2024-04-12T05:36:56.153 INFO:tasks.workunit.client.0.smithi032.stdout:5/57: dread - d2/f4 zero size 2024-04-12T05:36:56.160 INFO:tasks.workunit.client.0.smithi032.stdout:5/58: link d2/c11 d2/c13 0 2024-04-12T05:36:56.160 INFO:tasks.workunit.client.0.smithi032.stdout:0/51: dwrite d2/f6 [0,4194304] 0 2024-04-12T05:36:56.161 INFO:tasks.workunit.client.0.smithi032.stdout:9/54: dwrite d9/dc/f10 [0,4194304] 0 2024-04-12T05:36:56.162 INFO:tasks.workunit.client.0.smithi032.stdout:5/59: write d2/f3 [1998411,64617] 0 2024-04-12T05:36:56.163 INFO:tasks.workunit.client.0.smithi032.stdout:0/52: creat d2/f14 x:0 0 0 2024-04-12T05:36:56.165 INFO:tasks.workunit.client.0.smithi032.stdout:0/53: truncate d2/f7 682273 0 2024-04-12T05:36:56.165 INFO:tasks.workunit.client.0.smithi032.stdout:9/55: mknod d9/c18 0 2024-04-12T05:36:56.165 INFO:tasks.workunit.client.0.smithi032.stdout:5/60: write d2/f4 [239952,108964] 0 2024-04-12T05:36:56.167 INFO:tasks.workunit.client.0.smithi032.stdout:9/56: write d9/dc/ff [369228,80708] 0 2024-04-12T05:36:56.168 INFO:tasks.workunit.client.0.smithi032.stdout:9/57: dread - d9/dc/f16 zero size 2024-04-12T05:36:56.173 INFO:tasks.workunit.client.0.smithi032.stdout:0/54: dread d2/f6 [0,4194304] 0 2024-04-12T05:36:56.176 INFO:tasks.workunit.client.0.smithi032.stdout:9/58: dwrite f7 [0,4194304] 0 2024-04-12T05:36:56.177 INFO:tasks.workunit.client.0.smithi032.stdout:9/59: chown f7 4919 1 2024-04-12T05:36:56.178 INFO:tasks.workunit.client.0.smithi032.stdout:9/60: write d9/dc/f10 [2620354,77238] 0 2024-04-12T05:36:56.181 INFO:tasks.workunit.client.0.smithi032.stdout:9/61: mknod d9/d15/c19 0 2024-04-12T05:36:56.182 INFO:tasks.workunit.client.0.smithi032.stdout:0/55: dwrite d2/dc/fd [0,4194304] 0 2024-04-12T05:36:56.183 INFO:tasks.workunit.client.0.smithi032.stdout:0/56: rename d2 to d2/dc/d15 22 2024-04-12T05:36:56.184 INFO:tasks.workunit.client.0.smithi032.stdout:9/62: mkdir d9/d15/d1a 0 2024-04-12T05:36:56.184 INFO:tasks.workunit.client.0.smithi032.stdout:9/63: dread - d9/f11 zero size 2024-04-12T05:36:56.203 INFO:tasks.workunit.client.0.smithi032.stdout:0/57: truncate d2/dc/fd 2169090 0 2024-04-12T05:36:56.204 INFO:tasks.workunit.client.0.smithi032.stdout:0/58: creat d2/f16 x:0 0 0 2024-04-12T05:36:56.205 INFO:tasks.workunit.client.0.smithi032.stdout:0/59: fdatasync d2/f7 0 2024-04-12T05:36:56.208 INFO:tasks.workunit.client.0.smithi032.stdout:9/64: dwrite d9/fa [0,4194304] 0 2024-04-12T05:36:56.209 INFO:tasks.workunit.client.0.smithi032.stdout:0/60: unlink d2/f9 0 2024-04-12T05:36:56.210 INFO:tasks.workunit.client.0.smithi032.stdout:9/65: read d9/f13 [180752,98788] 0 2024-04-12T05:36:56.215 INFO:tasks.workunit.client.0.smithi032.stdout:9/66: rename d9/c12 to d9/d15/d1a/c1b 0 2024-04-12T05:36:56.218 INFO:tasks.workunit.client.0.smithi032.stdout:0/61: dwrite d2/dc/fd [0,4194304] 0 2024-04-12T05:36:56.218 INFO:tasks.workunit.client.0.smithi032.stdout:9/67: chown d9/dc/c17 8097 1 2024-04-12T05:36:56.218 INFO:tasks.workunit.client.0.smithi032.stdout:9/68: stat d9/dc/f10 0 2024-04-12T05:36:56.219 INFO:tasks.workunit.client.0.smithi032.stdout:9/69: rename d9 to d9/d15/d1a/d1c 22 2024-04-12T05:36:56.220 INFO:tasks.workunit.client.0.smithi032.stdout:0/62: mkdir d2/d8/d17 0 2024-04-12T05:36:56.220 INFO:tasks.workunit.client.0.smithi032.stdout:9/70: truncate d9/dc/ff 571615 0 2024-04-12T05:36:56.222 INFO:tasks.workunit.client.0.smithi032.stdout:0/63: creat d2/dc/f18 x:0 0 0 2024-04-12T05:36:56.222 INFO:tasks.workunit.client.0.smithi032.stdout:0/64: stat d2/d8/df 0 2024-04-12T05:36:56.225 INFO:tasks.workunit.client.0.smithi032.stdout:0/65: mknod d2/dc/c19 0 2024-04-12T05:36:56.226 INFO:tasks.workunit.client.0.smithi032.stdout:9/71: dread d9/dc/ff [0,4194304] 0 2024-04-12T05:36:56.227 INFO:tasks.workunit.client.0.smithi032.stdout:0/66: mknod d2/d8/c1a 0 2024-04-12T05:36:56.228 INFO:tasks.workunit.client.0.smithi032.stdout:0/67: dread - d2/f14 zero size 2024-04-12T05:36:56.228 INFO:tasks.workunit.client.0.smithi032.stdout:9/72: chown d9/dc/f10 3191470 1 2024-04-12T05:36:56.228 INFO:tasks.workunit.client.0.smithi032.stdout:9/73: dread - d9/dc/f16 zero size 2024-04-12T05:36:56.237 INFO:tasks.workunit.client.0.smithi032.stdout:0/68: dwrite d2/f14 [0,4194304] 0 2024-04-12T05:36:56.239 INFO:tasks.workunit.client.0.smithi032.stdout:9/74: dwrite d9/dc/f16 [0,4194304] 0 2024-04-12T05:36:56.241 INFO:tasks.workunit.client.0.smithi032.stdout:0/69: mkdir d2/d8/d17/d1b 0 2024-04-12T05:36:56.242 INFO:tasks.workunit.client.0.smithi032.stdout:9/75: truncate d9/fa 5063772 0 2024-04-12T05:36:56.245 INFO:tasks.workunit.client.0.smithi032.stdout:9/76: mkdir d9/d15/d1a/d1d 0 2024-04-12T05:36:56.247 INFO:tasks.workunit.client.0.smithi032.stdout:9/77: read d9/dc/f16 [1259222,87374] 0 2024-04-12T05:36:56.247 INFO:tasks.workunit.client.0.smithi032.stdout:9/78: chown d9/fa 757018 1 2024-04-12T05:36:56.249 INFO:tasks.workunit.client.0.smithi032.stdout:0/70: link d2/f7 d2/d8/d17/f1c 0 2024-04-12T05:36:56.249 INFO:tasks.workunit.client.0.smithi032.stdout:0/71: truncate d2/f16 521603 0 2024-04-12T05:36:56.250 INFO:tasks.workunit.client.0.smithi032.stdout:9/79: mkdir d9/d15/d1a/d1e 0 2024-04-12T05:36:56.250 INFO:tasks.workunit.client.0.smithi032.stdout:0/72: chown d2/fe 758 1 2024-04-12T05:36:56.251 INFO:tasks.workunit.client.0.smithi032.stdout:0/73: fsync d2/f6 0 2024-04-12T05:36:56.252 INFO:tasks.workunit.client.0.smithi032.stdout:9/80: mkdir d9/d1f 0 2024-04-12T05:36:56.253 INFO:tasks.workunit.client.0.smithi032.stdout:0/74: mknod d2/d8/c1d 0 2024-04-12T05:36:56.259 INFO:tasks.workunit.client.0.smithi032.stdout:0/75: dwrite d2/dc/fd [0,4194304] 0 2024-04-12T05:36:56.260 INFO:tasks.workunit.client.0.smithi032.stdout:0/76: chown d2/f16 346518206 1 2024-04-12T05:36:56.261 INFO:tasks.workunit.client.0.smithi032.stdout:0/77: truncate d2/d8/fa 766697 0 2024-04-12T05:36:56.267 INFO:tasks.workunit.client.0.smithi032.stdout:0/78: dwrite d2/f14 [0,4194304] 0 2024-04-12T05:36:56.268 INFO:tasks.workunit.client.0.smithi032.stdout:0/79: write d2/f6 [2443456,38256] 0 2024-04-12T05:36:56.272 INFO:tasks.workunit.client.0.smithi032.stdout:0/80: mknod d2/d8/d17/d1b/c1e 0 2024-04-12T05:36:56.274 INFO:tasks.workunit.client.0.smithi032.stdout:0/81: mknod d2/c1f 0 2024-04-12T05:36:56.275 INFO:tasks.workunit.client.0.smithi032.stdout:0/82: fsync d2/fe 0 2024-04-12T05:36:56.277 INFO:tasks.workunit.client.0.smithi032.stdout:0/83: mkdir d2/d8/d17/d1b/d20 0 2024-04-12T05:36:56.283 INFO:tasks.workunit.client.0.smithi032.stdout:0/84: dwrite d2/fe [0,4194304] 0 2024-04-12T05:36:56.286 INFO:tasks.workunit.client.0.smithi032.stdout:0/85: mknod d2/dc/c21 0 2024-04-12T05:36:56.288 INFO:tasks.workunit.client.0.smithi032.stdout:0/86: creat d2/d8/d17/f22 x:0 0 0 2024-04-12T05:36:56.289 INFO:tasks.workunit.client.0.smithi032.stdout:0/87: read d2/f7 [125938,27113] 0 2024-04-12T05:36:56.291 INFO:tasks.workunit.client.0.smithi032.stdout:0/88: creat d2/dc/f23 x:0 0 0 2024-04-12T05:36:56.293 INFO:tasks.workunit.client.0.smithi032.stdout:0/89: mkdir d2/d8/df/d24 0 2024-04-12T05:36:56.294 INFO:tasks.workunit.client.0.smithi032.stdout:0/90: write d2/f16 [109661,96037] 0 2024-04-12T05:36:56.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:56 smithi067 ceph-mon[49010]: pgmap v259: 65 pgs: 65 active+clean; 291 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 14 MiB/s wr, 296 op/s 2024-04-12T05:36:56.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:56 smithi032 ceph-mon[39731]: pgmap v259: 65 pgs: 65 active+clean; 291 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 14 MiB/s wr, 296 op/s 2024-04-12T05:36:58.516 INFO:tasks.workunit.client.0.smithi032.stdout:6/2: sync 2024-04-12T05:36:58.517 INFO:tasks.workunit.client.0.smithi032.stdout:7/9: sync 2024-04-12T05:36:58.517 INFO:tasks.workunit.client.0.smithi032.stdout:6/3: rename - no filename 2024-04-12T05:36:58.517 INFO:tasks.workunit.client.0.smithi032.stdout:8/0: sync 2024-04-12T05:36:58.517 INFO:tasks.workunit.client.0.smithi032.stdout:2/47: sync 2024-04-12T05:36:58.517 INFO:tasks.workunit.client.0.smithi032.stdout:2/48: readlink - no filename 2024-04-12T05:36:58.517 INFO:tasks.workunit.client.0.smithi032.stdout:1/16: sync 2024-04-12T05:36:58.517 INFO:tasks.workunit.client.0.smithi032.stdout:0/91: sync 2024-04-12T05:36:58.517 INFO:tasks.workunit.client.0.smithi032.stdout:4/30: sync 2024-04-12T05:36:58.517 INFO:tasks.workunit.client.0.smithi032.stdout:9/81: sync 2024-04-12T05:36:58.517 INFO:tasks.workunit.client.0.smithi032.stdout:3/52: sync 2024-04-12T05:36:58.517 INFO:tasks.workunit.client.0.smithi032.stdout:5/61: sync 2024-04-12T05:36:58.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:36:58 smithi067 ceph-mon[49010]: pgmap v260: 65 pgs: 65 active+clean; 339 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 19 MiB/s wr, 379 op/s 2024-04-12T05:36:58.525 INFO:tasks.workunit.client.0.smithi032.stdout:3/53: dwrite fb [0,4194304] 0 2024-04-12T05:36:58.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:36:58 smithi032 ceph-mon[39731]: pgmap v260: 65 pgs: 65 active+clean; 339 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 19 MiB/s wr, 379 op/s 2024-04-12T05:36:58.607 INFO:tasks.workunit.client.0.smithi032.stdout:7/10: mknod d0/c2 0 2024-04-12T05:36:58.607 INFO:tasks.workunit.client.0.smithi032.stdout:7/11: fdatasync - no filename 2024-04-12T05:36:58.607 INFO:tasks.workunit.client.0.smithi032.stdout:7/12: chown d0/l1 45962 1 2024-04-12T05:36:58.607 INFO:tasks.workunit.client.0.smithi032.stdout:7/13: dread - no filename 2024-04-12T05:36:58.608 INFO:tasks.workunit.client.0.smithi032.stdout:0/92: sync 2024-04-12T05:36:58.608 INFO:tasks.workunit.client.0.smithi032.stdout:0/93: write d2/d8/d17/f22 [872638,99442] 0 2024-04-12T05:36:58.611 INFO:tasks.workunit.client.0.smithi032.stdout:9/82: fsync d9/dc/f10 0 2024-04-12T05:36:58.612 INFO:tasks.workunit.client.0.smithi032.stdout:1/17: symlink d5/l7 0 2024-04-12T05:36:58.613 INFO:tasks.workunit.client.0.smithi032.stdout:8/1: mkdir d0 0 2024-04-12T05:36:58.613 INFO:tasks.workunit.client.0.smithi032.stdout:6/4: symlink l0 0 2024-04-12T05:36:58.614 INFO:tasks.workunit.client.0.smithi032.stdout:9/83: chown d9/dc/f16 0 1 2024-04-12T05:36:58.614 INFO:tasks.workunit.client.0.smithi032.stdout:4/31: creat d4/da/fd x:0 0 0 2024-04-12T05:36:58.615 INFO:tasks.workunit.client.0.smithi032.stdout:1/18: write d5/f6 [110056,126204] 0 2024-04-12T05:36:58.617 INFO:tasks.workunit.client.0.smithi032.stdout:5/62: creat d2/dd/f14 x:0 0 0 2024-04-12T05:36:58.617 INFO:tasks.workunit.client.0.smithi032.stdout:1/19: read d5/f6 [200658,99340] 0 2024-04-12T05:36:58.618 INFO:tasks.workunit.client.0.smithi032.stdout:5/63: chown d2/f3 10 1 2024-04-12T05:36:58.618 INFO:tasks.workunit.client.0.smithi032.stdout:5/64: chown d2/dd 1208884 1 2024-04-12T05:36:58.618 INFO:tasks.workunit.client.0.smithi032.stdout:6/5: rename l0 to l1 0 2024-04-12T05:36:58.618 INFO:tasks.workunit.client.0.smithi032.stdout:6/6: dread - no filename 2024-04-12T05:36:58.619 INFO:tasks.workunit.client.0.smithi032.stdout:4/32: symlink d4/le 0 2024-04-12T05:36:58.620 INFO:tasks.workunit.client.0.smithi032.stdout:4/33: write f3 [708669,35868] 0 2024-04-12T05:36:58.623 INFO:tasks.workunit.client.0.smithi032.stdout:9/84: creat d9/dc/f20 x:0 0 0 2024-04-12T05:36:58.624 INFO:tasks.workunit.client.0.smithi032.stdout:8/2: mknod d0/c1 0 2024-04-12T05:36:58.629 INFO:tasks.workunit.client.0.smithi032.stdout:8/3: creat d0/f2 x:0 0 0 2024-04-12T05:36:58.631 INFO:tasks.workunit.client.0.smithi032.stdout:9/85: rename d9/f11 to d9/d15/d1a/f21 0 2024-04-12T05:36:58.631 INFO:tasks.workunit.client.0.smithi032.stdout:5/65: dwrite d2/f3 [0,4194304] 0 2024-04-12T05:36:58.631 INFO:tasks.workunit.client.0.smithi032.stdout:1/20: link d5/l7 d5/l8 0 2024-04-12T05:36:58.633 INFO:tasks.workunit.client.0.smithi032.stdout:8/4: creat d0/f3 x:0 0 0 2024-04-12T05:36:58.634 INFO:tasks.workunit.client.0.smithi032.stdout:4/34: dwrite f3 [0,4194304] 0 2024-04-12T05:36:58.635 INFO:tasks.workunit.client.0.smithi032.stdout:8/5: read - d0/f2 zero size 2024-04-12T05:36:58.636 INFO:tasks.workunit.client.0.smithi032.stdout:8/6: fdatasync d0/f2 0 2024-04-12T05:36:58.636 INFO:tasks.workunit.client.0.smithi032.stdout:9/86: write d9/dc/f16 [1316524,52214] 0 2024-04-12T05:36:58.641 INFO:tasks.workunit.client.0.smithi032.stdout:1/21: link d5/l7 d5/l9 0 2024-04-12T05:36:58.644 INFO:tasks.workunit.client.0.smithi032.stdout:5/66: link d2/f9 d2/d7/f15 0 2024-04-12T05:36:58.645 INFO:tasks.workunit.client.0.smithi032.stdout:1/22: read d5/f6 [158462,4711] 0 2024-04-12T05:36:58.645 INFO:tasks.workunit.client.0.smithi032.stdout:5/67: stat d2/d7/ca 0 2024-04-12T05:36:58.646 INFO:tasks.workunit.client.0.smithi032.stdout:5/68: dread - d2/f9 zero size 2024-04-12T05:36:58.646 INFO:tasks.workunit.client.0.smithi032.stdout:9/87: dwrite d9/fa [0,4194304] 0 2024-04-12T05:36:58.667 INFO:tasks.workunit.client.0.smithi032.stdout:5/69: rmdir d2 39 2024-04-12T05:36:58.670 INFO:tasks.workunit.client.0.smithi032.stdout:1/23: dwrite d5/f6 [0,4194304] 0 2024-04-12T05:36:58.674 INFO:tasks.workunit.client.0.smithi032.stdout:9/88: creat d9/f22 x:0 0 0 2024-04-12T05:36:58.675 INFO:tasks.workunit.client.0.smithi032.stdout:9/89: chown d9/d15/d1a 4552885 1 2024-04-12T05:36:58.679 INFO:tasks.workunit.client.0.smithi032.stdout:9/90: dread d9/dc/f10 [0,4194304] 0 2024-04-12T05:36:58.681 INFO:tasks.workunit.client.0.smithi032.stdout:5/70: link d2/d7/f15 d2/d7/f16 0 2024-04-12T05:36:58.686 INFO:tasks.workunit.client.0.smithi032.stdout:9/91: rmdir d9/d15/d1a/d1e 0 2024-04-12T05:36:58.688 INFO:tasks.workunit.client.0.smithi032.stdout:5/71: dwrite d2/f5 [0,4194304] 0 2024-04-12T05:36:58.688 INFO:tasks.workunit.client.0.smithi032.stdout:9/92: chown d9/d15/c19 4 1 2024-04-12T05:36:58.689 INFO:tasks.workunit.client.0.smithi032.stdout:5/72: read d2/f4 [271328,86430] 0 2024-04-12T05:36:58.691 INFO:tasks.workunit.client.0.smithi032.stdout:5/73: write d2/dd/f14 [773091,115794] 0 2024-04-12T05:36:58.692 INFO:tasks.workunit.client.0.smithi032.stdout:5/74: chown d2/f6 88958041 1 2024-04-12T05:36:58.698 INFO:tasks.workunit.client.0.smithi032.stdout:0/94: sync 2024-04-12T05:36:58.700 INFO:tasks.workunit.client.0.smithi032.stdout:9/93: link f7 d9/dc/f23 0 2024-04-12T05:36:58.701 INFO:tasks.workunit.client.0.smithi032.stdout:0/95: fsync d2/d8/d17/f1c 0 2024-04-12T05:36:58.703 INFO:tasks.workunit.client.0.smithi032.stdout:5/75: mkdir d2/d17 0 2024-04-12T05:36:58.708 INFO:tasks.workunit.client.0.smithi032.stdout:9/94: dwrite d9/f22 [0,4194304] 0 2024-04-12T05:36:58.709 INFO:tasks.workunit.client.0.smithi032.stdout:0/96: symlink d2/d8/d17/d1b/d20/l25 0 2024-04-12T05:36:58.711 INFO:tasks.workunit.client.0.smithi032.stdout:9/95: dread f5 [0,4194304] 0 2024-04-12T05:36:58.714 INFO:tasks.workunit.client.0.smithi032.stdout:0/97: write d2/dc/f18 [429621,37770] 0 2024-04-12T05:36:58.715 INFO:tasks.workunit.client.0.smithi032.stdout:9/96: unlink d9/c18 0 2024-04-12T05:36:58.718 INFO:tasks.workunit.client.0.smithi032.stdout:5/76: dwrite d2/f4 [0,4194304] 0 2024-04-12T05:36:58.719 INFO:tasks.workunit.client.0.smithi032.stdout:5/77: fdatasync d2/f6 0 2024-04-12T05:36:58.720 INFO:tasks.workunit.client.0.smithi032.stdout:0/98: symlink d2/d8/d17/l26 0 2024-04-12T05:36:58.723 INFO:tasks.workunit.client.0.smithi032.stdout:5/78: rename d2/d7/lb to d2/d7/l18 0 2024-04-12T05:36:58.723 INFO:tasks.workunit.client.0.smithi032.stdout:0/99: dread d2/f7 [0,4194304] 0 2024-04-12T05:36:58.723 INFO:tasks.workunit.client.0.smithi032.stdout:9/97: dread d9/dc/f23 [0,4194304] 0 2024-04-12T05:36:58.724 INFO:tasks.workunit.client.0.smithi032.stdout:0/100: fsync d2/dc/f23 0 2024-04-12T05:36:58.724 INFO:tasks.workunit.client.0.smithi032.stdout:9/98: dread f5 [0,4194304] 0 2024-04-12T05:36:58.728 INFO:tasks.workunit.client.0.smithi032.stdout:0/101: symlink d2/d8/d17/d1b/l27 0 2024-04-12T05:36:58.729 INFO:tasks.workunit.client.0.smithi032.stdout:9/99: mkdir d9/d15/d1a/d24 0 2024-04-12T05:36:58.730 INFO:tasks.workunit.client.0.smithi032.stdout:0/102: dread d2/d8/fa [0,4194304] 0 2024-04-12T05:36:58.731 INFO:tasks.workunit.client.0.smithi032.stdout:5/79: rename d2/d7/f15 to d2/f19 0 2024-04-12T05:36:58.733 INFO:tasks.workunit.client.0.smithi032.stdout:5/80: write d2/f3 [5151449,96939] 0 2024-04-12T05:36:58.734 INFO:tasks.workunit.client.0.smithi032.stdout:0/103: dread d2/f7 [0,4194304] 0 2024-04-12T05:36:58.735 INFO:tasks.workunit.client.0.smithi032.stdout:5/81: write d2/f3 [1435413,110665] 0 2024-04-12T05:36:58.736 INFO:tasks.workunit.client.0.smithi032.stdout:9/100: dread d9/dc/f10 [0,4194304] 0 2024-04-12T05:36:58.738 INFO:tasks.workunit.client.0.smithi032.stdout:5/82: mkdir d2/dd/d1a 0 2024-04-12T05:36:58.741 INFO:tasks.workunit.client.0.smithi032.stdout:8/7: sync 2024-04-12T05:36:58.741 INFO:tasks.workunit.client.0.smithi032.stdout:4/35: sync 2024-04-12T05:36:58.742 INFO:tasks.workunit.client.0.smithi032.stdout:0/104: dwrite d2/fe [0,4194304] 0 2024-04-12T05:36:58.743 INFO:tasks.workunit.client.0.smithi032.stdout:4/36: fsync d4/fc 0 2024-04-12T05:36:58.744 INFO:tasks.workunit.client.0.smithi032.stdout:8/8: mknod d0/c4 0 2024-04-12T05:36:58.744 INFO:tasks.workunit.client.0.smithi032.stdout:8/9: stat d0/c1 0 2024-04-12T05:36:58.745 INFO:tasks.workunit.client.0.smithi032.stdout:4/37: creat d4/da/ff x:0 0 0 2024-04-12T05:36:58.746 INFO:tasks.workunit.client.0.smithi032.stdout:4/38: stat d4 0 2024-04-12T05:36:58.749 INFO:tasks.workunit.client.0.smithi032.stdout:8/10: creat d0/f5 x:0 0 0 2024-04-12T05:36:58.749 INFO:tasks.workunit.client.0.smithi032.stdout:8/11: stat d0/c4 0 2024-04-12T05:36:58.750 INFO:tasks.workunit.client.0.smithi032.stdout:8/12: write d0/f2 [201045,21646] 0 2024-04-12T05:36:58.756 INFO:tasks.workunit.client.0.smithi032.stdout:0/105: fdatasync d2/fe 0 2024-04-12T05:36:58.758 INFO:tasks.workunit.client.0.smithi032.stdout:0/106: mkdir d2/d28 0 2024-04-12T05:36:58.763 INFO:tasks.workunit.client.0.smithi032.stdout:0/107: dwrite d2/f14 [0,4194304] 0 2024-04-12T05:36:58.766 INFO:tasks.workunit.client.0.smithi032.stdout:0/108: rmdir d2/dc 39 2024-04-12T05:36:58.768 INFO:tasks.workunit.client.0.smithi032.stdout:0/109: stat d2/dc/c19 0 2024-04-12T05:36:58.771 INFO:tasks.workunit.client.0.smithi032.stdout:0/110: mkdir d2/d29 0 2024-04-12T05:36:58.772 INFO:tasks.workunit.client.0.smithi032.stdout:0/111: write d2/d8/d17/f22 [644656,33232] 0 2024-04-12T05:36:58.779 INFO:tasks.workunit.client.0.smithi032.stdout:0/112: dwrite d2/d8/d17/f22 [0,4194304] 0 2024-04-12T05:36:58.780 INFO:tasks.workunit.client.0.smithi032.stdout:0/113: write d2/dc/f18 [417406,122464] 0 2024-04-12T05:36:58.781 INFO:tasks.workunit.client.0.smithi032.stdout:0/114: write d2/d8/d17/f22 [448006,93230] 0 2024-04-12T05:36:58.783 INFO:tasks.workunit.client.0.smithi032.stdout:0/115: rmdir d2/d8/df 39 2024-04-12T05:36:58.784 INFO:tasks.workunit.client.0.smithi032.stdout:0/116: chown d2/d8/d17 5 1 2024-04-12T05:36:58.784 INFO:tasks.workunit.client.0.smithi032.stdout:0/117: fsync d2/dc/f18 0 2024-04-12T05:36:58.786 INFO:tasks.workunit.client.0.smithi032.stdout:0/118: dread d2/f7 [0,4194304] 0 2024-04-12T05:36:58.786 INFO:tasks.workunit.client.0.smithi032.stdout:0/119: stat d2/fe 0 2024-04-12T05:36:58.791 INFO:tasks.workunit.client.0.smithi032.stdout:0/120: rename d2/d8/d17/d1b/d20/l25 to d2/d29/l2a 0 2024-04-12T05:36:58.794 INFO:tasks.workunit.client.0.smithi032.stdout:0/121: rename d2/dc/f23 to d2/d8/d17/f2b 0 2024-04-12T05:36:58.921 INFO:tasks.workunit.client.0.smithi032.stdout:2/49: write dd/fe [248505,22610] 0 2024-04-12T05:36:58.987 INFO:tasks.workunit.client.0.smithi032.stdout:7/14: getdents d0 0 2024-04-12T05:36:58.992 INFO:tasks.workunit.client.0.smithi032.stdout:2/50: truncate f4 365119 0 2024-04-12T05:36:58.992 INFO:tasks.workunit.client.0.smithi032.stdout:7/15: rmdir d0 39 2024-04-12T05:36:58.993 INFO:tasks.workunit.client.0.smithi032.stdout:3/54: write f7 [1661947,67206] 0 2024-04-12T05:36:58.993 INFO:tasks.workunit.client.0.smithi032.stdout:2/51: symlink dd/l10 0 2024-04-12T05:36:58.993 INFO:tasks.workunit.client.0.smithi032.stdout:2/52: write f1 [1161634,67264] 0 2024-04-12T05:36:58.993 INFO:tasks.workunit.client.0.smithi032.stdout:3/55: unlink fb 0 2024-04-12T05:36:58.993 INFO:tasks.workunit.client.0.smithi032.stdout:7/16: creat d0/f3 x:0 0 0 2024-04-12T05:36:58.993 INFO:tasks.workunit.client.0.smithi032.stdout:2/53: mknod dd/c11 0 2024-04-12T05:36:58.993 INFO:tasks.workunit.client.0.smithi032.stdout:6/7: chown l1 16441 1 2024-04-12T05:36:58.993 INFO:tasks.workunit.client.0.smithi032.stdout:3/56: creat f10 x:0 0 0 2024-04-12T05:36:58.993 INFO:tasks.workunit.client.0.smithi032.stdout:2/54: dread f1 [0,4194304] 0 2024-04-12T05:36:58.993 INFO:tasks.workunit.client.0.smithi032.stdout:6/8: creat f2 x:0 0 0 2024-04-12T05:36:58.993 INFO:tasks.workunit.client.0.smithi032.stdout:3/57: symlink l11 0 2024-04-12T05:36:58.993 INFO:tasks.workunit.client.0.smithi032.stdout:7/17: link d0/f3 d0/f4 0 2024-04-12T05:36:58.993 INFO:tasks.workunit.client.0.smithi032.stdout:2/55: rmdir dd 39 2024-04-12T05:36:58.993 INFO:tasks.workunit.client.0.smithi032.stdout:2/56: write fa [3065319,111543] 0 2024-04-12T05:36:58.993 INFO:tasks.workunit.client.0.smithi032.stdout:6/9: creat f3 x:0 0 0 2024-04-12T05:36:58.994 INFO:tasks.workunit.client.0.smithi032.stdout:6/10: chown f3 52159 1 2024-04-12T05:36:58.994 INFO:tasks.workunit.client.0.smithi032.stdout:6/11: dread - f2 zero size 2024-04-12T05:36:58.994 INFO:tasks.workunit.client.0.smithi032.stdout:6/12: rmdir - no directory 2024-04-12T05:36:58.994 INFO:tasks.workunit.client.0.smithi032.stdout:3/58: mkdir d12 0 2024-04-12T05:36:58.994 INFO:tasks.workunit.client.0.smithi032.stdout:6/13: write f2 [640869,107209] 0 2024-04-12T05:36:58.994 INFO:tasks.workunit.client.0.smithi032.stdout:3/59: write f10 [579242,19095] 0 2024-04-12T05:36:58.994 INFO:tasks.workunit.client.0.smithi032.stdout:6/14: mkdir d4 0 2024-04-12T05:36:58.994 INFO:tasks.workunit.client.0.smithi032.stdout:3/60: dread f10 [0,4194304] 0 2024-04-12T05:36:58.994 INFO:tasks.workunit.client.0.smithi032.stdout:3/61: fdatasync fd 0 2024-04-12T05:36:58.994 INFO:tasks.workunit.client.0.smithi032.stdout:2/57: mknod dd/c12 0 2024-04-12T05:36:58.994 INFO:tasks.workunit.client.0.smithi032.stdout:6/15: dread f2 [0,4194304] 0 2024-04-12T05:36:58.994 INFO:tasks.workunit.client.0.smithi032.stdout:1/24: truncate d5/f6 1327239 0 2024-04-12T05:36:58.994 INFO:tasks.workunit.client.0.smithi032.stdout:2/58: dread fa [0,4194304] 0 2024-04-12T05:36:58.994 INFO:tasks.workunit.client.0.smithi032.stdout:1/25: symlink d5/la 0 2024-04-12T05:36:58.995 INFO:tasks.workunit.client.0.smithi032.stdout:9/101: fsync f7 0 2024-04-12T05:36:58.995 INFO:tasks.workunit.client.0.smithi032.stdout:9/102: write d9/dc/f20 [592564,456] 0 2024-04-12T05:36:58.995 INFO:tasks.workunit.client.0.smithi032.stdout:2/59: dwrite f6 [0,4194304] 0 2024-04-12T05:36:58.995 INFO:tasks.workunit.client.0.smithi032.stdout:9/103: chown d9/dc/f10 244 1 2024-04-12T05:36:58.995 INFO:tasks.workunit.client.0.smithi032.stdout:6/16: dwrite f2 [0,4194304] 0 2024-04-12T05:36:58.995 INFO:tasks.workunit.client.0.smithi032.stdout:2/60: readlink dd/l10 0 2024-04-12T05:36:58.995 INFO:tasks.workunit.client.0.smithi032.stdout:5/83: unlink d2/d7/l18 0 2024-04-12T05:36:58.995 INFO:tasks.workunit.client.0.smithi032.stdout:2/61: rename f4 to dd/f13 0 2024-04-12T05:36:58.995 INFO:tasks.workunit.client.0.smithi032.stdout:6/17: truncate f3 225334 0 2024-04-12T05:36:58.995 INFO:tasks.workunit.client.0.smithi032.stdout:5/84: mknod d2/d17/c1b 0 2024-04-12T05:36:58.995 INFO:tasks.workunit.client.0.smithi032.stdout:8/13: rmdir d0 39 2024-04-12T05:36:58.995 INFO:tasks.workunit.client.0.smithi032.stdout:2/62: dread f0 [0,4194304] 0 2024-04-12T05:36:58.995 INFO:tasks.workunit.client.0.smithi032.stdout:5/85: chown d2/f9 1213 1 2024-04-12T05:36:58.995 INFO:tasks.workunit.client.0.smithi032.stdout:4/39: getdents d4/da 0 2024-04-12T05:36:58.995 INFO:tasks.workunit.client.0.smithi032.stdout:0/122: readlink d2/d29/l2a 0 2024-04-12T05:36:58.996 INFO:tasks.workunit.client.0.smithi032.stdout:8/14: write d0/f3 [901724,79443] 0 2024-04-12T05:36:58.996 INFO:tasks.workunit.client.0.smithi032.stdout:2/63: rename fb to dd/f14 0 2024-04-12T05:36:58.996 INFO:tasks.workunit.client.0.smithi032.stdout:6/18: creat d4/f5 x:0 0 0 2024-04-12T05:36:58.996 INFO:tasks.workunit.client.0.smithi032.stdout:2/64: dread f0 [0,4194304] 0 2024-04-12T05:36:58.996 INFO:tasks.workunit.client.0.smithi032.stdout:7/18: fsync d0/f4 0 2024-04-12T05:36:58.996 INFO:tasks.workunit.client.0.smithi032.stdout:6/19: write d4/f5 [819679,99087] 0 2024-04-12T05:36:58.996 INFO:tasks.workunit.client.0.smithi032.stdout:5/86: rmdir d2/dd/d1a 0 2024-04-12T05:36:58.996 INFO:tasks.workunit.client.0.smithi032.stdout:0/123: mkdir d2/d28/d2c 0 2024-04-12T05:36:58.996 INFO:tasks.workunit.client.0.smithi032.stdout:7/19: dread - d0/f3 zero size 2024-04-12T05:36:58.996 INFO:tasks.workunit.client.0.smithi032.stdout:6/20: read d4/f5 [585132,127324] 0 2024-04-12T05:36:58.996 INFO:tasks.workunit.client.0.smithi032.stdout:4/40: dwrite d4/da/ff [0,4194304] 0 2024-04-12T05:36:58.996 INFO:tasks.workunit.client.0.smithi032.stdout:7/20: chown d0/f3 935 1 2024-04-12T05:36:58.996 INFO:tasks.workunit.client.0.smithi032.stdout:0/124: chown d2/d8/c1a 3 1 2024-04-12T05:36:58.996 INFO:tasks.workunit.client.0.smithi032.stdout:4/41: fsync d4/da/fd 0 2024-04-12T05:36:58.996 INFO:tasks.workunit.client.0.smithi032.stdout:8/15: dwrite d0/f3 [0,4194304] 0 2024-04-12T05:36:58.997 INFO:tasks.workunit.client.0.smithi032.stdout:5/87: creat d2/df/f1c x:0 0 0 2024-04-12T05:36:58.997 INFO:tasks.workunit.client.0.smithi032.stdout:1/26: write d5/f6 [1917817,99965] 0 2024-04-12T05:36:58.997 INFO:tasks.workunit.client.0.smithi032.stdout:5/88: write d2/f4 [651040,99309] 0 2024-04-12T05:36:58.998 INFO:tasks.workunit.client.0.smithi032.stdout:0/125: fdatasync d2/d8/d17/f1c 0 2024-04-12T05:36:59.002 INFO:tasks.workunit.client.0.smithi032.stdout:5/89: readlink d2/d7/lc 0 2024-04-12T05:36:59.002 INFO:tasks.workunit.client.0.smithi032.stdout:2/65: dwrite dd/f13 [0,4194304] 0 2024-04-12T05:36:59.005 INFO:tasks.workunit.client.0.smithi032.stdout:8/16: rename d0/f2 to d0/f6 0 2024-04-12T05:36:59.006 INFO:tasks.workunit.client.0.smithi032.stdout:1/27: creat d5/fb x:0 0 0 2024-04-12T05:36:59.008 INFO:tasks.workunit.client.0.smithi032.stdout:5/90: write d2/df/f1c [799527,67644] 0 2024-04-12T05:36:59.009 INFO:tasks.workunit.client.0.smithi032.stdout:0/126: creat d2/d29/f2d x:0 0 0 2024-04-12T05:36:59.010 INFO:tasks.workunit.client.0.smithi032.stdout:1/28: read d5/f6 [1013472,15373] 0 2024-04-12T05:36:59.011 INFO:tasks.workunit.client.0.smithi032.stdout:2/66: creat dd/f15 x:0 0 0 2024-04-12T05:36:59.013 INFO:tasks.workunit.client.0.smithi032.stdout:6/21: rmdir d4 39 2024-04-12T05:36:59.014 INFO:tasks.workunit.client.0.smithi032.stdout:2/67: readlink dd/l10 0 2024-04-12T05:36:59.015 INFO:tasks.workunit.client.0.smithi032.stdout:4/42: dwrite d4/da/ff [0,4194304] 0 2024-04-12T05:36:59.015 INFO:tasks.workunit.client.0.smithi032.stdout:7/21: dwrite d0/f3 [0,4194304] 0 2024-04-12T05:36:59.015 INFO:tasks.workunit.client.0.smithi032.stdout:5/91: write d2/f6 [490599,118758] 0 2024-04-12T05:36:59.019 INFO:tasks.workunit.client.0.smithi032.stdout:0/127: creat d2/d28/f2e x:0 0 0 2024-04-12T05:36:59.021 INFO:tasks.workunit.client.0.smithi032.stdout:5/92: readlink d2/d7/l12 0 2024-04-12T05:36:59.022 INFO:tasks.workunit.client.0.smithi032.stdout:7/22: chown d0/f3 0 1 2024-04-12T05:36:59.024 INFO:tasks.workunit.client.0.smithi032.stdout:4/43: write d4/da/fd [362883,65399] 0 2024-04-12T05:36:59.024 INFO:tasks.workunit.client.0.smithi032.stdout:2/68: dread dd/fe [0,4194304] 0 2024-04-12T05:36:59.027 INFO:tasks.workunit.client.0.smithi032.stdout:0/128: rename d2 to d2/d8/df/d24/d2f 22 2024-04-12T05:36:59.027 INFO:tasks.workunit.client.0.smithi032.stdout:2/69: chown dd 2 1 2024-04-12T05:36:59.029 INFO:tasks.workunit.client.0.smithi032.stdout:4/44: readlink d4/le 0 2024-04-12T05:36:59.032 INFO:tasks.workunit.client.0.smithi032.stdout:7/23: write d0/f4 [3291547,55462] 0 2024-04-12T05:36:59.032 INFO:tasks.workunit.client.0.smithi032.stdout:2/70: dread - dd/f14 zero size 2024-04-12T05:36:59.034 INFO:tasks.workunit.client.0.smithi032.stdout:4/45: chown l0 555529 1 2024-04-12T05:36:59.038 INFO:tasks.workunit.client.0.smithi032.stdout:2/71: unlink dd/l10 0 2024-04-12T05:36:59.043 INFO:tasks.workunit.client.0.smithi032.stdout:6/22: dwrite f2 [0,4194304] 0 2024-04-12T05:36:59.043 INFO:tasks.workunit.client.0.smithi032.stdout:8/17: dwrite d0/f6 [0,4194304] 0 2024-04-12T05:36:59.043 INFO:tasks.workunit.client.0.smithi032.stdout:4/46: mknod d4/c10 0 2024-04-12T05:36:59.043 INFO:tasks.workunit.client.0.smithi032.stdout:1/29: dwrite f4 [0,4194304] 0 2024-04-12T05:36:59.045 INFO:tasks.workunit.client.0.smithi032.stdout:6/23: fsync f3 0 2024-04-12T05:36:59.050 INFO:tasks.workunit.client.0.smithi032.stdout:0/129: dwrite d2/fe [0,4194304] 0 2024-04-12T05:36:59.050 INFO:tasks.workunit.client.0.smithi032.stdout:5/93: link d2/f19 d2/df/f1d 0 2024-04-12T05:36:59.050 INFO:tasks.workunit.client.0.smithi032.stdout:7/24: dread d0/f3 [0,4194304] 0 2024-04-12T05:36:59.051 INFO:tasks.workunit.client.0.smithi032.stdout:8/18: rmdir d0 39 2024-04-12T05:36:59.051 INFO:tasks.workunit.client.0.smithi032.stdout:4/47: mkdir d4/d11 0 2024-04-12T05:36:59.056 INFO:tasks.workunit.client.0.smithi032.stdout:4/48: write d4/da/fd [1251395,6910] 0 2024-04-12T05:36:59.056 INFO:tasks.workunit.client.0.smithi032.stdout:0/130: write d2/f14 [2166534,74411] 0 2024-04-12T05:36:59.057 INFO:tasks.workunit.client.0.smithi032.stdout:1/30: truncate d5/f6 409863 0 2024-04-12T05:36:59.061 INFO:tasks.workunit.client.0.smithi032.stdout:4/49: creat d4/f12 x:0 0 0 2024-04-12T05:36:59.062 INFO:tasks.workunit.client.0.smithi032.stdout:5/94: stat d2/f19 0 2024-04-12T05:36:59.063 INFO:tasks.workunit.client.0.smithi032.stdout:1/31: creat d5/fc x:0 0 0 2024-04-12T05:36:59.065 INFO:tasks.workunit.client.0.smithi032.stdout:4/50: write d4/f12 [570540,127135] 0 2024-04-12T05:36:59.070 INFO:tasks.workunit.client.0.smithi032.stdout:6/24: getdents d4 0 2024-04-12T05:36:59.071 INFO:tasks.workunit.client.0.smithi032.stdout:1/32: chown d5/l8 14062967 1 2024-04-12T05:36:59.072 INFO:tasks.workunit.client.0.smithi032.stdout:2/72: dwrite fc [0,4194304] 0 2024-04-12T05:36:59.073 INFO:tasks.workunit.client.0.smithi032.stdout:0/131: link d2/d8/d17/f22 d2/f30 0 2024-04-12T05:36:59.073 INFO:tasks.workunit.client.0.smithi032.stdout:2/73: chown dd 14629 1 2024-04-12T05:36:59.074 INFO:tasks.workunit.client.0.smithi032.stdout:4/51: rename d4/da to d4/d13 0 2024-04-12T05:36:59.076 INFO:tasks.workunit.client.0.smithi032.stdout:8/19: getdents d0 0 2024-04-12T05:36:59.078 INFO:tasks.workunit.client.0.smithi032.stdout:7/25: dwrite d0/f3 [4194304,4194304] 0 2024-04-12T05:36:59.079 INFO:tasks.workunit.client.0.smithi032.stdout:2/74: dread f2 [0,4194304] 0 2024-04-12T05:36:59.079 INFO:tasks.workunit.client.0.smithi032.stdout:6/25: creat d4/f6 x:0 0 0 2024-04-12T05:36:59.082 INFO:tasks.workunit.client.0.smithi032.stdout:7/26: write d0/f4 [2614703,110981] 0 2024-04-12T05:36:59.083 INFO:tasks.workunit.client.0.smithi032.stdout:7/27: readlink d0/l1 0 2024-04-12T05:36:59.088 INFO:tasks.workunit.client.0.smithi032.stdout:6/26: mknod d4/c7 0 2024-04-12T05:36:59.089 INFO:tasks.workunit.client.0.smithi032.stdout:0/132: dread d2/fe [0,4194304] 0 2024-04-12T05:36:59.089 INFO:tasks.workunit.client.0.smithi032.stdout:4/52: dread d4/fc [0,4194304] 0 2024-04-12T05:36:59.090 INFO:tasks.workunit.client.0.smithi032.stdout:6/27: read f3 [8861,48970] 0 2024-04-12T05:36:59.092 INFO:tasks.workunit.client.0.smithi032.stdout:8/20: link d0/f3 d0/f7 0 2024-04-12T05:36:59.095 INFO:tasks.workunit.client.0.smithi032.stdout:5/95: dwrite d2/df/f1d [0,4194304] 0 2024-04-12T05:36:59.096 INFO:tasks.workunit.client.0.smithi032.stdout:0/133: fsync d2/d8/d17/f2b 0 2024-04-12T05:36:59.098 INFO:tasks.workunit.client.0.smithi032.stdout:0/134: chown d2/d29/f2d 176826705 1 2024-04-12T05:36:59.099 INFO:tasks.workunit.client.0.smithi032.stdout:8/21: dread d0/f3 [0,4194304] 0 2024-04-12T05:36:59.102 INFO:tasks.workunit.client.0.smithi032.stdout:7/28: truncate d0/f3 2231592 0 2024-04-12T05:36:59.104 INFO:tasks.workunit.client.0.smithi032.stdout:5/96: mkdir d2/d7/d1e 0 2024-04-12T05:36:59.106 INFO:tasks.workunit.client.0.smithi032.stdout:4/53: rename d4/l8 to d4/d11/l14 0 2024-04-12T05:36:59.106 INFO:tasks.workunit.client.0.smithi032.stdout:0/135: fsync d2/fe 0 2024-04-12T05:36:59.107 INFO:tasks.workunit.client.0.smithi032.stdout:2/75: getdents dd 0 2024-04-12T05:36:59.107 INFO:tasks.workunit.client.0.smithi032.stdout:2/76: readlink - no filename 2024-04-12T05:36:59.108 INFO:tasks.workunit.client.0.smithi032.stdout:4/54: fsync d4/d13/fd 0 2024-04-12T05:36:59.109 INFO:tasks.workunit.client.0.smithi032.stdout:7/29: creat d0/f5 x:0 0 0 2024-04-12T05:36:59.110 INFO:tasks.workunit.client.0.smithi032.stdout:2/77: creat dd/f16 x:0 0 0 2024-04-12T05:36:59.110 INFO:tasks.workunit.client.0.smithi032.stdout:0/136: chown d2/d8/d17/d1b/l27 1 1 2024-04-12T05:36:59.111 INFO:tasks.workunit.client.0.smithi032.stdout:4/55: fdatasync d4/d13/fd 0 2024-04-12T05:36:59.112 INFO:tasks.workunit.client.0.smithi032.stdout:5/97: creat d2/d7/d1e/f1f x:0 0 0 2024-04-12T05:36:59.113 INFO:tasks.workunit.client.0.smithi032.stdout:8/22: write d0/f3 [3172067,96206] 0 2024-04-12T05:36:59.113 INFO:tasks.workunit.client.0.smithi032.stdout:6/28: dwrite d4/f5 [0,4194304] 0 2024-04-12T05:36:59.116 INFO:tasks.workunit.client.0.smithi032.stdout:7/30: unlink d0/c2 0 2024-04-12T05:36:59.117 INFO:tasks.workunit.client.0.smithi032.stdout:2/78: mkdir dd/d17 0 2024-04-12T05:36:59.118 INFO:tasks.workunit.client.0.smithi032.stdout:1/33: truncate d5/f6 552268 0 2024-04-12T05:36:59.119 INFO:tasks.workunit.client.0.smithi032.stdout:2/79: dread - dd/f15 zero size 2024-04-12T05:36:59.119 INFO:tasks.workunit.client.0.smithi032.stdout:5/98: symlink d2/d7/l20 0 2024-04-12T05:36:59.121 INFO:tasks.workunit.client.0.smithi032.stdout:0/137: fdatasync d2/d8/d17/f1c 0 2024-04-12T05:36:59.122 INFO:tasks.workunit.client.0.smithi032.stdout:8/23: creat d0/f8 x:0 0 0 2024-04-12T05:36:59.124 INFO:tasks.workunit.client.0.smithi032.stdout:4/56: dwrite d4/d13/fd [0,4194304] 0 2024-04-12T05:36:59.125 INFO:tasks.workunit.client.0.smithi032.stdout:7/31: mknod d0/c6 0 2024-04-12T05:36:59.125 INFO:tasks.workunit.client.0.smithi032.stdout:8/24: dread - d0/f8 zero size 2024-04-12T05:36:59.125 INFO:tasks.workunit.client.0.smithi032.stdout:0/138: truncate d2/d28/f2e 180451 0 2024-04-12T05:36:59.125 INFO:tasks.workunit.client.0.smithi032.stdout:1/34: rename d5/la to d5/ld 0 2024-04-12T05:36:59.127 INFO:tasks.workunit.client.0.smithi032.stdout:1/35: dread - d5/fb zero size 2024-04-12T05:36:59.128 INFO:tasks.workunit.client.0.smithi032.stdout:5/99: creat d2/d17/f21 x:0 0 0 2024-04-12T05:36:59.128 INFO:tasks.workunit.client.0.smithi032.stdout:0/139: chown d2/d8/df/l13 12710 1 2024-04-12T05:36:59.132 INFO:tasks.workunit.client.0.smithi032.stdout:5/100: write d2/f6 [3813018,33532] 0 2024-04-12T05:36:59.132 INFO:tasks.workunit.client.0.smithi032.stdout:0/140: write d2/f16 [299237,58301] 0 2024-04-12T05:36:59.132 INFO:tasks.workunit.client.0.smithi032.stdout:3/62: sync 2024-04-12T05:36:59.134 INFO:tasks.workunit.client.0.smithi032.stdout:4/57: symlink d4/d13/l15 0 2024-04-12T05:36:59.135 INFO:tasks.workunit.client.0.smithi032.stdout:7/32: rename d0/f4 to d0/f7 0 2024-04-12T05:36:59.136 INFO:tasks.workunit.client.0.smithi032.stdout:5/101: fdatasync d2/f19 0 2024-04-12T05:36:59.137 INFO:tasks.workunit.client.0.smithi032.stdout:2/80: getdents dd/d17 0 2024-04-12T05:36:59.137 INFO:tasks.workunit.client.0.smithi032.stdout:4/58: chown d4/c5 124174948 1 2024-04-12T05:36:59.137 INFO:tasks.workunit.client.0.smithi032.stdout:7/33: chown d0 0 1 2024-04-12T05:36:59.137 INFO:tasks.workunit.client.0.smithi032.stdout:1/36: mkdir d5/de 0 2024-04-12T05:36:59.139 INFO:tasks.workunit.client.0.smithi032.stdout:1/37: chown d5 5179626 1 2024-04-12T05:36:59.139 INFO:tasks.workunit.client.0.smithi032.stdout:8/25: dwrite d0/f5 [0,4194304] 0 2024-04-12T05:36:59.139 INFO:tasks.workunit.client.0.smithi032.stdout:4/59: read d4/d13/fd [3919338,25710] 0 2024-04-12T05:36:59.140 INFO:tasks.workunit.client.0.smithi032.stdout:5/102: write d2/d7/d1e/f1f [282401,130435] 0 2024-04-12T05:36:59.142 INFO:tasks.workunit.client.0.smithi032.stdout:6/29: truncate f2 854884 0 2024-04-12T05:36:59.143 INFO:tasks.workunit.client.0.smithi032.stdout:8/26: truncate d0/f8 955500 0 2024-04-12T05:36:59.144 INFO:tasks.workunit.client.0.smithi032.stdout:6/30: readlink l1 0 2024-04-12T05:36:59.145 INFO:tasks.workunit.client.0.smithi032.stdout:8/27: truncate d0/f3 5066644 0 2024-04-12T05:36:59.147 INFO:tasks.workunit.client.0.smithi032.stdout:7/34: fsync d0/f7 0 2024-04-12T05:36:59.148 INFO:tasks.workunit.client.0.smithi032.stdout:4/60: symlink d4/d11/l16 0 2024-04-12T05:36:59.148 INFO:tasks.workunit.client.0.smithi032.stdout:1/38: dread f4 [0,4194304] 0 2024-04-12T05:36:59.149 INFO:tasks.workunit.client.0.smithi032.stdout:5/103: mknod d2/d17/c22 0 2024-04-12T05:36:59.150 INFO:tasks.workunit.client.0.smithi032.stdout:3/63: creat d12/f13 x:0 0 0 2024-04-12T05:36:59.151 INFO:tasks.workunit.client.0.smithi032.stdout:7/35: dread d0/f7 [0,4194304] 0 2024-04-12T05:36:59.151 INFO:tasks.workunit.client.0.smithi032.stdout:5/104: read d2/f4 [4141538,54936] 0 2024-04-12T05:36:59.152 INFO:tasks.workunit.client.0.smithi032.stdout:0/141: write d2/d8/d17/f22 [2076416,67776] 0 2024-04-12T05:36:59.153 INFO:tasks.workunit.client.0.smithi032.stdout:8/28: mknod d0/c9 0 2024-04-12T05:36:59.154 INFO:tasks.workunit.client.0.smithi032.stdout:8/29: chown d0/f5 80820 1 2024-04-12T05:36:59.154 INFO:tasks.workunit.client.0.smithi032.stdout:2/81: getdents dd/d17 0 2024-04-12T05:36:59.155 INFO:tasks.workunit.client.0.smithi032.stdout:0/142: write d2/d28/f2e [874136,118045] 0 2024-04-12T05:36:59.155 INFO:tasks.workunit.client.0.smithi032.stdout:4/61: mknod d4/d11/c17 0 2024-04-12T05:36:59.156 INFO:tasks.workunit.client.0.smithi032.stdout:4/62: stat d4/d13/ff 0 2024-04-12T05:36:59.157 INFO:tasks.workunit.client.0.smithi032.stdout:1/39: creat d5/ff x:0 0 0 2024-04-12T05:36:59.158 INFO:tasks.workunit.client.0.smithi032.stdout:4/63: read d4/f12 [150179,3532] 0 2024-04-12T05:36:59.158 INFO:tasks.workunit.client.0.smithi032.stdout:7/36: creat d0/f8 x:0 0 0 2024-04-12T05:36:59.159 INFO:tasks.workunit.client.0.smithi032.stdout:5/105: rename d2/f19 to d2/d7/d1e/f23 0 2024-04-12T05:36:59.162 INFO:tasks.workunit.client.0.smithi032.stdout:0/143: creat d2/d8/df/f31 x:0 0 0 2024-04-12T05:36:59.164 INFO:tasks.workunit.client.0.smithi032.stdout:8/30: dwrite d0/f8 [0,4194304] 0 2024-04-12T05:36:59.165 INFO:tasks.workunit.client.0.smithi032.stdout:5/106: dread d2/f9 [0,4194304] 0 2024-04-12T05:36:59.167 INFO:tasks.workunit.client.0.smithi032.stdout:7/37: creat d0/f9 x:0 0 0 2024-04-12T05:36:59.168 INFO:tasks.workunit.client.0.smithi032.stdout:7/38: truncate d0/f5 1009096 0 2024-04-12T05:36:59.169 INFO:tasks.workunit.client.0.smithi032.stdout:6/31: truncate d4/f5 2647620 0 2024-04-12T05:36:59.172 INFO:tasks.workunit.client.0.smithi032.stdout:8/31: mknod d0/ca 0 2024-04-12T05:36:59.172 INFO:tasks.workunit.client.0.smithi032.stdout:0/144: dread d2/d8/fa [0,4194304] 0 2024-04-12T05:36:59.174 INFO:tasks.workunit.client.0.smithi032.stdout:1/40: link d5/fc d5/de/f10 0 2024-04-12T05:36:59.176 INFO:tasks.workunit.client.0.smithi032.stdout:6/32: symlink d4/l8 0 2024-04-12T05:36:59.177 INFO:tasks.workunit.client.0.smithi032.stdout:2/82: rmdir dd/d17 0 2024-04-12T05:36:59.178 INFO:tasks.workunit.client.0.smithi032.stdout:7/39: dwrite d0/f9 [0,4194304] 0 2024-04-12T05:36:59.179 INFO:tasks.workunit.client.0.smithi032.stdout:2/83: chown fc 30 1 2024-04-12T05:36:59.179 INFO:tasks.workunit.client.0.smithi032.stdout:8/32: mkdir d0/db 0 2024-04-12T05:36:59.180 INFO:tasks.workunit.client.0.smithi032.stdout:4/64: write d4/d13/fd [3805359,6029] 0 2024-04-12T05:36:59.182 INFO:tasks.workunit.client.0.smithi032.stdout:0/145: mkdir d2/d8/df/d32 0 2024-04-12T05:36:59.184 INFO:tasks.workunit.client.0.smithi032.stdout:6/33: unlink d4/c7 0 2024-04-12T05:36:59.185 INFO:tasks.workunit.client.0.smithi032.stdout:6/34: chown d4 2 1 2024-04-12T05:36:59.187 INFO:tasks.workunit.client.0.smithi032.stdout:0/146: write d2/d8/d17/f22 [4435425,11021] 0 2024-04-12T05:36:59.187 INFO:tasks.workunit.client.0.smithi032.stdout:1/41: dwrite d5/fb [0,4194304] 0 2024-04-12T05:36:59.189 INFO:tasks.workunit.client.0.smithi032.stdout:2/84: rename f6 to dd/f18 0 2024-04-12T05:36:59.189 INFO:tasks.workunit.client.0.smithi032.stdout:0/147: readlink d2/d8/d17/l26 0 2024-04-12T05:36:59.191 INFO:tasks.workunit.client.0.smithi032.stdout:0/148: write d2/d8/d17/f2b [725085,109764] 0 2024-04-12T05:36:59.191 INFO:tasks.workunit.client.0.smithi032.stdout:8/33: dread d0/f8 [0,4194304] 0 2024-04-12T05:36:59.192 INFO:tasks.workunit.client.0.smithi032.stdout:0/149: chown d2/d8/df 275892 1 2024-04-12T05:36:59.192 INFO:tasks.workunit.client.0.smithi032.stdout:8/34: chown d0/c4 1 1 2024-04-12T05:36:59.193 INFO:tasks.workunit.client.0.smithi032.stdout:4/65: unlink d4/fc 0 2024-04-12T05:36:59.193 INFO:tasks.workunit.client.0.smithi032.stdout:0/150: write d2/fe [4342236,26984] 0 2024-04-12T05:36:59.194 INFO:tasks.workunit.client.0.smithi032.stdout:5/107: creat d2/f24 x:0 0 0 2024-04-12T05:36:59.197 INFO:tasks.workunit.client.0.smithi032.stdout:7/40: link d0/f7 d0/fa 0 2024-04-12T05:36:59.201 INFO:tasks.workunit.client.0.smithi032.stdout:7/41: dread d0/fa [0,4194304] 0 2024-04-12T05:36:59.202 INFO:tasks.workunit.client.0.smithi032.stdout:2/85: mkdir dd/d19 0 2024-04-12T05:36:59.208 INFO:tasks.workunit.client.0.smithi032.stdout:6/35: rename f2 to d4/f9 0 2024-04-12T05:36:59.209 INFO:tasks.workunit.client.0.smithi032.stdout:7/42: creat d0/fb x:0 0 0 2024-04-12T05:36:59.210 INFO:tasks.workunit.client.0.smithi032.stdout:2/86: fsync dd/f14 0 2024-04-12T05:36:59.213 INFO:tasks.workunit.client.0.smithi032.stdout:5/108: rename d2/d7 to d2/df/d25 0 2024-04-12T05:36:59.216 INFO:tasks.workunit.client.0.smithi032.stdout:6/36: rename d4/l8 to d4/la 0 2024-04-12T05:36:59.217 INFO:tasks.workunit.client.0.smithi032.stdout:0/151: creat d2/d8/f33 x:0 0 0 2024-04-12T05:36:59.252 INFO:tasks.workunit.client.0.smithi032.stdout:9/104: sync 2024-04-12T05:36:59.264 INFO:tasks.workunit.client.0.smithi032.stdout:7/43: write d0/f3 [2707590,78492] 0 2024-04-12T05:36:59.264 INFO:tasks.workunit.client.0.smithi032.stdout:2/87: creat dd/f1a x:0 0 0 2024-04-12T05:36:59.265 INFO:tasks.workunit.client.0.smithi032.stdout:9/105: readlink l6 0 2024-04-12T05:36:59.265 INFO:tasks.workunit.client.0.smithi032.stdout:1/42: write d5/f6 [445575,6660] 0 2024-04-12T05:36:59.267 INFO:tasks.workunit.client.0.smithi032.stdout:0/152: stat d2/d8/d17/d1b/c1e 0 2024-04-12T05:36:59.268 INFO:tasks.workunit.client.0.smithi032.stdout:1/43: chown d5/l7 4796281 1 2024-04-12T05:36:59.269 INFO:tasks.workunit.client.0.smithi032.stdout:9/106: fsync d9/dc/f16 0 2024-04-12T05:36:59.270 INFO:tasks.workunit.client.0.smithi032.stdout:6/37: truncate f3 96165 0 2024-04-12T05:36:59.271 INFO:tasks.workunit.client.0.smithi032.stdout:0/153: write d2/f14 [3187994,130820] 0 2024-04-12T05:36:59.272 INFO:tasks.workunit.client.0.smithi032.stdout:7/44: rename d0/fb to d0/fc 0 2024-04-12T05:36:59.273 INFO:tasks.workunit.client.0.smithi032.stdout:5/109: dwrite d2/f6 [0,4194304] 0 2024-04-12T05:36:59.275 INFO:tasks.workunit.client.0.smithi032.stdout:9/107: dread d9/fa [4194304,4194304] 0 2024-04-12T05:36:59.275 INFO:tasks.workunit.client.0.smithi032.stdout:5/110: rename d2 to d2/df/d26 22 2024-04-12T05:36:59.276 INFO:tasks.workunit.client.0.smithi032.stdout:1/44: write f4 [38440,24433] 0 2024-04-12T05:36:59.277 INFO:tasks.workunit.client.0.smithi032.stdout:0/154: mknod d2/d8/df/c34 0 2024-04-12T05:36:59.279 INFO:tasks.workunit.client.0.smithi032.stdout:7/45: mkdir d0/dd 0 2024-04-12T05:36:59.279 INFO:tasks.workunit.client.0.smithi032.stdout:1/45: write f4 [1531197,22205] 0 2024-04-12T05:36:59.281 INFO:tasks.workunit.client.0.smithi032.stdout:5/111: symlink d2/d17/l27 0 2024-04-12T05:36:59.282 INFO:tasks.workunit.client.0.smithi032.stdout:2/88: dread dd/f13 [0,4194304] 0 2024-04-12T05:36:59.283 INFO:tasks.workunit.client.0.smithi032.stdout:1/46: chown d5/f6 51325581 1 2024-04-12T05:36:59.283 INFO:tasks.workunit.client.0.smithi032.stdout:7/46: mkdir d0/de 0 2024-04-12T05:36:59.286 INFO:tasks.workunit.client.0.smithi032.stdout:5/112: chown d2/df/d25/f16 1310156 1 2024-04-12T05:36:59.288 INFO:tasks.workunit.client.0.smithi032.stdout:2/89: mkdir dd/d1b 0 2024-04-12T05:36:59.288 INFO:tasks.workunit.client.0.smithi032.stdout:5/113: read d2/df/f1d [3580214,51189] 0 2024-04-12T05:36:59.289 INFO:tasks.workunit.client.0.smithi032.stdout:7/47: fdatasync d0/f7 0 2024-04-12T05:36:59.289 INFO:tasks.workunit.client.0.smithi032.stdout:0/155: dwrite d2/d8/d17/f22 [0,4194304] 0 2024-04-12T05:36:59.290 INFO:tasks.workunit.client.0.smithi032.stdout:9/108: getdents d9/d15/d1a 0 2024-04-12T05:36:59.291 INFO:tasks.workunit.client.0.smithi032.stdout:0/156: fdatasync d2/f6 0 2024-04-12T05:36:59.296 INFO:tasks.workunit.client.0.smithi032.stdout:9/109: fdatasync d9/dc/ff 0 2024-04-12T05:36:59.296 INFO:tasks.workunit.client.0.smithi032.stdout:7/48: mkdir d0/de/df 0 2024-04-12T05:36:59.297 INFO:tasks.workunit.client.0.smithi032.stdout:1/47: dread d5/fb [0,4194304] 0 2024-04-12T05:36:59.302 INFO:tasks.workunit.client.0.smithi032.stdout:7/49: chown d0/fc 19612716 1 2024-04-12T05:36:59.304 INFO:tasks.workunit.client.0.smithi032.stdout:1/48: write d5/f6 [1390538,131060] 0 2024-04-12T05:36:59.307 INFO:tasks.workunit.client.0.smithi032.stdout:7/50: mkdir d0/d10 0 2024-04-12T05:36:59.310 INFO:tasks.workunit.client.0.smithi032.stdout:2/90: dwrite dd/f14 [0,4194304] 0 2024-04-12T05:36:59.311 INFO:tasks.workunit.client.0.smithi032.stdout:6/38: dwrite d4/f5 [0,4194304] 0 2024-04-12T05:36:59.312 INFO:tasks.workunit.client.0.smithi032.stdout:5/114: dread d2/f4 [0,4194304] 0 2024-04-12T05:36:59.313 INFO:tasks.workunit.client.0.smithi032.stdout:1/49: creat d5/de/f11 x:0 0 0 2024-04-12T05:36:59.316 INFO:tasks.workunit.client.0.smithi032.stdout:0/157: dwrite d2/f14 [0,4194304] 0 2024-04-12T05:36:59.317 INFO:tasks.workunit.client.0.smithi032.stdout:1/50: symlink d5/l12 0 2024-04-12T05:36:59.318 INFO:tasks.workunit.client.0.smithi032.stdout:6/39: symlink d4/lb 0 2024-04-12T05:36:59.319 INFO:tasks.workunit.client.0.smithi032.stdout:9/110: dread d9/dc/f16 [0,4194304] 0 2024-04-12T05:36:59.320 INFO:tasks.workunit.client.0.smithi032.stdout:2/91: mkdir dd/d1b/d1c 0 2024-04-12T05:36:59.320 INFO:tasks.workunit.client.0.smithi032.stdout:2/92: chown f9 13 1 2024-04-12T05:36:59.320 INFO:tasks.workunit.client.0.smithi032.stdout:2/93: readlink - no filename 2024-04-12T05:36:59.322 INFO:tasks.workunit.client.0.smithi032.stdout:0/158: write d2/f16 [363435,125721] 0 2024-04-12T05:36:59.322 INFO:tasks.workunit.client.0.smithi032.stdout:7/51: truncate d0/f3 3728756 0 2024-04-12T05:36:59.322 INFO:tasks.workunit.client.0.smithi032.stdout:1/51: write d5/ff [285080,70179] 0 2024-04-12T05:36:59.323 INFO:tasks.workunit.client.0.smithi032.stdout:6/40: read - d4/f6 zero size 2024-04-12T05:36:59.323 INFO:tasks.workunit.client.0.smithi032.stdout:0/159: chown d2 21632 1 2024-04-12T05:36:59.324 INFO:tasks.workunit.client.0.smithi032.stdout:1/52: chown d5/l8 3 1 2024-04-12T05:36:59.325 INFO:tasks.workunit.client.0.smithi032.stdout:5/115: rename d2/f5 to d2/f28 0 2024-04-12T05:36:59.326 INFO:tasks.workunit.client.0.smithi032.stdout:0/160: chown d2/fe 7305 1 2024-04-12T05:36:59.328 INFO:tasks.workunit.client.0.smithi032.stdout:9/111: creat d9/dc/f25 x:0 0 0 2024-04-12T05:36:59.328 INFO:tasks.workunit.client.0.smithi032.stdout:1/53: dread - d5/fc zero size 2024-04-12T05:36:59.329 INFO:tasks.workunit.client.0.smithi032.stdout:5/116: chown d2/f4 9833503 1 2024-04-12T05:36:59.332 INFO:tasks.workunit.client.0.smithi032.stdout:7/52: dread d0/f5 [0,4194304] 0 2024-04-12T05:36:59.332 INFO:tasks.workunit.client.0.smithi032.stdout:0/161: rmdir d2/d8 39 2024-04-12T05:36:59.334 INFO:tasks.workunit.client.0.smithi032.stdout:7/53: chown d0/f8 7937 1 2024-04-12T05:36:59.335 INFO:tasks.workunit.client.0.smithi032.stdout:5/117: creat d2/df/d25/f29 x:0 0 0 2024-04-12T05:36:59.335 INFO:tasks.workunit.client.0.smithi032.stdout:6/41: rmdir d4 39 2024-04-12T05:36:59.336 INFO:tasks.workunit.client.0.smithi032.stdout:7/54: dread - d0/f8 zero size 2024-04-12T05:36:59.338 INFO:tasks.workunit.client.0.smithi032.stdout:7/55: chown d0 0 1 2024-04-12T05:36:59.338 INFO:tasks.workunit.client.0.smithi032.stdout:0/162: truncate d2/f7 1088490 0 2024-04-12T05:36:59.338 INFO:tasks.workunit.client.0.smithi032.stdout:2/94: dwrite fc [0,4194304] 0 2024-04-12T05:36:59.342 INFO:tasks.workunit.client.0.smithi032.stdout:6/42: mknod d4/cc 0 2024-04-12T05:36:59.346 INFO:tasks.workunit.client.0.smithi032.stdout:1/54: dwrite d5/fb [0,4194304] 0 2024-04-12T05:36:59.347 INFO:tasks.workunit.client.0.smithi032.stdout:0/163: rename d2/d29/f2d to d2/d28/d2c/f35 0 2024-04-12T05:36:59.348 INFO:tasks.workunit.client.0.smithi032.stdout:9/112: truncate d9/dc/f20 35874 0 2024-04-12T05:36:59.350 INFO:tasks.workunit.client.0.smithi032.stdout:7/56: creat d0/dd/f11 x:0 0 0 2024-04-12T05:36:59.351 INFO:tasks.workunit.client.0.smithi032.stdout:1/55: unlink d5/f6 0 2024-04-12T05:36:59.353 INFO:tasks.workunit.client.0.smithi032.stdout:3/64: sync 2024-04-12T05:36:59.354 INFO:tasks.workunit.client.0.smithi032.stdout:2/95: dread fc [0,4194304] 0 2024-04-12T05:36:59.355 INFO:tasks.workunit.client.0.smithi032.stdout:1/56: dread - d5/de/f10 zero size 2024-04-12T05:36:59.357 INFO:tasks.workunit.client.0.smithi032.stdout:5/118: dwrite d2/f28 [0,4194304] 0 2024-04-12T05:36:59.357 INFO:tasks.workunit.client.0.smithi032.stdout:2/96: chown dd/d1b 1 1 2024-04-12T05:36:59.359 INFO:tasks.workunit.client.0.smithi032.stdout:6/43: dwrite d4/f5 [0,4194304] 0 2024-04-12T05:36:59.360 INFO:tasks.workunit.client.0.smithi032.stdout:0/164: rename d2/c10 to d2/dc/c36 0 2024-04-12T05:36:59.361 INFO:tasks.workunit.client.0.smithi032.stdout:9/113: mknod d9/c26 0 2024-04-12T05:36:59.362 INFO:tasks.workunit.client.0.smithi032.stdout:2/97: fdatasync dd/f18 0 2024-04-12T05:36:59.363 INFO:tasks.workunit.client.0.smithi032.stdout:2/98: dread - dd/f1a zero size 2024-04-12T05:36:59.364 INFO:tasks.workunit.client.0.smithi032.stdout:6/44: mknod d4/cd 0 2024-04-12T05:36:59.367 INFO:tasks.workunit.client.0.smithi032.stdout:0/165: read d2/d8/d17/f2b [592045,8169] 0 2024-04-12T05:36:59.368 INFO:tasks.workunit.client.0.smithi032.stdout:2/99: rmdir dd 39 2024-04-12T05:36:59.371 INFO:tasks.workunit.client.0.smithi032.stdout:5/119: dwrite d2/df/d25/f16 [0,4194304] 0 2024-04-12T05:36:59.372 INFO:tasks.workunit.client.0.smithi032.stdout:5/120: stat d2/df/d25/f8 0 2024-04-12T05:36:59.373 INFO:tasks.workunit.client.0.smithi032.stdout:6/45: creat d4/fe x:0 0 0 2024-04-12T05:36:59.375 INFO:tasks.workunit.client.0.smithi032.stdout:9/114: mkdir d9/d15/d1a/d24/d27 0 2024-04-12T05:36:59.376 INFO:tasks.workunit.client.0.smithi032.stdout:7/57: write d0/f7 [4099632,14552] 0 2024-04-12T05:36:59.378 INFO:tasks.workunit.client.0.smithi032.stdout:2/100: stat dd/f18 0 2024-04-12T05:36:59.379 INFO:tasks.workunit.client.0.smithi032.stdout:5/121: dread d2/df/d25/d1e/f23 [0,4194304] 0 2024-04-12T05:36:59.380 INFO:tasks.workunit.client.0.smithi032.stdout:5/122: fdatasync d2/f3 0 2024-04-12T05:36:59.382 INFO:tasks.workunit.client.0.smithi032.stdout:3/65: truncate f7 518936 0 2024-04-12T05:36:59.382 INFO:tasks.workunit.client.0.smithi032.stdout:5/123: dread - d2/d17/f21 zero size 2024-04-12T05:36:59.384 INFO:tasks.workunit.client.0.smithi032.stdout:9/115: creat d9/d15/f28 x:0 0 0 2024-04-12T05:36:59.385 INFO:tasks.workunit.client.0.smithi032.stdout:7/58: unlink d0/fa 0 2024-04-12T05:36:59.386 INFO:tasks.workunit.client.0.smithi032.stdout:2/101: write fc [921308,65371] 0 2024-04-12T05:36:59.387 INFO:tasks.workunit.client.0.smithi032.stdout:1/57: dwrite d5/de/f10 [0,4194304] 0 2024-04-12T05:36:59.387 INFO:tasks.workunit.client.0.smithi032.stdout:7/59: chown d0/l1 113 1 2024-04-12T05:36:59.389 INFO:tasks.workunit.client.0.smithi032.stdout:0/166: creat d2/d8/f37 x:0 0 0 2024-04-12T05:36:59.389 INFO:tasks.workunit.client.0.smithi032.stdout:6/46: link d4/fe d4/ff 0 2024-04-12T05:36:59.391 INFO:tasks.workunit.client.0.smithi032.stdout:5/124: write f1 [2007357,100272] 0 2024-04-12T05:36:59.393 INFO:tasks.workunit.client.0.smithi032.stdout:7/60: creat d0/de/f12 x:0 0 0 2024-04-12T05:36:59.395 INFO:tasks.workunit.client.0.smithi032.stdout:0/167: truncate d2/dc/f18 1165664 0 2024-04-12T05:36:59.396 INFO:tasks.workunit.client.0.smithi032.stdout:6/47: creat d4/f10 x:0 0 0 2024-04-12T05:36:59.396 INFO:tasks.workunit.client.0.smithi032.stdout:3/66: dread fd [0,4194304] 0 2024-04-12T05:36:59.397 INFO:tasks.workunit.client.0.smithi032.stdout:5/125: dread - d2/df/d25/f29 zero size 2024-04-12T05:36:59.397 INFO:tasks.workunit.client.0.smithi032.stdout:2/102: getdents dd/d19 0 2024-04-12T05:36:59.398 INFO:tasks.workunit.client.0.smithi032.stdout:6/48: dread - d4/fe zero size 2024-04-12T05:36:59.400 INFO:tasks.workunit.client.0.smithi032.stdout:9/116: link d9/d15/c19 d9/d15/d1a/d1d/c29 0 2024-04-12T05:36:59.400 INFO:tasks.workunit.client.0.smithi032.stdout:6/49: chown l1 200651509 1 2024-04-12T05:36:59.401 INFO:tasks.workunit.client.0.smithi032.stdout:5/126: chown d2/df/d25/l12 725738 1 2024-04-12T05:36:59.402 INFO:tasks.workunit.client.0.smithi032.stdout:6/50: chown d4/ff 120064 1 2024-04-12T05:36:59.403 INFO:tasks.workunit.client.0.smithi032.stdout:1/58: dwrite d5/ff [0,4194304] 0 2024-04-12T05:36:59.405 INFO:tasks.workunit.client.0.smithi032.stdout:7/61: write d0/f3 [355754,16808] 0 2024-04-12T05:36:59.405 INFO:tasks.workunit.client.0.smithi032.stdout:6/51: dread - d4/ff zero size 2024-04-12T05:36:59.406 INFO:tasks.workunit.client.0.smithi032.stdout:9/117: unlink d9/f22 0 2024-04-12T05:36:59.407 INFO:tasks.workunit.client.0.smithi032.stdout:2/103: creat dd/d19/f1d x:0 0 0 2024-04-12T05:36:59.407 INFO:tasks.workunit.client.0.smithi032.stdout:0/168: dread d2/d8/d17/f1c [0,4194304] 0 2024-04-12T05:36:59.408 INFO:tasks.workunit.client.0.smithi032.stdout:9/118: stat d9/c26 0 2024-04-12T05:36:59.408 INFO:tasks.workunit.client.0.smithi032.stdout:1/59: rename f4 to d5/f13 0 2024-04-12T05:36:59.409 INFO:tasks.workunit.client.0.smithi032.stdout:0/169: fsync d2/f6 0 2024-04-12T05:36:59.409 INFO:tasks.workunit.client.0.smithi032.stdout:8/35: sync 2024-04-12T05:36:59.409 INFO:tasks.workunit.client.0.smithi032.stdout:4/66: sync 2024-04-12T05:36:59.410 INFO:tasks.workunit.client.0.smithi032.stdout:7/62: truncate d0/dd/f11 69465 0 2024-04-12T05:36:59.411 INFO:tasks.workunit.client.0.smithi032.stdout:6/52: unlink d4/fe 0 2024-04-12T05:36:59.412 INFO:tasks.workunit.client.0.smithi032.stdout:1/60: chown d5/fb 1013604 1 2024-04-12T05:36:59.413 INFO:tasks.workunit.client.0.smithi032.stdout:0/170: rmdir d2/d28 39 2024-04-12T05:36:59.414 INFO:tasks.workunit.client.0.smithi032.stdout:5/127: dread d2/df/d25/f16 [0,4194304] 0 2024-04-12T05:36:59.414 INFO:tasks.workunit.client.0.smithi032.stdout:4/67: write d4/d13/fd [2676480,44886] 0 2024-04-12T05:36:59.417 INFO:tasks.workunit.client.0.smithi032.stdout:7/63: mkdir d0/d13 0 2024-04-12T05:36:59.417 INFO:tasks.workunit.client.0.smithi032.stdout:4/68: chown d4/d11/c17 359421 1 2024-04-12T05:36:59.418 INFO:tasks.workunit.client.0.smithi032.stdout:7/64: chown d0/fc 0 1 2024-04-12T05:36:59.419 INFO:tasks.workunit.client.0.smithi032.stdout:4/69: chown d4/c9 3726356 1 2024-04-12T05:36:59.422 INFO:tasks.workunit.client.0.smithi032.stdout:9/119: dwrite d9/dc/f25 [0,4194304] 0 2024-04-12T05:36:59.425 INFO:tasks.workunit.client.0.smithi032.stdout:5/128: fdatasync d2/df/d25/f16 0 2024-04-12T05:36:59.427 INFO:tasks.workunit.client.0.smithi032.stdout:8/36: creat d0/db/fc x:0 0 0 2024-04-12T05:36:59.427 INFO:tasks.workunit.client.0.smithi032.stdout:8/37: readlink - no filename 2024-04-12T05:36:59.427 INFO:tasks.workunit.client.0.smithi032.stdout:0/171: write d2/fe [1215423,121410] 0 2024-04-12T05:36:59.429 INFO:tasks.workunit.client.0.smithi032.stdout:5/129: write d2/df/d25/f8 [281318,109019] 0 2024-04-12T05:36:59.430 INFO:tasks.workunit.client.0.smithi032.stdout:8/38: read d0/f8 [3779662,109505] 0 2024-04-12T05:36:59.432 INFO:tasks.workunit.client.0.smithi032.stdout:1/61: rename d5/ld to d5/de/l14 0 2024-04-12T05:36:59.433 INFO:tasks.workunit.client.0.smithi032.stdout:4/70: dwrite d4/d13/ff [4194304,4194304] 0 2024-04-12T05:36:59.434 INFO:tasks.workunit.client.0.smithi032.stdout:1/62: write d5/fc [5018757,47444] 0 2024-04-12T05:36:59.438 INFO:tasks.workunit.client.0.smithi032.stdout:2/104: truncate fa 4162213 0 2024-04-12T05:36:59.442 INFO:tasks.workunit.client.0.smithi032.stdout:7/65: creat d0/de/df/f14 x:0 0 0 2024-04-12T05:36:59.446 INFO:tasks.workunit.client.0.smithi032.stdout:8/39: fsync d0/db/fc 0 2024-04-12T05:36:59.447 INFO:tasks.workunit.client.0.smithi032.stdout:9/120: dread d9/fa [0,4194304] 0 2024-04-12T05:36:59.450 INFO:tasks.workunit.client.0.smithi032.stdout:9/121: read - d9/d15/f28 zero size 2024-04-12T05:36:59.450 INFO:tasks.workunit.client.0.smithi032.stdout:9/122: read d9/fa [1845489,113297] 0 2024-04-12T05:36:59.454 INFO:tasks.workunit.client.0.smithi032.stdout:6/53: write d4/ff [363260,50628] 0 2024-04-12T05:36:59.455 INFO:tasks.workunit.client.0.smithi032.stdout:7/66: dread d0/f7 [0,4194304] 0 2024-04-12T05:36:59.456 INFO:tasks.workunit.client.0.smithi032.stdout:2/105: rmdir dd 39 2024-04-12T05:36:59.480 INFO:tasks.workunit.client.0.smithi032.stdout:5/130: rename d2/c13 to d2/dd/c2a 0 2024-04-12T05:36:59.480 INFO:tasks.workunit.client.0.smithi032.stdout:0/172: mknod d2/d28/d2c/c38 0 2024-04-12T05:36:59.481 INFO:tasks.workunit.client.0.smithi032.stdout:8/40: mkdir d0/db/dd 0 2024-04-12T05:36:59.484 INFO:tasks.workunit.client.0.smithi032.stdout:0/173: chown d2/d8/df/d24 1 1 2024-04-12T05:36:59.490 INFO:tasks.workunit.client.0.smithi032.stdout:7/67: dwrite d0/f3 [0,4194304] 0 2024-04-12T05:36:59.490 INFO:tasks.workunit.client.0.smithi032.stdout:3/67: dwrite f7 [0,4194304] 0 2024-04-12T05:36:59.491 INFO:tasks.workunit.client.0.smithi032.stdout:7/68: readlink d0/l1 0 2024-04-12T05:36:59.492 INFO:tasks.workunit.client.0.smithi032.stdout:4/71: dwrite d4/f12 [0,4194304] 0 2024-04-12T05:36:59.497 INFO:tasks.workunit.client.0.smithi032.stdout:3/68: dread fd [0,4194304] 0 2024-04-12T05:36:59.497 INFO:tasks.workunit.client.0.smithi032.stdout:3/69: chown l11 1313286 1 2024-04-12T05:36:59.499 INFO:tasks.workunit.client.0.smithi032.stdout:3/70: write d12/f13 [533914,127650] 0 2024-04-12T05:36:59.601 INFO:tasks.workunit.client.0.smithi032.stdout:5/131: sync 2024-04-12T05:36:59.644 INFO:tasks.workunit.client.0.smithi032.stdout:9/123: symlink d9/l2a 0 2024-04-12T05:36:59.645 INFO:tasks.workunit.client.0.smithi032.stdout:8/41: creat d0/fe x:0 0 0 2024-04-12T05:36:59.653 INFO:tasks.workunit.client.0.smithi032.stdout:2/106: rename dd/f13 to dd/f1e 0 2024-04-12T05:36:59.654 INFO:tasks.workunit.client.0.smithi032.stdout:7/69: creat d0/de/f15 x:0 0 0 2024-04-12T05:36:59.655 INFO:tasks.workunit.client.0.smithi032.stdout:4/72: creat d4/f18 x:0 0 0 2024-04-12T05:36:59.655 INFO:tasks.workunit.client.0.smithi032.stdout:7/70: truncate d0/de/df/f14 40242 0 2024-04-12T05:36:59.655 INFO:tasks.workunit.client.0.smithi032.stdout:3/71: truncate f10 1530972 0 2024-04-12T05:36:59.656 INFO:tasks.workunit.client.0.smithi032.stdout:7/71: dread - d0/de/f15 zero size 2024-04-12T05:36:59.657 INFO:tasks.workunit.client.0.smithi032.stdout:1/63: getdents d5/de 0 2024-04-12T05:36:59.659 INFO:tasks.workunit.client.0.smithi032.stdout:6/54: mknod d4/c11 0 2024-04-12T05:36:59.663 INFO:tasks.workunit.client.0.smithi032.stdout:7/72: dwrite d0/f3 [0,4194304] 0 2024-04-12T05:36:59.664 INFO:tasks.workunit.client.0.smithi032.stdout:7/73: chown d0/dd 201 1 2024-04-12T05:36:59.664 INFO:tasks.workunit.client.0.smithi032.stdout:7/74: stat d0 0 2024-04-12T05:36:59.666 INFO:tasks.workunit.client.0.smithi032.stdout:7/75: dread d0/de/df/f14 [0,4194304] 0 2024-04-12T05:36:59.666 INFO:tasks.workunit.client.0.smithi032.stdout:7/76: stat d0/f5 0 2024-04-12T05:36:59.687 INFO:tasks.workunit.client.0.smithi032.stdout:0/174: mkdir d2/d8/df/d24/d39 0 2024-04-12T05:36:59.688 INFO:tasks.workunit.client.0.smithi032.stdout:0/175: dread d2/f7 [0,4194304] 0 2024-04-12T05:36:59.690 INFO:tasks.workunit.client.0.smithi032.stdout:5/132: symlink d2/l2b 0 2024-04-12T05:36:59.693 INFO:tasks.workunit.client.0.smithi032.stdout:1/64: mknod d5/de/c15 0 2024-04-12T05:36:59.694 INFO:tasks.workunit.client.0.smithi032.stdout:1/65: chown d5/l7 431300 1 2024-04-12T05:36:59.694 INFO:tasks.workunit.client.0.smithi032.stdout:1/66: stat l1 0 2024-04-12T05:36:59.695 INFO:tasks.workunit.client.0.smithi032.stdout:1/67: read d5/ff [2310982,97349] 0 2024-04-12T05:36:59.698 INFO:tasks.workunit.client.0.smithi032.stdout:7/77: rename d0/dd/f11 to d0/d13/f16 0 2024-04-12T05:36:59.699 INFO:tasks.workunit.client.0.smithi032.stdout:7/78: chown d0/d13/f16 13 1 2024-04-12T05:36:59.702 INFO:tasks.workunit.client.0.smithi032.stdout:0/176: unlink d2/dc/c19 0 2024-04-12T05:36:59.702 INFO:tasks.workunit.client.0.smithi032.stdout:0/177: chown d2/f7 6 1 2024-04-12T05:36:59.703 INFO:tasks.workunit.client.0.smithi032.stdout:0/178: stat d2/d8/fa 0 2024-04-12T05:36:59.703 INFO:tasks.workunit.client.0.smithi032.stdout:5/133: stat d2/dd/c2a 0 2024-04-12T05:36:59.703 INFO:tasks.workunit.client.0.smithi032.stdout:0/179: fsync d2/d8/f37 0 2024-04-12T05:36:59.704 INFO:tasks.workunit.client.0.smithi032.stdout:0/180: write d2/d8/df/f31 [522287,83197] 0 2024-04-12T05:36:59.706 INFO:tasks.workunit.client.0.smithi032.stdout:1/68: mknod d5/de/c16 0 2024-04-12T05:36:59.710 INFO:tasks.workunit.client.0.smithi032.stdout:7/79: mkdir d0/d17 0 2024-04-12T05:36:59.710 INFO:tasks.workunit.client.0.smithi032.stdout:5/134: dwrite d2/df/f1d [0,4194304] 0 2024-04-12T05:36:59.711 INFO:tasks.workunit.client.0.smithi032.stdout:5/135: dread - d2/f24 zero size 2024-04-12T05:36:59.712 INFO:tasks.workunit.client.0.smithi032.stdout:8/42: getdents d0/db 0 2024-04-12T05:36:59.713 INFO:tasks.workunit.client.0.smithi032.stdout:8/43: rename d0/db to d0/db/df 22 2024-04-12T05:36:59.714 INFO:tasks.workunit.client.0.smithi032.stdout:0/181: mknod d2/d29/c3a 0 2024-04-12T05:36:59.714 INFO:tasks.workunit.client.0.smithi032.stdout:8/44: write d0/f5 [4455286,17056] 0 2024-04-12T05:36:59.718 INFO:tasks.workunit.client.0.smithi032.stdout:4/73: getdents d4/d11 0 2024-04-12T05:36:59.719 INFO:tasks.workunit.client.0.smithi032.stdout:3/72: sync 2024-04-12T05:36:59.719 INFO:tasks.workunit.client.0.smithi032.stdout:5/136: creat d2/df/d25/f2c x:0 0 0 2024-04-12T05:36:59.722 INFO:tasks.workunit.client.0.smithi032.stdout:0/182: creat d2/d8/d17/f3b x:0 0 0 2024-04-12T05:36:59.724 INFO:tasks.workunit.client.0.smithi032.stdout:7/80: dwrite d0/d13/f16 [0,4194304] 0 2024-04-12T05:36:59.727 INFO:tasks.workunit.client.0.smithi032.stdout:5/137: unlink d2/dd/f14 0 2024-04-12T05:36:59.728 INFO:tasks.workunit.client.0.smithi032.stdout:0/183: dread d2/dc/fd [0,4194304] 0 2024-04-12T05:36:59.729 INFO:tasks.workunit.client.0.smithi032.stdout:8/45: symlink d0/db/dd/l10 0 2024-04-12T05:36:59.730 INFO:tasks.workunit.client.0.smithi032.stdout:8/46: fdatasync d0/db/fc 0 2024-04-12T05:36:59.735 INFO:tasks.workunit.client.0.smithi032.stdout:7/81: symlink d0/d13/l18 0 2024-04-12T05:36:59.735 INFO:tasks.workunit.client.0.smithi032.stdout:7/82: chown d0/l1 26 1 2024-04-12T05:36:59.738 INFO:tasks.workunit.client.0.smithi032.stdout:5/138: rename d2/f9 to d2/dd/f2d 0 2024-04-12T05:36:59.738 INFO:tasks.workunit.client.0.smithi032.stdout:5/139: chown d2/df/f1d 22857951 1 2024-04-12T05:36:59.739 INFO:tasks.workunit.client.0.smithi032.stdout:0/184: symlink d2/d28/d2c/l3c 0 2024-04-12T05:36:59.740 INFO:tasks.workunit.client.0.smithi032.stdout:0/185: chown d2/d28/f2e 2496437 1 2024-04-12T05:36:59.741 INFO:tasks.workunit.client.0.smithi032.stdout:8/47: mkdir d0/db/dd/d11 0 2024-04-12T05:36:59.745 INFO:tasks.workunit.client.0.smithi032.stdout:4/74: getdents d4/d13 0 2024-04-12T05:36:59.746 INFO:tasks.workunit.client.0.smithi032.stdout:5/140: dwrite d2/df/d25/f8 [0,4194304] 0 2024-04-12T05:36:59.747 INFO:tasks.workunit.client.0.smithi032.stdout:0/186: dread d2/f30 [0,4194304] 0 2024-04-12T05:36:59.749 INFO:tasks.workunit.client.0.smithi032.stdout:0/187: dread d2/f7 [0,4194304] 0 2024-04-12T05:36:59.754 INFO:tasks.workunit.client.0.smithi032.stdout:4/75: dread f3 [0,4194304] 0 2024-04-12T05:36:59.757 INFO:tasks.workunit.client.0.smithi032.stdout:0/188: dread d2/f16 [0,4194304] 0 2024-04-12T05:36:59.758 INFO:tasks.workunit.client.0.smithi032.stdout:5/141: dwrite d2/df/f1c [0,4194304] 0 2024-04-12T05:36:59.766 INFO:tasks.workunit.client.0.smithi032.stdout:5/142: dwrite d2/f28 [0,4194304] 0 2024-04-12T05:36:59.835 INFO:tasks.workunit.client.0.smithi032.stdout:4/76: rmdir d4 39 2024-04-12T05:36:59.839 INFO:tasks.workunit.client.0.smithi032.stdout:4/77: read d4/f12 [1000320,51845] 0 2024-04-12T05:36:59.842 INFO:tasks.workunit.client.0.smithi032.stdout:6/55: chown d4/f10 6293 1 2024-04-12T05:36:59.843 INFO:tasks.workunit.client.0.smithi032.stdout:8/48: sync 2024-04-12T05:36:59.844 INFO:tasks.workunit.client.0.smithi032.stdout:4/78: rename d4/d11 to d4/d13/d19 0 2024-04-12T05:36:59.844 INFO:tasks.workunit.client.0.smithi032.stdout:8/49: chown d0/db/dd/d11 1898 1 2024-04-12T05:36:59.845 INFO:tasks.workunit.client.0.smithi032.stdout:9/124: dwrite d9/dc/f23 [4194304,4194304] 0 2024-04-12T05:36:59.845 INFO:tasks.workunit.client.0.smithi032.stdout:6/56: symlink d4/l12 0 2024-04-12T05:36:59.846 INFO:tasks.workunit.client.0.smithi032.stdout:4/79: rename d4/d13 to d4/d13/d19/d1a 22 2024-04-12T05:36:59.849 INFO:tasks.workunit.client.0.smithi032.stdout:6/57: chown d4/ff 953749 1 2024-04-12T05:36:59.849 INFO:tasks.workunit.client.0.smithi032.stdout:8/50: symlink d0/db/dd/d11/l12 0 2024-04-12T05:36:59.852 INFO:tasks.workunit.client.0.smithi032.stdout:9/125: mknod d9/d15/d1a/d24/d27/c2b 0 2024-04-12T05:36:59.853 INFO:tasks.workunit.client.0.smithi032.stdout:8/51: creat d0/db/dd/d11/f13 x:0 0 0 2024-04-12T05:36:59.854 INFO:tasks.workunit.client.0.smithi032.stdout:8/52: chown d0 424450370 1 2024-04-12T05:36:59.856 INFO:tasks.workunit.client.0.smithi032.stdout:6/58: creat d4/f13 x:0 0 0 2024-04-12T05:36:59.856 INFO:tasks.workunit.client.0.smithi032.stdout:8/53: fsync d0/fe 0 2024-04-12T05:36:59.856 INFO:tasks.workunit.client.0.smithi032.stdout:4/80: dwrite d4/d13/fd [0,4194304] 0 2024-04-12T05:36:59.857 INFO:tasks.workunit.client.0.smithi032.stdout:8/54: dread - d0/fe zero size 2024-04-12T05:36:59.862 INFO:tasks.workunit.client.0.smithi032.stdout:9/126: rename d9/dc to d9/d15/d2c 0 2024-04-12T05:36:59.862 INFO:tasks.workunit.client.0.smithi032.stdout:8/55: write d0/db/fc [1039153,121061] 0 2024-04-12T05:36:59.862 INFO:tasks.workunit.client.0.smithi032.stdout:6/59: mkdir d4/d14 0 2024-04-12T05:36:59.862 INFO:tasks.workunit.client.0.smithi032.stdout:6/60: chown d4 92278 1 2024-04-12T05:36:59.866 INFO:tasks.workunit.client.0.smithi032.stdout:8/56: link d0/f3 d0/db/dd/f14 0 2024-04-12T05:36:59.869 INFO:tasks.workunit.client.0.smithi032.stdout:9/127: dwrite f5 [0,4194304] 0 2024-04-12T05:36:59.873 INFO:tasks.workunit.client.0.smithi032.stdout:9/128: write d9/d15/d2c/f14 [557851,16803] 0 2024-04-12T05:36:59.874 INFO:tasks.workunit.client.0.smithi032.stdout:1/69: getdents d5/de 0 2024-04-12T05:36:59.876 INFO:tasks.workunit.client.0.smithi032.stdout:6/61: dwrite d4/ff [0,4194304] 0 2024-04-12T05:36:59.878 INFO:tasks.workunit.client.0.smithi032.stdout:9/129: write d9/d15/f28 [78190,26881] 0 2024-04-12T05:36:59.881 INFO:tasks.workunit.client.0.smithi032.stdout:5/143: rmdir d2/df 39 2024-04-12T05:36:59.882 INFO:tasks.workunit.client.0.smithi032.stdout:6/62: fsync d4/f10 0 2024-04-12T05:36:59.884 INFO:tasks.workunit.client.0.smithi032.stdout:2/107: dwrite f8 [0,4194304] 0 2024-04-12T05:36:59.892 INFO:tasks.workunit.client.0.smithi032.stdout:9/130: rename d9/d15/d2c/f16 to d9/d15/d1a/d24/d27/f2d 0 2024-04-12T05:36:59.895 INFO:tasks.workunit.client.0.smithi032.stdout:0/189: sync 2024-04-12T05:36:59.896 INFO:tasks.workunit.client.0.smithi032.stdout:2/108: dread f8 [0,4194304] 0 2024-04-12T05:36:59.896 INFO:tasks.workunit.client.0.smithi032.stdout:7/83: truncate d0/d13/f16 2110499 0 2024-04-12T05:36:59.897 INFO:tasks.workunit.client.0.smithi032.stdout:2/109: chown f1 1 1 2024-04-12T05:36:59.897 INFO:tasks.workunit.client.0.smithi032.stdout:7/84: dread - d0/fc zero size 2024-04-12T05:36:59.897 INFO:tasks.workunit.client.0.smithi032.stdout:2/110: read - dd/f1a zero size 2024-04-12T05:36:59.898 INFO:tasks.workunit.client.0.smithi032.stdout:0/190: mkdir d2/d28/d2c/d3d 0 2024-04-12T05:36:59.899 INFO:tasks.workunit.client.0.smithi032.stdout:0/191: chown d2/d8/df/f31 323 1 2024-04-12T05:36:59.903 INFO:tasks.workunit.client.0.smithi032.stdout:0/192: mkdir d2/dc/d3e 0 2024-04-12T05:36:59.906 INFO:tasks.workunit.client.0.smithi032.stdout:9/131: getdents d9/d15/d1a/d1d 0 2024-04-12T05:36:59.906 INFO:tasks.workunit.client.0.smithi032.stdout:7/85: dread d0/f7 [0,4194304] 0 2024-04-12T05:36:59.908 INFO:tasks.workunit.client.0.smithi032.stdout:2/111: dwrite dd/f18 [0,4194304] 0 2024-04-12T05:36:59.911 INFO:tasks.workunit.client.0.smithi032.stdout:7/86: symlink d0/d13/l19 0 2024-04-12T05:36:59.914 INFO:tasks.workunit.client.0.smithi032.stdout:9/132: rename l6 to d9/d15/d1a/d1d/l2e 0 2024-04-12T05:36:59.915 INFO:tasks.workunit.client.0.smithi032.stdout:0/193: truncate d2/d8/d17/f1c 2036634 0 2024-04-12T05:36:59.915 INFO:tasks.workunit.client.0.smithi032.stdout:3/73: dwrite f10 [0,4194304] 0 2024-04-12T05:36:59.921 INFO:tasks.workunit.client.0.smithi032.stdout:6/63: getdents d4 0 2024-04-12T05:36:59.921 INFO:tasks.workunit.client.0.smithi032.stdout:3/74: rmdir d12 39 2024-04-12T05:36:59.922 INFO:tasks.workunit.client.0.smithi032.stdout:4/81: truncate d4/f12 2311577 0 2024-04-12T05:36:59.924 INFO:tasks.workunit.client.0.smithi032.stdout:2/112: mknod dd/d1b/d1c/c1f 0 2024-04-12T05:36:59.925 INFO:tasks.workunit.client.0.smithi032.stdout:2/113: truncate f9 242470 0 2024-04-12T05:36:59.927 INFO:tasks.workunit.client.0.smithi032.stdout:0/194: getdents d2/dc/d3e 0 2024-04-12T05:36:59.928 INFO:tasks.workunit.client.0.smithi032.stdout:3/75: write d12/f13 [175242,67789] 0 2024-04-12T05:36:59.928 INFO:tasks.workunit.client.0.smithi032.stdout:7/87: dwrite d0/f7 [0,4194304] 0 2024-04-12T05:36:59.929 INFO:tasks.workunit.client.0.smithi032.stdout:8/57: truncate d0/f8 460682 0 2024-04-12T05:36:59.931 INFO:tasks.workunit.client.0.smithi032.stdout:6/64: dread d4/ff [0,4194304] 0 2024-04-12T05:36:59.931 INFO:tasks.workunit.client.0.smithi032.stdout:4/82: mknod d4/d13/c1b 0 2024-04-12T05:36:59.931 INFO:tasks.workunit.client.0.smithi032.stdout:7/88: truncate d0/f8 96422 0 2024-04-12T05:36:59.931 INFO:tasks.workunit.client.0.smithi032.stdout:1/70: write d5/ff [2664283,5917] 0 2024-04-12T05:36:59.932 INFO:tasks.workunit.client.0.smithi032.stdout:2/114: fsync f1 0 2024-04-12T05:36:59.933 INFO:tasks.workunit.client.0.smithi032.stdout:0/195: dread d2/d28/f2e [0,4194304] 0 2024-04-12T05:36:59.935 INFO:tasks.workunit.client.0.smithi032.stdout:2/115: write dd/f1a [135808,121309] 0 2024-04-12T05:36:59.940 INFO:tasks.workunit.client.0.smithi032.stdout:1/71: dread d5/fc [0,4194304] 0 2024-04-12T05:36:59.940 INFO:tasks.workunit.client.0.smithi032.stdout:5/144: truncate d2/df/d25/d1e/f23 1280024 0 2024-04-12T05:36:59.941 INFO:tasks.workunit.client.0.smithi032.stdout:5/145: truncate f1 2152868 0 2024-04-12T05:36:59.941 INFO:tasks.workunit.client.0.smithi032.stdout:3/76: dread d12/f13 [0,4194304] 0 2024-04-12T05:36:59.942 INFO:tasks.workunit.client.0.smithi032.stdout:7/89: dread d0/de/df/f14 [0,4194304] 0 2024-04-12T05:36:59.945 INFO:tasks.workunit.client.0.smithi032.stdout:2/116: rmdir dd/d1b 39 2024-04-12T05:36:59.947 INFO:tasks.workunit.client.0.smithi032.stdout:6/65: dwrite d4/ff [0,4194304] 0 2024-04-12T05:36:59.948 INFO:tasks.workunit.client.0.smithi032.stdout:6/66: fdatasync d4/f6 0 2024-04-12T05:36:59.956 INFO:tasks.workunit.client.0.smithi032.stdout:7/90: creat d0/d13/f1a x:0 0 0 2024-04-12T05:36:59.959 INFO:tasks.workunit.client.0.smithi032.stdout:2/117: mkdir dd/d19/d20 0 2024-04-12T05:36:59.960 INFO:tasks.workunit.client.0.smithi032.stdout:8/58: link d0/ca d0/db/dd/c15 0 2024-04-12T05:36:59.962 INFO:tasks.workunit.client.0.smithi032.stdout:8/59: read d0/f3 [841781,65116] 0 2024-04-12T05:36:59.962 INFO:tasks.workunit.client.0.smithi032.stdout:8/60: write d0/db/fc [1712686,47209] 0 2024-04-12T05:36:59.968 INFO:tasks.workunit.client.0.smithi032.stdout:0/196: unlink d2/d8/d17/f1c 0 2024-04-12T05:36:59.971 INFO:tasks.workunit.client.0.smithi032.stdout:6/67: creat d4/d14/f15 x:0 0 0 2024-04-12T05:36:59.971 INFO:tasks.workunit.client.0.smithi032.stdout:4/83: truncate d4/d13/ff 3101049 0 2024-04-12T05:36:59.973 INFO:tasks.workunit.client.0.smithi032.stdout:5/146: link d2/df/d25/l20 d2/dd/l2e 0 2024-04-12T05:36:59.974 INFO:tasks.workunit.client.0.smithi032.stdout:5/147: fsync d2/d17/f21 0 2024-04-12T05:36:59.975 INFO:tasks.workunit.client.0.smithi032.stdout:6/68: dread d4/ff [0,4194304] 0 2024-04-12T05:36:59.975 INFO:tasks.workunit.client.0.smithi032.stdout:0/197: creat d2/d28/d2c/f3f x:0 0 0 2024-04-12T05:36:59.976 INFO:tasks.workunit.client.0.smithi032.stdout:1/72: getdents d5 0 2024-04-12T05:36:59.976 INFO:tasks.workunit.client.0.smithi032.stdout:0/198: dread - d2/d8/f37 zero size 2024-04-12T05:36:59.977 INFO:tasks.workunit.client.0.smithi032.stdout:0/199: chown d2/l3 18087413 1 2024-04-12T05:36:59.977 INFO:tasks.workunit.client.0.smithi032.stdout:3/77: truncate f10 2969427 0 2024-04-12T05:36:59.978 INFO:tasks.workunit.client.0.smithi032.stdout:3/78: chown l11 6649069 1 2024-04-12T05:36:59.979 INFO:tasks.workunit.client.0.smithi032.stdout:4/84: creat d4/d13/f1c x:0 0 0 2024-04-12T05:36:59.979 INFO:tasks.workunit.client.0.smithi032.stdout:7/91: write d0/de/df/f14 [147596,78858] 0 2024-04-12T05:36:59.981 INFO:tasks.workunit.client.0.smithi032.stdout:4/85: truncate d4/f18 372136 0 2024-04-12T05:36:59.981 INFO:tasks.workunit.client.0.smithi032.stdout:5/148: rmdir d2/df/d25 39 2024-04-12T05:36:59.982 INFO:tasks.workunit.client.0.smithi032.stdout:5/149: fsync d2/f3 0 2024-04-12T05:36:59.982 INFO:tasks.workunit.client.0.smithi032.stdout:3/79: dread f7 [0,4194304] 0 2024-04-12T05:36:59.982 INFO:tasks.workunit.client.0.smithi032.stdout:6/69: symlink d4/l16 0 2024-04-12T05:36:59.983 INFO:tasks.workunit.client.0.smithi032.stdout:3/80: stat fd 0 2024-04-12T05:36:59.983 INFO:tasks.workunit.client.0.smithi032.stdout:1/73: mkdir d5/de/d17 0 2024-04-12T05:36:59.987 INFO:tasks.workunit.client.0.smithi032.stdout:2/118: getdents dd 0 2024-04-12T05:36:59.989 INFO:tasks.workunit.client.0.smithi032.stdout:7/92: dread d0/f5 [0,4194304] 0 2024-04-12T05:36:59.992 INFO:tasks.workunit.client.0.smithi032.stdout:3/81: mkdir d12/d14 0 2024-04-12T05:36:59.993 INFO:tasks.workunit.client.0.smithi032.stdout:8/61: getdents d0/db/dd/d11 0 2024-04-12T05:36:59.993 INFO:tasks.workunit.client.0.smithi032.stdout:0/200: dwrite d2/f16 [0,4194304] 0 2024-04-12T05:36:59.996 INFO:tasks.workunit.client.0.smithi032.stdout:5/150: creat d2/df/d25/d1e/f2f x:0 0 0 2024-04-12T05:36:59.998 INFO:tasks.workunit.client.0.smithi032.stdout:2/119: rename dd/f1a to dd/d1b/d1c/f21 0 2024-04-12T05:36:59.999 INFO:tasks.workunit.client.0.smithi032.stdout:4/86: dwrite f3 [4194304,4194304] 0 2024-04-12T05:36:59.999 INFO:tasks.workunit.client.0.smithi032.stdout:6/70: symlink d4/d14/l17 0 2024-04-12T05:36:59.999 INFO:tasks.workunit.client.0.smithi032.stdout:8/62: rename d0/db/dd to d0/db/dd/d11/d16 22 2024-04-12T05:37:00.000 INFO:tasks.workunit.client.0.smithi032.stdout:7/93: creat d0/f1b x:0 0 0 2024-04-12T05:37:00.002 INFO:tasks.workunit.client.0.smithi032.stdout:8/63: readlink d0/db/dd/d11/l12 0 2024-04-12T05:37:00.002 INFO:tasks.workunit.client.0.smithi032.stdout:0/201: rmdir d2 39 2024-04-12T05:37:00.005 INFO:tasks.workunit.client.0.smithi032.stdout:5/151: dread d2/f3 [0,4194304] 0 2024-04-12T05:37:00.010 INFO:tasks.workunit.client.0.smithi032.stdout:0/202: readlink d2/d28/d2c/l3c 0 2024-04-12T05:37:00.012 INFO:tasks.workunit.client.0.smithi032.stdout:8/64: creat d0/db/dd/f17 x:0 0 0 2024-04-12T05:37:00.016 INFO:tasks.workunit.client.0.smithi032.stdout:3/82: rmdir d12/d14 0 2024-04-12T05:37:00.017 INFO:tasks.workunit.client.0.smithi032.stdout:8/65: dread d0/f7 [0,4194304] 0 2024-04-12T05:37:00.021 INFO:tasks.workunit.client.0.smithi032.stdout:0/203: unlink d2/dc/f18 0 2024-04-12T05:37:00.023 INFO:tasks.workunit.client.0.smithi032.stdout:3/83: rmdir d12 39 2024-04-12T05:37:00.024 INFO:tasks.workunit.client.0.smithi032.stdout:8/66: creat d0/db/f18 x:0 0 0 2024-04-12T05:37:00.025 INFO:tasks.workunit.client.0.smithi032.stdout:6/71: rename d4/l16 to d4/l18 0 2024-04-12T05:37:00.027 INFO:tasks.workunit.client.0.smithi032.stdout:5/152: dwrite d2/df/f1c [4194304,4194304] 0 2024-04-12T05:37:00.027 INFO:tasks.workunit.client.0.smithi032.stdout:2/120: fsync dd/f1e 0 2024-04-12T05:37:00.029 INFO:tasks.workunit.client.0.smithi032.stdout:4/87: getdents d4/d13 0 2024-04-12T05:37:00.031 INFO:tasks.workunit.client.0.smithi032.stdout:6/72: fdatasync d4/f6 0 2024-04-12T05:37:00.032 INFO:tasks.workunit.client.0.smithi032.stdout:7/94: dwrite d0/d13/f16 [0,4194304] 0 2024-04-12T05:37:00.033 INFO:tasks.workunit.client.0.smithi032.stdout:8/67: rmdir d0/db/dd/d11 39 2024-04-12T05:37:00.037 INFO:tasks.workunit.client.0.smithi032.stdout:2/121: fsync f0 0 2024-04-12T05:37:00.037 INFO:tasks.workunit.client.0.smithi032.stdout:3/84: dread f7 [0,4194304] 0 2024-04-12T05:37:00.037 INFO:tasks.workunit.client.0.smithi032.stdout:9/133: sync 2024-04-12T05:37:00.040 INFO:tasks.workunit.client.0.smithi032.stdout:5/153: symlink d2/l30 0 2024-04-12T05:37:00.040 INFO:tasks.workunit.client.0.smithi032.stdout:4/88: dread f3 [4194304,4194304] 0 2024-04-12T05:37:00.044 INFO:tasks.workunit.client.0.smithi032.stdout:0/204: getdents d2/d28/d2c 0 2024-04-12T05:37:00.046 INFO:tasks.workunit.client.0.smithi032.stdout:2/122: creat dd/d19/f22 x:0 0 0 2024-04-12T05:37:00.047 INFO:tasks.workunit.client.0.smithi032.stdout:8/68: link d0/f3 d0/db/f19 0 2024-04-12T05:37:00.048 INFO:tasks.workunit.client.0.smithi032.stdout:7/95: dwrite d0/de/f12 [0,4194304] 0 2024-04-12T05:37:00.049 INFO:tasks.workunit.client.0.smithi032.stdout:4/89: creat d4/f1d x:0 0 0 2024-04-12T05:37:00.049 INFO:tasks.workunit.client.0.smithi032.stdout:2/123: chown f0 89900655 1 2024-04-12T05:37:00.050 INFO:tasks.workunit.client.0.smithi032.stdout:9/134: mkdir d9/d1f/d2f 0 2024-04-12T05:37:00.055 INFO:tasks.workunit.client.0.smithi032.stdout:8/69: chown d0/db/dd/f17 618 1 2024-04-12T05:37:00.055 INFO:tasks.workunit.client.0.smithi032.stdout:6/73: dread d4/ff [0,4194304] 0 2024-04-12T05:37:00.055 INFO:tasks.workunit.client.0.smithi032.stdout:7/96: mkdir d0/d1c 0 2024-04-12T05:37:00.055 INFO:tasks.workunit.client.0.smithi032.stdout:9/135: creat d9/d15/f30 x:0 0 0 2024-04-12T05:37:00.056 INFO:tasks.workunit.client.0.smithi032.stdout:3/85: dwrite fd [0,4194304] 0 2024-04-12T05:37:00.066 INFO:tasks.workunit.client.0.smithi032.stdout:0/205: getdents d2/dc 0 2024-04-12T05:37:00.066 INFO:tasks.workunit.client.0.smithi032.stdout:3/86: dread d12/f13 [0,4194304] 0 2024-04-12T05:37:00.070 INFO:tasks.workunit.client.0.smithi032.stdout:6/74: creat d4/f19 x:0 0 0 2024-04-12T05:37:00.070 INFO:tasks.workunit.client.0.smithi032.stdout:7/97: link d0/d13/l19 d0/l1d 0 2024-04-12T05:37:00.070 INFO:tasks.workunit.client.0.smithi032.stdout:2/124: dwrite dd/ff [0,4194304] 0 2024-04-12T05:37:00.072 INFO:tasks.workunit.client.0.smithi032.stdout:4/90: truncate d4/d13/fd 1937971 0 2024-04-12T05:37:00.074 INFO:tasks.workunit.client.0.smithi032.stdout:8/70: dread d0/f3 [0,4194304] 0 2024-04-12T05:37:00.074 INFO:tasks.workunit.client.0.smithi032.stdout:3/87: symlink d12/l15 0 2024-04-12T05:37:00.077 INFO:tasks.workunit.client.0.smithi032.stdout:6/75: mkdir d4/d14/d1a 0 2024-04-12T05:37:00.078 INFO:tasks.workunit.client.0.smithi032.stdout:0/206: mknod d2/d8/df/d24/c40 0 2024-04-12T05:37:00.079 INFO:tasks.workunit.client.0.smithi032.stdout:5/154: dwrite d2/dd/f2d [0,4194304] 0 2024-04-12T05:37:00.080 INFO:tasks.workunit.client.0.smithi032.stdout:8/71: write d0/db/fc [737243,123500] 0 2024-04-12T05:37:00.081 INFO:tasks.workunit.client.0.smithi032.stdout:4/91: fsync d4/d13/ff 0 2024-04-12T05:37:00.082 INFO:tasks.workunit.client.0.smithi032.stdout:5/155: stat d2/dd/f2d 0 2024-04-12T05:37:00.084 INFO:tasks.workunit.client.0.smithi032.stdout:2/125: unlink dd/f18 0 2024-04-12T05:37:00.085 INFO:tasks.workunit.client.0.smithi032.stdout:7/98: write d0/f5 [591517,14114] 0 2024-04-12T05:37:00.085 INFO:tasks.workunit.client.0.smithi032.stdout:1/74: sync 2024-04-12T05:37:00.085 INFO:tasks.workunit.client.0.smithi032.stdout:3/88: rename fd to d12/f16 0 2024-04-12T05:37:00.086 INFO:tasks.workunit.client.0.smithi032.stdout:8/72: read d0/f5 [4185353,16119] 0 2024-04-12T05:37:00.086 INFO:tasks.workunit.client.0.smithi032.stdout:9/136: dwrite d9/d15/d1a/d24/d27/f2d [0,4194304] 0 2024-04-12T05:37:00.089 INFO:tasks.workunit.client.0.smithi032.stdout:1/75: dread - d5/de/f11 zero size 2024-04-12T05:37:00.090 INFO:tasks.workunit.client.0.smithi032.stdout:4/92: write d4/d13/f1c [123553,16963] 0 2024-04-12T05:37:00.092 INFO:tasks.workunit.client.0.smithi032.stdout:0/207: truncate d2/d8/df/f31 1545092 0 2024-04-12T05:37:00.092 INFO:tasks.workunit.client.0.smithi032.stdout:3/89: dread d12/f13 [0,4194304] 0 2024-04-12T05:37:00.094 INFO:tasks.workunit.client.0.smithi032.stdout:7/99: write d0/de/f12 [1955256,58248] 0 2024-04-12T05:37:00.095 INFO:tasks.workunit.client.0.smithi032.stdout:2/126: read - dd/f15 zero size 2024-04-12T05:37:00.097 INFO:tasks.workunit.client.0.smithi032.stdout:9/137: mkdir d9/d15/d1a/d31 0 2024-04-12T05:37:00.098 INFO:tasks.workunit.client.0.smithi032.stdout:4/93: fdatasync d4/d13/ff 0 2024-04-12T05:37:00.100 INFO:tasks.workunit.client.0.smithi032.stdout:6/76: creat d4/f1b x:0 0 0 2024-04-12T05:37:00.102 INFO:tasks.workunit.client.0.smithi032.stdout:3/90: rename lf to d12/l17 0 2024-04-12T05:37:00.104 INFO:tasks.workunit.client.0.smithi032.stdout:0/208: unlink d2/f16 0 2024-04-12T05:37:00.105 INFO:tasks.workunit.client.0.smithi032.stdout:8/73: dread d0/f3 [0,4194304] 0 2024-04-12T05:37:00.106 INFO:tasks.workunit.client.0.smithi032.stdout:6/77: write d4/f1b [919791,126957] 0 2024-04-12T05:37:00.107 INFO:tasks.workunit.client.0.smithi032.stdout:4/94: symlink d4/d13/d19/l1e 0 2024-04-12T05:37:00.112 INFO:tasks.workunit.client.0.smithi032.stdout:9/138: mkdir d9/d32 0 2024-04-12T05:37:00.112 INFO:tasks.workunit.client.0.smithi032.stdout:4/95: dread - d4/f1d zero size 2024-04-12T05:37:00.114 INFO:tasks.workunit.client.0.smithi032.stdout:1/76: dwrite d5/fb [0,4194304] 0 2024-04-12T05:37:00.115 INFO:tasks.workunit.client.0.smithi032.stdout:7/100: dwrite d0/de/f15 [0,4194304] 0 2024-04-12T05:37:00.115 INFO:tasks.workunit.client.0.smithi032.stdout:0/209: unlink d2/d8/d17/f3b 0 2024-04-12T05:37:00.119 INFO:tasks.workunit.client.0.smithi032.stdout:2/127: dread dd/f14 [0,4194304] 0 2024-04-12T05:37:00.141 INFO:tasks.workunit.client.0.smithi032.stdout:0/210: chown d2/f30 3304409 1 2024-04-12T05:37:00.141 INFO:tasks.workunit.client.0.smithi032.stdout:1/77: chown d5/de/c15 3 1 2024-04-12T05:37:00.141 INFO:tasks.workunit.client.0.smithi032.stdout:4/96: dread d4/f18 [0,4194304] 0 2024-04-12T05:37:00.142 INFO:tasks.workunit.client.0.smithi032.stdout:8/74: dread d0/db/fc [0,4194304] 0 2024-04-12T05:37:00.144 INFO:tasks.workunit.client.0.smithi032.stdout:2/128: chown f2 15771733 1 2024-04-12T05:37:00.145 INFO:tasks.workunit.client.0.smithi032.stdout:7/101: creat d0/de/df/f1e x:0 0 0 2024-04-12T05:37:00.146 INFO:tasks.workunit.client.0.smithi032.stdout:0/211: mkdir d2/dc/d41 0 2024-04-12T05:37:00.147 INFO:tasks.workunit.client.0.smithi032.stdout:1/78: write d5/f13 [5048461,96978] 0 2024-04-12T05:37:00.148 INFO:tasks.workunit.client.0.smithi032.stdout:6/78: dwrite d4/f1b [0,4194304] 0 2024-04-12T05:37:00.150 INFO:tasks.workunit.client.0.smithi032.stdout:7/102: dread - d0/f1b zero size 2024-04-12T05:37:00.151 INFO:tasks.workunit.client.0.smithi032.stdout:3/91: dwrite d12/f16 [4194304,4194304] 0 2024-04-12T05:37:00.152 INFO:tasks.workunit.client.0.smithi032.stdout:2/129: mknod dd/d19/c23 0 2024-04-12T05:37:00.159 INFO:tasks.workunit.client.0.smithi032.stdout:7/103: creat d0/d1c/f1f x:0 0 0 2024-04-12T05:37:00.163 INFO:tasks.workunit.client.0.smithi032.stdout:3/92: truncate f7 3084018 0 2024-04-12T05:37:00.163 INFO:tasks.workunit.client.0.smithi032.stdout:4/97: dwrite d4/d13/f1c [0,4194304] 0 2024-04-12T05:37:00.165 INFO:tasks.workunit.client.0.smithi032.stdout:8/75: dwrite d0/db/f18 [0,4194304] 0 2024-04-12T05:37:00.166 INFO:tasks.workunit.client.0.smithi032.stdout:8/76: chown d0/c1 1 1 2024-04-12T05:37:00.168 INFO:tasks.workunit.client.0.smithi032.stdout:7/104: rename d0/d17 to d0/de/df/d20 0 2024-04-12T05:37:00.172 INFO:tasks.workunit.client.0.smithi032.stdout:3/93: unlink l11 0 2024-04-12T05:37:00.173 INFO:tasks.workunit.client.0.smithi032.stdout:4/98: creat d4/d13/f1f x:0 0 0 2024-04-12T05:37:00.177 INFO:tasks.workunit.client.0.smithi032.stdout:8/77: mknod d0/c1a 0 2024-04-12T05:37:00.178 INFO:tasks.workunit.client.0.smithi032.stdout:7/105: mkdir d0/d13/d21 0 2024-04-12T05:37:00.179 INFO:tasks.workunit.client.0.smithi032.stdout:5/156: sync 2024-04-12T05:37:00.180 INFO:tasks.workunit.client.0.smithi032.stdout:4/99: rename d4/d13/d19/l16 to d4/l20 0 2024-04-12T05:37:00.181 INFO:tasks.workunit.client.0.smithi032.stdout:6/79: truncate d4/f1b 3163215 0 2024-04-12T05:37:00.184 INFO:tasks.workunit.client.0.smithi032.stdout:7/106: unlink d0/c6 0 2024-04-12T05:37:00.184 INFO:tasks.workunit.client.0.smithi032.stdout:2/130: dwrite f0 [0,4194304] 0 2024-04-12T05:37:00.185 INFO:tasks.workunit.client.0.smithi032.stdout:2/131: chown f2 18 1 2024-04-12T05:37:00.187 INFO:tasks.workunit.client.0.smithi032.stdout:0/212: truncate d2/d8/df/f31 1479711 0 2024-04-12T05:37:00.187 INFO:tasks.workunit.client.0.smithi032.stdout:4/100: dread d4/d13/ff [0,4194304] 0 2024-04-12T05:37:00.189 INFO:tasks.workunit.client.0.smithi032.stdout:7/107: unlink d0/f3 0 2024-04-12T05:37:00.191 INFO:tasks.workunit.client.0.smithi032.stdout:2/132: creat dd/d19/f24 x:0 0 0 2024-04-12T05:37:00.194 INFO:tasks.workunit.client.0.smithi032.stdout:4/101: symlink d4/d13/l21 0 2024-04-12T05:37:00.199 INFO:tasks.workunit.client.0.smithi032.stdout:3/94: chown f7 26 1 2024-04-12T05:37:00.202 INFO:tasks.workunit.client.0.smithi032.stdout:4/102: rmdir d4/d13/d19 39 2024-04-12T05:37:00.202 INFO:tasks.workunit.client.0.smithi032.stdout:8/78: dwrite d0/f5 [0,4194304] 0 2024-04-12T05:37:00.203 INFO:tasks.workunit.client.0.smithi032.stdout:0/213: mkdir d2/d8/df/d24/d39/d42 0 2024-04-12T05:37:00.204 INFO:tasks.workunit.client.0.smithi032.stdout:7/108: creat d0/d10/f22 x:0 0 0 2024-04-12T05:37:00.205 INFO:tasks.workunit.client.0.smithi032.stdout:7/109: chown d0/d13/d21 14 1 2024-04-12T05:37:00.206 INFO:tasks.workunit.client.0.smithi032.stdout:2/133: creat dd/d1b/d1c/f25 x:0 0 0 2024-04-12T05:37:00.206 INFO:tasks.workunit.client.0.smithi032.stdout:7/110: fdatasync d0/f8 0 2024-04-12T05:37:00.207 INFO:tasks.workunit.client.0.smithi032.stdout:7/111: readlink d0/d13/l18 0 2024-04-12T05:37:00.207 INFO:tasks.workunit.client.0.smithi032.stdout:7/112: stat d0/f8 0 2024-04-12T05:37:00.208 INFO:tasks.workunit.client.0.smithi032.stdout:5/157: getdents d2/df/d25 0 2024-04-12T05:37:00.208 INFO:tasks.workunit.client.0.smithi032.stdout:5/158: chown d2/f3 280 1 2024-04-12T05:37:00.209 INFO:tasks.workunit.client.0.smithi032.stdout:5/159: chown d2/f6 1195 1 2024-04-12T05:37:00.209 INFO:tasks.workunit.client.0.smithi032.stdout:3/95: unlink d12/f16 0 2024-04-12T05:37:00.215 INFO:tasks.workunit.client.0.smithi032.stdout:0/214: write d2/d28/d2c/f35 [1048547,6621] 0 2024-04-12T05:37:00.218 INFO:tasks.workunit.client.0.smithi032.stdout:7/113: creat d0/de/f23 x:0 0 0 2024-04-12T05:37:00.223 INFO:tasks.workunit.client.0.smithi032.stdout:3/96: rmdir d12 39 2024-04-12T05:37:00.224 INFO:tasks.workunit.client.0.smithi032.stdout:6/80: link d4/lb d4/l1c 0 2024-04-12T05:37:00.226 INFO:tasks.workunit.client.0.smithi032.stdout:8/79: mkdir d0/db/dd/d11/d1b 0 2024-04-12T05:37:00.230 INFO:tasks.workunit.client.0.smithi032.stdout:2/134: dread fa [0,4194304] 0 2024-04-12T05:37:00.230 INFO:tasks.workunit.client.0.smithi032.stdout:4/103: dwrite d4/f18 [0,4194304] 0 2024-04-12T05:37:00.231 INFO:tasks.workunit.client.0.smithi032.stdout:2/135: fdatasync dd/d1b/d1c/f25 0 2024-04-12T05:37:00.234 INFO:tasks.workunit.client.0.smithi032.stdout:5/160: truncate d2/df/d25/f8 3051444 0 2024-04-12T05:37:00.235 INFO:tasks.workunit.client.0.smithi032.stdout:3/97: chown f10 337144 1 2024-04-12T05:37:00.235 INFO:tasks.workunit.client.0.smithi032.stdout:0/215: creat d2/d28/d2c/d3d/f43 x:0 0 0 2024-04-12T05:37:00.236 INFO:tasks.workunit.client.0.smithi032.stdout:8/80: dread d0/f3 [0,4194304] 0 2024-04-12T05:37:00.236 INFO:tasks.workunit.client.0.smithi032.stdout:6/81: write d4/f5 [1216631,95928] 0 2024-04-12T05:37:00.237 INFO:tasks.workunit.client.0.smithi032.stdout:2/136: symlink dd/d19/l26 0 2024-04-12T05:37:00.238 INFO:tasks.workunit.client.0.smithi032.stdout:4/104: write f3 [5990018,46390] 0 2024-04-12T05:37:00.239 INFO:tasks.workunit.client.0.smithi032.stdout:7/114: dread - d0/d13/f1a zero size 2024-04-12T05:37:00.241 INFO:tasks.workunit.client.0.smithi032.stdout:5/161: mkdir d2/df/d31 0 2024-04-12T05:37:00.242 INFO:tasks.workunit.client.0.smithi032.stdout:7/115: chown d0/d13/l18 90 1 2024-04-12T05:37:00.246 INFO:tasks.workunit.client.0.smithi032.stdout:0/216: symlink d2/dc/d41/l44 0 2024-04-12T05:37:00.246 INFO:tasks.workunit.client.0.smithi032.stdout:2/137: creat dd/f27 x:0 0 0 2024-04-12T05:37:00.248 INFO:tasks.workunit.client.0.smithi032.stdout:4/105: symlink d4/d13/d19/l22 0 2024-04-12T05:37:00.249 INFO:tasks.workunit.client.0.smithi032.stdout:5/162: unlink d2/df/d25/lc 0 2024-04-12T05:37:00.251 INFO:tasks.workunit.client.0.smithi032.stdout:8/81: getdents d0/db 0 2024-04-12T05:37:00.251 INFO:tasks.workunit.client.0.smithi032.stdout:6/82: dwrite d4/f19 [0,4194304] 0 2024-04-12T05:37:00.251 INFO:tasks.workunit.client.0.smithi032.stdout:4/106: fsync d4/d13/f1c 0 2024-04-12T05:37:00.253 INFO:tasks.workunit.client.0.smithi032.stdout:7/116: creat d0/de/df/f24 x:0 0 0 2024-04-12T05:37:00.254 INFO:tasks.workunit.client.0.smithi032.stdout:0/217: creat d2/d8/d17/d1b/d20/f45 x:0 0 0 2024-04-12T05:37:00.257 INFO:tasks.workunit.client.0.smithi032.stdout:0/218: fsync d2/f6 0 2024-04-12T05:37:00.260 INFO:tasks.workunit.client.0.smithi032.stdout:8/82: creat d0/db/dd/d11/f1c x:0 0 0 2024-04-12T05:37:00.262 INFO:tasks.workunit.client.0.smithi032.stdout:3/98: dwrite f7 [0,4194304] 0 2024-04-12T05:37:00.266 INFO:tasks.workunit.client.0.smithi032.stdout:6/83: mknod d4/c1d 0 2024-04-12T05:37:00.267 INFO:tasks.workunit.client.0.smithi032.stdout:7/117: write d0/f8 [615172,109081] 0 2024-04-12T05:37:00.269 INFO:tasks.workunit.client.0.smithi032.stdout:9/139: sync 2024-04-12T05:37:00.270 INFO:tasks.workunit.client.0.smithi032.stdout:2/138: dwrite dd/d1b/d1c/f21 [0,4194304] 0 2024-04-12T05:37:00.271 INFO:tasks.workunit.client.0.smithi032.stdout:4/107: write d4/d13/fd [534199,20683] 0 2024-04-12T05:37:00.271 INFO:tasks.workunit.client.0.smithi032.stdout:6/84: write d4/f6 [1024350,51238] 0 2024-04-12T05:37:00.272 INFO:tasks.workunit.client.0.smithi032.stdout:5/163: dwrite d2/df/d25/d1e/f2f [0,4194304] 0 2024-04-12T05:37:00.274 INFO:tasks.workunit.client.0.smithi032.stdout:3/99: symlink d12/l18 0 2024-04-12T05:37:00.276 INFO:tasks.workunit.client.0.smithi032.stdout:7/118: mkdir d0/d13/d25 0 2024-04-12T05:37:00.278 INFO:tasks.workunit.client.0.smithi032.stdout:4/108: unlink d4/d13/c1b 0 2024-04-12T05:37:00.279 INFO:tasks.workunit.client.0.smithi032.stdout:9/140: mknod d9/d15/d2c/c33 0 2024-04-12T05:37:00.279 INFO:tasks.workunit.client.0.smithi032.stdout:0/219: dread d2/dc/fd [0,4194304] 0 2024-04-12T05:37:00.280 INFO:tasks.workunit.client.0.smithi032.stdout:2/139: truncate dd/d1b/d1c/f21 4382940 0 2024-04-12T05:37:00.280 INFO:tasks.workunit.client.0.smithi032.stdout:6/85: mknod d4/d14/c1e 0 2024-04-12T05:37:00.281 INFO:tasks.workunit.client.0.smithi032.stdout:8/83: link d0/db/dd/l10 d0/db/dd/d11/l1d 0 2024-04-12T05:37:00.282 INFO:tasks.workunit.client.0.smithi032.stdout:2/140: dread - dd/f15 zero size 2024-04-12T05:37:00.283 INFO:tasks.workunit.client.0.smithi032.stdout:7/119: mkdir d0/d13/d25/d26 0 2024-04-12T05:37:00.284 INFO:tasks.workunit.client.0.smithi032.stdout:4/109: mknod d4/d13/c23 0 2024-04-12T05:37:00.285 INFO:tasks.workunit.client.0.smithi032.stdout:4/110: chown d4/d13/ff 53 1 2024-04-12T05:37:00.288 INFO:tasks.workunit.client.0.smithi032.stdout:5/164: creat d2/df/f32 x:0 0 0 2024-04-12T05:37:00.288 INFO:tasks.workunit.client.0.smithi032.stdout:8/84: dwrite d0/f6 [0,4194304] 0 2024-04-12T05:37:00.289 INFO:tasks.workunit.client.0.smithi032.stdout:8/85: stat d0/c4 0 2024-04-12T05:37:00.290 INFO:tasks.workunit.client.0.smithi032.stdout:7/120: creat d0/d13/f27 x:0 0 0 2024-04-12T05:37:00.293 INFO:tasks.workunit.client.0.smithi032.stdout:6/86: symlink d4/d14/d1a/l1f 0 2024-04-12T05:37:00.293 INFO:tasks.workunit.client.0.smithi032.stdout:7/121: fdatasync d0/de/df/f1e 0 2024-04-12T05:37:00.295 INFO:tasks.workunit.client.0.smithi032.stdout:5/165: rename d2/f28 to d2/df/d31/f33 0 2024-04-12T05:37:00.295 INFO:tasks.workunit.client.0.smithi032.stdout:9/141: creat d9/f34 x:0 0 0 2024-04-12T05:37:00.296 INFO:tasks.workunit.client.0.smithi032.stdout:5/166: truncate d2/d17/f21 111093 0 2024-04-12T05:37:00.297 INFO:tasks.workunit.client.0.smithi032.stdout:6/87: fdatasync d4/ff 0 2024-04-12T05:37:00.298 INFO:tasks.workunit.client.0.smithi032.stdout:8/86: mkdir d0/db/dd/d11/d1b/d1e 0 2024-04-12T05:37:00.299 INFO:tasks.workunit.client.0.smithi032.stdout:8/87: read d0/f6 [1489159,83878] 0 2024-04-12T05:37:00.299 INFO:tasks.workunit.client.0.smithi032.stdout:7/122: mkdir d0/de/df/d20/d28 0 2024-04-12T05:37:00.300 INFO:tasks.workunit.client.0.smithi032.stdout:4/111: link d4/c10 d4/d13/d19/c24 0 2024-04-12T05:37:00.303 INFO:tasks.workunit.client.0.smithi032.stdout:6/88: chown d4/f10 88795542 1 2024-04-12T05:37:00.304 INFO:tasks.workunit.client.0.smithi032.stdout:8/88: stat d0/f8 0 2024-04-12T05:37:00.305 INFO:tasks.workunit.client.0.smithi032.stdout:8/89: stat d0/db/dd 0 2024-04-12T05:37:00.307 INFO:tasks.workunit.client.0.smithi032.stdout:7/123: rename d0/de/df/d20/d28 to d0/de/df/d29 0 2024-04-12T05:37:00.307 INFO:tasks.workunit.client.0.smithi032.stdout:8/90: readlink d0/db/dd/d11/l12 0 2024-04-12T05:37:00.308 INFO:tasks.workunit.client.0.smithi032.stdout:5/167: truncate d2/df/d25/f8 1903656 0 2024-04-12T05:37:00.309 INFO:tasks.workunit.client.0.smithi032.stdout:6/89: dwrite d4/f5 [0,4194304] 0 2024-04-12T05:37:00.310 INFO:tasks.workunit.client.0.smithi032.stdout:6/90: dread - d4/f10 zero size 2024-04-12T05:37:00.310 INFO:tasks.workunit.client.0.smithi032.stdout:6/91: dread - d4/f10 zero size 2024-04-12T05:37:00.311 INFO:tasks.workunit.client.0.smithi032.stdout:6/92: fsync d4/f19 0 2024-04-12T05:37:00.311 INFO:tasks.workunit.client.0.smithi032.stdout:4/112: mknod d4/d13/c25 0 2024-04-12T05:37:00.312 INFO:tasks.workunit.client.0.smithi032.stdout:4/113: chown d4/c9 1 1 2024-04-12T05:37:00.315 INFO:tasks.workunit.client.0.smithi032.stdout:8/91: truncate d0/db/dd/f14 3312282 0 2024-04-12T05:37:00.316 INFO:tasks.workunit.client.0.smithi032.stdout:8/92: chown d0/f5 8114 1 2024-04-12T05:37:00.318 INFO:tasks.workunit.client.0.smithi032.stdout:7/124: creat d0/dd/f2a x:0 0 0 2024-04-12T05:37:00.319 INFO:tasks.workunit.client.0.smithi032.stdout:6/93: symlink d4/l20 0 2024-04-12T05:37:00.319 INFO:tasks.workunit.client.0.smithi032.stdout:4/114: dwrite d4/f18 [0,4194304] 0 2024-04-12T05:37:00.320 INFO:tasks.workunit.client.0.smithi032.stdout:8/93: readlink d0/db/dd/l10 0 2024-04-12T05:37:00.321 INFO:tasks.workunit.client.0.smithi032.stdout:4/115: chown d4/d13/f1f 1209035 1 2024-04-12T05:37:00.322 INFO:tasks.workunit.client.0.smithi032.stdout:5/168: link d2/d17/f21 d2/f34 0 2024-04-12T05:37:00.323 INFO:tasks.workunit.client.0.smithi032.stdout:4/116: chown d4/d13/fd 181749546 1 2024-04-12T05:37:00.325 INFO:tasks.workunit.client.0.smithi032.stdout:6/94: mkdir d4/d21 0 2024-04-12T05:37:00.327 INFO:tasks.workunit.client.0.smithi032.stdout:5/169: write d2/df/d25/f16 [3704963,9988] 0 2024-04-12T05:37:00.327 INFO:tasks.workunit.client.0.smithi032.stdout:8/94: link d0/f5 d0/db/dd/d11/f1f 0 2024-04-12T05:37:00.332 INFO:tasks.workunit.client.0.smithi032.stdout:7/125: dwrite d0/de/df/f1e [0,4194304] 0 2024-04-12T05:37:00.332 INFO:tasks.workunit.client.0.smithi032.stdout:8/95: rmdir d0/db 39 2024-04-12T05:37:00.332 INFO:tasks.workunit.client.0.smithi032.stdout:6/95: chown d4/f9 89219406 1 2024-04-12T05:37:00.333 INFO:tasks.workunit.client.0.smithi032.stdout:8/96: read d0/f8 [414624,35319] 0 2024-04-12T05:37:00.336 INFO:tasks.workunit.client.0.smithi032.stdout:5/170: creat d2/df/d25/f35 x:0 0 0 2024-04-12T05:37:00.336 INFO:tasks.workunit.client.0.smithi032.stdout:5/171: stat d2/df/d25/ce 0 2024-04-12T05:37:00.340 INFO:tasks.workunit.client.0.smithi032.stdout:4/117: truncate d4/d13/ff 216258 0 2024-04-12T05:37:00.343 INFO:tasks.workunit.client.0.smithi032.stdout:6/96: rename d4/l1c to d4/d14/d1a/l22 0 2024-04-12T05:37:00.344 INFO:tasks.workunit.client.0.smithi032.stdout:7/126: getdents d0/de/df/d29 0 2024-04-12T05:37:00.344 INFO:tasks.workunit.client.0.smithi032.stdout:6/97: stat d4/d14/f15 0 2024-04-12T05:37:00.345 INFO:tasks.workunit.client.0.smithi032.stdout:7/127: chown d0/dd/f2a 1316 1 2024-04-12T05:37:00.345 INFO:tasks.workunit.client.0.smithi032.stdout:5/172: mknod d2/dd/c36 0 2024-04-12T05:37:00.348 INFO:tasks.workunit.client.0.smithi032.stdout:4/118: rename d4/d13/l15 to d4/l26 0 2024-04-12T05:37:00.353 INFO:tasks.workunit.client.0.smithi032.stdout:4/119: symlink d4/d13/d19/l27 0 2024-04-12T05:37:00.355 INFO:tasks.workunit.client.0.smithi032.stdout:7/128: symlink d0/d10/l2b 0 2024-04-12T05:37:00.358 INFO:tasks.workunit.client.0.smithi032.stdout:7/129: write d0/de/df/f14 [19168,117065] 0 2024-04-12T05:37:00.360 INFO:tasks.workunit.client.0.smithi032.stdout:4/120: dwrite d4/f1d [0,4194304] 0 2024-04-12T05:37:00.360 INFO:tasks.workunit.client.0.smithi032.stdout:5/173: chown d2/df/d25/f8 3145 1 2024-04-12T05:37:00.361 INFO:tasks.workunit.client.0.smithi032.stdout:4/121: fdatasync d4/f18 0 2024-04-12T05:37:00.365 INFO:tasks.workunit.client.0.smithi032.stdout:5/174: truncate d2/df/d25/f35 946356 0 2024-04-12T05:37:00.366 INFO:tasks.workunit.client.0.smithi032.stdout:4/122: unlink f3 0 2024-04-12T05:37:00.368 INFO:tasks.workunit.client.0.smithi032.stdout:4/123: chown d4/le 8372664 1 2024-04-12T05:37:00.370 INFO:tasks.workunit.client.0.smithi032.stdout:6/98: dwrite f3 [0,4194304] 0 2024-04-12T05:37:00.371 INFO:tasks.workunit.client.0.smithi032.stdout:4/124: chown d4/cb 6 1 2024-04-12T05:37:00.372 INFO:tasks.workunit.client.0.smithi032.stdout:6/99: fsync d4/f1b 0 2024-04-12T05:37:00.374 INFO:tasks.workunit.client.0.smithi032.stdout:4/125: chown d4/l7 2 1 2024-04-12T05:37:00.375 INFO:tasks.workunit.client.0.smithi032.stdout:6/100: chown d4/f5 2 1 2024-04-12T05:37:00.376 INFO:tasks.workunit.client.0.smithi032.stdout:5/175: dread d2/df/d25/d1e/f23 [0,4194304] 0 2024-04-12T05:37:00.378 INFO:tasks.workunit.client.0.smithi032.stdout:8/97: dwrite d0/f3 [0,4194304] 0 2024-04-12T05:37:00.379 INFO:tasks.workunit.client.0.smithi032.stdout:4/126: mkdir d4/d13/d19/d28 0 2024-04-12T05:37:00.379 INFO:tasks.workunit.client.0.smithi032.stdout:6/101: symlink d4/d14/d1a/l23 0 2024-04-12T05:37:00.382 INFO:tasks.workunit.client.0.smithi032.stdout:4/127: symlink d4/l29 0 2024-04-12T05:37:00.383 INFO:tasks.workunit.client.0.smithi032.stdout:6/102: chown d4/l18 1012 1 2024-04-12T05:37:00.386 INFO:tasks.workunit.client.0.smithi032.stdout:4/128: creat d4/d13/d19/f2a x:0 0 0 2024-04-12T05:37:00.387 INFO:tasks.workunit.client.0.smithi032.stdout:5/176: link d2/df/d25/ce d2/c37 0 2024-04-12T05:37:00.387 INFO:tasks.workunit.client.0.smithi032.stdout:6/103: truncate d4/ff 4180680 0 2024-04-12T05:37:00.389 INFO:tasks.workunit.client.0.smithi032.stdout:8/98: dwrite d0/db/dd/f14 [0,4194304] 0 2024-04-12T05:37:00.390 INFO:tasks.workunit.client.0.smithi032.stdout:6/104: unlink d4/f10 0 2024-04-12T05:37:00.391 INFO:tasks.workunit.client.0.smithi032.stdout:8/99: truncate d0/fe 829 0 2024-04-12T05:37:00.392 INFO:tasks.workunit.client.0.smithi032.stdout:4/129: dread d4/f18 [0,4194304] 0 2024-04-12T05:37:00.392 INFO:tasks.workunit.client.0.smithi032.stdout:5/177: rename d2/df/d25/l12 to d2/df/d31/l38 0 2024-04-12T05:37:00.393 INFO:tasks.workunit.client.0.smithi032.stdout:4/130: chown d4/d13/d19/l14 111624003 1 2024-04-12T05:37:00.394 INFO:tasks.workunit.client.0.smithi032.stdout:6/105: dread - d4/f13 zero size 2024-04-12T05:37:00.394 INFO:tasks.workunit.client.0.smithi032.stdout:4/131: read d4/f18 [694356,103178] 0 2024-04-12T05:37:00.395 INFO:tasks.workunit.client.0.smithi032.stdout:8/100: creat d0/db/f20 x:0 0 0 2024-04-12T05:37:00.397 INFO:tasks.workunit.client.0.smithi032.stdout:5/178: creat d2/d17/f39 x:0 0 0 2024-04-12T05:37:00.397 INFO:tasks.workunit.client.0.smithi032.stdout:1/79: sync 2024-04-12T05:37:00.398 INFO:tasks.workunit.client.0.smithi032.stdout:6/106: symlink d4/d14/d1a/l24 0 2024-04-12T05:37:00.399 INFO:tasks.workunit.client.0.smithi032.stdout:1/80: write d5/de/f11 [509740,3977] 0 2024-04-12T05:37:00.399 INFO:tasks.workunit.client.0.smithi032.stdout:4/132: symlink d4/l2b 0 2024-04-12T05:37:00.400 INFO:tasks.workunit.client.0.smithi032.stdout:5/179: write d2/df/d25/f2c [10932,24605] 0 2024-04-12T05:37:00.402 INFO:tasks.workunit.client.0.smithi032.stdout:6/107: mkdir d4/d14/d25 0 2024-04-12T05:37:00.403 INFO:tasks.workunit.client.0.smithi032.stdout:8/101: dread d0/db/dd/d11/f1f [0,4194304] 0 2024-04-12T05:37:00.404 INFO:tasks.workunit.client.0.smithi032.stdout:5/180: truncate d2/f24 834826 0 2024-04-12T05:37:00.404 INFO:tasks.workunit.client.0.smithi032.stdout:4/133: symlink d4/d13/d19/l2c 0 2024-04-12T05:37:00.404 INFO:tasks.workunit.client.0.smithi032.stdout:8/102: readlink d0/db/dd/d11/l1d 0 2024-04-12T05:37:00.405 INFO:tasks.workunit.client.0.smithi032.stdout:6/108: creat d4/d14/f26 x:0 0 0 2024-04-12T05:37:00.406 INFO:tasks.workunit.client.0.smithi032.stdout:5/181: chown d2/dd/l2e 32 1 2024-04-12T05:37:00.406 INFO:tasks.workunit.client.0.smithi032.stdout:4/134: truncate d4/d13/fd 2258712 0 2024-04-12T05:37:00.407 INFO:tasks.workunit.client.0.smithi032.stdout:6/109: write d4/f1b [3324463,63629] 0 2024-04-12T05:37:00.407 INFO:tasks.workunit.client.0.smithi032.stdout:8/103: unlink d0/f6 0 2024-04-12T05:37:00.409 INFO:tasks.workunit.client.0.smithi032.stdout:5/182: truncate d2/df/d25/f35 1333849 0 2024-04-12T05:37:00.409 INFO:tasks.workunit.client.0.smithi032.stdout:8/104: dread d0/f8 [0,4194304] 0 2024-04-12T05:37:00.409 INFO:tasks.workunit.client.0.smithi032.stdout:4/135: fsync d4/f12 0 2024-04-12T05:37:00.410 INFO:tasks.workunit.client.0.smithi032.stdout:6/110: mkdir d4/d21/d27 0 2024-04-12T05:37:00.411 INFO:tasks.workunit.client.0.smithi032.stdout:6/111: stat d4/f5 0 2024-04-12T05:37:00.412 INFO:tasks.workunit.client.0.smithi032.stdout:5/183: fsync d2/df/f1d 0 2024-04-12T05:37:00.412 INFO:tasks.workunit.client.0.smithi032.stdout:8/105: write d0/db/dd/f14 [3077166,72498] 0 2024-04-12T05:37:00.412 INFO:tasks.workunit.client.0.smithi032.stdout:5/184: fdatasync f1 0 2024-04-12T05:37:00.413 INFO:tasks.workunit.client.0.smithi032.stdout:4/136: symlink d4/d13/l2d 0 2024-04-12T05:37:00.413 INFO:tasks.workunit.client.0.smithi032.stdout:4/137: readlink d4/d13/l21 0 2024-04-12T05:37:00.414 INFO:tasks.workunit.client.0.smithi032.stdout:4/138: readlink d4/d13/l21 0 2024-04-12T05:37:00.425 INFO:tasks.workunit.client.0.smithi032.stdout:6/112: creat d4/d21/d27/f28 x:0 0 0 2024-04-12T05:37:00.425 INFO:tasks.workunit.client.0.smithi032.stdout:8/106: dwrite d0/db/f18 [0,4194304] 0 2024-04-12T05:37:00.426 INFO:tasks.workunit.client.0.smithi032.stdout:5/185: write d2/df/d31/f33 [1941404,97230] 0 2024-04-12T05:37:00.427 INFO:tasks.workunit.client.0.smithi032.stdout:6/113: readlink d4/d14/d1a/l1f 0 2024-04-12T05:37:00.430 INFO:tasks.workunit.client.0.smithi032.stdout:5/186: write d2/df/d25/f35 [746363,108921] 0 2024-04-12T05:37:00.431 INFO:tasks.workunit.client.0.smithi032.stdout:6/114: symlink d4/d21/l29 0 2024-04-12T05:37:00.433 INFO:tasks.workunit.client.0.smithi032.stdout:4/139: link d4/l2b d4/d13/l2e 0 2024-04-12T05:37:00.434 INFO:tasks.workunit.client.0.smithi032.stdout:8/107: creat d0/db/dd/f21 x:0 0 0 2024-04-12T05:37:00.436 INFO:tasks.workunit.client.0.smithi032.stdout:1/81: dwrite d5/fc [0,4194304] 0 2024-04-12T05:37:00.436 INFO:tasks.workunit.client.0.smithi032.stdout:5/187: rename d2/f3 to d2/dd/f3a 0 2024-04-12T05:37:00.437 INFO:tasks.workunit.client.0.smithi032.stdout:4/140: mkdir d4/d13/d19/d28/d2f 0 2024-04-12T05:37:00.440 INFO:tasks.workunit.client.0.smithi032.stdout:8/108: getdents d0/db 0 2024-04-12T05:37:00.446 INFO:tasks.workunit.client.0.smithi032.stdout:4/141: creat d4/d13/d19/f30 x:0 0 0 2024-04-12T05:37:00.446 INFO:tasks.workunit.client.0.smithi032.stdout:5/188: dread d2/df/d25/f8 [0,4194304] 0 2024-04-12T05:37:00.447 INFO:tasks.workunit.client.0.smithi032.stdout:8/109: mkdir d0/db/dd/d11/d1b/d22 0 2024-04-12T05:37:00.449 INFO:tasks.workunit.client.0.smithi032.stdout:6/115: dwrite d4/d21/d27/f28 [0,4194304] 0 2024-04-12T05:37:00.450 INFO:tasks.workunit.client.0.smithi032.stdout:6/116: chown d4/c1d 943742155 1 2024-04-12T05:37:00.451 INFO:tasks.workunit.client.0.smithi032.stdout:4/142: creat d4/f31 x:0 0 0 2024-04-12T05:37:00.452 INFO:tasks.workunit.client.0.smithi032.stdout:1/82: dwrite d5/de/f11 [0,4194304] 0 2024-04-12T05:37:00.454 INFO:tasks.workunit.client.0.smithi032.stdout:5/189: mknod d2/d17/c3b 0 2024-04-12T05:37:00.454 INFO:tasks.workunit.client.0.smithi032.stdout:8/110: mknod d0/c23 0 2024-04-12T05:37:00.458 INFO:tasks.workunit.client.0.smithi032.stdout:4/143: mkdir d4/d13/d19/d28/d32 0 2024-04-12T05:37:00.462 INFO:tasks.workunit.client.0.smithi032.stdout:6/117: rename d4/f13 to d4/f2a 0 2024-04-12T05:37:00.462 INFO:tasks.workunit.client.0.smithi032.stdout:1/83: dwrite d5/fb [0,4194304] 0 2024-04-12T05:37:00.463 INFO:tasks.workunit.client.0.smithi032.stdout:6/118: chown l1 1 1 2024-04-12T05:37:00.465 INFO:tasks.workunit.client.0.smithi032.stdout:5/190: getdents d2/dd 0 2024-04-12T05:37:00.470 INFO:tasks.workunit.client.0.smithi032.stdout:4/144: dread d4/d13/fd [0,4194304] 0 2024-04-12T05:37:00.470 INFO:tasks.workunit.client.0.smithi032.stdout:8/111: dwrite d0/db/f19 [0,4194304] 0 2024-04-12T05:37:00.470 INFO:tasks.workunit.client.0.smithi032.stdout:6/119: read d4/ff [2884421,68496] 0 2024-04-12T05:37:00.470 INFO:tasks.workunit.client.0.smithi032.stdout:6/120: fsync d4/d14/f26 0 2024-04-12T05:37:00.470 INFO:tasks.workunit.client.0.smithi032.stdout:5/191: mkdir d2/d3c 0 2024-04-12T05:37:00.472 INFO:tasks.workunit.client.0.smithi032.stdout:5/192: write f1 [1332789,21201] 0 2024-04-12T05:37:00.473 INFO:tasks.workunit.client.0.smithi032.stdout:5/193: fsync d2/f24 0 2024-04-12T05:37:00.474 INFO:tasks.workunit.client.0.smithi032.stdout:1/84: dread d5/de/f10 [0,4194304] 0 2024-04-12T05:37:00.476 INFO:tasks.workunit.client.0.smithi032.stdout:6/121: dread d4/f6 [0,4194304] 0 2024-04-12T05:37:00.477 INFO:tasks.workunit.client.0.smithi032.stdout:6/122: dread - d4/d14/f15 zero size 2024-04-12T05:37:00.478 INFO:tasks.workunit.client.0.smithi032.stdout:6/123: chown d4/d14/f26 0 1 2024-04-12T05:37:00.479 INFO:tasks.workunit.client.0.smithi032.stdout:8/112: creat d0/f24 x:0 0 0 2024-04-12T05:37:00.500 INFO:tasks.workunit.client.0.smithi032.stdout:1/85: rmdir d5/de 39 2024-04-12T05:37:00.501 INFO:tasks.workunit.client.0.smithi032.stdout:1/86: chown d5/ff 26458 1 2024-04-12T05:37:00.502 INFO:tasks.workunit.client.0.smithi032.stdout:5/194: dwrite d2/df/d25/d1e/f2f [0,4194304] 0 2024-04-12T05:37:00.502 INFO:tasks.workunit.client.0.smithi032.stdout:1/87: stat d5/fc 0 2024-04-12T05:37:00.503 INFO:tasks.workunit.client.0.smithi032.stdout:3/100: sync 2024-04-12T05:37:00.506 INFO:tasks.workunit.client.0.smithi032.stdout:5/195: dread d2/df/d25/f16 [0,4194304] 0 2024-04-12T05:37:00.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:00 smithi067 ceph-mon[49010]: pgmap v261: 65 pgs: 65 active+clean; 339 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 7.2 MiB/s wr, 231 op/s 2024-04-12T05:37:00.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:00 smithi032 ceph-mon[39731]: pgmap v261: 65 pgs: 65 active+clean; 339 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 7.2 MiB/s wr, 231 op/s 2024-04-12T05:37:00.594 INFO:tasks.workunit.client.0.smithi032.stdout:7/130: sync 2024-04-12T05:37:00.594 INFO:tasks.workunit.client.0.smithi032.stdout:2/141: sync 2024-04-12T05:37:00.595 INFO:tasks.workunit.client.0.smithi032.stdout:9/142: sync 2024-04-12T05:37:00.595 INFO:tasks.workunit.client.0.smithi032.stdout:0/220: sync 2024-04-12T05:37:00.614 INFO:tasks.workunit.client.0.smithi032.stdout:4/145: link d4/d13/ff d4/d13/f33 0 2024-04-12T05:37:00.615 INFO:tasks.workunit.client.0.smithi032.stdout:8/113: fdatasync d0/db/dd/d11/f1f 0 2024-04-12T05:37:00.615 INFO:tasks.workunit.client.0.smithi032.stdout:1/88: chown d5/de 1 1 2024-04-12T05:37:00.616 INFO:tasks.workunit.client.0.smithi032.stdout:3/101: mkdir d12/d19 0 2024-04-12T05:37:00.618 INFO:tasks.workunit.client.0.smithi032.stdout:7/131: rmdir d0/d10 39 2024-04-12T05:37:00.619 INFO:tasks.workunit.client.0.smithi032.stdout:5/196: sync 2024-04-12T05:37:00.624 INFO:tasks.workunit.client.0.smithi032.stdout:7/132: write d0/de/df/f14 [343732,87834] 0 2024-04-12T05:37:00.626 INFO:tasks.workunit.client.0.smithi032.stdout:6/124: getdents d4/d14 0 2024-04-12T05:37:00.627 INFO:tasks.workunit.client.0.smithi032.stdout:4/146: dwrite d4/f1d [0,4194304] 0 2024-04-12T05:37:00.628 INFO:tasks.workunit.client.0.smithi032.stdout:5/197: write d2/df/d25/d1e/f2f [4226907,51008] 0 2024-04-12T05:37:00.632 INFO:tasks.workunit.client.0.smithi032.stdout:9/143: link d9/d15/d1a/c1b d9/d15/d1a/d1d/c35 0 2024-04-12T05:37:00.633 INFO:tasks.workunit.client.0.smithi032.stdout:3/102: getdents d12/d19 0 2024-04-12T05:37:00.633 INFO:tasks.workunit.client.0.smithi032.stdout:7/133: write d0/de/f23 [348302,39193] 0 2024-04-12T05:37:00.634 INFO:tasks.workunit.client.0.smithi032.stdout:8/114: dwrite d0/db/f20 [0,4194304] 0 2024-04-12T05:37:00.636 INFO:tasks.workunit.client.0.smithi032.stdout:2/142: getdents dd 0 2024-04-12T05:37:00.638 INFO:tasks.workunit.client.0.smithi032.stdout:0/221: dwrite d2/d8/d17/f22 [4194304,4194304] 0 2024-04-12T05:37:00.640 INFO:tasks.workunit.client.0.smithi032.stdout:8/115: write d0/db/dd/f21 [874302,49895] 0 2024-04-12T05:37:00.645 INFO:tasks.workunit.client.0.smithi032.stdout:5/198: mknod d2/df/d25/d1e/c3d 0 2024-04-12T05:37:00.646 INFO:tasks.workunit.client.0.smithi032.stdout:5/199: chown d2 16542904 1 2024-04-12T05:37:00.646 INFO:tasks.workunit.client.0.smithi032.stdout:9/144: readlink d9/d15/d1a/d1d/l2e 0 2024-04-12T05:37:00.649 INFO:tasks.workunit.client.0.smithi032.stdout:2/143: dread f0 [0,4194304] 0 2024-04-12T05:37:00.652 INFO:tasks.workunit.client.0.smithi032.stdout:4/147: link d4/d13/d19/f30 d4/d13/d19/f34 0 2024-04-12T05:37:00.653 INFO:tasks.workunit.client.0.smithi032.stdout:3/103: link lc d12/l1a 0 2024-04-12T05:37:00.654 INFO:tasks.workunit.client.0.smithi032.stdout:9/145: mknod d9/d15/c36 0 2024-04-12T05:37:00.656 INFO:tasks.workunit.client.0.smithi032.stdout:6/125: getdents d4/d21/d27 0 2024-04-12T05:37:00.657 INFO:tasks.workunit.client.0.smithi032.stdout:5/200: dwrite d2/df/d25/d1e/f2f [0,4194304] 0 2024-04-12T05:37:00.658 INFO:tasks.workunit.client.0.smithi032.stdout:0/222: dread d2/f14 [0,4194304] 0 2024-04-12T05:37:00.658 INFO:tasks.workunit.client.0.smithi032.stdout:6/126: chown d4/f19 3 1 2024-04-12T05:37:00.659 INFO:tasks.workunit.client.0.smithi032.stdout:4/148: creat d4/f35 x:0 0 0 2024-04-12T05:37:00.660 INFO:tasks.workunit.client.0.smithi032.stdout:2/144: creat dd/d19/d20/f28 x:0 0 0 2024-04-12T05:37:00.661 INFO:tasks.workunit.client.0.smithi032.stdout:3/104: chown lc 1 1 2024-04-12T05:37:00.662 INFO:tasks.workunit.client.0.smithi032.stdout:5/201: symlink d2/d17/l3e 0 2024-04-12T05:37:00.665 INFO:tasks.workunit.client.0.smithi032.stdout:4/149: rmdir d4/d13/d19/d28 39 2024-04-12T05:37:00.668 INFO:tasks.workunit.client.0.smithi032.stdout:2/145: unlink f2 0 2024-04-12T05:37:00.668 INFO:tasks.workunit.client.0.smithi032.stdout:2/146: readlink dd/d19/l26 0 2024-04-12T05:37:00.674 INFO:tasks.workunit.client.0.smithi032.stdout:3/105: rename f7 to d12/f1b 0 2024-04-12T05:37:00.675 INFO:tasks.workunit.client.0.smithi032.stdout:2/147: creat dd/d1b/d1c/f29 x:0 0 0 2024-04-12T05:37:00.677 INFO:tasks.workunit.client.0.smithi032.stdout:2/148: write dd/d1b/d1c/f21 [797470,62487] 0 2024-04-12T05:37:00.678 INFO:tasks.workunit.client.0.smithi032.stdout:9/146: getdents d9/d15/d1a/d24/d27 0 2024-04-12T05:37:00.680 INFO:tasks.workunit.client.0.smithi032.stdout:4/150: dwrite d4/d13/d19/f34 [0,4194304] 0 2024-04-12T05:37:00.680 INFO:tasks.workunit.client.0.smithi032.stdout:9/147: chown d9/d15/d2c/c33 1 1 2024-04-12T05:37:00.682 INFO:tasks.workunit.client.0.smithi032.stdout:3/106: mknod d12/d19/c1c 0 2024-04-12T05:37:00.682 INFO:tasks.workunit.client.0.smithi032.stdout:5/202: creat d2/df/f3f x:0 0 0 2024-04-12T05:37:00.711 INFO:tasks.workunit.client.0.smithi032.stdout:9/148: truncate d9/d15/d2c/f10 4494924 0 2024-04-12T05:37:00.715 INFO:tasks.workunit.client.0.smithi032.stdout:4/151: dwrite d4/d13/d19/f30 [0,4194304] 0 2024-04-12T05:37:00.719 INFO:tasks.workunit.client.0.smithi032.stdout:3/107: dwrite d12/f13 [0,4194304] 0 2024-04-12T05:37:00.724 INFO:tasks.workunit.client.0.smithi032.stdout:3/108: write d12/f1b [3420078,41288] 0 2024-04-12T05:37:00.725 INFO:tasks.workunit.client.0.smithi032.stdout:3/109: read f10 [1056889,26101] 0 2024-04-12T05:37:00.725 INFO:tasks.workunit.client.0.smithi032.stdout:5/203: truncate d2/df/d25/d1e/f23 680303 0 2024-04-12T05:37:00.727 INFO:tasks.workunit.client.0.smithi032.stdout:5/204: read d2/df/f1c [5859408,82985] 0 2024-04-12T05:37:00.728 INFO:tasks.workunit.client.0.smithi032.stdout:5/205: chown d2/f4 0 1 2024-04-12T05:37:00.728 INFO:tasks.workunit.client.0.smithi032.stdout:5/206: readlink d2/df/d25/l20 0 2024-04-12T05:37:00.732 INFO:tasks.workunit.client.0.smithi032.stdout:5/207: chown d2/c37 121127 1 2024-04-12T05:37:00.736 INFO:tasks.workunit.client.0.smithi032.stdout:3/110: link d12/l15 d12/d19/l1d 0 2024-04-12T05:37:00.739 INFO:tasks.workunit.client.0.smithi032.stdout:3/111: dread d12/f1b [0,4194304] 0 2024-04-12T05:37:00.741 INFO:tasks.workunit.client.0.smithi032.stdout:3/112: dread f10 [0,4194304] 0 2024-04-12T05:37:00.743 INFO:tasks.workunit.client.0.smithi032.stdout:3/113: creat d12/d19/f1e x:0 0 0 2024-04-12T05:37:00.751 INFO:tasks.workunit.client.0.smithi032.stdout:8/116: truncate d0/db/dd/d11/f1f 420097 0 2024-04-12T05:37:00.751 INFO:tasks.workunit.client.0.smithi032.stdout:8/117: fdatasync d0/f24 0 2024-04-12T05:37:00.752 INFO:tasks.workunit.client.0.smithi032.stdout:8/118: write d0/f3 [3112712,56310] 0 2024-04-12T05:37:00.755 INFO:tasks.workunit.client.0.smithi032.stdout:8/119: mknod d0/c25 0 2024-04-12T05:37:00.756 INFO:tasks.workunit.client.0.smithi032.stdout:8/120: write d0/fe [889663,29218] 0 2024-04-12T05:37:00.763 INFO:tasks.workunit.client.0.smithi032.stdout:3/114: write d12/f13 [4892555,74093] 0 2024-04-12T05:37:00.765 INFO:tasks.workunit.client.0.smithi032.stdout:0/223: dwrite d2/dc/fd [0,4194304] 0 2024-04-12T05:37:00.765 INFO:tasks.workunit.client.0.smithi032.stdout:4/152: write d4/d13/f33 [382630,19036] 0 2024-04-12T05:37:00.767 INFO:tasks.workunit.client.0.smithi032.stdout:4/153: chown d4/cb 1 1 2024-04-12T05:37:00.768 INFO:tasks.workunit.client.0.smithi032.stdout:3/115: mkdir d12/d1f 0 2024-04-12T05:37:00.769 INFO:tasks.workunit.client.0.smithi032.stdout:5/208: truncate d2/df/d31/f33 3357187 0 2024-04-12T05:37:00.775 INFO:tasks.workunit.client.0.smithi032.stdout:0/224: creat d2/dc/d3e/f46 x:0 0 0 2024-04-12T05:37:00.776 INFO:tasks.workunit.client.0.smithi032.stdout:0/225: fsync d2/f30 0 2024-04-12T05:37:00.776 INFO:tasks.workunit.client.0.smithi032.stdout:5/209: truncate d2/d17/f21 928677 0 2024-04-12T05:37:00.778 INFO:tasks.workunit.client.0.smithi032.stdout:5/210: write d2/df/d25/d1e/f2f [3946140,92786] 0 2024-04-12T05:37:00.779 INFO:tasks.workunit.client.0.smithi032.stdout:3/116: rename d12/d19/l1d to d12/d19/l20 0 2024-04-12T05:37:00.779 INFO:tasks.workunit.client.0.smithi032.stdout:5/211: chown d2/f24 0 1 2024-04-12T05:37:00.780 INFO:tasks.workunit.client.0.smithi032.stdout:5/212: stat d2/df/d25/f8 0 2024-04-12T05:37:00.782 INFO:tasks.workunit.client.0.smithi032.stdout:3/117: creat d12/f21 x:0 0 0 2024-04-12T05:37:00.783 INFO:tasks.workunit.client.0.smithi032.stdout:1/89: sync 2024-04-12T05:37:00.786 INFO:tasks.workunit.client.0.smithi032.stdout:4/154: write d4/d13/fd [1348369,77960] 0 2024-04-12T05:37:00.789 INFO:tasks.workunit.client.0.smithi032.stdout:5/213: dread d2/df/f1d [0,4194304] 0 2024-04-12T05:37:00.790 INFO:tasks.workunit.client.0.smithi032.stdout:1/90: symlink d5/de/l18 0 2024-04-12T05:37:00.792 INFO:tasks.workunit.client.0.smithi032.stdout:8/121: dwrite d0/f8 [0,4194304] 0 2024-04-12T05:37:00.794 INFO:tasks.workunit.client.0.smithi032.stdout:9/149: dread d9/d15/d2c/f10 [0,4194304] 0 2024-04-12T05:37:00.795 INFO:tasks.workunit.client.0.smithi032.stdout:1/91: creat d5/de/f19 x:0 0 0 2024-04-12T05:37:00.799 INFO:tasks.workunit.client.0.smithi032.stdout:3/118: dwrite d12/f13 [4194304,4194304] 0 2024-04-12T05:37:00.807 INFO:tasks.workunit.client.0.smithi032.stdout:3/119: fdatasync f10 0 2024-04-12T05:37:00.807 INFO:tasks.workunit.client.0.smithi032.stdout:5/214: dwrite d2/f24 [0,4194304] 0 2024-04-12T05:37:00.808 INFO:tasks.workunit.client.0.smithi032.stdout:4/155: dread d4/f1d [0,4194304] 0 2024-04-12T05:37:00.810 INFO:tasks.workunit.client.0.smithi032.stdout:3/120: mknod d12/c22 0 2024-04-12T05:37:00.812 INFO:tasks.workunit.client.0.smithi032.stdout:5/215: dread - d2/df/d25/f29 zero size 2024-04-12T05:37:00.813 INFO:tasks.workunit.client.0.smithi032.stdout:8/122: dread d0/db/f18 [0,4194304] 0 2024-04-12T05:37:00.813 INFO:tasks.workunit.client.0.smithi032.stdout:1/92: getdents d5/de 0 2024-04-12T05:37:00.816 INFO:tasks.workunit.client.0.smithi032.stdout:9/150: dwrite d9/f34 [0,4194304] 0 2024-04-12T05:37:00.816 INFO:tasks.workunit.client.0.smithi032.stdout:1/93: stat d5/de/c15 0 2024-04-12T05:37:00.817 INFO:tasks.workunit.client.0.smithi032.stdout:9/151: stat d9/f13 0 2024-04-12T05:37:00.818 INFO:tasks.workunit.client.0.smithi032.stdout:1/94: stat d5/f13 0 2024-04-12T05:37:00.818 INFO:tasks.workunit.client.0.smithi032.stdout:9/152: chown d9/d1f/d2f 555 1 2024-04-12T05:37:00.820 INFO:tasks.workunit.client.0.smithi032.stdout:4/156: mknod d4/d13/d19/d28/d32/c36 0 2024-04-12T05:37:00.820 INFO:tasks.workunit.client.0.smithi032.stdout:4/157: fdatasync d4/d13/d19/f30 0 2024-04-12T05:37:00.822 INFO:tasks.workunit.client.0.smithi032.stdout:4/158: write d4/f35 [741311,39773] 0 2024-04-12T05:37:00.822 INFO:tasks.workunit.client.0.smithi032.stdout:5/216: rename d2/df/d25/f35 to d2/df/d25/d1e/f40 0 2024-04-12T05:37:00.824 INFO:tasks.workunit.client.0.smithi032.stdout:5/217: write d2/df/d25/f2c [287473,73318] 0 2024-04-12T05:37:00.826 INFO:tasks.workunit.client.0.smithi032.stdout:5/218: write d2/df/f3f [975614,8213] 0 2024-04-12T05:37:00.826 INFO:tasks.workunit.client.0.smithi032.stdout:4/159: mknod d4/c37 0 2024-04-12T05:37:00.827 INFO:tasks.workunit.client.0.smithi032.stdout:3/121: getdents d12/d19 0 2024-04-12T05:37:00.829 INFO:tasks.workunit.client.0.smithi032.stdout:4/160: chown d4/d13/d19/c17 1179552022 1 2024-04-12T05:37:00.829 INFO:tasks.workunit.client.0.smithi032.stdout:9/153: creat d9/d32/f37 x:0 0 0 2024-04-12T05:37:00.830 INFO:tasks.workunit.client.0.smithi032.stdout:5/219: mknod d2/df/d25/d1e/c41 0 2024-04-12T05:37:00.834 INFO:tasks.workunit.client.0.smithi032.stdout:1/95: truncate d5/fb 2052082 0 2024-04-12T05:37:00.835 INFO:tasks.workunit.client.0.smithi032.stdout:5/220: mkdir d2/df/d31/d42 0 2024-04-12T05:37:00.837 INFO:tasks.workunit.client.0.smithi032.stdout:1/96: chown d5/de/d17 15 1 2024-04-12T05:37:00.839 INFO:tasks.workunit.client.0.smithi032.stdout:9/154: rename d9/d15/d1a/d1d/l2e to d9/d15/l38 0 2024-04-12T05:37:00.840 INFO:tasks.workunit.client.0.smithi032.stdout:5/221: fsync d2/df/d25/f2c 0 2024-04-12T05:37:00.842 INFO:tasks.workunit.client.0.smithi032.stdout:3/122: dread d12/f13 [0,4194304] 0 2024-04-12T05:37:00.842 INFO:tasks.workunit.client.0.smithi032.stdout:4/161: dwrite d4/f31 [0,4194304] 0 2024-04-12T05:37:00.843 INFO:tasks.workunit.client.0.smithi032.stdout:5/222: read d2/f24 [1543334,85954] 0 2024-04-12T05:37:00.845 INFO:tasks.workunit.client.0.smithi032.stdout:9/155: creat d9/d1f/f39 x:0 0 0 2024-04-12T05:37:00.846 INFO:tasks.workunit.client.0.smithi032.stdout:8/123: dwrite d0/db/dd/d11/f1f [0,4194304] 0 2024-04-12T05:37:00.847 INFO:tasks.workunit.client.0.smithi032.stdout:6/127: sync 2024-04-12T05:37:00.847 INFO:tasks.workunit.client.0.smithi032.stdout:7/134: sync 2024-04-12T05:37:00.847 INFO:tasks.workunit.client.0.smithi032.stdout:2/149: sync 2024-04-12T05:37:00.848 INFO:tasks.workunit.client.0.smithi032.stdout:8/124: readlink d0/db/dd/l10 0 2024-04-12T05:37:00.853 INFO:tasks.workunit.client.0.smithi032.stdout:5/223: truncate d2/d17/f39 551930 0 2024-04-12T05:37:00.853 INFO:tasks.workunit.client.0.smithi032.stdout:2/150: dread - dd/d19/f22 zero size 2024-04-12T05:37:00.856 INFO:tasks.workunit.client.0.smithi032.stdout:5/224: fdatasync d2/df/f32 0 2024-04-12T05:37:00.858 INFO:tasks.workunit.client.0.smithi032.stdout:8/125: creat d0/db/dd/d11/f26 x:0 0 0 2024-04-12T05:37:00.858 INFO:tasks.workunit.client.0.smithi032.stdout:7/135: dread - d0/fc zero size 2024-04-12T05:37:00.860 INFO:tasks.workunit.client.0.smithi032.stdout:1/97: link d5/de/l18 d5/de/d17/l1a 0 2024-04-12T05:37:00.860 INFO:tasks.workunit.client.0.smithi032.stdout:2/151: mknod dd/d19/c2a 0 2024-04-12T05:37:00.861 INFO:tasks.workunit.client.0.smithi032.stdout:9/156: mknod d9/c3a 0 2024-04-12T05:37:00.862 INFO:tasks.workunit.client.0.smithi032.stdout:5/225: read d2/f6 [3246588,75065] 0 2024-04-12T05:37:00.862 INFO:tasks.workunit.client.0.smithi032.stdout:4/162: dwrite d4/d13/f1f [0,4194304] 0 2024-04-12T05:37:00.862 INFO:tasks.workunit.client.0.smithi032.stdout:3/123: link ce d12/d19/c23 0 2024-04-12T05:37:00.862 INFO:tasks.workunit.client.0.smithi032.stdout:1/98: stat d5/de/f11 0 2024-04-12T05:37:00.863 INFO:tasks.workunit.client.0.smithi032.stdout:2/152: fdatasync dd/fe 0 2024-04-12T05:37:00.864 INFO:tasks.workunit.client.0.smithi032.stdout:6/128: symlink d4/d14/d25/l2b 0 2024-04-12T05:37:00.867 INFO:tasks.workunit.client.0.smithi032.stdout:7/136: symlink d0/d1c/l2c 0 2024-04-12T05:37:00.870 INFO:tasks.workunit.client.0.smithi032.stdout:6/129: fdatasync d4/d14/f26 0 2024-04-12T05:37:00.871 INFO:tasks.workunit.client.0.smithi032.stdout:9/157: truncate d9/fa 2227505 0 2024-04-12T05:37:00.873 INFO:tasks.workunit.client.0.smithi032.stdout:7/137: symlink d0/d13/d25/l2d 0 2024-04-12T05:37:00.875 INFO:tasks.workunit.client.0.smithi032.stdout:2/153: dread dd/ff [0,4194304] 0 2024-04-12T05:37:00.875 INFO:tasks.workunit.client.0.smithi032.stdout:3/124: getdents d12/d1f 0 2024-04-12T05:37:00.878 INFO:tasks.workunit.client.0.smithi032.stdout:7/138: chown d0/de/df/d29 6 1 2024-04-12T05:37:00.879 INFO:tasks.workunit.client.0.smithi032.stdout:1/99: mkdir d5/de/d17/d1b 0 2024-04-12T05:37:00.879 INFO:tasks.workunit.client.0.smithi032.stdout:9/158: dread d9/d15/d2c/f20 [0,4194304] 0 2024-04-12T05:37:00.880 INFO:tasks.workunit.client.0.smithi032.stdout:2/154: write dd/d19/d20/f28 [790983,10059] 0 2024-04-12T05:37:00.880 INFO:tasks.workunit.client.0.smithi032.stdout:8/126: dwrite d0/db/dd/d11/f1c [0,4194304] 0 2024-04-12T05:37:00.881 INFO:tasks.workunit.client.0.smithi032.stdout:3/125: mkdir d12/d24 0 2024-04-12T05:37:00.882 INFO:tasks.workunit.client.0.smithi032.stdout:3/126: chown d12 16187822 1 2024-04-12T05:37:00.883 INFO:tasks.workunit.client.0.smithi032.stdout:1/100: dread - d5/de/f19 zero size 2024-04-12T05:37:00.885 INFO:tasks.workunit.client.0.smithi032.stdout:3/127: chown d12/d1f 3 1 2024-04-12T05:37:00.887 INFO:tasks.workunit.client.0.smithi032.stdout:9/159: symlink d9/d15/d1a/d24/l3b 0 2024-04-12T05:37:00.887 INFO:tasks.workunit.client.0.smithi032.stdout:5/226: dwrite d2/df/f1c [0,4194304] 0 2024-04-12T05:37:00.887 INFO:tasks.workunit.client.0.smithi032.stdout:7/139: link d0/d13/l18 d0/d13/d21/l2e 0 2024-04-12T05:37:00.887 INFO:tasks.workunit.client.0.smithi032.stdout:3/128: dread - d12/d19/f1e zero size 2024-04-12T05:37:00.889 INFO:tasks.workunit.client.0.smithi032.stdout:8/127: rename d0/f24 to d0/db/dd/f27 0 2024-04-12T05:37:00.889 INFO:tasks.workunit.client.0.smithi032.stdout:7/140: dread - d0/dd/f2a zero size 2024-04-12T05:37:00.892 INFO:tasks.workunit.client.0.smithi032.stdout:3/129: symlink d12/d19/l25 0 2024-04-12T05:37:00.893 INFO:tasks.workunit.client.0.smithi032.stdout:1/101: mknod d5/de/d17/d1b/c1c 0 2024-04-12T05:37:00.894 INFO:tasks.workunit.client.0.smithi032.stdout:7/141: dread d0/de/df/f14 [0,4194304] 0 2024-04-12T05:37:00.896 INFO:tasks.workunit.client.0.smithi032.stdout:8/128: creat d0/db/dd/d11/f28 x:0 0 0 2024-04-12T05:37:00.898 INFO:tasks.workunit.client.0.smithi032.stdout:3/130: creat d12/d1f/f26 x:0 0 0 2024-04-12T05:37:00.899 INFO:tasks.workunit.client.0.smithi032.stdout:8/129: fdatasync d0/db/f20 0 2024-04-12T05:37:00.900 INFO:tasks.workunit.client.0.smithi032.stdout:5/227: dwrite d2/f6 [0,4194304] 0 2024-04-12T05:37:00.903 INFO:tasks.workunit.client.0.smithi032.stdout:8/130: chown d0/db/dd/d11/l12 26946832 1 2024-04-12T05:37:00.905 INFO:tasks.workunit.client.0.smithi032.stdout:5/228: unlink d2/f6 0 2024-04-12T05:37:00.907 INFO:tasks.workunit.client.0.smithi032.stdout:7/142: dwrite d0/f8 [0,4194304] 0 2024-04-12T05:37:00.908 INFO:tasks.workunit.client.0.smithi032.stdout:7/143: chown d0/fc 619152 1 2024-04-12T05:37:00.908 INFO:tasks.workunit.client.0.smithi032.stdout:8/131: fsync d0/db/fc 0 2024-04-12T05:37:00.909 INFO:tasks.workunit.client.0.smithi032.stdout:7/144: write d0/d13/f1a [1021058,124683] 0 2024-04-12T05:37:00.909 INFO:tasks.workunit.client.0.smithi032.stdout:9/160: getdents d9/d15/d1a/d24 0 2024-04-12T05:37:00.912 INFO:tasks.workunit.client.0.smithi032.stdout:3/131: link d12/l18 d12/l27 0 2024-04-12T05:37:00.913 INFO:tasks.workunit.client.0.smithi032.stdout:3/132: rename d12/d1f to d12/d1f/d28 22 2024-04-12T05:37:00.915 INFO:tasks.workunit.client.0.smithi032.stdout:5/229: mkdir d2/df/d25/d43 0 2024-04-12T05:37:00.952 INFO:tasks.workunit.client.0.smithi032.stdout:0/226: sync 2024-04-12T05:37:00.965 INFO:tasks.workunit.client.0.smithi032.stdout:3/133: write d12/d19/f1e [30857,39767] 0 2024-04-12T05:37:00.967 INFO:tasks.workunit.client.0.smithi032.stdout:7/145: mknod d0/de/df/c2f 0 2024-04-12T05:37:00.967 INFO:tasks.workunit.client.0.smithi032.stdout:3/134: read f10 [1229622,32656] 0 2024-04-12T05:37:00.969 INFO:tasks.workunit.client.0.smithi032.stdout:8/132: dread d0/db/dd/d11/f1f [0,4194304] 0 2024-04-12T05:37:00.969 INFO:tasks.workunit.client.0.smithi032.stdout:0/227: creat d2/d29/f47 x:0 0 0 2024-04-12T05:37:00.970 INFO:tasks.workunit.client.0.smithi032.stdout:3/135: readlink d12/l1a 0 2024-04-12T05:37:00.998 INFO:tasks.workunit.client.0.smithi032.stdout:7/146: dread d0/de/f23 [0,4194304] 0 2024-04-12T05:37:01.000 INFO:tasks.workunit.client.0.smithi032.stdout:0/228: mkdir d2/d8/d48 0 2024-04-12T05:37:01.001 INFO:tasks.workunit.client.0.smithi032.stdout:8/133: mkdir d0/db/dd/d11/d1b/d22/d29 0 2024-04-12T05:37:01.001 INFO:tasks.workunit.client.0.smithi032.stdout:1/102: dread d5/ff [0,4194304] 0 2024-04-12T05:37:01.003 INFO:tasks.workunit.client.0.smithi032.stdout:7/147: mkdir d0/d13/d21/d30 0 2024-04-12T05:37:01.003 INFO:tasks.workunit.client.0.smithi032.stdout:3/136: link d12/l1a d12/l29 0 2024-04-12T05:37:01.005 INFO:tasks.workunit.client.0.smithi032.stdout:1/103: read d5/fc [356106,81072] 0 2024-04-12T05:37:01.006 INFO:tasks.workunit.client.0.smithi032.stdout:5/230: dwrite d2/df/d25/d1e/f2f [0,4194304] 0 2024-04-12T05:37:01.006 INFO:tasks.workunit.client.0.smithi032.stdout:8/134: creat d0/db/dd/d11/f2a x:0 0 0 2024-04-12T05:37:01.006 INFO:tasks.workunit.client.0.smithi032.stdout:3/137: fsync d12/d1f/f26 0 2024-04-12T05:37:01.008 INFO:tasks.workunit.client.0.smithi032.stdout:9/161: dread d9/fa [0,4194304] 0 2024-04-12T05:37:01.010 INFO:tasks.workunit.client.0.smithi032.stdout:0/229: write d2/d8/d17/f2b [1648555,45803] 0 2024-04-12T05:37:01.011 INFO:tasks.workunit.client.0.smithi032.stdout:5/231: write d2/df/d25/f29 [326379,98516] 0 2024-04-12T05:37:01.011 INFO:tasks.workunit.client.0.smithi032.stdout:1/104: creat d5/de/d17/d1b/f1d x:0 0 0 2024-04-12T05:37:01.012 INFO:tasks.workunit.client.0.smithi032.stdout:9/162: symlink d9/l3c 0 2024-04-12T05:37:01.014 INFO:tasks.workunit.client.0.smithi032.stdout:7/148: dread d0/de/f15 [0,4194304] 0 2024-04-12T05:37:01.016 INFO:tasks.workunit.client.0.smithi032.stdout:3/138: dread d12/f1b [0,4194304] 0 2024-04-12T05:37:01.017 INFO:tasks.workunit.client.0.smithi032.stdout:8/135: rename d0/db/dd/f21 to d0/db/dd/d11/d1b/d22/f2b 0 2024-04-12T05:37:01.017 INFO:tasks.workunit.client.0.smithi032.stdout:4/163: sync 2024-04-12T05:37:01.018 INFO:tasks.workunit.client.0.smithi032.stdout:1/105: write d5/fc [5910736,45603] 0 2024-04-12T05:37:01.018 INFO:tasks.workunit.client.0.smithi032.stdout:0/230: symlink d2/dc/d41/l49 0 2024-04-12T05:37:01.021 INFO:tasks.workunit.client.0.smithi032.stdout:1/106: dread - d5/de/f19 zero size 2024-04-12T05:37:01.022 INFO:tasks.workunit.client.0.smithi032.stdout:3/139: write d12/f13 [1757635,85905] 0 2024-04-12T05:37:01.022 INFO:tasks.workunit.client.0.smithi032.stdout:7/149: getdents d0/d13/d21/d30 0 2024-04-12T05:37:01.022 INFO:tasks.workunit.client.0.smithi032.stdout:4/164: chown d4/d13/d19/c24 2788036 1 2024-04-12T05:37:01.023 INFO:tasks.workunit.client.0.smithi032.stdout:0/231: creat d2/d29/f4a x:0 0 0 2024-04-12T05:37:01.023 INFO:tasks.workunit.client.0.smithi032.stdout:5/232: read d2/df/d31/f33 [1480512,54572] 0 2024-04-12T05:37:01.024 INFO:tasks.workunit.client.0.smithi032.stdout:3/140: write d12/f13 [5591808,49296] 0 2024-04-12T05:37:01.024 INFO:tasks.workunit.client.0.smithi032.stdout:8/136: rename d0/db/dd/d11/f1c to d0/db/dd/d11/f2c 0 2024-04-12T05:37:01.025 INFO:tasks.workunit.client.0.smithi032.stdout:0/232: dread - d2/d29/f47 zero size 2024-04-12T05:37:01.025 INFO:tasks.workunit.client.0.smithi032.stdout:7/150: chown d0/de/f23 67064 1 2024-04-12T05:37:01.026 INFO:tasks.workunit.client.0.smithi032.stdout:4/165: write d4/d13/d19/f2a [773512,130186] 0 2024-04-12T05:37:01.027 INFO:tasks.workunit.client.0.smithi032.stdout:4/166: fsync d4/f35 0 2024-04-12T05:37:01.029 INFO:tasks.workunit.client.0.smithi032.stdout:7/151: write d0/de/df/f1e [2025386,80868] 0 2024-04-12T05:37:01.029 INFO:tasks.workunit.client.0.smithi032.stdout:5/233: rmdir d2/df/d25 39 2024-04-12T05:37:01.031 INFO:tasks.workunit.client.0.smithi032.stdout:4/167: truncate d4/f35 1417765 0 2024-04-12T05:37:01.031 INFO:tasks.workunit.client.0.smithi032.stdout:1/107: mkdir d5/d1e 0 2024-04-12T05:37:01.031 INFO:tasks.workunit.client.0.smithi032.stdout:4/168: fdatasync d4/f18 0 2024-04-12T05:37:01.032 INFO:tasks.workunit.client.0.smithi032.stdout:9/163: truncate f5 378603 0 2024-04-12T05:37:01.037 INFO:tasks.workunit.client.0.smithi032.stdout:4/169: dread d4/f35 [0,4194304] 0 2024-04-12T05:37:01.039 INFO:tasks.workunit.client.0.smithi032.stdout:1/108: dwrite d5/de/f19 [0,4194304] 0 2024-04-12T05:37:01.039 INFO:tasks.workunit.client.0.smithi032.stdout:4/170: truncate d4/d13/ff 700710 0 2024-04-12T05:37:01.041 INFO:tasks.workunit.client.0.smithi032.stdout:7/152: unlink d0/de/f23 0 2024-04-12T05:37:01.043 INFO:tasks.workunit.client.0.smithi032.stdout:9/164: creat d9/d15/d1a/f3d x:0 0 0 2024-04-12T05:37:01.044 INFO:tasks.workunit.client.0.smithi032.stdout:3/141: link d12/d1f/f26 d12/f2a 0 2024-04-12T05:37:01.046 INFO:tasks.workunit.client.0.smithi032.stdout:8/137: symlink d0/db/dd/d11/d1b/d1e/l2d 0 2024-04-12T05:37:01.047 INFO:tasks.workunit.client.0.smithi032.stdout:9/165: dread d9/f34 [0,4194304] 0 2024-04-12T05:37:01.048 INFO:tasks.workunit.client.0.smithi032.stdout:5/234: fsync d2/f34 0 2024-04-12T05:37:01.050 INFO:tasks.workunit.client.0.smithi032.stdout:0/233: link d2/dc/d3e/f46 d2/d8/df/d24/f4b 0 2024-04-12T05:37:01.050 INFO:tasks.workunit.client.0.smithi032.stdout:4/171: creat d4/f38 x:0 0 0 2024-04-12T05:37:01.053 INFO:tasks.workunit.client.0.smithi032.stdout:8/138: fsync d0/db/dd/d11/d1b/d22/f2b 0 2024-04-12T05:37:01.054 INFO:tasks.workunit.client.0.smithi032.stdout:6/130: sync 2024-04-12T05:37:01.057 INFO:tasks.workunit.client.0.smithi032.stdout:7/153: mkdir d0/d13/d21/d30/d31 0 2024-04-12T05:37:01.060 INFO:tasks.workunit.client.0.smithi032.stdout:3/142: creat d12/d24/f2b x:0 0 0 2024-04-12T05:37:01.061 INFO:tasks.workunit.client.0.smithi032.stdout:9/166: creat d9/d1f/d2f/f3e x:0 0 0 2024-04-12T05:37:01.062 INFO:tasks.workunit.client.0.smithi032.stdout:4/172: dwrite d4/d13/d19/f2a [0,4194304] 0 2024-04-12T05:37:01.064 INFO:tasks.workunit.client.0.smithi032.stdout:7/154: symlink d0/de/df/d29/l32 0 2024-04-12T05:37:01.064 INFO:tasks.workunit.client.0.smithi032.stdout:8/139: dread d0/db/f19 [0,4194304] 0 2024-04-12T05:37:01.065 INFO:tasks.workunit.client.0.smithi032.stdout:8/140: stat d0/c1 0 2024-04-12T05:37:01.065 INFO:tasks.workunit.client.0.smithi032.stdout:3/143: creat d12/d24/f2c x:0 0 0 2024-04-12T05:37:01.066 INFO:tasks.workunit.client.0.smithi032.stdout:1/109: write d5/fb [2692486,57286] 0 2024-04-12T05:37:01.066 INFO:tasks.workunit.client.0.smithi032.stdout:0/234: write d2/d28/f2e [412030,94524] 0 2024-04-12T05:37:01.067 INFO:tasks.workunit.client.0.smithi032.stdout:6/131: creat d4/f2c x:0 0 0 2024-04-12T05:37:01.068 INFO:tasks.workunit.client.0.smithi032.stdout:7/155: symlink d0/de/l33 0 2024-04-12T05:37:01.069 INFO:tasks.workunit.client.0.smithi032.stdout:8/141: write d0/db/dd/d11/f13 [158865,5380] 0 2024-04-12T05:37:01.069 INFO:tasks.workunit.client.0.smithi032.stdout:9/167: chown d9/d15/d1a/c1b 10061 1 2024-04-12T05:37:01.069 INFO:tasks.workunit.client.0.smithi032.stdout:1/110: stat d5/de/d17 0 2024-04-12T05:37:01.070 INFO:tasks.workunit.client.0.smithi032.stdout:3/144: mkdir d12/d1f/d2d 0 2024-04-12T05:37:01.070 INFO:tasks.workunit.client.0.smithi032.stdout:5/235: dread d2/df/d31/f33 [0,4194304] 0 2024-04-12T05:37:01.072 INFO:tasks.workunit.client.0.smithi032.stdout:0/235: write d2/dc/fd [4900335,108138] 0 2024-04-12T05:37:01.072 INFO:tasks.workunit.client.0.smithi032.stdout:4/173: truncate d4/f18 2384001 0 2024-04-12T05:37:01.073 INFO:tasks.workunit.client.0.smithi032.stdout:3/145: chown d12/d1f/d2d 1 1 2024-04-12T05:37:01.075 INFO:tasks.workunit.client.0.smithi032.stdout:6/132: mknod d4/d21/d27/c2d 0 2024-04-12T05:37:01.076 INFO:tasks.workunit.client.0.smithi032.stdout:7/156: mkdir d0/d34 0 2024-04-12T05:37:01.076 INFO:tasks.workunit.client.0.smithi032.stdout:1/111: write d5/de/f10 [2058188,98925] 0 2024-04-12T05:37:01.078 INFO:tasks.workunit.client.0.smithi032.stdout:5/236: creat d2/dd/f44 x:0 0 0 2024-04-12T05:37:01.080 INFO:tasks.workunit.client.0.smithi032.stdout:8/142: write d0/db/dd/d11/f28 [227843,108965] 0 2024-04-12T05:37:01.080 INFO:tasks.workunit.client.0.smithi032.stdout:3/146: truncate d12/d19/f1e 225826 0 2024-04-12T05:37:01.080 INFO:tasks.workunit.client.0.smithi032.stdout:0/236: rename d2/l3 to d2/d29/l4c 0 2024-04-12T05:37:01.081 INFO:tasks.workunit.client.0.smithi032.stdout:3/147: chown d12/d24/f2b 5 1 2024-04-12T05:37:01.082 INFO:tasks.workunit.client.0.smithi032.stdout:7/157: mknod d0/d13/d21/c35 0 2024-04-12T05:37:01.087 INFO:tasks.workunit.client.0.smithi032.stdout:3/148: dread - d12/d24/f2c zero size 2024-04-12T05:37:01.087 INFO:tasks.workunit.client.0.smithi032.stdout:1/112: creat d5/de/f1f x:0 0 0 2024-04-12T05:37:01.087 INFO:tasks.workunit.client.0.smithi032.stdout:8/143: stat d0/ca 0 2024-04-12T05:37:01.087 INFO:tasks.workunit.client.0.smithi032.stdout:5/237: dread d2/df/d25/d1e/f23 [0,4194304] 0 2024-04-12T05:37:01.088 INFO:tasks.workunit.client.0.smithi032.stdout:2/155: sync 2024-04-12T05:37:01.091 INFO:tasks.workunit.client.0.smithi032.stdout:9/168: write d9/d15/d2c/ff [416906,28348] 0 2024-04-12T05:37:01.091 INFO:tasks.workunit.client.0.smithi032.stdout:0/237: dwrite d2/d8/f33 [0,4194304] 0 2024-04-12T05:37:01.091 INFO:tasks.workunit.client.0.smithi032.stdout:1/113: readlink d5/de/l18 0 2024-04-12T05:37:01.094 INFO:tasks.workunit.client.0.smithi032.stdout:0/238: chown d2/f6 2065295 1 2024-04-12T05:37:01.095 INFO:tasks.workunit.client.0.smithi032.stdout:6/133: getdents d4/d21/d27 0 2024-04-12T05:37:01.095 INFO:tasks.workunit.client.0.smithi032.stdout:8/144: dread d0/db/dd/d11/f13 [0,4194304] 0 2024-04-12T05:37:01.096 INFO:tasks.workunit.client.0.smithi032.stdout:1/114: write d5/de/f19 [22302,110200] 0 2024-04-12T05:37:01.099 INFO:tasks.workunit.client.0.smithi032.stdout:9/169: creat d9/d15/d1a/d24/f3f x:0 0 0 2024-04-12T05:37:01.101 INFO:tasks.workunit.client.0.smithi032.stdout:5/238: rmdir d2/df/d31/d42 0 2024-04-12T05:37:01.103 INFO:tasks.workunit.client.0.smithi032.stdout:6/134: chown d4/d14/l17 252 1 2024-04-12T05:37:01.107 INFO:tasks.workunit.client.0.smithi032.stdout:9/170: write d9/d15/d2c/f23 [1755943,56545] 0 2024-04-12T05:37:01.107 INFO:tasks.workunit.client.0.smithi032.stdout:1/115: creat d5/de/d17/f20 x:0 0 0 2024-04-12T05:37:01.108 INFO:tasks.workunit.client.0.smithi032.stdout:5/239: read d2/df/f1c [6085342,28328] 0 2024-04-12T05:37:01.109 INFO:tasks.workunit.client.0.smithi032.stdout:7/158: dwrite d0/fc [0,4194304] 0 2024-04-12T05:37:01.111 INFO:tasks.workunit.client.0.smithi032.stdout:0/239: getdents d2 0 2024-04-12T05:37:01.112 INFO:tasks.workunit.client.0.smithi032.stdout:2/156: dread dd/f14 [0,4194304] 0 2024-04-12T05:37:01.112 INFO:tasks.workunit.client.0.smithi032.stdout:6/135: mkdir d4/d21/d2e 0 2024-04-12T05:37:01.114 INFO:tasks.workunit.client.0.smithi032.stdout:1/116: write d5/f13 [4175936,64994] 0 2024-04-12T05:37:01.114 INFO:tasks.workunit.client.0.smithi032.stdout:5/240: chown d2/f34 126842 1 2024-04-12T05:37:01.115 INFO:tasks.workunit.client.0.smithi032.stdout:4/174: dwrite d4/f35 [0,4194304] 0 2024-04-12T05:37:01.116 INFO:tasks.workunit.client.0.smithi032.stdout:0/240: readlink d2/d29/l4c 0 2024-04-12T05:37:01.118 INFO:tasks.workunit.client.0.smithi032.stdout:7/159: creat d0/dd/f36 x:0 0 0 2024-04-12T05:37:01.120 INFO:tasks.workunit.client.0.smithi032.stdout:2/157: symlink dd/d19/l2b 0 2024-04-12T05:37:01.121 INFO:tasks.workunit.client.0.smithi032.stdout:6/136: creat d4/d21/f2f x:0 0 0 2024-04-12T05:37:01.122 INFO:tasks.workunit.client.0.smithi032.stdout:6/137: fdatasync d4/d21/f2f 0 2024-04-12T05:37:01.123 INFO:tasks.workunit.client.0.smithi032.stdout:8/145: truncate d0/f3 3246260 0 2024-04-12T05:37:01.126 INFO:tasks.workunit.client.0.smithi032.stdout:8/146: stat d0/db/dd/d11/f28 0 2024-04-12T05:37:01.127 INFO:tasks.workunit.client.0.smithi032.stdout:1/117: dwrite d5/fc [0,4194304] 0 2024-04-12T05:37:01.128 INFO:tasks.workunit.client.0.smithi032.stdout:4/175: creat d4/d13/d19/f39 x:0 0 0 2024-04-12T05:37:01.133 INFO:tasks.workunit.client.0.smithi032.stdout:0/241: getdents d2/d8/d17 0 2024-04-12T05:37:01.136 INFO:tasks.workunit.client.0.smithi032.stdout:4/176: symlink d4/d13/d19/l3a 0 2024-04-12T05:37:01.137 INFO:tasks.workunit.client.0.smithi032.stdout:1/118: creat d5/de/d17/d1b/f21 x:0 0 0 2024-04-12T05:37:01.138 INFO:tasks.workunit.client.0.smithi032.stdout:6/138: dwrite d4/f5 [0,4194304] 0 2024-04-12T05:37:01.140 INFO:tasks.workunit.client.0.smithi032.stdout:8/147: getdents d0/db 0 2024-04-12T05:37:01.142 INFO:tasks.workunit.client.0.smithi032.stdout:0/242: getdents d2/d8/d17 0 2024-04-12T05:37:01.142 INFO:tasks.workunit.client.0.smithi032.stdout:7/160: dwrite d0/f9 [0,4194304] 0 2024-04-12T05:37:01.145 INFO:tasks.workunit.client.0.smithi032.stdout:8/148: read - d0/db/dd/f17 zero size 2024-04-12T05:37:01.147 INFO:tasks.workunit.client.0.smithi032.stdout:4/177: dread d4/d13/fd [0,4194304] 0 2024-04-12T05:37:01.147 INFO:tasks.workunit.client.0.smithi032.stdout:7/161: chown d0/d1c/f1f 0 1 2024-04-12T05:37:01.150 INFO:tasks.workunit.client.0.smithi032.stdout:0/243: creat d2/d8/d17/d1b/f4d x:0 0 0 2024-04-12T05:37:01.150 INFO:tasks.workunit.client.0.smithi032.stdout:9/171: dwrite f5 [0,4194304] 0 2024-04-12T05:37:01.152 INFO:tasks.workunit.client.0.smithi032.stdout:0/244: stat d2/d8/f37 0 2024-04-12T05:37:01.154 INFO:tasks.workunit.client.0.smithi032.stdout:6/139: dwrite d4/f19 [0,4194304] 0 2024-04-12T05:37:01.155 INFO:tasks.workunit.client.0.smithi032.stdout:4/178: fsync d4/f18 0 2024-04-12T05:37:01.157 INFO:tasks.workunit.client.0.smithi032.stdout:0/245: symlink d2/d8/d17/d1b/d20/l4e 0 2024-04-12T05:37:01.158 INFO:tasks.workunit.client.0.smithi032.stdout:5/241: dread d2/dd/f3a [0,4194304] 0 2024-04-12T05:37:01.159 INFO:tasks.workunit.client.0.smithi032.stdout:9/172: link d9/d15/d2c/f23 d9/d15/d1a/d24/f40 0 2024-04-12T05:37:01.160 INFO:tasks.workunit.client.0.smithi032.stdout:6/140: truncate d4/f19 4910966 0 2024-04-12T05:37:01.163 INFO:tasks.workunit.client.0.smithi032.stdout:4/179: mkdir d4/d13/d19/d28/d2f/d3b 0 2024-04-12T05:37:01.166 INFO:tasks.workunit.client.0.smithi032.stdout:0/246: read d2/d8/fa [642897,48866] 0 2024-04-12T05:37:01.166 INFO:tasks.workunit.client.0.smithi032.stdout:6/141: chown d4/d14/f26 63374215 1 2024-04-12T05:37:01.170 INFO:tasks.workunit.client.0.smithi032.stdout:7/162: dwrite d0/d10/f22 [0,4194304] 0 2024-04-12T05:37:01.172 INFO:tasks.workunit.client.0.smithi032.stdout:7/163: write d0/d13/f27 [71219,37629] 0 2024-04-12T05:37:01.172 INFO:tasks.workunit.client.0.smithi032.stdout:9/173: rename d9/d1f/d2f to d9/d15/d2c/d41 0 2024-04-12T05:37:01.174 INFO:tasks.workunit.client.0.smithi032.stdout:1/119: rmdir d5/de 39 2024-04-12T05:37:01.175 INFO:tasks.workunit.client.0.smithi032.stdout:5/242: dread d2/f24 [0,4194304] 0 2024-04-12T05:37:01.175 INFO:tasks.workunit.client.0.smithi032.stdout:6/142: mknod d4/d21/c30 0 2024-04-12T05:37:01.175 INFO:tasks.workunit.client.0.smithi032.stdout:9/174: fdatasync d9/d15/d2c/f14 0 2024-04-12T05:37:01.176 INFO:tasks.workunit.client.0.smithi032.stdout:5/243: fdatasync d2/dd/f44 0 2024-04-12T05:37:01.177 INFO:tasks.workunit.client.0.smithi032.stdout:1/120: write d5/fc [1191485,102996] 0 2024-04-12T05:37:01.179 INFO:tasks.workunit.client.0.smithi032.stdout:4/180: creat d4/d13/d19/d28/d2f/d3b/f3c x:0 0 0 2024-04-12T05:37:01.191 INFO:tasks.workunit.client.0.smithi032.stdout:0/247: rmdir d2/d8/d17/d1b/d20 39 2024-04-12T05:37:01.197 INFO:tasks.workunit.client.0.smithi032.stdout:6/143: dwrite d4/f2c [0,4194304] 0 2024-04-12T05:37:01.198 INFO:tasks.workunit.client.0.smithi032.stdout:8/149: write d0/db/f19 [1150806,11017] 0 2024-04-12T05:37:01.199 INFO:tasks.workunit.client.0.smithi032.stdout:9/175: dwrite d9/d15/d1a/d24/f3f [0,4194304] 0 2024-04-12T05:37:01.200 INFO:tasks.workunit.client.0.smithi032.stdout:1/121: dread d5/ff [0,4194304] 0 2024-04-12T05:37:01.201 INFO:tasks.workunit.client.0.smithi032.stdout:7/164: dwrite d0/de/f15 [0,4194304] 0 2024-04-12T05:37:01.202 INFO:tasks.workunit.client.0.smithi032.stdout:5/244: dwrite d2/dd/f44 [0,4194304] 0 2024-04-12T05:37:01.203 INFO:tasks.workunit.client.0.smithi032.stdout:4/181: getdents d4/d13/d19/d28/d2f/d3b 0 2024-04-12T05:37:01.204 INFO:tasks.workunit.client.0.smithi032.stdout:9/176: chown d9/d15/d2c/d41/f3e 11 1 2024-04-12T05:37:01.205 INFO:tasks.workunit.client.0.smithi032.stdout:8/150: fsync d0/db/f18 0 2024-04-12T05:37:01.210 INFO:tasks.workunit.client.0.smithi032.stdout:4/182: write d4/d13/ff [125991,22784] 0 2024-04-12T05:37:01.211 INFO:tasks.workunit.client.0.smithi032.stdout:5/245: unlink f1 0 2024-04-12T05:37:01.213 INFO:tasks.workunit.client.0.smithi032.stdout:3/149: sync 2024-04-12T05:37:01.213 INFO:tasks.workunit.client.0.smithi032.stdout:4/183: fsync d4/f31 0 2024-04-12T05:37:01.218 INFO:tasks.workunit.client.0.smithi032.stdout:0/248: write d2/f7 [1787478,110575] 0 2024-04-12T05:37:01.220 INFO:tasks.workunit.client.0.smithi032.stdout:4/184: dread - d4/f38 zero size 2024-04-12T05:37:01.222 INFO:tasks.workunit.client.0.smithi032.stdout:7/165: dwrite d0/f9 [0,4194304] 0 2024-04-12T05:37:01.223 INFO:tasks.workunit.client.0.smithi032.stdout:0/249: symlink d2/d8/l4f 0 2024-04-12T05:37:01.225 INFO:tasks.workunit.client.0.smithi032.stdout:6/144: getdents d4/d14/d25 0 2024-04-12T05:37:01.226 INFO:tasks.workunit.client.0.smithi032.stdout:4/185: creat d4/f3d x:0 0 0 2024-04-12T05:37:01.226 INFO:tasks.workunit.client.0.smithi032.stdout:9/177: dwrite d9/d15/d2c/ff [0,4194304] 0 2024-04-12T05:37:01.228 INFO:tasks.workunit.client.0.smithi032.stdout:8/151: dread d0/f8 [0,4194304] 0 2024-04-12T05:37:01.231 INFO:tasks.workunit.client.0.smithi032.stdout:7/166: unlink d0/f1b 0 2024-04-12T05:37:01.234 INFO:tasks.workunit.client.0.smithi032.stdout:7/167: fsync d0/de/f15 0 2024-04-12T05:37:01.234 INFO:tasks.workunit.client.0.smithi032.stdout:4/186: creat d4/d13/d19/d28/d32/f3e x:0 0 0 2024-04-12T05:37:01.236 INFO:tasks.workunit.client.0.smithi032.stdout:8/152: creat d0/db/f2e x:0 0 0 2024-04-12T05:37:01.237 INFO:tasks.workunit.client.0.smithi032.stdout:7/168: write d0/de/f15 [4156234,63031] 0 2024-04-12T05:37:01.237 INFO:tasks.workunit.client.0.smithi032.stdout:0/250: dwrite d2/d28/d2c/d3d/f43 [0,4194304] 0 2024-04-12T05:37:01.238 INFO:tasks.workunit.client.0.smithi032.stdout:1/122: truncate d5/f13 4610253 0 2024-04-12T05:37:01.238 INFO:tasks.workunit.client.0.smithi032.stdout:9/178: dread d9/d15/f28 [0,4194304] 0 2024-04-12T05:37:01.242 INFO:tasks.workunit.client.0.smithi032.stdout:6/145: link d4/d21/f2f d4/d14/d1a/f31 0 2024-04-12T05:37:01.242 INFO:tasks.workunit.client.0.smithi032.stdout:0/251: chown d2/d28/d2c/f3f 1000860 1 2024-04-12T05:37:01.243 INFO:tasks.workunit.client.0.smithi032.stdout:3/150: truncate d12/f13 984983 0 2024-04-12T05:37:01.245 INFO:tasks.workunit.client.0.smithi032.stdout:0/252: fdatasync d2/d8/f37 0 2024-04-12T05:37:01.246 INFO:tasks.workunit.client.0.smithi032.stdout:3/151: stat f10 0 2024-04-12T05:37:01.248 INFO:tasks.workunit.client.0.smithi032.stdout:1/123: dread - d5/de/d17/d1b/f1d zero size 2024-04-12T05:37:01.250 INFO:tasks.workunit.client.0.smithi032.stdout:4/187: dwrite d4/d13/f33 [0,4194304] 0 2024-04-12T05:37:01.255 INFO:tasks.workunit.client.0.smithi032.stdout:9/179: dread d9/d15/d2c/ff [0,4194304] 0 2024-04-12T05:37:01.255 INFO:tasks.workunit.client.0.smithi032.stdout:6/146: mkdir d4/d21/d2e/d32 0 2024-04-12T05:37:01.256 INFO:tasks.workunit.client.0.smithi032.stdout:4/188: write d4/d13/d19/f39 [241019,51220] 0 2024-04-12T05:37:01.256 INFO:tasks.workunit.client.0.smithi032.stdout:1/124: symlink d5/de/d17/l22 0 2024-04-12T05:37:01.256 INFO:tasks.workunit.client.0.smithi032.stdout:9/180: chown d9/d1f 25853 1 2024-04-12T05:37:01.256 INFO:tasks.workunit.client.0.smithi032.stdout:8/153: rename d0/f8 to d0/f2f 0 2024-04-12T05:37:01.258 INFO:tasks.workunit.client.0.smithi032.stdout:7/169: link d0/f5 d0/f37 0 2024-04-12T05:37:01.258 INFO:tasks.workunit.client.0.smithi032.stdout:3/152: dread f10 [0,4194304] 0 2024-04-12T05:37:01.259 INFO:tasks.workunit.client.0.smithi032.stdout:4/189: chown d4/f38 3860 1 2024-04-12T05:37:01.259 INFO:tasks.workunit.client.0.smithi032.stdout:6/147: truncate d4/f5 4665221 0 2024-04-12T05:37:01.260 INFO:tasks.workunit.client.0.smithi032.stdout:9/181: mknod d9/d1f/c42 0 2024-04-12T05:37:01.261 INFO:tasks.workunit.client.0.smithi032.stdout:3/153: dread - d12/d1f/f26 zero size 2024-04-12T05:37:01.265 INFO:tasks.workunit.client.0.smithi032.stdout:3/154: truncate d12/d24/f2b 202419 0 2024-04-12T05:37:01.267 INFO:tasks.workunit.client.0.smithi032.stdout:8/154: symlink d0/db/dd/d11/d1b/d1e/l30 0 2024-04-12T05:37:01.268 INFO:tasks.workunit.client.0.smithi032.stdout:7/170: dwrite d0/fc [0,4194304] 0 2024-04-12T05:37:01.269 INFO:tasks.workunit.client.0.smithi032.stdout:1/125: rename d5/fc to d5/f23 0 2024-04-12T05:37:01.270 INFO:tasks.workunit.client.0.smithi032.stdout:9/182: creat d9/d15/f43 x:0 0 0 2024-04-12T05:37:01.271 INFO:tasks.workunit.client.0.smithi032.stdout:1/126: write d5/de/f11 [4456890,68562] 0 2024-04-12T05:37:01.272 INFO:tasks.workunit.client.0.smithi032.stdout:0/253: write d2/d8/fa [387031,15981] 0 2024-04-12T05:37:01.273 INFO:tasks.workunit.client.0.smithi032.stdout:3/155: rename d12/d19 to d12/d2e 0 2024-04-12T05:37:01.274 INFO:tasks.workunit.client.0.smithi032.stdout:6/148: write d4/f2a [650212,10928] 0 2024-04-12T05:37:01.274 INFO:tasks.workunit.client.0.smithi032.stdout:8/155: dread d0/f5 [0,4194304] 0 2024-04-12T05:37:01.275 INFO:tasks.workunit.client.0.smithi032.stdout:1/127: truncate d5/de/d17/f20 91072 0 2024-04-12T05:37:01.282 INFO:tasks.workunit.client.0.smithi032.stdout:8/156: fsync d0/f5 0 2024-04-12T05:37:01.283 INFO:tasks.workunit.client.0.smithi032.stdout:1/128: dwrite d5/de/d17/f20 [0,4194304] 0 2024-04-12T05:37:01.285 INFO:tasks.workunit.client.0.smithi032.stdout:0/254: rename d2/d28/d2c to d2/d28/d50 0 2024-04-12T05:37:01.287 INFO:tasks.workunit.client.0.smithi032.stdout:6/149: mknod d4/d21/d2e/c33 0 2024-04-12T05:37:01.290 INFO:tasks.workunit.client.0.smithi032.stdout:7/171: mknod d0/c38 0 2024-04-12T05:37:01.290 INFO:tasks.workunit.client.0.smithi032.stdout:0/255: fdatasync d2/d28/d50/d3d/f43 0 2024-04-12T05:37:01.291 INFO:tasks.workunit.client.0.smithi032.stdout:4/190: dwrite d4/f18 [0,4194304] 0 2024-04-12T05:37:01.293 INFO:tasks.workunit.client.0.smithi032.stdout:9/183: link d9/d15/d1a/f21 d9/d15/f44 0 2024-04-12T05:37:01.295 INFO:tasks.workunit.client.0.smithi032.stdout:1/129: symlink d5/de/d17/d1b/l24 0 2024-04-12T05:37:01.295 INFO:tasks.workunit.client.0.smithi032.stdout:9/184: stat d9/fa 0 2024-04-12T05:37:01.298 INFO:tasks.workunit.client.0.smithi032.stdout:7/172: mknod d0/de/df/c39 0 2024-04-12T05:37:01.299 INFO:tasks.workunit.client.0.smithi032.stdout:8/157: creat d0/db/dd/d11/d1b/d22/d29/f31 x:0 0 0 2024-04-12T05:37:01.299 INFO:tasks.workunit.client.0.smithi032.stdout:9/185: write f5 [3333331,83320] 0 2024-04-12T05:37:01.303 INFO:tasks.workunit.client.0.smithi032.stdout:0/256: dwrite d2/f30 [4194304,4194304] 0 2024-04-12T05:37:01.304 INFO:tasks.workunit.client.0.smithi032.stdout:4/191: getdents d4/d13/d19/d28/d32 0 2024-04-12T05:37:01.306 INFO:tasks.workunit.client.0.smithi032.stdout:6/150: write d4/ff [4749037,110473] 0 2024-04-12T05:37:01.306 INFO:tasks.workunit.client.0.smithi032.stdout:8/158: fsync d0/db/dd/d11/f2a 0 2024-04-12T05:37:01.311 INFO:tasks.workunit.client.0.smithi032.stdout:7/173: link d0/d10/l2b d0/d1c/l3a 0 2024-04-12T05:37:01.315 INFO:tasks.workunit.client.0.smithi032.stdout:3/156: dwrite f10 [0,4194304] 0 2024-04-12T05:37:01.315 INFO:tasks.workunit.client.0.smithi032.stdout:6/151: rename d4/l12 to d4/d21/l34 0 2024-04-12T05:37:01.316 INFO:tasks.workunit.client.0.smithi032.stdout:1/130: dwrite d5/f23 [0,4194304] 0 2024-04-12T05:37:01.320 INFO:tasks.workunit.client.0.smithi032.stdout:8/159: symlink d0/db/dd/d11/d1b/d1e/l32 0 2024-04-12T05:37:01.320 INFO:tasks.workunit.client.0.smithi032.stdout:7/174: creat d0/dd/f3b x:0 0 0 2024-04-12T05:37:01.320 INFO:tasks.workunit.client.0.smithi032.stdout:6/152: symlink d4/d14/d25/l35 0 2024-04-12T05:37:01.323 INFO:tasks.workunit.client.0.smithi032.stdout:3/157: fdatasync d12/d24/f2b 0 2024-04-12T05:37:01.326 INFO:tasks.workunit.client.0.smithi032.stdout:9/186: dwrite d9/d15/d2c/d41/f3e [0,4194304] 0 2024-04-12T05:37:01.328 INFO:tasks.workunit.client.0.smithi032.stdout:1/131: rmdir d5/d1e 0 2024-04-12T05:37:01.329 INFO:tasks.workunit.client.0.smithi032.stdout:6/153: truncate d4/d14/d1a/f31 884940 0 2024-04-12T05:37:01.330 INFO:tasks.workunit.client.0.smithi032.stdout:3/158: mkdir d12/d2f 0 2024-04-12T05:37:01.336 INFO:tasks.workunit.client.0.smithi032.stdout:2/158: sync 2024-04-12T05:37:01.339 INFO:tasks.workunit.client.0.smithi032.stdout:0/257: dwrite d2/dc/d3e/f46 [0,4194304] 0 2024-04-12T05:37:01.340 INFO:tasks.workunit.client.0.smithi032.stdout:3/159: creat d12/d24/f30 x:0 0 0 2024-04-12T05:37:01.341 INFO:tasks.workunit.client.0.smithi032.stdout:2/159: chown dd/c11 536 1 2024-04-12T05:37:01.343 INFO:tasks.workunit.client.0.smithi032.stdout:7/175: dwrite d0/fc [4194304,4194304] 0 2024-04-12T05:37:01.345 INFO:tasks.workunit.client.0.smithi032.stdout:3/160: fsync d12/d24/f2b 0 2024-04-12T05:37:01.346 INFO:tasks.workunit.client.0.smithi032.stdout:9/187: getdents d9 0 2024-04-12T05:37:01.349 INFO:tasks.workunit.client.0.smithi032.stdout:3/161: stat d12/d2e/l25 0 2024-04-12T05:37:01.351 INFO:tasks.workunit.client.0.smithi032.stdout:7/176: fsync d0/dd/f3b 0 2024-04-12T05:37:01.351 INFO:tasks.workunit.client.0.smithi032.stdout:2/160: write dd/ff [4354244,88242] 0 2024-04-12T05:37:01.352 INFO:tasks.workunit.client.0.smithi032.stdout:8/160: dwrite d0/db/dd/d11/f26 [0,4194304] 0 2024-04-12T05:37:01.352 INFO:tasks.workunit.client.0.smithi032.stdout:6/154: getdents d4 0 2024-04-12T05:37:01.356 INFO:tasks.workunit.client.0.smithi032.stdout:1/132: truncate d5/de/f19 2591414 0 2024-04-12T05:37:01.358 INFO:tasks.workunit.client.0.smithi032.stdout:3/162: write d12/d24/f2c [650032,113901] 0 2024-04-12T05:37:01.360 INFO:tasks.workunit.client.0.smithi032.stdout:7/177: symlink d0/d1c/l3c 0 2024-04-12T05:37:01.361 INFO:tasks.workunit.client.0.smithi032.stdout:6/155: creat d4/d21/d2e/d32/f36 x:0 0 0 2024-04-12T05:37:01.362 INFO:tasks.workunit.client.0.smithi032.stdout:8/161: symlink d0/db/dd/d11/l33 0 2024-04-12T05:37:01.362 INFO:tasks.workunit.client.0.smithi032.stdout:4/192: dread d4/d13/f1c [0,4194304] 0 2024-04-12T05:37:01.364 INFO:tasks.workunit.client.0.smithi032.stdout:6/156: fsync d4/f19 0 2024-04-12T05:37:01.365 INFO:tasks.workunit.client.0.smithi032.stdout:3/163: chown d12/l15 32 1 2024-04-12T05:37:01.365 INFO:tasks.workunit.client.0.smithi032.stdout:0/258: dwrite d2/d8/d17/f22 [0,4194304] 0 2024-04-12T05:37:01.365 INFO:tasks.workunit.client.0.smithi032.stdout:9/188: getdents d9/d15/d2c/d41 0 2024-04-12T05:37:01.365 INFO:tasks.workunit.client.0.smithi032.stdout:1/133: truncate d5/de/f1f 461590 0 2024-04-12T05:37:01.368 INFO:tasks.workunit.client.0.smithi032.stdout:7/178: truncate d0/dd/f3b 803741 0 2024-04-12T05:37:01.368 INFO:tasks.workunit.client.0.smithi032.stdout:9/189: dread - d9/d32/f37 zero size 2024-04-12T05:37:01.369 INFO:tasks.workunit.client.0.smithi032.stdout:0/259: read d2/fe [1953477,9611] 0 2024-04-12T05:37:01.369 INFO:tasks.workunit.client.0.smithi032.stdout:4/193: rename d4/d13 to d4/d13/d19/d28/d2f/d3f 22 2024-04-12T05:37:01.369 INFO:tasks.workunit.client.0.smithi032.stdout:8/162: fdatasync d0/db/dd/d11/d1b/d22/f2b 0 2024-04-12T05:37:01.369 INFO:tasks.workunit.client.0.smithi032.stdout:6/157: mkdir d4/d14/d1a/d37 0 2024-04-12T05:37:01.372 INFO:tasks.workunit.client.0.smithi032.stdout:3/164: fsync d12/f13 0 2024-04-12T05:37:01.373 INFO:tasks.workunit.client.0.smithi032.stdout:7/179: write d0/dd/f36 [543720,40435] 0 2024-04-12T05:37:01.374 INFO:tasks.workunit.client.0.smithi032.stdout:6/158: readlink d4/d14/d1a/l24 0 2024-04-12T05:37:01.378 INFO:tasks.workunit.client.0.smithi032.stdout:2/161: dwrite dd/ff [0,4194304] 0 2024-04-12T05:37:01.378 INFO:tasks.workunit.client.0.smithi032.stdout:8/163: mknod d0/db/dd/d11/d1b/c34 0 2024-04-12T05:37:01.378 INFO:tasks.workunit.client.0.smithi032.stdout:0/260: truncate d2/d8/f33 4630649 0 2024-04-12T05:37:01.381 INFO:tasks.workunit.client.0.smithi032.stdout:3/165: creat d12/d1f/f31 x:0 0 0 2024-04-12T05:37:01.382 INFO:tasks.workunit.client.0.smithi032.stdout:0/261: stat d2/d8/c1a 0 2024-04-12T05:37:01.387 INFO:tasks.workunit.client.0.smithi032.stdout:8/164: mkdir d0/db/dd/d11/d1b/d22/d35 0 2024-04-12T05:37:01.388 INFO:tasks.workunit.client.0.smithi032.stdout:0/262: mkdir d2/d28/d50/d51 0 2024-04-12T05:37:01.389 INFO:tasks.workunit.client.0.smithi032.stdout:9/190: dwrite d9/d15/f30 [0,4194304] 0 2024-04-12T05:37:01.389 INFO:tasks.workunit.client.0.smithi032.stdout:3/166: truncate d12/f13 1220417 0 2024-04-12T05:37:01.392 INFO:tasks.workunit.client.0.smithi032.stdout:7/180: dread d0/f9 [0,4194304] 0 2024-04-12T05:37:01.393 INFO:tasks.workunit.client.0.smithi032.stdout:3/167: rmdir d12/d2e 39 2024-04-12T05:37:01.394 INFO:tasks.workunit.client.0.smithi032.stdout:6/159: dwrite d4/f5 [0,4194304] 0 2024-04-12T05:37:01.396 INFO:tasks.workunit.client.0.smithi032.stdout:0/263: unlink d2/d8/d17/l26 0 2024-04-12T05:37:01.396 INFO:tasks.workunit.client.0.smithi032.stdout:8/165: mkdir d0/d36 0 2024-04-12T05:37:01.398 INFO:tasks.workunit.client.0.smithi032.stdout:2/162: dwrite f8 [0,4194304] 0 2024-04-12T05:37:01.405 INFO:tasks.workunit.client.0.smithi032.stdout:8/166: write d0/f7 [731083,69130] 0 2024-04-12T05:37:01.405 INFO:tasks.workunit.client.0.smithi032.stdout:0/264: creat d2/d8/df/f52 x:0 0 0 2024-04-12T05:37:01.408 INFO:tasks.workunit.client.0.smithi032.stdout:9/191: rename d9/d15/d1a/f21 to d9/d15/d2c/d41/f45 0 2024-04-12T05:37:01.410 INFO:tasks.workunit.client.0.smithi032.stdout:9/192: stat d9/f13 0 2024-04-12T05:37:01.411 INFO:tasks.workunit.client.0.smithi032.stdout:2/163: unlink dd/d1b/d1c/f21 0 2024-04-12T05:37:01.411 INFO:tasks.workunit.client.0.smithi032.stdout:8/167: symlink d0/d36/l37 0 2024-04-12T05:37:01.414 INFO:tasks.workunit.client.0.smithi032.stdout:2/164: chown c3 1939286 1 2024-04-12T05:37:01.416 INFO:tasks.workunit.client.0.smithi032.stdout:6/160: dwrite d4/f1b [0,4194304] 0 2024-04-12T05:37:01.418 INFO:tasks.workunit.client.0.smithi032.stdout:3/168: dwrite d12/d24/f2c [0,4194304] 0 2024-04-12T05:37:01.419 INFO:tasks.workunit.client.0.smithi032.stdout:9/193: write d9/d32/f37 [599234,22082] 0 2024-04-12T05:37:01.421 INFO:tasks.workunit.client.0.smithi032.stdout:7/181: dwrite d0/dd/f3b [0,4194304] 0 2024-04-12T05:37:01.421 INFO:tasks.workunit.client.0.smithi032.stdout:0/265: link d2/d28/d50/d3d/f43 d2/dc/d3e/f53 0 2024-04-12T05:37:01.422 INFO:tasks.workunit.client.0.smithi032.stdout:8/168: write d0/db/dd/d11/f1f [2368789,51919] 0 2024-04-12T05:37:01.424 INFO:tasks.workunit.client.0.smithi032.stdout:9/194: stat d9/d15/f28 0 2024-04-12T05:37:01.424 INFO:tasks.workunit.client.0.smithi032.stdout:6/161: chown d4/cc 0 1 2024-04-12T05:37:01.425 INFO:tasks.workunit.client.0.smithi032.stdout:3/169: write d12/d1f/f31 [712296,25654] 0 2024-04-12T05:37:01.426 INFO:tasks.workunit.client.0.smithi032.stdout:0/266: chown d2/d8/df/d24/f4b 57 1 2024-04-12T05:37:01.428 INFO:tasks.workunit.client.0.smithi032.stdout:8/169: mkdir d0/db/d38 0 2024-04-12T05:37:01.429 INFO:tasks.workunit.client.0.smithi032.stdout:7/182: rmdir d0/de/df/d29 39 2024-04-12T05:37:01.431 INFO:tasks.workunit.client.0.smithi032.stdout:6/162: truncate d4/d21/d2e/d32/f36 255484 0 2024-04-12T05:37:01.434 INFO:tasks.workunit.client.0.smithi032.stdout:3/170: unlink d12/d24/f30 0 2024-04-12T05:37:01.435 INFO:tasks.workunit.client.0.smithi032.stdout:8/170: dread d0/db/dd/f14 [0,4194304] 0 2024-04-12T05:37:01.436 INFO:tasks.workunit.client.0.smithi032.stdout:0/267: rename d2/d8/fa to d2/d28/d50/f54 0 2024-04-12T05:37:01.444 INFO:tasks.workunit.client.0.smithi032.stdout:8/171: mkdir d0/db/d39 0 2024-04-12T05:37:01.445 INFO:tasks.workunit.client.0.smithi032.stdout:0/268: fdatasync d2/dc/d3e/f53 0 2024-04-12T05:37:01.447 INFO:tasks.workunit.client.0.smithi032.stdout:7/183: mknod d0/de/df/d29/c3d 0 2024-04-12T05:37:01.449 INFO:tasks.workunit.client.0.smithi032.stdout:9/195: dwrite d9/d15/d2c/f23 [8388608,4194304] 0 2024-04-12T05:37:01.449 INFO:tasks.workunit.client.0.smithi032.stdout:7/184: chown d0/f9 57745 1 2024-04-12T05:37:01.450 INFO:tasks.workunit.client.0.smithi032.stdout:7/185: dread - d0/d1c/f1f zero size 2024-04-12T05:37:01.451 INFO:tasks.workunit.client.0.smithi032.stdout:8/172: creat d0/d36/f3a x:0 0 0 2024-04-12T05:37:01.452 INFO:tasks.workunit.client.0.smithi032.stdout:0/269: fsync d2/dc/d3e/f53 0 2024-04-12T05:37:01.453 INFO:tasks.workunit.client.0.smithi032.stdout:8/173: fdatasync d0/db/dd/d11/f2a 0 2024-04-12T05:37:01.455 INFO:tasks.workunit.client.0.smithi032.stdout:9/196: dread d9/fa [0,4194304] 0 2024-04-12T05:37:01.457 INFO:tasks.workunit.client.0.smithi032.stdout:0/270: fsync d2/d8/d17/d1b/d20/f45 0 2024-04-12T05:37:01.458 INFO:tasks.workunit.client.0.smithi032.stdout:3/171: link d12/l27 d12/l32 0 2024-04-12T05:37:01.459 INFO:tasks.workunit.client.0.smithi032.stdout:6/163: truncate d4/f6 932612 0 2024-04-12T05:37:01.460 INFO:tasks.workunit.client.0.smithi032.stdout:8/174: rename d0/db/dd/d11/l1d to d0/db/dd/d11/d1b/d22/d35/l3b 0 2024-04-12T05:37:01.462 INFO:tasks.workunit.client.0.smithi032.stdout:5/246: sync 2024-04-12T05:37:01.463 INFO:tasks.workunit.client.0.smithi032.stdout:5/247: fsync d2/df/f3f 0 2024-04-12T05:37:01.466 INFO:tasks.workunit.client.0.smithi032.stdout:3/172: symlink d12/d2e/l33 0 2024-04-12T05:37:01.471 INFO:tasks.workunit.client.0.smithi032.stdout:5/248: truncate d2/d17/f21 156915 0 2024-04-12T05:37:01.472 INFO:tasks.workunit.client.0.smithi032.stdout:6/164: write d4/d14/d1a/f31 [1234266,111196] 0 2024-04-12T05:37:01.472 INFO:tasks.workunit.client.0.smithi032.stdout:8/175: write d0/db/dd/f14 [2946117,5868] 0 2024-04-12T05:37:01.472 INFO:tasks.workunit.client.0.smithi032.stdout:3/173: dwrite d12/f21 [0,4194304] 0 2024-04-12T05:37:01.474 INFO:tasks.workunit.client.0.smithi032.stdout:9/197: getdents d9/d15/d2c 0 2024-04-12T05:37:01.476 INFO:tasks.workunit.client.0.smithi032.stdout:0/271: link d2/d29/l2a d2/dc/l55 0 2024-04-12T05:37:01.476 INFO:tasks.workunit.client.0.smithi032.stdout:3/174: creat d12/d24/f34 x:0 0 0 2024-04-12T05:37:01.478 INFO:tasks.workunit.client.0.smithi032.stdout:0/272: chown d2/d28/d50/c38 0 1 2024-04-12T05:37:01.478 INFO:tasks.workunit.client.0.smithi032.stdout:5/249: dread d2/df/d25/f2c [0,4194304] 0 2024-04-12T05:37:01.478 INFO:tasks.workunit.client.0.smithi032.stdout:3/175: rename d12/d2e to d12/d2e/d35 22 2024-04-12T05:37:01.480 INFO:tasks.workunit.client.0.smithi032.stdout:8/176: write d0/fe [84596,41468] 0 2024-04-12T05:37:01.482 INFO:tasks.workunit.client.0.smithi032.stdout:6/165: link d4/f2a d4/d21/d27/f38 0 2024-04-12T05:37:01.482 INFO:tasks.workunit.client.0.smithi032.stdout:3/176: truncate d12/d24/f2b 354185 0 2024-04-12T05:37:01.482 INFO:tasks.workunit.client.0.smithi032.stdout:9/198: dread d9/d15/f28 [0,4194304] 0 2024-04-12T05:37:01.483 INFO:tasks.workunit.client.0.smithi032.stdout:6/166: write d4/d21/d27/f28 [928994,39664] 0 2024-04-12T05:37:01.483 INFO:tasks.workunit.client.0.smithi032.stdout:0/273: mknod d2/d28/d50/d51/c56 0 2024-04-12T05:37:01.485 INFO:tasks.workunit.client.0.smithi032.stdout:6/167: stat d4/d14 0 2024-04-12T05:37:01.486 INFO:tasks.workunit.client.0.smithi032.stdout:0/274: fdatasync d2/d8/f33 0 2024-04-12T05:37:01.488 INFO:tasks.workunit.client.0.smithi032.stdout:5/250: mknod d2/d3c/c45 0 2024-04-12T05:37:01.494 INFO:tasks.workunit.client.0.smithi032.stdout:0/275: mknod d2/d8/d17/d1b/d20/c57 0 2024-04-12T05:37:01.494 INFO:tasks.workunit.client.0.smithi032.stdout:8/177: dwrite d0/f7 [0,4194304] 0 2024-04-12T05:37:01.494 INFO:tasks.workunit.client.0.smithi032.stdout:5/251: write d2/df/d25/f29 [1224364,14240] 0 2024-04-12T05:37:01.497 INFO:tasks.workunit.client.0.smithi032.stdout:0/276: write d2/d8/d17/f2b [1923637,52419] 0 2024-04-12T05:37:01.498 INFO:tasks.workunit.client.0.smithi032.stdout:6/168: truncate d4/f5 103355 0 2024-04-12T05:37:01.498 INFO:tasks.workunit.client.0.smithi032.stdout:5/252: creat d2/df/d31/f46 x:0 0 0 2024-04-12T05:37:01.499 INFO:tasks.workunit.client.0.smithi032.stdout:9/199: dread d9/d15/d2c/f23 [4194304,4194304] 0 2024-04-12T05:37:01.499 INFO:tasks.workunit.client.0.smithi032.stdout:3/177: dread d12/f13 [0,4194304] 0 2024-04-12T05:37:01.506 INFO:tasks.workunit.client.0.smithi032.stdout:0/277: mknod d2/d28/d50/d51/c58 0 2024-04-12T05:37:01.507 INFO:tasks.workunit.client.0.smithi032.stdout:9/200: dread - d9/d15/d1a/f3d zero size 2024-04-12T05:37:01.507 INFO:tasks.workunit.client.0.smithi032.stdout:8/178: getdents d0/db/dd/d11 0 2024-04-12T05:37:01.508 INFO:tasks.workunit.client.0.smithi032.stdout:3/178: write d12/f21 [2763720,86117] 0 2024-04-12T05:37:01.514 INFO:tasks.workunit.client.0.smithi032.stdout:6/169: dwrite d4/d21/f2f [0,4194304] 0 2024-04-12T05:37:01.514 INFO:tasks.workunit.client.0.smithi032.stdout:8/179: unlink d0/db/dd/d11/d1b/d1e/l32 0 2024-04-12T05:37:01.515 INFO:tasks.workunit.client.0.smithi032.stdout:3/179: dread d12/d1f/f31 [0,4194304] 0 2024-04-12T05:37:01.518 INFO:tasks.workunit.client.0.smithi032.stdout:6/170: chown d4/d14/d1a/l22 6 1 2024-04-12T05:37:01.519 INFO:tasks.workunit.client.0.smithi032.stdout:3/180: readlink d12/l27 0 2024-04-12T05:37:01.521 INFO:tasks.workunit.client.0.smithi032.stdout:3/181: stat d12/d24/f2b 0 2024-04-12T05:37:01.521 INFO:tasks.workunit.client.0.smithi032.stdout:6/171: dread d4/d21/d2e/d32/f36 [0,4194304] 0 2024-04-12T05:37:01.522 INFO:tasks.workunit.client.0.smithi032.stdout:9/201: link d9/d15/d2c/f10 d9/d1f/f46 0 2024-04-12T05:37:01.523 INFO:tasks.workunit.client.0.smithi032.stdout:8/180: mkdir d0/db/dd/d3c 0 2024-04-12T05:37:01.523 INFO:tasks.workunit.client.0.smithi032.stdout:0/278: write d2/d8/df/f31 [1213807,72080] 0 2024-04-12T05:37:01.525 INFO:tasks.workunit.client.0.smithi032.stdout:3/182: creat d12/d2e/f36 x:0 0 0 2024-04-12T05:37:01.525 INFO:tasks.workunit.client.0.smithi032.stdout:9/202: creat d9/d15/d1a/d24/f47 x:0 0 0 2024-04-12T05:37:01.527 INFO:tasks.workunit.client.0.smithi032.stdout:6/172: fsync d4/f6 0 2024-04-12T05:37:01.528 INFO:tasks.workunit.client.0.smithi032.stdout:0/279: mknod d2/dc/d41/c59 0 2024-04-12T05:37:01.528 INFO:tasks.workunit.client.0.smithi032.stdout:6/173: write d4/f19 [3694115,22186] 0 2024-04-12T05:37:01.530 INFO:tasks.workunit.client.0.smithi032.stdout:0/280: write d2/d8/d17/f22 [7106451,66322] 0 2024-04-12T05:37:01.535 INFO:tasks.workunit.client.0.smithi032.stdout:6/174: fsync d4/d21/d2e/d32/f36 0 2024-04-12T05:37:01.537 INFO:tasks.workunit.client.0.smithi032.stdout:3/183: symlink d12/d1f/d2d/l37 0 2024-04-12T05:37:01.537 INFO:tasks.workunit.client.0.smithi032.stdout:8/181: link d0/db/dd/d11/d1b/d1e/l2d d0/d36/l3d 0 2024-04-12T05:37:01.539 INFO:tasks.workunit.client.0.smithi032.stdout:6/175: creat d4/d21/d2e/f39 x:0 0 0 2024-04-12T05:37:01.540 INFO:tasks.workunit.client.0.smithi032.stdout:0/281: symlink d2/d8/l5a 0 2024-04-12T05:37:01.541 INFO:tasks.workunit.client.0.smithi032.stdout:3/184: creat d12/d24/f38 x:0 0 0 2024-04-12T05:37:01.542 INFO:tasks.workunit.client.0.smithi032.stdout:8/182: mknod d0/db/c3e 0 2024-04-12T05:37:01.542 INFO:tasks.workunit.client.0.smithi032.stdout:0/282: mknod d2/d8/df/d24/d39/c5b 0 2024-04-12T05:37:01.543 INFO:tasks.workunit.client.0.smithi032.stdout:6/176: mknod d4/d14/d1a/d37/c3a 0 2024-04-12T05:37:01.543 INFO:tasks.workunit.client.0.smithi032.stdout:8/183: readlink d0/db/dd/d11/d1b/d1e/l30 0 2024-04-12T05:37:01.543 INFO:tasks.workunit.client.0.smithi032.stdout:3/185: truncate d12/f13 1844486 0 2024-04-12T05:37:01.544 INFO:tasks.workunit.client.0.smithi032.stdout:6/177: write d4/f19 [2946673,21375] 0 2024-04-12T05:37:01.545 INFO:tasks.workunit.client.0.smithi032.stdout:0/283: creat d2/d8/df/d24/f5c x:0 0 0 2024-04-12T05:37:01.547 INFO:tasks.workunit.client.0.smithi032.stdout:3/186: mknod d12/d1f/d2d/c39 0 2024-04-12T05:37:01.548 INFO:tasks.workunit.client.0.smithi032.stdout:6/178: creat d4/d21/f3b x:0 0 0 2024-04-12T05:37:01.549 INFO:tasks.workunit.client.0.smithi032.stdout:8/184: creat d0/db/d38/f3f x:0 0 0 2024-04-12T05:37:01.552 INFO:tasks.workunit.client.0.smithi032.stdout:0/284: rename d2/d8/df/d24/f4b to d2/d29/f5d 0 2024-04-12T05:37:01.552 INFO:tasks.workunit.client.0.smithi032.stdout:6/179: dread d4/f1b [0,4194304] 0 2024-04-12T05:37:01.555 INFO:tasks.workunit.client.0.smithi032.stdout:0/285: write d2/d28/d50/f35 [443759,25951] 0 2024-04-12T05:37:01.557 INFO:tasks.workunit.client.0.smithi032.stdout:8/185: dwrite d0/fe [0,4194304] 0 2024-04-12T05:37:01.558 INFO:tasks.workunit.client.0.smithi032.stdout:6/180: mkdir d4/d14/d3c 0 2024-04-12T05:37:01.560 INFO:tasks.workunit.client.0.smithi032.stdout:0/286: write d2/d28/d50/f54 [1347942,104493] 0 2024-04-12T05:37:01.563 INFO:tasks.workunit.client.0.smithi032.stdout:8/186: fdatasync d0/db/dd/d11/f13 0 2024-04-12T05:37:01.564 INFO:tasks.workunit.client.0.smithi032.stdout:3/187: dwrite d12/f1b [4194304,4194304] 0 2024-04-12T05:37:01.564 INFO:tasks.workunit.client.0.smithi032.stdout:6/181: symlink d4/d21/d2e/d32/l3d 0 2024-04-12T05:37:01.570 INFO:tasks.workunit.client.0.smithi032.stdout:8/187: creat d0/db/dd/d11/f40 x:0 0 0 2024-04-12T05:37:01.571 INFO:tasks.workunit.client.0.smithi032.stdout:6/182: mknod d4/d21/c3e 0 2024-04-12T05:37:01.573 INFO:tasks.workunit.client.0.smithi032.stdout:0/287: rmdir d2/d8/df/d32 0 2024-04-12T05:37:01.573 INFO:tasks.workunit.client.0.smithi032.stdout:0/288: chown d2/dc/d41/l44 4355431 1 2024-04-12T05:37:01.575 INFO:tasks.workunit.client.0.smithi032.stdout:0/289: creat d2/d8/df/d24/f5e x:0 0 0 2024-04-12T05:37:01.576 INFO:tasks.workunit.client.0.smithi032.stdout:0/290: read - d2/d29/f47 zero size 2024-04-12T05:37:01.579 INFO:tasks.workunit.client.0.smithi032.stdout:0/291: getdents d2/dc/d3e 0 2024-04-12T05:37:01.579 INFO:tasks.workunit.client.0.smithi032.stdout:0/292: chown d2/f6 350 1 2024-04-12T05:37:01.580 INFO:tasks.workunit.client.0.smithi032.stdout:0/293: truncate d2/d28/d50/f35 1836010 0 2024-04-12T05:37:01.581 INFO:tasks.workunit.client.0.smithi032.stdout:0/294: dread - d2/d8/d17/d1b/d20/f45 zero size 2024-04-12T05:37:01.582 INFO:tasks.workunit.client.0.smithi032.stdout:0/295: write d2/d8/df/d24/f5c [746103,109361] 0 2024-04-12T05:37:01.585 INFO:tasks.workunit.client.0.smithi032.stdout:0/296: creat d2/d8/df/f5f x:0 0 0 2024-04-12T05:37:01.587 INFO:tasks.workunit.client.0.smithi032.stdout:0/297: rename d2/d8/df/d24/d39 to d2/d8/df/d24/d60 0 2024-04-12T05:37:01.590 INFO:tasks.workunit.client.0.smithi032.stdout:0/298: getdents d2/d8/df/d24/d60 0 2024-04-12T05:37:01.590 INFO:tasks.workunit.client.0.smithi032.stdout:0/299: chown d2/d29/f47 82882236 1 2024-04-12T05:37:01.593 INFO:tasks.workunit.client.0.smithi032.stdout:0/300: rmdir d2/d8/df/d24/d60/d42 0 2024-04-12T05:37:01.595 INFO:tasks.workunit.client.0.smithi032.stdout:0/301: write d2/d8/d17/f22 [90041,87954] 0 2024-04-12T05:37:01.596 INFO:tasks.workunit.client.0.smithi032.stdout:0/302: creat d2/dc/f61 x:0 0 0 2024-04-12T05:37:01.599 INFO:tasks.workunit.client.0.smithi032.stdout:0/303: getdents d2/d28/d50 0 2024-04-12T05:37:01.600 INFO:tasks.workunit.client.0.smithi032.stdout:0/304: dread - d2/dc/f61 zero size 2024-04-12T05:37:01.603 INFO:tasks.workunit.client.0.smithi032.stdout:0/305: dread d2/dc/d3e/f53 [0,4194304] 0 2024-04-12T05:37:01.609 INFO:tasks.workunit.client.0.smithi032.stdout:0/306: dwrite d2/d29/f4a [0,4194304] 0 2024-04-12T05:37:01.612 INFO:tasks.workunit.client.0.smithi032.stdout:0/307: rename d2/dc/c36 to d2/d8/d48/c62 0 2024-04-12T05:37:01.616 INFO:tasks.workunit.client.0.smithi032.stdout:1/134: sync 2024-04-12T05:37:01.616 INFO:tasks.workunit.client.0.smithi032.stdout:0/308: rename d2/d28/d50/d3d to d2/d8/df/d24/d60/d63 0 2024-04-12T05:37:01.619 INFO:tasks.workunit.client.0.smithi032.stdout:1/135: getdents d5/de/d17 0 2024-04-12T05:37:01.623 INFO:tasks.workunit.client.0.smithi032.stdout:1/136: dread d5/f13 [4194304,4194304] 0 2024-04-12T05:37:01.624 INFO:tasks.workunit.client.0.smithi032.stdout:1/137: rmdir d5/de/d17/d1b 39 2024-04-12T05:37:01.626 INFO:tasks.workunit.client.0.smithi032.stdout:1/138: rmdir d5/de/d17/d1b 39 2024-04-12T05:37:01.719 INFO:tasks.workunit.client.0.smithi032.stdout:2/165: sync 2024-04-12T05:37:01.719 INFO:tasks.workunit.client.0.smithi032.stdout:4/194: sync 2024-04-12T05:37:01.719 INFO:tasks.workunit.client.0.smithi032.stdout:5/253: sync 2024-04-12T05:37:01.719 INFO:tasks.workunit.client.0.smithi032.stdout:7/186: sync 2024-04-12T05:37:01.720 INFO:tasks.workunit.client.0.smithi032.stdout:4/195: write d4/f1d [3815830,78851] 0 2024-04-12T05:37:01.721 INFO:tasks.workunit.client.0.smithi032.stdout:5/254: fdatasync d2/df/d25/d1e/f40 0 2024-04-12T05:37:01.723 INFO:tasks.workunit.client.0.smithi032.stdout:7/187: creat d0/f3e x:0 0 0 2024-04-12T05:37:01.723 INFO:tasks.workunit.client.0.smithi032.stdout:5/255: creat d2/dd/f47 x:0 0 0 2024-04-12T05:37:01.724 INFO:tasks.workunit.client.0.smithi032.stdout:4/196: symlink d4/d13/d19/d28/d2f/d3b/l40 0 2024-04-12T05:37:01.724 INFO:tasks.workunit.client.0.smithi032.stdout:2/166: getdents dd/d1b 0 2024-04-12T05:37:01.725 INFO:tasks.workunit.client.0.smithi032.stdout:2/167: dread - dd/f15 zero size 2024-04-12T05:37:01.726 INFO:tasks.workunit.client.0.smithi032.stdout:5/256: truncate d2/df/d25/d1e/f40 2256520 0 2024-04-12T05:37:01.727 INFO:tasks.workunit.client.0.smithi032.stdout:2/168: write dd/d19/f1d [1047416,20180] 0 2024-04-12T05:37:01.727 INFO:tasks.workunit.client.0.smithi032.stdout:4/197: mknod d4/d13/d19/c41 0 2024-04-12T05:37:01.728 INFO:tasks.workunit.client.0.smithi032.stdout:5/257: mkdir d2/d3c/d48 0 2024-04-12T05:37:01.728 INFO:tasks.workunit.client.0.smithi032.stdout:4/198: fsync d4/d13/d19/d28/d2f/d3b/f3c 0 2024-04-12T05:37:01.733 INFO:tasks.workunit.client.0.smithi032.stdout:4/199: creat d4/d13/d19/d28/f42 x:0 0 0 2024-04-12T05:37:01.733 INFO:tasks.workunit.client.0.smithi032.stdout:4/200: chown d4/d13/f33 0 1 2024-04-12T05:37:01.733 INFO:tasks.workunit.client.0.smithi032.stdout:5/258: mkdir d2/df/d25/d43/d49 0 2024-04-12T05:37:01.735 INFO:tasks.workunit.client.0.smithi032.stdout:7/188: truncate d0/d13/f16 2785714 0 2024-04-12T05:37:01.736 INFO:tasks.workunit.client.0.smithi032.stdout:7/189: write d0/de/f12 [1901218,64979] 0 2024-04-12T05:37:01.740 INFO:tasks.workunit.client.0.smithi032.stdout:7/190: creat d0/de/df/f3f x:0 0 0 2024-04-12T05:37:01.741 INFO:tasks.workunit.client.0.smithi032.stdout:4/201: rmdir d4/d13/d19 39 2024-04-12T05:37:01.744 INFO:tasks.workunit.client.0.smithi032.stdout:5/259: write d2/df/d31/f33 [238901,58480] 0 2024-04-12T05:37:01.744 INFO:tasks.workunit.client.0.smithi032.stdout:7/191: dread d0/d13/f1a [0,4194304] 0 2024-04-12T05:37:01.745 INFO:tasks.workunit.client.0.smithi032.stdout:4/202: creat d4/d13/f43 x:0 0 0 2024-04-12T05:37:01.746 INFO:tasks.workunit.client.0.smithi032.stdout:5/260: fdatasync d2/f4 0 2024-04-12T05:37:01.749 INFO:tasks.workunit.client.0.smithi032.stdout:4/203: truncate d4/d13/d19/d28/f42 908523 0 2024-04-12T05:37:01.751 INFO:tasks.workunit.client.0.smithi032.stdout:5/261: unlink d2/dd/c36 0 2024-04-12T05:37:01.752 INFO:tasks.workunit.client.0.smithi032.stdout:7/192: getdents d0/d13/d21/d30 0 2024-04-12T05:37:01.752 INFO:tasks.workunit.client.0.smithi032.stdout:4/204: mkdir d4/d13/d44 0 2024-04-12T05:37:01.753 INFO:tasks.workunit.client.0.smithi032.stdout:7/193: symlink d0/de/l40 0 2024-04-12T05:37:01.755 INFO:tasks.workunit.client.0.smithi032.stdout:4/205: mknod d4/d13/c45 0 2024-04-12T05:37:01.756 INFO:tasks.workunit.client.0.smithi032.stdout:7/194: mknod d0/d13/d21/d30/d31/c41 0 2024-04-12T05:37:01.759 INFO:tasks.workunit.client.0.smithi032.stdout:7/195: write d0/d13/f1a [2007964,1544] 0 2024-04-12T05:37:01.760 INFO:tasks.workunit.client.0.smithi032.stdout:5/262: truncate d2/d17/f39 32522 0 2024-04-12T05:37:01.761 INFO:tasks.workunit.client.0.smithi032.stdout:7/196: rmdir d0/de/df 39 2024-04-12T05:37:01.762 INFO:tasks.workunit.client.0.smithi032.stdout:4/206: dwrite d4/f18 [0,4194304] 0 2024-04-12T05:37:01.766 INFO:tasks.workunit.client.0.smithi032.stdout:5/263: mknod d2/df/c4a 0 2024-04-12T05:37:01.766 INFO:tasks.workunit.client.0.smithi032.stdout:7/197: mknod d0/de/df/d20/c42 0 2024-04-12T05:37:01.768 INFO:tasks.workunit.client.0.smithi032.stdout:4/207: dread d4/d13/ff [0,4194304] 0 2024-04-12T05:37:01.770 INFO:tasks.workunit.client.0.smithi032.stdout:5/264: truncate d2/df/d25/f16 1626967 0 2024-04-12T05:37:01.772 INFO:tasks.workunit.client.0.smithi032.stdout:5/265: chown d2/c37 2 1 2024-04-12T05:37:01.773 INFO:tasks.workunit.client.0.smithi032.stdout:4/208: dread d4/f18 [0,4194304] 0 2024-04-12T05:37:01.777 INFO:tasks.workunit.client.0.smithi032.stdout:4/209: mkdir d4/d13/d19/d28/d2f/d46 0 2024-04-12T05:37:01.777 INFO:tasks.workunit.client.0.smithi032.stdout:5/266: dread d2/df/f3f [0,4194304] 0 2024-04-12T05:37:01.780 INFO:tasks.workunit.client.0.smithi032.stdout:4/210: creat d4/d13/d19/d28/f47 x:0 0 0 2024-04-12T05:37:01.781 INFO:tasks.workunit.client.0.smithi032.stdout:5/267: mkdir d2/d3c/d48/d4b 0 2024-04-12T05:37:01.783 INFO:tasks.workunit.client.0.smithi032.stdout:3/188: sync 2024-04-12T05:37:01.783 INFO:tasks.workunit.client.0.smithi032.stdout:9/203: sync 2024-04-12T05:37:01.783 INFO:tasks.workunit.client.0.smithi032.stdout:8/188: sync 2024-04-12T05:37:01.783 INFO:tasks.workunit.client.0.smithi032.stdout:6/183: sync 2024-04-12T05:37:01.783 INFO:tasks.workunit.client.0.smithi032.stdout:1/139: sync 2024-04-12T05:37:01.783 INFO:tasks.workunit.client.0.smithi032.stdout:0/309: sync 2024-04-12T05:37:01.784 INFO:tasks.workunit.client.0.smithi032.stdout:6/184: dread - d4/d14/f26 zero size 2024-04-12T05:37:01.784 INFO:tasks.workunit.client.0.smithi032.stdout:3/189: write d12/d24/f38 [192598,129744] 0 2024-04-12T05:37:01.786 INFO:tasks.workunit.client.0.smithi032.stdout:9/204: creat d9/d15/f48 x:0 0 0 2024-04-12T05:37:01.787 INFO:tasks.workunit.client.0.smithi032.stdout:9/205: chown d9/d15 88965 1 2024-04-12T05:37:01.787 INFO:tasks.workunit.client.0.smithi032.stdout:9/206: readlink d9/l3c 0 2024-04-12T05:37:01.787 INFO:tasks.workunit.client.0.smithi032.stdout:1/140: truncate d5/de/d17/d1b/f1d 267690 0 2024-04-12T05:37:01.788 INFO:tasks.workunit.client.0.smithi032.stdout:5/268: link d2/d3c/c45 d2/d3c/c4c 0 2024-04-12T05:37:01.790 INFO:tasks.workunit.client.0.smithi032.stdout:3/190: rename d12/d1f to d12/d2e/d3a 0 2024-04-12T05:37:01.791 INFO:tasks.workunit.client.0.smithi032.stdout:1/141: chown d5/de/d17/d1b/c1c 315359241 1 2024-04-12T05:37:01.793 INFO:tasks.workunit.client.0.smithi032.stdout:5/269: write d2/df/d31/f33 [3791318,69621] 0 2024-04-12T05:37:01.794 INFO:tasks.workunit.client.0.smithi032.stdout:3/191: stat d12/l1a 0 2024-04-12T05:37:01.795 INFO:tasks.workunit.client.0.smithi032.stdout:5/270: readlink d2/df/d25/l20 0 2024-04-12T05:37:01.798 INFO:tasks.workunit.client.0.smithi032.stdout:6/185: dread d4/f1b [0,4194304] 0 2024-04-12T05:37:01.798 INFO:tasks.workunit.client.0.smithi032.stdout:9/207: dwrite d9/d15/f30 [0,4194304] 0 2024-04-12T05:37:01.798 INFO:tasks.workunit.client.0.smithi032.stdout:3/192: chown c9 17 1 2024-04-12T05:37:01.799 INFO:tasks.workunit.client.0.smithi032.stdout:1/142: creat d5/de/d17/f25 x:0 0 0 2024-04-12T05:37:01.800 INFO:tasks.workunit.client.0.smithi032.stdout:4/211: dread d4/f31 [0,4194304] 0 2024-04-12T05:37:01.801 INFO:tasks.workunit.client.0.smithi032.stdout:1/143: fdatasync d5/fb 0 2024-04-12T05:37:01.804 INFO:tasks.workunit.client.0.smithi032.stdout:5/271: symlink d2/df/d25/d43/d49/l4d 0 2024-04-12T05:37:01.807 INFO:tasks.workunit.client.0.smithi032.stdout:4/212: rmdir d4/d13/d19/d28/d32 39 2024-04-12T05:37:01.808 INFO:tasks.workunit.client.0.smithi032.stdout:0/310: dwrite d2/dc/d3e/f53 [0,4194304] 0 2024-04-12T05:37:01.809 INFO:tasks.workunit.client.0.smithi032.stdout:6/186: getdents d4 0 2024-04-12T05:37:01.812 INFO:tasks.workunit.client.0.smithi032.stdout:1/144: creat d5/de/d17/f26 x:0 0 0 2024-04-12T05:37:01.813 INFO:tasks.workunit.client.0.smithi032.stdout:0/311: mkdir d2/d8/d17/d64 0 2024-04-12T05:37:01.813 INFO:tasks.workunit.client.0.smithi032.stdout:6/187: write d4/d21/d2e/f39 [112108,111476] 0 2024-04-12T05:37:01.813 INFO:tasks.workunit.client.0.smithi032.stdout:4/213: symlink d4/d13/d44/l48 0 2024-04-12T05:37:01.814 INFO:tasks.workunit.client.0.smithi032.stdout:9/208: dwrite d9/d15/d1a/d24/d27/f2d [0,4194304] 0 2024-04-12T05:37:01.815 INFO:tasks.workunit.client.0.smithi032.stdout:6/188: stat d4/d14/d25/l35 0 2024-04-12T05:37:01.820 INFO:tasks.workunit.client.0.smithi032.stdout:6/189: write d4/d21/f3b [371290,25893] 0 2024-04-12T05:37:01.822 INFO:tasks.workunit.client.0.smithi032.stdout:1/145: mkdir d5/d27 0 2024-04-12T05:37:01.822 INFO:tasks.workunit.client.0.smithi032.stdout:5/272: link d2/df/d25/ca d2/df/d25/d43/d49/c4e 0 2024-04-12T05:37:01.825 INFO:tasks.workunit.client.0.smithi032.stdout:3/193: truncate d12/d24/f2c 2848860 0 2024-04-12T05:37:01.825 INFO:tasks.workunit.client.0.smithi032.stdout:6/190: chown d4/d14/f15 2141671 1 2024-04-12T05:37:01.825 INFO:tasks.workunit.client.0.smithi032.stdout:0/312: dread d2/dc/d3e/f53 [0,4194304] 0 2024-04-12T05:37:01.829 INFO:tasks.workunit.client.0.smithi032.stdout:1/146: rename d5/fb to d5/de/f28 0 2024-04-12T05:37:01.829 INFO:tasks.workunit.client.0.smithi032.stdout:6/191: dread - d4/d14/f26 zero size 2024-04-12T05:37:01.829 INFO:tasks.workunit.client.0.smithi032.stdout:5/273: chown d2/df/d31/f46 47 1 2024-04-12T05:37:01.829 INFO:tasks.workunit.client.0.smithi032.stdout:9/209: dwrite d9/d15/d2c/f14 [0,4194304] 0 2024-04-12T05:37:01.830 INFO:tasks.workunit.client.0.smithi032.stdout:3/194: creat d12/d2e/d3a/f3b x:0 0 0 2024-04-12T05:37:01.832 INFO:tasks.workunit.client.0.smithi032.stdout:1/147: fsync d5/de/f11 0 2024-04-12T05:37:01.833 INFO:tasks.workunit.client.0.smithi032.stdout:6/192: chown d4/lb 58 1 2024-04-12T05:37:01.833 INFO:tasks.workunit.client.0.smithi032.stdout:0/313: creat d2/d29/f65 x:0 0 0 2024-04-12T05:37:01.833 INFO:tasks.workunit.client.0.smithi032.stdout:4/214: dwrite d4/d13/fd [0,4194304] 0 2024-04-12T05:37:01.835 INFO:tasks.workunit.client.0.smithi032.stdout:5/274: chown d2/df/d25/f29 12349 1 2024-04-12T05:37:01.837 INFO:tasks.workunit.client.0.smithi032.stdout:5/275: readlink d2/dd/l2e 0 2024-04-12T05:37:01.838 INFO:tasks.workunit.client.0.smithi032.stdout:9/210: creat d9/d1f/f49 x:0 0 0 2024-04-12T05:37:01.838 INFO:tasks.workunit.client.0.smithi032.stdout:4/215: chown d4/d13/d19/c17 0 1 2024-04-12T05:37:01.839 INFO:tasks.workunit.client.0.smithi032.stdout:6/193: write d4/d21/d27/f28 [1435495,86904] 0 2024-04-12T05:37:01.840 INFO:tasks.workunit.client.0.smithi032.stdout:2/169: sync 2024-04-12T05:37:01.842 INFO:tasks.workunit.client.0.smithi032.stdout:4/216: read - d4/f3d zero size 2024-04-12T05:37:01.842 INFO:tasks.workunit.client.0.smithi032.stdout:2/170: chown fa 166546 1 2024-04-12T05:37:01.844 INFO:tasks.workunit.client.0.smithi032.stdout:5/276: chown d2/df/d25/f16 2678642 1 2024-04-12T05:37:01.845 INFO:tasks.workunit.client.0.smithi032.stdout:9/211: unlink d9/l2a 0 2024-04-12T05:37:01.847 INFO:tasks.workunit.client.0.smithi032.stdout:3/195: link ce d12/d24/c3c 0 2024-04-12T05:37:01.850 INFO:tasks.workunit.client.0.smithi032.stdout:3/196: readlink d12/l32 0 2024-04-12T05:37:01.850 INFO:tasks.workunit.client.0.smithi032.stdout:2/171: mkdir dd/d1b/d1c/d2c 0 2024-04-12T05:37:01.851 INFO:tasks.workunit.client.0.smithi032.stdout:0/314: dwrite d2/d28/d50/f35 [0,4194304] 0 2024-04-12T05:37:01.851 INFO:tasks.workunit.client.0.smithi032.stdout:4/217: mknod d4/d13/d19/d28/d2f/d46/c49 0 2024-04-12T05:37:01.852 INFO:tasks.workunit.client.0.smithi032.stdout:1/148: dwrite d5/ff [0,4194304] 0 2024-04-12T05:37:01.852 INFO:tasks.workunit.client.0.smithi032.stdout:5/277: symlink d2/d3c/d48/d4b/l4f 0 2024-04-12T05:37:01.853 INFO:tasks.workunit.client.0.smithi032.stdout:0/315: stat d2/dc/c21 0 2024-04-12T05:37:01.854 INFO:tasks.workunit.client.0.smithi032.stdout:1/149: chown d5/de/d17/d1b 12592749 1 2024-04-12T05:37:01.854 INFO:tasks.workunit.client.0.smithi032.stdout:3/197: rmdir d12/d2e/d3a 39 2024-04-12T05:37:01.855 INFO:tasks.workunit.client.0.smithi032.stdout:2/172: creat dd/d19/f2d x:0 0 0 2024-04-12T05:37:01.856 INFO:tasks.workunit.client.0.smithi032.stdout:5/278: mkdir d2/df/d25/d50 0 2024-04-12T05:37:01.857 INFO:tasks.workunit.client.0.smithi032.stdout:4/218: write d4/d13/f1c [1252941,103416] 0 2024-04-12T05:37:01.859 INFO:tasks.workunit.client.0.smithi032.stdout:1/150: rmdir d5/de 39 2024-04-12T05:37:01.862 INFO:tasks.workunit.client.0.smithi032.stdout:2/173: rename dd/f15 to dd/f2e 0 2024-04-12T05:37:01.866 INFO:tasks.workunit.client.0.smithi032.stdout:5/279: truncate d2/df/f1d 1376096 0 2024-04-12T05:37:01.868 INFO:tasks.workunit.client.0.smithi032.stdout:3/198: dread d12/d24/f2c [0,4194304] 0 2024-04-12T05:37:01.869 INFO:tasks.workunit.client.0.smithi032.stdout:1/151: getdents d5/d27 0 2024-04-12T05:37:01.870 INFO:tasks.workunit.client.0.smithi032.stdout:6/194: dread d4/d21/d27/f28 [0,4194304] 0 2024-04-12T05:37:01.871 INFO:tasks.workunit.client.0.smithi032.stdout:5/280: unlink d2/df/d25/l20 0 2024-04-12T05:37:01.873 INFO:tasks.workunit.client.0.smithi032.stdout:1/152: fdatasync d5/de/f1f 0 2024-04-12T05:37:01.874 INFO:tasks.workunit.client.0.smithi032.stdout:9/212: dread d9/d15/d1a/d24/d27/f2d [0,4194304] 0 2024-04-12T05:37:01.875 INFO:tasks.workunit.client.0.smithi032.stdout:6/195: rename d4/d21/l29 to d4/d14/d3c/l3f 0 2024-04-12T05:37:01.877 INFO:tasks.workunit.client.0.smithi032.stdout:1/153: fdatasync d5/ff 0 2024-04-12T05:37:01.877 INFO:tasks.workunit.client.0.smithi032.stdout:5/281: dread d2/dd/f3a [4194304,4194304] 0 2024-04-12T05:37:01.879 INFO:tasks.workunit.client.0.smithi032.stdout:6/196: write d4/f2c [2861597,72359] 0 2024-04-12T05:37:01.880 INFO:tasks.workunit.client.0.smithi032.stdout:2/174: dread fa [0,4194304] 0 2024-04-12T05:37:01.882 INFO:tasks.workunit.client.0.smithi032.stdout:3/199: dwrite d12/d2e/f36 [0,4194304] 0 2024-04-12T05:37:01.887 INFO:tasks.workunit.client.0.smithi032.stdout:9/213: dread d9/d15/d2c/d41/f3e [0,4194304] 0 2024-04-12T05:37:01.889 INFO:tasks.workunit.client.0.smithi032.stdout:6/197: dread d4/f2c [0,4194304] 0 2024-04-12T05:37:01.891 INFO:tasks.workunit.client.0.smithi032.stdout:1/154: dwrite d5/f23 [0,4194304] 0 2024-04-12T05:37:01.896 INFO:tasks.workunit.client.0.smithi032.stdout:6/198: dread d4/d21/f3b [0,4194304] 0 2024-04-12T05:37:01.897 INFO:tasks.workunit.client.0.smithi032.stdout:9/214: dwrite f5 [0,4194304] 0 2024-04-12T05:37:01.897 INFO:tasks.workunit.client.0.smithi032.stdout:2/175: link c3 dd/d1b/d1c/c2f 0 2024-04-12T05:37:01.898 INFO:tasks.workunit.client.0.smithi032.stdout:6/199: write d4/d14/f26 [891244,101358] 0 2024-04-12T05:37:01.899 INFO:tasks.workunit.client.0.smithi032.stdout:6/200: readlink l1 0 2024-04-12T05:37:01.900 INFO:tasks.workunit.client.0.smithi032.stdout:1/155: unlink d5/de/d17/d1b/c1c 0 2024-04-12T05:37:01.901 INFO:tasks.workunit.client.0.smithi032.stdout:1/156: write d5/de/f1f [176497,66569] 0 2024-04-12T05:37:01.904 INFO:tasks.workunit.client.0.smithi032.stdout:2/176: rename dd/d19/c2a to dd/d1b/d1c/c30 0 2024-04-12T05:37:01.905 INFO:tasks.workunit.client.0.smithi032.stdout:6/201: symlink d4/d21/d2e/l40 0 2024-04-12T05:37:01.906 INFO:tasks.workunit.client.0.smithi032.stdout:2/177: readlink dd/d19/l2b 0 2024-04-12T05:37:01.911 INFO:tasks.workunit.client.0.smithi032.stdout:2/178: mknod dd/d1b/d1c/c31 0 2024-04-12T05:37:01.913 INFO:tasks.workunit.client.0.smithi032.stdout:9/215: dwrite d9/d15/d2c/f23 [12582912,4194304] 0 2024-04-12T05:37:01.913 INFO:tasks.workunit.client.0.smithi032.stdout:2/179: stat dd/d19/c23 0 2024-04-12T05:37:01.914 INFO:tasks.workunit.client.0.smithi032.stdout:1/157: dwrite d5/de/d17/f20 [4194304,4194304] 0 2024-04-12T05:37:01.914 INFO:tasks.workunit.client.0.smithi032.stdout:7/198: sync 2024-04-12T05:37:01.915 INFO:tasks.workunit.client.0.smithi032.stdout:8/189: sync 2024-04-12T05:37:01.918 INFO:tasks.workunit.client.0.smithi032.stdout:9/216: symlink d9/d15/d1a/d24/d27/l4a 0 2024-04-12T05:37:01.921 INFO:tasks.workunit.client.0.smithi032.stdout:8/190: symlink d0/db/dd/d11/d1b/l41 0 2024-04-12T05:37:01.921 INFO:tasks.workunit.client.0.smithi032.stdout:6/202: dwrite d4/d21/f2f [0,4194304] 0 2024-04-12T05:37:01.921 INFO:tasks.workunit.client.0.smithi032.stdout:1/158: mknod d5/de/d17/c29 0 2024-04-12T05:37:01.922 INFO:tasks.workunit.client.0.smithi032.stdout:2/180: dread f1 [0,4194304] 0 2024-04-12T05:37:01.924 INFO:tasks.workunit.client.0.smithi032.stdout:2/181: chown dd/ff 71 1 2024-04-12T05:37:01.929 INFO:tasks.workunit.client.0.smithi032.stdout:3/200: write d12/d24/f2c [3587688,22069] 0 2024-04-12T05:37:01.930 INFO:tasks.workunit.client.0.smithi032.stdout:6/203: dread d4/f5 [0,4194304] 0 2024-04-12T05:37:01.934 INFO:tasks.workunit.client.0.smithi032.stdout:3/201: dread d12/d24/f38 [0,4194304] 0 2024-04-12T05:37:01.938 INFO:tasks.workunit.client.0.smithi032.stdout:2/182: write f0 [1711320,61391] 0 2024-04-12T05:37:01.939 INFO:tasks.workunit.client.0.smithi032.stdout:8/191: dread d0/db/dd/d11/d1b/d22/f2b [0,4194304] 0 2024-04-12T05:37:01.942 INFO:tasks.workunit.client.0.smithi032.stdout:3/202: fdatasync d12/d2e/d3a/f3b 0 2024-04-12T05:37:01.945 INFO:tasks.workunit.client.0.smithi032.stdout:8/192: dread - d0/db/f2e zero size 2024-04-12T05:37:01.947 INFO:tasks.workunit.client.0.smithi032.stdout:9/217: dwrite d9/d1f/f49 [0,4194304] 0 2024-04-12T05:37:01.950 INFO:tasks.workunit.client.0.smithi032.stdout:6/204: dwrite d4/d14/f15 [0,4194304] 0 2024-04-12T05:37:01.952 INFO:tasks.workunit.client.0.smithi032.stdout:1/159: dwrite d5/de/d17/f20 [8388608,4194304] 0 2024-04-12T05:37:01.953 INFO:tasks.workunit.client.0.smithi032.stdout:3/203: rename d12/d24/f34 to d12/d2f/f3d 0 2024-04-12T05:37:01.956 INFO:tasks.workunit.client.0.smithi032.stdout:1/160: chown d5/de/c15 634996 1 2024-04-12T05:37:01.958 INFO:tasks.workunit.client.0.smithi032.stdout:3/204: mknod d12/d2e/d3a/c3e 0 2024-04-12T05:37:01.960 INFO:tasks.workunit.client.0.smithi032.stdout:7/199: dread d0/f7 [0,4194304] 0 2024-04-12T05:37:01.961 INFO:tasks.workunit.client.0.smithi032.stdout:8/193: truncate d0/db/dd/d11/d1b/d22/f2b 655416 0 2024-04-12T05:37:01.963 INFO:tasks.workunit.client.0.smithi032.stdout:3/205: mknod d12/d2f/c3f 0 2024-04-12T05:37:01.964 INFO:tasks.workunit.client.0.smithi032.stdout:3/206: fdatasync d12/f21 0 2024-04-12T05:37:01.965 INFO:tasks.workunit.client.0.smithi032.stdout:7/200: creat d0/de/f43 x:0 0 0 2024-04-12T05:37:01.967 INFO:tasks.workunit.client.0.smithi032.stdout:1/161: dwrite d5/ff [0,4194304] 0 2024-04-12T05:37:01.969 INFO:tasks.workunit.client.0.smithi032.stdout:3/207: fdatasync d12/d2e/d3a/f31 0 2024-04-12T05:37:01.970 INFO:tasks.workunit.client.0.smithi032.stdout:1/162: rmdir d5 39 2024-04-12T05:37:01.971 INFO:tasks.workunit.client.0.smithi032.stdout:3/208: truncate d12/d24/f38 940970 0 2024-04-12T05:37:01.971 INFO:tasks.workunit.client.0.smithi032.stdout:3/209: stat d12/d2f 0 2024-04-12T05:37:01.972 INFO:tasks.workunit.client.0.smithi032.stdout:9/218: truncate d9/d15/d1a/d24/f40 10978561 0 2024-04-12T05:37:01.973 INFO:tasks.workunit.client.0.smithi032.stdout:9/219: chown d9/d15/f30 1940454 1 2024-04-12T05:37:01.976 INFO:tasks.workunit.client.0.smithi032.stdout:6/205: truncate d4/d14/d1a/f31 1680850 0 2024-04-12T05:37:01.977 INFO:tasks.workunit.client.0.smithi032.stdout:1/163: mkdir d5/de/d2a 0 2024-04-12T05:37:01.980 INFO:tasks.workunit.client.0.smithi032.stdout:9/220: rename d9/d15/d2c/f25 to d9/d15/d1a/d24/f4b 0 2024-04-12T05:37:01.983 INFO:tasks.workunit.client.0.smithi032.stdout:7/201: dread d0/f8 [0,4194304] 0 2024-04-12T05:37:01.984 INFO:tasks.workunit.client.0.smithi032.stdout:9/221: chown d9/d15/d1a/c1b 13941 1 2024-04-12T05:37:01.984 INFO:tasks.workunit.client.0.smithi032.stdout:1/164: creat d5/de/d2a/f2b x:0 0 0 2024-04-12T05:37:01.985 INFO:tasks.workunit.client.0.smithi032.stdout:3/210: dwrite d12/d2f/f3d [0,4194304] 0 2024-04-12T05:37:01.986 INFO:tasks.workunit.client.0.smithi032.stdout:6/206: dread d4/f2c [0,4194304] 0 2024-04-12T05:37:01.991 INFO:tasks.workunit.client.0.smithi032.stdout:3/211: truncate d12/d2e/d3a/f26 553760 0 2024-04-12T05:37:01.994 INFO:tasks.workunit.client.0.smithi032.stdout:1/165: mkdir d5/d2c 0 2024-04-12T05:37:01.994 INFO:tasks.workunit.client.0.smithi032.stdout:3/212: read d12/d2e/f1e [21953,112381] 0 2024-04-12T05:37:01.995 INFO:tasks.workunit.client.0.smithi032.stdout:1/166: readlink d5/de/l18 0 2024-04-12T05:37:01.996 INFO:tasks.workunit.client.0.smithi032.stdout:7/202: dwrite d0/de/df/f14 [0,4194304] 0 2024-04-12T05:37:02.003 INFO:tasks.workunit.client.0.smithi032.stdout:9/222: getdents d9/d15/d2c/d41 0 2024-04-12T05:37:02.003 INFO:tasks.workunit.client.0.smithi032.stdout:3/213: dwrite d12/d2f/f3d [0,4194304] 0 2024-04-12T05:37:02.006 INFO:tasks.workunit.client.0.smithi032.stdout:6/207: getdents d4/d21/d27 0 2024-04-12T05:37:02.007 INFO:tasks.workunit.client.0.smithi032.stdout:1/167: creat d5/d2c/f2d x:0 0 0 2024-04-12T05:37:02.010 INFO:tasks.workunit.client.0.smithi032.stdout:6/208: creat d4/d14/d3c/f41 x:0 0 0 2024-04-12T05:37:02.014 INFO:tasks.workunit.client.0.smithi032.stdout:7/203: link d0/dd/f3b d0/de/f44 0 2024-04-12T05:37:02.016 INFO:tasks.workunit.client.0.smithi032.stdout:1/168: mknod d5/d27/c2e 0 2024-04-12T05:37:02.018 INFO:tasks.workunit.client.0.smithi032.stdout:7/204: creat d0/d13/d21/f45 x:0 0 0 2024-04-12T05:37:02.018 INFO:tasks.workunit.client.0.smithi032.stdout:1/169: creat d5/d2c/f2f x:0 0 0 2024-04-12T05:37:02.019 INFO:tasks.workunit.client.0.smithi032.stdout:7/205: fsync d0/fc 0 2024-04-12T05:37:02.021 INFO:tasks.workunit.client.0.smithi032.stdout:1/170: chown d5/de/f11 228043 1 2024-04-12T05:37:02.023 INFO:tasks.workunit.client.0.smithi032.stdout:1/171: chown d5/d27/c2e 53319211 1 2024-04-12T05:37:02.025 INFO:tasks.workunit.client.0.smithi032.stdout:6/209: dwrite d4/d21/f3b [0,4194304] 0 2024-04-12T05:37:02.026 INFO:tasks.workunit.client.0.smithi032.stdout:7/206: write d0/f37 [119833,3224] 0 2024-04-12T05:37:02.030 INFO:tasks.workunit.client.0.smithi032.stdout:3/214: dwrite d12/f2a [0,4194304] 0 2024-04-12T05:37:02.031 INFO:tasks.workunit.client.0.smithi032.stdout:6/210: mknod d4/d14/d1a/c42 0 2024-04-12T05:37:02.032 INFO:tasks.workunit.client.0.smithi032.stdout:1/172: getdents d5/d27 0 2024-04-12T05:37:02.032 INFO:tasks.workunit.client.0.smithi032.stdout:6/211: chown d4/d21/d27/f28 0 1 2024-04-12T05:37:02.034 INFO:tasks.workunit.client.0.smithi032.stdout:3/215: mknod d12/d24/c40 0 2024-04-12T05:37:02.035 INFO:tasks.workunit.client.0.smithi032.stdout:6/212: chown d4/d14/d3c/l3f 5 1 2024-04-12T05:37:02.036 INFO:tasks.workunit.client.0.smithi032.stdout:7/207: write d0/dd/f3b [1012281,83543] 0 2024-04-12T05:37:02.037 INFO:tasks.workunit.client.0.smithi032.stdout:3/216: creat d12/d2f/f41 x:0 0 0 2024-04-12T05:37:02.039 INFO:tasks.workunit.client.0.smithi032.stdout:1/173: creat d5/f30 x:0 0 0 2024-04-12T05:37:02.040 INFO:tasks.workunit.client.0.smithi032.stdout:7/208: unlink d0/de/l40 0 2024-04-12T05:37:02.042 INFO:tasks.workunit.client.0.smithi032.stdout:1/174: mknod d5/de/d2a/c31 0 2024-04-12T05:37:02.045 INFO:tasks.workunit.client.0.smithi032.stdout:6/213: dwrite d4/d14/d3c/f41 [0,4194304] 0 2024-04-12T05:37:02.045 INFO:tasks.workunit.client.0.smithi032.stdout:1/175: readlink d5/de/d17/d1b/l24 0 2024-04-12T05:37:02.046 INFO:tasks.workunit.client.0.smithi032.stdout:7/209: link d0/f8 d0/de/df/d29/f46 0 2024-04-12T05:37:02.047 INFO:tasks.workunit.client.0.smithi032.stdout:9/223: dread d9/d15/d1a/d24/f40 [0,4194304] 0 2024-04-12T05:37:02.048 INFO:tasks.workunit.client.0.smithi032.stdout:9/224: fsync d9/d15/f48 0 2024-04-12T05:37:02.050 INFO:tasks.workunit.client.0.smithi032.stdout:9/225: mknod d9/d15/d1a/c4c 0 2024-04-12T05:37:02.051 INFO:tasks.workunit.client.0.smithi032.stdout:6/214: link d4/d14/d25/l2b d4/d14/d25/l43 0 2024-04-12T05:37:02.051 INFO:tasks.workunit.client.0.smithi032.stdout:1/176: dread d5/de/f10 [0,4194304] 0 2024-04-12T05:37:02.052 INFO:tasks.workunit.client.0.smithi032.stdout:0/316: sync 2024-04-12T05:37:02.053 INFO:tasks.workunit.client.0.smithi032.stdout:7/210: truncate d0/fc 2982397 0 2024-04-12T05:37:02.054 INFO:tasks.workunit.client.0.smithi032.stdout:1/177: symlink d5/de/l32 0 2024-04-12T05:37:02.054 INFO:tasks.workunit.client.0.smithi032.stdout:1/178: chown d5 237377 1 2024-04-12T05:37:02.055 INFO:tasks.workunit.client.0.smithi032.stdout:1/179: chown d5/f23 148479323 1 2024-04-12T05:37:02.056 INFO:tasks.workunit.client.0.smithi032.stdout:9/226: rename d9/d1f/f49 to d9/d15/d1a/d1d/f4d 0 2024-04-12T05:37:02.058 INFO:tasks.workunit.client.0.smithi032.stdout:7/211: mkdir d0/de/d47 0 2024-04-12T05:37:02.059 INFO:tasks.workunit.client.0.smithi032.stdout:6/215: link d4/cc d4/c44 0 2024-04-12T05:37:02.061 INFO:tasks.workunit.client.0.smithi032.stdout:9/227: symlink d9/d15/d1a/d31/l4e 0 2024-04-12T05:37:02.061 INFO:tasks.workunit.client.0.smithi032.stdout:6/216: write d4/d14/d3c/f41 [188112,115170] 0 2024-04-12T05:37:02.065 INFO:tasks.workunit.client.0.smithi032.stdout:1/180: getdents d5/de/d17 0 2024-04-12T05:37:02.066 INFO:tasks.workunit.client.0.smithi032.stdout:6/217: truncate d4/f9 1002058 0 2024-04-12T05:37:02.067 INFO:tasks.workunit.client.0.smithi032.stdout:7/212: dwrite d0/f9 [0,4194304] 0 2024-04-12T05:37:02.069 INFO:tasks.workunit.client.0.smithi032.stdout:1/181: creat d5/de/d17/d1b/f33 x:0 0 0 2024-04-12T05:37:02.069 INFO:tasks.workunit.client.0.smithi032.stdout:1/182: rename d5 to d5/d2c/d34 22 2024-04-12T05:37:02.070 INFO:tasks.workunit.client.0.smithi032.stdout:6/218: symlink d4/d14/d1a/d37/l45 0 2024-04-12T05:37:02.137 INFO:tasks.workunit.client.0.smithi032.stdout:9/228: getdents d9/d15/d1a/d24/d27 0 2024-04-12T05:37:02.139 INFO:tasks.workunit.client.0.smithi032.stdout:1/183: truncate d5/f13 1299189 0 2024-04-12T05:37:02.183 INFO:tasks.workunit.client.0.smithi032.stdout:4/219: sync 2024-04-12T05:37:02.183 INFO:tasks.workunit.client.0.smithi032.stdout:5/282: sync 2024-04-12T05:37:02.186 INFO:tasks.workunit.client.0.smithi032.stdout:7/213: dwrite d0/f37 [0,4194304] 0 2024-04-12T05:37:02.188 INFO:tasks.workunit.client.0.smithi032.stdout:1/184: fsync d5/f23 0 2024-04-12T05:37:02.189 INFO:tasks.workunit.client.0.smithi032.stdout:6/219: truncate d4/d14/d1a/f31 661860 0 2024-04-12T05:37:02.189 INFO:tasks.workunit.client.0.smithi032.stdout:5/283: creat d2/df/d25/d43/f51 x:0 0 0 2024-04-12T05:37:02.191 INFO:tasks.workunit.client.0.smithi032.stdout:4/220: symlink d4/d13/d19/d28/d2f/d46/l4a 0 2024-04-12T05:37:02.191 INFO:tasks.workunit.client.0.smithi032.stdout:5/284: readlink d2/d17/l3e 0 2024-04-12T05:37:02.191 INFO:tasks.workunit.client.0.smithi032.stdout:7/214: symlink d0/d1c/l48 0 2024-04-12T05:37:02.191 INFO:tasks.workunit.client.0.smithi032.stdout:4/221: fdatasync d4/d13/f43 0 2024-04-12T05:37:02.193 INFO:tasks.workunit.client.0.smithi032.stdout:9/229: link d9/d15/d1a/d24/d27/l4a d9/d15/d1a/d24/d27/l4f 0 2024-04-12T05:37:02.194 INFO:tasks.workunit.client.0.smithi032.stdout:1/185: getdents d5/d27 0 2024-04-12T05:37:02.195 INFO:tasks.workunit.client.0.smithi032.stdout:7/215: symlink d0/d13/d21/d30/d31/l49 0 2024-04-12T05:37:02.197 INFO:tasks.workunit.client.0.smithi032.stdout:4/222: write d4/f18 [1248063,75517] 0 2024-04-12T05:37:02.197 INFO:tasks.workunit.client.0.smithi032.stdout:7/216: write d0/f37 [3234590,128085] 0 2024-04-12T05:37:02.199 INFO:tasks.workunit.client.0.smithi032.stdout:5/285: rename d2/df/d31/l38 to d2/df/d25/l52 0 2024-04-12T05:37:02.200 INFO:tasks.workunit.client.0.smithi032.stdout:6/220: getdents d4/d14/d1a 0 2024-04-12T05:37:02.201 INFO:tasks.workunit.client.0.smithi032.stdout:5/286: chown d2/d17/c3b 731 1 2024-04-12T05:37:02.201 INFO:tasks.workunit.client.0.smithi032.stdout:4/223: write d4/d13/f33 [3071006,25977] 0 2024-04-12T05:37:02.202 INFO:tasks.workunit.client.0.smithi032.stdout:6/221: fsync d4/d21/f3b 0 2024-04-12T05:37:02.202 INFO:tasks.workunit.client.0.smithi032.stdout:1/186: rename d5/d27 to d5/de/d17/d35 0 2024-04-12T05:37:02.203 INFO:tasks.workunit.client.0.smithi032.stdout:4/224: write d4/f1d [3843397,901] 0 2024-04-12T05:37:02.204 INFO:tasks.workunit.client.0.smithi032.stdout:1/187: stat d5/de/f28 0 2024-04-12T05:37:02.206 INFO:tasks.workunit.client.0.smithi032.stdout:4/225: truncate d4/d13/d19/d28/f47 795693 0 2024-04-12T05:37:02.206 INFO:tasks.workunit.client.0.smithi032.stdout:4/226: read - d4/f38 zero size 2024-04-12T05:37:02.209 INFO:tasks.workunit.client.0.smithi032.stdout:6/222: creat d4/d14/f46 x:0 0 0 2024-04-12T05:37:02.212 INFO:tasks.workunit.client.0.smithi032.stdout:6/223: write d4/d21/d2e/d32/f36 [490797,18165] 0 2024-04-12T05:37:02.213 INFO:tasks.workunit.client.0.smithi032.stdout:7/217: rename d0/d13/d21/l2e to d0/d34/l4a 0 2024-04-12T05:37:02.214 INFO:tasks.workunit.client.0.smithi032.stdout:9/230: getdents d9/d32 0 2024-04-12T05:37:02.215 INFO:tasks.workunit.client.0.smithi032.stdout:4/227: dwrite d4/f18 [0,4194304] 0 2024-04-12T05:37:02.215 INFO:tasks.workunit.client.0.smithi032.stdout:7/218: fdatasync d0/d1c/f1f 0 2024-04-12T05:37:02.216 INFO:tasks.workunit.client.0.smithi032.stdout:5/287: mkdir d2/dd/d53 0 2024-04-12T05:37:02.219 INFO:tasks.workunit.client.0.smithi032.stdout:5/288: fsync d2/df/d25/d1e/f2f 0 2024-04-12T05:37:02.220 INFO:tasks.workunit.client.0.smithi032.stdout:6/224: dread d4/d14/d3c/f41 [0,4194304] 0 2024-04-12T05:37:02.221 INFO:tasks.workunit.client.0.smithi032.stdout:7/219: rename d0/dd/f36 to d0/d13/f4b 0 2024-04-12T05:37:02.225 INFO:tasks.workunit.client.0.smithi032.stdout:8/194: sync 2024-04-12T05:37:02.225 INFO:tasks.workunit.client.0.smithi032.stdout:2/183: sync 2024-04-12T05:37:02.225 INFO:tasks.workunit.client.0.smithi032.stdout:9/231: truncate d9/f13 114554 0 2024-04-12T05:37:02.229 INFO:tasks.workunit.client.0.smithi032.stdout:6/225: mkdir d4/d14/d1a/d37/d47 0 2024-04-12T05:37:02.232 INFO:tasks.workunit.client.0.smithi032.stdout:7/220: write d0/f37 [4633113,120472] 0 2024-04-12T05:37:02.233 INFO:tasks.workunit.client.0.smithi032.stdout:8/195: dwrite d0/f7 [0,4194304] 0 2024-04-12T05:37:02.235 INFO:tasks.workunit.client.0.smithi032.stdout:7/221: chown d0/de/df/f24 244 1 2024-04-12T05:37:02.238 INFO:tasks.workunit.client.0.smithi032.stdout:8/196: symlink d0/db/dd/d11/d1b/d22/d35/l42 0 2024-04-12T05:37:02.241 INFO:tasks.workunit.client.0.smithi032.stdout:5/289: creat d2/df/f54 x:0 0 0 2024-04-12T05:37:02.242 INFO:tasks.workunit.client.0.smithi032.stdout:1/188: fdatasync d5/de/d17/d1b/f1d 0 2024-04-12T05:37:02.243 INFO:tasks.workunit.client.0.smithi032.stdout:8/197: dread d0/db/f19 [0,4194304] 0 2024-04-12T05:37:02.243 INFO:tasks.workunit.client.0.smithi032.stdout:7/222: chown d0/d1c/l3a 6833487 1 2024-04-12T05:37:02.244 INFO:tasks.workunit.client.0.smithi032.stdout:2/184: mkdir dd/d1b/d1c/d2c/d32 0 2024-04-12T05:37:02.245 INFO:tasks.workunit.client.0.smithi032.stdout:8/198: write d0/db/dd/d11/f2a [539251,122081] 0 2024-04-12T05:37:02.247 INFO:tasks.workunit.client.0.smithi032.stdout:1/189: mknod d5/de/c36 0 2024-04-12T05:37:02.250 INFO:tasks.workunit.client.0.smithi032.stdout:2/185: stat c3 0 2024-04-12T05:37:02.252 INFO:tasks.workunit.client.0.smithi032.stdout:7/223: rename d0/d13/d25 to d0/d1c/d4c 0 2024-04-12T05:37:02.252 INFO:tasks.workunit.client.0.smithi032.stdout:4/228: truncate d4/d13/ff 1456859 0 2024-04-12T05:37:02.256 INFO:tasks.workunit.client.0.smithi032.stdout:7/224: creat d0/d1c/d4c/f4d x:0 0 0 2024-04-12T05:37:02.257 INFO:tasks.workunit.client.0.smithi032.stdout:8/199: mknod d0/c43 0 2024-04-12T05:37:02.259 INFO:tasks.workunit.client.0.smithi032.stdout:2/186: creat dd/d1b/d1c/d2c/d32/f33 x:0 0 0 2024-04-12T05:37:02.262 INFO:tasks.workunit.client.0.smithi032.stdout:7/225: write d0/de/df/f14 [4498156,26464] 0 2024-04-12T05:37:02.263 INFO:tasks.workunit.client.0.smithi032.stdout:4/229: rename d4/d13/c45 to d4/d13/d19/c4b 0 2024-04-12T05:37:02.264 INFO:tasks.workunit.client.0.smithi032.stdout:9/232: dwrite d9/d15/f44 [0,4194304] 0 2024-04-12T05:37:02.267 INFO:tasks.workunit.client.0.smithi032.stdout:7/226: read d0/d13/f27 [63059,35719] 0 2024-04-12T05:37:02.268 INFO:tasks.workunit.client.0.smithi032.stdout:8/200: mkdir d0/db/dd/d11/d1b/d22/d35/d44 0 2024-04-12T05:37:02.268 INFO:tasks.workunit.client.0.smithi032.stdout:9/233: chown d9/d15/d1a 998 1 2024-04-12T05:37:02.270 INFO:tasks.workunit.client.0.smithi032.stdout:6/226: dwrite d4/f1b [0,4194304] 0 2024-04-12T05:37:02.270 INFO:tasks.workunit.client.0.smithi032.stdout:7/227: fsync d0/de/df/f1e 0 2024-04-12T05:37:02.276 INFO:tasks.workunit.client.0.smithi032.stdout:1/190: dread d5/de/f11 [0,4194304] 0 2024-04-12T05:37:02.278 INFO:tasks.workunit.client.0.smithi032.stdout:6/227: creat d4/d14/d3c/f48 x:0 0 0 2024-04-12T05:37:02.278 INFO:tasks.workunit.client.0.smithi032.stdout:4/230: dwrite d4/f1d [0,4194304] 0 2024-04-12T05:37:02.278 INFO:tasks.workunit.client.0.smithi032.stdout:7/228: chown d0/l1 625000 1 2024-04-12T05:37:02.279 INFO:tasks.workunit.client.0.smithi032.stdout:9/234: truncate d9/f34 5054928 0 2024-04-12T05:37:02.281 INFO:tasks.workunit.client.0.smithi032.stdout:1/191: mknod d5/c37 0 2024-04-12T05:37:02.281 INFO:tasks.workunit.client.0.smithi032.stdout:9/235: fsync f5 0 2024-04-12T05:37:02.286 INFO:tasks.workunit.client.0.smithi032.stdout:9/236: dread d9/d15/f28 [0,4194304] 0 2024-04-12T05:37:02.290 INFO:tasks.workunit.client.0.smithi032.stdout:4/231: truncate d4/d13/d19/d28/d2f/d3b/f3c 86174 0 2024-04-12T05:37:02.290 INFO:tasks.workunit.client.0.smithi032.stdout:1/192: unlink d5/de/f19 0 2024-04-12T05:37:02.292 INFO:tasks.workunit.client.0.smithi032.stdout:6/228: dread d4/d14/f15 [0,4194304] 0 2024-04-12T05:37:02.294 INFO:tasks.workunit.client.0.smithi032.stdout:2/187: dwrite f1 [0,4194304] 0 2024-04-12T05:37:02.297 INFO:tasks.workunit.client.0.smithi032.stdout:7/229: dread d0/f5 [0,4194304] 0 2024-04-12T05:37:02.297 INFO:tasks.workunit.client.0.smithi032.stdout:6/229: unlink d4/la 0 2024-04-12T05:37:02.298 INFO:tasks.workunit.client.0.smithi032.stdout:1/193: rename d5/d2c/f2f to d5/de/d17/d1b/f38 0 2024-04-12T05:37:02.302 INFO:tasks.workunit.client.0.smithi032.stdout:6/230: rmdir d4/d21/d2e 39 2024-04-12T05:37:02.304 INFO:tasks.workunit.client.0.smithi032.stdout:1/194: read d5/de/d17/d1b/f1d [203696,58376] 0 2024-04-12T05:37:02.305 INFO:tasks.workunit.client.0.smithi032.stdout:2/188: link dd/d1b/d1c/d2c/d32/f33 dd/d1b/f34 0 2024-04-12T05:37:02.305 INFO:tasks.workunit.client.0.smithi032.stdout:9/237: dread d9/d15/d2c/d41/f45 [0,4194304] 0 2024-04-12T05:37:02.306 INFO:tasks.workunit.client.0.smithi032.stdout:4/232: dwrite d4/d13/d19/f30 [0,4194304] 0 2024-04-12T05:37:02.307 INFO:tasks.workunit.client.0.smithi032.stdout:6/231: creat d4/d14/d1a/f49 x:0 0 0 2024-04-12T05:37:02.309 INFO:tasks.workunit.client.0.smithi032.stdout:4/233: read d4/d13/d19/f34 [2736953,89695] 0 2024-04-12T05:37:02.309 INFO:tasks.workunit.client.0.smithi032.stdout:7/230: dread d0/f9 [0,4194304] 0 2024-04-12T05:37:02.310 INFO:tasks.workunit.client.0.smithi032.stdout:1/195: mkdir d5/de/d2a/d39 0 2024-04-12T05:37:02.313 INFO:tasks.workunit.client.0.smithi032.stdout:7/231: stat d0/d13/d21/d30/d31/c41 0 2024-04-12T05:37:02.316 INFO:tasks.workunit.client.0.smithi032.stdout:1/196: dread d5/de/d17/d1b/f1d [0,4194304] 0 2024-04-12T05:37:02.318 INFO:tasks.workunit.client.0.smithi032.stdout:2/189: dwrite dd/ff [0,4194304] 0 2024-04-12T05:37:02.318 INFO:tasks.workunit.client.0.smithi032.stdout:7/232: write d0/de/df/f14 [1287402,94331] 0 2024-04-12T05:37:02.320 INFO:tasks.workunit.client.0.smithi032.stdout:6/232: write d4/d14/d1a/f31 [421038,6755] 0 2024-04-12T05:37:02.323 INFO:tasks.workunit.client.0.smithi032.stdout:1/197: truncate d5/d2c/f2d 165061 0 2024-04-12T05:37:02.324 INFO:tasks.workunit.client.0.smithi032.stdout:7/233: creat d0/d1c/f4e x:0 0 0 2024-04-12T05:37:02.324 INFO:tasks.workunit.client.0.smithi032.stdout:6/233: mknod d4/d14/c4a 0 2024-04-12T05:37:02.327 INFO:tasks.workunit.client.0.smithi032.stdout:4/234: dwrite d4/d13/d19/f2a [0,4194304] 0 2024-04-12T05:37:02.331 INFO:tasks.workunit.client.0.smithi032.stdout:7/234: write d0/de/f43 [1022598,108913] 0 2024-04-12T05:37:02.333 INFO:tasks.workunit.client.0.smithi032.stdout:9/238: dwrite d9/d15/d1a/d24/f40 [8388608,4194304] 0 2024-04-12T05:37:02.336 INFO:tasks.workunit.client.0.smithi032.stdout:1/198: rename d5/de/c16 to d5/de/d17/d35/c3a 0 2024-04-12T05:37:02.337 INFO:tasks.workunit.client.0.smithi032.stdout:2/190: dwrite dd/f16 [0,4194304] 0 2024-04-12T05:37:02.338 INFO:tasks.workunit.client.0.smithi032.stdout:7/235: rmdir d0/d13/d21/d30 39 2024-04-12T05:37:02.339 INFO:tasks.workunit.client.0.smithi032.stdout:7/236: stat d0/de/df/d20 0 2024-04-12T05:37:02.339 INFO:tasks.workunit.client.0.smithi032.stdout:6/234: rmdir d4 39 2024-04-12T05:37:02.342 INFO:tasks.workunit.client.0.smithi032.stdout:1/199: readlink d5/de/d17/l22 0 2024-04-12T05:37:02.344 INFO:tasks.workunit.client.0.smithi032.stdout:2/191: write dd/f14 [2820209,26114] 0 2024-04-12T05:37:02.345 INFO:tasks.workunit.client.0.smithi032.stdout:2/192: chown dd 105630 1 2024-04-12T05:37:02.345 INFO:tasks.workunit.client.0.smithi032.stdout:7/237: symlink d0/d13/d21/l4f 0 2024-04-12T05:37:02.346 INFO:tasks.workunit.client.0.smithi032.stdout:4/235: getdents d4/d13/d19/d28/d2f/d46 0 2024-04-12T05:37:02.348 INFO:tasks.workunit.client.0.smithi032.stdout:1/200: mkdir d5/de/d17/d35/d3b 0 2024-04-12T05:37:02.349 INFO:tasks.workunit.client.0.smithi032.stdout:2/193: truncate dd/d1b/f34 919162 0 2024-04-12T05:37:02.350 INFO:tasks.workunit.client.0.smithi032.stdout:7/238: write d0/d13/d21/f45 [453469,61852] 0 2024-04-12T05:37:02.351 INFO:tasks.workunit.client.0.smithi032.stdout:9/239: dwrite d9/d15/d2c/d41/f45 [0,4194304] 0 2024-04-12T05:37:02.351 INFO:tasks.workunit.client.0.smithi032.stdout:4/236: truncate d4/f38 284551 0 2024-04-12T05:37:02.353 INFO:tasks.workunit.client.0.smithi032.stdout:1/201: creat d5/d2c/f3c x:0 0 0 2024-04-12T05:37:02.354 INFO:tasks.workunit.client.0.smithi032.stdout:6/235: getdents d4/d14/d1a 0 2024-04-12T05:37:02.355 INFO:tasks.workunit.client.0.smithi032.stdout:4/237: unlink d4/d13/l21 0 2024-04-12T05:37:02.355 INFO:tasks.workunit.client.0.smithi032.stdout:2/194: write dd/d1b/d1c/f25 [128161,77549] 0 2024-04-12T05:37:02.357 INFO:tasks.workunit.client.0.smithi032.stdout:6/236: dread - d4/d14/f46 zero size 2024-04-12T05:37:02.357 INFO:tasks.workunit.client.0.smithi032.stdout:2/195: stat dd/d19/l26 0 2024-04-12T05:37:02.358 INFO:tasks.workunit.client.0.smithi032.stdout:6/237: dread - d4/d14/f46 zero size 2024-04-12T05:37:02.358 INFO:tasks.workunit.client.0.smithi032.stdout:3/217: sync 2024-04-12T05:37:02.359 INFO:tasks.workunit.client.0.smithi032.stdout:0/317: sync 2024-04-12T05:37:02.359 INFO:tasks.workunit.client.0.smithi032.stdout:1/202: creat d5/de/d17/d35/f3d x:0 0 0 2024-04-12T05:37:02.361 INFO:tasks.workunit.client.0.smithi032.stdout:0/318: mkdir d2/d8/d66 0 2024-04-12T05:37:02.364 INFO:tasks.workunit.client.0.smithi032.stdout:2/196: rename dd/d19/f1d to dd/d1b/d1c/d2c/d32/f35 0 2024-04-12T05:37:02.364 INFO:tasks.workunit.client.0.smithi032.stdout:2/197: chown f8 13791141 1 2024-04-12T05:37:02.366 INFO:tasks.workunit.client.0.smithi032.stdout:1/203: mkdir d5/de/d17/d35/d3e 0 2024-04-12T05:37:02.367 INFO:tasks.workunit.client.0.smithi032.stdout:1/204: truncate d5/de/f28 3141138 0 2024-04-12T05:37:02.368 INFO:tasks.workunit.client.0.smithi032.stdout:0/319: write d2/dc/d3e/f53 [2955922,124950] 0 2024-04-12T05:37:02.369 INFO:tasks.workunit.client.0.smithi032.stdout:3/218: creat d12/d2e/f42 x:0 0 0 2024-04-12T05:37:02.370 INFO:tasks.workunit.client.0.smithi032.stdout:3/219: dread - d12/d2e/f42 zero size 2024-04-12T05:37:02.371 INFO:tasks.workunit.client.0.smithi032.stdout:2/198: unlink dd/d1b/d1c/c1f 0 2024-04-12T05:37:02.372 INFO:tasks.workunit.client.0.smithi032.stdout:2/199: write fc [1358351,1345] 0 2024-04-12T05:37:02.372 INFO:tasks.workunit.client.0.smithi032.stdout:7/239: fdatasync d0/d13/d21/f45 0 2024-04-12T05:37:02.373 INFO:tasks.workunit.client.0.smithi032.stdout:1/205: creat d5/de/d2a/f3f x:0 0 0 2024-04-12T05:37:02.373 INFO:tasks.workunit.client.0.smithi032.stdout:0/320: dread d2/d8/df/f31 [0,4194304] 0 2024-04-12T05:37:02.374 INFO:tasks.workunit.client.0.smithi032.stdout:7/240: stat d0/de/df/d29/f46 0 2024-04-12T05:37:02.375 INFO:tasks.workunit.client.0.smithi032.stdout:1/206: dread - d5/de/d2a/f2b zero size 2024-04-12T05:37:02.375 INFO:tasks.workunit.client.0.smithi032.stdout:0/321: write d2/d29/f4a [1789650,122272] 0 2024-04-12T05:37:02.377 INFO:tasks.workunit.client.0.smithi032.stdout:6/238: getdents d4/d21/d2e/d32 0 2024-04-12T05:37:02.377 INFO:tasks.workunit.client.0.smithi032.stdout:1/207: rename d5/de/d2a to d5/de/d2a/d40 22 2024-04-12T05:37:02.377 INFO:tasks.workunit.client.0.smithi032.stdout:9/240: write d9/d1f/f46 [922571,1355] 0 2024-04-12T05:37:02.381 INFO:tasks.workunit.client.0.smithi032.stdout:1/208: write d5/de/d17/d1b/f38 [795165,7972] 0 2024-04-12T05:37:02.418 INFO:tasks.workunit.client.0.smithi032.stdout:6/239: rename d4/d14/f15 to d4/d21/d27/f4b 0 2024-04-12T05:37:02.421 INFO:tasks.workunit.client.0.smithi032.stdout:6/240: read d4/f5 [61316,2786] 0 2024-04-12T05:37:02.422 INFO:tasks.workunit.client.0.smithi032.stdout:0/322: dwrite d2/d8/df/f52 [0,4194304] 0 2024-04-12T05:37:02.459 INFO:tasks.workunit.client.0.smithi032.stdout:5/290: sync 2024-04-12T05:37:02.460 INFO:tasks.workunit.client.0.smithi032.stdout:9/241: dwrite d9/d15/f43 [0,4194304] 0 2024-04-12T05:37:02.461 INFO:tasks.workunit.client.0.smithi032.stdout:0/323: symlink d2/dc/d41/l67 0 2024-04-12T05:37:02.462 INFO:tasks.workunit.client.0.smithi032.stdout:9/242: write f5 [909522,103600] 0 2024-04-12T05:37:02.463 INFO:tasks.workunit.client.0.smithi032.stdout:5/291: write d2/df/d31/f46 [892770,36275] 0 2024-04-12T05:37:02.465 INFO:tasks.workunit.client.0.smithi032.stdout:7/241: dread d0/d13/f16 [0,4194304] 0 2024-04-12T05:37:02.467 INFO:tasks.workunit.client.0.smithi032.stdout:5/292: fsync d2/df/d25/d1e/f1f 0 2024-04-12T05:37:02.468 INFO:tasks.workunit.client.0.smithi032.stdout:0/324: link d2/d29/f4a d2/f68 0 2024-04-12T05:37:02.470 INFO:tasks.workunit.client.0.smithi032.stdout:6/241: dwrite d4/d14/d3c/f41 [4194304,4194304] 0 2024-04-12T05:37:02.470 INFO:tasks.workunit.client.0.smithi032.stdout:9/243: creat d9/d15/d1a/d24/d27/f50 x:0 0 0 2024-04-12T05:37:02.472 INFO:tasks.workunit.client.0.smithi032.stdout:0/325: truncate d2/d28/d50/f3f 1007943 0 2024-04-12T05:37:02.480 INFO:tasks.workunit.client.0.smithi032.stdout:5/293: rename d2/dd/d53 to d2/df/d25/d50/d55 0 2024-04-12T05:37:02.484 INFO:tasks.workunit.client.0.smithi032.stdout:9/244: dwrite d9/d1f/f46 [0,4194304] 0 2024-04-12T05:37:02.484 INFO:tasks.workunit.client.0.smithi032.stdout:0/326: creat d2/d8/d17/d1b/d20/f69 x:0 0 0 2024-04-12T05:37:02.490 INFO:tasks.workunit.client.0.smithi032.stdout:0/327: write d2/d28/d50/f35 [4417,78368] 0 2024-04-12T05:37:02.492 INFO:tasks.workunit.client.0.smithi032.stdout:6/242: dread f3 [0,4194304] 0 2024-04-12T05:37:02.493 INFO:tasks.workunit.client.0.smithi032.stdout:9/245: creat d9/d15/d2c/d41/f51 x:0 0 0 2024-04-12T05:37:02.493 INFO:tasks.workunit.client.0.smithi032.stdout:7/242: dwrite d0/d13/f27 [0,4194304] 0 2024-04-12T05:37:02.496 INFO:tasks.workunit.client.0.smithi032.stdout:6/243: rmdir d4/d21 39 2024-04-12T05:37:02.498 INFO:tasks.workunit.client.0.smithi032.stdout:0/328: creat d2/d8/d17/f6a x:0 0 0 2024-04-12T05:37:02.498 INFO:tasks.workunit.client.0.smithi032.stdout:9/246: stat d9/d15/d1a/d1d/c29 0 2024-04-12T05:37:02.498 INFO:tasks.workunit.client.0.smithi032.stdout:5/294: link d2/dd/f2d d2/d3c/f56 0 2024-04-12T05:37:02.500 INFO:tasks.workunit.client.0.smithi032.stdout:9/247: stat d9/le 0 2024-04-12T05:37:02.501 INFO:tasks.workunit.client.0.smithi032.stdout:5/295: symlink d2/d17/l57 0 2024-04-12T05:37:02.503 INFO:tasks.workunit.client.0.smithi032.stdout:9/248: readlink d9/l3c 0 2024-04-12T05:37:02.508 INFO:tasks.workunit.client.0.smithi032.stdout:6/244: rename d4/d21/d2e/l40 to d4/d14/l4c 0 2024-04-12T05:37:02.509 INFO:tasks.workunit.client.0.smithi032.stdout:0/329: dwrite d2/f6 [0,4194304] 0 2024-04-12T05:37:02.509 INFO:tasks.workunit.client.0.smithi032.stdout:5/296: fsync d2/f4 0 2024-04-12T05:37:02.511 INFO:tasks.workunit.client.0.smithi032.stdout:7/243: truncate d0/de/df/f14 2144098 0 2024-04-12T05:37:02.515 INFO:tasks.workunit.client.0.smithi032.stdout:5/297: symlink d2/l58 0 2024-04-12T05:37:02.517 INFO:tasks.workunit.client.0.smithi032.stdout:9/249: dwrite d9/d15/d2c/d41/f45 [0,4194304] 0 2024-04-12T05:37:02.518 INFO:tasks.workunit.client.0.smithi032.stdout:0/330: rename d2/d8/f33 to d2/d8/d17/f6b 0 2024-04-12T05:37:02.523 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:02 smithi067 ceph-mon[49010]: pgmap v262: 65 pgs: 65 active+clean; 471 MiB data, 3.0 GiB used, 533 GiB / 536 GiB avail; 2.0 MiB/s rd, 29 MiB/s wr, 283 op/s 2024-04-12T05:37:02.524 INFO:tasks.workunit.client.0.smithi032.stdout:7/244: mknod d0/c50 0 2024-04-12T05:37:02.524 INFO:tasks.workunit.client.0.smithi032.stdout:7/245: chown d0/d13 7 1 2024-04-12T05:37:02.525 INFO:tasks.workunit.client.0.smithi032.stdout:9/250: creat d9/d15/d2c/f52 x:0 0 0 2024-04-12T05:37:02.525 INFO:tasks.workunit.client.0.smithi032.stdout:5/298: read d2/dd/f2d [94104,122329] 0 2024-04-12T05:37:02.525 INFO:tasks.workunit.client.0.smithi032.stdout:0/331: rmdir d2/d8/df/d24/d60 39 2024-04-12T05:37:02.528 INFO:tasks.workunit.client.0.smithi032.stdout:0/332: write d2/d28/d50/f3f [1837326,62413] 0 2024-04-12T05:37:02.530 INFO:tasks.workunit.client.0.smithi032.stdout:9/251: dread f5 [0,4194304] 0 2024-04-12T05:37:02.531 INFO:tasks.workunit.client.0.smithi032.stdout:5/299: creat d2/df/d25/d1e/f59 x:0 0 0 2024-04-12T05:37:02.533 INFO:tasks.workunit.client.0.smithi032.stdout:9/252: write d9/d15/d1a/d1d/f4d [1360000,72700] 0 2024-04-12T05:37:02.533 INFO:tasks.workunit.client.0.smithi032.stdout:7/246: dread d0/de/f44 [0,4194304] 0 2024-04-12T05:37:02.534 INFO:tasks.workunit.client.0.smithi032.stdout:0/333: creat d2/d8/f6c x:0 0 0 2024-04-12T05:37:02.536 INFO:tasks.workunit.client.0.smithi032.stdout:9/253: creat d9/d32/f53 x:0 0 0 2024-04-12T05:37:02.537 INFO:tasks.workunit.client.0.smithi032.stdout:5/300: read d2/df/d25/d1e/f2f [3171864,89290] 0 2024-04-12T05:37:02.545 INFO:tasks.workunit.client.0.smithi032.stdout:0/334: dread d2/f68 [0,4194304] 0 2024-04-12T05:37:02.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:02 smithi032 ceph-mon[39731]: pgmap v262: 65 pgs: 65 active+clean; 471 MiB data, 3.0 GiB used, 533 GiB / 536 GiB avail; 2.0 MiB/s rd, 29 MiB/s wr, 283 op/s 2024-04-12T05:37:02.593 INFO:tasks.workunit.client.0.smithi032.stdout:4/238: sync 2024-04-12T05:37:02.594 INFO:tasks.workunit.client.0.smithi032.stdout:8/201: sync 2024-04-12T05:37:02.599 INFO:tasks.workunit.client.0.smithi032.stdout:8/202: dwrite d0/db/fc [0,4194304] 0 2024-04-12T05:37:02.617 INFO:tasks.workunit.client.0.smithi032.stdout:1/209: sync 2024-04-12T05:37:02.617 INFO:tasks.workunit.client.0.smithi032.stdout:2/200: sync 2024-04-12T05:37:02.617 INFO:tasks.workunit.client.0.smithi032.stdout:3/220: sync 2024-04-12T05:37:02.619 INFO:tasks.workunit.client.0.smithi032.stdout:3/221: truncate d12/d2e/d3a/f3b 84690 0 2024-04-12T05:37:02.620 INFO:tasks.workunit.client.0.smithi032.stdout:3/222: dread d12/d2e/d3a/f3b [0,4194304] 0 2024-04-12T05:37:02.620 INFO:tasks.workunit.client.0.smithi032.stdout:3/223: fsync d12/f1b 0 2024-04-12T05:37:02.621 INFO:tasks.workunit.client.0.smithi032.stdout:3/224: write f10 [2488052,59467] 0 2024-04-12T05:37:02.627 INFO:tasks.workunit.client.0.smithi032.stdout:3/225: dwrite d12/d2f/f3d [0,4194304] 0 2024-04-12T05:37:02.628 INFO:tasks.workunit.client.0.smithi032.stdout:3/226: write d12/d2e/d3a/f31 [1053856,129619] 0 2024-04-12T05:37:02.654 INFO:tasks.workunit.client.0.smithi032.stdout:9/254: mknod d9/d32/c54 0 2024-04-12T05:37:02.655 INFO:tasks.workunit.client.0.smithi032.stdout:9/255: write d9/d15/d1a/d24/d27/f50 [427274,15032] 0 2024-04-12T05:37:02.656 INFO:tasks.workunit.client.0.smithi032.stdout:9/256: stat d9/d15/f30 0 2024-04-12T05:37:02.656 INFO:tasks.workunit.client.0.smithi032.stdout:5/301: creat d2/df/d25/d50/d55/f5a x:0 0 0 2024-04-12T05:37:02.657 INFO:tasks.workunit.client.0.smithi032.stdout:6/245: truncate d4/d14/d3c/f41 4576206 0 2024-04-12T05:37:02.658 INFO:tasks.workunit.client.0.smithi032.stdout:6/246: read - d4/d14/d3c/f48 zero size 2024-04-12T05:37:02.660 INFO:tasks.workunit.client.0.smithi032.stdout:0/335: creat d2/d8/d17/d1b/d20/f6d x:0 0 0 2024-04-12T05:37:02.664 INFO:tasks.workunit.client.0.smithi032.stdout:8/203: creat d0/db/dd/d11/d1b/d22/f45 x:0 0 0 2024-04-12T05:37:02.669 INFO:tasks.workunit.client.0.smithi032.stdout:3/227: symlink d12/d2f/l43 0 2024-04-12T05:37:02.670 INFO:tasks.workunit.client.0.smithi032.stdout:9/257: mkdir d9/d15/d1a/d1d/d55 0 2024-04-12T05:37:02.671 INFO:tasks.workunit.client.0.smithi032.stdout:6/247: mkdir d4/d21/d2e/d4d 0 2024-04-12T05:37:02.674 INFO:tasks.workunit.client.0.smithi032.stdout:7/247: dwrite d0/d13/f16 [0,4194304] 0 2024-04-12T05:37:02.675 INFO:tasks.workunit.client.0.smithi032.stdout:1/210: mknod d5/de/d17/d35/d3e/c41 0 2024-04-12T05:37:02.679 INFO:tasks.workunit.client.0.smithi032.stdout:2/201: dread dd/f1e [0,4194304] 0 2024-04-12T05:37:02.679 INFO:tasks.workunit.client.0.smithi032.stdout:2/202: fsync dd/f14 0 2024-04-12T05:37:02.679 INFO:tasks.workunit.client.0.smithi032.stdout:2/203: chown dd/d19/d20 183 1 2024-04-12T05:37:02.680 INFO:tasks.workunit.client.0.smithi032.stdout:4/239: rename d4/d13/d19/d28/d2f/d46/c49 to d4/d13/c4c 0 2024-04-12T05:37:02.681 INFO:tasks.workunit.client.0.smithi032.stdout:6/248: rmdir d4/d14 39 2024-04-12T05:37:02.681 INFO:tasks.workunit.client.0.smithi032.stdout:4/240: chown d4/d13/d19/d28/d2f/d46/l4a 13317 1 2024-04-12T05:37:02.683 INFO:tasks.workunit.client.0.smithi032.stdout:7/248: mkdir d0/dd/d51 0 2024-04-12T05:37:02.684 INFO:tasks.workunit.client.0.smithi032.stdout:1/211: mknod d5/de/d17/d35/c42 0 2024-04-12T05:37:02.686 INFO:tasks.workunit.client.0.smithi032.stdout:3/228: mknod d12/d2e/c44 0 2024-04-12T05:37:02.687 INFO:tasks.workunit.client.0.smithi032.stdout:2/204: mkdir dd/d1b/d1c/d2c/d36 0 2024-04-12T05:37:02.689 INFO:tasks.workunit.client.0.smithi032.stdout:5/302: rename d2/d17 to d2/d3c/d5b 0 2024-04-12T05:37:02.691 INFO:tasks.workunit.client.0.smithi032.stdout:2/205: dread dd/d1b/d1c/d2c/d32/f33 [0,4194304] 0 2024-04-12T05:37:02.693 INFO:tasks.workunit.client.0.smithi032.stdout:1/212: creat d5/d2c/f43 x:0 0 0 2024-04-12T05:37:02.693 INFO:tasks.workunit.client.0.smithi032.stdout:3/229: dwrite d12/d2e/f36 [0,4194304] 0 2024-04-12T05:37:02.694 INFO:tasks.workunit.client.0.smithi032.stdout:3/230: write d12/d24/f2b [493578,109273] 0 2024-04-12T05:37:02.697 INFO:tasks.workunit.client.0.smithi032.stdout:6/249: mknod d4/d21/c4e 0 2024-04-12T05:37:02.699 INFO:tasks.workunit.client.0.smithi032.stdout:0/336: getdents d2/d8/df 0 2024-04-12T05:37:02.700 INFO:tasks.workunit.client.0.smithi032.stdout:0/337: fsync d2/d28/f2e 0 2024-04-12T05:37:02.702 INFO:tasks.workunit.client.0.smithi032.stdout:8/204: rename d0/db/dd/d11/d1b/d1e/l30 to d0/db/dd/d11/d1b/d22/l46 0 2024-04-12T05:37:02.703 INFO:tasks.workunit.client.0.smithi032.stdout:4/241: mkdir d4/d13/d19/d28/d4d 0 2024-04-12T05:37:02.704 INFO:tasks.workunit.client.0.smithi032.stdout:7/249: creat d0/d1c/d4c/d26/f52 x:0 0 0 2024-04-12T05:37:02.707 INFO:tasks.workunit.client.0.smithi032.stdout:3/231: chown d12/f13 0 1 2024-04-12T05:37:02.710 INFO:tasks.workunit.client.0.smithi032.stdout:9/258: rename d9/d32/f37 to d9/d15/d1a/f56 0 2024-04-12T05:37:02.710 INFO:tasks.workunit.client.0.smithi032.stdout:8/205: symlink d0/db/d38/l47 0 2024-04-12T05:37:02.710 INFO:tasks.workunit.client.0.smithi032.stdout:9/259: stat l8 0 2024-04-12T05:37:02.712 INFO:tasks.workunit.client.0.smithi032.stdout:8/206: readlink d0/d36/l37 0 2024-04-12T05:37:02.712 INFO:tasks.workunit.client.0.smithi032.stdout:4/242: write d4/d13/d19/f34 [1883500,98183] 0 2024-04-12T05:37:02.712 INFO:tasks.workunit.client.0.smithi032.stdout:2/206: creat dd/d1b/d1c/d2c/d36/f37 x:0 0 0 2024-04-12T05:37:02.713 INFO:tasks.workunit.client.0.smithi032.stdout:6/250: mknod d4/d14/d3c/c4f 0 2024-04-12T05:37:02.716 INFO:tasks.workunit.client.0.smithi032.stdout:2/207: read - dd/d19/f2d zero size 2024-04-12T05:37:02.716 INFO:tasks.workunit.client.0.smithi032.stdout:3/232: dread d12/d2e/f1e [0,4194304] 0 2024-04-12T05:37:02.716 INFO:tasks.workunit.client.0.smithi032.stdout:5/303: dwrite d2/dd/f3a [0,4194304] 0 2024-04-12T05:37:02.718 INFO:tasks.workunit.client.0.smithi032.stdout:7/250: link d0/de/df/c39 d0/d13/d21/d30/d31/c53 0 2024-04-12T05:37:02.719 INFO:tasks.workunit.client.0.smithi032.stdout:0/338: rename d2/dc/c21 to d2/d8/df/d24/c6e 0 2024-04-12T05:37:02.719 INFO:tasks.workunit.client.0.smithi032.stdout:4/243: dread - d4/d13/d19/d28/d32/f3e zero size 2024-04-12T05:37:02.721 INFO:tasks.workunit.client.0.smithi032.stdout:3/233: write d12/d2f/f3d [1237398,45176] 0 2024-04-12T05:37:02.727 INFO:tasks.workunit.client.0.smithi032.stdout:7/251: write d0/d1c/f1f [857464,113132] 0 2024-04-12T05:37:02.727 INFO:tasks.workunit.client.0.smithi032.stdout:6/251: dread d4/d21/d27/f38 [0,4194304] 0 2024-04-12T05:37:02.727 INFO:tasks.workunit.client.0.smithi032.stdout:5/304: creat d2/df/f5c x:0 0 0 2024-04-12T05:37:02.728 INFO:tasks.workunit.client.0.smithi032.stdout:0/339: symlink d2/l6f 0 2024-04-12T05:37:02.728 INFO:tasks.workunit.client.0.smithi032.stdout:6/252: fdatasync d4/ff 0 2024-04-12T05:37:02.728 INFO:tasks.workunit.client.0.smithi032.stdout:4/244: creat d4/d13/d19/d28/d2f/f4e x:0 0 0 2024-04-12T05:37:02.729 INFO:tasks.workunit.client.0.smithi032.stdout:8/207: dwrite d0/f3 [0,4194304] 0 2024-04-12T05:37:02.730 INFO:tasks.workunit.client.0.smithi032.stdout:0/340: chown d2/d8/d17/d1b/c1e 900921 1 2024-04-12T05:37:02.730 INFO:tasks.workunit.client.0.smithi032.stdout:3/234: unlink d12/d2e/d3a/d2d/l37 0 2024-04-12T05:37:02.731 INFO:tasks.workunit.client.0.smithi032.stdout:8/208: fdatasync d0/db/dd/d11/f2a 0 2024-04-12T05:37:02.733 INFO:tasks.workunit.client.0.smithi032.stdout:7/252: mknod d0/c54 0 2024-04-12T05:37:02.734 INFO:tasks.workunit.client.0.smithi032.stdout:4/245: readlink d4/d13/l2e 0 2024-04-12T05:37:02.738 INFO:tasks.workunit.client.0.smithi032.stdout:0/341: stat d2/d29/l2a 0 2024-04-12T05:37:02.738 INFO:tasks.workunit.client.0.smithi032.stdout:0/342: chown d2/d28/f2e 179301 1 2024-04-12T05:37:02.739 INFO:tasks.workunit.client.0.smithi032.stdout:5/305: mkdir d2/dd/d5d 0 2024-04-12T05:37:02.740 INFO:tasks.workunit.client.0.smithi032.stdout:5/306: fdatasync d2/df/d25/d43/f51 0 2024-04-12T05:37:02.744 INFO:tasks.workunit.client.0.smithi032.stdout:7/253: creat d0/dd/f55 x:0 0 0 2024-04-12T05:37:02.747 INFO:tasks.workunit.client.0.smithi032.stdout:0/343: write d2/d8/d17/f6b [4840592,95593] 0 2024-04-12T05:37:02.748 INFO:tasks.workunit.client.0.smithi032.stdout:4/246: dread d4/d13/f1c [0,4194304] 0 2024-04-12T05:37:02.751 INFO:tasks.workunit.client.0.smithi032.stdout:0/344: creat d2/d29/f70 x:0 0 0 2024-04-12T05:37:02.753 INFO:tasks.workunit.client.0.smithi032.stdout:7/254: mknod d0/dd/d51/c56 0 2024-04-12T05:37:02.753 INFO:tasks.workunit.client.0.smithi032.stdout:0/345: read d2/f68 [2762531,1476] 0 2024-04-12T05:37:02.754 INFO:tasks.workunit.client.0.smithi032.stdout:6/253: dwrite d4/f5 [0,4194304] 0 2024-04-12T05:37:02.754 INFO:tasks.workunit.client.0.smithi032.stdout:3/235: getdents d12/d24 0 2024-04-12T05:37:02.755 INFO:tasks.workunit.client.0.smithi032.stdout:5/307: link d2/df/f1c d2/d3c/d5b/f5e 0 2024-04-12T05:37:02.756 INFO:tasks.workunit.client.0.smithi032.stdout:4/247: chown d4/d13/f33 1833626 1 2024-04-12T05:37:02.757 INFO:tasks.workunit.client.0.smithi032.stdout:8/209: link d0/db/dd/d11/d1b/d22/l46 d0/db/dd/d11/d1b/d22/d35/l48 0 2024-04-12T05:37:02.757 INFO:tasks.workunit.client.0.smithi032.stdout:4/248: fsync d4/d13/f1f 0 2024-04-12T05:37:02.760 INFO:tasks.workunit.client.0.smithi032.stdout:4/249: chown d4/d13/d19/d28 0 1 2024-04-12T05:37:02.761 INFO:tasks.workunit.client.0.smithi032.stdout:3/236: creat d12/d2e/d3a/d2d/f45 x:0 0 0 2024-04-12T05:37:02.764 INFO:tasks.workunit.client.0.smithi032.stdout:5/308: symlink d2/df/d25/d50/d55/l5f 0 2024-04-12T05:37:02.766 INFO:tasks.workunit.client.0.smithi032.stdout:0/346: creat d2/d8/df/d24/d60/d63/f71 x:0 0 0 2024-04-12T05:37:02.767 INFO:tasks.workunit.client.0.smithi032.stdout:8/210: symlink d0/db/dd/d11/d1b/l49 0 2024-04-12T05:37:02.769 INFO:tasks.workunit.client.0.smithi032.stdout:8/211: write d0/db/dd/d11/f40 [338923,45455] 0 2024-04-12T05:37:02.769 INFO:tasks.workunit.client.0.smithi032.stdout:7/255: creat d0/d13/d21/f57 x:0 0 0 2024-04-12T05:37:02.771 INFO:tasks.workunit.client.0.smithi032.stdout:0/347: mknod d2/d8/d17/d1b/d20/c72 0 2024-04-12T05:37:02.771 INFO:tasks.workunit.client.0.smithi032.stdout:4/250: mknod d4/d13/d19/d28/c4f 0 2024-04-12T05:37:02.776 INFO:tasks.workunit.client.0.smithi032.stdout:0/348: unlink d2/d8/c1d 0 2024-04-12T05:37:02.776 INFO:tasks.workunit.client.0.smithi032.stdout:6/254: truncate d4/d14/f26 947714 0 2024-04-12T05:37:02.778 INFO:tasks.workunit.client.0.smithi032.stdout:5/309: creat d2/f60 x:0 0 0 2024-04-12T05:37:02.778 INFO:tasks.workunit.client.0.smithi032.stdout:0/349: dread - d2/d8/d17/d1b/d20/f69 zero size 2024-04-12T05:37:02.782 INFO:tasks.workunit.client.0.smithi032.stdout:3/237: rename ce to d12/d2e/c46 0 2024-04-12T05:37:02.782 INFO:tasks.workunit.client.0.smithi032.stdout:8/212: dread d0/db/dd/d11/f26 [0,4194304] 0 2024-04-12T05:37:02.783 INFO:tasks.workunit.client.0.smithi032.stdout:8/213: stat d0/ca 0 2024-04-12T05:37:02.784 INFO:tasks.workunit.client.0.smithi032.stdout:1/213: sync 2024-04-12T05:37:02.789 INFO:tasks.workunit.client.0.smithi032.stdout:0/350: dwrite d2/d29/f70 [0,4194304] 0 2024-04-12T05:37:02.790 INFO:tasks.workunit.client.0.smithi032.stdout:4/251: link d4/c10 d4/d13/d19/d28/d2f/c50 0 2024-04-12T05:37:02.793 INFO:tasks.workunit.client.0.smithi032.stdout:8/214: creat d0/d36/f4a x:0 0 0 2024-04-12T05:37:02.793 INFO:tasks.workunit.client.0.smithi032.stdout:1/214: symlink d5/de/d17/d1b/l44 0 2024-04-12T05:37:02.796 INFO:tasks.workunit.client.0.smithi032.stdout:5/310: rename d2/df/d25/l52 to d2/dd/l61 0 2024-04-12T05:37:02.796 INFO:tasks.workunit.client.0.smithi032.stdout:6/255: dwrite d4/f2c [0,4194304] 0 2024-04-12T05:37:02.797 INFO:tasks.workunit.client.0.smithi032.stdout:3/238: dread d12/d2e/f36 [0,4194304] 0 2024-04-12T05:37:02.798 INFO:tasks.workunit.client.0.smithi032.stdout:4/252: creat d4/d13/d19/d28/d2f/d46/f51 x:0 0 0 2024-04-12T05:37:02.801 INFO:tasks.workunit.client.0.smithi032.stdout:0/351: rename d2/d8/l5a to d2/d8/df/l73 0 2024-04-12T05:37:02.801 INFO:tasks.workunit.client.0.smithi032.stdout:7/256: dwrite d0/fc [0,4194304] 0 2024-04-12T05:37:02.804 INFO:tasks.workunit.client.0.smithi032.stdout:0/352: chown d2/c1f 407133557 1 2024-04-12T05:37:02.804 INFO:tasks.workunit.client.0.smithi032.stdout:3/239: creat d12/d24/f47 x:0 0 0 2024-04-12T05:37:02.806 INFO:tasks.workunit.client.0.smithi032.stdout:5/311: mknod d2/df/c62 0 2024-04-12T05:37:02.808 INFO:tasks.workunit.client.0.smithi032.stdout:5/312: readlink d2/df/d25/d50/d55/l5f 0 2024-04-12T05:37:02.810 INFO:tasks.workunit.client.0.smithi032.stdout:1/215: dwrite d5/de/d17/f26 [0,4194304] 0 2024-04-12T05:37:02.811 INFO:tasks.workunit.client.0.smithi032.stdout:5/313: write d2/df/f32 [237975,65700] 0 2024-04-12T05:37:02.812 INFO:tasks.workunit.client.0.smithi032.stdout:5/314: truncate d2/f60 761867 0 2024-04-12T05:37:02.812 INFO:tasks.workunit.client.0.smithi032.stdout:7/257: creat d0/d13/d21/d30/d31/f58 x:0 0 0 2024-04-12T05:37:02.815 INFO:tasks.workunit.client.0.smithi032.stdout:3/240: fdatasync d12/d2e/f1e 0 2024-04-12T05:37:02.815 INFO:tasks.workunit.client.0.smithi032.stdout:6/256: write d4/f2a [771015,27483] 0 2024-04-12T05:37:02.819 INFO:tasks.workunit.client.0.smithi032.stdout:4/253: write d4/d13/ff [1905972,52994] 0 2024-04-12T05:37:02.821 INFO:tasks.workunit.client.0.smithi032.stdout:3/241: creat d12/d2f/f48 x:0 0 0 2024-04-12T05:37:02.823 INFO:tasks.workunit.client.0.smithi032.stdout:3/242: dread - d12/d2e/f42 zero size 2024-04-12T05:37:02.825 INFO:tasks.workunit.client.0.smithi032.stdout:0/353: link d2/c12 d2/d8/d17/d1b/c74 0 2024-04-12T05:37:02.825 INFO:tasks.workunit.client.0.smithi032.stdout:2/208: sync 2024-04-12T05:37:02.825 INFO:tasks.workunit.client.0.smithi032.stdout:9/260: sync 2024-04-12T05:37:02.825 INFO:tasks.workunit.client.0.smithi032.stdout:7/258: link d0/d1c/d4c/l2d d0/l59 0 2024-04-12T05:37:02.827 INFO:tasks.workunit.client.0.smithi032.stdout:4/254: rename d4/f31 to d4/d13/d19/d28/f52 0 2024-04-12T05:37:02.828 INFO:tasks.workunit.client.0.smithi032.stdout:3/243: mknod d12/d2e/d3a/d2d/c49 0 2024-04-12T05:37:02.828 INFO:tasks.workunit.client.0.smithi032.stdout:2/209: write dd/f27 [899468,24825] 0 2024-04-12T05:37:02.829 INFO:tasks.workunit.client.0.smithi032.stdout:0/354: unlink d2/f68 0 2024-04-12T05:37:02.832 INFO:tasks.workunit.client.0.smithi032.stdout:4/255: chown d4/d13/c23 140 1 2024-04-12T05:37:02.833 INFO:tasks.workunit.client.0.smithi032.stdout:9/261: rename d9/d15/d1a/d1d/d55 to d9/d15/d1a/d57 0 2024-04-12T05:37:02.834 INFO:tasks.workunit.client.0.smithi032.stdout:6/257: dwrite d4/d21/d27/f4b [0,4194304] 0 2024-04-12T05:37:02.834 INFO:tasks.workunit.client.0.smithi032.stdout:8/215: truncate d0/db/dd/d11/f40 50378 0 2024-04-12T05:37:02.835 INFO:tasks.workunit.client.0.smithi032.stdout:5/315: dread d2/df/d25/d1e/f23 [0,4194304] 0 2024-04-12T05:37:02.835 INFO:tasks.workunit.client.0.smithi032.stdout:7/259: getdents d0/de/df/d20 0 2024-04-12T05:37:02.837 INFO:tasks.workunit.client.0.smithi032.stdout:2/210: mknod dd/d19/d20/c38 0 2024-04-12T05:37:02.837 INFO:tasks.workunit.client.0.smithi032.stdout:9/262: write d9/d15/f43 [1217828,43822] 0 2024-04-12T05:37:02.839 INFO:tasks.workunit.client.0.smithi032.stdout:0/355: mknod d2/dc/d3e/c75 0 2024-04-12T05:37:02.842 INFO:tasks.workunit.client.0.smithi032.stdout:6/258: rmdir d4/d14 39 2024-04-12T05:37:02.846 INFO:tasks.workunit.client.0.smithi032.stdout:2/211: symlink dd/d19/d20/l39 0 2024-04-12T05:37:02.847 INFO:tasks.workunit.client.0.smithi032.stdout:3/244: dwrite d12/d24/f38 [0,4194304] 0 2024-04-12T05:37:02.847 INFO:tasks.workunit.client.0.smithi032.stdout:0/356: mknod d2/c76 0 2024-04-12T05:37:02.849 INFO:tasks.workunit.client.0.smithi032.stdout:9/263: mkdir d9/d58 0 2024-04-12T05:37:02.850 INFO:tasks.workunit.client.0.smithi032.stdout:1/216: dread d5/de/f28 [0,4194304] 0 2024-04-12T05:37:02.857 INFO:tasks.workunit.client.0.smithi032.stdout:8/216: dwrite d0/fe [0,4194304] 0 2024-04-12T05:37:02.857 INFO:tasks.workunit.client.0.smithi032.stdout:0/357: write d2/d8/d17/f6b [941394,125915] 0 2024-04-12T05:37:02.860 INFO:tasks.workunit.client.0.smithi032.stdout:5/316: getdents d2/df/d31 0 2024-04-12T05:37:02.867 INFO:tasks.workunit.client.0.smithi032.stdout:4/256: dwrite d4/d13/f1c [0,4194304] 0 2024-04-12T05:37:02.868 INFO:tasks.workunit.client.0.smithi032.stdout:6/259: mknod d4/d21/d2e/d4d/c50 0 2024-04-12T05:37:02.868 INFO:tasks.workunit.client.0.smithi032.stdout:7/260: dwrite d0/de/f12 [0,4194304] 0 2024-04-12T05:37:02.869 INFO:tasks.workunit.client.0.smithi032.stdout:5/317: chown d2/df/f3f 4337 1 2024-04-12T05:37:02.874 INFO:tasks.workunit.client.0.smithi032.stdout:1/217: dwrite d5/ff [0,4194304] 0 2024-04-12T05:37:02.877 INFO:tasks.workunit.client.0.smithi032.stdout:2/212: dwrite dd/fe [0,4194304] 0 2024-04-12T05:37:02.877 INFO:tasks.workunit.client.0.smithi032.stdout:4/257: readlink d4/d13/d19/d28/d2f/d46/l4a 0 2024-04-12T05:37:02.878 INFO:tasks.workunit.client.0.smithi032.stdout:7/261: dread - d0/d13/d21/d30/d31/f58 zero size 2024-04-12T05:37:02.880 INFO:tasks.workunit.client.0.smithi032.stdout:5/318: mknod d2/df/d25/d1e/c63 0 2024-04-12T05:37:02.882 INFO:tasks.workunit.client.0.smithi032.stdout:3/245: link d12/d24/f38 d12/d2e/f4a 0 2024-04-12T05:37:02.884 INFO:tasks.workunit.client.0.smithi032.stdout:8/217: mknod d0/db/dd/d11/d1b/d22/d35/d44/c4b 0 2024-04-12T05:37:02.891 INFO:tasks.workunit.client.0.smithi032.stdout:0/358: getdents d2/d8/df/d24/d60/d63 0 2024-04-12T05:37:02.892 INFO:tasks.workunit.client.0.smithi032.stdout:7/262: mknod d0/d1c/d4c/d26/c5a 0 2024-04-12T05:37:02.894 INFO:tasks.workunit.client.0.smithi032.stdout:9/264: dwrite d9/d15/f28 [0,4194304] 0 2024-04-12T05:37:02.894 INFO:tasks.workunit.client.0.smithi032.stdout:3/246: truncate d12/d24/f47 863358 0 2024-04-12T05:37:02.895 INFO:tasks.workunit.client.0.smithi032.stdout:8/218: unlink d0/db/f18 0 2024-04-12T05:37:02.896 INFO:tasks.workunit.client.0.smithi032.stdout:6/260: dwrite d4/f2c [0,4194304] 0 2024-04-12T05:37:02.899 INFO:tasks.workunit.client.0.smithi032.stdout:1/218: dread d5/de/f10 [0,4194304] 0 2024-04-12T05:37:02.902 INFO:tasks.workunit.client.0.smithi032.stdout:6/261: rmdir d4/d21/d2e 39 2024-04-12T05:37:02.902 INFO:tasks.workunit.client.0.smithi032.stdout:0/359: write d2/dc/d3e/f46 [3724967,86779] 0 2024-04-12T05:37:02.903 INFO:tasks.workunit.client.0.smithi032.stdout:3/247: symlink d12/d2e/d3a/d2d/l4b 0 2024-04-12T05:37:02.904 INFO:tasks.workunit.client.0.smithi032.stdout:2/213: dwrite dd/d1b/d1c/f25 [0,4194304] 0 2024-04-12T05:37:02.904 INFO:tasks.workunit.client.0.smithi032.stdout:8/219: dread - d0/db/d38/f3f zero size 2024-04-12T05:37:02.911 INFO:tasks.workunit.client.0.smithi032.stdout:5/319: rmdir d2/df 39 2024-04-12T05:37:02.913 INFO:tasks.workunit.client.0.smithi032.stdout:9/265: truncate d9/fa 1880577 0 2024-04-12T05:37:02.913 INFO:tasks.workunit.client.0.smithi032.stdout:0/360: chown d2/d8/df/c34 1 1 2024-04-12T05:37:02.915 INFO:tasks.workunit.client.0.smithi032.stdout:6/262: creat d4/d14/f51 x:0 0 0 2024-04-12T05:37:02.916 INFO:tasks.workunit.client.0.smithi032.stdout:3/248: creat d12/d24/f4c x:0 0 0 2024-04-12T05:37:02.918 INFO:tasks.workunit.client.0.smithi032.stdout:7/263: dwrite d0/de/df/f24 [0,4194304] 0 2024-04-12T05:37:02.919 INFO:tasks.workunit.client.0.smithi032.stdout:4/258: dwrite d4/d13/d19/d28/d2f/f4e [0,4194304] 0 2024-04-12T05:37:02.919 INFO:tasks.workunit.client.0.smithi032.stdout:2/214: unlink dd/d1b/d1c/f25 0 2024-04-12T05:37:02.920 INFO:tasks.workunit.client.0.smithi032.stdout:1/219: rename d5/l7 to d5/de/d17/d35/d3e/l45 0 2024-04-12T05:37:02.921 INFO:tasks.workunit.client.0.smithi032.stdout:6/263: write d4/d21/d27/f38 [1191135,101233] 0 2024-04-12T05:37:02.931 INFO:tasks.workunit.client.0.smithi032.stdout:5/320: write d2/df/f54 [753489,74765] 0 2024-04-12T05:37:02.931 INFO:tasks.workunit.client.0.smithi032.stdout:3/249: creat d12/d24/f4d x:0 0 0 2024-04-12T05:37:02.931 INFO:tasks.workunit.client.0.smithi032.stdout:2/215: fsync dd/d1b/d1c/f29 0 2024-04-12T05:37:02.931 INFO:tasks.workunit.client.0.smithi032.stdout:8/220: rmdir d0/db/dd/d3c 0 2024-04-12T05:37:02.931 INFO:tasks.workunit.client.0.smithi032.stdout:4/259: symlink d4/d13/d19/d28/d2f/l53 0 2024-04-12T05:37:02.935 INFO:tasks.workunit.client.0.smithi032.stdout:7/264: dwrite d0/de/f44 [0,4194304] 0 2024-04-12T05:37:02.935 INFO:tasks.workunit.client.0.smithi032.stdout:5/321: chown d2/df/d25/d43/d49/c4e 410732 1 2024-04-12T05:37:02.936 INFO:tasks.workunit.client.0.smithi032.stdout:1/220: mknod d5/c46 0 2024-04-12T05:37:02.937 INFO:tasks.workunit.client.0.smithi032.stdout:2/216: mkdir dd/d1b/d1c/d2c/d32/d3a 0 2024-04-12T05:37:02.938 INFO:tasks.workunit.client.0.smithi032.stdout:7/265: fsync d0/d13/d21/f45 0 2024-04-12T05:37:02.940 INFO:tasks.workunit.client.0.smithi032.stdout:6/264: mknod d4/d21/c52 0 2024-04-12T05:37:02.942 INFO:tasks.workunit.client.0.smithi032.stdout:5/322: write d2/df/d31/f46 [1687120,77596] 0 2024-04-12T05:37:02.942 INFO:tasks.workunit.client.0.smithi032.stdout:3/250: getdents d12 0 2024-04-12T05:37:02.946 INFO:tasks.workunit.client.0.smithi032.stdout:3/251: dread - d12/d2e/f42 zero size 2024-04-12T05:37:02.947 INFO:tasks.workunit.client.0.smithi032.stdout:8/221: read d0/f5 [132218,101837] 0 2024-04-12T05:37:02.948 INFO:tasks.workunit.client.0.smithi032.stdout:5/323: read - d2/dd/f47 zero size 2024-04-12T05:37:02.948 INFO:tasks.workunit.client.0.smithi032.stdout:4/260: dwrite d4/d13/d19/d28/d32/f3e [0,4194304] 0 2024-04-12T05:37:02.950 INFO:tasks.workunit.client.0.smithi032.stdout:5/324: chown d2/d3c/d48/d4b/l4f 8 1 2024-04-12T05:37:02.950 INFO:tasks.workunit.client.0.smithi032.stdout:6/265: rename d4/f19 to d4/d14/d3c/f53 0 2024-04-12T05:37:02.952 INFO:tasks.workunit.client.0.smithi032.stdout:1/221: link d5/de/d2a/f3f d5/f47 0 2024-04-12T05:37:02.954 INFO:tasks.workunit.client.0.smithi032.stdout:6/266: dread - d4/d14/f46 zero size 2024-04-12T05:37:02.954 INFO:tasks.workunit.client.0.smithi032.stdout:5/325: chown d2/f4 0 1 2024-04-12T05:37:02.955 INFO:tasks.workunit.client.0.smithi032.stdout:3/252: creat d12/d2e/d3a/d2d/f4e x:0 0 0 2024-04-12T05:37:02.959 INFO:tasks.workunit.client.0.smithi032.stdout:4/261: creat d4/f54 x:0 0 0 2024-04-12T05:37:02.960 INFO:tasks.workunit.client.0.smithi032.stdout:5/326: chown d2/df/c62 113767694 1 2024-04-12T05:37:02.960 INFO:tasks.workunit.client.0.smithi032.stdout:7/266: dread d0/de/df/f24 [0,4194304] 0 2024-04-12T05:37:02.961 INFO:tasks.workunit.client.0.smithi032.stdout:3/253: fdatasync d12/d2e/d3a/f3b 0 2024-04-12T05:37:02.962 INFO:tasks.workunit.client.0.smithi032.stdout:8/222: unlink d0/db/dd/d11/f40 0 2024-04-12T05:37:02.963 INFO:tasks.workunit.client.0.smithi032.stdout:9/266: truncate d9/d15/d1a/d24/d27/f2d 1103542 0 2024-04-12T05:37:02.966 INFO:tasks.workunit.client.0.smithi032.stdout:1/222: dread d5/de/f1f [0,4194304] 0 2024-04-12T05:37:02.967 INFO:tasks.workunit.client.0.smithi032.stdout:2/217: truncate dd/f16 1513019 0 2024-04-12T05:37:02.970 INFO:tasks.workunit.client.0.smithi032.stdout:2/218: truncate dd/f27 1219347 0 2024-04-12T05:37:02.971 INFO:tasks.workunit.client.0.smithi032.stdout:7/267: mknod d0/d13/d21/d30/d31/c5b 0 2024-04-12T05:37:02.972 INFO:tasks.workunit.client.0.smithi032.stdout:3/254: dwrite d12/d2f/f3d [4194304,4194304] 0 2024-04-12T05:37:02.972 INFO:tasks.workunit.client.0.smithi032.stdout:8/223: dread d0/f3 [0,4194304] 0 2024-04-12T05:37:02.973 INFO:tasks.workunit.client.0.smithi032.stdout:6/267: mknod d4/d21/d2e/c54 0 2024-04-12T05:37:02.977 INFO:tasks.workunit.client.0.smithi032.stdout:9/267: link d9/d15/d2c/f52 d9/d15/d2c/d41/f59 0 2024-04-12T05:37:02.978 INFO:tasks.workunit.client.0.smithi032.stdout:5/327: creat d2/df/f64 x:0 0 0 2024-04-12T05:37:02.979 INFO:tasks.workunit.client.0.smithi032.stdout:1/223: mkdir d5/d48 0 2024-04-12T05:37:02.981 INFO:tasks.workunit.client.0.smithi032.stdout:8/224: symlink d0/db/d39/l4c 0 2024-04-12T05:37:02.983 INFO:tasks.workunit.client.0.smithi032.stdout:5/328: readlink d2/dd/l61 0 2024-04-12T05:37:02.984 INFO:tasks.workunit.client.0.smithi032.stdout:2/219: getdents dd/d1b/d1c 0 2024-04-12T05:37:02.987 INFO:tasks.workunit.client.0.smithi032.stdout:6/268: dwrite d4/d14/f46 [0,4194304] 0 2024-04-12T05:37:02.988 INFO:tasks.workunit.client.0.smithi032.stdout:3/255: dwrite d12/d2e/f36 [0,4194304] 0 2024-04-12T05:37:02.989 INFO:tasks.workunit.client.0.smithi032.stdout:8/225: symlink d0/db/dd/d11/d1b/d1e/l4d 0 2024-04-12T05:37:02.989 INFO:tasks.workunit.client.0.smithi032.stdout:1/224: write d5/de/d17/f25 [985265,52563] 0 2024-04-12T05:37:02.992 INFO:tasks.workunit.client.0.smithi032.stdout:5/329: chown d2/dd/c2a 1 1 2024-04-12T05:37:02.993 INFO:tasks.workunit.client.0.smithi032.stdout:5/330: chown d2 809520 1 2024-04-12T05:37:02.994 INFO:tasks.workunit.client.0.smithi032.stdout:4/262: truncate d4/d13/d19/f30 1172625 0 2024-04-12T05:37:02.994 INFO:tasks.workunit.client.0.smithi032.stdout:3/256: mknod d12/d2e/d3a/d2d/c4f 0 2024-04-12T05:37:02.995 INFO:tasks.workunit.client.0.smithi032.stdout:8/226: chown d0/db/dd/d11/d1b/d22/d29/f31 3832 1 2024-04-12T05:37:03.000 INFO:tasks.workunit.client.0.smithi032.stdout:3/257: rmdir d12/d24 39 2024-04-12T05:37:03.002 INFO:tasks.workunit.client.0.smithi032.stdout:2/220: dread fa [0,4194304] 0 2024-04-12T05:37:03.002 INFO:tasks.workunit.client.0.smithi032.stdout:8/227: creat d0/db/f4e x:0 0 0 2024-04-12T05:37:03.004 INFO:tasks.workunit.client.0.smithi032.stdout:4/263: symlink d4/d13/d19/d28/d2f/d3b/l55 0 2024-04-12T05:37:03.005 INFO:tasks.workunit.client.0.smithi032.stdout:6/269: dwrite d4/d14/d1a/f49 [0,4194304] 0 2024-04-12T05:37:03.006 INFO:tasks.workunit.client.0.smithi032.stdout:4/264: fsync d4/d13/f1f 0 2024-04-12T05:37:03.010 INFO:tasks.workunit.client.0.smithi032.stdout:2/221: symlink dd/d1b/d1c/l3b 0 2024-04-12T05:37:03.012 INFO:tasks.workunit.client.0.smithi032.stdout:3/258: write d12/d2e/f42 [123175,72907] 0 2024-04-12T05:37:03.012 INFO:tasks.workunit.client.0.smithi032.stdout:5/331: getdents d2/df/d25 0 2024-04-12T05:37:03.013 INFO:tasks.workunit.client.0.smithi032.stdout:6/270: mkdir d4/d21/d2e/d4d/d55 0 2024-04-12T05:37:03.013 INFO:tasks.workunit.client.0.smithi032.stdout:1/225: dread d5/f13 [0,4194304] 0 2024-04-12T05:37:03.014 INFO:tasks.workunit.client.0.smithi032.stdout:4/265: dread d4/d13/d19/d28/d2f/d3b/f3c [0,4194304] 0 2024-04-12T05:37:03.015 INFO:tasks.workunit.client.0.smithi032.stdout:1/226: dread - d5/d2c/f43 zero size 2024-04-12T05:37:03.015 INFO:tasks.workunit.client.0.smithi032.stdout:8/228: rename d0/db/dd/d11/f2a to d0/db/dd/d11/d1b/d22/f4f 0 2024-04-12T05:37:03.017 INFO:tasks.workunit.client.0.smithi032.stdout:4/266: chown d4/d13/d19/d28/d2f 13 1 2024-04-12T05:37:03.017 INFO:tasks.workunit.client.0.smithi032.stdout:8/229: readlink d0/db/dd/d11/d1b/d22/d35/l42 0 2024-04-12T05:37:03.017 INFO:tasks.workunit.client.0.smithi032.stdout:2/222: mknod dd/d1b/c3c 0 2024-04-12T05:37:03.018 INFO:tasks.workunit.client.0.smithi032.stdout:2/223: write dd/f14 [4539232,106262] 0 2024-04-12T05:37:03.022 INFO:tasks.workunit.client.0.smithi032.stdout:8/230: dread d0/fe [0,4194304] 0 2024-04-12T05:37:03.023 INFO:tasks.workunit.client.0.smithi032.stdout:8/231: write d0/db/dd/f17 [653216,123709] 0 2024-04-12T05:37:03.025 INFO:tasks.workunit.client.0.smithi032.stdout:5/332: mkdir d2/d65 0 2024-04-12T05:37:03.025 INFO:tasks.workunit.client.0.smithi032.stdout:5/333: dread - d2/df/f5c zero size 2024-04-12T05:37:03.028 INFO:tasks.workunit.client.0.smithi032.stdout:2/224: mknod dd/d1b/c3d 0 2024-04-12T05:37:03.029 INFO:tasks.workunit.client.0.smithi032.stdout:6/271: creat d4/d14/d1a/d37/d47/f56 x:0 0 0 2024-04-12T05:37:03.032 INFO:tasks.workunit.client.0.smithi032.stdout:2/225: rmdir dd/d1b 39 2024-04-12T05:37:03.033 INFO:tasks.workunit.client.0.smithi032.stdout:6/272: mkdir d4/d14/d1a/d37/d47/d57 0 2024-04-12T05:37:03.033 INFO:tasks.workunit.client.0.smithi032.stdout:5/334: symlink d2/dd/l66 0 2024-04-12T05:37:03.038 INFO:tasks.workunit.client.0.smithi032.stdout:5/335: fdatasync d2/df/d25/f2c 0 2024-04-12T05:37:03.038 INFO:tasks.workunit.client.0.smithi032.stdout:1/227: dwrite d5/f13 [0,4194304] 0 2024-04-12T05:37:03.038 INFO:tasks.workunit.client.0.smithi032.stdout:6/273: dread - d4/d14/d3c/f48 zero size 2024-04-12T05:37:03.040 INFO:tasks.workunit.client.0.smithi032.stdout:3/259: dread d12/d2e/d3a/f26 [0,4194304] 0 2024-04-12T05:37:03.042 INFO:tasks.workunit.client.0.smithi032.stdout:2/226: rename fc to dd/d1b/d1c/d2c/f3e 0 2024-04-12T05:37:03.043 INFO:tasks.workunit.client.0.smithi032.stdout:8/232: dwrite d0/db/dd/d11/d1b/d22/f4f [0,4194304] 0 2024-04-12T05:37:03.045 INFO:tasks.workunit.client.0.smithi032.stdout:2/227: symlink dd/d1b/l3f 0 2024-04-12T05:37:03.045 INFO:tasks.workunit.client.0.smithi032.stdout:5/336: dread d2/d3c/d5b/f39 [0,4194304] 0 2024-04-12T05:37:03.046 INFO:tasks.workunit.client.0.smithi032.stdout:1/228: mkdir d5/de/d17/d35/d3b/d49 0 2024-04-12T05:37:03.046 INFO:tasks.workunit.client.0.smithi032.stdout:0/361: sync 2024-04-12T05:37:03.050 INFO:tasks.workunit.client.0.smithi032.stdout:3/260: creat d12/d2e/f50 x:0 0 0 2024-04-12T05:37:03.052 INFO:tasks.workunit.client.0.smithi032.stdout:0/362: rmdir d2/d8/df/d24 39 2024-04-12T05:37:03.055 INFO:tasks.workunit.client.0.smithi032.stdout:0/363: fdatasync d2/d8/df/f52 0 2024-04-12T05:37:03.056 INFO:tasks.workunit.client.0.smithi032.stdout:2/228: dwrite dd/d19/d20/f28 [0,4194304] 0 2024-04-12T05:37:03.056 INFO:tasks.workunit.client.0.smithi032.stdout:8/233: link d0/db/fc d0/db/dd/d11/d1b/d22/d35/d44/f50 0 2024-04-12T05:37:03.059 INFO:tasks.workunit.client.0.smithi032.stdout:2/229: chown dd/d19 3153 1 2024-04-12T05:37:03.060 INFO:tasks.workunit.client.0.smithi032.stdout:2/230: write dd/d1b/d1c/f29 [695087,2955] 0 2024-04-12T05:37:03.060 INFO:tasks.workunit.client.0.smithi032.stdout:3/261: truncate d12/d24/f38 3124147 0 2024-04-12T05:37:03.064 INFO:tasks.workunit.client.0.smithi032.stdout:0/364: write d2/d8/df/d24/d60/d63/f43 [2861652,99181] 0 2024-04-12T05:37:03.066 INFO:tasks.workunit.client.0.smithi032.stdout:6/274: dread d4/d21/f3b [0,4194304] 0 2024-04-12T05:37:03.066 INFO:tasks.workunit.client.0.smithi032.stdout:2/231: dread - dd/d1b/d1c/d2c/d36/f37 zero size 2024-04-12T05:37:03.067 INFO:tasks.workunit.client.0.smithi032.stdout:3/262: symlink d12/d2f/l51 0 2024-04-12T05:37:03.068 INFO:tasks.workunit.client.0.smithi032.stdout:0/365: mkdir d2/d29/d77 0 2024-04-12T05:37:03.069 INFO:tasks.workunit.client.0.smithi032.stdout:6/275: write d4/f2c [1098386,100919] 0 2024-04-12T05:37:03.069 INFO:tasks.workunit.client.0.smithi032.stdout:2/232: rmdir dd 39 2024-04-12T05:37:03.070 INFO:tasks.workunit.client.0.smithi032.stdout:1/229: write d5/f47 [30861,12052] 0 2024-04-12T05:37:03.072 INFO:tasks.workunit.client.0.smithi032.stdout:8/234: dwrite d0/db/dd/d11/f28 [0,4194304] 0 2024-04-12T05:37:03.073 INFO:tasks.workunit.client.0.smithi032.stdout:5/337: write d2/d3c/f56 [624757,9335] 0 2024-04-12T05:37:03.073 INFO:tasks.workunit.client.0.smithi032.stdout:0/366: rmdir d2/d8/d17/d1b 39 2024-04-12T05:37:03.076 INFO:tasks.workunit.client.0.smithi032.stdout:2/233: mknod dd/d19/c40 0 2024-04-12T05:37:03.076 INFO:tasks.workunit.client.0.smithi032.stdout:6/276: write d4/d21/d27/f4b [1758059,78486] 0 2024-04-12T05:37:03.078 INFO:tasks.workunit.client.0.smithi032.stdout:3/263: symlink d12/l52 0 2024-04-12T05:37:03.078 INFO:tasks.workunit.client.0.smithi032.stdout:2/234: stat dd/d19/c40 0 2024-04-12T05:37:03.082 INFO:tasks.workunit.client.0.smithi032.stdout:5/338: creat d2/df/d25/f67 x:0 0 0 2024-04-12T05:37:03.084 INFO:tasks.workunit.client.0.smithi032.stdout:8/235: mkdir d0/db/dd/d11/d1b/d22/d51 0 2024-04-12T05:37:03.084 INFO:tasks.workunit.client.0.smithi032.stdout:0/367: write d2/d8/d17/d1b/d20/f6d [526765,83918] 0 2024-04-12T05:37:03.085 INFO:tasks.workunit.client.0.smithi032.stdout:6/277: mknod d4/c58 0 2024-04-12T05:37:03.086 INFO:tasks.workunit.client.0.smithi032.stdout:3/264: symlink d12/d2e/l53 0 2024-04-12T05:37:03.086 INFO:tasks.workunit.client.0.smithi032.stdout:2/235: dread dd/d1b/d1c/d2c/d32/f33 [0,4194304] 0 2024-04-12T05:37:03.088 INFO:tasks.workunit.client.0.smithi032.stdout:1/230: dwrite d5/de/d2a/f3f [0,4194304] 0 2024-04-12T05:37:03.089 INFO:tasks.workunit.client.0.smithi032.stdout:3/265: fdatasync d12/d2e/f42 0 2024-04-12T05:37:03.092 INFO:tasks.workunit.client.0.smithi032.stdout:8/236: write d0/db/dd/d11/f13 [60653,98716] 0 2024-04-12T05:37:03.093 INFO:tasks.workunit.client.0.smithi032.stdout:8/237: read d0/f3 [3492121,59023] 0 2024-04-12T05:37:03.094 INFO:tasks.workunit.client.0.smithi032.stdout:1/231: creat d5/de/f4a x:0 0 0 2024-04-12T05:37:03.095 INFO:tasks.workunit.client.0.smithi032.stdout:5/339: truncate d2/d3c/d5b/f21 690046 0 2024-04-12T05:37:03.097 INFO:tasks.workunit.client.0.smithi032.stdout:2/236: truncate dd/d1b/f34 640065 0 2024-04-12T05:37:03.097 INFO:tasks.workunit.client.0.smithi032.stdout:0/368: dwrite d2/d8/d17/f22 [4194304,4194304] 0 2024-04-12T05:37:03.099 INFO:tasks.workunit.client.0.smithi032.stdout:3/266: mknod d12/d24/c54 0 2024-04-12T05:37:03.100 INFO:tasks.workunit.client.0.smithi032.stdout:5/340: write d2/df/d25/d1e/f23 [141559,18489] 0 2024-04-12T05:37:03.101 INFO:tasks.workunit.client.0.smithi032.stdout:6/278: mkdir d4/d21/d2e/d4d/d55/d59 0 2024-04-12T05:37:03.104 INFO:tasks.workunit.client.0.smithi032.stdout:8/238: mknod d0/c52 0 2024-04-12T05:37:03.105 INFO:tasks.workunit.client.0.smithi032.stdout:5/341: chown d2/d3c/d5b/l27 1 1 2024-04-12T05:37:03.106 INFO:tasks.workunit.client.0.smithi032.stdout:3/267: mknod d12/c55 0 2024-04-12T05:37:03.108 INFO:tasks.workunit.client.0.smithi032.stdout:2/237: getdents dd/d1b 0 2024-04-12T05:37:03.109 INFO:tasks.workunit.client.0.smithi032.stdout:6/279: mknod d4/d21/d2e/d4d/d55/c5a 0 2024-04-12T05:37:03.112 INFO:tasks.workunit.client.0.smithi032.stdout:2/238: fdatasync dd/f2e 0 2024-04-12T05:37:03.112 INFO:tasks.workunit.client.0.smithi032.stdout:1/232: dwrite d5/de/d17/d1b/f33 [0,4194304] 0 2024-04-12T05:37:03.113 INFO:tasks.workunit.client.0.smithi032.stdout:3/268: write d12/d24/f2b [1090557,53874] 0 2024-04-12T05:37:03.115 INFO:tasks.workunit.client.0.smithi032.stdout:5/342: creat d2/f68 x:0 0 0 2024-04-12T05:37:03.117 INFO:tasks.workunit.client.0.smithi032.stdout:0/369: dwrite d2/d8/f37 [0,4194304] 0 2024-04-12T05:37:03.120 INFO:tasks.workunit.client.0.smithi032.stdout:2/239: rmdir dd 39 2024-04-12T05:37:03.120 INFO:tasks.workunit.client.0.smithi032.stdout:0/370: write d2/d28/d50/f35 [2020189,34630] 0 2024-04-12T05:37:03.124 INFO:tasks.workunit.client.0.smithi032.stdout:3/269: write d12/d24/f4c [269630,7859] 0 2024-04-12T05:37:03.125 INFO:tasks.workunit.client.0.smithi032.stdout:0/371: rmdir d2/d8 39 2024-04-12T05:37:03.127 INFO:tasks.workunit.client.0.smithi032.stdout:6/280: dread d4/f9 [0,4194304] 0 2024-04-12T05:37:03.127 INFO:tasks.workunit.client.0.smithi032.stdout:8/239: dread d0/db/dd/f14 [0,4194304] 0 2024-04-12T05:37:03.130 INFO:tasks.workunit.client.0.smithi032.stdout:2/240: chown dd/d1b/d1c/c31 70890173 1 2024-04-12T05:37:03.133 INFO:tasks.workunit.client.0.smithi032.stdout:0/372: symlink d2/d28/d50/d51/l78 0 2024-04-12T05:37:03.134 INFO:tasks.workunit.client.0.smithi032.stdout:5/343: dwrite d2/dd/f47 [0,4194304] 0 2024-04-12T05:37:03.138 INFO:tasks.workunit.client.0.smithi032.stdout:3/270: creat d12/f56 x:0 0 0 2024-04-12T05:37:03.140 INFO:tasks.workunit.client.0.smithi032.stdout:2/241: read dd/d1b/d1c/d2c/d32/f35 [948332,72194] 0 2024-04-12T05:37:03.141 INFO:tasks.workunit.client.0.smithi032.stdout:1/233: dwrite d5/de/d2a/f3f [4194304,4194304] 0 2024-04-12T05:37:03.142 INFO:tasks.workunit.client.0.smithi032.stdout:0/373: read - d2/d8/d17/d1b/f4d zero size 2024-04-12T05:37:03.143 INFO:tasks.workunit.client.0.smithi032.stdout:7/268: sync 2024-04-12T05:37:03.144 INFO:tasks.workunit.client.0.smithi032.stdout:1/234: fsync d5/f13 0 2024-04-12T05:37:03.147 INFO:tasks.workunit.client.0.smithi032.stdout:8/240: rename d0/db/dd/c15 to d0/db/dd/c53 0 2024-04-12T05:37:03.149 INFO:tasks.workunit.client.0.smithi032.stdout:6/281: dwrite d4/d14/f46 [0,4194304] 0 2024-04-12T05:37:03.151 INFO:tasks.workunit.client.0.smithi032.stdout:2/242: symlink dd/l41 0 2024-04-12T05:37:03.151 INFO:tasks.workunit.client.0.smithi032.stdout:6/282: dread - d4/d14/f51 zero size 2024-04-12T05:37:03.153 INFO:tasks.workunit.client.0.smithi032.stdout:7/269: creat d0/d1c/d4c/d26/f5c x:0 0 0 2024-04-12T05:37:03.155 INFO:tasks.workunit.client.0.smithi032.stdout:0/374: write d2/d8/df/f31 [200413,75626] 0 2024-04-12T05:37:03.155 INFO:tasks.workunit.client.0.smithi032.stdout:5/344: getdents d2 0 2024-04-12T05:37:03.157 INFO:tasks.workunit.client.0.smithi032.stdout:3/271: write d12/d24/f38 [462770,2016] 0 2024-04-12T05:37:03.158 INFO:tasks.workunit.client.0.smithi032.stdout:8/241: mkdir d0/db/dd/d11/d1b/d22/d54 0 2024-04-12T05:37:03.162 INFO:tasks.workunit.client.0.smithi032.stdout:1/235: dwrite d5/f23 [0,4194304] 0 2024-04-12T05:37:03.163 INFO:tasks.workunit.client.0.smithi032.stdout:5/345: stat d2/df/d25/d1e/c63 0 2024-04-12T05:37:03.164 INFO:tasks.workunit.client.0.smithi032.stdout:3/272: fsync d12/d2f/f48 0 2024-04-12T05:37:03.165 INFO:tasks.workunit.client.0.smithi032.stdout:6/283: dread d4/d21/d27/f38 [0,4194304] 0 2024-04-12T05:37:03.167 INFO:tasks.workunit.client.0.smithi032.stdout:8/242: fdatasync d0/db/dd/d11/f2c 0 2024-04-12T05:37:03.168 INFO:tasks.workunit.client.0.smithi032.stdout:0/375: mkdir d2/d8/df/d24/d60/d63/d79 0 2024-04-12T05:37:03.170 INFO:tasks.workunit.client.0.smithi032.stdout:3/273: creat d12/d2f/f57 x:0 0 0 2024-04-12T05:37:03.171 INFO:tasks.workunit.client.0.smithi032.stdout:1/236: write d5/de/d17/d1b/f33 [796630,29489] 0 2024-04-12T05:37:03.171 INFO:tasks.workunit.client.0.smithi032.stdout:5/346: rmdir d2/df/d25/d1e 39 2024-04-12T05:37:03.174 INFO:tasks.workunit.client.0.smithi032.stdout:2/243: dwrite f1 [0,4194304] 0 2024-04-12T05:37:03.175 INFO:tasks.workunit.client.0.smithi032.stdout:8/243: fsync d0/fe 0 2024-04-12T05:37:03.176 INFO:tasks.workunit.client.0.smithi032.stdout:1/237: dread - d5/d2c/f43 zero size 2024-04-12T05:37:03.176 INFO:tasks.workunit.client.0.smithi032.stdout:9/268: sync 2024-04-12T05:37:03.176 INFO:tasks.workunit.client.0.smithi032.stdout:4/267: sync 2024-04-12T05:37:03.186 INFO:tasks.workunit.client.0.smithi032.stdout:1/238: mknod d5/de/d2a/c4b 0 2024-04-12T05:37:03.186 INFO:tasks.workunit.client.0.smithi032.stdout:5/347: link d2/d3c/d5b/c3b d2/d3c/c69 0 2024-04-12T05:37:03.188 INFO:tasks.workunit.client.0.smithi032.stdout:0/376: creat d2/d8/d17/d1b/f7a x:0 0 0 2024-04-12T05:37:03.189 INFO:tasks.workunit.client.0.smithi032.stdout:8/244: write d0/db/dd/d11/f2c [1590412,89906] 0 2024-04-12T05:37:03.191 INFO:tasks.workunit.client.0.smithi032.stdout:7/270: dwrite d0/f8 [0,4194304] 0 2024-04-12T05:37:03.196 INFO:tasks.workunit.client.0.smithi032.stdout:5/348: readlink d2/df/d25/d43/d49/l4d 0 2024-04-12T05:37:03.199 INFO:tasks.workunit.client.0.smithi032.stdout:4/268: dwrite d4/d13/f1c [0,4194304] 0 2024-04-12T05:37:03.199 INFO:tasks.workunit.client.0.smithi032.stdout:2/244: truncate dd/f27 516575 0 2024-04-12T05:37:03.200 INFO:tasks.workunit.client.0.smithi032.stdout:9/269: rmdir d9/d15/d1a/d57 0 2024-04-12T05:37:03.201 INFO:tasks.workunit.client.0.smithi032.stdout:6/284: dread d4/f1b [0,4194304] 0 2024-04-12T05:37:03.201 INFO:tasks.workunit.client.0.smithi032.stdout:3/274: dwrite d12/d24/f4c [0,4194304] 0 2024-04-12T05:37:03.201 INFO:tasks.workunit.client.0.smithi032.stdout:2/245: stat f8 0 2024-04-12T05:37:03.203 INFO:tasks.workunit.client.0.smithi032.stdout:4/269: fdatasync d4/d13/d19/d28/f42 0 2024-04-12T05:37:03.204 INFO:tasks.workunit.client.0.smithi032.stdout:3/275: stat d12/d2e/f50 0 2024-04-12T05:37:03.204 INFO:tasks.workunit.client.0.smithi032.stdout:1/239: creat d5/de/d17/d35/d3e/f4c x:0 0 0 2024-04-12T05:37:03.206 INFO:tasks.workunit.client.0.smithi032.stdout:6/285: write d4/d21/d2e/f39 [5956,96056] 0 2024-04-12T05:37:03.208 INFO:tasks.workunit.client.0.smithi032.stdout:3/276: write d12/d2e/f36 [4202009,124765] 0 2024-04-12T05:37:03.209 INFO:tasks.workunit.client.0.smithi032.stdout:6/286: readlink d4/l18 0 2024-04-12T05:37:03.210 INFO:tasks.workunit.client.0.smithi032.stdout:1/240: chown d5/de/l32 190454 1 2024-04-12T05:37:03.210 INFO:tasks.workunit.client.0.smithi032.stdout:9/270: chown d9/d1f/f39 2 1 2024-04-12T05:37:03.213 INFO:tasks.workunit.client.0.smithi032.stdout:9/271: chown d9/d15/d2c/f52 7209220 1 2024-04-12T05:37:03.214 INFO:tasks.workunit.client.0.smithi032.stdout:5/349: mknod d2/df/d25/d1e/c6a 0 2024-04-12T05:37:03.215 INFO:tasks.workunit.client.0.smithi032.stdout:4/270: mknod d4/d13/d19/d28/d32/c56 0 2024-04-12T05:37:03.216 INFO:tasks.workunit.client.0.smithi032.stdout:7/271: dwrite d0/de/df/f24 [0,4194304] 0 2024-04-12T05:37:03.217 INFO:tasks.workunit.client.0.smithi032.stdout:8/245: fdatasync d0/db/dd/d11/d1b/d22/f2b 0 2024-04-12T05:37:03.217 INFO:tasks.workunit.client.0.smithi032.stdout:3/277: rmdir d12/d2e/d3a 39 2024-04-12T05:37:03.218 INFO:tasks.workunit.client.0.smithi032.stdout:6/287: mknod d4/d14/d1a/d37/d47/c5b 0 2024-04-12T05:37:03.219 INFO:tasks.workunit.client.0.smithi032.stdout:1/241: dread d5/de/d17/f25 [0,4194304] 0 2024-04-12T05:37:03.219 INFO:tasks.workunit.client.0.smithi032.stdout:4/271: chown d4/d13/d19/d28/f47 3701976 1 2024-04-12T05:37:03.221 INFO:tasks.workunit.client.0.smithi032.stdout:5/350: chown d2/c11 276 1 2024-04-12T05:37:03.223 INFO:tasks.workunit.client.0.smithi032.stdout:8/246: write d0/db/d38/f3f [626877,53278] 0 2024-04-12T05:37:03.224 INFO:tasks.workunit.client.0.smithi032.stdout:6/288: creat d4/d14/d3c/f5c x:0 0 0 2024-04-12T05:37:03.225 INFO:tasks.workunit.client.0.smithi032.stdout:4/272: rename d4/d13/d19/d28/d4d to d4/d13/d19/d28/d4d/d57 22 2024-04-12T05:37:03.228 INFO:tasks.workunit.client.0.smithi032.stdout:5/351: creat d2/d3c/d48/d4b/f6b x:0 0 0 2024-04-12T05:37:03.229 INFO:tasks.workunit.client.0.smithi032.stdout:6/289: write d4/d14/f46 [1588960,124174] 0 2024-04-12T05:37:03.230 INFO:tasks.workunit.client.0.smithi032.stdout:3/278: symlink d12/d2e/l58 0 2024-04-12T05:37:03.231 INFO:tasks.workunit.client.0.smithi032.stdout:8/247: write d0/d36/f3a [482546,36818] 0 2024-04-12T05:37:03.233 INFO:tasks.workunit.client.0.smithi032.stdout:2/246: write fa [3900886,28182] 0 2024-04-12T05:37:03.237 INFO:tasks.workunit.client.0.smithi032.stdout:6/290: chown d4/d14/c4a 469 1 2024-04-12T05:37:03.238 INFO:tasks.workunit.client.0.smithi032.stdout:8/248: creat d0/db/f55 x:0 0 0 2024-04-12T05:37:03.239 INFO:tasks.workunit.client.0.smithi032.stdout:2/247: mkdir dd/d1b/d42 0 2024-04-12T05:37:03.240 INFO:tasks.workunit.client.0.smithi032.stdout:7/272: dread d0/fc [0,4194304] 0 2024-04-12T05:37:03.240 INFO:tasks.workunit.client.0.smithi032.stdout:1/242: dwrite d5/d2c/f2d [0,4194304] 0 2024-04-12T05:37:03.241 INFO:tasks.workunit.client.0.smithi032.stdout:5/352: getdents d2/df/d25/d50 0 2024-04-12T05:37:03.242 INFO:tasks.workunit.client.0.smithi032.stdout:2/248: write f1 [56001,90258] 0 2024-04-12T05:37:03.244 INFO:tasks.workunit.client.0.smithi032.stdout:0/377: dread d2/dc/d3e/f53 [0,4194304] 0 2024-04-12T05:37:03.246 INFO:tasks.workunit.client.0.smithi032.stdout:5/353: readlink d2/df/d25/d50/d55/l5f 0 2024-04-12T05:37:03.246 INFO:tasks.workunit.client.0.smithi032.stdout:8/249: mknod d0/db/dd/d11/d1b/d22/d35/c56 0 2024-04-12T05:37:03.248 INFO:tasks.workunit.client.0.smithi032.stdout:2/249: creat dd/d1b/d1c/f43 x:0 0 0 2024-04-12T05:37:03.249 INFO:tasks.workunit.client.0.smithi032.stdout:7/273: dread d0/d13/f4b [0,4194304] 0 2024-04-12T05:37:03.249 INFO:tasks.workunit.client.0.smithi032.stdout:8/250: chown d0/db/dd/d11/f13 711 1 2024-04-12T05:37:03.251 INFO:tasks.workunit.client.0.smithi032.stdout:6/291: link d4/d21/d27/f4b d4/d14/d1a/d37/f5d 0 2024-04-12T05:37:03.254 INFO:tasks.workunit.client.0.smithi032.stdout:5/354: mkdir d2/d65/d6c 0 2024-04-12T05:37:03.255 INFO:tasks.workunit.client.0.smithi032.stdout:9/272: dread d9/d15/d1a/d1d/f4d [0,4194304] 0 2024-04-12T05:37:03.256 INFO:tasks.workunit.client.0.smithi032.stdout:2/250: mknod dd/c44 0 2024-04-12T05:37:03.259 INFO:tasks.workunit.client.0.smithi032.stdout:6/292: truncate d4/ff 4351795 0 2024-04-12T05:37:03.261 INFO:tasks.workunit.client.0.smithi032.stdout:0/378: getdents d2/d8/d17/d1b 0 2024-04-12T05:37:03.263 INFO:tasks.workunit.client.0.smithi032.stdout:9/273: mkdir d9/d15/d1a/d1d/d5a 0 2024-04-12T05:37:03.266 INFO:tasks.workunit.client.0.smithi032.stdout:8/251: getdents d0 0 2024-04-12T05:37:03.267 INFO:tasks.workunit.client.0.smithi032.stdout:7/274: dread d0/d13/f27 [0,4194304] 0 2024-04-12T05:37:03.267 INFO:tasks.workunit.client.0.smithi032.stdout:6/293: symlink d4/d14/d1a/d37/d47/l5e 0 2024-04-12T05:37:03.268 INFO:tasks.workunit.client.0.smithi032.stdout:1/243: truncate d5/de/d17/d1b/f33 1829508 0 2024-04-12T05:37:03.269 INFO:tasks.workunit.client.0.smithi032.stdout:9/274: chown d9/d15/d1a/d24/f47 1 1 2024-04-12T05:37:03.271 INFO:tasks.workunit.client.0.smithi032.stdout:0/379: mknod d2/d29/d77/c7b 0 2024-04-12T05:37:03.271 INFO:tasks.workunit.client.0.smithi032.stdout:5/355: dwrite d2/df/d25/d50/d55/f5a [0,4194304] 0 2024-04-12T05:37:03.272 INFO:tasks.workunit.client.0.smithi032.stdout:9/275: chown d9/d15/d1a 277785 1 2024-04-12T05:37:03.273 INFO:tasks.workunit.client.0.smithi032.stdout:7/275: write d0/de/df/f24 [358224,29478] 0 2024-04-12T05:37:03.278 INFO:tasks.workunit.client.0.smithi032.stdout:5/356: chown d2/d3c/d48/d4b/f6b 5 1 2024-04-12T05:37:03.278 INFO:tasks.workunit.client.0.smithi032.stdout:6/294: creat d4/d14/d1a/d37/f5f x:0 0 0 2024-04-12T05:37:03.281 INFO:tasks.workunit.client.0.smithi032.stdout:8/252: rename d0/db/dd/d11/f26 to d0/db/dd/d11/d1b/d1e/f57 0 2024-04-12T05:37:03.281 INFO:tasks.workunit.client.0.smithi032.stdout:2/251: dread dd/f14 [0,4194304] 0 2024-04-12T05:37:03.282 INFO:tasks.workunit.client.0.smithi032.stdout:5/357: rmdir d2/d65 39 2024-04-12T05:37:03.284 INFO:tasks.workunit.client.0.smithi032.stdout:1/244: dwrite d5/de/f10 [0,4194304] 0 2024-04-12T05:37:03.284 INFO:tasks.workunit.client.0.smithi032.stdout:7/276: rename d0/de/f43 to d0/d34/f5d 0 2024-04-12T05:37:03.286 INFO:tasks.workunit.client.0.smithi032.stdout:8/253: mkdir d0/d58 0 2024-04-12T05:37:03.287 INFO:tasks.workunit.client.0.smithi032.stdout:6/295: mknod d4/d21/d2e/d4d/c60 0 2024-04-12T05:37:03.288 INFO:tasks.workunit.client.0.smithi032.stdout:0/380: link d2/dc/f61 d2/d8/df/d24/d60/d63/d79/f7c 0 2024-04-12T05:37:03.290 INFO:tasks.workunit.client.0.smithi032.stdout:6/296: fsync d4/d14/f51 0 2024-04-12T05:37:03.291 INFO:tasks.workunit.client.0.smithi032.stdout:9/276: dread d9/d15/d1a/d24/d27/f2d [0,4194304] 0 2024-04-12T05:37:03.294 INFO:tasks.workunit.client.0.smithi032.stdout:7/277: link d0/de/f12 d0/d1c/d4c/f5e 0 2024-04-12T05:37:03.295 INFO:tasks.workunit.client.0.smithi032.stdout:1/245: dwrite d5/f23 [0,4194304] 0 2024-04-12T05:37:03.295 INFO:tasks.workunit.client.0.smithi032.stdout:0/381: symlink d2/d28/d50/d51/l7d 0 2024-04-12T05:37:03.297 INFO:tasks.workunit.client.0.smithi032.stdout:6/297: write d4/d21/d27/f28 [609721,117570] 0 2024-04-12T05:37:03.300 INFO:tasks.workunit.client.0.smithi032.stdout:6/298: dread d4/f9 [0,4194304] 0 2024-04-12T05:37:03.304 INFO:tasks.workunit.client.0.smithi032.stdout:9/277: symlink d9/d58/l5b 0 2024-04-12T05:37:03.306 INFO:tasks.workunit.client.0.smithi032.stdout:6/299: unlink d4/d14/d1a/l23 0 2024-04-12T05:37:03.307 INFO:tasks.workunit.client.0.smithi032.stdout:6/300: stat d4/d14/d1a/d37/d47/f56 0 2024-04-12T05:37:03.309 INFO:tasks.workunit.client.0.smithi032.stdout:0/382: dread d2/f6 [0,4194304] 0 2024-04-12T05:37:03.310 INFO:tasks.workunit.client.0.smithi032.stdout:1/246: truncate d5/de/d17/d1b/f33 2218853 0 2024-04-12T05:37:03.310 INFO:tasks.workunit.client.0.smithi032.stdout:0/383: readlink d2/dc/d41/l67 0 2024-04-12T05:37:03.311 INFO:tasks.workunit.client.0.smithi032.stdout:2/252: truncate dd/d1b/d1c/d2c/f3e 2046585 0 2024-04-12T05:37:03.312 INFO:tasks.workunit.client.0.smithi032.stdout:8/254: dread d0/d36/f3a [0,4194304] 0 2024-04-12T05:37:03.313 INFO:tasks.workunit.client.0.smithi032.stdout:6/301: creat d4/d21/d2e/d32/f61 x:0 0 0 2024-04-12T05:37:03.313 INFO:tasks.workunit.client.0.smithi032.stdout:2/253: write dd/d1b/d1c/f43 [680063,22924] 0 2024-04-12T05:37:03.314 INFO:tasks.workunit.client.0.smithi032.stdout:2/254: truncate f9 1197575 0 2024-04-12T05:37:03.315 INFO:tasks.workunit.client.0.smithi032.stdout:0/384: write d2/fe [5250746,13952] 0 2024-04-12T05:37:03.317 INFO:tasks.workunit.client.0.smithi032.stdout:1/247: rename d5/de/d2a/d39 to d5/d2c/d4d 0 2024-04-12T05:37:03.317 INFO:tasks.workunit.client.0.smithi032.stdout:5/358: write d2/d3c/d5b/f21 [1386457,27007] 0 2024-04-12T05:37:03.318 INFO:tasks.workunit.client.0.smithi032.stdout:9/278: truncate d9/f34 123237 0 2024-04-12T05:37:03.319 INFO:tasks.workunit.client.0.smithi032.stdout:1/248: truncate d5/de/f1f 569554 0 2024-04-12T05:37:03.320 INFO:tasks.workunit.client.0.smithi032.stdout:5/359: chown d2/d3c/d48/d4b/l4f 40079 1 2024-04-12T05:37:03.321 INFO:tasks.workunit.client.0.smithi032.stdout:6/302: mknod d4/d21/c62 0 2024-04-12T05:37:03.323 INFO:tasks.workunit.client.0.smithi032.stdout:9/279: symlink d9/d15/d1a/d31/l5c 0 2024-04-12T05:37:03.324 INFO:tasks.workunit.client.0.smithi032.stdout:1/249: dread d5/f13 [0,4194304] 0 2024-04-12T05:37:03.325 INFO:tasks.workunit.client.0.smithi032.stdout:1/250: chown d5/de/d17/d1b/f21 1118 1 2024-04-12T05:37:03.326 INFO:tasks.workunit.client.0.smithi032.stdout:8/255: rename d0/db/dd/d11/d1b/d1e/f57 to d0/db/dd/d11/d1b/d22/d51/f59 0 2024-04-12T05:37:03.327 INFO:tasks.workunit.client.0.smithi032.stdout:0/385: link d2/d8/c11 d2/d28/d50/d51/c7e 0 2024-04-12T05:37:03.328 INFO:tasks.workunit.client.0.smithi032.stdout:0/386: chown d2/c76 332275014 1 2024-04-12T05:37:03.330 INFO:tasks.workunit.client.0.smithi032.stdout:9/280: creat d9/d1f/f5d x:0 0 0 2024-04-12T05:37:03.330 INFO:tasks.workunit.client.0.smithi032.stdout:6/303: symlink d4/l63 0 2024-04-12T05:37:03.333 INFO:tasks.workunit.client.0.smithi032.stdout:8/256: mknod d0/db/dd/d11/d1b/d22/d35/c5a 0 2024-04-12T05:37:03.338 INFO:tasks.workunit.client.0.smithi032.stdout:5/360: symlink d2/df/l6d 0 2024-04-12T05:37:03.339 INFO:tasks.workunit.client.0.smithi032.stdout:5/361: write d2/df/d25/d50/d55/f5a [548861,86558] 0 2024-04-12T05:37:03.339 INFO:tasks.workunit.client.0.smithi032.stdout:9/281: creat d9/d15/d2c/f5e x:0 0 0 2024-04-12T05:37:03.339 INFO:tasks.workunit.client.0.smithi032.stdout:6/304: mknod d4/d14/c64 0 2024-04-12T05:37:03.339 INFO:tasks.workunit.client.0.smithi032.stdout:9/282: chown d9/d15/d2c/d41/f3e 0 1 2024-04-12T05:37:03.339 INFO:tasks.workunit.client.0.smithi032.stdout:2/255: write dd/d1b/d1c/d2c/f3e [305820,124645] 0 2024-04-12T05:37:03.340 INFO:tasks.workunit.client.0.smithi032.stdout:0/387: mknod d2/d8/d17/d64/c7f 0 2024-04-12T05:37:03.342 INFO:tasks.workunit.client.0.smithi032.stdout:0/388: read d2/f7 [301512,97323] 0 2024-04-12T05:37:03.344 INFO:tasks.workunit.client.0.smithi032.stdout:6/305: dread d4/d21/f2f [0,4194304] 0 2024-04-12T05:37:03.345 INFO:tasks.workunit.client.0.smithi032.stdout:6/306: chown d4/d14 46586273 1 2024-04-12T05:37:03.347 INFO:tasks.workunit.client.0.smithi032.stdout:2/256: symlink dd/d1b/d1c/d2c/d32/l45 0 2024-04-12T05:37:03.348 INFO:tasks.workunit.client.0.smithi032.stdout:0/389: mknod d2/d8/d48/c80 0 2024-04-12T05:37:03.349 INFO:tasks.workunit.client.0.smithi032.stdout:8/257: truncate d0/db/dd/f14 2723205 0 2024-04-12T05:37:03.349 INFO:tasks.workunit.client.0.smithi032.stdout:9/283: symlink d9/d32/l5f 0 2024-04-12T05:37:03.350 INFO:tasks.workunit.client.0.smithi032.stdout:6/307: creat d4/d21/d2e/d4d/d55/f65 x:0 0 0 2024-04-12T05:37:03.352 INFO:tasks.workunit.client.0.smithi032.stdout:9/284: dread d9/d15/d1a/d24/d27/f2d [0,4194304] 0 2024-04-12T05:37:03.352 INFO:tasks.workunit.client.0.smithi032.stdout:0/390: symlink d2/d28/d50/l81 0 2024-04-12T05:37:03.353 INFO:tasks.workunit.client.0.smithi032.stdout:9/285: readlink d9/d58/l5b 0 2024-04-12T05:37:03.353 INFO:tasks.workunit.client.0.smithi032.stdout:5/362: rename d2/df/d25/ca to d2/dd/c6e 0 2024-04-12T05:37:03.355 INFO:tasks.workunit.client.0.smithi032.stdout:8/258: symlink d0/l5b 0 2024-04-12T05:37:03.355 INFO:tasks.workunit.client.0.smithi032.stdout:9/286: write d9/d15/d2c/d41/f45 [4457792,90488] 0 2024-04-12T05:37:03.355 INFO:tasks.workunit.client.0.smithi032.stdout:5/363: chown d2/df/f64 11469 1 2024-04-12T05:37:03.355 INFO:tasks.workunit.client.0.smithi032.stdout:6/308: symlink d4/d21/d2e/d4d/d55/d59/l66 0 2024-04-12T05:37:03.356 INFO:tasks.workunit.client.0.smithi032.stdout:5/364: dread - d2/df/f5c zero size 2024-04-12T05:37:03.358 INFO:tasks.workunit.client.0.smithi032.stdout:2/257: getdents dd 0 2024-04-12T05:37:03.359 INFO:tasks.workunit.client.0.smithi032.stdout:5/365: fdatasync d2/d3c/d5b/f21 0 2024-04-12T05:37:03.360 INFO:tasks.workunit.client.0.smithi032.stdout:2/258: chown dd/d1b/d1c/c30 27901146 1 2024-04-12T05:37:03.361 INFO:tasks.workunit.client.0.smithi032.stdout:2/259: write dd/d1b/d1c/f29 [868411,67541] 0 2024-04-12T05:37:03.362 INFO:tasks.workunit.client.0.smithi032.stdout:9/287: dread d9/d15/d1a/d24/d27/f50 [0,4194304] 0 2024-04-12T05:37:03.363 INFO:tasks.workunit.client.0.smithi032.stdout:6/309: symlink d4/d14/d1a/d37/d47/d57/l67 0 2024-04-12T05:37:03.364 INFO:tasks.workunit.client.0.smithi032.stdout:8/259: truncate d0/d36/f3a 975888 0 2024-04-12T05:37:03.366 INFO:tasks.workunit.client.0.smithi032.stdout:2/260: unlink dd/d19/f2d 0 2024-04-12T05:37:03.367 INFO:tasks.workunit.client.0.smithi032.stdout:2/261: chown dd/d1b/d42 98738 1 2024-04-12T05:37:03.367 INFO:tasks.workunit.client.0.smithi032.stdout:6/310: chown d4/cc 1519464 1 2024-04-12T05:37:03.368 INFO:tasks.workunit.client.0.smithi032.stdout:2/262: rename dd/d19 to dd/d19/d20/d46 22 2024-04-12T05:37:03.368 INFO:tasks.workunit.client.0.smithi032.stdout:5/366: mkdir d2/dd/d6f 0 2024-04-12T05:37:03.368 INFO:tasks.workunit.client.0.smithi032.stdout:2/263: readlink dd/l41 0 2024-04-12T05:37:03.369 INFO:tasks.workunit.client.0.smithi032.stdout:0/391: write d2/d8/df/d24/d60/d63/d79/f7c [282599,115104] 0 2024-04-12T05:37:03.371 INFO:tasks.workunit.client.0.smithi032.stdout:0/392: fsync d2/d8/d17/d1b/f7a 0 2024-04-12T05:37:03.372 INFO:tasks.workunit.client.0.smithi032.stdout:2/264: creat dd/d1b/d1c/d2c/d32/f47 x:0 0 0 2024-04-12T05:37:03.374 INFO:tasks.workunit.client.0.smithi032.stdout:0/393: write d2/d8/d17/f22 [7629041,11215] 0 2024-04-12T05:37:03.378 INFO:tasks.workunit.client.0.smithi032.stdout:6/311: dwrite d4/d21/d27/f38 [0,4194304] 0 2024-04-12T05:37:03.379 INFO:tasks.workunit.client.0.smithi032.stdout:2/265: creat dd/d19/d20/f48 x:0 0 0 2024-04-12T05:37:03.380 INFO:tasks.workunit.client.0.smithi032.stdout:8/260: creat d0/db/dd/f5c x:0 0 0 2024-04-12T05:37:03.381 INFO:tasks.workunit.client.0.smithi032.stdout:6/312: fsync d4/d21/f3b 0 2024-04-12T05:37:03.383 INFO:tasks.workunit.client.0.smithi032.stdout:5/367: truncate d2/dd/f47 2502204 0 2024-04-12T05:37:03.386 INFO:tasks.workunit.client.0.smithi032.stdout:9/288: dwrite d9/d15/d2c/f20 [0,4194304] 0 2024-04-12T05:37:03.387 INFO:tasks.workunit.client.0.smithi032.stdout:9/289: chown f5 7 1 2024-04-12T05:37:03.388 INFO:tasks.workunit.client.0.smithi032.stdout:9/290: chown d9/d15/d1a/d24/f4b 11965011 1 2024-04-12T05:37:03.388 INFO:tasks.workunit.client.0.smithi032.stdout:2/266: rename dd/d19/d20/c38 to dd/d1b/c49 0 2024-04-12T05:37:03.392 INFO:tasks.workunit.client.0.smithi032.stdout:8/261: mknod d0/db/dd/c5d 0 2024-04-12T05:37:03.393 INFO:tasks.workunit.client.0.smithi032.stdout:0/394: link d2/d29/f4a d2/d8/d66/f82 0 2024-04-12T05:37:03.394 INFO:tasks.workunit.client.0.smithi032.stdout:5/368: creat d2/df/d25/d43/d49/f70 x:0 0 0 2024-04-12T05:37:03.397 INFO:tasks.workunit.client.0.smithi032.stdout:0/395: mkdir d2/d28/d50/d51/d83 0 2024-04-12T05:37:03.398 INFO:tasks.workunit.client.0.smithi032.stdout:8/262: mknod d0/db/dd/d11/d1b/d1e/c5e 0 2024-04-12T05:37:03.400 INFO:tasks.workunit.client.0.smithi032.stdout:2/267: dread dd/d1b/d1c/d2c/d32/f35 [0,4194304] 0 2024-04-12T05:37:03.403 INFO:tasks.workunit.client.0.smithi032.stdout:0/396: creat d2/d29/f84 x:0 0 0 2024-04-12T05:37:03.404 INFO:tasks.workunit.client.0.smithi032.stdout:9/291: dwrite d9/d15/d2c/f5e [0,4194304] 0 2024-04-12T05:37:03.404 INFO:tasks.workunit.client.0.smithi032.stdout:4/273: sync 2024-04-12T05:37:03.404 INFO:tasks.workunit.client.0.smithi032.stdout:8/263: mkdir d0/db/dd/d11/d1b/d5f 0 2024-04-12T05:37:03.406 INFO:tasks.workunit.client.0.smithi032.stdout:8/264: fsync d0/d36/f4a 0 2024-04-12T05:37:03.407 INFO:tasks.workunit.client.0.smithi032.stdout:0/397: chown d2/d8/d17/d1b/l27 0 1 2024-04-12T05:37:03.407 INFO:tasks.workunit.client.0.smithi032.stdout:5/369: truncate d2/df/d25/d1e/f2f 3583305 0 2024-04-12T05:37:03.409 INFO:tasks.workunit.client.0.smithi032.stdout:2/268: mkdir dd/d1b/d42/d4a 0 2024-04-12T05:37:03.409 INFO:tasks.workunit.client.0.smithi032.stdout:0/398: dread - d2/d29/f65 zero size 2024-04-12T05:37:03.411 INFO:tasks.workunit.client.0.smithi032.stdout:6/313: dwrite d4/d21/d27/f4b [0,4194304] 0 2024-04-12T05:37:03.411 INFO:tasks.workunit.client.0.smithi032.stdout:9/292: mknod d9/d15/d1a/c60 0 2024-04-12T05:37:03.412 INFO:tasks.workunit.client.0.smithi032.stdout:0/399: dread - d2/d8/d17/d1b/d20/f69 zero size 2024-04-12T05:37:03.413 INFO:tasks.workunit.client.0.smithi032.stdout:8/265: creat d0/db/f60 x:0 0 0 2024-04-12T05:37:03.416 INFO:tasks.workunit.client.0.smithi032.stdout:2/269: creat dd/d19/d20/f4b x:0 0 0 2024-04-12T05:37:03.416 INFO:tasks.workunit.client.0.smithi032.stdout:8/266: chown d0/db/dd/d11/d1b/d1e 49428277 1 2024-04-12T05:37:03.418 INFO:tasks.workunit.client.0.smithi032.stdout:0/400: write d2/d8/df/f5f [138278,75291] 0 2024-04-12T05:37:03.421 INFO:tasks.workunit.client.0.smithi032.stdout:9/293: rename d9/d15/d2c/d41/f3e to d9/d15/d1a/d1d/d5a/f61 0 2024-04-12T05:37:03.421 INFO:tasks.workunit.client.0.smithi032.stdout:0/401: readlink d2/dc/d41/l49 0 2024-04-12T05:37:03.422 INFO:tasks.workunit.client.0.smithi032.stdout:5/370: dread d2/df/d25/d50/d55/f5a [0,4194304] 0 2024-04-12T05:37:03.423 INFO:tasks.workunit.client.0.smithi032.stdout:9/294: stat d9/d15/f28 0 2024-04-12T05:37:03.423 INFO:tasks.workunit.client.0.smithi032.stdout:2/270: dread dd/f27 [0,4194304] 0 2024-04-12T05:37:03.426 INFO:tasks.workunit.client.0.smithi032.stdout:8/267: rename d0/db/dd/f14 to d0/db/d38/f61 0 2024-04-12T05:37:03.426 INFO:tasks.workunit.client.0.smithi032.stdout:6/314: dwrite d4/d21/d27/f28 [0,4194304] 0 2024-04-12T05:37:03.428 INFO:tasks.workunit.client.0.smithi032.stdout:0/402: creat d2/d8/df/d24/d60/d63/f85 x:0 0 0 2024-04-12T05:37:03.428 INFO:tasks.workunit.client.0.smithi032.stdout:2/271: mknod dd/d1b/d42/d4a/c4c 0 2024-04-12T05:37:03.429 INFO:tasks.workunit.client.0.smithi032.stdout:5/371: creat d2/d65/d6c/f71 x:0 0 0 2024-04-12T05:37:03.430 INFO:tasks.workunit.client.0.smithi032.stdout:6/315: mknod d4/d14/d1a/d37/c68 0 2024-04-12T05:37:03.431 INFO:tasks.workunit.client.0.smithi032.stdout:8/268: mkdir d0/db/dd/d11/d1b/d1e/d62 0 2024-04-12T05:37:03.431 INFO:tasks.workunit.client.0.smithi032.stdout:2/272: rmdir dd/d1b/d42/d4a 39 2024-04-12T05:37:03.432 INFO:tasks.workunit.client.0.smithi032.stdout:6/316: write d4/d21/d27/f28 [2108072,77586] 0 2024-04-12T05:37:03.434 INFO:tasks.workunit.client.0.smithi032.stdout:8/269: mkdir d0/db/d38/d63 0 2024-04-12T05:37:03.434 INFO:tasks.workunit.client.0.smithi032.stdout:6/317: chown d4/d14/c4a 357 1 2024-04-12T05:37:03.435 INFO:tasks.workunit.client.0.smithi032.stdout:5/372: rename d2/d3c/c4c to d2/d3c/d48/c72 0 2024-04-12T05:37:03.435 INFO:tasks.workunit.client.0.smithi032.stdout:6/318: fsync d4/d14/d1a/d37/f5f 0 2024-04-12T05:37:03.436 INFO:tasks.workunit.client.0.smithi032.stdout:0/403: getdents d2/d8 0 2024-04-12T05:37:03.440 INFO:tasks.workunit.client.0.smithi032.stdout:5/373: unlink d2/df/d25/d43/d49/f70 0 2024-04-12T05:37:03.443 INFO:tasks.workunit.client.0.smithi032.stdout:6/319: dread d4/f6 [0,4194304] 0 2024-04-12T05:37:03.446 INFO:tasks.workunit.client.0.smithi032.stdout:2/273: dwrite dd/d1b/d1c/d2c/d32/f35 [0,4194304] 0 2024-04-12T05:37:03.447 INFO:tasks.workunit.client.0.smithi032.stdout:5/374: symlink d2/df/d25/l73 0 2024-04-12T05:37:03.448 INFO:tasks.workunit.client.0.smithi032.stdout:8/270: dwrite d0/db/dd/d11/f13 [0,4194304] 0 2024-04-12T05:37:03.449 INFO:tasks.workunit.client.0.smithi032.stdout:6/320: mkdir d4/d14/d1a/d37/d69 0 2024-04-12T05:37:03.452 INFO:tasks.workunit.client.0.smithi032.stdout:6/321: chown d4/d14/d1a/c42 12 1 2024-04-12T05:37:03.453 INFO:tasks.workunit.client.0.smithi032.stdout:5/375: write d2/df/d25/d1e/f1f [35730,34795] 0 2024-04-12T05:37:03.454 INFO:tasks.workunit.client.0.smithi032.stdout:0/404: dread d2/d8/df/d24/d60/d63/f43 [0,4194304] 0 2024-04-12T05:37:03.457 INFO:tasks.workunit.client.0.smithi032.stdout:5/376: write d2/f60 [1423825,25679] 0 2024-04-12T05:37:03.459 INFO:tasks.workunit.client.0.smithi032.stdout:5/377: stat d2/df/d25/d1e/c3d 0 2024-04-12T05:37:03.460 INFO:tasks.workunit.client.0.smithi032.stdout:8/271: dwrite d0/d36/f4a [0,4194304] 0 2024-04-12T05:37:03.461 INFO:tasks.workunit.client.0.smithi032.stdout:5/378: dread - d2/d65/d6c/f71 zero size 2024-04-12T05:37:03.463 INFO:tasks.workunit.client.0.smithi032.stdout:6/322: dread d4/d14/f46 [0,4194304] 0 2024-04-12T05:37:03.466 INFO:tasks.workunit.client.0.smithi032.stdout:0/405: mknod d2/d29/d77/c86 0 2024-04-12T05:37:03.468 INFO:tasks.workunit.client.0.smithi032.stdout:5/379: rename d2/df/d25/l73 to d2/d3c/d48/l74 0 2024-04-12T05:37:03.468 INFO:tasks.workunit.client.0.smithi032.stdout:6/323: mknod d4/d21/d2e/d4d/d55/d59/c6a 0 2024-04-12T05:37:03.472 INFO:tasks.workunit.client.0.smithi032.stdout:8/272: rename d0/f5 to d0/db/dd/d11/d1b/d22/d54/f64 0 2024-04-12T05:37:03.472 INFO:tasks.workunit.client.0.smithi032.stdout:5/380: mkdir d2/d3c/d5b/d75 0 2024-04-12T05:37:03.474 INFO:tasks.workunit.client.0.smithi032.stdout:2/274: dwrite dd/d1b/d1c/d2c/d32/f35 [4194304,4194304] 0 2024-04-12T05:37:03.476 INFO:tasks.workunit.client.0.smithi032.stdout:0/406: rename d2/d8/df/d24/d60/d63 to d2/d28/d87 0 2024-04-12T05:37:03.477 INFO:tasks.workunit.client.0.smithi032.stdout:5/381: creat d2/df/d25/d50/f76 x:0 0 0 2024-04-12T05:37:03.478 INFO:tasks.workunit.client.0.smithi032.stdout:8/273: mknod d0/db/dd/d11/d1b/d1e/c65 0 2024-04-12T05:37:03.479 INFO:tasks.workunit.client.0.smithi032.stdout:6/324: rename d4/d21/d2e/d4d/d55/d59/l66 to d4/d21/d2e/d4d/d55/l6b 0 2024-04-12T05:37:03.482 INFO:tasks.workunit.client.0.smithi032.stdout:8/274: symlink d0/db/dd/d11/d1b/d22/d35/l66 0 2024-04-12T05:37:03.484 INFO:tasks.workunit.client.0.smithi032.stdout:2/275: dwrite dd/d19/d20/f48 [0,4194304] 0 2024-04-12T05:37:03.484 INFO:tasks.workunit.client.0.smithi032.stdout:5/382: rename d2/f34 to d2/d65/d6c/f77 0 2024-04-12T05:37:03.487 INFO:tasks.workunit.client.0.smithi032.stdout:2/276: write dd/d1b/d1c/f43 [1276057,20357] 0 2024-04-12T05:37:03.487 INFO:tasks.workunit.client.0.smithi032.stdout:5/383: dread d2/d3c/d5b/f39 [0,4194304] 0 2024-04-12T05:37:03.492 INFO:tasks.workunit.client.0.smithi032.stdout:6/325: rename d4/d14/d25/l43 to d4/d14/d25/l6c 0 2024-04-12T05:37:03.493 INFO:tasks.workunit.client.0.smithi032.stdout:5/384: stat d2/c37 0 2024-04-12T05:37:03.494 INFO:tasks.workunit.client.0.smithi032.stdout:2/277: dwrite f8 [0,4194304] 0 2024-04-12T05:37:03.495 INFO:tasks.workunit.client.0.smithi032.stdout:6/326: dread d4/f9 [0,4194304] 0 2024-04-12T05:37:03.496 INFO:tasks.workunit.client.0.smithi032.stdout:6/327: write d4/f2a [542752,118481] 0 2024-04-12T05:37:03.498 INFO:tasks.workunit.client.0.smithi032.stdout:6/328: write d4/d14/d3c/f5c [639564,23308] 0 2024-04-12T05:37:03.498 INFO:tasks.workunit.client.0.smithi032.stdout:2/278: dread f9 [0,4194304] 0 2024-04-12T05:37:03.499 INFO:tasks.workunit.client.0.smithi032.stdout:8/275: rename d0/fe to d0/db/dd/d11/d1b/d5f/f67 0 2024-04-12T05:37:03.501 INFO:tasks.workunit.client.0.smithi032.stdout:2/279: dread dd/f27 [0,4194304] 0 2024-04-12T05:37:03.503 INFO:tasks.workunit.client.0.smithi032.stdout:5/385: read d2/df/d25/f8 [1866821,82791] 0 2024-04-12T05:37:03.552 INFO:tasks.workunit.client.0.smithi032.stdout:3/279: sync 2024-04-12T05:37:03.552 INFO:tasks.workunit.client.0.smithi032.stdout:7/278: sync 2024-04-12T05:37:03.573 INFO:tasks.workunit.client.0.smithi032.stdout:6/329: unlink d4/d14/d3c/l3f 0 2024-04-12T05:37:03.573 INFO:tasks.workunit.client.0.smithi032.stdout:3/280: fdatasync f10 0 2024-04-12T05:37:03.573 INFO:tasks.workunit.client.0.smithi032.stdout:2/280: creat dd/d19/d20/f4d x:0 0 0 2024-04-12T05:37:03.575 INFO:tasks.workunit.client.0.smithi032.stdout:5/386: mkdir d2/d65/d78 0 2024-04-12T05:37:03.575 INFO:tasks.workunit.client.0.smithi032.stdout:7/279: mknod d0/de/df/d20/c5f 0 2024-04-12T05:37:03.577 INFO:tasks.workunit.client.0.smithi032.stdout:7/280: readlink d0/d13/d21/d30/d31/l49 0 2024-04-12T05:37:03.577 INFO:tasks.workunit.client.0.smithi032.stdout:2/281: rmdir dd/d1b/d1c/d2c/d32 39 2024-04-12T05:37:03.578 INFO:tasks.workunit.client.0.smithi032.stdout:5/387: truncate d2/df/d25/d1e/f1f 1128264 0 2024-04-12T05:37:03.579 INFO:tasks.workunit.client.0.smithi032.stdout:7/281: write d0/de/df/f3f [610333,123874] 0 2024-04-12T05:37:03.580 INFO:tasks.workunit.client.0.smithi032.stdout:6/330: rename d4/d14/f26 to d4/d14/f6d 0 2024-04-12T05:37:03.582 INFO:tasks.workunit.client.0.smithi032.stdout:3/281: truncate d12/f1b 4521599 0 2024-04-12T05:37:03.583 INFO:tasks.workunit.client.0.smithi032.stdout:5/388: mknod d2/df/d25/d1e/c79 0 2024-04-12T05:37:03.585 INFO:tasks.workunit.client.0.smithi032.stdout:2/282: creat dd/d1b/d1c/d2c/d36/f4e x:0 0 0 2024-04-12T05:37:03.585 INFO:tasks.workunit.client.0.smithi032.stdout:8/276: getdents d0/db/dd/d11/d1b/d22/d54 0 2024-04-12T05:37:03.586 INFO:tasks.workunit.client.0.smithi032.stdout:2/283: write dd/d1b/d1c/f29 [1610613,120930] 0 2024-04-12T05:37:03.587 INFO:tasks.workunit.client.0.smithi032.stdout:7/282: creat d0/d10/f60 x:0 0 0 2024-04-12T05:37:03.587 INFO:tasks.workunit.client.0.smithi032.stdout:8/277: write d0/db/dd/d11/f2c [2498072,1222] 0 2024-04-12T05:37:03.589 INFO:tasks.workunit.client.0.smithi032.stdout:6/331: creat d4/d21/d2e/d4d/d55/f6e x:0 0 0 2024-04-12T05:37:03.590 INFO:tasks.workunit.client.0.smithi032.stdout:7/283: chown d0/d1c/d4c/f5e 6388 1 2024-04-12T05:37:03.590 INFO:tasks.workunit.client.0.smithi032.stdout:3/282: chown d12/d2e/d3a/d2d 1 1 2024-04-12T05:37:03.592 INFO:tasks.workunit.client.0.smithi032.stdout:5/389: rmdir d2/df/d25 39 2024-04-12T05:37:03.594 INFO:tasks.workunit.client.0.smithi032.stdout:2/284: mknod dd/d1b/d1c/d2c/d36/c4f 0 2024-04-12T05:37:03.597 INFO:tasks.workunit.client.0.smithi032.stdout:6/332: unlink d4/d14/d1a/d37/l45 0 2024-04-12T05:37:03.601 INFO:tasks.workunit.client.0.smithi032.stdout:2/285: dwrite dd/d1b/d1c/d2c/f3e [0,4194304] 0 2024-04-12T05:37:03.606 INFO:tasks.workunit.client.0.smithi032.stdout:3/283: creat d12/f59 x:0 0 0 2024-04-12T05:37:03.606 INFO:tasks.workunit.client.0.smithi032.stdout:8/278: write d0/db/f19 [2706423,106168] 0 2024-04-12T05:37:03.607 INFO:tasks.workunit.client.0.smithi032.stdout:7/284: truncate d0/de/f15 2814965 0 2024-04-12T05:37:03.609 INFO:tasks.workunit.client.0.smithi032.stdout:5/390: write d2/df/f1c [7514527,21277] 0 2024-04-12T05:37:03.610 INFO:tasks.workunit.client.0.smithi032.stdout:6/333: getdents d4/d21/d2e/d4d/d55 0 2024-04-12T05:37:03.611 INFO:tasks.workunit.client.0.smithi032.stdout:3/284: creat d12/d2e/f5a x:0 0 0 2024-04-12T05:37:03.613 INFO:tasks.workunit.client.0.smithi032.stdout:7/285: read - d0/d1c/d4c/d26/f5c zero size 2024-04-12T05:37:03.614 INFO:tasks.workunit.client.0.smithi032.stdout:8/279: chown d0/db/dd/d11/d1b/d22/d29/f31 584289820 1 2024-04-12T05:37:03.615 INFO:tasks.workunit.client.0.smithi032.stdout:3/285: mkdir d12/d2f/d5b 0 2024-04-12T05:37:03.616 INFO:tasks.workunit.client.0.smithi032.stdout:2/286: dwrite dd/d1b/d1c/d2c/d32/f35 [0,4194304] 0 2024-04-12T05:37:03.616 INFO:tasks.workunit.client.0.smithi032.stdout:3/286: stat d12/d24/f47 0 2024-04-12T05:37:03.617 INFO:tasks.workunit.client.0.smithi032.stdout:6/334: rename d4/d14/d3c/f5c to d4/d14/d1a/f6f 0 2024-04-12T05:37:03.619 INFO:tasks.workunit.client.0.smithi032.stdout:6/335: write d4/d21/d27/f4b [3282584,111439] 0 2024-04-12T05:37:03.620 INFO:tasks.workunit.client.0.smithi032.stdout:3/287: chown d12/d2e/c46 394473 1 2024-04-12T05:37:03.621 INFO:tasks.workunit.client.0.smithi032.stdout:6/336: stat d4/d14/d3c 0 2024-04-12T05:37:03.622 INFO:tasks.workunit.client.0.smithi032.stdout:8/280: unlink d0/db/dd/d11/d1b/d22/d35/l42 0 2024-04-12T05:37:03.624 INFO:tasks.workunit.client.0.smithi032.stdout:6/337: write d4/d21/d2e/d32/f36 [1400687,87442] 0 2024-04-12T05:37:03.625 INFO:tasks.workunit.client.0.smithi032.stdout:2/287: dread dd/d19/d20/f48 [0,4194304] 0 2024-04-12T05:37:03.630 INFO:tasks.workunit.client.0.smithi032.stdout:8/281: mknod d0/db/c68 0 2024-04-12T05:37:03.631 INFO:tasks.workunit.client.0.smithi032.stdout:5/391: dwrite d2/d3c/d5b/f39 [0,4194304] 0 2024-04-12T05:37:03.633 INFO:tasks.workunit.client.0.smithi032.stdout:6/338: write d4/f1b [2131060,68382] 0 2024-04-12T05:37:03.634 INFO:tasks.workunit.client.0.smithi032.stdout:2/288: mknod dd/d19/c50 0 2024-04-12T05:37:03.635 INFO:tasks.workunit.client.0.smithi032.stdout:3/288: write d12/d2e/d3a/f3b [876217,42516] 0 2024-04-12T05:37:03.636 INFO:tasks.workunit.client.0.smithi032.stdout:8/282: chown d0/db/dd/d11/l12 959645 1 2024-04-12T05:37:03.636 INFO:tasks.workunit.client.0.smithi032.stdout:2/289: stat dd/d1b/l3f 0 2024-04-12T05:37:03.641 INFO:tasks.workunit.client.0.smithi032.stdout:2/290: creat dd/d1b/d1c/f51 x:0 0 0 2024-04-12T05:37:03.642 INFO:tasks.workunit.client.0.smithi032.stdout:6/339: creat d4/d21/d2e/d4d/d55/f70 x:0 0 0 2024-04-12T05:37:03.645 INFO:tasks.workunit.client.0.smithi032.stdout:7/286: dread d0/d10/f22 [0,4194304] 0 2024-04-12T05:37:03.645 INFO:tasks.workunit.client.0.smithi032.stdout:8/283: rmdir d0/db/dd/d11/d1b/d1e/d62 0 2024-04-12T05:37:03.646 INFO:tasks.workunit.client.0.smithi032.stdout:5/392: dwrite d2/df/d31/f46 [0,4194304] 0 2024-04-12T05:37:03.647 INFO:tasks.workunit.client.0.smithi032.stdout:6/340: mknod d4/d14/d1a/d37/d69/c71 0 2024-04-12T05:37:03.649 INFO:tasks.workunit.client.0.smithi032.stdout:7/287: truncate d0/dd/f2a 924372 0 2024-04-12T05:37:03.652 INFO:tasks.workunit.client.0.smithi032.stdout:2/291: dread dd/d1b/d1c/f43 [0,4194304] 0 2024-04-12T05:37:03.653 INFO:tasks.workunit.client.0.smithi032.stdout:6/341: rename d4/d21/d27/f28 to d4/d21/d2e/d4d/d55/f72 0 2024-04-12T05:37:03.653 INFO:tasks.workunit.client.0.smithi032.stdout:7/288: dread d0/d13/f4b [0,4194304] 0 2024-04-12T05:37:03.654 INFO:tasks.workunit.client.0.smithi032.stdout:5/393: creat d2/df/d25/d43/f7a x:0 0 0 2024-04-12T05:37:03.656 INFO:tasks.workunit.client.0.smithi032.stdout:7/289: write d0/d10/f22 [1111650,37345] 0 2024-04-12T05:37:03.657 INFO:tasks.workunit.client.0.smithi032.stdout:5/394: read d2/df/f1c [5111299,31145] 0 2024-04-12T05:37:03.657 INFO:tasks.workunit.client.0.smithi032.stdout:6/342: mknod d4/d21/c73 0 2024-04-12T05:37:03.658 INFO:tasks.workunit.client.0.smithi032.stdout:2/292: symlink dd/d1b/d42/d4a/l52 0 2024-04-12T05:37:03.659 INFO:tasks.workunit.client.0.smithi032.stdout:2/293: chown dd/f16 468342682 1 2024-04-12T05:37:03.662 INFO:tasks.workunit.client.0.smithi032.stdout:6/343: mkdir d4/d74 0 2024-04-12T05:37:03.663 INFO:tasks.workunit.client.0.smithi032.stdout:7/290: creat d0/de/f61 x:0 0 0 2024-04-12T05:37:03.664 INFO:tasks.workunit.client.0.smithi032.stdout:2/294: creat dd/d19/d20/f53 x:0 0 0 2024-04-12T05:37:03.665 INFO:tasks.workunit.client.0.smithi032.stdout:2/295: chown dd/d19/l26 166051 1 2024-04-12T05:37:03.666 INFO:tasks.workunit.client.0.smithi032.stdout:8/284: dread d0/db/dd/d11/f1f [0,4194304] 0 2024-04-12T05:37:03.666 INFO:tasks.workunit.client.0.smithi032.stdout:5/395: creat d2/f7b x:0 0 0 2024-04-12T05:37:03.668 INFO:tasks.workunit.client.0.smithi032.stdout:7/291: creat d0/d10/f62 x:0 0 0 2024-04-12T05:37:03.671 INFO:tasks.workunit.client.0.smithi032.stdout:5/396: getdents d2/df/d31 0 2024-04-12T05:37:03.672 INFO:tasks.workunit.client.0.smithi032.stdout:3/289: dread d12/f1b [0,4194304] 0 2024-04-12T05:37:03.674 INFO:tasks.workunit.client.0.smithi032.stdout:5/397: chown d2/df/d25/d50 1056 1 2024-04-12T05:37:03.674 INFO:tasks.workunit.client.0.smithi032.stdout:3/290: fdatasync d12/d24/f38 0 2024-04-12T05:37:03.675 INFO:tasks.workunit.client.0.smithi032.stdout:7/292: dread d0/dd/f2a [0,4194304] 0 2024-04-12T05:37:03.676 INFO:tasks.workunit.client.0.smithi032.stdout:6/344: rename d4/d21/d2e/d4d to d4/d14/d3c/d75 0 2024-04-12T05:37:03.676 INFO:tasks.workunit.client.0.smithi032.stdout:8/285: dwrite d0/db/f2e [0,4194304] 0 2024-04-12T05:37:03.676 INFO:tasks.workunit.client.0.smithi032.stdout:5/398: readlink d2/d3c/d5b/l3e 0 2024-04-12T05:37:03.677 INFO:tasks.workunit.client.0.smithi032.stdout:2/296: getdents dd/d19 0 2024-04-12T05:37:03.678 INFO:tasks.workunit.client.0.smithi032.stdout:3/291: symlink d12/d24/l5c 0 2024-04-12T05:37:03.683 INFO:tasks.workunit.client.0.smithi032.stdout:5/399: fsync d2/df/f64 0 2024-04-12T05:37:03.683 INFO:tasks.workunit.client.0.smithi032.stdout:7/293: write d0/d1c/d4c/f4d [307437,28907] 0 2024-04-12T05:37:03.683 INFO:tasks.workunit.client.0.smithi032.stdout:8/286: truncate d0/db/f60 72123 0 2024-04-12T05:37:03.685 INFO:tasks.workunit.client.0.smithi032.stdout:5/400: creat d2/d3c/d48/d4b/f7c x:0 0 0 2024-04-12T05:37:03.687 INFO:tasks.workunit.client.0.smithi032.stdout:1/251: sync 2024-04-12T05:37:03.687 INFO:tasks.workunit.client.0.smithi032.stdout:4/274: sync 2024-04-12T05:37:03.691 INFO:tasks.workunit.client.0.smithi032.stdout:7/294: dread d0/de/df/f3f [0,4194304] 0 2024-04-12T05:37:03.691 INFO:tasks.workunit.client.0.smithi032.stdout:5/401: creat d2/df/d25/d43/d49/f7d x:0 0 0 2024-04-12T05:37:03.692 INFO:tasks.workunit.client.0.smithi032.stdout:2/297: dwrite dd/d19/d20/f53 [0,4194304] 0 2024-04-12T05:37:03.692 INFO:tasks.workunit.client.0.smithi032.stdout:6/345: dwrite d4/d14/f46 [0,4194304] 0 2024-04-12T05:37:03.693 INFO:tasks.workunit.client.0.smithi032.stdout:6/346: chown d4/d21/d2e 773303 1 2024-04-12T05:37:03.694 INFO:tasks.workunit.client.0.smithi032.stdout:8/287: rename d0/db/dd/d11/f13 to d0/db/dd/d11/d1b/d22/d51/f69 0 2024-04-12T05:37:03.698 INFO:tasks.workunit.client.0.smithi032.stdout:2/298: rename dd/c12 to dd/c54 0 2024-04-12T05:37:03.698 INFO:tasks.workunit.client.0.smithi032.stdout:6/347: fsync d4/f9 0 2024-04-12T05:37:03.700 INFO:tasks.workunit.client.0.smithi032.stdout:2/299: chown dd/d19/c23 239335 1 2024-04-12T05:37:03.701 INFO:tasks.workunit.client.0.smithi032.stdout:6/348: truncate d4/d14/f51 725730 0 2024-04-12T05:37:03.701 INFO:tasks.workunit.client.0.smithi032.stdout:4/275: read d4/d13/d19/f34 [16837,43025] 0 2024-04-12T05:37:03.706 INFO:tasks.workunit.client.0.smithi032.stdout:5/402: link d2/df/d25/d1e/f23 d2/df/d25/d50/f7e 0 2024-04-12T05:37:03.708 INFO:tasks.workunit.client.0.smithi032.stdout:8/288: symlink d0/d58/l6a 0 2024-04-12T05:37:03.710 INFO:tasks.workunit.client.0.smithi032.stdout:6/349: dwrite d4/f2c [4194304,4194304] 0 2024-04-12T05:37:03.711 INFO:tasks.workunit.client.0.smithi032.stdout:4/276: dwrite d4/d13/f1c [0,4194304] 0 2024-04-12T05:37:03.714 INFO:tasks.workunit.client.0.smithi032.stdout:1/252: rename d5/de/d17/d35/d3e/c41 to d5/c4e 0 2024-04-12T05:37:03.716 INFO:tasks.workunit.client.0.smithi032.stdout:1/253: write d5/de/f4a [42873,95787] 0 2024-04-12T05:37:03.716 INFO:tasks.workunit.client.0.smithi032.stdout:8/289: creat d0/db/f6b x:0 0 0 2024-04-12T05:37:03.716 INFO:tasks.workunit.client.0.smithi032.stdout:1/254: read - d5/d2c/f3c zero size 2024-04-12T05:37:03.718 INFO:tasks.workunit.client.0.smithi032.stdout:7/295: getdents d0/dd/d51 0 2024-04-12T05:37:03.718 INFO:tasks.workunit.client.0.smithi032.stdout:1/255: write d5/de/d2a/f2b [252057,125603] 0 2024-04-12T05:37:03.719 INFO:tasks.workunit.client.0.smithi032.stdout:7/296: fsync d0/dd/f55 0 2024-04-12T05:37:03.720 INFO:tasks.workunit.client.0.smithi032.stdout:6/350: mkdir d4/d14/d1a/d37/d69/d76 0 2024-04-12T05:37:03.721 INFO:tasks.workunit.client.0.smithi032.stdout:4/277: symlink d4/d13/d19/d28/d2f/d3b/l58 0 2024-04-12T05:37:03.721 INFO:tasks.workunit.client.0.smithi032.stdout:5/403: mknod d2/d3c/d5b/d75/c7f 0 2024-04-12T05:37:03.722 INFO:tasks.workunit.client.0.smithi032.stdout:8/290: creat d0/db/dd/d11/d1b/f6c x:0 0 0 2024-04-12T05:37:03.726 INFO:tasks.workunit.client.0.smithi032.stdout:1/256: dwrite d5/f23 [0,4194304] 0 2024-04-12T05:37:03.726 INFO:tasks.workunit.client.0.smithi032.stdout:7/297: rmdir d0/d10 39 2024-04-12T05:37:03.727 INFO:tasks.workunit.client.0.smithi032.stdout:1/257: dread - d5/de/d17/d35/d3e/f4c zero size 2024-04-12T05:37:03.729 INFO:tasks.workunit.client.0.smithi032.stdout:6/351: rename d4/c1d to d4/d14/d25/c77 0 2024-04-12T05:37:03.732 INFO:tasks.workunit.client.0.smithi032.stdout:8/291: mknod d0/d36/c6d 0 2024-04-12T05:37:03.734 INFO:tasks.workunit.client.0.smithi032.stdout:2/300: truncate dd/d1b/d1c/f29 1723592 0 2024-04-12T05:37:03.734 INFO:tasks.workunit.client.0.smithi032.stdout:8/292: write d0/db/f6b [62846,79279] 0 2024-04-12T05:37:03.736 INFO:tasks.workunit.client.0.smithi032.stdout:2/301: write dd/d1b/d1c/d2c/d32/f35 [5053474,10184] 0 2024-04-12T05:37:03.739 INFO:tasks.workunit.client.0.smithi032.stdout:2/302: write dd/d1b/d1c/d2c/d36/f37 [961805,72947] 0 2024-04-12T05:37:03.740 INFO:tasks.workunit.client.0.smithi032.stdout:4/278: creat d4/d13/d19/d28/f59 x:0 0 0 2024-04-12T05:37:03.742 INFO:tasks.workunit.client.0.smithi032.stdout:8/293: dwrite d0/db/f55 [0,4194304] 0 2024-04-12T05:37:03.746 INFO:tasks.workunit.client.0.smithi032.stdout:4/279: link d4/d13/fd d4/d13/d44/f5a 0 2024-04-12T05:37:03.748 INFO:tasks.workunit.client.0.smithi032.stdout:4/280: chown d4/d13/d19/d28/f42 268414926 1 2024-04-12T05:37:03.748 INFO:tasks.workunit.client.0.smithi032.stdout:2/303: dwrite dd/d19/d20/f48 [0,4194304] 0 2024-04-12T05:37:03.749 INFO:tasks.workunit.client.0.smithi032.stdout:8/294: link d0/db/dd/d11/d1b/d22/d29/f31 d0/db/f6e 0 2024-04-12T05:37:03.750 INFO:tasks.workunit.client.0.smithi032.stdout:2/304: read - dd/d19/f24 zero size 2024-04-12T05:37:03.755 INFO:tasks.workunit.client.0.smithi032.stdout:2/305: mknod dd/d19/c55 0 2024-04-12T05:37:03.759 INFO:tasks.workunit.client.0.smithi032.stdout:7/298: truncate d0/f8 3509514 0 2024-04-12T05:37:03.761 INFO:tasks.workunit.client.0.smithi032.stdout:1/258: truncate d5/de/d2a/f3f 7244553 0 2024-04-12T05:37:03.761 INFO:tasks.workunit.client.0.smithi032.stdout:8/295: dwrite d0/db/dd/d11/d1b/d22/d51/f59 [0,4194304] 0 2024-04-12T05:37:03.762 INFO:tasks.workunit.client.0.smithi032.stdout:1/259: chown d5/d48 478100 1 2024-04-12T05:37:03.763 INFO:tasks.workunit.client.0.smithi032.stdout:6/352: rename d4/d14/d25/c77 to d4/c78 0 2024-04-12T05:37:03.765 INFO:tasks.workunit.client.0.smithi032.stdout:8/296: fsync d0/db/dd/d11/d1b/d22/f45 0 2024-04-12T05:37:03.765 INFO:tasks.workunit.client.0.smithi032.stdout:1/260: mkdir d5/d2c/d4f 0 2024-04-12T05:37:03.765 INFO:tasks.workunit.client.0.smithi032.stdout:6/353: chown d4/d74 934451 1 2024-04-12T05:37:03.765 INFO:tasks.workunit.client.0.smithi032.stdout:7/299: creat d0/d10/f63 x:0 0 0 2024-04-12T05:37:03.767 INFO:tasks.workunit.client.0.smithi032.stdout:4/281: rename d4/c5 to d4/d13/d19/d28/d4d/c5b 0 2024-04-12T05:37:03.772 INFO:tasks.workunit.client.0.smithi032.stdout:1/261: creat d5/d2c/d4d/f50 x:0 0 0 2024-04-12T05:37:03.772 INFO:tasks.workunit.client.0.smithi032.stdout:5/404: dread d2/dd/f44 [0,4194304] 0 2024-04-12T05:37:03.773 INFO:tasks.workunit.client.0.smithi032.stdout:8/297: readlink d0/db/dd/d11/d1b/d22/l46 0 2024-04-12T05:37:03.774 INFO:tasks.workunit.client.0.smithi032.stdout:6/354: mkdir d4/d14/d1a/d79 0 2024-04-12T05:37:03.776 INFO:tasks.workunit.client.0.smithi032.stdout:2/306: rename dd/fe to dd/d1b/d1c/d2c/d32/d3a/f56 0 2024-04-12T05:37:03.777 INFO:tasks.workunit.client.0.smithi032.stdout:6/355: truncate d4/d21/d2e/d32/f61 143582 0 2024-04-12T05:37:03.778 INFO:tasks.workunit.client.0.smithi032.stdout:0/407: sync 2024-04-12T05:37:03.778 INFO:tasks.workunit.client.0.smithi032.stdout:9/295: sync 2024-04-12T05:37:03.778 INFO:tasks.workunit.client.0.smithi032.stdout:5/405: mkdir d2/df/d31/d80 0 2024-04-12T05:37:03.781 INFO:tasks.workunit.client.0.smithi032.stdout:8/298: dread d0/db/f55 [0,4194304] 0 2024-04-12T05:37:03.783 INFO:tasks.workunit.client.0.smithi032.stdout:4/282: dread d4/d13/d19/d28/d2f/f4e [0,4194304] 0 2024-04-12T05:37:03.785 INFO:tasks.workunit.client.0.smithi032.stdout:2/307: symlink dd/d19/l57 0 2024-04-12T05:37:03.785 INFO:tasks.workunit.client.0.smithi032.stdout:2/308: chown dd/f14 115486 1 2024-04-12T05:37:03.786 INFO:tasks.workunit.client.0.smithi032.stdout:1/262: symlink d5/de/d17/d35/d3b/d49/l51 0 2024-04-12T05:37:03.786 INFO:tasks.workunit.client.0.smithi032.stdout:7/300: rmdir d0 39 2024-04-12T05:37:03.787 INFO:tasks.workunit.client.0.smithi032.stdout:8/299: dread d0/db/dd/d11/d1b/d22/d54/f64 [0,4194304] 0 2024-04-12T05:37:03.787 INFO:tasks.workunit.client.0.smithi032.stdout:1/263: write d5/ff [4150737,43815] 0 2024-04-12T05:37:03.788 INFO:tasks.workunit.client.0.smithi032.stdout:6/356: creat d4/d21/d2e/d32/f7a x:0 0 0 2024-04-12T05:37:03.789 INFO:tasks.workunit.client.0.smithi032.stdout:0/408: creat d2/d8/f88 x:0 0 0 2024-04-12T05:37:03.793 INFO:tasks.workunit.client.0.smithi032.stdout:4/283: creat d4/d13/d44/f5c x:0 0 0 2024-04-12T05:37:03.796 INFO:tasks.workunit.client.0.smithi032.stdout:1/264: symlink d5/de/d17/d35/d3e/l52 0 2024-04-12T05:37:03.798 INFO:tasks.workunit.client.0.smithi032.stdout:1/265: write d5/de/d17/d1b/f21 [385123,93310] 0 2024-04-12T05:37:03.799 INFO:tasks.workunit.client.0.smithi032.stdout:1/266: stat d5/de/f1f 0 2024-04-12T05:37:03.801 INFO:tasks.workunit.client.0.smithi032.stdout:7/301: rename d0/d1c/f1f to d0/d13/f64 0 2024-04-12T05:37:03.804 INFO:tasks.workunit.client.0.smithi032.stdout:4/284: dread d4/d13/fd [0,4194304] 0 2024-04-12T05:37:03.806 INFO:tasks.workunit.client.0.smithi032.stdout:1/267: symlink d5/de/d17/d35/d3b/l53 0 2024-04-12T05:37:03.807 INFO:tasks.workunit.client.0.smithi032.stdout:7/302: mkdir d0/dd/d51/d65 0 2024-04-12T05:37:03.810 INFO:tasks.workunit.client.0.smithi032.stdout:8/300: truncate d0/db/f19 1865180 0 2024-04-12T05:37:03.812 INFO:tasks.workunit.client.0.smithi032.stdout:9/296: dwrite d9/d15/d2c/f52 [0,4194304] 0 2024-04-12T05:37:03.812 INFO:tasks.workunit.client.0.smithi032.stdout:6/357: getdents d4/d14/d1a/d37/d69 0 2024-04-12T05:37:03.813 INFO:tasks.workunit.client.0.smithi032.stdout:4/285: dread d4/d13/d19/d28/d2f/d3b/f3c [0,4194304] 0 2024-04-12T05:37:03.813 INFO:tasks.workunit.client.0.smithi032.stdout:7/303: write d0/d13/d21/f57 [853055,48087] 0 2024-04-12T05:37:03.815 INFO:tasks.workunit.client.0.smithi032.stdout:3/292: sync 2024-04-12T05:37:03.815 INFO:tasks.workunit.client.0.smithi032.stdout:1/268: rename d5/d48 to d5/d2c/d4d/d54 0 2024-04-12T05:37:03.815 INFO:tasks.workunit.client.0.smithi032.stdout:6/358: chown d4/d21 15862 1 2024-04-12T05:37:03.817 INFO:tasks.workunit.client.0.smithi032.stdout:5/406: dwrite d2/df/d25/d1e/f23 [0,4194304] 0 2024-04-12T05:37:03.821 INFO:tasks.workunit.client.0.smithi032.stdout:1/269: truncate d5/f30 242728 0 2024-04-12T05:37:03.822 INFO:tasks.workunit.client.0.smithi032.stdout:9/297: creat d9/d15/d2c/f62 x:0 0 0 2024-04-12T05:37:03.822 INFO:tasks.workunit.client.0.smithi032.stdout:7/304: truncate d0/d34/f5d 1880480 0 2024-04-12T05:37:03.824 INFO:tasks.workunit.client.0.smithi032.stdout:4/286: rmdir d4/d13/d19/d28 39 2024-04-12T05:37:03.824 INFO:tasks.workunit.client.0.smithi032.stdout:9/298: chown d9/d15 2 1 2024-04-12T05:37:03.827 INFO:tasks.workunit.client.0.smithi032.stdout:0/409: dwrite d2/d8/d17/d1b/d20/f45 [0,4194304] 0 2024-04-12T05:37:03.829 INFO:tasks.workunit.client.0.smithi032.stdout:1/270: creat d5/de/f55 x:0 0 0 2024-04-12T05:37:03.829 INFO:tasks.workunit.client.0.smithi032.stdout:5/407: creat d2/df/d25/d43/f81 x:0 0 0 2024-04-12T05:37:03.831 INFO:tasks.workunit.client.0.smithi032.stdout:8/301: write d0/db/f19 [1392567,15539] 0 2024-04-12T05:37:03.832 INFO:tasks.workunit.client.0.smithi032.stdout:3/293: link d12/c55 d12/d2f/c5d 0 2024-04-12T05:37:03.835 INFO:tasks.workunit.client.0.smithi032.stdout:3/294: fdatasync d12/d24/f2c 0 2024-04-12T05:37:03.835 INFO:tasks.workunit.client.0.smithi032.stdout:9/299: link d9/d15/d2c/f5e d9/d15/d1a/d1d/f63 0 2024-04-12T05:37:03.836 INFO:tasks.workunit.client.0.smithi032.stdout:8/302: write d0/db/dd/d11/d1b/d22/f45 [51794,26386] 0 2024-04-12T05:37:03.836 INFO:tasks.workunit.client.0.smithi032.stdout:7/305: dread d0/de/df/d29/f46 [0,4194304] 0 2024-04-12T05:37:03.838 INFO:tasks.workunit.client.0.smithi032.stdout:3/295: write d12/d2e/d3a/d2d/f4e [773224,911] 0 2024-04-12T05:37:03.838 INFO:tasks.workunit.client.0.smithi032.stdout:9/300: chown d9/d15/d1a/d24/d27/f2d 53 1 2024-04-12T05:37:03.838 INFO:tasks.workunit.client.0.smithi032.stdout:0/410: rename d2/d8/d17/d1b/f4d to d2/dc/d41/f89 0 2024-04-12T05:37:03.839 INFO:tasks.workunit.client.0.smithi032.stdout:5/408: mknod d2/df/c82 0 2024-04-12T05:37:03.840 INFO:tasks.workunit.client.0.smithi032.stdout:3/296: rename d12/d2e/d3a to d12/d2e/d3a/d2d/d5e 22 2024-04-12T05:37:03.841 INFO:tasks.workunit.client.0.smithi032.stdout:5/409: dread - d2/df/d25/d43/f81 zero size 2024-04-12T05:37:03.843 INFO:tasks.workunit.client.0.smithi032.stdout:8/303: rmdir d0/db 39 2024-04-12T05:37:03.845 INFO:tasks.workunit.client.0.smithi032.stdout:0/411: dwrite d2/d8/d17/f6b [0,4194304] 0 2024-04-12T05:37:03.847 INFO:tasks.workunit.client.0.smithi032.stdout:5/410: fsync d2/df/d25/f8 0 2024-04-12T05:37:03.848 INFO:tasks.workunit.client.0.smithi032.stdout:4/287: link d4/d13/d19/f34 d4/d13/d19/d28/d2f/d3b/f5d 0 2024-04-12T05:37:03.849 INFO:tasks.workunit.client.0.smithi032.stdout:8/304: read - d0/db/dd/f27 zero size 2024-04-12T05:37:03.849 INFO:tasks.workunit.client.0.smithi032.stdout:9/301: rename d9/d15/d1a/d24/d27/l4a to d9/d32/l64 0 2024-04-12T05:37:03.871 INFO:tasks.workunit.client.0.smithi032.stdout:6/359: truncate d4/d14/d3c/d75/d55/f72 846433 0 2024-04-12T05:37:03.873 INFO:tasks.workunit.client.0.smithi032.stdout:7/306: dwrite d0/d13/f4b [0,4194304] 0 2024-04-12T05:37:03.876 INFO:tasks.workunit.client.0.smithi032.stdout:9/302: mkdir d9/d15/d1a/d31/d65 0 2024-04-12T05:37:03.877 INFO:tasks.workunit.client.0.smithi032.stdout:4/288: rename d4/d13/d19/d28/f47 to d4/d13/d19/d28/d4d/f5e 0 2024-04-12T05:37:03.878 INFO:tasks.workunit.client.0.smithi032.stdout:6/360: rmdir d4 39 2024-04-12T05:37:03.879 INFO:tasks.workunit.client.0.smithi032.stdout:0/412: dwrite d2/d29/f47 [0,4194304] 0 2024-04-12T05:37:03.880 INFO:tasks.workunit.client.0.smithi032.stdout:5/411: creat d2/dd/f83 x:0 0 0 2024-04-12T05:37:03.885 INFO:tasks.workunit.client.0.smithi032.stdout:9/303: write d9/d15/f30 [4198053,106852] 0 2024-04-12T05:37:03.886 INFO:tasks.workunit.client.0.smithi032.stdout:4/289: rename d4/d13/f33 to d4/d13/d19/d28/d32/f5f 0 2024-04-12T05:37:03.887 INFO:tasks.workunit.client.0.smithi032.stdout:6/361: stat d4/d14/d3c/c4f 0 2024-04-12T05:37:03.890 INFO:tasks.workunit.client.0.smithi032.stdout:4/290: chown d4/d13/f1f 108274 1 2024-04-12T05:37:03.890 INFO:tasks.workunit.client.0.smithi032.stdout:1/271: dwrite d5/f47 [4194304,4194304] 0 2024-04-12T05:37:03.892 INFO:tasks.workunit.client.0.smithi032.stdout:8/305: dwrite d0/db/dd/f27 [0,4194304] 0 2024-04-12T05:37:03.893 INFO:tasks.workunit.client.0.smithi032.stdout:7/307: mknod d0/de/c66 0 2024-04-12T05:37:03.893 INFO:tasks.workunit.client.0.smithi032.stdout:1/272: write d5/d2c/f43 [612491,116826] 0 2024-04-12T05:37:03.896 INFO:tasks.workunit.client.0.smithi032.stdout:8/306: truncate d0/db/f6e 707321 0 2024-04-12T05:37:03.898 INFO:tasks.workunit.client.0.smithi032.stdout:5/412: rmdir d2/df/d25/d1e 39 2024-04-12T05:37:03.899 INFO:tasks.workunit.client.0.smithi032.stdout:9/304: mknod d9/d15/d2c/c66 0 2024-04-12T05:37:03.901 INFO:tasks.workunit.client.0.smithi032.stdout:7/308: symlink d0/d1c/l67 0 2024-04-12T05:37:03.902 INFO:tasks.workunit.client.0.smithi032.stdout:0/413: chown d2/d8/d17/d1b/d20/c72 28 1 2024-04-12T05:37:03.903 INFO:tasks.workunit.client.0.smithi032.stdout:1/273: creat d5/de/d17/d35/d3b/d49/f56 x:0 0 0 2024-04-12T05:37:03.903 INFO:tasks.workunit.client.0.smithi032.stdout:8/307: read d0/db/dd/d11/d1b/d22/f45 [55696,55955] 0 2024-04-12T05:37:03.906 INFO:tasks.workunit.client.0.smithi032.stdout:4/291: dwrite d4/d13/d19/d28/f42 [0,4194304] 0 2024-04-12T05:37:03.910 INFO:tasks.workunit.client.0.smithi032.stdout:9/305: chown d9/le 2 1 2024-04-12T05:37:03.910 INFO:tasks.workunit.client.0.smithi032.stdout:6/362: truncate d4/d14/f6d 1711122 0 2024-04-12T05:37:03.910 INFO:tasks.workunit.client.0.smithi032.stdout:5/413: chown d2/f24 57947 1 2024-04-12T05:37:03.913 INFO:tasks.workunit.client.0.smithi032.stdout:1/274: mknod d5/de/d17/d1b/c57 0 2024-04-12T05:37:03.915 INFO:tasks.workunit.client.0.smithi032.stdout:6/363: write d4/d14/d3c/f48 [485232,73642] 0 2024-04-12T05:37:03.916 INFO:tasks.workunit.client.0.smithi032.stdout:5/414: write d2/df/d25/d43/f51 [90579,115482] 0 2024-04-12T05:37:03.918 INFO:tasks.workunit.client.0.smithi032.stdout:7/309: link d0/d13/f1a d0/de/df/d20/f68 0 2024-04-12T05:37:03.918 INFO:tasks.workunit.client.0.smithi032.stdout:4/292: creat d4/d13/d19/d28/d2f/f60 x:0 0 0 2024-04-12T05:37:03.919 INFO:tasks.workunit.client.0.smithi032.stdout:6/364: readlink d4/d14/d1a/l1f 0 2024-04-12T05:37:03.921 INFO:tasks.workunit.client.0.smithi032.stdout:5/415: unlink d2/d3c/c69 0 2024-04-12T05:37:03.922 INFO:tasks.workunit.client.0.smithi032.stdout:0/414: dwrite d2/d8/d17/f2b [0,4194304] 0 2024-04-12T05:37:03.922 INFO:tasks.workunit.client.0.smithi032.stdout:7/310: symlink d0/d1c/l69 0 2024-04-12T05:37:03.923 INFO:tasks.workunit.client.0.smithi032.stdout:8/308: link d0/c25 d0/db/d38/d63/c6f 0 2024-04-12T05:37:03.927 INFO:tasks.workunit.client.0.smithi032.stdout:6/365: symlink d4/d14/d1a/d37/l7b 0 2024-04-12T05:37:03.928 INFO:tasks.workunit.client.0.smithi032.stdout:0/415: mkdir d2/d29/d8a 0 2024-04-12T05:37:03.930 INFO:tasks.workunit.client.0.smithi032.stdout:9/306: dwrite d9/d15/d1a/d1d/f4d [0,4194304] 0 2024-04-12T05:37:03.932 INFO:tasks.workunit.client.0.smithi032.stdout:1/275: dread d5/de/d17/d1b/f33 [0,4194304] 0 2024-04-12T05:37:03.932 INFO:tasks.workunit.client.0.smithi032.stdout:5/416: rename d2/df/d25/f8 to d2/d3c/d5b/d75/f84 0 2024-04-12T05:37:03.933 INFO:tasks.workunit.client.0.smithi032.stdout:7/311: dread d0/d13/f4b [0,4194304] 0 2024-04-12T05:37:03.936 INFO:tasks.workunit.client.0.smithi032.stdout:6/366: symlink d4/d21/d27/l7c 0 2024-04-12T05:37:03.941 INFO:tasks.workunit.client.0.smithi032.stdout:5/417: write d2/d3c/d5b/f5e [7196001,37925] 0 2024-04-12T05:37:03.941 INFO:tasks.workunit.client.0.smithi032.stdout:4/293: link d4/d13/d19/l2c d4/d13/d19/d28/d2f/d46/l61 0 2024-04-12T05:37:03.941 INFO:tasks.workunit.client.0.smithi032.stdout:5/418: chown d2/df/f1d 1 1 2024-04-12T05:37:03.942 INFO:tasks.workunit.client.0.smithi032.stdout:6/367: dread d4/d21/d2e/f39 [0,4194304] 0 2024-04-12T05:37:03.952 INFO:tasks.workunit.client.0.smithi032.stdout:9/307: symlink d9/l67 0 2024-04-12T05:37:03.952 INFO:tasks.workunit.client.0.smithi032.stdout:6/368: write d4/d21/d2e/d32/f36 [454071,99184] 0 2024-04-12T05:37:03.954 INFO:tasks.workunit.client.0.smithi032.stdout:1/276: unlink d5/l9 0 2024-04-12T05:37:03.956 INFO:tasks.workunit.client.0.smithi032.stdout:8/309: link d0/db/dd/d11/l33 d0/db/dd/d11/d1b/l70 0 2024-04-12T05:37:03.956 INFO:tasks.workunit.client.0.smithi032.stdout:6/369: chown d4/d14/d1a/d37/f5d 886892 1 2024-04-12T05:37:03.956 INFO:tasks.workunit.client.0.smithi032.stdout:9/308: write d9/d15/d2c/d41/f51 [988968,58270] 0 2024-04-12T05:37:03.958 INFO:tasks.workunit.client.0.smithi032.stdout:4/294: dread d4/f12 [0,4194304] 0 2024-04-12T05:37:03.959 INFO:tasks.workunit.client.0.smithi032.stdout:8/310: chown d0/c23 9 1 2024-04-12T05:37:03.960 INFO:tasks.workunit.client.0.smithi032.stdout:1/277: rmdir d5/de/d17/d35/d3b/d49 39 2024-04-12T05:37:03.962 INFO:tasks.workunit.client.0.smithi032.stdout:4/295: symlink d4/d13/d19/d28/d32/l62 0 2024-04-12T05:37:03.964 INFO:tasks.workunit.client.0.smithi032.stdout:0/416: truncate d2/d8/df/f31 1356494 0 2024-04-12T05:37:03.966 INFO:tasks.workunit.client.0.smithi032.stdout:6/370: creat d4/d21/f7d x:0 0 0 2024-04-12T05:37:03.966 INFO:tasks.workunit.client.0.smithi032.stdout:5/419: truncate d2/df/d31/f46 2704751 0 2024-04-12T05:37:03.968 INFO:tasks.workunit.client.0.smithi032.stdout:8/311: getdents d0/db/dd/d11/d1b/d22 0 2024-04-12T05:37:03.968 INFO:tasks.workunit.client.0.smithi032.stdout:6/371: dread d4/f6 [0,4194304] 0 2024-04-12T05:37:03.971 INFO:tasks.workunit.client.0.smithi032.stdout:5/420: symlink d2/dd/l85 0 2024-04-12T05:37:03.971 INFO:tasks.workunit.client.0.smithi032.stdout:4/296: dread d4/d13/d19/f30 [0,4194304] 0 2024-04-12T05:37:03.973 INFO:tasks.workunit.client.0.smithi032.stdout:6/372: dread d4/d14/f46 [0,4194304] 0 2024-04-12T05:37:03.977 INFO:tasks.workunit.client.0.smithi032.stdout:0/417: truncate d2/dc/d3e/f46 30808 0 2024-04-12T05:37:03.978 INFO:tasks.workunit.client.0.smithi032.stdout:5/421: mkdir d2/d65/d78/d86 0 2024-04-12T05:37:03.979 INFO:tasks.workunit.client.0.smithi032.stdout:6/373: creat d4/d14/d25/f7e x:0 0 0 2024-04-12T05:37:03.980 INFO:tasks.workunit.client.0.smithi032.stdout:5/422: creat d2/d3c/d5b/f87 x:0 0 0 2024-04-12T05:37:03.990 INFO:tasks.workunit.client.0.smithi032.stdout:8/312: dwrite d0/d36/f3a [0,4194304] 0 2024-04-12T05:37:03.992 INFO:tasks.workunit.client.0.smithi032.stdout:5/423: dwrite d2/d3c/d5b/f39 [0,4194304] 0 2024-04-12T05:37:03.992 INFO:tasks.workunit.client.0.smithi032.stdout:6/374: dread d4/d14/d1a/f49 [0,4194304] 0 2024-04-12T05:37:03.993 INFO:tasks.workunit.client.0.smithi032.stdout:8/313: dread - d0/db/dd/f5c zero size 2024-04-12T05:37:03.994 INFO:tasks.workunit.client.0.smithi032.stdout:5/424: write d2/df/f32 [667048,40371] 0 2024-04-12T05:37:03.996 INFO:tasks.workunit.client.0.smithi032.stdout:2/309: sync 2024-04-12T05:37:03.997 INFO:tasks.workunit.client.0.smithi032.stdout:5/425: chown d2/dd/f2d 1248251 1 2024-04-12T05:37:03.997 INFO:tasks.workunit.client.0.smithi032.stdout:8/314: creat d0/db/dd/d11/d1b/d5f/f71 x:0 0 0 2024-04-12T05:37:03.999 INFO:tasks.workunit.client.0.smithi032.stdout:2/310: creat dd/d1b/f58 x:0 0 0 2024-04-12T05:37:04.002 INFO:tasks.workunit.client.0.smithi032.stdout:5/426: creat d2/df/d31/d80/f88 x:0 0 0 2024-04-12T05:37:04.003 INFO:tasks.workunit.client.0.smithi032.stdout:6/375: link d4/d14/d3c/d75/d55/d59/c6a d4/d21/c7f 0 2024-04-12T05:37:04.006 INFO:tasks.workunit.client.0.smithi032.stdout:5/427: chown d2/df/d25/d1e/c6a 271 1 2024-04-12T05:37:04.007 INFO:tasks.workunit.client.0.smithi032.stdout:8/315: dwrite d0/db/dd/d11/d1b/d22/d51/f59 [0,4194304] 0 2024-04-12T05:37:04.012 INFO:tasks.workunit.client.0.smithi032.stdout:6/376: getdents d4/d21/d2e/d32 0 2024-04-12T05:37:04.012 INFO:tasks.workunit.client.0.smithi032.stdout:6/377: stat d4/d14/f46 0 2024-04-12T05:37:04.013 INFO:tasks.workunit.client.0.smithi032.stdout:8/316: mkdir d0/d72 0 2024-04-12T05:37:04.013 INFO:tasks.workunit.client.0.smithi032.stdout:2/311: write dd/d1b/f34 [552209,18676] 0 2024-04-12T05:37:04.014 INFO:tasks.workunit.client.0.smithi032.stdout:6/378: symlink d4/d21/d2e/d32/l80 0 2024-04-12T05:37:04.018 INFO:tasks.workunit.client.0.smithi032.stdout:6/379: write d4/d14/d3c/d75/d55/f70 [148807,96051] 0 2024-04-12T05:37:04.022 INFO:tasks.workunit.client.0.smithi032.stdout:5/428: dwrite d2/f24 [0,4194304] 0 2024-04-12T05:37:04.026 INFO:tasks.workunit.client.0.smithi032.stdout:5/429: write d2/df/d25/f16 [3213751,125404] 0 2024-04-12T05:37:04.028 INFO:tasks.workunit.client.0.smithi032.stdout:8/317: dwrite d0/db/dd/d11/d1b/d22/d54/f64 [0,4194304] 0 2024-04-12T05:37:04.030 INFO:tasks.workunit.client.0.smithi032.stdout:2/312: dread dd/f1e [0,4194304] 0 2024-04-12T05:37:04.032 INFO:tasks.workunit.client.0.smithi032.stdout:8/318: readlink d0/db/dd/d11/d1b/d22/d35/l48 0 2024-04-12T05:37:04.032 INFO:tasks.workunit.client.0.smithi032.stdout:5/430: mknod d2/dd/d6f/c89 0 2024-04-12T05:37:04.034 INFO:tasks.workunit.client.0.smithi032.stdout:6/380: dwrite d4/d21/d2e/d32/f61 [0,4194304] 0 2024-04-12T05:37:04.034 INFO:tasks.workunit.client.0.smithi032.stdout:2/313: dread - dd/d1b/d1c/d2c/d32/f47 zero size 2024-04-12T05:37:04.034 INFO:tasks.workunit.client.0.smithi032.stdout:8/319: stat d0/db/c3e 0 2024-04-12T05:37:04.035 INFO:tasks.workunit.client.0.smithi032.stdout:5/431: chown d2/df/f32 42 1 2024-04-12T05:37:04.036 INFO:tasks.workunit.client.0.smithi032.stdout:8/320: write d0/db/f2e [4022283,29541] 0 2024-04-12T05:37:04.039 INFO:tasks.workunit.client.0.smithi032.stdout:5/432: readlink d2/d3c/d48/d4b/l4f 0 2024-04-12T05:37:04.040 INFO:tasks.workunit.client.0.smithi032.stdout:2/314: symlink dd/d1b/d1c/d2c/l59 0 2024-04-12T05:37:04.041 INFO:tasks.workunit.client.0.smithi032.stdout:2/315: chown dd/d1b/d1c/d2c 0 1 2024-04-12T05:37:04.042 INFO:tasks.workunit.client.0.smithi032.stdout:8/321: mknod d0/db/dd/d11/d1b/d22/d51/c73 0 2024-04-12T05:37:04.042 INFO:tasks.workunit.client.0.smithi032.stdout:2/316: read - dd/f2e zero size 2024-04-12T05:37:04.049 INFO:tasks.workunit.client.0.smithi032.stdout:8/322: getdents d0/db/dd/d11/d1b/d22 0 2024-04-12T05:37:04.050 INFO:tasks.workunit.client.0.smithi032.stdout:6/381: dwrite d4/d21/d2e/f39 [0,4194304] 0 2024-04-12T05:37:04.051 INFO:tasks.workunit.client.0.smithi032.stdout:2/317: dread dd/d1b/d1c/f29 [0,4194304] 0 2024-04-12T05:37:04.052 INFO:tasks.workunit.client.0.smithi032.stdout:6/382: readlink d4/d21/d27/l7c 0 2024-04-12T05:37:04.053 INFO:tasks.workunit.client.0.smithi032.stdout:8/323: mknod d0/d36/c74 0 2024-04-12T05:37:04.054 INFO:tasks.workunit.client.0.smithi032.stdout:2/318: unlink dd/d1b/c3c 0 2024-04-12T05:37:04.054 INFO:tasks.workunit.client.0.smithi032.stdout:8/324: readlink d0/db/dd/d11/d1b/l49 0 2024-04-12T05:37:04.057 INFO:tasks.workunit.client.0.smithi032.stdout:8/325: chown d0/f3 2365058 1 2024-04-12T05:37:04.058 INFO:tasks.workunit.client.0.smithi032.stdout:2/319: fdatasync dd/d1b/d1c/f29 0 2024-04-12T05:37:04.060 INFO:tasks.workunit.client.0.smithi032.stdout:6/383: dwrite d4/d21/d2e/d32/f61 [0,4194304] 0 2024-04-12T05:37:04.062 INFO:tasks.workunit.client.0.smithi032.stdout:6/384: truncate d4/d21/f7d 65447 0 2024-04-12T05:37:04.064 INFO:tasks.workunit.client.0.smithi032.stdout:2/320: rename dd/l41 to dd/d1b/d1c/d2c/d32/l5a 0 2024-04-12T05:37:04.066 INFO:tasks.workunit.client.0.smithi032.stdout:2/321: write dd/d1b/f34 [1407493,103367] 0 2024-04-12T05:37:04.066 INFO:tasks.workunit.client.0.smithi032.stdout:2/322: stat dd/d19/c23 0 2024-04-12T05:37:04.066 INFO:tasks.workunit.client.0.smithi032.stdout:6/385: unlink d4/d14/f51 0 2024-04-12T05:37:04.070 INFO:tasks.workunit.client.0.smithi032.stdout:2/323: unlink dd/d1b/d1c/d2c/d36/f4e 0 2024-04-12T05:37:04.072 INFO:tasks.workunit.client.0.smithi032.stdout:2/324: write dd/d19/d20/f4d [124446,80773] 0 2024-04-12T05:37:04.073 INFO:tasks.workunit.client.0.smithi032.stdout:2/325: chown dd/f27 7150533 1 2024-04-12T05:37:04.074 INFO:tasks.workunit.client.0.smithi032.stdout:6/386: dwrite d4/d21/d2e/f39 [0,4194304] 0 2024-04-12T05:37:04.074 INFO:tasks.workunit.client.0.smithi032.stdout:2/326: fdatasync dd/ff 0 2024-04-12T05:37:04.075 INFO:tasks.workunit.client.0.smithi032.stdout:2/327: dread - dd/f2e zero size 2024-04-12T05:37:04.076 INFO:tasks.workunit.client.0.smithi032.stdout:8/326: write d0/db/dd/d11/d1b/d22/f45 [701342,110001] 0 2024-04-12T05:37:04.080 INFO:tasks.workunit.client.0.smithi032.stdout:3/297: sync 2024-04-12T05:37:04.080 INFO:tasks.workunit.client.0.smithi032.stdout:7/312: sync 2024-04-12T05:37:04.080 INFO:tasks.workunit.client.0.smithi032.stdout:7/313: stat d0/d34 0 2024-04-12T05:37:04.081 INFO:tasks.workunit.client.0.smithi032.stdout:2/328: dread dd/d1b/d1c/d2c/d32/f35 [0,4194304] 0 2024-04-12T05:37:04.089 INFO:tasks.workunit.client.0.smithi032.stdout:3/298: dread d12/f13 [0,4194304] 0 2024-04-12T05:37:04.090 INFO:tasks.workunit.client.0.smithi032.stdout:2/329: rename dd/d1b/d1c/d2c/d32/l5a to dd/d19/d20/l5b 0 2024-04-12T05:37:04.090 INFO:tasks.workunit.client.0.smithi032.stdout:6/387: link d4/d21/c7f d4/d14/d1a/d37/d47/d57/c81 0 2024-04-12T05:37:04.092 INFO:tasks.workunit.client.0.smithi032.stdout:7/314: creat d0/de/df/f6a x:0 0 0 2024-04-12T05:37:04.092 INFO:tasks.workunit.client.0.smithi032.stdout:8/327: getdents d0/db/dd/d11/d1b/d22/d51 0 2024-04-12T05:37:04.093 INFO:tasks.workunit.client.0.smithi032.stdout:3/299: mknod d12/d2e/d3a/c5f 0 2024-04-12T05:37:04.095 INFO:tasks.workunit.client.0.smithi032.stdout:8/328: chown d0/f2f 884537 1 2024-04-12T05:37:04.095 INFO:tasks.workunit.client.0.smithi032.stdout:5/433: sync 2024-04-12T05:37:04.095 INFO:tasks.workunit.client.0.smithi032.stdout:9/309: sync 2024-04-12T05:37:04.095 INFO:tasks.workunit.client.0.smithi032.stdout:0/418: sync 2024-04-12T05:37:04.095 INFO:tasks.workunit.client.0.smithi032.stdout:1/278: sync 2024-04-12T05:37:04.096 INFO:tasks.workunit.client.0.smithi032.stdout:4/297: sync 2024-04-12T05:37:04.096 INFO:tasks.workunit.client.0.smithi032.stdout:6/388: mknod d4/d14/d1a/d37/d69/d76/c82 0 2024-04-12T05:37:04.096 INFO:tasks.workunit.client.0.smithi032.stdout:2/330: dread dd/d1b/f34 [0,4194304] 0 2024-04-12T05:37:04.097 INFO:tasks.workunit.client.0.smithi032.stdout:7/315: creat d0/de/df/d20/f6b x:0 0 0 2024-04-12T05:37:04.098 INFO:tasks.workunit.client.0.smithi032.stdout:9/310: fdatasync d9/d15/f44 0 2024-04-12T05:37:04.099 INFO:tasks.workunit.client.0.smithi032.stdout:9/311: chown d9/d1f/c42 125677042 1 2024-04-12T05:37:04.102 INFO:tasks.workunit.client.0.smithi032.stdout:1/279: chown d5/l8 393829360 1 2024-04-12T05:37:04.103 INFO:tasks.workunit.client.0.smithi032.stdout:7/316: creat d0/dd/f6c x:0 0 0 2024-04-12T05:37:04.104 INFO:tasks.workunit.client.0.smithi032.stdout:9/312: symlink d9/d15/d1a/d24/d27/l68 0 2024-04-12T05:37:04.104 INFO:tasks.workunit.client.0.smithi032.stdout:3/300: dwrite d12/f2a [0,4194304] 0 2024-04-12T05:37:04.105 INFO:tasks.workunit.client.0.smithi032.stdout:0/419: mknod d2/d8/d17/d1b/c8b 0 2024-04-12T05:37:04.106 INFO:tasks.workunit.client.0.smithi032.stdout:6/389: getdents d4/d74 0 2024-04-12T05:37:04.111 INFO:tasks.workunit.client.0.smithi032.stdout:8/329: creat d0/db/dd/f75 x:0 0 0 2024-04-12T05:37:04.112 INFO:tasks.workunit.client.0.smithi032.stdout:5/434: dread d2/d3c/d5b/d75/f84 [0,4194304] 0 2024-04-12T05:37:04.118 INFO:tasks.workunit.client.0.smithi032.stdout:8/330: fdatasync d0/f2f 0 2024-04-12T05:37:04.119 INFO:tasks.workunit.client.0.smithi032.stdout:7/317: dwrite d0/d10/f62 [0,4194304] 0 2024-04-12T05:37:04.120 INFO:tasks.workunit.client.0.smithi032.stdout:3/301: dwrite d12/d24/f38 [0,4194304] 0 2024-04-12T05:37:04.123 INFO:tasks.workunit.client.0.smithi032.stdout:0/420: rename d2/d29/c3a to d2/d8/c8c 0 2024-04-12T05:37:04.123 INFO:tasks.workunit.client.0.smithi032.stdout:5/435: symlink d2/d65/d78/d86/l8a 0 2024-04-12T05:37:04.124 INFO:tasks.workunit.client.0.smithi032.stdout:0/421: chown d2/d28/d50 1 1 2024-04-12T05:37:04.124 INFO:tasks.workunit.client.0.smithi032.stdout:2/331: link dd/d19/d20/l5b dd/d1b/d1c/d2c/d32/l5c 0 2024-04-12T05:37:04.125 INFO:tasks.workunit.client.0.smithi032.stdout:8/331: mknod d0/db/d38/c76 0 2024-04-12T05:37:04.125 INFO:tasks.workunit.client.0.smithi032.stdout:2/332: chown dd/d19/l26 118208 1 2024-04-12T05:37:04.126 INFO:tasks.workunit.client.0.smithi032.stdout:9/313: truncate f5 3694566 0 2024-04-12T05:37:04.129 INFO:tasks.workunit.client.0.smithi032.stdout:5/436: dread d2/d3c/d5b/f39 [0,4194304] 0 2024-04-12T05:37:04.131 INFO:tasks.workunit.client.0.smithi032.stdout:0/422: creat d2/d28/d50/d51/f8d x:0 0 0 2024-04-12T05:37:04.133 INFO:tasks.workunit.client.0.smithi032.stdout:3/302: symlink d12/d2e/l60 0 2024-04-12T05:37:04.135 INFO:tasks.workunit.client.0.smithi032.stdout:2/333: mknod dd/d1b/c5d 0 2024-04-12T05:37:04.137 INFO:tasks.workunit.client.0.smithi032.stdout:5/437: dread d2/dd/f2d [0,4194304] 0 2024-04-12T05:37:04.139 INFO:tasks.workunit.client.0.smithi032.stdout:3/303: dread d12/f13 [0,4194304] 0 2024-04-12T05:37:04.139 INFO:tasks.workunit.client.0.smithi032.stdout:7/318: link d0/de/f12 d0/dd/d51/f6d 0 2024-04-12T05:37:04.140 INFO:tasks.workunit.client.0.smithi032.stdout:8/332: mkdir d0/db/d38/d63/d77 0 2024-04-12T05:37:04.140 INFO:tasks.workunit.client.0.smithi032.stdout:0/423: chown d2/d8/c1a 6 1 2024-04-12T05:37:04.141 INFO:tasks.workunit.client.0.smithi032.stdout:9/314: dread d9/d15/d2c/f23 [4194304,4194304] 0 2024-04-12T05:37:04.142 INFO:tasks.workunit.client.0.smithi032.stdout:6/390: dread d4/d14/f6d [0,4194304] 0 2024-04-12T05:37:04.143 INFO:tasks.workunit.client.0.smithi032.stdout:2/334: mknod dd/d1b/d42/c5e 0 2024-04-12T05:37:04.143 INFO:tasks.workunit.client.0.smithi032.stdout:1/280: dread d5/ff [0,4194304] 0 2024-04-12T05:37:04.145 INFO:tasks.workunit.client.0.smithi032.stdout:7/319: unlink d0/de/df/f6a 0 2024-04-12T05:37:04.146 INFO:tasks.workunit.client.0.smithi032.stdout:3/304: symlink d12/d2e/l61 0 2024-04-12T05:37:04.146 INFO:tasks.workunit.client.0.smithi032.stdout:2/335: truncate dd/d19/f22 652926 0 2024-04-12T05:37:04.147 INFO:tasks.workunit.client.0.smithi032.stdout:9/315: mkdir d9/d15/d1a/d69 0 2024-04-12T05:37:04.155 INFO:tasks.workunit.client.0.smithi032.stdout:8/333: creat d0/db/dd/d11/d1b/f78 x:0 0 0 2024-04-12T05:37:04.155 INFO:tasks.workunit.client.0.smithi032.stdout:6/391: mknod d4/d14/d1a/d37/c83 0 2024-04-12T05:37:04.156 INFO:tasks.workunit.client.0.smithi032.stdout:3/305: mknod d12/d2e/d3a/c62 0 2024-04-12T05:37:04.158 INFO:tasks.workunit.client.0.smithi032.stdout:1/281: dwrite d5/f30 [0,4194304] 0 2024-04-12T05:37:04.158 INFO:tasks.workunit.client.0.smithi032.stdout:7/320: write d0/f5 [5720002,103347] 0 2024-04-12T05:37:04.161 INFO:tasks.workunit.client.0.smithi032.stdout:0/424: dread d2/d8/df/f31 [0,4194304] 0 2024-04-12T05:37:04.163 INFO:tasks.workunit.client.0.smithi032.stdout:2/336: dwrite dd/d1b/f58 [0,4194304] 0 2024-04-12T05:37:04.165 INFO:tasks.workunit.client.0.smithi032.stdout:3/306: mkdir d12/d2e/d3a/d63 0 2024-04-12T05:37:04.168 INFO:tasks.workunit.client.0.smithi032.stdout:7/321: chown d0/d1c/d4c/d26/f5c 4567 1 2024-04-12T05:37:04.172 INFO:tasks.workunit.client.0.smithi032.stdout:8/334: dread d0/db/f2e [0,4194304] 0 2024-04-12T05:37:04.172 INFO:tasks.workunit.client.0.smithi032.stdout:0/425: creat d2/d28/f8e x:0 0 0 2024-04-12T05:37:04.174 INFO:tasks.workunit.client.0.smithi032.stdout:1/282: creat d5/de/d17/d35/f58 x:0 0 0 2024-04-12T05:37:04.175 INFO:tasks.workunit.client.0.smithi032.stdout:9/316: dread d9/d15/d1a/d1d/d5a/f61 [0,4194304] 0 2024-04-12T05:37:04.175 INFO:tasks.workunit.client.0.smithi032.stdout:3/307: rmdir d12/d2f 39 2024-04-12T05:37:04.176 INFO:tasks.workunit.client.0.smithi032.stdout:7/322: fsync d0/d13/f27 0 2024-04-12T05:37:04.177 INFO:tasks.workunit.client.0.smithi032.stdout:8/335: mknod d0/d58/c79 0 2024-04-12T05:37:04.179 INFO:tasks.workunit.client.0.smithi032.stdout:6/392: dwrite d4/d21/d2e/d32/f36 [0,4194304] 0 2024-04-12T05:37:04.180 INFO:tasks.workunit.client.0.smithi032.stdout:0/426: write d2/d28/d87/f43 [3840106,23586] 0 2024-04-12T05:37:04.182 INFO:tasks.workunit.client.0.smithi032.stdout:9/317: write d9/d32/f53 [243446,61588] 0 2024-04-12T05:37:04.182 INFO:tasks.workunit.client.0.smithi032.stdout:7/323: dread - d0/f3e zero size 2024-04-12T05:37:04.182 INFO:tasks.workunit.client.0.smithi032.stdout:2/337: dwrite dd/d19/d20/f53 [0,4194304] 0 2024-04-12T05:37:04.184 INFO:tasks.workunit.client.0.smithi032.stdout:2/338: chown dd/d1b/f58 380 1 2024-04-12T05:37:04.190 INFO:tasks.workunit.client.0.smithi032.stdout:2/339: stat dd/d1b/d1c/c31 0 2024-04-12T05:37:04.195 INFO:tasks.workunit.client.0.smithi032.stdout:2/340: creat dd/d19/f5f x:0 0 0 2024-04-12T05:37:04.197 INFO:tasks.workunit.client.0.smithi032.stdout:9/318: write d9/d15/d2c/f23 [2202103,46373] 0 2024-04-12T05:37:04.199 INFO:tasks.workunit.client.0.smithi032.stdout:2/341: stat dd/d19/c55 0 2024-04-12T05:37:04.204 INFO:tasks.workunit.client.0.smithi032.stdout:1/283: dwrite d5/de/d17/d1b/f1d [0,4194304] 0 2024-04-12T05:37:04.204 INFO:tasks.workunit.client.0.smithi032.stdout:3/308: dwrite d12/d24/f4d [0,4194304] 0 2024-04-12T05:37:04.205 INFO:tasks.workunit.client.0.smithi032.stdout:7/324: mknod d0/d1c/d4c/d26/c6e 0 2024-04-12T05:37:04.207 INFO:tasks.workunit.client.0.smithi032.stdout:9/319: fdatasync d9/f34 0 2024-04-12T05:37:04.214 INFO:tasks.workunit.client.0.smithi032.stdout:1/284: write d5/f47 [1260742,59271] 0 2024-04-12T05:37:04.214 INFO:tasks.workunit.client.0.smithi032.stdout:2/342: creat dd/d1b/d1c/d2c/d32/f60 x:0 0 0 2024-04-12T05:37:04.216 INFO:tasks.workunit.client.0.smithi032.stdout:6/393: dwrite d4/d14/f46 [0,4194304] 0 2024-04-12T05:37:04.220 INFO:tasks.workunit.client.0.smithi032.stdout:0/427: dwrite d2/dc/d41/f89 [0,4194304] 0 2024-04-12T05:37:04.225 INFO:tasks.workunit.client.0.smithi032.stdout:1/285: mknod d5/de/d17/d1b/c59 0 2024-04-12T05:37:04.227 INFO:tasks.workunit.client.0.smithi032.stdout:3/309: dwrite d12/d24/f4d [0,4194304] 0 2024-04-12T05:37:04.227 INFO:tasks.workunit.client.0.smithi032.stdout:8/336: dwrite d0/db/dd/d11/d1b/d22/d29/f31 [0,4194304] 0 2024-04-12T05:37:04.229 INFO:tasks.workunit.client.0.smithi032.stdout:7/325: rename d0/f9 to d0/f6f 0 2024-04-12T05:37:04.234 INFO:tasks.workunit.client.0.smithi032.stdout:0/428: symlink d2/d28/l8f 0 2024-04-12T05:37:04.235 INFO:tasks.workunit.client.0.smithi032.stdout:9/320: getdents d9/d1f 0 2024-04-12T05:37:04.235 INFO:tasks.workunit.client.0.smithi032.stdout:2/343: dwrite dd/d1b/f34 [0,4194304] 0 2024-04-12T05:37:04.237 INFO:tasks.workunit.client.0.smithi032.stdout:1/286: write d5/de/f28 [1268464,96365] 0 2024-04-12T05:37:04.238 INFO:tasks.workunit.client.0.smithi032.stdout:8/337: rmdir d0/d36 39 2024-04-12T05:37:04.238 INFO:tasks.workunit.client.0.smithi032.stdout:9/321: chown d9/d15/d1a/d24/d27/c2b 588640 1 2024-04-12T05:37:04.239 INFO:tasks.workunit.client.0.smithi032.stdout:0/429: chown d2/d8/df/d24/f5c 14 1 2024-04-12T05:37:04.241 INFO:tasks.workunit.client.0.smithi032.stdout:2/344: dread - dd/d1b/d1c/d2c/d32/f60 zero size 2024-04-12T05:37:04.241 INFO:tasks.workunit.client.0.smithi032.stdout:3/310: symlink d12/l64 0 2024-04-12T05:37:04.244 INFO:tasks.workunit.client.0.smithi032.stdout:6/394: getdents d4/d14/d3c 0 2024-04-12T05:37:04.245 INFO:tasks.workunit.client.0.smithi032.stdout:8/338: creat d0/d72/f7a x:0 0 0 2024-04-12T05:37:04.246 INFO:tasks.workunit.client.0.smithi032.stdout:6/395: chown d4/l20 466 1 2024-04-12T05:37:04.246 INFO:tasks.workunit.client.0.smithi032.stdout:0/430: truncate d2/d8/df/d24/f5c 1669576 0 2024-04-12T05:37:04.247 INFO:tasks.workunit.client.0.smithi032.stdout:8/339: dread - d0/db/f4e zero size 2024-04-12T05:37:04.248 INFO:tasks.workunit.client.0.smithi032.stdout:2/345: mknod dd/d1b/d1c/c61 0 2024-04-12T05:37:04.249 INFO:tasks.workunit.client.0.smithi032.stdout:7/326: getdents d0/d13/d21/d30/d31 0 2024-04-12T05:37:04.250 INFO:tasks.workunit.client.0.smithi032.stdout:1/287: dwrite d5/de/f11 [0,4194304] 0 2024-04-12T05:37:04.252 INFO:tasks.workunit.client.0.smithi032.stdout:6/396: symlink d4/d74/l84 0 2024-04-12T05:37:04.253 INFO:tasks.workunit.client.0.smithi032.stdout:1/288: truncate d5/d2c/f3c 723090 0 2024-04-12T05:37:04.255 INFO:tasks.workunit.client.0.smithi032.stdout:3/311: rename d12/l1a to d12/d2e/l65 0 2024-04-12T05:37:04.255 INFO:tasks.workunit.client.0.smithi032.stdout:0/431: rmdir d2/d8/d17 39 2024-04-12T05:37:04.258 INFO:tasks.workunit.client.0.smithi032.stdout:8/340: symlink d0/db/dd/d11/d1b/d22/d29/l7b 0 2024-04-12T05:37:04.259 INFO:tasks.workunit.client.0.smithi032.stdout:1/289: creat d5/d2c/d4d/d54/f5a x:0 0 0 2024-04-12T05:37:04.260 INFO:tasks.workunit.client.0.smithi032.stdout:8/341: write d0/db/f60 [314908,121597] 0 2024-04-12T05:37:04.262 INFO:tasks.workunit.client.0.smithi032.stdout:0/432: rmdir d2/d28/d50/d51 39 2024-04-12T05:37:04.263 INFO:tasks.workunit.client.0.smithi032.stdout:0/433: chown d2/dc/d41/l44 893033663 1 2024-04-12T05:37:04.266 INFO:tasks.workunit.client.0.smithi032.stdout:2/346: getdents dd/d1b/d1c/d2c/d32 0 2024-04-12T05:37:04.267 INFO:tasks.workunit.client.0.smithi032.stdout:2/347: readlink dd/d1b/l3f 0 2024-04-12T05:37:04.268 INFO:tasks.workunit.client.0.smithi032.stdout:8/342: unlink d0/d36/f3a 0 2024-04-12T05:37:04.268 INFO:tasks.workunit.client.0.smithi032.stdout:1/290: dwrite d5/de/f55 [0,4194304] 0 2024-04-12T05:37:04.269 INFO:tasks.workunit.client.0.smithi032.stdout:0/434: mkdir d2/d28/d87/d90 0 2024-04-12T05:37:04.270 INFO:tasks.workunit.client.0.smithi032.stdout:2/348: creat dd/d19/d20/f62 x:0 0 0 2024-04-12T05:37:04.273 INFO:tasks.workunit.client.0.smithi032.stdout:4/298: sync 2024-04-12T05:37:04.273 INFO:tasks.workunit.client.0.smithi032.stdout:0/435: stat d2/d8/d66 0 2024-04-12T05:37:04.274 INFO:tasks.workunit.client.0.smithi032.stdout:2/349: mknod dd/d1b/d1c/d2c/d32/c63 0 2024-04-12T05:37:04.275 INFO:tasks.workunit.client.0.smithi032.stdout:8/343: getdents d0/db/d38/d63/d77 0 2024-04-12T05:37:04.276 INFO:tasks.workunit.client.0.smithi032.stdout:4/299: stat d4/d13/d44/f5a 0 2024-04-12T05:37:04.276 INFO:tasks.workunit.client.0.smithi032.stdout:0/436: dread - d2/d28/d87/f71 zero size 2024-04-12T05:37:04.276 INFO:tasks.workunit.client.0.smithi032.stdout:2/350: fsync dd/d19/f22 0 2024-04-12T05:37:04.278 INFO:tasks.workunit.client.0.smithi032.stdout:8/344: write d0/db/f4e [34035,9337] 0 2024-04-12T05:37:04.281 INFO:tasks.workunit.client.0.smithi032.stdout:1/291: dwrite d5/de/d17/d1b/f1d [0,4194304] 0 2024-04-12T05:37:04.282 INFO:tasks.workunit.client.0.smithi032.stdout:4/300: creat d4/d13/d19/d28/d4d/f63 x:0 0 0 2024-04-12T05:37:04.288 INFO:tasks.workunit.client.0.smithi032.stdout:8/345: symlink d0/l7c 0 2024-04-12T05:37:04.288 INFO:tasks.workunit.client.0.smithi032.stdout:0/437: read d2/dc/d3e/f46 [27717,99722] 0 2024-04-12T05:37:04.288 INFO:tasks.workunit.client.0.smithi032.stdout:4/301: write d4/d13/ff [1091946,6802] 0 2024-04-12T05:37:04.288 INFO:tasks.workunit.client.0.smithi032.stdout:7/327: truncate d0/d1c/d4c/f4d 282609 0 2024-04-12T05:37:04.289 INFO:tasks.workunit.client.0.smithi032.stdout:6/397: truncate f3 3731557 0 2024-04-12T05:37:04.291 INFO:tasks.workunit.client.0.smithi032.stdout:2/351: dwrite dd/ff [0,4194304] 0 2024-04-12T05:37:04.291 INFO:tasks.workunit.client.0.smithi032.stdout:0/438: write d2/f14 [1898971,43448] 0 2024-04-12T05:37:04.291 INFO:tasks.workunit.client.0.smithi032.stdout:7/328: read d0/d10/f62 [2940150,63736] 0 2024-04-12T05:37:04.299 INFO:tasks.workunit.client.0.smithi032.stdout:0/439: fdatasync d2/fe 0 2024-04-12T05:37:04.301 INFO:tasks.workunit.client.0.smithi032.stdout:4/302: chown d4/d13/d19/d28/d32/c56 5227421 1 2024-04-12T05:37:04.302 INFO:tasks.workunit.client.0.smithi032.stdout:1/292: dwrite d5/de/d17/f26 [0,4194304] 0 2024-04-12T05:37:04.303 INFO:tasks.workunit.client.0.smithi032.stdout:7/329: mkdir d0/d10/d70 0 2024-04-12T05:37:04.305 INFO:tasks.workunit.client.0.smithi032.stdout:2/352: rename dd/d1b/d1c/d2c to dd/d1b/d1c/d64 0 2024-04-12T05:37:04.305 INFO:tasks.workunit.client.0.smithi032.stdout:3/312: dread d12/d2e/f36 [0,4194304] 0 2024-04-12T05:37:04.305 INFO:tasks.workunit.client.0.smithi032.stdout:1/293: readlink d5/de/d17/d35/d3e/l52 0 2024-04-12T05:37:04.308 INFO:tasks.workunit.client.0.smithi032.stdout:7/330: truncate d0/de/df/d20/f6b 17715 0 2024-04-12T05:37:04.309 INFO:tasks.workunit.client.0.smithi032.stdout:4/303: creat d4/d13/d19/d28/d2f/d46/f64 x:0 0 0 2024-04-12T05:37:04.310 INFO:tasks.workunit.client.0.smithi032.stdout:8/346: truncate d0/d36/f4a 847262 0 2024-04-12T05:37:04.311 INFO:tasks.workunit.client.0.smithi032.stdout:2/353: fdatasync dd/d1b/d1c/d64/d32/f35 0 2024-04-12T05:37:04.313 INFO:tasks.workunit.client.0.smithi032.stdout:1/294: write d5/de/d17/f20 [11656192,75045] 0 2024-04-12T05:37:04.313 INFO:tasks.workunit.client.0.smithi032.stdout:0/440: symlink d2/d8/d17/d1b/d20/l91 0 2024-04-12T05:37:04.315 INFO:tasks.workunit.client.0.smithi032.stdout:6/398: creat d4/d21/f85 x:0 0 0 2024-04-12T05:37:04.316 INFO:tasks.workunit.client.0.smithi032.stdout:7/331: mkdir d0/d1c/d71 0 2024-04-12T05:37:04.316 INFO:tasks.workunit.client.0.smithi032.stdout:6/399: chown l1 49397893 1 2024-04-12T05:37:04.317 INFO:tasks.workunit.client.0.smithi032.stdout:4/304: unlink d4/d13/d19/l14 0 2024-04-12T05:37:04.318 INFO:tasks.workunit.client.0.smithi032.stdout:8/347: mkdir d0/db/d7d 0 2024-04-12T05:37:04.320 INFO:tasks.workunit.client.0.smithi032.stdout:1/295: mkdir d5/de/d5b 0 2024-04-12T05:37:04.322 INFO:tasks.workunit.client.0.smithi032.stdout:2/354: dwrite dd/d19/f22 [0,4194304] 0 2024-04-12T05:37:04.322 INFO:tasks.workunit.client.0.smithi032.stdout:6/400: mknod d4/d14/c86 0 2024-04-12T05:37:04.323 INFO:tasks.workunit.client.0.smithi032.stdout:1/296: chown d5/de/l18 10 1 2024-04-12T05:37:04.327 INFO:tasks.workunit.client.0.smithi032.stdout:2/355: fsync dd/d19/d20/f4b 0 2024-04-12T05:37:04.329 INFO:tasks.workunit.client.0.smithi032.stdout:8/348: unlink d0/db/f19 0 2024-04-12T05:37:04.329 INFO:tasks.workunit.client.0.smithi032.stdout:3/313: creat d12/d2f/d5b/f66 x:0 0 0 2024-04-12T05:37:04.330 INFO:tasks.workunit.client.0.smithi032.stdout:0/441: creat d2/d28/d87/d90/f92 x:0 0 0 2024-04-12T05:37:04.331 INFO:tasks.workunit.client.0.smithi032.stdout:7/332: symlink d0/de/df/l72 0 2024-04-12T05:37:04.332 INFO:tasks.workunit.client.0.smithi032.stdout:4/305: dwrite d4/f54 [0,4194304] 0 2024-04-12T05:37:04.332 INFO:tasks.workunit.client.0.smithi032.stdout:2/356: fdatasync dd/d19/d20/f4b 0 2024-04-12T05:37:04.334 INFO:tasks.workunit.client.0.smithi032.stdout:7/333: write d0/de/f61 [1021374,18661] 0 2024-04-12T05:37:04.337 INFO:tasks.workunit.client.0.smithi032.stdout:6/401: mknod d4/d14/d1a/d37/d47/d57/c87 0 2024-04-12T05:37:04.337 INFO:tasks.workunit.client.0.smithi032.stdout:8/349: write d0/db/dd/d11/d1b/d22/d51/f59 [1949713,125606] 0 2024-04-12T05:37:04.404 INFO:tasks.workunit.client.0.smithi032.stdout:1/297: dread d5/de/d17/d1b/f1d [0,4194304] 0 2024-04-12T05:37:04.406 INFO:tasks.workunit.client.0.smithi032.stdout:8/350: rename d0 to d0/db/d38/d63/d77/d7e 22 2024-04-12T05:37:04.409 INFO:tasks.workunit.client.0.smithi032.stdout:6/402: mkdir d4/d14/d1a/d88 0 2024-04-12T05:37:04.409 INFO:tasks.workunit.client.0.smithi032.stdout:7/334: mkdir d0/d1c/d71/d73 0 2024-04-12T05:37:04.410 INFO:tasks.workunit.client.0.smithi032.stdout:2/357: dwrite dd/d19/d20/f62 [0,4194304] 0 2024-04-12T05:37:04.411 INFO:tasks.workunit.client.0.smithi032.stdout:1/298: creat d5/de/d17/d1b/f5c x:0 0 0 2024-04-12T05:37:04.412 INFO:tasks.workunit.client.0.smithi032.stdout:8/351: chown d0/c25 128568189 1 2024-04-12T05:37:04.413 INFO:tasks.workunit.client.0.smithi032.stdout:2/358: stat dd/f16 0 2024-04-12T05:37:04.413 INFO:tasks.workunit.client.0.smithi032.stdout:7/335: chown d0/d1c/d4c/d26/f5c 49797 1 2024-04-12T05:37:04.416 INFO:tasks.workunit.client.0.smithi032.stdout:6/403: unlink d4/d14/l4c 0 2024-04-12T05:37:04.420 INFO:tasks.workunit.client.0.smithi032.stdout:3/314: dread d12/f21 [0,4194304] 0 2024-04-12T05:37:04.420 INFO:tasks.workunit.client.0.smithi032.stdout:6/404: fsync d4/d21/f85 0 2024-04-12T05:37:04.421 INFO:tasks.workunit.client.0.smithi032.stdout:1/299: symlink d5/de/d17/d35/d3b/d49/l5d 0 2024-04-12T05:37:04.422 INFO:tasks.workunit.client.0.smithi032.stdout:4/306: dread d4/d13/d19/d28/d32/f3e [0,4194304] 0 2024-04-12T05:37:04.423 INFO:tasks.workunit.client.0.smithi032.stdout:7/336: dread d0/de/df/d20/f68 [0,4194304] 0 2024-04-12T05:37:04.424 INFO:tasks.workunit.client.0.smithi032.stdout:6/405: fdatasync d4/f2c 0 2024-04-12T05:37:04.424 INFO:tasks.workunit.client.0.smithi032.stdout:8/352: getdents d0/db/d38 0 2024-04-12T05:37:04.424 INFO:tasks.workunit.client.0.smithi032.stdout:3/315: fsync d12/d2e/f36 0 2024-04-12T05:37:04.427 INFO:tasks.workunit.client.0.smithi032.stdout:6/406: mknod d4/c89 0 2024-04-12T05:37:04.428 INFO:tasks.workunit.client.0.smithi032.stdout:2/359: dwrite dd/d1b/d1c/d64/d32/f33 [4194304,4194304] 0 2024-04-12T05:37:04.432 INFO:tasks.workunit.client.0.smithi032.stdout:7/337: rename d0/d1c/d71/d73 to d0/dd/d74 0 2024-04-12T05:37:04.433 INFO:tasks.workunit.client.0.smithi032.stdout:1/300: getdents d5/de/d17 0 2024-04-12T05:37:04.434 INFO:tasks.workunit.client.0.smithi032.stdout:8/353: mknod d0/db/dd/d11/d1b/d22/d35/c7f 0 2024-04-12T05:37:04.435 INFO:tasks.workunit.client.0.smithi032.stdout:3/316: dread d12/d2e/d3a/f31 [0,4194304] 0 2024-04-12T05:37:04.436 INFO:tasks.workunit.client.0.smithi032.stdout:6/407: write d4/d14/d1a/f6f [1619132,78072] 0 2024-04-12T05:37:04.436 INFO:tasks.workunit.client.0.smithi032.stdout:2/360: mkdir dd/d1b/d42/d4a/d65 0 2024-04-12T05:37:04.437 INFO:tasks.workunit.client.0.smithi032.stdout:8/354: chown d0/db/d39/l4c 316296 1 2024-04-12T05:37:04.437 INFO:tasks.workunit.client.0.smithi032.stdout:7/338: mknod d0/d13/d21/d30/c75 0 2024-04-12T05:37:04.439 INFO:tasks.workunit.client.0.smithi032.stdout:4/307: rename d4/d13/l2e to d4/d13/d19/d28/d32/l65 0 2024-04-12T05:37:04.439 INFO:tasks.workunit.client.0.smithi032.stdout:6/408: truncate d4/d14/d3c/d75/d55/f70 1062736 0 2024-04-12T05:37:04.440 INFO:tasks.workunit.client.0.smithi032.stdout:1/301: creat d5/de/d17/d35/d3b/d49/f5e x:0 0 0 2024-04-12T05:37:04.440 INFO:tasks.workunit.client.0.smithi032.stdout:7/339: readlink d0/d13/l18 0 2024-04-12T05:37:04.441 INFO:tasks.workunit.client.0.smithi032.stdout:4/308: write d4/d13/d19/f39 [747670,89205] 0 2024-04-12T05:37:04.443 INFO:tasks.workunit.client.0.smithi032.stdout:6/409: write d4/d14/d3c/d75/d55/f70 [451716,56181] 0 2024-04-12T05:37:04.445 INFO:tasks.workunit.client.0.smithi032.stdout:1/302: creat d5/de/d17/d1b/f5f x:0 0 0 2024-04-12T05:37:04.445 INFO:tasks.workunit.client.0.smithi032.stdout:5/438: sync 2024-04-12T05:37:04.447 INFO:tasks.workunit.client.0.smithi032.stdout:3/317: rename d12/d24/f2b to d12/d2e/f67 0 2024-04-12T05:37:04.447 INFO:tasks.workunit.client.0.smithi032.stdout:1/303: readlink d5/de/d17/d1b/l44 0 2024-04-12T05:37:04.447 INFO:tasks.workunit.client.0.smithi032.stdout:5/439: write d2/d3c/d5b/f87 [230484,121334] 0 2024-04-12T05:37:04.448 INFO:tasks.workunit.client.0.smithi032.stdout:4/309: dread d4/d13/d19/d28/d2f/d3b/f5d [0,4194304] 0 2024-04-12T05:37:04.450 INFO:tasks.workunit.client.0.smithi032.stdout:1/304: dread - d5/d2c/d4d/f50 zero size 2024-04-12T05:37:04.450 INFO:tasks.workunit.client.0.smithi032.stdout:5/440: chown d2/df/d25/ce 897983827 1 2024-04-12T05:37:04.453 INFO:tasks.workunit.client.0.smithi032.stdout:7/340: getdents d0/de/d47 0 2024-04-12T05:37:04.453 INFO:tasks.workunit.client.0.smithi032.stdout:1/305: dread - d5/de/d17/d35/d3e/f4c zero size 2024-04-12T05:37:04.453 INFO:tasks.workunit.client.0.smithi032.stdout:5/441: dread - d2/df/d25/d1e/f59 zero size 2024-04-12T05:37:04.454 INFO:tasks.workunit.client.0.smithi032.stdout:8/355: write d0/db/d38/f61 [923844,25538] 0 2024-04-12T05:37:04.454 INFO:tasks.workunit.client.0.smithi032.stdout:1/306: chown d5/d2c/d4d 2363 1 2024-04-12T05:37:04.456 INFO:tasks.workunit.client.0.smithi032.stdout:6/410: dwrite d4/d14/d1a/f49 [0,4194304] 0 2024-04-12T05:37:04.457 INFO:tasks.workunit.client.0.smithi032.stdout:2/361: rename dd/d19/c50 to dd/d1b/d1c/d64/c66 0 2024-04-12T05:37:04.458 INFO:tasks.workunit.client.0.smithi032.stdout:8/356: write d0/db/f4e [952441,32387] 0 2024-04-12T05:37:04.463 INFO:tasks.workunit.client.0.smithi032.stdout:6/411: write d4/d21/d2e/f39 [2190892,109523] 0 2024-04-12T05:37:04.463 INFO:tasks.workunit.client.0.smithi032.stdout:3/318: getdents d12/d2e/d3a/d2d 0 2024-04-12T05:37:04.465 INFO:tasks.workunit.client.0.smithi032.stdout:4/310: rename d4/d13/d19/d28/d2f/d3b/l58 to d4/l66 0 2024-04-12T05:37:04.469 INFO:tasks.workunit.client.0.smithi032.stdout:8/357: chown d0/db/dd/c5d 105164 1 2024-04-12T05:37:04.469 INFO:tasks.workunit.client.0.smithi032.stdout:6/412: stat d4/d14/d3c/d75/d55/l6b 0 2024-04-12T05:37:04.470 INFO:tasks.workunit.client.0.smithi032.stdout:4/311: chown d4/d13/f1f 3690 1 2024-04-12T05:37:04.472 INFO:tasks.workunit.client.0.smithi032.stdout:3/319: write d12/d2e/d3a/d2d/f45 [914712,15308] 0 2024-04-12T05:37:04.472 INFO:tasks.workunit.client.0.smithi032.stdout:5/442: write d2/df/d31/f46 [3654884,45968] 0 2024-04-12T05:37:04.472 INFO:tasks.workunit.client.0.smithi032.stdout:8/358: fdatasync d0/db/f20 0 2024-04-12T05:37:04.476 INFO:tasks.workunit.client.0.smithi032.stdout:6/413: mkdir d4/d14/d1a/d37/d69/d8a 0 2024-04-12T05:37:04.477 INFO:tasks.workunit.client.0.smithi032.stdout:1/307: dwrite d5/d2c/f2d [4194304,4194304] 0 2024-04-12T05:37:04.478 INFO:tasks.workunit.client.0.smithi032.stdout:5/443: chown d2/df/c62 1112 1 2024-04-12T05:37:04.480 INFO:tasks.workunit.client.0.smithi032.stdout:8/359: unlink d0/db/f20 0 2024-04-12T05:37:04.481 INFO:tasks.workunit.client.0.smithi032.stdout:3/320: mknod d12/d2e/c68 0 2024-04-12T05:37:04.481 INFO:tasks.workunit.client.0.smithi032.stdout:2/362: dwrite dd/d1b/f34 [0,4194304] 0 2024-04-12T05:37:04.483 INFO:tasks.workunit.client.0.smithi032.stdout:1/308: symlink d5/de/l60 0 2024-04-12T05:37:04.485 INFO:tasks.workunit.client.0.smithi032.stdout:7/341: dwrite d0/de/f12 [0,4194304] 0 2024-04-12T05:37:04.486 INFO:tasks.workunit.client.0.smithi032.stdout:2/363: chown f1 19963 1 2024-04-12T05:37:04.486 INFO:tasks.workunit.client.0.smithi032.stdout:1/309: chown d5/de/c36 991922960 1 2024-04-12T05:37:04.486 INFO:tasks.workunit.client.0.smithi032.stdout:3/321: mknod d12/d2e/d3a/d2d/c69 0 2024-04-12T05:37:04.487 INFO:tasks.workunit.client.0.smithi032.stdout:7/342: dread - d0/d10/f63 zero size 2024-04-12T05:37:04.492 INFO:tasks.workunit.client.0.smithi032.stdout:4/312: link d4/d13/d19/d28/d2f/d46/l61 d4/d13/d19/d28/d4d/l67 0 2024-04-12T05:37:04.492 INFO:tasks.workunit.client.0.smithi032.stdout:5/444: getdents d2/df/d25/d43 0 2024-04-12T05:37:04.493 INFO:tasks.workunit.client.0.smithi032.stdout:6/414: dread d4/d14/d3c/d75/d55/f72 [0,4194304] 0 2024-04-12T05:37:04.495 INFO:tasks.workunit.client.0.smithi032.stdout:6/415: fdatasync d4/d21/d2e/f39 0 2024-04-12T05:37:04.496 INFO:tasks.workunit.client.0.smithi032.stdout:9/322: sync 2024-04-12T05:37:04.498 INFO:tasks.workunit.client.0.smithi032.stdout:1/310: rename d5/d2c/d4d to d5/de/d17/d35/d61 0 2024-04-12T05:37:04.504 INFO:tasks.workunit.client.0.smithi032.stdout:9/323: creat d9/d15/f6a x:0 0 0 2024-04-12T05:37:04.505 INFO:tasks.workunit.client.0.smithi032.stdout:3/322: link d12/d2e/d3a/d2d/f45 d12/d2e/f6a 0 2024-04-12T05:37:04.505 INFO:tasks.workunit.client.0.smithi032.stdout:4/313: mkdir d4/d13/d19/d28/d68 0 2024-04-12T05:37:04.507 INFO:tasks.workunit.client.0.smithi032.stdout:1/311: creat d5/de/d17/d35/d3b/d49/f62 x:0 0 0 2024-04-12T05:37:04.510 INFO:tasks.workunit.client.0.smithi032.stdout:2/364: dwrite dd/d1b/d1c/d64/d32/f60 [0,4194304] 0 2024-04-12T05:37:04.513 INFO:tasks.workunit.client.0.smithi032.stdout:1/312: fdatasync d5/de/d17/f20 0 2024-04-12T05:37:04.513 INFO:tasks.workunit.client.0.smithi032.stdout:8/360: truncate d0/db/dd/d11/d1b/d22/d51/f69 2179441 0 2024-04-12T05:37:04.514 INFO:tasks.workunit.client.0.smithi032.stdout:5/445: dwrite d2/d3c/d48/d4b/f7c [0,4194304] 0 2024-04-12T05:37:04.514 INFO:tasks.workunit.client.0.smithi032.stdout:7/343: dread d0/f8 [0,4194304] 0 2024-04-12T05:37:04.516 INFO:tasks.workunit.client.0.smithi032.stdout:4/314: mknod d4/d13/d19/d28/d4d/c69 0 2024-04-12T05:37:04.519 INFO:tasks.workunit.client.0.smithi032.stdout:5/446: read d2/df/f54 [623352,5127] 0 2024-04-12T05:37:04.522 INFO:tasks.workunit.client.0.smithi032.stdout:8/361: symlink d0/db/d38/l80 0 2024-04-12T05:37:04.522 INFO:tasks.workunit.client.0.smithi032.stdout:2/365: mknod dd/d1b/d1c/d64/d32/d3a/c67 0 2024-04-12T05:37:04.524 INFO:tasks.workunit.client.0.smithi032.stdout:7/344: write d0/f6f [2260079,52546] 0 2024-04-12T05:37:04.526 INFO:tasks.workunit.client.0.smithi032.stdout:3/323: dwrite f10 [0,4194304] 0 2024-04-12T05:37:04.527 INFO:tasks.workunit.client.0.smithi032.stdout:5/447: truncate d2/f7b 942107 0 2024-04-12T05:37:04.529 INFO:tasks.workunit.client.0.smithi032.stdout:9/324: dwrite d9/d15/d2c/f23 [4194304,4194304] 0 2024-04-12T05:37:04.535 INFO:tasks.workunit.client.0.smithi032.stdout:1/313: dwrite d5/de/d17/d1b/f21 [0,4194304] 0 2024-04-12T05:37:04.537 INFO:tasks.workunit.client.0.smithi032.stdout:8/362: chown d0/db/dd/d11/d1b/d22/d35/d44/c4b 1 1 2024-04-12T05:37:04.541 INFO:tasks.workunit.client.0.smithi032.stdout:5/448: write d2/df/d25/d1e/f59 [987453,11355] 0 2024-04-12T05:37:04.543 INFO:tasks.workunit.client.0.smithi032.stdout:4/315: truncate d4/d13/d19/d28/d2f/d3b/f3c 543886 0 2024-04-12T05:37:04.547 INFO:tasks.workunit.client.0.smithi032.stdout:6/416: dread d4/d14/d3c/f41 [0,4194304] 0 2024-04-12T05:37:04.549 INFO:tasks.workunit.client.0.smithi032.stdout:2/366: dread dd/ff [0,4194304] 0 2024-04-12T05:37:04.551 INFO:tasks.workunit.client.0.smithi032.stdout:4/316: creat d4/d13/d44/f6a x:0 0 0 2024-04-12T05:37:04.552 INFO:tasks.workunit.client.0.smithi032.stdout:0/442: sync 2024-04-12T05:37:04.553 INFO:tasks.workunit.client.0.smithi032.stdout:7/345: getdents d0/d1c/d4c/d26 0 2024-04-12T05:37:04.554 INFO:tasks.workunit.client.0.smithi032.stdout:1/314: creat d5/de/d17/d35/d3b/f63 x:0 0 0 2024-04-12T05:37:04.555 INFO:tasks.workunit.client.0.smithi032.stdout:9/325: dwrite d9/d1f/f5d [0,4194304] 0 2024-04-12T05:37:04.559 INFO:tasks.workunit.client.0.smithi032.stdout:3/324: dread d12/f1b [0,4194304] 0 2024-04-12T05:37:04.567 INFO:tasks.workunit.client.0.smithi032.stdout:8/363: dread d0/db/dd/f27 [0,4194304] 0 2024-04-12T05:37:04.567 INFO:tasks.workunit.client.0.smithi032.stdout:2/367: dwrite f0 [4194304,4194304] 0 2024-04-12T05:37:04.567 INFO:tasks.workunit.client.0.smithi032.stdout:9/326: mkdir d9/d15/d1a/d1d/d5a/d6b 0 2024-04-12T05:37:04.569 INFO:tasks.workunit.client.0.smithi032.stdout:3/325: mknod d12/c6b 0 2024-04-12T05:37:04.570 INFO:tasks.workunit.client.0.smithi032.stdout:4/317: dread d4/d13/d19/d28/d32/f3e [0,4194304] 0 2024-04-12T05:37:04.570 INFO:tasks.workunit.client.0.smithi032.stdout:1/315: rename d5/de/d17/d35/d61/d54/f5a to d5/de/d17/d35/d61/d54/f64 0 2024-04-12T05:37:04.571 INFO:tasks.workunit.client.0.smithi032.stdout:7/346: rename d0/d1c/d4c/d26 to d0/d76 0 2024-04-12T05:37:04.572 INFO:tasks.workunit.client.0.smithi032.stdout:5/449: write d2/f4 [1742440,12560] 0 2024-04-12T05:37:04.580 INFO:tasks.workunit.client.0.smithi032.stdout:3/326: chown d12/d2f/c5d 433 1 2024-04-12T05:37:04.580 INFO:tasks.workunit.client.0.smithi032.stdout:8/364: rmdir d0/db/dd/d11/d1b/d22/d35/d44 39 2024-04-12T05:37:04.580 INFO:tasks.workunit.client.0.smithi032.stdout:1/316: dread - d5/de/d17/d35/d3e/f4c zero size 2024-04-12T05:37:04.581 INFO:tasks.workunit.client.0.smithi032.stdout:2/368: rename dd/d1b/d1c/d64/f3e to dd/f68 0 2024-04-12T05:37:04.583 INFO:tasks.workunit.client.0.smithi032.stdout:3/327: dread - d12/d2f/f41 zero size 2024-04-12T05:37:04.584 INFO:tasks.workunit.client.0.smithi032.stdout:5/450: write d2/df/d25/d43/d49/f7d [389327,33401] 0 2024-04-12T05:37:04.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:04 smithi032 ceph-mon[39731]: pgmap v263: 65 pgs: 65 active+clean; 627 MiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 3.6 MiB/s rd, 51 MiB/s wr, 247 op/s 2024-04-12T05:37:04.591 INFO:tasks.workunit.client.0.smithi032.stdout:2/369: unlink dd/d1b/d1c/d64/d32/d3a/c67 0 2024-04-12T05:37:04.591 INFO:tasks.workunit.client.0.smithi032.stdout:5/451: mknod d2/df/c8b 0 2024-04-12T05:37:04.592 INFO:tasks.workunit.client.0.smithi032.stdout:3/328: symlink d12/d2f/d5b/l6c 0 2024-04-12T05:37:04.595 INFO:tasks.workunit.client.0.smithi032.stdout:6/417: dwrite f3 [0,4194304] 0 2024-04-12T05:37:04.596 INFO:tasks.workunit.client.0.smithi032.stdout:7/347: dwrite d0/d10/f60 [0,4194304] 0 2024-04-12T05:37:04.597 INFO:tasks.workunit.client.0.smithi032.stdout:2/370: chown dd/d1b/d1c/d64 658222 1 2024-04-12T05:37:04.598 INFO:tasks.workunit.client.0.smithi032.stdout:1/317: read d5/de/f4a [38080,114918] 0 2024-04-12T05:37:04.600 INFO:tasks.workunit.client.0.smithi032.stdout:8/365: dread d0/db/dd/f27 [0,4194304] 0 2024-04-12T05:37:04.602 INFO:tasks.workunit.client.0.smithi032.stdout:1/318: fdatasync d5/de/f11 0 2024-04-12T05:37:04.603 INFO:tasks.workunit.client.0.smithi032.stdout:2/371: dread - dd/f2e zero size 2024-04-12T05:37:04.603 INFO:tasks.workunit.client.0.smithi032.stdout:0/443: dread d2/d8/d17/f2b [0,4194304] 0 2024-04-12T05:37:04.605 INFO:tasks.workunit.client.0.smithi032.stdout:4/318: dwrite d4/d13/d19/f30 [0,4194304] 0 2024-04-12T05:37:04.608 INFO:tasks.workunit.client.0.smithi032.stdout:6/418: chown d4/d14/d1a/d37/d47/d57 10121 1 2024-04-12T05:37:04.610 INFO:tasks.workunit.client.0.smithi032.stdout:2/372: fdatasync dd/d1b/d1c/d64/d32/f33 0 2024-04-12T05:37:04.611 INFO:tasks.workunit.client.0.smithi032.stdout:3/329: rename d12/d2e/c44 to d12/d2e/c6d 0 2024-04-12T05:37:04.612 INFO:tasks.workunit.client.0.smithi032.stdout:1/319: write d5/de/d17/d35/d61/d54/f64 [523904,110728] 0 2024-04-12T05:37:04.613 INFO:tasks.workunit.client.0.smithi032.stdout:6/419: symlink d4/d14/l8b 0 2024-04-12T05:37:04.614 INFO:tasks.workunit.client.0.smithi032.stdout:5/452: dwrite d2/d3c/d5b/f39 [0,4194304] 0 2024-04-12T05:37:04.615 INFO:tasks.workunit.client.0.smithi032.stdout:3/330: dread - d12/f59 zero size 2024-04-12T05:37:04.616 INFO:tasks.workunit.client.0.smithi032.stdout:4/319: creat d4/d13/d19/d28/d2f/d3b/f6b x:0 0 0 2024-04-12T05:37:04.619 INFO:tasks.workunit.client.0.smithi032.stdout:1/320: write d5/d2c/f3c [1517404,90690] 0 2024-04-12T05:37:04.619 INFO:tasks.workunit.client.0.smithi032.stdout:0/444: truncate d2/d8/df/f52 1004892 0 2024-04-12T05:37:04.621 INFO:tasks.workunit.client.0.smithi032.stdout:2/373: mknod dd/d1b/d1c/d64/d32/d3a/c69 0 2024-04-12T05:37:04.622 INFO:tasks.workunit.client.0.smithi032.stdout:5/453: fsync d2/d3c/d48/d4b/f7c 0 2024-04-12T05:37:04.623 INFO:tasks.workunit.client.0.smithi032.stdout:3/331: dread - d12/d2e/f5a zero size 2024-04-12T05:37:04.624 INFO:tasks.workunit.client.0.smithi032.stdout:6/420: rename d4/c89 to d4/d14/d1a/d88/c8c 0 2024-04-12T05:37:04.625 INFO:tasks.workunit.client.0.smithi032.stdout:8/366: dwrite d0/db/dd/f27 [0,4194304] 0 2024-04-12T05:37:04.626 INFO:tasks.workunit.client.0.smithi032.stdout:2/374: write dd/d19/d20/f53 [5227048,81072] 0 2024-04-12T05:37:04.628 INFO:tasks.workunit.client.0.smithi032.stdout:5/454: write d2/df/d25/d43/f81 [58054,42031] 0 2024-04-12T05:37:04.633 INFO:tasks.workunit.client.0.smithi032.stdout:4/320: rename d4/d13/d19/l22 to d4/d13/d19/d28/d2f/d3b/l6c 0 2024-04-12T05:37:04.639 INFO:tasks.workunit.client.0.smithi032.stdout:4/321: readlink d4/d13/d19/d28/d2f/d46/l4a 0 2024-04-12T05:37:04.640 INFO:tasks.workunit.client.0.smithi032.stdout:1/321: rename d5/de/d17/d35/f58 to d5/de/d2a/f65 0 2024-04-12T05:37:04.641 INFO:tasks.workunit.client.0.smithi032.stdout:2/375: dwrite dd/f2e [0,4194304] 0 2024-04-12T05:37:04.642 INFO:tasks.workunit.client.0.smithi032.stdout:3/332: creat d12/d2e/f6e x:0 0 0 2024-04-12T05:37:04.642 INFO:tasks.workunit.client.0.smithi032.stdout:1/322: readlink d5/de/l18 0 2024-04-12T05:37:04.643 INFO:tasks.workunit.client.0.smithi032.stdout:0/445: write d2/d29/f5d [721156,74256] 0 2024-04-12T05:37:04.646 INFO:tasks.workunit.client.0.smithi032.stdout:5/455: dwrite d2/df/d25/d43/f7a [0,4194304] 0 2024-04-12T05:37:04.646 INFO:tasks.workunit.client.0.smithi032.stdout:4/322: chown d4/d13/d19/d28/d2f/d3b/f6b 1279592 1 2024-04-12T05:37:04.649 INFO:tasks.workunit.client.0.smithi032.stdout:2/376: creat dd/d1b/d1c/d64/d32/d3a/f6a x:0 0 0 2024-04-12T05:37:04.656 INFO:tasks.workunit.client.0.smithi032.stdout:0/446: write d2/d8/d17/d1b/d20/f6d [758408,105459] 0 2024-04-12T05:37:04.657 INFO:tasks.workunit.client.0.smithi032.stdout:5/456: mknod d2/df/d25/d43/d49/c8c 0 2024-04-12T05:37:04.663 INFO:tasks.workunit.client.0.smithi032.stdout:8/367: dwrite d0/db/dd/d11/d1b/d22/f2b [0,4194304] 0 2024-04-12T05:37:04.670 INFO:tasks.workunit.client.0.smithi032.stdout:6/421: dwrite d4/f6 [0,4194304] 0 2024-04-12T05:37:04.674 INFO:tasks.workunit.client.0.smithi032.stdout:1/323: dwrite d5/de/d17/d1b/f1d [0,4194304] 0 2024-04-12T05:37:04.683 INFO:tasks.workunit.client.0.smithi032.stdout:3/333: dwrite d12/d2e/d3a/f26 [0,4194304] 0 2024-04-12T05:37:04.685 INFO:tasks.workunit.client.0.smithi032.stdout:4/323: dwrite d4/f35 [0,4194304] 0 2024-04-12T05:37:04.686 INFO:tasks.workunit.client.0.smithi032.stdout:2/377: dwrite dd/d1b/d1c/d64/d32/d3a/f6a [0,4194304] 0 2024-04-12T05:37:04.688 INFO:tasks.workunit.client.0.smithi032.stdout:0/447: readlink d2/d8/l4f 0 2024-04-12T05:37:04.688 INFO:tasks.workunit.client.0.smithi032.stdout:5/457: symlink d2/d65/d78/d86/l8d 0 2024-04-12T05:37:04.689 INFO:tasks.workunit.client.0.smithi032.stdout:3/334: truncate d12/d2e/d3a/f26 4405899 0 2024-04-12T05:37:04.690 INFO:tasks.workunit.client.0.smithi032.stdout:5/458: chown d2/d3c/d48 7867 1 2024-04-12T05:37:04.691 INFO:tasks.workunit.client.0.smithi032.stdout:0/448: read d2/d8/d17/f2b [3748069,82881] 0 2024-04-12T05:37:04.692 INFO:tasks.workunit.client.0.smithi032.stdout:2/378: truncate dd/d1b/d1c/f51 948526 0 2024-04-12T05:37:04.693 INFO:tasks.workunit.client.0.smithi032.stdout:6/422: mkdir d4/d14/d1a/d88/d8d 0 2024-04-12T05:37:04.694 INFO:tasks.workunit.client.0.smithi032.stdout:1/324: creat d5/de/f66 x:0 0 0 2024-04-12T05:37:04.694 INFO:tasks.workunit.client.0.smithi032.stdout:4/324: truncate d4/d13/d19/d28/d2f/f60 616343 0 2024-04-12T05:37:04.695 INFO:tasks.workunit.client.0.smithi032.stdout:3/335: chown d12/d2f/f48 12391760 1 2024-04-12T05:37:04.696 INFO:tasks.workunit.client.0.smithi032.stdout:4/325: stat d4/f1d 0 2024-04-12T05:37:04.696 INFO:tasks.workunit.client.0.smithi032.stdout:5/459: write d2/d3c/d48/d4b/f6b [582771,125028] 0 2024-04-12T05:37:04.699 INFO:tasks.workunit.client.0.smithi032.stdout:0/449: symlink d2/d28/d87/l93 0 2024-04-12T05:37:04.701 INFO:tasks.workunit.client.0.smithi032.stdout:1/325: chown d5/de/d17/d1b/f5f 3 1 2024-04-12T05:37:04.703 INFO:tasks.workunit.client.0.smithi032.stdout:1/326: chown d5/de/f11 84714 1 2024-04-12T05:37:04.706 INFO:tasks.workunit.client.0.smithi032.stdout:5/460: fsync d2/dd/f44 0 2024-04-12T05:37:04.707 INFO:tasks.workunit.client.0.smithi032.stdout:2/379: dread dd/d1b/f58 [0,4194304] 0 2024-04-12T05:37:04.708 INFO:tasks.workunit.client.0.smithi032.stdout:2/380: dread - dd/d19/f5f zero size 2024-04-12T05:37:04.713 INFO:tasks.workunit.client.0.smithi032.stdout:3/336: mknod d12/d2e/c6f 0 2024-04-12T05:37:04.715 INFO:tasks.workunit.client.0.smithi032.stdout:5/461: creat d2/df/d25/d50/f8e x:0 0 0 2024-04-12T05:37:04.716 INFO:tasks.workunit.client.0.smithi032.stdout:2/381: creat dd/d1b/d42/d4a/f6b x:0 0 0 2024-04-12T05:37:04.717 INFO:tasks.workunit.client.0.smithi032.stdout:1/327: mkdir d5/de/d17/d35/d3b/d67 0 2024-04-12T05:37:04.720 INFO:tasks.workunit.client.0.smithi032.stdout:2/382: read dd/d1b/d1c/f43 [370414,55184] 0 2024-04-12T05:37:04.722 INFO:tasks.workunit.client.0.smithi032.stdout:3/337: dwrite d12/d2e/f5a [0,4194304] 0 2024-04-12T05:37:04.723 INFO:tasks.workunit.client.0.smithi032.stdout:3/338: fdatasync d12/d2f/f3d 0 2024-04-12T05:37:04.726 INFO:tasks.workunit.client.0.smithi032.stdout:2/383: creat dd/d1b/f6c x:0 0 0 2024-04-12T05:37:04.731 INFO:tasks.workunit.client.0.smithi032.stdout:5/462: dwrite d2/dd/f83 [0,4194304] 0 2024-04-12T05:37:04.733 INFO:tasks.workunit.client.0.smithi032.stdout:4/326: rename d4/f12 to d4/d13/d19/d28/d2f/d46/f6d 0 2024-04-12T05:37:04.733 INFO:tasks.workunit.client.0.smithi032.stdout:1/328: dwrite d5/de/d17/d35/d3b/d49/f56 [0,4194304] 0 2024-04-12T05:37:04.735 INFO:tasks.workunit.client.0.smithi032.stdout:2/384: mkdir dd/d1b/d1c/d64/d32/d6d 0 2024-04-12T05:37:04.737 INFO:tasks.workunit.client.0.smithi032.stdout:4/327: mkdir d4/d13/d19/d28/d2f/d6e 0 2024-04-12T05:37:04.738 INFO:tasks.workunit.client.0.smithi032.stdout:1/329: read d5/de/d17/f20 [7334225,108002] 0 2024-04-12T05:37:04.738 INFO:tasks.workunit.client.0.smithi032.stdout:3/339: creat d12/f70 x:0 0 0 2024-04-12T05:37:04.739 INFO:tasks.workunit.client.0.smithi032.stdout:1/330: stat d5/de/f11 0 2024-04-12T05:37:04.745 INFO:tasks.workunit.client.0.smithi032.stdout:2/385: getdents dd/d1b/d42 0 2024-04-12T05:37:04.748 INFO:tasks.workunit.client.0.smithi032.stdout:6/423: truncate d4/f2a 3098670 0 2024-04-12T05:37:04.749 INFO:tasks.workunit.client.0.smithi032.stdout:3/340: link d12/l27 d12/d24/l71 0 2024-04-12T05:37:04.749 INFO:tasks.workunit.client.0.smithi032.stdout:4/328: dwrite d4/d13/fd [0,4194304] 0 2024-04-12T05:37:04.750 INFO:tasks.workunit.client.0.smithi032.stdout:3/341: stat d12/l52 0 2024-04-12T05:37:04.752 INFO:tasks.workunit.client.0.smithi032.stdout:3/342: dread - d12/d2f/f41 zero size 2024-04-12T05:37:04.753 INFO:tasks.workunit.client.0.smithi032.stdout:6/424: creat d4/d21/d27/f8e x:0 0 0 2024-04-12T05:37:04.754 INFO:tasks.workunit.client.0.smithi032.stdout:0/450: dwrite d2/d8/df/f31 [0,4194304] 0 2024-04-12T05:37:04.755 INFO:tasks.workunit.client.0.smithi032.stdout:0/451: fdatasync d2/d29/f65 0 2024-04-12T05:37:04.757 INFO:tasks.workunit.client.0.smithi032.stdout:2/386: link dd/d1b/c5d dd/d1b/d1c/d64/c6e 0 2024-04-12T05:37:04.757 INFO:tasks.workunit.client.0.smithi032.stdout:0/452: write d2/d8/d17/f22 [4422145,96515] 0 2024-04-12T05:37:04.758 INFO:tasks.workunit.client.0.smithi032.stdout:3/343: mkdir d12/d2e/d3a/d72 0 2024-04-12T05:37:04.762 INFO:tasks.workunit.client.0.smithi032.stdout:4/329: mknod d4/d13/d19/c6f 0 2024-04-12T05:37:04.762 INFO:tasks.workunit.client.0.smithi032.stdout:0/453: truncate d2/d28/d87/f71 810897 0 2024-04-12T05:37:04.763 INFO:tasks.workunit.client.0.smithi032.stdout:0/454: stat d2/d29/f65 0 2024-04-12T05:37:04.766 INFO:tasks.workunit.client.0.smithi032.stdout:4/330: truncate d4/d13/d19/d28/d4d/f63 570623 0 2024-04-12T05:37:04.766 INFO:tasks.workunit.client.0.smithi032.stdout:2/387: symlink dd/d1b/l6f 0 2024-04-12T05:37:04.766 INFO:tasks.workunit.client.0.smithi032.stdout:4/331: fsync d4/d13/d19/d28/d2f/d3b/f6b 0 2024-04-12T05:37:04.766 INFO:tasks.workunit.client.0.smithi032.stdout:0/455: stat d2/d8/d48 0 2024-04-12T05:37:04.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:04 smithi067 ceph-mon[49010]: pgmap v263: 65 pgs: 65 active+clean; 627 MiB data, 3.5 GiB used, 533 GiB / 536 GiB avail; 3.6 MiB/s rd, 51 MiB/s wr, 247 op/s 2024-04-12T05:37:04.770 INFO:tasks.workunit.client.0.smithi032.stdout:4/332: creat d4/d13/d19/d28/d2f/f70 x:0 0 0 2024-04-12T05:37:04.770 INFO:tasks.workunit.client.0.smithi032.stdout:4/333: chown d4/d13/d19/d28/d32/c56 60000063 1 2024-04-12T05:37:04.771 INFO:tasks.workunit.client.0.smithi032.stdout:2/388: creat dd/d1b/d1c/d64/f70 x:0 0 0 2024-04-12T05:37:04.772 INFO:tasks.workunit.client.0.smithi032.stdout:6/425: rmdir d4 39 2024-04-12T05:37:04.777 INFO:tasks.workunit.client.0.smithi032.stdout:4/334: creat d4/d13/f71 x:0 0 0 2024-04-12T05:37:04.778 INFO:tasks.workunit.client.0.smithi032.stdout:4/335: write d4/d13/d19/d28/d2f/f70 [608067,84851] 0 2024-04-12T05:37:04.782 INFO:tasks.workunit.client.0.smithi032.stdout:0/456: truncate d2/d8/d17/f6b 2419855 0 2024-04-12T05:37:04.784 INFO:tasks.workunit.client.0.smithi032.stdout:3/344: dwrite d12/f13 [0,4194304] 0 2024-04-12T05:37:04.787 INFO:tasks.workunit.client.0.smithi032.stdout:4/336: link d4/d13/d19/c24 d4/d13/d19/d28/d2f/d6e/c72 0 2024-04-12T05:37:04.796 INFO:tasks.workunit.client.0.smithi032.stdout:4/337: rmdir d4/d13/d19/d28/d4d 39 2024-04-12T05:37:04.798 INFO:tasks.workunit.client.0.smithi032.stdout:4/338: fdatasync d4/d13/d19/d28/d32/f5f 0 2024-04-12T05:37:04.798 INFO:tasks.workunit.client.0.smithi032.stdout:3/345: getdents d12 0 2024-04-12T05:37:04.799 INFO:tasks.workunit.client.0.smithi032.stdout:2/389: dread dd/f1e [0,4194304] 0 2024-04-12T05:37:04.799 INFO:tasks.workunit.client.0.smithi032.stdout:3/346: chown d12/d2f/l43 38971 1 2024-04-12T05:37:04.800 INFO:tasks.workunit.client.0.smithi032.stdout:3/347: dread - d12/d2f/d5b/f66 zero size 2024-04-12T05:37:04.800 INFO:tasks.workunit.client.0.smithi032.stdout:3/348: chown d12/f59 6884 1 2024-04-12T05:37:04.801 INFO:tasks.workunit.client.0.smithi032.stdout:2/390: unlink dd/d1b/d1c/d64/l59 0 2024-04-12T05:37:04.802 INFO:tasks.workunit.client.0.smithi032.stdout:2/391: mknod dd/d1b/d1c/d64/d36/c71 0 2024-04-12T05:37:04.806 INFO:tasks.workunit.client.0.smithi032.stdout:3/349: dwrite d12/f2a [0,4194304] 0 2024-04-12T05:37:04.807 INFO:tasks.workunit.client.0.smithi032.stdout:2/392: symlink dd/d1b/d1c/d64/d32/d6d/l72 0 2024-04-12T05:37:04.808 INFO:tasks.workunit.client.0.smithi032.stdout:2/393: chown dd/d1b/d42/d4a/f6b 5783925 1 2024-04-12T05:37:04.808 INFO:tasks.workunit.client.0.smithi032.stdout:3/350: chown d12/l29 1735106 1 2024-04-12T05:37:04.808 INFO:tasks.workunit.client.0.smithi032.stdout:2/394: write dd/d1b/f34 [1599292,61983] 0 2024-04-12T05:37:04.809 INFO:tasks.workunit.client.0.smithi032.stdout:2/395: stat dd/d19/f5f 0 2024-04-12T05:37:04.810 INFO:tasks.workunit.client.0.smithi032.stdout:3/351: mknod d12/d2e/d3a/d2d/c73 0 2024-04-12T05:37:04.812 INFO:tasks.workunit.client.0.smithi032.stdout:2/396: creat dd/d1b/f73 x:0 0 0 2024-04-12T05:37:04.813 INFO:tasks.workunit.client.0.smithi032.stdout:2/397: chown dd/d1b/d42/d4a/d65 0 1 2024-04-12T05:37:04.816 INFO:tasks.workunit.client.0.smithi032.stdout:3/352: dwrite d12/d24/f4d [0,4194304] 0 2024-04-12T05:37:04.818 INFO:tasks.workunit.client.0.smithi032.stdout:2/398: rename dd/ff to dd/d1b/d1c/f74 0 2024-04-12T05:37:04.819 INFO:tasks.workunit.client.0.smithi032.stdout:2/399: chown dd/d19/d20/f48 3358 1 2024-04-12T05:37:04.821 INFO:tasks.workunit.client.0.smithi032.stdout:2/400: chown dd/d1b/d42/c5e 55108668 1 2024-04-12T05:37:04.821 INFO:tasks.workunit.client.0.smithi032.stdout:3/353: dread d12/f21 [0,4194304] 0 2024-04-12T05:37:04.822 INFO:tasks.workunit.client.0.smithi032.stdout:2/401: write dd/d19/f5f [142986,84331] 0 2024-04-12T05:37:04.825 INFO:tasks.workunit.client.0.smithi032.stdout:3/354: getdents d12/d2e/d3a/d63 0 2024-04-12T05:37:04.825 INFO:tasks.workunit.client.0.smithi032.stdout:3/355: fdatasync f10 0 2024-04-12T05:37:04.826 INFO:tasks.workunit.client.0.smithi032.stdout:3/356: write d12/d24/f47 [430130,35254] 0 2024-04-12T05:37:04.827 INFO:tasks.workunit.client.0.smithi032.stdout:3/357: chown c9 47811835 1 2024-04-12T05:37:04.828 INFO:tasks.workunit.client.0.smithi032.stdout:2/402: link c7 dd/d1b/d1c/d64/c75 0 2024-04-12T05:37:04.829 INFO:tasks.workunit.client.0.smithi032.stdout:3/358: creat d12/d2e/f74 x:0 0 0 2024-04-12T05:37:04.832 INFO:tasks.workunit.client.0.smithi032.stdout:2/403: link dd/d1b/d1c/d64/f70 dd/d1b/d1c/f76 0 2024-04-12T05:37:04.833 INFO:tasks.workunit.client.0.smithi032.stdout:3/359: getdents d12/d2e/d3a/d72 0 2024-04-12T05:37:04.835 INFO:tasks.workunit.client.0.smithi032.stdout:2/404: creat dd/d1b/d1c/d64/d32/d6d/f77 x:0 0 0 2024-04-12T05:37:04.836 INFO:tasks.workunit.client.0.smithi032.stdout:3/360: rmdir d12/d2f 39 2024-04-12T05:37:04.836 INFO:tasks.workunit.client.0.smithi032.stdout:3/361: chown d12/d24/f4d 10672545 1 2024-04-12T05:37:04.837 INFO:tasks.workunit.client.0.smithi032.stdout:3/362: dread - d12/f59 zero size 2024-04-12T05:37:04.837 INFO:tasks.workunit.client.0.smithi032.stdout:3/363: stat d12/d2e/f36 0 2024-04-12T05:37:04.839 INFO:tasks.workunit.client.0.smithi032.stdout:3/364: dread - d12/d2f/d5b/f66 zero size 2024-04-12T05:37:04.842 INFO:tasks.workunit.client.0.smithi032.stdout:3/365: unlink d12/l27 0 2024-04-12T05:37:04.843 INFO:tasks.workunit.client.0.smithi032.stdout:3/366: chown d12/f70 1634 1 2024-04-12T05:37:04.843 INFO:tasks.workunit.client.0.smithi032.stdout:2/405: dread fa [0,4194304] 0 2024-04-12T05:37:04.887 INFO:tasks.workunit.client.0.smithi032.stdout:3/367: mknod d12/d24/c75 0 2024-04-12T05:37:04.887 INFO:tasks.workunit.client.0.smithi032.stdout:7/348: sync 2024-04-12T05:37:04.888 INFO:tasks.workunit.client.0.smithi032.stdout:9/327: sync 2024-04-12T05:37:04.888 INFO:tasks.workunit.client.0.smithi032.stdout:8/368: sync 2024-04-12T05:37:04.889 INFO:tasks.workunit.client.0.smithi032.stdout:2/406: dread dd/f27 [0,4194304] 0 2024-04-12T05:37:04.890 INFO:tasks.workunit.client.0.smithi032.stdout:3/368: truncate d12/d2e/f1e 1116332 0 2024-04-12T05:37:04.891 INFO:tasks.workunit.client.0.smithi032.stdout:7/349: rename d0/d13/d21/c35 to d0/d1c/d71/c77 0 2024-04-12T05:37:04.894 INFO:tasks.workunit.client.0.smithi032.stdout:2/407: write dd/d1b/d1c/d64/d36/f37 [1362851,39191] 0 2024-04-12T05:37:04.897 INFO:tasks.workunit.client.0.smithi032.stdout:9/328: dwrite d9/d15/f28 [0,4194304] 0 2024-04-12T05:37:04.901 INFO:tasks.workunit.client.0.smithi032.stdout:2/408: write dd/d1b/d1c/d64/d32/d6d/f77 [580411,19968] 0 2024-04-12T05:37:04.903 INFO:tasks.workunit.client.0.smithi032.stdout:3/369: dwrite d12/d24/f4c [0,4194304] 0 2024-04-12T05:37:04.907 INFO:tasks.workunit.client.0.smithi032.stdout:7/350: dwrite d0/d13/d21/d30/d31/f58 [0,4194304] 0 2024-04-12T05:37:04.907 INFO:tasks.workunit.client.0.smithi032.stdout:2/409: write dd/d1b/d1c/d64/d32/f47 [446382,25896] 0 2024-04-12T05:37:04.909 INFO:tasks.workunit.client.0.smithi032.stdout:2/410: chown dd/d1b/d1c/d64/d32/d6d/l72 4343 1 2024-04-12T05:37:04.910 INFO:tasks.workunit.client.0.smithi032.stdout:3/370: symlink d12/d2e/d3a/d63/l76 0 2024-04-12T05:37:04.912 INFO:tasks.workunit.client.0.smithi032.stdout:3/371: fsync d12/d24/f4d 0 2024-04-12T05:37:04.914 INFO:tasks.workunit.client.0.smithi032.stdout:3/372: dread - d12/d2e/f74 zero size 2024-04-12T05:37:04.916 INFO:tasks.workunit.client.0.smithi032.stdout:2/411: rename dd/d1b/d1c/f51 to dd/d19/f78 0 2024-04-12T05:37:04.917 INFO:tasks.workunit.client.0.smithi032.stdout:7/351: rmdir d0/dd/d74 0 2024-04-12T05:37:04.919 INFO:tasks.workunit.client.0.smithi032.stdout:9/329: dwrite f5 [0,4194304] 0 2024-04-12T05:37:04.921 INFO:tasks.workunit.client.0.smithi032.stdout:2/412: mknod dd/d1b/d42/c79 0 2024-04-12T05:37:04.922 INFO:tasks.workunit.client.0.smithi032.stdout:7/352: dread d0/de/df/d20/f6b [0,4194304] 0 2024-04-12T05:37:04.926 INFO:tasks.workunit.client.0.smithi032.stdout:9/330: rename d9/d15/d1a/d24/f3f to d9/d15/d1a/d1d/d5a/d6b/f6c 0 2024-04-12T05:37:04.927 INFO:tasks.workunit.client.0.smithi032.stdout:7/353: write d0/d34/f5d [2888956,123861] 0 2024-04-12T05:37:04.931 INFO:tasks.workunit.client.0.smithi032.stdout:9/331: link d9/d32/f53 d9/d15/d2c/f6d 0 2024-04-12T05:37:04.933 INFO:tasks.workunit.client.0.smithi032.stdout:2/413: rmdir dd/d19 39 2024-04-12T05:37:04.933 INFO:tasks.workunit.client.0.smithi032.stdout:7/354: dread d0/d10/f62 [0,4194304] 0 2024-04-12T05:37:04.938 INFO:tasks.workunit.client.0.smithi032.stdout:7/355: rename d0/c54 to d0/d76/c78 0 2024-04-12T05:37:04.942 INFO:tasks.workunit.client.0.smithi032.stdout:3/373: dwrite d12/d2e/d3a/d2d/f45 [0,4194304] 0 2024-04-12T05:37:04.945 INFO:tasks.workunit.client.0.smithi032.stdout:7/356: unlink d0/d13/d21/d30/d31/l49 0 2024-04-12T05:37:04.948 INFO:tasks.workunit.client.0.smithi032.stdout:2/414: link dd/d1b/d1c/d64/c6e dd/d1b/d1c/d64/d32/d3a/c7a 0 2024-04-12T05:37:04.951 INFO:tasks.workunit.client.0.smithi032.stdout:2/415: symlink dd/d1b/l7b 0 2024-04-12T05:37:04.952 INFO:tasks.workunit.client.0.smithi032.stdout:3/374: creat d12/f77 x:0 0 0 2024-04-12T05:37:04.954 INFO:tasks.workunit.client.0.smithi032.stdout:7/357: dwrite d0/d13/d21/f45 [0,4194304] 0 2024-04-12T05:37:04.955 INFO:tasks.workunit.client.0.smithi032.stdout:2/416: write dd/d1b/d1c/d64/d32/f33 [3275586,15062] 0 2024-04-12T05:37:04.956 INFO:tasks.workunit.client.0.smithi032.stdout:7/358: chown d0/de/l33 9983963 1 2024-04-12T05:37:04.960 INFO:tasks.workunit.client.0.smithi032.stdout:2/417: chown dd/d1b/d1c/d64/d32/d3a/f56 0 1 2024-04-12T05:37:04.963 INFO:tasks.workunit.client.0.smithi032.stdout:2/418: readlink dd/d19/l26 0 2024-04-12T05:37:04.965 INFO:tasks.workunit.client.0.smithi032.stdout:3/375: dwrite d12/d2f/f48 [0,4194304] 0 2024-04-12T05:37:04.965 INFO:tasks.workunit.client.0.smithi032.stdout:9/332: dread d9/d1f/f46 [0,4194304] 0 2024-04-12T05:37:04.970 INFO:tasks.workunit.client.0.smithi032.stdout:3/376: unlink lc 0 2024-04-12T05:37:04.971 INFO:tasks.workunit.client.0.smithi032.stdout:9/333: mkdir d9/d32/d6e 0 2024-04-12T05:37:04.971 INFO:tasks.workunit.client.0.smithi032.stdout:5/463: sync 2024-04-12T05:37:04.973 INFO:tasks.workunit.client.0.smithi032.stdout:2/419: rename dd/d1b/d1c/l3b to dd/d1b/d1c/d64/l7c 0 2024-04-12T05:37:04.973 INFO:tasks.workunit.client.0.smithi032.stdout:7/359: dwrite d0/f8 [0,4194304] 0 2024-04-12T05:37:04.977 INFO:tasks.workunit.client.0.smithi032.stdout:5/464: symlink d2/df/d25/l8f 0 2024-04-12T05:37:04.977 INFO:tasks.workunit.client.0.smithi032.stdout:3/377: dread d12/d2e/f67 [0,4194304] 0 2024-04-12T05:37:04.979 INFO:tasks.workunit.client.0.smithi032.stdout:2/420: fdatasync dd/d1b/d1c/d64/d32/d6d/f77 0 2024-04-12T05:37:04.983 INFO:tasks.workunit.client.0.smithi032.stdout:7/360: write d0/de/f61 [617020,58103] 0 2024-04-12T05:37:04.984 INFO:tasks.workunit.client.0.smithi032.stdout:5/465: symlink d2/d65/d78/l90 0 2024-04-12T05:37:04.985 INFO:tasks.workunit.client.0.smithi032.stdout:7/361: truncate d0/d13/d21/f57 1073204 0 2024-04-12T05:37:04.985 INFO:tasks.workunit.client.0.smithi032.stdout:5/466: chown d2/d3c/d5b 246 1 2024-04-12T05:37:04.987 INFO:tasks.workunit.client.0.smithi032.stdout:2/421: creat dd/d1b/d1c/d64/f7d x:0 0 0 2024-04-12T05:37:04.987 INFO:tasks.workunit.client.0.smithi032.stdout:2/422: chown f0 0 1 2024-04-12T05:37:04.988 INFO:tasks.workunit.client.0.smithi032.stdout:3/378: rmdir d12/d2e/d3a/d72 0 2024-04-12T05:37:04.989 INFO:tasks.workunit.client.0.smithi032.stdout:5/467: truncate d2/dd/f47 816433 0 2024-04-12T05:37:04.990 INFO:tasks.workunit.client.0.smithi032.stdout:2/423: rename dd/d19/c40 to dd/d19/c7e 0 2024-04-12T05:37:04.992 INFO:tasks.workunit.client.0.smithi032.stdout:5/468: unlink d2/df/d25/d43/d49/c8c 0 2024-04-12T05:37:04.993 INFO:tasks.workunit.client.0.smithi032.stdout:2/424: creat dd/d1b/d1c/d64/d32/d6d/f7f x:0 0 0 2024-04-12T05:37:04.994 INFO:tasks.workunit.client.0.smithi032.stdout:5/469: write d2/df/d25/f29 [1826263,84552] 0 2024-04-12T05:37:04.996 INFO:tasks.workunit.client.0.smithi032.stdout:9/334: dread d9/d15/d2c/f14 [0,4194304] 0 2024-04-12T05:37:05.003 INFO:tasks.workunit.client.0.smithi032.stdout:5/470: dwrite d2/d3c/d5b/f5e [4194304,4194304] 0 2024-04-12T05:37:05.008 INFO:tasks.workunit.client.0.smithi032.stdout:9/335: dread d9/f13 [0,4194304] 0 2024-04-12T05:37:05.010 INFO:tasks.workunit.client.0.smithi032.stdout:3/379: rename d12/d2e/l61 to d12/d2e/d3a/d2d/l78 0 2024-04-12T05:37:05.010 INFO:tasks.workunit.client.0.smithi032.stdout:2/425: dread f8 [0,4194304] 0 2024-04-12T05:37:05.011 INFO:tasks.workunit.client.0.smithi032.stdout:7/362: dread d0/de/df/f1e [0,4194304] 0 2024-04-12T05:37:05.014 INFO:tasks.workunit.client.0.smithi032.stdout:9/336: creat d9/d15/d1a/d1d/f6f x:0 0 0 2024-04-12T05:37:05.014 INFO:tasks.workunit.client.0.smithi032.stdout:0/457: sync 2024-04-12T05:37:05.014 INFO:tasks.workunit.client.0.smithi032.stdout:1/331: sync 2024-04-12T05:37:05.014 INFO:tasks.workunit.client.0.smithi032.stdout:4/339: sync 2024-04-12T05:37:05.014 INFO:tasks.workunit.client.0.smithi032.stdout:6/426: sync 2024-04-12T05:37:05.014 INFO:tasks.workunit.client.0.smithi032.stdout:8/369: sync 2024-04-12T05:37:05.015 INFO:tasks.workunit.client.0.smithi032.stdout:3/380: creat d12/d2e/d3a/f79 x:0 0 0 2024-04-12T05:37:05.018 INFO:tasks.workunit.client.0.smithi032.stdout:7/363: unlink d0/d1c/l2c 0 2024-04-12T05:37:05.019 INFO:tasks.workunit.client.0.smithi032.stdout:9/337: rmdir d9/d15 39 2024-04-12T05:37:05.019 INFO:tasks.workunit.client.0.smithi032.stdout:2/426: dwrite dd/d19/d20/f4d [0,4194304] 0 2024-04-12T05:37:05.022 INFO:tasks.workunit.client.0.smithi032.stdout:1/332: rename d5/ff to d5/de/d2a/f68 0 2024-04-12T05:37:05.022 INFO:tasks.workunit.client.0.smithi032.stdout:6/427: chown d4/d21/f2f 1 1 2024-04-12T05:37:05.025 INFO:tasks.workunit.client.0.smithi032.stdout:3/381: mknod d12/d2f/c7a 0 2024-04-12T05:37:05.027 INFO:tasks.workunit.client.0.smithi032.stdout:9/338: mknod d9/d1f/c70 0 2024-04-12T05:37:05.029 INFO:tasks.workunit.client.0.smithi032.stdout:0/458: dread d2/d8/d17/f2b [0,4194304] 0 2024-04-12T05:37:05.031 INFO:tasks.workunit.client.0.smithi032.stdout:5/471: truncate d2/dd/f83 2188222 0 2024-04-12T05:37:05.034 INFO:tasks.workunit.client.0.smithi032.stdout:6/428: mknod d4/d14/d1a/d37/d47/c8f 0 2024-04-12T05:37:05.035 INFO:tasks.workunit.client.0.smithi032.stdout:1/333: dread d5/de/d2a/f2b [0,4194304] 0 2024-04-12T05:37:05.035 INFO:tasks.workunit.client.0.smithi032.stdout:4/340: dwrite d4/d13/d19/d28/d2f/d46/f51 [0,4194304] 0 2024-04-12T05:37:05.036 INFO:tasks.workunit.client.0.smithi032.stdout:7/364: dwrite d0/d76/f52 [0,4194304] 0 2024-04-12T05:37:05.038 INFO:tasks.workunit.client.0.smithi032.stdout:3/382: creat d12/d2e/f7b x:0 0 0 2024-04-12T05:37:05.038 INFO:tasks.workunit.client.0.smithi032.stdout:0/459: mknod d2/d8/df/d24/d60/c94 0 2024-04-12T05:37:05.039 INFO:tasks.workunit.client.0.smithi032.stdout:2/427: creat dd/d1b/d42/f80 x:0 0 0 2024-04-12T05:37:05.040 INFO:tasks.workunit.client.0.smithi032.stdout:8/370: truncate d0/db/d38/f61 1301386 0 2024-04-12T05:37:05.041 INFO:tasks.workunit.client.0.smithi032.stdout:9/339: unlink d9/d15/d1a/d1d/f63 0 2024-04-12T05:37:05.046 INFO:tasks.workunit.client.0.smithi032.stdout:1/334: mknod d5/d2c/c69 0 2024-04-12T05:37:05.047 INFO:tasks.workunit.client.0.smithi032.stdout:0/460: dread d2/dc/d3e/f46 [0,4194304] 0 2024-04-12T05:37:05.048 INFO:tasks.workunit.client.0.smithi032.stdout:3/383: dwrite d12/d2e/f5a [0,4194304] 0 2024-04-12T05:37:05.049 INFO:tasks.workunit.client.0.smithi032.stdout:1/335: chown d5/de/d17/d35/d3b/d49 51744 1 2024-04-12T05:37:05.052 INFO:tasks.workunit.client.0.smithi032.stdout:2/428: creat dd/d1b/d1c/f81 x:0 0 0 2024-04-12T05:37:05.052 INFO:tasks.workunit.client.0.smithi032.stdout:9/340: mkdir d9/d15/d1a/d24/d71 0 2024-04-12T05:37:05.053 INFO:tasks.workunit.client.0.smithi032.stdout:8/371: symlink d0/db/dd/d11/d1b/d22/l81 0 2024-04-12T05:37:05.054 INFO:tasks.workunit.client.0.smithi032.stdout:7/365: symlink d0/dd/d51/d65/l79 0 2024-04-12T05:37:05.055 INFO:tasks.workunit.client.0.smithi032.stdout:9/341: write d9/d15/f43 [4750277,47483] 0 2024-04-12T05:37:05.058 INFO:tasks.workunit.client.0.smithi032.stdout:9/342: chown d9/d15 3 1 2024-04-12T05:37:05.059 INFO:tasks.workunit.client.0.smithi032.stdout:8/372: write d0/db/dd/d11/d1b/f78 [344514,55619] 0 2024-04-12T05:37:05.059 INFO:tasks.workunit.client.0.smithi032.stdout:6/429: rename d4/d14/c1e to d4/c90 0 2024-04-12T05:37:05.061 INFO:tasks.workunit.client.0.smithi032.stdout:6/430: fdatasync f3 0 2024-04-12T05:37:05.063 INFO:tasks.workunit.client.0.smithi032.stdout:6/431: stat l1 0 2024-04-12T05:37:05.063 INFO:tasks.workunit.client.0.smithi032.stdout:0/461: unlink d2/d28/d50/d51/c58 0 2024-04-12T05:37:05.064 INFO:tasks.workunit.client.0.smithi032.stdout:9/343: write d9/d15/d2c/d41/f59 [740478,118807] 0 2024-04-12T05:37:05.065 INFO:tasks.workunit.client.0.smithi032.stdout:2/429: dwrite dd/d1b/f6c [0,4194304] 0 2024-04-12T05:37:05.069 INFO:tasks.workunit.client.0.smithi032.stdout:3/384: dread d12/f2a [0,4194304] 0 2024-04-12T05:37:05.071 INFO:tasks.workunit.client.0.smithi032.stdout:4/341: getdents d4/d13/d19 0 2024-04-12T05:37:05.072 INFO:tasks.workunit.client.0.smithi032.stdout:3/385: truncate d12/d2f/f57 515251 0 2024-04-12T05:37:05.073 INFO:tasks.workunit.client.0.smithi032.stdout:2/430: dread dd/f2e [0,4194304] 0 2024-04-12T05:37:05.074 INFO:tasks.workunit.client.0.smithi032.stdout:3/386: read f10 [1896448,86085] 0 2024-04-12T05:37:05.074 INFO:tasks.workunit.client.0.smithi032.stdout:3/387: dread - d12/d2f/f41 zero size 2024-04-12T05:37:05.075 INFO:tasks.workunit.client.0.smithi032.stdout:2/431: write dd/d1b/d1c/d64/f7d [235786,119239] 0 2024-04-12T05:37:05.076 INFO:tasks.workunit.client.0.smithi032.stdout:0/462: creat d2/d28/d87/f95 x:0 0 0 2024-04-12T05:37:05.077 INFO:tasks.workunit.client.0.smithi032.stdout:2/432: write dd/d19/d20/f53 [4847990,105705] 0 2024-04-12T05:37:05.078 INFO:tasks.workunit.client.0.smithi032.stdout:6/432: unlink d4/d14/d1a/l1f 0 2024-04-12T05:37:05.084 INFO:tasks.workunit.client.0.smithi032.stdout:4/342: creat d4/d13/d19/d28/d2f/f73 x:0 0 0 2024-04-12T05:37:05.084 INFO:tasks.workunit.client.0.smithi032.stdout:4/343: fdatasync d4/d13/f1c 0 2024-04-12T05:37:05.085 INFO:tasks.workunit.client.0.smithi032.stdout:8/373: truncate d0/db/fc 1785245 0 2024-04-12T05:37:05.086 INFO:tasks.workunit.client.0.smithi032.stdout:8/374: fsync d0/db/dd/d11/d1b/f78 0 2024-04-12T05:37:05.090 INFO:tasks.workunit.client.0.smithi032.stdout:7/366: creat d0/de/df/f7a x:0 0 0 2024-04-12T05:37:05.092 INFO:tasks.workunit.client.0.smithi032.stdout:3/388: dwrite d12/d2e/d3a/f26 [0,4194304] 0 2024-04-12T05:37:05.092 INFO:tasks.workunit.client.0.smithi032.stdout:1/336: dread d5/d2c/f2d [4194304,4194304] 0 2024-04-12T05:37:05.097 INFO:tasks.workunit.client.0.smithi032.stdout:0/463: mknod d2/d28/d50/c96 0 2024-04-12T05:37:05.098 INFO:tasks.workunit.client.0.smithi032.stdout:9/344: link d9/d15/d2c/c17 d9/d15/d1a/d31/c72 0 2024-04-12T05:37:05.100 INFO:tasks.workunit.client.0.smithi032.stdout:7/367: unlink d0/f3e 0 2024-04-12T05:37:05.102 INFO:tasks.workunit.client.0.smithi032.stdout:4/344: mknod d4/d13/d19/d28/d2f/d6e/c74 0 2024-04-12T05:37:05.102 INFO:tasks.workunit.client.0.smithi032.stdout:8/375: rmdir d0/db/dd/d11/d1b/d22/d35/d44 39 2024-04-12T05:37:05.104 INFO:tasks.workunit.client.0.smithi032.stdout:0/464: rmdir d2/d28/d50/d51 39 2024-04-12T05:37:05.107 INFO:tasks.workunit.client.0.smithi032.stdout:1/337: symlink d5/d2c/d4f/l6a 0 2024-04-12T05:37:05.107 INFO:tasks.workunit.client.0.smithi032.stdout:1/338: chown d5/de/d17 48 1 2024-04-12T05:37:05.108 INFO:tasks.workunit.client.0.smithi032.stdout:6/433: getdents d4/d14/d1a/d37/d47 0 2024-04-12T05:37:05.109 INFO:tasks.workunit.client.0.smithi032.stdout:8/376: mkdir d0/db/dd/d11/d1b/d22/d54/d82 0 2024-04-12T05:37:05.110 INFO:tasks.workunit.client.0.smithi032.stdout:6/434: dread - d4/d21/d2e/d32/f7a zero size 2024-04-12T05:37:05.110 INFO:tasks.workunit.client.0.smithi032.stdout:4/345: mkdir d4/d13/d75 0 2024-04-12T05:37:05.113 INFO:tasks.workunit.client.0.smithi032.stdout:1/339: mknod d5/de/d5b/c6b 0 2024-04-12T05:37:05.113 INFO:tasks.workunit.client.0.smithi032.stdout:4/346: readlink d4/d13/d19/d28/d2f/d3b/l40 0 2024-04-12T05:37:05.114 INFO:tasks.workunit.client.0.smithi032.stdout:7/368: dread d0/d1c/d4c/f5e [0,4194304] 0 2024-04-12T05:37:05.116 INFO:tasks.workunit.client.0.smithi032.stdout:1/340: rmdir d5/de/d17 39 2024-04-12T05:37:05.119 INFO:tasks.workunit.client.0.smithi032.stdout:3/389: dread d12/d2e/f4a [0,4194304] 0 2024-04-12T05:37:05.119 INFO:tasks.workunit.client.0.smithi032.stdout:8/377: dwrite d0/db/dd/d11/f1f [0,4194304] 0 2024-04-12T05:37:05.125 INFO:tasks.workunit.client.0.smithi032.stdout:3/390: stat d12/d24/c3c 0 2024-04-12T05:37:05.126 INFO:tasks.workunit.client.0.smithi032.stdout:0/465: write d2/d8/d17/f6b [409551,41688] 0 2024-04-12T05:37:05.136 INFO:tasks.workunit.client.0.smithi032.stdout:3/391: mknod d12/d2e/c7c 0 2024-04-12T05:37:05.136 INFO:tasks.workunit.client.0.smithi032.stdout:7/369: dread d0/f7 [0,4194304] 0 2024-04-12T05:37:05.136 INFO:tasks.workunit.client.0.smithi032.stdout:1/341: dwrite d5/de/f4a [0,4194304] 0 2024-04-12T05:37:05.136 INFO:tasks.workunit.client.0.smithi032.stdout:8/378: rename d0/db/fc to d0/db/d39/f83 0 2024-04-12T05:37:05.136 INFO:tasks.workunit.client.0.smithi032.stdout:9/345: dwrite d9/f34 [0,4194304] 0 2024-04-12T05:37:05.137 INFO:tasks.workunit.client.0.smithi032.stdout:3/392: chown d12/l15 194 1 2024-04-12T05:37:05.137 INFO:tasks.workunit.client.0.smithi032.stdout:7/370: truncate d0/dd/f55 77017 0 2024-04-12T05:37:05.141 INFO:tasks.workunit.client.0.smithi032.stdout:8/379: mkdir d0/d36/d84 0 2024-04-12T05:37:05.143 INFO:tasks.workunit.client.0.smithi032.stdout:3/393: write d12/f59 [797834,58840] 0 2024-04-12T05:37:05.144 INFO:tasks.workunit.client.0.smithi032.stdout:3/394: write f10 [3556014,23505] 0 2024-04-12T05:37:05.150 INFO:tasks.workunit.client.0.smithi032.stdout:1/342: creat d5/de/d17/d35/d61/d54/f6c x:0 0 0 2024-04-12T05:37:05.150 INFO:tasks.workunit.client.0.smithi032.stdout:1/343: stat d5/de/d17/d1b/f33 0 2024-04-12T05:37:05.155 INFO:tasks.workunit.client.0.smithi032.stdout:8/380: symlink d0/db/dd/d11/d1b/d1e/l85 0 2024-04-12T05:37:05.155 INFO:tasks.workunit.client.0.smithi032.stdout:0/466: rename d2/f30 to d2/d8/d17/d1b/f97 0 2024-04-12T05:37:05.156 INFO:tasks.workunit.client.0.smithi032.stdout:1/344: mknod d5/de/d17/d1b/c6d 0 2024-04-12T05:37:05.157 INFO:tasks.workunit.client.0.smithi032.stdout:3/395: rename d12/d2e/f6e to d12/d2f/f7d 0 2024-04-12T05:37:05.160 INFO:tasks.workunit.client.0.smithi032.stdout:3/396: write d12/d2e/f4a [1119063,11318] 0 2024-04-12T05:37:05.161 INFO:tasks.workunit.client.0.smithi032.stdout:8/381: mknod d0/db/dd/d11/d1b/d22/d35/c86 0 2024-04-12T05:37:05.165 INFO:tasks.workunit.client.0.smithi032.stdout:7/371: write d0/f7 [450017,9789] 0 2024-04-12T05:37:05.165 INFO:tasks.workunit.client.0.smithi032.stdout:1/345: dwrite d5/de/d17/d1b/f21 [0,4194304] 0 2024-04-12T05:37:05.165 INFO:tasks.workunit.client.0.smithi032.stdout:0/467: truncate d2/fe 791965 0 2024-04-12T05:37:05.168 INFO:tasks.workunit.client.0.smithi032.stdout:8/382: write d0/db/dd/d11/d1b/d22/f4f [1965630,14020] 0 2024-04-12T05:37:05.170 INFO:tasks.workunit.client.0.smithi032.stdout:8/383: chown d0/d72 7730 1 2024-04-12T05:37:05.171 INFO:tasks.workunit.client.0.smithi032.stdout:1/346: symlink d5/l6e 0 2024-04-12T05:37:05.174 INFO:tasks.workunit.client.0.smithi032.stdout:3/397: dwrite d12/f2a [0,4194304] 0 2024-04-12T05:37:05.174 INFO:tasks.workunit.client.0.smithi032.stdout:1/347: chown d5/de/c36 70980262 1 2024-04-12T05:37:05.177 INFO:tasks.workunit.client.0.smithi032.stdout:8/384: rmdir d0/d58 39 2024-04-12T05:37:05.177 INFO:tasks.workunit.client.0.smithi032.stdout:9/346: dread d9/d15/d1a/d24/f40 [0,4194304] 0 2024-04-12T05:37:05.178 INFO:tasks.workunit.client.0.smithi032.stdout:0/468: creat d2/d8/d17/d1b/f98 x:0 0 0 2024-04-12T05:37:05.179 INFO:tasks.workunit.client.0.smithi032.stdout:8/385: chown d0/db/d39 85128299 1 2024-04-12T05:37:05.180 INFO:tasks.workunit.client.0.smithi032.stdout:7/372: dread d0/de/f61 [0,4194304] 0 2024-04-12T05:37:05.182 INFO:tasks.workunit.client.0.smithi032.stdout:1/348: truncate d5/de/d17/d35/d61/f50 240821 0 2024-04-12T05:37:05.183 INFO:tasks.workunit.client.0.smithi032.stdout:0/469: write d2/d29/f70 [2489855,95530] 0 2024-04-12T05:37:05.183 INFO:tasks.workunit.client.0.smithi032.stdout:3/398: dread d12/d2e/d3a/d2d/f45 [0,4194304] 0 2024-04-12T05:37:05.185 INFO:tasks.workunit.client.0.smithi032.stdout:1/349: read d5/de/d2a/f3f [2300862,17784] 0 2024-04-12T05:37:05.185 INFO:tasks.workunit.client.0.smithi032.stdout:3/399: write d12/d2f/d5b/f66 [952421,56546] 0 2024-04-12T05:37:05.186 INFO:tasks.workunit.client.0.smithi032.stdout:9/347: symlink d9/d15/d1a/l73 0 2024-04-12T05:37:05.188 INFO:tasks.workunit.client.0.smithi032.stdout:7/373: write d0/de/f12 [3344175,5508] 0 2024-04-12T05:37:05.188 INFO:tasks.workunit.client.0.smithi032.stdout:1/350: rename d5/d2c to d5/d2c/d6f 22 2024-04-12T05:37:05.190 INFO:tasks.workunit.client.0.smithi032.stdout:7/374: truncate d0/dd/f3b 4273119 0 2024-04-12T05:37:05.191 INFO:tasks.workunit.client.0.smithi032.stdout:0/470: mkdir d2/d29/d8a/d99 0 2024-04-12T05:37:05.191 INFO:tasks.workunit.client.0.smithi032.stdout:1/351: write d5/de/d17/d35/d3b/f63 [746550,19132] 0 2024-04-12T05:37:05.194 INFO:tasks.workunit.client.0.smithi032.stdout:3/400: dread d12/d2e/f1e [0,4194304] 0 2024-04-12T05:37:05.196 INFO:tasks.workunit.client.0.smithi032.stdout:3/401: symlink d12/d24/l7e 0 2024-04-12T05:37:05.198 INFO:tasks.workunit.client.0.smithi032.stdout:1/352: rename d5/de/d17/d35/f3d to d5/de/d17/d35/d3b/d49/f70 0 2024-04-12T05:37:05.199 INFO:tasks.workunit.client.0.smithi032.stdout:7/375: dwrite d0/d76/f52 [0,4194304] 0 2024-04-12T05:37:05.201 INFO:tasks.workunit.client.0.smithi032.stdout:3/402: creat d12/d2e/d3a/f7f x:0 0 0 2024-04-12T05:37:05.202 INFO:tasks.workunit.client.0.smithi032.stdout:7/376: write d0/de/f44 [3339652,30139] 0 2024-04-12T05:37:05.202 INFO:tasks.workunit.client.0.smithi032.stdout:3/403: fsync d12/f21 0 2024-04-12T05:37:05.203 INFO:tasks.workunit.client.0.smithi032.stdout:7/377: chown d0/d1c 3668844 1 2024-04-12T05:37:05.203 INFO:tasks.workunit.client.0.smithi032.stdout:3/404: chown d12/d2f/l43 1875 1 2024-04-12T05:37:05.205 INFO:tasks.workunit.client.0.smithi032.stdout:3/405: readlink d12/d24/l7e 0 2024-04-12T05:37:05.206 INFO:tasks.workunit.client.0.smithi032.stdout:7/378: mkdir d0/d10/d70/d7b 0 2024-04-12T05:37:05.207 INFO:tasks.workunit.client.0.smithi032.stdout:3/406: symlink d12/d2e/d3a/d63/l80 0 2024-04-12T05:37:05.208 INFO:tasks.workunit.client.0.smithi032.stdout:3/407: write d12/d2e/d3a/d2d/f4e [1022782,98668] 0 2024-04-12T05:37:05.209 INFO:tasks.workunit.client.0.smithi032.stdout:7/379: mkdir d0/d13/d21/d7c 0 2024-04-12T05:37:05.210 INFO:tasks.workunit.client.0.smithi032.stdout:3/408: truncate d12/d2e/f7b 283050 0 2024-04-12T05:37:05.212 INFO:tasks.workunit.client.0.smithi032.stdout:3/409: creat d12/d2f/f81 x:0 0 0 2024-04-12T05:37:05.215 INFO:tasks.workunit.client.0.smithi032.stdout:7/380: rename d0/dd/d51/f6d to d0/de/df/d29/f7d 0 2024-04-12T05:37:05.218 INFO:tasks.workunit.client.0.smithi032.stdout:3/410: unlink d12/f59 0 2024-04-12T05:37:05.219 INFO:tasks.workunit.client.0.smithi032.stdout:3/411: stat d12/d2e/d3a/d2d/c69 0 2024-04-12T05:37:05.220 INFO:tasks.workunit.client.0.smithi032.stdout:1/353: dread d5/de/f28 [0,4194304] 0 2024-04-12T05:37:05.223 INFO:tasks.workunit.client.0.smithi032.stdout:1/354: fsync d5/d2c/f2d 0 2024-04-12T05:37:05.224 INFO:tasks.workunit.client.0.smithi032.stdout:3/412: dwrite f10 [0,4194304] 0 2024-04-12T05:37:05.225 INFO:tasks.workunit.client.0.smithi032.stdout:3/413: dread - d12/d2f/f41 zero size 2024-04-12T05:37:05.226 INFO:tasks.workunit.client.0.smithi032.stdout:3/414: stat d12/d2e/c6f 0 2024-04-12T05:37:05.227 INFO:tasks.workunit.client.0.smithi032.stdout:3/415: readlink d12/d2e/l25 0 2024-04-12T05:37:05.227 INFO:tasks.workunit.client.0.smithi032.stdout:1/355: mknod d5/de/d17/d35/d3b/c71 0 2024-04-12T05:37:05.229 INFO:tasks.workunit.client.0.smithi032.stdout:1/356: mknod d5/de/d17/d35/d61/c72 0 2024-04-12T05:37:05.229 INFO:tasks.workunit.client.0.smithi032.stdout:1/357: stat d5/de/d5b 0 2024-04-12T05:37:05.230 INFO:tasks.workunit.client.0.smithi032.stdout:3/416: creat d12/d2e/f82 x:0 0 0 2024-04-12T05:37:05.232 INFO:tasks.workunit.client.0.smithi032.stdout:1/358: mkdir d5/de/d17/d35/d73 0 2024-04-12T05:37:05.233 INFO:tasks.workunit.client.0.smithi032.stdout:1/359: dread - d5/de/d17/d35/d3b/d49/f5e zero size 2024-04-12T05:37:05.234 INFO:tasks.workunit.client.0.smithi032.stdout:3/417: creat d12/d2e/d3a/d63/f83 x:0 0 0 2024-04-12T05:37:05.236 INFO:tasks.workunit.client.0.smithi032.stdout:3/418: mknod d12/d2e/c84 0 2024-04-12T05:37:05.238 INFO:tasks.workunit.client.0.smithi032.stdout:5/472: sync 2024-04-12T05:37:05.243 INFO:tasks.workunit.client.0.smithi032.stdout:5/473: mkdir d2/df/d25/d50/d91 0 2024-04-12T05:37:05.245 INFO:tasks.workunit.client.0.smithi032.stdout:3/419: dread d12/d2e/f4a [0,4194304] 0 2024-04-12T05:37:05.253 INFO:tasks.workunit.client.0.smithi032.stdout:3/420: dread d12/d2e/d3a/d2d/f4e [0,4194304] 0 2024-04-12T05:37:05.254 INFO:tasks.workunit.client.0.smithi032.stdout:5/474: getdents d2/d3c/d5b 0 2024-04-12T05:37:05.255 INFO:tasks.workunit.client.0.smithi032.stdout:9/348: write d9/d15/d2c/f6d [1250823,42141] 0 2024-04-12T05:37:05.257 INFO:tasks.workunit.client.0.smithi032.stdout:3/421: chown d12/d2e/c46 23168706 1 2024-04-12T05:37:05.257 INFO:tasks.workunit.client.0.smithi032.stdout:1/360: dread d5/de/f11 [0,4194304] 0 2024-04-12T05:37:05.259 INFO:tasks.workunit.client.0.smithi032.stdout:5/475: write d2/df/d31/d80/f88 [36202,114600] 0 2024-04-12T05:37:05.259 INFO:tasks.workunit.client.0.smithi032.stdout:8/386: dwrite d0/d36/f4a [0,4194304] 0 2024-04-12T05:37:05.261 INFO:tasks.workunit.client.0.smithi032.stdout:1/361: chown d5/de/d17/d35/d3b/d49/l5d 0 1 2024-04-12T05:37:05.261 INFO:tasks.workunit.client.0.smithi032.stdout:5/476: write d2/df/f5c [25019,2408] 0 2024-04-12T05:37:05.262 INFO:tasks.workunit.client.0.smithi032.stdout:8/387: readlink d0/db/dd/d11/d1b/d22/d29/l7b 0 2024-04-12T05:37:05.264 INFO:tasks.workunit.client.0.smithi032.stdout:5/477: fdatasync d2/df/d25/d50/f76 0 2024-04-12T05:37:05.265 INFO:tasks.workunit.client.0.smithi032.stdout:8/388: truncate d0/f2f 4227991 0 2024-04-12T05:37:05.270 INFO:tasks.workunit.client.0.smithi032.stdout:9/349: link d9/d15/d1a/f56 d9/d32/f74 0 2024-04-12T05:37:05.270 INFO:tasks.workunit.client.0.smithi032.stdout:5/478: mkdir d2/df/d31/d92 0 2024-04-12T05:37:05.275 INFO:tasks.workunit.client.0.smithi032.stdout:3/422: write d12/f1b [564985,99759] 0 2024-04-12T05:37:05.276 INFO:tasks.workunit.client.0.smithi032.stdout:5/479: mknod d2/dd/d6f/c93 0 2024-04-12T05:37:05.276 INFO:tasks.workunit.client.0.smithi032.stdout:8/389: dwrite d0/db/dd/d11/f2c [0,4194304] 0 2024-04-12T05:37:05.276 INFO:tasks.workunit.client.0.smithi032.stdout:1/362: dread d5/de/f10 [4194304,4194304] 0 2024-04-12T05:37:05.279 INFO:tasks.workunit.client.0.smithi032.stdout:7/381: dwrite d0/d1c/d4c/f4d [0,4194304] 0 2024-04-12T05:37:05.281 INFO:tasks.workunit.client.0.smithi032.stdout:9/350: rename d9/d15/d2c/f52 to d9/d15/d1a/d1d/d5a/f75 0 2024-04-12T05:37:05.281 INFO:tasks.workunit.client.0.smithi032.stdout:8/390: chown d0/db/f6e 310 1 2024-04-12T05:37:05.281 INFO:tasks.workunit.client.0.smithi032.stdout:1/363: chown d5/d2c/f43 500 1 2024-04-12T05:37:05.282 INFO:tasks.workunit.client.0.smithi032.stdout:5/480: creat d2/d65/d78/f94 x:0 0 0 2024-04-12T05:37:05.282 INFO:tasks.workunit.client.0.smithi032.stdout:3/423: chown d12/d2e/c6d 3 1 2024-04-12T05:37:05.289 INFO:tasks.workunit.client.0.smithi032.stdout:5/481: rename d2/l30 to d2/df/d25/d50/d55/l95 0 2024-04-12T05:37:05.291 INFO:tasks.workunit.client.0.smithi032.stdout:3/424: getdents d12/d2e/d3a 0 2024-04-12T05:37:05.292 INFO:tasks.workunit.client.0.smithi032.stdout:9/351: fdatasync d9/d15/d1a/d1d/d5a/f61 0 2024-04-12T05:37:05.293 INFO:tasks.workunit.client.0.smithi032.stdout:1/364: rename d5/de/d17/d35/d3b/d67 to d5/de/d17/d35/d73/d74 0 2024-04-12T05:37:05.293 INFO:tasks.workunit.client.0.smithi032.stdout:7/382: dwrite d0/de/f12 [0,4194304] 0 2024-04-12T05:37:05.294 INFO:tasks.workunit.client.0.smithi032.stdout:5/482: mkdir d2/df/d25/d43/d96 0 2024-04-12T05:37:05.294 INFO:tasks.workunit.client.0.smithi032.stdout:8/391: write d0/f3 [2230359,81793] 0 2024-04-12T05:37:05.296 INFO:tasks.workunit.client.0.smithi032.stdout:3/425: rmdir d12/d2f 39 2024-04-12T05:37:05.297 INFO:tasks.workunit.client.0.smithi032.stdout:9/352: chown d9/d15/d1a/d24/d27/l68 5311913 1 2024-04-12T05:37:05.297 INFO:tasks.workunit.client.0.smithi032.stdout:7/383: creat d0/d76/f7e x:0 0 0 2024-04-12T05:37:05.301 INFO:tasks.workunit.client.0.smithi032.stdout:1/365: dread d5/de/d17/f25 [0,4194304] 0 2024-04-12T05:37:05.303 INFO:tasks.workunit.client.0.smithi032.stdout:9/353: creat d9/d15/d1a/d24/d27/f76 x:0 0 0 2024-04-12T05:37:05.305 INFO:tasks.workunit.client.0.smithi032.stdout:3/426: mknod d12/d2f/c85 0 2024-04-12T05:37:05.305 INFO:tasks.workunit.client.0.smithi032.stdout:5/483: dread d2/df/d25/d1e/f40 [0,4194304] 0 2024-04-12T05:37:05.305 INFO:tasks.workunit.client.0.smithi032.stdout:7/384: mknod d0/c7f 0 2024-04-12T05:37:05.306 INFO:tasks.workunit.client.0.smithi032.stdout:7/385: readlink d0/de/l33 0 2024-04-12T05:37:05.307 INFO:tasks.workunit.client.0.smithi032.stdout:7/386: chown d0/l1 16354 1 2024-04-12T05:37:05.307 INFO:tasks.workunit.client.0.smithi032.stdout:1/366: creat d5/de/d17/d35/f75 x:0 0 0 2024-04-12T05:37:05.308 INFO:tasks.workunit.client.0.smithi032.stdout:9/354: creat d9/d15/d2c/f77 x:0 0 0 2024-04-12T05:37:05.309 INFO:tasks.workunit.client.0.smithi032.stdout:9/355: write d9/d15/f43 [5203219,25708] 0 2024-04-12T05:37:05.313 INFO:tasks.workunit.client.0.smithi032.stdout:3/427: getdents d12/d2e/d3a/d63 0 2024-04-12T05:37:05.314 INFO:tasks.workunit.client.0.smithi032.stdout:3/428: write d12/d2e/d3a/f7f [853763,93568] 0 2024-04-12T05:37:05.316 INFO:tasks.workunit.client.0.smithi032.stdout:7/387: write d0/dd/f2a [1692475,41022] 0 2024-04-12T05:37:05.317 INFO:tasks.workunit.client.0.smithi032.stdout:5/484: write d2/df/d25/d1e/f2f [4022130,122703] 0 2024-04-12T05:37:05.318 INFO:tasks.workunit.client.0.smithi032.stdout:3/429: creat d12/d2e/f86 x:0 0 0 2024-04-12T05:37:05.320 INFO:tasks.workunit.client.0.smithi032.stdout:9/356: truncate d9/d15/d1a/d1d/d5a/f75 3379881 0 2024-04-12T05:37:05.321 INFO:tasks.workunit.client.0.smithi032.stdout:8/392: dwrite d0/db/dd/d11/d1b/d22/d51/f69 [0,4194304] 0 2024-04-12T05:37:05.322 INFO:tasks.workunit.client.0.smithi032.stdout:5/485: truncate d2/df/f5c 288643 0 2024-04-12T05:37:05.323 INFO:tasks.workunit.client.0.smithi032.stdout:3/430: creat d12/d2f/d5b/f87 x:0 0 0 2024-04-12T05:37:05.326 INFO:tasks.workunit.client.0.smithi032.stdout:7/388: dread d0/d13/f1a [0,4194304] 0 2024-04-12T05:37:05.327 INFO:tasks.workunit.client.0.smithi032.stdout:7/389: read d0/f8 [646079,74180] 0 2024-04-12T05:37:05.327 INFO:tasks.workunit.client.0.smithi032.stdout:5/486: dread d2/df/d31/d80/f88 [0,4194304] 0 2024-04-12T05:37:05.333 INFO:tasks.workunit.client.0.smithi032.stdout:7/390: unlink d0/d13/d21/d30/d31/f58 0 2024-04-12T05:37:05.334 INFO:tasks.workunit.client.0.smithi032.stdout:9/357: link d9/d15/d2c/f5e d9/d15/d1a/d1d/f78 0 2024-04-12T05:37:05.334 INFO:tasks.workunit.client.0.smithi032.stdout:9/358: stat d9/d15/d1a/d1d/d5a/d6b 0 2024-04-12T05:37:05.336 INFO:tasks.workunit.client.0.smithi032.stdout:8/393: creat d0/db/dd/d11/d1b/d22/d35/f87 x:0 0 0 2024-04-12T05:37:05.336 INFO:tasks.workunit.client.0.smithi032.stdout:5/487: creat d2/dd/d5d/f97 x:0 0 0 2024-04-12T05:37:05.338 INFO:tasks.workunit.client.0.smithi032.stdout:5/488: fdatasync d2/df/d25/d1e/f1f 0 2024-04-12T05:37:05.338 INFO:tasks.workunit.client.0.smithi032.stdout:8/394: chown d0/db/dd/d11/d1b/d22/d35/l48 18 1 2024-04-12T05:37:05.338 INFO:tasks.workunit.client.0.smithi032.stdout:7/391: mknod d0/de/df/c80 0 2024-04-12T05:37:05.340 INFO:tasks.workunit.client.0.smithi032.stdout:9/359: getdents d9/d1f 0 2024-04-12T05:37:05.342 INFO:tasks.workunit.client.0.smithi032.stdout:5/489: creat d2/df/d25/f98 x:0 0 0 2024-04-12T05:37:05.342 INFO:tasks.workunit.client.0.smithi032.stdout:7/392: mkdir d0/d13/d21/d81 0 2024-04-12T05:37:05.343 INFO:tasks.workunit.client.0.smithi032.stdout:9/360: creat d9/d58/f79 x:0 0 0 2024-04-12T05:37:05.345 INFO:tasks.workunit.client.0.smithi032.stdout:8/395: dwrite d0/db/f60 [0,4194304] 0 2024-04-12T05:37:05.345 INFO:tasks.workunit.client.0.smithi032.stdout:7/393: read d0/f5 [5823170,77363] 0 2024-04-12T05:37:05.348 INFO:tasks.workunit.client.0.smithi032.stdout:9/361: read d9/d15/d1a/d24/f4b [2884,81098] 0 2024-04-12T05:37:05.350 INFO:tasks.workunit.client.0.smithi032.stdout:8/396: symlink d0/db/dd/d11/d1b/d22/d29/l88 0 2024-04-12T05:37:05.353 INFO:tasks.workunit.client.0.smithi032.stdout:3/431: truncate d12/f13 2490449 0 2024-04-12T05:37:05.354 INFO:tasks.workunit.client.0.smithi032.stdout:8/397: unlink d0/c1 0 2024-04-12T05:37:05.356 INFO:tasks.workunit.client.0.smithi032.stdout:8/398: chown d0/f2f 25656946 1 2024-04-12T05:37:05.356 INFO:tasks.workunit.client.0.smithi032.stdout:7/394: dwrite d0/d76/f7e [0,4194304] 0 2024-04-12T05:37:05.357 INFO:tasks.workunit.client.0.smithi032.stdout:7/395: dread - d0/de/df/f7a zero size 2024-04-12T05:37:05.358 INFO:tasks.workunit.client.0.smithi032.stdout:9/362: symlink d9/d15/d1a/d24/d27/l7a 0 2024-04-12T05:37:05.358 INFO:tasks.workunit.client.0.smithi032.stdout:3/432: symlink d12/l88 0 2024-04-12T05:37:05.361 INFO:tasks.workunit.client.0.smithi032.stdout:5/490: rmdir d2/df/d25 39 2024-04-12T05:37:05.361 INFO:tasks.workunit.client.0.smithi032.stdout:7/396: dread d0/de/df/d20/f6b [0,4194304] 0 2024-04-12T05:37:05.361 INFO:tasks.workunit.client.0.smithi032.stdout:8/399: getdents d0/db/d7d 0 2024-04-12T05:37:05.365 INFO:tasks.workunit.client.0.smithi032.stdout:3/433: mkdir d12/d2f/d89 0 2024-04-12T05:37:05.367 INFO:tasks.workunit.client.0.smithi032.stdout:9/363: symlink d9/d15/d1a/d24/d71/l7b 0 2024-04-12T05:37:05.368 INFO:tasks.workunit.client.0.smithi032.stdout:8/400: rename d0/db/dd/d11/f28 to d0/db/dd/d11/d1b/f89 0 2024-04-12T05:37:05.370 INFO:tasks.workunit.client.0.smithi032.stdout:3/434: creat d12/d2e/d3a/d63/f8a x:0 0 0 2024-04-12T05:37:05.370 INFO:tasks.workunit.client.0.smithi032.stdout:3/435: chown d12/d2e/c68 25954091 1 2024-04-12T05:37:05.373 INFO:tasks.workunit.client.0.smithi032.stdout:5/491: mknod d2/df/d25/d43/d49/c99 0 2024-04-12T05:37:05.375 INFO:tasks.workunit.client.0.smithi032.stdout:9/364: mknod d9/d15/d1a/d24/c7c 0 2024-04-12T05:37:05.378 INFO:tasks.workunit.client.0.smithi032.stdout:3/436: symlink d12/d2f/d5b/l8b 0 2024-04-12T05:37:05.378 INFO:tasks.workunit.client.0.smithi032.stdout:3/437: chown d12/d2e/c23 83821126 1 2024-04-12T05:37:05.381 INFO:tasks.workunit.client.0.smithi032.stdout:5/492: symlink d2/df/d25/l9a 0 2024-04-12T05:37:05.383 INFO:tasks.workunit.client.0.smithi032.stdout:3/438: symlink d12/l8c 0 2024-04-12T05:37:05.384 INFO:tasks.workunit.client.0.smithi032.stdout:3/439: stat d12/d2e/c68 0 2024-04-12T05:37:05.385 INFO:tasks.workunit.client.0.smithi032.stdout:8/401: getdents d0/db/dd/d11/d1b/d22/d54 0 2024-04-12T05:37:05.387 INFO:tasks.workunit.client.0.smithi032.stdout:5/493: dwrite d2/d3c/d5b/f5e [0,4194304] 0 2024-04-12T05:37:05.395 INFO:tasks.workunit.client.0.smithi032.stdout:5/494: write d2/f24 [2246970,93187] 0 2024-04-12T05:37:05.395 INFO:tasks.workunit.client.0.smithi032.stdout:9/365: dwrite d9/d15/d1a/f56 [0,4194304] 0 2024-04-12T05:37:05.396 INFO:tasks.workunit.client.0.smithi032.stdout:9/366: dread - d9/d58/f79 zero size 2024-04-12T05:37:05.400 INFO:tasks.workunit.client.0.smithi032.stdout:8/402: dwrite d0/db/dd/d11/d1b/f6c [0,4194304] 0 2024-04-12T05:37:05.404 INFO:tasks.workunit.client.0.smithi032.stdout:8/403: symlink d0/db/dd/d11/d1b/d5f/l8a 0 2024-04-12T05:37:05.406 INFO:tasks.workunit.client.0.smithi032.stdout:9/367: getdents d9/d15 0 2024-04-12T05:37:05.407 INFO:tasks.workunit.client.0.smithi032.stdout:8/404: symlink d0/db/dd/d11/d1b/d22/d51/l8b 0 2024-04-12T05:37:05.409 INFO:tasks.workunit.client.0.smithi032.stdout:9/368: creat d9/d15/d1a/d24/f7d x:0 0 0 2024-04-12T05:37:05.412 INFO:tasks.workunit.client.0.smithi032.stdout:9/369: rename d9/d15/d1a/f3d to d9/d15/d2c/d41/f7e 0 2024-04-12T05:37:05.415 INFO:tasks.workunit.client.0.smithi032.stdout:8/405: dwrite d0/f7 [0,4194304] 0 2024-04-12T05:37:05.418 INFO:tasks.workunit.client.0.smithi032.stdout:8/406: rename d0/l7c to d0/d36/l8c 0 2024-04-12T05:37:05.420 INFO:tasks.workunit.client.0.smithi032.stdout:8/407: write d0/db/f55 [79971,120798] 0 2024-04-12T05:37:05.421 INFO:tasks.workunit.client.0.smithi032.stdout:6/435: sync 2024-04-12T05:37:05.421 INFO:tasks.workunit.client.0.smithi032.stdout:2/433: sync 2024-04-12T05:37:05.423 INFO:tasks.workunit.client.0.smithi032.stdout:6/436: readlink d4/d14/d25/l2b 0 2024-04-12T05:37:05.424 INFO:tasks.workunit.client.0.smithi032.stdout:2/434: mkdir dd/d19/d82 0 2024-04-12T05:37:05.426 INFO:tasks.workunit.client.0.smithi032.stdout:8/408: rename d0/db/dd/d11/d1b/d22/d54/d82 to d0/db/d38/d63/d77/d8d 0 2024-04-12T05:37:05.429 INFO:tasks.workunit.client.0.smithi032.stdout:8/409: symlink d0/db/dd/d11/d1b/l8e 0 2024-04-12T05:37:05.429 INFO:tasks.workunit.client.0.smithi032.stdout:6/437: getdents d4/d14/d1a/d88 0 2024-04-12T05:37:05.432 INFO:tasks.workunit.client.0.smithi032.stdout:6/438: truncate d4/d14/d1a/d37/d47/f56 636087 0 2024-04-12T05:37:05.433 INFO:tasks.workunit.client.0.smithi032.stdout:6/439: chown d4/d14/d1a/d37/d69 96 1 2024-04-12T05:37:05.434 INFO:tasks.workunit.client.0.smithi032.stdout:8/410: dread d0/db/f60 [0,4194304] 0 2024-04-12T05:37:05.436 INFO:tasks.workunit.client.0.smithi032.stdout:5/495: dread d2/dd/f3a [0,4194304] 0 2024-04-12T05:37:05.437 INFO:tasks.workunit.client.0.smithi032.stdout:6/440: unlink d4/d14/d3c/f53 0 2024-04-12T05:37:05.437 INFO:tasks.workunit.client.0.smithi032.stdout:2/435: truncate f0 1829494 0 2024-04-12T05:37:05.438 INFO:tasks.workunit.client.0.smithi032.stdout:8/411: symlink d0/db/dd/d11/d1b/d22/l8f 0 2024-04-12T05:37:05.439 INFO:tasks.workunit.client.0.smithi032.stdout:2/436: write dd/d19/d20/f53 [4678752,26658] 0 2024-04-12T05:37:05.439 INFO:tasks.workunit.client.0.smithi032.stdout:8/412: chown d0/db/d7d 403172143 1 2024-04-12T05:37:05.439 INFO:tasks.workunit.client.0.smithi032.stdout:5/496: fsync d2/f68 0 2024-04-12T05:37:05.440 INFO:tasks.workunit.client.0.smithi032.stdout:6/441: creat d4/d74/f91 x:0 0 0 2024-04-12T05:37:05.441 INFO:tasks.workunit.client.0.smithi032.stdout:8/413: write d0/d36/f4a [4316002,64353] 0 2024-04-12T05:37:05.444 INFO:tasks.workunit.client.0.smithi032.stdout:8/414: chown d0/d36/l37 3146 1 2024-04-12T05:37:05.449 INFO:tasks.workunit.client.0.smithi032.stdout:6/442: getdents d4/d14/d25 0 2024-04-12T05:37:05.449 INFO:tasks.workunit.client.0.smithi032.stdout:8/415: write d0/db/dd/d11/f1f [5022141,47153] 0 2024-04-12T05:37:05.450 INFO:tasks.workunit.client.0.smithi032.stdout:2/437: dwrite dd/d19/f24 [0,4194304] 0 2024-04-12T05:37:05.452 INFO:tasks.workunit.client.0.smithi032.stdout:5/497: dwrite d2/f68 [0,4194304] 0 2024-04-12T05:37:05.453 INFO:tasks.workunit.client.0.smithi032.stdout:2/438: read - dd/d19/d20/f4b zero size 2024-04-12T05:37:05.455 INFO:tasks.workunit.client.0.smithi032.stdout:8/416: rmdir d0/db/dd/d11/d1b/d22/d29 39 2024-04-12T05:37:05.459 INFO:tasks.workunit.client.0.smithi032.stdout:2/439: mkdir dd/d19/d20/d83 0 2024-04-12T05:37:05.461 INFO:tasks.workunit.client.0.smithi032.stdout:6/443: dwrite d4/d21/f3b [0,4194304] 0 2024-04-12T05:37:05.465 INFO:tasks.workunit.client.0.smithi032.stdout:5/498: rename d2/df/d25/d50/d55 to d2/d9b 0 2024-04-12T05:37:05.467 INFO:tasks.workunit.client.0.smithi032.stdout:5/499: write d2/d3c/d5b/f5e [1974875,87600] 0 2024-04-12T05:37:05.467 INFO:tasks.workunit.client.0.smithi032.stdout:8/417: creat d0/db/dd/d11/d1b/d22/d35/d44/f90 x:0 0 0 2024-04-12T05:37:05.468 INFO:tasks.workunit.client.0.smithi032.stdout:6/444: mkdir d4/d14/d1a/d92 0 2024-04-12T05:37:05.470 INFO:tasks.workunit.client.0.smithi032.stdout:6/445: chown d4/d14/d1a/l22 13 1 2024-04-12T05:37:05.470 INFO:tasks.workunit.client.0.smithi032.stdout:8/418: rmdir d0/d36 39 2024-04-12T05:37:05.471 INFO:tasks.workunit.client.0.smithi032.stdout:8/419: chown d0/db/d38 2061724284 1 2024-04-12T05:37:05.472 INFO:tasks.workunit.client.0.smithi032.stdout:6/446: readlink d4/d21/d2e/d32/l3d 0 2024-04-12T05:37:05.472 INFO:tasks.workunit.client.0.smithi032.stdout:2/440: dwrite dd/d1b/d1c/f43 [0,4194304] 0 2024-04-12T05:37:05.473 INFO:tasks.workunit.client.0.smithi032.stdout:6/447: fsync d4/d14/d1a/f6f 0 2024-04-12T05:37:05.478 INFO:tasks.workunit.client.0.smithi032.stdout:6/448: mknod d4/d14/d1a/d79/c93 0 2024-04-12T05:37:05.479 INFO:tasks.workunit.client.0.smithi032.stdout:8/420: creat d0/db/dd/d11/d1b/d22/f91 x:0 0 0 2024-04-12T05:37:05.481 INFO:tasks.workunit.client.0.smithi032.stdout:6/449: unlink d4/d14/d3c/d75/d55/f6e 0 2024-04-12T05:37:05.482 INFO:tasks.workunit.client.0.smithi032.stdout:6/450: dread - d4/d21/d27/f8e zero size 2024-04-12T05:37:05.482 INFO:tasks.workunit.client.0.smithi032.stdout:8/421: dread d0/db/d38/f61 [0,4194304] 0 2024-04-12T05:37:05.483 INFO:tasks.workunit.client.0.smithi032.stdout:2/441: link dd/d1b/d1c/d64/d36/f37 dd/d1b/d42/f84 0 2024-04-12T05:37:05.484 INFO:tasks.workunit.client.0.smithi032.stdout:8/422: write d0/db/f6b [398755,38030] 0 2024-04-12T05:37:05.484 INFO:tasks.workunit.client.0.smithi032.stdout:2/442: write dd/d1b/f73 [103004,18294] 0 2024-04-12T05:37:05.485 INFO:tasks.workunit.client.0.smithi032.stdout:6/451: mkdir d4/d21/d2e/d32/d94 0 2024-04-12T05:37:05.486 INFO:tasks.workunit.client.0.smithi032.stdout:2/443: write dd/f1e [2137308,98557] 0 2024-04-12T05:37:05.486 INFO:tasks.workunit.client.0.smithi032.stdout:2/444: fsync f1 0 2024-04-12T05:37:05.490 INFO:tasks.workunit.client.0.smithi032.stdout:8/423: creat d0/d36/d84/f92 x:0 0 0 2024-04-12T05:37:05.492 INFO:tasks.workunit.client.0.smithi032.stdout:2/445: rmdir dd/d1b/d42/d4a/d65 0 2024-04-12T05:37:05.493 INFO:tasks.workunit.client.0.smithi032.stdout:8/424: rename d0/db/dd/d11/d1b/d5f/f71 to d0/db/dd/d11/d1b/d5f/f93 0 2024-04-12T05:37:05.496 INFO:tasks.workunit.client.0.smithi032.stdout:2/446: rename dd/d19/f22 to dd/d1b/d1c/d64/d36/f85 0 2024-04-12T05:37:05.498 INFO:tasks.workunit.client.0.smithi032.stdout:2/447: write dd/d1b/d1c/f43 [2135446,36719] 0 2024-04-12T05:37:05.498 INFO:tasks.workunit.client.0.smithi032.stdout:2/448: stat dd/d19/l57 0 2024-04-12T05:37:05.500 INFO:tasks.workunit.client.0.smithi032.stdout:8/425: dwrite d0/db/f6b [0,4194304] 0 2024-04-12T05:37:05.503 INFO:tasks.workunit.client.0.smithi032.stdout:2/449: symlink dd/d1b/d1c/d64/l86 0 2024-04-12T05:37:05.504 INFO:tasks.workunit.client.0.smithi032.stdout:8/426: mknod d0/db/dd/d11/d1b/d5f/c94 0 2024-04-12T05:37:05.505 INFO:tasks.workunit.client.0.smithi032.stdout:2/450: creat dd/d1b/d1c/f87 x:0 0 0 2024-04-12T05:37:05.505 INFO:tasks.workunit.client.0.smithi032.stdout:5/500: dread d2/d3c/d48/d4b/f7c [0,4194304] 0 2024-04-12T05:37:05.507 INFO:tasks.workunit.client.0.smithi032.stdout:8/427: mknod d0/d36/c95 0 2024-04-12T05:37:05.508 INFO:tasks.workunit.client.0.smithi032.stdout:5/501: write d2/d65/d6c/f77 [344658,26281] 0 2024-04-12T05:37:05.512 INFO:tasks.workunit.client.0.smithi032.stdout:5/502: symlink d2/d65/d78/d86/l9c 0 2024-04-12T05:37:05.513 INFO:tasks.workunit.client.0.smithi032.stdout:2/451: link dd/d1b/d1c/d64/c75 dd/d1b/d42/c88 0 2024-04-12T05:37:05.513 INFO:tasks.workunit.client.0.smithi032.stdout:8/428: dread d0/f3 [0,4194304] 0 2024-04-12T05:37:05.514 INFO:tasks.workunit.client.0.smithi032.stdout:6/452: dread d4/d14/f46 [0,4194304] 0 2024-04-12T05:37:05.514 INFO:tasks.workunit.client.0.smithi032.stdout:5/503: chown d2/df/d31/f46 254790101 1 2024-04-12T05:37:05.515 INFO:tasks.workunit.client.0.smithi032.stdout:2/452: symlink dd/d19/d20/d83/l89 0 2024-04-12T05:37:05.516 INFO:tasks.workunit.client.0.smithi032.stdout:2/453: fsync dd/d1b/f34 0 2024-04-12T05:37:05.517 INFO:tasks.workunit.client.0.smithi032.stdout:8/429: mkdir d0/db/dd/d11/d1b/d96 0 2024-04-12T05:37:05.520 INFO:tasks.workunit.client.0.smithi032.stdout:6/453: truncate d4/f2c 2612259 0 2024-04-12T05:37:05.521 INFO:tasks.workunit.client.0.smithi032.stdout:5/504: dread d2/df/d25/d1e/f59 [0,4194304] 0 2024-04-12T05:37:05.523 INFO:tasks.workunit.client.0.smithi032.stdout:8/430: creat d0/db/d38/d63/d77/f97 x:0 0 0 2024-04-12T05:37:05.525 INFO:tasks.workunit.client.0.smithi032.stdout:6/454: chown d4/d14/d1a/d37/d47/d57/l67 78662736 1 2024-04-12T05:37:05.525 INFO:tasks.workunit.client.0.smithi032.stdout:5/505: chown d2/d65/d78/l90 7646 1 2024-04-12T05:37:05.528 INFO:tasks.workunit.client.0.smithi032.stdout:5/506: symlink d2/d3c/l9d 0 2024-04-12T05:37:05.528 INFO:tasks.workunit.client.0.smithi032.stdout:2/454: dread dd/d19/d20/f53 [4194304,4194304] 0 2024-04-12T05:37:05.529 INFO:tasks.workunit.client.0.smithi032.stdout:6/455: chown d4/d74/f91 10111 1 2024-04-12T05:37:05.532 INFO:tasks.workunit.client.0.smithi032.stdout:6/456: chown d4/d14/d1a/d37/c68 7 1 2024-04-12T05:37:05.532 INFO:tasks.workunit.client.0.smithi032.stdout:6/457: chown d4/d14/d3c/c4f 3151126 1 2024-04-12T05:37:05.533 INFO:tasks.workunit.client.0.smithi032.stdout:2/455: unlink dd/d19/d20/f48 0 2024-04-12T05:37:05.534 INFO:tasks.workunit.client.0.smithi032.stdout:5/507: creat d2/d65/d78/f9e x:0 0 0 2024-04-12T05:37:05.537 INFO:tasks.workunit.client.0.smithi032.stdout:2/456: write dd/d1b/d1c/d64/d36/f85 [1875936,26748] 0 2024-04-12T05:37:05.537 INFO:tasks.workunit.client.0.smithi032.stdout:5/508: creat d2/dd/d5d/f9f x:0 0 0 2024-04-12T05:37:05.538 INFO:tasks.workunit.client.0.smithi032.stdout:6/458: symlink d4/d14/d1a/d92/l95 0 2024-04-12T05:37:05.541 INFO:tasks.workunit.client.0.smithi032.stdout:5/509: symlink d2/df/d25/d50/la0 0 2024-04-12T05:37:05.542 INFO:tasks.workunit.client.0.smithi032.stdout:6/459: mknod d4/d14/d1a/d37/d69/d76/c96 0 2024-04-12T05:37:05.543 INFO:tasks.workunit.client.0.smithi032.stdout:5/510: fsync d2/df/d25/d50/f8e 0 2024-04-12T05:37:05.545 INFO:tasks.workunit.client.0.smithi032.stdout:2/457: dwrite dd/d1b/f73 [0,4194304] 0 2024-04-12T05:37:05.546 INFO:tasks.workunit.client.0.smithi032.stdout:6/460: creat d4/d14/d1a/d92/f97 x:0 0 0 2024-04-12T05:37:05.552 INFO:tasks.workunit.client.0.smithi032.stdout:0/471: sync 2024-04-12T05:37:05.552 INFO:tasks.workunit.client.0.smithi032.stdout:4/347: sync 2024-04-12T05:37:05.553 INFO:tasks.workunit.client.0.smithi032.stdout:5/511: dwrite d2/df/d25/d43/d49/f7d [0,4194304] 0 2024-04-12T05:37:05.553 INFO:tasks.workunit.client.0.smithi032.stdout:4/348: dread - d4/f3d zero size 2024-04-12T05:37:05.555 INFO:tasks.workunit.client.0.smithi032.stdout:6/461: getdents d4/d14/d1a/d37 0 2024-04-12T05:37:05.557 INFO:tasks.workunit.client.0.smithi032.stdout:0/472: readlink d2/d28/d50/d51/l78 0 2024-04-12T05:37:05.557 INFO:tasks.workunit.client.0.smithi032.stdout:6/462: dread - d4/d21/f85 zero size 2024-04-12T05:37:05.562 INFO:tasks.workunit.client.0.smithi032.stdout:0/473: rmdir d2/d28/d50 39 2024-04-12T05:37:05.563 INFO:tasks.workunit.client.0.smithi032.stdout:5/512: dread d2/d3c/d5b/f5e [0,4194304] 0 2024-04-12T05:37:05.564 INFO:tasks.workunit.client.0.smithi032.stdout:6/463: creat d4/d14/d1a/d37/f98 x:0 0 0 2024-04-12T05:37:05.568 INFO:tasks.workunit.client.0.smithi032.stdout:4/349: dwrite d4/d13/d19/d28/d2f/d3b/f6b [0,4194304] 0 2024-04-12T05:37:05.572 INFO:tasks.workunit.client.0.smithi032.stdout:8/431: dwrite d0/db/dd/d11/d1b/d22/d35/d44/f50 [0,4194304] 0 2024-04-12T05:37:05.576 INFO:tasks.workunit.client.0.smithi032.stdout:2/458: dread dd/d1b/d1c/d64/d32/d3a/f6a [0,4194304] 0 2024-04-12T05:37:05.585 INFO:tasks.workunit.client.0.smithi032.stdout:6/464: write d4/d14/d3c/d75/d55/f72 [1226250,117451] 0 2024-04-12T05:37:05.586 INFO:tasks.workunit.client.0.smithi032.stdout:0/474: truncate d2/d28/d87/f71 331784 0 2024-04-12T05:37:05.590 INFO:tasks.workunit.client.0.smithi032.stdout:4/350: dwrite d4/f1d [4194304,4194304] 0 2024-04-12T05:37:05.590 INFO:tasks.workunit.client.0.smithi032.stdout:5/513: dread d2/f24 [0,4194304] 0 2024-04-12T05:37:05.592 INFO:tasks.workunit.client.0.smithi032.stdout:6/465: mkdir d4/d14/d3c/d75/d55/d99 0 2024-04-12T05:37:05.592 INFO:tasks.workunit.client.0.smithi032.stdout:0/475: read d2/fe [6612,43653] 0 2024-04-12T05:37:05.594 INFO:tasks.workunit.client.0.smithi032.stdout:5/514: fsync d2/dd/f3a 0 2024-04-12T05:37:05.597 INFO:tasks.workunit.client.0.smithi032.stdout:4/351: creat d4/d13/d19/d28/d4d/f76 x:0 0 0 2024-04-12T05:37:05.603 INFO:tasks.workunit.client.0.smithi032.stdout:4/352: symlink d4/d13/d19/l77 0 2024-04-12T05:37:05.603 INFO:tasks.workunit.client.0.smithi032.stdout:0/476: dwrite d2/f6 [0,4194304] 0 2024-04-12T05:37:05.604 INFO:tasks.workunit.client.0.smithi032.stdout:2/459: dread dd/d19/d20/f28 [0,4194304] 0 2024-04-12T05:37:05.605 INFO:tasks.workunit.client.0.smithi032.stdout:6/466: link d4/c44 d4/d14/d3c/d75/d55/c9a 0 2024-04-12T05:37:05.607 INFO:tasks.workunit.client.0.smithi032.stdout:5/515: dwrite d2/d3c/d5b/f87 [0,4194304] 0 2024-04-12T05:37:05.607 INFO:tasks.workunit.client.0.smithi032.stdout:0/477: creat d2/d29/f9a x:0 0 0 2024-04-12T05:37:05.609 INFO:tasks.workunit.client.0.smithi032.stdout:0/478: fdatasync d2/dc/f61 0 2024-04-12T05:37:05.609 INFO:tasks.workunit.client.0.smithi032.stdout:2/460: creat dd/d19/d82/f8a x:0 0 0 2024-04-12T05:37:05.612 INFO:tasks.workunit.client.0.smithi032.stdout:5/516: creat d2/d3c/d5b/d75/fa1 x:0 0 0 2024-04-12T05:37:05.614 INFO:tasks.workunit.client.0.smithi032.stdout:2/461: write fa [1931440,14961] 0 2024-04-12T05:37:05.615 INFO:tasks.workunit.client.0.smithi032.stdout:1/367: sync 2024-04-12T05:37:05.615 INFO:tasks.workunit.client.0.smithi032.stdout:7/397: sync 2024-04-12T05:37:05.615 INFO:tasks.workunit.client.0.smithi032.stdout:3/440: sync 2024-04-12T05:37:05.622 INFO:tasks.workunit.client.0.smithi032.stdout:5/517: mkdir d2/df/d31/da2 0 2024-04-12T05:37:05.623 INFO:tasks.workunit.client.0.smithi032.stdout:6/467: link d4/c11 d4/d14/d1a/d37/d47/d57/c9b 0 2024-04-12T05:37:05.624 INFO:tasks.workunit.client.0.smithi032.stdout:1/368: creat d5/de/d17/d35/d73/f76 x:0 0 0 2024-04-12T05:37:05.625 INFO:tasks.workunit.client.0.smithi032.stdout:3/441: creat d12/d2e/d3a/d63/f8d x:0 0 0 2024-04-12T05:37:05.630 INFO:tasks.workunit.client.0.smithi032.stdout:5/518: write d2/dd/d5d/f97 [937671,125135] 0 2024-04-12T05:37:05.632 INFO:tasks.workunit.client.0.smithi032.stdout:0/479: dread d2/d8/df/f31 [0,4194304] 0 2024-04-12T05:37:05.632 INFO:tasks.workunit.client.0.smithi032.stdout:7/398: dwrite d0/dd/f55 [0,4194304] 0 2024-04-12T05:37:05.635 INFO:tasks.workunit.client.0.smithi032.stdout:2/462: dwrite dd/d1b/d1c/f81 [0,4194304] 0 2024-04-12T05:37:05.637 INFO:tasks.workunit.client.0.smithi032.stdout:7/399: write d0/f7 [4970326,129113] 0 2024-04-12T05:37:05.641 INFO:tasks.workunit.client.0.smithi032.stdout:4/353: dread d4/d13/d19/d28/d2f/d3b/f3c [0,4194304] 0 2024-04-12T05:37:05.645 INFO:tasks.workunit.client.0.smithi032.stdout:8/432: dread d0/db/dd/d11/d1b/d22/d29/f31 [0,4194304] 0 2024-04-12T05:37:05.646 INFO:tasks.workunit.client.0.smithi032.stdout:2/463: chown dd/d1b/d1c/d64/c75 570893721 1 2024-04-12T05:37:05.649 INFO:tasks.workunit.client.0.smithi032.stdout:8/433: creat d0/f98 x:0 0 0 2024-04-12T05:37:05.649 INFO:tasks.workunit.client.0.smithi032.stdout:7/400: truncate d0/de/df/f7a 832063 0 2024-04-12T05:37:05.649 INFO:tasks.workunit.client.0.smithi032.stdout:0/480: truncate d2/d28/d87/f71 458310 0 2024-04-12T05:37:05.652 INFO:tasks.workunit.client.0.smithi032.stdout:5/519: dread d2/dd/f3a [0,4194304] 0 2024-04-12T05:37:05.653 INFO:tasks.workunit.client.0.smithi032.stdout:1/369: dread d5/f23 [0,4194304] 0 2024-04-12T05:37:05.654 INFO:tasks.workunit.client.0.smithi032.stdout:4/354: rename d4/d13/d19/d28/d2f/d46 to d4/d13/d19/d28/d4d/d78 0 2024-04-12T05:37:05.655 INFO:tasks.workunit.client.0.smithi032.stdout:8/434: mknod d0/c99 0 2024-04-12T05:37:05.656 INFO:tasks.workunit.client.0.smithi032.stdout:5/520: rmdir d2 39 2024-04-12T05:37:05.657 INFO:tasks.workunit.client.0.smithi032.stdout:6/468: dread d4/d21/d2e/d32/f36 [0,4194304] 0 2024-04-12T05:37:05.658 INFO:tasks.workunit.client.0.smithi032.stdout:7/401: symlink d0/d13/d21/d30/d31/l82 0 2024-04-12T05:37:05.658 INFO:tasks.workunit.client.0.smithi032.stdout:1/370: fsync d5/de/d17/f25 0 2024-04-12T05:37:05.659 INFO:tasks.workunit.client.0.smithi032.stdout:0/481: symlink d2/d8/df/d24/l9b 0 2024-04-12T05:37:05.660 INFO:tasks.workunit.client.0.smithi032.stdout:7/402: chown d0/de 5132668 1 2024-04-12T05:37:05.661 INFO:tasks.workunit.client.0.smithi032.stdout:2/464: link dd/d1b/d1c/d64/d32/f33 dd/d1b/d1c/d64/d36/f8b 0 2024-04-12T05:37:05.662 INFO:tasks.workunit.client.0.smithi032.stdout:8/435: rmdir d0/db/dd/d11/d1b/d22/d54 39 2024-04-12T05:37:05.662 INFO:tasks.workunit.client.0.smithi032.stdout:1/371: dread - d5/de/d17/d35/d3e/f4c zero size 2024-04-12T05:37:05.663 INFO:tasks.workunit.client.0.smithi032.stdout:5/521: rmdir d2/d65/d78/d86 39 2024-04-12T05:37:05.663 INFO:tasks.workunit.client.0.smithi032.stdout:6/469: symlink d4/d14/d3c/d75/d55/d99/l9c 0 2024-04-12T05:37:05.665 INFO:tasks.workunit.client.0.smithi032.stdout:7/403: symlink d0/d13/d21/d7c/l83 0 2024-04-12T05:37:05.667 INFO:tasks.workunit.client.0.smithi032.stdout:1/372: write d5/d2c/f3c [304424,117107] 0 2024-04-12T05:37:05.668 INFO:tasks.workunit.client.0.smithi032.stdout:8/436: symlink d0/db/dd/d11/d1b/d5f/l9a 0 2024-04-12T05:37:05.668 INFO:tasks.workunit.client.0.smithi032.stdout:0/482: dread d2/d28/f2e [0,4194304] 0 2024-04-12T05:37:05.668 INFO:tasks.workunit.client.0.smithi032.stdout:5/522: write d2/d65/d6c/f71 [536573,38908] 0 2024-04-12T05:37:05.669 INFO:tasks.workunit.client.0.smithi032.stdout:2/465: write dd/d1b/d1c/d64/d32/d3a/f6a [3705787,110767] 0 2024-04-12T05:37:05.669 INFO:tasks.workunit.client.0.smithi032.stdout:4/355: dread d4/d13/d19/d28/d2f/f4e [0,4194304] 0 2024-04-12T05:37:05.671 INFO:tasks.workunit.client.0.smithi032.stdout:7/404: chown d0/d13/l19 60564 1 2024-04-12T05:37:05.673 INFO:tasks.workunit.client.0.smithi032.stdout:2/466: mkdir dd/d8c 0 2024-04-12T05:37:05.674 INFO:tasks.workunit.client.0.smithi032.stdout:1/373: chown d5/de/d17/d1b/f33 41723 1 2024-04-12T05:37:05.674 INFO:tasks.workunit.client.0.smithi032.stdout:5/523: symlink d2/dd/d6f/la3 0 2024-04-12T05:37:05.675 INFO:tasks.workunit.client.0.smithi032.stdout:4/356: read d4/d13/d44/f5a [2944819,10384] 0 2024-04-12T05:37:05.677 INFO:tasks.workunit.client.0.smithi032.stdout:8/437: write d0/db/dd/d11/d1b/f89 [4887059,39449] 0 2024-04-12T05:37:05.679 INFO:tasks.workunit.client.0.smithi032.stdout:0/483: creat d2/d8/df/d24/f9c x:0 0 0 2024-04-12T05:37:05.680 INFO:tasks.workunit.client.0.smithi032.stdout:1/374: truncate d5/de/d2a/f65 279684 0 2024-04-12T05:37:05.680 INFO:tasks.workunit.client.0.smithi032.stdout:8/438: dread - d0/db/dd/f75 zero size 2024-04-12T05:37:05.682 INFO:tasks.workunit.client.0.smithi032.stdout:6/470: dwrite d4/d21/d27/f8e [0,4194304] 0 2024-04-12T05:37:05.682 INFO:tasks.workunit.client.0.smithi032.stdout:7/405: dread d0/dd/f2a [0,4194304] 0 2024-04-12T05:37:05.683 INFO:tasks.workunit.client.0.smithi032.stdout:5/524: creat d2/d3c/d48/d4b/fa4 x:0 0 0 2024-04-12T05:37:05.684 INFO:tasks.workunit.client.0.smithi032.stdout:4/357: rmdir d4/d13/d19/d28/d32 39 2024-04-12T05:37:05.685 INFO:tasks.workunit.client.0.smithi032.stdout:7/406: write d0/d10/f60 [2751055,96374] 0 2024-04-12T05:37:05.686 INFO:tasks.workunit.client.0.smithi032.stdout:8/439: chown d0/db/dd/d11/d1b/d22/d35/l48 228406 1 2024-04-12T05:37:05.687 INFO:tasks.workunit.client.0.smithi032.stdout:1/375: mkdir d5/de/d17/d35/d61/d77 0 2024-04-12T05:37:05.691 INFO:tasks.workunit.client.0.smithi032.stdout:5/525: truncate d2/df/d25/d1e/f1f 1189567 0 2024-04-12T05:37:05.692 INFO:tasks.workunit.client.0.smithi032.stdout:4/358: fsync d4/d13/d19/d28/d32/f5f 0 2024-04-12T05:37:05.694 INFO:tasks.workunit.client.0.smithi032.stdout:4/359: chown d4/c37 268 1 2024-04-12T05:37:05.694 INFO:tasks.workunit.client.0.smithi032.stdout:8/440: mknod d0/db/d39/c9b 0 2024-04-12T05:37:05.695 INFO:tasks.workunit.client.0.smithi032.stdout:6/471: getdents d4/d14/d1a/d37/d69/d8a 0 2024-04-12T05:37:05.696 INFO:tasks.workunit.client.0.smithi032.stdout:7/407: dwrite d0/dd/f3b [4194304,4194304] 0 2024-04-12T05:37:05.696 INFO:tasks.workunit.client.0.smithi032.stdout:0/484: mkdir d2/d28/d50/d51/d83/d9d 0 2024-04-12T05:37:05.703 INFO:tasks.workunit.client.0.smithi032.stdout:2/467: write dd/d1b/d1c/d64/d36/f37 [288513,86425] 0 2024-04-12T05:37:05.704 INFO:tasks.workunit.client.0.smithi032.stdout:5/526: unlink d2/dd/c6e 0 2024-04-12T05:37:05.704 INFO:tasks.workunit.client.0.smithi032.stdout:7/408: chown d0/d13/d21/l4f 599 1 2024-04-12T05:37:05.704 INFO:tasks.workunit.client.0.smithi032.stdout:6/472: chown d4/d14/d3c/d75/d55/l6b 0 1 2024-04-12T05:37:05.704 INFO:tasks.workunit.client.0.smithi032.stdout:4/360: rename d4/d13/d19/d28/d32/f3e to d4/f79 0 2024-04-12T05:37:05.704 INFO:tasks.workunit.client.0.smithi032.stdout:8/441: mkdir d0/db/d38/d63/d77/d9c 0 2024-04-12T05:37:05.704 INFO:tasks.workunit.client.0.smithi032.stdout:1/376: dread d5/d2c/f2d [4194304,4194304] 0 2024-04-12T05:37:05.705 INFO:tasks.workunit.client.0.smithi032.stdout:0/485: unlink d2/d8/d17/f22 0 2024-04-12T05:37:05.705 INFO:tasks.workunit.client.0.smithi032.stdout:8/442: write d0/db/f55 [1776480,23042] 0 2024-04-12T05:37:05.708 INFO:tasks.workunit.client.0.smithi032.stdout:7/409: unlink d0/d34/l4a 0 2024-04-12T05:37:05.710 INFO:tasks.workunit.client.0.smithi032.stdout:4/361: getdents d4/d13/d75 0 2024-04-12T05:37:05.710 INFO:tasks.workunit.client.0.smithi032.stdout:0/486: chown d2/d8/df/d24/f5e 14515497 1 2024-04-12T05:37:05.710 INFO:tasks.workunit.client.0.smithi032.stdout:1/377: dread - d5/de/d17/d35/d3b/d49/f5e zero size 2024-04-12T05:37:05.711 INFO:tasks.workunit.client.0.smithi032.stdout:7/410: chown d0/l1 1390 1 2024-04-12T05:37:05.711 INFO:tasks.workunit.client.0.smithi032.stdout:8/443: chown d0/db/d39/l4c 6302602 1 2024-04-12T05:37:05.712 INFO:tasks.workunit.client.0.smithi032.stdout:6/473: link d4/d21/d2e/c33 d4/d14/d3c/d75/d55/d59/c9d 0 2024-04-12T05:37:05.713 INFO:tasks.workunit.client.0.smithi032.stdout:1/378: readlink d5/de/d17/d35/d3b/l53 0 2024-04-12T05:37:05.713 INFO:tasks.workunit.client.0.smithi032.stdout:5/527: dread d2/df/d31/f46 [0,4194304] 0 2024-04-12T05:37:05.714 INFO:tasks.workunit.client.0.smithi032.stdout:1/379: chown d5/de/d17/d1b 462 1 2024-04-12T05:37:05.716 INFO:tasks.workunit.client.0.smithi032.stdout:2/468: write dd/f27 [228945,19506] 0 2024-04-12T05:37:05.717 INFO:tasks.workunit.client.0.smithi032.stdout:7/411: symlink d0/d34/l84 0 2024-04-12T05:37:05.717 INFO:tasks.workunit.client.0.smithi032.stdout:0/487: read d2/d8/df/f52 [704901,4686] 0 2024-04-12T05:37:05.719 INFO:tasks.workunit.client.0.smithi032.stdout:8/444: mkdir d0/d9d 0 2024-04-12T05:37:05.719 INFO:tasks.workunit.client.0.smithi032.stdout:1/380: write d5/de/d17/f20 [8054840,121705] 0 2024-04-12T05:37:05.722 INFO:tasks.workunit.client.0.smithi032.stdout:5/528: mknod d2/df/d25/d50/ca5 0 2024-04-12T05:37:05.723 INFO:tasks.workunit.client.0.smithi032.stdout:4/362: dwrite d4/d13/d19/d28/d4d/f76 [0,4194304] 0 2024-04-12T05:37:05.724 INFO:tasks.workunit.client.0.smithi032.stdout:8/445: write d0/db/dd/d11/d1b/d22/f91 [537044,64507] 0 2024-04-12T05:37:05.727 INFO:tasks.workunit.client.0.smithi032.stdout:9/370: sync 2024-04-12T05:37:05.729 INFO:tasks.workunit.client.0.smithi032.stdout:7/412: fdatasync d0/dd/f2a 0 2024-04-12T05:37:05.730 INFO:tasks.workunit.client.0.smithi032.stdout:0/488: symlink d2/d29/l9e 0 2024-04-12T05:37:05.731 INFO:tasks.workunit.client.0.smithi032.stdout:7/413: stat d0/d13/f27 0 2024-04-12T05:37:05.734 INFO:tasks.workunit.client.0.smithi032.stdout:6/474: dwrite d4/d14/d3c/f41 [0,4194304] 0 2024-04-12T05:37:05.735 INFO:tasks.workunit.client.0.smithi032.stdout:9/371: symlink d9/d15/d1a/d24/d27/l7f 0 2024-04-12T05:37:05.738 INFO:tasks.workunit.client.0.smithi032.stdout:7/414: creat d0/d1c/d71/f85 x:0 0 0 2024-04-12T05:37:05.740 INFO:tasks.workunit.client.0.smithi032.stdout:7/415: readlink d0/d1c/l3c 0 2024-04-12T05:37:05.742 INFO:tasks.workunit.client.0.smithi032.stdout:0/489: read d2/d8/f37 [1753552,27597] 0 2024-04-12T05:37:05.743 INFO:tasks.workunit.client.0.smithi032.stdout:2/469: truncate dd/f68 159997 0 2024-04-12T05:37:05.744 INFO:tasks.workunit.client.0.smithi032.stdout:4/363: creat d4/d13/d19/f7a x:0 0 0 2024-04-12T05:37:05.746 INFO:tasks.workunit.client.0.smithi032.stdout:1/381: getdents d5/de/d17/d1b 0 2024-04-12T05:37:05.746 INFO:tasks.workunit.client.0.smithi032.stdout:8/446: truncate d0/db/f6e 1006756 0 2024-04-12T05:37:05.749 INFO:tasks.workunit.client.0.smithi032.stdout:9/372: dwrite d9/d15/d2c/f62 [0,4194304] 0 2024-04-12T05:37:05.749 INFO:tasks.workunit.client.0.smithi032.stdout:6/475: dwrite d4/d14/d3c/d75/d55/f65 [0,4194304] 0 2024-04-12T05:37:05.749 INFO:tasks.workunit.client.0.smithi032.stdout:8/447: stat d0/d72 0 2024-04-12T05:37:05.750 INFO:tasks.workunit.client.0.smithi032.stdout:7/416: creat d0/d1c/d71/f86 x:0 0 0 2024-04-12T05:37:05.751 INFO:tasks.workunit.client.0.smithi032.stdout:2/470: readlink dd/d1b/d1c/d64/l7c 0 2024-04-12T05:37:05.753 INFO:tasks.workunit.client.0.smithi032.stdout:9/373: fdatasync d9/d15/d1a/f56 0 2024-04-12T05:37:05.758 INFO:tasks.workunit.client.0.smithi032.stdout:4/364: mknod d4/d13/d19/c7b 0 2024-04-12T05:37:05.758 INFO:tasks.workunit.client.0.smithi032.stdout:6/476: fsync d4/d14/d1a/d92/f97 0 2024-04-12T05:37:05.760 INFO:tasks.workunit.client.0.smithi032.stdout:7/417: mkdir d0/d13/d21/d30/d87 0 2024-04-12T05:37:05.762 INFO:tasks.workunit.client.0.smithi032.stdout:8/448: symlink d0/db/d38/d63/d77/l9e 0 2024-04-12T05:37:05.765 INFO:tasks.workunit.client.0.smithi032.stdout:0/490: dread d2/fe [0,4194304] 0 2024-04-12T05:37:05.767 INFO:tasks.workunit.client.0.smithi032.stdout:2/471: mkdir dd/d1b/d1c/d64/d32/d3a/d8d 0 2024-04-12T05:37:05.767 INFO:tasks.workunit.client.0.smithi032.stdout:8/449: read d0/db/f4e [350791,100478] 0 2024-04-12T05:37:05.770 INFO:tasks.workunit.client.0.smithi032.stdout:6/477: mkdir d4/d14/d1a/d92/d9e 0 2024-04-12T05:37:05.770 INFO:tasks.workunit.client.0.smithi032.stdout:1/382: dwrite d5/de/d17/d35/d3e/f4c [0,4194304] 0 2024-04-12T05:37:05.770 INFO:tasks.workunit.client.0.smithi032.stdout:9/374: dwrite d9/f34 [0,4194304] 0 2024-04-12T05:37:05.771 INFO:tasks.workunit.client.0.smithi032.stdout:4/365: mkdir d4/d13/d19/d28/d2f/d7c 0 2024-04-12T05:37:05.771 INFO:tasks.workunit.client.0.smithi032.stdout:0/491: write d2/d28/d87/f85 [71072,87320] 0 2024-04-12T05:37:05.771 INFO:tasks.workunit.client.0.smithi032.stdout:2/472: write dd/d19/d20/f62 [2408352,41597] 0 2024-04-12T05:37:05.773 INFO:tasks.workunit.client.0.smithi032.stdout:7/418: creat d0/d10/d70/d7b/f88 x:0 0 0 2024-04-12T05:37:05.778 INFO:tasks.workunit.client.0.smithi032.stdout:8/450: write d0/db/dd/d11/d1b/d22/d35/f87 [182686,27250] 0 2024-04-12T05:37:05.779 INFO:tasks.workunit.client.0.smithi032.stdout:0/492: write d2/d8/df/d24/f5e [9936,121059] 0 2024-04-12T05:37:05.780 INFO:tasks.workunit.client.0.smithi032.stdout:7/419: write d0/d13/f64 [1076730,80273] 0 2024-04-12T05:37:05.781 INFO:tasks.workunit.client.0.smithi032.stdout:1/383: truncate d5/de/d17/d1b/f5c 912409 0 2024-04-12T05:37:05.783 INFO:tasks.workunit.client.0.smithi032.stdout:2/473: symlink dd/d19/d20/d83/l8e 0 2024-04-12T05:37:05.783 INFO:tasks.workunit.client.0.smithi032.stdout:6/478: creat d4/d14/d3c/d75/d55/d59/f9f x:0 0 0 2024-04-12T05:37:05.784 INFO:tasks.workunit.client.0.smithi032.stdout:9/375: mkdir d9/d15/d1a/d31/d65/d80 0 2024-04-12T05:37:05.784 INFO:tasks.workunit.client.0.smithi032.stdout:7/420: fdatasync d0/d13/f16 0 2024-04-12T05:37:05.785 INFO:tasks.workunit.client.0.smithi032.stdout:4/366: rename d4/d13/d19/c41 to d4/c7d 0 2024-04-12T05:37:05.788 INFO:tasks.workunit.client.0.smithi032.stdout:7/421: write d0/d76/f7e [4957794,83573] 0 2024-04-12T05:37:05.789 INFO:tasks.workunit.client.0.smithi032.stdout:4/367: dread d4/f38 [0,4194304] 0 2024-04-12T05:37:05.789 INFO:tasks.workunit.client.0.smithi032.stdout:7/422: stat d0/de/f12 0 2024-04-12T05:37:05.790 INFO:tasks.workunit.client.0.smithi032.stdout:1/384: read d5/de/d17/d35/d61/f50 [18527,11058] 0 2024-04-12T05:37:05.791 INFO:tasks.workunit.client.0.smithi032.stdout:8/451: mknod d0/db/dd/d11/d1b/d22/d35/d44/c9f 0 2024-04-12T05:37:05.792 INFO:tasks.workunit.client.0.smithi032.stdout:1/385: fsync d5/de/f66 0 2024-04-12T05:37:05.793 INFO:tasks.workunit.client.0.smithi032.stdout:4/368: unlink d4/d13/d19/d28/d4d/f5e 0 2024-04-12T05:37:05.793 INFO:tasks.workunit.client.0.smithi032.stdout:7/423: chown d0/de/df/f1e 3984 1 2024-04-12T05:37:05.796 INFO:tasks.workunit.client.0.smithi032.stdout:7/424: chown d0/d13/d21 185 1 2024-04-12T05:37:05.797 INFO:tasks.workunit.client.0.smithi032.stdout:6/479: dread d4/d14/d1a/f6f [0,4194304] 0 2024-04-12T05:37:05.797 INFO:tasks.workunit.client.0.smithi032.stdout:8/452: symlink d0/db/d7d/la0 0 2024-04-12T05:37:05.798 INFO:tasks.workunit.client.0.smithi032.stdout:9/376: getdents d9/d15/d1a/d1d/d5a/d6b 0 2024-04-12T05:37:05.799 INFO:tasks.workunit.client.0.smithi032.stdout:1/386: mkdir d5/de/d17/d35/d78 0 2024-04-12T05:37:05.801 INFO:tasks.workunit.client.0.smithi032.stdout:4/369: chown d4/d13/d19/d28/d2f/c50 4017818 1 2024-04-12T05:37:05.803 INFO:tasks.workunit.client.0.smithi032.stdout:2/474: dwrite dd/d19/d20/f28 [0,4194304] 0 2024-04-12T05:37:05.804 INFO:tasks.workunit.client.0.smithi032.stdout:1/387: chown d5/de/d17/d35/c42 7961 1 2024-04-12T05:37:05.810 INFO:tasks.workunit.client.0.smithi032.stdout:2/475: mknod dd/d19/d20/d83/c8f 0 2024-04-12T05:37:05.814 INFO:tasks.workunit.client.0.smithi032.stdout:6/480: link d4/d14/d1a/d37/d69/d76/c82 d4/d14/d1a/d88/ca0 0 2024-04-12T05:37:05.815 INFO:tasks.workunit.client.0.smithi032.stdout:8/453: dwrite d0/db/dd/d11/d1b/d22/d35/d44/f50 [0,4194304] 0 2024-04-12T05:37:05.818 INFO:tasks.workunit.client.0.smithi032.stdout:7/425: dread d0/de/df/f14 [0,4194304] 0 2024-04-12T05:37:05.818 INFO:tasks.workunit.client.0.smithi032.stdout:3/442: sync 2024-04-12T05:37:05.818 INFO:tasks.workunit.client.0.smithi032.stdout:9/377: write d9/d15/d2c/d41/f59 [202329,98876] 0 2024-04-12T05:37:05.824 INFO:tasks.workunit.client.0.smithi032.stdout:0/493: dread d2/d8/d17/d1b/d20/f45 [0,4194304] 0 2024-04-12T05:37:05.824 INFO:tasks.workunit.client.0.smithi032.stdout:6/481: mkdir d4/d21/d2e/da1 0 2024-04-12T05:37:05.824 INFO:tasks.workunit.client.0.smithi032.stdout:0/494: chown d2/dc/d41/l44 0 1 2024-04-12T05:37:05.824 INFO:tasks.workunit.client.0.smithi032.stdout:4/370: dread d4/d13/d19/d28/d32/f5f [0,4194304] 0 2024-04-12T05:37:05.824 INFO:tasks.workunit.client.0.smithi032.stdout:1/388: link d5/de/f1f d5/de/d17/d35/d78/f79 0 2024-04-12T05:37:05.825 INFO:tasks.workunit.client.0.smithi032.stdout:7/426: creat d0/de/df/d29/f89 x:0 0 0 2024-04-12T05:37:05.826 INFO:tasks.workunit.client.0.smithi032.stdout:6/482: fsync d4/d21/d2e/f39 0 2024-04-12T05:37:05.829 INFO:tasks.workunit.client.0.smithi032.stdout:4/371: symlink d4/d13/d19/l7e 0 2024-04-12T05:37:05.831 INFO:tasks.workunit.client.0.smithi032.stdout:6/483: write d4/d14/d1a/f49 [4961746,125626] 0 2024-04-12T05:37:05.834 INFO:tasks.workunit.client.0.smithi032.stdout:0/495: rename d2/d28/d50/l3c to d2/d8/df/l9f 0 2024-04-12T05:37:05.836 INFO:tasks.workunit.client.0.smithi032.stdout:3/443: dwrite d12/d2e/d3a/d63/f83 [0,4194304] 0 2024-04-12T05:37:05.836 INFO:tasks.workunit.client.0.smithi032.stdout:6/484: write d4/d74/f91 [804834,30684] 0 2024-04-12T05:37:05.842 INFO:tasks.workunit.client.0.smithi032.stdout:8/454: truncate d0/db/dd/d11/d1b/f6c 3301889 0 2024-04-12T05:37:05.842 INFO:tasks.workunit.client.0.smithi032.stdout:1/389: rename d5/de/d17/d35/d61/d54/f64 to d5/de/d17/d35/d78/f7a 0 2024-04-12T05:37:05.843 INFO:tasks.workunit.client.0.smithi032.stdout:6/485: chown d4/d14/f46 88 1 2024-04-12T05:37:05.847 INFO:tasks.workunit.client.0.smithi032.stdout:2/476: dwrite dd/d1b/d1c/d64/d32/d3a/f56 [0,4194304] 0 2024-04-12T05:37:05.848 INFO:tasks.workunit.client.0.smithi032.stdout:7/427: dread d0/de/df/d29/f46 [0,4194304] 0 2024-04-12T05:37:05.848 INFO:tasks.workunit.client.0.smithi032.stdout:0/496: getdents d2/dc 0 2024-04-12T05:37:05.850 INFO:tasks.workunit.client.0.smithi032.stdout:3/444: rename d12/d24/c40 to d12/d2e/c8e 0 2024-04-12T05:37:05.850 INFO:tasks.workunit.client.0.smithi032.stdout:1/390: write d5/de/d2a/f65 [379930,41163] 0 2024-04-12T05:37:05.852 INFO:tasks.workunit.client.0.smithi032.stdout:6/486: rmdir d4/d14/d3c/d75/d55/d59 39 2024-04-12T05:37:05.852 INFO:tasks.workunit.client.0.smithi032.stdout:8/455: dread d0/db/f4e [0,4194304] 0 2024-04-12T05:37:05.855 INFO:tasks.workunit.client.0.smithi032.stdout:4/372: dread d4/f18 [0,4194304] 0 2024-04-12T05:37:05.856 INFO:tasks.workunit.client.0.smithi032.stdout:7/428: rename d0/d13/f1a to d0/d1c/d71/f8a 0 2024-04-12T05:37:05.859 INFO:tasks.workunit.client.0.smithi032.stdout:7/429: write d0/d13/f4b [1476922,33027] 0 2024-04-12T05:37:05.864 INFO:tasks.workunit.client.0.smithi032.stdout:4/373: creat d4/d13/d19/d28/d2f/d6e/f7f x:0 0 0 2024-04-12T05:37:05.866 INFO:tasks.workunit.client.0.smithi032.stdout:1/391: read d5/de/d17/d35/d3b/d49/f56 [3236120,108382] 0 2024-04-12T05:37:05.867 INFO:tasks.workunit.client.0.smithi032.stdout:3/445: write d12/d2e/f1e [75986,42742] 0 2024-04-12T05:37:05.872 INFO:tasks.workunit.client.0.smithi032.stdout:0/497: dwrite d2/f14 [0,4194304] 0 2024-04-12T05:37:05.872 INFO:tasks.workunit.client.0.smithi032.stdout:7/430: creat d0/de/df/f8b x:0 0 0 2024-04-12T05:37:05.879 INFO:tasks.workunit.client.0.smithi032.stdout:3/446: creat d12/d24/f8f x:0 0 0 2024-04-12T05:37:05.879 INFO:tasks.workunit.client.0.smithi032.stdout:8/456: dwrite d0/db/dd/d11/d1b/d22/f2b [4194304,4194304] 0 2024-04-12T05:37:05.879 INFO:tasks.workunit.client.0.smithi032.stdout:7/431: creat d0/d13/f8c x:0 0 0 2024-04-12T05:37:05.879 INFO:tasks.workunit.client.0.smithi032.stdout:1/392: rename d5/d2c/c69 to d5/de/d17/d35/d3e/c7b 0 2024-04-12T05:37:05.880 INFO:tasks.workunit.client.0.smithi032.stdout:6/487: dwrite d4/d14/d3c/d75/d55/f70 [0,4194304] 0 2024-04-12T05:37:05.880 INFO:tasks.workunit.client.0.smithi032.stdout:4/374: truncate d4/d13/d19/d28/d2f/d6e/f7f 446256 0 2024-04-12T05:37:05.883 INFO:tasks.workunit.client.0.smithi032.stdout:7/432: chown d0/d1c 6 1 2024-04-12T05:37:05.887 INFO:tasks.workunit.client.0.smithi032.stdout:6/488: fsync d4/d21/f2f 0 2024-04-12T05:37:05.889 INFO:tasks.workunit.client.0.smithi032.stdout:7/433: truncate d0/d13/f8c 674614 0 2024-04-12T05:37:05.890 INFO:tasks.workunit.client.0.smithi032.stdout:4/375: readlink d4/d13/d19/l27 0 2024-04-12T05:37:05.890 INFO:tasks.workunit.client.0.smithi032.stdout:1/393: creat d5/de/d17/d35/d78/f7c x:0 0 0 2024-04-12T05:37:05.890 INFO:tasks.workunit.client.0.smithi032.stdout:6/489: fsync d4/d74/f91 0 2024-04-12T05:37:05.891 INFO:tasks.workunit.client.0.smithi032.stdout:3/447: rename d12/d24/c75 to d12/d2e/d3a/d63/c90 0 2024-04-12T05:37:05.893 INFO:tasks.workunit.client.0.smithi032.stdout:1/394: chown d5/de/d17/d35/d3b/d49/f62 531518928 1 2024-04-12T05:37:05.895 INFO:tasks.workunit.client.0.smithi032.stdout:8/457: rename d0/db/dd/d11/d1b/d22/f45 to d0/db/d38/d63/fa1 0 2024-04-12T05:37:05.896 INFO:tasks.workunit.client.0.smithi032.stdout:3/448: creat d12/d2f/f91 x:0 0 0 2024-04-12T05:37:05.899 INFO:tasks.workunit.client.0.smithi032.stdout:1/395: rmdir d5/de/d5b 39 2024-04-12T05:37:05.901 INFO:tasks.workunit.client.0.smithi032.stdout:6/490: mknod d4/d14/d1a/d88/d8d/ca2 0 2024-04-12T05:37:05.902 INFO:tasks.workunit.client.0.smithi032.stdout:3/449: symlink d12/l92 0 2024-04-12T05:37:05.904 INFO:tasks.workunit.client.0.smithi032.stdout:7/434: getdents d0 0 2024-04-12T05:37:05.905 INFO:tasks.workunit.client.0.smithi032.stdout:6/491: creat d4/d74/fa3 x:0 0 0 2024-04-12T05:37:05.906 INFO:tasks.workunit.client.0.smithi032.stdout:4/376: dread d4/f54 [0,4194304] 0 2024-04-12T05:37:05.906 INFO:tasks.workunit.client.0.smithi032.stdout:1/396: getdents d5/de/d17/d35/d73/d74 0 2024-04-12T05:37:05.907 INFO:tasks.workunit.client.0.smithi032.stdout:7/435: creat d0/d34/f8d x:0 0 0 2024-04-12T05:37:05.908 INFO:tasks.workunit.client.0.smithi032.stdout:3/450: link d12/d2f/d5b/l8b d12/d24/l93 0 2024-04-12T05:37:05.908 INFO:tasks.workunit.client.0.smithi032.stdout:1/397: readlink d5/de/d17/d35/d3b/l53 0 2024-04-12T05:37:05.908 INFO:tasks.workunit.client.0.smithi032.stdout:7/436: fsync d0/d1c/d71/f85 0 2024-04-12T05:37:05.909 INFO:tasks.workunit.client.0.smithi032.stdout:4/377: rmdir d4/d13/d19/d28/d4d 39 2024-04-12T05:37:05.909 INFO:tasks.workunit.client.0.smithi032.stdout:0/498: dread d2/d28/d50/f35 [0,4194304] 0 2024-04-12T05:37:05.909 INFO:tasks.workunit.client.0.smithi032.stdout:3/451: fsync d12/d2e/f36 0 2024-04-12T05:37:05.910 INFO:tasks.workunit.client.0.smithi032.stdout:6/492: symlink d4/d14/d1a/d37/d47/la4 0 2024-04-12T05:37:05.912 INFO:tasks.workunit.client.0.smithi032.stdout:0/499: symlink d2/la0 0 2024-04-12T05:37:05.913 INFO:tasks.workunit.client.0.smithi032.stdout:7/437: symlink d0/d13/d21/d81/l8e 0 2024-04-12T05:37:05.915 INFO:tasks.workunit.client.0.smithi032.stdout:6/493: chown d4/c90 931612567 1 2024-04-12T05:37:05.917 INFO:tasks.workunit.client.0.smithi032.stdout:7/438: unlink d0/d1c/l48 0 2024-04-12T05:37:05.918 INFO:tasks.workunit.client.0.smithi032.stdout:6/494: write d4/d21/f3b [985474,101677] 0 2024-04-12T05:37:05.919 INFO:tasks.workunit.client.0.smithi032.stdout:0/500: dread d2/d28/d87/f85 [0,4194304] 0 2024-04-12T05:37:05.921 INFO:tasks.workunit.client.0.smithi032.stdout:3/452: dwrite d12/d2e/d3a/d63/f8a [0,4194304] 0 2024-04-12T05:37:05.922 INFO:tasks.workunit.client.0.smithi032.stdout:4/378: dread d4/f79 [0,4194304] 0 2024-04-12T05:37:05.923 INFO:tasks.workunit.client.0.smithi032.stdout:6/495: rmdir d4/d21/d27 39 2024-04-12T05:37:05.927 INFO:tasks.workunit.client.0.smithi032.stdout:0/501: truncate d2/d28/d87/d79/f7c 1386967 0 2024-04-12T05:37:05.930 INFO:tasks.workunit.client.0.smithi032.stdout:7/439: dread d0/de/df/d29/f46 [0,4194304] 0 2024-04-12T05:37:05.931 INFO:tasks.workunit.client.0.smithi032.stdout:4/379: unlink d4/d13/d19/d28/d4d/d78/l61 0 2024-04-12T05:37:05.931 INFO:tasks.workunit.client.0.smithi032.stdout:0/502: chown d2/dc/d41/c59 272009258 1 2024-04-12T05:37:05.932 INFO:tasks.workunit.client.0.smithi032.stdout:7/440: stat d0/d76 0 2024-04-12T05:37:05.934 INFO:tasks.workunit.client.0.smithi032.stdout:3/453: dwrite d12/d2f/f81 [0,4194304] 0 2024-04-12T05:37:05.934 INFO:tasks.workunit.client.0.smithi032.stdout:0/503: chown d2/d29/d77/c7b 82090 1 2024-04-12T05:37:05.935 INFO:tasks.workunit.client.0.smithi032.stdout:6/496: dread d4/d14/f46 [0,4194304] 0 2024-04-12T05:37:05.936 INFO:tasks.workunit.client.0.smithi032.stdout:4/380: creat d4/d13/d19/f80 x:0 0 0 2024-04-12T05:37:05.938 INFO:tasks.workunit.client.0.smithi032.stdout:7/441: rename d0/dd/f55 to d0/d10/d70/f8f 0 2024-04-12T05:37:05.939 INFO:tasks.workunit.client.0.smithi032.stdout:5/529: sync 2024-04-12T05:37:05.940 INFO:tasks.workunit.client.0.smithi032.stdout:0/504: creat d2/d28/d50/fa1 x:0 0 0 2024-04-12T05:37:05.941 INFO:tasks.workunit.client.0.smithi032.stdout:5/530: dread d2/df/f5c [0,4194304] 0 2024-04-12T05:37:05.942 INFO:tasks.workunit.client.0.smithi032.stdout:3/454: link d12/d2e/f42 d12/d2e/f94 0 2024-04-12T05:37:05.942 INFO:tasks.workunit.client.0.smithi032.stdout:7/442: symlink d0/de/df/l90 0 2024-04-12T05:37:05.944 INFO:tasks.workunit.client.0.smithi032.stdout:7/443: read d0/dd/f3b [6428865,77675] 0 2024-04-12T05:37:05.944 INFO:tasks.workunit.client.0.smithi032.stdout:3/455: mknod d12/d2f/c95 0 2024-04-12T05:37:05.946 INFO:tasks.workunit.client.0.smithi032.stdout:0/505: symlink d2/d8/df/la2 0 2024-04-12T05:37:05.946 INFO:tasks.workunit.client.0.smithi032.stdout:3/456: chown d12/d2e/d3a/d2d 1705 1 2024-04-12T05:37:05.947 INFO:tasks.workunit.client.0.smithi032.stdout:6/497: rename d4/d21/l34 to d4/d14/d3c/d75/d55/la5 0 2024-04-12T05:37:05.949 INFO:tasks.workunit.client.0.smithi032.stdout:3/457: write d12/d24/f47 [137548,44732] 0 2024-04-12T05:37:05.949 INFO:tasks.workunit.client.0.smithi032.stdout:4/381: rename d4/d13/d19/d28 to d4/d13/d19/d28/d4d/d78/d81 22 2024-04-12T05:37:05.951 INFO:tasks.workunit.client.0.smithi032.stdout:5/531: dwrite d2/df/d25/d50/f76 [0,4194304] 0 2024-04-12T05:37:05.952 INFO:tasks.workunit.client.0.smithi032.stdout:0/506: truncate d2/d28/d50/d51/f8d 189797 0 2024-04-12T05:37:05.954 INFO:tasks.workunit.client.0.smithi032.stdout:3/458: creat d12/d2e/d3a/f96 x:0 0 0 2024-04-12T05:37:05.954 INFO:tasks.workunit.client.0.smithi032.stdout:5/532: readlink d2/d3c/d5b/l3e 0 2024-04-12T05:37:05.959 INFO:tasks.workunit.client.0.smithi032.stdout:4/382: rename d4/d13/d19/d28/d2f/f73 to d4/d13/d19/d28/d2f/d7c/f82 0 2024-04-12T05:37:05.960 INFO:tasks.workunit.client.0.smithi032.stdout:6/498: dwrite d4/d74/f91 [0,4194304] 0 2024-04-12T05:37:05.961 INFO:tasks.workunit.client.0.smithi032.stdout:4/383: chown d4/d13/f1c 2822434 1 2024-04-12T05:37:05.962 INFO:tasks.workunit.client.0.smithi032.stdout:0/507: unlink d2/d8/df/f52 0 2024-04-12T05:37:05.963 INFO:tasks.workunit.client.0.smithi032.stdout:5/533: symlink d2/d3c/la6 0 2024-04-12T05:37:05.964 INFO:tasks.workunit.client.0.smithi032.stdout:3/459: creat d12/d2e/d3a/f97 x:0 0 0 2024-04-12T05:37:05.966 INFO:tasks.workunit.client.0.smithi032.stdout:3/460: chown d12/d2e/f82 2016 1 2024-04-12T05:37:05.967 INFO:tasks.workunit.client.0.smithi032.stdout:6/499: mknod d4/d14/d1a/d37/d69/ca6 0 2024-04-12T05:37:05.968 INFO:tasks.workunit.client.0.smithi032.stdout:4/384: write d4/f18 [5155757,38456] 0 2024-04-12T05:37:05.972 INFO:tasks.workunit.client.0.smithi032.stdout:3/461: mkdir d12/d2e/d98 0 2024-04-12T05:37:05.972 INFO:tasks.workunit.client.0.smithi032.stdout:7/444: dwrite d0/de/df/d20/f6b [0,4194304] 0 2024-04-12T05:37:05.974 INFO:tasks.workunit.client.0.smithi032.stdout:4/385: write d4/d13/d19/d28/d32/f5f [1787870,97964] 0 2024-04-12T05:37:05.975 INFO:tasks.workunit.client.0.smithi032.stdout:3/462: chown d12/d24/c54 46 1 2024-04-12T05:37:05.975 INFO:tasks.workunit.client.0.smithi032.stdout:6/500: mknod d4/d14/d3c/d75/d55/d99/ca7 0 2024-04-12T05:37:05.978 INFO:tasks.workunit.client.0.smithi032.stdout:7/445: truncate d0/d1c/d71/f85 163696 0 2024-04-12T05:37:05.979 INFO:tasks.workunit.client.0.smithi032.stdout:0/508: dwrite d2/d8/df/f31 [0,4194304] 0 2024-04-12T05:37:05.982 INFO:tasks.workunit.client.0.smithi032.stdout:7/446: mknod d0/d1c/d71/c91 0 2024-04-12T05:37:05.991 INFO:tasks.workunit.client.0.smithi032.stdout:3/463: rename d12/d2e/d3a/f97 to d12/d2e/f99 0 2024-04-12T05:37:05.991 INFO:tasks.workunit.client.0.smithi032.stdout:4/386: rename d4/d13/d19/d28/d4d/f76 to d4/d13/d19/d28/d4d/d78/f83 0 2024-04-12T05:37:05.991 INFO:tasks.workunit.client.0.smithi032.stdout:6/501: dwrite d4/d14/d25/f7e [0,4194304] 0 2024-04-12T05:37:05.993 INFO:tasks.workunit.client.0.smithi032.stdout:3/464: truncate d12/d2e/f36 3685522 0 2024-04-12T05:37:05.996 INFO:tasks.workunit.client.0.smithi032.stdout:0/509: rename d2/d8/d17/d1b/f97 to d2/d29/d8a/d99/fa3 0 2024-04-12T05:37:05.997 INFO:tasks.workunit.client.0.smithi032.stdout:4/387: mknod d4/d13/d19/d28/c84 0 2024-04-12T05:37:06.002 INFO:tasks.workunit.client.0.smithi032.stdout:3/465: rmdir d12/d2e/d3a 39 2024-04-12T05:37:06.002 INFO:tasks.workunit.client.0.smithi032.stdout:0/510: dread d2/d8/df/d24/f5c [0,4194304] 0 2024-04-12T05:37:06.003 INFO:tasks.workunit.client.0.smithi032.stdout:0/511: dread - d2/d28/d50/fa1 zero size 2024-04-12T05:37:06.004 INFO:tasks.workunit.client.0.smithi032.stdout:6/502: truncate d4/f2c 2896421 0 2024-04-12T05:37:06.004 INFO:tasks.workunit.client.0.smithi032.stdout:0/512: stat d2/d8/df/d24 0 2024-04-12T05:37:06.005 INFO:tasks.workunit.client.0.smithi032.stdout:0/513: write d2/d29/f65 [351372,97613] 0 2024-04-12T05:37:06.006 INFO:tasks.workunit.client.0.smithi032.stdout:4/388: creat d4/f85 x:0 0 0 2024-04-12T05:37:06.009 INFO:tasks.workunit.client.0.smithi032.stdout:7/447: dread d0/d10/f62 [0,4194304] 0 2024-04-12T05:37:06.012 INFO:tasks.workunit.client.0.smithi032.stdout:0/514: write d2/d29/d8a/d99/fa3 [3981186,105614] 0 2024-04-12T05:37:06.013 INFO:tasks.workunit.client.0.smithi032.stdout:3/466: creat d12/d2e/d98/f9a x:0 0 0 2024-04-12T05:37:06.015 INFO:tasks.workunit.client.0.smithi032.stdout:6/503: write d4/d14/f6d [353521,32727] 0 2024-04-12T05:37:06.015 INFO:tasks.workunit.client.0.smithi032.stdout:4/389: mknod d4/d13/d19/d28/d68/c86 0 2024-04-12T05:37:06.017 INFO:tasks.workunit.client.0.smithi032.stdout:0/515: mknod d2/d8/df/d24/d60/ca4 0 2024-04-12T05:37:06.017 INFO:tasks.workunit.client.0.smithi032.stdout:7/448: dwrite d0/de/df/f24 [0,4194304] 0 2024-04-12T05:37:06.019 INFO:tasks.workunit.client.0.smithi032.stdout:7/449: stat d0/d13/f8c 0 2024-04-12T05:37:06.022 INFO:tasks.workunit.client.0.smithi032.stdout:6/504: dread d4/d14/d3c/f48 [0,4194304] 0 2024-04-12T05:37:06.025 INFO:tasks.workunit.client.0.smithi032.stdout:7/450: dread d0/de/df/f14 [0,4194304] 0 2024-04-12T05:37:06.025 INFO:tasks.workunit.client.0.smithi032.stdout:3/467: dread d12/d2e/f67 [0,4194304] 0 2024-04-12T05:37:06.025 INFO:tasks.workunit.client.0.smithi032.stdout:0/516: getdents d2/d28/d50 0 2024-04-12T05:37:06.025 INFO:tasks.workunit.client.0.smithi032.stdout:6/505: symlink d4/d14/d1a/d37/d69/la8 0 2024-04-12T05:37:06.027 INFO:tasks.workunit.client.0.smithi032.stdout:0/517: chown d2/d8/df/d24/d60/c5b 1970 1 2024-04-12T05:37:06.028 INFO:tasks.workunit.client.0.smithi032.stdout:3/468: dread d12/d2f/f57 [0,4194304] 0 2024-04-12T05:37:06.028 INFO:tasks.workunit.client.0.smithi032.stdout:7/451: symlink d0/d13/d21/l92 0 2024-04-12T05:37:06.029 INFO:tasks.workunit.client.0.smithi032.stdout:6/506: creat d4/d21/d2e/d32/d94/fa9 x:0 0 0 2024-04-12T05:37:06.031 INFO:tasks.workunit.client.0.smithi032.stdout:3/469: rename d12/d2f/l51 to d12/d24/l9b 0 2024-04-12T05:37:06.033 INFO:tasks.workunit.client.0.smithi032.stdout:3/470: fdatasync d12/d2e/f99 0 2024-04-12T05:37:06.034 INFO:tasks.workunit.client.0.smithi032.stdout:3/471: stat d12/d2e/l53 0 2024-04-12T05:37:06.035 INFO:tasks.workunit.client.0.smithi032.stdout:0/518: dwrite d2/d8/df/d24/f5e [0,4194304] 0 2024-04-12T05:37:06.040 INFO:tasks.workunit.client.0.smithi032.stdout:3/472: dwrite d12/d2f/f81 [0,4194304] 0 2024-04-12T05:37:06.041 INFO:tasks.workunit.client.0.smithi032.stdout:0/519: link d2/d8/df/f5f d2/d8/d17/d64/fa5 0 2024-04-12T05:37:06.041 INFO:tasks.workunit.client.0.smithi032.stdout:0/520: readlink d2/d28/l8f 0 2024-04-12T05:37:06.044 INFO:tasks.workunit.client.0.smithi032.stdout:3/473: rename d12/d2e/l33 to d12/d2f/d89/l9c 0 2024-04-12T05:37:06.045 INFO:tasks.workunit.client.0.smithi032.stdout:0/521: creat d2/dc/fa6 x:0 0 0 2024-04-12T05:37:06.045 INFO:tasks.workunit.client.0.smithi032.stdout:3/474: chown d12/d2e/c68 901750 1 2024-04-12T05:37:06.045 INFO:tasks.workunit.client.0.smithi032.stdout:0/522: chown d2/d8/d17/d64/fa5 129191 1 2024-04-12T05:37:06.047 INFO:tasks.workunit.client.0.smithi032.stdout:0/523: mknod d2/d28/d87/ca7 0 2024-04-12T05:37:06.048 INFO:tasks.workunit.client.0.smithi032.stdout:3/475: truncate d12/d2e/f36 2352421 0 2024-04-12T05:37:06.049 INFO:tasks.workunit.client.0.smithi032.stdout:3/476: fsync d12/d2e/d3a/d63/f8d 0 2024-04-12T05:37:06.052 INFO:tasks.workunit.client.0.smithi032.stdout:3/477: unlink d12/l17 0 2024-04-12T05:37:06.053 INFO:tasks.workunit.client.0.smithi032.stdout:3/478: write d12/d2f/d5b/f66 [321277,41809] 0 2024-04-12T05:37:06.053 INFO:tasks.workunit.client.0.smithi032.stdout:3/479: stat d12/d2e/c7c 0 2024-04-12T05:37:06.058 INFO:tasks.workunit.client.0.smithi032.stdout:3/480: dwrite d12/d2f/f41 [0,4194304] 0 2024-04-12T05:37:06.061 INFO:tasks.workunit.client.0.smithi032.stdout:9/378: sync 2024-04-12T05:37:06.073 INFO:tasks.workunit.client.0.smithi032.stdout:9/379: dread d9/d15/d1a/d24/f40 [8388608,4194304] 0 2024-04-12T05:37:06.076 INFO:tasks.workunit.client.0.smithi032.stdout:9/380: symlink d9/d15/d1a/d24/d27/l81 0 2024-04-12T05:37:06.079 INFO:tasks.workunit.client.0.smithi032.stdout:9/381: unlink d9/d15/d2c/d41/f51 0 2024-04-12T05:37:06.080 INFO:tasks.workunit.client.0.smithi032.stdout:9/382: dread - d9/d15/f6a zero size 2024-04-12T05:37:06.082 INFO:tasks.workunit.client.0.smithi032.stdout:9/383: unlink d9/c3a 0 2024-04-12T05:37:06.103 INFO:tasks.workunit.client.0.smithi032.stdout:9/384: dread d9/d32/f74 [0,4194304] 0 2024-04-12T05:37:06.104 INFO:tasks.workunit.client.0.smithi032.stdout:9/385: stat d9/d1f/c70 0 2024-04-12T05:37:06.106 INFO:tasks.workunit.client.0.smithi032.stdout:9/386: symlink d9/d15/d2c/l82 0 2024-04-12T05:37:06.109 INFO:tasks.workunit.client.0.smithi032.stdout:9/387: getdents d9/d15/d1a/d1d 0 2024-04-12T05:37:06.110 INFO:tasks.workunit.client.0.smithi032.stdout:9/388: mkdir d9/d15/d1a/d1d/d5a/d6b/d83 0 2024-04-12T05:37:06.111 INFO:tasks.workunit.client.0.smithi032.stdout:9/389: chown d9/d15/d2c/f77 4209833 1 2024-04-12T05:37:06.115 INFO:tasks.workunit.client.0.smithi032.stdout:9/390: dwrite d9/d15/f28 [0,4194304] 0 2024-04-12T05:37:06.120 INFO:tasks.workunit.client.0.smithi032.stdout:9/391: link d9/d15/d1a/d24/f47 d9/d1f/f84 0 2024-04-12T05:37:06.141 INFO:tasks.workunit.client.0.smithi032.stdout:4/390: dread d4/d13/d44/f5a [0,4194304] 0 2024-04-12T05:37:06.144 INFO:tasks.workunit.client.0.smithi032.stdout:7/452: truncate d0/de/df/f24 987194 0 2024-04-12T05:37:06.145 INFO:tasks.workunit.client.0.smithi032.stdout:6/507: truncate d4/d21/d2e/d32/f36 4064768 0 2024-04-12T05:37:06.146 INFO:tasks.workunit.client.0.smithi032.stdout:6/508: write d4/f1b [1280202,17739] 0 2024-04-12T05:37:06.147 INFO:tasks.workunit.client.0.smithi032.stdout:7/453: creat d0/dd/d51/f93 x:0 0 0 2024-04-12T05:37:06.149 INFO:tasks.workunit.client.0.smithi032.stdout:7/454: write d0/de/df/d29/f89 [851130,115279] 0 2024-04-12T05:37:06.156 INFO:tasks.workunit.client.0.smithi032.stdout:6/509: unlink d4/d21/d27/f38 0 2024-04-12T05:37:06.157 INFO:tasks.workunit.client.0.smithi032.stdout:7/455: getdents d0/d13/d21/d7c 0 2024-04-12T05:37:06.158 INFO:tasks.workunit.client.0.smithi032.stdout:4/391: dread d4/d13/f1c [0,4194304] 0 2024-04-12T05:37:06.158 INFO:tasks.workunit.client.0.smithi032.stdout:6/510: truncate d4/d14/d1a/f49 5539607 0 2024-04-12T05:37:06.159 INFO:tasks.workunit.client.0.smithi032.stdout:9/392: getdents d9/d15/d2c 0 2024-04-12T05:37:06.160 INFO:tasks.workunit.client.0.smithi032.stdout:7/456: readlink d0/d1c/d4c/l2d 0 2024-04-12T05:37:06.160 INFO:tasks.workunit.client.0.smithi032.stdout:6/511: fsync d4/d21/d2e/d32/f61 0 2024-04-12T05:37:06.161 INFO:tasks.workunit.client.0.smithi032.stdout:4/392: mkdir d4/d87 0 2024-04-12T05:37:06.161 INFO:tasks.workunit.client.0.smithi032.stdout:7/457: fdatasync d0/de/df/f1e 0 2024-04-12T05:37:06.163 INFO:tasks.workunit.client.0.smithi032.stdout:9/393: symlink d9/d15/d1a/d1d/d5a/d6b/l85 0 2024-04-12T05:37:06.163 INFO:tasks.workunit.client.0.smithi032.stdout:7/458: write d0/d34/f8d [502562,113148] 0 2024-04-12T05:37:06.166 INFO:tasks.workunit.client.0.smithi032.stdout:9/394: rmdir d9/d15 39 2024-04-12T05:37:06.169 INFO:tasks.workunit.client.0.smithi032.stdout:7/459: rename d0/d76/f52 to d0/d1c/d71/f94 0 2024-04-12T05:37:06.170 INFO:tasks.workunit.client.0.smithi032.stdout:6/512: getdents d4/d14/d1a/d37/d69 0 2024-04-12T05:37:06.170 INFO:tasks.workunit.client.0.smithi032.stdout:4/393: dwrite d4/d13/d44/f6a [0,4194304] 0 2024-04-12T05:37:06.172 INFO:tasks.workunit.client.0.smithi032.stdout:9/395: mkdir d9/d32/d86 0 2024-04-12T05:37:06.172 INFO:tasks.workunit.client.0.smithi032.stdout:7/460: write d0/d1c/d4c/f4d [1994372,96140] 0 2024-04-12T05:37:06.175 INFO:tasks.workunit.client.0.smithi032.stdout:6/513: write d4/d21/d2e/f39 [1779437,35840] 0 2024-04-12T05:37:06.176 INFO:tasks.workunit.client.0.smithi032.stdout:7/461: symlink d0/d34/l95 0 2024-04-12T05:37:06.178 INFO:tasks.workunit.client.0.smithi032.stdout:6/514: dread - d4/d14/d1a/d92/f97 zero size 2024-04-12T05:37:06.181 INFO:tasks.workunit.client.0.smithi032.stdout:2/477: sync 2024-04-12T05:37:06.181 INFO:tasks.workunit.client.0.smithi032.stdout:4/394: dwrite d4/d13/f1c [0,4194304] 0 2024-04-12T05:37:06.181 INFO:tasks.workunit.client.0.smithi032.stdout:6/515: mknod d4/d14/d1a/d37/d47/d57/caa 0 2024-04-12T05:37:06.186 INFO:tasks.workunit.client.0.smithi032.stdout:4/395: unlink d4/f38 0 2024-04-12T05:37:06.194 INFO:tasks.workunit.client.0.smithi032.stdout:9/396: dwrite d9/d15/d2c/d41/f7e [0,4194304] 0 2024-04-12T05:37:06.194 INFO:tasks.workunit.client.0.smithi032.stdout:6/516: link d4/d21/c7f d4/d14/d3c/cab 0 2024-04-12T05:37:06.196 INFO:tasks.workunit.client.0.smithi032.stdout:2/478: dwrite dd/d1b/d1c/d64/d32/d3a/f6a [4194304,4194304] 0 2024-04-12T05:37:06.200 INFO:tasks.workunit.client.0.smithi032.stdout:2/479: fsync dd/d1b/d1c/f87 0 2024-04-12T05:37:06.204 INFO:tasks.workunit.client.0.smithi032.stdout:4/396: dwrite d4/d13/d19/d28/d2f/d7c/f82 [0,4194304] 0 2024-04-12T05:37:06.208 INFO:tasks.workunit.client.0.smithi032.stdout:2/480: mknod dd/d8c/c90 0 2024-04-12T05:37:06.209 INFO:tasks.workunit.client.0.smithi032.stdout:6/517: dwrite d4/d21/d2e/d32/f7a [0,4194304] 0 2024-04-12T05:37:06.210 INFO:tasks.workunit.client.0.smithi032.stdout:6/518: chown d4/d14 3 1 2024-04-12T05:37:06.213 INFO:tasks.workunit.client.0.smithi032.stdout:2/481: symlink dd/d19/d20/d83/l91 0 2024-04-12T05:37:06.213 INFO:tasks.workunit.client.0.smithi032.stdout:4/397: creat d4/d13/d19/d28/d2f/d3b/f88 x:0 0 0 2024-04-12T05:37:06.214 INFO:tasks.workunit.client.0.smithi032.stdout:6/519: creat d4/d14/d1a/d88/fac x:0 0 0 2024-04-12T05:37:06.216 INFO:tasks.workunit.client.0.smithi032.stdout:6/520: fsync d4/d14/d3c/f48 0 2024-04-12T05:37:06.221 INFO:tasks.workunit.client.0.smithi032.stdout:6/521: getdents d4/d14/d1a/d79 0 2024-04-12T05:37:06.225 INFO:tasks.workunit.client.0.smithi032.stdout:6/522: rename d4/d14/d3c/d75 to d4/d21/d27/dad 0 2024-04-12T05:37:06.226 INFO:tasks.workunit.client.0.smithi032.stdout:6/523: symlink d4/d21/d27/lae 0 2024-04-12T05:37:06.228 INFO:tasks.workunit.client.0.smithi032.stdout:6/524: chown d4/d21/d27/dad/d55/d59/f9f 1775 1 2024-04-12T05:37:06.229 INFO:tasks.workunit.client.0.smithi032.stdout:6/525: truncate d4/d74/fa3 1023055 0 2024-04-12T05:37:06.231 INFO:tasks.workunit.client.0.smithi032.stdout:6/526: creat d4/d21/d27/faf x:0 0 0 2024-04-12T05:37:06.232 INFO:tasks.workunit.client.0.smithi032.stdout:6/527: readlink d4/d21/d27/dad/d55/la5 0 2024-04-12T05:37:06.232 INFO:tasks.workunit.client.0.smithi032.stdout:6/528: readlink d4/d14/d25/l2b 0 2024-04-12T05:37:06.234 INFO:tasks.workunit.client.0.smithi032.stdout:6/529: symlink d4/d14/d1a/d79/lb0 0 2024-04-12T05:37:06.238 INFO:tasks.workunit.client.0.smithi032.stdout:6/530: dread d4/d21/d2e/d32/f36 [0,4194304] 0 2024-04-12T05:37:06.266 INFO:tasks.workunit.client.0.smithi032.stdout:8/458: sync 2024-04-12T05:37:06.280 INFO:tasks.workunit.client.0.smithi032.stdout:2/482: truncate dd/f68 2673036 0 2024-04-12T05:37:06.280 INFO:tasks.workunit.client.0.smithi032.stdout:9/397: dread d9/d1f/f5d [0,4194304] 0 2024-04-12T05:37:06.281 INFO:tasks.workunit.client.0.smithi032.stdout:8/459: chown d0/db/dd/d11/d1b/d22/d29/l7b 111461 1 2024-04-12T05:37:06.282 INFO:tasks.workunit.client.0.smithi032.stdout:6/531: creat d4/d14/d1a/d37/d69/d76/fb1 x:0 0 0 2024-04-12T05:37:06.283 INFO:tasks.workunit.client.0.smithi032.stdout:6/532: readlink d4/d21/d27/dad/d55/la5 0 2024-04-12T05:37:06.284 INFO:tasks.workunit.client.0.smithi032.stdout:2/483: fdatasync dd/d1b/d1c/f29 0 2024-04-12T05:37:06.284 INFO:tasks.workunit.client.0.smithi032.stdout:9/398: fsync d9/f13 0 2024-04-12T05:37:06.285 INFO:tasks.workunit.client.0.smithi032.stdout:8/460: creat d0/db/dd/d11/d1b/d1e/fa2 x:0 0 0 2024-04-12T05:37:06.289 INFO:tasks.workunit.client.0.smithi032.stdout:6/533: unlink d4/d14/d1a/d37/d47/c8f 0 2024-04-12T05:37:06.289 INFO:tasks.workunit.client.0.smithi032.stdout:2/484: fsync dd/d19/d20/f53 0 2024-04-12T05:37:06.292 INFO:tasks.workunit.client.0.smithi032.stdout:9/399: mkdir d9/d15/d1a/d31/d65/d80/d87 0 2024-04-12T05:37:06.293 INFO:tasks.workunit.client.0.smithi032.stdout:8/461: dwrite d0/db/dd/d11/d1b/d22/d35/d44/f50 [4194304,4194304] 0 2024-04-12T05:37:06.294 INFO:tasks.workunit.client.0.smithi032.stdout:6/534: mkdir d4/d14/d1a/d92/d9e/db2 0 2024-04-12T05:37:06.295 INFO:tasks.workunit.client.0.smithi032.stdout:2/485: mkdir dd/d1b/d1c/d64/d92 0 2024-04-12T05:37:06.296 INFO:tasks.workunit.client.0.smithi032.stdout:6/535: truncate d4/d14/d1a/d92/f97 74710 0 2024-04-12T05:37:06.296 INFO:tasks.workunit.client.0.smithi032.stdout:8/462: rename d0/db/dd/d11/d1b/d22/d51/l8b to d0/d72/la3 0 2024-04-12T05:37:06.297 INFO:tasks.workunit.client.0.smithi032.stdout:6/536: read d4/f9 [924178,41917] 0 2024-04-12T05:37:06.307 INFO:tasks.workunit.client.0.smithi032.stdout:8/463: dwrite d0/db/dd/d11/d1b/d22/d54/f64 [4194304,4194304] 0 2024-04-12T05:37:06.311 INFO:tasks.workunit.client.0.smithi032.stdout:8/464: mkdir d0/db/d38/d63/d77/d9c/da4 0 2024-04-12T05:37:06.312 INFO:tasks.workunit.client.0.smithi032.stdout:6/537: dread d4/d21/d27/f8e [0,4194304] 0 2024-04-12T05:37:06.313 INFO:tasks.workunit.client.0.smithi032.stdout:8/465: symlink d0/db/d7d/la5 0 2024-04-12T05:37:06.315 INFO:tasks.workunit.client.0.smithi032.stdout:6/538: creat d4/d14/d3c/fb3 x:0 0 0 2024-04-12T05:37:06.316 INFO:tasks.workunit.client.0.smithi032.stdout:8/466: unlink d0/db/d39/l4c 0 2024-04-12T05:37:06.316 INFO:tasks.workunit.client.0.smithi032.stdout:6/539: read d4/d21/d2e/d32/f7a [2592922,9236] 0 2024-04-12T05:37:06.316 INFO:tasks.workunit.client.0.smithi032.stdout:2/486: truncate dd/d1b/f73 1530675 0 2024-04-12T05:37:06.317 INFO:tasks.workunit.client.0.smithi032.stdout:2/487: chown dd/f16 141 1 2024-04-12T05:37:06.319 INFO:tasks.workunit.client.0.smithi032.stdout:8/467: symlink d0/db/d38/la6 0 2024-04-12T05:37:06.319 INFO:tasks.workunit.client.0.smithi032.stdout:2/488: truncate dd/d19/d20/f62 4557889 0 2024-04-12T05:37:06.320 INFO:tasks.workunit.client.0.smithi032.stdout:6/540: mkdir d4/d21/db4 0 2024-04-12T05:37:06.325 INFO:tasks.workunit.client.0.smithi032.stdout:8/468: dwrite d0/db/f6b [0,4194304] 0 2024-04-12T05:37:06.327 INFO:tasks.workunit.client.0.smithi032.stdout:6/541: truncate d4/d21/d2e/d32/f7a 3039591 0 2024-04-12T05:37:06.333 INFO:tasks.workunit.client.0.smithi032.stdout:8/469: rename d0/db/dd/c53 to d0/db/ca7 0 2024-04-12T05:37:06.334 INFO:tasks.workunit.client.0.smithi032.stdout:2/489: dwrite f9 [0,4194304] 0 2024-04-12T05:37:06.339 INFO:tasks.workunit.client.0.smithi032.stdout:6/542: dread f3 [0,4194304] 0 2024-04-12T05:37:06.342 INFO:tasks.workunit.client.0.smithi032.stdout:8/470: rmdir d0/d9d 0 2024-04-12T05:37:06.343 INFO:tasks.workunit.client.0.smithi032.stdout:8/471: write d0/db/dd/d11/f1f [3127343,73316] 0 2024-04-12T05:37:06.345 INFO:tasks.workunit.client.0.smithi032.stdout:6/543: creat d4/d14/d3c/fb5 x:0 0 0 2024-04-12T05:37:06.346 INFO:tasks.workunit.client.0.smithi032.stdout:6/544: fsync d4/d14/d25/f7e 0 2024-04-12T05:37:06.347 INFO:tasks.workunit.client.0.smithi032.stdout:8/472: mkdir d0/db/dd/d11/d1b/d22/d54/da8 0 2024-04-12T05:37:06.349 INFO:tasks.workunit.client.0.smithi032.stdout:6/545: truncate d4/f9 1255279 0 2024-04-12T05:37:06.350 INFO:tasks.workunit.client.0.smithi032.stdout:8/473: creat d0/db/dd/d11/d1b/d22/d51/fa9 x:0 0 0 2024-04-12T05:37:06.353 INFO:tasks.workunit.client.0.smithi032.stdout:8/474: creat d0/d72/faa x:0 0 0 2024-04-12T05:37:06.353 INFO:tasks.workunit.client.0.smithi032.stdout:2/490: truncate f8 1908921 0 2024-04-12T05:37:06.355 INFO:tasks.workunit.client.0.smithi032.stdout:6/546: dwrite d4/f5 [0,4194304] 0 2024-04-12T05:37:06.360 INFO:tasks.workunit.client.0.smithi032.stdout:6/547: unlink d4/d14/d1a/d37/d47/l5e 0 2024-04-12T05:37:06.362 INFO:tasks.workunit.client.0.smithi032.stdout:8/475: link d0/db/dd/d11/f2c d0/db/d38/d63/d77/d9c/fab 0 2024-04-12T05:37:06.362 INFO:tasks.workunit.client.0.smithi032.stdout:1/398: sync 2024-04-12T05:37:06.362 INFO:tasks.workunit.client.0.smithi032.stdout:5/534: sync 2024-04-12T05:37:06.363 INFO:tasks.workunit.client.0.smithi032.stdout:8/476: chown d0/db/c68 421 1 2024-04-12T05:37:06.364 INFO:tasks.workunit.client.0.smithi032.stdout:6/548: creat d4/d21/d2e/fb6 x:0 0 0 2024-04-12T05:37:06.368 INFO:tasks.workunit.client.0.smithi032.stdout:1/399: write d5/de/d2a/f68 [1285185,82015] 0 2024-04-12T05:37:06.368 INFO:tasks.workunit.client.0.smithi032.stdout:6/549: fsync d4/d14/d1a/d37/f5f 0 2024-04-12T05:37:06.370 INFO:tasks.workunit.client.0.smithi032.stdout:1/400: stat d5/de/d17/l22 0 2024-04-12T05:37:06.371 INFO:tasks.workunit.client.0.smithi032.stdout:6/550: mknod d4/d21/d27/cb7 0 2024-04-12T05:37:06.371 INFO:tasks.workunit.client.0.smithi032.stdout:5/535: dwrite d2/d3c/d5b/f87 [0,4194304] 0 2024-04-12T05:37:06.373 INFO:tasks.workunit.client.0.smithi032.stdout:6/551: rmdir d4 39 2024-04-12T05:37:06.375 INFO:tasks.workunit.client.0.smithi032.stdout:2/491: dread f8 [0,4194304] 0 2024-04-12T05:37:06.377 INFO:tasks.workunit.client.0.smithi032.stdout:5/536: rename d2/l2b to d2/df/d31/da2/la7 0 2024-04-12T05:37:06.380 INFO:tasks.workunit.client.0.smithi032.stdout:6/552: mknod d4/d14/d1a/d37/d69/d76/cb8 0 2024-04-12T05:37:06.380 INFO:tasks.workunit.client.0.smithi032.stdout:6/553: fsync d4/d21/d2e/f39 0 2024-04-12T05:37:06.381 INFO:tasks.workunit.client.0.smithi032.stdout:2/492: fdatasync dd/d1b/d1c/f76 0 2024-04-12T05:37:06.382 INFO:tasks.workunit.client.0.smithi032.stdout:6/554: chown d4/d14/d1a/d37 65257 1 2024-04-12T05:37:06.385 INFO:tasks.workunit.client.0.smithi032.stdout:6/555: creat d4/d14/d1a/d37/d47/d57/fb9 x:0 0 0 2024-04-12T05:37:06.386 INFO:tasks.workunit.client.0.smithi032.stdout:2/493: getdents dd/d1b/d1c/d64/d32/d6d 0 2024-04-12T05:37:06.387 INFO:tasks.workunit.client.0.smithi032.stdout:6/556: creat d4/d21/d2e/d32/fba x:0 0 0 2024-04-12T05:37:06.389 INFO:tasks.workunit.client.0.smithi032.stdout:2/494: unlink dd/d1b/d1c/d64/d36/c4f 0 2024-04-12T05:37:06.390 INFO:tasks.workunit.client.0.smithi032.stdout:2/495: write dd/d1b/d1c/d64/f7d [292686,59894] 0 2024-04-12T05:37:06.390 INFO:tasks.workunit.client.0.smithi032.stdout:6/557: rmdir d4/d21/d27/dad/d55/d59 39 2024-04-12T05:37:06.391 INFO:tasks.workunit.client.0.smithi032.stdout:6/558: fsync d4/d21/d27/dad/d55/f70 0 2024-04-12T05:37:06.391 INFO:tasks.workunit.client.0.smithi032.stdout:2/496: write dd/d1b/d1c/f43 [1604199,99823] 0 2024-04-12T05:37:06.392 INFO:tasks.workunit.client.0.smithi032.stdout:6/559: stat d4/d14/d1a/d88 0 2024-04-12T05:37:06.393 INFO:tasks.workunit.client.0.smithi032.stdout:6/560: chown d4/d21/c4e 30264 1 2024-04-12T05:37:06.395 INFO:tasks.workunit.client.0.smithi032.stdout:6/561: dread d4/d14/d1a/d92/f97 [0,4194304] 0 2024-04-12T05:37:06.398 INFO:tasks.workunit.client.0.smithi032.stdout:2/497: truncate dd/d1b/d1c/d64/d32/f35 4412659 0 2024-04-12T05:37:06.400 INFO:tasks.workunit.client.0.smithi032.stdout:2/498: write dd/d1b/d1c/f74 [1468250,50073] 0 2024-04-12T05:37:06.401 INFO:tasks.workunit.client.0.smithi032.stdout:6/562: dwrite d4/d21/d27/dad/d55/f70 [0,4194304] 0 2024-04-12T05:37:06.402 INFO:tasks.workunit.client.0.smithi032.stdout:6/563: chown d4/d21/d27/lae 1 1 2024-04-12T05:37:06.404 INFO:tasks.workunit.client.0.smithi032.stdout:6/564: read - d4/d14/d1a/d37/d69/d76/fb1 zero size 2024-04-12T05:37:06.407 INFO:tasks.workunit.client.0.smithi032.stdout:6/565: symlink d4/d14/d1a/d37/lbb 0 2024-04-12T05:37:06.407 INFO:tasks.workunit.client.0.smithi032.stdout:2/499: dread dd/d1b/d42/f84 [0,4194304] 0 2024-04-12T05:37:06.408 INFO:tasks.workunit.client.0.smithi032.stdout:6/566: read - d4/d21/d2e/d32/d94/fa9 zero size 2024-04-12T05:37:06.416 INFO:tasks.workunit.client.0.smithi032.stdout:2/500: dwrite dd/d19/d20/f62 [4194304,4194304] 0 2024-04-12T05:37:06.416 INFO:tasks.workunit.client.0.smithi032.stdout:6/567: dwrite d4/d14/d3c/f48 [0,4194304] 0 2024-04-12T05:37:06.418 INFO:tasks.workunit.client.0.smithi032.stdout:6/568: write d4/d14/d1a/d37/d47/f56 [371916,73941] 0 2024-04-12T05:37:06.423 INFO:tasks.workunit.client.0.smithi032.stdout:2/501: mknod dd/d1b/d42/c93 0 2024-04-12T05:37:06.424 INFO:tasks.workunit.client.0.smithi032.stdout:6/569: symlink d4/d74/lbc 0 2024-04-12T05:37:06.431 INFO:tasks.workunit.client.0.smithi032.stdout:6/570: link d4/d21/d27/dad/d55/d99/ca7 d4/cbd 0 2024-04-12T05:37:06.433 INFO:tasks.workunit.client.0.smithi032.stdout:6/571: symlink d4/d21/lbe 0 2024-04-12T05:37:06.438 INFO:tasks.workunit.client.0.smithi032.stdout:2/502: dread dd/d1b/d1c/d64/d36/f8b [4194304,4194304] 0 2024-04-12T05:37:06.439 INFO:tasks.workunit.client.0.smithi032.stdout:6/572: dwrite d4/d14/d1a/d37/f98 [0,4194304] 0 2024-04-12T05:37:06.443 INFO:tasks.workunit.client.0.smithi032.stdout:2/503: mknod dd/d19/d82/c94 0 2024-04-12T05:37:06.448 INFO:tasks.workunit.client.0.smithi032.stdout:2/504: rename dd/d1b/d1c/d64/c6e to dd/d1b/d1c/d64/d92/c95 0 2024-04-12T05:37:06.452 INFO:tasks.workunit.client.0.smithi032.stdout:2/505: link dd/d1b/d1c/d64/d36/f85 dd/d1b/d1c/d64/f96 0 2024-04-12T05:37:06.453 INFO:tasks.workunit.client.0.smithi032.stdout:2/506: chown dd/d1b/d42/c93 52299 1 2024-04-12T05:37:06.454 INFO:tasks.workunit.client.0.smithi032.stdout:2/507: truncate dd/d19/d20/f28 5099843 0 2024-04-12T05:37:06.456 INFO:tasks.workunit.client.0.smithi032.stdout:2/508: symlink dd/d1b/d1c/l97 0 2024-04-12T05:37:06.457 INFO:tasks.workunit.client.0.smithi032.stdout:2/509: truncate dd/d1b/d42/f84 1523675 0 2024-04-12T05:37:06.458 INFO:tasks.workunit.client.0.smithi032.stdout:2/510: stat dd/d1b/d1c/d64/d36/c71 0 2024-04-12T05:37:06.459 INFO:tasks.workunit.client.0.smithi032.stdout:2/511: write dd/d1b/d1c/f74 [3938580,70147] 0 2024-04-12T05:37:06.462 INFO:tasks.workunit.client.0.smithi032.stdout:6/573: dread d4/d21/d2e/d32/f61 [0,4194304] 0 2024-04-12T05:37:06.464 INFO:tasks.workunit.client.0.smithi032.stdout:6/574: creat d4/d21/d2e/fbf x:0 0 0 2024-04-12T05:37:06.465 INFO:tasks.workunit.client.0.smithi032.stdout:6/575: dread - d4/d14/d1a/d37/d47/d57/fb9 zero size 2024-04-12T05:37:06.467 INFO:tasks.workunit.client.0.smithi032.stdout:2/512: truncate dd/f27 496306 0 2024-04-12T05:37:06.467 INFO:tasks.workunit.client.0.smithi032.stdout:6/576: creat d4/d14/d1a/d88/fc0 x:0 0 0 2024-04-12T05:37:06.469 INFO:tasks.workunit.client.0.smithi032.stdout:6/577: creat d4/d14/fc1 x:0 0 0 2024-04-12T05:37:06.471 INFO:tasks.workunit.client.0.smithi032.stdout:2/513: rename dd/d1b/f34 to dd/d1b/d1c/d64/f98 0 2024-04-12T05:37:06.471 INFO:tasks.workunit.client.0.smithi032.stdout:6/578: creat d4/d74/fc2 x:0 0 0 2024-04-12T05:37:06.472 INFO:tasks.workunit.client.0.smithi032.stdout:2/514: mknod dd/d1b/d1c/d64/d92/c99 0 2024-04-12T05:37:06.473 INFO:tasks.workunit.client.0.smithi032.stdout:6/579: symlink d4/d21/d2e/lc3 0 2024-04-12T05:37:06.474 INFO:tasks.workunit.client.0.smithi032.stdout:2/515: fsync dd/d1b/d1c/f76 0 2024-04-12T05:37:06.474 INFO:tasks.workunit.client.0.smithi032.stdout:6/580: write d4/d14/d1a/d37/f5d [3366849,18780] 0 2024-04-12T05:37:06.477 INFO:tasks.workunit.client.0.smithi032.stdout:2/516: chown dd/d1b/c5d 122 1 2024-04-12T05:37:06.479 INFO:tasks.workunit.client.0.smithi032.stdout:6/581: chown d4/cbd 555640375 1 2024-04-12T05:37:06.483 INFO:tasks.workunit.client.0.smithi032.stdout:6/582: dread d4/d14/d1a/f6f [0,4194304] 0 2024-04-12T05:37:06.489 INFO:tasks.workunit.client.0.smithi032.stdout:2/517: dwrite dd/d1b/d1c/f76 [0,4194304] 0 2024-04-12T05:37:06.490 INFO:tasks.workunit.client.0.smithi032.stdout:6/583: dwrite d4/d21/d2e/d32/f61 [0,4194304] 0 2024-04-12T05:37:06.493 INFO:tasks.workunit.client.0.smithi032.stdout:2/518: write dd/d1b/d1c/d64/f98 [719225,46576] 0 2024-04-12T05:37:06.498 INFO:tasks.workunit.client.0.smithi032.stdout:6/584: dwrite d4/d21/f85 [0,4194304] 0 2024-04-12T05:37:06.500 INFO:tasks.workunit.client.0.smithi032.stdout:2/519: symlink dd/d1b/d1c/d64/d32/d3a/d8d/l9a 0 2024-04-12T05:37:06.503 INFO:tasks.workunit.client.0.smithi032.stdout:0/524: sync 2024-04-12T05:37:06.503 INFO:tasks.workunit.client.0.smithi032.stdout:3/481: sync 2024-04-12T05:37:06.509 INFO:tasks.workunit.client.0.smithi032.stdout:0/525: symlink d2/d29/d8a/d99/la8 0 2024-04-12T05:37:06.510 INFO:tasks.workunit.client.0.smithi032.stdout:0/526: chown d2/d8/d17/f2b 529 1 2024-04-12T05:37:06.511 INFO:tasks.workunit.client.0.smithi032.stdout:3/482: getdents d12/d2f/d5b 0 2024-04-12T05:37:06.512 INFO:tasks.workunit.client.0.smithi032.stdout:3/483: write d12/d2e/d98/f9a [457808,62477] 0 2024-04-12T05:37:06.513 INFO:tasks.workunit.client.0.smithi032.stdout:0/527: symlink d2/d28/d87/d79/la9 0 2024-04-12T05:37:06.513 INFO:tasks.workunit.client.0.smithi032.stdout:2/520: fsync dd/d1b/d1c/f76 0 2024-04-12T05:37:06.514 INFO:tasks.workunit.client.0.smithi032.stdout:2/521: fdatasync f1 0 2024-04-12T05:37:06.514 INFO:tasks.workunit.client.0.smithi032.stdout:3/484: fsync d12/f56 0 2024-04-12T05:37:06.515 INFO:tasks.workunit.client.0.smithi032.stdout:0/528: symlink d2/d29/d8a/laa 0 2024-04-12T05:37:06.515 INFO:tasks.workunit.client.0.smithi032.stdout:2/522: stat dd/f1e 0 2024-04-12T05:37:06.516 INFO:tasks.workunit.client.0.smithi032.stdout:0/529: truncate d2/dc/fa6 996541 0 2024-04-12T05:37:06.516 INFO:tasks.workunit.client.0.smithi032.stdout:3/485: mkdir d12/d2f/d89/d9d 0 2024-04-12T05:37:06.517 INFO:tasks.workunit.client.0.smithi032.stdout:2/523: creat dd/d8c/f9b x:0 0 0 2024-04-12T05:37:06.518 INFO:tasks.workunit.client.0.smithi032.stdout:0/530: creat d2/d28/d87/d79/fab x:0 0 0 2024-04-12T05:37:06.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:06 smithi067 ceph-mon[49010]: pgmap v264: 65 pgs: 65 active+clean; 689 MiB data, 3.7 GiB used, 533 GiB / 536 GiB avail; 5.9 MiB/s rd, 63 MiB/s wr, 206 op/s 2024-04-12T05:37:06.520 INFO:tasks.workunit.client.0.smithi032.stdout:2/524: creat dd/d19/d20/d83/f9c x:0 0 0 2024-04-12T05:37:06.521 INFO:tasks.workunit.client.0.smithi032.stdout:0/531: rename d2/d28/d50 to d2/dc/d3e/dac 0 2024-04-12T05:37:06.526 INFO:tasks.workunit.client.0.smithi032.stdout:2/525: rename dd/d1b to dd/d19/d9d 0 2024-04-12T05:37:06.527 INFO:tasks.workunit.client.0.smithi032.stdout:6/585: dwrite d4/d14/d1a/d92/f97 [0,4194304] 0 2024-04-12T05:37:06.529 INFO:tasks.workunit.client.0.smithi032.stdout:3/486: write d12/d2e/f42 [44906,119477] 0 2024-04-12T05:37:06.530 INFO:tasks.workunit.client.0.smithi032.stdout:2/526: symlink dd/d19/d9d/d1c/d64/d36/l9e 0 2024-04-12T05:37:06.535 INFO:tasks.workunit.client.0.smithi032.stdout:0/532: getdents d2/dc 0 2024-04-12T05:37:06.536 INFO:tasks.workunit.client.0.smithi032.stdout:3/487: symlink d12/l9e 0 2024-04-12T05:37:06.536 INFO:tasks.workunit.client.0.smithi032.stdout:6/586: dwrite d4/d14/d1a/f31 [0,4194304] 0 2024-04-12T05:37:06.537 INFO:tasks.workunit.client.0.smithi032.stdout:0/533: truncate d2/d8/f88 767611 0 2024-04-12T05:37:06.537 INFO:tasks.workunit.client.0.smithi032.stdout:3/488: chown d12/l88 973 1 2024-04-12T05:37:06.538 INFO:tasks.workunit.client.0.smithi032.stdout:2/527: rename dd/d19/d9d/f6c to dd/d19/d9d/d1c/d64/d32/f9f 0 2024-04-12T05:37:06.539 INFO:tasks.workunit.client.0.smithi032.stdout:2/528: stat dd/d19/d9d/d1c/f76 0 2024-04-12T05:37:06.541 INFO:tasks.workunit.client.0.smithi032.stdout:6/587: mkdir d4/d14/d1a/dc4 0 2024-04-12T05:37:06.543 INFO:tasks.workunit.client.0.smithi032.stdout:0/534: mknod d2/dc/d3e/dac/d51/d83/cad 0 2024-04-12T05:37:06.544 INFO:tasks.workunit.client.0.smithi032.stdout:6/588: symlink d4/d14/d3c/lc5 0 2024-04-12T05:37:06.545 INFO:tasks.workunit.client.0.smithi032.stdout:3/489: creat d12/f9f x:0 0 0 2024-04-12T05:37:06.545 INFO:tasks.workunit.client.0.smithi032.stdout:6/589: write d4/d74/fc2 [898185,1946] 0 2024-04-12T05:37:06.546 INFO:tasks.workunit.client.0.smithi032.stdout:3/490: dread - d12/d2e/f86 zero size 2024-04-12T05:37:06.547 INFO:tasks.workunit.client.0.smithi032.stdout:2/529: creat dd/d19/d20/fa0 x:0 0 0 2024-04-12T05:37:06.549 INFO:tasks.workunit.client.0.smithi032.stdout:0/535: mknod d2/dc/d3e/dac/d51/d83/d9d/cae 0 2024-04-12T05:37:06.549 INFO:tasks.workunit.client.0.smithi032.stdout:2/530: chown dd/d19/d20/d83/c8f 8933849 1 2024-04-12T05:37:06.549 INFO:tasks.workunit.client.0.smithi032.stdout:3/491: symlink d12/d2e/d3a/d63/la0 0 2024-04-12T05:37:06.551 INFO:tasks.workunit.client.0.smithi032.stdout:3/492: chown d12/d2e/f50 72928 1 2024-04-12T05:37:06.551 INFO:tasks.workunit.client.0.smithi032.stdout:6/590: dread d4/d21/d2e/d32/f61 [0,4194304] 0 2024-04-12T05:37:06.552 INFO:tasks.workunit.client.0.smithi032.stdout:4/398: sync 2024-04-12T05:37:06.552 INFO:tasks.workunit.client.0.smithi032.stdout:7/462: sync 2024-04-12T05:37:06.552 INFO:tasks.workunit.client.0.smithi032.stdout:9/400: sync 2024-04-12T05:37:06.554 INFO:tasks.workunit.client.0.smithi032.stdout:2/531: getdents dd 0 2024-04-12T05:37:06.555 INFO:tasks.workunit.client.0.smithi032.stdout:6/591: unlink d4/lb 0 2024-04-12T05:37:06.555 INFO:tasks.workunit.client.0.smithi032.stdout:4/399: write d4/d13/f43 [59523,129729] 0 2024-04-12T05:37:06.556 INFO:tasks.workunit.client.0.smithi032.stdout:9/401: stat d9/l3c 0 2024-04-12T05:37:06.558 INFO:tasks.workunit.client.0.smithi032.stdout:7/463: creat d0/dd/d51/d65/f96 x:0 0 0 2024-04-12T05:37:06.558 INFO:tasks.workunit.client.0.smithi032.stdout:2/532: mkdir dd/d19/d9d/d42/d4a/da1 0 2024-04-12T05:37:06.559 INFO:tasks.workunit.client.0.smithi032.stdout:0/536: dwrite d2/d29/f47 [0,4194304] 0 2024-04-12T05:37:06.559 INFO:tasks.workunit.client.0.smithi032.stdout:4/400: mknod d4/d13/d19/d28/d32/c89 0 2024-04-12T05:37:06.560 INFO:tasks.workunit.client.0.smithi032.stdout:2/533: stat dd/d19/d9d/d1c/f74 0 2024-04-12T05:37:06.561 INFO:tasks.workunit.client.0.smithi032.stdout:9/402: readlink d9/d15/d1a/d24/d27/l4f 0 2024-04-12T05:37:06.565 INFO:tasks.workunit.client.0.smithi032.stdout:4/401: mknod d4/d87/c8a 0 2024-04-12T05:37:06.565 INFO:tasks.workunit.client.0.smithi032.stdout:0/537: fdatasync d2/d8/d17/d64/fa5 0 2024-04-12T05:37:06.566 INFO:tasks.workunit.client.0.smithi032.stdout:0/538: chown d2/dc 1725 1 2024-04-12T05:37:06.569 INFO:tasks.workunit.client.0.smithi032.stdout:2/534: creat dd/d19/d9d/d1c/d64/d32/d3a/d8d/fa2 x:0 0 0 2024-04-12T05:37:06.571 INFO:tasks.workunit.client.0.smithi032.stdout:7/464: dwrite d0/dd/d51/d65/f96 [0,4194304] 0 2024-04-12T05:37:06.574 INFO:tasks.workunit.client.0.smithi032.stdout:0/539: rename d2/d29/d77/c7b to d2/d29/caf 0 2024-04-12T05:37:06.575 INFO:tasks.workunit.client.0.smithi032.stdout:4/402: truncate d4/d13/d19/d28/d4d/d78/f64 95083 0 2024-04-12T05:37:06.576 INFO:tasks.workunit.client.0.smithi032.stdout:7/465: chown d0/de/df/d29/f89 129 1 2024-04-12T05:37:06.577 INFO:tasks.workunit.client.0.smithi032.stdout:2/535: unlink dd/d19/d9d/d1c/d64/d36/f37 0 2024-04-12T05:37:06.577 INFO:tasks.workunit.client.0.smithi032.stdout:0/540: fsync d2/d29/f47 0 2024-04-12T05:37:06.577 INFO:tasks.workunit.client.0.smithi032.stdout:9/403: dwrite d9/d15/d1a/d1d/d5a/f61 [0,4194304] 0 2024-04-12T05:37:06.580 INFO:tasks.workunit.client.0.smithi032.stdout:9/404: fdatasync d9/d15/d2c/f10 0 2024-04-12T05:37:06.582 INFO:tasks.workunit.client.0.smithi032.stdout:7/466: chown d0/d13/l18 18608 1 2024-04-12T05:37:06.589 INFO:tasks.workunit.client.0.smithi032.stdout:0/541: creat d2/d8/df/d24/d60/fb0 x:0 0 0 2024-04-12T05:37:06.589 INFO:tasks.workunit.client.0.smithi032.stdout:2/536: mknod dd/d19/d9d/d42/d4a/da1/ca3 0 2024-04-12T05:37:06.589 INFO:tasks.workunit.client.0.smithi032.stdout:7/467: dread - d0/de/df/f8b zero size 2024-04-12T05:37:06.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:06 smithi032 ceph-mon[39731]: pgmap v264: 65 pgs: 65 active+clean; 689 MiB data, 3.7 GiB used, 533 GiB / 536 GiB avail; 5.9 MiB/s rd, 63 MiB/s wr, 206 op/s 2024-04-12T05:37:06.590 INFO:tasks.workunit.client.0.smithi032.stdout:4/403: dread d4/d13/f1c [0,4194304] 0 2024-04-12T05:37:06.594 INFO:tasks.workunit.client.0.smithi032.stdout:4/404: stat d4/l66 0 2024-04-12T05:37:06.597 INFO:tasks.workunit.client.0.smithi032.stdout:7/468: dread d0/de/df/f3f [0,4194304] 0 2024-04-12T05:37:06.603 INFO:tasks.workunit.client.0.smithi032.stdout:2/537: dread dd/f68 [0,4194304] 0 2024-04-12T05:37:06.603 INFO:tasks.workunit.client.0.smithi032.stdout:7/469: write d0/d1c/d4c/f5e [849325,129106] 0 2024-04-12T05:37:06.603 INFO:tasks.workunit.client.0.smithi032.stdout:2/538: fsync dd/d19/d9d/d1c/d64/d32/f33 0 2024-04-12T05:37:06.603 INFO:tasks.workunit.client.0.smithi032.stdout:9/405: truncate d9/d15/d2c/d41/f59 1157747 0 2024-04-12T05:37:06.603 INFO:tasks.workunit.client.0.smithi032.stdout:9/406: stat d9/d15/f48 0 2024-04-12T05:37:06.605 INFO:tasks.workunit.client.0.smithi032.stdout:2/539: symlink dd/d19/la4 0 2024-04-12T05:37:06.607 INFO:tasks.workunit.client.0.smithi032.stdout:0/542: dwrite d2/fe [0,4194304] 0 2024-04-12T05:37:06.609 INFO:tasks.workunit.client.0.smithi032.stdout:9/407: mkdir d9/d15/d1a/d24/d27/d88 0 2024-04-12T05:37:06.611 INFO:tasks.workunit.client.0.smithi032.stdout:7/470: dwrite d0/d13/d21/f57 [0,4194304] 0 2024-04-12T05:37:06.612 INFO:tasks.workunit.client.0.smithi032.stdout:0/543: write d2/d8/df/f31 [3763447,8142] 0 2024-04-12T05:37:06.614 INFO:tasks.workunit.client.0.smithi032.stdout:0/544: dread - d2/d8/d17/f6a zero size 2024-04-12T05:37:06.614 INFO:tasks.workunit.client.0.smithi032.stdout:2/540: read dd/d19/d9d/d1c/d64/d32/f47 [408437,40232] 0 2024-04-12T05:37:06.616 INFO:tasks.workunit.client.0.smithi032.stdout:9/408: mknod d9/d15/d1a/c89 0 2024-04-12T05:37:06.618 INFO:tasks.workunit.client.0.smithi032.stdout:7/471: read d0/de/f12 [2098878,45415] 0 2024-04-12T05:37:06.620 INFO:tasks.workunit.client.0.smithi032.stdout:0/545: rename d2/d28/d87/f85 to d2/d28/d87/d90/fb1 0 2024-04-12T05:37:06.621 INFO:tasks.workunit.client.0.smithi032.stdout:2/541: mkdir dd/d19/d9d/d1c/d64/d32/d6d/da5 0 2024-04-12T05:37:06.623 INFO:tasks.workunit.client.0.smithi032.stdout:2/542: fsync dd/d19/d9d/d1c/d64/f7d 0 2024-04-12T05:37:06.624 INFO:tasks.workunit.client.0.smithi032.stdout:0/546: getdents d2/d8 0 2024-04-12T05:37:06.626 INFO:tasks.workunit.client.0.smithi032.stdout:2/543: creat dd/d19/d9d/d1c/fa6 x:0 0 0 2024-04-12T05:37:06.628 INFO:tasks.workunit.client.0.smithi032.stdout:0/547: link d2/d29/d77/c86 d2/dc/d3e/dac/d51/cb2 0 2024-04-12T05:37:06.630 INFO:tasks.workunit.client.0.smithi032.stdout:2/544: mkdir dd/d19/d9d/d42/da7 0 2024-04-12T05:37:06.630 INFO:tasks.workunit.client.0.smithi032.stdout:7/472: dread d0/de/df/d29/f7d [0,4194304] 0 2024-04-12T05:37:06.631 INFO:tasks.workunit.client.0.smithi032.stdout:0/548: dread - d2/d8/d17/d1b/f7a zero size 2024-04-12T05:37:06.632 INFO:tasks.workunit.client.0.smithi032.stdout:7/473: mknod d0/d76/c97 0 2024-04-12T05:37:06.632 INFO:tasks.workunit.client.0.smithi032.stdout:0/549: write d2/dc/fa6 [1272510,66663] 0 2024-04-12T05:37:06.633 INFO:tasks.workunit.client.0.smithi032.stdout:9/409: write d9/d15/d1a/d24/d27/f2d [2121416,84287] 0 2024-04-12T05:37:06.634 INFO:tasks.workunit.client.0.smithi032.stdout:9/410: dread - d9/d15/d1a/d1d/f6f zero size 2024-04-12T05:37:06.634 INFO:tasks.workunit.client.0.smithi032.stdout:2/545: creat dd/d19/d9d/d1c/d64/d32/d3a/fa8 x:0 0 0 2024-04-12T05:37:06.635 INFO:tasks.workunit.client.0.smithi032.stdout:7/474: creat d0/dd/f98 x:0 0 0 2024-04-12T05:37:06.637 INFO:tasks.workunit.client.0.smithi032.stdout:9/411: rmdir d9 39 2024-04-12T05:37:06.639 INFO:tasks.workunit.client.0.smithi032.stdout:0/550: symlink d2/d8/d17/lb3 0 2024-04-12T05:37:06.643 INFO:tasks.workunit.client.0.smithi032.stdout:7/475: dwrite d0/d10/d70/d7b/f88 [0,4194304] 0 2024-04-12T05:37:06.644 INFO:tasks.workunit.client.0.smithi032.stdout:0/551: creat d2/dc/d3e/dac/d51/d83/fb4 x:0 0 0 2024-04-12T05:37:06.646 INFO:tasks.workunit.client.0.smithi032.stdout:9/412: rename d9/d15/d1a/d24/l3b to d9/d15/d1a/d24/l8a 0 2024-04-12T05:37:06.646 INFO:tasks.workunit.client.0.smithi032.stdout:2/546: write dd/d19/d9d/d1c/d64/d32/f35 [794257,38744] 0 2024-04-12T05:37:06.647 INFO:tasks.workunit.client.0.smithi032.stdout:0/552: mkdir d2/d29/d77/db5 0 2024-04-12T05:37:06.649 INFO:tasks.workunit.client.0.smithi032.stdout:0/553: dread - d2/d8/d17/f6a zero size 2024-04-12T05:37:06.651 INFO:tasks.workunit.client.0.smithi032.stdout:9/413: write d9/d15/d2c/f14 [1030937,26642] 0 2024-04-12T05:37:06.652 INFO:tasks.workunit.client.0.smithi032.stdout:7/476: dwrite d0/d1c/f4e [0,4194304] 0 2024-04-12T05:37:06.654 INFO:tasks.workunit.client.0.smithi032.stdout:0/554: symlink d2/d8/df/lb6 0 2024-04-12T05:37:06.656 INFO:tasks.workunit.client.0.smithi032.stdout:9/414: creat d9/d15/d1a/d24/f8b x:0 0 0 2024-04-12T05:37:06.658 INFO:tasks.workunit.client.0.smithi032.stdout:7/477: rename d0/de/f61 to d0/d10/f99 0 2024-04-12T05:37:06.659 INFO:tasks.workunit.client.0.smithi032.stdout:2/547: dread dd/f14 [0,4194304] 0 2024-04-12T05:37:06.659 INFO:tasks.workunit.client.0.smithi032.stdout:0/555: truncate d2/dc/fd 1560896 0 2024-04-12T05:37:06.663 INFO:tasks.workunit.client.0.smithi032.stdout:0/556: rmdir d2/d28/d87/d79 39 2024-04-12T05:37:06.666 INFO:tasks.workunit.client.0.smithi032.stdout:7/478: rename d0/dd/d51/c56 to d0/d13/d21/d30/d87/c9a 0 2024-04-12T05:37:06.667 INFO:tasks.workunit.client.0.smithi032.stdout:2/548: creat dd/d19/d9d/d1c/d64/d32/d3a/d8d/fa9 x:0 0 0 2024-04-12T05:37:06.671 INFO:tasks.workunit.client.0.smithi032.stdout:1/401: sync 2024-04-12T05:37:06.671 INFO:tasks.workunit.client.0.smithi032.stdout:8/477: sync 2024-04-12T05:37:06.671 INFO:tasks.workunit.client.0.smithi032.stdout:5/537: sync 2024-04-12T05:37:06.671 INFO:tasks.workunit.client.0.smithi032.stdout:0/557: symlink d2/dc/d3e/dac/d51/lb7 0 2024-04-12T05:37:06.671 INFO:tasks.workunit.client.0.smithi032.stdout:9/415: write d9/d15/d1a/d1d/d5a/f75 [584436,54072] 0 2024-04-12T05:37:06.672 INFO:tasks.workunit.client.0.smithi032.stdout:7/479: symlink d0/d13/d21/d30/l9b 0 2024-04-12T05:37:06.677 INFO:tasks.workunit.client.0.smithi032.stdout:0/558: write d2/d28/f2e [637261,11618] 0 2024-04-12T05:37:06.677 INFO:tasks.workunit.client.0.smithi032.stdout:9/416: stat d9/d15/d1a/d31/d65/d80 0 2024-04-12T05:37:06.678 INFO:tasks.workunit.client.0.smithi032.stdout:9/417: readlink d9/l67 0 2024-04-12T05:37:06.679 INFO:tasks.workunit.client.0.smithi032.stdout:2/549: mknod dd/d19/d9d/d1c/d64/d36/caa 0 2024-04-12T05:37:06.682 INFO:tasks.workunit.client.0.smithi032.stdout:8/478: write d0/db/dd/d11/d1b/d22/d35/d44/f90 [582323,79241] 0 2024-04-12T05:37:06.688 INFO:tasks.workunit.client.0.smithi032.stdout:9/418: write d9/d58/f79 [936060,3939] 0 2024-04-12T05:37:06.688 INFO:tasks.workunit.client.0.smithi032.stdout:2/550: read dd/d19/d20/f28 [2494752,23480] 0 2024-04-12T05:37:06.690 INFO:tasks.workunit.client.0.smithi032.stdout:0/559: rename d2/d28/d87/ca7 to d2/d28/d87/cb8 0 2024-04-12T05:37:06.693 INFO:tasks.workunit.client.0.smithi032.stdout:5/538: dwrite d2/d65/d78/f9e [0,4194304] 0 2024-04-12T05:37:06.693 INFO:tasks.workunit.client.0.smithi032.stdout:1/402: dwrite d5/de/d2a/f65 [0,4194304] 0 2024-04-12T05:37:06.694 INFO:tasks.workunit.client.0.smithi032.stdout:8/479: symlink d0/db/dd/lac 0 2024-04-12T05:37:06.701 INFO:tasks.workunit.client.0.smithi032.stdout:0/560: creat d2/d28/d87/d79/fb9 x:0 0 0 2024-04-12T05:37:06.702 INFO:tasks.workunit.client.0.smithi032.stdout:7/480: dwrite d0/d10/d70/f8f [4194304,4194304] 0 2024-04-12T05:37:06.703 INFO:tasks.workunit.client.0.smithi032.stdout:5/539: truncate d2/d3c/f56 1296135 0 2024-04-12T05:37:06.707 INFO:tasks.workunit.client.0.smithi032.stdout:8/480: unlink d0/db/dd/d11/d1b/d22/d35/c56 0 2024-04-12T05:37:06.707 INFO:tasks.workunit.client.0.smithi032.stdout:0/561: write d2/d29/f65 [104190,94582] 0 2024-04-12T05:37:06.710 INFO:tasks.workunit.client.0.smithi032.stdout:5/540: mknod d2/df/d25/d50/ca8 0 2024-04-12T05:37:06.710 INFO:tasks.workunit.client.0.smithi032.stdout:7/481: creat d0/d13/d21/d81/f9c x:0 0 0 2024-04-12T05:37:06.712 INFO:tasks.workunit.client.0.smithi032.stdout:9/419: dread d9/d15/d1a/f56 [0,4194304] 0 2024-04-12T05:37:06.713 INFO:tasks.workunit.client.0.smithi032.stdout:1/403: dread d5/d2c/f3c [0,4194304] 0 2024-04-12T05:37:06.714 INFO:tasks.workunit.client.0.smithi032.stdout:7/482: write d0/de/df/d20/f6b [1715806,41409] 0 2024-04-12T05:37:06.716 INFO:tasks.workunit.client.0.smithi032.stdout:2/551: dwrite dd/d19/d20/f28 [4194304,4194304] 0 2024-04-12T05:37:06.717 INFO:tasks.workunit.client.0.smithi032.stdout:5/541: chown d2/df/d25/d43/d49/f7d 217758289 1 2024-04-12T05:37:06.722 INFO:tasks.workunit.client.0.smithi032.stdout:1/404: creat d5/de/d17/d35/d61/d77/f7d x:0 0 0 2024-04-12T05:37:06.722 INFO:tasks.workunit.client.0.smithi032.stdout:7/483: rename d0/d10/f99 to d0/d13/d21/d81/f9d 0 2024-04-12T05:37:06.723 INFO:tasks.workunit.client.0.smithi032.stdout:9/420: dread d9/d15/d2c/d41/f7e [0,4194304] 0 2024-04-12T05:37:06.724 INFO:tasks.workunit.client.0.smithi032.stdout:5/542: mkdir d2/df/da9 0 2024-04-12T05:37:06.724 INFO:tasks.workunit.client.0.smithi032.stdout:1/405: fdatasync d5/de/d17/d35/d78/f7c 0 2024-04-12T05:37:06.727 INFO:tasks.workunit.client.0.smithi032.stdout:1/406: write d5/de/f55 [4256350,879] 0 2024-04-12T05:37:06.728 INFO:tasks.workunit.client.0.smithi032.stdout:7/484: mknod d0/d13/d21/d30/d31/c9e 0 2024-04-12T05:37:06.729 INFO:tasks.workunit.client.0.smithi032.stdout:5/543: mkdir d2/df/d25/d50/daa 0 2024-04-12T05:37:06.731 INFO:tasks.workunit.client.0.smithi032.stdout:7/485: write d0/d13/f16 [1121490,117516] 0 2024-04-12T05:37:06.732 INFO:tasks.workunit.client.0.smithi032.stdout:5/544: write d2/f60 [1342271,99213] 0 2024-04-12T05:37:06.734 INFO:tasks.workunit.client.0.smithi032.stdout:2/552: getdents dd/d19 0 2024-04-12T05:37:06.736 INFO:tasks.workunit.client.0.smithi032.stdout:9/421: dwrite d9/d15/d1a/d1d/d5a/f75 [0,4194304] 0 2024-04-12T05:37:06.737 INFO:tasks.workunit.client.0.smithi032.stdout:5/545: dread - d2/df/d25/d50/f8e zero size 2024-04-12T05:37:06.738 INFO:tasks.workunit.client.0.smithi032.stdout:1/407: rename d5/de/f55 to d5/de/f7e 0 2024-04-12T05:37:06.739 INFO:tasks.workunit.client.0.smithi032.stdout:9/422: chown d9/d15/d1a/d24/d71 56187771 1 2024-04-12T05:37:06.739 INFO:tasks.workunit.client.0.smithi032.stdout:2/553: mknod dd/cab 0 2024-04-12T05:37:06.741 INFO:tasks.workunit.client.0.smithi032.stdout:0/562: dwrite d2/d29/f4a [0,4194304] 0 2024-04-12T05:37:06.741 INFO:tasks.workunit.client.0.smithi032.stdout:7/486: rename d0/de/df/d29/c3d to d0/dd/d51/c9f 0 2024-04-12T05:37:06.742 INFO:tasks.workunit.client.0.smithi032.stdout:1/408: read - d5/de/d17/d35/d3b/d49/f5e zero size 2024-04-12T05:37:06.744 INFO:tasks.workunit.client.0.smithi032.stdout:0/563: chown d2/dc/d3e/c75 24113592 1 2024-04-12T05:37:06.746 INFO:tasks.workunit.client.0.smithi032.stdout:5/546: rename d2/df/d25/f67 to d2/df/d25/d50/fab 0 2024-04-12T05:37:06.747 INFO:tasks.workunit.client.0.smithi032.stdout:1/409: creat d5/d2c/f7f x:0 0 0 2024-04-12T05:37:06.748 INFO:tasks.workunit.client.0.smithi032.stdout:2/554: link dd/cab dd/cac 0 2024-04-12T05:37:06.752 INFO:tasks.workunit.client.0.smithi032.stdout:1/410: creat d5/de/d2a/f80 x:0 0 0 2024-04-12T05:37:06.753 INFO:tasks.workunit.client.0.smithi032.stdout:9/423: dread d9/d15/d1a/d24/d27/f2d [0,4194304] 0 2024-04-12T05:37:06.753 INFO:tasks.workunit.client.0.smithi032.stdout:2/555: fsync dd/d19/d9d/d1c/d64/d32/f9f 0 2024-04-12T05:37:06.756 INFO:tasks.workunit.client.0.smithi032.stdout:0/564: getdents d2/dc/d3e/dac/d51 0 2024-04-12T05:37:06.757 INFO:tasks.workunit.client.0.smithi032.stdout:5/547: creat d2/fac x:0 0 0 2024-04-12T05:37:06.758 INFO:tasks.workunit.client.0.smithi032.stdout:2/556: write dd/d19/d82/f8a [234547,34962] 0 2024-04-12T05:37:06.759 INFO:tasks.workunit.client.0.smithi032.stdout:7/487: dwrite d0/de/df/d29/f7d [0,4194304] 0 2024-04-12T05:37:06.759 INFO:tasks.workunit.client.0.smithi032.stdout:9/424: rename d9/d15/d2c/f6d to d9/d32/d86/f8c 0 2024-04-12T05:37:06.760 INFO:tasks.workunit.client.0.smithi032.stdout:5/548: write d2/df/f3f [1632389,37189] 0 2024-04-12T05:37:06.763 INFO:tasks.workunit.client.0.smithi032.stdout:7/488: creat d0/d10/fa0 x:0 0 0 2024-04-12T05:37:06.764 INFO:tasks.workunit.client.0.smithi032.stdout:1/411: link d5/de/d17/d35/d78/f7a d5/de/d17/d35/d3b/f81 0 2024-04-12T05:37:06.765 INFO:tasks.workunit.client.0.smithi032.stdout:9/425: mknod d9/c8d 0 2024-04-12T05:37:06.767 INFO:tasks.workunit.client.0.smithi032.stdout:5/549: creat d2/df/d31/da2/fad x:0 0 0 2024-04-12T05:37:06.767 INFO:tasks.workunit.client.0.smithi032.stdout:7/489: mkdir d0/d13/d21/d30/da1 0 2024-04-12T05:37:06.767 INFO:tasks.workunit.client.0.smithi032.stdout:9/426: chown d9/d32/f53 8951 1 2024-04-12T05:37:06.769 INFO:tasks.workunit.client.0.smithi032.stdout:1/412: mknod d5/de/d17/d35/d78/c82 0 2024-04-12T05:37:06.770 INFO:tasks.workunit.client.0.smithi032.stdout:1/413: readlink d5/de/d17/l22 0 2024-04-12T05:37:06.770 INFO:tasks.workunit.client.0.smithi032.stdout:2/557: dwrite dd/d19/d9d/d1c/d64/d32/f33 [4194304,4194304] 0 2024-04-12T05:37:06.773 INFO:tasks.workunit.client.0.smithi032.stdout:9/427: creat d9/d32/d6e/f8e x:0 0 0 2024-04-12T05:37:06.775 INFO:tasks.workunit.client.0.smithi032.stdout:1/414: creat d5/de/d17/d35/d78/f83 x:0 0 0 2024-04-12T05:37:06.776 INFO:tasks.workunit.client.0.smithi032.stdout:1/415: readlink d5/de/d17/l22 0 2024-04-12T05:37:06.779 INFO:tasks.workunit.client.0.smithi032.stdout:9/428: getdents d9/d1f 0 2024-04-12T05:37:06.780 INFO:tasks.workunit.client.0.smithi032.stdout:1/416: write d5/de/d17/d1b/f38 [1033682,22400] 0 2024-04-12T05:37:06.780 INFO:tasks.workunit.client.0.smithi032.stdout:9/429: chown d9/d1f 480958 1 2024-04-12T05:37:06.783 INFO:tasks.workunit.client.0.smithi032.stdout:1/417: chown d5/c4e 914989037 1 2024-04-12T05:37:06.783 INFO:tasks.workunit.client.0.smithi032.stdout:2/558: dwrite dd/d19/d9d/d42/f80 [0,4194304] 0 2024-04-12T05:37:06.783 INFO:tasks.workunit.client.0.smithi032.stdout:9/430: symlink d9/d15/l8f 0 2024-04-12T05:37:06.785 INFO:tasks.workunit.client.0.smithi032.stdout:1/418: mknod d5/d2c/c84 0 2024-04-12T05:37:06.786 INFO:tasks.workunit.client.0.smithi032.stdout:2/559: symlink dd/d19/d20/lad 0 2024-04-12T05:37:06.815 INFO:tasks.workunit.client.0.smithi032.stdout:3/493: sync 2024-04-12T05:37:06.831 INFO:tasks.workunit.client.0.smithi032.stdout:9/431: creat d9/d15/d1a/d24/d27/f90 x:0 0 0 2024-04-12T05:37:06.832 INFO:tasks.workunit.client.0.smithi032.stdout:1/419: write d5/de/d17/d35/d78/f7a [1271469,22588] 0 2024-04-12T05:37:06.834 INFO:tasks.workunit.client.0.smithi032.stdout:3/494: write d12/d2f/f91 [550407,93509] 0 2024-04-12T05:37:06.836 INFO:tasks.workunit.client.0.smithi032.stdout:1/420: chown d5/de/d17/d1b/f5f 0 1 2024-04-12T05:37:06.837 INFO:tasks.workunit.client.0.smithi032.stdout:2/560: creat dd/d19/d9d/fae x:0 0 0 2024-04-12T05:37:06.837 INFO:tasks.workunit.client.0.smithi032.stdout:9/432: creat d9/d32/d6e/f91 x:0 0 0 2024-04-12T05:37:06.837 INFO:tasks.workunit.client.0.smithi032.stdout:3/495: rmdir d12/d2e/d3a/d63 39 2024-04-12T05:37:06.839 INFO:tasks.workunit.client.0.smithi032.stdout:7/490: dread d0/d76/f7e [0,4194304] 0 2024-04-12T05:37:06.840 INFO:tasks.workunit.client.0.smithi032.stdout:5/550: dread d2/df/d25/d43/f7a [0,4194304] 0 2024-04-12T05:37:06.841 INFO:tasks.workunit.client.0.smithi032.stdout:1/421: creat d5/de/d17/d1b/f85 x:0 0 0 2024-04-12T05:37:06.842 INFO:tasks.workunit.client.0.smithi032.stdout:9/433: mknod d9/d15/d2c/c92 0 2024-04-12T05:37:06.843 INFO:tasks.workunit.client.0.smithi032.stdout:3/496: symlink d12/la1 0 2024-04-12T05:37:06.845 INFO:tasks.workunit.client.0.smithi032.stdout:3/497: dread - d12/d2e/f86 zero size 2024-04-12T05:37:06.845 INFO:tasks.workunit.client.0.smithi032.stdout:9/434: chown d9/d15/d1a/d24/d27/l4f 3315 1 2024-04-12T05:37:06.847 INFO:tasks.workunit.client.0.smithi032.stdout:2/561: dwrite dd/d19/d9d/d1c/d64/d36/f8b [0,4194304] 0 2024-04-12T05:37:06.847 INFO:tasks.workunit.client.0.smithi032.stdout:3/498: stat d12/d2e/d3a/d2d/l4b 0 2024-04-12T05:37:06.850 INFO:tasks.workunit.client.0.smithi032.stdout:1/422: rename d5/de/d17/d1b/f33 to d5/de/d17/f86 0 2024-04-12T05:37:06.850 INFO:tasks.workunit.client.0.smithi032.stdout:7/491: mknod d0/d13/d21/ca2 0 2024-04-12T05:37:06.850 INFO:tasks.workunit.client.0.smithi032.stdout:5/551: creat d2/df/d25/d50/daa/fae x:0 0 0 2024-04-12T05:37:06.851 INFO:tasks.workunit.client.0.smithi032.stdout:2/562: readlink dd/d19/d9d/d1c/d64/d32/d3a/d8d/l9a 0 2024-04-12T05:37:06.851 INFO:tasks.workunit.client.0.smithi032.stdout:7/492: write d0/f7 [101540,128460] 0 2024-04-12T05:37:06.852 INFO:tasks.workunit.client.0.smithi032.stdout:1/423: write d5/de/d17/d35/d3b/f63 [648062,39365] 0 2024-04-12T05:37:06.854 INFO:tasks.workunit.client.0.smithi032.stdout:7/493: write d0/de/df/f7a [1406641,97573] 0 2024-04-12T05:37:06.854 INFO:tasks.workunit.client.0.smithi032.stdout:1/424: chown d5/de/d17/d35/f75 83132 1 2024-04-12T05:37:06.855 INFO:tasks.workunit.client.0.smithi032.stdout:3/499: read d12/d2e/d3a/f7f [630615,109569] 0 2024-04-12T05:37:06.856 INFO:tasks.workunit.client.0.smithi032.stdout:5/552: readlink d2/df/d25/d43/d49/l4d 0 2024-04-12T05:37:06.857 INFO:tasks.workunit.client.0.smithi032.stdout:2/563: rmdir dd/d19/d9d/d1c/d64/d32/d6d 39 2024-04-12T05:37:06.857 INFO:tasks.workunit.client.0.smithi032.stdout:3/500: fsync d12/d2e/d3a/f96 0 2024-04-12T05:37:06.859 INFO:tasks.workunit.client.0.smithi032.stdout:3/501: write d12/d2f/d5b/f87 [87030,117741] 0 2024-04-12T05:37:06.859 INFO:tasks.workunit.client.0.smithi032.stdout:1/425: fsync d5/de/f1f 0 2024-04-12T05:37:06.861 INFO:tasks.workunit.client.0.smithi032.stdout:5/553: write d2/d9b/f5a [3470492,16988] 0 2024-04-12T05:37:06.864 INFO:tasks.workunit.client.0.smithi032.stdout:3/502: symlink d12/d2e/la2 0 2024-04-12T05:37:06.865 INFO:tasks.workunit.client.0.smithi032.stdout:1/426: mknod d5/de/d17/d35/d73/d74/c87 0 2024-04-12T05:37:06.866 INFO:tasks.workunit.client.0.smithi032.stdout:1/427: chown d5/d2c/d4f 23 1 2024-04-12T05:37:06.867 INFO:tasks.workunit.client.0.smithi032.stdout:7/494: dwrite d0/d76/f7e [0,4194304] 0 2024-04-12T05:37:06.869 INFO:tasks.workunit.client.0.smithi032.stdout:3/503: mknod d12/ca3 0 2024-04-12T05:37:06.874 INFO:tasks.workunit.client.0.smithi032.stdout:9/435: dread d9/d15/f43 [0,4194304] 0 2024-04-12T05:37:06.875 INFO:tasks.workunit.client.0.smithi032.stdout:1/428: fdatasync d5/de/d17/f86 0 2024-04-12T05:37:06.877 INFO:tasks.workunit.client.0.smithi032.stdout:5/554: getdents d2/df/d31/d80 0 2024-04-12T05:37:06.878 INFO:tasks.workunit.client.0.smithi032.stdout:9/436: creat d9/d32/d6e/f93 x:0 0 0 2024-04-12T05:37:06.879 INFO:tasks.workunit.client.0.smithi032.stdout:1/429: truncate d5/de/d17/d35/d78/f7c 592149 0 2024-04-12T05:37:06.880 INFO:tasks.workunit.client.0.smithi032.stdout:5/555: creat d2/dd/d5d/faf x:0 0 0 2024-04-12T05:37:06.881 INFO:tasks.workunit.client.0.smithi032.stdout:1/430: mknod d5/d2c/d4f/c88 0 2024-04-12T05:37:06.882 INFO:tasks.workunit.client.0.smithi032.stdout:1/431: write d5/de/f11 [4067277,49813] 0 2024-04-12T05:37:06.883 INFO:tasks.workunit.client.0.smithi032.stdout:9/437: rename d9/d15/f30 to d9/d15/d1a/d24/d71/f94 0 2024-04-12T05:37:06.887 INFO:tasks.workunit.client.0.smithi032.stdout:1/432: write d5/f23 [275170,83104] 0 2024-04-12T05:37:06.887 INFO:tasks.workunit.client.0.smithi032.stdout:9/438: symlink d9/d15/d1a/d1d/d5a/d6b/d83/l95 0 2024-04-12T05:37:06.894 INFO:tasks.workunit.client.0.smithi032.stdout:1/433: getdents d5/de/d17/d1b 0 2024-04-12T05:37:06.896 INFO:tasks.workunit.client.0.smithi032.stdout:1/434: write d5/de/d17/f20 [9528971,99028] 0 2024-04-12T05:37:06.898 INFO:tasks.workunit.client.0.smithi032.stdout:3/504: dread d12/d2e/d3a/f26 [0,4194304] 0 2024-04-12T05:37:06.899 INFO:tasks.workunit.client.0.smithi032.stdout:5/556: dwrite d2/d3c/d5b/d75/f84 [0,4194304] 0 2024-04-12T05:37:06.901 INFO:tasks.workunit.client.0.smithi032.stdout:7/495: dread d0/dd/f3b [4194304,4194304] 0 2024-04-12T05:37:06.903 INFO:tasks.workunit.client.0.smithi032.stdout:3/505: write d12/d2e/d3a/f26 [1752532,93300] 0 2024-04-12T05:37:06.903 INFO:tasks.workunit.client.0.smithi032.stdout:5/557: rename d2/df/d25/d1e/f59 to d2/df/d25/d50/fb0 0 2024-04-12T05:37:06.907 INFO:tasks.workunit.client.0.smithi032.stdout:3/506: creat d12/d24/fa4 x:0 0 0 2024-04-12T05:37:06.908 INFO:tasks.workunit.client.0.smithi032.stdout:1/435: dread d5/de/d17/d1b/f38 [0,4194304] 0 2024-04-12T05:37:06.910 INFO:tasks.workunit.client.0.smithi032.stdout:3/507: symlink d12/d24/la5 0 2024-04-12T05:37:06.911 INFO:tasks.workunit.client.0.smithi032.stdout:3/508: rmdir d12/d2e 39 2024-04-12T05:37:06.913 INFO:tasks.workunit.client.0.smithi032.stdout:1/436: link d5/de/f7e d5/de/d2a/f89 0 2024-04-12T05:37:06.915 INFO:tasks.workunit.client.0.smithi032.stdout:5/558: dread d2/df/d25/f16 [0,4194304] 0 2024-04-12T05:37:06.916 INFO:tasks.workunit.client.0.smithi032.stdout:3/509: dread d12/d2f/d5b/f66 [0,4194304] 0 2024-04-12T05:37:06.920 INFO:tasks.workunit.client.0.smithi032.stdout:5/559: link d2/df/d25/d1e/f40 d2/df/fb1 0 2024-04-12T05:37:06.920 INFO:tasks.workunit.client.0.smithi032.stdout:3/510: mkdir d12/d2e/d3a/d63/da6 0 2024-04-12T05:37:06.921 INFO:tasks.workunit.client.0.smithi032.stdout:1/437: dwrite d5/de/d17/d35/d3b/f63 [0,4194304] 0 2024-04-12T05:37:06.924 INFO:tasks.workunit.client.0.smithi032.stdout:1/438: creat d5/de/d17/d35/d73/f8a x:0 0 0 2024-04-12T05:37:06.928 INFO:tasks.workunit.client.0.smithi032.stdout:5/560: dwrite d2/df/d25/f29 [0,4194304] 0 2024-04-12T05:37:06.931 INFO:tasks.workunit.client.0.smithi032.stdout:3/511: write d12/f13 [2251214,115648] 0 2024-04-12T05:37:06.934 INFO:tasks.workunit.client.0.smithi032.stdout:5/561: mknod d2/dd/cb2 0 2024-04-12T05:37:06.934 INFO:tasks.workunit.client.0.smithi032.stdout:1/439: link d5/l12 d5/de/d17/d35/d3b/d49/l8b 0 2024-04-12T05:37:06.936 INFO:tasks.workunit.client.0.smithi032.stdout:1/440: chown d5/de/d17/d1b/c57 26 1 2024-04-12T05:37:06.938 INFO:tasks.workunit.client.0.smithi032.stdout:3/512: dwrite d12/d2f/d5b/f87 [0,4194304] 0 2024-04-12T05:37:06.939 INFO:tasks.workunit.client.0.smithi032.stdout:4/405: sync 2024-04-12T05:37:06.939 INFO:tasks.workunit.client.0.smithi032.stdout:8/481: sync 2024-04-12T05:37:06.939 INFO:tasks.workunit.client.0.smithi032.stdout:6/592: sync 2024-04-12T05:37:06.939 INFO:tasks.workunit.client.0.smithi032.stdout:1/441: rename d5/de/c36 to d5/de/d2a/c8c 0 2024-04-12T05:37:06.942 INFO:tasks.workunit.client.0.smithi032.stdout:4/406: rename d4/d13/d19/c7b to d4/d87/c8b 0 2024-04-12T05:37:06.944 INFO:tasks.workunit.client.0.smithi032.stdout:1/442: mknod d5/d2c/d4f/c8d 0 2024-04-12T05:37:06.945 INFO:tasks.workunit.client.0.smithi032.stdout:6/593: mkdir d4/d14/d1a/d79/dc6 0 2024-04-12T05:37:06.946 INFO:tasks.workunit.client.0.smithi032.stdout:3/513: dread d12/d2e/d3a/f7f [0,4194304] 0 2024-04-12T05:37:06.946 INFO:tasks.workunit.client.0.smithi032.stdout:8/482: rename d0/db/d38/d63/d77/d9c to d0/d36/d84/dad 0 2024-04-12T05:37:06.949 INFO:tasks.workunit.client.0.smithi032.stdout:4/407: symlink d4/d13/d75/l8c 0 2024-04-12T05:37:06.950 INFO:tasks.workunit.client.0.smithi032.stdout:4/408: chown d4/d13/f1f 406761193 1 2024-04-12T05:37:06.950 INFO:tasks.workunit.client.0.smithi032.stdout:3/514: creat d12/d2f/d89/fa7 x:0 0 0 2024-04-12T05:37:06.952 INFO:tasks.workunit.client.0.smithi032.stdout:6/594: creat d4/d21/db4/fc7 x:0 0 0 2024-04-12T05:37:06.952 INFO:tasks.workunit.client.0.smithi032.stdout:8/483: mknod d0/db/dd/d11/d1b/d96/cae 0 2024-04-12T05:37:06.953 INFO:tasks.workunit.client.0.smithi032.stdout:4/409: mknod d4/d13/d75/c8d 0 2024-04-12T05:37:06.954 INFO:tasks.workunit.client.0.smithi032.stdout:1/443: rmdir d5/de/d2a 39 2024-04-12T05:37:06.955 INFO:tasks.workunit.client.0.smithi032.stdout:4/410: write d4/d13/d19/d28/d2f/f70 [1444771,103693] 0 2024-04-12T05:37:06.955 INFO:tasks.workunit.client.0.smithi032.stdout:3/515: unlink d12/d2e/d3a/d63/l80 0 2024-04-12T05:37:06.956 INFO:tasks.workunit.client.0.smithi032.stdout:3/516: stat d12/d2f/c85 0 2024-04-12T05:37:06.956 INFO:tasks.workunit.client.0.smithi032.stdout:4/411: truncate d4/d13/f71 542139 0 2024-04-12T05:37:06.958 INFO:tasks.workunit.client.0.smithi032.stdout:4/412: truncate d4/d13/d19/d28/f42 4893014 0 2024-04-12T05:37:06.958 INFO:tasks.workunit.client.0.smithi032.stdout:8/484: unlink d0/db/f2e 0 2024-04-12T05:37:06.960 INFO:tasks.workunit.client.0.smithi032.stdout:6/595: creat d4/d14/d1a/d79/dc6/fc8 x:0 0 0 2024-04-12T05:37:06.962 INFO:tasks.workunit.client.0.smithi032.stdout:4/413: chown d4/d13/d19/c6f 1 1 2024-04-12T05:37:06.962 INFO:tasks.workunit.client.0.smithi032.stdout:6/596: fdatasync d4/d21/d2e/d32/fba 0 2024-04-12T05:37:06.963 INFO:tasks.workunit.client.0.smithi032.stdout:8/485: rename d0/db/dd/d11/d1b/d1e to d0/db/dd/d11/d1b/d5f/daf 0 2024-04-12T05:37:06.964 INFO:tasks.workunit.client.0.smithi032.stdout:1/444: symlink d5/de/d2a/l8e 0 2024-04-12T05:37:06.964 INFO:tasks.workunit.client.0.smithi032.stdout:6/597: write d4/d14/d3c/f41 [5412869,113657] 0 2024-04-12T05:37:06.967 INFO:tasks.workunit.client.0.smithi032.stdout:4/414: symlink d4/d13/d19/d28/d2f/d3b/l8e 0 2024-04-12T05:37:06.969 INFO:tasks.workunit.client.0.smithi032.stdout:6/598: write d4/d21/d2e/d32/fba [865756,52847] 0 2024-04-12T05:37:06.970 INFO:tasks.workunit.client.0.smithi032.stdout:3/517: dwrite d12/d2e/d3a/d2d/f45 [0,4194304] 0 2024-04-12T05:37:06.971 INFO:tasks.workunit.client.0.smithi032.stdout:8/486: creat d0/db/dd/d11/d1b/d22/d54/da8/fb0 x:0 0 0 2024-04-12T05:37:06.972 INFO:tasks.workunit.client.0.smithi032.stdout:4/415: rmdir d4/d13/d19/d28/d4d/d78 39 2024-04-12T05:37:06.973 INFO:tasks.workunit.client.0.smithi032.stdout:3/518: rmdir d12/d2e 39 2024-04-12T05:37:06.974 INFO:tasks.workunit.client.0.smithi032.stdout:6/599: mknod d4/d21/d27/cc9 0 2024-04-12T05:37:06.977 INFO:tasks.workunit.client.0.smithi032.stdout:4/416: symlink d4/l8f 0 2024-04-12T05:37:06.978 INFO:tasks.workunit.client.0.smithi032.stdout:3/519: read - d12/d2e/d3a/f79 zero size 2024-04-12T05:37:06.979 INFO:tasks.workunit.client.0.smithi032.stdout:1/445: dread d5/de/d17/d1b/f1d [0,4194304] 0 2024-04-12T05:37:06.979 INFO:tasks.workunit.client.0.smithi032.stdout:6/600: mkdir d4/d21/d2e/dca 0 2024-04-12T05:37:06.980 INFO:tasks.workunit.client.0.smithi032.stdout:3/520: chown d12/d2e/f50 583418 1 2024-04-12T05:37:06.982 INFO:tasks.workunit.client.0.smithi032.stdout:4/417: mknod d4/d13/d75/c90 0 2024-04-12T05:37:06.985 INFO:tasks.workunit.client.0.smithi032.stdout:6/601: mknod d4/d14/d1a/dc4/ccb 0 2024-04-12T05:37:06.988 INFO:tasks.workunit.client.0.smithi032.stdout:1/446: dwrite d5/d2c/f43 [0,4194304] 0 2024-04-12T05:37:06.989 INFO:tasks.workunit.client.0.smithi032.stdout:8/487: write d0/db/dd/d11/d1b/f6c [247280,31872] 0 2024-04-12T05:37:06.991 INFO:tasks.workunit.client.0.smithi032.stdout:6/602: creat d4/d14/d1a/d88/fcc x:0 0 0 2024-04-12T05:37:06.992 INFO:tasks.workunit.client.0.smithi032.stdout:8/488: fsync d0/f2f 0 2024-04-12T05:37:06.993 INFO:tasks.workunit.client.0.smithi032.stdout:4/418: dwrite d4/d13/f43 [0,4194304] 0 2024-04-12T05:37:06.995 INFO:tasks.workunit.client.0.smithi032.stdout:4/419: write d4/d13/d19/f80 [518962,8240] 0 2024-04-12T05:37:06.996 INFO:tasks.workunit.client.0.smithi032.stdout:1/447: dread d5/de/d17/d1b/f38 [0,4194304] 0 2024-04-12T05:37:06.999 INFO:tasks.workunit.client.0.smithi032.stdout:4/420: chown d4/d13/d19/d28 26090 1 2024-04-12T05:37:07.003 INFO:tasks.workunit.client.0.smithi032.stdout:4/421: unlink d4/c37 0 2024-04-12T05:37:07.007 INFO:tasks.workunit.client.0.smithi032.stdout:6/603: dwrite d4/d14/d1a/d79/dc6/fc8 [0,4194304] 0 2024-04-12T05:37:07.008 INFO:tasks.workunit.client.0.smithi032.stdout:8/489: dwrite d0/db/dd/d11/d1b/d22/d51/fa9 [0,4194304] 0 2024-04-12T05:37:07.008 INFO:tasks.workunit.client.0.smithi032.stdout:4/422: write d4/d13/f1c [885528,91592] 0 2024-04-12T05:37:07.010 INFO:tasks.workunit.client.0.smithi032.stdout:4/423: chown d4/d13/l2d 446835 1 2024-04-12T05:37:07.014 INFO:tasks.workunit.client.0.smithi032.stdout:6/604: creat d4/d21/db4/fcd x:0 0 0 2024-04-12T05:37:07.015 INFO:tasks.workunit.client.0.smithi032.stdout:8/490: creat d0/d36/d84/dad/da4/fb1 x:0 0 0 2024-04-12T05:37:07.016 INFO:tasks.workunit.client.0.smithi032.stdout:1/448: dread d5/d2c/f2d [4194304,4194304] 0 2024-04-12T05:37:07.017 INFO:tasks.workunit.client.0.smithi032.stdout:4/424: creat d4/d13/d44/f91 x:0 0 0 2024-04-12T05:37:07.017 INFO:tasks.workunit.client.0.smithi032.stdout:6/605: write d4/d21/f85 [4972442,29672] 0 2024-04-12T05:37:07.021 INFO:tasks.workunit.client.0.smithi032.stdout:8/491: read d0/f3 [2499845,100911] 0 2024-04-12T05:37:07.023 INFO:tasks.workunit.client.0.smithi032.stdout:6/606: symlink d4/d21/db4/lce 0 2024-04-12T05:37:07.023 INFO:tasks.workunit.client.0.smithi032.stdout:4/425: fdatasync d4/d13/d44/f5a 0 2024-04-12T05:37:07.025 INFO:tasks.workunit.client.0.smithi032.stdout:1/449: dwrite d5/de/d17/d35/d3b/f63 [0,4194304] 0 2024-04-12T05:37:07.027 INFO:tasks.workunit.client.0.smithi032.stdout:6/607: mknod d4/d21/d2e/d32/d94/ccf 0 2024-04-12T05:37:07.029 INFO:tasks.workunit.client.0.smithi032.stdout:8/492: mknod d0/d58/cb2 0 2024-04-12T05:37:07.034 INFO:tasks.workunit.client.0.smithi032.stdout:1/450: dwrite d5/de/d17/d35/d78/f83 [0,4194304] 0 2024-04-12T05:37:07.035 INFO:tasks.workunit.client.0.smithi032.stdout:1/451: chown d5/d2c 78 1 2024-04-12T05:37:07.038 INFO:tasks.workunit.client.0.smithi032.stdout:8/493: getdents d0/db/dd/d11/d1b/d22 0 2024-04-12T05:37:07.039 INFO:tasks.workunit.client.0.smithi032.stdout:1/452: mkdir d5/de/d17/d35/d61/d77/d8f 0 2024-04-12T05:37:07.040 INFO:tasks.workunit.client.0.smithi032.stdout:1/453: read d5/d2c/f43 [1886562,115175] 0 2024-04-12T05:37:07.040 INFO:tasks.workunit.client.0.smithi032.stdout:8/494: write d0/db/dd/d11/d1b/d22/d51/fa9 [1460276,51122] 0 2024-04-12T05:37:07.042 INFO:tasks.workunit.client.0.smithi032.stdout:6/608: dread d4/d21/d27/dad/d55/f65 [0,4194304] 0 2024-04-12T05:37:07.046 INFO:tasks.workunit.client.0.smithi032.stdout:1/454: symlink d5/de/d17/d35/d61/l90 0 2024-04-12T05:37:07.049 INFO:tasks.workunit.client.0.smithi032.stdout:4/426: truncate d4/f18 4581408 0 2024-04-12T05:37:07.051 INFO:tasks.workunit.client.0.smithi032.stdout:6/609: dwrite d4/d21/d27/dad/d55/f72 [0,4194304] 0 2024-04-12T05:37:07.052 INFO:tasks.workunit.client.0.smithi032.stdout:6/610: chown l1 21 1 2024-04-12T05:37:07.054 INFO:tasks.workunit.client.0.smithi032.stdout:4/427: read d4/d13/d19/f30 [1054381,9134] 0 2024-04-12T05:37:07.057 INFO:tasks.workunit.client.0.smithi032.stdout:6/611: write d4/f5 [1108330,123694] 0 2024-04-12T05:37:07.057 INFO:tasks.workunit.client.0.smithi032.stdout:4/428: fsync d4/d13/f71 0 2024-04-12T05:37:07.059 INFO:tasks.workunit.client.0.smithi032.stdout:1/455: dwrite d5/de/d17/d35/d73/f8a [0,4194304] 0 2024-04-12T05:37:07.061 INFO:tasks.workunit.client.0.smithi032.stdout:4/429: fdatasync d4/d13/fd 0 2024-04-12T05:37:07.061 INFO:tasks.workunit.client.0.smithi032.stdout:6/612: read - d4/d21/d2e/fbf zero size 2024-04-12T05:37:07.066 INFO:tasks.workunit.client.0.smithi032.stdout:4/430: write d4/d13/d19/d28/d2f/f60 [1504767,8339] 0 2024-04-12T05:37:07.068 INFO:tasks.workunit.client.0.smithi032.stdout:0/565: sync 2024-04-12T05:37:07.068 INFO:tasks.workunit.client.0.smithi032.stdout:8/495: dwrite d0/db/f6e [0,4194304] 0 2024-04-12T05:37:07.068 INFO:tasks.workunit.client.0.smithi032.stdout:1/456: dread d5/de/d17/f25 [0,4194304] 0 2024-04-12T05:37:07.073 INFO:tasks.workunit.client.0.smithi032.stdout:0/566: creat d2/d29/d8a/d99/fba x:0 0 0 2024-04-12T05:37:07.074 INFO:tasks.workunit.client.0.smithi032.stdout:1/457: unlink d5/de/d17/d35/d3b/f81 0 2024-04-12T05:37:07.077 INFO:tasks.workunit.client.0.smithi032.stdout:4/431: rename d4/d13/d19/f80 to d4/d13/d19/d28/d4d/f92 0 2024-04-12T05:37:07.079 INFO:tasks.workunit.client.0.smithi032.stdout:1/458: mknod d5/de/d17/d35/d78/c91 0 2024-04-12T05:37:07.081 INFO:tasks.workunit.client.0.smithi032.stdout:8/496: dwrite d0/d36/d84/dad/da4/fb1 [0,4194304] 0 2024-04-12T05:37:07.082 INFO:tasks.workunit.client.0.smithi032.stdout:0/567: creat d2/d8/d17/d1b/fbb x:0 0 0 2024-04-12T05:37:07.084 INFO:tasks.workunit.client.0.smithi032.stdout:0/568: readlink d2/dc/d41/l67 0 2024-04-12T05:37:07.085 INFO:tasks.workunit.client.0.smithi032.stdout:8/497: write d0/f2f [1414089,121838] 0 2024-04-12T05:37:07.085 INFO:tasks.workunit.client.0.smithi032.stdout:6/613: dread d4/d21/d27/f4b [0,4194304] 0 2024-04-12T05:37:07.090 INFO:tasks.workunit.client.0.smithi032.stdout:0/569: write d2/d29/f47 [4156962,66190] 0 2024-04-12T05:37:07.091 INFO:tasks.workunit.client.0.smithi032.stdout:8/498: mkdir d0/d36/d84/db3 0 2024-04-12T05:37:07.092 INFO:tasks.workunit.client.0.smithi032.stdout:6/614: chown d4/d14/d1a/d37/d69/d76 429 1 2024-04-12T05:37:07.092 INFO:tasks.workunit.client.0.smithi032.stdout:4/432: dwrite d4/f85 [0,4194304] 0 2024-04-12T05:37:07.096 INFO:tasks.workunit.client.0.smithi032.stdout:4/433: stat d4/d13/d19/d28/d2f/d6e/c74 0 2024-04-12T05:37:07.098 INFO:tasks.workunit.client.0.smithi032.stdout:6/615: mknod d4/d14/d1a/d37/d47/cd0 0 2024-04-12T05:37:07.098 INFO:tasks.workunit.client.0.smithi032.stdout:1/459: dread d5/de/f28 [0,4194304] 0 2024-04-12T05:37:07.098 INFO:tasks.workunit.client.0.smithi032.stdout:8/499: creat d0/db/dd/d11/d1b/d22/fb4 x:0 0 0 2024-04-12T05:37:07.099 INFO:tasks.workunit.client.0.smithi032.stdout:0/570: rename d2/d8/d17/d1b/f7a to d2/dc/d41/fbc 0 2024-04-12T05:37:07.100 INFO:tasks.workunit.client.0.smithi032.stdout:6/616: stat d4/c78 0 2024-04-12T05:37:07.101 INFO:tasks.workunit.client.0.smithi032.stdout:4/434: mknod d4/d13/c93 0 2024-04-12T05:37:07.101 INFO:tasks.workunit.client.0.smithi032.stdout:0/571: dread - d2/d8/df/d24/d60/fb0 zero size 2024-04-12T05:37:07.106 INFO:tasks.workunit.client.0.smithi032.stdout:1/460: dwrite d5/de/d17/d1b/f21 [4194304,4194304] 0 2024-04-12T05:37:07.107 INFO:tasks.workunit.client.0.smithi032.stdout:6/617: rmdir d4 39 2024-04-12T05:37:07.112 INFO:tasks.workunit.client.0.smithi032.stdout:1/461: write d5/de/d17/d1b/f38 [1622774,105675] 0 2024-04-12T05:37:07.114 INFO:tasks.workunit.client.0.smithi032.stdout:1/462: chown d5/de/l32 174622 1 2024-04-12T05:37:07.114 INFO:tasks.workunit.client.0.smithi032.stdout:0/572: dread d2/dc/d3e/f46 [0,4194304] 0 2024-04-12T05:37:07.114 INFO:tasks.workunit.client.0.smithi032.stdout:8/500: write d0/db/dd/d11/d1b/d22/d51/f59 [1987368,77289] 0 2024-04-12T05:37:07.116 INFO:tasks.workunit.client.0.smithi032.stdout:8/501: write d0/db/dd/f5c [504495,109665] 0 2024-04-12T05:37:07.117 INFO:tasks.workunit.client.0.smithi032.stdout:1/463: mknod d5/de/d17/d35/d73/c92 0 2024-04-12T05:37:07.125 INFO:tasks.workunit.client.0.smithi032.stdout:1/464: write d5/de/f7e [4950326,109454] 0 2024-04-12T05:37:07.126 INFO:tasks.workunit.client.0.smithi032.stdout:4/435: dread d4/f35 [0,4194304] 0 2024-04-12T05:37:07.126 INFO:tasks.workunit.client.0.smithi032.stdout:0/573: dwrite d2/d8/d17/d1b/d20/f69 [0,4194304] 0 2024-04-12T05:37:07.127 INFO:tasks.workunit.client.0.smithi032.stdout:8/502: rename d0/db/dd/d11/d1b/d22/l81 to d0/db/dd/d11/d1b/d5f/lb5 0 2024-04-12T05:37:07.129 INFO:tasks.workunit.client.0.smithi032.stdout:6/618: dread d4/f6 [0,4194304] 0 2024-04-12T05:37:07.129 INFO:tasks.workunit.client.0.smithi032.stdout:1/465: mkdir d5/de/d17/d93 0 2024-04-12T05:37:07.133 INFO:tasks.workunit.client.0.smithi032.stdout:6/619: fsync d4/d14/d1a/d92/f97 0 2024-04-12T05:37:07.133 INFO:tasks.workunit.client.0.smithi032.stdout:1/466: creat d5/de/d17/d35/d73/f94 x:0 0 0 2024-04-12T05:37:07.135 INFO:tasks.workunit.client.0.smithi032.stdout:6/620: chown d4/d14/d1a/d88/c8c 7639662 1 2024-04-12T05:37:07.136 INFO:tasks.workunit.client.0.smithi032.stdout:1/467: chown d5/de/d2a/l8e 7259 1 2024-04-12T05:37:07.138 INFO:tasks.workunit.client.0.smithi032.stdout:6/621: dread d4/d21/f7d [0,4194304] 0 2024-04-12T05:37:07.140 INFO:tasks.workunit.client.0.smithi032.stdout:6/622: chown d4/d21/c3e 77 1 2024-04-12T05:37:07.141 INFO:tasks.workunit.client.0.smithi032.stdout:0/574: dread d2/d8/df/f31 [0,4194304] 0 2024-04-12T05:37:07.143 INFO:tasks.workunit.client.0.smithi032.stdout:0/575: chown d2/d8/df/d24/d60 2 1 2024-04-12T05:37:07.143 INFO:tasks.workunit.client.0.smithi032.stdout:6/623: write d4/d14/d1a/d37/d69/d76/fb1 [296655,34676] 0 2024-04-12T05:37:07.143 INFO:tasks.workunit.client.0.smithi032.stdout:1/468: dwrite d5/de/d2a/f89 [0,4194304] 0 2024-04-12T05:37:07.146 INFO:tasks.workunit.client.0.smithi032.stdout:8/503: write d0/db/dd/d11/d1b/d5f/f67 [3705637,121683] 0 2024-04-12T05:37:07.148 INFO:tasks.workunit.client.0.smithi032.stdout:1/469: truncate d5/de/d17/d35/d3b/d49/f62 759031 0 2024-04-12T05:37:07.149 INFO:tasks.workunit.client.0.smithi032.stdout:6/624: creat d4/d14/d1a/d92/d9e/fd1 x:0 0 0 2024-04-12T05:37:07.150 INFO:tasks.workunit.client.0.smithi032.stdout:6/625: stat d4/d14/d1a/d37/d69/ca6 0 2024-04-12T05:37:07.151 INFO:tasks.workunit.client.0.smithi032.stdout:8/504: creat d0/d36/d84/dad/fb6 x:0 0 0 2024-04-12T05:37:07.151 INFO:tasks.workunit.client.0.smithi032.stdout:6/626: stat d4/d21/c30 0 2024-04-12T05:37:07.153 INFO:tasks.workunit.client.0.smithi032.stdout:8/505: mknod d0/d36/d84/dad/cb7 0 2024-04-12T05:37:07.155 INFO:tasks.workunit.client.0.smithi032.stdout:6/627: unlink d4/d14/d1a/dc4/ccb 0 2024-04-12T05:37:07.155 INFO:tasks.workunit.client.0.smithi032.stdout:8/506: fdatasync d0/d36/d84/f92 0 2024-04-12T05:37:07.159 INFO:tasks.workunit.client.0.smithi032.stdout:6/628: write d4/d21/d27/dad/d55/f70 [2600640,90962] 0 2024-04-12T05:37:07.159 INFO:tasks.workunit.client.0.smithi032.stdout:0/576: dwrite d2/d29/f5d [0,4194304] 0 2024-04-12T05:37:07.161 INFO:tasks.workunit.client.0.smithi032.stdout:6/629: chown d4/d21/d2e/da1 396752319 1 2024-04-12T05:37:07.161 INFO:tasks.workunit.client.0.smithi032.stdout:8/507: getdents d0/db/d39 0 2024-04-12T05:37:07.164 INFO:tasks.workunit.client.0.smithi032.stdout:6/630: symlink d4/d21/d2e/d32/d94/ld2 0 2024-04-12T05:37:07.171 INFO:tasks.workunit.client.0.smithi032.stdout:0/577: dwrite d2/dc/d3e/dac/d51/f8d [0,4194304] 0 2024-04-12T05:37:07.172 INFO:tasks.workunit.client.0.smithi032.stdout:8/508: creat d0/db/dd/d11/d1b/d22/d51/fb8 x:0 0 0 2024-04-12T05:37:07.172 INFO:tasks.workunit.client.0.smithi032.stdout:1/470: dwrite d5/d2c/f43 [0,4194304] 0 2024-04-12T05:37:07.174 INFO:tasks.workunit.client.0.smithi032.stdout:0/578: stat d2/d8/df/f31 0 2024-04-12T05:37:07.179 INFO:tasks.workunit.client.0.smithi032.stdout:8/509: creat d0/db/d7d/fb9 x:0 0 0 2024-04-12T05:37:07.180 INFO:tasks.workunit.client.0.smithi032.stdout:1/471: creat d5/de/d5b/f95 x:0 0 0 2024-04-12T05:37:07.181 INFO:tasks.workunit.client.0.smithi032.stdout:6/631: dwrite d4/d14/d1a/d37/f5f [0,4194304] 0 2024-04-12T05:37:07.183 INFO:tasks.workunit.client.0.smithi032.stdout:1/472: write d5/de/d17/d35/d78/f83 [3239906,5039] 0 2024-04-12T05:37:07.184 INFO:tasks.workunit.client.0.smithi032.stdout:0/579: link d2/d8/df/d24/d60/c94 d2/d28/d87/d79/cbd 0 2024-04-12T05:37:07.184 INFO:tasks.workunit.client.0.smithi032.stdout:0/580: readlink d2/dc/d41/l49 0 2024-04-12T05:37:07.186 INFO:tasks.workunit.client.0.smithi032.stdout:8/510: readlink d0/db/dd/d11/d1b/d22/d29/l7b 0 2024-04-12T05:37:07.187 INFO:tasks.workunit.client.0.smithi032.stdout:6/632: creat d4/d14/d1a/d79/fd3 x:0 0 0 2024-04-12T05:37:07.200 INFO:tasks.workunit.client.0.smithi032.stdout:8/511: dread d0/db/dd/d11/d1b/d5f/f67 [0,4194304] 0 2024-04-12T05:37:07.205 INFO:tasks.workunit.client.0.smithi032.stdout:8/512: rename d0/db/dd/d11/d1b/d22/d29/l7b to d0/d72/lba 0 2024-04-12T05:37:07.206 INFO:tasks.workunit.client.0.smithi032.stdout:8/513: truncate d0/db/f60 4433196 0 2024-04-12T05:37:07.209 INFO:tasks.workunit.client.0.smithi032.stdout:8/514: fsync d0/db/dd/d11/d1b/d22/d35/f87 0 2024-04-12T05:37:07.210 INFO:tasks.workunit.client.0.smithi032.stdout:8/515: chown d0/db/dd/d11/d1b/d22/d35/d44/c9f 2715555 1 2024-04-12T05:37:07.211 INFO:tasks.workunit.client.0.smithi032.stdout:8/516: mknod d0/db/d39/cbb 0 2024-04-12T05:37:07.214 INFO:tasks.workunit.client.0.smithi032.stdout:8/517: write d0/db/dd/d11/d1b/d5f/f93 [734492,47293] 0 2024-04-12T05:37:07.215 INFO:tasks.workunit.client.0.smithi032.stdout:8/518: chown d0/db/dd/d11/d1b/d22/d35/d44/c9f 1011644987 1 2024-04-12T05:37:07.216 INFO:tasks.workunit.client.0.smithi032.stdout:6/633: dread d4/d21/d2e/f39 [0,4194304] 0 2024-04-12T05:37:07.217 INFO:tasks.workunit.client.0.smithi032.stdout:8/519: creat d0/d36/d84/dad/da4/fbc x:0 0 0 2024-04-12T05:37:07.218 INFO:tasks.workunit.client.0.smithi032.stdout:8/520: dread - d0/d36/d84/dad/fb6 zero size 2024-04-12T05:37:07.219 INFO:tasks.workunit.client.0.smithi032.stdout:8/521: dread - d0/f98 zero size 2024-04-12T05:37:07.221 INFO:tasks.workunit.client.0.smithi032.stdout:5/562: sync 2024-04-12T05:37:07.221 INFO:tasks.workunit.client.0.smithi032.stdout:9/439: sync 2024-04-12T05:37:07.221 INFO:tasks.workunit.client.0.smithi032.stdout:7/496: sync 2024-04-12T05:37:07.221 INFO:tasks.workunit.client.0.smithi032.stdout:2/564: sync 2024-04-12T05:37:07.222 INFO:tasks.workunit.client.0.smithi032.stdout:7/497: stat d0/d1c/d71/f86 0 2024-04-12T05:37:07.223 INFO:tasks.workunit.client.0.smithi032.stdout:9/440: creat d9/d15/d2c/d41/f96 x:0 0 0 2024-04-12T05:37:07.225 INFO:tasks.workunit.client.0.smithi032.stdout:2/565: write dd/d19/d9d/d1c/d64/d32/d6d/f77 [1509087,31091] 0 2024-04-12T05:37:07.226 INFO:tasks.workunit.client.0.smithi032.stdout:8/522: rmdir d0/d36/d84/dad 39 2024-04-12T05:37:07.229 INFO:tasks.workunit.client.0.smithi032.stdout:9/441: creat d9/d32/d86/f97 x:0 0 0 2024-04-12T05:37:07.230 INFO:tasks.workunit.client.0.smithi032.stdout:5/563: dread d2/df/d31/f46 [0,4194304] 0 2024-04-12T05:37:07.230 INFO:tasks.workunit.client.0.smithi032.stdout:8/523: rmdir d0/db/dd/d11/d1b/d22/d51 39 2024-04-12T05:37:07.231 INFO:tasks.workunit.client.0.smithi032.stdout:2/566: write dd/d19/d9d/d42/d4a/f6b [806199,29656] 0 2024-04-12T05:37:07.231 INFO:tasks.workunit.client.0.smithi032.stdout:7/498: truncate d0/d1c/f4e 2016143 0 2024-04-12T05:37:07.232 INFO:tasks.workunit.client.0.smithi032.stdout:8/524: fdatasync d0/d72/f7a 0 2024-04-12T05:37:07.233 INFO:tasks.workunit.client.0.smithi032.stdout:2/567: chown dd/d19/la4 6065855 1 2024-04-12T05:37:07.233 INFO:tasks.workunit.client.0.smithi032.stdout:6/634: dread d4/d14/d1a/d37/f98 [0,4194304] 0 2024-04-12T05:37:07.234 INFO:tasks.workunit.client.0.smithi032.stdout:8/525: readlink d0/db/d7d/la0 0 2024-04-12T05:37:07.234 INFO:tasks.workunit.client.0.smithi032.stdout:7/499: symlink d0/d1c/d71/la3 0 2024-04-12T05:37:07.235 INFO:tasks.workunit.client.0.smithi032.stdout:2/568: chown dd/d19/d9d/d42/c93 124400 1 2024-04-12T05:37:07.237 INFO:tasks.workunit.client.0.smithi032.stdout:5/564: truncate d2/d3c/d48/d4b/f7c 1795143 0 2024-04-12T05:37:07.238 INFO:tasks.workunit.client.0.smithi032.stdout:8/526: creat d0/db/d38/d63/fbd x:0 0 0 2024-04-12T05:37:07.239 INFO:tasks.workunit.client.0.smithi032.stdout:8/527: stat d0/db/dd/d11/d1b/d22/d35/d44/f90 0 2024-04-12T05:37:07.240 INFO:tasks.workunit.client.0.smithi032.stdout:8/528: chown d0/db/dd/d11/d1b/d22/d29 739769 1 2024-04-12T05:37:07.241 INFO:tasks.workunit.client.0.smithi032.stdout:7/500: rename d0/c7f to d0/d10/d70/d7b/ca4 0 2024-04-12T05:37:07.242 INFO:tasks.workunit.client.0.smithi032.stdout:2/569: symlink dd/d19/d82/laf 0 2024-04-12T05:37:07.243 INFO:tasks.workunit.client.0.smithi032.stdout:5/565: chown d2/d65/d78/d86/l8a 971781 1 2024-04-12T05:37:07.244 INFO:tasks.workunit.client.0.smithi032.stdout:5/566: fsync d2/df/d31/da2/fad 0 2024-04-12T05:37:07.248 INFO:tasks.workunit.client.0.smithi032.stdout:2/570: symlink dd/d19/d82/lb0 0 2024-04-12T05:37:07.249 INFO:tasks.workunit.client.0.smithi032.stdout:6/635: link d4/d74/lbc d4/d14/d1a/d92/d9e/ld4 0 2024-04-12T05:37:07.252 INFO:tasks.workunit.client.0.smithi032.stdout:8/529: mkdir d0/db/dd/d11/d1b/d22/d51/dbe 0 2024-04-12T05:37:07.253 INFO:tasks.workunit.client.0.smithi032.stdout:2/571: symlink dd/d19/d82/lb1 0 2024-04-12T05:37:07.254 INFO:tasks.workunit.client.0.smithi032.stdout:5/567: creat d2/df/d31/d92/fb3 x:0 0 0 2024-04-12T05:37:07.255 INFO:tasks.workunit.client.0.smithi032.stdout:7/501: symlink d0/d13/d21/d30/la5 0 2024-04-12T05:37:07.256 INFO:tasks.workunit.client.0.smithi032.stdout:5/568: write d2/d3c/d5b/f21 [399382,79173] 0 2024-04-12T05:37:07.257 INFO:tasks.workunit.client.0.smithi032.stdout:8/530: dread d0/db/dd/d11/d1b/d5f/f67 [0,4194304] 0 2024-04-12T05:37:07.259 INFO:tasks.workunit.client.0.smithi032.stdout:7/502: creat d0/d76/fa6 x:0 0 0 2024-04-12T05:37:07.259 INFO:tasks.workunit.client.0.smithi032.stdout:8/531: fdatasync d0/db/dd/f27 0 2024-04-12T05:37:07.260 INFO:tasks.workunit.client.0.smithi032.stdout:2/572: link dd/d19/d20/l5b dd/d19/d82/lb2 0 2024-04-12T05:37:07.260 INFO:tasks.workunit.client.0.smithi032.stdout:5/569: unlink d2/df/d31/da2/fad 0 2024-04-12T05:37:07.263 INFO:tasks.workunit.client.0.smithi032.stdout:5/570: truncate d2/d3c/d48/d4b/f6b 718504 0 2024-04-12T05:37:07.263 INFO:tasks.workunit.client.0.smithi032.stdout:7/503: read d0/de/df/f1e [3822323,67191] 0 2024-04-12T05:37:07.265 INFO:tasks.workunit.client.0.smithi032.stdout:7/504: write d0/dd/f6c [689327,4073] 0 2024-04-12T05:37:07.266 INFO:tasks.workunit.client.0.smithi032.stdout:5/571: symlink d2/d3c/d48/d4b/lb4 0 2024-04-12T05:37:07.267 INFO:tasks.workunit.client.0.smithi032.stdout:7/505: chown d0/d13/f8c 373 1 2024-04-12T05:37:07.268 INFO:tasks.workunit.client.0.smithi032.stdout:5/572: rmdir d2/d3c/d5b 39 2024-04-12T05:37:07.269 INFO:tasks.workunit.client.0.smithi032.stdout:7/506: symlink d0/dd/la7 0 2024-04-12T05:37:07.271 INFO:tasks.workunit.client.0.smithi032.stdout:7/507: fdatasync d0/de/df/f3f 0 2024-04-12T05:37:07.273 INFO:tasks.workunit.client.0.smithi032.stdout:8/532: dread d0/db/dd/d11/d1b/d22/d51/f59 [0,4194304] 0 2024-04-12T05:37:07.276 INFO:tasks.workunit.client.0.smithi032.stdout:5/573: dwrite d2/f7b [0,4194304] 0 2024-04-12T05:37:07.276 INFO:tasks.workunit.client.0.smithi032.stdout:2/573: dread dd/d19/d9d/d1c/f43 [0,4194304] 0 2024-04-12T05:37:07.279 INFO:tasks.workunit.client.0.smithi032.stdout:8/533: mknod d0/db/dd/d11/d1b/d22/d35/cbf 0 2024-04-12T05:37:07.279 INFO:tasks.workunit.client.0.smithi032.stdout:8/534: chown d0/d58/cb2 184349181 1 2024-04-12T05:37:07.280 INFO:tasks.workunit.client.0.smithi032.stdout:8/535: write d0/f98 [681271,35135] 0 2024-04-12T05:37:07.281 INFO:tasks.workunit.client.0.smithi032.stdout:2/574: rename dd/d19/d9d/d1c/f43 to dd/d19/d9d/d42/d4a/fb3 0 2024-04-12T05:37:07.288 INFO:tasks.workunit.client.0.smithi032.stdout:8/536: mknod d0/db/dd/cc0 0 2024-04-12T05:37:07.289 INFO:tasks.workunit.client.0.smithi032.stdout:5/574: getdents d2/df/d31 0 2024-04-12T05:37:07.291 INFO:tasks.workunit.client.0.smithi032.stdout:8/537: stat d0/d36/d84/dad/cb7 0 2024-04-12T05:37:07.292 INFO:tasks.workunit.client.0.smithi032.stdout:5/575: creat d2/d9b/fb5 x:0 0 0 2024-04-12T05:37:07.294 INFO:tasks.workunit.client.0.smithi032.stdout:2/575: dwrite dd/d19/d9d/d42/d4a/fb3 [0,4194304] 0 2024-04-12T05:37:07.294 INFO:tasks.workunit.client.0.smithi032.stdout:5/576: chown d2/df/f64 19 1 2024-04-12T05:37:07.295 INFO:tasks.workunit.client.0.smithi032.stdout:7/508: dread d0/de/df/d20/f6b [0,4194304] 0 2024-04-12T05:37:07.296 INFO:tasks.workunit.client.0.smithi032.stdout:5/577: rmdir d2/df/d31 39 2024-04-12T05:37:07.298 INFO:tasks.workunit.client.0.smithi032.stdout:7/509: write d0/d10/fa0 [967796,81410] 0 2024-04-12T05:37:07.298 INFO:tasks.workunit.client.0.smithi032.stdout:8/538: write d0/db/d38/f61 [3111070,128460] 0 2024-04-12T05:37:07.306 INFO:tasks.workunit.client.0.smithi032.stdout:8/539: getdents d0/d36/d84/dad/da4 0 2024-04-12T05:37:07.309 INFO:tasks.workunit.client.0.smithi032.stdout:8/540: symlink d0/db/dd/d11/d1b/d22/d29/lc1 0 2024-04-12T05:37:07.311 INFO:tasks.workunit.client.0.smithi032.stdout:8/541: mknod d0/db/dd/d11/d1b/d22/d35/cc2 0 2024-04-12T05:37:07.314 INFO:tasks.workunit.client.0.smithi032.stdout:8/542: symlink d0/db/dd/d11/d1b/d5f/daf/lc3 0 2024-04-12T05:37:07.315 INFO:tasks.workunit.client.0.smithi032.stdout:8/543: dread - d0/db/dd/f75 zero size 2024-04-12T05:37:07.333 INFO:tasks.workunit.client.0.smithi032.stdout:4/436: sync 2024-04-12T05:37:07.333 INFO:tasks.workunit.client.0.smithi032.stdout:3/521: sync 2024-04-12T05:37:07.345 INFO:tasks.workunit.client.0.smithi032.stdout:7/510: dwrite d0/f8 [4194304,4194304] 0 2024-04-12T05:37:07.347 INFO:tasks.workunit.client.0.smithi032.stdout:3/522: chown d12/d2e/l25 1615962 1 2024-04-12T05:37:07.348 INFO:tasks.workunit.client.0.smithi032.stdout:8/544: unlink d0/c1a 0 2024-04-12T05:37:07.352 INFO:tasks.workunit.client.0.smithi032.stdout:7/511: dread d0/dd/f3b [4194304,4194304] 0 2024-04-12T05:37:07.353 INFO:tasks.workunit.client.0.smithi032.stdout:3/523: rename d12/d2f/d89/l9c to d12/d24/la8 0 2024-04-12T05:37:07.354 INFO:tasks.workunit.client.0.smithi032.stdout:2/576: dread dd/d19/d9d/f58 [0,4194304] 0 2024-04-12T05:37:07.354 INFO:tasks.workunit.client.0.smithi032.stdout:4/437: dwrite d4/f54 [0,4194304] 0 2024-04-12T05:37:07.355 INFO:tasks.workunit.client.0.smithi032.stdout:7/512: stat d0/de/df/d20/f6b 0 2024-04-12T05:37:07.356 INFO:tasks.workunit.client.0.smithi032.stdout:8/545: truncate d0/db/dd/d11/d1b/d5f/f93 982339 0 2024-04-12T05:37:07.357 INFO:tasks.workunit.client.0.smithi032.stdout:4/438: chown d4/d13/c93 37877425 1 2024-04-12T05:37:07.358 INFO:tasks.workunit.client.0.smithi032.stdout:3/524: creat d12/d2e/d3a/fa9 x:0 0 0 2024-04-12T05:37:07.359 INFO:tasks.workunit.client.0.smithi032.stdout:2/577: unlink dd/d19/d9d/d1c/d64/d36/f8b 0 2024-04-12T05:37:07.360 INFO:tasks.workunit.client.0.smithi032.stdout:4/439: fdatasync d4/d13/d19/d28/d2f/f60 0 2024-04-12T05:37:07.360 INFO:tasks.workunit.client.0.smithi032.stdout:3/525: rmdir d12/d2e 39 2024-04-12T05:37:07.362 INFO:tasks.workunit.client.0.smithi032.stdout:4/440: rmdir d4/d87 39 2024-04-12T05:37:07.362 INFO:tasks.workunit.client.0.smithi032.stdout:3/526: fdatasync d12/d2f/d89/fa7 0 2024-04-12T05:37:07.363 INFO:tasks.workunit.client.0.smithi032.stdout:7/513: creat d0/d13/d21/d7c/fa8 x:0 0 0 2024-04-12T05:37:07.364 INFO:tasks.workunit.client.0.smithi032.stdout:2/578: truncate dd/d19/d9d/d1c/d64/f7d 1212325 0 2024-04-12T05:37:07.366 INFO:tasks.workunit.client.0.smithi032.stdout:4/441: mkdir d4/d13/d19/d28/d2f/d94 0 2024-04-12T05:37:07.366 INFO:tasks.workunit.client.0.smithi032.stdout:2/579: chown dd/d19/d9d/l6f 0 1 2024-04-12T05:37:07.367 INFO:tasks.workunit.client.0.smithi032.stdout:3/527: rmdir d12/d2e/d3a/d2d 39 2024-04-12T05:37:07.368 INFO:tasks.workunit.client.0.smithi032.stdout:2/580: chown dd/d19/l57 6393933 1 2024-04-12T05:37:07.368 INFO:tasks.workunit.client.0.smithi032.stdout:8/546: dwrite d0/db/dd/d11/d1b/d22/f2b [4194304,4194304] 0 2024-04-12T05:37:07.369 INFO:tasks.workunit.client.0.smithi032.stdout:7/514: rename d0/de/df/c80 to d0/de/df/d29/ca9 0 2024-04-12T05:37:07.370 INFO:tasks.workunit.client.0.smithi032.stdout:3/528: mkdir d12/daa 0 2024-04-12T05:37:07.372 INFO:tasks.workunit.client.0.smithi032.stdout:2/581: creat dd/d19/d9d/d1c/d64/d36/fb4 x:0 0 0 2024-04-12T05:37:07.373 INFO:tasks.workunit.client.0.smithi032.stdout:8/547: fsync d0/db/dd/d11/d1b/d5f/daf/fa2 0 2024-04-12T05:37:07.373 INFO:tasks.workunit.client.0.smithi032.stdout:4/442: creat d4/d13/d19/d28/d4d/d78/f95 x:0 0 0 2024-04-12T05:37:07.374 INFO:tasks.workunit.client.0.smithi032.stdout:7/515: mknod d0/d1c/d4c/caa 0 2024-04-12T05:37:07.377 INFO:tasks.workunit.client.0.smithi032.stdout:3/529: dread d12/d2e/f1e [0,4194304] 0 2024-04-12T05:37:07.377 INFO:tasks.workunit.client.0.smithi032.stdout:2/582: fdatasync dd/d19/d9d/d1c/d64/d32/f47 0 2024-04-12T05:37:07.379 INFO:tasks.workunit.client.0.smithi032.stdout:8/548: creat d0/d36/d84/db3/fc4 x:0 0 0 2024-04-12T05:37:07.383 INFO:tasks.workunit.client.0.smithi032.stdout:3/530: symlink d12/d2e/d3a/d2d/lab 0 2024-04-12T05:37:07.384 INFO:tasks.workunit.client.0.smithi032.stdout:8/549: write d0/d36/d84/db3/fc4 [183711,96155] 0 2024-04-12T05:37:07.384 INFO:tasks.workunit.client.0.smithi032.stdout:4/443: getdents d4/d13/d19/d28/d2f/d3b 0 2024-04-12T05:37:07.386 INFO:tasks.workunit.client.0.smithi032.stdout:7/516: dwrite d0/de/df/f1e [4194304,4194304] 0 2024-04-12T05:37:07.387 INFO:tasks.workunit.client.0.smithi032.stdout:4/444: chown d4/l29 21 1 2024-04-12T05:37:07.391 INFO:tasks.workunit.client.0.smithi032.stdout:7/517: write d0/d13/d21/f57 [3876573,117785] 0 2024-04-12T05:37:07.391 INFO:tasks.workunit.client.0.smithi032.stdout:2/583: dwrite dd/d19/d82/f8a [0,4194304] 0 2024-04-12T05:37:07.392 INFO:tasks.workunit.client.0.smithi032.stdout:3/531: mkdir d12/d2e/d3a/d63/da6/dac 0 2024-04-12T05:37:07.394 INFO:tasks.workunit.client.0.smithi032.stdout:8/550: mknod d0/db/dd/d11/d1b/d96/cc5 0 2024-04-12T05:37:07.395 INFO:tasks.workunit.client.0.smithi032.stdout:4/445: chown d4/d13/d19/l77 110713054 1 2024-04-12T05:37:07.396 INFO:tasks.workunit.client.0.smithi032.stdout:7/518: creat d0/d1c/d4c/fab x:0 0 0 2024-04-12T05:37:07.397 INFO:tasks.workunit.client.0.smithi032.stdout:3/532: creat d12/d2e/d98/fad x:0 0 0 2024-04-12T05:37:07.400 INFO:tasks.workunit.client.0.smithi032.stdout:2/584: rename dd/d19/d9d/d1c/d64/d32/d3a/c69 to dd/d19/d9d/d42/d4a/cb5 0 2024-04-12T05:37:07.400 INFO:tasks.workunit.client.0.smithi032.stdout:2/585: readlink dd/d19/d9d/d1c/d64/d32/d6d/l72 0 2024-04-12T05:37:07.401 INFO:tasks.workunit.client.0.smithi032.stdout:8/551: readlink d0/db/dd/d11/d1b/l70 0 2024-04-12T05:37:07.405 INFO:tasks.workunit.client.0.smithi032.stdout:8/552: mkdir d0/db/dd/d11/d1b/d5f/daf/dc6 0 2024-04-12T05:37:07.407 INFO:tasks.workunit.client.0.smithi032.stdout:4/446: rename d4/d13/d19/d28 to d4/d87/d96 0 2024-04-12T05:37:07.409 INFO:tasks.workunit.client.0.smithi032.stdout:3/533: creat d12/d2e/d3a/fae x:0 0 0 2024-04-12T05:37:07.410 INFO:tasks.workunit.client.0.smithi032.stdout:4/447: write d4/f79 [4595548,64537] 0 2024-04-12T05:37:07.410 INFO:tasks.workunit.client.0.smithi032.stdout:8/553: creat d0/db/dd/d11/d1b/d22/d54/da8/fc7 x:0 0 0 2024-04-12T05:37:07.412 INFO:tasks.workunit.client.0.smithi032.stdout:8/554: creat d0/d72/fc8 x:0 0 0 2024-04-12T05:37:07.414 INFO:tasks.workunit.client.0.smithi032.stdout:4/448: symlink d4/d87/d96/l97 0 2024-04-12T05:37:07.418 INFO:tasks.workunit.client.0.smithi032.stdout:2/586: write dd/d19/d9d/d1c/d64/f96 [716644,102366] 0 2024-04-12T05:37:07.420 INFO:tasks.workunit.client.0.smithi032.stdout:2/587: write dd/d19/d20/f28 [5383443,89984] 0 2024-04-12T05:37:07.420 INFO:tasks.workunit.client.0.smithi032.stdout:4/449: rename d4/d13/d19/f39 to d4/f98 0 2024-04-12T05:37:07.423 INFO:tasks.workunit.client.0.smithi032.stdout:3/534: dread d12/f21 [0,4194304] 0 2024-04-12T05:37:07.424 INFO:tasks.workunit.client.0.smithi032.stdout:4/450: chown d4/d87/d96/d2f/d3b 3336 1 2024-04-12T05:37:07.424 INFO:tasks.workunit.client.0.smithi032.stdout:8/555: dwrite d0/db/dd/d11/d1b/d5f/f67 [0,4194304] 0 2024-04-12T05:37:07.425 INFO:tasks.workunit.client.0.smithi032.stdout:2/588: creat dd/d19/d9d/d1c/fb6 x:0 0 0 2024-04-12T05:37:07.426 INFO:tasks.workunit.client.0.smithi032.stdout:3/535: fsync d12/d2e/d3a/fa9 0 2024-04-12T05:37:07.428 INFO:tasks.workunit.client.0.smithi032.stdout:8/556: creat d0/db/dd/d11/d1b/fc9 x:0 0 0 2024-04-12T05:37:07.429 INFO:tasks.workunit.client.0.smithi032.stdout:4/451: symlink d4/d87/d96/d2f/d7c/l99 0 2024-04-12T05:37:07.430 INFO:tasks.workunit.client.0.smithi032.stdout:4/452: write d4/d13/d44/f91 [902611,8441] 0 2024-04-12T05:37:07.431 INFO:tasks.workunit.client.0.smithi032.stdout:4/453: stat d4/d13/f71 0 2024-04-12T05:37:07.431 INFO:tasks.workunit.client.0.smithi032.stdout:8/557: creat d0/d36/fca x:0 0 0 2024-04-12T05:37:07.432 INFO:tasks.workunit.client.0.smithi032.stdout:3/536: getdents d12/d2e/d98 0 2024-04-12T05:37:07.433 INFO:tasks.workunit.client.0.smithi032.stdout:8/558: readlink d0/db/dd/d11/d1b/d5f/lb5 0 2024-04-12T05:37:07.435 INFO:tasks.workunit.client.0.smithi032.stdout:4/454: dread d4/f98 [0,4194304] 0 2024-04-12T05:37:07.437 INFO:tasks.workunit.client.0.smithi032.stdout:3/537: creat d12/d2e/d3a/d63/da6/dac/faf x:0 0 0 2024-04-12T05:37:07.439 INFO:tasks.workunit.client.0.smithi032.stdout:3/538: write d12/f56 [986920,77864] 0 2024-04-12T05:37:07.441 INFO:tasks.workunit.client.0.smithi032.stdout:8/559: dwrite d0/d72/f7a [0,4194304] 0 2024-04-12T05:37:07.442 INFO:tasks.workunit.client.0.smithi032.stdout:3/539: mknod d12/d2e/d3a/cb0 0 2024-04-12T05:37:07.447 INFO:tasks.workunit.client.0.smithi032.stdout:3/540: dread d12/d2e/f42 [0,4194304] 0 2024-04-12T05:37:07.451 INFO:tasks.workunit.client.0.smithi032.stdout:3/541: unlink d12/d2e/d3a/f7f 0 2024-04-12T05:37:07.452 INFO:tasks.workunit.client.0.smithi032.stdout:8/560: truncate d0/db/dd/d11/d1b/d22/d29/f31 1202339 0 2024-04-12T05:37:07.453 INFO:tasks.workunit.client.0.smithi032.stdout:8/561: fsync d0/d72/fc8 0 2024-04-12T05:37:07.455 INFO:tasks.workunit.client.0.smithi032.stdout:8/562: mkdir d0/d36/d84/dad/da4/dcb 0 2024-04-12T05:37:07.455 INFO:tasks.workunit.client.0.smithi032.stdout:3/542: link d12/d2e/d3a/c62 d12/d2e/cb1 0 2024-04-12T05:37:07.458 INFO:tasks.workunit.client.0.smithi032.stdout:3/543: rename d12/d2e/f5a to d12/d2e/d3a/d63/fb2 0 2024-04-12T05:37:07.461 INFO:tasks.workunit.client.0.smithi032.stdout:4/455: dread d4/d13/f1c [0,4194304] 0 2024-04-12T05:37:07.461 INFO:tasks.workunit.client.0.smithi032.stdout:3/544: rename d12/d2f/d5b to d12/daa/db3 0 2024-04-12T05:37:07.462 INFO:tasks.workunit.client.0.smithi032.stdout:4/456: chown d4/d87/d96/d2f/d3b/f88 3 1 2024-04-12T05:37:07.462 INFO:tasks.workunit.client.0.smithi032.stdout:3/545: fdatasync d12/d2e/d3a/d63/da6/dac/faf 0 2024-04-12T05:37:07.463 INFO:tasks.workunit.client.0.smithi032.stdout:3/546: chown d12/d2e/d3a/d2d/f45 0 1 2024-04-12T05:37:07.464 INFO:tasks.workunit.client.0.smithi032.stdout:8/563: unlink d0/db/f6e 0 2024-04-12T05:37:07.465 INFO:tasks.workunit.client.0.smithi032.stdout:4/457: mkdir d4/d87/d96/d2f/d9a 0 2024-04-12T05:37:07.465 INFO:tasks.workunit.client.0.smithi032.stdout:4/458: stat d4/cb 0 2024-04-12T05:37:07.466 INFO:tasks.workunit.client.0.smithi032.stdout:4/459: readlink d4/l29 0 2024-04-12T05:37:07.467 INFO:tasks.workunit.client.0.smithi032.stdout:3/547: unlink d12/d2e/d3a/d63/la0 0 2024-04-12T05:37:07.470 INFO:tasks.workunit.client.0.smithi032.stdout:4/460: mkdir d4/d87/d96/d68/d9b 0 2024-04-12T05:37:07.471 INFO:tasks.workunit.client.0.smithi032.stdout:3/548: mknod d12/d2f/d89/cb4 0 2024-04-12T05:37:07.471 INFO:tasks.workunit.client.0.smithi032.stdout:1/473: sync 2024-04-12T05:37:07.474 INFO:tasks.workunit.client.0.smithi032.stdout:4/461: rmdir d4/d87 39 2024-04-12T05:37:07.476 INFO:tasks.workunit.client.0.smithi032.stdout:1/474: rename d5/de/d17/l22 to d5/de/d17/d35/d61/d77/l96 0 2024-04-12T05:37:07.480 INFO:tasks.workunit.client.0.smithi032.stdout:8/564: truncate d0/db/dd/d11/d1b/d5f/f67 3766484 0 2024-04-12T05:37:07.481 INFO:tasks.workunit.client.0.smithi032.stdout:3/549: creat d12/d2e/fb5 x:0 0 0 2024-04-12T05:37:07.483 INFO:tasks.workunit.client.0.smithi032.stdout:8/565: mknod d0/ccc 0 2024-04-12T05:37:07.484 INFO:tasks.workunit.client.0.smithi032.stdout:1/475: link d5/de/d17/d35/d78/f7a d5/de/d2a/f97 0 2024-04-12T05:37:07.484 INFO:tasks.workunit.client.0.smithi032.stdout:3/550: write d12/d2e/d3a/f26 [2642112,35790] 0 2024-04-12T05:37:07.486 INFO:tasks.workunit.client.0.smithi032.stdout:4/462: dwrite d4/d87/d96/d2f/f70 [0,4194304] 0 2024-04-12T05:37:07.488 INFO:tasks.workunit.client.0.smithi032.stdout:3/551: creat d12/d2f/d89/d9d/fb6 x:0 0 0 2024-04-12T05:37:07.489 INFO:tasks.workunit.client.0.smithi032.stdout:4/463: write d4/d87/d96/f52 [4257787,73671] 0 2024-04-12T05:37:07.490 INFO:tasks.workunit.client.0.smithi032.stdout:1/476: rename d5/de/f11 to d5/de/d17/d35/d3b/f98 0 2024-04-12T05:37:07.495 INFO:tasks.workunit.client.0.smithi032.stdout:4/464: dwrite d4/d87/d96/d4d/d78/f51 [0,4194304] 0 2024-04-12T05:37:07.496 INFO:tasks.workunit.client.0.smithi032.stdout:1/477: write d5/de/d17/d35/d3b/d49/f56 [2400806,7068] 0 2024-04-12T05:37:07.499 INFO:tasks.workunit.client.0.smithi032.stdout:8/566: rename d0/db/dd/d11/d1b/d22/f2b to d0/db/dd/fcd 0 2024-04-12T05:37:07.501 INFO:tasks.workunit.client.0.smithi032.stdout:8/567: fsync d0/db/dd/d11/d1b/f78 0 2024-04-12T05:37:07.506 INFO:tasks.workunit.client.0.smithi032.stdout:4/465: mknod d4/d87/d96/d2f/d94/c9c 0 2024-04-12T05:37:07.509 INFO:tasks.workunit.client.0.smithi032.stdout:4/466: mknod d4/d87/d96/d68/c9d 0 2024-04-12T05:37:07.511 INFO:tasks.workunit.client.0.smithi032.stdout:4/467: creat d4/d87/d96/d2f/d3b/f9e x:0 0 0 2024-04-12T05:37:07.512 INFO:tasks.workunit.client.0.smithi032.stdout:3/552: truncate d12/d2f/f81 1853407 0 2024-04-12T05:37:07.514 INFO:tasks.workunit.client.0.smithi032.stdout:8/568: dread d0/d36/f4a [0,4194304] 0 2024-04-12T05:37:07.519 INFO:tasks.workunit.client.0.smithi032.stdout:4/468: rename d4/d87/d96/d4d/l67 to d4/d13/d44/l9f 0 2024-04-12T05:37:07.525 INFO:tasks.workunit.client.0.smithi032.stdout:8/569: dread d0/f2f [0,4194304] 0 2024-04-12T05:37:07.525 INFO:tasks.workunit.client.0.smithi032.stdout:3/553: dread d12/d2e/d3a/f26 [0,4194304] 0 2024-04-12T05:37:07.527 INFO:tasks.workunit.client.0.smithi032.stdout:4/469: write d4/f18 [16543,4996] 0 2024-04-12T05:37:07.532 INFO:tasks.workunit.client.0.smithi032.stdout:1/478: dread d5/f47 [4194304,4194304] 0 2024-04-12T05:37:07.533 INFO:tasks.workunit.client.0.smithi032.stdout:3/554: dwrite d12/d2e/f86 [0,4194304] 0 2024-04-12T05:37:07.533 INFO:tasks.workunit.client.0.smithi032.stdout:1/479: fsync d5/de/f7e 0 2024-04-12T05:37:07.536 INFO:tasks.workunit.client.0.smithi032.stdout:1/480: symlink d5/de/d17/d35/d61/d77/l99 0 2024-04-12T05:37:07.538 INFO:tasks.workunit.client.0.smithi032.stdout:3/555: getdents d12/d2f/d89/d9d 0 2024-04-12T05:37:07.540 INFO:tasks.workunit.client.0.smithi032.stdout:3/556: write d12/d2f/f41 [795619,14543] 0 2024-04-12T05:37:07.543 INFO:tasks.workunit.client.0.smithi032.stdout:1/481: dwrite d5/d2c/f43 [4194304,4194304] 0 2024-04-12T05:37:07.544 INFO:tasks.workunit.client.0.smithi032.stdout:3/557: creat d12/d2e/d3a/d63/da6/fb7 x:0 0 0 2024-04-12T05:37:07.548 INFO:tasks.workunit.client.0.smithi032.stdout:0/581: sync 2024-04-12T05:37:07.549 INFO:tasks.workunit.client.0.smithi032.stdout:9/442: sync 2024-04-12T05:37:07.549 INFO:tasks.workunit.client.0.smithi032.stdout:6/636: sync 2024-04-12T05:37:07.549 INFO:tasks.workunit.client.0.smithi032.stdout:3/558: getdents d12/d2f/d89/d9d 0 2024-04-12T05:37:07.550 INFO:tasks.workunit.client.0.smithi032.stdout:1/482: getdents d5/de/d17/d35/d3b 0 2024-04-12T05:37:07.551 INFO:tasks.workunit.client.0.smithi032.stdout:6/637: truncate d4/d14/d1a/d92/d9e/fd1 370397 0 2024-04-12T05:37:07.551 INFO:tasks.workunit.client.0.smithi032.stdout:3/559: readlink d12/d2f/l43 0 2024-04-12T05:37:07.552 INFO:tasks.workunit.client.0.smithi032.stdout:1/483: readlink d5/d2c/d4f/l6a 0 2024-04-12T05:37:07.553 INFO:tasks.workunit.client.0.smithi032.stdout:3/560: fdatasync d12/d2f/d89/fa7 0 2024-04-12T05:37:07.555 INFO:tasks.workunit.client.0.smithi032.stdout:1/484: write d5/de/f28 [2066210,2351] 0 2024-04-12T05:37:07.555 INFO:tasks.workunit.client.0.smithi032.stdout:0/582: link d2/dc/d3e/c75 d2/d28/cbe 0 2024-04-12T05:37:07.555 INFO:tasks.workunit.client.0.smithi032.stdout:3/561: write d12/d2e/d3a/d63/da6/fb7 [236264,31046] 0 2024-04-12T05:37:07.556 INFO:tasks.workunit.client.0.smithi032.stdout:6/638: read d4/d21/d2e/d32/f61 [754413,40462] 0 2024-04-12T05:37:07.557 INFO:tasks.workunit.client.0.smithi032.stdout:0/583: fsync d2/d28/d87/d90/f92 0 2024-04-12T05:37:07.558 INFO:tasks.workunit.client.0.smithi032.stdout:1/485: mknod d5/de/d17/c9a 0 2024-04-12T05:37:07.559 INFO:tasks.workunit.client.0.smithi032.stdout:3/562: mknod d12/daa/db3/cb8 0 2024-04-12T05:37:07.560 INFO:tasks.workunit.client.0.smithi032.stdout:1/486: fdatasync d5/de/d17/d35/d78/f7c 0 2024-04-12T05:37:07.561 INFO:tasks.workunit.client.0.smithi032.stdout:6/639: write d4/d21/d2e/d32/f61 [4567138,96551] 0 2024-04-12T05:37:07.562 INFO:tasks.workunit.client.0.smithi032.stdout:4/470: dread d4/d87/d96/f52 [0,4194304] 0 2024-04-12T05:37:07.562 INFO:tasks.workunit.client.0.smithi032.stdout:0/584: rename d2/fe to d2/d29/d8a/d99/fbf 0 2024-04-12T05:37:07.563 INFO:tasks.workunit.client.0.smithi032.stdout:1/487: chown d5/de/d17/d35/d61/d77/l99 456 1 2024-04-12T05:37:07.565 INFO:tasks.workunit.client.0.smithi032.stdout:4/471: creat d4/d13/d44/fa0 x:0 0 0 2024-04-12T05:37:07.567 INFO:tasks.workunit.client.0.smithi032.stdout:1/488: dread d5/d2c/f3c [0,4194304] 0 2024-04-12T05:37:07.570 INFO:tasks.workunit.client.0.smithi032.stdout:6/640: creat d4/d14/d1a/d37/d69/d8a/fd5 x:0 0 0 2024-04-12T05:37:07.571 INFO:tasks.workunit.client.0.smithi032.stdout:0/585: rmdir d2/d29/d8a/d99 39 2024-04-12T05:37:07.575 INFO:tasks.workunit.client.0.smithi032.stdout:3/563: rename d12/d2e/f6a to d12/d2e/fb9 0 2024-04-12T05:37:07.576 INFO:tasks.workunit.client.0.smithi032.stdout:1/489: mknod d5/de/d17/d35/d61/d77/c9b 0 2024-04-12T05:37:07.579 INFO:tasks.workunit.client.0.smithi032.stdout:6/641: unlink d4/d14/d1a/d37/d69/ca6 0 2024-04-12T05:37:07.582 INFO:tasks.workunit.client.0.smithi032.stdout:3/564: creat d12/d2f/d89/fba x:0 0 0 2024-04-12T05:37:07.584 INFO:tasks.workunit.client.0.smithi032.stdout:0/586: creat d2/d8/d66/fc0 x:0 0 0 2024-04-12T05:37:07.586 INFO:tasks.workunit.client.0.smithi032.stdout:4/472: creat d4/d13/d19/fa1 x:0 0 0 2024-04-12T05:37:07.587 INFO:tasks.workunit.client.0.smithi032.stdout:1/490: mkdir d5/de/d17/d35/d61/d77/d8f/d9c 0 2024-04-12T05:37:07.587 INFO:tasks.workunit.client.0.smithi032.stdout:3/565: mkdir d12/d24/dbb 0 2024-04-12T05:37:07.588 INFO:tasks.workunit.client.0.smithi032.stdout:0/587: unlink d2/d8/df/c34 0 2024-04-12T05:37:07.595 INFO:tasks.workunit.client.0.smithi032.stdout:3/566: chown d12/d2e/d3a/f3b 13668 1 2024-04-12T05:37:07.595 INFO:tasks.workunit.client.0.smithi032.stdout:0/588: chown d2/d29/f65 4247030 1 2024-04-12T05:37:07.596 INFO:tasks.workunit.client.0.smithi032.stdout:6/642: dwrite d4/d21/d27/f4b [0,4194304] 0 2024-04-12T05:37:07.598 INFO:tasks.workunit.client.0.smithi032.stdout:0/589: dread - d2/d8/d66/fc0 zero size 2024-04-12T05:37:07.598 INFO:tasks.workunit.client.0.smithi032.stdout:1/491: getdents d5/de/d17 0 2024-04-12T05:37:07.599 INFO:tasks.workunit.client.0.smithi032.stdout:6/643: chown d4/d21/d2e/d32/d94/fa9 6471 1 2024-04-12T05:37:07.601 INFO:tasks.workunit.client.0.smithi032.stdout:1/492: chown d5/de/d2a/f3f 302894431 1 2024-04-12T05:37:07.601 INFO:tasks.workunit.client.0.smithi032.stdout:0/590: chown d2/d28/f8e 437 1 2024-04-12T05:37:07.602 INFO:tasks.workunit.client.0.smithi032.stdout:0/591: readlink d2/d29/d8a/laa 0 2024-04-12T05:37:07.603 INFO:tasks.workunit.client.0.smithi032.stdout:1/493: truncate d5/de/d5b/f95 32634 0 2024-04-12T05:37:07.604 INFO:tasks.workunit.client.0.smithi032.stdout:3/567: rename d12/d24/la8 to d12/d2e/d3a/d63/da6/dac/lbc 0 2024-04-12T05:37:07.604 INFO:tasks.workunit.client.0.smithi032.stdout:6/644: truncate d4/f2a 2057666 0 2024-04-12T05:37:07.606 INFO:tasks.workunit.client.0.smithi032.stdout:6/645: chown d4/d14/d1a/d37/d47/la4 773600 1 2024-04-12T05:37:07.607 INFO:tasks.workunit.client.0.smithi032.stdout:3/568: unlink d12/daa/db3/l6c 0 2024-04-12T05:37:07.608 INFO:tasks.workunit.client.0.smithi032.stdout:1/494: mknod d5/de/c9d 0 2024-04-12T05:37:07.608 INFO:tasks.workunit.client.0.smithi032.stdout:6/646: mknod d4/d74/cd6 0 2024-04-12T05:37:07.610 INFO:tasks.workunit.client.0.smithi032.stdout:3/569: creat d12/d2e/d3a/d2d/fbd x:0 0 0 2024-04-12T05:37:07.610 INFO:tasks.workunit.client.0.smithi032.stdout:6/647: stat d4/d14/d1a/d37/d47/c5b 0 2024-04-12T05:37:07.612 INFO:tasks.workunit.client.0.smithi032.stdout:0/592: dread d2/d8/df/d24/f5c [0,4194304] 0 2024-04-12T05:37:07.612 INFO:tasks.workunit.client.0.smithi032.stdout:4/473: dread d4/d87/d96/d2f/d3b/f6b [0,4194304] 0 2024-04-12T05:37:07.613 INFO:tasks.workunit.client.0.smithi032.stdout:0/593: write d2/f6 [3551319,119448] 0 2024-04-12T05:37:07.615 INFO:tasks.workunit.client.0.smithi032.stdout:6/648: creat d4/d21/fd7 x:0 0 0 2024-04-12T05:37:07.616 INFO:tasks.workunit.client.0.smithi032.stdout:4/474: mkdir d4/d87/d96/d2f/d3b/da2 0 2024-04-12T05:37:07.622 INFO:tasks.workunit.client.0.smithi032.stdout:3/570: dwrite d12/d2f/f91 [0,4194304] 0 2024-04-12T05:37:07.622 INFO:tasks.workunit.client.0.smithi032.stdout:0/594: rename d2/d8/df/la2 to d2/dc/d41/lc1 0 2024-04-12T05:37:07.624 INFO:tasks.workunit.client.0.smithi032.stdout:6/649: symlink d4/d14/d1a/d88/ld8 0 2024-04-12T05:37:07.624 INFO:tasks.workunit.client.0.smithi032.stdout:4/475: dwrite d4/f54 [0,4194304] 0 2024-04-12T05:37:07.627 INFO:tasks.workunit.client.0.smithi032.stdout:3/571: rmdir d12/d2e/d3a/d63/da6 39 2024-04-12T05:37:07.628 INFO:tasks.workunit.client.0.smithi032.stdout:3/572: stat d12/d2f/c5d 0 2024-04-12T05:37:07.628 INFO:tasks.workunit.client.0.smithi032.stdout:0/595: fdatasync d2/d28/d87/d90/fb1 0 2024-04-12T05:37:07.629 INFO:tasks.workunit.client.0.smithi032.stdout:1/495: truncate d5/de/d17/d1b/f38 1265340 0 2024-04-12T05:37:07.633 INFO:tasks.workunit.client.0.smithi032.stdout:6/650: mkdir d4/d14/d1a/dd9 0 2024-04-12T05:37:07.634 INFO:tasks.workunit.client.0.smithi032.stdout:4/476: getdents d4/d87/d96/d2f/d3b/da2 0 2024-04-12T05:37:07.637 INFO:tasks.workunit.client.0.smithi032.stdout:1/496: truncate d5/de/d17/d35/d61/f50 1022648 0 2024-04-12T05:37:07.639 INFO:tasks.workunit.client.0.smithi032.stdout:0/596: dwrite d2/d29/f47 [4194304,4194304] 0 2024-04-12T05:37:07.640 INFO:tasks.workunit.client.0.smithi032.stdout:6/651: chown d4/d21/d27/dad/d55/d59/f9f 36329 1 2024-04-12T05:37:07.641 INFO:tasks.workunit.client.0.smithi032.stdout:4/477: mknod d4/d87/d96/d2f/d3b/ca3 0 2024-04-12T05:37:07.642 INFO:tasks.workunit.client.0.smithi032.stdout:1/497: mknod d5/de/d17/d35/d73/d74/c9e 0 2024-04-12T05:37:07.644 INFO:tasks.workunit.client.0.smithi032.stdout:0/597: truncate d2/dc/fa6 1936949 0 2024-04-12T05:37:07.646 INFO:tasks.workunit.client.0.smithi032.stdout:3/573: dread d12/d2e/d3a/d63/f8a [0,4194304] 0 2024-04-12T05:37:07.648 INFO:tasks.workunit.client.0.smithi032.stdout:0/598: dread d2/d28/f2e [0,4194304] 0 2024-04-12T05:37:07.649 INFO:tasks.workunit.client.0.smithi032.stdout:6/652: truncate d4/ff 68606 0 2024-04-12T05:37:07.650 INFO:tasks.workunit.client.0.smithi032.stdout:4/478: truncate d4/d87/d96/d4d/d78/f6d 1663834 0 2024-04-12T05:37:07.651 INFO:tasks.workunit.client.0.smithi032.stdout:1/498: symlink d5/de/l9f 0 2024-04-12T05:37:07.652 INFO:tasks.workunit.client.0.smithi032.stdout:1/499: fdatasync d5/de/d17/d35/d3b/d49/f56 0 2024-04-12T05:37:07.652 INFO:tasks.workunit.client.0.smithi032.stdout:0/599: mkdir d2/d29/d77/dc2 0 2024-04-12T05:37:07.653 INFO:tasks.workunit.client.0.smithi032.stdout:6/653: mknod d4/d14/d1a/d37/d47/cda 0 2024-04-12T05:37:07.653 INFO:tasks.workunit.client.0.smithi032.stdout:1/500: stat d5/de/d17/d1b 0 2024-04-12T05:37:07.653 INFO:tasks.workunit.client.0.smithi032.stdout:0/600: readlink d2/d8/df/d24/l9b 0 2024-04-12T05:37:07.656 INFO:tasks.workunit.client.0.smithi032.stdout:6/654: rename d4/d21/d27 to d4/d21/d27/dad/d55/d59/ddb 22 2024-04-12T05:37:07.656 INFO:tasks.workunit.client.0.smithi032.stdout:1/501: write d5/de/d17/d35/d78/f83 [3527083,30879] 0 2024-04-12T05:37:07.656 INFO:tasks.workunit.client.0.smithi032.stdout:0/601: mkdir d2/d8/df/d24/dc3 0 2024-04-12T05:37:07.659 INFO:tasks.workunit.client.0.smithi032.stdout:6/655: truncate d4/d21/fd7 881460 0 2024-04-12T05:37:07.662 INFO:tasks.workunit.client.0.smithi032.stdout:1/502: truncate d5/de/d17/d35/d78/f83 4885240 0 2024-04-12T05:37:07.662 INFO:tasks.workunit.client.0.smithi032.stdout:4/479: dwrite d4/d87/d96/d2f/d3b/f5d [0,4194304] 0 2024-04-12T05:37:07.664 INFO:tasks.workunit.client.0.smithi032.stdout:6/656: symlink d4/d14/d25/ldc 0 2024-04-12T05:37:07.665 INFO:tasks.workunit.client.0.smithi032.stdout:1/503: write d5/de/f28 [3303122,95048] 0 2024-04-12T05:37:07.666 INFO:tasks.workunit.client.0.smithi032.stdout:4/480: write d4/d87/d96/d2f/f70 [1631210,81450] 0 2024-04-12T05:37:07.666 INFO:tasks.workunit.client.0.smithi032.stdout:6/657: chown d4/d14/c64 29 1 2024-04-12T05:37:07.667 INFO:tasks.workunit.client.0.smithi032.stdout:1/504: stat d5/de/d17/d35/d3b/d49 0 2024-04-12T05:37:07.668 INFO:tasks.workunit.client.0.smithi032.stdout:4/481: truncate d4/d13/f43 4219290 0 2024-04-12T05:37:07.673 INFO:tasks.workunit.client.0.smithi032.stdout:4/482: symlink d4/d87/d96/d4d/la4 0 2024-04-12T05:37:07.675 INFO:tasks.workunit.client.0.smithi032.stdout:6/658: dwrite d4/d21/db4/fcd [0,4194304] 0 2024-04-12T05:37:07.676 INFO:tasks.workunit.client.0.smithi032.stdout:4/483: rmdir d4/d13/d19 39 2024-04-12T05:37:07.678 INFO:tasks.workunit.client.0.smithi032.stdout:5/578: sync 2024-04-12T05:37:07.678 INFO:tasks.workunit.client.0.smithi032.stdout:7/519: sync 2024-04-12T05:37:07.678 INFO:tasks.workunit.client.0.smithi032.stdout:2/589: sync 2024-04-12T05:37:07.678 INFO:tasks.workunit.client.0.smithi032.stdout:3/574: dread d12/d2f/f41 [0,4194304] 0 2024-04-12T05:37:07.682 INFO:tasks.workunit.client.0.smithi032.stdout:7/520: chown d0/d13/d21/l92 262 1 2024-04-12T05:37:07.682 INFO:tasks.workunit.client.0.smithi032.stdout:5/579: dread - d2/df/d25/d50/f8e zero size 2024-04-12T05:37:07.685 INFO:tasks.workunit.client.0.smithi032.stdout:0/602: dwrite d2/d8/df/f5f [0,4194304] 0 2024-04-12T05:37:07.685 INFO:tasks.workunit.client.0.smithi032.stdout:3/575: fsync d12/d2e/d3a/d63/da6/fb7 0 2024-04-12T05:37:07.687 INFO:tasks.workunit.client.0.smithi032.stdout:5/580: symlink d2/dd/d6f/lb6 0 2024-04-12T05:37:07.688 INFO:tasks.workunit.client.0.smithi032.stdout:4/484: dread d4/f54 [0,4194304] 0 2024-04-12T05:37:07.690 INFO:tasks.workunit.client.0.smithi032.stdout:0/603: dread - d2/d8/f6c zero size 2024-04-12T05:37:07.691 INFO:tasks.workunit.client.0.smithi032.stdout:3/576: mkdir d12/d2e/d3a/dbe 0 2024-04-12T05:37:07.694 INFO:tasks.workunit.client.0.smithi032.stdout:5/581: read d2/df/f5c [44363,79569] 0 2024-04-12T05:37:07.694 INFO:tasks.workunit.client.0.smithi032.stdout:2/590: rename dd/d19/d9d/c5d to dd/d19/d9d/d1c/d64/cb7 0 2024-04-12T05:37:07.696 INFO:tasks.workunit.client.0.smithi032.stdout:3/577: symlink d12/d2f/d89/d9d/lbf 0 2024-04-12T05:37:07.696 INFO:tasks.workunit.client.0.smithi032.stdout:7/521: dwrite d0/de/df/d29/f46 [0,4194304] 0 2024-04-12T05:37:07.698 INFO:tasks.workunit.client.0.smithi032.stdout:5/582: creat d2/d9b/fb7 x:0 0 0 2024-04-12T05:37:07.700 INFO:tasks.workunit.client.0.smithi032.stdout:3/578: symlink d12/daa/lc0 0 2024-04-12T05:37:07.700 INFO:tasks.workunit.client.0.smithi032.stdout:0/604: getdents d2/d29/d77 0 2024-04-12T05:37:07.702 INFO:tasks.workunit.client.0.smithi032.stdout:7/522: mknod d0/d13/d21/cac 0 2024-04-12T05:37:07.704 INFO:tasks.workunit.client.0.smithi032.stdout:3/579: read d12/d2e/d3a/d63/f8a [1646137,20391] 0 2024-04-12T05:37:07.708 INFO:tasks.workunit.client.0.smithi032.stdout:0/605: creat d2/d8/d17/fc4 x:0 0 0 2024-04-12T05:37:07.708 INFO:tasks.workunit.client.0.smithi032.stdout:7/523: creat d0/dd/d51/d65/fad x:0 0 0 2024-04-12T05:37:07.709 INFO:tasks.workunit.client.0.smithi032.stdout:5/583: dwrite d2/df/d25/f98 [0,4194304] 0 2024-04-12T05:37:07.710 INFO:tasks.workunit.client.0.smithi032.stdout:3/580: chown d12/l64 9561 1 2024-04-12T05:37:07.711 INFO:tasks.workunit.client.0.smithi032.stdout:4/485: dwrite d4/d13/d19/f2a [0,4194304] 0 2024-04-12T05:37:07.712 INFO:tasks.workunit.client.0.smithi032.stdout:7/524: truncate d0/dd/f98 679150 0 2024-04-12T05:37:07.714 INFO:tasks.workunit.client.0.smithi032.stdout:5/584: truncate d2/df/f3f 1958263 0 2024-04-12T05:37:07.715 INFO:tasks.workunit.client.0.smithi032.stdout:4/486: write d4/d87/d96/d4d/d78/f64 [482674,52109] 0 2024-04-12T05:37:07.715 INFO:tasks.workunit.client.0.smithi032.stdout:7/525: chown d0/d13/d21/d30/d31/l82 4488 1 2024-04-12T05:37:07.717 INFO:tasks.workunit.client.0.smithi032.stdout:4/487: chown d4/d13/f1f 368176061 1 2024-04-12T05:37:07.720 INFO:tasks.workunit.client.0.smithi032.stdout:5/585: symlink d2/d65/d78/lb8 0 2024-04-12T05:37:07.721 INFO:tasks.workunit.client.0.smithi032.stdout:4/488: mknod d4/d13/d44/ca5 0 2024-04-12T05:37:07.727 INFO:tasks.workunit.client.0.smithi032.stdout:3/581: dwrite d12/d2e/f67 [0,4194304] 0 2024-04-12T05:37:07.727 INFO:tasks.workunit.client.0.smithi032.stdout:4/489: symlink d4/d13/d44/la6 0 2024-04-12T05:37:07.731 INFO:tasks.workunit.client.0.smithi032.stdout:3/582: dread - d12/d2e/f99 zero size 2024-04-12T05:37:07.734 INFO:tasks.workunit.client.0.smithi032.stdout:7/526: dwrite d0/d13/d21/d7c/fa8 [0,4194304] 0 2024-04-12T05:37:07.734 INFO:tasks.workunit.client.0.smithi032.stdout:0/606: dread d2/d29/f70 [0,4194304] 0 2024-04-12T05:37:07.739 INFO:tasks.workunit.client.0.smithi032.stdout:7/527: creat d0/d76/fae x:0 0 0 2024-04-12T05:37:07.739 INFO:tasks.workunit.client.0.smithi032.stdout:4/490: write d4/d13/f1c [2211329,49402] 0 2024-04-12T05:37:07.741 INFO:tasks.workunit.client.0.smithi032.stdout:3/583: truncate d12/daa/db3/f87 2893805 0 2024-04-12T05:37:07.742 INFO:tasks.workunit.client.0.smithi032.stdout:4/491: write d4/f79 [3056142,30388] 0 2024-04-12T05:37:07.743 INFO:tasks.workunit.client.0.smithi032.stdout:5/586: dwrite d2/f24 [0,4194304] 0 2024-04-12T05:37:07.744 INFO:tasks.workunit.client.0.smithi032.stdout:0/607: getdents d2/dc 0 2024-04-12T05:37:07.746 INFO:tasks.workunit.client.0.smithi032.stdout:7/528: getdents d0 0 2024-04-12T05:37:07.748 INFO:tasks.workunit.client.0.smithi032.stdout:0/608: fsync d2/dc/d3e/dac/d51/f8d 0 2024-04-12T05:37:07.751 INFO:tasks.workunit.client.0.smithi032.stdout:4/492: readlink d4/d87/d96/d32/l65 0 2024-04-12T05:37:07.751 INFO:tasks.workunit.client.0.smithi032.stdout:3/584: chown d12/d2e/d3a/d63/da6/dac/lbc 133919055 1 2024-04-12T05:37:07.755 INFO:tasks.workunit.client.0.smithi032.stdout:0/609: getdents d2 0 2024-04-12T05:37:07.755 INFO:tasks.workunit.client.0.smithi032.stdout:4/493: chown d4/d87/d96/d2f/d7c/l99 28304463 1 2024-04-12T05:37:07.757 INFO:tasks.workunit.client.0.smithi032.stdout:7/529: dwrite d0/de/df/f8b [0,4194304] 0 2024-04-12T05:37:07.762 INFO:tasks.workunit.client.0.smithi032.stdout:3/585: mknod d12/d24/dbb/cc1 0 2024-04-12T05:37:07.762 INFO:tasks.workunit.client.0.smithi032.stdout:5/587: link d2/dd/f47 d2/dd/fb9 0 2024-04-12T05:37:07.766 INFO:tasks.workunit.client.0.smithi032.stdout:3/586: creat d12/d24/dbb/fc2 x:0 0 0 2024-04-12T05:37:07.768 INFO:tasks.workunit.client.0.smithi032.stdout:4/494: getdents d4/d13 0 2024-04-12T05:37:07.768 INFO:tasks.workunit.client.0.smithi032.stdout:5/588: mkdir d2/df/d31/da2/dba 0 2024-04-12T05:37:07.773 INFO:tasks.workunit.client.0.smithi032.stdout:3/587: getdents d12/d2e 0 2024-04-12T05:37:07.774 INFO:tasks.workunit.client.0.smithi032.stdout:5/589: rename d2/df/d25/f16 to d2/d3c/d48/d4b/fbb 0 2024-04-12T05:37:07.774 INFO:tasks.workunit.client.0.smithi032.stdout:7/530: dread d0/d10/f62 [0,4194304] 0 2024-04-12T05:37:07.774 INFO:tasks.workunit.client.0.smithi032.stdout:4/495: creat d4/d87/d96/d2f/d9a/fa7 x:0 0 0 2024-04-12T05:37:07.775 INFO:tasks.workunit.client.0.smithi032.stdout:4/496: fdatasync d4/d13/f43 0 2024-04-12T05:37:07.776 INFO:tasks.workunit.client.0.smithi032.stdout:5/590: truncate d2/d3c/d48/d4b/fa4 585032 0 2024-04-12T05:37:07.777 INFO:tasks.workunit.client.0.smithi032.stdout:8/570: sync 2024-04-12T05:37:07.777 INFO:tasks.workunit.client.0.smithi032.stdout:9/443: sync 2024-04-12T05:37:07.779 INFO:tasks.workunit.client.0.smithi032.stdout:3/588: getdents d12 0 2024-04-12T05:37:07.779 INFO:tasks.workunit.client.0.smithi032.stdout:7/531: dread d0/d13/d21/d81/f9d [0,4194304] 0 2024-04-12T05:37:07.781 INFO:tasks.workunit.client.0.smithi032.stdout:9/444: creat d9/d15/d1a/d31/f98 x:0 0 0 2024-04-12T05:37:07.782 INFO:tasks.workunit.client.0.smithi032.stdout:5/591: mknod d2/df/d25/d43/d96/cbc 0 2024-04-12T05:37:07.782 INFO:tasks.workunit.client.0.smithi032.stdout:8/571: creat d0/db/dd/d11/d1b/d22/d54/da8/fce x:0 0 0 2024-04-12T05:37:07.784 INFO:tasks.workunit.client.0.smithi032.stdout:4/497: creat d4/d13/d19/fa8 x:0 0 0 2024-04-12T05:37:07.790 INFO:tasks.workunit.client.0.smithi032.stdout:7/532: dwrite d0/dd/f2a [0,4194304] 0 2024-04-12T05:37:07.790 INFO:tasks.workunit.client.0.smithi032.stdout:5/592: truncate d2/dd/f44 3040265 0 2024-04-12T05:37:07.794 INFO:tasks.workunit.client.0.smithi032.stdout:4/498: unlink d4/d13/d19/f34 0 2024-04-12T05:37:07.796 INFO:tasks.workunit.client.0.smithi032.stdout:3/589: dwrite d12/d2e/f42 [0,4194304] 0 2024-04-12T05:37:07.797 INFO:tasks.workunit.client.0.smithi032.stdout:4/499: chown d4/d13/d19/f7a 8780023 1 2024-04-12T05:37:07.798 INFO:tasks.workunit.client.0.smithi032.stdout:9/445: dwrite d9/d1f/f39 [0,4194304] 0 2024-04-12T05:37:07.800 INFO:tasks.workunit.client.0.smithi032.stdout:4/500: write d4/d13/d19/f2a [2950714,90580] 0 2024-04-12T05:37:07.800 INFO:tasks.workunit.client.0.smithi032.stdout:9/446: chown d9/d32/d86/f8c 96369 1 2024-04-12T05:37:07.807 INFO:tasks.workunit.client.0.smithi032.stdout:5/593: creat d2/dd/d6f/fbd x:0 0 0 2024-04-12T05:37:07.808 INFO:tasks.workunit.client.0.smithi032.stdout:7/533: dwrite d0/f5 [4194304,4194304] 0 2024-04-12T05:37:07.809 INFO:tasks.workunit.client.0.smithi032.stdout:4/501: symlink d4/d87/d96/d68/la9 0 2024-04-12T05:37:07.810 INFO:tasks.workunit.client.0.smithi032.stdout:3/590: mknod d12/d2e/d3a/cc3 0 2024-04-12T05:37:07.811 INFO:tasks.workunit.client.0.smithi032.stdout:9/447: creat d9/d15/d1a/d69/f99 x:0 0 0 2024-04-12T05:37:07.813 INFO:tasks.workunit.client.0.smithi032.stdout:5/594: rename d2/df/c82 to d2/df/d31/d80/cbe 0 2024-04-12T05:37:07.814 INFO:tasks.workunit.client.0.smithi032.stdout:9/448: dread - d9/d15/d1a/d69/f99 zero size 2024-04-12T05:37:07.817 INFO:tasks.workunit.client.0.smithi032.stdout:7/534: getdents d0/d1c 0 2024-04-12T05:37:07.820 INFO:tasks.workunit.client.0.smithi032.stdout:5/595: getdents d2/df/d31 0 2024-04-12T05:37:07.821 INFO:tasks.workunit.client.0.smithi032.stdout:7/535: write d0/d1c/d4c/f4d [3265320,129563] 0 2024-04-12T05:37:07.827 INFO:tasks.workunit.client.0.smithi032.stdout:3/591: dwrite d12/d2f/d89/d9d/fb6 [0,4194304] 0 2024-04-12T05:37:07.827 INFO:tasks.workunit.client.0.smithi032.stdout:5/596: chown d2/d3c/d48/d4b/fa4 88512587 1 2024-04-12T05:37:07.829 INFO:tasks.workunit.client.0.smithi032.stdout:7/536: dread d0/d13/d21/d81/f9d [0,4194304] 0 2024-04-12T05:37:07.829 INFO:tasks.workunit.client.0.smithi032.stdout:8/572: truncate d0/db/dd/d11/d1b/d22/f4f 3562758 0 2024-04-12T05:37:07.830 INFO:tasks.workunit.client.0.smithi032.stdout:5/597: dread - d2/dd/d5d/f9f zero size 2024-04-12T05:37:07.830 INFO:tasks.workunit.client.0.smithi032.stdout:4/502: dread d4/d13/d44/f6a [0,4194304] 0 2024-04-12T05:37:07.830 INFO:tasks.workunit.client.0.smithi032.stdout:9/449: dwrite d9/d15/d2c/f77 [0,4194304] 0 2024-04-12T05:37:07.834 INFO:tasks.workunit.client.0.smithi032.stdout:7/537: symlink d0/d1c/d71/laf 0 2024-04-12T05:37:07.836 INFO:tasks.workunit.client.0.smithi032.stdout:9/450: mknod d9/d15/d1a/d31/c9a 0 2024-04-12T05:37:07.839 INFO:tasks.workunit.client.0.smithi032.stdout:7/538: unlink d0/c50 0 2024-04-12T05:37:07.839 INFO:tasks.workunit.client.0.smithi032.stdout:3/592: dwrite d12/d2f/f41 [4194304,4194304] 0 2024-04-12T05:37:07.842 INFO:tasks.workunit.client.0.smithi032.stdout:5/598: creat d2/df/d31/da2/dba/fbf x:0 0 0 2024-04-12T05:37:07.842 INFO:tasks.workunit.client.0.smithi032.stdout:7/539: stat d0/d13/f27 0 2024-04-12T05:37:07.844 INFO:tasks.workunit.client.0.smithi032.stdout:3/593: creat d12/d2f/fc4 x:0 0 0 2024-04-12T05:37:07.845 INFO:tasks.workunit.client.0.smithi032.stdout:3/594: dread - d12/d2e/d98/fad zero size 2024-04-12T05:37:07.846 INFO:tasks.workunit.client.0.smithi032.stdout:7/540: truncate d0/d10/f62 3011217 0 2024-04-12T05:37:07.846 INFO:tasks.workunit.client.0.smithi032.stdout:5/599: write d2/df/f1c [4885706,69561] 0 2024-04-12T05:37:07.848 INFO:tasks.workunit.client.0.smithi032.stdout:3/595: dread - d12/d2e/d3a/fae zero size 2024-04-12T05:37:07.849 INFO:tasks.workunit.client.0.smithi032.stdout:4/503: truncate d4/d13/d19/f2a 807762 0 2024-04-12T05:37:07.850 INFO:tasks.workunit.client.0.smithi032.stdout:3/596: write d12/d24/f8f [160275,91687] 0 2024-04-12T05:37:07.851 INFO:tasks.workunit.client.0.smithi032.stdout:3/597: chown d12/d2e/c6f 3428998 1 2024-04-12T05:37:07.852 INFO:tasks.workunit.client.0.smithi032.stdout:5/600: mkdir d2/df/d31/da2/dc0 0 2024-04-12T05:37:07.853 INFO:tasks.workunit.client.0.smithi032.stdout:9/451: dread d9/d15/d1a/d24/f40 [4194304,4194304] 0 2024-04-12T05:37:07.854 INFO:tasks.workunit.client.0.smithi032.stdout:9/452: chown d9/d1f/f84 1805381951 1 2024-04-12T05:37:07.855 INFO:tasks.workunit.client.0.smithi032.stdout:4/504: mkdir d4/d87/d96/d2f/d3b/daa 0 2024-04-12T05:37:07.855 INFO:tasks.workunit.client.0.smithi032.stdout:4/505: readlink d4/d13/d19/l77 0 2024-04-12T05:37:07.856 INFO:tasks.workunit.client.0.smithi032.stdout:3/598: dread d12/d2e/f94 [0,4194304] 0 2024-04-12T05:37:07.857 INFO:tasks.workunit.client.0.smithi032.stdout:5/601: unlink d2/dd/d6f/la3 0 2024-04-12T05:37:07.858 INFO:tasks.workunit.client.0.smithi032.stdout:9/453: creat d9/d32/d86/f9b x:0 0 0 2024-04-12T05:37:07.858 INFO:tasks.workunit.client.0.smithi032.stdout:3/599: read d12/d2e/f7b [149878,90366] 0 2024-04-12T05:37:07.859 INFO:tasks.workunit.client.0.smithi032.stdout:7/541: getdents d0/d13/d21/d30 0 2024-04-12T05:37:07.859 INFO:tasks.workunit.client.0.smithi032.stdout:5/602: write d2/df/d25/f29 [3042210,86124] 0 2024-04-12T05:37:07.861 INFO:tasks.workunit.client.0.smithi032.stdout:1/505: sync 2024-04-12T05:37:07.861 INFO:tasks.workunit.client.0.smithi032.stdout:4/506: symlink d4/d87/d96/d68/d9b/lab 0 2024-04-12T05:37:07.863 INFO:tasks.workunit.client.0.smithi032.stdout:9/454: unlink d9/d15/d2c/f14 0 2024-04-12T05:37:07.869 INFO:tasks.workunit.client.0.smithi032.stdout:3/600: rename d12/d2f/d89/d9d/lbf to d12/lc5 0 2024-04-12T05:37:07.870 INFO:tasks.workunit.client.0.smithi032.stdout:8/573: dread d0/db/dd/d11/d1b/d22/f4f [0,4194304] 0 2024-04-12T05:37:07.872 INFO:tasks.workunit.client.0.smithi032.stdout:1/506: mkdir d5/de/d17/d35/d61/d77/d8f/da0 0 2024-04-12T05:37:07.872 INFO:tasks.workunit.client.0.smithi032.stdout:4/507: dread d4/d13/f1f [0,4194304] 0 2024-04-12T05:37:07.879 INFO:tasks.workunit.client.0.smithi032.stdout:1/507: rename d5/de/f7e to d5/de/d17/d35/fa1 0 2024-04-12T05:37:07.880 INFO:tasks.workunit.client.0.smithi032.stdout:4/508: read d4/d87/d96/d2f/d3b/f5d [2677477,88270] 0 2024-04-12T05:37:07.881 INFO:tasks.workunit.client.0.smithi032.stdout:7/542: link d0/d1c/d71/f8a d0/de/d47/fb0 0 2024-04-12T05:37:07.881 INFO:tasks.workunit.client.0.smithi032.stdout:8/574: dwrite d0/f2f [0,4194304] 0 2024-04-12T05:37:07.884 INFO:tasks.workunit.client.0.smithi032.stdout:5/603: getdents d2/d3c/d48/d4b 0 2024-04-12T05:37:07.885 INFO:tasks.workunit.client.0.smithi032.stdout:4/509: rename d4/d13/d19/f7a to d4/d13/d75/fac 0 2024-04-12T05:37:07.887 INFO:tasks.workunit.client.0.smithi032.stdout:8/575: mknod d0/d36/ccf 0 2024-04-12T05:37:07.888 INFO:tasks.workunit.client.0.smithi032.stdout:8/576: write d0/db/dd/d11/d1b/d5f/f93 [588793,45791] 0 2024-04-12T05:37:07.890 INFO:tasks.workunit.client.0.smithi032.stdout:7/543: rename d0/d13/l19 to d0/de/d47/lb1 0 2024-04-12T05:37:07.892 INFO:tasks.workunit.client.0.smithi032.stdout:4/510: stat d4/d87/d96/d4d/c5b 0 2024-04-12T05:37:07.895 INFO:tasks.workunit.client.0.smithi032.stdout:5/604: rename d2/d65 to d2/df/da9/dc1 0 2024-04-12T05:37:07.897 INFO:tasks.workunit.client.0.smithi032.stdout:3/601: mknod d12/daa/db3/cc6 0 2024-04-12T05:37:07.898 INFO:tasks.workunit.client.0.smithi032.stdout:7/544: creat d0/d13/d21/d7c/fb2 x:0 0 0 2024-04-12T05:37:07.900 INFO:tasks.workunit.client.0.smithi032.stdout:8/577: dread d0/db/dd/d11/f2c [0,4194304] 0 2024-04-12T05:37:07.901 INFO:tasks.workunit.client.0.smithi032.stdout:3/602: chown d12/d2e/c46 18000911 1 2024-04-12T05:37:07.901 INFO:tasks.workunit.client.0.smithi032.stdout:4/511: mknod d4/cad 0 2024-04-12T05:37:07.902 INFO:tasks.workunit.client.0.smithi032.stdout:4/512: stat d4/d87/d96/d2f/d3b/l55 0 2024-04-12T05:37:07.903 INFO:tasks.workunit.client.0.smithi032.stdout:7/545: mknod d0/de/df/d20/cb3 0 2024-04-12T05:37:07.904 INFO:tasks.workunit.client.0.smithi032.stdout:5/605: write d2/df/f5c [1217708,51269] 0 2024-04-12T05:37:07.905 INFO:tasks.workunit.client.0.smithi032.stdout:7/546: chown d0/l1d 16229955 1 2024-04-12T05:37:07.906 INFO:tasks.workunit.client.0.smithi032.stdout:4/513: creat d4/d87/d96/d2f/d6e/fae x:0 0 0 2024-04-12T05:37:07.906 INFO:tasks.workunit.client.0.smithi032.stdout:3/603: symlink d12/lc7 0 2024-04-12T05:37:07.907 INFO:tasks.workunit.client.0.smithi032.stdout:4/514: dread - d4/d13/d44/f5c zero size 2024-04-12T05:37:07.907 INFO:tasks.workunit.client.0.smithi032.stdout:8/578: link d0/db/dd/d11/d1b/d22/d54/da8/fce d0/db/dd/d11/d1b/d5f/daf/dc6/fd0 0 2024-04-12T05:37:07.908 INFO:tasks.workunit.client.0.smithi032.stdout:5/606: symlink d2/d3c/d48/lc2 0 2024-04-12T05:37:07.910 INFO:tasks.workunit.client.0.smithi032.stdout:7/547: rename d0/d10/l2b to d0/d10/d70/d7b/lb4 0 2024-04-12T05:37:07.910 INFO:tasks.workunit.client.0.smithi032.stdout:7/548: fdatasync d0/dd/f98 0 2024-04-12T05:37:07.911 INFO:tasks.workunit.client.0.smithi032.stdout:8/579: mknod d0/d58/cd1 0 2024-04-12T05:37:07.913 INFO:tasks.workunit.client.0.smithi032.stdout:4/515: symlink d4/d87/d96/d2f/d3b/da2/laf 0 2024-04-12T05:37:07.913 INFO:tasks.workunit.client.0.smithi032.stdout:7/549: rmdir d0/d13/d21 39 2024-04-12T05:37:07.916 INFO:tasks.workunit.client.0.smithi032.stdout:5/607: rename d2/df/d25/d1e/f23 to d2/df/d31/d92/fc3 0 2024-04-12T05:37:07.917 INFO:tasks.workunit.client.0.smithi032.stdout:4/516: creat d4/d87/d96/d68/fb0 x:0 0 0 2024-04-12T05:37:07.918 INFO:tasks.workunit.client.0.smithi032.stdout:4/517: write d4/d13/d44/f91 [1733846,64505] 0 2024-04-12T05:37:07.919 INFO:tasks.workunit.client.0.smithi032.stdout:7/550: chown d0/d13/d21/d30/c75 1 1 2024-04-12T05:37:07.919 INFO:tasks.workunit.client.0.smithi032.stdout:8/580: mknod d0/db/dd/d11/d1b/d22/cd2 0 2024-04-12T05:37:07.921 INFO:tasks.workunit.client.0.smithi032.stdout:5/608: mkdir d2/d3c/d48/d4b/dc4 0 2024-04-12T05:37:07.921 INFO:tasks.workunit.client.0.smithi032.stdout:8/581: write d0/db/dd/d11/d1b/d22/d51/fa9 [2093176,1676] 0 2024-04-12T05:37:07.922 INFO:tasks.workunit.client.0.smithi032.stdout:8/582: fdatasync d0/db/f6b 0 2024-04-12T05:37:07.924 INFO:tasks.workunit.client.0.smithi032.stdout:7/551: mknod d0/de/df/d20/cb5 0 2024-04-12T05:37:07.927 INFO:tasks.workunit.client.0.smithi032.stdout:4/518: mkdir d4/d87/d96/d2f/d3b/daa/db1 0 2024-04-12T05:37:07.928 INFO:tasks.workunit.client.0.smithi032.stdout:7/552: symlink d0/de/d47/lb6 0 2024-04-12T05:37:07.932 INFO:tasks.workunit.client.0.smithi032.stdout:7/553: rmdir d0/d13/d21/d30/d87 39 2024-04-12T05:37:07.933 INFO:tasks.workunit.client.0.smithi032.stdout:3/604: dwrite d12/d2f/f81 [0,4194304] 0 2024-04-12T05:37:07.933 INFO:tasks.workunit.client.0.smithi032.stdout:5/609: truncate d2/df/d25/d50/f7e 55534 0 2024-04-12T05:37:07.934 INFO:tasks.workunit.client.0.smithi032.stdout:8/583: dread d0/db/dd/f27 [0,4194304] 0 2024-04-12T05:37:07.935 INFO:tasks.workunit.client.0.smithi032.stdout:4/519: rename d4/d87/d96/d2f/d6e/c74 to d4/d87/d96/d2f/d9a/cb2 0 2024-04-12T05:37:07.936 INFO:tasks.workunit.client.0.smithi032.stdout:5/610: dread - d2/df/da9/dc1/d78/f94 zero size 2024-04-12T05:37:07.938 INFO:tasks.workunit.client.0.smithi032.stdout:3/605: chown d12/d2e/f74 187196 1 2024-04-12T05:37:07.938 INFO:tasks.workunit.client.0.smithi032.stdout:3/606: fsync d12/d2f/f3d 0 2024-04-12T05:37:07.942 INFO:tasks.workunit.client.0.smithi032.stdout:7/554: link d0/l59 d0/dd/d51/d65/lb7 0 2024-04-12T05:37:07.945 INFO:tasks.workunit.client.0.smithi032.stdout:2/591: sync 2024-04-12T05:37:07.945 INFO:tasks.workunit.client.0.smithi032.stdout:0/610: sync 2024-04-12T05:37:07.945 INFO:tasks.workunit.client.0.smithi032.stdout:6/659: sync 2024-04-12T05:37:07.945 INFO:tasks.workunit.client.0.smithi032.stdout:4/520: creat d4/d87/d96/d2f/d3b/daa/db1/fb3 x:0 0 0 2024-04-12T05:37:07.946 INFO:tasks.workunit.client.0.smithi032.stdout:2/592: fdatasync dd/f1e 0 2024-04-12T05:37:07.946 INFO:tasks.workunit.client.0.smithi032.stdout:3/607: write d12/daa/db3/f66 [575106,87155] 0 2024-04-12T05:37:07.947 INFO:tasks.workunit.client.0.smithi032.stdout:8/584: rename d0/db/dd/d11/d1b/d5f/c94 to d0/cd3 0 2024-04-12T05:37:07.948 INFO:tasks.workunit.client.0.smithi032.stdout:6/660: readlink d4/d21/d27/dad/d55/d99/l9c 0 2024-04-12T05:37:07.949 INFO:tasks.workunit.client.0.smithi032.stdout:4/521: read d4/d87/d96/d4d/f92 [369791,116920] 0 2024-04-12T05:37:07.949 INFO:tasks.workunit.client.0.smithi032.stdout:7/555: fsync d0/de/f15 0 2024-04-12T05:37:07.950 INFO:tasks.workunit.client.0.smithi032.stdout:0/611: symlink d2/dc/d41/lc5 0 2024-04-12T05:37:07.952 INFO:tasks.workunit.client.0.smithi032.stdout:3/608: chown d12/l32 367 1 2024-04-12T05:37:07.952 INFO:tasks.workunit.client.0.smithi032.stdout:5/611: rename d2/fac to d2/df/d31/fc5 0 2024-04-12T05:37:07.953 INFO:tasks.workunit.client.0.smithi032.stdout:4/522: write d4/d87/d96/d2f/d3b/f88 [177420,110814] 0 2024-04-12T05:37:07.955 INFO:tasks.workunit.client.0.smithi032.stdout:3/609: fsync d12/d2e/f67 0 2024-04-12T05:37:07.959 INFO:tasks.workunit.client.0.smithi032.stdout:5/612: chown d2/df/da9/dc1/d78/f9e 92 1 2024-04-12T05:37:07.959 INFO:tasks.workunit.client.0.smithi032.stdout:7/556: creat d0/de/fb8 x:0 0 0 2024-04-12T05:37:07.961 INFO:tasks.workunit.client.0.smithi032.stdout:0/612: creat d2/d8/d17/d1b/fc6 x:0 0 0 2024-04-12T05:37:07.962 INFO:tasks.workunit.client.0.smithi032.stdout:4/523: truncate d4/d87/d96/f59 55039 0 2024-04-12T05:37:07.963 INFO:tasks.workunit.client.0.smithi032.stdout:8/585: dread d0/db/dd/d11/d1b/d22/f4f [0,4194304] 0 2024-04-12T05:37:07.964 INFO:tasks.workunit.client.0.smithi032.stdout:2/593: dread dd/f14 [0,4194304] 0 2024-04-12T05:37:07.964 INFO:tasks.workunit.client.0.smithi032.stdout:6/661: dwrite d4/d14/d25/f7e [0,4194304] 0 2024-04-12T05:37:07.965 INFO:tasks.workunit.client.0.smithi032.stdout:7/557: mknod d0/d13/d21/d81/cb9 0 2024-04-12T05:37:07.967 INFO:tasks.workunit.client.0.smithi032.stdout:5/613: truncate d2/df/d25/f2c 378894 0 2024-04-12T05:37:07.968 INFO:tasks.workunit.client.0.smithi032.stdout:6/662: fsync d4/d21/d2e/fb6 0 2024-04-12T05:37:07.969 INFO:tasks.workunit.client.0.smithi032.stdout:8/586: fsync d0/d36/d84/dad/fb6 0 2024-04-12T05:37:07.972 INFO:tasks.workunit.client.0.smithi032.stdout:7/558: symlink d0/d13/d21/lba 0 2024-04-12T05:37:07.973 INFO:tasks.workunit.client.0.smithi032.stdout:6/663: creat d4/d21/d2e/fdd x:0 0 0 2024-04-12T05:37:07.973 INFO:tasks.workunit.client.0.smithi032.stdout:3/610: dread f10 [0,4194304] 0 2024-04-12T05:37:07.974 INFO:tasks.workunit.client.0.smithi032.stdout:2/594: write dd/f68 [1287907,34342] 0 2024-04-12T05:37:07.976 INFO:tasks.workunit.client.0.smithi032.stdout:7/559: chown d0/de/df/d29/l32 494 1 2024-04-12T05:37:07.976 INFO:tasks.workunit.client.0.smithi032.stdout:8/587: chown d0/db/dd/d11/d1b/d22/d35/l3b 18569 1 2024-04-12T05:37:07.977 INFO:tasks.workunit.client.0.smithi032.stdout:5/614: getdents d2/df/d25/d50/daa 0 2024-04-12T05:37:07.978 INFO:tasks.workunit.client.0.smithi032.stdout:7/560: dread - d0/d10/f63 zero size 2024-04-12T05:37:07.979 INFO:tasks.workunit.client.0.smithi032.stdout:3/611: mknod d12/d2e/d3a/d2d/cc8 0 2024-04-12T05:37:07.979 INFO:tasks.workunit.client.0.smithi032.stdout:9/455: sync 2024-04-12T05:37:07.980 INFO:tasks.workunit.client.0.smithi032.stdout:5/615: dread - d2/dd/d5d/f9f zero size 2024-04-12T05:37:07.981 INFO:tasks.workunit.client.0.smithi032.stdout:6/664: creat d4/d14/d1a/d92/d9e/db2/fde x:0 0 0 2024-04-12T05:37:07.981 INFO:tasks.workunit.client.0.smithi032.stdout:9/456: stat d9/f34 0 2024-04-12T05:37:07.982 INFO:tasks.workunit.client.0.smithi032.stdout:6/665: readlink d4/l20 0 2024-04-12T05:37:07.982 INFO:tasks.workunit.client.0.smithi032.stdout:2/595: creat dd/d19/d9d/d42/da7/fb8 x:0 0 0 2024-04-12T05:37:07.983 INFO:tasks.workunit.client.0.smithi032.stdout:4/524: dread d4/d13/f43 [0,4194304] 0 2024-04-12T05:37:07.983 INFO:tasks.workunit.client.0.smithi032.stdout:7/561: truncate d0/de/f15 876069 0 2024-04-12T05:37:07.983 INFO:tasks.workunit.client.0.smithi032.stdout:3/612: write d12/d24/dbb/fc2 [903956,115180] 0 2024-04-12T05:37:07.984 INFO:tasks.workunit.client.0.smithi032.stdout:8/588: rmdir d0/db/d38/d63/d77/d8d 0 2024-04-12T05:37:07.985 INFO:tasks.workunit.client.0.smithi032.stdout:3/613: stat d12/d2e/f7b 0 2024-04-12T05:37:07.985 INFO:tasks.workunit.client.0.smithi032.stdout:6/666: mknod d4/d21/d2e/d32/d94/cdf 0 2024-04-12T05:37:08.040 INFO:tasks.workunit.client.0.smithi032.stdout:9/457: symlink d9/d15/d1a/d1d/d5a/d6b/l9c 0 2024-04-12T05:37:08.041 INFO:tasks.workunit.client.0.smithi032.stdout:4/525: mknod d4/d87/d96/d2f/d3b/cb4 0 2024-04-12T05:37:08.043 INFO:tasks.workunit.client.0.smithi032.stdout:2/596: symlink dd/d19/d9d/d1c/d64/d32/lb9 0 2024-04-12T05:37:08.043 INFO:tasks.workunit.client.0.smithi032.stdout:0/613: dread d2/d29/f5d [0,4194304] 0 2024-04-12T05:37:08.044 INFO:tasks.workunit.client.0.smithi032.stdout:3/614: mkdir d12/d2e/d3a/d63/da6/dac/dc9 0 2024-04-12T05:37:08.044 INFO:tasks.workunit.client.0.smithi032.stdout:9/458: write d9/d15/d2c/f62 [2405588,83319] 0 2024-04-12T05:37:08.046 INFO:tasks.workunit.client.0.smithi032.stdout:4/526: chown d4/d13/f1c 65824307 1 2024-04-12T05:37:08.047 INFO:tasks.workunit.client.0.smithi032.stdout:2/597: fsync dd/d19/d20/d83/f9c 0 2024-04-12T05:37:08.048 INFO:tasks.workunit.client.0.smithi032.stdout:6/667: rename d4/d14/d1a/d37/d47/d57 to d4/de0 0 2024-04-12T05:37:08.048 INFO:tasks.workunit.client.0.smithi032.stdout:9/459: fdatasync d9/d15/d2c/d41/f96 0 2024-04-12T05:37:08.049 INFO:tasks.workunit.client.0.smithi032.stdout:0/614: read d2/d29/f65 [322575,5699] 0 2024-04-12T05:37:08.050 INFO:tasks.workunit.client.0.smithi032.stdout:7/562: dwrite d0/f7 [0,4194304] 0 2024-04-12T05:37:08.050 INFO:tasks.workunit.client.0.smithi032.stdout:8/589: truncate d0/db/dd/d11/d1b/d22/d35/f87 597002 0 2024-04-12T05:37:08.053 INFO:tasks.workunit.client.0.smithi032.stdout:3/615: symlink d12/d2e/d3a/dbe/lca 0 2024-04-12T05:37:08.053 INFO:tasks.workunit.client.0.smithi032.stdout:3/616: stat d12/d2e 0 2024-04-12T05:37:08.053 INFO:tasks.workunit.client.0.smithi032.stdout:6/668: rmdir d4/d14/d3c 39 2024-04-12T05:37:08.055 INFO:tasks.workunit.client.0.smithi032.stdout:3/617: fdatasync d12/d2e/d98/f9a 0 2024-04-12T05:37:08.055 INFO:tasks.workunit.client.0.smithi032.stdout:6/669: fsync d4/d74/f91 0 2024-04-12T05:37:08.055 INFO:tasks.workunit.client.0.smithi032.stdout:4/527: mkdir d4/d87/d96/d4d/db5 0 2024-04-12T05:37:08.056 INFO:tasks.workunit.client.0.smithi032.stdout:6/670: readlink d4/de0/l67 0 2024-04-12T05:37:08.058 INFO:tasks.workunit.client.0.smithi032.stdout:3/618: read d12/d2f/f48 [2964779,8466] 0 2024-04-12T05:37:08.059 INFO:tasks.workunit.client.0.smithi032.stdout:7/563: dread d0/de/df/f14 [0,4194304] 0 2024-04-12T05:37:08.062 INFO:tasks.workunit.client.0.smithi032.stdout:9/460: fsync d9/d15/d2c/f5e 0 2024-04-12T05:37:08.063 INFO:tasks.workunit.client.0.smithi032.stdout:9/461: chown d9/le 119 1 2024-04-12T05:37:08.065 INFO:tasks.workunit.client.0.smithi032.stdout:4/528: creat d4/d87/d96/d2f/d6e/fb6 x:0 0 0 2024-04-12T05:37:08.065 INFO:tasks.workunit.client.0.smithi032.stdout:6/671: mknod d4/d14/d1a/d92/d9e/db2/ce1 0 2024-04-12T05:37:08.066 INFO:tasks.workunit.client.0.smithi032.stdout:4/529: dread - d4/d87/d96/d2f/d6e/fae zero size 2024-04-12T05:37:08.067 INFO:tasks.workunit.client.0.smithi032.stdout:3/619: dread d12/d2e/d3a/f31 [0,4194304] 0 2024-04-12T05:37:08.068 INFO:tasks.workunit.client.0.smithi032.stdout:2/598: link dd/d19/l2b dd/d19/d9d/d42/d4a/lba 0 2024-04-12T05:37:08.070 INFO:tasks.workunit.client.0.smithi032.stdout:1/508: sync 2024-04-12T05:37:08.070 INFO:tasks.workunit.client.0.smithi032.stdout:4/530: dread d4/f98 [0,4194304] 0 2024-04-12T05:37:08.071 INFO:tasks.workunit.client.0.smithi032.stdout:7/564: rename d0/d13/d21/d30/d87 to d0/d76/dbb 0 2024-04-12T05:37:08.072 INFO:tasks.workunit.client.0.smithi032.stdout:6/672: mknod d4/d21/d27/dad/ce2 0 2024-04-12T05:37:08.073 INFO:tasks.workunit.client.0.smithi032.stdout:3/620: mkdir d12/d2e/dcb 0 2024-04-12T05:37:08.075 INFO:tasks.workunit.client.0.smithi032.stdout:2/599: mknod dd/d19/d20/cbb 0 2024-04-12T05:37:08.075 INFO:tasks.workunit.client.0.smithi032.stdout:9/462: dread d9/d15/d1a/f56 [0,4194304] 0 2024-04-12T05:37:08.076 INFO:tasks.workunit.client.0.smithi032.stdout:8/590: getdents d0/db/dd/d11/d1b/d22 0 2024-04-12T05:37:08.078 INFO:tasks.workunit.client.0.smithi032.stdout:1/509: symlink d5/de/d17/d35/d61/d77/d8f/d9c/la2 0 2024-04-12T05:37:08.079 INFO:tasks.workunit.client.0.smithi032.stdout:2/600: fdatasync dd/d19/f78 0 2024-04-12T05:37:08.080 INFO:tasks.workunit.client.0.smithi032.stdout:6/673: mknod d4/ce3 0 2024-04-12T05:37:08.081 INFO:tasks.workunit.client.0.smithi032.stdout:8/591: dread - d0/d36/d84/dad/fb6 zero size 2024-04-12T05:37:08.081 INFO:tasks.workunit.client.0.smithi032.stdout:9/463: write d9/d15/d1a/d31/f98 [143526,100431] 0 2024-04-12T05:37:08.081 INFO:tasks.workunit.client.0.smithi032.stdout:4/531: creat d4/fb7 x:0 0 0 2024-04-12T05:37:08.085 INFO:tasks.workunit.client.0.smithi032.stdout:7/565: dwrite d0/d1c/d71/f86 [0,4194304] 0 2024-04-12T05:37:08.085 INFO:tasks.workunit.client.0.smithi032.stdout:6/674: dread - d4/d21/d2e/fb6 zero size 2024-04-12T05:37:08.086 INFO:tasks.workunit.client.0.smithi032.stdout:9/464: stat d9/d15/d2c/f77 0 2024-04-12T05:37:08.088 INFO:tasks.workunit.client.0.smithi032.stdout:1/510: unlink d5/de/d17/d35/d3b/d49/f5e 0 2024-04-12T05:37:08.089 INFO:tasks.workunit.client.0.smithi032.stdout:6/675: readlink d4/d21/lbe 0 2024-04-12T05:37:08.089 INFO:tasks.workunit.client.0.smithi032.stdout:4/532: write d4/d87/d96/d4d/d78/f64 [448292,56825] 0 2024-04-12T05:37:08.090 INFO:tasks.workunit.client.0.smithi032.stdout:3/621: dwrite d12/d2e/d3a/f3b [0,4194304] 0 2024-04-12T05:37:08.090 INFO:tasks.workunit.client.0.smithi032.stdout:2/601: creat dd/d19/d9d/d42/d4a/da1/fbc x:0 0 0 2024-04-12T05:37:08.090 INFO:tasks.workunit.client.0.smithi032.stdout:7/566: write d0/d1c/d71/f85 [1209004,27444] 0 2024-04-12T05:37:08.093 INFO:tasks.workunit.client.0.smithi032.stdout:8/592: mkdir d0/db/d38/d63/d77/dd4 0 2024-04-12T05:37:08.094 INFO:tasks.workunit.client.0.smithi032.stdout:8/593: chown d0/db/f55 6857271 1 2024-04-12T05:37:08.096 INFO:tasks.workunit.client.0.smithi032.stdout:1/511: fdatasync d5/f47 0 2024-04-12T05:37:08.096 INFO:tasks.workunit.client.0.smithi032.stdout:2/602: fdatasync dd/d19/d9d/d1c/d64/d32/d3a/fa8 0 2024-04-12T05:37:08.099 INFO:tasks.workunit.client.0.smithi032.stdout:3/622: creat d12/daa/db3/fcc x:0 0 0 2024-04-12T05:37:08.099 INFO:tasks.workunit.client.0.smithi032.stdout:3/623: chown d12/d24/f4c 3494315 1 2024-04-12T05:37:08.102 INFO:tasks.workunit.client.0.smithi032.stdout:7/567: rmdir d0/dd/d51/d65 39 2024-04-12T05:37:08.103 INFO:tasks.workunit.client.0.smithi032.stdout:6/676: fdatasync d4/f2a 0 2024-04-12T05:37:08.104 INFO:tasks.workunit.client.0.smithi032.stdout:9/465: link d9/d15/d1a/f56 d9/d15/d2c/d41/f9d 0 2024-04-12T05:37:08.106 INFO:tasks.workunit.client.0.smithi032.stdout:1/512: dwrite d5/f23 [4194304,4194304] 0 2024-04-12T05:37:08.107 INFO:tasks.workunit.client.0.smithi032.stdout:8/594: mkdir d0/db/d38/d63/d77/dd4/dd5 0 2024-04-12T05:37:08.107 INFO:tasks.workunit.client.0.smithi032.stdout:9/466: write f5 [3468866,45872] 0 2024-04-12T05:37:08.110 INFO:tasks.workunit.client.0.smithi032.stdout:1/513: dread - d5/de/d17/d35/d61/d77/f7d zero size 2024-04-12T05:37:08.112 INFO:tasks.workunit.client.0.smithi032.stdout:4/533: creat d4/d13/d19/fb8 x:0 0 0 2024-04-12T05:37:08.113 INFO:tasks.workunit.client.0.smithi032.stdout:1/514: dread - d5/de/d17/d1b/f85 zero size 2024-04-12T05:37:08.114 INFO:tasks.workunit.client.0.smithi032.stdout:8/595: creat d0/db/dd/d11/d1b/d22/fd6 x:0 0 0 2024-04-12T05:37:08.115 INFO:tasks.workunit.client.0.smithi032.stdout:2/603: dread fa [0,4194304] 0 2024-04-12T05:37:08.116 INFO:tasks.workunit.client.0.smithi032.stdout:1/515: chown d5/c46 696 1 2024-04-12T05:37:08.116 INFO:tasks.workunit.client.0.smithi032.stdout:6/677: unlink d4/d21/d27/dad/d55/c9a 0 2024-04-12T05:37:08.117 INFO:tasks.workunit.client.0.smithi032.stdout:7/568: creat d0/dd/d51/fbc x:0 0 0 2024-04-12T05:37:08.117 INFO:tasks.workunit.client.0.smithi032.stdout:4/534: mknod d4/d87/d96/d68/cb9 0 2024-04-12T05:37:08.117 INFO:tasks.workunit.client.0.smithi032.stdout:8/596: symlink d0/d36/ld7 0 2024-04-12T05:37:08.118 INFO:tasks.workunit.client.0.smithi032.stdout:2/604: write dd/d19/d20/f62 [4096510,86781] 0 2024-04-12T05:37:08.119 INFO:tasks.workunit.client.0.smithi032.stdout:9/467: getdents d9/d32/d6e 0 2024-04-12T05:37:08.120 INFO:tasks.workunit.client.0.smithi032.stdout:7/569: write d0/de/fb8 [460020,102180] 0 2024-04-12T05:37:08.121 INFO:tasks.workunit.client.0.smithi032.stdout:6/678: truncate d4/d21/d2e/fb6 646969 0 2024-04-12T05:37:08.122 INFO:tasks.workunit.client.0.smithi032.stdout:4/535: mkdir d4/d87/d96/d4d/d78/dba 0 2024-04-12T05:37:08.122 INFO:tasks.workunit.client.0.smithi032.stdout:7/570: readlink d0/de/d47/lb1 0 2024-04-12T05:37:08.123 INFO:tasks.workunit.client.0.smithi032.stdout:7/571: stat d0/d13/d21/lba 0 2024-04-12T05:37:08.125 INFO:tasks.workunit.client.0.smithi032.stdout:9/468: creat d9/d15/d1a/f9e x:0 0 0 2024-04-12T05:37:08.128 INFO:tasks.workunit.client.0.smithi032.stdout:7/572: unlink d0/d10/fa0 0 2024-04-12T05:37:08.128 INFO:tasks.workunit.client.0.smithi032.stdout:9/469: write d9/d1f/f39 [2261037,96744] 0 2024-04-12T05:37:08.129 INFO:tasks.workunit.client.0.smithi032.stdout:2/605: rename dd/d19/d9d/d42/d4a/lba to dd/d19/d82/lbd 0 2024-04-12T05:37:08.131 INFO:tasks.workunit.client.0.smithi032.stdout:6/679: dwrite d4/d14/d1a/d88/fc0 [0,4194304] 0 2024-04-12T05:37:08.132 INFO:tasks.workunit.client.0.smithi032.stdout:6/680: chown d4/d14/d1a/d37/d47 4019282 1 2024-04-12T05:37:08.132 INFO:tasks.workunit.client.0.smithi032.stdout:8/597: rmdir d0/db/dd/d11/d1b/d22/d54/da8 39 2024-04-12T05:37:08.137 INFO:tasks.workunit.client.0.smithi032.stdout:4/536: symlink d4/d87/d96/d4d/lbb 0 2024-04-12T05:37:08.138 INFO:tasks.workunit.client.0.smithi032.stdout:9/470: creat d9/d15/d2c/f9f x:0 0 0 2024-04-12T05:37:08.139 INFO:tasks.workunit.client.0.smithi032.stdout:2/606: mknod dd/d19/d9d/d42/d4a/cbe 0 2024-04-12T05:37:08.141 INFO:tasks.workunit.client.0.smithi032.stdout:8/598: dwrite d0/d72/faa [0,4194304] 0 2024-04-12T05:37:08.143 INFO:tasks.workunit.client.0.smithi032.stdout:6/681: creat d4/d21/d27/dad/d55/fe4 x:0 0 0 2024-04-12T05:37:08.146 INFO:tasks.workunit.client.0.smithi032.stdout:4/537: truncate d4/d87/d96/d2f/d3b/f5d 1413074 0 2024-04-12T05:37:08.146 INFO:tasks.workunit.client.0.smithi032.stdout:8/599: rmdir d0/db/d39 39 2024-04-12T05:37:08.146 INFO:tasks.workunit.client.0.smithi032.stdout:2/607: mknod dd/d19/d9d/d1c/d64/cbf 0 2024-04-12T05:37:08.148 INFO:tasks.workunit.client.0.smithi032.stdout:8/600: chown d0/l5b 4 1 2024-04-12T05:37:08.148 INFO:tasks.workunit.client.0.smithi032.stdout:4/538: fsync d4/d87/d96/f59 0 2024-04-12T05:37:08.150 INFO:tasks.workunit.client.0.smithi032.stdout:7/573: dread d0/de/f15 [0,4194304] 0 2024-04-12T05:37:08.154 INFO:tasks.workunit.client.0.smithi032.stdout:4/539: write d4/d87/d96/d32/f5f [2812218,63824] 0 2024-04-12T05:37:08.154 INFO:tasks.workunit.client.0.smithi032.stdout:9/471: dread d9/d15/d1a/d24/d27/f2d [0,4194304] 0 2024-04-12T05:37:08.156 INFO:tasks.workunit.client.0.smithi032.stdout:2/608: rename dd/d19/d9d/d42/c79 to dd/d19/d9d/d1c/d64/d32/d3a/cc0 0 2024-04-12T05:37:08.156 INFO:tasks.workunit.client.0.smithi032.stdout:6/682: dread d4/d14/d1a/d88/fc0 [0,4194304] 0 2024-04-12T05:37:08.157 INFO:tasks.workunit.client.0.smithi032.stdout:8/601: symlink d0/db/dd/d11/d1b/d22/d54/ld8 0 2024-04-12T05:37:08.159 INFO:tasks.workunit.client.0.smithi032.stdout:7/574: creat d0/d13/d21/d30/d31/fbd x:0 0 0 2024-04-12T05:37:08.161 INFO:tasks.workunit.client.0.smithi032.stdout:9/472: symlink d9/d15/d1a/d1d/d5a/la0 0 2024-04-12T05:37:08.162 INFO:tasks.workunit.client.0.smithi032.stdout:9/473: read - d9/d15/d1a/d24/f7d zero size 2024-04-12T05:37:08.163 INFO:tasks.workunit.client.0.smithi032.stdout:9/474: chown d9/d15/d2c/d41/f7e 157672 1 2024-04-12T05:37:08.163 INFO:tasks.workunit.client.0.smithi032.stdout:6/683: write d4/d14/d1a/d37/f98 [4669150,85533] 0 2024-04-12T05:37:08.163 INFO:tasks.workunit.client.0.smithi032.stdout:9/475: stat d9/d1f/f5d 0 2024-04-12T05:37:08.169 INFO:tasks.workunit.client.0.smithi032.stdout:4/540: rename d4/d87/d96/d2f/d3b/daa/db1 to d4/d87/d96/d4d/dbc 0 2024-04-12T05:37:08.170 INFO:tasks.workunit.client.0.smithi032.stdout:2/609: creat dd/d19/d9d/d1c/d64/fc1 x:0 0 0 2024-04-12T05:37:08.171 INFO:tasks.workunit.client.0.smithi032.stdout:2/610: readlink dd/d19/d82/laf 0 2024-04-12T05:37:08.173 INFO:tasks.workunit.client.0.smithi032.stdout:7/575: rename d0/d76 to d0/d34/dbe 0 2024-04-12T05:37:08.178 INFO:tasks.workunit.client.0.smithi032.stdout:8/602: rename d0/db/d38/f3f to d0/db/dd/d11/d1b/d22/d54/da8/fd9 0 2024-04-12T05:37:08.183 INFO:tasks.workunit.client.0.smithi032.stdout:4/541: write d4/d87/d96/d2f/d3b/f3c [951327,61608] 0 2024-04-12T05:37:08.186 INFO:tasks.workunit.client.0.smithi032.stdout:2/611: creat dd/d19/d9d/d1c/d64/fc2 x:0 0 0 2024-04-12T05:37:08.187 INFO:tasks.workunit.client.0.smithi032.stdout:4/542: rmdir d4/d13/d75 39 2024-04-12T05:37:08.191 INFO:tasks.workunit.client.0.smithi032.stdout:6/684: rename d4/d21/d27/cc9 to d4/d14/d1a/ce5 0 2024-04-12T05:37:08.192 INFO:tasks.workunit.client.0.smithi032.stdout:7/576: getdents d0/d13/d21/d81 0 2024-04-12T05:37:08.194 INFO:tasks.workunit.client.0.smithi032.stdout:8/603: link d0/db/dd/d11/d1b/d5f/l8a d0/db/dd/d11/d1b/d22/d51/lda 0 2024-04-12T05:37:08.199 INFO:tasks.workunit.client.0.smithi032.stdout:9/476: rename d9/d15/d1a/d24/d27/f2d to d9/fa1 0 2024-04-12T05:37:08.199 INFO:tasks.workunit.client.0.smithi032.stdout:4/543: dwrite d4/d87/d96/d2f/f4e [0,4194304] 0 2024-04-12T05:37:08.199 INFO:tasks.workunit.client.0.smithi032.stdout:6/685: rename d4/d14/d1a/d79 to d4/d14/d1a/d79/de6 22 2024-04-12T05:37:08.204 INFO:tasks.workunit.client.0.smithi032.stdout:7/577: truncate d0/d10/f60 600043 0 2024-04-12T05:37:08.205 INFO:tasks.workunit.client.0.smithi032.stdout:9/477: truncate d9/d32/f74 985577 0 2024-04-12T05:37:08.206 INFO:tasks.workunit.client.0.smithi032.stdout:7/578: readlink d0/l1 0 2024-04-12T05:37:08.207 INFO:tasks.workunit.client.0.smithi032.stdout:4/544: readlink d4/d13/d19/l3a 0 2024-04-12T05:37:08.209 INFO:tasks.workunit.client.0.smithi032.stdout:9/478: write d9/d15/f6a [895962,10409] 0 2024-04-12T05:37:08.209 INFO:tasks.workunit.client.0.smithi032.stdout:7/579: mkdir d0/d34/dbf 0 2024-04-12T05:37:08.210 INFO:tasks.workunit.client.0.smithi032.stdout:4/545: chown d4/d87/d96/d4d/dbc 325 1 2024-04-12T05:37:08.210 INFO:tasks.workunit.client.0.smithi032.stdout:8/604: dwrite d0/d36/d84/dad/fab [0,4194304] 0 2024-04-12T05:37:08.213 INFO:tasks.workunit.client.0.smithi032.stdout:7/580: creat d0/d34/fc0 x:0 0 0 2024-04-12T05:37:08.217 INFO:tasks.workunit.client.0.smithi032.stdout:8/605: readlink d0/db/dd/d11/d1b/d22/d35/l48 0 2024-04-12T05:37:08.218 INFO:tasks.workunit.client.0.smithi032.stdout:6/686: dread d4/f9 [0,4194304] 0 2024-04-12T05:37:08.220 INFO:tasks.workunit.client.0.smithi032.stdout:7/581: symlink d0/d13/d21/d30/lc1 0 2024-04-12T05:37:08.222 INFO:tasks.workunit.client.0.smithi032.stdout:9/479: dwrite f5 [0,4194304] 0 2024-04-12T05:37:08.223 INFO:tasks.workunit.client.0.smithi032.stdout:8/606: chown d0/db/dd/d11/d1b/d22/d54/da8/fc7 5427 1 2024-04-12T05:37:08.226 INFO:tasks.workunit.client.0.smithi032.stdout:4/546: dwrite d4/d13/f71 [0,4194304] 0 2024-04-12T05:37:08.226 INFO:tasks.workunit.client.0.smithi032.stdout:9/480: symlink d9/d15/la2 0 2024-04-12T05:37:08.234 INFO:tasks.workunit.client.0.smithi032.stdout:7/582: dwrite d0/fc [0,4194304] 0 2024-04-12T05:37:08.240 INFO:tasks.workunit.client.0.smithi032.stdout:8/607: truncate d0/db/dd/d11/d1b/d5f/daf/dc6/fd0 982969 0 2024-04-12T05:37:08.240 INFO:tasks.workunit.client.0.smithi032.stdout:7/583: read d0/d13/f64 [992661,22986] 0 2024-04-12T05:37:08.244 INFO:tasks.workunit.client.0.smithi032.stdout:7/584: write d0/de/df/f7a [448418,17370] 0 2024-04-12T05:37:08.246 INFO:tasks.workunit.client.0.smithi032.stdout:8/608: mknod d0/db/dd/d11/d1b/d22/d54/cdb 0 2024-04-12T05:37:08.247 INFO:tasks.workunit.client.0.smithi032.stdout:6/687: dread d4/d14/d1a/f31 [0,4194304] 0 2024-04-12T05:37:08.247 INFO:tasks.workunit.client.0.smithi032.stdout:4/547: dwrite d4/d13/f1c [0,4194304] 0 2024-04-12T05:37:08.249 INFO:tasks.workunit.client.0.smithi032.stdout:8/609: dread - d0/db/d38/d63/fbd zero size 2024-04-12T05:37:08.251 INFO:tasks.workunit.client.0.smithi032.stdout:7/585: symlink d0/dd/d51/d65/lc2 0 2024-04-12T05:37:08.251 INFO:tasks.workunit.client.0.smithi032.stdout:8/610: write d0/db/dd/d11/f2c [3757844,86247] 0 2024-04-12T05:37:08.257 INFO:tasks.workunit.client.0.smithi032.stdout:6/688: getdents d4/d14/d1a/d37/d69/d8a 0 2024-04-12T05:37:08.258 INFO:tasks.workunit.client.0.smithi032.stdout:8/611: mkdir d0/db/dd/d11/d1b/d22/ddc 0 2024-04-12T05:37:08.260 INFO:tasks.workunit.client.0.smithi032.stdout:4/548: getdents d4/d87/d96/d2f/d3b 0 2024-04-12T05:37:08.261 INFO:tasks.workunit.client.0.smithi032.stdout:7/586: dwrite d0/d13/d21/f57 [0,4194304] 0 2024-04-12T05:37:08.263 INFO:tasks.workunit.client.0.smithi032.stdout:9/481: dread d9/d15/d2c/f5e [0,4194304] 0 2024-04-12T05:37:08.264 INFO:tasks.workunit.client.0.smithi032.stdout:6/689: rename d4/d14/d1a to d4/de0/de7 0 2024-04-12T05:37:08.266 INFO:tasks.workunit.client.0.smithi032.stdout:8/612: mkdir d0/db/dd/ddd 0 2024-04-12T05:37:08.267 INFO:tasks.workunit.client.0.smithi032.stdout:7/587: mknod d0/d13/d21/d81/cc3 0 2024-04-12T05:37:08.269 INFO:tasks.workunit.client.0.smithi032.stdout:9/482: rename d9/d15/d1a/d1d/c35 to d9/d32/ca3 0 2024-04-12T05:37:08.271 INFO:tasks.workunit.client.0.smithi032.stdout:7/588: creat d0/fc4 x:0 0 0 2024-04-12T05:37:08.272 INFO:tasks.workunit.client.0.smithi032.stdout:7/589: dread - d0/d34/fc0 zero size 2024-04-12T05:37:08.273 INFO:tasks.workunit.client.0.smithi032.stdout:8/613: dwrite d0/db/d38/d63/fbd [0,4194304] 0 2024-04-12T05:37:08.274 INFO:tasks.workunit.client.0.smithi032.stdout:7/590: chown d0/de/df/d20/f6b 981372091 1 2024-04-12T05:37:08.275 INFO:tasks.workunit.client.0.smithi032.stdout:6/690: rename d4/de0/de7/d92/f97 to d4/d14/d25/fe8 0 2024-04-12T05:37:08.277 INFO:tasks.workunit.client.0.smithi032.stdout:9/483: creat d9/d15/d1a/d1d/d5a/fa4 x:0 0 0 2024-04-12T05:37:08.279 INFO:tasks.workunit.client.0.smithi032.stdout:9/484: write d9/d15/d2c/f20 [1258885,72333] 0 2024-04-12T05:37:08.279 INFO:tasks.workunit.client.0.smithi032.stdout:5/616: sync 2024-04-12T05:37:08.279 INFO:tasks.workunit.client.0.smithi032.stdout:0/615: sync 2024-04-12T05:37:08.281 INFO:tasks.workunit.client.0.smithi032.stdout:7/591: unlink d0/d13/d21/f57 0 2024-04-12T05:37:08.282 INFO:tasks.workunit.client.0.smithi032.stdout:7/592: readlink d0/d13/l18 0 2024-04-12T05:37:08.283 INFO:tasks.workunit.client.0.smithi032.stdout:4/549: rename d4/d87/d96/d4d/d78/f51 to d4/d13/d75/fbd 0 2024-04-12T05:37:08.287 INFO:tasks.workunit.client.0.smithi032.stdout:6/691: dread d4/d21/d27/f4b [0,4194304] 0 2024-04-12T05:37:08.288 INFO:tasks.workunit.client.0.smithi032.stdout:9/485: mknod d9/d15/d1a/d24/d71/ca5 0 2024-04-12T05:37:08.289 INFO:tasks.workunit.client.0.smithi032.stdout:9/486: chown d9/d15/d1a/c60 4187020 1 2024-04-12T05:37:08.290 INFO:tasks.workunit.client.0.smithi032.stdout:9/487: stat d9/d15/d2c/f62 0 2024-04-12T05:37:08.291 INFO:tasks.workunit.client.0.smithi032.stdout:0/616: dread d2/dc/d3e/dac/f35 [0,4194304] 0 2024-04-12T05:37:08.291 INFO:tasks.workunit.client.0.smithi032.stdout:5/617: mkdir d2/d3c/dc6 0 2024-04-12T05:37:08.293 INFO:tasks.workunit.client.0.smithi032.stdout:0/617: truncate d2/dc/d3e/dac/fa1 348469 0 2024-04-12T05:37:08.293 INFO:tasks.workunit.client.0.smithi032.stdout:0/618: readlink d2/d29/l4c 0 2024-04-12T05:37:08.295 INFO:tasks.workunit.client.0.smithi032.stdout:6/692: creat d4/de0/de7/dc4/fe9 x:0 0 0 2024-04-12T05:37:08.296 INFO:tasks.workunit.client.0.smithi032.stdout:4/550: mknod d4/d87/d96/d4d/cbe 0 2024-04-12T05:37:08.297 INFO:tasks.workunit.client.0.smithi032.stdout:8/614: getdents d0 0 2024-04-12T05:37:08.301 INFO:tasks.workunit.client.0.smithi032.stdout:0/619: rename d2/d28/d87/d90 to d2/d29/d77/dc7 0 2024-04-12T05:37:08.307 INFO:tasks.workunit.client.0.smithi032.stdout:5/618: mkdir d2/df/da9/dc1/d78/dc7 0 2024-04-12T05:37:08.308 INFO:tasks.workunit.client.0.smithi032.stdout:0/620: fsync d2/dc/d41/fbc 0 2024-04-12T05:37:08.308 INFO:tasks.workunit.client.0.smithi032.stdout:9/488: dread d9/f34 [0,4194304] 0 2024-04-12T05:37:08.309 INFO:tasks.workunit.client.0.smithi032.stdout:7/593: truncate d0/d13/f4b 3222620 0 2024-04-12T05:37:08.310 INFO:tasks.workunit.client.0.smithi032.stdout:7/594: readlink d0/d13/d21/l92 0 2024-04-12T05:37:08.311 INFO:tasks.workunit.client.0.smithi032.stdout:8/615: rename d0/db/dd/d11/d1b/d22/d29/lc1 to d0/db/dd/d11/d1b/d96/lde 0 2024-04-12T05:37:08.312 INFO:tasks.workunit.client.0.smithi032.stdout:8/616: fsync d0/db/dd/d11/d1b/f6c 0 2024-04-12T05:37:08.313 INFO:tasks.workunit.client.0.smithi032.stdout:4/551: creat d4/d87/d96/fbf x:0 0 0 2024-04-12T05:37:08.314 INFO:tasks.workunit.client.0.smithi032.stdout:0/621: mkdir d2/d8/df/d24/d60/dc8 0 2024-04-12T05:37:08.317 INFO:tasks.workunit.client.0.smithi032.stdout:9/489: rename d9/d15/d1a/d24/f7d to d9/d15/d1a/d31/d65/fa6 0 2024-04-12T05:37:08.319 INFO:tasks.workunit.client.0.smithi032.stdout:4/552: unlink d4/d87/d96/d68/c86 0 2024-04-12T05:37:08.322 INFO:tasks.workunit.client.0.smithi032.stdout:9/490: write f7 [3070160,56332] 0 2024-04-12T05:37:08.324 INFO:tasks.workunit.client.0.smithi032.stdout:0/622: dwrite d2/d29/f47 [8388608,4194304] 0 2024-04-12T05:37:08.325 INFO:tasks.workunit.client.0.smithi032.stdout:6/693: dwrite d4/d21/f7d [0,4194304] 0 2024-04-12T05:37:08.331 INFO:tasks.workunit.client.0.smithi032.stdout:0/623: creat d2/d29/d77/fc9 x:0 0 0 2024-04-12T05:37:08.333 INFO:tasks.workunit.client.0.smithi032.stdout:6/694: mknod d4/d14/d25/cea 0 2024-04-12T05:37:08.334 INFO:tasks.workunit.client.0.smithi032.stdout:6/695: write d4/d74/f91 [5174515,119826] 0 2024-04-12T05:37:08.335 INFO:tasks.workunit.client.0.smithi032.stdout:7/595: link d0/de/d47/fb0 d0/d13/fc5 0 2024-04-12T05:37:08.336 INFO:tasks.workunit.client.0.smithi032.stdout:7/596: dread - d0/d13/d21/d81/f9c zero size 2024-04-12T05:37:08.337 INFO:tasks.workunit.client.0.smithi032.stdout:0/624: rename d2/dc/d3e/dac/d51/d83/d9d/cae to d2/dc/cca 0 2024-04-12T05:37:08.342 INFO:tasks.workunit.client.0.smithi032.stdout:9/491: creat d9/fa7 x:0 0 0 2024-04-12T05:37:08.380 INFO:tasks.workunit.client.0.smithi032.stdout:3/624: sync 2024-04-12T05:37:08.380 INFO:tasks.workunit.client.0.smithi032.stdout:1/516: sync 2024-04-12T05:37:08.380 INFO:tasks.workunit.client.0.smithi032.stdout:6/696: mkdir d4/de0/deb 0 2024-04-12T05:37:08.381 INFO:tasks.workunit.client.0.smithi032.stdout:0/625: rename d2/d29/d77/dc7/f92 to d2/dc/d3e/dac/d51/fcb 0 2024-04-12T05:37:08.382 INFO:tasks.workunit.client.0.smithi032.stdout:9/492: chown d9/fa1 1567495 1 2024-04-12T05:37:08.382 INFO:tasks.workunit.client.0.smithi032.stdout:8/617: dread d0/db/dd/d11/d1b/d22/d51/fa9 [0,4194304] 0 2024-04-12T05:37:08.383 INFO:tasks.workunit.client.0.smithi032.stdout:6/697: write d4/d14/d25/f7e [4443501,125478] 0 2024-04-12T05:37:08.383 INFO:tasks.workunit.client.0.smithi032.stdout:3/625: write d12/d2f/fc4 [104466,46418] 0 2024-04-12T05:37:08.384 INFO:tasks.workunit.client.0.smithi032.stdout:0/626: read d2/f14 [513159,106489] 0 2024-04-12T05:37:08.384 INFO:tasks.workunit.client.0.smithi032.stdout:7/597: chown d0/de/df/d20/f68 0 1 2024-04-12T05:37:08.384 INFO:tasks.workunit.client.0.smithi032.stdout:9/493: dread - d9/d32/d86/f9b zero size 2024-04-12T05:37:08.386 INFO:tasks.workunit.client.0.smithi032.stdout:6/698: chown d4/de0/de7/d37/lbb 6791601 1 2024-04-12T05:37:08.388 INFO:tasks.workunit.client.0.smithi032.stdout:0/627: mkdir d2/d29/dcc 0 2024-04-12T05:37:08.388 INFO:tasks.workunit.client.0.smithi032.stdout:6/699: read - d4/d21/d2e/d32/d94/fa9 zero size 2024-04-12T05:37:08.389 INFO:tasks.workunit.client.0.smithi032.stdout:0/628: fsync d2/f6 0 2024-04-12T05:37:08.389 INFO:tasks.workunit.client.0.smithi032.stdout:3/626: dread d12/daa/db3/f66 [0,4194304] 0 2024-04-12T05:37:08.391 INFO:tasks.workunit.client.0.smithi032.stdout:8/618: write d0/db/dd/d11/d1b/d22/d54/da8/fb0 [538423,112160] 0 2024-04-12T05:37:08.394 INFO:tasks.workunit.client.0.smithi032.stdout:9/494: creat d9/d15/d1a/d24/d27/d88/fa8 x:0 0 0 2024-04-12T05:37:08.395 INFO:tasks.workunit.client.0.smithi032.stdout:0/629: symlink d2/d8/d17/d1b/d20/lcd 0 2024-04-12T05:37:08.395 INFO:tasks.workunit.client.0.smithi032.stdout:6/700: mkdir d4/de0/deb/dec 0 2024-04-12T05:37:08.396 INFO:tasks.workunit.client.0.smithi032.stdout:9/495: mkdir d9/d58/da9 0 2024-04-12T05:37:08.399 INFO:tasks.workunit.client.0.smithi032.stdout:7/598: dread d0/de/df/f1e [0,4194304] 0 2024-04-12T05:37:08.400 INFO:tasks.workunit.client.0.smithi032.stdout:0/630: write d2/d29/f5d [2388534,98994] 0 2024-04-12T05:37:08.401 INFO:tasks.workunit.client.0.smithi032.stdout:9/496: chown d9/d15/d1a/c4c 8730 1 2024-04-12T05:37:08.402 INFO:tasks.workunit.client.0.smithi032.stdout:6/701: symlink d4/de0/de7/d37/d69/led 0 2024-04-12T05:37:08.405 INFO:tasks.workunit.client.0.smithi032.stdout:8/619: dwrite d0/db/dd/d11/d1b/fc9 [0,4194304] 0 2024-04-12T05:37:08.406 INFO:tasks.workunit.client.0.smithi032.stdout:6/702: chown d4/d21/d2e/lc3 78 1 2024-04-12T05:37:08.407 INFO:tasks.workunit.client.0.smithi032.stdout:7/599: creat d0/d1c/fc6 x:0 0 0 2024-04-12T05:37:08.408 INFO:tasks.workunit.client.0.smithi032.stdout:3/627: dwrite d12/d2e/d3a/d63/fb2 [0,4194304] 0 2024-04-12T05:37:08.416 INFO:tasks.workunit.client.0.smithi032.stdout:6/703: mknod d4/d21/d2e/cee 0 2024-04-12T05:37:08.416 INFO:tasks.workunit.client.0.smithi032.stdout:0/631: dread d2/d8/d66/f82 [0,4194304] 0 2024-04-12T05:37:08.418 INFO:tasks.workunit.client.0.smithi032.stdout:8/620: mknod d0/db/cdf 0 2024-04-12T05:37:08.419 INFO:tasks.workunit.client.0.smithi032.stdout:8/621: fdatasync d0/d72/fc8 0 2024-04-12T05:37:08.422 INFO:tasks.workunit.client.0.smithi032.stdout:7/600: link d0/f6f d0/d1c/d4c/fc7 0 2024-04-12T05:37:08.423 INFO:tasks.workunit.client.0.smithi032.stdout:6/704: mknod d4/de0/de7/d88/cef 0 2024-04-12T05:37:08.424 INFO:tasks.workunit.client.0.smithi032.stdout:0/632: rename d2/d8/df/d24/d60/fb0 to d2/d28/d87/d79/fce 0 2024-04-12T05:37:08.425 INFO:tasks.workunit.client.0.smithi032.stdout:9/497: link d9/d15/d1a/c60 d9/d15/d1a/d31/d65/d80/d87/caa 0 2024-04-12T05:37:08.425 INFO:tasks.workunit.client.0.smithi032.stdout:0/633: chown d2/dc 6948 1 2024-04-12T05:37:08.427 INFO:tasks.workunit.client.0.smithi032.stdout:8/622: write d0/d36/f4a [3735733,64661] 0 2024-04-12T05:37:08.429 INFO:tasks.workunit.client.0.smithi032.stdout:3/628: getdents d12/d2e 0 2024-04-12T05:37:08.430 INFO:tasks.workunit.client.0.smithi032.stdout:9/498: mkdir d9/d15/d2c/dab 0 2024-04-12T05:37:08.430 INFO:tasks.workunit.client.0.smithi032.stdout:0/634: mknod d2/d29/d77/dc7/ccf 0 2024-04-12T05:37:08.432 INFO:tasks.workunit.client.0.smithi032.stdout:7/601: getdents d0/d34/dbe/dbb 0 2024-04-12T05:37:08.432 INFO:tasks.workunit.client.0.smithi032.stdout:8/623: fdatasync d0/d36/d84/dad/da4/fb1 0 2024-04-12T05:37:08.432 INFO:tasks.workunit.client.0.smithi032.stdout:3/629: mkdir d12/d2e/dcd 0 2024-04-12T05:37:08.435 INFO:tasks.workunit.client.0.smithi032.stdout:6/705: dwrite d4/de0/de7/d37/f98 [4194304,4194304] 0 2024-04-12T05:37:08.438 INFO:tasks.workunit.client.0.smithi032.stdout:7/602: write d0/d1c/d71/f86 [4451745,27919] 0 2024-04-12T05:37:08.438 INFO:tasks.workunit.client.0.smithi032.stdout:3/630: read d12/d2e/f7b [110330,34779] 0 2024-04-12T05:37:08.439 INFO:tasks.workunit.client.0.smithi032.stdout:8/624: mknod d0/db/dd/d11/d1b/d96/ce0 0 2024-04-12T05:37:08.443 INFO:tasks.workunit.client.0.smithi032.stdout:6/706: truncate d4/d14/f6d 2115244 0 2024-04-12T05:37:08.448 INFO:tasks.workunit.client.0.smithi032.stdout:8/625: creat d0/d36/d84/db3/fe1 x:0 0 0 2024-04-12T05:37:08.448 INFO:tasks.workunit.client.0.smithi032.stdout:9/499: dwrite d9/d15/d1a/d24/d27/f90 [0,4194304] 0 2024-04-12T05:37:08.449 INFO:tasks.workunit.client.0.smithi032.stdout:6/707: rmdir d4/de0/de7/d37/d47 39 2024-04-12T05:37:08.449 INFO:tasks.workunit.client.0.smithi032.stdout:0/635: dwrite d2/d28/d87/d79/fce [0,4194304] 0 2024-04-12T05:37:08.454 INFO:tasks.workunit.client.0.smithi032.stdout:8/626: mkdir d0/d36/de2 0 2024-04-12T05:37:08.456 INFO:tasks.workunit.client.0.smithi032.stdout:6/708: creat d4/d21/d2e/ff0 x:0 0 0 2024-04-12T05:37:08.458 INFO:tasks.workunit.client.0.smithi032.stdout:3/631: dwrite d12/d2e/d3a/d63/f8d [0,4194304] 0 2024-04-12T05:37:08.458 INFO:tasks.workunit.client.0.smithi032.stdout:7/603: write d0/d10/f62 [2986188,112260] 0 2024-04-12T05:37:08.460 INFO:tasks.workunit.client.0.smithi032.stdout:8/627: dread - d0/d36/d84/f92 zero size 2024-04-12T05:37:08.462 INFO:tasks.workunit.client.0.smithi032.stdout:3/632: stat d12/d2e/c68 0 2024-04-12T05:37:08.462 INFO:tasks.workunit.client.0.smithi032.stdout:9/500: write d9/d15/d1a/d1d/d5a/d6b/f6c [1920807,89500] 0 2024-04-12T05:37:08.462 INFO:tasks.workunit.client.0.smithi032.stdout:6/709: read - d4/de0/de7/d88/fcc zero size 2024-04-12T05:37:08.464 INFO:tasks.workunit.client.0.smithi032.stdout:8/628: write d0/f2f [2232708,4225] 0 2024-04-12T05:37:08.466 INFO:tasks.workunit.client.0.smithi032.stdout:3/633: write f10 [4264975,55926] 0 2024-04-12T05:37:08.467 INFO:tasks.workunit.client.0.smithi032.stdout:5/619: sync 2024-04-12T05:37:08.467 INFO:tasks.workunit.client.0.smithi032.stdout:2/612: sync 2024-04-12T05:37:08.468 INFO:tasks.workunit.client.0.smithi032.stdout:9/501: mknod d9/d15/d1a/d24/d71/cac 0 2024-04-12T05:37:08.469 INFO:tasks.workunit.client.0.smithi032.stdout:5/620: fdatasync d2/f60 0 2024-04-12T05:37:08.471 INFO:tasks.workunit.client.0.smithi032.stdout:6/710: rename d4/d14/fc1 to d4/de0/deb/dec/ff1 0 2024-04-12T05:37:08.471 INFO:tasks.workunit.client.0.smithi032.stdout:0/636: dwrite d2/d8/df/d24/f9c [0,4194304] 0 2024-04-12T05:37:08.473 INFO:tasks.workunit.client.0.smithi032.stdout:3/634: creat d12/d2f/d89/d9d/fce x:0 0 0 2024-04-12T05:37:08.473 INFO:tasks.workunit.client.0.smithi032.stdout:8/629: write d0/db/dd/f17 [1107182,94553] 0 2024-04-12T05:37:08.474 INFO:tasks.workunit.client.0.smithi032.stdout:2/613: creat dd/d19/d9d/d42/d4a/fc3 x:0 0 0 2024-04-12T05:37:08.476 INFO:tasks.workunit.client.0.smithi032.stdout:5/621: truncate d2/df/d25/d1e/f2f 4868180 0 2024-04-12T05:37:08.478 INFO:tasks.workunit.client.0.smithi032.stdout:6/711: write d4/d21/d27/dad/d55/fe4 [269732,77573] 0 2024-04-12T05:37:08.478 INFO:tasks.workunit.client.0.smithi032.stdout:8/630: chown d0/c43 61003 1 2024-04-12T05:37:08.480 INFO:tasks.workunit.client.0.smithi032.stdout:5/622: fdatasync d2/df/fb1 0 2024-04-12T05:37:08.481 INFO:tasks.workunit.client.0.smithi032.stdout:8/631: fsync d0/d36/d84/dad/fb6 0 2024-04-12T05:37:08.483 INFO:tasks.workunit.client.0.smithi032.stdout:8/632: chown d0/db/dd/f75 360772 1 2024-04-12T05:37:08.484 INFO:tasks.workunit.client.0.smithi032.stdout:6/712: write d4/d21/d27/f8e [4784040,86942] 0 2024-04-12T05:37:08.485 INFO:tasks.workunit.client.0.smithi032.stdout:2/614: dread f8 [0,4194304] 0 2024-04-12T05:37:08.487 INFO:tasks.workunit.client.0.smithi032.stdout:8/633: read d0/db/dd/d11/d1b/d22/d51/f59 [3413991,103998] 0 2024-04-12T05:37:08.487 INFO:tasks.workunit.client.0.smithi032.stdout:0/637: getdents d2/d29/d77/dc7 0 2024-04-12T05:37:08.492 INFO:tasks.workunit.client.0.smithi032.stdout:7/604: dread d0/d34/dbe/f7e [0,4194304] 0 2024-04-12T05:37:08.493 INFO:tasks.workunit.client.0.smithi032.stdout:7/605: chown d0/d13/d21/d7c/fa8 1531637 1 2024-04-12T05:37:08.494 INFO:tasks.workunit.client.0.smithi032.stdout:7/606: fsync d0/d13/d21/d7c/fb2 0 2024-04-12T05:37:08.500 INFO:tasks.workunit.client.0.smithi032.stdout:5/623: dread d2/f4 [0,4194304] 0 2024-04-12T05:37:08.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:08 smithi067 ceph-mon[49010]: pgmap v265: 65 pgs: 65 active+clean; 1.1 GiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 33 MiB/s rd, 152 MiB/s wr, 370 op/s 2024-04-12T05:37:08.572 INFO:tasks.workunit.client.0.smithi032.stdout:1/517: sync 2024-04-12T05:37:08.572 INFO:tasks.workunit.client.0.smithi032.stdout:4/553: sync 2024-04-12T05:37:08.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:08 smithi032 ceph-mon[39731]: pgmap v265: 65 pgs: 65 active+clean; 1.1 GiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 33 MiB/s rd, 152 MiB/s wr, 370 op/s 2024-04-12T05:37:08.592 INFO:tasks.workunit.client.0.smithi032.stdout:6/713: chown d4/de0/de7/f49 50382436 1 2024-04-12T05:37:08.594 INFO:tasks.workunit.client.0.smithi032.stdout:3/635: rename d12/d2e/d3a/d63/da6/fb7 to d12/d2e/d3a/d63/fcf 0 2024-04-12T05:37:08.594 INFO:tasks.workunit.client.0.smithi032.stdout:6/714: write d4/de0/de7/d88/fac [229762,76389] 0 2024-04-12T05:37:08.599 INFO:tasks.workunit.client.0.smithi032.stdout:9/502: sync 2024-04-12T05:37:08.599 INFO:tasks.workunit.client.0.smithi032.stdout:2/615: creat dd/d19/d9d/d42/d4a/fc4 x:0 0 0 2024-04-12T05:37:08.600 INFO:tasks.workunit.client.0.smithi032.stdout:8/634: creat d0/db/dd/d11/d1b/d5f/fe3 x:0 0 0 2024-04-12T05:37:08.601 INFO:tasks.workunit.client.0.smithi032.stdout:7/607: symlink d0/de/df/d29/lc8 0 2024-04-12T05:37:08.602 INFO:tasks.workunit.client.0.smithi032.stdout:5/624: write d2/df/fb1 [443831,1887] 0 2024-04-12T05:37:08.604 INFO:tasks.workunit.client.0.smithi032.stdout:1/518: write d5/de/d17/d1b/f1d [4513641,90417] 0 2024-04-12T05:37:08.605 INFO:tasks.workunit.client.0.smithi032.stdout:5/625: chown d2/d3c/d48/d4b/dc4 4288 1 2024-04-12T05:37:08.608 INFO:tasks.workunit.client.0.smithi032.stdout:6/715: truncate d4/de0/de7/d92/d9e/db2/fde 181052 0 2024-04-12T05:37:08.609 INFO:tasks.workunit.client.0.smithi032.stdout:0/638: link d2/la0 d2/dc/d3e/dac/ld0 0 2024-04-12T05:37:08.611 INFO:tasks.workunit.client.0.smithi032.stdout:3/636: rename d12/f77 to d12/d2e/d3a/d63/da6/dac/fd0 0 2024-04-12T05:37:08.611 INFO:tasks.workunit.client.0.smithi032.stdout:2/616: write dd/d19/d9d/d42/f84 [1580748,120587] 0 2024-04-12T05:37:08.611 INFO:tasks.workunit.client.0.smithi032.stdout:5/626: truncate d2/d9b/fb7 443655 0 2024-04-12T05:37:08.613 INFO:tasks.workunit.client.0.smithi032.stdout:6/716: write d4/de0/de7/dc4/fe9 [295350,23630] 0 2024-04-12T05:37:08.613 INFO:tasks.workunit.client.0.smithi032.stdout:1/519: creat d5/de/d17/d35/d3e/fa3 x:0 0 0 2024-04-12T05:37:08.615 INFO:tasks.workunit.client.0.smithi032.stdout:9/503: truncate d9/d15/d2c/d41/f45 2327647 0 2024-04-12T05:37:08.616 INFO:tasks.workunit.client.0.smithi032.stdout:4/554: dwrite d4/d13/d44/fa0 [0,4194304] 0 2024-04-12T05:37:08.618 INFO:tasks.workunit.client.0.smithi032.stdout:7/608: rename d0/d34/dbe/fae to d0/d10/d70/d7b/fc9 0 2024-04-12T05:37:08.618 INFO:tasks.workunit.client.0.smithi032.stdout:5/627: dread - d2/df/d31/d92/fb3 zero size 2024-04-12T05:37:08.619 INFO:tasks.workunit.client.0.smithi032.stdout:3/637: write d12/d2e/d3a/f96 [33776,90604] 0 2024-04-12T05:37:08.620 INFO:tasks.workunit.client.0.smithi032.stdout:2/617: unlink dd/d19/d82/f8a 0 2024-04-12T05:37:08.620 INFO:tasks.workunit.client.0.smithi032.stdout:0/639: dread d2/dc/fa6 [0,4194304] 0 2024-04-12T05:37:08.623 INFO:tasks.workunit.client.0.smithi032.stdout:0/640: truncate d2/dc/d3e/f46 4382192 0 2024-04-12T05:37:08.626 INFO:tasks.workunit.client.0.smithi032.stdout:1/520: rmdir d5/de/d17/d35/d78 39 2024-04-12T05:37:08.630 INFO:tasks.workunit.client.0.smithi032.stdout:5/628: dread d2/d3c/f56 [0,4194304] 0 2024-04-12T05:37:08.631 INFO:tasks.workunit.client.0.smithi032.stdout:8/635: rename d0/d36/d84/dad/fb6 to d0/db/dd/d11/d1b/d22/d35/d44/fe4 0 2024-04-12T05:37:08.632 INFO:tasks.workunit.client.0.smithi032.stdout:4/555: dread d4/f35 [0,4194304] 0 2024-04-12T05:37:08.632 INFO:tasks.workunit.client.0.smithi032.stdout:3/638: creat d12/d2f/fd1 x:0 0 0 2024-04-12T05:37:08.633 INFO:tasks.workunit.client.0.smithi032.stdout:8/636: chown d0/db/dd/d11/d1b/d22/d54/da8 380262 1 2024-04-12T05:37:08.633 INFO:tasks.workunit.client.0.smithi032.stdout:7/609: mknod d0/de/d47/cca 0 2024-04-12T05:37:08.635 INFO:tasks.workunit.client.0.smithi032.stdout:7/610: write d0/d13/d21/f45 [2711798,127756] 0 2024-04-12T05:37:08.635 INFO:tasks.workunit.client.0.smithi032.stdout:8/637: truncate d0/db/d38/d63/fa1 1245697 0 2024-04-12T05:37:08.636 INFO:tasks.workunit.client.0.smithi032.stdout:2/618: mkdir dd/d19/d9d/d42/da7/dc5 0 2024-04-12T05:37:08.642 INFO:tasks.workunit.client.0.smithi032.stdout:0/641: write d2/dc/d3e/dac/d51/f8d [1948382,113979] 0 2024-04-12T05:37:08.643 INFO:tasks.workunit.client.0.smithi032.stdout:3/639: creat d12/d2f/d89/fd2 x:0 0 0 2024-04-12T05:37:08.643 INFO:tasks.workunit.client.0.smithi032.stdout:3/640: readlink d12/d2e/l60 0 2024-04-12T05:37:08.644 INFO:tasks.workunit.client.0.smithi032.stdout:4/556: mkdir d4/d87/d96/d2f/d6e/dc0 0 2024-04-12T05:37:08.646 INFO:tasks.workunit.client.0.smithi032.stdout:6/717: creat d4/ff2 x:0 0 0 2024-04-12T05:37:08.646 INFO:tasks.workunit.client.0.smithi032.stdout:4/557: truncate d4/d13/d19/fa1 520092 0 2024-04-12T05:37:08.648 INFO:tasks.workunit.client.0.smithi032.stdout:8/638: creat d0/d58/fe5 x:0 0 0 2024-04-12T05:37:08.649 INFO:tasks.workunit.client.0.smithi032.stdout:9/504: dwrite d9/d15/f44 [0,4194304] 0 2024-04-12T05:37:08.652 INFO:tasks.workunit.client.0.smithi032.stdout:1/521: symlink d5/de/d17/d35/d61/d77/d8f/da0/la4 0 2024-04-12T05:37:08.653 INFO:tasks.workunit.client.0.smithi032.stdout:3/641: truncate d12/f13 3313687 0 2024-04-12T05:37:08.654 INFO:tasks.workunit.client.0.smithi032.stdout:6/718: mknod d4/d74/cf3 0 2024-04-12T05:37:08.655 INFO:tasks.workunit.client.0.smithi032.stdout:6/719: readlink d4/d21/d2e/lc3 0 2024-04-12T05:37:08.657 INFO:tasks.workunit.client.0.smithi032.stdout:8/639: unlink d0/db/dd/d11/d1b/d22/d54/da8/fb0 0 2024-04-12T05:37:08.659 INFO:tasks.workunit.client.0.smithi032.stdout:5/629: creat d2/fc8 x:0 0 0 2024-04-12T05:37:08.660 INFO:tasks.workunit.client.0.smithi032.stdout:1/522: creat d5/de/d17/d35/d61/d77/d8f/d9c/fa5 x:0 0 0 2024-04-12T05:37:08.662 INFO:tasks.workunit.client.0.smithi032.stdout:4/558: fdatasync d4/d13/d19/f2a 0 2024-04-12T05:37:08.664 INFO:tasks.workunit.client.0.smithi032.stdout:9/505: link d9/d32/f53 d9/d15/d1a/d31/d65/d80/fad 0 2024-04-12T05:37:08.665 INFO:tasks.workunit.client.0.smithi032.stdout:0/642: creat d2/d29/d8a/fd1 x:0 0 0 2024-04-12T05:37:08.666 INFO:tasks.workunit.client.0.smithi032.stdout:8/640: symlink d0/db/dd/d11/d1b/d5f/le6 0 2024-04-12T05:37:08.667 INFO:tasks.workunit.client.0.smithi032.stdout:3/642: mknod d12/d2e/d3a/d63/cd3 0 2024-04-12T05:37:08.668 INFO:tasks.workunit.client.0.smithi032.stdout:7/611: getdents d0/d10 0 2024-04-12T05:37:08.669 INFO:tasks.workunit.client.0.smithi032.stdout:1/523: mknod d5/de/d17/d35/d73/ca6 0 2024-04-12T05:37:08.671 INFO:tasks.workunit.client.0.smithi032.stdout:2/619: truncate dd/d19/d9d/d42/d4a/fb3 1513485 0 2024-04-12T05:37:08.671 INFO:tasks.workunit.client.0.smithi032.stdout:1/524: write d5/de/d17/d1b/f1d [768780,87774] 0 2024-04-12T05:37:08.672 INFO:tasks.workunit.client.0.smithi032.stdout:8/641: dread d0/db/d38/d63/fa1 [0,4194304] 0 2024-04-12T05:37:08.672 INFO:tasks.workunit.client.0.smithi032.stdout:4/559: creat d4/d87/d96/d2f/fc1 x:0 0 0 2024-04-12T05:37:08.673 INFO:tasks.workunit.client.0.smithi032.stdout:8/642: fdatasync d0/d36/f4a 0 2024-04-12T05:37:08.674 INFO:tasks.workunit.client.0.smithi032.stdout:6/720: mkdir d4/d21/d27/dad/d55/df4 0 2024-04-12T05:37:08.676 INFO:tasks.workunit.client.0.smithi032.stdout:0/643: symlink d2/dc/d3e/dac/d51/d83/ld2 0 2024-04-12T05:37:08.678 INFO:tasks.workunit.client.0.smithi032.stdout:8/643: dread d0/db/dd/d11/d1b/d22/d51/fa9 [0,4194304] 0 2024-04-12T05:37:08.680 INFO:tasks.workunit.client.0.smithi032.stdout:3/643: unlink d12/d2e/f86 0 2024-04-12T05:37:08.680 INFO:tasks.workunit.client.0.smithi032.stdout:0/644: dread d2/d29/f47 [8388608,4194304] 0 2024-04-12T05:37:08.681 INFO:tasks.workunit.client.0.smithi032.stdout:7/612: creat d0/d34/fcb x:0 0 0 2024-04-12T05:37:08.683 INFO:tasks.workunit.client.0.smithi032.stdout:5/630: link d2/df/d25/d1e/f40 d2/df/d25/d50/daa/fc9 0 2024-04-12T05:37:08.684 INFO:tasks.workunit.client.0.smithi032.stdout:5/631: dread - d2/df/da9/dc1/d78/f94 zero size 2024-04-12T05:37:08.685 INFO:tasks.workunit.client.0.smithi032.stdout:2/620: mkdir dd/d19/d20/dc6 0 2024-04-12T05:37:08.688 INFO:tasks.workunit.client.0.smithi032.stdout:1/525: rename d5/f23 to d5/de/d17/fa7 0 2024-04-12T05:37:08.689 INFO:tasks.workunit.client.0.smithi032.stdout:2/621: dread dd/d19/d9d/d1c/d64/f7d [0,4194304] 0 2024-04-12T05:37:08.690 INFO:tasks.workunit.client.0.smithi032.stdout:4/560: rmdir d4/d87/d96/d68 39 2024-04-12T05:37:08.696 INFO:tasks.workunit.client.0.smithi032.stdout:4/561: dwrite d4/d87/d96/d2f/d6e/f7f [0,4194304] 0 2024-04-12T05:37:08.703 INFO:tasks.workunit.client.0.smithi032.stdout:1/526: symlink d5/de/d17/d35/d61/d77/d8f/la8 0 2024-04-12T05:37:08.704 INFO:tasks.workunit.client.0.smithi032.stdout:8/644: dwrite d0/db/dd/d11/d1b/d22/d51/fa9 [0,4194304] 0 2024-04-12T05:37:08.708 INFO:tasks.workunit.client.0.smithi032.stdout:0/645: fdatasync d2/dc/d3e/dac/f35 0 2024-04-12T05:37:08.709 INFO:tasks.workunit.client.0.smithi032.stdout:7/613: dread d0/d1c/d71/f8a [0,4194304] 0 2024-04-12T05:37:08.710 INFO:tasks.workunit.client.0.smithi032.stdout:4/562: mkdir d4/d87/d96/d2f/d7c/dc2 0 2024-04-12T05:37:08.714 INFO:tasks.workunit.client.0.smithi032.stdout:4/563: chown d4/d13/d19/f2a 400077 1 2024-04-12T05:37:08.714 INFO:tasks.workunit.client.0.smithi032.stdout:1/527: dwrite d5/de/d17/d1b/f1d [0,4194304] 0 2024-04-12T05:37:08.714 INFO:tasks.workunit.client.0.smithi032.stdout:8/645: dread d0/db/dd/d11/f2c [0,4194304] 0 2024-04-12T05:37:08.716 INFO:tasks.workunit.client.0.smithi032.stdout:1/528: chown d5/de/d17/d35/d3b/d49/l51 425889821 1 2024-04-12T05:37:08.718 INFO:tasks.workunit.client.0.smithi032.stdout:0/646: dwrite d2/d28/d87/f43 [0,4194304] 0 2024-04-12T05:37:08.720 INFO:tasks.workunit.client.0.smithi032.stdout:6/721: rename d4/de0/de7/d88/ca0 to d4/de0/de7/d92/d9e/cf5 0 2024-04-12T05:37:08.721 INFO:tasks.workunit.client.1.smithi067.stdout:5/44: sync 2024-04-12T05:37:08.722 INFO:tasks.workunit.client.1.smithi067.stdout:6/3: sync 2024-04-12T05:37:08.722 INFO:tasks.workunit.client.1.smithi067.stdout:1/58: sync 2024-04-12T05:37:08.722 INFO:tasks.workunit.client.1.smithi067.stdout:7/24: sync 2024-04-12T05:37:08.722 INFO:tasks.workunit.client.1.smithi067.stdout:9/9: sync 2024-04-12T05:37:08.722 INFO:tasks.workunit.client.1.smithi067.stdout:3/16: sync 2024-04-12T05:37:08.723 INFO:tasks.workunit.client.0.smithi032.stdout:2/622: mkdir dd/d19/d9d/d42/dc7 0 2024-04-12T05:37:08.723 INFO:tasks.workunit.client.1.smithi067.stdout:4/2: sync 2024-04-12T05:37:08.723 INFO:tasks.workunit.client.1.smithi067.stdout:4/3: dwrite - no filename 2024-04-12T05:37:08.724 INFO:tasks.workunit.client.1.smithi067.stdout:0/122: sync 2024-04-12T05:37:08.724 INFO:tasks.workunit.client.1.smithi067.stdout:2/28: sync 2024-04-12T05:37:08.724 INFO:tasks.workunit.client.1.smithi067.stdout:8/25: sync 2024-04-12T05:37:08.724 INFO:tasks.workunit.client.1.smithi067.stdout:4/4: chown . 1 1 2024-04-12T05:37:08.724 INFO:tasks.workunit.client.1.smithi067.stdout:4/5: fsync - no filename 2024-04-12T05:37:08.724 INFO:tasks.workunit.client.0.smithi032.stdout:2/623: chown dd/d19/d20/f62 13859 1 2024-04-12T05:37:08.724 INFO:tasks.workunit.client.1.smithi067.stdout:6/4: chown f0 16342988 1 2024-04-12T05:37:08.725 INFO:tasks.workunit.client.1.smithi067.stdout:1/59: rmdir d4/d9 39 2024-04-12T05:37:08.727 INFO:tasks.workunit.client.0.smithi032.stdout:9/506: write d9/d15/d1a/f56 [376909,91087] 0 2024-04-12T05:37:08.727 INFO:tasks.workunit.client.0.smithi032.stdout:6/722: dwrite d4/d21/d27/dad/d55/fe4 [0,4194304] 0 2024-04-12T05:37:08.729 INFO:tasks.workunit.client.1.smithi067.stdout:5/45: mknod d0/d6/d8/cf 0 2024-04-12T05:37:08.729 INFO:tasks.workunit.client.1.smithi067.stdout:1/60: readlink d4/lf 0 2024-04-12T05:37:08.729 INFO:tasks.workunit.client.0.smithi032.stdout:9/507: write d9/d32/d6e/f93 [63235,10572] 0 2024-04-12T05:37:08.730 INFO:tasks.workunit.client.0.smithi032.stdout:7/614: symlink d0/lcc 0 2024-04-12T05:37:08.730 INFO:tasks.workunit.client.1.smithi067.stdout:2/29: stat d5/c8 0 2024-04-12T05:37:08.731 INFO:tasks.workunit.client.1.smithi067.stdout:3/17: rename f3 to f5 0 2024-04-12T05:37:08.732 INFO:tasks.workunit.client.0.smithi032.stdout:4/564: write d4/f54 [607124,72698] 0 2024-04-12T05:37:08.733 INFO:tasks.workunit.client.1.smithi067.stdout:0/123: symlink dd/d12/d14/l27 0 2024-04-12T05:37:08.734 INFO:tasks.workunit.client.0.smithi032.stdout:1/529: mkdir d5/de/d17/d35/d73/da9 0 2024-04-12T05:37:08.735 INFO:tasks.workunit.client.1.smithi067.stdout:8/26: rename f2 to d6/f7 0 2024-04-12T05:37:08.735 INFO:tasks.workunit.client.1.smithi067.stdout:5/46: dread - d0/fc zero size 2024-04-12T05:37:08.737 INFO:tasks.workunit.client.1.smithi067.stdout:1/61: chown d4/d9/fa 7001878 1 2024-04-12T05:37:08.737 INFO:tasks.workunit.client.1.smithi067.stdout:1/62: readlink d4/lf 0 2024-04-12T05:37:08.739 INFO:tasks.workunit.client.0.smithi032.stdout:5/632: rename d2/fc8 to d2/df/da9/dc1/d6c/fca 0 2024-04-12T05:37:08.739 INFO:tasks.workunit.client.1.smithi067.stdout:4/6: getdents . 0 2024-04-12T05:37:08.741 INFO:tasks.workunit.client.1.smithi067.stdout:0/124: unlink dd/d12/d14/l27 0 2024-04-12T05:37:08.742 INFO:tasks.workunit.client.0.smithi032.stdout:9/508: mknod d9/d15/d1a/d24/cae 0 2024-04-12T05:37:08.742 INFO:tasks.workunit.client.0.smithi032.stdout:8/646: symlink d0/db/dd/ddd/le7 0 2024-04-12T05:37:08.742 INFO:tasks.workunit.client.1.smithi067.stdout:8/27: mkdir d6/d8 0 2024-04-12T05:37:08.744 INFO:tasks.workunit.client.1.smithi067.stdout:5/47: creat d0/d6/f10 x:0 0 0 2024-04-12T05:37:08.744 INFO:tasks.workunit.client.1.smithi067.stdout:7/25: link d0/d2/f3 d0/f9 0 2024-04-12T05:37:08.748 INFO:tasks.workunit.client.0.smithi032.stdout:6/723: dread d4/de0/de7/d37/d47/f56 [0,4194304] 0 2024-04-12T05:37:08.748 INFO:tasks.workunit.client.0.smithi032.stdout:5/633: unlink d2/df/f3f 0 2024-04-12T05:37:08.751 INFO:tasks.workunit.client.1.smithi067.stdout:6/5: write f0 [242453,106444] 0 2024-04-12T05:37:08.752 INFO:tasks.workunit.client.1.smithi067.stdout:4/7: creat f0 x:0 0 0 2024-04-12T05:37:08.754 INFO:tasks.workunit.client.0.smithi032.stdout:7/615: truncate d0/d13/f27 1299090 0 2024-04-12T05:37:08.755 INFO:tasks.workunit.client.0.smithi032.stdout:4/565: symlink d4/d87/d96/d2f/d6e/dc0/lc3 0 2024-04-12T05:37:08.755 INFO:tasks.workunit.client.1.smithi067.stdout:8/28: creat d6/d8/f9 x:0 0 0 2024-04-12T05:37:08.756 INFO:tasks.workunit.client.0.smithi032.stdout:7/616: dread d0/d13/f8c [0,4194304] 0 2024-04-12T05:37:08.757 INFO:tasks.workunit.client.1.smithi067.stdout:6/6: dread f0 [0,4194304] 0 2024-04-12T05:37:08.757 INFO:tasks.workunit.client.1.smithi067.stdout:3/18: read f5 [2093829,51662] 0 2024-04-12T05:37:08.757 INFO:tasks.workunit.client.1.smithi067.stdout:5/48: dwrite d0/d6/f10 [0,4194304] 0 2024-04-12T05:37:08.759 INFO:tasks.workunit.client.0.smithi032.stdout:1/530: rename d5/de/d17/d35/d3e/c7b to d5/de/d17/d35/d61/d77/d8f/da0/caa 0 2024-04-12T05:37:08.760 INFO:tasks.workunit.client.1.smithi067.stdout:1/63: creat d4/f10 x:0 0 0 2024-04-12T05:37:08.761 INFO:tasks.workunit.client.1.smithi067.stdout:7/26: creat d0/d2/d5/fa x:0 0 0 2024-04-12T05:37:08.762 INFO:tasks.workunit.client.0.smithi032.stdout:1/531: write d5/de/d2a/f65 [902570,59411] 0 2024-04-12T05:37:08.762 INFO:tasks.workunit.client.1.smithi067.stdout:0/125: write f1 [4782944,117385] 0 2024-04-12T05:37:08.763 INFO:tasks.workunit.client.0.smithi032.stdout:6/724: mkdir d4/de0/de7/d92/d9e/df6 0 2024-04-12T05:37:08.764 INFO:tasks.workunit.client.1.smithi067.stdout:8/29: chown d6/d8/f9 46256267 1 2024-04-12T05:37:08.765 INFO:tasks.workunit.client.1.smithi067.stdout:0/126: chown dd/d12/d14 76695 1 2024-04-12T05:37:08.766 INFO:tasks.workunit.client.1.smithi067.stdout:6/7: dread f0 [0,4194304] 0 2024-04-12T05:37:08.768 INFO:tasks.workunit.client.1.smithi067.stdout:4/8: unlink f0 0 2024-04-12T05:37:08.769 INFO:tasks.workunit.client.1.smithi067.stdout:2/30: dwrite f3 [0,4194304] 0 2024-04-12T05:37:08.770 INFO:tasks.workunit.client.0.smithi032.stdout:8/647: mknod d0/d36/d84/dad/da4/dcb/ce8 0 2024-04-12T05:37:08.770 INFO:tasks.workunit.client.0.smithi032.stdout:4/566: dread d4/f85 [0,4194304] 0 2024-04-12T05:37:08.771 INFO:tasks.workunit.client.1.smithi067.stdout:7/27: unlink d0/f4 0 2024-04-12T05:37:08.771 INFO:tasks.workunit.client.1.smithi067.stdout:1/64: truncate d4/d9/fa 1068105 0 2024-04-12T05:37:08.772 INFO:tasks.workunit.client.0.smithi032.stdout:0/647: truncate d2/d28/d87/d79/fce 1966292 0 2024-04-12T05:37:08.772 INFO:tasks.workunit.client.1.smithi067.stdout:8/30: creat d6/d8/fa x:0 0 0 2024-04-12T05:37:08.773 INFO:tasks.workunit.client.1.smithi067.stdout:6/8: write f0 [1068446,53627] 0 2024-04-12T05:37:08.773 INFO:tasks.workunit.client.0.smithi032.stdout:2/624: getdents dd/d19/d9d/d1c/d64/d32 0 2024-04-12T05:37:08.775 INFO:tasks.workunit.client.0.smithi032.stdout:7/617: write d0/de/df/d20/f6b [3787348,130230] 0 2024-04-12T05:37:08.775 INFO:tasks.workunit.client.1.smithi067.stdout:4/9: symlink l1 0 2024-04-12T05:37:08.777 INFO:tasks.workunit.client.0.smithi032.stdout:6/725: rmdir d4/d21/db4 39 2024-04-12T05:37:08.777 INFO:tasks.workunit.client.0.smithi032.stdout:4/567: truncate d4/d87/d96/f59 490848 0 2024-04-12T05:37:08.779 INFO:tasks.workunit.client.0.smithi032.stdout:5/634: mknod d2/d3c/d48/d4b/dc4/ccb 0 2024-04-12T05:37:08.781 INFO:tasks.workunit.client.0.smithi032.stdout:1/532: fdatasync d5/de/d17/f86 0 2024-04-12T05:37:08.781 INFO:tasks.workunit.client.1.smithi067.stdout:1/65: symlink d4/d9/l11 0 2024-04-12T05:37:08.782 INFO:tasks.workunit.client.1.smithi067.stdout:3/19: write f5 [3382041,120110] 0 2024-04-12T05:37:08.785 INFO:tasks.workunit.client.1.smithi067.stdout:2/31: dread f4 [0,4194304] 0 2024-04-12T05:37:08.785 INFO:tasks.workunit.client.0.smithi032.stdout:8/648: unlink d0/db/dd/d11/d1b/d22/d51/fb8 0 2024-04-12T05:37:08.786 INFO:tasks.workunit.client.1.smithi067.stdout:6/9: symlink l1 0 2024-04-12T05:37:08.786 INFO:tasks.workunit.client.0.smithi032.stdout:7/618: symlink d0/d1c/d4c/lcd 0 2024-04-12T05:37:08.787 INFO:tasks.workunit.client.1.smithi067.stdout:8/31: read d6/f7 [4003442,52876] 0 2024-04-12T05:37:08.787 INFO:tasks.workunit.client.0.smithi032.stdout:2/625: dwrite f1 [0,4194304] 0 2024-04-12T05:37:08.790 INFO:tasks.workunit.client.0.smithi032.stdout:6/726: rmdir d4/d21 39 2024-04-12T05:37:08.790 INFO:tasks.workunit.client.1.smithi067.stdout:4/10: rename l1 to l2 0 2024-04-12T05:37:08.790 INFO:tasks.workunit.client.1.smithi067.stdout:4/11: truncate - no filename 2024-04-12T05:37:08.792 INFO:tasks.workunit.client.0.smithi032.stdout:2/626: write dd/d19/d9d/d42/da7/fb8 [412084,92075] 0 2024-04-12T05:37:08.795 INFO:tasks.workunit.client.0.smithi032.stdout:0/648: symlink d2/d8/df/ld3 0 2024-04-12T05:37:08.795 INFO:tasks.workunit.client.0.smithi032.stdout:4/568: creat d4/d87/d96/d2f/d3b/fc4 x:0 0 0 2024-04-12T05:37:08.796 INFO:tasks.workunit.client.1.smithi067.stdout:5/49: link d0/d6/d8/l9 d0/d6/d8/l11 0 2024-04-12T05:37:08.796 INFO:tasks.workunit.client.0.smithi032.stdout:5/635: mknod d2/dd/d5d/ccc 0 2024-04-12T05:37:08.797 INFO:tasks.workunit.client.0.smithi032.stdout:1/533: rmdir d5/de/d17/d35/d3b 39 2024-04-12T05:37:08.798 INFO:tasks.workunit.client.1.smithi067.stdout:2/32: mknod d5/cb 0 2024-04-12T05:37:08.799 INFO:tasks.workunit.client.1.smithi067.stdout:8/32: truncate f4 2425012 0 2024-04-12T05:37:08.800 INFO:tasks.workunit.client.1.smithi067.stdout:6/10: write f0 [715343,109773] 0 2024-04-12T05:37:08.800 INFO:tasks.workunit.client.0.smithi032.stdout:8/649: creat d0/d36/d84/db3/fe9 x:0 0 0 2024-04-12T05:37:08.802 INFO:tasks.workunit.client.1.smithi067.stdout:4/12: creat f3 x:0 0 0 2024-04-12T05:37:08.803 INFO:tasks.workunit.client.0.smithi032.stdout:0/649: symlink d2/d29/ld4 0 2024-04-12T05:37:08.804 INFO:tasks.workunit.client.0.smithi032.stdout:6/727: mknod d4/d21/d2e/cf7 0 2024-04-12T05:37:08.805 INFO:tasks.workunit.client.0.smithi032.stdout:7/619: dwrite d0/de/df/d29/f46 [4194304,4194304] 0 2024-04-12T05:37:08.805 INFO:tasks.workunit.client.1.smithi067.stdout:5/50: rename d0/fd to d0/d6/f12 0 2024-04-12T05:37:08.805 INFO:tasks.workunit.client.1.smithi067.stdout:6/11: read f0 [148416,73183] 0 2024-04-12T05:37:08.805 INFO:tasks.workunit.client.0.smithi032.stdout:8/650: write d0/db/dd/f27 [3361788,74373] 0 2024-04-12T05:37:08.805 INFO:tasks.workunit.client.1.smithi067.stdout:0/127: dwrite dd/d12/f21 [0,4194304] 0 2024-04-12T05:37:08.806 INFO:tasks.workunit.client.0.smithi032.stdout:2/627: fsync f0 0 2024-04-12T05:37:08.807 INFO:tasks.workunit.client.1.smithi067.stdout:4/13: write f3 [465930,106360] 0 2024-04-12T05:37:08.808 INFO:tasks.workunit.client.0.smithi032.stdout:2/628: dread - dd/d19/d20/fa0 zero size 2024-04-12T05:37:08.808 INFO:tasks.workunit.client.0.smithi032.stdout:7/620: chown d0/d13/d21/d30/la5 1 1 2024-04-12T05:37:08.809 INFO:tasks.workunit.client.0.smithi032.stdout:8/651: chown d0/db/dd/d11/d1b/d22/d35/c7f 9560 1 2024-04-12T05:37:08.811 INFO:tasks.workunit.client.0.smithi032.stdout:8/652: readlink d0/d36/l37 0 2024-04-12T05:37:08.811 INFO:tasks.workunit.client.0.smithi032.stdout:2/629: chown dd/d19/d20/d83 221275 1 2024-04-12T05:37:08.816 INFO:tasks.workunit.client.0.smithi032.stdout:8/653: truncate d0/d36/d84/db3/fe9 467155 0 2024-04-12T05:37:08.817 INFO:tasks.workunit.client.0.smithi032.stdout:8/654: stat d0/db/dd/d11/d1b/d22/d35/d44/fe4 0 2024-04-12T05:37:08.817 INFO:tasks.workunit.client.0.smithi032.stdout:4/569: truncate d4/d13/d44/f5a 3103260 0 2024-04-12T05:37:08.818 INFO:tasks.workunit.client.1.smithi067.stdout:8/33: unlink d6/d8/fa 0 2024-04-12T05:37:08.818 INFO:tasks.workunit.client.1.smithi067.stdout:3/20: link l4 l6 0 2024-04-12T05:37:08.818 INFO:tasks.workunit.client.0.smithi032.stdout:7/621: mknod d0/d13/d21/d7c/cce 0 2024-04-12T05:37:08.819 INFO:tasks.workunit.client.0.smithi032.stdout:7/622: chown d0/d13/f16 23070 1 2024-04-12T05:37:08.820 INFO:tasks.workunit.client.1.smithi067.stdout:4/14: dread f3 [0,4194304] 0 2024-04-12T05:37:08.821 INFO:tasks.workunit.client.1.smithi067.stdout:5/51: rename d0/d6/d8/cf to d0/da/de/c13 0 2024-04-12T05:37:08.822 INFO:tasks.workunit.client.1.smithi067.stdout:6/12: symlink l2 0 2024-04-12T05:37:08.823 INFO:tasks.workunit.client.0.smithi032.stdout:7/623: fdatasync d0/fc4 0 2024-04-12T05:37:08.823 INFO:tasks.workunit.client.0.smithi032.stdout:5/636: getdents d2/dd/d6f 0 2024-04-12T05:37:08.823 INFO:tasks.workunit.client.0.smithi032.stdout:4/570: mkdir d4/d87/d96/d32/dc5 0 2024-04-12T05:37:08.824 INFO:tasks.workunit.client.0.smithi032.stdout:7/624: creat d0/de/df/d29/fcf x:0 0 0 2024-04-12T05:37:08.824 INFO:tasks.workunit.client.1.smithi067.stdout:0/128: dread - dd/d12/d14/d1a/f1e zero size 2024-04-12T05:37:08.824 INFO:tasks.workunit.client.1.smithi067.stdout:4/15: chown f3 489287380 1 2024-04-12T05:37:08.825 INFO:tasks.workunit.client.1.smithi067.stdout:1/66: write d4/fc [1014393,56053] 0 2024-04-12T05:37:08.825 INFO:tasks.workunit.client.0.smithi032.stdout:4/571: write d4/d87/d96/d2f/d6e/f7f [1705807,15190] 0 2024-04-12T05:37:08.825 INFO:tasks.workunit.client.1.smithi067.stdout:5/52: dread d0/f7 [0,4194304] 0 2024-04-12T05:37:08.826 INFO:tasks.workunit.client.0.smithi032.stdout:6/728: rename d4/de0/deb/dec/ff1 to d4/de0/de7/ff8 0 2024-04-12T05:37:08.827 INFO:tasks.workunit.client.0.smithi032.stdout:4/572: truncate d4/d13/f71 5096082 0 2024-04-12T05:37:08.828 INFO:tasks.workunit.client.0.smithi032.stdout:4/573: stat d4/d87/d96/d2f/d3b/l55 0 2024-04-12T05:37:08.828 INFO:tasks.workunit.client.0.smithi032.stdout:7/625: symlink d0/dd/ld0 0 2024-04-12T05:37:08.830 INFO:tasks.workunit.client.1.smithi067.stdout:7/28: dwrite d0/f8 [0,4194304] 0 2024-04-12T05:37:08.831 INFO:tasks.workunit.client.0.smithi032.stdout:5/637: dread d2/df/f32 [0,4194304] 0 2024-04-12T05:37:08.833 INFO:tasks.workunit.client.0.smithi032.stdout:2/630: rename dd/d19/d9d/l6f to dd/d19/d9d/d1c/d64/d32/d6d/da5/lc8 0 2024-04-12T05:37:08.834 INFO:tasks.workunit.client.1.smithi067.stdout:5/53: dread d0/d6/f10 [0,4194304] 0 2024-04-12T05:37:08.834 INFO:tasks.workunit.client.0.smithi032.stdout:4/574: rmdir d4/d87/d96/d32 39 2024-04-12T05:37:08.836 INFO:tasks.workunit.client.0.smithi032.stdout:4/575: write d4/d87/d96/d4d/dbc/fb3 [963332,61974] 0 2024-04-12T05:37:08.837 INFO:tasks.workunit.client.0.smithi032.stdout:7/626: creat d0/d34/dbe/dbb/fd1 x:0 0 0 2024-04-12T05:37:08.838 INFO:tasks.workunit.client.0.smithi032.stdout:6/729: mknod d4/d14/d3c/cf9 0 2024-04-12T05:37:08.839 INFO:tasks.workunit.client.0.smithi032.stdout:3/644: sync 2024-04-12T05:37:08.840 INFO:tasks.workunit.client.0.smithi032.stdout:5/638: unlink d2/df/da9/dc1/d6c/f71 0 2024-04-12T05:37:08.841 INFO:tasks.workunit.client.1.smithi067.stdout:4/16: chown l2 257 1 2024-04-12T05:37:08.841 INFO:tasks.workunit.client.1.smithi067.stdout:4/17: chown f3 6438 1 2024-04-12T05:37:08.842 INFO:tasks.workunit.client.0.smithi032.stdout:8/655: rename d0/d36/fca to d0/d58/fea 0 2024-04-12T05:37:08.844 INFO:tasks.workunit.client.1.smithi067.stdout:2/33: link f4 d5/da/fc 0 2024-04-12T05:37:08.844 INFO:tasks.workunit.client.0.smithi032.stdout:2/631: rmdir dd/d19/d9d/d1c/d64/d32/d3a 39 2024-04-12T05:37:08.846 INFO:tasks.workunit.client.0.smithi032.stdout:4/576: mkdir d4/d87/d96/d2f/d94/dc6 0 2024-04-12T05:37:08.850 INFO:tasks.workunit.client.0.smithi032.stdout:7/627: mkdir d0/d10/d70/d7b/dd2 0 2024-04-12T05:37:08.850 INFO:tasks.workunit.client.0.smithi032.stdout:6/730: symlink d4/d21/d2e/lfa 0 2024-04-12T05:37:08.851 INFO:tasks.workunit.client.1.smithi067.stdout:6/13: write f0 [1861498,89111] 0 2024-04-12T05:37:08.851 INFO:tasks.workunit.client.1.smithi067.stdout:0/129: write dd/f1c [3344022,79798] 0 2024-04-12T05:37:08.851 INFO:tasks.workunit.client.1.smithi067.stdout:7/29: read - d0/d2/d5/fa zero size 2024-04-12T05:37:08.852 INFO:tasks.workunit.client.0.smithi032.stdout:1/534: write d5/de/d17/d35/d61/f50 [1700175,123100] 0 2024-04-12T05:37:08.852 INFO:tasks.workunit.client.0.smithi032.stdout:5/639: dread d2/df/d25/d1e/f40 [0,4194304] 0 2024-04-12T05:37:08.852 INFO:tasks.workunit.client.0.smithi032.stdout:3/645: dread d12/d2e/d3a/d2d/f4e [0,4194304] 0 2024-04-12T05:37:08.853 INFO:tasks.workunit.client.0.smithi032.stdout:5/640: readlink d2/df/d25/l9a 0 2024-04-12T05:37:08.854 INFO:tasks.workunit.client.1.smithi067.stdout:6/14: write f0 [1452323,59894] 0 2024-04-12T05:37:08.854 INFO:tasks.workunit.client.0.smithi032.stdout:6/731: chown d4/d21/d2e/cee 50 1 2024-04-12T05:37:08.855 INFO:tasks.workunit.client.1.smithi067.stdout:3/21: truncate f5 2304419 0 2024-04-12T05:37:08.856 INFO:tasks.workunit.client.1.smithi067.stdout:0/130: fdatasync dd/f13 0 2024-04-12T05:37:08.857 INFO:tasks.workunit.client.1.smithi067.stdout:7/30: mkdir d0/d2/d5/db 0 2024-04-12T05:37:08.857 INFO:tasks.workunit.client.0.smithi032.stdout:3/646: chown d12/d24/f2c 3 1 2024-04-12T05:37:08.857 INFO:tasks.workunit.client.0.smithi032.stdout:2/632: mknod dd/d19/d9d/d1c/d64/d92/cc9 0 2024-04-12T05:37:08.859 INFO:tasks.workunit.client.1.smithi067.stdout:1/67: write d4/f8 [1701278,76133] 0 2024-04-12T05:37:08.859 INFO:tasks.workunit.client.1.smithi067.stdout:3/22: mknod c7 0 2024-04-12T05:37:08.861 INFO:tasks.workunit.client.1.smithi067.stdout:2/34: write f4 [3411646,110287] 0 2024-04-12T05:37:08.861 INFO:tasks.workunit.client.1.smithi067.stdout:0/131: chown dd/f1c 909 1 2024-04-12T05:37:08.863 INFO:tasks.workunit.client.0.smithi032.stdout:3/647: dread - d12/d2e/d3a/d63/da6/dac/fd0 zero size 2024-04-12T05:37:08.864 INFO:tasks.workunit.client.1.smithi067.stdout:6/15: link f0 f3 0 2024-04-12T05:37:08.864 INFO:tasks.workunit.client.0.smithi032.stdout:1/535: dwrite d5/de/d17/d35/d73/f94 [0,4194304] 0 2024-04-12T05:37:08.866 INFO:tasks.workunit.client.1.smithi067.stdout:1/68: mkdir d4/d9/d12 0 2024-04-12T05:37:08.867 INFO:tasks.workunit.client.0.smithi032.stdout:1/536: chown d5/de/d17/f26 145 1 2024-04-12T05:37:08.870 INFO:tasks.workunit.client.0.smithi032.stdout:6/732: fsync d4/f6 0 2024-04-12T05:37:08.873 INFO:tasks.workunit.client.1.smithi067.stdout:5/54: dwrite d0/fb [0,4194304] 0 2024-04-12T05:37:08.873 INFO:tasks.workunit.client.1.smithi067.stdout:8/34: dwrite f4 [0,4194304] 0 2024-04-12T05:37:08.875 INFO:tasks.workunit.client.1.smithi067.stdout:6/16: creat f4 x:0 0 0 2024-04-12T05:37:08.875 INFO:tasks.workunit.client.0.smithi032.stdout:2/633: dread fa [0,4194304] 0 2024-04-12T05:37:08.877 INFO:tasks.workunit.client.0.smithi032.stdout:4/577: symlink d4/d87/d96/d32/dc5/lc7 0 2024-04-12T05:37:08.878 INFO:tasks.workunit.client.0.smithi032.stdout:6/733: chown d4/de0/de7/d79/c93 1177 1 2024-04-12T05:37:08.879 INFO:tasks.workunit.client.0.smithi032.stdout:7/628: creat d0/d13/d21/d30/fd3 x:0 0 0 2024-04-12T05:37:08.881 INFO:tasks.workunit.client.1.smithi067.stdout:0/132: dread f1 [0,4194304] 0 2024-04-12T05:37:08.882 INFO:tasks.workunit.client.0.smithi032.stdout:5/641: dwrite d2/d3c/d5b/f5e [8388608,4194304] 0 2024-04-12T05:37:08.884 INFO:tasks.workunit.client.0.smithi032.stdout:8/656: getdents d0/db/dd/d11 0 2024-04-12T05:37:08.885 INFO:tasks.workunit.client.1.smithi067.stdout:1/69: dread d4/f8 [0,4194304] 0 2024-04-12T05:37:08.886 INFO:tasks.workunit.client.1.smithi067.stdout:8/35: symlink d6/d8/lb 0 2024-04-12T05:37:08.886 INFO:tasks.workunit.client.1.smithi067.stdout:2/35: creat d5/da/fd x:0 0 0 2024-04-12T05:37:08.889 INFO:tasks.workunit.client.0.smithi032.stdout:3/648: truncate d12/d24/f2c 9791 0 2024-04-12T05:37:08.889 INFO:tasks.workunit.client.1.smithi067.stdout:2/36: dread - d5/da/fd zero size 2024-04-12T05:37:08.891 INFO:tasks.workunit.client.1.smithi067.stdout:8/36: rmdir d6 39 2024-04-12T05:37:08.894 INFO:tasks.workunit.client.1.smithi067.stdout:7/31: dwrite d0/d2/d5/fa [0,4194304] 0 2024-04-12T05:37:08.894 INFO:tasks.workunit.client.0.smithi032.stdout:1/537: dwrite d5/de/d17/d1b/f21 [0,4194304] 0 2024-04-12T05:37:08.895 INFO:tasks.workunit.client.1.smithi067.stdout:2/37: chown f4 32 1 2024-04-12T05:37:08.896 INFO:tasks.workunit.client.0.smithi032.stdout:3/649: readlink d12/daa/db3/l8b 0 2024-04-12T05:37:08.896 INFO:tasks.workunit.client.0.smithi032.stdout:7/629: rename d0/de/fb8 to d0/d10/fd4 0 2024-04-12T05:37:08.897 INFO:tasks.workunit.client.1.smithi067.stdout:1/70: creat d4/f13 x:0 0 0 2024-04-12T05:37:08.897 INFO:tasks.workunit.client.1.smithi067.stdout:2/38: fsync f4 0 2024-04-12T05:37:08.897 INFO:tasks.workunit.client.0.smithi032.stdout:7/630: chown d0/d34 309058198 1 2024-04-12T05:37:08.898 INFO:tasks.workunit.client.1.smithi067.stdout:0/133: dwrite dd/d12/f21 [0,4194304] 0 2024-04-12T05:37:08.898 INFO:tasks.workunit.client.1.smithi067.stdout:3/23: write f1 [1827944,66867] 0 2024-04-12T05:37:08.900 INFO:tasks.workunit.client.1.smithi067.stdout:3/24: stat c7 0 2024-04-12T05:37:08.900 INFO:tasks.workunit.client.1.smithi067.stdout:7/32: chown d0/f9 10220464 1 2024-04-12T05:37:08.901 INFO:tasks.workunit.client.0.smithi032.stdout:3/650: write d12/d24/f8f [1276351,43702] 0 2024-04-12T05:37:08.904 INFO:tasks.workunit.client.0.smithi032.stdout:5/642: mkdir d2/df/d31/da2/dc0/dcd 0 2024-04-12T05:37:08.906 INFO:tasks.workunit.client.0.smithi032.stdout:2/634: mknod dd/d19/d9d/d1c/d64/d32/d6d/da5/cca 0 2024-04-12T05:37:08.907 INFO:tasks.workunit.client.1.smithi067.stdout:5/55: getdents d0/da 0 2024-04-12T05:37:08.907 INFO:tasks.workunit.client.1.smithi067.stdout:2/39: stat d5/da/fc 0 2024-04-12T05:37:08.908 INFO:tasks.workunit.client.1.smithi067.stdout:1/71: truncate d4/f10 954409 0 2024-04-12T05:37:08.908 INFO:tasks.workunit.client.0.smithi032.stdout:8/657: rename d0/db/dd/d11/d1b/d22/d35/d44 to d0/db/dd/d11/d1b/d5f/deb 0 2024-04-12T05:37:08.908 INFO:tasks.workunit.client.1.smithi067.stdout:8/37: mkdir d6/dc 0 2024-04-12T05:37:08.909 INFO:tasks.workunit.client.0.smithi032.stdout:6/734: dwrite d4/d21/d2e/ff0 [0,4194304] 0 2024-04-12T05:37:08.909 INFO:tasks.workunit.client.0.smithi032.stdout:5/643: chown d2/df/da9/dc1 319 1 2024-04-12T05:37:08.911 INFO:tasks.workunit.client.0.smithi032.stdout:3/651: chown d12/d2e/d3a/d2d 69530 1 2024-04-12T05:37:08.912 INFO:tasks.workunit.client.1.smithi067.stdout:3/25: write f1 [1245426,127022] 0 2024-04-12T05:37:08.915 INFO:tasks.workunit.client.0.smithi032.stdout:7/631: dread d0/d13/fc5 [0,4194304] 0 2024-04-12T05:37:08.915 INFO:tasks.workunit.client.0.smithi032.stdout:5/644: chown d2/df/d25/d50/ca5 54175 1 2024-04-12T05:37:08.915 INFO:tasks.workunit.client.1.smithi067.stdout:1/72: truncate d4/d9/fd 171111 0 2024-04-12T05:37:08.916 INFO:tasks.workunit.client.0.smithi032.stdout:6/735: unlink d4/d21/fd7 0 2024-04-12T05:37:08.917 INFO:tasks.workunit.client.1.smithi067.stdout:3/26: readlink l6 0 2024-04-12T05:37:08.918 INFO:tasks.workunit.client.0.smithi032.stdout:3/652: write d12/d2e/d3a/d63/da6/dac/faf [382656,88405] 0 2024-04-12T05:37:08.918 INFO:tasks.workunit.client.1.smithi067.stdout:5/56: link d0/f1 d0/da/de/f14 0 2024-04-12T05:37:08.919 INFO:tasks.workunit.client.0.smithi032.stdout:1/538: link d5/d2c/d4f/c88 d5/de/d2a/cab 0 2024-04-12T05:37:08.921 INFO:tasks.workunit.client.0.smithi032.stdout:2/635: rename dd/d19/d9d/d1c/d64/d32 to dd/d19/d9d/d1c/d64/d32/dcb 22 2024-04-12T05:37:08.922 INFO:tasks.workunit.client.1.smithi067.stdout:8/38: rmdir d6/dc 0 2024-04-12T05:37:08.922 INFO:tasks.workunit.client.1.smithi067.stdout:2/40: dwrite f4 [4194304,4194304] 0 2024-04-12T05:37:08.923 INFO:tasks.workunit.client.0.smithi032.stdout:4/578: truncate d4/d87/d96/d2f/f70 1743653 0 2024-04-12T05:37:08.924 INFO:tasks.workunit.client.0.smithi032.stdout:7/632: unlink d0/d13/d21/lba 0 2024-04-12T05:37:08.928 INFO:tasks.workunit.client.1.smithi067.stdout:8/39: stat d6/d8 0 2024-04-12T05:37:08.929 INFO:tasks.workunit.client.1.smithi067.stdout:2/41: mkdir d5/da/de 0 2024-04-12T05:37:08.934 INFO:tasks.workunit.client.0.smithi032.stdout:7/633: chown d0/d34/f8d 7 1 2024-04-12T05:37:08.934 INFO:tasks.workunit.client.0.smithi032.stdout:8/658: dread d0/db/dd/d11/d1b/f6c [0,4194304] 0 2024-04-12T05:37:08.934 INFO:tasks.workunit.client.0.smithi032.stdout:4/579: dread - d4/d87/d96/d2f/d3b/fc4 zero size 2024-04-12T05:37:08.934 INFO:tasks.workunit.client.1.smithi067.stdout:1/73: dread d4/f6 [0,4194304] 0 2024-04-12T05:37:08.935 INFO:tasks.workunit.client.0.smithi032.stdout:1/539: fsync d5/de/d17/d35/d78/f79 0 2024-04-12T05:37:08.936 INFO:tasks.workunit.client.1.smithi067.stdout:7/33: dwrite d0/f8 [0,4194304] 0 2024-04-12T05:37:08.936 INFO:tasks.workunit.client.0.smithi032.stdout:6/736: dwrite d4/d14/d3c/fb5 [0,4194304] 0 2024-04-12T05:37:08.938 INFO:tasks.workunit.client.0.smithi032.stdout:4/580: fsync d4/d87/d96/f42 0 2024-04-12T05:37:08.939 INFO:tasks.workunit.client.0.smithi032.stdout:5/645: dwrite d2/df/da9/dc1/d78/f9e [4194304,4194304] 0 2024-04-12T05:37:08.939 INFO:tasks.workunit.client.0.smithi032.stdout:7/634: read d0/dd/f3b [3515245,37285] 0 2024-04-12T05:37:08.940 INFO:tasks.workunit.client.0.smithi032.stdout:3/653: getdents d12/daa 0 2024-04-12T05:37:08.943 INFO:tasks.workunit.client.0.smithi032.stdout:3/654: chown d12/d2e/f99 21389570 1 2024-04-12T05:37:08.943 INFO:tasks.workunit.client.0.smithi032.stdout:6/737: mkdir d4/de0/de7/d37/dfb 0 2024-04-12T05:37:08.944 INFO:tasks.workunit.client.0.smithi032.stdout:3/655: chown d12/d2e/dcb 112593756 1 2024-04-12T05:37:08.944 INFO:tasks.workunit.client.1.smithi067.stdout:2/42: creat d5/da/de/ff x:0 0 0 2024-04-12T05:37:08.945 INFO:tasks.workunit.client.0.smithi032.stdout:8/659: creat d0/db/dd/fec x:0 0 0 2024-04-12T05:37:08.947 INFO:tasks.workunit.client.0.smithi032.stdout:7/635: symlink d0/de/df/d20/ld5 0 2024-04-12T05:37:08.949 INFO:tasks.workunit.client.1.smithi067.stdout:2/43: write f4 [7058890,120955] 0 2024-04-12T05:37:08.949 INFO:tasks.workunit.client.1.smithi067.stdout:3/27: dread f5 [0,4194304] 0 2024-04-12T05:37:08.954 INFO:tasks.workunit.client.0.smithi032.stdout:3/656: mkdir d12/d24/dbb/dd4 0 2024-04-12T05:37:08.954 INFO:tasks.workunit.client.0.smithi032.stdout:5/646: dwrite d2/df/da9/dc1/d78/f94 [0,4194304] 0 2024-04-12T05:37:08.960 INFO:tasks.workunit.client.1.smithi067.stdout:8/40: dwrite f4 [0,4194304] 0 2024-04-12T05:37:08.963 INFO:tasks.workunit.client.1.smithi067.stdout:3/28: mknod c8 0 2024-04-12T05:37:08.964 INFO:tasks.workunit.client.1.smithi067.stdout:8/41: mknod d6/d8/cd 0 2024-04-12T05:37:08.965 INFO:tasks.workunit.client.1.smithi067.stdout:1/74: dwrite d4/f13 [0,4194304] 0 2024-04-12T05:37:08.967 INFO:tasks.workunit.client.0.smithi032.stdout:7/636: symlink d0/d34/dbe/ld6 0 2024-04-12T05:37:08.968 INFO:tasks.workunit.client.1.smithi067.stdout:5/57: write d0/d6/f12 [667744,91747] 0 2024-04-12T05:37:08.969 INFO:tasks.workunit.client.0.smithi032.stdout:8/660: mkdir d0/db/dd/d11/d1b/d22/d29/ded 0 2024-04-12T05:37:08.970 INFO:tasks.workunit.client.1.smithi067.stdout:5/58: read d0/f7 [884519,5723] 0 2024-04-12T05:37:08.970 INFO:tasks.workunit.client.1.smithi067.stdout:8/42: mkdir d6/de 0 2024-04-12T05:37:08.972 INFO:tasks.workunit.client.1.smithi067.stdout:3/29: creat f9 x:0 0 0 2024-04-12T05:37:08.972 INFO:tasks.workunit.client.1.smithi067.stdout:1/75: mkdir d4/d9/d14 0 2024-04-12T05:37:08.973 INFO:tasks.workunit.client.1.smithi067.stdout:2/44: link d5/l6 d5/l10 0 2024-04-12T05:37:08.973 INFO:tasks.workunit.client.0.smithi032.stdout:5/647: creat d2/dd/d5d/fce x:0 0 0 2024-04-12T05:37:08.974 INFO:tasks.workunit.client.1.smithi067.stdout:2/45: fsync d5/da/fd 0 2024-04-12T05:37:08.974 INFO:tasks.workunit.client.0.smithi032.stdout:6/738: link d4/de0/de7/d37/f5d d4/d21/d27/dad/ffc 0 2024-04-12T05:37:08.976 INFO:tasks.workunit.client.1.smithi067.stdout:5/59: creat d0/da/de/f15 x:0 0 0 2024-04-12T05:37:08.980 INFO:tasks.workunit.client.0.smithi032.stdout:5/648: creat d2/df/d31/fcf x:0 0 0 2024-04-12T05:37:08.980 INFO:tasks.workunit.client.1.smithi067.stdout:2/46: dread f3 [0,4194304] 0 2024-04-12T05:37:08.981 INFO:tasks.workunit.client.0.smithi032.stdout:3/657: mkdir d12/d2e/d3a/d63/dd5 0 2024-04-12T05:37:08.983 INFO:tasks.workunit.client.1.smithi067.stdout:2/47: write f3 [4752560,102569] 0 2024-04-12T05:37:08.984 INFO:tasks.workunit.client.0.smithi032.stdout:5/649: symlink d2/dd/d5d/ld0 0 2024-04-12T05:37:08.986 INFO:tasks.workunit.client.0.smithi032.stdout:7/637: rename d0/de/df/f1e to d0/dd/d51/fd7 0 2024-04-12T05:37:08.988 INFO:tasks.workunit.client.0.smithi032.stdout:6/739: getdents d4/de0/deb/dec 0 2024-04-12T05:37:08.990 INFO:tasks.workunit.client.1.smithi067.stdout:2/48: fdatasync d5/da/fc 0 2024-04-12T05:37:08.991 INFO:tasks.workunit.client.1.smithi067.stdout:7/34: truncate d0/f8 3444460 0 2024-04-12T05:37:08.992 INFO:tasks.workunit.client.0.smithi032.stdout:2/636: dread dd/d19/d9d/d1c/f74 [0,4194304] 0 2024-04-12T05:37:08.992 INFO:tasks.workunit.client.0.smithi032.stdout:5/650: read d2/df/d25/d1e/f1f [872880,48160] 0 2024-04-12T05:37:08.994 INFO:tasks.workunit.client.0.smithi032.stdout:7/638: write d0/d34/dbe/f5c [134126,5759] 0 2024-04-12T05:37:08.995 INFO:tasks.workunit.client.0.smithi032.stdout:6/740: mknod d4/de0/deb/cfd 0 2024-04-12T05:37:08.997 INFO:tasks.workunit.client.1.smithi067.stdout:7/35: unlink d0/d2/d5/fa 0 2024-04-12T05:37:08.997 INFO:tasks.workunit.client.1.smithi067.stdout:8/43: write f5 [726755,34649] 0 2024-04-12T05:37:08.998 INFO:tasks.workunit.client.1.smithi067.stdout:7/36: read - d0/d2/f3 zero size 2024-04-12T05:37:08.998 INFO:tasks.workunit.client.1.smithi067.stdout:8/44: chown d6/d8/f9 384 1 2024-04-12T05:37:08.999 INFO:tasks.workunit.client.1.smithi067.stdout:8/45: stat d6 0 2024-04-12T05:37:08.999 INFO:tasks.workunit.client.1.smithi067.stdout:1/76: write d4/fe [584613,23176] 0 2024-04-12T05:37:09.001 INFO:tasks.workunit.client.1.smithi067.stdout:7/37: unlink d0/d2/f3 0 2024-04-12T05:37:09.002 INFO:tasks.workunit.client.0.smithi032.stdout:2/637: write dd/d19/f5f [856008,81512] 0 2024-04-12T05:37:09.002 INFO:tasks.workunit.client.0.smithi032.stdout:3/658: dwrite d12/d2f/d89/d9d/fb6 [0,4194304] 0 2024-04-12T05:37:09.002 INFO:tasks.workunit.client.0.smithi032.stdout:6/741: write d4/f1b [1020261,32078] 0 2024-04-12T05:37:09.002 INFO:tasks.workunit.client.1.smithi067.stdout:3/30: truncate f1 1096849 0 2024-04-12T05:37:09.004 INFO:tasks.workunit.client.1.smithi067.stdout:3/31: truncate f9 585350 0 2024-04-12T05:37:09.004 INFO:tasks.workunit.client.1.smithi067.stdout:3/32: write f9 [1549543,89848] 0 2024-04-12T05:37:09.006 INFO:tasks.workunit.client.0.smithi032.stdout:8/661: dread d0/db/dd/d11/f1f [4194304,4194304] 0 2024-04-12T05:37:09.011 INFO:tasks.workunit.client.1.smithi067.stdout:8/46: mkdir d6/de/df 0 2024-04-12T05:37:09.011 INFO:tasks.workunit.client.0.smithi032.stdout:4/581: dwrite d4/d13/d19/f30 [0,4194304] 0 2024-04-12T05:37:09.012 INFO:tasks.workunit.client.0.smithi032.stdout:4/582: fdatasync d4/d13/d19/fa1 0 2024-04-12T05:37:09.013 INFO:tasks.workunit.client.1.smithi067.stdout:1/77: rename d4/f13 to d4/f15 0 2024-04-12T05:37:09.014 INFO:tasks.workunit.client.0.smithi032.stdout:8/662: dread d0/db/f4e [0,4194304] 0 2024-04-12T05:37:09.014 INFO:tasks.workunit.client.1.smithi067.stdout:5/60: dwrite d0/f1 [4194304,4194304] 0 2024-04-12T05:37:09.016 INFO:tasks.workunit.client.1.smithi067.stdout:5/61: write d0/f1 [391778,127376] 0 2024-04-12T05:37:09.019 INFO:tasks.workunit.client.0.smithi032.stdout:6/742: dwrite d4/d21/d2e/d32/fba [0,4194304] 0 2024-04-12T05:37:09.020 INFO:tasks.workunit.client.0.smithi032.stdout:6/743: chown d4/f1b 5803075 1 2024-04-12T05:37:09.026 INFO:tasks.workunit.client.1.smithi067.stdout:1/78: creat d4/d9/d12/f16 x:0 0 0 2024-04-12T05:37:09.027 INFO:tasks.workunit.client.1.smithi067.stdout:1/79: write d4/f10 [7506,127223] 0 2024-04-12T05:37:09.028 INFO:tasks.workunit.client.1.smithi067.stdout:1/80: chown d4/f7 11407492 1 2024-04-12T05:37:09.032 INFO:tasks.workunit.client.1.smithi067.stdout:3/33: write f5 [985219,51421] 0 2024-04-12T05:37:09.036 INFO:tasks.workunit.client.1.smithi067.stdout:1/81: link d4/d9/d12/f16 d4/d9/f17 0 2024-04-12T05:37:09.037 INFO:tasks.workunit.client.1.smithi067.stdout:8/47: write d6/f7 [3905536,48169] 0 2024-04-12T05:37:09.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.068+0000 7fe4352a9700 1 -- 172.21.15.32:0/2551353764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4300fbcd0 msgr2=0x7fe4300fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:09.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.068+0000 7fe4352a9700 1 --2- 172.21.15.32:0/2551353764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4300fbcd0 0x7fe4300fc0e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe420004940 tx=0x7fe420009e30 comp rx=0 tx=0).stop 2024-04-12T05:37:09.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.068+0000 7fe4352a9700 1 -- 172.21.15.32:0/2551353764 shutdown_connections 2024-04-12T05:37:09.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.068+0000 7fe4352a9700 1 --2- 172.21.15.32:0/2551353764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4300fcf80 0x7fe4300fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:09.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.068+0000 7fe4352a9700 1 --2- 172.21.15.32:0/2551353764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4300fbcd0 0x7fe4300fc0e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:09.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.068+0000 7fe4352a9700 1 -- 172.21.15.32:0/2551353764 >> 172.21.15.32:0/2551353764 conn(0x7fe4300f7450 msgr2=0x7fe4300f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:09.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.069+0000 7fe4352a9700 1 -- 172.21.15.32:0/2551353764 shutdown_connections 2024-04-12T05:37:09.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.069+0000 7fe4352a9700 1 -- 172.21.15.32:0/2551353764 wait complete. 2024-04-12T05:37:09.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.069+0000 7fe4352a9700 1 Processor -- start 2024-04-12T05:37:09.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.069+0000 7fe4352a9700 1 -- start start 2024-04-12T05:37:09.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.070+0000 7fe4352a9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4300fbcd0 0x7fe43010c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:37:09.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.070+0000 7fe4352a9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4300fcf80 0x7fe43010cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:37:09.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.070+0000 7fe4352a9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe43010d1e0 con 0x7fe4300fbcd0 2024-04-12T05:37:09.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.070+0000 7fe4352a9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe43010d350 con 0x7fe4300fcf80 2024-04-12T05:37:09.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.070+0000 7fe42effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4300fbcd0 0x7fe43010c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:09.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.070+0000 7fe42effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4300fbcd0 0x7fe43010c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56550/0 (socket says 172.21.15.32:56550) 2024-04-12T05:37:09.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.070+0000 7fe42effd700 1 -- 172.21.15.32:0/2846387143 learned_addr learned my addr 172.21.15.32:0/2846387143 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:37:09.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.070+0000 7fe42e7fc700 1 --2- 172.21.15.32:0/2846387143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4300fcf80 0x7fe43010cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:09.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.070+0000 7fe42effd700 1 -- 172.21.15.32:0/2846387143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4300fcf80 msgr2=0x7fe43010cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:09.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.070+0000 7fe42effd700 1 --2- 172.21.15.32:0/2846387143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4300fcf80 0x7fe43010cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:09.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.070+0000 7fe42effd700 1 -- 172.21.15.32:0/2846387143 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe420005040 con 0x7fe4300fbcd0 2024-04-12T05:37:09.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.070+0000 7fe42effd700 1 --2- 172.21.15.32:0/2846387143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4300fbcd0 0x7fe43010c6a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe4200137b0 tx=0x7fe420009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:09.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.070+0000 7fe4177fe700 1 -- 172.21.15.32:0/2846387143 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe42000e070 con 0x7fe4300fbcd0 2024-04-12T05:37:09.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.070+0000 7fe4177fe700 1 -- 172.21.15.32:0/2846387143 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe42001c400 con 0x7fe4300fbcd0 2024-04-12T05:37:09.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.070+0000 7fe4177fe700 1 -- 172.21.15.32:0/2846387143 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe42000b650 con 0x7fe4300fbcd0 2024-04-12T05:37:09.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.071+0000 7fe4352a9700 1 -- 172.21.15.32:0/2846387143 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe430066270 con 0x7fe4300fbcd0 2024-04-12T05:37:09.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.071+0000 7fe4352a9700 1 -- 172.21.15.32:0/2846387143 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4300667d0 con 0x7fe4300fbcd0 2024-04-12T05:37:09.071 INFO:tasks.workunit.client.1.smithi067.stdout:9/10: sync 2024-04-12T05:37:09.071 INFO:tasks.workunit.client.1.smithi067.stdout:9/11: dread - no filename 2024-04-12T05:37:09.071 INFO:tasks.workunit.client.1.smithi067.stdout:9/12: dread - no filename 2024-04-12T05:37:09.071 INFO:tasks.workunit.client.1.smithi067.stdout:9/13: truncate - no filename 2024-04-12T05:37:09.071 INFO:tasks.workunit.client.1.smithi067.stdout:9/14: truncate - no filename 2024-04-12T05:37:09.071 INFO:tasks.workunit.client.1.smithi067.stdout:5/62: fdatasync d0/f1 0 2024-04-12T05:37:09.071 INFO:tasks.workunit.client.1.smithi067.stdout:7/38: dwrite d0/f9 [0,4194304] 0 2024-04-12T05:37:09.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.072+0000 7fe4352a9700 1 -- 172.21.15.32:0/2846387143 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe43004c710 con 0x7fe4300fbcd0 2024-04-12T05:37:09.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.072+0000 7fe4177fe700 1 -- 172.21.15.32:0/2846387143 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fe42001c9e0 con 0x7fe4300fbcd0 2024-04-12T05:37:09.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.073+0000 7fe4177fe700 1 --2- 172.21.15.32:0/2846387143 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fe4180692a0 0x7fe41806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:37:09.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.073+0000 7fe4177fe700 1 -- 172.21.15.32:0/2846387143 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fe420088760 con 0x7fe4300fbcd0 2024-04-12T05:37:09.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.073+0000 7fe42e7fc700 1 --2- 172.21.15.32:0/2846387143 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fe4180692a0 0x7fe41806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:09.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.074+0000 7fe42e7fc700 1 --2- 172.21.15.32:0/2846387143 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fe4180692a0 0x7fe41806b750 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe424009b80 tx=0x7fe42400b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:09.073 INFO:tasks.workunit.client.1.smithi067.stdout:9/15: stat c0 0 2024-04-12T05:37:09.073 INFO:tasks.workunit.client.1.smithi067.stdout:1/82: write d4/f7 [4224478,119123] 0 2024-04-12T05:37:09.073 INFO:tasks.workunit.client.1.smithi067.stdout:5/63: dread - d0/fc zero size 2024-04-12T05:37:09.074 INFO:tasks.workunit.client.0.smithi032.stdout:7/639: unlink d0/dd/f98 0 2024-04-12T05:37:09.074 INFO:tasks.workunit.client.1.smithi067.stdout:3/34: fsync f1 0 2024-04-12T05:37:09.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.079+0000 7fe4177fe700 1 -- 172.21.15.32:0/2846387143 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fe420058570 con 0x7fe4300fbcd0 2024-04-12T05:37:09.076 INFO:tasks.workunit.client.1.smithi067.stdout:1/83: stat d4/d9/d12 0 2024-04-12T05:37:09.077 INFO:tasks.workunit.client.1.smithi067.stdout:9/16: mkdir d1 0 2024-04-12T05:37:09.078 INFO:tasks.workunit.client.1.smithi067.stdout:8/48: dwrite f4 [0,4194304] 0 2024-04-12T05:37:09.078 INFO:tasks.workunit.client.1.smithi067.stdout:3/35: symlink la 0 2024-04-12T05:37:09.080 INFO:tasks.workunit.client.1.smithi067.stdout:3/36: readlink l4 0 2024-04-12T05:37:09.080 INFO:tasks.workunit.client.1.smithi067.stdout:7/39: read d0/f9 [1474763,119369] 0 2024-04-12T05:37:09.081 INFO:tasks.workunit.client.0.smithi032.stdout:2/638: chown dd/d19/d9d/d1c/d64/d32/d3a/fa8 194 1 2024-04-12T05:37:09.081 INFO:tasks.workunit.client.0.smithi032.stdout:3/659: symlink d12/d2f/d89/ld6 0 2024-04-12T05:37:09.081 INFO:tasks.workunit.client.0.smithi032.stdout:2/639: write dd/d19/f24 [5027456,10864] 0 2024-04-12T05:37:09.081 INFO:tasks.workunit.client.1.smithi067.stdout:1/84: symlink d4/d9/l18 0 2024-04-12T05:37:09.084 INFO:tasks.workunit.client.0.smithi032.stdout:4/583: chown d4/d13/d19/c24 0 1 2024-04-12T05:37:09.084 INFO:tasks.workunit.client.1.smithi067.stdout:9/17: creat d1/f2 x:0 0 0 2024-04-12T05:37:09.085 INFO:tasks.workunit.client.1.smithi067.stdout:8/49: unlink f1 0 2024-04-12T05:37:09.086 INFO:tasks.workunit.client.1.smithi067.stdout:3/37: symlink lb 0 2024-04-12T05:37:09.086 INFO:tasks.workunit.client.0.smithi032.stdout:4/584: chown d4/d13/d75/c90 45 1 2024-04-12T05:37:09.087 INFO:tasks.workunit.client.1.smithi067.stdout:5/64: dwrite d0/fb [0,4194304] 0 2024-04-12T05:37:09.088 INFO:tasks.workunit.client.1.smithi067.stdout:3/38: write f5 [1799857,110151] 0 2024-04-12T05:37:09.088 INFO:tasks.workunit.client.0.smithi032.stdout:7/640: write d0/d10/fd4 [1533066,81110] 0 2024-04-12T05:37:09.089 INFO:tasks.workunit.client.1.smithi067.stdout:3/39: stat la 0 2024-04-12T05:37:09.089 INFO:tasks.workunit.client.1.smithi067.stdout:5/65: write d0/fb [2175576,6023] 0 2024-04-12T05:37:09.090 INFO:tasks.workunit.client.1.smithi067.stdout:5/66: stat d0/d6 0 2024-04-12T05:37:09.091 INFO:tasks.workunit.client.0.smithi032.stdout:7/641: dread - d0/d34/dbe/fa6 zero size 2024-04-12T05:37:09.093 INFO:tasks.workunit.client.0.smithi032.stdout:7/642: chown d0/d1c/d71 16 1 2024-04-12T05:37:09.096 INFO:tasks.workunit.client.1.smithi067.stdout:1/85: mkdir d4/d9/d12/d19 0 2024-04-12T05:37:09.098 INFO:tasks.workunit.client.0.smithi032.stdout:4/585: dwrite d4/d13/d44/fa0 [4194304,4194304] 0 2024-04-12T05:37:09.100 INFO:tasks.workunit.client.1.smithi067.stdout:8/50: symlink d6/l10 0 2024-04-12T05:37:09.101 INFO:tasks.workunit.client.1.smithi067.stdout:3/40: mkdir dc 0 2024-04-12T05:37:09.101 INFO:tasks.workunit.client.1.smithi067.stdout:3/41: stat f1 0 2024-04-12T05:37:09.103 INFO:tasks.workunit.client.1.smithi067.stdout:7/40: dwrite d0/f9 [0,4194304] 0 2024-04-12T05:37:09.104 INFO:tasks.workunit.client.0.smithi032.stdout:8/663: unlink d0/db/dd/d11/d1b/d22/d51/lda 0 2024-04-12T05:37:09.105 INFO:tasks.workunit.client.0.smithi032.stdout:5/651: creat d2/df/d25/fd1 x:0 0 0 2024-04-12T05:37:09.105 INFO:tasks.workunit.client.0.smithi032.stdout:7/643: rmdir d0/de/d47 39 2024-04-12T05:37:09.106 INFO:tasks.workunit.client.0.smithi032.stdout:4/586: symlink d4/d87/d96/d2f/lc8 0 2024-04-12T05:37:09.106 INFO:tasks.workunit.client.0.smithi032.stdout:6/744: link d4/d14/d25/l35 d4/d21/d2e/dca/lfe 0 2024-04-12T05:37:09.109 INFO:tasks.workunit.client.0.smithi032.stdout:7/644: mkdir d0/d1c/d4c/dd8 0 2024-04-12T05:37:09.109 INFO:tasks.workunit.client.0.smithi032.stdout:8/664: chown d0/db/ca7 21 1 2024-04-12T05:37:09.110 INFO:tasks.workunit.client.0.smithi032.stdout:5/652: mknod d2/d3c/d5b/cd2 0 2024-04-12T05:37:09.112 INFO:tasks.workunit.client.1.smithi067.stdout:8/51: creat d6/de/f11 x:0 0 0 2024-04-12T05:37:09.113 INFO:tasks.workunit.client.0.smithi032.stdout:5/653: readlink d2/dd/d6f/lb6 0 2024-04-12T05:37:09.114 INFO:tasks.workunit.client.1.smithi067.stdout:9/18: fsync d1/f2 0 2024-04-12T05:37:09.114 INFO:tasks.workunit.client.1.smithi067.stdout:1/86: mkdir d4/d9/d12/d19/d1a 0 2024-04-12T05:37:09.115 INFO:tasks.workunit.client.1.smithi067.stdout:3/42: mknod dc/cd 0 2024-04-12T05:37:09.115 INFO:tasks.workunit.client.0.smithi032.stdout:6/745: mknod d4/de0/deb/dec/cff 0 2024-04-12T05:37:09.116 INFO:tasks.workunit.client.1.smithi067.stdout:1/87: chown d4/f8 0 1 2024-04-12T05:37:09.116 INFO:tasks.workunit.client.1.smithi067.stdout:3/43: chown f5 0 1 2024-04-12T05:37:09.117 INFO:tasks.workunit.client.0.smithi032.stdout:8/665: mknod d0/db/dd/d11/cee 0 2024-04-12T05:37:09.117 INFO:tasks.workunit.client.1.smithi067.stdout:7/41: mkdir d0/d2/d5/dc 0 2024-04-12T05:37:09.117 INFO:tasks.workunit.client.1.smithi067.stdout:3/44: truncate f5 2382572 0 2024-04-12T05:37:09.118 INFO:tasks.workunit.client.1.smithi067.stdout:7/42: chown d0/d2 7222 1 2024-04-12T05:37:09.118 INFO:tasks.workunit.client.1.smithi067.stdout:9/19: mkdir d1/d3 0 2024-04-12T05:37:09.119 INFO:tasks.workunit.client.0.smithi032.stdout:5/654: creat d2/df/da9/dc1/fd3 x:0 0 0 2024-04-12T05:37:09.119 INFO:tasks.workunit.client.0.smithi032.stdout:4/587: dwrite d4/d13/d19/f30 [4194304,4194304] 0 2024-04-12T05:37:09.120 INFO:tasks.workunit.client.0.smithi032.stdout:8/666: write d0/f7 [1857103,113683] 0 2024-04-12T05:37:09.120 INFO:tasks.workunit.client.1.smithi067.stdout:1/88: unlink c1 0 2024-04-12T05:37:09.120 INFO:tasks.workunit.client.1.smithi067.stdout:5/67: link d0/f2 d0/da/de/f16 0 2024-04-12T05:37:09.123 INFO:tasks.workunit.client.1.smithi067.stdout:1/89: dread d4/f8 [0,4194304] 0 2024-04-12T05:37:09.124 INFO:tasks.workunit.client.1.smithi067.stdout:7/43: mkdir d0/d2/dd 0 2024-04-12T05:37:09.124 INFO:tasks.workunit.client.0.smithi032.stdout:5/655: truncate d2/df/d31/fc5 826658 0 2024-04-12T05:37:09.126 INFO:tasks.workunit.client.0.smithi032.stdout:2/640: write dd/d19/d9d/f73 [820651,64683] 0 2024-04-12T05:37:09.128 INFO:tasks.workunit.client.0.smithi032.stdout:2/641: fdatasync f9 0 2024-04-12T05:37:09.132 INFO:tasks.workunit.client.1.smithi067.stdout:5/68: read d0/da/de/f16 [647356,1213] 0 2024-04-12T05:37:09.133 INFO:tasks.workunit.client.1.smithi067.stdout:1/90: rename d4/fc to d4/d9/d12/f1b 0 2024-04-12T05:37:09.136 INFO:tasks.workunit.client.1.smithi067.stdout:5/69: dread d0/f1 [4194304,4194304] 0 2024-04-12T05:37:09.137 INFO:tasks.workunit.client.1.smithi067.stdout:3/45: getdents dc 0 2024-04-12T05:37:09.140 INFO:tasks.workunit.client.1.smithi067.stdout:1/91: creat d4/d9/f1c x:0 0 0 2024-04-12T05:37:09.140 INFO:tasks.workunit.client.0.smithi032.stdout:5/656: truncate d2/dd/f83 982514 0 2024-04-12T05:37:09.140 INFO:tasks.workunit.client.0.smithi032.stdout:8/667: dwrite d0/db/dd/d11/d1b/d22/fb4 [0,4194304] 0 2024-04-12T05:37:09.143 INFO:tasks.workunit.client.0.smithi032.stdout:7/645: dwrite d0/de/df/d20/f68 [0,4194304] 0 2024-04-12T05:37:09.144 INFO:tasks.workunit.client.0.smithi032.stdout:2/642: creat dd/d19/d9d/d42/da7/dc5/fcc x:0 0 0 2024-04-12T05:37:09.145 INFO:tasks.workunit.client.1.smithi067.stdout:3/46: rmdir dc 39 2024-04-12T05:37:09.145 INFO:tasks.workunit.client.0.smithi032.stdout:8/668: chown d0/db/d38/l47 81 1 2024-04-12T05:37:09.145 INFO:tasks.workunit.client.1.smithi067.stdout:8/52: truncate f5 405412 0 2024-04-12T05:37:09.146 INFO:tasks.workunit.client.1.smithi067.stdout:8/53: chown d6/de 142190 1 2024-04-12T05:37:09.150 INFO:tasks.workunit.client.1.smithi067.stdout:5/70: unlink d0/d6/d8/l11 0 2024-04-12T05:37:09.151 INFO:tasks.workunit.client.0.smithi032.stdout:2/643: write dd/d19/d9d/d1c/d64/d32/d6d/f7f [175044,114041] 0 2024-04-12T05:37:09.152 INFO:tasks.workunit.client.0.smithi032.stdout:7/646: link d0/dd/f2a d0/d1c/d4c/fd9 0 2024-04-12T05:37:09.154 INFO:tasks.workunit.client.0.smithi032.stdout:4/588: dread d4/d13/f71 [0,4194304] 0 2024-04-12T05:37:09.154 INFO:tasks.workunit.client.1.smithi067.stdout:3/47: chown dc/cd 1588754 1 2024-04-12T05:37:09.154 INFO:tasks.workunit.client.0.smithi032.stdout:4/589: fsync d4/f18 0 2024-04-12T05:37:09.158 INFO:tasks.workunit.client.0.smithi032.stdout:5/657: write d2/d3c/d5b/d75/f84 [1128517,70103] 0 2024-04-12T05:37:09.159 INFO:tasks.workunit.client.0.smithi032.stdout:6/746: dread d4/d14/d25/fe8 [0,4194304] 0 2024-04-12T05:37:09.159 INFO:tasks.workunit.client.1.smithi067.stdout:8/54: dwrite f4 [4194304,4194304] 0 2024-04-12T05:37:09.160 INFO:tasks.workunit.client.0.smithi032.stdout:5/658: stat d2/df/d25/d50/fab 0 2024-04-12T05:37:09.161 INFO:tasks.workunit.client.1.smithi067.stdout:7/44: dwrite d0/d2/f7 [0,4194304] 0 2024-04-12T05:37:09.161 INFO:tasks.workunit.client.0.smithi032.stdout:7/647: creat d0/de/d47/fda x:0 0 0 2024-04-12T05:37:09.163 INFO:tasks.workunit.client.0.smithi032.stdout:5/659: write d2/dd/d5d/faf [134421,62115] 0 2024-04-12T05:37:09.165 INFO:tasks.workunit.client.0.smithi032.stdout:5/660: fdatasync d2/dd/f3a 0 2024-04-12T05:37:09.167 INFO:tasks.workunit.client.0.smithi032.stdout:5/661: stat d2/f24 0 2024-04-12T05:37:09.168 INFO:tasks.workunit.client.1.smithi067.stdout:5/71: symlink d0/da/de/l17 0 2024-04-12T05:37:09.168 INFO:tasks.workunit.client.1.smithi067.stdout:1/92: link d4/d9/fa d4/f1d 0 2024-04-12T05:37:09.168 INFO:tasks.workunit.client.1.smithi067.stdout:3/48: dread f9 [0,4194304] 0 2024-04-12T05:37:09.168 INFO:tasks.workunit.client.0.smithi032.stdout:8/669: dread d0/db/dd/f17 [0,4194304] 0 2024-04-12T05:37:09.169 INFO:tasks.workunit.client.0.smithi032.stdout:4/590: mkdir d4/d13/dc9 0 2024-04-12T05:37:09.169 INFO:tasks.workunit.client.1.smithi067.stdout:3/49: unlink l4 0 2024-04-12T05:37:09.170 INFO:tasks.workunit.client.0.smithi032.stdout:6/747: mknod d4/de0/de7/dd9/c100 0 2024-04-12T05:37:09.171 INFO:tasks.workunit.client.1.smithi067.stdout:1/93: creat d4/d9/d12/d19/f1e x:0 0 0 2024-04-12T05:37:09.171 INFO:tasks.workunit.client.0.smithi032.stdout:7/648: creat d0/de/df/d29/fdb x:0 0 0 2024-04-12T05:37:09.172 INFO:tasks.workunit.client.0.smithi032.stdout:5/662: mkdir d2/d3c/d5b/dd4 0 2024-04-12T05:37:09.172 INFO:tasks.workunit.client.0.smithi032.stdout:4/591: fdatasync d4/d87/d96/d2f/d9a/fa7 0 2024-04-12T05:37:09.172 INFO:tasks.workunit.client.0.smithi032.stdout:8/670: fsync d0/db/dd/f27 0 2024-04-12T05:37:09.177 INFO:tasks.workunit.client.1.smithi067.stdout:7/45: link d0/f1 d0/d2/d5/fe 0 2024-04-12T05:37:09.177 INFO:tasks.workunit.client.1.smithi067.stdout:1/94: write d4/d9/d12/f1b [1646428,122250] 0 2024-04-12T05:37:09.178 INFO:tasks.workunit.client.0.smithi032.stdout:2/644: truncate fa 2444531 0 2024-04-12T05:37:09.179 INFO:tasks.workunit.client.1.smithi067.stdout:5/72: dwrite d0/da/de/f14 [4194304,4194304] 0 2024-04-12T05:37:09.180 INFO:tasks.workunit.client.0.smithi032.stdout:4/592: creat d4/d13/d44/fca x:0 0 0 2024-04-12T05:37:09.184 INFO:tasks.workunit.client.1.smithi067.stdout:5/73: chown d0/d6/d8/l9 40 1 2024-04-12T05:37:09.185 INFO:tasks.workunit.client.1.smithi067.stdout:3/50: dwrite f5 [0,4194304] 0 2024-04-12T05:37:09.188 INFO:tasks.workunit.client.1.smithi067.stdout:3/51: write f5 [2504523,114061] 0 2024-04-12T05:37:09.192 INFO:tasks.workunit.client.0.smithi032.stdout:8/671: dwrite d0/db/dd/d11/d1b/d22/d54/da8/fc7 [0,4194304] 0 2024-04-12T05:37:09.193 INFO:tasks.workunit.client.0.smithi032.stdout:9/509: sync 2024-04-12T05:37:09.195 INFO:tasks.workunit.client.1.smithi067.stdout:3/52: creat dc/fe x:0 0 0 2024-04-12T05:37:09.195 INFO:tasks.workunit.client.1.smithi067.stdout:1/95: dwrite d4/d9/d12/d19/f1e [0,4194304] 0 2024-04-12T05:37:09.198 INFO:tasks.workunit.client.0.smithi032.stdout:9/510: rmdir d9/d15/d1a/d24/d71 39 2024-04-12T05:37:09.200 INFO:tasks.workunit.client.0.smithi032.stdout:9/511: stat d9/d15/d1a/d31/f98 0 2024-04-12T05:37:09.200 INFO:tasks.workunit.client.0.smithi032.stdout:6/748: dread d4/d74/f91 [0,4194304] 0 2024-04-12T05:37:09.202 INFO:tasks.workunit.client.1.smithi067.stdout:3/53: chown f1 63 1 2024-04-12T05:37:09.203 INFO:tasks.workunit.client.0.smithi032.stdout:9/512: readlink d9/d15/d1a/d1d/d5a/d6b/l85 0 2024-04-12T05:37:09.204 INFO:tasks.workunit.client.1.smithi067.stdout:1/96: getdents d4/d9/d12/d19/d1a 0 2024-04-12T05:37:09.205 INFO:tasks.workunit.client.1.smithi067.stdout:3/54: chown lb 13293 1 2024-04-12T05:37:09.208 INFO:tasks.workunit.client.0.smithi032.stdout:7/649: write d0/d1c/d4c/fd9 [5117135,40951] 0 2024-04-12T05:37:09.208 INFO:tasks.workunit.client.0.smithi032.stdout:4/593: dwrite d4/d87/d96/d2f/d3b/f5d [8388608,4194304] 0 2024-04-12T05:37:09.209 INFO:tasks.workunit.client.1.smithi067.stdout:1/97: fdatasync d4/d9/d12/f16 0 2024-04-12T05:37:09.213 INFO:tasks.workunit.client.0.smithi032.stdout:7/650: dread - d0/d13/d21/d81/f9c zero size 2024-04-12T05:37:09.216 INFO:tasks.workunit.client.1.smithi067.stdout:3/55: dwrite f5 [0,4194304] 0 2024-04-12T05:37:09.218 INFO:tasks.workunit.client.1.smithi067.stdout:1/98: mknod d4/d9/d12/d19/c1f 0 2024-04-12T05:37:09.218 INFO:tasks.workunit.client.1.smithi067.stdout:1/99: readlink d4/d9/l11 0 2024-04-12T05:37:09.219 INFO:tasks.workunit.client.1.smithi067.stdout:1/100: dread - d4/d9/f1c zero size 2024-04-12T05:37:09.219 INFO:tasks.workunit.client.0.smithi032.stdout:9/513: rename d9/d32/d86/f97 to d9/d15/d1a/d1d/faf 0 2024-04-12T05:37:09.219 INFO:tasks.workunit.client.0.smithi032.stdout:4/594: fdatasync d4/d13/f71 0 2024-04-12T05:37:09.225 INFO:tasks.workunit.client.1.smithi067.stdout:1/101: mkdir d4/d20 0 2024-04-12T05:37:09.229 INFO:tasks.workunit.client.0.smithi032.stdout:9/514: rename d9/d15/d1a/d24/d27/l7a to d9/d15/lb0 0 2024-04-12T05:37:09.230 INFO:tasks.workunit.client.0.smithi032.stdout:2/645: truncate dd/d19/d9d/d1c/d64/d32/f35 1945046 0 2024-04-12T05:37:09.230 INFO:tasks.workunit.client.1.smithi067.stdout:5/74: dwrite d0/fc [0,4194304] 0 2024-04-12T05:37:09.230 INFO:tasks.workunit.client.0.smithi032.stdout:2/646: chown c7 1104751 1 2024-04-12T05:37:09.231 INFO:tasks.workunit.client.0.smithi032.stdout:6/749: dread d4/d21/d27/dad/d55/f72 [0,4194304] 0 2024-04-12T05:37:09.232 INFO:tasks.workunit.client.1.smithi067.stdout:5/75: dread d0/f7 [0,4194304] 0 2024-04-12T05:37:09.232 INFO:tasks.workunit.client.1.smithi067.stdout:1/102: fdatasync d4/d9/fb 0 2024-04-12T05:37:09.234 INFO:tasks.workunit.client.0.smithi032.stdout:6/750: read d4/d21/d27/dad/d55/f70 [486766,117251] 0 2024-04-12T05:37:09.234 INFO:tasks.workunit.client.1.smithi067.stdout:5/76: write d0/fb [2924300,89897] 0 2024-04-12T05:37:09.236 INFO:tasks.workunit.client.0.smithi032.stdout:2/647: mknod dd/d19/d9d/d1c/d64/d32/d3a/d8d/ccd 0 2024-04-12T05:37:09.239 INFO:tasks.workunit.client.1.smithi067.stdout:5/77: creat d0/da/de/f18 x:0 0 0 2024-04-12T05:37:09.240 INFO:tasks.workunit.client.1.smithi067.stdout:5/78: truncate d0/da/de/f18 740835 0 2024-04-12T05:37:09.240 INFO:tasks.workunit.client.0.smithi032.stdout:6/751: mkdir d4/de0/de7/d79/dc6/d101 0 2024-04-12T05:37:09.242 INFO:tasks.workunit.client.1.smithi067.stdout:1/103: rmdir d4/d9/d12/d19/d1a 0 2024-04-12T05:37:09.242 INFO:tasks.workunit.client.1.smithi067.stdout:1/104: stat d4/d9/d12/f16 0 2024-04-12T05:37:09.243 INFO:tasks.workunit.client.1.smithi067.stdout:5/79: creat d0/d6/d8/f19 x:0 0 0 2024-04-12T05:37:09.258 INFO:tasks.workunit.client.1.smithi067.stdout:5/80: dwrite d0/da/de/f15 [0,4194304] 0 2024-04-12T05:37:09.259 INFO:tasks.workunit.client.1.smithi067.stdout:3/56: dwrite f9 [0,4194304] 0 2024-04-12T05:37:09.260 INFO:tasks.workunit.client.1.smithi067.stdout:5/81: symlink d0/da/de/l1a 0 2024-04-12T05:37:09.263 INFO:tasks.workunit.client.1.smithi067.stdout:5/82: creat d0/da/f1b x:0 0 0 2024-04-12T05:37:09.265 INFO:tasks.workunit.client.1.smithi067.stdout:3/57: mkdir dc/df 0 2024-04-12T05:37:09.271 INFO:tasks.workunit.client.1.smithi067.stdout:5/83: dwrite d0/fc [0,4194304] 0 2024-04-12T05:37:09.276 INFO:tasks.workunit.client.1.smithi067.stdout:5/84: link d0/d6/f12 d0/f1c 0 2024-04-12T05:37:09.278 INFO:tasks.workunit.client.1.smithi067.stdout:5/85: creat d0/d6/f1d x:0 0 0 2024-04-12T05:37:09.282 INFO:tasks.workunit.client.1.smithi067.stdout:5/86: link d0/da/de/l17 d0/d6/d8/l1e 0 2024-04-12T05:37:09.286 INFO:tasks.workunit.client.1.smithi067.stdout:5/87: dread d0/da/de/f15 [0,4194304] 0 2024-04-12T05:37:09.291 INFO:tasks.workunit.client.1.smithi067.stdout:3/58: truncate f5 4046487 0 2024-04-12T05:37:09.292 INFO:tasks.workunit.client.1.smithi067.stdout:3/59: chown la 57851303 1 2024-04-12T05:37:09.292 INFO:tasks.workunit.client.1.smithi067.stdout:5/88: dread d0/f1 [4194304,4194304] 0 2024-04-12T05:37:09.293 INFO:tasks.workunit.client.1.smithi067.stdout:5/89: stat d0/da/de/f15 0 2024-04-12T05:37:09.295 INFO:tasks.workunit.client.1.smithi067.stdout:5/90: mknod d0/da/c1f 0 2024-04-12T05:37:09.297 INFO:tasks.workunit.client.1.smithi067.stdout:5/91: write d0/da/de/f14 [5056642,79307] 0 2024-04-12T05:37:09.300 INFO:tasks.workunit.client.1.smithi067.stdout:5/92: mkdir d0/d6/d20 0 2024-04-12T05:37:09.302 INFO:tasks.workunit.client.1.smithi067.stdout:5/93: write d0/fc [1599630,3974] 0 2024-04-12T05:37:09.305 INFO:tasks.workunit.client.1.smithi067.stdout:3/60: dwrite f5 [4194304,4194304] 0 2024-04-12T05:37:09.306 INFO:tasks.workunit.client.1.smithi067.stdout:5/94: unlink d0/f1 0 2024-04-12T05:37:09.309 INFO:tasks.workunit.client.1.smithi067.stdout:5/95: symlink d0/d6/d20/l21 0 2024-04-12T05:37:09.310 INFO:tasks.workunit.client.1.smithi067.stdout:5/96: chown d0/d6/f12 55722 1 2024-04-12T05:37:09.310 INFO:tasks.workunit.client.1.smithi067.stdout:5/97: chown d0/d6 4263183 1 2024-04-12T05:37:09.312 INFO:tasks.workunit.client.1.smithi067.stdout:4/18: sync 2024-04-12T05:37:09.313 INFO:tasks.workunit.client.1.smithi067.stdout:5/98: symlink d0/da/de/l22 0 2024-04-12T05:37:09.313 INFO:tasks.workunit.client.1.smithi067.stdout:4/19: fsync f3 0 2024-04-12T05:37:09.314 INFO:tasks.workunit.client.0.smithi032.stdout:7/651: dread d0/d10/d70/d7b/f88 [0,4194304] 0 2024-04-12T05:37:09.316 INFO:tasks.workunit.client.0.smithi032.stdout:9/515: rmdir d9/d15/d1a/d24/d27 39 2024-04-12T05:37:09.316 INFO:tasks.workunit.client.0.smithi032.stdout:4/595: truncate d4/d87/d96/d2f/f4e 3874847 0 2024-04-12T05:37:09.319 INFO:tasks.workunit.client.0.smithi032.stdout:2/648: link dd/d19/l2b dd/d19/lce 0 2024-04-12T05:37:09.321 INFO:tasks.workunit.client.1.smithi067.stdout:3/61: fsync f5 0 2024-04-12T05:37:09.321 INFO:tasks.workunit.client.0.smithi032.stdout:9/516: write d9/d32/d6e/f91 [256167,27446] 0 2024-04-12T05:37:09.321 INFO:tasks.workunit.client.0.smithi032.stdout:6/752: dwrite d4/d14/d25/f7e [0,4194304] 0 2024-04-12T05:37:09.321 INFO:tasks.workunit.client.0.smithi032.stdout:7/652: creat d0/d10/d70/d7b/dd2/fdc x:0 0 0 2024-04-12T05:37:09.322 INFO:tasks.workunit.client.0.smithi032.stdout:4/596: dread - d4/d87/d96/d2f/d3b/fc4 zero size 2024-04-12T05:37:09.324 INFO:tasks.workunit.client.1.smithi067.stdout:5/99: link d0/da/de/c13 d0/d6/d20/c23 0 2024-04-12T05:37:09.325 INFO:tasks.workunit.client.0.smithi032.stdout:2/649: fdatasync dd/d19/d9d/d1c/d64/f70 0 2024-04-12T05:37:09.327 INFO:tasks.workunit.client.0.smithi032.stdout:9/517: symlink d9/d15/d1a/d1d/d5a/lb1 0 2024-04-12T05:37:09.328 INFO:tasks.workunit.client.0.smithi032.stdout:7/653: symlink d0/dd/d51/ldd 0 2024-04-12T05:37:09.329 INFO:tasks.workunit.client.0.smithi032.stdout:4/597: write d4/d13/d44/fca [448391,91690] 0 2024-04-12T05:37:09.330 INFO:tasks.workunit.client.0.smithi032.stdout:7/654: chown d0/d1c/d4c 623 1 2024-04-12T05:37:09.331 INFO:tasks.workunit.client.0.smithi032.stdout:6/753: getdents d4/d21/d27 0 2024-04-12T05:37:09.333 INFO:tasks.workunit.client.0.smithi032.stdout:6/754: write f3 [4842201,83115] 0 2024-04-12T05:37:09.334 INFO:tasks.workunit.client.1.smithi067.stdout:3/62: dwrite f1 [0,4194304] 0 2024-04-12T05:37:09.338 INFO:tasks.workunit.client.0.smithi032.stdout:9/518: rename d9/le to d9/d15/d1a/d24/d71/lb2 0 2024-04-12T05:37:09.339 INFO:tasks.workunit.client.0.smithi032.stdout:7/655: symlink d0/dd/d51/lde 0 2024-04-12T05:37:09.341 INFO:tasks.workunit.client.0.smithi032.stdout:7/656: write d0/dd/f6c [1327411,13583] 0 2024-04-12T05:37:09.341 INFO:tasks.workunit.client.0.smithi032.stdout:2/650: link dd/d19/d9d/d1c/d64/d32/d3a/f6a dd/d19/d9d/d42/d4a/fcf 0 2024-04-12T05:37:09.342 INFO:tasks.workunit.client.0.smithi032.stdout:6/755: symlink d4/d21/d27/dad/d55/d99/l102 0 2024-04-12T05:37:09.344 INFO:tasks.workunit.client.0.smithi032.stdout:4/598: creat d4/d13/fcb x:0 0 0 2024-04-12T05:37:09.344 INFO:tasks.workunit.client.0.smithi032.stdout:9/519: mknod d9/d1f/cb3 0 2024-04-12T05:37:09.345 INFO:tasks.workunit.client.1.smithi067.stdout:4/20: dwrite f3 [0,4194304] 0 2024-04-12T05:37:09.345 INFO:tasks.workunit.client.0.smithi032.stdout:2/651: truncate dd/d19/d20/fa0 949516 0 2024-04-12T05:37:09.345 INFO:tasks.workunit.client.0.smithi032.stdout:9/520: chown d9/d15 144 1 2024-04-12T05:37:09.346 INFO:tasks.workunit.client.0.smithi032.stdout:7/657: creat d0/d13/d21/d81/fdf x:0 0 0 2024-04-12T05:37:09.348 INFO:tasks.workunit.client.0.smithi032.stdout:4/599: creat d4/d87/d96/d2f/d6e/fcc x:0 0 0 2024-04-12T05:37:09.351 INFO:tasks.workunit.client.0.smithi032.stdout:4/600: write d4/d13/d19/fa8 [144003,54429] 0 2024-04-12T05:37:09.351 INFO:tasks.workunit.client.0.smithi032.stdout:2/652: mknod dd/d19/d9d/d1c/d64/d32/d3a/d8d/cd0 0 2024-04-12T05:37:09.353 INFO:tasks.workunit.client.0.smithi032.stdout:6/756: link d4/d21/d27/dad/d55/d99/l102 d4/de0/de7/l103 0 2024-04-12T05:37:09.357 INFO:tasks.workunit.client.1.smithi067.stdout:5/100: dwrite d0/da/de/f15 [0,4194304] 0 2024-04-12T05:37:09.357 INFO:tasks.workunit.client.0.smithi032.stdout:4/601: fsync d4/d13/fd 0 2024-04-12T05:37:09.358 INFO:tasks.workunit.client.0.smithi032.stdout:6/757: symlink d4/de0/de7/d79/l104 0 2024-04-12T05:37:09.359 INFO:tasks.workunit.client.0.smithi032.stdout:7/658: dwrite d0/d13/d21/d30/d31/fbd [0,4194304] 0 2024-04-12T05:37:09.359 INFO:tasks.workunit.client.1.smithi067.stdout:5/101: read d0/d6/f10 [1041983,5151] 0 2024-04-12T05:37:09.362 INFO:tasks.workunit.client.1.smithi067.stdout:5/102: rmdir d0 39 2024-04-12T05:37:09.363 INFO:tasks.workunit.client.0.smithi032.stdout:4/602: write d4/d87/d96/d4d/d78/f64 [1413971,95554] 0 2024-04-12T05:37:09.363 INFO:tasks.workunit.client.0.smithi032.stdout:2/653: rename dd/d19/d9d/d1c/d64/d32/d6d/l72 to dd/d19/d9d/d42/da7/dc5/ld1 0 2024-04-12T05:37:09.363 INFO:tasks.workunit.client.0.smithi032.stdout:7/659: chown d0/fc 62872515 1 2024-04-12T05:37:09.377 INFO:tasks.workunit.client.0.smithi032.stdout:6/758: dwrite d4/de0/de7/d88/fcc [0,4194304] 0 2024-04-12T05:37:09.378 INFO:tasks.workunit.client.0.smithi032.stdout:7/660: dread d0/d10/d70/d7b/f88 [0,4194304] 0 2024-04-12T05:37:09.384 INFO:tasks.workunit.client.0.smithi032.stdout:2/654: dread dd/d19/d9d/d1c/f29 [0,4194304] 0 2024-04-12T05:37:09.384 INFO:tasks.workunit.client.0.smithi032.stdout:6/759: symlink d4/d21/l105 0 2024-04-12T05:37:09.385 INFO:tasks.workunit.client.0.smithi032.stdout:4/603: link d4/d87/d96/d2f/d3b/da2/laf d4/d87/d96/lcd 0 2024-04-12T05:37:09.385 INFO:tasks.workunit.client.0.smithi032.stdout:7/661: mkdir d0/d13/d21/d30/de0 0 2024-04-12T05:37:09.386 INFO:tasks.workunit.client.1.smithi067.stdout:5/103: write d0/da/de/f14 [5844398,59924] 0 2024-04-12T05:37:09.386 INFO:tasks.workunit.client.0.smithi032.stdout:9/521: dread d9/d15/d1a/d1d/f4d [0,4194304] 0 2024-04-12T05:37:09.388 INFO:tasks.workunit.client.0.smithi032.stdout:2/655: mknod dd/d19/d20/cd2 0 2024-04-12T05:37:09.388 INFO:tasks.workunit.client.0.smithi032.stdout:6/760: write d4/de0/de7/d79/dc6/fc8 [3671711,83288] 0 2024-04-12T05:37:09.389 INFO:tasks.workunit.client.1.smithi067.stdout:5/104: symlink d0/d6/l24 0 2024-04-12T05:37:09.389 INFO:tasks.workunit.client.0.smithi032.stdout:4/604: write d4/d87/d96/d2f/d3b/f9e [73795,31689] 0 2024-04-12T05:37:09.390 INFO:tasks.workunit.client.0.smithi032.stdout:7/662: read d0/d1c/d71/f94 [2580129,75847] 0 2024-04-12T05:37:09.391 INFO:tasks.workunit.client.1.smithi067.stdout:5/105: chown d0/da/de/l22 434622 1 2024-04-12T05:37:09.392 INFO:tasks.workunit.client.0.smithi032.stdout:9/522: fdatasync d9/d15/d1a/d24/d27/d88/fa8 0 2024-04-12T05:37:09.393 INFO:tasks.workunit.client.0.smithi032.stdout:7/663: stat d0/d13/d21/d30/fd3 0 2024-04-12T05:37:09.393 INFO:tasks.workunit.client.0.smithi032.stdout:6/761: chown d4/de0/de7/d88/fc0 1884 1 2024-04-12T05:37:09.395 INFO:tasks.workunit.client.1.smithi067.stdout:5/106: truncate d0/da/de/f16 4830895 0 2024-04-12T05:37:09.395 INFO:tasks.workunit.client.0.smithi032.stdout:4/605: mknod d4/d87/d96/d4d/db5/cce 0 2024-04-12T05:37:09.396 INFO:tasks.workunit.client.0.smithi032.stdout:6/762: chown d4/de0/de7/d88/fac 97458576 1 2024-04-12T05:37:09.397 INFO:tasks.workunit.client.1.smithi067.stdout:5/107: symlink d0/d6/l25 0 2024-04-12T05:37:09.399 INFO:tasks.workunit.client.0.smithi032.stdout:9/523: creat d9/d15/d1a/d31/d65/d80/fb4 x:0 0 0 2024-04-12T05:37:09.400 INFO:tasks.workunit.client.0.smithi032.stdout:4/606: mkdir d4/d87/d96/d4d/db5/dcf 0 2024-04-12T05:37:09.401 INFO:tasks.workunit.client.0.smithi032.stdout:6/763: dwrite d4/f6 [0,4194304] 0 2024-04-12T05:37:09.403 INFO:tasks.workunit.client.0.smithi032.stdout:6/764: chown d4/d21/d27/f8e 7042 1 2024-04-12T05:37:09.404 INFO:tasks.workunit.client.0.smithi032.stdout:2/656: rename dd/d19/d20/lad to dd/d19/d9d/d1c/d64/d32/ld3 0 2024-04-12T05:37:09.406 INFO:tasks.workunit.client.1.smithi067.stdout:5/108: truncate d0/f2 2686498 0 2024-04-12T05:37:09.406 INFO:tasks.workunit.client.0.smithi032.stdout:7/664: getdents d0/d13/d21/d30/d31 0 2024-04-12T05:37:09.409 INFO:tasks.workunit.client.0.smithi032.stdout:9/524: rmdir d9/d15/d1a/d31 39 2024-04-12T05:37:09.412 INFO:tasks.workunit.client.0.smithi032.stdout:4/607: dwrite d4/d13/d44/f91 [0,4194304] 0 2024-04-12T05:37:09.412 INFO:tasks.workunit.client.0.smithi032.stdout:9/525: fdatasync d9/d15/d1a/d24/d27/f90 0 2024-04-12T05:37:09.412 INFO:tasks.workunit.client.0.smithi032.stdout:7/665: rmdir d0/d34/dbe/dbb 39 2024-04-12T05:37:09.417 INFO:tasks.workunit.client.0.smithi032.stdout:2/657: write fa [1258614,71001] 0 2024-04-12T05:37:09.419 INFO:tasks.workunit.client.0.smithi032.stdout:4/608: symlink d4/d87/d96/d2f/d7c/dc2/ld0 0 2024-04-12T05:37:09.419 INFO:tasks.workunit.client.0.smithi032.stdout:2/658: write dd/d19/d20/fa0 [738957,100005] 0 2024-04-12T05:37:09.425 INFO:tasks.workunit.client.0.smithi032.stdout:0/650: sync 2024-04-12T05:37:09.425 INFO:tasks.workunit.client.0.smithi032.stdout:1/540: sync 2024-04-12T05:37:09.426 INFO:tasks.workunit.client.0.smithi032.stdout:0/651: chown d2/d29/d8a/laa 364 1 2024-04-12T05:37:09.428 INFO:tasks.workunit.client.0.smithi032.stdout:6/765: getdents d4/de0/de7/d92 0 2024-04-12T05:37:09.429 INFO:tasks.workunit.client.0.smithi032.stdout:2/659: mknod dd/d19/cd4 0 2024-04-12T05:37:09.429 INFO:tasks.workunit.client.1.smithi067.stdout:5/109: dwrite d0/f7 [0,4194304] 0 2024-04-12T05:37:09.429 INFO:tasks.workunit.client.0.smithi032.stdout:4/609: symlink d4/d87/d96/d2f/d7c/ld1 0 2024-04-12T05:37:09.430 INFO:tasks.workunit.client.0.smithi032.stdout:7/666: getdents d0/dd/d51 0 2024-04-12T05:37:09.431 INFO:tasks.workunit.client.0.smithi032.stdout:0/652: chown d2/d8/c11 29 1 2024-04-12T05:37:09.431 INFO:tasks.workunit.client.0.smithi032.stdout:9/526: link d9/d15/d1a/d24/d71/l7b d9/d15/d1a/d24/d27/lb5 0 2024-04-12T05:37:09.432 INFO:tasks.workunit.client.0.smithi032.stdout:4/610: truncate d4/f54 4674938 0 2024-04-12T05:37:09.434 INFO:tasks.workunit.client.1.smithi067.stdout:5/110: getdents d0/d6/d8 0 2024-04-12T05:37:09.434 INFO:tasks.workunit.client.0.smithi032.stdout:7/667: mkdir d0/de/d47/de1 0 2024-04-12T05:37:09.437 INFO:tasks.workunit.client.0.smithi032.stdout:4/611: read - d4/d87/d96/d2f/d9a/fa7 zero size 2024-04-12T05:37:09.437 INFO:tasks.workunit.client.1.smithi067.stdout:5/111: mknod d0/d6/d20/c26 0 2024-04-12T05:37:09.437 INFO:tasks.workunit.client.0.smithi032.stdout:1/541: getdents d5/de/d17/d35/d73/da9 0 2024-04-12T05:37:09.440 INFO:tasks.workunit.client.0.smithi032.stdout:7/668: creat d0/dd/fe2 x:0 0 0 2024-04-12T05:37:09.442 INFO:tasks.workunit.client.0.smithi032.stdout:0/653: truncate d2/d28/d87/d79/f7c 1582443 0 2024-04-12T05:37:09.443 INFO:tasks.workunit.client.1.smithi067.stdout:5/112: dread d0/f2 [0,4194304] 0 2024-04-12T05:37:09.443 INFO:tasks.workunit.client.1.smithi067.stdout:5/113: chown d0 1987611 1 2024-04-12T05:37:09.443 INFO:tasks.workunit.client.0.smithi032.stdout:4/612: read - d4/d13/d19/fb8 zero size 2024-04-12T05:37:09.443 INFO:tasks.workunit.client.0.smithi032.stdout:1/542: write d5/de/f10 [6448829,113632] 0 2024-04-12T05:37:09.444 INFO:tasks.workunit.client.0.smithi032.stdout:6/766: getdents d4/de0/de7/d37/d69/d76 0 2024-04-12T05:37:09.444 INFO:tasks.workunit.client.1.smithi067.stdout:5/114: chown d0/d6/l24 3579835 1 2024-04-12T05:37:09.447 INFO:tasks.workunit.client.1.smithi067.stdout:5/115: rename d0/da/de to d0/da/d27 0 2024-04-12T05:37:09.448 INFO:tasks.workunit.client.0.smithi032.stdout:7/669: stat d0/d13/d21/d30/d31/fbd 0 2024-04-12T05:37:09.449 INFO:tasks.workunit.client.1.smithi067.stdout:5/116: mkdir d0/d6/d20/d28 0 2024-04-12T05:37:09.451 INFO:tasks.workunit.client.0.smithi032.stdout:4/613: mkdir d4/d87/d96/d4d/dbc/dd2 0 2024-04-12T05:37:09.451 INFO:tasks.workunit.client.0.smithi032.stdout:7/670: stat d0/de/df/d20/f68 0 2024-04-12T05:37:09.452 INFO:tasks.workunit.client.1.smithi067.stdout:5/117: getdents d0/d6/d20/d28 0 2024-04-12T05:37:09.453 INFO:tasks.workunit.client.1.smithi067.stdout:5/118: fsync d0/da/d27/f18 0 2024-04-12T05:37:09.454 INFO:tasks.workunit.client.0.smithi032.stdout:9/527: dwrite d9/d15/d2c/f62 [4194304,4194304] 0 2024-04-12T05:37:09.454 INFO:tasks.workunit.client.1.smithi067.stdout:5/119: chown d0/d6/l24 4163 1 2024-04-12T05:37:09.456 INFO:tasks.workunit.client.0.smithi032.stdout:6/767: dread d4/de0/de7/dc4/fe9 [0,4194304] 0 2024-04-12T05:37:09.456 INFO:tasks.workunit.client.1.smithi067.stdout:5/120: stat d0/d6/d8/l1e 0 2024-04-12T05:37:09.458 INFO:tasks.workunit.client.1.smithi067.stdout:5/121: mknod d0/d6/d20/d28/c29 0 2024-04-12T05:37:09.459 INFO:tasks.workunit.client.0.smithi032.stdout:7/671: chown d0/de/df/l90 93799 1 2024-04-12T05:37:09.462 INFO:tasks.workunit.client.0.smithi032.stdout:7/672: chown d0/d10/d70 1004409 1 2024-04-12T05:37:09.464 INFO:tasks.workunit.client.0.smithi032.stdout:2/660: dread dd/d19/d20/f28 [4194304,4194304] 0 2024-04-12T05:37:09.464 INFO:tasks.workunit.client.1.smithi067.stdout:5/122: getdents d0/da/d27 0 2024-04-12T05:37:09.464 INFO:tasks.workunit.client.1.smithi067.stdout:5/123: chown d0/da/c1f 466620 1 2024-04-12T05:37:09.466 INFO:tasks.workunit.client.0.smithi032.stdout:6/768: unlink d4/d21/d2e/lfa 0 2024-04-12T05:37:09.467 INFO:tasks.workunit.client.1.smithi067.stdout:5/124: creat d0/d6/d8/f2a x:0 0 0 2024-04-12T05:37:09.468 INFO:tasks.workunit.client.0.smithi032.stdout:1/543: dwrite d5/de/f10 [4194304,4194304] 0 2024-04-12T05:37:09.468 INFO:tasks.workunit.client.0.smithi032.stdout:8/672: sync 2024-04-12T05:37:09.468 INFO:tasks.workunit.client.0.smithi032.stdout:3/660: sync 2024-04-12T05:37:09.469 INFO:tasks.workunit.client.0.smithi032.stdout:5/663: sync 2024-04-12T05:37:09.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.471+0000 7fe4352a9700 1 -- 172.21.15.32:0/2846387143 --> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe430066f60 con 0x7fe4180692a0 2024-04-12T05:37:09.470 INFO:tasks.workunit.client.1.smithi067.stdout:5/125: creat d0/d6/f2b x:0 0 0 2024-04-12T05:37:09.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.473+0000 7fe4177fe700 1 -- 172.21.15.32:0/2846387143 <== mgr.14208 v2:172.21.15.32:6800/595990262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7fe430066f60 con 0x7fe4180692a0 2024-04-12T05:37:09.473 INFO:tasks.workunit.client.0.smithi032.stdout:0/654: dwrite d2/d28/d87/f71 [0,4194304] 0 2024-04-12T05:37:09.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.477+0000 7fe4352a9700 1 -- 172.21.15.32:0/2846387143 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fe4180692a0 msgr2=0x7fe41806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:09.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.477+0000 7fe4352a9700 1 --2- 172.21.15.32:0/2846387143 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fe4180692a0 0x7fe41806b750 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe424009b80 tx=0x7fe42400b040 comp rx=0 tx=0).stop 2024-04-12T05:37:09.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.477+0000 7fe4352a9700 1 -- 172.21.15.32:0/2846387143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4300fbcd0 msgr2=0x7fe43010c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:09.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.477+0000 7fe4352a9700 1 --2- 172.21.15.32:0/2846387143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4300fbcd0 0x7fe43010c6a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe4200137b0 tx=0x7fe420009e30 comp rx=0 tx=0).stop 2024-04-12T05:37:09.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.477+0000 7fe4352a9700 1 -- 172.21.15.32:0/2846387143 shutdown_connections 2024-04-12T05:37:09.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.477+0000 7fe4352a9700 1 --2- 172.21.15.32:0/2846387143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4300fbcd0 0x7fe43010c6a0 secure :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe4200137b0 tx=0x7fe420009e30 comp rx=0 tx=0).stop 2024-04-12T05:37:09.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.477+0000 7fe4352a9700 1 --2- 172.21.15.32:0/2846387143 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fe4180692a0 0x7fe41806b750 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:09.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.477+0000 7fe4352a9700 1 --2- 172.21.15.32:0/2846387143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4300fcf80 0x7fe43010cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:09.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.477+0000 7fe4352a9700 1 -- 172.21.15.32:0/2846387143 >> 172.21.15.32:0/2846387143 conn(0x7fe4300f7450 msgr2=0x7fe4300f8b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:09.476 INFO:tasks.workunit.client.0.smithi032.stdout:7/673: rename d0/d1c/d4c/f5e to d0/fe3 0 2024-04-12T05:37:09.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.479+0000 7fe4352a9700 1 -- 172.21.15.32:0/2846387143 shutdown_connections 2024-04-12T05:37:09.477 INFO:tasks.workunit.client.0.smithi032.stdout:8/673: chown d0/c23 94290842 1 2024-04-12T05:37:09.478 INFO:tasks.workunit.client.1.smithi067.stdout:5/126: dwrite d0/d6/f2b [0,4194304] 0 2024-04-12T05:37:09.479 INFO:tasks.workunit.client.0.smithi032.stdout:4/614: dread d4/d13/fd [0,4194304] 0 2024-04-12T05:37:09.479 INFO:tasks.workunit.client.0.smithi032.stdout:6/769: creat d4/d21/d27/dad/d55/d99/f106 x:0 0 0 2024-04-12T05:37:09.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.483+0000 7fe4352a9700 1 -- 172.21.15.32:0/2846387143 wait complete. 2024-04-12T05:37:09.484 INFO:tasks.workunit.client.0.smithi032.stdout:8/674: read - d0/db/d38/d63/d77/f97 zero size 2024-04-12T05:37:09.485 INFO:tasks.workunit.client.0.smithi032.stdout:7/674: unlink d0/dd/fe2 0 2024-04-12T05:37:09.486 INFO:tasks.workunit.client.0.smithi032.stdout:2/661: mkdir dd/d19/d9d/d42/d4a/da1/dd5 0 2024-04-12T05:37:09.489 INFO:tasks.workunit.client.1.smithi067.stdout:5/127: getdents d0/d6/d20/d28 0 2024-04-12T05:37:09.490 INFO:tasks.workunit.client.1.smithi067.stdout:5/128: readlink d0/d6/l24 0 2024-04-12T05:37:09.492 INFO:tasks.workunit.client.0.smithi032.stdout:9/528: dread d9/d1f/f39 [0,4194304] 0 2024-04-12T05:37:09.493 INFO:tasks.workunit.client.0.smithi032.stdout:8/675: creat d0/db/dd/d11/d1b/d96/fef x:0 0 0 2024-04-12T05:37:09.493 INFO:tasks.workunit.client.0.smithi032.stdout:7/675: chown d0/d13/d21/d81/cc3 1443 1 2024-04-12T05:37:09.497 INFO:tasks.workunit.client.0.smithi032.stdout:0/655: symlink d2/d29/d8a/ld5 0 2024-04-12T05:37:09.498 INFO:tasks.workunit.client.0.smithi032.stdout:1/544: link d5/d2c/d4f/l6a d5/d2c/d4f/lac 0 2024-04-12T05:37:09.499 INFO:tasks.workunit.client.0.smithi032.stdout:7/676: write d0/d1c/fc6 [354312,76577] 0 2024-04-12T05:37:09.501 INFO:tasks.workunit.client.0.smithi032.stdout:2/662: rename dd/d19/d20/d83/l91 to dd/d19/d20/d83/ld6 0 2024-04-12T05:37:09.502 INFO:tasks.workunit.client.1.smithi067.stdout:5/129: getdents d0/d6 0 2024-04-12T05:37:09.503 INFO:tasks.workunit.client.0.smithi032.stdout:5/664: dwrite d2/df/d25/d43/d49/f7d [0,4194304] 0 2024-04-12T05:37:09.504 INFO:tasks.workunit.client.0.smithi032.stdout:4/615: creat d4/d87/d96/d4d/fd3 x:0 0 0 2024-04-12T05:37:09.505 INFO:tasks.workunit.client.1.smithi067.stdout:5/130: write d0/da/d27/f15 [3297206,94653] 0 2024-04-12T05:37:09.506 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:37:09.512 INFO:tasks.workunit.client.0.smithi032.stdout:2/663: creat dd/d8c/fd7 x:0 0 0 2024-04-12T05:37:09.512 INFO:tasks.workunit.client.0.smithi032.stdout:4/616: unlink d4/d13/d19/fb8 0 2024-04-12T05:37:09.513 INFO:tasks.workunit.client.0.smithi032.stdout:0/656: write d2/d28/f8e [327912,16148] 0 2024-04-12T05:37:09.513 INFO:tasks.workunit.client.0.smithi032.stdout:9/529: dwrite d9/d15/d2c/f23 [8388608,4194304] 0 2024-04-12T05:37:09.513 INFO:tasks.workunit.client.0.smithi032.stdout:8/676: creat d0/db/dd/ff0 x:0 0 0 2024-04-12T05:37:09.514 INFO:tasks.workunit.client.0.smithi032.stdout:6/770: dread d4/de0/de7/d88/fc0 [0,4194304] 0 2024-04-12T05:37:09.514 INFO:tasks.workunit.client.0.smithi032.stdout:3/661: dread d12/d2f/f91 [0,4194304] 0 2024-04-12T05:37:09.515 INFO:tasks.workunit.client.1.smithi067.stdout:5/131: write d0/d6/f12 [1276184,59588] 0 2024-04-12T05:37:09.516 INFO:tasks.workunit.client.1.smithi067.stdout:5/132: write d0/d6/f12 [265574,119187] 0 2024-04-12T05:37:09.518 INFO:tasks.workunit.client.0.smithi032.stdout:8/677: readlink d0/db/d7d/la0 0 2024-04-12T05:37:09.519 INFO:tasks.workunit.client.1.smithi067.stdout:5/133: mkdir d0/d6/d8/d2c 0 2024-04-12T05:37:09.521 INFO:tasks.workunit.client.1.smithi067.stdout:5/134: mkdir d0/da/d2d 0 2024-04-12T05:37:09.521 INFO:tasks.workunit.client.1.smithi067.stdout:5/135: readlink d0/d6/d20/l21 0 2024-04-12T05:37:09.522 INFO:tasks.workunit.client.1.smithi067.stdout:5/136: chown d0/d6/d8/l1e 86333590 1 2024-04-12T05:37:09.522 INFO:tasks.workunit.client.1.smithi067.stdout:5/137: dread - d0/d6/d8/f19 zero size 2024-04-12T05:37:09.523 INFO:tasks.workunit.client.1.smithi067.stdout:5/138: stat d0/d6/f12 0 2024-04-12T05:37:09.523 INFO:tasks.workunit.client.1.smithi067.stdout:5/139: stat d0/d6/d8 0 2024-04-12T05:37:09.529 INFO:tasks.workunit.client.0.smithi032.stdout:7/677: dwrite d0/de/df/d29/f89 [0,4194304] 0 2024-04-12T05:37:09.530 INFO:tasks.workunit.client.1.smithi067.stdout:5/140: dwrite d0/d6/f1d [0,4194304] 0 2024-04-12T05:37:09.531 INFO:tasks.workunit.client.0.smithi032.stdout:2/664: dwrite dd/d19/d9d/d1c/f81 [4194304,4194304] 0 2024-04-12T05:37:09.534 INFO:tasks.workunit.client.1.smithi067.stdout:6/17: sync 2024-04-12T05:37:09.534 INFO:tasks.workunit.client.1.smithi067.stdout:6/18: write f4 [290109,80174] 0 2024-04-12T05:37:09.536 INFO:tasks.workunit.client.0.smithi032.stdout:0/657: dread d2/d8/d66/f82 [0,4194304] 0 2024-04-12T05:37:09.537 INFO:tasks.workunit.client.0.smithi032.stdout:7/678: write d0/de/df/f7a [2461049,59506] 0 2024-04-12T05:37:09.537 INFO:tasks.workunit.client.1.smithi067.stdout:6/19: stat f0 0 2024-04-12T05:37:09.539 INFO:tasks.workunit.client.1.smithi067.stdout:6/20: readlink l2 0 2024-04-12T05:37:09.540 INFO:tasks.workunit.client.0.smithi032.stdout:3/662: symlink d12/d2e/d3a/dbe/ld7 0 2024-04-12T05:37:09.541 INFO:tasks.workunit.client.0.smithi032.stdout:7/679: write d0/dd/f6c [1769513,108845] 0 2024-04-12T05:37:09.545 INFO:tasks.workunit.client.0.smithi032.stdout:0/658: unlink d2/d8/d66/f82 0 2024-04-12T05:37:09.545 INFO:tasks.workunit.client.0.smithi032.stdout:9/530: getdents d9/d15/d2c/dab 0 2024-04-12T05:37:09.546 INFO:tasks.workunit.client.1.smithi067.stdout:6/21: symlink l5 0 2024-04-12T05:37:09.547 INFO:tasks.workunit.client.0.smithi032.stdout:0/659: write d2/dc/d3e/f46 [2228618,60359] 0 2024-04-12T05:37:09.548 INFO:tasks.workunit.client.0.smithi032.stdout:2/665: unlink dd/d19/d9d/d1c/d64/d32/d3a/cc0 0 2024-04-12T05:37:09.548 INFO:tasks.workunit.client.0.smithi032.stdout:5/665: dread d2/d3c/f56 [0,4194304] 0 2024-04-12T05:37:09.549 INFO:tasks.workunit.client.0.smithi032.stdout:7/680: creat d0/d13/d21/d7c/fe4 x:0 0 0 2024-04-12T05:37:09.550 INFO:tasks.workunit.client.0.smithi032.stdout:3/663: symlink d12/d2e/d3a/d2d/ld8 0 2024-04-12T05:37:09.552 INFO:tasks.workunit.client.0.smithi032.stdout:8/678: link d0/db/dd/d11/d1b/d22/d35/c7f d0/db/d7d/cf1 0 2024-04-12T05:37:09.553 INFO:tasks.workunit.client.0.smithi032.stdout:2/666: mkdir dd/d19/d9d/d42/d4a/da1/dd8 0 2024-04-12T05:37:09.554 INFO:tasks.workunit.client.0.smithi032.stdout:4/617: getdents d4/d87/d96/d2f/d6e/dc0 0 2024-04-12T05:37:09.557 INFO:tasks.workunit.client.1.smithi067.stdout:6/22: dwrite f4 [0,4194304] 0 2024-04-12T05:37:09.557 INFO:tasks.workunit.client.0.smithi032.stdout:4/618: stat d4/d87/d96/d4d/d78 0 2024-04-12T05:37:09.559 INFO:tasks.workunit.client.0.smithi032.stdout:4/619: dread - d4/d13/d44/f5c zero size 2024-04-12T05:37:09.559 INFO:tasks.workunit.client.0.smithi032.stdout:1/545: dread d5/de/d17/d1b/f38 [0,4194304] 0 2024-04-12T05:37:09.560 INFO:tasks.workunit.client.1.smithi067.stdout:6/23: fdatasync f0 0 2024-04-12T05:37:09.560 INFO:tasks.workunit.client.1.smithi067.stdout:6/24: rmdir - no directory 2024-04-12T05:37:09.561 INFO:tasks.workunit.client.0.smithi032.stdout:3/664: fdatasync d12/d2e/f94 0 2024-04-12T05:37:09.561 INFO:tasks.workunit.client.0.smithi032.stdout:5/666: symlink d2/df/d25/d50/d91/ld5 0 2024-04-12T05:37:09.564 INFO:tasks.workunit.client.0.smithi032.stdout:3/665: readlink d12/daa/lc0 0 2024-04-12T05:37:09.565 INFO:tasks.workunit.client.0.smithi032.stdout:8/679: symlink d0/db/dd/d11/d1b/d22/d29/ded/lf2 0 2024-04-12T05:37:09.567 INFO:tasks.workunit.client.1.smithi067.stdout:6/25: link f3 f6 0 2024-04-12T05:37:09.567 INFO:tasks.workunit.client.0.smithi032.stdout:9/531: dread d9/d15/d1a/d24/f4b [0,4194304] 0 2024-04-12T05:37:09.567 INFO:tasks.workunit.client.0.smithi032.stdout:1/546: readlink d5/de/d17/d35/d61/d77/d8f/da0/la4 0 2024-04-12T05:37:09.570 INFO:tasks.workunit.client.0.smithi032.stdout:5/667: fsync d2/df/d25/d43/d49/f7d 0 2024-04-12T05:37:09.571 INFO:tasks.workunit.client.0.smithi032.stdout:2/667: dread dd/f68 [0,4194304] 0 2024-04-12T05:37:09.572 INFO:tasks.workunit.client.0.smithi032.stdout:4/620: mknod d4/d87/d96/d2f/d7c/dc2/cd4 0 2024-04-12T05:37:09.572 INFO:tasks.workunit.client.0.smithi032.stdout:1/547: read d5/de/d2a/f2b [341130,72114] 0 2024-04-12T05:37:09.574 INFO:tasks.workunit.client.0.smithi032.stdout:6/771: dread d4/d21/d2e/d32/f36 [0,4194304] 0 2024-04-12T05:37:09.575 INFO:tasks.workunit.client.1.smithi067.stdout:6/26: write f0 [498498,29158] 0 2024-04-12T05:37:09.575 INFO:tasks.workunit.client.0.smithi032.stdout:9/532: creat d9/d58/fb6 x:0 0 0 2024-04-12T05:37:09.577 INFO:tasks.workunit.client.0.smithi032.stdout:6/772: dread - d4/de0/fb9 zero size 2024-04-12T05:37:09.577 INFO:tasks.workunit.client.1.smithi067.stdout:5/141: dwrite d0/d6/f1d [4194304,4194304] 0 2024-04-12T05:37:09.577 INFO:tasks.workunit.client.1.smithi067.stdout:6/27: creat f7 x:0 0 0 2024-04-12T05:37:09.578 INFO:tasks.workunit.client.0.smithi032.stdout:5/668: mkdir d2/d3c/d5b/d75/dd6 0 2024-04-12T05:37:09.579 INFO:tasks.workunit.client.0.smithi032.stdout:9/533: dread d9/d32/d6e/f93 [0,4194304] 0 2024-04-12T05:37:09.581 INFO:tasks.workunit.client.1.smithi067.stdout:5/142: write d0/f7 [5163440,43589] 0 2024-04-12T05:37:09.582 INFO:tasks.workunit.client.0.smithi032.stdout:8/680: creat d0/d36/de2/ff3 x:0 0 0 2024-04-12T05:37:09.582 INFO:tasks.workunit.client.0.smithi032.stdout:2/668: creat dd/d19/d9d/d42/da7/dc5/fd9 x:0 0 0 2024-04-12T05:37:09.591 INFO:tasks.workunit.client.0.smithi032.stdout:9/534: rmdir d9/d15/d1a/d1d/d5a 39 2024-04-12T05:37:09.591 INFO:tasks.workunit.client.1.smithi067.stdout:6/28: dwrite f0 [0,4194304] 0 2024-04-12T05:37:09.592 INFO:tasks.workunit.client.1.smithi067.stdout:5/143: creat d0/da/f2e x:0 0 0 2024-04-12T05:37:09.594 INFO:tasks.workunit.client.0.smithi032.stdout:6/773: symlink d4/d21/db4/l107 0 2024-04-12T05:37:09.595 INFO:tasks.workunit.client.0.smithi032.stdout:1/548: dread d5/f47 [0,4194304] 0 2024-04-12T05:37:09.596 INFO:tasks.workunit.client.1.smithi067.stdout:5/144: symlink d0/da/d27/l2f 0 2024-04-12T05:37:09.596 INFO:tasks.workunit.client.0.smithi032.stdout:6/774: write d4/ff2 [347689,122288] 0 2024-04-12T05:37:09.599 INFO:tasks.workunit.client.1.smithi067.stdout:6/29: dread f4 [0,4194304] 0 2024-04-12T05:37:09.600 INFO:tasks.workunit.client.1.smithi067.stdout:6/30: readlink l1 0 2024-04-12T05:37:09.600 INFO:tasks.workunit.client.0.smithi032.stdout:3/666: getdents d12/d2e 0 2024-04-12T05:37:09.600 INFO:tasks.workunit.client.0.smithi032.stdout:4/621: dwrite d4/d87/d96/d68/fb0 [0,4194304] 0 2024-04-12T05:37:09.604 INFO:tasks.workunit.client.1.smithi067.stdout:6/31: creat f8 x:0 0 0 2024-04-12T05:37:09.614 INFO:tasks.workunit.client.1.smithi067.stdout:6/32: dwrite f8 [0,4194304] 0 2024-04-12T05:37:09.616 INFO:tasks.workunit.client.1.smithi067.stdout:6/33: write f6 [1561085,102689] 0 2024-04-12T05:37:09.617 INFO:tasks.workunit.client.0.smithi032.stdout:3/667: creat d12/d2f/d89/fd9 x:0 0 0 2024-04-12T05:37:09.620 INFO:tasks.workunit.client.1.smithi067.stdout:6/34: creat f9 x:0 0 0 2024-04-12T05:37:09.621 INFO:tasks.workunit.client.0.smithi032.stdout:6/775: creat d4/de0/de7/d88/d8d/f108 x:0 0 0 2024-04-12T05:37:09.622 INFO:tasks.workunit.client.0.smithi032.stdout:4/622: truncate d4/d87/d96/d2f/d6e/fb6 121081 0 2024-04-12T05:37:09.623 INFO:tasks.workunit.client.0.smithi032.stdout:5/669: write d2/d3c/d48/d4b/fbb [679465,14983] 0 2024-04-12T05:37:09.623 INFO:tasks.workunit.client.1.smithi067.stdout:6/35: dread f4 [0,4194304] 0 2024-04-12T05:37:09.623 INFO:tasks.workunit.client.0.smithi032.stdout:8/681: dread d0/f7 [0,4194304] 0 2024-04-12T05:37:09.624 INFO:tasks.workunit.client.0.smithi032.stdout:7/681: dwrite d0/de/f15 [0,4194304] 0 2024-04-12T05:37:09.626 INFO:tasks.workunit.client.0.smithi032.stdout:9/535: rename d9/f13 to d9/d15/d1a/d31/d65/fb7 0 2024-04-12T05:37:09.628 INFO:tasks.workunit.client.1.smithi067.stdout:6/36: dread f8 [0,4194304] 0 2024-04-12T05:37:09.629 INFO:tasks.workunit.client.0.smithi032.stdout:5/670: chown d2/df/d25/d50/fab 288835 1 2024-04-12T05:37:09.629 INFO:tasks.workunit.client.0.smithi032.stdout:4/623: read d4/d87/d96/d68/fb0 [1072539,103293] 0 2024-04-12T05:37:09.629 INFO:tasks.workunit.client.0.smithi032.stdout:8/682: dread - d0/db/dd/f75 zero size 2024-04-12T05:37:09.631 INFO:tasks.workunit.client.0.smithi032.stdout:6/776: symlink d4/de0/de7/d92/d9e/l109 0 2024-04-12T05:37:09.631 INFO:tasks.workunit.client.0.smithi032.stdout:3/668: chown d12/d24/f2c 47723 1 2024-04-12T05:37:09.633 INFO:tasks.workunit.client.0.smithi032.stdout:8/683: mkdir d0/df4 0 2024-04-12T05:37:09.634 INFO:tasks.workunit.client.1.smithi067.stdout:6/37: creat fa x:0 0 0 2024-04-12T05:37:09.634 INFO:tasks.workunit.client.1.smithi067.stdout:6/38: rmdir - no directory 2024-04-12T05:37:09.637 INFO:tasks.workunit.client.0.smithi032.stdout:4/624: dread - d4/d87/d96/d4d/d78/f95 zero size 2024-04-12T05:37:09.638 INFO:tasks.workunit.client.1.smithi067.stdout:6/39: chown f4 2697560 1 2024-04-12T05:37:09.641 INFO:tasks.workunit.client.0.smithi032.stdout:2/669: dwrite dd/d19/d9d/d42/d4a/fcf [4194304,4194304] 0 2024-04-12T05:37:09.641 INFO:tasks.workunit.client.1.smithi067.stdout:5/145: dwrite d0/da/d27/f16 [0,4194304] 0 2024-04-12T05:37:09.642 INFO:tasks.workunit.client.0.smithi032.stdout:5/671: rename d2/dd/d5d/f9f to d2/df/d31/da2/fd7 0 2024-04-12T05:37:09.643 INFO:tasks.workunit.client.0.smithi032.stdout:6/777: symlink d4/de0/de7/d92/l10a 0 2024-04-12T05:37:09.647 INFO:tasks.workunit.client.0.smithi032.stdout:4/625: rmdir d4/d87/d96/d4d/db5 39 2024-04-12T05:37:09.652 INFO:tasks.workunit.client.0.smithi032.stdout:7/682: dwrite d0/de/df/f24 [0,4194304] 0 2024-04-12T05:37:09.652 INFO:tasks.workunit.client.0.smithi032.stdout:3/669: dwrite d12/d2e/f82 [0,4194304] 0 2024-04-12T05:37:09.653 INFO:tasks.workunit.client.1.smithi067.stdout:5/146: dwrite d0/fb [0,4194304] 0 2024-04-12T05:37:09.656 INFO:tasks.workunit.client.1.smithi067.stdout:5/147: symlink d0/da/d27/l30 0 2024-04-12T05:37:09.662 INFO:tasks.workunit.client.1.smithi067.stdout:5/148: dread d0/d6/f12 [0,4194304] 0 2024-04-12T05:37:09.665 INFO:tasks.workunit.client.0.smithi032.stdout:9/536: dread d9/d15/d1a/d24/d27/f90 [0,4194304] 0 2024-04-12T05:37:09.666 INFO:tasks.workunit.client.0.smithi032.stdout:5/672: creat d2/d9b/fd8 x:0 0 0 2024-04-12T05:37:09.667 INFO:tasks.workunit.client.1.smithi067.stdout:5/149: read d0/da/d27/f18 [729873,99456] 0 2024-04-12T05:37:09.667 INFO:tasks.workunit.client.0.smithi032.stdout:7/683: rmdir d0/de/df 39 2024-04-12T05:37:09.669 INFO:tasks.workunit.client.0.smithi032.stdout:3/670: chown d12/d2e/l65 21 1 2024-04-12T05:37:09.672 INFO:tasks.workunit.client.1.smithi067.stdout:5/150: symlink d0/da/d2d/l31 0 2024-04-12T05:37:09.672 INFO:tasks.workunit.client.0.smithi032.stdout:9/537: dread d9/d15/f6a [0,4194304] 0 2024-04-12T05:37:09.672 INFO:tasks.workunit.client.0.smithi032.stdout:9/538: rename d9 to d9/d15/db8 22 2024-04-12T05:37:09.674 INFO:tasks.workunit.client.1.smithi067.stdout:5/151: truncate d0/da/f1b 36962 0 2024-04-12T05:37:09.676 INFO:tasks.workunit.client.1.smithi067.stdout:5/152: unlink d0/da/d27/l30 0 2024-04-12T05:37:09.678 INFO:tasks.workunit.client.1.smithi067.stdout:5/153: rename d0/d6 to d0/d6/d8/d2c/d32 22 2024-04-12T05:37:09.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.684+0000 7f1024459700 1 -- 172.21.15.32:0/393434991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f101c0fce70 msgr2=0x7f101c0fd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:09.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.684+0000 7f1024459700 1 --2- 172.21.15.32:0/393434991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f101c0fce70 0x7f101c0fd280 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f1018004940 tx=0x7f1018009e30 comp rx=0 tx=0).stop 2024-04-12T05:37:09.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.684+0000 7f1024459700 1 -- 172.21.15.32:0/393434991 shutdown_connections 2024-04-12T05:37:09.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.684+0000 7f1024459700 1 --2- 172.21.15.32:0/393434991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f101c0fdfe0 0x7f101c0fe450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:09.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.684+0000 7f1024459700 1 --2- 172.21.15.32:0/393434991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f101c0fce70 0x7f101c0fd280 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:09.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.684+0000 7f1024459700 1 -- 172.21.15.32:0/393434991 >> 172.21.15.32:0/393434991 conn(0x7f101c0f8550 msgr2=0x7f101c0fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:09.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.684+0000 7f1024459700 1 -- 172.21.15.32:0/393434991 shutdown_connections 2024-04-12T05:37:09.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.685+0000 7f1024459700 1 -- 172.21.15.32:0/393434991 wait complete. 2024-04-12T05:37:09.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.685+0000 7f1024459700 1 Processor -- start 2024-04-12T05:37:09.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.685+0000 7f1024459700 1 -- start start 2024-04-12T05:37:09.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.685+0000 7f1024459700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f101c0fce70 0x7f101c071c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:37:09.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.685+0000 7f1024459700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f101c0fdfe0 0x7f101c070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:37:09.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.685+0000 7f1024459700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101c070850 con 0x7f101c0fdfe0 2024-04-12T05:37:09.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.685+0000 7f1024459700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101c0709c0 con 0x7f101c0fce70 2024-04-12T05:37:09.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.686+0000 7f10219f4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f101c0fdfe0 0x7f101c070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:09.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.686+0000 7f10219f4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f101c0fdfe0 0x7f101c070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56566/0 (socket says 172.21.15.32:56566) 2024-04-12T05:37:09.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.686+0000 7f10219f4700 1 -- 172.21.15.32:0/3443969063 learned_addr learned my addr 172.21.15.32:0/3443969063 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:37:09.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.686+0000 7f10219f4700 1 -- 172.21.15.32:0/3443969063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f101c0fce70 msgr2=0x7f101c071c60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T05:37:09.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.686+0000 7f10219f4700 1 --2- 172.21.15.32:0/3443969063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f101c0fce70 0x7f101c071c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:09.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.686+0000 7f10219f4700 1 -- 172.21.15.32:0/3443969063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1018005040 con 0x7f101c0fdfe0 2024-04-12T05:37:09.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.686+0000 7f10219f4700 1 --2- 172.21.15.32:0/3443969063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f101c0fdfe0 0x7f101c070310 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f100c00b790 tx=0x7f100c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:09.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.686+0000 7f1012ffd700 1 -- 172.21.15.32:0/3443969063 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f100c00f770 con 0x7f101c0fdfe0 2024-04-12T05:37:09.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.686+0000 7f1012ffd700 1 -- 172.21.15.32:0/3443969063 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f100c00fd80 con 0x7f101c0fdfe0 2024-04-12T05:37:09.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.686+0000 7f1012ffd700 1 -- 172.21.15.32:0/3443969063 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f100c00d310 con 0x7f101c0fdfe0 2024-04-12T05:37:09.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.686+0000 7f1024459700 1 -- 172.21.15.32:0/3443969063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f101c070ca0 con 0x7f101c0fdfe0 2024-04-12T05:37:09.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.687+0000 7f1024459700 1 -- 172.21.15.32:0/3443969063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f101c115960 con 0x7f101c0fdfe0 2024-04-12T05:37:09.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.687+0000 7f1024459700 1 -- 172.21.15.32:0/3443969063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f101c04c710 con 0x7f101c0fdfe0 2024-04-12T05:37:09.686 INFO:tasks.workunit.client.0.smithi032.stdout:3/671: dread d12/d2e/d3a/d63/fcf [0,4194304] 0 2024-04-12T05:37:09.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.688+0000 7f1012ffd700 1 -- 172.21.15.32:0/3443969063 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f100c010420 con 0x7f101c0fdfe0 2024-04-12T05:37:09.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.688+0000 7f1012ffd700 1 --2- 172.21.15.32:0/3443969063 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f10080692a0 0x7f100806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:37:09.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.689+0000 7f1012ffd700 1 -- 172.21.15.32:0/3443969063 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f100c087220 con 0x7f101c0fdfe0 2024-04-12T05:37:09.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.689+0000 7f10221f5700 1 --2- 172.21.15.32:0/3443969063 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f10080692a0 0x7f100806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:09.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.689+0000 7f10221f5700 1 --2- 172.21.15.32:0/3443969063 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f10080692a0 0x7f100806b750 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f101800b900 tx=0x7f1018009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:09.688 INFO:tasks.workunit.client.1.smithi067.stdout:5/154: dread d0/fc [0,4194304] 0 2024-04-12T05:37:09.691 INFO:tasks.workunit.client.1.smithi067.stdout:5/155: fsync d0/da/f2e 0 2024-04-12T05:37:09.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:09.694+0000 7f1012ffd700 1 -- 172.21.15.32:0/3443969063 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f100c056ef0 con 0x7f101c0fdfe0 2024-04-12T05:37:09.693 INFO:tasks.workunit.client.1.smithi067.stdout:5/156: readlink d0/da/d27/l22 0 2024-04-12T05:37:09.696 INFO:tasks.workunit.client.1.smithi067.stdout:5/157: mkdir d0/d6/d33 0 2024-04-12T05:37:09.697 INFO:tasks.workunit.client.1.smithi067.stdout:5/158: stat d0/d6/d8/l5 0 2024-04-12T05:37:09.698 INFO:tasks.workunit.client.1.smithi067.stdout:0/134: sync 2024-04-12T05:37:09.701 INFO:tasks.workunit.client.0.smithi032.stdout:6/778: dread d4/d21/d2e/ff0 [0,4194304] 0 2024-04-12T05:37:09.706 INFO:tasks.workunit.client.1.smithi067.stdout:5/159: rename d0/d6/l25 to d0/d6/d8/l34 0 2024-04-12T05:37:09.708 INFO:tasks.workunit.client.1.smithi067.stdout:5/160: mknod d0/da/d27/c35 0 2024-04-12T05:37:09.710 INFO:tasks.workunit.client.1.smithi067.stdout:5/161: mkdir d0/d36 0 2024-04-12T05:37:09.712 INFO:tasks.workunit.client.1.smithi067.stdout:5/162: fdatasync d0/fc 0 2024-04-12T05:37:09.713 INFO:tasks.workunit.client.0.smithi032.stdout:2/670: fdatasync dd/d19/d9d/d42/d4a/fb3 0 2024-04-12T05:37:09.713 INFO:tasks.workunit.client.1.smithi067.stdout:0/135: write dd/d12/d14/d1a/f1e [290739,126747] 0 2024-04-12T05:37:09.714 INFO:tasks.workunit.client.1.smithi067.stdout:5/163: write d0/fb [2531618,129397] 0 2024-04-12T05:37:09.720 INFO:tasks.workunit.client.1.smithi067.stdout:0/136: write f1 [5607106,85525] 0 2024-04-12T05:37:09.726 INFO:tasks.workunit.client.1.smithi067.stdout:5/164: dwrite d0/f1c [0,4194304] 0 2024-04-12T05:37:09.727 INFO:tasks.workunit.client.1.smithi067.stdout:0/137: mknod dd/d12/d14/d1a/c28 0 2024-04-12T05:37:09.732 INFO:tasks.workunit.client.0.smithi032.stdout:6/779: creat d4/de0/de7/d88/f10b x:0 0 0 2024-04-12T05:37:09.733 INFO:tasks.workunit.client.0.smithi032.stdout:6/780: dread - d4/de0/de7/d37/d69/d8a/fd5 zero size 2024-04-12T05:37:09.734 INFO:tasks.workunit.client.1.smithi067.stdout:5/165: read d0/da/d27/f16 [3737339,37879] 0 2024-04-12T05:37:09.737 INFO:tasks.workunit.client.1.smithi067.stdout:0/138: creat dd/d12/f29 x:0 0 0 2024-04-12T05:37:09.738 INFO:tasks.workunit.client.0.smithi032.stdout:2/671: symlink dd/d19/d9d/d42/d4a/da1/lda 0 2024-04-12T05:37:09.738 INFO:tasks.workunit.client.0.smithi032.stdout:5/673: truncate d2/dd/f3a 3972184 0 2024-04-12T05:37:09.739 INFO:tasks.workunit.client.0.smithi032.stdout:5/674: write d2/df/f5c [485702,15989] 0 2024-04-12T05:37:09.741 INFO:tasks.workunit.client.0.smithi032.stdout:5/675: chown d2/df/da9/dc1 0 1 2024-04-12T05:37:09.743 INFO:tasks.workunit.client.1.smithi067.stdout:0/139: creat dd/d12/f2a x:0 0 0 2024-04-12T05:37:09.746 INFO:tasks.workunit.client.0.smithi032.stdout:2/672: fsync dd/f14 0 2024-04-12T05:37:09.746 INFO:tasks.workunit.client.1.smithi067.stdout:5/166: write d0/d6/f10 [4220596,84431] 0 2024-04-12T05:37:09.747 INFO:tasks.workunit.client.0.smithi032.stdout:6/781: dread d4/de0/de7/d37/f5d [0,4194304] 0 2024-04-12T05:37:09.748 INFO:tasks.workunit.client.0.smithi032.stdout:4/626: link d4/d87/d96/d4d/db5/cce d4/d87/d96/d4d/dbc/cd5 0 2024-04-12T05:37:09.750 INFO:tasks.workunit.client.0.smithi032.stdout:5/676: creat d2/df/d25/d50/fd9 x:0 0 0 2024-04-12T05:37:09.755 INFO:tasks.workunit.client.1.smithi067.stdout:0/140: dwrite dd/d12/f2a [0,4194304] 0 2024-04-12T05:37:09.755 INFO:tasks.workunit.client.0.smithi032.stdout:4/627: creat d4/d13/d44/fd6 x:0 0 0 2024-04-12T05:37:09.756 INFO:tasks.workunit.client.0.smithi032.stdout:2/673: dwrite dd/d19/d9d/d1c/d64/d32/d3a/fa8 [0,4194304] 0 2024-04-12T05:37:09.756 INFO:tasks.workunit.client.1.smithi067.stdout:0/141: write dd/f1c [334246,45618] 0 2024-04-12T05:37:09.757 INFO:tasks.workunit.client.1.smithi067.stdout:5/167: link d0/da/d27/f14 d0/d6/d8/d2c/f37 0 2024-04-12T05:37:09.757 INFO:tasks.workunit.client.0.smithi032.stdout:3/672: getdents d12/d2e/d3a/d63 0 2024-04-12T05:37:09.758 INFO:tasks.workunit.client.0.smithi032.stdout:4/628: chown d4/d87/d96/d32/dc5 691 1 2024-04-12T05:37:09.759 INFO:tasks.workunit.client.0.smithi032.stdout:4/629: stat d4/d87/d96/d68/d9b 0 2024-04-12T05:37:09.761 INFO:tasks.workunit.client.1.smithi067.stdout:0/142: creat dd/d12/d14/d1a/f2b x:0 0 0 2024-04-12T05:37:09.761 INFO:tasks.workunit.client.1.smithi067.stdout:5/168: mkdir d0/da/d2d/d38 0 2024-04-12T05:37:09.764 INFO:tasks.workunit.client.1.smithi067.stdout:0/143: write f1 [6612520,98663] 0 2024-04-12T05:37:09.766 INFO:tasks.workunit.client.0.smithi032.stdout:9/539: link d9/d32/f53 d9/d15/d1a/d1d/fb9 0 2024-04-12T05:37:09.768 INFO:tasks.workunit.client.1.smithi067.stdout:5/169: symlink d0/da/d2d/l39 0 2024-04-12T05:37:09.768 INFO:tasks.workunit.client.0.smithi032.stdout:0/660: sync 2024-04-12T05:37:09.769 INFO:tasks.workunit.client.0.smithi032.stdout:4/630: dwrite d4/f3d [0,4194304] 0 2024-04-12T05:37:09.769 INFO:tasks.workunit.client.1.smithi067.stdout:0/144: rename la to dd/l2c 0 2024-04-12T05:37:09.772 INFO:tasks.workunit.client.0.smithi032.stdout:2/674: mknod dd/cdb 0 2024-04-12T05:37:09.776 INFO:tasks.workunit.client.0.smithi032.stdout:3/673: write d12/d2e/d3a/d2d/f45 [4232979,82940] 0 2024-04-12T05:37:09.779 INFO:tasks.workunit.client.0.smithi032.stdout:0/661: dwrite d2/d29/f5d [0,4194304] 0 2024-04-12T05:37:09.780 INFO:tasks.workunit.client.1.smithi067.stdout:5/170: truncate d0/da/d27/f14 1588875 0 2024-04-12T05:37:09.781 INFO:tasks.workunit.client.0.smithi032.stdout:6/782: link d4/d21/d27/lae d4/de0/de7/d79/l10c 0 2024-04-12T05:37:09.785 INFO:tasks.workunit.client.0.smithi032.stdout:4/631: creat d4/d87/d96/d68/fd7 x:0 0 0 2024-04-12T05:37:09.786 INFO:tasks.workunit.client.1.smithi067.stdout:0/145: readlink dd/l2c 0 2024-04-12T05:37:09.790 INFO:tasks.workunit.client.0.smithi032.stdout:9/540: mkdir d9/d15/d1a/dba 0 2024-04-12T05:37:09.793 INFO:tasks.workunit.client.0.smithi032.stdout:2/675: getdents dd/d19/d9d/d42/d4a/da1/dd5 0 2024-04-12T05:37:09.794 INFO:tasks.workunit.client.0.smithi032.stdout:9/541: dread - d9/d15/d1a/d24/d27/d88/fa8 zero size 2024-04-12T05:37:09.795 INFO:tasks.workunit.client.0.smithi032.stdout:3/674: dwrite d12/d2e/d3a/d63/fcf [0,4194304] 0 2024-04-12T05:37:09.795 INFO:tasks.workunit.client.0.smithi032.stdout:6/783: chown d4/f2c 22 1 2024-04-12T05:37:09.796 INFO:tasks.workunit.client.1.smithi067.stdout:0/146: mkdir dd/d12/d14/d2d 0 2024-04-12T05:37:09.796 INFO:tasks.workunit.client.1.smithi067.stdout:0/147: chown dd/d12/d14/f15 86293 1 2024-04-12T05:37:09.797 INFO:tasks.workunit.client.0.smithi032.stdout:0/662: mkdir d2/d28/dd6 0 2024-04-12T05:37:09.800 INFO:tasks.workunit.client.1.smithi067.stdout:0/148: creat dd/d12/f2e x:0 0 0 2024-04-12T05:37:09.803 INFO:tasks.workunit.client.0.smithi032.stdout:3/675: creat d12/d24/fda x:0 0 0 2024-04-12T05:37:09.806 INFO:tasks.workunit.client.0.smithi032.stdout:9/542: dwrite d9/fa7 [0,4194304] 0 2024-04-12T05:37:09.806 INFO:tasks.workunit.client.1.smithi067.stdout:0/149: truncate dd/d12/d14/d1a/f2b 243617 0 2024-04-12T05:37:09.807 INFO:tasks.workunit.client.0.smithi032.stdout:9/543: stat d9/d15/d2c/c66 0 2024-04-12T05:37:09.815 INFO:tasks.workunit.client.1.smithi067.stdout:0/150: rename dd/d12/d14/f15 to dd/d12/d14/d2d/f2f 0 2024-04-12T05:37:09.816 INFO:tasks.workunit.client.0.smithi032.stdout:9/544: mknod d9/d15/d1a/cbb 0 2024-04-12T05:37:09.817 INFO:tasks.workunit.client.0.smithi032.stdout:9/545: fdatasync d9/d15/d2c/f62 0 2024-04-12T05:37:09.819 INFO:tasks.workunit.client.0.smithi032.stdout:9/546: dread - d9/d15/d1a/d24/d27/d88/fa8 zero size 2024-04-12T05:37:09.819 INFO:tasks.workunit.client.1.smithi067.stdout:0/151: symlink dd/d12/l30 0 2024-04-12T05:37:09.819 INFO:tasks.workunit.client.0.smithi032.stdout:3/676: rmdir d12/d24/dbb/dd4 0 2024-04-12T05:37:09.820 INFO:tasks.workunit.client.0.smithi032.stdout:3/677: truncate d12/d2e/d98/fad 725033 0 2024-04-12T05:37:09.822 INFO:tasks.workunit.client.0.smithi032.stdout:3/678: write d12/d2f/f41 [8144037,104860] 0 2024-04-12T05:37:09.822 INFO:tasks.workunit.client.0.smithi032.stdout:9/547: creat d9/d15/d1a/d31/d65/fbc x:0 0 0 2024-04-12T05:37:09.824 INFO:tasks.workunit.client.1.smithi067.stdout:0/152: link l4 dd/d12/d14/d1a/l31 0 2024-04-12T05:37:09.825 INFO:tasks.workunit.client.0.smithi032.stdout:1/549: sync 2024-04-12T05:37:09.829 INFO:tasks.workunit.client.0.smithi032.stdout:6/784: creat d4/d14/f10d x:0 0 0 2024-04-12T05:37:09.832 INFO:tasks.workunit.client.0.smithi032.stdout:1/550: link d5/de/d17/d35/d3e/fa3 d5/de/d17/d35/d78/fad 0 2024-04-12T05:37:09.834 INFO:tasks.workunit.client.0.smithi032.stdout:6/785: readlink d4/de0/de7/l103 0 2024-04-12T05:37:09.835 INFO:tasks.workunit.client.1.smithi067.stdout:0/153: dwrite dd/f1c [0,4194304] 0 2024-04-12T05:37:09.836 INFO:tasks.workunit.client.0.smithi032.stdout:6/786: symlink d4/d74/l10e 0 2024-04-12T05:37:09.839 INFO:tasks.workunit.client.0.smithi032.stdout:6/787: dread - d4/d14/f10d zero size 2024-04-12T05:37:09.840 INFO:tasks.workunit.client.0.smithi032.stdout:1/551: dwrite d5/de/d17/d35/f75 [0,4194304] 0 2024-04-12T05:37:09.844 INFO:tasks.workunit.client.0.smithi032.stdout:6/788: creat d4/de0/de7/d37/d69/d8a/f10f x:0 0 0 2024-04-12T05:37:09.845 INFO:tasks.workunit.client.0.smithi032.stdout:6/789: chown d4/d21/d27/f8e 49978247 1 2024-04-12T05:37:09.849 INFO:tasks.workunit.client.1.smithi067.stdout:0/154: dwrite f1 [4194304,4194304] 0 2024-04-12T05:37:09.849 INFO:tasks.workunit.client.0.smithi032.stdout:6/790: write d4/de0/de7/f31 [2555362,23567] 0 2024-04-12T05:37:09.852 INFO:tasks.workunit.client.0.smithi032.stdout:5/677: dwrite d2/df/d25/d1e/f40 [0,4194304] 0 2024-04-12T05:37:09.854 INFO:tasks.workunit.client.1.smithi067.stdout:0/155: dread dd/d12/d14/d1a/f1e [0,4194304] 0 2024-04-12T05:37:09.858 INFO:tasks.workunit.client.0.smithi032.stdout:8/684: link d0/db/dd/f17 d0/db/dd/d11/d1b/ff5 0 2024-04-12T05:37:09.858 INFO:tasks.workunit.client.1.smithi067.stdout:0/156: creat dd/d12/d14/d2d/f32 x:0 0 0 2024-04-12T05:37:09.859 INFO:tasks.workunit.client.0.smithi032.stdout:4/632: write d4/d87/d96/d2f/f4e [3770791,45239] 0 2024-04-12T05:37:09.860 INFO:tasks.workunit.client.0.smithi032.stdout:4/633: fsync d4/d13/d44/f5c 0 2024-04-12T05:37:09.860 INFO:tasks.workunit.client.0.smithi032.stdout:8/685: truncate d0/db/dd/d11/d1b/ff5 1573063 0 2024-04-12T05:37:09.861 INFO:tasks.workunit.client.0.smithi032.stdout:6/791: creat d4/d21/d27/dad/d55/f110 x:0 0 0 2024-04-12T05:37:09.861 INFO:tasks.workunit.client.1.smithi067.stdout:0/157: creat dd/d12/d14/f33 x:0 0 0 2024-04-12T05:37:09.862 INFO:tasks.workunit.client.0.smithi032.stdout:4/634: read - d4/d87/d96/d2f/d6e/fae zero size 2024-04-12T05:37:09.865 INFO:tasks.workunit.client.0.smithi032.stdout:7/684: rename d0/de/df/l72 to d0/d1c/le5 0 2024-04-12T05:37:09.867 INFO:tasks.workunit.client.0.smithi032.stdout:6/792: symlink d4/d21/d27/dad/d55/df4/l111 0 2024-04-12T05:37:09.869 INFO:tasks.workunit.client.0.smithi032.stdout:8/686: chown d0/d36/d84/db3/fe9 2423086 1 2024-04-12T05:37:09.869 INFO:tasks.workunit.client.0.smithi032.stdout:4/635: creat d4/d13/d75/fd8 x:0 0 0 2024-04-12T05:37:09.872 INFO:tasks.workunit.client.0.smithi032.stdout:2/676: rename dd/d19/d9d/d1c/d64/l7c to dd/d19/d9d/d42/ldc 0 2024-04-12T05:37:09.872 INFO:tasks.workunit.client.1.smithi067.stdout:0/158: write dd/f13 [817840,22410] 0 2024-04-12T05:37:09.874 INFO:tasks.workunit.client.0.smithi032.stdout:9/548: truncate d9/d15/d1a/d24/f4b 959663 0 2024-04-12T05:37:09.876 INFO:tasks.workunit.client.0.smithi032.stdout:6/793: creat d4/de0/de7/d37/d69/f112 x:0 0 0 2024-04-12T05:37:09.877 INFO:tasks.workunit.client.1.smithi067.stdout:0/159: rename dd/d12/d14/d1a/c26 to dd/d12/c34 0 2024-04-12T05:37:09.879 INFO:tasks.workunit.client.0.smithi032.stdout:3/679: rename d12/d2f/d89/fa7 to d12/d2e/dcb/fdb 0 2024-04-12T05:37:09.880 INFO:tasks.workunit.client.0.smithi032.stdout:0/663: dwrite d2/d29/f47 [8388608,4194304] 0 2024-04-12T05:37:09.882 INFO:tasks.workunit.client.0.smithi032.stdout:4/636: mknod d4/d87/d96/d2f/d6e/cd9 0 2024-04-12T05:37:09.882 INFO:tasks.workunit.client.1.smithi067.stdout:0/160: rename dd/l10 to dd/d12/l35 0 2024-04-12T05:37:09.882 INFO:tasks.workunit.client.0.smithi032.stdout:2/677: unlink dd/d19/d9d/d1c/c61 0 2024-04-12T05:37:09.885 INFO:tasks.workunit.client.0.smithi032.stdout:5/678: dread d2/d9b/f5a [0,4194304] 0 2024-04-12T05:37:09.886 INFO:tasks.workunit.client.1.smithi067.stdout:0/161: symlink dd/d12/l36 0 2024-04-12T05:37:09.892 INFO:tasks.workunit.client.0.smithi032.stdout:7/685: link d0/d1c/d71/c77 d0/d10/d70/ce6 0 2024-04-12T05:37:09.893 INFO:tasks.workunit.client.0.smithi032.stdout:2/678: read dd/d19/d20/f53 [2182695,80600] 0 2024-04-12T05:37:09.893 INFO:tasks.workunit.client.0.smithi032.stdout:8/687: dwrite d0/db/dd/d11/d1b/d22/fd6 [0,4194304] 0 2024-04-12T05:37:09.895 INFO:tasks.workunit.client.0.smithi032.stdout:6/794: mkdir d4/de0/de7/d92/d9e/df6/d113 0 2024-04-12T05:37:09.896 INFO:tasks.workunit.client.0.smithi032.stdout:0/664: fdatasync d2/f14 0 2024-04-12T05:37:09.899 INFO:tasks.workunit.client.1.smithi067.stdout:0/162: getdents dd/d12/d14/d1a 0 2024-04-12T05:37:09.899 INFO:tasks.workunit.client.0.smithi032.stdout:1/552: dwrite d5/de/d17/f26 [0,4194304] 0 2024-04-12T05:37:09.899 INFO:tasks.workunit.client.0.smithi032.stdout:2/679: stat dd/d19/d9d/d42/d4a/l52 0 2024-04-12T05:37:09.899 INFO:tasks.workunit.client.0.smithi032.stdout:5/679: fdatasync d2/df/d31/da2/fd7 0 2024-04-12T05:37:09.900 INFO:tasks.workunit.client.0.smithi032.stdout:9/549: write d9/fa [2880821,99134] 0 2024-04-12T05:37:09.900 INFO:tasks.workunit.client.0.smithi032.stdout:0/665: chown d2/dc/d3e/f53 44930538 1 2024-04-12T05:37:09.900 INFO:tasks.workunit.client.0.smithi032.stdout:8/688: rmdir d0/db/d7d 39 2024-04-12T05:37:09.902 INFO:tasks.workunit.client.1.smithi067.stdout:0/163: dread dd/d12/d14/d1a/f1e [0,4194304] 0 2024-04-12T05:37:09.903 INFO:tasks.workunit.client.0.smithi032.stdout:0/666: read - d2/d28/d87/f95 zero size 2024-04-12T05:37:09.904 INFO:tasks.workunit.client.0.smithi032.stdout:4/637: link d4/d87/d96/d2f/fc1 d4/d87/d96/d2f/fda 0 2024-04-12T05:37:09.904 INFO:tasks.workunit.client.0.smithi032.stdout:5/680: truncate d2/d3c/d48/d4b/f6b 1514136 0 2024-04-12T05:37:09.905 INFO:tasks.workunit.client.1.smithi067.stdout:0/164: dread dd/d12/d14/d1a/f1e [0,4194304] 0 2024-04-12T05:37:09.908 INFO:tasks.workunit.client.0.smithi032.stdout:7/686: mknod d0/d34/dbf/ce7 0 2024-04-12T05:37:09.909 INFO:tasks.workunit.client.0.smithi032.stdout:2/680: creat dd/d19/d9d/d42/d4a/fdd x:0 0 0 2024-04-12T05:37:09.909 INFO:tasks.workunit.client.0.smithi032.stdout:8/689: rmdir d0/db/dd/d11/d1b/d22/d35 39 2024-04-12T05:37:09.912 INFO:tasks.workunit.client.0.smithi032.stdout:4/638: mknod d4/d87/d96/d2f/d6e/cdb 0 2024-04-12T05:37:09.913 INFO:tasks.workunit.client.0.smithi032.stdout:4/639: write d4/f79 [1567436,17867] 0 2024-04-12T05:37:09.914 INFO:tasks.workunit.client.0.smithi032.stdout:4/640: chown d4/d87/d96/d4d/dbc 124575758 1 2024-04-12T05:37:09.914 INFO:tasks.workunit.client.0.smithi032.stdout:4/641: stat d4/f79 0 2024-04-12T05:37:09.915 INFO:tasks.workunit.client.0.smithi032.stdout:6/795: mknod d4/de0/de7/d79/dc6/d101/c114 0 2024-04-12T05:37:09.918 INFO:tasks.workunit.client.0.smithi032.stdout:4/642: stat d4/d87/d96/d4d/d78/f6d 0 2024-04-12T05:37:09.918 INFO:tasks.workunit.client.0.smithi032.stdout:9/550: link d9/d15/f48 d9/d15/d2c/dab/fbd 0 2024-04-12T05:37:09.919 INFO:tasks.workunit.client.0.smithi032.stdout:4/643: truncate d4/d13/d75/fac 885319 0 2024-04-12T05:37:09.920 INFO:tasks.workunit.client.0.smithi032.stdout:0/667: creat d2/d29/d8a/fd7 x:0 0 0 2024-04-12T05:37:09.920 INFO:tasks.workunit.client.0.smithi032.stdout:9/551: write d9/d15/d2c/f23 [11008563,69842] 0 2024-04-12T05:37:09.921 INFO:tasks.workunit.client.0.smithi032.stdout:6/796: creat d4/de0/f115 x:0 0 0 2024-04-12T05:37:09.921 INFO:tasks.workunit.client.0.smithi032.stdout:5/681: rmdir d2/df/da9/dc1/d78/dc7 0 2024-04-12T05:37:09.926 INFO:tasks.workunit.client.0.smithi032.stdout:6/797: readlink d4/d21/d2e/dca/lfe 0 2024-04-12T05:37:09.928 INFO:tasks.workunit.client.0.smithi032.stdout:9/552: creat d9/d15/d1a/d31/d65/d80/fbe x:0 0 0 2024-04-12T05:37:09.929 INFO:tasks.workunit.client.0.smithi032.stdout:8/690: creat d0/db/dd/d11/d1b/d22/d35/ff6 x:0 0 0 2024-04-12T05:37:09.930 INFO:tasks.workunit.client.0.smithi032.stdout:2/681: link dd/d19/d82/lb2 dd/d19/d9d/d1c/d64/d32/d6d/lde 0 2024-04-12T05:37:09.931 INFO:tasks.workunit.client.0.smithi032.stdout:6/798: creat d4/de0/de7/dc4/f116 x:0 0 0 2024-04-12T05:37:09.932 INFO:tasks.workunit.client.0.smithi032.stdout:0/668: creat d2/d8/d17/fd8 x:0 0 0 2024-04-12T05:37:09.932 INFO:tasks.workunit.client.0.smithi032.stdout:1/553: dwrite d5/de/d2a/f3f [8388608,4194304] 0 2024-04-12T05:37:09.932 INFO:tasks.workunit.client.0.smithi032.stdout:4/644: rename d4/d87/d96/d4d/cbe to d4/d87/d96/d4d/cdc 0 2024-04-12T05:37:09.934 INFO:tasks.workunit.client.0.smithi032.stdout:6/799: readlink d4/d14/d25/l6c 0 2024-04-12T05:37:09.938 INFO:tasks.workunit.client.0.smithi032.stdout:5/682: getdents d2/df/da9/dc1 0 2024-04-12T05:37:09.939 INFO:tasks.workunit.client.0.smithi032.stdout:4/645: write d4/d13/ff [394933,46668] 0 2024-04-12T05:37:09.941 INFO:tasks.workunit.client.0.smithi032.stdout:9/553: dread d9/d15/d1a/d24/d27/f90 [0,4194304] 0 2024-04-12T05:37:09.945 INFO:tasks.workunit.client.0.smithi032.stdout:8/691: dread d0/db/dd/d11/d1b/f89 [4194304,4194304] 0 2024-04-12T05:37:09.946 INFO:tasks.workunit.client.0.smithi032.stdout:0/669: truncate d2/dc/d3e/dac/f54 663084 0 2024-04-12T05:37:09.947 INFO:tasks.workunit.client.0.smithi032.stdout:6/800: link d4/d21/d27/dad/d55/f70 d4/d21/d2e/d32/f117 0 2024-04-12T05:37:09.947 INFO:tasks.workunit.client.0.smithi032.stdout:1/554: dread d5/de/d2a/f2b [0,4194304] 0 2024-04-12T05:37:09.950 INFO:tasks.workunit.client.0.smithi032.stdout:8/692: mkdir d0/d58/df7 0 2024-04-12T05:37:09.951 INFO:tasks.workunit.client.0.smithi032.stdout:1/555: chown d5/d2c/d4f/c8d 14859 1 2024-04-12T05:37:09.954 INFO:tasks.workunit.client.0.smithi032.stdout:5/683: dwrite d2/df/d25/f98 [4194304,4194304] 0 2024-04-12T05:37:09.956 INFO:tasks.workunit.client.0.smithi032.stdout:0/670: truncate d2/f6 612775 0 2024-04-12T05:37:09.959 INFO:tasks.workunit.client.0.smithi032.stdout:9/554: dwrite d9/d15/d2c/d41/f96 [0,4194304] 0 2024-04-12T05:37:09.961 INFO:tasks.workunit.client.0.smithi032.stdout:1/556: read d5/de/d2a/f97 [1164386,3380] 0 2024-04-12T05:37:09.961 INFO:tasks.workunit.client.0.smithi032.stdout:0/671: creat d2/fd9 x:0 0 0 2024-04-12T05:37:09.964 INFO:tasks.workunit.client.0.smithi032.stdout:2/682: dread dd/d19/d9d/d1c/d64/d32/d3a/f6a [0,4194304] 0 2024-04-12T05:37:09.966 INFO:tasks.workunit.client.0.smithi032.stdout:5/684: rmdir d2/d3c/d48/d4b 39 2024-04-12T05:37:09.969 INFO:tasks.workunit.client.0.smithi032.stdout:1/557: rmdir d5/de/d17/d35/d3e 39 2024-04-12T05:37:09.969 INFO:tasks.workunit.client.0.smithi032.stdout:2/683: rmdir dd 39 2024-04-12T05:37:09.970 INFO:tasks.workunit.client.0.smithi032.stdout:1/558: write d5/de/d17/d35/d78/f7c [474829,102765] 0 2024-04-12T05:37:09.972 INFO:tasks.workunit.client.0.smithi032.stdout:5/685: dread d2/df/d25/d43/f81 [0,4194304] 0 2024-04-12T05:37:09.974 INFO:tasks.workunit.client.0.smithi032.stdout:1/559: creat d5/de/d17/fae x:0 0 0 2024-04-12T05:37:09.975 INFO:tasks.workunit.client.0.smithi032.stdout:2/684: mkdir dd/d19/d82/ddf 0 2024-04-12T05:37:09.979 INFO:tasks.workunit.client.0.smithi032.stdout:6/801: dread d4/d14/f46 [0,4194304] 0 2024-04-12T05:37:09.980 INFO:tasks.workunit.client.0.smithi032.stdout:2/685: mkdir dd/d19/d9d/d1c/de0 0 2024-04-12T05:37:09.984 INFO:tasks.workunit.client.1.smithi067.stdout:2/49: sync 2024-04-12T05:37:09.984 INFO:tasks.workunit.client.1.smithi067.stdout:9/20: sync 2024-04-12T05:37:09.984 INFO:tasks.workunit.client.1.smithi067.stdout:9/21: dread - d1/f2 zero size 2024-04-12T05:37:09.985 INFO:tasks.workunit.client.0.smithi032.stdout:4/646: dread d4/d13/f1c [0,4194304] 0 2024-04-12T05:37:09.986 INFO:tasks.workunit.client.0.smithi032.stdout:5/686: dread d2/df/d31/f46 [0,4194304] 0 2024-04-12T05:37:09.987 INFO:tasks.workunit.client.1.smithi067.stdout:9/22: symlink d1/l4 0 2024-04-12T05:37:09.988 INFO:tasks.workunit.client.1.smithi067.stdout:2/50: truncate f4 106162 0 2024-04-12T05:37:09.990 INFO:tasks.workunit.client.0.smithi032.stdout:1/560: dwrite d5/de/d2a/f89 [0,4194304] 0 2024-04-12T05:37:09.990 INFO:tasks.workunit.client.0.smithi032.stdout:6/802: mkdir d4/de0/de7/d92/d9e/df6/d113/d118 0 2024-04-12T05:37:09.990 INFO:tasks.workunit.client.1.smithi067.stdout:2/51: mkdir d5/d11 0 2024-04-12T05:37:09.991 INFO:tasks.workunit.client.1.smithi067.stdout:2/52: read f3 [4283699,90347] 0 2024-04-12T05:37:09.992 INFO:tasks.workunit.client.0.smithi032.stdout:0/672: mknod d2/d29/d77/cda 0 2024-04-12T05:37:09.993 INFO:tasks.workunit.client.0.smithi032.stdout:2/686: unlink dd/d19/d9d/d1c/d64/d32/f60 0 2024-04-12T05:37:09.996 INFO:tasks.workunit.client.1.smithi067.stdout:9/23: rmdir d1/d3 0 2024-04-12T05:37:09.997 INFO:tasks.workunit.client.0.smithi032.stdout:5/687: rename d2/dd/l85 to d2/df/da9/dc1/lda 0 2024-04-12T05:37:09.998 INFO:tasks.workunit.client.1.smithi067.stdout:2/53: read d5/da/fc [77893,8042] 0 2024-04-12T05:37:09.999 INFO:tasks.workunit.client.0.smithi032.stdout:0/673: chown d2/la0 218072314 1 2024-04-12T05:37:09.999 INFO:tasks.workunit.client.1.smithi067.stdout:9/24: symlink d1/l5 0 2024-04-12T05:37:10.000 INFO:tasks.workunit.client.1.smithi067.stdout:9/25: chown d1 7607252 1 2024-04-12T05:37:10.002 INFO:tasks.workunit.client.0.smithi032.stdout:6/803: creat d4/de0/de7/d79/dc6/f119 x:0 0 0 2024-04-12T05:37:10.002 INFO:tasks.workunit.client.0.smithi032.stdout:1/561: dread d5/de/d17/d35/d78/f79 [0,4194304] 0 2024-04-12T05:37:10.002 INFO:tasks.workunit.client.0.smithi032.stdout:4/647: dwrite d4/d13/d19/fa8 [0,4194304] 0 2024-04-12T05:37:10.003 INFO:tasks.workunit.client.0.smithi032.stdout:2/687: mkdir dd/d19/d9d/d1c/de0/de1 0 2024-04-12T05:37:10.003 INFO:tasks.workunit.client.0.smithi032.stdout:5/688: creat d2/df/da9/dc1/fdb x:0 0 0 2024-04-12T05:37:10.003 INFO:tasks.workunit.client.1.smithi067.stdout:2/54: write d5/da/fc [690096,15578] 0 2024-04-12T05:37:10.005 INFO:tasks.workunit.client.0.smithi032.stdout:5/689: chown d2/f4 0 1 2024-04-12T05:37:10.008 INFO:tasks.workunit.client.0.smithi032.stdout:0/674: symlink d2/ldb 0 2024-04-12T05:37:10.008 INFO:tasks.workunit.client.0.smithi032.stdout:1/562: symlink d5/de/d17/d35/d73/d74/laf 0 2024-04-12T05:37:10.009 INFO:tasks.workunit.client.1.smithi067.stdout:2/55: creat d5/f12 x:0 0 0 2024-04-12T05:37:10.011 INFO:tasks.workunit.client.1.smithi067.stdout:9/26: dwrite d1/f2 [0,4194304] 0 2024-04-12T05:37:10.014 INFO:tasks.workunit.client.0.smithi032.stdout:1/563: symlink d5/de/d17/d1b/lb0 0 2024-04-12T05:37:10.015 INFO:tasks.workunit.client.0.smithi032.stdout:2/688: dwrite dd/d19/d9d/d1c/d64/fc1 [0,4194304] 0 2024-04-12T05:37:10.015 INFO:tasks.workunit.client.1.smithi067.stdout:2/56: creat d5/da/de/f13 x:0 0 0 2024-04-12T05:37:10.016 INFO:tasks.workunit.client.1.smithi067.stdout:2/57: write d5/da/de/f13 [323709,110544] 0 2024-04-12T05:37:10.016 INFO:tasks.workunit.client.1.smithi067.stdout:9/27: write d1/f2 [1929139,4916] 0 2024-04-12T05:37:10.018 INFO:tasks.workunit.client.0.smithi032.stdout:0/675: unlink d2/d8/df/l13 0 2024-04-12T05:37:10.020 INFO:tasks.workunit.client.1.smithi067.stdout:2/58: mkdir d5/da/de/d14 0 2024-04-12T05:37:10.022 INFO:tasks.workunit.client.1.smithi067.stdout:9/28: creat d1/f6 x:0 0 0 2024-04-12T05:37:10.025 INFO:tasks.workunit.client.0.smithi032.stdout:2/689: dread dd/d19/d9d/d1c/d64/d32/d3a/fa8 [0,4194304] 0 2024-04-12T05:37:10.025 INFO:tasks.workunit.client.0.smithi032.stdout:0/676: chown d2/dc/d3e/dac/d51/d83/fb4 92996290 1 2024-04-12T05:37:10.028 INFO:tasks.workunit.client.0.smithi032.stdout:1/564: dwrite d5/de/d17/d35/d3b/f63 [0,4194304] 0 2024-04-12T05:37:10.030 INFO:tasks.workunit.client.0.smithi032.stdout:2/690: dread - dd/d19/d9d/d42/d4a/fc4 zero size 2024-04-12T05:37:10.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.033+0000 7f1024459700 1 -- 172.21.15.32:0/3443969063 --> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f101c102340 con 0x7f10080692a0 2024-04-12T05:37:10.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.034+0000 7f1012ffd700 1 -- 172.21.15.32:0/3443969063 <== mgr.14208 v2:172.21.15.32:6800/595990262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f101c102340 con 0x7f10080692a0 2024-04-12T05:37:10.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.037+0000 7f1024459700 1 -- 172.21.15.32:0/3443969063 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f10080692a0 msgr2=0x7f100806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:10.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.037+0000 7f1024459700 1 --2- 172.21.15.32:0/3443969063 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f10080692a0 0x7f100806b750 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f101800b900 tx=0x7f1018009e30 comp rx=0 tx=0).stop 2024-04-12T05:37:10.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.037+0000 7f1024459700 1 -- 172.21.15.32:0/3443969063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f101c0fdfe0 msgr2=0x7f101c070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:10.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.037+0000 7f1024459700 1 --2- 172.21.15.32:0/3443969063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f101c0fdfe0 0x7f101c070310 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f100c00b790 tx=0x7f100c009250 comp rx=0 tx=0).stop 2024-04-12T05:37:10.035 INFO:tasks.workunit.client.0.smithi032.stdout:1/565: rename d5/de/d2a/f3f to d5/de/d17/d35/d78/fb1 0 2024-04-12T05:37:10.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.037+0000 7f1024459700 1 -- 172.21.15.32:0/3443969063 shutdown_connections 2024-04-12T05:37:10.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.037+0000 7f1024459700 1 --2- 172.21.15.32:0/3443969063 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f10080692a0 0x7f100806b750 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:10.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.037+0000 7f1024459700 1 --2- 172.21.15.32:0/3443969063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f101c0fce70 0x7f101c071c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:10.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.037+0000 7f1024459700 1 --2- 172.21.15.32:0/3443969063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f101c0fdfe0 0x7f101c070310 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:10.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.037+0000 7f1024459700 1 -- 172.21.15.32:0/3443969063 >> 172.21.15.32:0/3443969063 conn(0x7f101c0f8550 msgr2=0x7f101c0f9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:10.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.037+0000 7f1024459700 1 -- 172.21.15.32:0/3443969063 shutdown_connections 2024-04-12T05:37:10.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.037+0000 7f1024459700 1 -- 172.21.15.32:0/3443969063 wait complete. 2024-04-12T05:37:10.038 INFO:tasks.workunit.client.0.smithi032.stdout:6/804: truncate d4/d21/f3b 4151607 0 2024-04-12T05:37:10.040 INFO:tasks.workunit.client.0.smithi032.stdout:2/691: dwrite dd/d19/d9d/d42/f84 [0,4194304] 0 2024-04-12T05:37:10.041 INFO:tasks.workunit.client.1.smithi067.stdout:2/59: dread d5/da/de/f13 [0,4194304] 0 2024-04-12T05:37:10.041 INFO:tasks.workunit.client.1.smithi067.stdout:9/29: fsync d1/f6 0 2024-04-12T05:37:10.042 INFO:tasks.workunit.client.1.smithi067.stdout:9/30: write d1/f6 [278935,17699] 0 2024-04-12T05:37:10.043 INFO:tasks.workunit.client.1.smithi067.stdout:2/60: readlink d5/l6 0 2024-04-12T05:37:10.043 INFO:tasks.workunit.client.0.smithi032.stdout:0/677: rename d2/dc/d3e/dac/d51/d83/ld2 to d2/d29/d77/ldc 0 2024-04-12T05:37:10.043 INFO:tasks.workunit.client.0.smithi032.stdout:1/566: mknod d5/cb2 0 2024-04-12T05:37:10.044 INFO:tasks.workunit.client.0.smithi032.stdout:0/678: readlink d2/dc/d3e/dac/d51/l7d 0 2024-04-12T05:37:10.045 INFO:tasks.workunit.client.0.smithi032.stdout:2/692: creat dd/d8c/fe2 x:0 0 0 2024-04-12T05:37:10.046 INFO:tasks.workunit.client.0.smithi032.stdout:6/805: write d4/de0/de7/ff8 [756725,111838] 0 2024-04-12T05:37:10.047 INFO:tasks.workunit.client.0.smithi032.stdout:1/567: creat d5/de/d17/d35/d73/fb3 x:0 0 0 2024-04-12T05:37:10.047 INFO:tasks.workunit.client.0.smithi032.stdout:2/693: chown dd/d19/d20/fa0 12 1 2024-04-12T05:37:10.048 INFO:tasks.workunit.client.1.smithi067.stdout:2/61: link d5/l6 d5/d11/l15 0 2024-04-12T05:37:10.049 INFO:tasks.workunit.client.1.smithi067.stdout:2/62: chown d5/da/de 2095 1 2024-04-12T05:37:10.051 INFO:tasks.workunit.client.1.smithi067.stdout:2/63: chown d5/c8 7291 1 2024-04-12T05:37:10.051 INFO:tasks.workunit.client.0.smithi032.stdout:1/568: truncate d5/de/d2a/f89 6070247 0 2024-04-12T05:37:10.053 INFO:tasks.workunit.client.0.smithi032.stdout:0/679: creat d2/d8/df/fdd x:0 0 0 2024-04-12T05:37:10.053 INFO:tasks.workunit.client.0.smithi032.stdout:1/569: write d5/de/d17/d35/fa1 [121406,129487] 0 2024-04-12T05:37:10.054 INFO:tasks.workunit.client.1.smithi067.stdout:9/31: dwrite d1/f6 [0,4194304] 0 2024-04-12T05:37:10.054 INFO:tasks.workunit.client.0.smithi032.stdout:6/806: getdents d4/d21/d27/dad/d55/df4 0 2024-04-12T05:37:10.055 INFO:tasks.workunit.client.0.smithi032.stdout:6/807: chown d4/de0/de7/dd9/c100 620964 1 2024-04-12T05:37:10.056 INFO:tasks.workunit.client.1.smithi067.stdout:2/64: unlink d5/da/de/ff 0 2024-04-12T05:37:10.057 INFO:tasks.workunit.client.0.smithi032.stdout:2/694: dread dd/d19/d9d/f73 [0,4194304] 0 2024-04-12T05:37:10.059 INFO:tasks.workunit.client.1.smithi067.stdout:9/32: dread d1/f6 [0,4194304] 0 2024-04-12T05:37:10.059 INFO:tasks.workunit.client.0.smithi032.stdout:6/808: mknod d4/de0/de7/d92/d9e/c11a 0 2024-04-12T05:37:10.060 INFO:tasks.workunit.client.1.smithi067.stdout:2/65: unlink d5/da/fd 0 2024-04-12T05:37:10.061 INFO:tasks.workunit.client.0.smithi032.stdout:6/809: write d4/d21/f7d [1314038,66194] 0 2024-04-12T05:37:10.061 INFO:tasks.workunit.client.1.smithi067.stdout:2/66: read f4 [536621,112963] 0 2024-04-12T05:37:10.061 INFO:tasks.workunit.client.0.smithi032.stdout:0/680: rename d2/dc/d3e/dac/c96 to d2/d29/d8a/d99/cde 0 2024-04-12T05:37:10.063 INFO:tasks.workunit.client.0.smithi032.stdout:0/681: dread - d2/d29/f9a zero size 2024-04-12T05:37:10.064 INFO:tasks.workunit.client.1.smithi067.stdout:9/33: creat d1/f7 x:0 0 0 2024-04-12T05:37:10.066 INFO:tasks.workunit.client.0.smithi032.stdout:2/695: creat dd/d19/d9d/fe3 x:0 0 0 2024-04-12T05:37:10.067 INFO:tasks.workunit.client.1.smithi067.stdout:2/67: symlink d5/da/de/l16 0 2024-04-12T05:37:10.068 INFO:tasks.workunit.client.0.smithi032.stdout:6/810: creat d4/de0/deb/dec/f11b x:0 0 0 2024-04-12T05:37:10.069 INFO:tasks.workunit.client.1.smithi067.stdout:9/34: symlink d1/l8 0 2024-04-12T05:37:10.070 INFO:tasks.workunit.client.1.smithi067.stdout:9/35: write d1/f2 [4495026,105439] 0 2024-04-12T05:37:10.070 INFO:tasks.workunit.client.1.smithi067.stdout:9/36: fdatasync d1/f7 0 2024-04-12T05:37:10.071 INFO:tasks.workunit.client.0.smithi032.stdout:0/682: creat d2/d29/fdf x:0 0 0 2024-04-12T05:37:10.072 INFO:tasks.workunit.client.1.smithi067.stdout:9/37: write d1/f7 [204568,39523] 0 2024-04-12T05:37:10.072 INFO:tasks.workunit.client.1.smithi067.stdout:9/38: stat d1/l5 0 2024-04-12T05:37:10.073 INFO:tasks.workunit.client.1.smithi067.stdout:2/68: link d5/da/fc d5/da/f17 0 2024-04-12T05:37:10.073 INFO:tasks.workunit.client.0.smithi032.stdout:2/696: truncate dd/d19/d9d/d1c/d64/f7d 577352 0 2024-04-12T05:37:10.074 INFO:tasks.workunit.client.0.smithi032.stdout:1/570: mkdir d5/de/d17/d35/d3e/db4 0 2024-04-12T05:37:10.074 INFO:tasks.workunit.client.1.smithi067.stdout:9/39: creat d1/f9 x:0 0 0 2024-04-12T05:37:10.075 INFO:tasks.workunit.client.1.smithi067.stdout:2/69: mkdir d5/da/de/d18 0 2024-04-12T05:37:10.075 INFO:tasks.workunit.client.0.smithi032.stdout:6/811: symlink d4/d21/d27/dad/d55/df4/l11c 0 2024-04-12T05:37:10.075 INFO:tasks.workunit.client.0.smithi032.stdout:1/571: fdatasync d5/de/d17/fae 0 2024-04-12T05:37:10.077 INFO:tasks.workunit.client.0.smithi032.stdout:6/812: write d4/de0/de7/d37/f98 [8013667,4172] 0 2024-04-12T05:37:10.078 INFO:tasks.workunit.client.1.smithi067.stdout:2/70: mknod d5/d11/c19 0 2024-04-12T05:37:10.078 INFO:tasks.workunit.client.0.smithi032.stdout:2/697: creat dd/d19/d9d/d1c/d64/d32/d3a/d8d/fe4 x:0 0 0 2024-04-12T05:37:10.080 INFO:tasks.workunit.client.1.smithi067.stdout:9/40: dread d1/f2 [0,4194304] 0 2024-04-12T05:37:10.080 INFO:tasks.workunit.client.0.smithi032.stdout:6/813: write d4/d21/d27/dad/d55/d99/f106 [178675,25013] 0 2024-04-12T05:37:10.081 INFO:tasks.workunit.client.1.smithi067.stdout:2/71: dread d5/da/de/f13 [0,4194304] 0 2024-04-12T05:37:10.084 INFO:tasks.workunit.client.1.smithi067.stdout:9/41: rename d1/f9 to d1/fa 0 2024-04-12T05:37:10.084 INFO:tasks.workunit.client.0.smithi032.stdout:2/698: creat dd/d19/d9d/d1c/d64/d32/d6d/fe5 x:0 0 0 2024-04-12T05:37:10.085 INFO:tasks.workunit.client.1.smithi067.stdout:9/42: write d1/f7 [1292105,66143] 0 2024-04-12T05:37:10.085 INFO:tasks.workunit.client.1.smithi067.stdout:2/72: rename f3 to d5/da/de/d14/f1a 0 2024-04-12T05:37:10.086 INFO:tasks.workunit.client.1.smithi067.stdout:9/43: write d1/fa [760020,117682] 0 2024-04-12T05:37:10.087 INFO:tasks.workunit.client.0.smithi032.stdout:6/814: creat d4/de0/de7/d79/dc6/f11d x:0 0 0 2024-04-12T05:37:10.087 INFO:tasks.workunit.client.0.smithi032.stdout:6/815: chown d4/f2c 84884 1 2024-04-12T05:37:10.088 INFO:tasks.workunit.client.1.smithi067.stdout:2/73: stat d5/l6 0 2024-04-12T05:37:10.089 INFO:tasks.workunit.client.0.smithi032.stdout:6/816: chown d4/d21/d2e/d32/f61 8 1 2024-04-12T05:37:10.089 INFO:tasks.workunit.client.1.smithi067.stdout:2/74: stat d5/l9 0 2024-04-12T05:37:10.090 INFO:tasks.workunit.client.0.smithi032.stdout:6/817: write d4/d14/d3c/f41 [141740,79479] 0 2024-04-12T05:37:10.092 INFO:tasks.workunit.client.1.smithi067.stdout:2/75: unlink d5/da/fc 0 2024-04-12T05:37:10.094 INFO:tasks.workunit.client.0.smithi032.stdout:6/818: fsync d4/d21/d27/dad/ffc 0 2024-04-12T05:37:10.095 INFO:tasks.workunit.client.0.smithi032.stdout:0/683: dread d2/d28/d87/f43 [0,4194304] 0 2024-04-12T05:37:10.096 INFO:tasks.workunit.client.1.smithi067.stdout:2/76: fdatasync f4 0 2024-04-12T05:37:10.098 INFO:tasks.workunit.client.0.smithi032.stdout:0/684: mknod d2/d8/d48/ce0 0 2024-04-12T05:37:10.099 INFO:tasks.workunit.client.1.smithi067.stdout:2/77: mknod d5/d11/c1b 0 2024-04-12T05:37:10.101 INFO:tasks.workunit.client.1.smithi067.stdout:2/78: unlink d5/cb 0 2024-04-12T05:37:10.102 INFO:tasks.workunit.client.0.smithi032.stdout:0/685: getdents d2/d8/df/d24/d60 0 2024-04-12T05:37:10.104 INFO:tasks.workunit.client.0.smithi032.stdout:0/686: dread - d2/d8/d17/fc4 zero size 2024-04-12T05:37:10.105 INFO:tasks.workunit.client.0.smithi032.stdout:6/819: dwrite d4/de0/de7/d37/f5d [0,4194304] 0 2024-04-12T05:37:10.105 INFO:tasks.workunit.client.1.smithi067.stdout:2/79: symlink d5/da/de/l1c 0 2024-04-12T05:37:10.105 INFO:tasks.workunit.client.1.smithi067.stdout:2/80: dread - d5/f12 zero size 2024-04-12T05:37:10.105 INFO:tasks.workunit.client.1.smithi067.stdout:9/44: rmdir d1 39 2024-04-12T05:37:10.106 INFO:tasks.workunit.client.1.smithi067.stdout:2/81: truncate d5/f12 783821 0 2024-04-12T05:37:10.106 INFO:tasks.workunit.client.0.smithi032.stdout:0/687: write d2/d29/f70 [2992918,44645] 0 2024-04-12T05:37:10.107 INFO:tasks.workunit.client.1.smithi067.stdout:9/45: write d1/fa [950655,114993] 0 2024-04-12T05:37:10.111 INFO:tasks.workunit.client.1.smithi067.stdout:2/82: write d5/da/f17 [1264819,104903] 0 2024-04-12T05:37:10.113 INFO:tasks.workunit.client.0.smithi032.stdout:6/820: rename d4/c90 to d4/d21/c11e 0 2024-04-12T05:37:10.114 INFO:tasks.workunit.client.0.smithi032.stdout:6/821: stat d4/d21/d2e/c54 0 2024-04-12T05:37:10.114 INFO:tasks.workunit.client.1.smithi067.stdout:2/83: fdatasync d5/da/f17 0 2024-04-12T05:37:10.115 INFO:tasks.workunit.client.1.smithi067.stdout:2/84: chown f4 2 1 2024-04-12T05:37:10.116 INFO:tasks.workunit.client.0.smithi032.stdout:0/688: link d2/dc/d3e/f46 d2/d28/d87/d79/fe1 0 2024-04-12T05:37:10.116 INFO:tasks.workunit.client.1.smithi067.stdout:9/46: dread d1/fa [0,4194304] 0 2024-04-12T05:37:10.119 INFO:tasks.workunit.client.0.smithi032.stdout:0/689: mknod d2/d8/d66/ce2 0 2024-04-12T05:37:10.120 INFO:tasks.workunit.client.0.smithi032.stdout:3/680: sync 2024-04-12T05:37:10.121 INFO:tasks.workunit.client.0.smithi032.stdout:6/822: dwrite d4/de0/de7/dc4/f116 [0,4194304] 0 2024-04-12T05:37:10.121 INFO:tasks.workunit.client.1.smithi067.stdout:9/47: dread d1/f2 [0,4194304] 0 2024-04-12T05:37:10.121 INFO:tasks.workunit.client.1.smithi067.stdout:2/85: dread d5/da/de/d14/f1a [0,4194304] 0 2024-04-12T05:37:10.122 INFO:tasks.workunit.client.1.smithi067.stdout:8/55: sync 2024-04-12T05:37:10.122 INFO:tasks.workunit.client.1.smithi067.stdout:7/46: sync 2024-04-12T05:37:10.123 INFO:tasks.workunit.client.1.smithi067.stdout:8/56: readlink l3 0 2024-04-12T05:37:10.125 INFO:tasks.workunit.client.1.smithi067.stdout:9/48: symlink d1/lb 0 2024-04-12T05:37:10.125 INFO:tasks.workunit.client.1.smithi067.stdout:2/86: creat d5/da/f1d x:0 0 0 2024-04-12T05:37:10.126 INFO:tasks.workunit.client.1.smithi067.stdout:9/49: fdatasync d1/f7 0 2024-04-12T05:37:10.128 INFO:tasks.workunit.client.1.smithi067.stdout:7/47: creat d0/d2/ff x:0 0 0 2024-04-12T05:37:10.130 INFO:tasks.workunit.client.1.smithi067.stdout:2/87: creat d5/d11/f1e x:0 0 0 2024-04-12T05:37:10.130 INFO:tasks.workunit.client.1.smithi067.stdout:9/50: fsync d1/f2 0 2024-04-12T05:37:10.130 INFO:tasks.workunit.client.1.smithi067.stdout:8/57: dread f5 [0,4194304] 0 2024-04-12T05:37:10.131 INFO:tasks.workunit.client.1.smithi067.stdout:9/51: chown d1 2 1 2024-04-12T05:37:10.131 INFO:tasks.workunit.client.1.smithi067.stdout:2/88: write f4 [307999,19074] 0 2024-04-12T05:37:10.132 INFO:tasks.workunit.client.1.smithi067.stdout:9/52: readlink d1/l8 0 2024-04-12T05:37:10.135 INFO:tasks.workunit.client.1.smithi067.stdout:9/53: mknod d1/cc 0 2024-04-12T05:37:10.137 INFO:tasks.workunit.client.1.smithi067.stdout:2/89: dread d5/da/f17 [0,4194304] 0 2024-04-12T05:37:10.140 INFO:tasks.workunit.client.1.smithi067.stdout:2/90: dread d5/f12 [0,4194304] 0 2024-04-12T05:37:10.140 INFO:tasks.workunit.client.1.smithi067.stdout:9/54: truncate d1/f2 975275 0 2024-04-12T05:37:10.140 INFO:tasks.workunit.client.1.smithi067.stdout:2/91: rename d5 to d5/da/d1f 22 2024-04-12T05:37:10.141 INFO:tasks.workunit.client.1.smithi067.stdout:2/92: fdatasync d5/da/f1d 0 2024-04-12T05:37:10.142 INFO:tasks.workunit.client.1.smithi067.stdout:2/93: read d5/da/de/f13 [115101,55433] 0 2024-04-12T05:37:10.144 INFO:tasks.workunit.client.1.smithi067.stdout:9/55: link d1/f6 d1/fd 0 2024-04-12T05:37:10.145 INFO:tasks.workunit.client.1.smithi067.stdout:2/94: rename d5/da/f17 to d5/f20 0 2024-04-12T05:37:10.147 INFO:tasks.workunit.client.1.smithi067.stdout:9/56: rename d1/l8 to d1/le 0 2024-04-12T05:37:10.147 INFO:tasks.workunit.client.1.smithi067.stdout:7/48: truncate d0/f9 3338516 0 2024-04-12T05:37:10.148 INFO:tasks.workunit.client.1.smithi067.stdout:2/95: symlink d5/l21 0 2024-04-12T05:37:10.148 INFO:tasks.workunit.client.1.smithi067.stdout:8/58: write f5 [1161109,23305] 0 2024-04-12T05:37:10.149 INFO:tasks.workunit.client.1.smithi067.stdout:9/57: symlink d1/lf 0 2024-04-12T05:37:10.153 INFO:tasks.workunit.client.1.smithi067.stdout:8/59: dread f5 [0,4194304] 0 2024-04-12T05:37:10.153 INFO:tasks.workunit.client.1.smithi067.stdout:2/96: symlink d5/da/de/d14/l22 0 2024-04-12T05:37:10.153 INFO:tasks.workunit.client.1.smithi067.stdout:7/49: mkdir d0/d2/d5/db/d10 0 2024-04-12T05:37:10.154 INFO:tasks.workunit.client.1.smithi067.stdout:8/60: chown d6/d8 1753508 1 2024-04-12T05:37:10.154 INFO:tasks.workunit.client.1.smithi067.stdout:2/97: truncate f4 2185691 0 2024-04-12T05:37:10.156 INFO:tasks.workunit.client.1.smithi067.stdout:2/98: write d5/da/de/f13 [618256,70665] 0 2024-04-12T05:37:10.158 INFO:tasks.workunit.client.1.smithi067.stdout:7/50: fdatasync d0/d2/d5/fe 0 2024-04-12T05:37:10.162 INFO:tasks.workunit.client.1.smithi067.stdout:2/99: mkdir d5/d23 0 2024-04-12T05:37:10.169 INFO:tasks.workunit.client.1.smithi067.stdout:7/51: dread d0/f9 [0,4194304] 0 2024-04-12T05:37:10.171 INFO:tasks.workunit.client.1.smithi067.stdout:7/52: unlink d0/d2/f7 0 2024-04-12T05:37:10.174 INFO:tasks.workunit.client.1.smithi067.stdout:9/58: write d1/f2 [1742513,116387] 0 2024-04-12T05:37:10.175 INFO:tasks.workunit.client.1.smithi067.stdout:7/53: write d0/f1 [489145,85374] 0 2024-04-12T05:37:10.177 INFO:tasks.workunit.client.1.smithi067.stdout:2/100: getdents d5 0 2024-04-12T05:37:10.177 INFO:tasks.workunit.client.1.smithi067.stdout:7/54: write d0/f1 [3288487,56207] 0 2024-04-12T05:37:10.178 INFO:tasks.workunit.client.1.smithi067.stdout:9/59: mknod d1/c10 0 2024-04-12T05:37:10.178 INFO:tasks.workunit.client.1.smithi067.stdout:9/60: stat d1/fd 0 2024-04-12T05:37:10.180 INFO:tasks.workunit.client.1.smithi067.stdout:9/61: write d1/fa [920643,125637] 0 2024-04-12T05:37:10.182 INFO:tasks.workunit.client.1.smithi067.stdout:2/101: fsync d5/da/de/d14/f1a 0 2024-04-12T05:37:10.184 INFO:tasks.workunit.client.1.smithi067.stdout:2/102: chown d5/da/de/l16 207163 1 2024-04-12T05:37:10.185 INFO:tasks.workunit.client.1.smithi067.stdout:9/62: truncate d1/fd 4138764 0 2024-04-12T05:37:10.186 INFO:tasks.workunit.client.1.smithi067.stdout:7/55: dwrite d0/d2/ff [0,4194304] 0 2024-04-12T05:37:10.188 INFO:tasks.workunit.client.1.smithi067.stdout:2/103: creat d5/da/de/d14/f24 x:0 0 0 2024-04-12T05:37:10.191 INFO:tasks.workunit.client.1.smithi067.stdout:7/56: dread d0/f1 [0,4194304] 0 2024-04-12T05:37:10.192 INFO:tasks.workunit.client.1.smithi067.stdout:2/104: symlink d5/da/de/d14/l25 0 2024-04-12T05:37:10.196 INFO:tasks.workunit.client.1.smithi067.stdout:7/57: creat d0/d2/d5/f11 x:0 0 0 2024-04-12T05:37:10.196 INFO:tasks.workunit.client.1.smithi067.stdout:7/58: stat d0 0 2024-04-12T05:37:10.197 INFO:tasks.workunit.client.1.smithi067.stdout:2/105: creat d5/da/de/f26 x:0 0 0 2024-04-12T05:37:10.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.205+0000 7f2ae504b700 1 -- 172.21.15.32:0/1781027053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ae00fcfb0 msgr2=0x7f2ae00fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:10.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.205+0000 7f2ae504b700 1 --2- 172.21.15.32:0/1781027053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ae00fcfb0 0x7f2ae00fd420 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2ad00049c0 tx=0x7f2ad0009e30 comp rx=0 tx=0).stop 2024-04-12T05:37:10.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.205+0000 7f2ae504b700 1 -- 172.21.15.32:0/1781027053 shutdown_connections 2024-04-12T05:37:10.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.205+0000 7f2ae504b700 1 --2- 172.21.15.32:0/1781027053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ae00fcfb0 0x7f2ae00fd420 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:10.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.205+0000 7f2ae504b700 1 --2- 172.21.15.32:0/1781027053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae00fbd00 0x7f2ae00fc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:10.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.205+0000 7f2ae504b700 1 -- 172.21.15.32:0/1781027053 >> 172.21.15.32:0/1781027053 conn(0x7f2ae00f7480 msgr2=0x7f2ae00f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:10.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.205+0000 7f2ae504b700 1 -- 172.21.15.32:0/1781027053 shutdown_connections 2024-04-12T05:37:10.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.205+0000 7f2ae504b700 1 -- 172.21.15.32:0/1781027053 wait complete. 2024-04-12T05:37:10.205 INFO:tasks.workunit.client.1.smithi067.stdout:2/106: rename d5/da/de/l16 to d5/d11/l27 0 2024-04-12T05:37:10.205 INFO:tasks.workunit.client.1.smithi067.stdout:7/59: dwrite d0/d2/ff [0,4194304] 0 2024-04-12T05:37:10.205 INFO:tasks.workunit.client.1.smithi067.stdout:2/107: write d5/da/f1d [137796,47493] 0 2024-04-12T05:37:10.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.206+0000 7f2ae504b700 1 Processor -- start 2024-04-12T05:37:10.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.206+0000 7f2ae504b700 1 -- start start 2024-04-12T05:37:10.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.207+0000 7f2ae504b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae00fbd00 0x7f2ae0102d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:37:10.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.207+0000 7f2ae504b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ae00fcfb0 0x7f2ae01032d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:37:10.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.207+0000 7f2ae504b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae0103920 con 0x7f2ae00fcfb0 2024-04-12T05:37:10.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.207+0000 7f2ae504b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae0103a90 con 0x7f2ae00fbd00 2024-04-12T05:37:10.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.207+0000 7f2ade59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ae00fcfb0 0x7f2ae01032d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:10.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.207+0000 7f2ade59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ae00fcfb0 0x7f2ae01032d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56588/0 (socket says 172.21.15.32:56588) 2024-04-12T05:37:10.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.207+0000 7f2ade59c700 1 -- 172.21.15.32:0/2047238162 learned_addr learned my addr 172.21.15.32:0/2047238162 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:37:10.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.207+0000 7f2ade59c700 1 -- 172.21.15.32:0/2047238162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae00fbd00 msgr2=0x7f2ae0102d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:10.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.207+0000 7f2ade59c700 1 --2- 172.21.15.32:0/2047238162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae00fbd00 0x7f2ae0102d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:10.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.207+0000 7f2ade59c700 1 -- 172.21.15.32:0/2047238162 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad0005040 con 0x7f2ae00fcfb0 2024-04-12T05:37:10.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.208+0000 7f2ade59c700 1 --2- 172.21.15.32:0/2047238162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ae00fcfb0 0x7f2ae01032d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2ad0009040 tx=0x7f2ad0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:10.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.208+0000 7f2ac77fe700 1 -- 172.21.15.32:0/2047238162 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ad001cac0 con 0x7f2ae00fcfb0 2024-04-12T05:37:10.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.208+0000 7f2ae504b700 1 -- 172.21.15.32:0/2047238162 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ae012a470 con 0x7f2ae00fcfb0 2024-04-12T05:37:10.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.208+0000 7f2ac77fe700 1 -- 172.21.15.32:0/2047238162 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ad001cc20 con 0x7f2ae00fcfb0 2024-04-12T05:37:10.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.208+0000 7f2ac77fe700 1 -- 172.21.15.32:0/2047238162 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ad0016660 con 0x7f2ae00fcfb0 2024-04-12T05:37:10.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.208+0000 7f2ae504b700 1 -- 172.21.15.32:0/2047238162 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ae012a9a0 con 0x7f2ae00fcfb0 2024-04-12T05:37:10.207 INFO:tasks.workunit.client.1.smithi067.stdout:9/63: dread d1/fd [0,4194304] 0 2024-04-12T05:37:10.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.210+0000 7f2ae504b700 1 -- 172.21.15.32:0/2047238162 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ae006ea00 con 0x7f2ae00fcfb0 2024-04-12T05:37:10.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.210+0000 7f2ac77fe700 1 -- 172.21.15.32:0/2047238162 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f2ad0021030 con 0x7f2ae00fcfb0 2024-04-12T05:37:10.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.211+0000 7f2ac77fe700 1 --2- 172.21.15.32:0/2047238162 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f2ac806b4b0 0x7f2ac806d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:37:10.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.211+0000 7f2ac77fe700 1 -- 172.21.15.32:0/2047238162 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f2ad0088710 con 0x7f2ae00fcfb0 2024-04-12T05:37:10.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.211+0000 7f2aded9d700 1 --2- 172.21.15.32:0/2047238162 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f2ac806b4b0 0x7f2ac806d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:10.209 INFO:tasks.workunit.client.1.smithi067.stdout:2/108: creat d5/da/de/f28 x:0 0 0 2024-04-12T05:37:10.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.212+0000 7f2aded9d700 1 --2- 172.21.15.32:0/2047238162 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f2ac806b4b0 0x7f2ac806d960 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2ae0104290 tx=0x7f2ad8007b30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:10.210 INFO:tasks.workunit.client.1.smithi067.stdout:9/64: creat d1/f11 x:0 0 0 2024-04-12T05:37:10.211 INFO:tasks.workunit.client.1.smithi067.stdout:9/65: chown d1/f6 18229 1 2024-04-12T05:37:10.215 INFO:tasks.workunit.client.1.smithi067.stdout:7/60: creat d0/f12 x:0 0 0 2024-04-12T05:37:10.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.219+0000 7f2ac77fe700 1 -- 172.21.15.32:0/2047238162 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f2ad00549f0 con 0x7f2ae00fcfb0 2024-04-12T05:37:10.218 INFO:tasks.workunit.client.1.smithi067.stdout:2/109: dwrite d5/da/de/f26 [0,4194304] 0 2024-04-12T05:37:10.220 INFO:tasks.workunit.client.1.smithi067.stdout:2/110: write d5/da/de/d14/f24 [383106,37738] 0 2024-04-12T05:37:10.221 INFO:tasks.workunit.client.1.smithi067.stdout:9/66: dwrite d1/f11 [0,4194304] 0 2024-04-12T05:37:10.222 INFO:tasks.workunit.client.0.smithi032.stdout:3/681: dwrite d12/d2e/d3a/f96 [0,4194304] 0 2024-04-12T05:37:10.222 INFO:tasks.workunit.client.0.smithi032.stdout:0/690: dwrite d2/d28/d87/f71 [0,4194304] 0 2024-04-12T05:37:10.222 INFO:tasks.workunit.client.0.smithi032.stdout:0/691: readlink d2/dc/d41/l44 0 2024-04-12T05:37:10.222 INFO:tasks.workunit.client.0.smithi032.stdout:3/682: creat d12/d2f/fdc x:0 0 0 2024-04-12T05:37:10.223 INFO:tasks.workunit.client.1.smithi067.stdout:9/67: truncate d1/f11 4593702 0 2024-04-12T05:37:10.224 INFO:tasks.workunit.client.0.smithi032.stdout:0/692: write d2/d28/d87/f71 [3254632,66829] 0 2024-04-12T05:37:10.231 INFO:tasks.workunit.client.1.smithi067.stdout:7/61: creat d0/d2/f13 x:0 0 0 2024-04-12T05:37:10.232 INFO:tasks.workunit.client.1.smithi067.stdout:9/68: unlink d1/lf 0 2024-04-12T05:37:10.232 INFO:tasks.workunit.client.1.smithi067.stdout:2/111: creat d5/d11/f29 x:0 0 0 2024-04-12T05:37:10.232 INFO:tasks.workunit.client.0.smithi032.stdout:6/823: dwrite d4/de0/de7/d37/f98 [4194304,4194304] 0 2024-04-12T05:37:10.233 INFO:tasks.workunit.client.0.smithi032.stdout:0/693: write d2/d29/d8a/d99/fbf [146421,30211] 0 2024-04-12T05:37:10.237 INFO:tasks.workunit.client.0.smithi032.stdout:0/694: stat d2/d8/d17/fd8 0 2024-04-12T05:37:10.238 INFO:tasks.workunit.client.1.smithi067.stdout:2/112: creat d5/d11/f2a x:0 0 0 2024-04-12T05:37:10.239 INFO:tasks.workunit.client.1.smithi067.stdout:7/62: mknod d0/d2/d5/db/d10/c14 0 2024-04-12T05:37:10.242 INFO:tasks.workunit.client.1.smithi067.stdout:9/69: dwrite d1/f6 [0,4194304] 0 2024-04-12T05:37:10.244 INFO:tasks.workunit.client.0.smithi032.stdout:6/824: dread d4/ff [0,4194304] 0 2024-04-12T05:37:10.244 INFO:tasks.workunit.client.1.smithi067.stdout:7/63: symlink d0/d2/l15 0 2024-04-12T05:37:10.245 INFO:tasks.workunit.client.1.smithi067.stdout:2/113: dread d5/f20 [0,4194304] 0 2024-04-12T05:37:10.246 INFO:tasks.workunit.client.1.smithi067.stdout:2/114: dread - d5/d11/f29 zero size 2024-04-12T05:37:10.247 INFO:tasks.workunit.client.0.smithi032.stdout:0/695: getdents d2/d28/dd6 0 2024-04-12T05:37:10.248 INFO:tasks.workunit.client.1.smithi067.stdout:9/70: mkdir d1/d12 0 2024-04-12T05:37:10.249 INFO:tasks.workunit.client.1.smithi067.stdout:7/64: fsync d0/f1 0 2024-04-12T05:37:10.249 INFO:tasks.workunit.client.0.smithi032.stdout:0/696: write d2/d28/d87/f71 [5054498,69332] 0 2024-04-12T05:37:10.252 INFO:tasks.workunit.client.1.smithi067.stdout:9/71: stat d1/le 0 2024-04-12T05:37:10.252 INFO:tasks.workunit.client.0.smithi032.stdout:6/825: mknod d4/d21/d2e/dca/c11f 0 2024-04-12T05:37:10.256 INFO:tasks.workunit.client.0.smithi032.stdout:3/683: dread d12/d2f/f41 [0,4194304] 0 2024-04-12T05:37:10.256 INFO:tasks.workunit.client.0.smithi032.stdout:0/697: mknod d2/dc/d3e/dac/d51/ce3 0 2024-04-12T05:37:10.257 INFO:tasks.workunit.client.0.smithi032.stdout:6/826: symlink d4/d21/d27/dad/d55/d59/l120 0 2024-04-12T05:37:10.257 INFO:tasks.workunit.client.1.smithi067.stdout:9/72: dread d1/f11 [0,4194304] 0 2024-04-12T05:37:10.259 INFO:tasks.workunit.client.1.smithi067.stdout:9/73: write d1/fd [2770977,98678] 0 2024-04-12T05:37:10.262 INFO:tasks.workunit.client.0.smithi032.stdout:0/698: symlink d2/le4 0 2024-04-12T05:37:10.264 INFO:tasks.workunit.client.0.smithi032.stdout:6/827: fsync d4/d21/d27/dad/d55/f70 0 2024-04-12T05:37:10.266 INFO:tasks.workunit.client.0.smithi032.stdout:0/699: unlink d2/d28/f2e 0 2024-04-12T05:37:10.267 INFO:tasks.workunit.client.0.smithi032.stdout:6/828: stat d4/de0/de7/d79/dc6/f11d 0 2024-04-12T05:37:10.267 INFO:tasks.workunit.client.0.smithi032.stdout:7/687: sync 2024-04-12T05:37:10.267 INFO:tasks.workunit.client.0.smithi032.stdout:8/693: sync 2024-04-12T05:37:10.267 INFO:tasks.workunit.client.0.smithi032.stdout:3/684: dwrite d12/d2e/d3a/d63/da6/dac/faf [0,4194304] 0 2024-04-12T05:37:10.268 INFO:tasks.workunit.client.0.smithi032.stdout:7/688: write d0/de/f15 [4943748,20363] 0 2024-04-12T05:37:10.272 INFO:tasks.workunit.client.1.smithi067.stdout:9/74: mkdir d1/d13 0 2024-04-12T05:37:10.274 INFO:tasks.workunit.client.0.smithi032.stdout:7/689: rmdir d0/d10/d70/d7b 39 2024-04-12T05:37:10.278 INFO:tasks.workunit.client.1.smithi067.stdout:4/21: sync 2024-04-12T05:37:10.278 INFO:tasks.workunit.client.1.smithi067.stdout:3/63: sync 2024-04-12T05:37:10.278 INFO:tasks.workunit.client.0.smithi032.stdout:6/829: mkdir d4/de0/de7/dc4/d121 0 2024-04-12T05:37:10.279 INFO:tasks.workunit.client.1.smithi067.stdout:1/105: sync 2024-04-12T05:37:10.283 INFO:tasks.workunit.client.0.smithi032.stdout:8/694: mknod d0/d36/d84/cf8 0 2024-04-12T05:37:10.284 INFO:tasks.workunit.client.1.smithi067.stdout:7/65: getdents d0/d2 0 2024-04-12T05:37:10.285 INFO:tasks.workunit.client.1.smithi067.stdout:2/115: write d5/da/de/d14/f1a [2702014,27737] 0 2024-04-12T05:37:10.285 INFO:tasks.workunit.client.1.smithi067.stdout:9/75: creat d1/d12/f14 x:0 0 0 2024-04-12T05:37:10.285 INFO:tasks.workunit.client.1.smithi067.stdout:4/22: creat f4 x:0 0 0 2024-04-12T05:37:10.285 INFO:tasks.workunit.client.1.smithi067.stdout:3/64: dread f5 [4194304,4194304] 0 2024-04-12T05:37:10.286 INFO:tasks.workunit.client.1.smithi067.stdout:3/65: read f5 [7689194,111527] 0 2024-04-12T05:37:10.289 INFO:tasks.workunit.client.0.smithi032.stdout:7/690: mkdir d0/d34/dbe/dbb/de8 0 2024-04-12T05:37:10.289 INFO:tasks.workunit.client.1.smithi067.stdout:4/23: mknod c5 0 2024-04-12T05:37:10.290 INFO:tasks.workunit.client.1.smithi067.stdout:2/116: readlink d5/d11/l27 0 2024-04-12T05:37:10.290 INFO:tasks.workunit.client.0.smithi032.stdout:8/695: chown d0/f2f 37566155 1 2024-04-12T05:37:10.290 INFO:tasks.workunit.client.0.smithi032.stdout:0/700: dwrite d2/d28/d87/f43 [0,4194304] 0 2024-04-12T05:37:10.291 INFO:tasks.workunit.client.1.smithi067.stdout:9/76: mkdir d1/d12/d15 0 2024-04-12T05:37:10.291 INFO:tasks.workunit.client.1.smithi067.stdout:3/66: readlink l6 0 2024-04-12T05:37:10.292 INFO:tasks.workunit.client.1.smithi067.stdout:1/106: link d4/f1d d4/d9/d14/f21 0 2024-04-12T05:37:10.292 INFO:tasks.workunit.client.1.smithi067.stdout:7/66: mknod d0/d2/dd/c16 0 2024-04-12T05:37:10.293 INFO:tasks.workunit.client.0.smithi032.stdout:3/685: dwrite d12/d2e/f4a [0,4194304] 0 2024-04-12T05:37:10.294 INFO:tasks.workunit.client.1.smithi067.stdout:4/24: creat f6 x:0 0 0 2024-04-12T05:37:10.295 INFO:tasks.workunit.client.1.smithi067.stdout:7/67: dread - d0/d2/f13 zero size 2024-04-12T05:37:10.296 INFO:tasks.workunit.client.1.smithi067.stdout:9/77: mkdir d1/d12/d16 0 2024-04-12T05:37:10.296 INFO:tasks.workunit.client.1.smithi067.stdout:4/25: chown c5 42250 1 2024-04-12T05:37:10.297 INFO:tasks.workunit.client.1.smithi067.stdout:4/26: truncate f4 219016 0 2024-04-12T05:37:10.302 INFO:tasks.workunit.client.1.smithi067.stdout:3/67: mknod dc/df/c10 0 2024-04-12T05:37:10.303 INFO:tasks.workunit.client.0.smithi032.stdout:6/830: dread d4/d21/f2f [0,4194304] 0 2024-04-12T05:37:10.303 INFO:tasks.workunit.client.1.smithi067.stdout:6/40: sync 2024-04-12T05:37:10.303 INFO:tasks.workunit.client.1.smithi067.stdout:8/61: sync 2024-04-12T05:37:10.304 INFO:tasks.workunit.client.1.smithi067.stdout:5/171: sync 2024-04-12T05:37:10.304 INFO:tasks.workunit.client.1.smithi067.stdout:2/117: dwrite d5/da/de/d14/f24 [0,4194304] 0 2024-04-12T05:37:10.304 INFO:tasks.workunit.client.1.smithi067.stdout:4/27: creat f7 x:0 0 0 2024-04-12T05:37:10.308 INFO:tasks.workunit.client.1.smithi067.stdout:7/68: getdents d0/d2/d5/dc 0 2024-04-12T05:37:10.311 INFO:tasks.workunit.client.1.smithi067.stdout:3/68: mkdir dc/d11 0 2024-04-12T05:37:10.311 INFO:tasks.workunit.client.1.smithi067.stdout:0/165: sync 2024-04-12T05:37:10.312 INFO:tasks.workunit.client.1.smithi067.stdout:6/41: write f9 [62619,78775] 0 2024-04-12T05:37:10.316 INFO:tasks.workunit.client.1.smithi067.stdout:3/69: write f9 [1653895,3506] 0 2024-04-12T05:37:10.316 INFO:tasks.workunit.client.1.smithi067.stdout:5/172: rename d0/d6/d8 to d0/d3a 0 2024-04-12T05:37:10.318 INFO:tasks.workunit.client.1.smithi067.stdout:6/42: chown f0 52 1 2024-04-12T05:37:10.324 INFO:tasks.workunit.client.1.smithi067.stdout:0/166: creat dd/d12/f37 x:0 0 0 2024-04-12T05:37:10.327 INFO:tasks.workunit.client.1.smithi067.stdout:6/43: read f3 [2973372,100981] 0 2024-04-12T05:37:10.336 INFO:tasks.workunit.client.1.smithi067.stdout:2/118: dwrite d5/da/f1d [0,4194304] 0 2024-04-12T05:37:10.337 INFO:tasks.workunit.client.1.smithi067.stdout:8/62: dwrite d6/f7 [4194304,4194304] 0 2024-04-12T05:37:10.338 INFO:tasks.workunit.client.1.smithi067.stdout:4/28: getdents . 0 2024-04-12T05:37:10.338 INFO:tasks.workunit.client.1.smithi067.stdout:3/70: dwrite f9 [0,4194304] 0 2024-04-12T05:37:10.339 INFO:tasks.workunit.client.0.smithi032.stdout:7/691: mknod d0/dd/d51/ce9 0 2024-04-12T05:37:10.340 INFO:tasks.workunit.client.0.smithi032.stdout:3/686: creat d12/d2f/fdd x:0 0 0 2024-04-12T05:37:10.340 INFO:tasks.workunit.client.1.smithi067.stdout:5/173: dread d0/d6/f10 [0,4194304] 0 2024-04-12T05:37:10.348 INFO:tasks.workunit.client.0.smithi032.stdout:6/831: unlink d4/d21/c30 0 2024-04-12T05:37:10.348 INFO:tasks.workunit.client.0.smithi032.stdout:7/692: dread d0/de/df/d29/f89 [0,4194304] 0 2024-04-12T05:37:10.349 INFO:tasks.workunit.client.1.smithi067.stdout:6/44: creat fb x:0 0 0 2024-04-12T05:37:10.349 INFO:tasks.workunit.client.1.smithi067.stdout:1/107: truncate d4/d9/d12/f1b 991488 0 2024-04-12T05:37:10.349 INFO:tasks.workunit.client.1.smithi067.stdout:7/69: symlink d0/d2/d5/dc/l17 0 2024-04-12T05:37:10.349 INFO:tasks.workunit.client.1.smithi067.stdout:7/70: read - d0/d2/d5/f11 zero size 2024-04-12T05:37:10.350 INFO:tasks.workunit.client.0.smithi032.stdout:3/687: getdents d12/d2e/dcd 0 2024-04-12T05:37:10.355 INFO:tasks.workunit.client.0.smithi032.stdout:6/832: symlink d4/de0/de7/d92/d9e/db2/l122 0 2024-04-12T05:37:10.356 INFO:tasks.workunit.client.1.smithi067.stdout:3/71: dwrite f5 [4194304,4194304] 0 2024-04-12T05:37:10.359 INFO:tasks.workunit.client.0.smithi032.stdout:6/833: unlink d4/d21/c73 0 2024-04-12T05:37:10.360 INFO:tasks.workunit.client.1.smithi067.stdout:7/71: dread d0/d2/ff [0,4194304] 0 2024-04-12T05:37:10.360 INFO:tasks.workunit.client.1.smithi067.stdout:1/108: dread d4/f10 [0,4194304] 0 2024-04-12T05:37:10.361 INFO:tasks.workunit.client.0.smithi032.stdout:6/834: write d4/d14/d3c/fb3 [369717,124575] 0 2024-04-12T05:37:10.361 INFO:tasks.workunit.client.1.smithi067.stdout:1/109: stat d4/f10 0 2024-04-12T05:37:10.363 INFO:tasks.workunit.client.0.smithi032.stdout:6/835: chown d4/d21/f85 4 1 2024-04-12T05:37:10.365 INFO:tasks.workunit.client.0.smithi032.stdout:3/688: getdents d12/d2e 0 2024-04-12T05:37:10.366 INFO:tasks.workunit.client.1.smithi067.stdout:3/72: dread f1 [0,4194304] 0 2024-04-12T05:37:10.367 INFO:tasks.workunit.client.0.smithi032.stdout:3/689: mkdir d12/d24/dde 0 2024-04-12T05:37:10.374 INFO:tasks.workunit.client.0.smithi032.stdout:6/836: dwrite d4/de0/de7/ff8 [0,4194304] 0 2024-04-12T05:37:10.375 INFO:tasks.workunit.client.0.smithi032.stdout:8/696: dread d0/db/d39/f83 [4194304,4194304] 0 2024-04-12T05:37:10.376 INFO:tasks.workunit.client.0.smithi032.stdout:3/690: mknod d12/d2e/d3a/d63/dd5/cdf 0 2024-04-12T05:37:10.377 INFO:tasks.workunit.client.0.smithi032.stdout:3/691: readlink d12/d2f/d89/ld6 0 2024-04-12T05:37:10.377 INFO:tasks.workunit.client.0.smithi032.stdout:7/693: mkdir d0/dd/d51/dea 0 2024-04-12T05:37:10.379 INFO:tasks.workunit.client.0.smithi032.stdout:8/697: creat d0/d36/d84/dad/ff9 x:0 0 0 2024-04-12T05:37:10.382 INFO:tasks.workunit.client.0.smithi032.stdout:3/692: symlink d12/d2e/d3a/d63/le0 0 2024-04-12T05:37:10.390 INFO:tasks.workunit.client.0.smithi032.stdout:8/698: dwrite d0/f7 [0,4194304] 0 2024-04-12T05:37:10.391 INFO:tasks.workunit.client.0.smithi032.stdout:8/699: read - d0/db/dd/d11/d1b/d22/d35/ff6 zero size 2024-04-12T05:37:10.394 INFO:tasks.workunit.client.0.smithi032.stdout:8/700: stat d0/db/d7d/fb9 0 2024-04-12T05:37:10.396 INFO:tasks.workunit.client.0.smithi032.stdout:8/701: dread - d0/db/dd/d11/d1b/d96/fef zero size 2024-04-12T05:37:10.398 INFO:tasks.workunit.client.0.smithi032.stdout:7/694: write d0/d1c/d4c/fc7 [959285,110935] 0 2024-04-12T05:37:10.400 INFO:tasks.workunit.client.0.smithi032.stdout:8/702: creat d0/db/dd/d11/d1b/d5f/deb/ffa x:0 0 0 2024-04-12T05:37:10.404 INFO:tasks.workunit.client.0.smithi032.stdout:8/703: mkdir d0/db/d7d/dfb 0 2024-04-12T05:37:10.404 INFO:tasks.workunit.client.1.smithi067.stdout:0/167: creat dd/f38 x:0 0 0 2024-04-12T05:37:10.406 INFO:tasks.workunit.client.0.smithi032.stdout:8/704: symlink d0/d36/de2/lfc 0 2024-04-12T05:37:10.407 INFO:tasks.workunit.client.0.smithi032.stdout:8/705: chown d0/db/dd/d11/d1b/d22/d35 8078 1 2024-04-12T05:37:10.414 INFO:tasks.workunit.client.0.smithi032.stdout:7/695: dwrite d0/dd/f3b [0,4194304] 0 2024-04-12T05:37:10.415 INFO:tasks.workunit.client.0.smithi032.stdout:8/706: dwrite d0/db/d38/d63/fbd [0,4194304] 0 2024-04-12T05:37:10.418 INFO:tasks.workunit.client.0.smithi032.stdout:7/696: write d0/de/f44 [7890253,67798] 0 2024-04-12T05:37:10.419 INFO:tasks.workunit.client.0.smithi032.stdout:8/707: write d0/db/dd/d11/d1b/f78 [1011737,50745] 0 2024-04-12T05:37:10.424 INFO:tasks.workunit.client.0.smithi032.stdout:7/697: unlink d0/d10/d70/d7b/lb4 0 2024-04-12T05:37:10.428 INFO:tasks.workunit.client.1.smithi067.stdout:5/174: mknod d0/da/d2d/c3b 0 2024-04-12T05:37:10.429 INFO:tasks.workunit.client.1.smithi067.stdout:4/29: unlink f4 0 2024-04-12T05:37:10.431 INFO:tasks.workunit.client.1.smithi067.stdout:8/63: unlink f4 0 2024-04-12T05:37:10.435 INFO:tasks.workunit.client.1.smithi067.stdout:7/72: creat d0/d2/d5/db/f18 x:0 0 0 2024-04-12T05:37:10.437 INFO:tasks.workunit.client.0.smithi032.stdout:8/708: dread d0/db/dd/d11/d1b/ff5 [0,4194304] 0 2024-04-12T05:37:10.437 INFO:tasks.workunit.client.1.smithi067.stdout:0/168: rename f9 to dd/d12/d14/f39 0 2024-04-12T05:37:10.438 INFO:tasks.workunit.client.1.smithi067.stdout:2/119: dwrite f4 [0,4194304] 0 2024-04-12T05:37:10.438 INFO:tasks.workunit.client.0.smithi032.stdout:8/709: read - d0/d36/d84/db3/fe1 zero size 2024-04-12T05:37:10.441 INFO:tasks.workunit.client.1.smithi067.stdout:4/30: mkdir d8 0 2024-04-12T05:37:10.442 INFO:tasks.workunit.client.1.smithi067.stdout:7/73: truncate d0/d2/d5/f11 75943 0 2024-04-12T05:37:10.445 INFO:tasks.workunit.client.1.smithi067.stdout:7/74: chown d0/d2/d5 231456 1 2024-04-12T05:37:10.446 INFO:tasks.workunit.client.1.smithi067.stdout:1/110: getdents d4/d20 0 2024-04-12T05:37:10.446 INFO:tasks.workunit.client.1.smithi067.stdout:6/45: truncate f6 3230884 0 2024-04-12T05:37:10.446 INFO:tasks.workunit.client.1.smithi067.stdout:8/64: dwrite d6/d8/f9 [0,4194304] 0 2024-04-12T05:37:10.447 INFO:tasks.workunit.client.1.smithi067.stdout:1/111: chown d4/d9/f17 2028040 1 2024-04-12T05:37:10.450 INFO:tasks.workunit.client.0.smithi032.stdout:8/710: truncate d0/db/dd/d11/d1b/f6c 51186 0 2024-04-12T05:37:10.451 INFO:tasks.workunit.client.1.smithi067.stdout:2/120: symlink d5/l2b 0 2024-04-12T05:37:10.452 INFO:tasks.workunit.client.1.smithi067.stdout:2/121: chown d5/d11 0 1 2024-04-12T05:37:10.457 INFO:tasks.workunit.client.1.smithi067.stdout:9/78: sync 2024-04-12T05:37:10.460 INFO:tasks.workunit.client.0.smithi032.stdout:7/698: dread d0/d1c/f4e [0,4194304] 0 2024-04-12T05:37:10.460 INFO:tasks.workunit.client.1.smithi067.stdout:2/122: write d5/da/de/d14/f1a [1108598,20543] 0 2024-04-12T05:37:10.463 INFO:tasks.workunit.client.1.smithi067.stdout:9/79: write d1/f2 [1731226,119034] 0 2024-04-12T05:37:10.465 INFO:tasks.workunit.client.1.smithi067.stdout:6/46: dwrite f7 [0,4194304] 0 2024-04-12T05:37:10.469 INFO:tasks.workunit.client.1.smithi067.stdout:4/31: dwrite f6 [0,4194304] 0 2024-04-12T05:37:10.470 INFO:tasks.workunit.client.1.smithi067.stdout:4/32: fdatasync f7 0 2024-04-12T05:37:10.471 INFO:tasks.workunit.client.0.smithi032.stdout:7/699: rename d0/dd/d51/lde to d0/d1c/d71/leb 0 2024-04-12T05:37:10.480 INFO:tasks.workunit.client.1.smithi067.stdout:0/169: readlink dd/d12/d14/d1a/l31 0 2024-04-12T05:37:10.481 INFO:tasks.workunit.client.0.smithi032.stdout:8/711: dread d0/db/f55 [0,4194304] 0 2024-04-12T05:37:10.483 INFO:tasks.workunit.client.1.smithi067.stdout:3/73: rmdir dc/d11 0 2024-04-12T05:37:10.486 INFO:tasks.workunit.client.0.smithi032.stdout:8/712: write d0/db/f60 [5093390,33498] 0 2024-04-12T05:37:10.487 INFO:tasks.workunit.client.0.smithi032.stdout:8/713: chown d0/db/dd/d11/d1b/d22/d54/da8/fd9 76 1 2024-04-12T05:37:10.491 INFO:tasks.workunit.client.0.smithi032.stdout:8/714: getdents d0/d36/d84/dad/da4 0 2024-04-12T05:37:10.493 INFO:tasks.workunit.client.0.smithi032.stdout:8/715: mkdir d0/dfd 0 2024-04-12T05:37:10.495 INFO:tasks.workunit.client.0.smithi032.stdout:8/716: write d0/db/dd/d11/d1b/d5f/daf/fa2 [1010340,7943] 0 2024-04-12T05:37:10.504 INFO:tasks.workunit.client.0.smithi032.stdout:9/555: sync 2024-04-12T05:37:10.505 INFO:tasks.workunit.client.1.smithi067.stdout:7/75: mknod d0/d2/d5/dc/c19 0 2024-04-12T05:37:10.522 INFO:tasks.workunit.client.1.smithi067.stdout:7/76: dwrite d0/d2/d5/fe [0,4194304] 0 2024-04-12T05:37:10.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.548+0000 7f2ae504b700 1 -- 172.21.15.32:0/2047238162 --> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ae002b340 con 0x7f2ac806b4b0 2024-04-12T05:37:10.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.559+0000 7f2ac77fe700 1 -- 172.21.15.32:0/2047238162 <== mgr.14208 v2:172.21.15.32:6800/595990262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3344 (secure 0 0 0) 0x7f2ae002b340 con 0x7f2ac806b4b0 2024-04-12T05:37:10.558 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:37:10.559 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (6m) 2m ago 8m 27.0M - 0.25.0 c8568f914cd2 a5c616791a7e 2024-04-12T05:37:10.559 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (8m) 2m ago 8m 6656k - 17.2.7-827-gd3b5060a 21de9b7369ad a9cfc00c311b 2024-04-12T05:37:10.560 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (7m) 2m ago 7m 6651k - 17.2.7-827-gd3b5060a 21de9b7369ad 1f2eea4542d9 2024-04-12T05:37:10.560 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (6m) 2m ago 8m 87.0M - 9.4.7 954c08fa6188 8e5a7ebd77b5 2024-04-12T05:37:10.560 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (2m) 2m ago 2m 11.4M - 17.2.7-827-gd3b5060a 21de9b7369ad e7ada8e5259c 2024-04-12T05:37:10.560 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (2m) 2m ago 2m 14.0M - 17.2.7-827-gd3b5060a 21de9b7369ad 55124da9afdb 2024-04-12T05:37:10.560 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (2m) 2m ago 2m 13.3M - 17.2.7-827-gd3b5060a 21de9b7369ad 0c295d31a4a6 2024-04-12T05:37:10.560 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (2m) 2m ago 2m 16.8M - 17.2.7-827-gd3b5060a 21de9b7369ad 6eecd7deaac2 2024-04-12T05:37:10.560 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:9283 running (9m) 2m ago 9m 483M - 17.2.7-827-gd3b5060a 21de9b7369ad fdf859c0313c 2024-04-12T05:37:10.560 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283 running (7m) 2m ago 7m 435M - 17.2.7-827-gd3b5060a 21de9b7369ad 117ab3f1dbe0 2024-04-12T05:37:10.560 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (9m) 2m ago 10m 42.4M 2048M 17.2.7-827-gd3b5060a 21de9b7369ad 75aad95b984d 2024-04-12T05:37:10.561 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (7m) 2m ago 7m 31.6M 2048M 17.2.7-827-gd3b5060a 21de9b7369ad 157be8a8c6d8 2024-04-12T05:37:10.561 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (8m) 2m ago 8m 15.8M - 1.5.0 0da6a335fe13 ca3585dd7b65 2024-04-12T05:37:10.561 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (7m) 2m ago 7m 15.6M - 1.5.0 0da6a335fe13 c324d9436cca 2024-04-12T05:37:10.561 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (6m) 2m ago 6m 46.9M 4300M 17.2.7-827-gd3b5060a 21de9b7369ad 97446e42ceba 2024-04-12T05:37:10.561 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (6m) 2m ago 6m 47.6M 4300M 17.2.7-827-gd3b5060a 21de9b7369ad bf9f9a2db2d5 2024-04-12T05:37:10.561 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (5m) 2m ago 5m 44.1M 4300M 17.2.7-827-gd3b5060a 21de9b7369ad f055cf1ca665 2024-04-12T05:37:10.561 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (5m) 2m ago 5m 44.8M 5323M 17.2.7-827-gd3b5060a 21de9b7369ad 61b851235351 2024-04-12T05:37:10.561 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (4m) 2m ago 4m 42.6M 5323M 17.2.7-827-gd3b5060a 21de9b7369ad 2c84d9047e9c 2024-04-12T05:37:10.561 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (4m) 2m ago 4m 44.4M 5323M 17.2.7-827-gd3b5060a 21de9b7369ad e54839bb6055 2024-04-12T05:37:10.561 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (6m) 2m ago 7m 56.5M - 2.43.0 a07b618ecd1d 996307aec3d2 2024-04-12T05:37:10.562 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:10 smithi067 ceph-mon[49010]: pgmap v266: 65 pgs: 65 active+clean; 1.1 GiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 32 MiB/s rd, 147 MiB/s wr, 265 op/s 2024-04-12T05:37:10.562 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:10 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:37:10.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.561+0000 7f2ae504b700 1 -- 172.21.15.32:0/2047238162 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f2ac806b4b0 msgr2=0x7f2ac806d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:10.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.561+0000 7f2ae504b700 1 --2- 172.21.15.32:0/2047238162 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f2ac806b4b0 0x7f2ac806d960 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2ae0104290 tx=0x7f2ad8007b30 comp rx=0 tx=0).stop 2024-04-12T05:37:10.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.561+0000 7f2ae504b700 1 -- 172.21.15.32:0/2047238162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ae00fcfb0 msgr2=0x7f2ae01032d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:10.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.561+0000 7f2ae504b700 1 --2- 172.21.15.32:0/2047238162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ae00fcfb0 0x7f2ae01032d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2ad0009040 tx=0x7f2ad0009e30 comp rx=0 tx=0).stop 2024-04-12T05:37:10.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.561+0000 7f2ae504b700 1 -- 172.21.15.32:0/2047238162 shutdown_connections 2024-04-12T05:37:10.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.561+0000 7f2ae504b700 1 --2- 172.21.15.32:0/2047238162 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7f2ac806b4b0 0x7f2ac806d960 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:10.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.561+0000 7f2ae504b700 1 --2- 172.21.15.32:0/2047238162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae00fbd00 0x7f2ae0102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:10.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.561+0000 7f2ae504b700 1 --2- 172.21.15.32:0/2047238162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ae00fcfb0 0x7f2ae01032d0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:10.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.561+0000 7f2ae504b700 1 -- 172.21.15.32:0/2047238162 >> 172.21.15.32:0/2047238162 conn(0x7f2ae00f7480 msgr2=0x7f2ae00f8b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:10.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.562+0000 7f2ae504b700 1 -- 172.21.15.32:0/2047238162 shutdown_connections 2024-04-12T05:37:10.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.562+0000 7f2ae504b700 1 -- 172.21.15.32:0/2047238162 wait complete. 2024-04-12T05:37:10.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:10 smithi032 ceph-mon[39731]: pgmap v266: 65 pgs: 65 active+clean; 1.1 GiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 32 MiB/s rd, 147 MiB/s wr, 265 op/s 2024-04-12T05:37:10.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:10 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:37:10.636 INFO:tasks.workunit.client.1.smithi067.stdout:2/123: unlink d5/da/de/d14/l22 0 2024-04-12T05:37:10.637 INFO:tasks.workunit.client.1.smithi067.stdout:8/65: mkdir d6/de/df/d12 0 2024-04-12T05:37:10.637 INFO:tasks.workunit.client.1.smithi067.stdout:2/124: chown f4 11982727 1 2024-04-12T05:37:10.638 INFO:tasks.workunit.client.1.smithi067.stdout:5/175: write d0/da/d27/f14 [2022728,2911] 0 2024-04-12T05:37:10.638 INFO:tasks.workunit.client.1.smithi067.stdout:3/74: mknod dc/c12 0 2024-04-12T05:37:10.640 INFO:tasks.workunit.client.1.smithi067.stdout:9/80: write d1/f11 [144181,43068] 0 2024-04-12T05:37:10.641 INFO:tasks.workunit.client.1.smithi067.stdout:1/112: rmdir d4/d20 0 2024-04-12T05:37:10.642 INFO:tasks.workunit.client.1.smithi067.stdout:4/33: mkdir d8/d9 0 2024-04-12T05:37:10.645 INFO:tasks.workunit.client.1.smithi067.stdout:6/47: truncate f4 925557 0 2024-04-12T05:37:10.646 INFO:tasks.workunit.client.1.smithi067.stdout:1/113: truncate d4/d9/f1c 593203 0 2024-04-12T05:37:10.647 INFO:tasks.workunit.client.1.smithi067.stdout:2/125: truncate d5/da/f1d 4832290 0 2024-04-12T05:37:10.648 INFO:tasks.workunit.client.1.smithi067.stdout:8/66: creat d6/de/df/f13 x:0 0 0 2024-04-12T05:37:10.650 INFO:tasks.workunit.client.1.smithi067.stdout:5/176: creat d0/d6/d33/f3c x:0 0 0 2024-04-12T05:37:10.659 INFO:tasks.workunit.client.1.smithi067.stdout:8/67: read d6/d8/f9 [3521177,94724] 0 2024-04-12T05:37:10.659 INFO:tasks.workunit.client.1.smithi067.stdout:9/81: getdents d1/d12/d15 0 2024-04-12T05:37:10.659 INFO:tasks.workunit.client.1.smithi067.stdout:3/75: dwrite f9 [0,4194304] 0 2024-04-12T05:37:10.662 INFO:tasks.workunit.client.1.smithi067.stdout:0/170: dwrite dd/d12/f16 [0,4194304] 0 2024-04-12T05:37:10.662 INFO:tasks.workunit.client.1.smithi067.stdout:1/114: write d4/f6 [560411,64960] 0 2024-04-12T05:37:10.664 INFO:tasks.workunit.client.1.smithi067.stdout:6/48: rename f4 to fc 0 2024-04-12T05:37:10.669 INFO:tasks.workunit.client.1.smithi067.stdout:3/76: mknod dc/c13 0 2024-04-12T05:37:10.669 INFO:tasks.workunit.client.1.smithi067.stdout:0/171: write dd/d12/d14/d2d/f32 [56934,97773] 0 2024-04-12T05:37:10.672 INFO:tasks.workunit.client.1.smithi067.stdout:1/115: creat d4/d9/f22 x:0 0 0 2024-04-12T05:37:10.672 INFO:tasks.workunit.client.1.smithi067.stdout:4/34: dwrite f7 [0,4194304] 0 2024-04-12T05:37:10.672 INFO:tasks.workunit.client.1.smithi067.stdout:5/177: creat d0/d3a/d2c/f3d x:0 0 0 2024-04-12T05:37:10.677 INFO:tasks.workunit.client.0.smithi032.stdout:7/700: truncate d0/d10/d70/d7b/f88 4172868 0 2024-04-12T05:37:10.678 INFO:tasks.workunit.client.1.smithi067.stdout:8/68: link d6/f7 d6/d8/f14 0 2024-04-12T05:37:10.678 INFO:tasks.workunit.client.1.smithi067.stdout:5/178: stat d0/da 0 2024-04-12T05:37:10.678 INFO:tasks.workunit.client.1.smithi067.stdout:0/172: chown dd/d12/f2a 9889299 1 2024-04-12T05:37:10.678 INFO:tasks.workunit.client.1.smithi067.stdout:9/82: fdatasync d1/fa 0 2024-04-12T05:37:10.679 INFO:tasks.workunit.client.1.smithi067.stdout:5/179: write d0/da/f2e [334881,79344] 0 2024-04-12T05:37:10.680 INFO:tasks.workunit.client.0.smithi032.stdout:8/717: rename d0/db/dd/d11/d1b/d5f/le6 to d0/db/dd/d11/d1b/d5f/deb/lfe 0 2024-04-12T05:37:10.680 INFO:tasks.workunit.client.1.smithi067.stdout:8/69: chown d6/d8/f9 0 1 2024-04-12T05:37:10.687 INFO:tasks.workunit.client.1.smithi067.stdout:3/77: dread f1 [0,4194304] 0 2024-04-12T05:37:10.688 INFO:tasks.workunit.client.1.smithi067.stdout:0/173: truncate dd/d12/d14/f33 939193 0 2024-04-12T05:37:10.690 INFO:tasks.workunit.client.1.smithi067.stdout:7/77: sync 2024-04-12T05:37:10.693 INFO:tasks.workunit.client.1.smithi067.stdout:9/83: dread d1/f7 [0,4194304] 0 2024-04-12T05:37:10.694 INFO:tasks.workunit.client.1.smithi067.stdout:9/84: dread - d1/d12/f14 zero size 2024-04-12T05:37:10.695 INFO:tasks.workunit.client.1.smithi067.stdout:9/85: read - d1/d12/f14 zero size 2024-04-12T05:37:10.697 INFO:tasks.workunit.client.1.smithi067.stdout:9/86: write d1/f2 [32885,103260] 0 2024-04-12T05:37:10.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.683+0000 7fa6c6137700 1 -- 172.21.15.32:0/749141137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6c006e7a0 msgr2=0x7fa6c006ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:10.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.683+0000 7fa6c6137700 1 --2- 172.21.15.32:0/749141137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6c006e7a0 0x7fa6c006ebb0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0004940 tx=0x7fa6b0009e30 comp rx=0 tx=0).stop 2024-04-12T05:37:10.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.683+0000 7fa6c6137700 1 -- 172.21.15.32:0/749141137 shutdown_connections 2024-04-12T05:37:10.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.683+0000 7fa6c6137700 1 --2- 172.21.15.32:0/749141137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6c006f0f0 0x7fa6c0108fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:10.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.683+0000 7fa6c6137700 1 --2- 172.21.15.32:0/749141137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6c006e7a0 0x7fa6c006ebb0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:10.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.683+0000 7fa6c6137700 1 -- 172.21.15.32:0/749141137 >> 172.21.15.32:0/749141137 conn(0x7fa6c006a380 msgr2=0x7fa6c006c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:10.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.683+0000 7fa6c6137700 1 -- 172.21.15.32:0/749141137 shutdown_connections 2024-04-12T05:37:10.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.683+0000 7fa6c6137700 1 -- 172.21.15.32:0/749141137 wait complete. 2024-04-12T05:37:10.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.684+0000 7fa6c6137700 1 Processor -- start 2024-04-12T05:37:10.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.684+0000 7fa6c6137700 1 -- start start 2024-04-12T05:37:10.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.684+0000 7fa6c6137700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6c006e7a0 0x7fa6c0116120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:37:10.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.684+0000 7fa6c6137700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6c006f0f0 0x7fa6c0116660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:37:10.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.684+0000 7fa6c6137700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c0116cf0 con 0x7fa6c006e7a0 2024-04-12T05:37:10.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.684+0000 7fa6c6137700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c011b5f0 con 0x7fa6c006f0f0 2024-04-12T05:37:10.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.684+0000 7fa6bf7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6c006e7a0 0x7fa6c0116120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:10.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.684+0000 7fa6bf7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6c006e7a0 0x7fa6c0116120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56618/0 (socket says 172.21.15.32:56618) 2024-04-12T05:37:10.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.684+0000 7fa6bf7fe700 1 -- 172.21.15.32:0/1939609972 learned_addr learned my addr 172.21.15.32:0/1939609972 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:37:10.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.685+0000 7fa6bf7fe700 1 -- 172.21.15.32:0/1939609972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6c006f0f0 msgr2=0x7fa6c0116660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:10.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.685+0000 7fa6bf7fe700 1 --2- 172.21.15.32:0/1939609972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6c006f0f0 0x7fa6c0116660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:10.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.685+0000 7fa6bf7fe700 1 -- 172.21.15.32:0/1939609972 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b0005040 con 0x7fa6c006e7a0 2024-04-12T05:37:10.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.685+0000 7fa6bf7fe700 1 --2- 172.21.15.32:0/1939609972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6c006e7a0 0x7fa6c0116120 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa6b000b3b0 tx=0x7fa6b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:10.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.685+0000 7fa6a7fff700 1 -- 172.21.15.32:0/1939609972 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6b000e040 con 0x7fa6c006e7a0 2024-04-12T05:37:10.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.685+0000 7fa6a7fff700 1 -- 172.21.15.32:0/1939609972 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa6b000bd20 con 0x7fa6c006e7a0 2024-04-12T05:37:10.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.685+0000 7fa6a7fff700 1 -- 172.21.15.32:0/1939609972 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6b001c630 con 0x7fa6c006e7a0 2024-04-12T05:37:10.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.685+0000 7fa6c6137700 1 -- 172.21.15.32:0/1939609972 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c011b7f0 con 0x7fa6c006e7a0 2024-04-12T05:37:10.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.685+0000 7fa6c6137700 1 -- 172.21.15.32:0/1939609972 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c011bd60 con 0x7fa6c006e7a0 2024-04-12T05:37:10.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.687+0000 7fa6c6137700 1 -- 172.21.15.32:0/1939609972 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c01101a0 con 0x7fa6c006e7a0 2024-04-12T05:37:10.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.688+0000 7fa6a7fff700 1 -- 172.21.15.32:0/1939609972 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fa6b0013420 con 0x7fa6c006e7a0 2024-04-12T05:37:10.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.688+0000 7fa6a7fff700 1 --2- 172.21.15.32:0/1939609972 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fa6a80692a0 0x7fa6a806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:37:10.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.688+0000 7fa6a7fff700 1 -- 172.21.15.32:0/1939609972 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fa6b0087e80 con 0x7fa6c006e7a0 2024-04-12T05:37:10.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.688+0000 7fa6beffd700 1 --2- 172.21.15.32:0/1939609972 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fa6a80692a0 0x7fa6a806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:10.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.689+0000 7fa6beffd700 1 --2- 172.21.15.32:0/1939609972 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fa6a80692a0 0x7fa6a806b750 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4006fd0 tx=0x7fa6b4006db0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:10.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:10.696+0000 7fa6a7fff700 1 -- 172.21.15.32:0/1939609972 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fa6b0057c90 con 0x7fa6c006e7a0 2024-04-12T05:37:10.700 INFO:tasks.workunit.client.1.smithi067.stdout:9/87: fdatasync d1/fa 0 2024-04-12T05:37:10.701 INFO:tasks.workunit.client.0.smithi032.stdout:4/648: sync 2024-04-12T05:37:10.701 INFO:tasks.workunit.client.0.smithi032.stdout:5/690: sync 2024-04-12T05:37:10.701 INFO:tasks.workunit.client.0.smithi032.stdout:1/572: sync 2024-04-12T05:37:10.701 INFO:tasks.workunit.client.0.smithi032.stdout:1/573: write d5/d2c/f43 [8617727,16372] 0 2024-04-12T05:37:10.701 INFO:tasks.workunit.client.0.smithi032.stdout:1/574: chown d5/c4e 58 1 2024-04-12T05:37:10.741 INFO:tasks.workunit.client.1.smithi067.stdout:1/116: creat d4/d9/d12/d19/f23 x:0 0 0 2024-04-12T05:37:10.741 INFO:tasks.workunit.client.1.smithi067.stdout:4/35: mkdir d8/da 0 2024-04-12T05:37:10.741 INFO:tasks.workunit.client.1.smithi067.stdout:2/126: getdents d5/da/de 0 2024-04-12T05:37:10.742 INFO:tasks.workunit.client.1.smithi067.stdout:8/70: mknod d6/de/c15 0 2024-04-12T05:37:10.743 INFO:tasks.workunit.client.1.smithi067.stdout:4/36: write f3 [4105921,9602] 0 2024-04-12T05:37:10.744 INFO:tasks.workunit.client.1.smithi067.stdout:3/78: creat dc/df/f14 x:0 0 0 2024-04-12T05:37:10.746 INFO:tasks.workunit.client.1.smithi067.stdout:0/174: fdatasync dd/f18 0 2024-04-12T05:37:10.747 INFO:tasks.workunit.client.1.smithi067.stdout:7/78: creat d0/d2/d5/db/d10/f1a x:0 0 0 2024-04-12T05:37:10.749 INFO:tasks.workunit.client.1.smithi067.stdout:9/88: unlink d1/fa 0 2024-04-12T05:37:10.751 INFO:tasks.workunit.client.1.smithi067.stdout:0/175: write dd/f1c [3940608,74010] 0 2024-04-12T05:37:10.753 INFO:tasks.workunit.client.1.smithi067.stdout:1/117: symlink d4/d9/d12/l24 0 2024-04-12T05:37:10.754 INFO:tasks.workunit.client.1.smithi067.stdout:0/176: read dd/d12/f21 [405275,63901] 0 2024-04-12T05:37:10.754 INFO:tasks.workunit.client.1.smithi067.stdout:7/79: dread d0/d2/d5/f11 [0,4194304] 0 2024-04-12T05:37:10.755 INFO:tasks.workunit.client.1.smithi067.stdout:8/71: write d6/d8/f9 [534589,55848] 0 2024-04-12T05:37:10.759 INFO:tasks.workunit.client.1.smithi067.stdout:6/49: getdents . 0 2024-04-12T05:37:10.760 INFO:tasks.workunit.client.1.smithi067.stdout:4/37: symlink d8/d9/lb 0 2024-04-12T05:37:10.762 INFO:tasks.workunit.client.1.smithi067.stdout:4/38: chown l2 923 1 2024-04-12T05:37:10.762 INFO:tasks.workunit.client.1.smithi067.stdout:2/127: dread d5/da/f1d [0,4194304] 0 2024-04-12T05:37:10.762 INFO:tasks.workunit.client.1.smithi067.stdout:5/180: getdents d0/d6/d33 0 2024-04-12T05:37:10.764 INFO:tasks.workunit.client.1.smithi067.stdout:6/50: mkdir dd 0 2024-04-12T05:37:10.764 INFO:tasks.workunit.client.1.smithi067.stdout:4/39: chown l2 20982 1 2024-04-12T05:37:10.765 INFO:tasks.workunit.client.1.smithi067.stdout:6/51: dread - fb zero size 2024-04-12T05:37:10.767 INFO:tasks.workunit.client.1.smithi067.stdout:1/118: symlink d4/d9/d14/l25 0 2024-04-12T05:37:10.768 INFO:tasks.workunit.client.1.smithi067.stdout:7/80: truncate d0/f9 2173351 0 2024-04-12T05:37:10.770 INFO:tasks.workunit.client.1.smithi067.stdout:2/128: mknod d5/da/de/d14/c2c 0 2024-04-12T05:37:10.771 INFO:tasks.workunit.client.1.smithi067.stdout:5/181: mkdir d0/d6/d20/d3e 0 2024-04-12T05:37:10.773 INFO:tasks.workunit.client.1.smithi067.stdout:4/40: creat d8/d9/fc x:0 0 0 2024-04-12T05:37:10.774 INFO:tasks.workunit.client.1.smithi067.stdout:5/182: write d0/da/d27/f14 [2977944,42407] 0 2024-04-12T05:37:10.776 INFO:tasks.workunit.client.1.smithi067.stdout:4/41: dread - d8/d9/fc zero size 2024-04-12T05:37:10.778 INFO:tasks.workunit.client.1.smithi067.stdout:6/52: rename f8 to dd/fe 0 2024-04-12T05:37:10.779 INFO:tasks.workunit.client.1.smithi067.stdout:7/81: dread d0/d2/d5/fe [0,4194304] 0 2024-04-12T05:37:10.780 INFO:tasks.workunit.client.1.smithi067.stdout:1/119: read d4/d9/d14/f21 [1251282,58337] 0 2024-04-12T05:37:10.782 INFO:tasks.workunit.client.1.smithi067.stdout:1/120: chown d4/d9/f1c 138274 1 2024-04-12T05:37:10.783 INFO:tasks.workunit.client.1.smithi067.stdout:5/183: mkdir d0/da/d3f 0 2024-04-12T05:37:10.783 INFO:tasks.workunit.client.1.smithi067.stdout:1/121: read d4/f5 [307516,80120] 0 2024-04-12T05:37:10.785 INFO:tasks.workunit.client.1.smithi067.stdout:4/42: unlink f3 0 2024-04-12T05:37:10.786 INFO:tasks.workunit.client.1.smithi067.stdout:8/72: getdents d6/de/df 0 2024-04-12T05:37:10.788 INFO:tasks.workunit.client.1.smithi067.stdout:6/53: symlink dd/lf 0 2024-04-12T05:37:10.790 INFO:tasks.workunit.client.1.smithi067.stdout:2/129: rename c2 to d5/da/de/c2d 0 2024-04-12T05:37:10.792 INFO:tasks.workunit.client.1.smithi067.stdout:0/177: getdents dd/d12/d14/d2d 0 2024-04-12T05:37:10.796 INFO:tasks.workunit.client.1.smithi067.stdout:5/184: mkdir d0/d6/d40 0 2024-04-12T05:37:10.800 INFO:tasks.workunit.client.1.smithi067.stdout:6/54: creat dd/f10 x:0 0 0 2024-04-12T05:37:10.802 INFO:tasks.workunit.client.1.smithi067.stdout:4/43: dwrite d8/d9/fc [0,4194304] 0 2024-04-12T05:37:10.803 INFO:tasks.workunit.client.1.smithi067.stdout:5/185: stat d0/d6/f10 0 2024-04-12T05:37:10.803 INFO:tasks.workunit.client.1.smithi067.stdout:6/55: dread - fb zero size 2024-04-12T05:37:10.805 INFO:tasks.workunit.client.1.smithi067.stdout:1/122: read d4/d9/d12/f1b [571049,68290] 0 2024-04-12T05:37:10.805 INFO:tasks.workunit.client.1.smithi067.stdout:8/73: rename d6/de/f11 to d6/d8/f16 0 2024-04-12T05:37:10.808 INFO:tasks.workunit.client.1.smithi067.stdout:4/44: write d8/d9/fc [2917530,106352] 0 2024-04-12T05:37:10.810 INFO:tasks.workunit.client.1.smithi067.stdout:1/123: write d4/d9/d12/d19/f1e [2452161,8791] 0 2024-04-12T05:37:10.815 INFO:tasks.workunit.client.1.smithi067.stdout:5/186: unlink d0/d6/d20/c26 0 2024-04-12T05:37:10.823 INFO:tasks.workunit.client.1.smithi067.stdout:4/45: rename c5 to d8/cd 0 2024-04-12T05:37:10.824 INFO:tasks.workunit.client.1.smithi067.stdout:8/74: dwrite d6/d8/f9 [0,4194304] 0 2024-04-12T05:37:10.825 INFO:tasks.workunit.client.1.smithi067.stdout:1/124: mknod d4/d9/c26 0 2024-04-12T05:37:10.826 INFO:tasks.workunit.client.1.smithi067.stdout:5/187: unlink d0/fb 0 2024-04-12T05:37:10.826 INFO:tasks.workunit.client.1.smithi067.stdout:0/178: truncate f1 7366733 0 2024-04-12T05:37:10.829 INFO:tasks.workunit.client.1.smithi067.stdout:4/46: write f6 [3854540,27967] 0 2024-04-12T05:37:10.832 INFO:tasks.workunit.client.1.smithi067.stdout:5/188: mkdir d0/da/d41 0 2024-04-12T05:37:10.834 INFO:tasks.workunit.client.1.smithi067.stdout:8/75: mknod d6/d8/c17 0 2024-04-12T05:37:10.835 INFO:tasks.workunit.client.1.smithi067.stdout:4/47: write f7 [3856718,95099] 0 2024-04-12T05:37:10.837 INFO:tasks.workunit.client.1.smithi067.stdout:2/130: dwrite d5/da/de/d14/f24 [4194304,4194304] 0 2024-04-12T05:37:10.843 INFO:tasks.workunit.client.1.smithi067.stdout:2/131: dread - d5/da/de/f28 zero size 2024-04-12T05:37:10.845 INFO:tasks.workunit.client.1.smithi067.stdout:8/76: dread f5 [0,4194304] 0 2024-04-12T05:37:10.846 INFO:tasks.workunit.client.1.smithi067.stdout:4/48: creat d8/fe x:0 0 0 2024-04-12T05:37:10.846 INFO:tasks.workunit.client.1.smithi067.stdout:0/179: rename dd/f1c to dd/d12/f3a 0 2024-04-12T05:37:10.849 INFO:tasks.workunit.client.1.smithi067.stdout:5/189: dread d0/da/f2e [0,4194304] 0 2024-04-12T05:37:10.850 INFO:tasks.workunit.client.1.smithi067.stdout:2/132: rename d5/da/de/d14/f1a to d5/d11/f2e 0 2024-04-12T05:37:10.854 INFO:tasks.workunit.client.1.smithi067.stdout:5/190: symlink d0/d6/d33/l42 0 2024-04-12T05:37:10.860 INFO:tasks.workunit.client.1.smithi067.stdout:8/77: rename d6/de to d6/d8/d18 0 2024-04-12T05:37:10.861 INFO:tasks.workunit.client.1.smithi067.stdout:4/49: link d8/d9/fc d8/d9/ff 0 2024-04-12T05:37:10.862 INFO:tasks.workunit.client.1.smithi067.stdout:0/180: creat dd/d12/f3b x:0 0 0 2024-04-12T05:37:10.862 INFO:tasks.workunit.client.1.smithi067.stdout:5/191: chown d0/d6/d20/c23 6336355 1 2024-04-12T05:37:10.863 INFO:tasks.workunit.client.1.smithi067.stdout:4/50: chown d8/fe 0 1 2024-04-12T05:37:10.863 INFO:tasks.workunit.client.1.smithi067.stdout:0/181: fsync dd/f38 0 2024-04-12T05:37:10.864 INFO:tasks.workunit.client.1.smithi067.stdout:8/78: rename d6/d8/lb to d6/l19 0 2024-04-12T05:37:10.866 INFO:tasks.workunit.client.1.smithi067.stdout:5/192: creat d0/f43 x:0 0 0 2024-04-12T05:37:10.869 INFO:tasks.workunit.client.1.smithi067.stdout:0/182: creat dd/d12/d14/d2d/f3c x:0 0 0 2024-04-12T05:37:10.872 INFO:tasks.workunit.client.1.smithi067.stdout:5/193: write d0/d6/f12 [4500377,126177] 0 2024-04-12T05:37:10.880 INFO:tasks.workunit.client.0.smithi032.stdout:2/699: sync 2024-04-12T05:37:10.880 INFO:tasks.workunit.client.1.smithi067.stdout:5/194: symlink d0/d6/d33/l44 0 2024-04-12T05:37:10.881 INFO:tasks.workunit.client.0.smithi032.stdout:2/700: dread - dd/d19/d9d/d42/d4a/da1/fbc zero size 2024-04-12T05:37:10.885 INFO:tasks.workunit.client.1.smithi067.stdout:2/133: dwrite d5/da/f1d [4194304,4194304] 0 2024-04-12T05:37:10.887 INFO:tasks.workunit.client.1.smithi067.stdout:5/195: stat d0/d36 0 2024-04-12T05:37:10.892 INFO:tasks.workunit.client.1.smithi067.stdout:6/56: write f3 [3214646,100249] 0 2024-04-12T05:37:10.894 INFO:tasks.workunit.client.1.smithi067.stdout:2/134: symlink d5/da/de/l2f 0 2024-04-12T05:37:10.898 INFO:tasks.workunit.client.1.smithi067.stdout:4/51: dwrite d8/d9/fc [0,4194304] 0 2024-04-12T05:37:10.904 INFO:tasks.workunit.client.1.smithi067.stdout:8/79: dwrite d6/d8/d18/df/f13 [0,4194304] 0 2024-04-12T05:37:10.906 INFO:tasks.workunit.client.1.smithi067.stdout:0/183: dwrite dd/d12/d14/d2d/f3c [0,4194304] 0 2024-04-12T05:37:10.909 INFO:tasks.workunit.client.1.smithi067.stdout:7/82: dread d0/f9 [0,4194304] 0 2024-04-12T05:37:10.909 INFO:tasks.workunit.client.1.smithi067.stdout:4/52: mknod d8/d9/c10 0 2024-04-12T05:37:10.910 INFO:tasks.workunit.client.1.smithi067.stdout:8/80: mknod d6/d8/c1a 0 2024-04-12T05:37:10.915 INFO:tasks.workunit.client.1.smithi067.stdout:2/135: rmdir d5/da/de/d18 0 2024-04-12T05:37:10.918 INFO:tasks.workunit.client.1.smithi067.stdout:4/53: chown d8/d9 1708 1 2024-04-12T05:37:10.922 INFO:tasks.workunit.client.1.smithi067.stdout:2/136: chown d5/l21 9 1 2024-04-12T05:37:10.927 INFO:tasks.workunit.client.1.smithi067.stdout:6/57: dwrite dd/fe [0,4194304] 0 2024-04-12T05:37:10.927 INFO:tasks.workunit.client.1.smithi067.stdout:0/184: rename dd/d12/d14/d2d to dd/d12/d14/d1a/d3d 0 2024-04-12T05:37:10.929 INFO:tasks.workunit.client.1.smithi067.stdout:7/83: symlink d0/l1b 0 2024-04-12T05:37:10.929 INFO:tasks.workunit.client.1.smithi067.stdout:1/125: dwrite d4/d9/fd [0,4194304] 0 2024-04-12T05:37:10.930 INFO:tasks.workunit.client.1.smithi067.stdout:2/137: write d5/f20 [5163241,108661] 0 2024-04-12T05:37:10.937 INFO:tasks.workunit.client.1.smithi067.stdout:4/54: creat d8/d9/f11 x:0 0 0 2024-04-12T05:37:10.944 INFO:tasks.workunit.client.1.smithi067.stdout:7/84: creat d0/d2/d5/f1c x:0 0 0 2024-04-12T05:37:10.948 INFO:tasks.workunit.client.1.smithi067.stdout:1/126: rename d4/d9/l18 to d4/d9/d12/d19/l27 0 2024-04-12T05:37:10.951 INFO:tasks.workunit.client.1.smithi067.stdout:5/196: dwrite d0/da/d27/f16 [0,4194304] 0 2024-04-12T05:37:10.954 INFO:tasks.workunit.client.0.smithi032.stdout:3/693: sync 2024-04-12T05:37:10.954 INFO:tasks.workunit.client.0.smithi032.stdout:6/837: sync 2024-04-12T05:37:10.954 INFO:tasks.workunit.client.0.smithi032.stdout:0/701: sync 2024-04-12T05:37:10.955 INFO:tasks.workunit.client.1.smithi067.stdout:7/85: unlink d0/d2/f13 0 2024-04-12T05:37:10.955 INFO:tasks.workunit.client.1.smithi067.stdout:6/58: dread f3 [0,4194304] 0 2024-04-12T05:37:10.956 INFO:tasks.workunit.client.1.smithi067.stdout:4/55: symlink d8/d9/l12 0 2024-04-12T05:37:10.962 INFO:tasks.workunit.client.0.smithi032.stdout:6/838: dwrite d4/de0/fb9 [0,4194304] 0 2024-04-12T05:37:10.963 INFO:tasks.workunit.client.1.smithi067.stdout:8/81: dwrite d6/f7 [4194304,4194304] 0 2024-04-12T05:37:10.964 INFO:tasks.workunit.client.1.smithi067.stdout:5/197: stat d0/d3a/f19 0 2024-04-12T05:37:10.964 INFO:tasks.workunit.client.1.smithi067.stdout:8/82: chown d6 31741 1 2024-04-12T05:37:10.964 INFO:tasks.workunit.client.1.smithi067.stdout:2/138: link d5/c8 d5/da/c30 0 2024-04-12T05:37:10.965 INFO:tasks.workunit.client.1.smithi067.stdout:6/59: chown f6 0 1 2024-04-12T05:37:10.968 INFO:tasks.workunit.client.0.smithi032.stdout:0/702: dread d2/d8/df/d24/f5c [0,4194304] 0 2024-04-12T05:37:10.971 INFO:tasks.workunit.client.1.smithi067.stdout:0/185: dwrite dd/f18 [4194304,4194304] 0 2024-04-12T05:37:10.974 INFO:tasks.workunit.client.1.smithi067.stdout:1/127: write d4/f8 [695159,39153] 0 2024-04-12T05:37:10.974 INFO:tasks.workunit.client.1.smithi067.stdout:4/56: chown d8/cd 57600259 1 2024-04-12T05:37:10.976 INFO:tasks.workunit.client.1.smithi067.stdout:7/86: mkdir d0/d2/d5/d1d 0 2024-04-12T05:37:10.976 INFO:tasks.workunit.client.1.smithi067.stdout:1/128: dread - d4/d9/f17 zero size 2024-04-12T05:37:10.976 INFO:tasks.workunit.client.1.smithi067.stdout:6/60: mkdir dd/d11 0 2024-04-12T05:37:10.978 INFO:tasks.workunit.client.1.smithi067.stdout:6/61: chown dd/d11 655 1 2024-04-12T05:37:10.980 INFO:tasks.workunit.client.1.smithi067.stdout:8/83: read d6/f7 [2265487,71490] 0 2024-04-12T05:37:10.983 INFO:tasks.workunit.client.1.smithi067.stdout:4/57: rename l2 to d8/l13 0 2024-04-12T05:37:10.984 INFO:tasks.workunit.client.1.smithi067.stdout:2/139: dread d5/da/de/f13 [0,4194304] 0 2024-04-12T05:37:10.984 INFO:tasks.workunit.client.1.smithi067.stdout:5/198: creat d0/d36/f45 x:0 0 0 2024-04-12T05:37:10.986 INFO:tasks.workunit.client.1.smithi067.stdout:6/62: fsync f3 0 2024-04-12T05:37:10.986 INFO:tasks.workunit.client.1.smithi067.stdout:1/129: mkdir d4/d9/d12/d28 0 2024-04-12T05:37:10.987 INFO:tasks.workunit.client.1.smithi067.stdout:1/130: chown d4/d9/l11 873 1 2024-04-12T05:37:10.987 INFO:tasks.workunit.client.1.smithi067.stdout:6/63: stat dd/f10 0 2024-04-12T05:37:10.987 INFO:tasks.workunit.client.1.smithi067.stdout:8/84: rmdir d6/d8 39 2024-04-12T05:37:10.989 INFO:tasks.workunit.client.1.smithi067.stdout:1/131: read d4/f6 [618362,61052] 0 2024-04-12T05:37:10.989 INFO:tasks.workunit.client.1.smithi067.stdout:2/140: rename d5/da/de/f28 to d5/da/de/f31 0 2024-04-12T05:37:10.990 INFO:tasks.workunit.client.1.smithi067.stdout:6/64: write dd/f10 [408563,669] 0 2024-04-12T05:37:10.990 INFO:tasks.workunit.client.1.smithi067.stdout:0/186: creat dd/d12/f3e x:0 0 0 2024-04-12T05:37:10.994 INFO:tasks.workunit.client.1.smithi067.stdout:2/141: write d5/da/f1d [9170975,13934] 0 2024-04-12T05:37:10.994 INFO:tasks.workunit.client.1.smithi067.stdout:1/132: readlink d4/d9/d12/l24 0 2024-04-12T05:37:10.995 INFO:tasks.workunit.client.1.smithi067.stdout:6/65: unlink f3 0 2024-04-12T05:37:10.995 INFO:tasks.workunit.client.1.smithi067.stdout:5/199: symlink d0/da/d3f/l46 0 2024-04-12T05:37:10.996 INFO:tasks.workunit.client.1.smithi067.stdout:7/87: getdents d0/d2/d5 0 2024-04-12T05:37:10.997 INFO:tasks.workunit.client.1.smithi067.stdout:8/85: creat d6/d8/d18/df/f1b x:0 0 0 2024-04-12T05:37:10.998 INFO:tasks.workunit.client.1.smithi067.stdout:0/187: unlink dd/d12/c22 0 2024-04-12T05:37:10.999 INFO:tasks.workunit.client.1.smithi067.stdout:0/188: chown dd/d12/d14 154583826 1 2024-04-12T05:37:11.001 INFO:tasks.workunit.client.1.smithi067.stdout:2/142: rename d5/d11/f1e to d5/d11/f32 0 2024-04-12T05:37:11.002 INFO:tasks.workunit.client.1.smithi067.stdout:1/133: dread d4/d9/fb [0,4194304] 0 2024-04-12T05:37:11.003 INFO:tasks.workunit.client.1.smithi067.stdout:7/88: read d0/d2/ff [2053277,85743] 0 2024-04-12T05:37:11.004 INFO:tasks.workunit.client.1.smithi067.stdout:2/143: read d5/f20 [1572841,48361] 0 2024-04-12T05:37:11.005 INFO:tasks.workunit.client.1.smithi067.stdout:7/89: chown d0/d2/dd 217 1 2024-04-12T05:37:11.006 INFO:tasks.workunit.client.1.smithi067.stdout:0/189: creat dd/f3f x:0 0 0 2024-04-12T05:37:11.008 INFO:tasks.workunit.client.1.smithi067.stdout:2/144: write d5/da/f1d [19449,29898] 0 2024-04-12T05:37:11.009 INFO:tasks.workunit.client.1.smithi067.stdout:1/134: rename d4/f1d to d4/d9/f29 0 2024-04-12T05:37:11.010 INFO:tasks.workunit.client.1.smithi067.stdout:1/135: chown d4/d9/d12 18795179 1 2024-04-12T05:37:11.011 INFO:tasks.workunit.client.1.smithi067.stdout:7/90: rmdir d0/d2/dd 39 2024-04-12T05:37:11.011 INFO:tasks.workunit.client.1.smithi067.stdout:1/136: stat d4/d9 0 2024-04-12T05:37:11.011 INFO:tasks.workunit.client.1.smithi067.stdout:2/145: chown d5/da/de/d14 3831 1 2024-04-12T05:37:11.012 INFO:tasks.workunit.client.1.smithi067.stdout:8/86: dread d6/d8/d18/df/f13 [0,4194304] 0 2024-04-12T05:37:11.012 INFO:tasks.workunit.client.1.smithi067.stdout:0/190: mkdir dd/d40 0 2024-04-12T05:37:11.014 INFO:tasks.workunit.client.1.smithi067.stdout:5/200: dread d0/d3a/d2c/f37 [0,4194304] 0 2024-04-12T05:37:11.014 INFO:tasks.workunit.client.1.smithi067.stdout:2/146: mkdir d5/da/d33 0 2024-04-12T05:37:11.016 INFO:tasks.workunit.client.1.smithi067.stdout:8/87: fsync d6/d8/d18/df/f13 0 2024-04-12T05:37:11.017 INFO:tasks.workunit.client.1.smithi067.stdout:7/91: truncate d0/d2/d5/db/f18 717154 0 2024-04-12T05:37:11.017 INFO:tasks.workunit.client.1.smithi067.stdout:0/191: mkdir dd/d12/d14/d1a/d3d/d41 0 2024-04-12T05:37:11.018 INFO:tasks.workunit.client.1.smithi067.stdout:2/147: mkdir d5/da/de/d14/d34 0 2024-04-12T05:37:11.019 INFO:tasks.workunit.client.1.smithi067.stdout:5/201: write d0/d3a/d2c/f37 [3008561,128872] 0 2024-04-12T05:37:11.019 INFO:tasks.workunit.client.1.smithi067.stdout:1/137: dread d4/f8 [0,4194304] 0 2024-04-12T05:37:11.021 INFO:tasks.workunit.client.1.smithi067.stdout:7/92: readlink d0/d2/d5/dc/l17 0 2024-04-12T05:37:11.022 INFO:tasks.workunit.client.1.smithi067.stdout:8/88: creat d6/d8/d18/df/f1c x:0 0 0 2024-04-12T05:37:11.024 INFO:tasks.workunit.client.1.smithi067.stdout:5/202: mknod d0/d6/c47 0 2024-04-12T05:37:11.024 INFO:tasks.workunit.client.1.smithi067.stdout:8/89: write d6/d8/f9 [2016532,113744] 0 2024-04-12T05:37:11.026 INFO:tasks.workunit.client.1.smithi067.stdout:5/203: fdatasync d0/da/d27/f18 0 2024-04-12T05:37:11.027 INFO:tasks.workunit.client.1.smithi067.stdout:7/93: rmdir d0/d2 39 2024-04-12T05:37:11.029 INFO:tasks.workunit.client.1.smithi067.stdout:1/138: creat d4/d9/d14/f2a x:0 0 0 2024-04-12T05:37:11.031 INFO:tasks.workunit.client.1.smithi067.stdout:5/204: write d0/d6/d33/f3c [742339,108309] 0 2024-04-12T05:37:11.035 INFO:tasks.workunit.client.1.smithi067.stdout:7/94: mkdir d0/d2/d5/d1e 0 2024-04-12T05:37:11.036 INFO:tasks.workunit.client.1.smithi067.stdout:1/139: write d4/f8 [1764789,88455] 0 2024-04-12T05:37:11.036 INFO:tasks.workunit.client.1.smithi067.stdout:0/192: write dd/d12/d14/f39 [1119541,32126] 0 2024-04-12T05:37:11.037 INFO:tasks.workunit.client.1.smithi067.stdout:5/205: write d0/d3a/d2c/f3d [220861,30717] 0 2024-04-12T05:37:11.041 INFO:tasks.workunit.client.1.smithi067.stdout:5/206: chown d0/da 804 1 2024-04-12T05:37:11.041 INFO:tasks.workunit.client.1.smithi067.stdout:1/140: write d4/f8 [1734712,88385] 0 2024-04-12T05:37:11.042 INFO:tasks.workunit.client.1.smithi067.stdout:2/148: dwrite d5/d11/f2e [0,4194304] 0 2024-04-12T05:37:11.044 INFO:tasks.workunit.client.1.smithi067.stdout:8/90: dwrite d6/d8/f9 [0,4194304] 0 2024-04-12T05:37:11.051 INFO:tasks.workunit.client.1.smithi067.stdout:1/141: truncate d4/d9/f29 2310874 0 2024-04-12T05:37:11.053 INFO:tasks.workunit.client.1.smithi067.stdout:2/149: creat d5/da/d33/f35 x:0 0 0 2024-04-12T05:37:11.054 INFO:tasks.workunit.client.1.smithi067.stdout:5/207: dread d0/f2 [0,4194304] 0 2024-04-12T05:37:11.057 INFO:tasks.workunit.client.1.smithi067.stdout:1/142: write d4/d9/f1c [520339,108326] 0 2024-04-12T05:37:11.063 INFO:tasks.workunit.client.1.smithi067.stdout:2/150: link d5/f20 d5/da/de/d14/f36 0 2024-04-12T05:37:11.063 INFO:tasks.workunit.client.1.smithi067.stdout:1/143: read d4/d9/d12/f1b [879720,85550] 0 2024-04-12T05:37:11.064 INFO:tasks.workunit.client.1.smithi067.stdout:1/144: fdatasync d4/f5 0 2024-04-12T05:37:11.064 INFO:tasks.workunit.client.1.smithi067.stdout:5/208: write d0/da/f2e [1076908,20043] 0 2024-04-12T05:37:11.066 INFO:tasks.workunit.client.1.smithi067.stdout:1/145: chown d4/d9/d12/d28 12 1 2024-04-12T05:37:11.067 INFO:tasks.workunit.client.1.smithi067.stdout:5/209: dread - d0/f43 zero size 2024-04-12T05:37:11.068 INFO:tasks.workunit.client.1.smithi067.stdout:1/146: fdatasync d4/d9/f1c 0 2024-04-12T05:37:11.069 INFO:tasks.workunit.client.1.smithi067.stdout:2/151: creat d5/da/de/d14/d34/f37 x:0 0 0 2024-04-12T05:37:11.071 INFO:tasks.workunit.client.1.smithi067.stdout:2/152: read d5/d11/f2e [2320045,12066] 0 2024-04-12T05:37:11.074 INFO:tasks.workunit.client.1.smithi067.stdout:1/147: creat d4/d9/d12/d28/f2b x:0 0 0 2024-04-12T05:37:11.074 INFO:tasks.workunit.client.1.smithi067.stdout:2/153: write d5/da/de/f26 [1447724,51134] 0 2024-04-12T05:37:11.076 INFO:tasks.workunit.client.1.smithi067.stdout:2/154: chown d5/da 97 1 2024-04-12T05:37:11.077 INFO:tasks.workunit.client.1.smithi067.stdout:2/155: fdatasync d5/d11/f2a 0 2024-04-12T05:37:11.079 INFO:tasks.workunit.client.1.smithi067.stdout:2/156: readlink d5/d11/l27 0 2024-04-12T05:37:11.081 INFO:tasks.workunit.client.1.smithi067.stdout:5/210: dwrite d0/da/d27/f14 [0,4194304] 0 2024-04-12T05:37:11.085 INFO:tasks.workunit.client.1.smithi067.stdout:0/193: dread dd/d12/d14/d1a/d3d/f2f [0,4194304] 0 2024-04-12T05:37:11.087 INFO:tasks.workunit.client.1.smithi067.stdout:2/157: symlink d5/da/d33/l38 0 2024-04-12T05:37:11.088 INFO:tasks.workunit.client.1.smithi067.stdout:2/158: stat d5/d11/l27 0 2024-04-12T05:37:11.094 INFO:tasks.workunit.client.1.smithi067.stdout:0/194: dread dd/d12/f21 [0,4194304] 0 2024-04-12T05:37:11.095 INFO:tasks.workunit.client.1.smithi067.stdout:0/195: write dd/f13 [5199359,109855] 0 2024-04-12T05:37:11.097 INFO:tasks.workunit.client.1.smithi067.stdout:0/196: truncate dd/d12/f3a 4607356 0 2024-04-12T05:37:11.099 INFO:tasks.workunit.client.1.smithi067.stdout:0/197: stat dd/d12/d14/d1a/d3d/f32 0 2024-04-12T05:37:11.101 INFO:tasks.workunit.client.1.smithi067.stdout:0/198: read - dd/d12/f3e zero size 2024-04-12T05:37:11.102 INFO:tasks.workunit.client.1.smithi067.stdout:5/211: fsync d0/da/d27/f14 0 2024-04-12T05:37:11.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.106+0000 7fa6c6137700 1 -- 172.21.15.32:0/1939609972 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa6c002b3a0 con 0x7fa6c006e7a0 2024-04-12T05:37:11.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.107+0000 7fa6a7fff700 1 -- 172.21.15.32:0/1939609972 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7fa6b0013e60 con 0x7fa6c006e7a0 2024-04-12T05:37:11.104 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:37:11.104 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:37:11.104 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 2 2024-04-12T05:37:11.104 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:37:11.104 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:37:11.104 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 2 2024-04-12T05:37:11.104 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:37:11.104 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:37:11.105 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 6 2024-04-12T05:37:11.105 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:37:11.105 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:37:11.105 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 4 2024-04-12T05:37:11.105 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:37:11.105 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:37:11.105 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 14 2024-04-12T05:37:11.105 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:37:11.105 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:37:11.106 INFO:tasks.workunit.client.1.smithi067.stdout:5/212: write d0/fc [5125719,49371] 0 2024-04-12T05:37:11.106 INFO:tasks.workunit.client.1.smithi067.stdout:0/199: dread dd/d12/d14/d1a/d3d/f32 [0,4194304] 0 2024-04-12T05:37:11.106 INFO:tasks.workunit.client.1.smithi067.stdout:2/159: getdents d5/d11 0 2024-04-12T05:37:11.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.109+0000 7fa6c6137700 1 -- 172.21.15.32:0/1939609972 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fa6a80692a0 msgr2=0x7fa6a806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:11.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.109+0000 7fa6c6137700 1 --2- 172.21.15.32:0/1939609972 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fa6a80692a0 0x7fa6a806b750 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4006fd0 tx=0x7fa6b4006db0 comp rx=0 tx=0).stop 2024-04-12T05:37:11.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.109+0000 7fa6c6137700 1 -- 172.21.15.32:0/1939609972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6c006e7a0 msgr2=0x7fa6c0116120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:11.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.109+0000 7fa6c6137700 1 --2- 172.21.15.32:0/1939609972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6c006e7a0 0x7fa6c0116120 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa6b000b3b0 tx=0x7fa6b0009e30 comp rx=0 tx=0).stop 2024-04-12T05:37:11.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.109+0000 7fa6c6137700 1 -- 172.21.15.32:0/1939609972 shutdown_connections 2024-04-12T05:37:11.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.109+0000 7fa6c6137700 1 --2- 172.21.15.32:0/1939609972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6c006e7a0 0x7fa6c0116120 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:11.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.109+0000 7fa6c6137700 1 --2- 172.21.15.32:0/1939609972 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7fa6a80692a0 0x7fa6a806b750 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:11.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.109+0000 7fa6c6137700 1 --2- 172.21.15.32:0/1939609972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6c006f0f0 0x7fa6c0116660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:11.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.109+0000 7fa6c6137700 1 -- 172.21.15.32:0/1939609972 >> 172.21.15.32:0/1939609972 conn(0x7fa6c006a380 msgr2=0x7fa6c006cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:11.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.109+0000 7fa6c6137700 1 -- 172.21.15.32:0/1939609972 shutdown_connections 2024-04-12T05:37:11.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.109+0000 7fa6c6137700 1 -- 172.21.15.32:0/1939609972 wait complete. 2024-04-12T05:37:11.109 INFO:tasks.workunit.client.1.smithi067.stdout:0/200: creat dd/f42 x:0 0 0 2024-04-12T05:37:11.110 INFO:tasks.workunit.client.1.smithi067.stdout:2/160: mkdir d5/da/de/d39 0 2024-04-12T05:37:11.112 INFO:tasks.workunit.client.0.smithi032.stdout:9/556: unlink l8 0 2024-04-12T05:37:11.113 INFO:tasks.workunit.client.0.smithi032.stdout:7/701: stat d0/d1c/d4c/fc7 0 2024-04-12T05:37:11.114 INFO:tasks.workunit.client.1.smithi067.stdout:2/161: mkdir d5/d23/d3a 0 2024-04-12T05:37:11.114 INFO:tasks.workunit.client.1.smithi067.stdout:5/213: dwrite d0/f43 [0,4194304] 0 2024-04-12T05:37:11.114 INFO:tasks.workunit.client.0.smithi032.stdout:8/718: write d0/db/dd/d11/f1f [4238555,51027] 0 2024-04-12T05:37:11.115 INFO:tasks.workunit.client.0.smithi032.stdout:4/649: symlink d4/d87/d96/d2f/d7c/dc2/ldd 0 2024-04-12T05:37:11.117 INFO:tasks.workunit.client.1.smithi067.stdout:2/162: chown d5/l2b 249 1 2024-04-12T05:37:11.119 INFO:tasks.workunit.client.0.smithi032.stdout:1/575: creat d5/de/d17/d35/d61/d77/d8f/d9c/fb5 x:0 0 0 2024-04-12T05:37:11.119 INFO:tasks.workunit.client.1.smithi067.stdout:5/214: truncate d0/d3a/d2c/f37 4734668 0 2024-04-12T05:37:11.119 INFO:tasks.workunit.client.1.smithi067.stdout:2/163: chown d5/d23/d3a 42 1 2024-04-12T05:37:11.119 INFO:tasks.workunit.client.1.smithi067.stdout:0/201: getdents dd/d12/d14/d1a 0 2024-04-12T05:37:11.125 INFO:tasks.workunit.client.1.smithi067.stdout:0/202: creat dd/d12/d14/f43 x:0 0 0 2024-04-12T05:37:11.126 INFO:tasks.workunit.client.1.smithi067.stdout:5/215: truncate d0/da/d27/f18 724996 0 2024-04-12T05:37:11.127 INFO:tasks.workunit.client.1.smithi067.stdout:5/216: write d0/f43 [468041,91175] 0 2024-04-12T05:37:11.129 INFO:tasks.workunit.client.1.smithi067.stdout:0/203: creat dd/d40/f44 x:0 0 0 2024-04-12T05:37:11.129 INFO:tasks.workunit.client.0.smithi032.stdout:6/839: symlink d4/de0/de7/d79/l123 0 2024-04-12T05:37:11.131 INFO:tasks.workunit.client.0.smithi032.stdout:5/691: symlink d2/ldc 0 2024-04-12T05:37:11.131 INFO:tasks.workunit.client.1.smithi067.stdout:0/204: read dd/d12/d14/f33 [625626,15104] 0 2024-04-12T05:37:11.132 INFO:tasks.workunit.client.1.smithi067.stdout:0/205: rename dd/d12 to dd/d12/d14/d1a/d3d/d41/d45 22 2024-04-12T05:37:11.134 INFO:tasks.workunit.client.0.smithi032.stdout:7/702: creat d0/de/df/d29/fec x:0 0 0 2024-04-12T05:37:11.136 INFO:tasks.workunit.client.0.smithi032.stdout:8/719: truncate d0/f2f 4337880 0 2024-04-12T05:37:11.136 INFO:tasks.workunit.client.1.smithi067.stdout:0/206: symlink dd/d12/d14/d1a/d3d/l46 0 2024-04-12T05:37:11.138 INFO:tasks.workunit.client.0.smithi032.stdout:1/576: creat d5/de/d2a/fb6 x:0 0 0 2024-04-12T05:37:11.139 INFO:tasks.workunit.client.0.smithi032.stdout:3/694: write d12/d2e/d3a/d63/fcf [260805,33801] 0 2024-04-12T05:37:11.140 INFO:tasks.workunit.client.1.smithi067.stdout:0/207: write dd/d12/d14/d1a/f1e [137392,35153] 0 2024-04-12T05:37:11.140 INFO:tasks.workunit.client.1.smithi067.stdout:5/217: truncate d0/d6/d33/f3c 112225 0 2024-04-12T05:37:11.144 INFO:tasks.workunit.client.1.smithi067.stdout:5/218: mknod d0/da/d41/c48 0 2024-04-12T05:37:11.148 INFO:tasks.workunit.client.1.smithi067.stdout:5/219: rename d0/fc to d0/da/d3f/f49 0 2024-04-12T05:37:11.153 INFO:tasks.workunit.client.1.smithi067.stdout:5/220: dread d0/da/d3f/f49 [4194304,4194304] 0 2024-04-12T05:37:11.155 INFO:tasks.workunit.client.1.smithi067.stdout:5/221: truncate d0/da/f2e 2088365 0 2024-04-12T05:37:11.156 INFO:tasks.workunit.client.1.smithi067.stdout:5/222: fdatasync d0/da/d27/f15 0 2024-04-12T05:37:11.157 INFO:tasks.workunit.client.1.smithi067.stdout:0/208: dwrite f1 [0,4194304] 0 2024-04-12T05:37:11.158 INFO:tasks.workunit.client.1.smithi067.stdout:0/209: truncate dd/f42 794163 0 2024-04-12T05:37:11.159 INFO:tasks.workunit.client.1.smithi067.stdout:0/210: chown dd/l2c 113 1 2024-04-12T05:37:11.159 INFO:tasks.workunit.client.1.smithi067.stdout:5/223: mknod d0/d6/d33/c4a 0 2024-04-12T05:37:11.160 INFO:tasks.workunit.client.1.smithi067.stdout:0/211: write dd/d12/d14/d1a/f1e [568432,37760] 0 2024-04-12T05:37:11.164 INFO:tasks.workunit.client.1.smithi067.stdout:5/224: rename d0/da/d2d to d0/d6/d40/d4b 0 2024-04-12T05:37:11.165 INFO:tasks.workunit.client.1.smithi067.stdout:0/212: truncate dd/d12/f21 3879568 0 2024-04-12T05:37:11.166 INFO:tasks.workunit.client.1.smithi067.stdout:5/225: creat d0/d3a/f4c x:0 0 0 2024-04-12T05:37:11.166 INFO:tasks.workunit.client.1.smithi067.stdout:0/213: read dd/d12/d14/d1a/d3d/f3c [3647890,61647] 0 2024-04-12T05:37:11.169 INFO:tasks.workunit.client.1.smithi067.stdout:5/226: unlink d0/d6/f1d 0 2024-04-12T05:37:11.173 INFO:tasks.workunit.client.1.smithi067.stdout:5/227: mknod d0/d6/c4d 0 2024-04-12T05:37:11.173 INFO:tasks.workunit.client.1.smithi067.stdout:5/228: chown d0/d3a/d2c 172109 1 2024-04-12T05:37:11.176 INFO:tasks.workunit.client.1.smithi067.stdout:0/214: link dd/d12/d14/c1f dd/d12/d14/d1a/d3d/d41/c47 0 2024-04-12T05:37:11.179 INFO:tasks.workunit.client.1.smithi067.stdout:0/215: symlink dd/d12/l48 0 2024-04-12T05:37:11.181 INFO:tasks.workunit.client.1.smithi067.stdout:5/229: rename d0/d6/d20/d28 to d0/d6/d40/d4b/d38/d4e 0 2024-04-12T05:37:11.182 INFO:tasks.workunit.client.1.smithi067.stdout:5/230: write d0/d3a/f4c [778605,73007] 0 2024-04-12T05:37:11.187 INFO:tasks.workunit.client.1.smithi067.stdout:5/231: dread d0/d3a/d2c/f37 [0,4194304] 0 2024-04-12T05:37:11.188 INFO:tasks.workunit.client.1.smithi067.stdout:0/216: rmdir dd 39 2024-04-12T05:37:11.189 INFO:tasks.workunit.client.1.smithi067.stdout:5/232: write d0/f1c [2829167,7499] 0 2024-04-12T05:37:11.189 INFO:tasks.workunit.client.1.smithi067.stdout:5/233: fsync d0/da/d27/f15 0 2024-04-12T05:37:11.193 INFO:tasks.workunit.client.1.smithi067.stdout:5/234: mknod d0/d3a/d2c/c4f 0 2024-04-12T05:37:11.195 INFO:tasks.workunit.client.1.smithi067.stdout:0/217: mkdir dd/d12/d14/d49 0 2024-04-12T05:37:11.197 INFO:tasks.workunit.client.1.smithi067.stdout:5/235: dread d0/f7 [0,4194304] 0 2024-04-12T05:37:11.201 INFO:tasks.workunit.client.1.smithi067.stdout:0/218: rename dd/d12/l35 to dd/d12/d14/d1a/d3d/d41/l4a 0 2024-04-12T05:37:11.201 INFO:tasks.workunit.client.0.smithi032.stdout:0/703: mknod d2/d29/dcc/ce5 0 2024-04-12T05:37:11.203 INFO:tasks.workunit.client.1.smithi067.stdout:5/236: mknod d0/d6/d20/c50 0 2024-04-12T05:37:11.203 INFO:tasks.workunit.client.1.smithi067.stdout:3/79: sync 2024-04-12T05:37:11.204 INFO:tasks.workunit.client.1.smithi067.stdout:5/237: chown d0/f7 9760 1 2024-04-12T05:37:11.206 INFO:tasks.workunit.client.1.smithi067.stdout:3/80: symlink dc/df/l15 0 2024-04-12T05:37:11.206 INFO:tasks.workunit.client.1.smithi067.stdout:0/219: getdents dd/d12/d14/d1a 0 2024-04-12T05:37:11.206 INFO:tasks.workunit.client.0.smithi032.stdout:5/692: creat d2/df/d25/d50/daa/fdd x:0 0 0 2024-04-12T05:37:11.208 INFO:tasks.workunit.client.0.smithi032.stdout:7/703: write d0/dd/d51/d65/f96 [4856188,65118] 0 2024-04-12T05:37:11.211 INFO:tasks.workunit.client.1.smithi067.stdout:0/220: rename dd/d12/d14/f33 to dd/d12/d14/d1a/f4b 0 2024-04-12T05:37:11.212 INFO:tasks.workunit.client.1.smithi067.stdout:0/221: read - dd/d12/f3e zero size 2024-04-12T05:37:11.213 INFO:tasks.workunit.client.1.smithi067.stdout:0/222: fsync dd/d12/f29 0 2024-04-12T05:37:11.213 INFO:tasks.workunit.client.0.smithi032.stdout:0/704: dread d2/d8/df/f31 [0,4194304] 0 2024-04-12T05:37:11.215 INFO:tasks.workunit.client.0.smithi032.stdout:8/720: symlink d0/db/dd/d11/d1b/d5f/lff 0 2024-04-12T05:37:11.217 INFO:tasks.workunit.client.1.smithi067.stdout:0/223: fdatasync dd/d12/d14/d1a/d3d/f2f 0 2024-04-12T05:37:11.218 INFO:tasks.workunit.client.1.smithi067.stdout:3/81: rmdir dc 39 2024-04-12T05:37:11.219 INFO:tasks.workunit.client.1.smithi067.stdout:5/238: write d0/d6/d33/f3c [537689,38242] 0 2024-04-12T05:37:11.222 INFO:tasks.workunit.client.1.smithi067.stdout:3/82: chown dc/c12 47280311 1 2024-04-12T05:37:11.223 INFO:tasks.workunit.client.1.smithi067.stdout:5/239: symlink d0/d3a/l51 0 2024-04-12T05:37:11.225 INFO:tasks.workunit.client.1.smithi067.stdout:3/83: write f9 [3465729,121591] 0 2024-04-12T05:37:11.226 INFO:tasks.workunit.client.1.smithi067.stdout:0/224: dwrite dd/f13 [4194304,4194304] 0 2024-04-12T05:37:11.231 INFO:tasks.workunit.client.0.smithi032.stdout:6/840: dwrite d4/d21/d27/dad/d55/f70 [0,4194304] 0 2024-04-12T05:37:11.236 INFO:tasks.workunit.client.1.smithi067.stdout:3/84: mknod dc/df/c16 0 2024-04-12T05:37:11.237 INFO:tasks.workunit.client.1.smithi067.stdout:5/240: write d0/da/d27/f16 [380073,114513] 0 2024-04-12T05:37:11.238 INFO:tasks.workunit.client.1.smithi067.stdout:5/241: chown d0/d6/d40/d4b/c3b 1 1 2024-04-12T05:37:11.238 INFO:tasks.workunit.client.1.smithi067.stdout:3/85: mknod dc/df/c17 0 2024-04-12T05:37:11.241 INFO:tasks.workunit.client.0.smithi032.stdout:7/704: symlink d0/d13/d21/d7c/led 0 2024-04-12T05:37:11.244 INFO:tasks.workunit.client.0.smithi032.stdout:0/705: creat d2/d8/d48/fe6 x:0 0 0 2024-04-12T05:37:11.245 INFO:tasks.workunit.client.0.smithi032.stdout:8/721: rmdir d0/db/dd/d11/d1b/d22/d29 39 2024-04-12T05:37:11.245 INFO:tasks.workunit.client.0.smithi032.stdout:2/701: link dd/d19/lce dd/d19/d9d/d1c/d64/d92/le6 0 2024-04-12T05:37:11.245 INFO:tasks.workunit.client.0.smithi032.stdout:0/706: chown d2/dc/d41/lc5 197480925 1 2024-04-12T05:37:11.247 INFO:tasks.workunit.client.0.smithi032.stdout:4/650: link d4/d87/d96/d32/c36 d4/d13/cde 0 2024-04-12T05:37:11.248 INFO:tasks.workunit.client.0.smithi032.stdout:5/693: dread d2/dd/fb9 [0,4194304] 0 2024-04-12T05:37:11.248 INFO:tasks.workunit.client.1.smithi067.stdout:5/242: rename d0/d36/f45 to d0/d6/d33/f52 0 2024-04-12T05:37:11.251 INFO:tasks.workunit.client.1.smithi067.stdout:3/86: rename dc/df/f14 to dc/f18 0 2024-04-12T05:37:11.252 INFO:tasks.workunit.client.0.smithi032.stdout:5/694: write d2/df/d25/d50/daa/fdd [976969,120639] 0 2024-04-12T05:37:11.254 INFO:tasks.workunit.client.1.smithi067.stdout:5/243: unlink d0/da/f1b 0 2024-04-12T05:37:11.257 INFO:tasks.workunit.client.1.smithi067.stdout:5/244: write d0/d6/d33/f3c [518749,61252] 0 2024-04-12T05:37:11.261 INFO:tasks.workunit.client.1.smithi067.stdout:3/87: link la dc/l19 0 2024-04-12T05:37:11.262 INFO:tasks.workunit.client.1.smithi067.stdout:5/245: rename d0/d6/d33/c4a to d0/da/c53 0 2024-04-12T05:37:11.263 INFO:tasks.workunit.client.0.smithi032.stdout:3/695: getdents d12/d2e/d3a 0 2024-04-12T05:37:11.263 INFO:tasks.workunit.client.0.smithi032.stdout:8/722: symlink d0/d36/de2/l100 0 2024-04-12T05:37:11.264 INFO:tasks.workunit.client.0.smithi032.stdout:6/841: link d4/d21/db4/l107 d4/de0/deb/dec/l124 0 2024-04-12T05:37:11.265 INFO:tasks.workunit.client.0.smithi032.stdout:3/696: dread - d12/d2e/f50 zero size 2024-04-12T05:37:11.267 INFO:tasks.workunit.client.0.smithi032.stdout:5/695: dwrite d2/f60 [0,4194304] 0 2024-04-12T05:37:11.268 INFO:tasks.workunit.client.0.smithi032.stdout:0/707: mkdir d2/d8/d17/d1b/de7 0 2024-04-12T05:37:11.268 INFO:tasks.workunit.client.0.smithi032.stdout:8/723: stat d0/db/d7d/dfb 0 2024-04-12T05:37:11.269 INFO:tasks.workunit.client.0.smithi032.stdout:7/705: link d0/dd/ld0 d0/dd/d51/d65/lee 0 2024-04-12T05:37:11.272 INFO:tasks.workunit.client.0.smithi032.stdout:4/651: dread d4/d87/d96/d4d/d78/f83 [0,4194304] 0 2024-04-12T05:37:11.274 INFO:tasks.workunit.client.1.smithi067.stdout:3/88: dwrite dc/f18 [0,4194304] 0 2024-04-12T05:37:11.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.277+0000 7ff28f60e700 1 -- 172.21.15.32:0/2260689582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2880fe4a0 msgr2=0x7ff2880fe8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:11.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.277+0000 7ff28f60e700 1 --2- 172.21.15.32:0/2260689582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2880fe4a0 0x7ff2880fe8b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff27c004940 tx=0x7ff27c009e30 comp rx=0 tx=0).stop 2024-04-12T05:37:11.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.277+0000 7ff28f60e700 1 -- 172.21.15.32:0/2260689582 shutdown_connections 2024-04-12T05:37:11.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.277+0000 7ff28f60e700 1 --2- 172.21.15.32:0/2260689582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2880fedf0 0x7ff2880faf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:11.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.277+0000 7ff28f60e700 1 --2- 172.21.15.32:0/2260689582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2880fe4a0 0x7ff2880fe8b0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:11.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.277+0000 7ff28f60e700 1 -- 172.21.15.32:0/2260689582 >> 172.21.15.32:0/2260689582 conn(0x7ff2880f6ba0 msgr2=0x7ff2880f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:11.275 INFO:tasks.workunit.client.0.smithi032.stdout:4/652: write d4/d13/d19/fa1 [694963,24242] 0 2024-04-12T05:37:11.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.277+0000 7ff28f60e700 1 -- 172.21.15.32:0/2260689582 shutdown_connections 2024-04-12T05:37:11.276 INFO:tasks.workunit.client.0.smithi032.stdout:3/697: readlink d12/l92 0 2024-04-12T05:37:11.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.278+0000 7ff28f60e700 1 -- 172.21.15.32:0/2260689582 wait complete. 2024-04-12T05:37:11.277 INFO:tasks.workunit.client.0.smithi032.stdout:4/653: chown d4/l7 988413 1 2024-04-12T05:37:11.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.278+0000 7ff28f60e700 1 Processor -- start 2024-04-12T05:37:11.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.279+0000 7ff28f60e700 1 -- start start 2024-04-12T05:37:11.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.279+0000 7ff28f60e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2880fe4a0 0x7ff28810fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:37:11.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.279+0000 7ff28f60e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2880fedf0 0x7ff288110070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:37:11.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.279+0000 7ff28f60e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff288110670 con 0x7ff2880fe4a0 2024-04-12T05:37:11.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.279+0000 7ff28f60e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff288110ec0 con 0x7ff2880fedf0 2024-04-12T05:37:11.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.279+0000 7ff28d3aa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2880fe4a0 0x7ff28810fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:11.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.279+0000 7ff28d3aa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2880fe4a0 0x7ff28810fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56640/0 (socket says 172.21.15.32:56640) 2024-04-12T05:37:11.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.279+0000 7ff28d3aa700 1 -- 172.21.15.32:0/932033185 learned_addr learned my addr 172.21.15.32:0/932033185 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:37:11.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.279+0000 7ff28d3aa700 1 -- 172.21.15.32:0/932033185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2880fedf0 msgr2=0x7ff288110070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:11.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.280+0000 7ff28d3aa700 1 --2- 172.21.15.32:0/932033185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2880fedf0 0x7ff288110070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:11.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.280+0000 7ff28d3aa700 1 -- 172.21.15.32:0/932033185 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff27c005040 con 0x7ff2880fe4a0 2024-04-12T05:37:11.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.281+0000 7ff28d3aa700 1 --2- 172.21.15.32:0/932033185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2880fe4a0 0x7ff28810fb30 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7ff27c009fd0 tx=0x7ff27c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:11.280 INFO:tasks.workunit.client.0.smithi032.stdout:8/724: mknod d0/db/d39/c101 0 2024-04-12T05:37:11.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.283+0000 7ff285ffb700 1 -- 172.21.15.32:0/932033185 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff27c00e040 con 0x7ff2880fe4a0 2024-04-12T05:37:11.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.283+0000 7ff28f60e700 1 -- 172.21.15.32:0/932033185 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff288111140 con 0x7ff2880fe4a0 2024-04-12T05:37:11.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.283+0000 7ff28f60e700 1 -- 172.21.15.32:0/932033185 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff288111680 con 0x7ff2880fe4a0 2024-04-12T05:37:11.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.283+0000 7ff285ffb700 1 -- 172.21.15.32:0/932033185 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff27c004290 con 0x7ff2880fe4a0 2024-04-12T05:37:11.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.283+0000 7ff285ffb700 1 -- 172.21.15.32:0/932033185 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff27c013e80 con 0x7ff2880fe4a0 2024-04-12T05:37:11.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.285+0000 7ff28f60e700 1 -- 172.21.15.32:0/932033185 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2740052e0 con 0x7ff2880fe4a0 2024-04-12T05:37:11.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.287+0000 7ff285ffb700 1 -- 172.21.15.32:0/932033185 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ff27c00bd50 con 0x7ff2880fe4a0 2024-04-12T05:37:11.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.287+0000 7ff285ffb700 1 --2- 172.21.15.32:0/932033185 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7ff2700692b0 0x7ff27006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:37:11.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.288+0000 7ff285ffb700 1 -- 172.21.15.32:0/932033185 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ff27c088ab0 con 0x7ff2880fe4a0 2024-04-12T05:37:11.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.288+0000 7ff28cba9700 1 --2- 172.21.15.32:0/932033185 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7ff2700692b0 0x7ff27006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:11.286 INFO:tasks.workunit.client.0.smithi032.stdout:5/696: dwrite d2/d9b/f5a [0,4194304] 0 2024-04-12T05:37:11.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.288+0000 7ff28cba9700 1 --2- 172.21.15.32:0/932033185 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7ff2700692b0 0x7ff27006b760 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff280004d00 tx=0x7ff280009ce0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:11.287 INFO:tasks.workunit.client.0.smithi032.stdout:5/697: dread - d2/dd/d6f/fbd zero size 2024-04-12T05:37:11.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.296+0000 7ff285ffb700 1 -- 172.21.15.32:0/932033185 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ff27c0588c0 con 0x7ff2880fe4a0 2024-04-12T05:37:11.298 INFO:tasks.workunit.client.1.smithi067.stdout:5/246: truncate d0/da/d27/f16 3372564 0 2024-04-12T05:37:11.299 INFO:tasks.workunit.client.1.smithi067.stdout:5/247: rmdir d0/da/d41 39 2024-04-12T05:37:11.300 INFO:tasks.workunit.client.1.smithi067.stdout:5/248: fsync d0/f7 0 2024-04-12T05:37:11.302 INFO:tasks.workunit.client.1.smithi067.stdout:3/89: dwrite f1 [0,4194304] 0 2024-04-12T05:37:11.310 INFO:tasks.workunit.client.1.smithi067.stdout:5/249: fdatasync d0/da/d27/f18 0 2024-04-12T05:37:11.314 INFO:tasks.workunit.client.1.smithi067.stdout:5/250: rmdir d0/d36 0 2024-04-12T05:37:11.316 INFO:tasks.workunit.client.1.smithi067.stdout:3/90: dwrite f5 [0,4194304] 0 2024-04-12T05:37:11.316 INFO:tasks.workunit.client.1.smithi067.stdout:5/251: readlink d0/da/d27/l22 0 2024-04-12T05:37:11.322 INFO:tasks.workunit.client.1.smithi067.stdout:3/91: mknod dc/df/c1a 0 2024-04-12T05:37:11.324 INFO:tasks.workunit.client.0.smithi032.stdout:7/706: creat d0/d34/dbf/fef x:0 0 0 2024-04-12T05:37:11.326 INFO:tasks.workunit.client.0.smithi032.stdout:7/707: stat d0/d13/d21/d81/f9d 0 2024-04-12T05:37:11.326 INFO:tasks.workunit.client.1.smithi067.stdout:3/92: write f5 [5018385,122774] 0 2024-04-12T05:37:11.327 INFO:tasks.workunit.client.0.smithi032.stdout:3/698: symlink d12/d2e/dcb/le1 0 2024-04-12T05:37:11.327 INFO:tasks.workunit.client.1.smithi067.stdout:5/252: creat d0/d3a/f54 x:0 0 0 2024-04-12T05:37:11.330 INFO:tasks.workunit.client.1.smithi067.stdout:5/253: chown d0/d3a/f4c 10 1 2024-04-12T05:37:11.330 INFO:tasks.workunit.client.1.smithi067.stdout:5/254: fdatasync d0/d6/f12 0 2024-04-12T05:37:11.335 INFO:tasks.workunit.client.0.smithi032.stdout:0/708: getdents d2/d29/d77/dc2 0 2024-04-12T05:37:11.341 INFO:tasks.workunit.client.0.smithi032.stdout:3/699: truncate d12/d2e/d3a/d63/da6/dac/fd0 845063 0 2024-04-12T05:37:11.356 INFO:tasks.workunit.client.0.smithi032.stdout:3/700: dwrite d12/d2e/fb5 [0,4194304] 0 2024-04-12T05:37:11.357 INFO:tasks.workunit.client.0.smithi032.stdout:3/701: chown d12/d24/l5c 659795 1 2024-04-12T05:37:11.376 INFO:tasks.workunit.client.0.smithi032.stdout:7/708: mknod d0/de/df/cf0 0 2024-04-12T05:37:11.376 INFO:tasks.workunit.client.0.smithi032.stdout:4/654: creat d4/d13/d19/fdf x:0 0 0 2024-04-12T05:37:11.376 INFO:tasks.workunit.client.0.smithi032.stdout:7/709: fsync d0/d1c/d71/f85 0 2024-04-12T05:37:11.379 INFO:tasks.workunit.client.1.smithi067.stdout:4/58: sync 2024-04-12T05:37:11.379 INFO:tasks.workunit.client.1.smithi067.stdout:9/89: sync 2024-04-12T05:37:11.379 INFO:tasks.workunit.client.1.smithi067.stdout:7/95: sync 2024-04-12T05:37:11.379 INFO:tasks.workunit.client.1.smithi067.stdout:6/66: sync 2024-04-12T05:37:11.379 INFO:tasks.workunit.client.1.smithi067.stdout:6/67: fdatasync fa 0 2024-04-12T05:37:11.380 INFO:tasks.workunit.client.1.smithi067.stdout:7/96: chown d0/f12 4085 1 2024-04-12T05:37:11.381 INFO:tasks.workunit.client.1.smithi067.stdout:7/97: readlink d0/l1b 0 2024-04-12T05:37:11.382 INFO:tasks.workunit.client.1.smithi067.stdout:6/68: write fb [487756,19157] 0 2024-04-12T05:37:11.389 INFO:tasks.workunit.client.1.smithi067.stdout:9/90: dwrite d1/f2 [0,4194304] 0 2024-04-12T05:37:11.398 INFO:tasks.workunit.client.1.smithi067.stdout:3/93: symlink dc/df/l1b 0 2024-04-12T05:37:11.406 INFO:tasks.workunit.client.1.smithi067.stdout:4/59: mkdir d8/d14 0 2024-04-12T05:37:11.407 INFO:tasks.workunit.client.1.smithi067.stdout:7/98: unlink d0/d2/d5/dc/c19 0 2024-04-12T05:37:11.408 INFO:tasks.workunit.client.1.smithi067.stdout:3/94: creat dc/df/f1c x:0 0 0 2024-04-12T05:37:11.410 INFO:tasks.workunit.client.1.smithi067.stdout:7/99: readlink d0/l1b 0 2024-04-12T05:37:11.410 INFO:tasks.workunit.client.1.smithi067.stdout:9/91: symlink d1/d12/d15/l17 0 2024-04-12T05:37:11.411 INFO:tasks.workunit.client.1.smithi067.stdout:3/95: mkdir dc/d1d 0 2024-04-12T05:37:11.413 INFO:tasks.workunit.client.1.smithi067.stdout:4/60: mkdir d8/d14/d15 0 2024-04-12T05:37:11.415 INFO:tasks.workunit.client.1.smithi067.stdout:3/96: creat dc/df/f1e x:0 0 0 2024-04-12T05:37:11.416 INFO:tasks.workunit.client.1.smithi067.stdout:7/100: write d0/f8 [837714,16460] 0 2024-04-12T05:37:11.418 INFO:tasks.workunit.client.1.smithi067.stdout:3/97: write dc/df/f1c [849803,56172] 0 2024-04-12T05:37:11.445 INFO:tasks.workunit.client.1.smithi067.stdout:3/98: creat dc/d1d/f1f x:0 0 0 2024-04-12T05:37:11.446 INFO:tasks.workunit.client.0.smithi032.stdout:1/577: sync 2024-04-12T05:37:11.446 INFO:tasks.workunit.client.0.smithi032.stdout:9/557: sync 2024-04-12T05:37:11.447 INFO:tasks.workunit.client.1.smithi067.stdout:5/255: truncate d0/da/d3f/f49 2318020 0 2024-04-12T05:37:11.448 INFO:tasks.workunit.client.1.smithi067.stdout:3/99: symlink dc/df/l20 0 2024-04-12T05:37:11.450 INFO:tasks.workunit.client.1.smithi067.stdout:5/256: mknod d0/d6/c55 0 2024-04-12T05:37:11.453 INFO:tasks.workunit.client.1.smithi067.stdout:6/69: dwrite f0 [0,4194304] 0 2024-04-12T05:37:11.453 INFO:tasks.workunit.client.1.smithi067.stdout:9/92: truncate d1/f2 3682163 0 2024-04-12T05:37:11.455 INFO:tasks.workunit.client.1.smithi067.stdout:5/257: unlink d0/da/c53 0 2024-04-12T05:37:11.457 INFO:tasks.workunit.client.1.smithi067.stdout:7/101: truncate d0/f8 2082208 0 2024-04-12T05:37:11.457 INFO:tasks.workunit.client.1.smithi067.stdout:7/102: chown d0 6 1 2024-04-12T05:37:11.460 INFO:tasks.workunit.client.1.smithi067.stdout:7/103: write d0/d2/d5/db/f18 [990527,112748] 0 2024-04-12T05:37:11.463 INFO:tasks.workunit.client.1.smithi067.stdout:6/70: mknod dd/d11/c12 0 2024-04-12T05:37:11.466 INFO:tasks.workunit.client.1.smithi067.stdout:9/93: dwrite d1/f7 [0,4194304] 0 2024-04-12T05:37:11.467 INFO:tasks.workunit.client.1.smithi067.stdout:5/258: dread d0/d3a/f4c [0,4194304] 0 2024-04-12T05:37:11.474 INFO:tasks.workunit.client.1.smithi067.stdout:7/104: dread d0/d2/ff [0,4194304] 0 2024-04-12T05:37:11.477 INFO:tasks.workunit.client.1.smithi067.stdout:5/259: dread d0/da/d27/f14 [0,4194304] 0 2024-04-12T05:37:11.479 INFO:tasks.workunit.client.1.smithi067.stdout:7/105: creat d0/d2/d5/db/f1f x:0 0 0 2024-04-12T05:37:11.479 INFO:tasks.workunit.client.1.smithi067.stdout:6/71: rmdir dd 39 2024-04-12T05:37:11.479 INFO:tasks.workunit.client.1.smithi067.stdout:7/106: stat d0/d2/d5/d1d 0 2024-04-12T05:37:11.480 INFO:tasks.workunit.client.1.smithi067.stdout:6/72: dread - fa zero size 2024-04-12T05:37:11.480 INFO:tasks.workunit.client.1.smithi067.stdout:9/94: truncate d1/f7 76761 0 2024-04-12T05:37:11.482 INFO:tasks.workunit.client.1.smithi067.stdout:7/107: write d0/f1 [1345998,6159] 0 2024-04-12T05:37:11.485 INFO:tasks.workunit.client.1.smithi067.stdout:6/73: symlink dd/l13 0 2024-04-12T05:37:11.488 INFO:tasks.workunit.client.1.smithi067.stdout:6/74: fsync fa 0 2024-04-12T05:37:11.488 INFO:tasks.workunit.client.1.smithi067.stdout:5/260: truncate d0/d6/f12 1402365 0 2024-04-12T05:37:11.489 INFO:tasks.workunit.client.1.smithi067.stdout:5/261: readlink d0/d6/d20/l21 0 2024-04-12T05:37:11.497 INFO:tasks.workunit.client.1.smithi067.stdout:7/108: creat d0/d2/d5/d1e/f20 x:0 0 0 2024-04-12T05:37:11.497 INFO:tasks.workunit.client.1.smithi067.stdout:7/109: read d0/d2/d5/fe [2074771,115530] 0 2024-04-12T05:37:11.498 INFO:tasks.workunit.client.1.smithi067.stdout:6/75: dread dd/fe [0,4194304] 0 2024-04-12T05:37:11.504 INFO:tasks.workunit.client.1.smithi067.stdout:9/95: dwrite d1/f7 [0,4194304] 0 2024-04-12T05:37:11.505 INFO:tasks.workunit.client.1.smithi067.stdout:6/76: dread dd/f10 [0,4194304] 0 2024-04-12T05:37:11.508 INFO:tasks.workunit.client.1.smithi067.stdout:5/262: dread d0/f1c [0,4194304] 0 2024-04-12T05:37:11.511 INFO:tasks.workunit.client.1.smithi067.stdout:5/263: dread d0/d6/f12 [0,4194304] 0 2024-04-12T05:37:11.512 INFO:tasks.workunit.client.1.smithi067.stdout:6/77: creat dd/d11/f14 x:0 0 0 2024-04-12T05:37:11.514 INFO:tasks.workunit.client.1.smithi067.stdout:7/110: write d0/d2/d5/f11 [527661,86970] 0 2024-04-12T05:37:11.514 INFO:tasks.workunit.client.1.smithi067.stdout:9/96: getdents d1/d12 0 2024-04-12T05:37:11.516 INFO:tasks.workunit.client.1.smithi067.stdout:6/78: write dd/fe [823772,127983] 0 2024-04-12T05:37:11.518 INFO:tasks.workunit.client.1.smithi067.stdout:9/97: creat d1/d12/d16/f18 x:0 0 0 2024-04-12T05:37:11.520 INFO:tasks.workunit.client.1.smithi067.stdout:9/98: fsync d1/f11 0 2024-04-12T05:37:11.521 INFO:tasks.workunit.client.1.smithi067.stdout:7/111: creat d0/d2/f21 x:0 0 0 2024-04-12T05:37:11.521 INFO:tasks.workunit.client.1.smithi067.stdout:9/99: readlink d1/lb 0 2024-04-12T05:37:11.522 INFO:tasks.workunit.client.1.smithi067.stdout:7/112: read - d0/d2/f21 zero size 2024-04-12T05:37:11.523 INFO:tasks.workunit.client.1.smithi067.stdout:9/100: fdatasync d1/f6 0 2024-04-12T05:37:11.525 INFO:tasks.workunit.client.1.smithi067.stdout:6/79: dread dd/fe [0,4194304] 0 2024-04-12T05:37:11.525 INFO:tasks.workunit.client.1.smithi067.stdout:9/101: truncate d1/d12/f14 510175 0 2024-04-12T05:37:11.527 INFO:tasks.workunit.client.1.smithi067.stdout:9/102: chown d1/d12/d16/f18 976222291 1 2024-04-12T05:37:11.528 INFO:tasks.workunit.client.1.smithi067.stdout:7/113: rename d0/d2/d5/f1c to d0/d2/d5/d1e/f22 0 2024-04-12T05:37:11.531 INFO:tasks.workunit.client.1.smithi067.stdout:6/80: rename dd/lf to dd/l15 0 2024-04-12T05:37:11.531 INFO:tasks.workunit.client.1.smithi067.stdout:9/103: mknod d1/d12/c19 0 2024-04-12T05:37:11.535 INFO:tasks.workunit.client.1.smithi067.stdout:6/81: rename dd to dd/d11/d16 22 2024-04-12T05:37:11.535 INFO:tasks.workunit.client.1.smithi067.stdout:7/114: dread d0/d2/d5/db/f18 [0,4194304] 0 2024-04-12T05:37:11.535 INFO:tasks.workunit.client.1.smithi067.stdout:5/264: dread d0/da/d27/f15 [0,4194304] 0 2024-04-12T05:37:11.535 INFO:tasks.workunit.client.1.smithi067.stdout:9/104: unlink c0 0 2024-04-12T05:37:11.536 INFO:tasks.workunit.client.1.smithi067.stdout:7/115: dread - d0/d2/d5/db/d10/f1a zero size 2024-04-12T05:37:11.536 INFO:tasks.workunit.client.1.smithi067.stdout:9/105: stat d1/fd 0 2024-04-12T05:37:11.538 INFO:tasks.workunit.client.0.smithi032.stdout:2/702: sync 2024-04-12T05:37:11.539 INFO:tasks.workunit.client.1.smithi067.stdout:9/106: write d1/d12/f14 [245815,55908] 0 2024-04-12T05:37:11.540 INFO:tasks.workunit.client.1.smithi067.stdout:7/116: mkdir d0/d23 0 2024-04-12T05:37:11.546 INFO:tasks.workunit.client.1.smithi067.stdout:5/265: truncate d0/da/d27/f15 2353082 0 2024-04-12T05:37:11.549 INFO:tasks.workunit.client.1.smithi067.stdout:6/82: dwrite dd/fe [4194304,4194304] 0 2024-04-12T05:37:11.553 INFO:tasks.workunit.client.1.smithi067.stdout:6/83: write f0 [3316366,86035] 0 2024-04-12T05:37:11.559 INFO:tasks.workunit.client.1.smithi067.stdout:6/84: creat dd/f17 x:0 0 0 2024-04-12T05:37:11.559 INFO:tasks.workunit.client.1.smithi067.stdout:6/85: chown dd/l13 31704765 1 2024-04-12T05:37:11.560 INFO:tasks.workunit.client.1.smithi067.stdout:5/266: rename d0/da/d27/f16 to d0/d6/f56 0 2024-04-12T05:37:11.560 INFO:tasks.workunit.client.1.smithi067.stdout:7/117: link d0/d2/d5/dc/l17 d0/d2/d5/d1d/l24 0 2024-04-12T05:37:11.560 INFO:tasks.workunit.client.1.smithi067.stdout:6/86: fdatasync dd/d11/f14 0 2024-04-12T05:37:11.561 INFO:tasks.workunit.client.1.smithi067.stdout:7/118: readlink d0/l1b 0 2024-04-12T05:37:11.561 INFO:tasks.workunit.client.1.smithi067.stdout:6/87: readlink l5 0 2024-04-12T05:37:11.562 INFO:tasks.workunit.client.1.smithi067.stdout:6/88: read - dd/f17 zero size 2024-04-12T05:37:11.567 INFO:tasks.workunit.client.1.smithi067.stdout:5/267: creat d0/da/f57 x:0 0 0 2024-04-12T05:37:11.568 INFO:tasks.workunit.client.1.smithi067.stdout:7/119: symlink d0/d2/d5/l25 0 2024-04-12T05:37:11.571 INFO:tasks.workunit.client.1.smithi067.stdout:5/268: chown d0/da/d27/c13 1388256 1 2024-04-12T05:37:11.574 INFO:tasks.workunit.client.1.smithi067.stdout:6/89: rename dd/l15 to dd/d11/l18 0 2024-04-12T05:37:11.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.577+0000 7ff28f60e700 1 -- 172.21.15.32:0/932033185 --> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff274000bf0 con 0x7ff2700692b0 2024-04-12T05:37:11.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.578+0000 7ff285ffb700 1 -- 172.21.15.32:0/932033185 <== mgr.14208 v2:172.21.15.32:6800/595990262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+321 (secure 0 0 0) 0x7ff274000bf0 con 0x7ff2700692b0 2024-04-12T05:37:11.576 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:37:11.576 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:37:11.576 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:37:11.576 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-12T05:37:11.576 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [], 2024-04-12T05:37:11.576 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "", 2024-04-12T05:37:11.576 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mgr daemons", 2024-04-12T05:37:11.576 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:37:11.576 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:37:11.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.580+0000 7ff28f60e700 1 -- 172.21.15.32:0/932033185 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7ff2700692b0 msgr2=0x7ff27006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:11.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.580+0000 7ff28f60e700 1 --2- 172.21.15.32:0/932033185 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7ff2700692b0 0x7ff27006b760 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff280004d00 tx=0x7ff280009ce0 comp rx=0 tx=0).stop 2024-04-12T05:37:11.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.580+0000 7ff28f60e700 1 -- 172.21.15.32:0/932033185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2880fe4a0 msgr2=0x7ff28810fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:11.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.580+0000 7ff28f60e700 1 --2- 172.21.15.32:0/932033185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2880fe4a0 0x7ff28810fb30 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7ff27c009fd0 tx=0x7ff27c009e30 comp rx=0 tx=0).stop 2024-04-12T05:37:11.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.580+0000 7ff28f60e700 1 -- 172.21.15.32:0/932033185 shutdown_connections 2024-04-12T05:37:11.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.580+0000 7ff28f60e700 1 --2- 172.21.15.32:0/932033185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2880fe4a0 0x7ff28810fb30 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:11.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.580+0000 7ff28f60e700 1 --2- 172.21.15.32:0/932033185 >> [v2:172.21.15.32:6800/595990262,v1:172.21.15.32:6801/595990262] conn(0x7ff2700692b0 0x7ff27006b760 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:11.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.580+0000 7ff28f60e700 1 --2- 172.21.15.32:0/932033185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2880fedf0 0x7ff288110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:11.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.580+0000 7ff28f60e700 1 -- 172.21.15.32:0/932033185 >> 172.21.15.32:0/932033185 conn(0x7ff2880f6ba0 msgr2=0x7ff2880f8380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:11.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.581+0000 7ff28f60e700 1 -- 172.21.15.32:0/932033185 shutdown_connections 2024-04-12T05:37:11.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:11.581+0000 7ff28f60e700 1 -- 172.21.15.32:0/932033185 wait complete. 2024-04-12T05:37:11.580 INFO:tasks.workunit.client.1.smithi067.stdout:5/269: truncate d0/d3a/f2a 885737 0 2024-04-12T05:37:11.584 INFO:tasks.workunit.client.1.smithi067.stdout:7/120: dwrite d0/d2/d5/db/f18 [0,4194304] 0 2024-04-12T05:37:11.585 INFO:tasks.workunit.client.1.smithi067.stdout:5/270: read d0/d3a/f4c [209597,67278] 0 2024-04-12T05:37:11.588 INFO:tasks.workunit.client.1.smithi067.stdout:7/121: write d0/d2/d5/f11 [1047796,35043] 0 2024-04-12T05:37:11.588 INFO:tasks.workunit.client.1.smithi067.stdout:5/271: dread - d0/d3a/f54 zero size 2024-04-12T05:37:11.589 INFO:tasks.workunit.client.1.smithi067.stdout:6/90: dwrite f6 [4194304,4194304] 0 2024-04-12T05:37:11.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:11 smithi032 ceph-mon[39731]: from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:11.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:11 smithi032 ceph-mon[39731]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:11.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:11 smithi032 ceph-mon[39731]: fsmap cephfs:2 {0=cephfs.smithi032.pndtry=up:active,1=cephfs.smithi067.zesmgy=up:active} 2 up:standby-replay 2024-04-12T05:37:11.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:11 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:37:11.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:11 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:37:11.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:11 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:37:11.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:11 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/1939609972' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:37:11.591 INFO:tasks.workunit.client.1.smithi067.stdout:6/91: chown f9 61 1 2024-04-12T05:37:11.591 INFO:tasks.workunit.client.1.smithi067.stdout:5/272: chown d0/d6/c4d 112408 1 2024-04-12T05:37:11.591 INFO:tasks.workunit.client.1.smithi067.stdout:8/91: sync 2024-04-12T05:37:11.596 INFO:tasks.workunit.client.1.smithi067.stdout:6/92: rename dd/f10 to dd/d11/f19 0 2024-04-12T05:37:11.597 INFO:tasks.workunit.client.1.smithi067.stdout:5/273: creat d0/d6/d20/f58 x:0 0 0 2024-04-12T05:37:11.598 INFO:tasks.workunit.client.1.smithi067.stdout:7/122: dread d0/f1 [0,4194304] 0 2024-04-12T05:37:11.598 INFO:tasks.workunit.client.1.smithi067.stdout:6/93: mknod dd/d11/c1a 0 2024-04-12T05:37:11.600 INFO:tasks.workunit.client.1.smithi067.stdout:8/92: rename d6/d8/f9 to d6/d8/d18/df/d12/f1d 0 2024-04-12T05:37:11.601 INFO:tasks.workunit.client.1.smithi067.stdout:6/94: truncate fc 1307870 0 2024-04-12T05:37:11.604 INFO:tasks.workunit.client.1.smithi067.stdout:8/93: dread - d6/d8/f16 zero size 2024-04-12T05:37:11.605 INFO:tasks.workunit.client.1.smithi067.stdout:5/274: creat d0/da/d41/f59 x:0 0 0 2024-04-12T05:37:11.607 INFO:tasks.workunit.client.1.smithi067.stdout:5/275: chown d0/d3a/f54 22509092 1 2024-04-12T05:37:11.607 INFO:tasks.workunit.client.1.smithi067.stdout:7/123: dread d0/f9 [0,4194304] 0 2024-04-12T05:37:11.612 INFO:tasks.workunit.client.0.smithi032.stdout:5/698: sync 2024-04-12T05:37:11.619 INFO:tasks.workunit.client.1.smithi067.stdout:5/276: rename d0/da/d27/f18 to d0/da/f5a 0 2024-04-12T05:37:11.623 INFO:tasks.workunit.client.1.smithi067.stdout:6/95: dwrite dd/fe [0,4194304] 0 2024-04-12T05:37:11.623 INFO:tasks.workunit.client.1.smithi067.stdout:5/277: readlink d0/da/d27/l1a 0 2024-04-12T05:37:11.625 INFO:tasks.workunit.client.1.smithi067.stdout:7/124: dread d0/f1 [0,4194304] 0 2024-04-12T05:37:11.627 INFO:tasks.workunit.client.1.smithi067.stdout:7/125: readlink d0/l1b 0 2024-04-12T05:37:11.631 INFO:tasks.workunit.client.1.smithi067.stdout:8/94: dwrite d6/d8/d18/df/d12/f1d [0,4194304] 0 2024-04-12T05:37:11.632 INFO:tasks.workunit.client.1.smithi067.stdout:7/126: creat d0/d2/d5/db/d10/f26 x:0 0 0 2024-04-12T05:37:11.633 INFO:tasks.workunit.client.1.smithi067.stdout:7/127: chown d0/d2/d5/db/d10/f26 382169999 1 2024-04-12T05:37:11.638 INFO:tasks.workunit.client.1.smithi067.stdout:8/95: creat d6/d8/d18/df/d12/f1e x:0 0 0 2024-04-12T05:37:11.638 INFO:tasks.workunit.client.1.smithi067.stdout:7/128: truncate d0/f9 2113475 0 2024-04-12T05:37:11.642 INFO:tasks.workunit.client.1.smithi067.stdout:7/129: creat d0/d23/f27 x:0 0 0 2024-04-12T05:37:11.645 INFO:tasks.workunit.client.1.smithi067.stdout:7/130: creat d0/d2/d5/d1e/f28 x:0 0 0 2024-04-12T05:37:11.647 INFO:tasks.workunit.client.1.smithi067.stdout:8/96: dwrite d6/d8/f16 [0,4194304] 0 2024-04-12T05:37:11.648 INFO:tasks.workunit.client.1.smithi067.stdout:7/131: creat d0/d2/d5/f29 x:0 0 0 2024-04-12T05:37:11.650 INFO:tasks.workunit.client.1.smithi067.stdout:7/132: read d0/d2/d5/f11 [471681,85858] 0 2024-04-12T05:37:11.650 INFO:tasks.workunit.client.1.smithi067.stdout:7/133: chown d0/d2/f21 21 1 2024-04-12T05:37:11.657 INFO:tasks.workunit.client.1.smithi067.stdout:7/134: symlink d0/d2/d5/d1d/l2a 0 2024-04-12T05:37:11.657 INFO:tasks.workunit.client.1.smithi067.stdout:8/97: creat d6/d8/d18/df/d12/f1f x:0 0 0 2024-04-12T05:37:11.660 INFO:tasks.workunit.client.1.smithi067.stdout:6/96: truncate f6 5451557 0 2024-04-12T05:37:11.664 INFO:tasks.workunit.client.1.smithi067.stdout:7/135: mknod d0/d2/d5/c2b 0 2024-04-12T05:37:11.664 INFO:tasks.workunit.client.1.smithi067.stdout:5/278: write d0/d3a/d2c/f37 [1339322,22216] 0 2024-04-12T05:37:11.667 INFO:tasks.workunit.client.1.smithi067.stdout:7/136: readlink d0/d2/d5/d1d/l2a 0 2024-04-12T05:37:11.671 INFO:tasks.workunit.client.1.smithi067.stdout:8/98: dwrite d6/d8/d18/df/d12/f1f [0,4194304] 0 2024-04-12T05:37:11.675 INFO:tasks.workunit.client.1.smithi067.stdout:8/99: creat d6/d8/d18/df/f20 x:0 0 0 2024-04-12T05:37:11.677 INFO:tasks.workunit.client.1.smithi067.stdout:5/279: rename d0/da/d41/c48 to d0/d6/d40/c5b 0 2024-04-12T05:37:11.679 INFO:tasks.workunit.client.1.smithi067.stdout:5/280: mknod d0/d3a/c5c 0 2024-04-12T05:37:11.682 INFO:tasks.workunit.client.1.smithi067.stdout:6/97: dwrite dd/d11/f19 [0,4194304] 0 2024-04-12T05:37:11.687 INFO:tasks.workunit.client.1.smithi067.stdout:5/281: mknod d0/d6/c5d 0 2024-04-12T05:37:11.688 INFO:tasks.workunit.client.1.smithi067.stdout:8/100: getdents d6/d8/d18/df/d12 0 2024-04-12T05:37:11.692 INFO:tasks.workunit.client.1.smithi067.stdout:8/101: chown d6/d8/d18/df/d12/f1e 250110854 1 2024-04-12T05:37:11.699 INFO:tasks.workunit.client.1.smithi067.stdout:5/282: getdents d0/da/d27 0 2024-04-12T05:37:11.699 INFO:tasks.workunit.client.1.smithi067.stdout:0/225: sync 2024-04-12T05:37:11.700 INFO:tasks.workunit.client.1.smithi067.stdout:2/164: sync 2024-04-12T05:37:11.700 INFO:tasks.workunit.client.1.smithi067.stdout:1/148: sync 2024-04-12T05:37:11.702 INFO:tasks.workunit.client.1.smithi067.stdout:6/98: dwrite dd/f17 [0,4194304] 0 2024-04-12T05:37:11.703 INFO:tasks.workunit.client.1.smithi067.stdout:7/137: dwrite d0/f8 [0,4194304] 0 2024-04-12T05:37:11.713 INFO:tasks.workunit.client.1.smithi067.stdout:6/99: fsync fb 0 2024-04-12T05:37:11.713 INFO:tasks.workunit.client.1.smithi067.stdout:0/226: creat dd/d40/f4c x:0 0 0 2024-04-12T05:37:11.713 INFO:tasks.workunit.client.1.smithi067.stdout:5/283: fdatasync d0/da/d27/f14 0 2024-04-12T05:37:11.715 INFO:tasks.workunit.client.1.smithi067.stdout:5/284: rmdir d0/d3a/d2c 39 2024-04-12T05:37:11.716 INFO:tasks.workunit.client.1.smithi067.stdout:8/102: dwrite f5 [0,4194304] 0 2024-04-12T05:37:11.718 INFO:tasks.workunit.client.1.smithi067.stdout:0/227: truncate dd/d40/f44 655403 0 2024-04-12T05:37:11.720 INFO:tasks.workunit.client.1.smithi067.stdout:0/228: write dd/d12/f2e [537960,53233] 0 2024-04-12T05:37:11.729 INFO:tasks.workunit.client.1.smithi067.stdout:6/100: write f7 [1313447,51839] 0 2024-04-12T05:37:11.729 INFO:tasks.workunit.client.1.smithi067.stdout:8/103: dread d6/d8/d18/df/f13 [0,4194304] 0 2024-04-12T05:37:11.732 INFO:tasks.workunit.client.1.smithi067.stdout:5/285: dread d0/da/f5a [0,4194304] 0 2024-04-12T05:37:11.734 INFO:tasks.workunit.client.1.smithi067.stdout:7/138: write d0/f9 [3118039,55378] 0 2024-04-12T05:37:11.738 INFO:tasks.workunit.client.1.smithi067.stdout:0/229: write dd/d12/f21 [4531162,121831] 0 2024-04-12T05:37:11.742 INFO:tasks.workunit.client.1.smithi067.stdout:8/104: dwrite d6/d8/d18/df/f1b [0,4194304] 0 2024-04-12T05:37:11.752 INFO:tasks.workunit.client.1.smithi067.stdout:6/101: dread f6 [0,4194304] 0 2024-04-12T05:37:11.752 INFO:tasks.workunit.client.1.smithi067.stdout:5/286: getdents d0/d6/d20 0 2024-04-12T05:37:11.752 INFO:tasks.workunit.client.1.smithi067.stdout:1/149: dwrite d4/f10 [0,4194304] 0 2024-04-12T05:37:11.752 INFO:tasks.workunit.client.1.smithi067.stdout:0/230: fsync dd/d12/d14/d1a/d3d/f3c 0 2024-04-12T05:37:11.754 INFO:tasks.workunit.client.1.smithi067.stdout:6/102: readlink dd/d11/l18 0 2024-04-12T05:37:11.755 INFO:tasks.workunit.client.1.smithi067.stdout:2/165: dwrite d5/f12 [0,4194304] 0 2024-04-12T05:37:11.760 INFO:tasks.workunit.client.1.smithi067.stdout:1/150: write d4/d9/d14/f21 [260948,94787] 0 2024-04-12T05:37:11.760 INFO:tasks.workunit.client.1.smithi067.stdout:2/166: chown d5/da/d33 3572 1 2024-04-12T05:37:11.760 INFO:tasks.workunit.client.1.smithi067.stdout:5/287: dread d0/da/d27/f15 [0,4194304] 0 2024-04-12T05:37:11.760 INFO:tasks.workunit.client.1.smithi067.stdout:1/151: write d4/fe [855481,22500] 0 2024-04-12T05:37:11.762 INFO:tasks.workunit.client.1.smithi067.stdout:1/152: write d4/f10 [944988,23469] 0 2024-04-12T05:37:11.764 INFO:tasks.workunit.client.1.smithi067.stdout:1/153: stat d4/d9/fb 0 2024-04-12T05:37:11.765 INFO:tasks.workunit.client.1.smithi067.stdout:8/105: symlink d6/l21 0 2024-04-12T05:37:11.765 INFO:tasks.workunit.client.1.smithi067.stdout:6/103: symlink dd/l1b 0 2024-04-12T05:37:11.767 INFO:tasks.workunit.client.1.smithi067.stdout:0/231: unlink dd/d12/d14/d1a/d3d/f3c 0 2024-04-12T05:37:11.767 INFO:tasks.workunit.client.1.smithi067.stdout:5/288: dread - d0/d3a/f54 zero size 2024-04-12T05:37:11.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:11 smithi067 ceph-mon[49010]: from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:11.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:11 smithi067 ceph-mon[49010]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:11.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:11 smithi067 ceph-mon[49010]: fsmap cephfs:2 {0=cephfs.smithi032.pndtry=up:active,1=cephfs.smithi067.zesmgy=up:active} 2 up:standby-replay 2024-04-12T05:37:11.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:11 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:37:11.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:11 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:37:11.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:11 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:37:11.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:11 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/1939609972' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:37:11.771 INFO:tasks.workunit.client.1.smithi067.stdout:1/154: mkdir d4/d9/d12/d2c 0 2024-04-12T05:37:11.775 INFO:tasks.workunit.client.1.smithi067.stdout:7/139: getdents d0/d2/d5/dc 0 2024-04-12T05:37:11.776 INFO:tasks.workunit.client.1.smithi067.stdout:1/155: write d4/d9/d12/d19/f1e [675378,93662] 0 2024-04-12T05:37:11.782 INFO:tasks.workunit.client.1.smithi067.stdout:7/140: symlink d0/d2/d5/db/d10/l2c 0 2024-04-12T05:37:11.782 INFO:tasks.workunit.client.1.smithi067.stdout:8/106: truncate d6/d8/d18/df/f1b 630019 0 2024-04-12T05:37:11.783 INFO:tasks.workunit.client.1.smithi067.stdout:1/156: write d4/f15 [2329458,52846] 0 2024-04-12T05:37:11.785 INFO:tasks.workunit.client.1.smithi067.stdout:6/104: dwrite dd/d11/f14 [0,4194304] 0 2024-04-12T05:37:11.789 INFO:tasks.workunit.client.1.smithi067.stdout:7/141: fdatasync d0/d2/d5/fe 0 2024-04-12T05:37:11.789 INFO:tasks.workunit.client.1.smithi067.stdout:8/107: mknod d6/d8/d18/df/d12/c22 0 2024-04-12T05:37:11.791 INFO:tasks.workunit.client.1.smithi067.stdout:5/289: dwrite d0/d6/f2b [4194304,4194304] 0 2024-04-12T05:37:11.793 INFO:tasks.workunit.client.1.smithi067.stdout:1/157: symlink d4/d9/d12/d2c/l2d 0 2024-04-12T05:37:11.795 INFO:tasks.workunit.client.1.smithi067.stdout:1/158: fsync d4/fe 0 2024-04-12T05:37:11.795 INFO:tasks.workunit.client.1.smithi067.stdout:7/142: mknod d0/d2/d5/db/c2d 0 2024-04-12T05:37:11.796 INFO:tasks.workunit.client.1.smithi067.stdout:1/159: chown d4/d9/d12/d19/f1e 508119979 1 2024-04-12T05:37:11.798 INFO:tasks.workunit.client.1.smithi067.stdout:5/290: rmdir d0 39 2024-04-12T05:37:11.801 INFO:tasks.workunit.client.1.smithi067.stdout:7/143: fsync d0/d2/ff 0 2024-04-12T05:37:11.802 INFO:tasks.workunit.client.1.smithi067.stdout:1/160: creat d4/d9/d12/f2e x:0 0 0 2024-04-12T05:37:11.803 INFO:tasks.workunit.client.1.smithi067.stdout:1/161: readlink d4/d9/d12/l24 0 2024-04-12T05:37:11.805 INFO:tasks.workunit.client.1.smithi067.stdout:8/108: getdents d6/d8/d18/df/d12 0 2024-04-12T05:37:11.806 INFO:tasks.workunit.client.1.smithi067.stdout:8/109: write d6/d8/d18/df/f1c [465273,70756] 0 2024-04-12T05:37:11.806 INFO:tasks.workunit.client.1.smithi067.stdout:7/144: rename d0/d2/d5/l25 to d0/d2/d5/db/d10/l2e 0 2024-04-12T05:37:11.807 INFO:tasks.workunit.client.1.smithi067.stdout:1/162: rmdir d4/d9/d12/d28 39 2024-04-12T05:37:11.807 INFO:tasks.workunit.client.1.smithi067.stdout:1/163: chown d4/d9/d12 0 1 2024-04-12T05:37:11.808 INFO:tasks.workunit.client.1.smithi067.stdout:7/145: fdatasync d0/d2/d5/d1e/f22 0 2024-04-12T05:37:11.810 INFO:tasks.workunit.client.1.smithi067.stdout:7/146: creat d0/d2/d5/d1e/f2f x:0 0 0 2024-04-12T05:37:11.812 INFO:tasks.workunit.client.1.smithi067.stdout:1/164: getdents d4/d9/d14 0 2024-04-12T05:37:11.812 INFO:tasks.workunit.client.1.smithi067.stdout:7/147: readlink d0/d2/d5/dc/l17 0 2024-04-12T05:37:11.813 INFO:tasks.workunit.client.1.smithi067.stdout:8/110: dwrite d6/d8/d18/df/d12/f1d [0,4194304] 0 2024-04-12T05:37:11.816 INFO:tasks.workunit.client.1.smithi067.stdout:1/165: rename d4/d9/fa to d4/d9/f2f 0 2024-04-12T05:37:11.817 INFO:tasks.workunit.client.1.smithi067.stdout:8/111: creat d6/d8/d18/df/f23 x:0 0 0 2024-04-12T05:37:11.818 INFO:tasks.workunit.client.1.smithi067.stdout:1/166: symlink d4/d9/d14/l30 0 2024-04-12T05:37:11.819 INFO:tasks.workunit.client.1.smithi067.stdout:1/167: stat d4/f5 0 2024-04-12T05:37:11.820 INFO:tasks.workunit.client.1.smithi067.stdout:8/112: creat d6/d8/d18/df/f24 x:0 0 0 2024-04-12T05:37:11.820 INFO:tasks.workunit.client.1.smithi067.stdout:7/148: dwrite d0/d2/d5/d1e/f22 [0,4194304] 0 2024-04-12T05:37:11.821 INFO:tasks.workunit.client.1.smithi067.stdout:1/168: fsync d4/f6 0 2024-04-12T05:37:11.821 INFO:tasks.workunit.client.1.smithi067.stdout:7/149: stat d0/f8 0 2024-04-12T05:37:11.824 INFO:tasks.workunit.client.1.smithi067.stdout:1/169: write d4/d9/f2f [2746288,45818] 0 2024-04-12T05:37:11.827 INFO:tasks.workunit.client.1.smithi067.stdout:8/113: dwrite d6/d8/d18/df/d12/f1d [4194304,4194304] 0 2024-04-12T05:37:11.828 INFO:tasks.workunit.client.1.smithi067.stdout:8/114: chown d6/d8/c1a 55049 1 2024-04-12T05:37:11.830 INFO:tasks.workunit.client.1.smithi067.stdout:8/115: rmdir d6/d8 39 2024-04-12T05:37:11.833 INFO:tasks.workunit.client.1.smithi067.stdout:8/116: creat d6/d8/d18/f25 x:0 0 0 2024-04-12T05:37:11.834 INFO:tasks.workunit.client.1.smithi067.stdout:8/117: rename d6/d8/d18/df to d6/d8/d18/df/d26 22 2024-04-12T05:37:11.840 INFO:tasks.workunit.client.1.smithi067.stdout:8/118: dwrite d6/d8/d18/df/f20 [0,4194304] 0 2024-04-12T05:37:11.842 INFO:tasks.workunit.client.1.smithi067.stdout:8/119: write d6/d8/d18/df/d12/f1f [477045,82328] 0 2024-04-12T05:37:11.904 INFO:tasks.workunit.client.1.smithi067.stdout:9/107: sync 2024-04-12T05:37:11.905 INFO:tasks.workunit.client.1.smithi067.stdout:4/61: sync 2024-04-12T05:37:11.905 INFO:tasks.workunit.client.1.smithi067.stdout:3/100: sync 2024-04-12T05:37:11.909 INFO:tasks.workunit.client.1.smithi067.stdout:9/108: readlink d1/l5 0 2024-04-12T05:37:11.910 INFO:tasks.workunit.client.1.smithi067.stdout:3/101: unlink dc/fe 0 2024-04-12T05:37:11.916 INFO:tasks.workunit.client.1.smithi067.stdout:3/102: write f9 [3432125,34959] 0 2024-04-12T05:37:11.926 INFO:tasks.workunit.client.1.smithi067.stdout:9/109: write d1/f11 [1858958,29702] 0 2024-04-12T05:37:11.928 INFO:tasks.workunit.client.1.smithi067.stdout:3/103: write dc/f18 [5097539,77768] 0 2024-04-12T05:37:11.929 INFO:tasks.workunit.client.1.smithi067.stdout:6/105: dwrite fc [0,4194304] 0 2024-04-12T05:37:11.932 INFO:tasks.workunit.client.1.smithi067.stdout:9/110: rmdir d1/d12 39 2024-04-12T05:37:11.937 INFO:tasks.workunit.client.1.smithi067.stdout:9/111: chown d1/fd 137763437 1 2024-04-12T05:37:11.942 INFO:tasks.workunit.client.1.smithi067.stdout:4/62: dwrite d8/fe [0,4194304] 0 2024-04-12T05:37:11.944 INFO:tasks.workunit.client.0.smithi032.stdout:8/725: rename d0/d36/d84/dad/da4/fb1 to d0/db/dd/d11/d1b/d22/d29/ded/f102 0 2024-04-12T05:37:11.947 INFO:tasks.workunit.client.1.smithi067.stdout:9/112: write d1/f6 [1266556,29967] 0 2024-04-12T05:37:11.947 INFO:tasks.workunit.client.0.smithi032.stdout:6/842: truncate d4/d21/d27/dad/d55/f72 1828025 0 2024-04-12T05:37:11.948 INFO:tasks.workunit.client.1.smithi067.stdout:4/63: symlink d8/d14/l16 0 2024-04-12T05:37:11.950 INFO:tasks.workunit.client.0.smithi032.stdout:3/702: fsync d12/d2e/d3a/d63/f8d 0 2024-04-12T05:37:11.950 INFO:tasks.workunit.client.1.smithi067.stdout:7/150: rmdir d0/d2/d5/d1e 39 2024-04-12T05:37:11.951 INFO:tasks.workunit.client.0.smithi032.stdout:3/703: chown d12/d24/f47 0 1 2024-04-12T05:37:11.953 INFO:tasks.workunit.client.1.smithi067.stdout:3/104: dwrite dc/df/f1e [0,4194304] 0 2024-04-12T05:37:11.954 INFO:tasks.workunit.client.0.smithi032.stdout:1/578: symlink d5/de/d2a/lb7 0 2024-04-12T05:37:11.955 INFO:tasks.workunit.client.0.smithi032.stdout:9/558: rmdir d9/d15/d1a/d24 39 2024-04-12T05:37:11.958 INFO:tasks.workunit.client.0.smithi032.stdout:2/703: symlink dd/d19/d82/le7 0 2024-04-12T05:37:11.960 INFO:tasks.workunit.client.0.smithi032.stdout:5/699: stat d2/c11 0 2024-04-12T05:37:11.960 INFO:tasks.workunit.client.1.smithi067.stdout:5/291: dread d0/da/d3f/f49 [0,4194304] 0 2024-04-12T05:37:11.962 INFO:tasks.workunit.client.1.smithi067.stdout:5/292: stat d0/d6 0 2024-04-12T05:37:11.962 INFO:tasks.workunit.client.1.smithi067.stdout:8/120: getdents d6/d8/d18 0 2024-04-12T05:37:11.963 INFO:tasks.workunit.client.0.smithi032.stdout:5/700: write d2/df/d25/f2c [323572,122717] 0 2024-04-12T05:37:11.965 INFO:tasks.workunit.client.1.smithi067.stdout:8/121: dread - d6/d8/d18/df/d12/f1e zero size 2024-04-12T05:37:11.968 INFO:tasks.workunit.client.1.smithi067.stdout:9/113: dwrite d1/d12/f14 [0,4194304] 0 2024-04-12T05:37:11.970 INFO:tasks.workunit.client.0.smithi032.stdout:9/559: dread f7 [4194304,4194304] 0 2024-04-12T05:37:11.970 INFO:tasks.workunit.client.1.smithi067.stdout:9/114: fsync d1/f6 0 2024-04-12T05:37:11.972 INFO:tasks.workunit.client.0.smithi032.stdout:9/560: write d9/d15/d1a/d31/d65/fbc [271189,118183] 0 2024-04-12T05:37:11.974 INFO:tasks.workunit.client.1.smithi067.stdout:9/115: write d1/f11 [1209640,9613] 0 2024-04-12T05:37:11.984 INFO:tasks.workunit.client.0.smithi032.stdout:0/709: dread d2/dc/d41/f89 [0,4194304] 0 2024-04-12T05:37:12.137 INFO:tasks.workunit.client.0.smithi032.stdout:4/655: dwrite d4/d13/d19/fa8 [4194304,4194304] 0 2024-04-12T05:37:12.139 INFO:tasks.workunit.client.0.smithi032.stdout:4/656: write d4/d13/d44/fd6 [666799,127773] 0 2024-04-12T05:37:12.153 INFO:tasks.workunit.client.0.smithi032.stdout:8/726: fsync d0/db/dd/f17 0 2024-04-12T05:37:12.154 INFO:tasks.workunit.client.0.smithi032.stdout:8/727: chown d0/db/dd/d11/d1b/d5f/deb/f90 16201886 1 2024-04-12T05:37:12.155 INFO:tasks.workunit.client.0.smithi032.stdout:7/710: mkdir d0/d34/dbe/dbb/de8/df1 0 2024-04-12T05:37:12.156 INFO:tasks.workunit.client.0.smithi032.stdout:8/728: read d0/db/f6b [623630,108152] 0 2024-04-12T05:37:12.157 INFO:tasks.workunit.client.0.smithi032.stdout:5/701: creat d2/df/da9/dc1/fde x:0 0 0 2024-04-12T05:37:12.158 INFO:tasks.workunit.client.0.smithi032.stdout:9/561: mknod d9/d32/d6e/cbf 0 2024-04-12T05:37:12.158 INFO:tasks.workunit.client.0.smithi032.stdout:5/702: fdatasync d2/df/fb1 0 2024-04-12T05:37:12.164 INFO:tasks.workunit.client.0.smithi032.stdout:5/703: dwrite d2/df/d25/f29 [0,4194304] 0 2024-04-12T05:37:12.166 INFO:tasks.workunit.client.0.smithi032.stdout:6/843: creat d4/d21/d2e/da1/f125 x:0 0 0 2024-04-12T05:37:12.167 INFO:tasks.workunit.client.1.smithi067.stdout:2/167: sync 2024-04-12T05:37:12.177 INFO:tasks.workunit.client.0.smithi032.stdout:9/562: dread d9/d32/d86/f8c [0,4194304] 0 2024-04-12T05:37:12.178 INFO:tasks.workunit.client.0.smithi032.stdout:1/579: creat d5/fb8 x:0 0 0 2024-04-12T05:37:12.179 INFO:tasks.workunit.client.0.smithi032.stdout:6/844: creat d4/d21/d2e/d32/d94/f126 x:0 0 0 2024-04-12T05:37:12.180 INFO:tasks.workunit.client.0.smithi032.stdout:7/711: mknod d0/dd/d51/dea/cf2 0 2024-04-12T05:37:12.180 INFO:tasks.workunit.client.0.smithi032.stdout:6/845: fsync d4/de0/f115 0 2024-04-12T05:37:12.183 INFO:tasks.workunit.client.0.smithi032.stdout:4/657: link d4/d87/d96/d2f/d3b/cb4 d4/d87/d96/d4d/dbc/dd2/ce0 0 2024-04-12T05:37:12.185 INFO:tasks.workunit.client.0.smithi032.stdout:5/704: mkdir d2/dd/ddf 0 2024-04-12T05:37:12.187 INFO:tasks.workunit.client.0.smithi032.stdout:6/846: rmdir d4/d74 39 2024-04-12T05:37:12.187 INFO:tasks.workunit.client.0.smithi032.stdout:4/658: rename d4/d87/d96/d2f to d4/d87/d96/d2f/d94/de1 22 2024-04-12T05:37:12.187 INFO:tasks.workunit.client.0.smithi032.stdout:1/580: chown d5/d2c/d4f/c88 27 1 2024-04-12T05:37:12.188 INFO:tasks.workunit.client.0.smithi032.stdout:7/712: dwrite d0/de/df/f24 [0,4194304] 0 2024-04-12T05:37:12.189 INFO:tasks.workunit.client.0.smithi032.stdout:7/713: chown d0/fc4 13917 1 2024-04-12T05:37:12.195 INFO:tasks.workunit.client.0.smithi032.stdout:1/581: creat d5/d2c/d4f/fb9 x:0 0 0 2024-04-12T05:37:12.195 INFO:tasks.workunit.client.0.smithi032.stdout:4/659: stat d4/d87/d96/d2f/d9a/cb2 0 2024-04-12T05:37:12.197 INFO:tasks.workunit.client.0.smithi032.stdout:5/705: fsync d2/dd/f3a 0 2024-04-12T05:37:12.198 INFO:tasks.workunit.client.0.smithi032.stdout:6/847: mknod d4/d21/d2e/da1/c127 0 2024-04-12T05:37:12.202 INFO:tasks.workunit.client.0.smithi032.stdout:6/848: chown d4/de0/de7/d88/d8d/f108 32 1 2024-04-12T05:37:12.204 INFO:tasks.workunit.client.0.smithi032.stdout:0/710: dwrite d2/d8/f37 [0,4194304] 0 2024-04-12T05:37:12.206 INFO:tasks.workunit.client.0.smithi032.stdout:1/582: dwrite d5/de/d17/d1b/f1d [0,4194304] 0 2024-04-12T05:37:12.207 INFO:tasks.workunit.client.0.smithi032.stdout:4/660: symlink d4/d87/d96/le2 0 2024-04-12T05:37:12.209 INFO:tasks.workunit.client.0.smithi032.stdout:8/729: truncate d0/db/dd/d11/d1b/fc9 3582454 0 2024-04-12T05:37:12.209 INFO:tasks.workunit.client.0.smithi032.stdout:1/583: dread - d5/de/d17/d35/d61/d77/f7d zero size 2024-04-12T05:37:12.211 INFO:tasks.workunit.client.0.smithi032.stdout:0/711: symlink d2/d29/d77/dc7/le8 0 2024-04-12T05:37:12.216 INFO:tasks.workunit.client.0.smithi032.stdout:9/563: write d9/d15/d1a/d24/d27/f76 [604551,39003] 0 2024-04-12T05:37:12.216 INFO:tasks.workunit.client.0.smithi032.stdout:0/712: mknod d2/d29/d77/ce9 0 2024-04-12T05:37:12.219 INFO:tasks.workunit.client.0.smithi032.stdout:4/661: mkdir d4/d87/d96/d4d/db5/dcf/de3 0 2024-04-12T05:37:12.223 INFO:tasks.workunit.client.0.smithi032.stdout:6/849: truncate f3 3127289 0 2024-04-12T05:37:12.223 INFO:tasks.workunit.client.0.smithi032.stdout:8/730: dwrite d0/db/dd/f75 [0,4194304] 0 2024-04-12T05:37:12.227 INFO:tasks.workunit.client.0.smithi032.stdout:1/584: truncate d5/de/d17/d35/d78/fad 364573 0 2024-04-12T05:37:12.230 INFO:tasks.workunit.client.0.smithi032.stdout:9/564: rename d9/d15/la2 to d9/d15/d1a/dba/lc0 0 2024-04-12T05:37:12.231 INFO:tasks.workunit.client.0.smithi032.stdout:0/713: link d2/d29/d8a/fd7 d2/d8/df/d24/dc3/fea 0 2024-04-12T05:37:12.233 INFO:tasks.workunit.client.1.smithi067.stdout:0/232: sync 2024-04-12T05:37:12.233 INFO:tasks.workunit.client.1.smithi067.stdout:1/170: sync 2024-04-12T05:37:12.234 INFO:tasks.workunit.client.0.smithi032.stdout:5/706: dread d2/f68 [0,4194304] 0 2024-04-12T05:37:12.235 INFO:tasks.workunit.client.0.smithi032.stdout:1/585: creat d5/de/d17/d35/d73/fba x:0 0 0 2024-04-12T05:37:12.235 INFO:tasks.workunit.client.0.smithi032.stdout:4/662: dwrite d4/d13/d75/fac [0,4194304] 0 2024-04-12T05:37:12.239 INFO:tasks.workunit.client.0.smithi032.stdout:0/714: rename d2/d29/fdf to d2/d8/feb 0 2024-04-12T05:37:12.240 INFO:tasks.workunit.client.0.smithi032.stdout:1/586: write d5/de/d2a/f80 [98032,68200] 0 2024-04-12T05:37:12.243 INFO:tasks.workunit.client.0.smithi032.stdout:9/565: mkdir d9/d15/d1a/d24/d27/d88/dc1 0 2024-04-12T05:37:12.244 INFO:tasks.workunit.client.0.smithi032.stdout:0/715: truncate d2/d8/df/d24/dc3/fea 168224 0 2024-04-12T05:37:12.245 INFO:tasks.workunit.client.0.smithi032.stdout:1/587: symlink d5/de/d17/d35/d3b/lbb 0 2024-04-12T05:37:12.246 INFO:tasks.workunit.client.0.smithi032.stdout:4/663: dread d4/d87/d96/d4d/f63 [0,4194304] 0 2024-04-12T05:37:12.248 INFO:tasks.workunit.client.0.smithi032.stdout:8/731: getdents d0/db/dd/d11/d1b/d5f/daf/dc6 0 2024-04-12T05:37:12.250 INFO:tasks.workunit.client.0.smithi032.stdout:8/732: write d0/db/dd/f75 [1274286,67610] 0 2024-04-12T05:37:12.250 INFO:tasks.workunit.client.0.smithi032.stdout:1/588: truncate d5/de/d17/d35/d73/fb3 733806 0 2024-04-12T05:37:12.256 INFO:tasks.workunit.client.0.smithi032.stdout:9/566: truncate d9/d15/d2c/d41/f7e 3894995 0 2024-04-12T05:37:12.263 INFO:tasks.workunit.client.0.smithi032.stdout:8/733: dwrite d0/db/dd/fec [0,4194304] 0 2024-04-12T05:37:12.265 INFO:tasks.workunit.client.0.smithi032.stdout:9/567: write d9/d15/d1a/d69/f99 [440578,91308] 0 2024-04-12T05:37:12.270 INFO:tasks.workunit.client.0.smithi032.stdout:1/589: dread d5/de/f28 [0,4194304] 0 2024-04-12T05:37:12.270 INFO:tasks.workunit.client.0.smithi032.stdout:0/716: dwrite d2/d28/d87/d79/fce [0,4194304] 0 2024-04-12T05:37:12.271 INFO:tasks.workunit.client.0.smithi032.stdout:8/734: chown d0/db/dd/d11/d1b/f89 116767568 1 2024-04-12T05:37:12.273 INFO:tasks.workunit.client.0.smithi032.stdout:4/664: dwrite d4/d13/d44/f5a [0,4194304] 0 2024-04-12T05:37:12.273 INFO:tasks.workunit.client.0.smithi032.stdout:9/568: chown d9/d15/d1a/d24/d27/c2b 37585337 1 2024-04-12T05:37:12.280 INFO:tasks.workunit.client.0.smithi032.stdout:0/717: mknod d2/d8/d48/cec 0 2024-04-12T05:37:12.280 INFO:tasks.workunit.client.0.smithi032.stdout:5/707: dread d2/d3c/d5b/f87 [0,4194304] 0 2024-04-12T05:37:12.287 INFO:tasks.workunit.client.0.smithi032.stdout:1/590: dread d5/de/d17/f25 [0,4194304] 0 2024-04-12T05:37:12.288 INFO:tasks.workunit.client.0.smithi032.stdout:1/591: readlink d5/l6e 0 2024-04-12T05:37:12.290 INFO:tasks.workunit.client.0.smithi032.stdout:1/592: chown d5/de/d17/d35/d61/l90 5733041 1 2024-04-12T05:37:12.290 INFO:tasks.workunit.client.0.smithi032.stdout:9/569: creat d9/d15/d1a/d31/d65/d80/d87/fc2 x:0 0 0 2024-04-12T05:37:12.292 INFO:tasks.workunit.client.0.smithi032.stdout:8/735: symlink d0/db/d7d/dfb/l103 0 2024-04-12T05:37:12.293 INFO:tasks.workunit.client.0.smithi032.stdout:9/570: chown d9/d32/c54 96 1 2024-04-12T05:37:12.296 INFO:tasks.workunit.client.0.smithi032.stdout:5/708: rename d2/df/da9/dc1/lda to d2/d3c/d48/d4b/le0 0 2024-04-12T05:37:12.296 INFO:tasks.workunit.client.0.smithi032.stdout:4/665: dwrite d4/d87/d96/d2f/f70 [0,4194304] 0 2024-04-12T05:37:12.296 INFO:tasks.workunit.client.0.smithi032.stdout:0/718: write d2/dc/d3e/dac/f54 [523074,40554] 0 2024-04-12T05:37:12.299 INFO:tasks.workunit.client.0.smithi032.stdout:9/571: unlink d9/d15/l38 0 2024-04-12T05:37:12.301 INFO:tasks.workunit.client.0.smithi032.stdout:9/572: read - d9/d15/d1a/f9e zero size 2024-04-12T05:37:12.302 INFO:tasks.workunit.client.0.smithi032.stdout:4/666: mknod d4/d87/d96/d68/d9b/ce4 0 2024-04-12T05:37:12.304 INFO:tasks.workunit.client.0.smithi032.stdout:4/667: readlink d4/l7 0 2024-04-12T05:37:12.307 INFO:tasks.workunit.client.0.smithi032.stdout:5/709: dwrite d2/d9b/f5a [0,4194304] 0 2024-04-12T05:37:12.307 INFO:tasks.workunit.client.0.smithi032.stdout:0/719: rename d2/dc/d41/l44 to d2/d29/d8a/led 0 2024-04-12T05:37:12.311 INFO:tasks.workunit.client.0.smithi032.stdout:1/593: dwrite d5/de/d17/d35/d3b/d49/f70 [0,4194304] 0 2024-04-12T05:37:12.313 INFO:tasks.workunit.client.0.smithi032.stdout:5/710: mkdir d2/d3c/d48/d4b/dc4/de1 0 2024-04-12T05:37:12.314 INFO:tasks.workunit.client.0.smithi032.stdout:5/711: readlink d2/dd/l66 0 2024-04-12T05:37:12.319 INFO:tasks.workunit.client.0.smithi032.stdout:5/712: symlink d2/df/d31/le2 0 2024-04-12T05:37:12.320 INFO:tasks.workunit.client.0.smithi032.stdout:9/573: truncate d9/d15/d2c/f10 1018696 0 2024-04-12T05:37:12.320 INFO:tasks.workunit.client.0.smithi032.stdout:4/668: write d4/d13/d19/f2a [61726,67180] 0 2024-04-12T05:37:12.322 INFO:tasks.workunit.client.0.smithi032.stdout:1/594: dwrite d5/de/d2a/f65 [0,4194304] 0 2024-04-12T05:37:12.323 INFO:tasks.workunit.client.0.smithi032.stdout:4/669: fsync d4/d13/d44/f91 0 2024-04-12T05:37:12.326 INFO:tasks.workunit.client.0.smithi032.stdout:9/574: creat d9/d15/fc3 x:0 0 0 2024-04-12T05:37:12.326 INFO:tasks.workunit.client.0.smithi032.stdout:5/713: mkdir d2/df/d31/d80/de3 0 2024-04-12T05:37:12.327 INFO:tasks.workunit.client.0.smithi032.stdout:9/575: chown d9/d15/d2c/dab 303529 1 2024-04-12T05:37:12.328 INFO:tasks.workunit.client.0.smithi032.stdout:1/595: symlink d5/de/lbc 0 2024-04-12T05:37:12.328 INFO:tasks.workunit.client.0.smithi032.stdout:0/720: dwrite d2/d8/f6c [0,4194304] 0 2024-04-12T05:37:12.334 INFO:tasks.workunit.client.0.smithi032.stdout:5/714: rename d2/dd/d5d/f97 to d2/d3c/d5b/d75/dd6/fe4 0 2024-04-12T05:37:12.334 INFO:tasks.workunit.client.0.smithi032.stdout:0/721: readlink d2/d8/df/l9f 0 2024-04-12T05:37:12.335 INFO:tasks.workunit.client.0.smithi032.stdout:0/722: dread - d2/d8/d17/fc4 zero size 2024-04-12T05:37:12.337 INFO:tasks.workunit.client.0.smithi032.stdout:4/670: getdents d4/d13 0 2024-04-12T05:37:12.339 INFO:tasks.workunit.client.0.smithi032.stdout:9/576: rename d9/d15/d1a/d31/d65/d80/fbe to d9/d15/d2c/fc4 0 2024-04-12T05:37:12.340 INFO:tasks.workunit.client.0.smithi032.stdout:4/671: write d4/d87/d96/fbf [173751,118136] 0 2024-04-12T05:37:12.341 INFO:tasks.workunit.client.0.smithi032.stdout:4/672: readlink d4/d87/d96/d4d/d78/l4a 0 2024-04-12T05:37:12.341 INFO:tasks.workunit.client.0.smithi032.stdout:5/715: dwrite d2/d9b/fb7 [0,4194304] 0 2024-04-12T05:37:12.343 INFO:tasks.workunit.client.0.smithi032.stdout:5/716: chown d2/df/d25/d43 13 1 2024-04-12T05:37:12.346 INFO:tasks.workunit.client.0.smithi032.stdout:1/596: rename d5/de/d2a/f65 to d5/de/d17/d35/d3e/db4/fbd 0 2024-04-12T05:37:12.353 INFO:tasks.workunit.client.0.smithi032.stdout:4/673: symlink d4/d87/d96/d2f/d3b/da2/le5 0 2024-04-12T05:37:12.353 INFO:tasks.workunit.client.0.smithi032.stdout:5/717: symlink d2/df/d31/da2/dc0/le5 0 2024-04-12T05:37:12.355 INFO:tasks.workunit.client.0.smithi032.stdout:1/597: mkdir d5/de/dbe 0 2024-04-12T05:37:12.355 INFO:tasks.workunit.client.0.smithi032.stdout:0/723: link d2/d8/c11 d2/d29/d8a/cee 0 2024-04-12T05:37:12.356 INFO:tasks.workunit.client.0.smithi032.stdout:4/674: write d4/d87/d96/f52 [3348781,108158] 0 2024-04-12T05:37:12.357 INFO:tasks.workunit.client.0.smithi032.stdout:5/718: rename d2/df/d31/fc5 to d2/d3c/d48/d4b/dc4/fe6 0 2024-04-12T05:37:12.358 INFO:tasks.workunit.client.0.smithi032.stdout:1/598: chown d5/de/d17/d35/d3b/f98 372629 1 2024-04-12T05:37:12.360 INFO:tasks.workunit.client.0.smithi032.stdout:5/719: mkdir d2/df/d31/de7 0 2024-04-12T05:37:12.363 INFO:tasks.workunit.client.0.smithi032.stdout:5/720: chown d2/d3c/d48/d4b/dc4/ccb 99 1 2024-04-12T05:37:12.364 INFO:tasks.workunit.client.0.smithi032.stdout:0/724: dwrite d2/d29/d8a/d99/fba [0,4194304] 0 2024-04-12T05:37:12.364 INFO:tasks.workunit.client.0.smithi032.stdout:5/721: chown d2/d3c/d5b/c1b 0 1 2024-04-12T05:37:12.367 INFO:tasks.workunit.client.0.smithi032.stdout:0/725: fdatasync d2/dc/d41/f89 0 2024-04-12T05:37:12.375 INFO:tasks.workunit.client.0.smithi032.stdout:0/726: rename d2/d8/df/d24/f9c to d2/dc/d41/fef 0 2024-04-12T05:37:12.382 INFO:tasks.workunit.client.0.smithi032.stdout:1/599: write d5/de/f1f [1301215,70757] 0 2024-04-12T05:37:12.388 INFO:tasks.workunit.client.0.smithi032.stdout:1/600: creat d5/de/d17/d35/fbf x:0 0 0 2024-04-12T05:37:12.391 INFO:tasks.workunit.client.0.smithi032.stdout:1/601: dread - d5/de/d17/d35/d73/f76 zero size 2024-04-12T05:37:12.393 INFO:tasks.workunit.client.0.smithi032.stdout:0/727: dwrite d2/d8/d17/f2b [0,4194304] 0 2024-04-12T05:37:12.394 INFO:tasks.workunit.client.0.smithi032.stdout:5/722: dread d2/df/d25/d1e/f2f [0,4194304] 0 2024-04-12T05:37:12.396 INFO:tasks.workunit.client.0.smithi032.stdout:0/728: symlink d2/d8/lf0 0 2024-04-12T05:37:12.396 INFO:tasks.workunit.client.0.smithi032.stdout:5/723: write d2/d9b/fd8 [858525,105010] 0 2024-04-12T05:37:12.399 INFO:tasks.workunit.client.0.smithi032.stdout:0/729: creat d2/dc/d3e/ff1 x:0 0 0 2024-04-12T05:37:12.400 INFO:tasks.workunit.client.0.smithi032.stdout:5/724: creat d2/df/d31/d80/de3/fe8 x:0 0 0 2024-04-12T05:37:12.403 INFO:tasks.workunit.client.0.smithi032.stdout:5/725: truncate d2/df/da9/dc1/d6c/f77 2018344 0 2024-04-12T05:37:12.405 INFO:tasks.workunit.client.0.smithi032.stdout:5/726: mknod d2/ce9 0 2024-04-12T05:37:12.406 INFO:tasks.workunit.client.0.smithi032.stdout:5/727: write d2/df/da9/dc1/d78/f94 [3732467,123615] 0 2024-04-12T05:37:12.410 INFO:tasks.workunit.client.0.smithi032.stdout:5/728: rmdir d2/df/d31/da2/dc0/dcd 0 2024-04-12T05:37:12.423 INFO:tasks.workunit.client.0.smithi032.stdout:0/730: dwrite d2/f7 [0,4194304] 0 2024-04-12T05:37:12.432 INFO:tasks.workunit.client.0.smithi032.stdout:5/729: dread d2/df/d25/d1e/f40 [0,4194304] 0 2024-04-12T05:37:12.432 INFO:tasks.workunit.client.0.smithi032.stdout:0/731: dwrite d2/d28/f8e [0,4194304] 0 2024-04-12T05:37:12.434 INFO:tasks.workunit.client.0.smithi032.stdout:5/730: mknod d2/dd/d5d/cea 0 2024-04-12T05:37:12.436 INFO:tasks.workunit.client.0.smithi032.stdout:5/731: stat d2/df/c8b 0 2024-04-12T05:37:12.439 INFO:tasks.workunit.client.0.smithi032.stdout:0/732: fsync d2/d29/d8a/fd7 0 2024-04-12T05:37:12.443 INFO:tasks.workunit.client.0.smithi032.stdout:5/732: dwrite d2/df/d25/d50/f7e [0,4194304] 0 2024-04-12T05:37:12.446 INFO:tasks.workunit.client.0.smithi032.stdout:0/733: mkdir d2/d29/d8a/df2 0 2024-04-12T05:37:12.447 INFO:tasks.workunit.client.0.smithi032.stdout:5/733: write d2/df/da9/dc1/d6c/fca [377340,59701] 0 2024-04-12T05:37:12.449 INFO:tasks.workunit.client.0.smithi032.stdout:0/734: unlink d2/d8/d48/c80 0 2024-04-12T05:37:12.450 INFO:tasks.workunit.client.0.smithi032.stdout:0/735: chown d2/d28/dd6 24621 1 2024-04-12T05:37:12.454 INFO:tasks.workunit.client.0.smithi032.stdout:0/736: symlink d2/d29/d77/dc7/lf3 0 2024-04-12T05:37:12.455 INFO:tasks.workunit.client.0.smithi032.stdout:5/734: rename d2/df/d25/d50/daa/fae to d2/df/d25/d43/d49/feb 0 2024-04-12T05:37:12.457 INFO:tasks.workunit.client.0.smithi032.stdout:0/737: write d2/d8/df/d24/dc3/fea [618408,79434] 0 2024-04-12T05:37:12.459 INFO:tasks.workunit.client.0.smithi032.stdout:0/738: rmdir d2 39 2024-04-12T05:37:12.461 INFO:tasks.workunit.client.0.smithi032.stdout:0/739: fsync d2/dc/d3e/ff1 0 2024-04-12T05:37:12.468 INFO:tasks.workunit.client.0.smithi032.stdout:5/735: dread d2/d3c/d48/d4b/f7c [0,4194304] 0 2024-04-12T05:37:12.470 INFO:tasks.workunit.client.0.smithi032.stdout:5/736: symlink d2/lec 0 2024-04-12T05:37:12.471 INFO:tasks.workunit.client.0.smithi032.stdout:5/737: readlink d2/dd/l61 0 2024-04-12T05:37:12.472 INFO:tasks.workunit.client.0.smithi032.stdout:5/738: fsync d2/df/d25/f29 0 2024-04-12T05:37:12.473 INFO:tasks.workunit.client.0.smithi032.stdout:5/739: chown d2/df/d31/da2/dc0/le5 0 1 2024-04-12T05:37:12.473 INFO:tasks.workunit.client.0.smithi032.stdout:5/740: fdatasync d2/f60 0 2024-04-12T05:37:12.475 INFO:tasks.workunit.client.0.smithi032.stdout:5/741: truncate d2/df/d31/fcf 234415 0 2024-04-12T05:37:12.481 INFO:tasks.workunit.client.0.smithi032.stdout:5/742: truncate d2/df/f1c 7371666 0 2024-04-12T05:37:12.482 INFO:tasks.workunit.client.0.smithi032.stdout:5/743: chown d2/df/d31/da2/dba/fbf 23222666 1 2024-04-12T05:37:12.484 INFO:tasks.workunit.client.0.smithi032.stdout:5/744: symlink d2/df/d31/da2/dc0/led 0 2024-04-12T05:37:12.486 INFO:tasks.workunit.client.0.smithi032.stdout:5/745: fdatasync d2/d3c/d48/d4b/dc4/fe6 0 2024-04-12T05:37:12.497 INFO:tasks.workunit.client.0.smithi032.stdout:5/746: dwrite d2/dd/f3a [0,4194304] 0 2024-04-12T05:37:12.505 INFO:tasks.workunit.client.0.smithi032.stdout:5/747: dwrite d2/d3c/d5b/f21 [0,4194304] 0 2024-04-12T05:37:12.507 INFO:tasks.workunit.client.0.smithi032.stdout:5/748: write d2/df/da9/dc1/d78/f9e [2086201,41802] 0 2024-04-12T05:37:12.512 INFO:tasks.workunit.client.0.smithi032.stdout:5/749: chown d2/df/d25/ce 247740153 1 2024-04-12T05:37:12.514 INFO:tasks.workunit.client.0.smithi032.stdout:5/750: creat d2/df/d31/d92/fee x:0 0 0 2024-04-12T05:37:12.517 INFO:tasks.workunit.client.0.smithi032.stdout:5/751: creat d2/df/d25/d50/daa/fef x:0 0 0 2024-04-12T05:37:12.518 INFO:tasks.workunit.client.0.smithi032.stdout:5/752: chown d2/df/d25/f29 283418 1 2024-04-12T05:37:12.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:12 smithi067 ceph-mon[49010]: from='client.14608 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:12.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:12 smithi067 ceph-mon[49010]: pgmap v267: 65 pgs: 65 active+clean; 1.5 GiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 57 MiB/s rd, 200 MiB/s wr, 384 op/s 2024-04-12T05:37:12.523 INFO:tasks.workunit.client.0.smithi032.stdout:5/753: write d2/df/da9/dc1/fdb [316657,68921] 0 2024-04-12T05:37:12.523 INFO:tasks.workunit.client.0.smithi032.stdout:5/754: stat d2/df/d25/d1e/c79 0 2024-04-12T05:37:12.524 INFO:tasks.workunit.client.0.smithi032.stdout:5/755: rename d2/df/d31/d80/de3 to d2/d9b/df0 0 2024-04-12T05:37:12.534 INFO:tasks.workunit.client.0.smithi032.stdout:5/756: dwrite d2/d3c/d5b/d75/fa1 [0,4194304] 0 2024-04-12T05:37:12.536 INFO:tasks.workunit.client.0.smithi032.stdout:5/757: chown d2/df/d25/d50/daa/fdd 1298 1 2024-04-12T05:37:12.536 INFO:tasks.workunit.client.0.smithi032.stdout:5/758: dread - d2/df/d25/fd1 zero size 2024-04-12T05:37:12.540 INFO:tasks.workunit.client.0.smithi032.stdout:5/759: symlink d2/d3c/d5b/dd4/lf1 0 2024-04-12T05:37:12.541 INFO:tasks.workunit.client.0.smithi032.stdout:5/760: stat d2/df/d25/d43/d49 0 2024-04-12T05:37:12.546 INFO:tasks.workunit.client.0.smithi032.stdout:5/761: getdents d2/df/d25/d50 0 2024-04-12T05:37:12.547 INFO:tasks.workunit.client.0.smithi032.stdout:7/714: sync 2024-04-12T05:37:12.547 INFO:tasks.workunit.client.0.smithi032.stdout:3/704: sync 2024-04-12T05:37:12.547 INFO:tasks.workunit.client.0.smithi032.stdout:2/704: sync 2024-04-12T05:37:12.549 INFO:tasks.workunit.client.0.smithi032.stdout:2/705: read dd/d19/d9d/f58 [998851,97734] 0 2024-04-12T05:37:12.549 INFO:tasks.workunit.client.0.smithi032.stdout:5/762: mkdir d2/d3c/d5b/dd4/df2 0 2024-04-12T05:37:12.556 INFO:tasks.workunit.client.0.smithi032.stdout:7/715: dwrite d0/d13/d21/d7c/fa8 [4194304,4194304] 0 2024-04-12T05:37:12.556 INFO:tasks.workunit.client.0.smithi032.stdout:2/706: symlink dd/d19/d9d/d42/dc7/le8 0 2024-04-12T05:37:12.558 INFO:tasks.workunit.client.0.smithi032.stdout:5/763: dread d2/df/d25/d50/daa/fc9 [0,4194304] 0 2024-04-12T05:37:12.562 INFO:tasks.workunit.client.0.smithi032.stdout:2/707: dread - dd/d19/d9d/d1c/d64/fc2 zero size 2024-04-12T05:37:12.562 INFO:tasks.workunit.client.0.smithi032.stdout:5/764: write d2/df/d25/d1e/f40 [2640137,61082] 0 2024-04-12T05:37:12.570 INFO:tasks.workunit.client.0.smithi032.stdout:5/765: mkdir d2/df/da9/dc1/d6c/df3 0 2024-04-12T05:37:12.571 INFO:tasks.workunit.client.0.smithi032.stdout:5/766: read d2/dd/f3a [712519,97199] 0 2024-04-12T05:37:12.576 INFO:tasks.workunit.client.0.smithi032.stdout:2/708: rename dd/d19/d9d/d1c/d64/f96 to dd/d19/d9d/d42/fe9 0 2024-04-12T05:37:12.579 INFO:tasks.workunit.client.0.smithi032.stdout:2/709: truncate dd/f2e 2536117 0 2024-04-12T05:37:12.583 INFO:tasks.workunit.client.0.smithi032.stdout:5/767: dwrite d2/df/d31/f46 [4194304,4194304] 0 2024-04-12T05:37:12.584 INFO:tasks.workunit.client.0.smithi032.stdout:2/710: symlink dd/d19/d9d/d42/dc7/lea 0 2024-04-12T05:37:12.584 INFO:tasks.workunit.client.0.smithi032.stdout:5/768: read - d2/df/da9/dc1/fd3 zero size 2024-04-12T05:37:12.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:12 smithi032 ceph-mon[39731]: from='client.14608 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:12.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:12 smithi032 ceph-mon[39731]: pgmap v267: 65 pgs: 65 active+clean; 1.5 GiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 57 MiB/s rd, 200 MiB/s wr, 384 op/s 2024-04-12T05:37:12.594 INFO:tasks.workunit.client.0.smithi032.stdout:2/711: link dd/d19/d20/fa0 dd/d19/d9d/d42/dc7/feb 0 2024-04-12T05:37:12.597 INFO:tasks.workunit.client.0.smithi032.stdout:5/769: dwrite d2/df/f32 [0,4194304] 0 2024-04-12T05:37:12.606 INFO:tasks.workunit.client.0.smithi032.stdout:2/712: read dd/d19/d9d/d42/d4a/fb3 [1173439,111708] 0 2024-04-12T05:37:12.608 INFO:tasks.workunit.client.0.smithi032.stdout:5/770: dwrite d2/df/d25/d43/f81 [0,4194304] 0 2024-04-12T05:37:12.614 INFO:tasks.workunit.client.0.smithi032.stdout:2/713: mknod dd/d8c/cec 0 2024-04-12T05:37:12.620 INFO:tasks.workunit.client.0.smithi032.stdout:2/714: rename dd/d19/d9d/d1c/d64/d32/d3a/fa8 to dd/d19/d20/dc6/fed 0 2024-04-12T05:37:12.623 INFO:tasks.workunit.client.0.smithi032.stdout:5/771: getdents d2/df/d31 0 2024-04-12T05:37:12.623 INFO:tasks.workunit.client.0.smithi032.stdout:5/772: chown d2/d3c/la6 479 1 2024-04-12T05:37:12.634 INFO:tasks.workunit.client.0.smithi032.stdout:2/715: dread dd/d19/d9d/d1c/d64/d32/d3a/f56 [0,4194304] 0 2024-04-12T05:37:12.675 INFO:tasks.workunit.client.0.smithi032.stdout:5/773: truncate d2/dd/f47 1184877 0 2024-04-12T05:37:12.676 INFO:tasks.workunit.client.0.smithi032.stdout:2/716: chown dd/d19/d9d/d1c/d64/d32/lb9 890476526 1 2024-04-12T05:37:12.679 INFO:tasks.workunit.client.0.smithi032.stdout:5/774: rename d2/df/f54 to d2/df/da9/dc1/d6c/ff4 0 2024-04-12T05:37:12.680 INFO:tasks.workunit.client.0.smithi032.stdout:2/717: creat dd/d19/d20/fee x:0 0 0 2024-04-12T05:37:12.687 INFO:tasks.workunit.client.0.smithi032.stdout:5/775: dwrite d2/df/d25/d43/d49/feb [0,4194304] 0 2024-04-12T05:37:12.694 INFO:tasks.workunit.client.0.smithi032.stdout:5/776: mknod d2/df/cf5 0 2024-04-12T05:37:12.695 INFO:tasks.workunit.client.0.smithi032.stdout:2/718: dread dd/f2e [0,4194304] 0 2024-04-12T05:37:12.696 INFO:tasks.workunit.client.0.smithi032.stdout:5/777: write d2/d3c/d5b/d75/f84 [4739811,25430] 0 2024-04-12T05:37:12.703 INFO:tasks.workunit.client.0.smithi032.stdout:5/778: readlink d2/dd/l2e 0 2024-04-12T05:37:12.704 INFO:tasks.workunit.client.0.smithi032.stdout:2/719: chown dd/d19/d9d/d1c/d64/d32/ld3 30 1 2024-04-12T05:37:12.730 INFO:tasks.workunit.client.1.smithi067.stdout:6/106: truncate fc 3811012 0 2024-04-12T05:37:12.731 INFO:tasks.workunit.client.1.smithi067.stdout:6/107: write fb [1474092,117921] 0 2024-04-12T05:37:12.732 INFO:tasks.workunit.client.1.smithi067.stdout:4/64: rmdir d8 39 2024-04-12T05:37:12.732 INFO:tasks.workunit.client.1.smithi067.stdout:7/151: link d0/d2/ff d0/d2/d5/db/f30 0 2024-04-12T05:37:12.739 INFO:tasks.workunit.client.1.smithi067.stdout:7/152: dwrite d0/d23/f27 [0,4194304] 0 2024-04-12T05:37:12.741 INFO:tasks.workunit.client.1.smithi067.stdout:7/153: stat d0/d2/d5/db/f1f 0 2024-04-12T05:37:12.820 INFO:tasks.workunit.client.0.smithi032.stdout:5/779: mkdir d2/df/da9/dc1/d78/d86/df6 0 2024-04-12T05:37:12.822 INFO:tasks.workunit.client.0.smithi032.stdout:5/780: truncate d2/df/f5c 1802885 0 2024-04-12T05:37:12.823 INFO:tasks.workunit.client.0.smithi032.stdout:2/720: rename dd/d19/d9d/d1c/l97 to dd/d19/d9d/d1c/d64/d32/d3a/lef 0 2024-04-12T05:37:12.824 INFO:tasks.workunit.client.0.smithi032.stdout:5/781: mknod d2/df/da9/dc1/cf7 0 2024-04-12T05:37:12.831 INFO:tasks.workunit.client.0.smithi032.stdout:2/721: dwrite dd/d19/d9d/d42/d4a/fc4 [0,4194304] 0 2024-04-12T05:37:12.835 INFO:tasks.workunit.client.0.smithi032.stdout:5/782: dwrite d2/df/d25/d43/f81 [4194304,4194304] 0 2024-04-12T05:37:12.835 INFO:tasks.workunit.client.0.smithi032.stdout:2/722: creat dd/d19/d9d/d1c/ff0 x:0 0 0 2024-04-12T05:37:12.840 INFO:tasks.workunit.client.1.smithi067.stdout:3/105: truncate dc/f18 2550822 0 2024-04-12T05:37:12.844 INFO:tasks.workunit.client.0.smithi032.stdout:2/723: symlink dd/d19/d9d/d42/da7/lf1 0 2024-04-12T05:37:12.844 INFO:tasks.workunit.client.1.smithi067.stdout:9/116: rename d1/l4 to d1/d12/d15/l1a 0 2024-04-12T05:37:12.845 INFO:tasks.workunit.client.1.smithi067.stdout:2/168: chown d5/da/de/d14/f36 166 1 2024-04-12T05:37:12.845 INFO:tasks.workunit.client.1.smithi067.stdout:9/117: dread - d1/d12/d16/f18 zero size 2024-04-12T05:37:12.846 INFO:tasks.workunit.client.1.smithi067.stdout:1/171: creat d4/d9/f31 x:0 0 0 2024-04-12T05:37:12.848 INFO:tasks.workunit.client.1.smithi067.stdout:1/172: write d4/d9/d14/f2a [82950,79620] 0 2024-04-12T05:37:12.848 INFO:tasks.workunit.client.1.smithi067.stdout:6/108: fsync f0 0 2024-04-12T05:37:12.849 INFO:tasks.workunit.client.1.smithi067.stdout:6/109: chown l1 1070950 1 2024-04-12T05:37:12.851 INFO:tasks.workunit.client.1.smithi067.stdout:1/173: fdatasync d4/f15 0 2024-04-12T05:37:12.853 INFO:tasks.workunit.client.0.smithi032.stdout:2/724: dwrite dd/d19/d9d/d42/da7/fb8 [0,4194304] 0 2024-04-12T05:37:12.853 INFO:tasks.workunit.client.0.smithi032.stdout:5/783: rename d2/d3c/d48/d4b/f6b to d2/df/d31/da2/ff8 0 2024-04-12T05:37:12.853 INFO:tasks.workunit.client.1.smithi067.stdout:7/154: fsync d0/d2/d5/d1e/f28 0 2024-04-12T05:37:12.853 INFO:tasks.workunit.client.1.smithi067.stdout:3/106: creat dc/df/f21 x:0 0 0 2024-04-12T05:37:12.853 INFO:tasks.workunit.client.1.smithi067.stdout:6/110: write f6 [2063344,37683] 0 2024-04-12T05:37:12.854 INFO:tasks.workunit.client.1.smithi067.stdout:3/107: fsync f9 0 2024-04-12T05:37:12.854 INFO:tasks.workunit.client.1.smithi067.stdout:2/169: truncate f4 1944396 0 2024-04-12T05:37:12.856 INFO:tasks.workunit.client.1.smithi067.stdout:6/111: fdatasync f7 0 2024-04-12T05:37:12.857 INFO:tasks.workunit.client.1.smithi067.stdout:9/118: creat d1/d12/d16/f1b x:0 0 0 2024-04-12T05:37:12.857 INFO:tasks.workunit.client.0.smithi032.stdout:5/784: symlink d2/dd/d6f/lf9 0 2024-04-12T05:37:12.858 INFO:tasks.workunit.client.1.smithi067.stdout:0/233: creat dd/d12/d14/d49/f4d x:0 0 0 2024-04-12T05:37:12.858 INFO:tasks.workunit.client.0.smithi032.stdout:5/785: chown d2/df/d31/d92 1286286 1 2024-04-12T05:37:12.860 INFO:tasks.workunit.client.1.smithi067.stdout:8/122: creat d6/f27 x:0 0 0 2024-04-12T05:37:12.863 INFO:tasks.workunit.client.1.smithi067.stdout:5/293: getdents d0/da/d3f 0 2024-04-12T05:37:12.864 INFO:tasks.workunit.client.1.smithi067.stdout:3/108: fdatasync f1 0 2024-04-12T05:37:12.865 INFO:tasks.workunit.client.0.smithi032.stdout:6/850: sync 2024-04-12T05:37:12.866 INFO:tasks.workunit.client.1.smithi067.stdout:5/294: read d0/f1c [989480,101263] 0 2024-04-12T05:37:12.866 INFO:tasks.workunit.client.1.smithi067.stdout:4/65: dread d8/fe [0,4194304] 0 2024-04-12T05:37:12.867 INFO:tasks.workunit.client.1.smithi067.stdout:3/109: chown dc/df/l1b 24 1 2024-04-12T05:37:12.868 INFO:tasks.workunit.client.0.smithi032.stdout:2/725: dwrite dd/d19/d9d/f58 [0,4194304] 0 2024-04-12T05:37:12.870 INFO:tasks.workunit.client.0.smithi032.stdout:2/726: fsync dd/d19/d9d/d42/da7/fb8 0 2024-04-12T05:37:12.871 INFO:tasks.workunit.client.1.smithi067.stdout:1/174: dwrite d4/d9/fd [0,4194304] 0 2024-04-12T05:37:12.883 INFO:tasks.workunit.client.1.smithi067.stdout:1/175: dread d4/d9/d12/f1b [0,4194304] 0 2024-04-12T05:37:12.885 INFO:tasks.workunit.client.1.smithi067.stdout:1/176: dwrite d4/d9/f31 [0,4194304] 0 2024-04-12T05:37:13.038 INFO:tasks.workunit.client.0.smithi032.stdout:8/736: sync 2024-04-12T05:37:13.039 INFO:tasks.workunit.client.0.smithi032.stdout:9/577: sync 2024-04-12T05:37:13.081 INFO:tasks.workunit.client.0.smithi032.stdout:7/716: sync 2024-04-12T05:37:13.081 INFO:tasks.workunit.client.0.smithi032.stdout:0/740: sync 2024-04-12T05:37:13.081 INFO:tasks.workunit.client.0.smithi032.stdout:4/675: sync 2024-04-12T05:37:13.081 INFO:tasks.workunit.client.0.smithi032.stdout:1/602: sync 2024-04-12T05:37:13.085 INFO:tasks.workunit.client.0.smithi032.stdout:1/603: chown d5/de/d17/d35/d3b/d49/l51 0 1 2024-04-12T05:37:13.095 INFO:tasks.workunit.client.0.smithi032.stdout:4/676: dwrite d4/d87/d96/d2f/d6e/fcc [0,4194304] 0 2024-04-12T05:37:13.098 INFO:tasks.workunit.client.0.smithi032.stdout:4/677: write d4/d13/d44/fd6 [1644859,86618] 0 2024-04-12T05:37:13.100 INFO:tasks.workunit.client.0.smithi032.stdout:7/717: dread d0/de/df/d20/f6b [0,4194304] 0 2024-04-12T05:37:13.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:13 smithi067 ceph-mon[49010]: from='client.14618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:13.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:13 smithi032 ceph-mon[39731]: from='client.14618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:13.855 INFO:tasks.workunit.client.0.smithi032.stdout:5/786: truncate d2/df/d25/d50/fb0 1884392 0 2024-04-12T05:37:13.856 INFO:tasks.workunit.client.0.smithi032.stdout:5/787: dread - d2/dd/d6f/fbd zero size 2024-04-12T05:37:13.863 INFO:tasks.workunit.client.0.smithi032.stdout:5/788: dwrite d2/d3c/d48/d4b/fbb [0,4194304] 0 2024-04-12T05:37:13.866 INFO:tasks.workunit.client.0.smithi032.stdout:5/789: read - d2/dd/d5d/fce zero size 2024-04-12T05:37:13.867 INFO:tasks.workunit.client.0.smithi032.stdout:5/790: fdatasync d2/df/d25/d50/fd9 0 2024-04-12T05:37:13.896 INFO:tasks.workunit.client.0.smithi032.stdout:6/851: symlink d4/d14/d3c/l128 0 2024-04-12T05:37:13.908 INFO:tasks.workunit.client.0.smithi032.stdout:8/737: rename d0/db/dd/d11/d1b/d5f/daf/c5e to d0/db/dd/d11/d1b/d96/c104 0 2024-04-12T05:37:13.911 INFO:tasks.workunit.client.0.smithi032.stdout:0/741: unlink d2/d8/df/d24/d60/c5b 0 2024-04-12T05:37:13.916 INFO:tasks.workunit.client.0.smithi032.stdout:8/738: dread d0/db/dd/d11/d1b/f78 [0,4194304] 0 2024-04-12T05:37:13.916 INFO:tasks.workunit.client.0.smithi032.stdout:4/678: unlink d4/d87/d96/d32/c56 0 2024-04-12T05:37:13.918 INFO:tasks.workunit.client.0.smithi032.stdout:4/679: stat d4/d87/d96/d4d/d78/f64 0 2024-04-12T05:37:13.919 INFO:tasks.workunit.client.0.smithi032.stdout:4/680: stat d4/d13/ff 0 2024-04-12T05:37:13.919 INFO:tasks.workunit.client.0.smithi032.stdout:4/681: readlink d4/l8f 0 2024-04-12T05:37:13.921 INFO:tasks.workunit.client.0.smithi032.stdout:7/718: write d0/fe3 [3632757,56501] 0 2024-04-12T05:37:13.970 INFO:tasks.workunit.client.0.smithi032.stdout:5/791: mknod d2/df/da9/dc1/d78/d86/cfa 0 2024-04-12T05:37:13.971 INFO:tasks.workunit.client.0.smithi032.stdout:6/852: mknod d4/de0/c129 0 2024-04-12T05:37:13.972 INFO:tasks.workunit.client.0.smithi032.stdout:6/853: dread d4/ff [0,4194304] 0 2024-04-12T05:37:13.988 INFO:tasks.workunit.client.0.smithi032.stdout:4/682: rmdir d4/d87/d96/d2f/d9a 39 2024-04-12T05:37:13.990 INFO:tasks.workunit.client.0.smithi032.stdout:5/792: symlink d2/df/da9/lfb 0 2024-04-12T05:37:14.001 INFO:tasks.workunit.client.0.smithi032.stdout:8/739: rename d0/db/dd/d11/d1b/d5f/l8a to d0/db/dd/ddd/l105 0 2024-04-12T05:37:14.003 INFO:tasks.workunit.client.0.smithi032.stdout:7/719: mknod d0/d34/dbe/dbb/de8/df1/cf3 0 2024-04-12T05:37:14.005 INFO:tasks.workunit.client.0.smithi032.stdout:4/683: creat d4/d87/d96/d2f/d6e/dc0/fe6 x:0 0 0 2024-04-12T05:37:14.006 INFO:tasks.workunit.client.0.smithi032.stdout:4/684: write d4/d13/ff [3277659,16340] 0 2024-04-12T05:37:14.008 INFO:tasks.workunit.client.0.smithi032.stdout:5/793: unlink d2/df/d25/d50/d91/ld5 0 2024-04-12T05:37:14.011 INFO:tasks.workunit.client.0.smithi032.stdout:1/604: getdents d5/de/d17/d35/d3e/db4 0 2024-04-12T05:37:14.012 INFO:tasks.workunit.client.0.smithi032.stdout:1/605: dread - d5/de/d2a/fb6 zero size 2024-04-12T05:37:14.013 INFO:tasks.workunit.client.0.smithi032.stdout:1/606: chown d5/de/d17/d35/d3b/d49/f62 7525125 1 2024-04-12T05:37:14.015 INFO:tasks.workunit.client.0.smithi032.stdout:6/854: rename d4/d21/d2e/dca/c11f to d4/d21/d27/dad/c12a 0 2024-04-12T05:37:14.016 INFO:tasks.workunit.client.0.smithi032.stdout:6/855: stat d4/de0/de7/d37/d69/d76 0 2024-04-12T05:37:14.017 INFO:tasks.workunit.client.0.smithi032.stdout:6/856: write d4/d21/d2e/d32/f117 [2507145,103013] 0 2024-04-12T05:37:14.021 INFO:tasks.workunit.client.0.smithi032.stdout:9/578: getdents d9/d15/d1a/d1d/d5a 0 2024-04-12T05:37:14.025 INFO:tasks.workunit.client.0.smithi032.stdout:7/720: rename d0/d13/d21/d30/d31/c9e to d0/d34/cf4 0 2024-04-12T05:37:14.025 INFO:tasks.workunit.client.0.smithi032.stdout:8/740: getdents d0/db/dd/d11/d1b/d5f/daf/dc6 0 2024-04-12T05:37:14.027 INFO:tasks.workunit.client.0.smithi032.stdout:8/741: readlink d0/db/dd/d11/d1b/d22/d35/l66 0 2024-04-12T05:37:14.027 INFO:tasks.workunit.client.0.smithi032.stdout:6/857: symlink d4/d14/d25/l12b 0 2024-04-12T05:37:14.034 INFO:tasks.workunit.client.0.smithi032.stdout:2/727: truncate dd/d19/d9d/d1c/d64/fc1 2585253 0 2024-04-12T05:37:14.037 INFO:tasks.workunit.client.0.smithi032.stdout:6/858: dread d4/de0/de7/d92/d9e/fd1 [0,4194304] 0 2024-04-12T05:37:14.038 INFO:tasks.workunit.client.0.smithi032.stdout:9/579: creat d9/d15/d2c/dab/fc5 x:0 0 0 2024-04-12T05:37:14.040 INFO:tasks.workunit.client.0.smithi032.stdout:7/721: link d0/de/df/f24 d0/d34/dbe/dbb/ff5 0 2024-04-12T05:37:14.042 INFO:tasks.workunit.client.0.smithi032.stdout:7/722: dread - d0/d13/d21/d81/fdf zero size 2024-04-12T05:37:14.042 INFO:tasks.workunit.client.0.smithi032.stdout:0/742: dwrite d2/d8/df/d24/f5c [0,4194304] 0 2024-04-12T05:37:14.044 INFO:tasks.workunit.client.0.smithi032.stdout:2/728: mknod dd/d19/d9d/d42/da7/cf2 0 2024-04-12T05:37:14.044 INFO:tasks.workunit.client.0.smithi032.stdout:8/742: write d0/d58/fea [52763,42565] 0 2024-04-12T05:37:14.048 INFO:tasks.workunit.client.0.smithi032.stdout:7/723: truncate d0/d13/d21/f45 4452195 0 2024-04-12T05:37:14.050 INFO:tasks.workunit.client.0.smithi032.stdout:4/685: truncate d4/d87/d96/d2f/f4e 1548281 0 2024-04-12T05:37:14.052 INFO:tasks.workunit.client.0.smithi032.stdout:8/743: write d0/db/dd/d11/d1b/d22/fd6 [676442,115945] 0 2024-04-12T05:37:14.052 INFO:tasks.workunit.client.0.smithi032.stdout:0/743: symlink d2/dc/d3e/dac/d51/d83/lf4 0 2024-04-12T05:37:14.053 INFO:tasks.workunit.client.0.smithi032.stdout:1/607: dwrite d5/de/f66 [0,4194304] 0 2024-04-12T05:37:14.054 INFO:tasks.workunit.client.0.smithi032.stdout:7/724: write d0/d1c/d71/f8a [4318180,125564] 0 2024-04-12T05:37:14.060 INFO:tasks.workunit.client.0.smithi032.stdout:7/725: truncate d0/dd/f3b 9147141 0 2024-04-12T05:37:14.062 INFO:tasks.workunit.client.0.smithi032.stdout:1/608: fdatasync d5/de/d17/d35/d61/d77/d8f/d9c/fb5 0 2024-04-12T05:37:14.062 INFO:tasks.workunit.client.0.smithi032.stdout:6/859: symlink d4/d21/d27/dad/d55/l12c 0 2024-04-12T05:37:14.064 INFO:tasks.workunit.client.0.smithi032.stdout:0/744: fsync d2/d29/f5d 0 2024-04-12T05:37:14.068 INFO:tasks.workunit.client.0.smithi032.stdout:8/744: dread d0/db/dd/d11/d1b/d22/d54/da8/fd9 [0,4194304] 0 2024-04-12T05:37:14.070 INFO:tasks.workunit.client.0.smithi032.stdout:4/686: mkdir d4/d87/d96/d2f/d9a/de7 0 2024-04-12T05:37:14.071 INFO:tasks.workunit.client.0.smithi032.stdout:9/580: write d9/d15/d1a/d1d/fb9 [1538454,104593] 0 2024-04-12T05:37:14.072 INFO:tasks.workunit.client.0.smithi032.stdout:2/729: dread dd/d19/d9d/d1c/f81 [0,4194304] 0 2024-04-12T05:37:14.074 INFO:tasks.workunit.client.0.smithi032.stdout:1/609: rmdir d5/de/d17/d35/d3e/db4 39 2024-04-12T05:37:14.074 INFO:tasks.workunit.client.0.smithi032.stdout:1/610: fdatasync d5/d2c/f43 0 2024-04-12T05:37:14.076 INFO:tasks.workunit.client.0.smithi032.stdout:6/860: mknod d4/de0/de7/d88/c12d 0 2024-04-12T05:37:14.079 INFO:tasks.workunit.client.0.smithi032.stdout:7/726: creat d0/d34/dbe/ff6 x:0 0 0 2024-04-12T05:37:14.081 INFO:tasks.workunit.client.0.smithi032.stdout:4/687: fdatasync d4/f1d 0 2024-04-12T05:37:14.087 INFO:tasks.workunit.client.0.smithi032.stdout:8/745: symlink d0/d72/l106 0 2024-04-12T05:37:14.091 INFO:tasks.workunit.client.0.smithi032.stdout:8/746: truncate d0/db/f55 4401012 0 2024-04-12T05:37:14.091 INFO:tasks.workunit.client.0.smithi032.stdout:1/611: dwrite d5/de/f28 [0,4194304] 0 2024-04-12T05:37:14.092 INFO:tasks.workunit.client.0.smithi032.stdout:1/612: readlink d5/de/l18 0 2024-04-12T05:37:14.093 INFO:tasks.workunit.client.0.smithi032.stdout:8/747: chown d0/db/dd/d11/d1b/d22/d54/da8/fc7 308 1 2024-04-12T05:37:14.093 INFO:tasks.workunit.client.0.smithi032.stdout:7/727: creat d0/d10/d70/d7b/dd2/ff7 x:0 0 0 2024-04-12T05:37:14.097 INFO:tasks.workunit.client.0.smithi032.stdout:2/730: write f8 [1864640,93901] 0 2024-04-12T05:37:14.098 INFO:tasks.workunit.client.0.smithi032.stdout:6/861: write d4/d21/d2e/f39 [2463194,36168] 0 2024-04-12T05:37:14.104 INFO:tasks.workunit.client.0.smithi032.stdout:0/745: getdents d2/d8/d17/d1b/d20 0 2024-04-12T05:37:14.109 INFO:tasks.workunit.client.0.smithi032.stdout:4/688: rmdir d4/d87/d96/d2f/d9a/de7 0 2024-04-12T05:37:14.110 INFO:tasks.workunit.client.0.smithi032.stdout:2/731: creat dd/d19/d9d/d1c/d64/d36/ff3 x:0 0 0 2024-04-12T05:37:14.110 INFO:tasks.workunit.client.0.smithi032.stdout:9/581: getdents d9/d58 0 2024-04-12T05:37:14.111 INFO:tasks.workunit.client.0.smithi032.stdout:8/748: write d0/d36/d84/dad/fab [4809611,105999] 0 2024-04-12T05:37:14.114 INFO:tasks.workunit.client.0.smithi032.stdout:4/689: readlink d4/d13/d44/l9f 0 2024-04-12T05:37:14.116 INFO:tasks.workunit.client.0.smithi032.stdout:7/728: rename d0/d13/d21/f45 to d0/de/ff8 0 2024-04-12T05:37:14.118 INFO:tasks.workunit.client.0.smithi032.stdout:1/613: getdents d5/de/d17/d1b 0 2024-04-12T05:37:14.119 INFO:tasks.workunit.client.0.smithi032.stdout:0/746: creat d2/d28/dd6/ff5 x:0 0 0 2024-04-12T05:37:14.122 INFO:tasks.workunit.client.0.smithi032.stdout:4/690: mkdir d4/d87/d96/d2f/d3b/de8 0 2024-04-12T05:37:14.123 INFO:tasks.workunit.client.0.smithi032.stdout:6/862: dwrite d4/d21/d27/dad/d55/d59/f9f [0,4194304] 0 2024-04-12T05:37:14.127 INFO:tasks.workunit.client.0.smithi032.stdout:9/582: symlink d9/d15/d1a/d1d/d5a/d6b/d83/lc6 0 2024-04-12T05:37:14.127 INFO:tasks.workunit.client.0.smithi032.stdout:0/747: write d2/d28/dd6/ff5 [140650,46601] 0 2024-04-12T05:37:14.129 INFO:tasks.workunit.client.0.smithi032.stdout:2/732: rename dd/d19/f24 to dd/d19/d9d/d1c/de0/de1/ff4 0 2024-04-12T05:37:14.134 INFO:tasks.workunit.client.0.smithi032.stdout:7/729: creat d0/d1c/d71/ff9 x:0 0 0 2024-04-12T05:37:14.135 INFO:tasks.workunit.client.0.smithi032.stdout:6/863: mkdir d4/d21/d2e/d32/d12e 0 2024-04-12T05:37:14.136 INFO:tasks.workunit.client.0.smithi032.stdout:8/749: dwrite d0/d36/d84/dad/fab [4194304,4194304] 0 2024-04-12T05:37:14.137 INFO:tasks.workunit.client.0.smithi032.stdout:9/583: chown d9/d15/d1a/d31/f98 783600113 1 2024-04-12T05:37:14.140 INFO:tasks.workunit.client.0.smithi032.stdout:1/614: dwrite d5/de/d2a/f80 [0,4194304] 0 2024-04-12T05:37:14.142 INFO:tasks.workunit.client.0.smithi032.stdout:4/691: symlink d4/d87/d96/d4d/db5/dcf/de3/le9 0 2024-04-12T05:37:14.142 INFO:tasks.workunit.client.0.smithi032.stdout:2/733: write dd/d19/d20/f28 [1731640,102657] 0 2024-04-12T05:37:14.147 INFO:tasks.workunit.client.0.smithi032.stdout:0/748: creat d2/dc/d3e/dac/d51/d83/d9d/ff6 x:0 0 0 2024-04-12T05:37:14.148 INFO:tasks.workunit.client.0.smithi032.stdout:6/864: creat d4/d21/d27/dad/f12f x:0 0 0 2024-04-12T05:37:14.150 INFO:tasks.workunit.client.0.smithi032.stdout:2/734: chown dd/d19/d9d/d42/d4a 5 1 2024-04-12T05:37:14.150 INFO:tasks.workunit.client.0.smithi032.stdout:7/730: truncate d0/d10/f60 1557717 0 2024-04-12T05:37:14.154 INFO:tasks.workunit.client.0.smithi032.stdout:4/692: creat d4/d87/d96/d32/fea x:0 0 0 2024-04-12T05:37:14.156 INFO:tasks.workunit.client.0.smithi032.stdout:7/731: stat d0/d34/dbe/dbb/c9a 0 2024-04-12T05:37:14.156 INFO:tasks.workunit.client.0.smithi032.stdout:9/584: symlink d9/d15/d1a/d24/d27/d88/dc1/lc7 0 2024-04-12T05:37:14.159 INFO:tasks.workunit.client.0.smithi032.stdout:2/735: chown dd/d19/d9d/d1c/d64/d32/d6d/f77 474 1 2024-04-12T05:37:14.160 INFO:tasks.workunit.client.0.smithi032.stdout:8/750: dwrite d0/db/dd/d11/d1b/d5f/daf/fa2 [0,4194304] 0 2024-04-12T05:37:14.161 INFO:tasks.workunit.client.0.smithi032.stdout:1/615: link d5/de/d17/l1a d5/de/d17/d35/lc0 0 2024-04-12T05:37:14.161 INFO:tasks.workunit.client.0.smithi032.stdout:2/736: dread - dd/d8c/fd7 zero size 2024-04-12T05:37:14.161 INFO:tasks.workunit.client.0.smithi032.stdout:7/732: write d0/de/f12 [1477642,128457] 0 2024-04-12T05:37:14.165 INFO:tasks.workunit.client.0.smithi032.stdout:8/751: rmdir d0/d36/d84/dad 39 2024-04-12T05:37:14.167 INFO:tasks.workunit.client.0.smithi032.stdout:6/865: write d4/ff [607800,33376] 0 2024-04-12T05:37:14.169 INFO:tasks.workunit.client.0.smithi032.stdout:1/616: fdatasync d5/de/d17/d35/d3b/f63 0 2024-04-12T05:37:14.171 INFO:tasks.workunit.client.0.smithi032.stdout:0/749: write d2/d8/d17/d64/fa5 [4824422,92507] 0 2024-04-12T05:37:14.175 INFO:tasks.workunit.client.0.smithi032.stdout:4/693: getdents d4/d87/d96/d2f/d7c/dc2 0 2024-04-12T05:37:14.176 INFO:tasks.workunit.client.0.smithi032.stdout:1/617: chown d5/de/d5b/f95 0 1 2024-04-12T05:37:14.176 INFO:tasks.workunit.client.0.smithi032.stdout:0/750: fsync d2/d29/d8a/fd7 0 2024-04-12T05:37:14.176 INFO:tasks.workunit.client.0.smithi032.stdout:6/866: rmdir d4/d21/d2e/da1 39 2024-04-12T05:37:14.177 INFO:tasks.workunit.client.0.smithi032.stdout:2/737: creat dd/d19/d9d/d1c/d64/d32/ff5 x:0 0 0 2024-04-12T05:37:14.180 INFO:tasks.workunit.client.0.smithi032.stdout:7/733: rename d0/dd/d51/d65/lb7 to d0/d1c/lfa 0 2024-04-12T05:37:14.182 INFO:tasks.workunit.client.0.smithi032.stdout:8/752: dwrite d0/db/dd/d11/d1b/d22/d35/ff6 [0,4194304] 0 2024-04-12T05:37:14.182 INFO:tasks.workunit.client.0.smithi032.stdout:6/867: creat d4/de0/de7/d88/d8d/f130 x:0 0 0 2024-04-12T05:37:14.185 INFO:tasks.workunit.client.0.smithi032.stdout:0/751: rename d2/d8/d66/ce2 to d2/dc/d3e/cf7 0 2024-04-12T05:37:14.185 INFO:tasks.workunit.client.0.smithi032.stdout:4/694: creat d4/d87/d96/d4d/db5/dcf/de3/feb x:0 0 0 2024-04-12T05:37:14.194 INFO:tasks.workunit.client.0.smithi032.stdout:2/738: truncate dd/d19/d9d/fae 294772 0 2024-04-12T05:37:14.194 INFO:tasks.workunit.client.0.smithi032.stdout:4/695: write d4/d87/d96/d2f/d9a/fa7 [236357,109619] 0 2024-04-12T05:37:14.194 INFO:tasks.workunit.client.0.smithi032.stdout:6/868: mknod d4/de0/deb/c131 0 2024-04-12T05:37:14.194 INFO:tasks.workunit.client.0.smithi032.stdout:7/734: mkdir d0/d13/d21/d30/dfb 0 2024-04-12T05:37:14.196 INFO:tasks.workunit.client.0.smithi032.stdout:0/752: write d2/d28/d87/f95 [1000057,90051] 0 2024-04-12T05:37:14.197 INFO:tasks.workunit.client.0.smithi032.stdout:7/735: dread - d0/fc4 zero size 2024-04-12T05:37:14.198 INFO:tasks.workunit.client.0.smithi032.stdout:1/618: dwrite d5/f47 [0,4194304] 0 2024-04-12T05:37:14.200 INFO:tasks.workunit.client.0.smithi032.stdout:0/753: stat d2/d28/d87/l93 0 2024-04-12T05:37:14.202 INFO:tasks.workunit.client.0.smithi032.stdout:6/869: creat d4/d21/d2e/d32/d94/f132 x:0 0 0 2024-04-12T05:37:14.202 INFO:tasks.workunit.client.0.smithi032.stdout:7/736: chown d0/de/f44 89013638 1 2024-04-12T05:37:14.203 INFO:tasks.workunit.client.0.smithi032.stdout:8/753: dwrite d0/db/dd/d11/d1b/d22/d54/f64 [4194304,4194304] 0 2024-04-12T05:37:14.211 INFO:tasks.workunit.client.0.smithi032.stdout:2/739: write dd/d19/d20/f4d [5044168,21594] 0 2024-04-12T05:37:14.215 INFO:tasks.workunit.client.0.smithi032.stdout:8/754: dread d0/db/dd/fec [0,4194304] 0 2024-04-12T05:37:14.218 INFO:tasks.workunit.client.0.smithi032.stdout:4/696: truncate d4/d87/d96/d2f/d6e/fcc 1126945 0 2024-04-12T05:37:14.219 INFO:tasks.workunit.client.0.smithi032.stdout:0/754: link d2/f6 d2/d28/d87/d79/ff8 0 2024-04-12T05:37:14.223 INFO:tasks.workunit.client.0.smithi032.stdout:0/755: readlink d2/dc/d41/l67 0 2024-04-12T05:37:14.229 INFO:tasks.workunit.client.0.smithi032.stdout:7/737: dwrite d0/d34/fc0 [0,4194304] 0 2024-04-12T05:37:14.229 INFO:tasks.workunit.client.0.smithi032.stdout:1/619: dwrite d5/de/d17/d35/d3b/d49/f56 [0,4194304] 0 2024-04-12T05:37:14.233 INFO:tasks.workunit.client.0.smithi032.stdout:7/738: stat d0/d13/f16 0 2024-04-12T05:37:14.233 INFO:tasks.workunit.client.0.smithi032.stdout:6/870: dread d4/d21/d27/dad/d55/f65 [0,4194304] 0 2024-04-12T05:37:14.246 INFO:tasks.workunit.client.0.smithi032.stdout:4/697: dwrite d4/d87/d96/d2f/d3b/f3c [0,4194304] 0 2024-04-12T05:37:14.247 INFO:tasks.workunit.client.0.smithi032.stdout:1/620: dwrite d5/de/f1f [0,4194304] 0 2024-04-12T05:37:14.247 INFO:tasks.workunit.client.0.smithi032.stdout:4/698: dread - d4/d13/d44/f5c zero size 2024-04-12T05:37:14.255 INFO:tasks.workunit.client.0.smithi032.stdout:8/755: dread d0/db/f6b [0,4194304] 0 2024-04-12T05:37:14.256 INFO:tasks.workunit.client.0.smithi032.stdout:6/871: getdents d4/d14/d25 0 2024-04-12T05:37:14.257 INFO:tasks.workunit.client.0.smithi032.stdout:1/621: rmdir d5/de/d17/d35/d73/da9 0 2024-04-12T05:37:14.259 INFO:tasks.workunit.client.0.smithi032.stdout:4/699: write d4/d87/d96/d2f/fda [750763,119477] 0 2024-04-12T05:37:14.263 INFO:tasks.workunit.client.0.smithi032.stdout:8/756: creat d0/db/dd/d11/d1b/d22/d29/ded/f107 x:0 0 0 2024-04-12T05:37:14.264 INFO:tasks.workunit.client.0.smithi032.stdout:4/700: mkdir d4/d87/d96/d2f/d3b/da2/dec 0 2024-04-12T05:37:14.264 INFO:tasks.workunit.client.0.smithi032.stdout:1/622: dread d5/de/d5b/f95 [0,4194304] 0 2024-04-12T05:37:14.265 INFO:tasks.workunit.client.0.smithi032.stdout:6/872: rmdir d4/d21/d2e/d32/d12e 0 2024-04-12T05:37:14.266 INFO:tasks.workunit.client.0.smithi032.stdout:8/757: creat d0/d58/f108 x:0 0 0 2024-04-12T05:37:14.266 INFO:tasks.workunit.client.0.smithi032.stdout:1/623: stat d5/de/d17/d35/d73/fba 0 2024-04-12T05:37:14.268 INFO:tasks.workunit.client.0.smithi032.stdout:6/873: chown d4/de0/de7/d92/l10a 28028 1 2024-04-12T05:37:14.268 INFO:tasks.workunit.client.0.smithi032.stdout:1/624: chown d5/de/d17/c29 1762931338 1 2024-04-12T05:37:14.268 INFO:tasks.workunit.client.0.smithi032.stdout:4/701: symlink d4/d87/d96/d2f/d7c/led 0 2024-04-12T05:37:14.271 INFO:tasks.workunit.client.0.smithi032.stdout:4/702: write d4/d87/d96/d2f/d6e/dc0/fe6 [288604,110168] 0 2024-04-12T05:37:14.271 INFO:tasks.workunit.client.0.smithi032.stdout:6/874: fdatasync d4/de0/de7/d92/d9e/db2/fde 0 2024-04-12T05:37:14.273 INFO:tasks.workunit.client.0.smithi032.stdout:8/758: link d0/c52 d0/db/d38/d63/c109 0 2024-04-12T05:37:14.276 INFO:tasks.workunit.client.0.smithi032.stdout:1/625: mknod d5/de/d17/d35/d3e/db4/cc1 0 2024-04-12T05:37:14.278 INFO:tasks.workunit.client.0.smithi032.stdout:4/703: mknod d4/d13/cee 0 2024-04-12T05:37:14.279 INFO:tasks.workunit.client.0.smithi032.stdout:4/704: fsync d4/d87/d96/d4d/d78/f64 0 2024-04-12T05:37:14.282 INFO:tasks.workunit.client.0.smithi032.stdout:8/759: mkdir d0/d36/d10a 0 2024-04-12T05:37:14.283 INFO:tasks.workunit.client.0.smithi032.stdout:1/626: rmdir d5/de/d17/d35/d3b 39 2024-04-12T05:37:14.283 INFO:tasks.workunit.client.0.smithi032.stdout:8/760: write d0/d58/fea [180806,89697] 0 2024-04-12T05:37:14.286 INFO:tasks.workunit.client.0.smithi032.stdout:8/761: fsync d0/f7 0 2024-04-12T05:37:14.288 INFO:tasks.workunit.client.0.smithi032.stdout:6/875: getdents d4/de0/de7/d92/d9e/df6 0 2024-04-12T05:37:14.289 INFO:tasks.workunit.client.0.smithi032.stdout:1/627: truncate d5/de/d17/d35/d3e/f4c 1614096 0 2024-04-12T05:37:14.292 INFO:tasks.workunit.client.0.smithi032.stdout:6/876: rename d4/d21/d27/c2d to d4/d21/d27/dad/d55/df4/c133 0 2024-04-12T05:37:14.293 INFO:tasks.workunit.client.0.smithi032.stdout:8/762: symlink d0/db/dd/d11/d1b/d22/d35/l10b 0 2024-04-12T05:37:14.294 INFO:tasks.workunit.client.0.smithi032.stdout:1/628: truncate d5/f30 2036863 0 2024-04-12T05:37:14.297 INFO:tasks.workunit.client.0.smithi032.stdout:6/877: rename d4/d21/lbe to d4/de0/de7/d79/dc6/l134 0 2024-04-12T05:37:14.298 INFO:tasks.workunit.client.0.smithi032.stdout:1/629: creat d5/de/d17/d35/d73/fc2 x:0 0 0 2024-04-12T05:37:14.299 INFO:tasks.workunit.client.0.smithi032.stdout:6/878: fdatasync d4/d21/d2e/da1/f125 0 2024-04-12T05:37:14.300 INFO:tasks.workunit.client.0.smithi032.stdout:1/630: mknod d5/de/d17/cc3 0 2024-04-12T05:37:14.302 INFO:tasks.workunit.client.0.smithi032.stdout:6/879: symlink d4/de0/l135 0 2024-04-12T05:37:14.303 INFO:tasks.workunit.client.0.smithi032.stdout:1/631: creat d5/de/d17/d35/d61/fc4 x:0 0 0 2024-04-12T05:37:14.305 INFO:tasks.workunit.client.0.smithi032.stdout:6/880: mknod d4/de0/de7/d79/dc6/d101/c136 0 2024-04-12T05:37:14.306 INFO:tasks.workunit.client.0.smithi032.stdout:6/881: truncate d4/d14/d3c/fb3 662336 0 2024-04-12T05:37:14.306 INFO:tasks.workunit.client.0.smithi032.stdout:1/632: symlink d5/de/d17/d35/d73/lc5 0 2024-04-12T05:37:14.307 INFO:tasks.workunit.client.0.smithi032.stdout:1/633: dread - d5/d2c/d4f/fb9 zero size 2024-04-12T05:37:14.308 INFO:tasks.workunit.client.0.smithi032.stdout:6/882: fdatasync d4/d21/db4/fcd 0 2024-04-12T05:37:14.310 INFO:tasks.workunit.client.0.smithi032.stdout:1/634: fsync d5/de/d17/d35/d73/f76 0 2024-04-12T05:37:14.312 INFO:tasks.workunit.client.0.smithi032.stdout:6/883: mknod d4/de0/de7/d37/d69/d8a/c137 0 2024-04-12T05:37:14.314 INFO:tasks.workunit.client.0.smithi032.stdout:6/884: write d4/d21/d2e/ff0 [878240,35843] 0 2024-04-12T05:37:14.316 INFO:tasks.workunit.client.0.smithi032.stdout:6/885: read d4/de0/de7/d92/d9e/fd1 [18755,112625] 0 2024-04-12T05:37:14.322 INFO:tasks.workunit.client.0.smithi032.stdout:6/886: dread d4/d21/d27/dad/d55/d59/f9f [0,4194304] 0 2024-04-12T05:37:14.325 INFO:tasks.workunit.client.0.smithi032.stdout:6/887: unlink d4/d21/d2e/d32/d94/cdf 0 2024-04-12T05:37:14.327 INFO:tasks.workunit.client.0.smithi032.stdout:6/888: fdatasync d4/de0/de7/d79/dc6/fc8 0 2024-04-12T05:37:14.329 INFO:tasks.workunit.client.0.smithi032.stdout:6/889: mkdir d4/d21/d27/d138 0 2024-04-12T05:37:14.334 INFO:tasks.workunit.client.0.smithi032.stdout:6/890: write d4/de0/de7/f6f [2351565,35637] 0 2024-04-12T05:37:14.337 INFO:tasks.workunit.client.0.smithi032.stdout:6/891: unlink d4/d21/d2e/lc3 0 2024-04-12T05:37:14.350 INFO:tasks.workunit.client.0.smithi032.stdout:6/892: dread d4/d14/d25/fe8 [0,4194304] 0 2024-04-12T05:37:14.351 INFO:tasks.workunit.client.0.smithi032.stdout:6/893: chown d4/f6 2365 1 2024-04-12T05:37:14.353 INFO:tasks.workunit.client.0.smithi032.stdout:6/894: rmdir d4/de0/de7/d92/d9e 39 2024-04-12T05:37:14.360 INFO:tasks.workunit.client.0.smithi032.stdout:6/895: dwrite d4/de0/de7/d79/dc6/f119 [0,4194304] 0 2024-04-12T05:37:14.361 INFO:tasks.workunit.client.0.smithi032.stdout:6/896: readlink d4/d21/d27/dad/d55/l6b 0 2024-04-12T05:37:14.364 INFO:tasks.workunit.client.0.smithi032.stdout:6/897: creat d4/d21/d2e/da1/f139 x:0 0 0 2024-04-12T05:37:14.369 INFO:tasks.workunit.client.0.smithi032.stdout:6/898: link d4/de0/de7/d37/c68 d4/d14/d3c/c13a 0 2024-04-12T05:37:14.382 INFO:tasks.workunit.client.0.smithi032.stdout:6/899: dwrite d4/de0/de7/d88/fac [0,4194304] 0 2024-04-12T05:37:14.390 INFO:tasks.workunit.client.0.smithi032.stdout:6/900: dwrite d4/d21/d2e/d32/d94/f126 [0,4194304] 0 2024-04-12T05:37:14.397 INFO:tasks.workunit.client.0.smithi032.stdout:6/901: mkdir d4/de0/de7/d92/d9e/df6/d13b 0 2024-04-12T05:37:14.415 INFO:tasks.workunit.client.0.smithi032.stdout:6/902: dwrite d4/d21/d27/dad/d55/f72 [0,4194304] 0 2024-04-12T05:37:14.417 INFO:tasks.workunit.client.0.smithi032.stdout:6/903: write d4/de0/de7/d37/d69/d8a/f10f [580482,46822] 0 2024-04-12T05:37:14.428 INFO:tasks.workunit.client.0.smithi032.stdout:6/904: truncate d4/d21/d2e/f39 3636096 0 2024-04-12T05:37:14.429 INFO:tasks.workunit.client.0.smithi032.stdout:6/905: mknod d4/de0/de7/d37/c13c 0 2024-04-12T05:37:14.432 INFO:tasks.workunit.client.0.smithi032.stdout:6/906: creat d4/d21/d2e/f13d x:0 0 0 2024-04-12T05:37:14.435 INFO:tasks.workunit.client.0.smithi032.stdout:6/907: unlink d4/d21/d2e/fb6 0 2024-04-12T05:37:14.438 INFO:tasks.workunit.client.0.smithi032.stdout:6/908: truncate d4/de0/de7/d92/d9e/db2/fde 428086 0 2024-04-12T05:37:14.438 INFO:tasks.workunit.client.0.smithi032.stdout:6/909: dread - d4/de0/de7/d88/d8d/f130 zero size 2024-04-12T05:37:14.445 INFO:tasks.workunit.client.0.smithi032.stdout:6/910: dwrite d4/de0/de7/d88/fcc [0,4194304] 0 2024-04-12T05:37:14.458 INFO:tasks.workunit.client.0.smithi032.stdout:6/911: dread d4/d21/d2e/d32/f36 [0,4194304] 0 2024-04-12T05:37:14.461 INFO:tasks.workunit.client.0.smithi032.stdout:6/912: rmdir d4/de0/de7/d37/dfb 0 2024-04-12T05:37:14.465 INFO:tasks.workunit.client.0.smithi032.stdout:6/913: dread d4/d21/d27/dad/d55/f72 [0,4194304] 0 2024-04-12T05:37:14.466 INFO:tasks.workunit.client.0.smithi032.stdout:6/914: rmdir d4/d14/d3c 39 2024-04-12T05:37:14.470 INFO:tasks.workunit.client.0.smithi032.stdout:6/915: dread d4/d21/d27/dad/d55/f65 [0,4194304] 0 2024-04-12T05:37:14.473 INFO:tasks.workunit.client.0.smithi032.stdout:6/916: write d4/d21/d27/dad/d55/f72 [1786086,70507] 0 2024-04-12T05:37:14.474 INFO:tasks.workunit.client.0.smithi032.stdout:6/917: chown d4/d21/d2e/ff0 40124378 1 2024-04-12T05:37:14.476 INFO:tasks.workunit.client.0.smithi032.stdout:6/918: fsync d4/f9 0 2024-04-12T05:37:14.477 INFO:tasks.workunit.client.0.smithi032.stdout:6/919: chown d4/de0/de7/d92/l10a 4 1 2024-04-12T05:37:14.479 INFO:tasks.workunit.client.0.smithi032.stdout:6/920: mknod d4/d21/d2e/d32/c13e 0 2024-04-12T05:37:14.482 INFO:tasks.workunit.client.0.smithi032.stdout:6/921: link d4/de0/de7/d92/d9e/fd1 d4/de0/de7/d79/dc6/d101/f13f 0 2024-04-12T05:37:14.483 INFO:tasks.workunit.client.0.smithi032.stdout:6/922: fdatasync d4/d21/f2f 0 2024-04-12T05:37:14.484 INFO:tasks.workunit.client.0.smithi032.stdout:6/923: readlink d4/d21/d27/dad/d55/d99/l9c 0 2024-04-12T05:37:14.485 INFO:tasks.workunit.client.0.smithi032.stdout:6/924: chown d4/de0/de7/d37/d47/cd0 282 1 2024-04-12T05:37:14.486 INFO:tasks.workunit.client.0.smithi032.stdout:6/925: chown d4/cc 1 1 2024-04-12T05:37:14.491 INFO:tasks.workunit.client.0.smithi032.stdout:6/926: write d4/d21/f3b [4933619,57300] 0 2024-04-12T05:37:14.492 INFO:tasks.workunit.client.0.smithi032.stdout:6/927: chown d4/d21/d27/dad/d55/la5 40622605 1 2024-04-12T05:37:14.495 INFO:tasks.workunit.client.0.smithi032.stdout:6/928: truncate d4/de0/de7/d79/fd3 268462 0 2024-04-12T05:37:14.499 INFO:tasks.workunit.client.0.smithi032.stdout:6/929: rename d4/d21/d2e/da1/c127 to d4/de0/de7/d88/d8d/c140 0 2024-04-12T05:37:14.501 INFO:tasks.workunit.client.0.smithi032.stdout:6/930: creat d4/d21/d27/dad/d55/df4/f141 x:0 0 0 2024-04-12T05:37:14.504 INFO:tasks.workunit.client.0.smithi032.stdout:6/931: creat d4/d21/d2e/d32/f142 x:0 0 0 2024-04-12T05:37:14.511 INFO:tasks.workunit.client.0.smithi032.stdout:6/932: write d4/de0/de7/d37/d69/d76/fb1 [1209069,112911] 0 2024-04-12T05:37:14.518 INFO:tasks.workunit.client.0.smithi032.stdout:6/933: link d4/de0/de7/d92/d9e/fd1 d4/d21/f143 0 2024-04-12T05:37:14.521 INFO:tasks.workunit.client.0.smithi032.stdout:6/934: write d4/d74/f91 [1162605,121394] 0 2024-04-12T05:37:14.522 INFO:tasks.workunit.client.0.smithi032.stdout:6/935: dread - d4/de0/de7/d37/d69/d8a/fd5 zero size 2024-04-12T05:37:14.524 INFO:tasks.workunit.client.0.smithi032.stdout:6/936: stat d4/d21/d27/dad/d55/f72 0 2024-04-12T05:37:14.525 INFO:tasks.workunit.client.0.smithi032.stdout:6/937: dread - d4/d21/d27/dad/f12f zero size 2024-04-12T05:37:14.532 INFO:tasks.workunit.client.0.smithi032.stdout:6/938: dwrite d4/de0/de7/d79/dc6/f11d [0,4194304] 0 2024-04-12T05:37:14.536 INFO:tasks.workunit.client.0.smithi032.stdout:6/939: creat d4/de0/de7/dd9/f144 x:0 0 0 2024-04-12T05:37:14.547 INFO:tasks.workunit.client.0.smithi032.stdout:6/940: fdatasync d4/de0/de7/d79/dc6/f11d 0 2024-04-12T05:37:14.549 INFO:tasks.workunit.client.0.smithi032.stdout:6/941: mknod d4/d21/d27/c145 0 2024-04-12T05:37:14.555 INFO:tasks.workunit.client.0.smithi032.stdout:6/942: truncate d4/de0/de7/dc4/f116 3748981 0 2024-04-12T05:37:14.558 INFO:tasks.workunit.client.0.smithi032.stdout:6/943: creat d4/de0/de7/d37/d69/d76/f146 x:0 0 0 2024-04-12T05:37:14.563 INFO:tasks.workunit.client.0.smithi032.stdout:6/944: chown d4/de0/de7/d92/d9e/db2/ce1 10 1 2024-04-12T05:37:14.563 INFO:tasks.workunit.client.0.smithi032.stdout:6/945: write d4/d21/d27/dad/d55/df4/f141 [346715,112572] 0 2024-04-12T05:37:14.564 INFO:tasks.workunit.client.0.smithi032.stdout:6/946: creat d4/de0/de7/d92/d9e/df6/f147 x:0 0 0 2024-04-12T05:37:14.573 INFO:tasks.workunit.client.0.smithi032.stdout:6/947: dread d4/d21/d2e/ff0 [0,4194304] 0 2024-04-12T05:37:14.575 INFO:tasks.workunit.client.0.smithi032.stdout:6/948: symlink d4/d21/d2e/d32/d94/l148 0 2024-04-12T05:37:14.578 INFO:tasks.workunit.client.0.smithi032.stdout:6/949: mknod d4/de0/de7/d92/d9e/df6/d13b/c149 0 2024-04-12T05:37:14.582 INFO:tasks.workunit.client.0.smithi032.stdout:6/950: unlink d4/d21/d27/dad/d55/df4/f141 0 2024-04-12T05:37:14.589 INFO:tasks.workunit.client.1.smithi067.stdout:7/155: readlink d0/l6 0 2024-04-12T05:37:14.589 INFO:tasks.workunit.client.0.smithi032.stdout:6/951: dwrite d4/d21/d27/dad/d55/d99/f106 [0,4194304] 0 2024-04-12T05:37:14.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:14 smithi032 ceph-mon[39731]: pgmap v268: 65 pgs: 65 active+clean; 1.7 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 75 MiB/s rd, 218 MiB/s wr, 422 op/s 2024-04-12T05:37:14.590 INFO:tasks.workunit.client.1.smithi067.stdout:7/156: fsync d0/d2/d5/d1e/f20 0 2024-04-12T05:37:14.593 INFO:tasks.workunit.client.0.smithi032.stdout:6/952: mknod d4/de0/de7/dd9/c14a 0 2024-04-12T05:37:14.593 INFO:tasks.workunit.client.1.smithi067.stdout:7/157: dread - d0/f12 zero size 2024-04-12T05:37:14.596 INFO:tasks.workunit.client.1.smithi067.stdout:9/119: readlink d1/d12/d15/l1a 0 2024-04-12T05:37:14.598 INFO:tasks.workunit.client.0.smithi032.stdout:6/953: symlink d4/de0/de7/dd9/l14b 0 2024-04-12T05:37:14.600 INFO:tasks.workunit.client.0.smithi032.stdout:6/954: write d4/d21/d2e/f13d [530133,20126] 0 2024-04-12T05:37:14.601 INFO:tasks.workunit.client.0.smithi032.stdout:6/955: write d4/de0/de7/d37/d69/d8a/f10f [1205449,121597] 0 2024-04-12T05:37:14.603 INFO:tasks.workunit.client.1.smithi067.stdout:2/170: creat d5/da/de/d14/f3b x:0 0 0 2024-04-12T05:37:14.603 INFO:tasks.workunit.client.1.smithi067.stdout:6/112: rmdir dd/d11 39 2024-04-12T05:37:14.605 INFO:tasks.workunit.client.0.smithi032.stdout:6/956: creat d4/d21/d27/f14c x:0 0 0 2024-04-12T05:37:14.606 INFO:tasks.workunit.client.0.smithi032.stdout:6/957: write d4/ff [1467114,101558] 0 2024-04-12T05:37:14.613 INFO:tasks.workunit.client.1.smithi067.stdout:4/66: creat d8/d9/f17 x:0 0 0 2024-04-12T05:37:14.614 INFO:tasks.workunit.client.1.smithi067.stdout:0/234: dread dd/d12/d14/d1a/f4b [0,4194304] 0 2024-04-12T05:37:14.614 INFO:tasks.workunit.client.1.smithi067.stdout:3/110: creat dc/d1d/f22 x:0 0 0 2024-04-12T05:37:14.615 INFO:tasks.workunit.client.1.smithi067.stdout:8/123: mkdir d6/d8/d28 0 2024-04-12T05:37:14.618 INFO:tasks.workunit.client.0.smithi032.stdout:6/958: dwrite d4/d21/d2e/d32/d94/f126 [4194304,4194304] 0 2024-04-12T05:37:14.618 INFO:tasks.workunit.client.1.smithi067.stdout:0/235: readlink dd/d12/d14/d1a/d3d/l46 0 2024-04-12T05:37:14.619 INFO:tasks.workunit.client.1.smithi067.stdout:3/111: dread - dc/d1d/f22 zero size 2024-04-12T05:37:14.621 INFO:tasks.workunit.client.0.smithi032.stdout:6/959: write d4/d21/d2e/d32/f117 [666913,123601] 0 2024-04-12T05:37:14.624 INFO:tasks.workunit.client.1.smithi067.stdout:9/120: creat d1/d12/d15/f1c x:0 0 0 2024-04-12T05:37:14.624 INFO:tasks.workunit.client.1.smithi067.stdout:2/171: symlink d5/da/de/d14/l3c 0 2024-04-12T05:37:14.625 INFO:tasks.workunit.client.0.smithi032.stdout:6/960: symlink d4/d21/d2e/d32/d94/l14d 0 2024-04-12T05:37:14.629 INFO:tasks.workunit.client.1.smithi067.stdout:9/121: chown d1/f11 51382204 1 2024-04-12T05:37:14.632 INFO:tasks.workunit.client.1.smithi067.stdout:2/172: write d5/da/de/f26 [4709465,31269] 0 2024-04-12T05:37:14.636 INFO:tasks.workunit.client.1.smithi067.stdout:8/124: dwrite d6/d8/d18/df/f1c [0,4194304] 0 2024-04-12T05:37:14.636 INFO:tasks.workunit.client.1.smithi067.stdout:1/177: creat d4/d9/d12/d28/f32 x:0 0 0 2024-04-12T05:37:14.644 INFO:tasks.workunit.client.1.smithi067.stdout:0/236: rmdir dd/d12/d14/d1a/d3d/d41 39 2024-04-12T05:37:14.645 INFO:tasks.workunit.client.1.smithi067.stdout:1/178: read d4/d9/f31 [2539110,99571] 0 2024-04-12T05:37:14.649 INFO:tasks.workunit.client.1.smithi067.stdout:6/113: dwrite fc [0,4194304] 0 2024-04-12T05:37:14.651 INFO:tasks.workunit.client.1.smithi067.stdout:1/179: fdatasync d4/d9/f22 0 2024-04-12T05:37:14.651 INFO:tasks.workunit.client.1.smithi067.stdout:2/173: truncate d5/da/de/d14/d34/f37 1020392 0 2024-04-12T05:37:14.653 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:14 smithi067 ceph-mon[49010]: pgmap v268: 65 pgs: 65 active+clean; 1.7 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 75 MiB/s rd, 218 MiB/s wr, 422 op/s 2024-04-12T05:37:14.656 INFO:tasks.workunit.client.1.smithi067.stdout:5/295: getdents d0/d6 0 2024-04-12T05:37:14.657 INFO:tasks.workunit.client.1.smithi067.stdout:2/174: dread - d5/da/de/f31 zero size 2024-04-12T05:37:14.658 INFO:tasks.workunit.client.1.smithi067.stdout:0/237: dread - dd/d40/f4c zero size 2024-04-12T05:37:14.664 INFO:tasks.workunit.client.1.smithi067.stdout:0/238: chown dd/d12 0 1 2024-04-12T05:37:14.664 INFO:tasks.workunit.client.1.smithi067.stdout:4/67: truncate d8/d9/ff 632175 0 2024-04-12T05:37:14.669 INFO:tasks.workunit.client.1.smithi067.stdout:7/158: dwrite d0/d2/ff [0,4194304] 0 2024-04-12T05:37:14.683 INFO:tasks.workunit.client.1.smithi067.stdout:7/159: fdatasync d0/d2/f21 0 2024-04-12T05:37:14.683 INFO:tasks.workunit.client.1.smithi067.stdout:7/160: chown d0/d2/d5 4190760 1 2024-04-12T05:37:14.683 INFO:tasks.workunit.client.1.smithi067.stdout:9/122: dread d1/f6 [0,4194304] 0 2024-04-12T05:37:14.683 INFO:tasks.workunit.client.1.smithi067.stdout:7/161: fsync d0/d2/d5/f29 0 2024-04-12T05:37:14.683 INFO:tasks.workunit.client.1.smithi067.stdout:9/123: chown d1/d12 2012 1 2024-04-12T05:37:14.684 INFO:tasks.workunit.client.1.smithi067.stdout:1/180: rename d4/f15 to d4/d9/d12/d28/f33 0 2024-04-12T05:37:14.685 INFO:tasks.workunit.client.1.smithi067.stdout:2/175: mkdir d5/da/d3d 0 2024-04-12T05:37:14.688 INFO:tasks.workunit.client.1.smithi067.stdout:3/112: getdents dc/d1d 0 2024-04-12T05:37:14.688 INFO:tasks.workunit.client.1.smithi067.stdout:1/181: fsync d4/d9/d12/d28/f2b 0 2024-04-12T05:37:14.695 INFO:tasks.workunit.client.1.smithi067.stdout:4/68: unlink d8/d9/l12 0 2024-04-12T05:37:14.695 INFO:tasks.workunit.client.1.smithi067.stdout:5/296: dwrite d0/f1c [0,4194304] 0 2024-04-12T05:37:14.697 INFO:tasks.workunit.client.1.smithi067.stdout:6/114: mknod dd/d11/c1c 0 2024-04-12T05:37:14.710 INFO:tasks.workunit.client.1.smithi067.stdout:7/162: mknod d0/d2/d5/db/c31 0 2024-04-12T05:37:14.710 INFO:tasks.workunit.client.1.smithi067.stdout:9/124: dwrite d1/d12/f14 [0,4194304] 0 2024-04-12T05:37:14.710 INFO:tasks.workunit.client.1.smithi067.stdout:8/125: rmdir d6/d8/d28 0 2024-04-12T05:37:14.710 INFO:tasks.workunit.client.1.smithi067.stdout:4/69: dwrite f7 [4194304,4194304] 0 2024-04-12T05:37:14.711 INFO:tasks.workunit.client.1.smithi067.stdout:3/113: write dc/df/f1e [2877892,107003] 0 2024-04-12T05:37:14.713 INFO:tasks.workunit.client.1.smithi067.stdout:1/182: mkdir d4/d9/d34 0 2024-04-12T05:37:14.713 INFO:tasks.workunit.client.1.smithi067.stdout:5/297: mknod d0/da/d27/c5e 0 2024-04-12T05:37:14.723 INFO:tasks.workunit.client.1.smithi067.stdout:8/126: creat d6/d8/d18/df/d12/f29 x:0 0 0 2024-04-12T05:37:14.723 INFO:tasks.workunit.client.1.smithi067.stdout:7/163: symlink d0/d2/d5/db/d10/l32 0 2024-04-12T05:37:14.723 INFO:tasks.workunit.client.1.smithi067.stdout:7/164: write d0/d23/f27 [2000107,101642] 0 2024-04-12T05:37:14.725 INFO:tasks.workunit.client.1.smithi067.stdout:3/114: creat dc/f23 x:0 0 0 2024-04-12T05:37:14.725 INFO:tasks.workunit.client.1.smithi067.stdout:0/239: rename fb to dd/d12/f4e 0 2024-04-12T05:37:14.730 INFO:tasks.workunit.client.1.smithi067.stdout:9/125: truncate d1/f2 3365104 0 2024-04-12T05:37:14.730 INFO:tasks.workunit.client.1.smithi067.stdout:8/127: mknod d6/d8/d18/df/d12/c2a 0 2024-04-12T05:37:14.731 INFO:tasks.workunit.client.1.smithi067.stdout:4/70: creat d8/da/f18 x:0 0 0 2024-04-12T05:37:14.735 INFO:tasks.workunit.client.1.smithi067.stdout:4/71: dread - d8/d9/f17 zero size 2024-04-12T05:37:14.735 INFO:tasks.workunit.client.1.smithi067.stdout:6/115: link f9 dd/f1d 0 2024-04-12T05:37:14.735 INFO:tasks.workunit.client.1.smithi067.stdout:7/165: rmdir d0/d2/d5/d1d 39 2024-04-12T05:37:14.737 INFO:tasks.workunit.client.1.smithi067.stdout:3/115: unlink dc/df/f21 0 2024-04-12T05:37:14.738 INFO:tasks.workunit.client.1.smithi067.stdout:5/298: symlink d0/d3a/d2c/l5f 0 2024-04-12T05:37:14.739 INFO:tasks.workunit.client.1.smithi067.stdout:0/240: dwrite dd/d40/f4c [0,4194304] 0 2024-04-12T05:37:14.742 INFO:tasks.workunit.client.1.smithi067.stdout:6/116: unlink f7 0 2024-04-12T05:37:14.743 INFO:tasks.workunit.client.1.smithi067.stdout:2/176: rename d5/f20 to d5/da/f3e 0 2024-04-12T05:37:14.743 INFO:tasks.workunit.client.1.smithi067.stdout:6/117: readlink dd/l1b 0 2024-04-12T05:37:14.746 INFO:tasks.workunit.client.1.smithi067.stdout:8/128: symlink d6/l2b 0 2024-04-12T05:37:14.749 INFO:tasks.workunit.client.1.smithi067.stdout:5/299: creat d0/da/d41/f60 x:0 0 0 2024-04-12T05:37:14.752 INFO:tasks.workunit.client.1.smithi067.stdout:2/177: dread d5/da/de/d14/d34/f37 [0,4194304] 0 2024-04-12T05:37:14.753 INFO:tasks.workunit.client.1.smithi067.stdout:3/116: rename dc/d1d/f1f to dc/d1d/f24 0 2024-04-12T05:37:14.754 INFO:tasks.workunit.client.1.smithi067.stdout:0/241: rmdir dd/d12/d14/d1a 39 2024-04-12T05:37:14.755 INFO:tasks.workunit.client.1.smithi067.stdout:6/118: creat dd/d11/f1e x:0 0 0 2024-04-12T05:37:14.755 INFO:tasks.workunit.client.1.smithi067.stdout:0/242: fsync dd/d12/f29 0 2024-04-12T05:37:14.758 INFO:tasks.workunit.client.1.smithi067.stdout:3/117: fdatasync dc/d1d/f24 0 2024-04-12T05:37:14.762 INFO:tasks.workunit.client.1.smithi067.stdout:0/243: creat dd/d40/f4f x:0 0 0 2024-04-12T05:37:14.762 INFO:tasks.workunit.client.1.smithi067.stdout:7/166: getdents d0/d2/d5/db/d10 0 2024-04-12T05:37:14.765 INFO:tasks.workunit.client.1.smithi067.stdout:1/183: rename d4/f6 to d4/d9/d12/f35 0 2024-04-12T05:37:14.766 INFO:tasks.workunit.client.1.smithi067.stdout:5/300: mknod d0/da/d27/c61 0 2024-04-12T05:37:14.772 INFO:tasks.workunit.client.1.smithi067.stdout:4/72: rename d8/da to d8/da/d19 22 2024-04-12T05:37:14.772 INFO:tasks.workunit.client.1.smithi067.stdout:6/119: creat dd/f1f x:0 0 0 2024-04-12T05:37:14.778 INFO:tasks.workunit.client.1.smithi067.stdout:2/178: dwrite d5/da/de/f31 [0,4194304] 0 2024-04-12T05:37:14.779 INFO:tasks.workunit.client.1.smithi067.stdout:0/244: creat dd/d12/d14/d1a/d3d/d41/f50 x:0 0 0 2024-04-12T05:37:14.785 INFO:tasks.workunit.client.1.smithi067.stdout:3/118: dwrite f1 [0,4194304] 0 2024-04-12T05:37:14.799 INFO:tasks.workunit.client.1.smithi067.stdout:0/245: dread dd/d12/d14/d1a/f2b [0,4194304] 0 2024-04-12T05:37:14.799 INFO:tasks.workunit.client.1.smithi067.stdout:3/119: dread - dc/f23 zero size 2024-04-12T05:37:14.801 INFO:tasks.workunit.client.1.smithi067.stdout:3/120: dread - dc/f23 zero size 2024-04-12T05:37:14.801 INFO:tasks.workunit.client.1.smithi067.stdout:3/121: chown dc/df/c1a 246 1 2024-04-12T05:37:14.804 INFO:tasks.workunit.client.1.smithi067.stdout:7/167: dwrite d0/d2/d5/d1e/f22 [4194304,4194304] 0 2024-04-12T05:37:14.808 INFO:tasks.workunit.client.1.smithi067.stdout:1/184: creat d4/d9/d12/f36 x:0 0 0 2024-04-12T05:37:14.808 INFO:tasks.workunit.client.1.smithi067.stdout:4/73: unlink d8/d9/ff 0 2024-04-12T05:37:14.808 INFO:tasks.workunit.client.1.smithi067.stdout:8/129: rename d6/d8/d18/df/d12/f1e to d6/f2c 0 2024-04-12T05:37:14.811 INFO:tasks.workunit.client.1.smithi067.stdout:6/120: creat dd/f20 x:0 0 0 2024-04-12T05:37:14.813 INFO:tasks.workunit.client.1.smithi067.stdout:0/246: mkdir dd/d12/d14/d51 0 2024-04-12T05:37:14.816 INFO:tasks.workunit.client.1.smithi067.stdout:7/168: write d0/d2/d5/d1e/f20 [23551,93553] 0 2024-04-12T05:37:14.820 INFO:tasks.workunit.client.1.smithi067.stdout:3/122: rename dc/l19 to dc/df/l25 0 2024-04-12T05:37:14.821 INFO:tasks.workunit.client.1.smithi067.stdout:6/121: mkdir dd/d21 0 2024-04-12T05:37:14.821 INFO:tasks.workunit.client.1.smithi067.stdout:5/301: write d0/da/d27/f15 [919984,68816] 0 2024-04-12T05:37:14.826 INFO:tasks.workunit.client.1.smithi067.stdout:3/123: mkdir dc/d1d/d26 0 2024-04-12T05:37:14.829 INFO:tasks.workunit.client.1.smithi067.stdout:8/130: truncate d6/d8/d18/df/f13 1712764 0 2024-04-12T05:37:14.829 INFO:tasks.workunit.client.1.smithi067.stdout:1/185: dwrite d4/d9/d12/f36 [0,4194304] 0 2024-04-12T05:37:14.831 INFO:tasks.workunit.client.1.smithi067.stdout:4/74: rename d8/d9/fc to d8/d14/d15/f1a 0 2024-04-12T05:37:14.832 INFO:tasks.workunit.client.1.smithi067.stdout:7/169: write d0/d2/d5/db/d10/f26 [583088,90961] 0 2024-04-12T05:37:14.834 INFO:tasks.workunit.client.1.smithi067.stdout:4/75: chown f7 1 1 2024-04-12T05:37:14.834 INFO:tasks.workunit.client.1.smithi067.stdout:3/124: mkdir dc/df/d27 0 2024-04-12T05:37:14.834 INFO:tasks.workunit.client.0.smithi032.stdout:3/705: mkdir d12/d2f/de2 0 2024-04-12T05:37:14.836 INFO:tasks.workunit.client.1.smithi067.stdout:0/247: rename dd/d12/d14/d1a/c28 to dd/d12/d14/d51/c52 0 2024-04-12T05:37:14.837 INFO:tasks.workunit.client.1.smithi067.stdout:1/186: creat d4/d9/d12/d2c/f37 x:0 0 0 2024-04-12T05:37:14.842 INFO:tasks.workunit.client.1.smithi067.stdout:3/125: mkdir dc/d1d/d28 0 2024-04-12T05:37:14.843 INFO:tasks.workunit.client.1.smithi067.stdout:4/76: unlink d8/d14/l16 0 2024-04-12T05:37:14.843 INFO:tasks.workunit.client.1.smithi067.stdout:7/170: dwrite d0/f8 [0,4194304] 0 2024-04-12T05:37:14.844 INFO:tasks.workunit.client.1.smithi067.stdout:4/77: read f7 [7512973,41838] 0 2024-04-12T05:37:14.844 INFO:tasks.workunit.client.1.smithi067.stdout:4/78: fdatasync f6 0 2024-04-12T05:37:14.848 INFO:tasks.workunit.client.1.smithi067.stdout:5/302: rename d0/da/d27 to d0/d6/d20/d62 0 2024-04-12T05:37:14.855 INFO:tasks.workunit.client.1.smithi067.stdout:6/122: truncate dd/d11/f14 2390559 0 2024-04-12T05:37:14.855 INFO:tasks.workunit.client.1.smithi067.stdout:7/171: dread - d0/d2/d5/db/f1f zero size 2024-04-12T05:37:14.856 INFO:tasks.workunit.client.0.smithi032.stdout:3/706: getdents d12/d2e/d3a/d2d 0 2024-04-12T05:37:14.858 INFO:tasks.workunit.client.1.smithi067.stdout:6/123: dread - dd/d11/f1e zero size 2024-04-12T05:37:14.858 INFO:tasks.workunit.client.1.smithi067.stdout:3/126: symlink dc/d1d/d28/l29 0 2024-04-12T05:37:14.860 INFO:tasks.workunit.client.1.smithi067.stdout:8/131: rename d6/d8/c17 to d6/d8/d18/df/d12/c2d 0 2024-04-12T05:37:14.861 INFO:tasks.workunit.client.0.smithi032.stdout:3/707: creat d12/d2e/d3a/dbe/fe3 x:0 0 0 2024-04-12T05:37:14.865 INFO:tasks.workunit.client.1.smithi067.stdout:7/172: dread d0/f8 [0,4194304] 0 2024-04-12T05:37:14.866 INFO:tasks.workunit.client.1.smithi067.stdout:0/248: rename dd/d12/f37 to dd/d12/d14/d49/f53 0 2024-04-12T05:37:14.867 INFO:tasks.workunit.client.0.smithi032.stdout:3/708: creat d12/d2f/de2/fe4 x:0 0 0 2024-04-12T05:37:14.870 INFO:tasks.workunit.client.1.smithi067.stdout:8/132: dwrite d6/d8/d18/df/d12/f1f [0,4194304] 0 2024-04-12T05:37:14.870 INFO:tasks.workunit.client.0.smithi032.stdout:3/709: creat d12/d2f/de2/fe5 x:0 0 0 2024-04-12T05:37:14.873 INFO:tasks.workunit.client.1.smithi067.stdout:4/79: truncate f7 7181353 0 2024-04-12T05:37:14.873 INFO:tasks.workunit.client.1.smithi067.stdout:4/80: fdatasync f6 0 2024-04-12T05:37:14.874 INFO:tasks.workunit.client.1.smithi067.stdout:0/249: dwrite dd/f18 [0,4194304] 0 2024-04-12T05:37:14.876 INFO:tasks.workunit.client.1.smithi067.stdout:4/81: chown d8/cd 46 1 2024-04-12T05:37:14.876 INFO:tasks.workunit.client.1.smithi067.stdout:6/124: creat dd/f22 x:0 0 0 2024-04-12T05:37:14.943 INFO:tasks.workunit.client.1.smithi067.stdout:0/250: symlink dd/l54 0 2024-04-12T05:37:14.944 INFO:tasks.workunit.client.1.smithi067.stdout:6/125: read f0 [5131514,16340] 0 2024-04-12T05:37:14.947 INFO:tasks.workunit.client.1.smithi067.stdout:0/251: fdatasync dd/d12/d14/d1a/d3d/f32 0 2024-04-12T05:37:14.950 INFO:tasks.workunit.client.1.smithi067.stdout:6/126: getdents dd 0 2024-04-12T05:37:14.951 INFO:tasks.workunit.client.1.smithi067.stdout:0/252: mknod dd/d12/d14/d51/c55 0 2024-04-12T05:37:14.954 INFO:tasks.workunit.client.1.smithi067.stdout:3/127: dwrite dc/f18 [0,4194304] 0 2024-04-12T05:37:14.954 INFO:tasks.workunit.client.1.smithi067.stdout:4/82: dread d8/fe [0,4194304] 0 2024-04-12T05:37:14.956 INFO:tasks.workunit.client.1.smithi067.stdout:0/253: symlink dd/d12/l56 0 2024-04-12T05:37:14.957 INFO:tasks.workunit.client.1.smithi067.stdout:3/128: rmdir dc/d1d 39 2024-04-12T05:37:14.959 INFO:tasks.workunit.client.1.smithi067.stdout:3/129: readlink dc/df/l20 0 2024-04-12T05:37:14.960 INFO:tasks.workunit.client.0.smithi032.stdout:6/961: creat d4/d21/d2e/f14e x:0 0 0 2024-04-12T05:37:14.963 INFO:tasks.workunit.client.1.smithi067.stdout:6/127: dwrite dd/f1f [0,4194304] 0 2024-04-12T05:37:14.963 INFO:tasks.workunit.client.0.smithi032.stdout:3/710: creat d12/fe6 x:0 0 0 2024-04-12T05:37:14.965 INFO:tasks.workunit.client.1.smithi067.stdout:3/130: creat dc/d1d/d28/f2a x:0 0 0 2024-04-12T05:37:14.968 INFO:tasks.workunit.client.1.smithi067.stdout:6/128: mkdir dd/d21/d23 0 2024-04-12T05:37:14.968 INFO:tasks.workunit.client.0.smithi032.stdout:6/962: link d4/de0/de7/d37/l7b d4/d21/d27/dad/d55/d59/l14f 0 2024-04-12T05:37:14.974 INFO:tasks.workunit.client.1.smithi067.stdout:6/129: dread dd/f1f [0,4194304] 0 2024-04-12T05:37:14.974 INFO:tasks.workunit.client.0.smithi032.stdout:6/963: creat d4/de0/de7/d79/dc6/f150 x:0 0 0 2024-04-12T05:37:14.975 INFO:tasks.workunit.client.0.smithi032.stdout:6/964: stat d4/de0/de7/d79/dc6 0 2024-04-12T05:37:14.975 INFO:tasks.workunit.client.0.smithi032.stdout:3/711: rmdir d12/daa/db3 39 2024-04-12T05:37:14.977 INFO:tasks.workunit.client.0.smithi032.stdout:6/965: mknod d4/de0/de7/c151 0 2024-04-12T05:37:14.977 INFO:tasks.workunit.client.1.smithi067.stdout:6/130: creat dd/d21/f24 x:0 0 0 2024-04-12T05:37:14.980 INFO:tasks.workunit.client.1.smithi067.stdout:3/131: getdents dc/d1d/d28 0 2024-04-12T05:37:14.982 INFO:tasks.workunit.client.0.smithi032.stdout:3/712: rename d12/d2e/d3a/d2d/fbd to d12/daa/db3/fe7 0 2024-04-12T05:37:14.984 INFO:tasks.workunit.client.1.smithi067.stdout:3/132: rename lb to dc/df/d27/l2b 0 2024-04-12T05:37:14.986 INFO:tasks.workunit.client.1.smithi067.stdout:6/131: dwrite dd/d11/f19 [0,4194304] 0 2024-04-12T05:37:14.987 INFO:tasks.workunit.client.1.smithi067.stdout:6/132: chown dd/f1f 19 1 2024-04-12T05:37:14.987 INFO:tasks.workunit.client.0.smithi032.stdout:3/713: rename d12/d2e/d3a/fa9 to d12/d2e/d3a/d63/da6/dac/dc9/fe8 0 2024-04-12T05:37:14.987 INFO:tasks.workunit.client.1.smithi067.stdout:6/133: read dd/fe [7711171,107055] 0 2024-04-12T05:37:14.988 INFO:tasks.workunit.client.1.smithi067.stdout:3/133: creat dc/df/f2c x:0 0 0 2024-04-12T05:37:14.989 INFO:tasks.workunit.client.1.smithi067.stdout:3/134: chown dc/f18 1015422901 1 2024-04-12T05:37:14.990 INFO:tasks.workunit.client.1.smithi067.stdout:3/135: write dc/f23 [174211,41162] 0 2024-04-12T05:37:14.995 INFO:tasks.workunit.client.0.smithi032.stdout:6/966: write d4/d14/f6d [2337881,88915] 0 2024-04-12T05:37:14.996 INFO:tasks.workunit.client.1.smithi067.stdout:6/134: getdents dd/d11 0 2024-04-12T05:37:14.998 INFO:tasks.workunit.client.0.smithi032.stdout:6/967: mkdir d4/d14/d25/d152 0 2024-04-12T05:37:14.999 INFO:tasks.workunit.client.1.smithi067.stdout:3/136: fsync dc/df/f2c 0 2024-04-12T05:37:15.001 INFO:tasks.workunit.client.0.smithi032.stdout:6/968: symlink d4/de0/de7/d88/d8d/l153 0 2024-04-12T05:37:15.003 INFO:tasks.workunit.client.1.smithi067.stdout:3/137: creat dc/df/d27/f2d x:0 0 0 2024-04-12T05:37:15.005 INFO:tasks.workunit.client.0.smithi032.stdout:3/714: dread d12/d2e/d3a/f31 [0,4194304] 0 2024-04-12T05:37:15.008 INFO:tasks.workunit.client.1.smithi067.stdout:6/135: dwrite f6 [4194304,4194304] 0 2024-04-12T05:37:15.009 INFO:tasks.workunit.client.0.smithi032.stdout:3/715: link d12/d2e/c84 d12/d2e/dcd/ce9 0 2024-04-12T05:37:15.009 INFO:tasks.workunit.client.1.smithi067.stdout:3/138: link dc/d1d/d28/l29 dc/df/d27/l2e 0 2024-04-12T05:37:15.011 INFO:tasks.workunit.client.0.smithi032.stdout:6/969: dwrite d4/d21/db4/fcd [0,4194304] 0 2024-04-12T05:37:15.011 INFO:tasks.workunit.client.1.smithi067.stdout:6/136: creat dd/d21/d23/f25 x:0 0 0 2024-04-12T05:37:15.014 INFO:tasks.workunit.client.1.smithi067.stdout:3/139: truncate f9 4193346 0 2024-04-12T05:37:15.015 INFO:tasks.workunit.client.1.smithi067.stdout:6/137: fsync dd/fe 0 2024-04-12T05:37:15.016 INFO:tasks.workunit.client.1.smithi067.stdout:6/138: chown f0 3422 1 2024-04-12T05:37:15.018 INFO:tasks.workunit.client.1.smithi067.stdout:6/139: dread - dd/d11/f1e zero size 2024-04-12T05:37:15.018 INFO:tasks.workunit.client.0.smithi032.stdout:3/716: link d12/d2e/f42 d12/d2f/de2/fea 0 2024-04-12T05:37:15.019 INFO:tasks.workunit.client.1.smithi067.stdout:6/140: dread - dd/d21/d23/f25 zero size 2024-04-12T05:37:15.021 INFO:tasks.workunit.client.1.smithi067.stdout:6/141: rename dd/d21 to dd/d21/d26 22 2024-04-12T05:37:15.022 INFO:tasks.workunit.client.0.smithi032.stdout:3/717: rmdir d12/d2f/d89/d9d 39 2024-04-12T05:37:15.022 INFO:tasks.workunit.client.1.smithi067.stdout:6/142: stat dd/d11/c12 0 2024-04-12T05:37:15.026 INFO:tasks.workunit.client.1.smithi067.stdout:3/140: dwrite dc/d1d/f24 [0,4194304] 0 2024-04-12T05:37:15.027 INFO:tasks.workunit.client.1.smithi067.stdout:6/143: unlink dd/f1f 0 2024-04-12T05:37:15.030 INFO:tasks.workunit.client.1.smithi067.stdout:6/144: mkdir dd/d11/d27 0 2024-04-12T05:37:15.030 INFO:tasks.workunit.client.0.smithi032.stdout:3/718: dwrite d12/d2f/fdc [0,4194304] 0 2024-04-12T05:37:15.031 INFO:tasks.workunit.client.1.smithi067.stdout:3/141: creat dc/d1d/d26/f2f x:0 0 0 2024-04-12T05:37:15.033 INFO:tasks.workunit.client.1.smithi067.stdout:6/145: symlink dd/d21/d23/l28 0 2024-04-12T05:37:15.039 INFO:tasks.workunit.client.1.smithi067.stdout:3/142: chown c8 8551 1 2024-04-12T05:37:15.042 INFO:tasks.workunit.client.1.smithi067.stdout:3/143: creat dc/df/f30 x:0 0 0 2024-04-12T05:37:15.042 INFO:tasks.workunit.client.1.smithi067.stdout:6/146: dread dd/f1d [0,4194304] 0 2024-04-12T05:37:15.043 INFO:tasks.workunit.client.1.smithi067.stdout:3/144: chown dc/d1d/d28 1 1 2024-04-12T05:37:15.043 INFO:tasks.workunit.client.0.smithi032.stdout:6/970: dread d4/d21/f85 [0,4194304] 0 2024-04-12T05:37:15.045 INFO:tasks.workunit.client.0.smithi032.stdout:6/971: chown d4/d21/d2e/d32/d94/f132 1734068871 1 2024-04-12T05:37:15.046 INFO:tasks.workunit.client.1.smithi067.stdout:6/147: mknod dd/c29 0 2024-04-12T05:37:15.046 INFO:tasks.workunit.client.0.smithi032.stdout:3/719: dread d12/f21 [0,4194304] 0 2024-04-12T05:37:15.047 INFO:tasks.workunit.client.1.smithi067.stdout:6/148: write dd/f22 [964041,36168] 0 2024-04-12T05:37:15.048 INFO:tasks.workunit.client.1.smithi067.stdout:3/145: creat dc/d1d/f31 x:0 0 0 2024-04-12T05:37:15.050 INFO:tasks.workunit.client.0.smithi032.stdout:6/972: symlink d4/de0/de7/d92/d9e/df6/d113/d118/l154 0 2024-04-12T05:37:15.050 INFO:tasks.workunit.client.1.smithi067.stdout:6/149: symlink dd/l2a 0 2024-04-12T05:37:15.051 INFO:tasks.workunit.client.1.smithi067.stdout:6/150: rename dd/d11/d27 to dd/d11/d27/d2b 22 2024-04-12T05:37:15.052 INFO:tasks.workunit.client.0.smithi032.stdout:3/720: creat d12/d2f/d89/feb x:0 0 0 2024-04-12T05:37:15.053 INFO:tasks.workunit.client.1.smithi067.stdout:3/146: symlink dc/d1d/d26/l32 0 2024-04-12T05:37:15.055 INFO:tasks.workunit.client.0.smithi032.stdout:6/973: link d4/de0/de7/f49 d4/d21/d2e/f155 0 2024-04-12T05:37:15.056 INFO:tasks.workunit.client.1.smithi067.stdout:6/151: dread dd/d11/f14 [0,4194304] 0 2024-04-12T05:37:15.056 INFO:tasks.workunit.client.0.smithi032.stdout:3/721: rename d12/d2e/dcb/fdb to d12/d24/dde/fec 0 2024-04-12T05:37:15.056 INFO:tasks.workunit.client.0.smithi032.stdout:3/722: chown d12/d2f/d89/fba 29077692 1 2024-04-12T05:37:15.057 INFO:tasks.workunit.client.1.smithi067.stdout:3/147: truncate f9 4077364 0 2024-04-12T05:37:15.058 INFO:tasks.workunit.client.0.smithi032.stdout:6/974: write d4/d14/d3c/f41 [1383755,120208] 0 2024-04-12T05:37:15.058 INFO:tasks.workunit.client.1.smithi067.stdout:6/152: unlink dd/d11/c12 0 2024-04-12T05:37:15.060 INFO:tasks.workunit.client.0.smithi032.stdout:3/723: rename d12/d2f/d89/fd2 to d12/d2e/dcb/fed 0 2024-04-12T05:37:15.062 INFO:tasks.workunit.client.1.smithi067.stdout:6/153: symlink dd/d21/d23/l2c 0 2024-04-12T05:37:15.066 INFO:tasks.workunit.client.0.smithi032.stdout:3/724: rmdir d12/d2e/d3a/d63/da6/dac/dc9 39 2024-04-12T05:37:15.067 INFO:tasks.workunit.client.0.smithi032.stdout:6/975: dwrite d4/d21/d2e/d32/d94/f132 [0,4194304] 0 2024-04-12T05:37:15.067 INFO:tasks.workunit.client.0.smithi032.stdout:3/725: fdatasync d12/d2e/d3a/d63/f8d 0 2024-04-12T05:37:15.068 INFO:tasks.workunit.client.1.smithi067.stdout:3/148: dwrite dc/d1d/d26/f2f [0,4194304] 0 2024-04-12T05:37:15.070 INFO:tasks.workunit.client.1.smithi067.stdout:3/149: stat f1 0 2024-04-12T05:37:15.072 INFO:tasks.workunit.client.1.smithi067.stdout:6/154: dread dd/f17 [0,4194304] 0 2024-04-12T05:37:15.074 INFO:tasks.workunit.client.0.smithi032.stdout:3/726: rename d12/d2f/f3d to d12/d2f/d89/d9d/fee 0 2024-04-12T05:37:15.075 INFO:tasks.workunit.client.0.smithi032.stdout:6/976: getdents d4/de0/de7/d92/d9e/df6/d13b 0 2024-04-12T05:37:15.076 INFO:tasks.workunit.client.0.smithi032.stdout:6/977: stat d4/de0/de7/d88/c8c 0 2024-04-12T05:37:15.077 INFO:tasks.workunit.client.0.smithi032.stdout:6/978: stat d4/d21/d27/dad/d55/d99/l102 0 2024-04-12T05:37:15.079 INFO:tasks.workunit.client.0.smithi032.stdout:3/727: chown d12/d2e/d3a/d63/f8a 3524 1 2024-04-12T05:37:15.079 INFO:tasks.workunit.client.0.smithi032.stdout:6/979: mknod d4/d21/d2e/d32/c156 0 2024-04-12T05:37:15.082 INFO:tasks.workunit.client.1.smithi067.stdout:6/155: mkdir dd/d11/d2d 0 2024-04-12T05:37:15.082 INFO:tasks.workunit.client.1.smithi067.stdout:6/156: fsync f0 0 2024-04-12T05:37:15.083 INFO:tasks.workunit.client.0.smithi032.stdout:3/728: creat d12/fef x:0 0 0 2024-04-12T05:37:15.084 INFO:tasks.workunit.client.1.smithi067.stdout:6/157: truncate f9 72346 0 2024-04-12T05:37:15.085 INFO:tasks.workunit.client.1.smithi067.stdout:6/158: chown dd/d11/l18 7121029 1 2024-04-12T05:37:15.086 INFO:tasks.workunit.client.0.smithi032.stdout:6/980: dread d4/d21/d2e/f13d [0,4194304] 0 2024-04-12T05:37:15.089 INFO:tasks.workunit.client.0.smithi032.stdout:6/981: mkdir d4/de0/de7/d37/d69/d76/d157 0 2024-04-12T05:37:15.089 INFO:tasks.workunit.client.1.smithi067.stdout:6/159: dread dd/d11/f19 [0,4194304] 0 2024-04-12T05:37:15.089 INFO:tasks.workunit.client.0.smithi032.stdout:6/982: dread - d4/de0/de7/d88/f10b zero size 2024-04-12T05:37:15.090 INFO:tasks.workunit.client.1.smithi067.stdout:6/160: chown dd/d11/d2d 0 1 2024-04-12T05:37:15.091 INFO:tasks.workunit.client.0.smithi032.stdout:6/983: write d4/d21/f2f [15102,128307] 0 2024-04-12T05:37:15.091 INFO:tasks.workunit.client.1.smithi067.stdout:6/161: write fa [129146,94294] 0 2024-04-12T05:37:15.093 INFO:tasks.workunit.client.0.smithi032.stdout:3/729: rename d12/d2e to d12/d2f/d89/df0 0 2024-04-12T05:37:15.098 INFO:tasks.workunit.client.1.smithi067.stdout:6/162: creat dd/d11/d27/f2e x:0 0 0 2024-04-12T05:37:15.101 INFO:tasks.workunit.client.0.smithi032.stdout:3/730: truncate d12/f70 88842 0 2024-04-12T05:37:15.102 INFO:tasks.workunit.client.0.smithi032.stdout:3/731: stat d12/f21 0 2024-04-12T05:37:15.102 INFO:tasks.workunit.client.1.smithi067.stdout:6/163: creat dd/d11/d27/f2f x:0 0 0 2024-04-12T05:37:15.103 INFO:tasks.workunit.client.0.smithi032.stdout:6/984: dwrite d4/d74/fc2 [0,4194304] 0 2024-04-12T05:37:15.104 INFO:tasks.workunit.client.0.smithi032.stdout:3/732: write d12/d2f/fdd [219322,66589] 0 2024-04-12T05:37:15.106 INFO:tasks.workunit.client.1.smithi067.stdout:6/164: truncate dd/fe 4014321 0 2024-04-12T05:37:15.107 INFO:tasks.workunit.client.0.smithi032.stdout:6/985: symlink d4/d21/d27/dad/d55/df4/l158 0 2024-04-12T05:37:15.108 INFO:tasks.workunit.client.0.smithi032.stdout:3/733: symlink d12/d2f/d89/lf1 0 2024-04-12T05:37:15.112 INFO:tasks.workunit.client.1.smithi067.stdout:6/165: rmdir dd/d11/d2d 0 2024-04-12T05:37:15.113 INFO:tasks.workunit.client.0.smithi032.stdout:3/734: creat d12/d2f/d89/df0/d3a/d2d/ff2 x:0 0 0 2024-04-12T05:37:15.113 INFO:tasks.workunit.client.0.smithi032.stdout:6/986: creat d4/de0/de7/f159 x:0 0 0 2024-04-12T05:37:15.116 INFO:tasks.workunit.client.0.smithi032.stdout:6/987: fsync d4/d21/d2e/d32/f7a 0 2024-04-12T05:37:15.117 INFO:tasks.workunit.client.0.smithi032.stdout:3/735: creat d12/d2f/d89/df0/d3a/d63/da6/dac/dc9/ff3 x:0 0 0 2024-04-12T05:37:15.118 INFO:tasks.workunit.client.0.smithi032.stdout:3/736: chown d12/d2f/d89/df0/f4a 193372434 1 2024-04-12T05:37:15.118 INFO:tasks.workunit.client.1.smithi067.stdout:6/166: dwrite fa [0,4194304] 0 2024-04-12T05:37:15.121 INFO:tasks.workunit.client.0.smithi032.stdout:3/737: mkdir d12/d2f/d89/df0/d3a/d2d/df4 0 2024-04-12T05:37:15.123 INFO:tasks.workunit.client.0.smithi032.stdout:6/988: dwrite d4/d14/d3c/fb3 [0,4194304] 0 2024-04-12T05:37:15.126 INFO:tasks.workunit.client.0.smithi032.stdout:6/989: readlink d4/de0/de7/dd9/l14b 0 2024-04-12T05:37:15.128 INFO:tasks.workunit.client.0.smithi032.stdout:6/990: creat d4/de0/de7/d79/dc6/d101/f15a x:0 0 0 2024-04-12T05:37:15.130 INFO:tasks.workunit.client.0.smithi032.stdout:6/991: chown d4/de0/de7/d79/fd3 0 1 2024-04-12T05:37:15.133 INFO:tasks.workunit.client.0.smithi032.stdout:6/992: creat d4/d14/d3c/f15b x:0 0 0 2024-04-12T05:37:15.135 INFO:tasks.workunit.client.0.smithi032.stdout:6/993: fsync d4/d21/d27/dad/d55/f70 0 2024-04-12T05:37:15.137 INFO:tasks.workunit.client.1.smithi067.stdout:6/167: truncate fb 417758 0 2024-04-12T05:37:15.137 INFO:tasks.workunit.client.0.smithi032.stdout:3/738: write d12/d2f/d89/df0/f94 [590796,126055] 0 2024-04-12T05:37:15.138 INFO:tasks.workunit.client.0.smithi032.stdout:6/994: symlink d4/l15c 0 2024-04-12T05:37:15.139 INFO:tasks.workunit.client.1.smithi067.stdout:6/168: mkdir dd/d30 0 2024-04-12T05:37:15.140 INFO:tasks.workunit.client.1.smithi067.stdout:6/169: fsync dd/f22 0 2024-04-12T05:37:15.141 INFO:tasks.workunit.client.0.smithi032.stdout:3/739: mkdir d12/d2f/d89/df0/d3a/d63/da6/df5 0 2024-04-12T05:37:15.142 INFO:tasks.workunit.client.0.smithi032.stdout:3/740: fdatasync d12/d2f/d89/df0/f82 0 2024-04-12T05:37:15.142 INFO:tasks.workunit.client.0.smithi032.stdout:6/995: rename d4/de0/de7/d37/d47 to d4/d14/d25/d15d 0 2024-04-12T05:37:15.146 INFO:tasks.workunit.client.0.smithi032.stdout:6/996: unlink d4/d74/fc2 0 2024-04-12T05:37:15.147 INFO:tasks.workunit.client.1.smithi067.stdout:6/170: dwrite fc [0,4194304] 0 2024-04-12T05:37:15.149 INFO:tasks.workunit.client.1.smithi067.stdout:6/171: mknod dd/d11/c31 0 2024-04-12T05:37:15.153 INFO:tasks.workunit.client.1.smithi067.stdout:6/172: rmdir dd/d21 39 2024-04-12T05:37:15.158 INFO:tasks.workunit.client.1.smithi067.stdout:6/173: getdents dd/d11/d27 0 2024-04-12T05:37:15.158 INFO:tasks.workunit.client.0.smithi032.stdout:6/997: dread d4/de0/de7/d37/f5d [0,4194304] 0 2024-04-12T05:37:15.162 INFO:tasks.workunit.client.0.smithi032.stdout:6/998: symlink d4/de0/de7/d79/dc6/d101/l15e 0 2024-04-12T05:37:15.163 INFO:tasks.workunit.client.0.smithi032.stdout:3/741: dwrite d12/d2f/d89/df0/d3a/d63/fb2 [0,4194304] 0 2024-04-12T05:37:15.165 INFO:tasks.workunit.client.0.smithi032.stdout:3/742: chown d12/d2f/d89/df0/d3a/d2d/df4 58241 1 2024-04-12T05:37:15.165 INFO:tasks.workunit.client.0.smithi032.stdout:6/999: creat d4/d21/d27/f15f x:0 0 0 2024-04-12T05:37:15.166 INFO:tasks.workunit.client.1.smithi067.stdout:6/174: dwrite dd/f17 [0,4194304] 0 2024-04-12T05:37:15.166 INFO:tasks.workunit.client.0.smithi032.stdout:3/743: read d12/daa/db3/f66 [36505,1243] 0 2024-04-12T05:37:15.168 INFO:tasks.workunit.client.0.smithi032.stdout:3/744: fsync d12/d2f/d89/df0/d3a/fae 0 2024-04-12T05:37:15.171 INFO:tasks.workunit.client.0.smithi032.stdout:3/745: creat d12/daa/ff6 x:0 0 0 2024-04-12T05:37:15.171 INFO:tasks.workunit.client.0.smithi032.stdout:3/746: readlink d12/d24/l5c 0 2024-04-12T05:37:15.184 INFO:tasks.workunit.client.0.smithi032.stdout:3/747: dwrite d12/d2f/fc4 [0,4194304] 0 2024-04-12T05:37:15.189 INFO:tasks.workunit.client.0.smithi032.stdout:3/748: getdents d12/d2f/d89/df0/d3a/d63/da6 0 2024-04-12T05:37:15.190 INFO:tasks.workunit.client.0.smithi032.stdout:3/749: write d12/daa/db3/fcc [893427,33344] 0 2024-04-12T05:37:15.192 INFO:tasks.workunit.client.0.smithi032.stdout:3/750: mknod d12/d2f/cf7 0 2024-04-12T05:37:15.194 INFO:tasks.workunit.client.0.smithi032.stdout:3/751: symlink d12/d24/lf8 0 2024-04-12T05:37:15.196 INFO:tasks.workunit.client.0.smithi032.stdout:3/752: mknod d12/d2f/de2/cf9 0 2024-04-12T05:37:15.200 INFO:tasks.workunit.client.0.smithi032.stdout:3/753: getdents d12/d2f/d89/d9d 0 2024-04-12T05:37:15.204 INFO:tasks.workunit.client.0.smithi032.stdout:3/754: dread d12/f56 [0,4194304] 0 2024-04-12T05:37:15.205 INFO:tasks.workunit.client.0.smithi032.stdout:3/755: symlink d12/d2f/lfa 0 2024-04-12T05:37:15.212 INFO:tasks.workunit.client.0.smithi032.stdout:3/756: dwrite d12/d2f/d89/df0/d3a/f31 [0,4194304] 0 2024-04-12T05:37:15.221 INFO:tasks.workunit.client.0.smithi032.stdout:3/757: dwrite d12/f2a [0,4194304] 0 2024-04-12T05:37:15.222 INFO:tasks.workunit.client.0.smithi032.stdout:3/758: write d12/d2f/d89/fba [171028,10625] 0 2024-04-12T05:37:15.225 INFO:tasks.workunit.client.0.smithi032.stdout:3/759: rmdir d12/d24/dbb 39 2024-04-12T05:37:15.228 INFO:tasks.workunit.client.0.smithi032.stdout:3/760: creat d12/d2f/d89/df0/dcb/ffb x:0 0 0 2024-04-12T05:37:15.230 INFO:tasks.workunit.client.0.smithi032.stdout:3/761: rename d12/d2f/d89 to d12/d2f/d89/df0/d3a/dfc 22 2024-04-12T05:37:15.233 INFO:tasks.workunit.client.0.smithi032.stdout:3/762: mknod d12/d2f/d89/d9d/cfd 0 2024-04-12T05:37:15.234 INFO:tasks.workunit.client.0.smithi032.stdout:3/763: fdatasync d12/d2f/f91 0 2024-04-12T05:37:15.242 INFO:tasks.workunit.client.0.smithi032.stdout:3/764: dwrite d12/d2f/d89/df0/d3a/dbe/fe3 [0,4194304] 0 2024-04-12T05:37:15.243 INFO:tasks.workunit.client.0.smithi032.stdout:3/765: stat d12/c22 0 2024-04-12T05:37:15.248 INFO:tasks.workunit.client.0.smithi032.stdout:3/766: getdents d12/d2f/d89/df0 0 2024-04-12T05:37:15.254 INFO:tasks.workunit.client.0.smithi032.stdout:3/767: dwrite d12/d24/fa4 [0,4194304] 0 2024-04-12T05:37:15.256 INFO:tasks.workunit.client.0.smithi032.stdout:3/768: fdatasync d12/d2f/d89/df0/d3a/d2d/ff2 0 2024-04-12T05:37:15.259 INFO:tasks.workunit.client.0.smithi032.stdout:3/769: creat d12/d2f/d89/df0/dcb/ffe x:0 0 0 2024-04-12T05:37:15.262 INFO:tasks.workunit.client.0.smithi032.stdout:3/770: creat d12/d2f/d89/fff x:0 0 0 2024-04-12T05:37:15.271 INFO:tasks.workunit.client.0.smithi032.stdout:5/794: sync 2024-04-12T05:37:15.272 INFO:tasks.workunit.client.0.smithi032.stdout:5/795: write d2/df/d25/f98 [7070518,101634] 0 2024-04-12T05:37:15.273 INFO:tasks.workunit.client.0.smithi032.stdout:3/771: dread f10 [0,4194304] 0 2024-04-12T05:37:15.274 INFO:tasks.workunit.client.0.smithi032.stdout:3/772: truncate d12/d2f/fd1 792343 0 2024-04-12T05:37:15.275 INFO:tasks.workunit.client.0.smithi032.stdout:5/796: mknod d2/d3c/d5b/cfc 0 2024-04-12T05:37:15.276 INFO:tasks.workunit.client.0.smithi032.stdout:3/773: fdatasync d12/d24/dbb/fc2 0 2024-04-12T05:37:15.279 INFO:tasks.workunit.client.0.smithi032.stdout:3/774: dread d12/d2f/fd1 [0,4194304] 0 2024-04-12T05:37:15.280 INFO:tasks.workunit.client.0.smithi032.stdout:3/775: chown d12/d24/c54 0 1 2024-04-12T05:37:15.282 INFO:tasks.workunit.client.0.smithi032.stdout:5/797: write d2/d3c/d5b/f39 [3834500,24719] 0 2024-04-12T05:37:15.282 INFO:tasks.workunit.client.0.smithi032.stdout:3/776: truncate d12/d2f/d89/df0/f36 151998 0 2024-04-12T05:37:15.283 INFO:tasks.workunit.client.0.smithi032.stdout:3/777: chown d12/d2f/d89/df0/dcd 39 1 2024-04-12T05:37:15.285 INFO:tasks.workunit.client.0.smithi032.stdout:5/798: creat d2/df/d31/da2/dc0/ffd x:0 0 0 2024-04-12T05:37:15.286 INFO:tasks.workunit.client.0.smithi032.stdout:5/799: write d2/df/d31/d92/fc3 [4564734,89358] 0 2024-04-12T05:37:15.286 INFO:tasks.workunit.client.0.smithi032.stdout:3/778: symlink d12/d2f/d89/df0/d3a/d63/l100 0 2024-04-12T05:37:15.288 INFO:tasks.workunit.client.0.smithi032.stdout:3/779: chown d12/d2f/d89/df0/f67 442684 1 2024-04-12T05:37:15.291 INFO:tasks.workunit.client.0.smithi032.stdout:5/800: dread d2/d3c/d48/d4b/fbb [0,4194304] 0 2024-04-12T05:37:15.294 INFO:tasks.workunit.client.0.smithi032.stdout:5/801: mknod d2/d3c/d48/cfe 0 2024-04-12T05:37:15.299 INFO:tasks.workunit.client.0.smithi032.stdout:5/802: dwrite d2/d9b/fb7 [0,4194304] 0 2024-04-12T05:37:15.300 INFO:tasks.workunit.client.0.smithi032.stdout:5/803: fdatasync d2/dd/d5d/fce 0 2024-04-12T05:37:15.304 INFO:tasks.workunit.client.0.smithi032.stdout:5/804: creat d2/df/d31/d80/fff x:0 0 0 2024-04-12T05:37:15.305 INFO:tasks.workunit.client.0.smithi032.stdout:3/780: dread d12/d2f/d89/df0/d3a/d63/fcf [0,4194304] 0 2024-04-12T05:37:15.305 INFO:tasks.workunit.client.0.smithi032.stdout:5/805: write d2/df/d31/fcf [885741,92300] 0 2024-04-12T05:37:15.308 INFO:tasks.workunit.client.0.smithi032.stdout:5/806: creat d2/df/d25/d50/f100 x:0 0 0 2024-04-12T05:37:15.313 INFO:tasks.workunit.client.0.smithi032.stdout:5/807: dwrite d2/df/f5c [0,4194304] 0 2024-04-12T05:37:15.317 INFO:tasks.workunit.client.0.smithi032.stdout:5/808: mkdir d2/df/da9/dc1/d78/d101 0 2024-04-12T05:37:15.319 INFO:tasks.workunit.client.0.smithi032.stdout:5/809: symlink d2/df/da9/dc1/d78/l102 0 2024-04-12T05:37:15.320 INFO:tasks.workunit.client.0.smithi032.stdout:5/810: truncate d2/df/f5c 4237931 0 2024-04-12T05:37:15.325 INFO:tasks.workunit.client.0.smithi032.stdout:5/811: link d2/d3c/d48/l74 d2/df/da9/dc1/d78/d101/l103 0 2024-04-12T05:37:15.326 INFO:tasks.workunit.client.0.smithi032.stdout:5/812: write d2/d3c/d48/d4b/fbb [2998881,34482] 0 2024-04-12T05:37:15.330 INFO:tasks.workunit.client.0.smithi032.stdout:5/813: creat d2/df/d25/d43/d49/f104 x:0 0 0 2024-04-12T05:37:15.331 INFO:tasks.workunit.client.0.smithi032.stdout:5/814: write d2/df/da9/dc1/fdb [39274,1107] 0 2024-04-12T05:37:15.332 INFO:tasks.workunit.client.0.smithi032.stdout:5/815: fsync d2/df/d31/fcf 0 2024-04-12T05:37:15.339 INFO:tasks.workunit.client.0.smithi032.stdout:5/816: write d2/df/d25/d1e/f1f [1331202,100938] 0 2024-04-12T05:37:15.341 INFO:tasks.workunit.client.0.smithi032.stdout:5/817: symlink d2/df/d25/d50/daa/l105 0 2024-04-12T05:37:15.344 INFO:tasks.workunit.client.0.smithi032.stdout:5/818: dread d2/d9b/fb7 [0,4194304] 0 2024-04-12T05:37:15.345 INFO:tasks.workunit.client.0.smithi032.stdout:5/819: mknod d2/df/d31/de7/c106 0 2024-04-12T05:37:15.347 INFO:tasks.workunit.client.0.smithi032.stdout:5/820: fdatasync d2/f7b 0 2024-04-12T05:37:15.349 INFO:tasks.workunit.client.0.smithi032.stdout:5/821: creat d2/dd/ddf/f107 x:0 0 0 2024-04-12T05:37:15.381 INFO:tasks.workunit.client.0.smithi032.stdout:5/822: creat d2/df/d25/d43/d49/f108 x:0 0 0 2024-04-12T05:37:15.382 INFO:tasks.workunit.client.0.smithi032.stdout:5/823: dread d2/d3c/f56 [0,4194304] 0 2024-04-12T05:37:15.382 INFO:tasks.workunit.client.0.smithi032.stdout:5/824: chown d2/d3c/dc6 8148855 1 2024-04-12T05:37:15.384 INFO:tasks.workunit.client.0.smithi032.stdout:5/825: creat d2/df/da9/dc1/d78/d86/f109 x:0 0 0 2024-04-12T05:37:15.388 INFO:tasks.workunit.client.0.smithi032.stdout:5/826: dread d2/d3c/d5b/d75/dd6/fe4 [0,4194304] 0 2024-04-12T05:37:15.390 INFO:tasks.workunit.client.0.smithi032.stdout:5/827: truncate d2/df/d31/d80/f88 646358 0 2024-04-12T05:37:15.392 INFO:tasks.workunit.client.0.smithi032.stdout:5/828: symlink d2/d3c/d48/d4b/dc4/de1/l10a 0 2024-04-12T05:37:15.393 INFO:tasks.workunit.client.0.smithi032.stdout:5/829: chown d2/df/da9/dc1/d78 140279065 1 2024-04-12T05:37:15.399 INFO:tasks.workunit.client.0.smithi032.stdout:5/830: dwrite d2/d3c/f56 [0,4194304] 0 2024-04-12T05:37:15.400 INFO:tasks.workunit.client.0.smithi032.stdout:5/831: write d2/df/f1d [1192383,89294] 0 2024-04-12T05:37:15.403 INFO:tasks.workunit.client.0.smithi032.stdout:5/832: chown d2/d3c/d48/l74 11284005 1 2024-04-12T05:37:15.404 INFO:tasks.workunit.client.0.smithi032.stdout:5/833: write d2/dd/d6f/fbd [613299,60086] 0 2024-04-12T05:37:15.407 INFO:tasks.workunit.client.0.smithi032.stdout:5/834: creat d2/df/d25/d50/daa/f10b x:0 0 0 2024-04-12T05:37:15.410 INFO:tasks.workunit.client.0.smithi032.stdout:5/835: creat d2/d3c/d5b/dd4/f10c x:0 0 0 2024-04-12T05:37:15.413 INFO:tasks.workunit.client.0.smithi032.stdout:5/836: truncate d2/f24 2561706 0 2024-04-12T05:37:15.427 INFO:tasks.workunit.client.0.smithi032.stdout:5/837: dwrite d2/df/d25/d43/d49/f7d [0,4194304] 0 2024-04-12T05:37:15.430 INFO:tasks.workunit.client.0.smithi032.stdout:5/838: creat d2/d3c/d5b/d75/dd6/f10d x:0 0 0 2024-04-12T05:37:15.432 INFO:tasks.workunit.client.0.smithi032.stdout:5/839: symlink d2/dd/d6f/l10e 0 2024-04-12T05:37:15.442 INFO:tasks.workunit.client.0.smithi032.stdout:5/840: dwrite d2/dd/f44 [0,4194304] 0 2024-04-12T05:37:15.443 INFO:tasks.workunit.client.0.smithi032.stdout:5/841: dread - d2/df/d25/d50/daa/f10b zero size 2024-04-12T05:37:15.457 INFO:tasks.workunit.client.0.smithi032.stdout:5/842: dread d2/df/d25/d50/daa/fc9 [0,4194304] 0 2024-04-12T05:37:15.460 INFO:tasks.workunit.client.0.smithi032.stdout:5/843: dread d2/df/da9/dc1/fdb [0,4194304] 0 2024-04-12T05:37:15.462 INFO:tasks.workunit.client.0.smithi032.stdout:5/844: unlink d2/df/d31/d92/fee 0 2024-04-12T05:37:15.467 INFO:tasks.workunit.client.0.smithi032.stdout:5/845: dwrite d2/d9b/f5a [0,4194304] 0 2024-04-12T05:37:15.470 INFO:tasks.workunit.client.0.smithi032.stdout:5/846: rename d2/dd/d6f/c89 to d2/df/da9/c10f 0 2024-04-12T05:37:15.472 INFO:tasks.workunit.client.0.smithi032.stdout:5/847: mknod d2/df/d31/c110 0 2024-04-12T05:37:15.475 INFO:tasks.workunit.client.0.smithi032.stdout:5/848: creat d2/df/d25/d43/d49/f111 x:0 0 0 2024-04-12T05:37:15.477 INFO:tasks.workunit.client.0.smithi032.stdout:5/849: symlink d2/df/d25/d50/l112 0 2024-04-12T05:37:15.479 INFO:tasks.workunit.client.0.smithi032.stdout:5/850: creat d2/df/d31/de7/f113 x:0 0 0 2024-04-12T05:37:15.482 INFO:tasks.workunit.client.0.smithi032.stdout:5/851: link d2/df/da9/dc1/cf7 d2/dd/c114 0 2024-04-12T05:37:15.483 INFO:tasks.workunit.client.0.smithi032.stdout:5/852: chown d2/df/d25/d43/d49/f108 2971 1 2024-04-12T05:37:15.485 INFO:tasks.workunit.client.0.smithi032.stdout:5/853: mknod d2/df/d31/de7/c115 0 2024-04-12T05:37:15.486 INFO:tasks.workunit.client.0.smithi032.stdout:5/854: chown d2/df/da9/dc1/cf7 1436 1 2024-04-12T05:37:15.488 INFO:tasks.workunit.client.0.smithi032.stdout:5/855: getdents d2/d3c/d5b/dd4/df2 0 2024-04-12T05:37:15.504 INFO:tasks.workunit.client.0.smithi032.stdout:5/856: dwrite d2/dd/f83 [0,4194304] 0 2024-04-12T05:37:15.504 INFO:tasks.workunit.client.0.smithi032.stdout:5/857: chown d2/dd/d6f/fbd 58670 1 2024-04-12T05:37:15.508 INFO:tasks.workunit.client.0.smithi032.stdout:5/858: unlink d2/d9b/l5f 0 2024-04-12T05:37:15.512 INFO:tasks.workunit.client.0.smithi032.stdout:5/859: dread d2/dd/f83 [0,4194304] 0 2024-04-12T05:37:15.517 INFO:tasks.workunit.client.0.smithi032.stdout:5/860: link d2/d3c/d5b/d75/dd6/f10d d2/df/da9/f116 0 2024-04-12T05:37:15.521 INFO:tasks.workunit.client.0.smithi032.stdout:5/861: link d2/f68 d2/df/da9/dc1/d6c/df3/f117 0 2024-04-12T05:37:15.525 INFO:tasks.workunit.client.0.smithi032.stdout:5/862: dread d2/df/d31/d80/f88 [0,4194304] 0 2024-04-12T05:37:15.526 INFO:tasks.workunit.client.0.smithi032.stdout:5/863: write d2/df/d25/d43/d49/f104 [733053,105412] 0 2024-04-12T05:37:15.536 INFO:tasks.workunit.client.0.smithi032.stdout:5/864: dwrite d2/df/f1c [0,4194304] 0 2024-04-12T05:37:15.538 INFO:tasks.workunit.client.0.smithi032.stdout:5/865: read d2/df/d25/d43/d49/feb [2959264,7687] 0 2024-04-12T05:37:15.670 INFO:tasks.workunit.client.0.smithi032.stdout:4/705: sync 2024-04-12T05:37:15.671 INFO:tasks.workunit.client.0.smithi032.stdout:0/756: sync 2024-04-12T05:37:15.671 INFO:tasks.workunit.client.0.smithi032.stdout:3/781: sync 2024-04-12T05:37:15.671 INFO:tasks.workunit.client.0.smithi032.stdout:1/635: sync 2024-04-12T05:37:15.671 INFO:tasks.workunit.client.0.smithi032.stdout:7/739: sync 2024-04-12T05:37:15.671 INFO:tasks.workunit.client.0.smithi032.stdout:8/763: sync 2024-04-12T05:37:15.671 INFO:tasks.workunit.client.0.smithi032.stdout:2/740: sync 2024-04-12T05:37:15.671 INFO:tasks.workunit.client.0.smithi032.stdout:9/585: sync 2024-04-12T05:37:15.671 INFO:tasks.workunit.client.0.smithi032.stdout:8/764: stat d0/db/d7d 0 2024-04-12T05:37:15.672 INFO:tasks.workunit.client.0.smithi032.stdout:0/757: stat d2/d8/d17/d1b/d20/l4e 0 2024-04-12T05:37:15.673 INFO:tasks.workunit.client.0.smithi032.stdout:0/758: chown d2/d29/d8a/fd7 201641 1 2024-04-12T05:37:15.674 INFO:tasks.workunit.client.0.smithi032.stdout:2/741: truncate dd/d19/f78 585030 0 2024-04-12T05:37:15.723 INFO:tasks.workunit.client.0.smithi032.stdout:8/765: symlink d0/db/d38/l10c 0 2024-04-12T05:37:15.724 INFO:tasks.workunit.client.0.smithi032.stdout:3/782: truncate d12/f9f 955263 0 2024-04-12T05:37:15.728 INFO:tasks.workunit.client.0.smithi032.stdout:8/766: chown d0/db/d38/d63/fa1 3310 1 2024-04-12T05:37:15.729 INFO:tasks.workunit.client.0.smithi032.stdout:3/783: mkdir d12/d2f/de2/d101 0 2024-04-12T05:37:15.731 INFO:tasks.workunit.client.0.smithi032.stdout:1/636: write d5/de/f10 [4542608,117923] 0 2024-04-12T05:37:15.731 INFO:tasks.workunit.client.0.smithi032.stdout:4/706: dwrite d4/d87/d96/d4d/d78/f83 [0,4194304] 0 2024-04-12T05:37:15.733 INFO:tasks.workunit.client.0.smithi032.stdout:0/759: dwrite d2/d8/d48/fe6 [0,4194304] 0 2024-04-12T05:37:15.737 INFO:tasks.workunit.client.0.smithi032.stdout:4/707: dread - d4/fb7 zero size 2024-04-12T05:37:15.740 INFO:tasks.workunit.client.0.smithi032.stdout:3/784: symlink d12/d2f/d89/df0/d3a/d63/dd5/l102 0 2024-04-12T05:37:15.743 INFO:tasks.workunit.client.0.smithi032.stdout:8/767: dread d0/f98 [0,4194304] 0 2024-04-12T05:37:15.744 INFO:tasks.workunit.client.0.smithi032.stdout:0/760: truncate d2/d28/d87/d79/fce 5194456 0 2024-04-12T05:37:15.744 INFO:tasks.workunit.client.0.smithi032.stdout:2/742: link dd/d19/d9d/d1c/d64/d36/l9e dd/d19/d9d/d1c/d64/d32/lf6 0 2024-04-12T05:37:15.748 INFO:tasks.workunit.client.0.smithi032.stdout:1/637: write d5/de/d17/d35/d61/d77/d8f/d9c/fa5 [204598,10745] 0 2024-04-12T05:37:15.749 INFO:tasks.workunit.client.0.smithi032.stdout:8/768: symlink d0/db/d38/d63/d77/dd4/l10d 0 2024-04-12T05:37:15.750 INFO:tasks.workunit.client.0.smithi032.stdout:4/708: truncate d4/d13/d19/f30 6960212 0 2024-04-12T05:37:15.751 INFO:tasks.workunit.client.0.smithi032.stdout:0/761: creat d2/d8/df/d24/dc3/ff9 x:0 0 0 2024-04-12T05:37:15.752 INFO:tasks.workunit.client.0.smithi032.stdout:5/866: sync 2024-04-12T05:37:15.752 INFO:tasks.workunit.client.0.smithi032.stdout:2/743: truncate dd/d19/d9d/d1c/f76 944305 0 2024-04-12T05:37:15.752 INFO:tasks.workunit.client.0.smithi032.stdout:1/638: readlink d5/d2c/d4f/l6a 0 2024-04-12T05:37:15.756 INFO:tasks.workunit.client.0.smithi032.stdout:3/785: dwrite d12/d2f/d89/df0/dcb/fed [0,4194304] 0 2024-04-12T05:37:15.758 INFO:tasks.workunit.client.0.smithi032.stdout:1/639: fdatasync d5/de/d17/d35/d61/fc4 0 2024-04-12T05:37:15.759 INFO:tasks.workunit.client.0.smithi032.stdout:0/762: dread - d2/d29/d8a/fd1 zero size 2024-04-12T05:37:15.759 INFO:tasks.workunit.client.0.smithi032.stdout:5/867: symlink d2/dd/d5d/l118 0 2024-04-12T05:37:15.766 INFO:tasks.workunit.client.0.smithi032.stdout:4/709: write d4/d13/d75/fbd [429286,40957] 0 2024-04-12T05:37:15.767 INFO:tasks.workunit.client.0.smithi032.stdout:8/769: write d0/db/dd/d11/d1b/d96/fef [34305,7429] 0 2024-04-12T05:37:15.774 INFO:tasks.workunit.client.0.smithi032.stdout:0/763: creat d2/d29/d8a/d99/ffa x:0 0 0 2024-04-12T05:37:15.774 INFO:tasks.workunit.client.0.smithi032.stdout:1/640: dwrite d5/de/d17/d35/fbf [0,4194304] 0 2024-04-12T05:37:15.778 INFO:tasks.workunit.client.0.smithi032.stdout:2/744: rename dd/d19/d9d/d1c/d64/d92/c95 to dd/d19/d9d/d42/dc7/cf7 0 2024-04-12T05:37:15.778 INFO:tasks.workunit.client.0.smithi032.stdout:3/786: dwrite d12/d2f/d89/df0/d3a/f3b [0,4194304] 0 2024-04-12T05:37:15.782 INFO:tasks.workunit.client.0.smithi032.stdout:4/710: mknod d4/d87/d96/d4d/db5/dcf/de3/cef 0 2024-04-12T05:37:15.783 INFO:tasks.workunit.client.0.smithi032.stdout:1/641: creat d5/de/d17/d35/d3e/db4/fc6 x:0 0 0 2024-04-12T05:37:15.784 INFO:tasks.workunit.client.0.smithi032.stdout:1/642: chown d5/d2c/d4f/c88 4267149 1 2024-04-12T05:37:15.785 INFO:tasks.workunit.client.0.smithi032.stdout:0/764: mknod d2/dc/d3e/dac/d51/d83/cfb 0 2024-04-12T05:37:15.786 INFO:tasks.workunit.client.0.smithi032.stdout:2/745: fdatasync f1 0 2024-04-12T05:37:15.787 INFO:tasks.workunit.client.0.smithi032.stdout:4/711: creat d4/d87/ff0 x:0 0 0 2024-04-12T05:37:15.789 INFO:tasks.workunit.client.0.smithi032.stdout:8/770: write d0/db/f60 [1370283,82867] 0 2024-04-12T05:37:15.792 INFO:tasks.workunit.client.0.smithi032.stdout:8/771: chown d0/db/d38/l47 7371 1 2024-04-12T05:37:15.793 INFO:tasks.workunit.client.0.smithi032.stdout:4/712: creat d4/d87/d96/d4d/db5/dcf/de3/ff1 x:0 0 0 2024-04-12T05:37:15.793 INFO:tasks.workunit.client.0.smithi032.stdout:3/787: truncate d12/d2f/d89/df0/d3a/f26 4276453 0 2024-04-12T05:37:15.794 INFO:tasks.workunit.client.0.smithi032.stdout:0/765: link d2/d8/df/d24/c40 d2/d8/cfc 0 2024-04-12T05:37:15.795 INFO:tasks.workunit.client.0.smithi032.stdout:1/643: truncate d5/de/d17/d35/d3e/f4c 1746146 0 2024-04-12T05:37:15.797 INFO:tasks.workunit.client.0.smithi032.stdout:5/868: dread d2/df/da9/dc1/d78/f9e [4194304,4194304] 0 2024-04-12T05:37:15.798 INFO:tasks.workunit.client.0.smithi032.stdout:3/788: read d12/d2f/f48 [3338070,28849] 0 2024-04-12T05:37:15.799 INFO:tasks.workunit.client.0.smithi032.stdout:4/713: mknod d4/d87/d96/d4d/d78/dba/cf2 0 2024-04-12T05:37:15.800 INFO:tasks.workunit.client.0.smithi032.stdout:2/746: dread dd/d19/d20/f53 [4194304,4194304] 0 2024-04-12T05:37:15.800 INFO:tasks.workunit.client.0.smithi032.stdout:3/789: chown d12/c22 171638950 1 2024-04-12T05:37:15.800 INFO:tasks.workunit.client.0.smithi032.stdout:5/869: read - d2/df/da9/f116 zero size 2024-04-12T05:37:15.801 INFO:tasks.workunit.client.0.smithi032.stdout:0/766: link d2/d28/d87/d79/fce d2/d29/d77/db5/ffd 0 2024-04-12T05:37:15.803 INFO:tasks.workunit.client.0.smithi032.stdout:0/767: chown d2/d8/d17/fc4 20 1 2024-04-12T05:37:15.803 INFO:tasks.workunit.client.0.smithi032.stdout:8/772: truncate d0/db/dd/d11/d1b/d22/d35/f87 40320 0 2024-04-12T05:37:15.805 INFO:tasks.workunit.client.0.smithi032.stdout:3/790: symlink d12/d2f/d89/df0/d3a/d63/dd5/l103 0 2024-04-12T05:37:15.805 INFO:tasks.workunit.client.0.smithi032.stdout:1/644: rename d5/de/d17/d35/d73/fc2 to d5/de/d17/d35/d61/d77/d8f/fc7 0 2024-04-12T05:37:15.806 INFO:tasks.workunit.client.0.smithi032.stdout:7/740: sync 2024-04-12T05:37:15.807 INFO:tasks.workunit.client.0.smithi032.stdout:4/714: symlink d4/d13/d19/lf3 0 2024-04-12T05:37:15.807 INFO:tasks.workunit.client.0.smithi032.stdout:3/791: dread - d12/d24/dde/fec zero size 2024-04-12T05:37:15.810 INFO:tasks.workunit.client.0.smithi032.stdout:1/645: chown d5/de/d17/d1b/c6d 1640162 1 2024-04-12T05:37:15.811 INFO:tasks.workunit.client.0.smithi032.stdout:0/768: creat d2/d8/d17/d64/ffe x:0 0 0 2024-04-12T05:37:15.814 INFO:tasks.workunit.client.0.smithi032.stdout:7/741: creat d0/dd/ffc x:0 0 0 2024-04-12T05:37:15.817 INFO:tasks.workunit.client.0.smithi032.stdout:8/773: dwrite d0/db/f60 [4194304,4194304] 0 2024-04-12T05:37:15.817 INFO:tasks.workunit.client.0.smithi032.stdout:3/792: dread d12/d2f/d89/df0/f1e [0,4194304] 0 2024-04-12T05:37:15.817 INFO:tasks.workunit.client.0.smithi032.stdout:2/747: write dd/d19/d20/fa0 [290085,61056] 0 2024-04-12T05:37:15.818 INFO:tasks.workunit.client.0.smithi032.stdout:5/870: dread d2/df/da9/dc1/d6c/f77 [0,4194304] 0 2024-04-12T05:37:15.819 INFO:tasks.workunit.client.0.smithi032.stdout:4/715: creat d4/ff4 x:0 0 0 2024-04-12T05:37:15.821 INFO:tasks.workunit.client.0.smithi032.stdout:1/646: creat d5/de/d17/d93/fc8 x:0 0 0 2024-04-12T05:37:15.822 INFO:tasks.workunit.client.0.smithi032.stdout:7/742: creat d0/d1c/d71/ffd x:0 0 0 2024-04-12T05:37:15.825 INFO:tasks.workunit.client.0.smithi032.stdout:3/793: creat d12/d2f/d89/f104 x:0 0 0 2024-04-12T05:37:15.826 INFO:tasks.workunit.client.0.smithi032.stdout:0/769: symlink d2/d8/df/d24/d60/dc8/lff 0 2024-04-12T05:37:15.827 INFO:tasks.workunit.client.0.smithi032.stdout:8/774: read - d0/d36/d84/dad/ff9 zero size 2024-04-12T05:37:15.829 INFO:tasks.workunit.client.0.smithi032.stdout:5/871: mkdir d2/df/d119 0 2024-04-12T05:37:15.829 INFO:tasks.workunit.client.0.smithi032.stdout:0/770: chown d2/dc/f61 3765 1 2024-04-12T05:37:15.831 INFO:tasks.workunit.client.0.smithi032.stdout:4/716: rename d4/d87/d96/d4d/d78/f6d to d4/d87/d96/d4d/db5/ff5 0 2024-04-12T05:37:15.834 INFO:tasks.workunit.client.0.smithi032.stdout:8/775: write d0/db/dd/d11/d1b/d96/fef [40469,18605] 0 2024-04-12T05:37:15.834 INFO:tasks.workunit.client.0.smithi032.stdout:5/872: dread - d2/df/d25/d50/fd9 zero size 2024-04-12T05:37:15.839 INFO:tasks.workunit.client.0.smithi032.stdout:1/647: creat d5/de/d17/d35/d3e/fc9 x:0 0 0 2024-04-12T05:37:15.840 INFO:tasks.workunit.client.0.smithi032.stdout:3/794: dwrite d12/d2f/d89/fd9 [0,4194304] 0 2024-04-12T05:37:15.842 INFO:tasks.workunit.client.0.smithi032.stdout:8/776: chown d0/db/d38/d63/d77/dd4 41381016 1 2024-04-12T05:37:15.844 INFO:tasks.workunit.client.0.smithi032.stdout:1/648: write d5/de/d2a/fb6 [428964,16900] 0 2024-04-12T05:37:15.846 INFO:tasks.workunit.client.0.smithi032.stdout:0/771: write d2/d8/d17/d1b/fc6 [911201,40142] 0 2024-04-12T05:37:15.847 INFO:tasks.workunit.client.0.smithi032.stdout:9/586: sync 2024-04-12T05:37:15.848 INFO:tasks.workunit.client.0.smithi032.stdout:5/873: creat d2/df/da9/dc1/d6c/f11a x:0 0 0 2024-04-12T05:37:15.848 INFO:tasks.workunit.client.0.smithi032.stdout:3/795: fsync d12/d2f/fc4 0 2024-04-12T05:37:15.850 INFO:tasks.workunit.client.0.smithi032.stdout:8/777: fdatasync d0/db/d38/f61 0 2024-04-12T05:37:15.853 INFO:tasks.workunit.client.0.smithi032.stdout:1/649: mkdir d5/de/d17/d35/d61/dca 0 2024-04-12T05:37:15.855 INFO:tasks.workunit.client.0.smithi032.stdout:3/796: mkdir d12/d2f/d105 0 2024-04-12T05:37:15.856 INFO:tasks.workunit.client.0.smithi032.stdout:5/874: write d2/dd/d6f/fbd [732050,124511] 0 2024-04-12T05:37:15.856 INFO:tasks.workunit.client.0.smithi032.stdout:4/717: dread d4/d13/d19/f2a [0,4194304] 0 2024-04-12T05:37:15.857 INFO:tasks.workunit.client.0.smithi032.stdout:9/587: dread - d9/d32/d86/f9b zero size 2024-04-12T05:37:15.857 INFO:tasks.workunit.client.0.smithi032.stdout:8/778: dread - d0/d58/fe5 zero size 2024-04-12T05:37:15.860 INFO:tasks.workunit.client.0.smithi032.stdout:2/748: dwrite dd/d19/d9d/d42/da7/dc5/fcc [0,4194304] 0 2024-04-12T05:37:15.862 INFO:tasks.workunit.client.0.smithi032.stdout:7/743: dwrite d0/de/df/d29/fdb [0,4194304] 0 2024-04-12T05:37:15.863 INFO:tasks.workunit.client.0.smithi032.stdout:4/718: fsync d4/d87/d96/d2f/fda 0 2024-04-12T05:37:15.865 INFO:tasks.workunit.client.0.smithi032.stdout:8/779: chown d0/db/dd/d11/d1b/d22 120530 1 2024-04-12T05:37:15.866 INFO:tasks.workunit.client.0.smithi032.stdout:9/588: creat d9/d15/d2c/d41/fc8 x:0 0 0 2024-04-12T05:37:15.867 INFO:tasks.workunit.client.0.smithi032.stdout:1/650: creat d5/de/fcb x:0 0 0 2024-04-12T05:37:15.869 INFO:tasks.workunit.client.0.smithi032.stdout:5/875: creat d2/df/da9/dc1/d6c/df3/f11b x:0 0 0 2024-04-12T05:37:15.873 INFO:tasks.workunit.client.0.smithi032.stdout:9/589: chown d9/d15/d1a/d31/d65/fb7 38 1 2024-04-12T05:37:15.873 INFO:tasks.workunit.client.0.smithi032.stdout:3/797: dwrite d12/f56 [0,4194304] 0 2024-04-12T05:37:15.877 INFO:tasks.workunit.client.0.smithi032.stdout:7/744: creat d0/d13/d21/d30/dfb/ffe x:0 0 0 2024-04-12T05:37:15.879 INFO:tasks.workunit.client.0.smithi032.stdout:5/876: creat d2/df/da9/dc1/d78/d86/f11c x:0 0 0 2024-04-12T05:37:15.880 INFO:tasks.workunit.client.0.smithi032.stdout:7/745: dread - d0/d34/fcb zero size 2024-04-12T05:37:15.880 INFO:tasks.workunit.client.0.smithi032.stdout:8/780: write d0/db/dd/d11/d1b/d22/d54/da8/fd9 [1315450,99309] 0 2024-04-12T05:37:15.882 INFO:tasks.workunit.client.0.smithi032.stdout:9/590: read d9/d15/d2c/d41/f45 [3044644,75226] 0 2024-04-12T05:37:15.884 INFO:tasks.workunit.client.0.smithi032.stdout:9/591: readlink d9/d15/d1a/l73 0 2024-04-12T05:37:15.886 INFO:tasks.workunit.client.0.smithi032.stdout:3/798: write d12/d2f/f81 [49011,68893] 0 2024-04-12T05:37:15.887 INFO:tasks.workunit.client.0.smithi032.stdout:3/799: read - d12/fe6 zero size 2024-04-12T05:37:15.887 INFO:tasks.workunit.client.0.smithi032.stdout:7/746: rename d0/d1c to d0/dff 0 2024-04-12T05:37:15.889 INFO:tasks.workunit.client.0.smithi032.stdout:8/781: rmdir d0/db/d7d 39 2024-04-12T05:37:15.890 INFO:tasks.workunit.client.0.smithi032.stdout:5/877: truncate d2/df/da9/dc1/d6c/ff4 107408 0 2024-04-12T05:37:15.891 INFO:tasks.workunit.client.0.smithi032.stdout:3/800: creat d12/d2f/d89/df0/d98/f106 x:0 0 0 2024-04-12T05:37:15.892 INFO:tasks.workunit.client.0.smithi032.stdout:2/749: write dd/d19/d9d/d1c/fb6 [953129,28506] 0 2024-04-12T05:37:15.892 INFO:tasks.workunit.client.0.smithi032.stdout:7/747: fsync d0/d13/f64 0 2024-04-12T05:37:15.895 INFO:tasks.workunit.client.0.smithi032.stdout:4/719: write d4/d13/d44/fa0 [104976,88420] 0 2024-04-12T05:37:15.895 INFO:tasks.workunit.client.0.smithi032.stdout:3/801: fdatasync d12/d24/f8f 0 2024-04-12T05:37:15.898 INFO:tasks.workunit.client.0.smithi032.stdout:3/802: mkdir d12/d2f/de2/d107 0 2024-04-12T05:37:15.903 INFO:tasks.workunit.client.0.smithi032.stdout:9/592: truncate d9/d15/d1a/d1d/fb9 1527573 0 2024-04-12T05:37:15.904 INFO:tasks.workunit.client.0.smithi032.stdout:3/803: rename d12/d2f/fdc to d12/daa/f108 0 2024-04-12T05:37:15.904 INFO:tasks.workunit.client.0.smithi032.stdout:7/748: dread d0/dff/fc6 [0,4194304] 0 2024-04-12T05:37:15.905 INFO:tasks.workunit.client.0.smithi032.stdout:4/720: mknod d4/d87/d96/d4d/d78/cf6 0 2024-04-12T05:37:15.906 INFO:tasks.workunit.client.0.smithi032.stdout:8/782: write d0/db/dd/d11/d1b/f6c [744987,76182] 0 2024-04-12T05:37:15.909 INFO:tasks.workunit.client.0.smithi032.stdout:3/804: symlink d12/d2f/d89/df0/d3a/d63/dd5/l109 0 2024-04-12T05:37:15.909 INFO:tasks.workunit.client.0.smithi032.stdout:7/749: chown d0/d34/dbe/f5c 18989 1 2024-04-12T05:37:15.909 INFO:tasks.workunit.client.0.smithi032.stdout:9/593: rmdir d9/d15/d1a/d1d/d5a/d6b 39 2024-04-12T05:37:15.911 INFO:tasks.workunit.client.0.smithi032.stdout:8/783: mknod d0/d36/c10e 0 2024-04-12T05:37:15.913 INFO:tasks.workunit.client.0.smithi032.stdout:1/651: dread d5/de/d17/d35/fa1 [0,4194304] 0 2024-04-12T05:37:15.913 INFO:tasks.workunit.client.0.smithi032.stdout:5/878: truncate d2/df/da9/dc1/d6c/fca 69757 0 2024-04-12T05:37:15.916 INFO:tasks.workunit.client.0.smithi032.stdout:4/721: rmdir d4/d87/d96/d2f/d3b/de8 0 2024-04-12T05:37:15.918 INFO:tasks.workunit.client.0.smithi032.stdout:5/879: dread - d2/df/da9/dc1/d6c/df3/f11b zero size 2024-04-12T05:37:15.918 INFO:tasks.workunit.client.0.smithi032.stdout:2/750: dwrite dd/d19/d9d/d1c/d64/d32/d3a/d8d/fe4 [0,4194304] 0 2024-04-12T05:37:15.918 INFO:tasks.workunit.client.0.smithi032.stdout:7/750: dread - d0/d34/dbe/dbb/fd1 zero size 2024-04-12T05:37:15.922 INFO:tasks.workunit.client.0.smithi032.stdout:8/784: creat d0/d36/d10a/f10f x:0 0 0 2024-04-12T05:37:15.922 INFO:tasks.workunit.client.0.smithi032.stdout:9/594: dread d9/d32/f74 [0,4194304] 0 2024-04-12T05:37:15.924 INFO:tasks.workunit.client.0.smithi032.stdout:8/785: fsync d0/f2f 0 2024-04-12T05:37:15.928 INFO:tasks.workunit.client.0.smithi032.stdout:2/751: unlink dd/d19/d20/f53 0 2024-04-12T05:37:15.931 INFO:tasks.workunit.client.0.smithi032.stdout:3/805: dwrite d12/d2f/d89/df0/dcb/ffb [0,4194304] 0 2024-04-12T05:37:15.932 INFO:tasks.workunit.client.0.smithi032.stdout:4/722: dread d4/d87/d96/d2f/fda [0,4194304] 0 2024-04-12T05:37:15.933 INFO:tasks.workunit.client.0.smithi032.stdout:5/880: rename d2/df/d25/d43/f51 to d2/d3c/d48/f11d 0 2024-04-12T05:37:15.934 INFO:tasks.workunit.client.0.smithi032.stdout:2/752: mknod dd/d19/d9d/d42/da7/dc5/cf8 0 2024-04-12T05:37:15.937 INFO:tasks.workunit.client.0.smithi032.stdout:8/786: creat d0/db/dd/d11/d1b/d22/d54/f110 x:0 0 0 2024-04-12T05:37:15.937 INFO:tasks.workunit.client.0.smithi032.stdout:4/723: fdatasync d4/d13/d75/fbd 0 2024-04-12T05:37:15.938 INFO:tasks.workunit.client.0.smithi032.stdout:7/751: dread d0/d13/f27 [0,4194304] 0 2024-04-12T05:37:15.939 INFO:tasks.workunit.client.0.smithi032.stdout:1/652: dread d5/de/d17/d35/d3e/f4c [0,4194304] 0 2024-04-12T05:37:15.940 INFO:tasks.workunit.client.0.smithi032.stdout:9/595: link d9/d15/d2c/f20 d9/d15/d1a/d31/fc9 0 2024-04-12T05:37:15.941 INFO:tasks.workunit.client.0.smithi032.stdout:2/753: read dd/d19/d9d/d1c/d64/d32/d6d/f77 [177397,108342] 0 2024-04-12T05:37:15.942 INFO:tasks.workunit.client.0.smithi032.stdout:7/752: symlink d0/d10/l100 0 2024-04-12T05:37:15.943 INFO:tasks.workunit.client.0.smithi032.stdout:3/806: truncate d12/d2f/d89/df0/f7b 19566 0 2024-04-12T05:37:15.943 INFO:tasks.workunit.client.0.smithi032.stdout:8/787: chown d0/db/dd/d11/d1b/d22/d35/l10b 11178748 1 2024-04-12T05:37:15.943 INFO:tasks.workunit.client.0.smithi032.stdout:1/653: read d5/de/d17/fa7 [8299889,28089] 0 2024-04-12T05:37:15.944 INFO:tasks.workunit.client.0.smithi032.stdout:4/724: mknod d4/d87/d96/d4d/d78/dba/cf7 0 2024-04-12T05:37:15.947 INFO:tasks.workunit.client.0.smithi032.stdout:2/754: chown dd/d19/d9d/d42/d4a/fdd 27384142 1 2024-04-12T05:37:15.947 INFO:tasks.workunit.client.0.smithi032.stdout:3/807: dread - d12/d2f/d89/df0/d3a/d2d/ff2 zero size 2024-04-12T05:37:15.948 INFO:tasks.workunit.client.0.smithi032.stdout:5/881: dread d2/df/d25/d43/d49/f7d [0,4194304] 0 2024-04-12T05:37:15.949 INFO:tasks.workunit.client.0.smithi032.stdout:0/772: sync 2024-04-12T05:37:15.950 INFO:tasks.workunit.client.0.smithi032.stdout:2/755: truncate f0 136661 0 2024-04-12T05:37:15.952 INFO:tasks.workunit.client.0.smithi032.stdout:4/725: creat d4/d87/d96/d2f/ff8 x:0 0 0 2024-04-12T05:37:15.953 INFO:tasks.workunit.client.0.smithi032.stdout:5/882: rmdir d2/df/d31/d80 39 2024-04-12T05:37:15.954 INFO:tasks.workunit.client.0.smithi032.stdout:3/808: dread - d12/d2f/d89/df0/d3a/d63/da6/dac/dc9/ff3 zero size 2024-04-12T05:37:15.955 INFO:tasks.workunit.client.0.smithi032.stdout:8/788: truncate d0/db/dd/d11/d1b/fc9 501895 0 2024-04-12T05:37:15.960 INFO:tasks.workunit.client.0.smithi032.stdout:2/756: creat dd/d19/d82/ddf/ff9 x:0 0 0 2024-04-12T05:37:15.960 INFO:tasks.workunit.client.0.smithi032.stdout:7/753: dread d0/de/f12 [0,4194304] 0 2024-04-12T05:37:15.960 INFO:tasks.workunit.client.0.smithi032.stdout:1/654: dwrite d5/de/d17/d35/d3b/f98 [0,4194304] 0 2024-04-12T05:37:15.961 INFO:tasks.workunit.client.0.smithi032.stdout:3/809: chown d12/d2f/d89/df0/d3a/d63/da6/df5 172 1 2024-04-12T05:37:15.964 INFO:tasks.workunit.client.0.smithi032.stdout:0/773: getdents d2/d29/d8a/d99 0 2024-04-12T05:37:15.968 INFO:tasks.workunit.client.0.smithi032.stdout:2/757: mkdir dd/d19/d9d/d1c/d64/d92/dfa 0 2024-04-12T05:37:15.973 INFO:tasks.workunit.client.0.smithi032.stdout:3/810: dwrite d12/d2f/fc4 [0,4194304] 0 2024-04-12T05:37:15.974 INFO:tasks.workunit.client.0.smithi032.stdout:8/789: mkdir d0/db/d7d/d111 0 2024-04-12T05:37:15.977 INFO:tasks.workunit.client.0.smithi032.stdout:4/726: mknod d4/d87/d96/d4d/dbc/dd2/cf9 0 2024-04-12T05:37:15.977 INFO:tasks.workunit.client.0.smithi032.stdout:9/596: dwrite d9/d15/d2c/d41/f59 [0,4194304] 0 2024-04-12T05:37:15.978 INFO:tasks.workunit.client.0.smithi032.stdout:7/754: getdents d0/de/d47/de1 0 2024-04-12T05:37:15.981 INFO:tasks.workunit.client.0.smithi032.stdout:4/727: read d4/f98 [149322,15928] 0 2024-04-12T05:37:15.983 INFO:tasks.workunit.client.0.smithi032.stdout:0/774: rename d2/d29/d77/c86 to d2/dc/d3e/c100 0 2024-04-12T05:37:15.985 INFO:tasks.workunit.client.0.smithi032.stdout:7/755: fdatasync d0/d13/d21/d81/f9c 0 2024-04-12T05:37:15.987 INFO:tasks.workunit.client.0.smithi032.stdout:5/883: dwrite d2/df/d25/d50/f76 [0,4194304] 0 2024-04-12T05:37:15.988 INFO:tasks.workunit.client.0.smithi032.stdout:8/790: read d0/db/d38/f61 [3464771,45291] 0 2024-04-12T05:37:15.992 INFO:tasks.workunit.client.0.smithi032.stdout:9/597: truncate d9/d15/d2c/f10 1434572 0 2024-04-12T05:37:15.992 INFO:tasks.workunit.client.0.smithi032.stdout:1/655: write d5/de/d17/d35/d73/f94 [2174580,74452] 0 2024-04-12T05:37:15.997 INFO:tasks.workunit.client.0.smithi032.stdout:2/758: rename dd/d19/d9d/d1c/d64/d32/d3a/f6a to dd/d19/d9d/d1c/ffb 0 2024-04-12T05:37:15.999 INFO:tasks.workunit.client.0.smithi032.stdout:5/884: creat d2/d9b/f11e x:0 0 0 2024-04-12T05:37:15.999 INFO:tasks.workunit.client.0.smithi032.stdout:8/791: unlink d0/d58/fe5 0 2024-04-12T05:37:16.000 INFO:tasks.workunit.client.0.smithi032.stdout:9/598: symlink d9/d15/d1a/dba/lca 0 2024-04-12T05:37:16.002 INFO:tasks.workunit.client.0.smithi032.stdout:0/775: truncate d2/dc/fd 127802 0 2024-04-12T05:37:16.003 INFO:tasks.workunit.client.0.smithi032.stdout:3/811: rename d12/d2f/fc4 to d12/d2f/d89/df0/d3a/dbe/f10a 0 2024-04-12T05:37:16.003 INFO:tasks.workunit.client.0.smithi032.stdout:1/656: symlink d5/de/d5b/lcc 0 2024-04-12T05:37:16.004 INFO:tasks.workunit.client.0.smithi032.stdout:3/812: chown d12/d2f/d89/df0/l60 20 1 2024-04-12T05:37:16.005 INFO:tasks.workunit.client.0.smithi032.stdout:3/813: dread - d12/daa/db3/fe7 zero size 2024-04-12T05:37:16.005 INFO:tasks.workunit.client.0.smithi032.stdout:2/759: creat dd/d19/d9d/d1c/d64/d36/ffc x:0 0 0 2024-04-12T05:37:16.007 INFO:tasks.workunit.client.0.smithi032.stdout:9/599: symlink d9/d15/d1a/d31/lcb 0 2024-04-12T05:37:16.009 INFO:tasks.workunit.client.0.smithi032.stdout:3/814: truncate d12/d24/f4d 1079873 0 2024-04-12T05:37:16.011 INFO:tasks.workunit.client.0.smithi032.stdout:8/792: creat d0/dfd/f112 x:0 0 0 2024-04-12T05:37:16.013 INFO:tasks.workunit.client.0.smithi032.stdout:8/793: write d0/f2f [2360614,87260] 0 2024-04-12T05:37:16.016 INFO:tasks.workunit.client.0.smithi032.stdout:2/760: rename dd/d19/d20/cd2 to dd/d19/d9d/d42/d4a/da1/dd5/cfd 0 2024-04-12T05:37:16.017 INFO:tasks.workunit.client.0.smithi032.stdout:7/756: write d0/d13/f8c [1309284,111715] 0 2024-04-12T05:37:16.017 INFO:tasks.workunit.client.0.smithi032.stdout:0/776: dread d2/dc/d41/f89 [0,4194304] 0 2024-04-12T05:37:16.022 INFO:tasks.workunit.client.0.smithi032.stdout:3/815: dread d12/d2f/d89/df0/d3a/f96 [0,4194304] 0 2024-04-12T05:37:16.023 INFO:tasks.workunit.client.0.smithi032.stdout:8/794: truncate d0/d36/f4a 579252 0 2024-04-12T05:37:16.024 INFO:tasks.workunit.client.0.smithi032.stdout:5/885: getdents d2/d9b 0 2024-04-12T05:37:16.025 INFO:tasks.workunit.client.0.smithi032.stdout:4/728: truncate d4/d13/d19/fa8 1909261 0 2024-04-12T05:37:16.026 INFO:tasks.workunit.client.0.smithi032.stdout:2/761: dread - dd/d19/d20/d83/f9c zero size 2024-04-12T05:37:16.027 INFO:tasks.workunit.client.0.smithi032.stdout:0/777: creat d2/d29/d77/dc7/f101 x:0 0 0 2024-04-12T05:37:16.028 INFO:tasks.workunit.client.0.smithi032.stdout:1/657: getdents d5/de/d17/d35/d3e/db4 0 2024-04-12T05:37:16.029 INFO:tasks.workunit.client.0.smithi032.stdout:8/795: mknod d0/db/d38/d63/d77/c113 0 2024-04-12T05:37:16.030 INFO:tasks.workunit.client.0.smithi032.stdout:7/757: mknod d0/dd/d51/c101 0 2024-04-12T05:37:16.034 INFO:tasks.workunit.client.0.smithi032.stdout:3/816: dwrite d12/d2f/d89/df0/d3a/d2d/f4e [0,4194304] 0 2024-04-12T05:37:16.036 INFO:tasks.workunit.client.0.smithi032.stdout:1/658: creat d5/d2c/d4f/fcd x:0 0 0 2024-04-12T05:37:16.039 INFO:tasks.workunit.client.0.smithi032.stdout:1/659: stat d5/d2c/c84 0 2024-04-12T05:37:16.042 INFO:tasks.workunit.client.0.smithi032.stdout:7/758: readlink d0/dd/ld0 0 2024-04-12T05:37:16.043 INFO:tasks.workunit.client.0.smithi032.stdout:9/600: dwrite d9/d15/d1a/d24/f47 [0,4194304] 0 2024-04-12T05:37:16.043 INFO:tasks.workunit.client.0.smithi032.stdout:1/660: write d5/de/d17/d35/d3e/db4/fc6 [726815,12938] 0 2024-04-12T05:37:16.045 INFO:tasks.workunit.client.0.smithi032.stdout:1/661: chown d5/de/d17/d35/d61/d77/c9b 8616 1 2024-04-12T05:37:16.047 INFO:tasks.workunit.client.0.smithi032.stdout:0/778: fsync d2/d28/d87/d79/ff8 0 2024-04-12T05:37:16.049 INFO:tasks.workunit.client.0.smithi032.stdout:3/817: rmdir d12/d2f/d89/df0/d3a/d2d/df4 0 2024-04-12T05:37:16.050 INFO:tasks.workunit.client.0.smithi032.stdout:4/729: creat d4/d13/d19/ffa x:0 0 0 2024-04-12T05:37:16.053 INFO:tasks.workunit.client.0.smithi032.stdout:5/886: link d2/df/d31/d80/fff d2/df/d25/d43/f11f 0 2024-04-12T05:37:16.054 INFO:tasks.workunit.client.0.smithi032.stdout:3/818: unlink d12/d2f/d89/df0/d3a/d2d/c39 0 2024-04-12T05:37:16.055 INFO:tasks.workunit.client.0.smithi032.stdout:0/779: mknod d2/d8/d48/c102 0 2024-04-12T05:37:16.055 INFO:tasks.workunit.client.0.smithi032.stdout:2/762: dwrite dd/f16 [0,4194304] 0 2024-04-12T05:37:16.056 INFO:tasks.workunit.client.0.smithi032.stdout:8/796: creat d0/f114 x:0 0 0 2024-04-12T05:37:16.056 INFO:tasks.workunit.client.0.smithi032.stdout:5/887: chown d2/df/d31/da2/dba 56 1 2024-04-12T05:37:16.058 INFO:tasks.workunit.client.0.smithi032.stdout:5/888: dread - d2/df/da9/f116 zero size 2024-04-12T05:37:16.060 INFO:tasks.workunit.client.0.smithi032.stdout:5/889: write d2/df/da9/dc1/fd3 [381631,92396] 0 2024-04-12T05:37:16.062 INFO:tasks.workunit.client.0.smithi032.stdout:3/819: link d12/d2f/d89/cb4 d12/d2f/d105/c10b 0 2024-04-12T05:37:16.063 INFO:tasks.workunit.client.0.smithi032.stdout:3/820: truncate d12/d2f/d89/d9d/fce 419343 0 2024-04-12T05:37:16.071 INFO:tasks.workunit.client.0.smithi032.stdout:3/821: dwrite d12/d2f/d89/df0/d3a/d63/da6/dac/dc9/ff3 [0,4194304] 0 2024-04-12T05:37:16.108 INFO:tasks.workunit.client.0.smithi032.stdout:4/730: creat d4/d87/d96/d2f/d7c/ffb x:0 0 0 2024-04-12T05:37:16.108 INFO:tasks.workunit.client.0.smithi032.stdout:4/731: chown d4/d13/d75/fbd 4 1 2024-04-12T05:37:16.109 INFO:tasks.workunit.client.0.smithi032.stdout:9/601: truncate d9/d15/f48 13534 0 2024-04-12T05:37:16.111 INFO:tasks.workunit.client.0.smithi032.stdout:2/763: mknod dd/d19/d20/dc6/cfe 0 2024-04-12T05:37:16.131 INFO:tasks.workunit.client.0.smithi032.stdout:3/822: dread d12/f13 [0,4194304] 0 2024-04-12T05:37:16.136 INFO:tasks.workunit.client.0.smithi032.stdout:9/602: dread d9/d15/f43 [0,4194304] 0 2024-04-12T05:37:16.136 INFO:tasks.workunit.client.0.smithi032.stdout:7/759: creat d0/d13/d21/f102 x:0 0 0 2024-04-12T05:37:16.139 INFO:tasks.workunit.client.0.smithi032.stdout:9/603: truncate d9/d15/d2c/fc4 382865 0 2024-04-12T05:37:16.139 INFO:tasks.workunit.client.0.smithi032.stdout:4/732: mkdir d4/d87/d96/d4d/d78/dba/dfc 0 2024-04-12T05:37:16.142 INFO:tasks.workunit.client.0.smithi032.stdout:2/764: rmdir dd/d19/d9d/d1c/d64/d32/d6d 39 2024-04-12T05:37:16.142 INFO:tasks.workunit.client.0.smithi032.stdout:3/823: dwrite d12/d2f/d89/df0/f36 [0,4194304] 0 2024-04-12T05:37:16.143 INFO:tasks.workunit.client.0.smithi032.stdout:5/890: getdents d2/df/da9/dc1/d78/d86/df6 0 2024-04-12T05:37:16.144 INFO:tasks.workunit.client.0.smithi032.stdout:1/662: getdents d5/de/d17/d35/d73/d74 0 2024-04-12T05:37:16.145 INFO:tasks.workunit.client.0.smithi032.stdout:1/663: readlink l1 0 2024-04-12T05:37:16.146 INFO:tasks.workunit.client.0.smithi032.stdout:7/760: creat d0/dff/d4c/f103 x:0 0 0 2024-04-12T05:37:16.149 INFO:tasks.workunit.client.0.smithi032.stdout:0/780: rename d2/d29/d77/dc7/lf3 to d2/d8/df/d24/l103 0 2024-04-12T05:37:16.150 INFO:tasks.workunit.client.0.smithi032.stdout:1/664: fdatasync d5/de/d17/d35/d73/fba 0 2024-04-12T05:37:16.155 INFO:tasks.workunit.client.0.smithi032.stdout:8/797: dwrite d0/db/d38/d63/fa1 [0,4194304] 0 2024-04-12T05:37:16.158 INFO:tasks.workunit.client.0.smithi032.stdout:0/781: truncate d2/dc/d41/f89 910368 0 2024-04-12T05:37:16.160 INFO:tasks.workunit.client.0.smithi032.stdout:9/604: rename d9/d15/d1a/d24/d71 to d9/d32/d86/dcc 0 2024-04-12T05:37:16.162 INFO:tasks.workunit.client.0.smithi032.stdout:3/824: dwrite d12/f70 [0,4194304] 0 2024-04-12T05:37:16.162 INFO:tasks.workunit.client.0.smithi032.stdout:0/782: readlink d2/d29/d8a/led 0 2024-04-12T05:37:16.164 INFO:tasks.workunit.client.0.smithi032.stdout:0/783: chown d2/d29/d77/dc7/le8 504 1 2024-04-12T05:37:16.164 INFO:tasks.workunit.client.0.smithi032.stdout:8/798: symlink d0/db/dd/d11/d1b/d22/d54/da8/l115 0 2024-04-12T05:37:16.167 INFO:tasks.workunit.client.0.smithi032.stdout:8/799: chown d0/db/dd/c5d 246 1 2024-04-12T05:37:16.168 INFO:tasks.workunit.client.0.smithi032.stdout:1/665: truncate d5/de/d17/d35/d3b/d49/f62 1381030 0 2024-04-12T05:37:16.168 INFO:tasks.workunit.client.0.smithi032.stdout:8/800: stat d0/db 0 2024-04-12T05:37:16.171 INFO:tasks.workunit.client.0.smithi032.stdout:7/761: rename d0/d34/dbf/ce7 to d0/d13/d21/d30/de0/c104 0 2024-04-12T05:37:16.172 INFO:tasks.workunit.client.0.smithi032.stdout:4/733: write d4/f54 [4586156,21713] 0 2024-04-12T05:37:16.175 INFO:tasks.workunit.client.0.smithi032.stdout:3/825: dwrite d12/fe6 [0,4194304] 0 2024-04-12T05:37:16.177 INFO:tasks.workunit.client.0.smithi032.stdout:5/891: truncate d2/df/d25/d50/f7e 2945260 0 2024-04-12T05:37:16.180 INFO:tasks.workunit.client.0.smithi032.stdout:7/762: read - d0/d13/d21/d81/fdf zero size 2024-04-12T05:37:16.181 INFO:tasks.workunit.client.0.smithi032.stdout:1/666: creat d5/de/d17/fce x:0 0 0 2024-04-12T05:37:16.182 INFO:tasks.workunit.client.0.smithi032.stdout:9/605: dread d9/d15/d1a/d24/f4b [0,4194304] 0 2024-04-12T05:37:16.185 INFO:tasks.workunit.client.0.smithi032.stdout:4/734: dread d4/d13/d19/f2a [0,4194304] 0 2024-04-12T05:37:16.186 INFO:tasks.workunit.client.0.smithi032.stdout:7/763: fsync d0/fc 0 2024-04-12T05:37:16.186 INFO:tasks.workunit.client.0.smithi032.stdout:0/784: dwrite d2/d28/d87/f95 [0,4194304] 0 2024-04-12T05:37:16.188 INFO:tasks.workunit.client.0.smithi032.stdout:1/667: read - d5/d2c/d4f/fb9 zero size 2024-04-12T05:37:16.193 INFO:tasks.workunit.client.0.smithi032.stdout:1/668: chown d5/d2c/f7f 497 1 2024-04-12T05:37:16.195 INFO:tasks.workunit.client.0.smithi032.stdout:0/785: stat d2/d8/df/d24/d60/dc8/lff 0 2024-04-12T05:37:16.196 INFO:tasks.workunit.client.0.smithi032.stdout:2/765: dwrite dd/d19/d9d/d1c/d64/d32/d6d/fe5 [0,4194304] 0 2024-04-12T05:37:16.200 INFO:tasks.workunit.client.0.smithi032.stdout:9/606: dread d9/d15/d2c/d41/f59 [0,4194304] 0 2024-04-12T05:37:16.203 INFO:tasks.workunit.client.0.smithi032.stdout:3/826: getdents d12/d24 0 2024-04-12T05:37:16.203 INFO:tasks.workunit.client.0.smithi032.stdout:7/764: mkdir d0/dd/d51/dea/d105 0 2024-04-12T05:37:16.203 INFO:tasks.workunit.client.0.smithi032.stdout:8/801: truncate d0/db/dd/d11/f1f 7914045 0 2024-04-12T05:37:16.206 INFO:tasks.workunit.client.0.smithi032.stdout:8/802: stat d0/d36/d84/cf8 0 2024-04-12T05:37:16.207 INFO:tasks.workunit.client.0.smithi032.stdout:0/786: creat d2/d29/f104 x:0 0 0 2024-04-12T05:37:16.209 INFO:tasks.workunit.client.0.smithi032.stdout:4/735: write d4/f79 [3604871,126081] 0 2024-04-12T05:37:16.210 INFO:tasks.workunit.client.0.smithi032.stdout:3/827: creat d12/daa/f10c x:0 0 0 2024-04-12T05:37:16.214 INFO:tasks.workunit.client.0.smithi032.stdout:2/766: rmdir dd/d19/d9d/d42 39 2024-04-12T05:37:16.214 INFO:tasks.workunit.client.0.smithi032.stdout:5/892: dwrite d2/df/da9/dc1/d6c/fca [0,4194304] 0 2024-04-12T05:37:16.218 INFO:tasks.workunit.client.0.smithi032.stdout:5/893: chown d2/df/da9/dc1/d78/f94 3598077 1 2024-04-12T05:37:16.223 INFO:tasks.workunit.client.0.smithi032.stdout:1/669: write d5/de/d17/fae [1022645,89202] 0 2024-04-12T05:37:16.226 INFO:tasks.workunit.client.0.smithi032.stdout:2/767: chown dd/d19/d9d/d42/da7/dc5/cf8 59236 1 2024-04-12T05:37:16.227 INFO:tasks.workunit.client.0.smithi032.stdout:4/736: rename d4/d13/d44/la6 to d4/d87/d96/d2f/lfd 0 2024-04-12T05:37:16.231 INFO:tasks.workunit.client.0.smithi032.stdout:3/828: write d12/d24/f47 [1023747,25166] 0 2024-04-12T05:37:16.232 INFO:tasks.workunit.client.0.smithi032.stdout:4/737: chown d4/d87/d96/d4d/d78 2040 1 2024-04-12T05:37:16.232 INFO:tasks.workunit.client.0.smithi032.stdout:9/607: dwrite d9/d15/d1a/d31/d65/d80/fb4 [0,4194304] 0 2024-04-12T05:37:16.235 INFO:tasks.workunit.client.0.smithi032.stdout:7/765: dwrite d0/d13/f27 [0,4194304] 0 2024-04-12T05:37:16.236 INFO:tasks.workunit.client.0.smithi032.stdout:5/894: read d2/df/d31/d92/fc3 [1769125,24159] 0 2024-04-12T05:37:16.239 INFO:tasks.workunit.client.0.smithi032.stdout:2/768: creat dd/d19/d9d/d42/fff x:0 0 0 2024-04-12T05:37:16.244 INFO:tasks.workunit.client.0.smithi032.stdout:2/769: chown dd/d19 6 1 2024-04-12T05:37:16.244 INFO:tasks.workunit.client.0.smithi032.stdout:9/608: mknod d9/ccd 0 2024-04-12T05:37:16.247 INFO:tasks.workunit.client.0.smithi032.stdout:8/803: dwrite d0/db/dd/d11/d1b/d5f/f93 [0,4194304] 0 2024-04-12T05:37:16.252 INFO:tasks.workunit.client.0.smithi032.stdout:0/787: dwrite d2/f14 [0,4194304] 0 2024-04-12T05:37:16.254 INFO:tasks.workunit.client.0.smithi032.stdout:9/609: write d9/d15/d1a/d24/d27/f76 [169674,99565] 0 2024-04-12T05:37:16.259 INFO:tasks.workunit.client.0.smithi032.stdout:4/738: dwrite d4/d13/d75/fbd [0,4194304] 0 2024-04-12T05:37:16.259 INFO:tasks.workunit.client.0.smithi032.stdout:7/766: rename d0/dd/d51/dea/cf2 to d0/dff/d71/c106 0 2024-04-12T05:37:16.260 INFO:tasks.workunit.client.0.smithi032.stdout:1/670: dread d5/f13 [0,4194304] 0 2024-04-12T05:37:16.260 INFO:tasks.workunit.client.0.smithi032.stdout:0/788: creat d2/d29/dcc/f105 x:0 0 0 2024-04-12T05:37:16.263 INFO:tasks.workunit.client.0.smithi032.stdout:0/789: chown d2/d29/d77/db5/ffd 3372 1 2024-04-12T05:37:16.264 INFO:tasks.workunit.client.0.smithi032.stdout:8/804: truncate d0/db/f6b 88037 0 2024-04-12T05:37:16.265 INFO:tasks.workunit.client.0.smithi032.stdout:2/770: truncate dd/d19/d9d/d42/d4a/fcf 3857157 0 2024-04-12T05:37:16.265 INFO:tasks.workunit.client.0.smithi032.stdout:7/767: dread - d0/d13/d21/d7c/fb2 zero size 2024-04-12T05:37:16.273 INFO:tasks.workunit.client.0.smithi032.stdout:8/805: symlink d0/db/dd/d11/d1b/d22/d29/ded/l116 0 2024-04-12T05:37:16.274 INFO:tasks.workunit.client.0.smithi032.stdout:5/895: rename d2/d3c/d48/d4b to d2/df/da9/dc1/d78/d120 0 2024-04-12T05:37:16.275 INFO:tasks.workunit.client.0.smithi032.stdout:1/671: rename d5/de to d5/de/d17/d35/d61/d77/d8f/d9c/dcf 22 2024-04-12T05:37:16.276 INFO:tasks.workunit.client.0.smithi032.stdout:2/771: unlink f0 0 2024-04-12T05:37:16.279 INFO:tasks.workunit.client.0.smithi032.stdout:7/768: rename d0/de/df/c2f to d0/dd/d51/dea/d105/c107 0 2024-04-12T05:37:16.280 INFO:tasks.workunit.client.0.smithi032.stdout:9/610: dwrite d9/d15/d2c/d41/f96 [0,4194304] 0 2024-04-12T05:37:16.285 INFO:tasks.workunit.client.0.smithi032.stdout:7/769: truncate d0/dff/d4c/f103 360786 0 2024-04-12T05:37:16.285 INFO:tasks.workunit.client.0.smithi032.stdout:8/806: creat d0/db/dd/d11/d1b/d5f/daf/dc6/f117 x:0 0 0 2024-04-12T05:37:16.286 INFO:tasks.workunit.client.0.smithi032.stdout:1/672: creat d5/de/d17/d35/d61/d77/fd0 x:0 0 0 2024-04-12T05:37:16.288 INFO:tasks.workunit.client.0.smithi032.stdout:2/772: creat dd/d19/d9d/d42/d4a/f100 x:0 0 0 2024-04-12T05:37:16.289 INFO:tasks.workunit.client.0.smithi032.stdout:0/790: write d2/d8/d17/d1b/f98 [1042135,83142] 0 2024-04-12T05:37:16.291 INFO:tasks.workunit.client.0.smithi032.stdout:4/739: dwrite d4/d87/d96/d2f/d3b/f6b [0,4194304] 0 2024-04-12T05:37:16.294 INFO:tasks.workunit.client.0.smithi032.stdout:4/740: write d4/d87/d96/d2f/f70 [4762749,26384] 0 2024-04-12T05:37:16.295 INFO:tasks.workunit.client.0.smithi032.stdout:8/807: chown d0/db/d38/d63/c109 3711 1 2024-04-12T05:37:16.297 INFO:tasks.workunit.client.0.smithi032.stdout:4/741: dread - d4/d87/ff0 zero size 2024-04-12T05:37:16.297 INFO:tasks.workunit.client.0.smithi032.stdout:0/791: creat d2/d28/d87/d79/f106 x:0 0 0 2024-04-12T05:37:16.301 INFO:tasks.workunit.client.0.smithi032.stdout:1/673: rename d5/de/d17/d1b/l44 to d5/de/d17/d35/d61/d77/d8f/d9c/ld1 0 2024-04-12T05:37:16.301 INFO:tasks.workunit.client.0.smithi032.stdout:9/611: dwrite d9/d15/d1a/d1d/faf [0,4194304] 0 2024-04-12T05:37:16.302 INFO:tasks.workunit.client.0.smithi032.stdout:7/770: truncate d0/d13/f4b 534772 0 2024-04-12T05:37:16.306 INFO:tasks.workunit.client.0.smithi032.stdout:7/771: readlink d0/de/d47/lb1 0 2024-04-12T05:37:16.308 INFO:tasks.workunit.client.0.smithi032.stdout:4/742: creat d4/d87/d96/d4d/db5/dcf/ffe x:0 0 0 2024-04-12T05:37:16.310 INFO:tasks.workunit.client.0.smithi032.stdout:7/772: write d0/de/df/d29/fdb [2010103,47502] 0 2024-04-12T05:37:16.312 INFO:tasks.workunit.client.0.smithi032.stdout:5/896: dwrite d2/df/da9/dc1/d6c/df3/f117 [0,4194304] 0 2024-04-12T05:37:16.313 INFO:tasks.workunit.client.0.smithi032.stdout:4/743: dread d4/d13/d19/f2a [0,4194304] 0 2024-04-12T05:37:16.315 INFO:tasks.workunit.client.0.smithi032.stdout:7/773: rmdir d0/d34/dbe/dbb/de8/df1 39 2024-04-12T05:37:16.321 INFO:tasks.workunit.client.0.smithi032.stdout:9/612: truncate d9/d15/d2c/d41/f59 1494680 0 2024-04-12T05:37:16.322 INFO:tasks.workunit.client.1.smithi067.stdout:9/126: sync 2024-04-12T05:37:16.322 INFO:tasks.workunit.client.0.smithi032.stdout:4/744: unlink d4/d13/fcb 0 2024-04-12T05:37:16.324 INFO:tasks.workunit.client.1.smithi067.stdout:9/127: creat d1/d12/d15/f1d x:0 0 0 2024-04-12T05:37:16.328 INFO:tasks.workunit.client.0.smithi032.stdout:9/613: mkdir d9/d15/d1a/d24/d27/dce 0 2024-04-12T05:37:16.328 INFO:tasks.workunit.client.1.smithi067.stdout:9/128: rename d1/f2 to d1/d13/f1e 0 2024-04-12T05:37:16.330 INFO:tasks.workunit.client.0.smithi032.stdout:4/745: mkdir d4/d87/d96/d2f/d7c/dff 0 2024-04-12T05:37:16.331 INFO:tasks.workunit.client.0.smithi032.stdout:8/808: dwrite d0/db/dd/d11/d1b/d22/d54/da8/fc7 [0,4194304] 0 2024-04-12T05:37:16.331 INFO:tasks.workunit.client.0.smithi032.stdout:2/773: dwrite dd/d19/d9d/d1c/d64/f98 [0,4194304] 0 2024-04-12T05:37:16.333 INFO:tasks.workunit.client.0.smithi032.stdout:0/792: dwrite d2/d29/f9a [0,4194304] 0 2024-04-12T05:37:16.333 INFO:tasks.workunit.client.0.smithi032.stdout:9/614: chown d9/d32/d6e/f8e 48065574 1 2024-04-12T05:37:16.334 INFO:tasks.workunit.client.0.smithi032.stdout:4/746: fsync d4/d87/d96/d2f/fc1 0 2024-04-12T05:37:16.335 INFO:tasks.workunit.client.0.smithi032.stdout:2/774: dread - dd/d19/d9d/d1c/d64/d36/ffc zero size 2024-04-12T05:37:16.336 INFO:tasks.workunit.client.0.smithi032.stdout:9/615: chown d9/d15/d1a/d31/fc9 46658 1 2024-04-12T05:37:16.337 INFO:tasks.workunit.client.0.smithi032.stdout:8/809: symlink d0/db/d38/d63/d77/dd4/l118 0 2024-04-12T05:37:16.339 INFO:tasks.workunit.client.0.smithi032.stdout:9/616: mkdir d9/d15/d1a/d69/dcf 0 2024-04-12T05:37:16.340 INFO:tasks.workunit.client.0.smithi032.stdout:4/747: mkdir d4/d13/dc9/d100 0 2024-04-12T05:37:16.341 INFO:tasks.workunit.client.0.smithi032.stdout:8/810: mknod d0/db/dd/d11/d1b/d5f/deb/c119 0 2024-04-12T05:37:16.342 INFO:tasks.workunit.client.0.smithi032.stdout:2/775: unlink dd/d19/d9d/d42/d4a/da1/dd5/cfd 0 2024-04-12T05:37:16.343 INFO:tasks.workunit.client.0.smithi032.stdout:8/811: chown d0/db/dd/d11/d1b/d22/d35/l66 2 1 2024-04-12T05:37:16.343 INFO:tasks.workunit.client.0.smithi032.stdout:9/617: creat d9/d15/d1a/d24/d27/fd0 x:0 0 0 2024-04-12T05:37:16.353 INFO:tasks.workunit.client.0.smithi032.stdout:8/812: dread d0/db/dd/d11/d1b/d22/d35/ff6 [0,4194304] 0 2024-04-12T05:37:16.364 INFO:tasks.workunit.client.0.smithi032.stdout:8/813: rename d0/d72/f7a to d0/db/f11a 0 2024-04-12T05:37:16.366 INFO:tasks.workunit.client.0.smithi032.stdout:0/793: write d2/d29/d8a/d99/fbf [3038325,69185] 0 2024-04-12T05:37:16.373 INFO:tasks.workunit.client.0.smithi032.stdout:4/748: truncate d4/d87/d96/d2f/d3b/f3c 1351806 0 2024-04-12T05:37:16.375 INFO:tasks.workunit.client.0.smithi032.stdout:2/776: dwrite dd/d19/d9d/fe3 [0,4194304] 0 2024-04-12T05:37:16.375 INFO:tasks.workunit.client.0.smithi032.stdout:0/794: rename d2/d29/d8a/ld5 to d2/d8/df/l107 0 2024-04-12T05:37:16.376 INFO:tasks.workunit.client.0.smithi032.stdout:2/777: chown dd/d19/d82/c94 243 1 2024-04-12T05:37:16.378 INFO:tasks.workunit.client.0.smithi032.stdout:8/814: link d0/db/dd/f5c d0/db/dd/d11/d1b/d22/d35/f11b 0 2024-04-12T05:37:16.378 INFO:tasks.workunit.client.0.smithi032.stdout:0/795: write d2/d8/f6c [3992313,110537] 0 2024-04-12T05:37:16.379 INFO:tasks.workunit.client.0.smithi032.stdout:4/749: dread d4/d87/d96/d4d/f63 [0,4194304] 0 2024-04-12T05:37:16.380 INFO:tasks.workunit.client.0.smithi032.stdout:4/750: stat d4/d13/d44/fd6 0 2024-04-12T05:37:16.385 INFO:tasks.workunit.client.0.smithi032.stdout:2/778: fdatasync dd/d19/d9d/f73 0 2024-04-12T05:37:16.386 INFO:tasks.workunit.client.0.smithi032.stdout:0/796: truncate d2/fd9 964533 0 2024-04-12T05:37:16.386 INFO:tasks.workunit.client.0.smithi032.stdout:8/815: dwrite d0/db/dd/d11/f2c [4194304,4194304] 0 2024-04-12T05:37:16.390 INFO:tasks.workunit.client.0.smithi032.stdout:4/751: rename d4/d87/d96/d4d/c69 to d4/d87/d96/d2f/d3b/da2/dec/c101 0 2024-04-12T05:37:16.390 INFO:tasks.workunit.client.0.smithi032.stdout:0/797: rmdir d2/d29/d8a 39 2024-04-12T05:37:16.391 INFO:tasks.workunit.client.0.smithi032.stdout:8/816: mknod d0/db/d7d/dfb/c11c 0 2024-04-12T05:37:16.392 INFO:tasks.workunit.client.0.smithi032.stdout:4/752: read - d4/d87/d96/d4d/db5/dcf/ffe zero size 2024-04-12T05:37:16.394 INFO:tasks.workunit.client.0.smithi032.stdout:0/798: unlink d2/d8/c8c 0 2024-04-12T05:37:16.397 INFO:tasks.workunit.client.0.smithi032.stdout:8/817: mkdir d0/db/dd/d11/d1b/d22/d54/d11d 0 2024-04-12T05:37:16.399 INFO:tasks.workunit.client.0.smithi032.stdout:0/799: truncate d2/d8/d17/d1b/d20/f6d 655085 0 2024-04-12T05:37:16.408 INFO:tasks.workunit.client.0.smithi032.stdout:0/800: creat d2/d29/d8a/f108 x:0 0 0 2024-04-12T05:37:16.412 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:16 smithi067 ceph-mon[49010]: pgmap v269: 65 pgs: 65 active+clean; 1.8 GiB data, 6.9 GiB used, 529 GiB / 536 GiB avail; 75 MiB/s rd, 204 MiB/s wr, 410 op/s 2024-04-12T05:37:16.412 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:16 smithi067 ceph-mon[49010]: mgrmap e20: smithi032.hrwxrh(active, since 7m) 2024-04-12T05:37:16.419 INFO:tasks.workunit.client.0.smithi032.stdout:2/779: dwrite dd/f27 [0,4194304] 0 2024-04-12T05:37:16.421 INFO:tasks.workunit.client.0.smithi032.stdout:4/753: dwrite d4/d13/d75/fd8 [0,4194304] 0 2024-04-12T05:37:16.425 INFO:tasks.workunit.client.0.smithi032.stdout:2/780: fdatasync dd/d19/d20/f4d 0 2024-04-12T05:37:16.426 INFO:tasks.workunit.client.0.smithi032.stdout:8/818: dwrite d0/db/dd/f27 [4194304,4194304] 0 2024-04-12T05:37:16.433 INFO:tasks.workunit.client.0.smithi032.stdout:4/754: mkdir d4/d87/d96/d2f/d7c/d102 0 2024-04-12T05:37:16.434 INFO:tasks.workunit.client.0.smithi032.stdout:2/781: dread - dd/d19/d20/f4b zero size 2024-04-12T05:37:16.434 INFO:tasks.workunit.client.0.smithi032.stdout:8/819: read d0/d36/d84/db3/fe9 [312964,494] 0 2024-04-12T05:37:16.437 INFO:tasks.workunit.client.0.smithi032.stdout:4/755: creat d4/d87/d96/d4d/db5/f103 x:0 0 0 2024-04-12T05:37:16.438 INFO:tasks.workunit.client.0.smithi032.stdout:8/820: creat d0/df4/f11e x:0 0 0 2024-04-12T05:37:16.439 INFO:tasks.workunit.client.0.smithi032.stdout:8/821: chown d0/db/dd/ddd 866125 1 2024-04-12T05:37:16.441 INFO:tasks.workunit.client.0.smithi032.stdout:4/756: dread d4/f98 [0,4194304] 0 2024-04-12T05:37:16.441 INFO:tasks.workunit.client.0.smithi032.stdout:8/822: unlink d0/db/dd/fcd 0 2024-04-12T05:37:16.446 INFO:tasks.workunit.client.0.smithi032.stdout:4/757: link d4/d13/d44/f91 d4/d13/f104 0 2024-04-12T05:37:16.446 INFO:tasks.workunit.client.0.smithi032.stdout:4/758: chown d4/d87/d96 4 1 2024-04-12T05:37:16.447 INFO:tasks.workunit.client.0.smithi032.stdout:4/759: readlink d4/d87/d96/d2f/d7c/ld1 0 2024-04-12T05:37:16.448 INFO:tasks.workunit.client.0.smithi032.stdout:4/760: write d4/d13/d19/fa1 [1329823,102804] 0 2024-04-12T05:37:16.449 INFO:tasks.workunit.client.0.smithi032.stdout:4/761: chown d4/d87 7091 1 2024-04-12T05:37:16.452 INFO:tasks.workunit.client.0.smithi032.stdout:8/823: dread d0/db/dd/d11/d1b/d5f/daf/fa2 [0,4194304] 0 2024-04-12T05:37:16.455 INFO:tasks.workunit.client.0.smithi032.stdout:4/762: dread d4/d87/d96/d2f/f4e [0,4194304] 0 2024-04-12T05:37:16.456 INFO:tasks.workunit.client.0.smithi032.stdout:8/824: truncate d0/db/dd/d11/d1b/d22/d54/da8/fce 1412360 0 2024-04-12T05:37:16.457 INFO:tasks.workunit.client.0.smithi032.stdout:8/825: mkdir d0/db/d7d/d11f 0 2024-04-12T05:37:16.468 INFO:tasks.workunit.client.0.smithi032.stdout:4/763: dwrite d4/d13/f43 [0,4194304] 0 2024-04-12T05:37:16.468 INFO:tasks.workunit.client.0.smithi032.stdout:8/826: dwrite d0/d36/d84/f92 [0,4194304] 0 2024-04-12T05:37:16.470 INFO:tasks.workunit.client.0.smithi032.stdout:4/764: truncate d4/d13/d75/fbd 4933131 0 2024-04-12T05:37:16.471 INFO:tasks.workunit.client.0.smithi032.stdout:4/765: fdatasync d4/d13/d19/fa1 0 2024-04-12T05:37:16.474 INFO:tasks.workunit.client.0.smithi032.stdout:8/827: fdatasync d0/db/dd/d11/d1b/d22/d29/ded/f102 0 2024-04-12T05:37:16.476 INFO:tasks.workunit.client.0.smithi032.stdout:3/829: sync 2024-04-12T05:37:16.477 INFO:tasks.workunit.client.0.smithi032.stdout:8/828: symlink d0/db/dd/d11/d1b/d5f/l120 0 2024-04-12T05:37:16.478 INFO:tasks.workunit.client.0.smithi032.stdout:8/829: chown d0/db/dd/d11/d1b/l49 5282 1 2024-04-12T05:37:16.480 INFO:tasks.workunit.client.0.smithi032.stdout:8/830: stat d0/f3 0 2024-04-12T05:37:16.483 INFO:tasks.workunit.client.0.smithi032.stdout:8/831: chown d0/db/dd/d11/d1b/d22/d51/fa9 5 1 2024-04-12T05:37:16.485 INFO:tasks.workunit.client.0.smithi032.stdout:4/766: dwrite d4/d87/d96/f59 [0,4194304] 0 2024-04-12T05:37:16.488 INFO:tasks.workunit.client.0.smithi032.stdout:8/832: fdatasync d0/db/d38/d63/d77/f97 0 2024-04-12T05:37:16.492 INFO:tasks.workunit.client.0.smithi032.stdout:4/767: rename d4/d13/f1c to d4/d13/dc9/d100/f105 0 2024-04-12T05:37:16.493 INFO:tasks.workunit.client.0.smithi032.stdout:8/833: symlink d0/db/dd/d11/d1b/d5f/deb/l121 0 2024-04-12T05:37:16.497 INFO:tasks.workunit.client.0.smithi032.stdout:8/834: rename d0/f3 to d0/db/dd/d11/d1b/d22/d54/f122 0 2024-04-12T05:37:16.504 INFO:tasks.workunit.client.0.smithi032.stdout:4/768: dread d4/d87/d96/f42 [0,4194304] 0 2024-04-12T05:37:16.505 INFO:tasks.workunit.client.0.smithi032.stdout:4/769: chown d4/d13 14 1 2024-04-12T05:37:16.511 INFO:tasks.workunit.client.0.smithi032.stdout:4/770: dwrite d4/d13/d44/f5a [0,4194304] 0 2024-04-12T05:37:16.511 INFO:tasks.workunit.client.0.smithi032.stdout:4/771: readlink d4/d13/d75/l8c 0 2024-04-12T05:37:16.512 INFO:tasks.workunit.client.0.smithi032.stdout:4/772: write d4/d13/d19/fdf [831041,26946] 0 2024-04-12T05:37:16.515 INFO:tasks.workunit.client.0.smithi032.stdout:8/835: dread d0/db/dd/d11/d1b/d22/fb4 [0,4194304] 0 2024-04-12T05:37:16.517 INFO:tasks.workunit.client.0.smithi032.stdout:4/773: creat d4/d13/f106 x:0 0 0 2024-04-12T05:37:16.518 INFO:tasks.workunit.client.0.smithi032.stdout:8/836: dread - d0/d36/d84/db3/fe1 zero size 2024-04-12T05:37:16.521 INFO:tasks.workunit.client.0.smithi032.stdout:4/774: unlink d4/d87/d96/d4d/db5/dcf/ffe 0 2024-04-12T05:37:16.522 INFO:tasks.workunit.client.0.smithi032.stdout:4/775: readlink d4/d13/d19/l27 0 2024-04-12T05:37:16.524 INFO:tasks.workunit.client.0.smithi032.stdout:8/837: dwrite d0/db/dd/d11/d1b/ff5 [0,4194304] 0 2024-04-12T05:37:16.524 INFO:tasks.workunit.client.0.smithi032.stdout:4/776: chown d4/d87/d96/d2f/d6e/c72 13 1 2024-04-12T05:37:16.528 INFO:tasks.workunit.client.0.smithi032.stdout:8/838: creat d0/db/d38/d63/d77/dd4/f123 x:0 0 0 2024-04-12T05:37:16.528 INFO:tasks.workunit.client.0.smithi032.stdout:8/839: stat d0/d58/cd1 0 2024-04-12T05:37:16.533 INFO:tasks.workunit.client.0.smithi032.stdout:8/840: creat d0/db/dd/d11/d1b/d5f/daf/dc6/f124 x:0 0 0 2024-04-12T05:37:16.533 INFO:tasks.workunit.client.0.smithi032.stdout:4/777: link d4/d87/d96/d2f/d7c/ld1 d4/d87/l107 0 2024-04-12T05:37:16.535 INFO:tasks.workunit.client.0.smithi032.stdout:8/841: creat d0/db/dd/d11/f125 x:0 0 0 2024-04-12T05:37:16.535 INFO:tasks.workunit.client.0.smithi032.stdout:4/778: creat d4/d87/d96/d4d/db5/f108 x:0 0 0 2024-04-12T05:37:16.538 INFO:tasks.workunit.client.0.smithi032.stdout:4/779: symlink d4/d87/d96/d2f/d9a/l109 0 2024-04-12T05:37:16.540 INFO:tasks.workunit.client.0.smithi032.stdout:4/780: mkdir d4/d87/d96/d2f/d3b/da2/d10a 0 2024-04-12T05:37:16.545 INFO:tasks.workunit.client.0.smithi032.stdout:4/781: write d4/d87/d96/d2f/d3b/f3c [733687,49873] 0 2024-04-12T05:37:16.547 INFO:tasks.workunit.client.0.smithi032.stdout:8/842: dwrite d0/db/dd/d11/d1b/f78 [0,4194304] 0 2024-04-12T05:37:16.549 INFO:tasks.workunit.client.0.smithi032.stdout:8/843: fsync d0/d36/d84/f92 0 2024-04-12T05:37:16.552 INFO:tasks.workunit.client.0.smithi032.stdout:8/844: unlink d0/d36/de2/lfc 0 2024-04-12T05:37:16.554 INFO:tasks.workunit.client.0.smithi032.stdout:4/782: dwrite d4/d13/f106 [0,4194304] 0 2024-04-12T05:37:16.555 INFO:tasks.workunit.client.0.smithi032.stdout:4/783: chown d4/d87/d96/d2f/d9a/fa7 45895 1 2024-04-12T05:37:16.557 INFO:tasks.workunit.client.0.smithi032.stdout:8/845: mknod d0/db/dd/d11/d1b/d22/ddc/c126 0 2024-04-12T05:37:16.563 INFO:tasks.workunit.client.0.smithi032.stdout:4/784: rename d4/d87/d96/d32/l65 to d4/d87/d96/d2f/l10b 0 2024-04-12T05:37:16.569 INFO:tasks.workunit.client.0.smithi032.stdout:8/846: dwrite d0/d36/d84/dad/ff9 [0,4194304] 0 2024-04-12T05:37:16.570 INFO:tasks.workunit.client.0.smithi032.stdout:4/785: fsync d4/d13/f106 0 2024-04-12T05:37:16.581 INFO:tasks.workunit.client.0.smithi032.stdout:8/847: dread d0/d72/faa [0,4194304] 0 2024-04-12T05:37:16.585 INFO:tasks.workunit.client.0.smithi032.stdout:8/848: dread d0/db/dd/d11/d1b/f6c [0,4194304] 0 2024-04-12T05:37:16.587 INFO:tasks.workunit.client.0.smithi032.stdout:8/849: symlink d0/db/d7d/d111/l127 0 2024-04-12T05:37:16.588 INFO:tasks.workunit.client.0.smithi032.stdout:4/786: dread d4/d13/d19/fa8 [0,4194304] 0 2024-04-12T05:37:16.589 INFO:tasks.workunit.client.0.smithi032.stdout:4/787: rmdir d4/d87/d96/d2f/d7c/dc2 39 2024-04-12T05:37:16.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:16 smithi032 ceph-mon[39731]: pgmap v269: 65 pgs: 65 active+clean; 1.8 GiB data, 6.9 GiB used, 529 GiB / 536 GiB avail; 75 MiB/s rd, 204 MiB/s wr, 410 op/s 2024-04-12T05:37:16.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:16 smithi032 ceph-mon[39731]: mgrmap e20: smithi032.hrwxrh(active, since 7m) 2024-04-12T05:37:16.590 INFO:tasks.workunit.client.0.smithi032.stdout:4/788: chown d4/d13/d44/f91 41922 1 2024-04-12T05:37:16.593 INFO:tasks.workunit.client.0.smithi032.stdout:4/789: rmdir d4/d87 39 2024-04-12T05:37:16.595 INFO:tasks.workunit.client.0.smithi032.stdout:8/850: truncate d0/db/dd/f17 4156217 0 2024-04-12T05:37:16.596 INFO:tasks.workunit.client.0.smithi032.stdout:4/790: read d4/d87/d96/d4d/d78/f64 [1495697,32264] 0 2024-04-12T05:37:16.596 INFO:tasks.workunit.client.0.smithi032.stdout:8/851: stat d0/db/dd/d11/d1b/d22/ddc/c126 0 2024-04-12T05:37:16.602 INFO:tasks.workunit.client.0.smithi032.stdout:4/791: write d4/d87/d96/d2f/d6e/fcc [660150,109378] 0 2024-04-12T05:37:16.607 INFO:tasks.workunit.client.0.smithi032.stdout:8/852: dwrite d0/db/dd/d11/d1b/d5f/deb/f50 [0,4194304] 0 2024-04-12T05:37:16.609 INFO:tasks.workunit.client.0.smithi032.stdout:4/792: dread d4/d87/d96/d2f/d3b/f3c [0,4194304] 0 2024-04-12T05:37:16.610 INFO:tasks.workunit.client.0.smithi032.stdout:8/853: mknod d0/db/dd/c128 0 2024-04-12T05:37:16.612 INFO:tasks.workunit.client.0.smithi032.stdout:8/854: write d0/db/dd/d11/d1b/d96/fef [519962,47550] 0 2024-04-12T05:37:16.616 INFO:tasks.workunit.client.0.smithi032.stdout:8/855: mknod d0/db/dd/d11/d1b/d22/d54/d11d/c129 0 2024-04-12T05:37:16.617 INFO:tasks.workunit.client.0.smithi032.stdout:4/793: dread d4/d13/d44/fd6 [0,4194304] 0 2024-04-12T05:37:16.618 INFO:tasks.workunit.client.0.smithi032.stdout:8/856: mkdir d0/d36/d12a 0 2024-04-12T05:37:16.620 INFO:tasks.workunit.client.0.smithi032.stdout:4/794: dread - d4/d87/d96/d4d/fd3 zero size 2024-04-12T05:37:16.622 INFO:tasks.workunit.client.0.smithi032.stdout:8/857: dread d0/d36/f4a [0,4194304] 0 2024-04-12T05:37:16.622 INFO:tasks.workunit.client.0.smithi032.stdout:4/795: truncate d4/fb7 814991 0 2024-04-12T05:37:16.624 INFO:tasks.workunit.client.0.smithi032.stdout:8/858: write d0/db/dd/fec [4354042,50912] 0 2024-04-12T05:37:16.625 INFO:tasks.workunit.client.0.smithi032.stdout:1/674: sync 2024-04-12T05:37:16.625 INFO:tasks.workunit.client.0.smithi032.stdout:5/897: sync 2024-04-12T05:37:16.625 INFO:tasks.workunit.client.0.smithi032.stdout:7/774: sync 2024-04-12T05:37:16.626 INFO:tasks.workunit.client.0.smithi032.stdout:4/796: creat d4/d87/d96/d4d/db5/dcf/de3/f10c x:0 0 0 2024-04-12T05:37:16.627 INFO:tasks.workunit.client.0.smithi032.stdout:8/859: mkdir d0/db/dd/d12b 0 2024-04-12T05:37:16.627 INFO:tasks.workunit.client.0.smithi032.stdout:7/775: readlink d0/de/df/d29/lc8 0 2024-04-12T05:37:16.628 INFO:tasks.workunit.client.0.smithi032.stdout:4/797: dread - d4/d87/d96/d2f/d6e/fae zero size 2024-04-12T05:37:16.630 INFO:tasks.workunit.client.0.smithi032.stdout:5/898: creat d2/d3c/d5b/dd4/df2/f121 x:0 0 0 2024-04-12T05:37:16.631 INFO:tasks.workunit.client.0.smithi032.stdout:8/860: creat d0/db/dd/d11/d1b/d5f/deb/f12c x:0 0 0 2024-04-12T05:37:16.631 INFO:tasks.workunit.client.0.smithi032.stdout:7/776: truncate d0/d34/dbe/f5c 16428 0 2024-04-12T05:37:16.632 INFO:tasks.workunit.client.0.smithi032.stdout:4/798: dread d4/d87/d96/d2f/d3b/f3c [0,4194304] 0 2024-04-12T05:37:16.632 INFO:tasks.workunit.client.0.smithi032.stdout:1/675: getdents d5/de/d17/d35/d3b 0 2024-04-12T05:37:16.633 INFO:tasks.workunit.client.0.smithi032.stdout:8/861: chown d0/db/dd/d11/d1b/l49 0 1 2024-04-12T05:37:16.635 INFO:tasks.workunit.client.0.smithi032.stdout:1/676: chown d5/f30 542421 1 2024-04-12T05:37:16.636 INFO:tasks.workunit.client.0.smithi032.stdout:8/862: creat d0/db/d38/d63/d77/dd4/f12d x:0 0 0 2024-04-12T05:37:16.640 INFO:tasks.workunit.client.0.smithi032.stdout:8/863: dread d0/db/dd/d11/d1b/d5f/deb/f90 [0,4194304] 0 2024-04-12T05:37:16.700 INFO:tasks.workunit.client.0.smithi032.stdout:5/899: write d2/df/d31/d92/fc3 [2783888,46490] 0 2024-04-12T05:37:16.701 INFO:tasks.workunit.client.0.smithi032.stdout:7/777: write d0/d34/dbe/fa6 [812179,76185] 0 2024-04-12T05:37:16.701 INFO:tasks.workunit.client.0.smithi032.stdout:1/677: write d5/de/d2a/f2b [112711,107111] 0 2024-04-12T05:37:16.703 INFO:tasks.workunit.client.0.smithi032.stdout:7/778: chown d0/d13/d21/cac 0 1 2024-04-12T05:37:16.703 INFO:tasks.workunit.client.0.smithi032.stdout:1/678: chown d5/de/d17/fa7 265321412 1 2024-04-12T05:37:16.703 INFO:tasks.workunit.client.0.smithi032.stdout:5/900: write d2/df/d25/d50/f76 [2318397,94272] 0 2024-04-12T05:37:16.705 INFO:tasks.workunit.client.0.smithi032.stdout:5/901: write d2/df/d31/fcf [1672813,34344] 0 2024-04-12T05:37:16.708 INFO:tasks.workunit.client.0.smithi032.stdout:5/902: read d2/df/d25/d1e/f40 [331267,67280] 0 2024-04-12T05:37:16.708 INFO:tasks.workunit.client.0.smithi032.stdout:1/679: creat d5/de/d17/d35/d61/d77/d8f/fd2 x:0 0 0 2024-04-12T05:37:16.709 INFO:tasks.workunit.client.0.smithi032.stdout:8/864: write d0/db/dd/d11/d1b/d5f/deb/f90 [19069,55815] 0 2024-04-12T05:37:16.716 INFO:tasks.workunit.client.0.smithi032.stdout:7/779: dwrite d0/d13/f64 [0,4194304] 0 2024-04-12T05:37:16.717 INFO:tasks.workunit.client.0.smithi032.stdout:5/903: dwrite d2/dd/d5d/fce [0,4194304] 0 2024-04-12T05:37:16.717 INFO:tasks.workunit.client.0.smithi032.stdout:8/865: truncate d0/db/dd/f75 3001008 0 2024-04-12T05:37:16.718 INFO:tasks.workunit.client.0.smithi032.stdout:7/780: chown d0/de/df/d29/f46 1589372 1 2024-04-12T05:37:16.719 INFO:tasks.workunit.client.0.smithi032.stdout:8/866: chown d0/db/d38/l80 995085 1 2024-04-12T05:37:16.722 INFO:tasks.workunit.client.0.smithi032.stdout:7/781: truncate d0/dff/d4c/f4d 3627726 0 2024-04-12T05:37:16.727 INFO:tasks.workunit.client.0.smithi032.stdout:7/782: rename d0/dff/l69 to d0/dff/d4c/l108 0 2024-04-12T05:37:16.729 INFO:tasks.workunit.client.0.smithi032.stdout:7/783: chown d0/dff 320 1 2024-04-12T05:37:16.731 INFO:tasks.workunit.client.0.smithi032.stdout:7/784: chown d0/d13/f27 1071 1 2024-04-12T05:37:16.731 INFO:tasks.workunit.client.0.smithi032.stdout:5/904: dread d2/df/d25/f2c [0,4194304] 0 2024-04-12T05:37:16.731 INFO:tasks.workunit.client.0.smithi032.stdout:8/867: dwrite d0/db/d38/d63/d77/dd4/f123 [0,4194304] 0 2024-04-12T05:37:16.737 INFO:tasks.workunit.client.0.smithi032.stdout:5/905: dread d2/dd/d5d/faf [0,4194304] 0 2024-04-12T05:37:16.738 INFO:tasks.workunit.client.0.smithi032.stdout:8/868: unlink d0/d36/l37 0 2024-04-12T05:37:16.740 INFO:tasks.workunit.client.0.smithi032.stdout:5/906: mknod d2/df/d31/d92/c122 0 2024-04-12T05:37:16.745 INFO:tasks.workunit.client.0.smithi032.stdout:7/785: link d0/d10/d70/ce6 d0/dff/d71/c109 0 2024-04-12T05:37:16.748 INFO:tasks.workunit.client.0.smithi032.stdout:8/869: dwrite d0/db/dd/d11/d1b/d22/d35/ff6 [0,4194304] 0 2024-04-12T05:37:16.752 INFO:tasks.workunit.client.0.smithi032.stdout:9/618: sync 2024-04-12T05:37:16.753 INFO:tasks.workunit.client.0.smithi032.stdout:3/830: sync 2024-04-12T05:37:16.753 INFO:tasks.workunit.client.0.smithi032.stdout:2/782: sync 2024-04-12T05:37:16.753 INFO:tasks.workunit.client.0.smithi032.stdout:0/801: sync 2024-04-12T05:37:16.754 INFO:tasks.workunit.client.0.smithi032.stdout:3/831: chown d12/d2f/d89/df0/l25 4 1 2024-04-12T05:37:16.755 INFO:tasks.workunit.client.0.smithi032.stdout:9/619: write d9/d15/d2c/dab/fc5 [954695,38729] 0 2024-04-12T05:37:16.758 INFO:tasks.workunit.client.0.smithi032.stdout:3/832: symlink d12/d2f/d89/df0/dcd/l10d 0 2024-04-12T05:37:16.758 INFO:tasks.workunit.client.0.smithi032.stdout:9/620: mknod d9/d15/d1a/d31/d65/cd1 0 2024-04-12T05:37:16.765 INFO:tasks.workunit.client.0.smithi032.stdout:3/833: mkdir d12/d2f/d89/df0/d3a/d63/da6/dac/d10e 0 2024-04-12T05:37:16.766 INFO:tasks.workunit.client.0.smithi032.stdout:8/870: dwrite d0/db/d7d/fb9 [0,4194304] 0 2024-04-12T05:37:16.767 INFO:tasks.workunit.client.0.smithi032.stdout:9/621: rename d9/d15/d1a/d31 to d9/d15/d1a/dd2 0 2024-04-12T05:37:16.767 INFO:tasks.workunit.client.0.smithi032.stdout:7/786: dread d0/dff/f4e [0,4194304] 0 2024-04-12T05:37:16.769 INFO:tasks.workunit.client.0.smithi032.stdout:0/802: dwrite d2/d29/f47 [8388608,4194304] 0 2024-04-12T05:37:16.771 INFO:tasks.workunit.client.1.smithi067.stdout:5/303: sync 2024-04-12T05:37:16.771 INFO:tasks.workunit.client.1.smithi067.stdout:8/133: sync 2024-04-12T05:37:16.771 INFO:tasks.workunit.client.1.smithi067.stdout:1/187: sync 2024-04-12T05:37:16.771 INFO:tasks.workunit.client.1.smithi067.stdout:6/175: sync 2024-04-12T05:37:16.772 INFO:tasks.workunit.client.1.smithi067.stdout:4/83: sync 2024-04-12T05:37:16.772 INFO:tasks.workunit.client.1.smithi067.stdout:0/254: sync 2024-04-12T05:37:16.772 INFO:tasks.workunit.client.1.smithi067.stdout:7/173: sync 2024-04-12T05:37:16.772 INFO:tasks.workunit.client.1.smithi067.stdout:3/150: sync 2024-04-12T05:37:16.772 INFO:tasks.workunit.client.1.smithi067.stdout:2/179: sync 2024-04-12T05:37:16.772 INFO:tasks.workunit.client.1.smithi067.stdout:9/129: sync 2024-04-12T05:37:16.775 INFO:tasks.workunit.client.1.smithi067.stdout:9/130: dread - d1/d12/d15/f1d zero size 2024-04-12T05:37:16.776 INFO:tasks.workunit.client.1.smithi067.stdout:0/255: write dd/f18 [8088733,49225] 0 2024-04-12T05:37:16.776 INFO:tasks.workunit.client.1.smithi067.stdout:6/176: chown dd/d21/d23 195452 1 2024-04-12T05:37:16.778 INFO:tasks.workunit.client.1.smithi067.stdout:5/304: write d0/d3a/f4c [1848189,53377] 0 2024-04-12T05:37:16.779 INFO:tasks.workunit.client.1.smithi067.stdout:9/131: write d1/d12/d16/f18 [480218,74895] 0 2024-04-12T05:37:16.781 INFO:tasks.workunit.client.1.smithi067.stdout:6/177: read - dd/f20 zero size 2024-04-12T05:37:16.781 INFO:tasks.workunit.client.0.smithi032.stdout:0/803: truncate d2/d8/d17/f6a 551026 0 2024-04-12T05:37:16.782 INFO:tasks.workunit.client.0.smithi032.stdout:7/787: read d0/de/df/d29/f46 [3144876,6491] 0 2024-04-12T05:37:16.782 INFO:tasks.workunit.client.1.smithi067.stdout:6/178: read - dd/d21/f24 zero size 2024-04-12T05:37:16.784 INFO:tasks.workunit.client.1.smithi067.stdout:4/84: fsync f7 0 2024-04-12T05:37:16.785 INFO:tasks.workunit.client.0.smithi032.stdout:8/871: dwrite d0/f2f [4194304,4194304] 0 2024-04-12T05:37:16.785 INFO:tasks.workunit.client.0.smithi032.stdout:3/834: dwrite d12/d2f/d89/df0/dcb/ffb [4194304,4194304] 0 2024-04-12T05:37:16.786 INFO:tasks.workunit.client.1.smithi067.stdout:0/256: unlink dd/d40/f4f 0 2024-04-12T05:37:16.787 INFO:tasks.workunit.client.0.smithi032.stdout:3/835: write d12/daa/f10c [342950,61363] 0 2024-04-12T05:37:16.787 INFO:tasks.workunit.client.1.smithi067.stdout:9/132: dread d1/d12/f14 [0,4194304] 0 2024-04-12T05:37:16.788 INFO:tasks.workunit.client.1.smithi067.stdout:0/257: dread - dd/d12/f3e zero size 2024-04-12T05:37:16.789 INFO:tasks.workunit.client.1.smithi067.stdout:0/258: chown c5 48 1 2024-04-12T05:37:16.789 INFO:tasks.workunit.client.1.smithi067.stdout:0/259: stat dd/d40/f44 0 2024-04-12T05:37:16.797 INFO:tasks.workunit.client.0.smithi032.stdout:7/788: mknod d0/d34/dbe/dbb/de8/df1/c10a 0 2024-04-12T05:37:16.797 INFO:tasks.workunit.client.0.smithi032.stdout:8/872: dwrite d0/f2f [4194304,4194304] 0 2024-04-12T05:37:16.798 INFO:tasks.workunit.client.1.smithi067.stdout:7/174: mknod d0/d2/dd/c33 0 2024-04-12T05:37:16.798 INFO:tasks.workunit.client.1.smithi067.stdout:2/180: rename d5/l10 to d5/da/de/d39/l3f 0 2024-04-12T05:37:16.798 INFO:tasks.workunit.client.1.smithi067.stdout:2/181: readlink d5/l21 0 2024-04-12T05:37:16.799 INFO:tasks.workunit.client.1.smithi067.stdout:4/85: creat d8/d14/d15/f1b x:0 0 0 2024-04-12T05:37:16.801 INFO:tasks.workunit.client.0.smithi032.stdout:3/836: dwrite d12/d2f/d89/fff [0,4194304] 0 2024-04-12T05:37:16.802 INFO:tasks.workunit.client.1.smithi067.stdout:5/305: symlink d0/d3a/l63 0 2024-04-12T05:37:16.802 INFO:tasks.workunit.client.1.smithi067.stdout:8/134: mknod d6/c2e 0 2024-04-12T05:37:16.803 INFO:tasks.workunit.client.0.smithi032.stdout:7/789: symlink d0/de/df/d20/l10b 0 2024-04-12T05:37:16.803 INFO:tasks.workunit.client.1.smithi067.stdout:8/135: fsync f5 0 2024-04-12T05:37:16.804 INFO:tasks.workunit.client.0.smithi032.stdout:3/837: fsync d12/fe6 0 2024-04-12T05:37:16.804 INFO:tasks.workunit.client.1.smithi067.stdout:1/188: write d4/d9/f31 [3370423,68687] 0 2024-04-12T05:37:16.810 INFO:tasks.workunit.client.0.smithi032.stdout:3/838: rename d12/d2f/f48 to d12/d2f/f10f 0 2024-04-12T05:37:16.811 INFO:tasks.workunit.client.1.smithi067.stdout:8/136: write d6/d8/d18/f25 [830032,56162] 0 2024-04-12T05:37:16.811 INFO:tasks.workunit.client.1.smithi067.stdout:6/179: truncate dd/f1d 1078757 0 2024-04-12T05:37:16.811 INFO:tasks.workunit.client.1.smithi067.stdout:4/86: dread - d8/d9/f11 zero size 2024-04-12T05:37:16.811 INFO:tasks.workunit.client.1.smithi067.stdout:7/175: truncate d0/d2/d5/f11 479192 0 2024-04-12T05:37:16.812 INFO:tasks.workunit.client.0.smithi032.stdout:9/622: dread d9/fa [0,4194304] 0 2024-04-12T05:37:16.812 INFO:tasks.workunit.client.1.smithi067.stdout:2/182: mkdir d5/da/de/d14/d34/d40 0 2024-04-12T05:37:16.813 INFO:tasks.workunit.client.1.smithi067.stdout:4/87: dread - d8/d9/f11 zero size 2024-04-12T05:37:16.816 INFO:tasks.workunit.client.0.smithi032.stdout:7/790: dwrite d0/d34/dbe/fa6 [0,4194304] 0 2024-04-12T05:37:16.817 INFO:tasks.workunit.client.1.smithi067.stdout:3/151: getdents dc 0 2024-04-12T05:37:16.818 INFO:tasks.workunit.client.0.smithi032.stdout:3/839: dread d12/d2f/fdd [0,4194304] 0 2024-04-12T05:37:16.819 INFO:tasks.workunit.client.1.smithi067.stdout:4/88: chown f7 2467 1 2024-04-12T05:37:16.819 INFO:tasks.workunit.client.1.smithi067.stdout:3/152: write f5 [2057663,39949] 0 2024-04-12T05:37:16.819 INFO:tasks.workunit.client.1.smithi067.stdout:7/176: dread d0/d2/d5/db/f30 [0,4194304] 0 2024-04-12T05:37:16.820 INFO:tasks.workunit.client.0.smithi032.stdout:3/840: chown d12/d2f/d89/df0/d98/f9a 263 1 2024-04-12T05:37:16.821 INFO:tasks.workunit.client.0.smithi032.stdout:7/791: dread - d0/dd/d51/fbc zero size 2024-04-12T05:37:16.821 INFO:tasks.workunit.client.1.smithi067.stdout:6/180: rename dd/f22 to dd/d30/f32 0 2024-04-12T05:37:16.822 INFO:tasks.workunit.client.1.smithi067.stdout:4/89: dread - d8/d9/f11 zero size 2024-04-12T05:37:16.822 INFO:tasks.workunit.client.1.smithi067.stdout:3/153: read dc/df/f1e [1950193,109677] 0 2024-04-12T05:37:16.823 INFO:tasks.workunit.client.0.smithi032.stdout:3/841: write d12/d24/fa4 [3052715,53099] 0 2024-04-12T05:37:16.823 INFO:tasks.workunit.client.0.smithi032.stdout:9/623: mkdir d9/d15/d1a/d24/d27/dce/dd3 0 2024-04-12T05:37:16.823 INFO:tasks.workunit.client.1.smithi067.stdout:2/183: fsync d5/d11/f29 0 2024-04-12T05:37:16.825 INFO:tasks.workunit.client.1.smithi067.stdout:3/154: write dc/d1d/f24 [799069,76662] 0 2024-04-12T05:37:16.831 INFO:tasks.workunit.client.1.smithi067.stdout:7/177: mkdir d0/d2/d5/db/d34 0 2024-04-12T05:37:16.832 INFO:tasks.workunit.client.1.smithi067.stdout:4/90: unlink d8/d14/d15/f1b 0 2024-04-12T05:37:16.834 INFO:tasks.workunit.client.0.smithi032.stdout:7/792: link d0/de/f44 d0/de/df/f10c 0 2024-04-12T05:37:16.835 INFO:tasks.workunit.client.1.smithi067.stdout:9/133: dread d1/f11 [0,4194304] 0 2024-04-12T05:37:16.835 INFO:tasks.workunit.client.1.smithi067.stdout:4/91: dread - d8/d9/f17 zero size 2024-04-12T05:37:16.836 INFO:tasks.workunit.client.1.smithi067.stdout:7/178: dread d0/d2/d5/d1e/f20 [0,4194304] 0 2024-04-12T05:37:16.838 INFO:tasks.workunit.client.1.smithi067.stdout:3/155: dread dc/f18 [0,4194304] 0 2024-04-12T05:37:16.838 INFO:tasks.workunit.client.0.smithi032.stdout:3/842: dread d12/daa/db3/fcc [0,4194304] 0 2024-04-12T05:37:16.838 INFO:tasks.workunit.client.0.smithi032.stdout:7/793: mkdir d0/de/d10d 0 2024-04-12T05:37:16.839 INFO:tasks.workunit.client.0.smithi032.stdout:3/843: chown d12/d24/dbb/cc1 20335 1 2024-04-12T05:37:16.841 INFO:tasks.workunit.client.0.smithi032.stdout:7/794: creat d0/de/df/d20/f10e x:0 0 0 2024-04-12T05:37:16.842 INFO:tasks.workunit.client.1.smithi067.stdout:1/189: getdents d4/d9/d12/d2c 0 2024-04-12T05:37:16.843 INFO:tasks.workunit.client.0.smithi032.stdout:8/873: truncate d0/db/dd/fec 139901 0 2024-04-12T05:37:16.844 INFO:tasks.workunit.client.0.smithi032.stdout:3/844: creat d12/d2f/d89/df0/d3a/d63/dd5/f110 x:0 0 0 2024-04-12T05:37:16.845 INFO:tasks.workunit.client.0.smithi032.stdout:8/874: chown d0/db/d38/la6 10 1 2024-04-12T05:37:16.845 INFO:tasks.workunit.client.0.smithi032.stdout:7/795: mkdir d0/d10/d70/d7b/dd2/d10f 0 2024-04-12T05:37:16.846 INFO:tasks.workunit.client.1.smithi067.stdout:9/134: dread d1/fd [0,4194304] 0 2024-04-12T05:37:16.846 INFO:tasks.workunit.client.1.smithi067.stdout:4/92: mkdir d8/d14/d15/d1c 0 2024-04-12T05:37:16.847 INFO:tasks.workunit.client.0.smithi032.stdout:9/624: write d9/d15/d2c/dab/fbd [678423,83221] 0 2024-04-12T05:37:16.847 INFO:tasks.workunit.client.1.smithi067.stdout:3/156: dwrite f5 [8388608,4194304] 0 2024-04-12T05:37:16.849 INFO:tasks.workunit.client.1.smithi067.stdout:9/135: readlink d1/d12/d15/l17 0 2024-04-12T05:37:16.849 INFO:tasks.workunit.client.1.smithi067.stdout:4/93: read d8/fe [2920485,18931] 0 2024-04-12T05:37:16.851 INFO:tasks.workunit.client.1.smithi067.stdout:0/260: write dd/f42 [1754153,100757] 0 2024-04-12T05:37:16.851 INFO:tasks.workunit.client.0.smithi032.stdout:9/625: chown d9/d15/d1a/c4c 3094788 1 2024-04-12T05:37:16.852 INFO:tasks.workunit.client.1.smithi067.stdout:3/157: truncate dc/df/f30 166528 0 2024-04-12T05:37:16.852 INFO:tasks.workunit.client.1.smithi067.stdout:7/179: fdatasync d0/d2/ff 0 2024-04-12T05:37:16.853 INFO:tasks.workunit.client.1.smithi067.stdout:7/180: stat d0/d2 0 2024-04-12T05:37:16.854 INFO:tasks.workunit.client.1.smithi067.stdout:3/158: write dc/d1d/f31 [986475,81495] 0 2024-04-12T05:37:16.855 INFO:tasks.workunit.client.0.smithi032.stdout:8/875: dwrite d0/db/dd/d11/d1b/f78 [0,4194304] 0 2024-04-12T05:37:16.856 INFO:tasks.workunit.client.1.smithi067.stdout:7/181: fsync d0/d2/d5/db/d10/f1a 0 2024-04-12T05:37:16.856 INFO:tasks.workunit.client.1.smithi067.stdout:3/159: readlink dc/df/l20 0 2024-04-12T05:37:16.856 INFO:tasks.workunit.client.1.smithi067.stdout:9/136: mknod d1/d12/d16/c1f 0 2024-04-12T05:37:16.857 INFO:tasks.workunit.client.0.smithi032.stdout:4/799: sync 2024-04-12T05:37:16.859 INFO:tasks.workunit.client.0.smithi032.stdout:9/626: mkdir d9/d15/d1a/d1d/dd4 0 2024-04-12T05:37:16.861 INFO:tasks.workunit.client.1.smithi067.stdout:3/160: write dc/df/f30 [899587,84814] 0 2024-04-12T05:37:16.862 INFO:tasks.workunit.client.0.smithi032.stdout:7/796: write d0/d34/f8d [397509,49470] 0 2024-04-12T05:37:16.864 INFO:tasks.workunit.client.0.smithi032.stdout:7/797: chown d0/d13/d21/d30/da1 65678542 1 2024-04-12T05:37:16.866 INFO:tasks.workunit.client.1.smithi067.stdout:1/190: symlink d4/d9/d34/l38 0 2024-04-12T05:37:16.867 INFO:tasks.workunit.client.1.smithi067.stdout:5/306: write d0/da/f5a [356462,124889] 0 2024-04-12T05:37:16.867 INFO:tasks.workunit.client.0.smithi032.stdout:8/876: dread d0/db/d38/d63/d77/dd4/f123 [0,4194304] 0 2024-04-12T05:37:16.867 INFO:tasks.workunit.client.0.smithi032.stdout:7/798: creat d0/de/df/d29/f110 x:0 0 0 2024-04-12T05:37:16.868 INFO:tasks.workunit.client.0.smithi032.stdout:9/627: link d9/d15/d1a/d24/d27/c2b d9/d15/d2c/cd5 0 2024-04-12T05:37:16.870 INFO:tasks.workunit.client.1.smithi067.stdout:9/137: dread d1/fd [0,4194304] 0 2024-04-12T05:37:16.871 INFO:tasks.workunit.client.0.smithi032.stdout:9/628: fdatasync d9/d15/d1a/d1d/fb9 0 2024-04-12T05:37:16.872 INFO:tasks.workunit.client.1.smithi067.stdout:1/191: fsync d4/d9/d12/d2c/f37 0 2024-04-12T05:37:16.873 INFO:tasks.workunit.client.0.smithi032.stdout:9/629: chown d9/d15/d1a/d24/d27/dce/dd3 1679 1 2024-04-12T05:37:16.873 INFO:tasks.workunit.client.0.smithi032.stdout:9/630: fdatasync d9/d15/d2c/dab/fc5 0 2024-04-12T05:37:16.876 INFO:tasks.workunit.client.0.smithi032.stdout:3/845: write d12/d2f/d89/df0/f74 [171796,114753] 0 2024-04-12T05:37:16.877 INFO:tasks.workunit.client.0.smithi032.stdout:9/631: unlink d9/d15/d1a/cbb 0 2024-04-12T05:37:16.877 INFO:tasks.workunit.client.1.smithi067.stdout:2/184: truncate d5/f12 73237 0 2024-04-12T05:37:16.879 INFO:tasks.workunit.client.0.smithi032.stdout:4/800: dwrite d4/d87/d96/d2f/d3b/f88 [0,4194304] 0 2024-04-12T05:37:16.880 INFO:tasks.workunit.client.1.smithi067.stdout:4/94: rename d8/d14 to d8/da/d1d 0 2024-04-12T05:37:16.882 INFO:tasks.workunit.client.1.smithi067.stdout:3/161: dwrite dc/df/f1c [0,4194304] 0 2024-04-12T05:37:16.883 INFO:tasks.workunit.client.1.smithi067.stdout:9/138: mknod d1/d12/d15/c20 0 2024-04-12T05:37:16.886 INFO:tasks.workunit.client.0.smithi032.stdout:3/846: dwrite d12/d2f/f81 [0,4194304] 0 2024-04-12T05:37:16.887 INFO:tasks.workunit.client.0.smithi032.stdout:9/632: truncate d9/d32/d86/f9b 533012 0 2024-04-12T05:37:16.888 INFO:tasks.workunit.client.1.smithi067.stdout:7/182: rename d0/d2/dd/c33 to d0/d2/d5/dc/c35 0 2024-04-12T05:37:16.888 INFO:tasks.workunit.client.0.smithi032.stdout:4/801: creat d4/d87/d96/d2f/d94/f10d x:0 0 0 2024-04-12T05:37:16.889 INFO:tasks.workunit.client.0.smithi032.stdout:3/847: fdatasync d12/d2f/d89/df0/d3a/d2d/ff2 0 2024-04-12T05:37:16.891 INFO:tasks.workunit.client.1.smithi067.stdout:4/95: mknod d8/da/d1d/c1e 0 2024-04-12T05:37:16.891 INFO:tasks.workunit.client.1.smithi067.stdout:3/162: read dc/d1d/d26/f2f [4088045,86803] 0 2024-04-12T05:37:16.893 INFO:tasks.workunit.client.1.smithi067.stdout:2/185: stat d5/da/de/d14/d34/d40 0 2024-04-12T05:37:16.896 INFO:tasks.workunit.client.1.smithi067.stdout:6/181: dwrite fa [4194304,4194304] 0 2024-04-12T05:37:16.898 INFO:tasks.workunit.client.1.smithi067.stdout:8/137: dwrite d6/d8/d18/df/f1c [4194304,4194304] 0 2024-04-12T05:37:16.898 INFO:tasks.workunit.client.1.smithi067.stdout:3/163: chown f5 449 1 2024-04-12T05:37:16.900 INFO:tasks.workunit.client.1.smithi067.stdout:7/183: dread - d0/d2/d5/d1e/f2f zero size 2024-04-12T05:37:16.901 INFO:tasks.workunit.client.0.smithi032.stdout:9/633: write d9/d15/d1a/d24/d27/f76 [1655130,115800] 0 2024-04-12T05:37:16.902 INFO:tasks.workunit.client.0.smithi032.stdout:4/802: symlink d4/d87/d96/d4d/db5/l10e 0 2024-04-12T05:37:16.905 INFO:tasks.workunit.client.1.smithi067.stdout:4/96: creat d8/da/f1f x:0 0 0 2024-04-12T05:37:16.905 INFO:tasks.workunit.client.0.smithi032.stdout:8/877: dwrite d0/db/dd/d11/d1b/d5f/fe3 [0,4194304] 0 2024-04-12T05:37:16.908 INFO:tasks.workunit.client.1.smithi067.stdout:0/261: dwrite dd/f3f [0,4194304] 0 2024-04-12T05:37:16.908 INFO:tasks.workunit.client.0.smithi032.stdout:9/634: unlink d9/d1f/c70 0 2024-04-12T05:37:16.909 INFO:tasks.workunit.client.1.smithi067.stdout:1/192: rename d4/d9/f2f to d4/d9/d12/d28/f39 0 2024-04-12T05:37:16.913 INFO:tasks.workunit.client.0.smithi032.stdout:3/848: dwrite d12/d2f/d89/df0/d3a/dbe/fe3 [0,4194304] 0 2024-04-12T05:37:16.918 INFO:tasks.workunit.client.1.smithi067.stdout:5/307: dread d0/d6/f2b [0,4194304] 0 2024-04-12T05:37:16.919 INFO:tasks.workunit.client.0.smithi032.stdout:9/635: rename d9/fa7 to d9/d15/d1a/d24/d27/d88/dc1/fd6 0 2024-04-12T05:37:16.924 INFO:tasks.workunit.client.1.smithi067.stdout:8/138: mkdir d6/d2f 0 2024-04-12T05:37:16.925 INFO:tasks.workunit.client.1.smithi067.stdout:1/193: unlink d4/d9/d12/f35 0 2024-04-12T05:37:16.927 INFO:tasks.workunit.client.0.smithi032.stdout:8/878: dwrite d0/db/d39/f83 [4194304,4194304] 0 2024-04-12T05:37:16.927 INFO:tasks.workunit.client.0.smithi032.stdout:3/849: mkdir d12/d2f/de2/d107/d111 0 2024-04-12T05:37:16.927 INFO:tasks.workunit.client.1.smithi067.stdout:5/308: read d0/f7 [5003540,11374] 0 2024-04-12T05:37:16.928 INFO:tasks.workunit.client.0.smithi032.stdout:9/636: symlink d9/d32/d86/ld7 0 2024-04-12T05:37:16.929 INFO:tasks.workunit.client.1.smithi067.stdout:1/194: chown d4/d9/d12/d28/f39 0 1 2024-04-12T05:37:16.930 INFO:tasks.workunit.client.0.smithi032.stdout:3/850: fsync d12/d2f/d89/fba 0 2024-04-12T05:37:16.934 INFO:tasks.workunit.client.1.smithi067.stdout:3/164: dwrite dc/d1d/d26/f2f [0,4194304] 0 2024-04-12T05:37:16.934 INFO:tasks.workunit.client.1.smithi067.stdout:3/165: chown dc/d1d/d26/f2f 7582040 1 2024-04-12T05:37:16.934 INFO:tasks.workunit.client.0.smithi032.stdout:8/879: dread d0/db/d39/f83 [0,4194304] 0 2024-04-12T05:37:16.937 INFO:tasks.workunit.client.0.smithi032.stdout:3/851: unlink d12/d2f/de2/fea 0 2024-04-12T05:37:16.937 INFO:tasks.workunit.client.0.smithi032.stdout:9/637: getdents d9/d15/d1a/d1d 0 2024-04-12T05:37:16.939 INFO:tasks.workunit.client.0.smithi032.stdout:3/852: chown d12/d2f/d89/df0/f82 42898229 1 2024-04-12T05:37:16.939 INFO:tasks.workunit.client.0.smithi032.stdout:9/638: chown d9/d15/d1a/dba 1935263 1 2024-04-12T05:37:16.941 INFO:tasks.workunit.client.0.smithi032.stdout:9/639: chown d9/d32/l5f 22314 1 2024-04-12T05:37:16.941 INFO:tasks.workunit.client.0.smithi032.stdout:3/853: read - d12/d2f/d89/df0/d3a/d63/da6/dac/dc9/fe8 zero size 2024-04-12T05:37:16.942 INFO:tasks.workunit.client.0.smithi032.stdout:9/640: stat d9/d15/d1a/d24/d27/f50 0 2024-04-12T05:37:16.946 INFO:tasks.workunit.client.1.smithi067.stdout:7/184: truncate d0/d2/ff 1898665 0 2024-04-12T05:37:16.946 INFO:tasks.workunit.client.1.smithi067.stdout:6/182: rename dd/l1b to dd/l33 0 2024-04-12T05:37:16.946 INFO:tasks.workunit.client.1.smithi067.stdout:5/309: symlink d0/d3a/d2c/l64 0 2024-04-12T05:37:16.946 INFO:tasks.workunit.client.1.smithi067.stdout:7/185: chown d0/d2/d5/db/c2d 8724 1 2024-04-12T05:37:16.946 INFO:tasks.workunit.client.1.smithi067.stdout:3/166: dread dc/d1d/d26/f2f [0,4194304] 0 2024-04-12T05:37:16.946 INFO:tasks.workunit.client.1.smithi067.stdout:3/167: chown dc/df/c16 413726 1 2024-04-12T05:37:16.948 INFO:tasks.workunit.client.1.smithi067.stdout:8/139: rename d6/c2e to d6/d2f/c30 0 2024-04-12T05:37:16.954 INFO:tasks.workunit.client.1.smithi067.stdout:0/262: write dd/d12/d14/f43 [913821,107053] 0 2024-04-12T05:37:16.956 INFO:tasks.workunit.client.1.smithi067.stdout:4/97: write d8/d9/f11 [277423,46067] 0 2024-04-12T05:37:16.958 INFO:tasks.workunit.client.1.smithi067.stdout:9/139: getdents d1/d12/d16 0 2024-04-12T05:37:16.958 INFO:tasks.workunit.client.1.smithi067.stdout:2/186: write d5/d11/f2a [565942,76718] 0 2024-04-12T05:37:16.958 INFO:tasks.workunit.client.1.smithi067.stdout:5/310: creat d0/d6/d20/f65 x:0 0 0 2024-04-12T05:37:16.960 INFO:tasks.workunit.client.1.smithi067.stdout:4/98: truncate d8/da/f18 646414 0 2024-04-12T05:37:16.961 INFO:tasks.workunit.client.1.smithi067.stdout:2/187: stat d5/da/de/d14/d34/d40 0 2024-04-12T05:37:16.963 INFO:tasks.workunit.client.1.smithi067.stdout:5/311: truncate d0/da/d41/f60 35747 0 2024-04-12T05:37:16.963 INFO:tasks.workunit.client.1.smithi067.stdout:8/140: dread d6/d8/d18/f25 [0,4194304] 0 2024-04-12T05:37:16.966 INFO:tasks.workunit.client.1.smithi067.stdout:0/263: fsync dd/d12/d14/f39 0 2024-04-12T05:37:16.966 INFO:tasks.workunit.client.1.smithi067.stdout:9/140: mkdir d1/d12/d16/d21 0 2024-04-12T05:37:16.966 INFO:tasks.workunit.client.1.smithi067.stdout:1/195: rename d4/d9/d12/d19 to d4/d9/d3a 0 2024-04-12T05:37:16.969 INFO:tasks.workunit.client.1.smithi067.stdout:4/99: creat d8/d9/f20 x:0 0 0 2024-04-12T05:37:16.974 INFO:tasks.workunit.client.1.smithi067.stdout:2/188: write d5/d11/f29 [752911,8249] 0 2024-04-12T05:37:16.975 INFO:tasks.workunit.client.1.smithi067.stdout:0/264: dread dd/f3f [0,4194304] 0 2024-04-12T05:37:16.976 INFO:tasks.workunit.client.1.smithi067.stdout:2/189: write d5/da/de/f31 [257185,20665] 0 2024-04-12T05:37:16.978 INFO:tasks.workunit.client.1.smithi067.stdout:7/186: rename d0/d2/d5/d1e/f2f to d0/d2/d5/d1d/f36 0 2024-04-12T05:37:16.980 INFO:tasks.workunit.client.1.smithi067.stdout:9/141: unlink d1/d12/d15/l1a 0 2024-04-12T05:37:16.981 INFO:tasks.workunit.client.1.smithi067.stdout:4/100: stat d8/da/d1d/d15/f1a 0 2024-04-12T05:37:16.982 INFO:tasks.workunit.client.1.smithi067.stdout:4/101: stat d8/da/d1d/d15/d1c 0 2024-04-12T05:37:16.983 INFO:tasks.workunit.client.1.smithi067.stdout:5/312: truncate d0/f7 3630574 0 2024-04-12T05:37:16.983 INFO:tasks.workunit.client.1.smithi067.stdout:6/183: link f9 dd/d11/f34 0 2024-04-12T05:37:16.985 INFO:tasks.workunit.client.0.smithi032.stdout:5/907: sync 2024-04-12T05:37:16.985 INFO:tasks.workunit.client.0.smithi032.stdout:1/680: sync 2024-04-12T05:37:16.985 INFO:tasks.workunit.client.0.smithi032.stdout:2/783: sync 2024-04-12T05:37:16.986 INFO:tasks.workunit.client.1.smithi067.stdout:2/190: mknod d5/d11/c41 0 2024-04-12T05:37:16.988 INFO:tasks.workunit.client.1.smithi067.stdout:7/187: rename d0/d2/d5/db/d10/c14 to d0/d2/dd/c37 0 2024-04-12T05:37:16.990 INFO:tasks.workunit.client.1.smithi067.stdout:7/188: read - d0/d2/f21 zero size 2024-04-12T05:37:16.991 INFO:tasks.workunit.client.1.smithi067.stdout:6/184: mknod dd/d11/c35 0 2024-04-12T05:37:16.991 INFO:tasks.workunit.client.1.smithi067.stdout:4/102: creat d8/d9/f21 x:0 0 0 2024-04-12T05:37:16.992 INFO:tasks.workunit.client.1.smithi067.stdout:1/196: dread d4/d9/f1c [0,4194304] 0 2024-04-12T05:37:16.994 INFO:tasks.workunit.client.1.smithi067.stdout:3/168: link la dc/l33 0 2024-04-12T05:37:16.995 INFO:tasks.workunit.client.1.smithi067.stdout:9/142: symlink d1/d12/d16/d21/l22 0 2024-04-12T05:37:16.995 INFO:tasks.workunit.client.1.smithi067.stdout:2/191: dread - d5/da/d33/f35 zero size 2024-04-12T05:37:16.996 INFO:tasks.workunit.client.1.smithi067.stdout:5/313: mknod d0/d6/d20/d3e/c66 0 2024-04-12T05:37:16.998 INFO:tasks.workunit.client.1.smithi067.stdout:0/265: creat dd/d12/d14/f57 x:0 0 0 2024-04-12T05:37:17.001 INFO:tasks.workunit.client.0.smithi032.stdout:8/880: write d0/db/dd/d11/d1b/d22/fb4 [2788218,92881] 0 2024-04-12T05:37:17.002 INFO:tasks.workunit.client.1.smithi067.stdout:1/197: chown d4/d9/d12/f2e 1249505380 1 2024-04-12T05:37:17.002 INFO:tasks.workunit.client.1.smithi067.stdout:7/189: truncate d0/d2/d5/db/f1f 283739 0 2024-04-12T05:37:17.003 INFO:tasks.workunit.client.1.smithi067.stdout:9/143: mkdir d1/d12/d16/d23 0 2024-04-12T05:37:17.004 INFO:tasks.workunit.client.1.smithi067.stdout:8/141: dwrite d6/d8/d18/df/f13 [0,4194304] 0 2024-04-12T05:37:17.005 INFO:tasks.workunit.client.1.smithi067.stdout:6/185: rename dd/d11/f14 to dd/d21/d23/f36 0 2024-04-12T05:37:17.005 INFO:tasks.workunit.client.0.smithi032.stdout:1/681: creat d5/fd3 x:0 0 0 2024-04-12T05:37:17.007 INFO:tasks.workunit.client.1.smithi067.stdout:8/142: stat d6/d2f 0 2024-04-12T05:37:17.007 INFO:tasks.workunit.client.1.smithi067.stdout:9/144: read - d1/d12/d15/f1d zero size 2024-04-12T05:37:17.011 INFO:tasks.workunit.client.0.smithi032.stdout:8/881: creat d0/db/d38/d63/d77/f12e x:0 0 0 2024-04-12T05:37:17.014 INFO:tasks.workunit.client.0.smithi032.stdout:2/784: truncate dd/d19/d9d/d1c/ffb 1890254 0 2024-04-12T05:37:17.015 INFO:tasks.workunit.client.1.smithi067.stdout:5/314: symlink d0/d6/d40/d4b/l67 0 2024-04-12T05:37:17.015 INFO:tasks.workunit.client.0.smithi032.stdout:2/785: dread - dd/d19/d9d/d1c/d64/d36/ffc zero size 2024-04-12T05:37:17.015 INFO:tasks.workunit.client.0.smithi032.stdout:5/908: rename d2/dd/d6f/fbd to d2/df/f123 0 2024-04-12T05:37:17.016 INFO:tasks.workunit.client.1.smithi067.stdout:0/266: symlink dd/d12/d14/d49/l58 0 2024-04-12T05:37:17.016 INFO:tasks.workunit.client.1.smithi067.stdout:2/192: dread f4 [0,4194304] 0 2024-04-12T05:37:17.020 INFO:tasks.workunit.client.0.smithi032.stdout:9/641: rename d9/d15/d1a/d24/f8b to d9/d15/d1a/d24/d27/d88/dc1/fd8 0 2024-04-12T05:37:17.020 INFO:tasks.workunit.client.0.smithi032.stdout:3/854: truncate d12/d2f/d89/d9d/fce 233682 0 2024-04-12T05:37:17.020 INFO:tasks.workunit.client.1.smithi067.stdout:2/193: write d5/d11/f29 [1733978,42310] 0 2024-04-12T05:37:17.023 INFO:tasks.workunit.client.1.smithi067.stdout:0/267: fdatasync dd/d12/d14/d1a/d3d/f32 0 2024-04-12T05:37:17.023 INFO:tasks.workunit.client.0.smithi032.stdout:9/642: fdatasync d9/f34 0 2024-04-12T05:37:17.025 INFO:tasks.workunit.client.1.smithi067.stdout:8/143: dwrite d6/d8/d18/df/d12/f29 [0,4194304] 0 2024-04-12T05:37:17.026 INFO:tasks.workunit.client.1.smithi067.stdout:7/190: mkdir d0/d2/d5/d38 0 2024-04-12T05:37:17.029 INFO:tasks.workunit.client.0.smithi032.stdout:3/855: dwrite d12/d2f/d89/df0/f82 [0,4194304] 0 2024-04-12T05:37:17.030 INFO:tasks.workunit.client.1.smithi067.stdout:0/268: chown dd/d12/f21 1022334334 1 2024-04-12T05:37:17.034 INFO:tasks.workunit.client.1.smithi067.stdout:9/145: truncate d1/f6 1991587 0 2024-04-12T05:37:17.034 INFO:tasks.workunit.client.1.smithi067.stdout:1/198: readlink d4/d9/d3a/l27 0 2024-04-12T05:37:17.036 INFO:tasks.workunit.client.1.smithi067.stdout:7/191: write d0/d2/d5/db/f1f [144748,38457] 0 2024-04-12T05:37:17.037 INFO:tasks.workunit.client.0.smithi032.stdout:3/856: creat d12/d2f/de2/f112 x:0 0 0 2024-04-12T05:37:17.038 INFO:tasks.workunit.client.0.smithi032.stdout:3/857: fdatasync d12/d2f/d89/f104 0 2024-04-12T05:37:17.043 INFO:tasks.workunit.client.1.smithi067.stdout:4/103: dwrite d8/da/d1d/d15/f1a [0,4194304] 0 2024-04-12T05:37:17.043 INFO:tasks.workunit.client.1.smithi067.stdout:6/186: symlink dd/l37 0 2024-04-12T05:37:17.043 INFO:tasks.workunit.client.1.smithi067.stdout:2/194: rmdir d5/da/de/d14 39 2024-04-12T05:37:17.043 INFO:tasks.workunit.client.1.smithi067.stdout:4/104: chown d8/d9/f17 6 1 2024-04-12T05:37:17.044 INFO:tasks.workunit.client.0.smithi032.stdout:8/882: truncate d0/db/d38/f61 3790647 0 2024-04-12T05:37:17.046 INFO:tasks.workunit.client.0.smithi032.stdout:3/858: creat d12/d2f/d89/d9d/f113 x:0 0 0 2024-04-12T05:37:17.050 INFO:tasks.workunit.client.0.smithi032.stdout:8/883: symlink d0/d58/df7/l12f 0 2024-04-12T05:37:17.050 INFO:tasks.workunit.client.0.smithi032.stdout:3/859: stat d12/d2f/d89/df0/d3a/d63/fb2 0 2024-04-12T05:37:17.051 INFO:tasks.workunit.client.1.smithi067.stdout:4/105: read - d8/da/f1f zero size 2024-04-12T05:37:17.051 INFO:tasks.workunit.client.0.smithi032.stdout:8/884: chown d0/d58/cd1 9617 1 2024-04-12T05:37:17.054 INFO:tasks.workunit.client.0.smithi032.stdout:5/909: dwrite d2/df/d25/d50/f8e [0,4194304] 0 2024-04-12T05:37:17.057 INFO:tasks.workunit.client.1.smithi067.stdout:3/169: truncate dc/d1d/f24 749960 0 2024-04-12T05:37:17.058 INFO:tasks.workunit.client.1.smithi067.stdout:6/187: write dd/d11/f19 [171154,20113] 0 2024-04-12T05:37:17.058 INFO:tasks.workunit.client.1.smithi067.stdout:7/192: rmdir d0/d23 39 2024-04-12T05:37:17.060 INFO:tasks.workunit.client.1.smithi067.stdout:8/144: dwrite d6/d8/d18/df/d12/f1f [0,4194304] 0 2024-04-12T05:37:17.062 INFO:tasks.workunit.client.1.smithi067.stdout:8/145: dread - d6/f2c zero size 2024-04-12T05:37:17.063 INFO:tasks.workunit.client.0.smithi032.stdout:2/786: truncate dd/d19/d9d/d42/d4a/fcf 1075733 0 2024-04-12T05:37:17.066 INFO:tasks.workunit.client.0.smithi032.stdout:5/910: unlink d2/df/d25/f98 0 2024-04-12T05:37:17.067 INFO:tasks.workunit.client.1.smithi067.stdout:4/106: unlink d8/cd 0 2024-04-12T05:37:17.067 INFO:tasks.workunit.client.1.smithi067.stdout:9/146: write d1/d13/f1e [753397,96144] 0 2024-04-12T05:37:17.071 INFO:tasks.workunit.client.1.smithi067.stdout:9/147: write d1/d12/d16/f1b [420855,128332] 0 2024-04-12T05:37:17.072 INFO:tasks.workunit.client.1.smithi067.stdout:2/195: unlink d5/l2b 0 2024-04-12T05:37:17.073 INFO:tasks.workunit.client.1.smithi067.stdout:0/269: creat dd/d12/f59 x:0 0 0 2024-04-12T05:37:17.078 INFO:tasks.workunit.client.1.smithi067.stdout:7/193: mknod d0/d2/d5/d1e/c39 0 2024-04-12T05:37:17.079 INFO:tasks.workunit.client.1.smithi067.stdout:3/170: write dc/f23 [462834,111456] 0 2024-04-12T05:37:17.081 INFO:tasks.workunit.client.1.smithi067.stdout:5/315: dwrite d0/f7 [0,4194304] 0 2024-04-12T05:37:17.081 INFO:tasks.workunit.client.1.smithi067.stdout:6/188: symlink dd/d11/l38 0 2024-04-12T05:37:17.081 INFO:tasks.workunit.client.1.smithi067.stdout:0/270: write dd/f18 [3734589,8934] 0 2024-04-12T05:37:17.084 INFO:tasks.workunit.client.0.smithi032.stdout:2/787: dread dd/d19/d9d/d1c/d64/d36/f85 [0,4194304] 0 2024-04-12T05:37:17.084 INFO:tasks.workunit.client.1.smithi067.stdout:4/107: symlink d8/l22 0 2024-04-12T05:37:17.085 INFO:tasks.workunit.client.0.smithi032.stdout:2/788: write dd/d8c/fd7 [255476,129944] 0 2024-04-12T05:37:17.091 INFO:tasks.workunit.client.1.smithi067.stdout:1/199: link d4/d9/d12/d28/f39 d4/d9/d12/f3b 0 2024-04-12T05:37:17.091 INFO:tasks.workunit.client.1.smithi067.stdout:8/146: dwrite d6/d8/d18/df/f23 [0,4194304] 0 2024-04-12T05:37:17.091 INFO:tasks.workunit.client.0.smithi032.stdout:8/885: dread d0/db/dd/d11/d1b/d22/fd6 [0,4194304] 0 2024-04-12T05:37:17.091 INFO:tasks.workunit.client.0.smithi032.stdout:2/789: creat dd/d19/d9d/d1c/d64/d32/d6d/f101 x:0 0 0 2024-04-12T05:37:17.092 INFO:tasks.workunit.client.0.smithi032.stdout:2/790: write f8 [1836249,93818] 0 2024-04-12T05:37:17.093 INFO:tasks.workunit.client.0.smithi032.stdout:2/791: chown dd/d19/d82 29916058 1 2024-04-12T05:37:17.096 INFO:tasks.workunit.client.1.smithi067.stdout:7/194: creat d0/d2/d5/d1d/f3a x:0 0 0 2024-04-12T05:37:17.097 INFO:tasks.workunit.client.1.smithi067.stdout:2/196: symlink d5/da/d3d/l42 0 2024-04-12T05:37:17.098 INFO:tasks.workunit.client.1.smithi067.stdout:9/148: dwrite d1/d12/d15/f1c [0,4194304] 0 2024-04-12T05:37:17.100 INFO:tasks.workunit.client.0.smithi032.stdout:2/792: mknod dd/d19/c102 0 2024-04-12T05:37:17.101 INFO:tasks.workunit.client.1.smithi067.stdout:3/171: dread - dc/df/d27/f2d zero size 2024-04-12T05:37:17.103 INFO:tasks.workunit.client.0.smithi032.stdout:8/886: rename d0/db/dd/d11/d1b/d5f/daf/dc6 to d0/d36/d84/dad/d130 0 2024-04-12T05:37:17.106 INFO:tasks.workunit.client.1.smithi067.stdout:1/200: write d4/d9/d3a/f1e [2758713,67119] 0 2024-04-12T05:37:17.106 INFO:tasks.workunit.client.1.smithi067.stdout:6/189: stat dd/d21/d23/f36 0 2024-04-12T05:37:17.109 INFO:tasks.workunit.client.0.smithi032.stdout:2/793: getdents dd/d19/d9d/d42/d4a/da1/dd5 0 2024-04-12T05:37:17.109 INFO:tasks.workunit.client.1.smithi067.stdout:7/195: mknod d0/d2/d5/db/d10/c3b 0 2024-04-12T05:37:17.110 INFO:tasks.workunit.client.1.smithi067.stdout:0/271: rename dd/d12/f25 to dd/d12/d14/d51/f5a 0 2024-04-12T05:37:17.112 INFO:tasks.workunit.client.0.smithi032.stdout:8/887: truncate d0/db/dd/d11/d1b/d22/d54/da8/fce 687799 0 2024-04-12T05:37:17.112 INFO:tasks.workunit.client.0.smithi032.stdout:2/794: creat dd/d19/d9d/d42/da7/dc5/f103 x:0 0 0 2024-04-12T05:37:17.113 INFO:tasks.workunit.client.1.smithi067.stdout:5/316: creat d0/da/f68 x:0 0 0 2024-04-12T05:37:17.113 INFO:tasks.workunit.client.0.smithi032.stdout:8/888: chown d0/db/f11a 33005 1 2024-04-12T05:37:17.114 INFO:tasks.workunit.client.0.smithi032.stdout:2/795: fsync dd/d19/d20/fa0 0 2024-04-12T05:37:17.114 INFO:tasks.workunit.client.0.smithi032.stdout:2/796: readlink dd/d19/l26 0 2024-04-12T05:37:17.118 INFO:tasks.workunit.client.1.smithi067.stdout:0/272: mkdir dd/d5b 0 2024-04-12T05:37:17.120 INFO:tasks.workunit.client.1.smithi067.stdout:9/149: dwrite d1/d13/f1e [0,4194304] 0 2024-04-12T05:37:17.120 INFO:tasks.workunit.client.0.smithi032.stdout:8/889: symlink d0/db/dd/d11/d1b/d22/d51/l131 0 2024-04-12T05:37:17.122 INFO:tasks.workunit.client.1.smithi067.stdout:7/196: symlink d0/d2/d5/db/l3c 0 2024-04-12T05:37:17.122 INFO:tasks.workunit.client.1.smithi067.stdout:3/172: write dc/f18 [2616798,80222] 0 2024-04-12T05:37:17.122 INFO:tasks.workunit.client.0.smithi032.stdout:2/797: creat dd/d19/d20/dc6/f104 x:0 0 0 2024-04-12T05:37:17.123 INFO:tasks.workunit.client.0.smithi032.stdout:2/798: dread - dd/d19/d9d/d1c/d64/fc2 zero size 2024-04-12T05:37:17.125 INFO:tasks.workunit.client.1.smithi067.stdout:6/190: dread dd/d11/f34 [0,4194304] 0 2024-04-12T05:37:17.125 INFO:tasks.workunit.client.1.smithi067.stdout:3/173: write f1 [1774537,66855] 0 2024-04-12T05:37:17.126 INFO:tasks.workunit.client.1.smithi067.stdout:5/317: fdatasync d0/d6/f2b 0 2024-04-12T05:37:17.127 INFO:tasks.workunit.client.1.smithi067.stdout:4/108: rename d8/l13 to d8/da/d1d/d15/d1c/l23 0 2024-04-12T05:37:17.130 INFO:tasks.workunit.client.0.smithi032.stdout:8/890: mkdir d0/db/dd/d132 0 2024-04-12T05:37:17.131 INFO:tasks.workunit.client.0.smithi032.stdout:8/891: fsync d0/db/dd/d11/f125 0 2024-04-12T05:37:17.136 INFO:tasks.workunit.client.1.smithi067.stdout:6/191: dread f9 [0,4194304] 0 2024-04-12T05:37:17.137 INFO:tasks.workunit.client.1.smithi067.stdout:8/147: getdents d6 0 2024-04-12T05:37:17.138 INFO:tasks.workunit.client.1.smithi067.stdout:2/197: rename d5/da/d33 to d5/d43 0 2024-04-12T05:37:17.138 INFO:tasks.workunit.client.1.smithi067.stdout:5/318: mkdir d0/d6/d20/d3e/d69 0 2024-04-12T05:37:17.139 INFO:tasks.workunit.client.0.smithi032.stdout:8/892: truncate d0/db/dd/d11/d1b/d5f/deb/ffa 598164 0 2024-04-12T05:37:17.140 INFO:tasks.workunit.client.1.smithi067.stdout:3/174: dwrite dc/df/f1c [0,4194304] 0 2024-04-12T05:37:17.140 INFO:tasks.workunit.client.1.smithi067.stdout:5/319: chown d0/d3a/l5 412 1 2024-04-12T05:37:17.142 INFO:tasks.workunit.client.1.smithi067.stdout:5/320: readlink d0/d3a/l1e 0 2024-04-12T05:37:17.142 INFO:tasks.workunit.client.1.smithi067.stdout:3/175: rename dc/d1d to dc/d1d/d26/d34 22 2024-04-12T05:37:17.152 INFO:tasks.workunit.client.0.smithi032.stdout:2/799: write dd/d19/d9d/d42/d4a/fb3 [1766749,122557] 0 2024-04-12T05:37:17.154 INFO:tasks.workunit.client.0.smithi032.stdout:2/800: dread dd/d8c/fd7 [0,4194304] 0 2024-04-12T05:37:17.156 INFO:tasks.workunit.client.0.smithi032.stdout:2/801: write dd/d19/d9d/d42/d4a/f100 [738556,70478] 0 2024-04-12T05:37:17.157 INFO:tasks.workunit.client.0.smithi032.stdout:2/802: chown dd/d19/d9d/d42/d4a/da1/lda 417 1 2024-04-12T05:37:17.157 INFO:tasks.workunit.client.0.smithi032.stdout:2/803: readlink dd/d19/la4 0 2024-04-12T05:37:17.157 INFO:tasks.workunit.client.1.smithi067.stdout:4/109: dread f6 [0,4194304] 0 2024-04-12T05:37:17.159 INFO:tasks.workunit.client.0.smithi032.stdout:2/804: symlink dd/l105 0 2024-04-12T05:37:17.161 INFO:tasks.workunit.client.0.smithi032.stdout:2/805: write dd/d19/d9d/d1c/d64/d32/d3a/d8d/fe4 [1432006,65028] 0 2024-04-12T05:37:17.162 INFO:tasks.workunit.client.0.smithi032.stdout:8/893: write d0/db/dd/d11/d1b/d22/d51/f69 [2137073,90455] 0 2024-04-12T05:37:17.165 INFO:tasks.workunit.client.1.smithi067.stdout:7/197: dread d0/f1 [0,4194304] 0 2024-04-12T05:37:17.166 INFO:tasks.workunit.client.0.smithi032.stdout:8/894: write d0/db/f55 [3116856,87910] 0 2024-04-12T05:37:17.170 INFO:tasks.workunit.client.0.smithi032.stdout:8/895: dread d0/db/dd/d11/d1b/d5f/daf/fa2 [0,4194304] 0 2024-04-12T05:37:17.173 INFO:tasks.workunit.client.0.smithi032.stdout:8/896: creat d0/db/dd/d11/d1b/d22/d29/f133 x:0 0 0 2024-04-12T05:37:17.184 INFO:tasks.workunit.client.0.smithi032.stdout:2/806: dwrite dd/d19/d9d/fae [0,4194304] 0 2024-04-12T05:37:17.186 INFO:tasks.workunit.client.0.smithi032.stdout:8/897: dwrite d0/d36/de2/ff3 [0,4194304] 0 2024-04-12T05:37:17.189 INFO:tasks.workunit.client.0.smithi032.stdout:2/807: write dd/d19/d9d/d1c/d64/d36/f85 [5161531,37461] 0 2024-04-12T05:37:17.192 INFO:tasks.workunit.client.0.smithi032.stdout:8/898: readlink d0/db/dd/d11/d1b/d22/d29/ded/lf2 0 2024-04-12T05:37:17.203 INFO:tasks.workunit.client.0.smithi032.stdout:2/808: write dd/d19/d20/dc6/fed [4091256,24788] 0 2024-04-12T05:37:17.203 INFO:tasks.workunit.client.0.smithi032.stdout:8/899: write d0/f7 [2366459,65960] 0 2024-04-12T05:37:17.207 INFO:tasks.workunit.client.0.smithi032.stdout:2/809: mknod dd/d19/d9d/d42/dc7/c106 0 2024-04-12T05:37:17.209 INFO:tasks.workunit.client.0.smithi032.stdout:8/900: dwrite d0/db/d38/f61 [0,4194304] 0 2024-04-12T05:37:17.212 INFO:tasks.workunit.client.0.smithi032.stdout:2/810: dread dd/d19/d9d/d42/fe9 [4194304,4194304] 0 2024-04-12T05:37:17.214 INFO:tasks.workunit.client.0.smithi032.stdout:8/901: dread d0/db/dd/f17 [0,4194304] 0 2024-04-12T05:37:17.218 INFO:tasks.workunit.client.0.smithi032.stdout:2/811: unlink dd/d19/d9d/d1c/d64/d32/f47 0 2024-04-12T05:37:17.218 INFO:tasks.workunit.client.0.smithi032.stdout:2/812: fsync dd/d19/d20/f28 0 2024-04-12T05:37:17.219 INFO:tasks.workunit.client.0.smithi032.stdout:8/902: unlink d0/d36/de2/ff3 0 2024-04-12T05:37:17.219 INFO:tasks.workunit.client.0.smithi032.stdout:0/804: sync 2024-04-12T05:37:17.221 INFO:tasks.workunit.client.0.smithi032.stdout:2/813: read - dd/d19/d9d/d42/d4a/fc3 zero size 2024-04-12T05:37:17.222 INFO:tasks.workunit.client.0.smithi032.stdout:2/814: write dd/d19/d9d/d42/da7/dc5/f103 [152410,20870] 0 2024-04-12T05:37:17.222 INFO:tasks.workunit.client.0.smithi032.stdout:8/903: creat d0/db/d38/d63/d77/f134 x:0 0 0 2024-04-12T05:37:17.223 INFO:tasks.workunit.client.0.smithi032.stdout:2/815: write dd/d19/d9d/d1c/fb6 [1712613,21181] 0 2024-04-12T05:37:17.228 INFO:tasks.workunit.client.0.smithi032.stdout:0/805: mkdir d2/d8/d17/d1b/d109 0 2024-04-12T05:37:17.233 INFO:tasks.workunit.client.0.smithi032.stdout:2/816: fdatasync dd/d19/d9d/d1c/f76 0 2024-04-12T05:37:17.238 INFO:tasks.workunit.client.0.smithi032.stdout:0/806: truncate d2/dc/d3e/dac/fa1 633828 0 2024-04-12T05:37:17.239 INFO:tasks.workunit.client.1.smithi067.stdout:9/150: creat d1/d12/d16/d23/f24 x:0 0 0 2024-04-12T05:37:17.239 INFO:tasks.workunit.client.1.smithi067.stdout:6/192: mknod dd/d21/d23/c39 0 2024-04-12T05:37:17.240 INFO:tasks.workunit.client.1.smithi067.stdout:5/321: creat d0/d6/f6a x:0 0 0 2024-04-12T05:37:17.242 INFO:tasks.workunit.client.1.smithi067.stdout:1/201: write d4/d9/d12/f1b [1789081,18903] 0 2024-04-12T05:37:17.244 INFO:tasks.workunit.client.1.smithi067.stdout:7/198: symlink d0/d2/d5/db/d10/l3d 0 2024-04-12T05:37:17.249 INFO:tasks.workunit.client.0.smithi032.stdout:8/904: dwrite d0/db/dd/d11/d1b/f89 [0,4194304] 0 2024-04-12T05:37:17.249 INFO:tasks.workunit.client.1.smithi067.stdout:2/198: rmdir d5 39 2024-04-12T05:37:17.249 INFO:tasks.workunit.client.1.smithi067.stdout:2/199: stat c0 0 2024-04-12T05:37:17.254 INFO:tasks.workunit.client.0.smithi032.stdout:8/905: mkdir d0/db/dd/d11/d1b/d22/d35/d135 0 2024-04-12T05:37:17.258 INFO:tasks.workunit.client.0.smithi032.stdout:2/817: dread dd/d19/d9d/d1c/d64/d32/f9f [0,4194304] 0 2024-04-12T05:37:17.261 INFO:tasks.workunit.client.0.smithi032.stdout:2/818: truncate dd/d8c/fe2 725966 0 2024-04-12T05:37:17.262 INFO:tasks.workunit.client.0.smithi032.stdout:2/819: stat dd/d19/d9d/l7b 0 2024-04-12T05:37:17.263 INFO:tasks.workunit.client.1.smithi067.stdout:0/273: getdents dd/d12/d14/d51 0 2024-04-12T05:37:17.266 INFO:tasks.workunit.client.1.smithi067.stdout:6/193: read fb [80695,58126] 0 2024-04-12T05:37:17.266 INFO:tasks.workunit.client.1.smithi067.stdout:3/176: fsync dc/d1d/f24 0 2024-04-12T05:37:17.270 INFO:tasks.workunit.client.0.smithi032.stdout:2/820: write dd/d19/d9d/d1c/f87 [441780,83988] 0 2024-04-12T05:37:17.272 INFO:tasks.workunit.client.1.smithi067.stdout:0/274: rename dd/d12/f29 to dd/d12/d14/d51/f5c 0 2024-04-12T05:37:17.273 INFO:tasks.workunit.client.1.smithi067.stdout:5/322: dread d0/d6/d33/f3c [0,4194304] 0 2024-04-12T05:37:17.274 INFO:tasks.workunit.client.0.smithi032.stdout:2/821: creat dd/d19/f107 x:0 0 0 2024-04-12T05:37:17.275 INFO:tasks.workunit.client.0.smithi032.stdout:8/906: dwrite d0/db/dd/d11/d1b/d22/d54/f64 [4194304,4194304] 0 2024-04-12T05:37:17.276 INFO:tasks.workunit.client.1.smithi067.stdout:3/177: dread dc/f18 [0,4194304] 0 2024-04-12T05:37:17.279 INFO:tasks.workunit.client.1.smithi067.stdout:7/199: fdatasync d0/d2/d5/db/f30 0 2024-04-12T05:37:17.280 INFO:tasks.workunit.client.1.smithi067.stdout:7/200: stat d0/d2/d5/d1d/l2a 0 2024-04-12T05:37:17.283 INFO:tasks.workunit.client.1.smithi067.stdout:2/200: truncate d5/da/de/d14/f3b 651550 0 2024-04-12T05:37:17.283 INFO:tasks.workunit.client.1.smithi067.stdout:0/275: rmdir dd/d12/d14/d51 39 2024-04-12T05:37:17.285 INFO:tasks.workunit.client.0.smithi032.stdout:2/822: dwrite dd/d19/d20/f28 [4194304,4194304] 0 2024-04-12T05:37:17.287 INFO:tasks.workunit.client.1.smithi067.stdout:5/323: rename d0/d6/d20/f65 to d0/da/d41/f6b 0 2024-04-12T05:37:17.287 INFO:tasks.workunit.client.0.smithi032.stdout:8/907: dwrite d0/db/dd/d11/f125 [0,4194304] 0 2024-04-12T05:37:17.292 INFO:tasks.workunit.client.0.smithi032.stdout:2/823: chown dd/l105 31 1 2024-04-12T05:37:17.292 INFO:tasks.workunit.client.1.smithi067.stdout:7/201: dread d0/d2/d5/d1e/f20 [0,4194304] 0 2024-04-12T05:37:17.292 INFO:tasks.workunit.client.1.smithi067.stdout:2/201: rmdir d5/d23 39 2024-04-12T05:37:17.293 INFO:tasks.workunit.client.1.smithi067.stdout:0/276: dread dd/d12/d14/d1a/d3d/f32 [0,4194304] 0 2024-04-12T05:37:17.294 INFO:tasks.workunit.client.0.smithi032.stdout:2/824: stat dd/d19/d9d/d1c/d64/d32/d6d/f7f 0 2024-04-12T05:37:17.294 INFO:tasks.workunit.client.1.smithi067.stdout:7/202: write d0/d2/d5/fe [3323167,84258] 0 2024-04-12T05:37:17.300 INFO:tasks.workunit.client.1.smithi067.stdout:5/324: rename d0/d3a/f54 to d0/d6/d20/d3e/f6c 0 2024-04-12T05:37:17.300 INFO:tasks.workunit.client.0.smithi032.stdout:2/825: mknod dd/d19/d20/dc6/c108 0 2024-04-12T05:37:17.304 INFO:tasks.workunit.client.0.smithi032.stdout:8/908: rename d0/d36/d84 to d0/db/dd/d11/d1b/d22/d54/d136 0 2024-04-12T05:37:17.304 INFO:tasks.workunit.client.1.smithi067.stdout:8/148: truncate d6/d8/f14 4228972 0 2024-04-12T05:37:17.304 INFO:tasks.workunit.client.1.smithi067.stdout:9/151: write d1/d12/f14 [3348410,97669] 0 2024-04-12T05:37:17.304 INFO:tasks.workunit.client.1.smithi067.stdout:2/202: fdatasync d5/da/de/d14/d34/f37 0 2024-04-12T05:37:17.304 INFO:tasks.workunit.client.1.smithi067.stdout:7/203: creat d0/d2/d5/d1d/f3e x:0 0 0 2024-04-12T05:37:17.305 INFO:tasks.workunit.client.0.smithi032.stdout:2/826: creat dd/d19/d20/dc6/f109 x:0 0 0 2024-04-12T05:37:17.307 INFO:tasks.workunit.client.1.smithi067.stdout:6/194: truncate fc 670093 0 2024-04-12T05:37:17.308 INFO:tasks.workunit.client.0.smithi032.stdout:2/827: creat dd/d19/d82/f10a x:0 0 0 2024-04-12T05:37:17.309 INFO:tasks.workunit.client.1.smithi067.stdout:9/152: symlink d1/d12/d15/l25 0 2024-04-12T05:37:17.309 INFO:tasks.workunit.client.1.smithi067.stdout:5/325: creat d0/da/f6d x:0 0 0 2024-04-12T05:37:17.311 INFO:tasks.workunit.client.1.smithi067.stdout:5/326: chown d0/da/f6d 194014 1 2024-04-12T05:37:17.315 INFO:tasks.workunit.client.0.smithi032.stdout:8/909: write d0/db/dd/fec [256291,108324] 0 2024-04-12T05:37:17.316 INFO:tasks.workunit.client.1.smithi067.stdout:8/149: dread d6/d8/d18/df/f13 [0,4194304] 0 2024-04-12T05:37:17.316 INFO:tasks.workunit.client.1.smithi067.stdout:8/150: chown d6/d8/d18/df/d12/f1f 1 1 2024-04-12T05:37:17.318 INFO:tasks.workunit.client.0.smithi032.stdout:8/910: mkdir d0/dfd/d137 0 2024-04-12T05:37:17.320 INFO:tasks.workunit.client.1.smithi067.stdout:7/204: creat d0/d2/d5/db/d34/f3f x:0 0 0 2024-04-12T05:37:17.320 INFO:tasks.workunit.client.1.smithi067.stdout:9/153: fsync d1/f11 0 2024-04-12T05:37:17.320 INFO:tasks.workunit.client.1.smithi067.stdout:6/195: rename dd/l37 to dd/d30/l3a 0 2024-04-12T05:37:17.322 INFO:tasks.workunit.client.1.smithi067.stdout:9/154: fdatasync d1/d12/f14 0 2024-04-12T05:37:17.323 INFO:tasks.workunit.client.0.smithi032.stdout:8/911: mknod d0/db/dd/d11/d1b/d22/d54/d136/db3/c138 0 2024-04-12T05:37:17.323 INFO:tasks.workunit.client.1.smithi067.stdout:5/327: fsync d0/d6/f10 0 2024-04-12T05:37:17.325 INFO:tasks.workunit.client.0.smithi032.stdout:8/912: getdents d0/db/d7d/d11f 0 2024-04-12T05:37:17.326 INFO:tasks.workunit.client.1.smithi067.stdout:7/205: unlink d0/d2/dd/c16 0 2024-04-12T05:37:17.329 INFO:tasks.workunit.client.1.smithi067.stdout:5/328: unlink d0/d3a/c5c 0 2024-04-12T05:37:17.329 INFO:tasks.workunit.client.0.smithi032.stdout:8/913: fsync d0/db/dd/d11/d1b/fc9 0 2024-04-12T05:37:17.331 INFO:tasks.workunit.client.1.smithi067.stdout:6/196: creat dd/f3b x:0 0 0 2024-04-12T05:37:17.332 INFO:tasks.workunit.client.0.smithi032.stdout:8/914: mknod d0/d36/de2/c139 0 2024-04-12T05:37:17.333 INFO:tasks.workunit.client.1.smithi067.stdout:5/329: read - d0/da/f6d zero size 2024-04-12T05:37:17.335 INFO:tasks.workunit.client.0.smithi032.stdout:8/915: creat d0/db/dd/ddd/f13a x:0 0 0 2024-04-12T05:37:17.338 INFO:tasks.workunit.client.1.smithi067.stdout:0/277: dread dd/d12/d14/f39 [0,4194304] 0 2024-04-12T05:37:17.339 INFO:tasks.workunit.client.1.smithi067.stdout:8/151: dwrite d6/d8/d18/f25 [0,4194304] 0 2024-04-12T05:37:17.343 INFO:tasks.workunit.client.1.smithi067.stdout:2/203: dwrite d5/d43/f35 [0,4194304] 0 2024-04-12T05:37:17.344 INFO:tasks.workunit.client.0.smithi032.stdout:8/916: truncate d0/db/dd/d11/d1b/d5f/f93 3450342 0 2024-04-12T05:37:17.346 INFO:tasks.workunit.client.1.smithi067.stdout:9/155: link d1/f7 d1/d12/d16/d21/f26 0 2024-04-12T05:37:17.349 INFO:tasks.workunit.client.1.smithi067.stdout:6/197: getdents dd/d11/d27 0 2024-04-12T05:37:17.351 INFO:tasks.workunit.client.1.smithi067.stdout:0/278: mknod dd/d12/d14/d1a/d3d/c5d 0 2024-04-12T05:37:17.353 INFO:tasks.workunit.client.0.smithi032.stdout:8/917: unlink d0/db/dd/d11/d1b/fc9 0 2024-04-12T05:37:17.354 INFO:tasks.workunit.client.1.smithi067.stdout:2/204: write d5/da/de/d14/f24 [7764725,30539] 0 2024-04-12T05:37:17.355 INFO:tasks.workunit.client.0.smithi032.stdout:1/682: sync 2024-04-12T05:37:17.356 INFO:tasks.workunit.client.0.smithi032.stdout:7/799: sync 2024-04-12T05:37:17.356 INFO:tasks.workunit.client.0.smithi032.stdout:4/803: sync 2024-04-12T05:37:17.357 INFO:tasks.workunit.client.0.smithi032.stdout:7/800: readlink d0/dd/la7 0 2024-04-12T05:37:17.357 INFO:tasks.workunit.client.0.smithi032.stdout:1/683: chown d5/de/d2a/fb6 366067 1 2024-04-12T05:37:17.362 INFO:tasks.workunit.client.0.smithi032.stdout:4/804: mkdir d4/d87/d96/d2f/d3b/da2/d10f 0 2024-04-12T05:37:17.364 INFO:tasks.workunit.client.0.smithi032.stdout:7/801: unlink d0/de/df/d20/f10e 0 2024-04-12T05:37:17.364 INFO:tasks.workunit.client.0.smithi032.stdout:7/802: read - d0/d13/d21/d81/fdf zero size 2024-04-12T05:37:17.365 INFO:tasks.workunit.client.1.smithi067.stdout:7/206: dwrite d0/f8 [0,4194304] 0 2024-04-12T05:37:17.365 INFO:tasks.workunit.client.1.smithi067.stdout:0/279: mknod dd/d12/d14/d49/c5e 0 2024-04-12T05:37:17.365 INFO:tasks.workunit.client.1.smithi067.stdout:2/205: chown d5/d11/f32 3 1 2024-04-12T05:37:17.365 INFO:tasks.workunit.client.1.smithi067.stdout:7/207: read - d0/d2/d5/d1d/f3a zero size 2024-04-12T05:37:17.365 INFO:tasks.workunit.client.1.smithi067.stdout:7/208: chown d0/d2/d5 183825011 1 2024-04-12T05:37:17.365 INFO:tasks.workunit.client.1.smithi067.stdout:5/330: dread d0/d3a/f4c [0,4194304] 0 2024-04-12T05:37:17.366 INFO:tasks.workunit.client.0.smithi032.stdout:1/684: creat d5/de/d17/d35/d61/d77/fd4 x:0 0 0 2024-04-12T05:37:17.367 INFO:tasks.workunit.client.1.smithi067.stdout:2/206: creat d5/d43/f44 x:0 0 0 2024-04-12T05:37:17.368 INFO:tasks.workunit.client.1.smithi067.stdout:5/331: unlink d0/d3a/f2a 0 2024-04-12T05:37:17.368 INFO:tasks.workunit.client.1.smithi067.stdout:0/280: mkdir dd/d12/d14/d1a/d3d/d41/d5f 0 2024-04-12T05:37:17.368 INFO:tasks.workunit.client.1.smithi067.stdout:9/156: write d1/fd [2155883,69195] 0 2024-04-12T05:37:17.374 INFO:tasks.workunit.client.1.smithi067.stdout:0/281: stat dd/d12/d14/d1a/d3d/c5d 0 2024-04-12T05:37:17.376 INFO:tasks.workunit.client.1.smithi067.stdout:9/157: mkdir d1/d12/d16/d27 0 2024-04-12T05:37:17.377 INFO:tasks.workunit.client.0.smithi032.stdout:7/803: rename d0/de/df/d20/f68 to d0/dff/f111 0 2024-04-12T05:37:17.377 INFO:tasks.workunit.client.1.smithi067.stdout:7/209: getdents d0/d23 0 2024-04-12T05:37:17.380 INFO:tasks.workunit.client.0.smithi032.stdout:7/804: truncate d0/dff/d4c/fc7 345379 0 2024-04-12T05:37:17.381 INFO:tasks.workunit.client.1.smithi067.stdout:2/207: dread d5/da/f3e [0,4194304] 0 2024-04-12T05:37:17.381 INFO:tasks.workunit.client.1.smithi067.stdout:9/158: write d1/d12/d16/f1b [1022845,123986] 0 2024-04-12T05:37:17.381 INFO:tasks.workunit.client.0.smithi032.stdout:7/805: chown d0/d10/d70/d7b/dd2/fdc 1365042 1 2024-04-12T05:37:17.382 INFO:tasks.workunit.client.0.smithi032.stdout:1/685: dread d5/de/d17/d35/d73/f8a [0,4194304] 0 2024-04-12T05:37:17.384 INFO:tasks.workunit.client.1.smithi067.stdout:9/159: chown d1/d12 4545640 1 2024-04-12T05:37:17.384 INFO:tasks.workunit.client.1.smithi067.stdout:2/208: chown c0 11143 1 2024-04-12T05:37:17.384 INFO:tasks.workunit.client.0.smithi032.stdout:7/806: write d0/d13/f8c [26660,31536] 0 2024-04-12T05:37:17.387 INFO:tasks.workunit.client.1.smithi067.stdout:6/198: dwrite dd/d11/f34 [0,4194304] 0 2024-04-12T05:37:17.388 INFO:tasks.workunit.client.0.smithi032.stdout:1/686: mknod d5/de/d17/d35/d3e/db4/cd5 0 2024-04-12T05:37:17.390 INFO:tasks.workunit.client.1.smithi067.stdout:6/199: read - dd/d11/d27/f2e zero size 2024-04-12T05:37:17.391 INFO:tasks.workunit.client.0.smithi032.stdout:7/807: write d0/fe3 [2080038,87371] 0 2024-04-12T05:37:17.392 INFO:tasks.workunit.client.0.smithi032.stdout:7/808: readlink d0/d13/d21/d7c/led 0 2024-04-12T05:37:17.395 INFO:tasks.workunit.client.0.smithi032.stdout:1/687: mkdir d5/de/d17/dd6 0 2024-04-12T05:37:17.396 INFO:tasks.workunit.client.0.smithi032.stdout:8/918: truncate d0/db/dd/d11/d1b/d96/fef 465330 0 2024-04-12T05:37:17.398 INFO:tasks.workunit.client.1.smithi067.stdout:0/282: getdents dd/d12/d14/d51 0 2024-04-12T05:37:17.398 INFO:tasks.workunit.client.1.smithi067.stdout:5/332: truncate d0/d3a/f4c 1432355 0 2024-04-12T05:37:17.398 INFO:tasks.workunit.client.1.smithi067.stdout:2/209: symlink d5/d23/l45 0 2024-04-12T05:37:17.398 INFO:tasks.workunit.client.1.smithi067.stdout:9/160: getdents d1/d12/d16 0 2024-04-12T05:37:17.398 INFO:tasks.workunit.client.1.smithi067.stdout:6/200: read fc [245525,15933] 0 2024-04-12T05:37:17.398 INFO:tasks.workunit.client.1.smithi067.stdout:9/161: chown d1/d12/d16/d23/f24 983528 1 2024-04-12T05:37:17.399 INFO:tasks.workunit.client.0.smithi032.stdout:4/805: write d4/d13/d44/f6a [865608,99205] 0 2024-04-12T05:37:17.399 INFO:tasks.workunit.client.1.smithi067.stdout:0/283: dread - dd/d12/d14/f57 zero size 2024-04-12T05:37:17.399 INFO:tasks.workunit.client.1.smithi067.stdout:5/333: creat d0/d6/f6e x:0 0 0 2024-04-12T05:37:17.399 INFO:tasks.workunit.client.0.smithi032.stdout:1/688: fsync d5/de/f10 0 2024-04-12T05:37:17.400 INFO:tasks.workunit.client.0.smithi032.stdout:7/809: dwrite d0/d34/dbe/fa6 [0,4194304] 0 2024-04-12T05:37:17.401 INFO:tasks.workunit.client.1.smithi067.stdout:0/284: readlink dd/d12/d14/d1a/l1d 0 2024-04-12T05:37:17.403 INFO:tasks.workunit.client.0.smithi032.stdout:4/806: mknod d4/d87/d96/d68/c110 0 2024-04-12T05:37:17.408 INFO:tasks.workunit.client.0.smithi032.stdout:1/689: creat d5/de/d17/d35/d78/fd7 x:0 0 0 2024-04-12T05:37:17.413 INFO:tasks.workunit.client.1.smithi067.stdout:9/162: rename d1/d12/d16 to d1/d28 0 2024-04-12T05:37:17.413 INFO:tasks.workunit.client.1.smithi067.stdout:5/334: write d0/da/d3f/f49 [3045575,1939] 0 2024-04-12T05:37:17.415 INFO:tasks.workunit.client.1.smithi067.stdout:6/201: link dd/d21/d23/f25 dd/d21/d23/f3c 0 2024-04-12T05:37:17.416 INFO:tasks.workunit.client.1.smithi067.stdout:2/210: truncate d5/d11/f2a 256383 0 2024-04-12T05:37:17.416 INFO:tasks.workunit.client.0.smithi032.stdout:8/919: dread d0/db/dd/d11/d1b/d22/d29/ded/f102 [0,4194304] 0 2024-04-12T05:37:17.418 INFO:tasks.workunit.client.1.smithi067.stdout:9/163: unlink d1/f11 0 2024-04-12T05:37:17.418 INFO:tasks.workunit.client.0.smithi032.stdout:1/690: dwrite d5/de/d17/d35/d73/f94 [0,4194304] 0 2024-04-12T05:37:17.418 INFO:tasks.workunit.client.0.smithi032.stdout:8/920: stat d0/db/dd/d11/f2c 0 2024-04-12T05:37:17.419 INFO:tasks.workunit.client.1.smithi067.stdout:6/202: symlink dd/d11/l3d 0 2024-04-12T05:37:17.421 INFO:tasks.workunit.client.1.smithi067.stdout:2/211: chown d5/d11/l15 750062929 1 2024-04-12T05:37:17.421 INFO:tasks.workunit.client.0.smithi032.stdout:8/921: mknod d0/db/d7d/d11f/c13b 0 2024-04-12T05:37:17.422 INFO:tasks.workunit.client.0.smithi032.stdout:1/691: dread - d5/de/d17/d35/d61/d54/f6c zero size 2024-04-12T05:37:17.423 INFO:tasks.workunit.client.1.smithi067.stdout:9/164: symlink d1/d12/l29 0 2024-04-12T05:37:17.425 INFO:tasks.workunit.client.1.smithi067.stdout:9/165: write d1/d13/f1e [5081668,78823] 0 2024-04-12T05:37:17.425 INFO:tasks.workunit.client.1.smithi067.stdout:6/203: symlink dd/d11/d27/l3e 0 2024-04-12T05:37:17.425 INFO:tasks.workunit.client.0.smithi032.stdout:7/810: write d0/dd/f2a [5105293,30009] 0 2024-04-12T05:37:17.428 INFO:tasks.workunit.client.1.smithi067.stdout:2/212: unlink d5/da/de/d14/l25 0 2024-04-12T05:37:17.428 INFO:tasks.workunit.client.0.smithi032.stdout:4/807: truncate d4/d13/d44/fd6 1679825 0 2024-04-12T05:37:17.430 INFO:tasks.workunit.client.1.smithi067.stdout:6/204: mkdir dd/d21/d3f 0 2024-04-12T05:37:17.433 INFO:tasks.workunit.client.0.smithi032.stdout:7/811: dread d0/d34/dbe/fa6 [0,4194304] 0 2024-04-12T05:37:17.435 INFO:tasks.workunit.client.1.smithi067.stdout:6/205: dread dd/d21/d23/f36 [0,4194304] 0 2024-04-12T05:37:17.435 INFO:tasks.workunit.client.1.smithi067.stdout:9/166: symlink d1/d12/d15/l2a 0 2024-04-12T05:37:17.436 INFO:tasks.workunit.client.0.smithi032.stdout:1/692: rename d5/de/d17/d35/d3b/d49/l8b to d5/de/d17/d35/d78/ld8 0 2024-04-12T05:37:17.436 INFO:tasks.workunit.client.0.smithi032.stdout:8/922: link d0/db/dd/d11/d1b/d5f/f67 d0/db/d7d/d111/f13c 0 2024-04-12T05:37:17.440 INFO:tasks.workunit.client.0.smithi032.stdout:7/812: fsync d0/dd/d51/f93 0 2024-04-12T05:37:17.443 INFO:tasks.workunit.client.0.smithi032.stdout:7/813: stat d0/dd/d51/d65/l79 0 2024-04-12T05:37:17.443 INFO:tasks.workunit.client.0.smithi032.stdout:4/808: dwrite d4/d13/ff [4194304,4194304] 0 2024-04-12T05:37:17.445 INFO:tasks.workunit.client.0.smithi032.stdout:1/693: creat d5/de/d17/d35/d3b/d49/fd9 x:0 0 0 2024-04-12T05:37:17.446 INFO:tasks.workunit.client.0.smithi032.stdout:8/923: mkdir d0/db/dd/d11/d1b/d5f/d13d 0 2024-04-12T05:37:17.447 INFO:tasks.workunit.client.0.smithi032.stdout:7/814: creat d0/de/df/d29/f112 x:0 0 0 2024-04-12T05:37:17.453 INFO:tasks.workunit.client.1.smithi067.stdout:9/167: dwrite d1/d28/d23/f24 [0,4194304] 0 2024-04-12T05:37:17.462 INFO:tasks.workunit.client.1.smithi067.stdout:5/335: dwrite d0/d3a/f19 [0,4194304] 0 2024-04-12T05:37:17.462 INFO:tasks.workunit.client.1.smithi067.stdout:9/168: fsync d1/d12/f14 0 2024-04-12T05:37:17.465 INFO:tasks.workunit.client.0.smithi032.stdout:8/924: getdents d0/db/dd/d11/d1b/d22/d54/d136/dad/da4/dcb 0 2024-04-12T05:37:17.470 INFO:tasks.workunit.client.1.smithi067.stdout:2/213: dwrite d5/f12 [0,4194304] 0 2024-04-12T05:37:17.471 INFO:tasks.workunit.client.0.smithi032.stdout:8/925: dwrite d0/f2f [4194304,4194304] 0 2024-04-12T05:37:17.472 INFO:tasks.workunit.client.1.smithi067.stdout:6/206: dwrite fb [0,4194304] 0 2024-04-12T05:37:17.472 INFO:tasks.workunit.client.0.smithi032.stdout:8/926: fsync d0/db/d38/d63/fa1 0 2024-04-12T05:37:17.473 INFO:tasks.workunit.client.0.smithi032.stdout:8/927: readlink d0/db/d38/d63/d77/dd4/l118 0 2024-04-12T05:37:17.478 INFO:tasks.workunit.client.0.smithi032.stdout:8/928: unlink d0/db/dd/d11/d1b/d22/d35/ff6 0 2024-04-12T05:37:17.484 INFO:tasks.workunit.client.1.smithi067.stdout:9/169: symlink d1/l2b 0 2024-04-12T05:37:17.486 INFO:tasks.workunit.client.1.smithi067.stdout:9/170: chown d1/d28/d21 1421 1 2024-04-12T05:37:17.488 INFO:tasks.workunit.client.0.smithi032.stdout:4/809: dwrite d4/d87/d96/d4d/f92 [0,4194304] 0 2024-04-12T05:37:17.488 INFO:tasks.workunit.client.1.smithi067.stdout:6/207: creat dd/d21/d3f/f40 x:0 0 0 2024-04-12T05:37:17.489 INFO:tasks.workunit.client.0.smithi032.stdout:8/929: link d0/db/d7d/d11f/c13b d0/db/dd/d11/d1b/d22/d54/d136/dad/da4/c13e 0 2024-04-12T05:37:17.493 INFO:tasks.workunit.client.0.smithi032.stdout:7/815: write d0/de/df/d29/f89 [2937039,52854] 0 2024-04-12T05:37:17.493 INFO:tasks.workunit.client.0.smithi032.stdout:4/810: fdatasync d4/f3d 0 2024-04-12T05:37:17.493 INFO:tasks.workunit.client.1.smithi067.stdout:5/336: dwrite d0/da/d41/f6b [0,4194304] 0 2024-04-12T05:37:17.495 INFO:tasks.workunit.client.0.smithi032.stdout:4/811: fdatasync d4/f79 0 2024-04-12T05:37:17.495 INFO:tasks.workunit.client.1.smithi067.stdout:2/214: dwrite d5/da/de/d14/f3b [0,4194304] 0 2024-04-12T05:37:17.497 INFO:tasks.workunit.client.0.smithi032.stdout:9/643: sync 2024-04-12T05:37:17.497 INFO:tasks.workunit.client.0.smithi032.stdout:3/860: sync 2024-04-12T05:37:17.497 INFO:tasks.workunit.client.1.smithi067.stdout:6/208: mknod dd/d30/c41 0 2024-04-12T05:37:17.501 INFO:tasks.workunit.client.0.smithi032.stdout:7/816: dread d0/d34/dbe/fa6 [0,4194304] 0 2024-04-12T05:37:17.506 INFO:tasks.workunit.client.1.smithi067.stdout:6/209: creat dd/d21/d23/f42 x:0 0 0 2024-04-12T05:37:17.510 INFO:tasks.workunit.client.1.smithi067.stdout:2/215: mkdir d5/da/de/d14/d46 0 2024-04-12T05:37:17.511 INFO:tasks.workunit.client.1.smithi067.stdout:6/210: write dd/d11/d27/f2e [446754,122113] 0 2024-04-12T05:37:17.511 INFO:tasks.workunit.client.1.smithi067.stdout:5/337: dread - d0/d6/d20/f58 zero size 2024-04-12T05:37:17.514 INFO:tasks.workunit.client.1.smithi067.stdout:5/338: truncate d0/d6/f12 4838894 0 2024-04-12T05:37:17.514 INFO:tasks.workunit.client.1.smithi067.stdout:2/216: fsync d5/da/de/d14/d34/f37 0 2024-04-12T05:37:17.516 INFO:tasks.workunit.client.0.smithi032.stdout:1/694: dread d5/de/d17/f26 [0,4194304] 0 2024-04-12T05:37:17.516 INFO:tasks.workunit.client.0.smithi032.stdout:1/695: stat d5/d2c/f3c 0 2024-04-12T05:37:17.518 INFO:tasks.workunit.client.1.smithi067.stdout:5/339: symlink d0/d6/d20/d62/l6f 0 2024-04-12T05:37:17.520 INFO:tasks.workunit.client.1.smithi067.stdout:5/340: mkdir d0/d6/d70 0 2024-04-12T05:37:17.524 INFO:tasks.workunit.client.1.smithi067.stdout:5/341: creat d0/d6/d33/f71 x:0 0 0 2024-04-12T05:37:17.524 INFO:tasks.workunit.client.1.smithi067.stdout:2/217: write d5/da/de/d14/f3b [4292007,24317] 0 2024-04-12T05:37:17.525 INFO:tasks.workunit.client.1.smithi067.stdout:9/171: write d1/d28/d21/f26 [2700905,3429] 0 2024-04-12T05:37:17.528 INFO:tasks.workunit.client.0.smithi032.stdout:9/644: write d9/d58/f79 [307723,3681] 0 2024-04-12T05:37:17.528 INFO:tasks.workunit.client.0.smithi032.stdout:8/930: truncate d0/db/dd/d11/d1b/d22/d54/da8/fc7 2560381 0 2024-04-12T05:37:17.529 INFO:tasks.workunit.client.1.smithi067.stdout:2/218: read d5/da/de/f13 [70112,3174] 0 2024-04-12T05:37:17.529 INFO:tasks.workunit.client.0.smithi032.stdout:4/812: truncate d4/d87/d96/d32/f5f 7768754 0 2024-04-12T05:37:17.534 INFO:tasks.workunit.client.0.smithi032.stdout:4/813: mkdir d4/d13/d75/d111 0 2024-04-12T05:37:17.534 INFO:tasks.workunit.client.0.smithi032.stdout:3/861: dwrite d12/f2a [0,4194304] 0 2024-04-12T05:37:17.537 INFO:tasks.workunit.client.1.smithi067.stdout:4/110: sync 2024-04-12T05:37:17.537 INFO:tasks.workunit.client.0.smithi032.stdout:7/817: dwrite d0/d13/d21/d81/f9d [0,4194304] 0 2024-04-12T05:37:17.540 INFO:tasks.workunit.client.0.smithi032.stdout:9/645: dread d9/d15/d2c/dab/fbd [0,4194304] 0 2024-04-12T05:37:17.540 INFO:tasks.workunit.client.1.smithi067.stdout:9/172: dread d1/d13/f1e [0,4194304] 0 2024-04-12T05:37:17.540 INFO:tasks.workunit.client.1.smithi067.stdout:2/219: mkdir d5/da/de/d47 0 2024-04-12T05:37:17.544 INFO:tasks.workunit.client.1.smithi067.stdout:6/211: dwrite dd/fe [0,4194304] 0 2024-04-12T05:37:17.545 INFO:tasks.workunit.client.0.smithi032.stdout:3/862: rename d12/d2f/d89/f104 to d12/d2f/de2/d107/f114 0 2024-04-12T05:37:17.545 INFO:tasks.workunit.client.0.smithi032.stdout:9/646: chown d9/d15/d2c/d41 1563 1 2024-04-12T05:37:17.545 INFO:tasks.workunit.client.0.smithi032.stdout:3/863: write d12/daa/f108 [1085972,55296] 0 2024-04-12T05:37:17.548 INFO:tasks.workunit.client.0.smithi032.stdout:3/864: symlink d12/d24/dde/l115 0 2024-04-12T05:37:17.548 INFO:tasks.workunit.client.0.smithi032.stdout:4/814: mknod d4/d87/d96/d2f/d3b/c112 0 2024-04-12T05:37:17.549 INFO:tasks.workunit.client.1.smithi067.stdout:9/173: creat d1/d28/f2c x:0 0 0 2024-04-12T05:37:17.551 INFO:tasks.workunit.client.0.smithi032.stdout:3/865: creat d12/d2f/d89/df0/d98/f116 x:0 0 0 2024-04-12T05:37:17.551 INFO:tasks.workunit.client.1.smithi067.stdout:4/111: symlink d8/da/d1d/l24 0 2024-04-12T05:37:17.553 INFO:tasks.workunit.client.1.smithi067.stdout:4/112: readlink d8/d9/lb 0 2024-04-12T05:37:17.555 INFO:tasks.workunit.client.0.smithi032.stdout:3/866: truncate d12/d2f/d89/df0/f67 2870434 0 2024-04-12T05:37:17.555 INFO:tasks.workunit.client.1.smithi067.stdout:6/212: creat dd/d21/d23/f43 x:0 0 0 2024-04-12T05:37:17.556 INFO:tasks.workunit.client.0.smithi032.stdout:9/647: mkdir d9/d15/d1a/d24/d27/d88/dd9 0 2024-04-12T05:37:17.557 INFO:tasks.workunit.client.0.smithi032.stdout:3/867: mkdir d12/d117 0 2024-04-12T05:37:17.558 INFO:tasks.workunit.client.0.smithi032.stdout:4/815: rmdir d4/d87/d96/d2f/d3b 39 2024-04-12T05:37:17.559 INFO:tasks.workunit.client.1.smithi067.stdout:9/174: dread d1/d13/f1e [0,4194304] 0 2024-04-12T05:37:17.559 INFO:tasks.workunit.client.1.smithi067.stdout:5/342: dread d0/da/d3f/f49 [0,4194304] 0 2024-04-12T05:37:17.563 INFO:tasks.workunit.client.1.smithi067.stdout:2/220: link d5/d11/c1b d5/c48 0 2024-04-12T05:37:17.563 INFO:tasks.workunit.client.1.smithi067.stdout:5/343: readlink d0/d6/d33/l42 0 2024-04-12T05:37:17.564 INFO:tasks.workunit.client.1.smithi067.stdout:2/221: chown d5/d11/f2e 179 1 2024-04-12T05:37:17.564 INFO:tasks.workunit.client.1.smithi067.stdout:6/213: link dd/l2a dd/l44 0 2024-04-12T05:37:17.565 INFO:tasks.workunit.client.0.smithi032.stdout:7/818: getdents d0/d10/d70/d7b 0 2024-04-12T05:37:17.566 INFO:tasks.workunit.client.1.smithi067.stdout:9/175: rmdir d1/d28/d23 39 2024-04-12T05:37:17.567 INFO:tasks.workunit.client.0.smithi032.stdout:8/931: write d0/db/dd/d11/d1b/d22/d54/d136/dad/d130/fd0 [1258866,15133] 0 2024-04-12T05:37:17.568 INFO:tasks.workunit.client.0.smithi032.stdout:7/819: symlink d0/d34/dbf/l113 0 2024-04-12T05:37:17.568 INFO:tasks.workunit.client.1.smithi067.stdout:4/113: dwrite d8/da/f1f [0,4194304] 0 2024-04-12T05:37:17.569 INFO:tasks.workunit.client.0.smithi032.stdout:7/820: chown d0/dd/d51/d65/fad 0 1 2024-04-12T05:37:17.570 INFO:tasks.workunit.client.1.smithi067.stdout:5/344: creat d0/d6/d33/f72 x:0 0 0 2024-04-12T05:37:17.570 INFO:tasks.workunit.client.0.smithi032.stdout:9/648: dread d9/d15/d2c/d41/f96 [0,4194304] 0 2024-04-12T05:37:17.572 INFO:tasks.workunit.client.1.smithi067.stdout:6/214: rename l2 to dd/d11/l45 0 2024-04-12T05:37:17.573 INFO:tasks.workunit.client.0.smithi032.stdout:9/649: chown d9/d15/d1a/dd2/l5c 7465970 1 2024-04-12T05:37:17.573 INFO:tasks.workunit.client.0.smithi032.stdout:4/816: rename d4/d87/d96/d2f/d3b/da2/le5 to d4/d87/d96/d2f/d7c/dc2/l113 0 2024-04-12T05:37:17.576 INFO:tasks.workunit.client.1.smithi067.stdout:5/345: symlink d0/d3a/d2c/l73 0 2024-04-12T05:37:17.578 INFO:tasks.workunit.client.0.smithi032.stdout:4/817: mkdir d4/d87/d96/d4d/db5/dcf/d114 0 2024-04-12T05:37:17.579 INFO:tasks.workunit.client.0.smithi032.stdout:8/932: getdents d0/db/dd/d11/d1b/d22/d29 0 2024-04-12T05:37:17.579 INFO:tasks.workunit.client.0.smithi032.stdout:3/868: dread d12/d2f/d89/df0/f1e [0,4194304] 0 2024-04-12T05:37:17.580 INFO:tasks.workunit.client.1.smithi067.stdout:4/114: getdents d8/d9 0 2024-04-12T05:37:17.581 INFO:tasks.workunit.client.1.smithi067.stdout:9/176: getdents d1/d12/d15 0 2024-04-12T05:37:17.583 INFO:tasks.workunit.client.0.smithi032.stdout:8/933: creat d0/db/d38/d63/d77/f13f x:0 0 0 2024-04-12T05:37:17.584 INFO:tasks.workunit.client.1.smithi067.stdout:5/346: fdatasync d0/d3a/d2c/f3d 0 2024-04-12T05:37:17.588 INFO:tasks.workunit.client.1.smithi067.stdout:5/347: fsync d0/da/f6d 0 2024-04-12T05:37:17.588 INFO:tasks.workunit.client.1.smithi067.stdout:6/215: dwrite dd/d21/f24 [0,4194304] 0 2024-04-12T05:37:17.590 INFO:tasks.workunit.client.1.smithi067.stdout:5/348: readlink d0/d6/d40/d4b/l39 0 2024-04-12T05:37:17.592 INFO:tasks.workunit.client.1.smithi067.stdout:4/115: link d8/da/d1d/d15/f1a d8/d9/f25 0 2024-04-12T05:37:17.593 INFO:tasks.workunit.client.1.smithi067.stdout:6/216: creat dd/d11/d27/f46 x:0 0 0 2024-04-12T05:37:17.594 INFO:tasks.workunit.client.0.smithi032.stdout:3/869: rename d12/d2f/d89/df0/d98 to d12/d118 0 2024-04-12T05:37:17.595 INFO:tasks.workunit.client.0.smithi032.stdout:7/821: truncate d0/d34/dbe/fa6 2505388 0 2024-04-12T05:37:17.596 INFO:tasks.workunit.client.0.smithi032.stdout:3/870: chown d12/d2f/d89/df0/dcb/le1 373 1 2024-04-12T05:37:17.596 INFO:tasks.workunit.client.1.smithi067.stdout:5/349: fdatasync d0/d6/d33/f52 0 2024-04-12T05:37:17.596 INFO:tasks.workunit.client.1.smithi067.stdout:5/350: readlink d0/da/d3f/l46 0 2024-04-12T05:37:17.597 INFO:tasks.workunit.client.0.smithi032.stdout:3/871: write d12/fe6 [2796042,59404] 0 2024-04-12T05:37:17.597 INFO:tasks.workunit.client.1.smithi067.stdout:9/177: creat d1/d28/f2d x:0 0 0 2024-04-12T05:37:17.603 INFO:tasks.workunit.client.0.smithi032.stdout:7/822: truncate d0/dd/d51/f93 88522 0 2024-04-12T05:37:17.605 INFO:tasks.workunit.client.1.smithi067.stdout:6/217: dwrite dd/d21/f24 [0,4194304] 0 2024-04-12T05:37:17.605 INFO:tasks.workunit.client.0.smithi032.stdout:7/823: write d0/dd/f2a [3512171,93706] 0 2024-04-12T05:37:17.606 INFO:tasks.workunit.client.0.smithi032.stdout:7/824: chown d0/d34/dbe 51611 1 2024-04-12T05:37:17.607 INFO:tasks.workunit.client.1.smithi067.stdout:5/351: mknod d0/d6/d40/d4b/d38/d4e/c74 0 2024-04-12T05:37:17.608 INFO:tasks.workunit.client.0.smithi032.stdout:8/934: dwrite d0/db/dd/d11/d1b/d5f/f67 [0,4194304] 0 2024-04-12T05:37:17.609 INFO:tasks.workunit.client.1.smithi067.stdout:2/222: truncate d5/da/de/d14/f3b 2698651 0 2024-04-12T05:37:17.610 INFO:tasks.workunit.client.0.smithi032.stdout:8/935: chown d0/db/dd/d11/d1b/d22/d29/f133 15 1 2024-04-12T05:37:17.613 INFO:tasks.workunit.client.0.smithi032.stdout:3/872: link d12/d2f/d89/df0/d3a/f31 d12/d2f/d89/df0/d3a/d63/da6/f119 0 2024-04-12T05:37:17.615 INFO:tasks.workunit.client.1.smithi067.stdout:5/352: chown d0/d6/c4d 630094 1 2024-04-12T05:37:17.615 INFO:tasks.workunit.client.1.smithi067.stdout:5/353: read d0/da/d41/f6b [2778926,29274] 0 2024-04-12T05:37:17.615 INFO:tasks.workunit.client.1.smithi067.stdout:6/218: symlink dd/d21/d23/l47 0 2024-04-12T05:37:17.615 INFO:tasks.workunit.client.1.smithi067.stdout:6/219: readlink l1 0 2024-04-12T05:37:17.616 INFO:tasks.workunit.client.0.smithi032.stdout:8/936: write d0/db/dd/d11/d1b/d5f/deb/f50 [8309248,55209] 0 2024-04-12T05:37:17.616 INFO:tasks.workunit.client.1.smithi067.stdout:6/220: fsync dd/f3b 0 2024-04-12T05:37:17.617 INFO:tasks.workunit.client.1.smithi067.stdout:4/116: truncate d8/da/d1d/d15/f1a 3200685 0 2024-04-12T05:37:17.618 INFO:tasks.workunit.client.0.smithi032.stdout:4/818: dread d4/d13/ff [0,4194304] 0 2024-04-12T05:37:17.623 INFO:tasks.workunit.client.0.smithi032.stdout:8/937: symlink d0/d58/df7/l140 0 2024-04-12T05:37:17.624 INFO:tasks.workunit.client.0.smithi032.stdout:8/938: fsync d0/db/dd/d11/f2c 0 2024-04-12T05:37:17.626 INFO:tasks.workunit.client.1.smithi067.stdout:9/178: truncate d1/d28/f1b 907025 0 2024-04-12T05:37:17.627 INFO:tasks.workunit.client.1.smithi067.stdout:9/179: readlink d1/d12/l29 0 2024-04-12T05:37:17.631 INFO:tasks.workunit.client.0.smithi032.stdout:8/939: link d0/db/d38/d63/d77/dd4/f123 d0/db/d7d/dfb/f141 0 2024-04-12T05:37:17.633 INFO:tasks.workunit.client.0.smithi032.stdout:8/940: fdatasync d0/db/dd/d11/d1b/d22/d54/f122 0 2024-04-12T05:37:17.633 INFO:tasks.workunit.client.1.smithi067.stdout:9/180: dread d1/d12/d15/f1c [0,4194304] 0 2024-04-12T05:37:17.634 INFO:tasks.workunit.client.1.smithi067.stdout:2/223: write d5/da/f3e [2112195,44128] 0 2024-04-12T05:37:17.635 INFO:tasks.workunit.client.1.smithi067.stdout:4/117: read d8/da/d1d/d15/f1a [671662,36690] 0 2024-04-12T05:37:17.636 INFO:tasks.workunit.client.0.smithi032.stdout:3/873: write d12/d2f/d89/df0/f7b [406307,18025] 0 2024-04-12T05:37:17.640 INFO:tasks.workunit.client.1.smithi067.stdout:9/181: readlink d1/d12/l29 0 2024-04-12T05:37:17.640 INFO:tasks.workunit.client.0.smithi032.stdout:4/819: dread d4/d13/f71 [0,4194304] 0 2024-04-12T05:37:17.641 INFO:tasks.workunit.client.1.smithi067.stdout:4/118: write d8/da/f18 [930881,7820] 0 2024-04-12T05:37:17.643 INFO:tasks.workunit.client.1.smithi067.stdout:4/119: fdatasync d8/da/f1f 0 2024-04-12T05:37:17.649 INFO:tasks.workunit.client.1.smithi067.stdout:5/354: dread d0/d6/f10 [0,4194304] 0 2024-04-12T05:37:17.650 INFO:tasks.workunit.client.1.smithi067.stdout:4/120: rename d8/da/d1d/d15 to d8/da/d1d/d15/d1c/d26 22 2024-04-12T05:37:17.651 INFO:tasks.workunit.client.1.smithi067.stdout:9/182: link d1/d28/d21/f26 d1/d12/f2e 0 2024-04-12T05:37:17.653 INFO:tasks.workunit.client.1.smithi067.stdout:2/224: dwrite d5/d11/f29 [0,4194304] 0 2024-04-12T05:37:17.653 INFO:tasks.workunit.client.0.smithi032.stdout:4/820: fdatasync d4/f1d 0 2024-04-12T05:37:17.654 INFO:tasks.workunit.client.0.smithi032.stdout:8/941: write d0/db/d38/d63/d77/f97 [811015,8233] 0 2024-04-12T05:37:17.656 INFO:tasks.workunit.client.1.smithi067.stdout:5/355: dread d0/d3a/d2c/f3d [0,4194304] 0 2024-04-12T05:37:17.657 INFO:tasks.workunit.client.1.smithi067.stdout:2/225: write d5/d11/f2e [102220,124447] 0 2024-04-12T05:37:17.659 INFO:tasks.workunit.client.0.smithi032.stdout:8/942: unlink d0/db/dd/lac 0 2024-04-12T05:37:17.660 INFO:tasks.workunit.client.1.smithi067.stdout:4/121: dread d8/d9/f11 [0,4194304] 0 2024-04-12T05:37:17.660 INFO:tasks.workunit.client.1.smithi067.stdout:9/183: symlink d1/d13/l2f 0 2024-04-12T05:37:17.662 INFO:tasks.workunit.client.0.smithi032.stdout:4/821: dwrite d4/d13/d75/fd8 [0,4194304] 0 2024-04-12T05:37:17.664 INFO:tasks.workunit.client.0.smithi032.stdout:3/874: dread d12/d2f/d89/df0/d3a/d63/da6/f119 [0,4194304] 0 2024-04-12T05:37:17.665 INFO:tasks.workunit.client.0.smithi032.stdout:8/943: symlink d0/db/d7d/dfb/l142 0 2024-04-12T05:37:17.665 INFO:tasks.workunit.client.0.smithi032.stdout:5/911: sync 2024-04-12T05:37:17.666 INFO:tasks.workunit.client.0.smithi032.stdout:0/807: sync 2024-04-12T05:37:17.666 INFO:tasks.workunit.client.1.smithi067.stdout:4/122: mkdir d8/da/d1d/d27 0 2024-04-12T05:37:17.667 INFO:tasks.workunit.client.0.smithi032.stdout:3/875: chown d12/d2f/d89/df0/d3a/d63/dd5/cdf 5 1 2024-04-12T05:37:17.668 INFO:tasks.workunit.client.1.smithi067.stdout:2/226: link d5/da/de/d14/l3c d5/da/de/d14/l49 0 2024-04-12T05:37:17.668 INFO:tasks.workunit.client.0.smithi032.stdout:3/876: dread - d12/d2f/de2/fe5 zero size 2024-04-12T05:37:17.670 INFO:tasks.workunit.client.0.smithi032.stdout:0/808: mknod d2/dc/d3e/dac/d51/d83/c10a 0 2024-04-12T05:37:17.671 INFO:tasks.workunit.client.0.smithi032.stdout:0/809: write d2/d29/d8a/d99/fbf [571230,109801] 0 2024-04-12T05:37:17.672 INFO:tasks.workunit.client.0.smithi032.stdout:5/912: rename d2/df/d25/d43/d96 to d2/df/d25/d43/d124 0 2024-04-12T05:37:17.674 INFO:tasks.workunit.client.0.smithi032.stdout:8/944: rename d0/db/dd/d11/d1b/d22/d35/cc2 to d0/db/dd/d11/d1b/d22/d29/ded/c143 0 2024-04-12T05:37:17.676 INFO:tasks.workunit.client.0.smithi032.stdout:0/810: readlink d2/d8/d17/d1b/l27 0 2024-04-12T05:37:17.677 INFO:tasks.workunit.client.1.smithi067.stdout:9/184: dwrite d1/f6 [0,4194304] 0 2024-04-12T05:37:17.678 INFO:tasks.workunit.client.0.smithi032.stdout:8/945: dread d0/db/d7d/dfb/f141 [0,4194304] 0 2024-04-12T05:37:17.685 INFO:tasks.workunit.client.1.smithi067.stdout:4/123: rename d8/da/d1d/l24 to d8/da/d1d/d15/d1c/l28 0 2024-04-12T05:37:17.685 INFO:tasks.workunit.client.1.smithi067.stdout:2/227: write d5/d11/f32 [462175,12122] 0 2024-04-12T05:37:17.689 INFO:tasks.workunit.client.1.smithi067.stdout:4/124: mknod d8/d9/c29 0 2024-04-12T05:37:17.690 INFO:tasks.workunit.client.1.smithi067.stdout:5/356: truncate d0/da/f5a 57997 0 2024-04-12T05:37:17.693 INFO:tasks.workunit.client.1.smithi067.stdout:2/228: rename d5/da/de/l2f to d5/l4a 0 2024-04-12T05:37:17.693 INFO:tasks.workunit.client.1.smithi067.stdout:9/185: dwrite d1/d12/d15/f1d [0,4194304] 0 2024-04-12T05:37:17.694 INFO:tasks.workunit.client.1.smithi067.stdout:4/125: creat d8/d9/f2a x:0 0 0 2024-04-12T05:37:17.696 INFO:tasks.workunit.client.1.smithi067.stdout:4/126: chown d8/da/d1d/d15 1 1 2024-04-12T05:37:17.696 INFO:tasks.workunit.client.1.smithi067.stdout:2/229: write d5/da/de/f31 [4750567,20773] 0 2024-04-12T05:37:17.708 INFO:tasks.workunit.client.0.smithi032.stdout:0/811: write d2/d29/d8a/d99/fa3 [2222914,48783] 0 2024-04-12T05:37:17.710 INFO:tasks.workunit.client.1.smithi067.stdout:5/357: mkdir d0/d6/d20/d62/d75 0 2024-04-12T05:37:17.717 INFO:tasks.workunit.client.0.smithi032.stdout:3/877: dread d12/d2f/d89/df0/f36 [0,4194304] 0 2024-04-12T05:37:17.720 INFO:tasks.workunit.client.0.smithi032.stdout:4/822: dread d4/d13/d44/fa0 [0,4194304] 0 2024-04-12T05:37:17.725 INFO:tasks.workunit.client.0.smithi032.stdout:8/946: dread d0/db/f60 [4194304,4194304] 0 2024-04-12T05:37:17.725 INFO:tasks.workunit.client.0.smithi032.stdout:5/913: dread d2/f68 [0,4194304] 0 2024-04-12T05:37:17.730 INFO:tasks.workunit.client.0.smithi032.stdout:4/823: mknod d4/d13/dc9/c115 0 2024-04-12T05:37:17.731 INFO:tasks.workunit.client.0.smithi032.stdout:5/914: read d2/df/f32 [2444792,3464] 0 2024-04-12T05:37:17.733 INFO:tasks.workunit.client.0.smithi032.stdout:8/947: link d0/db/dd/d11/d1b/d5f/deb/f50 d0/d72/f144 0 2024-04-12T05:37:17.735 INFO:tasks.workunit.client.0.smithi032.stdout:4/824: creat d4/d87/d96/d32/dc5/f116 x:0 0 0 2024-04-12T05:37:17.736 INFO:tasks.workunit.client.0.smithi032.stdout:4/825: chown d4/d87/d96/d68/la9 29004 1 2024-04-12T05:37:17.736 INFO:tasks.workunit.client.0.smithi032.stdout:5/915: creat d2/df/da9/dc1/f125 x:0 0 0 2024-04-12T05:37:17.739 INFO:tasks.workunit.client.0.smithi032.stdout:5/916: creat d2/d9b/df0/f126 x:0 0 0 2024-04-12T05:37:17.740 INFO:tasks.workunit.client.0.smithi032.stdout:5/917: write d2/d9b/f5a [4324218,96623] 0 2024-04-12T05:37:17.741 INFO:tasks.workunit.client.0.smithi032.stdout:5/918: fdatasync d2/dd/d5d/fce 0 2024-04-12T05:37:17.742 INFO:tasks.workunit.client.0.smithi032.stdout:5/919: fdatasync d2/df/d31/fcf 0 2024-04-12T05:37:17.744 INFO:tasks.workunit.client.0.smithi032.stdout:0/812: dread d2/dc/d3e/dac/f35 [0,4194304] 0 2024-04-12T05:37:17.745 INFO:tasks.workunit.client.0.smithi032.stdout:5/920: write d2/df/fb1 [3921720,38222] 0 2024-04-12T05:37:17.745 INFO:tasks.workunit.client.0.smithi032.stdout:0/813: write d2/d28/d87/d79/f106 [461847,66900] 0 2024-04-12T05:37:17.749 INFO:tasks.workunit.client.0.smithi032.stdout:5/921: symlink d2/df/d31/da2/dc0/l127 0 2024-04-12T05:37:17.750 INFO:tasks.workunit.client.0.smithi032.stdout:0/814: rename d2/d28/dd6/ff5 to d2/d8/d17/d1b/d109/f10b 0 2024-04-12T05:37:17.751 INFO:tasks.workunit.client.0.smithi032.stdout:0/815: chown d2/dc/d3e/dac/d51 214989 1 2024-04-12T05:37:17.753 INFO:tasks.workunit.client.0.smithi032.stdout:0/816: mknod d2/d28/dd6/c10c 0 2024-04-12T05:37:17.754 INFO:tasks.workunit.client.0.smithi032.stdout:5/922: symlink d2/df/da9/dc1/d78/l128 0 2024-04-12T05:37:17.761 INFO:tasks.workunit.client.0.smithi032.stdout:8/948: write d0/db/dd/d11/d1b/d22/d51/f59 [342573,126514] 0 2024-04-12T05:37:17.764 INFO:tasks.workunit.client.0.smithi032.stdout:5/923: write d2/df/d25/d1e/f2f [5863451,100219] 0 2024-04-12T05:37:17.765 INFO:tasks.workunit.client.0.smithi032.stdout:8/949: link d0/db/dd/d11/d1b/d5f/deb/c4b d0/db/dd/d12b/c145 0 2024-04-12T05:37:17.767 INFO:tasks.workunit.client.1.smithi067.stdout:3/178: sync 2024-04-12T05:37:17.767 INFO:tasks.workunit.client.1.smithi067.stdout:1/202: sync 2024-04-12T05:37:17.770 INFO:tasks.workunit.client.0.smithi032.stdout:8/950: read d0/db/dd/d11/d1b/d22/d54/d136/db3/fc4 [74110,122612] 0 2024-04-12T05:37:17.771 INFO:tasks.workunit.client.0.smithi032.stdout:8/951: mknod d0/d36/de2/c146 0 2024-04-12T05:37:17.774 INFO:tasks.workunit.client.0.smithi032.stdout:8/952: chown d0/db/dd/d11/d1b/d96/cc5 674099 1 2024-04-12T05:37:17.775 INFO:tasks.workunit.client.0.smithi032.stdout:5/924: dwrite d2/df/da9/dc1/d6c/ff4 [0,4194304] 0 2024-04-12T05:37:17.780 INFO:tasks.workunit.client.0.smithi032.stdout:5/925: creat d2/d3c/f129 x:0 0 0 2024-04-12T05:37:17.783 INFO:tasks.workunit.client.0.smithi032.stdout:5/926: rename d2/df/d25/d43/d124/cbc to d2/dd/c12a 0 2024-04-12T05:37:17.785 INFO:tasks.workunit.client.0.smithi032.stdout:5/927: truncate d2/d3c/d5b/dd4/f10c 677449 0 2024-04-12T05:37:17.786 INFO:tasks.workunit.client.0.smithi032.stdout:8/953: write d0/db/d7d/dfb/f141 [2790651,41225] 0 2024-04-12T05:37:17.787 INFO:tasks.workunit.client.0.smithi032.stdout:5/928: symlink d2/df/d25/d1e/l12b 0 2024-04-12T05:37:17.798 INFO:tasks.workunit.client.0.smithi032.stdout:5/929: truncate d2/df/f123 677297 0 2024-04-12T05:37:17.800 INFO:tasks.workunit.client.0.smithi032.stdout:5/930: mknod d2/df/d31/da2/c12c 0 2024-04-12T05:37:17.801 INFO:tasks.workunit.client.0.smithi032.stdout:5/931: creat d2/df/d25/d50/d91/f12d x:0 0 0 2024-04-12T05:37:17.803 INFO:tasks.workunit.client.0.smithi032.stdout:5/932: write d2/df/d25/d43/d49/f7d [1873438,30187] 0 2024-04-12T05:37:17.805 INFO:tasks.workunit.client.0.smithi032.stdout:5/933: creat d2/df/da9/dc1/d6c/df3/f12e x:0 0 0 2024-04-12T05:37:17.808 INFO:tasks.workunit.client.0.smithi032.stdout:5/934: mknod d2/d3c/d5b/d75/c12f 0 2024-04-12T05:37:17.810 INFO:tasks.workunit.client.0.smithi032.stdout:5/935: dread d2/df/da9/dc1/fd3 [0,4194304] 0 2024-04-12T05:37:17.812 INFO:tasks.workunit.client.0.smithi032.stdout:5/936: write d2/d3c/d5b/d75/fa1 [4415097,119992] 0 2024-04-12T05:37:17.838 INFO:tasks.workunit.client.1.smithi067.stdout:4/127: creat d8/f2b x:0 0 0 2024-04-12T05:37:17.838 INFO:tasks.workunit.client.1.smithi067.stdout:9/186: symlink d1/d28/l30 0 2024-04-12T05:37:17.839 INFO:tasks.workunit.client.1.smithi067.stdout:4/128: write d8/da/f18 [126482,9587] 0 2024-04-12T05:37:17.840 INFO:tasks.workunit.client.0.smithi032.stdout:3/878: symlink d12/d2f/d89/df0/d3a/d2d/l11a 0 2024-04-12T05:37:17.840 INFO:tasks.workunit.client.1.smithi067.stdout:4/129: dread - d8/d9/f2a zero size 2024-04-12T05:37:17.841 INFO:tasks.workunit.client.1.smithi067.stdout:4/130: dread - d8/d9/f2a zero size 2024-04-12T05:37:17.842 INFO:tasks.workunit.client.1.smithi067.stdout:5/358: creat d0/d6/f76 x:0 0 0 2024-04-12T05:37:17.843 INFO:tasks.workunit.client.1.smithi067.stdout:1/203: fdatasync d4/d9/f1c 0 2024-04-12T05:37:17.845 INFO:tasks.workunit.client.1.smithi067.stdout:4/131: write d8/d9/f21 [133303,50788] 0 2024-04-12T05:37:17.847 INFO:tasks.workunit.client.1.smithi067.stdout:5/359: write d0/d6/f6a [87093,46149] 0 2024-04-12T05:37:17.853 INFO:tasks.workunit.client.1.smithi067.stdout:9/187: dwrite d1/d12/d15/f1d [0,4194304] 0 2024-04-12T05:37:17.854 INFO:tasks.workunit.client.1.smithi067.stdout:2/230: dwrite d5/da/f1d [4194304,4194304] 0 2024-04-12T05:37:17.854 INFO:tasks.workunit.client.1.smithi067.stdout:4/132: chown d8/da/d1d/d15/d1c/l28 155 1 2024-04-12T05:37:17.855 INFO:tasks.workunit.client.1.smithi067.stdout:9/188: readlink d1/d12/d15/l25 0 2024-04-12T05:37:17.855 INFO:tasks.workunit.client.0.smithi032.stdout:3/879: dwrite d12/d2f/d89/df0/d3a/d63/f83 [0,4194304] 0 2024-04-12T05:37:17.857 INFO:tasks.workunit.client.1.smithi067.stdout:9/189: chown d1/c10 3112 1 2024-04-12T05:37:17.857 INFO:tasks.workunit.client.1.smithi067.stdout:4/133: symlink d8/d9/l2c 0 2024-04-12T05:37:17.861 INFO:tasks.workunit.client.1.smithi067.stdout:5/360: link d0/d6/d20/c50 d0/da/d41/c77 0 2024-04-12T05:37:17.863 INFO:tasks.workunit.client.0.smithi032.stdout:3/880: symlink d12/d2f/d89/l11b 0 2024-04-12T05:37:17.864 INFO:tasks.workunit.client.1.smithi067.stdout:2/231: dread d5/d11/f32 [0,4194304] 0 2024-04-12T05:37:17.864 INFO:tasks.workunit.client.1.smithi067.stdout:4/134: chown d8/d9/f17 500 1 2024-04-12T05:37:17.865 INFO:tasks.workunit.client.1.smithi067.stdout:5/361: symlink d0/d6/l78 0 2024-04-12T05:37:17.866 INFO:tasks.workunit.client.0.smithi032.stdout:3/881: creat d12/d2f/f11c x:0 0 0 2024-04-12T05:37:17.870 INFO:tasks.workunit.client.1.smithi067.stdout:4/135: creat d8/da/d1d/d15/d1c/f2d x:0 0 0 2024-04-12T05:37:17.870 INFO:tasks.workunit.client.1.smithi067.stdout:5/362: dread d0/f7 [0,4194304] 0 2024-04-12T05:37:17.871 INFO:tasks.workunit.client.1.smithi067.stdout:4/136: read - d8/d9/f20 zero size 2024-04-12T05:37:17.872 INFO:tasks.workunit.client.0.smithi032.stdout:3/882: mknod d12/c11d 0 2024-04-12T05:37:17.873 INFO:tasks.workunit.client.1.smithi067.stdout:5/363: write d0/d6/f6a [507165,56731] 0 2024-04-12T05:37:17.873 INFO:tasks.workunit.client.0.smithi032.stdout:2/828: sync 2024-04-12T05:37:17.877 INFO:tasks.workunit.client.1.smithi067.stdout:5/364: unlink d0/d6/d20/c23 0 2024-04-12T05:37:17.877 INFO:tasks.workunit.client.0.smithi032.stdout:3/883: unlink d12/d2f/d89/lf1 0 2024-04-12T05:37:17.878 INFO:tasks.workunit.client.1.smithi067.stdout:5/365: truncate d0/da/f6d 190738 0 2024-04-12T05:37:17.880 INFO:tasks.workunit.client.0.smithi032.stdout:3/884: creat d12/d2f/d89/d9d/f11e x:0 0 0 2024-04-12T05:37:17.881 INFO:tasks.workunit.client.1.smithi067.stdout:4/137: link d8/da/d1d/c1e d8/da/d1d/d15/d1c/c2e 0 2024-04-12T05:37:17.888 INFO:tasks.workunit.client.1.smithi067.stdout:9/190: dread d1/d12/f2e [0,4194304] 0 2024-04-12T05:37:17.890 INFO:tasks.workunit.client.1.smithi067.stdout:9/191: fdatasync d1/fd 0 2024-04-12T05:37:17.892 INFO:tasks.workunit.client.1.smithi067.stdout:4/138: dwrite d8/fe [0,4194304] 0 2024-04-12T05:37:17.894 INFO:tasks.workunit.client.1.smithi067.stdout:4/139: read d8/d9/f11 [22428,33209] 0 2024-04-12T05:37:17.895 INFO:tasks.workunit.client.1.smithi067.stdout:2/232: dread d5/d11/f2e [0,4194304] 0 2024-04-12T05:37:17.896 INFO:tasks.workunit.client.1.smithi067.stdout:1/204: dread d4/d9/f31 [0,4194304] 0 2024-04-12T05:37:17.897 INFO:tasks.workunit.client.1.smithi067.stdout:0/285: sync 2024-04-12T05:37:17.897 INFO:tasks.workunit.client.1.smithi067.stdout:8/152: sync 2024-04-12T05:37:17.897 INFO:tasks.workunit.client.1.smithi067.stdout:7/210: sync 2024-04-12T05:37:17.898 INFO:tasks.workunit.client.1.smithi067.stdout:4/140: write d8/da/f1f [4573096,123334] 0 2024-04-12T05:37:17.899 INFO:tasks.workunit.client.1.smithi067.stdout:8/153: fdatasync d6/d8/d18/df/f23 0 2024-04-12T05:37:17.904 INFO:tasks.workunit.client.1.smithi067.stdout:9/192: dwrite d1/d12/f14 [0,4194304] 0 2024-04-12T05:37:17.969 INFO:tasks.workunit.client.1.smithi067.stdout:2/233: symlink d5/da/de/d14/l4b 0 2024-04-12T05:37:17.970 INFO:tasks.workunit.client.1.smithi067.stdout:5/366: write d0/d6/d20/d62/f14 [2070988,682] 0 2024-04-12T05:37:17.972 INFO:tasks.workunit.client.1.smithi067.stdout:1/205: creat d4/d9/d3a/f3c x:0 0 0 2024-04-12T05:37:17.972 INFO:tasks.workunit.client.1.smithi067.stdout:4/141: rmdir d8/da/d1d 39 2024-04-12T05:37:17.977 INFO:tasks.workunit.client.1.smithi067.stdout:1/206: dwrite d4/d9/d12/f36 [0,4194304] 0 2024-04-12T05:37:17.979 INFO:tasks.workunit.client.1.smithi067.stdout:1/207: write d4/d9/d3a/f1e [4405315,78835] 0 2024-04-12T05:37:18.004 INFO:tasks.workunit.client.0.smithi032.stdout:1/696: sync 2024-04-12T05:37:18.004 INFO:tasks.workunit.client.0.smithi032.stdout:9/650: sync 2024-04-12T05:37:18.005 INFO:tasks.workunit.client.0.smithi032.stdout:9/651: chown d9/f34 542 1 2024-04-12T05:37:18.005 INFO:tasks.workunit.client.0.smithi032.stdout:1/697: read - d5/de/d17/d35/d73/f76 zero size 2024-04-12T05:37:18.008 INFO:tasks.workunit.client.0.smithi032.stdout:1/698: mkdir d5/de/d17/d35/d3b/dda 0 2024-04-12T05:37:18.010 INFO:tasks.workunit.client.0.smithi032.stdout:9/652: getdents d9/d15/d1a/dba 0 2024-04-12T05:37:18.011 INFO:tasks.workunit.client.1.smithi067.stdout:5/367: chown d0/da/f5a 222 1 2024-04-12T05:37:18.014 INFO:tasks.workunit.client.1.smithi067.stdout:9/193: rename d1/d28/f2d to d1/d28/d23/f31 0 2024-04-12T05:37:18.015 INFO:tasks.workunit.client.0.smithi032.stdout:9/653: write d9/d15/d2c/dab/fbd [395051,96026] 0 2024-04-12T05:37:18.015 INFO:tasks.workunit.client.1.smithi067.stdout:4/142: mknod d8/da/d1d/d15/d1c/c2f 0 2024-04-12T05:37:18.016 INFO:tasks.workunit.client.0.smithi032.stdout:1/699: dread d5/de/d17/fae [0,4194304] 0 2024-04-12T05:37:18.017 INFO:tasks.workunit.client.1.smithi067.stdout:1/208: mkdir d4/d3d 0 2024-04-12T05:37:18.018 INFO:tasks.workunit.client.0.smithi032.stdout:9/654: creat d9/d15/d1a/d24/d27/d88/dd9/fda x:0 0 0 2024-04-12T05:37:18.020 INFO:tasks.workunit.client.1.smithi067.stdout:0/286: dwrite dd/d12/f3a [0,4194304] 0 2024-04-12T05:37:18.020 INFO:tasks.workunit.client.1.smithi067.stdout:2/234: getdents d5/d11 0 2024-04-12T05:37:18.022 INFO:tasks.workunit.client.1.smithi067.stdout:9/194: unlink d1/d12/d15/l2a 0 2024-04-12T05:37:18.024 INFO:tasks.workunit.client.0.smithi032.stdout:9/655: link d9/d15/d1a/d1d/faf d9/d15/d1a/d24/d27/d88/dc1/fdb 0 2024-04-12T05:37:18.025 INFO:tasks.workunit.client.1.smithi067.stdout:3/179: sync 2024-04-12T05:37:18.025 INFO:tasks.workunit.client.1.smithi067.stdout:6/221: sync 2024-04-12T05:37:18.027 INFO:tasks.workunit.client.1.smithi067.stdout:3/180: stat dc/d1d/f24 0 2024-04-12T05:37:18.028 INFO:tasks.workunit.client.1.smithi067.stdout:5/368: dread d0/f7 [0,4194304] 0 2024-04-12T05:37:18.028 INFO:tasks.workunit.client.1.smithi067.stdout:8/154: dread d6/d8/d18/df/f20 [0,4194304] 0 2024-04-12T05:37:18.030 INFO:tasks.workunit.client.1.smithi067.stdout:5/369: write d0/da/f68 [952420,123875] 0 2024-04-12T05:37:18.034 INFO:tasks.workunit.client.1.smithi067.stdout:2/235: mkdir d5/da/de/d14/d34/d4c 0 2024-04-12T05:37:18.035 INFO:tasks.workunit.client.1.smithi067.stdout:1/209: symlink d4/d9/d12/d28/l3e 0 2024-04-12T05:37:18.035 INFO:tasks.workunit.client.1.smithi067.stdout:6/222: truncate dd/d21/d23/f36 3277967 0 2024-04-12T05:37:18.035 INFO:tasks.workunit.client.1.smithi067.stdout:3/181: rmdir dc/d1d/d28 39 2024-04-12T05:37:18.035 INFO:tasks.workunit.client.1.smithi067.stdout:8/155: symlink d6/d8/d18/l31 0 2024-04-12T05:37:18.036 INFO:tasks.workunit.client.1.smithi067.stdout:6/223: stat dd/f17 0 2024-04-12T05:37:18.036 INFO:tasks.workunit.client.1.smithi067.stdout:5/370: creat d0/d6/d40/d4b/d38/f79 x:0 0 0 2024-04-12T05:37:18.037 INFO:tasks.workunit.client.1.smithi067.stdout:9/195: mkdir d1/d28/d27/d32 0 2024-04-12T05:37:18.038 INFO:tasks.workunit.client.1.smithi067.stdout:0/287: mknod dd/d12/d14/d1a/d3d/d41/d5f/c60 0 2024-04-12T05:37:18.039 INFO:tasks.workunit.client.0.smithi032.stdout:9/656: write d9/d1f/f39 [1781893,107706] 0 2024-04-12T05:37:18.039 INFO:tasks.workunit.client.1.smithi067.stdout:6/224: chown dd/d30 2 1 2024-04-12T05:37:18.040 INFO:tasks.workunit.client.1.smithi067.stdout:4/143: link d8/d9/f11 d8/da/f30 0 2024-04-12T05:37:18.042 INFO:tasks.workunit.client.1.smithi067.stdout:5/371: creat d0/da/d41/f7a x:0 0 0 2024-04-12T05:37:18.043 INFO:tasks.workunit.client.1.smithi067.stdout:6/225: fsync dd/d21/d3f/f40 0 2024-04-12T05:37:18.043 INFO:tasks.workunit.client.1.smithi067.stdout:1/210: symlink d4/d9/l3f 0 2024-04-12T05:37:18.045 INFO:tasks.workunit.client.1.smithi067.stdout:2/236: creat d5/da/de/d14/d34/d4c/f4d x:0 0 0 2024-04-12T05:37:18.048 INFO:tasks.workunit.client.1.smithi067.stdout:7/211: sync 2024-04-12T05:37:18.048 INFO:tasks.workunit.client.1.smithi067.stdout:4/144: readlink d8/da/d1d/d15/d1c/l23 0 2024-04-12T05:37:18.048 INFO:tasks.workunit.client.1.smithi067.stdout:3/182: link dc/df/l1b dc/df/l35 0 2024-04-12T05:37:18.052 INFO:tasks.workunit.client.1.smithi067.stdout:8/156: rename d6/d8/cd to d6/d2f/c32 0 2024-04-12T05:37:18.053 INFO:tasks.workunit.client.0.smithi032.stdout:9/657: dwrite d9/d15/d1a/d1d/faf [4194304,4194304] 0 2024-04-12T05:37:18.053 INFO:tasks.workunit.client.1.smithi067.stdout:5/372: symlink d0/d6/d40/d4b/d38/d4e/l7b 0 2024-04-12T05:37:18.053 INFO:tasks.workunit.client.1.smithi067.stdout:0/288: link dd/d12/f3e dd/d12/d14/f61 0 2024-04-12T05:37:18.056 INFO:tasks.workunit.client.0.smithi032.stdout:9/658: creat d9/d58/da9/fdc x:0 0 0 2024-04-12T05:37:18.057 INFO:tasks.workunit.client.0.smithi032.stdout:9/659: fsync d9/d15/d2c/d41/fc8 0 2024-04-12T05:37:18.059 INFO:tasks.workunit.client.1.smithi067.stdout:9/196: dread d1/d12/f2e [0,4194304] 0 2024-04-12T05:37:18.059 INFO:tasks.workunit.client.1.smithi067.stdout:2/237: creat d5/f4e x:0 0 0 2024-04-12T05:37:18.061 INFO:tasks.workunit.client.1.smithi067.stdout:7/212: creat d0/d2/d5/d1d/f40 x:0 0 0 2024-04-12T05:37:18.064 INFO:tasks.workunit.client.1.smithi067.stdout:7/213: stat d0/d2/dd 0 2024-04-12T05:37:18.065 INFO:tasks.workunit.client.1.smithi067.stdout:5/373: creat d0/d6/d40/d4b/d38/d4e/f7c x:0 0 0 2024-04-12T05:37:18.068 INFO:tasks.workunit.client.1.smithi067.stdout:3/183: creat dc/d1d/f36 x:0 0 0 2024-04-12T05:37:18.071 INFO:tasks.workunit.client.0.smithi032.stdout:9/660: write d9/d15/d1a/d24/d27/f50 [1137579,110033] 0 2024-04-12T05:37:18.073 INFO:tasks.workunit.client.1.smithi067.stdout:6/226: dwrite fa [0,4194304] 0 2024-04-12T05:37:18.074 INFO:tasks.workunit.client.1.smithi067.stdout:1/211: dwrite d4/d9/d12/f1b [0,4194304] 0 2024-04-12T05:37:18.076 INFO:tasks.workunit.client.1.smithi067.stdout:4/145: dwrite d8/fe [0,4194304] 0 2024-04-12T05:37:18.076 INFO:tasks.workunit.client.1.smithi067.stdout:2/238: unlink d5/da/de/d14/f3b 0 2024-04-12T05:37:18.077 INFO:tasks.workunit.client.1.smithi067.stdout:9/197: mknod d1/d13/c33 0 2024-04-12T05:37:18.088 INFO:tasks.workunit.client.1.smithi067.stdout:9/198: chown d1/d12/f14 2028 1 2024-04-12T05:37:18.088 INFO:tasks.workunit.client.1.smithi067.stdout:9/199: write d1/d28/d23/f24 [3286408,68319] 0 2024-04-12T05:37:18.088 INFO:tasks.workunit.client.1.smithi067.stdout:7/214: dread d0/f8 [0,4194304] 0 2024-04-12T05:37:18.088 INFO:tasks.workunit.client.1.smithi067.stdout:3/184: mknod dc/df/c37 0 2024-04-12T05:37:18.092 INFO:tasks.workunit.client.1.smithi067.stdout:6/227: dread dd/d30/f32 [0,4194304] 0 2024-04-12T05:37:18.092 INFO:tasks.workunit.client.1.smithi067.stdout:4/146: dread d8/d9/f21 [0,4194304] 0 2024-04-12T05:37:18.094 INFO:tasks.workunit.client.1.smithi067.stdout:6/228: dread - dd/f3b zero size 2024-04-12T05:37:18.096 INFO:tasks.workunit.client.1.smithi067.stdout:7/215: dread - d0/d2/d5/db/d34/f3f zero size 2024-04-12T05:37:18.096 INFO:tasks.workunit.client.1.smithi067.stdout:2/239: dread d5/da/f1d [4194304,4194304] 0 2024-04-12T05:37:18.096 INFO:tasks.workunit.client.1.smithi067.stdout:0/289: rmdir dd/d12 39 2024-04-12T05:37:18.099 INFO:tasks.workunit.client.1.smithi067.stdout:1/212: mknod d4/d9/d12/c40 0 2024-04-12T05:37:18.104 INFO:tasks.workunit.client.1.smithi067.stdout:7/216: chown d0/l1b 1051173 1 2024-04-12T05:37:18.105 INFO:tasks.workunit.client.1.smithi067.stdout:2/240: unlink d5/d11/l27 0 2024-04-12T05:37:18.106 INFO:tasks.workunit.client.1.smithi067.stdout:9/200: dread d1/d12/f2e [0,4194304] 0 2024-04-12T05:37:18.108 INFO:tasks.workunit.client.1.smithi067.stdout:5/374: getdents d0/d6/d40/d4b/d38/d4e 0 2024-04-12T05:37:18.110 INFO:tasks.workunit.client.1.smithi067.stdout:1/213: truncate d4/f10 3543573 0 2024-04-12T05:37:18.113 INFO:tasks.workunit.client.1.smithi067.stdout:6/229: symlink dd/d21/d3f/l48 0 2024-04-12T05:37:18.113 INFO:tasks.workunit.client.1.smithi067.stdout:3/185: dwrite dc/df/f30 [0,4194304] 0 2024-04-12T05:37:18.115 INFO:tasks.workunit.client.1.smithi067.stdout:9/201: mkdir d1/d13/d34 0 2024-04-12T05:37:18.115 INFO:tasks.workunit.client.1.smithi067.stdout:8/157: dwrite d6/d8/d18/df/d12/f1d [4194304,4194304] 0 2024-04-12T05:37:18.115 INFO:tasks.workunit.client.1.smithi067.stdout:2/241: mknod d5/da/de/d14/c4f 0 2024-04-12T05:37:18.115 INFO:tasks.workunit.client.0.smithi032.stdout:9/661: dread d9/d15/d1a/d1d/f78 [0,4194304] 0 2024-04-12T05:37:18.117 INFO:tasks.workunit.client.1.smithi067.stdout:0/290: creat dd/d12/d14/d1a/d3d/d41/f62 x:0 0 0 2024-04-12T05:37:18.117 INFO:tasks.workunit.client.1.smithi067.stdout:4/147: dread d8/fe [0,4194304] 0 2024-04-12T05:37:18.118 INFO:tasks.workunit.client.0.smithi032.stdout:0/817: sync 2024-04-12T05:37:18.119 INFO:tasks.workunit.client.0.smithi032.stdout:4/826: sync 2024-04-12T05:37:18.119 INFO:tasks.workunit.client.0.smithi032.stdout:7/825: sync 2024-04-12T05:37:18.119 INFO:tasks.workunit.client.1.smithi067.stdout:9/202: dread - d1/d28/f2c zero size 2024-04-12T05:37:18.121 INFO:tasks.workunit.client.0.smithi032.stdout:7/826: chown d0/de/df/d20/c42 427254737 1 2024-04-12T05:37:18.122 INFO:tasks.workunit.client.1.smithi067.stdout:1/214: chown d4/d9/d14/f21 0 1 2024-04-12T05:37:18.125 INFO:tasks.workunit.client.1.smithi067.stdout:2/242: creat d5/d11/f50 x:0 0 0 2024-04-12T05:37:18.125 INFO:tasks.workunit.client.1.smithi067.stdout:4/148: truncate d8/d9/f17 443109 0 2024-04-12T05:37:18.126 INFO:tasks.workunit.client.1.smithi067.stdout:0/291: truncate dd/d12/d14/d1a/d3d/d41/f62 387358 0 2024-04-12T05:37:18.126 INFO:tasks.workunit.client.1.smithi067.stdout:8/158: dread - d6/d8/d18/df/f24 zero size 2024-04-12T05:37:18.127 INFO:tasks.workunit.client.1.smithi067.stdout:4/149: chown d8/da/d1d/d27 36480170 1 2024-04-12T05:37:18.127 INFO:tasks.workunit.client.1.smithi067.stdout:0/292: stat dd/d40 0 2024-04-12T05:37:18.128 INFO:tasks.workunit.client.0.smithi032.stdout:4/827: dread d4/d13/d75/fd8 [0,4194304] 0 2024-04-12T05:37:18.130 INFO:tasks.workunit.client.0.smithi032.stdout:0/818: creat d2/d29/d8a/d99/f10d x:0 0 0 2024-04-12T05:37:18.130 INFO:tasks.workunit.client.1.smithi067.stdout:5/375: write d0/d3a/d2c/f3d [1002313,19067] 0 2024-04-12T05:37:18.131 INFO:tasks.workunit.client.1.smithi067.stdout:7/217: getdents d0/d2 0 2024-04-12T05:37:18.132 INFO:tasks.workunit.client.1.smithi067.stdout:6/230: dread fa [4194304,4194304] 0 2024-04-12T05:37:18.132 INFO:tasks.workunit.client.1.smithi067.stdout:7/218: readlink d0/l6 0 2024-04-12T05:37:18.134 INFO:tasks.workunit.client.0.smithi032.stdout:7/827: fdatasync d0/dd/d51/f93 0 2024-04-12T05:37:18.134 INFO:tasks.workunit.client.0.smithi032.stdout:9/662: dwrite d9/fa1 [0,4194304] 0 2024-04-12T05:37:18.135 INFO:tasks.workunit.client.0.smithi032.stdout:0/819: mknod d2/d8/df/d24/d60/dc8/c10e 0 2024-04-12T05:37:18.137 INFO:tasks.workunit.client.1.smithi067.stdout:3/186: dread dc/d1d/f24 [0,4194304] 0 2024-04-12T05:37:18.139 INFO:tasks.workunit.client.1.smithi067.stdout:8/159: creat d6/d8/d18/f33 x:0 0 0 2024-04-12T05:37:18.140 INFO:tasks.workunit.client.1.smithi067.stdout:2/243: rename d5/d43/f35 to d5/d23/f51 0 2024-04-12T05:37:18.140 INFO:tasks.workunit.client.0.smithi032.stdout:7/828: write d0/d34/dbe/f7e [3666488,6118] 0 2024-04-12T05:37:18.143 INFO:tasks.workunit.client.1.smithi067.stdout:0/293: symlink dd/d5b/l63 0 2024-04-12T05:37:18.143 INFO:tasks.workunit.client.1.smithi067.stdout:2/244: truncate d5/d11/f50 742609 0 2024-04-12T05:37:18.143 INFO:tasks.workunit.client.1.smithi067.stdout:7/219: unlink d0/d2/d5/db/c2d 0 2024-04-12T05:37:18.144 INFO:tasks.workunit.client.1.smithi067.stdout:7/220: dread - d0/d2/f21 zero size 2024-04-12T05:37:18.144 INFO:tasks.workunit.client.1.smithi067.stdout:2/245: chown d5/da/f3e 1002160 1 2024-04-12T05:37:18.144 INFO:tasks.workunit.client.1.smithi067.stdout:6/231: symlink dd/d21/d23/l49 0 2024-04-12T05:37:18.145 INFO:tasks.workunit.client.1.smithi067.stdout:5/376: mkdir d0/da/d41/d7d 0 2024-04-12T05:37:18.145 INFO:tasks.workunit.client.0.smithi032.stdout:4/828: link d4/l66 d4/d87/d96/d2f/l117 0 2024-04-12T05:37:18.146 INFO:tasks.workunit.client.0.smithi032.stdout:9/663: getdents d9/d32/d6e 0 2024-04-12T05:37:18.147 INFO:tasks.workunit.client.1.smithi067.stdout:2/246: chown d5/da/de/d14/d34/d4c 1 1 2024-04-12T05:37:18.148 INFO:tasks.workunit.client.1.smithi067.stdout:3/187: write dc/d1d/d28/f2a [168505,119967] 0 2024-04-12T05:37:18.148 INFO:tasks.workunit.client.0.smithi032.stdout:4/829: fsync d4/d13/d75/fac 0 2024-04-12T05:37:18.150 INFO:tasks.workunit.client.1.smithi067.stdout:7/221: creat d0/d23/f41 x:0 0 0 2024-04-12T05:37:18.150 INFO:tasks.workunit.client.1.smithi067.stdout:5/377: truncate d0/d6/d33/f71 712651 0 2024-04-12T05:37:18.150 INFO:tasks.workunit.client.0.smithi032.stdout:4/830: stat d4/d87/d96/d68 0 2024-04-12T05:37:18.151 INFO:tasks.workunit.client.0.smithi032.stdout:4/831: stat d4/f3d 0 2024-04-12T05:37:18.151 INFO:tasks.workunit.client.1.smithi067.stdout:6/232: mkdir dd/d11/d27/d4a 0 2024-04-12T05:37:18.151 INFO:tasks.workunit.client.1.smithi067.stdout:3/188: stat dc/d1d/d26/f2f 0 2024-04-12T05:37:18.151 INFO:tasks.workunit.client.1.smithi067.stdout:7/222: fdatasync d0/d2/d5/db/f1f 0 2024-04-12T05:37:18.152 INFO:tasks.workunit.client.1.smithi067.stdout:2/247: fsync d5/d11/f2e 0 2024-04-12T05:37:18.152 INFO:tasks.workunit.client.0.smithi032.stdout:9/664: creat d9/d15/d1a/fdd x:0 0 0 2024-04-12T05:37:18.152 INFO:tasks.workunit.client.1.smithi067.stdout:6/233: chown dd/d21/d23/l49 61 1 2024-04-12T05:37:18.154 INFO:tasks.workunit.client.1.smithi067.stdout:7/223: chown d0/d2/d5/db/d34/f3f 0 1 2024-04-12T05:37:18.154 INFO:tasks.workunit.client.1.smithi067.stdout:7/224: stat d0/f12 0 2024-04-12T05:37:18.156 INFO:tasks.workunit.client.1.smithi067.stdout:2/248: fsync d5/d11/f32 0 2024-04-12T05:37:18.156 INFO:tasks.workunit.client.1.smithi067.stdout:3/189: creat dc/d1d/d28/f38 x:0 0 0 2024-04-12T05:37:18.158 INFO:tasks.workunit.client.0.smithi032.stdout:4/832: getdents d4/d87/d96/d4d/db5/dcf/de3 0 2024-04-12T05:37:18.158 INFO:tasks.workunit.client.1.smithi067.stdout:8/160: fdatasync d6/d8/d18/df/d12/f1d 0 2024-04-12T05:37:18.159 INFO:tasks.workunit.client.1.smithi067.stdout:6/234: symlink dd/d11/d27/d4a/l4b 0 2024-04-12T05:37:18.160 INFO:tasks.workunit.client.1.smithi067.stdout:7/225: write d0/d2/d5/d1e/f22 [8267374,3819] 0 2024-04-12T05:37:18.160 INFO:tasks.workunit.client.0.smithi032.stdout:7/829: dread d0/de/d47/fb0 [0,4194304] 0 2024-04-12T05:37:18.162 INFO:tasks.workunit.client.1.smithi067.stdout:2/249: fdatasync d5/da/de/d14/f36 0 2024-04-12T05:37:18.165 INFO:tasks.workunit.client.0.smithi032.stdout:4/833: dread d4/d87/d96/d2f/d3b/f9e [0,4194304] 0 2024-04-12T05:37:18.166 INFO:tasks.workunit.client.1.smithi067.stdout:7/226: dread - d0/d2/d5/db/d10/f1a zero size 2024-04-12T05:37:18.167 INFO:tasks.workunit.client.1.smithi067.stdout:1/215: write d4/d9/f17 [570586,25249] 0 2024-04-12T05:37:18.167 INFO:tasks.workunit.client.1.smithi067.stdout:9/203: truncate d1/d12/d15/f1d 3128987 0 2024-04-12T05:37:18.168 INFO:tasks.workunit.client.0.smithi032.stdout:7/830: write d0/d10/d70/d7b/dd2/fdc [471798,120797] 0 2024-04-12T05:37:18.169 INFO:tasks.workunit.client.1.smithi067.stdout:2/250: write d5/d11/f50 [1618652,90644] 0 2024-04-12T05:37:18.169 INFO:tasks.workunit.client.0.smithi032.stdout:0/820: dwrite d2/d8/d17/fd8 [0,4194304] 0 2024-04-12T05:37:18.170 INFO:tasks.workunit.client.1.smithi067.stdout:4/150: write d8/da/d1d/d15/f1a [1084706,58176] 0 2024-04-12T05:37:18.171 INFO:tasks.workunit.client.1.smithi067.stdout:5/378: dwrite d0/f7 [0,4194304] 0 2024-04-12T05:37:18.172 INFO:tasks.workunit.client.0.smithi032.stdout:7/831: dread - d0/fc4 zero size 2024-04-12T05:37:18.172 INFO:tasks.workunit.client.1.smithi067.stdout:8/161: unlink d6/d2f/c32 0 2024-04-12T05:37:18.174 INFO:tasks.workunit.client.0.smithi032.stdout:7/832: dread - d0/de/df/d29/f112 zero size 2024-04-12T05:37:18.175 INFO:tasks.workunit.client.0.smithi032.stdout:4/834: getdents d4/d87/d96/d2f/d6e 0 2024-04-12T05:37:18.176 INFO:tasks.workunit.client.1.smithi067.stdout:4/151: fsync d8/d9/f25 0 2024-04-12T05:37:18.177 INFO:tasks.workunit.client.0.smithi032.stdout:4/835: read d4/f54 [807704,96631] 0 2024-04-12T05:37:18.179 INFO:tasks.workunit.client.0.smithi032.stdout:0/821: dwrite d2/d8/df/f5f [4194304,4194304] 0 2024-04-12T05:37:18.180 INFO:tasks.workunit.client.1.smithi067.stdout:3/190: dwrite dc/f23 [0,4194304] 0 2024-04-12T05:37:18.182 INFO:tasks.workunit.client.1.smithi067.stdout:0/294: write dd/f38 [690801,25509] 0 2024-04-12T05:37:18.182 INFO:tasks.workunit.client.1.smithi067.stdout:9/204: mknod d1/d28/d21/c35 0 2024-04-12T05:37:18.184 INFO:tasks.workunit.client.1.smithi067.stdout:7/227: rename d0/d2/d5/d38 to d0/d2/dd/d42 0 2024-04-12T05:37:18.186 INFO:tasks.workunit.client.1.smithi067.stdout:8/162: rmdir d6/d8/d18/df/d12 39 2024-04-12T05:37:18.187 INFO:tasks.workunit.client.1.smithi067.stdout:9/205: chown d1/d12/l29 154218 1 2024-04-12T05:37:18.188 INFO:tasks.workunit.client.1.smithi067.stdout:1/216: mknod d4/d9/c41 0 2024-04-12T05:37:18.190 INFO:tasks.workunit.client.0.smithi032.stdout:0/822: getdents d2/dc/d3e 0 2024-04-12T05:37:18.195 INFO:tasks.workunit.client.1.smithi067.stdout:8/163: readlink d6/d8/d18/l31 0 2024-04-12T05:37:18.198 INFO:tasks.workunit.client.1.smithi067.stdout:6/235: write dd/d21/d23/f3c [737780,104231] 0 2024-04-12T05:37:18.198 INFO:tasks.workunit.client.1.smithi067.stdout:6/236: chown dd/d11 5162534 1 2024-04-12T05:37:18.199 INFO:tasks.workunit.client.0.smithi032.stdout:4/836: dread d4/d13/d75/fac [0,4194304] 0 2024-04-12T05:37:18.200 INFO:tasks.workunit.client.1.smithi067.stdout:2/251: dwrite f4 [0,4194304] 0 2024-04-12T05:37:18.201 INFO:tasks.workunit.client.1.smithi067.stdout:3/191: creat dc/d1d/d28/f39 x:0 0 0 2024-04-12T05:37:18.201 INFO:tasks.workunit.client.0.smithi032.stdout:4/837: truncate d4/f1d 3161529 0 2024-04-12T05:37:18.203 INFO:tasks.workunit.client.1.smithi067.stdout:7/228: mkdir d0/d2/d5/d1d/d43 0 2024-04-12T05:37:18.205 INFO:tasks.workunit.client.1.smithi067.stdout:1/217: fdatasync d4/fe 0 2024-04-12T05:37:18.206 INFO:tasks.workunit.client.1.smithi067.stdout:6/237: creat dd/d21/f4c x:0 0 0 2024-04-12T05:37:18.208 INFO:tasks.workunit.client.1.smithi067.stdout:5/379: creat d0/d3a/f7e x:0 0 0 2024-04-12T05:37:18.210 INFO:tasks.workunit.client.1.smithi067.stdout:6/238: stat dd/d11/l3d 0 2024-04-12T05:37:18.210 INFO:tasks.workunit.client.1.smithi067.stdout:5/380: dread - d0/d6/f76 zero size 2024-04-12T05:37:18.213 INFO:tasks.workunit.client.1.smithi067.stdout:5/381: truncate d0/d6/d33/f71 1745263 0 2024-04-12T05:37:18.214 INFO:tasks.workunit.client.1.smithi067.stdout:7/229: mknod d0/d2/dd/d42/c44 0 2024-04-12T05:37:18.215 INFO:tasks.workunit.client.1.smithi067.stdout:2/252: creat d5/da/de/f52 x:0 0 0 2024-04-12T05:37:18.216 INFO:tasks.workunit.client.1.smithi067.stdout:1/218: mknod d4/d9/d34/c42 0 2024-04-12T05:37:18.219 INFO:tasks.workunit.client.1.smithi067.stdout:8/164: getdents d6/d8 0 2024-04-12T05:37:18.221 INFO:tasks.workunit.client.0.smithi032.stdout:4/838: write d4/d13/d44/fd6 [2517521,41707] 0 2024-04-12T05:37:18.223 INFO:tasks.workunit.client.1.smithi067.stdout:6/239: dwrite dd/d21/d23/f43 [0,4194304] 0 2024-04-12T05:37:18.225 INFO:tasks.workunit.client.0.smithi032.stdout:7/833: dwrite d0/de/df/d29/fcf [0,4194304] 0 2024-04-12T05:37:18.227 INFO:tasks.workunit.client.1.smithi067.stdout:3/192: link dc/d1d/d28/f2a dc/f3a 0 2024-04-12T05:37:18.227 INFO:tasks.workunit.client.1.smithi067.stdout:5/382: creat d0/d6/d40/f7f x:0 0 0 2024-04-12T05:37:18.228 INFO:tasks.workunit.client.1.smithi067.stdout:4/152: write d8/d9/f11 [214068,77871] 0 2024-04-12T05:37:18.229 INFO:tasks.workunit.client.1.smithi067.stdout:6/240: stat dd/d21/d23/l49 0 2024-04-12T05:37:18.232 INFO:tasks.workunit.client.0.smithi032.stdout:7/834: dread d0/de/df/f14 [0,4194304] 0 2024-04-12T05:37:18.235 INFO:tasks.workunit.client.1.smithi067.stdout:2/253: write d5/da/f3e [1108351,4190] 0 2024-04-12T05:37:18.235 INFO:tasks.workunit.client.0.smithi032.stdout:0/823: dread d2/d8/df/d24/f5c [0,4194304] 0 2024-04-12T05:37:18.238 INFO:tasks.workunit.client.1.smithi067.stdout:7/230: truncate d0/d2/d5/db/d34/f3f 844337 0 2024-04-12T05:37:18.240 INFO:tasks.workunit.client.1.smithi067.stdout:0/295: dread dd/d12/f2a [0,4194304] 0 2024-04-12T05:37:18.242 INFO:tasks.workunit.client.1.smithi067.stdout:3/193: symlink dc/d1d/d28/l3b 0 2024-04-12T05:37:18.243 INFO:tasks.workunit.client.0.smithi032.stdout:0/824: getdents d2/d8/df/d24 0 2024-04-12T05:37:18.245 INFO:tasks.workunit.client.0.smithi032.stdout:0/825: symlink d2/dc/d41/l10f 0 2024-04-12T05:37:18.248 INFO:tasks.workunit.client.0.smithi032.stdout:0/826: mknod d2/d28/d87/d79/c110 0 2024-04-12T05:37:18.251 INFO:tasks.workunit.client.1.smithi067.stdout:9/206: dwrite d1/d28/f1b [0,4194304] 0 2024-04-12T05:37:18.253 INFO:tasks.workunit.client.0.smithi032.stdout:4/839: write d4/d87/d96/d2f/d3b/f5d [2077218,4007] 0 2024-04-12T05:37:18.254 INFO:tasks.workunit.client.1.smithi067.stdout:2/254: creat d5/d43/f53 x:0 0 0 2024-04-12T05:37:18.255 INFO:tasks.workunit.client.1.smithi067.stdout:8/165: dwrite d6/d8/d18/df/f1c [4194304,4194304] 0 2024-04-12T05:37:18.256 INFO:tasks.workunit.client.1.smithi067.stdout:6/241: dread dd/d21/f24 [0,4194304] 0 2024-04-12T05:37:18.256 INFO:tasks.workunit.client.1.smithi067.stdout:4/153: mknod d8/da/d1d/c31 0 2024-04-12T05:37:18.257 INFO:tasks.workunit.client.1.smithi067.stdout:5/383: write d0/f2 [2328827,9766] 0 2024-04-12T05:37:18.257 INFO:tasks.workunit.client.1.smithi067.stdout:0/296: unlink dd/f3f 0 2024-04-12T05:37:18.258 INFO:tasks.workunit.client.0.smithi032.stdout:4/840: creat d4/d87/d96/d4d/d78/f118 x:0 0 0 2024-04-12T05:37:18.260 INFO:tasks.workunit.client.1.smithi067.stdout:4/154: dread - d8/d9/f20 zero size 2024-04-12T05:37:18.260 INFO:tasks.workunit.client.1.smithi067.stdout:3/194: rmdir dc 39 2024-04-12T05:37:18.263 INFO:tasks.workunit.client.0.smithi032.stdout:0/827: rename d2/dc to d2/d28/d87/d111 0 2024-04-12T05:37:18.269 INFO:tasks.workunit.client.1.smithi067.stdout:6/242: dread fc [0,4194304] 0 2024-04-12T05:37:18.269 INFO:tasks.workunit.client.1.smithi067.stdout:7/231: getdents d0/d2/d5/d1d/d43 0 2024-04-12T05:37:18.269 INFO:tasks.workunit.client.1.smithi067.stdout:0/297: mkdir dd/d12/d14/d1a/d3d/d41/d64 0 2024-04-12T05:37:18.270 INFO:tasks.workunit.client.0.smithi032.stdout:0/828: mkdir d2/d29/d77/dc7/d112 0 2024-04-12T05:37:18.270 INFO:tasks.workunit.client.0.smithi032.stdout:4/841: mkdir d4/d87/d96/d4d/db5/dcf/d114/d119 0 2024-04-12T05:37:18.271 INFO:tasks.workunit.client.1.smithi067.stdout:6/243: symlink dd/d11/d27/l4d 0 2024-04-12T05:37:18.271 INFO:tasks.workunit.client.0.smithi032.stdout:4/842: readlink d4/d87/d96/d2f/d3b/l6c 0 2024-04-12T05:37:18.272 INFO:tasks.workunit.client.1.smithi067.stdout:4/155: creat d8/da/d1d/d15/f32 x:0 0 0 2024-04-12T05:37:18.273 INFO:tasks.workunit.client.0.smithi032.stdout:4/843: dread - d4/d87/d96/d2f/d94/f10d zero size 2024-04-12T05:37:18.273 INFO:tasks.workunit.client.1.smithi067.stdout:9/207: mknod d1/c36 0 2024-04-12T05:37:18.274 INFO:tasks.workunit.client.0.smithi032.stdout:0/829: mknod d2/d28/c113 0 2024-04-12T05:37:18.274 INFO:tasks.workunit.client.0.smithi032.stdout:0/830: chown d2/d28/l8f 131 1 2024-04-12T05:37:18.277 INFO:tasks.workunit.client.0.smithi032.stdout:5/937: sync 2024-04-12T05:37:18.277 INFO:tasks.workunit.client.0.smithi032.stdout:8/954: sync 2024-04-12T05:37:18.277 INFO:tasks.workunit.client.1.smithi067.stdout:9/208: stat d1/f7 0 2024-04-12T05:37:18.279 INFO:tasks.workunit.client.0.smithi032.stdout:0/831: getdents d2/d29/d8a/d99 0 2024-04-12T05:37:18.280 INFO:tasks.workunit.client.1.smithi067.stdout:2/255: write d5/d23/f51 [363465,38692] 0 2024-04-12T05:37:18.280 INFO:tasks.workunit.client.0.smithi032.stdout:8/955: mknod d0/db/dd/d11/d1b/d96/c147 0 2024-04-12T05:37:18.281 INFO:tasks.workunit.client.0.smithi032.stdout:0/832: dread - d2/d8/df/d24/dc3/ff9 zero size 2024-04-12T05:37:18.284 INFO:tasks.workunit.client.1.smithi067.stdout:1/219: dwrite d4/d9/d12/d28/f33 [0,4194304] 0 2024-04-12T05:37:18.285 INFO:tasks.workunit.client.0.smithi032.stdout:8/956: mkdir d0/db/d39/d148 0 2024-04-12T05:37:18.286 INFO:tasks.workunit.client.0.smithi032.stdout:5/938: dwrite d2/df/d25/d50/d91/f12d [0,4194304] 0 2024-04-12T05:37:18.287 INFO:tasks.workunit.client.0.smithi032.stdout:8/957: truncate d0/f114 411512 0 2024-04-12T05:37:18.288 INFO:tasks.workunit.client.0.smithi032.stdout:0/833: mknod d2/d28/dd6/c114 0 2024-04-12T05:37:18.289 INFO:tasks.workunit.client.1.smithi067.stdout:5/384: dwrite d0/d6/d40/d4b/d38/d4e/f7c [0,4194304] 0 2024-04-12T05:37:18.292 INFO:tasks.workunit.client.1.smithi067.stdout:4/156: creat d8/f33 x:0 0 0 2024-04-12T05:37:18.292 INFO:tasks.workunit.client.0.smithi032.stdout:5/939: mknod d2/df/da9/dc1/d6c/c130 0 2024-04-12T05:37:18.293 INFO:tasks.workunit.client.0.smithi032.stdout:0/834: chown d2/d28/d87/d111/cca 148 1 2024-04-12T05:37:18.294 INFO:tasks.workunit.client.1.smithi067.stdout:7/232: rename d0/d2/d5/db/d10 to d0/d23/d45 0 2024-04-12T05:37:18.294 INFO:tasks.workunit.client.1.smithi067.stdout:5/385: fdatasync d0/d3a/f7e 0 2024-04-12T05:37:18.300 INFO:tasks.workunit.client.1.smithi067.stdout:8/166: write d6/d8/f14 [3387077,77054] 0 2024-04-12T05:37:18.300 INFO:tasks.workunit.client.1.smithi067.stdout:4/157: rmdir d8/da/d1d/d15/d1c 39 2024-04-12T05:37:18.300 INFO:tasks.workunit.client.0.smithi032.stdout:8/958: dread d0/db/d38/d63/d77/f97 [0,4194304] 0 2024-04-12T05:37:18.301 INFO:tasks.workunit.client.1.smithi067.stdout:6/244: dread dd/d21/f24 [0,4194304] 0 2024-04-12T05:37:18.301 INFO:tasks.workunit.client.1.smithi067.stdout:1/220: rename d4/d9/d12/d2c/l2d to d4/d9/d14/l43 0 2024-04-12T05:37:18.302 INFO:tasks.workunit.client.0.smithi032.stdout:5/940: creat d2/dd/f131 x:0 0 0 2024-04-12T05:37:18.303 INFO:tasks.workunit.client.1.smithi067.stdout:4/158: fsync d8/d9/f20 0 2024-04-12T05:37:18.304 INFO:tasks.workunit.client.0.smithi032.stdout:8/959: creat d0/db/dd/d11/d1b/d5f/f149 x:0 0 0 2024-04-12T05:37:18.304 INFO:tasks.workunit.client.1.smithi067.stdout:9/209: getdents d1/d13/d34 0 2024-04-12T05:37:18.304 INFO:tasks.workunit.client.1.smithi067.stdout:7/233: symlink d0/d23/l46 0 2024-04-12T05:37:18.305 INFO:tasks.workunit.client.1.smithi067.stdout:2/256: mknod d5/c54 0 2024-04-12T05:37:18.307 INFO:tasks.workunit.client.1.smithi067.stdout:5/386: chown d0/d3a/l34 1977 1 2024-04-12T05:37:18.309 INFO:tasks.workunit.client.1.smithi067.stdout:8/167: symlink d6/d8/d18/l34 0 2024-04-12T05:37:18.309 INFO:tasks.workunit.client.0.smithi032.stdout:5/941: symlink d2/df/d25/l132 0 2024-04-12T05:37:18.312 INFO:tasks.workunit.client.1.smithi067.stdout:6/245: stat dd/d11/l45 0 2024-04-12T05:37:18.313 INFO:tasks.workunit.client.1.smithi067.stdout:0/298: dwrite dd/d12/d14/d1a/f1e [0,4194304] 0 2024-04-12T05:37:18.314 INFO:tasks.workunit.client.0.smithi032.stdout:5/942: write d2/df/d25/f2c [196907,11836] 0 2024-04-12T05:37:18.315 INFO:tasks.workunit.client.1.smithi067.stdout:4/159: mknod d8/c34 0 2024-04-12T05:37:18.315 INFO:tasks.workunit.client.1.smithi067.stdout:9/210: symlink d1/d13/l37 0 2024-04-12T05:37:18.316 INFO:tasks.workunit.client.0.smithi032.stdout:4/844: dread d4/f3d [0,4194304] 0 2024-04-12T05:37:18.318 INFO:tasks.workunit.client.1.smithi067.stdout:1/221: mkdir d4/d44 0 2024-04-12T05:37:18.319 INFO:tasks.workunit.client.1.smithi067.stdout:3/195: dread f9 [0,4194304] 0 2024-04-12T05:37:18.322 INFO:tasks.workunit.client.1.smithi067.stdout:0/299: dread - dd/d12/d14/f61 zero size 2024-04-12T05:37:18.324 INFO:tasks.workunit.client.1.smithi067.stdout:7/234: dwrite d0/d2/d5/d1d/f3a [0,4194304] 0 2024-04-12T05:37:18.325 INFO:tasks.workunit.client.1.smithi067.stdout:6/246: rmdir dd/d21/d3f 39 2024-04-12T05:37:18.326 INFO:tasks.workunit.client.0.smithi032.stdout:8/960: write d0/db/dd/f5c [1199122,34766] 0 2024-04-12T05:37:18.328 INFO:tasks.workunit.client.1.smithi067.stdout:4/160: symlink d8/da/l35 0 2024-04-12T05:37:18.328 INFO:tasks.workunit.client.1.smithi067.stdout:9/211: write d1/d12/f14 [2600612,88698] 0 2024-04-12T05:37:18.331 INFO:tasks.workunit.client.1.smithi067.stdout:8/168: link d6/d8/d18/df/f23 d6/d8/d18/f35 0 2024-04-12T05:37:18.334 INFO:tasks.workunit.client.1.smithi067.stdout:6/247: write dd/d11/d27/f2f [178439,111803] 0 2024-04-12T05:37:18.337 INFO:tasks.workunit.client.1.smithi067.stdout:8/169: mkdir d6/d8/d18/d36 0 2024-04-12T05:37:18.340 INFO:tasks.workunit.client.0.smithi032.stdout:0/835: dwrite d2/d28/d87/d111/d41/fbc [0,4194304] 0 2024-04-12T05:37:18.341 INFO:tasks.workunit.client.1.smithi067.stdout:9/212: mknod d1/d28/d21/c38 0 2024-04-12T05:37:18.341 INFO:tasks.workunit.client.0.smithi032.stdout:8/961: rename d0/d36/c95 to d0/db/dd/d11/d1b/d22/d51/c14a 0 2024-04-12T05:37:18.341 INFO:tasks.workunit.client.1.smithi067.stdout:1/222: rename d4/d9/d12/f3b to d4/d9/f45 0 2024-04-12T05:37:18.342 INFO:tasks.workunit.client.0.smithi032.stdout:5/943: dread d2/df/fb1 [0,4194304] 0 2024-04-12T05:37:18.342 INFO:tasks.workunit.client.0.smithi032.stdout:4/845: dread d4/f85 [0,4194304] 0 2024-04-12T05:37:18.343 INFO:tasks.workunit.client.1.smithi067.stdout:4/161: mkdir d8/da/d1d/d15/d36 0 2024-04-12T05:37:18.343 INFO:tasks.workunit.client.1.smithi067.stdout:5/387: write d0/da/f5a [347458,124829] 0 2024-04-12T05:37:18.344 INFO:tasks.workunit.client.1.smithi067.stdout:3/196: rename dc/df/c37 to dc/df/c3c 0 2024-04-12T05:37:18.345 INFO:tasks.workunit.client.0.smithi032.stdout:4/846: chown d4/d87/d96/d2f/f4e 250820165 1 2024-04-12T05:37:18.346 INFO:tasks.workunit.client.0.smithi032.stdout:0/836: mkdir d2/d29/d77/dc7/d112/d115 0 2024-04-12T05:37:18.350 INFO:tasks.workunit.client.0.smithi032.stdout:4/847: fsync d4/f35 0 2024-04-12T05:37:18.351 INFO:tasks.workunit.client.1.smithi067.stdout:8/170: write d6/d8/d18/f35 [4492674,36736] 0 2024-04-12T05:37:18.351 INFO:tasks.workunit.client.1.smithi067.stdout:1/223: mkdir d4/d46 0 2024-04-12T05:37:18.351 INFO:tasks.workunit.client.1.smithi067.stdout:9/213: symlink d1/l39 0 2024-04-12T05:37:18.353 INFO:tasks.workunit.client.1.smithi067.stdout:1/224: mknod d4/d9/d34/c47 0 2024-04-12T05:37:18.354 INFO:tasks.workunit.client.0.smithi032.stdout:0/837: symlink d2/d28/d87/d111/d3e/dac/d51/l116 0 2024-04-12T05:37:18.357 INFO:tasks.workunit.client.0.smithi032.stdout:8/962: rename d0/db/dd/d11/d1b/f89 to d0/db/dd/d11/d1b/d22/d54/f14b 0 2024-04-12T05:37:18.358 INFO:tasks.workunit.client.1.smithi067.stdout:9/214: mknod d1/d28/d27/d32/c3a 0 2024-04-12T05:37:18.362 INFO:tasks.workunit.client.0.smithi032.stdout:4/848: link d4/d13/f71 d4/d87/d96/d4d/db5/f11a 0 2024-04-12T05:37:18.363 INFO:tasks.workunit.client.0.smithi032.stdout:0/838: creat d2/d8/d17/d1b/de7/f117 x:0 0 0 2024-04-12T05:37:18.363 INFO:tasks.workunit.client.1.smithi067.stdout:1/225: symlink d4/d9/d3a/l48 0 2024-04-12T05:37:18.363 INFO:tasks.workunit.client.1.smithi067.stdout:6/248: dwrite dd/d21/d23/f42 [0,4194304] 0 2024-04-12T05:37:18.365 INFO:tasks.workunit.client.1.smithi067.stdout:9/215: symlink d1/d13/l3b 0 2024-04-12T05:37:18.366 INFO:tasks.workunit.client.1.smithi067.stdout:4/162: dwrite d8/fe [0,4194304] 0 2024-04-12T05:37:18.366 INFO:tasks.workunit.client.0.smithi032.stdout:0/839: symlink d2/d29/l118 0 2024-04-12T05:37:18.370 INFO:tasks.workunit.client.1.smithi067.stdout:9/216: dread - d1/d28/d23/f31 zero size 2024-04-12T05:37:18.370 INFO:tasks.workunit.client.1.smithi067.stdout:8/171: symlink d6/d8/d18/df/d12/l37 0 2024-04-12T05:37:18.370 INFO:tasks.workunit.client.1.smithi067.stdout:9/217: dread - d1/d28/f2c zero size 2024-04-12T05:37:18.371 INFO:tasks.workunit.client.1.smithi067.stdout:8/172: fsync d6/d8/f14 0 2024-04-12T05:37:18.371 INFO:tasks.workunit.client.0.smithi032.stdout:4/849: rmdir d4/d87/d96/d2f/d7c/d102 0 2024-04-12T05:37:18.374 INFO:tasks.workunit.client.1.smithi067.stdout:1/226: rename d4/d9/c41 to d4/d44/c49 0 2024-04-12T05:37:18.374 INFO:tasks.workunit.client.0.smithi032.stdout:0/840: dwrite d2/d29/f104 [0,4194304] 0 2024-04-12T05:37:18.380 INFO:tasks.workunit.client.1.smithi067.stdout:8/173: rmdir d6/d8 39 2024-04-12T05:37:18.380 INFO:tasks.workunit.client.1.smithi067.stdout:1/227: dread - d4/d9/d12/d2c/f37 zero size 2024-04-12T05:37:18.381 INFO:tasks.workunit.client.0.smithi032.stdout:5/944: write d2/dd/d5d/faf [149185,18403] 0 2024-04-12T05:37:18.381 INFO:tasks.workunit.client.0.smithi032.stdout:4/850: fdatasync d4/d87/d96/d32/dc5/f116 0 2024-04-12T05:37:18.384 INFO:tasks.workunit.client.0.smithi032.stdout:5/945: unlink d2/d9b/fd8 0 2024-04-12T05:37:18.384 INFO:tasks.workunit.client.0.smithi032.stdout:0/841: mkdir d2/d8/d119 0 2024-04-12T05:37:18.385 INFO:tasks.workunit.client.0.smithi032.stdout:8/963: write d0/d36/f4a [605499,119075] 0 2024-04-12T05:37:18.388 INFO:tasks.workunit.client.0.smithi032.stdout:5/946: truncate d2/d3c/d5b/f39 5076273 0 2024-04-12T05:37:18.389 INFO:tasks.workunit.client.1.smithi067.stdout:4/163: dwrite d8/da/d1d/d15/f1a [0,4194304] 0 2024-04-12T05:37:18.389 INFO:tasks.workunit.client.1.smithi067.stdout:4/164: stat d8/da/f30 0 2024-04-12T05:37:18.390 INFO:tasks.workunit.client.0.smithi032.stdout:4/851: rename d4/d87/d96/d4d/db5/f103 to d4/d87/d96/d2f/d3b/da2/d10a/f11b 0 2024-04-12T05:37:18.390 INFO:tasks.workunit.client.1.smithi067.stdout:6/249: getdents dd/d21 0 2024-04-12T05:37:18.390 INFO:tasks.workunit.client.0.smithi032.stdout:5/947: stat d2/df/da9/dc1/fdb 0 2024-04-12T05:37:18.391 INFO:tasks.workunit.client.1.smithi067.stdout:1/228: dread - d4/d9/d3a/f23 zero size 2024-04-12T05:37:18.391 INFO:tasks.workunit.client.1.smithi067.stdout:9/218: dread d1/fd [0,4194304] 0 2024-04-12T05:37:18.391 INFO:tasks.workunit.client.1.smithi067.stdout:5/388: truncate d0/f1c 3980965 0 2024-04-12T05:37:18.392 INFO:tasks.workunit.client.0.smithi032.stdout:0/842: creat d2/d8/df/f11a x:0 0 0 2024-04-12T05:37:18.393 INFO:tasks.workunit.client.0.smithi032.stdout:8/964: getdents d0/db/dd/d11/d1b/d22/d35/d135 0 2024-04-12T05:37:18.393 INFO:tasks.workunit.client.1.smithi067.stdout:8/174: mkdir d6/d8/d18/d38 0 2024-04-12T05:37:18.395 INFO:tasks.workunit.client.0.smithi032.stdout:5/948: creat d2/df/d31/da2/dc0/f133 x:0 0 0 2024-04-12T05:37:18.397 INFO:tasks.workunit.client.1.smithi067.stdout:4/165: write d8/da/d1d/d15/d1c/f2d [571003,112864] 0 2024-04-12T05:37:18.398 INFO:tasks.workunit.client.0.smithi032.stdout:8/965: rmdir d0/db/dd/d11/d1b/d5f 39 2024-04-12T05:37:18.399 INFO:tasks.workunit.client.1.smithi067.stdout:9/219: symlink d1/d28/d27/l3c 0 2024-04-12T05:37:18.399 INFO:tasks.workunit.client.1.smithi067.stdout:8/175: write d6/d8/d18/df/d12/f1d [7039984,6743] 0 2024-04-12T05:37:18.401 INFO:tasks.workunit.client.0.smithi032.stdout:8/966: mknod d0/d36/de2/c14c 0 2024-04-12T05:37:18.403 INFO:tasks.workunit.client.0.smithi032.stdout:5/949: getdents d2/d3c/d5b 0 2024-04-12T05:37:18.403 INFO:tasks.workunit.client.0.smithi032.stdout:5/950: readlink d2/ldc 0 2024-04-12T05:37:18.404 INFO:tasks.workunit.client.0.smithi032.stdout:8/967: mkdir d0/d72/d14d 0 2024-04-12T05:37:18.405 INFO:tasks.workunit.client.0.smithi032.stdout:5/951: chown d2/d3c/d5b/d75/dd6/fe4 547 1 2024-04-12T05:37:18.405 INFO:tasks.workunit.client.1.smithi067.stdout:1/229: creat d4/d9/f4a x:0 0 0 2024-04-12T05:37:18.406 INFO:tasks.workunit.client.0.smithi032.stdout:5/952: fdatasync d2/df/d25/d50/f8e 0 2024-04-12T05:37:18.406 INFO:tasks.workunit.client.1.smithi067.stdout:8/176: fdatasync d6/d8/d18/df/d12/f1d 0 2024-04-12T05:37:18.407 INFO:tasks.workunit.client.1.smithi067.stdout:5/389: rename d0/d6/d20/d3e/f6c to d0/d6/d70/f80 0 2024-04-12T05:37:18.407 INFO:tasks.workunit.client.0.smithi032.stdout:8/968: symlink d0/d36/de2/l14e 0 2024-04-12T05:37:18.408 INFO:tasks.workunit.client.1.smithi067.stdout:4/166: mknod d8/da/d1d/d15/d36/c37 0 2024-04-12T05:37:18.408 INFO:tasks.workunit.client.1.smithi067.stdout:8/177: chown f5 63585 1 2024-04-12T05:37:18.408 INFO:tasks.workunit.client.1.smithi067.stdout:9/220: dread d1/d28/f18 [0,4194304] 0 2024-04-12T05:37:18.411 INFO:tasks.workunit.client.0.smithi032.stdout:5/953: mkdir d2/df/d31/da2/d134 0 2024-04-12T05:37:18.411 INFO:tasks.workunit.client.0.smithi032.stdout:4/852: write d4/d87/d96/fbf [118108,39804] 0 2024-04-12T05:37:18.412 INFO:tasks.workunit.client.1.smithi067.stdout:8/178: readlink d6/d8/d18/df/d12/l37 0 2024-04-12T05:37:18.413 INFO:tasks.workunit.client.1.smithi067.stdout:1/230: creat d4/d3d/f4b x:0 0 0 2024-04-12T05:37:18.414 INFO:tasks.workunit.client.1.smithi067.stdout:9/221: creat d1/d28/d23/f3d x:0 0 0 2024-04-12T05:37:18.414 INFO:tasks.workunit.client.0.smithi032.stdout:8/969: creat d0/db/d38/d63/d77/dd4/f14f x:0 0 0 2024-04-12T05:37:18.415 INFO:tasks.workunit.client.0.smithi032.stdout:0/843: write d2/d28/d87/d111/d3e/dac/f54 [1026745,96524] 0 2024-04-12T05:37:18.417 INFO:tasks.workunit.client.1.smithi067.stdout:4/167: mkdir d8/da/d38 0 2024-04-12T05:37:18.419 INFO:tasks.workunit.client.0.smithi032.stdout:8/970: chown d0/db/dd/d11/d1b/d22/d54/da8 498 1 2024-04-12T05:37:18.420 INFO:tasks.workunit.client.1.smithi067.stdout:4/168: readlink d8/da/d1d/d15/d1c/l28 0 2024-04-12T05:37:18.421 INFO:tasks.workunit.client.0.smithi032.stdout:0/844: fdatasync d2/d29/d77/dc7/fb1 0 2024-04-12T05:37:18.422 INFO:tasks.workunit.client.1.smithi067.stdout:1/231: chown d4/d9/d12/l24 23314788 1 2024-04-12T05:37:18.422 INFO:tasks.workunit.client.1.smithi067.stdout:6/250: dread dd/d11/f19 [0,4194304] 0 2024-04-12T05:37:18.423 INFO:tasks.workunit.client.0.smithi032.stdout:5/954: dwrite d2/df/d25/d1e/f1f [0,4194304] 0 2024-04-12T05:37:18.423 INFO:tasks.workunit.client.1.smithi067.stdout:5/390: read - d0/d6/d70/f80 zero size 2024-04-12T05:37:18.424 INFO:tasks.workunit.client.1.smithi067.stdout:6/251: chown dd/d21 1687596383 1 2024-04-12T05:37:18.424 INFO:tasks.workunit.client.1.smithi067.stdout:1/232: dread - d4/d9/d12/d2c/f37 zero size 2024-04-12T05:37:18.424 INFO:tasks.workunit.client.0.smithi032.stdout:0/845: write d2/d8/d17/d1b/de7/f117 [77871,118561] 0 2024-04-12T05:37:18.425 INFO:tasks.workunit.client.1.smithi067.stdout:4/169: mkdir d8/d39 0 2024-04-12T05:37:18.427 INFO:tasks.workunit.client.0.smithi032.stdout:8/971: write d0/db/dd/d11/d1b/d5f/deb/f90 [1207035,126492] 0 2024-04-12T05:37:18.431 INFO:tasks.workunit.client.0.smithi032.stdout:8/972: chown d0/db/dd/d11/d1b/d22/d35/f87 3731 1 2024-04-12T05:37:18.432 INFO:tasks.workunit.client.1.smithi067.stdout:9/222: link d1/d12/d15/f1d d1/d13/f3e 0 2024-04-12T05:37:18.432 INFO:tasks.workunit.client.0.smithi032.stdout:8/973: chown d0/db/dd/d11/d1b/d5f/deb/c9f 62791 1 2024-04-12T05:37:18.433 INFO:tasks.workunit.client.0.smithi032.stdout:8/974: read d0/db/dd/d11/d1b/d22/d54/f14b [2125031,66542] 0 2024-04-12T05:37:18.434 INFO:tasks.workunit.client.0.smithi032.stdout:0/846: link d2/d28/d87/d111/d3e/dac/d51/d83/cad d2/d29/d77/c11b 0 2024-04-12T05:37:18.434 INFO:tasks.workunit.client.1.smithi067.stdout:1/233: mkdir d4/d9/d4c 0 2024-04-12T05:37:18.434 INFO:tasks.workunit.client.0.smithi032.stdout:8/975: truncate d0/f2f 9042155 0 2024-04-12T05:37:18.435 INFO:tasks.workunit.client.0.smithi032.stdout:8/976: readlink d0/db/dd/d11/d1b/d22/d54/ld8 0 2024-04-12T05:37:18.436 INFO:tasks.workunit.client.0.smithi032.stdout:0/847: mkdir d2/d28/d87/d79/d11c 0 2024-04-12T05:37:18.439 INFO:tasks.workunit.client.1.smithi067.stdout:9/223: mkdir d1/d28/d21/d3f 0 2024-04-12T05:37:18.440 INFO:tasks.workunit.client.1.smithi067.stdout:4/170: rename d8/da/d1d/d15/f32 to d8/da/d38/f3a 0 2024-04-12T05:37:18.441 INFO:tasks.workunit.client.0.smithi032.stdout:0/848: symlink d2/d8/d119/l11d 0 2024-04-12T05:37:18.442 INFO:tasks.workunit.client.0.smithi032.stdout:0/849: fdatasync d2/d29/d8a/d99/fa3 0 2024-04-12T05:37:18.443 INFO:tasks.workunit.client.0.smithi032.stdout:0/850: stat d2/d29/dcc/f105 0 2024-04-12T05:37:18.444 INFO:tasks.workunit.client.1.smithi067.stdout:4/171: chown d8/d9 350 1 2024-04-12T05:37:18.444 INFO:tasks.workunit.client.0.smithi032.stdout:4/853: dread d4/d87/d96/d2f/f60 [0,4194304] 0 2024-04-12T05:37:18.444 INFO:tasks.workunit.client.1.smithi067.stdout:5/391: dwrite d0/d6/d33/f3c [0,4194304] 0 2024-04-12T05:37:18.446 INFO:tasks.workunit.client.1.smithi067.stdout:6/252: dread dd/d21/f24 [0,4194304] 0 2024-04-12T05:37:18.447 INFO:tasks.workunit.client.0.smithi032.stdout:0/851: write d2/d8/df/d24/f5c [590036,40112] 0 2024-04-12T05:37:18.447 INFO:tasks.workunit.client.0.smithi032.stdout:4/854: chown d4/d87/c8b 50 1 2024-04-12T05:37:18.450 INFO:tasks.workunit.client.1.smithi067.stdout:6/253: write dd/d21/d23/f3c [84294,11451] 0 2024-04-12T05:37:18.450 INFO:tasks.workunit.client.1.smithi067.stdout:4/172: creat d8/d39/f3b x:0 0 0 2024-04-12T05:37:18.452 INFO:tasks.workunit.client.0.smithi032.stdout:0/852: mknod d2/d29/d77/dc2/c11e 0 2024-04-12T05:37:18.455 INFO:tasks.workunit.client.0.smithi032.stdout:2/829: sync 2024-04-12T05:37:18.455 INFO:tasks.workunit.client.0.smithi032.stdout:9/665: sync 2024-04-12T05:37:18.455 INFO:tasks.workunit.client.0.smithi032.stdout:1/700: sync 2024-04-12T05:37:18.455 INFO:tasks.workunit.client.0.smithi032.stdout:3/885: sync 2024-04-12T05:37:18.456 INFO:tasks.workunit.client.1.smithi067.stdout:9/224: mknod d1/d28/d21/d3f/c40 0 2024-04-12T05:37:18.456 INFO:tasks.workunit.client.0.smithi032.stdout:0/853: readlink d2/d8/df/l73 0 2024-04-12T05:37:18.458 INFO:tasks.workunit.client.1.smithi067.stdout:4/173: mkdir d8/da/d38/d3c 0 2024-04-12T05:37:18.458 INFO:tasks.workunit.client.1.smithi067.stdout:9/225: chown d1/d13/l3b 1326 1 2024-04-12T05:37:18.459 INFO:tasks.workunit.client.0.smithi032.stdout:9/666: mkdir d9/d15/dde 0 2024-04-12T05:37:18.460 INFO:tasks.workunit.client.1.smithi067.stdout:8/179: dread d6/d8/f16 [0,4194304] 0 2024-04-12T05:37:18.460 INFO:tasks.workunit.client.1.smithi067.stdout:5/392: symlink d0/da/l81 0 2024-04-12T05:37:18.461 INFO:tasks.workunit.client.0.smithi032.stdout:3/886: chown d12/d2f/d89/df0/d3a/d63/f8d 429474 1 2024-04-12T05:37:18.461 INFO:tasks.workunit.client.0.smithi032.stdout:2/830: symlink dd/d19/d9d/d1c/de0/l10b 0 2024-04-12T05:37:18.461 INFO:tasks.workunit.client.1.smithi067.stdout:4/174: mkdir d8/d9/d3d 0 2024-04-12T05:37:18.462 INFO:tasks.workunit.client.1.smithi067.stdout:8/180: stat f5 0 2024-04-12T05:37:18.465 INFO:tasks.workunit.client.1.smithi067.stdout:9/226: write d1/d12/f2e [1843029,65030] 0 2024-04-12T05:37:18.465 INFO:tasks.workunit.client.1.smithi067.stdout:4/175: creat d8/d39/f3e x:0 0 0 2024-04-12T05:37:18.466 INFO:tasks.workunit.client.1.smithi067.stdout:6/254: getdents dd/d21/d23 0 2024-04-12T05:37:18.466 INFO:tasks.workunit.client.0.smithi032.stdout:1/701: rename d5/de/d17/d35/d61/fc4 to d5/de/d17/d35/d61/d77/d8f/d9c/fdb 0 2024-04-12T05:37:18.466 INFO:tasks.workunit.client.0.smithi032.stdout:5/955: dwrite d2/df/d25/fd1 [0,4194304] 0 2024-04-12T05:37:18.467 INFO:tasks.workunit.client.1.smithi067.stdout:8/181: mknod d6/d2f/c39 0 2024-04-12T05:37:18.467 INFO:tasks.workunit.client.0.smithi032.stdout:0/854: truncate d2/d28/d87/d79/fce 4982446 0 2024-04-12T05:37:18.468 INFO:tasks.workunit.client.1.smithi067.stdout:1/234: dread d4/d9/d14/f21 [0,4194304] 0 2024-04-12T05:37:18.469 INFO:tasks.workunit.client.1.smithi067.stdout:4/176: write d8/da/f18 [98575,105368] 0 2024-04-12T05:37:18.471 INFO:tasks.workunit.client.1.smithi067.stdout:5/393: rename d0/d6/d70 to d0/da/d3f/d82 0 2024-04-12T05:37:18.471 INFO:tasks.workunit.client.0.smithi032.stdout:0/855: dread - d2/d28/d87/d111/d3e/dac/d51/fcb zero size 2024-04-12T05:37:18.471 INFO:tasks.workunit.client.0.smithi032.stdout:2/831: rmdir dd/d19/d20 39 2024-04-12T05:37:18.472 INFO:tasks.workunit.client.0.smithi032.stdout:3/887: creat d12/d2f/d89/df0/dcd/f11f x:0 0 0 2024-04-12T05:37:18.474 INFO:tasks.workunit.client.1.smithi067.stdout:5/394: chown d0/d3a/d2c/f3d 2558 1 2024-04-12T05:37:18.475 INFO:tasks.workunit.client.1.smithi067.stdout:6/255: symlink dd/d21/l4e 0 2024-04-12T05:37:18.477 INFO:tasks.workunit.client.1.smithi067.stdout:9/227: creat d1/d28/d21/d3f/f41 x:0 0 0 2024-04-12T05:37:18.477 INFO:tasks.workunit.client.0.smithi032.stdout:9/667: rmdir d9/d15/dde 0 2024-04-12T05:37:18.478 INFO:tasks.workunit.client.1.smithi067.stdout:8/182: dread d6/d8/d18/df/d12/f1d [4194304,4194304] 0 2024-04-12T05:37:18.478 INFO:tasks.workunit.client.1.smithi067.stdout:1/235: mkdir d4/d9/d4d 0 2024-04-12T05:37:18.479 INFO:tasks.workunit.client.1.smithi067.stdout:4/177: truncate d8/d9/f21 493104 0 2024-04-12T05:37:18.479 INFO:tasks.workunit.client.0.smithi032.stdout:5/956: creat d2/df/d119/f135 x:0 0 0 2024-04-12T05:37:18.480 INFO:tasks.workunit.client.1.smithi067.stdout:6/256: symlink dd/d21/d3f/l4f 0 2024-04-12T05:37:18.481 INFO:tasks.workunit.client.1.smithi067.stdout:9/228: mknod d1/d28/d27/c42 0 2024-04-12T05:37:18.481 INFO:tasks.workunit.client.0.smithi032.stdout:1/702: unlink d5/de/d2a/c8c 0 2024-04-12T05:37:18.483 INFO:tasks.workunit.client.1.smithi067.stdout:8/183: fdatasync d6/f27 0 2024-04-12T05:37:18.483 INFO:tasks.workunit.client.0.smithi032.stdout:0/856: truncate d2/f6 118760 0 2024-04-12T05:37:18.485 INFO:tasks.workunit.client.0.smithi032.stdout:3/888: mknod d12/d2f/d89/df0/d3a/d63/da6/dac/d10e/c120 0 2024-04-12T05:37:18.485 INFO:tasks.workunit.client.0.smithi032.stdout:8/977: dwrite d0/db/dd/d11/d1b/d5f/f93 [0,4194304] 0 2024-04-12T05:37:18.488 INFO:tasks.workunit.client.1.smithi067.stdout:1/236: dread - d4/d9/f22 zero size 2024-04-12T05:37:18.488 INFO:tasks.workunit.client.1.smithi067.stdout:9/229: rename d1/d13/f1e to d1/d12/d15/f43 0 2024-04-12T05:37:18.492 INFO:tasks.workunit.client.0.smithi032.stdout:4/855: dwrite d4/f3d [0,4194304] 0 2024-04-12T05:37:18.492 INFO:tasks.workunit.client.0.smithi032.stdout:5/957: symlink d2/d3c/d5b/dd4/df2/l136 0 2024-04-12T05:37:18.494 INFO:tasks.workunit.client.1.smithi067.stdout:5/395: creat d0/d3a/f83 x:0 0 0 2024-04-12T05:37:18.495 INFO:tasks.workunit.client.1.smithi067.stdout:4/178: link f6 d8/d9/d3d/f3f 0 2024-04-12T05:37:18.497 INFO:tasks.workunit.client.1.smithi067.stdout:9/230: dread d1/d28/d23/f24 [0,4194304] 0 2024-04-12T05:37:18.499 INFO:tasks.workunit.client.0.smithi032.stdout:3/889: write d12/d2f/d89/df0/f36 [1269755,24929] 0 2024-04-12T05:37:18.499 INFO:tasks.workunit.client.0.smithi032.stdout:2/832: dread dd/d19/d9d/d42/fe9 [0,4194304] 0 2024-04-12T05:37:18.501 INFO:tasks.workunit.client.0.smithi032.stdout:1/703: fdatasync d5/de/d17/d35/d73/f76 0 2024-04-12T05:37:18.501 INFO:tasks.workunit.client.0.smithi032.stdout:3/890: fsync d12/d118/f9a 0 2024-04-12T05:37:18.501 INFO:tasks.workunit.client.1.smithi067.stdout:5/396: mknod d0/d6/d40/c84 0 2024-04-12T05:37:18.503 INFO:tasks.workunit.client.0.smithi032.stdout:8/978: truncate d0/db/d38/d63/d77/f134 200004 0 2024-04-12T05:37:18.503 INFO:tasks.workunit.client.0.smithi032.stdout:7/835: sync 2024-04-12T05:37:18.504 INFO:tasks.workunit.client.1.smithi067.stdout:8/184: mknod d6/d8/d18/df/c3a 0 2024-04-12T05:37:18.505 INFO:tasks.workunit.client.0.smithi032.stdout:1/704: chown d5/de/d17/d1b/l24 0 1 2024-04-12T05:37:18.505 INFO:tasks.workunit.client.1.smithi067.stdout:6/257: creat dd/f50 x:0 0 0 2024-04-12T05:37:18.506 INFO:tasks.workunit.client.0.smithi032.stdout:1/705: stat d5/de/d17/d35/d3e/db4/cc1 0 2024-04-12T05:37:18.507 INFO:tasks.workunit.client.1.smithi067.stdout:4/179: creat d8/da/d1d/d15/d1c/f40 x:0 0 0 2024-04-12T05:37:18.508 INFO:tasks.workunit.client.0.smithi032.stdout:0/857: rename d2/d8/d17/d1b/d20/lcd to d2/d29/d77/dc2/l11f 0 2024-04-12T05:37:18.509 INFO:tasks.workunit.client.0.smithi032.stdout:9/668: truncate d9/d1f/f84 3410974 0 2024-04-12T05:37:18.510 INFO:tasks.workunit.client.0.smithi032.stdout:2/833: truncate f9 1527096 0 2024-04-12T05:37:18.510 INFO:tasks.workunit.client.0.smithi032.stdout:3/891: rename d12/f13 to d12/d2f/d89/df0/f121 0 2024-04-12T05:37:18.511 INFO:tasks.workunit.client.1.smithi067.stdout:9/231: mkdir d1/d28/d44 0 2024-04-12T05:37:18.511 INFO:tasks.workunit.client.1.smithi067.stdout:6/258: write dd/fe [421453,69922] 0 2024-04-12T05:37:18.511 INFO:tasks.workunit.client.1.smithi067.stdout:8/185: truncate d6/d8/d18/f33 998127 0 2024-04-12T05:37:18.511 INFO:tasks.workunit.client.1.smithi067.stdout:9/232: rmdir d1/d12 39 2024-04-12T05:37:18.515 INFO:tasks.workunit.client.1.smithi067.stdout:6/259: mkdir dd/d11/d27/d4a/d51 0 2024-04-12T05:37:18.516 INFO:tasks.workunit.client.0.smithi032.stdout:0/858: unlink d2/d8/df/d24/f5c 0 2024-04-12T05:37:18.517 INFO:tasks.workunit.client.1.smithi067.stdout:4/180: unlink d8/da/d1d/d15/d1c/f2d 0 2024-04-12T05:37:18.519 INFO:tasks.workunit.client.1.smithi067.stdout:8/186: mknod d6/c3b 0 2024-04-12T05:37:18.519 INFO:tasks.workunit.client.1.smithi067.stdout:5/397: dwrite d0/d6/f6a [0,4194304] 0 2024-04-12T05:37:18.520 INFO:tasks.workunit.client.1.smithi067.stdout:6/260: stat dd/d30/f32 0 2024-04-12T05:37:18.520 INFO:tasks.workunit.client.0.smithi032.stdout:1/706: rename d5/f13 to d5/de/d5b/fdc 0 2024-04-12T05:37:18.521 INFO:tasks.workunit.client.1.smithi067.stdout:5/398: fsync d0/d3a/f19 0 2024-04-12T05:37:18.521 INFO:tasks.workunit.client.1.smithi067.stdout:8/187: stat d6/f7 0 2024-04-12T05:37:18.522 INFO:tasks.workunit.client.0.smithi032.stdout:3/892: getdents d12/d2f/d89/df0/d3a/d63/da6/dac/dc9 0 2024-04-12T05:37:18.522 INFO:tasks.workunit.client.1.smithi067.stdout:9/233: truncate d1/d12/d15/f43 5999693 0 2024-04-12T05:37:18.525 INFO:tasks.workunit.client.1.smithi067.stdout:5/399: mknod d0/d3a/d2c/c85 0 2024-04-12T05:37:18.526 INFO:tasks.workunit.client.0.smithi032.stdout:3/893: creat d12/d2f/d89/df0/dcd/f122 x:0 0 0 2024-04-12T05:37:18.527 INFO:tasks.workunit.client.0.smithi032.stdout:3/894: readlink d12/d2f/d89/df0/dcb/le1 0 2024-04-12T05:37:18.529 INFO:tasks.workunit.client.0.smithi032.stdout:3/895: write d12/d2f/d89/df0/f99 [148185,85135] 0 2024-04-12T05:37:18.530 INFO:tasks.workunit.client.1.smithi067.stdout:4/181: rename d8/d39/f3e to d8/d9/f41 0 2024-04-12T05:37:18.530 INFO:tasks.workunit.client.1.smithi067.stdout:8/188: mkdir d6/d8/d18/d3c 0 2024-04-12T05:37:18.532 INFO:tasks.workunit.client.1.smithi067.stdout:4/182: fdatasync d8/da/f1f 0 2024-04-12T05:37:18.532 INFO:tasks.workunit.client.0.smithi032.stdout:3/896: symlink d12/daa/db3/l123 0 2024-04-12T05:37:18.533 INFO:tasks.workunit.client.1.smithi067.stdout:8/189: chown d6/d8/d18/f25 407625691 1 2024-04-12T05:37:18.533 INFO:tasks.workunit.client.0.smithi032.stdout:8/979: truncate d0/db/d38/d63/fa1 1164931 0 2024-04-12T05:37:18.534 INFO:tasks.workunit.client.1.smithi067.stdout:9/234: dread d1/d12/d15/f43 [0,4194304] 0 2024-04-12T05:37:18.540 INFO:tasks.workunit.client.1.smithi067.stdout:5/400: link d0/d6/d20/d62/c61 d0/d3a/d2c/c86 0 2024-04-12T05:37:18.541 INFO:tasks.workunit.client.0.smithi032.stdout:9/669: write f7 [8015598,64425] 0 2024-04-12T05:37:18.542 INFO:tasks.workunit.client.1.smithi067.stdout:8/190: symlink d6/d8/d18/d3c/l3d 0 2024-04-12T05:37:18.542 INFO:tasks.workunit.client.1.smithi067.stdout:4/183: dread d8/da/f18 [0,4194304] 0 2024-04-12T05:37:18.545 INFO:tasks.workunit.client.1.smithi067.stdout:5/401: symlink d0/l87 0 2024-04-12T05:37:18.546 INFO:tasks.workunit.client.0.smithi032.stdout:2/834: write dd/d19/d9d/d1c/f76 [1209028,86428] 0 2024-04-12T05:37:18.547 INFO:tasks.workunit.client.0.smithi032.stdout:1/707: creat d5/de/d2a/fdd x:0 0 0 2024-04-12T05:37:18.547 INFO:tasks.workunit.client.0.smithi032.stdout:3/897: rmdir d12/daa/db3 39 2024-04-12T05:37:18.549 INFO:tasks.workunit.client.0.smithi032.stdout:5/958: link d2/df/da9/dc1/fde d2/df/d31/d80/f137 0 2024-04-12T05:37:18.551 INFO:tasks.workunit.client.1.smithi067.stdout:9/235: dwrite d1/d28/d21/d3f/f41 [0,4194304] 0 2024-04-12T05:37:18.551 INFO:tasks.workunit.client.1.smithi067.stdout:4/184: write d8/d9/f17 [422021,127206] 0 2024-04-12T05:37:18.552 INFO:tasks.workunit.client.0.smithi032.stdout:4/856: getdents d4/d13/d75 0 2024-04-12T05:37:18.553 INFO:tasks.workunit.client.1.smithi067.stdout:5/402: mkdir d0/d6/d33/d88 0 2024-04-12T05:37:18.555 INFO:tasks.workunit.client.1.smithi067.stdout:9/236: fdatasync d1/d12/f2e 0 2024-04-12T05:37:18.556 INFO:tasks.workunit.client.0.smithi032.stdout:2/835: dwrite dd/f16 [0,4194304] 0 2024-04-12T05:37:18.556 INFO:tasks.workunit.client.0.smithi032.stdout:3/898: creat d12/f124 x:0 0 0 2024-04-12T05:37:18.557 INFO:tasks.workunit.client.0.smithi032.stdout:8/980: creat d0/d58/f150 x:0 0 0 2024-04-12T05:37:18.557 INFO:tasks.workunit.client.1.smithi067.stdout:5/403: readlink d0/da/l81 0 2024-04-12T05:37:18.557 INFO:tasks.workunit.client.1.smithi067.stdout:4/185: fdatasync d8/da/d1d/d15/d1c/f40 0 2024-04-12T05:37:18.562 INFO:tasks.workunit.client.1.smithi067.stdout:8/191: dwrite d6/d8/d18/f35 [4194304,4194304] 0 2024-04-12T05:37:18.563 INFO:tasks.workunit.client.0.smithi032.stdout:3/899: dread d12/daa/f10c [0,4194304] 0 2024-04-12T05:37:18.563 INFO:tasks.workunit.client.0.smithi032.stdout:4/857: mkdir d4/d87/d96/d68/d9b/d11c 0 2024-04-12T05:37:18.563 INFO:tasks.workunit.client.0.smithi032.stdout:7/836: getdents d0/d13/d21/d30/de0 0 2024-04-12T05:37:18.569 INFO:tasks.workunit.client.0.smithi032.stdout:2/836: write dd/d19/d9d/d1c/d64/d36/f85 [3021125,36959] 0 2024-04-12T05:37:18.570 INFO:tasks.workunit.client.0.smithi032.stdout:1/708: write d5/de/d17/d35/d3b/f63 [3332676,10674] 0 2024-04-12T05:37:18.574 INFO:tasks.workunit.client.0.smithi032.stdout:8/981: fdatasync d0/db/f6b 0 2024-04-12T05:37:18.575 INFO:tasks.workunit.client.0.smithi032.stdout:3/900: dwrite d12/d24/f47 [0,4194304] 0 2024-04-12T05:37:18.577 INFO:tasks.workunit.client.0.smithi032.stdout:3/901: stat d12/d2f/d89 0 2024-04-12T05:37:18.577 INFO:tasks.workunit.client.0.smithi032.stdout:7/837: dread d0/de/df/d29/f7d [0,4194304] 0 2024-04-12T05:37:18.579 INFO:tasks.workunit.client.1.smithi067.stdout:9/237: mkdir d1/d28/d21/d45 0 2024-04-12T05:37:18.579 INFO:tasks.workunit.client.1.smithi067.stdout:5/404: creat d0/d6/d20/d3e/d69/f89 x:0 0 0 2024-04-12T05:37:18.579 INFO:tasks.workunit.client.1.smithi067.stdout:4/186: creat d8/d9/f42 x:0 0 0 2024-04-12T05:37:18.581 INFO:tasks.workunit.client.0.smithi032.stdout:3/902: mkdir d12/d2f/d89/df0/d3a/d63/da6/dac/dc9/d125 0 2024-04-12T05:37:18.582 INFO:tasks.workunit.client.0.smithi032.stdout:3/903: write d12/daa/ff6 [182842,35776] 0 2024-04-12T05:37:18.582 INFO:tasks.workunit.client.1.smithi067.stdout:4/187: write d8/f2b [949891,76130] 0 2024-04-12T05:37:18.583 INFO:tasks.workunit.client.0.smithi032.stdout:9/670: write d9/d15/d1a/d24/d27/d88/fa8 [612563,59671] 0 2024-04-12T05:37:18.584 INFO:tasks.workunit.client.0.smithi032.stdout:9/671: chown d9/d32/c54 54 1 2024-04-12T05:37:18.585 INFO:tasks.workunit.client.0.smithi032.stdout:1/709: creat d5/de/d17/d35/d61/d54/fde x:0 0 0 2024-04-12T05:37:18.589 INFO:tasks.workunit.client.0.smithi032.stdout:0/859: link d2/d29/d77/ce9 d2/d8/df/d24/c120 0 2024-04-12T05:37:18.589 INFO:tasks.workunit.client.1.smithi067.stdout:4/188: creat d8/d9/f43 x:0 0 0 2024-04-12T05:37:18.589 INFO:tasks.workunit.client.0.smithi032.stdout:8/982: mkdir d0/db/dd/d11/d1b/d22/d29/d151 0 2024-04-12T05:37:18.590 INFO:tasks.workunit.client.0.smithi032.stdout:3/904: truncate d12/d2f/d89/df0/d3a/d63/da6/dac/dc9/fe8 448916 0 2024-04-12T05:37:18.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:18 smithi032 ceph-mon[39731]: pgmap v270: 65 pgs: 65 active+clean; 2.1 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 86 MiB/s rd, 242 MiB/s wr, 498 op/s 2024-04-12T05:37:18.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:18 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:37:18.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:18 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:37:18.591 INFO:tasks.workunit.client.0.smithi032.stdout:9/672: mknod d9/d15/d1a/dd2/d65/d80/d87/cdf 0 2024-04-12T05:37:18.591 INFO:tasks.workunit.client.1.smithi067.stdout:9/238: link d1/d13/l37 d1/d12/d15/l46 0 2024-04-12T05:37:18.592 INFO:tasks.workunit.client.0.smithi032.stdout:1/710: creat d5/de/d17/d35/d61/fdf x:0 0 0 2024-04-12T05:37:18.593 INFO:tasks.workunit.client.0.smithi032.stdout:5/959: link d2/df/d25/d43/d49/c4e d2/d3c/d48/c138 0 2024-04-12T05:37:18.595 INFO:tasks.workunit.client.1.smithi067.stdout:9/239: rmdir d1/d12 39 2024-04-12T05:37:18.595 INFO:tasks.workunit.client.0.smithi032.stdout:8/983: creat d0/d58/df7/f152 x:0 0 0 2024-04-12T05:37:18.596 INFO:tasks.workunit.client.0.smithi032.stdout:0/860: mknod d2/d28/d87/d111/d41/c121 0 2024-04-12T05:37:18.597 INFO:tasks.workunit.client.0.smithi032.stdout:3/905: mknod d12/c126 0 2024-04-12T05:37:18.599 INFO:tasks.workunit.client.1.smithi067.stdout:9/240: stat d1/d28/f1b 0 2024-04-12T05:37:18.601 INFO:tasks.workunit.client.0.smithi032.stdout:7/838: dwrite d0/de/f12 [0,4194304] 0 2024-04-12T05:37:18.604 INFO:tasks.workunit.client.0.smithi032.stdout:3/906: dread d12/d2f/d89/df0/f99 [0,4194304] 0 2024-04-12T05:37:18.605 INFO:tasks.workunit.client.1.smithi067.stdout:9/241: creat d1/f47 x:0 0 0 2024-04-12T05:37:18.606 INFO:tasks.workunit.client.1.smithi067.stdout:8/192: dread d6/d8/d18/df/f1b [0,4194304] 0 2024-04-12T05:37:18.608 INFO:tasks.workunit.client.0.smithi032.stdout:4/858: dwrite d4/d87/d96/f42 [0,4194304] 0 2024-04-12T05:37:18.609 INFO:tasks.workunit.client.0.smithi032.stdout:3/907: creat d12/d2f/d89/df0/d3a/dbe/f127 x:0 0 0 2024-04-12T05:37:18.610 INFO:tasks.workunit.client.1.smithi067.stdout:5/405: dwrite d0/da/f57 [0,4194304] 0 2024-04-12T05:37:18.610 INFO:tasks.workunit.client.1.smithi067.stdout:8/193: write d6/f7 [1247503,72735] 0 2024-04-12T05:37:18.613 INFO:tasks.workunit.client.0.smithi032.stdout:3/908: creat d12/d2f/d89/df0/d3a/d2d/f128 x:0 0 0 2024-04-12T05:37:18.616 INFO:tasks.workunit.client.1.smithi067.stdout:4/189: dwrite f6 [0,4194304] 0 2024-04-12T05:37:18.617 INFO:tasks.workunit.client.0.smithi032.stdout:3/909: mkdir d12/d2f/d89/df0/d3a/d63/da6/dac/d129 0 2024-04-12T05:37:18.618 INFO:tasks.workunit.client.1.smithi067.stdout:5/406: write d0/d3a/d2c/f37 [2283628,46640] 0 2024-04-12T05:37:18.620 INFO:tasks.workunit.client.1.smithi067.stdout:4/190: dread - d8/da/d38/f3a zero size 2024-04-12T05:37:18.622 INFO:tasks.workunit.client.1.smithi067.stdout:8/194: chown d6/d8/d18/df 1806 1 2024-04-12T05:37:18.626 INFO:tasks.workunit.client.0.smithi032.stdout:3/910: dwrite d12/d2f/d89/df0/dcb/ffe [0,4194304] 0 2024-04-12T05:37:18.628 INFO:tasks.workunit.client.1.smithi067.stdout:5/407: creat d0/d6/d20/d3e/f8a x:0 0 0 2024-04-12T05:37:18.628 INFO:tasks.workunit.client.1.smithi067.stdout:4/191: rmdir d8/da/d1d 39 2024-04-12T05:37:18.629 INFO:tasks.workunit.client.0.smithi032.stdout:3/911: mkdir d12/daa/d12a 0 2024-04-12T05:37:18.630 INFO:tasks.workunit.client.1.smithi067.stdout:4/192: fsync d8/d9/f2a 0 2024-04-12T05:37:18.632 INFO:tasks.workunit.client.1.smithi067.stdout:5/408: creat d0/d3a/d2c/f8b x:0 0 0 2024-04-12T05:37:18.632 INFO:tasks.workunit.client.0.smithi032.stdout:3/912: stat d12/l18 0 2024-04-12T05:37:18.635 INFO:tasks.workunit.client.1.smithi067.stdout:5/409: symlink d0/da/d41/l8c 0 2024-04-12T05:37:18.638 INFO:tasks.workunit.client.1.smithi067.stdout:4/193: creat d8/da/d1d/f44 x:0 0 0 2024-04-12T05:37:18.640 INFO:tasks.workunit.client.0.smithi032.stdout:3/913: dwrite d12/d2f/d89/fd9 [4194304,4194304] 0 2024-04-12T05:37:18.644 INFO:tasks.workunit.client.0.smithi032.stdout:3/914: creat d12/daa/db3/f12b x:0 0 0 2024-04-12T05:37:18.645 INFO:tasks.workunit.client.0.smithi032.stdout:3/915: mkdir d12/d24/dde/d12c 0 2024-04-12T05:37:18.649 INFO:tasks.workunit.client.0.smithi032.stdout:3/916: unlink d12/l8c 0 2024-04-12T05:37:18.662 INFO:tasks.workunit.client.1.smithi067.stdout:8/195: truncate d6/d8/d18/df/d12/f29 2195116 0 2024-04-12T05:37:18.663 INFO:tasks.workunit.client.1.smithi067.stdout:8/196: fsync d6/d8/d18/f35 0 2024-04-12T05:37:18.665 INFO:tasks.workunit.client.0.smithi032.stdout:3/917: write d12/d24/f8f [2063831,129687] 0 2024-04-12T05:37:18.665 INFO:tasks.workunit.client.1.smithi067.stdout:8/197: read - d6/d8/d18/df/f24 zero size 2024-04-12T05:37:18.669 INFO:tasks.workunit.client.1.smithi067.stdout:5/410: dwrite d0/d6/f12 [0,4194304] 0 2024-04-12T05:37:18.669 INFO:tasks.workunit.client.1.smithi067.stdout:5/411: chown d0 0 1 2024-04-12T05:37:18.672 INFO:tasks.workunit.client.0.smithi032.stdout:3/918: link d12/d2f/d89/df0/f74 d12/d2f/d89/df0/d3a/dbe/f12d 0 2024-04-12T05:37:18.675 INFO:tasks.workunit.client.0.smithi032.stdout:3/919: mkdir d12/d2f/d105/d12e 0 2024-04-12T05:37:18.677 INFO:tasks.workunit.client.1.smithi067.stdout:2/257: sync 2024-04-12T05:37:18.678 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:18 smithi067 ceph-mon[49010]: pgmap v270: 65 pgs: 65 active+clean; 2.1 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 86 MiB/s rd, 242 MiB/s wr, 498 op/s 2024-04-12T05:37:18.679 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:18 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:37:18.679 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:18 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:37:18.679 INFO:tasks.workunit.client.1.smithi067.stdout:5/412: rename d0/d6/d20/d62/l17 to d0/d6/d20/d3e/l8d 0 2024-04-12T05:37:18.679 INFO:tasks.workunit.client.0.smithi032.stdout:9/673: rename d9/d15/d1a/dd2/fc9 to d9/d1f/fe0 0 2024-04-12T05:37:18.680 INFO:tasks.workunit.client.0.smithi032.stdout:3/920: mknod d12/d2f/d89/df0/d3a/d63/c12f 0 2024-04-12T05:37:18.680 INFO:tasks.workunit.client.0.smithi032.stdout:3/921: readlink d12/daa/db3/l8b 0 2024-04-12T05:37:18.682 INFO:tasks.workunit.client.1.smithi067.stdout:2/258: unlink d5/f4e 0 2024-04-12T05:37:18.683 INFO:tasks.workunit.client.1.smithi067.stdout:2/259: fdatasync d5/d11/f29 0 2024-04-12T05:37:18.683 INFO:tasks.workunit.client.0.smithi032.stdout:4/859: symlink d4/d87/d96/d2f/d94/l11d 0 2024-04-12T05:37:18.684 INFO:tasks.workunit.client.0.smithi032.stdout:9/674: mkdir d9/d15/d1a/d24/d27/de1 0 2024-04-12T05:37:18.685 INFO:tasks.workunit.client.1.smithi067.stdout:2/260: chown d5/da/de/f13 96 1 2024-04-12T05:37:18.688 INFO:tasks.workunit.client.0.smithi032.stdout:1/711: mkdir d5/de/d17/d35/d3b/dda/de0 0 2024-04-12T05:37:18.689 INFO:tasks.workunit.client.0.smithi032.stdout:3/922: creat d12/d2f/d89/df0/d3a/d63/da6/dac/f130 x:0 0 0 2024-04-12T05:37:18.690 INFO:tasks.workunit.client.0.smithi032.stdout:8/984: creat d0/db/dd/d11/d1b/d22/d35/d135/f153 x:0 0 0 2024-04-12T05:37:18.690 INFO:tasks.workunit.client.0.smithi032.stdout:7/839: symlink d0/d13/d21/l114 0 2024-04-12T05:37:18.692 INFO:tasks.workunit.client.0.smithi032.stdout:7/840: chown d0/d13/fc5 228 1 2024-04-12T05:37:18.693 INFO:tasks.workunit.client.0.smithi032.stdout:3/923: mkdir d12/d2f/d89/df0/d3a/d63/dd5/d131 0 2024-04-12T05:37:18.694 INFO:tasks.workunit.client.0.smithi032.stdout:4/860: fdatasync d4/d87/d96/d2f/d3b/fc4 0 2024-04-12T05:37:18.696 INFO:tasks.workunit.client.0.smithi032.stdout:1/712: readlink d5/de/d17/d35/lc0 0 2024-04-12T05:37:18.696 INFO:tasks.workunit.client.1.smithi067.stdout:5/413: truncate d0/da/f57 3144871 0 2024-04-12T05:37:18.696 INFO:tasks.workunit.client.1.smithi067.stdout:2/261: truncate d5/da/de/f31 4248970 0 2024-04-12T05:37:18.696 INFO:tasks.workunit.client.0.smithi032.stdout:3/924: dread d12/d118/f9a [0,4194304] 0 2024-04-12T05:37:18.697 INFO:tasks.workunit.client.0.smithi032.stdout:8/985: fdatasync d0/d72/faa 0 2024-04-12T05:37:18.699 INFO:tasks.workunit.client.0.smithi032.stdout:8/986: write d0/f2f [4003784,53302] 0 2024-04-12T05:37:18.699 INFO:tasks.workunit.client.1.smithi067.stdout:5/414: mkdir d0/d6/d40/d4b/d38/d8e 0 2024-04-12T05:37:18.700 INFO:tasks.workunit.client.0.smithi032.stdout:7/841: mkdir d0/dd/d51/dea/d105/d115 0 2024-04-12T05:37:18.700 INFO:tasks.workunit.client.1.smithi067.stdout:2/262: mkdir d5/da/d3d/d55 0 2024-04-12T05:37:18.700 INFO:tasks.workunit.client.0.smithi032.stdout:3/925: creat d12/d2f/d89/df0/dcb/f132 x:0 0 0 2024-04-12T05:37:18.703 INFO:tasks.workunit.client.1.smithi067.stdout:5/415: mkdir d0/d6/d40/d4b/d38/d8e/d8f 0 2024-04-12T05:37:18.703 INFO:tasks.workunit.client.1.smithi067.stdout:8/198: dread d6/d8/d18/df/d12/f29 [0,4194304] 0 2024-04-12T05:37:18.704 INFO:tasks.workunit.client.0.smithi032.stdout:0/861: rename d2/d8/df/d24/d60/c94 to d2/d28/d87/d111/d3e/c122 0 2024-04-12T05:37:18.704 INFO:tasks.workunit.client.1.smithi067.stdout:2/263: symlink d5/d23/d3a/l56 0 2024-04-12T05:37:18.705 INFO:tasks.workunit.client.0.smithi032.stdout:1/713: dwrite d5/de/d17/d35/d73/fba [0,4194304] 0 2024-04-12T05:37:18.705 INFO:tasks.workunit.client.1.smithi067.stdout:5/416: stat d0/d3a/l63 0 2024-04-12T05:37:18.706 INFO:tasks.workunit.client.0.smithi032.stdout:3/926: write d12/d2f/d89/df0/d3a/d63/da6/dac/dc9/ff3 [4126755,22998] 0 2024-04-12T05:37:18.711 INFO:tasks.workunit.client.1.smithi067.stdout:5/417: rename d0/f7 to d0/d6/d40/f90 0 2024-04-12T05:37:18.716 INFO:tasks.workunit.client.0.smithi032.stdout:7/842: creat d0/f116 x:0 0 0 2024-04-12T05:37:18.718 INFO:tasks.workunit.client.0.smithi032.stdout:0/862: chown d2/d28/d87/d111/d3e/dac/d51/cb2 37019 1 2024-04-12T05:37:18.719 INFO:tasks.workunit.client.0.smithi032.stdout:3/927: creat d12/d2f/d89/df0/d3a/d63/da6/dac/dc9/d125/f133 x:0 0 0 2024-04-12T05:37:18.720 INFO:tasks.workunit.client.0.smithi032.stdout:3/928: chown d12/d2f/f41 22593868 1 2024-04-12T05:37:18.720 INFO:tasks.workunit.client.1.smithi067.stdout:5/418: dwrite d0/f2 [0,4194304] 0 2024-04-12T05:37:18.722 INFO:tasks.workunit.client.0.smithi032.stdout:3/929: write d12/d2f/d89/df0/f7b [1102180,116792] 0 2024-04-12T05:37:18.724 INFO:tasks.workunit.client.1.smithi067.stdout:8/199: truncate d6/d8/d18/df/f13 2555876 0 2024-04-12T05:37:18.725 INFO:tasks.workunit.client.0.smithi032.stdout:7/843: dwrite d0/d34/dbe/f7e [4194304,4194304] 0 2024-04-12T05:37:18.726 INFO:tasks.workunit.client.1.smithi067.stdout:5/419: read d0/d3a/d2c/f37 [3940549,57296] 0 2024-04-12T05:37:18.726 INFO:tasks.workunit.client.1.smithi067.stdout:8/200: rename d6/d8 to d6/d8/d18/d3c/d3e 22 2024-04-12T05:37:18.729 INFO:tasks.workunit.client.0.smithi032.stdout:3/930: read d12/d2f/d89/df0/dcb/ffb [5272769,124356] 0 2024-04-12T05:37:18.730 INFO:tasks.workunit.client.0.smithi032.stdout:3/931: stat d12/d2f/d89/df0/d3a/d63/da6 0 2024-04-12T05:37:18.730 INFO:tasks.workunit.client.1.smithi067.stdout:8/201: creat d6/d8/d18/f3f x:0 0 0 2024-04-12T05:37:18.731 INFO:tasks.workunit.client.0.smithi032.stdout:9/675: truncate d9/d1f/f39 77712 0 2024-04-12T05:37:18.732 INFO:tasks.workunit.client.1.smithi067.stdout:8/202: write d6/d8/d18/f35 [9244918,15072] 0 2024-04-12T05:37:18.733 INFO:tasks.workunit.client.0.smithi032.stdout:3/932: chown d12/d2f/d89/fba 30215 1 2024-04-12T05:37:18.733 INFO:tasks.workunit.client.0.smithi032.stdout:4/861: fsync d4/d87/d96/d4d/db5/ff5 0 2024-04-12T05:37:18.735 INFO:tasks.workunit.client.0.smithi032.stdout:1/714: creat d5/de/d17/d35/d61/d77/d8f/da0/fe1 x:0 0 0 2024-04-12T05:37:18.736 INFO:tasks.workunit.client.0.smithi032.stdout:3/933: mkdir d12/d2f/d89/df0/d3a/d2d/d134 0 2024-04-12T05:37:18.736 INFO:tasks.workunit.client.0.smithi032.stdout:9/676: truncate d9/d15/d2c/d41/fc8 211066 0 2024-04-12T05:37:18.737 INFO:tasks.workunit.client.0.smithi032.stdout:1/715: dread - d5/de/d17/fce zero size 2024-04-12T05:37:18.738 INFO:tasks.workunit.client.0.smithi032.stdout:9/677: stat d9/d32/d6e/f93 0 2024-04-12T05:37:18.739 INFO:tasks.workunit.client.0.smithi032.stdout:3/934: fsync d12/d2f/d89/d9d/fb6 0 2024-04-12T05:37:18.742 INFO:tasks.workunit.client.0.smithi032.stdout:4/862: creat d4/d87/d96/d2f/d3b/da2/f11e x:0 0 0 2024-04-12T05:37:18.745 INFO:tasks.workunit.client.0.smithi032.stdout:0/863: dread d2/d8/d17/d1b/d20/f6d [0,4194304] 0 2024-04-12T05:37:18.748 INFO:tasks.workunit.client.0.smithi032.stdout:1/716: unlink d5/de/d17/d35/d61/d77/l99 0 2024-04-12T05:37:18.750 INFO:tasks.workunit.client.0.smithi032.stdout:0/864: read d2/d29/d8a/d99/fbf [2030738,112534] 0 2024-04-12T05:37:18.750 INFO:tasks.workunit.client.0.smithi032.stdout:4/863: chown d4/d13/c25 986800 1 2024-04-12T05:37:18.751 INFO:tasks.workunit.client.0.smithi032.stdout:4/864: stat d4/d13/cee 0 2024-04-12T05:37:18.752 INFO:tasks.workunit.client.0.smithi032.stdout:1/717: creat d5/de/d17/d35/d61/d77/d8f/d9c/fe2 x:0 0 0 2024-04-12T05:37:18.754 INFO:tasks.workunit.client.0.smithi032.stdout:7/844: truncate d0/de/df/d29/fcf 1923888 0 2024-04-12T05:37:18.755 INFO:tasks.workunit.client.0.smithi032.stdout:9/678: dread d9/d15/d1a/d1d/faf [0,4194304] 0 2024-04-12T05:37:18.757 INFO:tasks.workunit.client.0.smithi032.stdout:4/865: creat d4/d87/d96/d4d/d78/dba/f11f x:0 0 0 2024-04-12T05:37:18.757 INFO:tasks.workunit.client.0.smithi032.stdout:3/935: dread d12/d2f/d89/df0/f82 [0,4194304] 0 2024-04-12T05:37:18.757 INFO:tasks.workunit.client.0.smithi032.stdout:3/936: readlink d12/d2f/l43 0 2024-04-12T05:37:18.758 INFO:tasks.workunit.client.0.smithi032.stdout:1/718: write d5/de/d17/d35/d3b/d49/fd9 [940087,88327] 0 2024-04-12T05:37:18.759 INFO:tasks.workunit.client.0.smithi032.stdout:7/845: unlink d0/d34/dbe/c97 0 2024-04-12T05:37:18.759 INFO:tasks.workunit.client.0.smithi032.stdout:3/937: chown d12/d2f/d89/df0/d3a/d63/dd5/d131 25 1 2024-04-12T05:37:18.761 INFO:tasks.workunit.client.0.smithi032.stdout:9/679: fdatasync d9/d15/d2c/fc4 0 2024-04-12T05:37:18.762 INFO:tasks.workunit.client.0.smithi032.stdout:3/938: symlink d12/d2f/l135 0 2024-04-12T05:37:18.762 INFO:tasks.workunit.client.0.smithi032.stdout:9/680: chown d9/d15/d2c/d41/f45 564582 1 2024-04-12T05:37:18.762 INFO:tasks.workunit.client.0.smithi032.stdout:4/866: creat d4/d87/d96/d4d/d78/dba/f120 x:0 0 0 2024-04-12T05:37:18.764 INFO:tasks.workunit.client.0.smithi032.stdout:3/939: write d12/d2f/d89/df0/dcb/f132 [280229,18832] 0 2024-04-12T05:37:18.765 INFO:tasks.workunit.client.0.smithi032.stdout:7/846: dread - d0/de/df/d29/fec zero size 2024-04-12T05:37:18.766 INFO:tasks.workunit.client.0.smithi032.stdout:9/681: fsync d9/d15/d2c/f77 0 2024-04-12T05:37:18.769 INFO:tasks.workunit.client.0.smithi032.stdout:3/940: symlink d12/d24/dde/l136 0 2024-04-12T05:37:18.771 INFO:tasks.workunit.client.0.smithi032.stdout:0/865: dwrite d2/d29/d77/db5/ffd [0,4194304] 0 2024-04-12T05:37:18.775 INFO:tasks.workunit.client.0.smithi032.stdout:7/847: symlink d0/d34/dbe/l117 0 2024-04-12T05:37:18.775 INFO:tasks.workunit.client.0.smithi032.stdout:1/719: truncate d5/de/f28 165497 0 2024-04-12T05:37:18.779 INFO:tasks.workunit.client.0.smithi032.stdout:4/867: truncate d4/d87/d96/f42 2849908 0 2024-04-12T05:37:18.780 INFO:tasks.workunit.client.0.smithi032.stdout:0/866: unlink d2/d8/df/d24/c6e 0 2024-04-12T05:37:18.783 INFO:tasks.workunit.client.0.smithi032.stdout:9/682: dwrite d9/d15/d1a/dd2/d65/d80/fb4 [4194304,4194304] 0 2024-04-12T05:37:18.784 INFO:tasks.workunit.client.0.smithi032.stdout:4/868: fsync d4/d13/d19/f2a 0 2024-04-12T05:37:18.788 INFO:tasks.workunit.client.0.smithi032.stdout:3/941: write d12/d2f/d89/d9d/fce [659856,103434] 0 2024-04-12T05:37:18.788 INFO:tasks.workunit.client.0.smithi032.stdout:7/848: dwrite d0/d13/d21/d81/f9d [0,4194304] 0 2024-04-12T05:37:18.794 INFO:tasks.workunit.client.0.smithi032.stdout:9/683: mkdir d9/d15/d1a/d24/d27/de2 0 2024-04-12T05:37:18.798 INFO:tasks.workunit.client.0.smithi032.stdout:7/849: creat d0/de/d47/f118 x:0 0 0 2024-04-12T05:37:18.798 INFO:tasks.workunit.client.0.smithi032.stdout:0/867: write d2/fd9 [787078,35856] 0 2024-04-12T05:37:18.800 INFO:tasks.workunit.client.0.smithi032.stdout:3/942: dread d12/d2f/d89/df0/d3a/d63/da6/f119 [0,4194304] 0 2024-04-12T05:37:18.801 INFO:tasks.workunit.client.0.smithi032.stdout:9/684: rename d9/d15/d1a/fdd to d9/d15/d1a/dd2/fe3 0 2024-04-12T05:37:18.803 INFO:tasks.workunit.client.0.smithi032.stdout:0/868: mknod d2/d29/d8a/d99/c123 0 2024-04-12T05:37:18.805 INFO:tasks.workunit.client.0.smithi032.stdout:1/720: dwrite d5/de/d17/f20 [0,4194304] 0 2024-04-12T05:37:18.806 INFO:tasks.workunit.client.0.smithi032.stdout:0/869: write d2/fd9 [102186,54682] 0 2024-04-12T05:37:18.807 INFO:tasks.workunit.client.0.smithi032.stdout:1/721: fsync d5/de/d17/d35/d61/d77/d8f/fd2 0 2024-04-12T05:37:18.811 INFO:tasks.workunit.client.0.smithi032.stdout:1/722: write d5/de/d17/d35/d61/d54/fde [949066,48918] 0 2024-04-12T05:37:18.814 INFO:tasks.workunit.client.0.smithi032.stdout:0/870: write d2/d8/d17/d1b/d109/f10b [1107006,89063] 0 2024-04-12T05:37:18.816 INFO:tasks.workunit.client.0.smithi032.stdout:3/943: write d12/d2f/d89/df0/fb5 [1116937,103970] 0 2024-04-12T05:37:18.818 INFO:tasks.workunit.client.0.smithi032.stdout:7/850: dread d0/de/df/d29/f89 [0,4194304] 0 2024-04-12T05:37:18.819 INFO:tasks.workunit.client.0.smithi032.stdout:0/871: creat d2/d28/d87/f124 x:0 0 0 2024-04-12T05:37:18.820 INFO:tasks.workunit.client.0.smithi032.stdout:1/723: symlink d5/de/d17/d35/d3b/dda/le3 0 2024-04-12T05:37:18.822 INFO:tasks.workunit.client.0.smithi032.stdout:3/944: creat d12/d2f/d89/df0/d3a/d63/da6/df5/f137 x:0 0 0 2024-04-12T05:37:18.825 INFO:tasks.workunit.client.0.smithi032.stdout:0/872: symlink d2/d28/d87/d111/d3e/dac/d51/d83/d9d/l125 0 2024-04-12T05:37:18.826 INFO:tasks.workunit.client.0.smithi032.stdout:7/851: dread d0/d10/fd4 [0,4194304] 0 2024-04-12T05:37:18.830 INFO:tasks.workunit.client.0.smithi032.stdout:3/945: write d12/d2f/d89/df0/fb9 [4849598,44462] 0 2024-04-12T05:37:18.831 INFO:tasks.workunit.client.0.smithi032.stdout:9/685: dread d9/d1f/f5d [0,4194304] 0 2024-04-12T05:37:18.833 INFO:tasks.workunit.client.0.smithi032.stdout:3/946: creat d12/d2f/d89/df0/d3a/d63/dd5/f138 x:0 0 0 2024-04-12T05:37:18.834 INFO:tasks.workunit.client.0.smithi032.stdout:1/724: write d5/d2c/f7f [747894,6731] 0 2024-04-12T05:37:18.834 INFO:tasks.workunit.client.0.smithi032.stdout:7/852: dread d0/d13/f4b [0,4194304] 0 2024-04-12T05:37:18.836 INFO:tasks.workunit.client.0.smithi032.stdout:9/686: mknod d9/d15/d1a/d24/ce4 0 2024-04-12T05:37:18.842 INFO:tasks.workunit.client.0.smithi032.stdout:9/687: dread d9/d32/d6e/f91 [0,4194304] 0 2024-04-12T05:37:18.844 INFO:tasks.workunit.client.0.smithi032.stdout:0/873: dwrite d2/d29/f5d [4194304,4194304] 0 2024-04-12T05:37:18.844 INFO:tasks.workunit.client.0.smithi032.stdout:2/837: sync 2024-04-12T05:37:18.850 INFO:tasks.workunit.client.0.smithi032.stdout:1/725: dwrite d5/de/d17/d35/d73/fba [0,4194304] 0 2024-04-12T05:37:18.852 INFO:tasks.workunit.client.0.smithi032.stdout:7/853: dread d0/dff/d4c/f4d [0,4194304] 0 2024-04-12T05:37:18.854 INFO:tasks.workunit.client.0.smithi032.stdout:0/874: write d2/d8/d17/d1b/d109/f10b [1525764,63207] 0 2024-04-12T05:37:18.858 INFO:tasks.workunit.client.0.smithi032.stdout:3/947: dread d12/f70 [0,4194304] 0 2024-04-12T05:37:18.858 INFO:tasks.workunit.client.0.smithi032.stdout:9/688: rename d9/d15/d1a/d24/d27/l68 to d9/d15/d1a/d69/le5 0 2024-04-12T05:37:18.859 INFO:tasks.workunit.client.0.smithi032.stdout:0/875: creat d2/d29/d77/dc2/f126 x:0 0 0 2024-04-12T05:37:18.862 INFO:tasks.workunit.client.0.smithi032.stdout:0/876: write d2/fd9 [441141,69862] 0 2024-04-12T05:37:18.865 INFO:tasks.workunit.client.0.smithi032.stdout:9/689: getdents d9/d15/d1a/d1d 0 2024-04-12T05:37:18.865 INFO:tasks.workunit.client.0.smithi032.stdout:1/726: symlink d5/de/d17/d1b/le4 0 2024-04-12T05:37:18.866 INFO:tasks.workunit.client.0.smithi032.stdout:0/877: mknod d2/c127 0 2024-04-12T05:37:18.868 INFO:tasks.workunit.client.0.smithi032.stdout:2/838: dwrite dd/d19/d9d/d42/dc7/feb [0,4194304] 0 2024-04-12T05:37:18.869 INFO:tasks.workunit.client.0.smithi032.stdout:0/878: chown d2/d8/lf0 92 1 2024-04-12T05:37:18.875 INFO:tasks.workunit.client.0.smithi032.stdout:3/948: dwrite d12/d2f/fdd [0,4194304] 0 2024-04-12T05:37:18.875 INFO:tasks.workunit.client.0.smithi032.stdout:3/949: chown d12/d2f/d89/df0/d3a/dbe/ld7 234 1 2024-04-12T05:37:18.876 INFO:tasks.workunit.client.0.smithi032.stdout:3/950: readlink d12/d24/dde/l115 0 2024-04-12T05:37:18.880 INFO:tasks.workunit.client.0.smithi032.stdout:3/951: creat d12/d2f/d89/df0/d3a/d63/da6/df5/f139 x:0 0 0 2024-04-12T05:37:18.881 INFO:tasks.workunit.client.1.smithi067.stdout:7/235: sync 2024-04-12T05:37:18.882 INFO:tasks.workunit.client.1.smithi067.stdout:0/300: sync 2024-04-12T05:37:18.882 INFO:tasks.workunit.client.1.smithi067.stdout:3/197: sync 2024-04-12T05:37:18.883 INFO:tasks.workunit.client.0.smithi032.stdout:1/727: mkdir d5/de/d17/dd6/de5 0 2024-04-12T05:37:18.885 INFO:tasks.workunit.client.1.smithi067.stdout:3/198: creat dc/df/f3d x:0 0 0 2024-04-12T05:37:18.885 INFO:tasks.workunit.client.1.smithi067.stdout:7/236: truncate d0/f9 2854815 0 2024-04-12T05:37:18.886 INFO:tasks.workunit.client.0.smithi032.stdout:2/839: mkdir dd/d19/d9d/d1c/d64/d92/d10c 0 2024-04-12T05:37:18.887 INFO:tasks.workunit.client.1.smithi067.stdout:0/301: truncate dd/d12/f2e 158502 0 2024-04-12T05:37:18.887 INFO:tasks.workunit.client.0.smithi032.stdout:0/879: symlink d2/d29/d77/dc2/l128 0 2024-04-12T05:37:18.888 INFO:tasks.workunit.client.1.smithi067.stdout:3/199: write dc/df/f30 [1152457,70149] 0 2024-04-12T05:37:18.888 INFO:tasks.workunit.client.0.smithi032.stdout:1/728: fdatasync d5/de/d17/d35/d3b/d49/f70 0 2024-04-12T05:37:18.889 INFO:tasks.workunit.client.1.smithi067.stdout:0/302: write dd/f38 [607005,122706] 0 2024-04-12T05:37:18.890 INFO:tasks.workunit.client.0.smithi032.stdout:1/729: write d5/de/d17/fa7 [3494477,116226] 0 2024-04-12T05:37:18.892 INFO:tasks.workunit.client.1.smithi067.stdout:0/303: rmdir dd/d12 39 2024-04-12T05:37:18.894 INFO:tasks.workunit.client.0.smithi032.stdout:7/854: write d0/de/df/d20/f6b [4815839,21701] 0 2024-04-12T05:37:18.894 INFO:tasks.workunit.client.0.smithi032.stdout:0/880: read d2/d28/d87/f71 [5099466,48517] 0 2024-04-12T05:37:18.895 INFO:tasks.workunit.client.1.smithi067.stdout:3/200: stat dc/df/l25 0 2024-04-12T05:37:18.895 INFO:tasks.workunit.client.1.smithi067.stdout:3/201: readlink la 0 2024-04-12T05:37:18.897 INFO:tasks.workunit.client.1.smithi067.stdout:7/237: creat d0/d2/d5/f47 x:0 0 0 2024-04-12T05:37:18.899 INFO:tasks.workunit.client.1.smithi067.stdout:7/238: read - d0/d23/d45/f1a zero size 2024-04-12T05:37:18.900 INFO:tasks.workunit.client.1.smithi067.stdout:7/239: readlink d0/l1b 0 2024-04-12T05:37:18.900 INFO:tasks.workunit.client.1.smithi067.stdout:0/304: mknod dd/d12/d14/d1a/d3d/d41/c65 0 2024-04-12T05:37:18.902 INFO:tasks.workunit.client.0.smithi032.stdout:2/840: mknod dd/d19/d9d/d42/d4a/da1/dd8/c10d 0 2024-04-12T05:37:18.902 INFO:tasks.workunit.client.1.smithi067.stdout:0/305: read dd/d40/f4c [1312323,112172] 0 2024-04-12T05:37:18.903 INFO:tasks.workunit.client.0.smithi032.stdout:1/730: creat d5/de/d17/d35/d61/d77/d8f/da0/fe6 x:0 0 0 2024-04-12T05:37:18.903 INFO:tasks.workunit.client.0.smithi032.stdout:9/690: write d9/d15/d1a/d1d/f4d [3144606,46089] 0 2024-04-12T05:37:18.904 INFO:tasks.workunit.client.1.smithi067.stdout:7/240: unlink d0/d2/d5/d1e/c39 0 2024-04-12T05:37:18.904 INFO:tasks.workunit.client.0.smithi032.stdout:9/691: chown d9/l67 28128922 1 2024-04-12T05:37:18.906 INFO:tasks.workunit.client.1.smithi067.stdout:0/306: symlink dd/d40/l66 0 2024-04-12T05:37:18.908 INFO:tasks.workunit.client.1.smithi067.stdout:7/241: stat d0/d2/d5/f11 0 2024-04-12T05:37:18.910 INFO:tasks.workunit.client.1.smithi067.stdout:7/242: fdatasync d0/d2/d5/db/f1f 0 2024-04-12T05:37:18.911 INFO:tasks.workunit.client.1.smithi067.stdout:0/307: write dd/d40/f4c [2977650,46173] 0 2024-04-12T05:37:18.912 INFO:tasks.workunit.client.0.smithi032.stdout:0/881: getdents d2/d28/d87/d111/d3e/dac/d51/d83/d9d 0 2024-04-12T05:37:18.914 INFO:tasks.workunit.client.1.smithi067.stdout:7/243: dread - d0/d2/d5/d1d/f3e zero size 2024-04-12T05:37:18.915 INFO:tasks.workunit.client.0.smithi032.stdout:9/692: link d9/d15/c19 d9/d1f/ce6 0 2024-04-12T05:37:18.917 INFO:tasks.workunit.client.1.smithi067.stdout:3/202: dwrite f9 [4194304,4194304] 0 2024-04-12T05:37:18.918 INFO:tasks.workunit.client.0.smithi032.stdout:1/731: dread d5/f30 [0,4194304] 0 2024-04-12T05:37:18.918 INFO:tasks.workunit.client.1.smithi067.stdout:0/308: link dd/f42 dd/d40/f67 0 2024-04-12T05:37:18.919 INFO:tasks.workunit.client.0.smithi032.stdout:9/693: mknod d9/ce7 0 2024-04-12T05:37:18.919 INFO:tasks.workunit.client.1.smithi067.stdout:7/244: mknod d0/d2/d5/dc/c48 0 2024-04-12T05:37:18.923 INFO:tasks.workunit.client.0.smithi032.stdout:9/694: rmdir d9 39 2024-04-12T05:37:18.923 INFO:tasks.workunit.client.1.smithi067.stdout:3/203: chown dc/d1d/d28/l29 3 1 2024-04-12T05:37:18.924 INFO:tasks.workunit.client.1.smithi067.stdout:0/309: write dd/d12/d14/d51/f5c [256424,11671] 0 2024-04-12T05:37:18.924 INFO:tasks.workunit.client.0.smithi032.stdout:1/732: stat d5/de/d17/d35/d3b/dda/le3 0 2024-04-12T05:37:18.925 INFO:tasks.workunit.client.0.smithi032.stdout:7/855: dwrite d0/f6f [0,4194304] 0 2024-04-12T05:37:18.926 INFO:tasks.workunit.client.0.smithi032.stdout:2/841: dwrite dd/d19/d9d/d42/d4a/fdd [0,4194304] 0 2024-04-12T05:37:18.927 INFO:tasks.workunit.client.1.smithi067.stdout:3/204: write dc/df/f3d [92989,130404] 0 2024-04-12T05:37:18.932 INFO:tasks.workunit.client.0.smithi032.stdout:1/733: unlink d5/de/d17/d35/d3e/fa3 0 2024-04-12T05:37:18.935 INFO:tasks.workunit.client.1.smithi067.stdout:7/245: dwrite d0/d2/d5/f47 [0,4194304] 0 2024-04-12T05:37:18.941 INFO:tasks.workunit.client.0.smithi032.stdout:7/856: mkdir d0/dff/d71/d119 0 2024-04-12T05:37:18.942 INFO:tasks.workunit.client.0.smithi032.stdout:7/857: fdatasync d0/d13/f64 0 2024-04-12T05:37:18.944 INFO:tasks.workunit.client.0.smithi032.stdout:1/734: creat d5/de/d5b/fe7 x:0 0 0 2024-04-12T05:37:18.945 INFO:tasks.workunit.client.1.smithi067.stdout:0/310: write dd/d40/f44 [1452166,88081] 0 2024-04-12T05:37:18.947 INFO:tasks.workunit.client.0.smithi032.stdout:1/735: write d5/fd3 [535955,86813] 0 2024-04-12T05:37:18.949 INFO:tasks.workunit.client.1.smithi067.stdout:7/246: creat d0/d2/d5/d1d/d43/f49 x:0 0 0 2024-04-12T05:37:18.949 INFO:tasks.workunit.client.1.smithi067.stdout:3/205: dwrite dc/df/f30 [0,4194304] 0 2024-04-12T05:37:18.951 INFO:tasks.workunit.client.0.smithi032.stdout:9/695: creat d9/d15/d1a/d1d/d5a/fe8 x:0 0 0 2024-04-12T05:37:18.952 INFO:tasks.workunit.client.0.smithi032.stdout:2/842: dwrite dd/d19/d9d/d1c/fa6 [0,4194304] 0 2024-04-12T05:37:18.955 INFO:tasks.workunit.client.0.smithi032.stdout:1/736: symlink d5/de/d17/d35/d3b/d49/le8 0 2024-04-12T05:37:18.962 INFO:tasks.workunit.client.0.smithi032.stdout:2/843: rename dd/d19/d9d/d1c/d64/d32/f9f to dd/d19/d9d/d1c/d64/d32/d3a/f10e 0 2024-04-12T05:37:18.964 INFO:tasks.workunit.client.0.smithi032.stdout:2/844: dread - dd/d19/d20/dc6/f104 zero size 2024-04-12T05:37:18.967 INFO:tasks.workunit.client.0.smithi032.stdout:1/737: rename d5/de/d17/d35/d3b/d49/f56 to d5/de/d17/d35/d3b/dda/fe9 0 2024-04-12T05:37:18.968 INFO:tasks.workunit.client.0.smithi032.stdout:1/738: chown d5/de/d17/d35/d3b/lbb 0 1 2024-04-12T05:37:18.970 INFO:tasks.workunit.client.0.smithi032.stdout:2/845: symlink dd/d19/d9d/d42/l10f 0 2024-04-12T05:37:18.974 INFO:tasks.workunit.client.1.smithi067.stdout:0/311: dwrite dd/d12/f3e [0,4194304] 0 2024-04-12T05:37:18.976 INFO:tasks.workunit.client.0.smithi032.stdout:2/846: dread dd/d19/d9d/d1c/f29 [0,4194304] 0 2024-04-12T05:37:18.983 INFO:tasks.workunit.client.0.smithi032.stdout:7/858: write d0/de/df/f3f [248371,18804] 0 2024-04-12T05:37:18.985 INFO:tasks.workunit.client.1.smithi067.stdout:7/247: dwrite d0/d2/d5/f11 [0,4194304] 0 2024-04-12T05:37:18.986 INFO:tasks.workunit.client.0.smithi032.stdout:7/859: write d0/d34/fc0 [2859937,3317] 0 2024-04-12T05:37:18.989 INFO:tasks.workunit.client.1.smithi067.stdout:0/312: dread dd/f42 [0,4194304] 0 2024-04-12T05:37:18.990 INFO:tasks.workunit.client.0.smithi032.stdout:2/847: dwrite dd/d19/d9d/d1c/d64/fc2 [0,4194304] 0 2024-04-12T05:37:18.992 INFO:tasks.workunit.client.1.smithi067.stdout:0/313: truncate dd/d12/f3a 5131502 0 2024-04-12T05:37:18.994 INFO:tasks.workunit.client.0.smithi032.stdout:7/860: rename d0/de/df/d29/f46 to d0/dd/d51/dea/d105/f11a 0 2024-04-12T05:37:18.995 INFO:tasks.workunit.client.1.smithi067.stdout:7/248: mkdir d0/d2/d5/d4a 0 2024-04-12T05:37:18.998 INFO:tasks.workunit.client.0.smithi032.stdout:7/861: rmdir d0/d13 39 2024-04-12T05:37:18.999 INFO:tasks.workunit.client.1.smithi067.stdout:3/206: dread dc/df/f1c [0,4194304] 0 2024-04-12T05:37:19.000 INFO:tasks.workunit.client.1.smithi067.stdout:7/249: symlink d0/d2/dd/l4b 0 2024-04-12T05:37:19.002 INFO:tasks.workunit.client.0.smithi032.stdout:7/862: creat d0/dff/d71/d119/f11b x:0 0 0 2024-04-12T05:37:19.004 INFO:tasks.workunit.client.0.smithi032.stdout:9/696: write d9/d15/d2c/d41/f7e [3902619,53654] 0 2024-04-12T05:37:19.005 INFO:tasks.workunit.client.1.smithi067.stdout:3/207: mkdir dc/d1d/d28/d3e 0 2024-04-12T05:37:19.007 INFO:tasks.workunit.client.0.smithi032.stdout:2/848: dread dd/d19/d9d/d1c/f81 [0,4194304] 0 2024-04-12T05:37:19.007 INFO:tasks.workunit.client.0.smithi032.stdout:2/849: fdatasync dd/d19/d9d/d42/d4a/fb3 0 2024-04-12T05:37:19.009 INFO:tasks.workunit.client.0.smithi032.stdout:7/863: truncate d0/dff/d4c/f4d 1042433 0 2024-04-12T05:37:19.009 INFO:tasks.workunit.client.1.smithi067.stdout:3/208: creat dc/df/d27/f3f x:0 0 0 2024-04-12T05:37:19.013 INFO:tasks.workunit.client.0.smithi032.stdout:2/850: symlink dd/d19/d9d/d42/d4a/da1/dd8/l110 0 2024-04-12T05:37:19.015 INFO:tasks.workunit.client.1.smithi067.stdout:6/261: sync 2024-04-12T05:37:19.015 INFO:tasks.workunit.client.1.smithi067.stdout:1/237: sync 2024-04-12T05:37:19.015 INFO:tasks.workunit.client.0.smithi032.stdout:9/697: mkdir d9/d15/d1a/d24/d27/dce/dd3/de9 0 2024-04-12T05:37:19.017 INFO:tasks.workunit.client.0.smithi032.stdout:7/864: write d0/de/df/d29/f89 [452180,48703] 0 2024-04-12T05:37:19.017 INFO:tasks.workunit.client.1.smithi067.stdout:7/250: dwrite d0/d2/d5/db/f1f [0,4194304] 0 2024-04-12T05:37:19.019 INFO:tasks.workunit.client.1.smithi067.stdout:0/314: dwrite dd/d12/d14/d51/f5a [0,4194304] 0 2024-04-12T05:37:19.019 INFO:tasks.workunit.client.1.smithi067.stdout:7/251: fsync d0/f8 0 2024-04-12T05:37:19.019 INFO:tasks.workunit.client.0.smithi032.stdout:9/698: mknod d9/d15/d2c/d41/cea 0 2024-04-12T05:37:19.024 INFO:tasks.workunit.client.0.smithi032.stdout:9/699: mknod d9/d15/d1a/d24/d27/ceb 0 2024-04-12T05:37:19.025 INFO:tasks.workunit.client.0.smithi032.stdout:2/851: getdents dd 0 2024-04-12T05:37:19.027 INFO:tasks.workunit.client.0.smithi032.stdout:7/865: creat d0/d34/dbe/dbb/de8/f11c x:0 0 0 2024-04-12T05:37:19.027 INFO:tasks.workunit.client.0.smithi032.stdout:9/700: chown d9/d15/f44 3185536 1 2024-04-12T05:37:19.029 INFO:tasks.workunit.client.1.smithi067.stdout:1/238: rmdir d4/d9/d3a 39 2024-04-12T05:37:19.029 INFO:tasks.workunit.client.1.smithi067.stdout:1/239: write d4/d9/d12/d2c/f37 [375559,116081] 0 2024-04-12T05:37:19.030 INFO:tasks.workunit.client.0.smithi032.stdout:1/739: dwrite d5/de/d17/d1b/f38 [0,4194304] 0 2024-04-12T05:37:19.032 INFO:tasks.workunit.client.0.smithi032.stdout:5/960: sync 2024-04-12T05:37:19.040 INFO:tasks.workunit.client.1.smithi067.stdout:1/240: symlink d4/d46/l4e 0 2024-04-12T05:37:19.041 INFO:tasks.workunit.client.1.smithi067.stdout:6/262: link dd/l13 dd/d21/d3f/l52 0 2024-04-12T05:37:19.042 INFO:tasks.workunit.client.0.smithi032.stdout:2/852: link dd/d19/d20/dc6/f104 dd/d19/d9d/d42/d4a/da1/dd5/f111 0 2024-04-12T05:37:19.043 INFO:tasks.workunit.client.1.smithi067.stdout:7/252: unlink d0/d2/d5/d1e/f22 0 2024-04-12T05:37:19.043 INFO:tasks.workunit.client.1.smithi067.stdout:5/420: sync 2024-04-12T05:37:19.043 INFO:tasks.workunit.client.1.smithi067.stdout:4/194: sync 2024-04-12T05:37:19.043 INFO:tasks.workunit.client.1.smithi067.stdout:2/264: sync 2024-04-12T05:37:19.043 INFO:tasks.workunit.client.1.smithi067.stdout:9/242: sync 2024-04-12T05:37:19.043 INFO:tasks.workunit.client.1.smithi067.stdout:8/203: sync 2024-04-12T05:37:19.045 INFO:tasks.workunit.client.1.smithi067.stdout:0/315: getdents dd/d12/d14/d49 0 2024-04-12T05:37:19.048 INFO:tasks.workunit.client.1.smithi067.stdout:6/263: mknod dd/d21/d3f/c53 0 2024-04-12T05:37:19.048 INFO:tasks.workunit.client.1.smithi067.stdout:7/253: unlink d0/d2/ff 0 2024-04-12T05:37:19.049 INFO:tasks.workunit.client.1.smithi067.stdout:3/209: dwrite dc/f23 [4194304,4194304] 0 2024-04-12T05:37:19.055 INFO:tasks.workunit.client.1.smithi067.stdout:8/204: mknod d6/d8/d18/d3c/c40 0 2024-04-12T05:37:19.055 INFO:tasks.workunit.client.1.smithi067.stdout:9/243: creat d1/d28/d21/f48 x:0 0 0 2024-04-12T05:37:19.055 INFO:tasks.workunit.client.1.smithi067.stdout:4/195: chown d8/da/d1d/d15/d1c/f40 9930 1 2024-04-12T05:37:19.056 INFO:tasks.workunit.client.0.smithi032.stdout:7/866: link d0/d13/d21/d30/d31/fbd d0/dd/f11d 0 2024-04-12T05:37:19.057 INFO:tasks.workunit.client.1.smithi067.stdout:6/264: creat dd/d11/d27/f54 x:0 0 0 2024-04-12T05:37:19.057 INFO:tasks.workunit.client.0.smithi032.stdout:5/961: getdents d2/d3c/d48 0 2024-04-12T05:37:19.057 INFO:tasks.workunit.client.1.smithi067.stdout:1/241: write d4/d9/d12/f2e [871353,41240] 0 2024-04-12T05:37:19.059 INFO:tasks.workunit.client.1.smithi067.stdout:7/254: dread d0/d23/d45/f26 [0,4194304] 0 2024-04-12T05:37:19.059 INFO:tasks.workunit.client.0.smithi032.stdout:2/853: dread dd/f27 [0,4194304] 0 2024-04-12T05:37:19.059 INFO:tasks.workunit.client.1.smithi067.stdout:0/316: dread - dd/d12/d14/f57 zero size 2024-04-12T05:37:19.063 INFO:tasks.workunit.client.0.smithi032.stdout:5/962: dread - d2/df/da9/dc1/d78/d86/f109 zero size 2024-04-12T05:37:19.064 INFO:tasks.workunit.client.1.smithi067.stdout:9/244: truncate d1/d12/f2e 4822425 0 2024-04-12T05:37:19.064 INFO:tasks.workunit.client.1.smithi067.stdout:4/196: mkdir d8/da/d38/d45 0 2024-04-12T05:37:19.065 INFO:tasks.workunit.client.0.smithi032.stdout:9/701: dwrite d9/d32/d86/f9b [0,4194304] 0 2024-04-12T05:37:19.065 INFO:tasks.workunit.client.0.smithi032.stdout:7/867: creat d0/de/df/d20/f11e x:0 0 0 2024-04-12T05:37:19.066 INFO:tasks.workunit.client.0.smithi032.stdout:5/963: fsync d2/d9b/df0/f126 0 2024-04-12T05:37:19.067 INFO:tasks.workunit.client.1.smithi067.stdout:6/265: fdatasync dd/d11/d27/f46 0 2024-04-12T05:37:19.070 INFO:tasks.workunit.client.1.smithi067.stdout:5/421: rename d0/d6/d40/d4b/d38/d4e to d0/d3a/d2c/d91 0 2024-04-12T05:37:19.071 INFO:tasks.workunit.client.1.smithi067.stdout:4/197: readlink d8/da/l35 0 2024-04-12T05:37:19.073 INFO:tasks.workunit.client.0.smithi032.stdout:9/702: dread d9/d15/d1a/d1d/faf [0,4194304] 0 2024-04-12T05:37:19.074 INFO:tasks.workunit.client.0.smithi032.stdout:5/964: dread d2/df/d25/d50/d91/f12d [0,4194304] 0 2024-04-12T05:37:19.075 INFO:tasks.workunit.client.1.smithi067.stdout:5/422: chown d0/da/f5a 32271681 1 2024-04-12T05:37:19.075 INFO:tasks.workunit.client.0.smithi032.stdout:5/965: readlink d2/dd/l61 0 2024-04-12T05:37:19.076 INFO:tasks.workunit.client.1.smithi067.stdout:9/245: write d1/f7 [882043,20052] 0 2024-04-12T05:37:19.076 INFO:tasks.workunit.client.1.smithi067.stdout:6/266: creat dd/d21/d3f/f55 x:0 0 0 2024-04-12T05:37:19.080 INFO:tasks.workunit.client.1.smithi067.stdout:3/210: dread dc/d1d/f31 [0,4194304] 0 2024-04-12T05:37:19.082 INFO:tasks.workunit.client.0.smithi032.stdout:7/868: dread d0/dff/f4e [0,4194304] 0 2024-04-12T05:37:19.083 INFO:tasks.workunit.client.1.smithi067.stdout:4/198: write d8/da/f18 [149176,19925] 0 2024-04-12T05:37:19.083 INFO:tasks.workunit.client.1.smithi067.stdout:8/205: dwrite f5 [0,4194304] 0 2024-04-12T05:37:19.084 INFO:tasks.workunit.client.0.smithi032.stdout:1/740: dwrite d5/de/d17/d35/d78/f7a [0,4194304] 0 2024-04-12T05:37:19.084 INFO:tasks.workunit.client.1.smithi067.stdout:2/265: truncate d5/da/de/f26 4345859 0 2024-04-12T05:37:19.086 INFO:tasks.workunit.client.1.smithi067.stdout:5/423: mknod d0/da/d41/d7d/c92 0 2024-04-12T05:37:19.090 INFO:tasks.workunit.client.1.smithi067.stdout:4/199: write d8/d9/d3d/f3f [3854804,29014] 0 2024-04-12T05:37:19.091 INFO:tasks.workunit.client.0.smithi032.stdout:2/854: write dd/d19/d9d/d1c/ff0 [51944,117765] 0 2024-04-12T05:37:19.091 INFO:tasks.workunit.client.0.smithi032.stdout:7/869: mknod d0/d13/d21/d30/da1/c11f 0 2024-04-12T05:37:19.092 INFO:tasks.workunit.client.0.smithi032.stdout:1/741: creat d5/de/d17/d1b/fea x:0 0 0 2024-04-12T05:37:19.093 INFO:tasks.workunit.client.0.smithi032.stdout:1/742: read - d5/d2c/d4f/fb9 zero size 2024-04-12T05:37:19.094 INFO:tasks.workunit.client.1.smithi067.stdout:9/246: symlink d1/d28/l49 0 2024-04-12T05:37:19.094 INFO:tasks.workunit.client.0.smithi032.stdout:2/855: rmdir dd/d19/d9d/d42/da7 39 2024-04-12T05:37:19.095 INFO:tasks.workunit.client.1.smithi067.stdout:5/424: stat d0/da/d3f/f49 0 2024-04-12T05:37:19.095 INFO:tasks.workunit.client.0.smithi032.stdout:1/743: rmdir d5/de/d17/d35/d78 39 2024-04-12T05:37:19.097 INFO:tasks.workunit.client.0.smithi032.stdout:2/856: fsync dd/d19/d9d/d42/dc7/feb 0 2024-04-12T05:37:19.098 INFO:tasks.workunit.client.1.smithi067.stdout:6/267: dread dd/d11/f19 [0,4194304] 0 2024-04-12T05:37:19.098 INFO:tasks.workunit.client.1.smithi067.stdout:8/206: write d6/d8/d18/df/d12/f29 [406106,64249] 0 2024-04-12T05:37:19.099 INFO:tasks.workunit.client.0.smithi032.stdout:9/703: dwrite d9/d15/d2c/ff [0,4194304] 0 2024-04-12T05:37:19.103 INFO:tasks.workunit.client.0.smithi032.stdout:9/704: chown d9/d15/d1a/d24/d27/d88/dc1 8 1 2024-04-12T05:37:19.104 INFO:tasks.workunit.client.0.smithi032.stdout:5/966: dread d2/df/da9/dc1/d78/d120/f7c [0,4194304] 0 2024-04-12T05:37:19.105 INFO:tasks.workunit.client.0.smithi032.stdout:4/869: sync 2024-04-12T05:37:19.105 INFO:tasks.workunit.client.0.smithi032.stdout:8/987: sync 2024-04-12T05:37:19.110 INFO:tasks.workunit.client.1.smithi067.stdout:8/207: chown d6/l21 3 1 2024-04-12T05:37:19.110 INFO:tasks.workunit.client.1.smithi067.stdout:3/211: creat dc/f40 x:0 0 0 2024-04-12T05:37:19.110 INFO:tasks.workunit.client.1.smithi067.stdout:7/255: dwrite d0/d2/d5/d1d/f36 [0,4194304] 0 2024-04-12T05:37:19.110 INFO:tasks.workunit.client.1.smithi067.stdout:6/268: truncate fc 1279266 0 2024-04-12T05:37:19.110 INFO:tasks.workunit.client.1.smithi067.stdout:1/242: dread d4/d9/f31 [0,4194304] 0 2024-04-12T05:37:19.111 INFO:tasks.workunit.client.0.smithi032.stdout:1/744: symlink d5/de/d17/d35/d61/d77/d8f/d9c/leb 0 2024-04-12T05:37:19.111 INFO:tasks.workunit.client.1.smithi067.stdout:3/212: chown la 170 1 2024-04-12T05:37:19.112 INFO:tasks.workunit.client.1.smithi067.stdout:9/247: chown d1/d12/d15/f1d 75407374 1 2024-04-12T05:37:19.112 INFO:tasks.workunit.client.1.smithi067.stdout:5/425: creat d0/d6/d40/d4b/d38/d8e/d8f/f93 x:0 0 0 2024-04-12T05:37:19.112 INFO:tasks.workunit.client.1.smithi067.stdout:8/208: chown d6/d8/d18/df/d12/f1f 180006 1 2024-04-12T05:37:19.112 INFO:tasks.workunit.client.0.smithi032.stdout:9/705: fsync d9/d15/d1a/d1d/f6f 0 2024-04-12T05:37:19.113 INFO:tasks.workunit.client.1.smithi067.stdout:9/248: write d1/fd [3957330,51694] 0 2024-04-12T05:37:19.114 INFO:tasks.workunit.client.1.smithi067.stdout:7/256: creat d0/d23/f4c x:0 0 0 2024-04-12T05:37:19.114 INFO:tasks.workunit.client.1.smithi067.stdout:4/200: link d8/d9/f21 d8/d39/f46 0 2024-04-12T05:37:19.115 INFO:tasks.workunit.client.0.smithi032.stdout:5/967: rmdir d2/df/d31/da2/dc0 39 2024-04-12T05:37:19.116 INFO:tasks.workunit.client.0.smithi032.stdout:4/870: mkdir d4/d87/d96/d2f/d3b/da2/d121 0 2024-04-12T05:37:19.116 INFO:tasks.workunit.client.0.smithi032.stdout:5/968: chown d2/df/d25/d50/daa/f10b 28531 1 2024-04-12T05:37:19.116 INFO:tasks.workunit.client.0.smithi032.stdout:2/857: readlink dd/d19/d9d/d42/da7/dc5/ld1 0 2024-04-12T05:37:19.117 INFO:tasks.workunit.client.0.smithi032.stdout:1/745: unlink d5/de/d17/d35/d61/c72 0 2024-04-12T05:37:19.119 INFO:tasks.workunit.client.0.smithi032.stdout:9/706: mknod d9/d15/d1a/dd2/d65/cec 0 2024-04-12T05:37:19.120 INFO:tasks.workunit.client.0.smithi032.stdout:5/969: chown d2/dd/c2a 0 1 2024-04-12T05:37:19.120 INFO:tasks.workunit.client.1.smithi067.stdout:9/249: symlink d1/d28/d23/l4a 0 2024-04-12T05:37:19.120 INFO:tasks.workunit.client.1.smithi067.stdout:5/426: dread - d0/d6/d40/f7f zero size 2024-04-12T05:37:19.122 INFO:tasks.workunit.client.0.smithi032.stdout:2/858: fdatasync dd/d19/d9d/d42/d4a/f6b 0 2024-04-12T05:37:19.125 INFO:tasks.workunit.client.1.smithi067.stdout:5/427: dread - d0/d6/d40/d4b/d38/f79 zero size 2024-04-12T05:37:19.125 INFO:tasks.workunit.client.1.smithi067.stdout:8/209: creat d6/d8/d18/f41 x:0 0 0 2024-04-12T05:37:19.126 INFO:tasks.workunit.client.1.smithi067.stdout:6/269: read dd/f1d [3415635,31964] 0 2024-04-12T05:37:19.126 INFO:tasks.workunit.client.1.smithi067.stdout:5/428: chown d0/d6/d20/d3e/c66 64 1 2024-04-12T05:37:19.126 INFO:tasks.workunit.client.1.smithi067.stdout:7/257: creat d0/d23/f4d x:0 0 0 2024-04-12T05:37:19.126 INFO:tasks.workunit.client.0.smithi032.stdout:4/871: unlink d4/d87/d96/d2f/d3b/da2/d10a/f11b 0 2024-04-12T05:37:19.130 INFO:tasks.workunit.client.0.smithi032.stdout:7/870: write d0/d34/dbe/f5c [633568,1867] 0 2024-04-12T05:37:19.133 INFO:tasks.workunit.client.0.smithi032.stdout:2/859: creat dd/d19/d9d/d42/d4a/da1/dd5/f112 x:0 0 0 2024-04-12T05:37:19.135 INFO:tasks.workunit.client.0.smithi032.stdout:2/860: write dd/d19/d9d/fae [923651,94453] 0 2024-04-12T05:37:19.135 INFO:tasks.workunit.client.1.smithi067.stdout:9/250: chown d1/d28/d21/d3f/c40 3 1 2024-04-12T05:37:19.137 INFO:tasks.workunit.client.1.smithi067.stdout:4/201: link d8/f33 d8/da/d38/d45/f47 0 2024-04-12T05:37:19.138 INFO:tasks.workunit.client.1.smithi067.stdout:6/270: truncate dd/d11/f1e 745901 0 2024-04-12T05:37:19.138 INFO:tasks.workunit.client.1.smithi067.stdout:5/429: write d0/d6/d33/f52 [825133,89264] 0 2024-04-12T05:37:19.140 INFO:tasks.workunit.client.0.smithi032.stdout:1/746: creat d5/de/d17/d35/d3e/fec x:0 0 0 2024-04-12T05:37:19.141 INFO:tasks.workunit.client.1.smithi067.stdout:1/243: creat d4/d9/d12/d28/f4f x:0 0 0 2024-04-12T05:37:19.143 INFO:tasks.workunit.client.1.smithi067.stdout:8/210: mknod d6/d8/d18/c42 0 2024-04-12T05:37:19.144 INFO:tasks.workunit.client.0.smithi032.stdout:9/707: creat d9/d15/d1a/d24/fed x:0 0 0 2024-04-12T05:37:19.145 INFO:tasks.workunit.client.1.smithi067.stdout:4/202: truncate d8/f33 525657 0 2024-04-12T05:37:19.146 INFO:tasks.workunit.client.1.smithi067.stdout:9/251: unlink d1/l2b 0 2024-04-12T05:37:19.147 INFO:tasks.workunit.client.0.smithi032.stdout:8/988: dwrite d0/d72/fc8 [0,4194304] 0 2024-04-12T05:37:19.148 INFO:tasks.workunit.client.1.smithi067.stdout:3/213: getdents dc/df 0 2024-04-12T05:37:19.149 INFO:tasks.workunit.client.1.smithi067.stdout:2/266: dread d5/d11/f50 [0,4194304] 0 2024-04-12T05:37:19.152 INFO:tasks.workunit.client.0.smithi032.stdout:9/708: mkdir d9/d15/d2c/dee 0 2024-04-12T05:37:19.153 INFO:tasks.workunit.client.1.smithi067.stdout:3/214: fsync dc/d1d/f22 0 2024-04-12T05:37:19.153 INFO:tasks.workunit.client.1.smithi067.stdout:0/317: dread dd/f13 [4194304,4194304] 0 2024-04-12T05:37:19.155 INFO:tasks.workunit.client.0.smithi032.stdout:5/970: truncate d2/df/d25/d50/f76 3763212 0 2024-04-12T05:37:19.157 INFO:tasks.workunit.client.1.smithi067.stdout:4/203: chown d8/da/d1d/d15/f1a 793 1 2024-04-12T05:37:19.159 INFO:tasks.workunit.client.1.smithi067.stdout:6/271: dwrite dd/d11/d27/f2e [0,4194304] 0 2024-04-12T05:37:19.160 INFO:tasks.workunit.client.1.smithi067.stdout:7/258: rename d0/d2/d5/f47 to d0/d2/d5/db/f4e 0 2024-04-12T05:37:19.161 INFO:tasks.workunit.client.0.smithi032.stdout:7/871: getdents d0/dff/d4c 0 2024-04-12T05:37:19.162 INFO:tasks.workunit.client.0.smithi032.stdout:3/952: sync 2024-04-12T05:37:19.162 INFO:tasks.workunit.client.0.smithi032.stdout:0/882: sync 2024-04-12T05:37:19.162 INFO:tasks.workunit.client.0.smithi032.stdout:2/861: write dd/d19/d9d/d1c/d64/d36/fb4 [917259,59529] 0 2024-04-12T05:37:19.163 INFO:tasks.workunit.client.0.smithi032.stdout:4/872: dwrite d4/d87/d96/d4d/db5/ff5 [0,4194304] 0 2024-04-12T05:37:19.167 INFO:tasks.workunit.client.1.smithi067.stdout:0/318: truncate dd/d40/f44 2357134 0 2024-04-12T05:37:19.170 INFO:tasks.workunit.client.0.smithi032.stdout:9/709: dread d9/d15/d2c/d41/f96 [0,4194304] 0 2024-04-12T05:37:19.174 INFO:tasks.workunit.client.0.smithi032.stdout:0/883: fsync d2/d29/d8a/f108 0 2024-04-12T05:37:19.175 INFO:tasks.workunit.client.0.smithi032.stdout:1/747: truncate d5/de/d17/d35/d3b/f63 3315557 0 2024-04-12T05:37:19.175 INFO:tasks.workunit.client.0.smithi032.stdout:3/953: mknod d12/d2f/de2/c13a 0 2024-04-12T05:37:19.175 INFO:tasks.workunit.client.0.smithi032.stdout:7/872: readlink d0/de/d47/lb1 0 2024-04-12T05:37:19.175 INFO:tasks.workunit.client.0.smithi032.stdout:9/710: chown d9/d15/d1a/dba/lc0 5301297 1 2024-04-12T05:37:19.175 INFO:tasks.workunit.client.1.smithi067.stdout:9/252: dwrite d1/fd [0,4194304] 0 2024-04-12T05:37:19.176 INFO:tasks.workunit.client.0.smithi032.stdout:8/989: link d0/db/f4e d0/db/d7d/d11f/f154 0 2024-04-12T05:37:19.177 INFO:tasks.workunit.client.1.smithi067.stdout:5/430: dwrite d0/d3a/f19 [0,4194304] 0 2024-04-12T05:37:19.178 INFO:tasks.workunit.client.1.smithi067.stdout:6/272: chown dd/d30/l3a 31324335 1 2024-04-12T05:37:19.180 INFO:tasks.workunit.client.0.smithi032.stdout:8/990: chown d0/db/d38/d63/d77/f12e 0 1 2024-04-12T05:37:19.181 INFO:tasks.workunit.client.1.smithi067.stdout:3/215: rename dc/df/d27/l2b to dc/d1d/d28/d3e/l41 0 2024-04-12T05:37:19.182 INFO:tasks.workunit.client.1.smithi067.stdout:5/431: chown d0/d3a/f19 50 1 2024-04-12T05:37:19.184 INFO:tasks.workunit.client.1.smithi067.stdout:1/244: dwrite d4/d9/f29 [0,4194304] 0 2024-04-12T05:37:19.185 INFO:tasks.workunit.client.0.smithi032.stdout:1/748: unlink d5/de/d17/d35/d61/l90 0 2024-04-12T05:37:19.185 INFO:tasks.workunit.client.1.smithi067.stdout:3/216: read - dc/d1d/f36 zero size 2024-04-12T05:37:19.187 INFO:tasks.workunit.client.0.smithi032.stdout:1/749: fdatasync d5/de/d17/d35/fbf 0 2024-04-12T05:37:19.187 INFO:tasks.workunit.client.1.smithi067.stdout:5/432: dread - d0/d6/d33/f72 zero size 2024-04-12T05:37:19.190 INFO:tasks.workunit.client.1.smithi067.stdout:8/211: dwrite d6/d8/d18/df/f13 [0,4194304] 0 2024-04-12T05:37:19.191 INFO:tasks.workunit.client.0.smithi032.stdout:5/971: link d2/df/da9/dc1/d6c/c130 d2/df/d31/d80/c139 0 2024-04-12T05:37:19.192 INFO:tasks.workunit.client.0.smithi032.stdout:0/884: rmdir d2/d28/d87/d111/d41 39 2024-04-12T05:37:19.192 INFO:tasks.workunit.client.0.smithi032.stdout:4/873: creat d4/d87/d96/d4d/db5/dcf/d114/f122 x:0 0 0 2024-04-12T05:37:19.197 INFO:tasks.workunit.client.0.smithi032.stdout:1/750: creat d5/de/d17/fed x:0 0 0 2024-04-12T05:37:19.197 INFO:tasks.workunit.client.0.smithi032.stdout:9/711: creat d9/d15/d1a/d24/d27/de2/fef x:0 0 0 2024-04-12T05:37:19.200 INFO:tasks.workunit.client.0.smithi032.stdout:5/972: unlink d2/dd/d5d/cea 0 2024-04-12T05:37:19.202 INFO:tasks.workunit.client.1.smithi067.stdout:0/319: stat dd/d12/d14/d1a/d3d/d41/c47 0 2024-04-12T05:37:19.203 INFO:tasks.workunit.client.1.smithi067.stdout:7/259: creat d0/d2/d5/d1e/f4f x:0 0 0 2024-04-12T05:37:19.203 INFO:tasks.workunit.client.1.smithi067.stdout:9/253: symlink d1/d28/d21/l4b 0 2024-04-12T05:37:19.204 INFO:tasks.workunit.client.1.smithi067.stdout:7/260: fsync d0/f1 0 2024-04-12T05:37:19.204 INFO:tasks.workunit.client.1.smithi067.stdout:6/273: mkdir dd/d21/d23/d56 0 2024-04-12T05:37:19.204 INFO:tasks.workunit.client.0.smithi032.stdout:0/885: creat d2/d8/d17/d1b/d20/f129 x:0 0 0 2024-04-12T05:37:19.204 INFO:tasks.workunit.client.0.smithi032.stdout:5/973: write d2/f68 [2704531,123481] 0 2024-04-12T05:37:19.205 INFO:tasks.workunit.client.1.smithi067.stdout:4/204: rename d8/da/d38 to d8/d9/d3d/d48 0 2024-04-12T05:37:19.208 INFO:tasks.workunit.client.0.smithi032.stdout:9/712: creat d9/d32/d86/dcc/ff0 x:0 0 0 2024-04-12T05:37:19.209 INFO:tasks.workunit.client.1.smithi067.stdout:7/261: dread d0/d23/d45/f26 [0,4194304] 0 2024-04-12T05:37:19.211 INFO:tasks.workunit.client.1.smithi067.stdout:7/262: write d0/d2/d5/d1d/f3e [826045,24379] 0 2024-04-12T05:37:19.215 INFO:tasks.workunit.client.0.smithi032.stdout:0/886: dread d2/d29/d77/db5/ffd [0,4194304] 0 2024-04-12T05:37:19.215 INFO:tasks.workunit.client.0.smithi032.stdout:2/862: write dd/d19/d20/dc6/f104 [735394,128445] 0 2024-04-12T05:37:19.215 INFO:tasks.workunit.client.0.smithi032.stdout:5/974: dwrite d2/df/f1d [0,4194304] 0 2024-04-12T05:37:19.216 INFO:tasks.workunit.client.1.smithi067.stdout:3/217: mknod dc/d1d/d28/c42 0 2024-04-12T05:37:19.216 INFO:tasks.workunit.client.1.smithi067.stdout:5/433: rmdir d0/d6/d20/d3e 39 2024-04-12T05:37:19.216 INFO:tasks.workunit.client.1.smithi067.stdout:1/245: readlink d4/d9/d3a/l27 0 2024-04-12T05:37:19.216 INFO:tasks.workunit.client.1.smithi067.stdout:3/218: chown dc/df/c17 797 1 2024-04-12T05:37:19.217 INFO:tasks.workunit.client.1.smithi067.stdout:1/246: dread - d4/d9/d3a/f23 zero size 2024-04-12T05:37:19.218 INFO:tasks.workunit.client.1.smithi067.stdout:2/267: getdents d5/da/de/d14/d34 0 2024-04-12T05:37:19.219 INFO:tasks.workunit.client.0.smithi032.stdout:9/713: truncate d9/d15/d1a/dd2/d65/fa6 405127 0 2024-04-12T05:37:19.219 INFO:tasks.workunit.client.0.smithi032.stdout:2/863: write dd/d19/d20/dc6/f104 [264321,89779] 0 2024-04-12T05:37:19.219 INFO:tasks.workunit.client.1.smithi067.stdout:4/205: creat d8/da/d1d/f49 x:0 0 0 2024-04-12T05:37:19.219 INFO:tasks.workunit.client.1.smithi067.stdout:3/219: chown dc/d1d/d28/c42 68 1 2024-04-12T05:37:19.219 INFO:tasks.workunit.client.1.smithi067.stdout:5/434: stat d0/d6/d20/d62/d75 0 2024-04-12T05:37:19.220 INFO:tasks.workunit.client.0.smithi032.stdout:5/975: readlink d2/dd/l2e 0 2024-04-12T05:37:19.221 INFO:tasks.workunit.client.1.smithi067.stdout:4/206: readlink d8/da/l35 0 2024-04-12T05:37:19.222 INFO:tasks.workunit.client.0.smithi032.stdout:3/954: dread d12/d2f/d89/df0/f36 [0,4194304] 0 2024-04-12T05:37:19.223 INFO:tasks.workunit.client.1.smithi067.stdout:6/274: rename dd/d11/d27/f2e to dd/d11/d27/d4a/d51/f57 0 2024-04-12T05:37:19.224 INFO:tasks.workunit.client.0.smithi032.stdout:9/714: mknod d9/d15/d1a/d24/d27/d88/cf1 0 2024-04-12T05:37:19.225 INFO:tasks.workunit.client.1.smithi067.stdout:4/207: fdatasync d8/da/f1f 0 2024-04-12T05:37:19.225 INFO:tasks.workunit.client.0.smithi032.stdout:5/976: write d2/d9b/fb7 [4508672,56335] 0 2024-04-12T05:37:19.226 INFO:tasks.workunit.client.0.smithi032.stdout:7/873: dread d0/de/f44 [4194304,4194304] 0 2024-04-12T05:37:19.226 INFO:tasks.workunit.client.1.smithi067.stdout:6/275: write dd/f17 [3274720,86473] 0 2024-04-12T05:37:19.227 INFO:tasks.workunit.client.0.smithi032.stdout:0/887: creat d2/f12a x:0 0 0 2024-04-12T05:37:19.228 INFO:tasks.workunit.client.0.smithi032.stdout:9/715: rmdir d9/d32 39 2024-04-12T05:37:19.229 INFO:tasks.workunit.client.1.smithi067.stdout:9/254: symlink d1/d28/d27/d32/l4c 0 2024-04-12T05:37:19.231 INFO:tasks.workunit.client.0.smithi032.stdout:4/874: write d4/f35 [3971470,48656] 0 2024-04-12T05:37:19.232 INFO:tasks.workunit.client.1.smithi067.stdout:0/320: link dd/d12/d14/d49/f53 dd/d5b/f68 0 2024-04-12T05:37:19.234 INFO:tasks.workunit.client.0.smithi032.stdout:3/955: write d12/d2f/d89/df0/d3a/d63/fcf [4388617,20945] 0 2024-04-12T05:37:19.234 INFO:tasks.workunit.client.0.smithi032.stdout:4/875: chown d4/f3d 194150 1 2024-04-12T05:37:19.235 INFO:tasks.workunit.client.0.smithi032.stdout:2/864: dwrite dd/d19/d9d/d42/d4a/da1/dd5/f112 [0,4194304] 0 2024-04-12T05:37:19.238 INFO:tasks.workunit.client.0.smithi032.stdout:4/876: dread - d4/d87/d96/d2f/d3b/fc4 zero size 2024-04-12T05:37:19.238 INFO:tasks.workunit.client.0.smithi032.stdout:3/956: fdatasync d12/d2f/de2/fe4 0 2024-04-12T05:37:19.240 INFO:tasks.workunit.client.0.smithi032.stdout:9/716: dread - d9/d15/d1a/f9e zero size 2024-04-12T05:37:19.241 INFO:tasks.workunit.client.0.smithi032.stdout:2/865: chown dd/d19/d9d/d1c/d64/f98 26304901 1 2024-04-12T05:37:19.243 INFO:tasks.workunit.client.0.smithi032.stdout:4/877: write d4/ff4 [109164,105270] 0 2024-04-12T05:37:19.246 INFO:tasks.workunit.client.1.smithi067.stdout:8/212: dwrite d6/f27 [0,4194304] 0 2024-04-12T05:37:19.246 INFO:tasks.workunit.client.0.smithi032.stdout:9/717: write d9/d15/d2c/d41/f7e [4072131,31389] 0 2024-04-12T05:37:19.248 INFO:tasks.workunit.client.0.smithi032.stdout:3/957: mknod d12/d2f/d89/df0/d3a/d63/da6/dac/d129/c13b 0 2024-04-12T05:37:19.251 INFO:tasks.workunit.client.1.smithi067.stdout:6/276: creat dd/d11/d27/f58 x:0 0 0 2024-04-12T05:37:19.252 INFO:tasks.workunit.client.1.smithi067.stdout:5/435: dwrite d0/d3a/f7e [0,4194304] 0 2024-04-12T05:37:19.254 INFO:tasks.workunit.client.0.smithi032.stdout:4/878: mkdir d4/d87/d96/d2f/d94/dc6/d123 0 2024-04-12T05:37:19.255 INFO:tasks.workunit.client.0.smithi032.stdout:7/874: write d0/d34/dbe/dbb/fd1 [788075,18209] 0 2024-04-12T05:37:19.258 INFO:tasks.workunit.client.0.smithi032.stdout:2/866: link dd/d19/d20/dc6/fed dd/d19/d9d/d42/d4a/f113 0 2024-04-12T05:37:19.259 INFO:tasks.workunit.client.1.smithi067.stdout:3/220: dread dc/df/f30 [0,4194304] 0 2024-04-12T05:37:19.259 INFO:tasks.workunit.client.1.smithi067.stdout:4/208: dwrite d8/da/f30 [0,4194304] 0 2024-04-12T05:37:19.261 INFO:tasks.workunit.client.0.smithi032.stdout:5/977: dread d2/df/da9/dc1/d78/f94 [0,4194304] 0 2024-04-12T05:37:19.262 INFO:tasks.workunit.client.0.smithi032.stdout:2/867: stat dd/c54 0 2024-04-12T05:37:19.266 INFO:tasks.workunit.client.1.smithi067.stdout:4/209: write d8/da/d1d/d15/d1c/f40 [791157,76474] 0 2024-04-12T05:37:19.266 INFO:tasks.workunit.client.1.smithi067.stdout:9/255: dwrite d1/d12/f2e [0,4194304] 0 2024-04-12T05:37:19.267 INFO:tasks.workunit.client.1.smithi067.stdout:3/221: write dc/d1d/f22 [502035,48881] 0 2024-04-12T05:37:19.268 INFO:tasks.workunit.client.0.smithi032.stdout:1/751: dread d5/de/d17/d35/fbf [0,4194304] 0 2024-04-12T05:37:19.269 INFO:tasks.workunit.client.1.smithi067.stdout:2/268: mkdir d5/da/de/d14/d34/d40/d57 0 2024-04-12T05:37:19.271 INFO:tasks.workunit.client.1.smithi067.stdout:7/263: creat d0/d2/d5/f50 x:0 0 0 2024-04-12T05:37:19.271 INFO:tasks.workunit.client.1.smithi067.stdout:1/247: write d4/f10 [1951131,115867] 0 2024-04-12T05:37:19.273 INFO:tasks.workunit.client.0.smithi032.stdout:0/888: dwrite d2/d28/d87/d111/fd [0,4194304] 0 2024-04-12T05:37:19.275 INFO:tasks.workunit.client.1.smithi067.stdout:7/264: fsync d0/d23/f4d 0 2024-04-12T05:37:19.277 INFO:tasks.workunit.client.1.smithi067.stdout:5/436: mknod d0/d6/d33/c94 0 2024-04-12T05:37:19.278 INFO:tasks.workunit.client.0.smithi032.stdout:5/978: rename d2/df/d31 to d2/df/d25/d50/d91/d13a 0 2024-04-12T05:37:19.278 INFO:tasks.workunit.client.1.smithi067.stdout:6/277: symlink dd/d11/d27/d4a/l59 0 2024-04-12T05:37:19.281 INFO:tasks.workunit.client.0.smithi032.stdout:4/879: dread d4/d13/d19/f2a [0,4194304] 0 2024-04-12T05:37:19.283 INFO:tasks.workunit.client.0.smithi032.stdout:7/875: getdents d0/d10 0 2024-04-12T05:37:19.284 INFO:tasks.workunit.client.1.smithi067.stdout:4/210: stat d8/da/d1d/c1e 0 2024-04-12T05:37:19.285 INFO:tasks.workunit.client.0.smithi032.stdout:5/979: rename d2/dd/d5d/ccc to d2/dd/d5d/c13b 0 2024-04-12T05:37:19.288 INFO:tasks.workunit.client.1.smithi067.stdout:1/248: creat d4/d9/d12/d2c/f50 x:0 0 0 2024-04-12T05:37:19.292 INFO:tasks.workunit.client.1.smithi067.stdout:9/256: creat d1/d28/d21/d45/f4d x:0 0 0 2024-04-12T05:37:19.293 INFO:tasks.workunit.client.1.smithi067.stdout:7/265: dwrite d0/d23/f41 [0,4194304] 0 2024-04-12T05:37:19.293 INFO:tasks.workunit.client.0.smithi032.stdout:5/980: rename d2/df/d25/d50/f7e to d2/df/d25/d43/d124/f13c 0 2024-04-12T05:37:19.294 INFO:tasks.workunit.client.1.smithi067.stdout:9/257: write d1/d28/d21/f48 [1035461,19620] 0 2024-04-12T05:37:19.296 INFO:tasks.workunit.client.1.smithi067.stdout:7/266: readlink d0/d23/l46 0 2024-04-12T05:37:19.296 INFO:tasks.workunit.client.1.smithi067.stdout:0/321: getdents dd/d12/d14/d1a 0 2024-04-12T05:37:19.302 INFO:tasks.workunit.client.1.smithi067.stdout:4/211: unlink d8/da/d1d/d15/f1a 0 2024-04-12T05:37:19.303 INFO:tasks.workunit.client.1.smithi067.stdout:1/249: fdatasync d4/d9/d14/f2a 0 2024-04-12T05:37:19.308 INFO:tasks.workunit.client.1.smithi067.stdout:8/213: getdents d6/d8/d18/d3c 0 2024-04-12T05:37:19.310 INFO:tasks.workunit.client.0.smithi032.stdout:9/718: dread d9/d15/d2c/d41/f45 [0,4194304] 0 2024-04-12T05:37:19.310 INFO:tasks.workunit.client.1.smithi067.stdout:7/267: read d0/d2/d5/f11 [1472969,64475] 0 2024-04-12T05:37:19.311 INFO:tasks.workunit.client.0.smithi032.stdout:9/719: stat d9/d15/d1a/d1d/d5a/lb1 0 2024-04-12T05:37:19.311 INFO:tasks.workunit.client.1.smithi067.stdout:8/214: fsync d6/d8/d18/df/f1c 0 2024-04-12T05:37:19.312 INFO:tasks.workunit.client.1.smithi067.stdout:2/269: dwrite d5/d11/f50 [0,4194304] 0 2024-04-12T05:37:19.315 INFO:tasks.workunit.client.1.smithi067.stdout:5/437: creat d0/d6/d20/d3e/f95 x:0 0 0 2024-04-12T05:37:19.316 INFO:tasks.workunit.client.0.smithi032.stdout:2/868: dwrite dd/d19/d9d/d42/f80 [0,4194304] 0 2024-04-12T05:37:19.316 INFO:tasks.workunit.client.0.smithi032.stdout:5/981: fdatasync d2/df/d25/d50/d91/d13a/f46 0 2024-04-12T05:37:19.317 INFO:tasks.workunit.client.1.smithi067.stdout:2/270: write d5/d11/f2e [4816106,127675] 0 2024-04-12T05:37:19.321 INFO:tasks.workunit.client.1.smithi067.stdout:1/250: rename d4/d9/d34 to d4/d9/d3a/d51 0 2024-04-12T05:37:19.322 INFO:tasks.workunit.client.1.smithi067.stdout:3/222: link dc/df/f2c dc/d1d/f43 0 2024-04-12T05:37:19.322 INFO:tasks.workunit.client.0.smithi032.stdout:4/880: link d4/d13/fd d4/d87/d96/d2f/d7c/dc2/f124 0 2024-04-12T05:37:19.323 INFO:tasks.workunit.client.0.smithi032.stdout:0/889: write d2/d8/d17/f2b [2744916,76676] 0 2024-04-12T05:37:19.323 INFO:tasks.workunit.client.1.smithi067.stdout:0/322: write dd/d12/d14/d49/f4d [707964,1504] 0 2024-04-12T05:37:19.324 INFO:tasks.workunit.client.1.smithi067.stdout:3/223: stat dc/df/c3c 0 2024-04-12T05:37:19.324 INFO:tasks.workunit.client.1.smithi067.stdout:4/212: fsync d8/d9/f21 0 2024-04-12T05:37:19.327 INFO:tasks.workunit.client.1.smithi067.stdout:1/251: write d4/d3d/f4b [446406,76775] 0 2024-04-12T05:37:19.327 INFO:tasks.workunit.client.0.smithi032.stdout:2/869: mkdir dd/d114 0 2024-04-12T05:37:19.328 INFO:tasks.workunit.client.0.smithi032.stdout:1/752: dwrite d5/de/d17/d35/d73/f8a [0,4194304] 0 2024-04-12T05:37:19.329 INFO:tasks.workunit.client.1.smithi067.stdout:5/438: creat d0/d6/d40/d4b/d38/d8e/f96 x:0 0 0 2024-04-12T05:37:19.330 INFO:tasks.workunit.client.0.smithi032.stdout:0/890: dread d2/d29/d8a/fd7 [0,4194304] 0 2024-04-12T05:37:19.332 INFO:tasks.workunit.client.0.smithi032.stdout:0/891: dread - d2/d29/dcc/f105 zero size 2024-04-12T05:37:19.333 INFO:tasks.workunit.client.1.smithi067.stdout:6/278: creat dd/d11/d27/f5a x:0 0 0 2024-04-12T05:37:19.333 INFO:tasks.workunit.client.0.smithi032.stdout:1/753: write d5/d2c/f43 [2743138,56088] 0 2024-04-12T05:37:19.336 INFO:tasks.workunit.client.0.smithi032.stdout:7/876: dwrite d0/dff/d71/f86 [0,4194304] 0 2024-04-12T05:37:19.339 INFO:tasks.workunit.client.1.smithi067.stdout:2/271: dread - d5/da/de/d14/d34/d4c/f4d zero size 2024-04-12T05:37:19.339 INFO:tasks.workunit.client.1.smithi067.stdout:8/215: dwrite d6/f27 [0,4194304] 0 2024-04-12T05:37:19.339 INFO:tasks.workunit.client.0.smithi032.stdout:7/877: write d0/d34/dbe/f7e [7760645,81497] 0 2024-04-12T05:37:19.342 INFO:tasks.workunit.client.0.smithi032.stdout:9/720: creat d9/d32/d6e/ff2 x:0 0 0 2024-04-12T05:37:19.343 INFO:tasks.workunit.client.0.smithi032.stdout:2/870: mkdir dd/d19/d9d/d42/da7/d115 0 2024-04-12T05:37:19.344 INFO:tasks.workunit.client.1.smithi067.stdout:7/268: dwrite d0/d2/d5/d1d/f40 [0,4194304] 0 2024-04-12T05:37:19.353 INFO:tasks.workunit.client.1.smithi067.stdout:4/213: rename d8/da/d1d/d15/d1c/l28 to d8/d9/d3d/d48/l4a 0 2024-04-12T05:37:19.354 INFO:tasks.workunit.client.1.smithi067.stdout:8/216: readlink d6/d8/d18/l34 0 2024-04-12T05:37:19.354 INFO:tasks.workunit.client.1.smithi067.stdout:6/279: chown dd/d21/d23/l2c 27152 1 2024-04-12T05:37:19.355 INFO:tasks.workunit.client.0.smithi032.stdout:7/878: unlink d0/d34/fcb 0 2024-04-12T05:37:19.355 INFO:tasks.workunit.client.0.smithi032.stdout:4/881: creat d4/d87/d96/d2f/d94/dc6/d123/f125 x:0 0 0 2024-04-12T05:37:19.355 INFO:tasks.workunit.client.1.smithi067.stdout:9/258: dread d1/fd [0,4194304] 0 2024-04-12T05:37:19.356 INFO:tasks.workunit.client.1.smithi067.stdout:1/252: truncate d4/d9/d12/d28/f2b 330758 0 2024-04-12T05:37:19.356 INFO:tasks.workunit.client.0.smithi032.stdout:1/754: stat d5/de/d17/d35/d78/ld8 0 2024-04-12T05:37:19.356 INFO:tasks.workunit.client.0.smithi032.stdout:0/892: mknod d2/d8/df/d24/dc3/c12b 0 2024-04-12T05:37:19.358 INFO:tasks.workunit.client.0.smithi032.stdout:5/982: getdents d2/df/da9/dc1/d78/d120/dc4/de1 0 2024-04-12T05:37:19.359 INFO:tasks.workunit.client.1.smithi067.stdout:8/217: fdatasync d6/d8/f14 0 2024-04-12T05:37:19.360 INFO:tasks.workunit.client.0.smithi032.stdout:4/882: symlink d4/d87/d96/d4d/d78/l126 0 2024-04-12T05:37:19.363 INFO:tasks.workunit.client.0.smithi032.stdout:1/755: fdatasync d5/de/d17/d35/d61/d77/d8f/d9c/fdb 0 2024-04-12T05:37:19.363 INFO:tasks.workunit.client.1.smithi067.stdout:2/272: write d5/da/de/f13 [476464,26798] 0 2024-04-12T05:37:19.364 INFO:tasks.workunit.client.0.smithi032.stdout:5/983: rmdir d2/df/da9/dc1/d78/d101 39 2024-04-12T05:37:19.365 INFO:tasks.workunit.client.0.smithi032.stdout:1/756: write d5/de/d17/f20 [3007177,111920] 0 2024-04-12T05:37:19.374 INFO:tasks.workunit.client.1.smithi067.stdout:5/439: mknod d0/d6/d40/d4b/d38/d8e/d8f/c97 0 2024-04-12T05:37:19.374 INFO:tasks.workunit.client.1.smithi067.stdout:7/269: dread d0/d2/d5/d1d/f3a [0,4194304] 0 2024-04-12T05:37:19.375 INFO:tasks.workunit.client.1.smithi067.stdout:4/214: dread d8/f2b [0,4194304] 0 2024-04-12T05:37:19.376 INFO:tasks.workunit.client.0.smithi032.stdout:0/893: link d2/c127 d2/d29/d77/dc2/c12c 0 2024-04-12T05:37:19.377 INFO:tasks.workunit.client.1.smithi067.stdout:7/270: mkdir d0/d2/d5/d1e/d51 0 2024-04-12T05:37:19.377 INFO:tasks.workunit.client.0.smithi032.stdout:0/894: readlink d2/d28/d87/d79/la9 0 2024-04-12T05:37:19.378 INFO:tasks.workunit.client.0.smithi032.stdout:5/984: symlink d2/df/d25/d50/d91/d13a/l13d 0 2024-04-12T05:37:19.379 INFO:tasks.workunit.client.0.smithi032.stdout:1/757: symlink d5/de/d17/d35/d3b/dda/de0/lee 0 2024-04-12T05:37:19.379 INFO:tasks.workunit.client.1.smithi067.stdout:1/253: mkdir d4/d52 0 2024-04-12T05:37:19.379 INFO:tasks.workunit.client.0.smithi032.stdout:4/883: getdents d4/d87/d96/d2f/d94/dc6 0 2024-04-12T05:37:19.380 INFO:tasks.workunit.client.1.smithi067.stdout:8/218: symlink d6/d8/d18/d36/l43 0 2024-04-12T05:37:19.380 INFO:tasks.workunit.client.0.smithi032.stdout:5/985: stat d2/df/da9/dc1/d78/l90 0 2024-04-12T05:37:19.380 INFO:tasks.workunit.client.0.smithi032.stdout:0/895: write d2/d28/d87/d111/d3e/dac/f54 [691832,28408] 0 2024-04-12T05:37:19.381 INFO:tasks.workunit.client.0.smithi032.stdout:7/879: dread d0/d34/f8d [0,4194304] 0 2024-04-12T05:37:19.382 INFO:tasks.workunit.client.1.smithi067.stdout:2/273: symlink d5/da/de/l58 0 2024-04-12T05:37:19.382 INFO:tasks.workunit.client.1.smithi067.stdout:4/215: creat d8/da/d1d/d15/d1c/f4b x:0 0 0 2024-04-12T05:37:19.383 INFO:tasks.workunit.client.0.smithi032.stdout:4/884: read d4/d13/d44/f6a [2680584,57733] 0 2024-04-12T05:37:19.383 INFO:tasks.workunit.client.0.smithi032.stdout:0/896: fdatasync d2/d28/d87/d111/fd 0 2024-04-12T05:37:19.384 INFO:tasks.workunit.client.0.smithi032.stdout:5/986: unlink d2/f68 0 2024-04-12T05:37:19.385 INFO:tasks.workunit.client.0.smithi032.stdout:5/987: chown d2/df/d25/d1e 374808597 1 2024-04-12T05:37:19.385 INFO:tasks.workunit.client.1.smithi067.stdout:8/219: creat d6/d8/d18/d36/f44 x:0 0 0 2024-04-12T05:37:19.387 INFO:tasks.workunit.client.0.smithi032.stdout:7/880: fsync d0/d13/d21/d30/fd3 0 2024-04-12T05:37:19.389 INFO:tasks.workunit.client.0.smithi032.stdout:0/897: creat d2/d8/d48/f12d x:0 0 0 2024-04-12T05:37:19.390 INFO:tasks.workunit.client.0.smithi032.stdout:7/881: chown d0/d10/f63 252079 1 2024-04-12T05:37:19.390 INFO:tasks.workunit.client.0.smithi032.stdout:0/898: chown d2/f14 5 1 2024-04-12T05:37:19.392 INFO:tasks.workunit.client.0.smithi032.stdout:1/758: creat d5/de/fef x:0 0 0 2024-04-12T05:37:19.393 INFO:tasks.workunit.client.0.smithi032.stdout:0/899: chown d2/d28/d87/d111/d3e/f46 0 1 2024-04-12T05:37:19.395 INFO:tasks.workunit.client.0.smithi032.stdout:4/885: rename d4/d13/d44/f5a to d4/d87/d96/d2f/d94/f127 0 2024-04-12T05:37:19.401 INFO:tasks.workunit.client.0.smithi032.stdout:7/882: symlink d0/dff/d71/d119/l120 0 2024-04-12T05:37:19.403 INFO:tasks.workunit.client.0.smithi032.stdout:0/900: chown d2/d29/d77/ce9 26 1 2024-04-12T05:37:19.405 INFO:tasks.workunit.client.1.smithi067.stdout:2/274: dread d5/da/de/f26 [0,4194304] 0 2024-04-12T05:37:19.409 INFO:tasks.workunit.client.1.smithi067.stdout:7/271: creat d0/d2/f52 x:0 0 0 2024-04-12T05:37:19.409 INFO:tasks.workunit.client.1.smithi067.stdout:6/280: write fc [638241,116086] 0 2024-04-12T05:37:19.410 INFO:tasks.workunit.client.0.smithi032.stdout:7/883: symlink d0/d10/d70/d7b/l121 0 2024-04-12T05:37:19.411 INFO:tasks.workunit.client.1.smithi067.stdout:1/254: creat d4/d9/d12/f53 x:0 0 0 2024-04-12T05:37:19.412 INFO:tasks.workunit.client.0.smithi032.stdout:7/884: dread - d0/d34/dbe/dbb/de8/f11c zero size 2024-04-12T05:37:19.413 INFO:tasks.workunit.client.0.smithi032.stdout:7/885: chown d0/de/df/d29 2836 1 2024-04-12T05:37:19.413 INFO:tasks.workunit.client.1.smithi067.stdout:9/259: write d1/d28/d23/f24 [46524,94487] 0 2024-04-12T05:37:19.415 INFO:tasks.workunit.client.0.smithi032.stdout:0/901: rename d2/d29/d8a/df2 to d2/d29/d77/dc2/d12e 0 2024-04-12T05:37:19.416 INFO:tasks.workunit.client.0.smithi032.stdout:2/871: write dd/d8c/fe2 [1560275,86481] 0 2024-04-12T05:37:19.417 INFO:tasks.workunit.client.1.smithi067.stdout:6/281: creat dd/d11/f5b x:0 0 0 2024-04-12T05:37:19.417 INFO:tasks.workunit.client.1.smithi067.stdout:2/275: dwrite d5/da/de/d14/f36 [0,4194304] 0 2024-04-12T05:37:19.420 INFO:tasks.workunit.client.1.smithi067.stdout:1/255: truncate d4/fe 816010 0 2024-04-12T05:37:19.422 INFO:tasks.workunit.client.1.smithi067.stdout:7/272: dread d0/d2/d5/d1d/f36 [0,4194304] 0 2024-04-12T05:37:19.423 INFO:tasks.workunit.client.1.smithi067.stdout:7/273: chown d0/d23/d45/l3d 3363028 1 2024-04-12T05:37:19.424 INFO:tasks.workunit.client.0.smithi032.stdout:2/872: creat dd/d19/d9d/d1c/de0/f116 x:0 0 0 2024-04-12T05:37:19.424 INFO:tasks.workunit.client.1.smithi067.stdout:7/274: chown d0/d2/d5/d1e 15 1 2024-04-12T05:37:19.425 INFO:tasks.workunit.client.1.smithi067.stdout:5/440: truncate d0/d3a/d2c/f37 173392 0 2024-04-12T05:37:19.426 INFO:tasks.workunit.client.1.smithi067.stdout:5/441: truncate d0/da/f5a 1408158 0 2024-04-12T05:37:19.428 INFO:tasks.workunit.client.1.smithi067.stdout:5/442: read d0/da/f5a [696385,13624] 0 2024-04-12T05:37:19.429 INFO:tasks.workunit.client.1.smithi067.stdout:5/443: read - d0/d6/f76 zero size 2024-04-12T05:37:19.434 INFO:tasks.workunit.client.1.smithi067.stdout:2/276: dread d5/d11/f50 [0,4194304] 0 2024-04-12T05:37:19.435 INFO:tasks.workunit.client.0.smithi032.stdout:5/988: fsync d2/df/da9/dc1/d6c/df3/f117 0 2024-04-12T05:37:19.437 INFO:tasks.workunit.client.0.smithi032.stdout:2/873: mknod dd/d19/d20/dc6/c117 0 2024-04-12T05:37:19.437 INFO:tasks.workunit.client.1.smithi067.stdout:1/256: mknod d4/d9/c54 0 2024-04-12T05:37:19.440 INFO:tasks.workunit.client.0.smithi032.stdout:7/886: rename d0/d13/l18 to d0/de/df/l122 0 2024-04-12T05:37:19.442 INFO:tasks.workunit.client.1.smithi067.stdout:9/260: getdents d1/d12 0 2024-04-12T05:37:19.443 INFO:tasks.workunit.client.0.smithi032.stdout:2/874: mkdir dd/d118 0 2024-04-12T05:37:19.445 INFO:tasks.workunit.client.1.smithi067.stdout:6/282: getdents dd/d11/d27/d4a/d51 0 2024-04-12T05:37:19.448 INFO:tasks.workunit.client.0.smithi032.stdout:7/887: write d0/de/ff8 [5438056,58122] 0 2024-04-12T05:37:19.449 INFO:tasks.workunit.client.0.smithi032.stdout:1/759: truncate d5/de/d2a/f97 1175038 0 2024-04-12T05:37:19.452 INFO:tasks.workunit.client.0.smithi032.stdout:4/886: dwrite d4/d87/d96/d2f/d7c/dc2/f124 [0,4194304] 0 2024-04-12T05:37:19.452 INFO:tasks.workunit.client.1.smithi067.stdout:9/261: mkdir d1/d28/d27/d32/d4e 0 2024-04-12T05:37:19.454 INFO:tasks.workunit.client.1.smithi067.stdout:6/283: symlink dd/d21/l5c 0 2024-04-12T05:37:19.455 INFO:tasks.workunit.client.1.smithi067.stdout:9/262: mkdir d1/d28/d21/d3f/d4f 0 2024-04-12T05:37:19.457 INFO:tasks.workunit.client.1.smithi067.stdout:6/284: creat dd/d11/f5d x:0 0 0 2024-04-12T05:37:19.457 INFO:tasks.workunit.client.0.smithi032.stdout:5/989: dread d2/dd/d5d/fce [0,4194304] 0 2024-04-12T05:37:19.457 INFO:tasks.workunit.client.0.smithi032.stdout:0/902: dread d2/d8/d17/f6b [0,4194304] 0 2024-04-12T05:37:19.457 INFO:tasks.workunit.client.0.smithi032.stdout:4/887: readlink d4/d87/d96/d2f/lc8 0 2024-04-12T05:37:19.458 INFO:tasks.workunit.client.1.smithi067.stdout:9/263: write d1/d28/d21/f48 [882048,94673] 0 2024-04-12T05:37:19.458 INFO:tasks.workunit.client.1.smithi067.stdout:6/285: chown dd/d11/d27/d4a/d51/f57 1159 1 2024-04-12T05:37:19.462 INFO:tasks.workunit.client.0.smithi032.stdout:1/760: read - d5/de/d17/d35/d78/fd7 zero size 2024-04-12T05:37:19.463 INFO:tasks.workunit.client.0.smithi032.stdout:7/888: symlink d0/dd/d51/dea/d105/l123 0 2024-04-12T05:37:19.464 INFO:tasks.workunit.client.0.smithi032.stdout:2/875: dwrite dd/d19/d9d/d1c/d64/d32/ff5 [0,4194304] 0 2024-04-12T05:37:19.465 INFO:tasks.workunit.client.0.smithi032.stdout:0/903: symlink d2/d28/d87/d79/l12f 0 2024-04-12T05:37:19.467 INFO:tasks.workunit.client.0.smithi032.stdout:7/889: dread - d0/d10/f63 zero size 2024-04-12T05:37:19.469 INFO:tasks.workunit.client.0.smithi032.stdout:1/761: stat d5/de/d17/f26 0 2024-04-12T05:37:19.472 INFO:tasks.workunit.client.1.smithi067.stdout:8/220: rename d6/f7 to d6/d8/f45 0 2024-04-12T05:37:19.473 INFO:tasks.workunit.client.0.smithi032.stdout:5/990: rename d2/df/da9/dc1/d6c/ff4 to d2/dd/ddf/f13e 0 2024-04-12T05:37:19.473 INFO:tasks.workunit.client.0.smithi032.stdout:7/890: dread - d0/d13/d21/d7c/fe4 zero size 2024-04-12T05:37:19.475 INFO:tasks.workunit.client.0.smithi032.stdout:2/876: readlink dd/d19/d9d/d1c/d64/d32/d3a/lef 0 2024-04-12T05:37:19.475 INFO:tasks.workunit.client.1.smithi067.stdout:8/221: write f5 [3070345,8148] 0 2024-04-12T05:37:19.475 INFO:tasks.workunit.client.0.smithi032.stdout:7/891: fsync d0/dff/d4c/fc7 0 2024-04-12T05:37:19.476 INFO:tasks.workunit.client.1.smithi067.stdout:2/277: truncate d5/da/de/d14/f24 228217 0 2024-04-12T05:37:19.476 INFO:tasks.workunit.client.1.smithi067.stdout:5/444: write d0/d3a/f4c [507098,42708] 0 2024-04-12T05:37:19.477 INFO:tasks.workunit.client.0.smithi032.stdout:2/877: stat dd/d19/d9d/d42/d4a/da1/dd5 0 2024-04-12T05:37:19.479 INFO:tasks.workunit.client.1.smithi067.stdout:7/275: rmdir d0/d2/d5/dc 39 2024-04-12T05:37:19.479 INFO:tasks.workunit.client.1.smithi067.stdout:5/445: write d0/d6/f6e [202427,22846] 0 2024-04-12T05:37:19.479 INFO:tasks.workunit.client.1.smithi067.stdout:2/278: readlink d5/da/d3d/l42 0 2024-04-12T05:37:19.480 INFO:tasks.workunit.client.1.smithi067.stdout:6/286: truncate f9 1175312 0 2024-04-12T05:37:19.480 INFO:tasks.workunit.client.0.smithi032.stdout:1/762: creat d5/de/d17/d35/d3e/db4/ff0 x:0 0 0 2024-04-12T05:37:19.480 INFO:tasks.workunit.client.0.smithi032.stdout:4/888: mknod d4/d87/d96/d4d/d78/dba/dfc/c128 0 2024-04-12T05:37:19.481 INFO:tasks.workunit.client.1.smithi067.stdout:5/446: stat d0/d6/c47 0 2024-04-12T05:37:19.482 INFO:tasks.workunit.client.1.smithi067.stdout:2/279: chown d5/d11/f50 895761 1 2024-04-12T05:37:19.483 INFO:tasks.workunit.client.0.smithi032.stdout:7/892: unlink d0/d34/dbe/c6e 0 2024-04-12T05:37:19.486 INFO:tasks.workunit.client.0.smithi032.stdout:4/889: write d4/d13/d75/fbd [692624,23149] 0 2024-04-12T05:37:19.486 INFO:tasks.workunit.client.0.smithi032.stdout:0/904: write d2/d29/f70 [932278,58608] 0 2024-04-12T05:37:19.486 INFO:tasks.workunit.client.1.smithi067.stdout:5/447: symlink d0/da/d3f/l98 0 2024-04-12T05:37:19.487 INFO:tasks.workunit.client.0.smithi032.stdout:5/991: dwrite d2/d9b/f5a [0,4194304] 0 2024-04-12T05:37:19.489 INFO:tasks.workunit.client.1.smithi067.stdout:2/280: unlink d5/l9 0 2024-04-12T05:37:19.490 INFO:tasks.workunit.client.1.smithi067.stdout:7/276: dwrite d0/f12 [0,4194304] 0 2024-04-12T05:37:19.493 INFO:tasks.workunit.client.0.smithi032.stdout:1/763: truncate d5/de/d17/d35/d3e/db4/ff0 997926 0 2024-04-12T05:37:19.493 INFO:tasks.workunit.client.0.smithi032.stdout:7/893: creat d0/d13/d21/d81/f124 x:0 0 0 2024-04-12T05:37:19.496 INFO:tasks.workunit.client.0.smithi032.stdout:2/878: dread dd/d19/d9d/d42/d4a/fdd [0,4194304] 0 2024-04-12T05:37:19.497 INFO:tasks.workunit.client.0.smithi032.stdout:5/992: read - d2/df/da9/dc1/fde zero size 2024-04-12T05:37:19.499 INFO:tasks.workunit.client.0.smithi032.stdout:4/890: rmdir d4/d13/d75 39 2024-04-12T05:37:19.501 INFO:tasks.workunit.client.0.smithi032.stdout:7/894: read d0/de/d47/fb0 [870746,19942] 0 2024-04-12T05:37:19.504 INFO:tasks.workunit.client.0.smithi032.stdout:4/891: unlink d4/d87/d96/d2f/f60 0 2024-04-12T05:37:19.506 INFO:tasks.workunit.client.1.smithi067.stdout:8/222: write d6/d8/d18/df/f1b [1093066,96802] 0 2024-04-12T05:37:19.506 INFO:tasks.workunit.client.1.smithi067.stdout:5/448: rmdir d0/d6/d20/d62/d75 0 2024-04-12T05:37:19.506 INFO:tasks.workunit.client.0.smithi032.stdout:4/892: write d4/d87/d96/d2f/f70 [2656319,61914] 0 2024-04-12T05:37:19.508 INFO:tasks.workunit.client.0.smithi032.stdout:4/893: dread - d4/d87/d96/d4d/db5/dcf/de3/ff1 zero size 2024-04-12T05:37:19.508 INFO:tasks.workunit.client.0.smithi032.stdout:0/905: rename d2/c76 to d2/d28/d87/c130 0 2024-04-12T05:37:19.508 INFO:tasks.workunit.client.1.smithi067.stdout:2/281: write d5/d11/f2a [157587,17604] 0 2024-04-12T05:37:19.510 INFO:tasks.workunit.client.0.smithi032.stdout:7/895: getdents d0/dff/d4c 0 2024-04-12T05:37:19.510 INFO:tasks.workunit.client.1.smithi067.stdout:5/449: mkdir d0/d6/d40/d4b/d38/d8e/d8f/d99 0 2024-04-12T05:37:19.511 INFO:tasks.workunit.client.1.smithi067.stdout:2/282: write d5/da/de/f52 [135617,83629] 0 2024-04-12T05:37:19.511 INFO:tasks.workunit.client.0.smithi032.stdout:4/894: write d4/d87/d96/d4d/db5/dcf/de3/f10c [299878,11166] 0 2024-04-12T05:37:19.512 INFO:tasks.workunit.client.1.smithi067.stdout:2/283: stat d5/da/d3d/l42 0 2024-04-12T05:37:19.513 INFO:tasks.workunit.client.0.smithi032.stdout:5/993: rename d2/df/da9/dc1/d78/f94 to d2/df/d25/d50/d91/d13a/f13f 0 2024-04-12T05:37:19.514 INFO:tasks.workunit.client.0.smithi032.stdout:0/906: symlink d2/d28/d87/l131 0 2024-04-12T05:37:19.515 INFO:tasks.workunit.client.0.smithi032.stdout:7/896: creat d0/dd/d51/dea/f125 x:0 0 0 2024-04-12T05:37:19.515 INFO:tasks.workunit.client.0.smithi032.stdout:0/907: dread - d2/d28/d87/d111/d3e/ff1 zero size 2024-04-12T05:37:19.517 INFO:tasks.workunit.client.0.smithi032.stdout:4/895: mkdir d4/d87/d96/d68/d9b/d129 0 2024-04-12T05:37:19.518 INFO:tasks.workunit.client.0.smithi032.stdout:5/994: creat d2/dd/d6f/f140 x:0 0 0 2024-04-12T05:37:19.520 INFO:tasks.workunit.client.0.smithi032.stdout:1/764: write d5/de/d17/d35/d61/f50 [1385194,31420] 0 2024-04-12T05:37:19.524 INFO:tasks.workunit.client.0.smithi032.stdout:4/896: creat d4/d87/d96/d4d/d78/dba/dfc/f12a x:0 0 0 2024-04-12T05:37:19.526 INFO:tasks.workunit.client.0.smithi032.stdout:1/765: write d5/de/d17/d35/d3e/db4/ff0 [176249,62235] 0 2024-04-12T05:37:19.526 INFO:tasks.workunit.client.0.smithi032.stdout:7/897: dwrite d0/dd/ffc [0,4194304] 0 2024-04-12T05:37:19.528 INFO:tasks.workunit.client.0.smithi032.stdout:4/897: dread - d4/d87/d96/d4d/d78/dba/dfc/f12a zero size 2024-04-12T05:37:19.529 INFO:tasks.workunit.client.0.smithi032.stdout:0/908: write d2/d28/d87/d79/fab [139728,130263] 0 2024-04-12T05:37:19.531 INFO:tasks.workunit.client.0.smithi032.stdout:4/898: stat d4/d13/d44/f91 0 2024-04-12T05:37:19.532 INFO:tasks.workunit.client.0.smithi032.stdout:1/766: unlink d5/de/d17/d35/d61/d77/d8f/fc7 0 2024-04-12T05:37:19.534 INFO:tasks.workunit.client.0.smithi032.stdout:7/898: write d0/dd/d51/fd7 [6743641,129292] 0 2024-04-12T05:37:19.537 INFO:tasks.workunit.client.0.smithi032.stdout:0/909: write d2/d28/d87/d111/d41/fef [5127364,115168] 0 2024-04-12T05:37:19.538 INFO:tasks.workunit.client.0.smithi032.stdout:5/995: dwrite d2/f24 [0,4194304] 0 2024-04-12T05:37:19.538 INFO:tasks.workunit.client.0.smithi032.stdout:4/899: creat d4/d87/d96/f12b x:0 0 0 2024-04-12T05:37:19.539 INFO:tasks.workunit.client.0.smithi032.stdout:1/767: mknod d5/de/d17/d35/d3e/cf1 0 2024-04-12T05:37:19.540 INFO:tasks.workunit.client.0.smithi032.stdout:7/899: creat d0/de/df/d20/f126 x:0 0 0 2024-04-12T05:37:19.540 INFO:tasks.workunit.client.0.smithi032.stdout:4/900: dread - d4/d87/d96/d2f/ff8 zero size 2024-04-12T05:37:19.544 INFO:tasks.workunit.client.0.smithi032.stdout:5/996: write d2/df/d25/d50/fab [954541,73789] 0 2024-04-12T05:37:19.545 INFO:tasks.workunit.client.0.smithi032.stdout:1/768: dread - d5/de/d17/d35/d78/fd7 zero size 2024-04-12T05:37:19.547 INFO:tasks.workunit.client.0.smithi032.stdout:1/769: truncate d5/de/d17/d93/fc8 768477 0 2024-04-12T05:37:19.547 INFO:tasks.workunit.client.0.smithi032.stdout:7/900: rename d0/dd/d51/d65/f96 to d0/d13/d21/d30/dfb/f127 0 2024-04-12T05:37:19.548 INFO:tasks.workunit.client.0.smithi032.stdout:7/901: dread - d0/de/df/d29/fec zero size 2024-04-12T05:37:19.550 INFO:tasks.workunit.client.0.smithi032.stdout:5/997: creat d2/df/d25/d50/daa/f141 x:0 0 0 2024-04-12T05:37:19.551 INFO:tasks.workunit.client.0.smithi032.stdout:0/910: truncate d2/d29/d8a/fd7 4857 0 2024-04-12T05:37:19.552 INFO:tasks.workunit.client.0.smithi032.stdout:5/998: write d2/df/d25/d50/d91/d13a/de7/f113 [484991,46449] 0 2024-04-12T05:37:19.553 INFO:tasks.workunit.client.0.smithi032.stdout:0/911: chown d2/d8/d17/d1b/d109 21772 1 2024-04-12T05:37:19.554 INFO:tasks.workunit.client.0.smithi032.stdout:7/902: link d0/d34/dbe/ld6 d0/de/d47/de1/l128 0 2024-04-12T05:37:19.560 INFO:tasks.workunit.client.0.smithi032.stdout:7/903: mknod d0/dd/d51/c129 0 2024-04-12T05:37:19.561 INFO:tasks.workunit.client.0.smithi032.stdout:1/770: write d5/de/d17/d35/d78/f83 [3753365,22916] 0 2024-04-12T05:37:19.562 INFO:tasks.workunit.client.0.smithi032.stdout:1/771: fdatasync d5/de/d17/d35/d61/d77/d8f/d9c/fa5 0 2024-04-12T05:37:19.564 INFO:tasks.workunit.client.0.smithi032.stdout:0/912: write d2/d29/d8a/d99/fbf [1047508,45198] 0 2024-04-12T05:37:19.565 INFO:tasks.workunit.client.0.smithi032.stdout:5/999: truncate d2/d9b/f5a 3668022 0 2024-04-12T05:37:19.566 INFO:tasks.workunit.client.0.smithi032.stdout:1/772: truncate d5/f30 128014 0 2024-04-12T05:37:19.568 INFO:tasks.workunit.client.0.smithi032.stdout:0/913: mknod d2/d8/d119/c132 0 2024-04-12T05:37:19.569 INFO:tasks.workunit.client.0.smithi032.stdout:1/773: creat d5/d2c/ff2 x:0 0 0 2024-04-12T05:37:19.571 INFO:tasks.workunit.client.0.smithi032.stdout:7/904: dwrite d0/de/df/f7a [0,4194304] 0 2024-04-12T05:37:19.577 INFO:tasks.workunit.client.0.smithi032.stdout:7/905: rename d0/d13/d21/d30 to d0/dd/d51/d65/d12a 0 2024-04-12T05:37:19.582 INFO:tasks.workunit.client.0.smithi032.stdout:1/774: write d5/de/d17/d35/d61/d77/f7d [898356,23430] 0 2024-04-12T05:37:19.584 INFO:tasks.workunit.client.0.smithi032.stdout:1/775: dread - d5/de/d17/d35/d3e/fc9 zero size 2024-04-12T05:37:19.585 INFO:tasks.workunit.client.0.smithi032.stdout:1/776: readlink d5/de/d17/d1b/l24 0 2024-04-12T05:37:19.586 INFO:tasks.workunit.client.0.smithi032.stdout:1/777: chown d5/de/fcb 7167 1 2024-04-12T05:37:19.588 INFO:tasks.workunit.client.0.smithi032.stdout:0/914: dread d2/d28/d87/d111/d41/fef [0,4194304] 0 2024-04-12T05:37:19.593 INFO:tasks.workunit.client.0.smithi032.stdout:0/915: mkdir d2/d29/d77/dc7/d112/d115/d133 0 2024-04-12T05:37:19.594 INFO:tasks.workunit.client.0.smithi032.stdout:1/778: dwrite d5/de/d17/fce [0,4194304] 0 2024-04-12T05:37:19.595 INFO:tasks.workunit.client.0.smithi032.stdout:1/779: chown d5/fb8 999955177 1 2024-04-12T05:37:19.599 INFO:tasks.workunit.client.0.smithi032.stdout:1/780: creat d5/de/d2a/ff3 x:0 0 0 2024-04-12T05:37:19.601 INFO:tasks.workunit.client.0.smithi032.stdout:7/906: dread d0/d13/f16 [0,4194304] 0 2024-04-12T05:37:19.603 INFO:tasks.workunit.client.0.smithi032.stdout:1/781: creat d5/de/d17/d35/d3b/dda/de0/ff4 x:0 0 0 2024-04-12T05:37:19.604 INFO:tasks.workunit.client.0.smithi032.stdout:1/782: readlink d5/de/d17/d35/lc0 0 2024-04-12T05:37:19.606 INFO:tasks.workunit.client.0.smithi032.stdout:0/916: write d2/d28/d87/d79/ff8 [664577,59551] 0 2024-04-12T05:37:19.608 INFO:tasks.workunit.client.0.smithi032.stdout:1/783: unlink d5/de/d17/d35/d3b/lbb 0 2024-04-12T05:37:19.611 INFO:tasks.workunit.client.0.smithi032.stdout:7/907: getdents d0/d10/d70/d7b/dd2 0 2024-04-12T05:37:19.611 INFO:tasks.workunit.client.0.smithi032.stdout:7/908: chown d0/dff/d71/la3 812 1 2024-04-12T05:37:19.612 INFO:tasks.workunit.client.0.smithi032.stdout:7/909: dread - d0/d13/d21/d81/f124 zero size 2024-04-12T05:37:19.615 INFO:tasks.workunit.client.0.smithi032.stdout:1/784: link d5/de/d17/d35/d73/ca6 d5/de/d17/d35/d73/d74/cf5 0 2024-04-12T05:37:19.624 INFO:tasks.workunit.client.0.smithi032.stdout:0/917: dread d2/d8/d17/fd8 [0,4194304] 0 2024-04-12T05:37:19.624 INFO:tasks.workunit.client.0.smithi032.stdout:0/918: readlink d2/d28/d87/d111/d41/l10f 0 2024-04-12T05:37:19.625 INFO:tasks.workunit.client.0.smithi032.stdout:0/919: write d2/d8/d48/f12d [624533,28995] 0 2024-04-12T05:37:19.627 INFO:tasks.workunit.client.0.smithi032.stdout:1/785: dread d5/de/d17/d35/d78/fb1 [0,4194304] 0 2024-04-12T05:37:19.628 INFO:tasks.workunit.client.0.smithi032.stdout:1/786: dread - d5/de/d17/d35/d61/fdf zero size 2024-04-12T05:37:19.632 INFO:tasks.workunit.client.0.smithi032.stdout:1/787: rmdir d5/de/d2a 39 2024-04-12T05:37:19.635 INFO:tasks.workunit.client.0.smithi032.stdout:0/920: rename d2/d8/df/d24/c40 to d2/d29/d8a/c134 0 2024-04-12T05:37:19.666 INFO:tasks.workunit.client.1.smithi067.stdout:5/450: write d0/d6/d20/d62/f15 [2238957,84886] 0 2024-04-12T05:37:19.667 INFO:tasks.workunit.client.1.smithi067.stdout:8/223: rename d6/d8/d18/df/d12/c2d to d6/d8/d18/c46 0 2024-04-12T05:37:19.667 INFO:tasks.workunit.client.1.smithi067.stdout:7/277: creat d0/d2/d5/db/f53 x:0 0 0 2024-04-12T05:37:19.667 INFO:tasks.workunit.client.1.smithi067.stdout:5/451: chown d0/da/d3f/l98 0 1 2024-04-12T05:37:19.667 INFO:tasks.workunit.client.1.smithi067.stdout:8/224: chown d6/d8/d18/d3c/c40 5 1 2024-04-12T05:37:19.668 INFO:tasks.workunit.client.1.smithi067.stdout:2/284: write d5/da/de/d14/d34/d4c/f4d [378025,35396] 0 2024-04-12T05:37:19.670 INFO:tasks.workunit.client.1.smithi067.stdout:7/278: rename d0/d2/d5/d1d/l24 to d0/d23/l54 0 2024-04-12T05:37:19.672 INFO:tasks.workunit.client.1.smithi067.stdout:6/287: dwrite f9 [0,4194304] 0 2024-04-12T05:37:19.673 INFO:tasks.workunit.client.1.smithi067.stdout:2/285: fdatasync d5/da/de/d14/d34/f37 0 2024-04-12T05:37:19.674 INFO:tasks.workunit.client.1.smithi067.stdout:6/288: stat dd/d21/d23/f36 0 2024-04-12T05:37:19.675 INFO:tasks.workunit.client.0.smithi032.stdout:3/958: sync 2024-04-12T05:37:19.675 INFO:tasks.workunit.client.0.smithi032.stdout:8/991: sync 2024-04-12T05:37:19.678 INFO:tasks.workunit.client.0.smithi032.stdout:8/992: symlink d0/d36/d10a/l155 0 2024-04-12T05:37:19.679 INFO:tasks.workunit.client.1.smithi067.stdout:8/225: truncate f5 2567885 0 2024-04-12T05:37:19.682 INFO:tasks.workunit.client.0.smithi032.stdout:3/959: unlink d12/d2f/d89/df0/dcb/ffb 0 2024-04-12T05:37:19.684 INFO:tasks.workunit.client.1.smithi067.stdout:2/286: creat d5/d23/d3a/f59 x:0 0 0 2024-04-12T05:37:19.685 INFO:tasks.workunit.client.1.smithi067.stdout:7/279: stat d0/d2/d5/db/f30 0 2024-04-12T05:37:19.686 INFO:tasks.workunit.client.1.smithi067.stdout:6/289: mkdir dd/d21/d23/d5e 0 2024-04-12T05:37:19.688 INFO:tasks.workunit.client.1.smithi067.stdout:8/226: dwrite d6/d8/d18/f3f [0,4194304] 0 2024-04-12T05:37:19.689 INFO:tasks.workunit.client.1.smithi067.stdout:5/452: link d0/da/d3f/f49 d0/d6/d40/d4b/d38/d8e/d8f/d99/f9a 0 2024-04-12T05:37:19.690 INFO:tasks.workunit.client.0.smithi032.stdout:3/960: mknod d12/c13c 0 2024-04-12T05:37:19.694 INFO:tasks.workunit.client.1.smithi067.stdout:8/227: write d6/d8/d18/f25 [2614195,16204] 0 2024-04-12T05:37:19.695 INFO:tasks.workunit.client.0.smithi032.stdout:3/961: symlink d12/d2f/de2/d107/d111/l13d 0 2024-04-12T05:37:19.698 INFO:tasks.workunit.client.1.smithi067.stdout:6/290: creat dd/d30/f5f x:0 0 0 2024-04-12T05:37:19.698 INFO:tasks.workunit.client.1.smithi067.stdout:5/453: creat d0/da/d41/f9b x:0 0 0 2024-04-12T05:37:19.698 INFO:tasks.workunit.client.1.smithi067.stdout:6/291: read dd/d11/d27/f2f [196192,78034] 0 2024-04-12T05:37:19.700 INFO:tasks.workunit.client.1.smithi067.stdout:2/287: creat d5/da/de/d14/d46/f5a x:0 0 0 2024-04-12T05:37:19.703 INFO:tasks.workunit.client.0.smithi032.stdout:3/962: creat d12/d2f/d89/df0/d3a/d63/dd5/f13e x:0 0 0 2024-04-12T05:37:19.705 INFO:tasks.workunit.client.0.smithi032.stdout:3/963: dread - d12/d2f/d89/df0/dcd/f11f zero size 2024-04-12T05:37:19.708 INFO:tasks.workunit.client.1.smithi067.stdout:7/280: link d0/f1 d0/f55 0 2024-04-12T05:37:19.709 INFO:tasks.workunit.client.1.smithi067.stdout:6/292: symlink dd/d21/d23/l60 0 2024-04-12T05:37:19.711 INFO:tasks.workunit.client.1.smithi067.stdout:6/293: fdatasync dd/d21/d23/f43 0 2024-04-12T05:37:19.713 INFO:tasks.workunit.client.1.smithi067.stdout:6/294: write dd/f3b [694456,122075] 0 2024-04-12T05:37:19.715 INFO:tasks.workunit.client.1.smithi067.stdout:7/281: dread d0/d23/f41 [0,4194304] 0 2024-04-12T05:37:19.716 INFO:tasks.workunit.client.1.smithi067.stdout:7/282: dread - d0/d2/d5/f29 zero size 2024-04-12T05:37:19.718 INFO:tasks.workunit.client.1.smithi067.stdout:5/454: fdatasync d0/d6/d40/d4b/d38/d8e/d8f/d99/f9a 0 2024-04-12T05:37:19.724 INFO:tasks.workunit.client.1.smithi067.stdout:8/228: truncate d6/d8/d18/df/f23 5451341 0 2024-04-12T05:37:19.725 INFO:tasks.workunit.client.0.smithi032.stdout:3/964: symlink d12/d2f/d89/df0/d3a/d2d/d134/l13f 0 2024-04-12T05:37:19.725 INFO:tasks.workunit.client.1.smithi067.stdout:8/229: chown d6/l21 11288388 1 2024-04-12T05:37:19.726 INFO:tasks.workunit.client.0.smithi032.stdout:8/993: dread d0/db/dd/d11/f2c [4194304,4194304] 0 2024-04-12T05:37:19.732 INFO:tasks.workunit.client.0.smithi032.stdout:3/965: creat d12/d2f/d89/df0/d3a/d63/dd5/f140 x:0 0 0 2024-04-12T05:37:19.733 INFO:tasks.workunit.client.0.smithi032.stdout:3/966: chown d12/d24/lf8 55758547 1 2024-04-12T05:37:19.733 INFO:tasks.workunit.client.1.smithi067.stdout:3/224: sync 2024-04-12T05:37:19.735 INFO:tasks.workunit.client.1.smithi067.stdout:7/283: creat d0/d2/d5/dc/f56 x:0 0 0 2024-04-12T05:37:19.737 INFO:tasks.workunit.client.1.smithi067.stdout:8/230: unlink d6/d8/d18/l34 0 2024-04-12T05:37:19.738 INFO:tasks.workunit.client.1.smithi067.stdout:5/455: creat d0/d6/d40/d4b/d38/d8e/d8f/d99/f9c x:0 0 0 2024-04-12T05:37:19.739 INFO:tasks.workunit.client.0.smithi032.stdout:3/967: fdatasync d12/d2f/d89/df0/f4a 0 2024-04-12T05:37:19.742 INFO:tasks.workunit.client.1.smithi067.stdout:2/288: dwrite d5/da/de/d14/d34/f37 [0,4194304] 0 2024-04-12T05:37:19.743 INFO:tasks.workunit.client.1.smithi067.stdout:3/225: unlink dc/f23 0 2024-04-12T05:37:19.748 INFO:tasks.workunit.client.1.smithi067.stdout:7/284: mkdir d0/d2/d5/dc/d57 0 2024-04-12T05:37:19.750 INFO:tasks.workunit.client.1.smithi067.stdout:8/231: dwrite d6/d8/d18/f3f [0,4194304] 0 2024-04-12T05:37:19.753 INFO:tasks.workunit.client.1.smithi067.stdout:8/232: truncate d6/d8/d18/f41 697597 0 2024-04-12T05:37:19.757 INFO:tasks.workunit.client.1.smithi067.stdout:5/456: creat d0/d6/d33/f9d x:0 0 0 2024-04-12T05:37:19.757 INFO:tasks.workunit.client.1.smithi067.stdout:6/295: link dd/d21/d3f/l52 dd/d21/d3f/l61 0 2024-04-12T05:37:19.758 INFO:tasks.workunit.client.1.smithi067.stdout:2/289: mkdir d5/da/d5b 0 2024-04-12T05:37:19.759 INFO:tasks.workunit.client.0.smithi032.stdout:3/968: symlink d12/d2f/d89/df0/d3a/l141 0 2024-04-12T05:37:19.765 INFO:tasks.workunit.client.1.smithi067.stdout:6/296: rename dd/d11/d27/f2f to dd/d30/f62 0 2024-04-12T05:37:19.767 INFO:tasks.workunit.client.1.smithi067.stdout:2/290: chown d5/l4a 9 1 2024-04-12T05:37:19.768 INFO:tasks.workunit.client.0.smithi032.stdout:3/969: creat d12/d2f/d89/df0/d3a/d63/da6/df5/f142 x:0 0 0 2024-04-12T05:37:19.769 INFO:tasks.workunit.client.1.smithi067.stdout:5/457: mkdir d0/d6/d33/d9e 0 2024-04-12T05:37:19.770 INFO:tasks.workunit.client.1.smithi067.stdout:7/285: creat d0/d2/d5/d1e/d51/f58 x:0 0 0 2024-04-12T05:37:19.773 INFO:tasks.workunit.client.1.smithi067.stdout:6/297: mkdir dd/d21/d63 0 2024-04-12T05:37:19.775 INFO:tasks.workunit.client.1.smithi067.stdout:2/291: dwrite d5/d11/f2e [0,4194304] 0 2024-04-12T05:37:19.777 INFO:tasks.workunit.client.1.smithi067.stdout:2/292: truncate d5/d43/f44 535358 0 2024-04-12T05:37:19.777 INFO:tasks.workunit.client.1.smithi067.stdout:8/233: link d6/d8/d18/df/d12/f1f d6/d8/d18/df/f47 0 2024-04-12T05:37:19.779 INFO:tasks.workunit.client.0.smithi032.stdout:3/970: getdents d12/d2f/d89/df0/d3a/d63/da6/dac/dc9 0 2024-04-12T05:37:19.780 INFO:tasks.workunit.client.1.smithi067.stdout:6/298: mkdir dd/d21/d64 0 2024-04-12T05:37:19.780 INFO:tasks.workunit.client.1.smithi067.stdout:2/293: stat d5/da/de/d14/d34/d4c/f4d 0 2024-04-12T05:37:19.780 INFO:tasks.workunit.client.1.smithi067.stdout:3/226: getdents dc/d1d/d28/d3e 0 2024-04-12T05:37:19.782 INFO:tasks.workunit.client.1.smithi067.stdout:2/294: readlink d5/d43/l38 0 2024-04-12T05:37:19.783 INFO:tasks.workunit.client.0.smithi032.stdout:3/971: fdatasync d12/d24/f4d 0 2024-04-12T05:37:19.785 INFO:tasks.workunit.client.1.smithi067.stdout:6/299: rmdir dd 39 2024-04-12T05:37:19.785 INFO:tasks.workunit.client.1.smithi067.stdout:3/227: rmdir dc/d1d 39 2024-04-12T05:37:19.787 INFO:tasks.workunit.client.0.smithi032.stdout:3/972: dread - d12/d2f/d89/df0/d3a/d2d/f128 zero size 2024-04-12T05:37:19.789 INFO:tasks.workunit.client.1.smithi067.stdout:7/286: dwrite d0/d2/d5/f11 [4194304,4194304] 0 2024-04-12T05:37:19.793 INFO:tasks.workunit.client.0.smithi032.stdout:3/973: fdatasync d12/d2f/d89/df0/d3a/dbe/fe3 0 2024-04-12T05:37:19.794 INFO:tasks.workunit.client.0.smithi032.stdout:3/974: fdatasync d12/d2f/d89/df0/dcb/ffe 0 2024-04-12T05:37:19.796 INFO:tasks.workunit.client.1.smithi067.stdout:5/458: dread d0/d3a/d2c/d91/f7c [0,4194304] 0 2024-04-12T05:37:19.796 INFO:tasks.workunit.client.1.smithi067.stdout:2/295: mknod d5/d23/d3a/c5c 0 2024-04-12T05:37:19.796 INFO:tasks.workunit.client.1.smithi067.stdout:8/234: creat d6/f48 x:0 0 0 2024-04-12T05:37:19.798 INFO:tasks.workunit.client.1.smithi067.stdout:5/459: write d0/d6/f56 [915055,52904] 0 2024-04-12T05:37:19.799 INFO:tasks.workunit.client.1.smithi067.stdout:6/300: creat dd/d21/d23/f65 x:0 0 0 2024-04-12T05:37:19.800 INFO:tasks.workunit.client.1.smithi067.stdout:2/296: dread d5/da/de/f13 [0,4194304] 0 2024-04-12T05:37:19.802 INFO:tasks.workunit.client.1.smithi067.stdout:3/228: write dc/d1d/f36 [790494,72303] 0 2024-04-12T05:37:19.805 INFO:tasks.workunit.client.0.smithi032.stdout:9/721: sync 2024-04-12T05:37:19.806 INFO:tasks.workunit.client.1.smithi067.stdout:6/301: mkdir dd/d21/d66 0 2024-04-12T05:37:19.810 INFO:tasks.workunit.client.0.smithi032.stdout:9/722: chown d9/d58/f79 196 1 2024-04-12T05:37:19.811 INFO:tasks.workunit.client.1.smithi067.stdout:2/297: stat d5/da/de/f13 0 2024-04-12T05:37:19.811 INFO:tasks.workunit.client.1.smithi067.stdout:6/302: chown dd/d21/f24 143429222 1 2024-04-12T05:37:19.812 INFO:tasks.workunit.client.1.smithi067.stdout:8/235: rename d6/d8/d18/df/f13 to d6/d8/d18/df/f49 0 2024-04-12T05:37:19.813 INFO:tasks.workunit.client.1.smithi067.stdout:7/287: dwrite d0/d2/d5/d1e/d51/f58 [0,4194304] 0 2024-04-12T05:37:19.815 INFO:tasks.workunit.client.1.smithi067.stdout:6/303: fdatasync dd/f17 0 2024-04-12T05:37:19.816 INFO:tasks.workunit.client.1.smithi067.stdout:2/298: creat d5/da/de/d14/d34/d4c/f5d x:0 0 0 2024-04-12T05:37:19.818 INFO:tasks.workunit.client.0.smithi032.stdout:3/975: write d12/d2f/d89/d9d/fb6 [4917842,26413] 0 2024-04-12T05:37:19.819 INFO:tasks.workunit.client.1.smithi067.stdout:2/299: write d5/d11/f32 [387587,75956] 0 2024-04-12T05:37:19.821 INFO:tasks.workunit.client.1.smithi067.stdout:8/236: creat d6/d2f/f4a x:0 0 0 2024-04-12T05:37:19.821 INFO:tasks.workunit.client.1.smithi067.stdout:5/460: dwrite d0/d6/d40/d4b/d38/f79 [0,4194304] 0 2024-04-12T05:37:19.823 INFO:tasks.workunit.client.1.smithi067.stdout:2/300: write d5/d23/d3a/f59 [850921,22181] 0 2024-04-12T05:37:19.824 INFO:tasks.workunit.client.1.smithi067.stdout:8/237: readlink d6/d8/d18/d3c/l3d 0 2024-04-12T05:37:19.825 INFO:tasks.workunit.client.0.smithi032.stdout:3/976: write d12/d2f/d89/fd9 [6776907,78689] 0 2024-04-12T05:37:19.825 INFO:tasks.workunit.client.1.smithi067.stdout:3/229: rename dc/d1d/f36 to dc/d1d/f44 0 2024-04-12T05:37:19.826 INFO:tasks.workunit.client.1.smithi067.stdout:7/288: creat d0/d23/d45/f59 x:0 0 0 2024-04-12T05:37:19.829 INFO:tasks.workunit.client.1.smithi067.stdout:6/304: dwrite f6 [4194304,4194304] 0 2024-04-12T05:37:19.829 INFO:tasks.workunit.client.1.smithi067.stdout:5/461: creat d0/d6/d33/f9f x:0 0 0 2024-04-12T05:37:19.830 INFO:tasks.workunit.client.1.smithi067.stdout:3/230: write dc/d1d/d28/f38 [452227,24680] 0 2024-04-12T05:37:19.831 INFO:tasks.workunit.client.0.smithi032.stdout:2/879: sync 2024-04-12T05:37:19.831 INFO:tasks.workunit.client.0.smithi032.stdout:4/901: sync 2024-04-12T05:37:19.831 INFO:tasks.workunit.client.0.smithi032.stdout:7/910: sync 2024-04-12T05:37:19.833 INFO:tasks.workunit.client.1.smithi067.stdout:5/462: dread d0/da/d41/f60 [0,4194304] 0 2024-04-12T05:37:19.834 INFO:tasks.workunit.client.0.smithi032.stdout:3/977: rename d12/d2f/d89/df0/f67 to d12/daa/d12a/f143 0 2024-04-12T05:37:19.835 INFO:tasks.workunit.client.0.smithi032.stdout:7/911: symlink d0/dd/l12b 0 2024-04-12T05:37:19.836 INFO:tasks.workunit.client.0.smithi032.stdout:9/723: dread d9/f34 [0,4194304] 0 2024-04-12T05:37:19.836 INFO:tasks.workunit.client.1.smithi067.stdout:5/463: write d0/d3a/d2c/f8b [1020209,115193] 0 2024-04-12T05:37:19.836 INFO:tasks.workunit.client.1.smithi067.stdout:2/301: dwrite d5/da/de/f52 [0,4194304] 0 2024-04-12T05:37:19.837 INFO:tasks.workunit.client.1.smithi067.stdout:7/289: unlink d0/d23/f4c 0 2024-04-12T05:37:19.841 INFO:tasks.workunit.client.0.smithi032.stdout:9/724: fdatasync d9/fa1 0 2024-04-12T05:37:19.844 INFO:tasks.workunit.client.1.smithi067.stdout:8/238: symlink d6/d8/d18/df/l4b 0 2024-04-12T05:37:19.846 INFO:tasks.workunit.client.0.smithi032.stdout:2/880: rmdir dd/d19/d9d/d42/d4a/da1 39 2024-04-12T05:37:19.846 INFO:tasks.workunit.client.0.smithi032.stdout:4/902: creat d4/d87/d96/d4d/f12c x:0 0 0 2024-04-12T05:37:19.846 INFO:tasks.workunit.client.1.smithi067.stdout:2/302: unlink d5/d23/d3a/c5c 0 2024-04-12T05:37:19.846 INFO:tasks.workunit.client.1.smithi067.stdout:8/239: write d6/d8/d18/d36/f44 [845030,70518] 0 2024-04-12T05:37:19.848 INFO:tasks.workunit.client.0.smithi032.stdout:3/978: dwrite d12/d2f/f11c [0,4194304] 0 2024-04-12T05:37:19.849 INFO:tasks.workunit.client.0.smithi032.stdout:9/725: creat d9/d32/d86/ff3 x:0 0 0 2024-04-12T05:37:19.851 INFO:tasks.workunit.client.1.smithi067.stdout:5/464: unlink d0/d6/d33/l44 0 2024-04-12T05:37:19.852 INFO:tasks.workunit.client.0.smithi032.stdout:7/912: dread d0/dd/ffc [0,4194304] 0 2024-04-12T05:37:19.852 INFO:tasks.workunit.client.0.smithi032.stdout:2/881: fsync dd/d19/d9d/d1c/d64/f98 0 2024-04-12T05:37:19.852 INFO:tasks.workunit.client.1.smithi067.stdout:7/290: rename d0/d23/d45/l3d to d0/d2/d5/d1d/d43/l5a 0 2024-04-12T05:37:19.854 INFO:tasks.workunit.client.1.smithi067.stdout:8/240: write d6/d8/d18/f41 [967522,128725] 0 2024-04-12T05:37:19.856 INFO:tasks.workunit.client.1.smithi067.stdout:2/303: rename d5/l4a to d5/d43/l5e 0 2024-04-12T05:37:19.858 INFO:tasks.workunit.client.1.smithi067.stdout:6/305: dwrite fa [0,4194304] 0 2024-04-12T05:37:19.859 INFO:tasks.workunit.client.1.smithi067.stdout:2/304: write d5/d11/f29 [4099849,97463] 0 2024-04-12T05:37:19.860 INFO:tasks.workunit.client.1.smithi067.stdout:7/291: mknod d0/d2/dd/c5b 0 2024-04-12T05:37:19.862 INFO:tasks.workunit.client.0.smithi032.stdout:2/882: rmdir dd/d19/d9d/d42/da7/d115 0 2024-04-12T05:37:19.863 INFO:tasks.workunit.client.0.smithi032.stdout:3/979: mknod d12/d2f/d89/df0/d3a/c144 0 2024-04-12T05:37:19.864 INFO:tasks.workunit.client.1.smithi067.stdout:6/306: mkdir dd/d21/d23/d67 0 2024-04-12T05:37:19.864 INFO:tasks.workunit.client.0.smithi032.stdout:2/883: creat dd/d19/d82/f119 x:0 0 0 2024-04-12T05:37:19.866 INFO:tasks.workunit.client.0.smithi032.stdout:2/884: chown dd/d19/d9d/d1c/d64/d36/f85 2021950632 1 2024-04-12T05:37:19.867 INFO:tasks.workunit.client.1.smithi067.stdout:6/307: stat dd/d21/d23/f43 0 2024-04-12T05:37:19.868 INFO:tasks.workunit.client.1.smithi067.stdout:6/308: chown dd/d30/l3a 1320 1 2024-04-12T05:37:19.869 INFO:tasks.workunit.client.1.smithi067.stdout:5/465: getdents d0/d6/d40/d4b 0 2024-04-12T05:37:19.870 INFO:tasks.workunit.client.1.smithi067.stdout:7/292: dwrite d0/f8 [0,4194304] 0 2024-04-12T05:37:19.870 INFO:tasks.workunit.client.0.smithi032.stdout:2/885: dread - dd/d19/d9d/d42/d4a/da1/fbc zero size 2024-04-12T05:37:19.872 INFO:tasks.workunit.client.1.smithi067.stdout:6/309: truncate dd/d21/d23/f65 152530 0 2024-04-12T05:37:19.872 INFO:tasks.workunit.client.1.smithi067.stdout:2/305: creat d5/da/de/d14/d34/d4c/f5f x:0 0 0 2024-04-12T05:37:19.875 INFO:tasks.workunit.client.1.smithi067.stdout:7/293: mkdir d0/d2/d5/dc/d5c 0 2024-04-12T05:37:19.875 INFO:tasks.workunit.client.1.smithi067.stdout:2/306: chown d5/da/de/d14/l4b 124614 1 2024-04-12T05:37:19.875 INFO:tasks.workunit.client.0.smithi032.stdout:3/980: dread d12/fe6 [0,4194304] 0 2024-04-12T05:37:19.880 INFO:tasks.workunit.client.1.smithi067.stdout:6/310: rename dd/d11/f34 to dd/d21/d23/d67/f68 0 2024-04-12T05:37:19.880 INFO:tasks.workunit.client.0.smithi032.stdout:1/788: sync 2024-04-12T05:37:19.880 INFO:tasks.workunit.client.0.smithi032.stdout:0/921: sync 2024-04-12T05:37:19.883 INFO:tasks.workunit.client.1.smithi067.stdout:7/294: chown d0/d2/d5/db/l3c 0 1 2024-04-12T05:37:19.883 INFO:tasks.workunit.client.1.smithi067.stdout:6/311: fdatasync dd/d21/d23/f43 0 2024-04-12T05:37:19.884 INFO:tasks.workunit.client.0.smithi032.stdout:1/789: creat d5/de/d17/d35/d3b/dda/ff6 x:0 0 0 2024-04-12T05:37:19.885 INFO:tasks.workunit.client.0.smithi032.stdout:0/922: fsync d2/d8/d17/d1b/d20/f45 0 2024-04-12T05:37:19.886 INFO:tasks.workunit.client.1.smithi067.stdout:6/312: fsync f0 0 2024-04-12T05:37:19.886 INFO:tasks.workunit.client.1.smithi067.stdout:8/241: dread d6/d8/d18/f25 [0,4194304] 0 2024-04-12T05:37:19.886 INFO:tasks.workunit.client.1.smithi067.stdout:3/231: dwrite dc/df/f2c [0,4194304] 0 2024-04-12T05:37:19.887 INFO:tasks.workunit.client.0.smithi032.stdout:1/790: creat d5/de/d17/d35/d61/d77/d8f/ff7 x:0 0 0 2024-04-12T05:37:19.887 INFO:tasks.workunit.client.1.smithi067.stdout:7/295: chown d0/d2/d5/d1d/d43 614655 1 2024-04-12T05:37:19.888 INFO:tasks.workunit.client.0.smithi032.stdout:3/981: dread d12/d2f/f81 [0,4194304] 0 2024-04-12T05:37:19.889 INFO:tasks.workunit.client.0.smithi032.stdout:1/791: write d5/de/d17/d35/d78/f83 [868253,50704] 0 2024-04-12T05:37:19.889 INFO:tasks.workunit.client.0.smithi032.stdout:0/923: creat d2/d8/d119/f135 x:0 0 0 2024-04-12T05:37:19.892 INFO:tasks.workunit.client.1.smithi067.stdout:5/466: truncate d0/d3a/d2c/f3d 224262 0 2024-04-12T05:37:19.892 INFO:tasks.workunit.client.0.smithi032.stdout:1/792: dread - d5/de/d17/d35/d3b/dda/ff6 zero size 2024-04-12T05:37:19.893 INFO:tasks.workunit.client.1.smithi067.stdout:2/307: dwrite d5/da/de/f52 [0,4194304] 0 2024-04-12T05:37:19.893 INFO:tasks.workunit.client.1.smithi067.stdout:7/296: symlink d0/d2/d5/dc/l5d 0 2024-04-12T05:37:19.895 INFO:tasks.workunit.client.1.smithi067.stdout:2/308: readlink d5/l21 0 2024-04-12T05:37:19.897 INFO:tasks.workunit.client.1.smithi067.stdout:5/467: mknod d0/ca0 0 2024-04-12T05:37:19.900 INFO:tasks.workunit.client.1.smithi067.stdout:6/313: creat dd/d21/d23/d56/f69 x:0 0 0 2024-04-12T05:37:19.904 INFO:tasks.workunit.client.1.smithi067.stdout:5/468: creat d0/d6/d40/d4b/d38/d8e/d8f/d99/fa1 x:0 0 0 2024-04-12T05:37:19.907 INFO:tasks.workunit.client.0.smithi032.stdout:1/793: dwrite d5/de/d2a/fb6 [0,4194304] 0 2024-04-12T05:37:19.909 INFO:tasks.workunit.client.0.smithi032.stdout:3/982: getdents d12/d2f/d89/df0/d3a/d63/da6/dac/dc9/d125 0 2024-04-12T05:37:19.911 INFO:tasks.workunit.client.1.smithi067.stdout:5/469: dwrite d0/d6/d33/f3c [0,4194304] 0 2024-04-12T05:37:19.912 INFO:tasks.workunit.client.0.smithi032.stdout:2/886: dread dd/d19/d9d/d42/da7/dc5/fcc [0,4194304] 0 2024-04-12T05:37:19.916 INFO:tasks.workunit.client.1.smithi067.stdout:8/242: rename d6/l21 to d6/d8/d18/df/l4c 0 2024-04-12T05:37:19.918 INFO:tasks.workunit.client.0.smithi032.stdout:2/887: creat dd/d19/d9d/d42/da7/dc5/f11a x:0 0 0 2024-04-12T05:37:19.918 INFO:tasks.workunit.client.1.smithi067.stdout:5/470: unlink d0/d6/d20/d62/l2f 0 2024-04-12T05:37:19.922 INFO:tasks.workunit.client.1.smithi067.stdout:2/309: rename d5/da/de/d14/d34/d4c/f4d to d5/da/de/d14/d34/d40/d57/f60 0 2024-04-12T05:37:19.923 INFO:tasks.workunit.client.1.smithi067.stdout:6/314: write dd/d21/d23/f36 [3027172,67300] 0 2024-04-12T05:37:19.923 INFO:tasks.workunit.client.1.smithi067.stdout:7/297: dwrite d0/d23/f41 [0,4194304] 0 2024-04-12T05:37:19.924 INFO:tasks.workunit.client.1.smithi067.stdout:6/315: chown dd/f17 496 1 2024-04-12T05:37:19.925 INFO:tasks.workunit.client.1.smithi067.stdout:6/316: chown dd/d21/l4e 34 1 2024-04-12T05:37:19.926 INFO:tasks.workunit.client.1.smithi067.stdout:5/471: creat d0/da/d3f/d82/fa2 x:0 0 0 2024-04-12T05:37:19.929 INFO:tasks.workunit.client.1.smithi067.stdout:2/310: stat d5/da/c30 0 2024-04-12T05:37:19.935 INFO:tasks.workunit.client.1.smithi067.stdout:5/472: unlink d0/d3a/l9 0 2024-04-12T05:37:19.936 INFO:tasks.workunit.client.1.smithi067.stdout:8/243: write d6/d8/d18/df/f23 [1375681,69813] 0 2024-04-12T05:37:19.937 INFO:tasks.workunit.client.0.smithi032.stdout:3/983: truncate d12/d2f/d89/d9d/fb6 2713941 0 2024-04-12T05:37:19.937 INFO:tasks.workunit.client.1.smithi067.stdout:8/244: chown d6/d8/d18/d38 934 1 2024-04-12T05:37:19.940 INFO:tasks.workunit.client.1.smithi067.stdout:5/473: creat d0/d6/d20/d3e/fa3 x:0 0 0 2024-04-12T05:37:19.940 INFO:tasks.workunit.client.0.smithi032.stdout:3/984: mkdir d12/d24/dde/d145 0 2024-04-12T05:37:19.941 INFO:tasks.workunit.client.1.smithi067.stdout:2/311: rmdir d5/da/d5b 0 2024-04-12T05:37:19.942 INFO:tasks.workunit.client.0.smithi032.stdout:3/985: creat d12/d2f/de2/f146 x:0 0 0 2024-04-12T05:37:19.943 INFO:tasks.workunit.client.0.smithi032.stdout:3/986: write d12/d2f/d89/df0/d3a/dbe/f127 [367868,11029] 0 2024-04-12T05:37:19.944 INFO:tasks.workunit.client.1.smithi067.stdout:2/312: write d5/d11/f2a [280591,67300] 0 2024-04-12T05:37:19.945 INFO:tasks.workunit.client.1.smithi067.stdout:8/245: rmdir d6/d8/d18/d38 0 2024-04-12T05:37:19.946 INFO:tasks.workunit.client.1.smithi067.stdout:7/298: dwrite d0/d2/d5/fe [0,4194304] 0 2024-04-12T05:37:19.946 INFO:tasks.workunit.client.0.smithi032.stdout:3/987: dread - d12/d2f/d89/df0/d3a/fae zero size 2024-04-12T05:37:19.947 INFO:tasks.workunit.client.1.smithi067.stdout:5/474: getdents d0/d6/d40/d4b 0 2024-04-12T05:37:19.947 INFO:tasks.workunit.client.1.smithi067.stdout:6/317: dread dd/d21/d23/f36 [0,4194304] 0 2024-04-12T05:37:19.947 INFO:tasks.workunit.client.1.smithi067.stdout:8/246: fdatasync d6/d8/d18/f33 0 2024-04-12T05:37:19.950 INFO:tasks.workunit.client.1.smithi067.stdout:8/247: stat d6/d8/d18 0 2024-04-12T05:37:19.950 INFO:tasks.workunit.client.1.smithi067.stdout:2/313: creat d5/da/de/d14/d34/f61 x:0 0 0 2024-04-12T05:37:19.953 INFO:tasks.workunit.client.1.smithi067.stdout:6/318: mknod dd/d21/d23/c6a 0 2024-04-12T05:37:19.953 INFO:tasks.workunit.client.1.smithi067.stdout:7/299: creat d0/f5e x:0 0 0 2024-04-12T05:37:19.954 INFO:tasks.workunit.client.1.smithi067.stdout:7/300: write d0/d2/d5/f11 [3402344,72235] 0 2024-04-12T05:37:19.955 INFO:tasks.workunit.client.1.smithi067.stdout:5/475: rename d0/d6/d33/l42 to d0/da/d3f/la4 0 2024-04-12T05:37:19.959 INFO:tasks.workunit.client.1.smithi067.stdout:8/248: truncate d6/d8/f45 1970282 0 2024-04-12T05:37:19.960 INFO:tasks.workunit.client.1.smithi067.stdout:6/319: creat dd/d11/f6b x:0 0 0 2024-04-12T05:37:19.961 INFO:tasks.workunit.client.1.smithi067.stdout:5/476: readlink d0/d6/d20/d62/l22 0 2024-04-12T05:37:19.964 INFO:tasks.workunit.client.1.smithi067.stdout:7/301: mknod d0/d2/d5/db/d34/c5f 0 2024-04-12T05:37:19.964 INFO:tasks.workunit.client.1.smithi067.stdout:2/314: fsync d5/da/de/d14/f24 0 2024-04-12T05:37:19.966 INFO:tasks.workunit.client.1.smithi067.stdout:7/302: chown d0/d2/d5/fe 10262 1 2024-04-12T05:37:19.966 INFO:tasks.workunit.client.1.smithi067.stdout:5/477: dread - d0/d6/d33/f72 zero size 2024-04-12T05:37:19.968 INFO:tasks.workunit.client.1.smithi067.stdout:8/249: read d6/d8/d18/f33 [933719,85469] 0 2024-04-12T05:37:19.969 INFO:tasks.workunit.client.1.smithi067.stdout:6/320: rename dd/d21/d23/l49 to dd/l6c 0 2024-04-12T05:37:19.973 INFO:tasks.workunit.client.1.smithi067.stdout:6/321: chown dd/d11 39916967 1 2024-04-12T05:37:19.973 INFO:tasks.workunit.client.1.smithi067.stdout:5/478: truncate d0/d6/d33/f72 843737 0 2024-04-12T05:37:19.973 INFO:tasks.workunit.client.1.smithi067.stdout:8/250: chown d6/d8/d18/df/l4b 356737 1 2024-04-12T05:37:19.974 INFO:tasks.workunit.client.1.smithi067.stdout:2/315: fdatasync d5/da/de/f31 0 2024-04-12T05:37:19.976 INFO:tasks.workunit.client.1.smithi067.stdout:8/251: fsync d6/d8/f16 0 2024-04-12T05:37:19.978 INFO:tasks.workunit.client.1.smithi067.stdout:5/479: creat d0/d6/d20/d3e/d69/fa5 x:0 0 0 2024-04-12T05:37:19.981 INFO:tasks.workunit.client.1.smithi067.stdout:5/480: chown d0/d6/d33/d9e 0 1 2024-04-12T05:37:19.984 INFO:tasks.workunit.client.1.smithi067.stdout:6/322: dwrite fb [0,4194304] 0 2024-04-12T05:37:19.985 INFO:tasks.workunit.client.1.smithi067.stdout:7/303: getdents d0/d2/d5/dc 0 2024-04-12T05:37:19.990 INFO:tasks.workunit.client.0.smithi032.stdout:9/726: dread d9/d15/d2c/f10 [0,4194304] 0 2024-04-12T05:37:19.992 INFO:tasks.workunit.client.1.smithi067.stdout:6/323: dread - dd/d21/d3f/f40 zero size 2024-04-12T05:37:19.993 INFO:tasks.workunit.client.0.smithi032.stdout:9/727: symlink d9/d15/d1a/d24/d27/dce/dd3/lf4 0 2024-04-12T05:37:19.993 INFO:tasks.workunit.client.1.smithi067.stdout:5/481: dread d0/d6/d33/f72 [0,4194304] 0 2024-04-12T05:37:19.994 INFO:tasks.workunit.client.1.smithi067.stdout:7/304: symlink d0/d2/d5/d1d/l60 0 2024-04-12T05:37:19.995 INFO:tasks.workunit.client.1.smithi067.stdout:2/316: dread d5/da/de/d14/d34/f37 [0,4194304] 0 2024-04-12T05:37:19.996 INFO:tasks.workunit.client.0.smithi032.stdout:9/728: readlink d9/d15/d1a/d24/l8a 0 2024-04-12T05:37:19.997 INFO:tasks.workunit.client.1.smithi067.stdout:8/252: dwrite d6/d8/d18/df/d12/f1d [0,4194304] 0 2024-04-12T05:37:19.997 INFO:tasks.workunit.client.1.smithi067.stdout:6/324: unlink dd/d11/d27/d4a/l4b 0 2024-04-12T05:37:19.998 INFO:tasks.workunit.client.1.smithi067.stdout:2/317: write d5/da/de/f52 [4844700,121724] 0 2024-04-12T05:37:19.998 INFO:tasks.workunit.client.1.smithi067.stdout:5/482: creat d0/da/d3f/d82/fa6 x:0 0 0 2024-04-12T05:37:20.000 INFO:tasks.workunit.client.1.smithi067.stdout:7/305: mkdir d0/d2/d5/d61 0 2024-04-12T05:37:20.001 INFO:tasks.workunit.client.1.smithi067.stdout:6/325: mknod dd/d21/d3f/c6d 0 2024-04-12T05:37:20.002 INFO:tasks.workunit.client.1.smithi067.stdout:2/318: mknod d5/da/de/d14/d34/d40/c62 0 2024-04-12T05:37:20.002 INFO:tasks.workunit.client.1.smithi067.stdout:5/483: symlink d0/d6/d33/d9e/la7 0 2024-04-12T05:37:20.005 INFO:tasks.workunit.client.1.smithi067.stdout:5/484: write d0/d6/d40/f7f [198595,15341] 0 2024-04-12T05:37:20.007 INFO:tasks.workunit.client.1.smithi067.stdout:8/253: dwrite d6/f2c [0,4194304] 0 2024-04-12T05:37:20.009 INFO:tasks.workunit.client.1.smithi067.stdout:5/485: mkdir d0/d3a/da8 0 2024-04-12T05:37:20.009 INFO:tasks.workunit.client.1.smithi067.stdout:2/319: dread d5/da/de/d14/f36 [0,4194304] 0 2024-04-12T05:37:20.009 INFO:tasks.workunit.client.1.smithi067.stdout:8/254: symlink d6/d8/d18/d36/l4d 0 2024-04-12T05:37:20.010 INFO:tasks.workunit.client.1.smithi067.stdout:2/320: read d5/da/f3e [896099,21948] 0 2024-04-12T05:37:20.011 INFO:tasks.workunit.client.1.smithi067.stdout:5/486: creat d0/da/d41/d7d/fa9 x:0 0 0 2024-04-12T05:37:20.012 INFO:tasks.workunit.client.1.smithi067.stdout:8/255: mkdir d6/d8/d18/d4e 0 2024-04-12T05:37:20.013 INFO:tasks.workunit.client.1.smithi067.stdout:2/321: unlink d5/da/de/f31 0 2024-04-12T05:37:20.014 INFO:tasks.workunit.client.1.smithi067.stdout:5/487: mkdir d0/da/d3f/daa 0 2024-04-12T05:37:20.015 INFO:tasks.workunit.client.1.smithi067.stdout:8/256: mkdir d6/d8/d18/d3c/d4f 0 2024-04-12T05:37:20.016 INFO:tasks.workunit.client.1.smithi067.stdout:2/322: rename f4 to d5/da/de/d14/f63 0 2024-04-12T05:37:20.018 INFO:tasks.workunit.client.1.smithi067.stdout:5/488: mkdir d0/d3a/dab 0 2024-04-12T05:37:20.019 INFO:tasks.workunit.client.1.smithi067.stdout:2/323: creat d5/da/de/d47/f64 x:0 0 0 2024-04-12T05:37:20.020 INFO:tasks.workunit.client.1.smithi067.stdout:5/489: unlink d0/d3a/d2c/c85 0 2024-04-12T05:37:20.020 INFO:tasks.workunit.client.1.smithi067.stdout:2/324: creat d5/da/de/d47/f65 x:0 0 0 2024-04-12T05:37:20.022 INFO:tasks.workunit.client.1.smithi067.stdout:5/490: creat d0/da/d3f/d82/fac x:0 0 0 2024-04-12T05:37:20.022 INFO:tasks.workunit.client.1.smithi067.stdout:5/491: dread - d0/d6/d20/f58 zero size 2024-04-12T05:37:20.028 INFO:tasks.workunit.client.1.smithi067.stdout:5/492: rename d0/d3a/d2c/l64 to d0/d3a/lad 0 2024-04-12T05:37:20.028 INFO:tasks.workunit.client.1.smithi067.stdout:2/325: dwrite d5/da/de/d14/f63 [4194304,4194304] 0 2024-04-12T05:37:20.029 INFO:tasks.workunit.client.1.smithi067.stdout:5/493: chown d0/d6/f2b 2476466 1 2024-04-12T05:37:20.031 INFO:tasks.workunit.client.1.smithi067.stdout:5/494: mkdir d0/d6/dae 0 2024-04-12T05:37:20.036 INFO:tasks.workunit.client.1.smithi067.stdout:5/495: truncate d0/d6/f10 797348 0 2024-04-12T05:37:20.037 INFO:tasks.workunit.client.1.smithi067.stdout:2/326: dwrite d5/da/de/d14/d34/d4c/f5d [0,4194304] 0 2024-04-12T05:37:20.038 INFO:tasks.workunit.client.1.smithi067.stdout:2/327: fdatasync d5/d11/f29 0 2024-04-12T05:37:20.039 INFO:tasks.workunit.client.1.smithi067.stdout:5/496: write d0/d6/f6e [945530,46165] 0 2024-04-12T05:37:20.042 INFO:tasks.workunit.client.0.smithi032.stdout:2/888: rmdir dd/d19/d9d/d1c 39 2024-04-12T05:37:20.052 INFO:tasks.workunit.client.0.smithi032.stdout:2/889: creat dd/d19/d9d/d42/da7/f11b x:0 0 0 2024-04-12T05:37:20.054 INFO:tasks.workunit.client.1.smithi067.stdout:5/497: link d0/d6/d33/d9e/la7 d0/d6/d20/d3e/laf 0 2024-04-12T05:37:20.059 INFO:tasks.workunit.client.0.smithi032.stdout:9/729: dwrite d9/d15/d1a/dd2/fe3 [0,4194304] 0 2024-04-12T05:37:20.061 INFO:tasks.workunit.client.1.smithi067.stdout:5/498: rename d0/d6/d40/d4b/c3b to d0/d6/dae/cb0 0 2024-04-12T05:37:20.063 INFO:tasks.workunit.client.1.smithi067.stdout:2/328: dwrite d5/da/de/d47/f65 [0,4194304] 0 2024-04-12T05:37:20.066 INFO:tasks.workunit.client.0.smithi032.stdout:8/994: dwrite d0/db/dd/d11/d1b/d22/d54/d136/db3/fc4 [0,4194304] 0 2024-04-12T05:37:20.070 INFO:tasks.workunit.client.0.smithi032.stdout:9/730: creat d9/d32/ff5 x:0 0 0 2024-04-12T05:37:20.073 INFO:tasks.workunit.client.1.smithi067.stdout:5/499: rename d0/d3a/f19 to d0/d6/d40/d4b/fb1 0 2024-04-12T05:37:20.074 INFO:tasks.workunit.client.0.smithi032.stdout:2/890: rename dd/d19/d9d/d1c/d64/d36/fb4 to dd/d19/d9d/d1c/d64/d92/dfa/f11c 0 2024-04-12T05:37:20.078 INFO:tasks.workunit.client.1.smithi067.stdout:5/500: symlink d0/d6/d20/d3e/lb2 0 2024-04-12T05:37:20.079 INFO:tasks.workunit.client.0.smithi032.stdout:4/903: write d4/d87/d96/d2f/d3b/f9e [771208,22224] 0 2024-04-12T05:37:20.081 INFO:tasks.workunit.client.1.smithi067.stdout:2/329: dread d5/da/de/d47/f65 [0,4194304] 0 2024-04-12T05:37:20.082 INFO:tasks.workunit.client.0.smithi032.stdout:7/913: write d0/de/f44 [7733475,126867] 0 2024-04-12T05:37:20.082 INFO:tasks.workunit.client.0.smithi032.stdout:4/904: chown d4/d87/d96/d2f/d94/f127 3360937 1 2024-04-12T05:37:20.082 INFO:tasks.workunit.client.0.smithi032.stdout:7/914: readlink d0/dd/d51/d65/d12a/l9b 0 2024-04-12T05:37:20.083 INFO:tasks.workunit.client.1.smithi067.stdout:6/326: dread dd/f3b [0,4194304] 0 2024-04-12T05:37:20.083 INFO:tasks.workunit.client.1.smithi067.stdout:5/501: mkdir d0/d3a/d2c/db3 0 2024-04-12T05:37:20.084 INFO:tasks.workunit.client.0.smithi032.stdout:7/915: chown d0/d13/f8c 0 1 2024-04-12T05:37:20.084 INFO:tasks.workunit.client.1.smithi067.stdout:2/330: fdatasync d5/d11/f2e 0 2024-04-12T05:37:20.085 INFO:tasks.workunit.client.0.smithi032.stdout:2/891: creat dd/d19/d9d/d1c/d64/d92/d10c/f11d x:0 0 0 2024-04-12T05:37:20.087 INFO:tasks.workunit.client.1.smithi067.stdout:5/502: read d0/d6/d40/d4b/d38/f79 [335927,71283] 0 2024-04-12T05:37:20.087 INFO:tasks.workunit.client.0.smithi032.stdout:4/905: symlink d4/d87/d96/d32/l12d 0 2024-04-12T05:37:20.087 INFO:tasks.workunit.client.1.smithi067.stdout:6/327: read dd/d21/d23/f25 [110890,65959] 0 2024-04-12T05:37:20.089 INFO:tasks.workunit.client.1.smithi067.stdout:5/503: fdatasync d0/d6/f6e 0 2024-04-12T05:37:20.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:19 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:37:20.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:19 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:37:20.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:19 smithi032 ceph-mon[39731]: pgmap v271: 65 pgs: 65 active+clean; 2.1 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 59 MiB/s rd, 152 MiB/s wr, 321 op/s 2024-04-12T05:37:20.091 INFO:tasks.workunit.client.0.smithi032.stdout:4/906: creat d4/d87/d96/d2f/d6e/f12e x:0 0 0 2024-04-12T05:37:20.091 INFO:tasks.workunit.client.1.smithi067.stdout:2/331: mknod d5/d23/c66 0 2024-04-12T05:37:20.093 INFO:tasks.workunit.client.0.smithi032.stdout:0/924: write d2/d28/d87/d111/d3e/dac/f3f [2672205,112532] 0 2024-04-12T05:37:20.093 INFO:tasks.workunit.client.1.smithi067.stdout:2/332: read - d5/da/de/d14/d34/f61 zero size 2024-04-12T05:37:20.094 INFO:tasks.workunit.client.0.smithi032.stdout:0/925: chown d2/d29/ld4 439425 1 2024-04-12T05:37:20.097 INFO:tasks.workunit.client.0.smithi032.stdout:4/907: link d4/cad d4/d87/d96/d68/d9b/d11c/c12f 0 2024-04-12T05:37:20.098 INFO:tasks.workunit.client.1.smithi067.stdout:5/504: creat d0/da/d3f/d82/fb4 x:0 0 0 2024-04-12T05:37:20.099 INFO:tasks.workunit.client.0.smithi032.stdout:0/926: symlink d2/d8/d17/d1b/l136 0 2024-04-12T05:37:20.099 INFO:tasks.workunit.client.1.smithi067.stdout:2/333: creat d5/da/de/d47/f67 x:0 0 0 2024-04-12T05:37:20.100 INFO:tasks.workunit.client.0.smithi032.stdout:2/892: dread dd/d19/d9d/d1c/f76 [0,4194304] 0 2024-04-12T05:37:20.100 INFO:tasks.workunit.client.1.smithi067.stdout:6/328: getdents dd/d11/d27/d4a/d51 0 2024-04-12T05:37:20.100 INFO:tasks.workunit.client.0.smithi032.stdout:0/927: readlink d2/d8/df/d24/d60/dc8/lff 0 2024-04-12T05:37:20.101 INFO:tasks.workunit.client.0.smithi032.stdout:9/731: write d9/d15/d1a/d1d/fb9 [220443,81580] 0 2024-04-12T05:37:20.102 INFO:tasks.workunit.client.1.smithi067.stdout:2/334: write d5/da/f1d [2256941,129926] 0 2024-04-12T05:37:20.104 INFO:tasks.workunit.client.0.smithi032.stdout:2/893: fdatasync dd/d19/f5f 0 2024-04-12T05:37:20.104 INFO:tasks.workunit.client.0.smithi032.stdout:4/908: mknod d4/d87/d96/d2f/c130 0 2024-04-12T05:37:20.106 INFO:tasks.workunit.client.1.smithi067.stdout:6/329: creat dd/d21/d63/f6e x:0 0 0 2024-04-12T05:37:20.110 INFO:tasks.workunit.client.0.smithi032.stdout:2/894: rmdir dd/d19/d9d/d1c/d64/d92 39 2024-04-12T05:37:20.112 INFO:tasks.workunit.client.1.smithi067.stdout:5/505: getdents d0/d6/d33/d9e 0 2024-04-12T05:37:20.112 INFO:tasks.workunit.client.1.smithi067.stdout:2/335: link c0 d5/da/d3d/d55/c68 0 2024-04-12T05:37:20.113 INFO:tasks.workunit.client.1.smithi067.stdout:6/330: mkdir dd/d21/d64/d6f 0 2024-04-12T05:37:20.114 INFO:tasks.workunit.client.0.smithi032.stdout:2/895: truncate dd/d19/d9d/f73 1609701 0 2024-04-12T05:37:20.114 INFO:tasks.workunit.client.0.smithi032.stdout:2/896: readlink dd/d19/la4 0 2024-04-12T05:37:20.116 INFO:tasks.workunit.client.0.smithi032.stdout:0/928: write d2/d8/d17/f6a [426828,114657] 0 2024-04-12T05:37:20.118 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:19 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:37:20.118 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:19 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:37:20.118 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:19 smithi067 ceph-mon[49010]: pgmap v271: 65 pgs: 65 active+clean; 2.1 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 59 MiB/s rd, 152 MiB/s wr, 321 op/s 2024-04-12T05:37:20.118 INFO:tasks.workunit.client.1.smithi067.stdout:6/331: unlink fc 0 2024-04-12T05:37:20.121 INFO:tasks.workunit.client.0.smithi032.stdout:2/897: creat dd/d19/f11e x:0 0 0 2024-04-12T05:37:20.122 INFO:tasks.workunit.client.1.smithi067.stdout:2/336: dwrite d5/da/de/d14/f36 [8388608,4194304] 0 2024-04-12T05:37:20.122 INFO:tasks.workunit.client.0.smithi032.stdout:0/929: truncate d2/d29/d8a/d99/fba 2704527 0 2024-04-12T05:37:20.124 INFO:tasks.workunit.client.0.smithi032.stdout:9/732: dwrite d9/d15/d2c/f20 [0,4194304] 0 2024-04-12T05:37:20.125 INFO:tasks.workunit.client.0.smithi032.stdout:4/909: dread d4/f3d [0,4194304] 0 2024-04-12T05:37:20.127 INFO:tasks.workunit.client.0.smithi032.stdout:9/733: stat d9/d15/d1a/dd2/d65/cec 0 2024-04-12T05:37:20.132 INFO:tasks.workunit.client.1.smithi067.stdout:5/506: getdents d0 0 2024-04-12T05:37:20.134 INFO:tasks.workunit.client.0.smithi032.stdout:2/898: write fa [1062001,104788] 0 2024-04-12T05:37:20.135 INFO:tasks.workunit.client.1.smithi067.stdout:5/507: read d0/d3a/f7e [1553214,88986] 0 2024-04-12T05:37:20.135 INFO:tasks.workunit.client.0.smithi032.stdout:0/930: rename d2/c1f to d2/d29/d77/dc7/d112/c137 0 2024-04-12T05:37:20.137 INFO:tasks.workunit.client.0.smithi032.stdout:4/910: unlink d4/d13/c93 0 2024-04-12T05:37:20.137 INFO:tasks.workunit.client.1.smithi067.stdout:6/332: dwrite dd/d21/d23/f36 [0,4194304] 0 2024-04-12T05:37:20.139 INFO:tasks.workunit.client.1.smithi067.stdout:4/216: sync 2024-04-12T05:37:20.139 INFO:tasks.workunit.client.1.smithi067.stdout:0/323: sync 2024-04-12T05:37:20.141 INFO:tasks.workunit.client.0.smithi032.stdout:2/899: creat dd/d19/d9d/d1c/de0/f11f x:0 0 0 2024-04-12T05:37:20.143 INFO:tasks.workunit.client.1.smithi067.stdout:8/257: dwrite d6/d8/f45 [0,4194304] 0 2024-04-12T05:37:20.143 INFO:tasks.workunit.client.1.smithi067.stdout:2/337: creat d5/da/de/f69 x:0 0 0 2024-04-12T05:37:20.144 INFO:tasks.workunit.client.1.smithi067.stdout:4/217: dread - d8/da/d1d/f44 zero size 2024-04-12T05:37:20.146 INFO:tasks.workunit.client.1.smithi067.stdout:0/324: rmdir dd/d40 39 2024-04-12T05:37:20.149 INFO:tasks.workunit.client.0.smithi032.stdout:0/931: dwrite d2/f12a [0,4194304] 0 2024-04-12T05:37:20.150 INFO:tasks.workunit.client.1.smithi067.stdout:6/333: creat dd/d21/d64/d6f/f70 x:0 0 0 2024-04-12T05:37:20.152 INFO:tasks.workunit.client.1.smithi067.stdout:8/258: mkdir d6/d8/d50 0 2024-04-12T05:37:20.154 INFO:tasks.workunit.client.0.smithi032.stdout:2/900: unlink dd/d19/d9d/d1c/de0/f116 0 2024-04-12T05:37:20.154 INFO:tasks.workunit.client.1.smithi067.stdout:4/218: chown d8/d9/c29 4337326 1 2024-04-12T05:37:20.155 INFO:tasks.workunit.client.0.smithi032.stdout:2/901: dread - dd/d19/f107 zero size 2024-04-12T05:37:20.155 INFO:tasks.workunit.client.0.smithi032.stdout:0/932: stat d2/d29/f4a 0 2024-04-12T05:37:20.157 INFO:tasks.workunit.client.1.smithi067.stdout:4/219: dread d8/f33 [0,4194304] 0 2024-04-12T05:37:20.158 INFO:tasks.workunit.client.0.smithi032.stdout:9/734: dread d9/d15/d2c/d41/f7e [0,4194304] 0 2024-04-12T05:37:20.161 INFO:tasks.workunit.client.1.smithi067.stdout:2/338: write d5/da/de/d47/f65 [3035841,54465] 0 2024-04-12T05:37:20.162 INFO:tasks.workunit.client.1.smithi067.stdout:2/339: readlink d5/l21 0 2024-04-12T05:37:20.164 INFO:tasks.workunit.client.0.smithi032.stdout:2/902: creat dd/d114/f120 x:0 0 0 2024-04-12T05:37:20.164 INFO:tasks.workunit.client.0.smithi032.stdout:4/911: write d4/fb7 [1059399,20428] 0 2024-04-12T05:37:20.167 INFO:tasks.workunit.client.1.smithi067.stdout:5/508: truncate d0/d6/f12 3130235 0 2024-04-12T05:37:20.167 INFO:tasks.workunit.client.0.smithi032.stdout:2/903: chown dd/d19/l26 59 1 2024-04-12T05:37:20.167 INFO:tasks.workunit.client.0.smithi032.stdout:0/933: creat d2/d28/f138 x:0 0 0 2024-04-12T05:37:20.169 INFO:tasks.workunit.client.0.smithi032.stdout:4/912: fsync d4/d13/ff 0 2024-04-12T05:37:20.169 INFO:tasks.workunit.client.1.smithi067.stdout:0/325: rename dd/d12/d14/d1a/d3d/d41/d5f/c60 to dd/d12/d14/d1a/d3d/c69 0 2024-04-12T05:37:20.170 INFO:tasks.workunit.client.0.smithi032.stdout:2/904: rmdir dd/d19/d9d/d1c 39 2024-04-12T05:37:20.170 INFO:tasks.workunit.client.0.smithi032.stdout:0/934: dread d2/d29/d8a/fd7 [0,4194304] 0 2024-04-12T05:37:20.172 INFO:tasks.workunit.client.1.smithi067.stdout:2/340: dread d5/da/de/d14/d34/f37 [0,4194304] 0 2024-04-12T05:37:20.172 INFO:tasks.workunit.client.1.smithi067.stdout:0/326: chown dd/d12/l56 41032 1 2024-04-12T05:37:20.172 INFO:tasks.workunit.client.0.smithi032.stdout:4/913: write d4/d13/d19/f30 [507837,19834] 0 2024-04-12T05:37:20.174 INFO:tasks.workunit.client.1.smithi067.stdout:6/334: rename dd/d11/l3d to dd/d21/d3f/l71 0 2024-04-12T05:37:20.176 INFO:tasks.workunit.client.0.smithi032.stdout:0/935: symlink d2/d8/df/d24/d60/dc8/l139 0 2024-04-12T05:37:20.176 INFO:tasks.workunit.client.0.smithi032.stdout:4/914: dread - d4/d87/d96/d68/fd7 zero size 2024-04-12T05:37:20.177 INFO:tasks.workunit.client.1.smithi067.stdout:0/327: write dd/d40/f4c [136722,58307] 0 2024-04-12T05:37:20.177 INFO:tasks.workunit.client.0.smithi032.stdout:0/936: fsync d2/d28/d87/d79/f106 0 2024-04-12T05:37:20.178 INFO:tasks.workunit.client.1.smithi067.stdout:5/509: dread d0/d3a/f4c [0,4194304] 0 2024-04-12T05:37:20.178 INFO:tasks.workunit.client.1.smithi067.stdout:6/335: mknod dd/d21/d64/d6f/c72 0 2024-04-12T05:37:20.179 INFO:tasks.workunit.client.0.smithi032.stdout:2/905: mknod dd/d19/d9d/d1c/d64/d32/d6d/c121 0 2024-04-12T05:37:20.180 INFO:tasks.workunit.client.1.smithi067.stdout:8/259: dread d6/d8/d18/df/f20 [0,4194304] 0 2024-04-12T05:37:20.182 INFO:tasks.workunit.client.1.smithi067.stdout:6/336: truncate dd/d11/d27/f54 507003 0 2024-04-12T05:37:20.184 INFO:tasks.workunit.client.1.smithi067.stdout:2/341: rename d5/da/de/d47 to d5/d6a 0 2024-04-12T05:37:20.184 INFO:tasks.workunit.client.1.smithi067.stdout:0/328: truncate dd/d12/f2a 2803172 0 2024-04-12T05:37:20.185 INFO:tasks.workunit.client.1.smithi067.stdout:8/260: write d6/f2c [164608,128880] 0 2024-04-12T05:37:20.185 INFO:tasks.workunit.client.1.smithi067.stdout:6/337: fdatasync dd/d11/f1e 0 2024-04-12T05:37:20.187 INFO:tasks.workunit.client.1.smithi067.stdout:2/342: chown d5/d11 1 1 2024-04-12T05:37:20.190 INFO:tasks.workunit.client.1.smithi067.stdout:2/343: stat d5/d11/l15 0 2024-04-12T05:37:20.191 INFO:tasks.workunit.client.1.smithi067.stdout:8/261: truncate d6/f48 519542 0 2024-04-12T05:37:20.192 INFO:tasks.workunit.client.1.smithi067.stdout:4/220: dwrite d8/d9/d3d/d48/d45/f47 [0,4194304] 0 2024-04-12T05:37:20.193 INFO:tasks.workunit.client.0.smithi032.stdout:0/937: dwrite d2/f12a [0,4194304] 0 2024-04-12T05:37:20.195 INFO:tasks.workunit.client.1.smithi067.stdout:8/262: readlink d6/d8/d18/df/d12/l37 0 2024-04-12T05:37:20.198 INFO:tasks.workunit.client.0.smithi032.stdout:9/735: dwrite d9/f34 [4194304,4194304] 0 2024-04-12T05:37:20.202 INFO:tasks.workunit.client.0.smithi032.stdout:4/915: dwrite d4/d87/d96/d2f/d6e/fb6 [0,4194304] 0 2024-04-12T05:37:20.205 INFO:tasks.workunit.client.1.smithi067.stdout:0/329: dwrite dd/d12/f3a [0,4194304] 0 2024-04-12T05:37:20.209 INFO:tasks.workunit.client.0.smithi032.stdout:0/938: symlink d2/d28/d87/d111/d3e/dac/l13a 0 2024-04-12T05:37:20.211 INFO:tasks.workunit.client.0.smithi032.stdout:9/736: write d9/d15/d1a/d24/d27/d88/fa8 [295173,88865] 0 2024-04-12T05:37:20.217 INFO:tasks.workunit.client.1.smithi067.stdout:5/510: dread d0/f43 [0,4194304] 0 2024-04-12T05:37:20.217 INFO:tasks.workunit.client.1.smithi067.stdout:4/221: mknod d8/d9/d3d/d48/c4c 0 2024-04-12T05:37:20.220 INFO:tasks.workunit.client.0.smithi032.stdout:0/939: rmdir d2/d28/d87/d111/d3e/dac/d51/d83 39 2024-04-12T05:37:20.220 INFO:tasks.workunit.client.1.smithi067.stdout:6/338: truncate dd/f17 3681815 0 2024-04-12T05:37:20.221 INFO:tasks.workunit.client.1.smithi067.stdout:6/339: read - dd/d11/f6b zero size 2024-04-12T05:37:20.222 INFO:tasks.workunit.client.1.smithi067.stdout:2/344: dwrite d5/da/de/d14/d34/f61 [0,4194304] 0 2024-04-12T05:37:20.224 INFO:tasks.workunit.client.0.smithi032.stdout:9/737: mkdir d9/d15/d1a/d24/df6 0 2024-04-12T05:37:20.228 INFO:tasks.workunit.client.0.smithi032.stdout:1/794: sync 2024-04-12T05:37:20.228 INFO:tasks.workunit.client.0.smithi032.stdout:3/988: sync 2024-04-12T05:37:20.228 INFO:tasks.workunit.client.1.smithi067.stdout:8/263: rename d6/d8/d18/d3c/c40 to d6/d8/d18/d36/c51 0 2024-04-12T05:37:20.229 INFO:tasks.workunit.client.0.smithi032.stdout:9/738: mkdir d9/df7 0 2024-04-12T05:37:20.229 INFO:tasks.workunit.client.1.smithi067.stdout:8/264: chown d6/f48 26 1 2024-04-12T05:37:20.231 INFO:tasks.workunit.client.1.smithi067.stdout:8/265: write d6/d8/d18/d36/f44 [294201,50925] 0 2024-04-12T05:37:20.231 INFO:tasks.workunit.client.1.smithi067.stdout:5/511: mknod d0/d3a/d2c/d91/cb5 0 2024-04-12T05:37:20.232 INFO:tasks.workunit.client.0.smithi032.stdout:3/989: truncate d12/d2f/d89/feb 664988 0 2024-04-12T05:37:20.232 INFO:tasks.workunit.client.1.smithi067.stdout:4/222: creat d8/d9/d3d/f4d x:0 0 0 2024-04-12T05:37:20.233 INFO:tasks.workunit.client.1.smithi067.stdout:2/345: unlink d5/da/de/d14/d34/d4c/f5d 0 2024-04-12T05:37:20.236 INFO:tasks.workunit.client.0.smithi032.stdout:3/990: rename d12/d2f/d89/df0/d3a/d63/da6/df5/f139 to d12/d2f/de2/f147 0 2024-04-12T05:37:20.237 INFO:tasks.workunit.client.0.smithi032.stdout:9/739: mknod d9/d15/d1a/d24/d27/dce/dd3/de9/cf8 0 2024-04-12T05:37:20.238 INFO:tasks.workunit.client.1.smithi067.stdout:8/266: write d6/f2c [2308677,118066] 0 2024-04-12T05:37:20.240 INFO:tasks.workunit.client.0.smithi032.stdout:1/795: creat d5/de/d17/d35/ff8 x:0 0 0 2024-04-12T05:37:20.241 INFO:tasks.workunit.client.0.smithi032.stdout:3/991: truncate d12/d2f/d89/df0/d3a/d63/da6/df5/f142 43568 0 2024-04-12T05:37:20.241 INFO:tasks.workunit.client.1.smithi067.stdout:6/340: truncate dd/d30/f62 692761 0 2024-04-12T05:37:20.241 INFO:tasks.workunit.client.1.smithi067.stdout:4/223: fdatasync d8/d9/f11 0 2024-04-12T05:37:20.241 INFO:tasks.workunit.client.1.smithi067.stdout:5/512: truncate d0/d6/d33/f72 121862 0 2024-04-12T05:37:20.244 INFO:tasks.workunit.client.0.smithi032.stdout:9/740: link f7 d9/d15/d1a/dd2/d65/d80/d87/ff9 0 2024-04-12T05:37:20.245 INFO:tasks.workunit.client.1.smithi067.stdout:2/346: link d5/da/de/f13 d5/da/de/d14/d34/d4c/f6b 0 2024-04-12T05:37:20.245 INFO:tasks.workunit.client.0.smithi032.stdout:1/796: symlink d5/de/d5b/lf9 0 2024-04-12T05:37:20.245 INFO:tasks.workunit.client.1.smithi067.stdout:4/224: write d8/f33 [992560,33812] 0 2024-04-12T05:37:20.246 INFO:tasks.workunit.client.1.smithi067.stdout:6/341: readlink dd/d11/d27/d4a/l59 0 2024-04-12T05:37:20.247 INFO:tasks.workunit.client.0.smithi032.stdout:9/741: symlink d9/d32/lfa 0 2024-04-12T05:37:20.247 INFO:tasks.workunit.client.1.smithi067.stdout:0/330: dwrite dd/d40/f67 [0,4194304] 0 2024-04-12T05:37:20.250 INFO:tasks.workunit.client.1.smithi067.stdout:9/264: sync 2024-04-12T05:37:20.250 INFO:tasks.workunit.client.1.smithi067.stdout:1/257: sync 2024-04-12T05:37:20.250 INFO:tasks.workunit.client.0.smithi032.stdout:9/742: chown d9/d15/d2c/f20 0 1 2024-04-12T05:37:20.250 INFO:tasks.workunit.client.0.smithi032.stdout:1/797: chown d5/de/d17/d35/d78/f79 7668342 1 2024-04-12T05:37:20.251 INFO:tasks.workunit.client.0.smithi032.stdout:3/992: dwrite d12/d2f/d89/df0/f1e [0,4194304] 0 2024-04-12T05:37:20.254 INFO:tasks.workunit.client.0.smithi032.stdout:9/743: write d9/d32/d86/ff3 [199743,62670] 0 2024-04-12T05:37:20.255 INFO:tasks.workunit.client.0.smithi032.stdout:3/993: stat d12/daa/db3/f87 0 2024-04-12T05:37:20.257 INFO:tasks.workunit.client.0.smithi032.stdout:1/798: rename d5/de/d17/cc3 to d5/de/d2a/cfa 0 2024-04-12T05:37:20.259 INFO:tasks.workunit.client.1.smithi067.stdout:8/267: dwrite d6/f2c [0,4194304] 0 2024-04-12T05:37:20.259 INFO:tasks.workunit.client.1.smithi067.stdout:1/258: chown d4/f8 7288 1 2024-04-12T05:37:20.259 INFO:tasks.workunit.client.1.smithi067.stdout:6/342: truncate dd/d30/f32 1525080 0 2024-04-12T05:37:20.259 INFO:tasks.workunit.client.1.smithi067.stdout:0/331: rmdir dd 39 2024-04-12T05:37:20.259 INFO:tasks.workunit.client.1.smithi067.stdout:4/225: creat d8/d9/d3d/f4e x:0 0 0 2024-04-12T05:37:20.259 INFO:tasks.workunit.client.1.smithi067.stdout:1/259: chown d4/d9/d3a/l48 3578904 1 2024-04-12T05:37:20.259 INFO:tasks.workunit.client.1.smithi067.stdout:4/226: write d8/fe [444366,130157] 0 2024-04-12T05:37:20.263 INFO:tasks.workunit.client.1.smithi067.stdout:2/347: fsync d5/da/de/d14/d34/d40/d57/f60 0 2024-04-12T05:37:20.263 INFO:tasks.workunit.client.0.smithi032.stdout:0/940: truncate d2/d8/d17/f2b 3031324 0 2024-04-12T05:37:20.263 INFO:tasks.workunit.client.1.smithi067.stdout:5/513: getdents d0/d6/d20 0 2024-04-12T05:37:20.268 INFO:tasks.workunit.client.0.smithi032.stdout:1/799: rename d5/de/d17/d35/d73/d74/c9e to d5/de/d17/d35/d3e/db4/cfb 0 2024-04-12T05:37:20.269 INFO:tasks.workunit.client.0.smithi032.stdout:3/994: creat d12/d2f/d89/df0/d3a/f148 x:0 0 0 2024-04-12T05:37:20.269 INFO:tasks.workunit.client.1.smithi067.stdout:9/265: dread d1/d12/f2e [0,4194304] 0 2024-04-12T05:37:20.271 INFO:tasks.workunit.client.1.smithi067.stdout:1/260: mkdir d4/d9/d3a/d51/d55 0 2024-04-12T05:37:20.272 INFO:tasks.workunit.client.1.smithi067.stdout:9/266: write d1/d28/d23/f3d [863539,125907] 0 2024-04-12T05:37:20.273 INFO:tasks.workunit.client.1.smithi067.stdout:8/268: symlink d6/l52 0 2024-04-12T05:37:20.274 INFO:tasks.workunit.client.0.smithi032.stdout:3/995: creat d12/daa/db3/f149 x:0 0 0 2024-04-12T05:37:20.274 INFO:tasks.workunit.client.0.smithi032.stdout:0/941: link d2/d29/d8a/d99/c123 d2/d28/d87/c13b 0 2024-04-12T05:37:20.279 INFO:tasks.workunit.client.0.smithi032.stdout:0/942: write d2/d28/f138 [127204,77182] 0 2024-04-12T05:37:20.279 INFO:tasks.workunit.client.1.smithi067.stdout:4/227: creat d8/da/d1d/d15/d36/f4f x:0 0 0 2024-04-12T05:37:20.280 INFO:tasks.workunit.client.0.smithi032.stdout:3/996: dread d12/d2f/d89/df0/d3a/d63/da6/dac/dc9/fe8 [0,4194304] 0 2024-04-12T05:37:20.280 INFO:tasks.workunit.client.1.smithi067.stdout:6/343: dwrite dd/f3b [0,4194304] 0 2024-04-12T05:37:20.281 INFO:tasks.workunit.client.0.smithi032.stdout:3/997: readlink d12/d2f/d89/ld6 0 2024-04-12T05:37:20.282 INFO:tasks.workunit.client.0.smithi032.stdout:1/800: getdents d5/de/d17/d35/d61/d77/d8f 0 2024-04-12T05:37:20.282 INFO:tasks.workunit.client.0.smithi032.stdout:1/801: readlink d5/de/l60 0 2024-04-12T05:37:20.284 INFO:tasks.workunit.client.0.smithi032.stdout:3/998: creat d12/daa/d12a/f14a x:0 0 0 2024-04-12T05:37:20.285 INFO:tasks.workunit.client.0.smithi032.stdout:0/943: symlink d2/d28/d87/d111/d3e/dac/l13c 0 2024-04-12T05:37:20.286 INFO:tasks.workunit.client.1.smithi067.stdout:2/348: readlink d5/da/de/d14/l49 0 2024-04-12T05:37:20.288 INFO:tasks.workunit.client.1.smithi067.stdout:6/344: fdatasync dd/d11/d27/f46 0 2024-04-12T05:37:20.291 INFO:tasks.workunit.client.0.smithi032.stdout:0/944: rmdir d2/d29/d77/db5 39 2024-04-12T05:37:20.292 INFO:tasks.workunit.client.0.smithi032.stdout:9/744: dwrite d9/d32/f74 [0,4194304] 0 2024-04-12T05:37:20.293 INFO:tasks.workunit.client.1.smithi067.stdout:1/261: dwrite d4/d3d/f4b [0,4194304] 0 2024-04-12T05:37:20.294 INFO:tasks.workunit.client.0.smithi032.stdout:9/745: truncate d9/d15/f43 6014207 0 2024-04-12T05:37:20.295 INFO:tasks.workunit.client.1.smithi067.stdout:4/228: rename d8/da/d1d/d15/d1c/c2e to d8/d39/c50 0 2024-04-12T05:37:20.296 INFO:tasks.workunit.client.0.smithi032.stdout:9/746: chown d9/d1f/cb3 1082101 1 2024-04-12T05:37:20.298 INFO:tasks.workunit.client.0.smithi032.stdout:1/802: getdents d5/de/d5b 0 2024-04-12T05:37:20.299 INFO:tasks.workunit.client.1.smithi067.stdout:2/349: creat d5/da/de/d39/f6c x:0 0 0 2024-04-12T05:37:20.299 INFO:tasks.workunit.client.1.smithi067.stdout:6/345: creat dd/d11/d27/d4a/f73 x:0 0 0 2024-04-12T05:37:20.300 INFO:tasks.workunit.client.0.smithi032.stdout:1/803: chown d5/de/d2a/c31 7954 1 2024-04-12T05:37:20.302 INFO:tasks.workunit.client.0.smithi032.stdout:1/804: write d5/de/d17/d35/d61/d77/d8f/ff7 [336917,10051] 0 2024-04-12T05:37:20.303 INFO:tasks.workunit.client.1.smithi067.stdout:0/332: dread dd/d12/d14/d1a/f1e [0,4194304] 0 2024-04-12T05:37:20.303 INFO:tasks.workunit.client.1.smithi067.stdout:9/267: link d1/d28/f2c d1/d28/f50 0 2024-04-12T05:37:20.306 INFO:tasks.workunit.client.0.smithi032.stdout:9/747: mknod d9/d15/d1a/dd2/d65/d80/d87/cfb 0 2024-04-12T05:37:20.307 INFO:tasks.workunit.client.1.smithi067.stdout:4/229: unlink d8/d9/f25 0 2024-04-12T05:37:20.312 INFO:tasks.workunit.client.1.smithi067.stdout:4/230: creat d8/d39/f51 x:0 0 0 2024-04-12T05:37:20.314 INFO:tasks.workunit.client.1.smithi067.stdout:0/333: mknod dd/d12/d14/c6a 0 2024-04-12T05:37:20.315 INFO:tasks.workunit.client.1.smithi067.stdout:5/514: dwrite d0/da/d41/f6b [0,4194304] 0 2024-04-12T05:37:20.316 INFO:tasks.workunit.client.1.smithi067.stdout:6/346: link dd/d21/d3f/c53 dd/d21/d66/c74 0 2024-04-12T05:37:20.318 INFO:tasks.workunit.client.1.smithi067.stdout:2/350: getdents d5/d23/d3a 0 2024-04-12T05:37:20.321 INFO:tasks.workunit.client.1.smithi067.stdout:9/268: getdents d1/d28/d27/d32 0 2024-04-12T05:37:20.324 INFO:tasks.workunit.client.1.smithi067.stdout:5/515: creat d0/da/d3f/daa/fb6 x:0 0 0 2024-04-12T05:37:20.324 INFO:tasks.workunit.client.1.smithi067.stdout:9/269: write d1/f47 [774821,97576] 0 2024-04-12T05:37:20.325 INFO:tasks.workunit.client.1.smithi067.stdout:6/347: fsync dd/d11/d27/d4a/f73 0 2024-04-12T05:37:20.328 INFO:tasks.workunit.client.0.smithi032.stdout:3/999: dread d12/d2f/f10f [0,4194304] 0 2024-04-12T05:37:20.328 INFO:tasks.workunit.client.1.smithi067.stdout:1/262: truncate d4/d9/d12/f2e 231865 0 2024-04-12T05:37:20.329 INFO:tasks.workunit.client.0.smithi032.stdout:1/805: dwrite d5/de/d17/d35/d3b/d49/f62 [0,4194304] 0 2024-04-12T05:37:20.331 INFO:tasks.workunit.client.0.smithi032.stdout:9/748: dwrite d9/d15/d2c/f77 [0,4194304] 0 2024-04-12T05:37:20.332 INFO:tasks.workunit.client.1.smithi067.stdout:2/351: rmdir d5/da/de/d14/d34/d40 39 2024-04-12T05:37:20.333 INFO:tasks.workunit.client.0.smithi032.stdout:1/806: creat d5/de/d17/d93/ffc x:0 0 0 2024-04-12T05:37:20.335 INFO:tasks.workunit.client.0.smithi032.stdout:9/749: mknod d9/d15/d1a/dd2/cfc 0 2024-04-12T05:37:20.339 INFO:tasks.workunit.client.1.smithi067.stdout:1/263: symlink d4/d9/d12/d28/l56 0 2024-04-12T05:37:20.340 INFO:tasks.workunit.client.1.smithi067.stdout:8/269: dwrite f5 [0,4194304] 0 2024-04-12T05:37:20.344 INFO:tasks.workunit.client.1.smithi067.stdout:0/334: dwrite dd/d12/d14/d1a/d3d/d41/f62 [0,4194304] 0 2024-04-12T05:37:20.344 INFO:tasks.workunit.client.1.smithi067.stdout:2/352: creat d5/da/d3d/f6d x:0 0 0 2024-04-12T05:37:20.344 INFO:tasks.workunit.client.1.smithi067.stdout:5/516: mknod d0/d6/d40/d4b/d38/d8e/d8f/cb7 0 2024-04-12T05:37:20.346 INFO:tasks.workunit.client.1.smithi067.stdout:7/306: sync 2024-04-12T05:37:20.346 INFO:tasks.workunit.client.1.smithi067.stdout:3/232: sync 2024-04-12T05:37:20.346 INFO:tasks.workunit.client.1.smithi067.stdout:6/348: truncate dd/d11/f19 2539270 0 2024-04-12T05:37:20.349 INFO:tasks.workunit.client.1.smithi067.stdout:8/270: unlink d6/f48 0 2024-04-12T05:37:20.349 INFO:tasks.workunit.client.1.smithi067.stdout:5/517: unlink d0/d3a/l34 0 2024-04-12T05:37:20.353 INFO:tasks.workunit.client.1.smithi067.stdout:0/335: unlink dd/f42 0 2024-04-12T05:37:20.354 INFO:tasks.workunit.client.1.smithi067.stdout:1/264: rename d4/d9/f29 to d4/d9/d14/f57 0 2024-04-12T05:37:20.356 INFO:tasks.workunit.client.1.smithi067.stdout:3/233: fsync dc/df/f1e 0 2024-04-12T05:37:20.359 INFO:tasks.workunit.client.1.smithi067.stdout:6/349: write dd/d11/d27/f58 [94106,8615] 0 2024-04-12T05:37:20.361 INFO:tasks.workunit.client.1.smithi067.stdout:3/234: symlink dc/df/l45 0 2024-04-12T05:37:20.362 INFO:tasks.workunit.client.1.smithi067.stdout:1/265: write d4/d9/d12/d28/f2b [890572,55904] 0 2024-04-12T05:37:20.363 INFO:tasks.workunit.client.1.smithi067.stdout:5/518: fdatasync d0/da/f57 0 2024-04-12T05:37:20.363 INFO:tasks.workunit.client.1.smithi067.stdout:0/336: read dd/d12/f4e [1962994,41919] 0 2024-04-12T05:37:20.364 INFO:tasks.workunit.client.1.smithi067.stdout:7/307: dread d0/d23/d45/f26 [0,4194304] 0 2024-04-12T05:37:20.365 INFO:tasks.workunit.client.1.smithi067.stdout:3/235: fdatasync dc/df/f3d 0 2024-04-12T05:37:20.366 INFO:tasks.workunit.client.1.smithi067.stdout:9/270: dread d1/d28/d23/f24 [0,4194304] 0 2024-04-12T05:37:20.366 INFO:tasks.workunit.client.1.smithi067.stdout:6/350: write dd/d11/f5d [937986,102683] 0 2024-04-12T05:37:20.367 INFO:tasks.workunit.client.1.smithi067.stdout:2/353: unlink d5/da/de/d14/d34/d40/c62 0 2024-04-12T05:37:20.370 INFO:tasks.workunit.client.1.smithi067.stdout:3/236: creat dc/d1d/d26/f46 x:0 0 0 2024-04-12T05:37:20.372 INFO:tasks.workunit.client.1.smithi067.stdout:8/271: mknod d6/d8/d18/d4e/c53 0 2024-04-12T05:37:20.372 INFO:tasks.workunit.client.1.smithi067.stdout:5/519: fdatasync d0/d6/f12 0 2024-04-12T05:37:20.375 INFO:tasks.workunit.client.1.smithi067.stdout:2/354: rename d5/da/d3d/d55 to d5/d43/d6e 0 2024-04-12T05:37:20.381 INFO:tasks.workunit.client.1.smithi067.stdout:0/337: symlink dd/d12/d14/d1a/d3d/d41/d5f/l6b 0 2024-04-12T05:37:20.388 INFO:tasks.workunit.client.1.smithi067.stdout:7/308: dread d0/d2/d5/d1e/d51/f58 [0,4194304] 0 2024-04-12T05:37:20.390 INFO:tasks.workunit.client.1.smithi067.stdout:8/272: dwrite d6/f2c [4194304,4194304] 0 2024-04-12T05:37:20.396 INFO:tasks.workunit.client.1.smithi067.stdout:2/355: dwrite d5/da/de/d14/d34/d4c/f5f [0,4194304] 0 2024-04-12T05:37:20.398 INFO:tasks.workunit.client.1.smithi067.stdout:1/266: dread d4/d9/d14/f57 [0,4194304] 0 2024-04-12T05:37:20.399 INFO:tasks.workunit.client.1.smithi067.stdout:8/273: creat d6/d8/d18/d36/f54 x:0 0 0 2024-04-12T05:37:20.403 INFO:tasks.workunit.client.1.smithi067.stdout:8/274: fdatasync d6/d8/f14 0 2024-04-12T05:37:20.403 INFO:tasks.workunit.client.1.smithi067.stdout:7/309: rename d0/d2/d5/db/f18 to d0/d2/d5/db/d34/f62 0 2024-04-12T05:37:20.405 INFO:tasks.workunit.client.1.smithi067.stdout:7/310: chown d0/d2/d5/d1d/d43/f49 16681612 1 2024-04-12T05:37:20.407 INFO:tasks.workunit.client.1.smithi067.stdout:6/351: dread fa [4194304,4194304] 0 2024-04-12T05:37:20.408 INFO:tasks.workunit.client.1.smithi067.stdout:2/356: rename d5/da/de/d14/f24 to d5/d23/d3a/f6f 0 2024-04-12T05:37:20.409 INFO:tasks.workunit.client.1.smithi067.stdout:5/520: dwrite d0/d6/d40/d4b/d38/f79 [4194304,4194304] 0 2024-04-12T05:37:20.413 INFO:tasks.workunit.client.1.smithi067.stdout:1/267: creat d4/d46/f58 x:0 0 0 2024-04-12T05:37:20.415 INFO:tasks.workunit.client.1.smithi067.stdout:0/338: truncate dd/f18 6084848 0 2024-04-12T05:37:20.421 INFO:tasks.workunit.client.1.smithi067.stdout:7/311: dwrite d0/f5e [0,4194304] 0 2024-04-12T05:37:20.424 INFO:tasks.workunit.client.1.smithi067.stdout:6/352: dread dd/f3b [0,4194304] 0 2024-04-12T05:37:20.426 INFO:tasks.workunit.client.1.smithi067.stdout:1/268: truncate d4/d9/d12/f53 692315 0 2024-04-12T05:37:20.434 INFO:tasks.workunit.client.0.smithi032.stdout:1/807: write d5/de/d17/d35/d78/fd7 [57013,87920] 0 2024-04-12T05:37:20.436 INFO:tasks.workunit.client.0.smithi032.stdout:1/808: stat d5/de/d17/d1b/c6d 0 2024-04-12T05:37:20.440 INFO:tasks.workunit.client.0.smithi032.stdout:1/809: creat d5/de/d17/ffd x:0 0 0 2024-04-12T05:37:20.440 INFO:tasks.workunit.client.0.smithi032.stdout:9/750: dwrite d9/d32/f53 [0,4194304] 0 2024-04-12T05:37:20.442 INFO:tasks.workunit.client.1.smithi067.stdout:6/353: dread dd/d30/f62 [0,4194304] 0 2024-04-12T05:37:20.443 INFO:tasks.workunit.client.1.smithi067.stdout:5/521: dwrite d0/da/f57 [0,4194304] 0 2024-04-12T05:37:20.444 INFO:tasks.workunit.client.0.smithi032.stdout:1/810: creat d5/de/d17/d35/d73/d74/ffe x:0 0 0 2024-04-12T05:37:20.445 INFO:tasks.workunit.client.0.smithi032.stdout:1/811: chown d5/de/d2a/fdd 119280 1 2024-04-12T05:37:20.446 INFO:tasks.workunit.client.1.smithi067.stdout:1/269: dread d4/d9/d12/f16 [0,4194304] 0 2024-04-12T05:37:20.446 INFO:tasks.workunit.client.0.smithi032.stdout:9/751: rename d9/d15/d1a/dd2/d65/cec to d9/df7/cfd 0 2024-04-12T05:37:20.447 INFO:tasks.workunit.client.1.smithi067.stdout:0/339: dwrite dd/d12/d14/d49/f4d [0,4194304] 0 2024-04-12T05:37:20.448 INFO:tasks.workunit.client.1.smithi067.stdout:7/312: creat d0/d2/d5/dc/d5c/f63 x:0 0 0 2024-04-12T05:37:20.448 INFO:tasks.workunit.client.1.smithi067.stdout:8/275: dwrite d6/d8/d18/df/f47 [0,4194304] 0 2024-04-12T05:37:20.450 INFO:tasks.workunit.client.1.smithi067.stdout:6/354: stat dd/d11/l38 0 2024-04-12T05:37:20.451 INFO:tasks.workunit.client.0.smithi032.stdout:9/752: mkdir d9/d32/d86/dfe 0 2024-04-12T05:37:20.452 INFO:tasks.workunit.client.1.smithi067.stdout:0/340: chown c5 31777305 1 2024-04-12T05:37:20.454 INFO:tasks.workunit.client.0.smithi032.stdout:1/812: getdents d5/de/d17/d35/d78 0 2024-04-12T05:37:20.458 INFO:tasks.workunit.client.1.smithi067.stdout:5/522: rename d0/d6 to d0/d6/d20/d62/db8 22 2024-04-12T05:37:20.458 INFO:tasks.workunit.client.1.smithi067.stdout:7/313: mknod d0/d23/c64 0 2024-04-12T05:37:20.459 INFO:tasks.workunit.client.0.smithi032.stdout:9/753: mknod d9/d15/d1a/d1d/d5a/d6b/cff 0 2024-04-12T05:37:20.459 INFO:tasks.workunit.client.1.smithi067.stdout:6/355: mkdir dd/d21/d64/d75 0 2024-04-12T05:37:20.459 INFO:tasks.workunit.client.1.smithi067.stdout:1/270: dread d4/d9/f17 [0,4194304] 0 2024-04-12T05:37:20.462 INFO:tasks.workunit.client.1.smithi067.stdout:1/271: chown d4/lf 92050273 1 2024-04-12T05:37:20.464 INFO:tasks.workunit.client.1.smithi067.stdout:0/341: rename dd/d12/d14/d49/f53 to dd/d12/d14/d1a/f6c 0 2024-04-12T05:37:20.464 INFO:tasks.workunit.client.1.smithi067.stdout:8/276: dwrite d6/d8/d18/df/f47 [0,4194304] 0 2024-04-12T05:37:20.465 INFO:tasks.workunit.client.0.smithi032.stdout:1/813: dwrite d5/de/d17/d35/d3b/dda/de0/ff4 [0,4194304] 0 2024-04-12T05:37:20.466 INFO:tasks.workunit.client.1.smithi067.stdout:8/277: chown d6/d8/d18/l31 131680129 1 2024-04-12T05:37:20.466 INFO:tasks.workunit.client.1.smithi067.stdout:5/523: mkdir d0/d6/d40/d4b/d38/d8e/d8f/db9 0 2024-04-12T05:37:20.467 INFO:tasks.workunit.client.1.smithi067.stdout:0/342: chown dd/d12/d14/d1a/d3d/d41/c65 767368517 1 2024-04-12T05:37:20.468 INFO:tasks.workunit.client.1.smithi067.stdout:1/272: write d4/f5 [1458491,105385] 0 2024-04-12T05:37:20.472 INFO:tasks.workunit.client.1.smithi067.stdout:7/314: read d0/d23/f27 [554561,79172] 0 2024-04-12T05:37:20.476 INFO:tasks.workunit.client.1.smithi067.stdout:1/273: mkdir d4/d9/d12/d59 0 2024-04-12T05:37:20.483 INFO:tasks.workunit.client.0.smithi032.stdout:1/814: write d5/de/d17/d35/d61/d77/d8f/d9c/fb5 [209534,121121] 0 2024-04-12T05:37:20.484 INFO:tasks.workunit.client.1.smithi067.stdout:8/278: dwrite d6/d8/d18/d36/f44 [0,4194304] 0 2024-04-12T05:37:20.488 INFO:tasks.workunit.client.0.smithi032.stdout:1/815: mknod d5/de/d17/d35/d3b/cff 0 2024-04-12T05:37:20.488 INFO:tasks.workunit.client.0.smithi032.stdout:9/754: dwrite d9/d15/d1a/dd2/d65/fbc [0,4194304] 0 2024-04-12T05:37:20.489 INFO:tasks.workunit.client.1.smithi067.stdout:5/524: dwrite d0/d6/d20/d3e/d69/fa5 [0,4194304] 0 2024-04-12T05:37:20.493 INFO:tasks.workunit.client.1.smithi067.stdout:8/279: creat d6/d8/d18/df/d12/f55 x:0 0 0 2024-04-12T05:37:20.493 INFO:tasks.workunit.client.1.smithi067.stdout:5/525: chown d0/d3a/d2c/l73 874952001 1 2024-04-12T05:37:20.494 INFO:tasks.workunit.client.0.smithi032.stdout:1/816: unlink d5/de/d17/d35/d3e/l52 0 2024-04-12T05:37:20.494 INFO:tasks.workunit.client.1.smithi067.stdout:8/280: fdatasync f5 0 2024-04-12T05:37:20.496 INFO:tasks.workunit.client.1.smithi067.stdout:7/315: dwrite d0/d2/d5/fe [0,4194304] 0 2024-04-12T05:37:20.497 INFO:tasks.workunit.client.0.smithi032.stdout:8/995: sync 2024-04-12T05:37:20.498 INFO:tasks.workunit.client.0.smithi032.stdout:9/755: fdatasync d9/d15/d1a/d24/d27/d88/dc1/fd6 0 2024-04-12T05:37:20.500 INFO:tasks.workunit.client.0.smithi032.stdout:1/817: mkdir d5/de/d17/d35/d61/d77/d100 0 2024-04-12T05:37:20.503 INFO:tasks.workunit.client.1.smithi067.stdout:1/274: dwrite d4/d9/d12/d28/f39 [4194304,4194304] 0 2024-04-12T05:37:20.505 INFO:tasks.workunit.client.1.smithi067.stdout:5/526: write d0/d6/d40/f7f [630382,73238] 0 2024-04-12T05:37:20.506 INFO:tasks.workunit.client.1.smithi067.stdout:1/275: dread - d4/d9/f4a zero size 2024-04-12T05:37:20.514 INFO:tasks.workunit.client.0.smithi032.stdout:8/996: mknod d0/db/dd/d11/d1b/d22/d54/d136/c156 0 2024-04-12T05:37:20.515 INFO:tasks.workunit.client.0.smithi032.stdout:9/756: mknod d9/d15/d1a/d69/dcf/c100 0 2024-04-12T05:37:20.517 INFO:tasks.workunit.client.0.smithi032.stdout:9/757: readlink d9/d15/d1a/dd2/lcb 0 2024-04-12T05:37:20.518 INFO:tasks.workunit.client.0.smithi032.stdout:9/758: write d9/d32/d86/dcc/ff0 [731143,14507] 0 2024-04-12T05:37:20.520 INFO:tasks.workunit.client.1.smithi067.stdout:8/281: write d6/d8/d18/f41 [2077134,89836] 0 2024-04-12T05:37:20.520 INFO:tasks.workunit.client.1.smithi067.stdout:6/356: write dd/f17 [59041,70048] 0 2024-04-12T05:37:20.522 INFO:tasks.workunit.client.0.smithi032.stdout:9/759: creat d9/d15/d1a/d24/f101 x:0 0 0 2024-04-12T05:37:20.523 INFO:tasks.workunit.client.1.smithi067.stdout:7/316: unlink d0/f9 0 2024-04-12T05:37:20.523 INFO:tasks.workunit.client.1.smithi067.stdout:8/282: chown d6/d8/d18/df/d12/l37 85459445 1 2024-04-12T05:37:20.525 INFO:tasks.workunit.client.1.smithi067.stdout:1/276: rename d4/d9/d3a/l27 to d4/d9/d4d/l5a 0 2024-04-12T05:37:20.525 INFO:tasks.workunit.client.0.smithi032.stdout:9/760: mknod d9/d15/d1a/d24/d27/d88/dc1/c102 0 2024-04-12T05:37:20.526 INFO:tasks.workunit.client.1.smithi067.stdout:6/357: creat dd/d21/d23/d67/f76 x:0 0 0 2024-04-12T05:37:20.527 INFO:tasks.workunit.client.0.smithi032.stdout:9/761: dread - d9/d15/d1a/dd2/d65/d80/d87/fc2 zero size 2024-04-12T05:37:20.528 INFO:tasks.workunit.client.1.smithi067.stdout:5/527: unlink d0/d3a/d2c/d91/f7c 0 2024-04-12T05:37:20.529 INFO:tasks.workunit.client.0.smithi032.stdout:8/997: dread d0/db/dd/d11/d1b/d22/d54/f64 [0,4194304] 0 2024-04-12T05:37:20.529 INFO:tasks.workunit.client.1.smithi067.stdout:0/343: dwrite dd/f13 [4194304,4194304] 0 2024-04-12T05:37:20.530 INFO:tasks.workunit.client.1.smithi067.stdout:1/277: write d4/d9/d12/d2c/f37 [493107,90008] 0 2024-04-12T05:37:20.531 INFO:tasks.workunit.client.0.smithi032.stdout:9/762: creat d9/d15/d1a/d24/d27/de1/f103 x:0 0 0 2024-04-12T05:37:20.531 INFO:tasks.workunit.client.1.smithi067.stdout:7/317: truncate d0/d23/f27 1821574 0 2024-04-12T05:37:20.536 INFO:tasks.workunit.client.1.smithi067.stdout:0/344: creat dd/d12/d14/d49/f6d x:0 0 0 2024-04-12T05:37:20.537 INFO:tasks.workunit.client.0.smithi032.stdout:9/763: read d9/d15/d1a/dd2/d65/d80/d87/ff9 [5518492,45210] 0 2024-04-12T05:37:20.539 INFO:tasks.workunit.client.0.smithi032.stdout:1/818: write d5/de/d2a/f68 [3522517,33105] 0 2024-04-12T05:37:20.539 INFO:tasks.workunit.client.0.smithi032.stdout:8/998: dwrite d0/f2f [4194304,4194304] 0 2024-04-12T05:37:20.541 INFO:tasks.workunit.client.1.smithi067.stdout:6/358: read dd/d21/d23/f3c [266113,51608] 0 2024-04-12T05:37:20.542 INFO:tasks.workunit.client.0.smithi032.stdout:9/764: symlink d9/df7/l104 0 2024-04-12T05:37:20.542 INFO:tasks.workunit.client.1.smithi067.stdout:1/278: creat d4/d46/f5b x:0 0 0 2024-04-12T05:37:20.542 INFO:tasks.workunit.client.1.smithi067.stdout:8/283: dwrite d6/d8/d18/df/f47 [0,4194304] 0 2024-04-12T05:37:20.545 INFO:tasks.workunit.client.1.smithi067.stdout:1/279: write d4/d9/d14/f21 [1408781,66743] 0 2024-04-12T05:37:20.546 INFO:tasks.workunit.client.1.smithi067.stdout:6/359: truncate dd/d21/d63/f6e 837734 0 2024-04-12T05:37:20.548 INFO:tasks.workunit.client.1.smithi067.stdout:6/360: stat dd/d11/d27/d4a 0 2024-04-12T05:37:20.548 INFO:tasks.workunit.client.0.smithi032.stdout:9/765: chown d9/d1f/ce6 1032 1 2024-04-12T05:37:20.550 INFO:tasks.workunit.client.1.smithi067.stdout:0/345: mknod dd/d12/c6e 0 2024-04-12T05:37:20.552 INFO:tasks.workunit.client.0.smithi032.stdout:8/999: dread d0/db/dd/d11/d1b/d22/d35/f11b [0,4194304] 0 2024-04-12T05:37:20.553 INFO:tasks.workunit.client.0.smithi032.stdout:1/819: dread d5/de/d2a/f2b [0,4194304] 0 2024-04-12T05:37:20.556 INFO:tasks.workunit.client.1.smithi067.stdout:6/361: dwrite dd/d21/d23/f43 [0,4194304] 0 2024-04-12T05:37:20.557 INFO:tasks.workunit.client.0.smithi032.stdout:1/820: write d5/de/d17/d1b/fea [661323,57257] 0 2024-04-12T05:37:20.559 INFO:tasks.workunit.client.0.smithi032.stdout:9/766: dwrite d9/d15/f48 [0,4194304] 0 2024-04-12T05:37:20.559 INFO:tasks.workunit.client.1.smithi067.stdout:5/528: write d0/da/d41/f60 [151413,91838] 0 2024-04-12T05:37:20.559 INFO:tasks.workunit.client.1.smithi067.stdout:1/280: symlink d4/d44/l5c 0 2024-04-12T05:37:20.560 INFO:tasks.workunit.client.1.smithi067.stdout:8/284: getdents d6/d8/d18/d3c/d4f 0 2024-04-12T05:37:20.562 INFO:tasks.workunit.client.1.smithi067.stdout:7/318: creat d0/d2/d5/db/f65 x:0 0 0 2024-04-12T05:37:20.564 INFO:tasks.workunit.client.1.smithi067.stdout:6/362: mknod dd/d21/d23/d67/c77 0 2024-04-12T05:37:20.565 INFO:tasks.workunit.client.0.smithi032.stdout:1/821: creat d5/de/d17/d35/d61/d77/d8f/f101 x:0 0 0 2024-04-12T05:37:20.567 INFO:tasks.workunit.client.1.smithi067.stdout:6/363: read f0 [1521376,108529] 0 2024-04-12T05:37:20.571 INFO:tasks.workunit.client.1.smithi067.stdout:1/281: dread d4/f5 [4194304,4194304] 0 2024-04-12T05:37:20.572 INFO:tasks.workunit.client.0.smithi032.stdout:1/822: getdents d5/de/d17/dd6/de5 0 2024-04-12T05:37:20.572 INFO:tasks.workunit.client.1.smithi067.stdout:8/285: dread d6/d8/d18/df/f20 [0,4194304] 0 2024-04-12T05:37:20.573 INFO:tasks.workunit.client.0.smithi032.stdout:9/767: write d9/d15/f44 [4115057,15197] 0 2024-04-12T05:37:20.574 INFO:tasks.workunit.client.0.smithi032.stdout:9/768: fsync d9/d32/d86/f9b 0 2024-04-12T05:37:20.575 INFO:tasks.workunit.client.0.smithi032.stdout:1/823: creat d5/de/d17/d35/d61/f102 x:0 0 0 2024-04-12T05:37:20.576 INFO:tasks.workunit.client.1.smithi067.stdout:6/364: rename dd/c29 to dd/d11/d27/c78 0 2024-04-12T05:37:20.577 INFO:tasks.workunit.client.1.smithi067.stdout:0/346: write dd/d12/f2e [453653,15566] 0 2024-04-12T05:37:20.579 INFO:tasks.workunit.client.0.smithi032.stdout:1/824: mknod d5/de/d17/d35/d3b/dda/de0/c103 0 2024-04-12T05:37:20.580 INFO:tasks.workunit.client.1.smithi067.stdout:7/319: write d0/d2/d5/d1e/f20 [13613,80856] 0 2024-04-12T05:37:20.581 INFO:tasks.workunit.client.1.smithi067.stdout:0/347: mkdir dd/d6f 0 2024-04-12T05:37:20.582 INFO:tasks.workunit.client.0.smithi032.stdout:9/769: read f5 [2970426,65600] 0 2024-04-12T05:37:20.585 INFO:tasks.workunit.client.0.smithi032.stdout:9/770: dread - d9/d32/ff5 zero size 2024-04-12T05:37:20.585 INFO:tasks.workunit.client.1.smithi067.stdout:6/365: write dd/d11/d27/f5a [978322,127718] 0 2024-04-12T05:37:20.588 INFO:tasks.workunit.client.0.smithi032.stdout:1/825: dwrite d5/de/d17/d35/d61/d77/d8f/d9c/fa5 [0,4194304] 0 2024-04-12T05:37:20.589 INFO:tasks.workunit.client.1.smithi067.stdout:7/320: write d0/d2/d5/dc/d5c/f63 [633888,57778] 0 2024-04-12T05:37:20.589 INFO:tasks.workunit.client.1.smithi067.stdout:0/348: fdatasync dd/d12/d14/d1a/f1e 0 2024-04-12T05:37:20.592 INFO:tasks.workunit.client.1.smithi067.stdout:1/282: dwrite d4/d46/f5b [0,4194304] 0 2024-04-12T05:37:20.592 INFO:tasks.workunit.client.1.smithi067.stdout:6/366: dread - dd/d11/d27/d4a/f73 zero size 2024-04-12T05:37:20.595 INFO:tasks.workunit.client.1.smithi067.stdout:8/286: dread d6/d8/d18/f41 [0,4194304] 0 2024-04-12T05:37:20.596 INFO:tasks.workunit.client.0.smithi032.stdout:9/771: write d9/d1f/f39 [659307,33914] 0 2024-04-12T05:37:20.601 INFO:tasks.workunit.client.1.smithi067.stdout:1/283: mkdir d4/d9/d4c/d5d 0 2024-04-12T05:37:20.602 INFO:tasks.workunit.client.1.smithi067.stdout:8/287: unlink d6/d2f/f4a 0 2024-04-12T05:37:20.603 INFO:tasks.workunit.client.1.smithi067.stdout:5/529: dwrite d0/d6/d20/f58 [0,4194304] 0 2024-04-12T05:37:20.605 INFO:tasks.workunit.client.1.smithi067.stdout:0/349: creat dd/d40/f70 x:0 0 0 2024-04-12T05:37:20.606 INFO:tasks.workunit.client.1.smithi067.stdout:1/284: mknod d4/d3d/c5e 0 2024-04-12T05:37:20.607 INFO:tasks.workunit.client.0.smithi032.stdout:9/772: dwrite d9/d15/d1a/d24/d27/f50 [0,4194304] 0 2024-04-12T05:37:20.608 INFO:tasks.workunit.client.1.smithi067.stdout:0/350: read dd/f38 [528413,20971] 0 2024-04-12T05:37:20.609 INFO:tasks.workunit.client.0.smithi032.stdout:1/826: dread d5/de/d17/f20 [8388608,4194304] 0 2024-04-12T05:37:20.610 INFO:tasks.workunit.client.0.smithi032.stdout:1/827: stat d5/de/d17/f25 0 2024-04-12T05:37:20.611 INFO:tasks.workunit.client.1.smithi067.stdout:7/321: dwrite d0/d2/f21 [0,4194304] 0 2024-04-12T05:37:20.613 INFO:tasks.workunit.client.1.smithi067.stdout:0/351: dread - dd/d12/f59 zero size 2024-04-12T05:37:20.613 INFO:tasks.workunit.client.0.smithi032.stdout:9/773: mknod d9/d15/d1a/d1d/dd4/c105 0 2024-04-12T05:37:20.615 INFO:tasks.workunit.client.0.smithi032.stdout:1/828: mkdir d5/de/d17/d35/d61/dca/d104 0 2024-04-12T05:37:20.619 INFO:tasks.workunit.client.0.smithi032.stdout:9/774: rename d9/d15/d1a/dba/lca to d9/d32/d86/l106 0 2024-04-12T05:37:20.620 INFO:tasks.workunit.client.0.smithi032.stdout:1/829: mkdir d5/de/d17/d35/d61/d105 0 2024-04-12T05:37:20.624 INFO:tasks.workunit.client.0.smithi032.stdout:9/775: unlink d9/d58/f79 0 2024-04-12T05:37:20.624 INFO:tasks.workunit.client.1.smithi067.stdout:1/285: mkdir d4/d9/d5f 0 2024-04-12T05:37:20.625 INFO:tasks.workunit.client.1.smithi067.stdout:5/530: write d0/da/d41/f59 [545153,49474] 0 2024-04-12T05:37:20.625 INFO:tasks.workunit.client.0.smithi032.stdout:9/776: fsync d9/d15/d1a/d24/fed 0 2024-04-12T05:37:20.627 INFO:tasks.workunit.client.1.smithi067.stdout:5/531: stat d0/d6/d20/d62 0 2024-04-12T05:37:20.627 INFO:tasks.workunit.client.0.smithi032.stdout:1/830: symlink d5/de/d17/d35/d3b/l106 0 2024-04-12T05:37:20.628 INFO:tasks.workunit.client.1.smithi067.stdout:8/288: rename d6/d8/d18/c46 to d6/c56 0 2024-04-12T05:37:20.628 INFO:tasks.workunit.client.1.smithi067.stdout:0/352: fsync dd/d40/f44 0 2024-04-12T05:37:20.629 INFO:tasks.workunit.client.1.smithi067.stdout:5/532: write d0/d6/d33/f9f [399089,125375] 0 2024-04-12T05:37:20.629 INFO:tasks.workunit.client.1.smithi067.stdout:7/322: symlink d0/d2/d5/db/d34/l66 0 2024-04-12T05:37:20.631 INFO:tasks.workunit.client.1.smithi067.stdout:1/286: mkdir d4/d9/d3a/d51/d55/d60 0 2024-04-12T05:37:20.633 INFO:tasks.workunit.client.0.smithi032.stdout:9/777: rename d9/d15/d1a/d69/f99 to d9/d15/d1a/d24/f107 0 2024-04-12T05:37:20.635 INFO:tasks.workunit.client.1.smithi067.stdout:7/323: unlink d0/d23/l46 0 2024-04-12T05:37:20.637 INFO:tasks.workunit.client.1.smithi067.stdout:0/353: mknod dd/d12/d14/d1a/d3d/d41/c71 0 2024-04-12T05:37:20.638 INFO:tasks.workunit.client.0.smithi032.stdout:9/778: symlink d9/d15/d2c/dab/l108 0 2024-04-12T05:37:20.638 INFO:tasks.workunit.client.0.smithi032.stdout:1/831: dwrite d5/de/d17/d35/d3b/d49/fd9 [0,4194304] 0 2024-04-12T05:37:20.641 INFO:tasks.workunit.client.1.smithi067.stdout:6/367: dread dd/f1d [0,4194304] 0 2024-04-12T05:37:20.641 INFO:tasks.workunit.client.1.smithi067.stdout:8/289: dwrite d6/f2c [0,4194304] 0 2024-04-12T05:37:20.642 INFO:tasks.workunit.client.0.smithi032.stdout:9/779: mkdir d9/d15/d1a/d24/d27/dce/dd3/d109 0 2024-04-12T05:37:20.646 INFO:tasks.workunit.client.1.smithi067.stdout:0/354: write dd/d12/d14/d51/f5a [4495141,59204] 0 2024-04-12T05:37:20.646 INFO:tasks.workunit.client.0.smithi032.stdout:1/832: rename d5/d2c/ff2 to d5/de/d17/d35/f107 0 2024-04-12T05:37:20.649 INFO:tasks.workunit.client.0.smithi032.stdout:7/916: sync 2024-04-12T05:37:20.650 INFO:tasks.workunit.client.0.smithi032.stdout:2/906: sync 2024-04-12T05:37:20.650 INFO:tasks.workunit.client.1.smithi067.stdout:8/290: chown d6/d8/d18/df/f20 16 1 2024-04-12T05:37:20.650 INFO:tasks.workunit.client.1.smithi067.stdout:6/368: fdatasync dd/d21/d63/f6e 0 2024-04-12T05:37:20.650 INFO:tasks.workunit.client.1.smithi067.stdout:7/324: rename d0/d2/d5/dc to d0/d67 0 2024-04-12T05:37:20.651 INFO:tasks.workunit.client.0.smithi032.stdout:9/780: mknod d9/d15/d1a/d24/d27/c10a 0 2024-04-12T05:37:20.652 INFO:tasks.workunit.client.1.smithi067.stdout:0/355: fdatasync dd/d12/f4e 0 2024-04-12T05:37:20.652 INFO:tasks.workunit.client.0.smithi032.stdout:9/781: chown d9/d15/d1a/d24/d27/de1/f103 217544240 1 2024-04-12T05:37:20.653 INFO:tasks.workunit.client.0.smithi032.stdout:1/833: mkdir d5/de/d17/d35/d61/d105/d108 0 2024-04-12T05:37:20.653 INFO:tasks.workunit.client.0.smithi032.stdout:9/782: fsync d9/fa1 0 2024-04-12T05:37:20.654 INFO:tasks.workunit.client.0.smithi032.stdout:2/907: fsync dd/d19/d9d/d1c/d64/d32/f33 0 2024-04-12T05:37:20.654 INFO:tasks.workunit.client.1.smithi067.stdout:0/356: truncate dd/d5b/f68 889689 0 2024-04-12T05:37:20.656 INFO:tasks.workunit.client.0.smithi032.stdout:1/834: write d5/de/d17/d35/d61/f102 [977085,81045] 0 2024-04-12T05:37:20.656 INFO:tasks.workunit.client.1.smithi067.stdout:0/357: write dd/d12/f3a [2731764,11211] 0 2024-04-12T05:37:20.656 INFO:tasks.workunit.client.1.smithi067.stdout:6/369: chown dd/l2a 56863 1 2024-04-12T05:37:20.656 INFO:tasks.workunit.client.0.smithi032.stdout:7/917: creat d0/d10/d70/f12c x:0 0 0 2024-04-12T05:37:20.657 INFO:tasks.workunit.client.1.smithi067.stdout:4/231: sync 2024-04-12T05:37:20.657 INFO:tasks.workunit.client.1.smithi067.stdout:8/291: symlink d6/d8/l57 0 2024-04-12T05:37:20.658 INFO:tasks.workunit.client.0.smithi032.stdout:2/908: truncate f1 3425463 0 2024-04-12T05:37:20.659 INFO:tasks.workunit.client.1.smithi067.stdout:7/325: mknod d0/d2/d5/d61/c68 0 2024-04-12T05:37:20.664 INFO:tasks.workunit.client.1.smithi067.stdout:6/370: rmdir dd/d11 39 2024-04-12T05:37:20.668 INFO:tasks.workunit.client.1.smithi067.stdout:4/232: creat d8/da/d1d/d15/d1c/f52 x:0 0 0 2024-04-12T05:37:20.669 INFO:tasks.workunit.client.1.smithi067.stdout:1/287: write d4/d9/f1c [1118935,27376] 0 2024-04-12T05:37:20.669 INFO:tasks.workunit.client.0.smithi032.stdout:9/783: rename d9/d15/d1a/d1d/dd4 to d9/d15/d1a/d10b 0 2024-04-12T05:37:20.671 INFO:tasks.workunit.client.0.smithi032.stdout:9/784: chown d9/d15/d1a/d1d/d5a/fe8 89463250 1 2024-04-12T05:37:20.672 INFO:tasks.workunit.client.1.smithi067.stdout:5/533: dread d0/f2 [0,4194304] 0 2024-04-12T05:37:20.674 INFO:tasks.workunit.client.0.smithi032.stdout:7/918: rename d0/d13/fc5 to d0/dd/d51/dea/d105/f12d 0 2024-04-12T05:37:20.675 INFO:tasks.workunit.client.0.smithi032.stdout:9/785: dread d9/d32/d86/f8c [0,4194304] 0 2024-04-12T05:37:20.675 INFO:tasks.workunit.client.1.smithi067.stdout:1/288: read d4/d9/f45 [2685251,87817] 0 2024-04-12T05:37:20.676 INFO:tasks.workunit.client.0.smithi032.stdout:7/919: truncate d0/dff/d71/d119/f11b 4568 0 2024-04-12T05:37:20.678 INFO:tasks.workunit.client.0.smithi032.stdout:9/786: dread - d9/d15/d1a/d24/d27/d88/dd9/fda zero size 2024-04-12T05:37:20.678 INFO:tasks.workunit.client.1.smithi067.stdout:4/233: read - d8/d9/d3d/d48/f3a zero size 2024-04-12T05:37:20.678 INFO:tasks.workunit.client.0.smithi032.stdout:7/920: chown d0/dd/d51/dea/d105/l123 425544 1 2024-04-12T05:37:20.679 INFO:tasks.workunit.client.1.smithi067.stdout:8/292: dwrite d6/d8/d18/f3f [0,4194304] 0 2024-04-12T05:37:20.680 INFO:tasks.workunit.client.1.smithi067.stdout:6/371: rename dd/d21/d23/d67 to dd/d11/d27/d4a/d79 0 2024-04-12T05:37:20.681 INFO:tasks.workunit.client.0.smithi032.stdout:7/921: dread d0/dff/d4c/f103 [0,4194304] 0 2024-04-12T05:37:20.682 INFO:tasks.workunit.client.1.smithi067.stdout:5/534: rmdir d0/d6/d40/d4b/d38/d8e 39 2024-04-12T05:37:20.684 INFO:tasks.workunit.client.1.smithi067.stdout:1/289: mkdir d4/d9/d14/d61 0 2024-04-12T05:37:20.684 INFO:tasks.workunit.client.0.smithi032.stdout:1/835: rename d5/de/d17/d35/d78/fb1 to d5/de/d17/d35/d3b/f109 0 2024-04-12T05:37:20.686 INFO:tasks.workunit.client.1.smithi067.stdout:0/358: dwrite dd/d12/d14/d1a/f2b [0,4194304] 0 2024-04-12T05:37:20.690 INFO:tasks.workunit.client.1.smithi067.stdout:4/234: mknod d8/d9/d3d/d48/d3c/c53 0 2024-04-12T05:37:20.690 INFO:tasks.workunit.client.1.smithi067.stdout:8/293: symlink d6/d8/d18/d3c/d4f/l58 0 2024-04-12T05:37:20.691 INFO:tasks.workunit.client.1.smithi067.stdout:8/294: fdatasync f5 0 2024-04-12T05:37:20.694 INFO:tasks.workunit.client.1.smithi067.stdout:6/372: write dd/f3b [4166520,39700] 0 2024-04-12T05:37:20.694 INFO:tasks.workunit.client.1.smithi067.stdout:7/326: dread d0/f12 [0,4194304] 0 2024-04-12T05:37:20.694 INFO:tasks.workunit.client.1.smithi067.stdout:5/535: symlink d0/d6/d40/d4b/d38/lba 0 2024-04-12T05:37:20.695 INFO:tasks.workunit.client.1.smithi067.stdout:0/359: rmdir dd/d12/d14/d1a/d3d/d41 39 2024-04-12T05:37:20.697 INFO:tasks.workunit.client.1.smithi067.stdout:0/360: chown c7 285995114 1 2024-04-12T05:37:20.697 INFO:tasks.workunit.client.0.smithi032.stdout:2/909: rename dd/d19/d9d/d1c/d64/cbf to dd/d19/d9d/d1c/d64/d36/c122 0 2024-04-12T05:37:20.698 INFO:tasks.workunit.client.0.smithi032.stdout:4/916: sync 2024-04-12T05:37:20.698 INFO:tasks.workunit.client.0.smithi032.stdout:0/945: sync 2024-04-12T05:37:20.699 INFO:tasks.workunit.client.1.smithi067.stdout:6/373: mknod dd/d21/d63/c7a 0 2024-04-12T05:37:20.700 INFO:tasks.workunit.client.1.smithi067.stdout:4/235: mknod d8/da/d1d/d15/c54 0 2024-04-12T05:37:20.700 INFO:tasks.workunit.client.0.smithi032.stdout:1/836: creat d5/de/d17/dd6/f10a x:0 0 0 2024-04-12T05:37:20.701 INFO:tasks.workunit.client.1.smithi067.stdout:1/290: dwrite d4/d9/d12/f1b [0,4194304] 0 2024-04-12T05:37:20.703 INFO:tasks.workunit.client.0.smithi032.stdout:1/837: stat d5/de/d2a/ff3 0 2024-04-12T05:37:20.703 INFO:tasks.workunit.client.0.smithi032.stdout:7/922: mknod d0/dff/d4c/dd8/c12e 0 2024-04-12T05:37:20.705 INFO:tasks.workunit.client.0.smithi032.stdout:1/838: chown d5/d2c/d4f/c88 1539681 1 2024-04-12T05:37:20.705 INFO:tasks.workunit.client.0.smithi032.stdout:2/910: fsync dd/d19/d9d/f58 0 2024-04-12T05:37:20.707 INFO:tasks.workunit.client.0.smithi032.stdout:1/839: stat d5/l6e 0 2024-04-12T05:37:20.708 INFO:tasks.workunit.client.0.smithi032.stdout:2/911: read f8 [780993,55410] 0 2024-04-12T05:37:20.709 INFO:tasks.workunit.client.1.smithi067.stdout:6/374: chown dd/d21/d64/d6f/f70 716723724 1 2024-04-12T05:37:20.710 INFO:tasks.workunit.client.1.smithi067.stdout:7/327: creat d0/d2/d5/d1d/d43/f69 x:0 0 0 2024-04-12T05:37:20.710 INFO:tasks.workunit.client.1.smithi067.stdout:6/375: stat dd/d11/f5d 0 2024-04-12T05:37:20.710 INFO:tasks.workunit.client.1.smithi067.stdout:0/361: mkdir dd/d6f/d72 0 2024-04-12T05:37:20.710 INFO:tasks.workunit.client.1.smithi067.stdout:5/536: creat d0/da/fbb x:0 0 0 2024-04-12T05:37:20.710 INFO:tasks.workunit.client.1.smithi067.stdout:7/328: dread - d0/d2/d5/d1d/d43/f69 zero size 2024-04-12T05:37:20.711 INFO:tasks.workunit.client.1.smithi067.stdout:1/291: mknod d4/d9/d12/d2c/c62 0 2024-04-12T05:37:20.711 INFO:tasks.workunit.client.0.smithi032.stdout:0/946: mkdir d2/d28/dd6/d13d 0 2024-04-12T05:37:20.711 INFO:tasks.workunit.client.0.smithi032.stdout:4/917: dwrite d4/fb7 [0,4194304] 0 2024-04-12T05:37:20.713 INFO:tasks.workunit.client.1.smithi067.stdout:0/362: stat dd/d12/d14/d1a/l31 0 2024-04-12T05:37:20.713 INFO:tasks.workunit.client.1.smithi067.stdout:4/236: dread d8/d39/f46 [0,4194304] 0 2024-04-12T05:37:20.714 INFO:tasks.workunit.client.0.smithi032.stdout:4/918: write d4/d87/d96/d2f/f70 [5011366,122841] 0 2024-04-12T05:37:20.715 INFO:tasks.workunit.client.0.smithi032.stdout:9/787: getdents d9/d32/d86 0 2024-04-12T05:37:20.719 INFO:tasks.workunit.client.1.smithi067.stdout:8/295: dwrite d6/d8/d18/df/f24 [0,4194304] 0 2024-04-12T05:37:20.720 INFO:tasks.workunit.client.1.smithi067.stdout:4/237: readlink d8/d9/lb 0 2024-04-12T05:37:20.721 INFO:tasks.workunit.client.1.smithi067.stdout:5/537: creat d0/d6/d20/d3e/fbc x:0 0 0 2024-04-12T05:37:20.723 INFO:tasks.workunit.client.1.smithi067.stdout:5/538: readlink d0/d6/l78 0 2024-04-12T05:37:20.726 INFO:tasks.workunit.client.1.smithi067.stdout:4/238: chown d8/da/d1d/d15/c54 0 1 2024-04-12T05:37:20.726 INFO:tasks.workunit.client.1.smithi067.stdout:0/363: write dd/d12/f3e [2536025,30641] 0 2024-04-12T05:37:20.727 INFO:tasks.workunit.client.1.smithi067.stdout:1/292: getdents d4/d46 0 2024-04-12T05:37:20.729 INFO:tasks.workunit.client.1.smithi067.stdout:0/364: chown dd/d40/f70 11804908 1 2024-04-12T05:37:20.730 INFO:tasks.workunit.client.1.smithi067.stdout:8/296: rename d6/d8/f45 to d6/d8/d18/d36/f59 0 2024-04-12T05:37:20.731 INFO:tasks.workunit.client.1.smithi067.stdout:7/329: rmdir d0/d2/d5/d1d 39 2024-04-12T05:37:20.731 INFO:tasks.workunit.client.0.smithi032.stdout:4/919: fsync d4/d13/d44/f6a 0 2024-04-12T05:37:20.732 INFO:tasks.workunit.client.0.smithi032.stdout:0/947: dread d2/d8/f88 [0,4194304] 0 2024-04-12T05:37:20.733 INFO:tasks.workunit.client.0.smithi032.stdout:7/923: write d0/de/df/f14 [2383909,121721] 0 2024-04-12T05:37:20.740 INFO:tasks.workunit.client.1.smithi067.stdout:4/239: write d8/d9/f41 [935138,3032] 0 2024-04-12T05:37:20.742 INFO:tasks.workunit.client.1.smithi067.stdout:7/330: creat d0/d67/d5c/f6a x:0 0 0 2024-04-12T05:37:20.744 INFO:tasks.workunit.client.1.smithi067.stdout:4/240: fdatasync d8/da/d1d/d15/d36/f4f 0 2024-04-12T05:37:20.744 INFO:tasks.workunit.client.1.smithi067.stdout:1/293: link d4/d9/d12/f1b d4/d9/d3a/d51/f63 0 2024-04-12T05:37:20.744 INFO:tasks.workunit.client.1.smithi067.stdout:8/297: dwrite d6/d8/f14 [0,4194304] 0 2024-04-12T05:37:20.745 INFO:tasks.workunit.client.1.smithi067.stdout:5/539: mkdir d0/d6/d40/d4b/d38/d8e/d8f/db9/dbd 0 2024-04-12T05:37:20.750 INFO:tasks.workunit.client.1.smithi067.stdout:0/365: creat dd/d12/d14/d1a/d3d/f73 x:0 0 0 2024-04-12T05:37:20.751 INFO:tasks.workunit.client.1.smithi067.stdout:5/540: dread d0/d6/d33/f52 [0,4194304] 0 2024-04-12T05:37:20.752 INFO:tasks.workunit.client.1.smithi067.stdout:1/294: truncate d4/f7 6817049 0 2024-04-12T05:37:20.754 INFO:tasks.workunit.client.1.smithi067.stdout:5/541: write d0/da/d3f/d82/fa6 [33663,36644] 0 2024-04-12T05:37:20.754 INFO:tasks.workunit.client.1.smithi067.stdout:7/331: mkdir d0/d2/d5/d1d/d43/d6b 0 2024-04-12T05:37:20.755 INFO:tasks.workunit.client.0.smithi032.stdout:2/912: link dd/d19/d20/f4d dd/d19/d9d/d1c/de0/f123 0 2024-04-12T05:37:20.756 INFO:tasks.workunit.client.1.smithi067.stdout:1/295: fsync d4/f5 0 2024-04-12T05:37:20.756 INFO:tasks.workunit.client.0.smithi032.stdout:9/788: mknod d9/d32/d86/dcc/c10c 0 2024-04-12T05:37:20.757 INFO:tasks.workunit.client.1.smithi067.stdout:0/366: getdents dd/d6f/d72 0 2024-04-12T05:37:20.760 INFO:tasks.workunit.client.1.smithi067.stdout:4/241: link d8/d9/d3d/d48/d3c/c53 d8/da/d1d/d15/d1c/c55 0 2024-04-12T05:37:20.760 INFO:tasks.workunit.client.1.smithi067.stdout:1/296: write d4/d9/d12/f1b [1881637,38449] 0 2024-04-12T05:37:20.761 INFO:tasks.workunit.client.0.smithi032.stdout:4/920: fdatasync d4/d87/d96/d4d/f63 0 2024-04-12T05:37:20.761 INFO:tasks.workunit.client.0.smithi032.stdout:7/924: dread - d0/d34/dbe/ff6 zero size 2024-04-12T05:37:20.762 INFO:tasks.workunit.client.0.smithi032.stdout:0/948: creat d2/d28/d87/d79/f13e x:0 0 0 2024-04-12T05:37:20.763 INFO:tasks.workunit.client.1.smithi067.stdout:4/242: chown d8/fe 105 1 2024-04-12T05:37:20.763 INFO:tasks.workunit.client.1.smithi067.stdout:5/542: fsync d0/d6/d33/f72 0 2024-04-12T05:37:20.763 INFO:tasks.workunit.client.0.smithi032.stdout:2/913: dwrite dd/d8c/fe2 [0,4194304] 0 2024-04-12T05:37:20.765 INFO:tasks.workunit.client.0.smithi032.stdout:9/789: rename d9/d15/d2c/c92 to d9/d1f/c10d 0 2024-04-12T05:37:20.767 INFO:tasks.workunit.client.0.smithi032.stdout:1/840: link d5/de/d17/d35/c3a d5/de/d17/d35/d3e/db4/c10b 0 2024-04-12T05:37:20.768 INFO:tasks.workunit.client.1.smithi067.stdout:5/543: stat d0/d3a/d2c/db3 0 2024-04-12T05:37:20.770 INFO:tasks.workunit.client.1.smithi067.stdout:0/367: dwrite dd/d12/f59 [0,4194304] 0 2024-04-12T05:37:20.770 INFO:tasks.workunit.client.0.smithi032.stdout:2/914: readlink dd/d19/d82/lb2 0 2024-04-12T05:37:20.771 INFO:tasks.workunit.client.0.smithi032.stdout:2/915: chown dd/f1e 395660 1 2024-04-12T05:37:20.774 INFO:tasks.workunit.client.1.smithi067.stdout:0/368: fdatasync dd/d12/d14/d51/f5c 0 2024-04-12T05:37:20.777 INFO:tasks.workunit.client.1.smithi067.stdout:1/297: creat d4/d9/d12/d59/f64 x:0 0 0 2024-04-12T05:37:20.778 INFO:tasks.workunit.client.1.smithi067.stdout:8/298: dread d6/d8/d18/df/f1c [0,4194304] 0 2024-04-12T05:37:20.779 INFO:tasks.workunit.client.1.smithi067.stdout:4/243: creat d8/da/d1d/d27/f56 x:0 0 0 2024-04-12T05:37:20.779 INFO:tasks.workunit.client.0.smithi032.stdout:9/790: unlink d9/d15/d2c/d41/f9d 0 2024-04-12T05:37:20.779 INFO:tasks.workunit.client.0.smithi032.stdout:1/841: mkdir d5/de/d17/d93/d10c 0 2024-04-12T05:37:20.780 INFO:tasks.workunit.client.0.smithi032.stdout:7/925: mknod d0/dd/d51/d65/d12a/c12f 0 2024-04-12T05:37:20.780 INFO:tasks.workunit.client.1.smithi067.stdout:0/369: mknod dd/d12/d14/d51/c74 0 2024-04-12T05:37:20.783 INFO:tasks.workunit.client.0.smithi032.stdout:0/949: truncate d2/d28/d87/d111/d3e/dac/d51/d83/fb4 323466 0 2024-04-12T05:37:20.784 INFO:tasks.workunit.client.0.smithi032.stdout:9/791: read d9/d15/d1a/d24/f40 [8502177,86969] 0 2024-04-12T05:37:20.785 INFO:tasks.workunit.client.0.smithi032.stdout:0/950: dread d2/d29/d8a/fd7 [0,4194304] 0 2024-04-12T05:37:20.786 INFO:tasks.workunit.client.1.smithi067.stdout:0/370: write dd/d12/d14/d1a/d3d/f73 [841151,11303] 0 2024-04-12T05:37:20.786 INFO:tasks.workunit.client.0.smithi032.stdout:2/916: unlink dd/d19/d9d/f73 0 2024-04-12T05:37:20.787 INFO:tasks.workunit.client.0.smithi032.stdout:1/842: dread d5/de/d17/d35/d3e/db4/ff0 [0,4194304] 0 2024-04-12T05:37:20.788 INFO:tasks.workunit.client.1.smithi067.stdout:4/244: rmdir d8/da/d1d/d15/d1c 39 2024-04-12T05:37:20.791 INFO:tasks.workunit.client.1.smithi067.stdout:0/371: fdatasync dd/f13 0 2024-04-12T05:37:20.792 INFO:tasks.workunit.client.1.smithi067.stdout:8/299: dwrite d6/d8/d18/df/f24 [0,4194304] 0 2024-04-12T05:37:20.793 INFO:tasks.workunit.client.1.smithi067.stdout:5/544: truncate d0/d6/d33/f52 820787 0 2024-04-12T05:37:20.794 INFO:tasks.workunit.client.0.smithi032.stdout:2/917: dwrite dd/d19/d9d/d1c/fb6 [0,4194304] 0 2024-04-12T05:37:20.794 INFO:tasks.workunit.client.0.smithi032.stdout:1/843: mkdir d5/de/d17/d35/d61/d77/d8f/d9c/d10d 0 2024-04-12T05:37:20.795 INFO:tasks.workunit.client.0.smithi032.stdout:2/918: stat dd/d8c 0 2024-04-12T05:37:20.795 INFO:tasks.workunit.client.1.smithi067.stdout:0/372: stat dd/d12/f2a 0 2024-04-12T05:37:20.796 INFO:tasks.workunit.client.1.smithi067.stdout:1/298: write d4/d9/fb [165546,107103] 0 2024-04-12T05:37:20.796 INFO:tasks.workunit.client.0.smithi032.stdout:1/844: write d5/de/d17/d35/d3e/fc9 [774555,41150] 0 2024-04-12T05:37:20.797 INFO:tasks.workunit.client.1.smithi067.stdout:8/300: write d6/d8/d18/f33 [1817631,115101] 0 2024-04-12T05:37:20.797 INFO:tasks.workunit.client.1.smithi067.stdout:0/373: fsync dd/d12/f3a 0 2024-04-12T05:37:20.798 INFO:tasks.workunit.client.0.smithi032.stdout:9/792: symlink d9/d15/d1a/d24/d27/dce/dd3/d109/l10e 0 2024-04-12T05:37:20.798 INFO:tasks.workunit.client.0.smithi032.stdout:2/919: truncate dd/d19/d82/ddf/ff9 1021227 0 2024-04-12T05:37:20.798 INFO:tasks.workunit.client.1.smithi067.stdout:1/299: chown d4/d9/d12/d2c/c62 30845 1 2024-04-12T05:37:20.800 INFO:tasks.workunit.client.0.smithi032.stdout:1/845: unlink d5/f47 0 2024-04-12T05:37:20.802 INFO:tasks.workunit.client.0.smithi032.stdout:9/793: creat d9/d32/d6e/f10f x:0 0 0 2024-04-12T05:37:20.803 INFO:tasks.workunit.client.1.smithi067.stdout:5/545: creat d0/d6/d20/fbe x:0 0 0 2024-04-12T05:37:20.803 INFO:tasks.workunit.client.1.smithi067.stdout:1/300: unlink d4/lf 0 2024-04-12T05:37:20.803 INFO:tasks.workunit.client.0.smithi032.stdout:9/794: truncate d9/d1f/fe0 4324625 0 2024-04-12T05:37:20.805 INFO:tasks.workunit.client.0.smithi032.stdout:0/951: creat d2/d8/d17/d1b/f13f x:0 0 0 2024-04-12T05:37:20.805 INFO:tasks.workunit.client.1.smithi067.stdout:0/374: rename dd/d40 to dd/d12/d14/d1a/d75 0 2024-04-12T05:37:20.806 INFO:tasks.workunit.client.1.smithi067.stdout:4/245: getdents d8/da/d1d/d27 0 2024-04-12T05:37:20.807 INFO:tasks.workunit.client.1.smithi067.stdout:0/375: fsync f1 0 2024-04-12T05:37:20.808 INFO:tasks.workunit.client.0.smithi032.stdout:0/952: write d2/d28/d87/d79/fce [1272361,7948] 0 2024-04-12T05:37:20.809 INFO:tasks.workunit.client.1.smithi067.stdout:4/246: dread - d8/d9/d3d/d48/f3a zero size 2024-04-12T05:37:20.809 INFO:tasks.workunit.client.0.smithi032.stdout:0/953: dread - d2/d29/d8a/fd1 zero size 2024-04-12T05:37:20.811 INFO:tasks.workunit.client.1.smithi067.stdout:1/301: stat d4/d9/d3a/d51 0 2024-04-12T05:37:20.811 INFO:tasks.workunit.client.1.smithi067.stdout:4/247: chown d8/d9/d3d/d48/d3c 7 1 2024-04-12T05:37:20.812 INFO:tasks.workunit.client.1.smithi067.stdout:4/248: chown d8/da/d1d 55704183 1 2024-04-12T05:37:20.813 INFO:tasks.workunit.client.1.smithi067.stdout:1/302: write d4/d9/d12/d59/f64 [193684,6956] 0 2024-04-12T05:37:20.813 INFO:tasks.workunit.client.0.smithi032.stdout:0/954: creat d2/d8/df/d24/d60/dc8/f140 x:0 0 0 2024-04-12T05:37:20.814 INFO:tasks.workunit.client.0.smithi032.stdout:0/955: chown d2/f6 2084053315 1 2024-04-12T05:37:20.815 INFO:tasks.workunit.client.1.smithi067.stdout:0/376: creat dd/d12/d14/d1a/d75/f76 x:0 0 0 2024-04-12T05:37:20.815 INFO:tasks.workunit.client.0.smithi032.stdout:4/921: truncate d4/d87/d96/d2f/d3b/f3c 42106 0 2024-04-12T05:37:20.816 INFO:tasks.workunit.client.1.smithi067.stdout:4/249: dread - d8/da/d1d/d15/d1c/f52 zero size 2024-04-12T05:37:20.817 INFO:tasks.workunit.client.1.smithi067.stdout:0/377: dread - dd/d12/d14/d1a/d75/f76 zero size 2024-04-12T05:37:20.817 INFO:tasks.workunit.client.0.smithi032.stdout:0/956: creat d2/d29/dcc/f141 x:0 0 0 2024-04-12T05:37:20.818 INFO:tasks.workunit.client.0.smithi032.stdout:0/957: chown d2/d29/d8a/f108 1 1 2024-04-12T05:37:20.819 INFO:tasks.workunit.client.0.smithi032.stdout:7/926: write d0/dff/d71/f94 [3567747,73139] 0 2024-04-12T05:37:20.822 INFO:tasks.workunit.client.0.smithi032.stdout:7/927: fsync d0/de/df/f3f 0 2024-04-12T05:37:20.824 INFO:tasks.workunit.client.0.smithi032.stdout:0/958: symlink d2/d8/df/d24/d60/dc8/l142 0 2024-04-12T05:37:20.825 INFO:tasks.workunit.client.1.smithi067.stdout:4/250: dread d8/da/d1d/d15/d1c/f40 [0,4194304] 0 2024-04-12T05:37:20.826 INFO:tasks.workunit.client.0.smithi032.stdout:4/922: link d4/f18 d4/d87/d96/d4d/db5/dcf/de3/f131 0 2024-04-12T05:37:20.826 INFO:tasks.workunit.client.0.smithi032.stdout:9/795: dread d9/d1f/f5d [0,4194304] 0 2024-04-12T05:37:20.827 INFO:tasks.workunit.client.1.smithi067.stdout:1/303: dwrite d4/d9/d14/f21 [4194304,4194304] 0 2024-04-12T05:37:20.829 INFO:tasks.workunit.client.1.smithi067.stdout:5/546: mkdir d0/d6/d20/d3e/dbf 0 2024-04-12T05:37:20.829 INFO:tasks.workunit.client.0.smithi032.stdout:7/928: rmdir d0/d10/d70/d7b/dd2 39 2024-04-12T05:37:20.830 INFO:tasks.workunit.client.0.smithi032.stdout:2/920: write dd/d19/d9d/d42/da7/dc5/fd9 [140358,76719] 0 2024-04-12T05:37:20.830 INFO:tasks.workunit.client.0.smithi032.stdout:0/959: creat d2/d28/dd6/f143 x:0 0 0 2024-04-12T05:37:20.831 INFO:tasks.workunit.client.0.smithi032.stdout:1/846: write d5/de/d17/d1b/f85 [45627,98748] 0 2024-04-12T05:37:20.834 INFO:tasks.workunit.client.0.smithi032.stdout:9/796: creat d9/d15/d1a/d24/d27/d88/dc1/f110 x:0 0 0 2024-04-12T05:37:20.837 INFO:tasks.workunit.client.0.smithi032.stdout:1/847: chown d5/de/d17/d35/d61/d77/d8f/d9c/fb5 7 1 2024-04-12T05:37:20.838 INFO:tasks.workunit.client.0.smithi032.stdout:2/921: rmdir dd 39 2024-04-12T05:37:20.839 INFO:tasks.workunit.client.0.smithi032.stdout:9/797: creat d9/d32/f111 x:0 0 0 2024-04-12T05:37:20.839 INFO:tasks.workunit.client.1.smithi067.stdout:4/251: rename d8/d9/d3d/d48/d45/f47 to d8/d9/d3d/d48/f57 0 2024-04-12T05:37:20.841 INFO:tasks.workunit.client.0.smithi032.stdout:0/960: dwrite d2/d28/d87/d111/d41/fbc [0,4194304] 0 2024-04-12T05:37:20.844 INFO:tasks.workunit.client.1.smithi067.stdout:1/304: dwrite d4/d46/f5b [4194304,4194304] 0 2024-04-12T05:37:20.845 INFO:tasks.workunit.client.0.smithi032.stdout:2/922: dread dd/d19/d9d/d1c/f29 [0,4194304] 0 2024-04-12T05:37:20.846 INFO:tasks.workunit.client.0.smithi032.stdout:7/929: creat d0/de/f130 x:0 0 0 2024-04-12T05:37:20.847 INFO:tasks.workunit.client.1.smithi067.stdout:5/547: dwrite d0/da/f68 [0,4194304] 0 2024-04-12T05:37:20.849 INFO:tasks.workunit.client.0.smithi032.stdout:1/848: symlink d5/de/d17/d35/l10e 0 2024-04-12T05:37:20.849 INFO:tasks.workunit.client.0.smithi032.stdout:9/798: mknod d9/d15/d1a/d24/d27/dce/dd3/d109/c112 0 2024-04-12T05:37:20.849 INFO:tasks.workunit.client.0.smithi032.stdout:0/961: read - d2/d8/d17/fc4 zero size 2024-04-12T05:37:20.849 INFO:tasks.workunit.client.1.smithi067.stdout:1/305: chown d4/d9/l11 44636230 1 2024-04-12T05:37:20.850 INFO:tasks.workunit.client.1.smithi067.stdout:0/378: dwrite dd/d12/d14/d1a/d75/f67 [0,4194304] 0 2024-04-12T05:37:20.851 INFO:tasks.workunit.client.1.smithi067.stdout:1/306: chown d4/d9/d12/d2c 39842468 1 2024-04-12T05:37:20.853 INFO:tasks.workunit.client.1.smithi067.stdout:5/548: fsync d0/d6/d20/d3e/d69/f89 0 2024-04-12T05:37:20.857 INFO:tasks.workunit.client.0.smithi032.stdout:9/799: write d9/d15/d1a/dd2/d65/d80/fb4 [506487,107212] 0 2024-04-12T05:37:20.857 INFO:tasks.workunit.client.0.smithi032.stdout:0/962: rmdir d2/d29/d77/db5 39 2024-04-12T05:37:20.857 INFO:tasks.workunit.client.1.smithi067.stdout:5/549: dread - d0/da/d3f/d82/fb4 zero size 2024-04-12T05:37:20.863 INFO:tasks.workunit.client.0.smithi032.stdout:4/923: write d4/d87/d96/d4d/db5/f11a [90367,60906] 0 2024-04-12T05:37:20.863 INFO:tasks.workunit.client.0.smithi032.stdout:0/963: creat d2/d29/d77/dc2/f144 x:0 0 0 2024-04-12T05:37:20.865 INFO:tasks.workunit.client.0.smithi032.stdout:7/930: rename d0/f7 to d0/d10/d70/f131 0 2024-04-12T05:37:20.868 INFO:tasks.workunit.client.1.smithi067.stdout:1/307: dread d4/f5 [4194304,4194304] 0 2024-04-12T05:37:20.869 INFO:tasks.workunit.client.1.smithi067.stdout:0/379: truncate dd/d12/f4e 2255998 0 2024-04-12T05:37:20.870 INFO:tasks.workunit.client.0.smithi032.stdout:7/931: mkdir d0/dd/d51/d65/d12a/d31/d132 0 2024-04-12T05:37:20.871 INFO:tasks.workunit.client.1.smithi067.stdout:0/380: write dd/d12/d14/d51/f5a [5351162,52999] 0 2024-04-12T05:37:20.871 INFO:tasks.workunit.client.1.smithi067.stdout:0/381: chown dd/d12/f4e 0 1 2024-04-12T05:37:20.873 INFO:tasks.workunit.client.1.smithi067.stdout:1/308: creat d4/d9/d14/d61/f65 x:0 0 0 2024-04-12T05:37:20.874 INFO:tasks.workunit.client.0.smithi032.stdout:1/849: dread d5/de/d17/d35/d73/f8a [0,4194304] 0 2024-04-12T05:37:20.875 INFO:tasks.workunit.client.0.smithi032.stdout:7/932: creat d0/d13/d21/d81/f133 x:0 0 0 2024-04-12T05:37:20.876 INFO:tasks.workunit.client.0.smithi032.stdout:9/800: dwrite d9/d32/d86/dcc/f94 [0,4194304] 0 2024-04-12T05:37:20.877 INFO:tasks.workunit.client.1.smithi067.stdout:5/550: getdents d0/d6/d40/d4b/d38/d8e/d8f/db9 0 2024-04-12T05:37:20.877 INFO:tasks.workunit.client.1.smithi067.stdout:0/382: read dd/d12/d14/d1a/d3d/f2f [8850057,120408] 0 2024-04-12T05:37:20.879 INFO:tasks.workunit.client.0.smithi032.stdout:9/801: read d9/fa1 [22000,31850] 0 2024-04-12T05:37:20.879 INFO:tasks.workunit.client.1.smithi067.stdout:0/383: write dd/d12/d14/d1a/d3d/f73 [985158,74440] 0 2024-04-12T05:37:20.880 INFO:tasks.workunit.client.1.smithi067.stdout:5/551: unlink d0/da/l81 0 2024-04-12T05:37:20.881 INFO:tasks.workunit.client.0.smithi032.stdout:9/802: write d9/d32/d86/f9b [2726788,79425] 0 2024-04-12T05:37:20.881 INFO:tasks.workunit.client.1.smithi067.stdout:5/552: read d0/d3a/f4c [556910,28620] 0 2024-04-12T05:37:20.882 INFO:tasks.workunit.client.0.smithi032.stdout:7/933: symlink d0/dff/d4c/l134 0 2024-04-12T05:37:20.885 INFO:tasks.workunit.client.0.smithi032.stdout:4/924: write d4/d13/d44/f91 [3438397,106398] 0 2024-04-12T05:37:20.888 INFO:tasks.workunit.client.1.smithi067.stdout:0/384: creat dd/d12/d14/d51/f77 x:0 0 0 2024-04-12T05:37:20.889 INFO:tasks.workunit.client.0.smithi032.stdout:2/923: dwrite dd/d19/d9d/d42/da7/fb8 [0,4194304] 0 2024-04-12T05:37:20.889 INFO:tasks.workunit.client.1.smithi067.stdout:1/309: truncate d4/d9/d3a/f1e 3898411 0 2024-04-12T05:37:20.892 INFO:tasks.workunit.client.1.smithi067.stdout:5/553: dwrite d0/d6/f6a [0,4194304] 0 2024-04-12T05:37:20.894 INFO:tasks.workunit.client.1.smithi067.stdout:1/310: creat d4/d44/f66 x:0 0 0 2024-04-12T05:37:20.895 INFO:tasks.workunit.client.1.smithi067.stdout:1/311: stat d4/d9/l3f 0 2024-04-12T05:37:20.896 INFO:tasks.workunit.client.0.smithi032.stdout:0/964: dwrite d2/d8/df/f31 [4194304,4194304] 0 2024-04-12T05:37:20.897 INFO:tasks.workunit.client.1.smithi067.stdout:0/385: unlink dd/d12/d14/d1a/d75/f44 0 2024-04-12T05:37:20.898 INFO:tasks.workunit.client.1.smithi067.stdout:5/554: fsync d0/f43 0 2024-04-12T05:37:20.898 INFO:tasks.workunit.client.0.smithi032.stdout:9/803: dread d9/d1f/fe0 [0,4194304] 0 2024-04-12T05:37:20.899 INFO:tasks.workunit.client.0.smithi032.stdout:2/924: unlink dd/l105 0 2024-04-12T05:37:20.899 INFO:tasks.workunit.client.1.smithi067.stdout:5/555: write d0/da/d41/f9b [474966,40602] 0 2024-04-12T05:37:20.900 INFO:tasks.workunit.client.0.smithi032.stdout:4/925: creat d4/d87/d96/d2f/d3b/da2/dec/f132 x:0 0 0 2024-04-12T05:37:20.900 INFO:tasks.workunit.client.0.smithi032.stdout:9/804: stat d9/d32/d6e/cbf 0 2024-04-12T05:37:20.901 INFO:tasks.workunit.client.0.smithi032.stdout:0/965: chown d2/d8/df/l73 649 1 2024-04-12T05:37:20.901 INFO:tasks.workunit.client.1.smithi067.stdout:0/386: chown dd/d12/d14/d1a/d3d/d41/f50 300312 1 2024-04-12T05:37:20.901 INFO:tasks.workunit.client.1.smithi067.stdout:5/556: fsync d0/d6/d40/f7f 0 2024-04-12T05:37:20.901 INFO:tasks.workunit.client.0.smithi032.stdout:7/934: link d0/l1d d0/d34/dbe/dbb/l135 0 2024-04-12T05:37:20.903 INFO:tasks.workunit.client.0.smithi032.stdout:2/925: symlink dd/l124 0 2024-04-12T05:37:20.903 INFO:tasks.workunit.client.0.smithi032.stdout:7/935: stat d0/dd/ld0 0 2024-04-12T05:37:20.903 INFO:tasks.workunit.client.1.smithi067.stdout:5/557: fsync d0/da/d3f/d82/fac 0 2024-04-12T05:37:20.903 INFO:tasks.workunit.client.1.smithi067.stdout:5/558: read - d0/da/fbb zero size 2024-04-12T05:37:20.936 INFO:tasks.workunit.client.1.smithi067.stdout:0/387: unlink dd/d12/d14/d1a/d3d/d41/d5f/l6b 0 2024-04-12T05:37:20.939 INFO:tasks.workunit.client.0.smithi032.stdout:1/850: write d5/de/d17/d35/d78/f79 [3668114,112590] 0 2024-04-12T05:37:20.941 INFO:tasks.workunit.client.0.smithi032.stdout:4/926: unlink d4/l66 0 2024-04-12T05:37:20.942 INFO:tasks.workunit.client.1.smithi067.stdout:0/388: read dd/d12/f2e [377012,130473] 0 2024-04-12T05:37:20.942 INFO:tasks.workunit.client.0.smithi032.stdout:7/936: rmdir d0/de/df/d29 39 2024-04-12T05:37:20.950 INFO:tasks.workunit.client.0.smithi032.stdout:4/927: fsync d4/d87/d96/d4d/dbc/fb3 0 2024-04-12T05:37:20.951 INFO:tasks.workunit.client.0.smithi032.stdout:7/937: symlink d0/d10/l136 0 2024-04-12T05:37:20.952 INFO:tasks.workunit.client.1.smithi067.stdout:0/389: mkdir dd/d12/d14/d1a/d3d/d41/d5f/d78 0 2024-04-12T05:37:20.953 INFO:tasks.workunit.client.0.smithi032.stdout:2/926: creat dd/d19/d9d/d1c/d64/d92/f125 x:0 0 0 2024-04-12T05:37:20.953 INFO:tasks.workunit.client.1.smithi067.stdout:0/390: read - dd/d12/d14/d1a/d75/f76 zero size 2024-04-12T05:37:20.956 INFO:tasks.workunit.client.0.smithi032.stdout:0/966: write d2/d29/d8a/fd1 [489787,53659] 0 2024-04-12T05:37:20.957 INFO:tasks.workunit.client.0.smithi032.stdout:2/927: truncate dd/d19/d9d/d1c/de0/f11f 213684 0 2024-04-12T05:37:20.959 INFO:tasks.workunit.client.1.smithi067.stdout:0/391: rename dd/d12/d14/d1a/l1d to dd/d12/d14/d49/l79 0 2024-04-12T05:37:20.959 INFO:tasks.workunit.client.0.smithi032.stdout:4/928: unlink d4/d87/d96/d32/l12d 0 2024-04-12T05:37:20.960 INFO:tasks.workunit.client.0.smithi032.stdout:0/967: write d2/d29/d8a/d99/fbf [4584816,114230] 0 2024-04-12T05:37:20.960 INFO:tasks.workunit.client.0.smithi032.stdout:7/938: dwrite d0/d13/f8c [0,4194304] 0 2024-04-12T05:37:20.961 INFO:tasks.workunit.client.0.smithi032.stdout:1/851: write d5/de/d17/f26 [2551159,113101] 0 2024-04-12T05:37:20.963 INFO:tasks.workunit.client.0.smithi032.stdout:9/805: getdents d9/d15/d1a/d24/d27/d88 0 2024-04-12T05:37:20.963 INFO:tasks.workunit.client.0.smithi032.stdout:0/968: read - d2/d8/d17/fc4 zero size 2024-04-12T05:37:20.964 INFO:tasks.workunit.client.0.smithi032.stdout:4/929: chown d4/d87/d96/d4d/d78/dba/dfc/c128 3570 1 2024-04-12T05:37:20.968 INFO:tasks.workunit.client.0.smithi032.stdout:4/930: read d4/d87/d96/d4d/db5/f11a [3063953,62514] 0 2024-04-12T05:37:20.970 INFO:tasks.workunit.client.1.smithi067.stdout:0/392: dwrite dd/d12/f2e [0,4194304] 0 2024-04-12T05:37:20.970 INFO:tasks.workunit.client.0.smithi032.stdout:4/931: chown d4/d87/d96/d32/c89 676703149 1 2024-04-12T05:37:20.975 INFO:tasks.workunit.client.0.smithi032.stdout:9/806: dwrite d9/d15/d2c/d41/fc8 [0,4194304] 0 2024-04-12T05:37:20.978 INFO:tasks.workunit.client.1.smithi067.stdout:0/393: symlink dd/d12/d14/d51/l7a 0 2024-04-12T05:37:20.979 INFO:tasks.workunit.client.0.smithi032.stdout:2/928: rename dd/d19/d9d/d1c/d64/d32/d6d/da5/lc8 to dd/d118/l126 0 2024-04-12T05:37:20.990 INFO:tasks.workunit.client.0.smithi032.stdout:0/969: getdents d2/d29/d77/dc7 0 2024-04-12T05:37:20.991 INFO:tasks.workunit.client.1.smithi067.stdout:5/559: dwrite d0/d3a/d2c/f37 [0,4194304] 0 2024-04-12T05:37:20.992 INFO:tasks.workunit.client.1.smithi067.stdout:0/394: write dd/d12/d14/d1a/d3d/f2f [11015572,45855] 0 2024-04-12T05:37:20.992 INFO:tasks.workunit.client.0.smithi032.stdout:7/939: rename d0/de/df/d29/f7d to d0/d10/d70/d7b/f137 0 2024-04-12T05:37:20.994 INFO:tasks.workunit.client.0.smithi032.stdout:4/932: mknod d4/d87/d96/d2f/d7c/dff/c133 0 2024-04-12T05:37:20.997 INFO:tasks.workunit.client.1.smithi067.stdout:0/395: rmdir dd/d5b 39 2024-04-12T05:37:21.001 INFO:tasks.workunit.client.0.smithi032.stdout:0/970: unlink d2/d8/d17/f2b 0 2024-04-12T05:37:21.003 INFO:tasks.workunit.client.0.smithi032.stdout:2/929: rename dd/d19/d9d/d1c/d64/d32/d3a/c7a to dd/d19/d9d/d1c/c127 0 2024-04-12T05:37:21.003 INFO:tasks.workunit.client.1.smithi067.stdout:1/312: dread d4/d9/d3a/f1e [0,4194304] 0 2024-04-12T05:37:21.005 INFO:tasks.workunit.client.0.smithi032.stdout:0/971: rmdir d2/d8/df/d24/dc3 39 2024-04-12T05:37:21.005 INFO:tasks.workunit.client.0.smithi032.stdout:2/930: chown dd/d19/d9d/d42/d4a 1 1 2024-04-12T05:37:21.006 INFO:tasks.workunit.client.1.smithi067.stdout:5/560: dwrite d0/d3a/d2c/f37 [4194304,4194304] 0 2024-04-12T05:37:21.006 INFO:tasks.workunit.client.1.smithi067.stdout:5/561: fsync d0/da/d41/f6b 0 2024-04-12T05:37:21.007 INFO:tasks.workunit.client.1.smithi067.stdout:0/396: dread dd/d12/d14/f43 [0,4194304] 0 2024-04-12T05:37:21.007 INFO:tasks.workunit.client.1.smithi067.stdout:5/562: stat d0/d3a/d2c/d91/c29 0 2024-04-12T05:37:21.008 INFO:tasks.workunit.client.1.smithi067.stdout:1/313: mknod d4/d9/d3a/d51/d55/c67 0 2024-04-12T05:37:21.010 INFO:tasks.workunit.client.0.smithi032.stdout:4/933: rename d4/d87/d96/d2f/d7c/dc2/ld0 to d4/d13/dc9/d100/l134 0 2024-04-12T05:37:21.013 INFO:tasks.workunit.client.1.smithi067.stdout:0/397: symlink dd/d12/d14/d51/l7b 0 2024-04-12T05:37:21.014 INFO:tasks.workunit.client.1.smithi067.stdout:5/563: write d0/d6/d40/f7f [1221521,58074] 0 2024-04-12T05:37:21.018 INFO:tasks.workunit.client.0.smithi032.stdout:4/934: rename d4/d13/d75/fbd to d4/d87/d96/d4d/f135 0 2024-04-12T05:37:21.018 INFO:tasks.workunit.client.1.smithi067.stdout:5/564: creat d0/da/d3f/d82/fc0 x:0 0 0 2024-04-12T05:37:21.021 INFO:tasks.workunit.client.0.smithi032.stdout:2/931: readlink dd/d118/l126 0 2024-04-12T05:37:21.025 INFO:tasks.workunit.client.0.smithi032.stdout:1/852: dwrite d5/de/d17/d35/d3b/d49/fd9 [4194304,4194304] 0 2024-04-12T05:37:21.025 INFO:tasks.workunit.client.1.smithi067.stdout:0/398: dwrite dd/d12/f59 [0,4194304] 0 2024-04-12T05:37:21.029 INFO:tasks.workunit.client.0.smithi032.stdout:4/935: creat d4/d87/d96/d2f/d3b/da2/d10a/f136 x:0 0 0 2024-04-12T05:37:21.029 INFO:tasks.workunit.client.0.smithi032.stdout:9/807: dread d9/d15/d1a/d24/d27/d88/dc1/fdb [4194304,4194304] 0 2024-04-12T05:37:21.033 INFO:tasks.workunit.client.1.smithi067.stdout:0/399: rmdir dd/d12/d14/d1a/d3d 39 2024-04-12T05:37:21.035 INFO:tasks.workunit.client.0.smithi032.stdout:9/808: write d9/d15/d2c/d41/fc8 [5217068,34275] 0 2024-04-12T05:37:21.038 INFO:tasks.workunit.client.0.smithi032.stdout:0/972: truncate d2/d8/d17/d1b/de7/f117 11113 0 2024-04-12T05:37:21.039 INFO:tasks.workunit.client.1.smithi067.stdout:1/314: dread d4/d9/d12/f36 [0,4194304] 0 2024-04-12T05:37:21.039 INFO:tasks.workunit.client.0.smithi032.stdout:7/940: dwrite d0/f5 [0,4194304] 0 2024-04-12T05:37:21.040 INFO:tasks.workunit.client.1.smithi067.stdout:0/400: readlink dd/d12/d14/d1a/d3d/l46 0 2024-04-12T05:37:21.040 INFO:tasks.workunit.client.0.smithi032.stdout:7/941: dread - d0/d13/d21/d81/f124 zero size 2024-04-12T05:37:21.041 INFO:tasks.workunit.client.0.smithi032.stdout:0/973: rmdir d2/d29 39 2024-04-12T05:37:21.043 INFO:tasks.workunit.client.1.smithi067.stdout:5/565: dwrite d0/d6/d40/d4b/d38/d8e/d8f/d99/f9a [0,4194304] 0 2024-04-12T05:37:21.044 INFO:tasks.workunit.client.0.smithi032.stdout:7/942: symlink d0/d34/l138 0 2024-04-12T05:37:21.045 INFO:tasks.workunit.client.0.smithi032.stdout:2/932: dwrite dd/d19/d20/f62 [4194304,4194304] 0 2024-04-12T05:37:21.047 INFO:tasks.workunit.client.1.smithi067.stdout:1/315: creat d4/d9/f68 x:0 0 0 2024-04-12T05:37:21.048 INFO:tasks.workunit.client.0.smithi032.stdout:9/809: write d9/d15/d1a/d1d/d5a/d6b/f6c [2453638,68321] 0 2024-04-12T05:37:21.050 INFO:tasks.workunit.client.1.smithi067.stdout:0/401: dread dd/d12/d14/d1a/f1e [0,4194304] 0 2024-04-12T05:37:21.055 INFO:tasks.workunit.client.0.smithi032.stdout:9/810: symlink d9/d15/d1a/d24/l113 0 2024-04-12T05:37:21.056 INFO:tasks.workunit.client.1.smithi067.stdout:5/566: mknod d0/d6/dae/cc1 0 2024-04-12T05:37:21.056 INFO:tasks.workunit.client.1.smithi067.stdout:1/316: mknod d4/d9/d4d/c69 0 2024-04-12T05:37:21.057 INFO:tasks.workunit.client.0.smithi032.stdout:2/933: rename dd/d19/d9d/d1c/f76 to dd/d19/d9d/d1c/d64/d32/f128 0 2024-04-12T05:37:21.059 INFO:tasks.workunit.client.0.smithi032.stdout:0/974: getdents d2/d28/d87/d111/d3e/dac/d51 0 2024-04-12T05:37:21.059 INFO:tasks.workunit.client.1.smithi067.stdout:5/567: truncate d0/da/d3f/d82/fb4 100475 0 2024-04-12T05:37:21.062 INFO:tasks.workunit.client.0.smithi032.stdout:0/975: unlink d2/d8/l4f 0 2024-04-12T05:37:21.062 INFO:tasks.workunit.client.0.smithi032.stdout:7/943: dread d0/dd/d51/d65/d12a/d31/fbd [0,4194304] 0 2024-04-12T05:37:21.063 INFO:tasks.workunit.client.0.smithi032.stdout:1/853: dread d5/de/f1f [0,4194304] 0 2024-04-12T05:37:21.063 INFO:tasks.workunit.client.0.smithi032.stdout:2/934: mknod dd/d19/d9d/d1c/d64/d32/d6d/da5/c129 0 2024-04-12T05:37:21.065 INFO:tasks.workunit.client.1.smithi067.stdout:5/568: fdatasync d0/d6/d20/d62/f15 0 2024-04-12T05:37:21.067 INFO:tasks.workunit.client.0.smithi032.stdout:7/944: fsync d0/d10/d70/d7b/fc9 0 2024-04-12T05:37:21.068 INFO:tasks.workunit.client.0.smithi032.stdout:1/854: write d5/de/f10 [2867008,6588] 0 2024-04-12T05:37:21.069 INFO:tasks.workunit.client.1.smithi067.stdout:0/402: creat dd/d12/d14/d1a/d3d/f7c x:0 0 0 2024-04-12T05:37:21.072 INFO:tasks.workunit.client.1.smithi067.stdout:0/403: mknod dd/d12/d14/d1a/c7d 0 2024-04-12T05:37:21.074 INFO:tasks.workunit.client.0.smithi032.stdout:1/855: truncate d5/de/d5b/fe7 594688 0 2024-04-12T05:37:21.074 INFO:tasks.workunit.client.0.smithi032.stdout:0/976: dwrite d2/d28/d87/d79/fce [0,4194304] 0 2024-04-12T05:37:21.076 INFO:tasks.workunit.client.0.smithi032.stdout:2/935: dwrite dd/d19/d20/f62 [0,4194304] 0 2024-04-12T05:37:21.083 INFO:tasks.workunit.client.1.smithi067.stdout:1/317: truncate d4/d9/d12/d28/f2b 863699 0 2024-04-12T05:37:21.084 INFO:tasks.workunit.client.1.smithi067.stdout:1/318: chown d4/d9/f4a 255540050 1 2024-04-12T05:37:21.085 INFO:tasks.workunit.client.1.smithi067.stdout:5/569: truncate d0/d6/f6a 1071475 0 2024-04-12T05:37:21.085 INFO:tasks.workunit.client.0.smithi032.stdout:0/977: creat d2/d8/d17/d1b/de7/f145 x:0 0 0 2024-04-12T05:37:21.085 INFO:tasks.workunit.client.0.smithi032.stdout:2/936: symlink dd/d19/d9d/d1c/de0/de1/l12a 0 2024-04-12T05:37:21.087 INFO:tasks.workunit.client.0.smithi032.stdout:7/945: link d0/d34/dbe/dbb/l135 d0/d13/d21/l139 0 2024-04-12T05:37:21.088 INFO:tasks.workunit.client.0.smithi032.stdout:7/946: write d0/de/df/f14 [2462820,12779] 0 2024-04-12T05:37:21.088 INFO:tasks.workunit.client.1.smithi067.stdout:5/570: truncate d0/d6/d33/f9d 156579 0 2024-04-12T05:37:21.088 INFO:tasks.workunit.client.0.smithi032.stdout:2/937: creat dd/f12b x:0 0 0 2024-04-12T05:37:21.092 INFO:tasks.workunit.client.0.smithi032.stdout:7/947: mknod d0/d13/d21/d81/c13a 0 2024-04-12T05:37:21.093 INFO:tasks.workunit.client.0.smithi032.stdout:2/938: symlink dd/d19/d9d/d42/d4a/da1/l12c 0 2024-04-12T05:37:21.098 INFO:tasks.workunit.client.0.smithi032.stdout:0/978: getdents d2/d28/d87 0 2024-04-12T05:37:21.099 INFO:tasks.workunit.client.1.smithi067.stdout:0/404: dread dd/f13 [0,4194304] 0 2024-04-12T05:37:21.099 INFO:tasks.workunit.client.1.smithi067.stdout:5/571: rename d0/d6/f2b to d0/d6/d40/fc2 0 2024-04-12T05:37:21.099 INFO:tasks.workunit.client.1.smithi067.stdout:0/405: readlink dd/d12/l36 0 2024-04-12T05:37:21.102 INFO:tasks.workunit.client.0.smithi032.stdout:0/979: symlink d2/d8/df/d24/l146 0 2024-04-12T05:37:21.103 INFO:tasks.workunit.client.0.smithi032.stdout:9/811: dread d9/d15/f28 [0,4194304] 0 2024-04-12T05:37:21.103 INFO:tasks.workunit.client.0.smithi032.stdout:0/980: chown d2/d28/d87/d111/d3e/dac/f54 887 1 2024-04-12T05:37:21.104 INFO:tasks.workunit.client.1.smithi067.stdout:0/406: rmdir dd/d12 39 2024-04-12T05:37:21.109 INFO:tasks.workunit.client.0.smithi032.stdout:9/812: rename d9/d15/d2c/dee to d9/d32/d6e/d114 0 2024-04-12T05:37:21.110 INFO:tasks.workunit.client.1.smithi067.stdout:1/319: dread d4/d3d/f4b [0,4194304] 0 2024-04-12T05:37:21.110 INFO:tasks.workunit.client.0.smithi032.stdout:0/981: symlink d2/d29/d8a/d99/l147 0 2024-04-12T05:37:21.111 INFO:tasks.workunit.client.1.smithi067.stdout:0/407: creat dd/d12/d14/d1a/d3d/d41/d64/f7e x:0 0 0 2024-04-12T05:37:21.112 INFO:tasks.workunit.client.1.smithi067.stdout:1/320: symlink d4/d9/d12/d2c/l6a 0 2024-04-12T05:37:21.114 INFO:tasks.workunit.client.1.smithi067.stdout:0/408: symlink dd/d12/d14/d1a/d3d/d41/l7f 0 2024-04-12T05:37:21.114 INFO:tasks.workunit.client.0.smithi032.stdout:9/813: symlink d9/d15/d1a/dd2/d65/d80/d87/l115 0 2024-04-12T05:37:21.114 INFO:tasks.workunit.client.1.smithi067.stdout:1/321: write d4/d44/f66 [278416,4409] 0 2024-04-12T05:37:21.117 INFO:tasks.workunit.client.1.smithi067.stdout:0/409: symlink dd/d6f/l80 0 2024-04-12T05:37:21.117 INFO:tasks.workunit.client.0.smithi032.stdout:9/814: symlink d9/d15/d1a/d1d/d5a/d6b/d83/l116 0 2024-04-12T05:37:21.119 INFO:tasks.workunit.client.1.smithi067.stdout:5/572: dwrite d0/d6/d33/f52 [0,4194304] 0 2024-04-12T05:37:21.121 INFO:tasks.workunit.client.1.smithi067.stdout:1/322: rmdir d4/d9/d14/d61 39 2024-04-12T05:37:21.122 INFO:tasks.workunit.client.0.smithi032.stdout:9/815: mknod d9/d15/d1a/d24/d27/c117 0 2024-04-12T05:37:21.128 INFO:tasks.workunit.client.0.smithi032.stdout:2/939: dwrite dd/d19/d9d/d1c/d64/d32/d3a/f56 [0,4194304] 0 2024-04-12T05:37:21.128 INFO:tasks.workunit.client.0.smithi032.stdout:9/816: unlink d9/d15/d1a/d1d/f6f 0 2024-04-12T05:37:21.128 INFO:tasks.workunit.client.0.smithi032.stdout:7/948: dwrite d0/dff/fc6 [0,4194304] 0 2024-04-12T05:37:21.129 INFO:tasks.workunit.client.1.smithi067.stdout:5/573: mknod d0/d6/d20/d3e/cc3 0 2024-04-12T05:37:21.129 INFO:tasks.workunit.client.1.smithi067.stdout:1/323: mknod d4/d9/d14/d61/c6b 0 2024-04-12T05:37:21.129 INFO:tasks.workunit.client.1.smithi067.stdout:0/410: creat dd/d12/d14/d1a/d3d/f81 x:0 0 0 2024-04-12T05:37:21.129 INFO:tasks.workunit.client.1.smithi067.stdout:5/574: write d0/da/fbb [402864,100228] 0 2024-04-12T05:37:21.132 INFO:tasks.workunit.client.1.smithi067.stdout:1/324: truncate d4/d9/f1c 1325289 0 2024-04-12T05:37:21.132 INFO:tasks.workunit.client.1.smithi067.stdout:5/575: dread - d0/d6/d40/d4b/d38/d8e/f96 zero size 2024-04-12T05:37:21.132 INFO:tasks.workunit.client.0.smithi032.stdout:7/949: write d0/d13/f27 [2584704,47007] 0 2024-04-12T05:37:21.135 INFO:tasks.workunit.client.1.smithi067.stdout:0/411: dread - dd/d12/f3b zero size 2024-04-12T05:37:21.137 INFO:tasks.workunit.client.1.smithi067.stdout:5/576: symlink d0/da/d3f/daa/lc4 0 2024-04-12T05:37:21.138 INFO:tasks.workunit.client.0.smithi032.stdout:9/817: mknod d9/d15/d1a/d24/d27/dce/c118 0 2024-04-12T05:37:21.138 INFO:tasks.workunit.client.1.smithi067.stdout:6/376: sync 2024-04-12T05:37:21.138 INFO:tasks.workunit.client.1.smithi067.stdout:2/357: sync 2024-04-12T05:37:21.139 INFO:tasks.workunit.client.1.smithi067.stdout:9/271: sync 2024-04-12T05:37:21.139 INFO:tasks.workunit.client.1.smithi067.stdout:7/332: sync 2024-04-12T05:37:21.139 INFO:tasks.workunit.client.1.smithi067.stdout:3/237: sync 2024-04-12T05:37:21.142 INFO:tasks.workunit.client.0.smithi032.stdout:7/950: mkdir d0/dd/d51/d65/d12a/d31/d13b 0 2024-04-12T05:37:21.143 INFO:tasks.workunit.client.0.smithi032.stdout:9/818: symlink d9/d32/d86/l119 0 2024-04-12T05:37:21.144 INFO:tasks.workunit.client.0.smithi032.stdout:0/982: dread d2/d8/d17/d1b/d20/f45 [0,4194304] 0 2024-04-12T05:37:21.144 INFO:tasks.workunit.client.0.smithi032.stdout:2/940: dwrite dd/d19/d9d/d42/d4a/fb3 [0,4194304] 0 2024-04-12T05:37:21.144 INFO:tasks.workunit.client.1.smithi067.stdout:2/358: fdatasync d5/d11/f2a 0 2024-04-12T05:37:21.145 INFO:tasks.workunit.client.1.smithi067.stdout:1/325: rename d4/d9/f31 to d4/d9/d12/d2c/f6c 0 2024-04-12T05:37:21.149 INFO:tasks.workunit.client.1.smithi067.stdout:7/333: mknod d0/d2/d5/d1e/d51/c6c 0 2024-04-12T05:37:21.150 INFO:tasks.workunit.client.0.smithi032.stdout:2/941: readlink dd/d19/l2b 0 2024-04-12T05:37:21.152 INFO:tasks.workunit.client.1.smithi067.stdout:5/577: rename d0/d3a/l5 to d0/d3a/dab/lc5 0 2024-04-12T05:37:21.154 INFO:tasks.workunit.client.0.smithi032.stdout:9/819: symlink d9/d15/d2c/d41/l11a 0 2024-04-12T05:37:21.156 INFO:tasks.workunit.client.1.smithi067.stdout:6/377: dread dd/d30/f32 [0,4194304] 0 2024-04-12T05:37:21.159 INFO:tasks.workunit.client.1.smithi067.stdout:1/326: unlink d4/d9/d12/c40 0 2024-04-12T05:37:21.159 INFO:tasks.workunit.client.0.smithi032.stdout:0/983: mknod d2/c148 0 2024-04-12T05:37:21.160 INFO:tasks.workunit.client.0.smithi032.stdout:7/951: dwrite d0/dff/fc6 [4194304,4194304] 0 2024-04-12T05:37:21.160 INFO:tasks.workunit.client.1.smithi067.stdout:2/359: dread d5/da/de/f52 [4194304,4194304] 0 2024-04-12T05:37:21.161 INFO:tasks.workunit.client.1.smithi067.stdout:2/360: fsync d5/d11/f2a 0 2024-04-12T05:37:21.163 INFO:tasks.workunit.client.1.smithi067.stdout:0/412: dwrite dd/d12/d14/d1a/d75/f4c [0,4194304] 0 2024-04-12T05:37:21.164 INFO:tasks.workunit.client.0.smithi032.stdout:0/984: creat d2/d29/dcc/f149 x:0 0 0 2024-04-12T05:37:21.164 INFO:tasks.workunit.client.0.smithi032.stdout:9/820: mknod d9/d32/d86/dcc/c11b 0 2024-04-12T05:37:21.166 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:21 smithi067 ceph-mon[49010]: Standby manager daemon smithi067.dwichg started 2024-04-12T05:37:21.167 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:21 smithi067 ceph-mon[49010]: from='mgr.? 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi067.dwichg/crt"}]: dispatch 2024-04-12T05:37:21.167 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:21 smithi067 ceph-mon[49010]: from='mgr.? 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T05:37:21.167 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:21 smithi067 ceph-mon[49010]: from='mgr.? 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi067.dwichg/key"}]: dispatch 2024-04-12T05:37:21.167 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:21 smithi067 ceph-mon[49010]: from='mgr.? 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T05:37:21.167 INFO:tasks.workunit.client.0.smithi032.stdout:2/942: dwrite dd/d19/d9d/d42/d4a/f113 [0,4194304] 0 2024-04-12T05:37:21.169 INFO:tasks.workunit.client.0.smithi032.stdout:2/943: truncate dd/d19/d9d/d42/da7/dc5/fd9 752498 0 2024-04-12T05:37:21.171 INFO:tasks.workunit.client.0.smithi032.stdout:0/985: mknod d2/d8/df/d24/d60/dc8/c14a 0 2024-04-12T05:37:21.173 INFO:tasks.workunit.client.0.smithi032.stdout:0/986: dread - d2/d8/d119/f135 zero size 2024-04-12T05:37:21.176 INFO:tasks.workunit.client.0.smithi032.stdout:9/821: rename d9/d15/d2c/f62 to d9/df7/f11c 0 2024-04-12T05:37:21.176 INFO:tasks.workunit.client.0.smithi032.stdout:0/987: mknod d2/d29/dcc/c14b 0 2024-04-12T05:37:21.181 INFO:tasks.workunit.client.0.smithi032.stdout:0/988: mknod d2/d28/d87/d111/d3e/c14c 0 2024-04-12T05:37:21.182 INFO:tasks.workunit.client.0.smithi032.stdout:9/822: truncate d9/d32/d6e/f8e 221561 0 2024-04-12T05:37:21.184 INFO:tasks.workunit.client.0.smithi032.stdout:9/823: write d9/d32/d86/dcc/ff0 [1352528,80610] 0 2024-04-12T05:37:21.185 INFO:tasks.workunit.client.0.smithi032.stdout:9/824: chown d9/d1f/cb3 29833075 1 2024-04-12T05:37:21.186 INFO:tasks.workunit.client.0.smithi032.stdout:0/989: rename d2/d28/d87/d111/cca to d2/d8/d17/d1b/de7/c14d 0 2024-04-12T05:37:21.188 INFO:tasks.workunit.client.0.smithi032.stdout:0/990: fsync d2/d8/feb 0 2024-04-12T05:37:21.190 INFO:tasks.workunit.client.0.smithi032.stdout:9/825: unlink d9/d1f/c10d 0 2024-04-12T05:37:21.192 INFO:tasks.workunit.client.0.smithi032.stdout:0/991: creat d2/d8/df/d24/dc3/f14e x:0 0 0 2024-04-12T05:37:21.193 INFO:tasks.workunit.client.0.smithi032.stdout:9/826: truncate d9/d15/d1a/dd2/d65/fb7 1977657 0 2024-04-12T05:37:21.194 INFO:tasks.workunit.client.0.smithi032.stdout:0/992: getdents d2/d28/d87/d79/d11c 0 2024-04-12T05:37:21.196 INFO:tasks.workunit.client.0.smithi032.stdout:9/827: creat d9/d15/d1a/d1d/d5a/d6b/d83/f11d x:0 0 0 2024-04-12T05:37:21.196 INFO:tasks.workunit.client.0.smithi032.stdout:9/828: symlink d9/d58/l11e 2 2024-04-12T05:37:21.198 INFO:tasks.workunit.client.0.smithi032.stdout:0/993: dread d2/d8/d17/f6a [0,4194304] 0 2024-04-12T05:37:21.199 INFO:tasks.workunit.client.0.smithi032.stdout:0/994: fsync d2/d8/d17/d64/fa5 0 2024-04-12T05:37:21.201 INFO:tasks.workunit.client.0.smithi032.stdout:9/829: dread d9/d15/d2c/f77 [0,4194304] 0 2024-04-12T05:37:21.201 INFO:tasks.workunit.client.0.smithi032.stdout:7/952: dread d0/d10/d70/f8f [0,4194304] 0 2024-04-12T05:37:21.202 INFO:tasks.workunit.client.0.smithi032.stdout:0/995: mkdir d2/d29/d77/dc7/d112/d14f 0 2024-04-12T05:37:21.203 INFO:tasks.workunit.client.0.smithi032.stdout:9/830: mkdir d9/d58/d11f 0 2024-04-12T05:37:21.204 INFO:tasks.workunit.client.0.smithi032.stdout:7/953: mknod d0/dff/d4c/c13c 0 2024-04-12T05:37:21.205 INFO:tasks.workunit.client.0.smithi032.stdout:0/996: creat d2/d28/dd6/d13d/f150 x:0 0 0 2024-04-12T05:37:21.206 INFO:tasks.workunit.client.0.smithi032.stdout:9/831: symlink d9/d32/d86/l120 0 2024-04-12T05:37:21.207 INFO:tasks.workunit.client.0.smithi032.stdout:7/954: mknod d0/d34/dbe/dbb/de8/df1/c13d 0 2024-04-12T05:37:21.209 INFO:tasks.workunit.client.1.smithi067.stdout:7/334: creat d0/d67/d5c/f6d x:0 0 0 2024-04-12T05:37:21.209 INFO:tasks.workunit.client.0.smithi032.stdout:0/997: rename d2/d8/df/f5f to d2/d29/d77/dc2/d12e/f151 0 2024-04-12T05:37:21.210 INFO:tasks.workunit.client.1.smithi067.stdout:5/578: mkdir d0/d3a/d2c/dc6 0 2024-04-12T05:37:21.210 INFO:tasks.workunit.client.0.smithi032.stdout:0/998: write d2/d8/df/d24/dc3/f14e [700427,130225] 0 2024-04-12T05:37:21.211 INFO:tasks.workunit.client.0.smithi032.stdout:7/955: symlink d0/dd/d51/d65/d12a/dfb/l13e 0 2024-04-12T05:37:21.211 INFO:tasks.workunit.client.0.smithi032.stdout:9/832: truncate d9/d15/d1a/d24/f4b 452669 0 2024-04-12T05:37:21.212 INFO:tasks.workunit.client.1.smithi067.stdout:9/272: rmdir d1/d28/d27/d32/d4e 0 2024-04-12T05:37:21.216 INFO:tasks.workunit.client.1.smithi067.stdout:6/378: rename dd/d21/d63 to dd/d30/d7b 0 2024-04-12T05:37:21.220 INFO:tasks.workunit.client.1.smithi067.stdout:7/335: rmdir d0/d23/d45 39 2024-04-12T05:37:21.222 INFO:tasks.workunit.client.0.smithi032.stdout:0/999: dwrite d2/d8/df/d24/dc3/ff9 [0,4194304] 0 2024-04-12T05:37:21.223 INFO:tasks.workunit.client.1.smithi067.stdout:5/579: write d0/d6/d40/fc2 [380556,98802] 0 2024-04-12T05:37:21.224 INFO:tasks.workunit.client.1.smithi067.stdout:3/238: getdents dc/df/d27 0 2024-04-12T05:37:21.226 INFO:tasks.workunit.client.1.smithi067.stdout:3/239: truncate dc/d1d/d28/f38 1371406 0 2024-04-12T05:37:21.229 INFO:tasks.workunit.client.1.smithi067.stdout:1/327: rename d4/f10 to d4/d9/d14/d61/f6d 0 2024-04-12T05:37:21.232 INFO:tasks.workunit.client.0.smithi032.stdout:7/956: dwrite d0/d10/d70/d7b/dd2/ff7 [0,4194304] 0 2024-04-12T05:37:21.233 INFO:tasks.workunit.client.0.smithi032.stdout:7/957: write d0/f5 [6324302,49095] 0 2024-04-12T05:37:21.235 INFO:tasks.workunit.client.1.smithi067.stdout:6/379: fdatasync dd/d11/d27/d4a/d79/f68 0 2024-04-12T05:37:21.235 INFO:tasks.workunit.client.1.smithi067.stdout:5/580: dwrite d0/d6/f6e [0,4194304] 0 2024-04-12T05:37:21.235 INFO:tasks.workunit.client.1.smithi067.stdout:0/413: creat dd/d6f/d72/f82 x:0 0 0 2024-04-12T05:37:21.239 INFO:tasks.workunit.client.0.smithi032.stdout:7/958: creat d0/d34/dbe/dbb/f13f x:0 0 0 2024-04-12T05:37:21.239 INFO:tasks.workunit.client.1.smithi067.stdout:6/380: chown dd/d21/d23/f42 93849730 1 2024-04-12T05:37:21.240 INFO:tasks.workunit.client.1.smithi067.stdout:3/240: dwrite dc/df/f2c [4194304,4194304] 0 2024-04-12T05:37:21.240 INFO:tasks.workunit.client.1.smithi067.stdout:5/581: stat d0/da/d3f/daa/lc4 0 2024-04-12T05:37:21.241 INFO:tasks.workunit.client.1.smithi067.stdout:0/414: write dd/d12/d14/d1a/f2b [1412829,111786] 0 2024-04-12T05:37:21.242 INFO:tasks.workunit.client.1.smithi067.stdout:5/582: chown d0/d3a/l51 4670599 1 2024-04-12T05:37:21.248 INFO:tasks.workunit.client.0.smithi032.stdout:9/833: dread d9/f34 [0,4194304] 0 2024-04-12T05:37:21.249 INFO:tasks.workunit.client.1.smithi067.stdout:2/361: link d5/da/d3d/l42 d5/d23/d3a/l70 0 2024-04-12T05:37:21.250 INFO:tasks.workunit.client.1.smithi067.stdout:9/273: link d1/d13/l3b d1/d28/d23/l51 0 2024-04-12T05:37:21.250 INFO:tasks.workunit.client.1.smithi067.stdout:6/381: dwrite dd/d21/d23/f43 [0,4194304] 0 2024-04-12T05:37:21.252 INFO:tasks.workunit.client.1.smithi067.stdout:9/274: chown d1/d12/d15 61 1 2024-04-12T05:37:21.253 INFO:tasks.workunit.client.1.smithi067.stdout:3/241: stat dc/f18 0 2024-04-12T05:37:21.254 INFO:tasks.workunit.client.1.smithi067.stdout:5/583: dwrite d0/d6/d33/f52 [0,4194304] 0 2024-04-12T05:37:21.254 INFO:tasks.workunit.client.0.smithi032.stdout:7/959: dwrite d0/de/df/d29/f89 [0,4194304] 0 2024-04-12T05:37:21.255 INFO:tasks.workunit.client.0.smithi032.stdout:9/834: dread - d9/d15/d1a/d24/d27/d88/dc1/fd8 zero size 2024-04-12T05:37:21.258 INFO:tasks.workunit.client.1.smithi067.stdout:7/336: creat d0/d23/f6e x:0 0 0 2024-04-12T05:37:21.258 INFO:tasks.workunit.client.0.smithi032.stdout:7/960: dread - d0/dd/d51/d65/d12a/fd3 zero size 2024-04-12T05:37:21.259 INFO:tasks.workunit.client.1.smithi067.stdout:7/337: fdatasync d0/d2/f52 0 2024-04-12T05:37:21.259 INFO:tasks.workunit.client.0.smithi032.stdout:9/835: rename d9/d15/d1a to d9/d15/d1a/d24/d27/d88/dd9/d121 22 2024-04-12T05:37:21.260 INFO:tasks.workunit.client.1.smithi067.stdout:7/338: rename d0/d67 to d0/d67/d6f 22 2024-04-12T05:37:21.264 INFO:tasks.workunit.client.1.smithi067.stdout:2/362: unlink d5/d6a/f64 0 2024-04-12T05:37:21.264 INFO:tasks.workunit.client.1.smithi067.stdout:0/415: mkdir dd/d12/d14/d1a/d3d/d41/d5f/d78/d83 0 2024-04-12T05:37:21.264 INFO:tasks.workunit.client.1.smithi067.stdout:6/382: mknod dd/d21/d23/d56/c7c 0 2024-04-12T05:37:21.264 INFO:tasks.workunit.client.1.smithi067.stdout:3/242: mknod dc/d1d/d26/c47 0 2024-04-12T05:37:21.266 INFO:tasks.workunit.client.1.smithi067.stdout:3/243: read - dc/df/d27/f3f zero size 2024-04-12T05:37:21.267 INFO:tasks.workunit.client.1.smithi067.stdout:1/328: getdents d4/d9/d12/d28 0 2024-04-12T05:37:21.268 INFO:tasks.workunit.client.1.smithi067.stdout:7/339: chown d0/d2/d5/db/f1f 66824 1 2024-04-12T05:37:21.270 INFO:tasks.workunit.client.0.smithi032.stdout:7/961: link d0/fc4 d0/de/d47/de1/f140 0 2024-04-12T05:37:21.276 INFO:tasks.workunit.client.0.smithi032.stdout:7/962: read d0/dd/d51/d65/d12a/dfb/f127 [827544,113393] 0 2024-04-12T05:37:21.278 INFO:tasks.workunit.client.0.smithi032.stdout:7/963: read d0/de/f44 [6139324,128627] 0 2024-04-12T05:37:21.287 INFO:tasks.workunit.client.0.smithi032.stdout:7/964: dwrite d0/de/d47/de1/f140 [0,4194304] 0 2024-04-12T05:37:21.303 INFO:tasks.workunit.client.1.smithi067.stdout:8/301: sync 2024-04-12T05:37:21.304 INFO:tasks.workunit.client.0.smithi032.stdout:7/965: getdents d0/d13 0 2024-04-12T05:37:21.305 INFO:tasks.workunit.client.0.smithi032.stdout:7/966: write d0/d34/dbe/f5c [1509410,51618] 0 2024-04-12T05:37:21.308 INFO:tasks.workunit.client.0.smithi032.stdout:7/967: symlink d0/de/df/d29/l141 0 2024-04-12T05:37:21.309 INFO:tasks.workunit.client.0.smithi032.stdout:7/968: write d0/dff/fc6 [7942801,23567] 0 2024-04-12T05:37:21.312 INFO:tasks.workunit.client.1.smithi067.stdout:9/275: link d1/d12/d15/l46 d1/d13/l52 0 2024-04-12T05:37:21.313 INFO:tasks.workunit.client.1.smithi067.stdout:6/383: dwrite f0 [4194304,4194304] 0 2024-04-12T05:37:21.314 INFO:tasks.workunit.client.0.smithi032.stdout:7/969: creat d0/de/df/f142 x:0 0 0 2024-04-12T05:37:21.315 INFO:tasks.workunit.client.1.smithi067.stdout:0/416: dwrite dd/d12/d14/d51/f5c [0,4194304] 0 2024-04-12T05:37:21.317 INFO:tasks.workunit.client.0.smithi032.stdout:7/970: creat d0/d13/d21/d81/f143 x:0 0 0 2024-04-12T05:37:21.321 INFO:tasks.workunit.client.1.smithi067.stdout:7/340: symlink d0/d23/d45/l70 0 2024-04-12T05:37:21.323 INFO:tasks.workunit.client.1.smithi067.stdout:2/363: dread d5/d23/f51 [0,4194304] 0 2024-04-12T05:37:21.324 INFO:tasks.workunit.client.1.smithi067.stdout:8/302: dread d6/d8/d18/df/f20 [0,4194304] 0 2024-04-12T05:37:21.324 INFO:tasks.workunit.client.1.smithi067.stdout:0/417: write dd/d12/d14/d1a/d75/f4c [3297826,63749] 0 2024-04-12T05:37:21.328 INFO:tasks.workunit.client.0.smithi032.stdout:7/971: dwrite d0/d13/d21/d81/f124 [0,4194304] 0 2024-04-12T05:37:21.330 INFO:tasks.workunit.client.1.smithi067.stdout:6/384: getdents dd/d21/d23/d56 0 2024-04-12T05:37:21.330 INFO:tasks.workunit.client.1.smithi067.stdout:2/364: read d5/da/f1d [4624234,7878] 0 2024-04-12T05:37:21.331 INFO:tasks.workunit.client.1.smithi067.stdout:7/341: unlink d0/d2/d5/f29 0 2024-04-12T05:37:21.333 INFO:tasks.workunit.client.1.smithi067.stdout:1/329: dwrite d4/d9/f17 [0,4194304] 0 2024-04-12T05:37:21.336 INFO:tasks.workunit.client.0.smithi032.stdout:7/972: rename d0/d10/d70/d7b/dd2/d10f to d0/dd/d51/d65/d12a/de0/d144 0 2024-04-12T05:37:21.338 INFO:tasks.workunit.client.0.smithi032.stdout:7/973: dread - d0/d34/dbf/fef zero size 2024-04-12T05:37:21.338 INFO:tasks.workunit.client.0.smithi032.stdout:7/974: dread - d0/d10/f63 zero size 2024-04-12T05:37:21.341 INFO:tasks.workunit.client.0.smithi032.stdout:7/975: fsync d0/de/df/f3f 0 2024-04-12T05:37:21.341 INFO:tasks.workunit.client.1.smithi067.stdout:1/330: symlink d4/d9/d3a/d51/d55/l6e 0 2024-04-12T05:37:21.343 INFO:tasks.workunit.client.1.smithi067.stdout:9/276: dwrite d1/d12/d15/f1d [0,4194304] 0 2024-04-12T05:37:21.345 INFO:tasks.workunit.client.1.smithi067.stdout:7/342: rename d0/d2/d5/d61 to d0/d2/dd/d42/d71 0 2024-04-12T05:37:21.347 INFO:tasks.workunit.client.1.smithi067.stdout:1/331: fsync d4/d9/f45 0 2024-04-12T05:37:21.347 INFO:tasks.workunit.client.1.smithi067.stdout:6/385: link dd/d11/d27/d4a/l59 dd/d30/d7b/l7d 0 2024-04-12T05:37:21.349 INFO:tasks.workunit.client.1.smithi067.stdout:9/277: write d1/d28/d23/f24 [4791418,13637] 0 2024-04-12T05:37:21.353 INFO:tasks.workunit.client.1.smithi067.stdout:2/365: dwrite d5/da/de/d14/d34/d40/d57/f60 [0,4194304] 0 2024-04-12T05:37:21.354 INFO:tasks.workunit.client.1.smithi067.stdout:6/386: symlink dd/d21/d64/d6f/l7e 0 2024-04-12T05:37:21.355 INFO:tasks.workunit.client.1.smithi067.stdout:9/278: creat d1/d12/f53 x:0 0 0 2024-04-12T05:37:21.357 INFO:tasks.workunit.client.1.smithi067.stdout:2/366: truncate d5/da/de/f52 5241001 0 2024-04-12T05:37:21.357 INFO:tasks.workunit.client.1.smithi067.stdout:6/387: symlink dd/d21/l7f 0 2024-04-12T05:37:21.360 INFO:tasks.workunit.client.1.smithi067.stdout:1/332: dwrite d4/d9/d3a/d51/f63 [0,4194304] 0 2024-04-12T05:37:21.361 INFO:tasks.workunit.client.1.smithi067.stdout:6/388: write dd/d11/d27/d4a/f73 [403793,14155] 0 2024-04-12T05:37:21.364 INFO:tasks.workunit.client.1.smithi067.stdout:2/367: dread d5/d43/f44 [0,4194304] 0 2024-04-12T05:37:21.364 INFO:tasks.workunit.client.1.smithi067.stdout:8/303: dread d6/f2c [4194304,4194304] 0 2024-04-12T05:37:21.364 INFO:tasks.workunit.client.1.smithi067.stdout:6/389: dread - dd/d21/d23/d56/f69 zero size 2024-04-12T05:37:21.366 INFO:tasks.workunit.client.1.smithi067.stdout:9/279: link d1/l5 d1/d28/d21/l54 0 2024-04-12T05:37:21.366 INFO:tasks.workunit.client.1.smithi067.stdout:0/418: truncate dd/d12/d14/d1a/d3d/f2f 8977834 0 2024-04-12T05:37:21.367 INFO:tasks.workunit.client.1.smithi067.stdout:0/419: chown dd 10231788 1 2024-04-12T05:37:21.369 INFO:tasks.workunit.client.1.smithi067.stdout:8/304: unlink d6/l52 0 2024-04-12T05:37:21.369 INFO:tasks.workunit.client.0.smithi032.stdout:7/976: dwrite d0/d34/dbe/fa6 [0,4194304] 0 2024-04-12T05:37:21.371 INFO:tasks.workunit.client.1.smithi067.stdout:7/343: getdents d0/d2/dd/d42 0 2024-04-12T05:37:21.374 INFO:tasks.workunit.client.1.smithi067.stdout:2/368: dwrite d5/da/f3e [0,4194304] 0 2024-04-12T05:37:21.377 INFO:tasks.workunit.client.1.smithi067.stdout:8/305: creat d6/d2f/f5a x:0 0 0 2024-04-12T05:37:21.377 INFO:tasks.workunit.client.1.smithi067.stdout:2/369: dread - d5/da/d3d/f6d zero size 2024-04-12T05:37:21.377 INFO:tasks.workunit.client.1.smithi067.stdout:9/280: mknod d1/d28/d44/c55 0 2024-04-12T05:37:21.379 INFO:tasks.workunit.client.0.smithi032.stdout:7/977: dwrite d0/de/d47/de1/f140 [0,4194304] 0 2024-04-12T05:37:21.384 INFO:tasks.workunit.client.1.smithi067.stdout:2/370: chown d5/da/de/d14/d34/d40/d57 1 1 2024-04-12T05:37:21.384 INFO:tasks.workunit.client.1.smithi067.stdout:2/371: readlink d5/d23/l45 0 2024-04-12T05:37:21.385 INFO:tasks.workunit.client.0.smithi032.stdout:7/978: creat d0/dd/d51/d65/d12a/d31/d13b/f145 x:0 0 0 2024-04-12T05:37:21.385 INFO:tasks.workunit.client.1.smithi067.stdout:2/372: stat d5/d23/d3a 0 2024-04-12T05:37:21.386 INFO:tasks.workunit.client.1.smithi067.stdout:7/344: unlink d0/d2/dd/d42/d71/c68 0 2024-04-12T05:37:21.386 INFO:tasks.workunit.client.1.smithi067.stdout:1/333: dread d4/d9/d12/d2c/f6c [0,4194304] 0 2024-04-12T05:37:21.387 INFO:tasks.workunit.client.0.smithi032.stdout:7/979: write d0/d34/dbe/dbb/de8/f11c [540002,98013] 0 2024-04-12T05:37:21.389 INFO:tasks.workunit.client.1.smithi067.stdout:0/420: link dd/d12/f59 dd/d5b/f84 0 2024-04-12T05:37:21.390 INFO:tasks.workunit.client.1.smithi067.stdout:6/390: getdents dd/d21/d3f 0 2024-04-12T05:37:21.392 INFO:tasks.workunit.client.1.smithi067.stdout:1/334: dread d4/d9/d12/d59/f64 [0,4194304] 0 2024-04-12T05:37:21.393 INFO:tasks.workunit.client.1.smithi067.stdout:2/373: creat d5/d11/f71 x:0 0 0 2024-04-12T05:37:21.395 INFO:tasks.workunit.client.1.smithi067.stdout:1/335: write d4/d46/f58 [273450,57415] 0 2024-04-12T05:37:21.397 INFO:tasks.workunit.client.0.smithi032.stdout:7/980: symlink d0/d10/d70/l146 0 2024-04-12T05:37:21.397 INFO:tasks.workunit.client.1.smithi067.stdout:6/391: mkdir dd/d21/d23/d80 0 2024-04-12T05:37:21.398 INFO:tasks.workunit.client.1.smithi067.stdout:6/392: truncate dd/d30/d7b/f6e 1799132 0 2024-04-12T05:37:21.399 INFO:tasks.workunit.client.1.smithi067.stdout:2/374: rmdir d5/d23 39 2024-04-12T05:37:21.400 INFO:tasks.workunit.client.1.smithi067.stdout:2/375: fdatasync d5/d11/f2a 0 2024-04-12T05:37:21.401 INFO:tasks.workunit.client.1.smithi067.stdout:7/345: creat d0/f72 x:0 0 0 2024-04-12T05:37:21.401 INFO:tasks.workunit.client.1.smithi067.stdout:0/421: mknod dd/d12/d14/d1a/d3d/d41/d5f/d78/d83/c85 0 2024-04-12T05:37:21.402 INFO:tasks.workunit.client.0.smithi032.stdout:7/981: rename d0/d34/f5d to d0/dd/d51/dea/d105/f147 0 2024-04-12T05:37:21.402 INFO:tasks.workunit.client.1.smithi067.stdout:1/336: mkdir d4/d9/d12/d6f 0 2024-04-12T05:37:21.403 INFO:tasks.workunit.client.1.smithi067.stdout:2/376: readlink d5/da/de/d14/l49 0 2024-04-12T05:37:21.403 INFO:tasks.workunit.client.1.smithi067.stdout:7/346: chown d0/d2/d5/fe 6343103 1 2024-04-12T05:37:21.404 INFO:tasks.workunit.client.1.smithi067.stdout:0/422: chown dd/d12/d14/d1a/d3d/d41/d5f/d78 66813772 1 2024-04-12T05:37:21.405 INFO:tasks.workunit.client.1.smithi067.stdout:6/393: dread f6 [4194304,4194304] 0 2024-04-12T05:37:21.405 INFO:tasks.workunit.client.1.smithi067.stdout:2/377: readlink d5/da/d3d/l42 0 2024-04-12T05:37:21.406 INFO:tasks.workunit.client.1.smithi067.stdout:7/347: chown d0/d23/d45/f1a 1507104698 1 2024-04-12T05:37:21.406 INFO:tasks.workunit.client.1.smithi067.stdout:1/337: creat d4/d52/f70 x:0 0 0 2024-04-12T05:37:21.407 INFO:tasks.workunit.client.1.smithi067.stdout:6/394: readlink dd/d21/l5c 0 2024-04-12T05:37:21.408 INFO:tasks.workunit.client.1.smithi067.stdout:8/306: dread d6/d8/d18/df/f49 [0,4194304] 0 2024-04-12T05:37:21.409 INFO:tasks.workunit.client.1.smithi067.stdout:7/348: creat d0/d23/d45/f73 x:0 0 0 2024-04-12T05:37:21.410 INFO:tasks.workunit.client.1.smithi067.stdout:8/307: write d6/d8/d18/df/d12/f1f [1289066,69804] 0 2024-04-12T05:37:21.411 INFO:tasks.workunit.client.1.smithi067.stdout:6/395: chown dd/d11/d27/d4a/l59 171585 1 2024-04-12T05:37:21.412 INFO:tasks.workunit.client.1.smithi067.stdout:7/349: write d0/d23/d45/f1a [694127,4708] 0 2024-04-12T05:37:21.412 INFO:tasks.workunit.client.0.smithi032.stdout:7/982: dread d0/de/f12 [0,4194304] 0 2024-04-12T05:37:21.413 INFO:tasks.workunit.client.0.smithi032.stdout:7/983: write d0/f116 [719008,53647] 0 2024-04-12T05:37:21.414 INFO:tasks.workunit.client.1.smithi067.stdout:2/378: link d5/da/de/d14/f63 d5/d43/d6e/f72 0 2024-04-12T05:37:21.415 INFO:tasks.workunit.client.1.smithi067.stdout:2/379: fsync d5/da/de/f69 0 2024-04-12T05:37:21.419 INFO:tasks.workunit.client.1.smithi067.stdout:8/308: write d6/f2c [2030973,25792] 0 2024-04-12T05:37:21.421 INFO:tasks.workunit.client.0.smithi032.stdout:7/984: truncate d0/de/f44 7431055 0 2024-04-12T05:37:21.423 INFO:tasks.workunit.client.1.smithi067.stdout:6/396: symlink dd/d21/d64/d6f/l81 0 2024-04-12T05:37:21.423 INFO:tasks.workunit.client.0.smithi032.stdout:7/985: read d0/d13/f64 [1764451,51321] 0 2024-04-12T05:37:21.424 INFO:tasks.workunit.client.1.smithi067.stdout:8/309: write d6/d8/d18/df/d12/f1d [9356378,100116] 0 2024-04-12T05:37:21.424 INFO:tasks.workunit.client.1.smithi067.stdout:7/350: read d0/d2/d5/db/f30 [658495,127942] 0 2024-04-12T05:37:21.424 INFO:tasks.workunit.client.1.smithi067.stdout:2/380: creat d5/d11/f73 x:0 0 0 2024-04-12T05:37:21.425 INFO:tasks.workunit.client.1.smithi067.stdout:8/310: readlink l3 0 2024-04-12T05:37:21.425 INFO:tasks.workunit.client.1.smithi067.stdout:9/281: truncate d1/d28/d21/f26 3532464 0 2024-04-12T05:37:21.426 INFO:tasks.workunit.client.0.smithi032.stdout:7/986: mknod d0/dd/d51/d65/d12a/da1/c148 0 2024-04-12T05:37:21.428 INFO:tasks.workunit.client.1.smithi067.stdout:7/351: readlink d0/d2/d5/db/l3c 0 2024-04-12T05:37:21.428 INFO:tasks.workunit.client.1.smithi067.stdout:6/397: chown dd/l33 439 1 2024-04-12T05:37:21.428 INFO:tasks.workunit.client.1.smithi067.stdout:9/282: fsync d1/d28/f18 0 2024-04-12T05:37:21.430 INFO:tasks.workunit.client.1.smithi067.stdout:7/352: write d0/d2/f21 [3338002,125656] 0 2024-04-12T05:37:21.431 INFO:tasks.workunit.client.1.smithi067.stdout:9/283: write d1/d28/d23/f24 [139776,95572] 0 2024-04-12T05:37:21.437 INFO:tasks.workunit.client.0.smithi032.stdout:4/936: sync 2024-04-12T05:37:21.438 INFO:tasks.workunit.client.1.smithi067.stdout:0/423: dread dd/d12/d14/d49/f4d [0,4194304] 0 2024-04-12T05:37:21.440 INFO:tasks.workunit.client.1.smithi067.stdout:8/311: creat d6/d8/d18/df/f5b x:0 0 0 2024-04-12T05:37:21.442 INFO:tasks.workunit.client.1.smithi067.stdout:1/338: truncate d4/d9/f1c 389921 0 2024-04-12T05:37:21.442 INFO:tasks.workunit.client.1.smithi067.stdout:9/284: symlink d1/d12/l56 0 2024-04-12T05:37:21.443 INFO:tasks.workunit.client.0.smithi032.stdout:7/987: dwrite d0/dd/d51/d65/d12a/fd3 [0,4194304] 0 2024-04-12T05:37:21.450 INFO:tasks.workunit.client.0.smithi032.stdout:4/937: truncate d4/d13/d44/fca 1182915 0 2024-04-12T05:37:21.455 INFO:tasks.workunit.client.1.smithi067.stdout:2/381: getdents d5/da/de/d14/d34 0 2024-04-12T05:37:21.456 INFO:tasks.workunit.client.0.smithi032.stdout:4/938: rename d4/d87/d96/d2f/d7c/l99 to d4/d13/d44/l137 0 2024-04-12T05:37:21.456 INFO:tasks.workunit.client.1.smithi067.stdout:6/398: rmdir dd/d21/d64/d75 0 2024-04-12T05:37:21.458 INFO:tasks.workunit.client.0.smithi032.stdout:4/939: unlink d4/l20 0 2024-04-12T05:37:21.458 INFO:tasks.workunit.client.1.smithi067.stdout:7/353: dwrite d0/d2/d5/db/f65 [0,4194304] 0 2024-04-12T05:37:21.460 INFO:tasks.workunit.client.1.smithi067.stdout:2/382: creat d5/da/f74 x:0 0 0 2024-04-12T05:37:21.461 INFO:tasks.workunit.client.1.smithi067.stdout:1/339: creat d4/d9/d4c/d5d/f71 x:0 0 0 2024-04-12T05:37:21.462 INFO:tasks.workunit.client.0.smithi032.stdout:4/940: unlink d4/d13/d44/fd6 0 2024-04-12T05:37:21.465 INFO:tasks.workunit.client.0.smithi032.stdout:4/941: mknod d4/d87/d96/d32/dc5/c138 0 2024-04-12T05:37:21.465 INFO:tasks.workunit.client.1.smithi067.stdout:9/285: creat d1/d28/d21/d3f/d4f/f57 x:0 0 0 2024-04-12T05:37:21.466 INFO:tasks.workunit.client.0.smithi032.stdout:4/942: readlink d4/l29 0 2024-04-12T05:37:21.469 INFO:tasks.workunit.client.1.smithi067.stdout:1/340: dread d4/d9/d14/f57 [0,4194304] 0 2024-04-12T05:37:21.470 INFO:tasks.workunit.client.0.smithi032.stdout:7/988: truncate d0/dff/d71/f8a 3052022 0 2024-04-12T05:37:21.470 INFO:tasks.workunit.client.1.smithi067.stdout:6/399: link dd/f1d dd/d11/d27/d4a/f82 0 2024-04-12T05:37:21.472 INFO:tasks.workunit.client.1.smithi067.stdout:9/286: symlink d1/d28/d21/d3f/l58 0 2024-04-12T05:37:21.472 INFO:tasks.workunit.client.1.smithi067.stdout:8/312: dread f5 [0,4194304] 0 2024-04-12T05:37:21.473 INFO:tasks.workunit.client.0.smithi032.stdout:4/943: link d4/d87/d96/d4d/db5/dcf/de3/f10c d4/d87/d96/f139 0 2024-04-12T05:37:21.474 INFO:tasks.workunit.client.1.smithi067.stdout:7/354: link d0/d2/d5/d1e/f4f d0/d2/f74 0 2024-04-12T05:37:21.475 INFO:tasks.workunit.client.0.smithi032.stdout:4/944: dread - d4/d87/d96/d4d/d78/dba/dfc/f12a zero size 2024-04-12T05:37:21.475 INFO:tasks.workunit.client.1.smithi067.stdout:2/383: getdents d5/da/de/d14/d34/d4c 0 2024-04-12T05:37:21.476 INFO:tasks.workunit.client.1.smithi067.stdout:7/355: dread - d0/d2/d5/f50 zero size 2024-04-12T05:37:21.476 INFO:tasks.workunit.client.0.smithi032.stdout:7/989: link d0/d10/d70/d7b/fc9 d0/dd/d51/f149 0 2024-04-12T05:37:21.477 INFO:tasks.workunit.client.1.smithi067.stdout:6/400: rename dd/d30/f5f to dd/d21/d66/f83 0 2024-04-12T05:37:21.477 INFO:tasks.workunit.client.0.smithi032.stdout:7/990: chown d0/d10/f60 1335642 1 2024-04-12T05:37:21.479 INFO:tasks.workunit.client.1.smithi067.stdout:8/313: creat d6/d8/d18/d36/f5c x:0 0 0 2024-04-12T05:37:21.479 INFO:tasks.workunit.client.1.smithi067.stdout:1/341: mknod d4/d9/d3a/d51/d55/d60/c72 0 2024-04-12T05:37:21.482 INFO:tasks.workunit.client.0.smithi032.stdout:4/945: dread d4/d87/d96/d2f/f4e [0,4194304] 0 2024-04-12T05:37:21.483 INFO:tasks.workunit.client.1.smithi067.stdout:9/287: rename d1/d12/l56 to d1/d13/l59 0 2024-04-12T05:37:21.484 INFO:tasks.workunit.client.1.smithi067.stdout:0/424: dread dd/d12/d14/d1a/d3d/d41/f62 [0,4194304] 0 2024-04-12T05:37:21.485 INFO:tasks.workunit.client.0.smithi032.stdout:4/946: mknod d4/d87/d96/d2f/d7c/dc2/c13a 0 2024-04-12T05:37:21.486 INFO:tasks.workunit.client.1.smithi067.stdout:2/384: dwrite d5/d6a/f65 [4194304,4194304] 0 2024-04-12T05:37:21.488 INFO:tasks.workunit.client.1.smithi067.stdout:6/401: creat dd/d30/d7b/f84 x:0 0 0 2024-04-12T05:37:21.489 INFO:tasks.workunit.client.1.smithi067.stdout:8/314: rmdir d6/d8/d18/df 39 2024-04-12T05:37:21.489 INFO:tasks.workunit.client.1.smithi067.stdout:0/425: write dd/f13 [7521900,61222] 0 2024-04-12T05:37:21.491 INFO:tasks.workunit.client.0.smithi032.stdout:7/991: write d0/d34/dbe/dbb/ff5 [4794825,38836] 0 2024-04-12T05:37:21.493 INFO:tasks.workunit.client.1.smithi067.stdout:0/426: dread - dd/d12/d14/d51/f77 zero size 2024-04-12T05:37:21.494 INFO:tasks.workunit.client.1.smithi067.stdout:8/315: mkdir d6/d2f/d5d 0 2024-04-12T05:37:21.496 INFO:tasks.workunit.client.1.smithi067.stdout:6/402: symlink dd/d11/l85 0 2024-04-12T05:37:21.496 INFO:tasks.workunit.client.1.smithi067.stdout:1/342: dwrite d4/d9/d12/d2c/f6c [0,4194304] 0 2024-04-12T05:37:21.497 INFO:tasks.workunit.client.1.smithi067.stdout:6/403: chown dd/d21/d64/d6f 3 1 2024-04-12T05:37:21.498 INFO:tasks.workunit.client.1.smithi067.stdout:9/288: symlink d1/d13/d34/l5a 0 2024-04-12T05:37:21.500 INFO:tasks.workunit.client.1.smithi067.stdout:2/385: creat d5/da/de/d14/d34/d4c/f75 x:0 0 0 2024-04-12T05:37:21.501 INFO:tasks.workunit.client.0.smithi032.stdout:4/947: dwrite d4/d87/d96/d68/fd7 [0,4194304] 0 2024-04-12T05:37:21.503 INFO:tasks.workunit.client.1.smithi067.stdout:1/343: write d4/d9/d3a/f1e [3883442,34684] 0 2024-04-12T05:37:21.503 INFO:tasks.workunit.client.0.smithi032.stdout:7/992: rename d0/de/df to d0/d13/d21/d81/d14a 0 2024-04-12T05:37:21.504 INFO:tasks.workunit.client.1.smithi067.stdout:7/356: write d0/d2/d5/db/f30 [2140246,62902] 0 2024-04-12T05:37:21.506 INFO:tasks.workunit.client.1.smithi067.stdout:6/404: readlink dd/d21/d3f/l52 0 2024-04-12T05:37:21.506 INFO:tasks.workunit.client.1.smithi067.stdout:9/289: rmdir d1/d28/d21/d3f 39 2024-04-12T05:37:21.511 INFO:tasks.workunit.client.0.smithi032.stdout:4/948: link d4/d13/d19/ffa d4/d87/d96/d2f/d3b/da2/d121/f13b 0 2024-04-12T05:37:21.511 INFO:tasks.workunit.client.0.smithi032.stdout:7/993: dread d0/d10/d70/f8f [4194304,4194304] 0 2024-04-12T05:37:21.511 INFO:tasks.workunit.client.1.smithi067.stdout:7/357: mknod d0/d67/d5c/c75 0 2024-04-12T05:37:21.511 INFO:tasks.workunit.client.1.smithi067.stdout:7/358: dread - d0/f72 zero size 2024-04-12T05:37:21.512 INFO:tasks.workunit.client.1.smithi067.stdout:4/252: sync 2024-04-12T05:37:21.514 INFO:tasks.workunit.client.0.smithi032.stdout:4/949: truncate d4/d13/d44/f6a 4620835 0 2024-04-12T05:37:21.515 INFO:tasks.workunit.client.1.smithi067.stdout:8/316: creat d6/d8/d18/f5e x:0 0 0 2024-04-12T05:37:21.518 INFO:tasks.workunit.client.0.smithi032.stdout:4/950: creat d4/d87/d96/d4d/dbc/dd2/f13c x:0 0 0 2024-04-12T05:37:21.519 INFO:tasks.workunit.client.0.smithi032.stdout:4/951: rename d4 to d4/d87/d96/d68/d13d 22 2024-04-12T05:37:21.519 INFO:tasks.workunit.client.1.smithi067.stdout:2/386: dwrite d5/d43/f53 [0,4194304] 0 2024-04-12T05:37:21.520 INFO:tasks.workunit.client.0.smithi032.stdout:4/952: chown d4/d87/d96/d4d/la4 7724 1 2024-04-12T05:37:21.522 INFO:tasks.workunit.client.1.smithi067.stdout:4/253: mkdir d8/da/d1d/d15/d58 0 2024-04-12T05:37:21.523 INFO:tasks.workunit.client.0.smithi032.stdout:4/953: mknod d4/d13/d75/c13e 0 2024-04-12T05:37:21.523 INFO:tasks.workunit.client.1.smithi067.stdout:1/344: dwrite d4/d9/d3a/f1e [4194304,4194304] 0 2024-04-12T05:37:21.527 INFO:tasks.workunit.client.1.smithi067.stdout:6/405: truncate dd/fe 3324874 0 2024-04-12T05:37:21.531 INFO:tasks.workunit.client.1.smithi067.stdout:4/254: creat d8/da/d1d/d15/d58/f59 x:0 0 0 2024-04-12T05:37:21.531 INFO:tasks.workunit.client.0.smithi032.stdout:4/954: fsync d4/d87/d96/f42 0 2024-04-12T05:37:21.531 INFO:tasks.workunit.client.0.smithi032.stdout:4/955: readlink d4/l8f 0 2024-04-12T05:37:21.532 INFO:tasks.workunit.client.1.smithi067.stdout:7/359: dwrite d0/d2/d5/db/f4e [4194304,4194304] 0 2024-04-12T05:37:21.533 INFO:tasks.workunit.client.1.smithi067.stdout:8/317: creat d6/d8/d18/df/f5f x:0 0 0 2024-04-12T05:37:21.534 INFO:tasks.workunit.client.1.smithi067.stdout:8/318: truncate d6/f27 4699089 0 2024-04-12T05:37:21.539 INFO:tasks.workunit.client.1.smithi067.stdout:4/255: write d8/d9/f42 [783151,122723] 0 2024-04-12T05:37:21.539 INFO:tasks.workunit.client.0.smithi032.stdout:7/994: dread d0/d13/d21/d81/d14a/d20/f6b [0,4194304] 0 2024-04-12T05:37:21.542 INFO:tasks.workunit.client.1.smithi067.stdout:8/319: unlink d6/d8/d18/c15 0 2024-04-12T05:37:21.543 INFO:tasks.workunit.client.1.smithi067.stdout:7/360: unlink d0/d2/d5/db/f53 0 2024-04-12T05:37:21.545 INFO:tasks.workunit.client.1.smithi067.stdout:4/256: fdatasync d8/d9/f2a 0 2024-04-12T05:37:21.546 INFO:tasks.workunit.client.1.smithi067.stdout:7/361: fdatasync d0/d2/f74 0 2024-04-12T05:37:21.548 INFO:tasks.workunit.client.0.smithi032.stdout:4/956: dwrite d4/d13/d19/ffa [0,4194304] 0 2024-04-12T05:37:21.548 INFO:tasks.workunit.client.1.smithi067.stdout:6/406: getdents dd/d21/d23 0 2024-04-12T05:37:21.549 INFO:tasks.workunit.client.1.smithi067.stdout:9/290: dread d1/d12/f14 [0,4194304] 0 2024-04-12T05:37:21.549 INFO:tasks.workunit.client.1.smithi067.stdout:6/407: fsync dd/f17 0 2024-04-12T05:37:21.551 INFO:tasks.workunit.client.1.smithi067.stdout:4/257: creat d8/da/d1d/d15/d36/f5a x:0 0 0 2024-04-12T05:37:21.552 INFO:tasks.workunit.client.0.smithi032.stdout:4/957: stat d4/d13/d19/c17 0 2024-04-12T05:37:21.554 INFO:tasks.workunit.client.0.smithi032.stdout:7/995: dwrite d0/d13/d21/d81/d14a/d20/f6b [4194304,4194304] 0 2024-04-12T05:37:21.556 INFO:tasks.workunit.client.1.smithi067.stdout:9/291: fsync d1/d12/d15/f43 0 2024-04-12T05:37:21.558 INFO:tasks.workunit.client.1.smithi067.stdout:4/258: mknod d8/d9/d3d/d48/d3c/c5b 0 2024-04-12T05:37:21.560 INFO:tasks.workunit.client.0.smithi032.stdout:4/958: rename d4/d87/d96/d4d/f63 to d4/d87/d96/d2f/d9a/f13f 0 2024-04-12T05:37:21.561 INFO:tasks.workunit.client.0.smithi032.stdout:4/959: dread - d4/d87/d96/d4d/f12c zero size 2024-04-12T05:37:21.562 INFO:tasks.workunit.client.0.smithi032.stdout:4/960: chown d4/d13/d44/l9f 5718 1 2024-04-12T05:37:21.565 INFO:tasks.workunit.client.1.smithi067.stdout:8/320: getdents d6/d8/d18 0 2024-04-12T05:37:21.566 INFO:tasks.workunit.client.1.smithi067.stdout:7/362: dwrite d0/f12 [4194304,4194304] 0 2024-04-12T05:37:21.566 INFO:tasks.workunit.client.1.smithi067.stdout:9/292: readlink d1/d12/d15/l17 0 2024-04-12T05:37:21.566 INFO:tasks.workunit.client.1.smithi067.stdout:2/387: truncate d5/da/de/d14/d34/f37 3529864 0 2024-04-12T05:37:21.567 INFO:tasks.workunit.client.1.smithi067.stdout:6/408: symlink dd/d11/d27/d4a/d79/l86 0 2024-04-12T05:37:21.567 INFO:tasks.workunit.client.1.smithi067.stdout:8/321: chown d6/d8/d50 425674 1 2024-04-12T05:37:21.568 INFO:tasks.workunit.client.1.smithi067.stdout:1/345: truncate d4/d9/d3a/d51/f63 2974853 0 2024-04-12T05:37:21.568 INFO:tasks.workunit.client.1.smithi067.stdout:2/388: readlink d5/da/de/d14/l49 0 2024-04-12T05:37:21.569 INFO:tasks.workunit.client.1.smithi067.stdout:4/259: truncate f7 3783814 0 2024-04-12T05:37:21.569 INFO:tasks.workunit.client.1.smithi067.stdout:1/346: chown d4/d46 24104184 1 2024-04-12T05:37:21.570 INFO:tasks.workunit.client.1.smithi067.stdout:7/363: truncate d0/d2/d5/fe 4817831 0 2024-04-12T05:37:21.571 INFO:tasks.workunit.client.0.smithi032.stdout:4/961: rename d4/d87/d96/d4d/dbc/cd5 to d4/d87/d96/d4d/d78/dba/c140 0 2024-04-12T05:37:21.573 INFO:tasks.workunit.client.0.smithi032.stdout:7/996: dwrite d0/dd/d51/dea/d105/f147 [0,4194304] 0 2024-04-12T05:37:21.574 INFO:tasks.workunit.client.1.smithi067.stdout:2/389: write d5/da/de/d14/d34/d4c/f5f [2459748,24663] 0 2024-04-12T05:37:21.574 INFO:tasks.workunit.client.1.smithi067.stdout:8/322: unlink f5 0 2024-04-12T05:37:21.575 INFO:tasks.workunit.client.0.smithi032.stdout:7/997: fsync d0/de/f130 0 2024-04-12T05:37:21.576 INFO:tasks.workunit.client.1.smithi067.stdout:8/323: stat d6/d2f/d5d 0 2024-04-12T05:37:21.577 INFO:tasks.workunit.client.0.smithi032.stdout:4/962: creat d4/d13/d19/f141 x:0 0 0 2024-04-12T05:37:21.577 INFO:tasks.workunit.client.1.smithi067.stdout:6/409: creat dd/d11/d27/f87 x:0 0 0 2024-04-12T05:37:21.578 INFO:tasks.workunit.client.0.smithi032.stdout:7/998: creat d0/dd/d51/d65/d12a/d31/d13b/f14b x:0 0 0 2024-04-12T05:37:21.578 INFO:tasks.workunit.client.1.smithi067.stdout:7/364: symlink d0/d67/l76 0 2024-04-12T05:37:21.580 INFO:tasks.workunit.client.0.smithi032.stdout:7/999: fdatasync d0/d13/d21/d81/d14a/d29/f110 0 2024-04-12T05:37:21.580 INFO:tasks.workunit.client.1.smithi067.stdout:9/293: dwrite d1/f47 [0,4194304] 0 2024-04-12T05:37:21.580 INFO:tasks.workunit.client.1.smithi067.stdout:4/260: creat d8/d9/d3d/d48/d45/f5c x:0 0 0 2024-04-12T05:37:21.581 INFO:tasks.workunit.client.0.smithi032.stdout:4/963: truncate d4/d87/d96/d4d/db5/dcf/d114/f122 490757 0 2024-04-12T05:37:21.585 INFO:tasks.workunit.client.1.smithi067.stdout:7/365: write d0/d67/d5c/f6a [558349,9506] 0 2024-04-12T05:37:21.587 INFO:tasks.workunit.client.1.smithi067.stdout:1/347: fdatasync d4/d9/d12/d59/f64 0 2024-04-12T05:37:21.589 INFO:tasks.workunit.client.0.smithi032.stdout:1/856: sync 2024-04-12T05:37:21.593 INFO:tasks.workunit.client.1.smithi067.stdout:4/261: fsync d8/d9/f21 0 2024-04-12T05:37:21.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:21 smithi032 ceph-mon[39731]: Standby manager daemon smithi067.dwichg started 2024-04-12T05:37:21.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:21 smithi032 ceph-mon[39731]: from='mgr.? 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi067.dwichg/crt"}]: dispatch 2024-04-12T05:37:21.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:21 smithi032 ceph-mon[39731]: from='mgr.? 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T05:37:21.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:21 smithi032 ceph-mon[39731]: from='mgr.? 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi067.dwichg/key"}]: dispatch 2024-04-12T05:37:21.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:21 smithi032 ceph-mon[39731]: from='mgr.? 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T05:37:21.595 INFO:tasks.workunit.client.0.smithi032.stdout:4/964: symlink d4/d87/d96/d2f/d94/dc6/d123/l142 0 2024-04-12T05:37:21.595 INFO:tasks.workunit.client.1.smithi067.stdout:8/324: dwrite d6/f27 [0,4194304] 0 2024-04-12T05:37:21.595 INFO:tasks.workunit.client.1.smithi067.stdout:6/410: rename dd/d30 to dd/d21/d64/d88 0 2024-04-12T05:37:21.597 INFO:tasks.workunit.client.1.smithi067.stdout:2/390: dwrite d5/d43/f53 [0,4194304] 0 2024-04-12T05:37:21.598 INFO:tasks.workunit.client.0.smithi032.stdout:4/965: write d4/d87/d96/d4d/db5/f108 [740684,10569] 0 2024-04-12T05:37:21.598 INFO:tasks.workunit.client.1.smithi067.stdout:9/294: dread d1/d28/f18 [0,4194304] 0 2024-04-12T05:37:21.602 INFO:tasks.workunit.client.0.smithi032.stdout:1/857: mkdir d5/de/d17/d35/d61/d77/d8f/da0/d10f 0 2024-04-12T05:37:21.602 INFO:tasks.workunit.client.0.smithi032.stdout:4/966: readlink d4/d87/d96/d2f/l10b 0 2024-04-12T05:37:21.605 INFO:tasks.workunit.client.1.smithi067.stdout:7/366: dwrite d0/d23/f6e [0,4194304] 0 2024-04-12T05:37:21.607 INFO:tasks.workunit.client.1.smithi067.stdout:1/348: dread d4/d9/d3a/f1e [0,4194304] 0 2024-04-12T05:37:21.607 INFO:tasks.workunit.client.1.smithi067.stdout:2/391: creat d5/d43/d6e/f76 x:0 0 0 2024-04-12T05:37:21.608 INFO:tasks.workunit.client.0.smithi032.stdout:1/858: mknod d5/de/d17/d35/d3b/d49/c110 0 2024-04-12T05:37:21.608 INFO:tasks.workunit.client.1.smithi067.stdout:7/367: write d0/d67/f56 [573292,111492] 0 2024-04-12T05:37:21.610 INFO:tasks.workunit.client.1.smithi067.stdout:8/325: creat d6/d8/d18/d4e/f60 x:0 0 0 2024-04-12T05:37:21.611 INFO:tasks.workunit.client.1.smithi067.stdout:9/295: chown d1/d28/d21/d3f/d4f/f57 52127 1 2024-04-12T05:37:21.611 INFO:tasks.workunit.client.0.smithi032.stdout:4/967: creat d4/f143 x:0 0 0 2024-04-12T05:37:21.611 INFO:tasks.workunit.client.1.smithi067.stdout:7/368: chown d0/d23/d45/l70 2132615 1 2024-04-12T05:37:21.612 INFO:tasks.workunit.client.1.smithi067.stdout:4/262: mkdir d8/da/d1d/d15/d36/d5d 0 2024-04-12T05:37:21.614 INFO:tasks.workunit.client.1.smithi067.stdout:1/349: mkdir d4/d9/d4c/d5d/d73 0 2024-04-12T05:37:21.614 INFO:tasks.workunit.client.1.smithi067.stdout:4/263: dread - d8/d39/f51 zero size 2024-04-12T05:37:21.616 INFO:tasks.workunit.client.1.smithi067.stdout:1/350: readlink d4/d9/d12/l24 0 2024-04-12T05:37:21.616 INFO:tasks.workunit.client.0.smithi032.stdout:4/968: unlink d4/d87/d96/d4d/d78/dba/cf2 0 2024-04-12T05:37:21.616 INFO:tasks.workunit.client.0.smithi032.stdout:1/859: mkdir d5/de/d17/d93/d10c/d111 0 2024-04-12T05:37:21.617 INFO:tasks.workunit.client.1.smithi067.stdout:2/392: rename d5/d6a/f67 to d5/d6a/f77 0 2024-04-12T05:37:21.617 INFO:tasks.workunit.client.1.smithi067.stdout:2/393: chown d5/d11/f2e 342 1 2024-04-12T05:37:21.618 INFO:tasks.workunit.client.1.smithi067.stdout:9/296: creat d1/d13/f5b x:0 0 0 2024-04-12T05:37:21.618 INFO:tasks.workunit.client.1.smithi067.stdout:2/394: dread - d5/d6a/f77 zero size 2024-04-12T05:37:21.619 INFO:tasks.workunit.client.1.smithi067.stdout:8/326: mknod d6/d8/d18/d36/c61 0 2024-04-12T05:37:21.620 INFO:tasks.workunit.client.0.smithi032.stdout:4/969: truncate d4/d87/d96/d2f/d3b/da2/dec/f132 380987 0 2024-04-12T05:37:21.621 INFO:tasks.workunit.client.1.smithi067.stdout:2/395: chown d5/da/de/d14/d34/d40/d57 86240577 1 2024-04-12T05:37:21.622 INFO:tasks.workunit.client.1.smithi067.stdout:7/369: mkdir d0/d2/dd/d42/d71/d77 0 2024-04-12T05:37:21.623 INFO:tasks.workunit.client.0.smithi032.stdout:1/860: rename d5/de/d17/d35/d61/d54 to d5/de/d17/d112 0 2024-04-12T05:37:21.624 INFO:tasks.workunit.client.0.smithi032.stdout:4/970: symlink d4/d87/d96/d2f/d3b/da2/l144 0 2024-04-12T05:37:21.624 INFO:tasks.workunit.client.0.smithi032.stdout:1/861: truncate d5/de/d2a/f2b 1130180 0 2024-04-12T05:37:21.626 INFO:tasks.workunit.client.0.smithi032.stdout:4/971: write d4/d87/d96/d2f/d7c/dc2/f124 [3525618,127257] 0 2024-04-12T05:37:21.626 INFO:tasks.workunit.client.1.smithi067.stdout:9/297: read d1/d28/f1b [2151746,48107] 0 2024-04-12T05:37:21.628 INFO:tasks.workunit.client.1.smithi067.stdout:6/411: rename dd/fe to dd/f89 0 2024-04-12T05:37:21.628 INFO:tasks.workunit.client.1.smithi067.stdout:8/327: dread d6/d8/d18/df/f24 [0,4194304] 0 2024-04-12T05:37:21.629 INFO:tasks.workunit.client.0.smithi032.stdout:1/862: mknod d5/de/d17/d93/d10c/c113 0 2024-04-12T05:37:21.630 INFO:tasks.workunit.client.1.smithi067.stdout:8/328: chown d6/l10 24 1 2024-04-12T05:37:21.630 INFO:tasks.workunit.client.0.smithi032.stdout:4/972: chown d4/d13/dc9/d100/f105 56 1 2024-04-12T05:37:21.631 INFO:tasks.workunit.client.0.smithi032.stdout:1/863: write d5/de/d17/d35/d61/d77/d8f/d9c/fe2 [474773,98512] 0 2024-04-12T05:37:21.631 INFO:tasks.workunit.client.1.smithi067.stdout:4/264: symlink d8/da/d1d/d15/l5e 0 2024-04-12T05:37:21.633 INFO:tasks.workunit.client.1.smithi067.stdout:6/412: dread f6 [4194304,4194304] 0 2024-04-12T05:37:21.635 INFO:tasks.workunit.client.1.smithi067.stdout:2/396: mkdir d5/d6a/d78 0 2024-04-12T05:37:21.638 INFO:tasks.workunit.client.1.smithi067.stdout:6/413: dread dd/d21/d23/f25 [0,4194304] 0 2024-04-12T05:37:21.639 INFO:tasks.workunit.client.0.smithi032.stdout:1/864: link d5/de/d17/d35/d61/d77/fd0 d5/de/d17/d112/f114 0 2024-04-12T05:37:21.639 INFO:tasks.workunit.client.1.smithi067.stdout:9/298: creat d1/d12/d15/f5c x:0 0 0 2024-04-12T05:37:21.640 INFO:tasks.workunit.client.0.smithi032.stdout:4/973: getdents d4/d87/d96/d2f/d3b/da2/d121 0 2024-04-12T05:37:21.640 INFO:tasks.workunit.client.1.smithi067.stdout:9/299: chown d1/cc 16113338 1 2024-04-12T05:37:21.641 INFO:tasks.workunit.client.1.smithi067.stdout:4/265: mkdir d8/da/d1d/d15/d1c/d5f 0 2024-04-12T05:37:21.642 INFO:tasks.workunit.client.0.smithi032.stdout:1/865: unlink d5/de/d17/d1b/l24 0 2024-04-12T05:37:21.645 INFO:tasks.workunit.client.0.smithi032.stdout:1/866: mkdir d5/de/d17/d35/d61/d77/d8f/d115 0 2024-04-12T05:37:21.648 INFO:tasks.workunit.client.1.smithi067.stdout:6/414: creat dd/d21/d23/d56/f8a x:0 0 0 2024-04-12T05:37:21.649 INFO:tasks.workunit.client.0.smithi032.stdout:1/867: unlink d5/de/d5b/lf9 0 2024-04-12T05:37:21.649 INFO:tasks.workunit.client.1.smithi067.stdout:9/300: rename d1/d28/d27/c42 to d1/d28/d27/c5d 0 2024-04-12T05:37:21.650 INFO:tasks.workunit.client.1.smithi067.stdout:2/397: dwrite d5/d11/f32 [0,4194304] 0 2024-04-12T05:37:21.652 INFO:tasks.workunit.client.1.smithi067.stdout:1/351: truncate d4/d9/d3a/d51/f63 1916164 0 2024-04-12T05:37:21.652 INFO:tasks.workunit.client.1.smithi067.stdout:7/370: dread d0/f8 [0,4194304] 0 2024-04-12T05:37:21.652 INFO:tasks.workunit.client.0.smithi032.stdout:1/868: symlink d5/de/d17/d35/d61/d105/d108/l116 0 2024-04-12T05:37:21.654 INFO:tasks.workunit.client.1.smithi067.stdout:6/415: mkdir dd/d11/d27/d4a/d8b 0 2024-04-12T05:37:21.656 INFO:tasks.workunit.client.1.smithi067.stdout:9/301: write d1/d28/d21/d3f/f41 [239562,69570] 0 2024-04-12T05:37:21.657 INFO:tasks.workunit.client.1.smithi067.stdout:8/329: dread d6/d8/d18/f25 [0,4194304] 0 2024-04-12T05:37:21.657 INFO:tasks.workunit.client.1.smithi067.stdout:4/266: link d8/d9/d3d/d48/d3c/c5b d8/c60 0 2024-04-12T05:37:21.658 INFO:tasks.workunit.client.0.smithi032.stdout:4/974: write d4/d13/d44/f5c [578892,4777] 0 2024-04-12T05:37:21.658 INFO:tasks.workunit.client.1.smithi067.stdout:1/352: write d4/d9/f22 [595425,11513] 0 2024-04-12T05:37:21.659 INFO:tasks.workunit.client.0.smithi032.stdout:4/975: stat d4/d87/d96/d4d/dbc/dd2 0 2024-04-12T05:37:21.659 INFO:tasks.workunit.client.1.smithi067.stdout:8/330: stat d6/d8/d18/df/f5f 0 2024-04-12T05:37:21.660 INFO:tasks.workunit.client.1.smithi067.stdout:2/398: dread d5/d43/f44 [0,4194304] 0 2024-04-12T05:37:21.661 INFO:tasks.workunit.client.1.smithi067.stdout:9/302: mkdir d1/d28/d5e 0 2024-04-12T05:37:21.661 INFO:tasks.workunit.client.1.smithi067.stdout:7/371: creat d0/d2/f78 x:0 0 0 2024-04-12T05:37:21.662 INFO:tasks.workunit.client.1.smithi067.stdout:9/303: fsync d1/f47 0 2024-04-12T05:37:21.663 INFO:tasks.workunit.client.1.smithi067.stdout:6/416: creat dd/d11/d27/d4a/f8c x:0 0 0 2024-04-12T05:37:21.663 INFO:tasks.workunit.client.1.smithi067.stdout:2/399: stat d5/da/de/d39/f6c 0 2024-04-12T05:37:21.664 INFO:tasks.workunit.client.0.smithi032.stdout:4/976: truncate d4/d13/d44/f6a 3341503 0 2024-04-12T05:37:21.664 INFO:tasks.workunit.client.1.smithi067.stdout:8/331: creat d6/d8/d18/f62 x:0 0 0 2024-04-12T05:37:21.665 INFO:tasks.workunit.client.0.smithi032.stdout:4/977: dread - d4/d87/d96/d4d/d78/dba/f120 zero size 2024-04-12T05:37:21.666 INFO:tasks.workunit.client.1.smithi067.stdout:2/400: write d5/d11/f32 [1286097,89194] 0 2024-04-12T05:37:21.667 INFO:tasks.workunit.client.1.smithi067.stdout:1/353: mknod d4/d9/d3a/d51/d55/d60/c74 0 2024-04-12T05:37:21.667 INFO:tasks.workunit.client.1.smithi067.stdout:2/401: write d5/d43/d6e/f76 [466780,18902] 0 2024-04-12T05:37:21.668 INFO:tasks.workunit.client.0.smithi032.stdout:1/869: dread d5/de/d17/d35/d3b/f109 [0,4194304] 0 2024-04-12T05:37:21.668 INFO:tasks.workunit.client.1.smithi067.stdout:7/372: fsync d0/f8 0 2024-04-12T05:37:21.669 INFO:tasks.workunit.client.0.smithi032.stdout:4/978: symlink d4/d87/d96/d2f/d94/dc6/l145 0 2024-04-12T05:37:21.669 INFO:tasks.workunit.client.1.smithi067.stdout:4/267: creat d8/da/d1d/d15/d1c/d5f/f61 x:0 0 0 2024-04-12T05:37:21.670 INFO:tasks.workunit.client.1.smithi067.stdout:2/402: stat d5/da/de/d14/d34/d40 0 2024-04-12T05:37:21.671 INFO:tasks.workunit.client.0.smithi032.stdout:4/979: chown d4/d87/d96/d2f/d9a/fa7 68 1 2024-04-12T05:37:21.673 INFO:tasks.workunit.client.1.smithi067.stdout:2/403: write d5/d6a/f65 [3256430,122086] 0 2024-04-12T05:37:21.673 INFO:tasks.workunit.client.1.smithi067.stdout:6/417: write dd/d21/d64/d88/f32 [667822,6837] 0 2024-04-12T05:37:21.673 INFO:tasks.workunit.client.0.smithi032.stdout:1/870: rmdir d5/de/d17/d35/d61/d77/d8f/d115 0 2024-04-12T05:37:21.676 INFO:tasks.workunit.client.1.smithi067.stdout:1/354: unlink d4/d9/d3a/f23 0 2024-04-12T05:37:21.677 INFO:tasks.workunit.client.0.smithi032.stdout:4/980: rename d4/d87/d96/d2f/d94/c9c to d4/d87/d96/d4d/db5/c146 0 2024-04-12T05:37:21.677 INFO:tasks.workunit.client.1.smithi067.stdout:1/355: chown d4/d9/f17 2 1 2024-04-12T05:37:21.678 INFO:tasks.workunit.client.1.smithi067.stdout:4/268: write d8/d9/f43 [498062,32870] 0 2024-04-12T05:37:21.680 INFO:tasks.workunit.client.1.smithi067.stdout:7/373: rename d0/d67/d5c to d0/d2/dd/d79 0 2024-04-12T05:37:21.681 INFO:tasks.workunit.client.1.smithi067.stdout:9/304: mkdir d1/d5f 0 2024-04-12T05:37:21.683 INFO:tasks.workunit.client.0.smithi032.stdout:1/871: dwrite d5/d2c/d4f/fb9 [0,4194304] 0 2024-04-12T05:37:21.684 INFO:tasks.workunit.client.0.smithi032.stdout:4/981: link d4/d13/d19/l27 d4/d87/d96/d2f/d7c/dff/l147 0 2024-04-12T05:37:21.684 INFO:tasks.workunit.client.1.smithi067.stdout:8/332: mkdir d6/d8/d18/d63 0 2024-04-12T05:37:21.685 INFO:tasks.workunit.client.1.smithi067.stdout:2/404: symlink d5/da/de/d14/d46/l79 0 2024-04-12T05:37:21.687 INFO:tasks.workunit.client.1.smithi067.stdout:6/418: truncate dd/d21/d64/d88/f62 333009 0 2024-04-12T05:37:21.689 INFO:tasks.workunit.client.1.smithi067.stdout:4/269: creat d8/da/d1d/f62 x:0 0 0 2024-04-12T05:37:21.691 INFO:tasks.workunit.client.0.smithi032.stdout:4/982: creat d4/d87/d96/d2f/d3b/da2/d121/f148 x:0 0 0 2024-04-12T05:37:21.692 INFO:tasks.workunit.client.0.smithi032.stdout:1/872: creat d5/de/d17/d93/d10c/d111/f117 x:0 0 0 2024-04-12T05:37:21.694 INFO:tasks.workunit.client.1.smithi067.stdout:2/405: mkdir d5/d11/d7a 0 2024-04-12T05:37:21.695 INFO:tasks.workunit.client.1.smithi067.stdout:6/419: symlink dd/d21/d23/l8d 0 2024-04-12T05:37:21.698 INFO:tasks.workunit.client.0.smithi032.stdout:4/983: getdents d4/d87/d96/d2f/d3b/da2 0 2024-04-12T05:37:21.699 INFO:tasks.workunit.client.1.smithi067.stdout:8/333: symlink d6/d2f/d5d/l64 0 2024-04-12T05:37:21.700 INFO:tasks.workunit.client.1.smithi067.stdout:2/406: creat d5/da/de/d14/f7b x:0 0 0 2024-04-12T05:37:21.701 INFO:tasks.workunit.client.0.smithi032.stdout:4/984: truncate d4/f1d 3308696 0 2024-04-12T05:37:21.703 INFO:tasks.workunit.client.1.smithi067.stdout:1/356: write d4/d9/d3a/d51/f63 [1864979,113785] 0 2024-04-12T05:37:21.704 INFO:tasks.workunit.client.1.smithi067.stdout:7/374: link d0/d2/dd/d79/c75 d0/d2/d5/d1d/d43/c7a 0 2024-04-12T05:37:21.704 INFO:tasks.workunit.client.1.smithi067.stdout:9/305: dread d1/d12/f2e [0,4194304] 0 2024-04-12T05:37:21.706 INFO:tasks.workunit.client.1.smithi067.stdout:9/306: dread - d1/d28/d23/f31 zero size 2024-04-12T05:37:21.706 INFO:tasks.workunit.client.1.smithi067.stdout:8/334: write d6/d8/d18/d36/f59 [4978767,131070] 0 2024-04-12T05:37:21.706 INFO:tasks.workunit.client.1.smithi067.stdout:2/407: readlink d5/d23/d3a/l56 0 2024-04-12T05:37:21.708 INFO:tasks.workunit.client.0.smithi032.stdout:4/985: write d4/d13/f71 [2416296,101359] 0 2024-04-12T05:37:21.711 INFO:tasks.workunit.client.1.smithi067.stdout:9/307: read d1/d28/d23/f24 [1324043,53249] 0 2024-04-12T05:37:21.712 INFO:tasks.workunit.client.1.smithi067.stdout:2/408: creat d5/da/de/d14/d34/d40/f7c x:0 0 0 2024-04-12T05:37:21.713 INFO:tasks.workunit.client.1.smithi067.stdout:8/335: rmdir d6/d8/d18/df/d12 39 2024-04-12T05:37:21.713 INFO:tasks.workunit.client.1.smithi067.stdout:2/409: stat d5/d43/d6e/f76 0 2024-04-12T05:37:21.714 INFO:tasks.workunit.client.0.smithi032.stdout:4/986: rename d4/d87/d96/d2f/d9a/f13f to d4/d87/d96/d4d/dbc/f149 0 2024-04-12T05:37:21.719 INFO:tasks.workunit.client.1.smithi067.stdout:6/420: fsync dd/d21/d64/d88/f62 0 2024-04-12T05:37:21.719 INFO:tasks.workunit.client.1.smithi067.stdout:2/410: write d5/da/de/d39/f6c [666034,33358] 0 2024-04-12T05:37:21.720 INFO:tasks.workunit.client.1.smithi067.stdout:6/421: read dd/d21/d64/d88/f32 [1092755,40964] 0 2024-04-12T05:37:21.721 INFO:tasks.workunit.client.0.smithi032.stdout:4/987: mknod d4/d87/d96/d2f/d9a/c14a 0 2024-04-12T05:37:21.721 INFO:tasks.workunit.client.1.smithi067.stdout:9/308: dwrite d1/d12/d15/f43 [0,4194304] 0 2024-04-12T05:37:21.721 INFO:tasks.workunit.client.0.smithi032.stdout:4/988: dread d4/d87/d96/d2f/d9a/fa7 [0,4194304] 0 2024-04-12T05:37:21.722 INFO:tasks.workunit.client.1.smithi067.stdout:8/336: rename d6/d8/d18/df/f5b to d6/d8/d18/d3c/d4f/f65 0 2024-04-12T05:37:21.724 INFO:tasks.workunit.client.1.smithi067.stdout:9/309: creat d1/d28/d21/d3f/d4f/f60 x:0 0 0 2024-04-12T05:37:21.726 INFO:tasks.workunit.client.1.smithi067.stdout:6/422: mknod dd/d21/d23/d5e/c8e 0 2024-04-12T05:37:21.728 INFO:tasks.workunit.client.0.smithi032.stdout:4/989: getdents d4/d87/d96/d2f/d6e 0 2024-04-12T05:37:21.729 INFO:tasks.workunit.client.1.smithi067.stdout:2/411: dread d5/da/de/d39/f6c [0,4194304] 0 2024-04-12T05:37:21.730 INFO:tasks.workunit.client.1.smithi067.stdout:6/423: rename dd/d21/d23/f36 to dd/d11/d27/f8f 0 2024-04-12T05:37:21.731 INFO:tasks.workunit.client.1.smithi067.stdout:9/310: truncate d1/f7 3373922 0 2024-04-12T05:37:21.736 INFO:tasks.workunit.client.1.smithi067.stdout:2/412: rename d5/da/de/f52 to d5/da/d3d/f7d 0 2024-04-12T05:37:21.737 INFO:tasks.workunit.client.1.smithi067.stdout:9/311: creat d1/d12/f61 x:0 0 0 2024-04-12T05:37:21.742 INFO:tasks.workunit.client.0.smithi032.stdout:4/990: dwrite d4/d13/d19/f30 [0,4194304] 0 2024-04-12T05:37:21.743 INFO:tasks.workunit.client.0.smithi032.stdout:4/991: write d4/f79 [1708583,96143] 0 2024-04-12T05:37:21.747 INFO:tasks.workunit.client.1.smithi067.stdout:8/337: truncate d6/d8/d18/df/d12/f1d 1051510 0 2024-04-12T05:37:21.748 INFO:tasks.workunit.client.1.smithi067.stdout:8/338: chown d6/d8 11603 1 2024-04-12T05:37:21.750 INFO:tasks.workunit.client.1.smithi067.stdout:9/312: mknod d1/d28/d5e/c62 0 2024-04-12T05:37:21.751 INFO:tasks.workunit.client.1.smithi067.stdout:8/339: write d6/d8/d18/d36/f54 [79312,122022] 0 2024-04-12T05:37:21.755 INFO:tasks.workunit.client.1.smithi067.stdout:2/413: rename d5/da/de/d14/d46/l79 to d5/l7e 0 2024-04-12T05:37:21.756 INFO:tasks.workunit.client.1.smithi067.stdout:9/313: creat d1/d28/d23/f63 x:0 0 0 2024-04-12T05:37:21.756 INFO:tasks.workunit.client.1.smithi067.stdout:8/340: chown d6/d8/d18/d4e 17 1 2024-04-12T05:37:21.757 INFO:tasks.workunit.client.1.smithi067.stdout:2/414: chown d5/d6a/f65 109 1 2024-04-12T05:37:21.760 INFO:tasks.workunit.client.1.smithi067.stdout:8/341: stat d6/d8/d18/df/d12/f1f 0 2024-04-12T05:37:21.761 INFO:tasks.workunit.client.1.smithi067.stdout:2/415: unlink d5/d6a/f77 0 2024-04-12T05:37:21.763 INFO:tasks.workunit.client.1.smithi067.stdout:8/342: dread - d6/d8/d18/f62 zero size 2024-04-12T05:37:21.766 INFO:tasks.workunit.client.1.smithi067.stdout:9/314: rmdir d1/d28 39 2024-04-12T05:37:21.767 INFO:tasks.workunit.client.1.smithi067.stdout:6/424: dwrite dd/f89 [0,4194304] 0 2024-04-12T05:37:21.768 INFO:tasks.workunit.client.1.smithi067.stdout:6/425: read f6 [4585031,99911] 0 2024-04-12T05:37:21.769 INFO:tasks.workunit.client.1.smithi067.stdout:8/343: write d6/d8/d18/f3f [2563263,24832] 0 2024-04-12T05:37:21.773 INFO:tasks.workunit.client.0.smithi032.stdout:4/992: rename d4/d87/d96/d4d/db5/dcf/de3/ff1 to d4/d87/d96/d4d/d78/f14b 0 2024-04-12T05:37:21.774 INFO:tasks.workunit.client.1.smithi067.stdout:9/315: write d1/d28/d23/f63 [627217,108005] 0 2024-04-12T05:37:21.774 INFO:tasks.workunit.client.1.smithi067.stdout:6/426: read - dd/d11/d27/f87 zero size 2024-04-12T05:37:21.775 INFO:tasks.workunit.client.0.smithi032.stdout:4/993: write d4/d87/d96/d4d/d78/f118 [354934,55606] 0 2024-04-12T05:37:21.776 INFO:tasks.workunit.client.1.smithi067.stdout:8/344: dread - d6/d2f/f5a zero size 2024-04-12T05:37:21.776 INFO:tasks.workunit.client.1.smithi067.stdout:6/427: readlink dd/d21/d64/d6f/l7e 0 2024-04-12T05:37:21.778 INFO:tasks.workunit.client.1.smithi067.stdout:6/428: write dd/d21/d23/f42 [4863918,39149] 0 2024-04-12T05:37:21.778 INFO:tasks.workunit.client.1.smithi067.stdout:9/316: symlink d1/l64 0 2024-04-12T05:37:21.780 INFO:tasks.workunit.client.1.smithi067.stdout:8/345: symlink d6/d8/d18/d4e/l66 0 2024-04-12T05:37:21.782 INFO:tasks.workunit.client.1.smithi067.stdout:9/317: truncate d1/d28/d23/f3d 1981741 0 2024-04-12T05:37:21.784 INFO:tasks.workunit.client.1.smithi067.stdout:9/318: dread - d1/d28/f50 zero size 2024-04-12T05:37:21.786 INFO:tasks.workunit.client.1.smithi067.stdout:6/429: truncate dd/d11/d27/d4a/f82 940962 0 2024-04-12T05:37:21.791 INFO:tasks.workunit.client.1.smithi067.stdout:2/416: dread d5/d11/f29 [0,4194304] 0 2024-04-12T05:37:21.793 INFO:tasks.workunit.client.1.smithi067.stdout:8/346: dwrite d6/d8/d18/f62 [0,4194304] 0 2024-04-12T05:37:21.796 INFO:tasks.workunit.client.1.smithi067.stdout:6/430: mkdir dd/d21/d64/d6f/d90 0 2024-04-12T05:37:21.796 INFO:tasks.workunit.client.1.smithi067.stdout:9/319: creat d1/d5f/f65 x:0 0 0 2024-04-12T05:37:21.801 INFO:tasks.workunit.client.1.smithi067.stdout:6/431: creat dd/d11/f91 x:0 0 0 2024-04-12T05:37:21.801 INFO:tasks.workunit.client.1.smithi067.stdout:9/320: mkdir d1/d66 0 2024-04-12T05:37:21.804 INFO:tasks.workunit.client.1.smithi067.stdout:9/321: rmdir d1 39 2024-04-12T05:37:21.811 INFO:tasks.workunit.client.1.smithi067.stdout:8/347: dwrite d6/d8/d18/df/d12/f55 [0,4194304] 0 2024-04-12T05:37:21.811 INFO:tasks.workunit.client.1.smithi067.stdout:6/432: stat f9 0 2024-04-12T05:37:21.813 INFO:tasks.workunit.client.1.smithi067.stdout:2/417: dwrite d5/da/de/d14/d34/d40/f7c [0,4194304] 0 2024-04-12T05:37:21.814 INFO:tasks.workunit.client.0.smithi032.stdout:4/994: dwrite d4/d87/d96/d2f/d3b/f3c [0,4194304] 0 2024-04-12T05:37:21.817 INFO:tasks.workunit.client.1.smithi067.stdout:6/433: mknod dd/d21/d64/d6f/c92 0 2024-04-12T05:37:21.819 INFO:tasks.workunit.client.1.smithi067.stdout:9/322: rename d1/d13/f3e to d1/d28/d5e/f67 0 2024-04-12T05:37:21.819 INFO:tasks.workunit.client.0.smithi032.stdout:4/995: rename d4/d87/d96/d4d/f92 to d4/d87/d96/d2f/d3b/da2/d10a/f14c 0 2024-04-12T05:37:21.827 INFO:tasks.workunit.client.0.smithi032.stdout:2/944: sync 2024-04-12T05:37:21.827 INFO:tasks.workunit.client.0.smithi032.stdout:4/996: mknod d4/d87/d96/d4d/db5/dcf/d114/d119/c14d 0 2024-04-12T05:37:21.828 INFO:tasks.workunit.client.1.smithi067.stdout:2/418: symlink d5/da/de/d14/d46/l7f 0 2024-04-12T05:37:21.828 INFO:tasks.workunit.client.1.smithi067.stdout:8/348: truncate d6/d8/d18/df/f20 3639341 0 2024-04-12T05:37:21.828 INFO:tasks.workunit.client.1.smithi067.stdout:2/419: chown d5/da/de/d14/l4b 214398 1 2024-04-12T05:37:21.828 INFO:tasks.workunit.client.1.smithi067.stdout:2/420: truncate d5/da/de/f13 1056323 0 2024-04-12T05:37:21.829 INFO:tasks.workunit.client.1.smithi067.stdout:9/323: mkdir d1/d28/d21/d68 0 2024-04-12T05:37:21.830 INFO:tasks.workunit.client.1.smithi067.stdout:9/324: read d1/d12/d15/f1c [672059,61261] 0 2024-04-12T05:37:21.831 INFO:tasks.workunit.client.0.smithi032.stdout:4/997: truncate d4/d87/d96/d2f/d3b/fc4 761088 0 2024-04-12T05:37:21.831 INFO:tasks.workunit.client.1.smithi067.stdout:8/349: mkdir d6/d8/d18/d63/d67 0 2024-04-12T05:37:21.832 INFO:tasks.workunit.client.0.smithi032.stdout:2/945: dread dd/d8c/fe2 [0,4194304] 0 2024-04-12T05:37:21.832 INFO:tasks.workunit.client.1.smithi067.stdout:2/421: symlink d5/d6a/d78/l80 0 2024-04-12T05:37:21.833 INFO:tasks.workunit.client.1.smithi067.stdout:9/325: unlink d1/d12/d15/c20 0 2024-04-12T05:37:21.833 INFO:tasks.workunit.client.0.smithi032.stdout:4/998: read d4/d87/d96/d32/f5f [4850857,8594] 0 2024-04-12T05:37:21.834 INFO:tasks.workunit.client.1.smithi067.stdout:8/350: creat d6/d8/d18/d36/f68 x:0 0 0 2024-04-12T05:37:21.835 INFO:tasks.workunit.client.1.smithi067.stdout:2/422: symlink d5/da/de/d14/d34/d40/d57/l81 0 2024-04-12T05:37:21.838 INFO:tasks.workunit.client.0.smithi032.stdout:2/946: dread dd/d19/d9d/d42/d4a/f100 [0,4194304] 0 2024-04-12T05:37:21.840 INFO:tasks.workunit.client.1.smithi067.stdout:8/351: rename d6/d2f to d6/d8/d18/d63/d67/d69 0 2024-04-12T05:37:21.841 INFO:tasks.workunit.client.1.smithi067.stdout:8/352: readlink d6/d8/d18/d3c/d4f/l58 0 2024-04-12T05:37:21.841 INFO:tasks.workunit.client.0.smithi032.stdout:2/947: mknod dd/d19/d9d/d42/d4a/da1/dd8/c12d 0 2024-04-12T05:37:21.841 INFO:tasks.workunit.client.1.smithi067.stdout:8/353: chown d6/d8/d18 453414 1 2024-04-12T05:37:21.843 INFO:tasks.workunit.client.1.smithi067.stdout:9/326: symlink d1/d28/d21/l69 0 2024-04-12T05:37:21.847 INFO:tasks.workunit.client.1.smithi067.stdout:8/354: rename d6/d8/d18/d4e/f60 to d6/d8/f6a 0 2024-04-12T05:37:21.847 INFO:tasks.workunit.client.0.smithi032.stdout:4/999: dread d4/d87/d96/d2f/d94/f127 [0,4194304] 0 2024-04-12T05:37:21.853 INFO:tasks.workunit.client.1.smithi067.stdout:2/423: dread d5/da/de/d14/d34/d4c/f5f [0,4194304] 0 2024-04-12T05:37:21.853 INFO:tasks.workunit.client.1.smithi067.stdout:2/424: chown d5/da/c30 857 1 2024-04-12T05:37:21.854 INFO:tasks.workunit.client.0.smithi032.stdout:2/948: dwrite dd/d19/d9d/d1c/d64/d36/ff3 [0,4194304] 0 2024-04-12T05:37:21.856 INFO:tasks.workunit.client.1.smithi067.stdout:9/327: truncate d1/d28/d5e/f67 3729223 0 2024-04-12T05:37:21.858 INFO:tasks.workunit.client.1.smithi067.stdout:2/425: write d5/d11/f50 [4867000,126337] 0 2024-04-12T05:37:21.859 INFO:tasks.workunit.client.1.smithi067.stdout:9/328: symlink d1/d13/l6a 0 2024-04-12T05:37:21.861 INFO:tasks.workunit.client.1.smithi067.stdout:9/329: dread - d1/d28/d23/f31 zero size 2024-04-12T05:37:21.862 INFO:tasks.workunit.client.1.smithi067.stdout:2/426: mknod d5/d23/c82 0 2024-04-12T05:37:21.863 INFO:tasks.workunit.client.0.smithi032.stdout:2/949: dwrite dd/d19/d9d/d1c/d64/d92/f125 [0,4194304] 0 2024-04-12T05:37:21.865 INFO:tasks.workunit.client.1.smithi067.stdout:2/427: symlink d5/da/l83 0 2024-04-12T05:37:21.866 INFO:tasks.workunit.client.1.smithi067.stdout:9/330: fdatasync d1/d28/d5e/f67 0 2024-04-12T05:37:21.867 INFO:tasks.workunit.client.1.smithi067.stdout:2/428: write d5/d11/f71 [974168,16766] 0 2024-04-12T05:37:21.867 INFO:tasks.workunit.client.0.smithi032.stdout:2/950: creat dd/d19/d9d/d1c/de0/de1/f12e x:0 0 0 2024-04-12T05:37:21.874 INFO:tasks.workunit.client.1.smithi067.stdout:2/429: dread d5/d23/d3a/f6f [0,4194304] 0 2024-04-12T05:37:21.875 INFO:tasks.workunit.client.0.smithi032.stdout:2/951: link dd/d118/l126 dd/d19/d9d/d1c/d64/d92/l12f 0 2024-04-12T05:37:21.876 INFO:tasks.workunit.client.1.smithi067.stdout:2/430: creat d5/d43/f84 x:0 0 0 2024-04-12T05:37:21.877 INFO:tasks.workunit.client.0.smithi032.stdout:2/952: write dd/d19/d9d/d1c/de0/f11f [694817,1839] 0 2024-04-12T05:37:21.882 INFO:tasks.workunit.client.1.smithi067.stdout:2/431: dread d5/d23/d3a/f59 [0,4194304] 0 2024-04-12T05:37:21.882 INFO:tasks.workunit.client.1.smithi067.stdout:9/331: dwrite d1/d28/d21/f26 [0,4194304] 0 2024-04-12T05:37:21.882 INFO:tasks.workunit.client.0.smithi032.stdout:2/953: rmdir dd/d19/d9d/d1c/de0/de1 39 2024-04-12T05:37:21.885 INFO:tasks.workunit.client.0.smithi032.stdout:2/954: mknod dd/d19/d9d/d42/d4a/da1/c130 0 2024-04-12T05:37:21.886 INFO:tasks.workunit.client.1.smithi067.stdout:2/432: creat d5/d6a/f85 x:0 0 0 2024-04-12T05:37:21.889 INFO:tasks.workunit.client.1.smithi067.stdout:2/433: mknod d5/d6a/c86 0 2024-04-12T05:37:21.890 INFO:tasks.workunit.client.1.smithi067.stdout:9/332: dwrite d1/d12/d15/f5c [0,4194304] 0 2024-04-12T05:37:21.891 INFO:tasks.workunit.client.0.smithi032.stdout:2/955: dread dd/d19/d9d/d42/da7/dc5/f103 [0,4194304] 0 2024-04-12T05:37:21.892 INFO:tasks.workunit.client.1.smithi067.stdout:2/434: truncate d5/d43/d6e/f76 1205709 0 2024-04-12T05:37:21.892 INFO:tasks.workunit.client.1.smithi067.stdout:9/333: read d1/d28/d23/f24 [2592733,39631] 0 2024-04-12T05:37:21.892 INFO:tasks.workunit.client.1.smithi067.stdout:2/435: readlink d5/d23/d3a/l70 0 2024-04-12T05:37:21.893 INFO:tasks.workunit.client.0.smithi032.stdout:2/956: write dd/d19/d9d/d1c/f81 [6786152,109320] 0 2024-04-12T05:37:21.896 INFO:tasks.workunit.client.1.smithi067.stdout:2/436: creat d5/d11/f87 x:0 0 0 2024-04-12T05:37:21.898 INFO:tasks.workunit.client.0.smithi032.stdout:2/957: creat dd/d19/d9d/d1c/de0/f131 x:0 0 0 2024-04-12T05:37:21.899 INFO:tasks.workunit.client.1.smithi067.stdout:2/437: dread d5/d43/f44 [0,4194304] 0 2024-04-12T05:37:21.899 INFO:tasks.workunit.client.1.smithi067.stdout:2/438: chown d5 4104 1 2024-04-12T05:37:21.901 INFO:tasks.workunit.client.1.smithi067.stdout:2/439: write d5/d43/f53 [2837158,116610] 0 2024-04-12T05:37:21.902 INFO:tasks.workunit.client.1.smithi067.stdout:9/334: dwrite d1/d28/d23/f3d [0,4194304] 0 2024-04-12T05:37:21.904 INFO:tasks.workunit.client.0.smithi032.stdout:2/958: dwrite dd/d19/d20/f62 [0,4194304] 0 2024-04-12T05:37:21.905 INFO:tasks.workunit.client.1.smithi067.stdout:2/440: creat d5/d6a/f88 x:0 0 0 2024-04-12T05:37:21.905 INFO:tasks.workunit.client.0.smithi032.stdout:2/959: write dd/d19/d82/f10a [198709,120380] 0 2024-04-12T05:37:21.906 INFO:tasks.workunit.client.1.smithi067.stdout:9/335: unlink d1/c36 0 2024-04-12T05:37:21.908 INFO:tasks.workunit.client.1.smithi067.stdout:2/441: creat d5/d6a/d78/f89 x:0 0 0 2024-04-12T05:37:21.909 INFO:tasks.workunit.client.0.smithi032.stdout:2/960: mknod dd/d19/d9d/d42/da7/c132 0 2024-04-12T05:37:21.911 INFO:tasks.workunit.client.1.smithi067.stdout:2/442: rmdir d5/da/d3d 39 2024-04-12T05:37:21.913 INFO:tasks.workunit.client.1.smithi067.stdout:9/336: dread d1/f47 [0,4194304] 0 2024-04-12T05:37:21.915 INFO:tasks.workunit.client.1.smithi067.stdout:2/443: truncate d5/d11/f71 1087439 0 2024-04-12T05:37:21.917 INFO:tasks.workunit.client.1.smithi067.stdout:2/444: creat d5/d11/f8a x:0 0 0 2024-04-12T05:37:21.919 INFO:tasks.workunit.client.1.smithi067.stdout:9/337: creat d1/d28/d21/f6b x:0 0 0 2024-04-12T05:37:21.920 INFO:tasks.workunit.client.1.smithi067.stdout:9/338: chown d1/d28/d5e 4 1 2024-04-12T05:37:21.920 INFO:tasks.workunit.client.1.smithi067.stdout:2/445: symlink d5/da/de/d14/d46/l8b 0 2024-04-12T05:37:21.920 INFO:tasks.workunit.client.1.smithi067.stdout:9/339: chown d1/d28/c1f 58 1 2024-04-12T05:37:21.922 INFO:tasks.workunit.client.1.smithi067.stdout:2/446: write d5/da/d3d/f6d [992771,99798] 0 2024-04-12T05:37:21.923 INFO:tasks.workunit.client.1.smithi067.stdout:9/340: unlink d1/d28/f1b 0 2024-04-12T05:37:21.923 INFO:tasks.workunit.client.1.smithi067.stdout:2/447: readlink d5/d23/d3a/l56 0 2024-04-12T05:37:21.924 INFO:tasks.workunit.client.1.smithi067.stdout:2/448: fdatasync d5/d43/f53 0 2024-04-12T05:37:21.925 INFO:tasks.workunit.client.1.smithi067.stdout:9/341: truncate d1/f7 4807540 0 2024-04-12T05:37:21.925 INFO:tasks.workunit.client.1.smithi067.stdout:9/342: chown d1/cc 259650 1 2024-04-12T05:37:21.926 INFO:tasks.workunit.client.1.smithi067.stdout:9/343: write d1/d28/d5e/f67 [1397108,69184] 0 2024-04-12T05:37:21.928 INFO:tasks.workunit.client.1.smithi067.stdout:9/344: creat d1/d13/d34/f6c x:0 0 0 2024-04-12T05:37:21.930 INFO:tasks.workunit.client.0.smithi032.stdout:2/961: dwrite dd/d19/d9d/d1c/d64/d32/d6d/f7f [0,4194304] 0 2024-04-12T05:37:21.930 INFO:tasks.workunit.client.1.smithi067.stdout:9/345: symlink d1/d28/d21/d45/l6d 0 2024-04-12T05:37:21.932 INFO:tasks.workunit.client.1.smithi067.stdout:9/346: fsync d1/d28/f50 0 2024-04-12T05:37:21.937 INFO:tasks.workunit.client.1.smithi067.stdout:9/347: getdents d1/d5f 0 2024-04-12T05:37:21.937 INFO:tasks.workunit.client.1.smithi067.stdout:9/348: chown d1/d28/l30 25112081 1 2024-04-12T05:37:21.939 INFO:tasks.workunit.client.1.smithi067.stdout:9/349: write d1/d28/d21/d3f/f41 [3181584,45502] 0 2024-04-12T05:37:21.939 INFO:tasks.workunit.client.0.smithi032.stdout:2/962: link dd/d19/d20/l5b dd/d19/d9d/d42/l133 0 2024-04-12T05:37:21.944 INFO:tasks.workunit.client.1.smithi067.stdout:9/350: rmdir d1/d28/d21 39 2024-04-12T05:37:21.945 INFO:tasks.workunit.client.0.smithi032.stdout:2/963: dwrite dd/d19/d9d/d42/da7/dc5/fd9 [0,4194304] 0 2024-04-12T05:37:21.947 INFO:tasks.workunit.client.1.smithi067.stdout:9/351: unlink d1/d28/d27/d32/c3a 0 2024-04-12T05:37:21.947 INFO:tasks.workunit.client.0.smithi032.stdout:2/964: write dd/d19/d9d/d1c/d64/d92/dfa/f11c [1824963,52703] 0 2024-04-12T05:37:21.950 INFO:tasks.workunit.client.0.smithi032.stdout:2/965: write dd/d19/d9d/d1c/d64/d92/f125 [1855553,10220] 0 2024-04-12T05:37:21.951 INFO:tasks.workunit.client.1.smithi067.stdout:9/352: creat d1/d66/f6e x:0 0 0 2024-04-12T05:37:21.953 INFO:tasks.workunit.client.1.smithi067.stdout:9/353: stat d1/d28/d21/c38 0 2024-04-12T05:37:21.955 INFO:tasks.workunit.client.1.smithi067.stdout:9/354: chown d1/d28/d21/l54 5526853 1 2024-04-12T05:37:21.956 INFO:tasks.workunit.client.0.smithi032.stdout:2/966: link dd/d19/d9d/d42/da7/dc5/fd9 dd/d19/d9d/d1c/f134 0 2024-04-12T05:37:21.959 INFO:tasks.workunit.client.0.smithi032.stdout:2/967: creat dd/d118/f135 x:0 0 0 2024-04-12T05:37:21.960 INFO:tasks.workunit.client.1.smithi067.stdout:9/355: creat d1/d28/d5e/f6f x:0 0 0 2024-04-12T05:37:21.960 INFO:tasks.workunit.client.0.smithi032.stdout:2/968: read - dd/d19/d9d/d1c/d64/d92/d10c/f11d zero size 2024-04-12T05:37:21.963 INFO:tasks.workunit.client.1.smithi067.stdout:9/356: unlink d1/d28/d21/d45/f4d 0 2024-04-12T05:37:21.964 INFO:tasks.workunit.client.1.smithi067.stdout:9/357: write d1/d13/d34/f6c [952076,73676] 0 2024-04-12T05:37:21.966 INFO:tasks.workunit.client.0.smithi032.stdout:2/969: write dd/d19/d20/f4d [5891085,66862] 0 2024-04-12T05:37:21.968 INFO:tasks.workunit.client.1.smithi067.stdout:9/358: unlink d1/d28/d21/f6b 0 2024-04-12T05:37:21.969 INFO:tasks.workunit.client.0.smithi032.stdout:2/970: mkdir dd/d19/d9d/d1c/d64/d36/d136 0 2024-04-12T05:37:21.970 INFO:tasks.workunit.client.1.smithi067.stdout:9/359: dread d1/d28/f18 [0,4194304] 0 2024-04-12T05:37:21.971 INFO:tasks.workunit.client.0.smithi032.stdout:2/971: symlink dd/d19/d9d/d1c/l137 0 2024-04-12T05:37:21.972 INFO:tasks.workunit.client.1.smithi067.stdout:9/360: unlink d1/d28/l49 0 2024-04-12T05:37:21.977 INFO:tasks.workunit.client.0.smithi032.stdout:2/972: truncate dd/d19/d9d/d1c/d64/d92/dfa/f11c 1861695 0 2024-04-12T05:37:21.981 INFO:tasks.workunit.client.0.smithi032.stdout:2/973: getdents dd/d19/d9d/d1c/d64/d92/d10c 0 2024-04-12T05:37:21.984 INFO:tasks.workunit.client.0.smithi032.stdout:2/974: mknod dd/d19/d9d/d1c/d64/d32/d6d/c138 0 2024-04-12T05:37:21.985 INFO:tasks.workunit.client.1.smithi067.stdout:9/361: dread d1/d12/d15/f1c [0,4194304] 0 2024-04-12T05:37:21.988 INFO:tasks.workunit.client.0.smithi032.stdout:2/975: link dd/d19/d9d/d1c/d64/d92/d10c/f11d dd/d8c/f139 0 2024-04-12T05:37:21.989 INFO:tasks.workunit.client.1.smithi067.stdout:9/362: dread d1/d12/d15/f1d [0,4194304] 0 2024-04-12T05:37:21.991 INFO:tasks.workunit.client.1.smithi067.stdout:9/363: write d1/d28/d5e/f67 [3760244,120633] 0 2024-04-12T05:37:21.993 INFO:tasks.workunit.client.1.smithi067.stdout:9/364: rmdir d1/d13/d34 39 2024-04-12T05:37:21.994 INFO:tasks.workunit.client.1.smithi067.stdout:9/365: dread - d1/d28/f50 zero size 2024-04-12T05:37:21.998 INFO:tasks.workunit.client.1.smithi067.stdout:9/366: creat d1/d13/d34/f70 x:0 0 0 2024-04-12T05:37:22.000 INFO:tasks.workunit.client.1.smithi067.stdout:9/367: chown d1/d12/d15/l46 16 1 2024-04-12T05:37:22.002 INFO:tasks.workunit.client.1.smithi067.stdout:9/368: creat d1/d28/d27/f71 x:0 0 0 2024-04-12T05:37:22.004 INFO:tasks.workunit.client.0.smithi032.stdout:2/976: dread dd/d19/d9d/d1c/f74 [0,4194304] 0 2024-04-12T05:37:22.006 INFO:tasks.workunit.client.0.smithi032.stdout:2/977: fsync dd/d19/d9d/d42/d4a/fc4 0 2024-04-12T05:37:22.018 INFO:tasks.workunit.client.1.smithi067.stdout:5/584: sync 2024-04-12T05:37:22.019 INFO:tasks.workunit.client.1.smithi067.stdout:5/585: write d0/d6/d20/d3e/f8a [78703,21006] 0 2024-04-12T05:37:22.024 INFO:tasks.workunit.client.1.smithi067.stdout:5/586: dread d0/d6/f10 [0,4194304] 0 2024-04-12T05:37:22.025 INFO:tasks.workunit.client.1.smithi067.stdout:5/587: readlink d0/da/d3f/la4 0 2024-04-12T05:37:22.027 INFO:tasks.workunit.client.1.smithi067.stdout:5/588: creat d0/d3a/d2c/d91/fc7 x:0 0 0 2024-04-12T05:37:22.058 INFO:tasks.workunit.client.0.smithi032.stdout:9/836: sync 2024-04-12T05:37:22.061 INFO:tasks.workunit.client.1.smithi067.stdout:5/589: dread d0/d6/d20/f58 [0,4194304] 0 2024-04-12T05:37:22.062 INFO:tasks.workunit.client.1.smithi067.stdout:5/590: stat d0/d6/d40/d4b/d38/f79 0 2024-04-12T05:37:22.067 INFO:tasks.workunit.client.1.smithi067.stdout:5/591: dread d0/da/f5a [0,4194304] 0 2024-04-12T05:37:22.072 INFO:tasks.workunit.client.1.smithi067.stdout:5/592: truncate d0/d6/d33/f71 1158866 0 2024-04-12T05:37:22.073 INFO:tasks.workunit.client.0.smithi032.stdout:9/837: dread d9/d15/f44 [0,4194304] 0 2024-04-12T05:37:22.075 INFO:tasks.workunit.client.1.smithi067.stdout:3/244: sync 2024-04-12T05:37:22.075 INFO:tasks.workunit.client.1.smithi067.stdout:5/593: unlink d0/d6/d20/d62/f14 0 2024-04-12T05:37:22.075 INFO:tasks.workunit.client.0.smithi032.stdout:9/838: mknod d9/d58/c122 0 2024-04-12T05:37:22.075 INFO:tasks.workunit.client.0.smithi032.stdout:9/839: stat d9/d15/d2c/d41/f96 0 2024-04-12T05:37:22.077 INFO:tasks.workunit.client.0.smithi032.stdout:9/840: fsync d9/d15/d1a/d1d/fb9 0 2024-04-12T05:37:22.079 INFO:tasks.workunit.client.0.smithi032.stdout:9/841: read d9/d15/d1a/d24/d27/d88/fa8 [28078,11317] 0 2024-04-12T05:37:22.084 INFO:tasks.workunit.client.0.smithi032.stdout:9/842: truncate d9/d15/d2c/dab/fc5 621986 0 2024-04-12T05:37:22.084 INFO:tasks.workunit.client.1.smithi067.stdout:3/245: truncate dc/d1d/f43 5923239 0 2024-04-12T05:37:22.087 INFO:tasks.workunit.client.0.smithi032.stdout:9/843: creat d9/d1f/f123 x:0 0 0 2024-04-12T05:37:22.089 INFO:tasks.workunit.client.0.smithi032.stdout:9/844: mkdir d9/df7/d124 0 2024-04-12T05:37:22.089 INFO:tasks.workunit.client.1.smithi067.stdout:3/246: creat dc/f48 x:0 0 0 2024-04-12T05:37:22.090 INFO:tasks.workunit.client.1.smithi067.stdout:5/594: dread d0/d6/f56 [0,4194304] 0 2024-04-12T05:37:22.090 INFO:tasks.workunit.client.0.smithi032.stdout:9/845: write d9/d15/d1a/d24/d27/f76 [102178,93170] 0 2024-04-12T05:37:22.091 INFO:tasks.workunit.client.1.smithi067.stdout:3/247: rmdir dc/d1d 39 2024-04-12T05:37:22.092 INFO:tasks.workunit.client.1.smithi067.stdout:5/595: rmdir d0/da/d3f/d82 39 2024-04-12T05:37:22.094 INFO:tasks.workunit.client.0.smithi032.stdout:9/846: dread d9/d15/d2c/d41/f45 [0,4194304] 0 2024-04-12T05:37:22.186 INFO:tasks.workunit.client.0.smithi032.stdout:1/873: sync 2024-04-12T05:37:22.189 INFO:tasks.workunit.client.0.smithi032.stdout:1/874: symlink d5/de/d17/d35/d3b/dda/de0/l118 0 2024-04-12T05:37:22.193 INFO:tasks.workunit.client.0.smithi032.stdout:1/875: dread d5/de/d17/d35/d61/d77/f7d [0,4194304] 0 2024-04-12T05:37:22.195 INFO:tasks.workunit.client.0.smithi032.stdout:1/876: write d5/de/d17/d1b/fea [1311335,55239] 0 2024-04-12T05:37:22.197 INFO:tasks.workunit.client.0.smithi032.stdout:1/877: truncate d5/fb8 224532 0 2024-04-12T05:37:22.198 INFO:tasks.workunit.client.0.smithi032.stdout:1/878: write d5/de/d17/d35/d61/f50 [2015652,2589] 0 2024-04-12T05:37:22.202 INFO:tasks.workunit.client.0.smithi032.stdout:1/879: link d5/de/d17/d35/d3e/l45 d5/de/d17/d35/d3b/l119 0 2024-04-12T05:37:22.213 INFO:tasks.workunit.client.0.smithi032.stdout:1/880: dwrite d5/de/d17/d1b/f21 [0,4194304] 0 2024-04-12T05:37:22.219 INFO:tasks.workunit.client.1.smithi067.stdout:3/248: fsync dc/d1d/f44 0 2024-04-12T05:37:22.220 INFO:tasks.workunit.client.0.smithi032.stdout:1/881: dwrite d5/de/d17/d35/ff8 [0,4194304] 0 2024-04-12T05:37:22.225 INFO:tasks.workunit.client.0.smithi032.stdout:1/882: creat d5/de/d17/d35/d61/d105/d108/f11a x:0 0 0 2024-04-12T05:37:22.226 INFO:tasks.workunit.client.1.smithi067.stdout:5/596: dwrite d0/da/d3f/d82/fb4 [0,4194304] 0 2024-04-12T05:37:22.230 INFO:tasks.workunit.client.1.smithi067.stdout:3/249: link dc/df/l35 dc/d1d/l49 0 2024-04-12T05:37:22.232 INFO:tasks.workunit.client.0.smithi032.stdout:1/883: dwrite d5/de/f10 [4194304,4194304] 0 2024-04-12T05:37:22.233 INFO:tasks.workunit.client.1.smithi067.stdout:3/250: fdatasync dc/df/f1e 0 2024-04-12T05:37:22.234 INFO:tasks.workunit.client.0.smithi032.stdout:1/884: stat d5/de/d17/d35/d3b/dda 0 2024-04-12T05:37:22.234 INFO:tasks.workunit.client.1.smithi067.stdout:3/251: dread - dc/d1d/d26/f46 zero size 2024-04-12T05:37:22.238 INFO:tasks.workunit.client.0.smithi032.stdout:1/885: creat d5/de/f11b x:0 0 0 2024-04-12T05:37:22.243 INFO:tasks.workunit.client.0.smithi032.stdout:1/886: mkdir d5/de/d17/d35/d61/d77/d8f/da0/d10f/d11c 0 2024-04-12T05:37:22.247 INFO:tasks.workunit.client.1.smithi067.stdout:0/427: sync 2024-04-12T05:37:22.247 INFO:tasks.workunit.client.1.smithi067.stdout:1/357: sync 2024-04-12T05:37:22.249 INFO:tasks.workunit.client.1.smithi067.stdout:0/428: symlink dd/d12/d14/d1a/d3d/d41/d64/l86 0 2024-04-12T05:37:22.251 INFO:tasks.workunit.client.1.smithi067.stdout:1/358: creat d4/d9/f75 x:0 0 0 2024-04-12T05:37:22.255 INFO:tasks.workunit.client.1.smithi067.stdout:0/429: write dd/d12/f16 [3070365,126454] 0 2024-04-12T05:37:22.256 INFO:tasks.workunit.client.0.smithi032.stdout:1/887: dwrite d5/de/f66 [0,4194304] 0 2024-04-12T05:37:22.258 INFO:tasks.workunit.client.0.smithi032.stdout:1/888: dread d5/de/d5b/fe7 [0,4194304] 0 2024-04-12T05:37:22.259 INFO:tasks.workunit.client.1.smithi067.stdout:1/359: dwrite d4/d9/d3a/d51/f63 [0,4194304] 0 2024-04-12T05:37:22.262 INFO:tasks.workunit.client.1.smithi067.stdout:1/360: read d4/d3d/f4b [3014568,47564] 0 2024-04-12T05:37:22.265 INFO:tasks.workunit.client.0.smithi032.stdout:1/889: symlink d5/de/d17/d35/d3e/l11d 0 2024-04-12T05:37:22.267 INFO:tasks.workunit.client.0.smithi032.stdout:2/978: sync 2024-04-12T05:37:22.270 INFO:tasks.workunit.client.0.smithi032.stdout:1/890: creat d5/de/d17/d35/d61/d77/d100/f11e x:0 0 0 2024-04-12T05:37:22.271 INFO:tasks.workunit.client.0.smithi032.stdout:1/891: stat d5/de/d17/fae 0 2024-04-12T05:37:22.271 INFO:tasks.workunit.client.0.smithi032.stdout:2/979: unlink dd/d19/d9d/d1c/de0/f131 0 2024-04-12T05:37:22.271 INFO:tasks.workunit.client.1.smithi067.stdout:3/252: dwrite dc/d1d/d28/f2a [0,4194304] 0 2024-04-12T05:37:22.272 INFO:tasks.workunit.client.0.smithi032.stdout:2/980: fdatasync dd/d19/f107 0 2024-04-12T05:37:22.272 INFO:tasks.workunit.client.1.smithi067.stdout:1/361: dwrite d4/d9/f17 [0,4194304] 0 2024-04-12T05:37:22.277 INFO:tasks.workunit.client.0.smithi032.stdout:2/981: rename dd/d19/d9d/d1c/f134 to dd/d19/d82/f13a 0 2024-04-12T05:37:22.277 INFO:tasks.workunit.client.1.smithi067.stdout:3/253: creat dc/d1d/d28/d3e/f4a x:0 0 0 2024-04-12T05:37:22.278 INFO:tasks.workunit.client.0.smithi032.stdout:2/982: write dd/d19/d9d/d42/f80 [3059076,30375] 0 2024-04-12T05:37:22.279 INFO:tasks.workunit.client.0.smithi032.stdout:1/892: dread d5/de/d17/d112/fde [0,4194304] 0 2024-04-12T05:37:22.279 INFO:tasks.workunit.client.1.smithi067.stdout:1/362: getdents d4/d46 0 2024-04-12T05:37:22.280 INFO:tasks.workunit.client.0.smithi032.stdout:2/983: read - dd/d118/f135 zero size 2024-04-12T05:37:22.281 INFO:tasks.workunit.client.1.smithi067.stdout:3/254: symlink dc/df/l4b 0 2024-04-12T05:37:22.281 INFO:tasks.workunit.client.0.smithi032.stdout:2/984: chown dd/d19/d9d/d1c/d64/d32/d3a/f56 8175 1 2024-04-12T05:37:22.283 INFO:tasks.workunit.client.1.smithi067.stdout:3/255: creat dc/d1d/d28/f4c x:0 0 0 2024-04-12T05:37:22.284 INFO:tasks.workunit.client.0.smithi032.stdout:2/985: symlink dd/d19/d9d/d1c/d64/d92/l13b 0 2024-04-12T05:37:22.285 INFO:tasks.workunit.client.0.smithi032.stdout:2/986: chown dd/d19/d9d/d1c/d64/d32/d3a/d8d 1226286 1 2024-04-12T05:37:22.286 INFO:tasks.workunit.client.1.smithi067.stdout:1/363: link d4/d9/c54 d4/d9/d12/c76 0 2024-04-12T05:37:22.288 INFO:tasks.workunit.client.0.smithi032.stdout:2/987: dread - dd/d19/d9d/d1c/d64/d36/ffc zero size 2024-04-12T05:37:22.288 INFO:tasks.workunit.client.1.smithi067.stdout:3/256: mkdir dc/d1d/d4d 0 2024-04-12T05:37:22.289 INFO:tasks.workunit.client.1.smithi067.stdout:0/430: dread dd/d12/f2e [0,4194304] 0 2024-04-12T05:37:22.290 INFO:tasks.workunit.client.1.smithi067.stdout:3/257: mknod dc/d1d/d4d/c4e 0 2024-04-12T05:37:22.296 INFO:tasks.workunit.client.1.smithi067.stdout:1/364: dwrite d4/d44/f66 [0,4194304] 0 2024-04-12T05:37:22.296 INFO:tasks.workunit.client.1.smithi067.stdout:3/258: link dc/df/l25 dc/d1d/d28/l4f 0 2024-04-12T05:37:22.298 INFO:tasks.workunit.client.1.smithi067.stdout:1/365: creat d4/d3d/f77 x:0 0 0 2024-04-12T05:37:22.298 INFO:tasks.workunit.client.1.smithi067.stdout:3/259: mkdir dc/d50 0 2024-04-12T05:37:22.302 INFO:tasks.workunit.client.1.smithi067.stdout:3/260: mknod dc/d1d/d28/d3e/c51 0 2024-04-12T05:37:22.302 INFO:tasks.workunit.client.1.smithi067.stdout:0/431: dread dd/d12/f2a [0,4194304] 0 2024-04-12T05:37:22.303 INFO:tasks.workunit.client.1.smithi067.stdout:1/366: creat d4/d9/f78 x:0 0 0 2024-04-12T05:37:22.304 INFO:tasks.workunit.client.1.smithi067.stdout:3/261: mkdir dc/d1d/d26/d52 0 2024-04-12T05:37:22.306 INFO:tasks.workunit.client.1.smithi067.stdout:0/432: rename dd/d12/l36 to dd/d12/d14/d51/l87 0 2024-04-12T05:37:22.310 INFO:tasks.workunit.client.1.smithi067.stdout:1/367: rmdir d4/d9/d4c/d5d/d73 0 2024-04-12T05:37:22.314 INFO:tasks.workunit.client.1.smithi067.stdout:0/433: dwrite dd/d12/d14/d1a/f1e [4194304,4194304] 0 2024-04-12T05:37:22.315 INFO:tasks.workunit.client.1.smithi067.stdout:1/368: dread d4/d46/f58 [0,4194304] 0 2024-04-12T05:37:22.317 INFO:tasks.workunit.client.1.smithi067.stdout:0/434: truncate dd/d12/d14/d1a/d3d/d41/d64/f7e 523077 0 2024-04-12T05:37:22.318 INFO:tasks.workunit.client.1.smithi067.stdout:3/262: truncate dc/d1d/d26/f2f 2210985 0 2024-04-12T05:37:22.322 INFO:tasks.workunit.client.1.smithi067.stdout:3/263: unlink dc/df/c17 0 2024-04-12T05:37:22.323 INFO:tasks.workunit.client.1.smithi067.stdout:0/435: mknod dd/d12/d14/d1a/d3d/d41/d64/c88 0 2024-04-12T05:37:22.324 INFO:tasks.workunit.client.1.smithi067.stdout:0/436: chown dd/d12/d14/d1a/c7d 6 1 2024-04-12T05:37:22.325 INFO:tasks.workunit.client.1.smithi067.stdout:1/369: rename d4/fe to d4/d9/d14/d61/f79 0 2024-04-12T05:37:22.326 INFO:tasks.workunit.client.1.smithi067.stdout:3/264: rmdir dc/d1d/d28/d3e 39 2024-04-12T05:37:22.329 INFO:tasks.workunit.client.1.smithi067.stdout:3/265: mkdir dc/d1d/d28/d3e/d53 0 2024-04-12T05:37:22.340 INFO:tasks.workunit.client.1.smithi067.stdout:0/437: dread dd/d12/d14/d1a/d75/f67 [0,4194304] 0 2024-04-12T05:37:22.341 INFO:tasks.workunit.client.1.smithi067.stdout:0/438: stat c6 0 2024-04-12T05:37:22.341 INFO:tasks.workunit.client.1.smithi067.stdout:0/439: chown dd/d5b/l63 249492 1 2024-04-12T05:37:22.342 INFO:tasks.workunit.client.1.smithi067.stdout:0/440: write dd/d12/d14/d1a/d3d/f32 [71756,40978] 0 2024-04-12T05:37:22.376 INFO:tasks.workunit.client.1.smithi067.stdout:7/375: sync 2024-04-12T05:37:22.377 INFO:tasks.workunit.client.1.smithi067.stdout:4/270: sync 2024-04-12T05:37:22.377 INFO:tasks.workunit.client.1.smithi067.stdout:6/434: sync 2024-04-12T05:37:22.377 INFO:tasks.workunit.client.1.smithi067.stdout:4/271: chown d8/d39/f46 10262 1 2024-04-12T05:37:22.386 INFO:tasks.workunit.client.0.smithi032.stdout:9/847: sync 2024-04-12T05:37:22.387 INFO:tasks.workunit.client.0.smithi032.stdout:9/848: chown d9/d15/d1a/c4c 0 1 2024-04-12T05:37:22.389 INFO:tasks.workunit.client.0.smithi032.stdout:9/849: creat d9/d32/d86/f125 x:0 0 0 2024-04-12T05:37:22.392 INFO:tasks.workunit.client.0.smithi032.stdout:9/850: rename d9/d15/d1a/d24/d27/dce/dd3/d109/c112 to d9/d15/d1a/d69/c126 0 2024-04-12T05:37:22.404 INFO:tasks.workunit.client.0.smithi032.stdout:9/851: write d9/d1f/f5d [2293888,71484] 0 2024-04-12T05:37:22.405 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:22 smithi067 ceph-mon[49010]: pgmap v272: 65 pgs: 65 active+clean; 2.4 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 83 MiB/s rd, 207 MiB/s wr, 433 op/s 2024-04-12T05:37:22.405 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:22 smithi067 ceph-mon[49010]: mgrmap e21: smithi032.hrwxrh(active, since 7m), standbys: smithi067.dwichg 2024-04-12T05:37:22.405 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:22 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr metadata", "who": "smithi067.dwichg", "id": "smithi067.dwichg"}]: dispatch 2024-04-12T05:37:22.406 INFO:tasks.workunit.client.0.smithi032.stdout:9/852: creat d9/d15/d1a/d1d/f127 x:0 0 0 2024-04-12T05:37:22.413 INFO:tasks.workunit.client.0.smithi032.stdout:9/853: dwrite d9/d32/f53 [0,4194304] 0 2024-04-12T05:37:22.420 INFO:tasks.workunit.client.0.smithi032.stdout:9/854: dwrite d9/d15/f48 [0,4194304] 0 2024-04-12T05:37:22.423 INFO:tasks.workunit.client.1.smithi067.stdout:3/266: rename dc/df/l20 to dc/d50/l54 0 2024-04-12T05:37:22.429 INFO:tasks.workunit.client.1.smithi067.stdout:0/441: fdatasync dd/d12/d14/d1a/f4b 0 2024-04-12T05:37:22.431 INFO:tasks.workunit.client.0.smithi032.stdout:9/855: dwrite d9/d15/d1a/d24/fed [0,4194304] 0 2024-04-12T05:37:22.432 INFO:tasks.workunit.client.1.smithi067.stdout:1/370: dread d4/d9/d14/d61/f79 [0,4194304] 0 2024-04-12T05:37:22.433 INFO:tasks.workunit.client.1.smithi067.stdout:1/371: fsync d4/d9/f4a 0 2024-04-12T05:37:22.435 INFO:tasks.workunit.client.0.smithi032.stdout:9/856: mknod d9/d32/d6e/d114/c128 0 2024-04-12T05:37:22.435 INFO:tasks.workunit.client.1.smithi067.stdout:1/372: read d4/d9/d12/f53 [218060,30690] 0 2024-04-12T05:37:22.442 INFO:tasks.workunit.client.1.smithi067.stdout:6/435: mkdir dd/d21/d23/d80/d93 0 2024-04-12T05:37:22.444 INFO:tasks.workunit.client.1.smithi067.stdout:3/267: link dc/d1d/f44 dc/d1d/d4d/f55 0 2024-04-12T05:37:22.448 INFO:tasks.workunit.client.1.smithi067.stdout:0/442: dread dd/d12/f3e [0,4194304] 0 2024-04-12T05:37:22.449 INFO:tasks.workunit.client.1.smithi067.stdout:0/443: stat dd/f38 0 2024-04-12T05:37:22.450 INFO:tasks.workunit.client.1.smithi067.stdout:6/436: rename dd/d21/d3f/l4f to dd/d21/d3f/l94 0 2024-04-12T05:37:22.450 INFO:tasks.workunit.client.0.smithi032.stdout:9/857: dwrite d9/d1f/fe0 [0,4194304] 0 2024-04-12T05:37:22.451 INFO:tasks.workunit.client.1.smithi067.stdout:3/268: mknod dc/d1d/d28/d3e/c56 0 2024-04-12T05:37:22.454 INFO:tasks.workunit.client.1.smithi067.stdout:1/373: mkdir d4/d9/d14/d7a 0 2024-04-12T05:37:22.454 INFO:tasks.workunit.client.1.smithi067.stdout:7/376: dread d0/d2/d5/db/f1f [0,4194304] 0 2024-04-12T05:37:22.455 INFO:tasks.workunit.client.1.smithi067.stdout:0/444: creat dd/d12/d14/d1a/d3d/d41/f89 x:0 0 0 2024-04-12T05:37:22.456 INFO:tasks.workunit.client.1.smithi067.stdout:7/377: write d0/d23/f6e [2671200,126230] 0 2024-04-12T05:37:22.457 INFO:tasks.workunit.client.1.smithi067.stdout:6/437: creat dd/d11/d27/d4a/f95 x:0 0 0 2024-04-12T05:37:22.458 INFO:tasks.workunit.client.1.smithi067.stdout:3/269: symlink dc/df/d27/l57 0 2024-04-12T05:37:22.460 INFO:tasks.workunit.client.1.smithi067.stdout:1/374: mkdir d4/d3d/d7b 0 2024-04-12T05:37:22.463 INFO:tasks.workunit.client.1.smithi067.stdout:6/438: fdatasync dd/d11/d27/f8f 0 2024-04-12T05:37:22.464 INFO:tasks.workunit.client.1.smithi067.stdout:3/270: write dc/d1d/d4d/f55 [1471945,24169] 0 2024-04-12T05:37:22.464 INFO:tasks.workunit.client.1.smithi067.stdout:1/375: unlink d4/d46/f5b 0 2024-04-12T05:37:22.465 INFO:tasks.workunit.client.1.smithi067.stdout:0/445: creat dd/d12/d14/d1a/d75/f8a x:0 0 0 2024-04-12T05:37:22.466 INFO:tasks.workunit.client.1.smithi067.stdout:6/439: rmdir dd/d21/d3f 39 2024-04-12T05:37:22.467 INFO:tasks.workunit.client.1.smithi067.stdout:7/378: dwrite d0/d2/dd/d79/f63 [0,4194304] 0 2024-04-12T05:37:22.468 INFO:tasks.workunit.client.1.smithi067.stdout:1/376: read - d4/d9/d12/d28/f32 zero size 2024-04-12T05:37:22.471 INFO:tasks.workunit.client.1.smithi067.stdout:3/271: getdents dc/d50 0 2024-04-12T05:37:22.472 INFO:tasks.workunit.client.1.smithi067.stdout:0/446: truncate dd/d12/d14/d1a/d75/f70 165407 0 2024-04-12T05:37:22.478 INFO:tasks.workunit.client.1.smithi067.stdout:6/440: getdents dd/d11/d27/d4a 0 2024-04-12T05:37:22.479 INFO:tasks.workunit.client.1.smithi067.stdout:3/272: creat dc/d1d/d28/d3e/d53/f58 x:0 0 0 2024-04-12T05:37:22.480 INFO:tasks.workunit.client.1.smithi067.stdout:0/447: rmdir dd 39 2024-04-12T05:37:22.484 INFO:tasks.workunit.client.1.smithi067.stdout:6/441: chown dd/d21/d66/c74 781029 1 2024-04-12T05:37:22.484 INFO:tasks.workunit.client.1.smithi067.stdout:3/273: readlink dc/d1d/l49 0 2024-04-12T05:37:22.485 INFO:tasks.workunit.client.1.smithi067.stdout:7/379: truncate d0/d2/d5/fe 866526 0 2024-04-12T05:37:22.490 INFO:tasks.workunit.client.1.smithi067.stdout:1/377: dwrite d4/d9/d12/f1b [4194304,4194304] 0 2024-04-12T05:37:22.491 INFO:tasks.workunit.client.1.smithi067.stdout:0/448: chown dd/d12/d14/d1a/d3d/d41/d5f/d78/d83/c85 1 1 2024-04-12T05:37:22.493 INFO:tasks.workunit.client.1.smithi067.stdout:6/442: link dd/d11/f5d dd/d21/d64/d88/f96 0 2024-04-12T05:37:22.496 INFO:tasks.workunit.client.1.smithi067.stdout:3/274: dwrite dc/d1d/d28/f4c [0,4194304] 0 2024-04-12T05:37:22.498 INFO:tasks.workunit.client.1.smithi067.stdout:1/378: rmdir d4/d9/d12/d28 39 2024-04-12T05:37:22.499 INFO:tasks.workunit.client.1.smithi067.stdout:1/379: read d4/d9/d12/f1b [5617568,117145] 0 2024-04-12T05:37:22.499 INFO:tasks.workunit.client.1.smithi067.stdout:0/449: creat dd/d6f/d72/f8b x:0 0 0 2024-04-12T05:37:22.501 INFO:tasks.workunit.client.1.smithi067.stdout:7/380: rename d0/d2/d5/db/l3c to d0/d2/dd/l7b 0 2024-04-12T05:37:22.505 INFO:tasks.workunit.client.1.smithi067.stdout:3/275: symlink dc/d50/l59 0 2024-04-12T05:37:22.505 INFO:tasks.workunit.client.1.smithi067.stdout:7/381: truncate d0/d2/f21 4584334 0 2024-04-12T05:37:22.506 INFO:tasks.workunit.client.1.smithi067.stdout:3/276: readlink dc/df/l4b 0 2024-04-12T05:37:22.507 INFO:tasks.workunit.client.1.smithi067.stdout:7/382: chown d0/d23/d45/f59 558523236 1 2024-04-12T05:37:22.509 INFO:tasks.workunit.client.1.smithi067.stdout:6/443: rename dd/d11/d27/l4d to dd/d21/d64/d88/l97 0 2024-04-12T05:37:22.511 INFO:tasks.workunit.client.1.smithi067.stdout:3/277: mkdir dc/d50/d5a 0 2024-04-12T05:37:22.512 INFO:tasks.workunit.client.1.smithi067.stdout:7/383: creat d0/d2/d5/d1e/f7c x:0 0 0 2024-04-12T05:37:22.514 INFO:tasks.workunit.client.1.smithi067.stdout:1/380: link d4/d9/d3a/d51/l38 d4/d9/d12/d2c/l7c 0 2024-04-12T05:37:22.515 INFO:tasks.workunit.client.1.smithi067.stdout:6/444: mkdir dd/d21/d64/d6f/d90/d98 0 2024-04-12T05:37:22.516 INFO:tasks.workunit.client.1.smithi067.stdout:0/450: getdents dd/d12/d14/d1a 0 2024-04-12T05:37:22.517 INFO:tasks.workunit.client.1.smithi067.stdout:6/445: write dd/d11/d27/d4a/f73 [1194055,61463] 0 2024-04-12T05:37:22.518 INFO:tasks.workunit.client.1.smithi067.stdout:3/278: rename dc/df/l15 to dc/d1d/d28/l5b 0 2024-04-12T05:37:22.519 INFO:tasks.workunit.client.1.smithi067.stdout:3/279: write dc/d1d/d28/f4c [2131389,42361] 0 2024-04-12T05:37:22.520 INFO:tasks.workunit.client.1.smithi067.stdout:0/451: symlink dd/d12/d14/d1a/d3d/l8c 0 2024-04-12T05:37:22.521 INFO:tasks.workunit.client.1.smithi067.stdout:1/381: read d4/d9/d12/d59/f64 [186516,43163] 0 2024-04-12T05:37:22.522 INFO:tasks.workunit.client.1.smithi067.stdout:8/355: sync 2024-04-12T05:37:22.522 INFO:tasks.workunit.client.1.smithi067.stdout:9/369: sync 2024-04-12T05:37:22.522 INFO:tasks.workunit.client.1.smithi067.stdout:5/597: sync 2024-04-12T05:37:22.522 INFO:tasks.workunit.client.1.smithi067.stdout:2/449: sync 2024-04-12T05:37:22.523 INFO:tasks.workunit.client.1.smithi067.stdout:0/452: stat dd/d12/d14/c23 0 2024-04-12T05:37:22.525 INFO:tasks.workunit.client.1.smithi067.stdout:6/446: link dd/d21/d3f/f55 dd/d21/d23/d56/f99 0 2024-04-12T05:37:22.525 INFO:tasks.workunit.client.1.smithi067.stdout:1/382: dread - d4/d9/d14/d61/f65 zero size 2024-04-12T05:37:22.526 INFO:tasks.workunit.client.1.smithi067.stdout:3/280: creat dc/d1d/d28/d3e/d53/f5c x:0 0 0 2024-04-12T05:37:22.527 INFO:tasks.workunit.client.1.smithi067.stdout:5/598: unlink d0/da/d3f/d82/fac 0 2024-04-12T05:37:22.528 INFO:tasks.workunit.client.1.smithi067.stdout:1/383: dread d4/d9/d12/d59/f64 [0,4194304] 0 2024-04-12T05:37:22.528 INFO:tasks.workunit.client.1.smithi067.stdout:9/370: creat d1/d28/d44/f72 x:0 0 0 2024-04-12T05:37:22.530 INFO:tasks.workunit.client.1.smithi067.stdout:8/356: mknod d6/d8/d18/d3c/d4f/c6b 0 2024-04-12T05:37:22.530 INFO:tasks.workunit.client.1.smithi067.stdout:2/450: mkdir d5/da/d8c 0 2024-04-12T05:37:22.531 INFO:tasks.workunit.client.1.smithi067.stdout:8/357: chown d6/d8/c1a 213673288 1 2024-04-12T05:37:22.532 INFO:tasks.workunit.client.1.smithi067.stdout:6/447: creat dd/d11/d27/f9a x:0 0 0 2024-04-12T05:37:22.534 INFO:tasks.workunit.client.1.smithi067.stdout:3/281: symlink dc/d1d/d28/d3e/l5d 0 2024-04-12T05:37:22.535 INFO:tasks.workunit.client.1.smithi067.stdout:5/599: creat d0/d6/fc8 x:0 0 0 2024-04-12T05:37:22.537 INFO:tasks.workunit.client.1.smithi067.stdout:0/453: creat dd/d12/d14/f8d x:0 0 0 2024-04-12T05:37:22.537 INFO:tasks.workunit.client.1.smithi067.stdout:7/384: dwrite d0/d2/d5/d1d/f3a [0,4194304] 0 2024-04-12T05:37:22.537 INFO:tasks.workunit.client.1.smithi067.stdout:2/451: symlink d5/d6a/d78/l8d 0 2024-04-12T05:37:22.538 INFO:tasks.workunit.client.1.smithi067.stdout:9/371: dread d1/d12/f2e [0,4194304] 0 2024-04-12T05:37:22.539 INFO:tasks.workunit.client.1.smithi067.stdout:8/358: creat d6/d8/d18/d36/f6c x:0 0 0 2024-04-12T05:37:22.539 INFO:tasks.workunit.client.1.smithi067.stdout:3/282: write dc/df/d27/f3f [556255,73432] 0 2024-04-12T05:37:22.541 INFO:tasks.workunit.client.1.smithi067.stdout:7/385: write d0/d2/d5/d1d/d43/f69 [696269,97548] 0 2024-04-12T05:37:22.543 INFO:tasks.workunit.client.1.smithi067.stdout:8/359: chown d6/d8/d18/d36/c61 6608956 1 2024-04-12T05:37:22.543 INFO:tasks.workunit.client.1.smithi067.stdout:4/272: sync 2024-04-12T05:37:22.544 INFO:tasks.workunit.client.1.smithi067.stdout:5/600: stat d0/d6/d40/c5b 0 2024-04-12T05:37:22.545 INFO:tasks.workunit.client.1.smithi067.stdout:4/273: rename d8 to d8/d9/d3d/d63 22 2024-04-12T05:37:22.546 INFO:tasks.workunit.client.1.smithi067.stdout:0/454: dread dd/d12/f3e [0,4194304] 0 2024-04-12T05:37:22.546 INFO:tasks.workunit.client.1.smithi067.stdout:2/452: unlink d5/da/de/d39/l3f 0 2024-04-12T05:37:22.547 INFO:tasks.workunit.client.1.smithi067.stdout:1/384: creat d4/d9/d12/f7d x:0 0 0 2024-04-12T05:37:22.548 INFO:tasks.workunit.client.1.smithi067.stdout:5/601: read d0/d6/d33/f3c [2366636,50497] 0 2024-04-12T05:37:22.549 INFO:tasks.workunit.client.1.smithi067.stdout:3/283: creat dc/d1d/d26/f5e x:0 0 0 2024-04-12T05:37:22.551 INFO:tasks.workunit.client.1.smithi067.stdout:1/385: fdatasync d4/d9/d3a/f3c 0 2024-04-12T05:37:22.551 INFO:tasks.workunit.client.0.smithi032.stdout:1/893: sync 2024-04-12T05:37:22.553 INFO:tasks.workunit.client.0.smithi032.stdout:1/894: symlink d5/de/d5b/l11f 0 2024-04-12T05:37:22.555 INFO:tasks.workunit.client.0.smithi032.stdout:1/895: truncate d5/de/d17/d35/d61/d77/d100/f11e 343382 0 2024-04-12T05:37:22.556 INFO:tasks.workunit.client.0.smithi032.stdout:1/896: stat d5/de/dbe 0 2024-04-12T05:37:22.556 INFO:tasks.workunit.client.1.smithi067.stdout:7/386: unlink d0/d2/d5/d1d/d43/l5a 0 2024-04-12T05:37:22.556 INFO:tasks.workunit.client.1.smithi067.stdout:9/372: link d1/d12/f14 d1/d28/d44/f73 0 2024-04-12T05:37:22.558 INFO:tasks.workunit.client.0.smithi032.stdout:1/897: unlink d5/de/d17/d1b/f21 0 2024-04-12T05:37:22.558 INFO:tasks.workunit.client.0.smithi032.stdout:1/898: readlink d5/de/d17/d35/d3b/d49/l51 0 2024-04-12T05:37:22.560 INFO:tasks.workunit.client.1.smithi067.stdout:6/448: getdents dd/d21/d23 0 2024-04-12T05:37:22.563 INFO:tasks.workunit.client.0.smithi032.stdout:1/899: getdents d5/de/d17/d35/d61/d77 0 2024-04-12T05:37:22.563 INFO:tasks.workunit.client.1.smithi067.stdout:1/386: creat d4/d9/d12/d2c/f7e x:0 0 0 2024-04-12T05:37:22.564 INFO:tasks.workunit.client.0.smithi032.stdout:1/900: chown d5/de/d5b/fe7 87079916 1 2024-04-12T05:37:22.565 INFO:tasks.workunit.client.1.smithi067.stdout:4/274: dwrite d8/d9/f41 [0,4194304] 0 2024-04-12T05:37:22.565 INFO:tasks.workunit.client.0.smithi032.stdout:1/901: chown d5/de/d17/d93/fc8 3 1 2024-04-12T05:37:22.566 INFO:tasks.workunit.client.0.smithi032.stdout:1/902: chown d5/de/fef 99 1 2024-04-12T05:37:22.567 INFO:tasks.workunit.client.1.smithi067.stdout:7/387: mkdir d0/d2/d5/d1d/d7d 0 2024-04-12T05:37:22.568 INFO:tasks.workunit.client.1.smithi067.stdout:9/373: write d1/d28/d44/f73 [411571,100100] 0 2024-04-12T05:37:22.568 INFO:tasks.workunit.client.1.smithi067.stdout:6/449: rmdir dd/d21/d23 39 2024-04-12T05:37:22.572 INFO:tasks.workunit.client.1.smithi067.stdout:2/453: dwrite d5/d6a/f88 [0,4194304] 0 2024-04-12T05:37:22.572 INFO:tasks.workunit.client.1.smithi067.stdout:3/284: truncate dc/d1d/d26/f2f 188615 0 2024-04-12T05:37:22.574 INFO:tasks.workunit.client.1.smithi067.stdout:5/602: dwrite d0/da/d41/f7a [0,4194304] 0 2024-04-12T05:37:22.577 INFO:tasks.workunit.client.1.smithi067.stdout:4/275: mkdir d8/d9/d3d/d64 0 2024-04-12T05:37:22.578 INFO:tasks.workunit.client.0.smithi032.stdout:1/903: truncate d5/de/d17/d35/d61/f102 95138 0 2024-04-12T05:37:22.578 INFO:tasks.workunit.client.1.smithi067.stdout:7/388: write d0/d2/dd/d79/f63 [1090297,50864] 0 2024-04-12T05:37:22.580 INFO:tasks.workunit.client.0.smithi032.stdout:1/904: chown d5/de/d17/d35/d61/d77/d8f/da0/fe6 845772437 1 2024-04-12T05:37:22.581 INFO:tasks.workunit.client.1.smithi067.stdout:2/454: creat d5/da/d3d/f8e x:0 0 0 2024-04-12T05:37:22.581 INFO:tasks.workunit.client.1.smithi067.stdout:3/285: write dc/d1d/d28/f39 [934561,74672] 0 2024-04-12T05:37:22.584 INFO:tasks.workunit.client.0.smithi032.stdout:1/905: creat d5/de/d17/d35/d61/d77/f120 x:0 0 0 2024-04-12T05:37:22.585 INFO:tasks.workunit.client.0.smithi032.stdout:9/858: sync 2024-04-12T05:37:22.586 INFO:tasks.workunit.client.0.smithi032.stdout:2/988: sync 2024-04-12T05:37:22.586 INFO:tasks.workunit.client.0.smithi032.stdout:1/906: stat d5/de/d17/d35/d3b/dda/de0 0 2024-04-12T05:37:22.589 INFO:tasks.workunit.client.1.smithi067.stdout:0/455: dwrite dd/d12/f59 [0,4194304] 0 2024-04-12T05:37:22.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:22 smithi032 ceph-mon[39731]: pgmap v272: 65 pgs: 65 active+clean; 2.4 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 83 MiB/s rd, 207 MiB/s wr, 433 op/s 2024-04-12T05:37:22.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:22 smithi032 ceph-mon[39731]: mgrmap e21: smithi032.hrwxrh(active, since 7m), standbys: smithi067.dwichg 2024-04-12T05:37:22.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:22 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr metadata", "who": "smithi067.dwichg", "id": "smithi067.dwichg"}]: dispatch 2024-04-12T05:37:22.591 INFO:tasks.workunit.client.1.smithi067.stdout:2/455: chown d5/da/c30 1787304 1 2024-04-12T05:37:22.591 INFO:tasks.workunit.client.1.smithi067.stdout:1/387: dread d4/d9/d12/d28/f2b [0,4194304] 0 2024-04-12T05:37:22.592 INFO:tasks.workunit.client.1.smithi067.stdout:7/389: unlink d0/f55 0 2024-04-12T05:37:22.595 INFO:tasks.workunit.client.1.smithi067.stdout:5/603: dread d0/da/d41/f9b [0,4194304] 0 2024-04-12T05:37:22.596 INFO:tasks.workunit.client.1.smithi067.stdout:3/286: mkdir dc/d5f 0 2024-04-12T05:37:22.597 INFO:tasks.workunit.client.0.smithi032.stdout:2/989: dwrite dd/d19/d9d/d1c/f87 [0,4194304] 0 2024-04-12T05:37:22.598 INFO:tasks.workunit.client.1.smithi067.stdout:9/374: dwrite d1/d28/d21/d3f/d4f/f60 [0,4194304] 0 2024-04-12T05:37:22.601 INFO:tasks.workunit.client.1.smithi067.stdout:0/456: creat dd/d5b/f8e x:0 0 0 2024-04-12T05:37:22.602 INFO:tasks.workunit.client.1.smithi067.stdout:6/450: link dd/d21/d66/c74 dd/d21/d64/c9b 0 2024-04-12T05:37:22.611 INFO:tasks.workunit.client.0.smithi032.stdout:9/859: write d9/fa [1145030,17034] 0 2024-04-12T05:37:22.612 INFO:tasks.workunit.client.0.smithi032.stdout:2/990: rename dd/f16 to dd/d19/d9d/d42/d4a/f13c 0 2024-04-12T05:37:22.612 INFO:tasks.workunit.client.1.smithi067.stdout:3/287: readlink dc/l33 0 2024-04-12T05:37:22.612 INFO:tasks.workunit.client.1.smithi067.stdout:9/375: fsync d1/fd 0 2024-04-12T05:37:22.613 INFO:tasks.workunit.client.1.smithi067.stdout:6/451: write dd/d11/d27/f54 [976945,15536] 0 2024-04-12T05:37:22.616 INFO:tasks.workunit.client.0.smithi032.stdout:2/991: creat dd/d8c/f13d x:0 0 0 2024-04-12T05:37:22.618 INFO:tasks.workunit.client.1.smithi067.stdout:1/388: mknod d4/d9/d12/d6f/c7f 0 2024-04-12T05:37:22.618 INFO:tasks.workunit.client.1.smithi067.stdout:5/604: dwrite d0/da/d41/f60 [0,4194304] 0 2024-04-12T05:37:22.619 INFO:tasks.workunit.client.0.smithi032.stdout:9/860: link d9/df7/f11c d9/d58/da9/f129 0 2024-04-12T05:37:22.620 INFO:tasks.workunit.client.0.smithi032.stdout:1/907: dwrite d5/de/d17/d35/d78/f7a [0,4194304] 0 2024-04-12T05:37:22.621 INFO:tasks.workunit.client.1.smithi067.stdout:3/288: creat dc/d50/f60 x:0 0 0 2024-04-12T05:37:22.622 INFO:tasks.workunit.client.0.smithi032.stdout:9/861: truncate d9/d32/d6e/f10f 137742 0 2024-04-12T05:37:22.624 INFO:tasks.workunit.client.1.smithi067.stdout:4/276: getdents d8/d9/d3d 0 2024-04-12T05:37:22.625 INFO:tasks.workunit.client.1.smithi067.stdout:7/390: dwrite d0/d2/d5/db/d34/f3f [0,4194304] 0 2024-04-12T05:37:22.630 INFO:tasks.workunit.client.1.smithi067.stdout:4/277: chown d8/d9/d3d/f3f 1420705 1 2024-04-12T05:37:22.631 INFO:tasks.workunit.client.0.smithi032.stdout:9/862: truncate d9/d15/d1a/d24/f107 1212086 0 2024-04-12T05:37:22.631 INFO:tasks.workunit.client.1.smithi067.stdout:2/456: getdents d5/d43 0 2024-04-12T05:37:22.631 INFO:tasks.workunit.client.1.smithi067.stdout:7/391: readlink d0/d2/d5/db/d34/l66 0 2024-04-12T05:37:22.632 INFO:tasks.workunit.client.0.smithi032.stdout:2/992: fsync dd/d19/d9d/d42/d4a/f13c 0 2024-04-12T05:37:22.632 INFO:tasks.workunit.client.0.smithi032.stdout:9/863: chown d9/d1f/fe0 4859049 1 2024-04-12T05:37:22.636 INFO:tasks.workunit.client.1.smithi067.stdout:2/457: fsync d5/da/de/d14/d34/f61 0 2024-04-12T05:37:22.637 INFO:tasks.workunit.client.1.smithi067.stdout:9/376: dwrite d1/d12/d15/f5c [0,4194304] 0 2024-04-12T05:37:22.637 INFO:tasks.workunit.client.0.smithi032.stdout:2/993: mkdir dd/d19/d9d/d42/dc7/d13e 0 2024-04-12T05:37:22.639 INFO:tasks.workunit.client.1.smithi067.stdout:1/389: rename d4/d9/d14/d61 to d4/d52/d80 0 2024-04-12T05:37:22.639 INFO:tasks.workunit.client.1.smithi067.stdout:4/278: read d8/f33 [323965,85057] 0 2024-04-12T05:37:22.639 INFO:tasks.workunit.client.0.smithi032.stdout:9/864: fsync d9/d15/d2c/dab/fc5 0 2024-04-12T05:37:22.643 INFO:tasks.workunit.client.0.smithi032.stdout:2/994: mknod dd/d19/d9d/d1c/c13f 0 2024-04-12T05:37:22.645 INFO:tasks.workunit.client.1.smithi067.stdout:2/458: mkdir d5/d6a/d8f 0 2024-04-12T05:37:22.646 INFO:tasks.workunit.client.1.smithi067.stdout:0/457: getdents dd/d5b 0 2024-04-12T05:37:22.647 INFO:tasks.workunit.client.1.smithi067.stdout:9/377: creat d1/d12/f74 x:0 0 0 2024-04-12T05:37:22.647 INFO:tasks.workunit.client.0.smithi032.stdout:9/865: getdents d9/d15/d1a/d24/d27/d88 0 2024-04-12T05:37:22.648 INFO:tasks.workunit.client.1.smithi067.stdout:1/390: truncate d4/d9/f68 983959 0 2024-04-12T05:37:22.651 INFO:tasks.workunit.client.1.smithi067.stdout:5/605: write d0/d6/f6a [299639,67884] 0 2024-04-12T05:37:22.651 INFO:tasks.workunit.client.0.smithi032.stdout:2/995: dwrite dd/d19/f11e [0,4194304] 0 2024-04-12T05:37:22.652 INFO:tasks.workunit.client.1.smithi067.stdout:2/459: fdatasync d5/d43/d6e/f72 0 2024-04-12T05:37:22.653 INFO:tasks.workunit.client.0.smithi032.stdout:9/866: fsync d9/d15/d1a/d24/d27/fd0 0 2024-04-12T05:37:22.654 INFO:tasks.workunit.client.1.smithi067.stdout:7/392: dwrite d0/d23/f41 [4194304,4194304] 0 2024-04-12T05:37:22.655 INFO:tasks.workunit.client.1.smithi067.stdout:3/289: dwrite dc/df/f1c [0,4194304] 0 2024-04-12T05:37:22.655 INFO:tasks.workunit.client.0.smithi032.stdout:2/996: mknod dd/d19/d82/c140 0 2024-04-12T05:37:22.656 INFO:tasks.workunit.client.1.smithi067.stdout:0/458: mknod dd/d6f/d72/c8f 0 2024-04-12T05:37:22.657 INFO:tasks.workunit.client.0.smithi032.stdout:9/867: chown d9/d15/d2c/f77 702 1 2024-04-12T05:37:22.659 INFO:tasks.workunit.client.1.smithi067.stdout:4/279: getdents d8 0 2024-04-12T05:37:22.661 INFO:tasks.workunit.client.1.smithi067.stdout:5/606: write d0/d6/d20/d3e/f95 [896810,1687] 0 2024-04-12T05:37:22.661 INFO:tasks.workunit.client.0.smithi032.stdout:9/868: fsync d9/d58/fb6 0 2024-04-12T05:37:22.667 INFO:tasks.workunit.client.0.smithi032.stdout:2/997: truncate dd/d19/d9d/d42/fff 97573 0 2024-04-12T05:37:22.667 INFO:tasks.workunit.client.1.smithi067.stdout:3/290: fsync dc/d1d/d4d/f55 0 2024-04-12T05:37:22.671 INFO:tasks.workunit.client.1.smithi067.stdout:6/452: dwrite dd/d11/f19 [0,4194304] 0 2024-04-12T05:37:22.672 INFO:tasks.workunit.client.0.smithi032.stdout:9/869: creat d9/d15/d1a/f12a x:0 0 0 2024-04-12T05:37:22.675 INFO:tasks.workunit.client.1.smithi067.stdout:2/460: mkdir d5/da/de/d14/d46/d90 0 2024-04-12T05:37:22.676 INFO:tasks.workunit.client.1.smithi067.stdout:9/378: dread d1/d28/d21/d3f/d4f/f60 [0,4194304] 0 2024-04-12T05:37:22.677 INFO:tasks.workunit.client.1.smithi067.stdout:4/280: creat d8/da/d1d/d15/d1c/f65 x:0 0 0 2024-04-12T05:37:22.678 INFO:tasks.workunit.client.0.smithi032.stdout:2/998: dwrite dd/d19/d9d/d1c/d64/d36/f85 [4194304,4194304] 0 2024-04-12T05:37:22.679 INFO:tasks.workunit.client.1.smithi067.stdout:0/459: unlink dd/d12/d14/d1a/d3d/f2f 0 2024-04-12T05:37:22.679 INFO:tasks.workunit.client.0.smithi032.stdout:9/870: mkdir d9/d15/d1a/d1d/d5a/d12b 0 2024-04-12T05:37:22.683 INFO:tasks.workunit.client.1.smithi067.stdout:5/607: dwrite d0/f2 [0,4194304] 0 2024-04-12T05:37:22.684 INFO:tasks.workunit.client.1.smithi067.stdout:1/391: dwrite d4/d46/f58 [0,4194304] 0 2024-04-12T05:37:22.684 INFO:tasks.workunit.client.0.smithi032.stdout:2/999: chown dd/d19/d82/lb2 5 1 2024-04-12T05:37:22.685 INFO:tasks.workunit.client.1.smithi067.stdout:2/461: write d5/da/de/d14/f7b [712523,70184] 0 2024-04-12T05:37:22.686 INFO:tasks.workunit.client.1.smithi067.stdout:9/379: mkdir d1/d28/d21/d45/d75 0 2024-04-12T05:37:22.693 INFO:tasks.workunit.client.1.smithi067.stdout:3/291: dread dc/d1d/d28/f4c [0,4194304] 0 2024-04-12T05:37:22.694 INFO:tasks.workunit.client.1.smithi067.stdout:5/608: readlink d0/d3a/d2c/d91/l7b 0 2024-04-12T05:37:22.695 INFO:tasks.workunit.client.1.smithi067.stdout:4/281: mkdir d8/da/d1d/d15/d36/d5d/d66 0 2024-04-12T05:37:22.698 INFO:tasks.workunit.client.1.smithi067.stdout:3/292: mknod dc/d50/c61 0 2024-04-12T05:37:22.702 INFO:tasks.workunit.client.1.smithi067.stdout:6/453: write dd/d21/d23/f3c [1599099,12356] 0 2024-04-12T05:37:22.703 INFO:tasks.workunit.client.0.smithi032.stdout:9/871: dread d9/d15/f28 [0,4194304] 0 2024-04-12T05:37:22.704 INFO:tasks.workunit.client.1.smithi067.stdout:7/393: dread d0/f12 [0,4194304] 0 2024-04-12T05:37:22.705 INFO:tasks.workunit.client.1.smithi067.stdout:5/609: rename d0/da/d3f/daa/fb6 to d0/d3a/da8/fc9 0 2024-04-12T05:37:22.708 INFO:tasks.workunit.client.0.smithi032.stdout:9/872: read d9/d15/d1a/d24/d27/d88/dc1/fdb [7543171,66160] 0 2024-04-12T05:37:22.709 INFO:tasks.workunit.client.1.smithi067.stdout:4/282: creat d8/d9/f67 x:0 0 0 2024-04-12T05:37:22.710 INFO:tasks.workunit.client.1.smithi067.stdout:3/293: unlink dc/c13 0 2024-04-12T05:37:22.713 INFO:tasks.workunit.client.1.smithi067.stdout:6/454: truncate dd/d11/d27/f8f 4084046 0 2024-04-12T05:37:22.714 INFO:tasks.workunit.client.0.smithi032.stdout:9/873: unlink d9/c26 0 2024-04-12T05:37:22.714 INFO:tasks.workunit.client.1.smithi067.stdout:5/610: truncate d0/d6/d33/f9f 858384 0 2024-04-12T05:37:22.714 INFO:tasks.workunit.client.1.smithi067.stdout:7/394: rename d0/d2/d5/db to d0/d2/dd/d42/d7e 0 2024-04-12T05:37:22.717 INFO:tasks.workunit.client.1.smithi067.stdout:1/392: dwrite d4/d52/d80/f79 [0,4194304] 0 2024-04-12T05:37:22.717 INFO:tasks.workunit.client.1.smithi067.stdout:3/294: chown dc/d1d/d28/d3e/d53/f58 4220089 1 2024-04-12T05:37:22.719 INFO:tasks.workunit.client.0.smithi032.stdout:9/874: rename d9/d15/d1a/d1d/d5a/f75 to d9/d32/d86/f12c 0 2024-04-12T05:37:22.720 INFO:tasks.workunit.client.1.smithi067.stdout:5/611: dread - d0/d6/d20/d3e/fbc zero size 2024-04-12T05:37:22.722 INFO:tasks.workunit.client.1.smithi067.stdout:0/460: dread f1 [0,4194304] 0 2024-04-12T05:37:22.723 INFO:tasks.workunit.client.0.smithi032.stdout:9/875: symlink d9/d15/d1a/d24/d27/d88/dc1/l12d 0 2024-04-12T05:37:22.724 INFO:tasks.workunit.client.0.smithi032.stdout:9/876: chown d9/d15/d1a/d24/d27/d88/dc1/lc7 1664645 1 2024-04-12T05:37:22.725 INFO:tasks.workunit.client.1.smithi067.stdout:7/395: mknod d0/d2/d5/d1d/c7f 0 2024-04-12T05:37:22.725 INFO:tasks.workunit.client.1.smithi067.stdout:2/462: dread d5/d43/d6e/f72 [0,4194304] 0 2024-04-12T05:37:22.726 INFO:tasks.workunit.client.1.smithi067.stdout:6/455: creat dd/d21/f9c x:0 0 0 2024-04-12T05:37:22.729 INFO:tasks.workunit.client.1.smithi067.stdout:1/393: fdatasync d4/f7 0 2024-04-12T05:37:22.730 INFO:tasks.workunit.client.1.smithi067.stdout:7/396: write d0/d23/f6e [3232795,47858] 0 2024-04-12T05:37:22.730 INFO:tasks.workunit.client.1.smithi067.stdout:5/612: symlink d0/d6/d20/lca 0 2024-04-12T05:37:22.733 INFO:tasks.workunit.client.1.smithi067.stdout:4/283: dwrite d8/d9/d3d/d48/f57 [0,4194304] 0 2024-04-12T05:37:22.733 INFO:tasks.workunit.client.1.smithi067.stdout:3/295: rename dc/f48 to dc/d1d/d28/f62 0 2024-04-12T05:37:22.734 INFO:tasks.workunit.client.0.smithi032.stdout:9/877: link d9/d15/d1a/d24/d27/dce/dd3/de9/cf8 d9/c12e 0 2024-04-12T05:37:22.735 INFO:tasks.workunit.client.0.smithi032.stdout:9/878: write d9/d32/d86/ff3 [788068,42527] 0 2024-04-12T05:37:22.735 INFO:tasks.workunit.client.1.smithi067.stdout:6/456: creat dd/d21/d64/d6f/d90/f9d x:0 0 0 2024-04-12T05:37:22.737 INFO:tasks.workunit.client.1.smithi067.stdout:2/463: mknod d5/d6a/d78/c91 0 2024-04-12T05:37:22.743 INFO:tasks.workunit.client.1.smithi067.stdout:5/613: creat d0/d3a/dab/fcb x:0 0 0 2024-04-12T05:37:22.743 INFO:tasks.workunit.client.1.smithi067.stdout:7/397: mkdir d0/d23/d80 0 2024-04-12T05:37:22.743 INFO:tasks.workunit.client.1.smithi067.stdout:0/461: dwrite dd/d5b/f8e [0,4194304] 0 2024-04-12T05:37:22.743 INFO:tasks.workunit.client.1.smithi067.stdout:1/394: dread d4/d9/d12/f2e [0,4194304] 0 2024-04-12T05:37:22.744 INFO:tasks.workunit.client.0.smithi032.stdout:9/879: creat d9/d15/d1a/d1d/d5a/d6b/f12f x:0 0 0 2024-04-12T05:37:22.744 INFO:tasks.workunit.client.1.smithi067.stdout:7/398: write d0/f8 [1945754,116754] 0 2024-04-12T05:37:22.748 INFO:tasks.workunit.client.1.smithi067.stdout:1/395: read d4/d9/d12/f1b [3946853,39170] 0 2024-04-12T05:37:22.749 INFO:tasks.workunit.client.1.smithi067.stdout:6/457: mkdir dd/d21/d23/d9e 0 2024-04-12T05:37:22.750 INFO:tasks.workunit.client.1.smithi067.stdout:4/284: mkdir d8/da/d1d/d15/d1c/d68 0 2024-04-12T05:37:22.752 INFO:tasks.workunit.client.0.smithi032.stdout:9/880: dwrite d9/d15/d1a/d24/fed [0,4194304] 0 2024-04-12T05:37:22.753 INFO:tasks.workunit.client.1.smithi067.stdout:6/458: dread dd/d21/d23/f65 [0,4194304] 0 2024-04-12T05:37:22.758 INFO:tasks.workunit.client.1.smithi067.stdout:0/462: creat dd/d12/d14/d1a/d3d/d41/d5f/f90 x:0 0 0 2024-04-12T05:37:22.759 INFO:tasks.workunit.client.1.smithi067.stdout:3/296: write f1 [1884315,933] 0 2024-04-12T05:37:22.760 INFO:tasks.workunit.client.1.smithi067.stdout:6/459: dwrite dd/d11/d27/d4a/f73 [0,4194304] 0 2024-04-12T05:37:22.761 INFO:tasks.workunit.client.0.smithi032.stdout:1/908: sync 2024-04-12T05:37:22.766 INFO:tasks.workunit.client.1.smithi067.stdout:4/285: mkdir d8/da/d1d/d15/d58/d69 0 2024-04-12T05:37:22.768 INFO:tasks.workunit.client.0.smithi032.stdout:9/881: creat d9/d15/d1a/dd2/d65/d80/f130 x:0 0 0 2024-04-12T05:37:22.769 INFO:tasks.workunit.client.1.smithi067.stdout:3/297: dread - dc/df/d27/f2d zero size 2024-04-12T05:37:22.769 INFO:tasks.workunit.client.0.smithi032.stdout:1/909: creat d5/de/d17/dd6/f121 x:0 0 0 2024-04-12T05:37:22.769 INFO:tasks.workunit.client.1.smithi067.stdout:3/298: chown dc/d1d/d26/d52 85 1 2024-04-12T05:37:22.771 INFO:tasks.workunit.client.1.smithi067.stdout:2/464: link d5/da/c30 d5/d6a/d78/c92 0 2024-04-12T05:37:22.771 INFO:tasks.workunit.client.1.smithi067.stdout:1/396: creat d4/d9/d12/f81 x:0 0 0 2024-04-12T05:37:22.771 INFO:tasks.workunit.client.1.smithi067.stdout:6/460: mkdir dd/d9f 0 2024-04-12T05:37:22.773 INFO:tasks.workunit.client.1.smithi067.stdout:4/286: unlink f6 0 2024-04-12T05:37:22.774 INFO:tasks.workunit.client.1.smithi067.stdout:1/397: write d4/d9/d12/f7d [861823,107525] 0 2024-04-12T05:37:22.774 INFO:tasks.workunit.client.0.smithi032.stdout:9/882: rmdir d9/d58/d11f 0 2024-04-12T05:37:22.775 INFO:tasks.workunit.client.1.smithi067.stdout:3/299: creat dc/d1d/d28/d3e/f63 x:0 0 0 2024-04-12T05:37:22.775 INFO:tasks.workunit.client.0.smithi032.stdout:9/883: chown d9/d15/d1a/dd2/d65 677386 1 2024-04-12T05:37:22.778 INFO:tasks.workunit.client.1.smithi067.stdout:2/465: readlink d5/l6 0 2024-04-12T05:37:22.778 INFO:tasks.workunit.client.0.smithi032.stdout:9/884: creat d9/d32/d86/f131 x:0 0 0 2024-04-12T05:37:22.779 INFO:tasks.workunit.client.1.smithi067.stdout:4/287: write d8/da/d1d/f44 [455348,22169] 0 2024-04-12T05:37:22.779 INFO:tasks.workunit.client.1.smithi067.stdout:0/463: link dd/d12/d14/f39 dd/d12/d14/d51/f91 0 2024-04-12T05:37:22.780 INFO:tasks.workunit.client.1.smithi067.stdout:5/614: link d0/d6/d20/d62/c13 d0/d6/d20/d3e/ccc 0 2024-04-12T05:37:22.780 INFO:tasks.workunit.client.1.smithi067.stdout:1/398: write d4/d9/d4c/d5d/f71 [446849,23683] 0 2024-04-12T05:37:22.782 INFO:tasks.workunit.client.1.smithi067.stdout:2/466: chown d5/f12 254710 1 2024-04-12T05:37:22.785 INFO:tasks.workunit.client.1.smithi067.stdout:7/399: truncate d0/d2/d5/d1e/d51/f58 2470885 0 2024-04-12T05:37:22.788 INFO:tasks.workunit.client.1.smithi067.stdout:1/399: creat d4/d9/d3a/d51/f82 x:0 0 0 2024-04-12T05:37:22.788 INFO:tasks.workunit.client.1.smithi067.stdout:2/467: mkdir d5/d23/d93 0 2024-04-12T05:37:22.790 INFO:tasks.workunit.client.1.smithi067.stdout:5/615: write d0/da/d41/f59 [1100206,47575] 0 2024-04-12T05:37:22.791 INFO:tasks.workunit.client.0.smithi032.stdout:1/910: dread d5/de/d17/d35/d78/f83 [0,4194304] 0 2024-04-12T05:37:22.792 INFO:tasks.workunit.client.0.smithi032.stdout:1/911: chown d5/de/d17/d1b/lb0 264334 1 2024-04-12T05:37:22.793 INFO:tasks.workunit.client.1.smithi067.stdout:6/461: dwrite dd/d21/d23/d56/f69 [0,4194304] 0 2024-04-12T05:37:22.794 INFO:tasks.workunit.client.1.smithi067.stdout:3/300: dread dc/d1d/d4d/f55 [0,4194304] 0 2024-04-12T05:37:22.796 INFO:tasks.workunit.client.1.smithi067.stdout:2/468: stat d5/da/de/f13 0 2024-04-12T05:37:22.798 INFO:tasks.workunit.client.1.smithi067.stdout:1/400: readlink d4/d9/d12/d2c/l7c 0 2024-04-12T05:37:22.801 INFO:tasks.workunit.client.1.smithi067.stdout:0/464: rename dd/d12/d14/d1a/d75 to dd/d92 0 2024-04-12T05:37:22.804 INFO:tasks.workunit.client.1.smithi067.stdout:2/469: write d5/d43/d6e/f76 [222288,34686] 0 2024-04-12T05:37:22.805 INFO:tasks.workunit.client.1.smithi067.stdout:4/288: dwrite d8/d39/f51 [0,4194304] 0 2024-04-12T05:37:22.806 INFO:tasks.workunit.client.0.smithi032.stdout:9/885: dwrite d9/d15/d1a/dd2/f98 [0,4194304] 0 2024-04-12T05:37:22.808 INFO:tasks.workunit.client.1.smithi067.stdout:3/301: read dc/d1d/f24 [362524,65787] 0 2024-04-12T05:37:22.809 INFO:tasks.workunit.client.0.smithi032.stdout:9/886: write d9/d58/da9/fdc [146398,29965] 0 2024-04-12T05:37:22.811 INFO:tasks.workunit.client.1.smithi067.stdout:0/465: write dd/d6f/d72/f8b [564518,128897] 0 2024-04-12T05:37:22.814 INFO:tasks.workunit.client.1.smithi067.stdout:5/616: dread d0/da/d41/f60 [0,4194304] 0 2024-04-12T05:37:22.815 INFO:tasks.workunit.client.1.smithi067.stdout:7/400: dwrite d0/d23/f41 [0,4194304] 0 2024-04-12T05:37:22.815 INFO:tasks.workunit.client.1.smithi067.stdout:3/302: fsync dc/df/d27/f2d 0 2024-04-12T05:37:22.817 INFO:tasks.workunit.client.1.smithi067.stdout:5/617: stat d0/d6/d33/d9e 0 2024-04-12T05:37:22.817 INFO:tasks.workunit.client.1.smithi067.stdout:7/401: chown d0/d23/d45/l70 2450 1 2024-04-12T05:37:22.818 INFO:tasks.workunit.client.1.smithi067.stdout:7/402: readlink d0/d23/d45/l32 0 2024-04-12T05:37:22.819 INFO:tasks.workunit.client.0.smithi032.stdout:9/887: creat d9/d15/d1a/d24/df6/f132 x:0 0 0 2024-04-12T05:37:22.820 INFO:tasks.workunit.client.0.smithi032.stdout:9/888: stat d9/d15/d1a/d24/d27/fd0 0 2024-04-12T05:37:22.822 INFO:tasks.workunit.client.1.smithi067.stdout:2/470: fdatasync d5/da/de/d14/d34/d4c/f6b 0 2024-04-12T05:37:22.824 INFO:tasks.workunit.client.0.smithi032.stdout:9/889: creat d9/d32/d6e/d114/f133 x:0 0 0 2024-04-12T05:37:22.824 INFO:tasks.workunit.client.1.smithi067.stdout:6/462: getdents dd/d21/d23/d56 0 2024-04-12T05:37:22.826 INFO:tasks.workunit.client.1.smithi067.stdout:1/401: dread d4/d9/d12/d2c/f37 [0,4194304] 0 2024-04-12T05:37:22.827 INFO:tasks.workunit.client.1.smithi067.stdout:3/303: dwrite dc/d1d/d26/f46 [0,4194304] 0 2024-04-12T05:37:22.830 INFO:tasks.workunit.client.0.smithi032.stdout:9/890: link d9/d15/lb0 d9/d58/l134 0 2024-04-12T05:37:22.832 INFO:tasks.workunit.client.1.smithi067.stdout:6/463: mknod dd/d21/d64/d88/d7b/ca0 0 2024-04-12T05:37:22.834 INFO:tasks.workunit.client.1.smithi067.stdout:5/618: creat d0/d3a/d2c/fcd x:0 0 0 2024-04-12T05:37:22.837 INFO:tasks.workunit.client.1.smithi067.stdout:1/402: rename d4/d9/d12/d28/f2b to d4/d9/d12/d28/f83 0 2024-04-12T05:37:22.837 INFO:tasks.workunit.client.1.smithi067.stdout:6/464: symlink dd/d11/d27/d4a/d51/la1 0 2024-04-12T05:37:22.837 INFO:tasks.workunit.client.1.smithi067.stdout:0/466: truncate dd/d12/d14/d1a/d3d/d41/f50 811921 0 2024-04-12T05:37:22.838 INFO:tasks.workunit.client.1.smithi067.stdout:2/471: creat d5/da/de/f94 x:0 0 0 2024-04-12T05:37:22.839 INFO:tasks.workunit.client.1.smithi067.stdout:7/403: getdents d0/d2/dd/d42 0 2024-04-12T05:37:22.839 INFO:tasks.workunit.client.1.smithi067.stdout:1/403: chown d4/d9/d12/d28 2738814 1 2024-04-12T05:37:22.840 INFO:tasks.workunit.client.1.smithi067.stdout:3/304: rmdir dc/d5f 0 2024-04-12T05:37:22.840 INFO:tasks.workunit.client.1.smithi067.stdout:0/467: chown dd/d5b/f84 11 1 2024-04-12T05:37:22.841 INFO:tasks.workunit.client.1.smithi067.stdout:7/404: dread - d0/d2/f52 zero size 2024-04-12T05:37:22.843 INFO:tasks.workunit.client.1.smithi067.stdout:2/472: creat d5/da/de/d39/f95 x:0 0 0 2024-04-12T05:37:22.843 INFO:tasks.workunit.client.1.smithi067.stdout:6/465: symlink dd/d21/la2 0 2024-04-12T05:37:22.844 INFO:tasks.workunit.client.1.smithi067.stdout:1/404: mknod d4/d9/d3a/d51/c84 0 2024-04-12T05:37:22.846 INFO:tasks.workunit.client.1.smithi067.stdout:3/305: creat dc/d1d/d28/f64 x:0 0 0 2024-04-12T05:37:22.846 INFO:tasks.workunit.client.1.smithi067.stdout:5/619: dwrite d0/d3a/dab/fcb [0,4194304] 0 2024-04-12T05:37:22.847 INFO:tasks.workunit.client.1.smithi067.stdout:7/405: symlink d0/d2/d5/d1e/l81 0 2024-04-12T05:37:22.848 INFO:tasks.workunit.client.1.smithi067.stdout:1/405: stat d4/d9/d3a/d51/c84 0 2024-04-12T05:37:22.848 INFO:tasks.workunit.client.1.smithi067.stdout:0/468: chown dd/d12/d14/d1a/d3d/d41/d5f/d78/d83/c85 243865442 1 2024-04-12T05:37:22.850 INFO:tasks.workunit.client.1.smithi067.stdout:3/306: dread - dc/d1d/d28/d3e/f4a zero size 2024-04-12T05:37:22.850 INFO:tasks.workunit.client.1.smithi067.stdout:2/473: rmdir d5/d11/d7a 0 2024-04-12T05:37:22.852 INFO:tasks.workunit.client.1.smithi067.stdout:4/289: write d8/d9/d3d/f3f [4670743,120310] 0 2024-04-12T05:37:22.852 INFO:tasks.workunit.client.1.smithi067.stdout:7/406: mkdir d0/d2/dd/d42/d82 0 2024-04-12T05:37:22.857 INFO:tasks.workunit.client.1.smithi067.stdout:5/620: rename d0/d6/d33/f72 to d0/da/fce 0 2024-04-12T05:37:22.859 INFO:tasks.workunit.client.1.smithi067.stdout:3/307: dwrite dc/df/f1c [0,4194304] 0 2024-04-12T05:37:22.860 INFO:tasks.workunit.client.1.smithi067.stdout:7/407: symlink d0/d2/dd/d42/l83 0 2024-04-12T05:37:22.861 INFO:tasks.workunit.client.1.smithi067.stdout:2/474: mkdir d5/da/de/d14/d34/d40/d57/d96 0 2024-04-12T05:37:22.866 INFO:tasks.workunit.client.1.smithi067.stdout:2/475: dread d5/da/de/f13 [0,4194304] 0 2024-04-12T05:37:22.867 INFO:tasks.workunit.client.1.smithi067.stdout:6/466: rename dd/d21/d64/d88/d7b/c7a to dd/d21/d64/d6f/d90/d98/ca3 0 2024-04-12T05:37:22.870 INFO:tasks.workunit.client.1.smithi067.stdout:0/469: creat dd/d12/d14/d1a/f93 x:0 0 0 2024-04-12T05:37:22.872 INFO:tasks.workunit.client.1.smithi067.stdout:5/621: symlink d0/d6/d20/lcf 0 2024-04-12T05:37:22.874 INFO:tasks.workunit.client.1.smithi067.stdout:2/476: fdatasync d5/d43/f44 0 2024-04-12T05:37:22.875 INFO:tasks.workunit.client.1.smithi067.stdout:6/467: rmdir dd/d21/d66 39 2024-04-12T05:37:22.878 INFO:tasks.workunit.client.1.smithi067.stdout:7/408: creat d0/d2/d5/d1d/d7d/f84 x:0 0 0 2024-04-12T05:37:22.879 INFO:tasks.workunit.client.1.smithi067.stdout:0/470: rmdir dd/d12/d14/d1a/d3d 39 2024-04-12T05:37:22.880 INFO:tasks.workunit.client.1.smithi067.stdout:1/406: write d4/d9/d12/d28/f83 [1357923,85561] 0 2024-04-12T05:37:22.882 INFO:tasks.workunit.client.1.smithi067.stdout:5/622: chown d0/d3a/d2c/l73 388300320 1 2024-04-12T05:37:22.882 INFO:tasks.workunit.client.1.smithi067.stdout:0/471: write dd/d12/d14/d49/f6d [867918,24813] 0 2024-04-12T05:37:22.884 INFO:tasks.workunit.client.1.smithi067.stdout:6/468: unlink dd/d11/d27/d4a/f73 0 2024-04-12T05:37:22.885 INFO:tasks.workunit.client.1.smithi067.stdout:1/407: write d4/d3d/f77 [328508,106927] 0 2024-04-12T05:37:22.885 INFO:tasks.workunit.client.1.smithi067.stdout:3/308: truncate f1 442753 0 2024-04-12T05:37:22.886 INFO:tasks.workunit.client.1.smithi067.stdout:2/477: unlink d5/c48 0 2024-04-12T05:37:22.890 INFO:tasks.workunit.client.1.smithi067.stdout:6/469: unlink dd/d21/d23/f42 0 2024-04-12T05:37:22.890 INFO:tasks.workunit.client.1.smithi067.stdout:0/472: fdatasync dd/d12/d14/d1a/d3d/f81 0 2024-04-12T05:37:22.891 INFO:tasks.workunit.client.1.smithi067.stdout:7/409: dwrite d0/d23/d45/f73 [0,4194304] 0 2024-04-12T05:37:22.892 INFO:tasks.workunit.client.1.smithi067.stdout:1/408: rmdir d4/d9/d12 39 2024-04-12T05:37:22.898 INFO:tasks.workunit.client.1.smithi067.stdout:5/623: dread d0/d6/d33/f9f [0,4194304] 0 2024-04-12T05:37:22.898 INFO:tasks.workunit.client.1.smithi067.stdout:6/470: readlink dd/l6c 0 2024-04-12T05:37:22.899 INFO:tasks.workunit.client.1.smithi067.stdout:3/309: rename dc/df/l4b to dc/d50/l65 0 2024-04-12T05:37:22.900 INFO:tasks.workunit.client.1.smithi067.stdout:7/410: truncate d0/d2/d5/d1d/d7d/f84 649173 0 2024-04-12T05:37:22.901 INFO:tasks.workunit.client.1.smithi067.stdout:1/409: unlink d4/d9/d14/l30 0 2024-04-12T05:37:22.902 INFO:tasks.workunit.client.1.smithi067.stdout:3/310: write f9 [5536330,103186] 0 2024-04-12T05:37:22.903 INFO:tasks.workunit.client.1.smithi067.stdout:2/478: dwrite d5/d6a/d78/f89 [0,4194304] 0 2024-04-12T05:37:22.905 INFO:tasks.workunit.client.1.smithi067.stdout:0/473: chown dd/d12/d14/d51/l87 857222 1 2024-04-12T05:37:22.905 INFO:tasks.workunit.client.1.smithi067.stdout:7/411: write d0/d23/d45/f59 [317144,82360] 0 2024-04-12T05:37:22.907 INFO:tasks.workunit.client.1.smithi067.stdout:6/471: fdatasync dd/d11/d27/d4a/f82 0 2024-04-12T05:37:22.909 INFO:tasks.workunit.client.1.smithi067.stdout:3/311: write dc/d1d/d28/f64 [80974,44606] 0 2024-04-12T05:37:22.915 INFO:tasks.workunit.client.1.smithi067.stdout:3/312: read dc/d1d/d28/f38 [1048926,33176] 0 2024-04-12T05:37:22.916 INFO:tasks.workunit.client.1.smithi067.stdout:7/412: rename d0/d2/dd/d42/d71/d77 to d0/d67/d57/d85 0 2024-04-12T05:37:22.918 INFO:tasks.workunit.client.1.smithi067.stdout:1/410: dwrite d4/f5 [0,4194304] 0 2024-04-12T05:37:22.920 INFO:tasks.workunit.client.1.smithi067.stdout:6/472: link dd/d21/d23/l28 dd/la4 0 2024-04-12T05:37:22.921 INFO:tasks.workunit.client.1.smithi067.stdout:2/479: link d5/da/de/c2d d5/da/de/d14/d34/d40/c97 0 2024-04-12T05:37:22.922 INFO:tasks.workunit.client.1.smithi067.stdout:3/313: creat dc/d1d/d28/d3e/f66 x:0 0 0 2024-04-12T05:37:22.926 INFO:tasks.workunit.client.1.smithi067.stdout:7/413: creat d0/d2/d5/d4a/f86 x:0 0 0 2024-04-12T05:37:22.928 INFO:tasks.workunit.client.1.smithi067.stdout:1/411: getdents d4/d9 0 2024-04-12T05:37:22.928 INFO:tasks.workunit.client.1.smithi067.stdout:0/474: dwrite dd/d12/d14/d1a/d3d/f7c [0,4194304] 0 2024-04-12T05:37:22.929 INFO:tasks.workunit.client.1.smithi067.stdout:6/473: symlink dd/d21/d64/la5 0 2024-04-12T05:37:22.931 INFO:tasks.workunit.client.1.smithi067.stdout:5/624: dread d0/d6/d40/d4b/fb1 [0,4194304] 0 2024-04-12T05:37:22.931 INFO:tasks.workunit.client.0.smithi032.stdout:1/912: sync 2024-04-12T05:37:22.932 INFO:tasks.workunit.client.1.smithi067.stdout:2/480: creat d5/d23/d93/f98 x:0 0 0 2024-04-12T05:37:22.932 INFO:tasks.workunit.client.0.smithi032.stdout:1/913: read - d5/de/d2a/fdd zero size 2024-04-12T05:37:22.935 INFO:tasks.workunit.client.0.smithi032.stdout:1/914: write d5/de/d17/ffd [463473,20661] 0 2024-04-12T05:37:22.935 INFO:tasks.workunit.client.1.smithi067.stdout:5/625: truncate d0/d6/d40/f90 4433381 0 2024-04-12T05:37:22.935 INFO:tasks.workunit.client.1.smithi067.stdout:2/481: dread d5/d43/f44 [0,4194304] 0 2024-04-12T05:37:22.937 INFO:tasks.workunit.client.1.smithi067.stdout:3/314: rename dc/d1d/f31 to dc/d1d/d28/d3e/f67 0 2024-04-12T05:37:22.938 INFO:tasks.workunit.client.0.smithi032.stdout:1/915: truncate d5/de/d17/d35/d61/d77/d8f/d9c/fb5 1101601 0 2024-04-12T05:37:22.939 INFO:tasks.workunit.client.1.smithi067.stdout:5/626: symlink d0/d6/ld0 0 2024-04-12T05:37:22.941 INFO:tasks.workunit.client.1.smithi067.stdout:3/315: chown dc/d1d/d28/d3e/f63 2747 1 2024-04-12T05:37:22.941 INFO:tasks.workunit.client.1.smithi067.stdout:6/474: creat dd/fa6 x:0 0 0 2024-04-12T05:37:22.942 INFO:tasks.workunit.client.0.smithi032.stdout:1/916: creat d5/de/d17/d93/d10c/f122 x:0 0 0 2024-04-12T05:37:22.942 INFO:tasks.workunit.client.1.smithi067.stdout:2/482: chown d5/d43/d6e/c68 260481093 1 2024-04-12T05:37:22.943 INFO:tasks.workunit.client.1.smithi067.stdout:0/475: dwrite dd/d12/d14/d1a/d3d/d41/d64/f7e [0,4194304] 0 2024-04-12T05:37:22.946 INFO:tasks.workunit.client.0.smithi032.stdout:1/917: symlink d5/de/d17/dd6/l123 0 2024-04-12T05:37:22.947 INFO:tasks.workunit.client.1.smithi067.stdout:3/316: chown c7 78943 1 2024-04-12T05:37:22.947 INFO:tasks.workunit.client.1.smithi067.stdout:7/414: getdents d0/d2/dd/d42/d7e 0 2024-04-12T05:37:22.948 INFO:tasks.workunit.client.1.smithi067.stdout:3/317: chown dc/f3a 27421 1 2024-04-12T05:37:22.949 INFO:tasks.workunit.client.1.smithi067.stdout:2/483: creat d5/d6a/d8f/f99 x:0 0 0 2024-04-12T05:37:22.951 INFO:tasks.workunit.client.1.smithi067.stdout:0/476: symlink dd/d12/d14/d1a/d3d/l94 0 2024-04-12T05:37:22.951 INFO:tasks.workunit.client.1.smithi067.stdout:5/627: dread d0/d3a/dab/fcb [0,4194304] 0 2024-04-12T05:37:22.952 INFO:tasks.workunit.client.1.smithi067.stdout:5/628: read - d0/d3a/f83 zero size 2024-04-12T05:37:22.953 INFO:tasks.workunit.client.1.smithi067.stdout:6/475: rename dd/d21/d23/d5e to dd/d21/d64/d6f/da7 0 2024-04-12T05:37:22.956 INFO:tasks.workunit.client.1.smithi067.stdout:3/318: creat dc/d1d/d28/d3e/d53/f68 x:0 0 0 2024-04-12T05:37:22.957 INFO:tasks.workunit.client.1.smithi067.stdout:1/412: truncate d4/d9/d12/d28/f83 185097 0 2024-04-12T05:37:22.958 INFO:tasks.workunit.client.1.smithi067.stdout:0/477: symlink dd/d12/d14/d1a/d3d/d41/l95 0 2024-04-12T05:37:22.959 INFO:tasks.workunit.client.0.smithi032.stdout:1/918: dwrite d5/d2c/d4f/fcd [0,4194304] 0 2024-04-12T05:37:22.961 INFO:tasks.workunit.client.0.smithi032.stdout:1/919: chown d5/de/d17/l1a 60 1 2024-04-12T05:37:22.964 INFO:tasks.workunit.client.1.smithi067.stdout:7/415: unlink d0/d2/d5/d1e/d51/f58 0 2024-04-12T05:37:22.968 INFO:tasks.workunit.client.1.smithi067.stdout:6/476: mknod dd/d21/d66/ca8 0 2024-04-12T05:37:22.970 INFO:tasks.workunit.client.1.smithi067.stdout:1/413: dwrite d4/d52/f70 [0,4194304] 0 2024-04-12T05:37:22.970 INFO:tasks.workunit.client.1.smithi067.stdout:5/629: getdents d0/d6/d40/d4b/d38 0 2024-04-12T05:37:22.971 INFO:tasks.workunit.client.0.smithi032.stdout:1/920: dwrite d5/de/d17/d35/d3e/db4/fbd [0,4194304] 0 2024-04-12T05:37:22.975 INFO:tasks.workunit.client.1.smithi067.stdout:3/319: dwrite dc/d1d/d26/f46 [0,4194304] 0 2024-04-12T05:37:22.975 INFO:tasks.workunit.client.0.smithi032.stdout:1/921: mkdir d5/de/d17/dd6/d124 0 2024-04-12T05:37:22.986 INFO:tasks.workunit.client.0.smithi032.stdout:1/922: dwrite d5/de/d17/d35/d78/f79 [4194304,4194304] 0 2024-04-12T05:37:22.986 INFO:tasks.workunit.client.1.smithi067.stdout:5/630: truncate d0/d3a/f4c 2446560 0 2024-04-12T05:37:22.986 INFO:tasks.workunit.client.1.smithi067.stdout:2/484: dread d5/d43/f53 [0,4194304] 0 2024-04-12T05:37:22.987 INFO:tasks.workunit.client.1.smithi067.stdout:3/320: mkdir dc/d1d/d28/d3e/d69 0 2024-04-12T05:37:22.987 INFO:tasks.workunit.client.0.smithi032.stdout:1/923: chown d5/de/d17/d35/d3b/cff 374 1 2024-04-12T05:37:22.987 INFO:tasks.workunit.client.1.smithi067.stdout:0/478: truncate dd/d12/d14/d1a/f4b 1856354 0 2024-04-12T05:37:22.988 INFO:tasks.workunit.client.1.smithi067.stdout:3/321: chown dc/d1d/d28/f4c 345 1 2024-04-12T05:37:22.989 INFO:tasks.workunit.client.1.smithi067.stdout:6/477: rename dd/d21/d3f/l61 to dd/d11/d27/d4a/d79/la9 0 2024-04-12T05:37:22.991 INFO:tasks.workunit.client.1.smithi067.stdout:1/414: symlink d4/d9/d14/l85 0 2024-04-12T05:37:22.993 INFO:tasks.workunit.client.1.smithi067.stdout:5/631: fdatasync d0/d6/d33/f71 0 2024-04-12T05:37:22.994 INFO:tasks.workunit.client.0.smithi032.stdout:1/924: dread d5/de/d17/d35/d61/d77/d8f/d9c/fe2 [0,4194304] 0 2024-04-12T05:37:22.994 INFO:tasks.workunit.client.1.smithi067.stdout:2/485: rename c0 to d5/da/d8c/c9a 0 2024-04-12T05:37:22.997 INFO:tasks.workunit.client.1.smithi067.stdout:7/416: truncate d0/d2/f21 3233622 0 2024-04-12T05:37:23.001 INFO:tasks.workunit.client.1.smithi067.stdout:3/322: rename dc/d1d/d28/d3e/d53/f68 to dc/df/d27/f6a 0 2024-04-12T05:37:23.001 INFO:tasks.workunit.client.1.smithi067.stdout:5/632: read d0/d3a/f7e [685389,96791] 0 2024-04-12T05:37:23.001 INFO:tasks.workunit.client.1.smithi067.stdout:1/415: symlink d4/d9/d12/d59/l86 0 2024-04-12T05:37:23.001 INFO:tasks.workunit.client.0.smithi032.stdout:9/891: sync 2024-04-12T05:37:23.002 INFO:tasks.workunit.client.0.smithi032.stdout:9/892: readlink d9/d32/l5f 0 2024-04-12T05:37:23.003 INFO:tasks.workunit.client.0.smithi032.stdout:1/925: rename d5/de/d17/d35/d3b/dda/ff6 to d5/de/d17/d35/d61/d77/d8f/d9c/f125 0 2024-04-12T05:37:23.004 INFO:tasks.workunit.client.1.smithi067.stdout:0/479: dwrite dd/d12/f3a [4194304,4194304] 0 2024-04-12T05:37:23.005 INFO:tasks.workunit.client.1.smithi067.stdout:3/323: chown dc/d1d/d28/d3e/l41 11606097 1 2024-04-12T05:37:23.006 INFO:tasks.workunit.client.1.smithi067.stdout:2/486: dread d5/da/de/d14/f63 [0,4194304] 0 2024-04-12T05:37:23.006 INFO:tasks.workunit.client.1.smithi067.stdout:1/416: stat d4/d44/c49 0 2024-04-12T05:37:23.007 INFO:tasks.workunit.client.0.smithi032.stdout:1/926: rename d5/de/d17/dd6/d124 to d5/de/d17/d35/d73/d126 0 2024-04-12T05:37:23.007 INFO:tasks.workunit.client.1.smithi067.stdout:7/417: creat d0/d2/d5/f87 x:0 0 0 2024-04-12T05:37:23.008 INFO:tasks.workunit.client.0.smithi032.stdout:1/927: stat d5/de/d17/d35/d3b/dda 0 2024-04-12T05:37:23.008 INFO:tasks.workunit.client.1.smithi067.stdout:3/324: mkdir dc/df/d27/d6b 0 2024-04-12T05:37:23.009 INFO:tasks.workunit.client.1.smithi067.stdout:1/417: fsync d4/d9/d12/d2c/f50 0 2024-04-12T05:37:23.011 INFO:tasks.workunit.client.1.smithi067.stdout:3/325: write dc/f40 [714278,124201] 0 2024-04-12T05:37:23.012 INFO:tasks.workunit.client.0.smithi032.stdout:1/928: mkdir d5/de/d17/d35/d61/d77/d8f/da0/d127 0 2024-04-12T05:37:23.015 INFO:tasks.workunit.client.0.smithi032.stdout:1/929: unlink d5/l6e 0 2024-04-12T05:37:23.015 INFO:tasks.workunit.client.1.smithi067.stdout:2/487: getdents d5/d6a/d8f 0 2024-04-12T05:37:23.017 INFO:tasks.workunit.client.1.smithi067.stdout:3/326: mknod dc/d50/c6c 0 2024-04-12T05:37:23.017 INFO:tasks.workunit.client.0.smithi032.stdout:1/930: creat d5/de/d17/d35/d61/d105/f128 x:0 0 0 2024-04-12T05:37:23.018 INFO:tasks.workunit.client.1.smithi067.stdout:3/327: write dc/d1d/d28/d3e/f4a [264628,11357] 0 2024-04-12T05:37:23.019 INFO:tasks.workunit.client.1.smithi067.stdout:1/418: creat d4/d9/d12/d59/f87 x:0 0 0 2024-04-12T05:37:23.021 INFO:tasks.workunit.client.0.smithi032.stdout:1/931: truncate d5/fb8 1112480 0 2024-04-12T05:37:23.021 INFO:tasks.workunit.client.1.smithi067.stdout:3/328: write dc/d1d/f44 [305063,87927] 0 2024-04-12T05:37:23.021 INFO:tasks.workunit.client.1.smithi067.stdout:1/419: chown d4/d9/d12/f81 13609688 1 2024-04-12T05:37:23.022 INFO:tasks.workunit.client.1.smithi067.stdout:3/329: dread - dc/d1d/d28/d3e/f66 zero size 2024-04-12T05:37:23.023 INFO:tasks.workunit.client.0.smithi032.stdout:9/893: write f5 [50408,79511] 0 2024-04-12T05:37:23.025 INFO:tasks.workunit.client.0.smithi032.stdout:1/932: mknod d5/de/d17/d35/d61/d77/d8f/c129 0 2024-04-12T05:37:23.026 INFO:tasks.workunit.client.0.smithi032.stdout:9/894: unlink d9/d15/d2c/f23 0 2024-04-12T05:37:23.027 INFO:tasks.workunit.client.1.smithi067.stdout:0/480: rename dd/d12/d14/d1a/d3d/l94 to dd/d12/d14/l96 0 2024-04-12T05:37:23.027 INFO:tasks.workunit.client.1.smithi067.stdout:0/481: chown dd/d12 1690662 1 2024-04-12T05:37:23.029 INFO:tasks.workunit.client.0.smithi032.stdout:9/895: mkdir d9/d15/d1a/d1d/d5a/d135 0 2024-04-12T05:37:23.029 INFO:tasks.workunit.client.1.smithi067.stdout:3/330: creat dc/d1d/d28/d3e/f6d x:0 0 0 2024-04-12T05:37:23.030 INFO:tasks.workunit.client.0.smithi032.stdout:9/896: write f5 [2661044,31003] 0 2024-04-12T05:37:23.030 INFO:tasks.workunit.client.1.smithi067.stdout:5/633: link d0/d6/d20/d62/c61 d0/d6/d20/d3e/dbf/cd1 0 2024-04-12T05:37:23.031 INFO:tasks.workunit.client.1.smithi067.stdout:6/478: write dd/d11/d27/f8f [116424,26984] 0 2024-04-12T05:37:23.033 INFO:tasks.workunit.client.1.smithi067.stdout:7/418: link d0/d2/d5/d1d/d43/c7a d0/d67/d57/d85/c88 0 2024-04-12T05:37:23.034 INFO:tasks.workunit.client.1.smithi067.stdout:7/419: chown d0/d2/dd/d42/d7e/d34 3729 1 2024-04-12T05:37:23.035 INFO:tasks.workunit.client.0.smithi032.stdout:9/897: link d9/d15/d1a/d24/d27/c117 d9/d15/d1a/dd2/c136 0 2024-04-12T05:37:23.037 INFO:tasks.workunit.client.0.smithi032.stdout:9/898: mknod d9/d32/d86/dfe/c137 0 2024-04-12T05:37:23.037 INFO:tasks.workunit.client.1.smithi067.stdout:0/482: rename dd/d12/d14/f39 to dd/d6f/d72/f97 0 2024-04-12T05:37:23.040 INFO:tasks.workunit.client.1.smithi067.stdout:7/420: creat d0/d23/d45/f89 x:0 0 0 2024-04-12T05:37:23.041 INFO:tasks.workunit.client.1.smithi067.stdout:5/634: mknod d0/d6/d20/cd2 0 2024-04-12T05:37:23.043 INFO:tasks.workunit.client.0.smithi032.stdout:1/933: dread d5/de/d17/d35/fa1 [0,4194304] 0 2024-04-12T05:37:23.043 INFO:tasks.workunit.client.1.smithi067.stdout:6/479: rename dd/d11/d27/f5a to dd/d21/d64/d88/d7b/faa 0 2024-04-12T05:37:23.044 INFO:tasks.workunit.client.1.smithi067.stdout:3/331: link dc/d50/l54 dc/df/l6e 0 2024-04-12T05:37:23.046 INFO:tasks.workunit.client.1.smithi067.stdout:3/332: write dc/d1d/d28/f64 [796600,122737] 0 2024-04-12T05:37:23.048 INFO:tasks.workunit.client.1.smithi067.stdout:0/483: dread dd/d12/d14/d1a/f4b [0,4194304] 0 2024-04-12T05:37:23.048 INFO:tasks.workunit.client.0.smithi032.stdout:1/934: dread d5/de/d17/d35/d78/f83 [0,4194304] 0 2024-04-12T05:37:23.049 INFO:tasks.workunit.client.1.smithi067.stdout:5/635: rename d0/d6/d20/d3e/d69 to d0/d6/d33/dd3 0 2024-04-12T05:37:23.050 INFO:tasks.workunit.client.0.smithi032.stdout:1/935: stat d5/de/d17/d112/f114 0 2024-04-12T05:37:23.050 INFO:tasks.workunit.client.0.smithi032.stdout:1/936: chown d5/de/c15 650316 1 2024-04-12T05:37:23.051 INFO:tasks.workunit.client.0.smithi032.stdout:9/899: truncate d9/d15/d1a/dd2/f98 3249459 0 2024-04-12T05:37:23.051 INFO:tasks.workunit.client.0.smithi032.stdout:1/937: read d5/de/d17/d35/d3e/db4/fbd [1243619,122423] 0 2024-04-12T05:37:23.053 INFO:tasks.workunit.client.0.smithi032.stdout:9/900: dread - d9/d15/d1a/d1d/d5a/fe8 zero size 2024-04-12T05:37:23.053 INFO:tasks.workunit.client.0.smithi032.stdout:1/938: readlink d5/de/d17/d35/d73/lc5 0 2024-04-12T05:37:23.055 INFO:tasks.workunit.client.1.smithi067.stdout:6/480: rename dd/d21/d64/d6f/l81 to dd/d21/d23/d9e/lab 0 2024-04-12T05:37:23.057 INFO:tasks.workunit.client.1.smithi067.stdout:5/636: creat d0/d6/d40/fd4 x:0 0 0 2024-04-12T05:37:23.057 INFO:tasks.workunit.client.1.smithi067.stdout:3/333: mknod dc/df/d27/d6b/c6f 0 2024-04-12T05:37:23.057 INFO:tasks.workunit.client.0.smithi032.stdout:9/901: unlink d9/d15/d1a/dd2/d65/d80/d87/cfb 0 2024-04-12T05:37:23.057 INFO:tasks.workunit.client.1.smithi067.stdout:6/481: chown dd/d21/d64/d6f/f70 10970403 1 2024-04-12T05:37:23.058 INFO:tasks.workunit.client.0.smithi032.stdout:9/902: dread - d9/d32/f111 zero size 2024-04-12T05:37:23.058 INFO:tasks.workunit.client.1.smithi067.stdout:7/421: dread d0/d2/dd/d42/d7e/f65 [0,4194304] 0 2024-04-12T05:37:23.059 INFO:tasks.workunit.client.1.smithi067.stdout:6/482: chown dd/d21/d64/d88 5 1 2024-04-12T05:37:23.060 INFO:tasks.workunit.client.0.smithi032.stdout:1/939: write d5/de/d17/d1b/f1d [2917877,125140] 0 2024-04-12T05:37:23.060 INFO:tasks.workunit.client.1.smithi067.stdout:5/637: chown d0/d6/d40/d4b/d38/d8e/d8f/d99/f9a 996 1 2024-04-12T05:37:23.060 INFO:tasks.workunit.client.1.smithi067.stdout:0/484: rename dd/d92/f8a to dd/d6f/f98 0 2024-04-12T05:37:23.063 INFO:tasks.workunit.client.0.smithi032.stdout:9/903: fsync d9/d15/d1a/dd2/d65/d80/d87/ff9 0 2024-04-12T05:37:23.064 INFO:tasks.workunit.client.1.smithi067.stdout:5/638: symlink d0/da/d41/ld5 0 2024-04-12T05:37:23.065 INFO:tasks.workunit.client.0.smithi032.stdout:1/940: mkdir d5/de/d17/d35/d61/d12a 0 2024-04-12T05:37:23.066 INFO:tasks.workunit.client.1.smithi067.stdout:3/334: rename dc/d1d/d28/d3e/c51 to dc/d1d/d28/d3e/d69/c70 0 2024-04-12T05:37:23.066 INFO:tasks.workunit.client.0.smithi032.stdout:1/941: fdatasync d5/de/d17/fa7 0 2024-04-12T05:37:23.067 INFO:tasks.workunit.client.1.smithi067.stdout:6/483: dwrite dd/f17 [0,4194304] 0 2024-04-12T05:37:23.068 INFO:tasks.workunit.client.0.smithi032.stdout:9/904: creat d9/d15/d1a/dd2/d65/f138 x:0 0 0 2024-04-12T05:37:23.069 INFO:tasks.workunit.client.1.smithi067.stdout:3/335: stat dc/df 0 2024-04-12T05:37:23.069 INFO:tasks.workunit.client.1.smithi067.stdout:0/485: symlink dd/d12/d14/d1a/d3d/d41/d5f/d78/l99 0 2024-04-12T05:37:23.069 INFO:tasks.workunit.client.0.smithi032.stdout:1/942: fdatasync d5/de/f4a 0 2024-04-12T05:37:23.070 INFO:tasks.workunit.client.1.smithi067.stdout:6/484: fsync dd/d11/d27/d4a/f95 0 2024-04-12T05:37:23.071 INFO:tasks.workunit.client.1.smithi067.stdout:3/336: chown dc/df/d27/f6a 113 1 2024-04-12T05:37:23.071 INFO:tasks.workunit.client.1.smithi067.stdout:5/639: dread d0/d6/f10 [0,4194304] 0 2024-04-12T05:37:23.072 INFO:tasks.workunit.client.0.smithi032.stdout:1/943: creat d5/de/d17/f12b x:0 0 0 2024-04-12T05:37:23.073 INFO:tasks.workunit.client.1.smithi067.stdout:3/337: write dc/d1d/d26/f5e [974923,125332] 0 2024-04-12T05:37:23.074 INFO:tasks.workunit.client.1.smithi067.stdout:0/486: creat dd/d12/d14/d1a/d3d/d41/d5f/d78/d83/f9a x:0 0 0 2024-04-12T05:37:23.076 INFO:tasks.workunit.client.0.smithi032.stdout:1/944: mkdir d5/de/dbe/d12c 0 2024-04-12T05:37:23.077 INFO:tasks.workunit.client.1.smithi067.stdout:6/485: unlink dd/d21/d64/d88/d7b/f84 0 2024-04-12T05:37:23.078 INFO:tasks.workunit.client.1.smithi067.stdout:6/486: dread - dd/d21/f9c zero size 2024-04-12T05:37:23.079 INFO:tasks.workunit.client.1.smithi067.stdout:3/338: stat dc/d1d/d28/l5b 0 2024-04-12T05:37:23.081 INFO:tasks.workunit.client.1.smithi067.stdout:7/422: write d0/d23/f27 [1654418,4050] 0 2024-04-12T05:37:23.081 INFO:tasks.workunit.client.0.smithi032.stdout:1/945: creat d5/de/d17/d35/d61/d77/d8f/d9c/d10d/f12d x:0 0 0 2024-04-12T05:37:23.082 INFO:tasks.workunit.client.1.smithi067.stdout:0/487: mkdir dd/d12/d14/d49/d9b 0 2024-04-12T05:37:23.083 INFO:tasks.workunit.client.1.smithi067.stdout:5/640: symlink d0/d6/d20/d3e/dbf/ld6 0 2024-04-12T05:37:23.084 INFO:tasks.workunit.client.1.smithi067.stdout:7/423: chown d0/d2/d5/d1d/f36 264 1 2024-04-12T05:37:23.085 INFO:tasks.workunit.client.0.smithi032.stdout:1/946: unlink d5/de/d17/d1b/f5f 0 2024-04-12T05:37:23.086 INFO:tasks.workunit.client.1.smithi067.stdout:7/424: fsync d0/d2/d5/d1d/f3a 0 2024-04-12T05:37:23.086 INFO:tasks.workunit.client.0.smithi032.stdout:1/947: chown d5/de/f66 70 1 2024-04-12T05:37:23.087 INFO:tasks.workunit.client.1.smithi067.stdout:3/339: dread dc/df/d27/f3f [0,4194304] 0 2024-04-12T05:37:23.087 INFO:tasks.workunit.client.1.smithi067.stdout:5/641: symlink d0/d6/d33/ld7 0 2024-04-12T05:37:23.087 INFO:tasks.workunit.client.1.smithi067.stdout:6/487: dread dd/d21/d23/f25 [0,4194304] 0 2024-04-12T05:37:23.087 INFO:tasks.workunit.client.0.smithi032.stdout:9/905: dwrite d9/d15/d1a/f56 [0,4194304] 0 2024-04-12T05:37:23.088 INFO:tasks.workunit.client.1.smithi067.stdout:7/425: stat d0/d2/d5/d1e/l81 0 2024-04-12T05:37:23.088 INFO:tasks.workunit.client.1.smithi067.stdout:3/340: dread - dc/df/d27/f6a zero size 2024-04-12T05:37:23.089 INFO:tasks.workunit.client.0.smithi032.stdout:1/948: unlink d5/c4e 0 2024-04-12T05:37:23.090 INFO:tasks.workunit.client.1.smithi067.stdout:5/642: chown d0/d3a/d2c/d91/l7b 64 1 2024-04-12T05:37:23.093 INFO:tasks.workunit.client.1.smithi067.stdout:7/426: symlink d0/d23/d45/l8a 0 2024-04-12T05:37:23.093 INFO:tasks.workunit.client.0.smithi032.stdout:1/949: symlink d5/de/d17/d35/d3b/dda/de0/l12e 0 2024-04-12T05:37:23.094 INFO:tasks.workunit.client.0.smithi032.stdout:9/906: dwrite d9/d32/d86/dcc/f94 [0,4194304] 0 2024-04-12T05:37:23.098 INFO:tasks.workunit.client.1.smithi067.stdout:3/341: dread dc/df/f1c [0,4194304] 0 2024-04-12T05:37:23.099 INFO:tasks.workunit.client.0.smithi032.stdout:1/950: dread d5/de/d17/d35/d78/fad [0,4194304] 0 2024-04-12T05:37:23.104 INFO:tasks.workunit.client.0.smithi032.stdout:1/951: creat d5/de/d17/d35/d3b/dda/f12f x:0 0 0 2024-04-12T05:37:23.106 INFO:tasks.workunit.client.0.smithi032.stdout:1/952: dread d5/de/d17/d35/d61/d77/d8f/d9c/fe2 [0,4194304] 0 2024-04-12T05:37:23.107 INFO:tasks.workunit.client.1.smithi067.stdout:0/488: dread dd/d92/f4c [0,4194304] 0 2024-04-12T05:37:23.107 INFO:tasks.workunit.client.1.smithi067.stdout:3/342: mknod dc/d1d/d28/d3e/d53/c71 0 2024-04-12T05:37:23.108 INFO:tasks.workunit.client.1.smithi067.stdout:0/489: stat dd/d5b 0 2024-04-12T05:37:23.108 INFO:tasks.workunit.client.0.smithi032.stdout:1/953: truncate d5/de/d17/ffd 1097079 0 2024-04-12T05:37:23.109 INFO:tasks.workunit.client.1.smithi067.stdout:0/490: stat dd/d12/d14/d1a/d3d/d41/l7f 0 2024-04-12T05:37:23.110 INFO:tasks.workunit.client.1.smithi067.stdout:7/427: rmdir d0/d2/d5/d1d/d43/d6b 0 2024-04-12T05:37:23.111 INFO:tasks.workunit.client.1.smithi067.stdout:0/491: chown dd/d12/f21 82210561 1 2024-04-12T05:37:23.111 INFO:tasks.workunit.client.0.smithi032.stdout:1/954: unlink d5/de/d17/d35/d3b/dda/f12f 0 2024-04-12T05:37:23.117 INFO:tasks.workunit.client.1.smithi067.stdout:8/360: sync 2024-04-12T05:37:23.117 INFO:tasks.workunit.client.1.smithi067.stdout:7/428: read d0/d2/f21 [929542,48541] 0 2024-04-12T05:37:23.118 INFO:tasks.workunit.client.1.smithi067.stdout:6/488: dread fb [0,4194304] 0 2024-04-12T05:37:23.119 INFO:tasks.workunit.client.1.smithi067.stdout:5/643: dwrite d0/da/fce [0,4194304] 0 2024-04-12T05:37:23.119 INFO:tasks.workunit.client.1.smithi067.stdout:6/489: readlink dd/d11/l85 0 2024-04-12T05:37:23.120 INFO:tasks.workunit.client.1.smithi067.stdout:7/429: fdatasync d0/d2/d5/f50 0 2024-04-12T05:37:23.122 INFO:tasks.workunit.client.1.smithi067.stdout:8/361: symlink d6/d8/d18/d4e/l6d 0 2024-04-12T05:37:23.122 INFO:tasks.workunit.client.1.smithi067.stdout:3/343: dread dc/d1d/d28/d3e/f67 [0,4194304] 0 2024-04-12T05:37:23.123 INFO:tasks.workunit.client.1.smithi067.stdout:5/644: unlink d0/d6/d20/d62/l6f 0 2024-04-12T05:37:23.123 INFO:tasks.workunit.client.1.smithi067.stdout:6/490: write dd/d11/d27/f54 [397478,74204] 0 2024-04-12T05:37:23.126 INFO:tasks.workunit.client.1.smithi067.stdout:6/491: stat dd/d21/d64/d6f/c72 0 2024-04-12T05:37:23.126 INFO:tasks.workunit.client.1.smithi067.stdout:3/344: read - dc/d1d/d28/d3e/f66 zero size 2024-04-12T05:37:23.127 INFO:tasks.workunit.client.0.smithi032.stdout:1/955: write d5/de/d17/d35/d61/f102 [20804,93752] 0 2024-04-12T05:37:23.128 INFO:tasks.workunit.client.1.smithi067.stdout:8/362: fsync d6/d8/d18/d3c/d4f/f65 0 2024-04-12T05:37:23.128 INFO:tasks.workunit.client.1.smithi067.stdout:7/430: mknod d0/d2/d5/d1d/d7d/c8b 0 2024-04-12T05:37:23.128 INFO:tasks.workunit.client.1.smithi067.stdout:5/645: symlink d0/d3a/dab/ld8 0 2024-04-12T05:37:23.129 INFO:tasks.workunit.client.1.smithi067.stdout:6/492: creat dd/d11/fac x:0 0 0 2024-04-12T05:37:23.130 INFO:tasks.workunit.client.1.smithi067.stdout:6/493: dread - dd/fa6 zero size 2024-04-12T05:37:23.131 INFO:tasks.workunit.client.1.smithi067.stdout:3/345: write dc/d1d/d28/f62 [105499,79736] 0 2024-04-12T05:37:23.132 INFO:tasks.workunit.client.0.smithi032.stdout:9/907: dread d9/fa [0,4194304] 0 2024-04-12T05:37:23.133 INFO:tasks.workunit.client.1.smithi067.stdout:8/363: mkdir d6/d8/d6e 0 2024-04-12T05:37:23.133 INFO:tasks.workunit.client.0.smithi032.stdout:1/956: mknod d5/de/d17/d35/d3b/c130 0 2024-04-12T05:37:23.134 INFO:tasks.workunit.client.1.smithi067.stdout:7/431: symlink d0/d2/dd/d79/l8c 0 2024-04-12T05:37:23.135 INFO:tasks.workunit.client.0.smithi032.stdout:9/908: mknod d9/d32/d86/dfe/c139 0 2024-04-12T05:37:23.136 INFO:tasks.workunit.client.1.smithi067.stdout:6/494: mknod dd/d21/d66/cad 0 2024-04-12T05:37:23.138 INFO:tasks.workunit.client.0.smithi032.stdout:1/957: creat d5/de/d17/d35/d78/f131 x:0 0 0 2024-04-12T05:37:23.140 INFO:tasks.workunit.client.1.smithi067.stdout:5/646: creat d0/d6/d40/d4b/d38/d8e/d8f/db9/dbd/fd9 x:0 0 0 2024-04-12T05:37:23.140 INFO:tasks.workunit.client.1.smithi067.stdout:7/432: mknod d0/d2/d5/c8d 0 2024-04-12T05:37:23.142 INFO:tasks.workunit.client.0.smithi032.stdout:1/958: mkdir d5/de/d17/d93/d132 0 2024-04-12T05:37:23.142 INFO:tasks.workunit.client.1.smithi067.stdout:6/495: unlink dd/d11/l18 0 2024-04-12T05:37:23.142 INFO:tasks.workunit.client.0.smithi032.stdout:9/909: dwrite d9/d32/d86/dcc/f94 [0,4194304] 0 2024-04-12T05:37:23.152 INFO:tasks.workunit.client.0.smithi032.stdout:9/910: symlink d9/d15/d1a/d24/d27/dce/dd3/de9/l13a 0 2024-04-12T05:37:23.153 INFO:tasks.workunit.client.1.smithi067.stdout:0/492: dwrite dd/d12/d14/d1a/f4b [0,4194304] 0 2024-04-12T05:37:23.153 INFO:tasks.workunit.client.1.smithi067.stdout:3/346: dwrite dc/d1d/d28/d3e/f6d [0,4194304] 0 2024-04-12T05:37:23.153 INFO:tasks.workunit.client.1.smithi067.stdout:5/647: creat d0/d6/d40/d4b/fda x:0 0 0 2024-04-12T05:37:23.153 INFO:tasks.workunit.client.0.smithi032.stdout:1/959: link d5/de/f10 d5/de/d17/d35/d61/f133 0 2024-04-12T05:37:23.155 INFO:tasks.workunit.client.0.smithi032.stdout:9/911: symlink d9/d32/d6e/l13b 0 2024-04-12T05:37:23.157 INFO:tasks.workunit.client.1.smithi067.stdout:8/364: dwrite d6/d8/f14 [0,4194304] 0 2024-04-12T05:37:23.161 INFO:tasks.workunit.client.0.smithi032.stdout:1/960: truncate d5/de/d17/d35/d61/d77/d8f/da0/fe1 863089 0 2024-04-12T05:37:23.164 INFO:tasks.workunit.client.0.smithi032.stdout:1/961: chown d5/d2c/d4f/c8d 12 1 2024-04-12T05:37:23.167 INFO:tasks.workunit.client.0.smithi032.stdout:9/912: rename d9/d15/d1a/d24/d27/de1 to d9/d32/d6e/d13c 0 2024-04-12T05:37:23.168 INFO:tasks.workunit.client.1.smithi067.stdout:7/433: dwrite d0/d23/d45/f59 [0,4194304] 0 2024-04-12T05:37:23.169 INFO:tasks.workunit.client.1.smithi067.stdout:6/496: dwrite dd/d11/f91 [0,4194304] 0 2024-04-12T05:37:23.170 INFO:tasks.workunit.client.1.smithi067.stdout:7/434: stat d0/d2/d5/d1e/f7c 0 2024-04-12T05:37:23.171 INFO:tasks.workunit.client.1.smithi067.stdout:8/365: mkdir d6/d6f 0 2024-04-12T05:37:23.172 INFO:tasks.workunit.client.1.smithi067.stdout:7/435: fdatasync d0/d2/dd/d79/f6d 0 2024-04-12T05:37:23.176 INFO:tasks.workunit.client.1.smithi067.stdout:6/497: symlink dd/d21/d64/d6f/d90/d98/lae 0 2024-04-12T05:37:23.177 INFO:tasks.workunit.client.0.smithi032.stdout:9/913: rename d9/d15/d1a/d24/d27/d88/dc1/fd6 to d9/d15/d1a/d24/d27/dce/dd3/f13d 0 2024-04-12T05:37:23.180 INFO:tasks.workunit.client.1.smithi067.stdout:5/648: getdents d0/d6/d20 0 2024-04-12T05:37:23.180 INFO:tasks.workunit.client.1.smithi067.stdout:3/347: write f5 [7020837,30402] 0 2024-04-12T05:37:23.180 INFO:tasks.workunit.client.1.smithi067.stdout:8/366: creat d6/d8/d18/d36/f70 x:0 0 0 2024-04-12T05:37:23.180 INFO:tasks.workunit.client.1.smithi067.stdout:7/436: rename d0/d2/d5/f87 to d0/d67/d57/d85/f8e 0 2024-04-12T05:37:23.182 INFO:tasks.workunit.client.1.smithi067.stdout:8/367: chown d6/d8/d6e 14693 1 2024-04-12T05:37:23.182 INFO:tasks.workunit.client.0.smithi032.stdout:9/914: read - d9/d15/d1a/dd2/d65/d80/d87/fc2 zero size 2024-04-12T05:37:23.182 INFO:tasks.workunit.client.0.smithi032.stdout:1/962: dread d5/de/d17/d35/d73/f8a [0,4194304] 0 2024-04-12T05:37:23.183 INFO:tasks.workunit.client.0.smithi032.stdout:9/915: mknod d9/d15/d2c/d41/c13e 0 2024-04-12T05:37:23.184 INFO:tasks.workunit.client.1.smithi067.stdout:7/437: creat d0/d2/dd/d42/d7e/d34/f8f x:0 0 0 2024-04-12T05:37:23.185 INFO:tasks.workunit.client.0.smithi032.stdout:1/963: rename d5/de/d17/f26 to d5/de/d17/d93/d10c/f134 0 2024-04-12T05:37:23.186 INFO:tasks.workunit.client.1.smithi067.stdout:5/649: symlink d0/d6/d40/d4b/d38/d8e/d8f/ldb 0 2024-04-12T05:37:23.187 INFO:tasks.workunit.client.0.smithi032.stdout:9/916: fdatasync d9/d15/fc3 0 2024-04-12T05:37:23.187 INFO:tasks.workunit.client.1.smithi067.stdout:3/348: rename dc/df/f1e to dc/d1d/d26/f72 0 2024-04-12T05:37:23.188 INFO:tasks.workunit.client.0.smithi032.stdout:1/964: fsync d5/de/d2a/f80 0 2024-04-12T05:37:23.189 INFO:tasks.workunit.client.0.smithi032.stdout:9/917: dread - d9/d15/d1a/d24/df6/f132 zero size 2024-04-12T05:37:23.189 INFO:tasks.workunit.client.0.smithi032.stdout:9/918: chown d9/ccd 28 1 2024-04-12T05:37:23.190 INFO:tasks.workunit.client.1.smithi067.stdout:8/368: creat d6/d8/d18/d63/d67/d69/f71 x:0 0 0 2024-04-12T05:37:23.191 INFO:tasks.workunit.client.0.smithi032.stdout:9/919: mkdir d9/d15/d1a/dba/d13f 0 2024-04-12T05:37:23.192 INFO:tasks.workunit.client.1.smithi067.stdout:5/650: mkdir d0/d6/d40/d4b/d38/d8e/d8f/db9/dbd/ddc 0 2024-04-12T05:37:23.193 INFO:tasks.workunit.client.1.smithi067.stdout:3/349: mkdir dc/d1d/d28/d3e/d69/d73 0 2024-04-12T05:37:23.195 INFO:tasks.workunit.client.0.smithi032.stdout:1/965: dwrite d5/de/d17/d93/fc8 [0,4194304] 0 2024-04-12T05:37:23.197 INFO:tasks.workunit.client.1.smithi067.stdout:3/350: write dc/d1d/d28/f62 [156832,88954] 0 2024-04-12T05:37:23.201 INFO:tasks.workunit.client.0.smithi032.stdout:1/966: creat d5/de/d17/dd6/de5/f135 x:0 0 0 2024-04-12T05:37:23.202 INFO:tasks.workunit.client.1.smithi067.stdout:7/438: link d0/d23/d45/l70 d0/d2/l90 0 2024-04-12T05:37:23.205 INFO:tasks.workunit.client.0.smithi032.stdout:9/920: link d9/d15/d1a/d1d/d5a/d6b/d83/l116 d9/l140 0 2024-04-12T05:37:23.206 INFO:tasks.workunit.client.1.smithi067.stdout:8/369: dwrite d6/d8/d18/d36/f59 [0,4194304] 0 2024-04-12T05:37:23.209 INFO:tasks.workunit.client.1.smithi067.stdout:0/493: dwrite dd/d12/d14/d1a/d3d/d41/f50 [0,4194304] 0 2024-04-12T05:37:23.210 INFO:tasks.workunit.client.1.smithi067.stdout:5/651: dread d0/da/fce [0,4194304] 0 2024-04-12T05:37:23.211 INFO:tasks.workunit.client.0.smithi032.stdout:1/967: getdents d5/de/d17 0 2024-04-12T05:37:23.211 INFO:tasks.workunit.client.1.smithi067.stdout:7/439: symlink d0/d2/d5/d4a/l91 0 2024-04-12T05:37:23.212 INFO:tasks.workunit.client.1.smithi067.stdout:8/370: stat d6/d8/d18/d63/d67/d69/c39 0 2024-04-12T05:37:23.216 INFO:tasks.workunit.client.0.smithi032.stdout:9/921: getdents d9/d15/d1a/d24/d27/dce 0 2024-04-12T05:37:23.218 INFO:tasks.workunit.client.0.smithi032.stdout:9/922: chown d9/d32/d86/dcc/c10c 6667158 1 2024-04-12T05:37:23.219 INFO:tasks.workunit.client.0.smithi032.stdout:1/968: rename d5/d2c/f43 to d5/de/d17/d35/d61/d77/d8f/f136 0 2024-04-12T05:37:23.220 INFO:tasks.workunit.client.1.smithi067.stdout:0/494: chown dd/d12/d14/d51/f5c 41773 1 2024-04-12T05:37:23.220 INFO:tasks.workunit.client.1.smithi067.stdout:0/495: chown dd/d92 1789 1 2024-04-12T05:37:23.220 INFO:tasks.workunit.client.1.smithi067.stdout:6/498: dwrite dd/d21/d23/f65 [0,4194304] 0 2024-04-12T05:37:23.220 INFO:tasks.workunit.client.1.smithi067.stdout:3/351: write dc/d1d/f24 [499297,89365] 0 2024-04-12T05:37:23.221 INFO:tasks.workunit.client.1.smithi067.stdout:5/652: read d0/da/d41/f6b [1750598,126739] 0 2024-04-12T05:37:23.223 INFO:tasks.workunit.client.0.smithi032.stdout:9/923: mkdir d9/d15/d1a/d24/d27/dce/dd3/d109/d141 0 2024-04-12T05:37:23.229 INFO:tasks.workunit.client.0.smithi032.stdout:9/924: creat d9/d15/d2c/d41/f142 x:0 0 0 2024-04-12T05:37:23.230 INFO:tasks.workunit.client.0.smithi032.stdout:9/925: write d9/d1f/f5d [4188602,80268] 0 2024-04-12T05:37:23.231 INFO:tasks.workunit.client.1.smithi067.stdout:8/371: rmdir d6/d8/d18 39 2024-04-12T05:37:23.232 INFO:tasks.workunit.client.0.smithi032.stdout:1/969: link d5/de/d17/d35/d73/fba d5/de/d17/d35/d61/d77/d8f/d9c/d10d/f137 0 2024-04-12T05:37:23.233 INFO:tasks.workunit.client.0.smithi032.stdout:9/926: readlink d9/d15/d1a/d69/le5 0 2024-04-12T05:37:23.235 INFO:tasks.workunit.client.1.smithi067.stdout:3/352: fdatasync dc/d1d/d28/d3e/f4a 0 2024-04-12T05:37:23.235 INFO:tasks.workunit.client.1.smithi067.stdout:6/499: write dd/d11/f91 [3592259,33002] 0 2024-04-12T05:37:23.236 INFO:tasks.workunit.client.0.smithi032.stdout:9/927: mknod d9/d15/d1a/d24/d27/d88/dd9/c143 0 2024-04-12T05:37:23.237 INFO:tasks.workunit.client.0.smithi032.stdout:1/970: creat d5/de/d17/d35/d3b/f138 x:0 0 0 2024-04-12T05:37:23.239 INFO:tasks.workunit.client.1.smithi067.stdout:7/440: dwrite d0/f12 [4194304,4194304] 0 2024-04-12T05:37:23.241 INFO:tasks.workunit.client.0.smithi032.stdout:9/928: rename d9/d32/d86/dcc/c11b to d9/d15/d1a/dd2/d65/c144 0 2024-04-12T05:37:23.241 INFO:tasks.workunit.client.1.smithi067.stdout:8/372: write d6/d8/d18/f5e [137068,73965] 0 2024-04-12T05:37:23.242 INFO:tasks.workunit.client.0.smithi032.stdout:1/971: mknod d5/de/d17/d35/d73/c139 0 2024-04-12T05:37:23.243 INFO:tasks.workunit.client.0.smithi032.stdout:9/929: stat d9/d15/d1a/dd2/d65/fa6 0 2024-04-12T05:37:23.243 INFO:tasks.workunit.client.1.smithi067.stdout:3/353: mkdir dc/d1d/d28/d3e/d74 0 2024-04-12T05:37:23.244 INFO:tasks.workunit.client.1.smithi067.stdout:0/496: rename dd/d12/d14/d1a/d3d/d41 to dd/d9c 0 2024-04-12T05:37:23.245 INFO:tasks.workunit.client.1.smithi067.stdout:6/500: mknod dd/d11/d27/caf 0 2024-04-12T05:37:23.247 INFO:tasks.workunit.client.1.smithi067.stdout:3/354: write dc/d1d/d4d/f55 [2497539,5610] 0 2024-04-12T05:37:23.249 INFO:tasks.workunit.client.1.smithi067.stdout:5/653: dwrite d0/da/d41/f7a [0,4194304] 0 2024-04-12T05:37:23.254 INFO:tasks.workunit.client.0.smithi032.stdout:1/972: rmdir d5/de/d17/d35/d61/dca/d104 0 2024-04-12T05:37:23.255 INFO:tasks.workunit.client.0.smithi032.stdout:1/973: write d5/de/d17/ffd [139756,112272] 0 2024-04-12T05:37:23.256 INFO:tasks.workunit.client.1.smithi067.stdout:8/373: rmdir d6/d6f 0 2024-04-12T05:37:23.260 INFO:tasks.workunit.client.0.smithi032.stdout:1/974: mkdir d5/de/d17/d35/d61/d105/d108/d13a 0 2024-04-12T05:37:23.265 INFO:tasks.workunit.client.1.smithi067.stdout:7/441: dwrite d0/d23/f27 [0,4194304] 0 2024-04-12T05:37:23.265 INFO:tasks.workunit.client.0.smithi032.stdout:9/930: getdents d9/d32/d86/dcc 0 2024-04-12T05:37:23.268 INFO:tasks.workunit.client.0.smithi032.stdout:9/931: dread - d9/d32/f111 zero size 2024-04-12T05:37:23.268 INFO:tasks.workunit.client.0.smithi032.stdout:1/975: dread d5/d2c/f3c [0,4194304] 0 2024-04-12T05:37:23.270 INFO:tasks.workunit.client.1.smithi067.stdout:3/355: dwrite dc/d1d/d4d/f55 [0,4194304] 0 2024-04-12T05:37:23.270 INFO:tasks.workunit.client.1.smithi067.stdout:8/374: creat d6/d8/f72 x:0 0 0 2024-04-12T05:37:23.270 INFO:tasks.workunit.client.1.smithi067.stdout:7/442: chown d0/d67/c35 1265693386 1 2024-04-12T05:37:23.273 INFO:tasks.workunit.client.1.smithi067.stdout:6/501: dwrite dd/d11/d27/f46 [0,4194304] 0 2024-04-12T05:37:23.273 INFO:tasks.workunit.client.1.smithi067.stdout:5/654: mkdir d0/d3a/d2c/dc6/ddd 0 2024-04-12T05:37:23.274 INFO:tasks.workunit.client.0.smithi032.stdout:1/976: unlink d5/de/d17/d35/d61/d77/d8f/d9c/fa5 0 2024-04-12T05:37:23.275 INFO:tasks.workunit.client.1.smithi067.stdout:0/497: dread dd/d12/f16 [0,4194304] 0 2024-04-12T05:37:23.276 INFO:tasks.workunit.client.0.smithi032.stdout:9/932: dwrite d9/d1f/f5d [0,4194304] 0 2024-04-12T05:37:23.278 INFO:tasks.workunit.client.1.smithi067.stdout:7/443: truncate d0/d23/d45/f1a 1486459 0 2024-04-12T05:37:23.280 INFO:tasks.workunit.client.1.smithi067.stdout:8/375: rename d6/l10 to d6/d8/d50/l73 0 2024-04-12T05:37:23.281 INFO:tasks.workunit.client.1.smithi067.stdout:6/502: symlink dd/d21/d64/d6f/d90/lb0 0 2024-04-12T05:37:23.282 INFO:tasks.workunit.client.0.smithi032.stdout:9/933: mkdir d9/d15/d2c/d41/d145 0 2024-04-12T05:37:23.283 INFO:tasks.workunit.client.1.smithi067.stdout:5/655: rmdir d0/da/d3f/d82 39 2024-04-12T05:37:23.284 INFO:tasks.workunit.client.1.smithi067.stdout:8/376: write d6/f27 [5686296,109768] 0 2024-04-12T05:37:23.286 INFO:tasks.workunit.client.1.smithi067.stdout:0/498: rmdir dd/d6f/d72 39 2024-04-12T05:37:23.286 INFO:tasks.workunit.client.0.smithi032.stdout:9/934: mkdir d9/d15/d1a/d24/d27/d88/dd9/d146 0 2024-04-12T05:37:23.288 INFO:tasks.workunit.client.0.smithi032.stdout:1/977: rename d5/de/d17/d35/d3e/l45 to d5/de/d17/d35/d3b/l13b 0 2024-04-12T05:37:23.288 INFO:tasks.workunit.client.0.smithi032.stdout:9/935: chown d9/d15/d1a/d24/d27/l7f 400591269 1 2024-04-12T05:37:23.291 INFO:tasks.workunit.client.1.smithi067.stdout:9/380: sync 2024-04-12T05:37:23.291 INFO:tasks.workunit.client.1.smithi067.stdout:4/290: sync 2024-04-12T05:37:23.292 INFO:tasks.workunit.client.1.smithi067.stdout:7/444: link d0/d2/d5/d1d/f3e d0/d2/d5/d1e/f92 0 2024-04-12T05:37:23.292 INFO:tasks.workunit.client.0.smithi032.stdout:1/978: truncate d5/de/d5b/f95 581029 0 2024-04-12T05:37:23.294 INFO:tasks.workunit.client.0.smithi032.stdout:9/936: getdents d9/d15/d1a/d1d/d5a/d135 0 2024-04-12T05:37:23.294 INFO:tasks.workunit.client.0.smithi032.stdout:9/937: stat d9/d32/lfa 0 2024-04-12T05:37:23.295 INFO:tasks.workunit.client.1.smithi067.stdout:0/499: symlink dd/d12/d14/d49/l9d 0 2024-04-12T05:37:23.295 INFO:tasks.workunit.client.1.smithi067.stdout:5/656: mknod d0/da/cde 0 2024-04-12T05:37:23.296 INFO:tasks.workunit.client.1.smithi067.stdout:8/377: symlink d6/d8/l74 0 2024-04-12T05:37:23.298 INFO:tasks.workunit.client.1.smithi067.stdout:9/381: mkdir d1/d12/d76 0 2024-04-12T05:37:23.299 INFO:tasks.workunit.client.0.smithi032.stdout:1/979: rename d5/de/d17/d35/d61/d77/d8f/d9c/d10d/f137 to d5/de/d17/d35/d61/d105/d108/f13c 0 2024-04-12T05:37:23.300 INFO:tasks.workunit.client.1.smithi067.stdout:5/657: read - d0/da/d41/d7d/fa9 zero size 2024-04-12T05:37:23.301 INFO:tasks.workunit.client.0.smithi032.stdout:9/938: getdents d9/d15/d1a/d24/d27/dce 0 2024-04-12T05:37:23.303 INFO:tasks.workunit.client.0.smithi032.stdout:9/939: creat d9/d15/d1a/d24/df6/f147 x:0 0 0 2024-04-12T05:37:23.304 INFO:tasks.workunit.client.1.smithi067.stdout:9/382: mknod d1/d12/c77 0 2024-04-12T05:37:23.304 INFO:tasks.workunit.client.1.smithi067.stdout:5/658: chown d0/f1c 11448 1 2024-04-12T05:37:23.304 INFO:tasks.workunit.client.1.smithi067.stdout:0/500: creat dd/d12/d14/f9e x:0 0 0 2024-04-12T05:37:23.305 INFO:tasks.workunit.client.0.smithi032.stdout:1/980: dread d5/de/d17/d35/d78/f7c [0,4194304] 0 2024-04-12T05:37:23.305 INFO:tasks.workunit.client.0.smithi032.stdout:9/940: creat d9/d15/d1a/d1d/f148 x:0 0 0 2024-04-12T05:37:23.307 INFO:tasks.workunit.client.1.smithi067.stdout:7/445: dwrite d0/d23/d45/f1a [0,4194304] 0 2024-04-12T05:37:23.307 INFO:tasks.workunit.client.1.smithi067.stdout:4/291: getdents d8/d9/d3d/d48/d3c 0 2024-04-12T05:37:23.307 INFO:tasks.workunit.client.0.smithi032.stdout:9/941: dread - d9/d15/d1a/d24/d27/d88/dc1/f110 zero size 2024-04-12T05:37:23.308 INFO:tasks.workunit.client.1.smithi067.stdout:6/503: dread f0 [4194304,4194304] 0 2024-04-12T05:37:23.309 INFO:tasks.workunit.client.1.smithi067.stdout:9/383: write d1/fd [2669350,102395] 0 2024-04-12T05:37:23.311 INFO:tasks.workunit.client.1.smithi067.stdout:1/420: sync 2024-04-12T05:37:23.311 INFO:tasks.workunit.client.1.smithi067.stdout:2/488: sync 2024-04-12T05:37:23.312 INFO:tasks.workunit.client.1.smithi067.stdout:5/659: dread d0/d6/f10 [0,4194304] 0 2024-04-12T05:37:23.313 INFO:tasks.workunit.client.1.smithi067.stdout:7/446: symlink d0/d2/dd/d42/d82/l93 0 2024-04-12T05:37:23.314 INFO:tasks.workunit.client.1.smithi067.stdout:1/421: chown d4/d44/l5c 29 1 2024-04-12T05:37:23.315 INFO:tasks.workunit.client.1.smithi067.stdout:0/501: write dd/d12/d14/d1a/f93 [369800,84307] 0 2024-04-12T05:37:23.317 INFO:tasks.workunit.client.1.smithi067.stdout:9/384: read - d1/d5f/f65 zero size 2024-04-12T05:37:23.317 INFO:tasks.workunit.client.1.smithi067.stdout:7/447: chown d0/d2/dd/d42/d82 5370 1 2024-04-12T05:37:23.318 INFO:tasks.workunit.client.1.smithi067.stdout:3/356: dwrite dc/d1d/f43 [0,4194304] 0 2024-04-12T05:37:23.321 INFO:tasks.workunit.client.1.smithi067.stdout:8/378: truncate d6/d8/d18/f35 4700464 0 2024-04-12T05:37:23.325 INFO:tasks.workunit.client.1.smithi067.stdout:1/422: creat d4/d9/d12/d2c/f88 x:0 0 0 2024-04-12T05:37:23.325 INFO:tasks.workunit.client.1.smithi067.stdout:2/489: readlink d5/da/de/d14/d34/d40/d57/l81 0 2024-04-12T05:37:23.326 INFO:tasks.workunit.client.1.smithi067.stdout:5/660: unlink d0/d6/d20/cd2 0 2024-04-12T05:37:23.326 INFO:tasks.workunit.client.1.smithi067.stdout:9/385: rmdir d1/d28/d21 39 2024-04-12T05:37:23.332 INFO:tasks.workunit.client.0.smithi032.stdout:1/981: dwrite d5/de/f28 [0,4194304] 0 2024-04-12T05:37:23.333 INFO:tasks.workunit.client.0.smithi032.stdout:9/942: dwrite d9/d15/d2c/d41/f59 [0,4194304] 0 2024-04-12T05:37:23.335 INFO:tasks.workunit.client.1.smithi067.stdout:6/504: symlink dd/d11/d27/d4a/d8b/lb1 0 2024-04-12T05:37:23.336 INFO:tasks.workunit.client.0.smithi032.stdout:1/982: dread - d5/de/d17/d35/d3b/f138 zero size 2024-04-12T05:37:23.338 INFO:tasks.workunit.client.1.smithi067.stdout:4/292: dwrite d8/da/d1d/d15/d1c/f52 [0,4194304] 0 2024-04-12T05:37:23.339 INFO:tasks.workunit.client.1.smithi067.stdout:8/379: creat d6/d8/d18/d4e/f75 x:0 0 0 2024-04-12T05:37:23.342 INFO:tasks.workunit.client.1.smithi067.stdout:7/448: creat d0/d2/dd/d42/d7e/f94 x:0 0 0 2024-04-12T05:37:23.344 INFO:tasks.workunit.client.1.smithi067.stdout:0/502: write dd/d5b/f68 [67767,96316] 0 2024-04-12T05:37:23.348 INFO:tasks.workunit.client.0.smithi032.stdout:9/943: rename d9/d1f to d9/d15/d1a/d1d/d5a/d149 0 2024-04-12T05:37:23.350 INFO:tasks.workunit.client.1.smithi067.stdout:5/661: rmdir d0/da/d3f/d82 39 2024-04-12T05:37:23.354 INFO:tasks.workunit.client.1.smithi067.stdout:7/449: mknod d0/d2/dd/c95 0 2024-04-12T05:37:23.355 INFO:tasks.workunit.client.1.smithi067.stdout:0/503: fdatasync dd/d12/d14/f43 0 2024-04-12T05:37:23.357 INFO:tasks.workunit.client.1.smithi067.stdout:3/357: dwrite dc/d1d/d28/d3e/f6d [0,4194304] 0 2024-04-12T05:37:23.358 INFO:tasks.workunit.client.0.smithi032.stdout:1/983: dread d5/f30 [0,4194304] 0 2024-04-12T05:37:23.360 INFO:tasks.workunit.client.0.smithi032.stdout:1/984: dread d5/de/d17/d35/d61/d77/f7d [0,4194304] 0 2024-04-12T05:37:23.365 INFO:tasks.workunit.client.1.smithi067.stdout:1/423: dwrite d4/d9/d12/d59/f64 [0,4194304] 0 2024-04-12T05:37:23.365 INFO:tasks.workunit.client.1.smithi067.stdout:7/450: mkdir d0/d2/dd/d42/d82/d96 0 2024-04-12T05:37:23.366 INFO:tasks.workunit.client.1.smithi067.stdout:3/358: mknod dc/df/c75 0 2024-04-12T05:37:23.368 INFO:tasks.workunit.client.1.smithi067.stdout:2/490: dwrite d5/d23/d93/f98 [0,4194304] 0 2024-04-12T05:37:23.370 INFO:tasks.workunit.client.0.smithi032.stdout:1/985: rename d5/de/d17/d35/d61/d105 to d5/de/d17/d35/d3e/d13d 0 2024-04-12T05:37:23.377 INFO:tasks.workunit.client.1.smithi067.stdout:0/504: read dd/d6f/d72/f97 [1926620,122741] 0 2024-04-12T05:37:23.377 INFO:tasks.workunit.client.0.smithi032.stdout:1/986: fdatasync d5/de/d17/fae 0 2024-04-12T05:37:23.378 INFO:tasks.workunit.client.0.smithi032.stdout:9/944: getdents d9/d15/d1a/d1d/d5a 0 2024-04-12T05:37:23.380 INFO:tasks.workunit.client.0.smithi032.stdout:1/987: write d5/de/d17/d93/ffc [655651,9944] 0 2024-04-12T05:37:23.381 INFO:tasks.workunit.client.1.smithi067.stdout:1/424: creat d4/d9/d4c/f89 x:0 0 0 2024-04-12T05:37:23.384 INFO:tasks.workunit.client.0.smithi032.stdout:9/945: creat d9/d15/d1a/d24/f14a x:0 0 0 2024-04-12T05:37:23.384 INFO:tasks.workunit.client.0.smithi032.stdout:1/988: creat d5/de/d17/d35/d3e/db4/f13e x:0 0 0 2024-04-12T05:37:23.384 INFO:tasks.workunit.client.1.smithi067.stdout:9/386: dread d1/d12/d15/f5c [0,4194304] 0 2024-04-12T05:37:23.386 INFO:tasks.workunit.client.1.smithi067.stdout:1/425: readlink d4/d9/d14/l43 0 2024-04-12T05:37:23.387 INFO:tasks.workunit.client.0.smithi032.stdout:9/946: chown d9/d15/d1a/d1d/d5a/d149/f46 1018403 1 2024-04-12T05:37:23.387 INFO:tasks.workunit.client.1.smithi067.stdout:6/505: dwrite dd/f1d [0,4194304] 0 2024-04-12T05:37:23.393 INFO:tasks.workunit.client.0.smithi032.stdout:9/947: creat d9/d15/d2c/f14b x:0 0 0 2024-04-12T05:37:23.395 INFO:tasks.workunit.client.0.smithi032.stdout:1/989: truncate d5/de/d17/d35/d3b/d49/fd9 1897735 0 2024-04-12T05:37:23.397 INFO:tasks.workunit.client.1.smithi067.stdout:2/491: getdents d5/d6a 0 2024-04-12T05:37:23.397 INFO:tasks.workunit.client.1.smithi067.stdout:7/451: dread d0/d2/d5/fe [0,4194304] 0 2024-04-12T05:37:23.397 INFO:tasks.workunit.client.0.smithi032.stdout:9/948: rename d9/d32/c54 to d9/d32/d6e/d114/c14c 0 2024-04-12T05:37:23.398 INFO:tasks.workunit.client.1.smithi067.stdout:5/662: link d0/d6/d20/d3e/c66 d0/d6/d20/d62/cdf 0 2024-04-12T05:37:23.401 INFO:tasks.workunit.client.1.smithi067.stdout:0/505: dwrite dd/d12/d14/f61 [4194304,4194304] 0 2024-04-12T05:37:23.401 INFO:tasks.workunit.client.0.smithi032.stdout:1/990: mkdir d5/de/d17/d35/d3b/d13f 0 2024-04-12T05:37:23.401 INFO:tasks.workunit.client.0.smithi032.stdout:9/949: creat d9/d15/d1a/dd2/d65/d80/d87/f14d x:0 0 0 2024-04-12T05:37:23.404 INFO:tasks.workunit.client.0.smithi032.stdout:9/950: rmdir d9/d32/d86/dfe 39 2024-04-12T05:37:23.405 INFO:tasks.workunit.client.1.smithi067.stdout:9/387: rmdir d1/d28/d21 39 2024-04-12T05:37:23.406 INFO:tasks.workunit.client.1.smithi067.stdout:5/663: creat d0/d6/d40/d4b/fe0 x:0 0 0 2024-04-12T05:37:23.407 INFO:tasks.workunit.client.1.smithi067.stdout:6/506: dwrite dd/d11/f6b [0,4194304] 0 2024-04-12T05:37:23.408 INFO:tasks.workunit.client.1.smithi067.stdout:2/492: write d5/da/de/d14/d34/d40/d57/f60 [1679026,64621] 0 2024-04-12T05:37:23.408 INFO:tasks.workunit.client.0.smithi032.stdout:1/991: link d5/de/d17/d35/d61/d77/d8f/da0/caa d5/de/d17/d35/d3e/d13d/c140 0 2024-04-12T05:37:23.411 INFO:tasks.workunit.client.0.smithi032.stdout:1/992: stat d5/de/d17/d35/d73/c139 0 2024-04-12T05:37:23.411 INFO:tasks.workunit.client.0.smithi032.stdout:1/993: readlink d5/d2c/d4f/l6a 0 2024-04-12T05:37:23.412 INFO:tasks.workunit.client.1.smithi067.stdout:9/388: creat d1/d12/d76/f78 x:0 0 0 2024-04-12T05:37:23.413 INFO:tasks.workunit.client.1.smithi067.stdout:6/507: rmdir dd/d11/d27/d4a/d51 39 2024-04-12T05:37:23.416 INFO:tasks.workunit.client.0.smithi032.stdout:1/994: write d5/de/d17/d35/ff8 [4687891,54900] 0 2024-04-12T05:37:23.418 INFO:tasks.workunit.client.1.smithi067.stdout:5/664: creat d0/da/d41/fe1 x:0 0 0 2024-04-12T05:37:23.418 INFO:tasks.workunit.client.1.smithi067.stdout:7/452: unlink d0/d2/d5/d1d/d43/f69 0 2024-04-12T05:37:23.422 INFO:tasks.workunit.client.1.smithi067.stdout:6/508: symlink dd/d21/d66/lb2 0 2024-04-12T05:37:23.422 INFO:tasks.workunit.client.1.smithi067.stdout:1/426: dwrite d4/d9/d12/d28/f83 [0,4194304] 0 2024-04-12T05:37:23.423 INFO:tasks.workunit.client.0.smithi032.stdout:1/995: dread d5/de/d17/d35/d73/f8a [0,4194304] 0 2024-04-12T05:37:23.424 INFO:tasks.workunit.client.1.smithi067.stdout:8/380: dread d6/d8/d18/df/f23 [0,4194304] 0 2024-04-12T05:37:23.429 INFO:tasks.workunit.client.1.smithi067.stdout:7/453: write d0/d2/d5/d4a/f86 [722264,98365] 0 2024-04-12T05:37:23.430 INFO:tasks.workunit.client.1.smithi067.stdout:9/389: creat d1/f79 x:0 0 0 2024-04-12T05:37:23.432 INFO:tasks.workunit.client.1.smithi067.stdout:0/506: write dd/d12/f2a [299656,86322] 0 2024-04-12T05:37:23.433 INFO:tasks.workunit.client.1.smithi067.stdout:6/509: creat dd/d11/d27/d4a/d79/fb3 x:0 0 0 2024-04-12T05:37:23.434 INFO:tasks.workunit.client.1.smithi067.stdout:2/493: dwrite d5/da/de/f69 [0,4194304] 0 2024-04-12T05:37:23.443 INFO:tasks.workunit.client.1.smithi067.stdout:9/390: truncate d1/f7 2489810 0 2024-04-12T05:37:23.446 INFO:tasks.workunit.client.1.smithi067.stdout:2/494: creat d5/d11/f9b x:0 0 0 2024-04-12T05:37:23.446 INFO:tasks.workunit.client.1.smithi067.stdout:7/454: fsync d0/d2/dd/d42/d7e/f30 0 2024-04-12T05:37:23.447 INFO:tasks.workunit.client.1.smithi067.stdout:1/427: rename d4/d9/d3a/f3c to d4/d9/d14/f8a 0 2024-04-12T05:37:23.448 INFO:tasks.workunit.client.1.smithi067.stdout:0/507: mknod dd/d12/d14/c9f 0 2024-04-12T05:37:23.449 INFO:tasks.workunit.client.1.smithi067.stdout:9/391: mkdir d1/d66/d7a 0 2024-04-12T05:37:23.451 INFO:tasks.workunit.client.1.smithi067.stdout:7/455: fsync d0/d2/dd/d79/f63 0 2024-04-12T05:37:23.455 INFO:tasks.workunit.client.1.smithi067.stdout:9/392: stat d1/d28/d44/f73 0 2024-04-12T05:37:23.455 INFO:tasks.workunit.client.1.smithi067.stdout:7/456: fsync d0/d23/f27 0 2024-04-12T05:37:23.460 INFO:tasks.workunit.client.0.smithi032.stdout:1/996: dread d5/de/d17/d35/d3b/dda/de0/ff4 [0,4194304] 0 2024-04-12T05:37:23.461 INFO:tasks.workunit.client.1.smithi067.stdout:0/508: rename l4 to dd/d9c/d5f/d78/d83/la0 0 2024-04-12T05:37:23.462 INFO:tasks.workunit.client.1.smithi067.stdout:5/665: dwrite d0/da/f5a [0,4194304] 0 2024-04-12T05:37:23.463 INFO:tasks.workunit.client.1.smithi067.stdout:8/381: dread d6/d8/d18/d36/f44 [0,4194304] 0 2024-04-12T05:37:23.464 INFO:tasks.workunit.client.1.smithi067.stdout:2/495: creat d5/f9c x:0 0 0 2024-04-12T05:37:23.465 INFO:tasks.workunit.client.1.smithi067.stdout:6/510: dwrite dd/d21/d64/d88/d7b/faa [0,4194304] 0 2024-04-12T05:37:23.465 INFO:tasks.workunit.client.1.smithi067.stdout:7/457: unlink d0/d2/d5/d4a/f86 0 2024-04-12T05:37:23.470 INFO:tasks.workunit.client.1.smithi067.stdout:5/666: write d0/d6/d20/d3e/fbc [132597,122281] 0 2024-04-12T05:37:23.470 INFO:tasks.workunit.client.1.smithi067.stdout:1/428: link d4/d9/d12/d59/f87 d4/d9/d3a/f8b 0 2024-04-12T05:37:23.471 INFO:tasks.workunit.client.1.smithi067.stdout:0/509: chown dd/d12/d14/d51/f91 5048635 1 2024-04-12T05:37:23.473 INFO:tasks.workunit.client.1.smithi067.stdout:2/496: creat d5/da/de/d14/d34/d4c/f9d x:0 0 0 2024-04-12T05:37:23.474 INFO:tasks.workunit.client.1.smithi067.stdout:7/458: rmdir d0/d2/d5/d1d/d43 39 2024-04-12T05:37:23.475 INFO:tasks.workunit.client.1.smithi067.stdout:8/382: write d6/d8/d18/d63/d67/d69/f5a [74185,106480] 0 2024-04-12T05:37:23.478 INFO:tasks.workunit.client.1.smithi067.stdout:9/393: dwrite d1/d28/d44/f73 [0,4194304] 0 2024-04-12T05:37:23.478 INFO:tasks.workunit.client.1.smithi067.stdout:0/510: unlink dd/d12/d14/d51/f5a 0 2024-04-12T05:37:23.481 INFO:tasks.workunit.client.1.smithi067.stdout:7/459: fdatasync d0/d2/d5/f50 0 2024-04-12T05:37:23.481 INFO:tasks.workunit.client.1.smithi067.stdout:8/383: chown d6/d8/d18/d3c/d4f/f65 14 1 2024-04-12T05:37:23.481 INFO:tasks.workunit.client.1.smithi067.stdout:1/429: creat d4/d9/d12/d6f/f8c x:0 0 0 2024-04-12T05:37:23.483 INFO:tasks.workunit.client.1.smithi067.stdout:6/511: dread dd/d11/d27/f54 [0,4194304] 0 2024-04-12T05:37:23.487 INFO:tasks.workunit.client.1.smithi067.stdout:5/667: creat d0/d6/d33/d88/fe2 x:0 0 0 2024-04-12T05:37:23.489 INFO:tasks.workunit.client.1.smithi067.stdout:1/430: creat d4/d9/d12/d2c/f8d x:0 0 0 2024-04-12T05:37:23.494 INFO:tasks.workunit.client.1.smithi067.stdout:7/460: creat d0/d23/f97 x:0 0 0 2024-04-12T05:37:23.495 INFO:tasks.workunit.client.1.smithi067.stdout:6/512: rename dd/d21/d23/l47 to dd/d21/d64/d6f/d90/d98/lb4 0 2024-04-12T05:37:23.497 INFO:tasks.workunit.client.1.smithi067.stdout:1/431: write d4/d9/d14/f8a [1021673,5750] 0 2024-04-12T05:37:23.497 INFO:tasks.workunit.client.1.smithi067.stdout:9/394: dwrite d1/d12/d15/f1d [4194304,4194304] 0 2024-04-12T05:37:23.500 INFO:tasks.workunit.client.1.smithi067.stdout:7/461: write d0/d2/d5/d1e/f7c [218948,59487] 0 2024-04-12T05:37:23.501 INFO:tasks.workunit.client.1.smithi067.stdout:0/511: truncate dd/d9c/f50 1887846 0 2024-04-12T05:37:23.504 INFO:tasks.workunit.client.1.smithi067.stdout:8/384: dwrite d6/d8/d18/df/d12/f29 [0,4194304] 0 2024-04-12T05:37:23.511 INFO:tasks.workunit.client.1.smithi067.stdout:5/668: rename d0/da/d41/d7d/fa9 to d0/d3a/d2c/dc6/ddd/fe3 0 2024-04-12T05:37:23.512 INFO:tasks.workunit.client.1.smithi067.stdout:2/497: dread d5/f12 [0,4194304] 0 2024-04-12T05:37:23.514 INFO:tasks.workunit.client.1.smithi067.stdout:7/462: symlink d0/d67/d57/d85/l98 0 2024-04-12T05:37:23.515 INFO:tasks.workunit.client.1.smithi067.stdout:6/513: dwrite dd/d21/d64/d6f/f70 [0,4194304] 0 2024-04-12T05:37:23.518 INFO:tasks.workunit.client.1.smithi067.stdout:1/432: dwrite d4/d9/d12/d2c/f8d [0,4194304] 0 2024-04-12T05:37:23.521 INFO:tasks.workunit.client.1.smithi067.stdout:2/498: symlink d5/d6a/d78/l9e 0 2024-04-12T05:37:23.521 INFO:tasks.workunit.client.1.smithi067.stdout:8/385: symlink d6/d8/d18/l76 0 2024-04-12T05:37:23.523 INFO:tasks.workunit.client.1.smithi067.stdout:9/395: link d1/d13/l2f d1/d5f/l7b 0 2024-04-12T05:37:23.524 INFO:tasks.workunit.client.1.smithi067.stdout:1/433: fdatasync d4/d9/d12/d2c/f6c 0 2024-04-12T05:37:23.531 INFO:tasks.workunit.client.1.smithi067.stdout:2/499: creat d5/da/d3d/f9f x:0 0 0 2024-04-12T05:37:23.535 INFO:tasks.workunit.client.1.smithi067.stdout:5/669: truncate d0/d3a/d2c/f3d 738919 0 2024-04-12T05:37:23.536 INFO:tasks.workunit.client.1.smithi067.stdout:7/463: truncate d0/d2/d5/d1e/f4f 868928 0 2024-04-12T05:37:23.540 INFO:tasks.workunit.client.1.smithi067.stdout:6/514: dwrite dd/d11/d27/d4a/d79/fb3 [0,4194304] 0 2024-04-12T05:37:23.543 INFO:tasks.workunit.client.1.smithi067.stdout:0/512: dwrite dd/d12/d14/d49/f4d [0,4194304] 0 2024-04-12T05:37:23.544 INFO:tasks.workunit.client.1.smithi067.stdout:1/434: mknod d4/d3d/d7b/c8e 0 2024-04-12T05:37:23.546 INFO:tasks.workunit.client.1.smithi067.stdout:9/396: link d1/d28/d23/f3d d1/d13/f7c 0 2024-04-12T05:37:23.549 INFO:tasks.workunit.client.1.smithi067.stdout:6/515: unlink dd/d21/d3f/f55 0 2024-04-12T05:37:23.550 INFO:tasks.workunit.client.1.smithi067.stdout:8/386: dwrite d6/d8/d18/df/d12/f55 [0,4194304] 0 2024-04-12T05:37:23.551 INFO:tasks.workunit.client.1.smithi067.stdout:2/500: link d5/da/de/d39/f6c d5/fa0 0 2024-04-12T05:37:23.554 INFO:tasks.workunit.client.1.smithi067.stdout:2/501: readlink d5/d23/d3a/l70 0 2024-04-12T05:37:23.559 INFO:tasks.workunit.client.1.smithi067.stdout:6/516: mkdir dd/d21/d64/db5 0 2024-04-12T05:37:23.559 INFO:tasks.workunit.client.1.smithi067.stdout:1/435: mkdir d4/d8f 0 2024-04-12T05:37:23.559 INFO:tasks.workunit.client.1.smithi067.stdout:8/387: write d6/d8/d18/f33 [2019692,5241] 0 2024-04-12T05:37:23.560 INFO:tasks.workunit.client.1.smithi067.stdout:9/397: creat d1/d28/d21/d45/d75/f7d x:0 0 0 2024-04-12T05:37:23.561 INFO:tasks.workunit.client.1.smithi067.stdout:7/464: dread d0/d2/f74 [0,4194304] 0 2024-04-12T05:37:23.563 INFO:tasks.workunit.client.1.smithi067.stdout:0/513: dwrite dd/d9c/d5f/f90 [0,4194304] 0 2024-04-12T05:37:23.563 INFO:tasks.workunit.client.1.smithi067.stdout:8/388: dread - d6/d8/d18/d36/f5c zero size 2024-04-12T05:37:23.564 INFO:tasks.workunit.client.1.smithi067.stdout:9/398: chown d1/d28/d21/d3f/f41 61389242 1 2024-04-12T05:37:23.564 INFO:tasks.workunit.client.1.smithi067.stdout:1/436: mkdir d4/d9/d3a/d51/d55/d60/d90 0 2024-04-12T05:37:23.564 INFO:tasks.workunit.client.1.smithi067.stdout:0/514: dread - dd/d12/d14/f9e zero size 2024-04-12T05:37:23.568 INFO:tasks.workunit.client.1.smithi067.stdout:1/437: write d4/d9/d12/f81 [468884,101470] 0 2024-04-12T05:37:23.569 INFO:tasks.workunit.client.1.smithi067.stdout:7/465: rename d0/d2/d5/f11 to d0/d23/d80/f99 0 2024-04-12T05:37:23.570 INFO:tasks.workunit.client.1.smithi067.stdout:8/389: read d6/d8/d18/f3f [3875466,58275] 0 2024-04-12T05:37:23.571 INFO:tasks.workunit.client.1.smithi067.stdout:8/390: fsync d6/d8/d18/d63/d67/d69/f71 0 2024-04-12T05:37:23.574 INFO:tasks.workunit.client.1.smithi067.stdout:9/399: unlink d1/d12/d76/f78 0 2024-04-12T05:37:23.575 INFO:tasks.workunit.client.1.smithi067.stdout:1/438: dread d4/d9/f22 [0,4194304] 0 2024-04-12T05:37:23.578 INFO:tasks.workunit.client.1.smithi067.stdout:2/502: write d5/da/de/f26 [853792,3245] 0 2024-04-12T05:37:23.579 INFO:tasks.workunit.client.1.smithi067.stdout:7/466: symlink d0/d2/dd/d79/l9a 0 2024-04-12T05:37:23.580 INFO:tasks.workunit.client.1.smithi067.stdout:8/391: dwrite d6/d8/d18/f62 [4194304,4194304] 0 2024-04-12T05:37:23.582 INFO:tasks.workunit.client.1.smithi067.stdout:9/400: chown d1/d13/l59 124744 1 2024-04-12T05:37:23.582 INFO:tasks.workunit.client.1.smithi067.stdout:9/401: stat d1/d12/d15/f1c 0 2024-04-12T05:37:23.583 INFO:tasks.workunit.client.1.smithi067.stdout:9/402: chown d1/d28/d21/d68 625635250 1 2024-04-12T05:37:23.584 INFO:tasks.workunit.client.1.smithi067.stdout:9/403: read d1/d28/d23/f24 [4294074,110138] 0 2024-04-12T05:37:23.585 INFO:tasks.workunit.client.1.smithi067.stdout:9/404: readlink d1/d13/l37 0 2024-04-12T05:37:23.586 INFO:tasks.workunit.client.1.smithi067.stdout:1/439: creat d4/d9/d3a/d51/d55/d60/f91 x:0 0 0 2024-04-12T05:37:23.587 INFO:tasks.workunit.client.1.smithi067.stdout:1/440: write d4/d9/d4c/f89 [939819,71320] 0 2024-04-12T05:37:23.587 INFO:tasks.workunit.client.1.smithi067.stdout:2/503: symlink d5/d43/d6e/la1 0 2024-04-12T05:37:23.591 INFO:tasks.workunit.client.1.smithi067.stdout:8/392: symlink d6/d8/d18/d63/l77 0 2024-04-12T05:37:23.595 INFO:tasks.workunit.client.1.smithi067.stdout:0/515: rename dd/d9c/l4a to dd/d92/la1 0 2024-04-12T05:37:23.596 INFO:tasks.workunit.client.1.smithi067.stdout:8/393: chown d6/d8/d50/l73 48221 1 2024-04-12T05:37:23.596 INFO:tasks.workunit.client.1.smithi067.stdout:1/441: mknod d4/d9/d14/c92 0 2024-04-12T05:37:23.597 INFO:tasks.workunit.client.1.smithi067.stdout:1/442: chown d4/d9/d12/d2c/c62 15 1 2024-04-12T05:37:23.598 INFO:tasks.workunit.client.1.smithi067.stdout:1/443: chown d4/d9/d3a/d51/f82 511357 1 2024-04-12T05:37:23.599 INFO:tasks.workunit.client.1.smithi067.stdout:0/516: mkdir dd/d12/d14/d51/da2 0 2024-04-12T05:37:23.599 INFO:tasks.workunit.client.1.smithi067.stdout:1/444: write d4/d9/d12/d2c/f6c [2121464,86892] 0 2024-04-12T05:37:23.602 INFO:tasks.workunit.client.1.smithi067.stdout:9/405: rename d1/d28/d27/c5d to d1/d66/c7e 0 2024-04-12T05:37:23.603 INFO:tasks.workunit.client.1.smithi067.stdout:8/394: creat d6/d8/d6e/f78 x:0 0 0 2024-04-12T05:37:23.604 INFO:tasks.workunit.client.1.smithi067.stdout:0/517: creat dd/d6f/d72/fa3 x:0 0 0 2024-04-12T05:37:23.609 INFO:tasks.workunit.client.1.smithi067.stdout:7/467: dwrite d0/d2/d5/d1e/f28 [0,4194304] 0 2024-04-12T05:37:23.612 INFO:tasks.workunit.client.1.smithi067.stdout:0/518: symlink dd/d9c/d5f/d78/la4 0 2024-04-12T05:37:23.617 INFO:tasks.workunit.client.1.smithi067.stdout:2/504: dwrite d5/d23/d3a/f6f [0,4194304] 0 2024-04-12T05:37:23.617 INFO:tasks.workunit.client.1.smithi067.stdout:9/406: truncate d1/d28/d23/f3d 3127369 0 2024-04-12T05:37:23.617 INFO:tasks.workunit.client.1.smithi067.stdout:1/445: dwrite d4/d9/d4c/f89 [0,4194304] 0 2024-04-12T05:37:23.620 INFO:tasks.workunit.client.1.smithi067.stdout:8/395: rename d6/d8/d18/df/f47 to d6/d8/d18/d63/d67/d69/f79 0 2024-04-12T05:37:23.622 INFO:tasks.workunit.client.1.smithi067.stdout:1/446: write d4/d9/d12/f7d [1299693,72388] 0 2024-04-12T05:37:23.624 INFO:tasks.workunit.client.1.smithi067.stdout:2/505: mkdir d5/da/de/d14/d34/d40/d57/da2 0 2024-04-12T05:37:23.628 INFO:tasks.workunit.client.1.smithi067.stdout:8/396: rename d6/d8/l74 to d6/d8/d18/d4e/l7a 0 2024-04-12T05:37:23.629 INFO:tasks.workunit.client.1.smithi067.stdout:9/407: dwrite d1/d28/d27/f71 [0,4194304] 0 2024-04-12T05:37:23.632 INFO:tasks.workunit.client.1.smithi067.stdout:0/519: getdents dd/d6f/d72 0 2024-04-12T05:37:23.637 INFO:tasks.workunit.client.1.smithi067.stdout:2/506: creat d5/da/de/d14/d34/fa3 x:0 0 0 2024-04-12T05:37:23.637 INFO:tasks.workunit.client.1.smithi067.stdout:7/468: getdents d0/d2/dd/d42/d71 0 2024-04-12T05:37:23.644 INFO:tasks.workunit.client.1.smithi067.stdout:7/469: dwrite d0/d67/f56 [0,4194304] 0 2024-04-12T05:37:23.645 INFO:tasks.workunit.client.1.smithi067.stdout:2/507: mkdir d5/d43/da4 0 2024-04-12T05:37:23.647 INFO:tasks.workunit.client.1.smithi067.stdout:9/408: unlink d1/d5f/l7b 0 2024-04-12T05:37:23.648 INFO:tasks.workunit.client.1.smithi067.stdout:0/520: creat dd/d12/d14/d49/d9b/fa5 x:0 0 0 2024-04-12T05:37:23.651 INFO:tasks.workunit.client.1.smithi067.stdout:7/470: creat d0/d2/d5/d1e/f9b x:0 0 0 2024-04-12T05:37:23.651 INFO:tasks.workunit.client.1.smithi067.stdout:1/447: truncate d4/d9/f45 2397697 0 2024-04-12T05:37:23.653 INFO:tasks.workunit.client.1.smithi067.stdout:8/397: getdents d6/d8/d18/d63/d67/d69/d5d 0 2024-04-12T05:37:23.657 INFO:tasks.workunit.client.1.smithi067.stdout:7/471: creat d0/d23/d80/f9c x:0 0 0 2024-04-12T05:37:23.657 INFO:tasks.workunit.client.1.smithi067.stdout:2/508: dwrite d5/d11/f2a [0,4194304] 0 2024-04-12T05:37:23.659 INFO:tasks.workunit.client.1.smithi067.stdout:1/448: creat d4/d9/d3a/d51/d55/f93 x:0 0 0 2024-04-12T05:37:23.663 INFO:tasks.workunit.client.1.smithi067.stdout:9/409: dread d1/d12/d15/f43 [4194304,4194304] 0 2024-04-12T05:37:23.664 INFO:tasks.workunit.client.1.smithi067.stdout:0/521: dwrite dd/d9c/d64/f7e [0,4194304] 0 2024-04-12T05:37:23.668 INFO:tasks.workunit.client.1.smithi067.stdout:2/509: mkdir d5/da/de/da5 0 2024-04-12T05:37:23.669 INFO:tasks.workunit.client.1.smithi067.stdout:1/449: write d4/d9/d12/f2e [1006975,36070] 0 2024-04-12T05:37:23.677 INFO:tasks.workunit.client.1.smithi067.stdout:1/450: creat d4/d3d/d7b/f94 x:0 0 0 2024-04-12T05:37:23.677 INFO:tasks.workunit.client.1.smithi067.stdout:8/398: link d6/d8/d18/d36/c61 d6/d8/d18/d63/d67/c7b 0 2024-04-12T05:37:23.678 INFO:tasks.workunit.client.1.smithi067.stdout:9/410: mkdir d1/d28/d21/d7f 0 2024-04-12T05:37:23.679 INFO:tasks.workunit.client.1.smithi067.stdout:8/399: dread - d6/d8/d18/d3c/d4f/f65 zero size 2024-04-12T05:37:23.680 INFO:tasks.workunit.client.1.smithi067.stdout:7/472: link d0/d23/c64 d0/d23/d45/c9d 0 2024-04-12T05:37:23.683 INFO:tasks.workunit.client.1.smithi067.stdout:1/451: rename d4/d9/f68 to d4/d3d/d7b/f95 0 2024-04-12T05:37:23.683 INFO:tasks.workunit.client.1.smithi067.stdout:9/411: fsync d1/d12/d15/f5c 0 2024-04-12T05:37:23.684 INFO:tasks.workunit.client.1.smithi067.stdout:9/412: write d1/d12/d15/f5c [4936106,86917] 0 2024-04-12T05:37:23.687 INFO:tasks.workunit.client.1.smithi067.stdout:8/400: unlink d6/d8/d18/df/f1b 0 2024-04-12T05:37:23.688 INFO:tasks.workunit.client.1.smithi067.stdout:7/473: rename d0/d2/d5/d1e/f7c to d0/d23/d45/f9e 0 2024-04-12T05:37:23.689 INFO:tasks.workunit.client.1.smithi067.stdout:9/413: dread d1/d12/d15/f1d [4194304,4194304] 0 2024-04-12T05:37:23.689 INFO:tasks.workunit.client.1.smithi067.stdout:7/474: write d0/f8 [392310,81877] 0 2024-04-12T05:37:23.691 INFO:tasks.workunit.client.1.smithi067.stdout:0/522: write dd/d12/f4e [1505244,92140] 0 2024-04-12T05:37:23.692 INFO:tasks.workunit.client.1.smithi067.stdout:2/510: truncate d5/d23/d3a/f6f 3835346 0 2024-04-12T05:37:23.693 INFO:tasks.workunit.client.1.smithi067.stdout:8/401: rmdir d6/d8/d18/d3c 39 2024-04-12T05:37:23.695 INFO:tasks.workunit.client.1.smithi067.stdout:0/523: chown dd/d12/d14/c9f 208439828 1 2024-04-12T05:37:23.698 INFO:tasks.workunit.client.1.smithi067.stdout:7/475: rename d0/d2/d5/d1d/l60 to d0/d2/dd/d42/d82/l9f 0 2024-04-12T05:37:23.700 INFO:tasks.workunit.client.1.smithi067.stdout:0/524: write dd/d12/d14/f9e [730206,105933] 0 2024-04-12T05:37:23.700 INFO:tasks.workunit.client.1.smithi067.stdout:8/402: unlink d6/d8/f16 0 2024-04-12T05:37:23.702 INFO:tasks.workunit.client.1.smithi067.stdout:1/452: dwrite d4/d52/d80/f6d [0,4194304] 0 2024-04-12T05:37:23.706 INFO:tasks.workunit.client.1.smithi067.stdout:7/476: mkdir d0/d23/d80/da0 0 2024-04-12T05:37:23.707 INFO:tasks.workunit.client.1.smithi067.stdout:9/414: dwrite d1/d28/d27/f71 [0,4194304] 0 2024-04-12T05:37:23.709 INFO:tasks.workunit.client.1.smithi067.stdout:2/511: dwrite d5/da/de/d14/d46/f5a [0,4194304] 0 2024-04-12T05:37:23.709 INFO:tasks.workunit.client.1.smithi067.stdout:8/403: dread - d6/d8/d18/d3c/d4f/f65 zero size 2024-04-12T05:37:23.714 INFO:tasks.workunit.client.1.smithi067.stdout:0/525: link dd/f13 dd/d12/d14/d49/fa6 0 2024-04-12T05:37:23.714 INFO:tasks.workunit.client.1.smithi067.stdout:2/512: dread d5/d11/f2a [0,4194304] 0 2024-04-12T05:37:23.716 INFO:tasks.workunit.client.1.smithi067.stdout:7/477: mkdir d0/d2/d5/da1 0 2024-04-12T05:37:23.716 INFO:tasks.workunit.client.1.smithi067.stdout:7/478: chown d0/d23/d80/da0 3072816 1 2024-04-12T05:37:23.717 INFO:tasks.workunit.client.1.smithi067.stdout:2/513: creat d5/d6a/fa6 x:0 0 0 2024-04-12T05:37:23.723 INFO:tasks.workunit.client.1.smithi067.stdout:7/479: rename d0/d2/d5/d1e to d0/d2/dd/da2 0 2024-04-12T05:37:23.727 INFO:tasks.workunit.client.1.smithi067.stdout:2/514: dread d5/f12 [0,4194304] 0 2024-04-12T05:37:23.727 INFO:tasks.workunit.client.1.smithi067.stdout:0/526: dwrite dd/d12/f2a [0,4194304] 0 2024-04-12T05:37:23.728 INFO:tasks.workunit.client.1.smithi067.stdout:0/527: stat dd/d12/d14/d49 0 2024-04-12T05:37:23.729 INFO:tasks.workunit.client.1.smithi067.stdout:7/480: mkdir d0/d2/dd/d42/d7e/d34/da3 0 2024-04-12T05:37:23.729 INFO:tasks.workunit.client.1.smithi067.stdout:9/415: dread d1/d12/d15/f43 [0,4194304] 0 2024-04-12T05:37:23.735 INFO:tasks.workunit.client.1.smithi067.stdout:7/481: mknod d0/d67/d57/d85/ca4 0 2024-04-12T05:37:23.737 INFO:tasks.workunit.client.1.smithi067.stdout:9/416: unlink d1/d66/c7e 0 2024-04-12T05:37:23.738 INFO:tasks.workunit.client.1.smithi067.stdout:2/515: dwrite d5/d11/f8a [0,4194304] 0 2024-04-12T05:37:23.740 INFO:tasks.workunit.client.1.smithi067.stdout:7/482: mknod d0/d23/ca5 0 2024-04-12T05:37:23.741 INFO:tasks.workunit.client.1.smithi067.stdout:2/516: readlink d5/da/de/d14/l4b 0 2024-04-12T05:37:23.742 INFO:tasks.workunit.client.1.smithi067.stdout:0/528: dwrite dd/d12/d14/d51/f5c [0,4194304] 0 2024-04-12T05:37:23.742 INFO:tasks.workunit.client.1.smithi067.stdout:9/417: truncate d1/d5f/f65 254212 0 2024-04-12T05:37:23.743 INFO:tasks.workunit.client.1.smithi067.stdout:9/418: readlink d1/d13/l59 0 2024-04-12T05:37:23.749 INFO:tasks.workunit.client.1.smithi067.stdout:9/419: write d1/d28/d23/f31 [429809,40536] 0 2024-04-12T05:37:23.756 INFO:tasks.workunit.client.1.smithi067.stdout:9/420: creat d1/d28/d21/d45/d75/f80 x:0 0 0 2024-04-12T05:37:23.758 INFO:tasks.workunit.client.1.smithi067.stdout:2/517: getdents d5/da/de/d14/d34 0 2024-04-12T05:37:23.758 INFO:tasks.workunit.client.1.smithi067.stdout:9/421: fsync d1/fd 0 2024-04-12T05:37:23.758 INFO:tasks.workunit.client.1.smithi067.stdout:0/529: link c8 dd/d12/d14/ca7 0 2024-04-12T05:37:23.761 INFO:tasks.workunit.client.1.smithi067.stdout:7/483: dwrite d0/d2/dd/da2/f4f [0,4194304] 0 2024-04-12T05:37:23.763 INFO:tasks.workunit.client.1.smithi067.stdout:7/484: write d0/d2/f78 [572906,82431] 0 2024-04-12T05:37:23.766 INFO:tasks.workunit.client.1.smithi067.stdout:2/518: unlink d5/da/f74 0 2024-04-12T05:37:23.766 INFO:tasks.workunit.client.1.smithi067.stdout:9/422: creat d1/d28/d27/d32/f81 x:0 0 0 2024-04-12T05:37:23.766 INFO:tasks.workunit.client.1.smithi067.stdout:0/530: symlink dd/d6f/la8 0 2024-04-12T05:37:23.768 INFO:tasks.workunit.client.1.smithi067.stdout:7/485: write d0/d2/d5/d1d/d43/f49 [71264,20537] 0 2024-04-12T05:37:23.769 INFO:tasks.workunit.client.1.smithi067.stdout:0/531: symlink dd/d12/d14/d49/la9 0 2024-04-12T05:37:23.770 INFO:tasks.workunit.client.1.smithi067.stdout:7/486: fsync d0/d23/f97 0 2024-04-12T05:37:23.770 INFO:tasks.workunit.client.1.smithi067.stdout:9/423: symlink d1/d28/d44/l82 0 2024-04-12T05:37:23.771 INFO:tasks.workunit.client.1.smithi067.stdout:2/519: truncate d5/da/de/d14/d34/d4c/f6b 360570 0 2024-04-12T05:37:23.771 INFO:tasks.workunit.client.1.smithi067.stdout:9/424: chown d1/d28/d44/l82 225 1 2024-04-12T05:37:23.772 INFO:tasks.workunit.client.1.smithi067.stdout:7/487: write d0/d2/d5/d1d/f40 [1347227,58320] 0 2024-04-12T05:37:23.774 INFO:tasks.workunit.client.1.smithi067.stdout:9/425: write d1/f47 [190317,68178] 0 2024-04-12T05:37:23.776 INFO:tasks.workunit.client.1.smithi067.stdout:0/532: mknod dd/d12/d14/d1a/caa 0 2024-04-12T05:37:23.779 INFO:tasks.workunit.client.1.smithi067.stdout:0/533: chown dd/d12/d14/d1a/l31 0 1 2024-04-12T05:37:23.780 INFO:tasks.workunit.client.1.smithi067.stdout:9/426: fsync d1/d13/f7c 0 2024-04-12T05:37:23.781 INFO:tasks.workunit.client.1.smithi067.stdout:2/520: truncate d5/d23/d3a/f6f 2209132 0 2024-04-12T05:37:23.784 INFO:tasks.workunit.client.1.smithi067.stdout:7/488: rename d0/d2/dd/d42/d7e/d34/f62 to d0/d2/dd/fa6 0 2024-04-12T05:37:23.785 INFO:tasks.workunit.client.1.smithi067.stdout:0/534: mkdir dd/d12/d14/d49/dab 0 2024-04-12T05:37:23.788 INFO:tasks.workunit.client.1.smithi067.stdout:9/427: mknod d1/d28/d44/c83 0 2024-04-12T05:37:23.789 INFO:tasks.workunit.client.1.smithi067.stdout:9/428: write d1/d28/d23/f63 [1707180,119527] 0 2024-04-12T05:37:23.790 INFO:tasks.workunit.client.1.smithi067.stdout:7/489: creat d0/d67/d57/fa7 x:0 0 0 2024-04-12T05:37:23.796 INFO:tasks.workunit.client.1.smithi067.stdout:2/521: dread d5/da/de/d14/d34/d40/d57/f60 [0,4194304] 0 2024-04-12T05:37:23.796 INFO:tasks.workunit.client.1.smithi067.stdout:0/535: unlink dd/d12/d14/d1a/f2b 0 2024-04-12T05:37:23.798 INFO:tasks.workunit.client.1.smithi067.stdout:9/429: dwrite d1/d28/d27/d32/f81 [0,4194304] 0 2024-04-12T05:37:23.803 INFO:tasks.workunit.client.1.smithi067.stdout:0/536: rename dd/d12/d14/d51/c74 to dd/d9c/d5f/cac 0 2024-04-12T05:37:23.804 INFO:tasks.workunit.client.1.smithi067.stdout:9/430: mknod d1/d28/d21/d68/c84 0 2024-04-12T05:37:23.808 INFO:tasks.workunit.client.1.smithi067.stdout:7/490: dwrite d0/d2/dd/d42/d7e/f1f [0,4194304] 0 2024-04-12T05:37:23.811 INFO:tasks.workunit.client.0.smithi032.stdout:9/951: sync 2024-04-12T05:37:23.811 INFO:tasks.workunit.client.0.smithi032.stdout:1/997: sync 2024-04-12T05:37:23.812 INFO:tasks.workunit.client.1.smithi067.stdout:9/431: getdents d1/d28/d23 0 2024-04-12T05:37:23.813 INFO:tasks.workunit.client.0.smithi032.stdout:9/952: write d9/d32/f111 [25425,97402] 0 2024-04-12T05:37:23.813 INFO:tasks.workunit.client.0.smithi032.stdout:1/998: write d5/de/d17/dd6/de5/f135 [667107,41742] 0 2024-04-12T05:37:23.816 INFO:tasks.workunit.client.1.smithi067.stdout:7/491: dwrite d0/d2/dd/d42/d7e/f1f [0,4194304] 0 2024-04-12T05:37:23.822 INFO:tasks.workunit.client.0.smithi032.stdout:9/953: dread d9/d15/d2c/d41/f59 [0,4194304] 0 2024-04-12T05:37:23.823 INFO:tasks.workunit.client.1.smithi067.stdout:9/432: dwrite d1/d66/f6e [0,4194304] 0 2024-04-12T05:37:23.825 INFO:tasks.workunit.client.1.smithi067.stdout:2/522: dread d5/da/f1d [0,4194304] 0 2024-04-12T05:37:23.826 INFO:tasks.workunit.client.0.smithi032.stdout:1/999: dwrite d5/de/d17/d35/d73/d74/ffe [0,4194304] 0 2024-04-12T05:37:23.827 INFO:tasks.workunit.client.0.smithi032.stdout:9/954: write d9/d32/d86/ff3 [480423,77955] 0 2024-04-12T05:37:23.832 INFO:tasks.workunit.client.0.smithi032.stdout:9/955: creat d9/d58/f14e x:0 0 0 2024-04-12T05:37:23.832 INFO:tasks.workunit.client.1.smithi067.stdout:9/433: mkdir d1/d28/d21/d68/d85 0 2024-04-12T05:37:23.834 INFO:tasks.workunit.client.1.smithi067.stdout:2/523: write d5/d23/f51 [224379,39252] 0 2024-04-12T05:37:23.836 INFO:tasks.workunit.client.0.smithi032.stdout:9/956: rmdir d9/d15/d1a/d1d/d5a 39 2024-04-12T05:37:23.836 INFO:tasks.workunit.client.1.smithi067.stdout:9/434: mkdir d1/d5f/d86 0 2024-04-12T05:37:23.838 INFO:tasks.workunit.client.1.smithi067.stdout:2/524: symlink d5/la7 0 2024-04-12T05:37:23.842 INFO:tasks.workunit.client.0.smithi032.stdout:9/957: symlink d9/d58/da9/l14f 0 2024-04-12T05:37:23.843 INFO:tasks.workunit.client.0.smithi032.stdout:9/958: creat d9/d15/d1a/dba/f150 x:0 0 0 2024-04-12T05:37:23.843 INFO:tasks.workunit.client.1.smithi067.stdout:0/537: dread dd/d92/f4c [0,4194304] 0 2024-04-12T05:37:23.843 INFO:tasks.workunit.client.1.smithi067.stdout:9/435: creat d1/d13/f87 x:0 0 0 2024-04-12T05:37:23.843 INFO:tasks.workunit.client.1.smithi067.stdout:9/436: chown d1/d13 7 1 2024-04-12T05:37:23.844 INFO:tasks.workunit.client.1.smithi067.stdout:9/437: readlink d1/l64 0 2024-04-12T05:37:23.844 INFO:tasks.workunit.client.1.smithi067.stdout:0/538: mkdir dd/dad 0 2024-04-12T05:37:23.844 INFO:tasks.workunit.client.0.smithi032.stdout:9/959: chown d9/d15/d1a/d1d/f4d 55376745 1 2024-04-12T05:37:23.845 INFO:tasks.workunit.client.1.smithi067.stdout:9/438: mknod d1/d28/d21/d45/d75/c88 0 2024-04-12T05:37:23.845 INFO:tasks.workunit.client.1.smithi067.stdout:2/525: link d5/d23/c66 d5/da/de/d14/d46/ca8 0 2024-04-12T05:37:23.846 INFO:tasks.workunit.client.1.smithi067.stdout:2/526: chown d5/d6a 7 1 2024-04-12T05:37:23.847 INFO:tasks.workunit.client.1.smithi067.stdout:0/539: mknod dd/d9c/d5f/d78/cae 0 2024-04-12T05:37:23.847 INFO:tasks.workunit.client.0.smithi032.stdout:9/960: creat d9/d15/d1a/d69/dcf/f151 x:0 0 0 2024-04-12T05:37:23.848 INFO:tasks.workunit.client.1.smithi067.stdout:0/540: fsync dd/d12/f3e 0 2024-04-12T05:37:23.848 INFO:tasks.workunit.client.1.smithi067.stdout:2/527: write d5/d11/f87 [66528,3982] 0 2024-04-12T05:37:23.853 INFO:tasks.workunit.client.1.smithi067.stdout:7/492: dread d0/d23/f41 [4194304,4194304] 0 2024-04-12T05:37:23.853 INFO:tasks.workunit.client.0.smithi032.stdout:9/961: creat d9/d32/d86/f152 x:0 0 0 2024-04-12T05:37:23.853 INFO:tasks.workunit.client.1.smithi067.stdout:0/541: truncate dd/d12/d14/d49/fa6 4206772 0 2024-04-12T05:37:23.854 INFO:tasks.workunit.client.0.smithi032.stdout:9/962: dread - d9/d15/d1a/dd2/d65/d80/f130 zero size 2024-04-12T05:37:23.854 INFO:tasks.workunit.client.1.smithi067.stdout:7/493: write d0/d2/d5/d1d/f40 [3326976,9644] 0 2024-04-12T05:37:23.856 INFO:tasks.workunit.client.0.smithi032.stdout:9/963: write d9/d15/d1a/d24/df6/f132 [133337,51101] 0 2024-04-12T05:37:23.859 INFO:tasks.workunit.client.1.smithi067.stdout:0/542: rename dd/d12/d14/f9e to dd/d9c/d5f/d78/d83/faf 0 2024-04-12T05:37:23.861 INFO:tasks.workunit.client.1.smithi067.stdout:7/494: unlink d0/d2/dd/d42/d7e/d34/f3f 0 2024-04-12T05:37:23.862 INFO:tasks.workunit.client.1.smithi067.stdout:7/495: chown d0/d67/d57/d85/f8e 65030 1 2024-04-12T05:37:23.863 INFO:tasks.workunit.client.0.smithi032.stdout:9/964: link d9/d15/d1a/c1b d9/d15/d1a/d24/d27/dce/c153 0 2024-04-12T05:37:23.864 INFO:tasks.workunit.client.1.smithi067.stdout:0/543: dread dd/d12/f2a [0,4194304] 0 2024-04-12T05:37:23.866 INFO:tasks.workunit.client.1.smithi067.stdout:2/528: dread d5/d11/f2e [0,4194304] 0 2024-04-12T05:37:23.866 INFO:tasks.workunit.client.1.smithi067.stdout:7/496: mknod d0/d2/dd/da2/ca8 0 2024-04-12T05:37:23.866 INFO:tasks.workunit.client.0.smithi032.stdout:9/965: write d9/d15/d1a/d1d/d5a/d149/f5d [1349280,105013] 0 2024-04-12T05:37:23.867 INFO:tasks.workunit.client.1.smithi067.stdout:9/439: getdents d1/d28/d21/d45/d75 0 2024-04-12T05:37:23.869 INFO:tasks.workunit.client.1.smithi067.stdout:2/529: mknod d5/da/d3d/ca9 0 2024-04-12T05:37:23.870 INFO:tasks.workunit.client.1.smithi067.stdout:9/440: write d1/d28/d21/f48 [144311,77246] 0 2024-04-12T05:37:23.871 INFO:tasks.workunit.client.1.smithi067.stdout:7/497: rename d0/d23/f97 to d0/d2/d5/d4a/fa9 0 2024-04-12T05:37:23.872 INFO:tasks.workunit.client.1.smithi067.stdout:9/441: chown d1/d13/d34 24276660 1 2024-04-12T05:37:23.875 INFO:tasks.workunit.client.1.smithi067.stdout:7/498: rmdir d0/d67 39 2024-04-12T05:37:23.876 INFO:tasks.workunit.client.1.smithi067.stdout:9/442: dread d1/d28/d44/f73 [0,4194304] 0 2024-04-12T05:37:23.877 INFO:tasks.workunit.client.1.smithi067.stdout:9/443: rename d1 to d1/d28/d21/d3f/d89 22 2024-04-12T05:37:23.879 INFO:tasks.workunit.client.1.smithi067.stdout:2/530: link d5/d6a/fa6 d5/d6a/d78/faa 0 2024-04-12T05:37:23.881 INFO:tasks.workunit.client.1.smithi067.stdout:2/531: read - d5/d43/f84 zero size 2024-04-12T05:37:23.881 INFO:tasks.workunit.client.1.smithi067.stdout:9/444: creat d1/d12/f8a x:0 0 0 2024-04-12T05:37:23.882 INFO:tasks.workunit.client.1.smithi067.stdout:7/499: dwrite d0/d2/dd/d42/d7e/d34/f8f [0,4194304] 0 2024-04-12T05:37:23.883 INFO:tasks.workunit.client.1.smithi067.stdout:7/500: fsync d0/f72 0 2024-04-12T05:37:23.885 INFO:tasks.workunit.client.1.smithi067.stdout:7/501: chown d0/d2/dd/da2/f20 1061429048 1 2024-04-12T05:37:23.886 INFO:tasks.workunit.client.1.smithi067.stdout:2/532: rename d5/d11/f73 to d5/d23/fab 0 2024-04-12T05:37:23.888 INFO:tasks.workunit.client.1.smithi067.stdout:0/544: dwrite f1 [4194304,4194304] 0 2024-04-12T05:37:23.891 INFO:tasks.workunit.client.1.smithi067.stdout:0/545: unlink dd/d12/d14/d49/l9d 0 2024-04-12T05:37:23.893 INFO:tasks.workunit.client.1.smithi067.stdout:0/546: chown dd/d12/d14/d1a/f93 100998297 1 2024-04-12T05:37:23.894 INFO:tasks.workunit.client.1.smithi067.stdout:2/533: link d5/d23/d3a/f59 d5/da/de/d14/d34/fac 0 2024-04-12T05:37:23.895 INFO:tasks.workunit.client.1.smithi067.stdout:7/502: link d0/d2/d5/c8d d0/d2/dd/da2/d51/caa 0 2024-04-12T05:37:23.896 INFO:tasks.workunit.client.1.smithi067.stdout:7/503: read d0/d23/f41 [1651894,63973] 0 2024-04-12T05:37:23.899 INFO:tasks.workunit.client.1.smithi067.stdout:7/504: creat d0/d2/dd/d42/d82/d96/fab x:0 0 0 2024-04-12T05:37:23.900 INFO:tasks.workunit.client.1.smithi067.stdout:7/505: chown d0/d2/dd/d79/f6d 31377929 1 2024-04-12T05:37:23.901 INFO:tasks.workunit.client.1.smithi067.stdout:7/506: chown d0/d2/f52 131755783 1 2024-04-12T05:37:23.903 INFO:tasks.workunit.client.1.smithi067.stdout:7/507: creat d0/d2/dd/d42/d71/fac x:0 0 0 2024-04-12T05:37:23.904 INFO:tasks.workunit.client.1.smithi067.stdout:2/534: write d5/da/de/f13 [744880,33475] 0 2024-04-12T05:37:23.906 INFO:tasks.workunit.client.1.smithi067.stdout:7/508: creat d0/d67/d57/d85/fad x:0 0 0 2024-04-12T05:37:23.906 INFO:tasks.workunit.client.1.smithi067.stdout:2/535: mknod d5/d6a/d78/cad 0 2024-04-12T05:37:23.907 INFO:tasks.workunit.client.1.smithi067.stdout:2/536: dread - d5/da/de/d14/d34/fa3 zero size 2024-04-12T05:37:23.908 INFO:tasks.workunit.client.1.smithi067.stdout:7/509: dread d0/d2/d5/fe [0,4194304] 0 2024-04-12T05:37:23.909 INFO:tasks.workunit.client.1.smithi067.stdout:2/537: write d5/da/d3d/f6d [1530955,7658] 0 2024-04-12T05:37:23.911 INFO:tasks.workunit.client.1.smithi067.stdout:2/538: mkdir d5/da/dae 0 2024-04-12T05:37:23.917 INFO:tasks.workunit.client.1.smithi067.stdout:2/539: truncate d5/d43/d6e/f76 531413 0 2024-04-12T05:37:23.919 INFO:tasks.workunit.client.1.smithi067.stdout:2/540: read d5/da/de/d14/d34/f61 [3131743,26431] 0 2024-04-12T05:37:23.919 INFO:tasks.workunit.client.1.smithi067.stdout:2/541: dread - d5/d43/f84 zero size 2024-04-12T05:37:23.922 INFO:tasks.workunit.client.1.smithi067.stdout:2/542: creat d5/da/d8c/faf x:0 0 0 2024-04-12T05:37:23.922 INFO:tasks.workunit.client.1.smithi067.stdout:2/543: fdatasync d5/da/de/d14/d34/fa3 0 2024-04-12T05:37:23.924 INFO:tasks.workunit.client.1.smithi067.stdout:2/544: readlink d5/l7e 0 2024-04-12T05:37:23.924 INFO:tasks.workunit.client.1.smithi067.stdout:3/359: sync 2024-04-12T05:37:23.924 INFO:tasks.workunit.client.1.smithi067.stdout:4/293: sync 2024-04-12T05:37:23.926 INFO:tasks.workunit.client.1.smithi067.stdout:2/545: rmdir d5/da/de/d14 39 2024-04-12T05:37:23.927 INFO:tasks.workunit.client.1.smithi067.stdout:3/360: mkdir dc/d1d/d28/d3e/d53/d76 0 2024-04-12T05:37:23.929 INFO:tasks.workunit.client.1.smithi067.stdout:7/510: dread d0/d2/dd/d79/f63 [0,4194304] 0 2024-04-12T05:37:23.929 INFO:tasks.workunit.client.1.smithi067.stdout:3/361: chown dc/df/d27/l2e 348063 1 2024-04-12T05:37:23.930 INFO:tasks.workunit.client.1.smithi067.stdout:4/294: mkdir d8/da/d1d/d15/d36/d5d/d66/d6a 0 2024-04-12T05:37:23.932 INFO:tasks.workunit.client.1.smithi067.stdout:2/546: mknod d5/da/de/d14/d34/d40/d57/cb0 0 2024-04-12T05:37:23.932 INFO:tasks.workunit.client.1.smithi067.stdout:2/547: chown d5/da/d3d 11 1 2024-04-12T05:37:23.933 INFO:tasks.workunit.client.1.smithi067.stdout:3/362: creat dc/df/d27/d6b/f77 x:0 0 0 2024-04-12T05:37:23.934 INFO:tasks.workunit.client.1.smithi067.stdout:2/548: fsync d5/da/de/d14/d34/d4c/f6b 0 2024-04-12T05:37:23.934 INFO:tasks.workunit.client.1.smithi067.stdout:7/511: write d0/d67/d57/d85/f8e [318078,93880] 0 2024-04-12T05:37:23.935 INFO:tasks.workunit.client.1.smithi067.stdout:3/363: dread - dc/d1d/d28/d3e/f66 zero size 2024-04-12T05:37:23.935 INFO:tasks.workunit.client.1.smithi067.stdout:4/295: write d8/d9/d3d/d48/f3a [528766,112368] 0 2024-04-12T05:37:23.940 INFO:tasks.workunit.client.1.smithi067.stdout:7/512: unlink d0/d2/d5/d1d/f3e 0 2024-04-12T05:37:23.942 INFO:tasks.workunit.client.1.smithi067.stdout:2/549: link d5/d43/l5e d5/d6a/d78/lb1 0 2024-04-12T05:37:23.944 INFO:tasks.workunit.client.1.smithi067.stdout:3/364: dwrite dc/d1d/d4d/f55 [0,4194304] 0 2024-04-12T05:37:23.944 INFO:tasks.workunit.client.1.smithi067.stdout:2/550: mknod d5/d6a/cb2 0 2024-04-12T05:37:23.945 INFO:tasks.workunit.client.1.smithi067.stdout:2/551: dread - d5/da/de/d14/d34/fa3 zero size 2024-04-12T05:37:23.956 INFO:tasks.workunit.client.1.smithi067.stdout:4/296: dread d8/f33 [0,4194304] 0 2024-04-12T05:37:23.957 INFO:tasks.workunit.client.1.smithi067.stdout:2/552: write d5/da/de/d14/d34/fac [1758270,34782] 0 2024-04-12T05:37:23.957 INFO:tasks.workunit.client.1.smithi067.stdout:4/297: fdatasync d8/da/d1d/d15/d1c/d5f/f61 0 2024-04-12T05:37:23.960 INFO:tasks.workunit.client.1.smithi067.stdout:4/298: truncate d8/da/d1d/d15/d1c/f65 430239 0 2024-04-12T05:37:23.962 INFO:tasks.workunit.client.1.smithi067.stdout:2/553: write d5/da/de/d14/d34/d4c/f5f [3796097,50923] 0 2024-04-12T05:37:23.964 INFO:tasks.workunit.client.1.smithi067.stdout:4/299: creat d8/d9/f6b x:0 0 0 2024-04-12T05:37:23.965 INFO:tasks.workunit.client.1.smithi067.stdout:2/554: creat d5/da/de/d14/d34/fb3 x:0 0 0 2024-04-12T05:37:23.967 INFO:tasks.workunit.client.1.smithi067.stdout:2/555: truncate d5/da/de/f13 1011417 0 2024-04-12T05:37:23.969 INFO:tasks.workunit.client.1.smithi067.stdout:4/300: getdents d8/da/d1d/d15/d36 0 2024-04-12T05:37:23.970 INFO:tasks.workunit.client.1.smithi067.stdout:2/556: creat d5/fb4 x:0 0 0 2024-04-12T05:37:23.972 INFO:tasks.workunit.client.1.smithi067.stdout:4/301: symlink d8/da/d1d/d15/d36/d5d/d66/d6a/l6c 0 2024-04-12T05:37:23.972 INFO:tasks.workunit.client.1.smithi067.stdout:2/557: write d5/da/de/d14/d34/d40/f7c [1065114,26752] 0 2024-04-12T05:37:23.974 INFO:tasks.workunit.client.1.smithi067.stdout:2/558: write d5/da/de/d14/d34/d4c/f5f [1286141,48373] 0 2024-04-12T05:37:23.978 INFO:tasks.workunit.client.1.smithi067.stdout:2/559: creat d5/da/de/d14/d34/d4c/fb5 x:0 0 0 2024-04-12T05:37:23.979 INFO:tasks.workunit.client.1.smithi067.stdout:4/302: dwrite d8/da/d1d/f62 [0,4194304] 0 2024-04-12T05:37:23.982 INFO:tasks.workunit.client.1.smithi067.stdout:2/560: unlink d5/da/de/d14/c4f 0 2024-04-12T05:37:23.987 INFO:tasks.workunit.client.1.smithi067.stdout:4/303: dwrite d8/d9/f67 [0,4194304] 0 2024-04-12T05:37:23.988 INFO:tasks.workunit.client.1.smithi067.stdout:4/304: truncate d8/d9/f6b 933665 0 2024-04-12T05:37:23.992 INFO:tasks.workunit.client.1.smithi067.stdout:4/305: mkdir d8/da/d1d/d15/d58/d69/d6d 0 2024-04-12T05:37:23.995 INFO:tasks.workunit.client.1.smithi067.stdout:2/561: dwrite d5/d43/d6e/f76 [0,4194304] 0 2024-04-12T05:37:23.997 INFO:tasks.workunit.client.1.smithi067.stdout:4/306: creat d8/d9/d3d/d48/d45/f6e x:0 0 0 2024-04-12T05:37:23.998 INFO:tasks.workunit.client.1.smithi067.stdout:2/562: truncate d5/da/d8c/faf 1023369 0 2024-04-12T05:37:24.005 INFO:tasks.workunit.client.1.smithi067.stdout:4/307: creat d8/da/d1d/d15/d36/d5d/f6f x:0 0 0 2024-04-12T05:37:24.006 INFO:tasks.workunit.client.1.smithi067.stdout:2/563: dwrite d5/d6a/f88 [0,4194304] 0 2024-04-12T05:37:24.008 INFO:tasks.workunit.client.1.smithi067.stdout:4/308: write d8/d9/d3d/d48/d45/f5c [506338,74836] 0 2024-04-12T05:37:24.008 INFO:tasks.workunit.client.1.smithi067.stdout:2/564: mkdir d5/d23/d3a/db6 0 2024-04-12T05:37:24.016 INFO:tasks.workunit.client.1.smithi067.stdout:4/309: dwrite d8/da/d1d/d15/d58/f59 [0,4194304] 0 2024-04-12T05:37:24.017 INFO:tasks.workunit.client.1.smithi067.stdout:4/310: dread - d8/d9/d3d/d48/d45/f6e zero size 2024-04-12T05:37:24.020 INFO:tasks.workunit.client.1.smithi067.stdout:4/311: write d8/da/d1d/d15/d1c/f65 [757899,73085] 0 2024-04-12T05:37:24.029 INFO:tasks.workunit.client.1.smithi067.stdout:4/312: dwrite d8/d9/d3d/f4d [0,4194304] 0 2024-04-12T05:37:24.034 INFO:tasks.workunit.client.1.smithi067.stdout:4/313: getdents d8/da/d1d/d15/d58/d69 0 2024-04-12T05:37:24.034 INFO:tasks.workunit.client.1.smithi067.stdout:2/565: dread d5/da/d3d/f7d [0,4194304] 0 2024-04-12T05:37:24.037 INFO:tasks.workunit.client.1.smithi067.stdout:4/314: symlink d8/da/d1d/d15/d36/d5d/d66/d6a/l70 0 2024-04-12T05:37:24.038 INFO:tasks.workunit.client.1.smithi067.stdout:4/315: truncate d8/d9/f42 1002636 0 2024-04-12T05:37:24.038 INFO:tasks.workunit.client.1.smithi067.stdout:2/566: mknod d5/da/de/d14/d34/d4c/cb7 0 2024-04-12T05:37:24.041 INFO:tasks.workunit.client.1.smithi067.stdout:4/316: getdents d8/da/d1d/d15/d58/d69 0 2024-04-12T05:37:24.042 INFO:tasks.workunit.client.1.smithi067.stdout:4/317: readlink d8/da/d1d/d15/d36/d5d/d66/d6a/l70 0 2024-04-12T05:37:24.044 INFO:tasks.workunit.client.1.smithi067.stdout:2/567: write d5/d23/d3a/f6f [1055881,92042] 0 2024-04-12T05:37:24.044 INFO:tasks.workunit.client.1.smithi067.stdout:4/318: mkdir d8/da/d1d/d15/d36/d5d/d66/d71 0 2024-04-12T05:37:24.047 INFO:tasks.workunit.client.1.smithi067.stdout:4/319: truncate d8/d9/d3d/d48/d45/f6e 124260 0 2024-04-12T05:37:24.047 INFO:tasks.workunit.client.1.smithi067.stdout:2/568: rename d5/d11/c41 to d5/da/cb8 0 2024-04-12T05:37:24.048 INFO:tasks.workunit.client.1.smithi067.stdout:2/569: stat d5/da/de/l1c 0 2024-04-12T05:37:24.051 INFO:tasks.workunit.client.1.smithi067.stdout:4/320: dread d8/da/f18 [0,4194304] 0 2024-04-12T05:37:24.053 INFO:tasks.workunit.client.1.smithi067.stdout:4/321: chown d8/da/d1d/d15/d1c/f52 5783671 1 2024-04-12T05:37:24.053 INFO:tasks.workunit.client.1.smithi067.stdout:8/404: sync 2024-04-12T05:37:24.053 INFO:tasks.workunit.client.1.smithi067.stdout:1/453: sync 2024-04-12T05:37:24.053 INFO:tasks.workunit.client.1.smithi067.stdout:5/670: sync 2024-04-12T05:37:24.054 INFO:tasks.workunit.client.1.smithi067.stdout:6/517: sync 2024-04-12T05:37:24.056 INFO:tasks.workunit.client.1.smithi067.stdout:2/570: dwrite d5/da/de/d14/d34/d4c/f75 [0,4194304] 0 2024-04-12T05:37:24.057 INFO:tasks.workunit.client.1.smithi067.stdout:8/405: readlink d6/d8/d18/d63/d67/d69/d5d/l64 0 2024-04-12T05:37:24.057 INFO:tasks.workunit.client.1.smithi067.stdout:4/322: mknod d8/da/d1d/d15/d36/d5d/d66/c72 0 2024-04-12T05:37:24.058 INFO:tasks.workunit.client.1.smithi067.stdout:6/518: mknod dd/d11/cb6 0 2024-04-12T05:37:24.061 INFO:tasks.workunit.client.1.smithi067.stdout:8/406: symlink d6/d8/d18/d4e/l7c 0 2024-04-12T05:37:24.062 INFO:tasks.workunit.client.1.smithi067.stdout:8/407: chown d6/d8/d18/d63 1353 1 2024-04-12T05:37:24.063 INFO:tasks.workunit.client.1.smithi067.stdout:4/323: rename d8/da/d1d/c31 to d8/da/d1d/d15/d36/c73 0 2024-04-12T05:37:24.064 INFO:tasks.workunit.client.1.smithi067.stdout:2/571: dread d5/da/de/d14/d34/d4c/f75 [0,4194304] 0 2024-04-12T05:37:24.065 INFO:tasks.workunit.client.1.smithi067.stdout:8/408: symlink d6/d8/d18/d63/d67/d69/l7d 0 2024-04-12T05:37:24.070 INFO:tasks.workunit.client.1.smithi067.stdout:4/324: creat d8/da/d1d/d15/d36/d5d/f74 x:0 0 0 2024-04-12T05:37:24.075 INFO:tasks.workunit.client.1.smithi067.stdout:5/671: dread d0/d3a/d2c/f37 [0,4194304] 0 2024-04-12T05:37:24.077 INFO:tasks.workunit.client.1.smithi067.stdout:6/519: dwrite dd/d21/d23/f3c [0,4194304] 0 2024-04-12T05:37:24.079 INFO:tasks.workunit.client.0.smithi032.stdout:9/966: sync 2024-04-12T05:37:24.082 INFO:tasks.workunit.client.1.smithi067.stdout:2/572: dwrite d5/da/de/d39/f95 [0,4194304] 0 2024-04-12T05:37:24.083 INFO:tasks.workunit.client.0.smithi032.stdout:9/967: truncate d9/d58/fb6 592788 0 2024-04-12T05:37:24.083 INFO:tasks.workunit.client.1.smithi067.stdout:4/325: dread d8/d9/f67 [0,4194304] 0 2024-04-12T05:37:24.084 INFO:tasks.workunit.client.1.smithi067.stdout:5/672: truncate d0/f1c 1623307 0 2024-04-12T05:37:24.093 INFO:tasks.workunit.client.1.smithi067.stdout:6/520: read dd/d11/d27/f54 [568744,119863] 0 2024-04-12T05:37:24.093 INFO:tasks.workunit.client.1.smithi067.stdout:8/409: dwrite d6/d8/d18/f3f [0,4194304] 0 2024-04-12T05:37:24.094 INFO:tasks.workunit.client.1.smithi067.stdout:4/326: dread - d8/d9/d3d/f4e zero size 2024-04-12T05:37:24.094 INFO:tasks.workunit.client.1.smithi067.stdout:8/410: chown d6/d8/d18/f35 36 1 2024-04-12T05:37:24.094 INFO:tasks.workunit.client.1.smithi067.stdout:2/573: link d5/d6a/d78/l9e d5/d23/d3a/lb9 0 2024-04-12T05:37:24.095 INFO:tasks.workunit.client.0.smithi032.stdout:9/968: getdents d9/d15/d1a/d24/d27/dce/dd3/d109 0 2024-04-12T05:37:24.105 INFO:tasks.workunit.client.1.smithi067.stdout:8/411: symlink d6/d8/d18/df/l7e 0 2024-04-12T05:37:24.105 INFO:tasks.workunit.client.1.smithi067.stdout:2/574: mkdir d5/da/dae/dba 0 2024-04-12T05:37:24.107 INFO:tasks.workunit.client.1.smithi067.stdout:4/327: rename d8/d9/d3d/d48/d3c to d8/da/d1d/d15/d75 0 2024-04-12T05:37:24.112 INFO:tasks.workunit.client.1.smithi067.stdout:5/673: truncate d0/da/d3f/d82/fb4 1086138 0 2024-04-12T05:37:24.117 INFO:tasks.workunit.client.1.smithi067.stdout:5/674: write d0/da/fce [4086710,76436] 0 2024-04-12T05:37:24.117 INFO:tasks.workunit.client.1.smithi067.stdout:2/575: truncate d5/da/de/f13 913043 0 2024-04-12T05:37:24.122 INFO:tasks.workunit.client.1.smithi067.stdout:6/521: dwrite dd/d11/f5d [0,4194304] 0 2024-04-12T05:37:24.123 INFO:tasks.workunit.client.1.smithi067.stdout:4/328: dwrite d8/d9/d3d/f4e [0,4194304] 0 2024-04-12T05:37:24.127 INFO:tasks.workunit.client.1.smithi067.stdout:8/412: dread d6/d8/d18/df/d12/f1f [0,4194304] 0 2024-04-12T05:37:24.127 INFO:tasks.workunit.client.1.smithi067.stdout:5/675: unlink d0/d6/d40/d4b/d38/d8e/d8f/f93 0 2024-04-12T05:37:24.128 INFO:tasks.workunit.client.1.smithi067.stdout:2/576: stat d5/da/de/d14/d34/d40/c97 0 2024-04-12T05:37:24.129 INFO:tasks.workunit.client.1.smithi067.stdout:4/329: unlink d8/fe 0 2024-04-12T05:37:24.131 INFO:tasks.workunit.client.1.smithi067.stdout:4/330: readlink d8/da/l35 0 2024-04-12T05:37:24.133 INFO:tasks.workunit.client.1.smithi067.stdout:6/522: rmdir dd/d21/d64/d6f/d90 39 2024-04-12T05:37:24.133 INFO:tasks.workunit.client.0.smithi032.stdout:9/969: dread d9/df7/f11c [4194304,4194304] 0 2024-04-12T05:37:24.134 INFO:tasks.workunit.client.1.smithi067.stdout:6/523: fdatasync dd/d11/d27/f8f 0 2024-04-12T05:37:24.135 INFO:tasks.workunit.client.1.smithi067.stdout:2/577: write d5/d11/f2a [1618627,130103] 0 2024-04-12T05:37:24.136 INFO:tasks.workunit.client.0.smithi032.stdout:9/970: write d9/d15/d1a/dd2/fe3 [3766690,97581] 0 2024-04-12T05:37:24.136 INFO:tasks.workunit.client.1.smithi067.stdout:8/413: read d6/d8/d18/df/d12/f29 [3907042,13451] 0 2024-04-12T05:37:24.136 INFO:tasks.workunit.client.1.smithi067.stdout:4/331: rmdir d8/da/d1d/d15/d58 39 2024-04-12T05:37:24.137 INFO:tasks.workunit.client.1.smithi067.stdout:4/332: stat d8/d9/f6b 0 2024-04-12T05:37:24.139 INFO:tasks.workunit.client.1.smithi067.stdout:2/578: readlink d5/d6a/d78/l9e 0 2024-04-12T05:37:24.140 INFO:tasks.workunit.client.1.smithi067.stdout:8/414: creat d6/d8/d18/d36/f7f x:0 0 0 2024-04-12T05:37:24.141 INFO:tasks.workunit.client.1.smithi067.stdout:4/333: creat d8/da/d1d/d15/d36/f76 x:0 0 0 2024-04-12T05:37:24.142 INFO:tasks.workunit.client.0.smithi032.stdout:9/971: unlink d9/d15/d1a/dd2/d65/d80/d87/f14d 0 2024-04-12T05:37:24.143 INFO:tasks.workunit.client.1.smithi067.stdout:4/334: chown d8/f2b 212 1 2024-04-12T05:37:24.144 INFO:tasks.workunit.client.0.smithi032.stdout:9/972: truncate d9/d32/d6e/f91 913357 0 2024-04-12T05:37:24.144 INFO:tasks.workunit.client.1.smithi067.stdout:6/524: dwrite dd/d11/f91 [0,4194304] 0 2024-04-12T05:37:24.146 INFO:tasks.workunit.client.1.smithi067.stdout:4/335: mknod d8/c77 0 2024-04-12T05:37:24.146 INFO:tasks.workunit.client.1.smithi067.stdout:8/415: dread - d6/d8/d18/d63/d67/d69/f71 zero size 2024-04-12T05:37:24.147 INFO:tasks.workunit.client.0.smithi032.stdout:9/973: unlink d9/d58/da9/l14f 0 2024-04-12T05:37:24.148 INFO:tasks.workunit.client.1.smithi067.stdout:4/336: chown d8/d9/f67 844 1 2024-04-12T05:37:24.149 INFO:tasks.workunit.client.0.smithi032.stdout:9/974: stat d9/d15/d1a/dd2/d65/d80/d87/fc2 0 2024-04-12T05:37:24.151 INFO:tasks.workunit.client.1.smithi067.stdout:8/416: write d6/d8/d18/d3c/d4f/f65 [395339,112379] 0 2024-04-12T05:37:24.152 INFO:tasks.workunit.client.0.smithi032.stdout:9/975: mkdir d9/d15/d1a/d154 0 2024-04-12T05:37:24.153 INFO:tasks.workunit.client.1.smithi067.stdout:4/337: read - d8/da/d1d/d15/d36/f4f zero size 2024-04-12T05:37:24.155 INFO:tasks.workunit.client.0.smithi032.stdout:9/976: creat d9/d15/d1a/d69/dcf/f155 x:0 0 0 2024-04-12T05:37:24.156 INFO:tasks.workunit.client.1.smithi067.stdout:2/579: dwrite d5/da/f3e [0,4194304] 0 2024-04-12T05:37:24.156 INFO:tasks.workunit.client.1.smithi067.stdout:6/525: dread dd/d21/d64/d88/d7b/f6e [0,4194304] 0 2024-04-12T05:37:24.158 INFO:tasks.workunit.client.0.smithi032.stdout:9/977: mkdir d9/d15/d1a/dd2/d65/d80/d87/d156 0 2024-04-12T05:37:24.159 INFO:tasks.workunit.client.1.smithi067.stdout:4/338: fsync d8/d9/d3d/f4d 0 2024-04-12T05:37:24.159 INFO:tasks.workunit.client.0.smithi032.stdout:9/978: chown d9/d15/d1a/d1d/d5a/d6b/l9c 238474 1 2024-04-12T05:37:24.162 INFO:tasks.workunit.client.1.smithi067.stdout:5/676: dwrite d0/d6/d33/f71 [0,4194304] 0 2024-04-12T05:37:24.162 INFO:tasks.workunit.client.0.smithi032.stdout:9/979: fsync d9/d15/d1a/dd2/d65/d80/d87/ff9 0 2024-04-12T05:37:24.164 INFO:tasks.workunit.client.1.smithi067.stdout:6/526: creat dd/d11/fb7 x:0 0 0 2024-04-12T05:37:24.166 INFO:tasks.workunit.client.0.smithi032.stdout:9/980: mknod d9/d15/d1a/dd2/d65/d80/c157 0 2024-04-12T05:37:24.170 INFO:tasks.workunit.client.1.smithi067.stdout:5/677: mknod d0/d6/d33/d9e/ce4 0 2024-04-12T05:37:24.170 INFO:tasks.workunit.client.1.smithi067.stdout:2/580: getdents d5/da/dae 0 2024-04-12T05:37:24.172 INFO:tasks.workunit.client.0.smithi032.stdout:9/981: creat d9/d15/d1a/d24/d27/d88/dc1/f158 x:0 0 0 2024-04-12T05:37:24.172 INFO:tasks.workunit.client.1.smithi067.stdout:4/339: creat d8/da/d1d/d15/d58/f78 x:0 0 0 2024-04-12T05:37:24.174 INFO:tasks.workunit.client.0.smithi032.stdout:9/982: getdents d9/d15/d1a/dba/d13f 0 2024-04-12T05:37:24.174 INFO:tasks.workunit.client.1.smithi067.stdout:5/678: rename d0/d6/d40/d4b/l31 to d0/le5 0 2024-04-12T05:37:24.175 INFO:tasks.workunit.client.1.smithi067.stdout:5/679: rename d0/d6 to d0/d6/d40/d4b/de6 22 2024-04-12T05:37:24.176 INFO:tasks.workunit.client.0.smithi032.stdout:9/983: chown d9/d15/d1a/d1d/d5a/d149/ce6 119811504 1 2024-04-12T05:37:24.177 INFO:tasks.workunit.client.1.smithi067.stdout:2/581: link d5/da/de/d14/f7b d5/da/dae/fbb 0 2024-04-12T05:37:24.178 INFO:tasks.workunit.client.1.smithi067.stdout:5/680: stat d0/d3a/d2c/c4f 0 2024-04-12T05:37:24.180 INFO:tasks.workunit.client.1.smithi067.stdout:9/445: sync 2024-04-12T05:37:24.180 INFO:tasks.workunit.client.1.smithi067.stdout:0/547: sync 2024-04-12T05:37:24.181 INFO:tasks.workunit.client.1.smithi067.stdout:2/582: mkdir d5/da/de/d14/d34/d4c/dbc 0 2024-04-12T05:37:24.182 INFO:tasks.workunit.client.1.smithi067.stdout:4/340: dwrite d8/da/d1d/d15/d58/f59 [0,4194304] 0 2024-04-12T05:37:24.183 INFO:tasks.workunit.client.1.smithi067.stdout:0/548: chown dd/d12/f16 169083 1 2024-04-12T05:37:24.184 INFO:tasks.workunit.client.1.smithi067.stdout:8/417: dread d6/d8/d18/df/f49 [0,4194304] 0 2024-04-12T05:37:24.185 INFO:tasks.workunit.client.0.smithi032.stdout:9/984: dwrite d9/d15/d1a/d1d/d5a/d149/f123 [0,4194304] 0 2024-04-12T05:37:24.185 INFO:tasks.workunit.client.1.smithi067.stdout:4/341: write d8/da/f1f [881196,77840] 0 2024-04-12T05:37:24.189 INFO:tasks.workunit.client.1.smithi067.stdout:9/446: rename d1/d28/d23/f3d to d1/d28/d21/d45/f8b 0 2024-04-12T05:37:24.189 INFO:tasks.workunit.client.0.smithi032.stdout:9/985: mknod d9/d32/d6e/d13c/c159 0 2024-04-12T05:37:24.192 INFO:tasks.workunit.client.1.smithi067.stdout:8/418: mknod d6/d8/d18/d36/c80 0 2024-04-12T05:37:24.194 INFO:tasks.workunit.client.1.smithi067.stdout:4/342: rename d8/da/d1d/d15/d1c/f40 to d8/d39/f79 0 2024-04-12T05:37:24.196 INFO:tasks.workunit.client.1.smithi067.stdout:8/419: write d6/d8/d18/df/d12/f55 [4728004,36924] 0 2024-04-12T05:37:24.197 INFO:tasks.workunit.client.1.smithi067.stdout:0/549: creat dd/d12/d14/d1a/fb0 x:0 0 0 2024-04-12T05:37:24.198 INFO:tasks.workunit.client.1.smithi067.stdout:4/343: mkdir d8/d39/d7a 0 2024-04-12T05:37:24.200 INFO:tasks.workunit.client.1.smithi067.stdout:8/420: fdatasync d6/d8/d18/f5e 0 2024-04-12T05:37:24.201 INFO:tasks.workunit.client.1.smithi067.stdout:4/344: fdatasync d8/d9/f67 0 2024-04-12T05:37:24.202 INFO:tasks.workunit.client.1.smithi067.stdout:9/447: dwrite d1/d28/d27/d32/f81 [4194304,4194304] 0 2024-04-12T05:37:24.202 INFO:tasks.workunit.client.1.smithi067.stdout:0/550: creat dd/d12/d14/d49/fb1 x:0 0 0 2024-04-12T05:37:24.204 INFO:tasks.workunit.client.1.smithi067.stdout:5/681: dread d0/d6/d33/dd3/fa5 [0,4194304] 0 2024-04-12T05:37:24.205 INFO:tasks.workunit.client.1.smithi067.stdout:8/421: chown d6/d8/d18/d3c/d4f/l58 329 1 2024-04-12T05:37:24.205 INFO:tasks.workunit.client.1.smithi067.stdout:9/448: write d1/d13/d34/f6c [1228099,119629] 0 2024-04-12T05:37:24.206 INFO:tasks.workunit.client.1.smithi067.stdout:4/345: chown d8/da/d1d/d15/d36/d5d/d66/d6a/l6c 4400 1 2024-04-12T05:37:24.207 INFO:tasks.workunit.client.1.smithi067.stdout:8/422: write d6/d8/d18/f62 [3954932,27688] 0 2024-04-12T05:37:24.208 INFO:tasks.workunit.client.1.smithi067.stdout:5/682: truncate d0/f43 3477803 0 2024-04-12T05:37:24.210 INFO:tasks.workunit.client.1.smithi067.stdout:9/449: creat d1/d28/d27/d32/f8c x:0 0 0 2024-04-12T05:37:24.215 INFO:tasks.workunit.client.1.smithi067.stdout:8/423: truncate d6/d8/d6e/f78 450169 0 2024-04-12T05:37:24.215 INFO:tasks.workunit.client.1.smithi067.stdout:9/450: mkdir d1/d28/d44/d8d 0 2024-04-12T05:37:24.216 INFO:tasks.workunit.client.1.smithi067.stdout:0/551: dwrite dd/d12/f4e [0,4194304] 0 2024-04-12T05:37:24.217 INFO:tasks.workunit.client.1.smithi067.stdout:4/346: rename d8/da/d1d/d15/d58/d69 to d8/da/d1d/d15/d36/d5d/d66/d7b 0 2024-04-12T05:37:24.219 INFO:tasks.workunit.client.0.smithi032.stdout:9/986: dread d9/d15/d1a/d1d/d5a/d149/f84 [0,4194304] 0 2024-04-12T05:37:24.224 INFO:tasks.workunit.client.0.smithi032.stdout:9/987: creat d9/df7/d124/f15a x:0 0 0 2024-04-12T05:37:24.226 INFO:tasks.workunit.client.0.smithi032.stdout:9/988: creat d9/d58/f15b x:0 0 0 2024-04-12T05:37:24.227 INFO:tasks.workunit.client.1.smithi067.stdout:8/424: dwrite d6/f27 [0,4194304] 0 2024-04-12T05:37:24.227 INFO:tasks.workunit.client.1.smithi067.stdout:0/552: rename dd/d12/f3a to dd/d12/fb2 0 2024-04-12T05:37:24.229 INFO:tasks.workunit.client.0.smithi032.stdout:9/989: dread d9/d58/da9/fdc [0,4194304] 0 2024-04-12T05:37:24.233 INFO:tasks.workunit.client.0.smithi032.stdout:9/990: mknod d9/d15/d1a/d24/d27/dce/dd3/d109/d141/c15c 0 2024-04-12T05:37:24.236 INFO:tasks.workunit.client.1.smithi067.stdout:4/347: dwrite d8/d39/f51 [0,4194304] 0 2024-04-12T05:37:24.236 INFO:tasks.workunit.client.1.smithi067.stdout:8/425: write d6/d8/d18/d36/f59 [1542033,12633] 0 2024-04-12T05:37:24.237 INFO:tasks.workunit.client.1.smithi067.stdout:0/553: truncate dd/d12/d14/f8d 434225 0 2024-04-12T05:37:24.240 INFO:tasks.workunit.client.1.smithi067.stdout:0/554: read dd/d12/d14/d1a/d3d/f32 [46351,2886] 0 2024-04-12T05:37:24.241 INFO:tasks.workunit.client.0.smithi032.stdout:9/991: dwrite d9/d15/d1a/d24/f14a [0,4194304] 0 2024-04-12T05:37:24.242 INFO:tasks.workunit.client.1.smithi067.stdout:0/555: write dd/d12/d14/d1a/f4b [1151731,63801] 0 2024-04-12T05:37:24.245 INFO:tasks.workunit.client.1.smithi067.stdout:8/426: mknod d6/d8/d6e/c81 0 2024-04-12T05:37:24.246 INFO:tasks.workunit.client.1.smithi067.stdout:5/683: dread d0/da/f68 [0,4194304] 0 2024-04-12T05:37:24.247 INFO:tasks.workunit.client.1.smithi067.stdout:9/451: write d1/d12/f2e [526285,36852] 0 2024-04-12T05:37:24.251 INFO:tasks.workunit.client.0.smithi032.stdout:9/992: dwrite d9/d15/d1a/d1d/d5a/d6b/f12f [0,4194304] 0 2024-04-12T05:37:24.252 INFO:tasks.workunit.client.0.smithi032.stdout:9/993: chown d9/d15/c19 483641 1 2024-04-12T05:37:24.253 INFO:tasks.workunit.client.1.smithi067.stdout:8/427: mkdir d6/d8/d18/d3c/d4f/d82 0 2024-04-12T05:37:24.254 INFO:tasks.workunit.client.0.smithi032.stdout:9/994: mkdir d9/d32/d6e/d15d 0 2024-04-12T05:37:24.257 INFO:tasks.workunit.client.0.smithi032.stdout:9/995: chown d9/c12e 303 1 2024-04-12T05:37:24.258 INFO:tasks.workunit.client.1.smithi067.stdout:9/452: rename d1/d28/d21/d68/d85 to d1/d66/d8e 0 2024-04-12T05:37:24.259 INFO:tasks.workunit.client.1.smithi067.stdout:8/428: read d6/d8/d18/f41 [2093288,30169] 0 2024-04-12T05:37:24.260 INFO:tasks.workunit.client.1.smithi067.stdout:0/556: creat dd/d12/fb3 x:0 0 0 2024-04-12T05:37:24.262 INFO:tasks.workunit.client.1.smithi067.stdout:9/453: unlink d1/d28/d21/f26 0 2024-04-12T05:37:24.263 INFO:tasks.workunit.client.0.smithi032.stdout:9/996: rmdir d9/d15/d1a/dba/d13f 0 2024-04-12T05:37:24.266 INFO:tasks.workunit.client.0.smithi032.stdout:9/997: mkdir d9/d15/d1a/d154/d15e 0 2024-04-12T05:37:24.268 INFO:tasks.workunit.client.1.smithi067.stdout:4/348: write f7 [4350593,112037] 0 2024-04-12T05:37:24.269 INFO:tasks.workunit.client.1.smithi067.stdout:5/684: write d0/da/f2e [2399595,127026] 0 2024-04-12T05:37:24.269 INFO:tasks.workunit.client.1.smithi067.stdout:4/349: fdatasync d8/d9/f2a 0 2024-04-12T05:37:24.270 INFO:tasks.workunit.client.1.smithi067.stdout:7/513: sync 2024-04-12T05:37:24.271 INFO:tasks.workunit.client.1.smithi067.stdout:9/454: rename d1/d28/d23/l4a to d1/d28/d21/d45/d75/l8f 0 2024-04-12T05:37:24.271 INFO:tasks.workunit.client.1.smithi067.stdout:1/454: sync 2024-04-12T05:37:24.271 INFO:tasks.workunit.client.1.smithi067.stdout:3/365: sync 2024-04-12T05:37:24.272 INFO:tasks.workunit.client.1.smithi067.stdout:8/429: creat d6/f83 x:0 0 0 2024-04-12T05:37:24.272 INFO:tasks.workunit.client.1.smithi067.stdout:4/350: read d8/da/d1d/f62 [901274,130537] 0 2024-04-12T05:37:24.275 INFO:tasks.workunit.client.1.smithi067.stdout:9/455: readlink d1/d28/d21/l4b 0 2024-04-12T05:37:24.276 INFO:tasks.workunit.client.0.smithi032.stdout:9/998: write d9/d32/d6e/f93 [823048,123377] 0 2024-04-12T05:37:24.280 INFO:tasks.workunit.client.1.smithi067.stdout:4/351: truncate d8/da/d1d/d15/d36/f76 99993 0 2024-04-12T05:37:24.280 INFO:tasks.workunit.client.1.smithi067.stdout:9/456: creat d1/d28/d23/f90 x:0 0 0 2024-04-12T05:37:24.281 INFO:tasks.workunit.client.1.smithi067.stdout:1/455: truncate d4/d9/d12/d2c/f88 300174 0 2024-04-12T05:37:24.281 INFO:tasks.workunit.client.1.smithi067.stdout:8/430: unlink d6/d8/d18/d36/c61 0 2024-04-12T05:37:24.282 INFO:tasks.workunit.client.1.smithi067.stdout:8/431: stat d6 0 2024-04-12T05:37:24.282 INFO:tasks.workunit.client.1.smithi067.stdout:5/685: dwrite d0/da/d3f/d82/fc0 [0,4194304] 0 2024-04-12T05:37:24.284 INFO:tasks.workunit.client.1.smithi067.stdout:9/457: write d1/d28/d5e/f6f [406135,21359] 0 2024-04-12T05:37:24.285 INFO:tasks.workunit.client.1.smithi067.stdout:0/557: dread dd/d5b/f84 [0,4194304] 0 2024-04-12T05:37:24.285 INFO:tasks.workunit.client.1.smithi067.stdout:8/432: readlink d6/d8/d18/d4e/l66 0 2024-04-12T05:37:24.285 INFO:tasks.workunit.client.1.smithi067.stdout:5/686: chown d0/d6/d33/f9f 14465396 1 2024-04-12T05:37:24.286 INFO:tasks.workunit.client.1.smithi067.stdout:9/458: chown d1/d12/d15/l17 3 1 2024-04-12T05:37:24.292 INFO:tasks.workunit.client.1.smithi067.stdout:0/558: dwrite dd/d12/d14/f43 [0,4194304] 0 2024-04-12T05:37:24.301 INFO:tasks.workunit.client.1.smithi067.stdout:0/559: dread dd/d12/d14/d1a/d3d/f73 [0,4194304] 0 2024-04-12T05:37:24.327 INFO:tasks.workunit.client.1.smithi067.stdout:4/352: creat d8/da/d1d/d15/d75/f7c x:0 0 0 2024-04-12T05:37:24.331 INFO:tasks.workunit.client.1.smithi067.stdout:5/687: rename d0/d6/f6a to d0/d6/d40/d4b/d38/d8e/d8f/fe7 0 2024-04-12T05:37:24.332 INFO:tasks.workunit.client.1.smithi067.stdout:3/366: truncate dc/df/f30 3992068 0 2024-04-12T05:37:24.333 INFO:tasks.workunit.client.1.smithi067.stdout:3/367: write dc/f40 [1510589,64956] 0 2024-04-12T05:37:24.334 INFO:tasks.workunit.client.1.smithi067.stdout:0/560: creat dd/d12/d14/d51/fb4 x:0 0 0 2024-04-12T05:37:24.335 INFO:tasks.workunit.client.1.smithi067.stdout:8/433: dwrite d6/d8/d18/d63/d67/d69/f79 [4194304,4194304] 0 2024-04-12T05:37:24.335 INFO:tasks.workunit.client.1.smithi067.stdout:1/456: rmdir d4/d9/d4c 39 2024-04-12T05:37:24.339 INFO:tasks.workunit.client.1.smithi067.stdout:4/353: write d8/d9/d3d/d48/f57 [4395774,121717] 0 2024-04-12T05:37:24.339 INFO:tasks.workunit.client.1.smithi067.stdout:1/457: fdatasync d4/d9/d12/d28/f83 0 2024-04-12T05:37:24.343 INFO:tasks.workunit.client.1.smithi067.stdout:9/459: truncate d1/d13/f7c 603594 0 2024-04-12T05:37:24.343 INFO:tasks.workunit.client.1.smithi067.stdout:9/460: readlink d1/d28/d21/d45/l6d 0 2024-04-12T05:37:24.346 INFO:tasks.workunit.client.1.smithi067.stdout:5/688: mknod d0/d6/d20/d3e/ce8 0 2024-04-12T05:37:24.347 INFO:tasks.workunit.client.1.smithi067.stdout:5/689: read d0/d3a/d2c/f37 [6972735,115165] 0 2024-04-12T05:37:24.349 INFO:tasks.workunit.client.1.smithi067.stdout:3/368: fsync dc/df/d27/f3f 0 2024-04-12T05:37:24.350 INFO:tasks.workunit.client.1.smithi067.stdout:0/561: rmdir dd/d12/d14 39 2024-04-12T05:37:24.352 INFO:tasks.workunit.client.1.smithi067.stdout:8/434: unlink d6/d8/d18/df/d12/f29 0 2024-04-12T05:37:24.353 INFO:tasks.workunit.client.1.smithi067.stdout:8/435: chown d6/d8/d18/d4e/l66 28805 1 2024-04-12T05:37:24.354 INFO:tasks.workunit.client.1.smithi067.stdout:8/436: read - d6/d8/d18/d36/f5c zero size 2024-04-12T05:37:24.357 INFO:tasks.workunit.client.1.smithi067.stdout:1/458: write d4/d9/d3a/d51/f82 [919413,27913] 0 2024-04-12T05:37:24.358 INFO:tasks.workunit.client.1.smithi067.stdout:4/354: mkdir d8/da/d1d/d27/d7d 0 2024-04-12T05:37:24.364 INFO:tasks.workunit.client.1.smithi067.stdout:1/459: chown d4/d9/fb 1 1 2024-04-12T05:37:24.364 INFO:tasks.workunit.client.1.smithi067.stdout:2/583: sync 2024-04-12T05:37:24.364 INFO:tasks.workunit.client.1.smithi067.stdout:6/527: sync 2024-04-12T05:37:24.364 INFO:tasks.workunit.client.1.smithi067.stdout:4/355: rmdir d8/d9/d3d 39 2024-04-12T05:37:24.365 INFO:tasks.workunit.client.1.smithi067.stdout:2/584: chown d5/d43 4 1 2024-04-12T05:37:24.365 INFO:tasks.workunit.client.1.smithi067.stdout:9/461: symlink d1/d28/d44/d8d/l91 0 2024-04-12T05:37:24.365 INFO:tasks.workunit.client.1.smithi067.stdout:5/690: truncate d0/d6/f12 1563811 0 2024-04-12T05:37:24.366 INFO:tasks.workunit.client.1.smithi067.stdout:1/460: dread - d4/d52/d80/f65 zero size 2024-04-12T05:37:24.369 INFO:tasks.workunit.client.1.smithi067.stdout:9/462: write d1/d28/d21/d3f/f41 [4878120,87430] 0 2024-04-12T05:37:24.369 INFO:tasks.workunit.client.1.smithi067.stdout:4/356: rmdir d8/da/d1d/d15/d1c/d5f 39 2024-04-12T05:37:24.371 INFO:tasks.workunit.client.1.smithi067.stdout:0/562: creat dd/d12/d14/d51/da2/fb5 x:0 0 0 2024-04-12T05:37:24.372 INFO:tasks.workunit.client.1.smithi067.stdout:2/585: symlink d5/da/de/d14/d34/d40/lbd 0 2024-04-12T05:37:24.374 INFO:tasks.workunit.client.1.smithi067.stdout:3/369: write dc/d1d/d26/f2f [638498,54118] 0 2024-04-12T05:37:24.379 INFO:tasks.workunit.client.1.smithi067.stdout:6/528: dwrite dd/d11/f19 [0,4194304] 0 2024-04-12T05:37:24.380 INFO:tasks.workunit.client.1.smithi067.stdout:6/529: chown dd/d11/l85 922 1 2024-04-12T05:37:24.381 INFO:tasks.workunit.client.1.smithi067.stdout:3/370: chown dc/df/l1b 12854020 1 2024-04-12T05:37:24.384 INFO:tasks.workunit.client.1.smithi067.stdout:8/437: dwrite d6/d8/d18/d63/d67/d69/f79 [0,4194304] 0 2024-04-12T05:37:24.384 INFO:tasks.workunit.client.1.smithi067.stdout:6/530: dread - dd/d11/fac zero size 2024-04-12T05:37:24.385 INFO:tasks.workunit.client.1.smithi067.stdout:0/563: rename dd/d12/d14/d51/f91 to dd/d12/d14/d49/d9b/fb6 0 2024-04-12T05:37:24.396 INFO:tasks.workunit.client.1.smithi067.stdout:2/586: dwrite d5/d11/f87 [0,4194304] 0 2024-04-12T05:37:24.398 INFO:tasks.workunit.client.1.smithi067.stdout:3/371: write dc/d1d/d28/d3e/d53/f58 [198831,4234] 0 2024-04-12T05:37:24.401 INFO:tasks.workunit.client.1.smithi067.stdout:9/463: dread d1/fd [0,4194304] 0 2024-04-12T05:37:24.406 INFO:tasks.workunit.client.1.smithi067.stdout:1/461: dwrite d4/d9/d14/f8a [0,4194304] 0 2024-04-12T05:37:24.407 INFO:tasks.workunit.client.1.smithi067.stdout:4/357: dwrite d8/d9/f17 [0,4194304] 0 2024-04-12T05:37:24.436 INFO:tasks.workunit.client.1.smithi067.stdout:0/564: dwrite dd/d6f/f98 [0,4194304] 0 2024-04-12T05:37:24.437 INFO:tasks.workunit.client.1.smithi067.stdout:0/565: chown dd/d9c/d5f 54199 1 2024-04-12T05:37:24.438 INFO:tasks.workunit.client.1.smithi067.stdout:8/438: rename d6/d8/d18/d36/f44 to d6/d8/d6e/f84 0 2024-04-12T05:37:24.443 INFO:tasks.workunit.client.1.smithi067.stdout:7/514: sync 2024-04-12T05:37:24.448 INFO:tasks.workunit.client.1.smithi067.stdout:3/372: mkdir dc/d1d/d28/d3e/d53/d78 0 2024-04-12T05:37:24.451 INFO:tasks.workunit.client.1.smithi067.stdout:9/464: dwrite d1/d12/d15/f43 [0,4194304] 0 2024-04-12T05:37:24.455 INFO:tasks.workunit.client.1.smithi067.stdout:2/587: mknod d5/da/de/d14/d34/d40/d57/cbe 0 2024-04-12T05:37:24.455 INFO:tasks.workunit.client.1.smithi067.stdout:2/588: stat d5/f12 0 2024-04-12T05:37:24.459 INFO:tasks.workunit.client.1.smithi067.stdout:4/358: fdatasync d8/d9/f21 0 2024-04-12T05:37:24.462 INFO:tasks.workunit.client.1.smithi067.stdout:6/531: getdents dd/d21/d23/d80/d93 0 2024-04-12T05:37:24.463 INFO:tasks.workunit.client.1.smithi067.stdout:7/515: dread d0/d2/dd/d42/d7e/f65 [0,4194304] 0 2024-04-12T05:37:24.465 INFO:tasks.workunit.client.1.smithi067.stdout:6/532: write dd/d11/f5d [676117,52867] 0 2024-04-12T05:37:24.467 INFO:tasks.workunit.client.1.smithi067.stdout:0/566: write dd/d12/f59 [3944153,38774] 0 2024-04-12T05:37:24.474 INFO:tasks.workunit.client.1.smithi067.stdout:9/465: mknod d1/d28/d44/c92 0 2024-04-12T05:37:24.474 INFO:tasks.workunit.client.1.smithi067.stdout:1/462: rename d4/d9/d3a/d51/d55/d60/f91 to d4/d46/f96 0 2024-04-12T05:37:24.478 INFO:tasks.workunit.client.1.smithi067.stdout:5/691: sync 2024-04-12T05:37:24.479 INFO:tasks.workunit.client.1.smithi067.stdout:7/516: symlink d0/d67/d57/d85/lae 0 2024-04-12T05:37:24.480 INFO:tasks.workunit.client.1.smithi067.stdout:8/439: symlink d6/d8/d18/l85 0 2024-04-12T05:37:24.483 INFO:tasks.workunit.client.1.smithi067.stdout:6/533: stat dd/d21/d64/d6f/d90 0 2024-04-12T05:37:24.484 INFO:tasks.workunit.client.1.smithi067.stdout:0/567: unlink dd/d12/d14/d51/c55 0 2024-04-12T05:37:24.485 INFO:tasks.workunit.client.1.smithi067.stdout:6/534: write dd/d11/d27/f9a [52636,27157] 0 2024-04-12T05:37:24.489 INFO:tasks.workunit.client.1.smithi067.stdout:4/359: creat d8/d9/d3d/d48/d45/f7e x:0 0 0 2024-04-12T05:37:24.490 INFO:tasks.workunit.client.1.smithi067.stdout:8/440: dwrite d6/d8/d18/d36/f5c [0,4194304] 0 2024-04-12T05:37:24.490 INFO:tasks.workunit.client.1.smithi067.stdout:9/466: creat d1/d28/d21/d68/f93 x:0 0 0 2024-04-12T05:37:24.491 INFO:tasks.workunit.client.1.smithi067.stdout:4/360: write d8/da/d1d/d15/d75/f7c [841740,37403] 0 2024-04-12T05:37:24.492 INFO:tasks.workunit.client.1.smithi067.stdout:9/467: write d1/d28/d21/d45/d75/f80 [539284,86356] 0 2024-04-12T05:37:24.493 INFO:tasks.workunit.client.1.smithi067.stdout:4/361: dread - d8/da/d1d/d27/f56 zero size 2024-04-12T05:37:24.498 INFO:tasks.workunit.client.1.smithi067.stdout:0/568: unlink dd/d92/f70 0 2024-04-12T05:37:24.500 INFO:tasks.workunit.client.1.smithi067.stdout:6/535: creat dd/d21/d64/d6f/da7/fb8 x:0 0 0 2024-04-12T05:37:24.501 INFO:tasks.workunit.client.1.smithi067.stdout:6/536: readlink dd/d11/l85 0 2024-04-12T05:37:24.505 INFO:tasks.workunit.client.1.smithi067.stdout:7/517: mknod d0/d2/d5/da1/caf 0 2024-04-12T05:37:24.506 INFO:tasks.workunit.client.1.smithi067.stdout:4/362: mknod d8/d39/c7f 0 2024-04-12T05:37:24.508 INFO:tasks.workunit.client.1.smithi067.stdout:0/569: symlink dd/d5b/lb7 0 2024-04-12T05:37:24.508 INFO:tasks.workunit.client.1.smithi067.stdout:1/463: link d4/d9/d12/d6f/c7f d4/d9/d3a/c97 0 2024-04-12T05:37:24.509 INFO:tasks.workunit.client.1.smithi067.stdout:0/570: fdatasync dd/d12/d14/f8d 0 2024-04-12T05:37:24.511 INFO:tasks.workunit.client.1.smithi067.stdout:9/468: dread d1/d28/d21/d3f/d4f/f60 [0,4194304] 0 2024-04-12T05:37:24.512 INFO:tasks.workunit.client.0.smithi032.stdout:9/999: sync 2024-04-12T05:37:24.512 INFO:tasks.workunit.client.1.smithi067.stdout:9/469: dread - d1/d28/d21/d45/d75/f7d zero size 2024-04-12T05:37:24.513 INFO:tasks.workunit.client.1.smithi067.stdout:5/692: creat d0/d3a/d2c/fe9 x:0 0 0 2024-04-12T05:37:24.517 INFO:tasks.workunit.client.1.smithi067.stdout:6/537: dread dd/d21/d64/d88/d7b/faa [0,4194304] 0 2024-04-12T05:37:24.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:24 smithi067 ceph-mon[49010]: pgmap v273: 65 pgs: 65 active+clean; 2.6 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 73 MiB/s rd, 197 MiB/s wr, 439 op/s 2024-04-12T05:37:24.519 INFO:tasks.workunit.client.1.smithi067.stdout:8/441: rename d6/d8/d18/d3c to d6/d86 0 2024-04-12T05:37:24.521 INFO:tasks.workunit.client.1.smithi067.stdout:8/442: write d6/d8/d18/d63/d67/d69/f79 [4920100,75478] 0 2024-04-12T05:37:24.522 INFO:tasks.workunit.client.1.smithi067.stdout:1/464: truncate d4/d9/d14/f21 703713 0 2024-04-12T05:37:24.527 INFO:tasks.workunit.client.1.smithi067.stdout:9/470: rename d1/c10 to d1/d13/d34/c94 0 2024-04-12T05:37:24.527 INFO:tasks.workunit.client.1.smithi067.stdout:9/471: chown d1/l5 3 1 2024-04-12T05:37:24.528 INFO:tasks.workunit.client.1.smithi067.stdout:9/472: fdatasync d1/d28/d21/d3f/d4f/f57 0 2024-04-12T05:37:24.530 INFO:tasks.workunit.client.1.smithi067.stdout:9/473: write d1/d28/d21/d45/d75/f80 [568027,68479] 0 2024-04-12T05:37:24.531 INFO:tasks.workunit.client.1.smithi067.stdout:9/474: truncate d1/d12/f53 655594 0 2024-04-12T05:37:24.532 INFO:tasks.workunit.client.0.smithi032.stderr:+ rm -rf -- ./tmp.GNFw4y39uT 2024-04-12T05:37:24.533 INFO:tasks.workunit.client.1.smithi067.stdout:9/475: write d1/d28/d21/d68/f93 [12189,63948] 0 2024-04-12T05:37:24.537 INFO:tasks.workunit.client.1.smithi067.stdout:4/363: getdents d8/da/d1d/d15/d1c 0 2024-04-12T05:37:24.539 INFO:tasks.workunit.client.1.smithi067.stdout:6/538: rename dd/d21/d64/d88/d7b/faa to dd/d21/d64/d6f/d90/d98/fb9 0 2024-04-12T05:37:24.541 INFO:tasks.workunit.client.1.smithi067.stdout:4/364: dread d8/d9/d3d/f4d [0,4194304] 0 2024-04-12T05:37:24.543 INFO:tasks.workunit.client.1.smithi067.stdout:4/365: write d8/da/d1d/d15/d58/f59 [2201388,123471] 0 2024-04-12T05:37:24.545 INFO:tasks.workunit.client.1.smithi067.stdout:4/366: stat d8/d9/d3d/d48/f57 0 2024-04-12T05:37:24.546 INFO:tasks.workunit.client.1.smithi067.stdout:0/571: link c8 dd/d12/d14/d1a/cb8 0 2024-04-12T05:37:24.548 INFO:tasks.workunit.client.1.smithi067.stdout:0/572: read dd/d5b/f8e [791307,33650] 0 2024-04-12T05:37:24.548 INFO:tasks.workunit.client.1.smithi067.stdout:6/539: mkdir dd/d21/d23/d80/d93/dba 0 2024-04-12T05:37:24.548 INFO:tasks.workunit.client.1.smithi067.stdout:4/367: write d8/da/d1d/d15/d1c/f4b [885628,110690] 0 2024-04-12T05:37:24.551 INFO:tasks.workunit.client.1.smithi067.stdout:6/540: read dd/d21/d64/d88/f96 [995268,79922] 0 2024-04-12T05:37:24.552 INFO:tasks.workunit.client.1.smithi067.stdout:4/368: truncate d8/d39/f79 1234845 0 2024-04-12T05:37:24.555 INFO:tasks.workunit.client.1.smithi067.stdout:0/573: dread dd/d6f/d72/f8b [0,4194304] 0 2024-04-12T05:37:24.555 INFO:tasks.workunit.client.1.smithi067.stdout:2/589: mkdir d5/dbf 0 2024-04-12T05:37:24.556 INFO:tasks.workunit.client.1.smithi067.stdout:6/541: symlink dd/d21/d3f/lbb 0 2024-04-12T05:37:24.557 INFO:tasks.workunit.client.1.smithi067.stdout:4/369: symlink d8/da/d1d/d15/d1c/d5f/l80 0 2024-04-12T05:37:24.557 INFO:tasks.workunit.client.1.smithi067.stdout:6/542: fsync dd/f89 0 2024-04-12T05:37:24.558 INFO:tasks.workunit.client.1.smithi067.stdout:0/574: chown dd/d12/d14/d1a/d3d/l8c 1 1 2024-04-12T05:37:24.558 INFO:tasks.workunit.client.1.smithi067.stdout:2/590: write d5/da/d3d/f6d [1605704,32643] 0 2024-04-12T05:37:24.561 INFO:tasks.workunit.client.1.smithi067.stdout:6/543: readlink dd/d21/d64/d6f/d90/lb0 0 2024-04-12T05:37:24.563 INFO:tasks.workunit.client.1.smithi067.stdout:6/544: truncate dd/d21/d66/f83 734368 0 2024-04-12T05:37:24.563 INFO:tasks.workunit.client.1.smithi067.stdout:0/575: creat dd/d12/d14/d51/fb9 x:0 0 0 2024-04-12T05:37:24.564 INFO:tasks.workunit.client.1.smithi067.stdout:4/370: creat d8/da/d1d/d15/d36/d5d/d66/d7b/d6d/f81 x:0 0 0 2024-04-12T05:37:24.569 INFO:tasks.workunit.client.1.smithi067.stdout:6/545: symlink dd/d21/d23/d80/d93/lbc 0 2024-04-12T05:37:24.572 INFO:tasks.workunit.client.1.smithi067.stdout:0/576: fsync dd/d12/d14/d1a/d3d/f32 0 2024-04-12T05:37:24.573 INFO:tasks.workunit.client.1.smithi067.stdout:2/591: dwrite d5/d11/f29 [0,4194304] 0 2024-04-12T05:37:24.574 INFO:tasks.workunit.client.1.smithi067.stdout:3/373: dwrite f1 [0,4194304] 0 2024-04-12T05:37:24.575 INFO:tasks.workunit.client.1.smithi067.stdout:0/577: fsync dd/d5b/f84 0 2024-04-12T05:37:24.576 INFO:tasks.workunit.client.1.smithi067.stdout:2/592: dread - d5/da/d3d/f8e zero size 2024-04-12T05:37:24.577 INFO:tasks.workunit.client.1.smithi067.stdout:6/546: creat dd/d11/d27/d4a/d8b/fbd x:0 0 0 2024-04-12T05:37:24.578 INFO:tasks.workunit.client.1.smithi067.stdout:3/374: symlink dc/d50/l79 0 2024-04-12T05:37:24.582 INFO:tasks.workunit.client.1.smithi067.stdout:3/375: fsync f9 0 2024-04-12T05:37:24.583 INFO:tasks.workunit.client.1.smithi067.stdout:6/547: fsync dd/d21/d64/d6f/d90/f9d 0 2024-04-12T05:37:24.583 INFO:tasks.workunit.client.1.smithi067.stdout:3/376: dread - dc/d50/f60 zero size 2024-04-12T05:37:24.586 INFO:tasks.workunit.client.1.smithi067.stdout:3/377: read dc/d1d/d28/f2a [3142107,94877] 0 2024-04-12T05:37:24.589 INFO:tasks.workunit.client.1.smithi067.stdout:6/548: link dd/d11/l38 dd/d21/d64/lbe 0 2024-04-12T05:37:24.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:24 smithi032 ceph-mon[39731]: pgmap v273: 65 pgs: 65 active+clean; 2.6 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 73 MiB/s rd, 197 MiB/s wr, 439 op/s 2024-04-12T05:37:24.591 INFO:tasks.workunit.client.1.smithi067.stdout:5/693: truncate d0/da/f2e 1498364 0 2024-04-12T05:37:24.591 INFO:tasks.workunit.client.1.smithi067.stdout:1/465: write d4/d9/d12/d28/f33 [436527,30585] 0 2024-04-12T05:37:24.592 INFO:tasks.workunit.client.1.smithi067.stdout:5/694: chown d0/d3a/d2c/fe9 491910 1 2024-04-12T05:37:24.596 INFO:tasks.workunit.client.1.smithi067.stdout:6/549: unlink dd/d21/d64/d6f/d90/d98/ca3 0 2024-04-12T05:37:24.597 INFO:tasks.workunit.client.1.smithi067.stdout:8/443: truncate d6/d8/d18/f62 1324213 0 2024-04-12T05:37:24.600 INFO:tasks.workunit.client.1.smithi067.stdout:1/466: dread d4/d9/f22 [0,4194304] 0 2024-04-12T05:37:24.604 INFO:tasks.workunit.client.1.smithi067.stdout:4/371: fsync d8/d39/f79 0 2024-04-12T05:37:24.607 INFO:tasks.workunit.client.1.smithi067.stdout:1/467: chown d4/d9/d12/d59/l86 12032 1 2024-04-12T05:37:24.609 INFO:tasks.workunit.client.1.smithi067.stdout:5/695: dwrite d0/d6/d20/d3e/fbc [0,4194304] 0 2024-04-12T05:37:24.611 INFO:tasks.workunit.client.1.smithi067.stdout:6/550: truncate dd/d21/d64/d6f/d90/f9d 48124 0 2024-04-12T05:37:24.613 INFO:tasks.workunit.client.1.smithi067.stdout:9/476: dwrite d1/d28/f50 [0,4194304] 0 2024-04-12T05:37:24.616 INFO:tasks.workunit.client.1.smithi067.stdout:8/444: unlink d6/d8/d18/d63/d67/c7b 0 2024-04-12T05:37:24.618 INFO:tasks.workunit.client.1.smithi067.stdout:2/593: dread d5/d11/f2a [0,4194304] 0 2024-04-12T05:37:24.620 INFO:tasks.workunit.client.1.smithi067.stdout:1/468: symlink d4/d9/d3a/d51/l98 0 2024-04-12T05:37:24.621 INFO:tasks.workunit.client.1.smithi067.stdout:4/372: dread d8/d9/d3d/d48/f3a [0,4194304] 0 2024-04-12T05:37:24.623 INFO:tasks.workunit.client.1.smithi067.stdout:9/477: unlink d1/d28/c1f 0 2024-04-12T05:37:24.630 INFO:tasks.workunit.client.1.smithi067.stdout:1/469: mknod d4/d9/d3a/d51/d55/c99 0 2024-04-12T05:37:24.635 INFO:tasks.workunit.client.1.smithi067.stdout:5/696: dwrite d0/d6/d20/fbe [0,4194304] 0 2024-04-12T05:37:24.636 INFO:tasks.workunit.client.1.smithi067.stdout:1/470: chown d4/d3d/f77 429659169 1 2024-04-12T05:37:24.637 INFO:tasks.workunit.client.1.smithi067.stdout:0/578: dwrite dd/d12/d14/d49/d9b/fb6 [0,4194304] 0 2024-04-12T05:37:24.644 INFO:tasks.workunit.client.1.smithi067.stdout:0/579: dread - dd/d6f/d72/fa3 zero size 2024-04-12T05:37:24.644 INFO:tasks.workunit.client.1.smithi067.stdout:2/594: dread d5/da/de/d39/f95 [0,4194304] 0 2024-04-12T05:37:24.644 INFO:tasks.workunit.client.1.smithi067.stdout:4/373: dwrite d8/d9/d3d/d48/d45/f7e [0,4194304] 0 2024-04-12T05:37:24.644 INFO:tasks.workunit.client.1.smithi067.stdout:1/471: mkdir d4/d9/d3a/d51/d55/d9a 0 2024-04-12T05:37:24.653 INFO:tasks.workunit.client.1.smithi067.stdout:1/472: truncate d4/d9/d12/f81 1242370 0 2024-04-12T05:37:24.655 INFO:tasks.workunit.client.1.smithi067.stdout:8/445: dread d6/d8/d18/f25 [0,4194304] 0 2024-04-12T05:37:24.655 INFO:tasks.workunit.client.1.smithi067.stdout:5/697: symlink d0/d6/d40/d4b/d38/lea 0 2024-04-12T05:37:24.657 INFO:tasks.workunit.client.1.smithi067.stdout:0/580: creat dd/d6f/fba x:0 0 0 2024-04-12T05:37:24.658 INFO:tasks.workunit.client.1.smithi067.stdout:2/595: dwrite d5/d43/d6e/f72 [4194304,4194304] 0 2024-04-12T05:37:24.658 INFO:tasks.workunit.client.1.smithi067.stdout:4/374: getdents d8/da/d1d/d15/d36/d5d/d66/d71 0 2024-04-12T05:37:24.658 INFO:tasks.workunit.client.1.smithi067.stdout:8/446: write d6/f27 [2121247,119911] 0 2024-04-12T05:37:24.660 INFO:tasks.workunit.client.1.smithi067.stdout:1/473: write d4/d9/d12/f1b [5804352,125456] 0 2024-04-12T05:37:24.660 INFO:tasks.workunit.client.1.smithi067.stdout:4/375: chown d8/d9/d3d 399721008 1 2024-04-12T05:37:24.662 INFO:tasks.workunit.client.1.smithi067.stdout:4/376: read - d8/da/d1d/d27/f56 zero size 2024-04-12T05:37:24.676 INFO:tasks.workunit.client.1.smithi067.stdout:4/377: getdents d8/da/d1d/d15/d36/d5d/d66 0 2024-04-12T05:37:24.676 INFO:tasks.workunit.client.1.smithi067.stdout:1/474: dwrite d4/d9/d12/f16 [0,4194304] 0 2024-04-12T05:37:24.679 INFO:tasks.workunit.client.1.smithi067.stdout:5/698: unlink d0/d6/d40/d4b/d38/d8e/d8f/c97 0 2024-04-12T05:37:24.679 INFO:tasks.workunit.client.1.smithi067.stdout:8/447: dwrite d6/d8/d18/d36/f68 [0,4194304] 0 2024-04-12T05:37:24.682 INFO:tasks.workunit.client.1.smithi067.stdout:2/596: dwrite d5/da/de/f69 [4194304,4194304] 0 2024-04-12T05:37:24.682 INFO:tasks.workunit.client.1.smithi067.stdout:1/475: fdatasync d4/d46/f96 0 2024-04-12T05:37:24.689 INFO:tasks.workunit.client.1.smithi067.stdout:8/448: dread d6/d8/d18/f5e [0,4194304] 0 2024-04-12T05:37:24.696 INFO:tasks.workunit.client.1.smithi067.stdout:2/597: truncate d5/da/de/f94 520349 0 2024-04-12T05:37:24.700 INFO:tasks.workunit.client.1.smithi067.stdout:4/378: dwrite d8/d9/d3d/f4d [0,4194304] 0 2024-04-12T05:37:24.701 INFO:tasks.workunit.client.1.smithi067.stdout:8/449: creat d6/d8/d18/d36/f87 x:0 0 0 2024-04-12T05:37:24.706 INFO:tasks.workunit.client.1.smithi067.stdout:5/699: dwrite d0/f2 [0,4194304] 0 2024-04-12T05:37:24.709 INFO:tasks.workunit.client.1.smithi067.stdout:1/476: dwrite d4/d9/d12/d2c/f7e [0,4194304] 0 2024-04-12T05:37:24.709 INFO:tasks.workunit.client.1.smithi067.stdout:5/700: fdatasync d0/d6/fc8 0 2024-04-12T05:37:24.717 INFO:tasks.workunit.client.1.smithi067.stdout:2/598: rename d5/d23/c82 to d5/da/de/d14/d46/cc0 0 2024-04-12T05:37:24.718 INFO:tasks.workunit.client.1.smithi067.stdout:1/477: creat d4/d9/d12/d28/f9b x:0 0 0 2024-04-12T05:37:24.722 INFO:tasks.workunit.client.1.smithi067.stdout:8/450: rename d6/c56 to d6/d8/d18/d63/c88 0 2024-04-12T05:37:24.725 INFO:tasks.workunit.client.1.smithi067.stdout:1/478: chown d4/d9/d4d/c69 15059992 1 2024-04-12T05:37:24.725 INFO:tasks.workunit.client.1.smithi067.stdout:4/379: write d8/da/d1d/f49 [971790,29329] 0 2024-04-12T05:37:24.727 INFO:tasks.workunit.client.1.smithi067.stdout:8/451: dread - d6/d8/d18/d63/d67/d69/f71 zero size 2024-04-12T05:37:24.728 INFO:tasks.workunit.client.1.smithi067.stdout:1/479: chown d4/d46/l4e 5053 1 2024-04-12T05:37:24.729 INFO:tasks.workunit.client.1.smithi067.stdout:2/599: dwrite d5/d11/f8a [0,4194304] 0 2024-04-12T05:37:24.736 INFO:tasks.workunit.client.1.smithi067.stdout:4/380: truncate d8/da/d1d/d15/d36/f4f 273501 0 2024-04-12T05:37:24.736 INFO:tasks.workunit.client.1.smithi067.stdout:8/452: write d6/d8/d18/d36/f7f [352612,75639] 0 2024-04-12T05:37:24.736 INFO:tasks.workunit.client.1.smithi067.stdout:1/480: write d4/d9/d12/d2c/f8d [1943227,70816] 0 2024-04-12T05:37:24.739 INFO:tasks.workunit.client.1.smithi067.stdout:4/381: chown d8/f2b 11057 1 2024-04-12T05:37:24.739 INFO:tasks.workunit.client.1.smithi067.stdout:5/701: dwrite d0/d6/d33/f9f [0,4194304] 0 2024-04-12T05:37:24.741 INFO:tasks.workunit.client.1.smithi067.stdout:1/481: symlink d4/d9/d3a/l9c 0 2024-04-12T05:37:24.746 INFO:tasks.workunit.client.1.smithi067.stdout:8/453: write d6/d8/d18/d36/f70 [694741,81180] 0 2024-04-12T05:37:24.749 INFO:tasks.workunit.client.1.smithi067.stdout:1/482: link d4/d46/l4e d4/d8f/l9d 0 2024-04-12T05:37:24.751 INFO:tasks.workunit.client.1.smithi067.stdout:4/382: dread d8/da/d1d/d15/d1c/f65 [0,4194304] 0 2024-04-12T05:37:24.755 INFO:tasks.workunit.client.1.smithi067.stdout:1/483: fsync d4/d9/d12/f53 0 2024-04-12T05:37:24.755 INFO:tasks.workunit.client.1.smithi067.stdout:5/702: getdents d0/d6/d33/dd3 0 2024-04-12T05:37:24.756 INFO:tasks.workunit.client.1.smithi067.stdout:8/454: dwrite d6/d8/d6e/f84 [0,4194304] 0 2024-04-12T05:37:24.758 INFO:tasks.workunit.client.1.smithi067.stdout:5/703: stat d0/d6/c4d 0 2024-04-12T05:37:24.759 INFO:tasks.workunit.client.1.smithi067.stdout:1/484: dread d4/d9/f22 [0,4194304] 0 2024-04-12T05:37:24.763 INFO:tasks.workunit.client.1.smithi067.stdout:4/383: dwrite d8/d9/d3d/f3f [0,4194304] 0 2024-04-12T05:37:24.769 INFO:tasks.workunit.client.1.smithi067.stdout:8/455: dread d6/d8/d18/d36/f5c [0,4194304] 0 2024-04-12T05:37:24.769 INFO:tasks.workunit.client.1.smithi067.stdout:4/384: chown d8/d9/d3d/d64 545358359 1 2024-04-12T05:37:24.770 INFO:tasks.workunit.client.1.smithi067.stdout:5/704: dwrite d0/d6/d20/d62/f15 [0,4194304] 0 2024-04-12T05:37:24.772 INFO:tasks.workunit.client.1.smithi067.stdout:1/485: fdatasync d4/d9/f45 0 2024-04-12T05:37:24.773 INFO:tasks.workunit.client.1.smithi067.stdout:8/456: unlink d6/d8/d18/df/f24 0 2024-04-12T05:37:24.775 INFO:tasks.workunit.client.1.smithi067.stdout:8/457: chown d6/l2b 40712881 1 2024-04-12T05:37:24.775 INFO:tasks.workunit.client.1.smithi067.stdout:1/486: getdents d4/d9/d14/d7a 0 2024-04-12T05:37:24.776 INFO:tasks.workunit.client.1.smithi067.stdout:8/458: stat d6/d8/d18/d4e 0 2024-04-12T05:37:24.794 INFO:tasks.workunit.client.1.smithi067.stdout:7/518: sync 2024-04-12T05:37:24.794 INFO:tasks.workunit.client.1.smithi067.stdout:4/385: dwrite d8/d9/f67 [0,4194304] 0 2024-04-12T05:37:24.797 INFO:tasks.workunit.client.1.smithi067.stdout:7/519: stat d0/d23/d45/c3b 0 2024-04-12T05:37:24.801 INFO:tasks.workunit.client.1.smithi067.stdout:8/459: creat d6/d8/d18/d63/d67/d69/f89 x:0 0 0 2024-04-12T05:37:24.803 INFO:tasks.workunit.client.1.smithi067.stdout:4/386: symlink d8/da/d1d/d15/d1c/d68/l82 0 2024-04-12T05:37:24.804 INFO:tasks.workunit.client.1.smithi067.stdout:5/705: dwrite d0/d6/d40/d4b/fda [0,4194304] 0 2024-04-12T05:37:24.805 INFO:tasks.workunit.client.1.smithi067.stdout:1/487: dread d4/d9/f45 [0,4194304] 0 2024-04-12T05:37:24.807 INFO:tasks.workunit.client.1.smithi067.stdout:4/387: stat d8/da/d1d/d15/d36/d5d/d66/d71 0 2024-04-12T05:37:24.815 INFO:tasks.workunit.client.1.smithi067.stdout:5/706: write d0/da/fce [4798036,24958] 0 2024-04-12T05:37:24.815 INFO:tasks.workunit.client.1.smithi067.stdout:1/488: read d4/d9/f1c [166120,26517] 0 2024-04-12T05:37:24.815 INFO:tasks.workunit.client.1.smithi067.stdout:4/388: truncate d8/da/d1d/d15/d36/d5d/f6f 377847 0 2024-04-12T05:37:24.815 INFO:tasks.workunit.client.1.smithi067.stdout:5/707: mknod d0/d3a/d2c/dc6/ddd/ceb 0 2024-04-12T05:37:24.819 INFO:tasks.workunit.client.1.smithi067.stdout:5/708: mkdir d0/da/dec 0 2024-04-12T05:37:24.823 INFO:tasks.workunit.client.1.smithi067.stdout:4/389: dread d8/d39/f51 [0,4194304] 0 2024-04-12T05:37:24.825 INFO:tasks.workunit.client.1.smithi067.stdout:5/709: fdatasync d0/d3a/d2c/f37 0 2024-04-12T05:37:24.828 INFO:tasks.workunit.client.1.smithi067.stdout:7/520: truncate d0/d23/d45/f59 134776 0 2024-04-12T05:37:24.829 INFO:tasks.workunit.client.1.smithi067.stdout:1/489: dwrite d4/d9/d4c/f89 [0,4194304] 0 2024-04-12T05:37:24.833 INFO:tasks.workunit.client.1.smithi067.stdout:5/710: symlink d0/da/led 0 2024-04-12T05:37:24.834 INFO:tasks.workunit.client.1.smithi067.stdout:7/521: write d0/d23/d45/f1a [3701428,119704] 0 2024-04-12T05:37:24.837 INFO:tasks.workunit.client.1.smithi067.stdout:5/711: dread - d0/d6/d40/d4b/fe0 zero size 2024-04-12T05:37:24.842 INFO:tasks.workunit.client.1.smithi067.stdout:4/390: dwrite d8/da/d1d/d15/d1c/f52 [0,4194304] 0 2024-04-12T05:37:24.845 INFO:tasks.workunit.client.1.smithi067.stdout:1/490: link d4/d46/l4e d4/d46/l9e 0 2024-04-12T05:37:24.845 INFO:tasks.workunit.client.1.smithi067.stdout:4/391: readlink d8/l22 0 2024-04-12T05:37:24.847 INFO:tasks.workunit.client.1.smithi067.stdout:8/460: dwrite d6/d8/d18/df/f20 [0,4194304] 0 2024-04-12T05:37:24.847 INFO:tasks.workunit.client.1.smithi067.stdout:1/491: stat d4/d9/d3a/d51/c42 0 2024-04-12T05:37:24.851 INFO:tasks.workunit.client.1.smithi067.stdout:1/492: chown d4/d9/d3a/d51/d55 445 1 2024-04-12T05:37:24.851 INFO:tasks.workunit.client.1.smithi067.stdout:4/392: write d8/d39/f51 [3897477,75047] 0 2024-04-12T05:37:24.853 INFO:tasks.workunit.client.1.smithi067.stdout:1/493: fsync d4/d9/d12/f81 0 2024-04-12T05:37:24.858 INFO:tasks.workunit.client.1.smithi067.stdout:4/393: stat d8/da/d1d/d15/d1c/d68 0 2024-04-12T05:37:24.859 INFO:tasks.workunit.client.1.smithi067.stdout:1/494: dread d4/d9/d12/f53 [0,4194304] 0 2024-04-12T05:37:24.859 INFO:tasks.workunit.client.1.smithi067.stdout:5/712: dwrite d0/d6/d40/f7f [0,4194304] 0 2024-04-12T05:37:24.862 INFO:tasks.workunit.client.1.smithi067.stdout:8/461: creat d6/f8a x:0 0 0 2024-04-12T05:37:24.868 INFO:tasks.workunit.client.1.smithi067.stdout:7/522: truncate d0/d23/f41 853295 0 2024-04-12T05:37:24.868 INFO:tasks.workunit.client.1.smithi067.stdout:4/394: mknod d8/d9/c83 0 2024-04-12T05:37:24.868 INFO:tasks.workunit.client.1.smithi067.stdout:5/713: fdatasync d0/d6/d40/f90 0 2024-04-12T05:37:24.869 INFO:tasks.workunit.client.1.smithi067.stdout:8/462: mkdir d6/d8/d8b 0 2024-04-12T05:37:24.873 INFO:tasks.workunit.client.1.smithi067.stdout:4/395: write d8/da/d1d/d15/d36/f5a [499097,98874] 0 2024-04-12T05:37:24.873 INFO:tasks.workunit.client.1.smithi067.stdout:8/463: fsync d6/d8/d6e/f84 0 2024-04-12T05:37:24.874 INFO:tasks.workunit.client.1.smithi067.stdout:5/714: write d0/da/d41/f60 [2953373,121764] 0 2024-04-12T05:37:24.875 INFO:tasks.workunit.client.1.smithi067.stdout:7/523: mkdir d0/d2/dd/da2/d51/db0 0 2024-04-12T05:37:24.876 INFO:tasks.workunit.client.1.smithi067.stdout:1/495: dwrite d4/d9/d12/d28/f83 [0,4194304] 0 2024-04-12T05:37:24.877 INFO:tasks.workunit.client.1.smithi067.stdout:7/524: fsync d0/f12 0 2024-04-12T05:37:24.884 INFO:tasks.workunit.client.1.smithi067.stdout:4/396: symlink d8/da/d1d/d15/d36/d5d/d66/l84 0 2024-04-12T05:37:24.884 INFO:tasks.workunit.client.1.smithi067.stdout:1/496: dread d4/f5 [0,4194304] 0 2024-04-12T05:37:24.889 INFO:tasks.workunit.client.1.smithi067.stdout:1/497: dread - d4/d9/f75 zero size 2024-04-12T05:37:24.892 INFO:tasks.workunit.client.1.smithi067.stdout:7/525: dwrite d0/d2/dd/d42/d7e/f94 [0,4194304] 0 2024-04-12T05:37:24.897 INFO:tasks.workunit.client.1.smithi067.stdout:5/715: dwrite d0/d6/d40/d4b/d38/d8e/d8f/db9/dbd/fd9 [0,4194304] 0 2024-04-12T05:37:24.897 INFO:tasks.workunit.client.1.smithi067.stdout:4/397: truncate d8/d9/f21 511442 0 2024-04-12T05:37:24.898 INFO:tasks.workunit.client.1.smithi067.stdout:5/716: readlink d0/d6/d40/d4b/l39 0 2024-04-12T05:37:24.899 INFO:tasks.workunit.client.1.smithi067.stdout:8/464: truncate d6/d8/d18/d36/f68 3043883 0 2024-04-12T05:37:24.901 INFO:tasks.workunit.client.1.smithi067.stdout:8/465: stat d6/d8/d18/d36/f59 0 2024-04-12T05:37:24.901 INFO:tasks.workunit.client.1.smithi067.stdout:4/398: read d8/d9/d3d/d48/f57 [1239793,24569] 0 2024-04-12T05:37:24.902 INFO:tasks.workunit.client.1.smithi067.stdout:1/498: fsync d4/d9/fd 0 2024-04-12T05:37:24.906 INFO:tasks.workunit.client.1.smithi067.stdout:7/526: mknod d0/d2/cb1 0 2024-04-12T05:37:24.916 INFO:tasks.workunit.client.1.smithi067.stdout:1/499: dwrite d4/d9/d3a/d51/d55/f93 [0,4194304] 0 2024-04-12T05:37:24.916 INFO:tasks.workunit.client.1.smithi067.stdout:7/527: mknod d0/d2/dd/da2/d51/db0/cb2 0 2024-04-12T05:37:24.917 INFO:tasks.workunit.client.1.smithi067.stdout:4/399: rmdir d8/d39/d7a 0 2024-04-12T05:37:24.920 INFO:tasks.workunit.client.1.smithi067.stdout:5/717: link d0/d6/d20/d3e/l8d d0/d6/d20/lee 0 2024-04-12T05:37:24.921 INFO:tasks.workunit.client.1.smithi067.stdout:1/500: symlink d4/d9/d4c/d5d/l9f 0 2024-04-12T05:37:24.926 INFO:tasks.workunit.client.1.smithi067.stdout:1/501: write d4/d52/f70 [1778376,30586] 0 2024-04-12T05:37:24.936 INFO:tasks.workunit.client.1.smithi067.stdout:7/528: dwrite d0/d2/dd/d42/d7e/f30 [0,4194304] 0 2024-04-12T05:37:24.938 INFO:tasks.workunit.client.1.smithi067.stdout:4/400: dwrite d8/da/d1d/d15/d1c/f4b [0,4194304] 0 2024-04-12T05:37:24.938 INFO:tasks.workunit.client.1.smithi067.stdout:5/718: write d0/da/d3f/d82/f80 [532228,4121] 0 2024-04-12T05:37:24.949 INFO:tasks.workunit.client.1.smithi067.stdout:5/719: mkdir d0/d6/d40/d4b/d38/d8e/d8f/db9/dbd/def 0 2024-04-12T05:37:24.951 INFO:tasks.workunit.client.1.smithi067.stdout:8/466: dread d6/d8/d18/f3f [0,4194304] 0 2024-04-12T05:37:24.954 INFO:tasks.workunit.client.1.smithi067.stdout:1/502: dwrite d4/d52/d80/f6d [0,4194304] 0 2024-04-12T05:37:24.959 INFO:tasks.workunit.client.1.smithi067.stdout:0/581: sync 2024-04-12T05:37:24.959 INFO:tasks.workunit.client.1.smithi067.stdout:9/478: sync 2024-04-12T05:37:24.959 INFO:tasks.workunit.client.1.smithi067.stdout:3/378: sync 2024-04-12T05:37:24.959 INFO:tasks.workunit.client.1.smithi067.stdout:6/551: sync 2024-04-12T05:37:24.961 INFO:tasks.workunit.client.1.smithi067.stdout:0/582: chown dd/d9c/l7f 36923 1 2024-04-12T05:37:24.961 INFO:tasks.workunit.client.1.smithi067.stdout:5/720: rename d0/d6/d20/c50 to d0/d6/d20/cf0 0 2024-04-12T05:37:24.961 INFO:tasks.workunit.client.1.smithi067.stdout:0/583: fdatasync dd/d9c/d5f/f90 0 2024-04-12T05:37:24.965 INFO:tasks.workunit.client.1.smithi067.stdout:9/479: chown d1/d5f 27090701 1 2024-04-12T05:37:24.965 INFO:tasks.workunit.client.1.smithi067.stdout:7/529: truncate d0/d67/f56 201416 0 2024-04-12T05:37:24.970 INFO:tasks.workunit.client.1.smithi067.stdout:5/721: chown d0/d3a/d2c/f3d 57 1 2024-04-12T05:37:24.971 INFO:tasks.workunit.client.1.smithi067.stdout:3/379: chown dc/df/d27/f6a 18 1 2024-04-12T05:37:24.971 INFO:tasks.workunit.client.1.smithi067.stdout:9/480: fsync d1/d28/d21/d45/d75/f80 0 2024-04-12T05:37:24.979 INFO:tasks.workunit.client.1.smithi067.stdout:0/584: dread dd/d12/d14/d49/f6d [0,4194304] 0 2024-04-12T05:37:24.984 INFO:tasks.workunit.client.1.smithi067.stdout:4/401: dwrite d8/d39/f3b [0,4194304] 0 2024-04-12T05:37:24.986 INFO:tasks.workunit.client.1.smithi067.stdout:1/503: dwrite d4/d3d/f4b [4194304,4194304] 0 2024-04-12T05:37:24.986 INFO:tasks.workunit.client.1.smithi067.stdout:9/481: truncate d1/fd 3150600 0 2024-04-12T05:37:24.986 INFO:tasks.workunit.client.1.smithi067.stdout:3/380: symlink dc/d50/l7a 0 2024-04-12T05:37:24.994 INFO:tasks.workunit.client.1.smithi067.stdout:3/381: write f1 [3232272,48629] 0 2024-04-12T05:37:24.995 INFO:tasks.workunit.client.1.smithi067.stdout:9/482: fsync d1/d28/d27/d32/f81 0 2024-04-12T05:37:24.995 INFO:tasks.workunit.client.1.smithi067.stdout:7/530: dwrite d0/d2/f74 [0,4194304] 0 2024-04-12T05:37:25.001 INFO:tasks.workunit.client.1.smithi067.stdout:4/402: rmdir d8/da/d1d/d15/d36/d5d/d66/d6a 39 2024-04-12T05:37:25.005 INFO:tasks.workunit.client.1.smithi067.stdout:3/382: rmdir dc/d50 39 2024-04-12T05:37:25.009 INFO:tasks.workunit.client.1.smithi067.stdout:5/722: dwrite d0/d6/d33/d88/fe2 [0,4194304] 0 2024-04-12T05:37:25.014 INFO:tasks.workunit.client.1.smithi067.stdout:1/504: link d4/d9/f17 d4/d8f/fa0 0 2024-04-12T05:37:25.017 INFO:tasks.workunit.client.1.smithi067.stdout:4/403: chown d8/da/d1d/c1e 2624903 1 2024-04-12T05:37:25.017 INFO:tasks.workunit.client.1.smithi067.stdout:0/585: truncate dd/d9c/d64/f7e 3902251 0 2024-04-12T05:37:25.018 INFO:tasks.workunit.client.1.smithi067.stdout:9/483: link d1/d28/d5e/f67 d1/d12/f95 0 2024-04-12T05:37:25.022 INFO:tasks.workunit.client.1.smithi067.stdout:3/383: write dc/d1d/d28/f2a [4443434,49946] 0 2024-04-12T05:37:25.022 INFO:tasks.workunit.client.1.smithi067.stdout:6/552: dread fa [4194304,4194304] 0 2024-04-12T05:37:25.022 INFO:tasks.workunit.client.1.smithi067.stdout:9/484: dread - d1/d13/f87 zero size 2024-04-12T05:37:25.022 INFO:tasks.workunit.client.1.smithi067.stdout:4/404: creat d8/da/d1d/f85 x:0 0 0 2024-04-12T05:37:25.028 INFO:tasks.workunit.client.1.smithi067.stdout:0/586: creat dd/d6f/d72/fbb x:0 0 0 2024-04-12T05:37:25.028 INFO:tasks.workunit.client.1.smithi067.stdout:5/723: dwrite d0/d6/d40/d4b/fda [0,4194304] 0 2024-04-12T05:37:25.028 INFO:tasks.workunit.client.1.smithi067.stdout:9/485: write d1/d28/f2c [1424124,14444] 0 2024-04-12T05:37:25.031 INFO:tasks.workunit.client.1.smithi067.stdout:4/405: rmdir d8/d9/d3d/d48/d45 39 2024-04-12T05:37:25.033 INFO:tasks.workunit.client.1.smithi067.stdout:1/505: dwrite d4/d9/d12/d28/f83 [0,4194304] 0 2024-04-12T05:37:25.033 INFO:tasks.workunit.client.1.smithi067.stdout:3/384: mknod dc/d1d/d26/d52/c7b 0 2024-04-12T05:37:25.037 INFO:tasks.workunit.client.1.smithi067.stdout:7/531: dread d0/d2/dd/d42/d7e/f4e [0,4194304] 0 2024-04-12T05:37:25.039 INFO:tasks.workunit.client.1.smithi067.stdout:0/587: mknod dd/d9c/d64/cbc 0 2024-04-12T05:37:25.041 INFO:tasks.workunit.client.1.smithi067.stdout:3/385: mknod dc/df/d27/d6b/c7c 0 2024-04-12T05:37:25.041 INFO:tasks.workunit.client.1.smithi067.stdout:7/532: write d0/f12 [5184311,2048] 0 2024-04-12T05:37:25.041 INFO:tasks.workunit.client.1.smithi067.stdout:1/506: dread d4/d9/f1c [0,4194304] 0 2024-04-12T05:37:25.042 INFO:tasks.workunit.client.1.smithi067.stdout:3/386: readlink dc/df/l45 0 2024-04-12T05:37:25.044 INFO:tasks.workunit.client.1.smithi067.stdout:1/507: dread - d4/d52/d80/f65 zero size 2024-04-12T05:37:25.048 INFO:tasks.workunit.client.1.smithi067.stdout:6/553: symlink dd/d11/d27/d4a/d51/lbf 0 2024-04-12T05:37:25.048 INFO:tasks.workunit.client.1.smithi067.stdout:9/486: creat d1/d28/d21/d7f/f96 x:0 0 0 2024-04-12T05:37:25.049 INFO:tasks.workunit.client.1.smithi067.stdout:0/588: rmdir dd/d12/d14/d1a/d3d 39 2024-04-12T05:37:25.049 INFO:tasks.workunit.client.1.smithi067.stdout:7/533: rename d0/d2/dd/c95 to d0/d67/d57/cb3 0 2024-04-12T05:37:25.050 INFO:tasks.workunit.client.1.smithi067.stdout:5/724: write d0/d6/d33/f3c [976635,125675] 0 2024-04-12T05:37:25.050 INFO:tasks.workunit.client.1.smithi067.stdout:1/508: rmdir d4/d3d/d7b 39 2024-04-12T05:37:25.052 INFO:tasks.workunit.client.1.smithi067.stdout:3/387: dread dc/d1d/f24 [0,4194304] 0 2024-04-12T05:37:25.054 INFO:tasks.workunit.client.1.smithi067.stdout:0/589: mkdir dd/dbd 0 2024-04-12T05:37:25.056 INFO:tasks.workunit.client.1.smithi067.stdout:1/509: symlink d4/d9/d4d/la1 0 2024-04-12T05:37:25.057 INFO:tasks.workunit.client.1.smithi067.stdout:1/510: fdatasync d4/d9/d12/d2c/f88 0 2024-04-12T05:37:25.058 INFO:tasks.workunit.client.1.smithi067.stdout:9/487: link d1/d28/d5e/f6f d1/d28/d21/d3f/f97 0 2024-04-12T05:37:25.059 INFO:tasks.workunit.client.1.smithi067.stdout:4/406: dread d8/da/f1f [0,4194304] 0 2024-04-12T05:37:25.060 INFO:tasks.workunit.client.1.smithi067.stdout:0/590: mkdir dd/dbd/dbe 0 2024-04-12T05:37:25.060 INFO:tasks.workunit.client.1.smithi067.stdout:1/511: fsync d4/d9/d4c/d5d/f71 0 2024-04-12T05:37:25.060 INFO:tasks.workunit.client.1.smithi067.stdout:5/725: getdents d0/d6/dae 0 2024-04-12T05:37:25.061 INFO:tasks.workunit.client.1.smithi067.stdout:6/554: dread dd/d11/d27/f58 [0,4194304] 0 2024-04-12T05:37:25.061 INFO:tasks.workunit.client.1.smithi067.stdout:3/388: link la dc/d1d/d4d/l7d 0 2024-04-12T05:37:25.061 INFO:tasks.workunit.client.1.smithi067.stdout:9/488: chown d1/d12/d15/l17 3 1 2024-04-12T05:37:25.062 INFO:tasks.workunit.client.1.smithi067.stdout:0/591: dread - dd/d6f/d72/f82 zero size 2024-04-12T05:37:25.064 INFO:tasks.workunit.client.1.smithi067.stdout:4/407: creat d8/da/d1d/d27/f86 x:0 0 0 2024-04-12T05:37:25.065 INFO:tasks.workunit.client.1.smithi067.stdout:1/512: dread d4/f5 [0,4194304] 0 2024-04-12T05:37:25.065 INFO:tasks.workunit.client.1.smithi067.stdout:5/726: mknod d0/d3a/d2c/d91/cf1 0 2024-04-12T05:37:25.066 INFO:tasks.workunit.client.1.smithi067.stdout:9/489: symlink d1/d28/d21/d45/l98 0 2024-04-12T05:37:25.067 INFO:tasks.workunit.client.1.smithi067.stdout:6/555: mknod dd/d11/d27/d4a/d51/cc0 0 2024-04-12T05:37:25.068 INFO:tasks.workunit.client.1.smithi067.stdout:5/727: mknod d0/cf2 0 2024-04-12T05:37:25.073 INFO:tasks.workunit.client.1.smithi067.stdout:6/556: mkdir dd/d11/dc1 0 2024-04-12T05:37:25.073 INFO:tasks.workunit.client.1.smithi067.stdout:9/490: dread d1/d5f/f65 [0,4194304] 0 2024-04-12T05:37:25.074 INFO:tasks.workunit.client.1.smithi067.stdout:1/513: truncate d4/d8f/fa0 56270 0 2024-04-12T05:37:25.075 INFO:tasks.workunit.client.1.smithi067.stdout:7/534: write d0/d2/d5/d1d/f36 [4223795,82809] 0 2024-04-12T05:37:25.076 INFO:tasks.workunit.client.1.smithi067.stdout:9/491: readlink d1/d28/d21/l4b 0 2024-04-12T05:37:25.080 INFO:tasks.workunit.client.1.smithi067.stdout:7/535: chown d0/d2/dd/d79/l8c 30580 1 2024-04-12T05:37:25.081 INFO:tasks.workunit.client.1.smithi067.stdout:4/408: dwrite d8/da/d1d/d15/d58/f78 [0,4194304] 0 2024-04-12T05:37:25.081 INFO:tasks.workunit.client.1.smithi067.stdout:5/728: write d0/d6/d40/d4b/d38/d8e/d8f/fe7 [1077035,37755] 0 2024-04-12T05:37:25.081 INFO:tasks.workunit.client.1.smithi067.stdout:9/492: chown d1/d12/d15/l25 2685 1 2024-04-12T05:37:25.086 INFO:tasks.workunit.client.1.smithi067.stdout:9/493: read d1/d28/d21/d3f/f97 [297294,6278] 0 2024-04-12T05:37:25.087 INFO:tasks.workunit.client.1.smithi067.stdout:3/389: truncate dc/d1d/f43 2173676 0 2024-04-12T05:37:25.087 INFO:tasks.workunit.client.1.smithi067.stdout:1/514: rename d4/d44/l5c to d4/d9/d3a/la2 0 2024-04-12T05:37:25.089 INFO:tasks.workunit.client.1.smithi067.stdout:0/592: truncate dd/d12/f2a 2804160 0 2024-04-12T05:37:25.089 INFO:tasks.workunit.client.1.smithi067.stdout:7/536: readlink d0/d67/l76 0 2024-04-12T05:37:25.090 INFO:tasks.workunit.client.1.smithi067.stdout:5/729: dread - d0/d6/d40/fd4 zero size 2024-04-12T05:37:25.093 INFO:tasks.workunit.client.1.smithi067.stdout:6/557: creat dd/d21/d23/d80/d93/dba/fc2 x:0 0 0 2024-04-12T05:37:25.095 INFO:tasks.workunit.client.1.smithi067.stdout:5/730: read - d0/d3a/d2c/dc6/ddd/fe3 zero size 2024-04-12T05:37:25.095 INFO:tasks.workunit.client.1.smithi067.stdout:9/494: creat d1/d28/d21/d45/f99 x:0 0 0 2024-04-12T05:37:25.097 INFO:tasks.workunit.client.1.smithi067.stdout:0/593: write dd/d6f/d72/fa3 [663713,32966] 0 2024-04-12T05:37:25.099 INFO:tasks.workunit.client.1.smithi067.stdout:3/390: symlink dc/d1d/d28/d3e/d53/l7e 0 2024-04-12T05:37:25.099 INFO:tasks.workunit.client.1.smithi067.stdout:0/594: write dd/d12/fb3 [940574,6457] 0 2024-04-12T05:37:25.100 INFO:tasks.workunit.client.1.smithi067.stdout:1/515: truncate d4/d9/fd 838725 0 2024-04-12T05:37:25.101 INFO:tasks.workunit.client.1.smithi067.stdout:5/731: chown d0/d6/d33/f71 30123 1 2024-04-12T05:37:25.101 INFO:tasks.workunit.client.1.smithi067.stdout:7/537: rename d0/d2 to d0/d23/d80/da0/db4 0 2024-04-12T05:37:25.102 INFO:tasks.workunit.client.1.smithi067.stdout:4/409: creat d8/da/d1d/d15/d36/d5d/d66/d7b/f87 x:0 0 0 2024-04-12T05:37:25.105 INFO:tasks.workunit.client.1.smithi067.stdout:1/516: chown d4/d9/d14/f8a 6111 1 2024-04-12T05:37:25.105 INFO:tasks.workunit.client.1.smithi067.stdout:3/391: creat dc/d1d/d28/d3e/f7f x:0 0 0 2024-04-12T05:37:25.105 INFO:tasks.workunit.client.1.smithi067.stdout:7/538: read d0/f8 [410434,130361] 0 2024-04-12T05:37:25.109 INFO:tasks.workunit.client.1.smithi067.stdout:1/517: chown d4/d3d/f4b 3063711 1 2024-04-12T05:37:25.111 INFO:tasks.workunit.client.1.smithi067.stdout:5/732: rmdir d0/da/d3f/d82 39 2024-04-12T05:37:25.112 INFO:tasks.workunit.client.1.smithi067.stdout:3/392: write dc/d1d/d28/d3e/d53/f5c [898420,66054] 0 2024-04-12T05:37:25.113 INFO:tasks.workunit.client.1.smithi067.stdout:1/518: write d4/d9/d3a/d51/f82 [639951,120572] 0 2024-04-12T05:37:25.117 INFO:tasks.workunit.client.1.smithi067.stdout:5/733: symlink d0/d6/dae/lf3 0 2024-04-12T05:37:25.118 INFO:tasks.workunit.client.1.smithi067.stdout:4/410: dwrite d8/da/d1d/d15/d58/f59 [0,4194304] 0 2024-04-12T05:37:25.118 INFO:tasks.workunit.client.1.smithi067.stdout:3/393: mknod dc/d1d/d4d/c80 0 2024-04-12T05:37:25.120 INFO:tasks.workunit.client.1.smithi067.stdout:7/539: truncate d0/d23/d80/da0/db4/dd/d42/d7e/f94 4579000 0 2024-04-12T05:37:25.120 INFO:tasks.workunit.client.1.smithi067.stdout:9/495: truncate d1/f47 3165988 0 2024-04-12T05:37:25.120 INFO:tasks.workunit.client.1.smithi067.stdout:4/411: chown d8/da/d1d/d27 13 1 2024-04-12T05:37:25.122 INFO:tasks.workunit.client.1.smithi067.stdout:5/734: symlink d0/d6/d20/lf4 0 2024-04-12T05:37:25.127 INFO:tasks.workunit.client.1.smithi067.stdout:3/394: dread - dc/d1d/d28/d3e/f63 zero size 2024-04-12T05:37:25.127 INFO:tasks.workunit.client.1.smithi067.stdout:7/540: write d0/d23/d45/f9e [633085,40014] 0 2024-04-12T05:37:25.127 INFO:tasks.workunit.client.1.smithi067.stdout:0/595: dread dd/d12/f4e [0,4194304] 0 2024-04-12T05:37:25.127 INFO:tasks.workunit.client.1.smithi067.stdout:4/412: mkdir d8/da/d1d/d15/d36/d88 0 2024-04-12T05:37:25.129 INFO:tasks.workunit.client.1.smithi067.stdout:3/395: stat dc/d50 0 2024-04-12T05:37:25.129 INFO:tasks.workunit.client.1.smithi067.stdout:5/735: unlink d0/d6/d33/c94 0 2024-04-12T05:37:25.133 INFO:tasks.workunit.client.1.smithi067.stdout:9/496: getdents d1/d28/d21/d45/d75 0 2024-04-12T05:37:25.134 INFO:tasks.workunit.client.1.smithi067.stdout:3/396: readlink dc/d1d/d28/l3b 0 2024-04-12T05:37:25.134 INFO:tasks.workunit.client.1.smithi067.stdout:7/541: unlink d0/d23/d80/da0/db4/d5/d1d/f36 0 2024-04-12T05:37:25.134 INFO:tasks.workunit.client.1.smithi067.stdout:1/519: dwrite d4/d9/d12/f36 [0,4194304] 0 2024-04-12T05:37:25.134 INFO:tasks.workunit.client.1.smithi067.stdout:0/596: write dd/d9c/d5f/d78/d83/f9a [302331,25975] 0 2024-04-12T05:37:25.135 INFO:tasks.workunit.client.1.smithi067.stdout:3/397: fsync dc/d1d/f44 0 2024-04-12T05:37:25.142 INFO:tasks.workunit.client.1.smithi067.stdout:1/520: symlink d4/d9/d3a/la3 0 2024-04-12T05:37:25.142 INFO:tasks.workunit.client.1.smithi067.stdout:4/413: write d8/d9/f20 [756593,39244] 0 2024-04-12T05:37:25.146 INFO:tasks.workunit.client.1.smithi067.stdout:7/542: chown d0/d23/d80/da0/db4/d5/d4a/l91 3094 1 2024-04-12T05:37:25.147 INFO:tasks.workunit.client.1.smithi067.stdout:5/736: dwrite d0/da/fbb [0,4194304] 0 2024-04-12T05:37:25.148 INFO:tasks.workunit.client.1.smithi067.stdout:9/497: creat d1/f9a x:0 0 0 2024-04-12T05:37:25.152 INFO:tasks.workunit.client.1.smithi067.stdout:3/398: truncate dc/d1d/d28/d3e/f7f 360536 0 2024-04-12T05:37:25.153 INFO:tasks.workunit.client.1.smithi067.stdout:1/521: mknod d4/d9/d12/d2c/ca4 0 2024-04-12T05:37:25.153 INFO:tasks.workunit.client.1.smithi067.stdout:0/597: link dd/d12/f21 dd/d12/d14/fbf 0 2024-04-12T05:37:25.154 INFO:tasks.workunit.client.1.smithi067.stdout:7/543: mknod d0/d67/d57/cb5 0 2024-04-12T05:37:25.156 INFO:tasks.workunit.client.1.smithi067.stdout:5/737: stat d0/d6/d20/d3e/f95 0 2024-04-12T05:37:25.157 INFO:tasks.workunit.client.1.smithi067.stdout:1/522: chown d4/d9/d3a/d51/c47 5372397 1 2024-04-12T05:37:25.159 INFO:tasks.workunit.client.1.smithi067.stdout:0/598: write dd/d6f/fba [667147,28863] 0 2024-04-12T05:37:25.159 INFO:tasks.workunit.client.1.smithi067.stdout:4/414: dwrite d8/da/d1d/f49 [0,4194304] 0 2024-04-12T05:37:25.162 INFO:tasks.workunit.client.1.smithi067.stdout:1/523: symlink d4/d9/d12/d2c/la5 0 2024-04-12T05:37:25.162 INFO:tasks.workunit.client.1.smithi067.stdout:9/498: dread d1/d28/d21/d45/d75/f80 [0,4194304] 0 2024-04-12T05:37:25.164 INFO:tasks.workunit.client.1.smithi067.stdout:0/599: mknod dd/d9c/d5f/cc0 0 2024-04-12T05:37:25.166 INFO:tasks.workunit.client.1.smithi067.stdout:1/524: mknod d4/d46/ca6 0 2024-04-12T05:37:25.167 INFO:tasks.workunit.client.1.smithi067.stdout:4/415: write d8/d9/d3d/d48/d45/f6e [865865,102337] 0 2024-04-12T05:37:25.167 INFO:tasks.workunit.client.1.smithi067.stdout:9/499: truncate d1/d12/f74 970898 0 2024-04-12T05:37:25.168 INFO:tasks.workunit.client.1.smithi067.stdout:0/600: symlink dd/d5b/lc1 0 2024-04-12T05:37:25.170 INFO:tasks.workunit.client.1.smithi067.stdout:1/525: chown d4/d9/f45 1073 1 2024-04-12T05:37:25.171 INFO:tasks.workunit.client.1.smithi067.stdout:5/738: dwrite d0/d6/d40/d4b/d38/d8e/d8f/fe7 [0,4194304] 0 2024-04-12T05:37:25.173 INFO:tasks.workunit.client.1.smithi067.stdout:9/500: creat d1/d28/d21/f9b x:0 0 0 2024-04-12T05:37:25.173 INFO:tasks.workunit.client.1.smithi067.stdout:0/601: read dd/d12/d14/d1a/f93 [406974,49777] 0 2024-04-12T05:37:25.175 INFO:tasks.workunit.client.1.smithi067.stdout:4/416: rename d8/da/d1d/f44 to d8/da/d1d/d15/d36/f89 0 2024-04-12T05:37:25.176 INFO:tasks.workunit.client.1.smithi067.stdout:9/501: chown d1/d28/d27/d32/f8c 8 1 2024-04-12T05:37:25.178 INFO:tasks.workunit.client.1.smithi067.stdout:9/502: read - d1/d28/d44/f72 zero size 2024-04-12T05:37:25.180 INFO:tasks.workunit.client.1.smithi067.stdout:4/417: symlink d8/da/d1d/d15/d36/d5d/l8a 0 2024-04-12T05:37:25.181 INFO:tasks.workunit.client.1.smithi067.stdout:9/503: symlink d1/d28/d21/d45/l9c 0 2024-04-12T05:37:25.182 INFO:tasks.workunit.client.1.smithi067.stdout:5/739: read d0/d6/d20/d3e/fbc [1483681,83114] 0 2024-04-12T05:37:25.183 INFO:tasks.workunit.client.1.smithi067.stdout:0/602: dread dd/d12/d14/d1a/f93 [0,4194304] 0 2024-04-12T05:37:25.183 INFO:tasks.workunit.client.1.smithi067.stdout:5/740: stat d0/d3a/d2c/dc6 0 2024-04-12T05:37:25.184 INFO:tasks.workunit.client.1.smithi067.stdout:4/418: rename d8/d9/d3d/d48 to d8/da/d8b 0 2024-04-12T05:37:25.185 INFO:tasks.workunit.client.1.smithi067.stdout:0/603: write dd/d12/d14/f8d [725459,38195] 0 2024-04-12T05:37:25.186 INFO:tasks.workunit.client.1.smithi067.stdout:4/419: rmdir d8/da 39 2024-04-12T05:37:25.190 INFO:tasks.workunit.client.1.smithi067.stdout:5/741: mkdir d0/d6/d33/d88/df5 0 2024-04-12T05:37:25.191 INFO:tasks.workunit.client.1.smithi067.stdout:9/504: dwrite d1/d28/d27/f71 [0,4194304] 0 2024-04-12T05:37:25.191 INFO:tasks.workunit.client.1.smithi067.stdout:4/420: write d8/da/d1d/d15/d36/f76 [259434,63209] 0 2024-04-12T05:37:25.192 INFO:tasks.workunit.client.1.smithi067.stdout:0/604: creat dd/dbd/dbe/fc2 x:0 0 0 2024-04-12T05:37:25.193 INFO:tasks.workunit.client.1.smithi067.stdout:9/505: readlink d1/d13/l59 0 2024-04-12T05:37:25.194 INFO:tasks.workunit.client.1.smithi067.stdout:5/742: symlink d0/lf6 0 2024-04-12T05:37:25.199 INFO:tasks.workunit.client.1.smithi067.stdout:4/421: write d8/d9/d3d/f4d [2993997,59431] 0 2024-04-12T05:37:25.199 INFO:tasks.workunit.client.1.smithi067.stdout:9/506: write d1/d12/f8a [523490,125877] 0 2024-04-12T05:37:25.199 INFO:tasks.workunit.client.1.smithi067.stdout:0/605: chown dd/d12/d14/d1a/d3d/f7c 308816 1 2024-04-12T05:37:25.200 INFO:tasks.workunit.client.1.smithi067.stdout:5/743: rmdir d0/d6/d40/d4b/d38/d8e/d8f 39 2024-04-12T05:37:25.200 INFO:tasks.workunit.client.1.smithi067.stdout:0/606: chown dd/d12/d14/d49/f4d 21761 1 2024-04-12T05:37:25.202 INFO:tasks.workunit.client.1.smithi067.stdout:4/422: mkdir d8/da/d8b/d8c 0 2024-04-12T05:37:25.205 INFO:tasks.workunit.client.1.smithi067.stdout:0/607: symlink dd/d6f/lc3 0 2024-04-12T05:37:25.205 INFO:tasks.workunit.client.1.smithi067.stdout:4/423: chown d8/f33 16 1 2024-04-12T05:37:25.207 INFO:tasks.workunit.client.1.smithi067.stdout:9/507: dwrite d1/d13/f5b [0,4194304] 0 2024-04-12T05:37:25.212 INFO:tasks.workunit.client.1.smithi067.stdout:1/526: dread d4/f7 [0,4194304] 0 2024-04-12T05:37:25.213 INFO:tasks.workunit.client.1.smithi067.stdout:4/424: symlink d8/da/d1d/d15/d1c/d5f/l8d 0 2024-04-12T05:37:25.214 INFO:tasks.workunit.client.1.smithi067.stdout:0/608: dread dd/d6f/d72/f8b [0,4194304] 0 2024-04-12T05:37:25.216 INFO:tasks.workunit.client.1.smithi067.stdout:5/744: dwrite d0/d6/f6e [0,4194304] 0 2024-04-12T05:37:25.216 INFO:tasks.workunit.client.1.smithi067.stdout:0/609: write dd/d12/f59 [2973439,26489] 0 2024-04-12T05:37:25.221 INFO:tasks.workunit.client.1.smithi067.stdout:5/745: dread d0/da/f6d [0,4194304] 0 2024-04-12T05:37:25.222 INFO:tasks.workunit.client.1.smithi067.stdout:5/746: chown d0/d6/d20/d3e/dbf 976269899 1 2024-04-12T05:37:25.224 INFO:tasks.workunit.client.1.smithi067.stdout:5/747: chown d0/d6/d20/d3e/dbf 23 1 2024-04-12T05:37:25.225 INFO:tasks.workunit.client.1.smithi067.stdout:4/425: dwrite d8/d9/f41 [0,4194304] 0 2024-04-12T05:37:25.229 INFO:tasks.workunit.client.1.smithi067.stdout:4/426: mkdir d8/da/d1d/d15/d1c/d68/d8e 0 2024-04-12T05:37:25.243 INFO:tasks.workunit.client.1.smithi067.stdout:5/748: write d0/d6/f12 [658282,48783] 0 2024-04-12T05:37:25.243 INFO:tasks.workunit.client.1.smithi067.stdout:0/610: dread dd/d12/d14/fbf [0,4194304] 0 2024-04-12T05:37:25.244 INFO:tasks.workunit.client.1.smithi067.stdout:5/749: read - d0/d3a/d2c/fe9 zero size 2024-04-12T05:37:25.246 INFO:tasks.workunit.client.1.smithi067.stdout:0/611: creat dd/d9c/d5f/fc4 x:0 0 0 2024-04-12T05:37:25.248 INFO:tasks.workunit.client.1.smithi067.stdout:0/612: mknod dd/d6f/d72/cc5 0 2024-04-12T05:37:25.249 INFO:tasks.workunit.client.1.smithi067.stdout:0/613: readlink dd/d9c/l7f 0 2024-04-12T05:37:25.252 INFO:tasks.workunit.client.1.smithi067.stdout:5/750: dwrite d0/d3a/f4c [0,4194304] 0 2024-04-12T05:37:25.254 INFO:tasks.workunit.client.1.smithi067.stdout:4/427: dread d8/d9/f11 [0,4194304] 0 2024-04-12T05:37:25.260 INFO:tasks.workunit.client.1.smithi067.stdout:4/428: creat d8/da/d1d/d15/d36/d5d/f8f x:0 0 0 2024-04-12T05:37:25.267 INFO:tasks.workunit.client.1.smithi067.stdout:0/614: dwrite dd/d6f/d72/f82 [0,4194304] 0 2024-04-12T05:37:25.267 INFO:tasks.workunit.client.1.smithi067.stdout:5/751: dwrite d0/d6/f12 [0,4194304] 0 2024-04-12T05:37:25.270 INFO:tasks.workunit.client.1.smithi067.stdout:4/429: mknod d8/da/d1d/d15/d36/d88/c90 0 2024-04-12T05:37:25.271 INFO:tasks.workunit.client.1.smithi067.stdout:5/752: symlink d0/d3a/da8/lf7 0 2024-04-12T05:37:25.277 INFO:tasks.workunit.client.1.smithi067.stdout:5/753: symlink d0/da/lf8 0 2024-04-12T05:37:25.277 INFO:tasks.workunit.client.1.smithi067.stdout:4/430: write d8/da/d1d/d15/d1c/d5f/f61 [831013,130939] 0 2024-04-12T05:37:25.278 INFO:tasks.workunit.client.1.smithi067.stdout:5/754: symlink d0/d3a/dab/lf9 0 2024-04-12T05:37:25.281 INFO:tasks.workunit.client.1.smithi067.stdout:0/615: dwrite dd/d12/d14/d51/fb9 [0,4194304] 0 2024-04-12T05:37:25.281 INFO:tasks.workunit.client.1.smithi067.stdout:4/431: mkdir d8/da/d1d/d27/d91 0 2024-04-12T05:37:25.285 INFO:tasks.workunit.client.1.smithi067.stdout:4/432: chown d8/da/d1d/d15/d58/f78 1063 1 2024-04-12T05:37:25.286 INFO:tasks.workunit.client.1.smithi067.stdout:5/755: creat d0/d3a/d2c/db3/ffa x:0 0 0 2024-04-12T05:37:25.291 INFO:tasks.workunit.client.1.smithi067.stdout:0/616: dwrite f1 [4194304,4194304] 0 2024-04-12T05:37:25.294 INFO:tasks.workunit.client.1.smithi067.stdout:5/756: truncate d0/da/d3f/d82/fb4 1008682 0 2024-04-12T05:37:25.294 INFO:tasks.workunit.client.1.smithi067.stdout:5/757: stat d0 0 2024-04-12T05:37:25.301 INFO:tasks.workunit.client.1.smithi067.stdout:5/758: creat d0/d6/d40/d4b/d38/d8e/ffb x:0 0 0 2024-04-12T05:37:25.307 INFO:tasks.workunit.client.1.smithi067.stdout:5/759: getdents d0/d6/d20 0 2024-04-12T05:37:25.312 INFO:tasks.workunit.client.1.smithi067.stdout:4/433: dread d8/da/d1d/f62 [0,4194304] 0 2024-04-12T05:37:25.314 INFO:tasks.workunit.client.1.smithi067.stdout:5/760: dwrite d0/d6/d40/d4b/d38/d8e/f96 [0,4194304] 0 2024-04-12T05:37:25.319 INFO:tasks.workunit.client.1.smithi067.stdout:4/434: getdents d8/d9 0 2024-04-12T05:37:25.319 INFO:tasks.workunit.client.1.smithi067.stdout:5/761: chown d0/d3a/d2c/f3d 107 1 2024-04-12T05:37:25.323 INFO:tasks.workunit.client.1.smithi067.stdout:4/435: mkdir d8/da/d1d/d15/d36/d5d/d66/d71/d92 0 2024-04-12T05:37:25.324 INFO:tasks.workunit.client.1.smithi067.stdout:5/762: write d0/da/d3f/d82/fa2 [416806,1925] 0 2024-04-12T05:37:25.326 INFO:tasks.workunit.client.1.smithi067.stdout:4/436: truncate d8/da/d1d/d15/d36/d5d/f6f 890076 0 2024-04-12T05:37:25.327 INFO:tasks.workunit.client.1.smithi067.stdout:4/437: read - d8/da/d1d/d15/d36/d5d/d66/d7b/f87 zero size 2024-04-12T05:37:25.328 INFO:tasks.workunit.client.1.smithi067.stdout:5/763: unlink d0/le5 0 2024-04-12T05:37:25.328 INFO:tasks.workunit.client.1.smithi067.stdout:4/438: write d8/d9/f20 [1755060,26890] 0 2024-04-12T05:37:25.329 INFO:tasks.workunit.client.1.smithi067.stdout:5/764: chown d0/d6/d20/f58 13055 1 2024-04-12T05:37:25.334 INFO:tasks.workunit.client.1.smithi067.stdout:5/765: symlink d0/d6/d40/d4b/d38/d8e/d8f/lfc 0 2024-04-12T05:37:25.336 INFO:tasks.workunit.client.1.smithi067.stdout:4/439: dwrite d8/d9/d3d/f4e [0,4194304] 0 2024-04-12T05:37:25.338 INFO:tasks.workunit.client.1.smithi067.stdout:4/440: chown d8/da/d1d/d15/d36/f89 1783 1 2024-04-12T05:37:25.340 INFO:tasks.workunit.client.1.smithi067.stdout:5/766: creat d0/d6/d40/d4b/d38/d8e/d8f/db9/dbd/ffd x:0 0 0 2024-04-12T05:37:25.341 INFO:tasks.workunit.client.1.smithi067.stdout:5/767: chown d0/d3a/da8 6710426 1 2024-04-12T05:37:25.341 INFO:tasks.workunit.client.1.smithi067.stdout:4/441: getdents d8/da/d1d/d27/d91 0 2024-04-12T05:37:25.344 INFO:tasks.workunit.client.1.smithi067.stdout:5/768: creat d0/d6/d33/dd3/ffe x:0 0 0 2024-04-12T05:37:25.346 INFO:tasks.workunit.client.1.smithi067.stdout:4/442: dread d8/da/f30 [0,4194304] 0 2024-04-12T05:37:25.350 INFO:tasks.workunit.client.1.smithi067.stdout:4/443: creat d8/d9/d3d/f93 x:0 0 0 2024-04-12T05:37:25.356 INFO:tasks.workunit.client.1.smithi067.stdout:4/444: link d8/da/d8b/l4a d8/da/d1d/d15/l94 0 2024-04-12T05:37:25.358 INFO:tasks.workunit.client.1.smithi067.stdout:5/769: dread d0/d6/d40/f90 [0,4194304] 0 2024-04-12T05:37:25.358 INFO:tasks.workunit.client.1.smithi067.stdout:4/445: mknod d8/da/d1d/d27/d91/c95 0 2024-04-12T05:37:25.359 INFO:tasks.workunit.client.1.smithi067.stdout:5/770: dread - d0/d3a/da8/fc9 zero size 2024-04-12T05:37:25.360 INFO:tasks.workunit.client.1.smithi067.stdout:4/446: mkdir d8/da/d1d/d15/d36/d5d/d66/d96 0 2024-04-12T05:37:25.361 INFO:tasks.workunit.client.1.smithi067.stdout:5/771: creat d0/d6/d40/d4b/d38/d8e/d8f/db9/dbd/fff x:0 0 0 2024-04-12T05:37:25.388 INFO:tasks.workunit.client.1.smithi067.stdout:2/600: sync 2024-04-12T05:37:25.388 INFO:tasks.workunit.client.1.smithi067.stdout:5/772: truncate d0/d3a/f7e 2137521 0 2024-04-12T05:37:25.388 INFO:tasks.workunit.client.1.smithi067.stdout:4/447: creat d8/d9/d3d/d64/f97 x:0 0 0 2024-04-12T05:37:25.389 INFO:tasks.workunit.client.1.smithi067.stdout:2/601: readlink d5/d43/l38 0 2024-04-12T05:37:25.391 INFO:tasks.workunit.client.1.smithi067.stdout:2/602: write d5/da/de/d14/d34/fb3 [710701,101683] 0 2024-04-12T05:37:25.393 INFO:tasks.workunit.client.1.smithi067.stdout:5/773: rename d0/da/fce to d0/d3a/d2c/f100 0 2024-04-12T05:37:25.396 INFO:tasks.workunit.client.1.smithi067.stdout:2/603: dread d5/da/d8c/faf [0,4194304] 0 2024-04-12T05:37:25.396 INFO:tasks.workunit.client.1.smithi067.stdout:4/448: dwrite d8/d9/f67 [0,4194304] 0 2024-04-12T05:37:25.400 INFO:tasks.workunit.client.1.smithi067.stdout:2/604: unlink d5/f9c 0 2024-04-12T05:37:25.402 INFO:tasks.workunit.client.1.smithi067.stdout:2/605: read d5/da/de/d14/f63 [2179297,25722] 0 2024-04-12T05:37:25.403 INFO:tasks.workunit.client.1.smithi067.stdout:5/774: link d0/lf6 d0/d6/d40/d4b/d38/d8e/d8f/l101 0 2024-04-12T05:37:25.405 INFO:tasks.workunit.client.1.smithi067.stdout:2/606: mknod d5/da/de/d14/cc1 0 2024-04-12T05:37:25.405 INFO:tasks.workunit.client.1.smithi067.stdout:2/607: chown d5/da/de 25 1 2024-04-12T05:37:25.406 INFO:tasks.workunit.client.1.smithi067.stdout:5/775: creat d0/d3a/d2c/dc6/ddd/f102 x:0 0 0 2024-04-12T05:37:25.407 INFO:tasks.workunit.client.1.smithi067.stdout:4/449: dread d8/d9/f42 [0,4194304] 0 2024-04-12T05:37:25.408 INFO:tasks.workunit.client.1.smithi067.stdout:2/608: mkdir d5/da/d8c/dc2 0 2024-04-12T05:37:25.411 INFO:tasks.workunit.client.1.smithi067.stdout:4/450: symlink d8/da/d1d/d15/d58/l98 0 2024-04-12T05:37:25.411 INFO:tasks.workunit.client.1.smithi067.stdout:5/776: getdents d0/d3a/d2c/dc6 0 2024-04-12T05:37:25.412 INFO:tasks.workunit.client.1.smithi067.stdout:2/609: getdents d5/da/de/d14/d34/d4c/dbc 0 2024-04-12T05:37:25.413 INFO:tasks.workunit.client.1.smithi067.stdout:5/777: write d0/d6/d33/f9f [2129058,35048] 0 2024-04-12T05:37:25.415 INFO:tasks.workunit.client.1.smithi067.stdout:5/778: fsync d0/da/d41/f7a 0 2024-04-12T05:37:25.416 INFO:tasks.workunit.client.1.smithi067.stdout:2/610: mknod d5/da/de/d14/d46/d90/cc3 0 2024-04-12T05:37:25.417 INFO:tasks.workunit.client.1.smithi067.stdout:4/451: dread d8/da/d8b/f57 [4194304,4194304] 0 2024-04-12T05:37:25.418 INFO:tasks.workunit.client.1.smithi067.stdout:4/452: chown d8/l22 648915 1 2024-04-12T05:37:25.419 INFO:tasks.workunit.client.1.smithi067.stdout:5/779: unlink d0/da/d3f/d82/fa6 0 2024-04-12T05:37:25.420 INFO:tasks.workunit.client.1.smithi067.stdout:5/780: stat d0/d3a/dab/ld8 0 2024-04-12T05:37:25.421 INFO:tasks.workunit.client.1.smithi067.stdout:4/453: mknod d8/d39/c99 0 2024-04-12T05:37:25.425 INFO:tasks.workunit.client.1.smithi067.stdout:4/454: creat d8/da/d1d/d15/d36/d5d/d66/d71/f9a x:0 0 0 2024-04-12T05:37:25.428 INFO:tasks.workunit.client.1.smithi067.stdout:5/781: dwrite d0/d6/d40/f90 [0,4194304] 0 2024-04-12T05:37:25.433 INFO:tasks.workunit.client.1.smithi067.stdout:4/455: dread d8/f2b [0,4194304] 0 2024-04-12T05:37:25.439 INFO:tasks.workunit.client.1.smithi067.stdout:4/456: rename d8/da/d1d/d15/d36/f89 to d8/da/d1d/d15/d58/f9b 0 2024-04-12T05:37:25.440 INFO:tasks.workunit.client.1.smithi067.stdout:4/457: fdatasync d8/d9/d3d/f4d 0 2024-04-12T05:37:25.445 INFO:tasks.workunit.client.1.smithi067.stdout:5/782: dread d0/da/d41/f6b [0,4194304] 0 2024-04-12T05:37:25.447 INFO:tasks.workunit.client.1.smithi067.stdout:5/783: fsync d0/d3a/d2c/fe9 0 2024-04-12T05:37:25.448 INFO:tasks.workunit.client.1.smithi067.stdout:2/611: dwrite d5/f12 [0,4194304] 0 2024-04-12T05:37:25.456 INFO:tasks.workunit.client.1.smithi067.stdout:5/784: rmdir d0/d6/d40/d4b/d38/d8e/d8f/d99 39 2024-04-12T05:37:25.457 INFO:tasks.workunit.client.1.smithi067.stdout:4/458: dwrite d8/da/d1d/d27/f56 [0,4194304] 0 2024-04-12T05:37:25.459 INFO:tasks.workunit.client.1.smithi067.stdout:5/785: fsync d0/d6/d40/d4b/d38/d8e/d8f/db9/dbd/ffd 0 2024-04-12T05:37:25.463 INFO:tasks.workunit.client.1.smithi067.stdout:5/786: fsync d0/d6/d33/f9f 0 2024-04-12T05:37:25.463 INFO:tasks.workunit.client.1.smithi067.stdout:4/459: chown d8/da/d1d/d15/l94 31 1 2024-04-12T05:37:25.467 INFO:tasks.workunit.client.1.smithi067.stdout:4/460: creat d8/da/d1d/d15/d36/d5d/d66/f9c x:0 0 0 2024-04-12T05:37:25.467 INFO:tasks.workunit.client.1.smithi067.stdout:4/461: chown d8/da/d1d/f85 2335 1 2024-04-12T05:37:25.475 INFO:tasks.workunit.client.1.smithi067.stdout:4/462: rmdir d8/da/d1d/d15/d36/d5d 39 2024-04-12T05:37:25.477 INFO:tasks.workunit.client.1.smithi067.stdout:4/463: fdatasync d8/f33 0 2024-04-12T05:37:25.480 INFO:tasks.workunit.client.1.smithi067.stdout:4/464: unlink d8/d9/c83 0 2024-04-12T05:37:25.483 INFO:tasks.workunit.client.1.smithi067.stdout:4/465: readlink d8/da/d1d/d15/d36/d5d/d66/d6a/l70 0 2024-04-12T05:37:25.489 INFO:tasks.workunit.client.1.smithi067.stdout:4/466: mknod d8/d9/d3d/c9d 0 2024-04-12T05:37:25.489 INFO:tasks.workunit.client.1.smithi067.stdout:4/467: chown d8/da/d1d/f85 2 1 2024-04-12T05:37:25.496 INFO:tasks.workunit.client.1.smithi067.stdout:4/468: dwrite d8/d39/f3b [4194304,4194304] 0 2024-04-12T05:37:25.501 INFO:tasks.workunit.client.1.smithi067.stdout:4/469: rename d8/da/d1d/d15/d36/d5d/d66/d7b/f87 to d8/da/d1d/d15/d75/f9e 0 2024-04-12T05:37:25.506 INFO:tasks.workunit.client.1.smithi067.stdout:4/470: rename d8/da/d1d/d15/d1c to d8/da/d9f 0 2024-04-12T05:37:25.508 INFO:tasks.workunit.client.1.smithi067.stdout:4/471: mkdir d8/da/d1d/d15/d36/d88/da0 0 2024-04-12T05:37:25.509 INFO:tasks.workunit.client.1.smithi067.stdout:4/472: chown d8/da/d1d/d15/d36/d5d/d66/d6a 12647 1 2024-04-12T05:37:25.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:25 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:37:25.581 INFO:tasks.workunit.client.1.smithi067.stdout:6/558: sync 2024-04-12T05:37:25.581 INFO:tasks.workunit.client.1.smithi067.stdout:3/399: sync 2024-04-12T05:37:25.581 INFO:tasks.workunit.client.1.smithi067.stdout:8/467: sync 2024-04-12T05:37:25.586 INFO:tasks.workunit.client.1.smithi067.stdout:6/559: rename dd/d21/d23 to dd/d11/d27/d4a/d79/dc3 0 2024-04-12T05:37:25.587 INFO:tasks.workunit.client.1.smithi067.stdout:8/468: rename d6/d8/d18/d63 to d6/d8/d18/d63/d67/d69/d8c 22 2024-04-12T05:37:25.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:25 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:37:25.592 INFO:tasks.workunit.client.1.smithi067.stdout:3/400: dwrite dc/d1d/d28/d3e/f4a [0,4194304] 0 2024-04-12T05:37:25.592 INFO:tasks.workunit.client.1.smithi067.stdout:8/469: creat d6/d8/d8b/f8d x:0 0 0 2024-04-12T05:37:25.597 INFO:tasks.workunit.client.1.smithi067.stdout:8/470: getdents d6/d8/d18/df 0 2024-04-12T05:37:25.607 INFO:tasks.workunit.client.1.smithi067.stdout:3/401: dread dc/d1d/d28/f39 [0,4194304] 0 2024-04-12T05:37:25.607 INFO:tasks.workunit.client.1.smithi067.stdout:8/471: rename d6/d8/d18/l76 to d6/d8/d18/d63/l8e 0 2024-04-12T05:37:25.607 INFO:tasks.workunit.client.1.smithi067.stdout:3/402: dread - dc/df/d27/f2d zero size 2024-04-12T05:37:25.610 INFO:tasks.workunit.client.1.smithi067.stdout:8/472: rename d6/d8/d18/l31 to d6/d8/d18/d63/d67/d69/l8f 0 2024-04-12T05:37:25.612 INFO:tasks.workunit.client.1.smithi067.stdout:3/403: getdents dc/d50 0 2024-04-12T05:37:25.612 INFO:tasks.workunit.client.1.smithi067.stdout:8/473: write d6/f2c [8547762,39079] 0 2024-04-12T05:37:25.619 INFO:tasks.workunit.client.1.smithi067.stdout:8/474: rename d6/d8/d18/d63/d67/d69/f5a to d6/d8/d18/df/f90 0 2024-04-12T05:37:25.619 INFO:tasks.workunit.client.1.smithi067.stdout:3/404: link dc/d1d/d26/d52/c7b dc/df/d27/d6b/c81 0 2024-04-12T05:37:25.621 INFO:tasks.workunit.client.1.smithi067.stdout:6/560: dread dd/f89 [0,4194304] 0 2024-04-12T05:37:25.624 INFO:tasks.workunit.client.1.smithi067.stdout:3/405: rmdir dc/d1d/d28/d3e/d74 0 2024-04-12T05:37:25.625 INFO:tasks.workunit.client.1.smithi067.stdout:6/561: mknod dd/d21/d64/d6f/da7/cc4 0 2024-04-12T05:37:25.627 INFO:tasks.workunit.client.1.smithi067.stdout:3/406: creat dc/df/f82 x:0 0 0 2024-04-12T05:37:25.631 INFO:tasks.workunit.client.1.smithi067.stdout:8/475: dwrite d6/d8/d18/d63/d67/d69/f79 [4194304,4194304] 0 2024-04-12T05:37:25.634 INFO:tasks.workunit.client.1.smithi067.stdout:3/407: link f9 dc/d1d/d28/d3e/d69/f83 0 2024-04-12T05:37:25.636 INFO:tasks.workunit.client.1.smithi067.stdout:6/562: dwrite f9 [0,4194304] 0 2024-04-12T05:37:25.641 INFO:tasks.workunit.client.1.smithi067.stdout:8/476: creat d6/d8/d18/d63/d67/d69/d5d/f91 x:0 0 0 2024-04-12T05:37:25.642 INFO:tasks.workunit.client.1.smithi067.stdout:6/563: write dd/d21/d66/f83 [1121663,79959] 0 2024-04-12T05:37:25.644 INFO:tasks.workunit.client.1.smithi067.stdout:8/477: stat d6/d8/d18/d36/l43 0 2024-04-12T05:37:25.649 INFO:tasks.workunit.client.1.smithi067.stdout:8/478: fdatasync d6/d8/d18/df/f5f 0 2024-04-12T05:37:25.649 INFO:tasks.workunit.client.1.smithi067.stdout:3/408: dwrite dc/df/d27/d6b/f77 [0,4194304] 0 2024-04-12T05:37:25.651 INFO:tasks.workunit.client.1.smithi067.stdout:8/479: write d6/d8/d18/f5e [819194,126398] 0 2024-04-12T05:37:25.652 INFO:tasks.workunit.client.1.smithi067.stdout:8/480: chown d6/d86/d4f 185 1 2024-04-12T05:37:25.653 INFO:tasks.workunit.client.1.smithi067.stdout:3/409: mkdir dc/d1d/d28/d3e/d84 0 2024-04-12T05:37:25.656 INFO:tasks.workunit.client.1.smithi067.stdout:3/410: write dc/d1d/d28/d3e/f63 [980727,99579] 0 2024-04-12T05:37:25.661 INFO:tasks.workunit.client.1.smithi067.stdout:3/411: creat dc/d1d/d28/d3e/d84/f85 x:0 0 0 2024-04-12T05:37:25.663 INFO:tasks.workunit.client.1.smithi067.stdout:8/481: dwrite d6/d8/d6e/f84 [0,4194304] 0 2024-04-12T05:37:25.664 INFO:tasks.workunit.client.1.smithi067.stdout:6/564: dread dd/d21/d64/d6f/d90/d98/fb9 [0,4194304] 0 2024-04-12T05:37:25.668 INFO:tasks.workunit.client.1.smithi067.stdout:3/412: link dc/d50/l54 dc/df/l86 0 2024-04-12T05:37:25.670 INFO:tasks.workunit.client.1.smithi067.stdout:8/482: link d6/d8/d18/d36/f70 d6/d8/f92 0 2024-04-12T05:37:25.673 INFO:tasks.workunit.client.1.smithi067.stdout:3/413: rename dc/df/f3d to dc/f87 0 2024-04-12T05:37:25.676 INFO:tasks.workunit.client.1.smithi067.stdout:8/483: creat d6/d8/d18/df/f93 x:0 0 0 2024-04-12T05:37:25.677 INFO:tasks.workunit.client.1.smithi067.stdout:3/414: creat dc/d1d/d4d/f88 x:0 0 0 2024-04-12T05:37:25.678 INFO:tasks.workunit.client.1.smithi067.stdout:8/484: chown d6/d8/d18/l85 30 1 2024-04-12T05:37:25.678 INFO:tasks.workunit.client.1.smithi067.stdout:6/565: dread f0 [4194304,4194304] 0 2024-04-12T05:37:25.679 INFO:tasks.workunit.client.1.smithi067.stdout:8/485: chown d6/d8/f72 48355 1 2024-04-12T05:37:25.680 INFO:tasks.workunit.client.1.smithi067.stdout:8/486: write d6/f2c [1958742,42831] 0 2024-04-12T05:37:25.681 INFO:tasks.workunit.client.1.smithi067.stdout:6/566: symlink dd/d21/d64/d88/d7b/lc5 0 2024-04-12T05:37:25.683 INFO:tasks.workunit.client.1.smithi067.stdout:3/415: dread dc/d1d/d28/d3e/f4a [0,4194304] 0 2024-04-12T05:37:25.684 INFO:tasks.workunit.client.1.smithi067.stdout:6/567: getdents dd/d11/dc1 0 2024-04-12T05:37:25.686 INFO:tasks.workunit.client.1.smithi067.stdout:6/568: chown dd/d21/d64/d88/d7b/lc5 5217 1 2024-04-12T05:37:25.686 INFO:tasks.workunit.client.1.smithi067.stdout:3/416: mkdir dc/d1d/d89 0 2024-04-12T05:37:25.688 INFO:tasks.workunit.client.1.smithi067.stdout:6/569: chown dd/d21/d64/d88/l97 14598206 1 2024-04-12T05:37:25.688 INFO:tasks.workunit.client.1.smithi067.stdout:6/570: readlink dd/d21/d3f/lbb 0 2024-04-12T05:37:25.690 INFO:tasks.workunit.client.1.smithi067.stdout:6/571: write dd/d11/fac [830824,59790] 0 2024-04-12T05:37:25.692 INFO:tasks.workunit.client.1.smithi067.stdout:3/417: write dc/df/d27/f2d [50642,25982] 0 2024-04-12T05:37:25.699 INFO:tasks.workunit.client.1.smithi067.stdout:6/572: dwrite dd/d11/d27/d4a/f8c [0,4194304] 0 2024-04-12T05:37:25.707 INFO:tasks.workunit.client.1.smithi067.stdout:6/573: truncate dd/d11/d27/d4a/d51/f57 266705 0 2024-04-12T05:37:25.709 INFO:tasks.workunit.client.1.smithi067.stdout:3/418: dread dc/f18 [0,4194304] 0 2024-04-12T05:37:25.710 INFO:tasks.workunit.client.1.smithi067.stdout:6/574: symlink dd/d21/d64/d6f/d90/lc6 0 2024-04-12T05:37:25.716 INFO:tasks.workunit.client.1.smithi067.stdout:3/419: dread dc/df/d27/d6b/f77 [0,4194304] 0 2024-04-12T05:37:25.716 INFO:tasks.workunit.client.1.smithi067.stdout:6/575: dread dd/d21/d66/f83 [0,4194304] 0 2024-04-12T05:37:25.718 INFO:tasks.workunit.client.1.smithi067.stdout:3/420: dread dc/d1d/f24 [0,4194304] 0 2024-04-12T05:37:25.719 INFO:tasks.workunit.client.1.smithi067.stdout:3/421: truncate f1 5091653 0 2024-04-12T05:37:25.721 INFO:tasks.workunit.client.1.smithi067.stdout:7/544: sync 2024-04-12T05:37:25.723 INFO:tasks.workunit.client.1.smithi067.stdout:3/422: rename dc/d1d/d28/d3e/d69/d73 to dc/d1d/d28/d3e/d8a 0 2024-04-12T05:37:25.726 INFO:tasks.workunit.client.1.smithi067.stdout:3/423: rmdir dc/d1d/d28 39 2024-04-12T05:37:25.726 INFO:tasks.workunit.client.1.smithi067.stdout:6/576: dwrite dd/d11/d27/d4a/d79/dc3/d80/d93/dba/fc2 [0,4194304] 0 2024-04-12T05:37:25.735 INFO:tasks.workunit.client.1.smithi067.stdout:3/424: dread dc/df/d27/d6b/f77 [0,4194304] 0 2024-04-12T05:37:25.735 INFO:tasks.workunit.client.1.smithi067.stdout:7/545: dwrite d0/f5e [0,4194304] 0 2024-04-12T05:37:25.742 INFO:tasks.workunit.client.1.smithi067.stdout:6/577: dwrite dd/f20 [0,4194304] 0 2024-04-12T05:37:25.747 INFO:tasks.workunit.client.1.smithi067.stdout:6/578: truncate dd/d11/d27/d4a/d79/dc3/d80/d93/dba/fc2 5127714 0 2024-04-12T05:37:25.748 INFO:tasks.workunit.client.1.smithi067.stdout:3/425: creat dc/d1d/d28/d3e/d53/d78/f8b x:0 0 0 2024-04-12T05:37:25.748 INFO:tasks.workunit.client.1.smithi067.stdout:7/546: dread d0/d23/d80/da0/db4/d5/fe [0,4194304] 0 2024-04-12T05:37:25.750 INFO:tasks.workunit.client.1.smithi067.stdout:7/547: write d0/d67/d57/fa7 [1003235,10174] 0 2024-04-12T05:37:25.751 INFO:tasks.workunit.client.1.smithi067.stdout:6/579: mkdir dd/d11/d27/d4a/dc7 0 2024-04-12T05:37:25.753 INFO:tasks.workunit.client.1.smithi067.stdout:6/580: fsync dd/d11/fb7 0 2024-04-12T05:37:25.761 INFO:tasks.workunit.client.1.smithi067.stdout:7/548: dwrite d0/d23/d80/da0/db4/dd/d42/d7e/f1f [0,4194304] 0 2024-04-12T05:37:25.765 INFO:tasks.workunit.client.1.smithi067.stdout:6/581: link dd/d21/d66/c74 dd/d21/d64/d88/cc8 0 2024-04-12T05:37:25.767 INFO:tasks.workunit.client.1.smithi067.stdout:7/549: mknod d0/d23/d80/da0/db4/dd/da2/cb6 0 2024-04-12T05:37:25.769 INFO:tasks.workunit.client.1.smithi067.stdout:7/550: dread - d0/d23/d80/f9c zero size 2024-04-12T05:37:25.773 INFO:tasks.workunit.client.1.smithi067.stdout:3/426: dwrite dc/d1d/d26/f72 [0,4194304] 0 2024-04-12T05:37:25.776 INFO:tasks.workunit.client.1.smithi067.stdout:6/582: dwrite dd/d11/d27/d4a/d79/f76 [0,4194304] 0 2024-04-12T05:37:25.777 INFO:tasks.workunit.client.1.smithi067.stdout:3/427: write dc/d1d/d28/d3e/f6d [2996313,108336] 0 2024-04-12T05:37:25.783 INFO:tasks.workunit.client.1.smithi067.stdout:6/583: unlink dd/d21/d64/d88/d7b/ca0 0 2024-04-12T05:37:25.785 INFO:tasks.workunit.client.1.smithi067.stdout:7/551: dread d0/d23/d80/da0/db4/dd/da2/f92 [0,4194304] 0 2024-04-12T05:37:25.786 INFO:tasks.workunit.client.1.smithi067.stdout:6/584: write dd/d11/d27/d4a/d79/dc3/d80/d93/dba/fc2 [4699089,55868] 0 2024-04-12T05:37:25.787 INFO:tasks.workunit.client.1.smithi067.stdout:3/428: creat dc/f8c x:0 0 0 2024-04-12T05:37:25.789 INFO:tasks.workunit.client.1.smithi067.stdout:7/552: read d0/d23/d80/f99 [3367461,61551] 0 2024-04-12T05:37:25.791 INFO:tasks.workunit.client.1.smithi067.stdout:3/429: write dc/d1d/d28/d3e/d53/d78/f8b [872190,109697] 0 2024-04-12T05:37:25.794 INFO:tasks.workunit.client.1.smithi067.stdout:6/585: dread dd/d11/d27/d4a/d79/f68 [0,4194304] 0 2024-04-12T05:37:25.794 INFO:tasks.workunit.client.1.smithi067.stdout:7/553: write d0/d23/d80/da0/db4/d5/f50 [100929,78475] 0 2024-04-12T05:37:25.796 INFO:tasks.workunit.client.1.smithi067.stdout:3/430: fdatasync dc/d1d/d28/d3e/d84/f85 0 2024-04-12T05:37:25.796 INFO:tasks.workunit.client.1.smithi067.stdout:3/431: chown f1 27680 1 2024-04-12T05:37:25.798 INFO:tasks.workunit.client.1.smithi067.stdout:7/554: write d0/d67/d57/d85/f8e [1333360,116397] 0 2024-04-12T05:37:25.800 INFO:tasks.workunit.client.1.smithi067.stdout:3/432: mkdir dc/df/d8d 0 2024-04-12T05:37:25.801 INFO:tasks.workunit.client.1.smithi067.stdout:6/586: creat dd/d21/d64/db5/fc9 x:0 0 0 2024-04-12T05:37:25.801 INFO:tasks.workunit.client.1.smithi067.stdout:7/555: mknod d0/d23/d80/da0/db4/d5/d1d/d7d/cb7 0 2024-04-12T05:37:25.802 INFO:tasks.workunit.client.1.smithi067.stdout:3/433: chown dc/df/d27/l57 260736025 1 2024-04-12T05:37:25.803 INFO:tasks.workunit.client.1.smithi067.stdout:6/587: fdatasync dd/d11/f1e 0 2024-04-12T05:37:25.807 INFO:tasks.workunit.client.1.smithi067.stdout:6/588: creat dd/d11/d27/fca x:0 0 0 2024-04-12T05:37:25.815 INFO:tasks.workunit.client.1.smithi067.stdout:7/556: getdents d0/d67/d57/d85 0 2024-04-12T05:37:25.815 INFO:tasks.workunit.client.1.smithi067.stdout:6/589: creat dd/d21/fcb x:0 0 0 2024-04-12T05:37:25.816 INFO:tasks.workunit.client.1.smithi067.stdout:7/557: chown d0/d23/d80/da0/db4/dd 9967736 1 2024-04-12T05:37:25.816 INFO:tasks.workunit.client.1.smithi067.stdout:3/434: write dc/df/f2c [1902121,67946] 0 2024-04-12T05:37:25.816 INFO:tasks.workunit.client.1.smithi067.stdout:6/590: chown dd/d21/d3f/l71 330171 1 2024-04-12T05:37:25.817 INFO:tasks.workunit.client.1.smithi067.stdout:7/558: fdatasync d0/d23/d80/da0/db4/d5/d1d/f40 0 2024-04-12T05:37:25.817 INFO:tasks.workunit.client.1.smithi067.stdout:3/435: creat dc/f8e x:0 0 0 2024-04-12T05:37:25.820 INFO:tasks.workunit.client.1.smithi067.stdout:3/436: creat dc/f8f x:0 0 0 2024-04-12T05:37:25.823 INFO:tasks.workunit.client.1.smithi067.stdout:6/591: dread dd/d21/d64/d88/f32 [0,4194304] 0 2024-04-12T05:37:25.826 INFO:tasks.workunit.client.1.smithi067.stdout:7/559: dwrite d0/d23/d80/da0/db4/dd/d79/f6a [0,4194304] 0 2024-04-12T05:37:25.828 INFO:tasks.workunit.client.1.smithi067.stdout:6/592: write dd/d21/d64/d88/f96 [1141971,14180] 0 2024-04-12T05:37:25.830 INFO:tasks.workunit.client.1.smithi067.stdout:7/560: write d0/d23/d45/f26 [869983,73853] 0 2024-04-12T05:37:25.837 INFO:tasks.workunit.client.1.smithi067.stdout:3/437: dread dc/d1d/d26/f46 [0,4194304] 0 2024-04-12T05:37:25.839 INFO:tasks.workunit.client.1.smithi067.stdout:6/593: truncate fb 2844378 0 2024-04-12T05:37:25.840 INFO:tasks.workunit.client.1.smithi067.stdout:3/438: mkdir dc/d1d/d26/d52/d90 0 2024-04-12T05:37:25.843 INFO:tasks.workunit.client.1.smithi067.stdout:3/439: mknod dc/d1d/d28/d3e/d53/d78/c91 0 2024-04-12T05:37:25.843 INFO:tasks.workunit.client.1.smithi067.stdout:7/561: write d0/d23/d80/da0/db4/dd/da2/f92 [1836336,128482] 0 2024-04-12T05:37:25.846 INFO:tasks.workunit.client.1.smithi067.stdout:6/594: rename dd/d11/d27/d4a/d79/dc3/d80 to dd/d21/d64/d88/d7b/dcc 0 2024-04-12T05:37:25.847 INFO:tasks.workunit.client.1.smithi067.stdout:3/440: read dc/d1d/f22 [118507,47047] 0 2024-04-12T05:37:25.848 INFO:tasks.workunit.client.1.smithi067.stdout:7/562: mknod d0/d23/d80/da0/db4/dd/da2/d51/cb8 0 2024-04-12T05:37:25.849 INFO:tasks.workunit.client.1.smithi067.stdout:3/441: write dc/f8c [444685,17214] 0 2024-04-12T05:37:25.849 INFO:tasks.workunit.client.1.smithi067.stdout:6/595: truncate dd/d11/d27/d4a/d8b/fbd 111558 0 2024-04-12T05:37:25.850 INFO:tasks.workunit.client.1.smithi067.stdout:7/563: fsync d0/d23/d80/da0/db4/d5/d1d/d7d/f84 0 2024-04-12T05:37:25.853 INFO:tasks.workunit.client.1.smithi067.stdout:7/564: chown d0/d67/d57/d85/l98 1270 1 2024-04-12T05:37:25.853 INFO:tasks.workunit.client.1.smithi067.stdout:3/442: symlink dc/d1d/d28/d3e/d69/l92 0 2024-04-12T05:37:25.854 INFO:tasks.workunit.client.1.smithi067.stdout:7/565: fsync d0/d67/d57/d85/fad 0 2024-04-12T05:37:25.861 INFO:tasks.workunit.client.1.smithi067.stdout:3/443: getdents dc/d1d/d28/d3e/d69 0 2024-04-12T05:37:25.865 INFO:tasks.workunit.client.1.smithi067.stdout:7/566: dwrite d0/d23/d45/f73 [0,4194304] 0 2024-04-12T05:37:25.866 INFO:tasks.workunit.client.1.smithi067.stdout:6/596: dwrite dd/d11/f5b [0,4194304] 0 2024-04-12T05:37:25.870 INFO:tasks.workunit.client.1.smithi067.stdout:7/567: chown d0/d23/d80/da0/db4/dd/d42/d7e/d34/c5f 181325 1 2024-04-12T05:37:25.873 INFO:tasks.workunit.client.1.smithi067.stdout:3/444: dwrite dc/f8f [0,4194304] 0 2024-04-12T05:37:25.880 INFO:tasks.workunit.client.1.smithi067.stdout:3/445: link dc/d1d/d28/f64 dc/df/d8d/f93 0 2024-04-12T05:37:25.881 INFO:tasks.workunit.client.1.smithi067.stdout:6/597: dwrite dd/f3b [0,4194304] 0 2024-04-12T05:37:25.886 INFO:tasks.workunit.client.1.smithi067.stdout:6/598: mkdir dd/d21/d64/d88/d7b/dcd 0 2024-04-12T05:37:25.887 INFO:tasks.workunit.client.1.smithi067.stdout:6/599: write dd/d11/d27/f46 [1332262,8638] 0 2024-04-12T05:37:25.888 INFO:tasks.workunit.client.1.smithi067.stdout:3/446: rmdir dc/d1d/d26/d52/d90 0 2024-04-12T05:37:25.895 INFO:tasks.workunit.client.1.smithi067.stdout:6/600: link dd/d11/d27/d4a/d51/f57 dd/d21/d64/fce 0 2024-04-12T05:37:25.895 INFO:tasks.workunit.client.1.smithi067.stdout:6/601: chown dd/d11/d27/fca 4392026 1 2024-04-12T05:37:25.896 INFO:tasks.workunit.client.1.smithi067.stdout:6/602: truncate dd/fa6 459576 0 2024-04-12T05:37:25.910 INFO:tasks.workunit.client.1.smithi067.stdout:6/603: dread dd/d11/d27/d4a/d79/dc3/f3c [0,4194304] 0 2024-04-12T05:37:25.914 INFO:tasks.workunit.client.1.smithi067.stdout:6/604: write dd/d21/f4c [594772,105346] 0 2024-04-12T05:37:25.916 INFO:tasks.workunit.client.1.smithi067.stdout:6/605: mkdir dd/d21/d64/dcf 0 2024-04-12T05:37:25.917 INFO:tasks.workunit.client.1.smithi067.stdout:6/606: write dd/d11/f5d [2968969,4949] 0 2024-04-12T05:37:25.921 INFO:tasks.workunit.client.1.smithi067.stdout:6/607: rename dd/d21/f9c to dd/d11/d27/d4a/d51/fd0 0 2024-04-12T05:37:25.922 INFO:tasks.workunit.client.1.smithi067.stdout:6/608: truncate dd/d21/d64/d88/f96 4626395 0 2024-04-12T05:37:25.924 INFO:tasks.workunit.client.1.smithi067.stdout:6/609: symlink dd/d11/d27/d4a/d8b/ld1 0 2024-04-12T05:37:25.930 INFO:tasks.workunit.client.1.smithi067.stdout:6/610: dwrite dd/d11/d27/d4a/d79/dc3/f43 [0,4194304] 0 2024-04-12T05:37:25.934 INFO:tasks.workunit.client.1.smithi067.stdout:6/611: rename dd/d21/d66 to dd/d21/d64/d88/d7b/dd2 0 2024-04-12T05:37:25.937 INFO:tasks.workunit.client.1.smithi067.stdout:6/612: mknod dd/d21/d64/d88/d7b/dcc/d93/dba/cd3 0 2024-04-12T05:37:25.941 INFO:tasks.workunit.client.1.smithi067.stdout:6/613: rename dd/d21/f4c to dd/d11/d27/d4a/d79/dc3/fd4 0 2024-04-12T05:37:26.152 INFO:tasks.workunit.client.1.smithi067.stdout:1/527: sync 2024-04-12T05:37:26.152 INFO:tasks.workunit.client.1.smithi067.stdout:9/508: sync 2024-04-12T05:37:26.152 INFO:tasks.workunit.client.1.smithi067.stdout:0/617: sync 2024-04-12T05:37:26.153 INFO:tasks.workunit.client.1.smithi067.stdout:9/509: write d1/d66/f6e [4860622,39494] 0 2024-04-12T05:37:26.158 INFO:tasks.workunit.client.1.smithi067.stdout:9/510: rename d1/cc to d1/d28/d21/d3f/d4f/c9d 0 2024-04-12T05:37:26.159 INFO:tasks.workunit.client.1.smithi067.stdout:1/528: link d4/d9/d12/f53 d4/d9/d4c/fa7 0 2024-04-12T05:37:26.160 INFO:tasks.workunit.client.1.smithi067.stdout:0/618: truncate dd/d12/d14/d49/fa6 4589615 0 2024-04-12T05:37:26.161 INFO:tasks.workunit.client.1.smithi067.stdout:0/619: chown f1 25 1 2024-04-12T05:37:26.162 INFO:tasks.workunit.client.1.smithi067.stdout:1/529: mknod d4/d9/d3a/ca8 0 2024-04-12T05:37:26.165 INFO:tasks.workunit.client.1.smithi067.stdout:9/511: truncate d1/f6 3118500 0 2024-04-12T05:37:26.165 INFO:tasks.workunit.client.1.smithi067.stdout:9/512: chown d1/d12 579 1 2024-04-12T05:37:26.167 INFO:tasks.workunit.client.1.smithi067.stdout:1/530: symlink d4/d9/la9 0 2024-04-12T05:37:26.169 INFO:tasks.workunit.client.1.smithi067.stdout:0/620: rename dd/d9c/d5f/d78/cae to dd/cc6 0 2024-04-12T05:37:26.170 INFO:tasks.workunit.client.1.smithi067.stdout:0/621: read dd/d6f/fba [379090,23300] 0 2024-04-12T05:37:26.171 INFO:tasks.workunit.client.1.smithi067.stdout:1/531: chown d4/d9/d12/f53 10 1 2024-04-12T05:37:26.172 INFO:tasks.workunit.client.1.smithi067.stdout:0/622: chown dd/d6f/d72/f82 7565 1 2024-04-12T05:37:26.174 INFO:tasks.workunit.client.1.smithi067.stdout:0/623: dread - dd/d12/d14/f57 zero size 2024-04-12T05:37:26.176 INFO:tasks.workunit.client.1.smithi067.stdout:1/532: creat d4/d9/d3a/d51/d55/d9a/faa x:0 0 0 2024-04-12T05:37:26.178 INFO:tasks.workunit.client.1.smithi067.stdout:0/624: rmdir dd/d9c/d5f/d78 39 2024-04-12T05:37:26.180 INFO:tasks.workunit.client.1.smithi067.stdout:9/513: write d1/f6 [1132549,62175] 0 2024-04-12T05:37:26.183 INFO:tasks.workunit.client.1.smithi067.stdout:9/514: chown d1/d28/d21/l54 13426322 1 2024-04-12T05:37:26.184 INFO:tasks.workunit.client.1.smithi067.stdout:1/533: rename d4/d9/d12/d28 to d4/d3d/dab 0 2024-04-12T05:37:26.187 INFO:tasks.workunit.client.1.smithi067.stdout:9/515: write d1/d13/d34/f70 [583315,67468] 0 2024-04-12T05:37:26.188 INFO:tasks.workunit.client.1.smithi067.stdout:0/625: dwrite dd/d12/fb3 [0,4194304] 0 2024-04-12T05:37:26.189 INFO:tasks.workunit.client.1.smithi067.stdout:9/516: fsync d1/d66/f6e 0 2024-04-12T05:37:26.191 INFO:tasks.workunit.client.1.smithi067.stdout:1/534: write d4/d9/d14/f21 [1276536,20340] 0 2024-04-12T05:37:26.195 INFO:tasks.workunit.client.1.smithi067.stdout:0/626: write dd/d12/d14/d1a/d3d/f7c [1038564,28552] 0 2024-04-12T05:37:26.200 INFO:tasks.workunit.client.1.smithi067.stdout:1/535: read d4/d9/d14/f57 [750322,63059] 0 2024-04-12T05:37:26.203 INFO:tasks.workunit.client.1.smithi067.stdout:9/517: dwrite d1/d28/d27/f71 [0,4194304] 0 2024-04-12T05:37:26.204 INFO:tasks.workunit.client.1.smithi067.stdout:1/536: creat d4/d9/d12/d2c/fac x:0 0 0 2024-04-12T05:37:26.205 INFO:tasks.workunit.client.1.smithi067.stdout:9/518: chown d1/d66 813743 1 2024-04-12T05:37:26.208 INFO:tasks.workunit.client.1.smithi067.stdout:9/519: creat d1/d28/d44/f9e x:0 0 0 2024-04-12T05:37:26.209 INFO:tasks.workunit.client.1.smithi067.stdout:9/520: chown d1/d28/d21/c35 80918 1 2024-04-12T05:37:26.212 INFO:tasks.workunit.client.1.smithi067.stdout:0/627: dread dd/d12/d14/f43 [0,4194304] 0 2024-04-12T05:37:26.213 INFO:tasks.workunit.client.1.smithi067.stdout:0/628: stat dd/d6f/d72/fa3 0 2024-04-12T05:37:26.216 INFO:tasks.workunit.client.1.smithi067.stdout:9/521: fsync d1/d28/d44/f9e 0 2024-04-12T05:37:26.216 INFO:tasks.workunit.client.1.smithi067.stdout:1/537: truncate d4/d9/d12/f2e 576432 0 2024-04-12T05:37:26.216 INFO:tasks.workunit.client.1.smithi067.stdout:9/522: readlink d1/l5 0 2024-04-12T05:37:26.223 INFO:tasks.workunit.client.1.smithi067.stdout:9/523: readlink d1/d28/d23/l51 0 2024-04-12T05:37:26.223 INFO:tasks.workunit.client.1.smithi067.stdout:1/538: write d4/d3d/d7b/f95 [1549162,117060] 0 2024-04-12T05:37:26.223 INFO:tasks.workunit.client.1.smithi067.stdout:0/629: truncate dd/d5b/f84 3052140 0 2024-04-12T05:37:26.223 INFO:tasks.workunit.client.1.smithi067.stdout:9/524: creat d1/d28/d21/d45/f9f x:0 0 0 2024-04-12T05:37:26.224 INFO:tasks.workunit.client.1.smithi067.stdout:1/539: mkdir d4/d9/d3a/d51/d55/dad 0 2024-04-12T05:37:26.225 INFO:tasks.workunit.client.1.smithi067.stdout:0/630: rename dd/d9c/d5f/d78 to dd/d9c/d5f/d78/d83/dc7 22 2024-04-12T05:37:26.226 INFO:tasks.workunit.client.1.smithi067.stdout:9/525: fsync d1/f47 0 2024-04-12T05:37:26.230 INFO:tasks.workunit.client.1.smithi067.stdout:1/540: dread d4/d9/f45 [0,4194304] 0 2024-04-12T05:37:26.234 INFO:tasks.workunit.client.1.smithi067.stdout:1/541: creat d4/d3d/fae x:0 0 0 2024-04-12T05:37:26.235 INFO:tasks.workunit.client.1.smithi067.stdout:0/631: dwrite dd/d6f/d72/f97 [4194304,4194304] 0 2024-04-12T05:37:26.237 INFO:tasks.workunit.client.1.smithi067.stdout:0/632: write dd/dbd/dbe/fc2 [646906,3941] 0 2024-04-12T05:37:26.238 INFO:tasks.workunit.client.1.smithi067.stdout:0/633: dread - dd/d6f/d72/fbb zero size 2024-04-12T05:37:26.239 INFO:tasks.workunit.client.1.smithi067.stdout:1/542: rename d4/d3d/c5e to d4/d9/d3a/d51/d55/caf 0 2024-04-12T05:37:26.246 INFO:tasks.workunit.client.1.smithi067.stdout:1/543: creat d4/d9/d12/d2c/fb0 x:0 0 0 2024-04-12T05:37:26.248 INFO:tasks.workunit.client.1.smithi067.stdout:1/544: stat d4/d3d/dab/f4f 0 2024-04-12T05:37:26.250 INFO:tasks.workunit.client.1.smithi067.stdout:1/545: creat d4/d3d/dab/fb1 x:0 0 0 2024-04-12T05:37:26.253 INFO:tasks.workunit.client.1.smithi067.stdout:0/634: dwrite dd/d12/d14/d49/d9b/fa5 [0,4194304] 0 2024-04-12T05:37:26.256 INFO:tasks.workunit.client.1.smithi067.stdout:0/635: rename dd/d12/d14 to dd/d12/d14/d1a/dc8 22 2024-04-12T05:37:26.259 INFO:tasks.workunit.client.1.smithi067.stdout:0/636: creat dd/d12/d14/fc9 x:0 0 0 2024-04-12T05:37:26.260 INFO:tasks.workunit.client.1.smithi067.stdout:0/637: write dd/d6f/d72/fa3 [1207940,77436] 0 2024-04-12T05:37:26.264 INFO:tasks.workunit.client.1.smithi067.stdout:0/638: creat dd/d12/d14/d1a/fca x:0 0 0 2024-04-12T05:37:26.265 INFO:tasks.workunit.client.1.smithi067.stdout:0/639: dread - dd/d12/d14/d49/fb1 zero size 2024-04-12T05:37:26.267 INFO:tasks.workunit.client.1.smithi067.stdout:0/640: write dd/d9c/f62 [81909,6419] 0 2024-04-12T05:37:26.270 INFO:tasks.workunit.client.1.smithi067.stdout:1/546: write d4/d9/d12/f2e [86335,52589] 0 2024-04-12T05:37:26.272 INFO:tasks.workunit.client.1.smithi067.stdout:1/547: read - d4/d9/d3a/d51/d55/d9a/faa zero size 2024-04-12T05:37:26.272 INFO:tasks.workunit.client.1.smithi067.stdout:1/548: readlink d4/d9/d12/d2c/l7c 0 2024-04-12T05:37:26.275 INFO:tasks.workunit.client.1.smithi067.stdout:1/549: mknod d4/d9/d3a/d51/d55/d9a/cb2 0 2024-04-12T05:37:26.277 INFO:tasks.workunit.client.1.smithi067.stdout:1/550: creat d4/d52/d80/fb3 x:0 0 0 2024-04-12T05:37:26.279 INFO:tasks.workunit.client.1.smithi067.stdout:1/551: chown d4/d9/d12/f53 37346 1 2024-04-12T05:37:26.279 INFO:tasks.workunit.client.1.smithi067.stdout:1/552: stat d4/d9/d12/f2e 0 2024-04-12T05:37:26.281 INFO:tasks.workunit.client.1.smithi067.stdout:1/553: read d4/d9/d12/d59/f64 [1699141,84411] 0 2024-04-12T05:37:26.283 INFO:tasks.workunit.client.1.smithi067.stdout:1/554: creat d4/d9/d4d/fb4 x:0 0 0 2024-04-12T05:37:26.286 INFO:tasks.workunit.client.1.smithi067.stdout:1/555: symlink d4/d9/d3a/d51/d55/d9a/lb5 0 2024-04-12T05:37:26.287 INFO:tasks.workunit.client.1.smithi067.stdout:1/556: truncate d4/d46/f96 577180 0 2024-04-12T05:37:26.290 INFO:tasks.workunit.client.1.smithi067.stdout:1/557: getdents d4/d46 0 2024-04-12T05:37:26.298 INFO:tasks.workunit.client.1.smithi067.stdout:1/558: dwrite d4/d9/f4a [0,4194304] 0 2024-04-12T05:37:26.303 INFO:tasks.workunit.client.1.smithi067.stdout:1/559: dread d4/d9/d14/f2a [0,4194304] 0 2024-04-12T05:37:26.305 INFO:tasks.workunit.client.1.smithi067.stdout:1/560: write d4/d3d/dab/f39 [905119,126229] 0 2024-04-12T05:37:26.307 INFO:tasks.workunit.client.1.smithi067.stdout:1/561: write d4/d9/d4c/f89 [4751875,73210] 0 2024-04-12T05:37:26.324 INFO:tasks.workunit.client.1.smithi067.stdout:1/562: dread d4/d9/d3a/f1e [4194304,4194304] 0 2024-04-12T05:37:26.326 INFO:tasks.workunit.client.1.smithi067.stdout:1/563: read d4/d9/d12/f81 [599865,67568] 0 2024-04-12T05:37:26.327 INFO:tasks.workunit.client.1.smithi067.stdout:1/564: chown d4/d9/d12/d2c/la5 826433854 1 2024-04-12T05:37:26.328 INFO:tasks.workunit.client.1.smithi067.stdout:1/565: chown d4/d9/d12/d2c/l7c 606495316 1 2024-04-12T05:37:26.334 INFO:tasks.workunit.client.1.smithi067.stdout:1/566: dwrite d4/d9/d3a/f1e [4194304,4194304] 0 2024-04-12T05:37:26.337 INFO:tasks.workunit.client.1.smithi067.stdout:1/567: creat d4/d9/d3a/d51/d55/dad/fb6 x:0 0 0 2024-04-12T05:37:26.341 INFO:tasks.workunit.client.1.smithi067.stdout:1/568: mknod d4/d44/cb7 0 2024-04-12T05:37:26.347 INFO:tasks.workunit.client.1.smithi067.stdout:1/569: dwrite d4/d9/d4c/f89 [0,4194304] 0 2024-04-12T05:37:26.352 INFO:tasks.workunit.client.1.smithi067.stdout:1/570: creat d4/d3d/fb8 x:0 0 0 2024-04-12T05:37:26.353 INFO:tasks.workunit.client.1.smithi067.stdout:1/571: chown d4/d9/d3a/d51/d55/d9a/faa 4101 1 2024-04-12T05:37:26.354 INFO:tasks.workunit.client.1.smithi067.stdout:1/572: chown d4/d9/d3a/d51/d55/d9a/cb2 11838 1 2024-04-12T05:37:26.357 INFO:tasks.workunit.client.1.smithi067.stdout:1/573: symlink d4/d9/d4c/lb9 0 2024-04-12T05:37:26.359 INFO:tasks.workunit.client.1.smithi067.stdout:1/574: symlink d4/d9/d3a/d51/d55/lba 0 2024-04-12T05:37:26.368 INFO:tasks.workunit.client.1.smithi067.stdout:1/575: write d4/d9/d12/f16 [182316,61777] 0 2024-04-12T05:37:26.369 INFO:tasks.workunit.client.1.smithi067.stdout:4/473: sync 2024-04-12T05:37:26.369 INFO:tasks.workunit.client.1.smithi067.stdout:5/787: sync 2024-04-12T05:37:26.370 INFO:tasks.workunit.client.1.smithi067.stdout:2/612: sync 2024-04-12T05:37:26.371 INFO:tasks.workunit.client.1.smithi067.stdout:1/576: write d4/d9/f78 [497953,60872] 0 2024-04-12T05:37:26.373 INFO:tasks.workunit.client.1.smithi067.stdout:2/613: readlink d5/da/de/d14/d34/d40/d57/l81 0 2024-04-12T05:37:26.375 INFO:tasks.workunit.client.1.smithi067.stdout:2/614: mkdir d5/da/de/dc4 0 2024-04-12T05:37:26.380 INFO:tasks.workunit.client.1.smithi067.stdout:4/474: dwrite d8/f2b [0,4194304] 0 2024-04-12T05:37:26.384 INFO:tasks.workunit.client.1.smithi067.stdout:2/615: dread d5/da/de/f13 [0,4194304] 0 2024-04-12T05:37:26.385 INFO:tasks.workunit.client.1.smithi067.stdout:4/475: unlink d8/da/d1d/d15/d36/d5d/f6f 0 2024-04-12T05:37:26.387 INFO:tasks.workunit.client.1.smithi067.stdout:2/616: chown d5/da/de/l58 13770 1 2024-04-12T05:37:26.389 INFO:tasks.workunit.client.1.smithi067.stdout:5/788: dread d0/d6/d33/dd3/fa5 [0,4194304] 0 2024-04-12T05:37:26.392 INFO:tasks.workunit.client.1.smithi067.stdout:2/617: creat d5/d23/d93/fc5 x:0 0 0 2024-04-12T05:37:26.393 INFO:tasks.workunit.client.1.smithi067.stdout:4/476: write d8/d39/f46 [394167,104047] 0 2024-04-12T05:37:26.396 INFO:tasks.workunit.client.1.smithi067.stdout:5/789: getdents d0/d3a 0 2024-04-12T05:37:26.396 INFO:tasks.workunit.client.1.smithi067.stdout:2/618: mkdir d5/da/de/dc4/dc6 0 2024-04-12T05:37:26.397 INFO:tasks.workunit.client.1.smithi067.stdout:5/790: dread - d0/d6/f76 zero size 2024-04-12T05:37:26.397 INFO:tasks.workunit.client.1.smithi067.stdout:2/619: chown d5/d11/f2a 71 1 2024-04-12T05:37:26.398 INFO:tasks.workunit.client.1.smithi067.stdout:4/477: creat d8/da/d1d/d15/d36/d5d/d66/d71/d92/fa1 x:0 0 0 2024-04-12T05:37:26.399 INFO:tasks.workunit.client.1.smithi067.stdout:5/791: readlink d0/da/d41/l8c 0 2024-04-12T05:37:26.401 INFO:tasks.workunit.client.1.smithi067.stdout:2/620: fdatasync d5/d6a/d78/faa 0 2024-04-12T05:37:26.402 INFO:tasks.workunit.client.1.smithi067.stdout:4/478: mkdir d8/da/d1d/d27/d7d/da2 0 2024-04-12T05:37:26.403 INFO:tasks.workunit.client.1.smithi067.stdout:2/621: fdatasync d5/da/de/f69 0 2024-04-12T05:37:26.404 INFO:tasks.workunit.client.1.smithi067.stdout:4/479: write d8/d9/d3d/d64/f97 [93256,12737] 0 2024-04-12T05:37:26.405 INFO:tasks.workunit.client.1.smithi067.stdout:5/792: dread d0/da/d3f/d82/f80 [0,4194304] 0 2024-04-12T05:37:26.407 INFO:tasks.workunit.client.1.smithi067.stdout:2/622: truncate d5/da/f1d 7138198 0 2024-04-12T05:37:26.409 INFO:tasks.workunit.client.1.smithi067.stdout:5/793: mknod d0/d6/d40/d4b/d38/d8e/c103 0 2024-04-12T05:37:26.413 INFO:tasks.workunit.client.1.smithi067.stdout:5/794: link d0/da/d41/f60 d0/f104 0 2024-04-12T05:37:26.415 INFO:tasks.workunit.client.1.smithi067.stdout:5/795: read - d0/d6/d40/d4b/fe0 zero size 2024-04-12T05:37:26.421 INFO:tasks.workunit.client.1.smithi067.stdout:4/480: dread d8/da/d1d/d15/d58/f78 [0,4194304] 0 2024-04-12T05:37:26.421 INFO:tasks.workunit.client.1.smithi067.stdout:5/796: write d0/d6/d40/d4b/d38/f79 [3996104,89038] 0 2024-04-12T05:37:26.427 INFO:tasks.workunit.client.1.smithi067.stdout:5/797: write d0/d6/f10 [1786993,37724] 0 2024-04-12T05:37:26.483 INFO:tasks.workunit.client.1.smithi067.stdout:4/481: truncate d8/da/d1d/d15/d58/f78 2872500 0 2024-04-12T05:37:26.486 INFO:tasks.workunit.client.1.smithi067.stdout:5/798: dread d0/da/d3f/d82/fa2 [0,4194304] 0 2024-04-12T05:37:26.488 INFO:tasks.workunit.client.1.smithi067.stdout:4/482: dread d8/d39/f3b [4194304,4194304] 0 2024-04-12T05:37:26.492 INFO:tasks.workunit.client.1.smithi067.stdout:4/483: mkdir d8/d9/d3d/da3 0 2024-04-12T05:37:26.497 INFO:tasks.workunit.client.1.smithi067.stdout:4/484: creat d8/da/d1d/d15/d36/d5d/d66/fa4 x:0 0 0 2024-04-12T05:37:26.498 INFO:tasks.workunit.client.1.smithi067.stdout:4/485: chown d8/da/d9f/d68/d8e 10 1 2024-04-12T05:37:26.499 INFO:tasks.workunit.client.1.smithi067.stdout:4/486: readlink d8/d9/lb 0 2024-04-12T05:37:26.505 INFO:tasks.workunit.client.1.smithi067.stdout:5/799: dread d0/d6/d40/d4b/d38/d8e/d8f/fe7 [0,4194304] 0 2024-04-12T05:37:26.506 INFO:tasks.workunit.client.1.smithi067.stdout:5/800: chown d0/d6/c55 5808 1 2024-04-12T05:37:26.507 INFO:tasks.workunit.client.1.smithi067.stdout:4/487: truncate d8/da/d1d/d15/d36/f76 181540 0 2024-04-12T05:37:26.509 INFO:tasks.workunit.client.1.smithi067.stdout:5/801: symlink d0/d3a/d2c/dc6/l105 0 2024-04-12T05:37:26.510 INFO:tasks.workunit.client.1.smithi067.stdout:4/488: mknod d8/da/d1d/d15/d36/ca5 0 2024-04-12T05:37:26.512 INFO:tasks.workunit.client.1.smithi067.stdout:4/489: dread - d8/da/d1d/d15/d36/d5d/d66/d71/f9a zero size 2024-04-12T05:37:26.514 INFO:tasks.workunit.client.1.smithi067.stdout:5/802: dread d0/d6/f10 [0,4194304] 0 2024-04-12T05:37:26.516 INFO:tasks.workunit.client.1.smithi067.stdout:4/490: rmdir d8/da/d1d/d27/d7d/da2 0 2024-04-12T05:37:26.517 INFO:tasks.workunit.client.1.smithi067.stdout:4/491: readlink d8/da/d9f/l23 0 2024-04-12T05:37:26.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:26 smithi067 ceph-mon[49010]: pgmap v274: 65 pgs: 65 active+clean; 2.7 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 60 MiB/s rd, 177 MiB/s wr, 372 op/s 2024-04-12T05:37:26.522 INFO:tasks.workunit.client.1.smithi067.stdout:5/803: write d0/d6/d33/dd3/f89 [52874,113722] 0 2024-04-12T05:37:26.525 INFO:tasks.workunit.client.1.smithi067.stdout:5/804: readlink d0/d6/d20/lee 0 2024-04-12T05:37:26.526 INFO:tasks.workunit.client.1.smithi067.stdout:4/492: unlink d8/da/d1d/d15/d36/f76 0 2024-04-12T05:37:26.527 INFO:tasks.workunit.client.1.smithi067.stdout:4/493: chown d8/da/d1d/d15/d36/d5d/d66/d71/d92 1 1 2024-04-12T05:37:26.528 INFO:tasks.workunit.client.1.smithi067.stdout:5/805: rename d0/d6/d40/d4b/d38/f79 to d0/d6/d33/d9e/f106 0 2024-04-12T05:37:26.529 INFO:tasks.workunit.client.1.smithi067.stdout:5/806: chown d0/d6/f56 10064 1 2024-04-12T05:37:26.530 INFO:tasks.workunit.client.1.smithi067.stdout:4/494: mknod d8/da/d1d/d15/d36/d5d/d66/d96/ca6 0 2024-04-12T05:37:26.530 INFO:tasks.workunit.client.1.smithi067.stdout:4/495: fdatasync d8/d39/f46 0 2024-04-12T05:37:26.531 INFO:tasks.workunit.client.1.smithi067.stdout:5/807: symlink d0/d3a/d2c/dc6/l107 0 2024-04-12T05:37:26.531 INFO:tasks.workunit.client.1.smithi067.stdout:5/808: chown d0/d6/dae/cc1 23 1 2024-04-12T05:37:26.532 INFO:tasks.workunit.client.1.smithi067.stdout:5/809: write d0/f1c [4787743,9745] 0 2024-04-12T05:37:26.534 INFO:tasks.workunit.client.1.smithi067.stdout:4/496: creat d8/da/d8b/fa7 x:0 0 0 2024-04-12T05:37:26.534 INFO:tasks.workunit.client.1.smithi067.stdout:8/487: sync 2024-04-12T05:37:26.534 INFO:tasks.workunit.client.1.smithi067.stdout:7/568: sync 2024-04-12T05:37:26.535 INFO:tasks.workunit.client.1.smithi067.stdout:3/447: sync 2024-04-12T05:37:26.536 INFO:tasks.workunit.client.1.smithi067.stdout:4/497: dread - d8/da/d1d/d27/f86 zero size 2024-04-12T05:37:26.537 INFO:tasks.workunit.client.1.smithi067.stdout:7/569: fsync d0/d23/f6e 0 2024-04-12T05:37:26.538 INFO:tasks.workunit.client.1.smithi067.stdout:4/498: chown d8/da/d8b/d8c 12944361 1 2024-04-12T05:37:26.540 INFO:tasks.workunit.client.1.smithi067.stdout:3/448: fdatasync dc/d1d/f22 0 2024-04-12T05:37:26.540 INFO:tasks.workunit.client.1.smithi067.stdout:8/488: symlink d6/d8/d18/d63/d67/d69/l94 0 2024-04-12T05:37:26.543 INFO:tasks.workunit.client.1.smithi067.stdout:4/499: unlink d8/da/d8b/d45/f7e 0 2024-04-12T05:37:26.546 INFO:tasks.workunit.client.1.smithi067.stdout:7/570: rename d0/f5e to d0/d23/d80/da0/db4/d5/d1d/d43/fb9 0 2024-04-12T05:37:26.547 INFO:tasks.workunit.client.1.smithi067.stdout:3/449: mkdir dc/d1d/d28/d3e/d53/d94 0 2024-04-12T05:37:26.550 INFO:tasks.workunit.client.1.smithi067.stdout:7/571: readlink d0/d23/d80/da0/db4/dd/d42/l83 0 2024-04-12T05:37:26.551 INFO:tasks.workunit.client.1.smithi067.stdout:8/489: dwrite d6/d8/d18/f33 [0,4194304] 0 2024-04-12T05:37:26.553 INFO:tasks.workunit.client.1.smithi067.stdout:5/810: dread d0/da/d3f/d82/fc0 [0,4194304] 0 2024-04-12T05:37:26.554 INFO:tasks.workunit.client.1.smithi067.stdout:4/500: dread d8/da/d1d/d27/f56 [0,4194304] 0 2024-04-12T05:37:26.555 INFO:tasks.workunit.client.1.smithi067.stdout:3/450: creat dc/d1d/d28/d3e/d8a/f95 x:0 0 0 2024-04-12T05:37:26.556 INFO:tasks.workunit.client.1.smithi067.stdout:7/572: read - d0/d23/d80/da0/db4/dd/d79/f6d zero size 2024-04-12T05:37:26.558 INFO:tasks.workunit.client.1.smithi067.stdout:3/451: creat dc/df/d27/f96 x:0 0 0 2024-04-12T05:37:26.558 INFO:tasks.workunit.client.1.smithi067.stdout:5/811: rmdir d0/da/d41/d7d 39 2024-04-12T05:37:26.559 INFO:tasks.workunit.client.1.smithi067.stdout:8/490: rmdir d6/d8/d18/df/d12 39 2024-04-12T05:37:26.560 INFO:tasks.workunit.client.1.smithi067.stdout:3/452: truncate dc/df/f82 372025 0 2024-04-12T05:37:26.561 INFO:tasks.workunit.client.1.smithi067.stdout:4/501: creat d8/da/d1d/d27/d91/fa8 x:0 0 0 2024-04-12T05:37:26.564 INFO:tasks.workunit.client.1.smithi067.stdout:7/573: creat d0/d23/d80/da0/db4/dd/da2/d51/db0/fba x:0 0 0 2024-04-12T05:37:26.568 INFO:tasks.workunit.client.1.smithi067.stdout:8/491: creat d6/f95 x:0 0 0 2024-04-12T05:37:26.572 INFO:tasks.workunit.client.1.smithi067.stdout:8/492: mkdir d6/d8/d50/d96 0 2024-04-12T05:37:26.573 INFO:tasks.workunit.client.1.smithi067.stdout:4/502: dwrite d8/d9/f20 [0,4194304] 0 2024-04-12T05:37:26.573 INFO:tasks.workunit.client.1.smithi067.stdout:8/493: chown d6/d8/c1a 22570 1 2024-04-12T05:37:26.574 INFO:tasks.workunit.client.1.smithi067.stdout:3/453: rename dc/d1d/d26/l32 to dc/l97 0 2024-04-12T05:37:26.575 INFO:tasks.workunit.client.1.smithi067.stdout:5/812: mknod d0/d6/d40/d4b/d38/d8e/d8f/db9/dbd/def/c108 0 2024-04-12T05:37:26.576 INFO:tasks.workunit.client.1.smithi067.stdout:4/503: truncate d8/da/d1d/d27/f86 941993 0 2024-04-12T05:37:26.577 INFO:tasks.workunit.client.1.smithi067.stdout:4/504: chown d8/d9/d3d/d64 205 1 2024-04-12T05:37:26.578 INFO:tasks.workunit.client.1.smithi067.stdout:4/505: stat d8/da/d9f 0 2024-04-12T05:37:26.579 INFO:tasks.workunit.client.1.smithi067.stdout:4/506: readlink d8/da/d9f/l23 0 2024-04-12T05:37:26.579 INFO:tasks.workunit.client.1.smithi067.stdout:3/454: rename dc/d50/d5a to dc/d1d/d28/d3e/d8a/d98 0 2024-04-12T05:37:26.580 INFO:tasks.workunit.client.1.smithi067.stdout:4/507: stat d8/da/d9f/d68/d8e 0 2024-04-12T05:37:26.581 INFO:tasks.workunit.client.1.smithi067.stdout:8/494: mknod d6/d86/d4f/d82/c97 0 2024-04-12T05:37:26.585 INFO:tasks.workunit.client.1.smithi067.stdout:3/455: write dc/d1d/d28/d3e/d8a/f95 [888187,54432] 0 2024-04-12T05:37:26.585 INFO:tasks.workunit.client.1.smithi067.stdout:4/508: symlink d8/d39/la9 0 2024-04-12T05:37:26.585 INFO:tasks.workunit.client.1.smithi067.stdout:5/813: dwrite d0/da/fbb [0,4194304] 0 2024-04-12T05:37:26.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:26 smithi032 ceph-mon[39731]: pgmap v274: 65 pgs: 65 active+clean; 2.7 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 60 MiB/s rd, 177 MiB/s wr, 372 op/s 2024-04-12T05:37:26.591 INFO:tasks.workunit.client.1.smithi067.stdout:4/509: readlink d8/da/d1d/d15/d58/l98 0 2024-04-12T05:37:26.598 INFO:tasks.workunit.client.1.smithi067.stdout:7/574: dwrite d0/d23/d80/da0/db4/d5/fe [0,4194304] 0 2024-04-12T05:37:26.598 INFO:tasks.workunit.client.1.smithi067.stdout:4/510: read - d8/da/d1d/d15/d36/d5d/f8f zero size 2024-04-12T05:37:26.598 INFO:tasks.workunit.client.1.smithi067.stdout:5/814: mkdir d0/d6/d40/d4b/d38/d8e/d8f/d99/d109 0 2024-04-12T05:37:26.605 INFO:tasks.workunit.client.1.smithi067.stdout:3/456: dwrite dc/df/d27/f2d [0,4194304] 0 2024-04-12T05:37:26.608 INFO:tasks.workunit.client.1.smithi067.stdout:4/511: rename d8/da/d9f/d5f to d8/da/d1d/d15/d75/daa 0 2024-04-12T05:37:26.613 INFO:tasks.workunit.client.1.smithi067.stdout:3/457: getdents dc/d1d/d89 0 2024-04-12T05:37:26.614 INFO:tasks.workunit.client.1.smithi067.stdout:3/458: readlink dc/df/l35 0 2024-04-12T05:37:26.615 INFO:tasks.workunit.client.1.smithi067.stdout:5/815: dwrite d0/d6/d20/f58 [0,4194304] 0 2024-04-12T05:37:26.615 INFO:tasks.workunit.client.1.smithi067.stdout:4/512: getdents d8/da/d1d/d15/d36/d5d 0 2024-04-12T05:37:26.616 INFO:tasks.workunit.client.1.smithi067.stdout:5/816: chown d0/d3a/lad 62957 1 2024-04-12T05:37:26.617 INFO:tasks.workunit.client.1.smithi067.stdout:5/817: chown d0/d3a/d2c/d91/c29 53687 1 2024-04-12T05:37:26.619 INFO:tasks.workunit.client.1.smithi067.stdout:4/513: rmdir d8/da/d1d/d15/d36/d5d/d66/d71/d92 39 2024-04-12T05:37:26.620 INFO:tasks.workunit.client.1.smithi067.stdout:5/818: mkdir d0/d6/d40/d10a 0 2024-04-12T05:37:26.621 INFO:tasks.workunit.client.1.smithi067.stdout:4/514: truncate d8/d9/f6b 1687177 0 2024-04-12T05:37:26.624 INFO:tasks.workunit.client.1.smithi067.stdout:4/515: write d8/d39/f3b [3267669,77204] 0 2024-04-12T05:37:26.627 INFO:tasks.workunit.client.1.smithi067.stdout:5/819: dwrite d0/f2 [0,4194304] 0 2024-04-12T05:37:26.630 INFO:tasks.workunit.client.1.smithi067.stdout:5/820: creat d0/d3a/d2c/dc6/f10b x:0 0 0 2024-04-12T05:37:26.631 INFO:tasks.workunit.client.1.smithi067.stdout:5/821: readlink d0/d6/d20/lf4 0 2024-04-12T05:37:26.633 INFO:tasks.workunit.client.1.smithi067.stdout:7/575: dread d0/d23/d80/da0/db4/dd/d42/d7e/f4e [4194304,4194304] 0 2024-04-12T05:37:26.636 INFO:tasks.workunit.client.1.smithi067.stdout:7/576: dread - d0/d23/d80/da0/db4/dd/d79/f6d zero size 2024-04-12T05:37:26.636 INFO:tasks.workunit.client.1.smithi067.stdout:5/822: link d0/d3a/dab/ld8 d0/d6/d40/d10a/l10c 0 2024-04-12T05:37:26.637 INFO:tasks.workunit.client.1.smithi067.stdout:5/823: dread - d0/d3a/d2c/d91/fc7 zero size 2024-04-12T05:37:26.641 INFO:tasks.workunit.client.1.smithi067.stdout:5/824: mkdir d0/d6/d40/d4b/d38/d8e/d8f/db9/d10d 0 2024-04-12T05:37:26.642 INFO:tasks.workunit.client.1.smithi067.stdout:5/825: dread d0/da/d3f/d82/f80 [0,4194304] 0 2024-04-12T05:37:26.644 INFO:tasks.workunit.client.1.smithi067.stdout:5/826: mknod d0/d6/d40/d4b/d38/d8e/d8f/db9/dbd/c10e 0 2024-04-12T05:37:26.646 INFO:tasks.workunit.client.1.smithi067.stdout:7/577: dread d0/d23/d45/f59 [0,4194304] 0 2024-04-12T05:37:26.648 INFO:tasks.workunit.client.1.smithi067.stdout:5/827: creat d0/d6/d40/d4b/d38/d8e/d8f/db9/dbd/ddc/f10f x:0 0 0 2024-04-12T05:37:26.648 INFO:tasks.workunit.client.1.smithi067.stdout:7/578: chown d0/d23/d80/da0/db4/l90 0 1 2024-04-12T05:37:26.649 INFO:tasks.workunit.client.1.smithi067.stdout:5/828: read d0/f104 [905422,36741] 0 2024-04-12T05:37:26.650 INFO:tasks.workunit.client.1.smithi067.stdout:7/579: chown d0/d23/f41 782 1 2024-04-12T05:37:26.652 INFO:tasks.workunit.client.1.smithi067.stdout:5/829: readlink d0/d6/d40/d4b/d38/lba 0 2024-04-12T05:37:26.657 INFO:tasks.workunit.client.1.smithi067.stdout:7/580: rename d0/d67/d57/d85 to d0/d23/d80/da0/db4/d5/d1d/d7d/dbb 0 2024-04-12T05:37:26.657 INFO:tasks.workunit.client.1.smithi067.stdout:5/830: symlink d0/d6/d40/d4b/l110 0 2024-04-12T05:37:26.658 INFO:tasks.workunit.client.1.smithi067.stdout:5/831: truncate d0/d6/fc8 265572 0 2024-04-12T05:37:26.659 INFO:tasks.workunit.client.1.smithi067.stdout:7/581: truncate d0/d67/f56 233530 0 2024-04-12T05:37:26.659 INFO:tasks.workunit.client.1.smithi067.stdout:7/582: stat d0/d23/d45/f89 0 2024-04-12T05:37:26.661 INFO:tasks.workunit.client.1.smithi067.stdout:7/583: creat d0/d67/d57/fbc x:0 0 0 2024-04-12T05:37:26.662 INFO:tasks.workunit.client.1.smithi067.stdout:5/832: getdents d0/d3a/dab 0 2024-04-12T05:37:26.663 INFO:tasks.workunit.client.1.smithi067.stdout:7/584: write d0/d23/d80/da0/db4/dd/d42/d7e/f1f [4400833,16867] 0 2024-04-12T05:37:26.663 INFO:tasks.workunit.client.1.smithi067.stdout:5/833: write d0/d6/d20/d3e/f95 [915467,105158] 0 2024-04-12T05:37:26.664 INFO:tasks.workunit.client.1.smithi067.stdout:5/834: stat d0/f2 0 2024-04-12T05:37:26.664 INFO:tasks.workunit.client.1.smithi067.stdout:7/585: chown d0/d23/d80/da0/db4/dd/c5b 7196999 1 2024-04-12T05:37:26.735 INFO:tasks.workunit.client.1.smithi067.stdout:9/526: sync 2024-04-12T05:37:26.735 INFO:tasks.workunit.client.1.smithi067.stdout:6/614: sync 2024-04-12T05:37:26.735 INFO:tasks.workunit.client.1.smithi067.stdout:0/641: sync 2024-04-12T05:37:26.736 INFO:tasks.workunit.client.1.smithi067.stdout:7/586: chown d0/d23/d80/da0/db4/dd/d79/l8c 32 1 2024-04-12T05:37:26.737 INFO:tasks.workunit.client.1.smithi067.stdout:8/495: write d6/d8/f92 [1102333,35203] 0 2024-04-12T05:37:26.738 INFO:tasks.workunit.client.1.smithi067.stdout:9/527: fdatasync d1/d66/f6e 0 2024-04-12T05:37:26.742 INFO:tasks.workunit.client.1.smithi067.stdout:0/642: rmdir dd 39 2024-04-12T05:37:26.742 INFO:tasks.workunit.client.1.smithi067.stdout:8/496: write d6/d8/d18/df/f93 [778574,69036] 0 2024-04-12T05:37:26.745 INFO:tasks.workunit.client.1.smithi067.stdout:7/587: chown d0/d23/d80/da0/db4/dd/da2/d51/caa 35277 1 2024-04-12T05:37:26.748 INFO:tasks.workunit.client.1.smithi067.stdout:8/497: write d6/d8/d18/d36/f54 [515916,118516] 0 2024-04-12T05:37:26.748 INFO:tasks.workunit.client.1.smithi067.stdout:3/459: dwrite f9 [4194304,4194304] 0 2024-04-12T05:37:26.748 INFO:tasks.workunit.client.1.smithi067.stdout:0/643: write dd/d12/d14/d49/d9b/fb6 [1521262,63687] 0 2024-04-12T05:37:26.749 INFO:tasks.workunit.client.1.smithi067.stdout:9/528: link d1/d28/d21/d3f/f97 d1/d28/d44/d8d/fa0 0 2024-04-12T05:37:26.749 INFO:tasks.workunit.client.1.smithi067.stdout:6/615: dread dd/f1d [0,4194304] 0 2024-04-12T05:37:26.753 INFO:tasks.workunit.client.1.smithi067.stdout:0/644: fdatasync dd/d12/d14/f61 0 2024-04-12T05:37:26.754 INFO:tasks.workunit.client.1.smithi067.stdout:3/460: symlink dc/d50/l99 0 2024-04-12T05:37:26.755 INFO:tasks.workunit.client.1.smithi067.stdout:7/588: mkdir d0/d23/d80/da0/db4/dd/d42/d71/dbd 0 2024-04-12T05:37:26.758 INFO:tasks.workunit.client.1.smithi067.stdout:3/461: mkdir dc/d1d/d26/d52/d9a 0 2024-04-12T05:37:26.759 INFO:tasks.workunit.client.1.smithi067.stdout:9/529: getdents d1/d28/d44 0 2024-04-12T05:37:26.760 INFO:tasks.workunit.client.1.smithi067.stdout:3/462: write dc/d1d/f43 [2611510,18927] 0 2024-04-12T05:37:26.760 INFO:tasks.workunit.client.1.smithi067.stdout:8/498: dread d6/d8/d18/d36/f7f [0,4194304] 0 2024-04-12T05:37:26.761 INFO:tasks.workunit.client.1.smithi067.stdout:7/589: symlink d0/d23/d80/da0/db4/d5/d1d/d7d/dbb/lbe 0 2024-04-12T05:37:26.763 INFO:tasks.workunit.client.1.smithi067.stdout:0/645: getdents dd/d12/d14/d1a/d3d 0 2024-04-12T05:37:26.765 INFO:tasks.workunit.client.1.smithi067.stdout:8/499: read d6/d8/d18/d63/d67/d69/f79 [456611,53314] 0 2024-04-12T05:37:26.765 INFO:tasks.workunit.client.1.smithi067.stdout:9/530: creat d1/d5f/d86/fa1 x:0 0 0 2024-04-12T05:37:26.766 INFO:tasks.workunit.client.1.smithi067.stdout:0/646: truncate dd/d12/d14/d49/fb1 600650 0 2024-04-12T05:37:26.767 INFO:tasks.workunit.client.1.smithi067.stdout:7/590: creat d0/d23/d80/da0/db4/d5/d4a/fbf x:0 0 0 2024-04-12T05:37:26.769 INFO:tasks.workunit.client.1.smithi067.stdout:3/463: mkdir dc/d1d/d89/d9b 0 2024-04-12T05:37:26.770 INFO:tasks.workunit.client.1.smithi067.stdout:8/500: creat d6/d8/d8b/f98 x:0 0 0 2024-04-12T05:37:26.771 INFO:tasks.workunit.client.1.smithi067.stdout:9/531: mknod d1/d28/d5e/ca2 0 2024-04-12T05:37:26.772 INFO:tasks.workunit.client.1.smithi067.stdout:0/647: rename dd/d12/d14/f57 to dd/dbd/fcb 0 2024-04-12T05:37:26.774 INFO:tasks.workunit.client.1.smithi067.stdout:8/501: unlink d6/f27 0 2024-04-12T05:37:26.775 INFO:tasks.workunit.client.1.smithi067.stdout:8/502: chown d6/d8/d18 44830 1 2024-04-12T05:37:26.776 INFO:tasks.workunit.client.1.smithi067.stdout:3/464: mkdir dc/d1d/d26/d52/d9a/d9c 0 2024-04-12T05:37:26.777 INFO:tasks.workunit.client.1.smithi067.stdout:8/503: read d6/d8/d6e/f84 [564016,62606] 0 2024-04-12T05:37:26.778 INFO:tasks.workunit.client.1.smithi067.stdout:7/591: rename d0/d23/d80/da0/db4/d5/d1d/d7d/cb7 to d0/d23/d80/da0/db4/dd/d79/cc0 0 2024-04-12T05:37:26.778 INFO:tasks.workunit.client.1.smithi067.stdout:0/648: rmdir dd/d9c/d5f/d78/d83 39 2024-04-12T05:37:26.778 INFO:tasks.workunit.client.1.smithi067.stdout:9/532: dwrite d1/d28/f50 [0,4194304] 0 2024-04-12T05:37:26.779 INFO:tasks.workunit.client.1.smithi067.stdout:3/465: chown dc/d1d/d28/d3e/d53/d94 77 1 2024-04-12T05:37:26.779 INFO:tasks.workunit.client.1.smithi067.stdout:8/504: chown d6/d8/d18/d36 0 1 2024-04-12T05:37:26.780 INFO:tasks.workunit.client.1.smithi067.stdout:9/533: dread - d1/d28/d21/f9b zero size 2024-04-12T05:37:26.783 INFO:tasks.workunit.client.1.smithi067.stdout:9/534: read - d1/d13/f87 zero size 2024-04-12T05:37:26.784 INFO:tasks.workunit.client.1.smithi067.stdout:3/466: mknod dc/d1d/d26/d52/d9a/c9d 0 2024-04-12T05:37:26.786 INFO:tasks.workunit.client.1.smithi067.stdout:8/505: rename d6/d8/d18/d36/f6c to d6/d86/f99 0 2024-04-12T05:37:26.789 INFO:tasks.workunit.client.1.smithi067.stdout:3/467: creat dc/d1d/d26/f9e x:0 0 0 2024-04-12T05:37:26.790 INFO:tasks.workunit.client.1.smithi067.stdout:0/649: creat dd/d12/d14/d1a/fcc x:0 0 0 2024-04-12T05:37:26.792 INFO:tasks.workunit.client.1.smithi067.stdout:8/506: rename d6/d8/d18/df to d6/d8/d18/d63/d9a 0 2024-04-12T05:37:26.793 INFO:tasks.workunit.client.1.smithi067.stdout:9/535: dwrite d1/d28/f50 [4194304,4194304] 0 2024-04-12T05:37:26.796 INFO:tasks.workunit.client.1.smithi067.stdout:9/536: dread - d1/d28/d27/d32/f8c zero size 2024-04-12T05:37:26.803 INFO:tasks.workunit.client.1.smithi067.stdout:9/537: rmdir d1/d28 39 2024-04-12T05:37:26.805 INFO:tasks.workunit.client.1.smithi067.stdout:7/592: dwrite d0/f8 [0,4194304] 0 2024-04-12T05:37:26.807 INFO:tasks.workunit.client.1.smithi067.stdout:8/507: rmdir d6/d8/d50/d96 0 2024-04-12T05:37:26.815 INFO:tasks.workunit.client.1.smithi067.stdout:7/593: creat d0/d23/d80/da0/db4/dd/d42/d71/dbd/fc1 x:0 0 0 2024-04-12T05:37:26.820 INFO:tasks.workunit.client.1.smithi067.stdout:7/594: dwrite d0/d23/d80/da0/db4/d5/f50 [0,4194304] 0 2024-04-12T05:37:26.826 INFO:tasks.workunit.client.1.smithi067.stdout:3/468: dwrite dc/df/f30 [0,4194304] 0 2024-04-12T05:37:26.827 INFO:tasks.workunit.client.1.smithi067.stdout:3/469: fsync dc/d1d/f44 0 2024-04-12T05:37:26.835 INFO:tasks.workunit.client.1.smithi067.stdout:7/595: link d0/d23/d80/da0/db4/dd/da2/d51/db0/cb2 d0/d23/d80/da0/db4/d5/d1d/cc2 0 2024-04-12T05:37:26.836 INFO:tasks.workunit.client.1.smithi067.stdout:3/470: dwrite dc/f3a [0,4194304] 0 2024-04-12T05:37:26.837 INFO:tasks.workunit.client.1.smithi067.stdout:7/596: chown d0/d67/d57/cb5 436223 1 2024-04-12T05:37:26.840 INFO:tasks.workunit.client.1.smithi067.stdout:7/597: mkdir d0/d23/d80/da0/dc3 0 2024-04-12T05:37:26.842 INFO:tasks.workunit.client.1.smithi067.stdout:3/471: symlink dc/d1d/d26/d52/l9f 0 2024-04-12T05:37:26.844 INFO:tasks.workunit.client.1.smithi067.stdout:7/598: creat d0/d23/d80/da0/db4/d5/d1d/d7d/dbb/fc4 x:0 0 0 2024-04-12T05:37:26.844 INFO:tasks.workunit.client.1.smithi067.stdout:3/472: readlink dc/df/l86 0 2024-04-12T05:37:26.846 INFO:tasks.workunit.client.1.smithi067.stdout:3/473: write dc/d1d/d28/d3e/f63 [2042024,65808] 0 2024-04-12T05:37:26.846 INFO:tasks.workunit.client.1.smithi067.stdout:7/599: mknod d0/d23/cc5 0 2024-04-12T05:37:26.854 INFO:tasks.workunit.client.1.smithi067.stdout:0/650: write dd/d12/d14/d1a/d3d/f32 [972244,46666] 0 2024-04-12T05:37:26.855 INFO:tasks.workunit.client.1.smithi067.stdout:7/600: dwrite d0/d67/d57/fa7 [0,4194304] 0 2024-04-12T05:37:26.857 INFO:tasks.workunit.client.1.smithi067.stdout:0/651: symlink dd/dbd/lcd 0 2024-04-12T05:37:26.858 INFO:tasks.workunit.client.1.smithi067.stdout:7/601: fdatasync d0/d23/d80/da0/db4/dd/da2/f92 0 2024-04-12T05:37:26.865 INFO:tasks.workunit.client.1.smithi067.stdout:7/602: mknod d0/d23/d80/da0/db4/dd/d79/cc6 0 2024-04-12T05:37:26.865 INFO:tasks.workunit.client.1.smithi067.stdout:9/538: write d1/d28/d21/d45/f8b [160410,45449] 0 2024-04-12T05:37:26.868 INFO:tasks.workunit.client.1.smithi067.stdout:0/652: dwrite dd/d12/d14/d51/fb4 [0,4194304] 0 2024-04-12T05:37:26.869 INFO:tasks.workunit.client.1.smithi067.stdout:3/474: write dc/f18 [4225364,6309] 0 2024-04-12T05:37:26.876 INFO:tasks.workunit.client.1.smithi067.stdout:0/653: write dd/d12/f4e [1466523,102313] 0 2024-04-12T05:37:26.876 INFO:tasks.workunit.client.1.smithi067.stdout:9/539: link d1/f7 d1/d28/d5e/fa3 0 2024-04-12T05:37:26.880 INFO:tasks.workunit.client.1.smithi067.stdout:9/540: write d1/d28/d23/f63 [2028193,124314] 0 2024-04-12T05:37:26.880 INFO:tasks.workunit.client.1.smithi067.stdout:3/475: link dc/d1d/d26/d52/l9f dc/d1d/d28/d3e/d84/la0 0 2024-04-12T05:37:26.881 INFO:tasks.workunit.client.1.smithi067.stdout:7/603: dwrite d0/d23/d80/da0/db4/d5/d1d/f3a [0,4194304] 0 2024-04-12T05:37:26.884 INFO:tasks.workunit.client.1.smithi067.stdout:3/476: rmdir dc/d1d 39 2024-04-12T05:37:26.889 INFO:tasks.workunit.client.1.smithi067.stdout:3/477: fsync dc/d1d/d28/d3e/f4a 0 2024-04-12T05:37:26.893 INFO:tasks.workunit.client.1.smithi067.stdout:9/541: getdents d1/d28/d44/d8d 0 2024-04-12T05:37:26.894 INFO:tasks.workunit.client.1.smithi067.stdout:7/604: link d0/d23/d80/da0/db4/dd/c5b d0/d23/d80/da0/db4/d5/d1d/cc7 0 2024-04-12T05:37:26.899 INFO:tasks.workunit.client.1.smithi067.stdout:0/654: dwrite dd/d12/f59 [0,4194304] 0 2024-04-12T05:37:26.899 INFO:tasks.workunit.client.1.smithi067.stdout:7/605: unlink d0/d23/d80/da0/db4/f78 0 2024-04-12T05:37:26.901 INFO:tasks.workunit.client.1.smithi067.stdout:7/606: fsync d0/d23/d45/f89 0 2024-04-12T05:37:26.902 INFO:tasks.workunit.client.1.smithi067.stdout:9/542: rename d1/d28/d21/d68/c84 to d1/d28/d27/ca4 0 2024-04-12T05:37:26.903 INFO:tasks.workunit.client.1.smithi067.stdout:3/478: dwrite dc/d1d/d28/f62 [0,4194304] 0 2024-04-12T05:37:26.904 INFO:tasks.workunit.client.1.smithi067.stdout:0/655: symlink dd/d5b/lce 0 2024-04-12T05:37:26.908 INFO:tasks.workunit.client.1.smithi067.stdout:9/543: unlink d1/d28/d23/l51 0 2024-04-12T05:37:26.909 INFO:tasks.workunit.client.1.smithi067.stdout:3/479: mknod dc/d1d/d4d/ca1 0 2024-04-12T05:37:26.913 INFO:tasks.workunit.client.1.smithi067.stdout:9/544: creat d1/d28/d5e/fa5 x:0 0 0 2024-04-12T05:37:26.913 INFO:tasks.workunit.client.1.smithi067.stdout:9/545: dread - d1/d28/d21/d3f/d4f/f57 zero size 2024-04-12T05:37:26.916 INFO:tasks.workunit.client.1.smithi067.stdout:3/480: creat dc/d1d/d28/d3e/d8a/d98/fa2 x:0 0 0 2024-04-12T05:37:26.917 INFO:tasks.workunit.client.1.smithi067.stdout:7/607: link d0/d23/d80/da0/db4/dd/c37 d0/d23/d80/da0/db4/dd/da2/d51/cc8 0 2024-04-12T05:37:26.920 INFO:tasks.workunit.client.1.smithi067.stdout:9/546: dread d1/d28/f50 [0,4194304] 0 2024-04-12T05:37:26.922 INFO:tasks.workunit.client.1.smithi067.stdout:3/481: mkdir dc/d1d/d89/da3 0 2024-04-12T05:37:26.923 INFO:tasks.workunit.client.1.smithi067.stdout:7/608: creat d0/d23/d80/da0/db4/dd/da2/d51/db0/fc9 x:0 0 0 2024-04-12T05:37:26.923 INFO:tasks.workunit.client.1.smithi067.stdout:3/482: chown dc/df/d8d/f93 3030 1 2024-04-12T05:37:26.924 INFO:tasks.workunit.client.1.smithi067.stdout:9/547: chown d1/d13/l2f 3189695 1 2024-04-12T05:37:26.927 INFO:tasks.workunit.client.1.smithi067.stdout:9/548: fsync d1/d28/d21/d45/f9f 0 2024-04-12T05:37:26.927 INFO:tasks.workunit.client.1.smithi067.stdout:7/609: creat d0/d67/d57/fca x:0 0 0 2024-04-12T05:37:26.928 INFO:tasks.workunit.client.1.smithi067.stdout:9/549: stat d1/d28/d5e/fa3 0 2024-04-12T05:37:26.929 INFO:tasks.workunit.client.1.smithi067.stdout:0/656: getdents dd/d5b 0 2024-04-12T05:37:26.931 INFO:tasks.workunit.client.1.smithi067.stdout:3/483: dread dc/df/d8d/f93 [0,4194304] 0 2024-04-12T05:37:26.937 INFO:tasks.workunit.client.1.smithi067.stdout:9/550: dread d1/d28/d21/d45/d75/f80 [0,4194304] 0 2024-04-12T05:37:26.938 INFO:tasks.workunit.client.1.smithi067.stdout:7/610: dwrite d0/d23/d45/f1a [4194304,4194304] 0 2024-04-12T05:37:26.941 INFO:tasks.workunit.client.1.smithi067.stdout:3/484: link dc/c12 dc/d1d/d26/ca4 0 2024-04-12T05:37:26.941 INFO:tasks.workunit.client.1.smithi067.stdout:0/657: dread dd/d6f/d72/f97 [4194304,4194304] 0 2024-04-12T05:37:26.942 INFO:tasks.workunit.client.1.smithi067.stdout:9/551: creat d1/d28/d21/d45/d75/fa6 x:0 0 0 2024-04-12T05:37:26.944 INFO:tasks.workunit.client.1.smithi067.stdout:0/658: write dd/d12/f2e [4793941,113464] 0 2024-04-12T05:37:26.949 INFO:tasks.workunit.client.1.smithi067.stdout:9/552: getdents d1/d12/d15 0 2024-04-12T05:37:26.951 INFO:tasks.workunit.client.1.smithi067.stdout:7/611: truncate d0/d23/d80/f99 67212 0 2024-04-12T05:37:26.954 INFO:tasks.workunit.client.1.smithi067.stdout:9/553: mkdir d1/d28/d21/d7f/da7 0 2024-04-12T05:37:26.960 INFO:tasks.workunit.client.1.smithi067.stdout:3/485: dwrite dc/d1d/d4d/f88 [0,4194304] 0 2024-04-12T05:37:26.963 INFO:tasks.workunit.client.1.smithi067.stdout:0/659: dwrite dd/d12/d14/d1a/d3d/f32 [0,4194304] 0 2024-04-12T05:37:26.965 INFO:tasks.workunit.client.1.smithi067.stdout:9/554: write d1/f47 [555851,67516] 0 2024-04-12T05:37:26.966 INFO:tasks.workunit.client.1.smithi067.stdout:3/486: fdatasync dc/d1d/d28/d3e/d53/d78/f8b 0 2024-04-12T05:37:26.968 INFO:tasks.workunit.client.1.smithi067.stdout:7/612: truncate d0/d23/d80/da0/db4/d5/d4a/fa9 801447 0 2024-04-12T05:37:26.969 INFO:tasks.workunit.client.1.smithi067.stdout:9/555: write d1/d12/f53 [30242,84520] 0 2024-04-12T05:37:26.971 INFO:tasks.workunit.client.1.smithi067.stdout:3/487: creat dc/d1d/d4d/fa5 x:0 0 0 2024-04-12T05:37:26.972 INFO:tasks.workunit.client.1.smithi067.stdout:7/613: creat d0/d23/d80/fcb x:0 0 0 2024-04-12T05:37:26.976 INFO:tasks.workunit.client.1.smithi067.stdout:7/614: creat d0/d23/d80/da0/db4/dd/da2/d51/db0/fcc x:0 0 0 2024-04-12T05:37:26.977 INFO:tasks.workunit.client.1.smithi067.stdout:9/556: getdents d1/d13/d34 0 2024-04-12T05:37:26.977 INFO:tasks.workunit.client.1.smithi067.stdout:7/615: write d0/d23/d45/f26 [733316,82306] 0 2024-04-12T05:37:26.984 INFO:tasks.workunit.client.1.smithi067.stdout:7/616: rename d0/d23/d80/da0/db4/d5/d4a/fbf to d0/d23/d80/da0/db4/dd/fcd 0 2024-04-12T05:37:26.985 INFO:tasks.workunit.client.1.smithi067.stdout:3/488: write dc/d1d/d28/f64 [1553814,97948] 0 2024-04-12T05:37:26.985 INFO:tasks.workunit.client.1.smithi067.stdout:0/660: truncate dd/d12/d14/d1a/f1e 834412 0 2024-04-12T05:37:26.985 INFO:tasks.workunit.client.1.smithi067.stdout:9/557: dwrite d1/fd [0,4194304] 0 2024-04-12T05:37:26.987 INFO:tasks.workunit.client.1.smithi067.stdout:3/489: dread - dc/d1d/d4d/fa5 zero size 2024-04-12T05:37:26.988 INFO:tasks.workunit.client.1.smithi067.stdout:3/490: fsync f5 0 2024-04-12T05:37:26.988 INFO:tasks.workunit.client.1.smithi067.stdout:7/617: dread - d0/d23/d80/f9c zero size 2024-04-12T05:37:26.988 INFO:tasks.workunit.client.1.smithi067.stdout:0/661: stat dd/d12/d14/d51/l7a 0 2024-04-12T05:37:26.988 INFO:tasks.workunit.client.1.smithi067.stdout:1/577: sync 2024-04-12T05:37:26.988 INFO:tasks.workunit.client.1.smithi067.stdout:2/623: sync 2024-04-12T05:37:26.990 INFO:tasks.workunit.client.1.smithi067.stdout:9/558: mknod d1/d28/d21/d7f/ca8 0 2024-04-12T05:37:26.990 INFO:tasks.workunit.client.1.smithi067.stdout:1/578: chown d4/d9/d12/d59 652 1 2024-04-12T05:37:26.991 INFO:tasks.workunit.client.1.smithi067.stdout:3/491: dread - dc/d1d/d28/d3e/d84/f85 zero size 2024-04-12T05:37:26.993 INFO:tasks.workunit.client.1.smithi067.stdout:9/559: fsync d1/d12/d15/f5c 0 2024-04-12T05:37:26.993 INFO:tasks.workunit.client.1.smithi067.stdout:0/662: write dd/d12/d14/d51/da2/fb5 [974925,74940] 0 2024-04-12T05:37:26.994 INFO:tasks.workunit.client.1.smithi067.stdout:2/624: mknod d5/da/de/dc4/dc6/cc7 0 2024-04-12T05:37:26.995 INFO:tasks.workunit.client.1.smithi067.stdout:2/625: readlink d5/d11/l15 0 2024-04-12T05:37:26.996 INFO:tasks.workunit.client.1.smithi067.stdout:7/618: creat d0/d23/d80/da0/db4/dd/d42/fce x:0 0 0 2024-04-12T05:37:26.996 INFO:tasks.workunit.client.1.smithi067.stdout:3/492: mkdir dc/d50/da6 0 2024-04-12T05:37:26.997 INFO:tasks.workunit.client.1.smithi067.stdout:0/663: dread - dd/d12/d14/d1a/fca zero size 2024-04-12T05:37:27.000 INFO:tasks.workunit.client.1.smithi067.stdout:2/626: creat d5/d6a/d8f/fc8 x:0 0 0 2024-04-12T05:37:27.004 INFO:tasks.workunit.client.1.smithi067.stdout:3/493: mkdir dc/d1d/d28/d3e/d53/d76/da7 0 2024-04-12T05:37:27.008 INFO:tasks.workunit.client.1.smithi067.stdout:0/664: getdents dd/dbd/dbe 0 2024-04-12T05:37:27.009 INFO:tasks.workunit.client.1.smithi067.stdout:1/579: dwrite d4/d9/d14/f57 [0,4194304] 0 2024-04-12T05:37:27.010 INFO:tasks.workunit.client.1.smithi067.stdout:3/494: creat dc/df/d27/fa8 x:0 0 0 2024-04-12T05:37:27.021 INFO:tasks.workunit.client.1.smithi067.stdout:0/665: creat dd/d12/d14/d49/dab/fcf x:0 0 0 2024-04-12T05:37:27.022 INFO:tasks.workunit.client.1.smithi067.stdout:2/627: truncate d5/da/de/d14/d34/d4c/f6b 1806906 0 2024-04-12T05:37:27.023 INFO:tasks.workunit.client.1.smithi067.stdout:7/619: dwrite d0/d23/d80/da0/db4/dd/d42/d7e/f4e [0,4194304] 0 2024-04-12T05:37:27.023 INFO:tasks.workunit.client.1.smithi067.stdout:9/560: dwrite d1/d28/d21/d45/d75/fa6 [0,4194304] 0 2024-04-12T05:37:27.029 INFO:tasks.workunit.client.1.smithi067.stdout:0/666: write dd/d6f/d72/f82 [897228,22146] 0 2024-04-12T05:37:27.031 INFO:tasks.workunit.client.1.smithi067.stdout:2/628: rmdir d5/da/d8c 39 2024-04-12T05:37:27.039 INFO:tasks.workunit.client.1.smithi067.stdout:9/561: dread d1/d13/d34/f70 [0,4194304] 0 2024-04-12T05:37:27.040 INFO:tasks.workunit.client.1.smithi067.stdout:1/580: dwrite d4/d9/d12/d2c/f8d [0,4194304] 0 2024-04-12T05:37:27.042 INFO:tasks.workunit.client.1.smithi067.stdout:9/562: fdatasync d1/f6 0 2024-04-12T05:37:27.043 INFO:tasks.workunit.client.1.smithi067.stdout:1/581: readlink d4/d9/d3a/d51/d55/d9a/lb5 0 2024-04-12T05:37:27.047 INFO:tasks.workunit.client.1.smithi067.stdout:0/667: write dd/d12/d14/d49/d9b/fb6 [2323080,107153] 0 2024-04-12T05:37:27.047 INFO:tasks.workunit.client.1.smithi067.stdout:7/620: mkdir d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf 0 2024-04-12T05:37:27.048 INFO:tasks.workunit.client.1.smithi067.stdout:3/495: dwrite dc/d1d/d28/f4c [0,4194304] 0 2024-04-12T05:37:27.051 INFO:tasks.workunit.client.1.smithi067.stdout:1/582: link d4/d8f/fa0 d4/d8f/fbb 0 2024-04-12T05:37:27.053 INFO:tasks.workunit.client.1.smithi067.stdout:2/629: link d5/d6a/d8f/f99 d5/da/dae/dba/fc9 0 2024-04-12T05:37:27.055 INFO:tasks.workunit.client.1.smithi067.stdout:3/496: rmdir dc/df/d27 39 2024-04-12T05:37:27.066 INFO:tasks.workunit.client.1.smithi067.stdout:2/630: rename d5/d23/d3a/l56 to d5/da/de/d14/d46/lca 0 2024-04-12T05:37:27.069 INFO:tasks.workunit.client.1.smithi067.stdout:7/621: rmdir d0/d23/d80/da0/db4/dd/d42/d71/dbd 39 2024-04-12T05:37:27.071 INFO:tasks.workunit.client.1.smithi067.stdout:2/631: creat d5/da/de/d14/d34/d40/fcb x:0 0 0 2024-04-12T05:37:27.071 INFO:tasks.workunit.client.1.smithi067.stdout:1/583: link d4/d3d/dab/f32 d4/d9/d14/fbc 0 2024-04-12T05:37:27.076 INFO:tasks.workunit.client.1.smithi067.stdout:2/632: unlink d5/da/c30 0 2024-04-12T05:37:27.076 INFO:tasks.workunit.client.1.smithi067.stdout:3/497: dwrite dc/d1d/d26/f46 [0,4194304] 0 2024-04-12T05:37:27.082 INFO:tasks.workunit.client.1.smithi067.stdout:9/563: truncate d1/d12/f74 799384 0 2024-04-12T05:37:27.086 INFO:tasks.workunit.client.1.smithi067.stdout:1/584: dread d4/d9/f45 [0,4194304] 0 2024-04-12T05:37:27.087 INFO:tasks.workunit.client.1.smithi067.stdout:7/622: rename d0/d23/d80/da0/db4/dd/da2/f4f to d0/d23/d80/da0/db4/d5/d1d/d43/fd0 0 2024-04-12T05:37:27.088 INFO:tasks.workunit.client.1.smithi067.stdout:3/498: mknod dc/d1d/d28/d3e/ca9 0 2024-04-12T05:37:27.088 INFO:tasks.workunit.client.1.smithi067.stdout:9/564: creat d1/d12/d15/fa9 x:0 0 0 2024-04-12T05:37:27.091 INFO:tasks.workunit.client.1.smithi067.stdout:9/565: chown d1/d12/f2e 0 1 2024-04-12T05:37:27.092 INFO:tasks.workunit.client.1.smithi067.stdout:3/499: chown dc/d1d/d28/d3e/d53 112022978 1 2024-04-12T05:37:27.092 INFO:tasks.workunit.client.1.smithi067.stdout:2/633: mkdir d5/dbf/dcc 0 2024-04-12T05:37:27.093 INFO:tasks.workunit.client.1.smithi067.stdout:0/668: dread dd/f18 [4194304,4194304] 0 2024-04-12T05:37:27.093 INFO:tasks.workunit.client.1.smithi067.stdout:9/566: readlink d1/d28/l30 0 2024-04-12T05:37:27.103 INFO:tasks.workunit.client.1.smithi067.stdout:9/567: unlink d1/d12/f14 0 2024-04-12T05:37:27.107 INFO:tasks.workunit.client.1.smithi067.stdout:7/623: truncate d0/d23/d80/f99 809561 0 2024-04-12T05:37:27.107 INFO:tasks.workunit.client.1.smithi067.stdout:3/500: mkdir dc/df/d27/daa 0 2024-04-12T05:37:27.110 INFO:tasks.workunit.client.1.smithi067.stdout:0/669: write dd/f18 [7026973,68920] 0 2024-04-12T05:37:27.110 INFO:tasks.workunit.client.1.smithi067.stdout:3/501: write dc/d1d/f43 [3386661,39232] 0 2024-04-12T05:37:27.114 INFO:tasks.workunit.client.1.smithi067.stdout:1/585: write d4/d9/f1c [811688,11734] 0 2024-04-12T05:37:27.114 INFO:tasks.workunit.client.1.smithi067.stdout:1/586: truncate d4/d9/d12/d6f/f8c 138118 0 2024-04-12T05:37:27.115 INFO:tasks.workunit.client.1.smithi067.stdout:1/587: chown d4/d9/d4c/lb9 3866174 1 2024-04-12T05:37:27.119 INFO:tasks.workunit.client.1.smithi067.stdout:9/568: unlink d1/d28/d21/d7f/f96 0 2024-04-12T05:37:27.119 INFO:tasks.workunit.client.1.smithi067.stdout:1/588: symlink d4/d9/d3a/d51/d55/d60/lbd 0 2024-04-12T05:37:27.120 INFO:tasks.workunit.client.1.smithi067.stdout:3/502: write f9 [5749527,122354] 0 2024-04-12T05:37:27.123 INFO:tasks.workunit.client.1.smithi067.stdout:7/624: fdatasync d0/d23/d80/da0/db4/dd/d42/d71/dbd/fc1 0 2024-04-12T05:37:27.124 INFO:tasks.workunit.client.1.smithi067.stdout:0/670: creat dd/d12/d14/d49/fd0 x:0 0 0 2024-04-12T05:37:27.124 INFO:tasks.workunit.client.1.smithi067.stdout:9/569: write d1/d12/f61 [387763,12269] 0 2024-04-12T05:37:27.125 INFO:tasks.workunit.client.1.smithi067.stdout:3/503: write dc/df/f2c [2734872,5932] 0 2024-04-12T05:37:27.126 INFO:tasks.workunit.client.1.smithi067.stdout:1/589: mknod d4/d44/cbe 0 2024-04-12T05:37:27.127 INFO:tasks.workunit.client.1.smithi067.stdout:4/516: sync 2024-04-12T05:37:27.127 INFO:tasks.workunit.client.1.smithi067.stdout:5/835: sync 2024-04-12T05:37:27.128 INFO:tasks.workunit.client.1.smithi067.stdout:9/570: fdatasync d1/d28/d23/f31 0 2024-04-12T05:37:27.134 INFO:tasks.workunit.client.1.smithi067.stdout:4/517: mkdir d8/dab 0 2024-04-12T05:37:27.137 INFO:tasks.workunit.client.1.smithi067.stdout:5/836: readlink d0/d3a/dab/lf9 0 2024-04-12T05:37:27.137 INFO:tasks.workunit.client.1.smithi067.stdout:9/571: creat d1/d28/d21/d3f/d4f/faa x:0 0 0 2024-04-12T05:37:27.137 INFO:tasks.workunit.client.1.smithi067.stdout:2/634: dread d5/da/de/d14/f63 [4194304,4194304] 0 2024-04-12T05:37:27.138 INFO:tasks.workunit.client.1.smithi067.stdout:0/671: rename dd/d12/d14/d49/fb1 to dd/fd1 0 2024-04-12T05:37:27.143 INFO:tasks.workunit.client.1.smithi067.stdout:3/504: link dc/d1d/d28/f64 dc/df/d8d/fab 0 2024-04-12T05:37:27.143 INFO:tasks.workunit.client.1.smithi067.stdout:2/635: symlink d5/d43/lcd 0 2024-04-12T05:37:27.145 INFO:tasks.workunit.client.1.smithi067.stdout:9/572: creat d1/d13/d34/fab x:0 0 0 2024-04-12T05:37:27.145 INFO:tasks.workunit.client.1.smithi067.stdout:1/590: dread d4/d9/d14/f21 [0,4194304] 0 2024-04-12T05:37:27.146 INFO:tasks.workunit.client.1.smithi067.stdout:7/625: dwrite d0/d23/f6e [4194304,4194304] 0 2024-04-12T05:37:27.147 INFO:tasks.workunit.client.1.smithi067.stdout:5/837: truncate d0/da/f57 3607990 0 2024-04-12T05:37:27.148 INFO:tasks.workunit.client.1.smithi067.stdout:0/672: truncate dd/f38 123145 0 2024-04-12T05:37:27.152 INFO:tasks.workunit.client.1.smithi067.stdout:3/505: mknod dc/d1d/d26/d52/d9a/cac 0 2024-04-12T05:37:27.152 INFO:tasks.workunit.client.1.smithi067.stdout:7/626: mkdir d0/d23/d45/dd1 0 2024-04-12T05:37:27.153 INFO:tasks.workunit.client.1.smithi067.stdout:2/636: read d5/da/de/d14/d34/f61 [4044452,62926] 0 2024-04-12T05:37:27.153 INFO:tasks.workunit.client.1.smithi067.stdout:7/627: fdatasync d0/f72 0 2024-04-12T05:37:27.155 INFO:tasks.workunit.client.1.smithi067.stdout:9/573: symlink d1/d28/d5e/lac 0 2024-04-12T05:37:27.156 INFO:tasks.workunit.client.1.smithi067.stdout:0/673: rmdir dd/d9c/d5f/d78 39 2024-04-12T05:37:27.158 INFO:tasks.workunit.client.1.smithi067.stdout:1/591: mkdir d4/d3d/dbf 0 2024-04-12T05:37:27.160 INFO:tasks.workunit.client.1.smithi067.stdout:5/838: mkdir d0/da/d111 0 2024-04-12T05:37:27.161 INFO:tasks.workunit.client.1.smithi067.stdout:2/637: symlink d5/da/d3d/lce 0 2024-04-12T05:37:27.162 INFO:tasks.workunit.client.1.smithi067.stdout:3/506: link dc/d1d/d26/f72 dc/d1d/d28/d3e/d53/d76/fad 0 2024-04-12T05:37:27.163 INFO:tasks.workunit.client.1.smithi067.stdout:3/507: chown dc/d1d/d28 1875198800 1 2024-04-12T05:37:27.166 INFO:tasks.workunit.client.1.smithi067.stdout:9/574: dread d1/d5f/f65 [0,4194304] 0 2024-04-12T05:37:27.171 INFO:tasks.workunit.client.1.smithi067.stdout:2/638: rename d5/da/de/d14/d34/d40/d57/cb0 to d5/d6a/d8f/ccf 0 2024-04-12T05:37:27.174 INFO:tasks.workunit.client.1.smithi067.stdout:3/508: dwrite dc/d1d/d26/f2f [0,4194304] 0 2024-04-12T05:37:27.174 INFO:tasks.workunit.client.1.smithi067.stdout:0/674: symlink dd/d12/ld2 0 2024-04-12T05:37:27.177 INFO:tasks.workunit.client.1.smithi067.stdout:3/509: fsync dc/d1d/d28/d3e/d53/d78/f8b 0 2024-04-12T05:37:27.180 INFO:tasks.workunit.client.1.smithi067.stdout:7/628: dread d0/d23/d80/da0/db4/dd/d42/d7e/d34/f8f [0,4194304] 0 2024-04-12T05:37:27.184 INFO:tasks.workunit.client.1.smithi067.stdout:9/575: rename d1/d28/d27/ca4 to d1/d28/d21/d45/d75/cad 0 2024-04-12T05:37:27.187 INFO:tasks.workunit.client.1.smithi067.stdout:3/510: unlink dc/d1d/d28/d3e/d69/f83 0 2024-04-12T05:37:27.191 INFO:tasks.workunit.client.1.smithi067.stdout:5/839: truncate d0/d6/d40/f7f 1749339 0 2024-04-12T05:37:27.193 INFO:tasks.workunit.client.1.smithi067.stdout:9/576: write d1/d12/d15/f1c [2122311,2656] 0 2024-04-12T05:37:27.194 INFO:tasks.workunit.client.1.smithi067.stdout:2/639: truncate d5/da/d3d/f6d 478425 0 2024-04-12T05:37:27.196 INFO:tasks.workunit.client.1.smithi067.stdout:3/511: mknod dc/d1d/d28/d3e/d53/d76/cae 0 2024-04-12T05:37:27.199 INFO:tasks.workunit.client.1.smithi067.stdout:2/640: read d5/d43/f53 [185561,70147] 0 2024-04-12T05:37:27.199 INFO:tasks.workunit.client.1.smithi067.stdout:7/629: link d0/d23/d80/da0/db4/dd/c37 d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/cd2 0 2024-04-12T05:37:27.199 INFO:tasks.workunit.client.1.smithi067.stdout:3/512: stat dc/d1d/f22 0 2024-04-12T05:37:27.202 INFO:tasks.workunit.client.1.smithi067.stdout:3/513: fdatasync dc/d1d/d28/d3e/f66 0 2024-04-12T05:37:27.203 INFO:tasks.workunit.client.1.smithi067.stdout:5/840: dwrite d0/d3a/d2c/d91/fc7 [0,4194304] 0 2024-04-12T05:37:27.203 INFO:tasks.workunit.client.1.smithi067.stdout:9/577: write d1/d28/d21/d45/d75/fa6 [1003941,4960] 0 2024-04-12T05:37:27.205 INFO:tasks.workunit.client.1.smithi067.stdout:0/675: truncate dd/d12/d14/d49/d9b/fb6 5876591 0 2024-04-12T05:37:27.209 INFO:tasks.workunit.client.1.smithi067.stdout:3/514: rmdir dc/df/d8d 39 2024-04-12T05:37:27.210 INFO:tasks.workunit.client.1.smithi067.stdout:3/515: chown dc/d1d/d28/d3e/f67 110 1 2024-04-12T05:37:27.216 INFO:tasks.workunit.client.1.smithi067.stdout:2/641: truncate d5/d23/d3a/f59 230399 0 2024-04-12T05:37:27.219 INFO:tasks.workunit.client.1.smithi067.stdout:5/841: dread d0/d6/d40/d4b/d38/d8e/d8f/fe7 [0,4194304] 0 2024-04-12T05:37:27.222 INFO:tasks.workunit.client.1.smithi067.stdout:9/578: rename d1/d5f/d86 to d1/d28/d21/dae 0 2024-04-12T05:37:27.222 INFO:tasks.workunit.client.1.smithi067.stdout:7/630: dwrite d0/d23/d80/da0/db4/d5/d1d/d7d/dbb/f8e [0,4194304] 0 2024-04-12T05:37:27.224 INFO:tasks.workunit.client.1.smithi067.stdout:3/516: creat dc/faf x:0 0 0 2024-04-12T05:37:27.228 INFO:tasks.workunit.client.1.smithi067.stdout:9/579: write d1/d28/d21/d45/d75/f7d [141811,177] 0 2024-04-12T05:37:27.229 INFO:tasks.workunit.client.1.smithi067.stdout:5/842: link d0/d3a/d2c/dc6/f10b d0/d6/d40/d10a/f112 0 2024-04-12T05:37:27.229 INFO:tasks.workunit.client.1.smithi067.stdout:9/580: chown d1/d12/f53 22361389 1 2024-04-12T05:37:27.230 INFO:tasks.workunit.client.1.smithi067.stdout:7/631: readlink d0/d23/d80/da0/db4/dd/da2/l81 0 2024-04-12T05:37:27.237 INFO:tasks.workunit.client.1.smithi067.stdout:9/581: rename d1/d12/f95 to d1/d28/d21/d3f/d4f/faf 0 2024-04-12T05:37:27.238 INFO:tasks.workunit.client.1.smithi067.stdout:7/632: mkdir d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3 0 2024-04-12T05:37:27.240 INFO:tasks.workunit.client.1.smithi067.stdout:5/843: dwrite d0/da/d41/f7a [0,4194304] 0 2024-04-12T05:37:27.241 INFO:tasks.workunit.client.1.smithi067.stdout:2/642: write d5/da/d3d/f6d [687789,60022] 0 2024-04-12T05:37:27.244 INFO:tasks.workunit.client.1.smithi067.stdout:9/582: dread d1/d28/d23/f31 [0,4194304] 0 2024-04-12T05:37:27.246 INFO:tasks.workunit.client.1.smithi067.stdout:9/583: readlink d1/l39 0 2024-04-12T05:37:27.246 INFO:tasks.workunit.client.1.smithi067.stdout:5/844: dread - d0/d6/d40/d4b/d38/d8e/d8f/db9/dbd/fff zero size 2024-04-12T05:37:27.255 INFO:tasks.workunit.client.1.smithi067.stdout:0/676: dread dd/d6f/d72/f97 [0,4194304] 0 2024-04-12T05:37:27.257 INFO:tasks.workunit.client.1.smithi067.stdout:2/643: rename d5/d43/f84 to d5/d6a/d78/fd0 0 2024-04-12T05:37:27.258 INFO:tasks.workunit.client.1.smithi067.stdout:5/845: symlink d0/d6/d33/d88/l113 0 2024-04-12T05:37:27.258 INFO:tasks.workunit.client.1.smithi067.stdout:9/584: write d1/d28/d23/f90 [320530,108644] 0 2024-04-12T05:37:27.259 INFO:tasks.workunit.client.1.smithi067.stdout:5/846: stat d0 0 2024-04-12T05:37:27.260 INFO:tasks.workunit.client.1.smithi067.stdout:7/633: dwrite d0/d23/d80/da0/db4/d5/d1d/f3a [0,4194304] 0 2024-04-12T05:37:27.262 INFO:tasks.workunit.client.1.smithi067.stdout:9/585: fsync d1/d28/d27/d32/f8c 0 2024-04-12T05:37:27.263 INFO:tasks.workunit.client.1.smithi067.stdout:5/847: rmdir d0/d6 39 2024-04-12T05:37:27.264 INFO:tasks.workunit.client.1.smithi067.stdout:7/634: creat d0/d67/fd4 x:0 0 0 2024-04-12T05:37:27.267 INFO:tasks.workunit.client.1.smithi067.stdout:7/635: readlink d0/d23/d45/l2c 0 2024-04-12T05:37:27.268 INFO:tasks.workunit.client.1.smithi067.stdout:0/677: dwrite dd/d6f/d72/f82 [0,4194304] 0 2024-04-12T05:37:27.271 INFO:tasks.workunit.client.1.smithi067.stdout:5/848: rmdir d0/d6/d40/d4b 39 2024-04-12T05:37:27.274 INFO:tasks.workunit.client.1.smithi067.stdout:7/636: symlink d0/d23/d80/da0/db4/dd/d42/d82/ld5 0 2024-04-12T05:37:27.274 INFO:tasks.workunit.client.1.smithi067.stdout:9/586: unlink d1/d28/d44/l82 0 2024-04-12T05:37:27.274 INFO:tasks.workunit.client.1.smithi067.stdout:2/644: getdents d5/da/de/d14/d46/d90 0 2024-04-12T05:37:27.275 INFO:tasks.workunit.client.1.smithi067.stdout:5/849: mkdir d0/d3a/d114 0 2024-04-12T05:37:27.277 INFO:tasks.workunit.client.1.smithi067.stdout:0/678: getdents dd/dbd 0 2024-04-12T05:37:27.278 INFO:tasks.workunit.client.1.smithi067.stdout:9/587: creat d1/d12/d15/fb0 x:0 0 0 2024-04-12T05:37:27.279 INFO:tasks.workunit.client.1.smithi067.stdout:9/588: chown d1/f47 12961274 1 2024-04-12T05:37:27.280 INFO:tasks.workunit.client.1.smithi067.stdout:9/589: dread - d1/d13/d34/fab zero size 2024-04-12T05:37:27.280 INFO:tasks.workunit.client.1.smithi067.stdout:2/645: mkdir d5/d23/d93/dd1 0 2024-04-12T05:37:27.283 INFO:tasks.workunit.client.1.smithi067.stdout:0/679: unlink dd/d9c/d5f/cc0 0 2024-04-12T05:37:27.288 INFO:tasks.workunit.client.1.smithi067.stdout:0/680: write dd/d6f/fba [589819,96510] 0 2024-04-12T05:37:27.289 INFO:tasks.workunit.client.1.smithi067.stdout:9/590: unlink d1/d28/d21/d3f/f97 0 2024-04-12T05:37:27.290 INFO:tasks.workunit.client.1.smithi067.stdout:0/681: mknod dd/dbd/dbe/cd3 0 2024-04-12T05:37:27.291 INFO:tasks.workunit.client.1.smithi067.stdout:2/646: dread d5/d11/f71 [0,4194304] 0 2024-04-12T05:37:27.292 INFO:tasks.workunit.client.1.smithi067.stdout:5/850: dread d0/d3a/f4c [0,4194304] 0 2024-04-12T05:37:27.295 INFO:tasks.workunit.client.1.smithi067.stdout:2/647: creat d5/da/de/d14/d46/fd2 x:0 0 0 2024-04-12T05:37:27.298 INFO:tasks.workunit.client.1.smithi067.stdout:9/591: dwrite d1/d28/d21/d45/f99 [0,4194304] 0 2024-04-12T05:37:27.298 INFO:tasks.workunit.client.1.smithi067.stdout:2/648: symlink d5/dbf/dcc/ld3 0 2024-04-12T05:37:27.299 INFO:tasks.workunit.client.1.smithi067.stdout:5/851: getdents d0/d6/d40/d4b/d38/d8e 0 2024-04-12T05:37:27.301 INFO:tasks.workunit.client.1.smithi067.stdout:9/592: truncate d1/d13/d34/fab 581771 0 2024-04-12T05:37:27.302 INFO:tasks.workunit.client.1.smithi067.stdout:5/852: truncate d0/d3a/d2c/fcd 55726 0 2024-04-12T05:37:27.302 INFO:tasks.workunit.client.1.smithi067.stdout:2/649: creat d5/da/de/d14/d46/d90/fd4 x:0 0 0 2024-04-12T05:37:27.302 INFO:tasks.workunit.client.1.smithi067.stdout:9/593: symlink d1/d12/d76/lb1 0 2024-04-12T05:37:27.304 INFO:tasks.workunit.client.1.smithi067.stdout:9/594: fdatasync d1/d28/d21/d3f/d4f/f57 0 2024-04-12T05:37:27.305 INFO:tasks.workunit.client.1.smithi067.stdout:8/508: sync 2024-04-12T05:37:27.305 INFO:tasks.workunit.client.1.smithi067.stdout:6/616: sync 2024-04-12T05:37:27.306 INFO:tasks.workunit.client.1.smithi067.stdout:9/595: truncate d1/d28/d21/d3f/d4f/faa 781710 0 2024-04-12T05:37:27.307 INFO:tasks.workunit.client.1.smithi067.stdout:6/617: readlink dd/d21/d3f/lbb 0 2024-04-12T05:37:27.310 INFO:tasks.workunit.client.1.smithi067.stdout:8/509: read - d6/d8/d18/d4e/f75 zero size 2024-04-12T05:37:27.312 INFO:tasks.workunit.client.1.smithi067.stdout:0/682: dread dd/d12/d14/d1a/f93 [0,4194304] 0 2024-04-12T05:37:27.313 INFO:tasks.workunit.client.1.smithi067.stdout:0/683: readlink dd/d6f/la8 0 2024-04-12T05:37:27.315 INFO:tasks.workunit.client.1.smithi067.stdout:8/510: creat d6/d8/d18/d36/f9b x:0 0 0 2024-04-12T05:37:27.316 INFO:tasks.workunit.client.1.smithi067.stdout:8/511: write d6/d8/f92 [970805,87200] 0 2024-04-12T05:37:27.319 INFO:tasks.workunit.client.1.smithi067.stdout:6/618: dread dd/d11/fac [0,4194304] 0 2024-04-12T05:37:27.320 INFO:tasks.workunit.client.1.smithi067.stdout:6/619: readlink dd/d21/d64/d6f/d90/lc6 0 2024-04-12T05:37:27.320 INFO:tasks.workunit.client.1.smithi067.stdout:6/620: chown dd/l6c 1 1 2024-04-12T05:37:27.324 INFO:tasks.workunit.client.1.smithi067.stdout:5/853: dread d0/d6/d40/d4b/fda [0,4194304] 0 2024-04-12T05:37:27.325 INFO:tasks.workunit.client.1.smithi067.stdout:5/854: chown d0/d6/d40/c5b 886934 1 2024-04-12T05:37:27.328 INFO:tasks.workunit.client.1.smithi067.stdout:6/621: dwrite dd/d11/d27/d4a/d79/dc3/fd4 [0,4194304] 0 2024-04-12T05:37:27.331 INFO:tasks.workunit.client.1.smithi067.stdout:5/855: rename d0/da/d3f to d0/d6/d20/d3e/dbf/d115 0 2024-04-12T05:37:27.333 INFO:tasks.workunit.client.1.smithi067.stdout:8/512: fdatasync d6/d8/f92 0 2024-04-12T05:37:27.334 INFO:tasks.workunit.client.1.smithi067.stdout:5/856: fdatasync d0/da/d41/f9b 0 2024-04-12T05:37:27.336 INFO:tasks.workunit.client.1.smithi067.stdout:8/513: write d6/d8/d18/d63/d9a/f5f [322435,52883] 0 2024-04-12T05:37:27.339 INFO:tasks.workunit.client.1.smithi067.stdout:7/637: truncate d0/d23/d80/da0/db4/d5/d1d/d43/fd0 275013 0 2024-04-12T05:37:27.340 INFO:tasks.workunit.client.1.smithi067.stdout:8/514: fdatasync d6/d86/f99 0 2024-04-12T05:37:27.341 INFO:tasks.workunit.client.1.smithi067.stdout:7/638: stat d0/d23/d80/da0/db4/f21 0 2024-04-12T05:37:27.341 INFO:tasks.workunit.client.1.smithi067.stdout:5/857: mknod d0/da/d41/d7d/c116 0 2024-04-12T05:37:27.341 INFO:tasks.workunit.client.1.smithi067.stdout:8/515: write d6/f95 [344672,96117] 0 2024-04-12T05:37:27.344 INFO:tasks.workunit.client.1.smithi067.stdout:8/516: symlink d6/d86/d4f/l9c 0 2024-04-12T05:37:27.345 INFO:tasks.workunit.client.1.smithi067.stdout:9/596: rmdir d1/d12/d76 39 2024-04-12T05:37:27.345 INFO:tasks.workunit.client.1.smithi067.stdout:5/858: mknod d0/da/dec/c117 0 2024-04-12T05:37:27.350 INFO:tasks.workunit.client.1.smithi067.stdout:7/639: rename d0/d23/d80/da0/db4/d5/d1d/d7d/dbb to d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/dd6 0 2024-04-12T05:37:27.350 INFO:tasks.workunit.client.1.smithi067.stdout:8/517: creat d6/d86/d4f/d82/f9d x:0 0 0 2024-04-12T05:37:27.355 INFO:tasks.workunit.client.1.smithi067.stdout:5/859: dread d0/d3a/d2c/f100 [4194304,4194304] 0 2024-04-12T05:37:27.356 INFO:tasks.workunit.client.1.smithi067.stdout:2/650: dwrite d5/d23/d3a/f59 [0,4194304] 0 2024-04-12T05:37:27.360 INFO:tasks.workunit.client.1.smithi067.stdout:5/860: mknod d0/d3a/d114/c118 0 2024-04-12T05:37:27.365 INFO:tasks.workunit.client.1.smithi067.stdout:6/622: dwrite dd/d11/d27/d4a/d79/dc3/d56/f99 [0,4194304] 0 2024-04-12T05:37:27.367 INFO:tasks.workunit.client.1.smithi067.stdout:5/861: unlink d0/d6/d40/d4b/d38/d8e/c103 0 2024-04-12T05:37:27.369 INFO:tasks.workunit.client.1.smithi067.stdout:6/623: chown fb 30242 1 2024-04-12T05:37:27.371 INFO:tasks.workunit.client.1.smithi067.stdout:5/862: mkdir d0/d3a/d2c/db3/d119 0 2024-04-12T05:37:27.372 INFO:tasks.workunit.client.1.smithi067.stdout:6/624: mkdir dd/d21/d3f/dd5 0 2024-04-12T05:37:27.373 INFO:tasks.workunit.client.1.smithi067.stdout:7/640: dwrite d0/d23/d45/f59 [0,4194304] 0 2024-04-12T05:37:27.373 INFO:tasks.workunit.client.1.smithi067.stdout:2/651: truncate d5/d11/f2a 709166 0 2024-04-12T05:37:27.378 INFO:tasks.workunit.client.1.smithi067.stdout:7/641: chown d0/d23/d80/da0/db4/dd/da2/d51/caa 22368 1 2024-04-12T05:37:27.381 INFO:tasks.workunit.client.1.smithi067.stdout:2/652: write d5/da/de/d39/f95 [4683469,120603] 0 2024-04-12T05:37:27.382 INFO:tasks.workunit.client.1.smithi067.stdout:5/863: rename d0/d6/d20/d3e/l8d to d0/d6/d40/d4b/l11a 0 2024-04-12T05:37:27.382 INFO:tasks.workunit.client.1.smithi067.stdout:7/642: read d0/d23/d80/da0/db4/d5/d1d/d7d/f84 [172069,89680] 0 2024-04-12T05:37:27.384 INFO:tasks.workunit.client.1.smithi067.stdout:2/653: fdatasync d5/d11/f71 0 2024-04-12T05:37:27.389 INFO:tasks.workunit.client.1.smithi067.stdout:5/864: chown d0/d6/d40/d4b/d38/d8e/d8f 0 1 2024-04-12T05:37:27.390 INFO:tasks.workunit.client.1.smithi067.stdout:5/865: readlink d0/d6/d33/ld7 0 2024-04-12T05:37:27.391 INFO:tasks.workunit.client.1.smithi067.stdout:2/654: getdents d5/da/de/dc4 0 2024-04-12T05:37:27.392 INFO:tasks.workunit.client.1.smithi067.stdout:2/655: fsync d5/f12 0 2024-04-12T05:37:27.392 INFO:tasks.workunit.client.1.smithi067.stdout:5/866: read d0/d6/d33/dd3/f89 [62963,118683] 0 2024-04-12T05:37:27.394 INFO:tasks.workunit.client.1.smithi067.stdout:7/643: fsync d0/d23/d45/f59 0 2024-04-12T05:37:27.395 INFO:tasks.workunit.client.1.smithi067.stdout:5/867: creat d0/d6/d40/d4b/f11b x:0 0 0 2024-04-12T05:37:27.399 INFO:tasks.workunit.client.1.smithi067.stdout:7/644: mknod d0/d23/d80/da0/db4/d5/d4a/cd7 0 2024-04-12T05:37:27.402 INFO:tasks.workunit.client.1.smithi067.stdout:5/868: creat d0/d6/d40/d4b/d38/d8e/f11c x:0 0 0 2024-04-12T05:37:27.405 INFO:tasks.workunit.client.1.smithi067.stdout:7/645: mkdir d0/d23/d80/da0/db4/dd/d42/d7e/dd8 0 2024-04-12T05:37:27.406 INFO:tasks.workunit.client.1.smithi067.stdout:5/869: chown d0/d3a/d2c/dc6/ddd/fe3 2330904 1 2024-04-12T05:37:27.407 INFO:tasks.workunit.client.1.smithi067.stdout:6/625: dwrite dd/d21/d3f/f40 [0,4194304] 0 2024-04-12T05:37:27.408 INFO:tasks.workunit.client.1.smithi067.stdout:5/870: fsync d0/d6/d40/fd4 0 2024-04-12T05:37:27.410 INFO:tasks.workunit.client.1.smithi067.stdout:2/656: dread d5/da/de/d14/d34/d4c/f5f [0,4194304] 0 2024-04-12T05:37:27.412 INFO:tasks.workunit.client.1.smithi067.stdout:6/626: write dd/d11/d27/f54 [193585,34632] 0 2024-04-12T05:37:27.420 INFO:tasks.workunit.client.1.smithi067.stdout:5/871: dwrite d0/d6/d40/fd4 [0,4194304] 0 2024-04-12T05:37:27.424 INFO:tasks.workunit.client.1.smithi067.stdout:7/646: truncate d0/d23/d80/da0/db4/dd/d42/d7e/d34/f8f 2630554 0 2024-04-12T05:37:27.426 INFO:tasks.workunit.client.1.smithi067.stdout:5/872: creat d0/d6/f11d x:0 0 0 2024-04-12T05:37:27.427 INFO:tasks.workunit.client.1.smithi067.stdout:6/627: dwrite fa [0,4194304] 0 2024-04-12T05:37:27.427 INFO:tasks.workunit.client.1.smithi067.stdout:7/647: mkdir d0/d23/d80/dd9 0 2024-04-12T05:37:27.434 INFO:tasks.workunit.client.1.smithi067.stdout:7/648: rename d0/d23/d80/da0/db4/dd/c37 to d0/d23/d80/da0/db4/dd/cda 0 2024-04-12T05:37:27.438 INFO:tasks.workunit.client.1.smithi067.stdout:7/649: mknod d0/d23/cdb 0 2024-04-12T05:37:27.441 INFO:tasks.workunit.client.1.smithi067.stdout:7/650: rmdir d0/d67 39 2024-04-12T05:37:27.448 INFO:tasks.workunit.client.1.smithi067.stdout:5/873: dread d0/f43 [0,4194304] 0 2024-04-12T05:37:27.448 INFO:tasks.workunit.client.1.smithi067.stdout:7/651: write d0/d23/d80/da0/db4/dd/d79/f6d [1033369,69860] 0 2024-04-12T05:37:27.448 INFO:tasks.workunit.client.1.smithi067.stdout:6/628: dread dd/d11/d27/d4a/d79/dc3/f43 [0,4194304] 0 2024-04-12T05:37:27.448 INFO:tasks.workunit.client.1.smithi067.stdout:5/874: read d0/d6/d33/f52 [1087307,88499] 0 2024-04-12T05:37:27.448 INFO:tasks.workunit.client.1.smithi067.stdout:6/629: getdents dd/d11/dc1 0 2024-04-12T05:37:27.449 INFO:tasks.workunit.client.1.smithi067.stdout:5/875: write d0/d6/d33/f71 [2601631,112336] 0 2024-04-12T05:37:27.455 INFO:tasks.workunit.client.1.smithi067.stdout:7/652: link d0/d67/c35 d0/d23/d80/da0/db4/dd/cdc 0 2024-04-12T05:37:27.457 INFO:tasks.workunit.client.1.smithi067.stdout:5/876: write d0/d6/f76 [453425,42205] 0 2024-04-12T05:37:27.458 INFO:tasks.workunit.client.1.smithi067.stdout:7/653: creat d0/d23/d80/da0/db4/dd/d79/fdd x:0 0 0 2024-04-12T05:37:27.467 INFO:tasks.workunit.client.1.smithi067.stdout:7/654: dwrite d0/d23/d80/da0/db4/d5/f50 [0,4194304] 0 2024-04-12T05:37:27.468 INFO:tasks.workunit.client.1.smithi067.stdout:5/877: dwrite d0/d6/d40/d4b/d38/d8e/ffb [0,4194304] 0 2024-04-12T05:37:27.470 INFO:tasks.workunit.client.1.smithi067.stdout:5/878: write d0/d6/d40/d4b/fe0 [642039,70408] 0 2024-04-12T05:37:27.474 INFO:tasks.workunit.client.1.smithi067.stdout:5/879: write d0/d3a/d2c/dc6/ddd/f102 [802514,63118] 0 2024-04-12T05:37:27.475 INFO:tasks.workunit.client.1.smithi067.stdout:7/655: mknod d0/d23/d80/da0/db4/d5/d1d/cde 0 2024-04-12T05:37:27.479 INFO:tasks.workunit.client.1.smithi067.stdout:7/656: mkdir d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/dd6/ddf 0 2024-04-12T05:37:27.480 INFO:tasks.workunit.client.1.smithi067.stdout:5/880: creat d0/d6/d40/d4b/d38/d8e/d8f/db9/f11e x:0 0 0 2024-04-12T05:37:27.482 INFO:tasks.workunit.client.1.smithi067.stdout:7/657: symlink d0/d23/d80/da0/db4/d5/da1/le0 0 2024-04-12T05:37:27.483 INFO:tasks.workunit.client.1.smithi067.stdout:5/881: chown d0/d6/d20/d3e/dbf/d115/daa/lc4 113948806 1 2024-04-12T05:37:27.491 INFO:tasks.workunit.client.1.smithi067.stdout:7/658: dwrite d0/f1 [0,4194304] 0 2024-04-12T05:37:27.492 INFO:tasks.workunit.client.1.smithi067.stdout:5/882: dwrite d0/da/d41/f59 [0,4194304] 0 2024-04-12T05:37:27.495 INFO:tasks.workunit.client.1.smithi067.stdout:7/659: mkdir d0/d23/d80/da0/db4/dd/da2/d51/db0/de1 0 2024-04-12T05:37:27.506 INFO:tasks.workunit.client.1.smithi067.stdout:7/660: dwrite d0/d23/d45/f9e [0,4194304] 0 2024-04-12T05:37:27.512 INFO:tasks.workunit.client.1.smithi067.stdout:7/661: dread d0/d23/d80/da0/db4/dd/d42/d7e/d34/f8f [0,4194304] 0 2024-04-12T05:37:27.516 INFO:tasks.workunit.client.1.smithi067.stdout:7/662: creat d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/dd6/ddf/fe2 x:0 0 0 2024-04-12T05:37:27.518 INFO:tasks.workunit.client.1.smithi067.stdout:7/663: fdatasync d0/d23/d80/da0/db4/dd/da2/d51/db0/fcc 0 2024-04-12T05:37:27.526 INFO:tasks.workunit.client.1.smithi067.stdout:7/664: dwrite d0/d23/d80/fcb [0,4194304] 0 2024-04-12T05:37:27.532 INFO:tasks.workunit.client.1.smithi067.stdout:7/665: link d0/d23/d80/da0/db4/d5/da1/le0 d0/d23/d80/da0/db4/d5/d4a/le3 0 2024-04-12T05:37:27.535 INFO:tasks.workunit.client.1.smithi067.stdout:7/666: truncate d0/d23/d80/da0/db4/dd/da2/d51/db0/fba 213691 0 2024-04-12T05:37:27.537 INFO:tasks.workunit.client.1.smithi067.stdout:7/667: unlink d0/d23/f4d 0 2024-04-12T05:37:27.539 INFO:tasks.workunit.client.1.smithi067.stdout:7/668: mkdir d0/d23/d80/da0/db4/dd/d42/d82/d96/de4 0 2024-04-12T05:37:27.541 INFO:tasks.workunit.client.1.smithi067.stdout:7/669: read - d0/d23/d80/da0/db4/dd/da2/f9b zero size 2024-04-12T05:37:27.543 INFO:tasks.workunit.client.1.smithi067.stdout:7/670: mkdir d0/d23/d80/da0/db4/d5/d4a/de5 0 2024-04-12T05:37:27.546 INFO:tasks.workunit.client.1.smithi067.stdout:7/671: symlink d0/d23/d80/da0/db4/d5/d1d/d43/le6 0 2024-04-12T05:37:27.549 INFO:tasks.workunit.client.1.smithi067.stdout:7/672: mkdir d0/d23/d80/da0/db4/dd/da2/d51/db0/de7 0 2024-04-12T05:37:27.550 INFO:tasks.workunit.client.1.smithi067.stdout:7/673: write d0/d23/d80/f9c [223418,12846] 0 2024-04-12T05:37:27.564 INFO:tasks.workunit.client.1.smithi067.stdout:7/674: dread d0/d23/d80/da0/db4/dd/d79/f6a [0,4194304] 0 2024-04-12T05:37:27.566 INFO:tasks.workunit.client.1.smithi067.stdout:7/675: mknod d0/d23/d80/da0/db4/dd/d42/d7e/d34/da3/ce8 0 2024-04-12T05:37:27.571 INFO:tasks.workunit.client.1.smithi067.stdout:7/676: rename d0/d23/d80/da0/db4/dd/da2/d51/db0 to d0/d23/d80/da0/db4/d5/d1d/de9 0 2024-04-12T05:37:27.575 INFO:tasks.workunit.client.1.smithi067.stdout:7/677: dread d0/d23/d80/f99 [0,4194304] 0 2024-04-12T05:37:27.583 INFO:tasks.workunit.client.1.smithi067.stdout:7/678: dwrite d0/d23/d80/da0/db4/dd/d42/d7e/f94 [0,4194304] 0 2024-04-12T05:37:27.587 INFO:tasks.workunit.client.1.smithi067.stdout:7/679: mkdir d0/d23/d80/da0/db4/dd/d42/d7e/d34/da3/dea 0 2024-04-12T05:37:27.589 INFO:tasks.workunit.client.1.smithi067.stdout:7/680: mkdir d0/d23/d80/da0/deb 0 2024-04-12T05:37:27.607 INFO:tasks.workunit.client.1.smithi067.stdout:7/681: dread d0/d23/d80/da0/db4/dd/d79/f63 [0,4194304] 0 2024-04-12T05:37:27.608 INFO:tasks.workunit.client.1.smithi067.stdout:7/682: rmdir d0 39 2024-04-12T05:37:27.612 INFO:tasks.workunit.client.1.smithi067.stdout:7/683: unlink d0/d23/d80/da0/db4/d5/d4a/le3 0 2024-04-12T05:37:27.614 INFO:tasks.workunit.client.1.smithi067.stdout:7/684: truncate d0/d23/d80/da0/db4/dd/da2/f9b 507329 0 2024-04-12T05:37:27.620 INFO:tasks.workunit.client.1.smithi067.stdout:7/685: rename d0/d23/d80/dd9 to d0/d23/d80/da0/db4/d5/d1d/dec 0 2024-04-12T05:37:27.621 INFO:tasks.workunit.client.1.smithi067.stdout:7/686: read - d0/d23/d80/da0/db4/f52 zero size 2024-04-12T05:37:27.622 INFO:tasks.workunit.client.1.smithi067.stdout:7/687: truncate d0/d23/d80/da0/db4/dd/fcd 665656 0 2024-04-12T05:37:27.629 INFO:tasks.workunit.client.1.smithi067.stdout:7/688: rename d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/dd6/fad to d0/d23/d80/da0/db4/d5/d1d/fed 0 2024-04-12T05:37:27.631 INFO:tasks.workunit.client.1.smithi067.stdout:7/689: read - d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/dd6/fc4 zero size 2024-04-12T05:37:27.634 INFO:tasks.workunit.client.1.smithi067.stdout:7/690: creat d0/d23/d80/da0/db4/fee x:0 0 0 2024-04-12T05:37:27.639 INFO:tasks.workunit.client.1.smithi067.stdout:7/691: link d0/d23/d80/da0/db4/dd/d42/d71/dbd/fc1 d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/dd6/ddf/fef 0 2024-04-12T05:37:27.640 INFO:tasks.workunit.client.1.smithi067.stdout:7/692: chown d0/d23/d80/da0/db4/d5/d1d/de9 1934 1 2024-04-12T05:37:27.735 INFO:tasks.workunit.client.1.smithi067.stdout:1/592: sync 2024-04-12T05:37:27.735 INFO:tasks.workunit.client.1.smithi067.stdout:4/518: sync 2024-04-12T05:37:27.736 INFO:tasks.workunit.client.1.smithi067.stdout:1/593: fsync d4/d9/d3a/d51/f63 0 2024-04-12T05:37:27.740 INFO:tasks.workunit.client.1.smithi067.stdout:4/519: creat d8/da/d9f/d68/d8e/fac x:0 0 0 2024-04-12T05:37:27.746 INFO:tasks.workunit.client.1.smithi067.stdout:4/520: rename d8/d9/f20 to d8/da/d1d/d27/d91/fad 0 2024-04-12T05:37:27.750 INFO:tasks.workunit.client.1.smithi067.stdout:1/594: dread d4/d52/d80/f79 [0,4194304] 0 2024-04-12T05:37:27.751 INFO:tasks.workunit.client.1.smithi067.stdout:1/595: write d4/d9/d12/d2c/fb0 [324182,71730] 0 2024-04-12T05:37:27.758 INFO:tasks.workunit.client.1.smithi067.stdout:4/521: dwrite d8/f33 [4194304,4194304] 0 2024-04-12T05:37:27.758 INFO:tasks.workunit.client.1.smithi067.stdout:1/596: write d4/d3d/dab/f32 [457661,71113] 0 2024-04-12T05:37:27.761 INFO:tasks.workunit.client.1.smithi067.stdout:1/597: symlink d4/d46/lc0 0 2024-04-12T05:37:27.762 INFO:tasks.workunit.client.1.smithi067.stdout:4/522: rename d8/da/d9f/d68/l82 to d8/da/d8b/d8c/lae 0 2024-04-12T05:37:27.766 INFO:tasks.workunit.client.1.smithi067.stdout:4/523: creat d8/da/d1d/d15/d36/d5d/faf x:0 0 0 2024-04-12T05:37:27.770 INFO:tasks.workunit.client.1.smithi067.stdout:1/598: dwrite d4/d9/d12/d2c/f88 [0,4194304] 0 2024-04-12T05:37:27.771 INFO:tasks.workunit.client.1.smithi067.stdout:4/524: fdatasync d8/d9/d3d/f4e 0 2024-04-12T05:37:27.776 INFO:tasks.workunit.client.1.smithi067.stdout:4/525: symlink d8/da/d1d/d15/d36/d5d/d66/d6a/lb0 0 2024-04-12T05:37:27.778 INFO:tasks.workunit.client.1.smithi067.stdout:4/526: fdatasync d8/da/d9f/f52 0 2024-04-12T05:37:27.778 INFO:tasks.workunit.client.1.smithi067.stdout:1/599: dread d4/d9/f22 [0,4194304] 0 2024-04-12T05:37:27.779 INFO:tasks.workunit.client.1.smithi067.stdout:4/527: write f7 [1362433,1640] 0 2024-04-12T05:37:27.780 INFO:tasks.workunit.client.1.smithi067.stdout:1/600: stat d4/d9/d14/f21 0 2024-04-12T05:37:27.781 INFO:tasks.workunit.client.1.smithi067.stdout:4/528: write d8/da/d1d/d27/d91/fa8 [600185,69390] 0 2024-04-12T05:37:27.782 INFO:tasks.workunit.client.1.smithi067.stdout:4/529: write d8/da/f30 [700133,23361] 0 2024-04-12T05:37:27.784 INFO:tasks.workunit.client.1.smithi067.stdout:1/601: unlink d4/d3d/dab/f33 0 2024-04-12T05:37:27.785 INFO:tasks.workunit.client.1.smithi067.stdout:4/530: rmdir d8/da/d8b/d45 39 2024-04-12T05:37:27.787 INFO:tasks.workunit.client.1.smithi067.stdout:1/602: fdatasync d4/f8 0 2024-04-12T05:37:27.795 INFO:tasks.workunit.client.1.smithi067.stdout:4/531: dwrite d8/da/d1d/d27/d91/fa8 [0,4194304] 0 2024-04-12T05:37:27.800 INFO:tasks.workunit.client.1.smithi067.stdout:1/603: dwrite d4/d8f/fa0 [0,4194304] 0 2024-04-12T05:37:27.802 INFO:tasks.workunit.client.1.smithi067.stdout:1/604: chown d4/d9/d3a/d51/c84 1 1 2024-04-12T05:37:27.803 INFO:tasks.workunit.client.1.smithi067.stdout:1/605: dread - d4/d3d/dab/fb1 zero size 2024-04-12T05:37:27.805 INFO:tasks.workunit.client.1.smithi067.stdout:4/532: truncate d8/d9/d3d/d64/f97 75982 0 2024-04-12T05:37:27.808 INFO:tasks.workunit.client.1.smithi067.stdout:4/533: dread - d8/da/d1d/d15/d36/d5d/f74 zero size 2024-04-12T05:37:27.810 INFO:tasks.workunit.client.1.smithi067.stdout:1/606: dread d4/d9/d12/d2c/f88 [0,4194304] 0 2024-04-12T05:37:27.813 INFO:tasks.workunit.client.1.smithi067.stdout:1/607: creat d4/d9/d4c/fc1 x:0 0 0 2024-04-12T05:37:27.814 INFO:tasks.workunit.client.1.smithi067.stdout:1/608: readlink d4/d9/d12/d59/l86 0 2024-04-12T05:37:27.814 INFO:tasks.workunit.client.1.smithi067.stdout:4/534: rename d8/d39/f51 to d8/da/d1d/d15/d58/fb1 0 2024-04-12T05:37:27.817 INFO:tasks.workunit.client.1.smithi067.stdout:4/535: read d8/da/d1d/d15/d58/f9b [332748,54379] 0 2024-04-12T05:37:27.819 INFO:tasks.workunit.client.1.smithi067.stdout:1/609: getdents d4/d9/d14 0 2024-04-12T05:37:27.821 INFO:tasks.workunit.client.1.smithi067.stdout:4/536: write d8/da/d8b/d45/f6e [1132803,124244] 0 2024-04-12T05:37:27.824 INFO:tasks.workunit.client.1.smithi067.stdout:4/537: fsync d8/da/f18 0 2024-04-12T05:37:27.828 INFO:tasks.workunit.client.1.smithi067.stdout:4/538: rename d8/d39/c50 to d8/da/d1d/d15/d75/cb2 0 2024-04-12T05:37:27.836 INFO:tasks.workunit.client.1.smithi067.stdout:4/539: dwrite d8/da/d1d/d27/d91/fad [0,4194304] 0 2024-04-12T05:37:27.836 INFO:tasks.workunit.client.1.smithi067.stdout:1/610: dread d4/d9/d3a/d51/f63 [4194304,4194304] 0 2024-04-12T05:37:27.840 INFO:tasks.workunit.client.1.smithi067.stdout:1/611: fdatasync d4/d9/f22 0 2024-04-12T05:37:27.840 INFO:tasks.workunit.client.1.smithi067.stdout:4/540: creat d8/da/d1d/d15/d75/daa/fb3 x:0 0 0 2024-04-12T05:37:27.842 INFO:tasks.workunit.client.1.smithi067.stdout:1/612: read d4/d9/d4c/f89 [2864906,108061] 0 2024-04-12T05:37:27.842 INFO:tasks.workunit.client.1.smithi067.stdout:4/541: write d8/da/d1d/d15/d36/d5d/faf [974451,79751] 0 2024-04-12T05:37:27.843 INFO:tasks.workunit.client.1.smithi067.stdout:1/613: fdatasync d4/d3d/dab/f83 0 2024-04-12T05:37:27.847 INFO:tasks.workunit.client.1.smithi067.stdout:1/614: creat d4/d52/fc2 x:0 0 0 2024-04-12T05:37:27.850 INFO:tasks.workunit.client.1.smithi067.stdout:1/615: mknod d4/d9/d12/d59/cc3 0 2024-04-12T05:37:27.852 INFO:tasks.workunit.client.1.smithi067.stdout:1/616: chown d4/d9/d4d 61 1 2024-04-12T05:37:27.852 INFO:tasks.workunit.client.1.smithi067.stdout:4/542: dwrite d8/da/d1d/d15/d58/f59 [0,4194304] 0 2024-04-12T05:37:27.854 INFO:tasks.workunit.client.1.smithi067.stdout:1/617: unlink d4/d44/c49 0 2024-04-12T05:37:27.856 INFO:tasks.workunit.client.1.smithi067.stdout:1/618: stat d4/d9/d4d 0 2024-04-12T05:37:27.857 INFO:tasks.workunit.client.1.smithi067.stdout:4/543: rmdir d8/da/d1d/d27/d7d 0 2024-04-12T05:37:27.873 INFO:tasks.workunit.client.1.smithi067.stdout:1/619: write d4/d3d/dab/f4f [1011876,71522] 0 2024-04-12T05:37:27.875 INFO:tasks.workunit.client.1.smithi067.stdout:1/620: mkdir d4/d9/d3a/dc4 0 2024-04-12T05:37:27.877 INFO:tasks.workunit.client.1.smithi067.stdout:1/621: symlink d4/d52/lc5 0 2024-04-12T05:37:27.883 INFO:tasks.workunit.client.1.smithi067.stdout:1/622: truncate d4/d9/d4c/f89 3871097 0 2024-04-12T05:37:27.884 INFO:tasks.workunit.client.1.smithi067.stdout:4/544: dread d8/d9/d3d/f3f [0,4194304] 0 2024-04-12T05:37:27.886 INFO:tasks.workunit.client.1.smithi067.stdout:4/545: mkdir d8/da/d9f/d68/db4 0 2024-04-12T05:37:27.886 INFO:tasks.workunit.client.1.smithi067.stdout:1/623: symlink d4/d9/d3a/d51/d55/d60/d90/lc6 0 2024-04-12T05:37:27.887 INFO:tasks.workunit.client.1.smithi067.stdout:4/546: chown d8/d9/f67 0 1 2024-04-12T05:37:27.888 INFO:tasks.workunit.client.1.smithi067.stdout:1/624: creat d4/d52/fc7 x:0 0 0 2024-04-12T05:37:27.890 INFO:tasks.workunit.client.1.smithi067.stdout:4/547: mkdir d8/da/d9f/db5 0 2024-04-12T05:37:27.892 INFO:tasks.workunit.client.1.smithi067.stdout:4/548: symlink d8/da/d9f/lb6 0 2024-04-12T05:37:27.894 INFO:tasks.workunit.client.1.smithi067.stdout:1/625: dwrite d4/d9/d12/d2c/f6c [0,4194304] 0 2024-04-12T05:37:27.896 INFO:tasks.workunit.client.1.smithi067.stdout:1/626: chown d4/d9/d3a/d51/d55/lba 7834587 1 2024-04-12T05:37:27.896 INFO:tasks.workunit.client.1.smithi067.stdout:4/549: unlink d8/da/d9f/f65 0 2024-04-12T05:37:27.899 INFO:tasks.workunit.client.1.smithi067.stdout:1/627: write d4/d9/f75 [1046636,82784] 0 2024-04-12T05:37:27.900 INFO:tasks.workunit.client.1.smithi067.stdout:4/550: creat d8/d9/d3d/d64/fb7 x:0 0 0 2024-04-12T05:37:27.900 INFO:tasks.workunit.client.1.smithi067.stdout:1/628: read - d4/d3d/fae zero size 2024-04-12T05:37:27.903 INFO:tasks.workunit.client.1.smithi067.stdout:4/551: truncate d8/da/d1d/d15/d36/f5a 1026051 0 2024-04-12T05:37:27.903 INFO:tasks.workunit.client.1.smithi067.stdout:1/629: creat d4/d9/d3a/d51/d55/fc8 x:0 0 0 2024-04-12T05:37:27.911 INFO:tasks.workunit.client.1.smithi067.stdout:4/552: dread d8/da/d8b/f57 [4194304,4194304] 0 2024-04-12T05:37:27.924 INFO:tasks.workunit.client.1.smithi067.stdout:4/553: dread d8/da/d1d/f49 [0,4194304] 0 2024-04-12T05:37:27.925 INFO:tasks.workunit.client.1.smithi067.stdout:4/554: write f7 [1698277,126882] 0 2024-04-12T05:37:27.929 INFO:tasks.workunit.client.1.smithi067.stdout:4/555: mkdir d8/da/d1d/d15/d36/db8 0 2024-04-12T05:37:27.936 INFO:tasks.workunit.client.1.smithi067.stdout:4/556: dwrite d8/f33 [4194304,4194304] 0 2024-04-12T05:37:27.938 INFO:tasks.workunit.client.1.smithi067.stdout:4/557: write d8/da/d1d/d15/d36/d5d/faf [424226,104265] 0 2024-04-12T05:37:27.945 INFO:tasks.workunit.client.1.smithi067.stdout:4/558: dread d8/d9/f43 [0,4194304] 0 2024-04-12T05:37:27.947 INFO:tasks.workunit.client.1.smithi067.stdout:4/559: creat d8/d39/fb9 x:0 0 0 2024-04-12T05:37:27.999 INFO:tasks.workunit.client.1.smithi067.stdout:3/517: sync 2024-04-12T05:37:28.001 INFO:tasks.workunit.client.1.smithi067.stdout:3/518: mknod dc/d1d/d26/d52/d9a/d9c/cb0 0 2024-04-12T05:37:28.001 INFO:tasks.workunit.client.1.smithi067.stdout:3/519: chown dc/d1d/d28 3 1 2024-04-12T05:37:28.005 INFO:tasks.workunit.client.1.smithi067.stdout:3/520: link dc/f3a dc/d1d/d26/fb1 0 2024-04-12T05:37:28.007 INFO:tasks.workunit.client.1.smithi067.stdout:3/521: symlink dc/df/lb2 0 2024-04-12T05:37:28.009 INFO:tasks.workunit.client.1.smithi067.stdout:3/522: creat dc/df/d27/daa/fb3 x:0 0 0 2024-04-12T05:37:28.010 INFO:tasks.workunit.client.1.smithi067.stdout:3/523: rmdir dc/d50 39 2024-04-12T05:37:28.016 INFO:tasks.workunit.client.1.smithi067.stdout:3/524: dwrite dc/f18 [0,4194304] 0 2024-04-12T05:37:28.019 INFO:tasks.workunit.client.1.smithi067.stdout:3/525: creat dc/d1d/d28/d3e/d53/d76/fb4 x:0 0 0 2024-04-12T05:37:28.159 INFO:tasks.workunit.client.1.smithi067.stdout:8/518: sync 2024-04-12T05:37:28.160 INFO:tasks.workunit.client.1.smithi067.stdout:9/597: sync 2024-04-12T05:37:28.160 INFO:tasks.workunit.client.1.smithi067.stdout:0/684: sync 2024-04-12T05:37:28.160 INFO:tasks.workunit.client.1.smithi067.stdout:5/883: sync 2024-04-12T05:37:28.162 INFO:tasks.workunit.client.1.smithi067.stdout:9/598: write d1/d13/d34/f6c [2091432,13348] 0 2024-04-12T05:37:28.162 INFO:tasks.workunit.client.1.smithi067.stdout:8/519: rmdir d6/d8/d50 39 2024-04-12T05:37:28.163 INFO:tasks.workunit.client.1.smithi067.stdout:5/884: chown d0/d6/d20/d3e/cc3 66 1 2024-04-12T05:37:28.167 INFO:tasks.workunit.client.1.smithi067.stdout:5/885: creat d0/d3a/d2c/dc6/f11f x:0 0 0 2024-04-12T05:37:28.168 INFO:tasks.workunit.client.1.smithi067.stdout:8/520: mkdir d6/d8/d18/d4e/d9e 0 2024-04-12T05:37:28.169 INFO:tasks.workunit.client.1.smithi067.stdout:5/886: write d0/d6/d40/f90 [1221531,53956] 0 2024-04-12T05:37:28.172 INFO:tasks.workunit.client.1.smithi067.stdout:5/887: read d0/d6/fc8 [174354,3754] 0 2024-04-12T05:37:28.179 INFO:tasks.workunit.client.1.smithi067.stdout:8/521: symlink d6/d8/d18/d4e/d9e/l9f 0 2024-04-12T05:37:28.183 INFO:tasks.workunit.client.1.smithi067.stdout:8/522: write d6/d8/d8b/f8d [674267,35453] 0 2024-04-12T05:37:28.186 INFO:tasks.workunit.client.1.smithi067.stdout:5/888: dwrite d0/da/d41/f7a [0,4194304] 0 2024-04-12T05:37:28.188 INFO:tasks.workunit.client.1.smithi067.stdout:9/599: dread d1/d28/d21/d45/d75/fa6 [0,4194304] 0 2024-04-12T05:37:28.190 INFO:tasks.workunit.client.1.smithi067.stdout:8/523: creat d6/d86/d4f/fa0 x:0 0 0 2024-04-12T05:37:28.190 INFO:tasks.workunit.client.1.smithi067.stdout:5/889: chown d0/d6/d33 11476362 1 2024-04-12T05:37:28.192 INFO:tasks.workunit.client.1.smithi067.stdout:9/600: fdatasync d1/d28/d44/f9e 0 2024-04-12T05:37:28.193 INFO:tasks.workunit.client.1.smithi067.stdout:6/630: sync 2024-04-12T05:37:28.193 INFO:tasks.workunit.client.1.smithi067.stdout:1/630: sync 2024-04-12T05:37:28.194 INFO:tasks.workunit.client.1.smithi067.stdout:2/657: sync 2024-04-12T05:37:28.194 INFO:tasks.workunit.client.1.smithi067.stdout:7/693: sync 2024-04-12T05:37:28.195 INFO:tasks.workunit.client.1.smithi067.stdout:4/560: sync 2024-04-12T05:37:28.196 INFO:tasks.workunit.client.1.smithi067.stdout:8/524: dread - d6/d8/d18/d63/d67/d69/d5d/f91 zero size 2024-04-12T05:37:28.196 INFO:tasks.workunit.client.1.smithi067.stdout:5/890: mkdir d0/d6/d40/d120 0 2024-04-12T05:37:28.197 INFO:tasks.workunit.client.1.smithi067.stdout:1/631: write d4/d44/f66 [2374314,29963] 0 2024-04-12T05:37:28.197 INFO:tasks.workunit.client.1.smithi067.stdout:9/601: write d1/d28/d21/d3f/f41 [2276277,86173] 0 2024-04-12T05:37:28.204 INFO:tasks.workunit.client.1.smithi067.stdout:6/631: creat dd/d21/d64/d6f/d90/d98/fd6 x:0 0 0 2024-04-12T05:37:28.205 INFO:tasks.workunit.client.1.smithi067.stdout:9/602: chown d1/d28/d21/d45/f99 3169 1 2024-04-12T05:37:28.206 INFO:tasks.workunit.client.1.smithi067.stdout:2/658: creat d5/da/de/d14/d46/d90/fd5 x:0 0 0 2024-04-12T05:37:28.208 INFO:tasks.workunit.client.1.smithi067.stdout:8/525: chown d6/d8/d18/d63/d9a/f1c 65897787 1 2024-04-12T05:37:28.211 INFO:tasks.workunit.client.1.smithi067.stdout:1/632: rename d4/d3d/f77 to d4/d9/d14/d7a/fc9 0 2024-04-12T05:37:28.211 INFO:tasks.workunit.client.1.smithi067.stdout:7/694: mknod d0/d23/d80/da0/deb/cf0 0 2024-04-12T05:37:28.218 INFO:tasks.workunit.client.1.smithi067.stdout:0/685: dread dd/d12/fb2 [0,4194304] 0 2024-04-12T05:37:28.218 INFO:tasks.workunit.client.1.smithi067.stdout:9/603: mknod d1/d28/cb2 0 2024-04-12T05:37:28.226 INFO:tasks.workunit.client.1.smithi067.stdout:4/561: dread d8/da/d1d/d15/d36/f5a [0,4194304] 0 2024-04-12T05:37:28.226 INFO:tasks.workunit.client.1.smithi067.stdout:8/526: unlink d6/d86/d4f/d82/c97 0 2024-04-12T05:37:28.226 INFO:tasks.workunit.client.1.smithi067.stdout:1/633: unlink d4/d9/d3a/d51/d55/d60/c74 0 2024-04-12T05:37:28.228 INFO:tasks.workunit.client.1.smithi067.stdout:8/527: fdatasync d6/d8/d18/d4e/f75 0 2024-04-12T05:37:28.231 INFO:tasks.workunit.client.1.smithi067.stdout:9/604: rename d1/d28/d23 to d1/d28/d21/d45/d75/db3 0 2024-04-12T05:37:28.231 INFO:tasks.workunit.client.1.smithi067.stdout:1/634: write d4/d3d/f4b [6113636,110288] 0 2024-04-12T05:37:28.231 INFO:tasks.workunit.client.1.smithi067.stdout:1/635: chown d4/f5 43839 1 2024-04-12T05:37:28.234 INFO:tasks.workunit.client.1.smithi067.stdout:8/528: write d6/d8/d18/d36/f54 [1157153,50714] 0 2024-04-12T05:37:28.234 INFO:tasks.workunit.client.1.smithi067.stdout:7/695: creat d0/d23/d80/da0/db4/dd/d42/d7e/ff1 x:0 0 0 2024-04-12T05:37:28.235 INFO:tasks.workunit.client.1.smithi067.stdout:8/529: readlink d6/d8/l57 0 2024-04-12T05:37:28.236 INFO:tasks.workunit.client.1.smithi067.stdout:9/605: rename d1/d28/f50 to d1/d28/d21/d3f/d4f/fb4 0 2024-04-12T05:37:28.238 INFO:tasks.workunit.client.1.smithi067.stdout:0/686: creat dd/fd4 x:0 0 0 2024-04-12T05:37:28.239 INFO:tasks.workunit.client.1.smithi067.stdout:8/530: mkdir d6/d8/d8b/da1 0 2024-04-12T05:37:28.240 INFO:tasks.workunit.client.1.smithi067.stdout:7/696: mknod d0/d23/d80/da0/db4/dd/cf2 0 2024-04-12T05:37:28.242 INFO:tasks.workunit.client.1.smithi067.stdout:9/606: creat d1/d28/d5e/fb5 x:0 0 0 2024-04-12T05:37:28.243 INFO:tasks.workunit.client.1.smithi067.stdout:7/697: write d0/d23/d80/fcb [4401519,45092] 0 2024-04-12T05:37:28.245 INFO:tasks.workunit.client.1.smithi067.stdout:8/531: unlink d6/d8/c1a 0 2024-04-12T05:37:28.247 INFO:tasks.workunit.client.1.smithi067.stdout:9/607: write d1/d13/d34/f6c [1973674,87560] 0 2024-04-12T05:37:28.248 INFO:tasks.workunit.client.1.smithi067.stdout:4/562: getdents d8 0 2024-04-12T05:37:28.252 INFO:tasks.workunit.client.1.smithi067.stdout:2/659: write d5/da/dae/dba/fc9 [1017847,27349] 0 2024-04-12T05:37:28.252 INFO:tasks.workunit.client.1.smithi067.stdout:6/632: truncate dd/d11/d27/f54 378444 0 2024-04-12T05:37:28.258 INFO:tasks.workunit.client.1.smithi067.stdout:7/698: mknod d0/d23/d80/da0/db4/d5/d4a/cf3 0 2024-04-12T05:37:28.258 INFO:tasks.workunit.client.1.smithi067.stdout:8/532: dread - d6/d86/d4f/d82/f9d zero size 2024-04-12T05:37:28.261 INFO:tasks.workunit.client.1.smithi067.stdout:1/636: dread d4/d9/f45 [0,4194304] 0 2024-04-12T05:37:28.265 INFO:tasks.workunit.client.1.smithi067.stdout:0/687: dread dd/d12/d14/f43 [0,4194304] 0 2024-04-12T05:37:28.265 INFO:tasks.workunit.client.1.smithi067.stdout:9/608: dread d1/d12/f53 [0,4194304] 0 2024-04-12T05:37:28.268 INFO:tasks.workunit.client.1.smithi067.stdout:4/563: rename d8/da/d1d/d15/d36 to d8/dba 0 2024-04-12T05:37:28.269 INFO:tasks.workunit.client.1.smithi067.stdout:8/533: mknod d6/d8/d18/d4e/d9e/ca2 0 2024-04-12T05:37:28.269 INFO:tasks.workunit.client.1.smithi067.stdout:7/699: creat d0/d23/d80/da0/db4/dd/d42/d82/ff4 x:0 0 0 2024-04-12T05:37:28.275 INFO:tasks.workunit.client.1.smithi067.stdout:2/660: dwrite d5/da/de/d14/d34/d40/fcb [0,4194304] 0 2024-04-12T05:37:28.278 INFO:tasks.workunit.client.1.smithi067.stdout:0/688: creat dd/d12/d14/d49/fd5 x:0 0 0 2024-04-12T05:37:28.278 INFO:tasks.workunit.client.1.smithi067.stdout:9/609: mknod d1/d28/d21/d7f/cb6 0 2024-04-12T05:37:28.280 INFO:tasks.workunit.client.1.smithi067.stdout:2/661: readlink d5/da/de/d14/l3c 0 2024-04-12T05:37:28.281 INFO:tasks.workunit.client.1.smithi067.stdout:1/637: creat d4/fca x:0 0 0 2024-04-12T05:37:28.283 INFO:tasks.workunit.client.1.smithi067.stdout:0/689: mknod dd/d12/d14/d49/dab/cd6 0 2024-04-12T05:37:28.285 INFO:tasks.workunit.client.1.smithi067.stdout:8/534: rename d6/d8/d18/d63/d9a/d12/c2a to d6/d86/d4f/d82/ca3 0 2024-04-12T05:37:28.286 INFO:tasks.workunit.client.1.smithi067.stdout:7/700: dwrite d0/d67/d57/fca [0,4194304] 0 2024-04-12T05:37:28.288 INFO:tasks.workunit.client.1.smithi067.stdout:9/610: mkdir d1/d13/db7 0 2024-04-12T05:37:28.292 INFO:tasks.workunit.client.1.smithi067.stdout:8/535: creat d6/d86/fa4 x:0 0 0 2024-04-12T05:37:28.292 INFO:tasks.workunit.client.1.smithi067.stdout:9/611: stat d1/d28/d27/d32 0 2024-04-12T05:37:28.294 INFO:tasks.workunit.client.1.smithi067.stdout:2/662: write d5/da/de/d14/d34/d4c/f9d [767916,62861] 0 2024-04-12T05:37:28.299 INFO:tasks.workunit.client.1.smithi067.stdout:6/633: truncate dd/d11/d27/d4a/d79/dc3/f43 2102176 0 2024-04-12T05:37:28.299 INFO:tasks.workunit.client.1.smithi067.stdout:2/663: readlink d5/d6a/d78/l80 0 2024-04-12T05:37:28.300 INFO:tasks.workunit.client.1.smithi067.stdout:4/564: write d8/dba/d5d/d66/d71/f9a [865763,31197] 0 2024-04-12T05:37:28.301 INFO:tasks.workunit.client.1.smithi067.stdout:8/536: chown d6/d8/d18/d36/f54 498544864 1 2024-04-12T05:37:28.303 INFO:tasks.workunit.client.1.smithi067.stdout:7/701: creat d0/d23/d80/da0/db4/d5/d1d/ff5 x:0 0 0 2024-04-12T05:37:28.304 INFO:tasks.workunit.client.1.smithi067.stdout:0/690: dwrite dd/d9c/d5f/f90 [0,4194304] 0 2024-04-12T05:37:28.305 INFO:tasks.workunit.client.1.smithi067.stdout:1/638: dwrite d4/d9/d3a/d51/f63 [4194304,4194304] 0 2024-04-12T05:37:28.308 INFO:tasks.workunit.client.1.smithi067.stdout:9/612: creat d1/d28/fb8 x:0 0 0 2024-04-12T05:37:28.308 INFO:tasks.workunit.client.1.smithi067.stdout:8/537: write d6/d8/d18/d4e/f75 [278219,53471] 0 2024-04-12T05:37:28.309 INFO:tasks.workunit.client.1.smithi067.stdout:2/664: write d5/da/de/d14/d46/f5a [305535,68595] 0 2024-04-12T05:37:28.311 INFO:tasks.workunit.client.1.smithi067.stdout:3/526: sync 2024-04-12T05:37:28.312 INFO:tasks.workunit.client.1.smithi067.stdout:0/691: chown dd/d12/d14/d49/fd0 0 1 2024-04-12T05:37:28.317 INFO:tasks.workunit.client.1.smithi067.stdout:1/639: chown d4/d9/c54 76 1 2024-04-12T05:37:28.317 INFO:tasks.workunit.client.1.smithi067.stdout:0/692: chown dd/d12/d14/f43 173551 1 2024-04-12T05:37:28.318 INFO:tasks.workunit.client.1.smithi067.stdout:7/702: write d0/d23/d80/da0/db4/dd/d42/d82/d96/fab [170410,45113] 0 2024-04-12T05:37:28.320 INFO:tasks.workunit.client.1.smithi067.stdout:2/665: mkdir d5/da/de/d39/dd6 0 2024-04-12T05:37:28.321 INFO:tasks.workunit.client.1.smithi067.stdout:9/613: read d1/d28/d5e/f6f [216722,34866] 0 2024-04-12T05:37:28.322 INFO:tasks.workunit.client.1.smithi067.stdout:1/640: write d4/d9/d3a/d51/d55/f93 [1622886,63541] 0 2024-04-12T05:37:28.326 INFO:tasks.workunit.client.1.smithi067.stdout:7/703: write d0/d23/d80/da0/db4/f52 [836524,215] 0 2024-04-12T05:37:28.326 INFO:tasks.workunit.client.1.smithi067.stdout:4/565: rename d8/dba/d5d/f74 to d8/dba/d5d/d66/d7b/fbb 0 2024-04-12T05:37:28.326 INFO:tasks.workunit.client.1.smithi067.stdout:2/666: dread - d5/fb4 zero size 2024-04-12T05:37:28.327 INFO:tasks.workunit.client.1.smithi067.stdout:3/527: mknod dc/df/d27/daa/cb5 0 2024-04-12T05:37:28.332 INFO:tasks.workunit.client.1.smithi067.stdout:7/704: fsync d0/d23/d45/f73 0 2024-04-12T05:37:28.333 INFO:tasks.workunit.client.1.smithi067.stdout:1/641: read d4/d46/f58 [3233629,71712] 0 2024-04-12T05:37:28.336 INFO:tasks.workunit.client.1.smithi067.stdout:0/693: creat dd/d9c/d5f/d78/d83/fd7 x:0 0 0 2024-04-12T05:37:28.337 INFO:tasks.workunit.client.1.smithi067.stdout:5/891: sync 2024-04-12T05:37:28.342 INFO:tasks.workunit.client.1.smithi067.stdout:7/705: write d0/d23/d45/f26 [656223,80516] 0 2024-04-12T05:37:28.342 INFO:tasks.workunit.client.1.smithi067.stdout:6/634: write dd/d11/d27/f54 [837324,125537] 0 2024-04-12T05:37:28.342 INFO:tasks.workunit.client.1.smithi067.stdout:9/614: write d1/d12/f74 [1037915,129877] 0 2024-04-12T05:37:28.343 INFO:tasks.workunit.client.1.smithi067.stdout:5/892: readlink d0/d6/d20/d3e/lb2 0 2024-04-12T05:37:28.347 INFO:tasks.workunit.client.1.smithi067.stdout:0/694: mknod dd/d6f/d72/cd8 0 2024-04-12T05:37:28.347 INFO:tasks.workunit.client.1.smithi067.stdout:8/538: dread d6/d8/d18/d63/d9a/f1c [4194304,4194304] 0 2024-04-12T05:37:28.348 INFO:tasks.workunit.client.1.smithi067.stdout:2/667: dwrite d5/da/de/d14/d34/f61 [4194304,4194304] 0 2024-04-12T05:37:28.350 INFO:tasks.workunit.client.1.smithi067.stdout:1/642: write d4/f7 [7705482,70501] 0 2024-04-12T05:37:28.355 INFO:tasks.workunit.client.1.smithi067.stdout:5/893: fsync d0/d6/d33/f71 0 2024-04-12T05:37:28.362 INFO:tasks.workunit.client.1.smithi067.stdout:9/615: unlink d1/d28/d21/dae/fa1 0 2024-04-12T05:37:28.363 INFO:tasks.workunit.client.1.smithi067.stdout:4/566: dwrite d8/da/d1d/d27/d91/fa8 [4194304,4194304] 0 2024-04-12T05:37:28.363 INFO:tasks.workunit.client.1.smithi067.stdout:6/635: mknod dd/d11/d27/d4a/d8b/cd7 0 2024-04-12T05:37:28.364 INFO:tasks.workunit.client.1.smithi067.stdout:1/643: rmdir d4/d46 39 2024-04-12T05:37:28.366 INFO:tasks.workunit.client.1.smithi067.stdout:3/528: dwrite dc/df/d27/f6a [0,4194304] 0 2024-04-12T05:37:28.368 INFO:tasks.workunit.client.1.smithi067.stdout:5/894: mknod d0/da/d41/d7d/c121 0 2024-04-12T05:37:28.370 INFO:tasks.workunit.client.1.smithi067.stdout:6/636: stat dd/d11/d27/d4a/d79 0 2024-04-12T05:37:28.370 INFO:tasks.workunit.client.1.smithi067.stdout:5/895: dread - d0/d6/d40/d4b/f11b zero size 2024-04-12T05:37:28.373 INFO:tasks.workunit.client.1.smithi067.stdout:3/529: chown dc/d1d/d28/d3e/ca9 30 1 2024-04-12T05:37:28.375 INFO:tasks.workunit.client.1.smithi067.stdout:4/567: symlink d8/dba/d5d/d66/lbc 0 2024-04-12T05:37:28.378 INFO:tasks.workunit.client.1.smithi067.stdout:9/616: creat d1/d13/fb9 x:0 0 0 2024-04-12T05:37:28.378 INFO:tasks.workunit.client.1.smithi067.stdout:2/668: write d5/d43/d6e/f72 [10282782,18747] 0 2024-04-12T05:37:28.379 INFO:tasks.workunit.client.1.smithi067.stdout:0/695: truncate dd/d5b/f68 748410 0 2024-04-12T05:37:28.380 INFO:tasks.workunit.client.1.smithi067.stdout:4/568: read - d8/dba/d5d/d66/f9c zero size 2024-04-12T05:37:28.381 INFO:tasks.workunit.client.1.smithi067.stdout:7/706: dread d0/d23/d45/f59 [0,4194304] 0 2024-04-12T05:37:28.381 INFO:tasks.workunit.client.1.smithi067.stdout:5/896: rename d0/d6/d20/d62/c5e to d0/d6/d40/d4b/c122 0 2024-04-12T05:37:28.381 INFO:tasks.workunit.client.1.smithi067.stdout:8/539: dwrite d6/d8/d6e/f78 [0,4194304] 0 2024-04-12T05:37:28.383 INFO:tasks.workunit.client.1.smithi067.stdout:1/644: getdents d4 0 2024-04-12T05:37:28.384 INFO:tasks.workunit.client.1.smithi067.stdout:9/617: dread - d1/d28/d5e/fb5 zero size 2024-04-12T05:37:28.388 INFO:tasks.workunit.client.1.smithi067.stdout:9/618: readlink d1/d13/l37 0 2024-04-12T05:37:28.388 INFO:tasks.workunit.client.1.smithi067.stdout:8/540: read d6/d86/d4f/f65 [56892,109240] 0 2024-04-12T05:37:28.389 INFO:tasks.workunit.client.1.smithi067.stdout:2/669: rename d5/d11/f71 to d5/dbf/dcc/fd7 0 2024-04-12T05:37:28.389 INFO:tasks.workunit.client.1.smithi067.stdout:4/569: read - d8/d9/f2a zero size 2024-04-12T05:37:28.391 INFO:tasks.workunit.client.1.smithi067.stdout:5/897: mknod d0/d6/d33/c123 0 2024-04-12T05:37:28.393 INFO:tasks.workunit.client.1.smithi067.stdout:2/670: chown d5/da/de/d14/d46/d90/fd5 993092 1 2024-04-12T05:37:28.394 INFO:tasks.workunit.client.1.smithi067.stdout:3/530: dread dc/d1d/f22 [0,4194304] 0 2024-04-12T05:37:28.394 INFO:tasks.workunit.client.1.smithi067.stdout:0/696: mkdir dd/dad/dd9 0 2024-04-12T05:37:28.396 INFO:tasks.workunit.client.1.smithi067.stdout:2/671: truncate d5/d43/f53 4807765 0 2024-04-12T05:37:28.396 INFO:tasks.workunit.client.1.smithi067.stdout:9/619: creat d1/d28/d21/d68/fba x:0 0 0 2024-04-12T05:37:28.397 INFO:tasks.workunit.client.1.smithi067.stdout:4/570: mkdir d8/dba/d5d/d66/d7b/dbd 0 2024-04-12T05:37:28.398 INFO:tasks.workunit.client.1.smithi067.stdout:9/620: dread - d1/d28/d21/f9b zero size 2024-04-12T05:37:28.399 INFO:tasks.workunit.client.1.smithi067.stdout:8/541: chown d6/d8/d18/d63/d67/d69/l8f 1256 1 2024-04-12T05:37:28.399 INFO:tasks.workunit.client.1.smithi067.stdout:4/571: write d8/dba/d5d/faf [467732,24938] 0 2024-04-12T05:37:28.400 INFO:tasks.workunit.client.1.smithi067.stdout:7/707: rename d0/d23/d80/da0/db4/d5/da1/caf to d0/d23/d45/dd1/cf6 0 2024-04-12T05:37:28.402 INFO:tasks.workunit.client.1.smithi067.stdout:5/898: mknod d0/d3a/d2c/db3/c124 0 2024-04-12T05:37:28.402 INFO:tasks.workunit.client.1.smithi067.stdout:1/645: write d4/d9/d14/f2a [118060,62483] 0 2024-04-12T05:37:28.403 INFO:tasks.workunit.client.1.smithi067.stdout:0/697: mknod dd/d12/d14/d51/da2/cda 0 2024-04-12T05:37:28.404 INFO:tasks.workunit.client.1.smithi067.stdout:3/531: creat dc/d1d/d28/d3e/d53/d76/fb6 x:0 0 0 2024-04-12T05:37:28.407 INFO:tasks.workunit.client.1.smithi067.stdout:8/542: rename d6/d8/d18/d4e/l6d to d6/d8/d18/d4e/d9e/la5 0 2024-04-12T05:37:28.409 INFO:tasks.workunit.client.1.smithi067.stdout:3/532: truncate dc/d1d/d28/d3e/d53/f58 602007 0 2024-04-12T05:37:28.409 INFO:tasks.workunit.client.1.smithi067.stdout:2/672: mknod d5/d43/da4/cd8 0 2024-04-12T05:37:28.411 INFO:tasks.workunit.client.1.smithi067.stdout:9/621: creat d1/d13/db7/fbb x:0 0 0 2024-04-12T05:37:28.411 INFO:tasks.workunit.client.1.smithi067.stdout:4/572: creat d8/da/d9f/db5/fbe x:0 0 0 2024-04-12T05:37:28.411 INFO:tasks.workunit.client.1.smithi067.stdout:1/646: dwrite d4/d9/d14/f8a [0,4194304] 0 2024-04-12T05:37:28.412 INFO:tasks.workunit.client.1.smithi067.stdout:7/708: mkdir d0/d23/d80/da0/dc3/df7 0 2024-04-12T05:37:28.413 INFO:tasks.workunit.client.1.smithi067.stdout:3/533: dread - dc/df/d27/f96 zero size 2024-04-12T05:37:28.413 INFO:tasks.workunit.client.1.smithi067.stdout:5/899: mkdir d0/d6/d40/d120/d125 0 2024-04-12T05:37:28.414 INFO:tasks.workunit.client.1.smithi067.stdout:9/622: readlink d1/d12/l29 0 2024-04-12T05:37:28.416 INFO:tasks.workunit.client.1.smithi067.stdout:8/543: rmdir d6/d8/d18/d63/d67 39 2024-04-12T05:37:28.416 INFO:tasks.workunit.client.1.smithi067.stdout:2/673: write d5/da/de/f69 [5455309,107371] 0 2024-04-12T05:37:28.421 INFO:tasks.workunit.client.1.smithi067.stdout:1/647: write d4/d3d/dab/f32 [112138,99947] 0 2024-04-12T05:37:28.421 INFO:tasks.workunit.client.1.smithi067.stdout:9/623: write d1/f6 [2487312,60010] 0 2024-04-12T05:37:28.422 INFO:tasks.workunit.client.1.smithi067.stdout:0/698: dwrite dd/d6f/d72/f8b [0,4194304] 0 2024-04-12T05:37:28.422 INFO:tasks.workunit.client.1.smithi067.stdout:7/709: chown d0/d23/d80/da0/db4/dd/d42/d7e/d34/f8f 1250 1 2024-04-12T05:37:28.423 INFO:tasks.workunit.client.1.smithi067.stdout:9/624: fsync d1/d12/d15/f5c 0 2024-04-12T05:37:28.425 INFO:tasks.workunit.client.1.smithi067.stdout:7/710: chown d0/d23/d80/da0/db4/dd/d42/d7e/d34/f8f 399594081 1 2024-04-12T05:37:28.427 INFO:tasks.workunit.client.1.smithi067.stdout:0/699: chown dd/d12/d14/d51/da2/cda 0 1 2024-04-12T05:37:28.428 INFO:tasks.workunit.client.1.smithi067.stdout:4/573: rename d8/d9/d3d/d64 to d8/da/d9f/d68/dbf 0 2024-04-12T05:37:28.429 INFO:tasks.workunit.client.1.smithi067.stdout:5/900: mknod d0/d6/d33/d88/df5/c126 0 2024-04-12T05:37:28.433 INFO:tasks.workunit.client.1.smithi067.stdout:1/648: creat d4/d9/d3a/fcb x:0 0 0 2024-04-12T05:37:28.433 INFO:tasks.workunit.client.1.smithi067.stdout:4/574: write d8/d9/f21 [271624,87983] 0 2024-04-12T05:37:28.437 INFO:tasks.workunit.client.1.smithi067.stdout:8/544: unlink d6/d8/d18/d63/d9a/d12/f1d 0 2024-04-12T05:37:28.437 INFO:tasks.workunit.client.1.smithi067.stdout:2/674: fdatasync d5/da/de/d14/d34/d4c/f6b 0 2024-04-12T05:37:28.439 INFO:tasks.workunit.client.1.smithi067.stdout:9/625: symlink d1/d13/lbc 0 2024-04-12T05:37:28.439 INFO:tasks.workunit.client.1.smithi067.stdout:7/711: dwrite d0/f1 [0,4194304] 0 2024-04-12T05:37:28.439 INFO:tasks.workunit.client.1.smithi067.stdout:4/575: stat d8/d9/d3d/f4d 0 2024-04-12T05:37:28.441 INFO:tasks.workunit.client.1.smithi067.stdout:3/534: rename dc/c12 to dc/df/d27/daa/cb7 0 2024-04-12T05:37:28.444 INFO:tasks.workunit.client.1.smithi067.stdout:0/700: dwrite dd/d12/d14/d49/dab/fcf [0,4194304] 0 2024-04-12T05:37:28.447 INFO:tasks.workunit.client.1.smithi067.stdout:1/649: symlink d4/d9/d14/lcc 0 2024-04-12T05:37:28.453 INFO:tasks.workunit.client.1.smithi067.stdout:2/675: mknod d5/d43/d6e/cd9 0 2024-04-12T05:37:28.453 INFO:tasks.workunit.client.1.smithi067.stdout:9/626: symlink d1/d28/d21/d3f/lbd 0 2024-04-12T05:37:28.454 INFO:tasks.workunit.client.1.smithi067.stdout:3/535: stat dc/d1d/d28/d3e/d53/d94 0 2024-04-12T05:37:28.454 INFO:tasks.workunit.client.1.smithi067.stdout:5/901: getdents d0/d6/d33/d88 0 2024-04-12T05:37:28.455 INFO:tasks.workunit.client.1.smithi067.stdout:2/676: fdatasync d5/da/de/d14/d46/d90/fd4 0 2024-04-12T05:37:28.456 INFO:tasks.workunit.client.1.smithi067.stdout:4/576: creat d8/da/d1d/d27/d91/fc0 x:0 0 0 2024-04-12T05:37:28.457 INFO:tasks.workunit.client.1.smithi067.stdout:7/712: mkdir d0/d23/d80/da0/db4/dd/da2/df8 0 2024-04-12T05:37:28.458 INFO:tasks.workunit.client.1.smithi067.stdout:9/627: creat d1/d28/d21/d3f/d4f/fbe x:0 0 0 2024-04-12T05:37:28.459 INFO:tasks.workunit.client.1.smithi067.stdout:4/577: chown d8 19972 1 2024-04-12T05:37:28.459 INFO:tasks.workunit.client.1.smithi067.stdout:1/650: truncate d4/d9/d12/d59/f87 791700 0 2024-04-12T05:37:28.459 INFO:tasks.workunit.client.1.smithi067.stdout:5/902: mknod d0/d6/d40/d4b/c127 0 2024-04-12T05:37:28.460 INFO:tasks.workunit.client.1.smithi067.stdout:7/713: dread - d0/d67/fd4 zero size 2024-04-12T05:37:28.461 INFO:tasks.workunit.client.1.smithi067.stdout:8/545: truncate d6/d8/d18/d63/d67/d69/f79 3113540 0 2024-04-12T05:37:28.464 INFO:tasks.workunit.client.1.smithi067.stdout:1/651: chown d4/d3d/dab/l3e 10233 1 2024-04-12T05:37:28.464 INFO:tasks.workunit.client.1.smithi067.stdout:3/536: rename dc/d1d/d28/c42 to dc/d1d/cb8 0 2024-04-12T05:37:28.466 INFO:tasks.workunit.client.1.smithi067.stdout:4/578: creat d8/fc1 x:0 0 0 2024-04-12T05:37:28.466 INFO:tasks.workunit.client.1.smithi067.stdout:7/714: truncate d0/d67/fd4 528476 0 2024-04-12T05:37:28.467 INFO:tasks.workunit.client.1.smithi067.stdout:2/677: mkdir d5/da/de/d14/d46/d90/dda 0 2024-04-12T05:37:28.470 INFO:tasks.workunit.client.1.smithi067.stdout:1/652: creat d4/d9/d4c/fcd x:0 0 0 2024-04-12T05:37:28.471 INFO:tasks.workunit.client.1.smithi067.stdout:9/628: dwrite d1/d28/d21/d3f/f41 [0,4194304] 0 2024-04-12T05:37:28.472 INFO:tasks.workunit.client.1.smithi067.stdout:2/678: truncate d5/d6a/d8f/fc8 1045730 0 2024-04-12T05:37:28.477 INFO:tasks.workunit.client.1.smithi067.stdout:8/546: chown d6/d8/d18/d63/d9a/d12/f1f 52151 1 2024-04-12T05:37:28.479 INFO:tasks.workunit.client.1.smithi067.stdout:4/579: creat d8/da/d9f/d68/d8e/fc2 x:0 0 0 2024-04-12T05:37:28.480 INFO:tasks.workunit.client.1.smithi067.stdout:9/629: unlink d1/d28/d5e/fa3 0 2024-04-12T05:37:28.481 INFO:tasks.workunit.client.1.smithi067.stdout:8/547: chown d6/d8/d18/f5e 0 1 2024-04-12T05:37:28.486 INFO:tasks.workunit.client.1.smithi067.stdout:3/537: write dc/d1d/d26/fb1 [4452267,130755] 0 2024-04-12T05:37:28.487 INFO:tasks.workunit.client.1.smithi067.stdout:0/701: dwrite dd/d9c/f50 [0,4194304] 0 2024-04-12T05:37:28.493 INFO:tasks.workunit.client.1.smithi067.stdout:4/580: mknod d8/cc3 0 2024-04-12T05:37:28.494 INFO:tasks.workunit.client.1.smithi067.stdout:8/548: dread - d6/d8/d18/d63/d67/d69/f71 zero size 2024-04-12T05:37:28.494 INFO:tasks.workunit.client.1.smithi067.stdout:7/715: getdents d0/d23/d80/da0/db4/dd/d42 0 2024-04-12T05:37:28.495 INFO:tasks.workunit.client.1.smithi067.stdout:9/630: read d1/f7 [2199353,29983] 0 2024-04-12T05:37:28.501 INFO:tasks.workunit.client.1.smithi067.stdout:0/702: chown dd/d12/d14/d51/da2 37696177 1 2024-04-12T05:37:28.501 INFO:tasks.workunit.client.1.smithi067.stdout:3/538: write dc/d1d/d26/fb1 [4387033,106494] 0 2024-04-12T05:37:28.504 INFO:tasks.workunit.client.1.smithi067.stdout:5/903: dwrite d0/d6/d40/d4b/fb1 [4194304,4194304] 0 2024-04-12T05:37:28.506 INFO:tasks.workunit.client.1.smithi067.stdout:9/631: write d1/d13/f5b [4173584,128347] 0 2024-04-12T05:37:28.508 INFO:tasks.workunit.client.1.smithi067.stdout:1/653: dwrite d4/d9/d14/f8a [4194304,4194304] 0 2024-04-12T05:37:28.512 INFO:tasks.workunit.client.1.smithi067.stdout:7/716: write d0/d23/d45/f59 [4066286,57025] 0 2024-04-12T05:37:28.520 INFO:tasks.workunit.client.1.smithi067.stdout:3/539: mkdir dc/d1d/d89/da3/db9 0 2024-04-12T05:37:28.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:28 smithi067 ceph-mon[49010]: pgmap v275: 65 pgs: 65 active+clean; 3.2 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 93 MiB/s rd, 260 MiB/s wr, 562 op/s 2024-04-12T05:37:28.520 INFO:tasks.workunit.client.1.smithi067.stdout:2/679: dread d5/da/de/d14/f63 [0,4194304] 0 2024-04-12T05:37:28.523 INFO:tasks.workunit.client.1.smithi067.stdout:0/703: truncate dd/d12/d14/d49/f4d 4842206 0 2024-04-12T05:37:28.524 INFO:tasks.workunit.client.1.smithi067.stdout:9/632: symlink d1/d13/lbf 0 2024-04-12T05:37:28.526 INFO:tasks.workunit.client.1.smithi067.stdout:4/581: dread d8/d9/f6b [0,4194304] 0 2024-04-12T05:37:28.527 INFO:tasks.workunit.client.1.smithi067.stdout:5/904: symlink d0/d3a/d2c/l128 0 2024-04-12T05:37:28.530 INFO:tasks.workunit.client.1.smithi067.stdout:0/704: symlink dd/dbd/dbe/ldb 0 2024-04-12T05:37:28.531 INFO:tasks.workunit.client.1.smithi067.stdout:1/654: getdents d4/d52/d80 0 2024-04-12T05:37:28.532 INFO:tasks.workunit.client.1.smithi067.stdout:8/549: dwrite d6/d8/d18/d36/f5c [0,4194304] 0 2024-04-12T05:37:28.532 INFO:tasks.workunit.client.1.smithi067.stdout:4/582: chown d8/da/f1f 15451 1 2024-04-12T05:37:28.536 INFO:tasks.workunit.client.1.smithi067.stdout:4/583: stat d8/da/d8b/fa7 0 2024-04-12T05:37:28.536 INFO:tasks.workunit.client.1.smithi067.stdout:7/717: dread d0/d23/d80/da0/db4/d5/d1d/d43/fd0 [0,4194304] 0 2024-04-12T05:37:28.537 INFO:tasks.workunit.client.1.smithi067.stdout:8/550: chown d6/d8/f72 6179092 1 2024-04-12T05:37:28.541 INFO:tasks.workunit.client.1.smithi067.stdout:3/540: rename dc/d1d/d28/d3e/ca9 to dc/d1d/d28/d3e/d53/d94/cba 0 2024-04-12T05:37:28.541 INFO:tasks.workunit.client.1.smithi067.stdout:0/705: read dd/d12/d14/d1a/d3d/f7c [2454413,19110] 0 2024-04-12T05:37:28.545 INFO:tasks.workunit.client.1.smithi067.stdout:2/680: dwrite d5/da/dae/dba/fc9 [0,4194304] 0 2024-04-12T05:37:28.551 INFO:tasks.workunit.client.1.smithi067.stdout:5/905: creat d0/d6/d40/d4b/d38/d8e/d8f/d99/d109/f129 x:0 0 0 2024-04-12T05:37:28.557 INFO:tasks.workunit.client.1.smithi067.stdout:0/706: symlink dd/ldc 0 2024-04-12T05:37:28.558 INFO:tasks.workunit.client.1.smithi067.stdout:4/584: creat d8/da/fc4 x:0 0 0 2024-04-12T05:37:28.560 INFO:tasks.workunit.client.1.smithi067.stdout:8/551: creat d6/d8/d18/d63/d67/fa6 x:0 0 0 2024-04-12T05:37:28.561 INFO:tasks.workunit.client.1.smithi067.stdout:1/655: dwrite d4/d46/f58 [4194304,4194304] 0 2024-04-12T05:37:28.564 INFO:tasks.workunit.client.1.smithi067.stdout:5/906: unlink d0/d6/d40/d4b/d38/d8e/d8f/db9/dbd/c10e 0 2024-04-12T05:37:28.568 INFO:tasks.workunit.client.1.smithi067.stdout:9/633: dread d1/d28/d44/f73 [0,4194304] 0 2024-04-12T05:37:28.568 INFO:tasks.workunit.client.1.smithi067.stdout:1/656: rmdir d4/d3d/d7b 39 2024-04-12T05:37:28.569 INFO:tasks.workunit.client.1.smithi067.stdout:3/541: link dc/d50/c6c dc/df/d8d/cbb 0 2024-04-12T05:37:28.570 INFO:tasks.workunit.client.1.smithi067.stdout:7/718: dwrite d0/d23/d80/da0/db4/dd/da2/f28 [0,4194304] 0 2024-04-12T05:37:28.576 INFO:tasks.workunit.client.1.smithi067.stdout:8/552: truncate d6/d86/d4f/fa0 674189 0 2024-04-12T05:37:28.576 INFO:tasks.workunit.client.1.smithi067.stdout:1/657: fsync d4/d9/d12/f2e 0 2024-04-12T05:37:28.579 INFO:tasks.workunit.client.1.smithi067.stdout:3/542: read dc/df/d27/f3f [492547,22187] 0 2024-04-12T05:37:28.580 INFO:tasks.workunit.client.1.smithi067.stdout:5/907: rmdir d0/d6/d33/dd3 39 2024-04-12T05:37:28.580 INFO:tasks.workunit.client.1.smithi067.stdout:0/707: write dd/d9c/d64/f7e [2244635,53930] 0 2024-04-12T05:37:28.582 INFO:tasks.workunit.client.1.smithi067.stdout:4/585: dwrite d8/da/d1d/d15/d75/daa/f61 [0,4194304] 0 2024-04-12T05:37:28.588 INFO:tasks.workunit.client.1.smithi067.stdout:5/908: creat d0/da/d111/f12a x:0 0 0 2024-04-12T05:37:28.588 INFO:tasks.workunit.client.1.smithi067.stdout:1/658: link d4/d9/d3a/ca8 d4/d46/cce 0 2024-04-12T05:37:28.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:28 smithi032 ceph-mon[39731]: pgmap v275: 65 pgs: 65 active+clean; 3.2 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 93 MiB/s rd, 260 MiB/s wr, 562 op/s 2024-04-12T05:37:28.590 INFO:tasks.workunit.client.1.smithi067.stdout:7/719: dwrite d0/d23/d45/f73 [0,4194304] 0 2024-04-12T05:37:28.593 INFO:tasks.workunit.client.1.smithi067.stdout:5/909: stat d0/d3a/d2c/dc6/ddd/f102 0 2024-04-12T05:37:28.596 INFO:tasks.workunit.client.1.smithi067.stdout:1/659: unlink d4/d9/d3a/d51/d55/d60/d90/lc6 0 2024-04-12T05:37:28.597 INFO:tasks.workunit.client.1.smithi067.stdout:3/543: dwrite dc/d1d/d28/d3e/d53/d76/fb4 [0,4194304] 0 2024-04-12T05:37:28.598 INFO:tasks.workunit.client.1.smithi067.stdout:2/681: dread d5/da/de/d14/d46/f5a [0,4194304] 0 2024-04-12T05:37:28.598 INFO:tasks.workunit.client.1.smithi067.stdout:4/586: dwrite d8/d9/d3d/f93 [0,4194304] 0 2024-04-12T05:37:28.602 INFO:tasks.workunit.client.1.smithi067.stdout:4/587: stat d8/dba/d5d/d66/d71/f9a 0 2024-04-12T05:37:28.602 INFO:tasks.workunit.client.1.smithi067.stdout:3/544: rmdir dc/d1d/d28/d3e/d53/d94 39 2024-04-12T05:37:28.602 INFO:tasks.workunit.client.1.smithi067.stdout:7/720: link d0/d67/c48 d0/d23/d80/da0/db4/d5/d4a/de5/cf9 0 2024-04-12T05:37:28.603 INFO:tasks.workunit.client.1.smithi067.stdout:4/588: chown d8/da/d8b/l4a 7 1 2024-04-12T05:37:28.605 INFO:tasks.workunit.client.1.smithi067.stdout:1/660: dread d4/f8 [0,4194304] 0 2024-04-12T05:37:28.608 INFO:tasks.workunit.client.1.smithi067.stdout:3/545: getdents dc/d1d/d4d 0 2024-04-12T05:37:28.608 INFO:tasks.workunit.client.1.smithi067.stdout:1/661: chown d4/d46/f96 497203 1 2024-04-12T05:37:28.609 INFO:tasks.workunit.client.1.smithi067.stdout:3/546: chown dc/df/c1a 95250798 1 2024-04-12T05:37:28.611 INFO:tasks.workunit.client.1.smithi067.stdout:7/721: getdents d0/d23/d80/da0/db4/dd/d42/d7e/dd8 0 2024-04-12T05:37:28.613 INFO:tasks.workunit.client.1.smithi067.stdout:0/708: read dd/d9c/f62 [866642,26770] 0 2024-04-12T05:37:28.619 INFO:tasks.workunit.client.1.smithi067.stdout:4/589: dread d8/da/d8b/d45/f6e [0,4194304] 0 2024-04-12T05:37:28.620 INFO:tasks.workunit.client.1.smithi067.stdout:1/662: dwrite d4/d3d/dab/f32 [0,4194304] 0 2024-04-12T05:37:28.623 INFO:tasks.workunit.client.1.smithi067.stdout:1/663: truncate d4/d52/fc7 390274 0 2024-04-12T05:37:28.628 INFO:tasks.workunit.client.1.smithi067.stdout:8/553: write d6/d8/d18/d63/d9a/f1c [4307580,120190] 0 2024-04-12T05:37:28.632 INFO:tasks.workunit.client.1.smithi067.stdout:4/590: creat d8/dba/d5d/d66/d71/d92/fc5 x:0 0 0 2024-04-12T05:37:28.632 INFO:tasks.workunit.client.1.smithi067.stdout:0/709: dread dd/d12/d14/d49/f6d [0,4194304] 0 2024-04-12T05:37:28.634 INFO:tasks.workunit.client.1.smithi067.stdout:1/664: dwrite d4/d52/f70 [0,4194304] 0 2024-04-12T05:37:28.634 INFO:tasks.workunit.client.1.smithi067.stdout:0/710: stat dd/d12/f2e 0 2024-04-12T05:37:28.640 INFO:tasks.workunit.client.1.smithi067.stdout:1/665: write d4/d52/d80/f6d [701279,55704] 0 2024-04-12T05:37:28.645 INFO:tasks.workunit.client.1.smithi067.stdout:0/711: fsync dd/d12/d14/d1a/fb0 0 2024-04-12T05:37:28.647 INFO:tasks.workunit.client.1.smithi067.stdout:5/910: dwrite d0/d3a/d2c/dc6/f10b [0,4194304] 0 2024-04-12T05:37:28.647 INFO:tasks.workunit.client.1.smithi067.stdout:8/554: dwrite d6/d8/d18/d63/d9a/f5f [0,4194304] 0 2024-04-12T05:37:28.649 INFO:tasks.workunit.client.1.smithi067.stdout:0/712: chown dd/d12/d14/d51/l7a 2895 1 2024-04-12T05:37:28.649 INFO:tasks.workunit.client.1.smithi067.stdout:1/666: mkdir d4/d9/d12/dcf 0 2024-04-12T05:37:28.652 INFO:tasks.workunit.client.1.smithi067.stdout:0/713: mkdir dd/d9c/d64/ddd 0 2024-04-12T05:37:28.654 INFO:tasks.workunit.client.1.smithi067.stdout:1/667: chown d4/d9/d12/d59/l86 90 1 2024-04-12T05:37:28.659 INFO:tasks.workunit.client.1.smithi067.stdout:8/555: dwrite d6/d8/d18/d63/d9a/f5f [0,4194304] 0 2024-04-12T05:37:28.663 INFO:tasks.workunit.client.1.smithi067.stdout:0/714: creat dd/d12/d14/fde x:0 0 0 2024-04-12T05:37:28.663 INFO:tasks.workunit.client.1.smithi067.stdout:5/911: link d0/d6/d40/d10a/l10c d0/da/l12b 0 2024-04-12T05:37:28.664 INFO:tasks.workunit.client.1.smithi067.stdout:8/556: symlink d6/d8/d18/la7 0 2024-04-12T05:37:28.665 INFO:tasks.workunit.client.1.smithi067.stdout:5/912: chown d0/d6/d20/d3e/dbf/d115/la4 3 1 2024-04-12T05:37:28.667 INFO:tasks.workunit.client.1.smithi067.stdout:0/715: rename dd/d5b/lb7 to dd/dad/dd9/ldf 0 2024-04-12T05:37:28.668 INFO:tasks.workunit.client.1.smithi067.stdout:8/557: creat d6/d8/d18/d4e/fa8 x:0 0 0 2024-04-12T05:37:28.669 INFO:tasks.workunit.client.1.smithi067.stdout:8/558: chown d6/d86/d4f 2524094 1 2024-04-12T05:37:28.670 INFO:tasks.workunit.client.1.smithi067.stdout:5/913: creat d0/d6/d40/d10a/f12c x:0 0 0 2024-04-12T05:37:28.670 INFO:tasks.workunit.client.1.smithi067.stdout:0/716: mkdir dd/d9c/d5f/de0 0 2024-04-12T05:37:28.671 INFO:tasks.workunit.client.1.smithi067.stdout:8/559: truncate d6/d86/f99 866817 0 2024-04-12T05:37:28.673 INFO:tasks.workunit.client.1.smithi067.stdout:0/717: mknod dd/d92/ce1 0 2024-04-12T05:37:28.673 INFO:tasks.workunit.client.1.smithi067.stdout:5/914: creat d0/d6/d40/d4b/d38/d8e/d8f/db9/f12d x:0 0 0 2024-04-12T05:37:28.675 INFO:tasks.workunit.client.1.smithi067.stdout:0/718: mknod dd/dbd/dbe/ce2 0 2024-04-12T05:37:28.676 INFO:tasks.workunit.client.1.smithi067.stdout:5/915: symlink d0/d3a/d2c/dc6/l12e 0 2024-04-12T05:37:28.678 INFO:tasks.workunit.client.1.smithi067.stdout:5/916: readlink d0/d6/d20/d3e/laf 0 2024-04-12T05:37:28.682 INFO:tasks.workunit.client.1.smithi067.stdout:0/719: dwrite dd/d12/d14/f8d [0,4194304] 0 2024-04-12T05:37:28.684 INFO:tasks.workunit.client.1.smithi067.stdout:0/720: stat dd/dad/dd9 0 2024-04-12T05:37:28.691 INFO:tasks.workunit.client.1.smithi067.stdout:5/917: dwrite d0/d3a/d2c/dc6/ddd/fe3 [0,4194304] 0 2024-04-12T05:37:28.692 INFO:tasks.workunit.client.1.smithi067.stdout:8/560: fsync d6/d8/d18/d4e/fa8 0 2024-04-12T05:37:28.697 INFO:tasks.workunit.client.1.smithi067.stdout:1/668: dwrite d4/d9/d12/d59/f87 [0,4194304] 0 2024-04-12T05:37:28.703 INFO:tasks.workunit.client.1.smithi067.stdout:8/561: truncate d6/d8/d18/d63/d67/d69/d5d/f91 856213 0 2024-04-12T05:37:28.703 INFO:tasks.workunit.client.1.smithi067.stdout:1/669: rename d4/d44/cbe to d4/d9/d12/d6f/cd0 0 2024-04-12T05:37:28.707 INFO:tasks.workunit.client.1.smithi067.stdout:1/670: symlink d4/d9/d3a/d51/d55/d60/d90/ld1 0 2024-04-12T05:37:28.710 INFO:tasks.workunit.client.1.smithi067.stdout:8/562: creat d6/d8/d18/d63/d9a/d12/fa9 x:0 0 0 2024-04-12T05:37:28.710 INFO:tasks.workunit.client.1.smithi067.stdout:5/918: dwrite d0/da/fbb [0,4194304] 0 2024-04-12T05:37:28.712 INFO:tasks.workunit.client.1.smithi067.stdout:1/671: read d4/d8f/fa0 [1422849,101473] 0 2024-04-12T05:37:28.713 INFO:tasks.workunit.client.1.smithi067.stdout:1/672: chown d4/d8f/fbb 24731985 1 2024-04-12T05:37:28.713 INFO:tasks.workunit.client.1.smithi067.stdout:5/919: mkdir d0/d6/d40/d4b/d38/d8e/d8f/db9/dbd/ddc/d12f 0 2024-04-12T05:37:28.716 INFO:tasks.workunit.client.1.smithi067.stdout:1/673: mkdir d4/dd2 0 2024-04-12T05:37:28.716 INFO:tasks.workunit.client.1.smithi067.stdout:5/920: mkdir d0/d3a/d2c/dc6/d130 0 2024-04-12T05:37:28.724 INFO:tasks.workunit.client.1.smithi067.stdout:8/563: dread d6/d8/d18/f25 [0,4194304] 0 2024-04-12T05:37:28.727 INFO:tasks.workunit.client.1.smithi067.stdout:8/564: unlink d6/d8/f14 0 2024-04-12T05:37:28.728 INFO:tasks.workunit.client.1.smithi067.stdout:0/721: dread dd/d6f/f98 [0,4194304] 0 2024-04-12T05:37:28.729 INFO:tasks.workunit.client.1.smithi067.stdout:8/565: symlink d6/d86/laa 0 2024-04-12T05:37:28.731 INFO:tasks.workunit.client.1.smithi067.stdout:0/722: mkdir dd/d12/d14/de3 0 2024-04-12T05:37:28.734 INFO:tasks.workunit.client.1.smithi067.stdout:0/723: mknod dd/d12/d14/ce4 0 2024-04-12T05:37:28.735 INFO:tasks.workunit.client.1.smithi067.stdout:8/566: dread d6/d8/d18/f25 [0,4194304] 0 2024-04-12T05:37:28.736 INFO:tasks.workunit.client.1.smithi067.stdout:8/567: fdatasync d6/d8/d18/d36/f9b 0 2024-04-12T05:37:28.741 INFO:tasks.workunit.client.1.smithi067.stdout:1/674: creat d4/d9/d3a/fd3 x:0 0 0 2024-04-12T05:37:28.741 INFO:tasks.workunit.client.1.smithi067.stdout:0/724: dread dd/d9c/d5f/f90 [0,4194304] 0 2024-04-12T05:37:28.744 INFO:tasks.workunit.client.1.smithi067.stdout:0/725: rename dd/d12/l30 to dd/d9c/le5 0 2024-04-12T05:37:28.745 INFO:tasks.workunit.client.1.smithi067.stdout:0/726: chown dd/d5b/lce 88 1 2024-04-12T05:37:28.745 INFO:tasks.workunit.client.1.smithi067.stdout:5/921: dread d0/d6/f12 [0,4194304] 0 2024-04-12T05:37:28.747 INFO:tasks.workunit.client.1.smithi067.stdout:0/727: symlink dd/d5b/le6 0 2024-04-12T05:37:28.749 INFO:tasks.workunit.client.1.smithi067.stdout:1/675: rename d4/d9/d3a/l9c to d4/d3d/ld4 0 2024-04-12T05:37:28.755 INFO:tasks.workunit.client.1.smithi067.stdout:1/676: getdents d4/d46 0 2024-04-12T05:37:28.757 INFO:tasks.workunit.client.1.smithi067.stdout:8/568: dread d6/f2c [4194304,4194304] 0 2024-04-12T05:37:28.759 INFO:tasks.workunit.client.1.smithi067.stdout:1/677: unlink d4/d9/d3a/d51/c84 0 2024-04-12T05:37:28.764 INFO:tasks.workunit.client.1.smithi067.stdout:1/678: creat d4/d3d/dbf/fd5 x:0 0 0 2024-04-12T05:37:28.766 INFO:tasks.workunit.client.1.smithi067.stdout:5/922: dread d0/d6/d33/d88/fe2 [0,4194304] 0 2024-04-12T05:37:28.766 INFO:tasks.workunit.client.1.smithi067.stdout:1/679: fdatasync d4/d9/f1c 0 2024-04-12T05:37:28.837 INFO:tasks.workunit.client.1.smithi067.stdout:8/569: dwrite d6/d8/d18/f33 [0,4194304] 0 2024-04-12T05:37:28.838 INFO:tasks.workunit.client.1.smithi067.stdout:1/680: write d4/d9/d4d/fb4 [101994,117135] 0 2024-04-12T05:37:28.841 INFO:tasks.workunit.client.1.smithi067.stdout:8/570: write d6/d8/d18/d36/f54 [532373,127973] 0 2024-04-12T05:37:28.842 INFO:tasks.workunit.client.1.smithi067.stdout:1/681: mknod d4/d9/d12/d59/cd6 0 2024-04-12T05:37:28.843 INFO:tasks.workunit.client.1.smithi067.stdout:5/923: link d0/d3a/d2c/db3/c124 d0/d6/d40/d10a/c131 0 2024-04-12T05:37:28.844 INFO:tasks.workunit.client.1.smithi067.stdout:8/571: mknod d6/d8/d18/d63/cab 0 2024-04-12T05:37:28.844 INFO:tasks.workunit.client.1.smithi067.stdout:5/924: fdatasync d0/da/f5a 0 2024-04-12T05:37:28.850 INFO:tasks.workunit.client.1.smithi067.stdout:5/925: creat d0/d6/d40/d4b/d38/d8e/d8f/f132 x:0 0 0 2024-04-12T05:37:28.855 INFO:tasks.workunit.client.1.smithi067.stdout:1/682: truncate d4/d9/d12/f1b 5193998 0 2024-04-12T05:37:28.855 INFO:tasks.workunit.client.1.smithi067.stdout:5/926: dread d0/d6/d40/d4b/fe0 [0,4194304] 0 2024-04-12T05:37:28.857 INFO:tasks.workunit.client.1.smithi067.stdout:1/683: fsync d4/d9/d14/f2a 0 2024-04-12T05:37:28.857 INFO:tasks.workunit.client.1.smithi067.stdout:5/927: truncate d0/d6/d20/d3e/fbc 4366613 0 2024-04-12T05:37:28.858 INFO:tasks.workunit.client.1.smithi067.stdout:5/928: chown d0/d6/d33/f9f 84663398 1 2024-04-12T05:37:28.864 INFO:tasks.workunit.client.1.smithi067.stdout:1/684: dwrite d4/d9/d12/d2c/fb0 [0,4194304] 0 2024-04-12T05:37:28.869 INFO:tasks.workunit.client.1.smithi067.stdout:5/929: write d0/da/f2e [84101,79189] 0 2024-04-12T05:37:28.874 INFO:tasks.workunit.client.1.smithi067.stdout:1/685: dwrite d4/d9/f75 [0,4194304] 0 2024-04-12T05:37:28.875 INFO:tasks.workunit.client.1.smithi067.stdout:5/930: rmdir d0/d6/d40/d4b/d38/d8e/d8f 39 2024-04-12T05:37:28.878 INFO:tasks.workunit.client.1.smithi067.stdout:1/686: mknod d4/d9/cd7 0 2024-04-12T05:37:28.879 INFO:tasks.workunit.client.1.smithi067.stdout:5/931: creat d0/d3a/d2c/db3/f133 x:0 0 0 2024-04-12T05:37:28.880 INFO:tasks.workunit.client.1.smithi067.stdout:5/932: stat d0/f2 0 2024-04-12T05:37:28.882 INFO:tasks.workunit.client.1.smithi067.stdout:5/933: dread - d0/d6/d33/dd3/ffe zero size 2024-04-12T05:37:28.888 INFO:tasks.workunit.client.1.smithi067.stdout:1/687: truncate d4/f7 4791872 0 2024-04-12T05:37:28.889 INFO:tasks.workunit.client.1.smithi067.stdout:5/934: write d0/f43 [382176,90176] 0 2024-04-12T05:37:28.895 INFO:tasks.workunit.client.1.smithi067.stdout:1/688: dread d4/d9/d12/f7d [0,4194304] 0 2024-04-12T05:37:28.896 INFO:tasks.workunit.client.1.smithi067.stdout:5/935: dwrite d0/d6/f6e [0,4194304] 0 2024-04-12T05:37:28.898 INFO:tasks.workunit.client.1.smithi067.stdout:6/637: sync 2024-04-12T05:37:28.900 INFO:tasks.workunit.client.1.smithi067.stdout:1/689: creat d4/d9/d12/fd8 x:0 0 0 2024-04-12T05:37:28.902 INFO:tasks.workunit.client.1.smithi067.stdout:5/936: creat d0/d6/d40/d4b/d38/d8e/d8f/db9/d10d/f134 x:0 0 0 2024-04-12T05:37:28.902 INFO:tasks.workunit.client.1.smithi067.stdout:6/638: fsync dd/d21/d64/d88/f62 0 2024-04-12T05:37:28.908 INFO:tasks.workunit.client.1.smithi067.stdout:1/690: rename d4/d9/d4d/fb4 to d4/d9/fd9 0 2024-04-12T05:37:28.911 INFO:tasks.workunit.client.1.smithi067.stdout:1/691: unlink d4/d9/d12/d59/f87 0 2024-04-12T05:37:28.915 INFO:tasks.workunit.client.1.smithi067.stdout:1/692: rename d4/d9/f17 to d4/d9/d12/d59/fda 0 2024-04-12T05:37:28.916 INFO:tasks.workunit.client.1.smithi067.stdout:1/693: chown d4/d9/cd7 10100 1 2024-04-12T05:37:28.917 INFO:tasks.workunit.client.1.smithi067.stdout:6/639: write dd/f50 [471186,46712] 0 2024-04-12T05:37:28.922 INFO:tasks.workunit.client.1.smithi067.stdout:1/694: rmdir d4/d9/d3a/d51/d55/d60 39 2024-04-12T05:37:28.925 INFO:tasks.workunit.client.1.smithi067.stdout:5/937: dwrite d0/d3a/f83 [0,4194304] 0 2024-04-12T05:37:28.925 INFO:tasks.workunit.client.1.smithi067.stdout:6/640: write dd/d11/d27/d4a/d79/dc3/f25 [4603503,11225] 0 2024-04-12T05:37:28.932 INFO:tasks.workunit.client.1.smithi067.stdout:6/641: write dd/d21/d64/fce [1080942,88614] 0 2024-04-12T05:37:28.935 INFO:tasks.workunit.client.1.smithi067.stdout:5/938: link d0/d3a/d2c/d91/fc7 d0/d6/d33/f135 0 2024-04-12T05:37:28.938 INFO:tasks.workunit.client.1.smithi067.stdout:5/939: creat d0/d6/d40/d120/f136 x:0 0 0 2024-04-12T05:37:28.942 INFO:tasks.workunit.client.1.smithi067.stdout:5/940: unlink d0/d3a/f7e 0 2024-04-12T05:37:28.942 INFO:tasks.workunit.client.1.smithi067.stdout:6/642: dwrite dd/d11/d27/d4a/d8b/fbd [0,4194304] 0 2024-04-12T05:37:28.949 INFO:tasks.workunit.client.1.smithi067.stdout:5/941: creat d0/da/f137 x:0 0 0 2024-04-12T05:37:28.950 INFO:tasks.workunit.client.1.smithi067.stdout:6/643: getdents dd/d21/d64/db5 0 2024-04-12T05:37:28.954 INFO:tasks.workunit.client.1.smithi067.stdout:6/644: creat dd/d21/d64/d88/d7b/dcc/d93/fd8 x:0 0 0 2024-04-12T05:37:28.958 INFO:tasks.workunit.client.1.smithi067.stdout:5/942: dread d0/d3a/d2c/f8b [0,4194304] 0 2024-04-12T05:37:28.966 INFO:tasks.workunit.client.1.smithi067.stdout:6/645: dwrite dd/d11/f1e [0,4194304] 0 2024-04-12T05:37:28.969 INFO:tasks.workunit.client.1.smithi067.stdout:5/943: dwrite d0/d3a/d2c/f37 [4194304,4194304] 0 2024-04-12T05:37:28.973 INFO:tasks.workunit.client.1.smithi067.stdout:5/944: fdatasync d0/d6/d33/d9e/f106 0 2024-04-12T05:37:28.973 INFO:tasks.workunit.client.1.smithi067.stdout:6/646: read dd/d11/d27/d4a/d79/dc3/f43 [2015155,30237] 0 2024-04-12T05:37:28.976 INFO:tasks.workunit.client.1.smithi067.stdout:6/647: truncate dd/d11/fac 1835080 0 2024-04-12T05:37:28.981 INFO:tasks.workunit.client.1.smithi067.stdout:5/945: read d0/da/f5a [1595168,84540] 0 2024-04-12T05:37:28.984 INFO:tasks.workunit.client.1.smithi067.stdout:5/946: mkdir d0/d6/d40/d138 0 2024-04-12T05:37:28.986 INFO:tasks.workunit.client.1.smithi067.stdout:5/947: mknod d0/d6/d20/d3e/c139 0 2024-04-12T05:37:28.988 INFO:tasks.workunit.client.1.smithi067.stdout:5/948: symlink d0/l13a 0 2024-04-12T05:37:28.989 INFO:tasks.workunit.client.1.smithi067.stdout:5/949: dread - d0/d3a/d2c/fe9 zero size 2024-04-12T05:37:28.991 INFO:tasks.workunit.client.1.smithi067.stdout:5/950: truncate d0/d3a/d2c/fcd 932536 0 2024-04-12T05:37:28.992 INFO:tasks.workunit.client.1.smithi067.stdout:5/951: chown d0/d6/d20/d3e/dbf/d115/daa 0 1 2024-04-12T05:37:28.995 INFO:tasks.workunit.client.1.smithi067.stdout:5/952: creat d0/d6/d40/d4b/f13b x:0 0 0 2024-04-12T05:37:28.996 INFO:tasks.workunit.client.1.smithi067.stdout:5/953: fsync d0/d3a/f4c 0 2024-04-12T05:37:28.999 INFO:tasks.workunit.client.1.smithi067.stdout:6/648: dread dd/d11/d27/f46 [0,4194304] 0 2024-04-12T05:37:29.002 INFO:tasks.workunit.client.1.smithi067.stdout:6/649: unlink dd/d21/d64/d88/c41 0 2024-04-12T05:37:29.004 INFO:tasks.workunit.client.1.smithi067.stdout:5/954: getdents d0/d6/d40/d120 0 2024-04-12T05:37:29.006 INFO:tasks.workunit.client.1.smithi067.stdout:5/955: read d0/d3a/d2c/fcd [745704,39795] 0 2024-04-12T05:37:29.010 INFO:tasks.workunit.client.1.smithi067.stdout:5/956: rename d0/d6/d33/f9f to d0/d6/d20/d3e/dbf/f13c 0 2024-04-12T05:37:29.012 INFO:tasks.workunit.client.1.smithi067.stdout:5/957: unlink d0/d6/d40/d4b/f13b 0 2024-04-12T05:37:29.018 INFO:tasks.workunit.client.1.smithi067.stdout:6/650: dread dd/d11/f5b [0,4194304] 0 2024-04-12T05:37:29.019 INFO:tasks.workunit.client.1.smithi067.stdout:5/958: dwrite d0/da/d41/f7a [0,4194304] 0 2024-04-12T05:37:29.023 INFO:tasks.workunit.client.1.smithi067.stdout:6/651: rename dd/d11/d27/d4a/d79/fb3 to dd/d21/d3f/dd5/fd9 0 2024-04-12T05:37:29.025 INFO:tasks.workunit.client.1.smithi067.stdout:5/959: chown d0/d6/d20/d3e/dbf/d115/daa/lc4 104 1 2024-04-12T05:37:29.027 INFO:tasks.workunit.client.1.smithi067.stdout:5/960: readlink d0/d6/d40/d4b/d38/d8e/d8f/l101 0 2024-04-12T05:37:29.032 INFO:tasks.workunit.client.1.smithi067.stdout:5/961: unlink d0/da/led 0 2024-04-12T05:37:29.036 INFO:tasks.workunit.client.1.smithi067.stdout:5/962: dread d0/da/d41/f9b [0,4194304] 0 2024-04-12T05:37:29.038 INFO:tasks.workunit.client.1.smithi067.stdout:5/963: mkdir d0/d6/d40/d120/d125/d13d 0 2024-04-12T05:37:29.045 INFO:tasks.workunit.client.1.smithi067.stdout:5/964: dwrite d0/d6/d40/d4b/d38/d8e/ffb [0,4194304] 0 2024-04-12T05:37:29.049 INFO:tasks.workunit.client.1.smithi067.stdout:5/965: mkdir d0/d6/d20/d3e/dbf/d115/d82/d13e 0 2024-04-12T05:37:29.054 INFO:tasks.workunit.client.1.smithi067.stdout:5/966: mkdir d0/d6/d20/d62/d13f 0 2024-04-12T05:37:29.059 INFO:tasks.workunit.client.1.smithi067.stdout:5/967: write d0/d6/d40/d4b/d38/d8e/d8f/f132 [708718,17008] 0 2024-04-12T05:37:29.060 INFO:tasks.workunit.client.1.smithi067.stdout:5/968: readlink d0/d6/d20/d3e/dbf/ld6 0 2024-04-12T05:37:29.066 INFO:tasks.workunit.client.1.smithi067.stdout:5/969: write d0/d6/d40/d4b/d38/d8e/d8f/db9/d10d/f134 [477646,46161] 0 2024-04-12T05:37:29.067 INFO:tasks.workunit.client.1.smithi067.stdout:5/970: chown d0/d3a/d2c/f100 45416813 1 2024-04-12T05:37:29.079 INFO:tasks.workunit.client.1.smithi067.stdout:5/971: rename d0/d6/d40/c84 to d0/d6/d20/d3e/dbf/d115/d82/d13e/c140 0 2024-04-12T05:37:29.254 INFO:tasks.workunit.client.1.smithi067.stdout:3/547: sync 2024-04-12T05:37:29.254 INFO:tasks.workunit.client.1.smithi067.stdout:9/634: sync 2024-04-12T05:37:29.255 INFO:tasks.workunit.client.1.smithi067.stdout:2/682: sync 2024-04-12T05:37:29.255 INFO:tasks.workunit.client.1.smithi067.stdout:7/722: sync 2024-04-12T05:37:29.255 INFO:tasks.workunit.client.1.smithi067.stdout:9/635: dread - d1/d28/d5e/fb5 zero size 2024-04-12T05:37:29.256 INFO:tasks.workunit.client.1.smithi067.stdout:3/548: write dc/df/d27/f6a [3246977,120632] 0 2024-04-12T05:37:29.256 INFO:tasks.workunit.client.1.smithi067.stdout:2/683: mknod d5/d6a/d8f/cdb 0 2024-04-12T05:37:29.339 INFO:tasks.workunit.client.1.smithi067.stdout:3/549: mkdir dc/d1d/d28/d3e/d69/dbc 0 2024-04-12T05:37:29.339 INFO:tasks.workunit.client.1.smithi067.stdout:2/684: mkdir d5/da/de/dc4/dc6/ddc 0 2024-04-12T05:37:29.346 INFO:tasks.workunit.client.1.smithi067.stdout:9/636: dwrite d1/d28/f18 [0,4194304] 0 2024-04-12T05:37:29.348 INFO:tasks.workunit.client.1.smithi067.stdout:2/685: rename d5/da/de/c2d to d5/da/de/d14/d34/d4c/dbc/cdd 0 2024-04-12T05:37:29.350 INFO:tasks.workunit.client.1.smithi067.stdout:3/550: dwrite dc/d1d/d28/f4c [0,4194304] 0 2024-04-12T05:37:29.355 INFO:tasks.workunit.client.1.smithi067.stdout:3/551: chown dc/d50/l59 6147251 1 2024-04-12T05:37:29.357 INFO:tasks.workunit.client.1.smithi067.stdout:3/552: read dc/d1d/d26/f2f [3560557,37989] 0 2024-04-12T05:37:29.357 INFO:tasks.workunit.client.1.smithi067.stdout:2/686: mkdir d5/da/de/d39/dde 0 2024-04-12T05:37:29.360 INFO:tasks.workunit.client.1.smithi067.stdout:3/553: rmdir dc/d50 39 2024-04-12T05:37:29.361 INFO:tasks.workunit.client.1.smithi067.stdout:3/554: stat dc/d1d/d28/d3e/d53/d76/cae 0 2024-04-12T05:37:29.363 INFO:tasks.workunit.client.1.smithi067.stdout:3/555: write dc/df/d27/f96 [783154,7077] 0 2024-04-12T05:37:29.363 INFO:tasks.workunit.client.1.smithi067.stdout:2/687: mkdir d5/da/d8c/ddf 0 2024-04-12T05:37:29.365 INFO:tasks.workunit.client.1.smithi067.stdout:2/688: read d5/da/de/d14/f63 [1787778,57870] 0 2024-04-12T05:37:29.367 INFO:tasks.workunit.client.1.smithi067.stdout:2/689: dread - d5/d11/f9b zero size 2024-04-12T05:37:29.368 INFO:tasks.workunit.client.1.smithi067.stdout:3/556: mkdir dc/d50/da6/dbd 0 2024-04-12T05:37:29.374 INFO:tasks.workunit.client.1.smithi067.stdout:2/690: link d5/da/de/d14/f36 d5/da/de/fe0 0 2024-04-12T05:37:29.379 INFO:tasks.workunit.client.1.smithi067.stdout:3/557: dwrite dc/f8e [0,4194304] 0 2024-04-12T05:37:29.379 INFO:tasks.workunit.client.1.smithi067.stdout:2/691: getdents d5/da/de/d14/d34/d40/d57/d96 0 2024-04-12T05:37:29.383 INFO:tasks.workunit.client.1.smithi067.stdout:9/637: dwrite d1/d28/d21/d3f/d4f/faf [4194304,4194304] 0 2024-04-12T05:37:29.387 INFO:tasks.workunit.client.1.smithi067.stdout:3/558: dread dc/f8c [0,4194304] 0 2024-04-12T05:37:29.387 INFO:tasks.workunit.client.1.smithi067.stdout:9/638: readlink d1/d13/l37 0 2024-04-12T05:37:29.388 INFO:tasks.workunit.client.1.smithi067.stdout:2/692: dread d5/da/f3e [0,4194304] 0 2024-04-12T05:37:29.390 INFO:tasks.workunit.client.1.smithi067.stdout:2/693: fsync d5/f12 0 2024-04-12T05:37:29.406 INFO:tasks.workunit.client.1.smithi067.stdout:0/728: sync 2024-04-12T05:37:29.406 INFO:tasks.workunit.client.1.smithi067.stdout:8/572: sync 2024-04-12T05:37:29.406 INFO:tasks.workunit.client.1.smithi067.stdout:4/591: sync 2024-04-12T05:37:29.411 INFO:tasks.workunit.client.1.smithi067.stdout:8/573: write d6/d8/d6e/f78 [4580058,116530] 0 2024-04-12T05:37:29.426 INFO:tasks.workunit.client.1.smithi067.stdout:3/559: creat dc/df/d27/daa/fbe x:0 0 0 2024-04-12T05:37:29.431 INFO:tasks.workunit.client.1.smithi067.stdout:2/694: symlink d5/da/dae/le1 0 2024-04-12T05:37:29.432 INFO:tasks.workunit.client.1.smithi067.stdout:4/592: fdatasync d8/da/d1d/f49 0 2024-04-12T05:37:29.432 INFO:tasks.workunit.client.1.smithi067.stdout:0/729: dread dd/d12/f3e [4194304,4194304] 0 2024-04-12T05:37:29.434 INFO:tasks.workunit.client.1.smithi067.stdout:2/695: dread - d5/da/de/d14/d46/d90/fd5 zero size 2024-04-12T05:37:29.436 INFO:tasks.workunit.client.1.smithi067.stdout:0/730: chown dd/d9c/d5f/d78/d83/f9a 104108 1 2024-04-12T05:37:29.436 INFO:tasks.workunit.client.1.smithi067.stdout:4/593: write d8/da/d9f/db5/fbe [425886,60633] 0 2024-04-12T05:37:29.436 INFO:tasks.workunit.client.1.smithi067.stdout:9/639: symlink d1/d28/d21/d3f/lc0 0 2024-04-12T05:37:29.437 INFO:tasks.workunit.client.1.smithi067.stdout:3/560: read dc/d1d/d26/f2f [3107267,75748] 0 2024-04-12T05:37:29.449 INFO:tasks.workunit.client.1.smithi067.stdout:2/696: symlink d5/da/d8c/dc2/le2 0 2024-04-12T05:37:29.454 INFO:tasks.workunit.client.1.smithi067.stdout:8/574: dread d6/d8/d18/d36/f59 [4194304,4194304] 0 2024-04-12T05:37:29.462 INFO:tasks.workunit.client.1.smithi067.stdout:9/640: dwrite d1/d28/d21/d3f/d4f/faa [0,4194304] 0 2024-04-12T05:37:29.463 INFO:tasks.workunit.client.1.smithi067.stdout:2/697: chown d5/da/cb8 28767891 1 2024-04-12T05:37:29.468 INFO:tasks.workunit.client.1.smithi067.stdout:3/561: link dc/d1d/d28/f38 dc/d1d/d28/d3e/d53/d94/fbf 0 2024-04-12T05:37:29.468 INFO:tasks.workunit.client.1.smithi067.stdout:4/594: dread d8/da/d9f/d68/dbf/f97 [0,4194304] 0 2024-04-12T05:37:29.470 INFO:tasks.workunit.client.1.smithi067.stdout:9/641: stat d1/d28/d21/d3f/d4f/fbe 0 2024-04-12T05:37:29.475 INFO:tasks.workunit.client.1.smithi067.stdout:8/575: dread d6/d8/d8b/f8d [0,4194304] 0 2024-04-12T05:37:29.475 INFO:tasks.workunit.client.1.smithi067.stdout:2/698: mkdir d5/da/de/d14/d34/d4c/dbc/de3 0 2024-04-12T05:37:29.476 INFO:tasks.workunit.client.1.smithi067.stdout:3/562: unlink dc/df/d27/daa/cb7 0 2024-04-12T05:37:29.485 INFO:tasks.workunit.client.1.smithi067.stdout:8/576: read d6/d8/d18/d63/d9a/f23 [2072096,26613] 0 2024-04-12T05:37:29.486 INFO:tasks.workunit.client.1.smithi067.stdout:9/642: dread d1/d28/d5e/f67 [4194304,4194304] 0 2024-04-12T05:37:29.486 INFO:tasks.workunit.client.1.smithi067.stdout:2/699: dread d5/da/dae/fbb [0,4194304] 0 2024-04-12T05:37:29.487 INFO:tasks.workunit.client.1.smithi067.stdout:0/731: dread dd/d6f/d72/f82 [0,4194304] 0 2024-04-12T05:37:29.491 INFO:tasks.workunit.client.1.smithi067.stdout:9/643: fsync d1/d13/d34/fab 0 2024-04-12T05:37:29.491 INFO:tasks.workunit.client.1.smithi067.stdout:4/595: dwrite d8/dba/d5d/d66/d7b/fbb [0,4194304] 0 2024-04-12T05:37:29.494 INFO:tasks.workunit.client.1.smithi067.stdout:0/732: mknod dd/d12/d14/d1a/d3d/ce7 0 2024-04-12T05:37:29.500 INFO:tasks.workunit.client.1.smithi067.stdout:9/644: mkdir d1/d28/d44/d8d/dc1 0 2024-04-12T05:37:29.503 INFO:tasks.workunit.client.1.smithi067.stdout:0/733: chown dd/d9c/d5f/d78/d83/c85 15100 1 2024-04-12T05:37:29.504 INFO:tasks.workunit.client.1.smithi067.stdout:3/563: dwrite dc/df/d27/f2d [0,4194304] 0 2024-04-12T05:37:29.506 INFO:tasks.workunit.client.1.smithi067.stdout:3/564: write dc/f8e [2237287,16892] 0 2024-04-12T05:37:29.506 INFO:tasks.workunit.client.1.smithi067.stdout:2/700: write d5/da/de/d14/d34/d4c/f6b [1901135,19098] 0 2024-04-12T05:37:29.508 INFO:tasks.workunit.client.1.smithi067.stdout:9/645: truncate d1/fd 5140399 0 2024-04-12T05:37:29.512 INFO:tasks.workunit.client.1.smithi067.stdout:0/734: write dd/d12/d14/d49/dab/fcf [594270,45028] 0 2024-04-12T05:37:29.512 INFO:tasks.workunit.client.1.smithi067.stdout:4/596: dread d8/d9/d3d/f3f [4194304,4194304] 0 2024-04-12T05:37:29.514 INFO:tasks.workunit.client.1.smithi067.stdout:4/597: creat d8/da/d9f/d68/fc6 x:0 0 0 2024-04-12T05:37:29.517 INFO:tasks.workunit.client.1.smithi067.stdout:9/646: write d1/d28/d21/d3f/d4f/faa [131315,90095] 0 2024-04-12T05:37:29.520 INFO:tasks.workunit.client.1.smithi067.stdout:8/577: dwrite d6/d8/d8b/f8d [0,4194304] 0 2024-04-12T05:37:29.521 INFO:tasks.workunit.client.1.smithi067.stdout:4/598: write d8/da/d1d/d15/d75/daa/f61 [2819946,47800] 0 2024-04-12T05:37:29.525 INFO:tasks.workunit.client.1.smithi067.stdout:3/565: rename dc/df/d27/daa/cb5 to dc/d1d/d89/da3/cc0 0 2024-04-12T05:37:29.526 INFO:tasks.workunit.client.1.smithi067.stdout:6/652: sync 2024-04-12T05:37:29.526 INFO:tasks.workunit.client.1.smithi067.stdout:1/695: sync 2024-04-12T05:37:29.527 INFO:tasks.workunit.client.1.smithi067.stdout:2/701: dwrite d5/d11/f8a [0,4194304] 0 2024-04-12T05:37:29.529 INFO:tasks.workunit.client.1.smithi067.stdout:8/578: symlink d6/d86/lac 0 2024-04-12T05:37:29.530 INFO:tasks.workunit.client.1.smithi067.stdout:1/696: write d4/d9/f75 [873704,1500] 0 2024-04-12T05:37:29.533 INFO:tasks.workunit.client.1.smithi067.stdout:6/653: write dd/d11/d27/f54 [1698196,51694] 0 2024-04-12T05:37:29.537 INFO:tasks.workunit.client.1.smithi067.stdout:6/654: mknod dd/d21/d64/d88/cda 0 2024-04-12T05:37:29.537 INFO:tasks.workunit.client.1.smithi067.stdout:3/566: mknod dc/d1d/d28/d3e/d69/dbc/cc1 0 2024-04-12T05:37:29.538 INFO:tasks.workunit.client.1.smithi067.stdout:9/647: dwrite d1/d28/d21/d3f/d4f/faa [0,4194304] 0 2024-04-12T05:37:29.542 INFO:tasks.workunit.client.1.smithi067.stdout:6/655: mkdir dd/d21/d64/d88/d7b/dd2/ddb 0 2024-04-12T05:37:29.542 INFO:tasks.workunit.client.1.smithi067.stdout:9/648: stat d1/d28/fb8 0 2024-04-12T05:37:29.545 INFO:tasks.workunit.client.1.smithi067.stdout:1/697: write d4/d3d/dab/f39 [4550122,32605] 0 2024-04-12T05:37:29.549 INFO:tasks.workunit.client.1.smithi067.stdout:9/649: creat d1/d66/d8e/fc2 x:0 0 0 2024-04-12T05:37:29.550 INFO:tasks.workunit.client.1.smithi067.stdout:2/702: write d5/da/d8c/faf [944145,65876] 0 2024-04-12T05:37:29.553 INFO:tasks.workunit.client.1.smithi067.stdout:4/599: dwrite d8/f2b [0,4194304] 0 2024-04-12T05:37:29.558 INFO:tasks.workunit.client.1.smithi067.stdout:9/650: creat d1/d13/d34/fc3 x:0 0 0 2024-04-12T05:37:29.559 INFO:tasks.workunit.client.1.smithi067.stdout:4/600: mknod d8/da/d8b/d45/cc7 0 2024-04-12T05:37:29.562 INFO:tasks.workunit.client.1.smithi067.stdout:1/698: dwrite d4/d46/f96 [0,4194304] 0 2024-04-12T05:37:29.566 INFO:tasks.workunit.client.1.smithi067.stdout:9/651: chown d1/d12/f2e 6380457 1 2024-04-12T05:37:29.568 INFO:tasks.workunit.client.1.smithi067.stdout:4/601: write d8/d9/d3d/f93 [3734920,13891] 0 2024-04-12T05:37:29.572 INFO:tasks.workunit.client.1.smithi067.stdout:2/703: dwrite d5/d43/f53 [0,4194304] 0 2024-04-12T05:37:29.572 INFO:tasks.workunit.client.1.smithi067.stdout:1/699: chown d4/d9/d3a/la3 169986158 1 2024-04-12T05:37:29.574 INFO:tasks.workunit.client.1.smithi067.stdout:9/652: truncate d1/d28/d21/d68/fba 974419 0 2024-04-12T05:37:29.578 INFO:tasks.workunit.client.1.smithi067.stdout:4/602: symlink d8/dba/d5d/d66/d7b/d6d/lc8 0 2024-04-12T05:37:29.579 INFO:tasks.workunit.client.1.smithi067.stdout:9/653: creat d1/d13/db7/fc4 x:0 0 0 2024-04-12T05:37:29.580 INFO:tasks.workunit.client.1.smithi067.stdout:2/704: getdents d5/da/de/d14/d34/d4c/dbc/de3 0 2024-04-12T05:37:29.582 INFO:tasks.workunit.client.1.smithi067.stdout:6/656: dread dd/d11/d27/d4a/d79/dc3/d56/f99 [0,4194304] 0 2024-04-12T05:37:29.583 INFO:tasks.workunit.client.1.smithi067.stdout:6/657: dread - dd/d21/fcb zero size 2024-04-12T05:37:29.586 INFO:tasks.workunit.client.1.smithi067.stdout:9/654: unlink d1/d12/d15/f1c 0 2024-04-12T05:37:29.587 INFO:tasks.workunit.client.1.smithi067.stdout:2/705: symlink d5/d43/da4/le4 0 2024-04-12T05:37:29.587 INFO:tasks.workunit.client.1.smithi067.stdout:9/655: chown d1/d13/l2f 109568 1 2024-04-12T05:37:29.589 INFO:tasks.workunit.client.1.smithi067.stdout:2/706: write d5/d11/f29 [1536888,120663] 0 2024-04-12T05:37:29.589 INFO:tasks.workunit.client.1.smithi067.stdout:1/700: unlink d4/d8f/l9d 0 2024-04-12T05:37:29.591 INFO:tasks.workunit.client.1.smithi067.stdout:4/603: mknod d8/dba/d5d/d66/d7b/dbd/cc9 0 2024-04-12T05:37:29.595 INFO:tasks.workunit.client.1.smithi067.stdout:1/701: dread d4/d9/f78 [0,4194304] 0 2024-04-12T05:37:29.595 INFO:tasks.workunit.client.1.smithi067.stdout:4/604: creat d8/dba/d5d/d66/d7b/dbd/fca x:0 0 0 2024-04-12T05:37:29.596 INFO:tasks.workunit.client.1.smithi067.stdout:2/707: creat d5/fe5 x:0 0 0 2024-04-12T05:37:29.598 INFO:tasks.workunit.client.1.smithi067.stdout:1/702: rmdir d4/d52/d80 39 2024-04-12T05:37:29.601 INFO:tasks.workunit.client.1.smithi067.stdout:6/658: write dd/d11/d27/d4a/d79/dc3/d56/f69 [4857471,30677] 0 2024-04-12T05:37:29.604 INFO:tasks.workunit.client.1.smithi067.stdout:1/703: creat d4/d9/d12/d2c/fdb x:0 0 0 2024-04-12T05:37:29.610 INFO:tasks.workunit.client.1.smithi067.stdout:4/605: dwrite d8/d9/f2a [0,4194304] 0 2024-04-12T05:37:29.613 INFO:tasks.workunit.client.1.smithi067.stdout:1/704: truncate d4/f7 2646010 0 2024-04-12T05:37:29.613 INFO:tasks.workunit.client.1.smithi067.stdout:6/659: creat dd/d11/d27/d4a/d8b/fdc x:0 0 0 2024-04-12T05:37:29.614 INFO:tasks.workunit.client.1.smithi067.stdout:2/708: dwrite d5/da/de/d14/d34/d4c/f75 [0,4194304] 0 2024-04-12T05:37:29.615 INFO:tasks.workunit.client.1.smithi067.stdout:9/656: dwrite d1/d28/d44/d8d/fa0 [0,4194304] 0 2024-04-12T05:37:29.616 INFO:tasks.workunit.client.1.smithi067.stdout:1/705: chown d4/d3d/dab/f4f 5 1 2024-04-12T05:37:29.621 INFO:tasks.workunit.client.1.smithi067.stdout:4/606: unlink d8/da/d1d/f62 0 2024-04-12T05:37:29.622 INFO:tasks.workunit.client.1.smithi067.stdout:2/709: fdatasync d5/da/de/d14/d34/d40/f7c 0 2024-04-12T05:37:29.624 INFO:tasks.workunit.client.1.smithi067.stdout:9/657: mknod d1/d12/cc5 0 2024-04-12T05:37:29.626 INFO:tasks.workunit.client.1.smithi067.stdout:6/660: getdents dd/d11/d27/d4a/d79 0 2024-04-12T05:37:29.629 INFO:tasks.workunit.client.1.smithi067.stdout:6/661: fsync dd/f50 0 2024-04-12T05:37:29.630 INFO:tasks.workunit.client.1.smithi067.stdout:2/710: write d5/d23/d3a/f59 [2590047,88108] 0 2024-04-12T05:37:29.637 INFO:tasks.workunit.client.1.smithi067.stdout:4/607: dwrite d8/da/d8b/fa7 [0,4194304] 0 2024-04-12T05:37:29.637 INFO:tasks.workunit.client.1.smithi067.stdout:9/658: creat d1/d66/d8e/fc6 x:0 0 0 2024-04-12T05:37:29.641 INFO:tasks.workunit.client.1.smithi067.stdout:6/662: fdatasync dd/d11/d27/d4a/d8b/fbd 0 2024-04-12T05:37:29.641 INFO:tasks.workunit.client.1.smithi067.stdout:1/706: dwrite d4/d9/d3a/d51/d55/fc8 [0,4194304] 0 2024-04-12T05:37:29.644 INFO:tasks.workunit.client.1.smithi067.stdout:6/663: readlink dd/d11/d27/l3e 0 2024-04-12T05:37:29.646 INFO:tasks.workunit.client.1.smithi067.stdout:6/664: write dd/f3b [4435484,97779] 0 2024-04-12T05:37:29.647 INFO:tasks.workunit.client.1.smithi067.stdout:1/707: symlink d4/d3d/ldc 0 2024-04-12T05:37:29.648 INFO:tasks.workunit.client.1.smithi067.stdout:2/711: dread d5/d43/d6e/f72 [0,4194304] 0 2024-04-12T05:37:29.653 INFO:tasks.workunit.client.1.smithi067.stdout:1/708: mkdir d4/d9/d3a/d51/d55/dad/ddd 0 2024-04-12T05:37:29.656 INFO:tasks.workunit.client.1.smithi067.stdout:6/665: rename dd/d21/d64/d88/d7b/dcc/d93/dba to dd/d21/d64/d88/d7b/dd2/ddd 0 2024-04-12T05:37:29.658 INFO:tasks.workunit.client.1.smithi067.stdout:9/659: dwrite d1/d12/f53 [0,4194304] 0 2024-04-12T05:37:29.662 INFO:tasks.workunit.client.1.smithi067.stdout:2/712: dwrite d5/d11/f8a [0,4194304] 0 2024-04-12T05:37:29.665 INFO:tasks.workunit.client.1.smithi067.stdout:1/709: rename d4/d9/d5f to d4/d3d/d7b/dde 0 2024-04-12T05:37:29.667 INFO:tasks.workunit.client.1.smithi067.stdout:4/608: dread d8/da/d9f/f52 [0,4194304] 0 2024-04-12T05:37:29.667 INFO:tasks.workunit.client.1.smithi067.stdout:2/713: truncate d5/da/de/f94 1067503 0 2024-04-12T05:37:29.669 INFO:tasks.workunit.client.1.smithi067.stdout:1/710: symlink d4/d9/d4c/ldf 0 2024-04-12T05:37:29.671 INFO:tasks.workunit.client.1.smithi067.stdout:9/660: link d1/d28/d44/f72 d1/d28/d44/fc7 0 2024-04-12T05:37:29.672 INFO:tasks.workunit.client.1.smithi067.stdout:2/714: chown d5/d11/f2a 81406 1 2024-04-12T05:37:29.674 INFO:tasks.workunit.client.1.smithi067.stdout:9/661: symlink d1/d28/d21/d3f/lc8 0 2024-04-12T05:37:29.676 INFO:tasks.workunit.client.1.smithi067.stdout:4/609: truncate d8/d9/f42 743092 0 2024-04-12T05:37:29.678 INFO:tasks.workunit.client.1.smithi067.stdout:9/662: fdatasync d1/d12/f2e 0 2024-04-12T05:37:29.682 INFO:tasks.workunit.client.1.smithi067.stdout:9/663: creat d1/d28/d21/d3f/d4f/fc9 x:0 0 0 2024-04-12T05:37:29.685 INFO:tasks.workunit.client.1.smithi067.stdout:9/664: mkdir d1/d66/dca 0 2024-04-12T05:37:29.687 INFO:tasks.workunit.client.1.smithi067.stdout:4/610: dwrite d8/da/f18 [0,4194304] 0 2024-04-12T05:37:29.695 INFO:tasks.workunit.client.1.smithi067.stdout:4/611: chown d8/da/d9f/l23 10 1 2024-04-12T05:37:29.697 INFO:tasks.workunit.client.1.smithi067.stdout:1/711: truncate d4/d44/f66 3924614 0 2024-04-12T05:37:29.699 INFO:tasks.workunit.client.1.smithi067.stdout:1/712: dread - d4/d9/d12/fd8 zero size 2024-04-12T05:37:29.699 INFO:tasks.workunit.client.1.smithi067.stdout:2/715: write d5/da/de/d39/f6c [952321,105301] 0 2024-04-12T05:37:29.701 INFO:tasks.workunit.client.1.smithi067.stdout:9/665: dread d1/d28/d21/d45/d75/db3/f63 [0,4194304] 0 2024-04-12T05:37:29.702 INFO:tasks.workunit.client.1.smithi067.stdout:2/716: chown d5/d11/f50 553791 1 2024-04-12T05:37:29.712 INFO:tasks.workunit.client.1.smithi067.stdout:2/717: truncate d5/d23/d93/f98 1305441 0 2024-04-12T05:37:29.712 INFO:tasks.workunit.client.1.smithi067.stdout:9/666: mkdir d1/d28/d44/d8d/dc1/dcb 0 2024-04-12T05:37:29.714 INFO:tasks.workunit.client.1.smithi067.stdout:1/713: link d4/d9/d4c/fcd d4/d52/d80/fe0 0 2024-04-12T05:37:29.718 INFO:tasks.workunit.client.1.smithi067.stdout:9/667: rename d1/d28/d21/d45/d75/db3/f31 to d1/d5f/fcc 0 2024-04-12T05:37:29.719 INFO:tasks.workunit.client.1.smithi067.stdout:1/714: fdatasync d4/d9/d4c/fa7 0 2024-04-12T05:37:29.721 INFO:tasks.workunit.client.1.smithi067.stdout:9/668: write d1/d28/d27/f71 [3552808,36201] 0 2024-04-12T05:37:29.722 INFO:tasks.workunit.client.1.smithi067.stdout:4/612: dread d8/da/d1d/d27/d91/fa8 [4194304,4194304] 0 2024-04-12T05:37:29.723 INFO:tasks.workunit.client.1.smithi067.stdout:4/613: readlink d8/dba/d5d/d66/d6a/l70 0 2024-04-12T05:37:29.723 INFO:tasks.workunit.client.1.smithi067.stdout:1/715: mkdir d4/d9/d4d/de1 0 2024-04-12T05:37:29.726 INFO:tasks.workunit.client.1.smithi067.stdout:9/669: truncate d1/d28/d21/d45/d75/db3/f24 352459 0 2024-04-12T05:37:29.731 INFO:tasks.workunit.client.1.smithi067.stdout:4/614: fdatasync d8/da/d8b/d45/f6e 0 2024-04-12T05:37:29.734 INFO:tasks.workunit.client.1.smithi067.stdout:2/718: dread d5/d6a/f65 [4194304,4194304] 0 2024-04-12T05:37:29.737 INFO:tasks.workunit.client.1.smithi067.stdout:9/670: link d1/d28/d44/c92 d1/d28/d21/d68/ccd 0 2024-04-12T05:37:29.743 INFO:tasks.workunit.client.1.smithi067.stdout:4/615: dwrite d8/da/d1d/d15/d75/f9e [0,4194304] 0 2024-04-12T05:37:29.743 INFO:tasks.workunit.client.1.smithi067.stdout:1/716: write d4/d9/d12/d2c/f37 [1205161,5329] 0 2024-04-12T05:37:29.747 INFO:tasks.workunit.client.1.smithi067.stdout:2/719: symlink d5/d23/d93/dd1/le6 0 2024-04-12T05:37:29.748 INFO:tasks.workunit.client.1.smithi067.stdout:4/616: unlink d8/da/d9f/d68/fc6 0 2024-04-12T05:37:29.750 INFO:tasks.workunit.client.1.smithi067.stdout:4/617: fdatasync d8/da/d1d/d15/d75/f7c 0 2024-04-12T05:37:29.751 INFO:tasks.workunit.client.1.smithi067.stdout:9/671: dread d1/d28/d21/d3f/d4f/faf [4194304,4194304] 0 2024-04-12T05:37:29.753 INFO:tasks.workunit.client.1.smithi067.stdout:5/972: sync 2024-04-12T05:37:29.754 INFO:tasks.workunit.client.1.smithi067.stdout:7/723: sync 2024-04-12T05:37:29.755 INFO:tasks.workunit.client.1.smithi067.stdout:1/717: link d4/d9/d3a/fcb d4/d9/d12/d2c/fe2 0 2024-04-12T05:37:29.756 INFO:tasks.workunit.client.1.smithi067.stdout:4/618: mknod d8/da/d9f/d68/ccb 0 2024-04-12T05:37:29.758 INFO:tasks.workunit.client.1.smithi067.stdout:5/973: mknod d0/d6/d40/d10a/c141 0 2024-04-12T05:37:29.760 INFO:tasks.workunit.client.1.smithi067.stdout:9/672: rename d1/d28/d44/c55 to d1/d66/d8e/cce 0 2024-04-12T05:37:29.761 INFO:tasks.workunit.client.1.smithi067.stdout:7/724: creat d0/d23/d80/da0/db4/d5/d1d/de9/de7/ffa x:0 0 0 2024-04-12T05:37:29.796 INFO:tasks.workunit.client.1.smithi067.stdout:9/673: creat d1/d28/d21/dae/fcf x:0 0 0 2024-04-12T05:37:29.796 INFO:tasks.workunit.client.1.smithi067.stdout:2/720: write d5/da/de/d14/d34/f37 [3640045,94939] 0 2024-04-12T05:37:29.796 INFO:tasks.workunit.client.1.smithi067.stdout:5/974: mknod d0/d6/d33/c142 0 2024-04-12T05:37:29.799 INFO:tasks.workunit.client.1.smithi067.stdout:7/725: mkdir d0/d23/d80/da0/db4/dd/d42/d82/dfb 0 2024-04-12T05:37:29.800 INFO:tasks.workunit.client.1.smithi067.stdout:9/674: mknod d1/d12/d15/cd0 0 2024-04-12T05:37:29.801 INFO:tasks.workunit.client.1.smithi067.stdout:5/975: mkdir d0/d3a/d2c/dc6/ddd/d143 0 2024-04-12T05:37:29.808 INFO:tasks.workunit.client.1.smithi067.stdout:1/718: rename d4/d9/d14/f21 to d4/fe3 0 2024-04-12T05:37:29.808 INFO:tasks.workunit.client.1.smithi067.stdout:7/726: unlink d0/d23/d80/da0/db4/l90 0 2024-04-12T05:37:29.809 INFO:tasks.workunit.client.1.smithi067.stdout:5/976: creat d0/da/dec/f144 x:0 0 0 2024-04-12T05:37:29.809 INFO:tasks.workunit.client.1.smithi067.stdout:7/727: write d0/d23/d80/fcb [3427113,47354] 0 2024-04-12T05:37:29.810 INFO:tasks.workunit.client.1.smithi067.stdout:5/977: dread - d0/da/d111/f12a zero size 2024-04-12T05:37:29.812 INFO:tasks.workunit.client.1.smithi067.stdout:4/619: rename d8/dba/d5d/d66/d6a to d8/dba/d5d/d66/d7b/dbd/dcc 0 2024-04-12T05:37:29.816 INFO:tasks.workunit.client.1.smithi067.stdout:1/719: dread d4/fe3 [0,4194304] 0 2024-04-12T05:37:29.822 INFO:tasks.workunit.client.1.smithi067.stdout:2/721: link d5/da/de/d14/d34/d40/c97 d5/da/d8c/ddf/ce7 0 2024-04-12T05:37:29.827 INFO:tasks.workunit.client.1.smithi067.stdout:9/675: dread d1/d28/d27/f71 [0,4194304] 0 2024-04-12T05:37:29.830 INFO:tasks.workunit.client.1.smithi067.stdout:9/676: write d1/d28/d21/d3f/f41 [5861389,4926] 0 2024-04-12T05:37:29.832 INFO:tasks.workunit.client.1.smithi067.stdout:1/720: dread d4/d9/d3a/d51/f82 [0,4194304] 0 2024-04-12T05:37:29.833 INFO:tasks.workunit.client.1.smithi067.stdout:7/728: dread d0/d23/d80/da0/db4/d5/f50 [0,4194304] 0 2024-04-12T05:37:29.836 INFO:tasks.workunit.client.1.smithi067.stdout:9/677: creat d1/d28/fd1 x:0 0 0 2024-04-12T05:37:29.837 INFO:tasks.workunit.client.1.smithi067.stdout:1/721: mknod d4/d9/d14/ce4 0 2024-04-12T05:37:29.840 INFO:tasks.workunit.client.1.smithi067.stdout:7/729: truncate d0/d23/d80/da0/db4/dd/d42/d71/dbd/fc1 532160 0 2024-04-12T05:37:29.841 INFO:tasks.workunit.client.1.smithi067.stdout:9/678: rename d1/d28/d21/dae to d1/d28/d44/d8d/dd2 0 2024-04-12T05:37:29.842 INFO:tasks.workunit.client.1.smithi067.stdout:9/679: read d1/d28/d5e/f67 [4623069,125208] 0 2024-04-12T05:37:29.843 INFO:tasks.workunit.client.1.smithi067.stdout:7/730: rmdir d0/d23/d80/da0/dc3 39 2024-04-12T05:37:29.844 INFO:tasks.workunit.client.1.smithi067.stdout:1/722: creat d4/dd2/fe5 x:0 0 0 2024-04-12T05:37:29.847 INFO:tasks.workunit.client.1.smithi067.stdout:2/722: dread d5/da/d3d/f7d [0,4194304] 0 2024-04-12T05:37:29.850 INFO:tasks.workunit.client.1.smithi067.stdout:9/680: dread d1/d28/f18 [0,4194304] 0 2024-04-12T05:37:29.850 INFO:tasks.workunit.client.1.smithi067.stdout:1/723: rename d4/d9/d12/d2c/l6a to d4/d8f/le6 0 2024-04-12T05:37:29.851 INFO:tasks.workunit.client.1.smithi067.stdout:2/723: rmdir d5/d23/d3a 39 2024-04-12T05:37:29.854 INFO:tasks.workunit.client.1.smithi067.stdout:9/681: truncate d1/f79 892580 0 2024-04-12T05:37:29.855 INFO:tasks.workunit.client.1.smithi067.stdout:1/724: chown d4/d3d/dab/f83 154159 1 2024-04-12T05:37:29.856 INFO:tasks.workunit.client.1.smithi067.stdout:1/725: read - d4/d9/d3a/fcb zero size 2024-04-12T05:37:29.856 INFO:tasks.workunit.client.1.smithi067.stdout:2/724: mkdir d5/d43/de8 0 2024-04-12T05:37:29.860 INFO:tasks.workunit.client.1.smithi067.stdout:9/682: unlink d1/d28/d21/d68/ccd 0 2024-04-12T05:37:29.861 INFO:tasks.workunit.client.1.smithi067.stdout:2/725: truncate d5/da/de/d14/d46/d90/fd5 665547 0 2024-04-12T05:37:29.861 INFO:tasks.workunit.client.1.smithi067.stdout:9/683: read - d1/d28/d21/d3f/d4f/f57 zero size 2024-04-12T05:37:29.866 INFO:tasks.workunit.client.1.smithi067.stdout:2/726: creat d5/da/de/d14/d34/d40/d57/da2/fe9 x:0 0 0 2024-04-12T05:37:29.869 INFO:tasks.workunit.client.1.smithi067.stdout:9/684: write d1/f79 [157746,40908] 0 2024-04-12T05:37:29.869 INFO:tasks.workunit.client.1.smithi067.stdout:1/726: dwrite d4/d9/d14/d7a/fc9 [0,4194304] 0 2024-04-12T05:37:29.872 INFO:tasks.workunit.client.1.smithi067.stdout:1/727: fdatasync d4/d9/d14/f8a 0 2024-04-12T05:37:29.875 INFO:tasks.workunit.client.1.smithi067.stdout:2/727: symlink d5/da/de/d14/d34/d40/d57/d96/lea 0 2024-04-12T05:37:29.878 INFO:tasks.workunit.client.1.smithi067.stdout:2/728: creat d5/dbf/feb x:0 0 0 2024-04-12T05:37:29.880 INFO:tasks.workunit.client.1.smithi067.stdout:1/728: creat d4/d9/d3a/d51/d55/fe7 x:0 0 0 2024-04-12T05:37:29.882 INFO:tasks.workunit.client.1.smithi067.stdout:2/729: mknod d5/da/dae/dba/cec 0 2024-04-12T05:37:29.883 INFO:tasks.workunit.client.1.smithi067.stdout:9/685: write d1/d28/f2c [1326578,61086] 0 2024-04-12T05:37:29.891 INFO:tasks.workunit.client.1.smithi067.stdout:9/686: rename d1/d28/d5e/lac to d1/d5f/ld3 0 2024-04-12T05:37:29.891 INFO:tasks.workunit.client.1.smithi067.stdout:9/687: write d1/f6 [5959369,104802] 0 2024-04-12T05:37:29.894 INFO:tasks.workunit.client.1.smithi067.stdout:2/730: dwrite d5/da/de/d14/d34/fb3 [0,4194304] 0 2024-04-12T05:37:29.896 INFO:tasks.workunit.client.1.smithi067.stdout:2/731: dread d5/da/de/d14/f7b [0,4194304] 0 2024-04-12T05:37:29.898 INFO:tasks.workunit.client.1.smithi067.stdout:9/688: dread d1/d13/d34/f70 [0,4194304] 0 2024-04-12T05:37:29.899 INFO:tasks.workunit.client.1.smithi067.stdout:9/689: dread - d1/d28/d44/f72 zero size 2024-04-12T05:37:29.900 INFO:tasks.workunit.client.1.smithi067.stdout:2/732: mknod d5/d23/ced 0 2024-04-12T05:37:29.900 INFO:tasks.workunit.client.1.smithi067.stdout:1/729: dread d4/d9/d14/f8a [0,4194304] 0 2024-04-12T05:37:29.900 INFO:tasks.workunit.client.1.smithi067.stdout:9/690: fdatasync d1/f79 0 2024-04-12T05:37:29.902 INFO:tasks.workunit.client.1.smithi067.stdout:2/733: rmdir d5/d43/d6e 39 2024-04-12T05:37:29.904 INFO:tasks.workunit.client.1.smithi067.stdout:1/730: mkdir d4/d3d/dab/de8 0 2024-04-12T05:37:29.906 INFO:tasks.workunit.client.1.smithi067.stdout:9/691: creat d1/d13/db7/fd4 x:0 0 0 2024-04-12T05:37:29.907 INFO:tasks.workunit.client.1.smithi067.stdout:1/731: symlink d4/d3d/dab/de8/le9 0 2024-04-12T05:37:29.908 INFO:tasks.workunit.client.1.smithi067.stdout:1/732: read - d4/d9/d12/fd8 zero size 2024-04-12T05:37:29.910 INFO:tasks.workunit.client.1.smithi067.stdout:1/733: write d4/d9/d12/d2c/f8d [732767,88001] 0 2024-04-12T05:37:29.911 INFO:tasks.workunit.client.1.smithi067.stdout:9/692: dread d1/f79 [0,4194304] 0 2024-04-12T05:37:29.913 INFO:tasks.workunit.client.1.smithi067.stdout:9/693: symlink d1/d66/ld5 0 2024-04-12T05:37:29.913 INFO:tasks.workunit.client.1.smithi067.stdout:9/694: readlink d1/d12/d15/l17 0 2024-04-12T05:37:29.915 INFO:tasks.workunit.client.1.smithi067.stdout:1/734: write d4/d9/fb [909797,54022] 0 2024-04-12T05:37:29.917 INFO:tasks.workunit.client.1.smithi067.stdout:9/695: rename d1/d28/d21/d3f/lbd to d1/d28/d21/d68/ld6 0 2024-04-12T05:37:29.918 INFO:tasks.workunit.client.1.smithi067.stdout:9/696: fdatasync d1/d28/d21/d45/d75/f80 0 2024-04-12T05:37:29.918 INFO:tasks.workunit.client.1.smithi067.stdout:1/735: rmdir d4/d9/d3a/d51/d55/d9a 39 2024-04-12T05:37:29.919 INFO:tasks.workunit.client.1.smithi067.stdout:9/697: rmdir d1/d28/d27/d32 39 2024-04-12T05:37:29.921 INFO:tasks.workunit.client.1.smithi067.stdout:9/698: creat d1/d66/d8e/fd7 x:0 0 0 2024-04-12T05:37:29.923 INFO:tasks.workunit.client.1.smithi067.stdout:9/699: write d1/f6 [702242,107641] 0 2024-04-12T05:37:29.960 INFO:tasks.workunit.client.1.smithi067.stdout:0/735: sync 2024-04-12T05:37:29.962 INFO:tasks.workunit.client.1.smithi067.stdout:1/736: dwrite d4/d9/d12/f81 [0,4194304] 0 2024-04-12T05:37:29.962 INFO:tasks.workunit.client.1.smithi067.stdout:9/700: symlink d1/d5f/ld8 0 2024-04-12T05:37:29.966 INFO:tasks.workunit.client.1.smithi067.stdout:9/701: unlink d1/d66/d8e/fd7 0 2024-04-12T05:37:29.968 INFO:tasks.workunit.client.1.smithi067.stdout:9/702: readlink d1/d28/d21/l54 0 2024-04-12T05:37:29.969 INFO:tasks.workunit.client.1.smithi067.stdout:0/736: dwrite f1 [0,4194304] 0 2024-04-12T05:37:29.975 INFO:tasks.workunit.client.1.smithi067.stdout:9/703: dread d1/d12/d15/f1d [4194304,4194304] 0 2024-04-12T05:37:29.979 INFO:tasks.workunit.client.1.smithi067.stdout:0/737: write dd/d12/f16 [2075925,89233] 0 2024-04-12T05:37:29.981 INFO:tasks.workunit.client.1.smithi067.stdout:0/738: write dd/d12/f2e [3859986,106857] 0 2024-04-12T05:37:29.981 INFO:tasks.workunit.client.1.smithi067.stdout:9/704: symlink d1/d28/d27/ld9 0 2024-04-12T05:37:29.984 INFO:tasks.workunit.client.1.smithi067.stdout:9/705: fdatasync d1/d13/db7/fc4 0 2024-04-12T05:37:29.986 INFO:tasks.workunit.client.1.smithi067.stdout:1/737: dread d4/d44/f66 [0,4194304] 0 2024-04-12T05:37:29.988 INFO:tasks.workunit.client.1.smithi067.stdout:0/739: getdents dd/d9c/d5f/d78/d83 0 2024-04-12T05:37:29.989 INFO:tasks.workunit.client.1.smithi067.stdout:1/738: write d4/d9/d14/f57 [434242,101572] 0 2024-04-12T05:37:29.990 INFO:tasks.workunit.client.1.smithi067.stdout:0/740: write dd/d12/d14/d51/fb4 [3485383,26458] 0 2024-04-12T05:37:29.992 INFO:tasks.workunit.client.1.smithi067.stdout:9/706: dread d1/fd [4194304,4194304] 0 2024-04-12T05:37:29.995 INFO:tasks.workunit.client.1.smithi067.stdout:1/739: write d4/d9/f22 [1113518,86220] 0 2024-04-12T05:37:29.996 INFO:tasks.workunit.client.1.smithi067.stdout:9/707: truncate d1/d28/d21/d45/d75/db3/f90 749027 0 2024-04-12T05:37:29.996 INFO:tasks.workunit.client.1.smithi067.stdout:0/741: symlink dd/d12/d14/d1a/d3d/le8 0 2024-04-12T05:37:29.998 INFO:tasks.workunit.client.1.smithi067.stdout:1/740: getdents d4/d3d/d7b/dde 0 2024-04-12T05:37:29.999 INFO:tasks.workunit.client.1.smithi067.stdout:1/741: readlink d4/d9/d12/d2c/l7c 0 2024-04-12T05:37:30.000 INFO:tasks.workunit.client.1.smithi067.stdout:0/742: creat dd/d9c/d64/ddd/fe9 x:0 0 0 2024-04-12T05:37:30.002 INFO:tasks.workunit.client.1.smithi067.stdout:0/743: dread - dd/d12/d14/d49/fd5 zero size 2024-04-12T05:37:30.002 INFO:tasks.workunit.client.1.smithi067.stdout:1/742: unlink d4/d3d/fb8 0 2024-04-12T05:37:30.002 INFO:tasks.workunit.client.1.smithi067.stdout:9/708: link d1/d12/d15/cd0 d1/d13/d34/cda 0 2024-04-12T05:37:30.004 INFO:tasks.workunit.client.1.smithi067.stdout:1/743: mkdir d4/d8f/dea 0 2024-04-12T05:37:30.004 INFO:tasks.workunit.client.1.smithi067.stdout:9/709: mknod d1/d12/cdb 0 2024-04-12T05:37:30.006 INFO:tasks.workunit.client.1.smithi067.stdout:0/744: link dd/fd1 dd/dad/dd9/fea 0 2024-04-12T05:37:30.006 INFO:tasks.workunit.client.1.smithi067.stdout:9/710: rmdir d1 39 2024-04-12T05:37:30.007 INFO:tasks.workunit.client.1.smithi067.stdout:0/745: chown dd/d12/d14/d49/f4d 211360 1 2024-04-12T05:37:30.007 INFO:tasks.workunit.client.1.smithi067.stdout:1/744: creat d4/d8f/dea/feb x:0 0 0 2024-04-12T05:37:30.008 INFO:tasks.workunit.client.1.smithi067.stdout:0/746: write dd/d6f/fba [300993,34569] 0 2024-04-12T05:37:30.012 INFO:tasks.workunit.client.1.smithi067.stdout:0/747: symlink dd/d9c/d5f/d78/leb 0 2024-04-12T05:37:30.016 INFO:tasks.workunit.client.1.smithi067.stdout:9/711: dwrite d1/d28/d21/d3f/d4f/f60 [0,4194304] 0 2024-04-12T05:37:30.017 INFO:tasks.workunit.client.1.smithi067.stdout:0/748: rename dd/d9c/d64/cbc to dd/dbd/dbe/cec 0 2024-04-12T05:37:30.018 INFO:tasks.workunit.client.1.smithi067.stdout:9/712: rmdir d1/d12 39 2024-04-12T05:37:30.018 INFO:tasks.workunit.client.1.smithi067.stdout:0/749: stat dd/dad/dd9 0 2024-04-12T05:37:30.019 INFO:tasks.workunit.client.1.smithi067.stdout:9/713: chown d1/d28/d21/d45/d75/db3/f90 4742 1 2024-04-12T05:37:30.020 INFO:tasks.workunit.client.1.smithi067.stdout:0/750: readlink dd/d92/l66 0 2024-04-12T05:37:30.023 INFO:tasks.workunit.client.1.smithi067.stdout:0/751: mknod dd/d6f/d72/ced 0 2024-04-12T05:37:30.025 INFO:tasks.workunit.client.1.smithi067.stdout:0/752: chown dd/d12/d14/l96 122 1 2024-04-12T05:37:30.029 INFO:tasks.workunit.client.1.smithi067.stdout:0/753: rename dd/d12/d14/d1a/f4b to dd/d12/d14/d51/da2/fee 0 2024-04-12T05:37:30.034 INFO:tasks.workunit.client.1.smithi067.stdout:1/745: dread d4/d9/d12/d59/f64 [0,4194304] 0 2024-04-12T05:37:30.037 INFO:tasks.workunit.client.1.smithi067.stdout:1/746: rename d4/d3d/fae to d4/d3d/dab/fec 0 2024-04-12T05:37:30.040 INFO:tasks.workunit.client.1.smithi067.stdout:1/747: write d4/d8f/fbb [4652838,28749] 0 2024-04-12T05:37:30.042 INFO:tasks.workunit.client.1.smithi067.stdout:1/748: write d4/d9/d3a/d51/d55/dad/fb6 [20372,66669] 0 2024-04-12T05:37:30.044 INFO:tasks.workunit.client.1.smithi067.stdout:0/754: truncate dd/d6f/d72/fa3 1269556 0 2024-04-12T05:37:30.070 INFO:tasks.workunit.client.1.smithi067.stdout:0/755: dwrite dd/d6f/d72/fa3 [0,4194304] 0 2024-04-12T05:37:30.090 INFO:tasks.workunit.client.1.smithi067.stdout:0/756: dwrite dd/d12/d14/d51/da2/fee [0,4194304] 0 2024-04-12T05:37:30.093 INFO:tasks.workunit.client.1.smithi067.stdout:0/757: truncate dd/d12/d14/d1a/fca 223823 0 2024-04-12T05:37:30.105 INFO:tasks.workunit.client.1.smithi067.stdout:0/758: write dd/dbd/fcb [426806,26445] 0 2024-04-12T05:37:30.108 INFO:tasks.workunit.client.1.smithi067.stdout:0/759: unlink dd/d9c/d64/c88 0 2024-04-12T05:37:30.109 INFO:tasks.workunit.client.1.smithi067.stdout:0/760: chown dd/d6f/d72/c8f 236401271 1 2024-04-12T05:37:30.122 INFO:tasks.workunit.client.1.smithi067.stdout:0/761: dread dd/d12/d14/d49/dab/fcf [0,4194304] 0 2024-04-12T05:37:30.124 INFO:tasks.workunit.client.1.smithi067.stdout:0/762: symlink dd/d9c/d64/ddd/lef 0 2024-04-12T05:37:30.125 INFO:tasks.workunit.client.1.smithi067.stdout:0/763: write dd/d9c/d64/f7e [2060352,66173] 0 2024-04-12T05:37:30.127 INFO:tasks.workunit.client.1.smithi067.stdout:0/764: read dd/d12/d14/d49/f6d [701418,20502] 0 2024-04-12T05:37:30.131 INFO:tasks.workunit.client.1.smithi067.stdout:0/765: rename dd/d9c/d5f/cac to dd/d9c/d5f/cf0 0 2024-04-12T05:37:30.131 INFO:tasks.workunit.client.1.smithi067.stdout:0/766: fsync dd/d12/d14/fc9 0 2024-04-12T05:37:30.134 INFO:tasks.workunit.client.1.smithi067.stdout:0/767: creat dd/d12/d14/ff1 x:0 0 0 2024-04-12T05:37:30.144 INFO:tasks.workunit.client.1.smithi067.stdout:0/768: dwrite dd/d12/d14/d1a/f6c [0,4194304] 0 2024-04-12T05:37:30.211 INFO:tasks.workunit.client.1.smithi067.stdout:0/769: truncate dd/d9c/f62 1413003 0 2024-04-12T05:37:30.211 INFO:tasks.workunit.client.1.smithi067.stdout:0/770: truncate dd/d12/f2a 812768 0 2024-04-12T05:37:30.211 INFO:tasks.workunit.client.1.smithi067.stdout:0/771: truncate dd/d12/f16 2472370 0 2024-04-12T05:37:30.211 INFO:tasks.workunit.client.1.smithi067.stdout:0/772: mkdir dd/d12/d14/d1a/d3d/df2 0 2024-04-12T05:37:30.212 INFO:tasks.workunit.client.1.smithi067.stdout:0/773: write dd/d12/d14/d1a/f93 [383779,50430] 0 2024-04-12T05:37:30.212 INFO:tasks.workunit.client.1.smithi067.stdout:0/774: creat dd/d12/d14/ff3 x:0 0 0 2024-04-12T05:37:30.212 INFO:tasks.workunit.client.1.smithi067.stdout:0/775: symlink dd/d12/d14/de3/lf4 0 2024-04-12T05:37:30.212 INFO:tasks.workunit.client.1.smithi067.stdout:0/776: read dd/d6f/f98 [2757853,81391] 0 2024-04-12T05:37:30.212 INFO:tasks.workunit.client.1.smithi067.stdout:0/777: creat dd/d9c/d5f/de0/ff5 x:0 0 0 2024-04-12T05:37:30.212 INFO:tasks.workunit.client.1.smithi067.stdout:0/778: dwrite dd/d12/d14/ff1 [0,4194304] 0 2024-04-12T05:37:30.212 INFO:tasks.workunit.client.1.smithi067.stdout:0/779: write dd/d12/d14/f61 [3901295,19640] 0 2024-04-12T05:37:30.212 INFO:tasks.workunit.client.1.smithi067.stdout:0/780: creat dd/d9c/d5f/de0/ff6 x:0 0 0 2024-04-12T05:37:30.212 INFO:tasks.workunit.client.1.smithi067.stdout:0/781: chown dd/d12/d14/fc9 193 1 2024-04-12T05:37:30.212 INFO:tasks.workunit.client.1.smithi067.stdout:0/782: getdents dd/d6f 0 2024-04-12T05:37:30.212 INFO:tasks.workunit.client.1.smithi067.stdout:0/783: write dd/fd4 [101113,54095] 0 2024-04-12T05:37:30.213 INFO:tasks.workunit.client.1.smithi067.stdout:8/579: sync 2024-04-12T05:37:30.215 INFO:tasks.workunit.client.1.smithi067.stdout:8/580: creat d6/d8/d18/d63/d67/fad x:0 0 0 2024-04-12T05:37:30.219 INFO:tasks.workunit.client.1.smithi067.stdout:8/581: link d6/d8/d18/f62 d6/d8/d8b/fae 0 2024-04-12T05:37:30.224 INFO:tasks.workunit.client.1.smithi067.stdout:8/582: rename d6/d8/d18/d63/d67/d69/d5d/l64 to d6/d8/d18/d63/d67/laf 0 2024-04-12T05:37:30.228 INFO:tasks.workunit.client.1.smithi067.stdout:8/583: link d6/d8/d18/d63/d9a/d12/fa9 d6/fb0 0 2024-04-12T05:37:30.230 INFO:tasks.workunit.client.1.smithi067.stdout:8/584: fdatasync d6/d8/d6e/f84 0 2024-04-12T05:37:30.233 INFO:tasks.workunit.client.1.smithi067.stdout:8/585: symlink d6/d8/d18/d4e/d9e/lb1 0 2024-04-12T05:37:30.239 INFO:tasks.workunit.client.1.smithi067.stdout:8/586: dwrite d6/d8/d18/d63/d9a/f1c [4194304,4194304] 0 2024-04-12T05:37:30.240 INFO:tasks.workunit.client.1.smithi067.stdout:8/587: write d6/d86/f99 [13604,73680] 0 2024-04-12T05:37:30.247 INFO:tasks.workunit.client.1.smithi067.stdout:8/588: getdents d6/d8/d18/d63/d9a/d12 0 2024-04-12T05:37:30.249 INFO:tasks.workunit.client.1.smithi067.stdout:8/589: readlink d6/d8/d18/d63/d9a/l7e 0 2024-04-12T05:37:30.251 INFO:tasks.workunit.client.1.smithi067.stdout:8/590: mknod d6/d8/d18/d63/d9a/d12/cb2 0 2024-04-12T05:37:30.252 INFO:tasks.workunit.client.1.smithi067.stdout:8/591: fdatasync d6/d86/d4f/d82/f9d 0 2024-04-12T05:37:30.256 INFO:tasks.workunit.client.1.smithi067.stdout:8/592: dread d6/d8/d18/d36/f7f [0,4194304] 0 2024-04-12T05:37:30.256 INFO:tasks.workunit.client.1.smithi067.stdout:8/593: read - d6/d86/fa4 zero size 2024-04-12T05:37:30.261 INFO:tasks.workunit.client.1.smithi067.stdout:8/594: rename d6/d8/d8b/f98 to d6/d8/d18/d63/d9a/fb3 0 2024-04-12T05:37:30.264 INFO:tasks.workunit.client.1.smithi067.stdout:8/595: getdents d6/d8/d18/d63 0 2024-04-12T05:37:30.271 INFO:tasks.workunit.client.1.smithi067.stdout:8/596: dwrite d6/d86/d4f/d82/f9d [0,4194304] 0 2024-04-12T05:37:30.277 INFO:tasks.workunit.client.1.smithi067.stdout:8/597: truncate d6/d8/d18/d36/f70 159405 0 2024-04-12T05:37:30.279 INFO:tasks.workunit.client.1.smithi067.stdout:8/598: dread d6/d8/d18/d36/f7f [0,4194304] 0 2024-04-12T05:37:30.281 INFO:tasks.workunit.client.1.smithi067.stdout:8/599: write d6/d86/d4f/d82/f9d [1998950,39761] 0 2024-04-12T05:37:30.292 INFO:tasks.workunit.client.1.smithi067.stdout:8/600: dwrite d6/d8/d18/d36/f70 [0,4194304] 0 2024-04-12T05:37:30.295 INFO:tasks.workunit.client.1.smithi067.stdout:8/601: unlink d6/d8/d18/d63/d9a/d12/cb2 0 2024-04-12T05:37:30.302 INFO:tasks.workunit.client.1.smithi067.stdout:8/602: rename d6/d8/d18/d63/d9a/d12/f55 to d6/d8/d18/fb4 0 2024-04-12T05:37:30.303 INFO:tasks.workunit.client.1.smithi067.stdout:8/603: truncate d6/d8/d18/d63/d67/fad 560282 0 2024-04-12T05:37:30.310 INFO:tasks.workunit.client.1.smithi067.stdout:8/604: dread d6/d8/d18/f3f [0,4194304] 0 2024-04-12T05:37:30.312 INFO:tasks.workunit.client.1.smithi067.stdout:8/605: mknod d6/d8/d50/cb5 0 2024-04-12T05:37:30.313 INFO:tasks.workunit.client.1.smithi067.stdout:8/606: fdatasync d6/d86/fa4 0 2024-04-12T05:37:30.320 INFO:tasks.workunit.client.1.smithi067.stdout:8/607: dwrite d6/f95 [0,4194304] 0 2024-04-12T05:37:30.321 INFO:tasks.workunit.client.1.smithi067.stdout:8/608: chown d6/d86/d4f/c6b 30739 1 2024-04-12T05:37:30.328 INFO:tasks.workunit.client.1.smithi067.stdout:8/609: dwrite d6/d8/f92 [0,4194304] 0 2024-04-12T05:37:30.329 INFO:tasks.workunit.client.1.smithi067.stdout:8/610: stat d6/d8/d18/d63/d67/d69 0 2024-04-12T05:37:30.348 INFO:tasks.workunit.client.1.smithi067.stdout:8/611: dwrite d6/d8/d18/d36/f68 [0,4194304] 0 2024-04-12T05:37:30.352 INFO:tasks.workunit.client.1.smithi067.stdout:8/612: dread d6/d8/d18/d63/d67/d69/d5d/f91 [0,4194304] 0 2024-04-12T05:37:30.353 INFO:tasks.workunit.client.1.smithi067.stdout:8/613: truncate d6/d8/d18/d36/f7f 1286476 0 2024-04-12T05:37:30.354 INFO:tasks.workunit.client.1.smithi067.stdout:8/614: write d6/d8/d18/d63/d9a/f1c [1502918,60704] 0 2024-04-12T05:37:30.380 INFO:tasks.workunit.client.1.smithi067.stdout:8/615: dwrite d6/d8/d18/f35 [0,4194304] 0 2024-04-12T05:37:30.383 INFO:tasks.workunit.client.1.smithi067.stdout:8/616: unlink d6/d86/fa4 0 2024-04-12T05:37:30.383 INFO:tasks.workunit.client.1.smithi067.stdout:8/617: chown d6/d8/d8b 176 1 2024-04-12T05:37:30.386 INFO:tasks.workunit.client.1.smithi067.stdout:3/567: sync 2024-04-12T05:37:30.386 INFO:tasks.workunit.client.1.smithi067.stdout:6/666: sync 2024-04-12T05:37:30.386 INFO:tasks.workunit.client.1.smithi067.stdout:5/978: sync 2024-04-12T05:37:30.387 INFO:tasks.workunit.client.1.smithi067.stdout:8/618: dread d6/d8/d18/d63/d9a/f90 [0,4194304] 0 2024-04-12T05:37:30.388 INFO:tasks.workunit.client.1.smithi067.stdout:6/667: chown dd/d11/c31 58916268 1 2024-04-12T05:37:30.388 INFO:tasks.workunit.client.1.smithi067.stdout:6/668: chown dd/f20 132837895 1 2024-04-12T05:37:30.391 INFO:tasks.workunit.client.1.smithi067.stdout:6/669: mknod dd/d21/d3f/cde 0 2024-04-12T05:37:30.391 INFO:tasks.workunit.client.1.smithi067.stdout:3/568: read dc/df/f1c [4107995,88776] 0 2024-04-12T05:37:30.392 INFO:tasks.workunit.client.1.smithi067.stdout:5/979: rename d0/d6/d20/fbe to d0/d6/d20/d62/f145 0 2024-04-12T05:37:30.393 INFO:tasks.workunit.client.1.smithi067.stdout:6/670: symlink dd/d21/d64/d88/d7b/dd2/ddd/ldf 0 2024-04-12T05:37:30.393 INFO:tasks.workunit.client.1.smithi067.stdout:5/980: chown d0/d6/d40/d4b/l67 0 1 2024-04-12T05:37:30.403 INFO:tasks.workunit.client.1.smithi067.stdout:5/981: dwrite d0/d6/d20/d62/f15 [0,4194304] 0 2024-04-12T05:37:30.407 INFO:tasks.workunit.client.1.smithi067.stdout:6/671: dread dd/f89 [0,4194304] 0 2024-04-12T05:37:30.409 INFO:tasks.workunit.client.1.smithi067.stdout:6/672: chown dd/d21/la2 116608 1 2024-04-12T05:37:30.410 INFO:tasks.workunit.client.1.smithi067.stdout:5/982: truncate d0/d3a/f83 2537240 0 2024-04-12T05:37:30.411 INFO:tasks.workunit.client.1.smithi067.stdout:6/673: truncate dd/d21/d64/db5/fc9 775243 0 2024-04-12T05:37:30.412 INFO:tasks.workunit.client.1.smithi067.stdout:6/674: truncate dd/d11/fb7 290090 0 2024-04-12T05:37:30.413 INFO:tasks.workunit.client.1.smithi067.stdout:5/983: creat d0/d6/d20/d62/d13f/f146 x:0 0 0 2024-04-12T05:37:30.414 INFO:tasks.workunit.client.1.smithi067.stdout:6/675: write dd/d11/d27/f46 [846165,65653] 0 2024-04-12T05:37:30.418 INFO:tasks.workunit.client.1.smithi067.stdout:6/676: dread dd/d21/d64/db5/fc9 [0,4194304] 0 2024-04-12T05:37:30.419 INFO:tasks.workunit.client.1.smithi067.stdout:5/984: getdents d0/d6/d20/d3e 0 2024-04-12T05:37:30.420 INFO:tasks.workunit.client.1.smithi067.stdout:6/677: readlink dd/d21/d3f/l94 0 2024-04-12T05:37:30.422 INFO:tasks.workunit.client.1.smithi067.stdout:5/985: symlink d0/d6/d40/d4b/d38/l147 0 2024-04-12T05:37:30.422 INFO:tasks.workunit.client.1.smithi067.stdout:6/678: creat dd/d21/d3f/dd5/fe0 x:0 0 0 2024-04-12T05:37:30.425 INFO:tasks.workunit.client.1.smithi067.stdout:6/679: rmdir dd/d11/d27/d4a/d8b 39 2024-04-12T05:37:30.430 INFO:tasks.workunit.client.1.smithi067.stdout:5/986: dwrite d0/d6/d40/d4b/d38/d8e/f11c [0,4194304] 0 2024-04-12T05:37:30.430 INFO:tasks.workunit.client.1.smithi067.stdout:6/680: dread dd/f50 [0,4194304] 0 2024-04-12T05:37:30.434 INFO:tasks.workunit.client.1.smithi067.stdout:6/681: mknod dd/d21/d64/d88/d7b/dcc/ce1 0 2024-04-12T05:37:30.436 INFO:tasks.workunit.client.1.smithi067.stdout:5/987: dread d0/d6/d40/d4b/d38/d8e/d8f/f132 [0,4194304] 0 2024-04-12T05:37:30.436 INFO:tasks.workunit.client.1.smithi067.stdout:6/682: chown dd/d21/d64/d88/d7b/dd2/lb2 0 1 2024-04-12T05:37:30.444 INFO:tasks.workunit.client.1.smithi067.stdout:6/683: dwrite dd/d11/d27/f46 [0,4194304] 0 2024-04-12T05:37:30.451 INFO:tasks.workunit.client.1.smithi067.stdout:6/684: getdents dd 0 2024-04-12T05:37:30.456 INFO:tasks.workunit.client.1.smithi067.stdout:5/988: dwrite d0/d6/d40/d4b/d38/d8e/d8f/db9/dbd/fd9 [0,4194304] 0 2024-04-12T05:37:30.463 INFO:tasks.workunit.client.1.smithi067.stdout:6/685: link dd/d11/d27/d4a/d79/dc3/d56/c7c dd/d11/d27/d4a/d79/ce2 0 2024-04-12T05:37:30.468 INFO:tasks.workunit.client.1.smithi067.stdout:6/686: mkdir dd/d11/d27/d4a/d79/dc3/d56/de3 0 2024-04-12T05:37:30.473 INFO:tasks.workunit.client.1.smithi067.stdout:5/989: dread d0/f2 [0,4194304] 0 2024-04-12T05:37:30.474 INFO:tasks.workunit.client.1.smithi067.stdout:5/990: chown d0/d3a/d114/c118 287253607 1 2024-04-12T05:37:30.478 INFO:tasks.workunit.client.1.smithi067.stdout:5/991: creat d0/d6/d40/d4b/d38/d8e/d8f/db9/dbd/f148 x:0 0 0 2024-04-12T05:37:30.484 INFO:tasks.workunit.client.1.smithi067.stdout:6/687: dread dd/d21/d3f/f40 [0,4194304] 0 2024-04-12T05:37:30.487 INFO:tasks.workunit.client.1.smithi067.stdout:5/992: dwrite d0/d6/d20/d3e/f8a [0,4194304] 0 2024-04-12T05:37:30.488 INFO:tasks.workunit.client.1.smithi067.stdout:5/993: write d0/da/f2e [1393322,3712] 0 2024-04-12T05:37:30.488 INFO:tasks.workunit.client.1.smithi067.stdout:6/688: symlink dd/d21/d64/d6f/d90/le4 0 2024-04-12T05:37:30.494 INFO:tasks.workunit.client.1.smithi067.stdout:5/994: creat d0/d6/d20/d3e/dbf/d115/d82/d13e/f149 x:0 0 0 2024-04-12T05:37:30.496 INFO:tasks.workunit.client.1.smithi067.stdout:6/689: rename dd/d21/d64/d88/d7b/dd2/ddd to dd/d11/d27/d4a/d79/dc3/de5 0 2024-04-12T05:37:30.507 INFO:tasks.workunit.client.1.smithi067.stdout:6/690: dwrite dd/d21/d64/d6f/da7/fb8 [0,4194304] 0 2024-04-12T05:37:30.508 INFO:tasks.workunit.client.1.smithi067.stdout:6/691: truncate dd/d21/d3f/dd5/fe0 570070 0 2024-04-12T05:37:30.509 INFO:tasks.workunit.client.1.smithi067.stdout:6/692: write dd/f20 [1238469,37635] 0 2024-04-12T05:37:30.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:30 smithi067 ceph-mon[49010]: pgmap v276: 65 pgs: 65 active+clean; 3.2 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 80 MiB/s rd, 209 MiB/s wr, 458 op/s 2024-04-12T05:37:30.520 INFO:tasks.workunit.client.1.smithi067.stdout:6/693: truncate dd/d11/d27/d4a/d79/dc3/f3c 1911072 0 2024-04-12T05:37:30.520 INFO:tasks.workunit.client.1.smithi067.stdout:6/694: chown dd/f89 1 1 2024-04-12T05:37:30.522 INFO:tasks.workunit.client.1.smithi067.stdout:6/695: fdatasync dd/d11/d27/f9a 0 2024-04-12T05:37:30.525 INFO:tasks.workunit.client.1.smithi067.stdout:6/696: creat dd/d21/d3f/dd5/fe6 x:0 0 0 2024-04-12T05:37:30.528 INFO:tasks.workunit.client.1.smithi067.stdout:6/697: creat dd/d11/d27/d4a/d79/dc3/d56/de3/fe7 x:0 0 0 2024-04-12T05:37:30.531 INFO:tasks.workunit.client.1.smithi067.stdout:6/698: mknod dd/d9f/ce8 0 2024-04-12T05:37:30.535 INFO:tasks.workunit.client.1.smithi067.stdout:6/699: link dd/d21/d3f/l52 dd/le9 0 2024-04-12T05:37:30.542 INFO:tasks.workunit.client.1.smithi067.stdout:6/700: mkdir dd/d21/d64/d6f/dea 0 2024-04-12T05:37:30.544 INFO:tasks.workunit.client.1.smithi067.stdout:6/701: creat dd/d21/feb x:0 0 0 2024-04-12T05:37:30.547 INFO:tasks.workunit.client.1.smithi067.stdout:6/702: rmdir dd/d11/d27/d4a/d8b 39 2024-04-12T05:37:30.549 INFO:tasks.workunit.client.1.smithi067.stdout:6/703: dread dd/d21/d64/d6f/d90/f9d [0,4194304] 0 2024-04-12T05:37:30.561 INFO:tasks.workunit.client.1.smithi067.stdout:6/704: dread dd/d11/d27/d4a/d79/dc3/f65 [0,4194304] 0 2024-04-12T05:37:30.563 INFO:tasks.workunit.client.1.smithi067.stdout:6/705: read dd/d21/f24 [3536283,84216] 0 2024-04-12T05:37:30.569 INFO:tasks.workunit.client.1.smithi067.stdout:2/734: sync 2024-04-12T05:37:30.569 INFO:tasks.workunit.client.1.smithi067.stdout:7/731: sync 2024-04-12T05:37:30.569 INFO:tasks.workunit.client.1.smithi067.stdout:4/620: sync 2024-04-12T05:37:30.569 INFO:tasks.workunit.client.1.smithi067.stdout:9/714: sync 2024-04-12T05:37:30.571 INFO:tasks.workunit.client.1.smithi067.stdout:6/706: dwrite dd/d21/d64/d88/d7b/dcc/d93/fd8 [0,4194304] 0 2024-04-12T05:37:30.572 INFO:tasks.workunit.client.1.smithi067.stdout:6/707: readlink l1 0 2024-04-12T05:37:30.575 INFO:tasks.workunit.client.1.smithi067.stdout:6/708: chown dd/d21/d64/d6f/d90/d98/fb9 488 1 2024-04-12T05:37:30.576 INFO:tasks.workunit.client.1.smithi067.stdout:9/715: creat d1/d28/d27/fdc x:0 0 0 2024-04-12T05:37:30.577 INFO:tasks.workunit.client.1.smithi067.stdout:4/621: creat d8/dba/d5d/d66/d71/fcd x:0 0 0 2024-04-12T05:37:30.582 INFO:tasks.workunit.client.1.smithi067.stdout:6/709: rmdir dd/d21/d64/d6f/d90/d98 39 2024-04-12T05:37:30.583 INFO:tasks.workunit.client.1.smithi067.stdout:4/622: write d8/da/d1d/f49 [3966702,32543] 0 2024-04-12T05:37:30.585 INFO:tasks.workunit.client.1.smithi067.stdout:7/732: dwrite d0/d23/d80/da0/db4/f52 [0,4194304] 0 2024-04-12T05:37:30.586 INFO:tasks.workunit.client.1.smithi067.stdout:2/735: dwrite d5/d6a/fa6 [0,4194304] 0 2024-04-12T05:37:30.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:30 smithi032 ceph-mon[39731]: pgmap v276: 65 pgs: 65 active+clean; 3.2 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 80 MiB/s rd, 209 MiB/s wr, 458 op/s 2024-04-12T05:37:30.595 INFO:tasks.workunit.client.1.smithi067.stdout:4/623: write d8/dba/d5d/d66/d71/fcd [375271,2249] 0 2024-04-12T05:37:30.597 INFO:tasks.workunit.client.1.smithi067.stdout:7/733: mkdir d0/d23/d80/da0/dc3/df7/dfc 0 2024-04-12T05:37:30.598 INFO:tasks.workunit.client.1.smithi067.stdout:7/734: stat d0/f8 0 2024-04-12T05:37:30.599 INFO:tasks.workunit.client.1.smithi067.stdout:9/716: dwrite d1/d28/d21/f48 [0,4194304] 0 2024-04-12T05:37:30.601 INFO:tasks.workunit.client.1.smithi067.stdout:2/736: mknod d5/da/de/dc4/dc6/cee 0 2024-04-12T05:37:30.605 INFO:tasks.workunit.client.1.smithi067.stdout:6/710: dwrite dd/d21/d64/d88/d7b/f6e [0,4194304] 0 2024-04-12T05:37:30.607 INFO:tasks.workunit.client.1.smithi067.stdout:2/737: chown d5/da/de/d14/d34/d4c/f6b 14 1 2024-04-12T05:37:30.608 INFO:tasks.workunit.client.1.smithi067.stdout:9/717: mknod d1/cdd 0 2024-04-12T05:37:30.609 INFO:tasks.workunit.client.1.smithi067.stdout:7/735: write d0/d23/d80/da0/db4/dd/d42/d7e/f1f [1622399,5116] 0 2024-04-12T05:37:30.610 INFO:tasks.workunit.client.1.smithi067.stdout:9/718: write d1/d28/d21/d3f/f41 [412116,37858] 0 2024-04-12T05:37:30.612 INFO:tasks.workunit.client.1.smithi067.stdout:9/719: write d1/d28/d21/d45/d75/db3/f90 [1467912,13909] 0 2024-04-12T05:37:30.614 INFO:tasks.workunit.client.1.smithi067.stdout:2/738: fdatasync d5/d11/f9b 0 2024-04-12T05:37:30.617 INFO:tasks.workunit.client.1.smithi067.stdout:4/624: link d8/dba/d5d/d66/d7b/d6d/f81 d8/da/d1d/d15/d58/fce 0 2024-04-12T05:37:30.618 INFO:tasks.workunit.client.1.smithi067.stdout:7/736: symlink d0/lfd 0 2024-04-12T05:37:30.618 INFO:tasks.workunit.client.1.smithi067.stdout:9/720: chown d1/d12/cc5 12302194 1 2024-04-12T05:37:30.621 INFO:tasks.workunit.client.1.smithi067.stdout:7/737: fdatasync d0/d23/d80/da0/db4/dd/d42/d7e/f94 0 2024-04-12T05:37:30.623 INFO:tasks.workunit.client.1.smithi067.stdout:7/738: chown d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/dd6 14245183 1 2024-04-12T05:37:30.624 INFO:tasks.workunit.client.1.smithi067.stdout:9/721: symlink d1/d28/d21/d3f/d4f/lde 0 2024-04-12T05:37:30.625 INFO:tasks.workunit.client.1.smithi067.stdout:4/625: mkdir d8/da/d9f/dcf 0 2024-04-12T05:37:30.626 INFO:tasks.workunit.client.1.smithi067.stdout:6/711: dread dd/d11/f91 [0,4194304] 0 2024-04-12T05:37:30.627 INFO:tasks.workunit.client.1.smithi067.stdout:7/739: mknod d0/d23/d80/da0/db4/dd/d79/cfe 0 2024-04-12T05:37:30.628 INFO:tasks.workunit.client.1.smithi067.stdout:2/739: truncate d5/da/d3d/f6d 669484 0 2024-04-12T05:37:30.630 INFO:tasks.workunit.client.1.smithi067.stdout:4/626: symlink d8/dba/db8/ld0 0 2024-04-12T05:37:30.631 INFO:tasks.workunit.client.1.smithi067.stdout:6/712: symlink dd/d21/d64/d88/d7b/dcc/lec 0 2024-04-12T05:37:30.632 INFO:tasks.workunit.client.1.smithi067.stdout:2/740: dread - d5/da/d3d/f8e zero size 2024-04-12T05:37:30.635 INFO:tasks.workunit.client.1.smithi067.stdout:9/722: dwrite d1/d28/d5e/f6f [0,4194304] 0 2024-04-12T05:37:30.637 INFO:tasks.workunit.client.1.smithi067.stdout:6/713: chown dd/d21/d64/d6f/d90/d98 201378784 1 2024-04-12T05:37:30.640 INFO:tasks.workunit.client.1.smithi067.stdout:6/714: fsync dd/d21/d3f/dd5/fe6 0 2024-04-12T05:37:30.641 INFO:tasks.workunit.client.1.smithi067.stdout:2/741: mkdir d5/da/de/dc4/def 0 2024-04-12T05:37:30.647 INFO:tasks.workunit.client.1.smithi067.stdout:6/715: symlink dd/d11/d27/d4a/dc7/led 0 2024-04-12T05:37:30.649 INFO:tasks.workunit.client.1.smithi067.stdout:7/740: dwrite d0/d23/d80/da0/db4/d5/d1d/d43/fb9 [4194304,4194304] 0 2024-04-12T05:37:30.650 INFO:tasks.workunit.client.1.smithi067.stdout:4/627: dwrite d8/da/d9f/d68/dbf/f97 [0,4194304] 0 2024-04-12T05:37:30.650 INFO:tasks.workunit.client.1.smithi067.stdout:6/716: read - dd/d21/fcb zero size 2024-04-12T05:37:30.650 INFO:tasks.workunit.client.1.smithi067.stdout:2/742: rename d5/da/de/d39/f6c to d5/da/de/dc4/def/ff0 0 2024-04-12T05:37:30.655 INFO:tasks.workunit.client.1.smithi067.stdout:4/628: rmdir d8/da/d1d/d15 39 2024-04-12T05:37:30.658 INFO:tasks.workunit.client.1.smithi067.stdout:6/717: rename dd/d21/d64/d88/d7b/dcc/lec to dd/d21/d64/d6f/dea/lee 0 2024-04-12T05:37:30.662 INFO:tasks.workunit.client.1.smithi067.stdout:4/629: truncate d8/d39/f79 1560674 0 2024-04-12T05:37:30.665 INFO:tasks.workunit.client.1.smithi067.stdout:6/718: dread dd/d21/d64/d6f/da7/fb8 [0,4194304] 0 2024-04-12T05:37:30.666 INFO:tasks.workunit.client.1.smithi067.stdout:4/630: mkdir d8/dba/d5d/d66/d7b/dbd/dd1 0 2024-04-12T05:37:30.670 INFO:tasks.workunit.client.1.smithi067.stdout:6/719: truncate dd/d11/d27/f9a 375361 0 2024-04-12T05:37:30.670 INFO:tasks.workunit.client.1.smithi067.stdout:6/720: readlink l5 0 2024-04-12T05:37:30.674 INFO:tasks.workunit.client.1.smithi067.stdout:6/721: write dd/d11/d27/d4a/d51/f57 [786376,35230] 0 2024-04-12T05:37:30.675 INFO:tasks.workunit.client.1.smithi067.stdout:4/631: dwrite d8/dba/d5d/d66/d71/d92/fc5 [0,4194304] 0 2024-04-12T05:37:30.676 INFO:tasks.workunit.client.1.smithi067.stdout:2/743: dread d5/d6a/f88 [0,4194304] 0 2024-04-12T05:37:30.683 INFO:tasks.workunit.client.1.smithi067.stdout:7/741: truncate d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/dd6/f8e 2895246 0 2024-04-12T05:37:30.685 INFO:tasks.workunit.client.1.smithi067.stdout:4/632: rename d8/d9/f41 to d8/da/d9f/d68/d8e/fd2 0 2024-04-12T05:37:30.685 INFO:tasks.workunit.client.1.smithi067.stdout:2/744: rename d5/da/d8c to d5/da/d8c/ddf/df1 22 2024-04-12T05:37:30.690 INFO:tasks.workunit.client.1.smithi067.stdout:7/742: rename d0/d23/d80/da0/db4/dd/d79/cc6 to d0/d23/d45/cff 0 2024-04-12T05:37:30.692 INFO:tasks.workunit.client.1.smithi067.stdout:4/633: mknod d8/da/d1d/d15/d58/cd3 0 2024-04-12T05:37:30.694 INFO:tasks.workunit.client.1.smithi067.stdout:7/743: creat d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/dd6/ddf/f100 x:0 0 0 2024-04-12T05:37:30.694 INFO:tasks.workunit.client.1.smithi067.stdout:6/722: dread dd/d21/d64/d6f/d90/d98/fb9 [0,4194304] 0 2024-04-12T05:37:30.697 INFO:tasks.workunit.client.1.smithi067.stdout:4/634: mkdir d8/dba/d5d/d66/d96/dd4 0 2024-04-12T05:37:30.698 INFO:tasks.workunit.client.1.smithi067.stdout:2/745: write d5/d11/f32 [4393132,15854] 0 2024-04-12T05:37:30.702 INFO:tasks.workunit.client.1.smithi067.stdout:6/723: rename dd/d21/d64/d88/cc8 to dd/d21/d64/d88/d7b/dcc/d93/cef 0 2024-04-12T05:37:30.704 INFO:tasks.workunit.client.1.smithi067.stdout:2/746: mknod d5/da/dae/cf2 0 2024-04-12T05:37:30.706 INFO:tasks.workunit.client.1.smithi067.stdout:7/744: dwrite d0/d23/d80/da0/db4/d5/d1d/ff5 [0,4194304] 0 2024-04-12T05:37:30.709 INFO:tasks.workunit.client.1.smithi067.stdout:4/635: dwrite d8/da/d8b/fa7 [0,4194304] 0 2024-04-12T05:37:30.709 INFO:tasks.workunit.client.1.smithi067.stdout:6/724: creat dd/d11/d27/d4a/ff0 x:0 0 0 2024-04-12T05:37:30.715 INFO:tasks.workunit.client.1.smithi067.stdout:4/636: chown d8/da/d9f/db5/fbe 3 1 2024-04-12T05:37:30.715 INFO:tasks.workunit.client.1.smithi067.stdout:6/725: creat dd/d11/d27/ff1 x:0 0 0 2024-04-12T05:37:30.717 INFO:tasks.workunit.client.1.smithi067.stdout:7/745: dread d0/d23/d45/f26 [0,4194304] 0 2024-04-12T05:37:30.717 INFO:tasks.workunit.client.1.smithi067.stdout:6/726: creat dd/d21/d3f/dd5/ff2 x:0 0 0 2024-04-12T05:37:30.719 INFO:tasks.workunit.client.1.smithi067.stdout:7/746: write d0/d23/d80/da0/db4/d5/d1d/f3a [4883415,131003] 0 2024-04-12T05:37:30.723 INFO:tasks.workunit.client.1.smithi067.stdout:6/727: link dd/d11/f5b dd/d11/d27/ff3 0 2024-04-12T05:37:30.723 INFO:tasks.workunit.client.1.smithi067.stdout:7/747: rmdir d0/d23/d80/da0 39 2024-04-12T05:37:30.727 INFO:tasks.workunit.client.1.smithi067.stdout:6/728: write dd/d11/d27/d4a/d79/f68 [1870829,14412] 0 2024-04-12T05:37:30.727 INFO:tasks.workunit.client.1.smithi067.stdout:2/747: truncate d5/da/de/dc4/def/ff0 764164 0 2024-04-12T05:37:30.739 INFO:tasks.workunit.client.1.smithi067.stdout:4/637: dread d8/da/f1f [0,4194304] 0 2024-04-12T05:37:30.740 INFO:tasks.workunit.client.1.smithi067.stdout:6/729: truncate dd/d21/d3f/dd5/fd9 3352192 0 2024-04-12T05:37:30.743 INFO:tasks.workunit.client.1.smithi067.stdout:6/730: mkdir dd/df4 0 2024-04-12T05:37:30.744 INFO:tasks.workunit.client.1.smithi067.stdout:4/638: unlink d8/dba/d5d/d66/d71/d92/fa1 0 2024-04-12T05:37:30.744 INFO:tasks.workunit.client.1.smithi067.stdout:2/748: dwrite d5/d23/f51 [0,4194304] 0 2024-04-12T05:37:30.747 INFO:tasks.workunit.client.1.smithi067.stdout:2/749: write d5/d6a/fa6 [230524,33585] 0 2024-04-12T05:37:30.748 INFO:tasks.workunit.client.1.smithi067.stdout:6/731: creat dd/d21/d3f/dd5/ff5 x:0 0 0 2024-04-12T05:37:30.749 INFO:tasks.workunit.client.1.smithi067.stdout:2/750: readlink d5/d6a/d78/lb1 0 2024-04-12T05:37:30.751 INFO:tasks.workunit.client.1.smithi067.stdout:6/732: read dd/d11/f5d [767875,91625] 0 2024-04-12T05:37:30.757 INFO:tasks.workunit.client.1.smithi067.stdout:4/639: link d8/da/d9f/c2f d8/dba/d5d/d66/d7b/dbd/dd1/cd5 0 2024-04-12T05:37:30.758 INFO:tasks.workunit.client.1.smithi067.stdout:2/751: fsync d5/d23/d3a/f6f 0 2024-04-12T05:37:30.760 INFO:tasks.workunit.client.1.smithi067.stdout:4/640: symlink d8/dba/d5d/d66/d96/ld6 0 2024-04-12T05:37:30.764 INFO:tasks.workunit.client.1.smithi067.stdout:4/641: mknod d8/da/d1d/d15/d75/daa/cd7 0 2024-04-12T05:37:30.766 INFO:tasks.workunit.client.1.smithi067.stdout:4/642: chown d8/d9/f11 7 1 2024-04-12T05:37:30.767 INFO:tasks.workunit.client.1.smithi067.stdout:2/752: dwrite d5/da/dae/dba/fc9 [0,4194304] 0 2024-04-12T05:37:30.772 INFO:tasks.workunit.client.1.smithi067.stdout:4/643: rename d8/da/d1d/d15/d75/f9e to d8/da/d9f/d68/db4/fd8 0 2024-04-12T05:37:30.772 INFO:tasks.workunit.client.1.smithi067.stdout:2/753: creat d5/d43/d6e/ff3 x:0 0 0 2024-04-12T05:37:30.774 INFO:tasks.workunit.client.1.smithi067.stdout:6/733: dread dd/d21/d64/d88/f96 [0,4194304] 0 2024-04-12T05:37:30.777 INFO:tasks.workunit.client.1.smithi067.stdout:1/749: sync 2024-04-12T05:37:30.779 INFO:tasks.workunit.client.1.smithi067.stdout:2/754: truncate d5/da/f1d 2252339 0 2024-04-12T05:37:30.779 INFO:tasks.workunit.client.1.smithi067.stdout:2/755: readlink d5/d6a/d78/l9e 0 2024-04-12T05:37:30.780 INFO:tasks.workunit.client.1.smithi067.stdout:4/644: symlink d8/da/d9f/d68/db4/ld9 0 2024-04-12T05:37:30.781 INFO:tasks.workunit.client.1.smithi067.stdout:1/750: mknod d4/d9/d3a/d51/ced 0 2024-04-12T05:37:30.783 INFO:tasks.workunit.client.1.smithi067.stdout:6/734: write dd/d11/d27/ff3 [4185041,29499] 0 2024-04-12T05:37:30.784 INFO:tasks.workunit.client.1.smithi067.stdout:1/751: symlink d4/d8f/dea/lee 0 2024-04-12T05:37:30.788 INFO:tasks.workunit.client.1.smithi067.stdout:6/735: write dd/d11/fb7 [909297,71111] 0 2024-04-12T05:37:30.791 INFO:tasks.workunit.client.1.smithi067.stdout:6/736: fdatasync dd/f1d 0 2024-04-12T05:37:30.792 INFO:tasks.workunit.client.1.smithi067.stdout:4/645: dwrite d8/da/d1d/d15/d75/f7c [0,4194304] 0 2024-04-12T05:37:30.795 INFO:tasks.workunit.client.1.smithi067.stdout:6/737: write dd/d21/d3f/dd5/ff5 [389873,4836] 0 2024-04-12T05:37:30.798 INFO:tasks.workunit.client.1.smithi067.stdout:6/738: chown dd/d11/f1e 239664 1 2024-04-12T05:37:30.798 INFO:tasks.workunit.client.1.smithi067.stdout:4/646: write d8/dba/d5d/f8f [804182,98974] 0 2024-04-12T05:37:30.801 INFO:tasks.workunit.client.1.smithi067.stdout:6/739: write dd/d11/fb7 [1893509,75204] 0 2024-04-12T05:37:30.802 INFO:tasks.workunit.client.1.smithi067.stdout:2/756: dread d5/d23/d3a/f59 [0,4194304] 0 2024-04-12T05:37:30.805 INFO:tasks.workunit.client.1.smithi067.stdout:2/757: symlink d5/da/de/dc4/dc6/lf4 0 2024-04-12T05:37:30.805 INFO:tasks.workunit.client.1.smithi067.stdout:1/752: dwrite d4/d9/d3a/f8b [0,4194304] 0 2024-04-12T05:37:30.807 INFO:tasks.workunit.client.1.smithi067.stdout:6/740: creat dd/d11/dc1/ff6 x:0 0 0 2024-04-12T05:37:30.808 INFO:tasks.workunit.client.1.smithi067.stdout:2/758: rmdir d5/da/dae/dba 39 2024-04-12T05:37:30.810 INFO:tasks.workunit.client.1.smithi067.stdout:1/753: mknod d4/d9/d4d/de1/cef 0 2024-04-12T05:37:30.816 INFO:tasks.workunit.client.1.smithi067.stdout:6/741: dread dd/d21/d64/fce [0,4194304] 0 2024-04-12T05:37:30.816 INFO:tasks.workunit.client.1.smithi067.stdout:1/754: link d4/d9/d12/f2e d4/d9/d3a/dc4/ff0 0 2024-04-12T05:37:30.818 INFO:tasks.workunit.client.1.smithi067.stdout:6/742: chown dd/d11/d27/d4a/d51/f57 323649803 1 2024-04-12T05:37:30.820 INFO:tasks.workunit.client.1.smithi067.stdout:1/755: symlink d4/d3d/lf1 0 2024-04-12T05:37:30.820 INFO:tasks.workunit.client.1.smithi067.stdout:2/759: write d5/da/de/fe0 [7629282,105467] 0 2024-04-12T05:37:30.822 INFO:tasks.workunit.client.1.smithi067.stdout:6/743: mknod dd/d11/d27/d4a/cf7 0 2024-04-12T05:37:30.823 INFO:tasks.workunit.client.1.smithi067.stdout:1/756: stat d4/d9/d3a/la2 0 2024-04-12T05:37:30.823 INFO:tasks.workunit.client.1.smithi067.stdout:6/744: dread - dd/d11/d27/d4a/ff0 zero size 2024-04-12T05:37:30.824 INFO:tasks.workunit.client.1.smithi067.stdout:2/760: dread - d5/da/de/d14/d34/d4c/fb5 zero size 2024-04-12T05:37:30.831 INFO:tasks.workunit.client.1.smithi067.stdout:6/745: dwrite dd/d21/d64/d88/d7b/dcc/d93/fd8 [0,4194304] 0 2024-04-12T05:37:30.832 INFO:tasks.workunit.client.1.smithi067.stdout:2/761: truncate d5/da/de/d14/d46/f5a 1172371 0 2024-04-12T05:37:30.839 INFO:tasks.workunit.client.1.smithi067.stdout:6/746: dwrite dd/d21/d64/d6f/d90/d98/fb9 [0,4194304] 0 2024-04-12T05:37:30.852 INFO:tasks.workunit.client.1.smithi067.stdout:2/762: dread d5/da/de/d14/d34/f37 [0,4194304] 0 2024-04-12T05:37:30.855 INFO:tasks.workunit.client.1.smithi067.stdout:6/747: dwrite dd/d21/d3f/dd5/fd9 [0,4194304] 0 2024-04-12T05:37:30.862 INFO:tasks.workunit.client.1.smithi067.stdout:2/763: dwrite d5/d43/f53 [0,4194304] 0 2024-04-12T05:37:30.941 INFO:tasks.workunit.client.1.smithi067.stdout:2/764: mknod d5/da/d8c/ddf/cf5 0 2024-04-12T05:37:30.947 INFO:tasks.workunit.client.1.smithi067.stdout:6/748: dread dd/d11/d27/d4a/d79/f68 [0,4194304] 0 2024-04-12T05:37:30.956 INFO:tasks.workunit.client.1.smithi067.stdout:6/749: dread dd/d11/f5b [0,4194304] 0 2024-04-12T05:37:30.964 INFO:tasks.workunit.client.1.smithi067.stdout:6/750: write dd/d11/d27/d4a/d79/dc3/f3c [1733757,99957] 0 2024-04-12T05:37:30.971 INFO:tasks.workunit.client.1.smithi067.stdout:6/751: dwrite dd/f50 [0,4194304] 0 2024-04-12T05:37:30.972 INFO:tasks.workunit.client.1.smithi067.stdout:6/752: readlink dd/d11/d27/d4a/d79/l86 0 2024-04-12T05:37:30.980 INFO:tasks.workunit.client.1.smithi067.stdout:6/753: dwrite dd/d21/d3f/dd5/ff5 [0,4194304] 0 2024-04-12T05:37:30.984 INFO:tasks.workunit.client.1.smithi067.stdout:6/754: mkdir dd/d11/d27/d4a/df8 0 2024-04-12T05:37:30.987 INFO:tasks.workunit.client.1.smithi067.stdout:6/755: mkdir dd/d21/df9 0 2024-04-12T05:37:30.991 INFO:tasks.workunit.client.1.smithi067.stdout:6/756: rename dd/d11/d27/d4a/d79/dc3/l8d to dd/d21/d64/d88/d7b/dd2/lfa 0 2024-04-12T05:37:30.993 INFO:tasks.workunit.client.1.smithi067.stdout:6/757: truncate dd/d11/f6b 4166850 0 2024-04-12T05:37:30.997 INFO:tasks.workunit.client.1.smithi067.stdout:6/758: dread dd/d11/d27/ff3 [0,4194304] 0 2024-04-12T05:37:31.004 INFO:tasks.workunit.client.1.smithi067.stdout:6/759: dwrite dd/d21/d3f/dd5/ff5 [0,4194304] 0 2024-04-12T05:37:31.006 INFO:tasks.workunit.client.1.smithi067.stdout:6/760: write dd/d11/f5d [2671659,6785] 0 2024-04-12T05:37:31.011 INFO:tasks.workunit.client.1.smithi067.stdout:6/761: getdents dd/d21 0 2024-04-12T05:37:31.018 INFO:tasks.workunit.client.1.smithi067.stdout:6/762: dwrite dd/d21/d3f/dd5/fd9 [0,4194304] 0 2024-04-12T05:37:31.022 INFO:tasks.workunit.client.1.smithi067.stdout:6/763: getdents dd/d21/d3f/dd5 0 2024-04-12T05:37:31.029 INFO:tasks.workunit.client.1.smithi067.stdout:6/764: dwrite f0 [4194304,4194304] 0 2024-04-12T05:37:31.031 INFO:tasks.workunit.client.1.smithi067.stdout:6/765: write dd/d21/d3f/dd5/ff2 [758545,94535] 0 2024-04-12T05:37:31.033 INFO:tasks.workunit.client.1.smithi067.stdout:6/766: dread - dd/d11/d27/d4a/d79/dc3/d56/f8a zero size 2024-04-12T05:37:31.035 INFO:tasks.workunit.client.1.smithi067.stdout:0/784: sync 2024-04-12T05:37:31.035 INFO:tasks.workunit.client.1.smithi067.stdout:8/619: sync 2024-04-12T05:37:31.035 INFO:tasks.workunit.client.1.smithi067.stdout:3/569: sync 2024-04-12T05:37:31.035 INFO:tasks.workunit.client.1.smithi067.stdout:5/995: sync 2024-04-12T05:37:31.037 INFO:tasks.workunit.client.1.smithi067.stdout:0/785: readlink dd/d12/d14/l96 0 2024-04-12T05:37:31.037 INFO:tasks.workunit.client.1.smithi067.stdout:8/620: chown d6/f8a 272 1 2024-04-12T05:37:31.038 INFO:tasks.workunit.client.1.smithi067.stdout:6/767: unlink dd/d11/d27/d4a/d79/dc3/f3c 0 2024-04-12T05:37:31.044 INFO:tasks.workunit.client.1.smithi067.stdout:5/996: chown d0/d6/d40/d4b/d38/d8e/d8f/db9/dbd/fff 111 1 2024-04-12T05:37:31.045 INFO:tasks.workunit.client.1.smithi067.stdout:5/997: read d0/d6/d20/d3e/f8a [303294,65968] 0 2024-04-12T05:37:31.045 INFO:tasks.workunit.client.1.smithi067.stdout:8/621: creat d6/d8/d18/d63/fb6 x:0 0 0 2024-04-12T05:37:31.045 INFO:tasks.workunit.client.1.smithi067.stdout:6/768: truncate dd/d21/d64/db5/fc9 1210764 0 2024-04-12T05:37:31.046 INFO:tasks.workunit.client.1.smithi067.stdout:0/786: rename dd/d12/d14/d49/dab/cd6 to dd/dbd/dbe/cf7 0 2024-04-12T05:37:31.048 INFO:tasks.workunit.client.1.smithi067.stdout:3/570: dwrite dc/df/f30 [4194304,4194304] 0 2024-04-12T05:37:31.050 INFO:tasks.workunit.client.1.smithi067.stdout:0/787: read - dd/d12/d14/ff3 zero size 2024-04-12T05:37:31.051 INFO:tasks.workunit.client.1.smithi067.stdout:5/998: creat d0/d6/d20/f14a x:0 0 0 2024-04-12T05:37:31.054 INFO:tasks.workunit.client.1.smithi067.stdout:6/769: unlink dd/d21/d64/d88/d7b/l7d 0 2024-04-12T05:37:31.057 INFO:tasks.workunit.client.1.smithi067.stdout:0/788: creat dd/d12/d14/de3/ff8 x:0 0 0 2024-04-12T05:37:31.059 INFO:tasks.workunit.client.1.smithi067.stdout:6/770: stat dd/d21/d64/d88/d7b/dcd 0 2024-04-12T05:37:31.061 INFO:tasks.workunit.client.1.smithi067.stdout:8/622: truncate d6/d8/d18/d63/d9a/d12/fa9 952401 0 2024-04-12T05:37:31.062 INFO:tasks.workunit.client.1.smithi067.stdout:0/789: write dd/d12/f4e [2018190,43011] 0 2024-04-12T05:37:31.064 INFO:tasks.workunit.client.1.smithi067.stdout:3/571: rename dc/d1d/d28/d3e/d53/d78/c91 to dc/d1d/d28/d3e/cc2 0 2024-04-12T05:37:31.065 INFO:tasks.workunit.client.1.smithi067.stdout:0/790: chown dd/d12/f59 41 1 2024-04-12T05:37:31.074 INFO:tasks.workunit.client.1.smithi067.stdout:8/623: rmdir d6/d8/d18/d63/d67/d69/d5d 39 2024-04-12T05:37:31.086 INFO:tasks.workunit.client.1.smithi067.stdout:5/999: dwrite d0/d6/d40/d4b/d38/d8e/d8f/fe7 [4194304,4194304] 0 2024-04-12T05:37:31.092 INFO:tasks.workunit.client.1.smithi067.stdout:8/624: write d6/d8/d18/fb4 [921292,108607] 0 2024-04-12T05:37:31.093 INFO:tasks.workunit.client.1.smithi067.stdout:3/572: dwrite dc/d1d/d28/d3e/d8a/f95 [0,4194304] 0 2024-04-12T05:37:31.098 INFO:tasks.workunit.client.1.smithi067.stdout:0/791: dread dd/d5b/f84 [0,4194304] 0 2024-04-12T05:37:31.101 INFO:tasks.workunit.client.1.smithi067.stdout:8/625: symlink d6/d8/d18/d63/d9a/lb7 0 2024-04-12T05:37:31.105 INFO:tasks.workunit.client.1.smithi067.stdout:3/573: creat dc/df/fc3 x:0 0 0 2024-04-12T05:37:31.109 INFO:tasks.workunit.client.1.smithi067.stdout:8/626: symlink d6/d8/d18/d36/lb8 0 2024-04-12T05:37:31.110 INFO:tasks.workunit.client.1.smithi067.stdout:0/792: getdents dd/d9c/d64/ddd 0 2024-04-12T05:37:31.118 INFO:tasks.workunit.client.1.smithi067.stdout:3/574: truncate f1 1391511 0 2024-04-12T05:37:31.122 INFO:tasks.workunit.client.1.smithi067.stdout:8/627: dread d6/d8/d18/d36/f54 [0,4194304] 0 2024-04-12T05:37:31.123 INFO:tasks.workunit.client.1.smithi067.stdout:8/628: chown d6/d8/d18/d4e/d9e/ca2 12 1 2024-04-12T05:37:31.124 INFO:tasks.workunit.client.1.smithi067.stdout:3/575: fdatasync dc/d1d/f24 0 2024-04-12T05:37:31.131 INFO:tasks.workunit.client.1.smithi067.stdout:0/793: dwrite dd/d12/d14/d49/fd0 [0,4194304] 0 2024-04-12T05:37:31.132 INFO:tasks.workunit.client.1.smithi067.stdout:0/794: chown dd/d6f/d72/fbb 539843434 1 2024-04-12T05:37:31.133 INFO:tasks.workunit.client.1.smithi067.stdout:8/629: truncate d6/d8/d6e/f78 2084910 0 2024-04-12T05:37:31.133 INFO:tasks.workunit.client.1.smithi067.stdout:0/795: fsync dd/d12/f4e 0 2024-04-12T05:37:31.138 INFO:tasks.workunit.client.1.smithi067.stdout:0/796: write dd/d12/d14/d51/da2/fb5 [1345267,94322] 0 2024-04-12T05:37:31.138 INFO:tasks.workunit.client.1.smithi067.stdout:3/576: write dc/df/d27/d6b/f77 [1082507,4485] 0 2024-04-12T05:37:31.141 INFO:tasks.workunit.client.1.smithi067.stdout:8/630: creat d6/d86/fb9 x:0 0 0 2024-04-12T05:37:31.145 INFO:tasks.workunit.client.1.smithi067.stdout:8/631: fsync d6/d8/f6a 0 2024-04-12T05:37:31.150 INFO:tasks.workunit.client.1.smithi067.stdout:3/577: mknod dc/d1d/d28/cc4 0 2024-04-12T05:37:31.150 INFO:tasks.workunit.client.1.smithi067.stdout:0/797: creat dd/ff9 x:0 0 0 2024-04-12T05:37:31.153 INFO:tasks.workunit.client.1.smithi067.stdout:8/632: rename d6/d8/d18/d4e/d9e/lb1 to d6/d8/d18/d4e/d9e/lba 0 2024-04-12T05:37:31.163 INFO:tasks.workunit.client.1.smithi067.stdout:3/578: truncate dc/d1d/d28/f2a 4575390 0 2024-04-12T05:37:31.163 INFO:tasks.workunit.client.1.smithi067.stdout:0/798: write dd/d12/d14/d1a/f1e [1836866,112622] 0 2024-04-12T05:37:31.166 INFO:tasks.workunit.client.1.smithi067.stdout:8/633: dwrite d6/d8/d18/fb4 [0,4194304] 0 2024-04-12T05:37:31.167 INFO:tasks.workunit.client.1.smithi067.stdout:3/579: mkdir dc/df/d27/dc5 0 2024-04-12T05:37:31.172 INFO:tasks.workunit.client.1.smithi067.stdout:3/580: read dc/d1d/d28/d3e/d53/d76/fb4 [3812857,91826] 0 2024-04-12T05:37:31.176 INFO:tasks.workunit.client.1.smithi067.stdout:0/799: link dd/d12/d14/l17 dd/d9c/d5f/d78/lfa 0 2024-04-12T05:37:31.177 INFO:tasks.workunit.client.1.smithi067.stdout:0/800: chown dd/d9c/d5f/d78 10 1 2024-04-12T05:37:31.180 INFO:tasks.workunit.client.1.smithi067.stdout:0/801: fdatasync dd/d12/f16 0 2024-04-12T05:37:31.182 INFO:tasks.workunit.client.1.smithi067.stdout:3/581: rename dc/d50/f60 to dc/d1d/d26/fc6 0 2024-04-12T05:37:31.183 INFO:tasks.workunit.client.1.smithi067.stdout:0/802: rmdir dd/d9c 39 2024-04-12T05:37:31.185 INFO:tasks.workunit.client.1.smithi067.stdout:3/582: symlink dc/d1d/d28/d3e/d69/lc7 0 2024-04-12T05:37:31.186 INFO:tasks.workunit.client.1.smithi067.stdout:0/803: truncate dd/d92/f67 330779 0 2024-04-12T05:37:31.186 INFO:tasks.workunit.client.1.smithi067.stdout:3/583: truncate dc/d1d/f24 840783 0 2024-04-12T05:37:31.187 INFO:tasks.workunit.client.1.smithi067.stdout:0/804: chown dd/d12/d14/d1a/caa 12 1 2024-04-12T05:37:31.189 INFO:tasks.workunit.client.1.smithi067.stdout:3/584: rename dc/d1d/d89 to dc/df/d27/d6b/dc8 0 2024-04-12T05:37:31.190 INFO:tasks.workunit.client.1.smithi067.stdout:0/805: mknod dd/d12/d14/cfb 0 2024-04-12T05:37:31.191 INFO:tasks.workunit.client.1.smithi067.stdout:0/806: dread - dd/d9c/d5f/de0/ff5 zero size 2024-04-12T05:37:31.192 INFO:tasks.workunit.client.1.smithi067.stdout:3/585: mknod dc/d1d/cc9 0 2024-04-12T05:37:31.194 INFO:tasks.workunit.client.1.smithi067.stdout:3/586: mkdir dc/df/dca 0 2024-04-12T05:37:31.198 INFO:tasks.workunit.client.1.smithi067.stdout:0/807: dread dd/d12/fb2 [0,4194304] 0 2024-04-12T05:37:31.201 INFO:tasks.workunit.client.1.smithi067.stdout:3/587: mknod dc/d1d/d26/d52/ccb 0 2024-04-12T05:37:31.202 INFO:tasks.workunit.client.1.smithi067.stdout:0/808: creat dd/d9c/d5f/d78/d83/ffc x:0 0 0 2024-04-12T05:37:31.203 INFO:tasks.workunit.client.1.smithi067.stdout:0/809: stat dd/d9c/d5f/de0 0 2024-04-12T05:37:31.206 INFO:tasks.workunit.client.1.smithi067.stdout:3/588: creat dc/d1d/d28/fcc x:0 0 0 2024-04-12T05:37:31.206 INFO:tasks.workunit.client.1.smithi067.stdout:0/810: mkdir dd/d12/d14/de3/dfd 0 2024-04-12T05:37:31.208 INFO:tasks.workunit.client.1.smithi067.stdout:3/589: creat dc/d1d/d4d/fcd x:0 0 0 2024-04-12T05:37:31.212 INFO:tasks.workunit.client.1.smithi067.stdout:0/811: fdatasync dd/d12/d14/d1a/d3d/f32 0 2024-04-12T05:37:31.216 INFO:tasks.workunit.client.1.smithi067.stdout:0/812: mknod dd/d6f/d72/cfe 0 2024-04-12T05:37:31.224 INFO:tasks.workunit.client.1.smithi067.stdout:3/590: dwrite dc/d1d/d26/fb1 [4194304,4194304] 0 2024-04-12T05:37:31.225 INFO:tasks.workunit.client.1.smithi067.stdout:0/813: dwrite dd/d9c/d5f/f90 [0,4194304] 0 2024-04-12T05:37:31.228 INFO:tasks.workunit.client.1.smithi067.stdout:0/814: chown dd/d12/d14/d51/l7a 923 1 2024-04-12T05:37:31.231 INFO:tasks.workunit.client.1.smithi067.stdout:3/591: rename dc/df/d8d/cbb to dc/d1d/d26/d52/cce 0 2024-04-12T05:37:31.232 INFO:tasks.workunit.client.1.smithi067.stdout:3/592: chown dc/d1d/d28/d3e/d8a 12647 1 2024-04-12T05:37:31.234 INFO:tasks.workunit.client.1.smithi067.stdout:3/593: truncate dc/df/d27/daa/fbe 988542 0 2024-04-12T05:37:31.245 INFO:tasks.workunit.client.1.smithi067.stdout:3/594: dread f1 [0,4194304] 0 2024-04-12T05:37:31.245 INFO:tasks.workunit.client.1.smithi067.stdout:0/815: truncate dd/d6f/d72/f82 2063127 0 2024-04-12T05:37:31.248 INFO:tasks.workunit.client.1.smithi067.stdout:0/816: unlink dd/d9c/d5f/d78/d83/la0 0 2024-04-12T05:37:31.249 INFO:tasks.workunit.client.1.smithi067.stdout:3/595: symlink dc/d1d/d28/d3e/d53/lcf 0 2024-04-12T05:37:31.249 INFO:tasks.workunit.client.1.smithi067.stdout:0/817: write dd/d9c/d5f/de0/ff6 [504427,84771] 0 2024-04-12T05:37:31.251 INFO:tasks.workunit.client.1.smithi067.stdout:3/596: mknod dc/df/cd0 0 2024-04-12T05:37:31.252 INFO:tasks.workunit.client.1.smithi067.stdout:3/597: stat dc/d1d/d4d/l7d 0 2024-04-12T05:37:31.261 INFO:tasks.workunit.client.1.smithi067.stdout:0/818: truncate dd/d12/d14/d49/dab/fcf 2335652 0 2024-04-12T05:37:31.262 INFO:tasks.workunit.client.1.smithi067.stdout:0/819: stat dd/d12/d14/d49 0 2024-04-12T05:37:31.267 INFO:tasks.workunit.client.1.smithi067.stdout:3/598: dwrite dc/d1d/d26/f5e [0,4194304] 0 2024-04-12T05:37:31.269 INFO:tasks.workunit.client.1.smithi067.stdout:0/820: symlink dd/d9c/d5f/d78/lff 0 2024-04-12T05:37:31.272 INFO:tasks.workunit.client.1.smithi067.stdout:0/821: creat dd/dbd/f100 x:0 0 0 2024-04-12T05:37:31.275 INFO:tasks.workunit.client.1.smithi067.stdout:0/822: mkdir dd/d6f/d101 0 2024-04-12T05:37:31.276 INFO:tasks.workunit.client.1.smithi067.stdout:0/823: stat dd/d12/d14/f61 0 2024-04-12T05:37:31.278 INFO:tasks.workunit.client.1.smithi067.stdout:3/599: link dc/d1d/d26/f72 dc/fd1 0 2024-04-12T05:37:31.285 INFO:tasks.workunit.client.1.smithi067.stdout:3/600: truncate dc/d1d/d28/d3e/f6d 4170273 0 2024-04-12T05:37:31.287 INFO:tasks.workunit.client.1.smithi067.stdout:3/601: write dc/d1d/d28/d3e/d53/d76/fb4 [1370343,55310] 0 2024-04-12T05:37:31.289 INFO:tasks.workunit.client.1.smithi067.stdout:3/602: rmdir dc/df/d27/daa 39 2024-04-12T05:37:31.290 INFO:tasks.workunit.client.1.smithi067.stdout:3/603: chown dc/d1d/d28/d3e/d8a/d98/fa2 174621408 1 2024-04-12T05:37:31.290 INFO:tasks.workunit.client.1.smithi067.stdout:3/604: chown dc/df/f82 13140 1 2024-04-12T05:37:31.292 INFO:tasks.workunit.client.1.smithi067.stdout:3/605: dread f1 [0,4194304] 0 2024-04-12T05:37:31.294 INFO:tasks.workunit.client.1.smithi067.stdout:3/606: symlink dc/df/d27/daa/ld2 0 2024-04-12T05:37:31.297 INFO:tasks.workunit.client.1.smithi067.stdout:3/607: link dc/d50/l59 dc/d1d/d28/d3e/d53/ld3 0 2024-04-12T05:37:31.298 INFO:tasks.workunit.client.1.smithi067.stdout:3/608: chown dc/faf 0 1 2024-04-12T05:37:31.300 INFO:tasks.workunit.client.1.smithi067.stdout:3/609: creat dc/df/d27/dc5/fd4 x:0 0 0 2024-04-12T05:37:31.301 INFO:tasks.workunit.client.1.smithi067.stdout:3/610: symlink dc/d1d/d26/d52/d9a/ld5 0 2024-04-12T05:37:31.317 INFO:tasks.workunit.client.1.smithi067.stdout:3/611: write dc/d1d/d28/d3e/f7f [213338,22083] 0 2024-04-12T05:37:31.320 INFO:tasks.workunit.client.1.smithi067.stdout:3/612: creat dc/df/d27/d6b/dc8/da3/db9/fd6 x:0 0 0 2024-04-12T05:37:31.321 INFO:tasks.workunit.client.1.smithi067.stdout:3/613: write dc/d1d/d26/fb1 [5296237,120883] 0 2024-04-12T05:37:31.326 INFO:tasks.workunit.client.1.smithi067.stdout:3/614: getdents dc/d1d 0 2024-04-12T05:37:31.326 INFO:tasks.workunit.client.1.smithi067.stdout:3/615: readlink dc/d1d/d28/d3e/d53/lcf 0 2024-04-12T05:37:31.328 INFO:tasks.workunit.client.1.smithi067.stdout:3/616: mknod dc/d50/da6/cd7 0 2024-04-12T05:37:31.332 INFO:tasks.workunit.client.1.smithi067.stdout:3/617: dread dc/f3a [4194304,4194304] 0 2024-04-12T05:37:31.334 INFO:tasks.workunit.client.1.smithi067.stdout:3/618: mkdir dc/df/d27/dc5/dd8 0 2024-04-12T05:37:31.335 INFO:tasks.workunit.client.1.smithi067.stdout:3/619: fsync dc/d1d/d28/f39 0 2024-04-12T05:37:31.336 INFO:tasks.workunit.client.1.smithi067.stdout:3/620: dread - dc/d1d/d4d/fa5 zero size 2024-04-12T05:37:31.340 INFO:tasks.workunit.client.1.smithi067.stdout:3/621: write dc/f40 [461504,40836] 0 2024-04-12T05:37:31.341 INFO:tasks.workunit.client.1.smithi067.stdout:3/622: dread - dc/df/d27/daa/fb3 zero size 2024-04-12T05:37:31.342 INFO:tasks.workunit.client.1.smithi067.stdout:3/623: chown dc/d1d/d28/d3e/d8a/d98 7258432 1 2024-04-12T05:37:31.348 INFO:tasks.workunit.client.1.smithi067.stdout:3/624: creat dc/d1d/d28/d3e/d69/dbc/fd9 x:0 0 0 2024-04-12T05:37:31.357 INFO:tasks.workunit.client.1.smithi067.stdout:3/625: creat dc/d1d/d28/d3e/d53/d94/fda x:0 0 0 2024-04-12T05:37:31.358 INFO:tasks.workunit.client.1.smithi067.stdout:3/626: symlink dc/d50/ldb 0 2024-04-12T05:37:31.364 INFO:tasks.workunit.client.1.smithi067.stdout:3/627: dwrite dc/d1d/d28/f4c [0,4194304] 0 2024-04-12T05:37:31.371 INFO:tasks.workunit.client.1.smithi067.stdout:3/628: dwrite dc/d1d/d28/d3e/f66 [0,4194304] 0 2024-04-12T05:37:31.382 INFO:tasks.workunit.client.1.smithi067.stdout:3/629: link dc/df/cd0 dc/d1d/d28/d3e/d53/d78/cdc 0 2024-04-12T05:37:31.384 INFO:tasks.workunit.client.1.smithi067.stdout:3/630: creat dc/d1d/d4d/fdd x:0 0 0 2024-04-12T05:37:31.388 INFO:tasks.workunit.client.1.smithi067.stdout:3/631: dread dc/d1d/d28/f4c [0,4194304] 0 2024-04-12T05:37:31.425 INFO:tasks.workunit.client.1.smithi067.stdout:9/723: sync 2024-04-12T05:37:31.426 INFO:tasks.workunit.client.1.smithi067.stdout:7/748: sync 2024-04-12T05:37:31.427 INFO:tasks.workunit.client.1.smithi067.stdout:9/724: fsync d1/d28/d21/d45/d75/fa6 0 2024-04-12T05:37:31.428 INFO:tasks.workunit.client.1.smithi067.stdout:9/725: chown d1/d28/d21/d68 1 1 2024-04-12T05:37:31.439 INFO:tasks.workunit.client.1.smithi067.stdout:9/726: dwrite d1/d5f/f65 [0,4194304] 0 2024-04-12T05:37:31.441 INFO:tasks.workunit.client.1.smithi067.stdout:9/727: fsync d1/d28/d21/d45/f8b 0 2024-04-12T05:37:31.441 INFO:tasks.workunit.client.1.smithi067.stdout:7/749: dread d0/d23/d80/da0/db4/d5/f50 [0,4194304] 0 2024-04-12T05:37:31.442 INFO:tasks.workunit.client.1.smithi067.stdout:9/728: stat d1/d66/d8e/fc6 0 2024-04-12T05:37:31.451 INFO:tasks.workunit.client.1.smithi067.stdout:7/750: dwrite d0/f72 [0,4194304] 0 2024-04-12T05:37:31.456 INFO:tasks.workunit.client.1.smithi067.stdout:9/729: write d1/d28/d21/d45/d75/db3/f24 [113728,28644] 0 2024-04-12T05:37:31.457 INFO:tasks.workunit.client.1.smithi067.stdout:7/751: write d0/d23/d80/da0/db4/f74 [1155440,26103] 0 2024-04-12T05:37:31.457 INFO:tasks.workunit.client.1.smithi067.stdout:9/730: stat d1/d12/c19 0 2024-04-12T05:37:31.460 INFO:tasks.workunit.client.1.smithi067.stdout:9/731: symlink d1/d28/d21/d3f/d4f/ldf 0 2024-04-12T05:37:31.464 INFO:tasks.workunit.client.1.smithi067.stdout:9/732: rename d1/l39 to d1/d28/d44/d8d/dd2/le0 0 2024-04-12T05:37:31.465 INFO:tasks.workunit.client.1.smithi067.stdout:7/752: rmdir d0/d23/d80/da0/db4/dd/d42/d7e/dd8 0 2024-04-12T05:37:31.466 INFO:tasks.workunit.client.1.smithi067.stdout:9/733: fsync d1/d5f/fcc 0 2024-04-12T05:37:31.467 INFO:tasks.workunit.client.1.smithi067.stdout:9/734: chown d1/d28/d44/fc7 31153220 1 2024-04-12T05:37:31.471 INFO:tasks.workunit.client.1.smithi067.stdout:9/735: rename d1/d28/d5e/f6f to d1/d28/d44/d8d/dc1/dcb/fe1 0 2024-04-12T05:37:31.471 INFO:tasks.workunit.client.1.smithi067.stdout:9/736: stat d1/d13/l37 0 2024-04-12T05:37:31.474 INFO:tasks.workunit.client.1.smithi067.stdout:9/737: creat d1/d66/d7a/fe2 x:0 0 0 2024-04-12T05:37:31.476 INFO:tasks.workunit.client.1.smithi067.stdout:9/738: mkdir d1/d28/d44/d8d/dc1/de3 0 2024-04-12T05:37:31.478 INFO:tasks.workunit.client.1.smithi067.stdout:7/753: dread d0/f8 [0,4194304] 0 2024-04-12T05:37:31.480 INFO:tasks.workunit.client.1.smithi067.stdout:9/739: rename d1/d13/d34/fab to d1/d13/d34/fe4 0 2024-04-12T05:37:31.481 INFO:tasks.workunit.client.1.smithi067.stdout:7/754: mknod d0/d23/d45/dd1/c101 0 2024-04-12T05:37:31.484 INFO:tasks.workunit.client.1.smithi067.stdout:7/755: unlink d0/d23/d80/da0/db4/dd/d42/d82/ff4 0 2024-04-12T05:37:31.487 INFO:tasks.workunit.client.1.smithi067.stdout:9/740: dwrite d1/d12/f61 [0,4194304] 0 2024-04-12T05:37:31.489 INFO:tasks.workunit.client.1.smithi067.stdout:7/756: creat d0/d23/d80/da0/db4/f102 x:0 0 0 2024-04-12T05:37:31.491 INFO:tasks.workunit.client.1.smithi067.stdout:9/741: symlink d1/d28/d44/d8d/dc1/dcb/le5 0 2024-04-12T05:37:31.493 INFO:tasks.workunit.client.1.smithi067.stdout:7/757: truncate d0/d23/d80/da0/db4/dd/d79/fdd 41610 0 2024-04-12T05:37:31.494 INFO:tasks.workunit.client.1.smithi067.stdout:7/758: read d0/d23/d80/fcb [4439580,60132] 0 2024-04-12T05:37:31.499 INFO:tasks.workunit.client.1.smithi067.stdout:7/759: truncate d0/d23/d80/da0/db4/d5/d1d/f40 2550832 0 2024-04-12T05:37:31.501 INFO:tasks.workunit.client.1.smithi067.stdout:7/760: write d0/d23/d80/da0/db4/dd/d42/d7e/f94 [4875751,67502] 0 2024-04-12T05:37:31.504 INFO:tasks.workunit.client.1.smithi067.stdout:7/761: creat d0/d23/d80/da0/db4/d5/d1d/dec/f103 x:0 0 0 2024-04-12T05:37:31.506 INFO:tasks.workunit.client.1.smithi067.stdout:7/762: truncate d0/d23/d80/da0/db4/d5/d1d/de9/fba 224461 0 2024-04-12T05:37:31.508 INFO:tasks.workunit.client.1.smithi067.stdout:9/742: rename d1/d28/d44 to d1/d28/d21/d45/d75/de6 0 2024-04-12T05:37:31.508 INFO:tasks.workunit.client.1.smithi067.stdout:7/763: dread d0/d23/d80/da0/db4/dd/d79/fdd [0,4194304] 0 2024-04-12T05:37:31.511 INFO:tasks.workunit.client.1.smithi067.stdout:9/743: creat d1/fe7 x:0 0 0 2024-04-12T05:37:31.512 INFO:tasks.workunit.client.1.smithi067.stdout:7/764: mknod d0/d23/d80/da0/db4/d5/d4a/de5/c104 0 2024-04-12T05:37:31.514 INFO:tasks.workunit.client.1.smithi067.stdout:9/744: symlink d1/d28/d27/d32/le8 0 2024-04-12T05:37:31.515 INFO:tasks.workunit.client.1.smithi067.stdout:9/745: read d1/f79 [194873,91220] 0 2024-04-12T05:37:31.516 INFO:tasks.workunit.client.1.smithi067.stdout:9/746: stat d1/d28/d21/d45/d75/db3/f24 0 2024-04-12T05:37:31.517 INFO:tasks.workunit.client.1.smithi067.stdout:9/747: write d1/d28/fb8 [911683,9113] 0 2024-04-12T05:37:31.526 INFO:tasks.workunit.client.1.smithi067.stdout:9/748: dwrite d1/d28/d21/d3f/d4f/fbe [0,4194304] 0 2024-04-12T05:37:31.528 INFO:tasks.workunit.client.1.smithi067.stdout:7/765: dwrite d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/dd6/ddf/fef [0,4194304] 0 2024-04-12T05:37:31.532 INFO:tasks.workunit.client.1.smithi067.stdout:7/766: fsync d0/f12 0 2024-04-12T05:37:31.537 INFO:tasks.workunit.client.1.smithi067.stdout:9/749: dwrite d1/d28/d21/d45/d75/de6/f9e [0,4194304] 0 2024-04-12T05:37:31.541 INFO:tasks.workunit.client.1.smithi067.stdout:9/750: truncate d1/d28/d21/d45/d75/db3/f24 1021841 0 2024-04-12T05:37:31.541 INFO:tasks.workunit.client.1.smithi067.stdout:4/647: sync 2024-04-12T05:37:31.549 INFO:tasks.workunit.client.1.smithi067.stdout:7/767: dwrite d0/d23/d80/da0/db4/dd/d42/d71/fac [0,4194304] 0 2024-04-12T05:37:31.549 INFO:tasks.workunit.client.1.smithi067.stdout:9/751: rename d1/d28/d21/d45/d75/db3/f24 to d1/d66/fe9 0 2024-04-12T05:37:31.552 INFO:tasks.workunit.client.1.smithi067.stdout:9/752: truncate d1/d12/d15/fb0 138450 0 2024-04-12T05:37:31.553 INFO:tasks.workunit.client.1.smithi067.stdout:9/753: stat d1/d28/d21/d45/l6d 0 2024-04-12T05:37:31.555 INFO:tasks.workunit.client.1.smithi067.stdout:4/648: creat d8/dba/d5d/d66/d71/fda x:0 0 0 2024-04-12T05:37:31.557 INFO:tasks.workunit.client.1.smithi067.stdout:7/768: symlink d0/d23/d80/da0/db4/d5/d4a/l105 0 2024-04-12T05:37:31.559 INFO:tasks.workunit.client.1.smithi067.stdout:9/754: mkdir d1/d28/d21/d7f/da7/dea 0 2024-04-12T05:37:31.560 INFO:tasks.workunit.client.1.smithi067.stdout:9/755: stat d1/d28/d27/fdc 0 2024-04-12T05:37:31.561 INFO:tasks.workunit.client.1.smithi067.stdout:9/756: chown d1/d28/d21/l69 1945759 1 2024-04-12T05:37:31.562 INFO:tasks.workunit.client.1.smithi067.stdout:7/769: unlink d0/d67/l17 0 2024-04-12T05:37:31.564 INFO:tasks.workunit.client.1.smithi067.stdout:7/770: chown d0/d23/d80/da0/db4/f102 4 1 2024-04-12T05:37:31.569 INFO:tasks.workunit.client.1.smithi067.stdout:4/649: truncate d8/d9/f2a 3979572 0 2024-04-12T05:37:31.570 INFO:tasks.workunit.client.1.smithi067.stdout:9/757: write d1/d28/d21/d45/d75/de6/fc7 [520541,43796] 0 2024-04-12T05:37:31.572 INFO:tasks.workunit.client.1.smithi067.stdout:9/758: truncate d1/d28/d21/d3f/f41 6755198 0 2024-04-12T05:37:31.574 INFO:tasks.workunit.client.1.smithi067.stdout:9/759: stat d1/d28/d21/d3f/d4f/fc9 0 2024-04-12T05:37:31.575 INFO:tasks.workunit.client.1.smithi067.stdout:4/650: creat d8/dba/d5d/d66/d7b/fdb x:0 0 0 2024-04-12T05:37:31.576 INFO:tasks.workunit.client.1.smithi067.stdout:9/760: dread d1/d12/d15/fb0 [0,4194304] 0 2024-04-12T05:37:31.580 INFO:tasks.workunit.client.1.smithi067.stdout:9/761: creat d1/d12/d76/feb x:0 0 0 2024-04-12T05:37:31.581 INFO:tasks.workunit.client.1.smithi067.stdout:7/771: dread d0/d23/d80/da0/db4/f52 [0,4194304] 0 2024-04-12T05:37:31.584 INFO:tasks.workunit.client.1.smithi067.stdout:4/651: link d8/l22 d8/dba/d5d/d66/d7b/ldc 0 2024-04-12T05:37:31.586 INFO:tasks.workunit.client.1.smithi067.stdout:4/652: chown d8/da/d8b/l4a 26142 1 2024-04-12T05:37:31.588 INFO:tasks.workunit.client.1.smithi067.stdout:9/762: dwrite d1/d28/d21/d45/d75/de6/d8d/fa0 [0,4194304] 0 2024-04-12T05:37:31.590 INFO:tasks.workunit.client.1.smithi067.stdout:7/772: creat d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/f106 x:0 0 0 2024-04-12T05:37:31.597 INFO:tasks.workunit.client.1.smithi067.stdout:7/773: rename d0/d23/d80/da0/db4/d5/d1d/cde to d0/d23/d80/da0/db4/dd/d42/d7e/d34/da3/dea/c107 0 2024-04-12T05:37:31.598 INFO:tasks.workunit.client.1.smithi067.stdout:4/653: write d8/da/d1d/f85 [47133,38888] 0 2024-04-12T05:37:31.604 INFO:tasks.workunit.client.1.smithi067.stdout:7/774: rename d0/d23/d80/da0/db4/dd/d42/d7e/ff1 to d0/d23/d80/da0/db4/d5/d4a/f108 0 2024-04-12T05:37:31.605 INFO:tasks.workunit.client.1.smithi067.stdout:4/654: truncate d8/d39/f3b 4675718 0 2024-04-12T05:37:31.606 INFO:tasks.workunit.client.1.smithi067.stdout:7/775: fsync d0/d23/d45/f9e 0 2024-04-12T05:37:31.609 INFO:tasks.workunit.client.1.smithi067.stdout:4/655: creat d8/fdd x:0 0 0 2024-04-12T05:37:31.612 INFO:tasks.workunit.client.1.smithi067.stdout:7/776: mkdir d0/d23/d80/da0/db4/d109 0 2024-04-12T05:37:31.615 INFO:tasks.workunit.client.1.smithi067.stdout:4/656: creat d8/dba/d5d/d66/d71/fde x:0 0 0 2024-04-12T05:37:31.615 INFO:tasks.workunit.client.1.smithi067.stdout:7/777: read d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/dd6/ddf/fef [780690,12308] 0 2024-04-12T05:37:31.617 INFO:tasks.workunit.client.1.smithi067.stdout:4/657: write d8/dba/d5d/d66/d7b/fdb [314032,73701] 0 2024-04-12T05:37:31.618 INFO:tasks.workunit.client.1.smithi067.stdout:4/658: readlink d8/da/l35 0 2024-04-12T05:37:31.619 INFO:tasks.workunit.client.1.smithi067.stdout:7/778: rmdir d0/d23/d80/da0/db4/d5/d4a/de5 39 2024-04-12T05:37:31.619 INFO:tasks.workunit.client.1.smithi067.stdout:4/659: write d8/d9/d3d/f93 [4721197,64235] 0 2024-04-12T05:37:31.621 INFO:tasks.workunit.client.1.smithi067.stdout:7/779: read - d0/d23/d80/da0/db4/d5/d1d/de9/fcc zero size 2024-04-12T05:37:31.625 INFO:tasks.workunit.client.1.smithi067.stdout:4/660: rename d8/dba/d5d/d66/c72 to d8/da/cdf 0 2024-04-12T05:37:31.626 INFO:tasks.workunit.client.1.smithi067.stdout:7/780: symlink d0/d23/d80/da0/db4/dd/d42/d82/d96/de4/l10a 0 2024-04-12T05:37:31.628 INFO:tasks.workunit.client.1.smithi067.stdout:7/781: mknod d0/d23/d80/da0/db4/dd/d42/d7e/d34/c10b 0 2024-04-12T05:37:31.629 INFO:tasks.workunit.client.1.smithi067.stdout:4/661: truncate d8/dba/d5d/d66/d7b/d6d/f81 225802 0 2024-04-12T05:37:31.630 INFO:tasks.workunit.client.1.smithi067.stdout:7/782: dread - d0/d23/d80/da0/db4/d5/d4a/f108 zero size 2024-04-12T05:37:31.631 INFO:tasks.workunit.client.1.smithi067.stdout:4/662: symlink d8/da/d8b/le0 0 2024-04-12T05:37:31.639 INFO:tasks.workunit.client.1.smithi067.stdout:4/663: dwrite d8/f2b [0,4194304] 0 2024-04-12T05:37:31.639 INFO:tasks.workunit.client.1.smithi067.stdout:7/783: dwrite d0/d23/d80/da0/db4/d5/d1d/de9/fcc [0,4194304] 0 2024-04-12T05:37:31.640 INFO:tasks.workunit.client.1.smithi067.stdout:4/664: readlink d8/da/d9f/l23 0 2024-04-12T05:37:31.643 INFO:tasks.workunit.client.1.smithi067.stdout:4/665: rmdir d8/da/d8b 39 2024-04-12T05:37:31.645 INFO:tasks.workunit.client.1.smithi067.stdout:7/784: fdatasync d0/d23/d80/da0/db4/dd/d42/d7e/f30 0 2024-04-12T05:37:31.645 INFO:tasks.workunit.client.1.smithi067.stdout:4/666: write d8/fc1 [453107,81159] 0 2024-04-12T05:37:31.649 INFO:tasks.workunit.client.1.smithi067.stdout:4/667: dread d8/da/f1f [0,4194304] 0 2024-04-12T05:37:31.654 INFO:tasks.workunit.client.1.smithi067.stdout:4/668: symlink d8/da/d1d/d15/d75/le1 0 2024-04-12T05:37:31.655 INFO:tasks.workunit.client.1.smithi067.stdout:7/785: mkdir d0/d23/d80/da0/db4/d5/d4a/de5/d10c 0 2024-04-12T05:37:31.656 INFO:tasks.workunit.client.1.smithi067.stdout:4/669: chown d8/dba/d5d/d66/d7b/dbd/dcc/l70 8502228 1 2024-04-12T05:37:31.657 INFO:tasks.workunit.client.1.smithi067.stdout:4/670: chown d8/dba/d5d/d66/d71/d92 313181951 1 2024-04-12T05:37:31.660 INFO:tasks.workunit.client.1.smithi067.stdout:4/671: truncate d8/dba/d5d/d66/d7b/dbd/fca 1003621 0 2024-04-12T05:37:31.661 INFO:tasks.workunit.client.1.smithi067.stdout:4/672: fsync d8/da/d1d/d27/d91/fc0 0 2024-04-12T05:37:31.662 INFO:tasks.workunit.client.1.smithi067.stdout:7/786: rename d0/d23/d80/da0/db4/d5/d4a/de5/cf9 to d0/d23/d80/da0/db4/dd/d42/d82/dfb/c10d 0 2024-04-12T05:37:31.664 INFO:tasks.workunit.client.1.smithi067.stdout:4/673: symlink d8/da/d9f/d68/le2 0 2024-04-12T05:37:31.668 INFO:tasks.workunit.client.1.smithi067.stdout:7/787: write d0/d23/f27 [3108711,53495] 0 2024-04-12T05:37:31.671 INFO:tasks.workunit.client.1.smithi067.stdout:4/674: dwrite d8/da/d1d/d27/f86 [0,4194304] 0 2024-04-12T05:37:31.672 INFO:tasks.workunit.client.1.smithi067.stdout:7/788: dread - d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/dd6/ddf/f100 zero size 2024-04-12T05:37:31.675 INFO:tasks.workunit.client.1.smithi067.stdout:7/789: mknod d0/d23/d45/dd1/c10e 0 2024-04-12T05:37:31.678 INFO:tasks.workunit.client.1.smithi067.stdout:4/675: dwrite d8/fdd [0,4194304] 0 2024-04-12T05:37:31.679 INFO:tasks.workunit.client.1.smithi067.stdout:4/676: chown d8/da/l35 0 1 2024-04-12T05:37:31.684 INFO:tasks.workunit.client.1.smithi067.stdout:4/677: creat d8/da/d8b/d8c/fe3 x:0 0 0 2024-04-12T05:37:31.685 INFO:tasks.workunit.client.1.smithi067.stdout:7/790: dwrite d0/d23/d80/da0/db4/d5/d1d/d43/fb9 [0,4194304] 0 2024-04-12T05:37:31.687 INFO:tasks.workunit.client.1.smithi067.stdout:7/791: chown d0/d23/d80/da0/deb 1370 1 2024-04-12T05:37:31.689 INFO:tasks.workunit.client.1.smithi067.stdout:4/678: mknod d8/dba/d5d/d66/d7b/dbd/ce4 0 2024-04-12T05:37:31.695 INFO:tasks.workunit.client.1.smithi067.stdout:4/679: rename d8/da/d1d/d15/d58/cd3 to d8/da/d8b/d45/ce5 0 2024-04-12T05:37:31.695 INFO:tasks.workunit.client.1.smithi067.stdout:4/680: write f7 [997291,92704] 0 2024-04-12T05:37:31.699 INFO:tasks.workunit.client.1.smithi067.stdout:4/681: getdents d8/d39 0 2024-04-12T05:37:31.700 INFO:tasks.workunit.client.1.smithi067.stdout:4/682: fsync d8/d39/f46 0 2024-04-12T05:37:31.702 INFO:tasks.workunit.client.1.smithi067.stdout:7/792: dread d0/d23/f27 [0,4194304] 0 2024-04-12T05:37:31.703 INFO:tasks.workunit.client.1.smithi067.stdout:7/793: stat d0/d67 0 2024-04-12T05:37:31.703 INFO:tasks.workunit.client.1.smithi067.stdout:4/683: rmdir d8/dab 0 2024-04-12T05:37:31.705 INFO:tasks.workunit.client.1.smithi067.stdout:4/684: truncate d8/dba/d5d/faf 1744182 0 2024-04-12T05:37:31.707 INFO:tasks.workunit.client.1.smithi067.stdout:7/794: mknod d0/d23/d80/da0/db4/dd/d42/d71/dbd/c10f 0 2024-04-12T05:37:31.712 INFO:tasks.workunit.client.1.smithi067.stdout:7/795: creat d0/d23/d80/da0/db4/d5/d1d/f110 x:0 0 0 2024-04-12T05:37:31.716 INFO:tasks.workunit.client.1.smithi067.stdout:4/685: dwrite d8/d39/f79 [0,4194304] 0 2024-04-12T05:37:31.718 INFO:tasks.workunit.client.1.smithi067.stdout:4/686: write d8/dba/d5d/d66/d71/fda [666959,73270] 0 2024-04-12T05:37:31.720 INFO:tasks.workunit.client.1.smithi067.stdout:4/687: creat d8/da/d1d/d15/d58/fe6 x:0 0 0 2024-04-12T05:37:31.723 INFO:tasks.workunit.client.1.smithi067.stdout:4/688: mkdir d8/dba/d88/da0/de7 0 2024-04-12T05:37:31.725 INFO:tasks.workunit.client.1.smithi067.stdout:4/689: unlink d8/dba/d5d/faf 0 2024-04-12T05:37:31.729 INFO:tasks.workunit.client.1.smithi067.stdout:4/690: link d8/dba/d5d/d66/f9c d8/d9/fe8 0 2024-04-12T05:37:31.730 INFO:tasks.workunit.client.1.smithi067.stdout:4/691: write d8/d39/f79 [775810,46481] 0 2024-04-12T05:37:31.731 INFO:tasks.workunit.client.1.smithi067.stdout:4/692: truncate d8/f2b 4449648 0 2024-04-12T05:37:31.734 INFO:tasks.workunit.client.1.smithi067.stdout:4/693: mknod d8/da/d9f/dcf/ce9 0 2024-04-12T05:37:31.738 INFO:tasks.workunit.client.1.smithi067.stdout:4/694: link d8/da/d8b/d45/ce5 d8/da/d1d/d15/d75/cea 0 2024-04-12T05:37:31.740 INFO:tasks.workunit.client.1.smithi067.stdout:4/695: creat d8/dba/d5d/d66/d7b/dbd/feb x:0 0 0 2024-04-12T05:37:31.740 INFO:tasks.workunit.client.1.smithi067.stdout:4/696: dread - d8/da/fc4 zero size 2024-04-12T05:37:31.741 INFO:tasks.workunit.client.1.smithi067.stdout:4/697: chown d8/d9/d3d/f4e 117 1 2024-04-12T05:37:31.742 INFO:tasks.workunit.client.1.smithi067.stdout:4/698: mknod d8/cec 0 2024-04-12T05:37:31.743 INFO:tasks.workunit.client.1.smithi067.stdout:4/699: readlink d8/dba/d5d/d66/d7b/d6d/lc8 0 2024-04-12T05:37:31.745 INFO:tasks.workunit.client.1.smithi067.stdout:4/700: unlink d8/dba/c37 0 2024-04-12T05:37:31.746 INFO:tasks.workunit.client.1.smithi067.stdout:4/701: mkdir d8/dba/d88/ded 0 2024-04-12T05:37:31.748 INFO:tasks.workunit.client.1.smithi067.stdout:4/702: creat d8/dba/d88/da0/fee x:0 0 0 2024-04-12T05:37:31.750 INFO:tasks.workunit.client.1.smithi067.stdout:4/703: creat d8/da/d9f/d68/dbf/fef x:0 0 0 2024-04-12T05:37:31.755 INFO:tasks.workunit.client.1.smithi067.stdout:4/704: dwrite d8/da/d1d/f49 [0,4194304] 0 2024-04-12T05:37:31.758 INFO:tasks.workunit.client.1.smithi067.stdout:4/705: mknod d8/dba/d88/ded/cf0 0 2024-04-12T05:37:31.765 INFO:tasks.workunit.client.1.smithi067.stdout:4/706: rename d8/da/d1d/d15/d75/c53 to d8/dba/d5d/d66/d7b/dbd/dd1/cf1 0 2024-04-12T05:37:31.768 INFO:tasks.workunit.client.1.smithi067.stdout:4/707: dread d8/dba/d5d/d66/d7b/d6d/f81 [0,4194304] 0 2024-04-12T05:37:31.771 INFO:tasks.workunit.client.1.smithi067.stdout:4/708: dread d8/d9/f43 [0,4194304] 0 2024-04-12T05:37:31.781 INFO:tasks.workunit.client.1.smithi067.stdout:1/757: sync 2024-04-12T05:37:31.781 INFO:tasks.workunit.client.1.smithi067.stdout:6/771: sync 2024-04-12T05:37:31.781 INFO:tasks.workunit.client.1.smithi067.stdout:2/765: sync 2024-04-12T05:37:31.781 INFO:tasks.workunit.client.1.smithi067.stdout:4/709: rmdir d8/dba/d5d/d66/d7b 39 2024-04-12T05:37:31.783 INFO:tasks.workunit.client.1.smithi067.stdout:2/766: write d5/da/de/d14/f63 [4602725,85751] 0 2024-04-12T05:37:31.785 INFO:tasks.workunit.client.1.smithi067.stdout:2/767: stat d5/d6a/d78/lb1 0 2024-04-12T05:37:31.796 INFO:tasks.workunit.client.1.smithi067.stdout:4/710: dread d8/da/d1d/d27/f56 [0,4194304] 0 2024-04-12T05:37:31.796 INFO:tasks.workunit.client.1.smithi067.stdout:6/772: dread f0 [0,4194304] 0 2024-04-12T05:37:31.798 INFO:tasks.workunit.client.1.smithi067.stdout:1/758: dread d4/d9/d12/d2c/fb0 [0,4194304] 0 2024-04-12T05:37:31.799 INFO:tasks.workunit.client.1.smithi067.stdout:4/711: symlink d8/da/d1d/d15/d75/lf2 0 2024-04-12T05:37:31.804 INFO:tasks.workunit.client.1.smithi067.stdout:6/773: dwrite dd/d21/d3f/dd5/fe6 [0,4194304] 0 2024-04-12T05:37:32.068 INFO:tasks.workunit.client.1.smithi067.stdout:8/634: sync 2024-04-12T05:37:32.069 INFO:tasks.workunit.client.1.smithi067.stdout:3/632: sync 2024-04-12T05:37:32.069 INFO:tasks.workunit.client.1.smithi067.stdout:2/768: sync 2024-04-12T05:37:32.069 INFO:tasks.workunit.client.1.smithi067.stdout:9/763: sync 2024-04-12T05:37:32.069 INFO:tasks.workunit.client.1.smithi067.stdout:7/796: sync 2024-04-12T05:37:32.069 INFO:tasks.workunit.client.1.smithi067.stdout:0/824: sync 2024-04-12T05:37:32.074 INFO:tasks.workunit.client.1.smithi067.stdout:3/633: mkdir dc/df/d27/d6b/dc8/da3/dde 0 2024-04-12T05:37:32.074 INFO:tasks.workunit.client.1.smithi067.stdout:9/764: mknod d1/d28/d21/cec 0 2024-04-12T05:37:32.077 INFO:tasks.workunit.client.1.smithi067.stdout:7/797: creat d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/f111 x:0 0 0 2024-04-12T05:37:32.079 INFO:tasks.workunit.client.1.smithi067.stdout:2/769: rmdir d5/da/de/d39/dd6 0 2024-04-12T05:37:32.081 INFO:tasks.workunit.client.1.smithi067.stdout:2/770: chown d5/d6a 3936 1 2024-04-12T05:37:32.082 INFO:tasks.workunit.client.1.smithi067.stdout:8/635: write d6/d8/d18/d63/d9a/f49 [2669515,34777] 0 2024-04-12T05:37:32.083 INFO:tasks.workunit.client.1.smithi067.stdout:9/765: rename d1/d12/d76/lb1 to d1/d28/d27/led 0 2024-04-12T05:37:32.087 INFO:tasks.workunit.client.1.smithi067.stdout:3/634: dwrite dc/df/fc3 [0,4194304] 0 2024-04-12T05:37:32.088 INFO:tasks.workunit.client.1.smithi067.stdout:8/636: chown d6/d8/d18/d63/d67/d69/d5d 2 1 2024-04-12T05:37:32.090 INFO:tasks.workunit.client.1.smithi067.stdout:2/771: fsync d5/d23/d93/f98 0 2024-04-12T05:37:32.092 INFO:tasks.workunit.client.1.smithi067.stdout:0/825: dread dd/d12/d14/d51/da2/fee [0,4194304] 0 2024-04-12T05:37:32.093 INFO:tasks.workunit.client.1.smithi067.stdout:8/637: stat d6/d8/d18/d63/d67/fa6 0 2024-04-12T05:37:32.095 INFO:tasks.workunit.client.1.smithi067.stdout:7/798: dwrite d0/f8 [0,4194304] 0 2024-04-12T05:37:32.095 INFO:tasks.workunit.client.1.smithi067.stdout:9/766: creat d1/d28/d21/d45/d75/de6/d8d/dc1/fee x:0 0 0 2024-04-12T05:37:32.099 INFO:tasks.workunit.client.1.smithi067.stdout:3/635: mknod dc/df/dca/cdf 0 2024-04-12T05:37:32.101 INFO:tasks.workunit.client.1.smithi067.stdout:8/638: mkdir d6/d8/d18/d63/d9a/dbb 0 2024-04-12T05:37:32.110 INFO:tasks.workunit.client.1.smithi067.stdout:3/636: dread dc/d1d/d28/d3e/d53/f58 [0,4194304] 0 2024-04-12T05:37:32.110 INFO:tasks.workunit.client.1.smithi067.stdout:8/639: creat d6/d8/d18/d63/d9a/fbc x:0 0 0 2024-04-12T05:37:32.111 INFO:tasks.workunit.client.1.smithi067.stdout:7/799: truncate d0/d23/d80/da0/db4/d5/d1d/f40 387017 0 2024-04-12T05:37:32.114 INFO:tasks.workunit.client.1.smithi067.stdout:0/826: dread dd/d12/d14/f43 [0,4194304] 0 2024-04-12T05:37:32.114 INFO:tasks.workunit.client.1.smithi067.stdout:8/640: write d6/d86/fb9 [351770,81736] 0 2024-04-12T05:37:32.118 INFO:tasks.workunit.client.1.smithi067.stdout:8/641: unlink d6/d8/d18/f3f 0 2024-04-12T05:37:32.121 INFO:tasks.workunit.client.1.smithi067.stdout:7/800: unlink d0/d23/d80/da0/db4/dd/d79/l9a 0 2024-04-12T05:37:32.126 INFO:tasks.workunit.client.1.smithi067.stdout:0/827: creat dd/d12/d14/de3/dfd/f102 x:0 0 0 2024-04-12T05:37:32.126 INFO:tasks.workunit.client.1.smithi067.stdout:8/642: symlink d6/d86/d4f/lbd 0 2024-04-12T05:37:32.127 INFO:tasks.workunit.client.1.smithi067.stdout:9/767: dread d1/fd [0,4194304] 0 2024-04-12T05:37:32.137 INFO:tasks.workunit.client.1.smithi067.stdout:2/772: dread d5/d6a/d78/faa [0,4194304] 0 2024-04-12T05:37:32.138 INFO:tasks.workunit.client.1.smithi067.stdout:9/768: write d1/d66/d8e/fc6 [736715,61741] 0 2024-04-12T05:37:32.140 INFO:tasks.workunit.client.1.smithi067.stdout:3/637: truncate dc/d1d/d4d/f88 2150426 0 2024-04-12T05:37:32.142 INFO:tasks.workunit.client.1.smithi067.stdout:0/828: dwrite dd/d9c/d5f/f90 [0,4194304] 0 2024-04-12T05:37:32.144 INFO:tasks.workunit.client.1.smithi067.stdout:2/773: creat d5/da/dae/dba/ff6 x:0 0 0 2024-04-12T05:37:32.145 INFO:tasks.workunit.client.1.smithi067.stdout:7/801: dread d0/d23/d80/da0/db4/dd/d42/d71/fac [0,4194304] 0 2024-04-12T05:37:32.151 INFO:tasks.workunit.client.1.smithi067.stdout:9/769: creat d1/d66/dca/fef x:0 0 0 2024-04-12T05:37:32.152 INFO:tasks.workunit.client.1.smithi067.stdout:2/774: rename d5/da/d8c/dc2 to d5/da/d3d/df7 0 2024-04-12T05:37:32.158 INFO:tasks.workunit.client.1.smithi067.stdout:3/638: getdents dc/d1d/d28/d3e/d53 0 2024-04-12T05:37:32.159 INFO:tasks.workunit.client.1.smithi067.stdout:7/802: mkdir d0/d23/d80/da0/db4/d5/d1d/d43/d112 0 2024-04-12T05:37:32.161 INFO:tasks.workunit.client.1.smithi067.stdout:9/770: symlink d1/d28/d21/d45/lf0 0 2024-04-12T05:37:32.162 INFO:tasks.workunit.client.1.smithi067.stdout:1/759: sync 2024-04-12T05:37:32.163 INFO:tasks.workunit.client.1.smithi067.stdout:4/712: sync 2024-04-12T05:37:32.163 INFO:tasks.workunit.client.1.smithi067.stdout:6/774: sync 2024-04-12T05:37:32.165 INFO:tasks.workunit.client.1.smithi067.stdout:8/643: dwrite d6/fb0 [0,4194304] 0 2024-04-12T05:37:32.169 INFO:tasks.workunit.client.1.smithi067.stdout:0/829: rename dd/f13 to dd/d6f/d72/f103 0 2024-04-12T05:37:32.171 INFO:tasks.workunit.client.1.smithi067.stdout:3/639: mknod dc/ce0 0 2024-04-12T05:37:32.179 INFO:tasks.workunit.client.1.smithi067.stdout:9/771: mknod d1/d28/d21/d7f/da7/dea/cf1 0 2024-04-12T05:37:32.180 INFO:tasks.workunit.client.1.smithi067.stdout:9/772: fdatasync d1/d28/d27/fdc 0 2024-04-12T05:37:32.180 INFO:tasks.workunit.client.1.smithi067.stdout:4/713: symlink d8/da/d1d/d27/d91/lf3 0 2024-04-12T05:37:32.184 INFO:tasks.workunit.client.1.smithi067.stdout:6/775: mkdir dd/d21/d64/dcf/dfb 0 2024-04-12T05:37:32.185 INFO:tasks.workunit.client.1.smithi067.stdout:0/830: getdents dd/d12/d14/d1a/d3d/df2 0 2024-04-12T05:37:32.187 INFO:tasks.workunit.client.1.smithi067.stdout:2/775: dread d5/da/de/d14/d46/f5a [0,4194304] 0 2024-04-12T05:37:32.189 INFO:tasks.workunit.client.1.smithi067.stdout:3/640: rename dc/d1d/d28/d3e/d84/la0 to dc/df/dca/le1 0 2024-04-12T05:37:32.189 INFO:tasks.workunit.client.1.smithi067.stdout:9/773: truncate d1/d12/f8a 698172 0 2024-04-12T05:37:32.190 INFO:tasks.workunit.client.1.smithi067.stdout:4/714: truncate d8/d9/f17 2054855 0 2024-04-12T05:37:32.197 INFO:tasks.workunit.client.1.smithi067.stdout:2/776: mknod d5/da/de/dc4/dc6/cf8 0 2024-04-12T05:37:32.198 INFO:tasks.workunit.client.1.smithi067.stdout:7/803: link d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/dd6/fc4 d0/d23/d80/da0/db4/d5/d4a/de5/f113 0 2024-04-12T05:37:32.198 INFO:tasks.workunit.client.1.smithi067.stdout:6/776: chown dd/f20 133203 1 2024-04-12T05:37:32.201 INFO:tasks.workunit.client.1.smithi067.stdout:1/760: dwrite d4/d9/d12/f1b [4194304,4194304] 0 2024-04-12T05:37:32.209 INFO:tasks.workunit.client.1.smithi067.stdout:1/761: readlink d4/d9/d3a/d51/d55/lba 0 2024-04-12T05:37:32.211 INFO:tasks.workunit.client.1.smithi067.stdout:6/777: symlink dd/d21/d64/d88/d7b/dcc/lfc 0 2024-04-12T05:37:32.211 INFO:tasks.workunit.client.1.smithi067.stdout:8/644: rename d6/d8/d18/d63/cab to d6/d8/d18/d63/d9a/dbb/cbe 0 2024-04-12T05:37:32.211 INFO:tasks.workunit.client.1.smithi067.stdout:2/777: write d5/da/de/d14/d34/fb3 [3340730,55855] 0 2024-04-12T05:37:32.219 INFO:tasks.workunit.client.1.smithi067.stdout:4/715: rename d8/d9/d3d/f93 to d8/da/d1d/d27/d91/ff4 0 2024-04-12T05:37:32.221 INFO:tasks.workunit.client.1.smithi067.stdout:8/645: unlink d6/d8/d18/d36/f87 0 2024-04-12T05:37:32.226 INFO:tasks.workunit.client.1.smithi067.stdout:3/641: dwrite dc/df/d27/d6b/f77 [0,4194304] 0 2024-04-12T05:37:32.229 INFO:tasks.workunit.client.1.smithi067.stdout:7/804: dread d0/d23/d80/da0/db4/f74 [0,4194304] 0 2024-04-12T05:37:32.230 INFO:tasks.workunit.client.1.smithi067.stdout:1/762: creat d4/d9/d14/d7a/ff2 x:0 0 0 2024-04-12T05:37:32.231 INFO:tasks.workunit.client.1.smithi067.stdout:0/831: dread dd/d12/d14/d49/d9b/fb6 [0,4194304] 0 2024-04-12T05:37:32.233 INFO:tasks.workunit.client.1.smithi067.stdout:2/778: chown d5/d6a/d78/c92 7169190 1 2024-04-12T05:37:32.237 INFO:tasks.workunit.client.1.smithi067.stdout:8/646: write d6/d8/d18/d63/d9a/d12/fa9 [3859206,12339] 0 2024-04-12T05:37:32.241 INFO:tasks.workunit.client.1.smithi067.stdout:8/647: read d6/d86/fb9 [348477,10272] 0 2024-04-12T05:37:32.244 INFO:tasks.workunit.client.1.smithi067.stdout:6/778: dwrite fb [0,4194304] 0 2024-04-12T05:37:32.247 INFO:tasks.workunit.client.1.smithi067.stdout:4/716: link d8/d9/l2c d8/da/d9f/d68/db4/lf5 0 2024-04-12T05:37:32.247 INFO:tasks.workunit.client.1.smithi067.stdout:7/805: creat d0/d23/d80/da0/db4/d109/f114 x:0 0 0 2024-04-12T05:37:32.248 INFO:tasks.workunit.client.1.smithi067.stdout:1/763: symlink d4/d9/d12/dcf/lf3 0 2024-04-12T05:37:32.251 INFO:tasks.workunit.client.1.smithi067.stdout:7/806: chown d0/f1 70 1 2024-04-12T05:37:32.251 INFO:tasks.workunit.client.1.smithi067.stdout:2/779: mknod d5/da/de/da5/cf9 0 2024-04-12T05:37:32.251 INFO:tasks.workunit.client.1.smithi067.stdout:6/779: mkdir dd/d21/d64/d6f/d90/d98/dfd 0 2024-04-12T05:37:32.252 INFO:tasks.workunit.client.1.smithi067.stdout:1/764: chown d4/d9/d3a/d51 76 1 2024-04-12T05:37:32.253 INFO:tasks.workunit.client.1.smithi067.stdout:0/832: link dd/d12/d14/fc9 dd/d6f/f104 0 2024-04-12T05:37:32.258 INFO:tasks.workunit.client.1.smithi067.stdout:3/642: dwrite dc/d1d/d4d/f88 [0,4194304] 0 2024-04-12T05:37:32.258 INFO:tasks.workunit.client.1.smithi067.stdout:3/643: fsync dc/df/d27/daa/fb3 0 2024-04-12T05:37:32.259 INFO:tasks.workunit.client.1.smithi067.stdout:6/780: stat dd/d11/l85 0 2024-04-12T05:37:32.259 INFO:tasks.workunit.client.1.smithi067.stdout:4/717: symlink d8/dba/d88/lf6 0 2024-04-12T05:37:32.262 INFO:tasks.workunit.client.1.smithi067.stdout:3/644: dread - dc/faf zero size 2024-04-12T05:37:32.265 INFO:tasks.workunit.client.1.smithi067.stdout:0/833: readlink dd/d12/d14/d51/l7a 0 2024-04-12T05:37:32.265 INFO:tasks.workunit.client.1.smithi067.stdout:1/765: fdatasync d4/d9/d12/d2c/fe2 0 2024-04-12T05:37:32.268 INFO:tasks.workunit.client.1.smithi067.stdout:8/648: getdents d6/d8/d18/d36 0 2024-04-12T05:37:32.268 INFO:tasks.workunit.client.1.smithi067.stdout:2/780: unlink d5/d23/c66 0 2024-04-12T05:37:32.270 INFO:tasks.workunit.client.1.smithi067.stdout:6/781: creat dd/d11/d27/d4a/d79/dc3/ffe x:0 0 0 2024-04-12T05:37:32.271 INFO:tasks.workunit.client.1.smithi067.stdout:7/807: write d0/d23/d80/da0/db4/d5/d1d/d43/fd0 [1469210,12417] 0 2024-04-12T05:37:32.274 INFO:tasks.workunit.client.1.smithi067.stdout:6/782: chown dd/d21/d64/d6f/d90/f9d 12651 1 2024-04-12T05:37:32.276 INFO:tasks.workunit.client.1.smithi067.stdout:8/649: dread d6/d86/d4f/fa0 [0,4194304] 0 2024-04-12T05:37:32.282 INFO:tasks.workunit.client.1.smithi067.stdout:1/766: rename d4/d9/d12/dcf/lf3 to d4/d9/d4c/lf4 0 2024-04-12T05:37:32.282 INFO:tasks.workunit.client.1.smithi067.stdout:4/718: getdents d8/dba/d88/da0/de7 0 2024-04-12T05:37:32.284 INFO:tasks.workunit.client.1.smithi067.stdout:3/645: symlink dc/df/d27/dc5/dd8/le2 0 2024-04-12T05:37:32.287 INFO:tasks.workunit.client.1.smithi067.stdout:8/650: unlink d6/d8/d18/d63/d9a/f49 0 2024-04-12T05:37:32.294 INFO:tasks.workunit.client.1.smithi067.stdout:0/834: write dd/d12/d14/d49/fa6 [3139268,519] 0 2024-04-12T05:37:32.298 INFO:tasks.workunit.client.1.smithi067.stdout:2/781: dwrite d5/d11/f87 [4194304,4194304] 0 2024-04-12T05:37:32.298 INFO:tasks.workunit.client.1.smithi067.stdout:3/646: rename dc/d1d/f22 to dc/d50/da6/dbd/fe3 0 2024-04-12T05:37:32.303 INFO:tasks.workunit.client.1.smithi067.stdout:7/808: dread d0/d23/d80/da0/db4/d5/d1d/f40 [0,4194304] 0 2024-04-12T05:37:32.303 INFO:tasks.workunit.client.1.smithi067.stdout:6/783: dwrite f6 [0,4194304] 0 2024-04-12T05:37:32.305 INFO:tasks.workunit.client.1.smithi067.stdout:8/651: fsync d6/d8/d18/d63/d67/fa6 0 2024-04-12T05:37:32.305 INFO:tasks.workunit.client.1.smithi067.stdout:0/835: creat dd/d5b/f105 x:0 0 0 2024-04-12T05:37:32.309 INFO:tasks.workunit.client.1.smithi067.stdout:3/647: mkdir dc/de4 0 2024-04-12T05:37:32.310 INFO:tasks.workunit.client.1.smithi067.stdout:6/784: mkdir dd/d21/d64/db5/dff 0 2024-04-12T05:37:32.312 INFO:tasks.workunit.client.1.smithi067.stdout:0/836: write f1 [1922002,18844] 0 2024-04-12T05:37:32.312 INFO:tasks.workunit.client.1.smithi067.stdout:1/767: dwrite d4/d52/d80/f6d [0,4194304] 0 2024-04-12T05:37:32.317 INFO:tasks.workunit.client.1.smithi067.stdout:0/837: creat dd/d12/d14/d49/f106 x:0 0 0 2024-04-12T05:37:32.319 INFO:tasks.workunit.client.1.smithi067.stdout:0/838: truncate dd/d9c/d5f/d78/d83/fd7 920126 0 2024-04-12T05:37:32.320 INFO:tasks.workunit.client.1.smithi067.stdout:1/768: creat d4/d3d/d7b/ff5 x:0 0 0 2024-04-12T05:37:32.324 INFO:tasks.workunit.client.1.smithi067.stdout:6/785: mknod dd/d11/d27/d4a/d79/dc3/d56/de3/c100 0 2024-04-12T05:37:32.326 INFO:tasks.workunit.client.1.smithi067.stdout:0/839: rename dd/d5b/lc1 to dd/d6f/d72/l107 0 2024-04-12T05:37:32.327 INFO:tasks.workunit.client.1.smithi067.stdout:2/782: dread d5/dbf/dcc/fd7 [0,4194304] 0 2024-04-12T05:37:32.327 INFO:tasks.workunit.client.1.smithi067.stdout:3/648: link c7 dc/de4/ce5 0 2024-04-12T05:37:32.328 INFO:tasks.workunit.client.1.smithi067.stdout:1/769: mkdir d4/d9/d4c/df6 0 2024-04-12T05:37:32.329 INFO:tasks.workunit.client.1.smithi067.stdout:3/649: chown dc/f8e 27 1 2024-04-12T05:37:32.329 INFO:tasks.workunit.client.1.smithi067.stdout:4/719: getdents d8/dba/d5d/d66/d7b/dbd/dd1 0 2024-04-12T05:37:32.332 INFO:tasks.workunit.client.1.smithi067.stdout:0/840: read dd/d12/f16 [1508927,12362] 0 2024-04-12T05:37:32.334 INFO:tasks.workunit.client.1.smithi067.stdout:1/770: mkdir d4/d46/df7 0 2024-04-12T05:37:32.335 INFO:tasks.workunit.client.1.smithi067.stdout:7/809: dwrite d0/d23/d80/da0/db4/dd/d42/d71/dbd/fc1 [4194304,4194304] 0 2024-04-12T05:37:32.336 INFO:tasks.workunit.client.1.smithi067.stdout:8/652: dread d6/d8/d18/f41 [0,4194304] 0 2024-04-12T05:37:32.339 INFO:tasks.workunit.client.1.smithi067.stdout:1/771: readlink d4/d8f/dea/lee 0 2024-04-12T05:37:32.340 INFO:tasks.workunit.client.1.smithi067.stdout:8/653: fdatasync d6/d8/d18/d36/f5c 0 2024-04-12T05:37:32.342 INFO:tasks.workunit.client.1.smithi067.stdout:1/772: dread - d4/d9/d3a/d51/d55/fe7 zero size 2024-04-12T05:37:32.343 INFO:tasks.workunit.client.1.smithi067.stdout:1/773: dread - d4/d52/d80/fe0 zero size 2024-04-12T05:37:32.343 INFO:tasks.workunit.client.1.smithi067.stdout:3/650: mknod dc/d1d/d28/d3e/d69/ce6 0 2024-04-12T05:37:32.346 INFO:tasks.workunit.client.1.smithi067.stdout:4/720: rename d8/da/d9f/d68/d8e to d8/da/d9f/df7 0 2024-04-12T05:37:32.355 INFO:tasks.workunit.client.1.smithi067.stdout:0/841: write dd/d12/d14/d49/d9b/fa5 [3243873,127382] 0 2024-04-12T05:37:32.357 INFO:tasks.workunit.client.1.smithi067.stdout:7/810: getdents d0/d23/d45 0 2024-04-12T05:37:32.357 INFO:tasks.workunit.client.1.smithi067.stdout:1/774: dread - d4/d9/d3a/fd3 zero size 2024-04-12T05:37:32.358 INFO:tasks.workunit.client.1.smithi067.stdout:3/651: write dc/d1d/d26/f5e [2619170,85676] 0 2024-04-12T05:37:32.367 INFO:tasks.workunit.client.1.smithi067.stdout:4/721: dwrite d8/da/d1d/d27/d91/ff4 [0,4194304] 0 2024-04-12T05:37:32.367 INFO:tasks.workunit.client.1.smithi067.stdout:0/842: dwrite dd/d12/d14/d49/f4d [4194304,4194304] 0 2024-04-12T05:37:32.391 INFO:tasks.workunit.client.1.smithi067.stdout:1/775: truncate d4/d9/f4a 4393852 0 2024-04-12T05:37:32.393 INFO:tasks.workunit.client.1.smithi067.stdout:7/811: creat d0/d23/d80/da0/db4/dd/d42/d7e/d34/da3/dea/f115 x:0 0 0 2024-04-12T05:37:32.393 INFO:tasks.workunit.client.1.smithi067.stdout:3/652: mknod dc/d1d/d26/d52/ce7 0 2024-04-12T05:37:32.395 INFO:tasks.workunit.client.1.smithi067.stdout:0/843: mknod dd/d5b/c108 0 2024-04-12T05:37:32.395 INFO:tasks.workunit.client.1.smithi067.stdout:4/722: write d8/d9/d3d/f3f [44469,66183] 0 2024-04-12T05:37:32.397 INFO:tasks.workunit.client.1.smithi067.stdout:8/654: creat d6/d8/d18/d63/d9a/fbf x:0 0 0 2024-04-12T05:37:32.397 INFO:tasks.workunit.client.1.smithi067.stdout:4/723: stat d8/da/d8b/f57 0 2024-04-12T05:37:32.398 INFO:tasks.workunit.client.1.smithi067.stdout:3/653: chown dc/de4/ce5 108076996 1 2024-04-12T05:37:32.399 INFO:tasks.workunit.client.1.smithi067.stdout:8/655: read - d6/d8/d18/d63/d67/d69/f71 zero size 2024-04-12T05:37:32.399 INFO:tasks.workunit.client.1.smithi067.stdout:0/844: chown dd/d9c/f62 140678831 1 2024-04-12T05:37:32.402 INFO:tasks.workunit.client.1.smithi067.stdout:3/654: dread dc/f87 [0,4194304] 0 2024-04-12T05:37:32.402 INFO:tasks.workunit.client.1.smithi067.stdout:8/656: truncate d6/d8/d18/d63/d67/d69/f71 133897 0 2024-04-12T05:37:32.405 INFO:tasks.workunit.client.1.smithi067.stdout:0/845: fsync dd/d12/d14/d1a/fcc 0 2024-04-12T05:37:32.407 INFO:tasks.workunit.client.1.smithi067.stdout:8/657: symlink d6/d8/d6e/lc0 0 2024-04-12T05:37:32.409 INFO:tasks.workunit.client.1.smithi067.stdout:0/846: rmdir dd/d9c/d5f/d78/d83 39 2024-04-12T05:37:32.411 INFO:tasks.workunit.client.1.smithi067.stdout:8/658: unlink d6/d8/d18/d36/f9b 0 2024-04-12T05:37:32.414 INFO:tasks.workunit.client.1.smithi067.stdout:8/659: mknod d6/d86/d4f/d82/cc1 0 2024-04-12T05:37:32.417 INFO:tasks.workunit.client.1.smithi067.stdout:3/655: dread dc/f8f [0,4194304] 0 2024-04-12T05:37:32.418 INFO:tasks.workunit.client.1.smithi067.stdout:8/660: creat d6/d8/d18/d4e/d9e/fc2 x:0 0 0 2024-04-12T05:37:32.419 INFO:tasks.workunit.client.1.smithi067.stdout:8/661: read - d6/d8/d18/d4e/fa8 zero size 2024-04-12T05:37:32.420 INFO:tasks.workunit.client.1.smithi067.stdout:3/656: mkdir dc/df/d27/dc5/dd8/de8 0 2024-04-12T05:37:32.422 INFO:tasks.workunit.client.1.smithi067.stdout:8/662: mkdir d6/d8/d8b/dc3 0 2024-04-12T05:37:32.424 INFO:tasks.workunit.client.1.smithi067.stdout:1/776: truncate d4/d9/d12/d2c/f37 478676 0 2024-04-12T05:37:32.426 INFO:tasks.workunit.client.1.smithi067.stdout:3/657: fdatasync dc/d1d/d28/d3e/d53/d94/fbf 0 2024-04-12T05:37:32.427 INFO:tasks.workunit.client.1.smithi067.stdout:4/724: write d8/d39/f3b [3901860,6975] 0 2024-04-12T05:37:32.427 INFO:tasks.workunit.client.1.smithi067.stdout:8/663: mknod d6/d8/d18/d63/d9a/cc4 0 2024-04-12T05:37:32.430 INFO:tasks.workunit.client.1.smithi067.stdout:4/725: dread - d8/dba/d88/da0/fee zero size 2024-04-12T05:37:32.432 INFO:tasks.workunit.client.1.smithi067.stdout:8/664: stat d6/d8/d18/f5e 0 2024-04-12T05:37:32.432 INFO:tasks.workunit.client.1.smithi067.stdout:0/847: write dd/d6f/d72/fbb [539302,56606] 0 2024-04-12T05:37:32.434 INFO:tasks.workunit.client.1.smithi067.stdout:3/658: mknod dc/d50/da6/dbd/ce9 0 2024-04-12T05:37:32.436 INFO:tasks.workunit.client.1.smithi067.stdout:7/812: dwrite d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/dd6/f8e [0,4194304] 0 2024-04-12T05:37:32.437 INFO:tasks.workunit.client.1.smithi067.stdout:4/726: mkdir d8/dba/d5d/d66/df8 0 2024-04-12T05:37:32.438 INFO:tasks.workunit.client.1.smithi067.stdout:0/848: symlink dd/d12/d14/d51/da2/l109 0 2024-04-12T05:37:32.438 INFO:tasks.workunit.client.1.smithi067.stdout:1/777: dwrite d4/d52/d80/f6d [0,4194304] 0 2024-04-12T05:37:32.446 INFO:tasks.workunit.client.1.smithi067.stdout:7/813: dread - d0/d23/d80/da0/db4/d109/f114 zero size 2024-04-12T05:37:32.448 INFO:tasks.workunit.client.1.smithi067.stdout:3/659: rename dc/df/l1b to dc/d50/da6/lea 0 2024-04-12T05:37:32.448 INFO:tasks.workunit.client.1.smithi067.stdout:4/727: mknod d8/da/d8b/d45/cf9 0 2024-04-12T05:37:32.451 INFO:tasks.workunit.client.1.smithi067.stdout:0/849: mkdir dd/d12/d14/d1a/d10a 0 2024-04-12T05:37:32.452 INFO:tasks.workunit.client.1.smithi067.stdout:8/665: getdents d6/d8/d18/d4e/d9e 0 2024-04-12T05:37:32.452 INFO:tasks.workunit.client.1.smithi067.stdout:3/660: rmdir dc/df/d27 39 2024-04-12T05:37:32.452 INFO:tasks.workunit.client.1.smithi067.stdout:0/850: readlink dd/d12/d14/d49/l58 0 2024-04-12T05:37:32.453 INFO:tasks.workunit.client.1.smithi067.stdout:8/666: write d6/d8/d18/d36/f68 [4112947,48689] 0 2024-04-12T05:37:32.455 INFO:tasks.workunit.client.1.smithi067.stdout:3/661: dread dc/d1d/d28/d3e/d53/f58 [0,4194304] 0 2024-04-12T05:37:32.457 INFO:tasks.workunit.client.1.smithi067.stdout:0/851: write dd/d6f/fba [1293803,35115] 0 2024-04-12T05:37:32.457 INFO:tasks.workunit.client.1.smithi067.stdout:8/667: creat d6/d86/d4f/fc5 x:0 0 0 2024-04-12T05:37:32.457 INFO:tasks.workunit.client.1.smithi067.stdout:4/728: dread d8/d39/f46 [0,4194304] 0 2024-04-12T05:37:32.458 INFO:tasks.workunit.client.1.smithi067.stdout:9/774: sync 2024-04-12T05:37:32.459 INFO:tasks.workunit.client.1.smithi067.stdout:8/668: stat d6/d86/l3d 0 2024-04-12T05:37:32.460 INFO:tasks.workunit.client.1.smithi067.stdout:9/775: chown d1/d28/d21 15310917 1 2024-04-12T05:37:32.460 INFO:tasks.workunit.client.1.smithi067.stdout:4/729: rmdir d8/da/d8b/d45 39 2024-04-12T05:37:32.461 INFO:tasks.workunit.client.1.smithi067.stdout:8/669: dread - d6/d8/d18/d63/d9a/fbc zero size 2024-04-12T05:37:32.463 INFO:tasks.workunit.client.1.smithi067.stdout:1/778: dread d4/d9/d3a/d51/d55/fc8 [0,4194304] 0 2024-04-12T05:37:32.464 INFO:tasks.workunit.client.1.smithi067.stdout:9/776: mknod d1/d12/cf2 0 2024-04-12T05:37:32.464 INFO:tasks.workunit.client.1.smithi067.stdout:8/670: dread - d6/d8/d18/d4e/d9e/fc2 zero size 2024-04-12T05:37:32.465 INFO:tasks.workunit.client.1.smithi067.stdout:3/662: symlink dc/df/d27/d6b/dc8/da3/dde/leb 0 2024-04-12T05:37:32.467 INFO:tasks.workunit.client.1.smithi067.stdout:9/777: write d1/d28/d21/d3f/d4f/fb4 [7672268,112479] 0 2024-04-12T05:37:32.467 INFO:tasks.workunit.client.1.smithi067.stdout:8/671: chown d6/d8/d18/d63/d9a/d12/l37 229351 1 2024-04-12T05:37:32.467 INFO:tasks.workunit.client.1.smithi067.stdout:3/663: readlink dc/d1d/d28/d3e/l41 0 2024-04-12T05:37:32.468 INFO:tasks.workunit.client.1.smithi067.stdout:1/779: mkdir d4/d9/d12/d6f/df8 0 2024-04-12T05:37:32.470 INFO:tasks.workunit.client.1.smithi067.stdout:3/664: chown dc/d1d/d26/d52/ccb 269 1 2024-04-12T05:37:32.472 INFO:tasks.workunit.client.1.smithi067.stdout:9/778: unlink d1/d28/d21/d45/l6d 0 2024-04-12T05:37:32.476 INFO:tasks.workunit.client.1.smithi067.stdout:0/852: dwrite dd/d6f/f98 [0,4194304] 0 2024-04-12T05:37:32.478 INFO:tasks.workunit.client.1.smithi067.stdout:0/853: read - dd/d92/f76 zero size 2024-04-12T05:37:32.478 INFO:tasks.workunit.client.1.smithi067.stdout:9/779: chown d1/d28/d21/d45/d75/de6/d8d/dc1/dcb/fe1 53 1 2024-04-12T05:37:32.481 INFO:tasks.workunit.client.1.smithi067.stdout:8/672: write d6/d8/d18/d63/d67/d69/f79 [1602520,47676] 0 2024-04-12T05:37:32.481 INFO:tasks.workunit.client.1.smithi067.stdout:1/780: truncate d4/d9/d12/d2c/f88 1866025 0 2024-04-12T05:37:32.484 INFO:tasks.workunit.client.1.smithi067.stdout:9/780: write d1/d12/f53 [643929,44039] 0 2024-04-12T05:37:32.485 INFO:tasks.workunit.client.1.smithi067.stdout:0/854: mkdir dd/d6f/d72/d10b 0 2024-04-12T05:37:32.486 INFO:tasks.workunit.client.1.smithi067.stdout:9/781: chown d1/d13/db7/fd4 1878529 1 2024-04-12T05:37:32.487 INFO:tasks.workunit.client.1.smithi067.stdout:0/855: chown dd/d12/f3b 215879318 1 2024-04-12T05:37:32.490 INFO:tasks.workunit.client.1.smithi067.stdout:1/781: rename d4/d9/d4c/fc1 to d4/d9/d12/dcf/ff9 0 2024-04-12T05:37:32.491 INFO:tasks.workunit.client.1.smithi067.stdout:3/665: dwrite dc/d1d/d28/d3e/f63 [0,4194304] 0 2024-04-12T05:37:32.492 INFO:tasks.workunit.client.1.smithi067.stdout:8/673: mknod d6/d8/d8b/da1/cc6 0 2024-04-12T05:37:32.493 INFO:tasks.workunit.client.1.smithi067.stdout:1/782: write d4/d9/d14/d7a/ff2 [993824,17348] 0 2024-04-12T05:37:32.494 INFO:tasks.workunit.client.1.smithi067.stdout:0/856: unlink dd/d92/l66 0 2024-04-12T05:37:32.502 INFO:tasks.workunit.client.1.smithi067.stdout:9/782: dwrite d1/d28/d21/d3f/d4f/f60 [0,4194304] 0 2024-04-12T05:37:32.502 INFO:tasks.workunit.client.1.smithi067.stdout:8/674: rename d6/d8/d6e to d6/dc7 0 2024-04-12T05:37:32.504 INFO:tasks.workunit.client.1.smithi067.stdout:9/783: write d1/d28/d21/f9b [875374,75112] 0 2024-04-12T05:37:32.506 INFO:tasks.workunit.client.1.smithi067.stdout:0/857: rename dd/d9c/d5f/d78 to dd/d12/d14/d1a/d3d/df2/d10c 0 2024-04-12T05:37:32.508 INFO:tasks.workunit.client.1.smithi067.stdout:8/675: fdatasync d6/d8/d18/d63/d67/d69/f89 0 2024-04-12T05:37:32.510 INFO:tasks.workunit.client.1.smithi067.stdout:9/784: unlink d1/d12/c19 0 2024-04-12T05:37:32.512 INFO:tasks.workunit.client.1.smithi067.stdout:0/858: write dd/d5b/f84 [2582390,62899] 0 2024-04-12T05:37:32.516 INFO:tasks.workunit.client.1.smithi067.stdout:9/785: symlink d1/d13/lf3 0 2024-04-12T05:37:32.518 INFO:tasks.workunit.client.1.smithi067.stdout:0/859: readlink dd/d12/d14/d49/l79 0 2024-04-12T05:37:32.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:32 smithi067 ceph-mon[49010]: pgmap v277: 65 pgs: 65 active+clean; 3.2 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 94 MiB/s rd, 242 MiB/s wr, 577 op/s 2024-04-12T05:37:32.522 INFO:tasks.workunit.client.1.smithi067.stdout:9/786: write d1/d28/f18 [2815855,28846] 0 2024-04-12T05:37:32.523 INFO:tasks.workunit.client.1.smithi067.stdout:0/860: symlink dd/d5b/l10d 0 2024-04-12T05:37:32.524 INFO:tasks.workunit.client.1.smithi067.stdout:3/666: dwrite dc/df/d8d/f93 [0,4194304] 0 2024-04-12T05:37:32.526 INFO:tasks.workunit.client.1.smithi067.stdout:9/787: readlink d1/l5 0 2024-04-12T05:37:32.526 INFO:tasks.workunit.client.1.smithi067.stdout:1/783: dwrite d4/d3d/dab/f9b [0,4194304] 0 2024-04-12T05:37:32.527 INFO:tasks.workunit.client.1.smithi067.stdout:8/676: dwrite d6/d8/d18/d36/f70 [0,4194304] 0 2024-04-12T05:37:32.529 INFO:tasks.workunit.client.1.smithi067.stdout:0/861: write dd/d12/d14/d1a/d3d/f32 [2375623,32432] 0 2024-04-12T05:37:32.530 INFO:tasks.workunit.client.1.smithi067.stdout:9/788: write d1/d28/d21/d45/d75/de6/d8d/dd2/fcf [132589,116910] 0 2024-04-12T05:37:32.533 INFO:tasks.workunit.client.1.smithi067.stdout:1/784: unlink d4/d9/d3a/la2 0 2024-04-12T05:37:32.535 INFO:tasks.workunit.client.1.smithi067.stdout:0/862: fsync dd/d92/f4c 0 2024-04-12T05:37:32.537 INFO:tasks.workunit.client.1.smithi067.stdout:9/789: creat d1/d28/d21/d45/d75/db3/ff4 x:0 0 0 2024-04-12T05:37:32.542 INFO:tasks.workunit.client.1.smithi067.stdout:9/790: link d1/d13/fb9 d1/d28/d27/d32/ff5 0 2024-04-12T05:37:32.545 INFO:tasks.workunit.client.1.smithi067.stdout:1/785: mknod d4/d9/d14/d7a/cfa 0 2024-04-12T05:37:32.546 INFO:tasks.workunit.client.1.smithi067.stdout:8/677: write d6/f95 [4830467,35062] 0 2024-04-12T05:37:32.547 INFO:tasks.workunit.client.1.smithi067.stdout:9/791: write d1/d66/d8e/fc2 [8371,2696] 0 2024-04-12T05:37:32.549 INFO:tasks.workunit.client.1.smithi067.stdout:1/786: write d4/d46/f96 [4533636,6232] 0 2024-04-12T05:37:32.551 INFO:tasks.workunit.client.1.smithi067.stdout:0/863: dwrite dd/d12/d14/f43 [0,4194304] 0 2024-04-12T05:37:32.555 INFO:tasks.workunit.client.1.smithi067.stdout:0/864: read dd/d12/fb2 [5759685,68159] 0 2024-04-12T05:37:32.555 INFO:tasks.workunit.client.1.smithi067.stdout:8/678: unlink d6/d8/d18/d36/l43 0 2024-04-12T05:37:32.556 INFO:tasks.workunit.client.1.smithi067.stdout:9/792: creat d1/d28/d21/d45/d75/de6/d8d/dc1/ff6 x:0 0 0 2024-04-12T05:37:32.558 INFO:tasks.workunit.client.1.smithi067.stdout:9/793: write d1/d13/d34/f6c [182252,65381] 0 2024-04-12T05:37:32.558 INFO:tasks.workunit.client.1.smithi067.stdout:1/787: truncate d4/d52/d80/fb3 771386 0 2024-04-12T05:37:32.560 INFO:tasks.workunit.client.1.smithi067.stdout:8/679: rmdir d6/d8/d18/d63/d9a/d12 39 2024-04-12T05:37:32.561 INFO:tasks.workunit.client.1.smithi067.stdout:9/794: write d1/d28/d21/d45/d75/de6/d8d/dc1/fee [317544,61917] 0 2024-04-12T05:37:32.563 INFO:tasks.workunit.client.1.smithi067.stdout:0/865: rename dd/d12/d14/d49/la9 to dd/d6f/d72/l10e 0 2024-04-12T05:37:32.566 INFO:tasks.workunit.client.1.smithi067.stdout:8/680: symlink d6/d8/d18/d4e/d9e/lc8 0 2024-04-12T05:37:32.568 INFO:tasks.workunit.client.1.smithi067.stdout:1/788: dwrite d4/d9/d12/f81 [0,4194304] 0 2024-04-12T05:37:32.570 INFO:tasks.workunit.client.1.smithi067.stdout:3/667: dread dc/f3a [4194304,4194304] 0 2024-04-12T05:37:32.570 INFO:tasks.workunit.client.1.smithi067.stdout:9/795: truncate d1/d28/d27/fdc 62800 0 2024-04-12T05:37:32.576 INFO:tasks.workunit.client.1.smithi067.stdout:3/668: read dc/d1d/d28/d3e/f63 [2033338,34266] 0 2024-04-12T05:37:32.577 INFO:tasks.workunit.client.1.smithi067.stdout:0/866: fdatasync dd/d12/d14/d1a/d3d/f73 0 2024-04-12T05:37:32.581 INFO:tasks.workunit.client.1.smithi067.stdout:8/681: dwrite d6/d8/d18/f33 [0,4194304] 0 2024-04-12T05:37:32.583 INFO:tasks.workunit.client.1.smithi067.stdout:1/789: truncate d4/d9/d12/d6f/f8c 1081099 0 2024-04-12T05:37:32.586 INFO:tasks.workunit.client.1.smithi067.stdout:1/790: mkdir d4/d3d/d7b/dde/dfb 0 2024-04-12T05:37:32.588 INFO:tasks.workunit.client.1.smithi067.stdout:3/669: rename dc/d1d/d4d/fa5 to dc/df/d27/d6b/dc8/da3/fec 0 2024-04-12T05:37:32.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:32 smithi032 ceph-mon[39731]: pgmap v277: 65 pgs: 65 active+clean; 3.2 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 94 MiB/s rd, 242 MiB/s wr, 577 op/s 2024-04-12T05:37:32.590 INFO:tasks.workunit.client.1.smithi067.stdout:8/682: link d6/d8/d18/d4e/l7c d6/d86/d4f/d82/lc9 0 2024-04-12T05:37:32.591 INFO:tasks.workunit.client.1.smithi067.stdout:3/670: symlink dc/d1d/d28/d3e/d84/led 0 2024-04-12T05:37:32.591 INFO:tasks.workunit.client.1.smithi067.stdout:8/683: chown d6/d8/l57 0 1 2024-04-12T05:37:32.593 INFO:tasks.workunit.client.1.smithi067.stdout:8/684: symlink d6/lca 0 2024-04-12T05:37:32.593 INFO:tasks.workunit.client.1.smithi067.stdout:1/791: getdents d4/d9/d3a/d51/d55/d60 0 2024-04-12T05:37:32.595 INFO:tasks.workunit.client.1.smithi067.stdout:1/792: write d4/d9/d3a/d51/d55/f93 [3967436,84360] 0 2024-04-12T05:37:32.595 INFO:tasks.workunit.client.1.smithi067.stdout:3/671: creat dc/df/d27/d6b/dc8/da3/db9/fee x:0 0 0 2024-04-12T05:37:32.598 INFO:tasks.workunit.client.1.smithi067.stdout:3/672: chown dc/d1d/d28/d3e/d53/l7e 2534 1 2024-04-12T05:37:32.601 INFO:tasks.workunit.client.1.smithi067.stdout:1/793: rename d4/d9/d3a/d51/d55/d60/c72 to d4/d9/d12/d6f/df8/cfc 0 2024-04-12T05:37:32.604 INFO:tasks.workunit.client.1.smithi067.stdout:3/673: creat dc/df/d27/daa/fef x:0 0 0 2024-04-12T05:37:32.604 INFO:tasks.workunit.client.1.smithi067.stdout:1/794: readlink d4/d9/d3a/d51/l98 0 2024-04-12T05:37:32.605 INFO:tasks.workunit.client.1.smithi067.stdout:9/796: dwrite d1/d12/d15/fa9 [0,4194304] 0 2024-04-12T05:37:32.612 INFO:tasks.workunit.client.1.smithi067.stdout:1/795: rename d4/d9 to d4/d9/d4c/d5d/dfd 22 2024-04-12T05:37:32.613 INFO:tasks.workunit.client.1.smithi067.stdout:3/674: chown dc/df/d27/d6b/dc8/da3/cc0 1787717 1 2024-04-12T05:37:32.615 INFO:tasks.workunit.client.1.smithi067.stdout:0/867: dwrite dd/d9c/d5f/fc4 [0,4194304] 0 2024-04-12T05:37:32.616 INFO:tasks.workunit.client.1.smithi067.stdout:3/675: chown dc/d1d/d28/d3e/d8a 7 1 2024-04-12T05:37:32.616 INFO:tasks.workunit.client.1.smithi067.stdout:9/797: link d1/f79 d1/d12/ff7 0 2024-04-12T05:37:32.619 INFO:tasks.workunit.client.1.smithi067.stdout:0/868: write dd/d6f/f98 [3058764,58592] 0 2024-04-12T05:37:32.622 INFO:tasks.workunit.client.1.smithi067.stdout:8/685: dwrite d6/f83 [0,4194304] 0 2024-04-12T05:37:32.622 INFO:tasks.workunit.client.1.smithi067.stdout:0/869: chown dd/l19 9383 1 2024-04-12T05:37:32.624 INFO:tasks.workunit.client.1.smithi067.stdout:9/798: creat d1/d66/d8e/ff8 x:0 0 0 2024-04-12T05:37:32.627 INFO:tasks.workunit.client.1.smithi067.stdout:9/799: write d1/d28/d21/d68/fba [786072,87158] 0 2024-04-12T05:37:32.627 INFO:tasks.workunit.client.1.smithi067.stdout:8/686: unlink d6/fb0 0 2024-04-12T05:37:32.633 INFO:tasks.workunit.client.1.smithi067.stdout:0/870: fdatasync dd/d6f/d72/f82 0 2024-04-12T05:37:32.634 INFO:tasks.workunit.client.1.smithi067.stdout:9/800: creat d1/ff9 x:0 0 0 2024-04-12T05:37:32.634 INFO:tasks.workunit.client.1.smithi067.stdout:3/676: dwrite dc/faf [0,4194304] 0 2024-04-12T05:37:32.640 INFO:tasks.workunit.client.1.smithi067.stdout:0/871: getdents dd/d12/d14/d51/da2 0 2024-04-12T05:37:32.645 INFO:tasks.workunit.client.1.smithi067.stdout:9/801: rename d1/d28/d21/d45/d75/db3 to d1/d28/d27/dfa 0 2024-04-12T05:37:32.649 INFO:tasks.workunit.client.1.smithi067.stdout:0/872: dread dd/d6f/d72/f97 [4194304,4194304] 0 2024-04-12T05:37:32.649 INFO:tasks.workunit.client.1.smithi067.stdout:9/802: rename d1/d13/d34/fe4 to d1/d28/d21/d3f/ffb 0 2024-04-12T05:37:32.650 INFO:tasks.workunit.client.1.smithi067.stdout:3/677: link dc/d1d/d28/d3e/d53/l7e dc/d1d/d28/d3e/d53/d78/lf0 0 2024-04-12T05:37:32.651 INFO:tasks.workunit.client.1.smithi067.stdout:8/687: truncate d6/d8/d18/f35 3617793 0 2024-04-12T05:37:32.657 INFO:tasks.workunit.client.1.smithi067.stdout:9/803: unlink d1/d28/d21/d45/d75/de6/d8d/fa0 0 2024-04-12T05:37:32.657 INFO:tasks.workunit.client.1.smithi067.stdout:3/678: rmdir dc/d1d/d28 39 2024-04-12T05:37:32.658 INFO:tasks.workunit.client.1.smithi067.stdout:8/688: dread d6/d8/d18/d36/f54 [0,4194304] 0 2024-04-12T05:37:32.664 INFO:tasks.workunit.client.1.smithi067.stdout:8/689: symlink d6/d8/d18/d4e/lcb 0 2024-04-12T05:37:32.665 INFO:tasks.workunit.client.1.smithi067.stdout:8/690: write d6/d86/d4f/fc5 [369020,39832] 0 2024-04-12T05:37:32.667 INFO:tasks.workunit.client.1.smithi067.stdout:0/873: truncate dd/d12/d14/d1a/f1e 1739865 0 2024-04-12T05:37:32.669 INFO:tasks.workunit.client.1.smithi067.stdout:3/679: write dc/d1d/d28/d3e/f4a [3928549,122636] 0 2024-04-12T05:37:32.672 INFO:tasks.workunit.client.1.smithi067.stdout:0/874: rmdir dd/dbd 39 2024-04-12T05:37:32.673 INFO:tasks.workunit.client.1.smithi067.stdout:9/804: truncate d1/d66/d8e/fc6 74712 0 2024-04-12T05:37:32.676 INFO:tasks.workunit.client.1.smithi067.stdout:8/691: dwrite d6/d86/d4f/fc5 [0,4194304] 0 2024-04-12T05:37:32.677 INFO:tasks.workunit.client.1.smithi067.stdout:3/680: rename dc/df/d27/d6b to dc/d1d/d28/d3e/d53/d76/da7/df1 0 2024-04-12T05:37:32.678 INFO:tasks.workunit.client.1.smithi067.stdout:9/805: mkdir d1/d66/d8e/dfc 0 2024-04-12T05:37:32.684 INFO:tasks.workunit.client.1.smithi067.stdout:9/806: mknod d1/d13/db7/cfd 0 2024-04-12T05:37:32.684 INFO:tasks.workunit.client.1.smithi067.stdout:0/875: dwrite dd/d12/d14/ff3 [0,4194304] 0 2024-04-12T05:37:32.686 INFO:tasks.workunit.client.1.smithi067.stdout:0/876: stat dd/d12/d14/d1a/d3d/df2/d10c/la4 0 2024-04-12T05:37:32.688 INFO:tasks.workunit.client.1.smithi067.stdout:9/807: mkdir d1/d66/d7a/dfe 0 2024-04-12T05:37:32.690 INFO:tasks.workunit.client.1.smithi067.stdout:3/681: rename dc/fd1 to dc/d1d/d28/d3e/d53/d78/ff2 0 2024-04-12T05:37:32.691 INFO:tasks.workunit.client.1.smithi067.stdout:9/808: truncate d1/d66/d8e/fc2 252609 0 2024-04-12T05:37:32.696 INFO:tasks.workunit.client.1.smithi067.stdout:3/682: mkdir dc/d1d/d28/d3e/d53/d76/da7/df1/dc8/da3/df3 0 2024-04-12T05:37:32.696 INFO:tasks.workunit.client.1.smithi067.stdout:9/809: truncate d1/d13/f87 77485 0 2024-04-12T05:37:32.700 INFO:tasks.workunit.client.1.smithi067.stdout:9/810: mkdir d1/d28/d21/d45/dff 0 2024-04-12T05:37:32.702 INFO:tasks.workunit.client.1.smithi067.stdout:8/692: dread d6/d8/d18/f35 [0,4194304] 0 2024-04-12T05:37:32.705 INFO:tasks.workunit.client.1.smithi067.stdout:3/683: rename dc/d1d/d28/d3e/d8a/d98 to dc/df/df4 0 2024-04-12T05:37:32.705 INFO:tasks.workunit.client.1.smithi067.stdout:9/811: symlink d1/d66/d8e/dfc/l100 0 2024-04-12T05:37:32.707 INFO:tasks.workunit.client.1.smithi067.stdout:0/877: dread dd/d12/d14/d49/fd0 [0,4194304] 0 2024-04-12T05:37:32.709 INFO:tasks.workunit.client.1.smithi067.stdout:0/878: fdatasync dd/d12/d14/d1a/fb0 0 2024-04-12T05:37:32.712 INFO:tasks.workunit.client.1.smithi067.stdout:9/812: getdents d1/d12/d15 0 2024-04-12T05:37:32.713 INFO:tasks.workunit.client.1.smithi067.stdout:0/879: mkdir dd/d10f 0 2024-04-12T05:37:32.715 INFO:tasks.workunit.client.1.smithi067.stdout:8/693: truncate d6/d86/d4f/d82/f9d 4028546 0 2024-04-12T05:37:32.715 INFO:tasks.workunit.client.1.smithi067.stdout:0/880: fsync dd/d12/fb2 0 2024-04-12T05:37:32.716 INFO:tasks.workunit.client.1.smithi067.stdout:3/684: write dc/d1d/d28/d3e/d53/f5c [424959,130888] 0 2024-04-12T05:37:32.716 INFO:tasks.workunit.client.1.smithi067.stdout:0/881: stat f1 0 2024-04-12T05:37:32.716 INFO:tasks.workunit.client.1.smithi067.stdout:9/813: mkdir d1/d28/d21/d101 0 2024-04-12T05:37:32.720 INFO:tasks.workunit.client.1.smithi067.stdout:3/685: creat dc/df/df4/ff5 x:0 0 0 2024-04-12T05:37:32.725 INFO:tasks.workunit.client.1.smithi067.stdout:3/686: fdatasync dc/d1d/f44 0 2024-04-12T05:37:32.726 INFO:tasks.workunit.client.1.smithi067.stdout:0/882: dwrite dd/d12/d14/d1a/d3d/f32 [0,4194304] 0 2024-04-12T05:37:32.730 INFO:tasks.workunit.client.1.smithi067.stdout:3/687: stat dc/d1d/d26/d52/c7b 0 2024-04-12T05:37:32.731 INFO:tasks.workunit.client.1.smithi067.stdout:0/883: stat dd/dbd/dbe/ldb 0 2024-04-12T05:37:32.733 INFO:tasks.workunit.client.1.smithi067.stdout:9/814: dwrite d1/d12/d15/f43 [0,4194304] 0 2024-04-12T05:37:32.733 INFO:tasks.workunit.client.1.smithi067.stdout:3/688: mkdir dc/df/df4/df6 0 2024-04-12T05:37:32.735 INFO:tasks.workunit.client.1.smithi067.stdout:8/694: dread d6/dc7/f78 [0,4194304] 0 2024-04-12T05:37:32.738 INFO:tasks.workunit.client.1.smithi067.stdout:0/884: read dd/d12/d14/f8d [2399506,41896] 0 2024-04-12T05:37:32.740 INFO:tasks.workunit.client.1.smithi067.stdout:3/689: mknod dc/df/d27/dc5/dd8/de8/cf7 0 2024-04-12T05:37:32.742 INFO:tasks.workunit.client.1.smithi067.stdout:8/695: symlink d6/d8/d18/d63/d67/d69/lcc 0 2024-04-12T05:37:32.744 INFO:tasks.workunit.client.1.smithi067.stdout:3/690: dread dc/f87 [0,4194304] 0 2024-04-12T05:37:32.746 INFO:tasks.workunit.client.1.smithi067.stdout:9/815: dwrite d1/d28/d5e/fb5 [0,4194304] 0 2024-04-12T05:37:32.746 INFO:tasks.workunit.client.1.smithi067.stdout:0/885: write dd/d12/d14/d1a/f6c [176662,98204] 0 2024-04-12T05:37:32.750 INFO:tasks.workunit.client.1.smithi067.stdout:9/816: rmdir d1/d28/d21/d7f 39 2024-04-12T05:37:32.751 INFO:tasks.workunit.client.1.smithi067.stdout:0/886: stat dd/d12/d14/de3/lf4 0 2024-04-12T05:37:32.755 INFO:tasks.workunit.client.1.smithi067.stdout:0/887: getdents dd/d12/d14/d51/da2 0 2024-04-12T05:37:32.759 INFO:tasks.workunit.client.1.smithi067.stdout:9/817: link d1/d28/d21/d45/d75/de6/d8d/dc1/ff6 d1/d28/d21/d45/d75/de6/d8d/dc1/f102 0 2024-04-12T05:37:32.760 INFO:tasks.workunit.client.1.smithi067.stdout:3/691: dwrite dc/d1d/d28/d3e/f63 [0,4194304] 0 2024-04-12T05:37:32.761 INFO:tasks.workunit.client.1.smithi067.stdout:0/888: mkdir dd/d92/d110 0 2024-04-12T05:37:32.762 INFO:tasks.workunit.client.1.smithi067.stdout:8/696: dread d6/d8/d18/d36/f5c [0,4194304] 0 2024-04-12T05:37:32.765 INFO:tasks.workunit.client.1.smithi067.stdout:9/818: creat d1/d28/d21/d45/d75/de6/d8d/dc1/de3/f103 x:0 0 0 2024-04-12T05:37:32.769 INFO:tasks.workunit.client.1.smithi067.stdout:3/692: rmdir dc/df/df4 39 2024-04-12T05:37:32.769 INFO:tasks.workunit.client.1.smithi067.stdout:9/819: dread - d1/d28/d21/d3f/d4f/fc9 zero size 2024-04-12T05:37:32.769 INFO:tasks.workunit.client.1.smithi067.stdout:3/693: truncate dc/f40 1690731 0 2024-04-12T05:37:32.769 INFO:tasks.workunit.client.1.smithi067.stdout:9/820: fdatasync d1/d66/d8e/fc2 0 2024-04-12T05:37:32.771 INFO:tasks.workunit.client.1.smithi067.stdout:8/697: symlink d6/d8/d18/d63/d9a/lcd 0 2024-04-12T05:37:32.772 INFO:tasks.workunit.client.1.smithi067.stdout:9/821: creat d1/d28/d21/d101/f104 x:0 0 0 2024-04-12T05:37:32.774 INFO:tasks.workunit.client.1.smithi067.stdout:8/698: truncate d6/d8/d18/d4e/f75 733510 0 2024-04-12T05:37:32.777 INFO:tasks.workunit.client.1.smithi067.stdout:9/822: dread d1/f79 [0,4194304] 0 2024-04-12T05:37:32.778 INFO:tasks.workunit.client.1.smithi067.stdout:8/699: creat d6/d8/fce x:0 0 0 2024-04-12T05:37:32.779 INFO:tasks.workunit.client.1.smithi067.stdout:9/823: write d1/d28/d21/d45/d75/fa6 [2428811,44726] 0 2024-04-12T05:37:32.783 INFO:tasks.workunit.client.1.smithi067.stdout:3/694: dread dc/d1d/f44 [0,4194304] 0 2024-04-12T05:37:32.786 INFO:tasks.workunit.client.1.smithi067.stdout:3/695: write dc/df/d27/f96 [1431790,39156] 0 2024-04-12T05:37:32.790 INFO:tasks.workunit.client.1.smithi067.stdout:0/889: dread dd/d12/d14/d49/d9b/fa5 [0,4194304] 0 2024-04-12T05:37:32.791 INFO:tasks.workunit.client.1.smithi067.stdout:3/696: rename dc/d50/l79 to dc/d1d/d28/d3e/d53/d76/lf8 0 2024-04-12T05:37:32.793 INFO:tasks.workunit.client.1.smithi067.stdout:8/700: dwrite d6/d8/d18/d63/d9a/d12/fa9 [0,4194304] 0 2024-04-12T05:37:32.793 INFO:tasks.workunit.client.1.smithi067.stdout:3/697: write dc/df/df4/ff5 [734183,85110] 0 2024-04-12T05:37:32.794 INFO:tasks.workunit.client.1.smithi067.stdout:0/890: chown dd/d12/d14/d1a/cb8 2257787 1 2024-04-12T05:37:32.799 INFO:tasks.workunit.client.1.smithi067.stdout:9/824: dread d1/d12/f53 [0,4194304] 0 2024-04-12T05:37:32.801 INFO:tasks.workunit.client.1.smithi067.stdout:3/698: chown dc/d1d/d28/d3e/d53/d78/ff2 176 1 2024-04-12T05:37:32.804 INFO:tasks.workunit.client.1.smithi067.stdout:3/699: readlink dc/d1d/d28/d3e/l5d 0 2024-04-12T05:37:32.804 INFO:tasks.workunit.client.1.smithi067.stdout:9/825: getdents d1/d13/db7 0 2024-04-12T05:37:32.805 INFO:tasks.workunit.client.1.smithi067.stdout:8/701: dwrite d6/d8/d18/fb4 [0,4194304] 0 2024-04-12T05:37:32.813 INFO:tasks.workunit.client.1.smithi067.stdout:9/826: rename d1/d28/d27/d32 to d1/d28/d21/d45/d105 0 2024-04-12T05:37:32.813 INFO:tasks.workunit.client.1.smithi067.stdout:0/891: dwrite dd/d92/f4c [0,4194304] 0 2024-04-12T05:37:32.818 INFO:tasks.workunit.client.1.smithi067.stdout:3/700: dwrite dc/df/d27/f6a [4194304,4194304] 0 2024-04-12T05:37:32.819 INFO:tasks.workunit.client.1.smithi067.stdout:0/892: mknod dd/d9c/d64/ddd/c111 0 2024-04-12T05:37:32.821 INFO:tasks.workunit.client.1.smithi067.stdout:9/827: creat d1/d28/d21/d45/d75/de6/d8d/dd2/f106 x:0 0 0 2024-04-12T05:37:32.827 INFO:tasks.workunit.client.1.smithi067.stdout:3/701: dwrite dc/d1d/d28/f64 [0,4194304] 0 2024-04-12T05:37:32.831 INFO:tasks.workunit.client.1.smithi067.stdout:0/893: link dd/d12/d14/d49/l79 dd/d6f/l112 0 2024-04-12T05:37:32.831 INFO:tasks.workunit.client.1.smithi067.stdout:8/702: write d6/d8/d18/d63/d9a/f20 [1696605,85429] 0 2024-04-12T05:37:32.834 INFO:tasks.workunit.client.1.smithi067.stdout:3/702: fsync dc/f8f 0 2024-04-12T05:37:32.839 INFO:tasks.workunit.client.1.smithi067.stdout:3/703: mknod dc/df/dca/cf9 0 2024-04-12T05:37:32.840 INFO:tasks.workunit.client.1.smithi067.stdout:3/704: chown dc/d1d/d28/d3e/d84 352275989 1 2024-04-12T05:37:32.843 INFO:tasks.workunit.client.1.smithi067.stdout:3/705: symlink dc/d1d/d26/lfa 0 2024-04-12T05:37:32.848 INFO:tasks.workunit.client.1.smithi067.stdout:8/703: dread d6/d8/d18/f5e [0,4194304] 0 2024-04-12T05:37:32.848 INFO:tasks.workunit.client.1.smithi067.stdout:9/828: dwrite d1/d28/d21/d45/d75/de6/d8d/dc1/ff6 [0,4194304] 0 2024-04-12T05:37:32.853 INFO:tasks.workunit.client.1.smithi067.stdout:8/704: chown d6/d8/d18/d63/d9a/lcd 3235 1 2024-04-12T05:37:32.855 INFO:tasks.workunit.client.1.smithi067.stdout:0/894: dread dd/d12/d14/f8d [0,4194304] 0 2024-04-12T05:37:32.856 INFO:tasks.workunit.client.1.smithi067.stdout:9/829: creat d1/d66/d7a/f107 x:0 0 0 2024-04-12T05:37:32.857 INFO:tasks.workunit.client.1.smithi067.stdout:3/706: dwrite dc/d1d/d4d/fcd [0,4194304] 0 2024-04-12T05:37:32.860 INFO:tasks.workunit.client.1.smithi067.stdout:3/707: fdatasync dc/d1d/d28/f2a 0 2024-04-12T05:37:32.861 INFO:tasks.workunit.client.1.smithi067.stdout:0/895: write dd/d12/d14/d1a/d3d/df2/d10c/d83/fd7 [968034,91386] 0 2024-04-12T05:37:32.862 INFO:tasks.workunit.client.1.smithi067.stdout:8/705: creat d6/d8/d18/d63/d9a/fcf x:0 0 0 2024-04-12T05:37:32.867 INFO:tasks.workunit.client.1.smithi067.stdout:0/896: mknod dd/d12/d14/d1a/d10a/c113 0 2024-04-12T05:37:32.868 INFO:tasks.workunit.client.1.smithi067.stdout:0/897: stat dd/d6f/f104 0 2024-04-12T05:37:32.870 INFO:tasks.workunit.client.1.smithi067.stdout:8/706: dwrite d6/d8/d18/d63/d9a/f20 [0,4194304] 0 2024-04-12T05:37:32.872 INFO:tasks.workunit.client.1.smithi067.stdout:0/898: symlink dd/d6f/d72/d10b/l114 0 2024-04-12T05:37:32.875 INFO:tasks.workunit.client.1.smithi067.stdout:9/830: truncate d1/d12/f53 1815373 0 2024-04-12T05:37:32.875 INFO:tasks.workunit.client.1.smithi067.stdout:0/899: mknod dd/dbd/c115 0 2024-04-12T05:37:32.876 INFO:tasks.workunit.client.1.smithi067.stdout:8/707: symlink d6/d8/d18/ld0 0 2024-04-12T05:37:32.877 INFO:tasks.workunit.client.1.smithi067.stdout:8/708: dread - d6/d8/d18/d63/d9a/fb3 zero size 2024-04-12T05:37:32.877 INFO:tasks.workunit.client.1.smithi067.stdout:3/708: truncate dc/df/f30 507710 0 2024-04-12T05:37:32.878 INFO:tasks.workunit.client.1.smithi067.stdout:0/900: rmdir dd/d12/d14/d51/da2 39 2024-04-12T05:37:32.879 INFO:tasks.workunit.client.1.smithi067.stdout:0/901: truncate dd/d9c/d5f/de0/ff6 742377 0 2024-04-12T05:37:32.881 INFO:tasks.workunit.client.1.smithi067.stdout:8/709: creat d6/d8/d8b/da1/fd1 x:0 0 0 2024-04-12T05:37:32.881 INFO:tasks.workunit.client.1.smithi067.stdout:0/902: read dd/d6f/d72/f103 [3778375,44152] 0 2024-04-12T05:37:32.882 INFO:tasks.workunit.client.1.smithi067.stdout:8/710: chown d6/d8/d18/d63/d9a/cc4 72374539 1 2024-04-12T05:37:32.883 INFO:tasks.workunit.client.1.smithi067.stdout:0/903: write dd/d12/d14/fde [189658,23644] 0 2024-04-12T05:37:32.886 INFO:tasks.workunit.client.1.smithi067.stdout:9/831: getdents d1/d28/d21/d45 0 2024-04-12T05:37:32.886 INFO:tasks.workunit.client.1.smithi067.stdout:3/709: symlink dc/d1d/d26/d52/d9a/d9c/lfb 0 2024-04-12T05:37:32.891 INFO:tasks.workunit.client.1.smithi067.stdout:8/711: link d6/d8/fce d6/d86/d4f/fd2 0 2024-04-12T05:37:32.893 INFO:tasks.workunit.client.1.smithi067.stdout:3/710: unlink dc/d1d/d28/d3e/d53/d76/cae 0 2024-04-12T05:37:32.895 INFO:tasks.workunit.client.1.smithi067.stdout:0/904: creat dd/d9c/d5f/f116 x:0 0 0 2024-04-12T05:37:32.897 INFO:tasks.workunit.client.1.smithi067.stdout:8/712: mknod d6/d8/d8b/cd3 0 2024-04-12T05:37:32.898 INFO:tasks.workunit.client.1.smithi067.stdout:3/711: chown dc/d1d/d26/c47 2724559 1 2024-04-12T05:37:32.899 INFO:tasks.workunit.client.1.smithi067.stdout:3/712: fdatasync dc/faf 0 2024-04-12T05:37:32.903 INFO:tasks.workunit.client.1.smithi067.stdout:9/832: getdents d1/d28/d27 0 2024-04-12T05:37:32.903 INFO:tasks.workunit.client.1.smithi067.stdout:3/713: dread dc/df/d27/f3f [0,4194304] 0 2024-04-12T05:37:32.905 INFO:tasks.workunit.client.1.smithi067.stdout:0/905: rename f1 to dd/d92/f117 0 2024-04-12T05:37:32.911 INFO:tasks.workunit.client.1.smithi067.stdout:9/833: dwrite d1/d28/d21/d68/fba [0,4194304] 0 2024-04-12T05:37:32.912 INFO:tasks.workunit.client.1.smithi067.stdout:8/713: rename d6/d8/d18 to d6/d8/d50/dd4 0 2024-04-12T05:37:32.914 INFO:tasks.workunit.client.1.smithi067.stdout:0/906: mkdir dd/d12/d14/d49/d9b/d118 0 2024-04-12T05:37:32.917 INFO:tasks.workunit.client.1.smithi067.stdout:9/834: symlink d1/d28/d21/d45/d75/de6/d8d/dc1/l108 0 2024-04-12T05:37:32.920 INFO:tasks.workunit.client.1.smithi067.stdout:0/907: symlink dd/d5b/l119 0 2024-04-12T05:37:32.921 INFO:tasks.workunit.client.1.smithi067.stdout:8/714: dwrite d6/d8/d50/dd4/d63/d9a/d12/fa9 [0,4194304] 0 2024-04-12T05:37:32.924 INFO:tasks.workunit.client.1.smithi067.stdout:0/908: mknod dd/d12/d14/d1a/d3d/df2/d10c/d83/c11a 0 2024-04-12T05:37:32.930 INFO:tasks.workunit.client.1.smithi067.stdout:8/715: dread d6/d8/d50/dd4/d63/d67/d69/d5d/f91 [0,4194304] 0 2024-04-12T05:37:32.933 INFO:tasks.workunit.client.1.smithi067.stdout:3/714: dwrite dc/d50/da6/dbd/fe3 [0,4194304] 0 2024-04-12T05:37:32.938 INFO:tasks.workunit.client.1.smithi067.stdout:3/715: read - dc/d1d/d28/d3e/d53/d76/da7/df1/dc8/da3/db9/fee zero size 2024-04-12T05:37:32.941 INFO:tasks.workunit.client.1.smithi067.stdout:1/796: sync 2024-04-12T05:37:32.941 INFO:tasks.workunit.client.1.smithi067.stdout:6/786: sync 2024-04-12T05:37:32.941 INFO:tasks.workunit.client.1.smithi067.stdout:0/909: dwrite dd/d12/d14/d51/da2/fee [0,4194304] 0 2024-04-12T05:37:32.946 INFO:tasks.workunit.client.1.smithi067.stdout:3/716: mknod dc/d50/da6/dbd/cfc 0 2024-04-12T05:37:32.946 INFO:tasks.workunit.client.1.smithi067.stdout:7/814: sync 2024-04-12T05:37:32.946 INFO:tasks.workunit.client.1.smithi067.stdout:4/730: sync 2024-04-12T05:37:32.946 INFO:tasks.workunit.client.1.smithi067.stdout:0/910: dread - dd/ff9 zero size 2024-04-12T05:37:32.946 INFO:tasks.workunit.client.1.smithi067.stdout:2/783: sync 2024-04-12T05:37:32.947 INFO:tasks.workunit.client.1.smithi067.stdout:1/797: creat d4/d8f/dea/ffe x:0 0 0 2024-04-12T05:37:32.949 INFO:tasks.workunit.client.1.smithi067.stdout:6/787: mknod dd/df4/c101 0 2024-04-12T05:37:32.950 INFO:tasks.workunit.client.1.smithi067.stdout:2/784: write d5/da/dae/dba/ff6 [389950,61586] 0 2024-04-12T05:37:32.953 INFO:tasks.workunit.client.1.smithi067.stdout:4/731: dread d8/dba/d5d/d66/d7b/d6d/f81 [0,4194304] 0 2024-04-12T05:37:32.954 INFO:tasks.workunit.client.1.smithi067.stdout:0/911: mknod dd/dbd/dbe/c11b 0 2024-04-12T05:37:32.954 INFO:tasks.workunit.client.1.smithi067.stdout:2/785: chown d5/da/de/d39/dde 56386 1 2024-04-12T05:37:32.956 INFO:tasks.workunit.client.1.smithi067.stdout:7/815: rename d0/d23/d80/da0/db4/d5/d1d/de9/de1 to d0/d23/d80/da0/db4/d5/d1d/de9/de1/d116 22 2024-04-12T05:37:32.959 INFO:tasks.workunit.client.1.smithi067.stdout:1/798: write d4/d9/d12/d59/f64 [5083630,92993] 0 2024-04-12T05:37:32.961 INFO:tasks.workunit.client.1.smithi067.stdout:3/717: dread dc/df/df4/ff5 [0,4194304] 0 2024-04-12T05:37:32.962 INFO:tasks.workunit.client.1.smithi067.stdout:8/716: dread d6/d8/d50/dd4/d36/f68 [0,4194304] 0 2024-04-12T05:37:32.962 INFO:tasks.workunit.client.1.smithi067.stdout:6/788: creat dd/d21/d64/d6f/d90/d98/dfd/f102 x:0 0 0 2024-04-12T05:37:32.963 INFO:tasks.workunit.client.1.smithi067.stdout:0/912: link dd/d12/f3e dd/dad/dd9/f11c 0 2024-04-12T05:37:32.964 INFO:tasks.workunit.client.1.smithi067.stdout:9/835: dread d1/d28/d21/d3f/d4f/faa [0,4194304] 0 2024-04-12T05:37:32.965 INFO:tasks.workunit.client.1.smithi067.stdout:7/816: readlink d0/d23/d80/da0/db4/l15 0 2024-04-12T05:37:32.966 INFO:tasks.workunit.client.1.smithi067.stdout:2/786: symlink d5/da/de/d14/d34/d4c/lfa 0 2024-04-12T05:37:32.967 INFO:tasks.workunit.client.1.smithi067.stdout:6/789: dread - dd/d11/d27/fca zero size 2024-04-12T05:37:32.969 INFO:tasks.workunit.client.1.smithi067.stdout:7/817: readlink d0/d67/l5d 0 2024-04-12T05:37:32.969 INFO:tasks.workunit.client.1.smithi067.stdout:8/717: symlink d6/d8/d8b/dc3/ld5 0 2024-04-12T05:37:32.974 INFO:tasks.workunit.client.1.smithi067.stdout:6/790: write dd/d11/d27/ff1 [309350,12857] 0 2024-04-12T05:37:32.975 INFO:tasks.workunit.client.1.smithi067.stdout:3/718: mkdir dc/d1d/d28/d3e/d8a/dfd 0 2024-04-12T05:37:32.976 INFO:tasks.workunit.client.1.smithi067.stdout:0/913: dread dd/d12/d14/fde [0,4194304] 0 2024-04-12T05:37:32.977 INFO:tasks.workunit.client.1.smithi067.stdout:1/799: rename d4/d3d/ld4 to d4/d9/d3a/d51/d55/d60/d90/lff 0 2024-04-12T05:37:32.979 INFO:tasks.workunit.client.1.smithi067.stdout:8/718: write d6/d8/d50/dd4/d63/d9a/f90 [192272,63934] 0 2024-04-12T05:37:32.980 INFO:tasks.workunit.client.1.smithi067.stdout:7/818: mknod d0/d23/d80/da0/db4/d109/c117 0 2024-04-12T05:37:32.981 INFO:tasks.workunit.client.1.smithi067.stdout:6/791: fsync dd/d11/f19 0 2024-04-12T05:37:32.984 INFO:tasks.workunit.client.1.smithi067.stdout:6/792: fsync dd/d21/fcb 0 2024-04-12T05:37:32.984 INFO:tasks.workunit.client.1.smithi067.stdout:3/719: creat dc/d50/ffe x:0 0 0 2024-04-12T05:37:32.984 INFO:tasks.workunit.client.1.smithi067.stdout:4/732: dread d8/d39/f3b [0,4194304] 0 2024-04-12T05:37:32.985 INFO:tasks.workunit.client.1.smithi067.stdout:9/836: rmdir d1/d66/d7a/dfe 0 2024-04-12T05:37:32.985 INFO:tasks.workunit.client.1.smithi067.stdout:8/719: write d6/d8/d50/dd4/d63/d67/fa6 [662075,33832] 0 2024-04-12T05:37:32.986 INFO:tasks.workunit.client.1.smithi067.stdout:0/914: symlink dd/d5b/l11d 0 2024-04-12T05:37:32.990 INFO:tasks.workunit.client.1.smithi067.stdout:7/819: creat d0/d23/d45/dd1/f118 x:0 0 0 2024-04-12T05:37:32.991 INFO:tasks.workunit.client.1.smithi067.stdout:1/800: mkdir d4/d9/d3a/d51/d55/dad/ddd/d100 0 2024-04-12T05:37:32.992 INFO:tasks.workunit.client.1.smithi067.stdout:3/720: chown dc/d1d/d28/d3e/d69/dbc/cc1 30 1 2024-04-12T05:37:32.992 INFO:tasks.workunit.client.1.smithi067.stdout:6/793: rename dd/d11/d27/d4a/ff0 to dd/d21/d64/dcf/f103 0 2024-04-12T05:37:32.995 INFO:tasks.workunit.client.1.smithi067.stdout:2/787: creat d5/da/de/d14/d46/ffb x:0 0 0 2024-04-12T05:37:32.995 INFO:tasks.workunit.client.1.smithi067.stdout:3/721: stat dc/d1d/d28/f62 0 2024-04-12T05:37:32.998 INFO:tasks.workunit.client.1.smithi067.stdout:0/915: rename dd/d12/f21 to dd/d12/d14/d49/f11e 0 2024-04-12T05:37:33.000 INFO:tasks.workunit.client.1.smithi067.stdout:1/801: creat d4/d9/d12/d2c/f101 x:0 0 0 2024-04-12T05:37:33.003 INFO:tasks.workunit.client.1.smithi067.stdout:8/720: rename d6/d8/d50/dd4/d63/d67/d69/f71 to d6/d86/d4f/fd6 0 2024-04-12T05:37:33.006 INFO:tasks.workunit.client.1.smithi067.stdout:0/916: write dd/d12/d14/ff1 [1059614,11806] 0 2024-04-12T05:37:33.006 INFO:tasks.workunit.client.1.smithi067.stdout:7/820: readlink d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/dd6/l98 0 2024-04-12T05:37:33.007 INFO:tasks.workunit.client.1.smithi067.stdout:4/733: dwrite d8/dba/d5d/d66/d71/fcd [0,4194304] 0 2024-04-12T05:37:33.007 INFO:tasks.workunit.client.1.smithi067.stdout:2/788: read d5/da/f1d [1654322,85247] 0 2024-04-12T05:37:33.007 INFO:tasks.workunit.client.1.smithi067.stdout:6/794: symlink dd/d21/d64/d6f/d90/l104 0 2024-04-12T05:37:33.013 INFO:tasks.workunit.client.1.smithi067.stdout:3/722: creat dc/df/fff x:0 0 0 2024-04-12T05:37:33.013 INFO:tasks.workunit.client.1.smithi067.stdout:1/802: write d4/d9/d3a/d51/f82 [848159,389] 0 2024-04-12T05:37:33.018 INFO:tasks.workunit.client.1.smithi067.stdout:2/789: dread d5/da/dae/dba/ff6 [0,4194304] 0 2024-04-12T05:37:33.019 INFO:tasks.workunit.client.1.smithi067.stdout:3/723: write dc/d1d/d28/d3e/d53/d76/fb6 [374491,54412] 0 2024-04-12T05:37:33.020 INFO:tasks.workunit.client.1.smithi067.stdout:7/821: rmdir d0/d23/d45/dd1 39 2024-04-12T05:37:33.025 INFO:tasks.workunit.client.1.smithi067.stdout:8/721: rename d6/d86/d4f to d6/d8/d50/dd4/d63/d9a/dd7 0 2024-04-12T05:37:33.030 INFO:tasks.workunit.client.1.smithi067.stdout:0/917: dwrite dd/d12/d14/de3/ff8 [0,4194304] 0 2024-04-12T05:37:33.032 INFO:tasks.workunit.client.1.smithi067.stdout:7/822: write d0/d23/d45/f73 [1966302,46841] 0 2024-04-12T05:37:33.032 INFO:tasks.workunit.client.1.smithi067.stdout:3/724: creat dc/d1d/d28/d3e/d8a/f100 x:0 0 0 2024-04-12T05:37:33.034 INFO:tasks.workunit.client.1.smithi067.stdout:4/734: link d8/dba/d5d/d66/d7b/dbd/dcc/lb0 d8/dba/d5d/d66/d7b/dbd/dd1/lfa 0 2024-04-12T05:37:33.037 INFO:tasks.workunit.client.1.smithi067.stdout:8/722: chown d6/d8/d50/dd4/d63/d9a/cc4 2429728 1 2024-04-12T05:37:33.040 INFO:tasks.workunit.client.1.smithi067.stdout:9/837: dread d1/d28/d21/d45/d105/f81 [4194304,4194304] 0 2024-04-12T05:37:33.042 INFO:tasks.workunit.client.1.smithi067.stdout:3/725: unlink dc/df/l35 0 2024-04-12T05:37:33.049 INFO:tasks.workunit.client.1.smithi067.stdout:8/723: readlink d6/d8/d50/dd4/d4e/d9e/la5 0 2024-04-12T05:37:33.051 INFO:tasks.workunit.client.1.smithi067.stdout:2/790: dwrite d5/da/de/d14/d34/d4c/f5f [4194304,4194304] 0 2024-04-12T05:37:33.051 INFO:tasks.workunit.client.1.smithi067.stdout:6/795: dwrite dd/d11/d27/d4a/f8c [0,4194304] 0 2024-04-12T05:37:33.053 INFO:tasks.workunit.client.1.smithi067.stdout:3/726: mkdir dc/d101 0 2024-04-12T05:37:33.053 INFO:tasks.workunit.client.1.smithi067.stdout:1/803: dwrite d4/d9/d12/f53 [0,4194304] 0 2024-04-12T05:37:33.055 INFO:tasks.workunit.client.1.smithi067.stdout:4/735: write d8/dba/d5d/d66/fa4 [386381,97969] 0 2024-04-12T05:37:33.062 INFO:tasks.workunit.client.1.smithi067.stdout:3/727: chown dc/d1d/d28/d3e/d69/dbc 180 1 2024-04-12T05:37:33.067 INFO:tasks.workunit.client.1.smithi067.stdout:9/838: dread d1/d28/d21/d45/d75/de6/fc7 [0,4194304] 0 2024-04-12T05:37:33.067 INFO:tasks.workunit.client.1.smithi067.stdout:8/724: mkdir d6/d8/d50/dd4/d63/d9a/d12/dd8 0 2024-04-12T05:37:33.068 INFO:tasks.workunit.client.1.smithi067.stdout:4/736: write d8/da/d1d/d27/d91/fa8 [8969709,49331] 0 2024-04-12T05:37:33.069 INFO:tasks.workunit.client.1.smithi067.stdout:6/796: unlink dd/d21/d64/d6f/d90/lc6 0 2024-04-12T05:37:33.070 INFO:tasks.workunit.client.1.smithi067.stdout:0/918: dwrite dd/fd1 [0,4194304] 0 2024-04-12T05:37:33.071 INFO:tasks.workunit.client.1.smithi067.stdout:1/804: creat d4/d9/d4d/f102 x:0 0 0 2024-04-12T05:37:33.073 INFO:tasks.workunit.client.1.smithi067.stdout:4/737: stat d8/dba/d88/da0 0 2024-04-12T05:37:33.073 INFO:tasks.workunit.client.1.smithi067.stdout:7/823: dwrite d0/d23/d80/da0/db4/d109/f114 [0,4194304] 0 2024-04-12T05:37:33.083 INFO:tasks.workunit.client.1.smithi067.stdout:7/824: chown d0/d23/d80/da0/db4/d5/d1d/d43/fb9 10 1 2024-04-12T05:37:33.083 INFO:tasks.workunit.client.1.smithi067.stdout:4/738: fsync d8/d39/fb9 0 2024-04-12T05:37:33.085 INFO:tasks.workunit.client.1.smithi067.stdout:2/791: fsync d5/da/de/d14/d46/f5a 0 2024-04-12T05:37:33.086 INFO:tasks.workunit.client.1.smithi067.stdout:3/728: truncate dc/f8c 13457 0 2024-04-12T05:37:33.087 INFO:tasks.workunit.client.1.smithi067.stdout:0/919: readlink dd/d12/d14/d1a/d3d/df2/d10c/lfa 0 2024-04-12T05:37:33.088 INFO:tasks.workunit.client.1.smithi067.stdout:8/725: dread d6/d8/d50/dd4/d36/f70 [0,4194304] 0 2024-04-12T05:37:33.088 INFO:tasks.workunit.client.1.smithi067.stdout:9/839: dread d1/d28/d21/d45/d75/f7d [0,4194304] 0 2024-04-12T05:37:33.090 INFO:tasks.workunit.client.1.smithi067.stdout:9/840: readlink d1/l64 0 2024-04-12T05:37:33.092 INFO:tasks.workunit.client.1.smithi067.stdout:1/805: read d4/d52/d80/fb3 [332502,24121] 0 2024-04-12T05:37:33.094 INFO:tasks.workunit.client.1.smithi067.stdout:3/729: creat dc/df/d27/dc5/f102 x:0 0 0 2024-04-12T05:37:33.097 INFO:tasks.workunit.client.1.smithi067.stdout:6/797: link dd/d21/d64/d88/d7b/dcc/d93/lbc dd/d21/d64/l105 0 2024-04-12T05:37:33.098 INFO:tasks.workunit.client.1.smithi067.stdout:4/739: creat d8/dba/d5d/d66/d7b/d6d/ffb x:0 0 0 2024-04-12T05:37:33.098 INFO:tasks.workunit.client.1.smithi067.stdout:1/806: truncate d4/d9/d12/d2c/fb0 4202074 0 2024-04-12T05:37:33.099 INFO:tasks.workunit.client.1.smithi067.stdout:2/792: dread d5/da/d3d/f6d [0,4194304] 0 2024-04-12T05:37:33.102 INFO:tasks.workunit.client.1.smithi067.stdout:4/740: chown d8/da/d8b/d8c/fe3 972 1 2024-04-12T05:37:33.103 INFO:tasks.workunit.client.1.smithi067.stdout:2/793: write d5/da/d8c/faf [1437521,102579] 0 2024-04-12T05:37:33.106 INFO:tasks.workunit.client.1.smithi067.stdout:0/920: mkdir dd/d6f/d11f 0 2024-04-12T05:37:33.107 INFO:tasks.workunit.client.1.smithi067.stdout:2/794: stat d5/da/de/d14/f7b 0 2024-04-12T05:37:33.112 INFO:tasks.workunit.client.1.smithi067.stdout:6/798: symlink dd/d11/d27/d4a/d79/dc3/d9e/l106 0 2024-04-12T05:37:33.114 INFO:tasks.workunit.client.1.smithi067.stdout:3/730: rename dc/d1d/d28/d3e/d53/d78/cdc to dc/d1d/d28/d3e/d53/d76/da7/df1/dc8/da3/dde/c103 0 2024-04-12T05:37:33.115 INFO:tasks.workunit.client.1.smithi067.stdout:1/807: symlink d4/d9/d4c/df6/l103 0 2024-04-12T05:37:33.119 INFO:tasks.workunit.client.1.smithi067.stdout:9/841: getdents d1/d28/d21/d45/d75/de6/d8d/dc1/dcb 0 2024-04-12T05:37:33.120 INFO:tasks.workunit.client.1.smithi067.stdout:6/799: rmdir dd/d21/d64/d6f/d90/d98 39 2024-04-12T05:37:33.126 INFO:tasks.workunit.client.1.smithi067.stdout:8/726: dwrite d6/dc7/f84 [0,4194304] 0 2024-04-12T05:37:33.126 INFO:tasks.workunit.client.1.smithi067.stdout:2/795: write d5/da/de/f26 [4736930,28344] 0 2024-04-12T05:37:33.128 INFO:tasks.workunit.client.1.smithi067.stdout:4/741: creat d8/da/d1d/ffc x:0 0 0 2024-04-12T05:37:33.133 INFO:tasks.workunit.client.1.smithi067.stdout:1/808: creat d4/d9/d12/d59/f104 x:0 0 0 2024-04-12T05:37:33.136 INFO:tasks.workunit.client.1.smithi067.stdout:0/921: write dd/d12/d14/d49/dab/fcf [3014192,66119] 0 2024-04-12T05:37:33.137 INFO:tasks.workunit.client.1.smithi067.stdout:3/731: getdents dc/df/d27/daa 0 2024-04-12T05:37:33.143 INFO:tasks.workunit.client.1.smithi067.stdout:8/727: mkdir d6/d8/d50/dd4/d63/d9a/dbb/dd9 0 2024-04-12T05:37:33.143 INFO:tasks.workunit.client.1.smithi067.stdout:1/809: mknod d4/d9/d4c/df6/c105 0 2024-04-12T05:37:33.146 INFO:tasks.workunit.client.1.smithi067.stdout:0/922: symlink dd/dad/dd9/l120 0 2024-04-12T05:37:33.148 INFO:tasks.workunit.client.1.smithi067.stdout:3/732: symlink dc/d1d/d28/d3e/d8a/l104 0 2024-04-12T05:37:33.151 INFO:tasks.workunit.client.1.smithi067.stdout:9/842: dwrite d1/f6 [0,4194304] 0 2024-04-12T05:37:33.162 INFO:tasks.workunit.client.1.smithi067.stdout:3/733: write dc/df/fff [656230,73218] 0 2024-04-12T05:37:33.163 INFO:tasks.workunit.client.1.smithi067.stdout:4/742: dwrite d8/d9/f21 [0,4194304] 0 2024-04-12T05:37:33.175 INFO:tasks.workunit.client.1.smithi067.stdout:1/810: dwrite d4/d9/d12/d2c/f6c [0,4194304] 0 2024-04-12T05:37:33.175 INFO:tasks.workunit.client.1.smithi067.stdout:8/728: getdents d6/d8/d50/dd4/d63/d9a/dd7 0 2024-04-12T05:37:33.175 INFO:tasks.workunit.client.1.smithi067.stdout:0/923: dread dd/d9c/d5f/fc4 [0,4194304] 0 2024-04-12T05:37:33.176 INFO:tasks.workunit.client.1.smithi067.stdout:9/843: link d1/d13/lbc d1/d28/d21/d7f/l109 0 2024-04-12T05:37:33.179 INFO:tasks.workunit.client.1.smithi067.stdout:3/734: creat dc/d1d/d28/d3e/d53/d76/da7/df1/dc8/f105 x:0 0 0 2024-04-12T05:37:33.183 INFO:tasks.workunit.client.1.smithi067.stdout:6/800: dread dd/d11/d27/d4a/d79/f68 [0,4194304] 0 2024-04-12T05:37:33.186 INFO:tasks.workunit.client.1.smithi067.stdout:0/924: truncate dd/d12/f4e 3309854 0 2024-04-12T05:37:33.187 INFO:tasks.workunit.client.1.smithi067.stdout:4/743: creat d8/da/d1d/d15/ffd x:0 0 0 2024-04-12T05:37:33.188 INFO:tasks.workunit.client.1.smithi067.stdout:0/925: chown dd/d9c/d64/f7e 109702 1 2024-04-12T05:37:33.189 INFO:tasks.workunit.client.1.smithi067.stdout:6/801: chown dd/d11/d27/d4a/d51/lbf 336454 1 2024-04-12T05:37:33.190 INFO:tasks.workunit.client.1.smithi067.stdout:6/802: readlink dd/d21/d64/d6f/l7e 0 2024-04-12T05:37:33.191 INFO:tasks.workunit.client.1.smithi067.stdout:8/729: link d6/d8/d50/dd4/d63/d9a/l4b d6/d8/d50/dd4/d63/d67/d69/d5d/lda 0 2024-04-12T05:37:33.193 INFO:tasks.workunit.client.1.smithi067.stdout:3/735: getdents dc/d1d/d26/d52/d9a 0 2024-04-12T05:37:33.198 INFO:tasks.workunit.client.1.smithi067.stdout:0/926: creat dd/f121 x:0 0 0 2024-04-12T05:37:33.198 INFO:tasks.workunit.client.1.smithi067.stdout:8/730: creat d6/d8/d50/dd4/d63/d9a/d12/dd8/fdb x:0 0 0 2024-04-12T05:37:33.199 INFO:tasks.workunit.client.1.smithi067.stdout:3/736: symlink dc/d50/da6/l106 0 2024-04-12T05:37:33.204 INFO:tasks.workunit.client.1.smithi067.stdout:1/811: dread d4/d8f/fa0 [0,4194304] 0 2024-04-12T05:37:33.207 INFO:tasks.workunit.client.1.smithi067.stdout:3/737: dread dc/df/f30 [0,4194304] 0 2024-04-12T05:37:33.208 INFO:tasks.workunit.client.1.smithi067.stdout:1/812: dread - d4/d8f/dea/feb zero size 2024-04-12T05:37:33.211 INFO:tasks.workunit.client.1.smithi067.stdout:8/731: mknod d6/d8/d50/dd4/d63/d9a/cdc 0 2024-04-12T05:37:33.212 INFO:tasks.workunit.client.1.smithi067.stdout:1/813: write d4/d52/fc7 [910152,45387] 0 2024-04-12T05:37:33.213 INFO:tasks.workunit.client.1.smithi067.stdout:0/927: write dd/d12/f3e [1701671,113103] 0 2024-04-12T05:37:33.215 INFO:tasks.workunit.client.1.smithi067.stdout:1/814: creat d4/d9/d12/d6f/f106 x:0 0 0 2024-04-12T05:37:33.219 INFO:tasks.workunit.client.1.smithi067.stdout:3/738: creat dc/d1d/f107 x:0 0 0 2024-04-12T05:37:33.220 INFO:tasks.workunit.client.1.smithi067.stdout:0/928: mkdir dd/d12/d14/d49/d122 0 2024-04-12T05:37:33.222 INFO:tasks.workunit.client.1.smithi067.stdout:6/803: dwrite dd/d11/d27/d4a/d79/dc3/f43 [0,4194304] 0 2024-04-12T05:37:33.224 INFO:tasks.workunit.client.1.smithi067.stdout:8/732: rename d6/d8/d50/dd4/d63/d9a/l4b to d6/d8/d50/dd4/ldd 0 2024-04-12T05:37:33.225 INFO:tasks.workunit.client.1.smithi067.stdout:1/815: mknod d4/d9/d3a/d51/d55/c107 0 2024-04-12T05:37:33.227 INFO:tasks.workunit.client.1.smithi067.stdout:6/804: creat dd/d21/d64/d88/d7b/dcc/f107 x:0 0 0 2024-04-12T05:37:33.228 INFO:tasks.workunit.client.1.smithi067.stdout:4/744: dread d8/d9/d3d/f4e [0,4194304] 0 2024-04-12T05:37:33.230 INFO:tasks.workunit.client.1.smithi067.stdout:1/816: creat d4/d3d/d7b/dde/f108 x:0 0 0 2024-04-12T05:37:33.232 INFO:tasks.workunit.client.1.smithi067.stdout:8/733: dread d6/d8/f92 [0,4194304] 0 2024-04-12T05:37:33.233 INFO:tasks.workunit.client.1.smithi067.stdout:1/817: read - d4/d3d/dab/fb1 zero size 2024-04-12T05:37:33.235 INFO:tasks.workunit.client.1.smithi067.stdout:0/929: write dd/d12/f4e [448199,20980] 0 2024-04-12T05:37:33.239 INFO:tasks.workunit.client.1.smithi067.stdout:1/818: mknod d4/d9/d12/c109 0 2024-04-12T05:37:33.240 INFO:tasks.workunit.client.1.smithi067.stdout:4/745: dwrite d8/da/d1d/f85 [0,4194304] 0 2024-04-12T05:37:33.240 INFO:tasks.workunit.client.1.smithi067.stdout:0/930: unlink dd/d5b/c108 0 2024-04-12T05:37:33.241 INFO:tasks.workunit.client.1.smithi067.stdout:0/931: readlink dd/d12/l24 0 2024-04-12T05:37:33.241 INFO:tasks.workunit.client.1.smithi067.stdout:1/819: stat d4/d44/f66 0 2024-04-12T05:37:33.242 INFO:tasks.workunit.client.1.smithi067.stdout:8/734: link d6/d8/d50/cb5 d6/d8/d50/dd4/d63/d9a/d12/cde 0 2024-04-12T05:37:33.243 INFO:tasks.workunit.client.1.smithi067.stdout:0/932: dread - dd/ff9 zero size 2024-04-12T05:37:33.253 INFO:tasks.workunit.client.1.smithi067.stdout:0/933: chown dd/d12/d14/d1a/d3d/f7c 83245 1 2024-04-12T05:37:33.254 INFO:tasks.workunit.client.1.smithi067.stdout:8/735: dread d6/d8/d50/dd4/d63/d9a/f90 [0,4194304] 0 2024-04-12T05:37:33.257 INFO:tasks.workunit.client.1.smithi067.stdout:1/820: dwrite d4/d9/d14/fbc [0,4194304] 0 2024-04-12T05:37:33.261 INFO:tasks.workunit.client.1.smithi067.stdout:6/805: dread dd/f3b [0,4194304] 0 2024-04-12T05:37:33.263 INFO:tasks.workunit.client.1.smithi067.stdout:1/821: symlink d4/d9/d14/d7a/l10a 0 2024-04-12T05:37:33.265 INFO:tasks.workunit.client.1.smithi067.stdout:4/746: dread d8/da/d1d/d27/d91/fad [0,4194304] 0 2024-04-12T05:37:33.266 INFO:tasks.workunit.client.1.smithi067.stdout:6/806: mkdir dd/d21/d64/d88/d7b/dcc/d93/d108 0 2024-04-12T05:37:33.268 INFO:tasks.workunit.client.1.smithi067.stdout:1/822: mknod d4/d9/d4c/d5d/c10b 0 2024-04-12T05:37:33.271 INFO:tasks.workunit.client.1.smithi067.stdout:4/747: creat d8/da/d8b/ffe x:0 0 0 2024-04-12T05:37:33.274 INFO:tasks.workunit.client.1.smithi067.stdout:4/748: chown d8/da/d1d 2242 1 2024-04-12T05:37:33.274 INFO:tasks.workunit.client.1.smithi067.stdout:1/823: creat d4/d9/d3a/d51/d55/dad/f10c x:0 0 0 2024-04-12T05:37:33.275 INFO:tasks.workunit.client.1.smithi067.stdout:6/807: dread dd/f3b [0,4194304] 0 2024-04-12T05:37:33.277 INFO:tasks.workunit.client.1.smithi067.stdout:4/749: rmdir d8/d9 39 2024-04-12T05:37:33.278 INFO:tasks.workunit.client.1.smithi067.stdout:8/736: dwrite d6/d8/d50/dd4/d63/d9a/dd7/fd2 [0,4194304] 0 2024-04-12T05:37:33.279 INFO:tasks.workunit.client.1.smithi067.stdout:1/824: mkdir d4/d9/d12/d59/d10d 0 2024-04-12T05:37:33.279 INFO:tasks.workunit.client.1.smithi067.stdout:0/934: dread dd/d9c/f50 [0,4194304] 0 2024-04-12T05:37:33.280 INFO:tasks.workunit.client.1.smithi067.stdout:4/750: chown d8/da/l35 5051565 1 2024-04-12T05:37:33.282 INFO:tasks.workunit.client.1.smithi067.stdout:0/935: readlink dd/d12/d14/d49/l58 0 2024-04-12T05:37:33.283 INFO:tasks.workunit.client.1.smithi067.stdout:1/825: write d4/d9/d14/f57 [4299703,42123] 0 2024-04-12T05:37:33.285 INFO:tasks.workunit.client.1.smithi067.stdout:8/737: write d6/d8/d50/dd4/d36/f68 [1356382,1667] 0 2024-04-12T05:37:33.285 INFO:tasks.workunit.client.1.smithi067.stdout:4/751: write d8/da/d9f/d68/dbf/f97 [587423,39524] 0 2024-04-12T05:37:33.290 INFO:tasks.workunit.client.1.smithi067.stdout:8/738: fsync d6/dc7/f84 0 2024-04-12T05:37:33.293 INFO:tasks.workunit.client.1.smithi067.stdout:4/752: write d8/da/d1d/d27/d91/fad [1802403,48032] 0 2024-04-12T05:37:33.296 INFO:tasks.workunit.client.1.smithi067.stdout:6/808: dwrite dd/d11/d27/d4a/f8c [4194304,4194304] 0 2024-04-12T05:37:33.298 INFO:tasks.workunit.client.1.smithi067.stdout:8/739: write d6/d8/d50/dd4/d63/d9a/f90 [938643,53599] 0 2024-04-12T05:37:33.299 INFO:tasks.workunit.client.1.smithi067.stdout:0/936: rename dd/d12/d14/d1a/d3d/c69 to dd/d9c/d5f/c123 0 2024-04-12T05:37:33.300 INFO:tasks.workunit.client.1.smithi067.stdout:4/753: read d8/d9/f67 [986825,114175] 0 2024-04-12T05:37:33.302 INFO:tasks.workunit.client.1.smithi067.stdout:4/754: write d8/d39/fb9 [45099,7918] 0 2024-04-12T05:37:33.303 INFO:tasks.workunit.client.1.smithi067.stdout:8/740: unlink d6/d8/d50/dd4/d4e/c53 0 2024-04-12T05:37:33.305 INFO:tasks.workunit.client.1.smithi067.stdout:6/809: creat dd/d11/d27/d4a/d8b/f109 x:0 0 0 2024-04-12T05:37:33.316 INFO:tasks.workunit.client.1.smithi067.stdout:4/755: dwrite d8/dba/d5d/d66/d71/fde [0,4194304] 0 2024-04-12T05:37:33.319 INFO:tasks.workunit.client.1.smithi067.stdout:6/810: dwrite dd/f20 [0,4194304] 0 2024-04-12T05:37:33.323 INFO:tasks.workunit.client.1.smithi067.stdout:1/826: dread d4/d9/f4a [0,4194304] 0 2024-04-12T05:37:33.327 INFO:tasks.workunit.client.1.smithi067.stdout:4/756: mkdir d8/da/d1d/d15/d75/daa/dff 0 2024-04-12T05:37:33.328 INFO:tasks.workunit.client.1.smithi067.stdout:8/741: dwrite d6/f8a [0,4194304] 0 2024-04-12T05:37:33.328 INFO:tasks.workunit.client.1.smithi067.stdout:6/811: write dd/d21/f24 [3500508,77991] 0 2024-04-12T05:37:33.331 INFO:tasks.workunit.client.1.smithi067.stdout:8/742: write d6/dc7/f84 [342109,109208] 0 2024-04-12T05:37:33.332 INFO:tasks.workunit.client.1.smithi067.stdout:1/827: rmdir d4/d3d/d7b 39 2024-04-12T05:37:33.333 INFO:tasks.workunit.client.1.smithi067.stdout:8/743: fdatasync d6/d8/d50/dd4/d36/f68 0 2024-04-12T05:37:33.334 INFO:tasks.workunit.client.1.smithi067.stdout:4/757: mknod d8/da/d9f/dcf/c100 0 2024-04-12T05:37:33.334 INFO:tasks.workunit.client.1.smithi067.stdout:0/937: dread dd/d12/d14/d51/f5c [0,4194304] 0 2024-04-12T05:37:33.337 INFO:tasks.workunit.client.1.smithi067.stdout:8/744: write d6/d8/d8b/da1/fd1 [596721,31098] 0 2024-04-12T05:37:33.340 INFO:tasks.workunit.client.1.smithi067.stdout:4/758: mknod d8/dba/d88/c101 0 2024-04-12T05:37:33.342 INFO:tasks.workunit.client.1.smithi067.stdout:8/745: creat d6/d8/d50/dd4/d4e/d9e/fdf x:0 0 0 2024-04-12T05:37:33.345 INFO:tasks.workunit.client.1.smithi067.stdout:8/746: creat d6/d8/d50/fe0 x:0 0 0 2024-04-12T05:37:33.346 INFO:tasks.workunit.client.1.smithi067.stdout:8/747: write d6/d8/d8b/f8d [371928,98719] 0 2024-04-12T05:37:33.349 INFO:tasks.workunit.client.1.smithi067.stdout:1/828: truncate d4/fe3 4449700 0 2024-04-12T05:37:33.351 INFO:tasks.workunit.client.1.smithi067.stdout:8/748: creat d6/d8/d50/dd4/fe1 x:0 0 0 2024-04-12T05:37:33.352 INFO:tasks.workunit.client.1.smithi067.stdout:4/759: write d8/d9/f43 [369806,77383] 0 2024-04-12T05:37:33.354 INFO:tasks.workunit.client.1.smithi067.stdout:8/749: fsync d6/d8/d50/dd4/f41 0 2024-04-12T05:37:33.355 INFO:tasks.workunit.client.1.smithi067.stdout:1/829: fsync d4/d9/d3a/d51/d55/d9a/faa 0 2024-04-12T05:37:33.363 INFO:tasks.workunit.client.1.smithi067.stdout:4/760: creat d8/dba/d5d/d66/d7b/dbd/dcc/f102 x:0 0 0 2024-04-12T05:37:33.363 INFO:tasks.workunit.client.1.smithi067.stdout:6/812: dread dd/d21/d64/d6f/f70 [0,4194304] 0 2024-04-12T05:37:33.372 INFO:tasks.workunit.client.1.smithi067.stdout:6/813: dread dd/d11/d27/d4a/d51/f57 [0,4194304] 0 2024-04-12T05:37:33.375 INFO:tasks.workunit.client.1.smithi067.stdout:1/830: dwrite d4/d3d/dbf/fd5 [0,4194304] 0 2024-04-12T05:37:33.376 INFO:tasks.workunit.client.1.smithi067.stdout:4/761: dwrite d8/da/d8b/ffe [0,4194304] 0 2024-04-12T05:37:33.377 INFO:tasks.workunit.client.1.smithi067.stdout:1/831: readlink d4/d46/lc0 0 2024-04-12T05:37:33.383 INFO:tasks.workunit.client.1.smithi067.stdout:8/750: creat d6/d8/d50/dd4/d63/d9a/dd7/fe2 x:0 0 0 2024-04-12T05:37:33.384 INFO:tasks.workunit.client.1.smithi067.stdout:8/751: read - d6/d8/d50/fe0 zero size 2024-04-12T05:37:33.388 INFO:tasks.workunit.client.1.smithi067.stdout:6/814: creat dd/d21/d64/d6f/da7/f10a x:0 0 0 2024-04-12T05:37:33.389 INFO:tasks.workunit.client.1.smithi067.stdout:6/815: write dd/d11/d27/d4a/f8c [5331397,62729] 0 2024-04-12T05:37:33.390 INFO:tasks.workunit.client.1.smithi067.stdout:4/762: creat d8/dba/db8/f103 x:0 0 0 2024-04-12T05:37:33.391 INFO:tasks.workunit.client.1.smithi067.stdout:6/816: fdatasync dd/d11/f5d 0 2024-04-12T05:37:33.393 INFO:tasks.workunit.client.1.smithi067.stdout:8/752: mknod d6/d8/d50/dd4/d63/d9a/d12/dd8/ce3 0 2024-04-12T05:37:33.399 INFO:tasks.workunit.client.1.smithi067.stdout:6/817: creat dd/d21/d64/d6f/f10b x:0 0 0 2024-04-12T05:37:33.401 INFO:tasks.workunit.client.1.smithi067.stdout:4/763: dwrite d8/dba/d5d/d66/d71/fda [0,4194304] 0 2024-04-12T05:37:33.401 INFO:tasks.workunit.client.1.smithi067.stdout:8/753: truncate d6/d8/d50/dd4/d63/d67/d69/f89 156078 0 2024-04-12T05:37:33.404 INFO:tasks.workunit.client.1.smithi067.stdout:6/818: creat dd/d21/d64/d88/d7b/dcc/f10c x:0 0 0 2024-04-12T05:37:33.406 INFO:tasks.workunit.client.1.smithi067.stdout:4/764: mknod d8/dba/db8/c104 0 2024-04-12T05:37:33.408 INFO:tasks.workunit.client.1.smithi067.stdout:4/765: chown d8/da/d1d/d15/d75/le1 382017726 1 2024-04-12T05:37:33.411 INFO:tasks.workunit.client.1.smithi067.stdout:8/754: dwrite d6/d8/d50/dd4/d36/f7f [0,4194304] 0 2024-04-12T05:37:33.414 INFO:tasks.workunit.client.1.smithi067.stdout:4/766: mknod d8/c105 0 2024-04-12T05:37:33.414 INFO:tasks.workunit.client.1.smithi067.stdout:1/832: write d4/d9/d12/d2c/f50 [900066,121643] 0 2024-04-12T05:37:33.418 INFO:tasks.workunit.client.1.smithi067.stdout:1/833: creat d4/d44/f10e x:0 0 0 2024-04-12T05:37:33.418 INFO:tasks.workunit.client.1.smithi067.stdout:4/767: chown f7 527 1 2024-04-12T05:37:33.420 INFO:tasks.workunit.client.1.smithi067.stdout:1/834: read d4/d3d/dbf/fd5 [2257652,96104] 0 2024-04-12T05:37:33.420 INFO:tasks.workunit.client.1.smithi067.stdout:4/768: fsync d8/d9/f67 0 2024-04-12T05:37:33.423 INFO:tasks.workunit.client.1.smithi067.stdout:1/835: creat d4/d9/d3a/d51/f10f x:0 0 0 2024-04-12T05:37:33.424 INFO:tasks.workunit.client.1.smithi067.stdout:4/769: mknod d8/d9/d3d/da3/c106 0 2024-04-12T05:37:33.425 INFO:tasks.workunit.client.1.smithi067.stdout:4/770: chown d8/da/d9f/d68/db4/fd8 0 1 2024-04-12T05:37:33.426 INFO:tasks.workunit.client.1.smithi067.stdout:1/836: rmdir d4/d9/d4d 39 2024-04-12T05:37:33.426 INFO:tasks.workunit.client.1.smithi067.stdout:1/837: readlink d4/d9/d12/d59/l86 0 2024-04-12T05:37:33.427 INFO:tasks.workunit.client.1.smithi067.stdout:4/771: stat d8/da/d8b/d45 0 2024-04-12T05:37:33.428 INFO:tasks.workunit.client.1.smithi067.stdout:4/772: read d8/d39/f46 [3108186,109003] 0 2024-04-12T05:37:33.428 INFO:tasks.workunit.client.1.smithi067.stdout:1/838: unlink d4/d3d/ldc 0 2024-04-12T05:37:33.430 INFO:tasks.workunit.client.1.smithi067.stdout:1/839: truncate d4/d44/f10e 352649 0 2024-04-12T05:37:33.432 INFO:tasks.workunit.client.1.smithi067.stdout:4/773: creat d8/dba/d88/da0/de7/f107 x:0 0 0 2024-04-12T05:37:33.434 INFO:tasks.workunit.client.1.smithi067.stdout:4/774: chown d8/da/d1d/d15/d58/f78 7300 1 2024-04-12T05:37:33.438 INFO:tasks.workunit.client.1.smithi067.stdout:4/775: rename d8/da/d9f/d68/dbf/fef to d8/da/d1d/d15/d75/daa/f108 0 2024-04-12T05:37:33.443 INFO:tasks.workunit.client.1.smithi067.stdout:4/776: unlink d8/d9/d3d/f4e 0 2024-04-12T05:37:33.444 INFO:tasks.workunit.client.1.smithi067.stdout:1/840: dread d4/d44/f66 [0,4194304] 0 2024-04-12T05:37:33.451 INFO:tasks.workunit.client.1.smithi067.stdout:1/841: getdents d4/d9 0 2024-04-12T05:37:33.453 INFO:tasks.workunit.client.1.smithi067.stdout:1/842: mkdir d4/d8f/d110 0 2024-04-12T05:37:33.456 INFO:tasks.workunit.client.1.smithi067.stdout:4/777: dwrite d8/da/d1d/d27/d91/fc0 [0,4194304] 0 2024-04-12T05:37:33.458 INFO:tasks.workunit.client.1.smithi067.stdout:1/843: creat d4/d9/d3a/d51/d55/d60/f111 x:0 0 0 2024-04-12T05:37:33.462 INFO:tasks.workunit.client.1.smithi067.stdout:1/844: link d4/d9/d12/f16 d4/dd2/f112 0 2024-04-12T05:37:33.463 INFO:tasks.workunit.client.1.smithi067.stdout:1/845: chown d4/d44 1600174958 1 2024-04-12T05:37:33.473 INFO:tasks.workunit.client.1.smithi067.stdout:1/846: write d4/f8 [3073652,101865] 0 2024-04-12T05:37:33.475 INFO:tasks.workunit.client.1.smithi067.stdout:1/847: stat d4/d9/d4c/lb9 0 2024-04-12T05:37:33.478 INFO:tasks.workunit.client.1.smithi067.stdout:4/778: dwrite d8/da/d1d/d15/d58/f78 [0,4194304] 0 2024-04-12T05:37:33.479 INFO:tasks.workunit.client.1.smithi067.stdout:4/779: chown d8/dba/d88/lf6 503 1 2024-04-12T05:37:33.483 INFO:tasks.workunit.client.1.smithi067.stdout:4/780: creat d8/dba/d5d/d66/df8/f109 x:0 0 0 2024-04-12T05:37:33.483 INFO:tasks.workunit.client.1.smithi067.stdout:1/848: truncate d4/d9/d3a/d51/f63 5200635 0 2024-04-12T05:37:33.485 INFO:tasks.workunit.client.1.smithi067.stdout:4/781: mknod d8/da/d1d/d27/d91/c10a 0 2024-04-12T05:37:33.488 INFO:tasks.workunit.client.1.smithi067.stdout:4/782: symlink d8/dba/d5d/d66/d96/dd4/l10b 0 2024-04-12T05:37:33.492 INFO:tasks.workunit.client.1.smithi067.stdout:4/783: dread d8/d39/f46 [0,4194304] 0 2024-04-12T05:37:33.493 INFO:tasks.workunit.client.1.smithi067.stdout:4/784: fdatasync d8/dba/d5d/d66/d71/d92/fc5 0 2024-04-12T05:37:33.724 INFO:tasks.workunit.client.1.smithi067.stdout:7/825: sync 2024-04-12T05:37:33.725 INFO:tasks.workunit.client.1.smithi067.stdout:2/796: sync 2024-04-12T05:37:33.727 INFO:tasks.workunit.client.1.smithi067.stdout:2/797: unlink d5/da/f1d 0 2024-04-12T05:37:33.733 INFO:tasks.workunit.client.1.smithi067.stdout:7/826: getdents d0/d23/d80/da0/db4/dd/d42/d7e 0 2024-04-12T05:37:33.737 INFO:tasks.workunit.client.1.smithi067.stdout:7/827: mknod d0/d23/d80/da0/dc3/df7/dfc/c119 0 2024-04-12T05:37:33.739 INFO:tasks.workunit.client.1.smithi067.stdout:7/828: creat d0/d23/d80/da0/dc3/df7/dfc/f11a x:0 0 0 2024-04-12T05:37:33.740 INFO:tasks.workunit.client.1.smithi067.stdout:2/798: dread d5/d11/f29 [0,4194304] 0 2024-04-12T05:37:33.744 INFO:tasks.workunit.client.1.smithi067.stdout:2/799: getdents d5/dbf/dcc 0 2024-04-12T05:37:33.753 INFO:tasks.workunit.client.1.smithi067.stdout:7/829: dread d0/d23/d80/da0/db4/dd/d42/d7e/f65 [0,4194304] 0 2024-04-12T05:37:33.754 INFO:tasks.workunit.client.1.smithi067.stdout:7/830: chown d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf 536 1 2024-04-12T05:37:33.754 INFO:tasks.workunit.client.1.smithi067.stdout:7/831: truncate d0/d23/d45/f9e 4432849 0 2024-04-12T05:37:33.757 INFO:tasks.workunit.client.1.smithi067.stdout:7/832: mkdir d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/d11b 0 2024-04-12T05:37:33.767 INFO:tasks.workunit.client.1.smithi067.stdout:7/833: rename d0/d23/d45/dd1/cf6 to d0/d23/d80/da0/db4/dd/d42/d71/c11c 0 2024-04-12T05:37:33.781 INFO:tasks.workunit.client.1.smithi067.stdout:7/834: dread d0/d23/d80/da0/db4/d5/fe [0,4194304] 0 2024-04-12T05:37:33.799 INFO:tasks.workunit.client.1.smithi067.stdout:7/835: write d0/d67/d57/fca [183936,35167] 0 2024-04-12T05:37:33.800 INFO:tasks.workunit.client.1.smithi067.stdout:7/836: chown d0/f1 86 1 2024-04-12T05:37:33.811 INFO:tasks.workunit.client.1.smithi067.stdout:7/837: dwrite d0/d23/d80/da0/db4/dd/d79/fdd [0,4194304] 0 2024-04-12T05:37:33.817 INFO:tasks.workunit.client.1.smithi067.stdout:7/838: creat d0/d23/d80/da0/db4/d5/d4a/f11d x:0 0 0 2024-04-12T05:37:33.825 INFO:tasks.workunit.client.1.smithi067.stdout:7/839: dwrite d0/d23/d80/da0/db4/d5/d1d/de9/fba [0,4194304] 0 2024-04-12T05:37:33.829 INFO:tasks.workunit.client.1.smithi067.stdout:7/840: mknod d0/d23/d80/da0/db4/dd/d42/d7e/d34/c11e 0 2024-04-12T05:37:33.831 INFO:tasks.workunit.client.1.smithi067.stdout:7/841: dread - d0/d23/d80/da0/db4/d5/d4a/f108 zero size 2024-04-12T05:37:33.834 INFO:tasks.workunit.client.1.smithi067.stdout:7/842: mkdir d0/d23/d80/da0/db4/dd/d42/d82/d11f 0 2024-04-12T05:37:33.836 INFO:tasks.workunit.client.1.smithi067.stdout:7/843: readlink d0/d23/d80/da0/db4/dd/d42/d82/l9f 0 2024-04-12T05:37:33.838 INFO:tasks.workunit.client.1.smithi067.stdout:7/844: write d0/d23/d80/da0/db4/f52 [4762608,45767] 0 2024-04-12T05:37:33.842 INFO:tasks.workunit.client.1.smithi067.stdout:7/845: mknod d0/d23/d80/da0/db4/d5/d4a/de5/d10c/c120 0 2024-04-12T05:37:33.846 INFO:tasks.workunit.client.1.smithi067.stdout:9/844: sync 2024-04-12T05:37:33.846 INFO:tasks.workunit.client.1.smithi067.stdout:3/739: sync 2024-04-12T05:37:33.846 INFO:tasks.workunit.client.1.smithi067.stdout:7/846: creat d0/d23/d80/da0/db4/dd/da2/df8/f121 x:0 0 0 2024-04-12T05:37:33.846 INFO:tasks.workunit.client.1.smithi067.stdout:3/740: readlink la 0 2024-04-12T05:37:33.850 INFO:tasks.workunit.client.1.smithi067.stdout:3/741: creat dc/d1d/d28/d3e/d53/d76/da7/df1/f108 x:0 0 0 2024-04-12T05:37:33.852 INFO:tasks.workunit.client.1.smithi067.stdout:9/845: rename d1/d28/d21/d68 to d1/d28/d21/d45/d75/de6/d8d/d10a 0 2024-04-12T05:37:33.855 INFO:tasks.workunit.client.1.smithi067.stdout:3/742: creat dc/d1d/d28/d3e/d53/d76/da7/f109 x:0 0 0 2024-04-12T05:37:33.856 INFO:tasks.workunit.client.1.smithi067.stdout:3/743: fdatasync dc/df/d27/daa/fbe 0 2024-04-12T05:37:33.858 INFO:tasks.workunit.client.1.smithi067.stdout:9/846: rename d1/l64 to d1/d28/d5e/l10b 0 2024-04-12T05:37:33.859 INFO:tasks.workunit.client.1.smithi067.stdout:9/847: fdatasync d1/d13/f7c 0 2024-04-12T05:37:33.860 INFO:tasks.workunit.client.1.smithi067.stdout:3/744: dread dc/d50/da6/dbd/fe3 [0,4194304] 0 2024-04-12T05:37:33.863 INFO:tasks.workunit.client.1.smithi067.stdout:7/847: getdents d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3 0 2024-04-12T05:37:33.869 INFO:tasks.workunit.client.1.smithi067.stdout:9/848: getdents d1/d28/d21/d45/d75/de6 0 2024-04-12T05:37:33.870 INFO:tasks.workunit.client.1.smithi067.stdout:9/849: read d1/d28/d21/d45/d75/fa6 [512956,128643] 0 2024-04-12T05:37:33.872 INFO:tasks.workunit.client.1.smithi067.stdout:3/745: dread dc/d1d/d28/f2a [0,4194304] 0 2024-04-12T05:37:33.875 INFO:tasks.workunit.client.1.smithi067.stdout:9/850: rename d1/d12/cc5 to d1/d28/d21/d7f/da7/c10c 0 2024-04-12T05:37:33.878 INFO:tasks.workunit.client.1.smithi067.stdout:3/746: dread dc/d1d/d28/d3e/f67 [0,4194304] 0 2024-04-12T05:37:33.878 INFO:tasks.workunit.client.1.smithi067.stdout:9/851: unlink d1/d28/d5e/fb5 0 2024-04-12T05:37:33.881 INFO:tasks.workunit.client.1.smithi067.stdout:9/852: creat d1/d66/d8e/dfc/f10d x:0 0 0 2024-04-12T05:37:33.882 INFO:tasks.workunit.client.1.smithi067.stdout:3/747: creat dc/d1d/d26/d52/d9a/f10a x:0 0 0 2024-04-12T05:37:33.883 INFO:tasks.workunit.client.1.smithi067.stdout:9/853: mknod d1/d28/d21/d45/d75/de6/d8d/d10a/c10e 0 2024-04-12T05:37:33.884 INFO:tasks.workunit.client.1.smithi067.stdout:3/748: creat dc/d1d/d28/d3e/d53/d76/da7/f10b x:0 0 0 2024-04-12T05:37:33.885 INFO:tasks.workunit.client.1.smithi067.stdout:9/854: read d1/d28/d21/d45/f8b [460914,52992] 0 2024-04-12T05:37:33.885 INFO:tasks.workunit.client.1.smithi067.stdout:3/749: readlink dc/d1d/d4d/l7d 0 2024-04-12T05:37:33.886 INFO:tasks.workunit.client.1.smithi067.stdout:3/750: chown dc/d1d/d28/cc4 2 1 2024-04-12T05:37:33.890 INFO:tasks.workunit.client.1.smithi067.stdout:3/751: write dc/d1d/d28/d3e/d53/d76/da7/df1/dc8/da3/db9/fd6 [958434,47109] 0 2024-04-12T05:37:33.891 INFO:tasks.workunit.client.1.smithi067.stdout:9/855: dwrite d1/d13/f5b [0,4194304] 0 2024-04-12T05:37:33.893 INFO:tasks.workunit.client.1.smithi067.stdout:9/856: fsync d1/d66/d8e/dfc/f10d 0 2024-04-12T05:37:33.893 INFO:tasks.workunit.client.1.smithi067.stdout:3/752: dread - dc/d1d/d28/d3e/d53/d76/da7/df1/dc8/da3/fec zero size 2024-04-12T05:37:33.897 INFO:tasks.workunit.client.1.smithi067.stdout:9/857: truncate d1/d28/d21/d45/d75/de6/f9e 4498050 0 2024-04-12T05:37:33.899 INFO:tasks.workunit.client.1.smithi067.stdout:3/753: dwrite dc/d1d/d28/f64 [0,4194304] 0 2024-04-12T05:37:33.901 INFO:tasks.workunit.client.1.smithi067.stdout:9/858: rmdir d1/d28/d21/d7f/da7/dea 39 2024-04-12T05:37:33.906 INFO:tasks.workunit.client.1.smithi067.stdout:9/859: mkdir d1/d66/dca/d10f 0 2024-04-12T05:37:33.907 INFO:tasks.workunit.client.1.smithi067.stdout:9/860: write d1/d13/db7/fd4 [869104,108293] 0 2024-04-12T05:37:33.911 INFO:tasks.workunit.client.1.smithi067.stdout:3/754: rename dc/d1d/d28/d3e/d53/d76/da7/df1/dc8/da3/dde/leb to dc/d1d/d28/d3e/d69/dbc/l10c 0 2024-04-12T05:37:33.913 INFO:tasks.workunit.client.1.smithi067.stdout:9/861: chown d1/d66/d8e/cce 432292640 1 2024-04-12T05:37:33.918 INFO:tasks.workunit.client.1.smithi067.stdout:9/862: rename d1/d13/d34/f6c to d1/d28/d21/d45/dff/f110 0 2024-04-12T05:37:33.924 INFO:tasks.workunit.client.1.smithi067.stdout:9/863: creat d1/d28/d21/d101/f111 x:0 0 0 2024-04-12T05:37:33.925 INFO:tasks.workunit.client.1.smithi067.stdout:3/755: creat dc/d1d/d26/f10d x:0 0 0 2024-04-12T05:37:33.927 INFO:tasks.workunit.client.1.smithi067.stdout:3/756: dread dc/d1d/d28/d3e/f67 [0,4194304] 0 2024-04-12T05:37:33.927 INFO:tasks.workunit.client.1.smithi067.stdout:9/864: mknod d1/d28/d21/d7f/da7/c112 0 2024-04-12T05:37:33.928 INFO:tasks.workunit.client.1.smithi067.stdout:3/757: fsync dc/df/d27/f6a 0 2024-04-12T05:37:33.941 INFO:tasks.workunit.client.1.smithi067.stdout:9/865: truncate d1/d28/f18 168435 0 2024-04-12T05:37:33.942 INFO:tasks.workunit.client.1.smithi067.stdout:1/849: sync 2024-04-12T05:37:33.942 INFO:tasks.workunit.client.1.smithi067.stdout:8/755: sync 2024-04-12T05:37:33.942 INFO:tasks.workunit.client.1.smithi067.stdout:4/785: sync 2024-04-12T05:37:33.942 INFO:tasks.workunit.client.1.smithi067.stdout:6/819: sync 2024-04-12T05:37:33.942 INFO:tasks.workunit.client.1.smithi067.stdout:0/938: sync 2024-04-12T05:37:33.942 INFO:tasks.workunit.client.1.smithi067.stdout:2/800: sync 2024-04-12T05:37:33.945 INFO:tasks.workunit.client.1.smithi067.stdout:6/820: chown dd/d21/d3f/f40 7644 1 2024-04-12T05:37:33.945 INFO:tasks.workunit.client.1.smithi067.stdout:0/939: chown dd/d5b/f68 208007 1 2024-04-12T05:37:33.945 INFO:tasks.workunit.client.1.smithi067.stdout:3/758: dwrite dc/d1d/d28/d3e/f67 [0,4194304] 0 2024-04-12T05:37:33.947 INFO:tasks.workunit.client.1.smithi067.stdout:6/821: fdatasync dd/d11/dc1/ff6 0 2024-04-12T05:37:33.953 INFO:tasks.workunit.client.1.smithi067.stdout:4/786: rename d8/d9/fe8 to d8/dba/d5d/d66/d7b/f10c 0 2024-04-12T05:37:33.955 INFO:tasks.workunit.client.1.smithi067.stdout:3/759: creat dc/df/d8d/f10e x:0 0 0 2024-04-12T05:37:33.955 INFO:tasks.workunit.client.1.smithi067.stdout:4/787: read d8/d9/f67 [2322109,45206] 0 2024-04-12T05:37:33.958 INFO:tasks.workunit.client.1.smithi067.stdout:1/850: write d4/d9/d3a/d51/d55/d9a/faa [659815,36487] 0 2024-04-12T05:37:33.961 INFO:tasks.workunit.client.1.smithi067.stdout:6/822: creat dd/d21/d3f/f10d x:0 0 0 2024-04-12T05:37:33.962 INFO:tasks.workunit.client.1.smithi067.stdout:2/801: rename d5/d43/d6e/ff3 to d5/da/de/d14/d34/d4c/dbc/ffc 0 2024-04-12T05:37:33.964 INFO:tasks.workunit.client.1.smithi067.stdout:3/760: mknod dc/df/d27/dc5/c10f 0 2024-04-12T05:37:33.966 INFO:tasks.workunit.client.1.smithi067.stdout:8/756: link d6/f2c d6/d8/d50/dd4/d63/d9a/dbb/dd9/fe4 0 2024-04-12T05:37:33.967 INFO:tasks.workunit.client.1.smithi067.stdout:1/851: mkdir d4/d3d/dab/d113 0 2024-04-12T05:37:33.970 INFO:tasks.workunit.client.1.smithi067.stdout:2/802: rename d5/l21 to d5/da/d8c/ddf/lfd 0 2024-04-12T05:37:33.970 INFO:tasks.workunit.client.1.smithi067.stdout:4/788: dread f7 [0,4194304] 0 2024-04-12T05:37:33.971 INFO:tasks.workunit.client.1.smithi067.stdout:4/789: chown d8/dba/d5d/d66/d71/f9a 228089 1 2024-04-12T05:37:33.973 INFO:tasks.workunit.client.1.smithi067.stdout:0/940: creat dd/d12/d14/d1a/f124 x:0 0 0 2024-04-12T05:37:33.974 INFO:tasks.workunit.client.1.smithi067.stdout:0/941: chown dd/d5b/l63 45 1 2024-04-12T05:37:33.977 INFO:tasks.workunit.client.1.smithi067.stdout:6/823: unlink dd/d21/d64/d6f/d90/d98/lae 0 2024-04-12T05:37:33.978 INFO:tasks.workunit.client.1.smithi067.stdout:6/824: readlink dd/d21/d3f/l48 0 2024-04-12T05:37:33.979 INFO:tasks.workunit.client.1.smithi067.stdout:4/790: dread - d8/da/d1d/d15/d75/daa/fb3 zero size 2024-04-12T05:37:33.980 INFO:tasks.workunit.client.1.smithi067.stdout:8/757: dwrite d6/d8/d50/dd4/d36/f54 [0,4194304] 0 2024-04-12T05:37:33.981 INFO:tasks.workunit.client.1.smithi067.stdout:3/761: mknod dc/c110 0 2024-04-12T05:37:33.983 INFO:tasks.workunit.client.1.smithi067.stdout:1/852: mkdir d4/d52/d114 0 2024-04-12T05:37:33.983 INFO:tasks.workunit.client.1.smithi067.stdout:1/853: dread - d4/dd2/fe5 zero size 2024-04-12T05:37:33.992 INFO:tasks.workunit.client.1.smithi067.stdout:0/942: mkdir dd/d12/d125 0 2024-04-12T05:37:33.995 INFO:tasks.workunit.client.1.smithi067.stdout:1/854: chown d4/d46/l4e 239433 1 2024-04-12T05:37:33.996 INFO:tasks.workunit.client.1.smithi067.stdout:1/855: chown d4/d52/d80/fb3 1873 1 2024-04-12T05:37:33.996 INFO:tasks.workunit.client.1.smithi067.stdout:1/856: readlink d4/d9/d12/l24 0 2024-04-12T05:37:34.006 INFO:tasks.workunit.client.1.smithi067.stdout:3/762: link dc/df/d8d/fab dc/d1d/d26/d52/d9a/d9c/f111 0 2024-04-12T05:37:34.007 INFO:tasks.workunit.client.1.smithi067.stdout:6/825: write dd/d21/d64/db5/fc9 [1967492,107793] 0 2024-04-12T05:37:34.010 INFO:tasks.workunit.client.1.smithi067.stdout:2/803: dwrite d5/d11/f9b [0,4194304] 0 2024-04-12T05:37:34.013 INFO:tasks.workunit.client.1.smithi067.stdout:8/758: getdents d6/d8/d50/dd4/d63/d67/d69 0 2024-04-12T05:37:34.015 INFO:tasks.workunit.client.1.smithi067.stdout:4/791: dwrite d8/da/d9f/d68/dbf/fb7 [0,4194304] 0 2024-04-12T05:37:34.016 INFO:tasks.workunit.client.1.smithi067.stdout:0/943: rename dd/d12/d14/l96 to dd/d12/l126 0 2024-04-12T05:37:34.020 INFO:tasks.workunit.client.1.smithi067.stdout:1/857: creat d4/d9/d12/f115 x:0 0 0 2024-04-12T05:37:34.024 INFO:tasks.workunit.client.1.smithi067.stdout:6/826: unlink dd/d11/d27/d4a/d79/dc3/c6a 0 2024-04-12T05:37:34.026 INFO:tasks.workunit.client.1.smithi067.stdout:2/804: mkdir d5/da/de/d14/d34/d40/d57/dfe 0 2024-04-12T05:37:34.030 INFO:tasks.workunit.client.1.smithi067.stdout:8/759: dread d6/d8/d50/dd4/f41 [0,4194304] 0 2024-04-12T05:37:34.030 INFO:tasks.workunit.client.1.smithi067.stdout:0/944: rename dd/d12/d14/d49/c5e to dd/d6f/d72/d10b/c127 0 2024-04-12T05:37:34.034 INFO:tasks.workunit.client.1.smithi067.stdout:6/827: write dd/d11/d27/d4a/d51/f57 [1428895,64381] 0 2024-04-12T05:37:34.043 INFO:tasks.workunit.client.1.smithi067.stdout:3/763: link dc/d1d/d26/ca4 dc/d1d/d26/d52/d9a/c112 0 2024-04-12T05:37:34.044 INFO:tasks.workunit.client.1.smithi067.stdout:2/805: dread d5/d23/d3a/f6f [0,4194304] 0 2024-04-12T05:37:34.044 INFO:tasks.workunit.client.1.smithi067.stdout:7/848: sync 2024-04-12T05:37:34.045 INFO:tasks.workunit.client.1.smithi067.stdout:9/866: sync 2024-04-12T05:37:34.047 INFO:tasks.workunit.client.1.smithi067.stdout:4/792: getdents d8 0 2024-04-12T05:37:34.050 INFO:tasks.workunit.client.1.smithi067.stdout:6/828: write dd/d11/d27/d4a/d51/fd0 [898601,110468] 0 2024-04-12T05:37:34.053 INFO:tasks.workunit.client.1.smithi067.stdout:1/858: truncate d4/d9/d3a/d51/d55/f93 2119409 0 2024-04-12T05:37:34.054 INFO:tasks.workunit.client.1.smithi067.stdout:9/867: write d1/d28/d21/d45/d75/f80 [22202,56297] 0 2024-04-12T05:37:34.056 INFO:tasks.workunit.client.1.smithi067.stdout:3/764: truncate f9 3326152 0 2024-04-12T05:37:34.062 INFO:tasks.workunit.client.1.smithi067.stdout:7/849: creat d0/f122 x:0 0 0 2024-04-12T05:37:34.064 INFO:tasks.workunit.client.1.smithi067.stdout:0/945: dread dd/dad/dd9/f11c [0,4194304] 0 2024-04-12T05:37:34.066 INFO:tasks.workunit.client.1.smithi067.stdout:8/760: dread d6/d8/d50/dd4/d63/d9a/d12/fa9 [0,4194304] 0 2024-04-12T05:37:34.068 INFO:tasks.workunit.client.1.smithi067.stdout:4/793: rmdir d8/dba/d5d/d66/d7b/d6d 39 2024-04-12T05:37:34.071 INFO:tasks.workunit.client.1.smithi067.stdout:7/850: chown d0/d23/d80/da0/db4/dd/d42/d7e/d34/c11e 220063169 1 2024-04-12T05:37:34.072 INFO:tasks.workunit.client.1.smithi067.stdout:1/859: symlink d4/d9/d12/dcf/l116 0 2024-04-12T05:37:34.073 INFO:tasks.workunit.client.1.smithi067.stdout:7/851: readlink d0/l1b 0 2024-04-12T05:37:34.075 INFO:tasks.workunit.client.1.smithi067.stdout:2/806: link d5/da/de/d14/d34/d40/d57/l81 d5/da/de/d14/d34/d4c/lff 0 2024-04-12T05:37:34.077 INFO:tasks.workunit.client.1.smithi067.stdout:6/829: creat dd/d21/d64/d88/d7b/dcd/f10e x:0 0 0 2024-04-12T05:37:34.077 INFO:tasks.workunit.client.1.smithi067.stdout:2/807: chown d5/d6a/d78/fd0 648 1 2024-04-12T05:37:34.078 INFO:tasks.workunit.client.1.smithi067.stdout:2/808: stat d5/da/de/dc4/dc6/ddc 0 2024-04-12T05:37:34.079 INFO:tasks.workunit.client.1.smithi067.stdout:3/765: rename dc/d1d/d4d to dc/d1d/d28/d3e/d69/d113 0 2024-04-12T05:37:34.084 INFO:tasks.workunit.client.1.smithi067.stdout:9/868: dread d1/f47 [0,4194304] 0 2024-04-12T05:37:34.085 INFO:tasks.workunit.client.1.smithi067.stdout:0/946: creat dd/d9c/f128 x:0 0 0 2024-04-12T05:37:34.090 INFO:tasks.workunit.client.1.smithi067.stdout:2/809: rmdir d5/da/de/d14/d34/d4c/dbc 39 2024-04-12T05:37:34.090 INFO:tasks.workunit.client.1.smithi067.stdout:6/830: creat dd/d11/d27/f10f x:0 0 0 2024-04-12T05:37:34.091 INFO:tasks.workunit.client.1.smithi067.stdout:3/766: write dc/d1d/d28/d3e/d8a/f95 [386076,37026] 0 2024-04-12T05:37:34.092 INFO:tasks.workunit.client.1.smithi067.stdout:8/761: rename d6/d86/lac to d6/d8/d50/le5 0 2024-04-12T05:37:34.096 INFO:tasks.workunit.client.1.smithi067.stdout:8/762: write d6/d8/d50/dd4/d63/d9a/d12/dd8/fdb [426090,70647] 0 2024-04-12T05:37:34.099 INFO:tasks.workunit.client.1.smithi067.stdout:9/869: unlink d1/d28/d21/f9b 0 2024-04-12T05:37:34.105 INFO:tasks.workunit.client.1.smithi067.stdout:2/810: rmdir d5/d6a/d8f 39 2024-04-12T05:37:34.112 INFO:tasks.workunit.client.1.smithi067.stdout:7/852: dwrite d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/dd6/ddf/fe2 [0,4194304] 0 2024-04-12T05:37:34.114 INFO:tasks.workunit.client.1.smithi067.stdout:6/831: symlink dd/d11/d27/d4a/dc7/l110 0 2024-04-12T05:37:34.122 INFO:tasks.workunit.client.1.smithi067.stdout:2/811: mknod d5/d23/d93/dd1/c100 0 2024-04-12T05:37:34.124 INFO:tasks.workunit.client.1.smithi067.stdout:2/812: stat d5/d23/d3a/f6f 0 2024-04-12T05:37:34.128 INFO:tasks.workunit.client.1.smithi067.stdout:3/767: fsync dc/d1d/f43 0 2024-04-12T05:37:34.128 INFO:tasks.workunit.client.1.smithi067.stdout:8/763: dwrite d6/d8/d50/dd4/d63/d9a/d12/fa9 [4194304,4194304] 0 2024-04-12T05:37:34.133 INFO:tasks.workunit.client.1.smithi067.stdout:7/853: creat d0/d23/d80/da0/db4/dd/f123 x:0 0 0 2024-04-12T05:37:34.134 INFO:tasks.workunit.client.1.smithi067.stdout:8/764: truncate d6/f83 4806328 0 2024-04-12T05:37:34.135 INFO:tasks.workunit.client.1.smithi067.stdout:6/832: dwrite dd/d11/dc1/ff6 [0,4194304] 0 2024-04-12T05:37:34.137 INFO:tasks.workunit.client.1.smithi067.stdout:3/768: fsync dc/d1d/d28/d3e/d69/d113/fcd 0 2024-04-12T05:37:34.140 INFO:tasks.workunit.client.1.smithi067.stdout:6/833: stat dd/d21/d64/d88/d7b/dcd 0 2024-04-12T05:37:34.141 INFO:tasks.workunit.client.1.smithi067.stdout:0/947: dwrite dd/d12/d14/d49/f11e [0,4194304] 0 2024-04-12T05:37:34.145 INFO:tasks.workunit.client.1.smithi067.stdout:1/860: link d4/d9/d4c/lf4 d4/d9/d3a/d51/d55/dad/l117 0 2024-04-12T05:37:34.145 INFO:tasks.workunit.client.1.smithi067.stdout:4/794: rename d8/dba/d5d/d66/d71/d92/fc5 to d8/d9/d3d/f10d 0 2024-04-12T05:37:34.147 INFO:tasks.workunit.client.1.smithi067.stdout:2/813: fsync d5/da/de/d14/d34/d4c/f9d 0 2024-04-12T05:37:34.149 INFO:tasks.workunit.client.1.smithi067.stdout:4/795: chown d8/dba/d5d/d66 10823 1 2024-04-12T05:37:34.149 INFO:tasks.workunit.client.1.smithi067.stdout:6/834: dread dd/d21/d64/fce [0,4194304] 0 2024-04-12T05:37:34.150 INFO:tasks.workunit.client.1.smithi067.stdout:8/765: creat d6/d8/d50/fe6 x:0 0 0 2024-04-12T05:37:34.155 INFO:tasks.workunit.client.1.smithi067.stdout:7/854: fsync d0/d23/d80/da0/db4/dd/da2/f92 0 2024-04-12T05:37:34.155 INFO:tasks.workunit.client.1.smithi067.stdout:8/766: dread - d6/d8/d50/dd4/d4e/fa8 zero size 2024-04-12T05:37:34.155 INFO:tasks.workunit.client.1.smithi067.stdout:9/870: write d1/d66/d8e/fc6 [1081894,113535] 0 2024-04-12T05:37:34.157 INFO:tasks.workunit.client.1.smithi067.stdout:3/769: fdatasync dc/d1d/f44 0 2024-04-12T05:37:34.157 INFO:tasks.workunit.client.1.smithi067.stdout:7/855: chown d0/d23/d80/da0/db4/f74 0 1 2024-04-12T05:37:34.163 INFO:tasks.workunit.client.1.smithi067.stdout:2/814: chown d5/da/de/d14/d34/d4c/dbc/de3 6 1 2024-04-12T05:37:34.167 INFO:tasks.workunit.client.1.smithi067.stdout:6/835: dread - dd/d21/d64/d6f/d90/d98/fd6 zero size 2024-04-12T05:37:34.168 INFO:tasks.workunit.client.1.smithi067.stdout:9/871: dwrite d1/d28/d21/d45/f8b [0,4194304] 0 2024-04-12T05:37:34.169 INFO:tasks.workunit.client.1.smithi067.stdout:0/948: creat dd/d10f/f129 x:0 0 0 2024-04-12T05:37:34.169 INFO:tasks.workunit.client.1.smithi067.stdout:8/767: dread d6/dc7/f78 [0,4194304] 0 2024-04-12T05:37:34.173 INFO:tasks.workunit.client.1.smithi067.stdout:3/770: symlink dc/d1d/d28/d3e/d53/d76/da7/df1/dc8/da3/dde/l114 0 2024-04-12T05:37:34.174 INFO:tasks.workunit.client.1.smithi067.stdout:3/771: chown c8 2 1 2024-04-12T05:37:34.178 INFO:tasks.workunit.client.1.smithi067.stdout:7/856: rename d0/d67/d57/fca to d0/d23/d80/da0/db4/dd/da2/df8/f124 0 2024-04-12T05:37:34.180 INFO:tasks.workunit.client.1.smithi067.stdout:0/949: creat dd/d12/d14/d1a/d10a/f12a x:0 0 0 2024-04-12T05:37:34.182 INFO:tasks.workunit.client.1.smithi067.stdout:7/857: stat d0/d23/d80/da0/db4/d5/d4a/l91 0 2024-04-12T05:37:34.190 INFO:tasks.workunit.client.1.smithi067.stdout:2/815: dwrite d5/da/de/d14/d46/d90/fd5 [0,4194304] 0 2024-04-12T05:37:34.190 INFO:tasks.workunit.client.1.smithi067.stdout:0/950: unlink dd/d12/f16 0 2024-04-12T05:37:34.196 INFO:tasks.workunit.client.1.smithi067.stdout:8/768: rename d6/d8/d50/dd4/d63/d67 to d6/d8/d50/dd4/de7 0 2024-04-12T05:37:34.196 INFO:tasks.workunit.client.1.smithi067.stdout:0/951: chown dd/d12/d14/d1a/d3d/df2/d10c/lff 3 1 2024-04-12T05:37:34.198 INFO:tasks.workunit.client.1.smithi067.stdout:0/952: readlink dd/l19 0 2024-04-12T05:37:34.199 INFO:tasks.workunit.client.1.smithi067.stdout:9/872: dwrite d1/d28/d5e/f67 [4194304,4194304] 0 2024-04-12T05:37:34.201 INFO:tasks.workunit.client.1.smithi067.stdout:7/858: truncate d0/d23/d80/da0/db4/d5/d1d/d43/f49 18966 0 2024-04-12T05:37:34.204 INFO:tasks.workunit.client.1.smithi067.stdout:3/772: write dc/df/d8d/fab [2315074,99418] 0 2024-04-12T05:37:34.204 INFO:tasks.workunit.client.1.smithi067.stdout:0/953: chown dd/d6f/d72/cc5 13237569 1 2024-04-12T05:37:34.205 INFO:tasks.workunit.client.1.smithi067.stdout:1/861: dread d4/d9/d12/d2c/f7e [0,4194304] 0 2024-04-12T05:37:34.206 INFO:tasks.workunit.client.1.smithi067.stdout:3/773: stat dc/d50/l99 0 2024-04-12T05:37:34.207 INFO:tasks.workunit.client.1.smithi067.stdout:8/769: unlink d6/d8/d50/dd4/de7/d69/l7d 0 2024-04-12T05:37:34.208 INFO:tasks.workunit.client.1.smithi067.stdout:9/873: write d1/d28/d27/f71 [2047736,95087] 0 2024-04-12T05:37:34.213 INFO:tasks.workunit.client.1.smithi067.stdout:0/954: rmdir dd/d12/d14/d51 39 2024-04-12T05:37:34.213 INFO:tasks.workunit.client.1.smithi067.stdout:3/774: fdatasync dc/f18 0 2024-04-12T05:37:34.214 INFO:tasks.workunit.client.1.smithi067.stdout:1/862: mkdir d4/d3d/d7b/d118 0 2024-04-12T05:37:34.217 INFO:tasks.workunit.client.1.smithi067.stdout:8/770: truncate d6/d8/d50/dd4/f5e 1888347 0 2024-04-12T05:37:34.220 INFO:tasks.workunit.client.1.smithi067.stdout:9/874: symlink d1/d28/l113 0 2024-04-12T05:37:34.221 INFO:tasks.workunit.client.1.smithi067.stdout:0/955: fsync dd/d5b/f8e 0 2024-04-12T05:37:34.223 INFO:tasks.workunit.client.1.smithi067.stdout:1/863: dwrite d4/d9/f75 [0,4194304] 0 2024-04-12T05:37:34.225 INFO:tasks.workunit.client.1.smithi067.stdout:1/864: write d4/d9/d3a/d51/d55/dad/f10c [428858,89309] 0 2024-04-12T05:37:34.226 INFO:tasks.workunit.client.1.smithi067.stdout:1/865: chown d4/d9/d3a/fd3 1729243901 1 2024-04-12T05:37:34.228 INFO:tasks.workunit.client.1.smithi067.stdout:8/771: symlink d6/d8/d8b/dc3/le8 0 2024-04-12T05:37:34.231 INFO:tasks.workunit.client.1.smithi067.stdout:8/772: write d6/d8/d50/dd4/d63/d9a/f20 [2534557,9900] 0 2024-04-12T05:37:34.235 INFO:tasks.workunit.client.1.smithi067.stdout:9/875: unlink d1/d13/f5b 0 2024-04-12T05:37:34.236 INFO:tasks.workunit.client.1.smithi067.stdout:0/956: fsync dd/d12/d14/f8d 0 2024-04-12T05:37:34.238 INFO:tasks.workunit.client.1.smithi067.stdout:9/876: chown d1/d13/db7/fbb 206634936 1 2024-04-12T05:37:34.239 INFO:tasks.workunit.client.1.smithi067.stdout:3/775: symlink dc/df/df4/df6/l115 0 2024-04-12T05:37:34.241 INFO:tasks.workunit.client.1.smithi067.stdout:9/877: fsync d1/d28/d21/d45/d75/f80 0 2024-04-12T05:37:34.242 INFO:tasks.workunit.client.1.smithi067.stdout:1/866: dread d4/d8f/fa0 [0,4194304] 0 2024-04-12T05:37:34.243 INFO:tasks.workunit.client.1.smithi067.stdout:7/859: dwrite d0/d23/d80/da0/db4/dd/da2/f9b [0,4194304] 0 2024-04-12T05:37:34.247 INFO:tasks.workunit.client.1.smithi067.stdout:0/957: chown dd/d12/d14/d51/da2 32814 1 2024-04-12T05:37:34.260 INFO:tasks.workunit.client.1.smithi067.stdout:8/773: chown d6/d8/d50/dd4/de7/d69/c30 0 1 2024-04-12T05:37:34.261 INFO:tasks.workunit.client.1.smithi067.stdout:8/774: write d6/d8/d50/dd4/d36/f7f [2645407,92269] 0 2024-04-12T05:37:34.263 INFO:tasks.workunit.client.1.smithi067.stdout:9/878: rmdir d1/d66/d8e/dfc 39 2024-04-12T05:37:34.265 INFO:tasks.workunit.client.1.smithi067.stdout:9/879: write d1/d28/d21/d3f/d4f/fb4 [5641253,44893] 0 2024-04-12T05:37:34.271 INFO:tasks.workunit.client.1.smithi067.stdout:8/775: mkdir d6/d8/d50/dd4/d63/d9a/d12/de9 0 2024-04-12T05:37:34.272 INFO:tasks.workunit.client.1.smithi067.stdout:8/776: stat d6/d8/d50/dd4/d4e 0 2024-04-12T05:37:34.273 INFO:tasks.workunit.client.1.smithi067.stdout:9/880: mknod d1/d28/d21/d45/d75/de6/d8d/c114 0 2024-04-12T05:37:34.278 INFO:tasks.workunit.client.1.smithi067.stdout:7/860: dread d0/d23/f6e [0,4194304] 0 2024-04-12T05:37:34.279 INFO:tasks.workunit.client.1.smithi067.stdout:0/958: getdents dd/d6f/d72/d10b 0 2024-04-12T05:37:34.290 INFO:tasks.workunit.client.1.smithi067.stdout:9/881: dwrite d1/d12/f74 [0,4194304] 0 2024-04-12T05:37:34.294 INFO:tasks.workunit.client.1.smithi067.stdout:8/777: truncate d6/d8/d50/dd4/d63/d9a/d12/dd8/fdb 333399 0 2024-04-12T05:37:34.295 INFO:tasks.workunit.client.1.smithi067.stdout:9/882: mknod d1/d28/d21/d3f/d4f/c115 0 2024-04-12T05:37:34.297 INFO:tasks.workunit.client.1.smithi067.stdout:7/861: truncate d0/d23/d80/da0/db4/f74 189303 0 2024-04-12T05:37:34.299 INFO:tasks.workunit.client.1.smithi067.stdout:8/778: write d6/d8/d50/dd4/d4e/d9e/fc2 [417475,40077] 0 2024-04-12T05:37:34.300 INFO:tasks.workunit.client.1.smithi067.stdout:9/883: mknod d1/d13/c116 0 2024-04-12T05:37:34.301 INFO:tasks.workunit.client.1.smithi067.stdout:0/959: dwrite dd/d12/d14/d51/f77 [0,4194304] 0 2024-04-12T05:37:34.302 INFO:tasks.workunit.client.1.smithi067.stdout:0/960: fdatasync dd/d9c/d64/f7e 0 2024-04-12T05:37:34.302 INFO:tasks.workunit.client.1.smithi067.stdout:7/862: mkdir d0/d23/d80/da0/db4/d109/d125 0 2024-04-12T05:37:34.303 INFO:tasks.workunit.client.1.smithi067.stdout:8/779: chown d6/d8/d50/dd4/de7/d69/f89 48498860 1 2024-04-12T05:37:34.305 INFO:tasks.workunit.client.1.smithi067.stdout:0/961: readlink dd/d9c/l95 0 2024-04-12T05:37:34.308 INFO:tasks.workunit.client.1.smithi067.stdout:8/780: dread d6/d8/d50/dd4/f41 [0,4194304] 0 2024-04-12T05:37:34.308 INFO:tasks.workunit.client.1.smithi067.stdout:7/863: unlink d0/d23/d80/da0/db4/dd/d42/d7e/c31 0 2024-04-12T05:37:34.310 INFO:tasks.workunit.client.1.smithi067.stdout:8/781: mknod d6/d8/d50/dd4/d4e/d9e/cea 0 2024-04-12T05:37:34.311 INFO:tasks.workunit.client.1.smithi067.stdout:8/782: stat d6/d8/d50/dd4/d36/f7f 0 2024-04-12T05:37:34.311 INFO:tasks.workunit.client.1.smithi067.stdout:7/864: mkdir d0/d23/d80/da0/db4/dd/da2/df8/d126 0 2024-04-12T05:37:34.319 INFO:tasks.workunit.client.1.smithi067.stdout:7/865: dwrite d0/d23/d80/da0/db4/f52 [0,4194304] 0 2024-04-12T05:37:34.325 INFO:tasks.workunit.client.1.smithi067.stdout:7/866: chown d0/d23/d80/da0/db4/d5/d1d/f110 985 1 2024-04-12T05:37:34.325 INFO:tasks.workunit.client.1.smithi067.stdout:9/884: truncate d1/d28/d21/d3f/f41 6587453 0 2024-04-12T05:37:34.326 INFO:tasks.workunit.client.1.smithi067.stdout:7/867: truncate d0/d67/fd4 1130560 0 2024-04-12T05:37:34.327 INFO:tasks.workunit.client.1.smithi067.stdout:0/962: write dd/d12/d14/d51/f5c [4389470,109379] 0 2024-04-12T05:37:34.335 INFO:tasks.workunit.client.1.smithi067.stdout:0/963: unlink dd/d12/d14/d1a/d3d/df2/d10c/d83/faf 0 2024-04-12T05:37:34.337 INFO:tasks.workunit.client.1.smithi067.stdout:0/964: write dd/d12/f2e [1607436,128498] 0 2024-04-12T05:37:34.338 INFO:tasks.workunit.client.1.smithi067.stdout:7/868: mkdir d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/dd6/ddf/d127 0 2024-04-12T05:37:34.341 INFO:tasks.workunit.client.1.smithi067.stdout:0/965: mknod dd/dbd/c12b 0 2024-04-12T05:37:34.341 INFO:tasks.workunit.client.1.smithi067.stdout:8/783: dwrite d6/f2c [4194304,4194304] 0 2024-04-12T05:37:34.345 INFO:tasks.workunit.client.1.smithi067.stdout:0/966: rmdir dd/d12/d14/d49/d9b 39 2024-04-12T05:37:34.348 INFO:tasks.workunit.client.1.smithi067.stdout:7/869: rename d0/d23/d80/da0/db4/l15 to d0/d23/d80/da0/db4/dd/d42/d71/l128 0 2024-04-12T05:37:34.350 INFO:tasks.workunit.client.1.smithi067.stdout:0/967: symlink dd/d12/d14/d1a/d3d/l12c 0 2024-04-12T05:37:34.351 INFO:tasks.workunit.client.1.smithi067.stdout:8/784: link d6/d8/d50/dd4/d63/d9a/dd7/l9c d6/d8/d50/dd4/d63/d9a/dd7/d82/leb 0 2024-04-12T05:37:34.352 INFO:tasks.workunit.client.1.smithi067.stdout:8/785: read - d6/d8/d50/dd4/d4e/d9e/fdf zero size 2024-04-12T05:37:34.352 INFO:tasks.workunit.client.1.smithi067.stdout:7/870: mkdir d0/d23/d80/da0/db4/d5/d1d/de9/de1/d129 0 2024-04-12T05:37:34.353 INFO:tasks.workunit.client.1.smithi067.stdout:8/786: dread - d6/d8/d50/dd4/d63/d9a/fb3 zero size 2024-04-12T05:37:34.354 INFO:tasks.workunit.client.1.smithi067.stdout:7/871: write d0/d23/d45/dd1/f118 [673583,23757] 0 2024-04-12T05:37:34.355 INFO:tasks.workunit.client.1.smithi067.stdout:0/968: creat dd/d12/d14/d1a/d3d/df2/d10c/f12d x:0 0 0 2024-04-12T05:37:34.358 INFO:tasks.workunit.client.1.smithi067.stdout:7/872: mkdir d0/d23/d80/da0/db4/d5/d1d/d43/d12a 0 2024-04-12T05:37:34.360 INFO:tasks.workunit.client.1.smithi067.stdout:8/787: creat d6/d8/d50/dd4/d63/d9a/d12/de9/fec x:0 0 0 2024-04-12T05:37:34.361 INFO:tasks.workunit.client.1.smithi067.stdout:7/873: rmdir d0/d23/d80/da0/db4/dd/d79 39 2024-04-12T05:37:34.371 INFO:tasks.workunit.client.1.smithi067.stdout:8/788: dread d6/d8/d50/dd4/f5e [0,4194304] 0 2024-04-12T05:37:34.373 INFO:tasks.workunit.client.1.smithi067.stdout:8/789: stat d6/d8/d50/dd4/d63/d9a/dd7 0 2024-04-12T05:37:34.374 INFO:tasks.workunit.client.1.smithi067.stdout:7/874: dwrite d0/d23/d80/da0/db4/d5/d1d/fed [0,4194304] 0 2024-04-12T05:37:34.397 INFO:tasks.workunit.client.1.smithi067.stdout:8/790: dread d6/d8/fce [0,4194304] 0 2024-04-12T05:37:34.430 INFO:tasks.workunit.client.1.smithi067.stdout:7/875: unlink d0/d23/d80/da0/db4/d5/d4a/de5/d10c/c120 0 2024-04-12T05:37:34.433 INFO:tasks.workunit.client.1.smithi067.stdout:8/791: creat d6/d8/d8b/da1/fed x:0 0 0 2024-04-12T05:37:34.434 INFO:tasks.workunit.client.1.smithi067.stdout:8/792: chown d6/d8/d50/dd4/d4e 1840861 1 2024-04-12T05:37:34.435 INFO:tasks.workunit.client.1.smithi067.stdout:9/885: write d1/d12/f53 [2051264,39916] 0 2024-04-12T05:37:34.453 INFO:tasks.workunit.client.1.smithi067.stdout:9/886: creat d1/d12/f117 x:0 0 0 2024-04-12T05:37:34.454 INFO:tasks.workunit.client.1.smithi067.stdout:9/887: stat d1/d28/d21/d45/d75/de6/d8d/dd2/fcf 0 2024-04-12T05:37:34.455 INFO:tasks.workunit.client.1.smithi067.stdout:9/888: dread - d1/d13/fb9 zero size 2024-04-12T05:37:34.456 INFO:tasks.workunit.client.1.smithi067.stdout:8/793: getdents d6/d8/d50/dd4/de7/d69 0 2024-04-12T05:37:34.460 INFO:tasks.workunit.client.1.smithi067.stdout:0/969: write dd/d12/d14/d49/fa6 [4333697,109998] 0 2024-04-12T05:37:34.461 INFO:tasks.workunit.client.1.smithi067.stdout:0/970: chown dd/d12/d14/d49/f6d 114 1 2024-04-12T05:37:34.462 INFO:tasks.workunit.client.1.smithi067.stdout:8/794: write d6/d8/f92 [1190297,25984] 0 2024-04-12T05:37:34.462 INFO:tasks.workunit.client.1.smithi067.stdout:4/796: sync 2024-04-12T05:37:34.465 INFO:tasks.workunit.client.1.smithi067.stdout:0/971: write dd/d12/d14/d1a/f124 [369727,103756] 0 2024-04-12T05:37:34.466 INFO:tasks.workunit.client.1.smithi067.stdout:9/889: dread d1/d66/fe9 [0,4194304] 0 2024-04-12T05:37:34.469 INFO:tasks.workunit.client.1.smithi067.stdout:9/890: read d1/f6 [490515,34782] 0 2024-04-12T05:37:34.471 INFO:tasks.workunit.client.1.smithi067.stdout:4/797: dread d8/da/d1d/d15/d58/f78 [0,4194304] 0 2024-04-12T05:37:34.474 INFO:tasks.workunit.client.1.smithi067.stdout:4/798: dread - d8/dba/d5d/d66/f9c zero size 2024-04-12T05:37:34.476 INFO:tasks.workunit.client.1.smithi067.stdout:4/799: read d8/d9/d3d/f10d [2486737,128067] 0 2024-04-12T05:37:34.477 INFO:tasks.workunit.client.1.smithi067.stdout:4/800: chown d8/da/d9f/dcf/c100 4 1 2024-04-12T05:37:34.478 INFO:tasks.workunit.client.1.smithi067.stdout:9/891: chown d1/d28/d5e/l10b 365 1 2024-04-12T05:37:34.485 INFO:tasks.workunit.client.1.smithi067.stdout:4/801: unlink d8/dba/d5d/d66/d7b/dbd/fca 0 2024-04-12T05:37:34.490 INFO:tasks.workunit.client.1.smithi067.stdout:4/802: write d8/d9/d3d/f10d [2563615,91704] 0 2024-04-12T05:37:34.494 INFO:tasks.workunit.client.1.smithi067.stdout:4/803: truncate d8/d9/d3d/f4d 1698266 0 2024-04-12T05:37:34.498 INFO:tasks.workunit.client.1.smithi067.stdout:4/804: creat d8/dba/d5d/d66/d7b/dbd/f10e x:0 0 0 2024-04-12T05:37:34.503 INFO:tasks.workunit.client.1.smithi067.stdout:4/805: fsync d8/d9/f42 0 2024-04-12T05:37:34.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:34 smithi067 ceph-mon[49010]: pgmap v278: 65 pgs: 65 active+clean; 3.2 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 80 MiB/s rd, 218 MiB/s wr, 514 op/s 2024-04-12T05:37:34.520 INFO:tasks.workunit.client.1.smithi067.stdout:4/806: dread d8/da/f30 [0,4194304] 0 2024-04-12T05:37:34.548 INFO:tasks.workunit.client.1.smithi067.stdout:8/795: write d6/d8/d50/dd4/f5e [430398,85558] 0 2024-04-12T05:37:34.554 INFO:tasks.workunit.client.1.smithi067.stdout:8/796: link d6/d8/d50/dd4/d63/d9a/dbb/dd9/fe4 d6/d8/d50/dd4/d4e/fee 0 2024-04-12T05:37:34.555 INFO:tasks.workunit.client.1.smithi067.stdout:0/972: dwrite dd/dbd/dbe/fc2 [0,4194304] 0 2024-04-12T05:37:34.562 INFO:tasks.workunit.client.1.smithi067.stdout:8/797: write d6/d8/d50/fe0 [476154,19189] 0 2024-04-12T05:37:34.562 INFO:tasks.workunit.client.1.smithi067.stdout:0/973: creat dd/d12/d14/d49/dab/f12e x:0 0 0 2024-04-12T05:37:34.566 INFO:tasks.workunit.client.1.smithi067.stdout:0/974: symlink dd/d12/d14/l12f 0 2024-04-12T05:37:34.570 INFO:tasks.workunit.client.1.smithi067.stdout:9/892: dwrite d1/d12/ff7 [0,4194304] 0 2024-04-12T05:37:34.571 INFO:tasks.workunit.client.1.smithi067.stdout:0/975: creat dd/d9c/d64/ddd/f130 x:0 0 0 2024-04-12T05:37:34.575 INFO:tasks.workunit.client.1.smithi067.stdout:9/893: mkdir d1/d13/db7/d118 0 2024-04-12T05:37:34.584 INFO:tasks.workunit.client.1.smithi067.stdout:0/976: getdents dd/d92 0 2024-04-12T05:37:34.586 INFO:tasks.workunit.client.1.smithi067.stdout:0/977: mknod dd/dad/dd9/c131 0 2024-04-12T05:37:34.588 INFO:tasks.workunit.client.1.smithi067.stdout:0/978: creat dd/d9c/f132 x:0 0 0 2024-04-12T05:37:34.590 INFO:tasks.workunit.client.1.smithi067.stdout:0/979: write dd/d12/d14/d1a/d10a/f12a [565897,17057] 0 2024-04-12T05:37:34.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:34 smithi032 ceph-mon[39731]: pgmap v278: 65 pgs: 65 active+clean; 3.2 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 80 MiB/s rd, 218 MiB/s wr, 514 op/s 2024-04-12T05:37:34.594 INFO:tasks.workunit.client.1.smithi067.stdout:0/980: mknod dd/d9c/d5f/c133 0 2024-04-12T05:37:34.602 INFO:tasks.workunit.client.1.smithi067.stdout:0/981: dwrite dd/d12/d14/d49/d9b/fa5 [0,4194304] 0 2024-04-12T05:37:34.606 INFO:tasks.workunit.client.1.smithi067.stdout:0/982: creat dd/d9c/d64/ddd/f134 x:0 0 0 2024-04-12T05:37:34.612 INFO:tasks.workunit.client.1.smithi067.stdout:7/876: rename d0/d23/d80/da0/db4/f74 to d0/d23/d80/da0/db4/dd/d42/d7e/d34/da3/dea/f12b 0 2024-04-12T05:37:34.618 INFO:tasks.workunit.client.1.smithi067.stdout:7/877: mknod d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/d11b/c12c 0 2024-04-12T05:37:34.618 INFO:tasks.workunit.client.1.smithi067.stdout:7/878: chown d0/d67/d57/fa7 64 1 2024-04-12T05:37:34.622 INFO:tasks.workunit.client.1.smithi067.stdout:8/798: rename d6/d8/d50/dd4/d63 to d6/def 0 2024-04-12T05:37:34.626 INFO:tasks.workunit.client.1.smithi067.stdout:8/799: creat d6/dc7/ff0 x:0 0 0 2024-04-12T05:37:34.656 INFO:tasks.workunit.client.1.smithi067.stdout:0/983: dwrite dd/d12/d14/d51/fb9 [0,4194304] 0 2024-04-12T05:37:34.660 INFO:tasks.workunit.client.1.smithi067.stdout:0/984: readlink dd/d6f/lc3 0 2024-04-12T05:37:34.664 INFO:tasks.workunit.client.1.smithi067.stdout:8/800: rename d6/dc7 to d6/def/d9a/df1 0 2024-04-12T05:37:34.665 INFO:tasks.workunit.client.1.smithi067.stdout:8/801: dread - d6/d8/d50/dd4/d4e/fa8 zero size 2024-04-12T05:37:34.668 INFO:tasks.workunit.client.1.smithi067.stdout:0/985: unlink dd/d12/d14/d49/d9b/fa5 0 2024-04-12T05:37:34.668 INFO:tasks.workunit.client.1.smithi067.stdout:7/879: truncate d0/d23/d80/da0/db4/dd/d42/d71/fac 659858 0 2024-04-12T05:37:34.672 INFO:tasks.workunit.client.1.smithi067.stdout:0/986: creat dd/d6f/d72/f135 x:0 0 0 2024-04-12T05:37:34.674 INFO:tasks.workunit.client.1.smithi067.stdout:8/802: mkdir d6/df2 0 2024-04-12T05:37:34.675 INFO:tasks.workunit.client.1.smithi067.stdout:8/803: write d6/d8/d50/dd4/d36/f70 [3132551,54768] 0 2024-04-12T05:37:34.676 INFO:tasks.workunit.client.1.smithi067.stdout:0/987: symlink dd/d12/d14/d51/da2/l136 0 2024-04-12T05:37:34.677 INFO:tasks.workunit.client.1.smithi067.stdout:0/988: chown dd/d12/d14/d49/dab/fcf 894 1 2024-04-12T05:37:34.686 INFO:tasks.workunit.client.1.smithi067.stdout:8/804: mknod d6/d8/d50/dd4/cf3 0 2024-04-12T05:37:34.686 INFO:tasks.workunit.client.1.smithi067.stdout:7/880: write d0/d23/d80/da0/db4/dd/d79/f63 [3058835,10195] 0 2024-04-12T05:37:34.688 INFO:tasks.workunit.client.1.smithi067.stdout:0/989: mknod dd/d12/d14/d1a/c137 0 2024-04-12T05:37:34.689 INFO:tasks.workunit.client.1.smithi067.stdout:8/805: write d6/d8/d50/dd4/d36/f54 [1395072,69401] 0 2024-04-12T05:37:34.691 INFO:tasks.workunit.client.1.smithi067.stdout:0/990: unlink dd/d92/la1 0 2024-04-12T05:37:34.692 INFO:tasks.workunit.client.1.smithi067.stdout:8/806: read d6/d8/d50/dd4/d36/f68 [3145776,39506] 0 2024-04-12T05:37:34.697 INFO:tasks.workunit.client.1.smithi067.stdout:0/991: truncate dd/d12/d14/de3/dfd/f102 877089 0 2024-04-12T05:37:34.710 INFO:tasks.workunit.client.1.smithi067.stdout:8/807: dread d6/d8/f92 [0,4194304] 0 2024-04-12T05:37:34.712 INFO:tasks.workunit.client.1.smithi067.stdout:7/881: dread d0/d23/d80/da0/db4/d5/d1d/d43/fb9 [0,4194304] 0 2024-04-12T05:37:34.713 INFO:tasks.workunit.client.1.smithi067.stdout:8/808: mkdir d6/def/d9a/d12/df4 0 2024-04-12T05:37:34.714 INFO:tasks.workunit.client.1.smithi067.stdout:0/992: dread dd/d9c/d64/f7e [0,4194304] 0 2024-04-12T05:37:34.718 INFO:tasks.workunit.client.1.smithi067.stdout:8/809: symlink d6/d8/d50/lf5 0 2024-04-12T05:37:34.724 INFO:tasks.workunit.client.1.smithi067.stdout:7/882: dwrite d0/d23/d80/da0/db4/d5/d1d/de9/fcc [0,4194304] 0 2024-04-12T05:37:34.729 INFO:tasks.workunit.client.1.smithi067.stdout:0/993: creat dd/d12/d14/d51/f138 x:0 0 0 2024-04-12T05:37:34.733 INFO:tasks.workunit.client.1.smithi067.stdout:7/883: unlink d0/d23/d80/da0/db4/dd/d79/c75 0 2024-04-12T05:37:34.734 INFO:tasks.workunit.client.1.smithi067.stdout:7/884: readlink d0/d23/d45/l2e 0 2024-04-12T05:37:34.734 INFO:tasks.workunit.client.1.smithi067.stdout:0/994: mkdir dd/dad/d139 0 2024-04-12T05:37:34.736 INFO:tasks.workunit.client.1.smithi067.stdout:8/810: write d6/def/d9a/d12/dd8/fdb [879659,106851] 0 2024-04-12T05:37:34.737 INFO:tasks.workunit.client.1.smithi067.stdout:0/995: chown dd/d12/d14/d49/d9b/fb6 7171707 1 2024-04-12T05:37:34.739 INFO:tasks.workunit.client.1.smithi067.stdout:7/885: unlink d0/d23/d80/da0/db4/d5/d4a/f11d 0 2024-04-12T05:37:34.741 INFO:tasks.workunit.client.1.smithi067.stdout:8/811: mknod d6/d8/d50/dd4/d4e/cf6 0 2024-04-12T05:37:34.746 INFO:tasks.workunit.client.1.smithi067.stdout:8/812: rename d6/def/d9a/df1 to d6/d8/d8b/df7 0 2024-04-12T05:37:34.747 INFO:tasks.workunit.client.1.smithi067.stdout:7/886: mkdir d0/d23/d80/da0/db4/dd/d42/d7e/d12d 0 2024-04-12T05:37:34.748 INFO:tasks.workunit.client.1.smithi067.stdout:8/813: write d6/d8/d50/dd4/d4e/d9e/fc2 [1101793,53857] 0 2024-04-12T05:37:34.750 INFO:tasks.workunit.client.1.smithi067.stdout:8/814: read d6/d8/d50/dd4/de7/d69/f89 [45738,35776] 0 2024-04-12T05:37:34.755 INFO:tasks.workunit.client.1.smithi067.stdout:7/887: creat d0/d23/d80/da0/db4/dd/d42/d82/f12e x:0 0 0 2024-04-12T05:37:34.756 INFO:tasks.workunit.client.1.smithi067.stdout:8/815: dwrite d6/d8/d8b/da1/fd1 [0,4194304] 0 2024-04-12T05:37:34.756 INFO:tasks.workunit.client.1.smithi067.stdout:7/888: write d0/d67/fd4 [2029992,58426] 0 2024-04-12T05:37:34.757 INFO:tasks.workunit.client.1.smithi067.stdout:0/996: truncate dd/d12/d14/d1a/f6c 3895184 0 2024-04-12T05:37:34.758 INFO:tasks.workunit.client.1.smithi067.stdout:2/816: sync 2024-04-12T05:37:34.758 INFO:tasks.workunit.client.1.smithi067.stdout:6/836: sync 2024-04-12T05:37:34.765 INFO:tasks.workunit.client.1.smithi067.stdout:8/816: dread d6/def/d9a/d12/fa9 [0,4194304] 0 2024-04-12T05:37:34.768 INFO:tasks.workunit.client.1.smithi067.stdout:7/889: unlink d0/d23/d80/da0/db4/dd/d42/d82/d96/fab 0 2024-04-12T05:37:34.773 INFO:tasks.workunit.client.1.smithi067.stdout:6/837: mkdir dd/d21/d64/d6f/d90/d111 0 2024-04-12T05:37:34.778 INFO:tasks.workunit.client.1.smithi067.stdout:7/890: creat d0/d67/d57/f12f x:0 0 0 2024-04-12T05:37:34.778 INFO:tasks.workunit.client.1.smithi067.stdout:2/817: dread d5/da/de/d14/d34/f37 [0,4194304] 0 2024-04-12T05:37:34.785 INFO:tasks.workunit.client.1.smithi067.stdout:2/818: dwrite d5/d11/f32 [0,4194304] 0 2024-04-12T05:37:34.789 INFO:tasks.workunit.client.1.smithi067.stdout:7/891: rename d0/d23/d80/da0/db4/d5/d1d/d7d/c8b to d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/c130 0 2024-04-12T05:37:34.798 INFO:tasks.workunit.client.1.smithi067.stdout:8/817: dwrite d6/def/d9a/dd7/f65 [0,4194304] 0 2024-04-12T05:37:34.801 INFO:tasks.workunit.client.1.smithi067.stdout:0/997: dwrite dd/d12/d14/d1a/f6c [0,4194304] 0 2024-04-12T05:37:34.804 INFO:tasks.workunit.client.1.smithi067.stdout:6/838: dwrite dd/d21/d64/d6f/f70 [4194304,4194304] 0 2024-04-12T05:37:34.804 INFO:tasks.workunit.client.1.smithi067.stdout:2/819: write d5/d11/f2e [5559489,33454] 0 2024-04-12T05:37:34.806 INFO:tasks.workunit.client.1.smithi067.stdout:7/892: truncate d0/d23/d80/da0/db4/dd/f123 685972 0 2024-04-12T05:37:34.808 INFO:tasks.workunit.client.1.smithi067.stdout:0/998: chown dd/d12/d14/de3/dfd/f102 32788222 1 2024-04-12T05:37:34.810 INFO:tasks.workunit.client.1.smithi067.stdout:8/818: read d6/d8/d50/dd4/de7/fad [379807,606] 0 2024-04-12T05:37:34.810 INFO:tasks.workunit.client.1.smithi067.stdout:6/839: chown dd/d11/d27/d4a/d51/cc0 2943449 1 2024-04-12T05:37:34.814 INFO:tasks.workunit.client.1.smithi067.stdout:8/819: fdatasync d6/def/d9a/fbf 0 2024-04-12T05:37:34.814 INFO:tasks.workunit.client.1.smithi067.stdout:6/840: write dd/d21/d64/d88/f96 [1895870,30570] 0 2024-04-12T05:37:34.815 INFO:tasks.workunit.client.1.smithi067.stdout:2/820: rename d5/d6a/cb2 to d5/da/de/dc4/dc6/c101 0 2024-04-12T05:37:34.816 INFO:tasks.workunit.client.1.smithi067.stdout:3/776: sync 2024-04-12T05:37:34.816 INFO:tasks.workunit.client.1.smithi067.stdout:4/807: sync 2024-04-12T05:37:34.816 INFO:tasks.workunit.client.1.smithi067.stdout:1/867: sync 2024-04-12T05:37:34.816 INFO:tasks.workunit.client.1.smithi067.stdout:9/894: sync 2024-04-12T05:37:34.822 INFO:tasks.workunit.client.1.smithi067.stdout:6/841: mkdir dd/d21/d64/d88/d7b/dcc/d112 0 2024-04-12T05:37:34.824 INFO:tasks.workunit.client.1.smithi067.stdout:4/808: chown d8/da/d1d/d15/d58/f78 460628 1 2024-04-12T05:37:34.826 INFO:tasks.workunit.client.1.smithi067.stdout:0/999: dread dd/d12/fb2 [0,4194304] 0 2024-04-12T05:37:34.827 INFO:tasks.workunit.client.1.smithi067.stdout:2/821: creat d5/da/de/d14/d34/f102 x:0 0 0 2024-04-12T05:37:34.828 INFO:tasks.workunit.client.1.smithi067.stdout:9/895: rename d1/d12/d15/f43 to d1/d28/d21/d3f/f119 0 2024-04-12T05:37:34.829 INFO:tasks.workunit.client.1.smithi067.stdout:4/809: write d8/da/f18 [4103131,73498] 0 2024-04-12T05:37:34.833 INFO:tasks.workunit.client.1.smithi067.stdout:6/842: creat dd/d21/d64/d6f/dea/f113 x:0 0 0 2024-04-12T05:37:34.833 INFO:tasks.workunit.client.1.smithi067.stdout:1/868: dwrite d4/d9/d14/f57 [0,4194304] 0 2024-04-12T05:37:34.834 INFO:tasks.workunit.client.1.smithi067.stdout:9/896: truncate d1/fe7 96477 0 2024-04-12T05:37:34.835 INFO:tasks.workunit.client.1.smithi067.stdout:2/822: write d5/d6a/d78/fd0 [760354,1230] 0 2024-04-12T05:37:34.835 INFO:tasks.workunit.client.1.smithi067.stdout:4/810: chown d8/da/d1d/d15 1 1 2024-04-12T05:37:34.836 INFO:tasks.workunit.client.1.smithi067.stdout:8/820: write d6/d8/d50/dd4/d36/f59 [2723196,67715] 0 2024-04-12T05:37:34.848 INFO:tasks.workunit.client.1.smithi067.stdout:6/843: dread - dd/d11/d27/d4a/d79/dc3/d56/de3/fe7 zero size 2024-04-12T05:37:34.848 INFO:tasks.workunit.client.1.smithi067.stdout:9/897: creat d1/d28/d21/d45/d75/de6/d8d/d10a/f11a x:0 0 0 2024-04-12T05:37:34.851 INFO:tasks.workunit.client.1.smithi067.stdout:1/869: truncate d4/d9/d3a/d51/f63 5013544 0 2024-04-12T05:37:34.852 INFO:tasks.workunit.client.1.smithi067.stdout:7/893: dread d0/d23/d80/da0/db4/dd/d79/f63 [0,4194304] 0 2024-04-12T05:37:34.854 INFO:tasks.workunit.client.1.smithi067.stdout:3/777: dwrite dc/d1d/f43 [0,4194304] 0 2024-04-12T05:37:34.856 INFO:tasks.workunit.client.1.smithi067.stdout:2/823: dread d5/da/de/d14/d46/d90/fd5 [0,4194304] 0 2024-04-12T05:37:34.856 INFO:tasks.workunit.client.1.smithi067.stdout:4/811: getdents d8/dba/d88/da0/de7 0 2024-04-12T05:37:34.856 INFO:tasks.workunit.client.1.smithi067.stdout:9/898: rmdir d1/d28/d21/d45/d75 39 2024-04-12T05:37:34.862 INFO:tasks.workunit.client.1.smithi067.stdout:6/844: write dd/d11/d27/d4a/d79/dc3/d56/f99 [4521957,15863] 0 2024-04-12T05:37:34.865 INFO:tasks.workunit.client.1.smithi067.stdout:9/899: chown d1/d28/d21/d45/d105/f81 19792 1 2024-04-12T05:37:34.865 INFO:tasks.workunit.client.1.smithi067.stdout:1/870: getdents d4/d8f/d110 0 2024-04-12T05:37:34.866 INFO:tasks.workunit.client.1.smithi067.stdout:7/894: unlink d0/d23/d80/da0/db4/dd/cdc 0 2024-04-12T05:37:34.869 INFO:tasks.workunit.client.1.smithi067.stdout:2/824: creat d5/da/f103 x:0 0 0 2024-04-12T05:37:34.869 INFO:tasks.workunit.client.1.smithi067.stdout:1/871: dread - d4/d3d/d7b/dde/f108 zero size 2024-04-12T05:37:34.870 INFO:tasks.workunit.client.1.smithi067.stdout:7/895: chown d0/d23/d45/dd1/f118 193984 1 2024-04-12T05:37:34.870 INFO:tasks.workunit.client.1.smithi067.stdout:4/812: write d8/d9/f42 [217489,109646] 0 2024-04-12T05:37:34.873 INFO:tasks.workunit.client.1.smithi067.stdout:8/821: dwrite d6/d8/d8b/fae [0,4194304] 0 2024-04-12T05:37:34.875 INFO:tasks.workunit.client.1.smithi067.stdout:9/900: creat d1/d12/d15/f11b x:0 0 0 2024-04-12T05:37:34.879 INFO:tasks.workunit.client.1.smithi067.stdout:1/872: symlink d4/d9/d4d/l119 0 2024-04-12T05:37:34.883 INFO:tasks.workunit.client.1.smithi067.stdout:9/901: creat d1/d28/d21/d7f/f11c x:0 0 0 2024-04-12T05:37:34.884 INFO:tasks.workunit.client.1.smithi067.stdout:3/778: dwrite dc/f8c [0,4194304] 0 2024-04-12T05:37:34.889 INFO:tasks.workunit.client.1.smithi067.stdout:2/825: write d5/dbf/dcc/fd7 [1944626,48739] 0 2024-04-12T05:37:34.889 INFO:tasks.workunit.client.1.smithi067.stdout:1/873: chown d4/d3d/dab/fec 0 1 2024-04-12T05:37:34.891 INFO:tasks.workunit.client.1.smithi067.stdout:7/896: creat d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/f131 x:0 0 0 2024-04-12T05:37:34.891 INFO:tasks.workunit.client.1.smithi067.stdout:8/822: link d6/d8/f6a d6/def/d9a/ff8 0 2024-04-12T05:37:34.891 INFO:tasks.workunit.client.1.smithi067.stdout:3/779: read - dc/df/d27/daa/fb3 zero size 2024-04-12T05:37:34.892 INFO:tasks.workunit.client.1.smithi067.stdout:6/845: dwrite dd/d21/d64/d6f/d90/f9d [0,4194304] 0 2024-04-12T05:37:34.895 INFO:tasks.workunit.client.1.smithi067.stdout:8/823: chown d6/def/d9a/lcd 3 1 2024-04-12T05:37:34.896 INFO:tasks.workunit.client.1.smithi067.stdout:7/897: chown d0/d23/d80/da0/db4/d5/d4a/l105 127624748 1 2024-04-12T05:37:34.897 INFO:tasks.workunit.client.1.smithi067.stdout:3/780: rmdir dc/d1d/d26/d52/d9a 39 2024-04-12T05:37:34.898 INFO:tasks.workunit.client.1.smithi067.stdout:6/846: dread - dd/d11/d27/f87 zero size 2024-04-12T05:37:34.901 INFO:tasks.workunit.client.1.smithi067.stdout:9/902: link d1/d12/f74 d1/d28/d21/d3f/d4f/f11d 0 2024-04-12T05:37:34.903 INFO:tasks.workunit.client.1.smithi067.stdout:8/824: mkdir d6/def/d9a/d12/de9/df9 0 2024-04-12T05:37:34.904 INFO:tasks.workunit.client.1.smithi067.stdout:7/898: write d0/f1 [512546,101080] 0 2024-04-12T05:37:34.906 INFO:tasks.workunit.client.1.smithi067.stdout:3/781: creat dc/df/df4/df6/f116 x:0 0 0 2024-04-12T05:37:34.907 INFO:tasks.workunit.client.1.smithi067.stdout:4/813: dwrite d8/d9/f67 [0,4194304] 0 2024-04-12T05:37:34.910 INFO:tasks.workunit.client.1.smithi067.stdout:1/874: getdents d4/d9/d3a/dc4 0 2024-04-12T05:37:34.911 INFO:tasks.workunit.client.1.smithi067.stdout:1/875: chown d4/d52/d80 0 1 2024-04-12T05:37:34.914 INFO:tasks.workunit.client.1.smithi067.stdout:6/847: mknod dd/d21/d64/dcf/dfb/c114 0 2024-04-12T05:37:34.915 INFO:tasks.workunit.client.1.smithi067.stdout:2/826: dwrite d5/da/de/d14/d34/fac [0,4194304] 0 2024-04-12T05:37:34.915 INFO:tasks.workunit.client.1.smithi067.stdout:1/876: chown d4/d9/d3a/d51/d55/fe7 6 1 2024-04-12T05:37:34.918 INFO:tasks.workunit.client.1.smithi067.stdout:4/814: unlink d8/d9/f11 0 2024-04-12T05:37:34.925 INFO:tasks.workunit.client.1.smithi067.stdout:9/903: mkdir d1/d28/d21/d45/d75/de6/d8d/dc1/d11e 0 2024-04-12T05:37:34.926 INFO:tasks.workunit.client.1.smithi067.stdout:6/848: mknod dd/d11/d27/d4a/d79/dc3/d56/de3/c115 0 2024-04-12T05:37:34.927 INFO:tasks.workunit.client.1.smithi067.stdout:7/899: dwrite d0/d23/d80/da0/db4/d109/f114 [0,4194304] 0 2024-04-12T05:37:34.930 INFO:tasks.workunit.client.1.smithi067.stdout:6/849: chown dd/d21/d64/d88/d7b/f6e 25 1 2024-04-12T05:37:34.930 INFO:tasks.workunit.client.1.smithi067.stdout:2/827: rmdir d5/da/de/d14/d34/d40/d57 39 2024-04-12T05:37:34.933 INFO:tasks.workunit.client.1.smithi067.stdout:9/904: truncate d1/d28/d21/d45/d105/ff5 200795 0 2024-04-12T05:37:34.933 INFO:tasks.workunit.client.1.smithi067.stdout:6/850: write dd/d21/d64/d6f/d90/d98/fb9 [1250525,126436] 0 2024-04-12T05:37:34.934 INFO:tasks.workunit.client.1.smithi067.stdout:6/851: chown dd/d21/d3f/dd5/fe0 11586183 1 2024-04-12T05:37:34.936 INFO:tasks.workunit.client.1.smithi067.stdout:7/900: creat d0/f132 x:0 0 0 2024-04-12T05:37:34.942 INFO:tasks.workunit.client.1.smithi067.stdout:2/828: mkdir d5/da/de/d14/d34/d40/d57/d104 0 2024-04-12T05:37:34.943 INFO:tasks.workunit.client.1.smithi067.stdout:9/905: dwrite d1/d28/d21/d101/f111 [0,4194304] 0 2024-04-12T05:37:34.945 INFO:tasks.workunit.client.1.smithi067.stdout:4/815: link d8/da/d8b/d45/ce5 d8/da/d1d/d15/d75/c10f 0 2024-04-12T05:37:34.948 INFO:tasks.workunit.client.1.smithi067.stdout:7/901: creat d0/d23/d80/da0/db4/dd/d42/d82/d96/f133 x:0 0 0 2024-04-12T05:37:34.952 INFO:tasks.workunit.client.1.smithi067.stdout:9/906: creat d1/d13/db7/f11f x:0 0 0 2024-04-12T05:37:34.952 INFO:tasks.workunit.client.1.smithi067.stdout:3/782: write dc/d1d/d28/d3e/d53/d94/fbf [2220147,83289] 0 2024-04-12T05:37:34.954 INFO:tasks.workunit.client.1.smithi067.stdout:3/783: chown dc/f3a 0 1 2024-04-12T05:37:34.955 INFO:tasks.workunit.client.1.smithi067.stdout:4/816: mknod d8/dba/d5d/d66/d7b/dbd/c110 0 2024-04-12T05:37:34.956 INFO:tasks.workunit.client.1.smithi067.stdout:6/852: rmdir dd/d11/d27/d4a/d79/dc3 39 2024-04-12T05:37:34.957 INFO:tasks.workunit.client.1.smithi067.stdout:8/825: dwrite d6/d8/d50/dd4/de7/d69/d5d/f91 [0,4194304] 0 2024-04-12T05:37:34.961 INFO:tasks.workunit.client.1.smithi067.stdout:3/784: truncate dc/d1d/d28/d3e/d8a/f100 403547 0 2024-04-12T05:37:34.963 INFO:tasks.workunit.client.1.smithi067.stdout:3/785: stat dc/d1d/d28/d3e/d8a/dfd 0 2024-04-12T05:37:34.964 INFO:tasks.workunit.client.1.smithi067.stdout:1/877: dwrite d4/d9/d12/d2c/fe2 [0,4194304] 0 2024-04-12T05:37:34.968 INFO:tasks.workunit.client.1.smithi067.stdout:1/878: read d4/d9/d12/d2c/f50 [556320,82279] 0 2024-04-12T05:37:34.970 INFO:tasks.workunit.client.1.smithi067.stdout:8/826: unlink d6/f83 0 2024-04-12T05:37:34.971 INFO:tasks.workunit.client.1.smithi067.stdout:6/853: symlink dd/d11/d27/d4a/dc7/l116 0 2024-04-12T05:37:34.973 INFO:tasks.workunit.client.1.smithi067.stdout:6/854: write dd/d21/d3f/dd5/ff2 [210051,11169] 0 2024-04-12T05:37:34.973 INFO:tasks.workunit.client.1.smithi067.stdout:2/829: write d5/d43/d6e/f76 [1632533,110084] 0 2024-04-12T05:37:34.976 INFO:tasks.workunit.client.1.smithi067.stdout:6/855: read - dd/d21/fcb zero size 2024-04-12T05:37:34.978 INFO:tasks.workunit.client.1.smithi067.stdout:6/856: dread - dd/d21/d3f/f10d zero size 2024-04-12T05:37:34.978 INFO:tasks.workunit.client.1.smithi067.stdout:3/786: creat dc/d1d/d28/d3e/d53/d76/da7/df1/dc8/da3/db9/f117 x:0 0 0 2024-04-12T05:37:34.979 INFO:tasks.workunit.client.1.smithi067.stdout:3/787: fdatasync dc/f3a 0 2024-04-12T05:37:34.988 INFO:tasks.workunit.client.1.smithi067.stdout:2/830: creat d5/da/dae/dba/f105 x:0 0 0 2024-04-12T05:37:34.992 INFO:tasks.workunit.client.1.smithi067.stdout:4/817: chown d8/da/d1d/d15/d58/l98 450 1 2024-04-12T05:37:34.997 INFO:tasks.workunit.client.1.smithi067.stdout:7/902: dread d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/dd6/f8e [0,4194304] 0 2024-04-12T05:37:34.999 INFO:tasks.workunit.client.1.smithi067.stdout:6/857: symlink dd/d9f/l117 0 2024-04-12T05:37:35.001 INFO:tasks.workunit.client.1.smithi067.stdout:2/831: symlink d5/da/de/d14/d34/d40/l106 0 2024-04-12T05:37:35.004 INFO:tasks.workunit.client.1.smithi067.stdout:3/788: getdents dc/df/d27/dc5 0 2024-04-12T05:37:35.004 INFO:tasks.workunit.client.1.smithi067.stdout:8/827: dwrite d6/f2c [8388608,4194304] 0 2024-04-12T05:37:35.005 INFO:tasks.workunit.client.1.smithi067.stdout:2/832: rmdir d5/d11 39 2024-04-12T05:37:35.006 INFO:tasks.workunit.client.1.smithi067.stdout:4/818: chown d8/da/d9f/d68/db4/fd8 5 1 2024-04-12T05:37:35.006 INFO:tasks.workunit.client.1.smithi067.stdout:1/879: dwrite d4/d9/d3a/f1e [4194304,4194304] 0 2024-04-12T05:37:35.009 INFO:tasks.workunit.client.1.smithi067.stdout:6/858: creat dd/d11/d27/d4a/d8b/f118 x:0 0 0 2024-04-12T05:37:35.011 INFO:tasks.workunit.client.1.smithi067.stdout:8/828: fsync d6/d8/d50/dd4/d4e/f75 0 2024-04-12T05:37:35.017 INFO:tasks.workunit.client.1.smithi067.stdout:2/833: creat d5/da/de/dc4/def/f107 x:0 0 0 2024-04-12T05:37:35.018 INFO:tasks.workunit.client.1.smithi067.stdout:3/789: dwrite dc/d1d/d28/d3e/f4a [0,4194304] 0 2024-04-12T05:37:35.020 INFO:tasks.workunit.client.1.smithi067.stdout:7/903: getdents d0/d23/d80/da0/db4/dd/d79 0 2024-04-12T05:37:35.022 INFO:tasks.workunit.client.1.smithi067.stdout:4/819: mknod d8/da/c111 0 2024-04-12T05:37:35.025 INFO:tasks.workunit.client.1.smithi067.stdout:8/829: unlink d6/def/d9a/fbc 0 2024-04-12T05:37:35.027 INFO:tasks.workunit.client.1.smithi067.stdout:1/880: unlink d4/d9/d12/d2c/f37 0 2024-04-12T05:37:35.028 INFO:tasks.workunit.client.1.smithi067.stdout:2/834: mknod d5/da/de/d14/d34/c108 0 2024-04-12T05:37:35.028 INFO:tasks.workunit.client.1.smithi067.stdout:1/881: dread - d4/d3d/dab/fb1 zero size 2024-04-12T05:37:35.029 INFO:tasks.workunit.client.1.smithi067.stdout:2/835: chown d5/da/de/d14/d34 499466 1 2024-04-12T05:37:35.031 INFO:tasks.workunit.client.1.smithi067.stdout:3/790: dread f1 [0,4194304] 0 2024-04-12T05:37:35.033 INFO:tasks.workunit.client.1.smithi067.stdout:7/904: dread d0/d23/d80/da0/db4/d5/d1d/d43/f49 [0,4194304] 0 2024-04-12T05:37:35.034 INFO:tasks.workunit.client.1.smithi067.stdout:4/820: rename f7 to d8/dba/d5d/d66/d7b/dbd/f112 0 2024-04-12T05:37:35.035 INFO:tasks.workunit.client.1.smithi067.stdout:8/830: symlink d6/d8/d50/dd4/de7/d69/lfa 0 2024-04-12T05:37:35.036 INFO:tasks.workunit.client.1.smithi067.stdout:1/882: dwrite d4/d9/d14/fbc [0,4194304] 0 2024-04-12T05:37:35.038 INFO:tasks.workunit.client.1.smithi067.stdout:8/831: chown d6/d8/d50/dd4/d36/f59 1 1 2024-04-12T05:37:35.038 INFO:tasks.workunit.client.1.smithi067.stdout:2/836: creat d5/d43/d6e/f109 x:0 0 0 2024-04-12T05:37:35.039 INFO:tasks.workunit.client.1.smithi067.stdout:3/791: mkdir dc/d1d/d28/d3e/d53/d76/da7/df1/d118 0 2024-04-12T05:37:35.040 INFO:tasks.workunit.client.1.smithi067.stdout:2/837: readlink d5/da/d3d/lce 0 2024-04-12T05:37:35.042 INFO:tasks.workunit.client.1.smithi067.stdout:7/905: creat d0/d23/d80/da0/db4/d5/d1d/de9/de7/f134 x:0 0 0 2024-04-12T05:37:35.043 INFO:tasks.workunit.client.1.smithi067.stdout:1/883: write d4/d9/d12/dcf/ff9 [791884,50240] 0 2024-04-12T05:37:35.044 INFO:tasks.workunit.client.1.smithi067.stdout:6/859: getdents dd/d21/d64/d6f/d90 0 2024-04-12T05:37:35.045 INFO:tasks.workunit.client.1.smithi067.stdout:8/832: truncate d6/d8/d50/dd4/de7/fa6 1632531 0 2024-04-12T05:37:35.048 INFO:tasks.workunit.client.1.smithi067.stdout:2/838: mkdir d5/d6a/d78/d10a 0 2024-04-12T05:37:35.050 INFO:tasks.workunit.client.1.smithi067.stdout:3/792: creat dc/d1d/d28/d3e/d53/d76/da7/df1/dc8/da3/db9/f119 x:0 0 0 2024-04-12T05:37:35.052 INFO:tasks.workunit.client.1.smithi067.stdout:1/884: dread d4/d9/d3a/f1e [4194304,4194304] 0 2024-04-12T05:37:35.053 INFO:tasks.workunit.client.1.smithi067.stdout:6/860: creat dd/df4/f119 x:0 0 0 2024-04-12T05:37:35.054 INFO:tasks.workunit.client.1.smithi067.stdout:1/885: stat d4/d9/d12/d2c/fac 0 2024-04-12T05:37:35.054 INFO:tasks.workunit.client.1.smithi067.stdout:7/906: dread - d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/dd6/fc4 zero size 2024-04-12T05:37:35.057 INFO:tasks.workunit.client.1.smithi067.stdout:4/821: dread d8/f2b [0,4194304] 0 2024-04-12T05:37:35.063 INFO:tasks.workunit.client.1.smithi067.stdout:3/793: rename dc/de4/ce5 to dc/d50/da6/dbd/c11a 0 2024-04-12T05:37:35.067 INFO:tasks.workunit.client.1.smithi067.stdout:8/833: dread d6/d8/d50/dd4/d4e/fee [0,4194304] 0 2024-04-12T05:37:35.069 INFO:tasks.workunit.client.1.smithi067.stdout:1/886: readlink d4/d9/d4d/l5a 0 2024-04-12T05:37:35.074 INFO:tasks.workunit.client.1.smithi067.stdout:6/861: creat dd/d21/d64/d88/d7b/dcc/d93/f11a x:0 0 0 2024-04-12T05:37:35.078 INFO:tasks.workunit.client.1.smithi067.stdout:7/907: mkdir d0/d23/d80/da0/db4/d5/d4a/de5/d135 0 2024-04-12T05:37:35.080 INFO:tasks.workunit.client.1.smithi067.stdout:4/822: creat d8/da/d1d/d15/d75/daa/f113 x:0 0 0 2024-04-12T05:37:35.083 INFO:tasks.workunit.client.1.smithi067.stdout:7/908: chown d0/d23/d80/da0/db4/dd/d42/d82 201 1 2024-04-12T05:37:35.086 INFO:tasks.workunit.client.1.smithi067.stdout:2/839: dwrite d5/fa0 [0,4194304] 0 2024-04-12T05:37:35.088 INFO:tasks.workunit.client.1.smithi067.stdout:8/834: mknod d6/d8/d8b/cfb 0 2024-04-12T05:37:35.088 INFO:tasks.workunit.client.1.smithi067.stdout:2/840: chown d5/da/de/d14/d34/d4c/dbc/de3 34860981 1 2024-04-12T05:37:35.091 INFO:tasks.workunit.client.1.smithi067.stdout:6/862: creat dd/d21/d64/d6f/d90/f11b x:0 0 0 2024-04-12T05:37:35.095 INFO:tasks.workunit.client.1.smithi067.stdout:2/841: read d5/d6a/f88 [2008089,62107] 0 2024-04-12T05:37:35.096 INFO:tasks.workunit.client.1.smithi067.stdout:7/909: unlink d0/d23/d45/f9e 0 2024-04-12T05:37:35.096 INFO:tasks.workunit.client.1.smithi067.stdout:6/863: rename dd/d21/d64/d88/d7b to dd/d21/d64/d88/d7b/dcc/d93/d11c 22 2024-04-12T05:37:35.099 INFO:tasks.workunit.client.1.smithi067.stdout:1/887: fdatasync d4/d9/d12/d2c/f88 0 2024-04-12T05:37:35.099 INFO:tasks.workunit.client.1.smithi067.stdout:8/835: mkdir d6/def/d9a/dd7/dfc 0 2024-04-12T05:37:35.103 INFO:tasks.workunit.client.1.smithi067.stdout:7/910: truncate d0/d23/d80/da0/db4/d5/d1d/de9/fcc 4393974 0 2024-04-12T05:37:35.110 INFO:tasks.workunit.client.1.smithi067.stdout:2/842: rename d5/da/f3e to d5/da/de/d14/d34/d40/d57/da2/f10b 0 2024-04-12T05:37:35.115 INFO:tasks.workunit.client.1.smithi067.stdout:1/888: symlink d4/d9/d4c/df6/l11a 0 2024-04-12T05:37:35.123 INFO:tasks.workunit.client.1.smithi067.stdout:7/911: fdatasync d0/d23/d80/da0/db4/dd/da2/f28 0 2024-04-12T05:37:35.131 INFO:tasks.workunit.client.1.smithi067.stdout:1/889: fsync d4/d52/d80/fb3 0 2024-04-12T05:37:35.132 INFO:tasks.workunit.client.1.smithi067.stdout:6/864: dread fb [0,4194304] 0 2024-04-12T05:37:35.136 INFO:tasks.workunit.client.1.smithi067.stdout:6/865: chown dd/d21/l4e 49 1 2024-04-12T05:37:35.137 INFO:tasks.workunit.client.1.smithi067.stdout:4/823: dread d8/da/f18 [0,4194304] 0 2024-04-12T05:37:35.139 INFO:tasks.workunit.client.1.smithi067.stdout:2/843: mkdir d5/da/de/d14/d34/d40/d57/d104/d10c 0 2024-04-12T05:37:35.149 INFO:tasks.workunit.client.1.smithi067.stdout:1/890: rename d4/d9/d3a/d51/d55/fe7 to d4/d8f/d110/f11b 0 2024-04-12T05:37:35.155 INFO:tasks.workunit.client.1.smithi067.stdout:4/824: unlink d8/dba/d5d/d66/d7b/dbd/dcc/f102 0 2024-04-12T05:37:35.156 INFO:tasks.workunit.client.1.smithi067.stdout:2/844: symlink d5/da/de/d14/d34/d40/d57/da2/l10d 0 2024-04-12T05:37:35.163 INFO:tasks.workunit.client.1.smithi067.stdout:6/866: mknod dd/d11/d27/d4a/d79/c11d 0 2024-04-12T05:37:35.165 INFO:tasks.workunit.client.1.smithi067.stdout:2/845: write d5/d23/fab [474583,23371] 0 2024-04-12T05:37:35.167 INFO:tasks.workunit.client.1.smithi067.stdout:2/846: readlink d5/l7e 0 2024-04-12T05:37:35.169 INFO:tasks.workunit.client.1.smithi067.stdout:7/912: link d0/d23/d80/da0/db4/d5/d1d/de9/fcc d0/d23/d80/da0/db4/dd/d42/d7e/d34/f136 0 2024-04-12T05:37:35.170 INFO:tasks.workunit.client.1.smithi067.stdout:1/891: truncate d4/f7 761737 0 2024-04-12T05:37:35.170 INFO:tasks.workunit.client.1.smithi067.stdout:8/836: getdents d6/d8/d50/dd4/de7/d69 0 2024-04-12T05:37:35.171 INFO:tasks.workunit.client.1.smithi067.stdout:6/867: mknod dd/d11/d27/d4a/d8b/c11e 0 2024-04-12T05:37:35.173 INFO:tasks.workunit.client.1.smithi067.stdout:1/892: fsync d4/d9/d14/fbc 0 2024-04-12T05:37:35.174 INFO:tasks.workunit.client.1.smithi067.stdout:2/847: rename d5/d6a/f85 to d5/da/de/dc4/f10e 0 2024-04-12T05:37:35.174 INFO:tasks.workunit.client.1.smithi067.stdout:7/913: chown d0/d23/d80/da0/db4 505487 1 2024-04-12T05:37:35.175 INFO:tasks.workunit.client.1.smithi067.stdout:8/837: write d6/def/d9a/f1c [5996861,120682] 0 2024-04-12T05:37:35.177 INFO:tasks.workunit.client.1.smithi067.stdout:1/893: read d4/d9/d14/fbc [3673195,28077] 0 2024-04-12T05:37:35.180 INFO:tasks.workunit.client.1.smithi067.stdout:2/848: mknod d5/da/dae/dba/c10f 0 2024-04-12T05:37:35.181 INFO:tasks.workunit.client.1.smithi067.stdout:8/838: mknod d6/def/d9a/dbb/cfd 0 2024-04-12T05:37:35.183 INFO:tasks.workunit.client.1.smithi067.stdout:2/849: chown d5/da/de/d14/f36 7 1 2024-04-12T05:37:35.183 INFO:tasks.workunit.client.1.smithi067.stdout:7/914: getdents d0/d23/d80/da0/db4/dd/d42/d82/d11f 0 2024-04-12T05:37:35.187 INFO:tasks.workunit.client.1.smithi067.stdout:2/850: chown d5/d11/l15 29056 1 2024-04-12T05:37:35.187 INFO:tasks.workunit.client.1.smithi067.stdout:8/839: creat d6/def/d9a/d12/de9/ffe x:0 0 0 2024-04-12T05:37:35.188 INFO:tasks.workunit.client.1.smithi067.stdout:6/868: getdents dd/d21/d64/d6f/d90/d111 0 2024-04-12T05:37:35.188 INFO:tasks.workunit.client.1.smithi067.stdout:1/894: dwrite d4/fe3 [0,4194304] 0 2024-04-12T05:37:35.190 INFO:tasks.workunit.client.1.smithi067.stdout:2/851: rename d5 to d5/d43/de8/d110 22 2024-04-12T05:37:35.191 INFO:tasks.workunit.client.1.smithi067.stdout:9/907: sync 2024-04-12T05:37:35.195 INFO:tasks.workunit.client.1.smithi067.stdout:6/869: mkdir dd/d21/d64/dcf/dfb/d11f 0 2024-04-12T05:37:35.196 INFO:tasks.workunit.client.1.smithi067.stdout:1/895: write d4/d52/fc7 [484167,66480] 0 2024-04-12T05:37:35.199 INFO:tasks.workunit.client.1.smithi067.stdout:6/870: chown dd/d21/d64/d88/d7b/dcc/d112 13 1 2024-04-12T05:37:35.206 INFO:tasks.workunit.client.1.smithi067.stdout:1/896: stat d4/d9/d12/d2c/f101 0 2024-04-12T05:37:35.210 INFO:tasks.workunit.client.1.smithi067.stdout:7/915: link d0/d23/d80/da0/db4/dd/d42/d82/ld5 d0/d23/d80/da0/db4/d5/d1d/d7d/l137 0 2024-04-12T05:37:35.212 INFO:tasks.workunit.client.1.smithi067.stdout:1/897: mknod d4/d8f/d110/c11c 0 2024-04-12T05:37:35.215 INFO:tasks.workunit.client.1.smithi067.stdout:9/908: dwrite d1/d66/dca/fef [0,4194304] 0 2024-04-12T05:37:35.221 INFO:tasks.workunit.client.1.smithi067.stdout:7/916: mknod d0/d23/d80/da0/db4/dd/c138 0 2024-04-12T05:37:35.224 INFO:tasks.workunit.client.1.smithi067.stdout:8/840: truncate d6/def/d9a/dbb/dd9/fe4 8213808 0 2024-04-12T05:37:35.232 INFO:tasks.workunit.client.1.smithi067.stdout:1/898: dwrite d4/d52/fc7 [0,4194304] 0 2024-04-12T05:37:35.235 INFO:tasks.workunit.client.1.smithi067.stdout:2/852: dwrite d5/fb4 [0,4194304] 0 2024-04-12T05:37:35.239 INFO:tasks.workunit.client.1.smithi067.stdout:6/871: dread dd/d11/f19 [0,4194304] 0 2024-04-12T05:37:35.239 INFO:tasks.workunit.client.1.smithi067.stdout:1/899: mkdir d4/d46/d11d 0 2024-04-12T05:37:35.247 INFO:tasks.workunit.client.1.smithi067.stdout:7/917: dread d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/dd6/f8e [0,4194304] 0 2024-04-12T05:37:35.248 INFO:tasks.workunit.client.1.smithi067.stdout:8/841: dwrite d6/d8/d50/dd4/de7/d69/d5d/f91 [0,4194304] 0 2024-04-12T05:37:35.251 INFO:tasks.workunit.client.1.smithi067.stdout:1/900: rename d4/d9/d12/c109 to d4/d8f/dea/c11e 0 2024-04-12T05:37:35.252 INFO:tasks.workunit.client.1.smithi067.stdout:2/853: mknod d5/da/dae/c111 0 2024-04-12T05:37:35.257 INFO:tasks.workunit.client.1.smithi067.stdout:9/909: dread d1/d28/d21/d45/d75/de6/d8d/dc1/dcb/fe1 [0,4194304] 0 2024-04-12T05:37:35.257 INFO:tasks.workunit.client.1.smithi067.stdout:7/918: creat d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/f139 x:0 0 0 2024-04-12T05:37:35.260 INFO:tasks.workunit.client.1.smithi067.stdout:2/854: getdents d5/da/de/dc4 0 2024-04-12T05:37:35.260 INFO:tasks.workunit.client.1.smithi067.stdout:6/872: dwrite dd/d21/d64/d6f/d90/f9d [0,4194304] 0 2024-04-12T05:37:35.266 INFO:tasks.workunit.client.1.smithi067.stdout:7/919: mkdir d0/d23/d80/da0/db4/d5/d13a 0 2024-04-12T05:37:35.268 INFO:tasks.workunit.client.1.smithi067.stdout:2/855: truncate d5/d23/d93/fc5 803118 0 2024-04-12T05:37:35.269 INFO:tasks.workunit.client.1.smithi067.stdout:9/910: rename d1/d28/d21/d45/d75/de6/d8d/dc1/dcb/fe1 to d1/d28/d21/d45/d75/de6/d8d/dc1/dcb/f120 0 2024-04-12T05:37:35.276 INFO:tasks.workunit.client.1.smithi067.stdout:6/873: dwrite dd/d21/d3f/f40 [4194304,4194304] 0 2024-04-12T05:37:35.277 INFO:tasks.workunit.client.1.smithi067.stdout:9/911: truncate d1/d66/d7a/f107 419533 0 2024-04-12T05:37:35.283 INFO:tasks.workunit.client.1.smithi067.stdout:6/874: creat dd/d11/d27/d4a/df8/f120 x:0 0 0 2024-04-12T05:37:35.287 INFO:tasks.workunit.client.1.smithi067.stdout:9/912: creat d1/d28/d21/d7f/da7/dea/f121 x:0 0 0 2024-04-12T05:37:35.289 INFO:tasks.workunit.client.1.smithi067.stdout:6/875: fdatasync dd/d21/d64/d6f/da7/fb8 0 2024-04-12T05:37:35.291 INFO:tasks.workunit.client.1.smithi067.stdout:9/913: mkdir d1/d28/d21/d45/d75/de6/d8d/d122 0 2024-04-12T05:37:35.292 INFO:tasks.workunit.client.1.smithi067.stdout:9/914: fsync d1/d13/f7c 0 2024-04-12T05:37:35.295 INFO:tasks.workunit.client.1.smithi067.stdout:6/876: dread dd/d21/d3f/f40 [4194304,4194304] 0 2024-04-12T05:37:35.295 INFO:tasks.workunit.client.1.smithi067.stdout:9/915: creat d1/d28/d27/f123 x:0 0 0 2024-04-12T05:37:35.297 INFO:tasks.workunit.client.1.smithi067.stdout:6/877: truncate dd/d21/d64/dcf/f103 130628 0 2024-04-12T05:37:35.297 INFO:tasks.workunit.client.1.smithi067.stdout:9/916: dread d1/d66/d7a/f107 [0,4194304] 0 2024-04-12T05:37:35.304 INFO:tasks.workunit.client.1.smithi067.stdout:6/878: dwrite dd/d21/f24 [4194304,4194304] 0 2024-04-12T05:37:35.305 INFO:tasks.workunit.client.1.smithi067.stdout:6/879: fdatasync dd/d21/d3f/f10d 0 2024-04-12T05:37:35.306 INFO:tasks.workunit.client.1.smithi067.stdout:9/917: rename d1/f6 to d1/d28/d21/d3f/f124 0 2024-04-12T05:37:35.311 INFO:tasks.workunit.client.1.smithi067.stdout:7/920: truncate d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/dd6/ddf/fe2 147513 0 2024-04-12T05:37:35.315 INFO:tasks.workunit.client.1.smithi067.stdout:6/880: mknod dd/d21/d64/dcf/dfb/d11f/c121 0 2024-04-12T05:37:35.319 INFO:tasks.workunit.client.1.smithi067.stdout:7/921: symlink d0/d23/d80/da0/db4/d5/d1d/de9/de1/d129/l13b 0 2024-04-12T05:37:35.321 INFO:tasks.workunit.client.1.smithi067.stdout:6/881: creat dd/d11/d27/f122 x:0 0 0 2024-04-12T05:37:35.326 INFO:tasks.workunit.client.1.smithi067.stdout:9/918: truncate d1/d28/d27/dfa/f63 2303684 0 2024-04-12T05:37:35.333 INFO:tasks.workunit.client.1.smithi067.stdout:7/922: creat d0/d23/d80/da0/dc3/df7/f13c x:0 0 0 2024-04-12T05:37:35.333 INFO:tasks.workunit.client.1.smithi067.stdout:7/923: fdatasync d0/d23/d80/da0/dc3/df7/f13c 0 2024-04-12T05:37:35.420 INFO:tasks.workunit.client.1.smithi067.stdout:3/794: sync 2024-04-12T05:37:35.447 INFO:tasks.workunit.client.1.smithi067.stdout:7/924: creat d0/d23/d80/da0/db4/dd/d42/d82/dfb/f13d x:0 0 0 2024-04-12T05:37:35.452 INFO:tasks.workunit.client.1.smithi067.stdout:9/919: dwrite d1/d28/d21/d45/dff/f110 [0,4194304] 0 2024-04-12T05:37:35.500 INFO:tasks.workunit.client.1.smithi067.stdout:6/882: getdents dd/d21/d64/d6f/da7 0 2024-04-12T05:37:35.507 INFO:tasks.workunit.client.1.smithi067.stdout:6/883: stat dd/d21/d64/d6f/d90/lb0 0 2024-04-12T05:37:35.508 INFO:tasks.workunit.client.1.smithi067.stdout:3/795: link dc/df/l25 dc/df/d27/daa/l11b 0 2024-04-12T05:37:35.510 INFO:tasks.workunit.client.1.smithi067.stdout:7/925: rename d0/d23/d45/l2e to d0/d23/d80/da0/db4/d5/d4a/l13e 0 2024-04-12T05:37:35.511 INFO:tasks.workunit.client.1.smithi067.stdout:6/884: mknod dd/d11/dc1/c123 0 2024-04-12T05:37:35.513 INFO:tasks.workunit.client.1.smithi067.stdout:7/926: dread d0/d23/d80/da0/db4/dd/f123 [0,4194304] 0 2024-04-12T05:37:35.514 INFO:tasks.workunit.client.1.smithi067.stdout:6/885: write dd/d11/d27/ff1 [1334955,105192] 0 2024-04-12T05:37:35.515 INFO:tasks.workunit.client.1.smithi067.stdout:7/927: read - d0/d23/d80/da0/db4/d5/d1d/f110 zero size 2024-04-12T05:37:35.516 INFO:tasks.workunit.client.1.smithi067.stdout:6/886: fsync dd/d21/d64/d88/f96 0 2024-04-12T05:37:35.523 INFO:tasks.workunit.client.1.smithi067.stdout:7/928: mknod d0/d23/d80/da0/deb/c13f 0 2024-04-12T05:37:35.524 INFO:tasks.workunit.client.1.smithi067.stdout:9/920: rename d1/d28/d21/d45/d75/de6/c83 to d1/d28/d21/d45/c125 0 2024-04-12T05:37:35.526 INFO:tasks.workunit.client.1.smithi067.stdout:6/887: mkdir dd/d21/df9/d124 0 2024-04-12T05:37:35.527 INFO:tasks.workunit.client.1.smithi067.stdout:7/929: write d0/d23/d80/da0/db4/dd/f123 [97361,28841] 0 2024-04-12T05:37:35.529 INFO:tasks.workunit.client.1.smithi067.stdout:7/930: readlink d0/d23/d80/da0/db4/d5/d1d/l2a 0 2024-04-12T05:37:35.532 INFO:tasks.workunit.client.1.smithi067.stdout:9/921: creat d1/d13/db7/d118/f126 x:0 0 0 2024-04-12T05:37:35.533 INFO:tasks.workunit.client.1.smithi067.stdout:6/888: symlink dd/d11/d27/d4a/d79/dc3/de5/l125 0 2024-04-12T05:37:35.533 INFO:tasks.workunit.client.1.smithi067.stdout:9/922: fdatasync d1/d13/db7/f11f 0 2024-04-12T05:37:35.534 INFO:tasks.workunit.client.1.smithi067.stdout:4/825: sync 2024-04-12T05:37:35.535 INFO:tasks.workunit.client.1.smithi067.stdout:6/889: write dd/d11/d27/d4a/d79/dc3/ffe [336210,34328] 0 2024-04-12T05:37:35.538 INFO:tasks.workunit.client.1.smithi067.stdout:6/890: write fb [3865186,119578] 0 2024-04-12T05:37:35.539 INFO:tasks.workunit.client.1.smithi067.stdout:7/931: creat d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/dd6/f140 x:0 0 0 2024-04-12T05:37:35.543 INFO:tasks.workunit.client.1.smithi067.stdout:9/923: dwrite d1/d12/f53 [0,4194304] 0 2024-04-12T05:37:35.548 INFO:tasks.workunit.client.1.smithi067.stdout:7/932: rmdir d0/d23/d80/da0/db4/dd/d42/d82/d96/de4 39 2024-04-12T05:37:35.552 INFO:tasks.workunit.client.1.smithi067.stdout:7/933: read d0/d23/d80/da0/db4/d5/d1d/d43/fb9 [2920487,45723] 0 2024-04-12T05:37:35.553 INFO:tasks.workunit.client.1.smithi067.stdout:9/924: dwrite d1/d28/fd1 [0,4194304] 0 2024-04-12T05:37:35.556 INFO:tasks.workunit.client.1.smithi067.stdout:7/934: chown d0/d23/d80/da0/db4/d5/d1d/de9/fc9 48 1 2024-04-12T05:37:35.558 INFO:tasks.workunit.client.1.smithi067.stdout:9/925: write d1/d66/fe9 [1207674,82190] 0 2024-04-12T05:37:35.559 INFO:tasks.workunit.client.1.smithi067.stdout:6/891: dread dd/f17 [0,4194304] 0 2024-04-12T05:37:35.564 INFO:tasks.workunit.client.1.smithi067.stdout:7/935: write d0/d23/d80/da0/db4/dd/d79/f63 [1748481,24419] 0 2024-04-12T05:37:35.567 INFO:tasks.workunit.client.1.smithi067.stdout:7/936: chown d0/d23/d80/da0/db4/d5/d4a/fa9 7 1 2024-04-12T05:37:35.568 INFO:tasks.workunit.client.1.smithi067.stdout:6/892: dwrite dd/d21/d64/d6f/d90/f9d [0,4194304] 0 2024-04-12T05:37:35.570 INFO:tasks.workunit.client.1.smithi067.stdout:9/926: getdents d1/d5f 0 2024-04-12T05:37:35.576 INFO:tasks.workunit.client.1.smithi067.stdout:6/893: mkdir dd/d21/d64/d6f/d90/d98/d126 0 2024-04-12T05:37:35.578 INFO:tasks.workunit.client.1.smithi067.stdout:6/894: dread - dd/d21/d64/d88/d7b/dcd/f10e zero size 2024-04-12T05:37:35.584 INFO:tasks.workunit.client.1.smithi067.stdout:6/895: getdents dd/d11/d27/d4a/d79/dc3/d56/de3 0 2024-04-12T05:37:35.586 INFO:tasks.workunit.client.1.smithi067.stdout:6/896: dread - dd/d21/d64/d88/d7b/dcd/f10e zero size 2024-04-12T05:37:35.590 INFO:tasks.workunit.client.1.smithi067.stdout:6/897: dread dd/d11/d27/d4a/d51/fd0 [0,4194304] 0 2024-04-12T05:37:35.596 INFO:tasks.workunit.client.1.smithi067.stdout:3/796: write dc/df/d27/fa8 [469248,54744] 0 2024-04-12T05:37:35.604 INFO:tasks.workunit.client.1.smithi067.stdout:9/927: mknod d1/d28/d21/d45/d75/de6/d8d/dc1/d11e/c127 0 2024-04-12T05:37:35.607 INFO:tasks.workunit.client.1.smithi067.stdout:9/928: symlink d1/d28/d21/d7f/da7/l128 0 2024-04-12T05:37:35.609 INFO:tasks.workunit.client.1.smithi067.stdout:9/929: mknod d1/d13/c129 0 2024-04-12T05:37:35.612 INFO:tasks.workunit.client.1.smithi067.stdout:7/937: write d0/d67/d57/fa7 [2590017,119166] 0 2024-04-12T05:37:35.645 INFO:tasks.workunit.client.1.smithi067.stdout:8/842: sync 2024-04-12T05:37:35.646 INFO:tasks.workunit.client.1.smithi067.stdout:1/901: sync 2024-04-12T05:37:35.646 INFO:tasks.workunit.client.1.smithi067.stdout:2/856: sync 2024-04-12T05:37:35.652 INFO:tasks.workunit.client.1.smithi067.stdout:9/930: write d1/d13/fb9 [6202,42912] 0 2024-04-12T05:37:35.653 INFO:tasks.workunit.client.1.smithi067.stdout:1/902: chown d4/dd2/fe5 80 1 2024-04-12T05:37:35.655 INFO:tasks.workunit.client.1.smithi067.stdout:8/843: fsync d6/def/d9a/f5f 0 2024-04-12T05:37:35.657 INFO:tasks.workunit.client.1.smithi067.stdout:9/931: mkdir d1/d5f/d12a 0 2024-04-12T05:37:35.659 INFO:tasks.workunit.client.1.smithi067.stdout:3/797: dwrite dc/f18 [0,4194304] 0 2024-04-12T05:37:35.660 INFO:tasks.workunit.client.1.smithi067.stdout:2/857: write d5/d43/f44 [443067,17917] 0 2024-04-12T05:37:35.663 INFO:tasks.workunit.client.1.smithi067.stdout:9/932: truncate d1/d13/db7/fd4 1362732 0 2024-04-12T05:37:35.665 INFO:tasks.workunit.client.1.smithi067.stdout:3/798: fsync dc/d1d/d28/d3e/d69/d113/fcd 0 2024-04-12T05:37:35.667 INFO:tasks.workunit.client.1.smithi067.stdout:8/844: creat d6/def/d9a/dd7/d82/fff x:0 0 0 2024-04-12T05:37:35.667 INFO:tasks.workunit.client.1.smithi067.stdout:2/858: mkdir d5/da/d3d/df7/d112 0 2024-04-12T05:37:35.673 INFO:tasks.workunit.client.1.smithi067.stdout:8/845: mknod d6/d86/c100 0 2024-04-12T05:37:35.675 INFO:tasks.workunit.client.1.smithi067.stdout:2/859: creat d5/f113 x:0 0 0 2024-04-12T05:37:35.677 INFO:tasks.workunit.client.1.smithi067.stdout:7/938: dread d0/d23/d45/f1a [4194304,4194304] 0 2024-04-12T05:37:35.682 INFO:tasks.workunit.client.1.smithi067.stdout:9/933: dwrite d1/ff9 [0,4194304] 0 2024-04-12T05:37:35.683 INFO:tasks.workunit.client.1.smithi067.stdout:1/903: dwrite d4/d9/d12/f2e [0,4194304] 0 2024-04-12T05:37:35.685 INFO:tasks.workunit.client.1.smithi067.stdout:7/939: symlink d0/d23/d80/da0/db4/dd/d42/d7e/d34/l141 0 2024-04-12T05:37:35.687 INFO:tasks.workunit.client.1.smithi067.stdout:1/904: readlink d4/d46/lc0 0 2024-04-12T05:37:35.688 INFO:tasks.workunit.client.1.smithi067.stdout:2/860: symlink d5/da/de/d14/d34/d40/d57/d96/l114 0 2024-04-12T05:37:35.688 INFO:tasks.workunit.client.1.smithi067.stdout:9/934: stat d1/d12/d15/fb0 0 2024-04-12T05:37:35.690 INFO:tasks.workunit.client.1.smithi067.stdout:9/935: chown d1/d5f 6282 1 2024-04-12T05:37:35.691 INFO:tasks.workunit.client.1.smithi067.stdout:7/940: chown d0/d23/d80/da0/db4/dd/d42/d7e/f65 450836410 1 2024-04-12T05:37:35.695 INFO:tasks.workunit.client.1.smithi067.stdout:8/846: dwrite d6/d8/d50/dd4/d36/f59 [0,4194304] 0 2024-04-12T05:37:35.696 INFO:tasks.workunit.client.1.smithi067.stdout:3/799: dwrite dc/d1d/d26/f2f [0,4194304] 0 2024-04-12T05:37:35.697 INFO:tasks.workunit.client.1.smithi067.stdout:2/861: dread - d5/da/de/d14/d46/d90/fd4 zero size 2024-04-12T05:37:35.699 INFO:tasks.workunit.client.1.smithi067.stdout:7/941: readlink d0/d23/d80/da0/db4/d5/d4a/l105 0 2024-04-12T05:37:35.708 INFO:tasks.workunit.client.1.smithi067.stdout:4/826: sync 2024-04-12T05:37:35.709 INFO:tasks.workunit.client.1.smithi067.stdout:9/936: link d1/d28/d21/d45/lf0 d1/d12/d76/l12b 0 2024-04-12T05:37:35.710 INFO:tasks.workunit.client.1.smithi067.stdout:8/847: mknod d6/def/d9a/d12/de9/df9/c101 0 2024-04-12T05:37:35.711 INFO:tasks.workunit.client.1.smithi067.stdout:1/905: dwrite d4/d3d/dab/f39 [0,4194304] 0 2024-04-12T05:37:35.715 INFO:tasks.workunit.client.1.smithi067.stdout:8/848: read - d6/d8/f6a zero size 2024-04-12T05:37:35.716 INFO:tasks.workunit.client.1.smithi067.stdout:7/942: creat d0/d23/d80/da0/db4/dd/d42/f142 x:0 0 0 2024-04-12T05:37:35.717 INFO:tasks.workunit.client.1.smithi067.stdout:3/800: getdents dc/d1d/d28/d3e 0 2024-04-12T05:37:35.719 INFO:tasks.workunit.client.1.smithi067.stdout:8/849: fsync d6/d8/d50/fe6 0 2024-04-12T05:37:35.720 INFO:tasks.workunit.client.1.smithi067.stdout:9/937: creat d1/d28/d21/d45/d75/f12c x:0 0 0 2024-04-12T05:37:35.721 INFO:tasks.workunit.client.1.smithi067.stdout:1/906: truncate d4/d9/d4c/fcd 913379 0 2024-04-12T05:37:35.722 INFO:tasks.workunit.client.1.smithi067.stdout:3/801: rmdir dc/d1d/d26 39 2024-04-12T05:37:35.722 INFO:tasks.workunit.client.1.smithi067.stdout:7/943: symlink d0/d23/d80/da0/deb/l143 0 2024-04-12T05:37:35.724 INFO:tasks.workunit.client.1.smithi067.stdout:8/850: readlink d6/d8/d50/dd4/d4e/d9e/la5 0 2024-04-12T05:37:35.726 INFO:tasks.workunit.client.1.smithi067.stdout:4/827: dwrite d8/da/d1d/d27/d91/ff4 [4194304,4194304] 0 2024-04-12T05:37:35.726 INFO:tasks.workunit.client.1.smithi067.stdout:1/907: chown d4/d9/d3a/d51/d55/dad 39824756 1 2024-04-12T05:37:35.727 INFO:tasks.workunit.client.1.smithi067.stdout:9/938: chown d1/d28/d21/d45/d105/l4c 308462 1 2024-04-12T05:37:35.729 INFO:tasks.workunit.client.1.smithi067.stdout:8/851: mknod d6/d8/d8b/c102 0 2024-04-12T05:37:35.731 INFO:tasks.workunit.client.1.smithi067.stdout:4/828: fsync d8/dba/d5d/d66/d7b/dbd/f10e 0 2024-04-12T05:37:35.732 INFO:tasks.workunit.client.1.smithi067.stdout:1/908: unlink d4/d52/d80/f65 0 2024-04-12T05:37:35.735 INFO:tasks.workunit.client.1.smithi067.stdout:8/852: chown d6/d8/d50/dd4/d4e/l7c 356 1 2024-04-12T05:37:35.737 INFO:tasks.workunit.client.1.smithi067.stdout:1/909: fdatasync d4/fca 0 2024-04-12T05:37:35.740 INFO:tasks.workunit.client.1.smithi067.stdout:8/853: mknod d6/def/d9a/d12/c103 0 2024-04-12T05:37:35.744 INFO:tasks.workunit.client.1.smithi067.stdout:9/939: dwrite d1/d12/d15/f11b [0,4194304] 0 2024-04-12T05:37:35.745 INFO:tasks.workunit.client.1.smithi067.stdout:8/854: mkdir d6/d8/d50/dd4/de7/d104 0 2024-04-12T05:37:35.747 INFO:tasks.workunit.client.1.smithi067.stdout:9/940: fsync d1/d28/d27/dfa/ff4 0 2024-04-12T05:37:35.747 INFO:tasks.workunit.client.1.smithi067.stdout:8/855: fsync d6/d8/d50/dd4/de7/d69/f79 0 2024-04-12T05:37:35.749 INFO:tasks.workunit.client.1.smithi067.stdout:4/829: creat d8/da/d9f/d68/f114 x:0 0 0 2024-04-12T05:37:35.749 INFO:tasks.workunit.client.1.smithi067.stdout:9/941: stat d1/d28/d21/d45/d75/f7d 0 2024-04-12T05:37:35.750 INFO:tasks.workunit.client.1.smithi067.stdout:6/898: sync 2024-04-12T05:37:35.752 INFO:tasks.workunit.client.1.smithi067.stdout:8/856: creat d6/d8/d8b/da1/f105 x:0 0 0 2024-04-12T05:37:35.753 INFO:tasks.workunit.client.1.smithi067.stdout:1/910: read d4/d9/d3a/f8b [4164738,123558] 0 2024-04-12T05:37:35.755 INFO:tasks.workunit.client.1.smithi067.stdout:7/944: dread d0/d23/d80/da0/db4/dd/d42/d7e/f94 [0,4194304] 0 2024-04-12T05:37:35.756 INFO:tasks.workunit.client.1.smithi067.stdout:8/857: write d6/d8/d50/dd4/d36/f54 [3639366,19641] 0 2024-04-12T05:37:35.758 INFO:tasks.workunit.client.1.smithi067.stdout:8/858: chown d6/def/d9a/dd7/fa0 51 1 2024-04-12T05:37:35.759 INFO:tasks.workunit.client.1.smithi067.stdout:7/945: write d0/d23/d80/da0/db4/dd/d42/f142 [677135,126978] 0 2024-04-12T05:37:35.759 INFO:tasks.workunit.client.1.smithi067.stdout:9/942: creat d1/d28/d21/d45/d75/de6/d8d/f12d x:0 0 0 2024-04-12T05:37:35.763 INFO:tasks.workunit.client.1.smithi067.stdout:1/911: mknod d4/d9/d3a/d51/c11f 0 2024-04-12T05:37:35.764 INFO:tasks.workunit.client.1.smithi067.stdout:2/862: truncate d5/d6a/fa6 3737714 0 2024-04-12T05:37:35.764 INFO:tasks.workunit.client.1.smithi067.stdout:9/943: read d1/d28/d21/d3f/d4f/faf [7733126,95577] 0 2024-04-12T05:37:35.767 INFO:tasks.workunit.client.1.smithi067.stdout:8/859: dread d6/d8/d8b/fae [0,4194304] 0 2024-04-12T05:37:35.767 INFO:tasks.workunit.client.1.smithi067.stdout:4/830: link d8/dba/d88/ded/cf0 d8/dba/d88/da0/c115 0 2024-04-12T05:37:35.767 INFO:tasks.workunit.client.1.smithi067.stdout:2/863: read d5/da/dae/fbb [247821,28627] 0 2024-04-12T05:37:35.770 INFO:tasks.workunit.client.1.smithi067.stdout:3/802: truncate dc/d1d/d26/f5e 3356023 0 2024-04-12T05:37:35.771 INFO:tasks.workunit.client.1.smithi067.stdout:4/831: write d8/dba/db8/f103 [1043954,58363] 0 2024-04-12T05:37:35.771 INFO:tasks.workunit.client.1.smithi067.stdout:1/912: dread d4/d9/d12/d59/f64 [4194304,4194304] 0 2024-04-12T05:37:35.772 INFO:tasks.workunit.client.1.smithi067.stdout:3/803: fdatasync dc/d1d/d28/d3e/d69/d113/fcd 0 2024-04-12T05:37:35.778 INFO:tasks.workunit.client.1.smithi067.stdout:6/899: write dd/d11/f91 [2385757,17321] 0 2024-04-12T05:37:35.779 INFO:tasks.workunit.client.1.smithi067.stdout:9/944: fdatasync d1/d28/d21/d3f/d4f/faf 0 2024-04-12T05:37:35.781 INFO:tasks.workunit.client.1.smithi067.stdout:8/860: write d6/d8/d50/dd4/f62 [2012188,39290] 0 2024-04-12T05:37:35.785 INFO:tasks.workunit.client.1.smithi067.stdout:9/945: chown d1/d28/d21/d45/d75/de6/d8d/dc1/f102 62 1 2024-04-12T05:37:35.785 INFO:tasks.workunit.client.1.smithi067.stdout:4/832: symlink d8/da/d1d/d15/d58/l116 0 2024-04-12T05:37:35.785 INFO:tasks.workunit.client.1.smithi067.stdout:8/861: read d6/d8/d50/dd4/de7/d69/f79 [960485,22330] 0 2024-04-12T05:37:35.785 INFO:tasks.workunit.client.1.smithi067.stdout:1/913: rename d4/d9/d3a/d51/d55/dad/f10c to d4/d3d/dbf/f120 0 2024-04-12T05:37:35.786 INFO:tasks.workunit.client.1.smithi067.stdout:3/804: creat dc/d1d/d28/d3e/d53/d76/f11c x:0 0 0 2024-04-12T05:37:35.791 INFO:tasks.workunit.client.1.smithi067.stdout:7/946: creat d0/d23/d80/da0/db4/dd/d42/d71/f144 x:0 0 0 2024-04-12T05:37:35.796 INFO:tasks.workunit.client.1.smithi067.stdout:9/946: write d1/d28/d27/dfa/f63 [824415,111999] 0 2024-04-12T05:37:35.800 INFO:tasks.workunit.client.1.smithi067.stdout:1/914: dwrite d4/d9/d3a/d51/f82 [0,4194304] 0 2024-04-12T05:37:35.802 INFO:tasks.workunit.client.1.smithi067.stdout:8/862: dwrite d6/d8/d50/dd4/d36/f7f [0,4194304] 0 2024-04-12T05:37:35.804 INFO:tasks.workunit.client.1.smithi067.stdout:2/864: dwrite d5/f12 [0,4194304] 0 2024-04-12T05:37:35.807 INFO:tasks.workunit.client.1.smithi067.stdout:4/833: write d8/da/d9f/df7/fc2 [219108,90971] 0 2024-04-12T05:37:35.810 INFO:tasks.workunit.client.1.smithi067.stdout:6/900: creat dd/d21/d64/d6f/d90/f127 x:0 0 0 2024-04-12T05:37:35.811 INFO:tasks.workunit.client.1.smithi067.stdout:3/805: write dc/d1d/d28/d3e/d53/d78/ff2 [1566416,129478] 0 2024-04-12T05:37:35.813 INFO:tasks.workunit.client.1.smithi067.stdout:3/806: chown dc/d1d/d28/d3e/d53/d94 919 1 2024-04-12T05:37:35.813 INFO:tasks.workunit.client.1.smithi067.stdout:9/947: rmdir d1/d28/d5e 39 2024-04-12T05:37:35.814 INFO:tasks.workunit.client.1.smithi067.stdout:7/947: symlink d0/d23/d80/da0/l145 0 2024-04-12T05:37:35.817 INFO:tasks.workunit.client.1.smithi067.stdout:4/834: readlink d8/d9/lb 0 2024-04-12T05:37:35.821 INFO:tasks.workunit.client.1.smithi067.stdout:3/807: rmdir dc/d50 39 2024-04-12T05:37:35.823 INFO:tasks.workunit.client.1.smithi067.stdout:9/948: rename d1/d28/d21/d3f/d4f/f57 to d1/d28/d21/d45/d75/de6/d8d/dc1/de3/f12e 0 2024-04-12T05:37:35.826 INFO:tasks.workunit.client.1.smithi067.stdout:3/808: creat dc/d1d/d28/d3e/d8a/f11d x:0 0 0 2024-04-12T05:37:35.827 INFO:tasks.workunit.client.1.smithi067.stdout:6/901: creat dd/d21/d64/d6f/d90/d98/d126/f128 x:0 0 0 2024-04-12T05:37:35.828 INFO:tasks.workunit.client.1.smithi067.stdout:9/949: write d1/d66/d8e/ff8 [560664,37813] 0 2024-04-12T05:37:35.832 INFO:tasks.workunit.client.1.smithi067.stdout:2/865: dread d5/d6a/f88 [0,4194304] 0 2024-04-12T05:37:35.832 INFO:tasks.workunit.client.1.smithi067.stdout:8/863: write d6/d8/f6a [260929,100344] 0 2024-04-12T05:37:35.833 INFO:tasks.workunit.client.1.smithi067.stdout:1/915: write d4/d9/d14/f8a [6595512,50377] 0 2024-04-12T05:37:35.836 INFO:tasks.workunit.client.1.smithi067.stdout:7/948: link d0/d67/c35 d0/d23/d80/da0/db4/d5/d4a/c146 0 2024-04-12T05:37:35.837 INFO:tasks.workunit.client.1.smithi067.stdout:4/835: dwrite d8/da/f18 [4194304,4194304] 0 2024-04-12T05:37:35.839 INFO:tasks.workunit.client.1.smithi067.stdout:9/950: truncate d1/d66/d8e/ff8 995934 0 2024-04-12T05:37:35.839 INFO:tasks.workunit.client.1.smithi067.stdout:4/836: chown d8/da/d9f 16 1 2024-04-12T05:37:35.845 INFO:tasks.workunit.client.1.smithi067.stdout:6/902: chown dd/d21/d64/d88/d7b/dcc/d93/cef 1 1 2024-04-12T05:37:35.847 INFO:tasks.workunit.client.1.smithi067.stdout:2/866: creat d5/da/de/f115 x:0 0 0 2024-04-12T05:37:35.847 INFO:tasks.workunit.client.1.smithi067.stdout:3/809: write dc/d1d/d28/d3e/d53/d76/da7/df1/dc8/da3/fec [351514,30937] 0 2024-04-12T05:37:35.849 INFO:tasks.workunit.client.1.smithi067.stdout:4/837: read d8/dba/d5d/f8f [138875,79356] 0 2024-04-12T05:37:35.851 INFO:tasks.workunit.client.1.smithi067.stdout:6/903: mkdir dd/d11/d27/d4a/df8/d129 0 2024-04-12T05:37:35.853 INFO:tasks.workunit.client.1.smithi067.stdout:7/949: unlink d0/d23/cc5 0 2024-04-12T05:37:35.853 INFO:tasks.workunit.client.1.smithi067.stdout:3/810: chown dc/d1d/d28/d3e/cc2 9824 1 2024-04-12T05:37:35.855 INFO:tasks.workunit.client.1.smithi067.stdout:1/916: write d4/d9/d12/d2c/f8d [4783511,73020] 0 2024-04-12T05:37:35.862 INFO:tasks.workunit.client.1.smithi067.stdout:2/867: truncate d5/d11/f32 932249 0 2024-04-12T05:37:35.863 INFO:tasks.workunit.client.1.smithi067.stdout:6/904: creat dd/d21/d64/d88/d7b/dcc/f12a x:0 0 0 2024-04-12T05:37:35.863 INFO:tasks.workunit.client.1.smithi067.stdout:8/864: dread d6/d8/d50/dd4/f35 [0,4194304] 0 2024-04-12T05:37:35.869 INFO:tasks.workunit.client.1.smithi067.stdout:9/951: dwrite d1/d28/d21/d45/d105/f8c [0,4194304] 0 2024-04-12T05:37:35.870 INFO:tasks.workunit.client.1.smithi067.stdout:1/917: write d4/d3d/d7b/f94 [18156,112671] 0 2024-04-12T05:37:35.873 INFO:tasks.workunit.client.1.smithi067.stdout:3/811: link dc/df/d27/f3f dc/d1d/d28/d3e/d69/dbc/f11e 0 2024-04-12T05:37:35.875 INFO:tasks.workunit.client.1.smithi067.stdout:2/868: mkdir d5/da/de/d116 0 2024-04-12T05:37:35.878 INFO:tasks.workunit.client.1.smithi067.stdout:4/838: dread d8/da/d8b/d45/f6e [0,4194304] 0 2024-04-12T05:37:35.885 INFO:tasks.workunit.client.1.smithi067.stdout:3/812: creat dc/d1d/d28/d3e/f11f x:0 0 0 2024-04-12T05:37:35.885 INFO:tasks.workunit.client.1.smithi067.stdout:9/952: dread d1/d13/db7/fd4 [0,4194304] 0 2024-04-12T05:37:35.885 INFO:tasks.workunit.client.1.smithi067.stdout:8/865: dread d6/def/d9a/d12/dd8/fdb [0,4194304] 0 2024-04-12T05:37:35.886 INFO:tasks.workunit.client.1.smithi067.stdout:6/905: dwrite f9 [0,4194304] 0 2024-04-12T05:37:35.888 INFO:tasks.workunit.client.1.smithi067.stdout:4/839: symlink d8/dba/d88/da0/de7/l117 0 2024-04-12T05:37:35.889 INFO:tasks.workunit.client.1.smithi067.stdout:2/869: unlink d5/da/de/dc4/dc6/c101 0 2024-04-12T05:37:35.894 INFO:tasks.workunit.client.1.smithi067.stdout:1/918: truncate d4/dd2/f112 4176620 0 2024-04-12T05:37:35.895 INFO:tasks.workunit.client.1.smithi067.stdout:8/866: rmdir d6/d8/d50 39 2024-04-12T05:37:35.898 INFO:tasks.workunit.client.1.smithi067.stdout:7/950: dread d0/d23/d80/da0/db4/dd/da2/df8/f124 [0,4194304] 0 2024-04-12T05:37:35.906 INFO:tasks.workunit.client.1.smithi067.stdout:1/919: stat d4/d9/d3a/d51/d55/dad 0 2024-04-12T05:37:35.906 INFO:tasks.workunit.client.1.smithi067.stdout:3/813: write dc/d1d/d28/d3e/f6d [2317343,85270] 0 2024-04-12T05:37:35.906 INFO:tasks.workunit.client.1.smithi067.stdout:8/867: dread - d6/def/d9a/dd7/fe2 zero size 2024-04-12T05:37:35.906 INFO:tasks.workunit.client.1.smithi067.stdout:6/906: dwrite dd/d21/d64/d88/d7b/dcc/d93/fd8 [0,4194304] 0 2024-04-12T05:37:35.910 INFO:tasks.workunit.client.1.smithi067.stdout:4/840: truncate d8/da/d1d/d15/d58/f59 5118447 0 2024-04-12T05:37:35.911 INFO:tasks.workunit.client.1.smithi067.stdout:3/814: mkdir dc/d1d/d28/d3e/d69/d120 0 2024-04-12T05:37:35.913 INFO:tasks.workunit.client.1.smithi067.stdout:1/920: write d4/d9/d14/f2a [455244,83304] 0 2024-04-12T05:37:35.914 INFO:tasks.workunit.client.1.smithi067.stdout:7/951: dwrite d0/d23/d45/f73 [0,4194304] 0 2024-04-12T05:37:35.916 INFO:tasks.workunit.client.1.smithi067.stdout:3/815: dread - dc/df/d27/daa/fb3 zero size 2024-04-12T05:37:35.916 INFO:tasks.workunit.client.1.smithi067.stdout:8/868: fdatasync d6/d8/d50/dd4/f35 0 2024-04-12T05:37:35.917 INFO:tasks.workunit.client.1.smithi067.stdout:6/907: unlink dd/d21/d64/d88/d7b/f6e 0 2024-04-12T05:37:35.917 INFO:tasks.workunit.client.1.smithi067.stdout:9/953: getdents d1/d28/d21/d45/d75/de6/d8d/dc1/de3 0 2024-04-12T05:37:35.923 INFO:tasks.workunit.client.1.smithi067.stdout:2/870: dread d5/da/de/f13 [0,4194304] 0 2024-04-12T05:37:35.923 INFO:tasks.workunit.client.1.smithi067.stdout:4/841: truncate d8/dba/d5d/d66/fa4 669055 0 2024-04-12T05:37:35.924 INFO:tasks.workunit.client.1.smithi067.stdout:9/954: stat d1/d28/d21/d3f/d4f/f11d 0 2024-04-12T05:37:35.926 INFO:tasks.workunit.client.1.smithi067.stdout:1/921: rmdir d4/d52/d80 39 2024-04-12T05:37:35.929 INFO:tasks.workunit.client.1.smithi067.stdout:9/955: unlink d1/d28/d21/d3f/d4f/lde 0 2024-04-12T05:37:35.931 INFO:tasks.workunit.client.1.smithi067.stdout:8/869: truncate d6/d8/d50/dd4/d36/f7f 4962013 0 2024-04-12T05:37:35.931 INFO:tasks.workunit.client.1.smithi067.stdout:4/842: symlink d8/dba/d5d/d66/d96/l118 0 2024-04-12T05:37:35.932 INFO:tasks.workunit.client.1.smithi067.stdout:8/870: fdatasync d6/d8/d50/dd4/fe1 0 2024-04-12T05:37:35.933 INFO:tasks.workunit.client.1.smithi067.stdout:3/816: truncate dc/df/d27/f6a 4847375 0 2024-04-12T05:37:35.934 INFO:tasks.workunit.client.1.smithi067.stdout:9/956: unlink d1/d12/d15/fa9 0 2024-04-12T05:37:35.935 INFO:tasks.workunit.client.1.smithi067.stdout:8/871: stat d6/def/d9a/f1c 0 2024-04-12T05:37:35.936 INFO:tasks.workunit.client.1.smithi067.stdout:3/817: creat dc/d1d/d28/d3e/d8a/f121 x:0 0 0 2024-04-12T05:37:35.938 INFO:tasks.workunit.client.1.smithi067.stdout:7/952: truncate d0/d23/d80/da0/db4/dd/d42/d7e/d34/f136 1247477 0 2024-04-12T05:37:35.941 INFO:tasks.workunit.client.1.smithi067.stdout:3/818: write dc/d1d/d26/f2f [2282395,55006] 0 2024-04-12T05:37:35.942 INFO:tasks.workunit.client.1.smithi067.stdout:9/957: fsync d1/d13/db7/fd4 0 2024-04-12T05:37:35.942 INFO:tasks.workunit.client.1.smithi067.stdout:4/843: write d8/da/d1d/d15/d58/f78 [3983378,27884] 0 2024-04-12T05:37:35.946 INFO:tasks.workunit.client.1.smithi067.stdout:2/871: dread d5/d23/d93/f98 [0,4194304] 0 2024-04-12T05:37:35.946 INFO:tasks.workunit.client.1.smithi067.stdout:8/872: unlink d6/d8/d50/dd4/d4e/cf6 0 2024-04-12T05:37:35.948 INFO:tasks.workunit.client.1.smithi067.stdout:1/922: write d4/d3d/d7b/f95 [1063179,73293] 0 2024-04-12T05:37:35.954 INFO:tasks.workunit.client.1.smithi067.stdout:6/908: dwrite dd/d11/d27/d4a/d8b/fbd [0,4194304] 0 2024-04-12T05:37:35.956 INFO:tasks.workunit.client.1.smithi067.stdout:4/844: write d8/d9/f21 [4676667,113605] 0 2024-04-12T05:37:35.957 INFO:tasks.workunit.client.1.smithi067.stdout:8/873: creat d6/d8/d8b/da1/f106 x:0 0 0 2024-04-12T05:37:35.958 INFO:tasks.workunit.client.1.smithi067.stdout:2/872: mknod d5/da/de/d39/dde/c117 0 2024-04-12T05:37:35.959 INFO:tasks.workunit.client.1.smithi067.stdout:3/819: symlink dc/d1d/d28/d3e/d53/d76/da7/df1/dc8/da3/db9/l122 0 2024-04-12T05:37:35.961 INFO:tasks.workunit.client.1.smithi067.stdout:1/923: dread d4/f7 [0,4194304] 0 2024-04-12T05:37:35.963 INFO:tasks.workunit.client.1.smithi067.stdout:8/874: mknod d6/def/d9a/dbb/c107 0 2024-04-12T05:37:35.964 INFO:tasks.workunit.client.1.smithi067.stdout:7/953: write d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/dd6/f8e [970908,82777] 0 2024-04-12T05:37:35.964 INFO:tasks.workunit.client.1.smithi067.stdout:6/909: truncate dd/d11/d27/d4a/d79/dc3/d56/f69 3389733 0 2024-04-12T05:37:35.965 INFO:tasks.workunit.client.1.smithi067.stdout:3/820: write dc/df/df4/ff5 [679767,27060] 0 2024-04-12T05:37:35.965 INFO:tasks.workunit.client.1.smithi067.stdout:2/873: chown d5/da/de/d14/d46/d90/cc3 109 1 2024-04-12T05:37:35.966 INFO:tasks.workunit.client.1.smithi067.stdout:1/924: creat d4/d9/d12/d6f/f121 x:0 0 0 2024-04-12T05:37:35.968 INFO:tasks.workunit.client.1.smithi067.stdout:2/874: fsync d5/d6a/d78/fd0 0 2024-04-12T05:37:35.970 INFO:tasks.workunit.client.1.smithi067.stdout:1/925: write d4/dd2/fe5 [871174,122493] 0 2024-04-12T05:37:35.973 INFO:tasks.workunit.client.1.smithi067.stdout:7/954: creat d0/d23/d80/da0/db4/d5/d4a/de5/d10c/f147 x:0 0 0 2024-04-12T05:37:35.974 INFO:tasks.workunit.client.1.smithi067.stdout:3/821: rename dc/df/d8d/f10e to dc/d1d/d28/d3e/d53/d76/da7/df1/dc8/da3/dde/f123 0 2024-04-12T05:37:35.979 INFO:tasks.workunit.client.1.smithi067.stdout:6/910: write dd/d11/d27/f54 [1707320,81115] 0 2024-04-12T05:37:35.984 INFO:tasks.workunit.client.1.smithi067.stdout:2/875: getdents d5/d6a/d8f 0 2024-04-12T05:37:35.988 INFO:tasks.workunit.client.1.smithi067.stdout:7/955: creat d0/d23/d45/f148 x:0 0 0 2024-04-12T05:37:35.989 INFO:tasks.workunit.client.1.smithi067.stdout:6/911: symlink dd/d21/d64/d88/d7b/dcc/l12b 0 2024-04-12T05:37:35.990 INFO:tasks.workunit.client.1.smithi067.stdout:1/926: dwrite d4/d3d/dab/fb1 [0,4194304] 0 2024-04-12T05:37:35.992 INFO:tasks.workunit.client.1.smithi067.stdout:4/845: dread d8/d39/f79 [0,4194304] 0 2024-04-12T05:37:35.995 INFO:tasks.workunit.client.1.smithi067.stdout:6/912: readlink dd/la4 0 2024-04-12T05:37:35.995 INFO:tasks.workunit.client.1.smithi067.stdout:3/822: dread dc/d1d/d26/f2f [0,4194304] 0 2024-04-12T05:37:36.001 INFO:tasks.workunit.client.1.smithi067.stdout:2/876: write d5/da/de/d14/d34/d40/fcb [4867528,73679] 0 2024-04-12T05:37:36.003 INFO:tasks.workunit.client.1.smithi067.stdout:6/913: write dd/d11/d27/ff3 [2287399,71706] 0 2024-04-12T05:37:36.004 INFO:tasks.workunit.client.1.smithi067.stdout:3/823: creat dc/d50/da6/dbd/f124 x:0 0 0 2024-04-12T05:37:36.006 INFO:tasks.workunit.client.1.smithi067.stdout:4/846: creat d8/dba/d5d/d66/f119 x:0 0 0 2024-04-12T05:37:36.008 INFO:tasks.workunit.client.1.smithi067.stdout:1/927: dwrite d4/d9/d3a/d51/d55/fc8 [0,4194304] 0 2024-04-12T05:37:36.009 INFO:tasks.workunit.client.1.smithi067.stdout:6/914: stat dd/d21/d64/d88/d7b 0 2024-04-12T05:37:36.012 INFO:tasks.workunit.client.1.smithi067.stdout:4/847: readlink d8/da/d1d/d15/d75/daa/l80 0 2024-04-12T05:37:36.015 INFO:tasks.workunit.client.1.smithi067.stdout:6/915: fdatasync dd/d11/d27/d4a/d79/dc3/de5/fc2 0 2024-04-12T05:37:36.017 INFO:tasks.workunit.client.1.smithi067.stdout:6/916: stat f6 0 2024-04-12T05:37:36.023 INFO:tasks.workunit.client.1.smithi067.stdout:2/877: truncate d5/d11/f29 2166274 0 2024-04-12T05:37:36.023 INFO:tasks.workunit.client.1.smithi067.stdout:3/824: dwrite dc/d1d/d28/d3e/f67 [4194304,4194304] 0 2024-04-12T05:37:36.026 INFO:tasks.workunit.client.1.smithi067.stdout:4/848: rename d8/da/d1d/d15/d75/daa/l80 to d8/da/d9f/db5/l11a 0 2024-04-12T05:37:36.027 INFO:tasks.workunit.client.1.smithi067.stdout:7/956: dread d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/dd6/ddf/fef [0,4194304] 0 2024-04-12T05:37:36.028 INFO:tasks.workunit.client.1.smithi067.stdout:6/917: truncate dd/d21/d64/d88/f32 2403352 0 2024-04-12T05:37:36.028 INFO:tasks.workunit.client.1.smithi067.stdout:1/928: getdents d4/d9/d3a/d51/d55 0 2024-04-12T05:37:36.030 INFO:tasks.workunit.client.1.smithi067.stdout:4/849: chown d8/fc1 2056594 1 2024-04-12T05:37:36.036 INFO:tasks.workunit.client.1.smithi067.stdout:6/918: mkdir dd/d21/d64/d12c 0 2024-04-12T05:37:36.039 INFO:tasks.workunit.client.1.smithi067.stdout:7/957: creat d0/d23/d80/da0/db4/d5/d1d/de9/de1/f149 x:0 0 0 2024-04-12T05:37:36.040 INFO:tasks.workunit.client.1.smithi067.stdout:3/825: write dc/d1d/d28/d3e/d69/d113/f55 [1394686,64031] 0 2024-04-12T05:37:36.040 INFO:tasks.workunit.client.1.smithi067.stdout:1/929: creat d4/d3d/d7b/dde/f122 x:0 0 0 2024-04-12T05:37:36.041 INFO:tasks.workunit.client.1.smithi067.stdout:4/850: mkdir d8/da/d1d/d27/d91/d11b 0 2024-04-12T05:37:36.042 INFO:tasks.workunit.client.1.smithi067.stdout:6/919: symlink dd/d21/d3f/l12d 0 2024-04-12T05:37:36.093 INFO:tasks.workunit.client.1.smithi067.stdout:3/826: creat dc/d50/f125 x:0 0 0 2024-04-12T05:37:36.095 INFO:tasks.workunit.client.1.smithi067.stdout:1/930: creat d4/d9/d14/d7a/f123 x:0 0 0 2024-04-12T05:37:36.095 INFO:tasks.workunit.client.1.smithi067.stdout:4/851: readlink d8/da/d1d/d15/d58/l116 0 2024-04-12T05:37:36.096 INFO:tasks.workunit.client.1.smithi067.stdout:2/878: dwrite d5/d6a/d8f/fc8 [0,4194304] 0 2024-04-12T05:37:36.098 INFO:tasks.workunit.client.1.smithi067.stdout:6/920: write dd/d11/f1e [802269,9728] 0 2024-04-12T05:37:36.098 INFO:tasks.workunit.client.1.smithi067.stdout:4/852: fsync d8/d9/d3d/f3f 0 2024-04-12T05:37:36.100 INFO:tasks.workunit.client.1.smithi067.stdout:3/827: write dc/d1d/d28/f39 [1232572,72713] 0 2024-04-12T05:37:36.103 INFO:tasks.workunit.client.1.smithi067.stdout:3/828: dread - dc/df/d27/dc5/fd4 zero size 2024-04-12T05:37:36.105 INFO:tasks.workunit.client.1.smithi067.stdout:2/879: truncate d5/dbf/feb 688509 0 2024-04-12T05:37:36.106 INFO:tasks.workunit.client.1.smithi067.stdout:4/853: symlink d8/dba/d5d/d66/d96/dd4/l11c 0 2024-04-12T05:37:36.107 INFO:tasks.workunit.client.1.smithi067.stdout:3/829: mknod dc/df/df4/c126 0 2024-04-12T05:37:36.108 INFO:tasks.workunit.client.1.smithi067.stdout:7/958: getdents d0/d23/d80/da0/db4/dd/d42/d82/d96/de4 0 2024-04-12T05:37:36.109 INFO:tasks.workunit.client.1.smithi067.stdout:3/830: stat dc/d1d/d28/d3e/d8a/f121 0 2024-04-12T05:37:36.110 INFO:tasks.workunit.client.1.smithi067.stdout:1/931: dread d4/d3d/d7b/f95 [0,4194304] 0 2024-04-12T05:37:36.111 INFO:tasks.workunit.client.1.smithi067.stdout:4/854: creat d8/dba/d88/f11d x:0 0 0 2024-04-12T05:37:36.113 INFO:tasks.workunit.client.1.smithi067.stdout:6/921: dread dd/d11/fac [0,4194304] 0 2024-04-12T05:37:36.113 INFO:tasks.workunit.client.1.smithi067.stdout:7/959: chown d0/d23/cdb 1771000 1 2024-04-12T05:37:36.113 INFO:tasks.workunit.client.1.smithi067.stdout:1/932: chown d4/d9/d14/d7a/f123 1133003215 1 2024-04-12T05:37:36.115 INFO:tasks.workunit.client.1.smithi067.stdout:2/880: write d5/d11/f8a [2598614,98868] 0 2024-04-12T05:37:36.119 INFO:tasks.workunit.client.1.smithi067.stdout:1/933: creat d4/d9/d12/d6f/df8/f124 x:0 0 0 2024-04-12T05:37:36.121 INFO:tasks.workunit.client.1.smithi067.stdout:3/831: rename dc/cd to dc/d1d/d28/d3e/d53/d76/da7/df1/d118/c127 0 2024-04-12T05:37:36.122 INFO:tasks.workunit.client.1.smithi067.stdout:2/881: read d5/da/de/f94 [333376,17662] 0 2024-04-12T05:37:36.124 INFO:tasks.workunit.client.1.smithi067.stdout:7/960: fdatasync d0/d23/d80/da0/db4/dd/d79/f6a 0 2024-04-12T05:37:36.124 INFO:tasks.workunit.client.1.smithi067.stdout:3/832: dread - dc/d1d/d28/d3e/d53/d76/da7/df1/dc8/da3/dde/f123 zero size 2024-04-12T05:37:36.126 INFO:tasks.workunit.client.1.smithi067.stdout:6/922: getdents dd/d21/d64/d88/d7b/dd2/ddb 0 2024-04-12T05:37:36.132 INFO:tasks.workunit.client.1.smithi067.stdout:1/934: write d4/d9/d12/d2c/fdb [81323,75389] 0 2024-04-12T05:37:36.133 INFO:tasks.workunit.client.1.smithi067.stdout:3/833: mknod dc/d1d/d28/d3e/d53/d76/da7/df1/dc8/da3/dde/c128 0 2024-04-12T05:37:36.137 INFO:tasks.workunit.client.1.smithi067.stdout:4/855: truncate d8/dba/d5d/d66/d7b/fdb 28808 0 2024-04-12T05:37:36.142 INFO:tasks.workunit.client.1.smithi067.stdout:7/961: creat d0/d23/d80/da0/db4/d5/d1d/f14a x:0 0 0 2024-04-12T05:37:36.145 INFO:tasks.workunit.client.1.smithi067.stdout:3/834: stat dc/d1d/f24 0 2024-04-12T05:37:36.145 INFO:tasks.workunit.client.1.smithi067.stdout:6/923: dwrite dd/d21/d3f/dd5/fe0 [0,4194304] 0 2024-04-12T05:37:36.145 INFO:tasks.workunit.client.1.smithi067.stdout:1/935: write d4/d9/d12/f7d [1433345,125967] 0 2024-04-12T05:37:36.146 INFO:tasks.workunit.client.1.smithi067.stdout:4/856: creat d8/da/d9f/df7/f11e x:0 0 0 2024-04-12T05:37:36.150 INFO:tasks.workunit.client.1.smithi067.stdout:2/882: dwrite d5/da/de/d14/d34/f37 [0,4194304] 0 2024-04-12T05:37:36.150 INFO:tasks.workunit.client.1.smithi067.stdout:7/962: mkdir d0/d23/d80/da0/db4/dd/d42/d7e/d14b 0 2024-04-12T05:37:36.154 INFO:tasks.workunit.client.1.smithi067.stdout:1/936: write d4/d3d/dab/fb1 [3439639,130767] 0 2024-04-12T05:37:36.156 INFO:tasks.workunit.client.1.smithi067.stdout:2/883: chown d5/da/de/d14/d34/d40/d57/d104 59785 1 2024-04-12T05:37:36.161 INFO:tasks.workunit.client.1.smithi067.stdout:1/937: chown d4/d9/d12/d59/l86 97 1 2024-04-12T05:37:36.162 INFO:tasks.workunit.client.1.smithi067.stdout:2/884: write d5/da/de/d14/d34/fac [109541,44122] 0 2024-04-12T05:37:36.163 INFO:tasks.workunit.client.1.smithi067.stdout:7/963: creat d0/d23/d80/da0/db4/dd/da2/d51/f14c x:0 0 0 2024-04-12T05:37:36.170 INFO:tasks.workunit.client.1.smithi067.stdout:4/857: dwrite d8/da/d1d/d27/d91/ff4 [4194304,4194304] 0 2024-04-12T05:37:36.171 INFO:tasks.workunit.client.1.smithi067.stdout:6/924: rename dd/d11/d27/d4a/d79/dc3/c39 to dd/d21/d64/d88/d7b/dcc/d93/d108/c12e 0 2024-04-12T05:37:36.171 INFO:tasks.workunit.client.1.smithi067.stdout:3/835: dwrite dc/d1d/d28/d3e/d53/d76/da7/df1/dc8/da3/db9/f119 [0,4194304] 0 2024-04-12T05:37:36.174 INFO:tasks.workunit.client.1.smithi067.stdout:6/925: fsync dd/d11/d27/d4a/f82 0 2024-04-12T05:37:36.177 INFO:tasks.workunit.client.1.smithi067.stdout:3/836: write dc/d1d/d28/d3e/f4a [542472,88465] 0 2024-04-12T05:37:36.180 INFO:tasks.workunit.client.1.smithi067.stdout:4/858: symlink d8/dba/d88/ded/l11f 0 2024-04-12T05:37:36.180 INFO:tasks.workunit.client.1.smithi067.stdout:1/938: dread d4/d9/d3a/dc4/ff0 [0,4194304] 0 2024-04-12T05:37:36.181 INFO:tasks.workunit.client.1.smithi067.stdout:6/926: stat dd/d11/d27/d4a/d79/f68 0 2024-04-12T05:37:36.184 INFO:tasks.workunit.client.1.smithi067.stdout:4/859: write d8/da/d8b/d8c/fe3 [291770,14528] 0 2024-04-12T05:37:36.184 INFO:tasks.workunit.client.1.smithi067.stdout:1/939: fdatasync d4/d3d/dab/fec 0 2024-04-12T05:37:36.187 INFO:tasks.workunit.client.1.smithi067.stdout:3/837: unlink dc/d1d/d26/d52/ccb 0 2024-04-12T05:37:36.190 INFO:tasks.workunit.client.1.smithi067.stdout:4/860: mknod d8/da/d8b/d8c/c120 0 2024-04-12T05:37:36.191 INFO:tasks.workunit.client.1.smithi067.stdout:6/927: unlink dd/d21/d64/d6f/d90/le4 0 2024-04-12T05:37:36.192 INFO:tasks.workunit.client.1.smithi067.stdout:3/838: mknod dc/de4/c129 0 2024-04-12T05:37:36.194 INFO:tasks.workunit.client.1.smithi067.stdout:7/964: write d0/d23/d80/da0/db4/fee [53959,129160] 0 2024-04-12T05:37:36.196 INFO:tasks.workunit.client.1.smithi067.stdout:6/928: fdatasync dd/d21/d64/d88/f96 0 2024-04-12T05:37:36.197 INFO:tasks.workunit.client.1.smithi067.stdout:3/839: write dc/df/f2c [111484,12688] 0 2024-04-12T05:37:36.197 INFO:tasks.workunit.client.1.smithi067.stdout:4/861: dread d8/dba/d5d/d66/d7b/d6d/f81 [0,4194304] 0 2024-04-12T05:37:36.202 INFO:tasks.workunit.client.1.smithi067.stdout:4/862: write d8/da/d1d/d15/d58/fb1 [4921212,115595] 0 2024-04-12T05:37:36.202 INFO:tasks.workunit.client.1.smithi067.stdout:7/965: rmdir d0/d23/d80/da0/dc3 39 2024-04-12T05:37:36.202 INFO:tasks.workunit.client.1.smithi067.stdout:6/929: creat dd/d21/df9/d124/f12f x:0 0 0 2024-04-12T05:37:36.204 INFO:tasks.workunit.client.1.smithi067.stdout:7/966: readlink d0/l1b 0 2024-04-12T05:37:36.205 INFO:tasks.workunit.client.1.smithi067.stdout:6/930: write dd/f1d [1227599,45334] 0 2024-04-12T05:37:36.206 INFO:tasks.workunit.client.1.smithi067.stdout:2/885: dread d5/d11/f2e [0,4194304] 0 2024-04-12T05:37:36.207 INFO:tasks.workunit.client.1.smithi067.stdout:4/863: write d8/dba/d88/f11d [45359,64918] 0 2024-04-12T05:37:36.212 INFO:tasks.workunit.client.1.smithi067.stdout:6/931: chown dd/d11/d27/d4a/d79/dc3/l28 1654329 1 2024-04-12T05:37:36.213 INFO:tasks.workunit.client.1.smithi067.stdout:7/967: symlink d0/d23/d80/da0/db4/dd/d42/d71/dbd/l14d 0 2024-04-12T05:37:36.224 INFO:tasks.workunit.client.1.smithi067.stdout:6/932: write dd/d11/d27/d4a/d8b/f118 [187811,39316] 0 2024-04-12T05:37:36.224 INFO:tasks.workunit.client.1.smithi067.stdout:2/886: dwrite d5/da/dae/dba/f105 [0,4194304] 0 2024-04-12T05:37:36.226 INFO:tasks.workunit.client.1.smithi067.stdout:4/864: dwrite d8/da/d8b/d8c/fe3 [0,4194304] 0 2024-04-12T05:37:36.227 INFO:tasks.workunit.client.1.smithi067.stdout:6/933: dread - dd/d21/d64/d6f/d90/d98/fd6 zero size 2024-04-12T05:37:36.229 INFO:tasks.workunit.client.1.smithi067.stdout:7/968: rename d0/d23/d80/da0/db4/d5/d1d/ff5 to d0/d23/d80/da0/db4/d5/d1d/d43/d112/f14e 0 2024-04-12T05:37:36.230 INFO:tasks.workunit.client.1.smithi067.stdout:4/865: creat d8/dba/d88/da0/f121 x:0 0 0 2024-04-12T05:37:36.233 INFO:tasks.workunit.client.1.smithi067.stdout:2/887: rename d5/d43/f44 to d5/da/d3d/df7/f118 0 2024-04-12T05:37:36.233 INFO:tasks.workunit.client.1.smithi067.stdout:2/888: chown d5/d43/d6e/la1 883575 1 2024-04-12T05:37:36.234 INFO:tasks.workunit.client.1.smithi067.stdout:7/969: symlink d0/d23/d80/da0/db4/d5/d1d/de9/de1/l14f 0 2024-04-12T05:37:36.242 INFO:tasks.workunit.client.1.smithi067.stdout:4/866: creat d8/da/d9f/f122 x:0 0 0 2024-04-12T05:37:36.244 INFO:tasks.workunit.client.1.smithi067.stdout:2/889: dread d5/da/de/d14/d46/f5a [0,4194304] 0 2024-04-12T05:37:36.245 INFO:tasks.workunit.client.1.smithi067.stdout:4/867: write d8/d39/f3b [5239749,38783] 0 2024-04-12T05:37:36.245 INFO:tasks.workunit.client.1.smithi067.stdout:2/890: chown d5/da/dae/dba/c10f 449508 1 2024-04-12T05:37:36.248 INFO:tasks.workunit.client.1.smithi067.stdout:4/868: unlink d8/dba/d5d/d66/d7b/dbd/dcc/l6c 0 2024-04-12T05:37:36.250 INFO:tasks.workunit.client.1.smithi067.stdout:4/869: fsync d8/da/d8b/f3a 0 2024-04-12T05:37:36.251 INFO:tasks.workunit.client.1.smithi067.stdout:2/891: link d5/da/de/d14/l4b d5/d43/d6e/l119 0 2024-04-12T05:37:36.252 INFO:tasks.workunit.client.1.smithi067.stdout:2/892: fdatasync d5/da/de/d14/d34/fac 0 2024-04-12T05:37:36.254 INFO:tasks.workunit.client.1.smithi067.stdout:4/870: creat d8/da/d1d/d27/f123 x:0 0 0 2024-04-12T05:37:36.259 INFO:tasks.workunit.client.1.smithi067.stdout:4/871: unlink d8/da/d1d/d15/d58/f59 0 2024-04-12T05:37:36.260 INFO:tasks.workunit.client.1.smithi067.stdout:4/872: dread - d8/dba/d5d/d66/d7b/dbd/f10e zero size 2024-04-12T05:37:36.263 INFO:tasks.workunit.client.1.smithi067.stdout:4/873: unlink d8/c34 0 2024-04-12T05:37:36.263 INFO:tasks.workunit.client.1.smithi067.stdout:2/893: write d5/da/de/f69 [8134006,30035] 0 2024-04-12T05:37:36.266 INFO:tasks.workunit.client.1.smithi067.stdout:4/874: rename d8/da/d1d/d15/d58/f9b to d8/dba/d5d/d66/d7b/dbd/dd1/f124 0 2024-04-12T05:37:36.269 INFO:tasks.workunit.client.1.smithi067.stdout:4/875: creat d8/da/d9f/f125 x:0 0 0 2024-04-12T05:37:36.272 INFO:tasks.workunit.client.1.smithi067.stdout:4/876: rmdir d8/dba/d5d/d66/d71/d92 0 2024-04-12T05:37:36.276 INFO:tasks.workunit.client.1.smithi067.stdout:9/958: sync 2024-04-12T05:37:36.281 INFO:tasks.workunit.client.1.smithi067.stdout:4/877: dwrite d8/da/f18 [8388608,4194304] 0 2024-04-12T05:37:36.281 INFO:tasks.workunit.client.1.smithi067.stdout:9/959: dread d1/d28/d21/d45/d75/f7d [0,4194304] 0 2024-04-12T05:37:36.288 INFO:tasks.workunit.client.1.smithi067.stdout:2/894: dwrite d5/da/de/d14/d34/fa3 [0,4194304] 0 2024-04-12T05:37:36.289 INFO:tasks.workunit.client.1.smithi067.stdout:2/895: chown d5/d43/d6e 40404 1 2024-04-12T05:37:36.290 INFO:tasks.workunit.client.1.smithi067.stdout:9/960: mknod d1/d28/d21/d45/d75/de6/d8d/dc1/dcb/c12f 0 2024-04-12T05:37:36.291 INFO:tasks.workunit.client.1.smithi067.stdout:9/961: fdatasync d1/d12/f117 0 2024-04-12T05:37:36.293 INFO:tasks.workunit.client.1.smithi067.stdout:4/878: rename d8/dba/d5d/d66/d96/ca6 to d8/da/c126 0 2024-04-12T05:37:36.294 INFO:tasks.workunit.client.1.smithi067.stdout:2/896: rmdir d5/da/de/d14/d34/d4c/dbc 39 2024-04-12T05:37:36.295 INFO:tasks.workunit.client.1.smithi067.stdout:4/879: write d8/dba/d88/da0/fee [886088,92984] 0 2024-04-12T05:37:36.296 INFO:tasks.workunit.client.1.smithi067.stdout:9/962: symlink d1/d28/d21/d45/d75/l130 0 2024-04-12T05:37:36.300 INFO:tasks.workunit.client.1.smithi067.stdout:9/963: symlink d1/d28/d21/d3f/d4f/l131 0 2024-04-12T05:37:36.303 INFO:tasks.workunit.client.1.smithi067.stdout:2/897: dwrite d5/d11/f8a [0,4194304] 0 2024-04-12T05:37:36.305 INFO:tasks.workunit.client.1.smithi067.stdout:9/964: creat d1/d66/d7a/f132 x:0 0 0 2024-04-12T05:37:36.305 INFO:tasks.workunit.client.1.smithi067.stdout:4/880: write d8/dba/d5d/d66/f9c [666738,107989] 0 2024-04-12T05:37:36.312 INFO:tasks.workunit.client.1.smithi067.stdout:4/881: unlink d8/da/d9f/c55 0 2024-04-12T05:37:36.313 INFO:tasks.workunit.client.1.smithi067.stdout:4/882: chown d8/da/d1d/d27/f86 7 1 2024-04-12T05:37:36.314 INFO:tasks.workunit.client.1.smithi067.stdout:4/883: write d8/da/d8b/d8c/fe3 [963221,120202] 0 2024-04-12T05:37:36.315 INFO:tasks.workunit.client.1.smithi067.stdout:2/898: getdents d5/d23/d93 0 2024-04-12T05:37:36.315 INFO:tasks.workunit.client.1.smithi067.stdout:9/965: link d1/d28/d21/l22 d1/d28/d21/d45/dff/l133 0 2024-04-12T05:37:36.319 INFO:tasks.workunit.client.1.smithi067.stdout:4/884: truncate d8/d9/f6b 1097303 0 2024-04-12T05:37:36.320 INFO:tasks.workunit.client.1.smithi067.stdout:9/966: fsync d1/d13/db7/fd4 0 2024-04-12T05:37:36.321 INFO:tasks.workunit.client.1.smithi067.stdout:2/899: symlink d5/d23/l11a 0 2024-04-12T05:37:36.325 INFO:tasks.workunit.client.1.smithi067.stdout:9/967: link d1/d28/d21/c38 d1/d28/c134 0 2024-04-12T05:37:36.328 INFO:tasks.workunit.client.1.smithi067.stdout:4/885: dwrite d8/dba/d5d/f8f [0,4194304] 0 2024-04-12T05:37:36.333 INFO:tasks.workunit.client.1.smithi067.stdout:4/886: rename d8/d39/f3b to d8/d39/f127 0 2024-04-12T05:37:36.339 INFO:tasks.workunit.client.1.smithi067.stdout:4/887: rename d8/da/d1d/d15/l5e to d8/dba/d5d/d66/d96/l128 0 2024-04-12T05:37:36.340 INFO:tasks.workunit.client.1.smithi067.stdout:2/900: write d5/da/de/f94 [2033743,102771] 0 2024-04-12T05:37:36.342 INFO:tasks.workunit.client.1.smithi067.stdout:4/888: read d8/dba/db8/f103 [1008601,14230] 0 2024-04-12T05:37:36.343 INFO:tasks.workunit.client.1.smithi067.stdout:2/901: chown d5/da/dae/dba/cec 57 1 2024-04-12T05:37:36.347 INFO:tasks.workunit.client.1.smithi067.stdout:4/889: rename d8/dba/db8 to d8/dba/d88/da0/de7/d129 0 2024-04-12T05:37:36.348 INFO:tasks.workunit.client.1.smithi067.stdout:9/968: dwrite d1/d28/d21/d45/d75/de6/d8d/dc1/dcb/f120 [0,4194304] 0 2024-04-12T05:37:36.349 INFO:tasks.workunit.client.1.smithi067.stdout:2/902: symlink d5/da/de/d14/d46/l11b 0 2024-04-12T05:37:36.349 INFO:tasks.workunit.client.1.smithi067.stdout:9/969: fdatasync d1/d66/fe9 0 2024-04-12T05:37:36.355 INFO:tasks.workunit.client.1.smithi067.stdout:2/903: dread d5/da/dae/dba/ff6 [0,4194304] 0 2024-04-12T05:37:36.356 INFO:tasks.workunit.client.1.smithi067.stdout:9/970: creat d1/d13/f135 x:0 0 0 2024-04-12T05:37:36.358 INFO:tasks.workunit.client.1.smithi067.stdout:9/971: write d1/d28/d21/d45/d75/f80 [780040,35394] 0 2024-04-12T05:37:36.361 INFO:tasks.workunit.client.1.smithi067.stdout:4/890: write d8/d9/f2a [2392034,72775] 0 2024-04-12T05:37:36.364 INFO:tasks.workunit.client.1.smithi067.stdout:9/972: read d1/d28/d21/d3f/d4f/f11d [206007,76520] 0 2024-04-12T05:37:36.365 INFO:tasks.workunit.client.1.smithi067.stdout:4/891: stat d8/dba/d88/lf6 0 2024-04-12T05:37:36.365 INFO:tasks.workunit.client.1.smithi067.stdout:2/904: dwrite d5/d23/fab [0,4194304] 0 2024-04-12T05:37:36.370 INFO:tasks.workunit.client.1.smithi067.stdout:9/973: read d1/d28/d21/d45/d105/f81 [623684,32280] 0 2024-04-12T05:37:36.372 INFO:tasks.workunit.client.1.smithi067.stdout:9/974: write d1/d12/f117 [568980,87263] 0 2024-04-12T05:37:36.379 INFO:tasks.workunit.client.1.smithi067.stdout:2/905: dwrite d5/da/de/d14/d34/d4c/f6b [0,4194304] 0 2024-04-12T05:37:36.383 INFO:tasks.workunit.client.1.smithi067.stdout:9/975: dwrite d1/d28/fd1 [0,4194304] 0 2024-04-12T05:37:36.384 INFO:tasks.workunit.client.1.smithi067.stdout:2/906: mkdir d5/da/de/d14/d46/d11c 0 2024-04-12T05:37:36.389 INFO:tasks.workunit.client.1.smithi067.stdout:2/907: mknod d5/da/de/d14/d34/d40/d57/d96/c11d 0 2024-04-12T05:37:36.391 INFO:tasks.workunit.client.1.smithi067.stdout:9/976: rename d1/d28/d21/d7f/da7 to d1/d28/d21/d45/d75/de6/d8d/dc1/dcb/d136 0 2024-04-12T05:37:36.392 INFO:tasks.workunit.client.1.smithi067.stdout:2/908: rmdir d5/da/de/d14/d34/d40/d57/d104 39 2024-04-12T05:37:36.393 INFO:tasks.workunit.client.1.smithi067.stdout:9/977: dread d1/d28/d21/d45/d75/f7d [0,4194304] 0 2024-04-12T05:37:36.394 INFO:tasks.workunit.client.1.smithi067.stdout:9/978: chown d1/d13/lf3 45154935 1 2024-04-12T05:37:36.396 INFO:tasks.workunit.client.1.smithi067.stdout:2/909: creat d5/da/de/d14/d34/d40/d57/d96/f11e x:0 0 0 2024-04-12T05:37:36.397 INFO:tasks.workunit.client.1.smithi067.stdout:9/979: creat d1/d28/d21/d3f/f137 x:0 0 0 2024-04-12T05:37:36.397 INFO:tasks.workunit.client.1.smithi067.stdout:2/910: write d5/da/de/d14/d34/fa3 [1393881,79566] 0 2024-04-12T05:37:36.399 INFO:tasks.workunit.client.1.smithi067.stdout:2/911: read d5/d6a/f88 [874195,55189] 0 2024-04-12T05:37:36.407 INFO:tasks.workunit.client.1.smithi067.stdout:2/912: write d5/da/de/d14/d34/d40/d57/f60 [5084076,5280] 0 2024-04-12T05:37:36.408 INFO:tasks.workunit.client.1.smithi067.stdout:9/980: truncate d1/d28/d21/d45/d75/de6/d8d/dc1/fee 166737 0 2024-04-12T05:37:36.409 INFO:tasks.workunit.client.1.smithi067.stdout:2/913: write d5/d43/d6e/f109 [947426,86525] 0 2024-04-12T05:37:36.411 INFO:tasks.workunit.client.1.smithi067.stdout:9/981: symlink d1/d28/d21/d45/d75/de6/d8d/l138 0 2024-04-12T05:37:36.412 INFO:tasks.workunit.client.1.smithi067.stdout:2/914: creat d5/d43/de8/f11f x:0 0 0 2024-04-12T05:37:36.414 INFO:tasks.workunit.client.1.smithi067.stdout:9/982: symlink d1/d28/d21/d45/d75/de6/d8d/dc1/l139 0 2024-04-12T05:37:36.416 INFO:tasks.workunit.client.1.smithi067.stdout:2/915: dread d5/da/dae/fbb [0,4194304] 0 2024-04-12T05:37:36.417 INFO:tasks.workunit.client.1.smithi067.stdout:2/916: fdatasync d5/d23/d3a/f59 0 2024-04-12T05:37:36.418 INFO:tasks.workunit.client.1.smithi067.stdout:2/917: write d5/d43/f53 [1395024,58552] 0 2024-04-12T05:37:36.424 INFO:tasks.workunit.client.1.smithi067.stdout:9/983: truncate d1/d28/d21/d45/d75/de6/f9e 1822615 0 2024-04-12T05:37:36.429 INFO:tasks.workunit.client.1.smithi067.stdout:9/984: dread d1/d66/d8e/fc6 [0,4194304] 0 2024-04-12T05:37:36.434 INFO:tasks.workunit.client.1.smithi067.stdout:9/985: write d1/f9a [366742,90101] 0 2024-04-12T05:37:36.437 INFO:tasks.workunit.client.1.smithi067.stdout:9/986: unlink d1/d66/d8e/fc6 0 2024-04-12T05:37:36.439 INFO:tasks.workunit.client.1.smithi067.stdout:9/987: symlink d1/d13/db7/l13a 0 2024-04-12T05:37:36.441 INFO:tasks.workunit.client.1.smithi067.stdout:9/988: mknod d1/d28/d21/d45/d75/de6/d8d/dd2/c13b 0 2024-04-12T05:37:36.442 INFO:tasks.workunit.client.1.smithi067.stdout:9/989: dread - d1/d28/d21/d45/f9f zero size 2024-04-12T05:37:36.445 INFO:tasks.workunit.client.1.smithi067.stdout:8/875: sync 2024-04-12T05:37:36.453 INFO:tasks.workunit.client.1.smithi067.stdout:8/876: write d6/d8/d50/dd4/d4e/f75 [1727313,8211] 0 2024-04-12T05:37:36.455 INFO:tasks.workunit.client.1.smithi067.stdout:9/990: dwrite d1/d28/d27/f71 [0,4194304] 0 2024-04-12T05:37:36.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:36 smithi067 ceph-mon[49010]: pgmap v279: 65 pgs: 65 active+clean; 3.1 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 71 MiB/s rd, 183 MiB/s wr, 443 op/s 2024-04-12T05:37:36.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:36 smithi032 ceph-mon[39731]: pgmap v279: 65 pgs: 65 active+clean; 3.1 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 71 MiB/s rd, 183 MiB/s wr, 443 op/s 2024-04-12T05:37:36.593 INFO:tasks.workunit.client.1.smithi067.stdout:1/940: sync 2024-04-12T05:37:36.593 INFO:tasks.workunit.client.1.smithi067.stdout:7/970: sync 2024-04-12T05:37:36.594 INFO:tasks.workunit.client.1.smithi067.stdout:3/840: sync 2024-04-12T05:37:36.594 INFO:tasks.workunit.client.1.smithi067.stdout:6/934: sync 2024-04-12T05:37:36.597 INFO:tasks.workunit.client.1.smithi067.stdout:3/841: chown dc/df/d27/daa/fbe 4 1 2024-04-12T05:37:36.614 INFO:tasks.workunit.client.1.smithi067.stdout:7/971: rename d0/d23/d80/da0/db4/dd/d42/d71/dbd/c10f to d0/d23/d80/da0/dc3/c150 0 2024-04-12T05:37:36.615 INFO:tasks.workunit.client.1.smithi067.stdout:1/941: truncate d4/d52/d80/f6d 409246 0 2024-04-12T05:37:36.615 INFO:tasks.workunit.client.1.smithi067.stdout:6/935: dread dd/d21/d3f/dd5/fd9 [0,4194304] 0 2024-04-12T05:37:36.616 INFO:tasks.workunit.client.1.smithi067.stdout:7/972: chown d0/d23/d80/da0/l145 21969 1 2024-04-12T05:37:36.616 INFO:tasks.workunit.client.1.smithi067.stdout:3/842: getdents dc/d1d/d28/d3e/d8a 0 2024-04-12T05:37:36.619 INFO:tasks.workunit.client.1.smithi067.stdout:7/973: symlink d0/d23/d80/da0/db4/dd/da2/df8/l151 0 2024-04-12T05:37:36.621 INFO:tasks.workunit.client.1.smithi067.stdout:1/942: getdents d4/d9/d4d/de1 0 2024-04-12T05:37:36.621 INFO:tasks.workunit.client.1.smithi067.stdout:3/843: dread dc/d1d/d26/f2f [0,4194304] 0 2024-04-12T05:37:36.627 INFO:tasks.workunit.client.1.smithi067.stdout:1/943: rename d4/d3d/dab to d4/d9/d4c/d5d/d125 0 2024-04-12T05:37:36.629 INFO:tasks.workunit.client.1.smithi067.stdout:3/844: creat dc/d1d/d28/d3e/d53/d78/f12a x:0 0 0 2024-04-12T05:37:36.631 INFO:tasks.workunit.client.1.smithi067.stdout:7/974: write d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/dd6/ddf/fe2 [1012235,315] 0 2024-04-12T05:37:36.633 INFO:tasks.workunit.client.1.smithi067.stdout:3/845: creat dc/df/df4/df6/f12b x:0 0 0 2024-04-12T05:37:36.634 INFO:tasks.workunit.client.1.smithi067.stdout:1/944: rename d4/d8f/le6 to d4/d9/d3a/d51/d55/dad/ddd/l126 0 2024-04-12T05:37:36.635 INFO:tasks.workunit.client.1.smithi067.stdout:7/975: mknod d0/d23/d45/dd1/c152 0 2024-04-12T05:37:36.639 INFO:tasks.workunit.client.1.smithi067.stdout:3/846: truncate dc/d1d/d26/d52/d9a/f10a 32525 0 2024-04-12T05:37:36.640 INFO:tasks.workunit.client.1.smithi067.stdout:1/945: rename d4/d9/d3a/d51/d55/dad/l117 to d4/d9/d3a/d51/d55/dad/l127 0 2024-04-12T05:37:36.643 INFO:tasks.workunit.client.1.smithi067.stdout:1/946: unlink d4/d9/d12/d6f/df8/f124 0 2024-04-12T05:37:36.644 INFO:tasks.workunit.client.1.smithi067.stdout:3/847: symlink dc/l12c 0 2024-04-12T05:37:36.645 INFO:tasks.workunit.client.1.smithi067.stdout:3/848: chown dc/df/d27/daa 0 1 2024-04-12T05:37:36.646 INFO:tasks.workunit.client.1.smithi067.stdout:3/849: chown dc/d1d/d28/f62 738699536 1 2024-04-12T05:37:36.647 INFO:tasks.workunit.client.1.smithi067.stdout:1/947: rename d4/d9/d3a/c97 to d4/d3d/d7b/c128 0 2024-04-12T05:37:36.648 INFO:tasks.workunit.client.1.smithi067.stdout:3/850: mkdir dc/d1d/d28/d3e/d53/d76/da7/df1/dc8/da3/db9/d12d 0 2024-04-12T05:37:36.651 INFO:tasks.workunit.client.1.smithi067.stdout:1/948: rename d4/d9/d12/d59/cd6 to d4/d9/d3a/d51/d55/d9a/c129 0 2024-04-12T05:37:36.652 INFO:tasks.workunit.client.1.smithi067.stdout:3/851: symlink dc/d1d/d28/d3e/d53/d76/da7/df1/d118/l12e 0 2024-04-12T05:37:36.653 INFO:tasks.workunit.client.1.smithi067.stdout:3/852: dread - dc/d1d/f107 zero size 2024-04-12T05:37:36.654 INFO:tasks.workunit.client.1.smithi067.stdout:1/949: unlink d4/d46/lc0 0 2024-04-12T05:37:36.655 INFO:tasks.workunit.client.1.smithi067.stdout:7/976: dread d0/d23/d80/da0/db4/d5/d1d/d43/d112/f14e [0,4194304] 0 2024-04-12T05:37:36.655 INFO:tasks.workunit.client.1.smithi067.stdout:3/853: fdatasync dc/d1d/d28/d3e/d53/d78/ff2 0 2024-04-12T05:37:36.656 INFO:tasks.workunit.client.1.smithi067.stdout:1/950: mkdir d4/d3d/d7b/d118/d12a 0 2024-04-12T05:37:36.660 INFO:tasks.workunit.client.1.smithi067.stdout:1/951: unlink d4/d9/d3a/d51/d55/c107 0 2024-04-12T05:37:36.661 INFO:tasks.workunit.client.1.smithi067.stdout:7/977: rename d0/d23/d80/da0/db4/f21 to d0/d23/f153 0 2024-04-12T05:37:36.663 INFO:tasks.workunit.client.1.smithi067.stdout:1/952: chown d4/d9/d12/d59/fda 233060 1 2024-04-12T05:37:36.664 INFO:tasks.workunit.client.1.smithi067.stdout:7/978: symlink d0/d23/d80/da0/db4/d5/da1/l154 0 2024-04-12T05:37:36.669 INFO:tasks.workunit.client.1.smithi067.stdout:7/979: creat d0/d23/d80/da0/db4/d5/d13a/f155 x:0 0 0 2024-04-12T05:37:36.674 INFO:tasks.workunit.client.1.smithi067.stdout:1/953: write d4/f7 [380846,84301] 0 2024-04-12T05:37:36.679 INFO:tasks.workunit.client.1.smithi067.stdout:3/854: dwrite dc/d1d/d28/d3e/d53/f58 [0,4194304] 0 2024-04-12T05:37:36.680 INFO:tasks.workunit.client.1.smithi067.stdout:1/954: unlink d4/d3d/lf1 0 2024-04-12T05:37:36.680 INFO:tasks.workunit.client.1.smithi067.stdout:1/955: chown d4/d9/fd 6 1 2024-04-12T05:37:36.682 INFO:tasks.workunit.client.1.smithi067.stdout:3/855: rmdir dc/d50 39 2024-04-12T05:37:36.683 INFO:tasks.workunit.client.1.smithi067.stdout:1/956: symlink d4/d8f/d110/l12b 0 2024-04-12T05:37:36.687 INFO:tasks.workunit.client.1.smithi067.stdout:1/957: rename d4/d3d/d7b/d118 to d4/d8f/d110/d12c 0 2024-04-12T05:37:36.691 INFO:tasks.workunit.client.1.smithi067.stdout:3/856: dwrite dc/d1d/d28/f39 [0,4194304] 0 2024-04-12T05:37:36.697 INFO:tasks.workunit.client.1.smithi067.stdout:1/958: chown d4/d9/d3a/d51/d55/d60/d90/ld1 6319 1 2024-04-12T05:37:36.705 INFO:tasks.workunit.client.1.smithi067.stdout:1/959: dwrite d4/d9/d12/d2c/f101 [0,4194304] 0 2024-04-12T05:37:36.707 INFO:tasks.workunit.client.1.smithi067.stdout:1/960: write d4/d3d/d7b/ff5 [518639,73073] 0 2024-04-12T05:37:36.710 INFO:tasks.workunit.client.1.smithi067.stdout:1/961: dread d4/d9/d14/f2a [0,4194304] 0 2024-04-12T05:37:36.715 INFO:tasks.workunit.client.1.smithi067.stdout:3/857: dread f9 [0,4194304] 0 2024-04-12T05:37:36.717 INFO:tasks.workunit.client.1.smithi067.stdout:3/858: fdatasync dc/d1d/d28/d3e/d8a/f100 0 2024-04-12T05:37:36.718 INFO:tasks.workunit.client.1.smithi067.stdout:3/859: chown dc/d1d/d28/d3e/d53/d76/da7/df1/f77 15112 1 2024-04-12T05:37:36.720 INFO:tasks.workunit.client.1.smithi067.stdout:3/860: fdatasync dc/df/d8d/fab 0 2024-04-12T05:37:36.722 INFO:tasks.workunit.client.1.smithi067.stdout:3/861: write dc/df/fff [1131676,76376] 0 2024-04-12T05:37:36.725 INFO:tasks.workunit.client.1.smithi067.stdout:1/962: dwrite d4/d9/fd [0,4194304] 0 2024-04-12T05:37:36.726 INFO:tasks.workunit.client.1.smithi067.stdout:1/963: readlink d4/d9/d14/lcc 0 2024-04-12T05:37:36.730 INFO:tasks.workunit.client.1.smithi067.stdout:1/964: dread d4/d3d/d7b/f95 [0,4194304] 0 2024-04-12T05:37:36.733 INFO:tasks.workunit.client.1.smithi067.stdout:3/862: dwrite dc/d50/f125 [0,4194304] 0 2024-04-12T05:37:36.734 INFO:tasks.workunit.client.1.smithi067.stdout:3/863: fdatasync dc/d1d/d28/d3e/d8a/f11d 0 2024-04-12T05:37:36.737 INFO:tasks.workunit.client.1.smithi067.stdout:3/864: mknod dc/d1d/d28/d3e/d8a/dfd/c12f 0 2024-04-12T05:37:36.740 INFO:tasks.workunit.client.1.smithi067.stdout:1/965: dread d4/d9/f1c [0,4194304] 0 2024-04-12T05:37:36.743 INFO:tasks.workunit.client.1.smithi067.stdout:3/865: creat dc/d1d/d26/f130 x:0 0 0 2024-04-12T05:37:36.743 INFO:tasks.workunit.client.1.smithi067.stdout:1/966: mkdir d4/d12d 0 2024-04-12T05:37:36.744 INFO:tasks.workunit.client.1.smithi067.stdout:3/866: stat dc/d1d/d28/d3e/f66 0 2024-04-12T05:37:36.745 INFO:tasks.workunit.client.1.smithi067.stdout:3/867: chown dc/d1d/d28/f4c 415 1 2024-04-12T05:37:36.747 INFO:tasks.workunit.client.1.smithi067.stdout:1/967: rename d4/d9/d12/d59 to d4/d3d/d7b/dde/d12e 0 2024-04-12T05:37:36.748 INFO:tasks.workunit.client.1.smithi067.stdout:1/968: stat d4/d9/d12/d2c/fdb 0 2024-04-12T05:37:36.750 INFO:tasks.workunit.client.1.smithi067.stdout:3/868: mkdir dc/d1d/d28/d131 0 2024-04-12T05:37:36.753 INFO:tasks.workunit.client.1.smithi067.stdout:1/969: rename d4/d46 to d4/d3d/d7b/d12f 0 2024-04-12T05:37:36.754 INFO:tasks.workunit.client.1.smithi067.stdout:3/869: chown dc/l33 26290994 1 2024-04-12T05:37:36.756 INFO:tasks.workunit.client.1.smithi067.stdout:1/970: rename d4/dd2 to d4/d3d/dbf/d130 0 2024-04-12T05:37:36.758 INFO:tasks.workunit.client.1.smithi067.stdout:1/971: chown d4/d9/d3a/d51/d55/d9a 1129 1 2024-04-12T05:37:36.760 INFO:tasks.workunit.client.1.smithi067.stdout:1/972: fsync d4/d9/d3a/dc4/ff0 0 2024-04-12T05:37:36.762 INFO:tasks.workunit.client.1.smithi067.stdout:3/870: dread dc/d1d/f24 [0,4194304] 0 2024-04-12T05:37:36.762 INFO:tasks.workunit.client.1.smithi067.stdout:3/871: fdatasync dc/faf 0 2024-04-12T05:37:36.762 INFO:tasks.workunit.client.1.smithi067.stdout:1/973: mknod d4/d9/d14/d7a/c131 0 2024-04-12T05:37:36.765 INFO:tasks.workunit.client.1.smithi067.stdout:3/872: mknod dc/d1d/d26/c132 0 2024-04-12T05:37:36.766 INFO:tasks.workunit.client.1.smithi067.stdout:1/974: creat d4/d9/d3a/d51/d55/d60/f132 x:0 0 0 2024-04-12T05:37:36.769 INFO:tasks.workunit.client.1.smithi067.stdout:1/975: creat d4/d3d/d7b/d12f/f133 x:0 0 0 2024-04-12T05:37:36.770 INFO:tasks.workunit.client.1.smithi067.stdout:2/918: sync 2024-04-12T05:37:36.770 INFO:tasks.workunit.client.1.smithi067.stdout:9/991: sync 2024-04-12T05:37:36.770 INFO:tasks.workunit.client.1.smithi067.stdout:4/892: sync 2024-04-12T05:37:36.771 INFO:tasks.workunit.client.1.smithi067.stdout:8/877: sync 2024-04-12T05:37:36.772 INFO:tasks.workunit.client.1.smithi067.stdout:3/873: truncate dc/df/d27/fa8 320979 0 2024-04-12T05:37:36.776 INFO:tasks.workunit.client.1.smithi067.stdout:2/919: read d5/da/de/d14/d34/d4c/f9d [172318,2273] 0 2024-04-12T05:37:36.777 INFO:tasks.workunit.client.1.smithi067.stdout:4/893: creat d8/da/d1d/d15/f12a x:0 0 0 2024-04-12T05:37:36.778 INFO:tasks.workunit.client.1.smithi067.stdout:8/878: truncate d6/d8/d50/fe6 94728 0 2024-04-12T05:37:36.778 INFO:tasks.workunit.client.1.smithi067.stdout:1/976: dread d4/d9/d12/f2e [0,4194304] 0 2024-04-12T05:37:36.779 INFO:tasks.workunit.client.1.smithi067.stdout:3/874: truncate dc/d50/da6/dbd/fe3 4984843 0 2024-04-12T05:37:36.784 INFO:tasks.workunit.client.1.smithi067.stdout:3/875: write dc/d1d/d28/d3e/d53/d76/f11c [391056,84045] 0 2024-04-12T05:37:36.784 INFO:tasks.workunit.client.1.smithi067.stdout:9/992: dwrite d1/d28/d21/d45/dff/f110 [0,4194304] 0 2024-04-12T05:37:36.785 INFO:tasks.workunit.client.1.smithi067.stdout:3/876: write dc/df/df4/df6/f12b [326522,37393] 0 2024-04-12T05:37:36.786 INFO:tasks.workunit.client.1.smithi067.stdout:4/894: rmdir d8 39 2024-04-12T05:37:36.789 INFO:tasks.workunit.client.1.smithi067.stdout:8/879: mkdir d6/def/d9a/d12/d108 0 2024-04-12T05:37:36.791 INFO:tasks.workunit.client.1.smithi067.stdout:2/920: truncate d5/da/dae/dba/ff6 1311110 0 2024-04-12T05:37:36.791 INFO:tasks.workunit.client.1.smithi067.stdout:2/921: readlink d5/da/dae/le1 0 2024-04-12T05:37:36.792 INFO:tasks.workunit.client.1.smithi067.stdout:1/977: write d4/d8f/fa0 [4378239,99892] 0 2024-04-12T05:37:36.795 INFO:tasks.workunit.client.1.smithi067.stdout:8/880: creat d6/d8/d8b/dc3/f109 x:0 0 0 2024-04-12T05:37:36.797 INFO:tasks.workunit.client.1.smithi067.stdout:3/877: truncate dc/df/d27/f3f 351186 0 2024-04-12T05:37:36.797 INFO:tasks.workunit.client.1.smithi067.stdout:1/978: unlink d4/d9/d4d/de1/cef 0 2024-04-12T05:37:36.802 INFO:tasks.workunit.client.1.smithi067.stdout:2/922: dread d5/d11/f2a [0,4194304] 0 2024-04-12T05:37:36.802 INFO:tasks.workunit.client.1.smithi067.stdout:3/878: fdatasync dc/df/df4/df6/f12b 0 2024-04-12T05:37:36.802 INFO:tasks.workunit.client.1.smithi067.stdout:9/993: mkdir d1/d28/d27/d13c 0 2024-04-12T05:37:36.802 INFO:tasks.workunit.client.1.smithi067.stdout:1/979: symlink d4/d9/d4c/d5d/d125/d113/l134 0 2024-04-12T05:37:36.804 INFO:tasks.workunit.client.1.smithi067.stdout:8/881: link d6/d8/d50/dd4/de7/d69/l8f d6/d8/d8b/dc3/l10a 0 2024-04-12T05:37:36.807 INFO:tasks.workunit.client.1.smithi067.stdout:9/994: chown d1/d28/d21/d45/d75/de6/d8d/dd2/f106 3158 1 2024-04-12T05:37:36.810 INFO:tasks.workunit.client.1.smithi067.stdout:8/882: chown d6/d8/d8b/da1/cc6 223 1 2024-04-12T05:37:36.813 INFO:tasks.workunit.client.1.smithi067.stdout:9/995: chown d1/d28/d21/l54 35 1 2024-04-12T05:37:36.819 INFO:tasks.workunit.client.1.smithi067.stdout:4/895: truncate d8/da/d1d/f85 2652701 0 2024-04-12T05:37:36.820 INFO:tasks.workunit.client.1.smithi067.stdout:2/923: dwrite d5/d6a/d78/fd0 [0,4194304] 0 2024-04-12T05:37:36.822 INFO:tasks.workunit.client.1.smithi067.stdout:3/879: dwrite dc/d1d/d28/d3e/d53/d78/f12a [0,4194304] 0 2024-04-12T05:37:36.824 INFO:tasks.workunit.client.1.smithi067.stdout:1/980: dwrite d4/d9/d14/d7a/f123 [0,4194304] 0 2024-04-12T05:37:36.828 INFO:tasks.workunit.client.1.smithi067.stdout:2/924: fsync d5/d11/f50 0 2024-04-12T05:37:36.829 INFO:tasks.workunit.client.1.smithi067.stdout:2/925: stat d5/da/d3d/f7d 0 2024-04-12T05:37:36.832 INFO:tasks.workunit.client.1.smithi067.stdout:8/883: link d6/d8/d50/cb5 d6/d8/d50/c10b 0 2024-04-12T05:37:36.835 INFO:tasks.workunit.client.1.smithi067.stdout:3/880: creat dc/d1d/d28/d3e/d53/d78/f133 x:0 0 0 2024-04-12T05:37:36.841 INFO:tasks.workunit.client.1.smithi067.stdout:4/896: rename d8/d39/f79 to d8/da/f12b 0 2024-04-12T05:37:36.841 INFO:tasks.workunit.client.1.smithi067.stdout:2/926: mknod d5/da/de/d14/d34/d40/d57/d104/c120 0 2024-04-12T05:37:36.843 INFO:tasks.workunit.client.1.smithi067.stdout:3/881: creat dc/d1d/d26/d52/d9a/d9c/f134 x:0 0 0 2024-04-12T05:37:36.848 INFO:tasks.workunit.client.1.smithi067.stdout:9/996: rename d1/d28/d21/d45/d75/de6/d8d/dc1/l139 to d1/d66/dca/d10f/l13d 0 2024-04-12T05:37:36.851 INFO:tasks.workunit.client.1.smithi067.stdout:1/981: getdents d4/d8f 0 2024-04-12T05:37:36.853 INFO:tasks.workunit.client.1.smithi067.stdout:1/982: write d4/d9/d4c/d5d/d125/f32 [2170802,87512] 0 2024-04-12T05:37:36.855 INFO:tasks.workunit.client.1.smithi067.stdout:9/997: dread d1/d28/d21/d45/d75/de6/d8d/d10a/f93 [0,4194304] 0 2024-04-12T05:37:36.864 INFO:tasks.workunit.client.1.smithi067.stdout:1/983: dread d4/d9/d12/f2e [0,4194304] 0 2024-04-12T05:37:36.865 INFO:tasks.workunit.client.1.smithi067.stdout:8/884: dwrite d6/f2c [4194304,4194304] 0 2024-04-12T05:37:36.865 INFO:tasks.workunit.client.1.smithi067.stdout:4/897: dwrite d8/f2b [4194304,4194304] 0 2024-04-12T05:37:36.871 INFO:tasks.workunit.client.1.smithi067.stdout:8/885: write d6/d8/d50/dd4/d4e/fee [8757086,14226] 0 2024-04-12T05:37:36.875 INFO:tasks.workunit.client.1.smithi067.stdout:2/927: dread d5/da/de/d14/d34/d4c/f75 [0,4194304] 0 2024-04-12T05:37:36.875 INFO:tasks.workunit.client.1.smithi067.stdout:4/898: rmdir d8/dba/d88 39 2024-04-12T05:37:36.879 INFO:tasks.workunit.client.1.smithi067.stdout:8/886: creat d6/df2/f10c x:0 0 0 2024-04-12T05:37:36.880 INFO:tasks.workunit.client.1.smithi067.stdout:1/984: dwrite d4/d3d/d7b/d12f/f96 [0,4194304] 0 2024-04-12T05:37:36.880 INFO:tasks.workunit.client.1.smithi067.stdout:2/928: mknod d5/da/d3d/df7/d112/c121 0 2024-04-12T05:37:36.882 INFO:tasks.workunit.client.1.smithi067.stdout:1/985: stat d4/d3d/d7b/dde/f122 0 2024-04-12T05:37:36.882 INFO:tasks.workunit.client.1.smithi067.stdout:8/887: rmdir d6/d8/d8b 39 2024-04-12T05:37:36.884 INFO:tasks.workunit.client.1.smithi067.stdout:2/929: write d5/da/de/f69 [2318991,108343] 0 2024-04-12T05:37:36.885 INFO:tasks.workunit.client.1.smithi067.stdout:2/930: readlink d5/l7e 0 2024-04-12T05:37:36.887 INFO:tasks.workunit.client.1.smithi067.stdout:2/931: dread - d5/da/de/d14/d34/d4c/fb5 zero size 2024-04-12T05:37:36.887 INFO:tasks.workunit.client.1.smithi067.stdout:8/888: unlink d6/d86/f99 0 2024-04-12T05:37:36.890 INFO:tasks.workunit.client.1.smithi067.stdout:9/998: truncate d1/d28/d21/d45/d75/de6/d8d/dc1/ff6 2508531 0 2024-04-12T05:37:36.891 INFO:tasks.workunit.client.1.smithi067.stdout:8/889: stat d6/def/d9a/lcd 0 2024-04-12T05:37:36.894 INFO:tasks.workunit.client.1.smithi067.stdout:9/999: symlink d1/d13/d34/l13e 0 2024-04-12T05:37:36.895 INFO:tasks.workunit.client.1.smithi067.stdout:2/932: mknod d5/da/de/c122 0 2024-04-12T05:37:36.895 INFO:tasks.workunit.client.1.smithi067.stdout:8/890: chown d6/def/d9a/cdc 247568 1 2024-04-12T05:37:36.896 INFO:tasks.workunit.client.1.smithi067.stdout:4/899: dwrite d8/da/d1d/d27/f56 [0,4194304] 0 2024-04-12T05:37:36.897 INFO:tasks.workunit.client.1.smithi067.stdout:1/986: dwrite d4/d9/d4c/d5d/d125/fb1 [0,4194304] 0 2024-04-12T05:37:36.899 INFO:tasks.workunit.client.1.smithi067.stdout:1/987: read d4/f7 [234296,121708] 0 2024-04-12T05:37:36.900 INFO:tasks.workunit.client.1.smithi067.stdout:2/933: write d5/da/de/d14/f36 [13278705,1350] 0 2024-04-12T05:37:36.901 INFO:tasks.workunit.client.1.smithi067.stdout:8/891: rmdir d6/d8/d50/dd4/de7/d69/d5d 39 2024-04-12T05:37:36.902 INFO:tasks.workunit.client.1.smithi067.stdout:4/900: unlink d8/d9/d3d/c9d 0 2024-04-12T05:37:36.904 INFO:tasks.workunit.client.1.smithi067.stdout:8/892: write d6/d8/d50/dd4/f33 [938491,110462] 0 2024-04-12T05:37:36.905 INFO:tasks.workunit.client.1.smithi067.stdout:2/934: chown d5/da/d3d/f7d 6613 1 2024-04-12T05:37:36.907 INFO:tasks.workunit.client.1.smithi067.stdout:8/893: fdatasync d6/d8/d50/dd4/d4e/fee 0 2024-04-12T05:37:36.908 INFO:tasks.workunit.client.1.smithi067.stdout:4/901: write d8/da/d1d/d15/d75/daa/f113 [731916,18679] 0 2024-04-12T05:37:36.911 INFO:tasks.workunit.client.1.smithi067.stdout:8/894: stat d6/d8/d8b/da1/fed 0 2024-04-12T05:37:36.917 INFO:tasks.workunit.client.1.smithi067.stdout:8/895: unlink d6/def/d9a/d12/de9/ffe 0 2024-04-12T05:37:36.922 INFO:tasks.workunit.client.1.smithi067.stdout:8/896: creat d6/f10d x:0 0 0 2024-04-12T05:37:36.923 INFO:tasks.workunit.client.1.smithi067.stdout:1/988: write d4/d52/fc2 [594703,127701] 0 2024-04-12T05:37:36.924 INFO:tasks.workunit.client.1.smithi067.stdout:8/897: chown d6/d8/d50/dd4/d36/c51 386046740 1 2024-04-12T05:37:36.925 INFO:tasks.workunit.client.1.smithi067.stdout:2/935: truncate d5/da/de/d14/d34/d40/d57/da2/f10b 4019167 0 2024-04-12T05:37:36.928 INFO:tasks.workunit.client.1.smithi067.stdout:1/989: mknod d4/d9/d4c/df6/c135 0 2024-04-12T05:37:36.930 INFO:tasks.workunit.client.1.smithi067.stdout:1/990: chown d4/d52/d80/fb3 129351683 1 2024-04-12T05:37:36.930 INFO:tasks.workunit.client.1.smithi067.stdout:2/936: creat d5/da/de/dc4/def/f123 x:0 0 0 2024-04-12T05:37:36.935 INFO:tasks.workunit.client.1.smithi067.stdout:4/902: dwrite d8/da/d9f/df7/fac [0,4194304] 0 2024-04-12T05:37:36.938 INFO:tasks.workunit.client.1.smithi067.stdout:2/937: rmdir d5/da/dae/dba 39 2024-04-12T05:37:36.940 INFO:tasks.workunit.client.1.smithi067.stdout:8/898: truncate d6/d8/d8b/df7/f84 2066026 0 2024-04-12T05:37:36.941 INFO:tasks.workunit.client.1.smithi067.stdout:1/991: creat d4/d9/d12/f136 x:0 0 0 2024-04-12T05:37:36.941 INFO:tasks.workunit.client.1.smithi067.stdout:4/903: mknod d8/dba/d88/ded/c12c 0 2024-04-12T05:37:36.943 INFO:tasks.workunit.client.1.smithi067.stdout:8/899: mknod d6/d8/d50/dd4/d4e/d9e/c10e 0 2024-04-12T05:37:36.944 INFO:tasks.workunit.client.1.smithi067.stdout:8/900: dread - d6/def/d9a/dd7/fe2 zero size 2024-04-12T05:37:36.945 INFO:tasks.workunit.client.1.smithi067.stdout:1/992: unlink d4/d9/d14/lcc 0 2024-04-12T05:37:36.946 INFO:tasks.workunit.client.1.smithi067.stdout:4/904: creat d8/dba/d88/da0/f12d x:0 0 0 2024-04-12T05:37:36.948 INFO:tasks.workunit.client.1.smithi067.stdout:8/901: rmdir d6/d8/d50/dd4/de7/d69 39 2024-04-12T05:37:36.948 INFO:tasks.workunit.client.1.smithi067.stdout:4/905: write d8/da/d1d/d15/d58/fb1 [5529899,16890] 0 2024-04-12T05:37:36.950 INFO:tasks.workunit.client.1.smithi067.stdout:4/906: fsync d8/d9/f21 0 2024-04-12T05:37:36.955 INFO:tasks.workunit.client.1.smithi067.stdout:4/907: link d8/da/d1d/d15/d58/f78 d8/da/d1d/d27/d91/d11b/f12e 0 2024-04-12T05:37:36.956 INFO:tasks.workunit.client.1.smithi067.stdout:1/993: write d4/d9/d12/d2c/fac [610934,101729] 0 2024-04-12T05:37:36.956 INFO:tasks.workunit.client.1.smithi067.stdout:1/994: stat d4/d9/c26 0 2024-04-12T05:37:36.959 INFO:tasks.workunit.client.1.smithi067.stdout:1/995: fdatasync d4/d9/d4c/f89 0 2024-04-12T05:37:36.961 INFO:tasks.workunit.client.1.smithi067.stdout:1/996: write d4/d3d/d7b/dde/d12e/f104 [1014781,95910] 0 2024-04-12T05:37:36.963 INFO:tasks.workunit.client.1.smithi067.stdout:1/997: readlink d4/d9/d14/l85 0 2024-04-12T05:37:36.966 INFO:tasks.workunit.client.1.smithi067.stdout:1/998: getdents d4/d8f/dea 0 2024-04-12T05:37:36.971 INFO:tasks.workunit.client.1.smithi067.stdout:1/999: dwrite d4/f5 [0,4194304] 0 2024-04-12T05:37:37.078 INFO:tasks.workunit.client.1.smithi067.stdout:6/936: sync 2024-04-12T05:37:37.080 INFO:tasks.workunit.client.1.smithi067.stdout:6/937: getdents dd/d21/d64/db5/dff 0 2024-04-12T05:37:37.081 INFO:tasks.workunit.client.1.smithi067.stdout:6/938: write fb [4392114,45174] 0 2024-04-12T05:37:37.084 INFO:tasks.workunit.client.1.smithi067.stdout:6/939: creat dd/d21/d64/d88/d7b/dcc/d93/f130 x:0 0 0 2024-04-12T05:37:37.085 INFO:tasks.workunit.client.1.smithi067.stdout:6/940: write dd/d21/d64/d6f/d90/d98/fb9 [4358581,88746] 0 2024-04-12T05:37:37.088 INFO:tasks.workunit.client.1.smithi067.stdout:6/941: write dd/d11/d27/d4a/d51/fd0 [1984036,59737] 0 2024-04-12T05:37:37.089 INFO:tasks.workunit.client.1.smithi067.stdout:6/942: rmdir dd/d21 39 2024-04-12T05:37:37.091 INFO:tasks.workunit.client.1.smithi067.stdout:6/943: mkdir dd/d11/d27/d4a/df8/d129/d131 0 2024-04-12T05:37:37.095 INFO:tasks.workunit.client.1.smithi067.stdout:6/944: rename dd/d21/d3f/f10d to dd/d11/d27/d4a/df8/d129/d131/f132 0 2024-04-12T05:37:37.096 INFO:tasks.workunit.client.1.smithi067.stdout:6/945: chown dd/d21/d64/d88/d7b/dd2 101865032 1 2024-04-12T05:37:37.107 INFO:tasks.workunit.client.1.smithi067.stdout:6/946: dwrite dd/d21/d64/d88/d7b/dd2/f83 [0,4194304] 0 2024-04-12T05:37:37.111 INFO:tasks.workunit.client.1.smithi067.stdout:6/947: symlink dd/d21/d64/d88/l133 0 2024-04-12T05:37:37.117 INFO:tasks.workunit.client.1.smithi067.stdout:6/948: write dd/d21/d64/d88/f32 [3157789,11385] 0 2024-04-12T05:37:37.119 INFO:tasks.workunit.client.1.smithi067.stdout:6/949: chown dd/d11/d27/d4a/d79/f68 36623 1 2024-04-12T05:37:37.126 INFO:tasks.workunit.client.1.smithi067.stdout:6/950: dwrite dd/d21/d64/d6f/d90/f9d [0,4194304] 0 2024-04-12T05:37:37.129 INFO:tasks.workunit.client.1.smithi067.stdout:6/951: symlink dd/d21/df9/l134 0 2024-04-12T05:37:37.131 INFO:tasks.workunit.client.1.smithi067.stdout:6/952: mkdir dd/d21/df9/d135 0 2024-04-12T05:37:37.132 INFO:tasks.workunit.client.1.smithi067.stdout:6/953: dread - dd/d11/d27/fca zero size 2024-04-12T05:37:37.134 INFO:tasks.workunit.client.1.smithi067.stdout:6/954: write dd/d21/d64/dcf/f103 [592208,54204] 0 2024-04-12T05:37:37.137 INFO:tasks.workunit.client.1.smithi067.stdout:6/955: symlink dd/d21/d64/d6f/da7/l136 0 2024-04-12T05:37:37.139 INFO:tasks.workunit.client.1.smithi067.stdout:6/956: creat dd/d21/d3f/f137 x:0 0 0 2024-04-12T05:37:37.141 INFO:tasks.workunit.client.1.smithi067.stdout:6/957: mkdir dd/d21/d64/dcf/dfb/d138 0 2024-04-12T05:37:37.148 INFO:tasks.workunit.client.1.smithi067.stdout:6/958: dwrite dd/f3b [0,4194304] 0 2024-04-12T05:37:37.151 INFO:tasks.workunit.client.1.smithi067.stdout:6/959: creat dd/d11/d27/d4a/df8/d129/f139 x:0 0 0 2024-04-12T05:37:37.154 INFO:tasks.workunit.client.1.smithi067.stdout:6/960: symlink dd/d11/d27/d4a/d8b/l13a 0 2024-04-12T05:37:37.157 INFO:tasks.workunit.client.1.smithi067.stdout:6/961: getdents dd/d9f 0 2024-04-12T05:37:37.159 INFO:tasks.workunit.client.1.smithi067.stdout:6/962: fsync dd/d11/d27/f87 0 2024-04-12T05:37:37.163 INFO:tasks.workunit.client.1.smithi067.stdout:6/963: getdents dd/d21/d64/d88/d7b/dcc/d93 0 2024-04-12T05:37:37.163 INFO:tasks.workunit.client.1.smithi067.stdout:6/964: symlink dd/d21/d64/dcf/dfb/d138/l13b 2 2024-04-12T05:37:37.167 INFO:tasks.workunit.client.1.smithi067.stdout:6/965: getdents dd/d21/d64/d6f/da7 0 2024-04-12T05:37:37.170 INFO:tasks.workunit.client.1.smithi067.stdout:6/966: symlink dd/d11/d27/d4a/dc7/l13c 0 2024-04-12T05:37:37.183 INFO:tasks.workunit.client.1.smithi067.stdout:6/967: dread dd/d11/d27/d4a/d79/f68 [0,4194304] 0 2024-04-12T05:37:37.185 INFO:tasks.workunit.client.1.smithi067.stdout:6/968: dread - dd/d11/d27/d4a/d79/dc3/d56/de3/fe7 zero size 2024-04-12T05:37:37.186 INFO:tasks.workunit.client.1.smithi067.stdout:6/969: symlink dd/d21/d64/db5/l13d 0 2024-04-12T05:37:37.188 INFO:tasks.workunit.client.1.smithi067.stdout:6/970: truncate dd/d21/d64/d6f/d90/f127 74434 0 2024-04-12T05:37:37.191 INFO:tasks.workunit.client.1.smithi067.stdout:6/971: rename f9 to dd/d21/d64/f13e 0 2024-04-12T05:37:37.193 INFO:tasks.workunit.client.1.smithi067.stdout:6/972: creat dd/d21/f13f x:0 0 0 2024-04-12T05:37:37.201 INFO:tasks.workunit.client.1.smithi067.stdout:6/973: mknod dd/d21/d64/d88/d7b/dcc/d93/c140 0 2024-04-12T05:37:37.203 INFO:tasks.workunit.client.1.smithi067.stdout:6/974: truncate dd/d11/d27/d4a/df8/d129/d131/f132 990256 0 2024-04-12T05:37:37.204 INFO:tasks.workunit.client.1.smithi067.stdout:6/975: write dd/d21/d64/d6f/da7/fb8 [1985447,43914] 0 2024-04-12T05:37:37.205 INFO:tasks.workunit.client.1.smithi067.stdout:2/938: sync 2024-04-12T05:37:37.205 INFO:tasks.workunit.client.1.smithi067.stdout:7/980: sync 2024-04-12T05:37:37.205 INFO:tasks.workunit.client.1.smithi067.stdout:3/882: sync 2024-04-12T05:37:37.207 INFO:tasks.workunit.client.1.smithi067.stdout:6/976: mknod dd/d21/df9/c141 0 2024-04-12T05:37:37.209 INFO:tasks.workunit.client.1.smithi067.stdout:3/883: truncate dc/df/d27/daa/fef 3398 0 2024-04-12T05:37:37.209 INFO:tasks.workunit.client.1.smithi067.stdout:6/977: readlink dd/l13 0 2024-04-12T05:37:37.209 INFO:tasks.workunit.client.1.smithi067.stdout:2/939: symlink d5/da/dae/dba/l124 0 2024-04-12T05:37:37.209 INFO:tasks.workunit.client.1.smithi067.stdout:6/978: stat dd/d9f/l117 0 2024-04-12T05:37:37.214 INFO:tasks.workunit.client.1.smithi067.stdout:6/979: creat dd/d11/d27/d4a/d8b/f142 x:0 0 0 2024-04-12T05:37:37.216 INFO:tasks.workunit.client.1.smithi067.stdout:6/980: truncate dd/d21/d64/d88/d7b/dcc/f12a 759466 0 2024-04-12T05:37:37.218 INFO:tasks.workunit.client.1.smithi067.stdout:2/940: truncate d5/da/de/d14/d34/f37 2716817 0 2024-04-12T05:37:37.221 INFO:tasks.workunit.client.1.smithi067.stdout:2/941: write d5/d43/f53 [1103166,59829] 0 2024-04-12T05:37:37.222 INFO:tasks.workunit.client.1.smithi067.stdout:6/981: mkdir dd/d11/d27/d4a/d79/dc3/d56/d143 0 2024-04-12T05:37:37.227 INFO:tasks.workunit.client.1.smithi067.stdout:7/981: dwrite d0/d23/d80/da0/db4/dd/d42/d71/fac [0,4194304] 0 2024-04-12T05:37:37.229 INFO:tasks.workunit.client.1.smithi067.stdout:3/884: dwrite dc/d1d/d26/fc6 [0,4194304] 0 2024-04-12T05:37:37.229 INFO:tasks.workunit.client.1.smithi067.stdout:6/982: unlink dd/d21/d64/d6f/c92 0 2024-04-12T05:37:37.234 INFO:tasks.workunit.client.1.smithi067.stdout:2/942: rename d5/fe5 to d5/dbf/dcc/f125 0 2024-04-12T05:37:37.241 INFO:tasks.workunit.client.1.smithi067.stdout:2/943: creat d5/da/de/d14/f126 x:0 0 0 2024-04-12T05:37:37.244 INFO:tasks.workunit.client.1.smithi067.stdout:6/983: creat dd/d21/f144 x:0 0 0 2024-04-12T05:37:37.244 INFO:tasks.workunit.client.1.smithi067.stdout:7/982: dread d0/d67/f56 [0,4194304] 0 2024-04-12T05:37:37.245 INFO:tasks.workunit.client.1.smithi067.stdout:3/885: mknod dc/d1d/d28/d3e/d53/d76/da7/df1/dc8/da3/df3/c135 0 2024-04-12T05:37:37.248 INFO:tasks.workunit.client.1.smithi067.stdout:6/984: write dd/f1d [2546898,79733] 0 2024-04-12T05:37:37.249 INFO:tasks.workunit.client.1.smithi067.stdout:2/944: link d5/d43/d6e/cd9 d5/da/d8c/c127 0 2024-04-12T05:37:37.252 INFO:tasks.workunit.client.1.smithi067.stdout:4/908: sync 2024-04-12T05:37:37.253 INFO:tasks.workunit.client.1.smithi067.stdout:8/902: sync 2024-04-12T05:37:37.258 INFO:tasks.workunit.client.1.smithi067.stdout:2/945: truncate d5/d23/d3a/f59 1410836 0 2024-04-12T05:37:37.260 INFO:tasks.workunit.client.1.smithi067.stdout:4/909: fsync d8/d39/f127 0 2024-04-12T05:37:37.260 INFO:tasks.workunit.client.1.smithi067.stdout:7/983: write d0/d23/d80/da0/db4/dd/d42/d7e/f30 [1774888,6671] 0 2024-04-12T05:37:37.261 INFO:tasks.workunit.client.1.smithi067.stdout:8/903: mkdir d6/d8/d50/dd4/d36/d10f 0 2024-04-12T05:37:37.261 INFO:tasks.workunit.client.1.smithi067.stdout:6/985: write dd/d11/d27/d4a/d79/dc3/de5/fc2 [4955071,105454] 0 2024-04-12T05:37:37.262 INFO:tasks.workunit.client.1.smithi067.stdout:6/986: chown dd/d11/c1a 744900470 1 2024-04-12T05:37:37.263 INFO:tasks.workunit.client.1.smithi067.stdout:3/886: link dc/d1d/d28/d3e/d53/d76/da7/df1/dc8/da3/db9/l122 dc/d1d/d28/d3e/d53/d78/l136 0 2024-04-12T05:37:37.263 INFO:tasks.workunit.client.1.smithi067.stdout:6/987: write dd/d11/d27/f54 [2225366,123751] 0 2024-04-12T05:37:37.264 INFO:tasks.workunit.client.1.smithi067.stdout:3/887: chown dc/df/d27/l2e 614 1 2024-04-12T05:37:37.267 INFO:tasks.workunit.client.1.smithi067.stdout:7/984: creat d0/d23/d80/da0/db4/d5/d4a/de5/f156 x:0 0 0 2024-04-12T05:37:37.269 INFO:tasks.workunit.client.1.smithi067.stdout:6/988: fdatasync dd/d21/d64/d6f/d90/d98/dfd/f102 0 2024-04-12T05:37:37.270 INFO:tasks.workunit.client.1.smithi067.stdout:6/989: chown dd/d21/d64/d6f/f70 2063998 1 2024-04-12T05:37:37.273 INFO:tasks.workunit.client.1.smithi067.stdout:7/985: symlink d0/d23/d80/da0/db4/d109/d125/l157 0 2024-04-12T05:37:37.275 INFO:tasks.workunit.client.1.smithi067.stdout:3/888: write dc/d1d/d28/f62 [5023242,104926] 0 2024-04-12T05:37:37.277 INFO:tasks.workunit.client.1.smithi067.stdout:6/990: mkdir dd/d11/d27/d4a/d79/dc3/d9e/d145 0 2024-04-12T05:37:37.278 INFO:tasks.workunit.client.1.smithi067.stdout:2/946: getdents d5/da/de/d14/d34/d4c/dbc 0 2024-04-12T05:37:37.279 INFO:tasks.workunit.client.1.smithi067.stdout:4/910: link d8/d9/f17 d8/da/d8b/f12f 0 2024-04-12T05:37:37.282 INFO:tasks.workunit.client.1.smithi067.stdout:7/986: creat d0/d23/d80/da0/db4/d5/d1d/f158 x:0 0 0 2024-04-12T05:37:37.293 INFO:tasks.workunit.client.1.smithi067.stdout:7/987: symlink d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/l159 0 2024-04-12T05:37:37.293 INFO:tasks.workunit.client.1.smithi067.stdout:6/991: dwrite dd/d21/f13f [0,4194304] 0 2024-04-12T05:37:37.295 INFO:tasks.workunit.client.1.smithi067.stdout:2/947: dread d5/d11/f29 [0,4194304] 0 2024-04-12T05:37:37.300 INFO:tasks.workunit.client.1.smithi067.stdout:4/911: dwrite d8/dba/d5d/d66/d7b/dbd/f10e [0,4194304] 0 2024-04-12T05:37:37.302 INFO:tasks.workunit.client.1.smithi067.stdout:6/992: chown dd/l2a 9777736 1 2024-04-12T05:37:37.302 INFO:tasks.workunit.client.1.smithi067.stdout:2/948: mkdir d5/d43/d6e/d128 0 2024-04-12T05:37:37.304 INFO:tasks.workunit.client.1.smithi067.stdout:3/889: dwrite dc/d1d/d28/d3e/d69/d113/fdd [0,4194304] 0 2024-04-12T05:37:37.306 INFO:tasks.workunit.client.1.smithi067.stdout:2/949: chown d5/da/de/d14/d34/d4c/dbc 127144 1 2024-04-12T05:37:37.307 INFO:tasks.workunit.client.1.smithi067.stdout:4/912: readlink d8/dba/d5d/d66/d96/dd4/l10b 0 2024-04-12T05:37:37.307 INFO:tasks.workunit.client.1.smithi067.stdout:6/993: rmdir dd/d11/d27/d4a/d79/dc3/de5 39 2024-04-12T05:37:37.309 INFO:tasks.workunit.client.1.smithi067.stdout:4/913: truncate d8/da/d1d/ffc 872955 0 2024-04-12T05:37:37.314 INFO:tasks.workunit.client.1.smithi067.stdout:4/914: unlink d8/da/d8b/l4a 0 2024-04-12T05:37:37.315 INFO:tasks.workunit.client.1.smithi067.stdout:3/890: dwrite dc/d1d/d28/d3e/d8a/f95 [0,4194304] 0 2024-04-12T05:37:37.317 INFO:tasks.workunit.client.1.smithi067.stdout:7/988: rename d0/d23/d80/da0/db4/d109/c117 to d0/d23/d80/da0/db4/c15a 0 2024-04-12T05:37:37.321 INFO:tasks.workunit.client.1.smithi067.stdout:4/915: creat d8/da/d9f/d68/db4/f130 x:0 0 0 2024-04-12T05:37:37.323 INFO:tasks.workunit.client.1.smithi067.stdout:4/916: truncate d8/dba/d88/da0/f121 318183 0 2024-04-12T05:37:37.325 INFO:tasks.workunit.client.1.smithi067.stdout:7/989: chown d0/d23/d80/da0/db4/d5/d1d/d43/fd0 2006 1 2024-04-12T05:37:37.328 INFO:tasks.workunit.client.1.smithi067.stdout:7/990: write d0/d23/d80/da0/db4/d5/d1d/f110 [5292,100202] 0 2024-04-12T05:37:37.332 INFO:tasks.workunit.client.1.smithi067.stdout:3/891: getdents dc 0 2024-04-12T05:37:37.336 INFO:tasks.workunit.client.1.smithi067.stdout:2/950: dwrite d5/da/de/d14/d34/f61 [4194304,4194304] 0 2024-04-12T05:37:37.337 INFO:tasks.workunit.client.1.smithi067.stdout:7/991: creat d0/d23/d80/da0/db4/dd/d42/d7e/f15b x:0 0 0 2024-04-12T05:37:37.341 INFO:tasks.workunit.client.1.smithi067.stdout:2/951: write d5/d43/de8/f11f [692485,85087] 0 2024-04-12T05:37:37.345 INFO:tasks.workunit.client.1.smithi067.stdout:3/892: link dc/d1d/d28/d3e/d8a/f100 dc/d1d/d28/d3e/d53/d76/da7/df1/dc8/da3/db9/f137 0 2024-04-12T05:37:37.346 INFO:tasks.workunit.client.1.smithi067.stdout:4/917: dwrite d8/da/d9f/f52 [0,4194304] 0 2024-04-12T05:37:37.348 INFO:tasks.workunit.client.1.smithi067.stdout:2/952: dread d5/da/de/d14/d46/f5a [0,4194304] 0 2024-04-12T05:37:37.348 INFO:tasks.workunit.client.1.smithi067.stdout:7/992: rename d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/l159 to d0/d23/d80/da0/db4/dd/da2/l15c 0 2024-04-12T05:37:37.350 INFO:tasks.workunit.client.1.smithi067.stdout:3/893: dread - dc/d1d/d28/d3e/d53/d94/fda zero size 2024-04-12T05:37:37.354 INFO:tasks.workunit.client.1.smithi067.stdout:2/953: write d5/d43/f53 [1190053,85176] 0 2024-04-12T05:37:37.358 INFO:tasks.workunit.client.1.smithi067.stdout:3/894: truncate dc/d1d/d28/d3e/d53/d78/ff2 498560 0 2024-04-12T05:37:37.361 INFO:tasks.workunit.client.1.smithi067.stdout:3/895: symlink dc/d1d/d28/d3e/d53/d76/da7/df1/l138 0 2024-04-12T05:37:37.362 INFO:tasks.workunit.client.1.smithi067.stdout:3/896: write dc/d1d/f44 [186426,76493] 0 2024-04-12T05:37:37.365 INFO:tasks.workunit.client.1.smithi067.stdout:7/993: write d0/d23/f6e [8277669,68282] 0 2024-04-12T05:37:37.368 INFO:tasks.workunit.client.1.smithi067.stdout:3/897: unlink dc/d1d/d28/d3e/d53/d76/da7/df1/dc8/da3/dde/c103 0 2024-04-12T05:37:37.369 INFO:tasks.workunit.client.1.smithi067.stdout:4/918: truncate d8/dba/d5d/d66/f9c 322867 0 2024-04-12T05:37:37.370 INFO:tasks.workunit.client.1.smithi067.stdout:2/954: write d5/d6a/d78/f89 [1247015,107784] 0 2024-04-12T05:37:37.372 INFO:tasks.workunit.client.1.smithi067.stdout:4/919: chown d8/da/d9f/df7/fd2 283656150 1 2024-04-12T05:37:37.375 INFO:tasks.workunit.client.1.smithi067.stdout:2/955: mknod d5/d23/d93/dd1/c129 0 2024-04-12T05:37:37.376 INFO:tasks.workunit.client.1.smithi067.stdout:7/994: symlink d0/d23/d80/da0/db4/dd/d42/d71/dbd/dcf/dd3/dd6/ddf/d127/l15d 0 2024-04-12T05:37:37.378 INFO:tasks.workunit.client.1.smithi067.stdout:2/956: write d5/da/de/d14/d46/ffb [915877,106076] 0 2024-04-12T05:37:37.379 INFO:tasks.workunit.client.1.smithi067.stdout:3/898: dwrite dc/d1d/d28/d3e/d53/d76/f11c [0,4194304] 0 2024-04-12T05:37:37.380 INFO:tasks.workunit.client.1.smithi067.stdout:4/920: symlink d8/da/d1d/d15/d58/l131 0 2024-04-12T05:37:37.382 INFO:tasks.workunit.client.1.smithi067.stdout:7/995: fdatasync d0/d23/d80/da0/db4/dd/d79/f6d 0 2024-04-12T05:37:37.383 INFO:tasks.workunit.client.1.smithi067.stdout:4/921: read - d8/da/d1d/d15/d58/fe6 zero size 2024-04-12T05:37:37.385 INFO:tasks.workunit.client.1.smithi067.stdout:3/899: unlink dc/d1d/d26/f9e 0 2024-04-12T05:37:37.389 INFO:tasks.workunit.client.1.smithi067.stdout:7/996: truncate d0/d23/d80/da0/db4/dd/fcd 342351 0 2024-04-12T05:37:37.394 INFO:tasks.workunit.client.1.smithi067.stdout:3/900: dwrite dc/d50/f125 [0,4194304] 0 2024-04-12T05:37:37.396 INFO:tasks.workunit.client.1.smithi067.stdout:2/957: getdents d5/da/de/d14/d34/d40/d57/da2 0 2024-04-12T05:37:37.397 INFO:tasks.workunit.client.1.smithi067.stdout:2/958: stat d5/da/de/f26 0 2024-04-12T05:37:37.400 INFO:tasks.workunit.client.1.smithi067.stdout:4/922: getdents d8/da/d8b/d8c 0 2024-04-12T05:37:37.401 INFO:tasks.workunit.client.1.smithi067.stdout:7/997: mkdir d0/d23/d80/da0/db4/dd/d42/d82/d11f/d15e 0 2024-04-12T05:37:37.401 INFO:tasks.workunit.client.1.smithi067.stdout:3/901: rename dc/df/d27/dc5/c10f to dc/d1d/d28/d3e/d53/d76/da7/df1/d118/c139 0 2024-04-12T05:37:37.402 INFO:tasks.workunit.client.1.smithi067.stdout:2/959: truncate d5/da/de/d14/d34/fa3 5049677 0 2024-04-12T05:37:37.403 INFO:tasks.workunit.client.1.smithi067.stdout:7/998: read - d0/d23/d80/da0/dc3/df7/f13c zero size 2024-04-12T05:37:37.404 INFO:tasks.workunit.client.1.smithi067.stdout:4/923: creat d8/da/d9f/df7/f132 x:0 0 0 2024-04-12T05:37:37.405 INFO:tasks.workunit.client.1.smithi067.stdout:2/960: mkdir d5/da/de/dc4/def/d12a 0 2024-04-12T05:37:37.407 INFO:tasks.workunit.client.1.smithi067.stdout:2/961: readlink d5/da/de/d14/d34/d40/l106 0 2024-04-12T05:37:37.407 INFO:tasks.workunit.client.1.smithi067.stdout:3/902: truncate dc/d1d/f24 410812 0 2024-04-12T05:37:37.408 INFO:tasks.workunit.client.1.smithi067.stdout:7/999: read d0/d23/d45/f73 [1775517,27328] 0 2024-04-12T05:37:37.412 INFO:tasks.workunit.client.1.smithi067.stdout:2/962: truncate d5/d23/d93/f98 434558 0 2024-04-12T05:37:37.413 INFO:tasks.workunit.client.1.smithi067.stdout:4/924: dwrite d8/d9/d3d/f3f [0,4194304] 0 2024-04-12T05:37:37.415 INFO:tasks.workunit.client.1.smithi067.stdout:3/903: truncate dc/d1d/d28/d3e/d69/dbc/fd9 654963 0 2024-04-12T05:37:37.415 INFO:tasks.workunit.client.1.smithi067.stdout:4/925: dread - d8/da/d9f/df7/f11e zero size 2024-04-12T05:37:37.421 INFO:tasks.workunit.client.1.smithi067.stdout:3/904: getdents dc/df 0 2024-04-12T05:37:37.421 INFO:tasks.workunit.client.1.smithi067.stdout:2/963: truncate d5/da/de/d14/d34/d4c/f75 1654794 0 2024-04-12T05:37:37.423 INFO:tasks.workunit.client.1.smithi067.stdout:4/926: creat d8/d9/f133 x:0 0 0 2024-04-12T05:37:37.424 INFO:tasks.workunit.client.1.smithi067.stdout:3/905: mknod dc/d1d/d26/d52/d9a/d9c/c13a 0 2024-04-12T05:37:37.427 INFO:tasks.workunit.client.1.smithi067.stdout:3/906: mkdir dc/d1d/d26/d52/d9a/d13b 0 2024-04-12T05:37:37.429 INFO:tasks.workunit.client.1.smithi067.stdout:3/907: creat dc/df/d8d/f13c x:0 0 0 2024-04-12T05:37:37.431 INFO:tasks.workunit.client.1.smithi067.stdout:3/908: readlink dc/df/l45 0 2024-04-12T05:37:37.432 INFO:tasks.workunit.client.1.smithi067.stdout:4/927: dwrite d8/da/d1d/d15/ffd [0,4194304] 0 2024-04-12T05:37:37.433 INFO:tasks.workunit.client.1.smithi067.stdout:3/909: dread - dc/df/d27/daa/fb3 zero size 2024-04-12T05:37:37.439 INFO:tasks.workunit.client.1.smithi067.stdout:4/928: unlink d8/dba/d88/da0/fee 0 2024-04-12T05:37:37.440 INFO:tasks.workunit.client.1.smithi067.stdout:3/910: creat dc/d1d/d28/d3e/d53/d76/da7/df1/dc8/da3/f13d x:0 0 0 2024-04-12T05:37:37.441 INFO:tasks.workunit.client.1.smithi067.stdout:4/929: fdatasync d8/da/d9f/d68/f114 0 2024-04-12T05:37:37.442 INFO:tasks.workunit.client.1.smithi067.stdout:4/930: dread - d8/da/d9f/d68/f114 zero size 2024-04-12T05:37:37.444 INFO:tasks.workunit.client.1.smithi067.stdout:3/911: write dc/d1d/d26/fc6 [3498652,80429] 0 2024-04-12T05:37:37.444 INFO:tasks.workunit.client.1.smithi067.stdout:4/931: chown d8/dba/d5d/d66/d7b/d6d/ffb 35 1 2024-04-12T05:37:37.448 INFO:tasks.workunit.client.1.smithi067.stdout:2/964: dread d5/da/de/d14/d34/fac [0,4194304] 0 2024-04-12T05:37:37.451 INFO:tasks.workunit.client.1.smithi067.stdout:4/932: truncate d8/da/d1d/d15/f12a 180291 0 2024-04-12T05:37:37.463 INFO:tasks.workunit.client.1.smithi067.stdout:2/965: write d5/da/dae/dba/fc9 [569848,40706] 0 2024-04-12T05:37:37.463 INFO:tasks.workunit.client.1.smithi067.stdout:3/912: dwrite dc/d1d/d26/d52/d9a/d9c/f111 [0,4194304] 0 2024-04-12T05:37:37.476 INFO:tasks.workunit.client.1.smithi067.stdout:4/933: readlink d8/da/d1d/d15/l94 0 2024-04-12T05:37:37.476 INFO:tasks.workunit.client.1.smithi067.stdout:2/966: mkdir d5/da/d8c/d12b 0 2024-04-12T05:37:37.477 INFO:tasks.workunit.client.1.smithi067.stdout:4/934: chown d8/da/d1d/d15/d58/l98 82734253 1 2024-04-12T05:37:37.477 INFO:tasks.workunit.client.1.smithi067.stdout:3/913: fsync dc/d1d/d28/d3e/d84/f85 0 2024-04-12T05:37:37.479 INFO:tasks.workunit.client.1.smithi067.stdout:2/967: mkdir d5/da/de/dc4/def/d12c 0 2024-04-12T05:37:37.483 INFO:tasks.workunit.client.1.smithi067.stdout:4/935: creat d8/dba/d5d/d66/d71/f134 x:0 0 0 2024-04-12T05:37:37.484 INFO:tasks.workunit.client.1.smithi067.stdout:2/968: creat d5/da/d8c/d12b/f12d x:0 0 0 2024-04-12T05:37:37.485 INFO:tasks.workunit.client.1.smithi067.stdout:4/936: creat d8/dba/d5d/d66/d7b/d6d/f135 x:0 0 0 2024-04-12T05:37:37.488 INFO:tasks.workunit.client.1.smithi067.stdout:2/969: unlink d5/d6a/f65 0 2024-04-12T05:37:37.488 INFO:tasks.workunit.client.1.smithi067.stdout:2/970: chown d5/fa0 1519318 1 2024-04-12T05:37:37.489 INFO:tasks.workunit.client.1.smithi067.stdout:4/937: fsync d8/da/d8b/d45/f6e 0 2024-04-12T05:37:37.489 INFO:tasks.workunit.client.1.smithi067.stdout:3/914: link dc/d1d/d26/d52/d9a/c112 dc/d1d/d26/d52/d9a/d9c/c13e 0 2024-04-12T05:37:37.491 INFO:tasks.workunit.client.1.smithi067.stdout:2/971: symlink d5/d6a/d78/l12e 0 2024-04-12T05:37:37.493 INFO:tasks.workunit.client.1.smithi067.stdout:3/915: creat dc/df/df4/df6/f13f x:0 0 0 2024-04-12T05:37:37.494 INFO:tasks.workunit.client.1.smithi067.stdout:3/916: stat dc/d1d/d28/d3e/d69/d113/c4e 0 2024-04-12T05:37:37.495 INFO:tasks.workunit.client.1.smithi067.stdout:2/972: symlink d5/da/de/d39/l12f 0 2024-04-12T05:37:37.497 INFO:tasks.workunit.client.1.smithi067.stdout:3/917: mkdir dc/d1d/d26/d52/d140 0 2024-04-12T05:37:37.498 INFO:tasks.workunit.client.1.smithi067.stdout:3/918: readlink dc/df/d27/dc5/dd8/le2 0 2024-04-12T05:37:37.499 INFO:tasks.workunit.client.1.smithi067.stdout:3/919: write dc/d1d/d28/d3e/d53/d76/f11c [3239134,13707] 0 2024-04-12T05:37:37.500 INFO:tasks.workunit.client.1.smithi067.stdout:2/973: creat d5/da/de/d14/d34/d40/d57/dfe/f130 x:0 0 0 2024-04-12T05:37:37.503 INFO:tasks.workunit.client.1.smithi067.stdout:3/920: creat dc/d1d/d26/d52/d9a/f141 x:0 0 0 2024-04-12T05:37:37.507 INFO:tasks.workunit.client.1.smithi067.stdout:3/921: rename dc/d1d/d28/d3e/d69 to dc/df/dca/d142 0 2024-04-12T05:37:37.513 INFO:tasks.workunit.client.1.smithi067.stdout:3/922: dread dc/df/d27/f96 [0,4194304] 0 2024-04-12T05:37:37.516 INFO:tasks.workunit.client.1.smithi067.stdout:4/938: dwrite d8/da/f1f [4194304,4194304] 0 2024-04-12T05:37:37.518 INFO:tasks.workunit.client.1.smithi067.stdout:4/939: chown d8/da/d1d/d15/f12a 6 1 2024-04-12T05:37:37.518 INFO:tasks.workunit.client.1.smithi067.stdout:3/923: read dc/d1d/d28/d3e/d53/d78/ff2 [271465,83709] 0 2024-04-12T05:37:37.519 INFO:tasks.workunit.client.1.smithi067.stdout:4/940: readlink d8/dba/d5d/d66/d96/l118 0 2024-04-12T05:37:37.522 INFO:tasks.workunit.client.1.smithi067.stdout:3/924: mkdir dc/d1d/d28/d3e/d53/d76/da7/d143 0 2024-04-12T05:37:37.525 INFO:tasks.workunit.client.1.smithi067.stdout:3/925: mknod dc/d1d/d26/d52/d9a/c144 0 2024-04-12T05:37:37.526 INFO:tasks.workunit.client.1.smithi067.stdout:3/926: write dc/df/d27/daa/fef [689509,14441] 0 2024-04-12T05:37:37.529 INFO:tasks.workunit.client.1.smithi067.stdout:3/927: symlink dc/d1d/d28/d3e/d53/l145 0 2024-04-12T05:37:37.530 INFO:tasks.workunit.client.1.smithi067.stdout:3/928: rmdir dc 39 2024-04-12T05:37:37.532 INFO:tasks.workunit.client.1.smithi067.stdout:4/941: write d8/da/f30 [1380694,68611] 0 2024-04-12T05:37:37.533 INFO:tasks.workunit.client.1.smithi067.stdout:3/929: creat dc/df/df4/df6/f146 x:0 0 0 2024-04-12T05:37:37.534 INFO:tasks.workunit.client.1.smithi067.stdout:4/942: fsync d8/da/d1d/f49 0 2024-04-12T05:37:37.537 INFO:tasks.workunit.client.1.smithi067.stdout:4/943: mknod d8/da/c136 0 2024-04-12T05:37:37.538 INFO:tasks.workunit.client.1.smithi067.stdout:4/944: write d8/da/d1d/d27/d91/fad [2945927,60069] 0 2024-04-12T05:37:37.541 INFO:tasks.workunit.client.1.smithi067.stdout:4/945: mkdir d8/d137 0 2024-04-12T05:37:37.542 INFO:tasks.workunit.client.1.smithi067.stdout:3/930: link dc/df/dca/d142/d113/l7d dc/d1d/d28/d3e/d53/l147 0 2024-04-12T05:37:37.543 INFO:tasks.workunit.client.1.smithi067.stdout:4/946: mkdir d8/da/d1d/d27/d138 0 2024-04-12T05:37:37.546 INFO:tasks.workunit.client.1.smithi067.stdout:4/947: write d8/da/d1d/d27/d91/fad [1631991,10137] 0 2024-04-12T05:37:37.549 INFO:tasks.workunit.client.1.smithi067.stdout:4/948: creat d8/da/d9f/d68/dbf/f139 x:0 0 0 2024-04-12T05:37:37.550 INFO:tasks.workunit.client.1.smithi067.stdout:4/949: dread - d8/dba/d5d/d66/df8/f109 zero size 2024-04-12T05:37:37.550 INFO:tasks.workunit.client.1.smithi067.stdout:3/931: write dc/d1d/d26/f72 [1173390,59663] 0 2024-04-12T05:37:37.552 INFO:tasks.workunit.client.1.smithi067.stdout:3/932: fsync dc/d1d/f107 0 2024-04-12T05:37:37.553 INFO:tasks.workunit.client.1.smithi067.stdout:4/950: symlink d8/d137/l13a 0 2024-04-12T05:37:37.554 INFO:tasks.workunit.client.1.smithi067.stdout:3/933: creat dc/df/dca/f148 x:0 0 0 2024-04-12T05:37:37.556 INFO:tasks.workunit.client.1.smithi067.stdout:4/951: creat d8/da/d1d/d27/d91/d11b/f13b x:0 0 0 2024-04-12T05:37:37.558 INFO:tasks.workunit.client.1.smithi067.stdout:3/934: rename dc/d1d/d28/d3e/d53/d76/fb4 to dc/d1d/d28/d131/f149 0 2024-04-12T05:37:37.560 INFO:tasks.workunit.client.1.smithi067.stdout:3/935: write dc/d1d/d28/d3e/f6d [3494958,16664] 0 2024-04-12T05:37:37.561 INFO:tasks.workunit.client.1.smithi067.stdout:4/952: creat d8/da/d1d/d15/d75/daa/f13c x:0 0 0 2024-04-12T05:37:37.563 INFO:tasks.workunit.client.1.smithi067.stdout:4/953: write d8/da/d9f/df7/f132 [775612,130727] 0 2024-04-12T05:37:37.569 INFO:tasks.workunit.client.1.smithi067.stdout:3/936: dwrite dc/df/df4/df6/f146 [0,4194304] 0 2024-04-12T05:37:37.571 INFO:tasks.workunit.client.1.smithi067.stdout:3/937: write dc/df/f2c [1673254,37325] 0 2024-04-12T05:37:37.576 INFO:tasks.workunit.client.1.smithi067.stdout:3/938: mkdir dc/d1d/d28/d3e/d53/d76/da7/df1/dc8/da3/d14a 0 2024-04-12T05:37:37.579 INFO:tasks.workunit.client.1.smithi067.stdout:3/939: rmdir dc/df/d27 39 2024-04-12T05:37:37.582 INFO:tasks.workunit.client.1.smithi067.stdout:4/954: dwrite d8/dba/d5d/d66/d7b/fbb [0,4194304] 0 2024-04-12T05:37:37.583 INFO:tasks.workunit.client.1.smithi067.stdout:3/940: mkdir dc/d1d/d26/d52/d9a/d13b/d14b 0 2024-04-12T05:37:37.589 INFO:tasks.workunit.client.1.smithi067.stdout:4/955: rename d8/da/d8b/d45/f5c to d8/dba/f13d 0 2024-04-12T05:37:37.591 INFO:tasks.workunit.client.1.smithi067.stdout:3/941: creat dc/df/dca/d142/d113/f14c x:0 0 0 2024-04-12T05:37:37.592 INFO:tasks.workunit.client.1.smithi067.stdout:4/956: unlink d8/da/d1d/d15/d75/le1 0 2024-04-12T05:37:37.592 INFO:tasks.workunit.client.1.smithi067.stdout:3/942: mknod dc/de4/c14d 0 2024-04-12T05:37:37.594 INFO:tasks.workunit.client.1.smithi067.stdout:4/957: write d8/da/d9f/df7/f132 [1123923,65479] 0 2024-04-12T05:37:37.595 INFO:tasks.workunit.client.1.smithi067.stdout:3/943: symlink dc/d1d/d28/d3e/d84/l14e 0 2024-04-12T05:37:37.598 INFO:tasks.workunit.client.1.smithi067.stdout:4/958: creat d8/da/d9f/f13e x:0 0 0 2024-04-12T05:37:37.598 INFO:tasks.workunit.client.1.smithi067.stdout:3/944: creat dc/df/d8d/f14f x:0 0 0 2024-04-12T05:37:37.603 INFO:tasks.workunit.client.1.smithi067.stdout:4/959: getdents d8/da/d9f/d68/db4 0 2024-04-12T05:37:37.604 INFO:tasks.workunit.client.1.smithi067.stdout:3/945: getdents dc/d1d/d28/d3e 0 2024-04-12T05:37:37.606 INFO:tasks.workunit.client.1.smithi067.stdout:3/946: dread dc/d1d/d26/d52/d9a/f10a [0,4194304] 0 2024-04-12T05:37:37.611 INFO:tasks.workunit.client.1.smithi067.stdout:3/947: getdents dc/df/d27/dc5/dd8 0 2024-04-12T05:37:37.614 INFO:tasks.workunit.client.1.smithi067.stdout:4/960: truncate d8/da/d1d/d27/d91/ff4 3858994 0 2024-04-12T05:37:37.615 INFO:tasks.workunit.client.1.smithi067.stdout:4/961: truncate d8/dba/d5d/d66/df8/f109 559003 0 2024-04-12T05:37:37.616 INFO:tasks.workunit.client.1.smithi067.stdout:3/948: link dc/df/d8d/f13c dc/d50/da6/f150 0 2024-04-12T05:37:37.619 INFO:tasks.workunit.client.1.smithi067.stdout:3/949: creat dc/d1d/d28/d3e/d53/f151 x:0 0 0 2024-04-12T05:37:37.622 INFO:tasks.workunit.client.1.smithi067.stdout:4/962: dread d8/da/d9f/df7/fac [0,4194304] 0 2024-04-12T05:37:37.623 INFO:tasks.workunit.client.1.smithi067.stdout:3/950: mknod dc/df/d27/c152 0 2024-04-12T05:37:37.624 INFO:tasks.workunit.client.1.smithi067.stdout:3/951: stat dc/d1d/d28/d3e/d8a/dfd/c12f 0 2024-04-12T05:37:37.625 INFO:tasks.workunit.client.1.smithi067.stdout:4/963: read d8/da/d1d/d15/d75/f7c [395789,22747] 0 2024-04-12T05:37:37.629 INFO:tasks.workunit.client.1.smithi067.stdout:3/952: write dc/d1d/d28/d3e/d53/d78/f8b [1071032,113476] 0 2024-04-12T05:37:37.630 INFO:tasks.workunit.client.1.smithi067.stdout:4/964: link d8/da/d8b/d45/cf9 d8/da/c13f 0 2024-04-12T05:37:37.634 INFO:tasks.workunit.client.1.smithi067.stdout:3/953: link dc/d50/f125 dc/df/d27/daa/f153 0 2024-04-12T05:37:37.636 INFO:tasks.workunit.client.1.smithi067.stdout:3/954: write dc/df/d8d/f14f [342922,102985] 0 2024-04-12T05:37:37.638 INFO:tasks.workunit.client.1.smithi067.stdout:3/955: stat dc/d50/da6/f150 0 2024-04-12T05:37:37.638 INFO:tasks.workunit.client.1.smithi067.stdout:4/965: dwrite d8/dba/d88/da0/de7/d129/f103 [0,4194304] 0 2024-04-12T05:37:37.641 INFO:tasks.workunit.client.1.smithi067.stdout:4/966: symlink d8/dba/d88/da0/l140 0 2024-04-12T05:37:37.642 INFO:tasks.workunit.client.1.smithi067.stdout:4/967: write d8/da/d9f/f122 [534686,13179] 0 2024-04-12T05:37:37.643 INFO:tasks.workunit.client.1.smithi067.stdout:3/956: mknod dc/d1d/d26/c154 0 2024-04-12T05:37:37.648 INFO:tasks.workunit.client.1.smithi067.stdout:3/957: creat dc/df/d27/dc5/dd8/de8/f155 x:0 0 0 2024-04-12T05:37:37.657 INFO:tasks.workunit.client.1.smithi067.stdout:3/958: dwrite dc/df/dca/d142/d113/f14c [0,4194304] 0 2024-04-12T05:37:37.661 INFO:tasks.workunit.client.1.smithi067.stdout:3/959: write dc/d1d/d28/d3e/d53/d76/da7/df1/dc8/da3/db9/fee [228798,125144] 0 2024-04-12T05:37:37.661 INFO:tasks.workunit.client.1.smithi067.stdout:4/968: dread d8/da/d8b/ffe [0,4194304] 0 2024-04-12T05:37:37.665 INFO:tasks.workunit.client.1.smithi067.stdout:6/994: sync 2024-04-12T05:37:37.666 INFO:tasks.workunit.client.1.smithi067.stdout:8/904: sync 2024-04-12T05:37:37.670 INFO:tasks.workunit.client.1.smithi067.stdout:6/995: creat dd/d11/d27/d4a/d79/dc3/d9e/f146 x:0 0 0 2024-04-12T05:37:37.672 INFO:tasks.workunit.client.1.smithi067.stdout:8/905: rename d6/d8/f72 to d6/f110 0 2024-04-12T05:37:37.673 INFO:tasks.workunit.client.1.smithi067.stdout:4/969: dwrite d8/da/d9f/df7/f11e [0,4194304] 0 2024-04-12T05:37:37.674 INFO:tasks.workunit.client.1.smithi067.stdout:6/996: readlink dd/d21/d64/d88/d7b/lc5 0 2024-04-12T05:37:37.677 INFO:tasks.workunit.client.1.smithi067.stdout:8/906: symlink d6/def/d9a/d12/df4/l111 0 2024-04-12T05:37:37.679 INFO:tasks.workunit.client.1.smithi067.stdout:8/907: write d6/d8/d8b/da1/fed [194852,91696] 0 2024-04-12T05:37:37.679 INFO:tasks.workunit.client.1.smithi067.stdout:4/970: rename d8/dba/d88/da0/de7/d129/ld0 to d8/da/d9f/dcf/l141 0 2024-04-12T05:37:37.680 INFO:tasks.workunit.client.1.smithi067.stdout:4/971: fdatasync d8/d39/f46 0 2024-04-12T05:37:37.681 INFO:tasks.workunit.client.1.smithi067.stdout:8/908: fsync d6/d8/d50/dd4/fe1 0 2024-04-12T05:37:37.684 INFO:tasks.workunit.client.1.smithi067.stdout:8/909: fsync d6/def/fb6 0 2024-04-12T05:37:37.685 INFO:tasks.workunit.client.1.smithi067.stdout:4/972: symlink d8/dba/d5d/d66/df8/l142 0 2024-04-12T05:37:37.685 INFO:tasks.workunit.client.1.smithi067.stdout:8/910: chown d6/def/d9a/dd7/d82/cc1 61985 1 2024-04-12T05:37:37.687 INFO:tasks.workunit.client.1.smithi067.stdout:8/911: fdatasync d6/def/d9a/d12/de9/fec 0 2024-04-12T05:37:37.691 INFO:tasks.workunit.client.1.smithi067.stdout:8/912: link d6/def/d9a/ff8 d6/d8/d8b/df7/f112 0 2024-04-12T05:37:37.691 INFO:tasks.workunit.client.1.smithi067.stdout:6/997: truncate dd/d11/f5b 943586 0 2024-04-12T05:37:37.694 INFO:tasks.workunit.client.1.smithi067.stdout:6/998: mkdir dd/d11/d27/d147 0 2024-04-12T05:37:37.695 INFO:tasks.workunit.client.1.smithi067.stdout:6/999: dread - dd/d11/d27/d4a/df8/d129/f139 zero size 2024-04-12T05:37:37.700 INFO:tasks.workunit.client.1.smithi067.stdout:8/913: write d6/d8/d50/dd4/f35 [2299344,103945] 0 2024-04-12T05:37:37.706 INFO:tasks.workunit.client.1.smithi067.stdout:8/914: write d6/def/d9a/dd7/d82/f9d [1282308,76936] 0 2024-04-12T05:37:37.707 INFO:tasks.workunit.client.1.smithi067.stdout:8/915: stat d6/d8/d50/dd4/f62 0 2024-04-12T05:37:37.708 INFO:tasks.workunit.client.1.smithi067.stdout:8/916: fsync d6/def/d9a/f20 0 2024-04-12T05:37:37.711 INFO:tasks.workunit.client.1.smithi067.stdout:8/917: write d6/d8/d50/dd4/de7/d69/d5d/f91 [5118215,124462] 0 2024-04-12T05:37:37.718 INFO:tasks.workunit.client.1.smithi067.stdout:8/918: dread d6/d8/d8b/df7/f78 [0,4194304] 0 2024-04-12T05:37:37.721 INFO:tasks.workunit.client.1.smithi067.stdout:8/919: getdents d6/d8/d50/dd4/de7 0 2024-04-12T05:37:37.723 INFO:tasks.workunit.client.1.smithi067.stdout:8/920: mknod d6/def/d9a/c113 0 2024-04-12T05:37:37.724 INFO:tasks.workunit.client.1.smithi067.stdout:8/921: fdatasync d6/def/d9a/dbb/dd9/fe4 0 2024-04-12T05:37:37.725 INFO:tasks.workunit.client.1.smithi067.stdout:8/922: write d6/d8/d8b/da1/fed [917046,113137] 0 2024-04-12T05:37:37.729 INFO:tasks.workunit.client.1.smithi067.stdout:8/923: dread d6/d8/f6a [0,4194304] 0 2024-04-12T05:37:37.732 INFO:tasks.workunit.client.1.smithi067.stdout:8/924: rename d6/d8/d50/dd4/ld0 to d6/d8/d50/dd4/de7/d104/l114 0 2024-04-12T05:37:37.734 INFO:tasks.workunit.client.1.smithi067.stdout:8/925: symlink d6/d8/d50/dd4/d36/d10f/l115 0 2024-04-12T05:37:37.744 INFO:tasks.workunit.client.1.smithi067.stdout:8/926: dwrite d6/def/d9a/f93 [0,4194304] 0 2024-04-12T05:37:37.746 INFO:tasks.workunit.client.1.smithi067.stdout:8/927: write d6/def/d9a/dd7/f65 [3613361,9303] 0 2024-04-12T05:37:37.753 INFO:tasks.workunit.client.1.smithi067.stdout:8/928: getdents d6/def/d9a/d12/de9 0 2024-04-12T05:37:37.757 INFO:tasks.workunit.client.1.smithi067.stdout:8/929: write d6/def/d9a/f5f [1202825,82721] 0 2024-04-12T05:37:37.770 INFO:tasks.workunit.client.1.smithi067.stdout:8/930: dread d6/f8a [0,4194304] 0 2024-04-12T05:37:37.772 INFO:tasks.workunit.client.1.smithi067.stdout:8/931: rmdir d6/def/d9a 39 2024-04-12T05:37:37.774 INFO:tasks.workunit.client.1.smithi067.stdout:8/932: creat d6/d8/d50/dd4/f116 x:0 0 0 2024-04-12T05:37:37.777 INFO:tasks.workunit.client.1.smithi067.stdout:8/933: truncate d6/d8/d8b/df7/f84 1617246 0 2024-04-12T05:37:37.813 INFO:tasks.workunit.client.1.smithi067.stdout:2/974: sync 2024-04-12T05:37:37.818 INFO:tasks.workunit.client.1.smithi067.stdout:8/934: write d6/d8/d50/dd4/f33 [4608848,9607] 0 2024-04-12T05:37:37.818 INFO:tasks.workunit.client.1.smithi067.stdout:2/975: chown d5/d23/ced 1052020396 1 2024-04-12T05:37:37.820 INFO:tasks.workunit.client.1.smithi067.stdout:2/976: write d5/da/de/d14/d46/ffb [1759850,119563] 0 2024-04-12T05:37:37.826 INFO:tasks.workunit.client.1.smithi067.stdout:8/935: write d6/f110 [887166,34551] 0 2024-04-12T05:37:37.829 INFO:tasks.workunit.client.1.smithi067.stdout:8/936: read - d6/def/d9a/fcf zero size 2024-04-12T05:37:37.830 INFO:tasks.workunit.client.1.smithi067.stdout:2/977: dwrite d5/da/de/d14/d34/fa3 [4194304,4194304] 0 2024-04-12T05:37:37.833 INFO:tasks.workunit.client.1.smithi067.stdout:8/937: creat d6/def/d9a/d12/de9/f117 x:0 0 0 2024-04-12T05:37:37.838 INFO:tasks.workunit.client.1.smithi067.stdout:2/978: link d5/da/de/l1c d5/da/de/d14/d46/d90/l131 0 2024-04-12T05:37:37.839 INFO:tasks.workunit.client.1.smithi067.stdout:2/979: dread - d5/da/d3d/f9f zero size 2024-04-12T05:37:37.839 INFO:tasks.workunit.client.1.smithi067.stdout:8/938: link d6/def/fb6 d6/def/d9a/f118 0 2024-04-12T05:37:37.841 INFO:tasks.workunit.client.1.smithi067.stdout:8/939: creat d6/df2/f119 x:0 0 0 2024-04-12T05:37:37.842 INFO:tasks.workunit.client.1.smithi067.stdout:2/980: symlink d5/da/de/d14/d46/d90/l132 0 2024-04-12T05:37:37.843 INFO:tasks.workunit.client.1.smithi067.stdout:8/940: symlink d6/d86/l11a 0 2024-04-12T05:37:37.844 INFO:tasks.workunit.client.1.smithi067.stdout:8/941: stat d6/d86/laa 0 2024-04-12T05:37:37.846 INFO:tasks.workunit.client.1.smithi067.stdout:2/981: rename d5/d11/f8a to d5/da/dae/f133 0 2024-04-12T05:37:37.847 INFO:tasks.workunit.client.1.smithi067.stdout:8/942: mkdir d6/d8/d50/dd4/de7/d11b 0 2024-04-12T05:37:37.854 INFO:tasks.workunit.client.1.smithi067.stdout:2/982: dwrite d5/da/de/d14/d34/f102 [0,4194304] 0 2024-04-12T05:37:37.857 INFO:tasks.workunit.client.1.smithi067.stdout:8/943: dwrite d6/d8/d50/dd4/d36/f68 [0,4194304] 0 2024-04-12T05:37:37.858 INFO:tasks.workunit.client.1.smithi067.stdout:2/983: symlink d5/da/de/d39/dde/l134 0 2024-04-12T05:37:37.861 INFO:tasks.workunit.client.1.smithi067.stdout:2/984: stat d5/d11/c1b 0 2024-04-12T05:37:37.866 INFO:tasks.workunit.client.1.smithi067.stdout:8/944: dwrite d6/def/d9a/dd7/fc5 [0,4194304] 0 2024-04-12T05:37:37.872 INFO:tasks.workunit.client.1.smithi067.stdout:8/945: link d6/def/d9a/l7e d6/d8/d8b/l11c 0 2024-04-12T05:37:37.877 INFO:tasks.workunit.client.1.smithi067.stdout:8/946: rename d6/def/d9a/d12/l37 to d6/def/d9a/dd7/d82/l11d 0 2024-04-12T05:37:37.884 INFO:tasks.workunit.client.1.smithi067.stdout:2/985: dread d5/da/de/d14/d34/d40/d57/f60 [0,4194304] 0 2024-04-12T05:37:37.884 INFO:tasks.workunit.client.1.smithi067.stdout:8/947: dwrite d6/d8/d50/dd4/f33 [0,4194304] 0 2024-04-12T05:37:37.888 INFO:tasks.workunit.client.1.smithi067.stdout:2/986: symlink d5/d11/l135 0 2024-04-12T05:37:37.889 INFO:tasks.workunit.client.1.smithi067.stdout:2/987: stat d5/c54 0 2024-04-12T05:37:37.890 INFO:tasks.workunit.client.1.smithi067.stdout:2/988: dread - d5/da/de/dc4/def/f123 zero size 2024-04-12T05:37:37.891 INFO:tasks.workunit.client.1.smithi067.stdout:8/948: getdents d6/d8 0 2024-04-12T05:37:37.901 INFO:tasks.workunit.client.1.smithi067.stdout:8/949: rename d6/def/d9a/dd7/d82 to d6/d8/d50/dd4/de7/d11b/d11e 0 2024-04-12T05:37:37.902 INFO:tasks.workunit.client.1.smithi067.stdout:2/989: write d5/d23/d3a/f6f [1702758,81009] 0 2024-04-12T05:37:37.902 INFO:tasks.workunit.client.1.smithi067.stdout:8/950: fsync d6/def/d9a/dd7/fd6 0 2024-04-12T05:37:37.903 INFO:tasks.workunit.client.1.smithi067.stdout:2/990: read - d5/dbf/dcc/f125 zero size 2024-04-12T05:37:37.905 INFO:tasks.workunit.client.1.smithi067.stdout:8/951: creat d6/f11f x:0 0 0 2024-04-12T05:37:37.907 INFO:tasks.workunit.client.1.smithi067.stdout:2/991: rename d5/da/dae/f133 to d5/da/dae/f136 0 2024-04-12T05:37:37.909 INFO:tasks.workunit.client.1.smithi067.stdout:2/992: fdatasync d5/d11/f2a 0 2024-04-12T05:37:37.911 INFO:tasks.workunit.client.1.smithi067.stdout:2/993: unlink d5/d23/d93/fc5 0 2024-04-12T05:37:37.914 INFO:tasks.workunit.client.1.smithi067.stdout:2/994: rmdir d5/da/de/d14 39 2024-04-12T05:37:37.915 INFO:tasks.workunit.client.1.smithi067.stdout:2/995: dread - d5/da/de/dc4/def/f107 zero size 2024-04-12T05:37:37.921 INFO:tasks.workunit.client.1.smithi067.stdout:2/996: dread d5/d43/d6e/f72 [0,4194304] 0 2024-04-12T05:37:37.923 INFO:tasks.workunit.client.1.smithi067.stdout:2/997: creat d5/da/d3d/df7/d112/f137 x:0 0 0 2024-04-12T05:37:37.925 INFO:tasks.workunit.client.1.smithi067.stdout:2/998: fdatasync d5/d43/d6e/f72 0 2024-04-12T05:37:37.927 INFO:tasks.workunit.client.1.smithi067.stdout:2/999: creat d5/da/d8c/d12b/f138 x:0 0 0 2024-04-12T05:37:38.072 INFO:tasks.workunit.client.1.smithi067.stdout:3/960: sync 2024-04-12T05:37:38.075 INFO:tasks.workunit.client.1.smithi067.stdout:3/961: truncate dc/d1d/d28/d3e/d53/d76/da7/df1/dc8/da3/db9/f137 551891 0 2024-04-12T05:37:38.077 INFO:tasks.workunit.client.1.smithi067.stdout:3/962: symlink dc/d1d/d28/d3e/d53/d76/da7/df1/dc8/da3/dde/l156 0 2024-04-12T05:37:38.078 INFO:tasks.workunit.client.1.smithi067.stdout:3/963: chown dc/d1d/d26/d52 4 1 2024-04-12T05:37:38.082 INFO:tasks.workunit.client.1.smithi067.stdout:3/964: dread dc/d1d/d28/f64 [0,4194304] 0 2024-04-12T05:37:38.087 INFO:tasks.workunit.client.1.smithi067.stdout:3/965: rename dc/d1d/cc9 to dc/d1d/d28/d3e/d53/d76/da7/df1/dc8/da3/db9/d12d/c157 0 2024-04-12T05:37:38.089 INFO:tasks.workunit.client.1.smithi067.stdout:3/966: symlink dc/d1d/d28/d3e/d84/l158 0 2024-04-12T05:37:38.090 INFO:tasks.workunit.client.1.smithi067.stdout:3/967: mknod dc/d1d/d28/d131/c159 0 2024-04-12T05:37:38.094 INFO:tasks.workunit.client.1.smithi067.stdout:3/968: dread dc/d1d/d26/fc6 [0,4194304] 0 2024-04-12T05:37:38.097 INFO:tasks.workunit.client.1.smithi067.stdout:3/969: symlink dc/df/df4/l15a 0 2024-04-12T05:37:38.100 INFO:tasks.workunit.client.1.smithi067.stdout:3/970: dread dc/d1d/d28/d3e/f7f [0,4194304] 0 2024-04-12T05:37:38.105 INFO:tasks.workunit.client.1.smithi067.stdout:3/971: creat dc/df/d27/f15b x:0 0 0 2024-04-12T05:37:38.132 INFO:tasks.workunit.client.1.smithi067.stdout:3/972: dwrite dc/df/df4/fa2 [0,4194304] 0 2024-04-12T05:37:38.132 INFO:tasks.workunit.client.1.smithi067.stdout:3/973: chown dc/d50/da6/dbd 13442 1 2024-04-12T05:37:38.138 INFO:tasks.workunit.client.1.smithi067.stdout:4/973: sync 2024-04-12T05:37:38.142 INFO:tasks.workunit.client.1.smithi067.stdout:3/974: dwrite dc/d1d/d28/f2a [0,4194304] 0 2024-04-12T05:37:38.143 INFO:tasks.workunit.client.1.smithi067.stdout:8/952: sync 2024-04-12T05:37:38.149 INFO:tasks.workunit.client.1.smithi067.stdout:8/953: rename d6/f11f to d6/d8/d8b/f120 0 2024-04-12T05:37:38.154 INFO:tasks.workunit.client.1.smithi067.stdout:3/975: getdents dc/d1d/d28/d3e/d53/d76/da7 0 2024-04-12T05:37:38.154 INFO:tasks.workunit.client.1.smithi067.stdout:8/954: chown d6/def/d9a/dd7/fd6 146292250 1 2024-04-12T05:37:38.162 INFO:tasks.workunit.client.1.smithi067.stdout:3/976: rename dc/df/d27/dc5/dd8/le2 to dc/d1d/d28/d3e/d53/l15c 0 2024-04-12T05:37:38.162 INFO:tasks.workunit.client.1.smithi067.stdout:4/974: truncate d8/da/d8b/d8c/fe3 972627 0 2024-04-12T05:37:38.168 INFO:tasks.workunit.client.1.smithi067.stdout:8/955: dwrite d6/def/d9a/f90 [0,4194304] 0 2024-04-12T05:37:38.170 INFO:tasks.workunit.client.1.smithi067.stdout:8/956: fdatasync d6/d8/d8b/da1/fed 0 2024-04-12T05:37:38.181 INFO:tasks.workunit.client.1.smithi067.stdout:8/957: dwrite d6/d8/d50/fe6 [0,4194304] 0 2024-04-12T05:37:38.189 INFO:tasks.workunit.client.1.smithi067.stdout:8/958: creat d6/d8/d50/dd4/de7/f121 x:0 0 0 2024-04-12T05:37:38.190 INFO:tasks.workunit.client.1.smithi067.stdout:8/959: write d6/def/d9a/d12/f1f [2056545,31676] 0 2024-04-12T05:37:38.193 INFO:tasks.workunit.client.1.smithi067.stdout:8/960: unlink d6/def/d9a/d12/c103 0 2024-04-12T05:37:38.198 INFO:tasks.workunit.client.1.smithi067.stdout:4/975: dread d8/da/d9f/d68/dbf/fb7 [0,4194304] 0 2024-04-12T05:37:38.201 INFO:tasks.workunit.client.1.smithi067.stdout:4/976: mknod d8/d137/c143 0 2024-04-12T05:37:38.203 INFO:tasks.workunit.client.1.smithi067.stdout:8/961: dread d6/def/d9a/d12/fa9 [0,4194304] 0 2024-04-12T05:37:38.204 INFO:tasks.workunit.client.1.smithi067.stdout:8/962: write d6/def/d9a/fcf [629867,76397] 0 2024-04-12T05:37:38.206 INFO:tasks.workunit.client.1.smithi067.stdout:8/963: stat d6/d8/d8b/dc3/ld5 0 2024-04-12T05:37:38.206 INFO:tasks.workunit.client.1.smithi067.stdout:4/977: unlink d8/da/d1d/d15/d75/cea 0 2024-04-12T05:37:38.209 INFO:tasks.workunit.client.1.smithi067.stdout:4/978: symlink d8/da/d8b/d45/l144 0 2024-04-12T05:37:38.210 INFO:tasks.workunit.client.1.smithi067.stdout:8/964: creat d6/def/d9a/d12/d108/f122 x:0 0 0 2024-04-12T05:37:38.212 INFO:tasks.workunit.client.1.smithi067.stdout:8/965: truncate d6/d8/d50/dd4/fe1 871410 0 2024-04-12T05:37:38.212 INFO:tasks.workunit.client.1.smithi067.stdout:4/979: truncate d8/da/d1d/d27/d91/d11b/f12e 4368627 0 2024-04-12T05:37:38.213 INFO:tasks.workunit.client.1.smithi067.stdout:8/966: write d6/f95 [945479,18535] 0 2024-04-12T05:37:38.214 INFO:tasks.workunit.client.1.smithi067.stdout:4/980: chown d8/da/d8b/fa7 29367155 1 2024-04-12T05:37:38.222 INFO:tasks.workunit.client.1.smithi067.stdout:8/967: mkdir d6/d8/d50/dd4/de7/d104/d123 0 2024-04-12T05:37:38.223 INFO:tasks.workunit.client.1.smithi067.stdout:8/968: chown d6/d8/d8b/df7/c81 16023050 1 2024-04-12T05:37:38.224 INFO:tasks.workunit.client.1.smithi067.stdout:4/981: truncate d8/dba/d5d/d66/d71/fda 1657882 0 2024-04-12T05:37:38.224 INFO:tasks.workunit.client.1.smithi067.stdout:8/969: chown d6/d86/laa 6 1 2024-04-12T05:37:38.226 INFO:tasks.workunit.client.1.smithi067.stdout:4/982: mkdir d8/da/d9f/db5/d145 0 2024-04-12T05:37:38.229 INFO:tasks.workunit.client.1.smithi067.stdout:4/983: creat d8/dba/d5d/d66/d7b/d6d/f146 x:0 0 0 2024-04-12T05:37:38.230 INFO:tasks.workunit.client.1.smithi067.stdout:8/970: rename d6/d8/d50/dd4/de7/d11b/d11e/leb to d6/d8/d50/dd4/de7/d104/d123/l124 0 2024-04-12T05:37:38.239 INFO:tasks.workunit.client.1.smithi067.stdout:8/971: dwrite d6/def/d9a/f23 [0,4194304] 0 2024-04-12T05:37:38.246 INFO:tasks.workunit.client.1.smithi067.stdout:4/984: dwrite d8/dba/d5d/d66/d71/fda [0,4194304] 0 2024-04-12T05:37:38.247 INFO:tasks.workunit.client.1.smithi067.stdout:4/985: readlink d8/dba/d88/da0/l140 0 2024-04-12T05:37:38.254 INFO:tasks.workunit.client.1.smithi067.stdout:8/972: link d6/d8/d50/fe6 d6/d8/d50/dd4/de7/d69/d5d/f125 0 2024-04-12T05:37:38.255 INFO:tasks.workunit.client.1.smithi067.stdout:4/986: creat d8/da/d1d/f147 x:0 0 0 2024-04-12T05:37:38.260 INFO:tasks.workunit.client.1.smithi067.stdout:4/987: read d8/f33 [430254,115645] 0 2024-04-12T05:37:38.261 INFO:tasks.workunit.client.1.smithi067.stdout:8/973: write d6/def/d9a/f118 [774705,91379] 0 2024-04-12T05:37:38.269 INFO:tasks.workunit.client.1.smithi067.stdout:4/988: dwrite d8/da/d9f/df7/f132 [0,4194304] 0 2024-04-12T05:37:38.270 INFO:tasks.workunit.client.1.smithi067.stdout:8/974: dwrite d6/def/d9a/f23 [0,4194304] 0 2024-04-12T05:37:38.272 INFO:tasks.workunit.client.1.smithi067.stdout:4/989: creat d8/da/d1d/d15/d75/f148 x:0 0 0 2024-04-12T05:37:38.273 INFO:tasks.workunit.client.1.smithi067.stdout:8/975: symlink d6/d8/d8b/l126 0 2024-04-12T05:37:38.274 INFO:tasks.workunit.client.1.smithi067.stdout:8/976: write d6/f2c [8170153,48541] 0 2024-04-12T05:37:38.278 INFO:tasks.workunit.client.1.smithi067.stdout:4/990: creat d8/da/d1d/d15/f149 x:0 0 0 2024-04-12T05:37:38.281 INFO:tasks.workunit.client.1.smithi067.stdout:4/991: write d8/da/d9f/df7/fac [4612608,101974] 0 2024-04-12T05:37:38.285 INFO:tasks.workunit.client.1.smithi067.stdout:4/992: creat d8/da/d1d/f14a x:0 0 0 2024-04-12T05:37:38.288 INFO:tasks.workunit.client.1.smithi067.stdout:4/993: rename d8/da/d1d/d27/d91/c95 to d8/dba/d5d/d66/d7b/dbd/dd1/c14b 0 2024-04-12T05:37:38.289 INFO:tasks.workunit.client.1.smithi067.stdout:4/994: chown d8/dba/d5d/d66/d71/fde 251398141 1 2024-04-12T05:37:38.291 INFO:tasks.workunit.client.1.smithi067.stdout:4/995: chown d8/dba/d5d/d66/d7b/dbd/dd1/cd5 461 1 2024-04-12T05:37:38.292 INFO:tasks.workunit.client.1.smithi067.stdout:4/996: read - d8/dba/d5d/d66/d7b/d6d/f146 zero size 2024-04-12T05:37:38.296 INFO:tasks.workunit.client.1.smithi067.stdout:4/997: rename d8/da/d8b/f12f to d8/da/d1d/d27/d91/f14c 0 2024-04-12T05:37:38.298 INFO:tasks.workunit.client.1.smithi067.stdout:4/998: mknod d8/da/d1d/d15/d75/c14d 0 2024-04-12T05:37:38.299 INFO:tasks.workunit.client.1.smithi067.stdout:4/999: write d8/d9/f2a [3428992,30538] 0 2024-04-12T05:37:38.496 INFO:tasks.workunit.client.1.smithi067.stdout:3/977: sync 2024-04-12T05:37:38.497 INFO:tasks.workunit.client.1.smithi067.stdout:3/978: fsync dc/d1d/d28/f62 0 2024-04-12T05:37:38.509 INFO:tasks.workunit.client.1.smithi067.stdout:3/979: rmdir dc/df/d27/dc5/dd8/de8 39 2024-04-12T05:37:38.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:38 smithi067 ceph-mon[49010]: pgmap v280: 65 pgs: 65 active+clean; 2.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 90 MiB/s rd, 214 MiB/s wr, 567 op/s 2024-04-12T05:37:38.538 INFO:tasks.workunit.client.1.smithi067.stdout:3/980: dwrite dc/df/d27/fa8 [0,4194304] 0 2024-04-12T05:37:38.540 INFO:tasks.workunit.client.1.smithi067.stdout:3/981: mkdir dc/d50/da6/dbd/d15d 0 2024-04-12T05:37:38.544 INFO:tasks.workunit.client.1.smithi067.stdout:3/982: creat dc/d1d/d26/f15e x:0 0 0 2024-04-12T05:37:38.547 INFO:tasks.workunit.client.1.smithi067.stdout:3/983: mkdir dc/d1d/d28/d3e/d53/d76/da7/df1/dc8/da3/db9/d12d/d15f 0 2024-04-12T05:37:38.550 INFO:tasks.workunit.client.1.smithi067.stdout:3/984: mkdir dc/d1d/d28/d3e/d53/d76/da7/df1/dc8/da3/d160 0 2024-04-12T05:37:38.555 INFO:tasks.workunit.client.1.smithi067.stdout:3/985: dwrite dc/f3a [4194304,4194304] 0 2024-04-12T05:37:38.558 INFO:tasks.workunit.client.1.smithi067.stdout:3/986: mknod dc/d1d/d28/d131/c161 0 2024-04-12T05:37:38.560 INFO:tasks.workunit.client.1.smithi067.stdout:3/987: write dc/d1d/d28/d3e/d53/d76/fad [1601051,32978] 0 2024-04-12T05:37:38.563 INFO:tasks.workunit.client.1.smithi067.stdout:3/988: readlink dc/d1d/d28/d3e/d53/d76/lf8 0 2024-04-12T05:37:38.565 INFO:tasks.workunit.client.1.smithi067.stdout:3/989: symlink dc/df/d8d/l162 0 2024-04-12T05:37:38.567 INFO:tasks.workunit.client.1.smithi067.stdout:3/990: creat dc/d1d/d26/d52/d9a/f163 x:0 0 0 2024-04-12T05:37:38.568 INFO:tasks.workunit.client.1.smithi067.stdout:3/991: chown dc/d1d/d28/d3e/d53/d94 11302334 1 2024-04-12T05:37:38.571 INFO:tasks.workunit.client.1.smithi067.stdout:3/992: creat dc/d1d/d26/f164 x:0 0 0 2024-04-12T05:37:38.572 INFO:tasks.workunit.client.1.smithi067.stdout:8/977: sync 2024-04-12T05:37:38.573 INFO:tasks.workunit.client.1.smithi067.stdout:8/978: chown d6/d8/d50/dd4/d4e/d9e/cea 119 1 2024-04-12T05:37:38.573 INFO:tasks.workunit.client.1.smithi067.stdout:3/993: unlink dc/df/l6e 0 2024-04-12T05:37:38.579 INFO:tasks.workunit.client.1.smithi067.stdout:3/994: dread f1 [0,4194304] 0 2024-04-12T05:37:38.580 INFO:tasks.workunit.client.1.smithi067.stdout:8/979: dread d6/d8/d50/dd4/f5e [0,4194304] 0 2024-04-12T05:37:38.583 INFO:tasks.workunit.client.1.smithi067.stdout:8/980: creat d6/def/d9a/f127 x:0 0 0 2024-04-12T05:37:38.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:38 smithi032 ceph-mon[39731]: pgmap v280: 65 pgs: 65 active+clean; 2.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 90 MiB/s rd, 214 MiB/s wr, 567 op/s 2024-04-12T05:37:38.591 INFO:tasks.workunit.client.1.smithi067.stdout:8/981: truncate d6/d8/d50/dd4/f33 79072 0 2024-04-12T05:37:38.593 INFO:tasks.workunit.client.1.smithi067.stdout:8/982: creat d6/df2/f128 x:0 0 0 2024-04-12T05:37:38.593 INFO:tasks.workunit.client.1.smithi067.stdout:8/983: chown d6/def/d9a/f127 157806 1 2024-04-12T05:37:38.596 INFO:tasks.workunit.client.1.smithi067.stdout:8/984: unlink d6/def/l8e 0 2024-04-12T05:37:38.597 INFO:tasks.workunit.client.1.smithi067.stdout:8/985: stat d6/d8/d50/dd4/d4e/d9e/c10e 0 2024-04-12T05:37:38.610 INFO:tasks.workunit.client.1.smithi067.stdout:8/986: dread d6/f95 [0,4194304] 0 2024-04-12T05:37:38.611 INFO:tasks.workunit.client.1.smithi067.stdout:8/987: chown d6/def/d9a/d12/c22 47848277 1 2024-04-12T05:37:38.613 INFO:tasks.workunit.client.1.smithi067.stdout:8/988: write d6/d8/d50/dd4/d4e/fee [9347938,111531] 0 2024-04-12T05:37:38.616 INFO:tasks.workunit.client.1.smithi067.stdout:8/989: link d6/d8/d50/dd4/cf3 d6/df2/c129 0 2024-04-12T05:37:38.617 INFO:tasks.workunit.client.1.smithi067.stdout:8/990: chown d6/d8/d50/dd4/f35 86 1 2024-04-12T05:37:38.619 INFO:tasks.workunit.client.1.smithi067.stdout:8/991: mknod d6/d86/c12a 0 2024-04-12T05:37:38.620 INFO:tasks.workunit.client.1.smithi067.stdout:8/992: truncate d6/d8/d8b/da1/f105 453209 0 2024-04-12T05:37:38.622 INFO:tasks.workunit.client.1.smithi067.stdout:8/993: truncate d6/d8/d8b/f8d 3054912 0 2024-04-12T05:37:38.626 INFO:tasks.workunit.client.1.smithi067.stdout:8/994: rename d6/d8/d50/dd4/d36/f68 to d6/def/d9a/f12b 0 2024-04-12T05:37:38.628 INFO:tasks.workunit.client.1.smithi067.stdout:8/995: dread d6/d8/d8b/da1/f105 [0,4194304] 0 2024-04-12T05:37:38.630 INFO:tasks.workunit.client.1.smithi067.stdout:8/996: mknod d6/d8/c12c 0 2024-04-12T05:37:38.632 INFO:tasks.workunit.client.1.smithi067.stdout:8/997: creat d6/def/d9a/d12/de9/f12d x:0 0 0 2024-04-12T05:37:38.651 INFO:tasks.workunit.client.1.smithi067.stdout:8/998: dwrite d6/d8/d50/dd4/fb4 [0,4194304] 0 2024-04-12T05:37:38.659 INFO:tasks.workunit.client.1.smithi067.stdout:8/999: dwrite d6/d8/d50/dd4/de7/d11b/d11e/f9d [0,4194304] 0 2024-04-12T05:37:38.792 INFO:tasks.workunit.client.1.smithi067.stdout:3/995: sync 2024-04-12T05:37:38.792 INFO:tasks.workunit.client.1.smithi067.stdout:3/996: chown dc/l12c 4185486 1 2024-04-12T05:37:38.792 INFO:tasks.workunit.client.1.smithi067.stdout:3/997: chown dc/d1d/d26/f164 18685261 1 2024-04-12T05:37:38.794 INFO:tasks.workunit.client.1.smithi067.stdout:3/998: fdatasync dc/d1d/d26/d52/d9a/f141 0 2024-04-12T05:37:38.795 INFO:tasks.workunit.client.1.smithi067.stdout:3/999: mknod dc/df/dca/d142/d113/c165 0 2024-04-12T05:37:38.800 INFO:tasks.workunit.client.1.smithi067.stderr:+ rm -rf -- ./tmp.JyyLSyE8rf 2024-04-12T05:37:40.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: pgmap v281: 65 pgs: 65 active+clean; 2.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 55 MiB/s rd, 122 MiB/s wr, 368 op/s 2024-04-12T05:37:40.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:37:40.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:37:40.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:37:40.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:37:40.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:37:40.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:37:40.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr fail", "who": "smithi032.hrwxrh"}]: dispatch 2024-04-12T05:37:40.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: osdmap e47: 6 total, 6 up, 6 in 2024-04-12T05:37:40.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "mgr fail", "who": "smithi032.hrwxrh"}]': finished 2024-04-12T05:37:40.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: mgrmap e22: smithi067.dwichg(active, starting, since 0.00653367s) 2024-04-12T05:37:40.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-12T05:37:40.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-12T05:37:40.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.pndtry"}]: dispatch 2024-04-12T05:37:40.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.jehaaj"}]: dispatch 2024-04-12T05:37:40.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.zesmgy"}]: dispatch 2024-04-12T05:37:40.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.ymvqih"}]: dispatch 2024-04-12T05:37:40.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "mgr metadata", "who": "smithi067.dwichg", "id": "smithi067.dwichg"}]: dispatch 2024-04-12T05:37:40.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:37:40.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:37:40.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:37:40.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:37:40.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:37:40.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:37:40.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T05:37:40.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T05:37:40.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T05:37:40.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: Manager daemon smithi067.dwichg is now available 2024-04-12T05:37:40.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:37:40.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:37:40.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.dwichg/mirror_snapshot_schedule"}]: dispatch 2024-04-12T05:37:40.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.dwichg/mirror_snapshot_schedule"}]: dispatch 2024-04-12T05:37:40.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.dwichg/trash_purge_schedule"}]: dispatch 2024-04-12T05:37:40.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:40 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.dwichg/trash_purge_schedule"}]: dispatch 2024-04-12T05:37:40.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: pgmap v281: 65 pgs: 65 active+clean; 2.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 55 MiB/s rd, 122 MiB/s wr, 368 op/s 2024-04-12T05:37:40.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:37:40.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:37:40.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:37:40.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:37:40.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:37:40.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:37:40.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr fail", "who": "smithi032.hrwxrh"}]: dispatch 2024-04-12T05:37:40.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: osdmap e47: 6 total, 6 up, 6 in 2024-04-12T05:37:40.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: from='mgr.14208 172.21.15.32:0/3368729405' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "mgr fail", "who": "smithi032.hrwxrh"}]': finished 2024-04-12T05:37:40.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: mgrmap e22: smithi067.dwichg(active, starting, since 0.00653367s) 2024-04-12T05:37:40.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-12T05:37:40.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-12T05:37:40.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.pndtry"}]: dispatch 2024-04-12T05:37:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.jehaaj"}]: dispatch 2024-04-12T05:37:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.zesmgy"}]: dispatch 2024-04-12T05:37:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.ymvqih"}]: dispatch 2024-04-12T05:37:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "mgr metadata", "who": "smithi067.dwichg", "id": "smithi067.dwichg"}]: dispatch 2024-04-12T05:37:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:37:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:37:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:37:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:37:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:37:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:37:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T05:37:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T05:37:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T05:37:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: Manager daemon smithi067.dwichg is now available 2024-04-12T05:37:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:37:40.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:37:40.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.dwichg/mirror_snapshot_schedule"}]: dispatch 2024-04-12T05:37:40.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.dwichg/mirror_snapshot_schedule"}]: dispatch 2024-04-12T05:37:40.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.dwichg/trash_purge_schedule"}]: dispatch 2024-04-12T05:37:40.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:40 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.dwichg/trash_purge_schedule"}]: dispatch 2024-04-12T05:37:41.597 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:41 smithi032 ceph-mon[39731]: Deploying cephadm binary to smithi032 2024-04-12T05:37:41.598 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:41 smithi032 ceph-mon[39731]: Deploying cephadm binary to smithi067 2024-04-12T05:37:41.598 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:41 smithi032 ceph-mon[39731]: mgrmap e23: smithi067.dwichg(active, since 1.00996s) 2024-04-12T05:37:41.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.740+0000 7f03a57c4700 1 -- 172.21.15.32:0/2519944472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03a00fb390 msgr2=0x7f03a0102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:41.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.740+0000 7f03a57c4700 1 --2- 172.21.15.32:0/2519944472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03a00fb390 0x7f03a0102610 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0398009ce0 tx=0x7f0398004d10 comp rx=0 tx=0).stop 2024-04-12T05:37:41.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.741+0000 7f03a57c4700 1 -- 172.21.15.32:0/2519944472 shutdown_connections 2024-04-12T05:37:41.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.741+0000 7f03a57c4700 1 --2- 172.21.15.32:0/2519944472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03a00fb390 0x7f03a0102610 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:41.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.741+0000 7f03a57c4700 1 --2- 172.21.15.32:0/2519944472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03a00faa40 0x7f03a00fae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:41.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.741+0000 7f03a57c4700 1 -- 172.21.15.32:0/2519944472 >> 172.21.15.32:0/2519944472 conn(0x7f03a00f6770 msgr2=0x7f03a00f8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:41.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.741+0000 7f03a57c4700 1 -- 172.21.15.32:0/2519944472 shutdown_connections 2024-04-12T05:37:41.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.741+0000 7f03a57c4700 1 -- 172.21.15.32:0/2519944472 wait complete. 2024-04-12T05:37:41.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.742+0000 7f03a57c4700 1 Processor -- start 2024-04-12T05:37:41.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.742+0000 7f03a57c4700 1 -- start start 2024-04-12T05:37:41.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.742+0000 7f03a57c4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03a00faa40 0x7f03a006e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:37:41.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.742+0000 7f03a57c4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03a00fb390 0x7f03a006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:37:41.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.742+0000 7f03a57c4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a006ff50 con 0x7f03a00faa40 2024-04-12T05:37:41.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.742+0000 7f03a57c4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a006ef30 con 0x7f03a00fb390 2024-04-12T05:37:41.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.743+0000 7f039e7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03a00fb390 0x7f03a006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:41.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.743+0000 7f039e7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03a00fb390 0x7f03a006e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.32:41574/0 (socket says 172.21.15.32:41574) 2024-04-12T05:37:41.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.743+0000 7f039e7fc700 1 -- 172.21.15.32:0/2411838168 learned_addr learned my addr 172.21.15.32:0/2411838168 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:37:41.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.743+0000 7f039e7fc700 1 -- 172.21.15.32:0/2411838168 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03a00faa40 msgr2=0x7f03a006e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:41.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.743+0000 7f039e7fc700 1 --2- 172.21.15.32:0/2411838168 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03a00faa40 0x7f03a006e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:41.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.743+0000 7f039e7fc700 1 -- 172.21.15.32:0/2411838168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0398009940 con 0x7f03a00fb390 2024-04-12T05:37:41.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.743+0000 7f039e7fc700 1 --2- 172.21.15.32:0/2411838168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03a00fb390 0x7f03a006e9f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0398004e80 tx=0x7f0398004d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:41.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.744+0000 7f03877fe700 1 -- 172.21.15.32:0/2411838168 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f039800e050 con 0x7f03a00fb390 2024-04-12T05:37:41.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.744+0000 7f03877fe700 1 -- 172.21.15.32:0/2411838168 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f039800bec0 con 0x7f03a00fb390 2024-04-12T05:37:41.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.744+0000 7f03877fe700 1 -- 172.21.15.32:0/2411838168 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0398012900 con 0x7f03a00fb390 2024-04-12T05:37:41.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.744+0000 7f03a57c4700 1 -- 172.21.15.32:0/2411838168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03a006f1b0 con 0x7f03a00fb390 2024-04-12T05:37:41.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.744+0000 7f03a57c4700 1 -- 172.21.15.32:0/2411838168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03a0108bb0 con 0x7f03a00fb390 2024-04-12T05:37:41.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.745+0000 7f03a57c4700 1 -- 172.21.15.32:0/2411838168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03a004c710 con 0x7f03a00fb390 2024-04-12T05:37:41.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.745+0000 7f03877fe700 1 -- 172.21.15.32:0/2411838168 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 24) v1 ==== 45624+0+0 (secure 0 0 0) 0x7f0398019040 con 0x7f03a00fb390 2024-04-12T05:37:41.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.746+0000 7f03877fe700 1 --2- 172.21.15.32:0/2411838168 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f0388038610 0x7f038803aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:37:41.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.746+0000 7f03877fe700 1 -- 172.21.15.32:0/2411838168 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f039804dbe0 con 0x7f03a00fb390 2024-04-12T05:37:41.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.746+0000 7f039effd700 1 --2- 172.21.15.32:0/2411838168 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f0388038610 0x7f038803aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:41.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.746+0000 7f039effd700 1 --2- 172.21.15.32:0/2411838168 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f0388038610 0x7f038803aac0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0394006fd0 tx=0x7f0394006db0 comp rx=0 tx=0).ready entity=mgr.24399 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:41.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:41.751+0000 7f03877fe700 1 -- 172.21.15.32:0/2411838168 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f039801d620 con 0x7f03a00fb390 2024-04-12T05:37:41.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:41 smithi067 ceph-mon[49010]: Deploying cephadm binary to smithi032 2024-04-12T05:37:41.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:41 smithi067 ceph-mon[49010]: Deploying cephadm binary to smithi067 2024-04-12T05:37:41.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:41 smithi067 ceph-mon[49010]: mgrmap e23: smithi067.dwichg(active, since 1.00996s) 2024-04-12T05:37:42.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.023+0000 7f03a57c4700 1 -- 172.21.15.32:0/2411838168 --> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f03a0101f10 con 0x7f0388038610 2024-04-12T05:37:42.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.029+0000 7f03877fe700 1 -- 172.21.15.32:0/2411838168 <== mgr.24399 v2:172.21.15.67:6828/3925433122 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f03a0101f10 con 0x7f0388038610 2024-04-12T05:37:42.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.031+0000 7f03a57c4700 1 -- 172.21.15.32:0/2411838168 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f0388038610 msgr2=0x7f038803aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:42.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.031+0000 7f03a57c4700 1 --2- 172.21.15.32:0/2411838168 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f0388038610 0x7f038803aac0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0394006fd0 tx=0x7f0394006db0 comp rx=0 tx=0).stop 2024-04-12T05:37:42.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.031+0000 7f03a57c4700 1 -- 172.21.15.32:0/2411838168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03a00fb390 msgr2=0x7f03a006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:42.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.031+0000 7f03a57c4700 1 --2- 172.21.15.32:0/2411838168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03a00fb390 0x7f03a006e9f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0398004e80 tx=0x7f0398004d10 comp rx=0 tx=0).stop 2024-04-12T05:37:42.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.031+0000 7f03a57c4700 1 -- 172.21.15.32:0/2411838168 shutdown_connections 2024-04-12T05:37:42.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.031+0000 7f03a57c4700 1 --2- 172.21.15.32:0/2411838168 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f0388038610 0x7f038803aac0 secure :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0394006fd0 tx=0x7f0394006db0 comp rx=0 tx=0).stop 2024-04-12T05:37:42.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.031+0000 7f03a57c4700 1 --2- 172.21.15.32:0/2411838168 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03a00faa40 0x7f03a006e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:42.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.031+0000 7f03a57c4700 1 --2- 172.21.15.32:0/2411838168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03a00fb390 0x7f03a006e9f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:42.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.031+0000 7f03a57c4700 1 -- 172.21.15.32:0/2411838168 >> 172.21.15.32:0/2411838168 conn(0x7f03a00f6770 msgr2=0x7f03a01007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:42.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.031+0000 7f03a57c4700 1 -- 172.21.15.32:0/2411838168 shutdown_connections 2024-04-12T05:37:42.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.031+0000 7f03a57c4700 1 -- 172.21.15.32:0/2411838168 wait complete. 2024-04-12T05:37:42.047 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:37:42.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.190+0000 7eff6895c700 1 -- 172.21.15.32:0/3754815460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff640fcf90 msgr2=0x7eff640fd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:42.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.190+0000 7eff6895c700 1 --2- 172.21.15.32:0/3754815460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff640fcf90 0x7eff640fd400 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7eff580049c0 tx=0x7eff58009e30 comp rx=0 tx=0).stop 2024-04-12T05:37:42.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.190+0000 7eff6895c700 1 -- 172.21.15.32:0/3754815460 shutdown_connections 2024-04-12T05:37:42.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.190+0000 7eff6895c700 1 --2- 172.21.15.32:0/3754815460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff640fcf90 0x7eff640fd400 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:42.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.190+0000 7eff6895c700 1 --2- 172.21.15.32:0/3754815460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff640fbce0 0x7eff640fc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:42.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.190+0000 7eff6895c700 1 -- 172.21.15.32:0/3754815460 >> 172.21.15.32:0/3754815460 conn(0x7eff640f7440 msgr2=0x7eff640f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:42.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.190+0000 7eff6895c700 1 -- 172.21.15.32:0/3754815460 shutdown_connections 2024-04-12T05:37:42.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.190+0000 7eff6895c700 1 -- 172.21.15.32:0/3754815460 wait complete. 2024-04-12T05:37:42.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.191+0000 7eff6895c700 1 Processor -- start 2024-04-12T05:37:42.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.191+0000 7eff6895c700 1 -- start start 2024-04-12T05:37:42.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.191+0000 7eff6895c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff640fbce0 0x7eff6410b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:37:42.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.191+0000 7eff6895c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff640fcf90 0x7eff6410bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:37:42.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.191+0000 7eff6895c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6410c210 con 0x7eff640fcf90 2024-04-12T05:37:42.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.191+0000 7eff6895c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6410ca60 con 0x7eff640fbce0 2024-04-12T05:37:42.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.191+0000 7eff61d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff640fcf90 0x7eff6410bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:42.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.191+0000 7eff61d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff640fcf90 0x7eff6410bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60138/0 (socket says 172.21.15.32:60138) 2024-04-12T05:37:42.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.191+0000 7eff61d9b700 1 -- 172.21.15.32:0/2987899913 learned_addr learned my addr 172.21.15.32:0/2987899913 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:37:42.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.191+0000 7eff6259c700 1 --2- 172.21.15.32:0/2987899913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff640fbce0 0x7eff6410b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:42.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.192+0000 7eff61d9b700 1 -- 172.21.15.32:0/2987899913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff640fbce0 msgr2=0x7eff6410b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:42.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.192+0000 7eff61d9b700 1 --2- 172.21.15.32:0/2987899913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff640fbce0 0x7eff6410b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:42.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.192+0000 7eff61d9b700 1 -- 172.21.15.32:0/2987899913 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff58005040 con 0x7eff640fcf90 2024-04-12T05:37:42.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.192+0000 7eff61d9b700 1 --2- 172.21.15.32:0/2987899913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff640fcf90 0x7eff6410bbc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7eff58009fd0 tx=0x7eff58009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:42.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.192+0000 7eff52ffd700 1 -- 172.21.15.32:0/2987899913 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff58011480 con 0x7eff640fcf90 2024-04-12T05:37:42.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.192+0000 7eff52ffd700 1 -- 172.21.15.32:0/2987899913 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff58011a90 con 0x7eff640fcf90 2024-04-12T05:37:42.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.192+0000 7eff6895c700 1 -- 172.21.15.32:0/2987899913 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff6410cce0 con 0x7eff640fcf90 2024-04-12T05:37:42.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.192+0000 7eff52ffd700 1 -- 172.21.15.32:0/2987899913 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff5801a770 con 0x7eff640fcf90 2024-04-12T05:37:42.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.192+0000 7eff6895c700 1 -- 172.21.15.32:0/2987899913 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff6410d2a0 con 0x7eff640fcf90 2024-04-12T05:37:42.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.193+0000 7eff6895c700 1 -- 172.21.15.32:0/2987899913 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff6405f5f0 con 0x7eff640fcf90 2024-04-12T05:37:42.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.194+0000 7eff52ffd700 1 -- 172.21.15.32:0/2987899913 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 24) v1 ==== 45624+0+0 (secure 0 0 0) 0x7eff58011c00 con 0x7eff640fcf90 2024-04-12T05:37:42.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.194+0000 7eff52ffd700 1 --2- 172.21.15.32:0/2987899913 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7eff4c0385c0 0x7eff4c03aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:37:42.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.194+0000 7eff52ffd700 1 -- 172.21.15.32:0/2987899913 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7eff5804db50 con 0x7eff640fcf90 2024-04-12T05:37:42.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.194+0000 7eff6259c700 1 --2- 172.21.15.32:0/2987899913 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7eff4c0385c0 0x7eff4c03aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:42.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.195+0000 7eff6259c700 1 --2- 172.21.15.32:0/2987899913 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7eff4c0385c0 0x7eff4c03aa70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7eff6410b410 tx=0x7eff5400b040 comp rx=0 tx=0).ready entity=mgr.24399 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:42.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.199+0000 7eff52ffd700 1 -- 172.21.15.32:0/2987899913 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7eff58023970 con 0x7eff640fcf90 2024-04-12T05:37:42.322 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:42 smithi032 ceph-mon[39731]: pgmap v3: 65 pgs: 65 active+clean; 2.6 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail 2024-04-12T05:37:42.322 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:42 smithi032 ceph-mon[39731]: mgrmap e24: smithi067.dwichg(active, since 2s) 2024-04-12T05:37:42.322 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:42 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:42.322 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:42 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:42.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.522+0000 7eff6895c700 1 -- 172.21.15.32:0/2987899913 --> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7eff6410d4d0 con 0x7eff4c0385c0 2024-04-12T05:37:42.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.536+0000 7eff52ffd700 1 -- 172.21.15.32:0/2987899913 <== mgr.24399 v2:172.21.15.67:6828/3925433122 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7eff6410d4d0 con 0x7eff4c0385c0 2024-04-12T05:37:42.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.543+0000 7eff6895c700 1 -- 172.21.15.32:0/2987899913 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7eff4c0385c0 msgr2=0x7eff4c03aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:42.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.543+0000 7eff6895c700 1 --2- 172.21.15.32:0/2987899913 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7eff4c0385c0 0x7eff4c03aa70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7eff6410b410 tx=0x7eff5400b040 comp rx=0 tx=0).stop 2024-04-12T05:37:42.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.543+0000 7eff6895c700 1 -- 172.21.15.32:0/2987899913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff640fcf90 msgr2=0x7eff6410bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:42.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.543+0000 7eff6895c700 1 --2- 172.21.15.32:0/2987899913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff640fcf90 0x7eff6410bbc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7eff58009fd0 tx=0x7eff58009e30 comp rx=0 tx=0).stop 2024-04-12T05:37:42.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.545+0000 7eff6895c700 1 -- 172.21.15.32:0/2987899913 shutdown_connections 2024-04-12T05:37:42.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.545+0000 7eff6895c700 1 --2- 172.21.15.32:0/2987899913 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7eff4c0385c0 0x7eff4c03aa70 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:42.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.545+0000 7eff6895c700 1 --2- 172.21.15.32:0/2987899913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff640fbce0 0x7eff6410b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:42.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.545+0000 7eff6895c700 1 --2- 172.21.15.32:0/2987899913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff640fcf90 0x7eff6410bbc0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:42.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.545+0000 7eff6895c700 1 -- 172.21.15.32:0/2987899913 >> 172.21.15.32:0/2987899913 conn(0x7eff640f7440 msgr2=0x7eff640f8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:42.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.549+0000 7eff6895c700 1 -- 172.21.15.32:0/2987899913 shutdown_connections 2024-04-12T05:37:42.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.551+0000 7eff6895c700 1 -- 172.21.15.32:0/2987899913 wait complete. 2024-04-12T05:37:42.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.728+0000 7f64cf8bd700 1 -- 172.21.15.32:0/4293967114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64c80708f0 msgr2=0x7f64c8070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:42.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.728+0000 7f64cf8bd700 1 --2- 172.21.15.32:0/4293967114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64c80708f0 0x7f64c8070d60 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f64b8004940 tx=0x7f64b8009e30 comp rx=0 tx=0).stop 2024-04-12T05:37:42.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.728+0000 7f64cf8bd700 1 -- 172.21.15.32:0/4293967114 shutdown_connections 2024-04-12T05:37:42.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.728+0000 7f64cf8bd700 1 --2- 172.21.15.32:0/4293967114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64c80708f0 0x7f64c8070d60 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:42.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.728+0000 7f64cf8bd700 1 --2- 172.21.15.32:0/4293967114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64c8071ea0 0x7f64c80703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:42.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.728+0000 7f64cf8bd700 1 -- 172.21.15.32:0/4293967114 >> 172.21.15.32:0/4293967114 conn(0x7f64c80f8580 msgr2=0x7f64c80fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:42.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.729+0000 7f64cf8bd700 1 -- 172.21.15.32:0/4293967114 shutdown_connections 2024-04-12T05:37:42.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.729+0000 7f64cf8bd700 1 -- 172.21.15.32:0/4293967114 wait complete. 2024-04-12T05:37:42.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.730+0000 7f64cf8bd700 1 Processor -- start 2024-04-12T05:37:42.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.730+0000 7f64cf8bd700 1 -- start start 2024-04-12T05:37:42.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.730+0000 7f64cf8bd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64c80708f0 0x7f64c8107780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:37:42.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.730+0000 7f64cf8bd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64c8071ea0 0x7f64c8107cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:37:42.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.730+0000 7f64cf8bd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c8109190 con 0x7f64c80708f0 2024-04-12T05:37:42.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.730+0000 7f64cf8bd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c8109300 con 0x7f64c8071ea0 2024-04-12T05:37:42.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.731+0000 7f64cd659700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64c80708f0 0x7f64c8107780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:42.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.731+0000 7f64cd659700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64c80708f0 0x7f64c8107780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60154/0 (socket says 172.21.15.32:60154) 2024-04-12T05:37:42.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.731+0000 7f64cd659700 1 -- 172.21.15.32:0/3487522885 learned_addr learned my addr 172.21.15.32:0/3487522885 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:37:42.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.731+0000 7f64cd659700 1 -- 172.21.15.32:0/3487522885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64c8071ea0 msgr2=0x7f64c8107cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:42.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.731+0000 7f64cd659700 1 --2- 172.21.15.32:0/3487522885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64c8071ea0 0x7f64c8107cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:42.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.731+0000 7f64cd659700 1 -- 172.21.15.32:0/3487522885 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c4005040 con 0x7f64c80708f0 2024-04-12T05:37:42.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.731+0000 7f64cd659700 1 --2- 172.21.15.32:0/3487522885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64c80708f0 0x7f64c8107780 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f64c40096d0 tx=0x7f64c4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:42.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.731+0000 7f64bdffb700 1 -- 172.21.15.32:0/3487522885 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64c400e7e0 con 0x7f64c80708f0 2024-04-12T05:37:42.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.732+0000 7f64cf8bd700 1 -- 172.21.15.32:0/3487522885 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b8005040 con 0x7f64c80708f0 2024-04-12T05:37:42.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.732+0000 7f64cf8bd700 1 -- 172.21.15.32:0/3487522885 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64c80679e0 con 0x7f64c80708f0 2024-04-12T05:37:42.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.732+0000 7f64bdffb700 1 -- 172.21.15.32:0/3487522885 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f64c400f040 con 0x7f64c80708f0 2024-04-12T05:37:42.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.732+0000 7f64bdffb700 1 -- 172.21.15.32:0/3487522885 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64c4013d00 con 0x7f64c80708f0 2024-04-12T05:37:42.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.733+0000 7f64cf8bd700 1 -- 172.21.15.32:0/3487522885 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64ac0052e0 con 0x7f64c80708f0 2024-04-12T05:37:42.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.733+0000 7f64bdffb700 1 -- 172.21.15.32:0/3487522885 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 24) v1 ==== 45624+0+0 (secure 0 0 0) 0x7f64c4013500 con 0x7f64c80708f0 2024-04-12T05:37:42.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.734+0000 7f64bdffb700 1 --2- 172.21.15.32:0/3487522885 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f64b40385c0 0x7f64b403aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:37:42.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.734+0000 7f64cce58700 1 --2- 172.21.15.32:0/3487522885 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f64b40385c0 0x7f64b403aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:42.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.735+0000 7f64cce58700 1 --2- 172.21.15.32:0/3487522885 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f64b40385c0 0x7f64b403aa70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f64b8009fd0 tx=0x7f64b8009e30 comp rx=0 tx=0).ready entity=mgr.24399 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:42.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.735+0000 7f64bdffb700 1 -- 172.21.15.32:0/3487522885 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f64c401be60 con 0x7f64c80708f0 2024-04-12T05:37:42.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:42.744+0000 7f64bdffb700 1 -- 172.21.15.32:0/3487522885 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f64c401d460 con 0x7f64c80708f0 2024-04-12T05:37:42.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:42 smithi067 ceph-mon[49010]: pgmap v3: 65 pgs: 65 active+clean; 2.6 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail 2024-04-12T05:37:42.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:42 smithi067 ceph-mon[49010]: mgrmap e24: smithi067.dwichg(active, since 2s) 2024-04-12T05:37:42.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:42 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:42.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:42 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:43.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.075+0000 7f64cf8bd700 1 -- 172.21.15.32:0/3487522885 --> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f64ac000bf0 con 0x7f64b40385c0 2024-04-12T05:37:43.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.088+0000 7f64bdffb700 1 -- 172.21.15.32:0/3487522885 <== mgr.24399 v2:172.21.15.67:6828/3925433122 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3388 (secure 0 0 0) 0x7f64ac000bf0 con 0x7f64b40385c0 2024-04-12T05:37:43.087 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:37:43.087 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (7m) 2m ago 8m 27.0M - 0.25.0 c8568f914cd2 a5c616791a7e 2024-04-12T05:37:43.087 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (9m) 2m ago 9m 6656k - 17.2.7-827-gd3b5060a 21de9b7369ad a9cfc00c311b 2024-04-12T05:37:43.087 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (8m) 25s ago 8m 6651k - 17.2.7-827-gd3b5060a 21de9b7369ad 1f2eea4542d9 2024-04-12T05:37:43.087 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (7m) 2m ago 8m 87.0M - 9.4.7 954c08fa6188 8e5a7ebd77b5 2024-04-12T05:37:43.087 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (3m) 2m ago 3m 11.4M - 17.2.7-827-gd3b5060a 21de9b7369ad e7ada8e5259c 2024-04-12T05:37:43.087 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (3m) 2m ago 3m 14.0M - 17.2.7-827-gd3b5060a 21de9b7369ad 55124da9afdb 2024-04-12T05:37:43.087 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (3m) 25s ago 3m 89.3M - 17.2.7-827-gd3b5060a 21de9b7369ad 0c295d31a4a6 2024-04-12T05:37:43.087 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (3m) 25s ago 3m 148M - 17.2.7-827-gd3b5060a 21de9b7369ad 6eecd7deaac2 2024-04-12T05:37:43.087 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:9283 running (10m) 2m ago 10m 483M - 17.2.7-827-gd3b5060a 21de9b7369ad fdf859c0313c 2024-04-12T05:37:43.087 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283 running (58s) 25s ago 7m 73.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 b190c0307364 2024-04-12T05:37:43.087 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (10m) 2m ago 10m 42.4M 2048M 17.2.7-827-gd3b5060a 21de9b7369ad 75aad95b984d 2024-04-12T05:37:43.087 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (7m) 25s ago 7m 34.0M 2048M 17.2.7-827-gd3b5060a 21de9b7369ad 157be8a8c6d8 2024-04-12T05:37:43.087 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (9m) 2m ago 9m 15.8M - 1.5.0 0da6a335fe13 ca3585dd7b65 2024-04-12T05:37:43.088 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (8m) 25s ago 8m 15.6M - 1.5.0 0da6a335fe13 c324d9436cca 2024-04-12T05:37:43.088 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (7m) 2m ago 7m 46.9M 4300M 17.2.7-827-gd3b5060a 21de9b7369ad 97446e42ceba 2024-04-12T05:37:43.088 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (6m) 2m ago 6m 47.6M 4300M 17.2.7-827-gd3b5060a 21de9b7369ad bf9f9a2db2d5 2024-04-12T05:37:43.088 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (6m) 2m ago 6m 44.1M 4300M 17.2.7-827-gd3b5060a 21de9b7369ad f055cf1ca665 2024-04-12T05:37:43.088 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (5m) 25s ago 5m 220M 5323M 17.2.7-827-gd3b5060a 21de9b7369ad 61b851235351 2024-04-12T05:37:43.088 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (5m) 25s ago 5m 195M 5323M 17.2.7-827-gd3b5060a 21de9b7369ad 2c84d9047e9c 2024-04-12T05:37:43.088 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (4m) 25s ago 4m 219M 5323M 17.2.7-827-gd3b5060a 21de9b7369ad e54839bb6055 2024-04-12T05:37:43.088 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (7m) 2m ago 8m 56.5M - 2.43.0 a07b618ecd1d 996307aec3d2 2024-04-12T05:37:43.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.092+0000 7f64cf8bd700 1 -- 172.21.15.32:0/3487522885 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f64b40385c0 msgr2=0x7f64b403aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:43.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.092+0000 7f64cf8bd700 1 --2- 172.21.15.32:0/3487522885 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f64b40385c0 0x7f64b403aa70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f64b8009fd0 tx=0x7f64b8009e30 comp rx=0 tx=0).stop 2024-04-12T05:37:43.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.092+0000 7f64cf8bd700 1 -- 172.21.15.32:0/3487522885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64c80708f0 msgr2=0x7f64c8107780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:43.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.092+0000 7f64cf8bd700 1 --2- 172.21.15.32:0/3487522885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64c80708f0 0x7f64c8107780 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f64c40096d0 tx=0x7f64c4009e30 comp rx=0 tx=0).stop 2024-04-12T05:37:43.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.093+0000 7f64cf8bd700 1 -- 172.21.15.32:0/3487522885 shutdown_connections 2024-04-12T05:37:43.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.093+0000 7f64cf8bd700 1 --2- 172.21.15.32:0/3487522885 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f64b40385c0 0x7f64b403aa70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:43.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.093+0000 7f64cf8bd700 1 --2- 172.21.15.32:0/3487522885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64c80708f0 0x7f64c8107780 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:43.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.093+0000 7f64cf8bd700 1 --2- 172.21.15.32:0/3487522885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64c8071ea0 0x7f64c8107cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:43.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.093+0000 7f64cf8bd700 1 -- 172.21.15.32:0/3487522885 >> 172.21.15.32:0/3487522885 conn(0x7f64c80f8580 msgr2=0x7f64c80fa0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:43.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.094+0000 7f64cf8bd700 1 -- 172.21.15.32:0/3487522885 shutdown_connections 2024-04-12T05:37:43.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.094+0000 7f64cf8bd700 1 -- 172.21.15.32:0/3487522885 wait complete. 2024-04-12T05:37:43.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.233+0000 7ffa5c8cb700 1 -- 172.21.15.32:0/2455169663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa581023b0 msgr2=0x7ffa58104790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:43.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.233+0000 7ffa5c8cb700 1 --2- 172.21.15.32:0/2455169663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa581023b0 0x7ffa58104790 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c004970 tx=0x7ffa4c009e30 comp rx=0 tx=0).stop 2024-04-12T05:37:43.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.233+0000 7ffa5c8cb700 1 -- 172.21.15.32:0/2455169663 shutdown_connections 2024-04-12T05:37:43.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.233+0000 7ffa5c8cb700 1 --2- 172.21.15.32:0/2455169663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa581023b0 0x7ffa58104790 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:43.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.233+0000 7ffa5c8cb700 1 --2- 172.21.15.32:0/2455169663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa58066ef0 0x7ffa58101e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:43.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.233+0000 7ffa5c8cb700 1 -- 172.21.15.32:0/2455169663 >> 172.21.15.32:0/2455169663 conn(0x7ffa580f7480 msgr2=0x7ffa580f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:43.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.233+0000 7ffa5c8cb700 1 -- 172.21.15.32:0/2455169663 shutdown_connections 2024-04-12T05:37:43.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.233+0000 7ffa5c8cb700 1 -- 172.21.15.32:0/2455169663 wait complete. 2024-04-12T05:37:43.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.234+0000 7ffa5c8cb700 1 Processor -- start 2024-04-12T05:37:43.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.234+0000 7ffa5c8cb700 1 -- start start 2024-04-12T05:37:43.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.234+0000 7ffa5c8cb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa58066ef0 0x7ffa5810f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:37:43.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.234+0000 7ffa5c8cb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa581023b0 0x7ffa5810fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:37:43.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.234+0000 7ffa5c8cb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa581104d0 con 0x7ffa581023b0 2024-04-12T05:37:43.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.234+0000 7ffa5c8cb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa58110640 con 0x7ffa58066ef0 2024-04-12T05:37:43.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.234+0000 7ffa5659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa58066ef0 0x7ffa5810f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:43.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.234+0000 7ffa55d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa581023b0 0x7ffa5810fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:43.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.234+0000 7ffa5659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa58066ef0 0x7ffa5810f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.32:41620/0 (socket says 172.21.15.32:41620) 2024-04-12T05:37:43.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.234+0000 7ffa55d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa581023b0 0x7ffa5810fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60182/0 (socket says 172.21.15.32:60182) 2024-04-12T05:37:43.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.234+0000 7ffa5659c700 1 -- 172.21.15.32:0/3599422344 learned_addr learned my addr 172.21.15.32:0/3599422344 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:37:43.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.234+0000 7ffa55d9b700 1 -- 172.21.15.32:0/3599422344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa58066ef0 msgr2=0x7ffa5810f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:43.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.234+0000 7ffa55d9b700 1 --2- 172.21.15.32:0/3599422344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa58066ef0 0x7ffa5810f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:43.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.234+0000 7ffa55d9b700 1 -- 172.21.15.32:0/3599422344 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa4c005040 con 0x7ffa581023b0 2024-04-12T05:37:43.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.234+0000 7ffa5659c700 1 --2- 172.21.15.32:0/3599422344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa58066ef0 0x7ffa5810f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T05:37:43.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.235+0000 7ffa55d9b700 1 --2- 172.21.15.32:0/3599422344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa581023b0 0x7ffa5810fed0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c00b370 tx=0x7ffa4c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:43.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.235+0000 7ffa46ffd700 1 -- 172.21.15.32:0/3599422344 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa4c00e040 con 0x7ffa581023b0 2024-04-12T05:37:43.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.235+0000 7ffa5c8cb700 1 -- 172.21.15.32:0/3599422344 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa58115040 con 0x7ffa581023b0 2024-04-12T05:37:43.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.235+0000 7ffa46ffd700 1 -- 172.21.15.32:0/3599422344 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffa4c004290 con 0x7ffa581023b0 2024-04-12T05:37:43.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.235+0000 7ffa46ffd700 1 -- 172.21.15.32:0/3599422344 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa4c013420 con 0x7ffa581023b0 2024-04-12T05:37:43.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.236+0000 7ffa5c8cb700 1 -- 172.21.15.32:0/3599422344 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa58115600 con 0x7ffa581023b0 2024-04-12T05:37:43.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.236+0000 7ffa46ffd700 1 -- 172.21.15.32:0/3599422344 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 24) v1 ==== 45624+0+0 (secure 0 0 0) 0x7ffa4c0135a0 con 0x7ffa581023b0 2024-04-12T05:37:43.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.236+0000 7ffa46ffd700 1 --2- 172.21.15.32:0/3599422344 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7ffa400385f0 0x7ffa4003aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:37:43.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.237+0000 7ffa5659c700 1 --2- 172.21.15.32:0/3599422344 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7ffa400385f0 0x7ffa4003aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:43.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.237+0000 7ffa5659c700 1 --2- 172.21.15.32:0/3599422344 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7ffa400385f0 0x7ffa4003aaa0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ffa4800f610 tx=0x7ffa4800b040 comp rx=0 tx=0).ready entity=mgr.24399 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:43.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.237+0000 7ffa46ffd700 1 -- 172.21.15.32:0/3599422344 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7ffa4c0123c0 con 0x7ffa581023b0 2024-04-12T05:37:43.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.237+0000 7ffa5c8cb700 1 -- 172.21.15.32:0/3599422344 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa5804c710 con 0x7ffa581023b0 2024-04-12T05:37:43.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.244+0000 7ffa46ffd700 1 -- 172.21.15.32:0/3599422344 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7ffa4c012770 con 0x7ffa581023b0 2024-04-12T05:37:43.323 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:43 smithi032 ceph-mon[39731]: pgmap v4: 65 pgs: 65 active+clean; 2.6 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail 2024-04-12T05:37:43.323 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:43 smithi032 ceph-mon[39731]: from='client.24431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:43.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.502+0000 7ffa46ffd700 1 -- 172.21.15.32:0/3599422344 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mgrmap(e 25) v1 ==== 45680+0+0 (secure 0 0 0) 0x7ffa4c009660 con 0x7ffa581023b0 2024-04-12T05:37:43.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.664+0000 7ffa5c8cb700 1 -- 172.21.15.32:0/3599422344 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ffa58107900 con 0x7ffa581023b0 2024-04-12T05:37:43.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.664+0000 7ffa46ffd700 1 -- 172.21.15.32:0/3599422344 <== mon.0 v2:172.21.15.32:3300/0 8 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7ffa4c048070 con 0x7ffa581023b0 2024-04-12T05:37:43.664 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:37:43.664 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:37:43.664 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 2 2024-04-12T05:37:43.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:37:43.664 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:37:43.664 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 1 2024-04-12T05:37:43.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:37:43.664 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:37:43.664 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 6 2024-04-12T05:37:43.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:37:43.664 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:37:43.664 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 4 2024-04-12T05:37:43.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:37:43.665 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:37:43.665 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 12, 2024-04-12T05:37:43.665 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 1 2024-04-12T05:37:43.665 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:37:43.665 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:37:43.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.668+0000 7ffa5c8cb700 1 -- 172.21.15.32:0/3599422344 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7ffa400385f0 msgr2=0x7ffa4003aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:43.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.668+0000 7ffa5c8cb700 1 --2- 172.21.15.32:0/3599422344 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7ffa400385f0 0x7ffa4003aaa0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ffa4800f610 tx=0x7ffa4800b040 comp rx=0 tx=0).stop 2024-04-12T05:37:43.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.668+0000 7ffa5c8cb700 1 -- 172.21.15.32:0/3599422344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa581023b0 msgr2=0x7ffa5810fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:43.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.668+0000 7ffa5c8cb700 1 --2- 172.21.15.32:0/3599422344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa581023b0 0x7ffa5810fed0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c00b370 tx=0x7ffa4c009e30 comp rx=0 tx=0).stop 2024-04-12T05:37:43.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.668+0000 7ffa5c8cb700 1 -- 172.21.15.32:0/3599422344 shutdown_connections 2024-04-12T05:37:43.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.668+0000 7ffa5c8cb700 1 --2- 172.21.15.32:0/3599422344 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7ffa400385f0 0x7ffa4003aaa0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:43.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.668+0000 7ffa5c8cb700 1 --2- 172.21.15.32:0/3599422344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa58066ef0 0x7ffa5810f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:43.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.668+0000 7ffa5c8cb700 1 --2- 172.21.15.32:0/3599422344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa581023b0 0x7ffa5810fed0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:43.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.668+0000 7ffa5c8cb700 1 -- 172.21.15.32:0/3599422344 >> 172.21.15.32:0/3599422344 conn(0x7ffa580f7480 msgr2=0x7ffa580f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:43.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.668+0000 7ffa5c8cb700 1 -- 172.21.15.32:0/3599422344 shutdown_connections 2024-04-12T05:37:43.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.668+0000 7ffa5c8cb700 1 -- 172.21.15.32:0/3599422344 wait complete. 2024-04-12T05:37:43.767 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-12T05:37:43.768 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-12T05:37:43.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:43 smithi067 ceph-mon[49010]: pgmap v4: 65 pgs: 65 active+clean; 2.6 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail 2024-04-12T05:37:43.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:43 smithi067 ceph-mon[49010]: from='client.24431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:43.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.808+0000 7f17fccf9700 1 -- 172.21.15.32:0/140390621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f80fc6f0 msgr2=0x7f17f80fcb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:43.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.808+0000 7f17fccf9700 1 --2- 172.21.15.32:0/140390621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f80fc6f0 0x7f17f80fcb60 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f17ec004a50 tx=0x7f17ec00af40 comp rx=0 tx=0).stop 2024-04-12T05:37:43.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.808+0000 7f17fccf9700 1 -- 172.21.15.32:0/140390621 shutdown_connections 2024-04-12T05:37:43.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.808+0000 7f17fccf9700 1 --2- 172.21.15.32:0/140390621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f80fc6f0 0x7f17f80fcb60 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:43.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.808+0000 7f17fccf9700 1 --2- 172.21.15.32:0/140390621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17f80fb440 0x7f17f80fb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:43.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.808+0000 7f17fccf9700 1 -- 172.21.15.32:0/140390621 >> 172.21.15.32:0/140390621 conn(0x7f17f80f6ba0 msgr2=0x7f17f80f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:43.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.808+0000 7f17fccf9700 1 -- 172.21.15.32:0/140390621 shutdown_connections 2024-04-12T05:37:43.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.808+0000 7f17fccf9700 1 -- 172.21.15.32:0/140390621 wait complete. 2024-04-12T05:37:43.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.808+0000 7f17fccf9700 1 Processor -- start 2024-04-12T05:37:43.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.808+0000 7f17fccf9700 1 -- start start 2024-04-12T05:37:43.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.809+0000 7f17fccf9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17f80fb440 0x7f17f810b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:37:43.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.809+0000 7f17fccf9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f80fc6f0 0x7f17f810bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:37:43.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.809+0000 7f17fccf9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f810c170 con 0x7f17f80fc6f0 2024-04-12T05:37:43.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.809+0000 7f17fccf9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f810c9c0 con 0x7f17f80fb440 2024-04-12T05:37:43.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.809+0000 7f17f5d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f80fc6f0 0x7f17f810bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:43.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.809+0000 7f17f659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17f80fb440 0x7f17f810b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:43.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.809+0000 7f17f5d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f80fc6f0 0x7f17f810bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60198/0 (socket says 172.21.15.32:60198) 2024-04-12T05:37:43.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.809+0000 7f17f5d9b700 1 -- 172.21.15.32:0/4037771479 learned_addr learned my addr 172.21.15.32:0/4037771479 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:37:43.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.809+0000 7f17f5d9b700 1 -- 172.21.15.32:0/4037771479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17f80fb440 msgr2=0x7f17f810b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:43.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.809+0000 7f17f5d9b700 1 --2- 172.21.15.32:0/4037771479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17f80fb440 0x7f17f810b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:43.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.809+0000 7f17f5d9b700 1 -- 172.21.15.32:0/4037771479 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ec005040 con 0x7f17f80fc6f0 2024-04-12T05:37:43.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.809+0000 7f17f5d9b700 1 --2- 172.21.15.32:0/4037771479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f80fc6f0 0x7f17f810bb70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f17ec00afd0 tx=0x7f17ec00af40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:43.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.810+0000 7f17e6ffd700 1 -- 172.21.15.32:0/4037771479 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17ec00f040 con 0x7f17f80fc6f0 2024-04-12T05:37:43.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.810+0000 7f17e6ffd700 1 -- 172.21.15.32:0/4037771479 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f17ec00cad0 con 0x7f17f80fc6f0 2024-04-12T05:37:43.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.810+0000 7f17fccf9700 1 -- 172.21.15.32:0/4037771479 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f810cc40 con 0x7f17f80fc6f0 2024-04-12T05:37:43.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.810+0000 7f17e6ffd700 1 -- 172.21.15.32:0/4037771479 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17ec01d6c0 con 0x7f17f80fc6f0 2024-04-12T05:37:43.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.811+0000 7f17fccf9700 1 -- 172.21.15.32:0/4037771479 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f810d180 con 0x7f17f80fc6f0 2024-04-12T05:37:43.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.811+0000 7f17e6ffd700 1 -- 172.21.15.32:0/4037771479 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) v1 ==== 45680+0+0 (secure 0 0 0) 0x7f17ec0133c0 con 0x7f17f80fc6f0 2024-04-12T05:37:43.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.811+0000 7f17e6ffd700 1 --2- 172.21.15.32:0/4037771479 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f17e0038640 0x7f17e003aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:37:43.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.812+0000 7f17e6ffd700 1 -- 172.21.15.32:0/4037771479 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f17ec051170 con 0x7f17f80fc6f0 2024-04-12T05:37:43.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.812+0000 7f17f659c700 1 --2- 172.21.15.32:0/4037771479 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f17e0038640 0x7f17e003aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:43.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.812+0000 7f17fccf9700 1 -- 172.21.15.32:0/4037771479 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f805f5f0 con 0x7f17f80fc6f0 2024-04-12T05:37:43.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.812+0000 7f17f659c700 1 --2- 172.21.15.32:0/4037771479 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f17e0038640 0x7f17e003aaf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009fd0 tx=0x7f17e8011040 comp rx=0 tx=0).ready entity=mgr.24399 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:43.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:43.818+0000 7f17e6ffd700 1 -- 172.21.15.32:0/4037771479 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f17ec022020 con 0x7f17f80fc6f0 2024-04-12T05:37:44.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:44.134+0000 7f17fccf9700 1 -- 172.21.15.32:0/4037771479 --> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f17f810d430 con 0x7f17e0038640 2024-04-12T05:37:44.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:44.135+0000 7f17e6ffd700 1 -- 172.21.15.32:0/4037771479 <== mgr.24399 v2:172.21.15.67:6828/3925433122 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f17f810d430 con 0x7f17e0038640 2024-04-12T05:37:44.133 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:37:44.134 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:37:44.134 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:37:44.134 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-12T05:37:44.134 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [], 2024-04-12T05:37:44.134 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "1/2 daemons upgraded", 2024-04-12T05:37:44.134 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "", 2024-04-12T05:37:44.134 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:37:44.135 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:37:44.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:44.137+0000 7f17fccf9700 1 -- 172.21.15.32:0/4037771479 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f17e0038640 msgr2=0x7f17e003aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:44.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:44.137+0000 7f17fccf9700 1 --2- 172.21.15.32:0/4037771479 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f17e0038640 0x7f17e003aaf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009fd0 tx=0x7f17e8011040 comp rx=0 tx=0).stop 2024-04-12T05:37:44.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:44.137+0000 7f17fccf9700 1 -- 172.21.15.32:0/4037771479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f80fc6f0 msgr2=0x7f17f810bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:44.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:44.137+0000 7f17fccf9700 1 --2- 172.21.15.32:0/4037771479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f80fc6f0 0x7f17f810bb70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f17ec00afd0 tx=0x7f17ec00af40 comp rx=0 tx=0).stop 2024-04-12T05:37:44.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:44.138+0000 7f17fccf9700 1 -- 172.21.15.32:0/4037771479 shutdown_connections 2024-04-12T05:37:44.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:44.138+0000 7f17fccf9700 1 --2- 172.21.15.32:0/4037771479 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f17e0038640 0x7f17e003aaf0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:44.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:44.138+0000 7f17fccf9700 1 --2- 172.21.15.32:0/4037771479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17f80fb440 0x7f17f810b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:44.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:44.138+0000 7f17fccf9700 1 --2- 172.21.15.32:0/4037771479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f80fc6f0 0x7f17f810bb70 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:37:44.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:44.138+0000 7f17fccf9700 1 -- 172.21.15.32:0/4037771479 >> 172.21.15.32:0/4037771479 conn(0x7f17f80f6ba0 msgr2=0x7f17f80f82c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:44.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:44.138+0000 7f17fccf9700 1 -- 172.21.15.32:0/4037771479 shutdown_connections 2024-04-12T05:37:44.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:37:44.138+0000 7f17fccf9700 1 -- 172.21.15.32:0/4037771479 wait complete. 2024-04-12T05:37:44.405 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:44 smithi032 ceph-mon[39731]: from='client.14644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:44.405 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:44 smithi032 ceph-mon[39731]: from='client.14648 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:44.405 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:44 smithi032 ceph-mon[39731]: mgrmap e25: smithi067.dwichg(active, since 4s) 2024-04-12T05:37:44.405 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:44 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/3599422344' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:37:44.585 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:44 smithi067 ceph-mon[49010]: from='client.14644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:44.585 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:44 smithi067 ceph-mon[49010]: from='client.14648 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:44.585 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:44 smithi067 ceph-mon[49010]: mgrmap e25: smithi067.dwichg(active, since 4s) 2024-04-12T05:37:44.585 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:44 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/3599422344' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:37:45.459 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:45 smithi032 ceph-mon[39731]: pgmap v5: 65 pgs: 65 active+clean; 2.6 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail 2024-04-12T05:37:45.460 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:45 smithi032 ceph-mon[39731]: [12/Apr/2024:05:37:43] ENGINE Bus STARTING 2024-04-12T05:37:45.460 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:45 smithi032 ceph-mon[39731]: [12/Apr/2024:05:37:43] ENGINE Serving on http://172.21.15.67:8765 2024-04-12T05:37:45.460 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:45 smithi032 ceph-mon[39731]: [12/Apr/2024:05:37:43] ENGINE Serving on https://172.21.15.67:7150 2024-04-12T05:37:45.460 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:45 smithi032 ceph-mon[39731]: [12/Apr/2024:05:37:43] ENGINE Bus STARTED 2024-04-12T05:37:45.460 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:45 smithi032 ceph-mon[39731]: from='client.14656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:45.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:45 smithi067 ceph-mon[49010]: pgmap v5: 65 pgs: 65 active+clean; 2.6 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail 2024-04-12T05:37:45.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:45 smithi067 ceph-mon[49010]: [12/Apr/2024:05:37:43] ENGINE Bus STARTING 2024-04-12T05:37:45.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:45 smithi067 ceph-mon[49010]: [12/Apr/2024:05:37:43] ENGINE Serving on http://172.21.15.67:8765 2024-04-12T05:37:45.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:45 smithi067 ceph-mon[49010]: [12/Apr/2024:05:37:43] ENGINE Serving on https://172.21.15.67:7150 2024-04-12T05:37:45.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:45 smithi067 ceph-mon[49010]: [12/Apr/2024:05:37:43] ENGINE Bus STARTED 2024-04-12T05:37:45.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:45 smithi067 ceph-mon[49010]: from='client.14656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:47.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:46 smithi032 ceph-mon[39731]: pgmap v6: 65 pgs: 65 active+clean; 2.4 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 9.2 KiB/s rd, 31 KiB/s wr, 26 op/s 2024-04-12T05:37:47.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:46 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:47.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:46 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:47.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:46 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:47.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:46 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:47.229 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:46 smithi067 ceph-mon[49010]: pgmap v6: 65 pgs: 65 active+clean; 2.4 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 9.2 KiB/s rd, 31 KiB/s wr, 26 op/s 2024-04-12T05:37:47.230 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:46 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:47.230 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:46 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:47.230 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:46 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:47.230 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:46 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:49.020 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:48 smithi067 ceph-mon[49010]: pgmap v7: 65 pgs: 65 active+clean; 1.8 GiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 806 KiB/s rd, 892 KiB/s wr, 125 op/s 2024-04-12T05:37:49.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:48 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:49.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:48 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:49.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:48 smithi067 ceph-mon[49010]: Standby manager daemon smithi032.hrwxrh started 2024-04-12T05:37:49.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:48 smithi067 ceph-mon[49010]: from='mgr.? 172.21.15.32:0/1761616470' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi032.hrwxrh/crt"}]: dispatch 2024-04-12T05:37:49.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:48 smithi067 ceph-mon[49010]: from='mgr.? 172.21.15.32:0/1761616470' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T05:37:49.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:48 smithi067 ceph-mon[49010]: from='mgr.? 172.21.15.32:0/1761616470' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi032.hrwxrh/key"}]: dispatch 2024-04-12T05:37:49.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:48 smithi067 ceph-mon[49010]: from='mgr.? 172.21.15.32:0/1761616470' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T05:37:49.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:48 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:49.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:48 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:49.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:48 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:37:49.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:48 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:37:49.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:48 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:37:49.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:48 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:37:49.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:48 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:37:49.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:48 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:37:49.022 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:48 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:49.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:48 smithi032 ceph-mon[39731]: pgmap v7: 65 pgs: 65 active+clean; 1.8 GiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 806 KiB/s rd, 892 KiB/s wr, 125 op/s 2024-04-12T05:37:49.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:48 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:49.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:48 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:49.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:48 smithi032 ceph-mon[39731]: Standby manager daemon smithi032.hrwxrh started 2024-04-12T05:37:49.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:48 smithi032 ceph-mon[39731]: from='mgr.? 172.21.15.32:0/1761616470' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi032.hrwxrh/crt"}]: dispatch 2024-04-12T05:37:49.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:48 smithi032 ceph-mon[39731]: from='mgr.? 172.21.15.32:0/1761616470' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T05:37:49.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:48 smithi032 ceph-mon[39731]: from='mgr.? 172.21.15.32:0/1761616470' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi032.hrwxrh/key"}]: dispatch 2024-04-12T05:37:49.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:48 smithi032 ceph-mon[39731]: from='mgr.? 172.21.15.32:0/1761616470' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T05:37:49.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:48 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:49.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:48 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:49.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:48 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:37:49.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:48 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:37:49.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:48 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:37:49.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:48 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:37:49.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:48 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:37:49.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:48 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:37:49.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:48 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:50.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:49 smithi032 ceph-mon[39731]: Adjusting osd_memory_target on smithi032 to 1570M 2024-04-12T05:37:50.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:49 smithi032 ceph-mon[39731]: mgrmap e26: smithi067.dwichg(active, since 9s), standbys: smithi032.hrwxrh 2024-04-12T05:37:50.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:49 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "mgr metadata", "who": "smithi032.hrwxrh", "id": "smithi032.hrwxrh"}]: dispatch 2024-04-12T05:37:50.117 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:49 smithi067 ceph-mon[49010]: Adjusting osd_memory_target on smithi032 to 1570M 2024-04-12T05:37:50.117 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:49 smithi067 ceph-mon[49010]: mgrmap e26: smithi067.dwichg(active, since 9s), standbys: smithi032.hrwxrh 2024-04-12T05:37:50.117 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:49 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "mgr metadata", "who": "smithi032.hrwxrh", "id": "smithi032.hrwxrh"}]: dispatch 2024-04-12T05:37:51.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:50 smithi032 ceph-mon[39731]: pgmap v8: 65 pgs: 65 active+clean; 1.8 GiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 626 KiB/s rd, 694 KiB/s wr, 97 op/s 2024-04-12T05:37:51.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:50 smithi067 ceph-mon[49010]: pgmap v8: 65 pgs: 65 active+clean; 1.8 GiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 626 KiB/s rd, 694 KiB/s wr, 97 op/s 2024-04-12T05:37:52.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:51 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:52.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:51 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:52.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:51 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:37:52.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:51 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:37:52.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:51 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:37:52.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:51 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:37:52.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:51 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:37:52.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:51 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:37:52.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:51 smithi067 ceph-mon[49010]: Adjusting osd_memory_target on smithi067 to 2592M 2024-04-12T05:37:52.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:51 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:52.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:51 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:37:52.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:51 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:37:52.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:51 smithi067 ceph-mon[49010]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-12T05:37:52.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:51 smithi067 ceph-mon[49010]: Updating smithi067:/etc/ceph/ceph.conf 2024-04-12T05:37:52.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:51 smithi067 ceph-mon[49010]: Updating smithi067:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:37:52.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:51 smithi067 ceph-mon[49010]: Updating smithi032:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:37:52.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:51 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:52.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:51 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:52.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:51 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:37:52.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:51 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:37:52.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:51 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:37:52.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:51 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:37:52.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:51 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:37:52.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:51 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:37:52.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:51 smithi032 ceph-mon[39731]: Adjusting osd_memory_target on smithi067 to 2592M 2024-04-12T05:37:52.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:51 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:52.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:51 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:37:52.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:51 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:37:52.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:51 smithi032 ceph-mon[39731]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-12T05:37:52.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:51 smithi032 ceph-mon[39731]: Updating smithi067:/etc/ceph/ceph.conf 2024-04-12T05:37:52.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:51 smithi032 ceph-mon[39731]: Updating smithi067:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:37:52.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:51 smithi032 ceph-mon[39731]: Updating smithi032:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:37:53.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:53 smithi032 ceph-mon[39731]: pgmap v9: 65 pgs: 65 active+clean; 1.3 GiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 210 op/s 2024-04-12T05:37:53.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:53 smithi032 ceph-mon[39731]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:37:53.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:53 smithi032 ceph-mon[39731]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:37:53.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:53 smithi032 ceph-mon[39731]: Updating smithi067:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:37:53.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:53 smithi032 ceph-mon[39731]: Updating smithi032:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:37:53.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:53 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:53.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:53 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:53.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:53 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:53.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:53 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:53.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:53 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:53.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:53 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:53.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:53 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:53.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:53 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:53.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:53 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:53.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:53 smithi067 ceph-mon[49010]: pgmap v9: 65 pgs: 65 active+clean; 1.3 GiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 210 op/s 2024-04-12T05:37:53.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:53 smithi067 ceph-mon[49010]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:37:53.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:53 smithi067 ceph-mon[49010]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:37:53.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:53 smithi067 ceph-mon[49010]: Updating smithi067:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:37:53.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:53 smithi067 ceph-mon[49010]: Updating smithi032:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:37:53.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:53 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:53.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:53 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:53.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:53 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:53.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:53 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:53.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:53 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:53.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:53 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:53.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:53 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:53.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:53 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:53.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:53 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:37:54.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:54 smithi067 ceph-mon[49010]: Reconfiguring node-exporter.smithi032 (dependencies changed)... 2024-04-12T05:37:54.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:54 smithi067 ceph-mon[49010]: Reconfiguring daemon node-exporter.smithi032 on smithi032 2024-04-12T05:37:54.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:54 smithi032 ceph-mon[39731]: Reconfiguring node-exporter.smithi032 (dependencies changed)... 2024-04-12T05:37:54.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:54 smithi032 ceph-mon[39731]: Reconfiguring daemon node-exporter.smithi032 on smithi032 2024-04-12T05:37:55.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:55 smithi067 ceph-mon[49010]: pgmap v10: 65 pgs: 65 active+clean; 1.3 GiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 192 op/s 2024-04-12T05:37:55.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:55 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:37:55.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:55 smithi032 ceph-mon[39731]: pgmap v10: 65 pgs: 65 active+clean; 1.3 GiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 192 op/s 2024-04-12T05:37:55.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:55 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:37:57.333 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-12T05:37:57.333 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-04-12T05:37:57.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:57 smithi032 ceph-mon[39731]: pgmap v11: 65 pgs: 65 active+clean; 1.1 GiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 233 op/s 2024-04-12T05:37:57.404 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:57 smithi067 ceph-mon[49010]: pgmap v11: 65 pgs: 65 active+clean; 1.1 GiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 233 op/s 2024-04-12T05:37:59.443 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:37:59 smithi032 ceph-mon[39731]: pgmap v12: 65 pgs: 65 active+clean; 841 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 300 op/s 2024-04-12T05:37:59.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:37:59 smithi067 ceph-mon[49010]: pgmap v12: 65 pgs: 65 active+clean; 841 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 300 op/s 2024-04-12T05:38:01.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:01 smithi067 ceph-mon[49010]: pgmap v13: 65 pgs: 65 active+clean; 841 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 238 op/s 2024-04-12T05:38:01.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:01 smithi032 ceph-mon[39731]: pgmap v13: 65 pgs: 65 active+clean; 841 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 238 op/s 2024-04-12T05:38:03.137 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:03 smithi032 ceph-mon[39731]: pgmap v14: 65 pgs: 65 active+clean; 300 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 375 op/s 2024-04-12T05:38:03.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:03 smithi067 ceph-mon[49010]: pgmap v14: 65 pgs: 65 active+clean; 300 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 375 op/s 2024-04-12T05:38:05.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:04 smithi032 ceph-mon[39731]: pgmap v15: 65 pgs: 65 active+clean; 300 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 255 op/s 2024-04-12T05:38:05.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:04 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:05.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:04 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:05.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:04 smithi032 ceph-mon[39731]: Reconfiguring alertmanager.smithi032 (dependencies changed)... 2024-04-12T05:38:05.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:04 smithi032 ceph-mon[39731]: Reconfiguring daemon alertmanager.smithi032 on smithi032 2024-04-12T05:38:05.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:04 smithi067 ceph-mon[49010]: pgmap v15: 65 pgs: 65 active+clean; 300 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 255 op/s 2024-04-12T05:38:05.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:04 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:05.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:04 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:05.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:04 smithi067 ceph-mon[49010]: Reconfiguring alertmanager.smithi032 (dependencies changed)... 2024-04-12T05:38:05.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:04 smithi067 ceph-mon[49010]: Reconfiguring daemon alertmanager.smithi032 on smithi032 2024-04-12T05:38:07.073 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:06 smithi032 ceph-mon[39731]: pgmap v16: 65 pgs: 65 active+clean; 298 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 319 op/s 2024-04-12T05:38:07.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:06 smithi067 ceph-mon[49010]: pgmap v16: 65 pgs: 65 active+clean; 298 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 319 op/s 2024-04-12T05:38:09.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:08 smithi032 ceph-mon[39731]: pgmap v17: 65 pgs: 65 active+clean; 299 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 441 op/s 2024-04-12T05:38:09.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:08 smithi067 ceph-mon[49010]: pgmap v17: 65 pgs: 65 active+clean; 299 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 441 op/s 2024-04-12T05:38:10.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:09 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:38:10.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:09 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:38:11.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:10 smithi067 ceph-mon[49010]: pgmap v18: 65 pgs: 65 active+clean; 299 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 363 op/s 2024-04-12T05:38:11.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:10 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:11.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:10 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:11.325 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:10 smithi032 ceph-mon[39731]: pgmap v18: 65 pgs: 65 active+clean; 299 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 363 op/s 2024-04-12T05:38:11.325 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:10 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:11.325 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:10 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:12.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:11 smithi032 ceph-mon[39731]: Reconfiguring grafana.smithi032 (dependencies changed)... 2024-04-12T05:38:12.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:11 smithi032 ceph-mon[39731]: Reconfiguring daemon grafana.smithi032 on smithi032 2024-04-12T05:38:12.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:11 smithi067 ceph-mon[49010]: Reconfiguring grafana.smithi032 (dependencies changed)... 2024-04-12T05:38:12.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:11 smithi067 ceph-mon[49010]: Reconfiguring daemon grafana.smithi032 on smithi032 2024-04-12T05:38:13.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:12 smithi067 ceph-mon[49010]: pgmap v19: 65 pgs: 65 active+clean; 296 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 575 op/s 2024-04-12T05:38:13.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:12 smithi032 ceph-mon[39731]: pgmap v19: 65 pgs: 65 active+clean; 296 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 575 op/s 2024-04-12T05:38:14.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.300+0000 7f461ac69700 1 -- 172.21.15.32:0/763499161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f461406f9d0 msgr2=0x7f461406e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:14.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.300+0000 7f461ac69700 1 --2- 172.21.15.32:0/763499161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f461406f9d0 0x7f461406e460 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f460c004940 tx=0x7f460c009e30 comp rx=0 tx=0).stop 2024-04-12T05:38:14.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.300+0000 7f461ac69700 1 -- 172.21.15.32:0/763499161 shutdown_connections 2024-04-12T05:38:14.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.300+0000 7f461ac69700 1 --2- 172.21.15.32:0/763499161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f461406e9a0 0x7f461406ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:14.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.300+0000 7f461ac69700 1 --2- 172.21.15.32:0/763499161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f461406f9d0 0x7f461406e460 secure :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f460c004940 tx=0x7f460c009e30 comp rx=0 tx=0).stop 2024-04-12T05:38:14.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.300+0000 7f461ac69700 1 -- 172.21.15.32:0/763499161 >> 172.21.15.32:0/763499161 conn(0x7f4614069de0 msgr2=0x7f461406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:14.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.300+0000 7f461ac69700 1 -- 172.21.15.32:0/763499161 shutdown_connections 2024-04-12T05:38:14.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.301+0000 7f461ac69700 1 -- 172.21.15.32:0/763499161 wait complete. 2024-04-12T05:38:14.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.301+0000 7f461ac69700 1 Processor -- start 2024-04-12T05:38:14.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.301+0000 7f461ac69700 1 -- start start 2024-04-12T05:38:14.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.302+0000 7f461ac69700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f461406e9a0 0x7f46141140b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:14.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.302+0000 7f461ac69700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46141145f0 0x7f4614119580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:14.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.302+0000 7f461ac69700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4614119ba0 con 0x7f461406e9a0 2024-04-12T05:38:14.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.302+0000 7f461ac69700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4614114a60 con 0x7f46141145f0 2024-04-12T05:38:14.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.302+0000 7f4618a05700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f461406e9a0 0x7f46141140b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:14.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.302+0000 7f4618a05700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f461406e9a0 0x7f46141140b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56988/0 (socket says 172.21.15.32:56988) 2024-04-12T05:38:14.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.302+0000 7f4618a05700 1 -- 172.21.15.32:0/3694044174 learned_addr learned my addr 172.21.15.32:0/3694044174 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:38:14.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.302+0000 7f4618a05700 1 -- 172.21.15.32:0/3694044174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46141145f0 msgr2=0x7f4614119580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:14.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.302+0000 7f4618a05700 1 --2- 172.21.15.32:0/3694044174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46141145f0 0x7f4614119580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:14.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.302+0000 7f4618a05700 1 -- 172.21.15.32:0/3694044174 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f460c005040 con 0x7f461406e9a0 2024-04-12T05:38:14.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.303+0000 7f4618a05700 1 --2- 172.21.15.32:0/3694044174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f461406e9a0 0x7f46141140b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f460c00b3b0 tx=0x7f460c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:14.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.304+0000 7f46117fa700 1 -- 172.21.15.32:0/3694044174 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f460c00e040 con 0x7f461406e9a0 2024-04-12T05:38:14.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.304+0000 7f46117fa700 1 -- 172.21.15.32:0/3694044174 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f460c004290 con 0x7f461406e9a0 2024-04-12T05:38:14.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.304+0000 7f46117fa700 1 -- 172.21.15.32:0/3694044174 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f460c01c3f0 con 0x7f461406e9a0 2024-04-12T05:38:14.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.304+0000 7f461ac69700 1 -- 172.21.15.32:0/3694044174 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4614114ce0 con 0x7f461406e9a0 2024-04-12T05:38:14.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.304+0000 7f461ac69700 1 -- 172.21.15.32:0/3694044174 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f461412b0d0 con 0x7f461406e9a0 2024-04-12T05:38:14.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.306+0000 7f461ac69700 1 -- 172.21.15.32:0/3694044174 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f461404c710 con 0x7f461406e9a0 2024-04-12T05:38:14.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.306+0000 7f46117fa700 1 -- 172.21.15.32:0/3694044174 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 26) v1 ==== 88764+0+0 (secure 0 0 0) 0x7f460c013800 con 0x7f461406e9a0 2024-04-12T05:38:14.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.307+0000 7f46117fa700 1 --2- 172.21.15.32:0/3694044174 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f460006af80 0x7f460006d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:14.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.307+0000 7f46117fa700 1 -- 172.21.15.32:0/3694044174 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f460c08aab0 con 0x7f461406e9a0 2024-04-12T05:38:14.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.308+0000 7f4613fff700 1 --2- 172.21.15.32:0/3694044174 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f460006af80 0x7f460006d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:14.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.308+0000 7f4613fff700 1 --2- 172.21.15.32:0/3694044174 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f460006af80 0x7f460006d430 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f45fc006fd0 tx=0x7f45fc006db0 comp rx=0 tx=0).ready entity=mgr.24399 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:14.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.315+0000 7f46117fa700 1 -- 172.21.15.32:0/3694044174 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f460c058a50 con 0x7f461406e9a0 2024-04-12T05:38:14.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.615+0000 7f461ac69700 1 -- 172.21.15.32:0/3694044174 --> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f461410a500 con 0x7f460006af80 2024-04-12T05:38:14.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.617+0000 7f46117fa700 1 -- 172.21.15.32:0/3694044174 <== mgr.24399 v2:172.21.15.67:6828/3925433122 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f461410a500 con 0x7f460006af80 2024-04-12T05:38:14.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.619+0000 7f461ac69700 1 -- 172.21.15.32:0/3694044174 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f460006af80 msgr2=0x7f460006d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:14.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.619+0000 7f461ac69700 1 --2- 172.21.15.32:0/3694044174 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f460006af80 0x7f460006d430 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f45fc006fd0 tx=0x7f45fc006db0 comp rx=0 tx=0).stop 2024-04-12T05:38:14.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.619+0000 7f461ac69700 1 -- 172.21.15.32:0/3694044174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f461406e9a0 msgr2=0x7f46141140b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:14.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.619+0000 7f461ac69700 1 --2- 172.21.15.32:0/3694044174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f461406e9a0 0x7f46141140b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f460c00b3b0 tx=0x7f460c009e30 comp rx=0 tx=0).stop 2024-04-12T05:38:14.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.620+0000 7f461ac69700 1 -- 172.21.15.32:0/3694044174 shutdown_connections 2024-04-12T05:38:14.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.620+0000 7f461ac69700 1 --2- 172.21.15.32:0/3694044174 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f460006af80 0x7f460006d430 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:14.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.620+0000 7f461ac69700 1 --2- 172.21.15.32:0/3694044174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f461406e9a0 0x7f46141140b0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:14.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.620+0000 7f461ac69700 1 --2- 172.21.15.32:0/3694044174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46141145f0 0x7f4614119580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:14.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.620+0000 7f461ac69700 1 -- 172.21.15.32:0/3694044174 >> 172.21.15.32:0/3694044174 conn(0x7f4614069de0 msgr2=0x7f461406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:14.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.620+0000 7f461ac69700 1 -- 172.21.15.32:0/3694044174 shutdown_connections 2024-04-12T05:38:14.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.620+0000 7f461ac69700 1 -- 172.21.15.32:0/3694044174 wait complete. 2024-04-12T05:38:14.637 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:38:14.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.783+0000 7f60a45a1700 1 -- 172.21.15.32:0/2315283100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f609c0fff40 msgr2=0x7f609c102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:14.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.783+0000 7f60a45a1700 1 --2- 172.21.15.32:0/2315283100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f609c0fff40 0x7f609c102320 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6098004940 tx=0x7f6098009e30 comp rx=0 tx=0).stop 2024-04-12T05:38:14.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.783+0000 7f60a45a1700 1 -- 172.21.15.32:0/2315283100 shutdown_connections 2024-04-12T05:38:14.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.783+0000 7f60a45a1700 1 --2- 172.21.15.32:0/2315283100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f609c0fff40 0x7f609c102320 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:14.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.783+0000 7f60a45a1700 1 --2- 172.21.15.32:0/2315283100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f609c0662a0 0x7f609c0ffa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:14.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.783+0000 7f60a45a1700 1 -- 172.21.15.32:0/2315283100 >> 172.21.15.32:0/2315283100 conn(0x7f609c0f7430 msgr2=0x7f609c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:14.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.784+0000 7f60a45a1700 1 -- 172.21.15.32:0/2315283100 shutdown_connections 2024-04-12T05:38:14.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.784+0000 7f60a45a1700 1 -- 172.21.15.32:0/2315283100 wait complete. 2024-04-12T05:38:14.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.784+0000 7f60a45a1700 1 Processor -- start 2024-04-12T05:38:14.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.784+0000 7f60a45a1700 1 -- start start 2024-04-12T05:38:14.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.785+0000 7f60a45a1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f609c0662a0 0x7f609c10b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:14.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.785+0000 7f60a45a1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f609c0fff40 0x7f609c10bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:14.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.785+0000 7f60a45a1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609c10c360 con 0x7f609c0662a0 2024-04-12T05:38:14.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.785+0000 7f60a45a1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609c10cb20 con 0x7f609c0fff40 2024-04-12T05:38:14.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.785+0000 7f60a1b3c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f609c0fff40 0x7f609c10bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:14.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.785+0000 7f60a1b3c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f609c0fff40 0x7f609c10bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.32:46484/0 (socket says 172.21.15.32:46484) 2024-04-12T05:38:14.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.785+0000 7f60a1b3c700 1 -- 172.21.15.32:0/3036832872 learned_addr learned my addr 172.21.15.32:0/3036832872 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:38:14.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.785+0000 7f60a1b3c700 1 -- 172.21.15.32:0/3036832872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f609c0662a0 msgr2=0x7f609c10b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:14.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.785+0000 7f60a1b3c700 1 --2- 172.21.15.32:0/3036832872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f609c0662a0 0x7f609c10b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:14.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.785+0000 7f60a1b3c700 1 -- 172.21.15.32:0/3036832872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6098005040 con 0x7f609c0fff40 2024-04-12T05:38:14.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.785+0000 7f60a1b3c700 1 --2- 172.21.15.32:0/3036832872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f609c0fff40 0x7f609c10bc80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6098009380 tx=0x7f6098009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:14.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.786+0000 7f608effd700 1 -- 172.21.15.32:0/3036832872 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f609800e040 con 0x7f609c0fff40 2024-04-12T05:38:14.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.786+0000 7f608effd700 1 -- 172.21.15.32:0/3036832872 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f609800bd20 con 0x7f609c0fff40 2024-04-12T05:38:14.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.786+0000 7f608effd700 1 -- 172.21.15.32:0/3036832872 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f609801c630 con 0x7f609c0fff40 2024-04-12T05:38:14.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.786+0000 7f60a45a1700 1 -- 172.21.15.32:0/3036832872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f609c10cda0 con 0x7f609c0fff40 2024-04-12T05:38:14.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.786+0000 7f60a45a1700 1 -- 172.21.15.32:0/3036832872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f609c10d2e0 con 0x7f609c0fff40 2024-04-12T05:38:14.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.787+0000 7f60a45a1700 1 -- 172.21.15.32:0/3036832872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f609c0f8fb0 con 0x7f609c0fff40 2024-04-12T05:38:14.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.788+0000 7f608effd700 1 -- 172.21.15.32:0/3036832872 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 26) v1 ==== 88764+0+0 (secure 0 0 0) 0x7f6098013a40 con 0x7f609c0fff40 2024-04-12T05:38:14.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.789+0000 7f608effd700 1 --2- 172.21.15.32:0/3036832872 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f608806aec0 0x7f608806d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:14.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.789+0000 7f608effd700 1 -- 172.21.15.32:0/3036832872 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f609808a8d0 con 0x7f609c0fff40 2024-04-12T05:38:14.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.789+0000 7f60a233d700 1 --2- 172.21.15.32:0/3036832872 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f608806aec0 0x7f608806d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:14.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.789+0000 7f60a233d700 1 --2- 172.21.15.32:0/3036832872 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f608806aec0 0x7f608806d370 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6090006fd0 tx=0x7f6090006db0 comp rx=0 tx=0).ready entity=mgr.24399 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:14.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:14.794+0000 7f608effd700 1 -- 172.21.15.32:0/3036832872 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f6098054e40 con 0x7f609c0fff40 2024-04-12T05:38:14.965 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:14 smithi067 ceph-mon[49010]: pgmap v20: 65 pgs: 65 active+clean; 296 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 438 op/s 2024-04-12T05:38:15.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.092+0000 7f60a45a1700 1 -- 172.21.15.32:0/3036832872 --> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f609c10d5c0 con 0x7f608806aec0 2024-04-12T05:38:15.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.093+0000 7f608effd700 1 -- 172.21.15.32:0/3036832872 <== mgr.24399 v2:172.21.15.67:6828/3925433122 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f609c10d5c0 con 0x7f608806aec0 2024-04-12T05:38:15.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.097+0000 7f60a45a1700 1 -- 172.21.15.32:0/3036832872 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f608806aec0 msgr2=0x7f608806d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:15.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.097+0000 7f60a45a1700 1 --2- 172.21.15.32:0/3036832872 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f608806aec0 0x7f608806d370 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6090006fd0 tx=0x7f6090006db0 comp rx=0 tx=0).stop 2024-04-12T05:38:15.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.097+0000 7f60a45a1700 1 -- 172.21.15.32:0/3036832872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f609c0fff40 msgr2=0x7f609c10bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:15.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.097+0000 7f60a45a1700 1 --2- 172.21.15.32:0/3036832872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f609c0fff40 0x7f609c10bc80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6098009380 tx=0x7f6098009e30 comp rx=0 tx=0).stop 2024-04-12T05:38:15.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.097+0000 7f60a45a1700 1 -- 172.21.15.32:0/3036832872 shutdown_connections 2024-04-12T05:38:15.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.097+0000 7f60a45a1700 1 --2- 172.21.15.32:0/3036832872 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f608806aec0 0x7f608806d370 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:15.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.097+0000 7f60a45a1700 1 --2- 172.21.15.32:0/3036832872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f609c0662a0 0x7f609c10b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:15.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.097+0000 7f60a45a1700 1 --2- 172.21.15.32:0/3036832872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f609c0fff40 0x7f609c10bc80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:15.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.097+0000 7f60a45a1700 1 -- 172.21.15.32:0/3036832872 >> 172.21.15.32:0/3036832872 conn(0x7f609c0f7430 msgr2=0x7f609c0fbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:15.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.097+0000 7f60a45a1700 1 -- 172.21.15.32:0/3036832872 shutdown_connections 2024-04-12T05:38:15.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.097+0000 7f60a45a1700 1 -- 172.21.15.32:0/3036832872 wait complete. 2024-04-12T05:38:15.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.234+0000 7f1635245700 1 -- 172.21.15.32:0/4255639982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16300fbcd0 msgr2=0x7f16300fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:15.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.234+0000 7f1635245700 1 --2- 172.21.15.32:0/4255639982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16300fbcd0 0x7f16300fc0e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f16200049c0 tx=0x7f1620009e30 comp rx=0 tx=0).stop 2024-04-12T05:38:15.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.234+0000 7f1635245700 1 -- 172.21.15.32:0/4255639982 shutdown_connections 2024-04-12T05:38:15.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.234+0000 7f1635245700 1 --2- 172.21.15.32:0/4255639982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16300fcf80 0x7f16300fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:15.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.234+0000 7f1635245700 1 --2- 172.21.15.32:0/4255639982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16300fbcd0 0x7f16300fc0e0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:15.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.234+0000 7f1635245700 1 -- 172.21.15.32:0/4255639982 >> 172.21.15.32:0/4255639982 conn(0x7f16300f7430 msgr2=0x7f16300f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:15.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.234+0000 7f1635245700 1 -- 172.21.15.32:0/4255639982 shutdown_connections 2024-04-12T05:38:15.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.234+0000 7f1635245700 1 -- 172.21.15.32:0/4255639982 wait complete. 2024-04-12T05:38:15.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.235+0000 7f1635245700 1 Processor -- start 2024-04-12T05:38:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.235+0000 7f1635245700 1 -- start start 2024-04-12T05:38:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.235+0000 7f1635245700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16300fbcd0 0x7f163006e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.235+0000 7f1635245700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16300fcf80 0x7f163006ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.235+0000 7f1635245700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163006ef40 con 0x7f16300fbcd0 2024-04-12T05:38:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.235+0000 7f1635245700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163006f080 con 0x7f16300fcf80 2024-04-12T05:38:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.235+0000 7f162ed9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16300fbcd0 0x7f163006e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.235+0000 7f162ed9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16300fbcd0 0x7f163006e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57014/0 (socket says 172.21.15.32:57014) 2024-04-12T05:38:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.235+0000 7f162ed9d700 1 -- 172.21.15.32:0/3555138247 learned_addr learned my addr 172.21.15.32:0/3555138247 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:38:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.236+0000 7f162e59c700 1 --2- 172.21.15.32:0/3555138247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16300fcf80 0x7f163006ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.236+0000 7f162ed9d700 1 -- 172.21.15.32:0/3555138247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16300fcf80 msgr2=0x7f163006ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.236+0000 7f162ed9d700 1 --2- 172.21.15.32:0/3555138247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16300fcf80 0x7f163006ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.236+0000 7f162ed9d700 1 -- 172.21.15.32:0/3555138247 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1620005040 con 0x7f16300fbcd0 2024-04-12T05:38:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.236+0000 7f162e59c700 1 --2- 172.21.15.32:0/3555138247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16300fcf80 0x7f163006ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:38:15.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.236+0000 7f162ed9d700 1 --2- 172.21.15.32:0/3555138247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16300fbcd0 0x7f163006e4c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1620009fd0 tx=0x7f162000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:15.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.236+0000 7f16177fe700 1 -- 172.21.15.32:0/3555138247 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16200092b0 con 0x7f16300fbcd0 2024-04-12T05:38:15.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.236+0000 7f16177fe700 1 -- 172.21.15.32:0/3555138247 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f16200098c0 con 0x7f16300fbcd0 2024-04-12T05:38:15.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.236+0000 7f1635245700 1 -- 172.21.15.32:0/3555138247 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f163006f300 con 0x7f16300fbcd0 2024-04-12T05:38:15.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.236+0000 7f16177fe700 1 -- 172.21.15.32:0/3555138247 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1620021c70 con 0x7f16300fbcd0 2024-04-12T05:38:15.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.236+0000 7f1635245700 1 -- 172.21.15.32:0/3555138247 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1630119cc0 con 0x7f16300fbcd0 2024-04-12T05:38:15.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.237+0000 7f1635245700 1 -- 172.21.15.32:0/3555138247 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f163005f5f0 con 0x7f16300fbcd0 2024-04-12T05:38:15.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.238+0000 7f16177fe700 1 -- 172.21.15.32:0/3555138247 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 26) v1 ==== 88764+0+0 (secure 0 0 0) 0x7f1620027020 con 0x7f16300fbcd0 2024-04-12T05:38:15.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.239+0000 7f16177fe700 1 --2- 172.21.15.32:0/3555138247 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f161806aec0 0x7f161806d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:15.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.239+0000 7f16177fe700 1 -- 172.21.15.32:0/3555138247 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f1620093580 con 0x7f16300fbcd0 2024-04-12T05:38:15.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.239+0000 7f162e59c700 1 --2- 172.21.15.32:0/3555138247 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f161806aec0 0x7f161806d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:15.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.240+0000 7f162e59c700 1 --2- 172.21.15.32:0/3555138247 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f161806aec0 0x7f161806d370 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1624004fd0 tx=0x7f162400f040 comp rx=0 tx=0).ready entity=mgr.24399 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:15.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.246+0000 7f16177fe700 1 -- 172.21.15.32:0/3555138247 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f162005dae0 con 0x7f16300fbcd0 2024-04-12T05:38:15.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:14 smithi032 ceph-mon[39731]: pgmap v20: 65 pgs: 65 active+clean; 296 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 438 op/s 2024-04-12T05:38:15.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.538+0000 7f1635245700 1 -- 172.21.15.32:0/3555138247 --> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f16301013d0 con 0x7f161806aec0 2024-04-12T05:38:15.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.552+0000 7f16177fe700 1 -- 172.21.15.32:0/3555138247 <== mgr.24399 v2:172.21.15.67:6828/3925433122 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3388 (secure 0 0 0) 0x7f16301013d0 con 0x7f161806aec0 2024-04-12T05:38:15.551 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:38:15.551 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 starting - - - - 2024-04-12T05:38:15.551 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (9m) 27s ago 9m 6656k - 17.2.7-827-gd3b5060a 21de9b7369ad a9cfc00c311b 2024-04-12T05:38:15.551 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (8m) 29s ago 8m 6651k - 17.2.7-827-gd3b5060a 21de9b7369ad 1f2eea4542d9 2024-04-12T05:38:15.551 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 starting - - - - 2024-04-12T05:38:15.551 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (3m) 27s ago 3m 119M - 17.2.7-827-gd3b5060a 21de9b7369ad e7ada8e5259c 2024-04-12T05:38:15.551 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (3m) 27s ago 3m 205M - 17.2.7-827-gd3b5060a 21de9b7369ad 55124da9afdb 2024-04-12T05:38:15.551 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (3m) 29s ago 3m 98.4M - 17.2.7-827-gd3b5060a 21de9b7369ad 0c295d31a4a6 2024-04-12T05:38:15.551 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (3m) 29s ago 3m 153M - 17.2.7-827-gd3b5060a 21de9b7369ad 6eecd7deaac2 2024-04-12T05:38:15.551 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:9283 running (10m) 27s ago 10m 170M - 17.2.7-827-gd3b5060a 21de9b7369ad fdf859c0313c 2024-04-12T05:38:15.551 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283 running (91s) 29s ago 8m 480M - 18.2.2-1187-gd540ebac bb2c5eac26a4 b190c0307364 2024-04-12T05:38:15.551 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (11m) 27s ago 11m 49.3M 2048M 17.2.7-827-gd3b5060a 21de9b7369ad 75aad95b984d 2024-04-12T05:38:15.552 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (8m) 29s ago 8m 40.0M 2048M 17.2.7-827-gd3b5060a 21de9b7369ad 157be8a8c6d8 2024-04-12T05:38:15.552 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 starting - - - - 2024-04-12T05:38:15.552 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (8m) 29s ago 8m 15.8M - 1.5.0 0da6a335fe13 c324d9436cca 2024-04-12T05:38:15.552 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (7m) 27s ago 7m 227M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad 97446e42ceba 2024-04-12T05:38:15.552 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (7m) 27s ago 7m 242M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad bf9f9a2db2d5 2024-04-12T05:38:15.552 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (6m) 27s ago 6m 190M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad f055cf1ca665 2024-04-12T05:38:15.552 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (6m) 29s ago 6m 247M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad 61b851235351 2024-04-12T05:38:15.552 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (5m) 29s ago 5m 212M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad 2c84d9047e9c 2024-04-12T05:38:15.552 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (5m) 29s ago 5m 235M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad e54839bb6055 2024-04-12T05:38:15.552 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (7m) 27s ago 8m 58.7M - 2.43.0 a07b618ecd1d 996307aec3d2 2024-04-12T05:38:15.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.556+0000 7f1635245700 1 -- 172.21.15.32:0/3555138247 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f161806aec0 msgr2=0x7f161806d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:15.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.556+0000 7f1635245700 1 --2- 172.21.15.32:0/3555138247 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f161806aec0 0x7f161806d370 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1624004fd0 tx=0x7f162400f040 comp rx=0 tx=0).stop 2024-04-12T05:38:15.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.556+0000 7f1635245700 1 -- 172.21.15.32:0/3555138247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16300fbcd0 msgr2=0x7f163006e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:15.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.556+0000 7f1635245700 1 --2- 172.21.15.32:0/3555138247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16300fbcd0 0x7f163006e4c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1620009fd0 tx=0x7f162000b040 comp rx=0 tx=0).stop 2024-04-12T05:38:15.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.556+0000 7f1635245700 1 -- 172.21.15.32:0/3555138247 shutdown_connections 2024-04-12T05:38:15.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.556+0000 7f1635245700 1 --2- 172.21.15.32:0/3555138247 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f161806aec0 0x7f161806d370 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:15.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.556+0000 7f1635245700 1 --2- 172.21.15.32:0/3555138247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16300fbcd0 0x7f163006e4c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:15.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.556+0000 7f1635245700 1 --2- 172.21.15.32:0/3555138247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16300fcf80 0x7f163006ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:15.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.556+0000 7f1635245700 1 -- 172.21.15.32:0/3555138247 >> 172.21.15.32:0/3555138247 conn(0x7f16300f7430 msgr2=0x7f16300f8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:15.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.556+0000 7f1635245700 1 -- 172.21.15.32:0/3555138247 shutdown_connections 2024-04-12T05:38:15.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.557+0000 7f1635245700 1 -- 172.21.15.32:0/3555138247 wait complete. 2024-04-12T05:38:15.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.713+0000 7ff825247700 1 -- 172.21.15.32:0/1164050395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8200fbd00 msgr2=0x7ff8200fc110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:15.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.713+0000 7ff825247700 1 --2- 172.21.15.32:0/1164050395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8200fbd00 0x7ff8200fc110 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff814004940 tx=0x7ff814009e30 comp rx=0 tx=0).stop 2024-04-12T05:38:15.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.713+0000 7ff825247700 1 -- 172.21.15.32:0/1164050395 shutdown_connections 2024-04-12T05:38:15.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.713+0000 7ff825247700 1 --2- 172.21.15.32:0/1164050395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8200fcfb0 0x7ff8200fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:15.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.713+0000 7ff825247700 1 --2- 172.21.15.32:0/1164050395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8200fbd00 0x7ff8200fc110 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:15.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.713+0000 7ff825247700 1 -- 172.21.15.32:0/1164050395 >> 172.21.15.32:0/1164050395 conn(0x7ff8200f7440 msgr2=0x7ff8200f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:15.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.713+0000 7ff825247700 1 -- 172.21.15.32:0/1164050395 shutdown_connections 2024-04-12T05:38:15.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.713+0000 7ff825247700 1 -- 172.21.15.32:0/1164050395 wait complete. 2024-04-12T05:38:15.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.714+0000 7ff825247700 1 Processor -- start 2024-04-12T05:38:15.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.714+0000 7ff825247700 1 -- start start 2024-04-12T05:38:15.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.714+0000 7ff825247700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8200fbd00 0x7ff82010d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:15.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.714+0000 7ff825247700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8200fcfb0 0x7ff82010de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:15.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.714+0000 7ff825247700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82010e4a0 con 0x7ff8200fcfb0 2024-04-12T05:38:15.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.714+0000 7ff825247700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff820112df0 con 0x7ff8200fbd00 2024-04-12T05:38:15.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.714+0000 7ff81e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8200fcfb0 0x7ff82010de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:15.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.714+0000 7ff81ed9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8200fbd00 0x7ff82010d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:15.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.714+0000 7ff81e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8200fcfb0 0x7ff82010de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57018/0 (socket says 172.21.15.32:57018) 2024-04-12T05:38:15.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.714+0000 7ff81e59c700 1 -- 172.21.15.32:0/401055862 learned_addr learned my addr 172.21.15.32:0/401055862 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:38:15.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.715+0000 7ff81ed9d700 1 -- 172.21.15.32:0/401055862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8200fcfb0 msgr2=0x7ff82010de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:15.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.715+0000 7ff81ed9d700 1 --2- 172.21.15.32:0/401055862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8200fcfb0 0x7ff82010de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:15.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.715+0000 7ff81ed9d700 1 -- 172.21.15.32:0/401055862 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff814005040 con 0x7ff8200fbd00 2024-04-12T05:38:15.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.715+0000 7ff81ed9d700 1 --2- 172.21.15.32:0/401055862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8200fbd00 0x7ff82010d8d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff814009fd0 tx=0x7ff814009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:15.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.716+0000 7ff8077fe700 1 -- 172.21.15.32:0/401055862 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff81400e070 con 0x7ff8200fbd00 2024-04-12T05:38:15.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.716+0000 7ff8077fe700 1 -- 172.21.15.32:0/401055862 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff814013420 con 0x7ff8200fbd00 2024-04-12T05:38:15.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.716+0000 7ff825247700 1 -- 172.21.15.32:0/401055862 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff820112ff0 con 0x7ff8200fbd00 2024-04-12T05:38:15.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.716+0000 7ff8077fe700 1 -- 172.21.15.32:0/401055862 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff81401c640 con 0x7ff8200fbd00 2024-04-12T05:38:15.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.716+0000 7ff825247700 1 -- 172.21.15.32:0/401055862 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8200fe440 con 0x7ff8200fbd00 2024-04-12T05:38:15.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.717+0000 7ff825247700 1 -- 172.21.15.32:0/401055862 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff82004c710 con 0x7ff8200fbd00 2024-04-12T05:38:15.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.718+0000 7ff8077fe700 1 -- 172.21.15.32:0/401055862 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 26) v1 ==== 88764+0+0 (secure 0 0 0) 0x7ff814013a00 con 0x7ff8200fbd00 2024-04-12T05:38:15.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.718+0000 7ff8077fe700 1 --2- 172.21.15.32:0/401055862 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7ff80806af80 0x7ff80806d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:15.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.718+0000 7ff8077fe700 1 -- 172.21.15.32:0/401055862 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7ff81408ae30 con 0x7ff8200fbd00 2024-04-12T05:38:15.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.719+0000 7ff81e59c700 1 --2- 172.21.15.32:0/401055862 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7ff80806af80 0x7ff80806d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:15.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.722+0000 7ff81e59c700 1 --2- 172.21.15.32:0/401055862 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7ff80806af80 0x7ff80806d430 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff810009c10 tx=0x7ff810009250 comp rx=0 tx=0).ready entity=mgr.24399 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:15.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:15.727+0000 7ff8077fe700 1 -- 172.21.15.32:0/401055862 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7ff814058e80 con 0x7ff8200fbd00 2024-04-12T05:38:15.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:15 smithi032 ceph-mon[39731]: from='client.14660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:38:15.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:15 smithi032 ceph-mon[39731]: from='client.24449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:38:15.967 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:15 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:15.968 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:15 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:16.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.143+0000 7ff825247700 1 -- 172.21.15.32:0/401055862 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff820104730 con 0x7ff8200fbd00 2024-04-12T05:38:16.142 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:38:16.142 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:38:16.142 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 2 2024-04-12T05:38:16.142 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:38:16.142 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:38:16.142 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 1, 2024-04-12T05:38:16.142 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 1 2024-04-12T05:38:16.142 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:38:16.142 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:38:16.142 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 6 2024-04-12T05:38:16.142 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:38:16.142 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:38:16.143 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 4 2024-04-12T05:38:16.143 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:38:16.143 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:38:16.143 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 13, 2024-04-12T05:38:16.143 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 1 2024-04-12T05:38:16.143 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:38:16.143 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:38:16.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.143+0000 7ff8077fe700 1 -- 172.21.15.32:0/401055862 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+850 (secure 0 0 0) 0x7ff814058a10 con 0x7ff8200fbd00 2024-04-12T05:38:16.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.146+0000 7ff825247700 1 -- 172.21.15.32:0/401055862 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7ff80806af80 msgr2=0x7ff80806d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:16.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.146+0000 7ff825247700 1 --2- 172.21.15.32:0/401055862 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7ff80806af80 0x7ff80806d430 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff810009c10 tx=0x7ff810009250 comp rx=0 tx=0).stop 2024-04-12T05:38:16.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.146+0000 7ff825247700 1 -- 172.21.15.32:0/401055862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8200fbd00 msgr2=0x7ff82010d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:16.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.146+0000 7ff825247700 1 --2- 172.21.15.32:0/401055862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8200fbd00 0x7ff82010d8d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff814009fd0 tx=0x7ff814009e30 comp rx=0 tx=0).stop 2024-04-12T05:38:16.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.146+0000 7ff825247700 1 -- 172.21.15.32:0/401055862 shutdown_connections 2024-04-12T05:38:16.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.146+0000 7ff825247700 1 --2- 172.21.15.32:0/401055862 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7ff80806af80 0x7ff80806d430 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:16.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.146+0000 7ff825247700 1 --2- 172.21.15.32:0/401055862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8200fbd00 0x7ff82010d8d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:16.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.146+0000 7ff825247700 1 --2- 172.21.15.32:0/401055862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8200fcfb0 0x7ff82010de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:16.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.146+0000 7ff825247700 1 -- 172.21.15.32:0/401055862 >> 172.21.15.32:0/401055862 conn(0x7ff8200f7440 msgr2=0x7ff820066820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:16.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.149+0000 7ff825247700 1 -- 172.21.15.32:0/401055862 shutdown_connections 2024-04-12T05:38:16.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.149+0000 7ff825247700 1 -- 172.21.15.32:0/401055862 wait complete. 2024-04-12T05:38:16.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:15 smithi067 ceph-mon[49010]: from='client.14660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:38:16.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:15 smithi067 ceph-mon[49010]: from='client.24449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:38:16.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:15 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:16.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:15 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:16.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.307+0000 7f08d5c98700 1 -- 172.21.15.32:0/195187591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d00fbd00 msgr2=0x7f08d00fc110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:16.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.307+0000 7f08d5c98700 1 --2- 172.21.15.32:0/195187591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d00fbd00 0x7f08d00fc110 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f08c0004940 tx=0x7f08c0009e30 comp rx=0 tx=0).stop 2024-04-12T05:38:16.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.307+0000 7f08d5c98700 1 -- 172.21.15.32:0/195187591 shutdown_connections 2024-04-12T05:38:16.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.307+0000 7f08d5c98700 1 --2- 172.21.15.32:0/195187591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08d00fcfb0 0x7f08d00fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:16.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.307+0000 7f08d5c98700 1 --2- 172.21.15.32:0/195187591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d00fbd00 0x7f08d00fc110 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:16.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.307+0000 7f08d5c98700 1 -- 172.21.15.32:0/195187591 >> 172.21.15.32:0/195187591 conn(0x7f08d00f7440 msgr2=0x7f08d00f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:16.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.308+0000 7f08d5c98700 1 -- 172.21.15.32:0/195187591 shutdown_connections 2024-04-12T05:38:16.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.308+0000 7f08d5c98700 1 -- 172.21.15.32:0/195187591 wait complete. 2024-04-12T05:38:16.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.308+0000 7f08d5c98700 1 Processor -- start 2024-04-12T05:38:16.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.308+0000 7f08d5c98700 1 -- start start 2024-04-12T05:38:16.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.309+0000 7f08d5c98700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08d00fbd00 0x7f08d006f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:16.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.309+0000 7f08d5c98700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d00fcfb0 0x7f08d006fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:16.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.309+0000 7f08d5c98700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d0070050 con 0x7f08d00fcfb0 2024-04-12T05:38:16.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.309+0000 7f08d5c98700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d006dbc0 con 0x7f08d00fbd00 2024-04-12T05:38:16.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.309+0000 7f08cf7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08d00fbd00 0x7f08d006f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:16.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.309+0000 7f08cf7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08d00fbd00 0x7f08d006f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.32:46544/0 (socket says 172.21.15.32:46544) 2024-04-12T05:38:16.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.309+0000 7f08cf7fe700 1 -- 172.21.15.32:0/2365322571 learned_addr learned my addr 172.21.15.32:0/2365322571 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:38:16.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.309+0000 7f08c7fff700 1 --2- 172.21.15.32:0/2365322571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d00fcfb0 0x7f08d006fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:16.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.309+0000 7f08c7fff700 1 -- 172.21.15.32:0/2365322571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08d00fbd00 msgr2=0x7f08d006f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:16.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.309+0000 7f08c7fff700 1 --2- 172.21.15.32:0/2365322571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08d00fbd00 0x7f08d006f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:16.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.309+0000 7f08c7fff700 1 -- 172.21.15.32:0/2365322571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c0005040 con 0x7f08d00fcfb0 2024-04-12T05:38:16.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.310+0000 7f08c7fff700 1 --2- 172.21.15.32:0/2365322571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d00fcfb0 0x7f08d006fa50 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f08b8009fd0 tx=0x7f08b800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:16.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.310+0000 7f08ccff9700 1 -- 172.21.15.32:0/2365322571 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08b8004e90 con 0x7f08d00fcfb0 2024-04-12T05:38:16.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.310+0000 7f08ccff9700 1 -- 172.21.15.32:0/2365322571 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f08b800f040 con 0x7f08d00fcfb0 2024-04-12T05:38:16.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.310+0000 7f08ccff9700 1 -- 172.21.15.32:0/2365322571 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08b800c9b0 con 0x7f08d00fcfb0 2024-04-12T05:38:16.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.311+0000 7f08d5c98700 1 -- 172.21.15.32:0/2365322571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08d006de10 con 0x7f08d00fcfb0 2024-04-12T05:38:16.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.311+0000 7f08d5c98700 1 -- 172.21.15.32:0/2365322571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08d006e3d0 con 0x7f08d00fcfb0 2024-04-12T05:38:16.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.312+0000 7f08d5c98700 1 -- 172.21.15.32:0/2365322571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08d0102a50 con 0x7f08d00fcfb0 2024-04-12T05:38:16.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.313+0000 7f08ccff9700 1 -- 172.21.15.32:0/2365322571 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 26) v1 ==== 88764+0+0 (secure 0 0 0) 0x7f08b8007cc0 con 0x7f08d00fcfb0 2024-04-12T05:38:16.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.314+0000 7f08ccff9700 1 --2- 172.21.15.32:0/2365322571 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f08bc06aed0 0x7f08bc06d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:16.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.314+0000 7f08ccff9700 1 -- 172.21.15.32:0/2365322571 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f08b80892e0 con 0x7f08d00fcfb0 2024-04-12T05:38:16.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.314+0000 7f08cf7fe700 1 --2- 172.21.15.32:0/2365322571 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f08bc06aed0 0x7f08bc06d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:16.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.315+0000 7f08cf7fe700 1 --2- 172.21.15.32:0/2365322571 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f08bc06aed0 0x7f08bc06d380 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f08c0013600 tx=0x7f08c0009e30 comp rx=0 tx=0).ready entity=mgr.24399 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:16.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.323+0000 7f08ccff9700 1 -- 172.21.15.32:0/2365322571 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f08b804f560 con 0x7f08d00fcfb0 2024-04-12T05:38:16.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.692+0000 7f08d5c98700 1 -- 172.21.15.32:0/2365322571 --> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f08d006e810 con 0x7f08bc06aed0 2024-04-12T05:38:16.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.694+0000 7f08ccff9700 1 -- 172.21.15.32:0/2365322571 <== mgr.24399 v2:172.21.15.67:6828/3925433122 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f08d006e810 con 0x7f08bc06aed0 2024-04-12T05:38:16.692 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:38:16.692 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:38:16.692 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:38:16.692 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-12T05:38:16.693 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [], 2024-04-12T05:38:16.693 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "1/2 daemons upgraded", 2024-04-12T05:38:16.693 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "", 2024-04-12T05:38:16.693 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:38:16.693 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:38:16.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.698+0000 7f08d5c98700 1 -- 172.21.15.32:0/2365322571 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f08bc06aed0 msgr2=0x7f08bc06d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:16.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.698+0000 7f08d5c98700 1 --2- 172.21.15.32:0/2365322571 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f08bc06aed0 0x7f08bc06d380 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f08c0013600 tx=0x7f08c0009e30 comp rx=0 tx=0).stop 2024-04-12T05:38:16.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.698+0000 7f08d5c98700 1 -- 172.21.15.32:0/2365322571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d00fcfb0 msgr2=0x7f08d006fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:16.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.698+0000 7f08d5c98700 1 --2- 172.21.15.32:0/2365322571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d00fcfb0 0x7f08d006fa50 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f08b8009fd0 tx=0x7f08b800b040 comp rx=0 tx=0).stop 2024-04-12T05:38:16.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.698+0000 7f08d5c98700 1 -- 172.21.15.32:0/2365322571 shutdown_connections 2024-04-12T05:38:16.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.698+0000 7f08d5c98700 1 --2- 172.21.15.32:0/2365322571 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7f08bc06aed0 0x7f08bc06d380 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:16.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.698+0000 7f08d5c98700 1 --2- 172.21.15.32:0/2365322571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08d00fbd00 0x7f08d006f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:16.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.698+0000 7f08d5c98700 1 --2- 172.21.15.32:0/2365322571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d00fcfb0 0x7f08d006fa50 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:16.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.698+0000 7f08d5c98700 1 -- 172.21.15.32:0/2365322571 >> 172.21.15.32:0/2365322571 conn(0x7f08d00f7440 msgr2=0x7f08d00f8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:16.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.698+0000 7f08d5c98700 1 -- 172.21.15.32:0/2365322571 shutdown_connections 2024-04-12T05:38:16.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:16.698+0000 7f08d5c98700 1 -- 172.21.15.32:0/2365322571 wait complete. 2024-04-12T05:38:17.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:16 smithi067 ceph-mon[49010]: Reconfiguring prometheus.smithi032 (dependencies changed)... 2024-04-12T05:38:17.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:16 smithi067 ceph-mon[49010]: pgmap v21: 65 pgs: 65 active+clean; 294 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 503 op/s 2024-04-12T05:38:17.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:16 smithi067 ceph-mon[49010]: from='client.14664 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:38:17.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:16 smithi067 ceph-mon[49010]: Reconfiguring daemon prometheus.smithi032 on smithi032 2024-04-12T05:38:17.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:16 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/401055862' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:38:17.290 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:16 smithi032 ceph-mon[39731]: Reconfiguring prometheus.smithi032 (dependencies changed)... 2024-04-12T05:38:17.290 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:16 smithi032 ceph-mon[39731]: pgmap v21: 65 pgs: 65 active+clean; 294 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 503 op/s 2024-04-12T05:38:17.290 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:16 smithi032 ceph-mon[39731]: from='client.14664 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:38:17.291 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:16 smithi032 ceph-mon[39731]: Reconfiguring daemon prometheus.smithi032 on smithi032 2024-04-12T05:38:17.291 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:16 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/401055862' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:38:18.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:17 smithi032 ceph-mon[39731]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:38:18.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:17 smithi067 ceph-mon[49010]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:38:19.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:18 smithi067 ceph-mon[49010]: pgmap v22: 65 pgs: 65 active+clean; 296 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 626 op/s 2024-04-12T05:38:19.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:18 smithi032 ceph-mon[39731]: pgmap v22: 65 pgs: 65 active+clean; 296 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 626 op/s 2024-04-12T05:38:21.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:20 smithi067 ceph-mon[49010]: pgmap v23: 65 pgs: 65 active+clean; 296 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 463 op/s 2024-04-12T05:38:21.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:20 smithi032 ceph-mon[39731]: pgmap v23: 65 pgs: 65 active+clean; 296 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 463 op/s 2024-04-12T05:38:23.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:22 smithi067 ceph-mon[49010]: pgmap v24: 65 pgs: 65 active+clean; 287 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.4 MiB/s rd, 2.3 MiB/s wr, 663 op/s 2024-04-12T05:38:23.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:22 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:23.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:22 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:23.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:22 smithi067 ceph-mon[49010]: Reconfiguring node-exporter.smithi067 (dependencies changed)... 2024-04-12T05:38:23.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:22 smithi067 ceph-mon[49010]: Reconfiguring daemon node-exporter.smithi067 on smithi067 2024-04-12T05:38:23.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:22 smithi032 ceph-mon[39731]: pgmap v24: 65 pgs: 65 active+clean; 287 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.4 MiB/s rd, 2.3 MiB/s wr, 663 op/s 2024-04-12T05:38:23.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:22 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:23.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:22 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:23.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:22 smithi032 ceph-mon[39731]: Reconfiguring node-exporter.smithi067 (dependencies changed)... 2024-04-12T05:38:23.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:22 smithi032 ceph-mon[39731]: Reconfiguring daemon node-exporter.smithi067 on smithi067 2024-04-12T05:38:25.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:24 smithi067 ceph-mon[49010]: pgmap v25: 65 pgs: 65 active+clean; 287 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 451 op/s 2024-04-12T05:38:25.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:24 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:25.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:24 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:25.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:24 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T05:38:25.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:24 smithi067 ceph-mon[49010]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T05:38:25.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:24 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T05:38:25.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:24 smithi067 ceph-mon[49010]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T05:38:25.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:24 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:38:25.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:24 smithi067 ceph-mon[49010]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:38:25.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:24 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:38:25.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:24 smithi067 ceph-mon[49010]: Upgrade: Need to upgrade myself (mgr.smithi067.dwichg) 2024-04-12T05:38:25.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:24 smithi067 ceph-mon[49010]: Upgrade: Need to upgrade myself (mgr.smithi067.dwichg) 2024-04-12T05:38:25.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:24 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:38:25.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:24 smithi032 ceph-mon[39731]: pgmap v25: 65 pgs: 65 active+clean; 287 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 451 op/s 2024-04-12T05:38:25.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:24 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:25.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:24 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:25.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:24 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T05:38:25.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:24 smithi032 ceph-mon[39731]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T05:38:25.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:24 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T05:38:25.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:24 smithi032 ceph-mon[39731]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T05:38:25.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:24 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:38:25.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:24 smithi032 ceph-mon[39731]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:38:25.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:24 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:38:25.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:24 smithi032 ceph-mon[39731]: Upgrade: Need to upgrade myself (mgr.smithi067.dwichg) 2024-04-12T05:38:25.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:24 smithi032 ceph-mon[39731]: Upgrade: Need to upgrade myself (mgr.smithi067.dwichg) 2024-04-12T05:38:25.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:24 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:38:27.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:26 smithi067 ceph-mon[49010]: pgmap v26: 65 pgs: 65 active+clean; 288 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 524 op/s 2024-04-12T05:38:27.298 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:26 smithi032 ceph-mon[39731]: pgmap v26: 65 pgs: 65 active+clean; 288 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 524 op/s 2024-04-12T05:38:29.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:28 smithi067 ceph-mon[49010]: pgmap v27: 65 pgs: 65 active+clean; 286 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 611 op/s 2024-04-12T05:38:29.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:28 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:29.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:28 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi032.hrwxrh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:38:29.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:28 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi032.hrwxrh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:38:29.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:28 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:38:29.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:28 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:38:29.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:28 smithi032 ceph-mon[39731]: pgmap v27: 65 pgs: 65 active+clean; 286 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 611 op/s 2024-04-12T05:38:29.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:28 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:29.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:28 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi032.hrwxrh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:38:29.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:28 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi032.hrwxrh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:38:29.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:28 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:38:29.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:28 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:38:30.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:29 smithi067 ceph-mon[49010]: Upgrade: Updating mgr.smithi032.hrwxrh 2024-04-12T05:38:30.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:29 smithi067 ceph-mon[49010]: Deploying daemon mgr.smithi032.hrwxrh on smithi032 2024-04-12T05:38:30.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:29 smithi032 ceph-mon[39731]: Upgrade: Updating mgr.smithi032.hrwxrh 2024-04-12T05:38:30.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:29 smithi032 ceph-mon[39731]: Deploying daemon mgr.smithi032.hrwxrh on smithi032 2024-04-12T05:38:31.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:30 smithi032 ceph-mon[39731]: pgmap v28: 65 pgs: 65 active+clean; 286 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 424 op/s 2024-04-12T05:38:31.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:30 smithi067 ceph-mon[49010]: pgmap v28: 65 pgs: 65 active+clean; 286 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 424 op/s 2024-04-12T05:38:33.232 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:32 smithi032 ceph-mon[39731]: pgmap v29: 65 pgs: 65 active+clean; 284 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 618 op/s 2024-04-12T05:38:33.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:32 smithi067 ceph-mon[49010]: pgmap v29: 65 pgs: 65 active+clean; 284 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 618 op/s 2024-04-12T05:38:34.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:33 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:34.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:33 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:34.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:33 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:38:34.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:33 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:34.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:33 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:34.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:33 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:34.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:33 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:34.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:33 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:38:34.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:33 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:34.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:33 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:35.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:34 smithi032 ceph-mon[39731]: pgmap v30: 65 pgs: 65 active+clean; 284 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 418 op/s 2024-04-12T05:38:35.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:34 smithi067 ceph-mon[49010]: pgmap v30: 65 pgs: 65 active+clean; 284 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 418 op/s 2024-04-12T05:38:37.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:37 smithi032 ceph-mon[39731]: pgmap v31: 65 pgs: 65 active+clean; 279 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 520 op/s 2024-04-12T05:38:37.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:37 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:37.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:37 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:37.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:37 smithi067 ceph-mon[49010]: pgmap v31: 65 pgs: 65 active+clean; 279 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 520 op/s 2024-04-12T05:38:37.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:37 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:37.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:37 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:38.367 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-04-12T05:38:38.367 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-12T05:38:38.846 DEBUG:teuthology.parallel:result is None 2024-04-12T05:38:39.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:39 smithi032 ceph-mon[39731]: pgmap v32: 65 pgs: 65 active+clean; 284 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 523 op/s 2024-04-12T05:38:39.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:39 smithi067 ceph-mon[49010]: pgmap v32: 65 pgs: 65 active+clean; 284 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 523 op/s 2024-04-12T05:38:40.555 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:40 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:38:40.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:40 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:38:41.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:41 smithi032 ceph-mon[39731]: pgmap v33: 65 pgs: 65 active+clean; 284 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 372 op/s 2024-04-12T05:38:41.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:41 smithi067 ceph-mon[49010]: pgmap v33: 65 pgs: 65 active+clean; 284 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 372 op/s 2024-04-12T05:38:43.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:42 smithi067 ceph-mon[49010]: pgmap v34: 65 pgs: 65 active+clean; 279 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 552 op/s 2024-04-12T05:38:43.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:42 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:43.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:42 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:43.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:42 smithi067 ceph-mon[49010]: Standby manager daemon smithi032.hrwxrh restarted 2024-04-12T05:38:43.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:42 smithi067 ceph-mon[49010]: Standby manager daemon smithi032.hrwxrh started 2024-04-12T05:38:43.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:42 smithi067 ceph-mon[49010]: from='mgr.? 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi032.hrwxrh/crt"}]: dispatch 2024-04-12T05:38:43.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:42 smithi067 ceph-mon[49010]: from='mgr.? 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T05:38:43.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:42 smithi067 ceph-mon[49010]: from='mgr.? 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi032.hrwxrh/key"}]: dispatch 2024-04-12T05:38:43.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:42 smithi067 ceph-mon[49010]: from='mgr.? 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T05:38:43.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:42 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:43.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:42 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:43.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:42 smithi032 ceph-mon[39731]: pgmap v34: 65 pgs: 65 active+clean; 279 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 552 op/s 2024-04-12T05:38:43.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:42 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:43.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:42 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:43.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:42 smithi032 ceph-mon[39731]: Standby manager daemon smithi032.hrwxrh restarted 2024-04-12T05:38:43.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:42 smithi032 ceph-mon[39731]: Standby manager daemon smithi032.hrwxrh started 2024-04-12T05:38:43.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:42 smithi032 ceph-mon[39731]: from='mgr.? 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi032.hrwxrh/crt"}]: dispatch 2024-04-12T05:38:43.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:42 smithi032 ceph-mon[39731]: from='mgr.? 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T05:38:43.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:42 smithi032 ceph-mon[39731]: from='mgr.? 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi032.hrwxrh/key"}]: dispatch 2024-04-12T05:38:43.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:42 smithi032 ceph-mon[39731]: from='mgr.? 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T05:38:43.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:42 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:43.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:42 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:44.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:43 smithi067 ceph-mon[49010]: mgrmap e27: smithi067.dwichg(active, since 63s), standbys: smithi032.hrwxrh 2024-04-12T05:38:44.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:43 smithi032 ceph-mon[39731]: mgrmap e27: smithi067.dwichg(active, since 63s), standbys: smithi032.hrwxrh 2024-04-12T05:38:45.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:44 smithi067 ceph-mon[49010]: pgmap v35: 65 pgs: 65 active+clean; 279 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 358 op/s 2024-04-12T05:38:45.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:44 smithi032 ceph-mon[39731]: pgmap v35: 65 pgs: 65 active+clean; 279 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 358 op/s 2024-04-12T05:38:46.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.818+0000 7fb6f3093700 1 -- 172.21.15.32:0/3275415070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec0fbcd0 msgr2=0x7fb6ec0fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:46.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.818+0000 7fb6f3093700 1 --2- 172.21.15.32:0/3275415070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec0fbcd0 0x7fb6ec0fc0e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc009ce0 tx=0x7fb6dc004d10 comp rx=0 tx=0).stop 2024-04-12T05:38:46.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.818+0000 7fb6f3093700 1 -- 172.21.15.32:0/3275415070 shutdown_connections 2024-04-12T05:38:46.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.818+0000 7fb6f3093700 1 --2- 172.21.15.32:0/3275415070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6ec0fcf80 0x7fb6ec0fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:46.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.818+0000 7fb6f3093700 1 --2- 172.21.15.32:0/3275415070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec0fbcd0 0x7fb6ec0fc0e0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:46.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.818+0000 7fb6f3093700 1 -- 172.21.15.32:0/3275415070 >> 172.21.15.32:0/3275415070 conn(0x7fb6ec0f7430 msgr2=0x7fb6ec0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:46.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.818+0000 7fb6f3093700 1 -- 172.21.15.32:0/3275415070 shutdown_connections 2024-04-12T05:38:46.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.818+0000 7fb6f3093700 1 -- 172.21.15.32:0/3275415070 wait complete. 2024-04-12T05:38:46.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.819+0000 7fb6f3093700 1 Processor -- start 2024-04-12T05:38:46.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.819+0000 7fb6f3093700 1 -- start start 2024-04-12T05:38:46.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.819+0000 7fb6f3093700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec0fbcd0 0x7fb6ec10c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:46.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.819+0000 7fb6f3093700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6ec0fcf80 0x7fb6ec10cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:46.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.819+0000 7fb6f3093700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ec10d3d0 con 0x7fb6ec0fbcd0 2024-04-12T05:38:46.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.819+0000 7fb6f3093700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ec10dc20 con 0x7fb6ec0fcf80 2024-04-12T05:38:46.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.819+0000 7fb6f0e2f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec0fbcd0 0x7fb6ec10c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:46.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.819+0000 7fb6f0e2f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec0fbcd0 0x7fb6ec10c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57800/0 (socket says 172.21.15.32:57800) 2024-04-12T05:38:46.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.819+0000 7fb6f0e2f700 1 -- 172.21.15.32:0/1350824155 learned_addr learned my addr 172.21.15.32:0/1350824155 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:38:46.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.819+0000 7fb6ebfff700 1 --2- 172.21.15.32:0/1350824155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6ec0fcf80 0x7fb6ec10cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:46.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.820+0000 7fb6f0e2f700 1 -- 172.21.15.32:0/1350824155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6ec0fcf80 msgr2=0x7fb6ec10cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:46.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.820+0000 7fb6f0e2f700 1 --2- 172.21.15.32:0/1350824155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6ec0fcf80 0x7fb6ec10cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:46.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.820+0000 7fb6f0e2f700 1 -- 172.21.15.32:0/1350824155 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6dc009940 con 0x7fb6ec0fbcd0 2024-04-12T05:38:46.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.820+0000 7fb6ebfff700 1 --2- 172.21.15.32:0/1350824155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6ec0fcf80 0x7fb6ec10cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T05:38:46.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.820+0000 7fb6f0e2f700 1 --2- 172.21.15.32:0/1350824155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec0fbcd0 0x7fb6ec10c890 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc004e60 tx=0x7fb6dc004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:46.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.820+0000 7fb6e97fa700 1 -- 172.21.15.32:0/1350824155 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6dc00e040 con 0x7fb6ec0fbcd0 2024-04-12T05:38:46.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.820+0000 7fb6f3093700 1 -- 172.21.15.32:0/1350824155 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6ec10dea0 con 0x7fb6ec0fbcd0 2024-04-12T05:38:46.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.820+0000 7fb6e97fa700 1 -- 172.21.15.32:0/1350824155 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb6dc01cce0 con 0x7fb6ec0fbcd0 2024-04-12T05:38:46.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.820+0000 7fb6e97fa700 1 -- 172.21.15.32:0/1350824155 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6dc012b40 con 0x7fb6ec0fbcd0 2024-04-12T05:38:46.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.820+0000 7fb6f3093700 1 -- 172.21.15.32:0/1350824155 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6ec10e3e0 con 0x7fb6ec0fbcd0 2024-04-12T05:38:46.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.821+0000 7fb6f3093700 1 -- 172.21.15.32:0/1350824155 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6ec05f5f0 con 0x7fb6ec0fbcd0 2024-04-12T05:38:46.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.822+0000 7fb6e97fa700 1 -- 172.21.15.32:0/1350824155 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 27) v1 ==== 90805+0+0 (secure 0 0 0) 0x7fb6dc01c830 con 0x7fb6ec0fbcd0 2024-04-12T05:38:46.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.822+0000 7fb6e97fa700 1 --2- 172.21.15.32:0/1350824155 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7fb6d406c9c0 0x7fb6d406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:46.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.822+0000 7fb6e97fa700 1 -- 172.21.15.32:0/1350824155 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fb6dc08def0 con 0x7fb6ec0fbcd0 2024-04-12T05:38:46.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.822+0000 7fb6ebfff700 1 --2- 172.21.15.32:0/1350824155 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7fb6d406c9c0 0x7fb6d406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:46.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.823+0000 7fb6ebfff700 1 --2- 172.21.15.32:0/1350824155 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7fb6d406c9c0 0x7fb6d406ee70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb6e0009b80 tx=0x7fb6e000b040 comp rx=0 tx=0).ready entity=mgr.24399 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:46.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:46.828+0000 7fb6e97fa700 1 -- 172.21.15.32:0/1350824155 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fb6dc05bd50 con 0x7fb6ec0fbcd0 2024-04-12T05:38:47.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:46 smithi067 ceph-mon[49010]: pgmap v36: 65 pgs: 65 active+clean; 277 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 457 op/s 2024-04-12T05:38:47.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:47.060+0000 7fb6e97fa700 1 -- 172.21.15.32:0/1350824155 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mgrmap(e 28) v1 ==== 45203+0+0 (secure 0 0 0) 0x7fb6dc08c3a0 con 0x7fb6ec0fbcd0 2024-04-12T05:38:47.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:47.060+0000 7fb6e97fa700 1 -- 172.21.15.32:0/1350824155 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7fb6d406c9c0 msgr2=0x7fb6d406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:47.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:47.060+0000 7fb6e97fa700 1 --2- 172.21.15.32:0/1350824155 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7fb6d406c9c0 0x7fb6d406ee70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb6e0009b80 tx=0x7fb6e000b040 comp rx=0 tx=0).stop 2024-04-12T05:38:47.072 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:46 smithi032 ceph-mon[39731]: pgmap v36: 65 pgs: 65 active+clean; 277 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 457 op/s 2024-04-12T05:38:48.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.059+0000 7fb6e97fa700 1 -- 172.21.15.32:0/1350824155 <== mon.0 v2:172.21.15.32:3300/0 8 ==== mgrmap(e 29) v1 ==== 45330+0+0 (secure 0 0 0) 0x7fb6dc058690 con 0x7fb6ec0fbcd0 2024-04-12T05:38:48.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.059+0000 7fb6e97fa700 1 --2- 172.21.15.32:0/1350824155 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fb6d4005dc0 0x7fb6d4035c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:48.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.060+0000 7fb6e97fa700 1 -- 172.21.15.32:0/1350824155 --> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb6d4036710 con 0x7fb6d4005dc0 2024-04-12T05:38:48.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.068+0000 7fb6ebfff700 1 --2- 172.21.15.32:0/1350824155 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fb6d4005dc0 0x7fb6d4035c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:48.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.070+0000 7fb6ebfff700 1 --2- 172.21.15.32:0/1350824155 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fb6d4005dc0 0x7fb6d4035c50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb6e0004ed0 tx=0x7fb6e000b040 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:48.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.073+0000 7fb6e97fa700 1 -- 172.21.15.32:0/1350824155 <== mgr.24463 v2:172.21.15.32:6800/2564765508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7fb6d4036710 con 0x7fb6d4005dc0 2024-04-12T05:38:48.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.075+0000 7fb6f3093700 1 -- 172.21.15.32:0/1350824155 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fb6d4005dc0 msgr2=0x7fb6d4035c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:48.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.075+0000 7fb6f3093700 1 --2- 172.21.15.32:0/1350824155 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fb6d4005dc0 0x7fb6d4035c50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb6e0004ed0 tx=0x7fb6e000b040 comp rx=0 tx=0).stop 2024-04-12T05:38:48.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.075+0000 7fb6f3093700 1 -- 172.21.15.32:0/1350824155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec0fbcd0 msgr2=0x7fb6ec10c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:48.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.075+0000 7fb6f3093700 1 --2- 172.21.15.32:0/1350824155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec0fbcd0 0x7fb6ec10c890 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc004e60 tx=0x7fb6dc004d10 comp rx=0 tx=0).stop 2024-04-12T05:38:48.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.076+0000 7fb6f3093700 1 -- 172.21.15.32:0/1350824155 shutdown_connections 2024-04-12T05:38:48.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.076+0000 7fb6f3093700 1 --2- 172.21.15.32:0/1350824155 >> [v2:172.21.15.67:6828/3925433122,v1:172.21.15.67:6829/3925433122] conn(0x7fb6d406c9c0 0x7fb6d406ee70 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:48.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.076+0000 7fb6f3093700 1 --2- 172.21.15.32:0/1350824155 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fb6d4005dc0 0x7fb6d4035c50 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:48.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.076+0000 7fb6f3093700 1 --2- 172.21.15.32:0/1350824155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec0fbcd0 0x7fb6ec10c890 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:48.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.076+0000 7fb6f3093700 1 --2- 172.21.15.32:0/1350824155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6ec0fcf80 0x7fb6ec10cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:48.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.076+0000 7fb6f3093700 1 -- 172.21.15.32:0/1350824155 >> 172.21.15.32:0/1350824155 conn(0x7fb6ec0f7430 msgr2=0x7fb6ec0f8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:48.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.076+0000 7fb6f3093700 1 -- 172.21.15.32:0/1350824155 shutdown_connections 2024-04-12T05:38:48.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.076+0000 7fb6f3093700 1 -- 172.21.15.32:0/1350824155 wait complete. 2024-04-12T05:38:48.083 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:38:48.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.197+0000 7fa51a376700 1 -- 172.21.15.32:0/3695068304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa514066af0 msgr2=0x7fa5140ffad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:48.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.197+0000 7fa51a376700 1 --2- 172.21.15.32:0/3695068304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa514066af0 0x7fa5140ffad0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa504004940 tx=0x7fa504009e30 comp rx=0 tx=0).stop 2024-04-12T05:38:48.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.197+0000 7fa51a376700 1 -- 172.21.15.32:0/3695068304 shutdown_connections 2024-04-12T05:38:48.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.197+0000 7fa51a376700 1 --2- 172.21.15.32:0/3695068304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa514100010 0x7fa5141023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:48.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.197+0000 7fa51a376700 1 --2- 172.21.15.32:0/3695068304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa514066af0 0x7fa5140ffad0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:48.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.197+0000 7fa51a376700 1 -- 172.21.15.32:0/3695068304 >> 172.21.15.32:0/3695068304 conn(0x7fa5140f74e0 msgr2=0x7fa5140f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:48.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.197+0000 7fa51a376700 1 -- 172.21.15.32:0/3695068304 shutdown_connections 2024-04-12T05:38:48.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.197+0000 7fa51a376700 1 -- 172.21.15.32:0/3695068304 wait complete. 2024-04-12T05:38:48.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.198+0000 7fa51a376700 1 Processor -- start 2024-04-12T05:38:48.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.198+0000 7fa51a376700 1 -- start start 2024-04-12T05:38:48.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.198+0000 7fa51a376700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa514066af0 0x7fa51410d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:48.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.198+0000 7fa51a376700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa514100010 0x7fa51410de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:48.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.198+0000 7fa51a376700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51410e440 con 0x7fa514066af0 2024-04-12T05:38:48.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.198+0000 7fa51a376700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51410ec90 con 0x7fa514100010 2024-04-12T05:38:48.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.199+0000 7fa513fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa514066af0 0x7fa51410d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:48.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.199+0000 7fa513fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa514066af0 0x7fa51410d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57926/0 (socket says 172.21.15.32:57926) 2024-04-12T05:38:48.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.199+0000 7fa5137fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa514100010 0x7fa51410de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:48.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.199+0000 7fa513fff700 1 -- 172.21.15.32:0/1762057081 learned_addr learned my addr 172.21.15.32:0/1762057081 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:38:48.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.199+0000 7fa5137fe700 1 -- 172.21.15.32:0/1762057081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa514066af0 msgr2=0x7fa51410d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:48.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.199+0000 7fa5137fe700 1 --2- 172.21.15.32:0/1762057081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa514066af0 0x7fa51410d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:48.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.199+0000 7fa5137fe700 1 -- 172.21.15.32:0/1762057081 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa504005040 con 0x7fa514100010 2024-04-12T05:38:48.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.199+0000 7fa513fff700 1 --2- 172.21.15.32:0/1762057081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa514066af0 0x7fa51410d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T05:38:48.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.199+0000 7fa5137fe700 1 --2- 172.21.15.32:0/1762057081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa514100010 0x7fa51410de40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa508009fd0 tx=0x7fa50800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:48.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.199+0000 7fa510ff9700 1 -- 172.21.15.32:0/1762057081 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa508004e90 con 0x7fa514100010 2024-04-12T05:38:48.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.199+0000 7fa510ff9700 1 -- 172.21.15.32:0/1762057081 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa50800f040 con 0x7fa514100010 2024-04-12T05:38:48.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.199+0000 7fa510ff9700 1 -- 172.21.15.32:0/1762057081 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa50800c9b0 con 0x7fa514100010 2024-04-12T05:38:48.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.200+0000 7fa51a376700 1 -- 172.21.15.32:0/1762057081 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa51410ef10 con 0x7fa514100010 2024-04-12T05:38:48.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.200+0000 7fa51a376700 1 -- 172.21.15.32:0/1762057081 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa51410f450 con 0x7fa514100010 2024-04-12T05:38:48.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.200+0000 7fa510ff9700 1 -- 172.21.15.32:0/1762057081 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 29) v1 ==== 45330+0+0 (secure 0 0 0) 0x7fa50800cb10 con 0x7fa514100010 2024-04-12T05:38:48.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.201+0000 7fa510ff9700 1 --2- 172.21.15.32:0/1762057081 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fa4fc038230 0x7fa4fc03a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:48.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.201+0000 7fa510ff9700 1 -- 172.21.15.32:0/1762057081 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7fa50804c4d0 con 0x7fa514100010 2024-04-12T05:38:48.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.201+0000 7fa513fff700 1 --2- 172.21.15.32:0/1762057081 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fa4fc038230 0x7fa4fc03a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:48.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.201+0000 7fa51a376700 1 -- 172.21.15.32:0/1762057081 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa51405f5f0 con 0x7fa514100010 2024-04-12T05:38:48.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.201+0000 7fa513fff700 1 --2- 172.21.15.32:0/1762057081 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fa4fc038230 0x7fa4fc03a6e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa504004890 tx=0x7fa504009e30 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:48.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.206+0000 7fa510ff9700 1 -- 172.21.15.32:0/1762057081 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fa50801b2c0 con 0x7fa514100010 2024-04-12T05:38:48.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:48.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:48.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:38:48.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:38:48.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:48.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:38:48.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "mgr fail", "who": "smithi067.dwichg"}]: dispatch 2024-04-12T05:38:48.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "mgr fail", "who": "smithi067.dwichg"}]: dispatch 2024-04-12T05:38:48.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: osdmap e48: 6 total, 6 up, 6 in 2024-04-12T05:38:48.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' cmd='[{"prefix": "mgr fail", "who": "smithi067.dwichg"}]': finished 2024-04-12T05:38:48.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: mgrmap e28: smithi032.hrwxrh(active, starting, since 0.005627s) 2024-04-12T05:38:48.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-12T05:38:48.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-12T05:38:48.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.pndtry"}]: dispatch 2024-04-12T05:38:48.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.jehaaj"}]: dispatch 2024-04-12T05:38:48.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.zesmgy"}]: dispatch 2024-04-12T05:38:48.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.ymvqih"}]: dispatch 2024-04-12T05:38:48.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr metadata", "who": "smithi032.hrwxrh", "id": "smithi032.hrwxrh"}]: dispatch 2024-04-12T05:38:48.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T05:38:48.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T05:38:48.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T05:38:48.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:38:48.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:38:48.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:38:48.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:38:48.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:38:48.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:38:48.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: Manager daemon smithi032.hrwxrh is now available 2024-04-12T05:38:48.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:38:48.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:38:48.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.hrwxrh/mirror_snapshot_schedule"}]: dispatch 2024-04-12T05:38:48.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.hrwxrh/trash_purge_schedule"}]: dispatch 2024-04-12T05:38:48.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: [12/Apr/2024:05:38:47] ENGINE Bus STARTING 2024-04-12T05:38:48.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: [12/Apr/2024:05:38:47] ENGINE Serving on http://172.21.15.32:8765 2024-04-12T05:38:48.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: [12/Apr/2024:05:38:47] ENGINE Serving on https://172.21.15.32:7150 2024-04-12T05:38:48.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:47 smithi067 ceph-mon[49010]: [12/Apr/2024:05:38:47] ENGINE Bus STARTED 2024-04-12T05:38:48.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:48.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:48.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:38:48.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:38:48.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' 2024-04-12T05:38:48.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:38:48.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: from='mgr.24399 172.21.15.67:0/104270866' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "mgr fail", "who": "smithi067.dwichg"}]: dispatch 2024-04-12T05:38:48.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "mgr fail", "who": "smithi067.dwichg"}]: dispatch 2024-04-12T05:38:48.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: osdmap e48: 6 total, 6 up, 6 in 2024-04-12T05:38:48.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: from='mgr.24399 ' entity='mgr.smithi067.dwichg' cmd='[{"prefix": "mgr fail", "who": "smithi067.dwichg"}]': finished 2024-04-12T05:38:48.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: mgrmap e28: smithi032.hrwxrh(active, starting, since 0.005627s) 2024-04-12T05:38:48.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-12T05:38:48.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-12T05:38:48.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.pndtry"}]: dispatch 2024-04-12T05:38:48.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.jehaaj"}]: dispatch 2024-04-12T05:38:48.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.zesmgy"}]: dispatch 2024-04-12T05:38:48.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.ymvqih"}]: dispatch 2024-04-12T05:38:48.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr metadata", "who": "smithi032.hrwxrh", "id": "smithi032.hrwxrh"}]: dispatch 2024-04-12T05:38:48.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T05:38:48.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T05:38:48.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T05:38:48.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:38:48.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:38:48.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:38:48.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:38:48.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:38:48.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:38:48.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: Manager daemon smithi032.hrwxrh is now available 2024-04-12T05:38:48.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:38:48.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:38:48.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.hrwxrh/mirror_snapshot_schedule"}]: dispatch 2024-04-12T05:38:48.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.hrwxrh/trash_purge_schedule"}]: dispatch 2024-04-12T05:38:48.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: [12/Apr/2024:05:38:47] ENGINE Bus STARTING 2024-04-12T05:38:48.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: [12/Apr/2024:05:38:47] ENGINE Serving on http://172.21.15.32:8765 2024-04-12T05:38:48.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: [12/Apr/2024:05:38:47] ENGINE Serving on https://172.21.15.32:7150 2024-04-12T05:38:48.343 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:47 smithi032 ceph-mon[39731]: [12/Apr/2024:05:38:47] ENGINE Bus STARTED 2024-04-12T05:38:48.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.540+0000 7fa51a376700 1 -- 172.21.15.32:0/1762057081 --> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa51410f760 con 0x7fa4fc038230 2024-04-12T05:38:48.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.542+0000 7fa510ff9700 1 -- 172.21.15.32:0/1762057081 <== mgr.24463 v2:172.21.15.32:6800/2564765508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7fa51410f760 con 0x7fa4fc038230 2024-04-12T05:38:48.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.545+0000 7fa51a376700 1 -- 172.21.15.32:0/1762057081 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fa4fc038230 msgr2=0x7fa4fc03a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:48.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.545+0000 7fa51a376700 1 --2- 172.21.15.32:0/1762057081 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fa4fc038230 0x7fa4fc03a6e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa504004890 tx=0x7fa504009e30 comp rx=0 tx=0).stop 2024-04-12T05:38:48.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.545+0000 7fa51a376700 1 -- 172.21.15.32:0/1762057081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa514100010 msgr2=0x7fa51410de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:48.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.545+0000 7fa51a376700 1 --2- 172.21.15.32:0/1762057081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa514100010 0x7fa51410de40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa508009fd0 tx=0x7fa50800b040 comp rx=0 tx=0).stop 2024-04-12T05:38:48.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.545+0000 7fa51a376700 1 -- 172.21.15.32:0/1762057081 shutdown_connections 2024-04-12T05:38:48.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.545+0000 7fa51a376700 1 --2- 172.21.15.32:0/1762057081 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fa4fc038230 0x7fa4fc03a6e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:48.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.545+0000 7fa51a376700 1 --2- 172.21.15.32:0/1762057081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa514066af0 0x7fa51410d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:48.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.545+0000 7fa51a376700 1 --2- 172.21.15.32:0/1762057081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa514100010 0x7fa51410de40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:48.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.545+0000 7fa51a376700 1 -- 172.21.15.32:0/1762057081 >> 172.21.15.32:0/1762057081 conn(0x7fa5140f74e0 msgr2=0x7fa5140f78b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:48.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.545+0000 7fa51a376700 1 -- 172.21.15.32:0/1762057081 shutdown_connections 2024-04-12T05:38:48.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.545+0000 7fa51a376700 1 -- 172.21.15.32:0/1762057081 wait complete. 2024-04-12T05:38:48.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.695+0000 7fc8c0530700 1 -- 172.21.15.32:0/2845418378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8b8103450 msgr2=0x7fc8b8103860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:48.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.695+0000 7fc8c0530700 1 --2- 172.21.15.32:0/2845418378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8b8103450 0x7fc8b8103860 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc8b0004c20 tx=0x7fc8b0009ca0 comp rx=0 tx=0).stop 2024-04-12T05:38:48.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.697+0000 7fc8c0530700 1 -- 172.21.15.32:0/2845418378 shutdown_connections 2024-04-12T05:38:48.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.697+0000 7fc8c0530700 1 --2- 172.21.15.32:0/2845418378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8b81046b0 0x7fc8b8104b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:48.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.697+0000 7fc8c0530700 1 --2- 172.21.15.32:0/2845418378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8b8103450 0x7fc8b8103860 secure :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc8b0004c20 tx=0x7fc8b0009ca0 comp rx=0 tx=0).stop 2024-04-12T05:38:48.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.697+0000 7fc8c0530700 1 -- 172.21.15.32:0/2845418378 >> 172.21.15.32:0/2845418378 conn(0x7fc8b8066cb0 msgr2=0x7fc8b80670d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:48.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.699+0000 7fc8c0530700 1 -- 172.21.15.32:0/2845418378 shutdown_connections 2024-04-12T05:38:48.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.699+0000 7fc8c0530700 1 -- 172.21.15.32:0/2845418378 wait complete. 2024-04-12T05:38:48.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.700+0000 7fc8c0530700 1 Processor -- start 2024-04-12T05:38:48.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.700+0000 7fc8c0530700 1 -- start start 2024-04-12T05:38:48.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.700+0000 7fc8c0530700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8b81046b0 0x7fc8b807d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:48.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.700+0000 7fc8c0530700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8b807da10 0x7fc8b807de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:48.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.700+0000 7fc8c0530700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b807ee20 con 0x7fc8b807da10 2024-04-12T05:38:48.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.700+0000 7fc8c0530700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b81221b0 con 0x7fc8b81046b0 2024-04-12T05:38:48.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.700+0000 7fc8bdacb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8b807da10 0x7fc8b807de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:48.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.700+0000 7fc8bdacb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8b807da10 0x7fc8b807de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57948/0 (socket says 172.21.15.32:57948) 2024-04-12T05:38:48.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.700+0000 7fc8bdacb700 1 -- 172.21.15.32:0/713548775 learned_addr learned my addr 172.21.15.32:0/713548775 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:38:48.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.701+0000 7fc8bdacb700 1 -- 172.21.15.32:0/713548775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8b81046b0 msgr2=0x7fc8b807d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:48.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.701+0000 7fc8bdacb700 1 --2- 172.21.15.32:0/713548775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8b81046b0 0x7fc8b807d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:48.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.701+0000 7fc8bdacb700 1 -- 172.21.15.32:0/713548775 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b00048f0 con 0x7fc8b807da10 2024-04-12T05:38:48.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.701+0000 7fc8bdacb700 1 --2- 172.21.15.32:0/713548775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8b807da10 0x7fc8b807de80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc8a8002a50 tx=0x7fc8a8004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:48.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.701+0000 7fc8aeffd700 1 -- 172.21.15.32:0/713548775 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8a8007cc0 con 0x7fc8b807da10 2024-04-12T05:38:48.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.702+0000 7fc8c0530700 1 -- 172.21.15.32:0/713548775 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b8067a40 con 0x7fc8b807da10 2024-04-12T05:38:48.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.702+0000 7fc8aeffd700 1 -- 172.21.15.32:0/713548775 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc8a800f040 con 0x7fc8b807da10 2024-04-12T05:38:48.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.702+0000 7fc8c0530700 1 -- 172.21.15.32:0/713548775 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b8068000 con 0x7fc8b807da10 2024-04-12T05:38:48.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.703+0000 7fc8aeffd700 1 -- 172.21.15.32:0/713548775 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8a800daa0 con 0x7fc8b807da10 2024-04-12T05:38:48.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.704+0000 7fc8c0530700 1 -- 172.21.15.32:0/713548775 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8b804c710 con 0x7fc8b807da10 2024-04-12T05:38:48.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.704+0000 7fc8aeffd700 1 -- 172.21.15.32:0/713548775 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 29) v1 ==== 45330+0+0 (secure 0 0 0) 0x7fc8a8009760 con 0x7fc8b807da10 2024-04-12T05:38:48.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.705+0000 7fc8aeffd700 1 --2- 172.21.15.32:0/713548775 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fc8a4038220 0x7fc8a403a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:48.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.705+0000 7fc8aeffd700 1 -- 172.21.15.32:0/713548775 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7fc8a804b160 con 0x7fc8b807da10 2024-04-12T05:38:48.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.706+0000 7fc8be2cc700 1 --2- 172.21.15.32:0/713548775 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fc8a4038220 0x7fc8a403a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:48.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.708+0000 7fc8be2cc700 1 --2- 172.21.15.32:0/713548775 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fc8a4038220 0x7fc8a403a6d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc8b0009420 tx=0x7fc8b0009ca0 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:48.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.713+0000 7fc8aeffd700 1 -- 172.21.15.32:0/713548775 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fc8a8019490 con 0x7fc8b807da10 2024-04-12T05:38:48.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.959+0000 7fc8c0530700 1 -- 172.21.15.32:0/713548775 --> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8b81088f0 con 0x7fc8a4038220 2024-04-12T05:38:48.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.971+0000 7fc8aeffd700 1 -- 172.21.15.32:0/713548775 <== mgr.24463 v2:172.21.15.32:6800/2564765508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fc8b81088f0 con 0x7fc8a4038220 2024-04-12T05:38:48.968 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:38:48.969 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (39s) 7s ago 10m 19.2M - 0.25.0 c8568f914cd2 fe43361ddb75 2024-04-12T05:38:48.969 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (10m) 7s ago 10m 6656k - 17.2.7-827-gd3b5060a 21de9b7369ad a9cfc00c311b 2024-04-12T05:38:48.969 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (9m) 12s ago 9m 6651k - 17.2.7-827-gd3b5060a 21de9b7369ad 1f2eea4542d9 2024-04-12T05:38:48.969 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (34s) 7s ago 9m 75.7M - 9.4.7 954c08fa6188 2229cad57cd3 2024-04-12T05:38:48.969 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (4m) 7s ago 4m 83.7M - 17.2.7-827-gd3b5060a 21de9b7369ad e7ada8e5259c 2024-04-12T05:38:48.969 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (4m) 7s ago 4m 135M - 17.2.7-827-gd3b5060a 21de9b7369ad 55124da9afdb 2024-04-12T05:38:48.969 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (4m) 12s ago 4m 69.1M - 17.2.7-827-gd3b5060a 21de9b7369ad 0c295d31a4a6 2024-04-12T05:38:48.969 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (4m) 12s ago 4m 136M - 17.2.7-827-gd3b5060a 21de9b7369ad 6eecd7deaac2 2024-04-12T05:38:48.969 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (16s) 7s ago 11m 73.4M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:38:48.969 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283 running (2m) 12s ago 9m 497M - 18.2.2-1187-gd540ebac bb2c5eac26a4 b190c0307364 2024-04-12T05:38:48.969 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (11m) 7s ago 11m 50.1M 2048M 17.2.7-827-gd3b5060a 21de9b7369ad 75aad95b984d 2024-04-12T05:38:48.969 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (9m) 12s ago 8m 40.3M 2048M 17.2.7-827-gd3b5060a 21de9b7369ad 157be8a8c6d8 2024-04-12T05:38:48.969 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (45s) 7s ago 10m 7956k - 1.5.0 0da6a335fe13 f81b71b9b890 2024-04-12T05:38:48.969 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (25s) 12s ago 9m 7977k - 1.5.0 0da6a335fe13 7254c5121e36 2024-04-12T05:38:48.970 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (8m) 7s ago 8m 209M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad 97446e42ceba 2024-04-12T05:38:48.970 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (7m) 7s ago 7m 226M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad bf9f9a2db2d5 2024-04-12T05:38:48.970 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (7m) 7s ago 7m 183M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad f055cf1ca665 2024-04-12T05:38:48.970 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (6m) 12s ago 6m 243M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad 61b851235351 2024-04-12T05:38:48.970 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (6m) 12s ago 6m 223M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad 2c84d9047e9c 2024-04-12T05:38:48.970 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (5m) 12s ago 5m 214M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad e54839bb6055 2024-04-12T05:38:48.970 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (27s) 7s ago 9m 45.0M - 2.43.0 a07b618ecd1d 9ddf7c894c2a 2024-04-12T05:38:48.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.973+0000 7fc8c0530700 1 -- 172.21.15.32:0/713548775 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fc8a4038220 msgr2=0x7fc8a403a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:48.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.973+0000 7fc8c0530700 1 --2- 172.21.15.32:0/713548775 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fc8a4038220 0x7fc8a403a6d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc8b0009420 tx=0x7fc8b0009ca0 comp rx=0 tx=0).stop 2024-04-12T05:38:48.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.973+0000 7fc8c0530700 1 -- 172.21.15.32:0/713548775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8b807da10 msgr2=0x7fc8b807de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:48.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.973+0000 7fc8c0530700 1 --2- 172.21.15.32:0/713548775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8b807da10 0x7fc8b807de80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc8a8002a50 tx=0x7fc8a8004630 comp rx=0 tx=0).stop 2024-04-12T05:38:48.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.974+0000 7fc8c0530700 1 -- 172.21.15.32:0/713548775 shutdown_connections 2024-04-12T05:38:48.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.974+0000 7fc8c0530700 1 --2- 172.21.15.32:0/713548775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8b81046b0 0x7fc8b807d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:48.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.974+0000 7fc8c0530700 1 --2- 172.21.15.32:0/713548775 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fc8a4038220 0x7fc8a403a6d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:48.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.974+0000 7fc8c0530700 1 --2- 172.21.15.32:0/713548775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8b807da10 0x7fc8b807de80 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:48.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.974+0000 7fc8c0530700 1 -- 172.21.15.32:0/713548775 >> 172.21.15.32:0/713548775 conn(0x7fc8b8066cb0 msgr2=0x7fc8b8074150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:48.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.974+0000 7fc8c0530700 1 -- 172.21.15.32:0/713548775 shutdown_connections 2024-04-12T05:38:48.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:48.974+0000 7fc8c0530700 1 -- 172.21.15.32:0/713548775 wait complete. 2024-04-12T05:38:49.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.111+0000 7f0755e4f700 1 -- 172.21.15.32:0/3773346236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07500fcfb0 msgr2=0x7f07500fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:49.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.111+0000 7f0755e4f700 1 --2- 172.21.15.32:0/3773346236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07500fcfb0 0x7f07500fd420 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0744004940 tx=0x7f0744009e30 comp rx=0 tx=0).stop 2024-04-12T05:38:49.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.113+0000 7f0755e4f700 1 -- 172.21.15.32:0/3773346236 shutdown_connections 2024-04-12T05:38:49.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.113+0000 7f0755e4f700 1 --2- 172.21.15.32:0/3773346236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07500fcfb0 0x7f07500fd420 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:49.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.113+0000 7f0755e4f700 1 --2- 172.21.15.32:0/3773346236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07500fbd00 0x7f07500fc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:49.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.113+0000 7f0755e4f700 1 -- 172.21.15.32:0/3773346236 >> 172.21.15.32:0/3773346236 conn(0x7f07500f7480 msgr2=0x7f07500f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:49.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.113+0000 7f0755e4f700 1 -- 172.21.15.32:0/3773346236 shutdown_connections 2024-04-12T05:38:49.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.113+0000 7f0755e4f700 1 -- 172.21.15.32:0/3773346236 wait complete. 2024-04-12T05:38:49.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.113+0000 7f0755e4f700 1 Processor -- start 2024-04-12T05:38:49.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.114+0000 7f0755e4f700 1 -- start start 2024-04-12T05:38:49.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.114+0000 7f0755e4f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07500fbd00 0x7f0750113de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:49.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.114+0000 7f0755e4f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07500fcfb0 0x7f0750114320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:49.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.114+0000 7f0755e4f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0750114970 con 0x7f07500fbd00 2024-04-12T05:38:49.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.114+0000 7f0755e4f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0750119310 con 0x7f07500fcfb0 2024-04-12T05:38:49.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.114+0000 7f074f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07500fbd00 0x7f0750113de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:49.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.114+0000 7f074f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07500fbd00 0x7f0750113de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57954/0 (socket says 172.21.15.32:57954) 2024-04-12T05:38:49.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.114+0000 7f074f7fe700 1 -- 172.21.15.32:0/2350501045 learned_addr learned my addr 172.21.15.32:0/2350501045 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:38:49.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.114+0000 7f074effd700 1 --2- 172.21.15.32:0/2350501045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07500fcfb0 0x7f0750114320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:49.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.114+0000 7f074f7fe700 1 -- 172.21.15.32:0/2350501045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07500fcfb0 msgr2=0x7f0750114320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:49.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.114+0000 7f074f7fe700 1 --2- 172.21.15.32:0/2350501045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07500fcfb0 0x7f0750114320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:49.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.114+0000 7f074f7fe700 1 -- 172.21.15.32:0/2350501045 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0744005040 con 0x7f07500fbd00 2024-04-12T05:38:49.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.114+0000 7f074effd700 1 --2- 172.21.15.32:0/2350501045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07500fcfb0 0x7f0750114320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T05:38:49.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.115+0000 7f074f7fe700 1 --2- 172.21.15.32:0/2350501045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07500fbd00 0x7f0750113de0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f074000eb40 tx=0x7f074000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:49.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.115+0000 7f073ffff700 1 -- 172.21.15.32:0/2350501045 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0740007cc0 con 0x7f07500fbd00 2024-04-12T05:38:49.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.115+0000 7f073ffff700 1 -- 172.21.15.32:0/2350501045 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f074000f040 con 0x7f07500fbd00 2024-04-12T05:38:49.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.115+0000 7f0755e4f700 1 -- 172.21.15.32:0/2350501045 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0750119450 con 0x7f07500fbd00 2024-04-12T05:38:49.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.115+0000 7f0755e4f700 1 -- 172.21.15.32:0/2350501045 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0750119a10 con 0x7f07500fbd00 2024-04-12T05:38:49.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.116+0000 7f073ffff700 1 -- 172.21.15.32:0/2350501045 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f074000c930 con 0x7f07500fbd00 2024-04-12T05:38:49.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.117+0000 7f073ffff700 1 -- 172.21.15.32:0/2350501045 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 30) v1 ==== 45624+0+0 (secure 0 0 0) 0x7f074000cba0 con 0x7f07500fbd00 2024-04-12T05:38:49.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.117+0000 7f073ffff700 1 --2- 172.21.15.32:0/2350501045 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f07380385b0 0x7f073803aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:49.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.117+0000 7f073ffff700 1 -- 172.21.15.32:0/2350501045 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f074004cb70 con 0x7f07500fbd00 2024-04-12T05:38:49.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.117+0000 7f074effd700 1 --2- 172.21.15.32:0/2350501045 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f07380385b0 0x7f073803aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:49.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.117+0000 7f0755e4f700 1 -- 172.21.15.32:0/2350501045 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f075004c710 con 0x7f07500fbd00 2024-04-12T05:38:49.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.118+0000 7f074effd700 1 --2- 172.21.15.32:0/2350501045 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f07380385b0 0x7f073803aa60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f074400b0d0 tx=0x7f0744009e30 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:49.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.122+0000 7f073ffff700 1 -- 172.21.15.32:0/2350501045 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f0740019070 con 0x7f07500fbd00 2024-04-12T05:38:49.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:49 smithi032 ceph-mon[39731]: mgrmap e29: smithi032.hrwxrh(active, since 1.00793s) 2024-04-12T05:38:49.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:49 smithi032 ceph-mon[39731]: from='client.14678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:38:49.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:49 smithi032 ceph-mon[39731]: pgmap v3: 65 pgs: 65 active+clean; 275 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:38:49.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:49 smithi032 ceph-mon[39731]: from='client.24489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:38:49.352 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:49 smithi067 ceph-mon[49010]: mgrmap e29: smithi032.hrwxrh(active, since 1.00793s) 2024-04-12T05:38:49.352 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:49 smithi067 ceph-mon[49010]: from='client.14678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:38:49.352 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:49 smithi067 ceph-mon[49010]: pgmap v3: 65 pgs: 65 active+clean; 275 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:38:49.352 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:49 smithi067 ceph-mon[49010]: from='client.24489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:38:49.353 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-04-12T05:38:49.353 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2024-04-12T05:38:49.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.485+0000 7f0755e4f700 1 -- 172.21.15.32:0/2350501045 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0750119e10 con 0x7f07500fbd00 2024-04-12T05:38:49.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.486+0000 7f073ffff700 1 -- 172.21.15.32:0/2350501045 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7f074001b2c0 con 0x7f07500fbd00 2024-04-12T05:38:49.483 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:38:49.483 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:38:49.483 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 2 2024-04-12T05:38:49.483 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:38:49.483 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:38:49.483 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 1 2024-04-12T05:38:49.483 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:38:49.483 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:38:49.483 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 6 2024-04-12T05:38:49.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:38:49.484 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:38:49.484 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 4 2024-04-12T05:38:49.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:38:49.484 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:38:49.484 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 12, 2024-04-12T05:38:49.484 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 1 2024-04-12T05:38:49.484 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:38:49.484 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:38:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.488+0000 7f0755e4f700 1 -- 172.21.15.32:0/2350501045 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f07380385b0 msgr2=0x7f073803aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.488+0000 7f0755e4f700 1 --2- 172.21.15.32:0/2350501045 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f07380385b0 0x7f073803aa60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f074400b0d0 tx=0x7f0744009e30 comp rx=0 tx=0).stop 2024-04-12T05:38:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.488+0000 7f0755e4f700 1 -- 172.21.15.32:0/2350501045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07500fbd00 msgr2=0x7f0750113de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.488+0000 7f0755e4f700 1 --2- 172.21.15.32:0/2350501045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07500fbd00 0x7f0750113de0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f074000eb40 tx=0x7f074000b040 comp rx=0 tx=0).stop 2024-04-12T05:38:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.488+0000 7f0755e4f700 1 -- 172.21.15.32:0/2350501045 shutdown_connections 2024-04-12T05:38:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.488+0000 7f0755e4f700 1 --2- 172.21.15.32:0/2350501045 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f07380385b0 0x7f073803aa60 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.488+0000 7f0755e4f700 1 --2- 172.21.15.32:0/2350501045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07500fbd00 0x7f0750113de0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.488+0000 7f0755e4f700 1 --2- 172.21.15.32:0/2350501045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07500fcfb0 0x7f0750114320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.488+0000 7f0755e4f700 1 -- 172.21.15.32:0/2350501045 >> 172.21.15.32:0/2350501045 conn(0x7f07500f7480 msgr2=0x7f07500f8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.488+0000 7f0755e4f700 1 -- 172.21.15.32:0/2350501045 shutdown_connections 2024-04-12T05:38:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.488+0000 7f0755e4f700 1 -- 172.21.15.32:0/2350501045 wait complete. 2024-04-12T05:38:49.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.616+0000 7f748089b700 1 -- 172.21.15.32:0/3705437254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f747c073170 msgr2=0x7f747c073b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:49.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.616+0000 7f748089b700 1 --2- 172.21.15.32:0/3705437254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f747c073170 0x7f747c073b80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f746c004940 tx=0x7f746c009e30 comp rx=0 tx=0).stop 2024-04-12T05:38:49.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.616+0000 7f748089b700 1 -- 172.21.15.32:0/3705437254 shutdown_connections 2024-04-12T05:38:49.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.616+0000 7f748089b700 1 --2- 172.21.15.32:0/3705437254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f747c073170 0x7f747c073b80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:49.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.616+0000 7f748089b700 1 --2- 172.21.15.32:0/3705437254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f747c072820 0x7f747c072c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:49.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.616+0000 7f748089b700 1 -- 172.21.15.32:0/3705437254 >> 172.21.15.32:0/3705437254 conn(0x7f747c0fa950 msgr2=0x7f747c0fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:49.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.617+0000 7f748089b700 1 -- 172.21.15.32:0/3705437254 shutdown_connections 2024-04-12T05:38:49.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.617+0000 7f748089b700 1 -- 172.21.15.32:0/3705437254 wait complete. 2024-04-12T05:38:49.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.618+0000 7f748089b700 1 Processor -- start 2024-04-12T05:38:49.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.618+0000 7f748089b700 1 -- start start 2024-04-12T05:38:49.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.618+0000 7f748089b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f747c072820 0x7f747c11cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:49.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.618+0000 7f748089b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f747c11d110 0x7f747c11daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:49.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.618+0000 7f748089b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c11d6d0 con 0x7f747c11d110 2024-04-12T05:38:49.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.618+0000 7f748089b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c11e030 con 0x7f747c072820 2024-04-12T05:38:49.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.618+0000 7f747a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f747c072820 0x7f747c11cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:49.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.618+0000 7f747a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f747c072820 0x7f747c11cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.32:34364/0 (socket says 172.21.15.32:34364) 2024-04-12T05:38:49.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.618+0000 7f747a59c700 1 -- 172.21.15.32:0/1917513748 learned_addr learned my addr 172.21.15.32:0/1917513748 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:38:49.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.618+0000 7f7479d9b700 1 --2- 172.21.15.32:0/1917513748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f747c11d110 0x7f747c11daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:49.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.620+0000 7f747a59c700 1 -- 172.21.15.32:0/1917513748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f747c11d110 msgr2=0x7f747c11daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:49.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.620+0000 7f747a59c700 1 --2- 172.21.15.32:0/1917513748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f747c11d110 0x7f747c11daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:49.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.620+0000 7f747a59c700 1 -- 172.21.15.32:0/1917513748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f746c005040 con 0x7f747c072820 2024-04-12T05:38:49.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.620+0000 7f747a59c700 1 --2- 172.21.15.32:0/1917513748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f747c072820 0x7f747c11cbd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f747000b790 tx=0x7f7470009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:49.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.623+0000 7f746affd700 1 -- 172.21.15.32:0/1917513748 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f747000f6f0 con 0x7f747c072820 2024-04-12T05:38:49.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.623+0000 7f746affd700 1 -- 172.21.15.32:0/1917513748 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f747000fd00 con 0x7f747c072820 2024-04-12T05:38:49.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.623+0000 7f746affd700 1 -- 172.21.15.32:0/1917513748 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7470011e90 con 0x7f747c072820 2024-04-12T05:38:49.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.623+0000 7f748089b700 1 -- 172.21.15.32:0/1917513748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f747c11e2c0 con 0x7f747c072820 2024-04-12T05:38:49.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.624+0000 7f748089b700 1 -- 172.21.15.32:0/1917513748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f747c11e7b0 con 0x7f747c072820 2024-04-12T05:38:49.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.625+0000 7f748089b700 1 -- 172.21.15.32:0/1917513748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f747c05f5f0 con 0x7f747c072820 2024-04-12T05:38:49.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.625+0000 7f746affd700 1 -- 172.21.15.32:0/1917513748 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) v1 ==== 45624+0+0 (secure 0 0 0) 0x7f747000f850 con 0x7f747c072820 2024-04-12T05:38:49.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.626+0000 7f746affd700 1 --2- 172.21.15.32:0/1917513748 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f7464038600 0x7f746403aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:38:49.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.626+0000 7f746affd700 1 -- 172.21.15.32:0/1917513748 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f747004c820 con 0x7f747c072820 2024-04-12T05:38:49.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.626+0000 7f7479d9b700 1 --2- 172.21.15.32:0/1917513748 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f7464038600 0x7f746403aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:49.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.627+0000 7f7479d9b700 1 --2- 172.21.15.32:0/1917513748 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f7464038600 0x7f746403aab0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f746c009fd0 tx=0x7f746c009e30 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:49.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.638+0000 7f746affd700 1 -- 172.21.15.32:0/1917513748 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f7470011690 con 0x7f747c072820 2024-04-12T05:38:49.727 DEBUG:teuthology.parallel:result is None 2024-04-12T05:38:49.727 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-12T05:38:49.766 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-12T05:38:49.766 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1 2024-04-12T05:38:49.802 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.1/client.1 2024-04-12T05:38:49.802 DEBUG:teuthology.parallel:result is None 2024-04-12T05:38:49.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.899+0000 7f748089b700 1 -- 172.21.15.32:0/1917513748 --> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f747c0731e0 con 0x7f7464038600 2024-04-12T05:38:49.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.900+0000 7f746affd700 1 -- 172.21.15.32:0/1917513748 <== mgr.24463 v2:172.21.15.32:6800/2564765508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7f747c0731e0 con 0x7f7464038600 2024-04-12T05:38:49.897 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:38:49.897 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:38:49.897 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:38:49.897 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-12T05:38:49.897 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:38:49.898 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr" 2024-04-12T05:38:49.898 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:38:49.898 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "2/2 daemons upgraded", 2024-04-12T05:38:49.898 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "", 2024-04-12T05:38:49.898 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:38:49.898 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:38:49.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.902+0000 7f748089b700 1 -- 172.21.15.32:0/1917513748 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f7464038600 msgr2=0x7f746403aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:49.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.902+0000 7f748089b700 1 --2- 172.21.15.32:0/1917513748 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f7464038600 0x7f746403aab0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f746c009fd0 tx=0x7f746c009e30 comp rx=0 tx=0).stop 2024-04-12T05:38:49.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.902+0000 7f748089b700 1 -- 172.21.15.32:0/1917513748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f747c072820 msgr2=0x7f747c11cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:49.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.902+0000 7f748089b700 1 --2- 172.21.15.32:0/1917513748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f747c072820 0x7f747c11cbd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f747000b790 tx=0x7f7470009220 comp rx=0 tx=0).stop 2024-04-12T05:38:49.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.902+0000 7f748089b700 1 -- 172.21.15.32:0/1917513748 shutdown_connections 2024-04-12T05:38:49.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.902+0000 7f748089b700 1 --2- 172.21.15.32:0/1917513748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f747c072820 0x7f747c11cbd0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:49.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.902+0000 7f748089b700 1 --2- 172.21.15.32:0/1917513748 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f7464038600 0x7f746403aab0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:49.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.902+0000 7f748089b700 1 --2- 172.21.15.32:0/1917513748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f747c11d110 0x7f747c11daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:38:49.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.902+0000 7f748089b700 1 -- 172.21.15.32:0/1917513748 >> 172.21.15.32:0/1917513748 conn(0x7f747c0fa950 msgr2=0x7f747c0fd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:49.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.903+0000 7f748089b700 1 -- 172.21.15.32:0/1917513748 shutdown_connections 2024-04-12T05:38:49.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:38:49.903+0000 7f748089b700 1 -- 172.21.15.32:0/1917513748 wait complete. 2024-04-12T05:38:50.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:50 smithi032 ceph-mon[39731]: from='client.14706 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:38:50.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:50 smithi032 ceph-mon[39731]: mgrmap e30: smithi032.hrwxrh(active, since 2s) 2024-04-12T05:38:50.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:50 smithi032 ceph-mon[39731]: pgmap v4: 65 pgs: 65 active+clean; 275 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:38:50.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:50 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/2350501045' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:38:50.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:50 smithi067 ceph-mon[49010]: from='client.14706 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:38:50.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:50 smithi067 ceph-mon[49010]: mgrmap e30: smithi032.hrwxrh(active, since 2s) 2024-04-12T05:38:50.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:50 smithi067 ceph-mon[49010]: pgmap v4: 65 pgs: 65 active+clean; 275 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:38:50.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:50 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/2350501045' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:38:51.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:51 smithi067 ceph-mon[49010]: from='client.24503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:38:51.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:51 smithi032 ceph-mon[39731]: from='client.24503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:38:52.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:52 smithi032 ceph-mon[39731]: pgmap v5: 65 pgs: 65 active+clean; 275 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:38:52.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:52 smithi032 ceph-mon[39731]: mgrmap e31: smithi032.hrwxrh(active, since 4s) 2024-04-12T05:38:52.394 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:52 smithi067 ceph-mon[49010]: pgmap v5: 65 pgs: 65 active+clean; 275 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:38:52.395 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:52 smithi067 ceph-mon[49010]: mgrmap e31: smithi032.hrwxrh(active, since 4s) 2024-04-12T05:38:55.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:54 smithi067 ceph-mon[49010]: pgmap v6: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 357 KiB/s rd, 323 KiB/s wr, 138 op/s 2024-04-12T05:38:55.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:54 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:38:55.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:54 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:38:55.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:54 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:38:55.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:54 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:38:55.041 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:54 smithi032 ceph-mon[39731]: pgmap v6: 65 pgs: 65 active+clean; 271 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 357 KiB/s rd, 323 KiB/s wr, 138 op/s 2024-04-12T05:38:55.041 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:54 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:38:55.041 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:54 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:38:55.041 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:54 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:38:55.041 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:54 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:38:55.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:55 smithi032 ceph-mon[39731]: Standby manager daemon smithi067.dwichg started 2024-04-12T05:38:55.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:55 smithi032 ceph-mon[39731]: from='mgr.? 172.21.15.67:0/2125100359' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi067.dwichg/crt"}]: dispatch 2024-04-12T05:38:55.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:55 smithi032 ceph-mon[39731]: from='mgr.? 172.21.15.67:0/2125100359' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T05:38:55.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:55 smithi032 ceph-mon[39731]: from='mgr.? 172.21.15.67:0/2125100359' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi067.dwichg/key"}]: dispatch 2024-04-12T05:38:55.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:55 smithi032 ceph-mon[39731]: from='mgr.? 172.21.15.67:0/2125100359' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T05:38:55.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:55 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:38:55.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:55 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:38:56.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:55 smithi067 ceph-mon[49010]: Standby manager daemon smithi067.dwichg started 2024-04-12T05:38:56.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:55 smithi067 ceph-mon[49010]: from='mgr.? 172.21.15.67:0/2125100359' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi067.dwichg/crt"}]: dispatch 2024-04-12T05:38:56.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:55 smithi067 ceph-mon[49010]: from='mgr.? 172.21.15.67:0/2125100359' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T05:38:56.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:55 smithi067 ceph-mon[49010]: from='mgr.? 172.21.15.67:0/2125100359' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi067.dwichg/key"}]: dispatch 2024-04-12T05:38:56.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:55 smithi067 ceph-mon[49010]: from='mgr.? 172.21.15.67:0/2125100359' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T05:38:56.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:55 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:38:56.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:55 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:38:57.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:56 smithi067 ceph-mon[49010]: pgmap v7: 65 pgs: 65 active+clean; 269 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 256 KiB/s rd, 231 KiB/s wr, 138 op/s 2024-04-12T05:38:57.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:56 smithi067 ceph-mon[49010]: mgrmap e32: smithi032.hrwxrh(active, since 8s), standbys: smithi067.dwichg 2024-04-12T05:38:57.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:56 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr metadata", "who": "smithi067.dwichg", "id": "smithi067.dwichg"}]: dispatch 2024-04-12T05:38:57.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:56 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:38:57.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:56 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:38:57.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:56 smithi032 ceph-mon[39731]: pgmap v7: 65 pgs: 65 active+clean; 269 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 256 KiB/s rd, 231 KiB/s wr, 138 op/s 2024-04-12T05:38:57.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:56 smithi032 ceph-mon[39731]: mgrmap e32: smithi032.hrwxrh(active, since 8s), standbys: smithi067.dwichg 2024-04-12T05:38:57.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:56 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr metadata", "who": "smithi067.dwichg", "id": "smithi067.dwichg"}]: dispatch 2024-04-12T05:38:57.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:56 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:38:57.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:56 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:38:58.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:38:58 smithi032 ceph-mon[39731]: pgmap v8: 65 pgs: 65 active+clean; 269 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 199 KiB/s rd, 179 KiB/s wr, 108 op/s 2024-04-12T05:38:59.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:38:58 smithi067 ceph-mon[49010]: pgmap v8: 65 pgs: 65 active+clean; 269 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 199 KiB/s rd, 179 KiB/s wr, 108 op/s 2024-04-12T05:39:01.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:00 smithi067 ceph-mon[49010]: pgmap v9: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 198 KiB/s rd, 182 KiB/s wr, 91 op/s 2024-04-12T05:39:01.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:00 smithi032 ceph-mon[39731]: pgmap v9: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 198 KiB/s rd, 182 KiB/s wr, 91 op/s 2024-04-12T05:39:03.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:02 smithi067 ceph-mon[49010]: pgmap v10: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 182 KiB/s rd, 167 KiB/s wr, 84 op/s 2024-04-12T05:39:03.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:02 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:39:03.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:02 smithi032 ceph-mon[39731]: pgmap v10: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 182 KiB/s rd, 167 KiB/s wr, 84 op/s 2024-04-12T05:39:03.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:02 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:39:04.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:03 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:04.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:03 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:04.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:03 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:39:04.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:03 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:39:04.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:03 smithi067 ceph-mon[49010]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-12T05:39:04.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:03 smithi067 ceph-mon[49010]: Updating smithi067:/etc/ceph/ceph.conf 2024-04-12T05:39:04.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:03 smithi067 ceph-mon[49010]: pgmap v11: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 183 KiB/s rd, 167 KiB/s wr, 86 op/s 2024-04-12T05:39:04.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:03 smithi067 ceph-mon[49010]: Updating smithi067:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:39:04.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:03 smithi067 ceph-mon[49010]: Updating smithi032:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:39:04.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:03 smithi067 ceph-mon[49010]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:39:04.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:03 smithi067 ceph-mon[49010]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:39:04.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:03 smithi067 ceph-mon[49010]: Updating smithi067:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:39:04.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:03 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:04.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:03 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:04.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:03 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:39:04.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:03 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:39:04.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:03 smithi032 ceph-mon[39731]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-12T05:39:04.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:03 smithi032 ceph-mon[39731]: Updating smithi067:/etc/ceph/ceph.conf 2024-04-12T05:39:04.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:03 smithi032 ceph-mon[39731]: pgmap v11: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 183 KiB/s rd, 167 KiB/s wr, 86 op/s 2024-04-12T05:39:04.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:03 smithi032 ceph-mon[39731]: Updating smithi067:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:39:04.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:03 smithi032 ceph-mon[39731]: Updating smithi032:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:39:04.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:03 smithi032 ceph-mon[39731]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:39:04.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:03 smithi032 ceph-mon[39731]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:39:04.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:03 smithi032 ceph-mon[39731]: Updating smithi067:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:39:05.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:05 smithi067 ceph-mon[49010]: Updating smithi032:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:39:05.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:05 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:05.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:05 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:05.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:05 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:05.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:05 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:05.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:05 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:05.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:05 smithi067 ceph-mon[49010]: Reconfiguring prometheus.smithi032 (dependencies changed)... 2024-04-12T05:39:05.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:05 smithi067 ceph-mon[49010]: Reconfiguring daemon prometheus.smithi032 on smithi032 2024-04-12T05:39:05.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:05 smithi032 ceph-mon[39731]: Updating smithi032:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:39:05.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:05 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:05.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:05 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:05.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:05 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:05.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:05 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:05.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:05 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:05.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:05 smithi032 ceph-mon[39731]: Reconfiguring prometheus.smithi032 (dependencies changed)... 2024-04-12T05:39:05.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:05 smithi032 ceph-mon[39731]: Reconfiguring daemon prometheus.smithi032 on smithi032 2024-04-12T05:39:06.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:06 smithi067 ceph-mon[49010]: pgmap v12: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 34 KiB/s rd, 32 KiB/s wr, 28 op/s 2024-04-12T05:39:06.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:06 smithi032 ceph-mon[39731]: pgmap v12: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 34 KiB/s rd, 32 KiB/s wr, 28 op/s 2024-04-12T05:39:08.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:08 smithi067 ceph-mon[49010]: pgmap v13: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 33 KiB/s rd, 32 KiB/s wr, 5 op/s 2024-04-12T05:39:08.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:08 smithi032 ceph-mon[39731]: pgmap v13: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 33 KiB/s rd, 32 KiB/s wr, 5 op/s 2024-04-12T05:39:10.321 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:10 smithi032 ceph-mon[39731]: pgmap v14: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2024-04-12T05:39:10.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:10 smithi067 ceph-mon[49010]: pgmap v14: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2024-04-12T05:39:12.259 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:11 smithi032 ceph-mon[39731]: pgmap v15: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 341 B/s wr, 4 op/s 2024-04-12T05:39:12.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:11 smithi067 ceph-mon[49010]: pgmap v15: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 341 B/s wr, 4 op/s 2024-04-12T05:39:15.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:14 smithi067 ceph-mon[49010]: pgmap v16: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.8 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-12T05:39:15.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:14 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:15.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:14 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:15.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:14 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:39:15.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:14 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:39:15.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:14 smithi032 ceph-mon[39731]: pgmap v16: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.8 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-12T05:39:15.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:14 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:15.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:14 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:15.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:14 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:39:15.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:14 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:39:15.951 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:15 smithi067 ceph-mon[49010]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:39:16.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:15 smithi032 ceph-mon[39731]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:39:17.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:16 smithi067 ceph-mon[49010]: pgmap v17: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 426 B/s wr, 3 op/s 2024-04-12T05:39:17.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:16 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi067.dwichg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:39:17.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:16 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:39:17.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:16 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:39:17.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:16 smithi032 ceph-mon[39731]: pgmap v17: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 426 B/s wr, 3 op/s 2024-04-12T05:39:17.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:16 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi067.dwichg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:39:17.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:16 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:39:17.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:16 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:39:18.154 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:17 smithi067 ceph-mon[49010]: Upgrade: Updating mgr.smithi067.dwichg 2024-04-12T05:39:18.155 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:17 smithi067 ceph-mon[49010]: Deploying daemon mgr.smithi067.dwichg on smithi067 2024-04-12T05:39:18.155 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:17 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:39:18.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:17 smithi032 ceph-mon[39731]: Upgrade: Updating mgr.smithi067.dwichg 2024-04-12T05:39:18.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:17 smithi032 ceph-mon[39731]: Deploying daemon mgr.smithi067.dwichg on smithi067 2024-04-12T05:39:18.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:17 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:39:19.240 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:18 smithi067 ceph-mon[49010]: pgmap v18: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 426 B/s wr, 3 op/s 2024-04-12T05:39:19.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:18 smithi032 ceph-mon[39731]: pgmap v18: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 426 B/s wr, 3 op/s 2024-04-12T05:39:20.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.010+0000 7f0adb0fd700 1 -- 172.21.15.32:0/3899246590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ad40fbcd0 msgr2=0x7f0ad40fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:20.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.010+0000 7f0adb0fd700 1 --2- 172.21.15.32:0/3899246590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ad40fbcd0 0x7f0ad40fc0e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8004940 tx=0x7f0ac8009e30 comp rx=0 tx=0).stop 2024-04-12T05:39:20.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.010+0000 7f0adb0fd700 1 -- 172.21.15.32:0/3899246590 shutdown_connections 2024-04-12T05:39:20.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.010+0000 7f0adb0fd700 1 --2- 172.21.15.32:0/3899246590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ad40fcf80 0x7f0ad40fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:20.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.010+0000 7f0adb0fd700 1 --2- 172.21.15.32:0/3899246590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ad40fbcd0 0x7f0ad40fc0e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:20.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.010+0000 7f0adb0fd700 1 -- 172.21.15.32:0/3899246590 >> 172.21.15.32:0/3899246590 conn(0x7f0ad40f7430 msgr2=0x7f0ad40f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:20.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.010+0000 7f0adb0fd700 1 -- 172.21.15.32:0/3899246590 shutdown_connections 2024-04-12T05:39:20.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.010+0000 7f0adb0fd700 1 -- 172.21.15.32:0/3899246590 wait complete. 2024-04-12T05:39:20.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.011+0000 7f0adb0fd700 1 Processor -- start 2024-04-12T05:39:20.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.011+0000 7f0adb0fd700 1 -- start start 2024-04-12T05:39:20.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.011+0000 7f0adb0fd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ad40fbcd0 0x7f0ad41105d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:39:20.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.012+0000 7f0adb0fd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ad40fcf80 0x7f0ad4112b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:39:20.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.012+0000 7f0adb0fd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad4066350 con 0x7f0ad40fbcd0 2024-04-12T05:39:20.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.012+0000 7f0adb0fd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad4066490 con 0x7f0ad40fcf80 2024-04-12T05:39:20.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.012+0000 7f0ad8e99700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ad40fbcd0 0x7f0ad41105d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:20.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.012+0000 7f0ad8e99700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ad40fbcd0 0x7f0ad41105d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44128/0 (socket says 172.21.15.32:44128) 2024-04-12T05:39:20.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.012+0000 7f0ad8e99700 1 -- 172.21.15.32:0/1954193054 learned_addr learned my addr 172.21.15.32:0/1954193054 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:39:20.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.012+0000 7f0ad3fff700 1 --2- 172.21.15.32:0/1954193054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ad40fcf80 0x7f0ad4112b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:20.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.012+0000 7f0ad8e99700 1 -- 172.21.15.32:0/1954193054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ad40fcf80 msgr2=0x7f0ad4112b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:20.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.012+0000 7f0ad8e99700 1 --2- 172.21.15.32:0/1954193054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ad40fcf80 0x7f0ad4112b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:20.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.012+0000 7f0ad8e99700 1 -- 172.21.15.32:0/1954193054 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac8005040 con 0x7f0ad40fbcd0 2024-04-12T05:39:20.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.012+0000 7f0ad8e99700 1 --2- 172.21.15.32:0/1954193054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ad40fbcd0 0x7f0ad41105d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8009380 tx=0x7f0ac8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:20.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.012+0000 7f0ad17fa700 1 -- 172.21.15.32:0/1954193054 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ac800e040 con 0x7f0ad40fbcd0 2024-04-12T05:39:20.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.012+0000 7f0adb0fd700 1 -- 172.21.15.32:0/1954193054 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad4113060 con 0x7f0ad40fbcd0 2024-04-12T05:39:20.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.012+0000 7f0ad17fa700 1 -- 172.21.15.32:0/1954193054 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ac800bd20 con 0x7f0ad40fbcd0 2024-04-12T05:39:20.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.012+0000 7f0ad17fa700 1 -- 172.21.15.32:0/1954193054 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ac801c630 con 0x7f0ad40fbcd0 2024-04-12T05:39:20.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.012+0000 7f0adb0fd700 1 -- 172.21.15.32:0/1954193054 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad4113620 con 0x7f0ad40fbcd0 2024-04-12T05:39:20.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.014+0000 7f0adb0fd700 1 -- 172.21.15.32:0/1954193054 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad405f5f0 con 0x7f0ad40fbcd0 2024-04-12T05:39:20.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.014+0000 7f0ad17fa700 1 -- 172.21.15.32:0/1954193054 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 32) v1 ==== 90805+0+0 (secure 0 0 0) 0x7f0ac8013420 con 0x7f0ad40fbcd0 2024-04-12T05:39:20.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.015+0000 7f0ad17fa700 1 --2- 172.21.15.32:0/1954193054 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f0ac406c9c0 0x7f0ac406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:39:20.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.015+0000 7f0ad17fa700 1 -- 172.21.15.32:0/1954193054 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f0ac808c990 con 0x7f0ad40fbcd0 2024-04-12T05:39:20.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.015+0000 7f0ad3fff700 1 --2- 172.21.15.32:0/1954193054 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f0ac406c9c0 0x7f0ac406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:20.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.015+0000 7f0ad3fff700 1 --2- 172.21.15.32:0/1954193054 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f0ac406c9c0 0x7f0ac406ee70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0009fd0 tx=0x7f0ac000b040 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:20.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.021+0000 7f0ad17fa700 1 -- 172.21.15.32:0/1954193054 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f0ac8056c80 con 0x7f0ad40fbcd0 2024-04-12T05:39:20.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.249+0000 7f0adb0fd700 1 -- 172.21.15.32:0/1954193054 --> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ad41012e0 con 0x7f0ac406c9c0 2024-04-12T05:39:20.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.250+0000 7f0ad17fa700 1 -- 172.21.15.32:0/1954193054 <== mgr.24463 v2:172.21.15.32:6800/2564765508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+321 (secure 0 0 0) 0x7f0ad41012e0 con 0x7f0ac406c9c0 2024-04-12T05:39:20.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.252+0000 7f0adb0fd700 1 -- 172.21.15.32:0/1954193054 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f0ac406c9c0 msgr2=0x7f0ac406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:20.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.252+0000 7f0adb0fd700 1 --2- 172.21.15.32:0/1954193054 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f0ac406c9c0 0x7f0ac406ee70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0009fd0 tx=0x7f0ac000b040 comp rx=0 tx=0).stop 2024-04-12T05:39:20.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.252+0000 7f0adb0fd700 1 -- 172.21.15.32:0/1954193054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ad40fbcd0 msgr2=0x7f0ad41105d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:20.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.252+0000 7f0adb0fd700 1 --2- 172.21.15.32:0/1954193054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ad40fbcd0 0x7f0ad41105d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8009380 tx=0x7f0ac8009e30 comp rx=0 tx=0).stop 2024-04-12T05:39:20.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.252+0000 7f0adb0fd700 1 -- 172.21.15.32:0/1954193054 shutdown_connections 2024-04-12T05:39:20.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.252+0000 7f0adb0fd700 1 --2- 172.21.15.32:0/1954193054 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f0ac406c9c0 0x7f0ac406ee70 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:20.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.252+0000 7f0adb0fd700 1 --2- 172.21.15.32:0/1954193054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ad40fbcd0 0x7f0ad41105d0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:20.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.252+0000 7f0adb0fd700 1 --2- 172.21.15.32:0/1954193054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ad40fcf80 0x7f0ad4112b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:20.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.252+0000 7f0adb0fd700 1 -- 172.21.15.32:0/1954193054 >> 172.21.15.32:0/1954193054 conn(0x7f0ad40f7430 msgr2=0x7f0ad40f8b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:20.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.252+0000 7f0adb0fd700 1 -- 172.21.15.32:0/1954193054 shutdown_connections 2024-04-12T05:39:20.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.252+0000 7f0adb0fd700 1 -- 172.21.15.32:0/1954193054 wait complete. 2024-04-12T05:39:20.260 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:39:20.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.358+0000 7fa192393700 1 -- 172.21.15.32:0/3695011864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa18c0fcf80 msgr2=0x7fa18c0fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:20.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.358+0000 7fa192393700 1 --2- 172.21.15.32:0/3695011864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa18c0fcf80 0x7fa18c0fd3f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa180004940 tx=0x7fa180009e30 comp rx=0 tx=0).stop 2024-04-12T05:39:20.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.358+0000 7fa192393700 1 -- 172.21.15.32:0/3695011864 shutdown_connections 2024-04-12T05:39:20.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.358+0000 7fa192393700 1 --2- 172.21.15.32:0/3695011864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa18c0fcf80 0x7fa18c0fd3f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:20.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.358+0000 7fa192393700 1 --2- 172.21.15.32:0/3695011864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa18c0fbcd0 0x7fa18c0fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:20.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.358+0000 7fa192393700 1 -- 172.21.15.32:0/3695011864 >> 172.21.15.32:0/3695011864 conn(0x7fa18c0f7430 msgr2=0x7fa18c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:20.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.358+0000 7fa192393700 1 -- 172.21.15.32:0/3695011864 shutdown_connections 2024-04-12T05:39:20.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.358+0000 7fa192393700 1 -- 172.21.15.32:0/3695011864 wait complete. 2024-04-12T05:39:20.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.359+0000 7fa192393700 1 Processor -- start 2024-04-12T05:39:20.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.359+0000 7fa192393700 1 -- start start 2024-04-12T05:39:20.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.359+0000 7fa192393700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa18c0fbcd0 0x7fa18c10fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:39:20.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.359+0000 7fa192393700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa18c0fcf80 0x7fa18c110060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:39:20.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.359+0000 7fa192393700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18c110660 con 0x7fa18c0fbcd0 2024-04-12T05:39:20.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.359+0000 7fa192393700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18c110e60 con 0x7fa18c0fcf80 2024-04-12T05:39:20.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.359+0000 7fa18bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa18c0fbcd0 0x7fa18c10fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:20.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.359+0000 7fa18bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa18c0fbcd0 0x7fa18c10fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44138/0 (socket says 172.21.15.32:44138) 2024-04-12T05:39:20.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.359+0000 7fa18bfff700 1 -- 172.21.15.32:0/1600679199 learned_addr learned my addr 172.21.15.32:0/1600679199 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:39:20.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.359+0000 7fa18b7fe700 1 --2- 172.21.15.32:0/1600679199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa18c0fcf80 0x7fa18c110060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:20.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.359+0000 7fa18bfff700 1 -- 172.21.15.32:0/1600679199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa18c0fcf80 msgr2=0x7fa18c110060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:20.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.359+0000 7fa18bfff700 1 --2- 172.21.15.32:0/1600679199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa18c0fcf80 0x7fa18c110060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:20.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.359+0000 7fa18bfff700 1 -- 172.21.15.32:0/1600679199 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17c005040 con 0x7fa18c0fbcd0 2024-04-12T05:39:20.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.360+0000 7fa18bfff700 1 --2- 172.21.15.32:0/1600679199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa18c0fbcd0 0x7fa18c10fb20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa17c0096d0 tx=0x7fa17c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:20.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.360+0000 7fa188ff9700 1 -- 172.21.15.32:0/1600679199 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa17c00e840 con 0x7fa18c0fbcd0 2024-04-12T05:39:20.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.360+0000 7fa188ff9700 1 -- 172.21.15.32:0/1600679199 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa17c00f040 con 0x7fa18c0fbcd0 2024-04-12T05:39:20.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.360+0000 7fa192393700 1 -- 172.21.15.32:0/1600679199 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa180005040 con 0x7fa18c0fbcd0 2024-04-12T05:39:20.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.360+0000 7fa188ff9700 1 -- 172.21.15.32:0/1600679199 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa17c013d30 con 0x7fa18c0fbcd0 2024-04-12T05:39:20.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.360+0000 7fa192393700 1 -- 172.21.15.32:0/1600679199 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa18c111490 con 0x7fa18c0fbcd0 2024-04-12T05:39:20.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.361+0000 7fa192393700 1 -- 172.21.15.32:0/1600679199 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa18c101910 con 0x7fa18c0fbcd0 2024-04-12T05:39:20.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.362+0000 7fa188ff9700 1 -- 172.21.15.32:0/1600679199 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 32) v1 ==== 90805+0+0 (secure 0 0 0) 0x7fa17c00e9a0 con 0x7fa18c0fbcd0 2024-04-12T05:39:20.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.362+0000 7fa188ff9700 1 --2- 172.21.15.32:0/1600679199 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fa17406c9c0 0x7fa17406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:39:20.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.362+0000 7fa188ff9700 1 -- 172.21.15.32:0/1600679199 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7fa17c08c290 con 0x7fa18c0fbcd0 2024-04-12T05:39:20.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.362+0000 7fa18b7fe700 1 --2- 172.21.15.32:0/1600679199 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fa17406c9c0 0x7fa17406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:20.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.363+0000 7fa18b7fe700 1 --2- 172.21.15.32:0/1600679199 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fa17406c9c0 0x7fa17406ee70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa180004890 tx=0x7fa180009e30 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:20.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.366+0000 7fa188ff9700 1 -- 172.21.15.32:0/1600679199 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fa17c059f70 con 0x7fa18c0fbcd0 2024-04-12T05:39:20.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:20 smithi067 ceph-mon[49010]: pgmap v19: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-12T05:39:20.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:20 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:20.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:20 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:20.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:20 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:39:20.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:20 smithi032 ceph-mon[39731]: pgmap v19: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-12T05:39:20.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:20 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:20.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:20 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:20.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:20 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:39:20.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.597+0000 7fa192393700 1 -- 172.21.15.32:0/1600679199 --> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa18c104830 con 0x7fa17406c9c0 2024-04-12T05:39:20.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.598+0000 7fa188ff9700 1 -- 172.21.15.32:0/1600679199 <== mgr.24463 v2:172.21.15.32:6800/2564765508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+321 (secure 0 0 0) 0x7fa1800096f0 con 0x7fa17406c9c0 2024-04-12T05:39:20.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.600+0000 7fa192393700 1 -- 172.21.15.32:0/1600679199 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fa17406c9c0 msgr2=0x7fa17406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:20.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.600+0000 7fa192393700 1 --2- 172.21.15.32:0/1600679199 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fa17406c9c0 0x7fa17406ee70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa180004890 tx=0x7fa180009e30 comp rx=0 tx=0).stop 2024-04-12T05:39:20.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.600+0000 7fa192393700 1 -- 172.21.15.32:0/1600679199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa18c0fbcd0 msgr2=0x7fa18c10fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:20.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.600+0000 7fa192393700 1 --2- 172.21.15.32:0/1600679199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa18c0fbcd0 0x7fa18c10fb20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa17c0096d0 tx=0x7fa17c009e30 comp rx=0 tx=0).stop 2024-04-12T05:39:20.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.600+0000 7fa192393700 1 -- 172.21.15.32:0/1600679199 shutdown_connections 2024-04-12T05:39:20.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.600+0000 7fa192393700 1 --2- 172.21.15.32:0/1600679199 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fa17406c9c0 0x7fa17406ee70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:20.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.600+0000 7fa192393700 1 --2- 172.21.15.32:0/1600679199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa18c0fbcd0 0x7fa18c10fb20 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:20.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.600+0000 7fa192393700 1 --2- 172.21.15.32:0/1600679199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa18c0fcf80 0x7fa18c110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:20.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.600+0000 7fa192393700 1 -- 172.21.15.32:0/1600679199 >> 172.21.15.32:0/1600679199 conn(0x7fa18c0f7430 msgr2=0x7fa18c065c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:20.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.601+0000 7fa192393700 1 -- 172.21.15.32:0/1600679199 shutdown_connections 2024-04-12T05:39:20.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.601+0000 7fa192393700 1 -- 172.21.15.32:0/1600679199 wait complete. 2024-04-12T05:39:20.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.705+0000 7ff12df3b700 1 -- 172.21.15.32:0/2624990849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1280fbcd0 msgr2=0x7ff1280fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:20.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.705+0000 7ff12df3b700 1 --2- 172.21.15.32:0/2624990849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1280fbcd0 0x7ff1280fc0e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ff118004940 tx=0x7ff118009e30 comp rx=0 tx=0).stop 2024-04-12T05:39:20.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.705+0000 7ff12df3b700 1 -- 172.21.15.32:0/2624990849 shutdown_connections 2024-04-12T05:39:20.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.705+0000 7ff12df3b700 1 --2- 172.21.15.32:0/2624990849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1280fcf80 0x7ff1280fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:20.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.705+0000 7ff12df3b700 1 --2- 172.21.15.32:0/2624990849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1280fbcd0 0x7ff1280fc0e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:20.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.705+0000 7ff12df3b700 1 -- 172.21.15.32:0/2624990849 >> 172.21.15.32:0/2624990849 conn(0x7ff1280f7430 msgr2=0x7ff1280f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:20.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.705+0000 7ff12df3b700 1 -- 172.21.15.32:0/2624990849 shutdown_connections 2024-04-12T05:39:20.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.705+0000 7ff12df3b700 1 -- 172.21.15.32:0/2624990849 wait complete. 2024-04-12T05:39:20.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.705+0000 7ff12df3b700 1 Processor -- start 2024-04-12T05:39:20.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.705+0000 7ff12df3b700 1 -- start start 2024-04-12T05:39:20.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.706+0000 7ff12df3b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1280fbcd0 0x7ff1281129d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:39:20.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.706+0000 7ff12df3b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1280fcf80 0x7ff128114f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:39:20.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.706+0000 7ff12df3b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff128066240 con 0x7ff1280fcf80 2024-04-12T05:39:20.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.706+0000 7ff12df3b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff128066380 con 0x7ff1280fbcd0 2024-04-12T05:39:20.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.706+0000 7ff126ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1280fcf80 0x7ff128114f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:20.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.706+0000 7ff126ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1280fcf80 0x7ff128114f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44160/0 (socket says 172.21.15.32:44160) 2024-04-12T05:39:20.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.706+0000 7ff126ffd700 1 -- 172.21.15.32:0/1517085894 learned_addr learned my addr 172.21.15.32:0/1517085894 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:39:20.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.706+0000 7ff1277fe700 1 --2- 172.21.15.32:0/1517085894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1280fbcd0 0x7ff1281129d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:20.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.706+0000 7ff126ffd700 1 -- 172.21.15.32:0/1517085894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1280fbcd0 msgr2=0x7ff1281129d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:20.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.706+0000 7ff126ffd700 1 --2- 172.21.15.32:0/1517085894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1280fbcd0 0x7ff1281129d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:20.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.706+0000 7ff126ffd700 1 -- 172.21.15.32:0/1517085894 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff118005040 con 0x7ff1280fcf80 2024-04-12T05:39:20.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.706+0000 7ff126ffd700 1 --2- 172.21.15.32:0/1517085894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1280fcf80 0x7ff128114f20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff11000eb40 tx=0x7ff11000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:20.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.706+0000 7ff11ffff700 1 -- 172.21.15.32:0/1517085894 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff110007cc0 con 0x7ff1280fcf80 2024-04-12T05:39:20.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.706+0000 7ff12df3b700 1 -- 172.21.15.32:0/1517085894 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff128115460 con 0x7ff1280fcf80 2024-04-12T05:39:20.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.706+0000 7ff11ffff700 1 -- 172.21.15.32:0/1517085894 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff11000f040 con 0x7ff1280fcf80 2024-04-12T05:39:20.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.706+0000 7ff11ffff700 1 -- 172.21.15.32:0/1517085894 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff11000c990 con 0x7ff1280fcf80 2024-04-12T05:39:20.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.706+0000 7ff12df3b700 1 -- 172.21.15.32:0/1517085894 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff128115830 con 0x7ff1280fcf80 2024-04-12T05:39:20.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.708+0000 7ff12df3b700 1 -- 172.21.15.32:0/1517085894 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff12805f5f0 con 0x7ff1280fcf80 2024-04-12T05:39:20.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.708+0000 7ff11ffff700 1 -- 172.21.15.32:0/1517085894 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 32) v1 ==== 90805+0+0 (secure 0 0 0) 0x7ff11000cc00 con 0x7ff1280fcf80 2024-04-12T05:39:20.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.708+0000 7ff11ffff700 1 --2- 172.21.15.32:0/1517085894 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7ff11406c9c0 0x7ff11406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:39:20.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.708+0000 7ff11ffff700 1 -- 172.21.15.32:0/1517085894 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7ff11008b960 con 0x7ff1280fcf80 2024-04-12T05:39:20.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.708+0000 7ff1277fe700 1 --2- 172.21.15.32:0/1517085894 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7ff11406c9c0 0x7ff11406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:20.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.709+0000 7ff1277fe700 1 --2- 172.21.15.32:0/1517085894 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7ff11406c9c0 0x7ff11406ee70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff118013680 tx=0x7ff118009e30 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:20.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.713+0000 7ff11ffff700 1 -- 172.21.15.32:0/1517085894 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7ff110055c80 con 0x7ff1280fcf80 2024-04-12T05:39:20.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.929+0000 7ff12df3b700 1 -- 172.21.15.32:0/1517085894 --> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1281012e0 con 0x7ff11406c9c0 2024-04-12T05:39:20.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.938+0000 7ff11ffff700 1 -- 172.21.15.32:0/1517085894 <== mgr.24463 v2:172.21.15.32:6800/2564765508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7ff1281012e0 con 0x7ff11406c9c0 2024-04-12T05:39:20.937 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:39:20.937 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (71s) 27s ago 10m 19.2M - 0.25.0 c8568f914cd2 fe43361ddb75 2024-04-12T05:39:20.937 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (10m) 27s ago 10m 6656k - 17.2.7-827-gd3b5060a 21de9b7369ad a9cfc00c311b 2024-04-12T05:39:20.937 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (9m) 25s ago 9m 6651k - 17.2.7-827-gd3b5060a 21de9b7369ad 1f2eea4542d9 2024-04-12T05:39:20.937 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (66s) 27s ago 10m 80.0M - 9.4.7 954c08fa6188 2229cad57cd3 2024-04-12T05:39:20.937 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (4m) 27s ago 4m 72.0M - 17.2.7-827-gd3b5060a 21de9b7369ad e7ada8e5259c 2024-04-12T05:39:20.937 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (4m) 27s ago 4m 132M - 17.2.7-827-gd3b5060a 21de9b7369ad 55124da9afdb 2024-04-12T05:39:20.937 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (4m) 25s ago 4m 64.0M - 17.2.7-827-gd3b5060a 21de9b7369ad 0c295d31a4a6 2024-04-12T05:39:20.937 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (4m) 25s ago 4m 122M - 17.2.7-827-gd3b5060a 21de9b7369ad 6eecd7deaac2 2024-04-12T05:39:20.937 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (48s) 27s ago 12m 473M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:39:20.937 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 starting - - - - 2024-04-12T05:39:20.937 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (12m) 27s ago 12m 51.5M 2048M 17.2.7-827-gd3b5060a 21de9b7369ad 75aad95b984d 2024-04-12T05:39:20.937 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (9m) 25s ago 9m 39.5M 2048M 17.2.7-827-gd3b5060a 21de9b7369ad 157be8a8c6d8 2024-04-12T05:39:20.938 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (77s) 27s ago 10m 7558k - 1.5.0 0da6a335fe13 f81b71b9b890 2024-04-12T05:39:20.938 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (57s) 25s ago 9m 8400k - 1.5.0 0da6a335fe13 7254c5121e36 2024-04-12T05:39:20.938 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (8m) 27s ago 8m 218M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad 97446e42ceba 2024-04-12T05:39:20.938 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (8m) 27s ago 8m 233M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad bf9f9a2db2d5 2024-04-12T05:39:20.938 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (7m) 27s ago 7m 187M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad f055cf1ca665 2024-04-12T05:39:20.938 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (7m) 25s ago 7m 255M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad 61b851235351 2024-04-12T05:39:20.938 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (6m) 25s ago 6m 228M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad 2c84d9047e9c 2024-04-12T05:39:20.938 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (6m) 25s ago 6m 218M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad e54839bb6055 2024-04-12T05:39:20.938 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 starting - - - - 2024-04-12T05:39:20.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.941+0000 7ff12df3b700 1 -- 172.21.15.32:0/1517085894 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7ff11406c9c0 msgr2=0x7ff11406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:20.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.941+0000 7ff12df3b700 1 --2- 172.21.15.32:0/1517085894 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7ff11406c9c0 0x7ff11406ee70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff118013680 tx=0x7ff118009e30 comp rx=0 tx=0).stop 2024-04-12T05:39:20.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.941+0000 7ff12df3b700 1 -- 172.21.15.32:0/1517085894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1280fcf80 msgr2=0x7ff128114f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:20.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.941+0000 7ff12df3b700 1 --2- 172.21.15.32:0/1517085894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1280fcf80 0x7ff128114f20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff11000eb40 tx=0x7ff11000b040 comp rx=0 tx=0).stop 2024-04-12T05:39:20.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.941+0000 7ff12df3b700 1 -- 172.21.15.32:0/1517085894 shutdown_connections 2024-04-12T05:39:20.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.941+0000 7ff12df3b700 1 --2- 172.21.15.32:0/1517085894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1280fbcd0 0x7ff1281129d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:20.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.941+0000 7ff12df3b700 1 --2- 172.21.15.32:0/1517085894 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7ff11406c9c0 0x7ff11406ee70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:20.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.941+0000 7ff12df3b700 1 --2- 172.21.15.32:0/1517085894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1280fcf80 0x7ff128114f20 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:20.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.941+0000 7ff12df3b700 1 -- 172.21.15.32:0/1517085894 >> 172.21.15.32:0/1517085894 conn(0x7ff1280f7430 msgr2=0x7ff1280f8b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:20.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.941+0000 7ff12df3b700 1 -- 172.21.15.32:0/1517085894 shutdown_connections 2024-04-12T05:39:20.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:20.941+0000 7ff12df3b700 1 -- 172.21.15.32:0/1517085894 wait complete. 2024-04-12T05:39:21.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.047+0000 7f4fa0b43700 1 -- 172.21.15.32:0/2872049160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f9c0fbd00 msgr2=0x7f4f9c0fc110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:21.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.047+0000 7f4fa0b43700 1 --2- 172.21.15.32:0/2872049160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f9c0fbd00 0x7f4f9c0fc110 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c009ce0 tx=0x7f4f8c004d10 comp rx=0 tx=0).stop 2024-04-12T05:39:21.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.047+0000 7f4fa0b43700 1 -- 172.21.15.32:0/2872049160 shutdown_connections 2024-04-12T05:39:21.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.047+0000 7f4fa0b43700 1 --2- 172.21.15.32:0/2872049160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c0fcfb0 0x7f4f9c0fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:21.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.047+0000 7f4fa0b43700 1 --2- 172.21.15.32:0/2872049160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f9c0fbd00 0x7f4f9c0fc110 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:21.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.047+0000 7f4fa0b43700 1 -- 172.21.15.32:0/2872049160 >> 172.21.15.32:0/2872049160 conn(0x7f4f9c0f7480 msgr2=0x7f4f9c0f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:21.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.047+0000 7f4fa0b43700 1 -- 172.21.15.32:0/2872049160 shutdown_connections 2024-04-12T05:39:21.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.047+0000 7f4fa0b43700 1 -- 172.21.15.32:0/2872049160 wait complete. 2024-04-12T05:39:21.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.048+0000 7f4fa0b43700 1 Processor -- start 2024-04-12T05:39:21.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.048+0000 7f4fa0b43700 1 -- start start 2024-04-12T05:39:21.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.048+0000 7f4fa0b43700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f9c0fbd00 0x7f4f9c10faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:39:21.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.048+0000 7f4fa0b43700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c0fcfb0 0x7f4f9c10ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:39:21.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.048+0000 7f4fa0b43700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c1105e0 con 0x7f4f9c0fbd00 2024-04-12T05:39:21.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.048+0000 7f4fa0b43700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c110750 con 0x7f4f9c0fcfb0 2024-04-12T05:39:21.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.048+0000 7f4f9a59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f9c0fbd00 0x7f4f9c10faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:21.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.048+0000 7f4f9a59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f9c0fbd00 0x7f4f9c10faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44180/0 (socket says 172.21.15.32:44180) 2024-04-12T05:39:21.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.048+0000 7f4f9a59c700 1 -- 172.21.15.32:0/2052501576 learned_addr learned my addr 172.21.15.32:0/2052501576 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:39:21.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.048+0000 7f4f99d9b700 1 --2- 172.21.15.32:0/2052501576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c0fcfb0 0x7f4f9c10ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:21.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.048+0000 7f4f9a59c700 1 -- 172.21.15.32:0/2052501576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c0fcfb0 msgr2=0x7f4f9c10ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:21.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.048+0000 7f4f9a59c700 1 --2- 172.21.15.32:0/2052501576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c0fcfb0 0x7f4f9c10ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:21.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.048+0000 7f4f9a59c700 1 -- 172.21.15.32:0/2052501576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f8c009940 con 0x7f4f9c0fbd00 2024-04-12T05:39:21.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.048+0000 7f4f99d9b700 1 --2- 172.21.15.32:0/2052501576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c0fcfb0 0x7f4f9c10ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T05:39:21.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.048+0000 7f4f9a59c700 1 --2- 172.21.15.32:0/2052501576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f9c0fbd00 0x7f4f9c10faa0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c009310 tx=0x7f4f8c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:21.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.049+0000 7f4f92ffd700 1 -- 172.21.15.32:0/2052501576 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f8c00e040 con 0x7f4f9c0fbd00 2024-04-12T05:39:21.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.049+0000 7f4f92ffd700 1 -- 172.21.15.32:0/2052501576 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f8c01cce0 con 0x7f4f9c0fbd00 2024-04-12T05:39:21.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.049+0000 7f4fa0b43700 1 -- 172.21.15.32:0/2052501576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f9c115150 con 0x7f4f9c0fbd00 2024-04-12T05:39:21.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.049+0000 7f4f92ffd700 1 -- 172.21.15.32:0/2052501576 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f8c012bb0 con 0x7f4f9c0fbd00 2024-04-12T05:39:21.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.049+0000 7f4fa0b43700 1 -- 172.21.15.32:0/2052501576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f9c115710 con 0x7f4f9c0fbd00 2024-04-12T05:39:21.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.050+0000 7f4fa0b43700 1 -- 172.21.15.32:0/2052501576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f9c101940 con 0x7f4f9c0fbd00 2024-04-12T05:39:21.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.050+0000 7f4f92ffd700 1 -- 172.21.15.32:0/2052501576 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 32) v1 ==== 90805+0+0 (secure 0 0 0) 0x7f4f8c012410 con 0x7f4f9c0fbd00 2024-04-12T05:39:21.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.050+0000 7f4f92ffd700 1 --2- 172.21.15.32:0/2052501576 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f4f8806c9d0 0x7f4f8806ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:39:21.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.051+0000 7f4f92ffd700 1 -- 172.21.15.32:0/2052501576 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f4f8c08dff0 con 0x7f4f9c0fbd00 2024-04-12T05:39:21.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.051+0000 7f4f99d9b700 1 --2- 172.21.15.32:0/2052501576 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f4f8806c9d0 0x7f4f8806ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:21.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.051+0000 7f4f99d9b700 1 --2- 172.21.15.32:0/2052501576 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f4f8806c9d0 0x7f4f8806ee80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4f84004fd0 tx=0x7f4f8400b040 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:21.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.055+0000 7f4f92ffd700 1 -- 172.21.15.32:0/2052501576 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f4f8c05be20 con 0x7f4f9c0fbd00 2024-04-12T05:39:21.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.343+0000 7f4fa0b43700 1 -- 172.21.15.32:0/2052501576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4f9c02b2d0 con 0x7f4f9c0fbd00 2024-04-12T05:39:21.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.344+0000 7f4f92ffd700 1 -- 172.21.15.32:0/2052501576 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7f4f8c05b9b0 con 0x7f4f9c0fbd00 2024-04-12T05:39:21.342 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:39:21.342 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:39:21.342 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 2 2024-04-12T05:39:21.342 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:39:21.342 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:39:21.342 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:39:21.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:39:21.343 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:39:21.343 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 6 2024-04-12T05:39:21.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:39:21.343 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:39:21.343 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 4 2024-04-12T05:39:21.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:39:21.343 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:39:21.343 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 12, 2024-04-12T05:39:21.343 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:39:21.343 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:39:21.343 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:39:21.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.346+0000 7f4fa0b43700 1 -- 172.21.15.32:0/2052501576 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f4f8806c9d0 msgr2=0x7f4f8806ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:21.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.346+0000 7f4fa0b43700 1 --2- 172.21.15.32:0/2052501576 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f4f8806c9d0 0x7f4f8806ee80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4f84004fd0 tx=0x7f4f8400b040 comp rx=0 tx=0).stop 2024-04-12T05:39:21.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.346+0000 7f4fa0b43700 1 -- 172.21.15.32:0/2052501576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f9c0fbd00 msgr2=0x7f4f9c10faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:21.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.346+0000 7f4fa0b43700 1 --2- 172.21.15.32:0/2052501576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f9c0fbd00 0x7f4f9c10faa0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c009310 tx=0x7f4f8c004d10 comp rx=0 tx=0).stop 2024-04-12T05:39:21.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.346+0000 7f4fa0b43700 1 -- 172.21.15.32:0/2052501576 shutdown_connections 2024-04-12T05:39:21.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.346+0000 7f4fa0b43700 1 --2- 172.21.15.32:0/2052501576 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f4f8806c9d0 0x7f4f8806ee80 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:21.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.346+0000 7f4fa0b43700 1 --2- 172.21.15.32:0/2052501576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f9c0fbd00 0x7f4f9c10faa0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:21.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.346+0000 7f4fa0b43700 1 --2- 172.21.15.32:0/2052501576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c0fcfb0 0x7f4f9c10ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:21.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.346+0000 7f4fa0b43700 1 -- 172.21.15.32:0/2052501576 >> 172.21.15.32:0/2052501576 conn(0x7f4f9c0f7480 msgr2=0x7f4f9c066910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:21.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.347+0000 7f4fa0b43700 1 -- 172.21.15.32:0/2052501576 shutdown_connections 2024-04-12T05:39:21.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.347+0000 7f4fa0b43700 1 -- 172.21.15.32:0/2052501576 wait complete. 2024-04-12T05:39:21.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.459+0000 7fba4ed86700 1 -- 172.21.15.32:0/207654414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba480fff00 msgr2=0x7fba48102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:21.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.459+0000 7fba4ed86700 1 --2- 172.21.15.32:0/207654414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba480fff00 0x7fba48102320 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fba3c004940 tx=0x7fba3c009e30 comp rx=0 tx=0).stop 2024-04-12T05:39:21.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.459+0000 7fba4ed86700 1 -- 172.21.15.32:0/207654414 shutdown_connections 2024-04-12T05:39:21.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.459+0000 7fba4ed86700 1 --2- 172.21.15.32:0/207654414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba480fff00 0x7fba48102320 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:21.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.459+0000 7fba4ed86700 1 --2- 172.21.15.32:0/207654414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba480662e0 0x7fba480ff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:21.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.459+0000 7fba4ed86700 1 -- 172.21.15.32:0/207654414 >> 172.21.15.32:0/207654414 conn(0x7fba480f7430 msgr2=0x7fba480f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:21.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.459+0000 7fba4ed86700 1 -- 172.21.15.32:0/207654414 shutdown_connections 2024-04-12T05:39:21.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.459+0000 7fba4ed86700 1 -- 172.21.15.32:0/207654414 wait complete. 2024-04-12T05:39:21.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.459+0000 7fba4ed86700 1 Processor -- start 2024-04-12T05:39:21.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.460+0000 7fba4ed86700 1 -- start start 2024-04-12T05:39:21.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.460+0000 7fba4ed86700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba480662e0 0x7fba480fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:39:21.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.460+0000 7fba4ed86700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba480fff00 0x7fba480fb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:39:21.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.460+0000 7fba4ed86700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba480fbe50 con 0x7fba480fff00 2024-04-12T05:39:21.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.460+0000 7fba4ed86700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba480fbf90 con 0x7fba480662e0 2024-04-12T05:39:21.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.460+0000 7fba47fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba480fff00 0x7fba480fb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:21.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.460+0000 7fba47fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba480fff00 0x7fba480fb8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43242/0 (socket says 172.21.15.32:43242) 2024-04-12T05:39:21.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.460+0000 7fba47fff700 1 -- 172.21.15.32:0/1803108022 learned_addr learned my addr 172.21.15.32:0/1803108022 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:39:21.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.460+0000 7fba4cb22700 1 --2- 172.21.15.32:0/1803108022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba480662e0 0x7fba480fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:21.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.460+0000 7fba47fff700 1 -- 172.21.15.32:0/1803108022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba480662e0 msgr2=0x7fba480fd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:21.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.460+0000 7fba47fff700 1 --2- 172.21.15.32:0/1803108022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba480662e0 0x7fba480fd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:21.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.460+0000 7fba47fff700 1 -- 172.21.15.32:0/1803108022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba3c005040 con 0x7fba480fff00 2024-04-12T05:39:21.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.460+0000 7fba47fff700 1 --2- 172.21.15.32:0/1803108022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba480fff00 0x7fba480fb8c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fba3c0137b0 tx=0x7fba3c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:21.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.461+0000 7fba457fa700 1 -- 172.21.15.32:0/1803108022 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba3c00e070 con 0x7fba480fff00 2024-04-12T05:39:21.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.461+0000 7fba457fa700 1 -- 172.21.15.32:0/1803108022 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba3c01c400 con 0x7fba480fff00 2024-04-12T05:39:21.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.461+0000 7fba457fa700 1 -- 172.21.15.32:0/1803108022 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba3c00b650 con 0x7fba480fff00 2024-04-12T05:39:21.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.461+0000 7fba4ed86700 1 -- 172.21.15.32:0/1803108022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba480fc210 con 0x7fba480fff00 2024-04-12T05:39:21.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.461+0000 7fba4ed86700 1 -- 172.21.15.32:0/1803108022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba48115950 con 0x7fba480fff00 2024-04-12T05:39:21.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.462+0000 7fba4ed86700 1 -- 172.21.15.32:0/1803108022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba4805f5f0 con 0x7fba480fff00 2024-04-12T05:39:21.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.462+0000 7fba457fa700 1 -- 172.21.15.32:0/1803108022 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 32) v1 ==== 90805+0+0 (secure 0 0 0) 0x7fba3c01c9e0 con 0x7fba480fff00 2024-04-12T05:39:21.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.463+0000 7fba457fa700 1 --2- 172.21.15.32:0/1803108022 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fba3006c9c0 0x7fba3006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:39:21.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.463+0000 7fba4cb22700 1 --2- 172.21.15.32:0/1803108022 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fba3006c9c0 0x7fba3006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:21.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.463+0000 7fba457fa700 1 -- 172.21.15.32:0/1803108022 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7fba3c08d380 con 0x7fba480fff00 2024-04-12T05:39:21.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.463+0000 7fba4cb22700 1 --2- 172.21.15.32:0/1803108022 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fba3006c9c0 0x7fba3006ee70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fba480fd130 tx=0x7fba4000b040 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:21.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.468+0000 7fba457fa700 1 -- 172.21.15.32:0/1803108022 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fba3c057670 con 0x7fba480fff00 2024-04-12T05:39:21.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:21 smithi067 ceph-mon[49010]: from='client.14722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:21.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:21 smithi067 ceph-mon[49010]: from='client.14726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:21.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:21 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/2052501576' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:21.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.693+0000 7fba4ed86700 1 -- 172.21.15.32:0/1803108022 --> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fba48101b70 con 0x7fba3006c9c0 2024-04-12T05:39:21.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.694+0000 7fba457fa700 1 -- 172.21.15.32:0/1803108022 <== mgr.24463 v2:172.21.15.32:6800/2564765508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+321 (secure 0 0 0) 0x7fba48101b70 con 0x7fba3006c9c0 2024-04-12T05:39:21.692 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:39:21.692 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:39:21.692 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:39:21.692 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-12T05:39:21.692 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [], 2024-04-12T05:39:21.692 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "", 2024-04-12T05:39:21.692 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mgr daemons", 2024-04-12T05:39:21.692 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:39:21.692 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:39:21.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.696+0000 7fba4ed86700 1 -- 172.21.15.32:0/1803108022 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fba3006c9c0 msgr2=0x7fba3006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:21.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.696+0000 7fba4ed86700 1 --2- 172.21.15.32:0/1803108022 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fba3006c9c0 0x7fba3006ee70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fba480fd130 tx=0x7fba4000b040 comp rx=0 tx=0).stop 2024-04-12T05:39:21.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.696+0000 7fba4ed86700 1 -- 172.21.15.32:0/1803108022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba480fff00 msgr2=0x7fba480fb8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:21.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.696+0000 7fba4ed86700 1 --2- 172.21.15.32:0/1803108022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba480fff00 0x7fba480fb8c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fba3c0137b0 tx=0x7fba3c009e30 comp rx=0 tx=0).stop 2024-04-12T05:39:21.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.696+0000 7fba4ed86700 1 -- 172.21.15.32:0/1803108022 shutdown_connections 2024-04-12T05:39:21.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.696+0000 7fba4ed86700 1 --2- 172.21.15.32:0/1803108022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba480662e0 0x7fba480fd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:21.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.696+0000 7fba4ed86700 1 --2- 172.21.15.32:0/1803108022 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fba3006c9c0 0x7fba3006ee70 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:21.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.696+0000 7fba4ed86700 1 --2- 172.21.15.32:0/1803108022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba480fff00 0x7fba480fb8c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:21.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.696+0000 7fba4ed86700 1 -- 172.21.15.32:0/1803108022 >> 172.21.15.32:0/1803108022 conn(0x7fba480f7430 msgr2=0x7fba480f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:21.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.696+0000 7fba4ed86700 1 -- 172.21.15.32:0/1803108022 shutdown_connections 2024-04-12T05:39:21.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:21.696+0000 7fba4ed86700 1 -- 172.21.15.32:0/1803108022 wait complete. 2024-04-12T05:39:21.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:21 smithi032 ceph-mon[39731]: from='client.14722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:21.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:21 smithi032 ceph-mon[39731]: from='client.14726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:21.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:21 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/2052501576' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:22.727 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:22 smithi067 ceph-mon[49010]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:22.727 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:22 smithi067 ceph-mon[49010]: pgmap v20: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.8 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-12T05:39:22.727 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:22 smithi067 ceph-mon[49010]: from='client.14738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:22.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:22 smithi032 ceph-mon[39731]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:22.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:22 smithi032 ceph-mon[39731]: pgmap v20: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.8 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-12T05:39:22.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:22 smithi032 ceph-mon[39731]: from='client.14738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:23.916 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:23 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:23.916 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:23 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:23.916 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:23 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:23.916 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:23 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:24.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:23 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:24.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:23 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:24.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:23 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:24.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:23 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:25.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:24 smithi067 ceph-mon[49010]: pgmap v21: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-12T05:39:25.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:24 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:25.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:24 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:25.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:24 smithi032 ceph-mon[39731]: pgmap v21: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-12T05:39:25.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:24 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:25.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:24 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:27.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:26 smithi032 ceph-mon[39731]: pgmap v22: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 85 B/s wr, 3 op/s 2024-04-12T05:39:27.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:26 smithi032 ceph-mon[39731]: Standby manager daemon smithi067.dwichg restarted 2024-04-12T05:39:27.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:26 smithi032 ceph-mon[39731]: Standby manager daemon smithi067.dwichg started 2024-04-12T05:39:27.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:26 smithi032 ceph-mon[39731]: from='mgr.? 172.21.15.67:0/225113053' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi067.dwichg/crt"}]: dispatch 2024-04-12T05:39:27.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:26 smithi032 ceph-mon[39731]: from='mgr.? 172.21.15.67:0/225113053' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T05:39:27.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:26 smithi032 ceph-mon[39731]: from='mgr.? 172.21.15.67:0/225113053' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi067.dwichg/key"}]: dispatch 2024-04-12T05:39:27.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:26 smithi032 ceph-mon[39731]: from='mgr.? 172.21.15.67:0/225113053' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T05:39:27.225 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:26 smithi067 ceph-mon[49010]: pgmap v22: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 85 B/s wr, 3 op/s 2024-04-12T05:39:27.226 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:26 smithi067 ceph-mon[49010]: Standby manager daemon smithi067.dwichg restarted 2024-04-12T05:39:27.226 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:26 smithi067 ceph-mon[49010]: Standby manager daemon smithi067.dwichg started 2024-04-12T05:39:27.226 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:26 smithi067 ceph-mon[49010]: from='mgr.? 172.21.15.67:0/225113053' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi067.dwichg/crt"}]: dispatch 2024-04-12T05:39:27.226 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:26 smithi067 ceph-mon[49010]: from='mgr.? 172.21.15.67:0/225113053' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T05:39:27.226 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:26 smithi067 ceph-mon[49010]: from='mgr.? 172.21.15.67:0/225113053' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi067.dwichg/key"}]: dispatch 2024-04-12T05:39:27.226 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:26 smithi067 ceph-mon[49010]: from='mgr.? 172.21.15.67:0/225113053' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T05:39:28.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:27 smithi032 ceph-mon[39731]: mgrmap e33: smithi032.hrwxrh(active, since 39s), standbys: smithi067.dwichg 2024-04-12T05:39:28.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:27 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:28.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:27 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:28.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:27 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:39:28.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:27 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:39:28.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:27 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:28.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:27 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:39:28.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:27 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:28.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:27 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:28.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:27 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi032.hrwxrh"}]: dispatch 2024-04-12T05:39:28.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:27 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi032.hrwxrh"}]': finished 2024-04-12T05:39:28.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:27 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi067.dwichg"}]: dispatch 2024-04-12T05:39:28.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:27 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi067.dwichg"}]': finished 2024-04-12T05:39:28.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:27 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:28.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:27 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:28.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:27 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:28.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:27 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:28.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:27 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:28.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:27 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:28.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:27 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:28.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:27 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:28.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:27 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:28.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:27 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:28.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:27 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:28.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:27 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:28.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:27 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:28.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:27 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:28.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:27 smithi067 ceph-mon[49010]: mgrmap e33: smithi032.hrwxrh(active, since 39s), standbys: smithi067.dwichg 2024-04-12T05:39:28.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:27 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:28.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:27 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:28.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:27 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:39:28.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:27 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:39:28.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:27 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:28.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:27 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:39:28.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:27 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:28.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:27 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:28.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:27 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi032.hrwxrh"}]: dispatch 2024-04-12T05:39:28.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:27 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi032.hrwxrh"}]': finished 2024-04-12T05:39:28.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:27 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi067.dwichg"}]: dispatch 2024-04-12T05:39:28.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:27 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi067.dwichg"}]': finished 2024-04-12T05:39:28.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:27 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:28.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:27 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:28.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:27 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:28.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:27 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:28.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:27 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:28.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:27 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:28.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:27 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:28.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:27 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:28.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:27 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:28.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:27 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:28.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:27 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:28.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:27 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:28.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:27 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:28.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:27 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:29.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:28 smithi067 ceph-mon[49010]: pgmap v23: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 85 B/s wr, 3 op/s 2024-04-12T05:39:29.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:28 smithi067 ceph-mon[49010]: Upgrade: Setting container_image for all mgr 2024-04-12T05:39:29.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:28 smithi067 ceph-mon[49010]: Upgrade: Setting container_image for all rgw 2024-04-12T05:39:29.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:28 smithi067 ceph-mon[49010]: Upgrade: Setting container_image for all rbd-mirror 2024-04-12T05:39:29.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:28 smithi067 ceph-mon[49010]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-12T05:39:29.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:28 smithi067 ceph-mon[49010]: Upgrade: Setting container_image for all ceph-exporter 2024-04-12T05:39:29.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:28 smithi067 ceph-mon[49010]: Upgrade: Setting container_image for all iscsi 2024-04-12T05:39:29.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:28 smithi067 ceph-mon[49010]: Upgrade: Setting container_image for all nfs 2024-04-12T05:39:29.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:28 smithi067 ceph-mon[49010]: Upgrade: Setting container_image for all nvmeof 2024-04-12T05:39:29.273 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:28 smithi032 ceph-mon[39731]: pgmap v23: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 85 B/s wr, 3 op/s 2024-04-12T05:39:29.273 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:28 smithi032 ceph-mon[39731]: Upgrade: Setting container_image for all mgr 2024-04-12T05:39:29.273 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:28 smithi032 ceph-mon[39731]: Upgrade: Setting container_image for all rgw 2024-04-12T05:39:29.273 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:28 smithi032 ceph-mon[39731]: Upgrade: Setting container_image for all rbd-mirror 2024-04-12T05:39:29.273 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:28 smithi032 ceph-mon[39731]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-12T05:39:29.273 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:28 smithi032 ceph-mon[39731]: Upgrade: Setting container_image for all ceph-exporter 2024-04-12T05:39:29.273 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:28 smithi032 ceph-mon[39731]: Upgrade: Setting container_image for all iscsi 2024-04-12T05:39:29.273 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:28 smithi032 ceph-mon[39731]: Upgrade: Setting container_image for all nfs 2024-04-12T05:39:29.273 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:28 smithi032 ceph-mon[39731]: Upgrade: Setting container_image for all nvmeof 2024-04-12T05:39:31.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:30 smithi032 ceph-mon[39731]: pgmap v24: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-12T05:39:31.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:30 smithi032 ceph-mon[39731]: Upgrade: Updating node-exporter.smithi032 (1/2) 2024-04-12T05:39:31.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:30 smithi032 ceph-mon[39731]: Deploying daemon node-exporter.smithi032 on smithi032 2024-04-12T05:39:31.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:30 smithi067 ceph-mon[49010]: pgmap v24: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-12T05:39:31.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:30 smithi067 ceph-mon[49010]: Upgrade: Updating node-exporter.smithi032 (1/2) 2024-04-12T05:39:31.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:30 smithi067 ceph-mon[49010]: Deploying daemon node-exporter.smithi032 on smithi032 2024-04-12T05:39:33.149 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:32 smithi067 ceph-mon[49010]: pgmap v25: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:39:33.149 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:32 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:33.149 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:32 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:33.149 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:32 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:39:33.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:32 smithi032 ceph-mon[39731]: pgmap v25: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:39:33.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:32 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:33.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:32 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:33.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:32 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:39:35.121 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:34 smithi067 ceph-mon[49010]: pgmap v26: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:39:35.172 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:34 smithi067 ceph-mon[49010]: Upgrade: Updating node-exporter.smithi067 (2/2) 2024-04-12T05:39:35.172 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:34 smithi067 ceph-mon[49010]: Deploying daemon node-exporter.smithi067 on smithi067 2024-04-12T05:39:35.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:34 smithi032 ceph-mon[39731]: pgmap v26: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:39:35.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:34 smithi032 ceph-mon[39731]: Upgrade: Updating node-exporter.smithi067 (2/2) 2024-04-12T05:39:35.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:34 smithi032 ceph-mon[39731]: Deploying daemon node-exporter.smithi067 on smithi067 2024-04-12T05:39:37.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:36 smithi032 ceph-mon[39731]: pgmap v27: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:39:37.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:36 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:37.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:36 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:37.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:36 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:39:37.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:36 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:37.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:36 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:37.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:36 smithi067 ceph-mon[49010]: pgmap v27: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:39:37.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:36 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:37.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:36 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:37.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:36 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:39:37.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:36 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:37.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:36 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:38.961 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:38 smithi032 ceph-mon[39731]: pgmap v28: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:39:39.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:38 smithi067 ceph-mon[49010]: pgmap v28: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:39:40.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:40 smithi067 ceph-mon[49010]: pgmap v29: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:39:40.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:40.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:40.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:40.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:40.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:40 smithi032 ceph-mon[39731]: pgmap v29: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:39:40.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:40.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:40.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:40.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:42.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:42 smithi067 ceph-mon[49010]: pgmap v30: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:39:42.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:42 smithi032 ceph-mon[39731]: pgmap v30: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:39:44.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:44 smithi032 ceph-mon[39731]: pgmap v31: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:39:44.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:44 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:44.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:44 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:44.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:44 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:44.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:44 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:44.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:44 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:44.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:44 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:45.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:44 smithi067 ceph-mon[49010]: pgmap v31: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:39:45.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:44 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:45.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:44 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:45.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:44 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:45.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:44 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:45.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:44 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:45.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:44 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:46.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:46 smithi032 ceph-mon[39731]: pgmap v32: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:39:47.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:46 smithi067 ceph-mon[49010]: pgmap v32: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:39:47.676 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:47 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:39:48.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:47 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:39:48.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:48 smithi032 ceph-mon[39731]: pgmap v33: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:39:48.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:48 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:48.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:48 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:48.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:48 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:39:48.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:48 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:39:48.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:48 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:48.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:48 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:39:48.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:48 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:48.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:48 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:48.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:48 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:48.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:48 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:48.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:48 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:48.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:48 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:48.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:48 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:48.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:48 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:48.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:48 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:49.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:48 smithi067 ceph-mon[49010]: pgmap v33: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:39:49.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:48 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:49.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:48 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:49.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:48 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:39:49.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:48 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:39:49.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:48 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:49.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:48 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:39:49.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:48 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:49.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:48 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:49.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:48 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:49.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:48 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:49.020 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:48 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:49.020 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:48 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:49.020 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:48 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:49.020 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:48 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:49.020 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:48 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:50.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:50 smithi032 ceph-mon[39731]: pgmap v34: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:39:50.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:50 smithi032 ceph-mon[39731]: Upgrade: Updating prometheus.smithi032 2024-04-12T05:39:51.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:50 smithi067 ceph-mon[49010]: pgmap v34: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:39:51.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:50 smithi067 ceph-mon[49010]: Upgrade: Updating prometheus.smithi032 2024-04-12T05:39:51.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.806+0000 7fc555acb700 1 -- 172.21.15.32:0/2169146199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5500fcf00 msgr2=0x7fc5500fd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:51.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.806+0000 7fc555acb700 1 --2- 172.21.15.32:0/2169146199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5500fcf00 0x7fc5500fd310 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc540004940 tx=0x7fc540009e30 comp rx=0 tx=0).stop 2024-04-12T05:39:51.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.806+0000 7fc555acb700 1 -- 172.21.15.32:0/2169146199 shutdown_connections 2024-04-12T05:39:51.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.806+0000 7fc555acb700 1 --2- 172.21.15.32:0/2169146199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5500fe070 0x7fc5500fe4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:51.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.806+0000 7fc555acb700 1 --2- 172.21.15.32:0/2169146199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5500fcf00 0x7fc5500fd310 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:51.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.806+0000 7fc555acb700 1 -- 172.21.15.32:0/2169146199 >> 172.21.15.32:0/2169146199 conn(0x7fc5500f85a0 msgr2=0x7fc5500fa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:51.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.806+0000 7fc555acb700 1 -- 172.21.15.32:0/2169146199 shutdown_connections 2024-04-12T05:39:51.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.806+0000 7fc555acb700 1 -- 172.21.15.32:0/2169146199 wait complete. 2024-04-12T05:39:51.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.807+0000 7fc555acb700 1 Processor -- start 2024-04-12T05:39:51.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.807+0000 7fc555acb700 1 -- start start 2024-04-12T05:39:51.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.807+0000 7fc555acb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5500fcf00 0x7fc550071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:39:51.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.807+0000 7fc555acb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5500fe070 0x7fc550070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:39:51.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.807+0000 7fc555acb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc550070950 con 0x7fc5500fe070 2024-04-12T05:39:51.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.807+0000 7fc555acb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc550070ac0 con 0x7fc5500fcf00 2024-04-12T05:39:51.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.807+0000 7fc54effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5500fe070 0x7fc550070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:51.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.807+0000 7fc54effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5500fe070 0x7fc550070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40054/0 (socket says 172.21.15.32:40054) 2024-04-12T05:39:51.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.807+0000 7fc54effd700 1 -- 172.21.15.32:0/58998767 learned_addr learned my addr 172.21.15.32:0/58998767 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:39:51.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.808+0000 7fc54f7fe700 1 --2- 172.21.15.32:0/58998767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5500fcf00 0x7fc550071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:51.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.808+0000 7fc54effd700 1 -- 172.21.15.32:0/58998767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5500fcf00 msgr2=0x7fc550071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:51.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.808+0000 7fc54effd700 1 --2- 172.21.15.32:0/58998767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5500fcf00 0x7fc550071e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:51.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.808+0000 7fc54effd700 1 -- 172.21.15.32:0/58998767 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc540005040 con 0x7fc5500fe070 2024-04-12T05:39:51.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.808+0000 7fc54effd700 1 --2- 172.21.15.32:0/58998767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5500fe070 0x7fc550070310 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc54400b790 tx=0x7fc544009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:51.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.808+0000 7fc537fff700 1 -- 172.21.15.32:0/58998767 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc54400f770 con 0x7fc5500fe070 2024-04-12T05:39:51.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.808+0000 7fc555acb700 1 -- 172.21.15.32:0/58998767 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc550070da0 con 0x7fc5500fe070 2024-04-12T05:39:51.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.808+0000 7fc537fff700 1 -- 172.21.15.32:0/58998767 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc54400fd80 con 0x7fc5500fe070 2024-04-12T05:39:51.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.808+0000 7fc537fff700 1 -- 172.21.15.32:0/58998767 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc54400d310 con 0x7fc5500fe070 2024-04-12T05:39:51.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.808+0000 7fc555acb700 1 -- 172.21.15.32:0/58998767 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5501159e0 con 0x7fc5500fe070 2024-04-12T05:39:51.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.809+0000 7fc555acb700 1 -- 172.21.15.32:0/58998767 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc55005f5f0 con 0x7fc5500fe070 2024-04-12T05:39:51.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.810+0000 7fc537fff700 1 -- 172.21.15.32:0/58998767 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 33) v1 ==== 90805+0+0 (secure 0 0 0) 0x7fc544010420 con 0x7fc5500fe070 2024-04-12T05:39:51.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.810+0000 7fc537fff700 1 --2- 172.21.15.32:0/58998767 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fc53806c9c0 0x7fc53806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:39:51.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.810+0000 7fc537fff700 1 -- 172.21.15.32:0/58998767 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7fc54408bc80 con 0x7fc5500fe070 2024-04-12T05:39:51.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.810+0000 7fc54f7fe700 1 --2- 172.21.15.32:0/58998767 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fc53806c9c0 0x7fc53806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:51.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.810+0000 7fc54f7fe700 1 --2- 172.21.15.32:0/58998767 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fc53806c9c0 0x7fc53806ee70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc54000b900 tx=0x7fc540009e30 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:51.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:51.815+0000 7fc537fff700 1 -- 172.21.15.32:0/58998767 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fc544056080 con 0x7fc5500fe070 2024-04-12T05:39:51.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:51 smithi032 ceph-mon[39731]: Deploying daemon prometheus.smithi032 on smithi032 2024-04-12T05:39:52.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:51 smithi067 ceph-mon[49010]: Deploying daemon prometheus.smithi032 on smithi032 2024-04-12T05:39:52.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.039+0000 7fc555acb700 1 -- 172.21.15.32:0/58998767 --> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5501023d0 con 0x7fc53806c9c0 2024-04-12T05:39:52.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.040+0000 7fc537fff700 1 -- 172.21.15.32:0/58998767 <== mgr.24463 v2:172.21.15.32:6800/2564765508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7fc5501023d0 con 0x7fc53806c9c0 2024-04-12T05:39:52.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.042+0000 7fc555acb700 1 -- 172.21.15.32:0/58998767 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fc53806c9c0 msgr2=0x7fc53806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:52.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.042+0000 7fc555acb700 1 --2- 172.21.15.32:0/58998767 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fc53806c9c0 0x7fc53806ee70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc54000b900 tx=0x7fc540009e30 comp rx=0 tx=0).stop 2024-04-12T05:39:52.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.042+0000 7fc555acb700 1 -- 172.21.15.32:0/58998767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5500fe070 msgr2=0x7fc550070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:52.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.042+0000 7fc555acb700 1 --2- 172.21.15.32:0/58998767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5500fe070 0x7fc550070310 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc54400b790 tx=0x7fc544009250 comp rx=0 tx=0).stop 2024-04-12T05:39:52.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.042+0000 7fc555acb700 1 -- 172.21.15.32:0/58998767 shutdown_connections 2024-04-12T05:39:52.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.042+0000 7fc555acb700 1 --2- 172.21.15.32:0/58998767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5500fcf00 0x7fc550071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:52.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.042+0000 7fc555acb700 1 --2- 172.21.15.32:0/58998767 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fc53806c9c0 0x7fc53806ee70 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:52.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.042+0000 7fc555acb700 1 --2- 172.21.15.32:0/58998767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5500fe070 0x7fc550070310 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:52.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.042+0000 7fc555acb700 1 -- 172.21.15.32:0/58998767 >> 172.21.15.32:0/58998767 conn(0x7fc5500f85a0 msgr2=0x7fc5500f9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:52.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.042+0000 7fc555acb700 1 -- 172.21.15.32:0/58998767 shutdown_connections 2024-04-12T05:39:52.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.042+0000 7fc555acb700 1 -- 172.21.15.32:0/58998767 wait complete. 2024-04-12T05:39:52.050 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:39:52.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.152+0000 7fdace727700 1 -- 172.21.15.32:0/1475587719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac80fff00 msgr2=0x7fdac8102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:52.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.152+0000 7fdace727700 1 --2- 172.21.15.32:0/1475587719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac80fff00 0x7fdac8102320 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fdab0004940 tx=0x7fdab0009e30 comp rx=0 tx=0).stop 2024-04-12T05:39:52.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.152+0000 7fdace727700 1 -- 172.21.15.32:0/1475587719 shutdown_connections 2024-04-12T05:39:52.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.152+0000 7fdace727700 1 --2- 172.21.15.32:0/1475587719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac80fff00 0x7fdac8102320 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:52.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.152+0000 7fdace727700 1 --2- 172.21.15.32:0/1475587719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdac80662e0 0x7fdac80ff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:52.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.152+0000 7fdace727700 1 -- 172.21.15.32:0/1475587719 >> 172.21.15.32:0/1475587719 conn(0x7fdac80f7430 msgr2=0x7fdac80f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:52.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.152+0000 7fdace727700 1 -- 172.21.15.32:0/1475587719 shutdown_connections 2024-04-12T05:39:52.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.152+0000 7fdace727700 1 -- 172.21.15.32:0/1475587719 wait complete. 2024-04-12T05:39:52.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.152+0000 7fdace727700 1 Processor -- start 2024-04-12T05:39:52.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.153+0000 7fdace727700 1 -- start start 2024-04-12T05:39:52.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.153+0000 7fdace727700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac80662e0 0x7fdac80fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:39:52.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.153+0000 7fdace727700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdac80fff00 0x7fdac80fb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:39:52.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.153+0000 7fdace727700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac80fbe50 con 0x7fdac80662e0 2024-04-12T05:39:52.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.153+0000 7fdace727700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac80fbf90 con 0x7fdac80fff00 2024-04-12T05:39:52.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.153+0000 7fdac7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac80662e0 0x7fdac80fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:52.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.153+0000 7fdac7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac80662e0 0x7fdac80fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40066/0 (socket says 172.21.15.32:40066) 2024-04-12T05:39:52.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.153+0000 7fdac7fff700 1 -- 172.21.15.32:0/2857910987 learned_addr learned my addr 172.21.15.32:0/2857910987 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:39:52.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.153+0000 7fdac77fe700 1 --2- 172.21.15.32:0/2857910987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdac80fff00 0x7fdac80fb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:52.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.153+0000 7fdac7fff700 1 -- 172.21.15.32:0/2857910987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdac80fff00 msgr2=0x7fdac80fb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:52.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.153+0000 7fdac7fff700 1 --2- 172.21.15.32:0/2857910987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdac80fff00 0x7fdac80fb8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:52.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.153+0000 7fdac7fff700 1 -- 172.21.15.32:0/2857910987 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab0005040 con 0x7fdac80662e0 2024-04-12T05:39:52.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.153+0000 7fdac7fff700 1 --2- 172.21.15.32:0/2857910987 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac80662e0 0x7fdac80fd3a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fdab800eb40 tx=0x7fdab800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:52.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.154+0000 7fdac4ff9700 1 -- 172.21.15.32:0/2857910987 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdab8007cc0 con 0x7fdac80662e0 2024-04-12T05:39:52.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.154+0000 7fdace727700 1 -- 172.21.15.32:0/2857910987 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdac80fc210 con 0x7fdac80662e0 2024-04-12T05:39:52.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.154+0000 7fdac4ff9700 1 -- 172.21.15.32:0/2857910987 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdab800f040 con 0x7fdac80662e0 2024-04-12T05:39:52.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.154+0000 7fdac4ff9700 1 -- 172.21.15.32:0/2857910987 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdab8009570 con 0x7fdac80662e0 2024-04-12T05:39:52.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.154+0000 7fdace727700 1 -- 172.21.15.32:0/2857910987 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdac8115950 con 0x7fdac80662e0 2024-04-12T05:39:52.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.155+0000 7fdace727700 1 -- 172.21.15.32:0/2857910987 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdac805f5f0 con 0x7fdac80662e0 2024-04-12T05:39:52.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.155+0000 7fdac4ff9700 1 -- 172.21.15.32:0/2857910987 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 33) v1 ==== 90805+0+0 (secure 0 0 0) 0x7fdab80096d0 con 0x7fdac80662e0 2024-04-12T05:39:52.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.155+0000 7fdac4ff9700 1 --2- 172.21.15.32:0/2857910987 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fdab406c9c0 0x7fdab406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:39:52.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.156+0000 7fdac4ff9700 1 -- 172.21.15.32:0/2857910987 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7fdab808b8c0 con 0x7fdac80662e0 2024-04-12T05:39:52.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.156+0000 7fdac77fe700 1 --2- 172.21.15.32:0/2857910987 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fdab406c9c0 0x7fdab406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:52.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.156+0000 7fdac77fe700 1 --2- 172.21.15.32:0/2857910987 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fdab406c9c0 0x7fdab406ee70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdac80fd980 tx=0x7fdab0009e30 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:52.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.160+0000 7fdac4ff9700 1 -- 172.21.15.32:0/2857910987 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fdab8055be0 con 0x7fdac80662e0 2024-04-12T05:39:52.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.385+0000 7fdace727700 1 -- 172.21.15.32:0/2857910987 --> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdac8101b70 con 0x7fdab406c9c0 2024-04-12T05:39:52.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.386+0000 7fdac4ff9700 1 -- 172.21.15.32:0/2857910987 <== mgr.24463 v2:172.21.15.32:6800/2564765508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7fdac8101b70 con 0x7fdab406c9c0 2024-04-12T05:39:52.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.389+0000 7fdace727700 1 -- 172.21.15.32:0/2857910987 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fdab406c9c0 msgr2=0x7fdab406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:52.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.389+0000 7fdace727700 1 --2- 172.21.15.32:0/2857910987 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fdab406c9c0 0x7fdab406ee70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdac80fd980 tx=0x7fdab0009e30 comp rx=0 tx=0).stop 2024-04-12T05:39:52.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.389+0000 7fdace727700 1 -- 172.21.15.32:0/2857910987 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac80662e0 msgr2=0x7fdac80fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:52.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.389+0000 7fdace727700 1 --2- 172.21.15.32:0/2857910987 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac80662e0 0x7fdac80fd3a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fdab800eb40 tx=0x7fdab800b040 comp rx=0 tx=0).stop 2024-04-12T05:39:52.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.389+0000 7fdace727700 1 -- 172.21.15.32:0/2857910987 shutdown_connections 2024-04-12T05:39:52.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.389+0000 7fdace727700 1 --2- 172.21.15.32:0/2857910987 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fdab406c9c0 0x7fdab406ee70 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:52.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.389+0000 7fdace727700 1 --2- 172.21.15.32:0/2857910987 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac80662e0 0x7fdac80fd3a0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:52.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.389+0000 7fdace727700 1 --2- 172.21.15.32:0/2857910987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdac80fff00 0x7fdac80fb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:52.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.389+0000 7fdace727700 1 -- 172.21.15.32:0/2857910987 >> 172.21.15.32:0/2857910987 conn(0x7fdac80f7430 msgr2=0x7fdac80f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:52.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.389+0000 7fdace727700 1 -- 172.21.15.32:0/2857910987 shutdown_connections 2024-04-12T05:39:52.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.389+0000 7fdace727700 1 -- 172.21.15.32:0/2857910987 wait complete. 2024-04-12T05:39:52.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.492+0000 7f66c00e9700 1 -- 172.21.15.32:0/3520058257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66b80fcc00 msgr2=0x7f66b80fd070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:52.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.492+0000 7f66c00e9700 1 --2- 172.21.15.32:0/3520058257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66b80fcc00 0x7f66b80fd070 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f66a8004940 tx=0x7f66a8009e30 comp rx=0 tx=0).stop 2024-04-12T05:39:52.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.493+0000 7f66c00e9700 1 -- 172.21.15.32:0/3520058257 shutdown_connections 2024-04-12T05:39:52.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.493+0000 7f66c00e9700 1 --2- 172.21.15.32:0/3520058257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66b80fcc00 0x7f66b80fd070 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:52.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.493+0000 7f66c00e9700 1 --2- 172.21.15.32:0/3520058257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66b80fc2b0 0x7f66b80fc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:52.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.493+0000 7f66c00e9700 1 -- 172.21.15.32:0/3520058257 >> 172.21.15.32:0/3520058257 conn(0x7f66b80f6ba0 msgr2=0x7f66b80f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:52.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.493+0000 7f66c00e9700 1 -- 172.21.15.32:0/3520058257 shutdown_connections 2024-04-12T05:39:52.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.493+0000 7f66c00e9700 1 -- 172.21.15.32:0/3520058257 wait complete. 2024-04-12T05:39:52.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.493+0000 7f66c00e9700 1 Processor -- start 2024-04-12T05:39:52.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.493+0000 7f66c00e9700 1 -- start start 2024-04-12T05:39:52.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.493+0000 7f66c00e9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66b80fc2b0 0x7f66b8110690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:39:52.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.493+0000 7f66c00e9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66b80fcc00 0x7f66b8112be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:39:52.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.493+0000 7f66c00e9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b80fbd30 con 0x7f66b80fcc00 2024-04-12T05:39:52.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.493+0000 7f66c00e9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b80fbe70 con 0x7f66b80fc2b0 2024-04-12T05:39:52.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.494+0000 7f66bd684700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66b80fcc00 0x7f66b8112be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:52.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.494+0000 7f66bd684700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66b80fcc00 0x7f66b8112be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40080/0 (socket says 172.21.15.32:40080) 2024-04-12T05:39:52.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.494+0000 7f66bd684700 1 -- 172.21.15.32:0/721741172 learned_addr learned my addr 172.21.15.32:0/721741172 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:39:52.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.494+0000 7f66bde85700 1 --2- 172.21.15.32:0/721741172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66b80fc2b0 0x7f66b8110690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:52.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.494+0000 7f66bd684700 1 -- 172.21.15.32:0/721741172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66b80fc2b0 msgr2=0x7f66b8110690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:52.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.494+0000 7f66bd684700 1 --2- 172.21.15.32:0/721741172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66b80fc2b0 0x7f66b8110690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:52.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.494+0000 7f66bd684700 1 -- 172.21.15.32:0/721741172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a8005040 con 0x7f66b80fcc00 2024-04-12T05:39:52.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.494+0000 7f66bd684700 1 --2- 172.21.15.32:0/721741172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66b80fcc00 0x7f66b8112be0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009fd0 tx=0x7f66a8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:52.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.494+0000 7f66ae7fc700 1 -- 172.21.15.32:0/721741172 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66a800e040 con 0x7f66b80fcc00 2024-04-12T05:39:52.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.494+0000 7f66ae7fc700 1 -- 172.21.15.32:0/721741172 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f66a8004290 con 0x7f66b80fcc00 2024-04-12T05:39:52.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.494+0000 7f66ae7fc700 1 -- 172.21.15.32:0/721741172 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66a801c3f0 con 0x7f66b80fcc00 2024-04-12T05:39:52.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.494+0000 7f66c00e9700 1 -- 172.21.15.32:0/721741172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b81131e0 con 0x7f66b80fcc00 2024-04-12T05:39:52.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.494+0000 7f66c00e9700 1 -- 172.21.15.32:0/721741172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b8113770 con 0x7f66b80fcc00 2024-04-12T05:39:52.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.495+0000 7f66c00e9700 1 -- 172.21.15.32:0/721741172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66b805f5f0 con 0x7f66b80fcc00 2024-04-12T05:39:52.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.496+0000 7f66ae7fc700 1 -- 172.21.15.32:0/721741172 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 33) v1 ==== 90805+0+0 (secure 0 0 0) 0x7f66a800bc50 con 0x7f66b80fcc00 2024-04-12T05:39:52.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.496+0000 7f66ae7fc700 1 --2- 172.21.15.32:0/721741172 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f66a406c9c0 0x7f66a406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:39:52.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.496+0000 7f66ae7fc700 1 -- 172.21.15.32:0/721741172 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f66a808d170 con 0x7f66b80fcc00 2024-04-12T05:39:52.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.496+0000 7f66bde85700 1 --2- 172.21.15.32:0/721741172 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f66a406c9c0 0x7f66a406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:52.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.497+0000 7f66bde85700 1 --2- 172.21.15.32:0/721741172 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f66a406c9c0 0x7f66a406ee70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f66b4006fd0 tx=0x7f66b4008040 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:52.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.501+0000 7f66ae7fc700 1 -- 172.21.15.32:0/721741172 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f66a8057500 con 0x7f66b80fcc00 2024-04-12T05:39:52.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.718+0000 7f66c00e9700 1 -- 172.21.15.32:0/721741172 --> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f66b8101430 con 0x7f66a406c9c0 2024-04-12T05:39:52.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.729+0000 7f66ae7fc700 1 -- 172.21.15.32:0/721741172 <== mgr.24463 v2:172.21.15.32:6800/2564765508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f66b8101430 con 0x7f66a406c9c0 2024-04-12T05:39:52.727 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:39:52.727 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (103s) 8s ago 11m 19.3M - 0.25.0 c8568f914cd2 fe43361ddb75 2024-04-12T05:39:52.727 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (11m) 8s ago 11m 6656k - 17.2.7-827-gd3b5060a 21de9b7369ad a9cfc00c311b 2024-04-12T05:39:52.727 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (10m) 13s ago 10m 6651k - 17.2.7-827-gd3b5060a 21de9b7369ad 1f2eea4542d9 2024-04-12T05:39:52.727 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (97s) 8s ago 10m 80.2M - 9.4.7 954c08fa6188 2229cad57cd3 2024-04-12T05:39:52.727 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (5m) 8s ago 5m 64.5M - 17.2.7-827-gd3b5060a 21de9b7369ad e7ada8e5259c 2024-04-12T05:39:52.727 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (5m) 8s ago 5m 116M - 17.2.7-827-gd3b5060a 21de9b7369ad 55124da9afdb 2024-04-12T05:39:52.727 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (5m) 13s ago 5m 62.5M - 17.2.7-827-gd3b5060a 21de9b7369ad 0c295d31a4a6 2024-04-12T05:39:52.727 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (5m) 13s ago 5m 120M - 17.2.7-827-gd3b5060a 21de9b7369ad 6eecd7deaac2 2024-04-12T05:39:52.727 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (80s) 8s ago 12m 498M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:39:52.727 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (33s) 13s ago 10m 449M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:39:52.727 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (12m) 8s ago 12m 52.6M 2048M 17.2.7-827-gd3b5060a 21de9b7369ad 75aad95b984d 2024-04-12T05:39:52.728 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (10m) 13s ago 10m 42.3M 2048M 17.2.7-827-gd3b5060a 21de9b7369ad 157be8a8c6d8 2024-04-12T05:39:52.728 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (21s) 8s ago 11m 7218k - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:39:52.728 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (17s) 13s ago 10m 3594k - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:39:52.728 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (9m) 8s ago 9m 218M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad 97446e42ceba 2024-04-12T05:39:52.729 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (8m) 8s ago 8m 235M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad bf9f9a2db2d5 2024-04-12T05:39:52.729 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (8m) 8s ago 8m 188M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad f055cf1ca665 2024-04-12T05:39:52.729 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (7m) 13s ago 7m 255M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad 61b851235351 2024-04-12T05:39:52.729 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (7m) 13s ago 7m 227M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad 2c84d9047e9c 2024-04-12T05:39:52.729 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (7m) 13s ago 7m 217M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad e54839bb6055 2024-04-12T05:39:52.729 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (39s) 8s ago 10m 46.9M - 2.43.0 a07b618ecd1d dfe0a5407965 2024-04-12T05:39:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.731+0000 7f66c00e9700 1 -- 172.21.15.32:0/721741172 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f66a406c9c0 msgr2=0x7f66a406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.731+0000 7f66c00e9700 1 --2- 172.21.15.32:0/721741172 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f66a406c9c0 0x7f66a406ee70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f66b4006fd0 tx=0x7f66b4008040 comp rx=0 tx=0).stop 2024-04-12T05:39:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.731+0000 7f66c00e9700 1 -- 172.21.15.32:0/721741172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66b80fcc00 msgr2=0x7f66b8112be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.731+0000 7f66c00e9700 1 --2- 172.21.15.32:0/721741172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66b80fcc00 0x7f66b8112be0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009fd0 tx=0x7f66a8009e30 comp rx=0 tx=0).stop 2024-04-12T05:39:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.731+0000 7f66c00e9700 1 -- 172.21.15.32:0/721741172 shutdown_connections 2024-04-12T05:39:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.731+0000 7f66c00e9700 1 --2- 172.21.15.32:0/721741172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66b80fc2b0 0x7f66b8110690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.731+0000 7f66c00e9700 1 --2- 172.21.15.32:0/721741172 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f66a406c9c0 0x7f66a406ee70 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.731+0000 7f66c00e9700 1 --2- 172.21.15.32:0/721741172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66b80fcc00 0x7f66b8112be0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.731+0000 7f66c00e9700 1 -- 172.21.15.32:0/721741172 >> 172.21.15.32:0/721741172 conn(0x7f66b80f6ba0 msgr2=0x7f66b80f8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.731+0000 7f66c00e9700 1 -- 172.21.15.32:0/721741172 shutdown_connections 2024-04-12T05:39:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.731+0000 7f66c00e9700 1 -- 172.21.15.32:0/721741172 wait complete. 2024-04-12T05:39:52.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.839+0000 7fdfeed62700 1 -- 172.21.15.32:0/2070048649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfe80fcee0 msgr2=0x7fdfe80fd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:52.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.839+0000 7fdfeed62700 1 --2- 172.21.15.32:0/2070048649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfe80fcee0 0x7fdfe80fd310 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc004940 tx=0x7fdfdc009e30 comp rx=0 tx=0).stop 2024-04-12T05:39:52.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.839+0000 7fdfeed62700 1 -- 172.21.15.32:0/2070048649 shutdown_connections 2024-04-12T05:39:52.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.839+0000 7fdfeed62700 1 --2- 172.21.15.32:0/2070048649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfe80fe0d0 0x7fdfe80fe540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:52.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.839+0000 7fdfeed62700 1 --2- 172.21.15.32:0/2070048649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfe80fcee0 0x7fdfe80fd310 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:52.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.839+0000 7fdfeed62700 1 -- 172.21.15.32:0/2070048649 >> 172.21.15.32:0/2070048649 conn(0x7fdfe80f8560 msgr2=0x7fdfe80fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:52.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.839+0000 7fdfeed62700 1 -- 172.21.15.32:0/2070048649 shutdown_connections 2024-04-12T05:39:52.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.839+0000 7fdfeed62700 1 -- 172.21.15.32:0/2070048649 wait complete. 2024-04-12T05:39:52.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.840+0000 7fdfeed62700 1 Processor -- start 2024-04-12T05:39:52.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.840+0000 7fdfeed62700 1 -- start start 2024-04-12T05:39:52.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.840+0000 7fdfeed62700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfe80fcee0 0x7fdfe810d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:39:52.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.840+0000 7fdfeed62700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfe80fe0d0 0x7fdfe810df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:39:52.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.840+0000 7fdfeed62700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe810e510 con 0x7fdfe80fe0d0 2024-04-12T05:39:52.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.840+0000 7fdfeed62700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe810ecf0 con 0x7fdfe80fcee0 2024-04-12T05:39:52.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.840+0000 7fdfe7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfe80fe0d0 0x7fdfe810df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:52.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.840+0000 7fdfe7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfe80fe0d0 0x7fdfe810df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40102/0 (socket says 172.21.15.32:40102) 2024-04-12T05:39:52.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.840+0000 7fdfe7fff700 1 -- 172.21.15.32:0/161876692 learned_addr learned my addr 172.21.15.32:0/161876692 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:39:52.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.840+0000 7fdfecafe700 1 --2- 172.21.15.32:0/161876692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfe80fcee0 0x7fdfe810d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:52.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.840+0000 7fdfe7fff700 1 -- 172.21.15.32:0/161876692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfe80fcee0 msgr2=0x7fdfe810d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:52.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.840+0000 7fdfe7fff700 1 --2- 172.21.15.32:0/161876692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfe80fcee0 0x7fdfe810d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:52.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.840+0000 7fdfe7fff700 1 -- 172.21.15.32:0/161876692 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfdc005040 con 0x7fdfe80fe0d0 2024-04-12T05:39:52.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.841+0000 7fdfe7fff700 1 --2- 172.21.15.32:0/161876692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfe80fe0d0 0x7fdfe810df10 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4002a50 tx=0x7fdfd40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:52.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.841+0000 7fdfe57fa700 1 -- 172.21.15.32:0/161876692 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfd4007cc0 con 0x7fdfe80fe0d0 2024-04-12T05:39:52.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.841+0000 7fdfe57fa700 1 -- 172.21.15.32:0/161876692 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdfd400f040 con 0x7fdfe80fe0d0 2024-04-12T05:39:52.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.841+0000 7fdfeed62700 1 -- 172.21.15.32:0/161876692 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfe810efd0 con 0x7fdfe80fe0d0 2024-04-12T05:39:52.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.841+0000 7fdfe57fa700 1 -- 172.21.15.32:0/161876692 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfd4009540 con 0x7fdfe80fe0d0 2024-04-12T05:39:52.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.841+0000 7fdfeed62700 1 -- 172.21.15.32:0/161876692 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfe810f510 con 0x7fdfe80fe0d0 2024-04-12T05:39:52.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:52 smithi032 ceph-mon[39731]: pgmap v35: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:39:52.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.842+0000 7fdfeed62700 1 -- 172.21.15.32:0/161876692 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfe805f5f0 con 0x7fdfe80fe0d0 2024-04-12T05:39:52.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.843+0000 7fdfe57fa700 1 -- 172.21.15.32:0/161876692 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 33) v1 ==== 90805+0+0 (secure 0 0 0) 0x7fdfd40096a0 con 0x7fdfe80fe0d0 2024-04-12T05:39:52.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.843+0000 7fdfe57fa700 1 --2- 172.21.15.32:0/161876692 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fdfd806c9c0 0x7fdfd806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:39:52.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.843+0000 7fdfe57fa700 1 -- 172.21.15.32:0/161876692 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7fdfd408bb80 con 0x7fdfe80fe0d0 2024-04-12T05:39:52.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.843+0000 7fdfecafe700 1 --2- 172.21.15.32:0/161876692 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fdfd806c9c0 0x7fdfd806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:52.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.844+0000 7fdfecafe700 1 --2- 172.21.15.32:0/161876692 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fdfd806c9c0 0x7fdfd806ee70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc00b900 tx=0x7fdfdc009e30 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:52.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:52.847+0000 7fdfe57fa700 1 -- 172.21.15.32:0/161876692 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fdfd40598e0 con 0x7fdfe80fe0d0 2024-04-12T05:39:53.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:52 smithi067 ceph-mon[49010]: pgmap v35: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:39:53.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.139+0000 7fdfeed62700 1 -- 172.21.15.32:0/161876692 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fdfe804bed0 con 0x7fdfe80fe0d0 2024-04-12T05:39:53.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.139+0000 7fdfe57fa700 1 -- 172.21.15.32:0/161876692 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7fdfd4059470 con 0x7fdfe80fe0d0 2024-04-12T05:39:53.137 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:39:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:39:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 2 2024-04-12T05:39:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:39:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:39:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:39:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:39:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:39:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 6 2024-04-12T05:39:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:39:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:39:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 4 2024-04-12T05:39:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:39:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:39:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 12, 2024-04-12T05:39:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:39:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:39:53.138 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:39:53.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.141+0000 7fdfeed62700 1 -- 172.21.15.32:0/161876692 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fdfd806c9c0 msgr2=0x7fdfd806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:53.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.141+0000 7fdfeed62700 1 --2- 172.21.15.32:0/161876692 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fdfd806c9c0 0x7fdfd806ee70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc00b900 tx=0x7fdfdc009e30 comp rx=0 tx=0).stop 2024-04-12T05:39:53.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.141+0000 7fdfeed62700 1 -- 172.21.15.32:0/161876692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfe80fe0d0 msgr2=0x7fdfe810df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:53.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.141+0000 7fdfeed62700 1 --2- 172.21.15.32:0/161876692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfe80fe0d0 0x7fdfe810df10 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4002a50 tx=0x7fdfd40046b0 comp rx=0 tx=0).stop 2024-04-12T05:39:53.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.141+0000 7fdfeed62700 1 -- 172.21.15.32:0/161876692 shutdown_connections 2024-04-12T05:39:53.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.141+0000 7fdfeed62700 1 --2- 172.21.15.32:0/161876692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfe80fcee0 0x7fdfe810d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:53.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.141+0000 7fdfeed62700 1 --2- 172.21.15.32:0/161876692 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fdfd806c9c0 0x7fdfd806ee70 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:53.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.141+0000 7fdfeed62700 1 --2- 172.21.15.32:0/161876692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfe80fe0d0 0x7fdfe810df10 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:53.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.141+0000 7fdfeed62700 1 -- 172.21.15.32:0/161876692 >> 172.21.15.32:0/161876692 conn(0x7fdfe80f8560 msgr2=0x7fdfe80f9da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:53.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.141+0000 7fdfeed62700 1 -- 172.21.15.32:0/161876692 shutdown_connections 2024-04-12T05:39:53.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.141+0000 7fdfeed62700 1 -- 172.21.15.32:0/161876692 wait complete. 2024-04-12T05:39:53.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.246+0000 7ff5bdee8700 1 -- 172.21.15.32:0/103543779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5b80fdff0 msgr2=0x7ff5b80fe460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:53.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.246+0000 7ff5bdee8700 1 --2- 172.21.15.32:0/103543779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5b80fdff0 0x7ff5b80fe460 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac004940 tx=0x7ff5ac009e30 comp rx=0 tx=0).stop 2024-04-12T05:39:53.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.246+0000 7ff5bdee8700 1 -- 172.21.15.32:0/103543779 shutdown_connections 2024-04-12T05:39:53.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.246+0000 7ff5bdee8700 1 --2- 172.21.15.32:0/103543779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5b80fdff0 0x7ff5b80fe460 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:53.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.246+0000 7ff5bdee8700 1 --2- 172.21.15.32:0/103543779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5b80fce80 0x7ff5b80fd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:53.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.246+0000 7ff5bdee8700 1 -- 172.21.15.32:0/103543779 >> 172.21.15.32:0/103543779 conn(0x7ff5b80f8580 msgr2=0x7ff5b80fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:53.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.246+0000 7ff5bdee8700 1 -- 172.21.15.32:0/103543779 shutdown_connections 2024-04-12T05:39:53.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.246+0000 7ff5bdee8700 1 -- 172.21.15.32:0/103543779 wait complete. 2024-04-12T05:39:53.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.247+0000 7ff5bdee8700 1 Processor -- start 2024-04-12T05:39:53.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.247+0000 7ff5bdee8700 1 -- start start 2024-04-12T05:39:53.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.247+0000 7ff5bdee8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5b80fce80 0x7ff5b810d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:39:53.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.247+0000 7ff5bdee8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5b80fdff0 0x7ff5b810dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:39:53.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.247+0000 7ff5bdee8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b810e4a0 con 0x7ff5b80fdff0 2024-04-12T05:39:53.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.247+0000 7ff5bdee8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b810ecf0 con 0x7ff5b80fce80 2024-04-12T05:39:53.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.247+0000 7ff5b6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5b80fdff0 0x7ff5b810dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:53.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.247+0000 7ff5b6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5b80fdff0 0x7ff5b810dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40106/0 (socket says 172.21.15.32:40106) 2024-04-12T05:39:53.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.248+0000 7ff5b6ffd700 1 -- 172.21.15.32:0/1438206587 learned_addr learned my addr 172.21.15.32:0/1438206587 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:39:53.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.248+0000 7ff5b77fe700 1 --2- 172.21.15.32:0/1438206587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5b80fce80 0x7ff5b810d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:53.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.248+0000 7ff5b6ffd700 1 -- 172.21.15.32:0/1438206587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5b80fce80 msgr2=0x7ff5b810d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:53.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.248+0000 7ff5b6ffd700 1 --2- 172.21.15.32:0/1438206587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5b80fce80 0x7ff5b810d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:53.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.248+0000 7ff5b6ffd700 1 -- 172.21.15.32:0/1438206587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5ac005040 con 0x7ff5b80fdff0 2024-04-12T05:39:53.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.248+0000 7ff5b77fe700 1 --2- 172.21.15.32:0/1438206587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5b80fce80 0x7ff5b810d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:39:53.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.248+0000 7ff5b6ffd700 1 --2- 172.21.15.32:0/1438206587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5b80fdff0 0x7ff5b810dea0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac009380 tx=0x7ff5ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:53.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.248+0000 7ff59ffff700 1 -- 172.21.15.32:0/1438206587 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5ac00e040 con 0x7ff5b80fdff0 2024-04-12T05:39:53.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.248+0000 7ff5bdee8700 1 -- 172.21.15.32:0/1438206587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5b810ef70 con 0x7ff5b80fdff0 2024-04-12T05:39:53.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.248+0000 7ff59ffff700 1 -- 172.21.15.32:0/1438206587 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff5ac00bd20 con 0x7ff5b80fdff0 2024-04-12T05:39:53.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.248+0000 7ff59ffff700 1 -- 172.21.15.32:0/1438206587 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5ac01c630 con 0x7ff5b80fdff0 2024-04-12T05:39:53.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.248+0000 7ff5bdee8700 1 -- 172.21.15.32:0/1438206587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5b810f4b0 con 0x7ff5b80fdff0 2024-04-12T05:39:53.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.249+0000 7ff5bdee8700 1 -- 172.21.15.32:0/1438206587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5b805f5f0 con 0x7ff5b80fdff0 2024-04-12T05:39:53.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.250+0000 7ff59ffff700 1 -- 172.21.15.32:0/1438206587 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 33) v1 ==== 90805+0+0 (secure 0 0 0) 0x7ff5ac013420 con 0x7ff5b80fdff0 2024-04-12T05:39:53.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.250+0000 7ff59ffff700 1 --2- 172.21.15.32:0/1438206587 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7ff5a006c9c0 0x7ff5a006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:39:53.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.250+0000 7ff59ffff700 1 -- 172.21.15.32:0/1438206587 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7ff5ac08ca50 con 0x7ff5b80fdff0 2024-04-12T05:39:53.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.250+0000 7ff5b77fe700 1 --2- 172.21.15.32:0/1438206587 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7ff5a006c9c0 0x7ff5a006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:53.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.251+0000 7ff5b77fe700 1 --2- 172.21.15.32:0/1438206587 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7ff5a006c9c0 0x7ff5a006ee70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff5a8007fd0 tx=0x7ff5a8009040 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:53.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.255+0000 7ff59ffff700 1 -- 172.21.15.32:0/1438206587 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7ff5ac05a7d0 con 0x7ff5b80fdff0 2024-04-12T05:39:53.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.477+0000 7ff5bdee8700 1 -- 172.21.15.32:0/1438206587 --> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff5b810f770 con 0x7ff5a006c9c0 2024-04-12T05:39:53.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.478+0000 7ff59ffff700 1 -- 172.21.15.32:0/1438206587 <== mgr.24463 v2:172.21.15.32:6800/2564765508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7ff5b810f770 con 0x7ff5a006c9c0 2024-04-12T05:39:53.476 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:39:53.476 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:39:53.476 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:39:53.476 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-12T05:39:53.476 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:39:53.476 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr" 2024-04-12T05:39:53.476 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:39:53.476 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "2/2 daemons upgraded", 2024-04-12T05:39:53.477 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading prometheus daemons", 2024-04-12T05:39:53.477 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:39:53.477 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:39:53.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.480+0000 7ff5bdee8700 1 -- 172.21.15.32:0/1438206587 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7ff5a006c9c0 msgr2=0x7ff5a006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:53.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.480+0000 7ff5bdee8700 1 --2- 172.21.15.32:0/1438206587 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7ff5a006c9c0 0x7ff5a006ee70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff5a8007fd0 tx=0x7ff5a8009040 comp rx=0 tx=0).stop 2024-04-12T05:39:53.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.480+0000 7ff5bdee8700 1 -- 172.21.15.32:0/1438206587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5b80fdff0 msgr2=0x7ff5b810dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:53.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.480+0000 7ff5bdee8700 1 --2- 172.21.15.32:0/1438206587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5b80fdff0 0x7ff5b810dea0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac009380 tx=0x7ff5ac009e30 comp rx=0 tx=0).stop 2024-04-12T05:39:53.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.480+0000 7ff5bdee8700 1 -- 172.21.15.32:0/1438206587 shutdown_connections 2024-04-12T05:39:53.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.480+0000 7ff5bdee8700 1 --2- 172.21.15.32:0/1438206587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5b80fce80 0x7ff5b810d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:53.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.480+0000 7ff5bdee8700 1 --2- 172.21.15.32:0/1438206587 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7ff5a006c9c0 0x7ff5a006ee70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:53.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.480+0000 7ff5bdee8700 1 --2- 172.21.15.32:0/1438206587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5b80fdff0 0x7ff5b810dea0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:39:53.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.480+0000 7ff5bdee8700 1 -- 172.21.15.32:0/1438206587 >> 172.21.15.32:0/1438206587 conn(0x7ff5b80f8580 msgr2=0x7ff5b80fa090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:53.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.481+0000 7ff5bdee8700 1 -- 172.21.15.32:0/1438206587 shutdown_connections 2024-04-12T05:39:53.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:39:53.481+0000 7ff5bdee8700 1 -- 172.21.15.32:0/1438206587 wait complete. 2024-04-12T05:39:53.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:53 smithi032 ceph-mon[39731]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:53.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:53 smithi032 ceph-mon[39731]: from='client.14746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:53.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:53 smithi032 ceph-mon[39731]: from='client.14750 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:53.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:53 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/161876692' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:54.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:53 smithi067 ceph-mon[49010]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:54.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:53 smithi067 ceph-mon[49010]: from='client.14746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:54.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:53 smithi067 ceph-mon[49010]: from='client.14750 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:54.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:53 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/161876692' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:54.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:54 smithi032 ceph-mon[39731]: pgmap v36: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:39:54.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:54 smithi032 ceph-mon[39731]: from='client.14758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:55.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:54 smithi067 ceph-mon[49010]: pgmap v36: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:39:55.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:54 smithi067 ceph-mon[49010]: from='client.14758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:56.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:56 smithi032 ceph-mon[39731]: pgmap v37: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:39:56.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:56 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:56.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:56 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:56.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:56 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:39:57.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:56 smithi067 ceph-mon[49010]: pgmap v37: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:39:57.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:56 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:57.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:56 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:39:57.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:56 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:39:58.992 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:39:58 smithi032 ceph-mon[39731]: pgmap v38: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:39:59.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:39:58 smithi067 ceph-mon[49010]: pgmap v38: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:40:00.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:00 smithi032 ceph-mon[39731]: pgmap v39: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:40:00.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:00 smithi032 ceph-mon[39731]: overall HEALTH_OK 2024-04-12T05:40:01.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:00 smithi067 ceph-mon[49010]: pgmap v39: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:40:01.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:00 smithi067 ceph-mon[49010]: overall HEALTH_OK 2024-04-12T05:40:02.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:02 smithi067 ceph-mon[49010]: pgmap v40: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:40:02.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:02 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:02.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:02 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:02.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:02 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:02.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:02 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:02.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:02 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:40:02.524 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:02 smithi032 ceph-mon[39731]: pgmap v40: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:40:02.524 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:02 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:02.525 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:02 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:02.525 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:02 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:02.525 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:02 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:02.525 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:02 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:40:04.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:04 smithi067 ceph-mon[49010]: pgmap v41: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:40:04.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:04 smithi032 ceph-mon[39731]: pgmap v41: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:40:06.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:06 smithi032 ceph-mon[39731]: pgmap v42: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:40:06.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:06 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:06.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:06 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:06.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:06 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:40:06.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:06 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:40:06.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:06 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:06.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:06 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:40:06.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:06 smithi032 ceph-mon[39731]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:40:06.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:06 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:40:06.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:06 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:06.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:06 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:06.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:06 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:06.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:06 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:06.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:06 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:06.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:06 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:06.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:06 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:06.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:06 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:06.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:06 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:06.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:06 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:07.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:06 smithi067 ceph-mon[49010]: pgmap v42: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:40:07.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:06 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:07.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:06 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:07.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:06 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:40:07.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:06 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:40:07.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:06 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:07.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:06 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:40:07.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:06 smithi067 ceph-mon[49010]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:40:07.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:06 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:40:07.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:06 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:07.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:06 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:07.020 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:06 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:07.020 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:06 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:07.020 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:06 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:07.020 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:06 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:07.020 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:06 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:07.020 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:06 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:07.020 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:06 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:07.020 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:06 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:08.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:08 smithi032 ceph-mon[39731]: pgmap v43: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:40:08.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:08 smithi032 ceph-mon[39731]: Upgrade: Updating alertmanager.smithi032 2024-04-12T05:40:08.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:08 smithi032 ceph-mon[39731]: Deploying daemon alertmanager.smithi032 on smithi032 2024-04-12T05:40:09.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:08 smithi067 ceph-mon[49010]: pgmap v43: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:40:09.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:08 smithi067 ceph-mon[49010]: Upgrade: Updating alertmanager.smithi032 2024-04-12T05:40:09.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:08 smithi067 ceph-mon[49010]: Deploying daemon alertmanager.smithi032 on smithi032 2024-04-12T05:40:10.979 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:10 smithi032 ceph-mon[39731]: pgmap v44: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:40:11.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:10 smithi067 ceph-mon[49010]: pgmap v44: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:40:12.889 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:12 smithi032 ceph-mon[39731]: pgmap v45: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:40:13.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:12 smithi067 ceph-mon[49010]: pgmap v45: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:40:14.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:14 smithi067 ceph-mon[49010]: pgmap v46: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:40:14.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:14 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:14.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:14 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:14.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:14 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:40:14.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:14 smithi032 ceph-mon[39731]: pgmap v46: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:40:14.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:14 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:14.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:14 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:14.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:14 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:40:16.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:16 smithi067 ceph-mon[49010]: pgmap v47: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:40:16.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:16 smithi032 ceph-mon[39731]: pgmap v47: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:40:17.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:17 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:40:17.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:17 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:40:18.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:18 smithi067 ceph-mon[49010]: pgmap v48: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:40:18.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:18 smithi032 ceph-mon[39731]: pgmap v48: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:40:19.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:19 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:19.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:19 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:19.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:19 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:19.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:19 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:20.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:19 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:20.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:19 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:20.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:19 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:20.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:19 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:20.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:20 smithi032 ceph-mon[39731]: pgmap v49: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-12T05:40:20.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:20 smithi032 ceph-mon[39731]: mgrmap e34: smithi032.hrwxrh(active, since 92s), standbys: smithi067.dwichg 2024-04-12T05:40:21.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:20 smithi067 ceph-mon[49010]: pgmap v49: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-12T05:40:21.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:20 smithi067 ceph-mon[49010]: mgrmap e34: smithi032.hrwxrh(active, since 92s), standbys: smithi067.dwichg 2024-04-12T05:40:22.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:22 smithi032 ceph-mon[39731]: pgmap v50: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-12T05:40:23.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:22 smithi067 ceph-mon[49010]: pgmap v50: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-12T05:40:23.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.592+0000 7f34ee325700 1 -- 172.21.15.32:0/2142528980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34e80fee10 msgr2=0x7f34e80faf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:23.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.592+0000 7f34ee325700 1 --2- 172.21.15.32:0/2142528980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34e80fee10 0x7f34e80faf50 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f34dc004940 tx=0x7f34dc009e30 comp rx=0 tx=0).stop 2024-04-12T05:40:23.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.592+0000 7f34ee325700 1 -- 172.21.15.32:0/2142528980 shutdown_connections 2024-04-12T05:40:23.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.592+0000 7f34ee325700 1 --2- 172.21.15.32:0/2142528980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34e80fee10 0x7f34e80faf50 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:23.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.592+0000 7f34ee325700 1 --2- 172.21.15.32:0/2142528980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e80fe4c0 0x7f34e80fe8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:23.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.592+0000 7f34ee325700 1 -- 172.21.15.32:0/2142528980 >> 172.21.15.32:0/2142528980 conn(0x7f34e80f6ba0 msgr2=0x7f34e80f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:23.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.593+0000 7f34ee325700 1 -- 172.21.15.32:0/2142528980 shutdown_connections 2024-04-12T05:40:23.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.593+0000 7f34ee325700 1 -- 172.21.15.32:0/2142528980 wait complete. 2024-04-12T05:40:23.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.593+0000 7f34ee325700 1 Processor -- start 2024-04-12T05:40:23.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.593+0000 7f34ee325700 1 -- start start 2024-04-12T05:40:23.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.594+0000 7f34ee325700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e80fe4c0 0x7f34e810d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:40:23.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.594+0000 7f34ee325700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34e80fee10 0x7f34e810de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:40:23.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.594+0000 7f34ee325700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e810e470 con 0x7f34e80fee10 2024-04-12T05:40:23.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.594+0000 7f34ee325700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e810ecc0 con 0x7f34e80fe4c0 2024-04-12T05:40:23.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.594+0000 7f34e77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34e80fee10 0x7f34e810de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:23.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.594+0000 7f34e77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34e80fee10 0x7f34e810de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46984/0 (socket says 172.21.15.32:46984) 2024-04-12T05:40:23.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.594+0000 7f34e77fe700 1 -- 172.21.15.32:0/4103875764 learned_addr learned my addr 172.21.15.32:0/4103875764 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:40:23.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.594+0000 7f34e7fff700 1 --2- 172.21.15.32:0/4103875764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e80fe4c0 0x7f34e810d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:23.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.594+0000 7f34e77fe700 1 -- 172.21.15.32:0/4103875764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e80fe4c0 msgr2=0x7f34e810d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:23.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.594+0000 7f34e77fe700 1 --2- 172.21.15.32:0/4103875764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e80fe4c0 0x7f34e810d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:23.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.594+0000 7f34e77fe700 1 -- 172.21.15.32:0/4103875764 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34dc005040 con 0x7f34e80fee10 2024-04-12T05:40:23.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.594+0000 7f34e77fe700 1 --2- 172.21.15.32:0/4103875764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34e80fee10 0x7f34e810de70 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f34dc009fd0 tx=0x7f34dc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:23.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.594+0000 7f34e4ff9700 1 -- 172.21.15.32:0/4103875764 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34dc00e040 con 0x7f34e80fee10 2024-04-12T05:40:23.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.594+0000 7f34ee325700 1 -- 172.21.15.32:0/4103875764 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34e810ef40 con 0x7f34e80fee10 2024-04-12T05:40:23.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.594+0000 7f34e4ff9700 1 -- 172.21.15.32:0/4103875764 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34dc004290 con 0x7f34e80fee10 2024-04-12T05:40:23.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.594+0000 7f34e4ff9700 1 -- 172.21.15.32:0/4103875764 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34dc01c3f0 con 0x7f34e80fee10 2024-04-12T05:40:23.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.594+0000 7f34ee325700 1 -- 172.21.15.32:0/4103875764 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e810f480 con 0x7f34e80fee10 2024-04-12T05:40:23.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.595+0000 7f34ee325700 1 -- 172.21.15.32:0/4103875764 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34e805f5f0 con 0x7f34e80fee10 2024-04-12T05:40:23.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.596+0000 7f34e4ff9700 1 -- 172.21.15.32:0/4103875764 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 34) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f34dc013790 con 0x7f34e80fee10 2024-04-12T05:40:23.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.596+0000 7f34e4ff9700 1 --2- 172.21.15.32:0/4103875764 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f34d006c8b0 0x7f34d006ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:40:23.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.596+0000 7f34e4ff9700 1 -- 172.21.15.32:0/4103875764 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f34dc08cdb0 con 0x7f34e80fee10 2024-04-12T05:40:23.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.596+0000 7f34e7fff700 1 --2- 172.21.15.32:0/4103875764 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f34d006c8b0 0x7f34d006ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:23.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.597+0000 7f34e7fff700 1 --2- 172.21.15.32:0/4103875764 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f34d006c8b0 0x7f34d006ed60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f34d8004f40 tx=0x7f34d800b040 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:23.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.601+0000 7f34e4ff9700 1 -- 172.21.15.32:0/4103875764 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f34dc05adf0 con 0x7f34e80fee10 2024-04-12T05:40:23.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.847+0000 7f34ee325700 1 -- 172.21.15.32:0/4103875764 --> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f34e810f800 con 0x7f34d006c8b0 2024-04-12T05:40:23.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.848+0000 7f34e4ff9700 1 -- 172.21.15.32:0/4103875764 <== mgr.24463 v2:172.21.15.32:6800/2564765508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7f34e810f800 con 0x7f34d006c8b0 2024-04-12T05:40:23.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.850+0000 7f34ee325700 1 -- 172.21.15.32:0/4103875764 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f34d006c8b0 msgr2=0x7f34d006ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:23.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.850+0000 7f34ee325700 1 --2- 172.21.15.32:0/4103875764 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f34d006c8b0 0x7f34d006ed60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f34d8004f40 tx=0x7f34d800b040 comp rx=0 tx=0).stop 2024-04-12T05:40:23.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.850+0000 7f34ee325700 1 -- 172.21.15.32:0/4103875764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34e80fee10 msgr2=0x7f34e810de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:23.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.850+0000 7f34ee325700 1 --2- 172.21.15.32:0/4103875764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34e80fee10 0x7f34e810de70 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f34dc009fd0 tx=0x7f34dc009e30 comp rx=0 tx=0).stop 2024-04-12T05:40:23.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.850+0000 7f34ee325700 1 -- 172.21.15.32:0/4103875764 shutdown_connections 2024-04-12T05:40:23.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.850+0000 7f34ee325700 1 --2- 172.21.15.32:0/4103875764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e80fe4c0 0x7f34e810d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:23.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.850+0000 7f34ee325700 1 --2- 172.21.15.32:0/4103875764 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f34d006c8b0 0x7f34d006ed60 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:23.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.850+0000 7f34ee325700 1 --2- 172.21.15.32:0/4103875764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34e80fee10 0x7f34e810de70 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:23.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.850+0000 7f34ee325700 1 -- 172.21.15.32:0/4103875764 >> 172.21.15.32:0/4103875764 conn(0x7f34e80f6ba0 msgr2=0x7f34e80f8380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:23.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.850+0000 7f34ee325700 1 -- 172.21.15.32:0/4103875764 shutdown_connections 2024-04-12T05:40:23.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.850+0000 7f34ee325700 1 -- 172.21.15.32:0/4103875764 wait complete. 2024-04-12T05:40:23.858 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:40:23.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.961+0000 7f97f221b700 1 -- 172.21.15.32:0/1059106202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97ec0fce80 msgr2=0x7f97ec0fd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:23.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.961+0000 7f97f221b700 1 --2- 172.21.15.32:0/1059106202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97ec0fce80 0x7f97ec0fd290 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f97dc004940 tx=0x7f97dc009e30 comp rx=0 tx=0).stop 2024-04-12T05:40:23.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.961+0000 7f97f221b700 1 -- 172.21.15.32:0/1059106202 shutdown_connections 2024-04-12T05:40:23.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.961+0000 7f97f221b700 1 --2- 172.21.15.32:0/1059106202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97ec0fdff0 0x7f97ec0fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:23.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.961+0000 7f97f221b700 1 --2- 172.21.15.32:0/1059106202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97ec0fce80 0x7f97ec0fd290 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:23.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.961+0000 7f97f221b700 1 -- 172.21.15.32:0/1059106202 >> 172.21.15.32:0/1059106202 conn(0x7f97ec0f8580 msgr2=0x7f97ec0fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:23.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.961+0000 7f97f221b700 1 -- 172.21.15.32:0/1059106202 shutdown_connections 2024-04-12T05:40:23.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.961+0000 7f97f221b700 1 -- 172.21.15.32:0/1059106202 wait complete. 2024-04-12T05:40:23.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.962+0000 7f97f221b700 1 Processor -- start 2024-04-12T05:40:23.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.962+0000 7f97f221b700 1 -- start start 2024-04-12T05:40:23.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.962+0000 7f97f221b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97ec0fce80 0x7f97ec10d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:40:23.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.962+0000 7f97f221b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97ec0fdff0 0x7f97ec10de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:40:23.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.962+0000 7f97f221b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec10e410 con 0x7f97ec0fdff0 2024-04-12T05:40:23.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.962+0000 7f97f221b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec10ec60 con 0x7f97ec0fce80 2024-04-12T05:40:23.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.962+0000 7f97eaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97ec0fdff0 0x7f97ec10de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:23.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.962+0000 7f97eaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97ec0fdff0 0x7f97ec10de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47004/0 (socket says 172.21.15.32:47004) 2024-04-12T05:40:23.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.962+0000 7f97eaffd700 1 -- 172.21.15.32:0/907297750 learned_addr learned my addr 172.21.15.32:0/907297750 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:40:23.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.963+0000 7f97eb7fe700 1 --2- 172.21.15.32:0/907297750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97ec0fce80 0x7f97ec10d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:23.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.963+0000 7f97eaffd700 1 -- 172.21.15.32:0/907297750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97ec0fce80 msgr2=0x7f97ec10d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:23.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.963+0000 7f97eaffd700 1 --2- 172.21.15.32:0/907297750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97ec0fce80 0x7f97ec10d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:23.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.963+0000 7f97eaffd700 1 -- 172.21.15.32:0/907297750 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97dc005040 con 0x7f97ec0fdff0 2024-04-12T05:40:23.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.963+0000 7f97eaffd700 1 --2- 172.21.15.32:0/907297750 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97ec0fdff0 0x7f97ec10de10 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f97e000eb40 tx=0x7f97e000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:23.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.963+0000 7f97d3fff700 1 -- 172.21.15.32:0/907297750 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97e0007cc0 con 0x7f97ec0fdff0 2024-04-12T05:40:23.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.963+0000 7f97d3fff700 1 -- 172.21.15.32:0/907297750 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f97e000f040 con 0x7f97ec0fdff0 2024-04-12T05:40:23.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.963+0000 7f97f221b700 1 -- 172.21.15.32:0/907297750 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97ec10eee0 con 0x7f97ec0fdff0 2024-04-12T05:40:23.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.963+0000 7f97d3fff700 1 -- 172.21.15.32:0/907297750 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97e000c990 con 0x7f97ec0fdff0 2024-04-12T05:40:23.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.963+0000 7f97f221b700 1 -- 172.21.15.32:0/907297750 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97ec10f420 con 0x7f97ec0fdff0 2024-04-12T05:40:23.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.964+0000 7f97f221b700 1 -- 172.21.15.32:0/907297750 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97ec05f5f0 con 0x7f97ec0fdff0 2024-04-12T05:40:23.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.965+0000 7f97d3fff700 1 -- 172.21.15.32:0/907297750 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 34) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f97e000cc00 con 0x7f97ec0fdff0 2024-04-12T05:40:23.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.966+0000 7f97d3fff700 1 --2- 172.21.15.32:0/907297750 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f97d406c8a0 0x7f97d406ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:40:23.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.966+0000 7f97d3fff700 1 -- 172.21.15.32:0/907297750 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f97e008b6f0 con 0x7f97ec0fdff0 2024-04-12T05:40:23.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.966+0000 7f97eb7fe700 1 --2- 172.21.15.32:0/907297750 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f97d406c8a0 0x7f97d406ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:23.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.966+0000 7f97eb7fe700 1 --2- 172.21.15.32:0/907297750 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f97d406c8a0 0x7f97d406ed50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f97dc00b0d0 tx=0x7f97dc009e30 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:23.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:23.971+0000 7f97d3fff700 1 -- 172.21.15.32:0/907297750 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f97e00596c0 con 0x7f97ec0fdff0 2024-04-12T05:40:24.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.196+0000 7f97f221b700 1 -- 172.21.15.32:0/907297750 --> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f97ec10f750 con 0x7f97d406c8a0 2024-04-12T05:40:24.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.197+0000 7f97d3fff700 1 -- 172.21.15.32:0/907297750 <== mgr.24463 v2:172.21.15.32:6800/2564765508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7f97ec10f750 con 0x7f97d406c8a0 2024-04-12T05:40:24.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.200+0000 7f97f221b700 1 -- 172.21.15.32:0/907297750 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f97d406c8a0 msgr2=0x7f97d406ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:24.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.200+0000 7f97f221b700 1 --2- 172.21.15.32:0/907297750 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f97d406c8a0 0x7f97d406ed50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f97dc00b0d0 tx=0x7f97dc009e30 comp rx=0 tx=0).stop 2024-04-12T05:40:24.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.200+0000 7f97f221b700 1 -- 172.21.15.32:0/907297750 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97ec0fdff0 msgr2=0x7f97ec10de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:24.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.200+0000 7f97f221b700 1 --2- 172.21.15.32:0/907297750 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97ec0fdff0 0x7f97ec10de10 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f97e000eb40 tx=0x7f97e000b040 comp rx=0 tx=0).stop 2024-04-12T05:40:24.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.200+0000 7f97f221b700 1 -- 172.21.15.32:0/907297750 shutdown_connections 2024-04-12T05:40:24.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.200+0000 7f97f221b700 1 --2- 172.21.15.32:0/907297750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97ec0fce80 0x7f97ec10d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:24.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.200+0000 7f97f221b700 1 --2- 172.21.15.32:0/907297750 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f97d406c8a0 0x7f97d406ed50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:24.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.200+0000 7f97f221b700 1 --2- 172.21.15.32:0/907297750 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97ec0fdff0 0x7f97ec10de10 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:24.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.200+0000 7f97f221b700 1 -- 172.21.15.32:0/907297750 >> 172.21.15.32:0/907297750 conn(0x7f97ec0f8580 msgr2=0x7f97ec0f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:24.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.200+0000 7f97f221b700 1 -- 172.21.15.32:0/907297750 shutdown_connections 2024-04-12T05:40:24.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.200+0000 7f97f221b700 1 -- 172.21.15.32:0/907297750 wait complete. 2024-04-12T05:40:24.266 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:23 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:24.266 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:23 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:24.266 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:23 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:40:24.266 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:23 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:40:24.266 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:23 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:24.266 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:23 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T05:40:24.266 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:23 smithi032 ceph-mon[39731]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T05:40:24.266 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:23 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:40:24.266 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:23 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:24.266 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:23 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:24.267 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:23 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:24.267 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:23 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:24.267 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:23 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:24.267 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:23 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:24.267 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:23 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:24.267 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:23 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:24.267 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:23 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:24.267 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:23 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:24.267 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:23 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:24.267 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:23 smithi032 ceph-mon[39731]: pgmap v51: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-12T05:40:24.267 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:23 smithi032 ceph-mon[39731]: from='client.14762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:40:24.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:23 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:24.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:23 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:24.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:23 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:40:24.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:23 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:40:24.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:23 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:24.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:23 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T05:40:24.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:23 smithi067 ceph-mon[49010]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T05:40:24.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:23 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:40:24.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:23 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:24.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:23 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:24.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:23 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:24.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:23 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:24.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:23 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:24.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:23 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:24.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:23 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:24.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:23 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:24.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:23 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:24.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:23 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:24.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:23 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:24.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:23 smithi067 ceph-mon[49010]: pgmap v51: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-12T05:40:24.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:23 smithi067 ceph-mon[49010]: from='client.14762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:40:24.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.304+0000 7ff72322f700 1 -- 172.21.15.32:0/655707462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff71c0fe060 msgr2=0x7ff71c0fe4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:24.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.304+0000 7ff72322f700 1 --2- 172.21.15.32:0/655707462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff71c0fe060 0x7ff71c0fe4d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff708004940 tx=0x7ff708009e30 comp rx=0 tx=0).stop 2024-04-12T05:40:24.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.304+0000 7ff72322f700 1 -- 172.21.15.32:0/655707462 shutdown_connections 2024-04-12T05:40:24.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.304+0000 7ff72322f700 1 --2- 172.21.15.32:0/655707462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff71c0fe060 0x7ff71c0fe4d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:24.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.304+0000 7ff72322f700 1 --2- 172.21.15.32:0/655707462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff71c0fcef0 0x7ff71c0fd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:24.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.304+0000 7ff72322f700 1 -- 172.21.15.32:0/655707462 >> 172.21.15.32:0/655707462 conn(0x7ff71c0f85b0 msgr2=0x7ff71c0fa9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:24.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.304+0000 7ff72322f700 1 -- 172.21.15.32:0/655707462 shutdown_connections 2024-04-12T05:40:24.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.304+0000 7ff72322f700 1 -- 172.21.15.32:0/655707462 wait complete. 2024-04-12T05:40:24.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.304+0000 7ff72322f700 1 Processor -- start 2024-04-12T05:40:24.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.304+0000 7ff72322f700 1 -- start start 2024-04-12T05:40:24.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.305+0000 7ff72322f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff71c0fcef0 0x7ff71c071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:40:24.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.305+0000 7ff72322f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff71c0fe060 0x7ff71c070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:40:24.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.305+0000 7ff72322f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71c0708c0 con 0x7ff71c0fcef0 2024-04-12T05:40:24.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.305+0000 7ff72322f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71c070a30 con 0x7ff71c0fe060 2024-04-12T05:40:24.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.305+0000 7ff720fcb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff71c0fcef0 0x7ff71c071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:24.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.305+0000 7ff720fcb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff71c0fcef0 0x7ff71c071e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47020/0 (socket says 172.21.15.32:47020) 2024-04-12T05:40:24.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.305+0000 7ff720fcb700 1 -- 172.21.15.32:0/1195215482 learned_addr learned my addr 172.21.15.32:0/1195215482 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:40:24.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.305+0000 7ff71bfff700 1 --2- 172.21.15.32:0/1195215482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff71c0fe060 0x7ff71c070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:24.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.305+0000 7ff720fcb700 1 -- 172.21.15.32:0/1195215482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff71c0fe060 msgr2=0x7ff71c070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:24.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.305+0000 7ff720fcb700 1 --2- 172.21.15.32:0/1195215482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff71c0fe060 0x7ff71c070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:24.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.305+0000 7ff720fcb700 1 -- 172.21.15.32:0/1195215482 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff708005040 con 0x7ff71c0fcef0 2024-04-12T05:40:24.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.305+0000 7ff720fcb700 1 --2- 172.21.15.32:0/1195215482 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff71c0fcef0 0x7ff71c071e40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff710002a50 tx=0x7ff7100046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:24.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.306+0000 7ff7197fa700 1 -- 172.21.15.32:0/1195215482 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff710007cc0 con 0x7ff71c0fcef0 2024-04-12T05:40:24.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.306+0000 7ff7197fa700 1 -- 172.21.15.32:0/1195215482 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff71000f040 con 0x7ff71c0fcef0 2024-04-12T05:40:24.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.306+0000 7ff72322f700 1 -- 172.21.15.32:0/1195215482 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff71c070d10 con 0x7ff71c0fcef0 2024-04-12T05:40:24.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.306+0000 7ff7197fa700 1 -- 172.21.15.32:0/1195215482 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff71000d9c0 con 0x7ff71c0fcef0 2024-04-12T05:40:24.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.306+0000 7ff72322f700 1 -- 172.21.15.32:0/1195215482 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff71c115970 con 0x7ff71c0fcef0 2024-04-12T05:40:24.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.307+0000 7ff72322f700 1 -- 172.21.15.32:0/1195215482 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff71c05f5f0 con 0x7ff71c0fcef0 2024-04-12T05:40:24.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.307+0000 7ff7197fa700 1 -- 172.21.15.32:0/1195215482 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 34) v1 ==== 90703+0+0 (secure 0 0 0) 0x7ff71000db20 con 0x7ff71c0fcef0 2024-04-12T05:40:24.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.307+0000 7ff7197fa700 1 --2- 172.21.15.32:0/1195215482 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7ff70c06c8a0 0x7ff70c06ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:40:24.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.307+0000 7ff7197fa700 1 -- 172.21.15.32:0/1195215482 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7ff71008b890 con 0x7ff71c0fcef0 2024-04-12T05:40:24.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.308+0000 7ff71bfff700 1 --2- 172.21.15.32:0/1195215482 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7ff70c06c8a0 0x7ff70c06ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:24.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.308+0000 7ff71bfff700 1 --2- 172.21.15.32:0/1195215482 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7ff70c06c8a0 0x7ff70c06ed50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff70800b900 tx=0x7ff708009e30 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:24.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.312+0000 7ff7197fa700 1 -- 172.21.15.32:0/1195215482 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7ff710055d90 con 0x7ff71c0fcef0 2024-04-12T05:40:24.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.523+0000 7ff72322f700 1 -- 172.21.15.32:0/1195215482 --> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff71c102440 con 0x7ff70c06c8a0 2024-04-12T05:40:24.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.533+0000 7ff7197fa700 1 -- 172.21.15.32:0/1195215482 <== mgr.24463 v2:172.21.15.32:6800/2564765508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7ff71c102440 con 0x7ff70c06c8a0 2024-04-12T05:40:24.530 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:40:24.531 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (12s) 5s ago 11m 16.9M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:40:24.531 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (11m) 5s ago 11m 6656k - 17.2.7-827-gd3b5060a 21de9b7369ad a9cfc00c311b 2024-04-12T05:40:24.531 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (10m) 45s ago 10m 6651k - 17.2.7-827-gd3b5060a 21de9b7369ad 1f2eea4542d9 2024-04-12T05:40:24.531 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (2m) 5s ago 11m 82.3M - 9.4.7 954c08fa6188 2229cad57cd3 2024-04-12T05:40:24.531 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (5m) 5s ago 5m 64.5M - 17.2.7-827-gd3b5060a 21de9b7369ad e7ada8e5259c 2024-04-12T05:40:24.531 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (5m) 5s ago 5m 116M - 17.2.7-827-gd3b5060a 21de9b7369ad 55124da9afdb 2024-04-12T05:40:24.531 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (5m) 45s ago 5m 62.5M - 17.2.7-827-gd3b5060a 21de9b7369ad 0c295d31a4a6 2024-04-12T05:40:24.531 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (5m) 45s ago 5m 120M - 17.2.7-827-gd3b5060a 21de9b7369ad 6eecd7deaac2 2024-04-12T05:40:24.531 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (112s) 5s ago 13m 501M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:40:24.531 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (65s) 45s ago 10m 449M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:40:24.531 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (13m) 5s ago 13m 53.2M 2048M 17.2.7-827-gd3b5060a 21de9b7369ad 75aad95b984d 2024-04-12T05:40:24.531 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (10m) 45s ago 10m 42.3M 2048M 17.2.7-827-gd3b5060a 21de9b7369ad 157be8a8c6d8 2024-04-12T05:40:24.531 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (53s) 5s ago 11m 8212k - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:40:24.531 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (49s) 45s ago 10m 3594k - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:40:24.532 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (9m) 5s ago 9m 219M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad 97446e42ceba 2024-04-12T05:40:24.532 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (9m) 5s ago 9m 235M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad bf9f9a2db2d5 2024-04-12T05:40:24.532 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (8m) 5s ago 8m 189M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad f055cf1ca665 2024-04-12T05:40:24.532 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (8m) 45s ago 8m 255M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad 61b851235351 2024-04-12T05:40:24.532 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (7m) 45s ago 7m 227M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad 2c84d9047e9c 2024-04-12T05:40:24.532 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (7m) 45s ago 7m 217M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad e54839bb6055 2024-04-12T05:40:24.532 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (29s) 5s ago 11m 46.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:40:24.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.535+0000 7ff72322f700 1 -- 172.21.15.32:0/1195215482 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7ff70c06c8a0 msgr2=0x7ff70c06ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:24.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.535+0000 7ff72322f700 1 --2- 172.21.15.32:0/1195215482 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7ff70c06c8a0 0x7ff70c06ed50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff70800b900 tx=0x7ff708009e30 comp rx=0 tx=0).stop 2024-04-12T05:40:24.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.535+0000 7ff72322f700 1 -- 172.21.15.32:0/1195215482 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff71c0fcef0 msgr2=0x7ff71c071e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:24.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.535+0000 7ff72322f700 1 --2- 172.21.15.32:0/1195215482 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff71c0fcef0 0x7ff71c071e40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff710002a50 tx=0x7ff7100046b0 comp rx=0 tx=0).stop 2024-04-12T05:40:24.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.535+0000 7ff72322f700 1 -- 172.21.15.32:0/1195215482 shutdown_connections 2024-04-12T05:40:24.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.535+0000 7ff72322f700 1 --2- 172.21.15.32:0/1195215482 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7ff70c06c8a0 0x7ff70c06ed50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:24.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.535+0000 7ff72322f700 1 --2- 172.21.15.32:0/1195215482 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff71c0fcef0 0x7ff71c071e40 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:24.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.535+0000 7ff72322f700 1 --2- 172.21.15.32:0/1195215482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff71c0fe060 0x7ff71c070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:24.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.535+0000 7ff72322f700 1 -- 172.21.15.32:0/1195215482 >> 172.21.15.32:0/1195215482 conn(0x7ff71c0f85b0 msgr2=0x7ff71c0f9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:24.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.535+0000 7ff72322f700 1 -- 172.21.15.32:0/1195215482 shutdown_connections 2024-04-12T05:40:24.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.535+0000 7ff72322f700 1 -- 172.21.15.32:0/1195215482 wait complete. 2024-04-12T05:40:24.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.638+0000 7f646ac1a700 1 -- 172.21.15.32:0/1762634401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64640fff00 msgr2=0x7f6464102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:24.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.638+0000 7f646ac1a700 1 --2- 172.21.15.32:0/1762634401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64640fff00 0x7f6464102320 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f6450004940 tx=0x7f6450009e30 comp rx=0 tx=0).stop 2024-04-12T05:40:24.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.638+0000 7f646ac1a700 1 -- 172.21.15.32:0/1762634401 shutdown_connections 2024-04-12T05:40:24.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.638+0000 7f646ac1a700 1 --2- 172.21.15.32:0/1762634401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64640fff00 0x7f6464102320 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:24.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.638+0000 7f646ac1a700 1 --2- 172.21.15.32:0/1762634401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64640662e0 0x7f64640ff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:24.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.638+0000 7f646ac1a700 1 -- 172.21.15.32:0/1762634401 >> 172.21.15.32:0/1762634401 conn(0x7f64640f7430 msgr2=0x7f64640f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:24.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.638+0000 7f646ac1a700 1 -- 172.21.15.32:0/1762634401 shutdown_connections 2024-04-12T05:40:24.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.638+0000 7f646ac1a700 1 -- 172.21.15.32:0/1762634401 wait complete. 2024-04-12T05:40:24.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.639+0000 7f646ac1a700 1 Processor -- start 2024-04-12T05:40:24.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.639+0000 7f646ac1a700 1 -- start start 2024-04-12T05:40:24.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.639+0000 7f646ac1a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64640662e0 0x7f64640fd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:40:24.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.639+0000 7f646ac1a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64640fff00 0x7f64640fb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:40:24.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.639+0000 7f646ac1a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64640fbe70 con 0x7f64640fff00 2024-04-12T05:40:24.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.639+0000 7f646ac1a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64640fbfe0 con 0x7f64640662e0 2024-04-12T05:40:24.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.639+0000 7f6463fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64640fff00 0x7f64640fb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:24.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.639+0000 7f6463fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64640fff00 0x7f64640fb8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47034/0 (socket says 172.21.15.32:47034) 2024-04-12T05:40:24.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.639+0000 7f6463fff700 1 -- 172.21.15.32:0/1935269076 learned_addr learned my addr 172.21.15.32:0/1935269076 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:40:24.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.640+0000 7f64689b6700 1 --2- 172.21.15.32:0/1935269076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64640662e0 0x7f64640fd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:24.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.640+0000 7f6463fff700 1 -- 172.21.15.32:0/1935269076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64640662e0 msgr2=0x7f64640fd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:24.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.640+0000 7f6463fff700 1 --2- 172.21.15.32:0/1935269076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64640662e0 0x7f64640fd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:24.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.640+0000 7f6463fff700 1 -- 172.21.15.32:0/1935269076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6450005040 con 0x7f64640fff00 2024-04-12T05:40:24.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.640+0000 7f6463fff700 1 --2- 172.21.15.32:0/1935269076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64640fff00 0x7f64640fb8c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f6450009fd0 tx=0x7f6450009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:24.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.640+0000 7f64617fa700 1 -- 172.21.15.32:0/1935269076 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f645000e040 con 0x7f64640fff00 2024-04-12T05:40:24.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.640+0000 7f646ac1a700 1 -- 172.21.15.32:0/1935269076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64640fc260 con 0x7f64640fff00 2024-04-12T05:40:24.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.640+0000 7f64617fa700 1 -- 172.21.15.32:0/1935269076 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6450004290 con 0x7f64640fff00 2024-04-12T05:40:24.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.640+0000 7f64617fa700 1 -- 172.21.15.32:0/1935269076 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f645001c3f0 con 0x7f64640fff00 2024-04-12T05:40:24.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.640+0000 7f646ac1a700 1 -- 172.21.15.32:0/1935269076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6464115990 con 0x7f64640fff00 2024-04-12T05:40:24.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.642+0000 7f646ac1a700 1 -- 172.21.15.32:0/1935269076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f646405f5f0 con 0x7f64640fff00 2024-04-12T05:40:24.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.642+0000 7f64617fa700 1 -- 172.21.15.32:0/1935269076 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 34) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f645000bb80 con 0x7f64640fff00 2024-04-12T05:40:24.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.643+0000 7f64617fa700 1 --2- 172.21.15.32:0/1935269076 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f645406c8b0 0x7f645406ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:40:24.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.643+0000 7f64617fa700 1 -- 172.21.15.32:0/1935269076 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f645008ce80 con 0x7f64640fff00 2024-04-12T05:40:24.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.643+0000 7f64689b6700 1 --2- 172.21.15.32:0/1935269076 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f645406c8b0 0x7f645406ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:24.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.643+0000 7f64689b6700 1 --2- 172.21.15.32:0/1935269076 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f645406c8b0 0x7f645406ed60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f64640fd180 tx=0x7f645800b040 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:24.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.648+0000 7f64617fa700 1 -- 172.21.15.32:0/1935269076 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f6450057390 con 0x7f64640fff00 2024-04-12T05:40:24.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.946+0000 7f646ac1a700 1 -- 172.21.15.32:0/1935269076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6464115b20 con 0x7f64640fff00 2024-04-12T05:40:24.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.946+0000 7f64617fa700 1 -- 172.21.15.32:0/1935269076 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7f645005a9b0 con 0x7f64640fff00 2024-04-12T05:40:24.943 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:40:24.943 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:40:24.943 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 2 2024-04-12T05:40:24.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:40:24.943 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:40:24.943 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:40:24.944 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:40:24.944 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:40:24.944 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 6 2024-04-12T05:40:24.944 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:40:24.944 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:40:24.944 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 4 2024-04-12T05:40:24.944 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:40:24.944 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:40:24.944 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 12, 2024-04-12T05:40:24.944 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:40:24.944 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:40:24.944 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:40:24.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.948+0000 7f646ac1a700 1 -- 172.21.15.32:0/1935269076 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f645406c8b0 msgr2=0x7f645406ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:24.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.948+0000 7f646ac1a700 1 --2- 172.21.15.32:0/1935269076 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f645406c8b0 0x7f645406ed60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f64640fd180 tx=0x7f645800b040 comp rx=0 tx=0).stop 2024-04-12T05:40:24.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.948+0000 7f646ac1a700 1 -- 172.21.15.32:0/1935269076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64640fff00 msgr2=0x7f64640fb8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:24.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.948+0000 7f646ac1a700 1 --2- 172.21.15.32:0/1935269076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64640fff00 0x7f64640fb8c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f6450009fd0 tx=0x7f6450009e30 comp rx=0 tx=0).stop 2024-04-12T05:40:24.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.949+0000 7f646ac1a700 1 -- 172.21.15.32:0/1935269076 shutdown_connections 2024-04-12T05:40:24.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.949+0000 7f646ac1a700 1 --2- 172.21.15.32:0/1935269076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64640662e0 0x7f64640fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:24.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.949+0000 7f646ac1a700 1 --2- 172.21.15.32:0/1935269076 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f645406c8b0 0x7f645406ed60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:24.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.949+0000 7f646ac1a700 1 --2- 172.21.15.32:0/1935269076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64640fff00 0x7f64640fb8c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:24.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.949+0000 7f646ac1a700 1 -- 172.21.15.32:0/1935269076 >> 172.21.15.32:0/1935269076 conn(0x7f64640f7430 msgr2=0x7f64640f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:24.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.949+0000 7f646ac1a700 1 -- 172.21.15.32:0/1935269076 shutdown_connections 2024-04-12T05:40:24.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:24.949+0000 7f646ac1a700 1 -- 172.21.15.32:0/1935269076 wait complete. 2024-04-12T05:40:25.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.053+0000 7f607af18700 1 -- 172.21.15.32:0/3128114284 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60740fdff0 msgr2=0x7f60740fe460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:25.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.053+0000 7f607af18700 1 --2- 172.21.15.32:0/3128114284 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60740fdff0 0x7f60740fe460 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f6060004940 tx=0x7f6060009e30 comp rx=0 tx=0).stop 2024-04-12T05:40:25.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.053+0000 7f607af18700 1 -- 172.21.15.32:0/3128114284 shutdown_connections 2024-04-12T05:40:25.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.053+0000 7f607af18700 1 --2- 172.21.15.32:0/3128114284 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60740fdff0 0x7f60740fe460 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:25.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.053+0000 7f607af18700 1 --2- 172.21.15.32:0/3128114284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60740fce80 0x7f60740fd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:25.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.053+0000 7f607af18700 1 -- 172.21.15.32:0/3128114284 >> 172.21.15.32:0/3128114284 conn(0x7f60740f8580 msgr2=0x7f60740fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:25.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.053+0000 7f607af18700 1 -- 172.21.15.32:0/3128114284 shutdown_connections 2024-04-12T05:40:25.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.053+0000 7f607af18700 1 -- 172.21.15.32:0/3128114284 wait complete. 2024-04-12T05:40:25.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.053+0000 7f607af18700 1 Processor -- start 2024-04-12T05:40:25.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.053+0000 7f607af18700 1 -- start start 2024-04-12T05:40:25.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.054+0000 7f607af18700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60740fce80 0x7f607410d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:40:25.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.054+0000 7f607af18700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60740fdff0 0x7f607410dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:40:25.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.054+0000 7f607af18700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607410e4a0 con 0x7f60740fdff0 2024-04-12T05:40:25.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.054+0000 7f607af18700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607410ec80 con 0x7f60740fce80 2024-04-12T05:40:25.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.054+0000 7f6073fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60740fdff0 0x7f607410dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:25.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.054+0000 7f6073fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60740fdff0 0x7f607410dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47052/0 (socket says 172.21.15.32:47052) 2024-04-12T05:40:25.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.054+0000 7f6073fff700 1 -- 172.21.15.32:0/2963233444 learned_addr learned my addr 172.21.15.32:0/2963233444 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:40:25.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.054+0000 7f6078cb4700 1 --2- 172.21.15.32:0/2963233444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60740fce80 0x7f607410d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:25.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.054+0000 7f6073fff700 1 -- 172.21.15.32:0/2963233444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60740fce80 msgr2=0x7f607410d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:25.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.054+0000 7f6073fff700 1 --2- 172.21.15.32:0/2963233444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60740fce80 0x7f607410d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:25.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.054+0000 7f6073fff700 1 -- 172.21.15.32:0/2963233444 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6060005040 con 0x7f60740fdff0 2024-04-12T05:40:25.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.054+0000 7f6073fff700 1 --2- 172.21.15.32:0/2963233444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60740fdff0 0x7f607410dea0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f606000b3b0 tx=0x7f6060009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:25.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.054+0000 7f60717fa700 1 -- 172.21.15.32:0/2963233444 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f606000e040 con 0x7f60740fdff0 2024-04-12T05:40:25.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.055+0000 7f607af18700 1 -- 172.21.15.32:0/2963233444 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f607410ef00 con 0x7f60740fdff0 2024-04-12T05:40:25.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.055+0000 7f60717fa700 1 -- 172.21.15.32:0/2963233444 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f606000bc50 con 0x7f60740fdff0 2024-04-12T05:40:25.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.055+0000 7f60717fa700 1 -- 172.21.15.32:0/2963233444 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f606001c630 con 0x7f60740fdff0 2024-04-12T05:40:25.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.055+0000 7f607af18700 1 -- 172.21.15.32:0/2963233444 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f607410f440 con 0x7f60740fdff0 2024-04-12T05:40:25.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.056+0000 7f607af18700 1 -- 172.21.15.32:0/2963233444 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60580052e0 con 0x7f60740fdff0 2024-04-12T05:40:25.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.056+0000 7f60717fa700 1 -- 172.21.15.32:0/2963233444 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 34) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f6060013420 con 0x7f60740fdff0 2024-04-12T05:40:25.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.057+0000 7f60717fa700 1 --2- 172.21.15.32:0/2963233444 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f606406c8a0 0x7f606406ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:40:25.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.057+0000 7f60717fa700 1 -- 172.21.15.32:0/2963233444 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f606008c640 con 0x7f60740fdff0 2024-04-12T05:40:25.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.057+0000 7f6078cb4700 1 --2- 172.21.15.32:0/2963233444 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f606406c8a0 0x7f606406ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:25.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.057+0000 7f6078cb4700 1 --2- 172.21.15.32:0/2963233444 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f606406c8a0 0x7f606406ed50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6068004f40 tx=0x7f606800b040 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:25.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.061+0000 7f60717fa700 1 -- 172.21.15.32:0/2963233444 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f606005a5e0 con 0x7f60740fdff0 2024-04-12T05:40:25.161 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:24 smithi032 ceph-mon[39731]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:40:25.161 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:24 smithi032 ceph-mon[39731]: from='client.14770 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:40:25.162 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:24 smithi032 ceph-mon[39731]: Upgrade: Updating grafana.smithi032 2024-04-12T05:40:25.162 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:24 smithi032 ceph-mon[39731]: Deploying daemon grafana.smithi032 on smithi032 2024-04-12T05:40:25.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:24 smithi067 ceph-mon[49010]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:40:25.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:24 smithi067 ceph-mon[49010]: from='client.14770 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:40:25.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:24 smithi067 ceph-mon[49010]: Upgrade: Updating grafana.smithi032 2024-04-12T05:40:25.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:24 smithi067 ceph-mon[49010]: Deploying daemon grafana.smithi032 on smithi032 2024-04-12T05:40:25.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.292+0000 7f607af18700 1 -- 172.21.15.32:0/2963233444 --> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6058000bf0 con 0x7f606406c8a0 2024-04-12T05:40:25.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.293+0000 7f60717fa700 1 -- 172.21.15.32:0/2963233444 <== mgr.24463 v2:172.21.15.32:6800/2564765508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7f6058000bf0 con 0x7f606406c8a0 2024-04-12T05:40:25.290 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:40:25.290 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:40:25.290 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:40:25.290 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-12T05:40:25.290 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:40:25.290 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr" 2024-04-12T05:40:25.290 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:40:25.290 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "2/2 daemons upgraded", 2024-04-12T05:40:25.290 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading grafana daemons", 2024-04-12T05:40:25.290 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:40:25.290 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:40:25.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.295+0000 7f607af18700 1 -- 172.21.15.32:0/2963233444 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f606406c8a0 msgr2=0x7f606406ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:25.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.295+0000 7f607af18700 1 --2- 172.21.15.32:0/2963233444 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f606406c8a0 0x7f606406ed50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6068004f40 tx=0x7f606800b040 comp rx=0 tx=0).stop 2024-04-12T05:40:25.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.295+0000 7f607af18700 1 -- 172.21.15.32:0/2963233444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60740fdff0 msgr2=0x7f607410dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:25.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.295+0000 7f607af18700 1 --2- 172.21.15.32:0/2963233444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60740fdff0 0x7f607410dea0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f606000b3b0 tx=0x7f6060009e30 comp rx=0 tx=0).stop 2024-04-12T05:40:25.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.295+0000 7f607af18700 1 -- 172.21.15.32:0/2963233444 shutdown_connections 2024-04-12T05:40:25.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.295+0000 7f607af18700 1 --2- 172.21.15.32:0/2963233444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60740fce80 0x7f607410d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:25.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.295+0000 7f607af18700 1 --2- 172.21.15.32:0/2963233444 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f606406c8a0 0x7f606406ed50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:25.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.295+0000 7f607af18700 1 --2- 172.21.15.32:0/2963233444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60740fdff0 0x7f607410dea0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:25.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.295+0000 7f607af18700 1 -- 172.21.15.32:0/2963233444 >> 172.21.15.32:0/2963233444 conn(0x7f60740f8580 msgr2=0x7f60740fa090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:25.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.295+0000 7f607af18700 1 -- 172.21.15.32:0/2963233444 shutdown_connections 2024-04-12T05:40:25.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:25.295+0000 7f607af18700 1 -- 172.21.15.32:0/2963233444 wait complete. 2024-04-12T05:40:26.237 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:25 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/1935269076' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:26.238 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:25 smithi032 ceph-mon[39731]: pgmap v52: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-12T05:40:26.238 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:25 smithi032 ceph-mon[39731]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:40:26.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:25 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/1935269076' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:26.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:25 smithi067 ceph-mon[49010]: pgmap v52: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-12T05:40:26.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:25 smithi067 ceph-mon[49010]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:40:28.409 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:28 smithi032 ceph-mon[39731]: pgmap v53: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-12T05:40:28.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:28 smithi067 ceph-mon[49010]: pgmap v53: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-12T05:40:31.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:30 smithi067 ceph-mon[49010]: pgmap v54: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-12T05:40:31.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:30 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:31.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:30 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:31.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:30 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:40:31.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:30 smithi032 ceph-mon[39731]: pgmap v54: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-12T05:40:31.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:30 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:31.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:30 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:31.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:30 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:40:33.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:32 smithi067 ceph-mon[49010]: pgmap v55: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:40:33.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:32 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:40:33.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:32 smithi032 ceph-mon[39731]: pgmap v55: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:40:33.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:32 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:40:35.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:34 smithi067 ceph-mon[49010]: pgmap v56: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:40:35.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:34 smithi032 ceph-mon[39731]: pgmap v56: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:40:36.336 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:36 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:36.336 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:36 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:36.336 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:36 smithi032 ceph-mon[39731]: pgmap v57: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:40:36.336 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:36 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:36.336 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:36 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:36.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:36 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:36.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:36 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:36.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:36 smithi067 ceph-mon[49010]: pgmap v57: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:40:36.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:36 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:36.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:36 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:38.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:38 smithi067 ceph-mon[49010]: pgmap v58: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:40:38.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:38 smithi032 ceph-mon[39731]: pgmap v58: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:40:40.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: pgmap v59: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-12T05:40:40.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:40.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:40.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:40:40.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:40:40.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:40.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T05:40:40.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T05:40:40.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:40:40.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:40.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:40.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:40.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:40.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:40.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:40.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:40.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:40.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:40.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:40.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:40.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:40.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:40.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:40.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: Upgrade: Finalizing container_image settings 2024-04-12T05:40:40.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:40.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-12T05:40:40.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-12T05:40:40.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:40:40.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-12T05:40:40.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-12T05:40:40.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-12T05:40:40.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-12T05:40:40.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-12T05:40:40.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-12T05:40:40.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-12T05:40:40.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-12T05:40:40.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:40:40.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-12T05:40:40.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-12T05:40:40.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-12T05:40:40.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-12T05:40:40.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-12T05:40:40.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-12T05:40:40.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-12T05:40:40.593 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-12T05:40:40.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:40:40.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:40:40.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:40:40.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:40:40.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:40:40.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:40:40.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-04-12T05:40:40.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-04-12T05:40:40.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: Upgrade: Complete! 2024-04-12T05:40:40.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-12T05:40:40.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-12T05:40:40.594 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:40:40.595 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:40:40.595 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:40:40.595 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:40.595 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:40:40.595 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:40:40.595 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:40:40.595 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:40 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:40.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: pgmap v59: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-12T05:40:40.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:40.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:40.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:40:40.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:40:40.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:40.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T05:40:40.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T05:40:40.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:40:40.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:40.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:40.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:40.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:40.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:40.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:40.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:40.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:40.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:40.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:40.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:40.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:40.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:40.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:40.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: Upgrade: Finalizing container_image settings 2024-04-12T05:40:40.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-12T05:40:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-12T05:40:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:40:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-12T05:40:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-12T05:40:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-12T05:40:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-12T05:40:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-12T05:40:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-12T05:40:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-12T05:40:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-12T05:40:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:40:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-12T05:40:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-12T05:40:40.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-12T05:40:40.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-12T05:40:40.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-12T05:40:40.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-12T05:40:40.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-12T05:40:40.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-12T05:40:40.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:40:40.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:40:40.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:40:40.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:40:40.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:40:40.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:40:40.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-04-12T05:40:40.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-04-12T05:40:40.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: Upgrade: Complete! 2024-04-12T05:40:40.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-12T05:40:40.773 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-12T05:40:40.773 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:40:40.773 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:40:40.773 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:40:40.773 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:40.773 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:40:40.773 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:40:40.773 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:40:40.773 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:42.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:42 smithi032 ceph-mon[39731]: pgmap v60: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-12T05:40:42.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:42 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:42.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:42 smithi067 ceph-mon[49010]: pgmap v60: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-12T05:40:42.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:42 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:40:44.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:44 smithi032 ceph-mon[39731]: pgmap v61: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-12T05:40:44.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:44 smithi067 ceph-mon[49010]: pgmap v61: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-12T05:40:46.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:46 smithi032 ceph-mon[39731]: pgmap v62: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-12T05:40:46.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:46 smithi067 ceph-mon[49010]: pgmap v62: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-12T05:40:47.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:47 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:40:47.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:47 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:40:48.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:48 smithi032 ceph-mon[39731]: pgmap v63: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-12T05:40:48.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:48 smithi067 ceph-mon[49010]: pgmap v63: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-12T05:40:50.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:50 smithi032 ceph-mon[39731]: pgmap v64: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-12T05:40:50.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:50 smithi067 ceph-mon[49010]: pgmap v64: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-12T05:40:52.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:52 smithi032 ceph-mon[39731]: pgmap v65: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:40:52.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:52 smithi067 ceph-mon[49010]: pgmap v65: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:40:54.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:54 smithi032 ceph-mon[39731]: pgmap v66: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:40:54.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:54 smithi067 ceph-mon[49010]: pgmap v66: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:40:55.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.413+0000 7f3f26f9b700 1 -- 172.21.15.32:0/2968266476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f200fce40 msgr2=0x7f3f200fd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:55.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.413+0000 7f3f26f9b700 1 --2- 172.21.15.32:0/2968266476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f200fce40 0x7f3f200fd250 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3f14004940 tx=0x7f3f14009e30 comp rx=0 tx=0).stop 2024-04-12T05:40:55.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.413+0000 7f3f26f9b700 1 -- 172.21.15.32:0/2968266476 shutdown_connections 2024-04-12T05:40:55.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.413+0000 7f3f26f9b700 1 --2- 172.21.15.32:0/2968266476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f200fe040 0x7f3f200fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:55.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.413+0000 7f3f26f9b700 1 --2- 172.21.15.32:0/2968266476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f200fce40 0x7f3f200fd250 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:55.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.413+0000 7f3f26f9b700 1 -- 172.21.15.32:0/2968266476 >> 172.21.15.32:0/2968266476 conn(0x7f3f200f8550 msgr2=0x7f3f200fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:55.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.413+0000 7f3f26f9b700 1 -- 172.21.15.32:0/2968266476 shutdown_connections 2024-04-12T05:40:55.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.413+0000 7f3f26f9b700 1 -- 172.21.15.32:0/2968266476 wait complete. 2024-04-12T05:40:55.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.414+0000 7f3f26f9b700 1 Processor -- start 2024-04-12T05:40:55.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.414+0000 7f3f26f9b700 1 -- start start 2024-04-12T05:40:55.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.414+0000 7f3f26f9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f200fce40 0x7f3f2010d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:40:55.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.414+0000 7f3f26f9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f200fe040 0x7f3f2010de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:40:55.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.414+0000 7f3f26f9b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2010e490 con 0x7f3f200fce40 2024-04-12T05:40:55.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.414+0000 7f3f26f9b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2010ece0 con 0x7f3f200fe040 2024-04-12T05:40:55.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.414+0000 7f3f24d37700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f200fce40 0x7f3f2010d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:55.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.414+0000 7f3f24d37700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f200fce40 0x7f3f2010d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42660/0 (socket says 172.21.15.32:42660) 2024-04-12T05:40:55.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.414+0000 7f3f24d37700 1 -- 172.21.15.32:0/4013669936 learned_addr learned my addr 172.21.15.32:0/4013669936 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:40:55.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.415+0000 7f3f1ffff700 1 --2- 172.21.15.32:0/4013669936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f200fe040 0x7f3f2010de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:55.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.415+0000 7f3f24d37700 1 -- 172.21.15.32:0/4013669936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f200fe040 msgr2=0x7f3f2010de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:55.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.415+0000 7f3f24d37700 1 --2- 172.21.15.32:0/4013669936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f200fe040 0x7f3f2010de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:55.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.415+0000 7f3f24d37700 1 -- 172.21.15.32:0/4013669936 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f14005040 con 0x7f3f200fce40 2024-04-12T05:40:55.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.415+0000 7f3f1ffff700 1 --2- 172.21.15.32:0/4013669936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f200fe040 0x7f3f2010de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:40:55.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.415+0000 7f3f24d37700 1 --2- 172.21.15.32:0/4013669936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f200fce40 0x7f3f2010d950 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3f1400b900 tx=0x7f3f14009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:55.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.415+0000 7f3f1d7fa700 1 -- 172.21.15.32:0/4013669936 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f1400e040 con 0x7f3f200fce40 2024-04-12T05:40:55.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.415+0000 7f3f26f9b700 1 -- 172.21.15.32:0/4013669936 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f2010ef60 con 0x7f3f200fce40 2024-04-12T05:40:55.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.415+0000 7f3f1d7fa700 1 -- 172.21.15.32:0/4013669936 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f1400bc50 con 0x7f3f200fce40 2024-04-12T05:40:55.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.415+0000 7f3f1d7fa700 1 -- 172.21.15.32:0/4013669936 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f1401c630 con 0x7f3f200fce40 2024-04-12T05:40:55.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.415+0000 7f3f26f9b700 1 -- 172.21.15.32:0/4013669936 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f2010f4a0 con 0x7f3f200fce40 2024-04-12T05:40:55.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.416+0000 7f3f26f9b700 1 -- 172.21.15.32:0/4013669936 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f2005f5f0 con 0x7f3f200fce40 2024-04-12T05:40:55.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.417+0000 7f3f1d7fa700 1 -- 172.21.15.32:0/4013669936 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 34) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f3f140139d0 con 0x7f3f200fce40 2024-04-12T05:40:55.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.417+0000 7f3f1d7fa700 1 --2- 172.21.15.32:0/4013669936 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f3f1006c8a0 0x7f3f1006ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:40:55.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.417+0000 7f3f1d7fa700 1 -- 172.21.15.32:0/4013669936 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f3f1408ce80 con 0x7f3f200fce40 2024-04-12T05:40:55.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.417+0000 7f3f1ffff700 1 --2- 172.21.15.32:0/4013669936 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f3f1006c8a0 0x7f3f1006ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:55.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.418+0000 7f3f1ffff700 1 --2- 172.21.15.32:0/4013669936 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f3f1006c8a0 0x7f3f1006ed50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c009fd0 tx=0x7f3f0c00b040 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:55.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.422+0000 7f3f1d7fa700 1 -- 172.21.15.32:0/4013669936 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f3f1405ae20 con 0x7f3f200fce40 2024-04-12T05:40:55.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.643+0000 7f3f26f9b700 1 -- 172.21.15.32:0/4013669936 --> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f2010f7d0 con 0x7f3f1006c8a0 2024-04-12T05:40:55.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.644+0000 7f3f1d7fa700 1 -- 172.21.15.32:0/4013669936 <== mgr.24463 v2:172.21.15.32:6800/2564765508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f3f2010f7d0 con 0x7f3f1006c8a0 2024-04-12T05:40:55.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.646+0000 7f3f26f9b700 1 -- 172.21.15.32:0/4013669936 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f3f1006c8a0 msgr2=0x7f3f1006ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:55.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.646+0000 7f3f26f9b700 1 --2- 172.21.15.32:0/4013669936 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f3f1006c8a0 0x7f3f1006ed50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c009fd0 tx=0x7f3f0c00b040 comp rx=0 tx=0).stop 2024-04-12T05:40:55.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.646+0000 7f3f26f9b700 1 -- 172.21.15.32:0/4013669936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f200fce40 msgr2=0x7f3f2010d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:55.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.646+0000 7f3f26f9b700 1 --2- 172.21.15.32:0/4013669936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f200fce40 0x7f3f2010d950 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3f1400b900 tx=0x7f3f14009e30 comp rx=0 tx=0).stop 2024-04-12T05:40:55.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.646+0000 7f3f26f9b700 1 -- 172.21.15.32:0/4013669936 shutdown_connections 2024-04-12T05:40:55.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.647+0000 7f3f26f9b700 1 --2- 172.21.15.32:0/4013669936 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f3f1006c8a0 0x7f3f1006ed50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:55.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.647+0000 7f3f26f9b700 1 --2- 172.21.15.32:0/4013669936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f200fce40 0x7f3f2010d950 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:55.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.647+0000 7f3f26f9b700 1 --2- 172.21.15.32:0/4013669936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f200fe040 0x7f3f2010de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:55.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.647+0000 7f3f26f9b700 1 -- 172.21.15.32:0/4013669936 >> 172.21.15.32:0/4013669936 conn(0x7f3f200f8550 msgr2=0x7f3f200f9d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:55.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.647+0000 7f3f26f9b700 1 -- 172.21.15.32:0/4013669936 shutdown_connections 2024-04-12T05:40:55.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:55.647+0000 7f3f26f9b700 1 -- 172.21.15.32:0/4013669936 wait complete. 2024-04-12T05:40:56.473 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-04-12T05:40:56.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:56 smithi032 ceph-mon[39731]: pgmap v67: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:40:56.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:56 smithi032 ceph-mon[39731]: from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:40:56.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:56 smithi067 ceph-mon[49010]: pgmap v67: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:40:56.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:56 smithi067 ceph-mon[49010]: from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:40:56.801 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:40:58.482 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:58 smithi032 ceph-mon[39731]: pgmap v68: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:40:58.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:58 smithi067 ceph-mon[49010]: pgmap v68: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:40:58.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.780+0000 7fcf53dae700 1 -- 172.21.15.32:0/464047812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf4c102710 msgr2=0x7fcf4c0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:58.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.780+0000 7fcf53dae700 1 --2- 172.21.15.32:0/464047812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf4c102710 0x7fcf4c0fbd80 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fcf48004940 tx=0x7fcf48009e30 comp rx=0 tx=0).stop 2024-04-12T05:40:58.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.780+0000 7fcf53dae700 1 -- 172.21.15.32:0/464047812 shutdown_connections 2024-04-12T05:40:58.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.780+0000 7fcf53dae700 1 --2- 172.21.15.32:0/464047812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf4c102710 0x7fcf4c0fbd80 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:58.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.780+0000 7fcf53dae700 1 --2- 172.21.15.32:0/464047812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf4c101d70 0x7fcf4c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:58.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.780+0000 7fcf53dae700 1 -- 172.21.15.32:0/464047812 >> 172.21.15.32:0/464047812 conn(0x7fcf4c072bc0 msgr2=0x7fcf4c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:58.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.780+0000 7fcf53dae700 1 -- 172.21.15.32:0/464047812 shutdown_connections 2024-04-12T05:40:58.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.780+0000 7fcf53dae700 1 -- 172.21.15.32:0/464047812 wait complete. 2024-04-12T05:40:58.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.781+0000 7fcf53dae700 1 Processor -- start 2024-04-12T05:40:58.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.781+0000 7fcf53dae700 1 -- start start 2024-04-12T05:40:58.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.781+0000 7fcf53dae700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf4c101d70 0x7fcf4c112900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:40:58.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.781+0000 7fcf53dae700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf4c102710 0x7fcf4c112e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:40:58.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.781+0000 7fcf53dae700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4c113490 con 0x7fcf4c101d70 2024-04-12T05:40:58.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.781+0000 7fcf53dae700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4c10c880 con 0x7fcf4c102710 2024-04-12T05:40:58.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.781+0000 7fcf51b4a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf4c101d70 0x7fcf4c112900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:58.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.781+0000 7fcf51b4a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf4c101d70 0x7fcf4c112900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42680/0 (socket says 172.21.15.32:42680) 2024-04-12T05:40:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.781+0000 7fcf51b4a700 1 -- 172.21.15.32:0/3650104387 learned_addr learned my addr 172.21.15.32:0/3650104387 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:40:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.781+0000 7fcf51349700 1 --2- 172.21.15.32:0/3650104387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf4c102710 0x7fcf4c112e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.781+0000 7fcf51b4a700 1 -- 172.21.15.32:0/3650104387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf4c102710 msgr2=0x7fcf4c112e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.781+0000 7fcf51b4a700 1 --2- 172.21.15.32:0/3650104387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf4c102710 0x7fcf4c112e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.781+0000 7fcf51b4a700 1 -- 172.21.15.32:0/3650104387 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf48005040 con 0x7fcf4c101d70 2024-04-12T05:40:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.782+0000 7fcf51b4a700 1 --2- 172.21.15.32:0/3650104387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf4c101d70 0x7fcf4c112900 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fcf4000fd10 tx=0x7fcf400046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.782+0000 7fcf3e7fc700 1 -- 172.21.15.32:0/3650104387 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf4000dcf0 con 0x7fcf4c101d70 2024-04-12T05:40:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.782+0000 7fcf3e7fc700 1 -- 172.21.15.32:0/3650104387 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf4000de50 con 0x7fcf4c101d70 2024-04-12T05:40:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.782+0000 7fcf53dae700 1 -- 172.21.15.32:0/3650104387 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf4c10cb00 con 0x7fcf4c101d70 2024-04-12T05:40:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.782+0000 7fcf3e7fc700 1 -- 172.21.15.32:0/3650104387 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf400126a0 con 0x7fcf4c101d70 2024-04-12T05:40:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.782+0000 7fcf53dae700 1 -- 172.21.15.32:0/3650104387 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf4c10d090 con 0x7fcf4c101d70 2024-04-12T05:40:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.783+0000 7fcf53dae700 1 -- 172.21.15.32:0/3650104387 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf4c0fd4c0 con 0x7fcf4c101d70 2024-04-12T05:40:58.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.783+0000 7fcf3e7fc700 1 -- 172.21.15.32:0/3650104387 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 34) v1 ==== 90703+0+0 (secure 0 0 0) 0x7fcf40007ab0 con 0x7fcf4c101d70 2024-04-12T05:40:58.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.784+0000 7fcf3e7fc700 1 --2- 172.21.15.32:0/3650104387 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fcf3806c8a0 0x7fcf3806ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:40:58.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.784+0000 7fcf3e7fc700 1 -- 172.21.15.32:0/3650104387 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7fcf4008df00 con 0x7fcf4c101d70 2024-04-12T05:40:58.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.784+0000 7fcf51349700 1 --2- 172.21.15.32:0/3650104387 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fcf3806c8a0 0x7fcf3806ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:58.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.784+0000 7fcf51349700 1 --2- 172.21.15.32:0/3650104387 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fcf3806c8a0 0x7fcf3806ed50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c10db60 tx=0x7fcf48009e30 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:58.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:58.788+0000 7fcf3e7fc700 1 -- 172.21.15.32:0/3650104387 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fcf4005bea0 con 0x7fcf4c101d70 2024-04-12T05:40:59.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:59.076+0000 7fcf53dae700 1 -- 172.21.15.32:0/3650104387 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fcf4c04bed0 con 0x7fcf4c101d70 2024-04-12T05:40:59.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:59.077+0000 7fcf3e7fc700 1 -- 172.21.15.32:0/3650104387 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7fcf4005ba30 con 0x7fcf4c101d70 2024-04-12T05:40:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:59.078+0000 7fcf53dae700 1 -- 172.21.15.32:0/3650104387 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fcf3806c8a0 msgr2=0x7fcf3806ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:59.078+0000 7fcf53dae700 1 --2- 172.21.15.32:0/3650104387 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fcf3806c8a0 0x7fcf3806ed50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c10db60 tx=0x7fcf48009e30 comp rx=0 tx=0).stop 2024-04-12T05:40:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:59.078+0000 7fcf53dae700 1 -- 172.21.15.32:0/3650104387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf4c101d70 msgr2=0x7fcf4c112900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:59.078+0000 7fcf53dae700 1 --2- 172.21.15.32:0/3650104387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf4c101d70 0x7fcf4c112900 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fcf4000fd10 tx=0x7fcf400046b0 comp rx=0 tx=0).stop 2024-04-12T05:40:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:59.079+0000 7fcf53dae700 1 -- 172.21.15.32:0/3650104387 shutdown_connections 2024-04-12T05:40:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:59.079+0000 7fcf53dae700 1 --2- 172.21.15.32:0/3650104387 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fcf3806c8a0 0x7fcf3806ed50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:59.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:59.079+0000 7fcf53dae700 1 --2- 172.21.15.32:0/3650104387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf4c101d70 0x7fcf4c112900 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:59.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:59.079+0000 7fcf53dae700 1 --2- 172.21.15.32:0/3650104387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf4c102710 0x7fcf4c112e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:40:59.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:59.079+0000 7fcf53dae700 1 -- 172.21.15.32:0/3650104387 >> 172.21.15.32:0/3650104387 conn(0x7fcf4c072bc0 msgr2=0x7fcf4c0faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:59.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:59.079+0000 7fcf53dae700 1 -- 172.21.15.32:0/3650104387 shutdown_connections 2024-04-12T05:40:59.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:40:59.079+0000 7fcf53dae700 1 -- 172.21.15.32:0/3650104387 wait complete. 2024-04-12T05:40:59.086 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:40:59.508 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:40:59 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/3650104387' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:59.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:40:59 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/3650104387' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:59.947 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-04-12T05:41:00.234 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:41:00.685 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:00 smithi032 ceph-mon[39731]: pgmap v69: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:41:00.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:00 smithi067 ceph-mon[49010]: pgmap v69: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:41:01.555 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:01 smithi032 ceph-mon[39731]: fsmap cephfs:2 {0=cephfs.smithi032.pndtry=up:active,1=cephfs.smithi067.zesmgy=up:active} 2 up:standby-replay 2024-04-12T05:41:01.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:01 smithi067 ceph-mon[49010]: fsmap cephfs:2 {0=cephfs.smithi032.pndtry=up:active,1=cephfs.smithi067.zesmgy=up:active} 2 up:standby-replay 2024-04-12T05:41:02.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.116+0000 7f7c95dd6700 1 -- 172.21.15.32:0/1350143188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c90102710 msgr2=0x7f7c900fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:02.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.116+0000 7f7c95dd6700 1 --2- 172.21.15.32:0/1350143188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c90102710 0x7f7c900fbd80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7c80004940 tx=0x7f7c80009e30 comp rx=0 tx=0).stop 2024-04-12T05:41:02.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.116+0000 7f7c95dd6700 1 -- 172.21.15.32:0/1350143188 shutdown_connections 2024-04-12T05:41:02.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.116+0000 7f7c95dd6700 1 --2- 172.21.15.32:0/1350143188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c90102710 0x7f7c900fbd80 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:02.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.116+0000 7f7c95dd6700 1 --2- 172.21.15.32:0/1350143188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c90101d70 0x7f7c90102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:02.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.116+0000 7f7c95dd6700 1 -- 172.21.15.32:0/1350143188 >> 172.21.15.32:0/1350143188 conn(0x7f7c90072bc0 msgr2=0x7f7c90072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:02.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.116+0000 7f7c95dd6700 1 -- 172.21.15.32:0/1350143188 shutdown_connections 2024-04-12T05:41:02.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.116+0000 7f7c95dd6700 1 -- 172.21.15.32:0/1350143188 wait complete. 2024-04-12T05:41:02.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.117+0000 7f7c95dd6700 1 Processor -- start 2024-04-12T05:41:02.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.117+0000 7f7c95dd6700 1 -- start start 2024-04-12T05:41:02.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.117+0000 7f7c95dd6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c90101d70 0x7f7c90106c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:02.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.117+0000 7f7c95dd6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c90102710 0x7f7c90107150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:02.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.117+0000 7f7c95dd6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c901076e0 con 0x7f7c90101d70 2024-04-12T05:41:02.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.117+0000 7f7c95dd6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c90107820 con 0x7f7c90102710 2024-04-12T05:41:02.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.117+0000 7f7c8f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c90101d70 0x7f7c90106c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:02.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.117+0000 7f7c8f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c90101d70 0x7f7c90106c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42526/0 (socket says 172.21.15.32:42526) 2024-04-12T05:41:02.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.117+0000 7f7c8f7fe700 1 -- 172.21.15.32:0/95643314 learned_addr learned my addr 172.21.15.32:0/95643314 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:41:02.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.118+0000 7f7c8effd700 1 --2- 172.21.15.32:0/95643314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c90102710 0x7f7c90107150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:02.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.118+0000 7f7c8f7fe700 1 -- 172.21.15.32:0/95643314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c90102710 msgr2=0x7f7c90107150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:02.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.118+0000 7f7c8f7fe700 1 --2- 172.21.15.32:0/95643314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c90102710 0x7f7c90107150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:02.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.118+0000 7f7c8f7fe700 1 -- 172.21.15.32:0/95643314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c80005040 con 0x7f7c90101d70 2024-04-12T05:41:02.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.118+0000 7f7c8effd700 1 --2- 172.21.15.32:0/95643314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c90102710 0x7f7c90107150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:41:02.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.118+0000 7f7c8f7fe700 1 --2- 172.21.15.32:0/95643314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c90101d70 0x7f7c90106c10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7c8400c830 tx=0x7f7c84004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:02.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.118+0000 7f7c77fff700 1 -- 172.21.15.32:0/95643314 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c8400a760 con 0x7f7c90101d70 2024-04-12T05:41:02.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.118+0000 7f7c77fff700 1 -- 172.21.15.32:0/95643314 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c8400ad70 con 0x7f7c90101d70 2024-04-12T05:41:02.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.118+0000 7f7c95dd6700 1 -- 172.21.15.32:0/95643314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c90119db0 con 0x7f7c90101d70 2024-04-12T05:41:02.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.118+0000 7f7c77fff700 1 -- 172.21.15.32:0/95643314 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c840113a0 con 0x7f7c90101d70 2024-04-12T05:41:02.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.118+0000 7f7c95dd6700 1 -- 172.21.15.32:0/95643314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c9011a230 con 0x7f7c90101d70 2024-04-12T05:41:02.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.119+0000 7f7c95dd6700 1 -- 172.21.15.32:0/95643314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c900ff080 con 0x7f7c90101d70 2024-04-12T05:41:02.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.120+0000 7f7c77fff700 1 -- 172.21.15.32:0/95643314 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 34) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f7c84012400 con 0x7f7c90101d70 2024-04-12T05:41:02.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.120+0000 7f7c77fff700 1 --2- 172.21.15.32:0/95643314 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f7c7806c8a0 0x7f7c7806ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:02.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.120+0000 7f7c77fff700 1 -- 172.21.15.32:0/95643314 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f7c8408bb40 con 0x7f7c90101d70 2024-04-12T05:41:02.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.120+0000 7f7c8effd700 1 --2- 172.21.15.32:0/95643314 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f7c7806c8a0 0x7f7c7806ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:02.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.121+0000 7f7c8effd700 1 --2- 172.21.15.32:0/95643314 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f7c7806c8a0 0x7f7c7806ed50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7c8000b3b0 tx=0x7f7c80009e30 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:02.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.125+0000 7f7c77fff700 1 -- 172.21.15.32:0/95643314 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f7c84056050 con 0x7f7c90101d70 2024-04-12T05:41:02.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.415+0000 7f7c95dd6700 1 -- 172.21.15.32:0/95643314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7c900fd4c0 con 0x7f7c90101d70 2024-04-12T05:41:02.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.415+0000 7f7c77fff700 1 -- 172.21.15.32:0/95643314 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7f7c84059670 con 0x7f7c90101d70 2024-04-12T05:41:02.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.417+0000 7f7c95dd6700 1 -- 172.21.15.32:0/95643314 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f7c7806c8a0 msgr2=0x7f7c7806ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:02.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.417+0000 7f7c95dd6700 1 --2- 172.21.15.32:0/95643314 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f7c7806c8a0 0x7f7c7806ed50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7c8000b3b0 tx=0x7f7c80009e30 comp rx=0 tx=0).stop 2024-04-12T05:41:02.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.417+0000 7f7c95dd6700 1 -- 172.21.15.32:0/95643314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c90101d70 msgr2=0x7f7c90106c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:02.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.417+0000 7f7c95dd6700 1 --2- 172.21.15.32:0/95643314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c90101d70 0x7f7c90106c10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7c8400c830 tx=0x7f7c84004470 comp rx=0 tx=0).stop 2024-04-12T05:41:02.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.417+0000 7f7c95dd6700 1 -- 172.21.15.32:0/95643314 shutdown_connections 2024-04-12T05:41:02.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.417+0000 7f7c95dd6700 1 --2- 172.21.15.32:0/95643314 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f7c7806c8a0 0x7f7c7806ed50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:02.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.417+0000 7f7c95dd6700 1 --2- 172.21.15.32:0/95643314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c90101d70 0x7f7c90106c10 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:02.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.417+0000 7f7c95dd6700 1 --2- 172.21.15.32:0/95643314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c90102710 0x7f7c90107150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:02.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.417+0000 7f7c95dd6700 1 -- 172.21.15.32:0/95643314 >> 172.21.15.32:0/95643314 conn(0x7f7c90072bc0 msgr2=0x7f7c900fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:02.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.417+0000 7f7c95dd6700 1 -- 172.21.15.32:0/95643314 shutdown_connections 2024-04-12T05:41:02.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:02.418+0000 7f7c95dd6700 1 -- 172.21.15.32:0/95643314 wait complete. 2024-04-12T05:41:02.425 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)" 2024-04-12T05:41:02.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:02 smithi067 ceph-mon[49010]: pgmap v70: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:41:02.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:02 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:41:02.797 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:02 smithi032 ceph-mon[39731]: pgmap v70: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:41:02.797 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:02 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:41:03.116 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 2'"'"'' 2024-04-12T05:41:03.400 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:41:03.650 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:03 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/95643314' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:41:03.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:03 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/95643314' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:41:04.452 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:04 smithi032 ceph-mon[39731]: pgmap v71: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:41:04.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:04 smithi067 ceph-mon[49010]: pgmap v71: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:41:05.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.042+0000 7f3c306ab700 1 -- 172.21.15.32:0/2645340643 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c28102e10 msgr2=0x7f3c281031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:05.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.042+0000 7f3c306ab700 1 --2- 172.21.15.32:0/2645340643 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c28102e10 0x7f3c281031e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f3c18004940 tx=0x7f3c18009e30 comp rx=0 tx=0).stop 2024-04-12T05:41:05.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.042+0000 7f3c306ab700 1 -- 172.21.15.32:0/2645340643 shutdown_connections 2024-04-12T05:41:05.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.042+0000 7f3c306ab700 1 --2- 172.21.15.32:0/2645340643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c281037b0 0x7f3c280fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:05.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.042+0000 7f3c306ab700 1 --2- 172.21.15.32:0/2645340643 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c28102e10 0x7f3c281031e0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:05.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.042+0000 7f3c306ab700 1 -- 172.21.15.32:0/2645340643 >> 172.21.15.32:0/2645340643 conn(0x7f3c280f89a0 msgr2=0x7f3c280fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:05.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.042+0000 7f3c306ab700 1 -- 172.21.15.32:0/2645340643 shutdown_connections 2024-04-12T05:41:05.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.042+0000 7f3c306ab700 1 -- 172.21.15.32:0/2645340643 wait complete. 2024-04-12T05:41:05.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.043+0000 7f3c306ab700 1 Processor -- start 2024-04-12T05:41:05.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.043+0000 7f3c306ab700 1 -- start start 2024-04-12T05:41:05.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.043+0000 7f3c306ab700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c28102e10 0x7f3c28113ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:05.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.043+0000 7f3c306ab700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c281037b0 0x7f3c28113ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:05.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.043+0000 7f3c306ab700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c281146d0 con 0x7f3c281037b0 2024-04-12T05:41:05.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.043+0000 7f3c306ab700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2810da30 con 0x7f3c28102e10 2024-04-12T05:41:05.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.043+0000 7f3c2dc46700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c281037b0 0x7f3c28113ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:05.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.043+0000 7f3c2dc46700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c281037b0 0x7f3c28113ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42550/0 (socket says 172.21.15.32:42550) 2024-04-12T05:41:05.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.043+0000 7f3c2dc46700 1 -- 172.21.15.32:0/3393870526 learned_addr learned my addr 172.21.15.32:0/3393870526 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:41:05.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.043+0000 7f3c2e447700 1 --2- 172.21.15.32:0/3393870526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c28102e10 0x7f3c28113ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:05.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.044+0000 7f3c2dc46700 1 -- 172.21.15.32:0/3393870526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c28102e10 msgr2=0x7f3c28113ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:05.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.044+0000 7f3c2dc46700 1 --2- 172.21.15.32:0/3393870526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c28102e10 0x7f3c28113ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:05.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.044+0000 7f3c2dc46700 1 -- 172.21.15.32:0/3393870526 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c18005040 con 0x7f3c281037b0 2024-04-12T05:41:05.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.044+0000 7f3c2dc46700 1 --2- 172.21.15.32:0/3393870526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c281037b0 0x7f3c28113ff0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f3c2400b790 tx=0x7f3c24009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:05.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.044+0000 7f3c1effd700 1 -- 172.21.15.32:0/3393870526 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c2400f6f0 con 0x7f3c281037b0 2024-04-12T05:41:05.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.044+0000 7f3c1effd700 1 -- 172.21.15.32:0/3393870526 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c2400fd00 con 0x7f3c281037b0 2024-04-12T05:41:05.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.044+0000 7f3c306ab700 1 -- 172.21.15.32:0/3393870526 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c2810dd10 con 0x7f3c281037b0 2024-04-12T05:41:05.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.044+0000 7f3c1effd700 1 -- 172.21.15.32:0/3393870526 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c240103a0 con 0x7f3c281037b0 2024-04-12T05:41:05.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.044+0000 7f3c306ab700 1 -- 172.21.15.32:0/3393870526 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c2810e2d0 con 0x7f3c281037b0 2024-04-12T05:41:05.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.045+0000 7f3c306ab700 1 -- 172.21.15.32:0/3393870526 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c280fe560 con 0x7f3c281037b0 2024-04-12T05:41:05.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.046+0000 7f3c1effd700 1 -- 172.21.15.32:0/3393870526 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 34) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f3c2400f850 con 0x7f3c281037b0 2024-04-12T05:41:05.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.046+0000 7f3c1effd700 1 --2- 172.21.15.32:0/3393870526 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f3c1406c8b0 0x7f3c1406ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:05.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.046+0000 7f3c1effd700 1 -- 172.21.15.32:0/3393870526 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f3c2408b810 con 0x7f3c281037b0 2024-04-12T05:41:05.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.046+0000 7f3c2e447700 1 --2- 172.21.15.32:0/3393870526 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f3c1406c8b0 0x7f3c1406ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:05.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.046+0000 7f3c2e447700 1 --2- 172.21.15.32:0/3393870526 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f3c1406c8b0 0x7f3c1406ed60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3c1800b840 tx=0x7f3c18009e30 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:05.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.050+0000 7f3c1effd700 1 -- 172.21.15.32:0/3393870526 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f3c24059850 con 0x7f3c281037b0 2024-04-12T05:41:05.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.337+0000 7f3c306ab700 1 -- 172.21.15.32:0/3393870526 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3c28109e40 con 0x7f3c281037b0 2024-04-12T05:41:05.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.338+0000 7f3c1effd700 1 -- 172.21.15.32:0/3393870526 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7f3c240593e0 con 0x7f3c281037b0 2024-04-12T05:41:05.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.340+0000 7f3c306ab700 1 -- 172.21.15.32:0/3393870526 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f3c1406c8b0 msgr2=0x7f3c1406ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:05.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.340+0000 7f3c306ab700 1 --2- 172.21.15.32:0/3393870526 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f3c1406c8b0 0x7f3c1406ed60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3c1800b840 tx=0x7f3c18009e30 comp rx=0 tx=0).stop 2024-04-12T05:41:05.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.340+0000 7f3c306ab700 1 -- 172.21.15.32:0/3393870526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c281037b0 msgr2=0x7f3c28113ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:05.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.340+0000 7f3c306ab700 1 --2- 172.21.15.32:0/3393870526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c281037b0 0x7f3c28113ff0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f3c2400b790 tx=0x7f3c24009220 comp rx=0 tx=0).stop 2024-04-12T05:41:05.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.340+0000 7f3c306ab700 1 -- 172.21.15.32:0/3393870526 shutdown_connections 2024-04-12T05:41:05.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.340+0000 7f3c306ab700 1 --2- 172.21.15.32:0/3393870526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c28102e10 0x7f3c28113ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:05.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.340+0000 7f3c306ab700 1 --2- 172.21.15.32:0/3393870526 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f3c1406c8b0 0x7f3c1406ed60 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:05.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.340+0000 7f3c306ab700 1 --2- 172.21.15.32:0/3393870526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c281037b0 0x7f3c28113ff0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:05.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.340+0000 7f3c306ab700 1 -- 172.21.15.32:0/3393870526 >> 172.21.15.32:0/3393870526 conn(0x7f3c280f89a0 msgr2=0x7f3c280f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:05.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.340+0000 7f3c306ab700 1 -- 172.21.15.32:0/3393870526 shutdown_connections 2024-04-12T05:41:05.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:05.340+0000 7f3c306ab700 1 -- 172.21.15.32:0/3393870526 wait complete. 2024-04-12T05:41:05.347 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:41:05.611 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:05 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/3393870526' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:41:05.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:05 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/3393870526' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:41:05.932 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 2'"'"'' 2024-04-12T05:41:06.217 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:41:06.691 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:06 smithi032 ceph-mon[39731]: pgmap v72: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:41:06.691 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:06 smithi032 ceph-mon[39731]: fsmap cephfs:2 {0=cephfs.smithi032.pndtry=up:active,1=cephfs.smithi067.zesmgy=up:active} 2 up:standby-replay 2024-04-12T05:41:06.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:06 smithi067 ceph-mon[49010]: pgmap v72: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:41:06.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:06 smithi067 ceph-mon[49010]: fsmap cephfs:2 {0=cephfs.smithi032.pndtry=up:active,1=cephfs.smithi067.zesmgy=up:active} 2 up:standby-replay 2024-04-12T05:41:08.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.193+0000 7f0971f4b700 1 -- 172.21.15.32:0/2752132389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f096c102e10 msgr2=0x7f096c1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:08.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.193+0000 7f0971f4b700 1 --2- 172.21.15.32:0/2752132389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f096c102e10 0x7f096c1031e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f095c009ce0 tx=0x7f095c004d10 comp rx=0 tx=0).stop 2024-04-12T05:41:08.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.193+0000 7f0971f4b700 1 -- 172.21.15.32:0/2752132389 shutdown_connections 2024-04-12T05:41:08.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.193+0000 7f0971f4b700 1 --2- 172.21.15.32:0/2752132389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f096c1037b0 0x7f096c0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:08.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.193+0000 7f0971f4b700 1 --2- 172.21.15.32:0/2752132389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f096c102e10 0x7f096c1031e0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:08.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.193+0000 7f0971f4b700 1 -- 172.21.15.32:0/2752132389 >> 172.21.15.32:0/2752132389 conn(0x7f096c0f89a0 msgr2=0x7f096c0fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:08.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.193+0000 7f0971f4b700 1 -- 172.21.15.32:0/2752132389 shutdown_connections 2024-04-12T05:41:08.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.193+0000 7f0971f4b700 1 -- 172.21.15.32:0/2752132389 wait complete. 2024-04-12T05:41:08.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.194+0000 7f0971f4b700 1 Processor -- start 2024-04-12T05:41:08.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.194+0000 7f0971f4b700 1 -- start start 2024-04-12T05:41:08.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.194+0000 7f0971f4b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f096c102e10 0x7f096c113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:08.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.194+0000 7f0971f4b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f096c1037b0 0x7f096c114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:08.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.194+0000 7f0971f4b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096c114680 con 0x7f096c102e10 2024-04-12T05:41:08.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.194+0000 7f0971f4b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096c10da70 con 0x7f096c1037b0 2024-04-12T05:41:08.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.195+0000 7f096b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f096c102e10 0x7f096c113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:08.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.195+0000 7f096b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f096c102e10 0x7f096c113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42562/0 (socket says 172.21.15.32:42562) 2024-04-12T05:41:08.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.195+0000 7f096b7fe700 1 -- 172.21.15.32:0/2008449134 learned_addr learned my addr 172.21.15.32:0/2008449134 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:41:08.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.195+0000 7f096affd700 1 --2- 172.21.15.32:0/2008449134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f096c1037b0 0x7f096c114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:08.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.195+0000 7f096b7fe700 1 -- 172.21.15.32:0/2008449134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f096c1037b0 msgr2=0x7f096c114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:08.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.195+0000 7f096b7fe700 1 --2- 172.21.15.32:0/2008449134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f096c1037b0 0x7f096c114030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:08.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.195+0000 7f096b7fe700 1 -- 172.21.15.32:0/2008449134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095c009940 con 0x7f096c102e10 2024-04-12T05:41:08.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.195+0000 7f096affd700 1 --2- 172.21.15.32:0/2008449134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f096c1037b0 0x7f096c114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:41:08.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.195+0000 7f096b7fe700 1 --2- 172.21.15.32:0/2008449134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f096c102e10 0x7f096c113af0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f095c00b460 tx=0x7f095c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:08.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.195+0000 7f0963fff700 1 -- 172.21.15.32:0/2008449134 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f095c00e040 con 0x7f096c102e10 2024-04-12T05:41:08.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.195+0000 7f0971f4b700 1 -- 172.21.15.32:0/2008449134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f096c10dcf0 con 0x7f096c102e10 2024-04-12T05:41:08.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.195+0000 7f0963fff700 1 -- 172.21.15.32:0/2008449134 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f095c01cb00 con 0x7f096c102e10 2024-04-12T05:41:08.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.195+0000 7f0963fff700 1 -- 172.21.15.32:0/2008449134 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f095c012a00 con 0x7f096c102e10 2024-04-12T05:41:08.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.195+0000 7f0971f4b700 1 -- 172.21.15.32:0/2008449134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f096c10e2b0 con 0x7f096c102e10 2024-04-12T05:41:08.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.196+0000 7f0971f4b700 1 -- 172.21.15.32:0/2008449134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f096c0fe560 con 0x7f096c102e10 2024-04-12T05:41:08.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.197+0000 7f0963fff700 1 -- 172.21.15.32:0/2008449134 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 34) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f095c01cc70 con 0x7f096c102e10 2024-04-12T05:41:08.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.197+0000 7f0963fff700 1 --2- 172.21.15.32:0/2008449134 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f095806c8a0 0x7f095806ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:08.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.197+0000 7f0963fff700 1 -- 172.21.15.32:0/2008449134 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f095c08deb0 con 0x7f096c102e10 2024-04-12T05:41:08.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.197+0000 7f096affd700 1 --2- 172.21.15.32:0/2008449134 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f095806c8a0 0x7f095806ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:08.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.198+0000 7f096affd700 1 --2- 172.21.15.32:0/2008449134 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f095806c8a0 0x7f095806ed50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0954004600 tx=0x7f095400b040 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:08.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.202+0000 7f0963fff700 1 -- 172.21.15.32:0/2008449134 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f095c05bdd0 con 0x7f096c102e10 2024-04-12T05:41:08.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:08.423+0000 7f0971f4b700 1 -- 172.21.15.32:0/2008449134 --> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] -- mgr_command(tid 0: {"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42", "target": ["mon-mgr", ""]}) v1 -- 0x7f096c04bed0 con 0x7f095806c8a0 2024-04-12T05:41:08.657 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:08 smithi032 ceph-mon[39731]: pgmap v73: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:41:08.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:08 smithi067 ceph-mon[49010]: pgmap v73: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:41:09.494 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:09 smithi032 ceph-mon[39731]: from='client.14798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:41:09.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:09 smithi067 ceph-mon[49010]: from='client.14798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:41:09.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:09.842+0000 7f0963fff700 1 -- 172.21.15.32:0/2008449134 <== mgr.24463 v2:172.21.15.32:6800/2564765508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+5094 (secure 0 0 0) 0x7f096c04bed0 con 0x7f095806c8a0 2024-04-12T05:41:09.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:09.845+0000 7f0971f4b700 1 -- 172.21.15.32:0/2008449134 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f095806c8a0 msgr2=0x7f095806ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:09.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:09.845+0000 7f0971f4b700 1 --2- 172.21.15.32:0/2008449134 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f095806c8a0 0x7f095806ed50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0954004600 tx=0x7f095400b040 comp rx=0 tx=0).stop 2024-04-12T05:41:09.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:09.845+0000 7f0971f4b700 1 -- 172.21.15.32:0/2008449134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f096c102e10 msgr2=0x7f096c113af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:09.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:09.845+0000 7f0971f4b700 1 --2- 172.21.15.32:0/2008449134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f096c102e10 0x7f096c113af0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f095c00b460 tx=0x7f095c004d10 comp rx=0 tx=0).stop 2024-04-12T05:41:09.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:09.845+0000 7f0971f4b700 1 -- 172.21.15.32:0/2008449134 shutdown_connections 2024-04-12T05:41:09.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:09.845+0000 7f0971f4b700 1 --2- 172.21.15.32:0/2008449134 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f095806c8a0 0x7f095806ed50 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:09.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:09.845+0000 7f0971f4b700 1 --2- 172.21.15.32:0/2008449134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f096c102e10 0x7f096c113af0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:09.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:09.845+0000 7f0971f4b700 1 --2- 172.21.15.32:0/2008449134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f096c1037b0 0x7f096c114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:09.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:09.845+0000 7f0971f4b700 1 -- 172.21.15.32:0/2008449134 >> 172.21.15.32:0/2008449134 conn(0x7f096c0f89a0 msgr2=0x7f096c0f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:09.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:09.845+0000 7f0971f4b700 1 -- 172.21.15.32:0/2008449134 shutdown_connections 2024-04-12T05:41:09.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:09.845+0000 7f0971f4b700 1 -- 172.21.15.32:0/2008449134 wait complete. 2024-04-12T05:41:09.855 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:41:10.487 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:10 smithi032 ceph-mon[39731]: pgmap v74: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:41:10.511 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph orch ps' 2024-04-12T05:41:10.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:10 smithi067 ceph-mon[49010]: pgmap v74: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:41:10.799 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:41:12.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.697+0000 7f17eb207700 1 -- 172.21.15.32:0/2416098862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17e4102e10 msgr2=0x7f17e41031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:12.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.697+0000 7f17eb207700 1 --2- 172.21.15.32:0/2416098862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17e4102e10 0x7f17e41031e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f17d8004940 tx=0x7f17d8009e30 comp rx=0 tx=0).stop 2024-04-12T05:41:12.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.698+0000 7f17eb207700 1 -- 172.21.15.32:0/2416098862 shutdown_connections 2024-04-12T05:41:12.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.698+0000 7f17eb207700 1 --2- 172.21.15.32:0/2416098862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17e41037b0 0x7f17e40fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:12.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.698+0000 7f17eb207700 1 --2- 172.21.15.32:0/2416098862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17e4102e10 0x7f17e41031e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:12.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.698+0000 7f17eb207700 1 -- 172.21.15.32:0/2416098862 >> 172.21.15.32:0/2416098862 conn(0x7f17e40f8980 msgr2=0x7f17e40fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:12.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.698+0000 7f17eb207700 1 -- 172.21.15.32:0/2416098862 shutdown_connections 2024-04-12T05:41:12.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.698+0000 7f17eb207700 1 -- 172.21.15.32:0/2416098862 wait complete. 2024-04-12T05:41:12.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.698+0000 7f17eb207700 1 Processor -- start 2024-04-12T05:41:12.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.698+0000 7f17eb207700 1 -- start start 2024-04-12T05:41:12.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.699+0000 7f17eb207700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17e4102e10 0x7f17e4113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:12.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.699+0000 7f17eb207700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17e41037b0 0x7f17e4114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:12.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.699+0000 7f17eb207700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e41146f0 con 0x7f17e41037b0 2024-04-12T05:41:12.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.699+0000 7f17eb207700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e410da50 con 0x7f17e4102e10 2024-04-12T05:41:12.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.699+0000 7f17e3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17e41037b0 0x7f17e4114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:12.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.699+0000 7f17e3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17e41037b0 0x7f17e4114010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49028/0 (socket says 172.21.15.32:49028) 2024-04-12T05:41:12.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.699+0000 7f17e3fff700 1 -- 172.21.15.32:0/1738918782 learned_addr learned my addr 172.21.15.32:0/1738918782 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:41:12.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.699+0000 7f17e8fa3700 1 --2- 172.21.15.32:0/1738918782 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17e4102e10 0x7f17e4113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:12.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.699+0000 7f17e3fff700 1 -- 172.21.15.32:0/1738918782 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17e4102e10 msgr2=0x7f17e4113ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:12.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.699+0000 7f17e3fff700 1 --2- 172.21.15.32:0/1738918782 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17e4102e10 0x7f17e4113ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:12.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.699+0000 7f17e3fff700 1 -- 172.21.15.32:0/1738918782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d8005040 con 0x7f17e41037b0 2024-04-12T05:41:12.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.700+0000 7f17e3fff700 1 --2- 172.21.15.32:0/1738918782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17e41037b0 0x7f17e4114010 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f17d000b790 tx=0x7f17d0009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:12.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.700+0000 7f17e17fa700 1 -- 172.21.15.32:0/1738918782 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17d000f6f0 con 0x7f17e41037b0 2024-04-12T05:41:12.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.700+0000 7f17eb207700 1 -- 172.21.15.32:0/1738918782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e410dd30 con 0x7f17e41037b0 2024-04-12T05:41:12.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.700+0000 7f17e17fa700 1 -- 172.21.15.32:0/1738918782 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f17d000fd00 con 0x7f17e41037b0 2024-04-12T05:41:12.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.700+0000 7f17e17fa700 1 -- 172.21.15.32:0/1738918782 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17d00103a0 con 0x7f17e41037b0 2024-04-12T05:41:12.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.700+0000 7f17eb207700 1 -- 172.21.15.32:0/1738918782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e410e2f0 con 0x7f17e41037b0 2024-04-12T05:41:12.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.701+0000 7f17eb207700 1 -- 172.21.15.32:0/1738918782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e40fe560 con 0x7f17e41037b0 2024-04-12T05:41:12.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.701+0000 7f17e17fa700 1 -- 172.21.15.32:0/1738918782 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 34) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f17d000f850 con 0x7f17e41037b0 2024-04-12T05:41:12.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.702+0000 7f17e17fa700 1 --2- 172.21.15.32:0/1738918782 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f17d406c8b0 0x7f17d406ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:12.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.702+0000 7f17e17fa700 1 -- 172.21.15.32:0/1738918782 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f17d008b8e0 con 0x7f17e41037b0 2024-04-12T05:41:12.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.702+0000 7f17e8fa3700 1 --2- 172.21.15.32:0/1738918782 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f17d406c8b0 0x7f17d406ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:12.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.702+0000 7f17e8fa3700 1 --2- 172.21.15.32:0/1738918782 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f17d406c8b0 0x7f17d406ed60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f17d800b840 tx=0x7f17d8009e30 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:12.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.706+0000 7f17e17fa700 1 -- 172.21.15.32:0/1738918782 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f17d0059880 con 0x7f17e41037b0 2024-04-12T05:41:12.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:12 smithi067 ceph-mon[49010]: pgmap v75: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:41:12.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:12 smithi032 ceph-mon[39731]: pgmap v75: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:41:12.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.919+0000 7f17eb207700 1 -- 172.21.15.32:0/1738918782 --> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f17e4109e40 con 0x7f17d406c8b0 2024-04-12T05:41:12.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.929+0000 7f17e17fa700 1 -- 172.21.15.32:0/1738918782 <== mgr.24463 v2:172.21.15.32:6800/2564765508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f17e4109e40 con 0x7f17d406c8b0 2024-04-12T05:41:12.927 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:41:12.927 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (60s) 37s ago 12m 17.0M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:41:12.927 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (12m) 37s ago 12m 6656k - 17.2.7-827-gd3b5060a 21de9b7369ad a9cfc00c311b 2024-04-12T05:41:12.927 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (11m) 93s ago 11m 6651k - 17.2.7-827-gd3b5060a 21de9b7369ad 1f2eea4542d9 2024-04-12T05:41:12.927 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (43s) 37s ago 12m 44.0M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:41:12.927 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (6m) 37s ago 6m 64.5M - 17.2.7-827-gd3b5060a 21de9b7369ad e7ada8e5259c 2024-04-12T05:41:12.927 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (6m) 37s ago 6m 116M - 17.2.7-827-gd3b5060a 21de9b7369ad 55124da9afdb 2024-04-12T05:41:12.927 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (6m) 93s ago 6m 62.5M - 17.2.7-827-gd3b5060a 21de9b7369ad 0c295d31a4a6 2024-04-12T05:41:12.927 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (6m) 93s ago 6m 120M - 17.2.7-827-gd3b5060a 21de9b7369ad 6eecd7deaac2 2024-04-12T05:41:12.927 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (2m) 37s ago 13m 502M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:41:12.928 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (114s) 93s ago 11m 449M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:41:12.928 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (13m) 37s ago 14m 53.6M 2048M 17.2.7-827-gd3b5060a 21de9b7369ad 75aad95b984d 2024-04-12T05:41:12.928 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (11m) 93s ago 11m 42.3M 2048M 17.2.7-827-gd3b5060a 21de9b7369ad 157be8a8c6d8 2024-04-12T05:41:12.928 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (101s) 37s ago 12m 14.3M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:41:12.928 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (97s) 93s ago 11m 3594k - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:41:12.928 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (10m) 37s ago 10m 220M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad 97446e42ceba 2024-04-12T05:41:12.928 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (10m) 37s ago 10m 236M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad bf9f9a2db2d5 2024-04-12T05:41:12.928 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (9m) 37s ago 9m 189M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad f055cf1ca665 2024-04-12T05:41:12.928 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (9m) 93s ago 9m 255M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad 61b851235351 2024-04-12T05:41:12.928 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (8m) 93s ago 8m 227M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad 2c84d9047e9c 2024-04-12T05:41:12.928 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (8m) 93s ago 8m 217M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad e54839bb6055 2024-04-12T05:41:12.928 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (77s) 37s ago 11m 46.5M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:41:12.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.931+0000 7f17eb207700 1 -- 172.21.15.32:0/1738918782 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f17d406c8b0 msgr2=0x7f17d406ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:12.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.931+0000 7f17eb207700 1 --2- 172.21.15.32:0/1738918782 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f17d406c8b0 0x7f17d406ed60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f17d800b840 tx=0x7f17d8009e30 comp rx=0 tx=0).stop 2024-04-12T05:41:12.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.931+0000 7f17eb207700 1 -- 172.21.15.32:0/1738918782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17e41037b0 msgr2=0x7f17e4114010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:12.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.931+0000 7f17eb207700 1 --2- 172.21.15.32:0/1738918782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17e41037b0 0x7f17e4114010 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f17d000b790 tx=0x7f17d0009220 comp rx=0 tx=0).stop 2024-04-12T05:41:12.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.931+0000 7f17eb207700 1 -- 172.21.15.32:0/1738918782 shutdown_connections 2024-04-12T05:41:12.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.931+0000 7f17eb207700 1 --2- 172.21.15.32:0/1738918782 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17e4102e10 0x7f17e4113ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:12.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.931+0000 7f17eb207700 1 --2- 172.21.15.32:0/1738918782 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f17d406c8b0 0x7f17d406ed60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:12.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.931+0000 7f17eb207700 1 --2- 172.21.15.32:0/1738918782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17e41037b0 0x7f17e4114010 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:12.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.931+0000 7f17eb207700 1 -- 172.21.15.32:0/1738918782 >> 172.21.15.32:0/1738918782 conn(0x7f17e40f8980 msgr2=0x7f17e40f9970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:12.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.932+0000 7f17eb207700 1 -- 172.21.15.32:0/1738918782 shutdown_connections 2024-04-12T05:41:12.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:12.932+0000 7f17eb207700 1 -- 172.21.15.32:0/1738918782 wait complete. 2024-04-12T05:41:13.666 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-12T05:41:13.667 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-12T05:41:13.667 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs true' 2024-04-12T05:41:13.951 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:41:14.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:14 smithi032 ceph-mon[39731]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:41:14.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:14 smithi032 ceph-mon[39731]: pgmap v76: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:41:14.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:14 smithi067 ceph-mon[49010]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:41:14.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:14 smithi067 ceph-mon[49010]: pgmap v76: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:41:15.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.894+0000 7f3c60a06700 1 -- 172.21.15.32:0/813707138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c5c101d10 msgr2=0x7f3c5c1020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:15.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.894+0000 7f3c60a06700 1 --2- 172.21.15.32:0/813707138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c5c101d10 0x7f3c5c1020e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c004940 tx=0x7f3c4c009e30 comp rx=0 tx=0).stop 2024-04-12T05:41:15.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.894+0000 7f3c60a06700 1 -- 172.21.15.32:0/813707138 shutdown_connections 2024-04-12T05:41:15.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.894+0000 7f3c60a06700 1 --2- 172.21.15.32:0/813707138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5c1026b0 0x7f3c5c0fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:15.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.894+0000 7f3c60a06700 1 --2- 172.21.15.32:0/813707138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c5c101d10 0x7f3c5c1020e0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:15.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.894+0000 7f3c60a06700 1 -- 172.21.15.32:0/813707138 >> 172.21.15.32:0/813707138 conn(0x7f3c5c0f7880 msgr2=0x7f3c5c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:15.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.894+0000 7f3c60a06700 1 -- 172.21.15.32:0/813707138 shutdown_connections 2024-04-12T05:41:15.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.894+0000 7f3c60a06700 1 -- 172.21.15.32:0/813707138 wait complete. 2024-04-12T05:41:15.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.894+0000 7f3c60a06700 1 Processor -- start 2024-04-12T05:41:15.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.895+0000 7f3c60a06700 1 -- start start 2024-04-12T05:41:15.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.895+0000 7f3c60a06700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c5c101d10 0x7f3c5c113e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:15.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.895+0000 7f3c60a06700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5c1026b0 0x7f3c5c114370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:15.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.895+0000 7f3c60a06700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5c114a50 con 0x7f3c5c101d10 2024-04-12T05:41:15.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.895+0000 7f3c60a06700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5c10dd90 con 0x7f3c5c1026b0 2024-04-12T05:41:15.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.895+0000 7f3c5a59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c5c101d10 0x7f3c5c113e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:15.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.895+0000 7f3c5a59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c5c101d10 0x7f3c5c113e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49038/0 (socket says 172.21.15.32:49038) 2024-04-12T05:41:15.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.895+0000 7f3c5a59c700 1 -- 172.21.15.32:0/3828909278 learned_addr learned my addr 172.21.15.32:0/3828909278 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:41:15.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.896+0000 7f3c59d9b700 1 --2- 172.21.15.32:0/3828909278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5c1026b0 0x7f3c5c114370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:15.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.896+0000 7f3c5a59c700 1 -- 172.21.15.32:0/3828909278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5c1026b0 msgr2=0x7f3c5c114370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:15.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.896+0000 7f3c5a59c700 1 --2- 172.21.15.32:0/3828909278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5c1026b0 0x7f3c5c114370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:15.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.896+0000 7f3c5a59c700 1 -- 172.21.15.32:0/3828909278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c4c005040 con 0x7f3c5c101d10 2024-04-12T05:41:15.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.896+0000 7f3c59d9b700 1 --2- 172.21.15.32:0/3828909278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5c1026b0 0x7f3c5c114370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:41:15.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.896+0000 7f3c5a59c700 1 --2- 172.21.15.32:0/3828909278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c5c101d10 0x7f3c5c113e30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c009180 tx=0x7f3c4c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:15.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.896+0000 7f3c52ffd700 1 -- 172.21.15.32:0/3828909278 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c4c00e040 con 0x7f3c5c101d10 2024-04-12T05:41:15.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.896+0000 7f3c52ffd700 1 -- 172.21.15.32:0/3828909278 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c4c004290 con 0x7f3c5c101d10 2024-04-12T05:41:15.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.896+0000 7f3c52ffd700 1 -- 172.21.15.32:0/3828909278 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c4c01c3f0 con 0x7f3c5c101d10 2024-04-12T05:41:15.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.896+0000 7f3c60a06700 1 -- 172.21.15.32:0/3828909278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c5c10dfb0 con 0x7f3c5c101d10 2024-04-12T05:41:15.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.896+0000 7f3c60a06700 1 -- 172.21.15.32:0/3828909278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c5c10e4f0 con 0x7f3c5c101d10 2024-04-12T05:41:15.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.897+0000 7f3c60a06700 1 -- 172.21.15.32:0/3828909278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c5c0fd460 con 0x7f3c5c101d10 2024-04-12T05:41:15.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.898+0000 7f3c52ffd700 1 -- 172.21.15.32:0/3828909278 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 34) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f3c4c013790 con 0x7f3c5c101d10 2024-04-12T05:41:15.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.898+0000 7f3c52ffd700 1 --2- 172.21.15.32:0/3828909278 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f3c4806c8a0 0x7f3c4806ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:15.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.898+0000 7f3c52ffd700 1 -- 172.21.15.32:0/3828909278 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f3c4c08d840 con 0x7f3c5c101d10 2024-04-12T05:41:15.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.899+0000 7f3c59d9b700 1 --2- 172.21.15.32:0/3828909278 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f3c4806c8a0 0x7f3c4806ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:15.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.899+0000 7f3c59d9b700 1 --2- 172.21.15.32:0/3828909278 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f3c4806c8a0 0x7f3c4806ed50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c10f6d0 tx=0x7f3c44008040 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:15.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:15.902+0000 7f3c52ffd700 1 -- 172.21.15.32:0/3828909278 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f3c4c05b880 con 0x7f3c5c101d10 2024-04-12T05:41:16.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:16.103+0000 7f3c60a06700 1 -- 172.21.15.32:0/3828909278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7f3c5c05f5f0 con 0x7f3c5c101d10 2024-04-12T05:41:16.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:16.107+0000 7f3c52ffd700 1 -- 172.21.15.32:0/3828909278 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=0 v47)=0 v47) v1 ==== 125+0+0 (secure 0 0 0) 0x7f3c4c013420 con 0x7f3c5c101d10 2024-04-12T05:41:16.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:16.109+0000 7f3c60a06700 1 -- 172.21.15.32:0/3828909278 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f3c4806c8a0 msgr2=0x7f3c4806ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:16.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:16.109+0000 7f3c60a06700 1 --2- 172.21.15.32:0/3828909278 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f3c4806c8a0 0x7f3c4806ed50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c10f6d0 tx=0x7f3c44008040 comp rx=0 tx=0).stop 2024-04-12T05:41:16.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:16.109+0000 7f3c60a06700 1 -- 172.21.15.32:0/3828909278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c5c101d10 msgr2=0x7f3c5c113e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:16.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:16.109+0000 7f3c60a06700 1 --2- 172.21.15.32:0/3828909278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c5c101d10 0x7f3c5c113e30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c009180 tx=0x7f3c4c009e30 comp rx=0 tx=0).stop 2024-04-12T05:41:16.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:16.109+0000 7f3c60a06700 1 -- 172.21.15.32:0/3828909278 shutdown_connections 2024-04-12T05:41:16.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:16.109+0000 7f3c60a06700 1 --2- 172.21.15.32:0/3828909278 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f3c4806c8a0 0x7f3c4806ed50 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:16.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:16.109+0000 7f3c60a06700 1 --2- 172.21.15.32:0/3828909278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c5c101d10 0x7f3c5c113e30 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:16.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:16.109+0000 7f3c60a06700 1 --2- 172.21.15.32:0/3828909278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5c1026b0 0x7f3c5c114370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:16.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:16.109+0000 7f3c60a06700 1 -- 172.21.15.32:0/3828909278 >> 172.21.15.32:0/3828909278 conn(0x7f3c5c0f7880 msgr2=0x7f3c5c0f9070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:16.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:16.109+0000 7f3c60a06700 1 -- 172.21.15.32:0/3828909278 shutdown_connections 2024-04-12T05:41:16.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:16.109+0000 7f3c60a06700 1 -- 172.21.15.32:0/3828909278 wait complete. 2024-04-12T05:41:16.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:16 smithi067 ceph-mon[49010]: pgmap v77: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:41:16.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:16 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/3828909278' entity='client.admin' 2024-04-12T05:41:16.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:16 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:41:16.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:16 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:41:16.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:16 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:41:16.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:16 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:16.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:16 smithi032 ceph-mon[39731]: pgmap v77: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:41:16.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:16 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/3828909278' entity='client.admin' 2024-04-12T05:41:16.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:16 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:41:16.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:16 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:41:16.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:16 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:41:16.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:16 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:16.891 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-12T05:41:16.892 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-12T05:41:16.892 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-12T05:41:17.177 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:41:17.726 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:17 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:41:17.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:17 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:41:18.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:18 smithi032 ceph-mon[39731]: pgmap v78: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:41:18.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:18 smithi067 ceph-mon[49010]: pgmap v78: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:41:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.938+0000 7f82402c1700 1 -- 172.21.15.32:0/2205771326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8238102710 msgr2=0x7f82380fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.938+0000 7f82402c1700 1 --2- 172.21.15.32:0/2205771326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8238102710 0x7f82380fbd80 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f8234004940 tx=0x7f8234009e30 comp rx=0 tx=0).stop 2024-04-12T05:41:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.938+0000 7f82402c1700 1 -- 172.21.15.32:0/2205771326 shutdown_connections 2024-04-12T05:41:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.938+0000 7f82402c1700 1 --2- 172.21.15.32:0/2205771326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8238102710 0x7f82380fbd80 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.938+0000 7f82402c1700 1 --2- 172.21.15.32:0/2205771326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8238101d70 0x7f8238102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.938+0000 7f82402c1700 1 -- 172.21.15.32:0/2205771326 >> 172.21.15.32:0/2205771326 conn(0x7f8238072bc0 msgr2=0x7f8238072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.938+0000 7f82402c1700 1 -- 172.21.15.32:0/2205771326 shutdown_connections 2024-04-12T05:41:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.938+0000 7f82402c1700 1 -- 172.21.15.32:0/2205771326 wait complete. 2024-04-12T05:41:18.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.939+0000 7f82402c1700 1 Processor -- start 2024-04-12T05:41:18.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.939+0000 7f82402c1700 1 -- start start 2024-04-12T05:41:18.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.939+0000 7f82402c1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8238101d70 0x7f82381129c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:18.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.939+0000 7f82402c1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8238102710 0x7f8238112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:18.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.939+0000 7f82402c1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8238113550 con 0x7f8238102710 2024-04-12T05:41:18.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.939+0000 7f82402c1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f823810c940 con 0x7f8238101d70 2024-04-12T05:41:18.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.939+0000 7f823d85c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8238102710 0x7f8238112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:18.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.939+0000 7f823d85c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8238102710 0x7f8238112f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49064/0 (socket says 172.21.15.32:49064) 2024-04-12T05:41:18.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.939+0000 7f823d85c700 1 -- 172.21.15.32:0/4208176818 learned_addr learned my addr 172.21.15.32:0/4208176818 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:41:18.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.939+0000 7f823e05d700 1 --2- 172.21.15.32:0/4208176818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8238101d70 0x7f82381129c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:18.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.940+0000 7f823d85c700 1 -- 172.21.15.32:0/4208176818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8238101d70 msgr2=0x7f82381129c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:18.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.940+0000 7f823d85c700 1 --2- 172.21.15.32:0/4208176818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8238101d70 0x7f82381129c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:18.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.940+0000 7f823d85c700 1 -- 172.21.15.32:0/4208176818 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8234005040 con 0x7f8238102710 2024-04-12T05:41:18.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.940+0000 7f823d85c700 1 --2- 172.21.15.32:0/4208176818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8238102710 0x7f8238112f00 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f8234009390 tx=0x7f8234009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:18.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.940+0000 7f822effd700 1 -- 172.21.15.32:0/4208176818 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f823400e040 con 0x7f8238102710 2024-04-12T05:41:18.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.940+0000 7f822effd700 1 -- 172.21.15.32:0/4208176818 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8234013420 con 0x7f8238102710 2024-04-12T05:41:18.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.940+0000 7f82402c1700 1 -- 172.21.15.32:0/4208176818 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f823810cbc0 con 0x7f8238102710 2024-04-12T05:41:18.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.940+0000 7f822effd700 1 -- 172.21.15.32:0/4208176818 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f823401c630 con 0x7f8238102710 2024-04-12T05:41:18.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.940+0000 7f82402c1700 1 -- 172.21.15.32:0/4208176818 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f823810d180 con 0x7f8238102710 2024-04-12T05:41:18.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.941+0000 7f82402c1700 1 -- 172.21.15.32:0/4208176818 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82380ff080 con 0x7f8238102710 2024-04-12T05:41:18.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.941+0000 7f822effd700 1 -- 172.21.15.32:0/4208176818 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 34) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f8234013a00 con 0x7f8238102710 2024-04-12T05:41:18.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.942+0000 7f822effd700 1 --2- 172.21.15.32:0/4208176818 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f822406c8b0 0x7f822406ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:18.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.942+0000 7f822effd700 1 -- 172.21.15.32:0/4208176818 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f823408cb20 con 0x7f8238102710 2024-04-12T05:41:18.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.942+0000 7f823e05d700 1 --2- 172.21.15.32:0/4208176818 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f822406c8b0 0x7f822406ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:18.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.942+0000 7f823e05d700 1 --2- 172.21.15.32:0/4208176818 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f822406c8b0 0x7f822406ed60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8228004f40 tx=0x7f822800b040 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:18.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:18.946+0000 7f822effd700 1 -- 172.21.15.32:0/4208176818 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f823405ab60 con 0x7f8238102710 2024-04-12T05:41:19.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:19.146+0000 7f82402c1700 1 -- 172.21.15.32:0/4208176818 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f823810e010 con 0x7f8238102710 2024-04-12T05:41:19.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:19.147+0000 7f822effd700 1 -- 172.21.15.32:0/4208176818 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v47)=0 v47) v1 ==== 155+0+0 (secure 0 0 0) 0x7f823405a6f0 con 0x7f8238102710 2024-04-12T05:41:19.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:19.149+0000 7f82402c1700 1 -- 172.21.15.32:0/4208176818 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f822406c8b0 msgr2=0x7f822406ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:19.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:19.149+0000 7f82402c1700 1 --2- 172.21.15.32:0/4208176818 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f822406c8b0 0x7f822406ed60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8228004f40 tx=0x7f822800b040 comp rx=0 tx=0).stop 2024-04-12T05:41:19.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:19.149+0000 7f82402c1700 1 -- 172.21.15.32:0/4208176818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8238102710 msgr2=0x7f8238112f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:19.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:19.149+0000 7f82402c1700 1 --2- 172.21.15.32:0/4208176818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8238102710 0x7f8238112f00 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f8234009390 tx=0x7f8234009e30 comp rx=0 tx=0).stop 2024-04-12T05:41:19.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:19.149+0000 7f82402c1700 1 -- 172.21.15.32:0/4208176818 shutdown_connections 2024-04-12T05:41:19.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:19.149+0000 7f82402c1700 1 --2- 172.21.15.32:0/4208176818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8238101d70 0x7f82381129c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:19.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:19.149+0000 7f82402c1700 1 --2- 172.21.15.32:0/4208176818 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f822406c8b0 0x7f822406ed60 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:19.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:19.149+0000 7f82402c1700 1 --2- 172.21.15.32:0/4208176818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8238102710 0x7f8238112f00 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:19.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:19.149+0000 7f82402c1700 1 -- 172.21.15.32:0/4208176818 >> 172.21.15.32:0/4208176818 conn(0x7f8238072bc0 msgr2=0x7f82380fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:19.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:19.149+0000 7f82402c1700 1 -- 172.21.15.32:0/4208176818 shutdown_connections 2024-04-12T05:41:19.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:19.149+0000 7f82402c1700 1 -- 172.21.15.32:0/4208176818 wait complete. 2024-04-12T05:41:19.864 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-12T05:41:20.145 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:41:20.639 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:20 smithi032 ceph-mon[39731]: pgmap v79: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:41:20.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:20 smithi067 ceph-mon[49010]: pgmap v79: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:41:21.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.944+0000 7f202c1f8700 1 -- 172.21.15.32:0/1791694083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20240fb4a0 msgr2=0x7f20240fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:21.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.944+0000 7f202c1f8700 1 --2- 172.21.15.32:0/1791694083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20240fb4a0 0x7f20240fb910 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f2014004940 tx=0x7f2014009e30 comp rx=0 tx=0).stop 2024-04-12T05:41:21.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.944+0000 7f202c1f8700 1 -- 172.21.15.32:0/1791694083 shutdown_connections 2024-04-12T05:41:21.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.944+0000 7f202c1f8700 1 --2- 172.21.15.32:0/1791694083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20240fb4a0 0x7f20240fb910 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:21.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.944+0000 7f202c1f8700 1 --2- 172.21.15.32:0/1791694083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2024102e70 0x7f2024103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:21.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.944+0000 7f202c1f8700 1 -- 172.21.15.32:0/1791694083 >> 172.21.15.32:0/1791694083 conn(0x7f20240714c0 msgr2=0x7f20240718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:21.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.944+0000 7f202c1f8700 1 -- 172.21.15.32:0/1791694083 shutdown_connections 2024-04-12T05:41:21.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.944+0000 7f202c1f8700 1 -- 172.21.15.32:0/1791694083 wait complete. 2024-04-12T05:41:21.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.944+0000 7f202c1f8700 1 Processor -- start 2024-04-12T05:41:21.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.944+0000 7f202c1f8700 1 -- start start 2024-04-12T05:41:21.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.945+0000 7f202c1f8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20240fb4a0 0x7f20240fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:21.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.945+0000 7f202c1f8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2024102e70 0x7f20240fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:21.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.945+0000 7f202c1f8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20240fd150 con 0x7f20240fb4a0 2024-04-12T05:41:21.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.945+0000 7f202c1f8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20240fd290 con 0x7f2024102e70 2024-04-12T05:41:21.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.945+0000 7f2029f94700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20240fb4a0 0x7f20240fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:21.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.945+0000 7f2029f94700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20240fb4a0 0x7f20240fc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52512/0 (socket says 172.21.15.32:52512) 2024-04-12T05:41:21.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.945+0000 7f2029f94700 1 -- 172.21.15.32:0/2896824369 learned_addr learned my addr 172.21.15.32:0/2896824369 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:41:21.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.945+0000 7f2029793700 1 --2- 172.21.15.32:0/2896824369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2024102e70 0x7f20240fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:21.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.945+0000 7f2029f94700 1 -- 172.21.15.32:0/2896824369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2024102e70 msgr2=0x7f20240fcbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:21.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.945+0000 7f2029f94700 1 --2- 172.21.15.32:0/2896824369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2024102e70 0x7f20240fcbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:21.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.945+0000 7f2029f94700 1 -- 172.21.15.32:0/2896824369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2014005040 con 0x7f20240fb4a0 2024-04-12T05:41:21.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.945+0000 7f2029793700 1 --2- 172.21.15.32:0/2896824369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2024102e70 0x7f20240fcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:41:21.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.945+0000 7f2029f94700 1 --2- 172.21.15.32:0/2896824369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20240fb4a0 0x7f20240fc680 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2020002a50 tx=0x7f20200046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:21.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.945+0000 7f201a7fc700 1 -- 172.21.15.32:0/2896824369 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2020007cc0 con 0x7f20240fb4a0 2024-04-12T05:41:21.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.945+0000 7f201a7fc700 1 -- 172.21.15.32:0/2896824369 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f202000f040 con 0x7f20240fb4a0 2024-04-12T05:41:21.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.945+0000 7f202c1f8700 1 -- 172.21.15.32:0/2896824369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2024119d30 con 0x7f20240fb4a0 2024-04-12T05:41:21.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.945+0000 7f201a7fc700 1 -- 172.21.15.32:0/2896824369 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2020009540 con 0x7f20240fb4a0 2024-04-12T05:41:21.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.945+0000 7f202c1f8700 1 -- 172.21.15.32:0/2896824369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f202411a1b0 con 0x7f20240fb4a0 2024-04-12T05:41:21.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.946+0000 7f202c1f8700 1 -- 172.21.15.32:0/2896824369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2024100110 con 0x7f20240fb4a0 2024-04-12T05:41:21.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.947+0000 7f201a7fc700 1 -- 172.21.15.32:0/2896824369 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 34) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f20200096a0 con 0x7f20240fb4a0 2024-04-12T05:41:21.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.947+0000 7f201a7fc700 1 --2- 172.21.15.32:0/2896824369 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f201006c8a0 0x7f201006ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:21.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.947+0000 7f201a7fc700 1 -- 172.21.15.32:0/2896824369 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f202008b8f0 con 0x7f20240fb4a0 2024-04-12T05:41:21.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.947+0000 7f2029793700 1 --2- 172.21.15.32:0/2896824369 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f201006c8a0 0x7f201006ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:21.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.948+0000 7f2029793700 1 --2- 172.21.15.32:0/2896824369 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f201006c8a0 0x7f201006ed50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2014004560 tx=0x7f2014009e30 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:21.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:21.952+0000 7f201a7fc700 1 -- 172.21.15.32:0/2896824369 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f2020055d80 con 0x7f20240fb4a0 2024-04-12T05:41:22.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:22.154+0000 7f202c1f8700 1 -- 172.21.15.32:0/2896824369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f202404bed0 con 0x7f20240fb4a0 2024-04-12T05:41:22.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:22.155+0000 7f201a7fc700 1 -- 172.21.15.32:0/2896824369 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v47)=0 v47) v1 ==== 163+0+0 (secure 0 0 0) 0x7f20200593a0 con 0x7f20240fb4a0 2024-04-12T05:41:22.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:22.157+0000 7f202c1f8700 1 -- 172.21.15.32:0/2896824369 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f201006c8a0 msgr2=0x7f201006ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:22.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:22.157+0000 7f202c1f8700 1 --2- 172.21.15.32:0/2896824369 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f201006c8a0 0x7f201006ed50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2014004560 tx=0x7f2014009e30 comp rx=0 tx=0).stop 2024-04-12T05:41:22.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:22.157+0000 7f202c1f8700 1 -- 172.21.15.32:0/2896824369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20240fb4a0 msgr2=0x7f20240fc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:22.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:22.157+0000 7f202c1f8700 1 --2- 172.21.15.32:0/2896824369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20240fb4a0 0x7f20240fc680 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2020002a50 tx=0x7f20200046b0 comp rx=0 tx=0).stop 2024-04-12T05:41:22.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:22.157+0000 7f202c1f8700 1 -- 172.21.15.32:0/2896824369 shutdown_connections 2024-04-12T05:41:22.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:22.157+0000 7f202c1f8700 1 --2- 172.21.15.32:0/2896824369 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f201006c8a0 0x7f201006ed50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:22.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:22.157+0000 7f202c1f8700 1 --2- 172.21.15.32:0/2896824369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20240fb4a0 0x7f20240fc680 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:22.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:22.157+0000 7f202c1f8700 1 --2- 172.21.15.32:0/2896824369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2024102e70 0x7f20240fcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:22.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:22.157+0000 7f202c1f8700 1 -- 172.21.15.32:0/2896824369 >> 172.21.15.32:0/2896824369 conn(0x7f20240714c0 msgr2=0x7f20241060a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:22.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:22.157+0000 7f202c1f8700 1 -- 172.21.15.32:0/2896824369 shutdown_connections 2024-04-12T05:41:22.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:22.157+0000 7f202c1f8700 1 -- 172.21.15.32:0/2896824369 wait complete. 2024-04-12T05:41:22.574 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:22 smithi032 ceph-mon[39731]: pgmap v80: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:41:22.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:22 smithi067 ceph-mon[49010]: pgmap v80: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:41:22.953 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-12T05:41:23.241 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:41:24.579 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:24 smithi032 ceph-mon[39731]: pgmap v81: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:41:24.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:24 smithi067 ceph-mon[49010]: pgmap v81: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:41:25.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.178+0000 7f34480ee700 1 -- 172.21.15.32:0/103025381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3440102710 msgr2=0x7f34400fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:25.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.178+0000 7f34480ee700 1 --2- 172.21.15.32:0/103025381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3440102710 0x7f34400fbd80 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f343c004940 tx=0x7f343c009e30 comp rx=0 tx=0).stop 2024-04-12T05:41:25.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.178+0000 7f34480ee700 1 -- 172.21.15.32:0/103025381 shutdown_connections 2024-04-12T05:41:25.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.178+0000 7f34480ee700 1 --2- 172.21.15.32:0/103025381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3440102710 0x7f34400fbd80 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:25.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.178+0000 7f34480ee700 1 --2- 172.21.15.32:0/103025381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3440101d70 0x7f3440102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:25.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.178+0000 7f34480ee700 1 -- 172.21.15.32:0/103025381 >> 172.21.15.32:0/103025381 conn(0x7f3440072bc0 msgr2=0x7f3440072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:25.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.178+0000 7f34480ee700 1 -- 172.21.15.32:0/103025381 shutdown_connections 2024-04-12T05:41:25.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.178+0000 7f34480ee700 1 -- 172.21.15.32:0/103025381 wait complete. 2024-04-12T05:41:25.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.179+0000 7f34480ee700 1 Processor -- start 2024-04-12T05:41:25.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.179+0000 7f34480ee700 1 -- start start 2024-04-12T05:41:25.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.179+0000 7f34480ee700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3440101d70 0x7f344010ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:25.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.179+0000 7f34480ee700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3440102710 0x7f344010efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:25.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.179+0000 7f34480ee700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344010f5f0 con 0x7f3440102710 2024-04-12T05:41:25.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.179+0000 7f34480ee700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344010f730 con 0x7f3440101d70 2024-04-12T05:41:25.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.179+0000 7f3445689700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3440102710 0x7f344010efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:25.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.179+0000 7f3445689700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3440102710 0x7f344010efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52518/0 (socket says 172.21.15.32:52518) 2024-04-12T05:41:25.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.179+0000 7f3445689700 1 -- 172.21.15.32:0/3636893302 learned_addr learned my addr 172.21.15.32:0/3636893302 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:41:25.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.179+0000 7f3445e8a700 1 --2- 172.21.15.32:0/3636893302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3440101d70 0x7f344010ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:25.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.179+0000 7f3445689700 1 -- 172.21.15.32:0/3636893302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3440101d70 msgr2=0x7f344010ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:25.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.179+0000 7f3445689700 1 --2- 172.21.15.32:0/3636893302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3440101d70 0x7f344010ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:25.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.179+0000 7f3445689700 1 -- 172.21.15.32:0/3636893302 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f343c005040 con 0x7f3440102710 2024-04-12T05:41:25.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.179+0000 7f3445689700 1 --2- 172.21.15.32:0/3636893302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3440102710 0x7f344010efa0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f343c009380 tx=0x7f343c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:25.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.180+0000 7f34367fc700 1 -- 172.21.15.32:0/3636893302 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f343c00e040 con 0x7f3440102710 2024-04-12T05:41:25.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.180+0000 7f34367fc700 1 -- 172.21.15.32:0/3636893302 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f343c013420 con 0x7f3440102710 2024-04-12T05:41:25.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.180+0000 7f34367fc700 1 -- 172.21.15.32:0/3636893302 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f343c01c630 con 0x7f3440102710 2024-04-12T05:41:25.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.180+0000 7f34480ee700 1 -- 172.21.15.32:0/3636893302 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3440117ad0 con 0x7f3440102710 2024-04-12T05:41:25.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.180+0000 7f34480ee700 1 -- 172.21.15.32:0/3636893302 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3440117fb0 con 0x7f3440102710 2024-04-12T05:41:25.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.181+0000 7f34480ee700 1 -- 172.21.15.32:0/3636893302 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34400ff080 con 0x7f3440102710 2024-04-12T05:41:25.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.181+0000 7f34367fc700 1 -- 172.21.15.32:0/3636893302 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 34) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f343c0139e0 con 0x7f3440102710 2024-04-12T05:41:25.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.182+0000 7f34367fc700 1 --2- 172.21.15.32:0/3636893302 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f342c06c8a0 0x7f342c06ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:25.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.182+0000 7f34367fc700 1 -- 172.21.15.32:0/3636893302 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f343c08cd60 con 0x7f3440102710 2024-04-12T05:41:25.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.182+0000 7f3445e8a700 1 --2- 172.21.15.32:0/3636893302 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f342c06c8a0 0x7f342c06ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:25.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.182+0000 7f3445e8a700 1 --2- 172.21.15.32:0/3636893302 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f342c06c8a0 0x7f342c06ed50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3430004990 tx=0x7f3430009ee0 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:25.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.186+0000 7f34367fc700 1 -- 172.21.15.32:0/3636893302 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f343c091070 con 0x7f3440102710 2024-04-12T05:41:25.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.387+0000 7f34480ee700 1 -- 172.21.15.32:0/3636893302 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f3440108d10 con 0x7f3440102710 2024-04-12T05:41:25.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.387+0000 7f34367fc700 1 -- 172.21.15.32:0/3636893302 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v47)=0 v47) v1 ==== 135+0+0 (secure 0 0 0) 0x7f343c0571d0 con 0x7f3440102710 2024-04-12T05:41:25.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.389+0000 7f34480ee700 1 -- 172.21.15.32:0/3636893302 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f342c06c8a0 msgr2=0x7f342c06ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:25.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.389+0000 7f34480ee700 1 --2- 172.21.15.32:0/3636893302 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f342c06c8a0 0x7f342c06ed50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3430004990 tx=0x7f3430009ee0 comp rx=0 tx=0).stop 2024-04-12T05:41:25.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.389+0000 7f34480ee700 1 -- 172.21.15.32:0/3636893302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3440102710 msgr2=0x7f344010efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:25.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.389+0000 7f34480ee700 1 --2- 172.21.15.32:0/3636893302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3440102710 0x7f344010efa0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f343c009380 tx=0x7f343c009e30 comp rx=0 tx=0).stop 2024-04-12T05:41:25.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.389+0000 7f34480ee700 1 -- 172.21.15.32:0/3636893302 shutdown_connections 2024-04-12T05:41:25.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.389+0000 7f34480ee700 1 --2- 172.21.15.32:0/3636893302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3440101d70 0x7f344010ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:25.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.389+0000 7f34480ee700 1 --2- 172.21.15.32:0/3636893302 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f342c06c8a0 0x7f342c06ed50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:25.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.389+0000 7f34480ee700 1 --2- 172.21.15.32:0/3636893302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3440102710 0x7f344010efa0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:25.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.389+0000 7f34480ee700 1 -- 172.21.15.32:0/3636893302 >> 172.21.15.32:0/3636893302 conn(0x7f3440072bc0 msgr2=0x7f34400fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:25.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.389+0000 7f34480ee700 1 -- 172.21.15.32:0/3636893302 shutdown_connections 2024-04-12T05:41:25.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:25.389+0000 7f34480ee700 1 -- 172.21.15.32:0/3636893302 wait complete. 2024-04-12T05:41:26.202 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-12T05:41:26.488 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:41:26.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:26 smithi032 ceph-mon[39731]: pgmap v82: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:41:26.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:26 smithi067 ceph-mon[49010]: pgmap v82: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:41:28.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.447+0000 7ff4ebad8700 1 -- 172.21.15.32:0/441458191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4e40fb4a0 msgr2=0x7ff4e40fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:28.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.447+0000 7ff4ebad8700 1 --2- 172.21.15.32:0/441458191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4e40fb4a0 0x7ff4e40fb910 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0004940 tx=0x7ff4e0009e30 comp rx=0 tx=0).stop 2024-04-12T05:41:28.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.447+0000 7ff4ebad8700 1 -- 172.21.15.32:0/441458191 shutdown_connections 2024-04-12T05:41:28.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.447+0000 7ff4ebad8700 1 --2- 172.21.15.32:0/441458191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4e40fb4a0 0x7ff4e40fb910 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:28.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.447+0000 7ff4ebad8700 1 --2- 172.21.15.32:0/441458191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4e4102e70 0x7ff4e4103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:28.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.447+0000 7ff4ebad8700 1 -- 172.21.15.32:0/441458191 >> 172.21.15.32:0/441458191 conn(0x7ff4e40714c0 msgr2=0x7ff4e40718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:28.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.447+0000 7ff4ebad8700 1 -- 172.21.15.32:0/441458191 shutdown_connections 2024-04-12T05:41:28.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.447+0000 7ff4ebad8700 1 -- 172.21.15.32:0/441458191 wait complete. 2024-04-12T05:41:28.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.448+0000 7ff4ebad8700 1 Processor -- start 2024-04-12T05:41:28.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.448+0000 7ff4ebad8700 1 -- start start 2024-04-12T05:41:28.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.448+0000 7ff4ebad8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4e40fb4a0 0x7ff4e4113b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:28.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.448+0000 7ff4ebad8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4e4102e70 0x7ff4e4114070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:28.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.448+0000 7ff4ebad8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e4114750 con 0x7ff4e4102e70 2024-04-12T05:41:28.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.448+0000 7ff4ebad8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e410dab0 con 0x7ff4e40fb4a0 2024-04-12T05:41:28.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.448+0000 7ff4e9073700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4e4102e70 0x7ff4e4114070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:28.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.448+0000 7ff4e9073700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4e4102e70 0x7ff4e4114070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52536/0 (socket says 172.21.15.32:52536) 2024-04-12T05:41:28.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.448+0000 7ff4e9073700 1 -- 172.21.15.32:0/241719876 learned_addr learned my addr 172.21.15.32:0/241719876 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:41:28.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.448+0000 7ff4e9874700 1 --2- 172.21.15.32:0/241719876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4e40fb4a0 0x7ff4e4113b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:28.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.449+0000 7ff4e9073700 1 -- 172.21.15.32:0/241719876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4e40fb4a0 msgr2=0x7ff4e4113b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:28.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.449+0000 7ff4e9073700 1 --2- 172.21.15.32:0/241719876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4e40fb4a0 0x7ff4e4113b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:28.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.449+0000 7ff4e9073700 1 -- 172.21.15.32:0/241719876 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e0005040 con 0x7ff4e4102e70 2024-04-12T05:41:28.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.449+0000 7ff4e9874700 1 --2- 172.21.15.32:0/241719876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4e40fb4a0 0x7ff4e4113b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:41:28.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.449+0000 7ff4e9073700 1 --2- 172.21.15.32:0/241719876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4e4102e70 0x7ff4e4114070 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0009180 tx=0x7ff4e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:28.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.449+0000 7ff4da7fc700 1 -- 172.21.15.32:0/241719876 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4e000e040 con 0x7ff4e4102e70 2024-04-12T05:41:28.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.449+0000 7ff4da7fc700 1 -- 172.21.15.32:0/241719876 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff4e0004290 con 0x7ff4e4102e70 2024-04-12T05:41:28.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.449+0000 7ff4ebad8700 1 -- 172.21.15.32:0/241719876 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4e410dd30 con 0x7ff4e4102e70 2024-04-12T05:41:28.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.449+0000 7ff4da7fc700 1 -- 172.21.15.32:0/241719876 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4e001c3f0 con 0x7ff4e4102e70 2024-04-12T05:41:28.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.449+0000 7ff4ebad8700 1 -- 172.21.15.32:0/241719876 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4e410e270 con 0x7ff4e4102e70 2024-04-12T05:41:28.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.450+0000 7ff4ebad8700 1 -- 172.21.15.32:0/241719876 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4e4100170 con 0x7ff4e4102e70 2024-04-12T05:41:28.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.450+0000 7ff4da7fc700 1 -- 172.21.15.32:0/241719876 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 34) v1 ==== 90703+0+0 (secure 0 0 0) 0x7ff4e0013790 con 0x7ff4e4102e70 2024-04-12T05:41:28.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.451+0000 7ff4da7fc700 1 --2- 172.21.15.32:0/241719876 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7ff4d006c8a0 0x7ff4d006ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:28.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.451+0000 7ff4da7fc700 1 -- 172.21.15.32:0/241719876 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7ff4e008cd90 con 0x7ff4e4102e70 2024-04-12T05:41:28.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.451+0000 7ff4e9874700 1 --2- 172.21.15.32:0/241719876 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7ff4d006c8a0 0x7ff4d006ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:28.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.451+0000 7ff4e9874700 1 --2- 172.21.15.32:0/241719876 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7ff4d006c8a0 0x7ff4d006ed50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4009b10 tx=0x7ff4d4008040 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:28.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.455+0000 7ff4da7fc700 1 -- 172.21.15.32:0/241719876 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7ff4e005ad30 con 0x7ff4e4102e70 2024-04-12T05:41:28.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:28 smithi032 ceph-mon[39731]: pgmap v83: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:41:28.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.681+0000 7ff4ebad8700 1 -- 172.21.15.32:0/241719876 --> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4e410f1f0 con 0x7ff4d006c8a0 2024-04-12T05:41:28.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.686+0000 7ff4da7fc700 1 -- 172.21.15.32:0/241719876 <== mgr.24463 v2:172.21.15.32:6800/2564765508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7ff4e410f1f0 con 0x7ff4d006c8a0 2024-04-12T05:41:28.684 INFO:teuthology.orchestra.run.smithi032.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:41:28.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.688+0000 7ff4ebad8700 1 -- 172.21.15.32:0/241719876 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7ff4d006c8a0 msgr2=0x7ff4d006ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:28.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.688+0000 7ff4ebad8700 1 --2- 172.21.15.32:0/241719876 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7ff4d006c8a0 0x7ff4d006ed50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4009b10 tx=0x7ff4d4008040 comp rx=0 tx=0).stop 2024-04-12T05:41:28.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.688+0000 7ff4ebad8700 1 -- 172.21.15.32:0/241719876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4e4102e70 msgr2=0x7ff4e4114070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:28.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.688+0000 7ff4ebad8700 1 --2- 172.21.15.32:0/241719876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4e4102e70 0x7ff4e4114070 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0009180 tx=0x7ff4e0009e30 comp rx=0 tx=0).stop 2024-04-12T05:41:28.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.688+0000 7ff4ebad8700 1 -- 172.21.15.32:0/241719876 shutdown_connections 2024-04-12T05:41:28.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.688+0000 7ff4ebad8700 1 --2- 172.21.15.32:0/241719876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4e40fb4a0 0x7ff4e4113b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:28.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.688+0000 7ff4ebad8700 1 --2- 172.21.15.32:0/241719876 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7ff4d006c8a0 0x7ff4d006ed50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:28.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.688+0000 7ff4ebad8700 1 --2- 172.21.15.32:0/241719876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4e4102e70 0x7ff4e4114070 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:28.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.688+0000 7ff4ebad8700 1 -- 172.21.15.32:0/241719876 >> 172.21.15.32:0/241719876 conn(0x7ff4e40714c0 msgr2=0x7ff4e40fa7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:28.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.688+0000 7ff4ebad8700 1 -- 172.21.15.32:0/241719876 shutdown_connections 2024-04-12T05:41:28.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:28.688+0000 7ff4ebad8700 1 -- 172.21.15.32:0/241719876 wait complete. 2024-04-12T05:41:28.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:28 smithi067 ceph-mon[49010]: pgmap v83: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:41:29.436 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-12T05:41:29.437 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-12T05:41:29.437 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-12T05:41:30.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:29 smithi067 ceph-mon[49010]: from='client.14822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:41:30.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:29 smithi067 ceph-mon[49010]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:41:30.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:29 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:30.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:29 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:41:30.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:29 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:41:30.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:29 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:41:30.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:29 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:30.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:29 smithi067 ceph-mon[49010]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:41:30.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:29 smithi032 ceph-mon[39731]: from='client.14822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:41:30.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:29 smithi032 ceph-mon[39731]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:41:30.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:29 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:30.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:29 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:41:30.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:29 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:41:30.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:29 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:41:30.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:29 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:30.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:29 smithi032 ceph-mon[39731]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:41:30.116 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T05:41:30.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:30 smithi032 ceph-mon[39731]: pgmap v84: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:41:31.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:30 smithi067 ceph-mon[49010]: pgmap v84: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:41:32.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.673+0000 7f21ad667700 1 -- 172.21.15.32:0/2503884510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21a8102710 msgr2=0x7f21a80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:32.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.673+0000 7f21ad667700 1 --2- 172.21.15.32:0/2503884510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21a8102710 0x7f21a80fbd80 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f2190004940 tx=0x7f2190009e30 comp rx=0 tx=0).stop 2024-04-12T05:41:32.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.673+0000 7f21ad667700 1 -- 172.21.15.32:0/2503884510 shutdown_connections 2024-04-12T05:41:32.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.673+0000 7f21ad667700 1 --2- 172.21.15.32:0/2503884510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21a8102710 0x7f21a80fbd80 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:32.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.673+0000 7f21ad667700 1 --2- 172.21.15.32:0/2503884510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21a8101d70 0x7f21a8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:32.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.673+0000 7f21ad667700 1 -- 172.21.15.32:0/2503884510 >> 172.21.15.32:0/2503884510 conn(0x7f21a8072bc0 msgr2=0x7f21a8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:32.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.673+0000 7f21ad667700 1 -- 172.21.15.32:0/2503884510 shutdown_connections 2024-04-12T05:41:32.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.673+0000 7f21ad667700 1 -- 172.21.15.32:0/2503884510 wait complete. 2024-04-12T05:41:32.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.674+0000 7f21ad667700 1 Processor -- start 2024-04-12T05:41:32.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.674+0000 7f21ad667700 1 -- start start 2024-04-12T05:41:32.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.674+0000 7f21ad667700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21a8101d70 0x7f21a81129f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:32.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.674+0000 7f21ad667700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21a8102710 0x7f21a8112f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:32.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.674+0000 7f21ad667700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a8113610 con 0x7f21a8102710 2024-04-12T05:41:32.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.674+0000 7f21ad667700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a810c970 con 0x7f21a8101d70 2024-04-12T05:41:32.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.674+0000 7f21a67fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21a8102710 0x7f21a8112f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:32.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.674+0000 7f21a67fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21a8102710 0x7f21a8112f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52484/0 (socket says 172.21.15.32:52484) 2024-04-12T05:41:32.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.674+0000 7f21a67fc700 1 -- 172.21.15.32:0/3545252689 learned_addr learned my addr 172.21.15.32:0/3545252689 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:41:32.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.675+0000 7f21a6ffd700 1 --2- 172.21.15.32:0/3545252689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21a8101d70 0x7f21a81129f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:32.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.675+0000 7f21a67fc700 1 -- 172.21.15.32:0/3545252689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21a8101d70 msgr2=0x7f21a81129f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:32.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.675+0000 7f21a67fc700 1 --2- 172.21.15.32:0/3545252689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21a8101d70 0x7f21a81129f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:32.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.675+0000 7f21a67fc700 1 -- 172.21.15.32:0/3545252689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2190005040 con 0x7f21a8102710 2024-04-12T05:41:32.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.675+0000 7f21a67fc700 1 --2- 172.21.15.32:0/3545252689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21a8102710 0x7f21a8112f30 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f21900093b0 tx=0x7f2190009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:32.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.675+0000 7f219f7fe700 1 -- 172.21.15.32:0/3545252689 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f219000e040 con 0x7f21a8102710 2024-04-12T05:41:32.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.675+0000 7f219f7fe700 1 -- 172.21.15.32:0/3545252689 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f219000be30 con 0x7f21a8102710 2024-04-12T05:41:32.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.675+0000 7f21ad667700 1 -- 172.21.15.32:0/3545252689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21a810cbf0 con 0x7f21a8102710 2024-04-12T05:41:32.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.675+0000 7f219f7fe700 1 -- 172.21.15.32:0/3545252689 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2190013730 con 0x7f21a8102710 2024-04-12T05:41:32.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.675+0000 7f21ad667700 1 -- 172.21.15.32:0/3545252689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21a810d150 con 0x7f21a8102710 2024-04-12T05:41:32.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.676+0000 7f21ad667700 1 -- 172.21.15.32:0/3545252689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21a80fd4c0 con 0x7f21a8102710 2024-04-12T05:41:32.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.676+0000 7f219f7fe700 1 -- 172.21.15.32:0/3545252689 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 34) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f219001eb60 con 0x7f21a8102710 2024-04-12T05:41:32.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.677+0000 7f219f7fe700 1 --2- 172.21.15.32:0/3545252689 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f219406c8b0 0x7f219406ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:32.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.677+0000 7f219f7fe700 1 -- 172.21.15.32:0/3545252689 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f219008f9e0 con 0x7f21a8102710 2024-04-12T05:41:32.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.677+0000 7f21a6ffd700 1 --2- 172.21.15.32:0/3545252689 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f219406c8b0 0x7f219406ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:32.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.677+0000 7f21a6ffd700 1 --2- 172.21.15.32:0/3545252689 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f219406c8b0 0x7f219406ed60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f21a0006fd0 tx=0x7f21a0008040 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:32.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.681+0000 7f219f7fe700 1 -- 172.21.15.32:0/3545252689 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f219005d900 con 0x7f21a8102710 2024-04-12T05:41:32.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.906+0000 7f21ad667700 1 -- 172.21.15.32:0/3545252689 --> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f21a8108df0 con 0x7f219406c8b0 2024-04-12T05:41:32.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.907+0000 7f219f7fe700 1 -- 172.21.15.32:0/3545252689 <== mgr.24463 v2:172.21.15.32:6800/2564765508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f21a8108df0 con 0x7f219406c8b0 2024-04-12T05:41:32.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.909+0000 7f21ad667700 1 -- 172.21.15.32:0/3545252689 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f219406c8b0 msgr2=0x7f219406ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:32.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.909+0000 7f21ad667700 1 --2- 172.21.15.32:0/3545252689 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f219406c8b0 0x7f219406ed60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f21a0006fd0 tx=0x7f21a0008040 comp rx=0 tx=0).stop 2024-04-12T05:41:32.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.909+0000 7f21ad667700 1 -- 172.21.15.32:0/3545252689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21a8102710 msgr2=0x7f21a8112f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:32.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.909+0000 7f21ad667700 1 --2- 172.21.15.32:0/3545252689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21a8102710 0x7f21a8112f30 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f21900093b0 tx=0x7f2190009e30 comp rx=0 tx=0).stop 2024-04-12T05:41:32.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.909+0000 7f21ad667700 1 -- 172.21.15.32:0/3545252689 shutdown_connections 2024-04-12T05:41:32.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.909+0000 7f21ad667700 1 --2- 172.21.15.32:0/3545252689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21a8101d70 0x7f21a81129f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:32.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.909+0000 7f21ad667700 1 --2- 172.21.15.32:0/3545252689 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f219406c8b0 0x7f219406ed60 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:32.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.909+0000 7f21ad667700 1 --2- 172.21.15.32:0/3545252689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21a8102710 0x7f21a8112f30 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:32.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.909+0000 7f21ad667700 1 -- 172.21.15.32:0/3545252689 >> 172.21.15.32:0/3545252689 conn(0x7f21a8072bc0 msgr2=0x7f21a80fb060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:32.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.909+0000 7f21ad667700 1 -- 172.21.15.32:0/3545252689 shutdown_connections 2024-04-12T05:41:32.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:32.909+0000 7f21ad667700 1 -- 172.21.15.32:0/3545252689 wait complete. 2024-04-12T05:41:32.916 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:41:33.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.014+0000 7f82bfe24700 1 -- 172.21.15.32:0/2918736412 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82b8101d70 msgr2=0x7f82b8102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:33.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.014+0000 7f82bfe24700 1 --2- 172.21.15.32:0/2918736412 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82b8101d70 0x7f82b8102140 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f82b4004940 tx=0x7f82b4009e30 comp rx=0 tx=0).stop 2024-04-12T05:41:33.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.014+0000 7f82bfe24700 1 -- 172.21.15.32:0/2918736412 shutdown_connections 2024-04-12T05:41:33.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.014+0000 7f82bfe24700 1 --2- 172.21.15.32:0/2918736412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b8102710 0x7f82b80fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:33.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.014+0000 7f82bfe24700 1 --2- 172.21.15.32:0/2918736412 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82b8101d70 0x7f82b8102140 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:33.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.014+0000 7f82bfe24700 1 -- 172.21.15.32:0/2918736412 >> 172.21.15.32:0/2918736412 conn(0x7f82b8072bc0 msgr2=0x7f82b8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:33.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.014+0000 7f82bfe24700 1 -- 172.21.15.32:0/2918736412 shutdown_connections 2024-04-12T05:41:33.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.014+0000 7f82bfe24700 1 -- 172.21.15.32:0/2918736412 wait complete. 2024-04-12T05:41:33.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.015+0000 7f82bfe24700 1 Processor -- start 2024-04-12T05:41:33.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.015+0000 7f82bfe24700 1 -- start start 2024-04-12T05:41:33.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.015+0000 7f82bfe24700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82b8101d70 0x7f82b8113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:33.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.015+0000 7f82bfe24700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b8102710 0x7f82b8114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:33.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.015+0000 7f82bfe24700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b8114670 con 0x7f82b8101d70 2024-04-12T05:41:33.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.015+0000 7f82bfe24700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b810da60 con 0x7f82b8102710 2024-04-12T05:41:33.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.015+0000 7f82bdbc0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82b8101d70 0x7f82b8113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:33.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.015+0000 7f82bdbc0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82b8101d70 0x7f82b8113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52516/0 (socket says 172.21.15.32:52516) 2024-04-12T05:41:33.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.015+0000 7f82bdbc0700 1 -- 172.21.15.32:0/1556537564 learned_addr learned my addr 172.21.15.32:0/1556537564 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:41:33.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.015+0000 7f82bd3bf700 1 --2- 172.21.15.32:0/1556537564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b8102710 0x7f82b8114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:33.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.015+0000 7f82bdbc0700 1 -- 172.21.15.32:0/1556537564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b8102710 msgr2=0x7f82b8114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:33.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.015+0000 7f82bdbc0700 1 --2- 172.21.15.32:0/1556537564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b8102710 0x7f82b8114020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:33.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.015+0000 7f82bdbc0700 1 -- 172.21.15.32:0/1556537564 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b4005040 con 0x7f82b8101d70 2024-04-12T05:41:33.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.016+0000 7f82bdbc0700 1 --2- 172.21.15.32:0/1556537564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82b8101d70 0x7f82b8113ae0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f82b400b3b0 tx=0x7f82b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:33.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.016+0000 7f82ae7fc700 1 -- 172.21.15.32:0/1556537564 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82b400e040 con 0x7f82b8101d70 2024-04-12T05:41:33.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.016+0000 7f82bfe24700 1 -- 172.21.15.32:0/1556537564 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82b810dce0 con 0x7f82b8101d70 2024-04-12T05:41:33.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.016+0000 7f82ae7fc700 1 -- 172.21.15.32:0/1556537564 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f82b4013420 con 0x7f82b8101d70 2024-04-12T05:41:33.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.016+0000 7f82ae7fc700 1 -- 172.21.15.32:0/1556537564 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82b401c610 con 0x7f82b8101d70 2024-04-12T05:41:33.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.016+0000 7f82bfe24700 1 -- 172.21.15.32:0/1556537564 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b810e2a0 con 0x7f82b8101d70 2024-04-12T05:41:33.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.017+0000 7f82bfe24700 1 -- 172.21.15.32:0/1556537564 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82b80fd4c0 con 0x7f82b8101d70 2024-04-12T05:41:33.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.017+0000 7f82ae7fc700 1 -- 172.21.15.32:0/1556537564 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 34) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f82b4013a00 con 0x7f82b8101d70 2024-04-12T05:41:33.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.018+0000 7f82ae7fc700 1 --2- 172.21.15.32:0/1556537564 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f82a406c8a0 0x7f82a406ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:33.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.018+0000 7f82ae7fc700 1 -- 172.21.15.32:0/1556537564 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f82b408cc50 con 0x7f82b8101d70 2024-04-12T05:41:33.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.018+0000 7f82bd3bf700 1 --2- 172.21.15.32:0/1556537564 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f82a406c8a0 0x7f82a406ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:33.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.018+0000 7f82bd3bf700 1 --2- 172.21.15.32:0/1556537564 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f82a406c8a0 0x7f82a406ed50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f82b810ec20 tx=0x7f82a800b040 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:33.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:32 smithi067 ceph-mon[49010]: pgmap v85: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:41:33.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:32 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:41:33.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:32 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:33.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:32 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:41:33.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:32 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:33.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:32 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:41:33.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:32 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:33.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:32 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-12T05:41:33.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.022+0000 7f82ae7fc700 1 -- 172.21.15.32:0/1556537564 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f82b405abf0 con 0x7f82b8101d70 2024-04-12T05:41:33.056 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:32 smithi032 ceph-mon[39731]: pgmap v85: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:41:33.056 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:32 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:41:33.056 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:32 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:33.056 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:32 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:41:33.056 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:32 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:33.056 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:32 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:41:33.056 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:32 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:33.056 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:32 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-12T05:41:33.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.255+0000 7f82bfe24700 1 -- 172.21.15.32:0/1556537564 --> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f82b8108da0 con 0x7f82a406c8a0 2024-04-12T05:41:33.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.256+0000 7f82ae7fc700 1 -- 172.21.15.32:0/1556537564 <== mgr.24463 v2:172.21.15.32:6800/2564765508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f82b8108da0 con 0x7f82a406c8a0 2024-04-12T05:41:33.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.259+0000 7f82bfe24700 1 -- 172.21.15.32:0/1556537564 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f82a406c8a0 msgr2=0x7f82a406ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:33.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.259+0000 7f82bfe24700 1 --2- 172.21.15.32:0/1556537564 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f82a406c8a0 0x7f82a406ed50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f82b810ec20 tx=0x7f82a800b040 comp rx=0 tx=0).stop 2024-04-12T05:41:33.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.259+0000 7f82bfe24700 1 -- 172.21.15.32:0/1556537564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82b8101d70 msgr2=0x7f82b8113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:33.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.259+0000 7f82bfe24700 1 --2- 172.21.15.32:0/1556537564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82b8101d70 0x7f82b8113ae0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f82b400b3b0 tx=0x7f82b4009e30 comp rx=0 tx=0).stop 2024-04-12T05:41:33.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.259+0000 7f82bfe24700 1 -- 172.21.15.32:0/1556537564 shutdown_connections 2024-04-12T05:41:33.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.259+0000 7f82bfe24700 1 --2- 172.21.15.32:0/1556537564 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f82a406c8a0 0x7f82a406ed50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:33.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.259+0000 7f82bfe24700 1 --2- 172.21.15.32:0/1556537564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82b8101d70 0x7f82b8113ae0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:33.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.259+0000 7f82bfe24700 1 --2- 172.21.15.32:0/1556537564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b8102710 0x7f82b8114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:33.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.259+0000 7f82bfe24700 1 -- 172.21.15.32:0/1556537564 >> 172.21.15.32:0/1556537564 conn(0x7f82b8072bc0 msgr2=0x7f82b80659e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:33.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.259+0000 7f82bfe24700 1 -- 172.21.15.32:0/1556537564 shutdown_connections 2024-04-12T05:41:33.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.259+0000 7f82bfe24700 1 -- 172.21.15.32:0/1556537564 wait complete. 2024-04-12T05:41:33.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.364+0000 7f3cb866e700 1 -- 172.21.15.32:0/2663573332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cb01037b0 msgr2=0x7f3cb00fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:33.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.364+0000 7f3cb866e700 1 --2- 172.21.15.32:0/2663573332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cb01037b0 0x7f3cb00fce20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0004940 tx=0x7f3ca0009e30 comp rx=0 tx=0).stop 2024-04-12T05:41:33.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.364+0000 7f3cb866e700 1 -- 172.21.15.32:0/2663573332 shutdown_connections 2024-04-12T05:41:33.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.364+0000 7f3cb866e700 1 --2- 172.21.15.32:0/2663573332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cb01037b0 0x7f3cb00fce20 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:33.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.364+0000 7f3cb866e700 1 --2- 172.21.15.32:0/2663573332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cb0102e10 0x7f3cb01031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:33.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.364+0000 7f3cb866e700 1 -- 172.21.15.32:0/2663573332 >> 172.21.15.32:0/2663573332 conn(0x7f3cb00f89a0 msgr2=0x7f3cb00fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:33.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.364+0000 7f3cb866e700 1 -- 172.21.15.32:0/2663573332 shutdown_connections 2024-04-12T05:41:33.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.364+0000 7f3cb866e700 1 -- 172.21.15.32:0/2663573332 wait complete. 2024-04-12T05:41:33.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.364+0000 7f3cb866e700 1 Processor -- start 2024-04-12T05:41:33.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.365+0000 7f3cb866e700 1 -- start start 2024-04-12T05:41:33.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.365+0000 7f3cb866e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cb0102e10 0x7f3cb00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:33.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.365+0000 7f3cb866e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cb01037b0 0x7f3cb0070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:33.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.365+0000 7f3cb866e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb0070d90 con 0x7f3cb01037b0 2024-04-12T05:41:33.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.365+0000 7f3cb866e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb0070ed0 con 0x7f3cb0102e10 2024-04-12T05:41:33.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.365+0000 7f3cb5c09700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cb01037b0 0x7f3cb0070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:33.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.365+0000 7f3cb5c09700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cb01037b0 0x7f3cb0070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52534/0 (socket says 172.21.15.32:52534) 2024-04-12T05:41:33.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.365+0000 7f3cb5c09700 1 -- 172.21.15.32:0/3063859926 learned_addr learned my addr 172.21.15.32:0/3063859926 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:41:33.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.365+0000 7f3cb640a700 1 --2- 172.21.15.32:0/3063859926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cb0102e10 0x7f3cb00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:33.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.365+0000 7f3cb5c09700 1 -- 172.21.15.32:0/3063859926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cb0102e10 msgr2=0x7f3cb00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:33.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.365+0000 7f3cb5c09700 1 --2- 172.21.15.32:0/3063859926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cb0102e10 0x7f3cb00702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:33.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.365+0000 7f3cb5c09700 1 -- 172.21.15.32:0/3063859926 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca0005040 con 0x7f3cb01037b0 2024-04-12T05:41:33.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.365+0000 7f3cb5c09700 1 --2- 172.21.15.32:0/3063859926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cb01037b0 0x7f3cb0070800 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f3ca000b3b0 tx=0x7f3ca0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:33.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.366+0000 7f3ca6ffd700 1 -- 172.21.15.32:0/3063859926 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ca000e040 con 0x7f3cb01037b0 2024-04-12T05:41:33.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.366+0000 7f3ca6ffd700 1 -- 172.21.15.32:0/3063859926 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ca0009b10 con 0x7f3cb01037b0 2024-04-12T05:41:33.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.366+0000 7f3cb866e700 1 -- 172.21.15.32:0/3063859926 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cb0071120 con 0x7f3cb01037b0 2024-04-12T05:41:33.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.366+0000 7f3ca6ffd700 1 -- 172.21.15.32:0/3063859926 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ca0013420 con 0x7f3cb01037b0 2024-04-12T05:41:33.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.366+0000 7f3cb866e700 1 -- 172.21.15.32:0/3063859926 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cb011a200 con 0x7f3cb01037b0 2024-04-12T05:41:33.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.367+0000 7f3cb866e700 1 -- 172.21.15.32:0/3063859926 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cb00fe560 con 0x7f3cb01037b0 2024-04-12T05:41:33.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.367+0000 7f3ca6ffd700 1 -- 172.21.15.32:0/3063859926 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 34) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f3ca001c6c0 con 0x7f3cb01037b0 2024-04-12T05:41:33.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.368+0000 7f3ca6ffd700 1 --2- 172.21.15.32:0/3063859926 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f3c9c06c8a0 0x7f3c9c06ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:33.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.368+0000 7f3ca6ffd700 1 -- 172.21.15.32:0/3063859926 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f3ca008ccf0 con 0x7f3cb01037b0 2024-04-12T05:41:33.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.368+0000 7f3cb640a700 1 --2- 172.21.15.32:0/3063859926 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f3c9c06c8a0 0x7f3c9c06ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:33.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.368+0000 7f3cb640a700 1 --2- 172.21.15.32:0/3063859926 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f3c9c06c8a0 0x7f3c9c06ed50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3cac004f40 tx=0x7f3cac00b040 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:33.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.372+0000 7f3ca6ffd700 1 -- 172.21.15.32:0/3063859926 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f3ca0057350 con 0x7f3cb01037b0 2024-04-12T05:41:33.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.597+0000 7f3cb866e700 1 -- 172.21.15.32:0/3063859926 --> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3cb004bed0 con 0x7f3c9c06c8a0 2024-04-12T05:41:33.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.607+0000 7f3ca6ffd700 1 -- 172.21.15.32:0/3063859926 <== mgr.24463 v2:172.21.15.32:6800/2564765508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f3cb004bed0 con 0x7f3c9c06c8a0 2024-04-12T05:41:33.604 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:41:33.605 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (81s) 58s ago 12m 17.0M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:41:33.605 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (13m) 58s ago 13m 6656k - 17.2.7-827-gd3b5060a 21de9b7369ad a9cfc00c311b 2024-04-12T05:41:33.605 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (11m) 114s ago 11m 6651k - 17.2.7-827-gd3b5060a 21de9b7369ad 1f2eea4542d9 2024-04-12T05:41:33.605 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (64s) 58s ago 12m 44.0M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:41:33.605 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (7m) 58s ago 7m 64.5M - 17.2.7-827-gd3b5060a 21de9b7369ad e7ada8e5259c 2024-04-12T05:41:33.605 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (7m) 58s ago 7m 116M - 17.2.7-827-gd3b5060a 21de9b7369ad 55124da9afdb 2024-04-12T05:41:33.605 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (6m) 114s ago 6m 62.5M - 17.2.7-827-gd3b5060a 21de9b7369ad 0c295d31a4a6 2024-04-12T05:41:33.605 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (7m) 114s ago 7m 120M - 17.2.7-827-gd3b5060a 21de9b7369ad 6eecd7deaac2 2024-04-12T05:41:33.605 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (3m) 58s ago 14m 502M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:41:33.605 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (2m) 114s ago 11m 449M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:41:33.605 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (14m) 58s ago 14m 53.6M 2048M 17.2.7-827-gd3b5060a 21de9b7369ad 75aad95b984d 2024-04-12T05:41:33.605 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (11m) 114s ago 11m 42.3M 2048M 17.2.7-827-gd3b5060a 21de9b7369ad 157be8a8c6d8 2024-04-12T05:41:33.605 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (2m) 58s ago 12m 14.3M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:41:33.605 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (118s) 114s ago 11m 3594k - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:41:33.606 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (11m) 58s ago 10m 220M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad 97446e42ceba 2024-04-12T05:41:33.606 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (10m) 58s ago 10m 236M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad bf9f9a2db2d5 2024-04-12T05:41:33.606 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (10m) 58s ago 10m 189M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad f055cf1ca665 2024-04-12T05:41:33.606 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (9m) 114s ago 9m 255M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad 61b851235351 2024-04-12T05:41:33.606 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (9m) 114s ago 9m 227M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad 2c84d9047e9c 2024-04-12T05:41:33.606 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (8m) 114s ago 8m 217M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad e54839bb6055 2024-04-12T05:41:33.606 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (98s) 58s ago 12m 46.5M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:41:33.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.609+0000 7f3cb866e700 1 -- 172.21.15.32:0/3063859926 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f3c9c06c8a0 msgr2=0x7f3c9c06ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:33.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.609+0000 7f3cb866e700 1 --2- 172.21.15.32:0/3063859926 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f3c9c06c8a0 0x7f3c9c06ed50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3cac004f40 tx=0x7f3cac00b040 comp rx=0 tx=0).stop 2024-04-12T05:41:33.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.609+0000 7f3cb866e700 1 -- 172.21.15.32:0/3063859926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cb01037b0 msgr2=0x7f3cb0070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:33.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.609+0000 7f3cb866e700 1 --2- 172.21.15.32:0/3063859926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cb01037b0 0x7f3cb0070800 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f3ca000b3b0 tx=0x7f3ca0009e30 comp rx=0 tx=0).stop 2024-04-12T05:41:33.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.609+0000 7f3cb866e700 1 -- 172.21.15.32:0/3063859926 shutdown_connections 2024-04-12T05:41:33.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.609+0000 7f3cb866e700 1 --2- 172.21.15.32:0/3063859926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cb0102e10 0x7f3cb00702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:33.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.609+0000 7f3cb866e700 1 --2- 172.21.15.32:0/3063859926 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f3c9c06c8a0 0x7f3c9c06ed50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:33.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.609+0000 7f3cb866e700 1 --2- 172.21.15.32:0/3063859926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cb01037b0 0x7f3cb0070800 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:33.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.609+0000 7f3cb866e700 1 -- 172.21.15.32:0/3063859926 >> 172.21.15.32:0/3063859926 conn(0x7f3cb00f89a0 msgr2=0x7f3cb00f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:33.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.609+0000 7f3cb866e700 1 -- 172.21.15.32:0/3063859926 shutdown_connections 2024-04-12T05:41:33.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.609+0000 7f3cb866e700 1 -- 172.21.15.32:0/3063859926 wait complete. 2024-04-12T05:41:33.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.716+0000 7f0b7ef06700 1 -- 172.21.15.32:0/49241813 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b781037c0 msgr2=0x7f0b780fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:33.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.716+0000 7f0b7ef06700 1 --2- 172.21.15.32:0/49241813 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b781037c0 0x7f0b780fce30 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0b64004940 tx=0x7f0b64009e30 comp rx=0 tx=0).stop 2024-04-12T05:41:33.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.716+0000 7f0b7ef06700 1 -- 172.21.15.32:0/49241813 shutdown_connections 2024-04-12T05:41:33.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.716+0000 7f0b7ef06700 1 --2- 172.21.15.32:0/49241813 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b781037c0 0x7f0b780fce30 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:33.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.716+0000 7f0b7ef06700 1 --2- 172.21.15.32:0/49241813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b78102e20 0x7f0b781031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:33.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.716+0000 7f0b7ef06700 1 -- 172.21.15.32:0/49241813 >> 172.21.15.32:0/49241813 conn(0x7f0b780f89d0 msgr2=0x7f0b780fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:33.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.716+0000 7f0b7ef06700 1 -- 172.21.15.32:0/49241813 shutdown_connections 2024-04-12T05:41:33.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.716+0000 7f0b7ef06700 1 -- 172.21.15.32:0/49241813 wait complete. 2024-04-12T05:41:33.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.717+0000 7f0b7ef06700 1 Processor -- start 2024-04-12T05:41:33.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.717+0000 7f0b7ef06700 1 -- start start 2024-04-12T05:41:33.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.717+0000 7f0b7ef06700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b78102e20 0x7f0b78113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:33.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.717+0000 7f0b7ef06700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b781037c0 0x7f0b78114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:33.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.717+0000 7f0b7ef06700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b78114660 con 0x7f0b781037c0 2024-04-12T05:41:33.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.717+0000 7f0b7ef06700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7810da50 con 0x7f0b78102e20 2024-04-12T05:41:33.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.717+0000 7f0b77fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b781037c0 0x7f0b78114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:33.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.717+0000 7f0b77fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b781037c0 0x7f0b78114010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52540/0 (socket says 172.21.15.32:52540) 2024-04-12T05:41:33.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.717+0000 7f0b77fff700 1 -- 172.21.15.32:0/952839881 learned_addr learned my addr 172.21.15.32:0/952839881 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:41:33.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.717+0000 7f0b7cca2700 1 --2- 172.21.15.32:0/952839881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b78102e20 0x7f0b78113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:33.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.718+0000 7f0b77fff700 1 -- 172.21.15.32:0/952839881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b78102e20 msgr2=0x7f0b78113ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:33.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.718+0000 7f0b77fff700 1 --2- 172.21.15.32:0/952839881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b78102e20 0x7f0b78113ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:33.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.718+0000 7f0b77fff700 1 -- 172.21.15.32:0/952839881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b64005040 con 0x7f0b781037c0 2024-04-12T05:41:33.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.718+0000 7f0b77fff700 1 --2- 172.21.15.32:0/952839881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b781037c0 0x7f0b78114010 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0b64009390 tx=0x7f0b64009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:33.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.720+0000 7f0b757fa700 1 -- 172.21.15.32:0/952839881 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b6400e040 con 0x7f0b781037c0 2024-04-12T05:41:33.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.720+0000 7f0b7ef06700 1 -- 172.21.15.32:0/952839881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b7810dcd0 con 0x7f0b781037c0 2024-04-12T05:41:33.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.720+0000 7f0b7ef06700 1 -- 172.21.15.32:0/952839881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b7810e290 con 0x7f0b781037c0 2024-04-12T05:41:33.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.720+0000 7f0b757fa700 1 -- 172.21.15.32:0/952839881 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b64004290 con 0x7f0b781037c0 2024-04-12T05:41:33.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.720+0000 7f0b757fa700 1 -- 172.21.15.32:0/952839881 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b64013e70 con 0x7f0b781037c0 2024-04-12T05:41:33.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.721+0000 7f0b7ef06700 1 -- 172.21.15.32:0/952839881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b780fe570 con 0x7f0b781037c0 2024-04-12T05:41:33.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.721+0000 7f0b757fa700 1 -- 172.21.15.32:0/952839881 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 34) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f0b6401c440 con 0x7f0b781037c0 2024-04-12T05:41:33.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.721+0000 7f0b757fa700 1 --2- 172.21.15.32:0/952839881 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f0b6806c8b0 0x7f0b6806ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:33.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.721+0000 7f0b757fa700 1 -- 172.21.15.32:0/952839881 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f0b6408cbb0 con 0x7f0b781037c0 2024-04-12T05:41:33.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.722+0000 7f0b7cca2700 1 --2- 172.21.15.32:0/952839881 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f0b6806c8b0 0x7f0b6806ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:33.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.722+0000 7f0b7cca2700 1 --2- 172.21.15.32:0/952839881 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f0b6806c8b0 0x7f0b6806ed60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c0041d0 tx=0x7f0b6c0043c0 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:33.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:33.726+0000 7f0b757fa700 1 -- 172.21.15.32:0/952839881 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f0b64093080 con 0x7f0b781037c0 2024-04-12T05:41:34.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.017+0000 7f0b7ef06700 1 -- 172.21.15.32:0/952839881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0b78109dc0 con 0x7f0b781037c0 2024-04-12T05:41:34.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.018+0000 7f0b757fa700 1 -- 172.21.15.32:0/952839881 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7f0b6405ac60 con 0x7f0b781037c0 2024-04-12T05:41:34.016 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:41:34.016 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:41:34.016 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 2 2024-04-12T05:41:34.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:41:34.016 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:41:34.016 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:41:34.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:41:34.016 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:41:34.016 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 6 2024-04-12T05:41:34.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:41:34.017 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:41:34.017 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 4 2024-04-12T05:41:34.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:41:34.017 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:41:34.017 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 12, 2024-04-12T05:41:34.017 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:41:34.017 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:41:34.017 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:41:34.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:33 smithi067 ceph-mon[49010]: Upgrade: Target is version 18.2.2-1187-gd540ebac (reef) 2024-04-12T05:41:34.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:33 smithi067 ceph-mon[49010]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3, digests ['quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3'] 2024-04-12T05:41:34.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:33 smithi067 ceph-mon[49010]: Upgrade: Setting container_image for all mgr 2024-04-12T05:41:34.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.020+0000 7f0b7ef06700 1 -- 172.21.15.32:0/952839881 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f0b6806c8b0 msgr2=0x7f0b6806ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:34.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.020+0000 7f0b7ef06700 1 --2- 172.21.15.32:0/952839881 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f0b6806c8b0 0x7f0b6806ed60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c0041d0 tx=0x7f0b6c0043c0 comp rx=0 tx=0).stop 2024-04-12T05:41:34.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.020+0000 7f0b7ef06700 1 -- 172.21.15.32:0/952839881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b781037c0 msgr2=0x7f0b78114010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:34.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.020+0000 7f0b7ef06700 1 --2- 172.21.15.32:0/952839881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b781037c0 0x7f0b78114010 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0b64009390 tx=0x7f0b64009e30 comp rx=0 tx=0).stop 2024-04-12T05:41:34.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.020+0000 7f0b7ef06700 1 -- 172.21.15.32:0/952839881 shutdown_connections 2024-04-12T05:41:34.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.020+0000 7f0b7ef06700 1 --2- 172.21.15.32:0/952839881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b78102e20 0x7f0b78113ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:34.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.020+0000 7f0b7ef06700 1 --2- 172.21.15.32:0/952839881 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f0b6806c8b0 0x7f0b6806ed60 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:34.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.020+0000 7f0b7ef06700 1 --2- 172.21.15.32:0/952839881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b781037c0 0x7f0b78114010 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:34.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.020+0000 7f0b7ef06700 1 -- 172.21.15.32:0/952839881 >> 172.21.15.32:0/952839881 conn(0x7f0b780f89d0 msgr2=0x7f0b780f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:34.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.020+0000 7f0b7ef06700 1 -- 172.21.15.32:0/952839881 shutdown_connections 2024-04-12T05:41:34.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.020+0000 7f0b7ef06700 1 -- 172.21.15.32:0/952839881 wait complete. 2024-04-12T05:41:34.032 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:33 smithi032 ceph-mon[39731]: Upgrade: Target is version 18.2.2-1187-gd540ebac (reef) 2024-04-12T05:41:34.032 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:33 smithi032 ceph-mon[39731]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3, digests ['quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3'] 2024-04-12T05:41:34.033 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:33 smithi032 ceph-mon[39731]: Upgrade: Setting container_image for all mgr 2024-04-12T05:41:34.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.136+0000 7f4e50a51700 1 -- 172.21.15.32:0/3765874934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e4c0fdeb0 msgr2=0x7f4e4c0fe280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:34.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.136+0000 7f4e50a51700 1 --2- 172.21.15.32:0/3765874934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e4c0fdeb0 0x7f4e4c0fe280 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c004970 tx=0x7f4e3c009e30 comp rx=0 tx=0).stop 2024-04-12T05:41:34.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.137+0000 7f4e50a51700 1 -- 172.21.15.32:0/3765874934 shutdown_connections 2024-04-12T05:41:34.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.137+0000 7f4e50a51700 1 --2- 172.21.15.32:0/3765874934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e4c0fe850 0x7f4e4c106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:34.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.137+0000 7f4e50a51700 1 --2- 172.21.15.32:0/3765874934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e4c0fdeb0 0x7f4e4c0fe280 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:34.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.137+0000 7f4e50a51700 1 -- 172.21.15.32:0/3765874934 >> 172.21.15.32:0/3765874934 conn(0x7f4e4c072bc0 msgr2=0x7f4e4c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:34.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.137+0000 7f4e50a51700 1 -- 172.21.15.32:0/3765874934 shutdown_connections 2024-04-12T05:41:34.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.137+0000 7f4e50a51700 1 -- 172.21.15.32:0/3765874934 wait complete. 2024-04-12T05:41:34.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.137+0000 7f4e50a51700 1 Processor -- start 2024-04-12T05:41:34.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.137+0000 7f4e50a51700 1 -- start start 2024-04-12T05:41:34.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.137+0000 7f4e50a51700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e4c0fdeb0 0x7f4e4c115bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:34.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.137+0000 7f4e50a51700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e4c0fe850 0x7f4e4c116130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:34.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.137+0000 7f4e50a51700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4c116780 con 0x7f4e4c0fdeb0 2024-04-12T05:41:34.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.137+0000 7f4e50a51700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4c10fb90 con 0x7f4e4c0fe850 2024-04-12T05:41:34.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.138+0000 7f4e4a59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e4c0fdeb0 0x7f4e4c115bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:34.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.138+0000 7f4e4a59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e4c0fdeb0 0x7f4e4c115bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52552/0 (socket says 172.21.15.32:52552) 2024-04-12T05:41:34.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.138+0000 7f4e4a59c700 1 -- 172.21.15.32:0/592950362 learned_addr learned my addr 172.21.15.32:0/592950362 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:41:34.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.138+0000 7f4e49d9b700 1 --2- 172.21.15.32:0/592950362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e4c0fe850 0x7f4e4c116130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:34.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.138+0000 7f4e4a59c700 1 -- 172.21.15.32:0/592950362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e4c0fe850 msgr2=0x7f4e4c116130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:34.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.138+0000 7f4e4a59c700 1 --2- 172.21.15.32:0/592950362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e4c0fe850 0x7f4e4c116130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:34.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.138+0000 7f4e4a59c700 1 -- 172.21.15.32:0/592950362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e3c005040 con 0x7f4e4c0fdeb0 2024-04-12T05:41:34.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.138+0000 7f4e4a59c700 1 --2- 172.21.15.32:0/592950362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e4c0fdeb0 0x7f4e4c115bf0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c0043d0 tx=0x7f4e3c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:34.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.138+0000 7f4e3affd700 1 -- 172.21.15.32:0/592950362 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e3c00e040 con 0x7f4e4c0fdeb0 2024-04-12T05:41:34.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.138+0000 7f4e50a51700 1 -- 172.21.15.32:0/592950362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e4c10fe10 con 0x7f4e4c0fdeb0 2024-04-12T05:41:34.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.138+0000 7f4e3affd700 1 -- 172.21.15.32:0/592950362 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4e3c013420 con 0x7f4e4c0fdeb0 2024-04-12T05:41:34.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.138+0000 7f4e3affd700 1 -- 172.21.15.32:0/592950362 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e3c01c630 con 0x7f4e4c0fdeb0 2024-04-12T05:41:34.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.138+0000 7f4e50a51700 1 -- 172.21.15.32:0/592950362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e4c110350 con 0x7f4e4c0fdeb0 2024-04-12T05:41:34.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.139+0000 7f4e50a51700 1 -- 172.21.15.32:0/592950362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e4c1069f0 con 0x7f4e4c0fdeb0 2024-04-12T05:41:34.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.140+0000 7f4e3affd700 1 -- 172.21.15.32:0/592950362 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 34) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f4e3c013970 con 0x7f4e4c0fdeb0 2024-04-12T05:41:34.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.140+0000 7f4e3affd700 1 --2- 172.21.15.32:0/592950362 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f4e3406c8a0 0x7f4e3406ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:34.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.140+0000 7f4e3affd700 1 -- 172.21.15.32:0/592950362 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f4e3c08cd60 con 0x7f4e4c0fdeb0 2024-04-12T05:41:34.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.140+0000 7f4e49d9b700 1 --2- 172.21.15.32:0/592950362 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f4e3406c8a0 0x7f4e3406ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:34.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.141+0000 7f4e49d9b700 1 --2- 172.21.15.32:0/592950362 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f4e3406c8a0 0x7f4e3406ed50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c1114d0 tx=0x7f4e40009ee0 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:34.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.145+0000 7f4e3affd700 1 -- 172.21.15.32:0/592950362 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f4e3c057270 con 0x7f4e4c0fdeb0 2024-04-12T05:41:34.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.411+0000 7f4e50a51700 1 -- 172.21.15.32:0/592950362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4e4c109e20 con 0x7f4e4c0fdeb0 2024-04-12T05:41:34.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.411+0000 7f4e3affd700 1 -- 172.21.15.32:0/592950362 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1826 (secure 0 0 0) 0x7f4e3c004de0 con 0x7f4e4c0fdeb0 2024-04-12T05:41:34.409 INFO:teuthology.orchestra.run.smithi032.stdout:e22 2024-04-12T05:41:34.409 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:41:34.409 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:41:34.409 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:41:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:41:34.409 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:41:34.409 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:41:34.409 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 22 2024-04-12T05:41:34.409 INFO:teuthology.orchestra.run.smithi032.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-12T05:41:34.409 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:41:34.409 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:41:05.358509+0000 2024-04-12T05:41:34.409 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:41:34.409 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:41:34.409 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:41:34.410 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:41:34.410 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:41:34.410 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:41:34.410 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:41:34.410 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 46 2024-04-12T05:41:34.410 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:41:34.410 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:41:34.410 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:41:34.410 INFO:teuthology.orchestra.run.smithi032.stdout:up {0=14480,1=24281} 2024-04-12T05:41:34.410 INFO:teuthology.orchestra.run.smithi032.stdout:failed 2024-04-12T05:41:34.410 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:41:34.410 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:41:34.410 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:41:34.411 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:41:34.411 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:41:34.411 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:41:34.411 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:41:34.411 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{0:14480} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.32:6828/387754149,v1:172.21.15.32:6829/387754149] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:41:34.411 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{0:14490} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3678031004,v1:172.21.15.32:6827/3678031004] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:41:34.411 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{1:24281} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.67:6824/1291914599,v1:172.21.15.67:6825/1291914599] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:41:34.411 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{1:24295} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.67:6826/873395842,v1:172.21.15.67:6827/873395842] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:41:34.411 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:41:34.411 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:41:34.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.413+0000 7f4e50a51700 1 -- 172.21.15.32:0/592950362 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f4e3406c8a0 msgr2=0x7f4e3406ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:34.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.413+0000 7f4e50a51700 1 --2- 172.21.15.32:0/592950362 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f4e3406c8a0 0x7f4e3406ed50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c1114d0 tx=0x7f4e40009ee0 comp rx=0 tx=0).stop 2024-04-12T05:41:34.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.413+0000 7f4e50a51700 1 -- 172.21.15.32:0/592950362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e4c0fdeb0 msgr2=0x7f4e4c115bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:34.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.413+0000 7f4e50a51700 1 --2- 172.21.15.32:0/592950362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e4c0fdeb0 0x7f4e4c115bf0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c0043d0 tx=0x7f4e3c009e30 comp rx=0 tx=0).stop 2024-04-12T05:41:34.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.413+0000 7f4e50a51700 1 -- 172.21.15.32:0/592950362 shutdown_connections 2024-04-12T05:41:34.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.413+0000 7f4e50a51700 1 --2- 172.21.15.32:0/592950362 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f4e3406c8a0 0x7f4e3406ed50 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:34.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.413+0000 7f4e50a51700 1 --2- 172.21.15.32:0/592950362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e4c0fdeb0 0x7f4e4c115bf0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:34.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.413+0000 7f4e50a51700 1 --2- 172.21.15.32:0/592950362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e4c0fe850 0x7f4e4c116130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:34.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.413+0000 7f4e50a51700 1 -- 172.21.15.32:0/592950362 >> 172.21.15.32:0/592950362 conn(0x7f4e4c072bc0 msgr2=0x7f4e4c0fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:34.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.413+0000 7f4e50a51700 1 -- 172.21.15.32:0/592950362 shutdown_connections 2024-04-12T05:41:34.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.413+0000 7f4e50a51700 1 -- 172.21.15.32:0/592950362 wait complete. 2024-04-12T05:41:34.413 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 22 2024-04-12T05:41:34.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.519+0000 7fd32387f700 1 -- 172.21.15.32:0/3263251156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd31c0fde80 msgr2=0x7fd31c0fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:34.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.519+0000 7fd32387f700 1 --2- 172.21.15.32:0/3263251156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd31c0fde80 0x7fd31c0fe250 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd318004940 tx=0x7fd318009e30 comp rx=0 tx=0).stop 2024-04-12T05:41:34.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.519+0000 7fd32387f700 1 -- 172.21.15.32:0/3263251156 shutdown_connections 2024-04-12T05:41:34.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.519+0000 7fd32387f700 1 --2- 172.21.15.32:0/3263251156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd31c0fe820 0x7fd31c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:34.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.519+0000 7fd32387f700 1 --2- 172.21.15.32:0/3263251156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd31c0fde80 0x7fd31c0fe250 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:34.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.519+0000 7fd32387f700 1 -- 172.21.15.32:0/3263251156 >> 172.21.15.32:0/3263251156 conn(0x7fd31c0f7870 msgr2=0x7fd31c0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:34.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.519+0000 7fd32387f700 1 -- 172.21.15.32:0/3263251156 shutdown_connections 2024-04-12T05:41:34.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.519+0000 7fd32387f700 1 -- 172.21.15.32:0/3263251156 wait complete. 2024-04-12T05:41:34.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.520+0000 7fd32387f700 1 Processor -- start 2024-04-12T05:41:34.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.520+0000 7fd32387f700 1 -- start start 2024-04-12T05:41:34.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.520+0000 7fd32387f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd31c0fde80 0x7fd31c113830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:34.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.520+0000 7fd32387f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd31c0fe820 0x7fd31c113d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:34.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.520+0000 7fd32387f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c114450 con 0x7fd31c0fe820 2024-04-12T05:41:34.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.520+0000 7fd32387f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c10d9c0 con 0x7fd31c0fde80 2024-04-12T05:41:34.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.520+0000 7fd320e1a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd31c0fe820 0x7fd31c113d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:34.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.520+0000 7fd320e1a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd31c0fe820 0x7fd31c113d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52568/0 (socket says 172.21.15.32:52568) 2024-04-12T05:41:34.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.520+0000 7fd320e1a700 1 -- 172.21.15.32:0/1286477363 learned_addr learned my addr 172.21.15.32:0/1286477363 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:41:34.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.521+0000 7fd32161b700 1 --2- 172.21.15.32:0/1286477363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd31c0fde80 0x7fd31c113830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:34.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.521+0000 7fd320e1a700 1 -- 172.21.15.32:0/1286477363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd31c0fde80 msgr2=0x7fd31c113830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:34.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.521+0000 7fd320e1a700 1 --2- 172.21.15.32:0/1286477363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd31c0fde80 0x7fd31c113830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:34.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.521+0000 7fd320e1a700 1 -- 172.21.15.32:0/1286477363 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd318005040 con 0x7fd31c0fe820 2024-04-12T05:41:34.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.521+0000 7fd32161b700 1 --2- 172.21.15.32:0/1286477363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd31c0fde80 0x7fd31c113830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:41:34.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.521+0000 7fd320e1a700 1 --2- 172.21.15.32:0/1286477363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd31c0fe820 0x7fd31c113d70 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fd31000b790 tx=0x7fd310009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:34.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.521+0000 7fd30dffb700 1 -- 172.21.15.32:0/1286477363 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd31000f770 con 0x7fd31c0fe820 2024-04-12T05:41:34.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.521+0000 7fd30dffb700 1 -- 172.21.15.32:0/1286477363 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd31000fd80 con 0x7fd31c0fe820 2024-04-12T05:41:34.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.521+0000 7fd32387f700 1 -- 172.21.15.32:0/1286477363 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd31c10dca0 con 0x7fd31c0fe820 2024-04-12T05:41:34.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.521+0000 7fd30dffb700 1 -- 172.21.15.32:0/1286477363 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd31000d310 con 0x7fd31c0fe820 2024-04-12T05:41:34.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.521+0000 7fd32387f700 1 -- 172.21.15.32:0/1286477363 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd31c10e1e0 con 0x7fd31c0fe820 2024-04-12T05:41:34.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.522+0000 7fd32387f700 1 -- 172.21.15.32:0/1286477363 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd31c066af0 con 0x7fd31c0fe820 2024-04-12T05:41:34.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.523+0000 7fd30dffb700 1 -- 172.21.15.32:0/1286477363 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 34) v1 ==== 90703+0+0 (secure 0 0 0) 0x7fd310010420 con 0x7fd31c0fe820 2024-04-12T05:41:34.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.523+0000 7fd30dffb700 1 --2- 172.21.15.32:0/1286477363 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fd30806c8a0 0x7fd30806ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:34.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.523+0000 7fd30dffb700 1 -- 172.21.15.32:0/1286477363 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7fd31008bae0 con 0x7fd31c0fe820 2024-04-12T05:41:34.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.523+0000 7fd32161b700 1 --2- 172.21.15.32:0/1286477363 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fd30806c8a0 0x7fd30806ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:34.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.524+0000 7fd32161b700 1 --2- 172.21.15.32:0/1286477363 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fd30806c8a0 0x7fd30806ed50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd31800b3b0 tx=0x7fd318009e30 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:34.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.527+0000 7fd30dffb700 1 -- 172.21.15.32:0/1286477363 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fd310059a80 con 0x7fd31c0fe820 2024-04-12T05:41:34.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.763+0000 7fd32387f700 1 -- 172.21.15.32:0/1286477363 --> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd31c109df0 con 0x7fd30806c8a0 2024-04-12T05:41:34.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.764+0000 7fd30dffb700 1 -- 172.21.15.32:0/1286477363 <== mgr.24463 v2:172.21.15.32:6800/2564765508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7fd31c109df0 con 0x7fd30806c8a0 2024-04-12T05:41:34.762 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:41:34.762 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:41:34.762 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:41:34.762 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:41:34.763 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:41:34.763 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr" 2024-04-12T05:41:34.763 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:41:34.763 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "2/21 daemons upgraded", 2024-04-12T05:41:34.763 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mon daemons", 2024-04-12T05:41:34.763 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:41:34.763 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:41:34.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.766+0000 7fd32387f700 1 -- 172.21.15.32:0/1286477363 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fd30806c8a0 msgr2=0x7fd30806ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:34.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.766+0000 7fd32387f700 1 --2- 172.21.15.32:0/1286477363 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fd30806c8a0 0x7fd30806ed50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd31800b3b0 tx=0x7fd318009e30 comp rx=0 tx=0).stop 2024-04-12T05:41:34.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.766+0000 7fd32387f700 1 -- 172.21.15.32:0/1286477363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd31c0fe820 msgr2=0x7fd31c113d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:34.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.766+0000 7fd32387f700 1 --2- 172.21.15.32:0/1286477363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd31c0fe820 0x7fd31c113d70 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fd31000b790 tx=0x7fd310009250 comp rx=0 tx=0).stop 2024-04-12T05:41:34.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.766+0000 7fd32387f700 1 -- 172.21.15.32:0/1286477363 shutdown_connections 2024-04-12T05:41:34.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.766+0000 7fd32387f700 1 --2- 172.21.15.32:0/1286477363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd31c0fde80 0x7fd31c113830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:34.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.766+0000 7fd32387f700 1 --2- 172.21.15.32:0/1286477363 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7fd30806c8a0 0x7fd30806ed50 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:34.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.766+0000 7fd32387f700 1 --2- 172.21.15.32:0/1286477363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd31c0fe820 0x7fd31c113d70 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:34.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.766+0000 7fd32387f700 1 -- 172.21.15.32:0/1286477363 >> 172.21.15.32:0/1286477363 conn(0x7fd31c0f7870 msgr2=0x7fd31c0f8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:34.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.766+0000 7fd32387f700 1 -- 172.21.15.32:0/1286477363 shutdown_connections 2024-04-12T05:41:34.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.766+0000 7fd32387f700 1 -- 172.21.15.32:0/1286477363 wait complete. 2024-04-12T05:41:34.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.871+0000 7f0e38014700 1 -- 172.21.15.32:0/1058210443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e30101d70 msgr2=0x7f0e30102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:34.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.871+0000 7f0e38014700 1 --2- 172.21.15.32:0/1058210443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e30101d70 0x7f0e30102140 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c004940 tx=0x7f0e2c009e30 comp rx=0 tx=0).stop 2024-04-12T05:41:34.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.871+0000 7f0e38014700 1 -- 172.21.15.32:0/1058210443 shutdown_connections 2024-04-12T05:41:34.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.871+0000 7f0e38014700 1 --2- 172.21.15.32:0/1058210443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e30102710 0x7f0e300fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:34.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.871+0000 7f0e38014700 1 --2- 172.21.15.32:0/1058210443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e30101d70 0x7f0e30102140 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:34.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.871+0000 7f0e38014700 1 -- 172.21.15.32:0/1058210443 >> 172.21.15.32:0/1058210443 conn(0x7f0e30072bc0 msgr2=0x7f0e30072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:34.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.871+0000 7f0e38014700 1 -- 172.21.15.32:0/1058210443 shutdown_connections 2024-04-12T05:41:34.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.871+0000 7f0e38014700 1 -- 172.21.15.32:0/1058210443 wait complete. 2024-04-12T05:41:34.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.872+0000 7f0e38014700 1 Processor -- start 2024-04-12T05:41:34.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.872+0000 7f0e38014700 1 -- start start 2024-04-12T05:41:34.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.872+0000 7f0e38014700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e30101d70 0x7f0e30102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:34.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.872+0000 7f0e38014700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e30102710 0x7f0e30103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:34.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.872+0000 7f0e38014700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e30103750 con 0x7f0e30102710 2024-04-12T05:41:34.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.872+0000 7f0e38014700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e301038c0 con 0x7f0e30101d70 2024-04-12T05:41:34.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.872+0000 7f0e355af700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e30102710 0x7f0e30103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:34.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.872+0000 7f0e355af700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e30102710 0x7f0e30103110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52576/0 (socket says 172.21.15.32:52576) 2024-04-12T05:41:34.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.872+0000 7f0e355af700 1 -- 172.21.15.32:0/2099290950 learned_addr learned my addr 172.21.15.32:0/2099290950 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:41:34.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.872+0000 7f0e35db0700 1 --2- 172.21.15.32:0/2099290950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e30101d70 0x7f0e30102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:34.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.872+0000 7f0e355af700 1 -- 172.21.15.32:0/2099290950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e30101d70 msgr2=0x7f0e30102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:34.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.872+0000 7f0e355af700 1 --2- 172.21.15.32:0/2099290950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e30101d70 0x7f0e30102bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:34.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.872+0000 7f0e355af700 1 -- 172.21.15.32:0/2099290950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e2c005040 con 0x7f0e30102710 2024-04-12T05:41:34.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.872+0000 7f0e355af700 1 --2- 172.21.15.32:0/2099290950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e30102710 0x7f0e30103110 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0e2000b790 tx=0x7f0e20009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:34.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.873+0000 7f0e267fc700 1 -- 172.21.15.32:0/2099290950 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e2000f6f0 con 0x7f0e30102710 2024-04-12T05:41:34.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.873+0000 7f0e267fc700 1 -- 172.21.15.32:0/2099290950 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0e2000fd00 con 0x7f0e30102710 2024-04-12T05:41:34.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.873+0000 7f0e38014700 1 -- 172.21.15.32:0/2099290950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e30119e00 con 0x7f0e30102710 2024-04-12T05:41:34.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.873+0000 7f0e267fc700 1 -- 172.21.15.32:0/2099290950 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e200103a0 con 0x7f0e30102710 2024-04-12T05:41:34.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.873+0000 7f0e38014700 1 -- 172.21.15.32:0/2099290950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e3011a310 con 0x7f0e30102710 2024-04-12T05:41:34.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.874+0000 7f0e38014700 1 -- 172.21.15.32:0/2099290950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e300fd4c0 con 0x7f0e30102710 2024-04-12T05:41:34.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.874+0000 7f0e267fc700 1 -- 172.21.15.32:0/2099290950 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 34) v1 ==== 90703+0+0 (secure 0 0 0) 0x7f0e2000f850 con 0x7f0e30102710 2024-04-12T05:41:34.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.875+0000 7f0e267fc700 1 --2- 172.21.15.32:0/2099290950 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f0e1c06c970 0x7f0e1c06ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:41:34.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.875+0000 7f0e267fc700 1 -- 172.21.15.32:0/2099290950 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f0e2008b880 con 0x7f0e30102710 2024-04-12T05:41:34.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.875+0000 7f0e35db0700 1 --2- 172.21.15.32:0/2099290950 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f0e1c06c970 0x7f0e1c06ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:34.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.875+0000 7f0e35db0700 1 --2- 172.21.15.32:0/2099290950 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f0e1c06c970 0x7f0e1c06ee20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c004f90 tx=0x7f0e2c009e30 comp rx=0 tx=0).ready entity=mgr.24463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:34.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:34.879+0000 7f0e267fc700 1 -- 172.21.15.32:0/2099290950 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f0e20055d70 con 0x7f0e30102710 2024-04-12T05:41:35.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:34 smithi067 ceph-mon[49010]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:41:35.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:34 smithi067 ceph-mon[49010]: pgmap v86: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:41:35.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:34 smithi067 ceph-mon[49010]: from='client.14830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:41:35.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:34 smithi067 ceph-mon[49010]: from='client.14834 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:41:35.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:34 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/952839881' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:41:35.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:34 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:35.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:34 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T05:41:35.020 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:34 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T05:41:35.020 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:34 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:41:35.020 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:34 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/592950362' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:41:35.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:34 smithi032 ceph-mon[39731]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:41:35.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:34 smithi032 ceph-mon[39731]: pgmap v86: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:41:35.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:34 smithi032 ceph-mon[39731]: from='client.14830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:41:35.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:34 smithi032 ceph-mon[39731]: from='client.14834 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:41:35.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:34 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/952839881' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:41:35.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:34 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:35.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:34 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T05:41:35.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:34 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T05:41:35.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:34 smithi032 ceph-mon[39731]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:41:35.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:34 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/592950362' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:41:35.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:35.165+0000 7f0e38014700 1 -- 172.21.15.32:0/2099290950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0e3005f5f0 con 0x7f0e30102710 2024-04-12T05:41:35.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:35.166+0000 7f0e267fc700 1 -- 172.21.15.32:0/2099290950 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f0e20059390 con 0x7f0e30102710 2024-04-12T05:41:35.163 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_OK 2024-04-12T05:41:35.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:35.168+0000 7f0e38014700 1 -- 172.21.15.32:0/2099290950 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f0e1c06c970 msgr2=0x7f0e1c06ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:35.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:35.168+0000 7f0e38014700 1 --2- 172.21.15.32:0/2099290950 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f0e1c06c970 0x7f0e1c06ee20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c004f90 tx=0x7f0e2c009e30 comp rx=0 tx=0).stop 2024-04-12T05:41:35.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:35.168+0000 7f0e38014700 1 -- 172.21.15.32:0/2099290950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e30102710 msgr2=0x7f0e30103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:35.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:35.168+0000 7f0e38014700 1 --2- 172.21.15.32:0/2099290950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e30102710 0x7f0e30103110 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0e2000b790 tx=0x7f0e20009220 comp rx=0 tx=0).stop 2024-04-12T05:41:35.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:35.169+0000 7f0e38014700 1 -- 172.21.15.32:0/2099290950 shutdown_connections 2024-04-12T05:41:35.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:35.169+0000 7f0e38014700 1 --2- 172.21.15.32:0/2099290950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e30101d70 0x7f0e30102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:35.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:35.169+0000 7f0e38014700 1 --2- 172.21.15.32:0/2099290950 >> [v2:172.21.15.32:6800/2564765508,v1:172.21.15.32:6801/2564765508] conn(0x7f0e1c06c970 0x7f0e1c06ee20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:35.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:35.169+0000 7f0e38014700 1 --2- 172.21.15.32:0/2099290950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e30102710 0x7f0e30103110 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:41:35.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:35.169+0000 7f0e38014700 1 -- 172.21.15.32:0/2099290950 >> 172.21.15.32:0/2099290950 conn(0x7f0e30072bc0 msgr2=0x7f0e300658e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:35.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:35.169+0000 7f0e38014700 1 -- 172.21.15.32:0/2099290950 shutdown_connections 2024-04-12T05:41:35.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:41:35.169+0000 7f0e38014700 1 -- 172.21.15.32:0/2099290950 wait complete. 2024-04-12T05:41:35.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:35 smithi032 ceph-mon[39731]: Upgrade: Updating mon.smithi032 2024-04-12T05:41:35.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:35 smithi032 ceph-mon[39731]: Deploying daemon mon.smithi032 on smithi032 2024-04-12T05:41:35.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:35 smithi032 ceph-mon[39731]: from='client.14846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:41:35.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:35 smithi032 ceph-mon[39731]: from='client.? 172.21.15.32:0/2099290950' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:41:35.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:35 smithi032 systemd[1]: Stopping Ceph mon.smithi032 for 37b58ce2-f88d-11ee-bc8c-c7b262605968... 2024-04-12T05:41:36.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:35 smithi067 ceph-mon[49010]: Upgrade: Updating mon.smithi032 2024-04-12T05:41:36.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:35 smithi067 ceph-mon[49010]: Deploying daemon mon.smithi032 on smithi032 2024-04-12T05:41:36.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:35 smithi067 ceph-mon[49010]: from='client.14846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:41:36.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:35 smithi067 ceph-mon[49010]: from='client.? 172.21.15.32:0/2099290950' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:41:36.322 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:36 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-mon-smithi032[39708]: 2024-04-12T05:41:36.022+0000 7f36da648700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi032 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T05:41:36.323 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:36 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-mon-smithi032[39708]: 2024-04-12T05:41:36.022+0000 7f36da648700 -1 mon.smithi032@0(leader) e2 *** Got Signal Terminated *** 2024-04-12T05:41:36.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:36 smithi032 bash[146612]: ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-mon-smithi032 2024-04-12T05:41:37.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:36 smithi032 systemd[1]: ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@mon.smithi032.service: Succeeded. 2024-04-12T05:41:37.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:36 smithi032 systemd[1]: Stopped Ceph mon.smithi032 for 37b58ce2-f88d-11ee-bc8c-c7b262605968. 2024-04-12T05:41:37.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:36 smithi032 systemd[1]: Starting Ceph mon.smithi032 for 37b58ce2-f88d-11ee-bc8c-c7b262605968... 2024-04-12T05:41:37.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: set uid:gid to 167:167 (ceph:ceph) 2024-04-12T05:41:37.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable), process ceph-mon, pid 7 2024-04-12T05:41:37.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: pidfile_write: ignore empty --pid-file 2024-04-12T05:41:37.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: load: jerasure load: lrc 2024-04-12T05:41:37.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: RocksDB version: 7.9.2 2024-04-12T05:41:37.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Git sha 0 2024-04-12T05:41:37.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Compile date 2024-04-08 23:21:21 2024-04-12T05:41:37.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: DB SUMMARY 2024-04-12T05:41:37.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: DB Session ID: VYRY8YP313E4IM8N6EEW 2024-04-12T05:41:37.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: CURRENT file: CURRENT 2024-04-12T05:41:37.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: IDENTITY file: IDENTITY 2024-04-12T05:41:37.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1158 Bytes 2024-04-12T05:41:37.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi032/store.db dir, Total Num: 1, files: 000033.sst 2024-04-12T05:41:37.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi032/store.db: 000031.log size: 2342580 ; 2024-04-12T05:41:37.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.error_if_exists: 0 2024-04-12T05:41:37.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.create_if_missing: 0 2024-04-12T05:41:37.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.paranoid_checks: 1 2024-04-12T05:41:37.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-12T05:41:37.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-12T05:41:37.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-12T05:41:37.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.env: 0x55c310f127a0 2024-04-12T05:41:37.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.fs: PosixFileSystem 2024-04-12T05:41:37.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.info_log: 0x55c3130233a0 2024-04-12T05:41:37.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.max_file_opening_threads: 16 2024-04-12T05:41:37.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.statistics: (nil) 2024-04-12T05:41:37.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.use_fsync: 0 2024-04-12T05:41:37.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.max_log_file_size: 0 2024-04-12T05:41:37.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-12T05:41:37.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-12T05:41:37.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.keep_log_file_num: 1000 2024-04-12T05:41:37.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.recycle_log_file_num: 0 2024-04-12T05:41:37.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.allow_fallocate: 1 2024-04-12T05:41:37.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.allow_mmap_reads: 0 2024-04-12T05:41:37.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.allow_mmap_writes: 0 2024-04-12T05:41:37.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.use_direct_reads: 0 2024-04-12T05:41:37.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-12T05:41:37.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.create_missing_column_families: 0 2024-04-12T05:41:37.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.db_log_dir: 2024-04-12T05:41:37.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.wal_dir: 2024-04-12T05:41:37.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-12T05:41:37.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-12T05:41:37.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-12T05:41:37.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-12T05:41:37.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-12T05:41:37.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-12T05:41:37.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.advise_random_on_open: 1 2024-04-12T05:41:37.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.db_write_buffer_size: 0 2024-04-12T05:41:37.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.write_buffer_manager: 0x55c312298140 2024-04-12T05:41:37.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-12T05:41:37.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-12T05:41:37.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-12T05:41:37.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.rate_limiter: (nil) 2024-04-12T05:41:37.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-12T05:41:37.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.wal_recovery_mode: 2 2024-04-12T05:41:37.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.enable_thread_tracking: 0 2024-04-12T05:41:37.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.enable_pipelined_write: 0 2024-04-12T05:41:37.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.unordered_write: 0 2024-04-12T05:41:37.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-12T05:41:37.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-12T05:41:37.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-12T05:41:37.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-12T05:41:37.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.row_cache: None 2024-04-12T05:41:37.845 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.wal_filter: None 2024-04-12T05:41:37.845 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-12T05:41:37.845 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.allow_ingest_behind: 0 2024-04-12T05:41:37.845 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.two_write_queues: 0 2024-04-12T05:41:37.845 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.manual_wal_flush: 0 2024-04-12T05:41:37.845 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.wal_compression: 0 2024-04-12T05:41:37.845 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.atomic_flush: 0 2024-04-12T05:41:37.845 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-12T05:41:37.845 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-12T05:41:37.845 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-12T05:41:37.845 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.log_readahead_size: 0 2024-04-12T05:41:37.845 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-12T05:41:37.845 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.best_efforts_recovery: 0 2024-04-12T05:41:37.846 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-12T05:41:37.846 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-12T05:41:37.846 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.allow_data_in_errors: 0 2024-04-12T05:41:37.846 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.db_host_id: __hostname__ 2024-04-12T05:41:37.846 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-12T05:41:37.846 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.max_background_jobs: 2 2024-04-12T05:41:37.846 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.max_background_compactions: -1 2024-04-12T05:41:37.846 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.max_subcompactions: 1 2024-04-12T05:41:37.846 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-12T05:41:37.846 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-12T05:41:37.846 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-12T05:41:37.847 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.max_total_wal_size: 0 2024-04-12T05:41:37.847 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-12T05:41:37.847 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-12T05:41:37.847 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-12T05:41:37.847 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-12T05:41:37.847 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.max_open_files: -1 2024-04-12T05:41:37.847 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.bytes_per_sync: 0 2024-04-12T05:41:37.847 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-12T05:41:37.847 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-12T05:41:37.847 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.compaction_readahead_size: 0 2024-04-12T05:41:37.847 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.max_background_flushes: -1 2024-04-12T05:41:37.847 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Compression algorithms supported: 2024-04-12T05:41:37.847 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-12T05:41:37.847 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: kZSTD supported: 0 2024-04-12T05:41:37.848 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: kXpressCompression supported: 0 2024-04-12T05:41:37.848 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: kLZ4HCCompression supported: 1 2024-04-12T05:41:37.848 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: kZlibCompression supported: 1 2024-04-12T05:41:37.848 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: kSnappyCompression supported: 1 2024-04-12T05:41:37.848 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: kLZ4Compression supported: 1 2024-04-12T05:41:37.848 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: kBZip2Compression supported: 0 2024-04-12T05:41:37.848 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-12T05:41:37.848 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-12T05:41:37.848 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi032/store.db/MANIFEST-000015 2024-04-12T05:41:37.848 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-12T05:41:37.848 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-12T05:41:37.848 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.merge_operator: 2024-04-12T05:41:37.848 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.compaction_filter: None 2024-04-12T05:41:37.848 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.compaction_filter_factory: None 2024-04-12T05:41:37.849 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.sst_partitioner_factory: None 2024-04-12T05:41:37.849 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-12T05:41:37.849 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-12T05:41:37.849 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c3130234c0) 2024-04-12T05:41:37.849 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: cache_index_and_filter_blocks: 1 2024-04-12T05:41:37.849 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-12T05:41:37.849 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-12T05:41:37.850 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: pin_top_level_index_and_filter: 1 2024-04-12T05:41:37.850 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: index_type: 0 2024-04-12T05:41:37.850 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: data_block_index_type: 0 2024-04-12T05:41:37.850 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: index_shortening: 1 2024-04-12T05:41:37.850 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-12T05:41:37.850 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: checksum: 4 2024-04-12T05:41:37.850 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: no_block_cache: 0 2024-04-12T05:41:37.850 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_cache: 0x55c3123391f0 2024-04-12T05:41:37.850 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_cache_name: BinnedLRUCache 2024-04-12T05:41:37.850 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_cache_options: 2024-04-12T05:41:37.850 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: capacity : 536870912 2024-04-12T05:41:37.850 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: num_shard_bits : 4 2024-04-12T05:41:37.850 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: strict_capacity_limit : 0 2024-04-12T05:41:37.851 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: high_pri_pool_ratio: 0.000 2024-04-12T05:41:37.851 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_cache_compressed: (nil) 2024-04-12T05:41:37.851 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: persistent_cache: (nil) 2024-04-12T05:41:37.851 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_size: 4096 2024-04-12T05:41:37.851 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_size_deviation: 10 2024-04-12T05:41:37.851 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_restart_interval: 16 2024-04-12T05:41:37.851 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: index_block_restart_interval: 1 2024-04-12T05:41:37.851 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: metadata_block_size: 4096 2024-04-12T05:41:37.851 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: partition_filters: 0 2024-04-12T05:41:37.851 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: use_delta_encoding: 1 2024-04-12T05:41:37.851 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: filter_policy: bloomfilter 2024-04-12T05:41:37.851 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: whole_key_filtering: 1 2024-04-12T05:41:37.851 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: verify_compression: 0 2024-04-12T05:41:37.851 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: read_amp_bytes_per_bit: 0 2024-04-12T05:41:37.852 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: format_version: 5 2024-04-12T05:41:37.852 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: enable_index_compression: 1 2024-04-12T05:41:37.852 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: block_align: 0 2024-04-12T05:41:37.852 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: max_auto_readahead_size: 262144 2024-04-12T05:41:37.852 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: prepopulate_block_cache: 0 2024-04-12T05:41:37.852 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: initial_auto_readahead_size: 8192 2024-04-12T05:41:37.852 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: num_file_reads_for_auto_readahead: 2 2024-04-12T05:41:37.852 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.write_buffer_size: 33554432 2024-04-12T05:41:37.852 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.max_write_buffer_number: 2 2024-04-12T05:41:37.852 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.compression: NoCompression 2024-04-12T05:41:37.852 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.bottommost_compression: Disabled 2024-04-12T05:41:37.853 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.prefix_extractor: nullptr 2024-04-12T05:41:37.853 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-12T05:41:37.853 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.num_levels: 7 2024-04-12T05:41:37.853 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-12T05:41:37.853 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-12T05:41:37.853 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-12T05:41:37.853 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-12T05:41:37.853 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-12T05:41:37.853 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-12T05:41:37.853 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-12T05:41:37.853 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-12T05:41:37.853 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-12T05:41:37.853 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-12T05:41:37.853 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-12T05:41:37.853 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-12T05:41:37.854 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-12T05:41:37.854 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.compression_opts.level: 32767 2024-04-12T05:41:37.854 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.compression_opts.strategy: 0 2024-04-12T05:41:37.854 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-12T05:41:37.854 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-12T05:41:37.854 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-12T05:41:37.854 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-12T05:41:37.854 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.compression_opts.enabled: false 2024-04-12T05:41:37.854 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-12T05:41:37.854 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-12T05:41:37.854 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-12T05:41:37.854 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-12T05:41:37.854 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.target_file_size_base: 67108864 2024-04-12T05:41:37.855 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-12T05:41:37.855 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-12T05:41:37.855 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-12T05:41:37.855 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-12T05:41:37.855 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-12T05:41:37.855 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-12T05:41:37.855 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-12T05:41:37.855 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-12T05:41:37.855 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-12T05:41:37.855 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-12T05:41:37.855 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-12T05:41:37.855 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-12T05:41:37.855 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-12T05:41:37.855 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-12T05:41:37.855 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.arena_block_size: 1048576 2024-04-12T05:41:37.855 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-12T05:41:37.856 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-12T05:41:37.856 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.disable_auto_compactions: 0 2024-04-12T05:41:37.856 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-12T05:41:37.857 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-12T05:41:37.857 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-12T05:41:37.857 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-12T05:41:37.857 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-12T05:41:37.857 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-12T05:41:37.857 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-12T05:41:37.857 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-12T05:41:37.857 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-12T05:41:37.857 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-12T05:41:37.857 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.table_properties_collectors: 2024-04-12T05:41:37.857 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.inplace_update_support: 0 2024-04-12T05:41:37.858 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-12T05:41:37.858 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-12T05:41:37.858 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-12T05:41:37.858 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-12T05:41:37.858 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.bloom_locality: 0 2024-04-12T05:41:37.858 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.max_successive_merges: 0 2024-04-12T05:41:37.859 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-12T05:41:37.859 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.paranoid_file_checks: 0 2024-04-12T05:41:37.859 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.force_consistency_checks: 1 2024-04-12T05:41:37.859 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.report_bg_io_stats: 0 2024-04-12T05:41:37.859 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.ttl: 2592000 2024-04-12T05:41:37.859 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-12T05:41:37.859 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-12T05:41:37.859 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-12T05:41:37.859 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.enable_blob_files: false 2024-04-12T05:41:37.859 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.min_blob_size: 0 2024-04-12T05:41:37.859 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.blob_file_size: 268435456 2024-04-12T05:41:37.859 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-12T05:41:37.859 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-12T05:41:37.860 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-12T05:41:37.860 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-12T05:41:37.860 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-12T05:41:37.860 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.blob_file_starting_level: 0 2024-04-12T05:41:37.860 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-12T05:41:37.860 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 33.sst 2024-04-12T05:41:37.860 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-12T05:41:37.860 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi032/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 35, last_sequence is 11961, log_number is 31,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-12T05:41:37.860 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 31 2024-04-12T05:41:37.860 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 379414ef-b39f-4476-bfa9-d577bdbe733d 2024-04-12T05:41:37.861 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712900497578853, "job": 1, "event": "recovery_started", "wal_files": [31]} 2024-04-12T05:41:37.861 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #31 mode 2 2024-04-12T05:41:37.861 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712900497586553, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 36, "file_size": 2055093, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11962, "largest_seqno": 13751, "table_properties": {"data_size": 2046951, "index_size": 5078, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2053, "raw_key_size": 18792, "raw_average_key_size": 23, "raw_value_size": 2030045, "raw_average_value_size": 2543, "num_data_blocks": 233, "num_entries": 798, "num_filter_entries": 798, "num_deletions": 7, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712900497, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "379414ef-b39f-4476-bfa9-d577bdbe733d", "db_session_id": "VYRY8YP313E4IM8N6EEW", "orig_file_number": 36, "seqno_to_time_mapping": "N/A"}} 2024-04-12T05:41:37.861 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712900497586652, "job": 1, "event": "recovery_finished"} 2024-04-12T05:41:37.861 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: [db/version_set.cc:5047] Creating manifest 38 2024-04-12T05:41:37.861 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-12T05:41:37.861 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi032/store.db/000031.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-12T05:41:37.861 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c3123d8000 2024-04-12T05:41:37.861 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: DB pointer 0x55c3123c6000 2024-04-12T05:41:37.861 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-12T05:41:37.861 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-12T05:41:37.861 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** DB Stats ** 2024-04-12T05:41:37.861 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T05:41:37.861 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-12T05:41:37.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-12T05:41:37.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-12T05:41:37.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-12T05:41:37.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-12T05:41:37.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-12T05:41:37.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-12T05:41:37.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** Compaction Stats [default] ** 2024-04-12T05:41:37.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-12T05:41:37.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-12T05:41:37.862 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: L0 1/0 1.96 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 522.9 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-04-12T05:41:37.863 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: L6 1/0 6.75 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-12T05:41:37.863 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Sum 2/0 8.71 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 522.9 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-04-12T05:41:37.863 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 522.9 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-04-12T05:41:37.863 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-12T05:41:37.863 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** Compaction Stats [default] ** 2024-04-12T05:41:37.863 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-12T05:41:37.863 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T05:41:37.863 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 522.9 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-04-12T05:41:37.863 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-12T05:41:37.863 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-12T05:41:37.863 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-12T05:41:37.863 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T05:41:37.863 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-04-12T05:41:37.864 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-12T05:41:37.864 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-12T05:41:37.864 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-12T05:41:37.864 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-12T05:41:37.864 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Cumulative compaction: 0.00 GB write, 190.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T05:41:37.864 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Interval compaction: 0.00 GB write, 190.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T05:41:37.864 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-12T05:41:37.864 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Block cache BinnedLRUCache@0x55c3123391f0#7 capacity: 512.00 MB usage: 7.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-04-12T05:41:37.864 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.28 KB,0.000435114%) IndexBlock(1,5.08 KB,0.000968575%) Misc(1,0.00 KB,0%) 2024-04-12T05:41:37.864 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-12T05:41:37.864 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-12T05:41:37.865 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: starting mon.smithi032 rank 0 at public addrs [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] at bind addrs [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi032 fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 2024-04-12T05:41:37.865 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: mon.smithi032@-1(???) e2 preinit fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 2024-04-12T05:41:37.865 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: mon.smithi032@-1(???).mds e22 new map 2024-04-12T05:41:37.865 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: mon.smithi032@-1(???).mds e22 print_map 2024-04-12T05:41:37.865 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: e22 2024-04-12T05:41:37.865 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:41:37.865 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:41:37.865 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: legacy client fscid: 1 2024-04-12T05:41:37.865 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-12T05:41:37.865 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: Filesystem 'cephfs' (1) 2024-04-12T05:41:37.865 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: fs_name cephfs 2024-04-12T05:41:37.865 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: epoch 22 2024-04-12T05:41:37.866 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-12T05:41:37.866 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:41:37.866 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: modified 2024-04-12T05:41:05.358509+0000 2024-04-12T05:41:37.866 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: tableserver 0 2024-04-12T05:41:37.866 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: root 0 2024-04-12T05:41:37.866 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: session_timeout 60 2024-04-12T05:41:37.866 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: session_autoclose 300 2024-04-12T05:41:37.866 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: max_file_size 1099511627776 2024-04-12T05:41:37.866 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: max_xattr_size 65536 2024-04-12T05:41:37.866 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: required_client_features {} 2024-04-12T05:41:37.866 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: last_failure 0 2024-04-12T05:41:37.866 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: last_failure_osd_epoch 46 2024-04-12T05:41:37.866 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:41:37.867 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: max_mds 2 2024-04-12T05:41:37.867 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: in 0,1 2024-04-12T05:41:37.867 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: up {0=14480,1=24281} 2024-04-12T05:41:37.867 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: failed 2024-04-12T05:41:37.867 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: damaged 2024-04-12T05:41:37.867 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: stopped 2024-04-12T05:41:37.867 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: data_pools [3] 2024-04-12T05:41:37.867 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: metadata_pool 2 2024-04-12T05:41:37.867 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: inline_data disabled 2024-04-12T05:41:37.867 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: balancer 2024-04-12T05:41:37.867 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: bal_rank_mask -1 2024-04-12T05:41:37.868 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: standby_count_wanted 1 2024-04-12T05:41:37.868 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: [mds.cephfs.smithi032.pndtry{0:14480} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.32:6828/387754149,v1:172.21.15.32:6829/387754149] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:41:37.868 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: [mds.cephfs.smithi032.jehaaj{0:14490} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3678031004,v1:172.21.15.32:6827/3678031004] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:41:37.868 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: [mds.cephfs.smithi067.zesmgy{1:24281} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.67:6824/1291914599,v1:172.21.15.67:6825/1291914599] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:41:37.868 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: [mds.cephfs.smithi067.ymvqih{1:24295} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.67:6826/873395842,v1:172.21.15.67:6827/873395842] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:41:37.868 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-12T05:41:37.868 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout: 2024-04-12T05:41:37.868 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: mon.smithi032@-1(???).osd e48 crush map has features 3314933000852226048, adjusting msgr requires 2024-04-12T05:41:37.868 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: mon.smithi032@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-12T05:41:37.868 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: mon.smithi032@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-12T05:41:37.868 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: mon.smithi032@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-12T05:41:37.869 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 ceph-mon[146817]: mon.smithi032@-1(???).paxosservice(auth 1..21) refresh upgraded, format 0 -> 3 2024-04-12T05:41:37.869 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 bash[146779]: ad789df6d5c746654be5f7b05aa901f909d372e299ef5dbe9cc7730ae4c591ae 2024-04-12T05:41:37.869 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:37 smithi032 systemd[1]: Started Ceph mon.smithi032 for 37b58ce2-f88d-11ee-bc8c-c7b262605968. 2024-04-12T05:41:41.008 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:40 smithi032 ceph-mon[146817]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-12T05:41:41.008 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:40 smithi032 ceph-mon[146817]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-12T05:41:41.009 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:40 smithi032 ceph-mon[146817]: pgmap v88: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:41:41.009 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:40 smithi032 ceph-mon[146817]: pgmap v89: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:41:41.009 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:40 smithi032 ceph-mon[146817]: mon.smithi032 calling monitor election 2024-04-12T05:41:41.009 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:40 smithi032 ceph-mon[146817]: mon.smithi032 is new leader, mons smithi032,smithi067 in quorum (ranks 0,1) 2024-04-12T05:41:41.009 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:40 smithi032 ceph-mon[146817]: monmap e2: 2 mons at {smithi032=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0],smithi067=[v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-12T05:41:41.009 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:40 smithi032 ceph-mon[146817]: fsmap cephfs:2 {0=cephfs.smithi032.pndtry=up:active,1=cephfs.smithi067.zesmgy=up:active} 2 up:standby-replay 2024-04-12T05:41:41.009 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:40 smithi032 ceph-mon[146817]: osdmap e48: 6 total, 6 up, 6 in 2024-04-12T05:41:41.009 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:40 smithi032 ceph-mon[146817]: mgrmap e34: smithi032.hrwxrh(active, since 2m), standbys: smithi067.dwichg 2024-04-12T05:41:41.009 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:40 smithi032 ceph-mon[146817]: overall HEALTH_OK 2024-04-12T05:41:41.009 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:40 smithi032 ceph-mon[146817]: mgrmap e35: smithi032.hrwxrh(active, since 2m), standbys: smithi067.dwichg 2024-04-12T05:41:41.009 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:40 smithi032 ceph-mon[146817]: from='mgr.24463 ' entity='' 2024-04-12T05:41:41.010 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-12T05:41:41.010 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:40 smithi067 ceph-mon[49010]: from='mgr.24463 172.21.15.32:0/3457511384' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-12T05:41:41.010 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:40 smithi067 ceph-mon[49010]: pgmap v88: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:41:41.010 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:40 smithi067 ceph-mon[49010]: pgmap v89: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:41:41.010 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:40 smithi067 ceph-mon[49010]: mon.smithi032 calling monitor election 2024-04-12T05:41:41.010 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:40 smithi067 ceph-mon[49010]: mon.smithi032 is new leader, mons smithi032,smithi067 in quorum (ranks 0,1) 2024-04-12T05:41:41.010 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:40 smithi067 ceph-mon[49010]: monmap e2: 2 mons at {smithi032=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0],smithi067=[v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-12T05:41:41.010 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:40 smithi067 ceph-mon[49010]: fsmap cephfs:2 {0=cephfs.smithi032.pndtry=up:active,1=cephfs.smithi067.zesmgy=up:active} 2 up:standby-replay 2024-04-12T05:41:41.010 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:40 smithi067 ceph-mon[49010]: osdmap e48: 6 total, 6 up, 6 in 2024-04-12T05:41:41.010 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:40 smithi067 ceph-mon[49010]: mgrmap e34: smithi032.hrwxrh(active, since 2m), standbys: smithi067.dwichg 2024-04-12T05:41:41.011 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:40 smithi067 ceph-mon[49010]: overall HEALTH_OK 2024-04-12T05:41:41.011 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:40 smithi067 ceph-mon[49010]: mgrmap e35: smithi032.hrwxrh(active, since 2m), standbys: smithi067.dwichg 2024-04-12T05:41:41.011 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:40 smithi067 ceph-mon[49010]: from='mgr.24463 ' entity='' 2024-04-12T05:41:47.895 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:47 smithi032 ceph-mon[146817]: Active manager daemon smithi032.hrwxrh restarted 2024-04-12T05:41:47.895 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:47 smithi032 ceph-mon[146817]: Activating manager daemon smithi032.hrwxrh 2024-04-12T05:41:47.895 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:47 smithi032 ceph-mon[146817]: osdmap e49: 6 total, 6 up, 6 in 2024-04-12T05:41:47.895 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:47 smithi032 ceph-mon[146817]: mgrmap e36: smithi032.hrwxrh(active, starting, since 0.00560134s), standbys: smithi067.dwichg 2024-04-12T05:41:47.895 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:47 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-12T05:41:47.895 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:47 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-12T05:41:47.895 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:47 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.pndtry"}]: dispatch 2024-04-12T05:41:47.896 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:47 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.jehaaj"}]: dispatch 2024-04-12T05:41:47.896 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:47 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.zesmgy"}]: dispatch 2024-04-12T05:41:47.896 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:47 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.ymvqih"}]: dispatch 2024-04-12T05:41:47.896 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:47 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr metadata", "who": "smithi032.hrwxrh", "id": "smithi032.hrwxrh"}]: dispatch 2024-04-12T05:41:47.896 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:47 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr metadata", "who": "smithi067.dwichg", "id": "smithi067.dwichg"}]: dispatch 2024-04-12T05:41:47.896 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:47 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:41:47.896 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:47 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:41:47.896 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:47 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:41:47.896 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:47 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:41:47.896 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:47 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:41:47.897 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:47 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:41:47.897 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:47 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T05:41:47.897 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:47 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T05:41:47.897 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:47 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T05:41:47.897 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:47 smithi032 ceph-mon[146817]: Standby manager daemon smithi067.dwichg restarted 2024-04-12T05:41:47.897 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:47 smithi032 ceph-mon[146817]: Standby manager daemon smithi067.dwichg started 2024-04-12T05:41:47.897 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:47 smithi032 ceph-mon[146817]: from='mgr.? 172.21.15.67:0/3922970479' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi067.dwichg/crt"}]: dispatch 2024-04-12T05:41:47.897 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:47 smithi032 ceph-mon[146817]: from='mgr.? 172.21.15.67:0/3922970479' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T05:41:47.897 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:47 smithi032 ceph-mon[146817]: from='mgr.? 172.21.15.67:0/3922970479' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi067.dwichg/key"}]: dispatch 2024-04-12T05:41:47.897 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:47 smithi032 ceph-mon[146817]: from='mgr.? 172.21.15.67:0/3922970479' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T05:41:47.897 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:47 smithi032 ceph-mon[146817]: Manager daemon smithi032.hrwxrh is now available 2024-04-12T05:41:47.898 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:47 smithi067 ceph-mon[49010]: Active manager daemon smithi032.hrwxrh restarted 2024-04-12T05:41:47.898 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:47 smithi067 ceph-mon[49010]: Activating manager daemon smithi032.hrwxrh 2024-04-12T05:41:47.898 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:47 smithi067 ceph-mon[49010]: osdmap e49: 6 total, 6 up, 6 in 2024-04-12T05:41:47.898 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:47 smithi067 ceph-mon[49010]: mgrmap e36: smithi032.hrwxrh(active, starting, since 0.00560134s), standbys: smithi067.dwichg 2024-04-12T05:41:47.898 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:47 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-12T05:41:47.899 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:47 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-12T05:41:47.899 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:47 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.pndtry"}]: dispatch 2024-04-12T05:41:47.899 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:47 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.jehaaj"}]: dispatch 2024-04-12T05:41:47.899 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:47 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.zesmgy"}]: dispatch 2024-04-12T05:41:47.899 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:47 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.ymvqih"}]: dispatch 2024-04-12T05:41:47.899 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:47 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr metadata", "who": "smithi032.hrwxrh", "id": "smithi032.hrwxrh"}]: dispatch 2024-04-12T05:41:47.899 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:47 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr metadata", "who": "smithi067.dwichg", "id": "smithi067.dwichg"}]: dispatch 2024-04-12T05:41:47.899 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:47 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:41:47.899 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:47 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:41:47.899 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:47 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:41:47.899 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:47 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:41:47.899 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:47 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:41:47.899 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:47 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:41:47.900 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:47 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T05:41:47.900 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:47 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T05:41:47.900 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:47 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T05:41:47.900 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:47 smithi067 ceph-mon[49010]: Standby manager daemon smithi067.dwichg restarted 2024-04-12T05:41:47.900 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:47 smithi067 ceph-mon[49010]: Standby manager daemon smithi067.dwichg started 2024-04-12T05:41:47.900 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:47 smithi067 ceph-mon[49010]: from='mgr.? 172.21.15.67:0/3922970479' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi067.dwichg/crt"}]: dispatch 2024-04-12T05:41:47.900 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:47 smithi067 ceph-mon[49010]: from='mgr.? 172.21.15.67:0/3922970479' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T05:41:47.900 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:47 smithi067 ceph-mon[49010]: from='mgr.? 172.21.15.67:0/3922970479' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi067.dwichg/key"}]: dispatch 2024-04-12T05:41:47.900 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:47 smithi067 ceph-mon[49010]: from='mgr.? 172.21.15.67:0/3922970479' entity='mgr.smithi067.dwichg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T05:41:47.900 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:47 smithi067 ceph-mon[49010]: Manager daemon smithi032.hrwxrh is now available 2024-04-12T05:41:49.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:48 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:41:49.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:48 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:41:49.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:48 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.hrwxrh/mirror_snapshot_schedule"}]: dispatch 2024-04-12T05:41:49.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:48 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.hrwxrh/trash_purge_schedule"}]: dispatch 2024-04-12T05:41:49.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:48 smithi032 ceph-mon[146817]: [12/Apr/2024:05:41:48] ENGINE Bus STARTING 2024-04-12T05:41:49.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:48 smithi032 ceph-mon[146817]: [12/Apr/2024:05:41:48] ENGINE Serving on https://172.21.15.32:7150 2024-04-12T05:41:49.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:48 smithi032 ceph-mon[146817]: [12/Apr/2024:05:41:48] ENGINE Serving on http://172.21.15.32:8765 2024-04-12T05:41:49.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:48 smithi032 ceph-mon[146817]: [12/Apr/2024:05:41:48] ENGINE Bus STARTED 2024-04-12T05:41:49.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:48 smithi032 ceph-mon[146817]: mgrmap e37: smithi032.hrwxrh(active, since 1.00957s), standbys: smithi067.dwichg 2024-04-12T05:41:49.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:48 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:41:49.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:48 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:41:49.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:48 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.hrwxrh/mirror_snapshot_schedule"}]: dispatch 2024-04-12T05:41:49.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:48 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi032.hrwxrh/trash_purge_schedule"}]: dispatch 2024-04-12T05:41:49.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:48 smithi067 ceph-mon[49010]: [12/Apr/2024:05:41:48] ENGINE Bus STARTING 2024-04-12T05:41:49.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:48 smithi067 ceph-mon[49010]: [12/Apr/2024:05:41:48] ENGINE Serving on https://172.21.15.32:7150 2024-04-12T05:41:49.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:48 smithi067 ceph-mon[49010]: [12/Apr/2024:05:41:48] ENGINE Serving on http://172.21.15.32:8765 2024-04-12T05:41:49.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:48 smithi067 ceph-mon[49010]: [12/Apr/2024:05:41:48] ENGINE Bus STARTED 2024-04-12T05:41:49.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:48 smithi067 ceph-mon[49010]: mgrmap e37: smithi032.hrwxrh(active, since 1.00957s), standbys: smithi067.dwichg 2024-04-12T05:41:50.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:49 smithi067 ceph-mon[49010]: pgmap v3: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:41:50.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:49 smithi032 ceph-mon[146817]: pgmap v3: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:41:51.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:50 smithi032 ceph-mon[146817]: pgmap v4: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:41:51.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:50 smithi032 ceph-mon[146817]: mgrmap e38: smithi032.hrwxrh(active, since 3s), standbys: smithi067.dwichg 2024-04-12T05:41:51.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:50 smithi067 ceph-mon[49010]: pgmap v4: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:41:51.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:50 smithi067 ceph-mon[49010]: mgrmap e38: smithi032.hrwxrh(active, since 3s), standbys: smithi067.dwichg 2024-04-12T05:41:52.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:52 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:52.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:52 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:52.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:52 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:52.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:52 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:52.592 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:52 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:52.592 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:52 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:52.592 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:52 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:52.592 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:52 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:53.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:53 smithi067 ceph-mon[49010]: pgmap v5: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:41:53.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:53 smithi067 ceph-mon[49010]: mgrmap e39: smithi032.hrwxrh(active, since 4s), standbys: smithi067.dwichg 2024-04-12T05:41:53.591 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:53 smithi032 ceph-mon[146817]: pgmap v5: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:41:53.592 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:53 smithi032 ceph-mon[146817]: mgrmap e39: smithi032.hrwxrh(active, since 4s), standbys: smithi067.dwichg 2024-04-12T05:41:54.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:54 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:54.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:54 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:54.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:54 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:54.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:54 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:54.805 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:54 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:54.805 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:54 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:54.805 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:54 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:54.806 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:54 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:55.526 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:55 smithi032 ceph-mon[146817]: pgmap v6: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:41:55.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:55 smithi067 ceph-mon[49010]: pgmap v6: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:41:57.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:56 smithi067 ceph-mon[49010]: pgmap v7: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 31 KiB/s rd, 0 B/s wr, 19 op/s 2024-04-12T05:41:57.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:56 smithi067 ceph-mon[49010]: Detected new or changed devices on smithi067 2024-04-12T05:41:57.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:56 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:57.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:56 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:57.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:56 smithi032 ceph-mon[146817]: pgmap v7: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 31 KiB/s rd, 0 B/s wr, 19 op/s 2024-04-12T05:41:57.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:56 smithi032 ceph-mon[146817]: Detected new or changed devices on smithi067 2024-04-12T05:41:57.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:56 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:57.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:56 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:59.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:58 smithi067 ceph-mon[49010]: Detected new or changed devices on smithi032 2024-04-12T05:41:59.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:58 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:59.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:58 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:59.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:58 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:41:59.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:58 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:41:59.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:58 smithi032 ceph-mon[146817]: Detected new or changed devices on smithi032 2024-04-12T05:41:59.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:58 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:59.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:58 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:59.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:58 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:41:59.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:58 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:41:59.978 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:59 smithi067 ceph-mon[49010]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-12T05:41:59.979 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:59 smithi067 ceph-mon[49010]: Updating smithi067:/etc/ceph/ceph.conf 2024-04-12T05:41:59.979 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:59 smithi067 ceph-mon[49010]: pgmap v8: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-12T05:41:59.979 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:59 smithi067 ceph-mon[49010]: Updating smithi067:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:41:59.979 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:59 smithi067 ceph-mon[49010]: Updating smithi032:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:41:59.979 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:59 smithi067 ceph-mon[49010]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:41:59.979 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:59 smithi067 ceph-mon[49010]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:41:59.979 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:59 smithi067 ceph-mon[49010]: Updating smithi067:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:41:59.979 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:59 smithi067 ceph-mon[49010]: Updating smithi032:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:41:59.979 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:59 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:59.979 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:59 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:59.979 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:59 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:59.979 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:59 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:59.980 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:59 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:41:59.980 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:59 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:41:59.980 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:59 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:41:59.980 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:41:59 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-12T05:42:00.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:59 smithi032 ceph-mon[146817]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-12T05:42:00.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:59 smithi032 ceph-mon[146817]: Updating smithi067:/etc/ceph/ceph.conf 2024-04-12T05:42:00.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:59 smithi032 ceph-mon[146817]: pgmap v8: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-12T05:42:00.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:59 smithi032 ceph-mon[146817]: Updating smithi067:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:42:00.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:59 smithi032 ceph-mon[146817]: Updating smithi032:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:42:00.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:59 smithi032 ceph-mon[146817]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:42:00.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:59 smithi032 ceph-mon[146817]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:42:00.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:59 smithi032 ceph-mon[146817]: Updating smithi067:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:42:00.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:59 smithi032 ceph-mon[146817]: Updating smithi032:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:42:00.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:59 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:00.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:59 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:00.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:59 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:00.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:59 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:00.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:59 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:00.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:59 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:42:00.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:59 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:42:00.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:41:59 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-12T05:42:01.946 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:01 smithi067 ceph-mon[49010]: pgmap v9: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 21 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-12T05:42:01.946 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:01 smithi067 ceph-mon[49010]: Upgrade: Updating mon.smithi067 2024-04-12T05:42:01.946 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:01 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:01.947 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:01 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T05:42:01.947 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:01 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T05:42:01.947 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:01 smithi067 ceph-mon[49010]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:01.947 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:01 smithi067 ceph-mon[49010]: Deploying daemon mon.smithi067 on smithi067 2024-04-12T05:42:02.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:01 smithi032 ceph-mon[146817]: pgmap v9: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 21 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-12T05:42:02.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:01 smithi032 ceph-mon[146817]: Upgrade: Updating mon.smithi067 2024-04-12T05:42:02.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:01 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:02.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:01 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T05:42:02.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:01 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T05:42:02.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:01 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:02.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:01 smithi032 ceph-mon[146817]: Deploying daemon mon.smithi067 on smithi067 2024-04-12T05:42:02.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:02 smithi067 systemd[1]: Stopping Ceph mon.smithi067 for 37b58ce2-f88d-11ee-bc8c-c7b262605968... 2024-04-12T05:42:03.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:02 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-mon-smithi067[48953]: 2024-04-12T05:42:02.651+0000 7f3cd9bc7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi067 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T05:42:03.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:02 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-mon-smithi067[48953]: 2024-04-12T05:42:02.651+0000 7f3cd9bc7700 -1 mon.smithi067@1(peon) e2 *** Got Signal Terminated *** 2024-04-12T05:42:03.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:03 smithi067 bash[112785]: ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-mon-smithi067 2024-04-12T05:42:04.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:03 smithi067 systemd[1]: ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@mon.smithi067.service: Succeeded. 2024-04-12T05:42:04.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:03 smithi067 systemd[1]: Stopped Ceph mon.smithi067 for 37b58ce2-f88d-11ee-bc8c-c7b262605968. 2024-04-12T05:42:04.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:03 smithi067 systemd[1]: Starting Ceph mon.smithi067 for 37b58ce2-f88d-11ee-bc8c-c7b262605968... 2024-04-12T05:42:04.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: set uid:gid to 167:167 (ceph:ceph) 2024-04-12T05:42:04.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable), process ceph-mon, pid 7 2024-04-12T05:42:04.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: pidfile_write: ignore empty --pid-file 2024-04-12T05:42:04.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: load: jerasure load: lrc 2024-04-12T05:42:04.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: RocksDB version: 7.9.2 2024-04-12T05:42:04.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Git sha 0 2024-04-12T05:42:04.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Compile date 2024-04-08 23:21:21 2024-04-12T05:42:04.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: DB SUMMARY 2024-04-12T05:42:04.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: DB Session ID: 6RJUHWKYUU3ONC7PUHG2 2024-04-12T05:42:04.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: CURRENT file: CURRENT 2024-04-12T05:42:04.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: IDENTITY file: IDENTITY 2024-04-12T05:42:04.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1083 Bytes 2024-04-12T05:42:04.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi067/store.db dir, Total Num: 1, files: 000027.sst 2024-04-12T05:42:04.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi067/store.db: 000025.log size: 6153810 ; 2024-04-12T05:42:04.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.error_if_exists: 0 2024-04-12T05:42:04.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.create_if_missing: 0 2024-04-12T05:42:04.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.paranoid_checks: 1 2024-04-12T05:42:04.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-12T05:42:04.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-12T05:42:04.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-12T05:42:04.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.env: 0x55ab229867a0 2024-04-12T05:42:04.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.fs: PosixFileSystem 2024-04-12T05:42:04.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.info_log: 0x55ab24ff93a0 2024-04-12T05:42:04.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.max_file_opening_threads: 16 2024-04-12T05:42:04.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.statistics: (nil) 2024-04-12T05:42:04.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.use_fsync: 0 2024-04-12T05:42:04.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.max_log_file_size: 0 2024-04-12T05:42:04.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-12T05:42:04.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-12T05:42:04.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.keep_log_file_num: 1000 2024-04-12T05:42:04.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.recycle_log_file_num: 0 2024-04-12T05:42:04.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.allow_fallocate: 1 2024-04-12T05:42:04.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.allow_mmap_reads: 0 2024-04-12T05:42:04.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.allow_mmap_writes: 0 2024-04-12T05:42:04.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.use_direct_reads: 0 2024-04-12T05:42:04.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-12T05:42:04.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.create_missing_column_families: 0 2024-04-12T05:42:04.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.db_log_dir: 2024-04-12T05:42:04.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.wal_dir: 2024-04-12T05:42:04.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-12T05:42:04.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-12T05:42:04.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-12T05:42:04.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-12T05:42:04.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-12T05:42:04.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-12T05:42:04.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.advise_random_on_open: 1 2024-04-12T05:42:04.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.db_write_buffer_size: 0 2024-04-12T05:42:04.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.write_buffer_manager: 0x55ab2426e140 2024-04-12T05:42:04.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-12T05:42:04.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-12T05:42:04.773 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-12T05:42:04.773 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.rate_limiter: (nil) 2024-04-12T05:42:04.773 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-12T05:42:04.773 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.wal_recovery_mode: 2 2024-04-12T05:42:04.773 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.enable_thread_tracking: 0 2024-04-12T05:42:04.773 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.enable_pipelined_write: 0 2024-04-12T05:42:04.773 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.unordered_write: 0 2024-04-12T05:42:04.773 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-12T05:42:04.773 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-12T05:42:04.773 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-12T05:42:04.773 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-12T05:42:04.773 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.row_cache: None 2024-04-12T05:42:04.773 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.wal_filter: None 2024-04-12T05:42:04.773 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-12T05:42:04.773 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.allow_ingest_behind: 0 2024-04-12T05:42:04.774 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.two_write_queues: 0 2024-04-12T05:42:04.774 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.manual_wal_flush: 0 2024-04-12T05:42:04.774 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.wal_compression: 0 2024-04-12T05:42:04.774 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.atomic_flush: 0 2024-04-12T05:42:04.774 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-12T05:42:04.774 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-12T05:42:04.774 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-12T05:42:04.774 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.log_readahead_size: 0 2024-04-12T05:42:04.774 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-12T05:42:04.774 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.best_efforts_recovery: 0 2024-04-12T05:42:04.774 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-12T05:42:04.774 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-12T05:42:04.774 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.allow_data_in_errors: 0 2024-04-12T05:42:04.774 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.db_host_id: __hostname__ 2024-04-12T05:42:04.774 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-12T05:42:04.775 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.max_background_jobs: 2 2024-04-12T05:42:04.775 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.max_background_compactions: -1 2024-04-12T05:42:04.775 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.max_subcompactions: 1 2024-04-12T05:42:04.775 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-12T05:42:04.775 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-12T05:42:04.775 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-12T05:42:04.775 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.max_total_wal_size: 0 2024-04-12T05:42:04.775 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-12T05:42:04.775 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-12T05:42:04.775 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-12T05:42:04.775 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-12T05:42:04.775 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.max_open_files: -1 2024-04-12T05:42:04.775 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.bytes_per_sync: 0 2024-04-12T05:42:04.775 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-12T05:42:04.775 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-12T05:42:04.776 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.compaction_readahead_size: 0 2024-04-12T05:42:04.776 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.max_background_flushes: -1 2024-04-12T05:42:04.776 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Compression algorithms supported: 2024-04-12T05:42:04.776 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-12T05:42:04.776 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: kZSTD supported: 0 2024-04-12T05:42:04.776 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: kXpressCompression supported: 0 2024-04-12T05:42:04.776 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: kLZ4HCCompression supported: 1 2024-04-12T05:42:04.776 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: kZlibCompression supported: 1 2024-04-12T05:42:04.776 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: kSnappyCompression supported: 1 2024-04-12T05:42:04.776 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: kLZ4Compression supported: 1 2024-04-12T05:42:04.776 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: kBZip2Compression supported: 0 2024-04-12T05:42:04.776 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-12T05:42:04.776 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-12T05:42:04.776 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi067/store.db/MANIFEST-000009 2024-04-12T05:42:04.776 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-12T05:42:04.777 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-12T05:42:04.777 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.merge_operator: 2024-04-12T05:42:04.777 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.compaction_filter: None 2024-04-12T05:42:04.777 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.compaction_filter_factory: None 2024-04-12T05:42:04.777 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.sst_partitioner_factory: None 2024-04-12T05:42:04.777 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-12T05:42:04.777 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-12T05:42:04.777 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ab24ff94c0) 2024-04-12T05:42:04.777 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: cache_index_and_filter_blocks: 1 2024-04-12T05:42:04.777 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-12T05:42:04.777 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-12T05:42:04.777 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: pin_top_level_index_and_filter: 1 2024-04-12T05:42:04.777 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: index_type: 0 2024-04-12T05:42:04.777 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: data_block_index_type: 0 2024-04-12T05:42:04.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: index_shortening: 1 2024-04-12T05:42:04.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-12T05:42:04.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: checksum: 4 2024-04-12T05:42:04.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: no_block_cache: 0 2024-04-12T05:42:04.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: block_cache: 0x55ab2430f1f0 2024-04-12T05:42:04.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: block_cache_name: BinnedLRUCache 2024-04-12T05:42:04.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: block_cache_options: 2024-04-12T05:42:04.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: capacity : 536870912 2024-04-12T05:42:04.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: num_shard_bits : 4 2024-04-12T05:42:04.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: strict_capacity_limit : 0 2024-04-12T05:42:04.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: high_pri_pool_ratio: 0.000 2024-04-12T05:42:04.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: block_cache_compressed: (nil) 2024-04-12T05:42:04.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: persistent_cache: (nil) 2024-04-12T05:42:04.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: block_size: 4096 2024-04-12T05:42:04.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: block_size_deviation: 10 2024-04-12T05:42:04.779 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: block_restart_interval: 16 2024-04-12T05:42:04.779 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: index_block_restart_interval: 1 2024-04-12T05:42:04.779 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: metadata_block_size: 4096 2024-04-12T05:42:04.779 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: partition_filters: 0 2024-04-12T05:42:04.779 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: use_delta_encoding: 1 2024-04-12T05:42:04.779 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: filter_policy: bloomfilter 2024-04-12T05:42:04.779 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: whole_key_filtering: 1 2024-04-12T05:42:04.779 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: verify_compression: 0 2024-04-12T05:42:04.779 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: read_amp_bytes_per_bit: 0 2024-04-12T05:42:04.779 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: format_version: 5 2024-04-12T05:42:04.779 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: enable_index_compression: 1 2024-04-12T05:42:04.779 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: block_align: 0 2024-04-12T05:42:04.779 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: max_auto_readahead_size: 262144 2024-04-12T05:42:04.779 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: prepopulate_block_cache: 0 2024-04-12T05:42:04.779 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: initial_auto_readahead_size: 8192 2024-04-12T05:42:04.780 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: num_file_reads_for_auto_readahead: 2 2024-04-12T05:42:04.780 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.write_buffer_size: 33554432 2024-04-12T05:42:04.780 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.max_write_buffer_number: 2 2024-04-12T05:42:04.780 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.compression: NoCompression 2024-04-12T05:42:04.780 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.bottommost_compression: Disabled 2024-04-12T05:42:04.780 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.prefix_extractor: nullptr 2024-04-12T05:42:04.780 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-12T05:42:04.780 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.num_levels: 7 2024-04-12T05:42:04.780 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-12T05:42:04.780 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-12T05:42:04.780 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-12T05:42:04.780 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-12T05:42:04.780 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-12T05:42:04.780 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-12T05:42:04.781 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-12T05:42:04.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-12T05:42:04.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-12T05:42:04.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-12T05:42:04.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-12T05:42:04.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-12T05:42:04.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-12T05:42:04.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.compression_opts.level: 32767 2024-04-12T05:42:04.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.compression_opts.strategy: 0 2024-04-12T05:42:04.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-12T05:42:04.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-12T05:42:04.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-12T05:42:04.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-12T05:42:04.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.compression_opts.enabled: false 2024-04-12T05:42:04.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-12T05:42:04.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-12T05:42:04.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-12T05:42:04.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-12T05:42:04.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.target_file_size_base: 67108864 2024-04-12T05:42:04.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-12T05:42:04.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-12T05:42:04.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-12T05:42:04.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-12T05:42:04.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-12T05:42:04.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-12T05:42:04.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-12T05:42:04.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-12T05:42:04.784 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-12T05:42:04.784 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-12T05:42:04.784 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-12T05:42:04.784 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-12T05:42:04.784 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-12T05:42:04.784 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-12T05:42:04.784 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.arena_block_size: 1048576 2024-04-12T05:42:04.784 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-12T05:42:04.784 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-12T05:42:04.784 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.disable_auto_compactions: 0 2024-04-12T05:42:04.784 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-12T05:42:04.784 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-12T05:42:04.784 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-12T05:42:04.784 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-12T05:42:04.784 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-12T05:42:04.785 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-12T05:42:04.785 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-12T05:42:04.785 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-12T05:42:04.785 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-12T05:42:04.785 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-12T05:42:04.785 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.table_properties_collectors: 2024-04-12T05:42:04.785 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.inplace_update_support: 0 2024-04-12T05:42:04.785 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-12T05:42:04.785 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-12T05:42:04.785 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-12T05:42:04.785 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-12T05:42:04.785 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.bloom_locality: 0 2024-04-12T05:42:04.785 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.max_successive_merges: 0 2024-04-12T05:42:04.785 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-12T05:42:04.785 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.paranoid_file_checks: 0 2024-04-12T05:42:04.785 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.force_consistency_checks: 1 2024-04-12T05:42:04.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.report_bg_io_stats: 0 2024-04-12T05:42:04.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.ttl: 2592000 2024-04-12T05:42:04.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-12T05:42:04.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-12T05:42:04.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-12T05:42:04.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.enable_blob_files: false 2024-04-12T05:42:04.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.min_blob_size: 0 2024-04-12T05:42:04.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.blob_file_size: 268435456 2024-04-12T05:42:04.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-12T05:42:04.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-12T05:42:04.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-12T05:42:04.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-12T05:42:04.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-12T05:42:04.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.blob_file_starting_level: 0 2024-04-12T05:42:04.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-12T05:42:04.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-04-12T05:42:04.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-12T05:42:04.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi067/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 12203, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-12T05:42:04.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-04-12T05:42:04.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f3cd62b0-d715-40ed-b978-64d508432e3d 2024-04-12T05:42:04.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712900524432031, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-04-12T05:42:04.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-04-12T05:42:04.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712900524446340, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 3703684, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12208, "largest_seqno": 14947, "table_properties": {"data_size": 3692821, "index_size": 6642, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 3205, "raw_key_size": 27999, "raw_average_key_size": 22, "raw_value_size": 3668101, "raw_average_value_size": 2908, "num_data_blocks": 302, "num_entries": 1261, "num_filter_entries": 1261, "num_deletions": 258, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712900524, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f3cd62b0-d715-40ed-b978-64d508432e3d", "db_session_id": "6RJUHWKYUU3ONC7PUHG2", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-04-12T05:42:04.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712900524446448, "job": 1, "event": "recovery_finished"} 2024-04-12T05:42:04.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-04-12T05:42:04.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-12T05:42:04.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi067/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-12T05:42:04.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55ab243ae000 2024-04-12T05:42:04.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: DB pointer 0x55ab2439c000 2024-04-12T05:42:04.788 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-12T05:42:04.788 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-12T05:42:04.788 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: ** DB Stats ** 2024-04-12T05:42:04.788 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T05:42:04.788 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-12T05:42:04.788 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-12T05:42:04.788 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-12T05:42:04.788 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-12T05:42:04.788 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-12T05:42:04.788 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-12T05:42:04.788 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: 2024-04-12T05:42:04.788 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: ** Compaction Stats [default] ** 2024-04-12T05:42:04.788 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-12T05:42:04.788 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-12T05:42:04.788 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: L0 1/0 3.53 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 705.3 0.01 0.00 1 0.005 0 0 0.0 0.0 2024-04-12T05:42:04.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: L6 1/0 6.75 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-12T05:42:04.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Sum 2/0 10.28 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 705.3 0.01 0.00 1 0.005 0 0 0.0 0.0 2024-04-12T05:42:04.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 705.3 0.01 0.00 1 0.005 0 0 0.0 0.0 2024-04-12T05:42:04.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: 2024-04-12T05:42:04.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: ** Compaction Stats [default] ** 2024-04-12T05:42:04.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-12T05:42:04.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T05:42:04.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 705.3 0.01 0.00 1 0.005 0 0 0.0 0.0 2024-04-12T05:42:04.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: 2024-04-12T05:42:04.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-12T05:42:04.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: 2024-04-12T05:42:04.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T05:42:04.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-04-12T05:42:04.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-12T05:42:04.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-12T05:42:04.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-12T05:42:04.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-12T05:42:04.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Cumulative compaction: 0.00 GB write, 168.08 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T05:42:04.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Interval compaction: 0.00 GB write, 168.08 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T05:42:04.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-12T05:42:04.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Block cache BinnedLRUCache@0x55ab2430f1f0#7 capacity: 512.00 MB usage: 9.86 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-04-12T05:42:04.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,3.28 KB,0.000625849%) IndexBlock(1,6.58 KB,0.00125468%) Misc(1,0.00 KB,0%) 2024-04-12T05:42:04.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: 2024-04-12T05:42:04.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-12T05:42:04.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: starting mon.smithi067 rank 1 at public addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] at bind addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi067 fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 2024-04-12T05:42:04.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: mon.smithi067@-1(???) e2 preinit fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 2024-04-12T05:42:04.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: mon.smithi067@-1(???).mds e22 new map 2024-04-12T05:42:04.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: mon.smithi067@-1(???).mds e22 print_map 2024-04-12T05:42:04.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: e22 2024-04-12T05:42:04.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:42:04.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:42:04.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: legacy client fscid: 1 2024-04-12T05:42:04.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: 2024-04-12T05:42:04.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Filesystem 'cephfs' (1) 2024-04-12T05:42:04.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: fs_name cephfs 2024-04-12T05:42:04.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: epoch 22 2024-04-12T05:42:04.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-12T05:42:04.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:42:04.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: modified 2024-04-12T05:41:05.358509+0000 2024-04-12T05:42:04.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: tableserver 0 2024-04-12T05:42:04.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: root 0 2024-04-12T05:42:04.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: session_timeout 60 2024-04-12T05:42:04.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: session_autoclose 300 2024-04-12T05:42:04.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: max_file_size 1099511627776 2024-04-12T05:42:04.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: max_xattr_size 65536 2024-04-12T05:42:04.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: required_client_features {} 2024-04-12T05:42:04.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: last_failure 0 2024-04-12T05:42:04.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: last_failure_osd_epoch 46 2024-04-12T05:42:04.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:42:04.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: max_mds 2 2024-04-12T05:42:04.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: in 0,1 2024-04-12T05:42:04.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: up {0=14480,1=24281} 2024-04-12T05:42:04.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: failed 2024-04-12T05:42:04.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: damaged 2024-04-12T05:42:04.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: stopped 2024-04-12T05:42:04.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: data_pools [3] 2024-04-12T05:42:04.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: metadata_pool 2 2024-04-12T05:42:04.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: inline_data disabled 2024-04-12T05:42:04.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: balancer 2024-04-12T05:42:04.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: bal_rank_mask -1 2024-04-12T05:42:04.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: standby_count_wanted 1 2024-04-12T05:42:04.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: [mds.cephfs.smithi032.pndtry{0:14480} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.32:6828/387754149,v1:172.21.15.32:6829/387754149] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:42:04.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: [mds.cephfs.smithi032.jehaaj{0:14490} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3678031004,v1:172.21.15.32:6827/3678031004] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:42:04.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: [mds.cephfs.smithi067.zesmgy{1:24281} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.67:6824/1291914599,v1:172.21.15.67:6825/1291914599] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:42:04.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: [mds.cephfs.smithi067.ymvqih{1:24295} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.67:6826/873395842,v1:172.21.15.67:6827/873395842] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:42:04.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: 2024-04-12T05:42:04.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: 2024-04-12T05:42:04.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: mon.smithi067@-1(???).osd e49 crush map has features 3314933000852226048, adjusting msgr requires 2024-04-12T05:42:04.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: mon.smithi067@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-04-12T05:42:04.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: mon.smithi067@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-04-12T05:42:04.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: mon.smithi067@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-04-12T05:42:04.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 ceph-mon[112979]: mon.smithi067@-1(???).paxosservice(auth 1..22) refresh upgraded, format 0 -> 3 2024-04-12T05:42:04.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 bash[112941]: b4591addf064e247b5824e83b7cdadc9925d4666258f3ede97c0c8910294acdf 2024-04-12T05:42:04.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:04 smithi067 systemd[1]: Started Ceph mon.smithi067 for 37b58ce2-f88d-11ee-bc8c-c7b262605968. 2024-04-12T05:42:05.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.288+0000 7fd55a9fd700 1 -- 172.21.15.32:0/1264844919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd554102e00 msgr2=0x7fd5541031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:05.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.288+0000 7fd55a9fd700 1 --2- 172.21.15.32:0/1264844919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd554102e00 0x7fd5541031d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd544004940 tx=0x7fd544009e30 comp rx=0 tx=0).stop 2024-04-12T05:42:05.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.288+0000 7fd55a9fd700 1 -- 172.21.15.32:0/1264844919 shutdown_connections 2024-04-12T05:42:05.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.288+0000 7fd55a9fd700 1 --2- 172.21.15.32:0/1264844919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5541037a0 0x7fd5540fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:05.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.288+0000 7fd55a9fd700 1 --2- 172.21.15.32:0/1264844919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd554102e00 0x7fd5541031d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:05.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.288+0000 7fd55a9fd700 1 -- 172.21.15.32:0/1264844919 >> 172.21.15.32:0/1264844919 conn(0x7fd5540f89b0 msgr2=0x7fd5540fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:05.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.288+0000 7fd55a9fd700 1 -- 172.21.15.32:0/1264844919 shutdown_connections 2024-04-12T05:42:05.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.288+0000 7fd55a9fd700 1 -- 172.21.15.32:0/1264844919 wait complete. 2024-04-12T05:42:05.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.289+0000 7fd55a9fd700 1 Processor -- start 2024-04-12T05:42:05.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.289+0000 7fd55a9fd700 1 -- start start 2024-04-12T05:42:05.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.289+0000 7fd55a9fd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd554102e00 0x7fd554113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:05.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.289+0000 7fd55a9fd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5541037a0 0x7fd554114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:05.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.289+0000 7fd55a9fd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd554114680 con 0x7fd554102e00 2024-04-12T05:42:05.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.289+0000 7fd55a9fd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55410da70 con 0x7fd5541037a0 2024-04-12T05:42:05.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.289+0000 7fd553fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd554102e00 0x7fd554113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:05.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.290+0000 7fd553fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd554102e00 0x7fd554113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53090/0 (socket says 172.21.15.32:53090) 2024-04-12T05:42:05.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.290+0000 7fd553fff700 1 -- 172.21.15.32:0/555142669 learned_addr learned my addr 172.21.15.32:0/555142669 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:42:05.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.290+0000 7fd5537fe700 1 --2- 172.21.15.32:0/555142669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5541037a0 0x7fd554114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:05.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.290+0000 7fd5537fe700 1 -- 172.21.15.32:0/555142669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5541037a0 msgr2=0x7fd554114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-12T05:42:05.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.290+0000 7fd5537fe700 1 -- 172.21.15.32:0/555142669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5541037a0 msgr2=0x7fd554114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-12T05:42:05.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.290+0000 7fd5537fe700 1 --2- 172.21.15.32:0/555142669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5541037a0 0x7fd554114030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-12T05:42:05.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.290+0000 7fd553fff700 1 -- 172.21.15.32:0/555142669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5541037a0 msgr2=0x7fd554114030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T05:42:05.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.290+0000 7fd5537fe700 1 --2- 172.21.15.32:0/555142669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5541037a0 0x7fd554114030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-12T05:42:05.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.290+0000 7fd553fff700 1 --2- 172.21.15.32:0/555142669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5541037a0 0x7fd554114030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:05.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.290+0000 7fd553fff700 1 -- 172.21.15.32:0/555142669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd544005040 con 0x7fd554102e00 2024-04-12T05:42:05.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.290+0000 7fd553fff700 1 --2- 172.21.15.32:0/555142669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd554102e00 0x7fd554113af0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd54400b3b0 tx=0x7fd544009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:05.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.290+0000 7fd550ff9700 1 -- 172.21.15.32:0/555142669 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd54400e040 con 0x7fd554102e00 2024-04-12T05:42:05.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.290+0000 7fd550ff9700 1 -- 172.21.15.32:0/555142669 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd544013420 con 0x7fd554102e00 2024-04-12T05:42:05.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.290+0000 7fd55a9fd700 1 -- 172.21.15.32:0/555142669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd55410dcf0 con 0x7fd554102e00 2024-04-12T05:42:05.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.290+0000 7fd550ff9700 1 -- 172.21.15.32:0/555142669 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd54401c630 con 0x7fd554102e00 2024-04-12T05:42:05.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.290+0000 7fd55a9fd700 1 -- 172.21.15.32:0/555142669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd55410e2b0 con 0x7fd554102e00 2024-04-12T05:42:05.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.291+0000 7fd55a9fd700 1 -- 172.21.15.32:0/555142669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5540fe550 con 0x7fd554102e00 2024-04-12T05:42:05.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.292+0000 7fd550ff9700 1 -- 172.21.15.32:0/555142669 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7fd544013970 con 0x7fd554102e00 2024-04-12T05:42:05.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.292+0000 7fd550ff9700 1 --2- 172.21.15.32:0/555142669 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd53c06cd40 0x7fd53c06f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:05.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.292+0000 7fd550ff9700 1 -- 172.21.15.32:0/555142669 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fd54408d4e0 con 0x7fd554102e00 2024-04-12T05:42:05.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.292+0000 7fd5537fe700 1 --2- 172.21.15.32:0/555142669 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd53c06cd40 0x7fd53c06f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:05.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.293+0000 7fd5537fe700 1 --2- 172.21.15.32:0/555142669 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd53c06cd40 0x7fd53c06f1f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd55410ec30 tx=0x7fd548009f70 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:05.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.297+0000 7fd550ff9700 1 -- 172.21.15.32:0/555142669 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd54405ade0 con 0x7fd554102e00 2024-04-12T05:42:05.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:05 smithi032 ceph-mon[146817]: pgmap v10: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-12T05:42:05.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:05 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:42:05.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.521+0000 7fd55a9fd700 1 -- 172.21.15.32:0/555142669 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd554109e50 con 0x7fd53c06cd40 2024-04-12T05:42:05.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:05 smithi067 ceph-mon[112979]: pgmap v10: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-12T05:42:05.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:05 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:42:05.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.522+0000 7fd550ff9700 1 -- 172.21.15.32:0/555142669 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7fd554109e50 con 0x7fd53c06cd40 2024-04-12T05:42:05.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.524+0000 7fd55a9fd700 1 -- 172.21.15.32:0/555142669 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd53c06cd40 msgr2=0x7fd53c06f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:05.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.524+0000 7fd55a9fd700 1 --2- 172.21.15.32:0/555142669 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd53c06cd40 0x7fd53c06f1f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd55410ec30 tx=0x7fd548009f70 comp rx=0 tx=0).stop 2024-04-12T05:42:05.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.524+0000 7fd55a9fd700 1 -- 172.21.15.32:0/555142669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd554102e00 msgr2=0x7fd554113af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:05.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.524+0000 7fd55a9fd700 1 --2- 172.21.15.32:0/555142669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd554102e00 0x7fd554113af0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd54400b3b0 tx=0x7fd544009e30 comp rx=0 tx=0).stop 2024-04-12T05:42:05.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.524+0000 7fd55a9fd700 1 -- 172.21.15.32:0/555142669 shutdown_connections 2024-04-12T05:42:05.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.524+0000 7fd55a9fd700 1 --2- 172.21.15.32:0/555142669 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd53c06cd40 0x7fd53c06f1f0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:05.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.524+0000 7fd55a9fd700 1 --2- 172.21.15.32:0/555142669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd554102e00 0x7fd554113af0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:05.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.524+0000 7fd55a9fd700 1 --2- 172.21.15.32:0/555142669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5541037a0 0x7fd554114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:05.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.524+0000 7fd55a9fd700 1 -- 172.21.15.32:0/555142669 >> 172.21.15.32:0/555142669 conn(0x7fd5540f89b0 msgr2=0x7fd5540f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:05.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.525+0000 7fd55a9fd700 1 -- 172.21.15.32:0/555142669 shutdown_connections 2024-04-12T05:42:05.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.525+0000 7fd55a9fd700 1 -- 172.21.15.32:0/555142669 wait complete. 2024-04-12T05:42:05.531 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:42:05.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.628+0000 7f40d664b700 1 -- 172.21.15.32:0/600809598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d0102de0 msgr2=0x7f40d01031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:05.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.628+0000 7f40d664b700 1 --2- 172.21.15.32:0/600809598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d0102de0 0x7f40d01031b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f40c0004940 tx=0x7f40c0009e30 comp rx=0 tx=0).stop 2024-04-12T05:42:05.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.629+0000 7f40d664b700 1 -- 172.21.15.32:0/600809598 shutdown_connections 2024-04-12T05:42:05.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.629+0000 7f40d664b700 1 --2- 172.21.15.32:0/600809598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40d00656a0 0x7f40d0065af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:05.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.629+0000 7f40d664b700 1 --2- 172.21.15.32:0/600809598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d0102de0 0x7f40d01031b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:05.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.629+0000 7f40d664b700 1 -- 172.21.15.32:0/600809598 >> 172.21.15.32:0/600809598 conn(0x7f40d0072bc0 msgr2=0x7f40d0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:05.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.629+0000 7f40d664b700 1 -- 172.21.15.32:0/600809598 shutdown_connections 2024-04-12T05:42:05.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.629+0000 7f40d664b700 1 -- 172.21.15.32:0/600809598 wait complete. 2024-04-12T05:42:05.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.629+0000 7f40d664b700 1 Processor -- start 2024-04-12T05:42:05.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.629+0000 7f40d664b700 1 -- start start 2024-04-12T05:42:05.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.630+0000 7f40d664b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40d00656a0 0x7f40d0113a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:05.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.630+0000 7f40d664b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d0102de0 0x7f40d0113f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:05.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.630+0000 7f40d664b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d0114590 con 0x7f40d0102de0 2024-04-12T05:42:05.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.630+0000 7f40d664b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d010d980 con 0x7f40d00656a0 2024-04-12T05:42:05.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.630+0000 7f40cf7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d0102de0 0x7f40d0113f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:05.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.630+0000 7f40cf7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d0102de0 0x7f40d0113f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53112/0 (socket says 172.21.15.32:53112) 2024-04-12T05:42:05.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.630+0000 7f40cf7fe700 1 -- 172.21.15.32:0/636050052 learned_addr learned my addr 172.21.15.32:0/636050052 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:42:05.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.630+0000 7f40cffff700 1 --2- 172.21.15.32:0/636050052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40d00656a0 0x7f40d0113a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:05.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.630+0000 7f40cf7fe700 1 -- 172.21.15.32:0/636050052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40d00656a0 msgr2=0x7f40d0113a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:05.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.630+0000 7f40cf7fe700 1 --2- 172.21.15.32:0/636050052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40d00656a0 0x7f40d0113a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:05.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.630+0000 7f40cf7fe700 1 -- 172.21.15.32:0/636050052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c0005040 con 0x7f40d0102de0 2024-04-12T05:42:05.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.630+0000 7f40cf7fe700 1 --2- 172.21.15.32:0/636050052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d0102de0 0x7f40d0113f40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f40c400c830 tx=0x7f40c4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:05.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.630+0000 7f40ccff9700 1 -- 172.21.15.32:0/636050052 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40c400a760 con 0x7f40d0102de0 2024-04-12T05:42:05.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.630+0000 7f40ccff9700 1 -- 172.21.15.32:0/636050052 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f40c400ad70 con 0x7f40d0102de0 2024-04-12T05:42:05.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.630+0000 7f40d664b700 1 -- 172.21.15.32:0/636050052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40d010dc60 con 0x7f40d0102de0 2024-04-12T05:42:05.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.630+0000 7f40ccff9700 1 -- 172.21.15.32:0/636050052 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40c40113a0 con 0x7f40d0102de0 2024-04-12T05:42:05.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.630+0000 7f40d664b700 1 -- 172.21.15.32:0/636050052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40d010e220 con 0x7f40d0102de0 2024-04-12T05:42:05.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.631+0000 7f40d664b700 1 -- 172.21.15.32:0/636050052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40d00fe4c0 con 0x7f40d0102de0 2024-04-12T05:42:05.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.632+0000 7f40ccff9700 1 -- 172.21.15.32:0/636050052 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7f40c4012400 con 0x7f40d0102de0 2024-04-12T05:42:05.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.632+0000 7f40ccff9700 1 --2- 172.21.15.32:0/636050052 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f40b806cd40 0x7f40b806f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:05.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.632+0000 7f40ccff9700 1 -- 172.21.15.32:0/636050052 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f40c408c130 con 0x7f40d0102de0 2024-04-12T05:42:05.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.632+0000 7f40cffff700 1 --2- 172.21.15.32:0/636050052 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f40b806cd40 0x7f40b806f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:05.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.633+0000 7f40cffff700 1 --2- 172.21.15.32:0/636050052 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f40b806cd40 0x7f40b806f1f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f40c0004f90 tx=0x7f40c0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:05.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.637+0000 7f40ccff9700 1 -- 172.21.15.32:0/636050052 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f40c4059be0 con 0x7f40d0102de0 2024-04-12T05:42:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.857+0000 7f40d664b700 1 -- 172.21.15.32:0/636050052 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f40d0109cf0 con 0x7f40b806cd40 2024-04-12T05:42:05.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.858+0000 7f40ccff9700 1 -- 172.21.15.32:0/636050052 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f40d0109cf0 con 0x7f40b806cd40 2024-04-12T05:42:05.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.860+0000 7f40d664b700 1 -- 172.21.15.32:0/636050052 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f40b806cd40 msgr2=0x7f40b806f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:05.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.860+0000 7f40d664b700 1 --2- 172.21.15.32:0/636050052 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f40b806cd40 0x7f40b806f1f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f40c0004f90 tx=0x7f40c0009e30 comp rx=0 tx=0).stop 2024-04-12T05:42:05.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.860+0000 7f40d664b700 1 -- 172.21.15.32:0/636050052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d0102de0 msgr2=0x7f40d0113f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:05.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.860+0000 7f40d664b700 1 --2- 172.21.15.32:0/636050052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d0102de0 0x7f40d0113f40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f40c400c830 tx=0x7f40c4004470 comp rx=0 tx=0).stop 2024-04-12T05:42:05.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.860+0000 7f40d664b700 1 -- 172.21.15.32:0/636050052 shutdown_connections 2024-04-12T05:42:05.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.860+0000 7f40d664b700 1 --2- 172.21.15.32:0/636050052 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f40b806cd40 0x7f40b806f1f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:05.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.860+0000 7f40d664b700 1 --2- 172.21.15.32:0/636050052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40d00656a0 0x7f40d0113a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:05.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.860+0000 7f40d664b700 1 --2- 172.21.15.32:0/636050052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d0102de0 0x7f40d0113f40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:05.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.860+0000 7f40d664b700 1 -- 172.21.15.32:0/636050052 >> 172.21.15.32:0/636050052 conn(0x7f40d0072bc0 msgr2=0x7f40d0105b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:05.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.861+0000 7f40d664b700 1 -- 172.21.15.32:0/636050052 shutdown_connections 2024-04-12T05:42:05.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.861+0000 7f40d664b700 1 -- 172.21.15.32:0/636050052 wait complete. 2024-04-12T05:42:05.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.964+0000 7fa673c34700 1 -- 172.21.15.32:0/187278745 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa66c101d70 msgr2=0x7fa66c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:05.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.964+0000 7fa673c34700 1 --2- 172.21.15.32:0/187278745 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa66c101d70 0x7fa66c102140 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa668004940 tx=0x7fa668009e30 comp rx=0 tx=0).stop 2024-04-12T05:42:05.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.964+0000 7fa673c34700 1 -- 172.21.15.32:0/187278745 shutdown_connections 2024-04-12T05:42:05.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.964+0000 7fa673c34700 1 --2- 172.21.15.32:0/187278745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa66c102710 0x7fa66c0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:05.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.964+0000 7fa673c34700 1 --2- 172.21.15.32:0/187278745 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa66c101d70 0x7fa66c102140 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:05.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.964+0000 7fa673c34700 1 -- 172.21.15.32:0/187278745 >> 172.21.15.32:0/187278745 conn(0x7fa66c072bc0 msgr2=0x7fa66c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:05.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.964+0000 7fa673c34700 1 -- 172.21.15.32:0/187278745 shutdown_connections 2024-04-12T05:42:05.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.964+0000 7fa673c34700 1 -- 172.21.15.32:0/187278745 wait complete. 2024-04-12T05:42:05.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.965+0000 7fa673c34700 1 Processor -- start 2024-04-12T05:42:05.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.965+0000 7fa673c34700 1 -- start start 2024-04-12T05:42:05.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.965+0000 7fa673c34700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa66c101d70 0x7fa66c1129c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:05.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.965+0000 7fa673c34700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa66c102710 0x7fa66c112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:05.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.965+0000 7fa673c34700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa66c113550 con 0x7fa66c102710 2024-04-12T05:42:05.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.965+0000 7fa673c34700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa66c10c940 con 0x7fa66c101d70 2024-04-12T05:42:05.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.965+0000 7fa6711cf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa66c102710 0x7fa66c112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:05.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.965+0000 7fa6711cf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa66c102710 0x7fa66c112f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53136/0 (socket says 172.21.15.32:53136) 2024-04-12T05:42:05.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.965+0000 7fa6711cf700 1 -- 172.21.15.32:0/4176417232 learned_addr learned my addr 172.21.15.32:0/4176417232 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:42:05.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.965+0000 7fa6719d0700 1 --2- 172.21.15.32:0/4176417232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa66c101d70 0x7fa66c1129c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:05.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.965+0000 7fa6719d0700 1 -- 172.21.15.32:0/4176417232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa66c101d70 msgr2=0x7fa66c1129c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-12T05:42:05.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.965+0000 7fa6719d0700 1 -- 172.21.15.32:0/4176417232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa66c101d70 msgr2=0x7fa66c1129c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-12T05:42:05.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.965+0000 7fa6719d0700 1 --2- 172.21.15.32:0/4176417232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa66c101d70 0x7fa66c1129c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-12T05:42:05.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.965+0000 7fa6719d0700 1 --2- 172.21.15.32:0/4176417232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa66c101d70 0x7fa66c1129c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-12T05:42:05.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.965+0000 7fa6711cf700 1 -- 172.21.15.32:0/4176417232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa66c101d70 msgr2=0x7fa66c1129c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T05:42:05.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.965+0000 7fa6711cf700 1 --2- 172.21.15.32:0/4176417232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa66c101d70 0x7fa66c1129c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:05.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.965+0000 7fa6711cf700 1 -- 172.21.15.32:0/4176417232 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa668005040 con 0x7fa66c102710 2024-04-12T05:42:05.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.966+0000 7fa6711cf700 1 --2- 172.21.15.32:0/4176417232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa66c102710 0x7fa66c112f00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa660002a80 tx=0x7fa660004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:05.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.966+0000 7fa65e7fc700 1 -- 172.21.15.32:0/4176417232 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa660007cc0 con 0x7fa66c102710 2024-04-12T05:42:05.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.966+0000 7fa673c34700 1 -- 172.21.15.32:0/4176417232 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa66c10cc20 con 0x7fa66c102710 2024-04-12T05:42:05.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.966+0000 7fa65e7fc700 1 -- 172.21.15.32:0/4176417232 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa66000f040 con 0x7fa66c102710 2024-04-12T05:42:05.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.966+0000 7fa65e7fc700 1 -- 172.21.15.32:0/4176417232 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa66000da30 con 0x7fa66c102710 2024-04-12T05:42:05.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.966+0000 7fa673c34700 1 -- 172.21.15.32:0/4176417232 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa66c10d1e0 con 0x7fa66c102710 2024-04-12T05:42:05.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.967+0000 7fa673c34700 1 -- 172.21.15.32:0/4176417232 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa66c0ff080 con 0x7fa66c102710 2024-04-12T05:42:05.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.967+0000 7fa65e7fc700 1 -- 172.21.15.32:0/4176417232 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7fa66000dbd0 con 0x7fa66c102710 2024-04-12T05:42:05.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.968+0000 7fa65e7fc700 1 --2- 172.21.15.32:0/4176417232 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa65806cd40 0x7fa65806f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:05.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.968+0000 7fa65e7fc700 1 -- 172.21.15.32:0/4176417232 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fa66008b990 con 0x7fa66c102710 2024-04-12T05:42:05.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.968+0000 7fa6719d0700 1 --2- 172.21.15.32:0/4176417232 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa65806cd40 0x7fa65806f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:05.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.968+0000 7fa6719d0700 1 --2- 172.21.15.32:0/4176417232 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa65806cd40 0x7fa65806f1f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa668004f90 tx=0x7fa668009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:05.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:05.972+0000 7fa65e7fc700 1 -- 172.21.15.32:0/4176417232 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa660059210 con 0x7fa66c102710 2024-04-12T05:42:06.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.185+0000 7fa673c34700 1 -- 172.21.15.32:0/4176417232 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa66c108d50 con 0x7fa65806cd40 2024-04-12T05:42:06.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.195+0000 7fa65e7fc700 1 -- 172.21.15.32:0/4176417232 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fa66c108d50 con 0x7fa65806cd40 2024-04-12T05:42:06.195 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:42:06.195 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (113s) 12s ago 13m 19.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:42:06.195 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (13m) 12s ago 13m 6656k - 17.2.7-827-gd3b5060a 21de9b7369ad a9cfc00c311b 2024-04-12T05:42:06.195 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (12m) 14s ago 12m 6651k - 17.2.7-827-gd3b5060a 21de9b7369ad 1f2eea4542d9 2024-04-12T05:42:06.195 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (97s) 12s ago 13m 76.3M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:42:06.195 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (7m) 12s ago 7m 64.8M - 17.2.7-827-gd3b5060a 21de9b7369ad e7ada8e5259c 2024-04-12T05:42:06.195 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (7m) 12s ago 7m 116M - 17.2.7-827-gd3b5060a 21de9b7369ad 55124da9afdb 2024-04-12T05:42:06.195 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (7m) 14s ago 7m 62.6M - 17.2.7-827-gd3b5060a 21de9b7369ad 0c295d31a4a6 2024-04-12T05:42:06.195 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (7m) 14s ago 7m 120M - 17.2.7-827-gd3b5060a 21de9b7369ad 6eecd7deaac2 2024-04-12T05:42:06.195 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (3m) 12s ago 14m 470M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:42:06.195 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (2m) 14s ago 12m 446M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:42:06.195 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (29s) 12s ago 14m 27.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:42:06.195 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 starting - - - 2048M 2024-04-12T05:42:06.195 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (2m) 12s ago 13m 16.3M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:42:06.196 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (2m) 14s ago 12m 17.9M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:42:06.196 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (11m) 12s ago 11m 220M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad 97446e42ceba 2024-04-12T05:42:06.196 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (11m) 12s ago 11m 237M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad bf9f9a2db2d5 2024-04-12T05:42:06.196 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (10m) 12s ago 10m 190M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad f055cf1ca665 2024-04-12T05:42:06.196 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (10m) 14s ago 10m 257M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad 61b851235351 2024-04-12T05:42:06.196 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (9m) 14s ago 9m 229M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad 2c84d9047e9c 2024-04-12T05:42:06.196 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (9m) 14s ago 9m 219M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad e54839bb6055 2024-04-12T05:42:06.196 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (2m) 12s ago 12m 56.9M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:42:06.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.197+0000 7fa673c34700 1 -- 172.21.15.32:0/4176417232 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa65806cd40 msgr2=0x7fa65806f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:06.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.197+0000 7fa673c34700 1 --2- 172.21.15.32:0/4176417232 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa65806cd40 0x7fa65806f1f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa668004f90 tx=0x7fa668009e30 comp rx=0 tx=0).stop 2024-04-12T05:42:06.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.197+0000 7fa673c34700 1 -- 172.21.15.32:0/4176417232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa66c102710 msgr2=0x7fa66c112f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:06.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.197+0000 7fa673c34700 1 --2- 172.21.15.32:0/4176417232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa66c102710 0x7fa66c112f00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa660002a80 tx=0x7fa660004630 comp rx=0 tx=0).stop 2024-04-12T05:42:06.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.197+0000 7fa673c34700 1 -- 172.21.15.32:0/4176417232 shutdown_connections 2024-04-12T05:42:06.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.197+0000 7fa673c34700 1 --2- 172.21.15.32:0/4176417232 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa65806cd40 0x7fa65806f1f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:06.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.197+0000 7fa673c34700 1 --2- 172.21.15.32:0/4176417232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa66c101d70 0x7fa66c1129c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:06.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.197+0000 7fa673c34700 1 --2- 172.21.15.32:0/4176417232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa66c102710 0x7fa66c112f00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:06.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.197+0000 7fa673c34700 1 -- 172.21.15.32:0/4176417232 >> 172.21.15.32:0/4176417232 conn(0x7fa66c072bc0 msgr2=0x7fa66c0fb050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:06.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.198+0000 7fa673c34700 1 -- 172.21.15.32:0/4176417232 shutdown_connections 2024-04-12T05:42:06.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.198+0000 7fa673c34700 1 -- 172.21.15.32:0/4176417232 wait complete. 2024-04-12T05:42:06.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.300+0000 7f42f5b3c700 1 -- 172.21.15.32:0/1902090569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42f0102e10 msgr2=0x7f42f01031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:06.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.300+0000 7f42f5b3c700 1 --2- 172.21.15.32:0/1902090569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42f0102e10 0x7f42f01031e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f42e0004940 tx=0x7f42e0009e30 comp rx=0 tx=0).stop 2024-04-12T05:42:06.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.300+0000 7f42f5b3c700 1 -- 172.21.15.32:0/1902090569 shutdown_connections 2024-04-12T05:42:06.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.300+0000 7f42f5b3c700 1 --2- 172.21.15.32:0/1902090569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42f01037b0 0x7f42f00fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:06.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.300+0000 7f42f5b3c700 1 --2- 172.21.15.32:0/1902090569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42f0102e10 0x7f42f01031e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:06.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.300+0000 7f42f5b3c700 1 -- 172.21.15.32:0/1902090569 >> 172.21.15.32:0/1902090569 conn(0x7f42f00f89a0 msgr2=0x7f42f00fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:06.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.300+0000 7f42f5b3c700 1 -- 172.21.15.32:0/1902090569 shutdown_connections 2024-04-12T05:42:06.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.300+0000 7f42f5b3c700 1 -- 172.21.15.32:0/1902090569 wait complete. 2024-04-12T05:42:06.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.300+0000 7f42f5b3c700 1 Processor -- start 2024-04-12T05:42:06.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.301+0000 7f42f5b3c700 1 -- start start 2024-04-12T05:42:06.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.301+0000 7f42f5b3c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42f0102e10 0x7f42f0113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:06.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.301+0000 7f42f5b3c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42f01037b0 0x7f42f0114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:06.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.301+0000 7f42f5b3c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f0114680 con 0x7f42f0102e10 2024-04-12T05:42:06.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.301+0000 7f42f5b3c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f010da70 con 0x7f42f01037b0 2024-04-12T05:42:06.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.301+0000 7f42ef7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42f0102e10 0x7f42f0113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:06.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.301+0000 7f42ef7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42f0102e10 0x7f42f0113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53144/0 (socket says 172.21.15.32:53144) 2024-04-12T05:42:06.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.301+0000 7f42ef7fe700 1 -- 172.21.15.32:0/1454546213 learned_addr learned my addr 172.21.15.32:0/1454546213 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:42:06.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.301+0000 7f42ef7fe700 1 -- 172.21.15.32:0/1454546213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42f01037b0 msgr2=0x7f42f0114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:06.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.301+0000 7f42ef7fe700 1 --2- 172.21.15.32:0/1454546213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42f01037b0 0x7f42f0114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:06.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.301+0000 7f42ef7fe700 1 -- 172.21.15.32:0/1454546213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e0005040 con 0x7f42f0102e10 2024-04-12T05:42:06.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.302+0000 7f42ef7fe700 1 --2- 172.21.15.32:0/1454546213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42f0102e10 0x7f42f0113af0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f42e0009390 tx=0x7f42e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:06.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.302+0000 7f42d7fff700 1 -- 172.21.15.32:0/1454546213 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42e000e040 con 0x7f42f0102e10 2024-04-12T05:42:06.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.302+0000 7f42d7fff700 1 -- 172.21.15.32:0/1454546213 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f42e0013420 con 0x7f42f0102e10 2024-04-12T05:42:06.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.302+0000 7f42f5b3c700 1 -- 172.21.15.32:0/1454546213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f010dcf0 con 0x7f42f0102e10 2024-04-12T05:42:06.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.302+0000 7f42d7fff700 1 -- 172.21.15.32:0/1454546213 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42e001c630 con 0x7f42f0102e10 2024-04-12T05:42:06.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.302+0000 7f42f5b3c700 1 -- 172.21.15.32:0/1454546213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f010e2b0 con 0x7f42f0102e10 2024-04-12T05:42:06.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.303+0000 7f42f5b3c700 1 -- 172.21.15.32:0/1454546213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42f00fe560 con 0x7f42f0102e10 2024-04-12T05:42:06.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.303+0000 7f42d7fff700 1 -- 172.21.15.32:0/1454546213 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7f42e00139e0 con 0x7f42f0102e10 2024-04-12T05:42:06.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.304+0000 7f42d7fff700 1 --2- 172.21.15.32:0/1454546213 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f42d806cd40 0x7f42d806f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:06.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.304+0000 7f42d7fff700 1 -- 172.21.15.32:0/1454546213 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f42e008d3f0 con 0x7f42f0102e10 2024-04-12T05:42:06.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.304+0000 7f42eeffd700 1 --2- 172.21.15.32:0/1454546213 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f42d806cd40 0x7f42d806f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:06.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.304+0000 7f42eeffd700 1 --2- 172.21.15.32:0/1454546213 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f42d806cd40 0x7f42d806f1f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f42f010ec30 tx=0x7f42e4007cc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:06.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.308+0000 7f42d7fff700 1 -- 172.21.15.32:0/1454546213 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f42e005ae40 con 0x7f42f0102e10 2024-04-12T05:42:06.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:06 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-12T05:42:06.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:06 smithi032 ceph-mon[146817]: mon.smithi032 calling monitor election 2024-04-12T05:42:06.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:06 smithi032 ceph-mon[146817]: mon.smithi067 calling monitor election 2024-04-12T05:42:06.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:06 smithi032 ceph-mon[146817]: mon.smithi032 is new leader, mons smithi032,smithi067 in quorum (ranks 0,1) 2024-04-12T05:42:06.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:06 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-12T05:42:06.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:06 smithi032 ceph-mon[146817]: monmap e3: 2 mons at {smithi032=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0],smithi067=[v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-12T05:42:06.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:06 smithi032 ceph-mon[146817]: fsmap cephfs:2 {0=cephfs.smithi032.pndtry=up:active,1=cephfs.smithi067.zesmgy=up:active} 2 up:standby-replay 2024-04-12T05:42:06.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:06 smithi032 ceph-mon[146817]: osdmap e49: 6 total, 6 up, 6 in 2024-04-12T05:42:06.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:06 smithi032 ceph-mon[146817]: mgrmap e39: smithi032.hrwxrh(active, since 17s), standbys: smithi067.dwichg 2024-04-12T05:42:06.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:06 smithi032 ceph-mon[146817]: overall HEALTH_OK 2024-04-12T05:42:06.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:06 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:06.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:06 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:06.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:06 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:42:06.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:06 smithi032 ceph-mon[146817]: from='client.34128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:06.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:06 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi032"}]: dispatch 2024-04-12T05:42:06.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:06 smithi067 ceph-mon[112979]: mon.smithi032 calling monitor election 2024-04-12T05:42:06.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:06 smithi067 ceph-mon[112979]: mon.smithi067 calling monitor election 2024-04-12T05:42:06.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:06 smithi067 ceph-mon[112979]: mon.smithi032 is new leader, mons smithi032,smithi067 in quorum (ranks 0,1) 2024-04-12T05:42:06.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:06 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-12T05:42:06.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:06 smithi067 ceph-mon[112979]: monmap e3: 2 mons at {smithi032=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0],smithi067=[v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-12T05:42:06.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:06 smithi067 ceph-mon[112979]: fsmap cephfs:2 {0=cephfs.smithi032.pndtry=up:active,1=cephfs.smithi067.zesmgy=up:active} 2 up:standby-replay 2024-04-12T05:42:06.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:06 smithi067 ceph-mon[112979]: osdmap e49: 6 total, 6 up, 6 in 2024-04-12T05:42:06.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:06 smithi067 ceph-mon[112979]: mgrmap e39: smithi032.hrwxrh(active, since 17s), standbys: smithi067.dwichg 2024-04-12T05:42:06.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:06 smithi067 ceph-mon[112979]: overall HEALTH_OK 2024-04-12T05:42:06.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:06 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:06.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:06 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:06.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:06 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:42:06.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:06 smithi067 ceph-mon[112979]: from='client.34128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:06.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.599+0000 7f42f5b3c700 1 -- 172.21.15.32:0/1454546213 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f42f004bed0 con 0x7f42f0102e10 2024-04-12T05:42:06.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.599+0000 7f42d7fff700 1 -- 172.21.15.32:0/1454546213 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+742 (secure 0 0 0) 0x7f42e005a9d0 con 0x7f42f0102e10 2024-04-12T05:42:06.596 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:42:06.596 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:42:06.596 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:42:06.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:42:06.596 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:42:06.596 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:42:06.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:42:06.596 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:42:06.596 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 6 2024-04-12T05:42:06.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:42:06.596 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:42:06.597 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 4 2024-04-12T05:42:06.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:42:06.597 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:42:06.597 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 10, 2024-04-12T05:42:06.597 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:42:06.597 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:42:06.597 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:42:06.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.601+0000 7f42f5b3c700 1 -- 172.21.15.32:0/1454546213 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f42d806cd40 msgr2=0x7f42d806f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:06.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.601+0000 7f42f5b3c700 1 --2- 172.21.15.32:0/1454546213 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f42d806cd40 0x7f42d806f1f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f42f010ec30 tx=0x7f42e4007cc0 comp rx=0 tx=0).stop 2024-04-12T05:42:06.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.601+0000 7f42f5b3c700 1 -- 172.21.15.32:0/1454546213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42f0102e10 msgr2=0x7f42f0113af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:06.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.601+0000 7f42f5b3c700 1 --2- 172.21.15.32:0/1454546213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42f0102e10 0x7f42f0113af0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f42e0009390 tx=0x7f42e0009e30 comp rx=0 tx=0).stop 2024-04-12T05:42:06.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.601+0000 7f42f5b3c700 1 -- 172.21.15.32:0/1454546213 shutdown_connections 2024-04-12T05:42:06.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.601+0000 7f42f5b3c700 1 --2- 172.21.15.32:0/1454546213 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f42d806cd40 0x7f42d806f1f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:06.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.601+0000 7f42f5b3c700 1 --2- 172.21.15.32:0/1454546213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42f0102e10 0x7f42f0113af0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:06.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.601+0000 7f42f5b3c700 1 --2- 172.21.15.32:0/1454546213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42f01037b0 0x7f42f0114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:06.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.601+0000 7f42f5b3c700 1 -- 172.21.15.32:0/1454546213 >> 172.21.15.32:0/1454546213 conn(0x7f42f00f89a0 msgr2=0x7f42f00f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:06.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.602+0000 7f42f5b3c700 1 -- 172.21.15.32:0/1454546213 shutdown_connections 2024-04-12T05:42:06.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.602+0000 7f42f5b3c700 1 -- 172.21.15.32:0/1454546213 wait complete. 2024-04-12T05:42:06.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.707+0000 7f5b2c0f3700 1 -- 172.21.15.32:0/2568533166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b24101d70 msgr2=0x7f5b24102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:06.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.707+0000 7f5b2c0f3700 1 --2- 172.21.15.32:0/2568533166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b24101d70 0x7f5b24102140 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5b20004940 tx=0x7f5b20009e30 comp rx=0 tx=0).stop 2024-04-12T05:42:06.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.707+0000 7f5b2c0f3700 1 -- 172.21.15.32:0/2568533166 shutdown_connections 2024-04-12T05:42:06.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.707+0000 7f5b2c0f3700 1 --2- 172.21.15.32:0/2568533166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b24102710 0x7f5b240fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:06.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.707+0000 7f5b2c0f3700 1 --2- 172.21.15.32:0/2568533166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b24101d70 0x7f5b24102140 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:06.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.707+0000 7f5b2c0f3700 1 -- 172.21.15.32:0/2568533166 >> 172.21.15.32:0/2568533166 conn(0x7f5b24072bc0 msgr2=0x7f5b24072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:06.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.707+0000 7f5b2c0f3700 1 -- 172.21.15.32:0/2568533166 shutdown_connections 2024-04-12T05:42:06.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.707+0000 7f5b2c0f3700 1 -- 172.21.15.32:0/2568533166 wait complete. 2024-04-12T05:42:06.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.707+0000 7f5b2c0f3700 1 Processor -- start 2024-04-12T05:42:06.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.708+0000 7f5b2c0f3700 1 -- start start 2024-04-12T05:42:06.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.708+0000 7f5b2c0f3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b24101d70 0x7f5b24115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:06.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.708+0000 7f5b2c0f3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b24102710 0x7f5b24116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:06.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.708+0000 7f5b2c0f3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b24116850 con 0x7f5b24102710 2024-04-12T05:42:06.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.708+0000 7f5b2c0f3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2410fc20 con 0x7f5b24101d70 2024-04-12T05:42:06.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.708+0000 7f5b2968e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b24102710 0x7f5b24116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:06.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.708+0000 7f5b2968e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b24102710 0x7f5b24116200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53168/0 (socket says 172.21.15.32:53168) 2024-04-12T05:42:06.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.708+0000 7f5b2968e700 1 -- 172.21.15.32:0/2426611288 learned_addr learned my addr 172.21.15.32:0/2426611288 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:42:06.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.708+0000 7f5b29e8f700 1 --2- 172.21.15.32:0/2426611288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b24101d70 0x7f5b24115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:06.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.708+0000 7f5b29e8f700 1 -- 172.21.15.32:0/2426611288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b24101d70 msgr2=0x7f5b24115cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-12T05:42:06.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.708+0000 7f5b29e8f700 1 -- 172.21.15.32:0/2426611288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b24101d70 msgr2=0x7f5b24115cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-12T05:42:06.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.708+0000 7f5b29e8f700 1 --2- 172.21.15.32:0/2426611288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b24101d70 0x7f5b24115cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-12T05:42:06.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.708+0000 7f5b29e8f700 1 --2- 172.21.15.32:0/2426611288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b24101d70 0x7f5b24115cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-12T05:42:06.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.708+0000 7f5b2968e700 1 -- 172.21.15.32:0/2426611288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b24101d70 msgr2=0x7f5b24115cc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T05:42:06.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.708+0000 7f5b2968e700 1 --2- 172.21.15.32:0/2426611288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b24101d70 0x7f5b24115cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:06.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.708+0000 7f5b2968e700 1 -- 172.21.15.32:0/2426611288 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b20005040 con 0x7f5b24102710 2024-04-12T05:42:06.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.708+0000 7f5b2968e700 1 --2- 172.21.15.32:0/2426611288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b24102710 0x7f5b24116200 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5b18002a80 tx=0x7f5b18004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:06.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.708+0000 7f5b167fc700 1 -- 172.21.15.32:0/2426611288 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b18007cc0 con 0x7f5b24102710 2024-04-12T05:42:06.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.709+0000 7f5b167fc700 1 -- 172.21.15.32:0/2426611288 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b1800f040 con 0x7f5b24102710 2024-04-12T05:42:06.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.709+0000 7f5b2c0f3700 1 -- 172.21.15.32:0/2426611288 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b2410ff00 con 0x7f5b24102710 2024-04-12T05:42:06.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.709+0000 7f5b167fc700 1 -- 172.21.15.32:0/2426611288 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b1800da30 con 0x7f5b24102710 2024-04-12T05:42:06.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.709+0000 7f5b2c0f3700 1 -- 172.21.15.32:0/2426611288 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b241104c0 con 0x7f5b24102710 2024-04-12T05:42:06.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.709+0000 7f5b2c0f3700 1 -- 172.21.15.32:0/2426611288 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b240fd4c0 con 0x7f5b24102710 2024-04-12T05:42:06.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.710+0000 7f5b167fc700 1 -- 172.21.15.32:0/2426611288 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7f5b1800dbd0 con 0x7f5b24102710 2024-04-12T05:42:06.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.710+0000 7f5b167fc700 1 --2- 172.21.15.32:0/2426611288 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5b1006cd40 0x7f5b1006f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:06.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.710+0000 7f5b167fc700 1 -- 172.21.15.32:0/2426611288 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f5b1808b9c0 con 0x7f5b24102710 2024-04-12T05:42:06.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.711+0000 7f5b29e8f700 1 --2- 172.21.15.32:0/2426611288 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5b1006cd40 0x7f5b1006f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:06.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.711+0000 7f5b29e8f700 1 --2- 172.21.15.32:0/2426611288 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5b1006cd40 0x7f5b1006f1f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5b20004f90 tx=0x7f5b20009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:06.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.715+0000 7f5b167fc700 1 -- 172.21.15.32:0/2426611288 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5b18059220 con 0x7f5b24102710 2024-04-12T05:42:06.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.971+0000 7f5b2c0f3700 1 -- 172.21.15.32:0/2426611288 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5b2404bed0 con 0x7f5b24102710 2024-04-12T05:42:06.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.971+0000 7f5b167fc700 1 -- 172.21.15.32:0/2426611288 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1864 (secure 0 0 0) 0x7f5b18058db0 con 0x7f5b24102710 2024-04-12T05:42:06.968 INFO:teuthology.orchestra.run.smithi032.stdout:e22 2024-04-12T05:42:06.968 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:42:06.968 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:42:06.968 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:42:06.968 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:42:06.968 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:42:06.968 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:42:06.968 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 22 2024-04-12T05:42:06.968 INFO:teuthology.orchestra.run.smithi032.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-12T05:42:06.968 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:42:06.968 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:41:05.358509+0000 2024-04-12T05:42:06.969 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:42:06.969 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:42:06.969 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:42:06.969 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:42:06.969 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:42:06.969 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:42:06.969 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:42:06.969 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:42:06.969 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 46 2024-04-12T05:42:06.969 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:42:06.969 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:42:06.969 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:42:06.969 INFO:teuthology.orchestra.run.smithi032.stdout:up {0=14480,1=24281} 2024-04-12T05:42:06.969 INFO:teuthology.orchestra.run.smithi032.stdout:failed 2024-04-12T05:42:06.970 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:42:06.970 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:42:06.970 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:42:06.970 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:42:06.970 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:42:06.970 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:42:06.970 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:42:06.970 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:42:06.970 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{0:14480} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.32:6828/387754149,v1:172.21.15.32:6829/387754149] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:42:06.970 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{0:14490} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3678031004,v1:172.21.15.32:6827/3678031004] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:42:06.970 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{1:24281} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.67:6824/1291914599,v1:172.21.15.67:6825/1291914599] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:42:06.970 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{1:24295} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.67:6826/873395842,v1:172.21.15.67:6827/873395842] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:42:06.970 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:42:06.970 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:42:06.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.973+0000 7f5b2c0f3700 1 -- 172.21.15.32:0/2426611288 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5b1006cd40 msgr2=0x7f5b1006f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:06.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.973+0000 7f5b2c0f3700 1 --2- 172.21.15.32:0/2426611288 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5b1006cd40 0x7f5b1006f1f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5b20004f90 tx=0x7f5b20009e30 comp rx=0 tx=0).stop 2024-04-12T05:42:06.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.973+0000 7f5b2c0f3700 1 -- 172.21.15.32:0/2426611288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b24102710 msgr2=0x7f5b24116200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:06.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.973+0000 7f5b2c0f3700 1 --2- 172.21.15.32:0/2426611288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b24102710 0x7f5b24116200 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5b18002a80 tx=0x7f5b18004630 comp rx=0 tx=0).stop 2024-04-12T05:42:06.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.973+0000 7f5b2c0f3700 1 -- 172.21.15.32:0/2426611288 shutdown_connections 2024-04-12T05:42:06.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.973+0000 7f5b2c0f3700 1 --2- 172.21.15.32:0/2426611288 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5b1006cd40 0x7f5b1006f1f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:06.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.973+0000 7f5b2c0f3700 1 --2- 172.21.15.32:0/2426611288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b24101d70 0x7f5b24115cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:06.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.973+0000 7f5b2c0f3700 1 --2- 172.21.15.32:0/2426611288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b24102710 0x7f5b24116200 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:06.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.973+0000 7f5b2c0f3700 1 -- 172.21.15.32:0/2426611288 >> 172.21.15.32:0/2426611288 conn(0x7f5b24072bc0 msgr2=0x7f5b240659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:06.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.974+0000 7f5b2c0f3700 1 -- 172.21.15.32:0/2426611288 shutdown_connections 2024-04-12T05:42:06.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:06.974+0000 7f5b2c0f3700 1 -- 172.21.15.32:0/2426611288 wait complete. 2024-04-12T05:42:06.972 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 22 2024-04-12T05:42:07.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.078+0000 7f406a2e9700 1 -- 172.21.15.32:0/1243156469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40640fe7a0 msgr2=0x7f4064106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:07.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.078+0000 7f406a2e9700 1 --2- 172.21.15.32:0/1243156469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40640fe7a0 0x7f4064106e70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f404c004940 tx=0x7f404c009e30 comp rx=0 tx=0).stop 2024-04-12T05:42:07.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.078+0000 7f406a2e9700 1 -- 172.21.15.32:0/1243156469 shutdown_connections 2024-04-12T05:42:07.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.078+0000 7f406a2e9700 1 --2- 172.21.15.32:0/1243156469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40640fe7a0 0x7f4064106e70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:07.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.078+0000 7f406a2e9700 1 --2- 172.21.15.32:0/1243156469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40640fde00 0x7f40640fe1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:07.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.078+0000 7f406a2e9700 1 -- 172.21.15.32:0/1243156469 >> 172.21.15.32:0/1243156469 conn(0x7f40640f7880 msgr2=0x7f40640f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:07.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.078+0000 7f406a2e9700 1 -- 172.21.15.32:0/1243156469 shutdown_connections 2024-04-12T05:42:07.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.078+0000 7f406a2e9700 1 -- 172.21.15.32:0/1243156469 wait complete. 2024-04-12T05:42:07.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.079+0000 7f406a2e9700 1 Processor -- start 2024-04-12T05:42:07.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.079+0000 7f406a2e9700 1 -- start start 2024-04-12T05:42:07.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.079+0000 7f406a2e9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40640fde00 0x7f40640fbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:07.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.079+0000 7f406a2e9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40640fe7a0 0x7f40640fc200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:07.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.079+0000 7f406a2e9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40640fc740 con 0x7f40640fde00 2024-04-12T05:42:07.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.079+0000 7f406a2e9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40640fc880 con 0x7f40640fe7a0 2024-04-12T05:42:07.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.079+0000 7f4063fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40640fde00 0x7f40640fbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:07.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.079+0000 7f4063fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40640fde00 0x7f40640fbcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53192/0 (socket says 172.21.15.32:53192) 2024-04-12T05:42:07.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.079+0000 7f4063fff700 1 -- 172.21.15.32:0/1115742798 learned_addr learned my addr 172.21.15.32:0/1115742798 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:42:07.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.079+0000 7f405bfff700 1 --2- 172.21.15.32:0/1115742798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40640fe7a0 0x7f40640fc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:07.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.079+0000 7f405bfff700 1 -- 172.21.15.32:0/1115742798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40640fe7a0 msgr2=0x7f40640fc200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-12T05:42:07.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.079+0000 7f405bfff700 1 -- 172.21.15.32:0/1115742798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40640fe7a0 msgr2=0x7f40640fc200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-12T05:42:07.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.079+0000 7f405bfff700 1 --2- 172.21.15.32:0/1115742798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40640fe7a0 0x7f40640fc200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-12T05:42:07.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.079+0000 7f405bfff700 1 --2- 172.21.15.32:0/1115742798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40640fe7a0 0x7f40640fc200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-12T05:42:07.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.079+0000 7f4063fff700 1 -- 172.21.15.32:0/1115742798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40640fe7a0 msgr2=0x7f40640fc200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T05:42:07.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.079+0000 7f4063fff700 1 --2- 172.21.15.32:0/1115742798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40640fe7a0 0x7f40640fc200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:07.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.079+0000 7f4063fff700 1 -- 172.21.15.32:0/1115742798 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f404c005040 con 0x7f40640fde00 2024-04-12T05:42:07.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.079+0000 7f4063fff700 1 --2- 172.21.15.32:0/1115742798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40640fde00 0x7f40640fbcc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f40540041d0 tx=0x7f40540046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:07.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.080+0000 7f40617fa700 1 -- 172.21.15.32:0/1115742798 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4054007cc0 con 0x7f40640fde00 2024-04-12T05:42:07.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.080+0000 7f40617fa700 1 -- 172.21.15.32:0/1115742798 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f405400f040 con 0x7f40640fde00 2024-04-12T05:42:07.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.080+0000 7f40617fa700 1 -- 172.21.15.32:0/1115742798 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f405400dae0 con 0x7f40640fde00 2024-04-12T05:42:07.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.080+0000 7f406a2e9700 1 -- 172.21.15.32:0/1115742798 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40640fcb30 con 0x7f40640fde00 2024-04-12T05:42:07.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.080+0000 7f406a2e9700 1 -- 172.21.15.32:0/1115742798 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f406411a4a0 con 0x7f40640fde00 2024-04-12T05:42:07.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.081+0000 7f406a2e9700 1 -- 172.21.15.32:0/1115742798 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40640666f0 con 0x7f40640fde00 2024-04-12T05:42:07.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.081+0000 7f40617fa700 1 -- 172.21.15.32:0/1115742798 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7f405400dc40 con 0x7f40640fde00 2024-04-12T05:42:07.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.081+0000 7f40617fa700 1 --2- 172.21.15.32:0/1115742798 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f405006cd40 0x7f405006f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:07.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.082+0000 7f40617fa700 1 -- 172.21.15.32:0/1115742798 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f405408b7d0 con 0x7f40640fde00 2024-04-12T05:42:07.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.082+0000 7f405bfff700 1 --2- 172.21.15.32:0/1115742798 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f405006cd40 0x7f405006f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:07.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.082+0000 7f405bfff700 1 --2- 172.21.15.32:0/1115742798 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f405006cd40 0x7f405006f1f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f40640fd5b0 tx=0x7f404c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:07.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.086+0000 7f40617fa700 1 -- 172.21.15.32:0/1115742798 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4054055640 con 0x7f40640fde00 2024-04-12T05:42:07.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.309+0000 7f406a2e9700 1 -- 172.21.15.32:0/1115742798 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f40640fcb30 con 0x7f405006cd40 2024-04-12T05:42:07.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.311+0000 7f40617fa700 1 -- 172.21.15.32:0/1115742798 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f40640fcb30 con 0x7f405006cd40 2024-04-12T05:42:07.307 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:42:07.307 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:42:07.307 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:42:07.307 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:42:07.307 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:42:07.307 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr" 2024-04-12T05:42:07.307 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:42:07.307 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "3/21 daemons upgraded", 2024-04-12T05:42:07.308 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mon daemons", 2024-04-12T05:42:07.308 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:42:07.308 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:42:07.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.313+0000 7f406a2e9700 1 -- 172.21.15.32:0/1115742798 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f405006cd40 msgr2=0x7f405006f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:07.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.313+0000 7f406a2e9700 1 --2- 172.21.15.32:0/1115742798 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f405006cd40 0x7f405006f1f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f40640fd5b0 tx=0x7f404c009e30 comp rx=0 tx=0).stop 2024-04-12T05:42:07.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.313+0000 7f406a2e9700 1 -- 172.21.15.32:0/1115742798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40640fde00 msgr2=0x7f40640fbcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:07.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.313+0000 7f406a2e9700 1 --2- 172.21.15.32:0/1115742798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40640fde00 0x7f40640fbcc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f40540041d0 tx=0x7f40540046b0 comp rx=0 tx=0).stop 2024-04-12T05:42:07.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.313+0000 7f406a2e9700 1 -- 172.21.15.32:0/1115742798 shutdown_connections 2024-04-12T05:42:07.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.313+0000 7f406a2e9700 1 --2- 172.21.15.32:0/1115742798 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f405006cd40 0x7f405006f1f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:07.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.313+0000 7f406a2e9700 1 --2- 172.21.15.32:0/1115742798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40640fde00 0x7f40640fbcc0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:07.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.313+0000 7f406a2e9700 1 --2- 172.21.15.32:0/1115742798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40640fe7a0 0x7f40640fc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:07.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.313+0000 7f406a2e9700 1 -- 172.21.15.32:0/1115742798 >> 172.21.15.32:0/1115742798 conn(0x7f40640f7880 msgr2=0x7f40640f8ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:07.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.313+0000 7f406a2e9700 1 -- 172.21.15.32:0/1115742798 shutdown_connections 2024-04-12T05:42:07.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.313+0000 7f406a2e9700 1 -- 172.21.15.32:0/1115742798 wait complete. 2024-04-12T05:42:07.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:07 smithi032 ceph-mon[146817]: pgmap v12: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-12T05:42:07.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:07 smithi032 ceph-mon[146817]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:07.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:07 smithi032 ceph-mon[146817]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:07.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:07 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1454546213' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:42:07.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:07 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/2426611288' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:42:07.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.416+0000 7f1fe01e4700 1 -- 172.21.15.32:0/3382320044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fd8101d00 msgr2=0x7f1fd81020d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:07.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.416+0000 7f1fe01e4700 1 --2- 172.21.15.32:0/3382320044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fd8101d00 0x7f1fd81020d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8004940 tx=0x7f1fc8009e30 comp rx=0 tx=0).stop 2024-04-12T05:42:07.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.416+0000 7f1fe01e4700 1 -- 172.21.15.32:0/3382320044 shutdown_connections 2024-04-12T05:42:07.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.416+0000 7f1fe01e4700 1 --2- 172.21.15.32:0/3382320044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fd81026a0 0x7f1fd80fbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:07.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.416+0000 7f1fe01e4700 1 --2- 172.21.15.32:0/3382320044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fd8101d00 0x7f1fd81020d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:07.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.416+0000 7f1fe01e4700 1 -- 172.21.15.32:0/3382320044 >> 172.21.15.32:0/3382320044 conn(0x7f1fd80f78d0 msgr2=0x7f1fd80f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:07.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.416+0000 7f1fe01e4700 1 -- 172.21.15.32:0/3382320044 shutdown_connections 2024-04-12T05:42:07.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.416+0000 7f1fe01e4700 1 -- 172.21.15.32:0/3382320044 wait complete. 2024-04-12T05:42:07.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.416+0000 7f1fe01e4700 1 Processor -- start 2024-04-12T05:42:07.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.416+0000 7f1fe01e4700 1 -- start start 2024-04-12T05:42:07.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.417+0000 7f1fe01e4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fd8101d00 0x7f1fd8102b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:07.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.417+0000 7f1fe01e4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fd81026a0 0x7f1fd81030a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:07.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.417+0000 7f1fe01e4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd8103630 con 0x7f1fd81026a0 2024-04-12T05:42:07.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.417+0000 7f1fe01e4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd8103770 con 0x7f1fd8101d00 2024-04-12T05:42:07.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.417+0000 7f1fdd77f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fd81026a0 0x7f1fd81030a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:07.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.417+0000 7f1fdd77f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fd81026a0 0x7f1fd81030a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53198/0 (socket says 172.21.15.32:53198) 2024-04-12T05:42:07.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.417+0000 7f1fdd77f700 1 -- 172.21.15.32:0/744152471 learned_addr learned my addr 172.21.15.32:0/744152471 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:42:07.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.417+0000 7f1fddf80700 1 --2- 172.21.15.32:0/744152471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fd8101d00 0x7f1fd8102b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:07.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.417+0000 7f1fddf80700 1 -- 172.21.15.32:0/744152471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fd8101d00 msgr2=0x7f1fd8102b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-12T05:42:07.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.417+0000 7f1fddf80700 1 -- 172.21.15.32:0/744152471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fd8101d00 msgr2=0x7f1fd8102b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-12T05:42:07.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.417+0000 7f1fddf80700 1 --2- 172.21.15.32:0/744152471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fd8101d00 0x7f1fd8102b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-12T05:42:07.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.417+0000 7f1fddf80700 1 --2- 172.21.15.32:0/744152471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fd8101d00 0x7f1fd8102b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-12T05:42:07.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.417+0000 7f1fdd77f700 1 -- 172.21.15.32:0/744152471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fd8101d00 msgr2=0x7f1fd8102b60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T05:42:07.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.417+0000 7f1fdd77f700 1 --2- 172.21.15.32:0/744152471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fd8101d00 0x7f1fd8102b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:07.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.417+0000 7f1fdd77f700 1 -- 172.21.15.32:0/744152471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc8005040 con 0x7f1fd81026a0 2024-04-12T05:42:07.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.417+0000 7f1fdd77f700 1 --2- 172.21.15.32:0/744152471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fd81026a0 0x7f1fd81030a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4002a80 tx=0x7f1fd4004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:07.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.417+0000 7f1fce7fc700 1 -- 172.21.15.32:0/744152471 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fd4007cc0 con 0x7f1fd81026a0 2024-04-12T05:42:07.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.417+0000 7f1fce7fc700 1 -- 172.21.15.32:0/744152471 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1fd400f040 con 0x7f1fd81026a0 2024-04-12T05:42:07.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.418+0000 7f1fe01e4700 1 -- 172.21.15.32:0/744152471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fd8119e10 con 0x7f1fd81026a0 2024-04-12T05:42:07.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.418+0000 7f1fce7fc700 1 -- 172.21.15.32:0/744152471 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fd400da30 con 0x7f1fd81026a0 2024-04-12T05:42:07.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.418+0000 7f1fe01e4700 1 -- 172.21.15.32:0/744152471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd811a290 con 0x7f1fd81026a0 2024-04-12T05:42:07.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.419+0000 7f1fe01e4700 1 -- 172.21.15.32:0/744152471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fd80fd450 con 0x7f1fd81026a0 2024-04-12T05:42:07.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.419+0000 7f1fce7fc700 1 -- 172.21.15.32:0/744152471 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7f1fd400dbd0 con 0x7f1fd81026a0 2024-04-12T05:42:07.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.419+0000 7f1fce7fc700 1 --2- 172.21.15.32:0/744152471 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1fc406cd40 0x7f1fc406f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:07.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.419+0000 7f1fce7fc700 1 -- 172.21.15.32:0/744152471 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f1fd408b8d0 con 0x7f1fd81026a0 2024-04-12T05:42:07.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.419+0000 7f1fddf80700 1 --2- 172.21.15.32:0/744152471 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1fc406cd40 0x7f1fc406f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:07.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.420+0000 7f1fddf80700 1 --2- 172.21.15.32:0/744152471 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1fc406cd40 0x7f1fc406f1f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8004f90 tx=0x7f1fc8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:07.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.424+0000 7f1fce7fc700 1 -- 172.21.15.32:0/744152471 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1fd4055740 con 0x7f1fd81026a0 2024-04-12T05:42:07.435 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:07 smithi067 ceph-mon[112979]: pgmap v12: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-12T05:42:07.435 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:07 smithi067 ceph-mon[112979]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:07.435 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:07 smithi067 ceph-mon[112979]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:07.435 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:07 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1454546213' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:42:07.435 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:07 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/2426611288' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:42:07.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.711+0000 7f1fe01e4700 1 -- 172.21.15.32:0/744152471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1fd805f5f0 con 0x7f1fd81026a0 2024-04-12T05:42:07.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.712+0000 7f1fce7fc700 1 -- 172.21.15.32:0/744152471 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f1fd4058d60 con 0x7f1fd81026a0 2024-04-12T05:42:07.708 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_OK 2024-04-12T05:42:07.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.714+0000 7f1fe01e4700 1 -- 172.21.15.32:0/744152471 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1fc406cd40 msgr2=0x7f1fc406f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:07.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.714+0000 7f1fe01e4700 1 --2- 172.21.15.32:0/744152471 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1fc406cd40 0x7f1fc406f1f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8004f90 tx=0x7f1fc8009e30 comp rx=0 tx=0).stop 2024-04-12T05:42:07.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.714+0000 7f1fe01e4700 1 -- 172.21.15.32:0/744152471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fd81026a0 msgr2=0x7f1fd81030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:07.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.714+0000 7f1fe01e4700 1 --2- 172.21.15.32:0/744152471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fd81026a0 0x7f1fd81030a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4002a80 tx=0x7f1fd4004630 comp rx=0 tx=0).stop 2024-04-12T05:42:07.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.714+0000 7f1fe01e4700 1 -- 172.21.15.32:0/744152471 shutdown_connections 2024-04-12T05:42:07.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.714+0000 7f1fe01e4700 1 --2- 172.21.15.32:0/744152471 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1fc406cd40 0x7f1fc406f1f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:07.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.714+0000 7f1fe01e4700 1 --2- 172.21.15.32:0/744152471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fd8101d00 0x7f1fd8102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:07.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.714+0000 7f1fe01e4700 1 --2- 172.21.15.32:0/744152471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fd81026a0 0x7f1fd81030a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:07.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.714+0000 7f1fe01e4700 1 -- 172.21.15.32:0/744152471 >> 172.21.15.32:0/744152471 conn(0x7f1fd80f78d0 msgr2=0x7f1fd80f9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:07.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.714+0000 7f1fe01e4700 1 -- 172.21.15.32:0/744152471 shutdown_connections 2024-04-12T05:42:07.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:07.714+0000 7f1fe01e4700 1 -- 172.21.15.32:0/744152471 wait complete. 2024-04-12T05:42:08.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:08 smithi032 ceph-mon[146817]: from='client.34148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:08.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:08 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/744152471' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:42:08.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:08 smithi067 ceph-mon[112979]: from='client.34148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:08.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:08 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/744152471' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:42:09.454 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:09 smithi067 ceph-mon[112979]: pgmap v13: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:09.454 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:09.454 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:09.455 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:09.455 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:09.455 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:09 smithi032 ceph-mon[146817]: pgmap v13: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:09.455 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:09.456 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:09.456 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:09.456 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:11.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:11 smithi067 ceph-mon[112979]: pgmap v14: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:42:11.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:11 smithi032 ceph-mon[146817]: pgmap v14: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:42:13.760 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:13 smithi032 ceph-mon[146817]: pgmap v15: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:13.760 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:13 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:13.760 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:13 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:13.760 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:13 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:13.760 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:13 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:42:13.760 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:13 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:13.760 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:13 smithi032 ceph-mon[146817]: Reconfiguring mon.smithi032 (monmap changed)... 2024-04-12T05:42:13.761 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:13 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T05:42:13.761 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:13 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T05:42:13.761 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:13 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:13.761 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:13 smithi032 ceph-mon[146817]: Reconfiguring daemon mon.smithi032 on smithi032 2024-04-12T05:42:14.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:13 smithi067 ceph-mon[112979]: pgmap v15: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:14.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:13 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:14.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:13 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:14.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:13 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:14.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:13 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:42:14.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:13 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:14.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:13 smithi067 ceph-mon[112979]: Reconfiguring mon.smithi032 (monmap changed)... 2024-04-12T05:42:14.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:13 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T05:42:14.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:13 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T05:42:14.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:13 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:14.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:13 smithi067 ceph-mon[112979]: Reconfiguring daemon mon.smithi032 on smithi032 2024-04-12T05:42:15.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:15 smithi067 ceph-mon[112979]: pgmap v16: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:42:15.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:15 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:15.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:15 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:15.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:15 smithi067 ceph-mon[112979]: Reconfiguring mgr.smithi032.hrwxrh (monmap changed)... 2024-04-12T05:42:15.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:15 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi032.hrwxrh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:42:15.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:15 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:42:15.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:15 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:15.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:15 smithi067 ceph-mon[112979]: Reconfiguring daemon mgr.smithi032.hrwxrh on smithi032 2024-04-12T05:42:15.556 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:15 smithi032 ceph-mon[146817]: pgmap v16: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:42:15.556 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:15 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:15.556 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:15 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:15.556 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:15 smithi032 ceph-mon[146817]: Reconfiguring mgr.smithi032.hrwxrh (monmap changed)... 2024-04-12T05:42:15.556 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:15 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi032.hrwxrh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:42:15.556 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:15 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:42:15.557 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:15 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:15.557 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:15 smithi032 ceph-mon[146817]: Reconfiguring daemon mgr.smithi032.hrwxrh on smithi032 2024-04-12T05:42:17.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:16 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:17.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:16 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:17.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:16 smithi067 ceph-mon[112979]: Reconfiguring crash.smithi032 (monmap changed)... 2024-04-12T05:42:17.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:16 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi032", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-12T05:42:17.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:16 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:17.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:16 smithi067 ceph-mon[112979]: Reconfiguring daemon crash.smithi032 on smithi032 2024-04-12T05:42:17.038 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:16 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:17.038 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:16 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:17.038 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:16 smithi032 ceph-mon[146817]: Reconfiguring crash.smithi032 (monmap changed)... 2024-04-12T05:42:17.038 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:16 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi032", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-12T05:42:17.038 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:16 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:17.038 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:16 smithi032 ceph-mon[146817]: Reconfiguring daemon crash.smithi032 on smithi032 2024-04-12T05:42:17.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:17 smithi032 ceph-mon[146817]: pgmap v17: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:17.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:17 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:17.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:17 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:17.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:17 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-12T05:42:17.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:17 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:18.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:17 smithi067 ceph-mon[112979]: pgmap v17: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:18.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:17 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:18.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:17 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:18.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:17 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-12T05:42:18.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:17 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:18.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:18 smithi032 ceph-mon[146817]: Reconfiguring osd.0 (monmap changed)... 2024-04-12T05:42:18.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:18 smithi032 ceph-mon[146817]: Reconfiguring daemon osd.0 on smithi032 2024-04-12T05:42:18.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:42:19.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:18 smithi067 ceph-mon[112979]: Reconfiguring osd.0 (monmap changed)... 2024-04-12T05:42:19.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:18 smithi067 ceph-mon[112979]: Reconfiguring daemon osd.0 on smithi032 2024-04-12T05:42:19.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:42:20.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:19 smithi067 ceph-mon[112979]: pgmap v18: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:20.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:19 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:20.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:19 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:20.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:19 smithi067 ceph-mon[112979]: Reconfiguring osd.1 (monmap changed)... 2024-04-12T05:42:20.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:19 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-12T05:42:20.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:19 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:20.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:19 smithi032 ceph-mon[146817]: pgmap v18: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:20.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:19 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:20.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:19 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:20.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:19 smithi032 ceph-mon[146817]: Reconfiguring osd.1 (monmap changed)... 2024-04-12T05:42:20.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:19 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-12T05:42:20.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:19 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:20.970 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:20 smithi032 ceph-mon[146817]: Reconfiguring daemon osd.1 on smithi032 2024-04-12T05:42:20.971 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:20 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:20.971 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:20 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:20.971 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:20 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-12T05:42:20.971 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:20 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:21.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:20 smithi067 ceph-mon[112979]: Reconfiguring daemon osd.1 on smithi032 2024-04-12T05:42:21.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:20 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:21.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:20 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:21.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:20 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-12T05:42:21.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:20 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:22.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:21 smithi067 ceph-mon[112979]: pgmap v19: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:42:22.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:21 smithi067 ceph-mon[112979]: Reconfiguring osd.2 (monmap changed)... 2024-04-12T05:42:22.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:21 smithi067 ceph-mon[112979]: Reconfiguring daemon osd.2 on smithi032 2024-04-12T05:42:22.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:21 smithi032 ceph-mon[146817]: pgmap v19: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:42:22.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:21 smithi032 ceph-mon[146817]: Reconfiguring osd.2 (monmap changed)... 2024-04-12T05:42:22.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:21 smithi032 ceph-mon[146817]: Reconfiguring daemon osd.2 on smithi032 2024-04-12T05:42:23.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:22 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:23.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:22 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:23.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:22 smithi067 ceph-mon[112979]: Reconfiguring mds.cephfs.smithi032.jehaaj (monmap changed)... 2024-04-12T05:42:23.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:22 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.jehaaj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:42:23.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:22 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:23.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:22 smithi067 ceph-mon[112979]: Reconfiguring daemon mds.cephfs.smithi032.jehaaj on smithi032 2024-04-12T05:42:23.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:22 smithi067 ceph-mon[112979]: pgmap v20: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:23.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:22 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:23.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:22 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:23.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:22 smithi032 ceph-mon[146817]: Reconfiguring mds.cephfs.smithi032.jehaaj (monmap changed)... 2024-04-12T05:42:23.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:22 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.jehaaj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:42:23.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:22 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:23.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:22 smithi032 ceph-mon[146817]: Reconfiguring daemon mds.cephfs.smithi032.jehaaj on smithi032 2024-04-12T05:42:23.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:22 smithi032 ceph-mon[146817]: pgmap v20: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:24.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:24 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:24.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:24 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:24.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:24 smithi067 ceph-mon[112979]: Reconfiguring mds.cephfs.smithi032.pndtry (monmap changed)... 2024-04-12T05:42:24.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:24 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.pndtry", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:42:24.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:24 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:24.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:24 smithi067 ceph-mon[112979]: Reconfiguring daemon mds.cephfs.smithi032.pndtry on smithi032 2024-04-12T05:42:24.572 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:24 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:24.572 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:24 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:24.572 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:24 smithi032 ceph-mon[146817]: Reconfiguring mds.cephfs.smithi032.pndtry (monmap changed)... 2024-04-12T05:42:24.572 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:24 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.pndtry", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:42:24.572 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:24 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:24.572 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:24 smithi032 ceph-mon[146817]: Reconfiguring daemon mds.cephfs.smithi032.pndtry on smithi032 2024-04-12T05:42:25.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:25 smithi067 ceph-mon[112979]: pgmap v21: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:42:25.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:25 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:25.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:25 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:25.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:25 smithi067 ceph-mon[112979]: Reconfiguring crash.smithi067 (monmap changed)... 2024-04-12T05:42:25.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:25 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi067", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-12T05:42:25.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:25 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:25.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:25 smithi067 ceph-mon[112979]: Reconfiguring daemon crash.smithi067 on smithi067 2024-04-12T05:42:26.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:25 smithi032 ceph-mon[146817]: pgmap v21: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:42:26.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:25 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:26.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:25 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:26.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:25 smithi032 ceph-mon[146817]: Reconfiguring crash.smithi067 (monmap changed)... 2024-04-12T05:42:26.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:25 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi067", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-12T05:42:26.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:25 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:26.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:25 smithi032 ceph-mon[146817]: Reconfiguring daemon crash.smithi067 on smithi067 2024-04-12T05:42:27.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:27 smithi067 ceph-mon[112979]: pgmap v22: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:27.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:27 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:27.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:27 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:27.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:27 smithi067 ceph-mon[112979]: Reconfiguring mgr.smithi067.dwichg (monmap changed)... 2024-04-12T05:42:27.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:27 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi067.dwichg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:42:27.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:27 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:42:27.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:27 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:27.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:27 smithi067 ceph-mon[112979]: Reconfiguring daemon mgr.smithi067.dwichg on smithi067 2024-04-12T05:42:27.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:27 smithi032 ceph-mon[146817]: pgmap v22: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:27.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:27 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:27.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:27 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:27.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:27 smithi032 ceph-mon[146817]: Reconfiguring mgr.smithi067.dwichg (monmap changed)... 2024-04-12T05:42:27.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:27 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi067.dwichg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:42:27.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:27 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:42:27.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:27 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:27.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:27 smithi032 ceph-mon[146817]: Reconfiguring daemon mgr.smithi067.dwichg on smithi067 2024-04-12T05:42:29.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:28 smithi067 ceph-mon[112979]: pgmap v23: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:29.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:28 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:29.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:28 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:29.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:28 smithi067 ceph-mon[112979]: Reconfiguring mon.smithi067 (monmap changed)... 2024-04-12T05:42:29.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:28 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T05:42:29.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:28 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T05:42:29.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:28 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:29.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:28 smithi067 ceph-mon[112979]: Reconfiguring daemon mon.smithi067 on smithi067 2024-04-12T05:42:29.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:28 smithi032 ceph-mon[146817]: pgmap v23: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:29.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:28 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:29.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:28 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:29.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:28 smithi032 ceph-mon[146817]: Reconfiguring mon.smithi067 (monmap changed)... 2024-04-12T05:42:29.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:28 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T05:42:29.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:28 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T05:42:29.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:28 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:29.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:28 smithi032 ceph-mon[146817]: Reconfiguring daemon mon.smithi067 on smithi067 2024-04-12T05:42:30.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:30 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:30.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:30 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:30.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:30 smithi067 ceph-mon[112979]: Reconfiguring osd.3 (monmap changed)... 2024-04-12T05:42:30.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:30 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-12T05:42:30.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:30 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:30.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:30 smithi067 ceph-mon[112979]: Reconfiguring daemon osd.3 on smithi067 2024-04-12T05:42:30.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:30 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:30.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:30 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:30.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:30 smithi032 ceph-mon[146817]: Reconfiguring osd.3 (monmap changed)... 2024-04-12T05:42:30.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:30 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-12T05:42:30.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:30 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:30.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:30 smithi032 ceph-mon[146817]: Reconfiguring daemon osd.3 on smithi067 2024-04-12T05:42:31.423 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:31 smithi067 ceph-mon[112979]: pgmap v24: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:42:31.423 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:31 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:31.715 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:31 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:31.715 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:31 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-12T05:42:31.715 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:31 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:31.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:31 smithi032 ceph-mon[146817]: pgmap v24: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:42:31.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:31 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:31.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:31 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:31.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:31 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-12T05:42:31.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:31 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:32.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:32 smithi067 ceph-mon[112979]: Reconfiguring osd.4 (monmap changed)... 2024-04-12T05:42:32.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:32 smithi067 ceph-mon[112979]: Reconfiguring daemon osd.4 on smithi067 2024-04-12T05:42:32.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:32 smithi032 ceph-mon[146817]: Reconfiguring osd.4 (monmap changed)... 2024-04-12T05:42:32.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:32 smithi032 ceph-mon[146817]: Reconfiguring daemon osd.4 on smithi067 2024-04-12T05:42:34.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:33 smithi067 ceph-mon[112979]: pgmap v25: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:34.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:33 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:34.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:33 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:34.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:33 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-12T05:42:34.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:33 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:34.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:33 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:42:34.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:33 smithi032 ceph-mon[146817]: pgmap v25: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:34.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:33 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:34.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:33 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:34.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:33 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-12T05:42:34.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:33 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:34.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:33 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:42:34.995 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:34 smithi067 ceph-mon[112979]: Reconfiguring osd.5 (monmap changed)... 2024-04-12T05:42:34.995 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:34 smithi067 ceph-mon[112979]: Reconfiguring daemon osd.5 on smithi067 2024-04-12T05:42:34.995 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:34 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:34.995 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:34 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:34.995 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:34 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.zesmgy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:42:34.995 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:34 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:35.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:34 smithi032 ceph-mon[146817]: Reconfiguring osd.5 (monmap changed)... 2024-04-12T05:42:35.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:34 smithi032 ceph-mon[146817]: Reconfiguring daemon osd.5 on smithi067 2024-04-12T05:42:35.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:34 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:35.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:34 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:35.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:34 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.zesmgy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:42:35.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:34 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:36.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:35 smithi067 ceph-mon[112979]: pgmap v26: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:42:36.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:35 smithi067 ceph-mon[112979]: Reconfiguring mds.cephfs.smithi067.zesmgy (monmap changed)... 2024-04-12T05:42:36.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:35 smithi067 ceph-mon[112979]: Reconfiguring daemon mds.cephfs.smithi067.zesmgy on smithi067 2024-04-12T05:42:36.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:35 smithi032 ceph-mon[146817]: pgmap v26: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:42:36.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:35 smithi032 ceph-mon[146817]: Reconfiguring mds.cephfs.smithi067.zesmgy (monmap changed)... 2024-04-12T05:42:36.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:35 smithi032 ceph-mon[146817]: Reconfiguring daemon mds.cephfs.smithi067.zesmgy on smithi067 2024-04-12T05:42:37.004 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:36 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:37.004 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:36 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:37.005 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:36 smithi067 ceph-mon[112979]: Reconfiguring mds.cephfs.smithi067.ymvqih (monmap changed)... 2024-04-12T05:42:37.005 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:36 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.ymvqih", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:42:37.005 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:36 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:37.005 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:36 smithi067 ceph-mon[112979]: Reconfiguring daemon mds.cephfs.smithi067.ymvqih on smithi067 2024-04-12T05:42:37.005 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:36 smithi067 ceph-mon[112979]: pgmap v27: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:37.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:36 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:37.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:36 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:37.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:36 smithi032 ceph-mon[146817]: Reconfiguring mds.cephfs.smithi067.ymvqih (monmap changed)... 2024-04-12T05:42:37.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:36 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.ymvqih", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:42:37.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:36 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:37.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:36 smithi032 ceph-mon[146817]: Reconfiguring daemon mds.cephfs.smithi067.ymvqih on smithi067 2024-04-12T05:42:37.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:36 smithi032 ceph-mon[146817]: pgmap v27: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:37.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.828+0000 7f3e0b458700 1 -- 172.21.15.32:0/2940728868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e04102e00 msgr2=0x7f3e041031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:37.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.828+0000 7f3e0b458700 1 --2- 172.21.15.32:0/2940728868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e04102e00 0x7f3e041031d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3df8004940 tx=0x7f3df8009e30 comp rx=0 tx=0).stop 2024-04-12T05:42:37.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.828+0000 7f3e0b458700 1 -- 172.21.15.32:0/2940728868 shutdown_connections 2024-04-12T05:42:37.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.828+0000 7f3e0b458700 1 --2- 172.21.15.32:0/2940728868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e041037a0 0x7f3e040fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:37.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.828+0000 7f3e0b458700 1 --2- 172.21.15.32:0/2940728868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e04102e00 0x7f3e041031d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:37.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.828+0000 7f3e0b458700 1 -- 172.21.15.32:0/2940728868 >> 172.21.15.32:0/2940728868 conn(0x7f3e040f89b0 msgr2=0x7f3e040fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:37.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.828+0000 7f3e0b458700 1 -- 172.21.15.32:0/2940728868 shutdown_connections 2024-04-12T05:42:37.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.829+0000 7f3e0b458700 1 -- 172.21.15.32:0/2940728868 wait complete. 2024-04-12T05:42:37.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.829+0000 7f3e0b458700 1 Processor -- start 2024-04-12T05:42:37.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.829+0000 7f3e0b458700 1 -- start start 2024-04-12T05:42:37.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.829+0000 7f3e0b458700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e04102e00 0x7f3e04113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:37.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.829+0000 7f3e0b458700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e041037a0 0x7f3e04114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:37.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.829+0000 7f3e0b458700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e041146b0 con 0x7f3e04102e00 2024-04-12T05:42:37.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.829+0000 7f3e0b458700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0410daa0 con 0x7f3e041037a0 2024-04-12T05:42:37.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.829+0000 7f3e091f4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e04102e00 0x7f3e04113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:37.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.829+0000 7f3e091f4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e04102e00 0x7f3e04113b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41106/0 (socket says 172.21.15.32:41106) 2024-04-12T05:42:37.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.829+0000 7f3e091f4700 1 -- 172.21.15.32:0/3278261288 learned_addr learned my addr 172.21.15.32:0/3278261288 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:42:37.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.830+0000 7f3e091f4700 1 -- 172.21.15.32:0/3278261288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e041037a0 msgr2=0x7f3e04114060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:37.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.830+0000 7f3e089f3700 1 --2- 172.21.15.32:0/3278261288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e041037a0 0x7f3e04114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:37.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.830+0000 7f3e091f4700 1 --2- 172.21.15.32:0/3278261288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e041037a0 0x7f3e04114060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:37.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.830+0000 7f3e091f4700 1 -- 172.21.15.32:0/3278261288 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df8005040 con 0x7f3e04102e00 2024-04-12T05:42:37.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.830+0000 7f3e089f3700 1 --2- 172.21.15.32:0/3278261288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e041037a0 0x7f3e04114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:42:37.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.830+0000 7f3e091f4700 1 --2- 172.21.15.32:0/3278261288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e04102e00 0x7f3e04113b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3df8009360 tx=0x7f3df8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:37.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.830+0000 7f3df5ffb700 1 -- 172.21.15.32:0/3278261288 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3df800e040 con 0x7f3e04102e00 2024-04-12T05:42:37.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.830+0000 7f3e0b458700 1 -- 172.21.15.32:0/3278261288 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e0410dd20 con 0x7f3e04102e00 2024-04-12T05:42:37.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.830+0000 7f3df5ffb700 1 -- 172.21.15.32:0/3278261288 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3df8013420 con 0x7f3e04102e00 2024-04-12T05:42:37.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.830+0000 7f3df5ffb700 1 -- 172.21.15.32:0/3278261288 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3df801c5f0 con 0x7f3e04102e00 2024-04-12T05:42:37.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.830+0000 7f3e0b458700 1 -- 172.21.15.32:0/3278261288 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e0410e2e0 con 0x7f3e04102e00 2024-04-12T05:42:37.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.831+0000 7f3e0b458700 1 -- 172.21.15.32:0/3278261288 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e040fe550 con 0x7f3e04102e00 2024-04-12T05:42:37.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.832+0000 7f3df5ffb700 1 -- 172.21.15.32:0/3278261288 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7f3df8013590 con 0x7f3e04102e00 2024-04-12T05:42:37.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.832+0000 7f3df5ffb700 1 --2- 172.21.15.32:0/3278261288 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3df006cd40 0x7f3df006f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:37.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.832+0000 7f3df5ffb700 1 -- 172.21.15.32:0/3278261288 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f3df808d4e0 con 0x7f3e04102e00 2024-04-12T05:42:37.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.832+0000 7f3e089f3700 1 --2- 172.21.15.32:0/3278261288 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3df006cd40 0x7f3df006f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:37.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.833+0000 7f3e089f3700 1 --2- 172.21.15.32:0/3278261288 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3df006cd40 0x7f3df006f1f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3e00007fd0 tx=0x7f3e00009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:37.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:37.837+0000 7f3df5ffb700 1 -- 172.21.15.32:0/3278261288 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3df805ae90 con 0x7f3e04102e00 2024-04-12T05:42:38.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.060+0000 7f3e0b458700 1 -- 172.21.15.32:0/3278261288 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e04109dc0 con 0x7f3df006cd40 2024-04-12T05:42:38.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.061+0000 7f3df5ffb700 1 -- 172.21.15.32:0/3278261288 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7f3e04109dc0 con 0x7f3df006cd40 2024-04-12T05:42:38.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.063+0000 7f3e0b458700 1 -- 172.21.15.32:0/3278261288 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3df006cd40 msgr2=0x7f3df006f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:38.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.063+0000 7f3e0b458700 1 --2- 172.21.15.32:0/3278261288 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3df006cd40 0x7f3df006f1f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3e00007fd0 tx=0x7f3e00009040 comp rx=0 tx=0).stop 2024-04-12T05:42:38.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.064+0000 7f3e0b458700 1 -- 172.21.15.32:0/3278261288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e04102e00 msgr2=0x7f3e04113b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:38.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.064+0000 7f3e0b458700 1 --2- 172.21.15.32:0/3278261288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e04102e00 0x7f3e04113b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3df8009360 tx=0x7f3df8009e30 comp rx=0 tx=0).stop 2024-04-12T05:42:38.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.064+0000 7f3e0b458700 1 -- 172.21.15.32:0/3278261288 shutdown_connections 2024-04-12T05:42:38.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.064+0000 7f3e0b458700 1 --2- 172.21.15.32:0/3278261288 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3df006cd40 0x7f3df006f1f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:38.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.064+0000 7f3e0b458700 1 --2- 172.21.15.32:0/3278261288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e04102e00 0x7f3e04113b20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:38.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.064+0000 7f3e0b458700 1 --2- 172.21.15.32:0/3278261288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e041037a0 0x7f3e04114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:38.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.064+0000 7f3e0b458700 1 -- 172.21.15.32:0/3278261288 >> 172.21.15.32:0/3278261288 conn(0x7f3e040f89b0 msgr2=0x7f3e040f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:38.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.064+0000 7f3e0b458700 1 -- 172.21.15.32:0/3278261288 shutdown_connections 2024-04-12T05:42:38.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.064+0000 7f3e0b458700 1 -- 172.21.15.32:0/3278261288 wait complete. 2024-04-12T05:42:38.071 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:42:38.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.172+0000 7f48937bd700 1 -- 172.21.15.32:0/3727610616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f488c0fbc10 msgr2=0x7f488c0fbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:38.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.172+0000 7f48937bd700 1 --2- 172.21.15.32:0/3727610616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f488c0fbc10 0x7f488c0fbfe0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4888004940 tx=0x7f4888009e30 comp rx=0 tx=0).stop 2024-04-12T05:42:38.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.172+0000 7f48937bd700 1 -- 172.21.15.32:0/3727610616 shutdown_connections 2024-04-12T05:42:38.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.172+0000 7f48937bd700 1 --2- 172.21.15.32:0/3727610616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f488c0fc5b0 0x7f488c100bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:38.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.172+0000 7f48937bd700 1 --2- 172.21.15.32:0/3727610616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f488c0fbc10 0x7f488c0fbfe0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:38.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.172+0000 7f48937bd700 1 -- 172.21.15.32:0/3727610616 >> 172.21.15.32:0/3727610616 conn(0x7f488c0f7850 msgr2=0x7f488c0f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:38.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.172+0000 7f48937bd700 1 -- 172.21.15.32:0/3727610616 shutdown_connections 2024-04-12T05:42:38.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.172+0000 7f48937bd700 1 -- 172.21.15.32:0/3727610616 wait complete. 2024-04-12T05:42:38.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.172+0000 7f48937bd700 1 Processor -- start 2024-04-12T05:42:38.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.173+0000 7f48937bd700 1 -- start start 2024-04-12T05:42:38.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.173+0000 7f48937bd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f488c0fbc10 0x7f488c119f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:38.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.173+0000 7f48937bd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f488c0fc5b0 0x7f488c11a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:38.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.173+0000 7f48937bd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488c11aae0 con 0x7f488c0fbc10 2024-04-12T05:42:38.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.173+0000 7f48937bd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488c1140c0 con 0x7f488c0fc5b0 2024-04-12T05:42:38.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.173+0000 7f4891559700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f488c0fbc10 0x7f488c119f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:38.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.173+0000 7f4891559700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f488c0fbc10 0x7f488c119f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41120/0 (socket says 172.21.15.32:41120) 2024-04-12T05:42:38.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.173+0000 7f4891559700 1 -- 172.21.15.32:0/1069199891 learned_addr learned my addr 172.21.15.32:0/1069199891 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:42:38.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.173+0000 7f4891559700 1 -- 172.21.15.32:0/1069199891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f488c0fc5b0 msgr2=0x7f488c11a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:38.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.173+0000 7f4891559700 1 --2- 172.21.15.32:0/1069199891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f488c0fc5b0 0x7f488c11a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:38.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.173+0000 7f4891559700 1 -- 172.21.15.32:0/1069199891 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4888005040 con 0x7f488c0fbc10 2024-04-12T05:42:38.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.173+0000 7f4891559700 1 --2- 172.21.15.32:0/1069199891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f488c0fbc10 0x7f488c119f50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f488800b3b0 tx=0x7f4888009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:38.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.173+0000 7f4881ffb700 1 -- 172.21.15.32:0/1069199891 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f488800e040 con 0x7f488c0fbc10 2024-04-12T05:42:38.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.173+0000 7f4881ffb700 1 -- 172.21.15.32:0/1069199891 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4888013420 con 0x7f488c0fbc10 2024-04-12T05:42:38.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.173+0000 7f48937bd700 1 -- 172.21.15.32:0/1069199891 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f488c114340 con 0x7f488c0fbc10 2024-04-12T05:42:38.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.174+0000 7f4881ffb700 1 -- 172.21.15.32:0/1069199891 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f488801c630 con 0x7f488c0fbc10 2024-04-12T05:42:38.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.174+0000 7f48937bd700 1 -- 172.21.15.32:0/1069199891 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f488c114880 con 0x7f488c0fbc10 2024-04-12T05:42:38.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.175+0000 7f48937bd700 1 -- 172.21.15.32:0/1069199891 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f488c0fe2f0 con 0x7f488c0fbc10 2024-04-12T05:42:38.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.175+0000 7f4881ffb700 1 -- 172.21.15.32:0/1069199891 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7f4888013970 con 0x7f488c0fbc10 2024-04-12T05:42:38.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.176+0000 7f4881ffb700 1 --2- 172.21.15.32:0/1069199891 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f487806cd50 0x7f487806f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:38.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.176+0000 7f4881ffb700 1 -- 172.21.15.32:0/1069199891 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f488808d4e0 con 0x7f488c0fbc10 2024-04-12T05:42:38.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.178+0000 7f4890d58700 1 --2- 172.21.15.32:0/1069199891 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f487806cd50 0x7f487806f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:38.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.178+0000 7f4890d58700 1 --2- 172.21.15.32:0/1069199891 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f487806cd50 0x7f487806f200 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f487c009b80 tx=0x7f487c009d60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:38.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.181+0000 7f4881ffb700 1 -- 172.21.15.32:0/1069199891 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f488805ade0 con 0x7f488c0fbc10 2024-04-12T05:42:38.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.419+0000 7f48937bd700 1 -- 172.21.15.32:0/1069199891 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f488c1103e0 con 0x7f487806cd50 2024-04-12T05:42:38.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.420+0000 7f4881ffb700 1 -- 172.21.15.32:0/1069199891 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7f488c1103e0 con 0x7f487806cd50 2024-04-12T05:42:38.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.422+0000 7f48937bd700 1 -- 172.21.15.32:0/1069199891 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f487806cd50 msgr2=0x7f487806f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:38.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.422+0000 7f48937bd700 1 --2- 172.21.15.32:0/1069199891 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f487806cd50 0x7f487806f200 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f487c009b80 tx=0x7f487c009d60 comp rx=0 tx=0).stop 2024-04-12T05:42:38.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.422+0000 7f48937bd700 1 -- 172.21.15.32:0/1069199891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f488c0fbc10 msgr2=0x7f488c119f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:38.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.422+0000 7f48937bd700 1 --2- 172.21.15.32:0/1069199891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f488c0fbc10 0x7f488c119f50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f488800b3b0 tx=0x7f4888009e30 comp rx=0 tx=0).stop 2024-04-12T05:42:38.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.422+0000 7f48937bd700 1 -- 172.21.15.32:0/1069199891 shutdown_connections 2024-04-12T05:42:38.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.422+0000 7f48937bd700 1 --2- 172.21.15.32:0/1069199891 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f487806cd50 0x7f487806f200 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:38.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.422+0000 7f48937bd700 1 --2- 172.21.15.32:0/1069199891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f488c0fbc10 0x7f488c119f50 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:38.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.422+0000 7f48937bd700 1 --2- 172.21.15.32:0/1069199891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f488c0fc5b0 0x7f488c11a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:38.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.422+0000 7f48937bd700 1 -- 172.21.15.32:0/1069199891 >> 172.21.15.32:0/1069199891 conn(0x7f488c0f7850 msgr2=0x7f488c0f8e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:38.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.422+0000 7f48937bd700 1 -- 172.21.15.32:0/1069199891 shutdown_connections 2024-04-12T05:42:38.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.422+0000 7f48937bd700 1 -- 172.21.15.32:0/1069199891 wait complete. 2024-04-12T05:42:38.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.536+0000 7fe2a7f28700 1 -- 172.21.15.32:0/3674709350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2a01037a0 msgr2=0x7fe2a00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:38.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.536+0000 7fe2a7f28700 1 --2- 172.21.15.32:0/3674709350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2a01037a0 0x7fe2a00fce10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe294004940 tx=0x7fe294009e30 comp rx=0 tx=0).stop 2024-04-12T05:42:38.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.536+0000 7fe2a7f28700 1 -- 172.21.15.32:0/3674709350 shutdown_connections 2024-04-12T05:42:38.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.536+0000 7fe2a7f28700 1 --2- 172.21.15.32:0/3674709350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2a01037a0 0x7fe2a00fce10 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:38.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.536+0000 7fe2a7f28700 1 --2- 172.21.15.32:0/3674709350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2a0102e00 0x7fe2a01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:38.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.536+0000 7fe2a7f28700 1 -- 172.21.15.32:0/3674709350 >> 172.21.15.32:0/3674709350 conn(0x7fe2a00f89b0 msgr2=0x7fe2a00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:38.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.536+0000 7fe2a7f28700 1 -- 172.21.15.32:0/3674709350 shutdown_connections 2024-04-12T05:42:38.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.536+0000 7fe2a7f28700 1 -- 172.21.15.32:0/3674709350 wait complete. 2024-04-12T05:42:38.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.537+0000 7fe2a7f28700 1 Processor -- start 2024-04-12T05:42:38.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.537+0000 7fe2a7f28700 1 -- start start 2024-04-12T05:42:38.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.537+0000 7fe2a7f28700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2a0102e00 0x7fe2a0113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:38.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.537+0000 7fe2a7f28700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2a01037a0 0x7fe2a0114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:38.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.537+0000 7fe2a7f28700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a0114690 con 0x7fe2a01037a0 2024-04-12T05:42:38.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.537+0000 7fe2a7f28700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a010da80 con 0x7fe2a0102e00 2024-04-12T05:42:38.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.537+0000 7fe2a54c3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2a01037a0 0x7fe2a0114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:38.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.537+0000 7fe2a54c3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2a01037a0 0x7fe2a0114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41130/0 (socket says 172.21.15.32:41130) 2024-04-12T05:42:38.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.537+0000 7fe2a54c3700 1 -- 172.21.15.32:0/2293926099 learned_addr learned my addr 172.21.15.32:0/2293926099 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:42:38.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.537+0000 7fe2a5cc4700 1 --2- 172.21.15.32:0/2293926099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2a0102e00 0x7fe2a0113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:38.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.537+0000 7fe2a54c3700 1 -- 172.21.15.32:0/2293926099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2a0102e00 msgr2=0x7fe2a0113b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:38.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.537+0000 7fe2a54c3700 1 --2- 172.21.15.32:0/2293926099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2a0102e00 0x7fe2a0113b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:38.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.537+0000 7fe2a54c3700 1 -- 172.21.15.32:0/2293926099 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe294005040 con 0x7fe2a01037a0 2024-04-12T05:42:38.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.537+0000 7fe2a5cc4700 1 --2- 172.21.15.32:0/2293926099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2a0102e00 0x7fe2a0113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:42:38.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.538+0000 7fe2a54c3700 1 --2- 172.21.15.32:0/2293926099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2a01037a0 0x7fe2a0114040 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe294009380 tx=0x7fe294009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:38.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.538+0000 7fe29e7fc700 1 -- 172.21.15.32:0/2293926099 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe29400e040 con 0x7fe2a01037a0 2024-04-12T05:42:38.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.538+0000 7fe29e7fc700 1 -- 172.21.15.32:0/2293926099 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe294013420 con 0x7fe2a01037a0 2024-04-12T05:42:38.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.538+0000 7fe2a7f28700 1 -- 172.21.15.32:0/2293926099 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2a010dd00 con 0x7fe2a01037a0 2024-04-12T05:42:38.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.538+0000 7fe29e7fc700 1 -- 172.21.15.32:0/2293926099 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe29401c630 con 0x7fe2a01037a0 2024-04-12T05:42:38.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.538+0000 7fe2a7f28700 1 -- 172.21.15.32:0/2293926099 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2a010e2c0 con 0x7fe2a01037a0 2024-04-12T05:42:38.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.539+0000 7fe2a7f28700 1 -- 172.21.15.32:0/2293926099 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2a00fe550 con 0x7fe2a01037a0 2024-04-12T05:42:38.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.540+0000 7fe29e7fc700 1 -- 172.21.15.32:0/2293926099 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7fe2940139e0 con 0x7fe2a01037a0 2024-04-12T05:42:38.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.540+0000 7fe29e7fc700 1 --2- 172.21.15.32:0/2293926099 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe28806cd40 0x7fe28806f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:38.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.540+0000 7fe29e7fc700 1 -- 172.21.15.32:0/2293926099 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fe29408def0 con 0x7fe2a01037a0 2024-04-12T05:42:38.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.540+0000 7fe2a5cc4700 1 --2- 172.21.15.32:0/2293926099 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe28806cd40 0x7fe28806f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:38.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.541+0000 7fe2a5cc4700 1 --2- 172.21.15.32:0/2293926099 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe28806cd40 0x7fe28806f1f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe290001250 tx=0x7fe29000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:38.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.544+0000 7fe29e7fc700 1 -- 172.21.15.32:0/2293926099 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe29405b890 con 0x7fe2a01037a0 2024-04-12T05:42:38.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.764+0000 7fe2a7f28700 1 -- 172.21.15.32:0/2293926099 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2a0109e10 con 0x7fe28806cd40 2024-04-12T05:42:38.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:38 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:38.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:38 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:38.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:38 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:42:38.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:38 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:42:38.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:38 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:42:38.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:38 smithi067 ceph-mon[112979]: Upgrade: Setting container_image for all mon 2024-04-12T05:42:38.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:38 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:38.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:38 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi032"}]: dispatch 2024-04-12T05:42:38.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:38 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi032"}]': finished 2024-04-12T05:42:38.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:38 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi067"}]: dispatch 2024-04-12T05:42:38.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:38 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi067"}]': finished 2024-04-12T05:42:38.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.774+0000 7fe29e7fc700 1 -- 172.21.15.32:0/2293926099 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fe2a0109e10 con 0x7fe28806cd40 2024-04-12T05:42:38.771 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:42:38.771 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (2m) 45s ago 13m 19.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:42:38.772 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (14m) 45s ago 14m 6656k - 17.2.7-827-gd3b5060a 21de9b7369ad a9cfc00c311b 2024-04-12T05:42:38.772 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (13m) 30s ago 13m 6651k - 17.2.7-827-gd3b5060a 21de9b7369ad 1f2eea4542d9 2024-04-12T05:42:38.772 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (2m) 45s ago 13m 76.3M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:42:38.772 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (8m) 45s ago 8m 64.8M - 17.2.7-827-gd3b5060a 21de9b7369ad e7ada8e5259c 2024-04-12T05:42:38.772 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (8m) 45s ago 8m 116M - 17.2.7-827-gd3b5060a 21de9b7369ad 55124da9afdb 2024-04-12T05:42:38.772 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (8m) 30s ago 8m 62.6M - 17.2.7-827-gd3b5060a 21de9b7369ad 0c295d31a4a6 2024-04-12T05:42:38.773 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (8m) 30s ago 8m 120M - 17.2.7-827-gd3b5060a 21de9b7369ad 6eecd7deaac2 2024-04-12T05:42:38.773 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (4m) 45s ago 15m 470M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:42:38.773 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (3m) 30s ago 12m 447M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:42:38.773 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (61s) 45s ago 15m 27.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:42:38.773 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (34s) 30s ago 12m 22.4M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:42:38.774 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (3m) 45s ago 14m 16.3M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:42:38.774 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (3m) 30s ago 12m 18.0M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:42:38.774 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (12m) 45s ago 12m 220M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad 97446e42ceba 2024-04-12T05:42:38.774 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (11m) 45s ago 11m 237M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad bf9f9a2db2d5 2024-04-12T05:42:38.774 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (11m) 45s ago 11m 190M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad f055cf1ca665 2024-04-12T05:42:38.774 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (10m) 30s ago 10m 257M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad 61b851235351 2024-04-12T05:42:38.774 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (10m) 30s ago 10m 229M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad 2c84d9047e9c 2024-04-12T05:42:38.775 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (9m) 30s ago 9m 219M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad e54839bb6055 2024-04-12T05:42:38.775 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (2m) 45s ago 13m 56.9M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:42:38.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.776+0000 7fe2a7f28700 1 -- 172.21.15.32:0/2293926099 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe28806cd40 msgr2=0x7fe28806f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:38.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.776+0000 7fe2a7f28700 1 --2- 172.21.15.32:0/2293926099 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe28806cd40 0x7fe28806f1f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe290001250 tx=0x7fe29000b040 comp rx=0 tx=0).stop 2024-04-12T05:42:38.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.776+0000 7fe2a7f28700 1 -- 172.21.15.32:0/2293926099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2a01037a0 msgr2=0x7fe2a0114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:38.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.776+0000 7fe2a7f28700 1 --2- 172.21.15.32:0/2293926099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2a01037a0 0x7fe2a0114040 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe294009380 tx=0x7fe294009e30 comp rx=0 tx=0).stop 2024-04-12T05:42:38.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.776+0000 7fe2a7f28700 1 -- 172.21.15.32:0/2293926099 shutdown_connections 2024-04-12T05:42:38.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.776+0000 7fe2a7f28700 1 --2- 172.21.15.32:0/2293926099 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe28806cd40 0x7fe28806f1f0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:38.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.776+0000 7fe2a7f28700 1 --2- 172.21.15.32:0/2293926099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2a0102e00 0x7fe2a0113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:38.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.776+0000 7fe2a7f28700 1 --2- 172.21.15.32:0/2293926099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2a01037a0 0x7fe2a0114040 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:38.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.776+0000 7fe2a7f28700 1 -- 172.21.15.32:0/2293926099 >> 172.21.15.32:0/2293926099 conn(0x7fe2a00f89b0 msgr2=0x7fe2a00f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:38.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.776+0000 7fe2a7f28700 1 -- 172.21.15.32:0/2293926099 shutdown_connections 2024-04-12T05:42:38.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.776+0000 7fe2a7f28700 1 -- 172.21.15.32:0/2293926099 wait complete. 2024-04-12T05:42:38.798 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:38 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:38.798 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:38 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:38.799 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:38 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:42:38.799 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:38 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:42:38.799 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:38 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:42:38.799 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:38 smithi032 ceph-mon[146817]: Upgrade: Setting container_image for all mon 2024-04-12T05:42:38.799 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:38 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:38.799 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:38 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi032"}]: dispatch 2024-04-12T05:42:38.799 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:38 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi032"}]': finished 2024-04-12T05:42:38.799 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:38 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi067"}]: dispatch 2024-04-12T05:42:38.799 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:38 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi067"}]': finished 2024-04-12T05:42:38.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.880+0000 7f77de1a5700 1 -- 172.21.15.32:0/3817102687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77d8102710 msgr2=0x7f77d80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:38.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.880+0000 7f77de1a5700 1 --2- 172.21.15.32:0/3817102687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77d8102710 0x7f77d80fbd80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f77cc004940 tx=0x7f77cc009e30 comp rx=0 tx=0).stop 2024-04-12T05:42:38.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.880+0000 7f77de1a5700 1 -- 172.21.15.32:0/3817102687 shutdown_connections 2024-04-12T05:42:38.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.880+0000 7f77de1a5700 1 --2- 172.21.15.32:0/3817102687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77d8102710 0x7f77d80fbd80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:38.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.880+0000 7f77de1a5700 1 --2- 172.21.15.32:0/3817102687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77d8101d70 0x7f77d8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:38.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.880+0000 7f77de1a5700 1 -- 172.21.15.32:0/3817102687 >> 172.21.15.32:0/3817102687 conn(0x7f77d8072bc0 msgr2=0x7f77d8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:38.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.880+0000 7f77de1a5700 1 -- 172.21.15.32:0/3817102687 shutdown_connections 2024-04-12T05:42:38.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.880+0000 7f77de1a5700 1 -- 172.21.15.32:0/3817102687 wait complete. 2024-04-12T05:42:38.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.881+0000 7f77de1a5700 1 Processor -- start 2024-04-12T05:42:38.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.881+0000 7f77de1a5700 1 -- start start 2024-04-12T05:42:38.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.881+0000 7f77de1a5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77d8101d70 0x7f77d810ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:38.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.881+0000 7f77de1a5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77d8102710 0x7f77d810f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:38.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.881+0000 7f77de1a5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d810f680 con 0x7f77d8102710 2024-04-12T05:42:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.881+0000 7f77de1a5700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d810f7c0 con 0x7f77d8101d70 2024-04-12T05:42:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.881+0000 7f77d6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77d8102710 0x7f77d810f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.881+0000 7f77d6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77d8102710 0x7f77d810f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41144/0 (socket says 172.21.15.32:41144) 2024-04-12T05:42:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.881+0000 7f77d6ffd700 1 -- 172.21.15.32:0/800849950 learned_addr learned my addr 172.21.15.32:0/800849950 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:42:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.882+0000 7f77d77fe700 1 --2- 172.21.15.32:0/800849950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77d8101d70 0x7f77d810ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.882+0000 7f77d6ffd700 1 -- 172.21.15.32:0/800849950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77d8101d70 msgr2=0x7f77d810ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.882+0000 7f77d6ffd700 1 --2- 172.21.15.32:0/800849950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77d8101d70 0x7f77d810ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.882+0000 7f77d6ffd700 1 -- 172.21.15.32:0/800849950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77cc005040 con 0x7f77d8102710 2024-04-12T05:42:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.882+0000 7f77d77fe700 1 --2- 172.21.15.32:0/800849950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77d8101d70 0x7f77d810ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:42:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.882+0000 7f77d6ffd700 1 --2- 172.21.15.32:0/800849950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77d8102710 0x7f77d810f140 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f77cc009380 tx=0x7f77cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.882+0000 7f77bffff700 1 -- 172.21.15.32:0/800849950 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77cc00e040 con 0x7f77d8102710 2024-04-12T05:42:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.882+0000 7f77de1a5700 1 -- 172.21.15.32:0/800849950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77d8117ab0 con 0x7f77d8102710 2024-04-12T05:42:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.882+0000 7f77bffff700 1 -- 172.21.15.32:0/800849950 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f77cc013420 con 0x7f77d8102710 2024-04-12T05:42:38.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.882+0000 7f77bffff700 1 -- 172.21.15.32:0/800849950 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77cc01c630 con 0x7f77d8102710 2024-04-12T05:42:38.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.882+0000 7f77de1a5700 1 -- 172.21.15.32:0/800849950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77d8118040 con 0x7f77d8102710 2024-04-12T05:42:38.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.883+0000 7f77de1a5700 1 -- 172.21.15.32:0/800849950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77d80ff080 con 0x7f77d8102710 2024-04-12T05:42:38.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.884+0000 7f77bffff700 1 -- 172.21.15.32:0/800849950 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7f77cc0139e0 con 0x7f77d8102710 2024-04-12T05:42:38.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.884+0000 7f77bffff700 1 --2- 172.21.15.32:0/800849950 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f77c006cd40 0x7f77c006f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:38.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.884+0000 7f77bffff700 1 -- 172.21.15.32:0/800849950 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f77cc08d450 con 0x7f77d8102710 2024-04-12T05:42:38.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.884+0000 7f77d77fe700 1 --2- 172.21.15.32:0/800849950 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f77c006cd40 0x7f77c006f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:38.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.885+0000 7f77d77fe700 1 --2- 172.21.15.32:0/800849950 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f77c006cd40 0x7f77c006f1f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f77c8009fd0 tx=0x7f77c800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:38.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:38.889+0000 7f77bffff700 1 -- 172.21.15.32:0/800849950 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f77cc0572c0 con 0x7f77d8102710 2024-04-12T05:42:39.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.181+0000 7f77de1a5700 1 -- 172.21.15.32:0/800849950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f77d8065e00 con 0x7f77d8102710 2024-04-12T05:42:39.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.182+0000 7f77bffff700 1 -- 172.21.15.32:0/800849950 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+742 (secure 0 0 0) 0x7f77cc05a8e0 con 0x7f77d8102710 2024-04-12T05:42:39.179 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:42:39.179 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:42:39.179 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:42:39.179 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:42:39.179 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:42:39.179 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:42:39.179 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:42:39.179 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:42:39.179 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 6 2024-04-12T05:42:39.180 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:42:39.180 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:42:39.180 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 4 2024-04-12T05:42:39.180 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:42:39.180 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:42:39.180 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 10, 2024-04-12T05:42:39.180 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:42:39.180 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:42:39.180 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:42:39.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.184+0000 7f77de1a5700 1 -- 172.21.15.32:0/800849950 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f77c006cd40 msgr2=0x7f77c006f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:39.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.184+0000 7f77de1a5700 1 --2- 172.21.15.32:0/800849950 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f77c006cd40 0x7f77c006f1f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f77c8009fd0 tx=0x7f77c800b040 comp rx=0 tx=0).stop 2024-04-12T05:42:39.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.184+0000 7f77de1a5700 1 -- 172.21.15.32:0/800849950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77d8102710 msgr2=0x7f77d810f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:39.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.184+0000 7f77de1a5700 1 --2- 172.21.15.32:0/800849950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77d8102710 0x7f77d810f140 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f77cc009380 tx=0x7f77cc009e30 comp rx=0 tx=0).stop 2024-04-12T05:42:39.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.184+0000 7f77de1a5700 1 -- 172.21.15.32:0/800849950 shutdown_connections 2024-04-12T05:42:39.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.184+0000 7f77de1a5700 1 --2- 172.21.15.32:0/800849950 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f77c006cd40 0x7f77c006f1f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:39.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.184+0000 7f77de1a5700 1 --2- 172.21.15.32:0/800849950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77d8101d70 0x7f77d810ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:39.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.184+0000 7f77de1a5700 1 --2- 172.21.15.32:0/800849950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77d8102710 0x7f77d810f140 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:39.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.184+0000 7f77de1a5700 1 -- 172.21.15.32:0/800849950 >> 172.21.15.32:0/800849950 conn(0x7f77d8072bc0 msgr2=0x7f77d80fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:39.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.184+0000 7f77de1a5700 1 -- 172.21.15.32:0/800849950 shutdown_connections 2024-04-12T05:42:39.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.184+0000 7f77de1a5700 1 -- 172.21.15.32:0/800849950 wait complete. 2024-04-12T05:42:39.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.291+0000 7f40eba86700 1 -- 172.21.15.32:0/2107683246 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40e4101d70 msgr2=0x7f40e4102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:39.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.291+0000 7f40eba86700 1 --2- 172.21.15.32:0/2107683246 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40e4101d70 0x7f40e4102140 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f40e0004940 tx=0x7f40e0009e30 comp rx=0 tx=0).stop 2024-04-12T05:42:39.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.291+0000 7f40eba86700 1 -- 172.21.15.32:0/2107683246 shutdown_connections 2024-04-12T05:42:39.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.291+0000 7f40eba86700 1 --2- 172.21.15.32:0/2107683246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40e4102710 0x7f40e40fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:39.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.291+0000 7f40eba86700 1 --2- 172.21.15.32:0/2107683246 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40e4101d70 0x7f40e4102140 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:39.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.291+0000 7f40eba86700 1 -- 172.21.15.32:0/2107683246 >> 172.21.15.32:0/2107683246 conn(0x7f40e4072bc0 msgr2=0x7f40e4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:39.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.291+0000 7f40eba86700 1 -- 172.21.15.32:0/2107683246 shutdown_connections 2024-04-12T05:42:39.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.291+0000 7f40eba86700 1 -- 172.21.15.32:0/2107683246 wait complete. 2024-04-12T05:42:39.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.292+0000 7f40eba86700 1 Processor -- start 2024-04-12T05:42:39.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.292+0000 7f40eba86700 1 -- start start 2024-04-12T05:42:39.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.292+0000 7f40eba86700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40e4101d70 0x7f40e410c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:39.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.292+0000 7f40eba86700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40e4102710 0x7f40e410ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:39.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.292+0000 7f40eba86700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e410d230 con 0x7f40e4102710 2024-04-12T05:42:39.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.292+0000 7f40eba86700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e410d3a0 con 0x7f40e4101d70 2024-04-12T05:42:39.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.292+0000 7f40e9021700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40e4102710 0x7f40e410ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:39.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.292+0000 7f40e9021700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40e4102710 0x7f40e410ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41166/0 (socket says 172.21.15.32:41166) 2024-04-12T05:42:39.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.292+0000 7f40e9021700 1 -- 172.21.15.32:0/1289411402 learned_addr learned my addr 172.21.15.32:0/1289411402 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:42:39.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.292+0000 7f40e9822700 1 --2- 172.21.15.32:0/1289411402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40e4101d70 0x7f40e410c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:39.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.293+0000 7f40e9021700 1 -- 172.21.15.32:0/1289411402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40e4101d70 msgr2=0x7f40e410c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:39.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.293+0000 7f40e9021700 1 --2- 172.21.15.32:0/1289411402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40e4101d70 0x7f40e410c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:39.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.293+0000 7f40e9021700 1 -- 172.21.15.32:0/1289411402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e0005040 con 0x7f40e4102710 2024-04-12T05:42:39.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.293+0000 7f40e9021700 1 --2- 172.21.15.32:0/1289411402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40e4102710 0x7f40e410ccf0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f40d400eb40 tx=0x7f40d400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:39.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.293+0000 7f40da7fc700 1 -- 172.21.15.32:0/1289411402 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40d4007cc0 con 0x7f40e4102710 2024-04-12T05:42:39.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.293+0000 7f40da7fc700 1 -- 172.21.15.32:0/1289411402 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f40d400f040 con 0x7f40e4102710 2024-04-12T05:42:39.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.293+0000 7f40eba86700 1 -- 172.21.15.32:0/1289411402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40e410d620 con 0x7f40e4102710 2024-04-12T05:42:39.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.293+0000 7f40da7fc700 1 -- 172.21.15.32:0/1289411402 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40d4009570 con 0x7f40e4102710 2024-04-12T05:42:39.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.293+0000 7f40eba86700 1 -- 172.21.15.32:0/1289411402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40e411a1a0 con 0x7f40e4102710 2024-04-12T05:42:39.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.294+0000 7f40eba86700 1 -- 172.21.15.32:0/1289411402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40e40ff080 con 0x7f40e4102710 2024-04-12T05:42:39.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.295+0000 7f40da7fc700 1 -- 172.21.15.32:0/1289411402 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7f40d40096d0 con 0x7f40e4102710 2024-04-12T05:42:39.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.295+0000 7f40da7fc700 1 --2- 172.21.15.32:0/1289411402 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f40d006cd40 0x7f40d006f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:39.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.295+0000 7f40da7fc700 1 -- 172.21.15.32:0/1289411402 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f40d408ba70 con 0x7f40e4102710 2024-04-12T05:42:39.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.295+0000 7f40e9822700 1 --2- 172.21.15.32:0/1289411402 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f40d006cd40 0x7f40d006f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:39.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.296+0000 7f40e9822700 1 --2- 172.21.15.32:0/1289411402 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f40d006cd40 0x7f40d006f1f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f40e000b0d0 tx=0x7f40e0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:39.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.300+0000 7f40da7fc700 1 -- 172.21.15.32:0/1289411402 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f40d4059420 con 0x7f40e4102710 2024-04-12T05:42:39.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.560+0000 7f40eba86700 1 -- 172.21.15.32:0/1289411402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f40e40fd4c0 con 0x7f40e4102710 2024-04-12T05:42:39.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.561+0000 7f40da7fc700 1 -- 172.21.15.32:0/1289411402 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1864 (secure 0 0 0) 0x7f40d4019760 con 0x7f40e4102710 2024-04-12T05:42:39.559 INFO:teuthology.orchestra.run.smithi032.stdout:e22 2024-04-12T05:42:39.559 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:42:39.559 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:42:39.559 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:42:39.559 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:42:39.559 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:42:39.559 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:42:39.559 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 22 2024-04-12T05:42:39.559 INFO:teuthology.orchestra.run.smithi032.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-12T05:42:39.559 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:42:39.559 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:41:05.358509+0000 2024-04-12T05:42:39.559 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:42:39.559 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:42:39.559 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:42:39.560 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:42:39.560 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:42:39.560 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:42:39.560 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:42:39.560 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:42:39.560 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 46 2024-04-12T05:42:39.560 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:42:39.560 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:42:39.560 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:42:39.560 INFO:teuthology.orchestra.run.smithi032.stdout:up {0=14480,1=24281} 2024-04-12T05:42:39.560 INFO:teuthology.orchestra.run.smithi032.stdout:failed 2024-04-12T05:42:39.560 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:42:39.560 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:42:39.560 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:42:39.561 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:42:39.561 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:42:39.561 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:42:39.561 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:42:39.561 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:42:39.561 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{0:14480} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.32:6828/387754149,v1:172.21.15.32:6829/387754149] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:42:39.561 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{0:14490} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3678031004,v1:172.21.15.32:6827/3678031004] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:42:39.561 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{1:24281} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.67:6824/1291914599,v1:172.21.15.67:6825/1291914599] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:42:39.561 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{1:24295} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.67:6826/873395842,v1:172.21.15.67:6827/873395842] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:42:39.561 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:42:39.561 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:42:39.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.563+0000 7f40eba86700 1 -- 172.21.15.32:0/1289411402 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f40d006cd40 msgr2=0x7f40d006f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:39.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.563+0000 7f40eba86700 1 --2- 172.21.15.32:0/1289411402 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f40d006cd40 0x7f40d006f1f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f40e000b0d0 tx=0x7f40e0009e30 comp rx=0 tx=0).stop 2024-04-12T05:42:39.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.563+0000 7f40eba86700 1 -- 172.21.15.32:0/1289411402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40e4102710 msgr2=0x7f40e410ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:39.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.563+0000 7f40eba86700 1 --2- 172.21.15.32:0/1289411402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40e4102710 0x7f40e410ccf0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f40d400eb40 tx=0x7f40d400b040 comp rx=0 tx=0).stop 2024-04-12T05:42:39.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.563+0000 7f40eba86700 1 -- 172.21.15.32:0/1289411402 shutdown_connections 2024-04-12T05:42:39.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.563+0000 7f40eba86700 1 --2- 172.21.15.32:0/1289411402 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f40d006cd40 0x7f40d006f1f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:39.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.563+0000 7f40eba86700 1 --2- 172.21.15.32:0/1289411402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40e4101d70 0x7f40e410c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:39.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.563+0000 7f40eba86700 1 --2- 172.21.15.32:0/1289411402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40e4102710 0x7f40e410ccf0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:39.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.563+0000 7f40eba86700 1 -- 172.21.15.32:0/1289411402 >> 172.21.15.32:0/1289411402 conn(0x7f40e4072bc0 msgr2=0x7f40e40faf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:39.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.563+0000 7f40eba86700 1 -- 172.21.15.32:0/1289411402 shutdown_connections 2024-04-12T05:42:39.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.563+0000 7f40eba86700 1 -- 172.21.15.32:0/1289411402 wait complete. 2024-04-12T05:42:39.563 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 22 2024-04-12T05:42:39.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.674+0000 7fca630c0700 1 -- 172.21.15.32:0/478586484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca5c0fe830 msgr2=0x7fca5c106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:39.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.674+0000 7fca630c0700 1 --2- 172.21.15.32:0/478586484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca5c0fe830 0x7fca5c106f00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fca48004940 tx=0x7fca48009e30 comp rx=0 tx=0).stop 2024-04-12T05:42:39.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.674+0000 7fca630c0700 1 -- 172.21.15.32:0/478586484 shutdown_connections 2024-04-12T05:42:39.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.674+0000 7fca630c0700 1 --2- 172.21.15.32:0/478586484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca5c0fe830 0x7fca5c106f00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:39.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.674+0000 7fca630c0700 1 --2- 172.21.15.32:0/478586484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca5c0fde90 0x7fca5c0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:39.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.674+0000 7fca630c0700 1 -- 172.21.15.32:0/478586484 >> 172.21.15.32:0/478586484 conn(0x7fca5c072bc0 msgr2=0x7fca5c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:39.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.674+0000 7fca630c0700 1 -- 172.21.15.32:0/478586484 shutdown_connections 2024-04-12T05:42:39.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.674+0000 7fca630c0700 1 -- 172.21.15.32:0/478586484 wait complete. 2024-04-12T05:42:39.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.674+0000 7fca630c0700 1 Processor -- start 2024-04-12T05:42:39.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.674+0000 7fca630c0700 1 -- start start 2024-04-12T05:42:39.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.675+0000 7fca630c0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca5c0fde90 0x7fca5c113a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:39.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.675+0000 7fca630c0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca5c0fe830 0x7fca5c113fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:39.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.675+0000 7fca630c0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5c114610 con 0x7fca5c0fe830 2024-04-12T05:42:39.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.675+0000 7fca630c0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5c10da00 con 0x7fca5c0fde90 2024-04-12T05:42:39.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.675+0000 7fca5bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca5c0fe830 0x7fca5c113fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:39.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.675+0000 7fca5bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca5c0fe830 0x7fca5c113fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41178/0 (socket says 172.21.15.32:41178) 2024-04-12T05:42:39.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.675+0000 7fca5bfff700 1 -- 172.21.15.32:0/3751396025 learned_addr learned my addr 172.21.15.32:0/3751396025 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:42:39.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.675+0000 7fca60e5c700 1 --2- 172.21.15.32:0/3751396025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca5c0fde90 0x7fca5c113a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:39.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.675+0000 7fca5bfff700 1 -- 172.21.15.32:0/3751396025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca5c0fde90 msgr2=0x7fca5c113a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:39.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.675+0000 7fca5bfff700 1 --2- 172.21.15.32:0/3751396025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca5c0fde90 0x7fca5c113a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:39.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.675+0000 7fca5bfff700 1 -- 172.21.15.32:0/3751396025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca48005040 con 0x7fca5c0fe830 2024-04-12T05:42:39.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.675+0000 7fca5bfff700 1 --2- 172.21.15.32:0/3751396025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca5c0fe830 0x7fca5c113fc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fca48000c00 tx=0x7fca48009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:39.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.675+0000 7fca597fa700 1 -- 172.21.15.32:0/3751396025 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca48021070 con 0x7fca5c0fe830 2024-04-12T05:42:39.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.675+0000 7fca630c0700 1 -- 172.21.15.32:0/3751396025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca5c10dc80 con 0x7fca5c0fe830 2024-04-12T05:42:39.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.675+0000 7fca597fa700 1 -- 172.21.15.32:0/3751396025 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca48013e00 con 0x7fca5c0fe830 2024-04-12T05:42:39.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.675+0000 7fca597fa700 1 -- 172.21.15.32:0/3751396025 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca4800e070 con 0x7fca5c0fe830 2024-04-12T05:42:39.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.676+0000 7fca630c0700 1 -- 172.21.15.32:0/3751396025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca5c10e1c0 con 0x7fca5c0fe830 2024-04-12T05:42:39.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.676+0000 7fca630c0700 1 -- 172.21.15.32:0/3751396025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca5c101980 con 0x7fca5c0fe830 2024-04-12T05:42:39.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.677+0000 7fca597fa700 1 -- 172.21.15.32:0/3751396025 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7fca48013950 con 0x7fca5c0fe830 2024-04-12T05:42:39.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.677+0000 7fca597fa700 1 --2- 172.21.15.32:0/3751396025 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fca4c06cd40 0x7fca4c06f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:39.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.677+0000 7fca597fa700 1 -- 172.21.15.32:0/3751396025 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fca4808d240 con 0x7fca5c0fe830 2024-04-12T05:42:39.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.677+0000 7fca60e5c700 1 --2- 172.21.15.32:0/3751396025 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fca4c06cd40 0x7fca4c06f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:39.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.678+0000 7fca60e5c700 1 --2- 172.21.15.32:0/3751396025 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fca4c06cd40 0x7fca4c06f1f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fca50009b80 tx=0x7fca5000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:39.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.682+0000 7fca597fa700 1 -- 172.21.15.32:0/3751396025 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fca4805ab40 con 0x7fca5c0fe830 2024-04-12T05:42:39.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:39 smithi067 ceph-mon[112979]: pgmap v28: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:39.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:39 smithi067 ceph-mon[112979]: from='client.34156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:39.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:39 smithi067 ceph-mon[112979]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:39.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:39 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:39.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:39 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi032", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-12T05:42:39.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:39 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:39.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:39 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/800849950' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:42:39.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:39 smithi032 ceph-mon[146817]: pgmap v28: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:39.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:39 smithi032 ceph-mon[146817]: from='client.34156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:39.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:39 smithi032 ceph-mon[146817]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:39.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:39 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:39.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:39 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi032", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-12T05:42:39.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:39 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:39.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:39 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/800849950' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:42:39.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.911+0000 7fca630c0700 1 -- 172.21.15.32:0/3751396025 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fca5c10f0d0 con 0x7fca4c06cd40 2024-04-12T05:42:39.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.912+0000 7fca597fa700 1 -- 172.21.15.32:0/3751396025 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7fca5c10f0d0 con 0x7fca4c06cd40 2024-04-12T05:42:39.909 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:42:39.909 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:42:39.909 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:42:39.910 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:42:39.910 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:42:39.910 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:42:39.910 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr" 2024-04-12T05:42:39.910 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:42:39.910 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "4/21 daemons upgraded", 2024-04-12T05:42:39.910 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading crash daemons", 2024-04-12T05:42:39.910 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:42:39.910 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:42:39.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.914+0000 7fca630c0700 1 -- 172.21.15.32:0/3751396025 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fca4c06cd40 msgr2=0x7fca4c06f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:39.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.914+0000 7fca630c0700 1 --2- 172.21.15.32:0/3751396025 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fca4c06cd40 0x7fca4c06f1f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fca50009b80 tx=0x7fca5000b040 comp rx=0 tx=0).stop 2024-04-12T05:42:39.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.914+0000 7fca630c0700 1 -- 172.21.15.32:0/3751396025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca5c0fe830 msgr2=0x7fca5c113fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:39.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.914+0000 7fca630c0700 1 --2- 172.21.15.32:0/3751396025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca5c0fe830 0x7fca5c113fc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fca48000c00 tx=0x7fca48009e30 comp rx=0 tx=0).stop 2024-04-12T05:42:39.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.914+0000 7fca630c0700 1 -- 172.21.15.32:0/3751396025 shutdown_connections 2024-04-12T05:42:39.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.914+0000 7fca630c0700 1 --2- 172.21.15.32:0/3751396025 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fca4c06cd40 0x7fca4c06f1f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:39.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.914+0000 7fca630c0700 1 --2- 172.21.15.32:0/3751396025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca5c0fde90 0x7fca5c113a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:39.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.914+0000 7fca630c0700 1 --2- 172.21.15.32:0/3751396025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca5c0fe830 0x7fca5c113fc0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:39.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.914+0000 7fca630c0700 1 -- 172.21.15.32:0/3751396025 >> 172.21.15.32:0/3751396025 conn(0x7fca5c072bc0 msgr2=0x7fca5c0ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:39.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.914+0000 7fca630c0700 1 -- 172.21.15.32:0/3751396025 shutdown_connections 2024-04-12T05:42:39.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:39.914+0000 7fca630c0700 1 -- 172.21.15.32:0/3751396025 wait complete. 2024-04-12T05:42:40.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.017+0000 7effbfe06700 1 -- 172.21.15.32:0/1855790596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effb8102710 msgr2=0x7effb80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:40.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.017+0000 7effbfe06700 1 --2- 172.21.15.32:0/1855790596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effb8102710 0x7effb80fbd80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7effac004940 tx=0x7effac009e30 comp rx=0 tx=0).stop 2024-04-12T05:42:40.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.017+0000 7effbfe06700 1 -- 172.21.15.32:0/1855790596 shutdown_connections 2024-04-12T05:42:40.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.017+0000 7effbfe06700 1 --2- 172.21.15.32:0/1855790596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effb8102710 0x7effb80fbd80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:40.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.017+0000 7effbfe06700 1 --2- 172.21.15.32:0/1855790596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effb8101d70 0x7effb8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:40.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.017+0000 7effbfe06700 1 -- 172.21.15.32:0/1855790596 >> 172.21.15.32:0/1855790596 conn(0x7effb8072bc0 msgr2=0x7effb8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:40.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.017+0000 7effbfe06700 1 -- 172.21.15.32:0/1855790596 shutdown_connections 2024-04-12T05:42:40.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.017+0000 7effbfe06700 1 -- 172.21.15.32:0/1855790596 wait complete. 2024-04-12T05:42:40.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.018+0000 7effbfe06700 1 Processor -- start 2024-04-12T05:42:40.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.018+0000 7effbfe06700 1 -- start start 2024-04-12T05:42:40.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.018+0000 7effbfe06700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effb8101d70 0x7effb8113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:40.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.018+0000 7effbfe06700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effb8102710 0x7effb8114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:40.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.018+0000 7effbfe06700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb8114670 con 0x7effb8102710 2024-04-12T05:42:40.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.018+0000 7effbfe06700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb810da60 con 0x7effb8101d70 2024-04-12T05:42:40.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.018+0000 7effbd3a1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effb8102710 0x7effb8114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:40.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.018+0000 7effbd3a1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effb8102710 0x7effb8114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41198/0 (socket says 172.21.15.32:41198) 2024-04-12T05:42:40.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.018+0000 7effbd3a1700 1 -- 172.21.15.32:0/1467720911 learned_addr learned my addr 172.21.15.32:0/1467720911 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:42:40.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.018+0000 7effbdba2700 1 --2- 172.21.15.32:0/1467720911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effb8101d70 0x7effb8113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:40.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.018+0000 7effbd3a1700 1 -- 172.21.15.32:0/1467720911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effb8101d70 msgr2=0x7effb8113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:40.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.018+0000 7effbd3a1700 1 --2- 172.21.15.32:0/1467720911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effb8101d70 0x7effb8113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:40.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.018+0000 7effbd3a1700 1 -- 172.21.15.32:0/1467720911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effac005040 con 0x7effb8102710 2024-04-12T05:42:40.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.019+0000 7effbd3a1700 1 --2- 172.21.15.32:0/1467720911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effb8102710 0x7effb8114020 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7effac0093b0 tx=0x7effac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:40.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.019+0000 7effb67fc700 1 -- 172.21.15.32:0/1467720911 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effac00e040 con 0x7effb8102710 2024-04-12T05:42:40.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.019+0000 7effbfe06700 1 -- 172.21.15.32:0/1467720911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effb810dce0 con 0x7effb8102710 2024-04-12T05:42:40.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.019+0000 7effb67fc700 1 -- 172.21.15.32:0/1467720911 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7effac013420 con 0x7effb8102710 2024-04-12T05:42:40.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.019+0000 7effb67fc700 1 -- 172.21.15.32:0/1467720911 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effac01c630 con 0x7effb8102710 2024-04-12T05:42:40.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.019+0000 7effbfe06700 1 -- 172.21.15.32:0/1467720911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effb810e2a0 con 0x7effb8102710 2024-04-12T05:42:40.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.020+0000 7effbfe06700 1 -- 172.21.15.32:0/1467720911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa40052b0 con 0x7effb8102710 2024-04-12T05:42:40.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.020+0000 7effb67fc700 1 -- 172.21.15.32:0/1467720911 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7effac00ba80 con 0x7effb8102710 2024-04-12T05:42:40.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.021+0000 7effb67fc700 1 --2- 172.21.15.32:0/1467720911 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7effa006cd40 0x7effa006f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:42:40.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.021+0000 7effb67fc700 1 -- 172.21.15.32:0/1467720911 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6410+0+0 (secure 0 0 0) 0x7effac08d2c0 con 0x7effb8102710 2024-04-12T05:42:40.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.021+0000 7effbdba2700 1 --2- 172.21.15.32:0/1467720911 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7effa006cd40 0x7effa006f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:40.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.021+0000 7effbdba2700 1 --2- 172.21.15.32:0/1467720911 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7effa006cd40 0x7effa006f1f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7effa8004fd0 tx=0x7effa8009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:40.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.025+0000 7effb67fc700 1 -- 172.21.15.32:0/1467720911 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7effac05abc0 con 0x7effb8102710 2024-04-12T05:42:40.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.315+0000 7effbfe06700 1 -- 172.21.15.32:0/1467720911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7effa4005830 con 0x7effb8102710 2024-04-12T05:42:40.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.315+0000 7effb67fc700 1 -- 172.21.15.32:0/1467720911 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7effac05a750 con 0x7effb8102710 2024-04-12T05:42:40.312 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_OK 2024-04-12T05:42:40.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.317+0000 7effbfe06700 1 -- 172.21.15.32:0/1467720911 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7effa006cd40 msgr2=0x7effa006f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:40.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.317+0000 7effbfe06700 1 --2- 172.21.15.32:0/1467720911 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7effa006cd40 0x7effa006f1f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7effa8004fd0 tx=0x7effa8009040 comp rx=0 tx=0).stop 2024-04-12T05:42:40.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.317+0000 7effbfe06700 1 -- 172.21.15.32:0/1467720911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effb8102710 msgr2=0x7effb8114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:40.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.317+0000 7effbfe06700 1 --2- 172.21.15.32:0/1467720911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effb8102710 0x7effb8114020 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7effac0093b0 tx=0x7effac009e30 comp rx=0 tx=0).stop 2024-04-12T05:42:40.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.317+0000 7effbfe06700 1 -- 172.21.15.32:0/1467720911 shutdown_connections 2024-04-12T05:42:40.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.317+0000 7effbfe06700 1 --2- 172.21.15.32:0/1467720911 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7effa006cd40 0x7effa006f1f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:40.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.317+0000 7effbfe06700 1 --2- 172.21.15.32:0/1467720911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effb8101d70 0x7effb8113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:40.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.317+0000 7effbfe06700 1 --2- 172.21.15.32:0/1467720911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effb8102710 0x7effb8114020 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:42:40.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.317+0000 7effbfe06700 1 -- 172.21.15.32:0/1467720911 >> 172.21.15.32:0/1467720911 conn(0x7effb8072bc0 msgr2=0x7effb8065a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:40.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.317+0000 7effbfe06700 1 -- 172.21.15.32:0/1467720911 shutdown_connections 2024-04-12T05:42:40.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:42:40.317+0000 7effbfe06700 1 -- 172.21.15.32:0/1467720911 wait complete. 2024-04-12T05:42:40.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:40 smithi067 ceph-mon[112979]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:40.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:40 smithi067 ceph-mon[112979]: Upgrade: Updating crash.smithi032 (1/2) 2024-04-12T05:42:40.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:40 smithi067 ceph-mon[112979]: Deploying daemon crash.smithi032 on smithi032 2024-04-12T05:42:40.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:40 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1289411402' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:42:40.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:40 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1467720911' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:42:40.789 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:40 smithi032 ceph-mon[146817]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:40.789 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:40 smithi032 ceph-mon[146817]: Upgrade: Updating crash.smithi032 (1/2) 2024-04-12T05:42:40.789 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:40 smithi032 ceph-mon[146817]: Deploying daemon crash.smithi032 on smithi032 2024-04-12T05:42:40.789 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:40 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1289411402' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:42:40.789 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:40 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1467720911' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:42:41.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:41 smithi032 ceph-mon[146817]: pgmap v29: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:42:41.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:41 smithi032 ceph-mon[146817]: from='client.34176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:41.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:41 smithi067 ceph-mon[112979]: pgmap v29: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:42:41.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:41 smithi067 ceph-mon[112979]: from='client.34176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:43.760 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:43 smithi067 ceph-mon[112979]: pgmap v30: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:43.760 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:43 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:43.760 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:43 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:43.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:43 smithi032 ceph-mon[146817]: pgmap v30: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:43.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:43 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:43.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:43 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:45.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:45 smithi067 ceph-mon[112979]: pgmap v31: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:42:45.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:45 smithi067 ceph-mon[112979]: Upgrade: Updating crash.smithi067 (2/2) 2024-04-12T05:42:45.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:45 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:45.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:45 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi067", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-12T05:42:45.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:45 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:45.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:45 smithi067 ceph-mon[112979]: Deploying daemon crash.smithi067 on smithi067 2024-04-12T05:42:45.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:45 smithi032 ceph-mon[146817]: pgmap v31: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:42:45.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:45 smithi032 ceph-mon[146817]: Upgrade: Updating crash.smithi067 (2/2) 2024-04-12T05:42:45.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:45 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:45.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:45 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi067", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-12T05:42:45.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:45 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:45.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:45 smithi032 ceph-mon[146817]: Deploying daemon crash.smithi067 on smithi067 2024-04-12T05:42:47.490 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:47 smithi067 ceph-mon[112979]: pgmap v32: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:47.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:47 smithi032 ceph-mon[146817]: pgmap v32: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:48.656 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:48 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:48.656 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:48 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:48.656 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:48 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:42:48.656 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:48 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:42:48.981 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:48 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:48.981 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:48 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:48.981 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:48 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:42:48.981 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:48 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:42:49.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:49 smithi032 ceph-mon[146817]: pgmap v33: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:49.875 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:49 smithi067 ceph-mon[112979]: pgmap v33: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:51.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:51 smithi067 ceph-mon[112979]: pgmap v34: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:42:51.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:51 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:51.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:51 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:51.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:51 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:51.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:51 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:51.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:51 smithi032 ceph-mon[146817]: pgmap v34: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:42:51.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:51 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:51.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:51 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:51.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:51 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:51.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:51 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:53.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:53 smithi067 ceph-mon[112979]: pgmap v35: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:53.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:53 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:53.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:53 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:53.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:53 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:53.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:53 smithi032 ceph-mon[146817]: pgmap v35: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:53.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:53 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:53.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:53 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:53.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:53 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:54.672 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:54 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:54.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:54 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:56.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:55 smithi067 ceph-mon[112979]: pgmap v36: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:42:56.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:55 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:56.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:55 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:56.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:55 smithi032 ceph-mon[146817]: pgmap v36: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:42:56.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:55 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:56.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:55 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:58.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:57 smithi067 ceph-mon[112979]: pgmap v37: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:58.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:57 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:58.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:57 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:58.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:57 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:58.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:57 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:42:58.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:57 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:58.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:57 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:42:58.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:57 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:42:58.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:57 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:42:58.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:57 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:42:58.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:57 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:58.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:57 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi032"}]: dispatch 2024-04-12T05:42:58.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:57 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi032"}]': finished 2024-04-12T05:42:58.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:57 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi067"}]: dispatch 2024-04-12T05:42:58.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:57 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi067"}]': finished 2024-04-12T05:42:58.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:57 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-12T05:42:58.086 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:57 smithi032 ceph-mon[146817]: pgmap v37: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:58.086 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:57 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:58.086 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:57 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:58.086 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:57 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:58.086 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:57 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:42:58.086 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:57 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:58.086 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:57 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:42:58.086 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:57 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:42:58.086 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:57 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:42:58.086 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:57 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:42:58.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:57 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:58.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:57 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi032"}]: dispatch 2024-04-12T05:42:58.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:57 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi032"}]': finished 2024-04-12T05:42:58.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:57 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi067"}]: dispatch 2024-04-12T05:42:58.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:57 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi067"}]': finished 2024-04-12T05:42:58.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:57 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-12T05:42:58.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:58 smithi032 ceph-mon[146817]: Upgrade: Setting container_image for all crash 2024-04-12T05:42:58.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:58 smithi032 ceph-mon[146817]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-12T05:42:58.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:58 smithi032 ceph-mon[146817]: Upgrade: osd.0 is safe to restart 2024-04-12T05:42:59.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:58 smithi067 ceph-mon[112979]: Upgrade: Setting container_image for all crash 2024-04-12T05:42:59.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:58 smithi067 ceph-mon[112979]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-12T05:42:59.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:58 smithi067 ceph-mon[112979]: Upgrade: osd.0 is safe to restart 2024-04-12T05:42:59.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:59 smithi032 ceph-mon[146817]: pgmap v38: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:42:59.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:59 smithi032 ceph-mon[146817]: Upgrade: Updating osd.0 2024-04-12T05:42:59.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:59 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:42:59.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:59 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-12T05:42:59.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:59 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:42:59.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:42:59 smithi032 ceph-mon[146817]: Deploying daemon osd.0 on smithi032 2024-04-12T05:43:00.020 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:59 smithi067 ceph-mon[112979]: pgmap v38: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:43:00.020 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:59 smithi067 ceph-mon[112979]: Upgrade: Updating osd.0 2024-04-12T05:43:00.020 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:59 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:00.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:59 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-12T05:43:00.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:59 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:43:00.021 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:42:59 smithi067 ceph-mon[112979]: Deploying daemon osd.0 on smithi032 2024-04-12T05:43:00.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:00 smithi032 ceph-mon[146817]: pgmap v39: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:43:01.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:00 smithi067 ceph-mon[112979]: pgmap v39: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:43:02.544 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 05:43:02 smithi032 systemd[1]: Stopping Ceph osd.0 for 37b58ce2-f88d-11ee-bc8c-c7b262605968... 2024-04-12T05:43:02.838 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 05:43:02 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-0[63328]: 2024-04-12T05:43:02.546+0000 7f2c64cef700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T05:43:02.839 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 05:43:02 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-0[63328]: 2024-04-12T05:43:02.546+0000 7f2c64cef700 -1 osd.0 49 *** Got signal Terminated *** 2024-04-12T05:43:02.839 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 05:43:02 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-0[63328]: 2024-04-12T05:43:02.546+0000 7f2c64cef700 -1 osd.0 49 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T05:43:03.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:02 smithi067 ceph-mon[112979]: pgmap v40: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:43:03.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:02 smithi067 ceph-mon[112979]: osd.0 marked itself down and dead 2024-04-12T05:43:03.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:02 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:43:03.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:02 smithi032 ceph-mon[146817]: pgmap v40: 65 pgs: 65 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:43:03.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:02 smithi032 ceph-mon[146817]: osd.0 marked itself down and dead 2024-04-12T05:43:03.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:02 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:43:04.089 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 05:43:03 smithi032 bash[160705]: ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-0 2024-04-12T05:43:04.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:03 smithi032 ceph-mon[146817]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T05:43:04.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:03 smithi032 ceph-mon[146817]: osdmap e50: 6 total, 5 up, 6 in 2024-04-12T05:43:04.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:03 smithi067 ceph-mon[112979]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T05:43:04.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:03 smithi067 ceph-mon[112979]: osdmap e50: 6 total, 5 up, 6 in 2024-04-12T05:43:05.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:04 smithi032 ceph-mon[146817]: pgmap v42: 65 pgs: 34 peering, 31 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1 op/s 2024-04-12T05:43:05.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:04 smithi032 ceph-mon[146817]: Health check failed: Reduced data availability: 33 pgs peering (PG_AVAILABILITY) 2024-04-12T05:43:05.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:04 smithi032 ceph-mon[146817]: osdmap e51: 6 total, 5 up, 6 in 2024-04-12T05:43:05.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:04 smithi067 ceph-mon[112979]: pgmap v42: 65 pgs: 34 peering, 31 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1 op/s 2024-04-12T05:43:05.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:04 smithi067 ceph-mon[112979]: Health check failed: Reduced data availability: 33 pgs peering (PG_AVAILABILITY) 2024-04-12T05:43:05.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:04 smithi067 ceph-mon[112979]: osdmap e51: 6 total, 5 up, 6 in 2024-04-12T05:43:05.839 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 05:43:05 smithi032 systemd[1]: ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@osd.0.service: Succeeded. 2024-04-12T05:43:05.839 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 05:43:05 smithi032 systemd[1]: Stopped Ceph osd.0 for 37b58ce2-f88d-11ee-bc8c-c7b262605968. 2024-04-12T05:43:05.839 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 05:43:05 smithi032 systemd[1]: Starting Ceph osd.0 for 37b58ce2-f88d-11ee-bc8c-c7b262605968... 2024-04-12T05:43:06.582 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 05:43:06 smithi032 bash[160986]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-12T05:43:06.582 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 05:43:06 smithi032 bash[160986]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-12T05:43:06.582 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 05:43:06 smithi032 bash[160986]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-12T05:43:06.582 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 05:43:06 smithi032 bash[160986]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-12T05:43:06.582 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 05:43:06 smithi032 bash[160986]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-12T05:43:06.583 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 05:43:06 smithi032 bash[160986]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-12T05:43:06.583 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 05:43:06 smithi032 bash[160986]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-12T05:43:07.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:06 smithi067 ceph-mon[112979]: pgmap v44: 65 pgs: 34 peering, 31 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1 op/s 2024-04-12T05:43:07.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:06 smithi032 ceph-mon[146817]: pgmap v44: 65 pgs: 34 peering, 31 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1 op/s 2024-04-12T05:43:08.089 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 05:43:07 smithi032 ceph-osd[161290]: -- 172.21.15.32:0/2210984032 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564b553b11e0 con 0x564b55380c00 2024-04-12T05:43:08.089 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 05:43:07 smithi032 bash[161252]: ab15e07d77a28fc13e780d22921f7b9a96d8053dc03d6d5bb6b29b11ff9d0b1a 2024-04-12T05:43:08.089 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 05:43:07 smithi032 systemd[1]: Started Ceph osd.0 for 37b58ce2-f88d-11ee-bc8c-c7b262605968. 2024-04-12T05:43:08.589 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 05:43:08 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-0[161267]: 2024-04-12T05:43:08.258+0000 7f26cb1077c0 -1 Falling back to public interface 2024-04-12T05:43:09.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:08 smithi032 ceph-mon[146817]: pgmap v45: 65 pgs: 34 peering, 31 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:43:09.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:08 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:09.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:08 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:09.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:08 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:43:09.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:08 smithi067 ceph-mon[112979]: pgmap v45: 65 pgs: 34 peering, 31 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:43:09.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:08 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:09.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:08 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:09.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:08 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:43:10.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:09 smithi067 ceph-mon[112979]: Health check failed: Degraded data redundancy: 64/333 objects degraded (19.219%), 19 pgs degraded (PG_DEGRADED) 2024-04-12T05:43:10.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:09 smithi067 ceph-mon[112979]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 33 pgs peering) 2024-04-12T05:43:10.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:09 smithi032 ceph-mon[146817]: Health check failed: Degraded data redundancy: 64/333 objects degraded (19.219%), 19 pgs degraded (PG_DEGRADED) 2024-04-12T05:43:10.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:09 smithi032 ceph-mon[146817]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 33 pgs peering) 2024-04-12T05:43:10.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.436+0000 7fa4ff345700 1 -- 172.21.15.32:0/1792462421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4f8102710 msgr2=0x7fa4f80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:10.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.436+0000 7fa4ff345700 1 --2- 172.21.15.32:0/1792462421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4f8102710 0x7fa4f80fbd80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec004940 tx=0x7fa4ec009e30 comp rx=0 tx=0).stop 2024-04-12T05:43:10.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.436+0000 7fa4ff345700 1 -- 172.21.15.32:0/1792462421 shutdown_connections 2024-04-12T05:43:10.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.436+0000 7fa4ff345700 1 --2- 172.21.15.32:0/1792462421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4f8102710 0x7fa4f80fbd80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:10.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.436+0000 7fa4ff345700 1 --2- 172.21.15.32:0/1792462421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4f8101d70 0x7fa4f8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:10.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.436+0000 7fa4ff345700 1 -- 172.21.15.32:0/1792462421 >> 172.21.15.32:0/1792462421 conn(0x7fa4f8072bc0 msgr2=0x7fa4f8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:10.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.436+0000 7fa4ff345700 1 -- 172.21.15.32:0/1792462421 shutdown_connections 2024-04-12T05:43:10.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.436+0000 7fa4ff345700 1 -- 172.21.15.32:0/1792462421 wait complete. 2024-04-12T05:43:10.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.436+0000 7fa4ff345700 1 Processor -- start 2024-04-12T05:43:10.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.436+0000 7fa4ff345700 1 -- start start 2024-04-12T05:43:10.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.436+0000 7fa4ff345700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4f8101d70 0x7fa4f8113a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:10.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.437+0000 7fa4ff345700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4f8102710 0x7fa4f8113f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:10.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.437+0000 7fa4ff345700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f8114590 con 0x7fa4f8101d70 2024-04-12T05:43:10.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.437+0000 7fa4ff345700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f810d980 con 0x7fa4f8102710 2024-04-12T05:43:10.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.437+0000 7fa4fc8e0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4f8102710 0x7fa4f8113f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:10.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.437+0000 7fa4fc8e0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4f8102710 0x7fa4f8113f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.32:52900/0 (socket says 172.21.15.32:52900) 2024-04-12T05:43:10.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.437+0000 7fa4fc8e0700 1 -- 172.21.15.32:0/1879860643 learned_addr learned my addr 172.21.15.32:0/1879860643 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:43:10.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.437+0000 7fa4fd0e1700 1 --2- 172.21.15.32:0/1879860643 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4f8101d70 0x7fa4f8113a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:10.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.437+0000 7fa4fc8e0700 1 -- 172.21.15.32:0/1879860643 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4f8101d70 msgr2=0x7fa4f8113a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:10.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.437+0000 7fa4fc8e0700 1 --2- 172.21.15.32:0/1879860643 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4f8101d70 0x7fa4f8113a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:10.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.437+0000 7fa4fc8e0700 1 -- 172.21.15.32:0/1879860643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ec005040 con 0x7fa4f8102710 2024-04-12T05:43:10.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.437+0000 7fa4fc8e0700 1 --2- 172.21.15.32:0/1879860643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4f8102710 0x7fa4f8113f40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec009390 tx=0x7fa4ec009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:10.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.437+0000 7fa4f5ffb700 1 -- 172.21.15.32:0/1879860643 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4ec00e040 con 0x7fa4f8102710 2024-04-12T05:43:10.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.438+0000 7fa4ff345700 1 -- 172.21.15.32:0/1879860643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4f810dc00 con 0x7fa4f8102710 2024-04-12T05:43:10.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.438+0000 7fa4f5ffb700 1 -- 172.21.15.32:0/1879860643 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa4ec004290 con 0x7fa4f8102710 2024-04-12T05:43:10.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.438+0000 7fa4f5ffb700 1 -- 172.21.15.32:0/1879860643 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4ec013e70 con 0x7fa4f8102710 2024-04-12T05:43:10.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.438+0000 7fa4ff345700 1 -- 172.21.15.32:0/1879860643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4f810e1c0 con 0x7fa4f8102710 2024-04-12T05:43:10.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.439+0000 7fa4ff345700 1 -- 172.21.15.32:0/1879860643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4f80fd4c0 con 0x7fa4f8102710 2024-04-12T05:43:10.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.439+0000 7fa4f5ffb700 1 -- 172.21.15.32:0/1879860643 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7fa4ec013690 con 0x7fa4f8102710 2024-04-12T05:43:10.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.440+0000 7fa4f5ffb700 1 --2- 172.21.15.32:0/1879860643 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa4e006ccd0 0x7fa4e006f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:10.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.440+0000 7fa4f5ffb700 1 -- 172.21.15.32:0/1879860643 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fa4ec08d480 con 0x7fa4f8102710 2024-04-12T05:43:10.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.440+0000 7fa4fd0e1700 1 --2- 172.21.15.32:0/1879860643 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa4e006ccd0 0x7fa4e006f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:10.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.440+0000 7fa4fd0e1700 1 --2- 172.21.15.32:0/1879860643 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa4e006ccd0 0x7fa4e006f180 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa4e8007b20 tx=0x7fa4e8009e20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:10.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.445+0000 7fa4f5ffb700 1 -- 172.21.15.32:0/1879860643 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa4ec05ae30 con 0x7fa4f8102710 2024-04-12T05:43:10.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.668+0000 7fa4ff345700 1 -- 172.21.15.32:0/1879860643 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4f8108d10 con 0x7fa4e006ccd0 2024-04-12T05:43:10.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.669+0000 7fa4f5ffb700 1 -- 172.21.15.32:0/1879860643 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fa4f8108d10 con 0x7fa4e006ccd0 2024-04-12T05:43:10.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.671+0000 7fa4ff345700 1 -- 172.21.15.32:0/1879860643 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa4e006ccd0 msgr2=0x7fa4e006f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:10.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.671+0000 7fa4ff345700 1 --2- 172.21.15.32:0/1879860643 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa4e006ccd0 0x7fa4e006f180 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa4e8007b20 tx=0x7fa4e8009e20 comp rx=0 tx=0).stop 2024-04-12T05:43:10.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.671+0000 7fa4ff345700 1 -- 172.21.15.32:0/1879860643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4f8102710 msgr2=0x7fa4f8113f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:10.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.671+0000 7fa4ff345700 1 --2- 172.21.15.32:0/1879860643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4f8102710 0x7fa4f8113f40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec009390 tx=0x7fa4ec009e30 comp rx=0 tx=0).stop 2024-04-12T05:43:10.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.671+0000 7fa4ff345700 1 -- 172.21.15.32:0/1879860643 shutdown_connections 2024-04-12T05:43:10.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.671+0000 7fa4ff345700 1 --2- 172.21.15.32:0/1879860643 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa4e006ccd0 0x7fa4e006f180 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:10.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.671+0000 7fa4ff345700 1 --2- 172.21.15.32:0/1879860643 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4f8101d70 0x7fa4f8113a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:10.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.671+0000 7fa4ff345700 1 --2- 172.21.15.32:0/1879860643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4f8102710 0x7fa4f8113f40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:10.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.671+0000 7fa4ff345700 1 -- 172.21.15.32:0/1879860643 >> 172.21.15.32:0/1879860643 conn(0x7fa4f8072bc0 msgr2=0x7fa4f8065970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:10.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.671+0000 7fa4ff345700 1 -- 172.21.15.32:0/1879860643 shutdown_connections 2024-04-12T05:43:10.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.671+0000 7fa4ff345700 1 -- 172.21.15.32:0/1879860643 wait complete. 2024-04-12T05:43:10.681 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:43:10.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.782+0000 7f911f621700 1 -- 172.21.15.32:0/1918373328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91180fe820 msgr2=0x7f9118106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:10.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.782+0000 7f911f621700 1 --2- 172.21.15.32:0/1918373328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91180fe820 0x7f9118106e60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9114004940 tx=0x7f9114009e30 comp rx=0 tx=0).stop 2024-04-12T05:43:10.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.782+0000 7f911f621700 1 -- 172.21.15.32:0/1918373328 shutdown_connections 2024-04-12T05:43:10.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.782+0000 7f911f621700 1 --2- 172.21.15.32:0/1918373328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91180fe820 0x7f9118106e60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:10.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.782+0000 7f911f621700 1 --2- 172.21.15.32:0/1918373328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91180fde80 0x7f91180fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:10.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.782+0000 7f911f621700 1 -- 172.21.15.32:0/1918373328 >> 172.21.15.32:0/1918373328 conn(0x7f91180f78d0 msgr2=0x7f91180f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:10.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.782+0000 7f911f621700 1 -- 172.21.15.32:0/1918373328 shutdown_connections 2024-04-12T05:43:10.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.782+0000 7f911f621700 1 -- 172.21.15.32:0/1918373328 wait complete. 2024-04-12T05:43:10.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.783+0000 7f911f621700 1 Processor -- start 2024-04-12T05:43:10.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.783+0000 7f911f621700 1 -- start start 2024-04-12T05:43:10.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.783+0000 7f911f621700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91180fde80 0x7f9118113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:10.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.783+0000 7f911f621700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91180fe820 0x7f9118113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:10.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.783+0000 7f911f621700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91181145f0 con 0x7f91180fde80 2024-04-12T05:43:10.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.783+0000 7f911f621700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911810d9e0 con 0x7f91180fe820 2024-04-12T05:43:10.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.783+0000 7f911d3bd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91180fde80 0x7f9118113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:10.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.783+0000 7f911d3bd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91180fde80 0x7f9118113a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59014/0 (socket says 172.21.15.32:59014) 2024-04-12T05:43:10.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.783+0000 7f911d3bd700 1 -- 172.21.15.32:0/177355339 learned_addr learned my addr 172.21.15.32:0/177355339 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:43:10.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.783+0000 7f911cbbc700 1 --2- 172.21.15.32:0/177355339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91180fe820 0x7f9118113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:10.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.784+0000 7f911d3bd700 1 -- 172.21.15.32:0/177355339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91180fe820 msgr2=0x7f9118113fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:10.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.784+0000 7f911d3bd700 1 --2- 172.21.15.32:0/177355339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91180fe820 0x7f9118113fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:10.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.784+0000 7f911d3bd700 1 -- 172.21.15.32:0/177355339 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9114005040 con 0x7f91180fde80 2024-04-12T05:43:10.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.784+0000 7f911d3bd700 1 --2- 172.21.15.32:0/177355339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91180fde80 0x7f9118113a60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9108002a50 tx=0x7f91080046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:10.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.784+0000 7f910dffb700 1 -- 172.21.15.32:0/177355339 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9108007cc0 con 0x7f91180fde80 2024-04-12T05:43:10.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.784+0000 7f911f621700 1 -- 172.21.15.32:0/177355339 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f911810dcc0 con 0x7f91180fde80 2024-04-12T05:43:10.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.784+0000 7f910dffb700 1 -- 172.21.15.32:0/177355339 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f910800f040 con 0x7f91180fde80 2024-04-12T05:43:10.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.784+0000 7f910dffb700 1 -- 172.21.15.32:0/177355339 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f910800d9c0 con 0x7f91180fde80 2024-04-12T05:43:10.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.784+0000 7f911f621700 1 -- 172.21.15.32:0/177355339 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f911810e1d0 con 0x7f91180fde80 2024-04-12T05:43:10.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.786+0000 7f910dffb700 1 -- 172.21.15.32:0/177355339 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7f910800db20 con 0x7f91180fde80 2024-04-12T05:43:10.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.786+0000 7f911f621700 1 -- 172.21.15.32:0/177355339 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9118066710 con 0x7f91180fde80 2024-04-12T05:43:10.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.786+0000 7f910dffb700 1 --2- 172.21.15.32:0/177355339 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f910406cd40 0x7f910406f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:10.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.786+0000 7f910dffb700 1 -- 172.21.15.32:0/177355339 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f910808bfa0 con 0x7f91180fde80 2024-04-12T05:43:10.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.786+0000 7f911cbbc700 1 --2- 172.21.15.32:0/177355339 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f910406cd40 0x7f910406f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:10.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.787+0000 7f911cbbc700 1 --2- 172.21.15.32:0/177355339 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f910406cd40 0x7f910406f1f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f911810ecc0 tx=0x7f9114009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:10.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:10.794+0000 7f910dffb700 1 -- 172.21.15.32:0/177355339 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9108059820 con 0x7f91180fde80 2024-04-12T05:43:11.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.037+0000 7f911f621700 1 -- 172.21.15.32:0/177355339 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9118109d00 con 0x7f910406cd40 2024-04-12T05:43:11.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.038+0000 7f910dffb700 1 -- 172.21.15.32:0/177355339 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f9118109d00 con 0x7f910406cd40 2024-04-12T05:43:11.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.040+0000 7f911f621700 1 -- 172.21.15.32:0/177355339 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f910406cd40 msgr2=0x7f910406f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:11.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.040+0000 7f911f621700 1 --2- 172.21.15.32:0/177355339 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f910406cd40 0x7f910406f1f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f911810ecc0 tx=0x7f9114009e30 comp rx=0 tx=0).stop 2024-04-12T05:43:11.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.040+0000 7f911f621700 1 -- 172.21.15.32:0/177355339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91180fde80 msgr2=0x7f9118113a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:11.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.040+0000 7f911f621700 1 --2- 172.21.15.32:0/177355339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91180fde80 0x7f9118113a60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9108002a50 tx=0x7f91080046b0 comp rx=0 tx=0).stop 2024-04-12T05:43:11.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.041+0000 7f911f621700 1 -- 172.21.15.32:0/177355339 shutdown_connections 2024-04-12T05:43:11.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.041+0000 7f911f621700 1 --2- 172.21.15.32:0/177355339 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f910406cd40 0x7f910406f1f0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:11.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.041+0000 7f911f621700 1 --2- 172.21.15.32:0/177355339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91180fde80 0x7f9118113a60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:11.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.041+0000 7f911f621700 1 --2- 172.21.15.32:0/177355339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91180fe820 0x7f9118113fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:11.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.041+0000 7f911f621700 1 -- 172.21.15.32:0/177355339 >> 172.21.15.32:0/177355339 conn(0x7f91180f78d0 msgr2=0x7f91180f8f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:11.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.041+0000 7f911f621700 1 -- 172.21.15.32:0/177355339 shutdown_connections 2024-04-12T05:43:11.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.041+0000 7f911f621700 1 -- 172.21.15.32:0/177355339 wait complete. 2024-04-12T05:43:11.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.150+0000 7fed37bfd700 1 -- 172.21.15.32:0/1160700436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed30101d70 msgr2=0x7fed30102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:11.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.150+0000 7fed37bfd700 1 --2- 172.21.15.32:0/1160700436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed30101d70 0x7fed30102140 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fed2c004940 tx=0x7fed2c009e30 comp rx=0 tx=0).stop 2024-04-12T05:43:11.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.150+0000 7fed37bfd700 1 -- 172.21.15.32:0/1160700436 shutdown_connections 2024-04-12T05:43:11.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.150+0000 7fed37bfd700 1 --2- 172.21.15.32:0/1160700436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed30102710 0x7fed300fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:11.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.150+0000 7fed37bfd700 1 --2- 172.21.15.32:0/1160700436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed30101d70 0x7fed30102140 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:11.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.150+0000 7fed37bfd700 1 -- 172.21.15.32:0/1160700436 >> 172.21.15.32:0/1160700436 conn(0x7fed30072bc0 msgr2=0x7fed30072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:11.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.150+0000 7fed37bfd700 1 -- 172.21.15.32:0/1160700436 shutdown_connections 2024-04-12T05:43:11.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.150+0000 7fed37bfd700 1 -- 172.21.15.32:0/1160700436 wait complete. 2024-04-12T05:43:11.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.151+0000 7fed37bfd700 1 Processor -- start 2024-04-12T05:43:11.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.151+0000 7fed37bfd700 1 -- start start 2024-04-12T05:43:11.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.151+0000 7fed37bfd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed30101d70 0x7fed30102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:11.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.151+0000 7fed37bfd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed30102710 0x7fed30103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:11.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.151+0000 7fed37bfd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed30103650 con 0x7fed30102710 2024-04-12T05:43:11.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.151+0000 7fed37bfd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed30103790 con 0x7fed30101d70 2024-04-12T05:43:11.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.151+0000 7fed35198700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed30102710 0x7fed30103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:11.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.151+0000 7fed35198700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed30102710 0x7fed30103110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59032/0 (socket says 172.21.15.32:59032) 2024-04-12T05:43:11.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.151+0000 7fed35198700 1 -- 172.21.15.32:0/820475370 learned_addr learned my addr 172.21.15.32:0/820475370 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:43:11.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.151+0000 7fed35999700 1 --2- 172.21.15.32:0/820475370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed30101d70 0x7fed30102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:11.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.151+0000 7fed35198700 1 -- 172.21.15.32:0/820475370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed30101d70 msgr2=0x7fed30102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:11.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.151+0000 7fed35198700 1 --2- 172.21.15.32:0/820475370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed30101d70 0x7fed30102bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:11.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.151+0000 7fed35198700 1 -- 172.21.15.32:0/820475370 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed2c005040 con 0x7fed30102710 2024-04-12T05:43:11.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.152+0000 7fed35198700 1 --2- 172.21.15.32:0/820475370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed30102710 0x7fed30103110 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fed20002a50 tx=0x7fed20004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:11.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.153+0000 7fed267fc700 1 -- 172.21.15.32:0/820475370 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed20007cc0 con 0x7fed30102710 2024-04-12T05:43:11.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.153+0000 7fed267fc700 1 -- 172.21.15.32:0/820475370 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed2000f040 con 0x7fed30102710 2024-04-12T05:43:11.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.153+0000 7fed267fc700 1 -- 172.21.15.32:0/820475370 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed2000d920 con 0x7fed30102710 2024-04-12T05:43:11.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.153+0000 7fed37bfd700 1 -- 172.21.15.32:0/820475370 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed30103a40 con 0x7fed30102710 2024-04-12T05:43:11.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.153+0000 7fed37bfd700 1 -- 172.21.15.32:0/820475370 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed3011a210 con 0x7fed30102710 2024-04-12T05:43:11.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.154+0000 7fed37bfd700 1 -- 172.21.15.32:0/820475370 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed300fd4c0 con 0x7fed30102710 2024-04-12T05:43:11.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.154+0000 7fed267fc700 1 -- 172.21.15.32:0/820475370 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7fed20009680 con 0x7fed30102710 2024-04-12T05:43:11.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.155+0000 7fed267fc700 1 --2- 172.21.15.32:0/820475370 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fed1c06ce10 0x7fed1c06f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:11.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.155+0000 7fed267fc700 1 -- 172.21.15.32:0/820475370 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fed2008bc80 con 0x7fed30102710 2024-04-12T05:43:11.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.158+0000 7fed35999700 1 --2- 172.21.15.32:0/820475370 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fed1c06ce10 0x7fed1c06f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:11.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.158+0000 7fed35999700 1 --2- 172.21.15.32:0/820475370 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fed1c06ce10 0x7fed1c06f2c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fed2c004f90 tx=0x7fed2c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:11.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.159+0000 7fed267fc700 1 -- 172.21.15.32:0/820475370 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fed20055b90 con 0x7fed30102710 2024-04-12T05:43:11.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:10 smithi067 ceph-mon[112979]: pgmap v46: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 64/333 objects degraded (19.219%) 2024-04-12T05:43:11.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:10 smithi067 ceph-mon[112979]: from='client.44119 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:11.278 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:10 smithi032 ceph-mon[146817]: pgmap v46: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 64/333 objects degraded (19.219%) 2024-04-12T05:43:11.278 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:10 smithi032 ceph-mon[146817]: from='client.44119 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:11.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.387+0000 7fed37bfd700 1 -- 172.21.15.32:0/820475370 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fed30103a40 con 0x7fed1c06ce10 2024-04-12T05:43:11.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.397+0000 7fed267fc700 1 -- 172.21.15.32:0/820475370 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fed30103a40 con 0x7fed1c06ce10 2024-04-12T05:43:11.394 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:43:11.394 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (2m) 18s ago 14m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:43:11.394 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (29s) 18s ago 14m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:43:11.394 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (24s) 20s ago 13m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:43:11.394 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (2m) 18s ago 14m 76.5M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:43:11.394 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (8m) 18s ago 8m 64.7M - 17.2.7-827-gd3b5060a 21de9b7369ad e7ada8e5259c 2024-04-12T05:43:11.394 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (8m) 18s ago 8m 115M - 17.2.7-827-gd3b5060a 21de9b7369ad 55124da9afdb 2024-04-12T05:43:11.394 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (8m) 20s ago 8m 62.6M - 17.2.7-827-gd3b5060a 21de9b7369ad 0c295d31a4a6 2024-04-12T05:43:11.394 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (8m) 20s ago 8m 120M - 17.2.7-827-gd3b5060a 21de9b7369ad 6eecd7deaac2 2024-04-12T05:43:11.394 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (4m) 18s ago 15m 493M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:43:11.395 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (3m) 20s ago 13m 447M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:43:11.395 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (94s) 18s ago 16m 41.5M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:43:11.395 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (67s) 20s ago 13m 33.0M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:43:11.395 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (3m) 18s ago 14m 16.4M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:43:11.395 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (3m) 20s ago 13m 18.0M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:43:11.395 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 starting - - - 1570M 2024-04-12T05:43:11.395 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (12m) 18s ago 12m 238M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad bf9f9a2db2d5 2024-04-12T05:43:11.395 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (11m) 18s ago 11m 191M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad f055cf1ca665 2024-04-12T05:43:11.395 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (11m) 20s ago 11m 258M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad 61b851235351 2024-04-12T05:43:11.395 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (10m) 20s ago 10m 230M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad 2c84d9047e9c 2024-04-12T05:43:11.395 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (10m) 20s ago 10m 220M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad e54839bb6055 2024-04-12T05:43:11.395 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (3m) 18s ago 13m 63.4M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:43:11.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.399+0000 7fed37bfd700 1 -- 172.21.15.32:0/820475370 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fed1c06ce10 msgr2=0x7fed1c06f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:11.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.399+0000 7fed37bfd700 1 --2- 172.21.15.32:0/820475370 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fed1c06ce10 0x7fed1c06f2c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fed2c004f90 tx=0x7fed2c009e30 comp rx=0 tx=0).stop 2024-04-12T05:43:11.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.399+0000 7fed37bfd700 1 -- 172.21.15.32:0/820475370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed30102710 msgr2=0x7fed30103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:11.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.399+0000 7fed37bfd700 1 --2- 172.21.15.32:0/820475370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed30102710 0x7fed30103110 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fed20002a50 tx=0x7fed20004600 comp rx=0 tx=0).stop 2024-04-12T05:43:11.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.399+0000 7fed37bfd700 1 -- 172.21.15.32:0/820475370 shutdown_connections 2024-04-12T05:43:11.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.399+0000 7fed37bfd700 1 --2- 172.21.15.32:0/820475370 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fed1c06ce10 0x7fed1c06f2c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:11.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.399+0000 7fed37bfd700 1 --2- 172.21.15.32:0/820475370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed30101d70 0x7fed30102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:11.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.399+0000 7fed37bfd700 1 --2- 172.21.15.32:0/820475370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed30102710 0x7fed30103110 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:11.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.399+0000 7fed37bfd700 1 -- 172.21.15.32:0/820475370 >> 172.21.15.32:0/820475370 conn(0x7fed30072bc0 msgr2=0x7fed30065a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:11.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.399+0000 7fed37bfd700 1 -- 172.21.15.32:0/820475370 shutdown_connections 2024-04-12T05:43:11.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.399+0000 7fed37bfd700 1 -- 172.21.15.32:0/820475370 wait complete. 2024-04-12T05:43:11.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.518+0000 7f45ccbe7700 1 -- 172.21.15.32:0/979722690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45c80fe7b0 msgr2=0x7f45c8106df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:11.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.518+0000 7f45ccbe7700 1 --2- 172.21.15.32:0/979722690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45c80fe7b0 0x7f45c8106df0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f45bc004940 tx=0x7f45bc009e30 comp rx=0 tx=0).stop 2024-04-12T05:43:11.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.518+0000 7f45ccbe7700 1 -- 172.21.15.32:0/979722690 shutdown_connections 2024-04-12T05:43:11.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.518+0000 7f45ccbe7700 1 --2- 172.21.15.32:0/979722690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45c80fe7b0 0x7f45c8106df0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:11.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.518+0000 7f45ccbe7700 1 --2- 172.21.15.32:0/979722690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45c80fde10 0x7f45c80fe1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:11.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.518+0000 7f45ccbe7700 1 -- 172.21.15.32:0/979722690 >> 172.21.15.32:0/979722690 conn(0x7f45c80f7880 msgr2=0x7f45c80f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:11.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.518+0000 7f45ccbe7700 1 -- 172.21.15.32:0/979722690 shutdown_connections 2024-04-12T05:43:11.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.518+0000 7f45ccbe7700 1 -- 172.21.15.32:0/979722690 wait complete. 2024-04-12T05:43:11.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.519+0000 7f45ccbe7700 1 Processor -- start 2024-04-12T05:43:11.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.519+0000 7f45ccbe7700 1 -- start start 2024-04-12T05:43:11.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.519+0000 7f45ccbe7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45c80fde10 0x7f45c80fbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:11.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.519+0000 7f45ccbe7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45c80fe7b0 0x7f45c80fc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:11.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.519+0000 7f45ccbe7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c80fc6c0 con 0x7f45c80fe7b0 2024-04-12T05:43:11.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.519+0000 7f45ccbe7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c80fc800 con 0x7f45c80fde10 2024-04-12T05:43:11.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.519+0000 7f45c5d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45c80fe7b0 0x7f45c80fc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:11.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.519+0000 7f45c5d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45c80fe7b0 0x7f45c80fc180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54564/0 (socket says 172.21.15.32:54564) 2024-04-12T05:43:11.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.519+0000 7f45c5d9b700 1 -- 172.21.15.32:0/3677460283 learned_addr learned my addr 172.21.15.32:0/3677460283 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:43:11.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.520+0000 7f45c659c700 1 --2- 172.21.15.32:0/3677460283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45c80fde10 0x7f45c80fbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:11.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.520+0000 7f45c5d9b700 1 -- 172.21.15.32:0/3677460283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45c80fde10 msgr2=0x7f45c80fbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:11.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.520+0000 7f45c5d9b700 1 --2- 172.21.15.32:0/3677460283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45c80fde10 0x7f45c80fbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:11.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.520+0000 7f45c5d9b700 1 -- 172.21.15.32:0/3677460283 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45bc005040 con 0x7f45c80fe7b0 2024-04-12T05:43:11.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.520+0000 7f45c5d9b700 1 --2- 172.21.15.32:0/3677460283 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45c80fe7b0 0x7f45c80fc180 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f45bc000c00 tx=0x7f45bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:11.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.520+0000 7f45b6ffd700 1 -- 172.21.15.32:0/3677460283 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45bc021070 con 0x7f45c80fe7b0 2024-04-12T05:43:11.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.520+0000 7f45b6ffd700 1 -- 172.21.15.32:0/3677460283 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f45bc013e00 con 0x7f45c80fe7b0 2024-04-12T05:43:11.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.520+0000 7f45ccbe7700 1 -- 172.21.15.32:0/3677460283 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45c80fca50 con 0x7f45c80fe7b0 2024-04-12T05:43:11.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.520+0000 7f45b6ffd700 1 -- 172.21.15.32:0/3677460283 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45bc00e070 con 0x7f45c80fe7b0 2024-04-12T05:43:11.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.520+0000 7f45ccbe7700 1 -- 172.21.15.32:0/3677460283 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45c811a390 con 0x7f45c80fe7b0 2024-04-12T05:43:11.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.521+0000 7f45ccbe7700 1 -- 172.21.15.32:0/3677460283 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45c8066770 con 0x7f45c80fe7b0 2024-04-12T05:43:11.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.522+0000 7f45b6ffd700 1 -- 172.21.15.32:0/3677460283 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7f45bc013950 con 0x7f45c80fe7b0 2024-04-12T05:43:11.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.522+0000 7f45b6ffd700 1 --2- 172.21.15.32:0/3677460283 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f45b006cd40 0x7f45b006f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:11.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.522+0000 7f45b6ffd700 1 -- 172.21.15.32:0/3677460283 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f45bc08d1f0 con 0x7f45c80fe7b0 2024-04-12T05:43:11.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.523+0000 7f45c659c700 1 --2- 172.21.15.32:0/3677460283 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f45b006cd40 0x7f45b006f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:11.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.523+0000 7f45c659c700 1 --2- 172.21.15.32:0/3677460283 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f45b006cd40 0x7f45b006f1f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f45b8004600 tx=0x7f45b800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:11.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.528+0000 7f45b6ffd700 1 -- 172.21.15.32:0/3677460283 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f45bc057100 con 0x7f45c80fe7b0 2024-04-12T05:43:11.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.831+0000 7f45ccbe7700 1 -- 172.21.15.32:0/3677460283 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f45c804bed0 con 0x7f45c80fe7b0 2024-04-12T05:43:11.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.831+0000 7f45b6ffd700 1 -- 172.21.15.32:0/3677460283 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+741 (secure 0 0 0) 0x7f45bc05a720 con 0x7f45c80fe7b0 2024-04-12T05:43:11.829 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:43:11.829 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:43:11.829 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:43:11.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:43:11.829 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:43:11.829 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:43:11.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:43:11.829 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:43:11.829 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 5 2024-04-12T05:43:11.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:43:11.830 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:43:11.830 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 4 2024-04-12T05:43:11.830 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:43:11.830 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:43:11.830 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 9, 2024-04-12T05:43:11.830 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:43:11.830 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:43:11.830 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:43:11.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.834+0000 7f45ccbe7700 1 -- 172.21.15.32:0/3677460283 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f45b006cd40 msgr2=0x7f45b006f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:11.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.834+0000 7f45ccbe7700 1 --2- 172.21.15.32:0/3677460283 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f45b006cd40 0x7f45b006f1f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f45b8004600 tx=0x7f45b800b040 comp rx=0 tx=0).stop 2024-04-12T05:43:11.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.834+0000 7f45ccbe7700 1 -- 172.21.15.32:0/3677460283 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45c80fe7b0 msgr2=0x7f45c80fc180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:11.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.834+0000 7f45ccbe7700 1 --2- 172.21.15.32:0/3677460283 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45c80fe7b0 0x7f45c80fc180 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f45bc000c00 tx=0x7f45bc009e30 comp rx=0 tx=0).stop 2024-04-12T05:43:11.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.834+0000 7f45ccbe7700 1 -- 172.21.15.32:0/3677460283 shutdown_connections 2024-04-12T05:43:11.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.834+0000 7f45ccbe7700 1 --2- 172.21.15.32:0/3677460283 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f45b006cd40 0x7f45b006f1f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:11.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.834+0000 7f45ccbe7700 1 --2- 172.21.15.32:0/3677460283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45c80fde10 0x7f45c80fbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:11.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.834+0000 7f45ccbe7700 1 --2- 172.21.15.32:0/3677460283 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45c80fe7b0 0x7f45c80fc180 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:11.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.834+0000 7f45ccbe7700 1 -- 172.21.15.32:0/3677460283 >> 172.21.15.32:0/3677460283 conn(0x7f45c80f7880 msgr2=0x7f45c80f8ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:11.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.834+0000 7f45ccbe7700 1 -- 172.21.15.32:0/3677460283 shutdown_connections 2024-04-12T05:43:11.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.834+0000 7f45ccbe7700 1 -- 172.21.15.32:0/3677460283 wait complete. 2024-04-12T05:43:11.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.946+0000 7efe7ab57700 1 -- 172.21.15.32:0/2188114350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe74102710 msgr2=0x7efe740fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:11.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.946+0000 7efe7ab57700 1 --2- 172.21.15.32:0/2188114350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe74102710 0x7efe740fbd80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efe60004940 tx=0x7efe60009e30 comp rx=0 tx=0).stop 2024-04-12T05:43:11.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.946+0000 7efe7ab57700 1 -- 172.21.15.32:0/2188114350 shutdown_connections 2024-04-12T05:43:11.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.946+0000 7efe7ab57700 1 --2- 172.21.15.32:0/2188114350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe74102710 0x7efe740fbd80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:11.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.946+0000 7efe7ab57700 1 --2- 172.21.15.32:0/2188114350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe74101d70 0x7efe74102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:11.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.946+0000 7efe7ab57700 1 -- 172.21.15.32:0/2188114350 >> 172.21.15.32:0/2188114350 conn(0x7efe74072bc0 msgr2=0x7efe74072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:11.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.946+0000 7efe7ab57700 1 -- 172.21.15.32:0/2188114350 shutdown_connections 2024-04-12T05:43:11.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.946+0000 7efe7ab57700 1 -- 172.21.15.32:0/2188114350 wait complete. 2024-04-12T05:43:11.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.946+0000 7efe7ab57700 1 Processor -- start 2024-04-12T05:43:11.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.946+0000 7efe7ab57700 1 -- start start 2024-04-12T05:43:11.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.947+0000 7efe7ab57700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe74101d70 0x7efe74113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:11.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.947+0000 7efe7ab57700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe74102710 0x7efe74114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:11.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.947+0000 7efe7ab57700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe74114650 con 0x7efe74101d70 2024-04-12T05:43:11.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.947+0000 7efe7ab57700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7410da40 con 0x7efe74102710 2024-04-12T05:43:11.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.947+0000 7efe788f3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe74101d70 0x7efe74113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:11.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.947+0000 7efe788f3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe74101d70 0x7efe74113ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54584/0 (socket says 172.21.15.32:54584) 2024-04-12T05:43:11.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.947+0000 7efe788f3700 1 -- 172.21.15.32:0/4075266341 learned_addr learned my addr 172.21.15.32:0/4075266341 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:43:11.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.947+0000 7efe73fff700 1 --2- 172.21.15.32:0/4075266341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe74102710 0x7efe74114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:11.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.947+0000 7efe788f3700 1 -- 172.21.15.32:0/4075266341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe74102710 msgr2=0x7efe74114000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:11.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.947+0000 7efe788f3700 1 --2- 172.21.15.32:0/4075266341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe74102710 0x7efe74114000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:11.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.947+0000 7efe788f3700 1 -- 172.21.15.32:0/4075266341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe60005040 con 0x7efe74101d70 2024-04-12T05:43:11.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.948+0000 7efe788f3700 1 --2- 172.21.15.32:0/4075266341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe74101d70 0x7efe74113ac0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7efe68002a50 tx=0x7efe680046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:11.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.948+0000 7efe717fa700 1 -- 172.21.15.32:0/4075266341 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe68007cc0 con 0x7efe74101d70 2024-04-12T05:43:11.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.948+0000 7efe7ab57700 1 -- 172.21.15.32:0/4075266341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe7410dd20 con 0x7efe74101d70 2024-04-12T05:43:11.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.948+0000 7efe717fa700 1 -- 172.21.15.32:0/4075266341 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efe6800f040 con 0x7efe74101d70 2024-04-12T05:43:11.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.948+0000 7efe717fa700 1 -- 172.21.15.32:0/4075266341 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe6800d9c0 con 0x7efe74101d70 2024-04-12T05:43:11.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.948+0000 7efe7ab57700 1 -- 172.21.15.32:0/4075266341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe7410e2b0 con 0x7efe74101d70 2024-04-12T05:43:11.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.949+0000 7efe7ab57700 1 -- 172.21.15.32:0/4075266341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe740fd4c0 con 0x7efe74101d70 2024-04-12T05:43:11.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.950+0000 7efe717fa700 1 -- 172.21.15.32:0/4075266341 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7efe6800db20 con 0x7efe74101d70 2024-04-12T05:43:11.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.950+0000 7efe717fa700 1 --2- 172.21.15.32:0/4075266341 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7efe6406cd40 0x7efe6406f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:11.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.950+0000 7efe717fa700 1 -- 172.21.15.32:0/4075266341 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6410+0+0 (secure 0 0 0) 0x7efe6808bf50 con 0x7efe74101d70 2024-04-12T05:43:11.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.951+0000 7efe73fff700 1 --2- 172.21.15.32:0/4075266341 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7efe6406cd40 0x7efe6406f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:11.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.952+0000 7efe73fff700 1 --2- 172.21.15.32:0/4075266341 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7efe6406cd40 0x7efe6406f1f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efe60004890 tx=0x7efe60009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:11.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:11.956+0000 7efe717fa700 1 -- 172.21.15.32:0/4075266341 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7efe680bfab0 con 0x7efe74101d70 2024-04-12T05:43:12.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.208+0000 7efe7ab57700 1 -- 172.21.15.32:0/4075266341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7efe74108d60 con 0x7efe74101d70 2024-04-12T05:43:12.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.209+0000 7efe717fa700 1 -- 172.21.15.32:0/4075266341 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1864 (secure 0 0 0) 0x7efe680598c0 con 0x7efe74101d70 2024-04-12T05:43:12.206 INFO:teuthology.orchestra.run.smithi032.stdout:e22 2024-04-12T05:43:12.206 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:43:12.206 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:43:12.206 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:43:12.206 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:43:12.207 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:43:12.207 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:43:12.207 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 22 2024-04-12T05:43:12.207 INFO:teuthology.orchestra.run.smithi032.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-12T05:43:12.207 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:43:12.207 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:41:05.358509+0000 2024-04-12T05:43:12.207 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:43:12.207 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:43:12.207 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:43:12.207 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:43:12.207 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:43:12.207 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:43:12.207 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:43:12.207 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:43:12.207 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 46 2024-04-12T05:43:12.207 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:43:12.208 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:43:12.208 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:43:12.208 INFO:teuthology.orchestra.run.smithi032.stdout:up {0=14480,1=24281} 2024-04-12T05:43:12.208 INFO:teuthology.orchestra.run.smithi032.stdout:failed 2024-04-12T05:43:12.208 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:43:12.208 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:43:12.208 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:43:12.208 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:43:12.208 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:43:12.208 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:43:12.208 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:43:12.208 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:43:12.208 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{0:14480} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.32:6828/387754149,v1:172.21.15.32:6829/387754149] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:43:12.208 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{0:14490} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3678031004,v1:172.21.15.32:6827/3678031004] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:43:12.208 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{1:24281} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.67:6824/1291914599,v1:172.21.15.67:6825/1291914599] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:43:12.209 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{1:24295} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.67:6826/873395842,v1:172.21.15.67:6827/873395842] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:43:12.209 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:43:12.209 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:43:12.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.211+0000 7efe7ab57700 1 -- 172.21.15.32:0/4075266341 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7efe6406cd40 msgr2=0x7efe6406f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:12.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.211+0000 7efe7ab57700 1 --2- 172.21.15.32:0/4075266341 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7efe6406cd40 0x7efe6406f1f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efe60004890 tx=0x7efe60009e30 comp rx=0 tx=0).stop 2024-04-12T05:43:12.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.211+0000 7efe7ab57700 1 -- 172.21.15.32:0/4075266341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe74101d70 msgr2=0x7efe74113ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:12.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.211+0000 7efe7ab57700 1 --2- 172.21.15.32:0/4075266341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe74101d70 0x7efe74113ac0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7efe68002a50 tx=0x7efe680046b0 comp rx=0 tx=0).stop 2024-04-12T05:43:12.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.211+0000 7efe7ab57700 1 -- 172.21.15.32:0/4075266341 shutdown_connections 2024-04-12T05:43:12.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.211+0000 7efe7ab57700 1 --2- 172.21.15.32:0/4075266341 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7efe6406cd40 0x7efe6406f1f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:12.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.211+0000 7efe7ab57700 1 --2- 172.21.15.32:0/4075266341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe74101d70 0x7efe74113ac0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:12.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.211+0000 7efe7ab57700 1 --2- 172.21.15.32:0/4075266341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe74102710 0x7efe74114000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:12.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.211+0000 7efe7ab57700 1 -- 172.21.15.32:0/4075266341 >> 172.21.15.32:0/4075266341 conn(0x7efe74072bc0 msgr2=0x7efe740659e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:12.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.211+0000 7efe7ab57700 1 -- 172.21.15.32:0/4075266341 shutdown_connections 2024-04-12T05:43:12.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.211+0000 7efe7ab57700 1 -- 172.21.15.32:0/4075266341 wait complete. 2024-04-12T05:43:12.210 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 22 2024-04-12T05:43:12.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:11 smithi067 ceph-mon[112979]: from='client.34190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:12.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:11 smithi067 ceph-mon[112979]: from='client.34194 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:12.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:11 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3677460283' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:43:12.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.333+0000 7f1f842e7700 1 -- 172.21.15.32:0/936348102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f7c1026b0 msgr2=0x7f1f7c0fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:12.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.333+0000 7f1f842e7700 1 --2- 172.21.15.32:0/936348102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f7c1026b0 0x7f1f7c0fbd20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c004940 tx=0x7f1f6c009e30 comp rx=0 tx=0).stop 2024-04-12T05:43:12.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.333+0000 7f1f842e7700 1 -- 172.21.15.32:0/936348102 shutdown_connections 2024-04-12T05:43:12.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.333+0000 7f1f842e7700 1 --2- 172.21.15.32:0/936348102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f7c1026b0 0x7f1f7c0fbd20 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:12.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.333+0000 7f1f842e7700 1 --2- 172.21.15.32:0/936348102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f7c101d10 0x7f1f7c1020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:12.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.333+0000 7f1f842e7700 1 -- 172.21.15.32:0/936348102 >> 172.21.15.32:0/936348102 conn(0x7f1f7c0f7880 msgr2=0x7f1f7c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:12.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.333+0000 7f1f842e7700 1 -- 172.21.15.32:0/936348102 shutdown_connections 2024-04-12T05:43:12.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.333+0000 7f1f842e7700 1 -- 172.21.15.32:0/936348102 wait complete. 2024-04-12T05:43:12.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.334+0000 7f1f842e7700 1 Processor -- start 2024-04-12T05:43:12.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.334+0000 7f1f842e7700 1 -- start start 2024-04-12T05:43:12.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.334+0000 7f1f842e7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f7c101d10 0x7f1f7c113dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:12.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.334+0000 7f1f842e7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f7c1026b0 0x7f1f7c114310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:12.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.334+0000 7f1f842e7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c114960 con 0x7f1f7c101d10 2024-04-12T05:43:12.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.334+0000 7f1f842e7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c10dd30 con 0x7f1f7c1026b0 2024-04-12T05:43:12.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.334+0000 7f1f82083700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f7c101d10 0x7f1f7c113dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:12.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.334+0000 7f1f82083700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f7c101d10 0x7f1f7c113dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54600/0 (socket says 172.21.15.32:54600) 2024-04-12T05:43:12.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.334+0000 7f1f82083700 1 -- 172.21.15.32:0/1058996276 learned_addr learned my addr 172.21.15.32:0/1058996276 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:43:12.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.335+0000 7f1f81882700 1 --2- 172.21.15.32:0/1058996276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f7c1026b0 0x7f1f7c114310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:12.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.335+0000 7f1f82083700 1 -- 172.21.15.32:0/1058996276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f7c1026b0 msgr2=0x7f1f7c114310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:12.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.335+0000 7f1f82083700 1 --2- 172.21.15.32:0/1058996276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f7c1026b0 0x7f1f7c114310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:12.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.335+0000 7f1f82083700 1 -- 172.21.15.32:0/1058996276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f6c005040 con 0x7f1f7c101d10 2024-04-12T05:43:12.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.335+0000 7f1f81882700 1 --2- 172.21.15.32:0/1058996276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f7c1026b0 0x7f1f7c114310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:43:12.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.335+0000 7f1f82083700 1 --2- 172.21.15.32:0/1058996276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f7c101d10 0x7f1f7c113dd0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1f78002a50 tx=0x7f1f780046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:12.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.335+0000 7f1f72ffd700 1 -- 172.21.15.32:0/1058996276 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f78007cc0 con 0x7f1f7c101d10 2024-04-12T05:43:12.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.335+0000 7f1f842e7700 1 -- 172.21.15.32:0/1058996276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f7c10e010 con 0x7f1f7c101d10 2024-04-12T05:43:12.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.335+0000 7f1f72ffd700 1 -- 172.21.15.32:0/1058996276 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f7800f040 con 0x7f1f7c101d10 2024-04-12T05:43:12.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.335+0000 7f1f72ffd700 1 -- 172.21.15.32:0/1058996276 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f78009540 con 0x7f1f7c101d10 2024-04-12T05:43:12.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.335+0000 7f1f842e7700 1 -- 172.21.15.32:0/1058996276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f7c10e5d0 con 0x7f1f7c101d10 2024-04-12T05:43:12.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.336+0000 7f1f842e7700 1 -- 172.21.15.32:0/1058996276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f7c0fd460 con 0x7f1f7c101d10 2024-04-12T05:43:12.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.337+0000 7f1f72ffd700 1 -- 172.21.15.32:0/1058996276 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7f1f780096a0 con 0x7f1f7c101d10 2024-04-12T05:43:12.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.337+0000 7f1f72ffd700 1 --2- 172.21.15.32:0/1058996276 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1f6806cd40 0x7f1f6806f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:12.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.337+0000 7f1f72ffd700 1 -- 172.21.15.32:0/1058996276 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f1f7808c0b0 con 0x7f1f7c101d10 2024-04-12T05:43:12.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.338+0000 7f1f81882700 1 --2- 172.21.15.32:0/1058996276 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1f6806cd40 0x7f1f6806f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:12.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.338+0000 7f1f81882700 1 --2- 172.21.15.32:0/1058996276 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1f6806cd40 0x7f1f6806f1f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c009fd0 tx=0x7f1f6c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:12.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:11 smithi032 ceph-mon[146817]: from='client.34190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:12.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:11 smithi032 ceph-mon[146817]: from='client.34194 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:12.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:11 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3677460283' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:43:12.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.346+0000 7f1f72ffd700 1 -- 172.21.15.32:0/1058996276 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1f780599a0 con 0x7f1f7c101d10 2024-04-12T05:43:12.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.587+0000 7f1f842e7700 1 -- 172.21.15.32:0/1058996276 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f7c04bf40 con 0x7f1f6806cd40 2024-04-12T05:43:12.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.588+0000 7f1f72ffd700 1 -- 172.21.15.32:0/1058996276 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f1f7c04bf40 con 0x7f1f6806cd40 2024-04-12T05:43:12.585 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:43:12.585 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:43:12.585 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:43:12.585 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:43:12.585 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:43:12.585 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:43:12.586 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:43:12.586 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-12T05:43:12.586 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:43:12.586 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "6/21 daemons upgraded", 2024-04-12T05:43:12.586 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading osd daemons", 2024-04-12T05:43:12.586 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:43:12.586 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:43:12.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.590+0000 7f1f842e7700 1 -- 172.21.15.32:0/1058996276 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1f6806cd40 msgr2=0x7f1f6806f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:12.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.590+0000 7f1f842e7700 1 --2- 172.21.15.32:0/1058996276 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1f6806cd40 0x7f1f6806f1f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c009fd0 tx=0x7f1f6c009e30 comp rx=0 tx=0).stop 2024-04-12T05:43:12.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.590+0000 7f1f842e7700 1 -- 172.21.15.32:0/1058996276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f7c101d10 msgr2=0x7f1f7c113dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:12.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.590+0000 7f1f842e7700 1 --2- 172.21.15.32:0/1058996276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f7c101d10 0x7f1f7c113dd0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1f78002a50 tx=0x7f1f780046b0 comp rx=0 tx=0).stop 2024-04-12T05:43:12.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.590+0000 7f1f842e7700 1 -- 172.21.15.32:0/1058996276 shutdown_connections 2024-04-12T05:43:12.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.590+0000 7f1f842e7700 1 --2- 172.21.15.32:0/1058996276 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1f6806cd40 0x7f1f6806f1f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:12.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.590+0000 7f1f842e7700 1 --2- 172.21.15.32:0/1058996276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f7c101d10 0x7f1f7c113dd0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:12.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.590+0000 7f1f842e7700 1 --2- 172.21.15.32:0/1058996276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f7c1026b0 0x7f1f7c114310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:12.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.590+0000 7f1f842e7700 1 -- 172.21.15.32:0/1058996276 >> 172.21.15.32:0/1058996276 conn(0x7f1f7c0f7880 msgr2=0x7f1f7c0f8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:12.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.590+0000 7f1f842e7700 1 -- 172.21.15.32:0/1058996276 shutdown_connections 2024-04-12T05:43:12.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.590+0000 7f1f842e7700 1 -- 172.21.15.32:0/1058996276 wait complete. 2024-04-12T05:43:12.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.723+0000 7f27d4f1c700 1 -- 172.21.15.32:0/861647851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27d01037a0 msgr2=0x7f27d00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:12.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.723+0000 7f27d4f1c700 1 --2- 172.21.15.32:0/861647851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27d01037a0 0x7f27d00fce10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f27c0004940 tx=0x7f27c0009e30 comp rx=0 tx=0).stop 2024-04-12T05:43:12.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.723+0000 7f27d4f1c700 1 -- 172.21.15.32:0/861647851 shutdown_connections 2024-04-12T05:43:12.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.723+0000 7f27d4f1c700 1 --2- 172.21.15.32:0/861647851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27d01037a0 0x7f27d00fce10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:12.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.723+0000 7f27d4f1c700 1 --2- 172.21.15.32:0/861647851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d0102e00 0x7f27d01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:12.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.723+0000 7f27d4f1c700 1 -- 172.21.15.32:0/861647851 >> 172.21.15.32:0/861647851 conn(0x7f27d00f89b0 msgr2=0x7f27d00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:12.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.723+0000 7f27d4f1c700 1 -- 172.21.15.32:0/861647851 shutdown_connections 2024-04-12T05:43:12.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.723+0000 7f27d4f1c700 1 -- 172.21.15.32:0/861647851 wait complete. 2024-04-12T05:43:12.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.724+0000 7f27d4f1c700 1 Processor -- start 2024-04-12T05:43:12.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.724+0000 7f27d4f1c700 1 -- start start 2024-04-12T05:43:12.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.724+0000 7f27d4f1c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d0102e00 0x7f27d00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:12.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.724+0000 7f27d4f1c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27d01037a0 0x7f27d0070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:12.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.724+0000 7f27d4f1c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d0070d90 con 0x7f27d01037a0 2024-04-12T05:43:12.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.724+0000 7f27d4f1c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d0070ed0 con 0x7f27d0102e00 2024-04-12T05:43:12.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.724+0000 7f27cdd9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27d01037a0 0x7f27d0070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:12.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.724+0000 7f27cdd9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27d01037a0 0x7f27d0070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54616/0 (socket says 172.21.15.32:54616) 2024-04-12T05:43:12.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.724+0000 7f27cdd9b700 1 -- 172.21.15.32:0/1111236377 learned_addr learned my addr 172.21.15.32:0/1111236377 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:43:12.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.724+0000 7f27ce59c700 1 --2- 172.21.15.32:0/1111236377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d0102e00 0x7f27d00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:12.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.725+0000 7f27cdd9b700 1 -- 172.21.15.32:0/1111236377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d0102e00 msgr2=0x7f27d00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:12.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.725+0000 7f27cdd9b700 1 --2- 172.21.15.32:0/1111236377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d0102e00 0x7f27d00702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:12.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.725+0000 7f27cdd9b700 1 -- 172.21.15.32:0/1111236377 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c0005040 con 0x7f27d01037a0 2024-04-12T05:43:12.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.725+0000 7f27ce59c700 1 --2- 172.21.15.32:0/1111236377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d0102e00 0x7f27d00702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:43:12.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.725+0000 7f27cdd9b700 1 --2- 172.21.15.32:0/1111236377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27d01037a0 0x7f27d0070800 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f27c0000c00 tx=0x7f27c0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:12.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.725+0000 7f27c6ffd700 1 -- 172.21.15.32:0/1111236377 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27c000e070 con 0x7f27d01037a0 2024-04-12T05:43:12.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.725+0000 7f27c6ffd700 1 -- 172.21.15.32:0/1111236377 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f27c0013de0 con 0x7f27d01037a0 2024-04-12T05:43:12.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.725+0000 7f27c6ffd700 1 -- 172.21.15.32:0/1111236377 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27c001cde0 con 0x7f27d01037a0 2024-04-12T05:43:12.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.725+0000 7f27d4f1c700 1 -- 172.21.15.32:0/1111236377 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27d0071120 con 0x7f27d01037a0 2024-04-12T05:43:12.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.725+0000 7f27d4f1c700 1 -- 172.21.15.32:0/1111236377 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27d011a140 con 0x7f27d01037a0 2024-04-12T05:43:12.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.727+0000 7f27c4ff9700 1 -- 172.21.15.32:0/1111236377 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27b40052b0 con 0x7f27d01037a0 2024-04-12T05:43:12.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.727+0000 7f27c6ffd700 1 -- 172.21.15.32:0/1111236377 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 39) v1 ==== 90952+0+0 (secure 0 0 0) 0x7f27c001c600 con 0x7f27d01037a0 2024-04-12T05:43:12.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.728+0000 7f27c6ffd700 1 --2- 172.21.15.32:0/1111236377 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f27bc06cd40 0x7f27bc06f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:12.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.728+0000 7f27c6ffd700 1 -- 172.21.15.32:0/1111236377 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f27c008d6d0 con 0x7f27d01037a0 2024-04-12T05:43:12.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.728+0000 7f27ce59c700 1 --2- 172.21.15.32:0/1111236377 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f27bc06cd40 0x7f27bc06f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:12.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.728+0000 7f27ce59c700 1 --2- 172.21.15.32:0/1111236377 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f27bc06cd40 0x7f27bc06f1f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f27b8004600 tx=0x7f27b800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:12.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:12.736+0000 7f27c6ffd700 1 -- 172.21.15.32:0/1111236377 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f27c00575e0 con 0x7f27d01037a0 2024-04-12T05:43:13.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:13.060+0000 7f27c4ff9700 1 -- 172.21.15.32:0/1111236377 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f27b4005830 con 0x7f27d01037a0 2024-04-12T05:43:13.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:13.061+0000 7f27c6ffd700 1 -- 172.21.15.32:0/1111236377 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1328 (secure 0 0 0) 0x7f27c005ac00 con 0x7f27d01037a0 2024-04-12T05:43:13.058 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 64/333 objects degraded (19.219%), 19 pgs degraded 2024-04-12T05:43:13.059 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-12T05:43:13.059 INFO:teuthology.orchestra.run.smithi032.stdout: osd.0 (root=default,host=smithi032) is down 2024-04-12T05:43:13.059 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 64/333 objects degraded (19.219%), 19 pgs degraded 2024-04-12T05:43:13.059 INFO:teuthology.orchestra.run.smithi032.stdout: pg 1.0 is active+undersized+degraded, acting [3,1] 2024-04-12T05:43:13.059 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.0 is active+undersized+degraded, acting [3,1] 2024-04-12T05:43:13.059 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.1 is active+undersized+degraded, acting [2,1] 2024-04-12T05:43:13.060 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.2 is active+undersized+degraded, acting [5,1] 2024-04-12T05:43:13.060 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.4 is active+undersized+degraded, acting [1,4] 2024-04-12T05:43:13.060 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.5 is active+undersized+degraded, acting [3,4] 2024-04-12T05:43:13.060 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.8 is active+undersized+degraded, acting [3,5] 2024-04-12T05:43:13.060 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.9 is active+undersized+degraded, acting [1,4] 2024-04-12T05:43:13.060 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.e is active+undersized+degraded, acting [2,3] 2024-04-12T05:43:13.061 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.f is active+undersized+degraded, acting [4,5] 2024-04-12T05:43:13.061 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.10 is active+undersized+degraded, acting [2,1] 2024-04-12T05:43:13.061 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.12 is active+undersized+degraded, acting [3,1] 2024-04-12T05:43:13.061 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.13 is active+undersized+degraded, acting [4,2] 2024-04-12T05:43:13.061 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.15 is active+undersized+degraded, acting [1,3] 2024-04-12T05:43:13.061 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.19 is active+undersized+degraded, acting [4,2] 2024-04-12T05:43:13.061 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.1b is active+undersized+degraded, acting [1,5] 2024-04-12T05:43:13.061 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.1d is active+undersized+degraded, acting [3,5] 2024-04-12T05:43:13.061 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.1e is active+undersized+degraded, acting [2,5] 2024-04-12T05:43:13.061 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.1f is active+undersized+degraded, acting [3,4] 2024-04-12T05:43:13.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:13.063+0000 7f27c4ff9700 1 -- 172.21.15.32:0/1111236377 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f27bc06cd40 msgr2=0x7f27bc06f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:13.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:13.063+0000 7f27c4ff9700 1 --2- 172.21.15.32:0/1111236377 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f27bc06cd40 0x7f27bc06f1f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f27b8004600 tx=0x7f27b800b040 comp rx=0 tx=0).stop 2024-04-12T05:43:13.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:13.063+0000 7f27c4ff9700 1 -- 172.21.15.32:0/1111236377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27d01037a0 msgr2=0x7f27d0070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:13.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:13.063+0000 7f27c4ff9700 1 --2- 172.21.15.32:0/1111236377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27d01037a0 0x7f27d0070800 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f27c0000c00 tx=0x7f27c0009e30 comp rx=0 tx=0).stop 2024-04-12T05:43:13.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:13.063+0000 7f27c4ff9700 1 -- 172.21.15.32:0/1111236377 shutdown_connections 2024-04-12T05:43:13.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:13.063+0000 7f27c4ff9700 1 --2- 172.21.15.32:0/1111236377 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f27bc06cd40 0x7f27bc06f1f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:13.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:13.063+0000 7f27c4ff9700 1 --2- 172.21.15.32:0/1111236377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d0102e00 0x7f27d00702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:13.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:13.063+0000 7f27c4ff9700 1 --2- 172.21.15.32:0/1111236377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27d01037a0 0x7f27d0070800 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:13.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:13.063+0000 7f27c4ff9700 1 -- 172.21.15.32:0/1111236377 >> 172.21.15.32:0/1111236377 conn(0x7f27d00f89b0 msgr2=0x7f27d00f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:13.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:13.063+0000 7f27c4ff9700 1 -- 172.21.15.32:0/1111236377 shutdown_connections 2024-04-12T05:43:13.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:13.063+0000 7f27c4ff9700 1 -- 172.21.15.32:0/1111236377 wait complete. 2024-04-12T05:43:13.088 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 05:43:12 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-0[161267]: 2024-04-12T05:43:12.959+0000 7f26cb1077c0 -1 osd.0 49 log_to_monitors true 2024-04-12T05:43:13.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:12 smithi032 ceph-mon[146817]: pgmap v47: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 64/333 objects degraded (19.219%) 2024-04-12T05:43:13.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:12 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/4075266341' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:43:13.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:12 smithi032 ceph-mon[146817]: from='client.34206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:13.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:12 smithi067 ceph-mon[112979]: pgmap v47: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 64/333 objects degraded (19.219%) 2024-04-12T05:43:13.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:12 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/4075266341' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:43:13.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:12 smithi067 ceph-mon[112979]: from='client.34206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:14.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:13 smithi067 ceph-mon[112979]: from='osd.0 [v2:172.21.15.32:6802/1783741888,v1:172.21.15.32:6803/1783741888]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-12T05:43:14.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:13 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1111236377' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:43:14.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:13 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:14.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:13 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:14.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:13 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:14.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:13 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:14.339 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 05:43:13 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-0[161267]: 2024-04-12T05:43:13.939+0000 7f26bc561700 -1 osd.0 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T05:43:14.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:13 smithi032 ceph-mon[146817]: from='osd.0 [v2:172.21.15.32:6802/1783741888,v1:172.21.15.32:6803/1783741888]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-12T05:43:14.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:13 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1111236377' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:43:14.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:13 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:14.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:13 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:14.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:13 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:14.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:13 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:15.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:14 smithi067 ceph-mon[112979]: pgmap v48: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s; 64/333 objects degraded (19.219%) 2024-04-12T05:43:15.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:14 smithi067 ceph-mon[112979]: from='osd.0 [v2:172.21.15.32:6802/1783741888,v1:172.21.15.32:6803/1783741888]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-12T05:43:15.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:14 smithi067 ceph-mon[112979]: osdmap e52: 6 total, 5 up, 6 in 2024-04-12T05:43:15.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:14 smithi067 ceph-mon[112979]: from='osd.0 [v2:172.21.15.32:6802/1783741888,v1:172.21.15.32:6803/1783741888]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-12T05:43:15.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:14 smithi032 ceph-mon[146817]: pgmap v48: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 270 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s; 64/333 objects degraded (19.219%) 2024-04-12T05:43:15.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:14 smithi032 ceph-mon[146817]: from='osd.0 [v2:172.21.15.32:6802/1783741888,v1:172.21.15.32:6803/1783741888]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-12T05:43:15.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:14 smithi032 ceph-mon[146817]: osdmap e52: 6 total, 5 up, 6 in 2024-04-12T05:43:15.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:14 smithi032 ceph-mon[146817]: from='osd.0 [v2:172.21.15.32:6802/1783741888,v1:172.21.15.32:6803/1783741888]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-12T05:43:16.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:15 smithi067 ceph-mon[112979]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T05:43:16.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:15 smithi067 ceph-mon[112979]: osd.0 [v2:172.21.15.32:6802/1783741888,v1:172.21.15.32:6803/1783741888] boot 2024-04-12T05:43:16.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:15 smithi067 ceph-mon[112979]: osdmap e53: 6 total, 6 up, 6 in 2024-04-12T05:43:16.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:15 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:43:16.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:15 smithi067 ceph-mon[112979]: osdmap e54: 6 total, 6 up, 6 in 2024-04-12T05:43:16.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:15 smithi032 ceph-mon[146817]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T05:43:16.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:15 smithi032 ceph-mon[146817]: osd.0 [v2:172.21.15.32:6802/1783741888,v1:172.21.15.32:6803/1783741888] boot 2024-04-12T05:43:16.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:15 smithi032 ceph-mon[146817]: osdmap e53: 6 total, 6 up, 6 in 2024-04-12T05:43:16.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:15 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:43:16.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:15 smithi032 ceph-mon[146817]: osdmap e54: 6 total, 6 up, 6 in 2024-04-12T05:43:17.079 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:16 smithi032 ceph-mon[146817]: pgmap v51: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s; 64/333 objects degraded (19.219%) 2024-04-12T05:43:17.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:16 smithi067 ceph-mon[112979]: pgmap v51: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s; 64/333 objects degraded (19.219%) 2024-04-12T05:43:18.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:18.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:18.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:43:18.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:43:18.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:18.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:43:18.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:43:18.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:43:18.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:43:18.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-12T05:43:18.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:18 smithi032 ceph-mon[146817]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-12T05:43:18.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:18 smithi032 ceph-mon[146817]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-04-12T05:43:18.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:18.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:43:19.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:19.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:19.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:43:19.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:43:19.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:19.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:43:19.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:43:19.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:43:19.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:43:19.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-12T05:43:19.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:18 smithi067 ceph-mon[112979]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-12T05:43:19.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:18 smithi067 ceph-mon[112979]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-04-12T05:43:19.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:19.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:43:19.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:19 smithi032 ceph-mon[146817]: pgmap v53: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 64/333 objects degraded (19.219%) 2024-04-12T05:43:19.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:19 smithi032 ceph-mon[146817]: mgrmap e40: smithi032.hrwxrh(active, since 91s), standbys: smithi067.dwichg 2024-04-12T05:43:20.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:19 smithi067 ceph-mon[112979]: pgmap v53: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 64/333 objects degraded (19.219%) 2024-04-12T05:43:20.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:19 smithi067 ceph-mon[112979]: mgrmap e40: smithi032.hrwxrh(active, since 91s), standbys: smithi067.dwichg 2024-04-12T05:43:21.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:20 smithi067 ceph-mon[112979]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 64/333 objects degraded (19.219%), 19 pgs degraded) 2024-04-12T05:43:21.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:20 smithi067 ceph-mon[112979]: Cluster is now healthy 2024-04-12T05:43:21.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:20 smithi032 ceph-mon[146817]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 64/333 objects degraded (19.219%), 19 pgs degraded) 2024-04-12T05:43:21.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:20 smithi032 ceph-mon[146817]: Cluster is now healthy 2024-04-12T05:43:22.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:21 smithi067 ceph-mon[112979]: pgmap v54: 65 pgs: 65 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 853 B/s wr, 4 op/s 2024-04-12T05:43:22.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:21 smithi032 ceph-mon[146817]: pgmap v54: 65 pgs: 65 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 853 B/s wr, 4 op/s 2024-04-12T05:43:23.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:23 smithi032 ceph-mon[146817]: pgmap v55: 65 pgs: 65 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.5 KiB/s rd, 650 B/s wr, 3 op/s 2024-04-12T05:43:24.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:23 smithi067 ceph-mon[112979]: pgmap v55: 65 pgs: 65 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.5 KiB/s rd, 650 B/s wr, 3 op/s 2024-04-12T05:43:26.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:25 smithi067 ceph-mon[112979]: pgmap v56: 65 pgs: 65 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 576 B/s wr, 5 op/s 2024-04-12T05:43:26.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:25 smithi032 ceph-mon[146817]: pgmap v56: 65 pgs: 65 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 576 B/s wr, 5 op/s 2024-04-12T05:43:28.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:27 smithi067 ceph-mon[112979]: pgmap v57: 65 pgs: 65 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.2 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-12T05:43:28.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:27 smithi032 ceph-mon[146817]: pgmap v57: 65 pgs: 65 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.2 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-12T05:43:29.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:28 smithi067 ceph-mon[112979]: pgmap v58: 65 pgs: 65 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 431 B/s wr, 3 op/s 2024-04-12T05:43:29.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:28 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:29.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:28 smithi032 ceph-mon[146817]: pgmap v58: 65 pgs: 65 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 431 B/s wr, 3 op/s 2024-04-12T05:43:29.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:28 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:31.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:30 smithi067 ceph-mon[112979]: pgmap v59: 65 pgs: 65 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-12T05:43:31.341 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:30 smithi032 ceph-mon[146817]: pgmap v59: 65 pgs: 65 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-12T05:43:33.182 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:32 smithi032 ceph-mon[146817]: pgmap v60: 65 pgs: 65 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:43:33.182 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:32 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-12T05:43:33.182 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:32 smithi032 ceph-mon[146817]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-12T05:43:33.182 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:32 smithi032 ceph-mon[146817]: Upgrade: osd.1 is safe to restart 2024-04-12T05:43:33.182 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:32 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:33.182 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:32 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:43:33.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:32 smithi067 ceph-mon[112979]: pgmap v60: 65 pgs: 65 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:43:33.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:32 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-12T05:43:33.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:32 smithi067 ceph-mon[112979]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-12T05:43:33.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:32 smithi067 ceph-mon[112979]: Upgrade: osd.1 is safe to restart 2024-04-12T05:43:33.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:32 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:33.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:32 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:43:35.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:35 smithi067 ceph-mon[112979]: pgmap v61: 65 pgs: 65 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:43:35.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:35 smithi067 ceph-mon[112979]: Upgrade: Updating osd.1 2024-04-12T05:43:35.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:35 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:35.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:35 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-12T05:43:35.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:35 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:43:35.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:35 smithi067 ceph-mon[112979]: Deploying daemon osd.1 on smithi032 2024-04-12T05:43:35.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:35 smithi032 ceph-mon[146817]: pgmap v61: 65 pgs: 65 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:43:35.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:35 smithi032 ceph-mon[146817]: Upgrade: Updating osd.1 2024-04-12T05:43:35.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:35 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:35.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:35 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-12T05:43:35.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:35 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:43:35.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:35 smithi032 ceph-mon[146817]: Deploying daemon osd.1 on smithi032 2024-04-12T05:43:37.457 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:37 smithi032 ceph-mon[146817]: pgmap v62: 65 pgs: 65 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:43:37.730 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:43:37 smithi032 systemd[1]: Stopping Ceph osd.1 for 37b58ce2-f88d-11ee-bc8c-c7b262605968... 2024-04-12T05:43:37.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:37 smithi067 ceph-mon[112979]: pgmap v62: 65 pgs: 65 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:43:38.089 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:43:37 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-1[73003]: 2024-04-12T05:43:37.732+0000 7f48a4adc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T05:43:38.089 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:43:37 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-1[73003]: 2024-04-12T05:43:37.732+0000 7f48a4adc700 -1 osd.1 54 *** Got signal Terminated *** 2024-04-12T05:43:38.089 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:43:37 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-1[73003]: 2024-04-12T05:43:37.732+0000 7f48a4adc700 -1 osd.1 54 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T05:43:38.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:38 smithi032 ceph-mon[146817]: osd.1 marked itself down and dead 2024-04-12T05:43:38.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:38 smithi067 ceph-mon[112979]: osd.1 marked itself down and dead 2024-04-12T05:43:39.283 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:43:39 smithi032 bash[167356]: ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-1 2024-04-12T05:43:39.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:39 smithi032 ceph-mon[146817]: pgmap v63: 65 pgs: 65 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:43:39.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:39 smithi032 ceph-mon[146817]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T05:43:39.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:39 smithi032 ceph-mon[146817]: osdmap e55: 6 total, 5 up, 6 in 2024-04-12T05:43:39.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:39 smithi067 ceph-mon[112979]: pgmap v63: 65 pgs: 65 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:43:39.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:39 smithi067 ceph-mon[112979]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T05:43:39.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:39 smithi067 ceph-mon[112979]: osdmap e55: 6 total, 5 up, 6 in 2024-04-12T05:43:40.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:40 smithi032 ceph-mon[146817]: osdmap e56: 6 total, 5 up, 6 in 2024-04-12T05:43:40.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:40 smithi067 ceph-mon[112979]: osdmap e56: 6 total, 5 up, 6 in 2024-04-12T05:43:41.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:41 smithi032 ceph-mon[146817]: pgmap v66: 65 pgs: 34 peering, 31 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:43:41.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:41 smithi032 ceph-mon[146817]: Health check failed: Reduced data availability: 12 pgs peering (PG_AVAILABILITY) 2024-04-12T05:43:41.339 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:43:41 smithi032 systemd[1]: ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@osd.1.service: Succeeded. 2024-04-12T05:43:41.339 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:43:41 smithi032 systemd[1]: Stopped Ceph osd.1 for 37b58ce2-f88d-11ee-bc8c-c7b262605968. 2024-04-12T05:43:41.339 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:43:41 smithi032 systemd[1]: Starting Ceph osd.1 for 37b58ce2-f88d-11ee-bc8c-c7b262605968... 2024-04-12T05:43:41.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:41 smithi067 ceph-mon[112979]: pgmap v66: 65 pgs: 34 peering, 31 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:43:41.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:41 smithi067 ceph-mon[112979]: Health check failed: Reduced data availability: 12 pgs peering (PG_AVAILABILITY) 2024-04-12T05:43:42.573 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:43:42 smithi032 bash[167627]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-12T05:43:42.573 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:43:42 smithi032 bash[167627]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-12T05:43:42.573 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:43:42 smithi032 bash[167627]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-12T05:43:42.573 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:43:42 smithi032 bash[167627]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-12T05:43:42.573 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:43:42 smithi032 bash[167627]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-12T05:43:42.573 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:43:42 smithi032 bash[167627]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-12T05:43:42.573 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:43:42 smithi032 bash[167627]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-12T05:43:43.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.175+0000 7f399efe3700 1 -- 172.21.15.32:0/2211859228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3998102e10 msgr2=0x7f39981031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:43.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.175+0000 7f399efe3700 1 --2- 172.21.15.32:0/2211859228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3998102e10 0x7f39981031e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f398c004940 tx=0x7f398c009e30 comp rx=0 tx=0).stop 2024-04-12T05:43:43.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.175+0000 7f399efe3700 1 -- 172.21.15.32:0/2211859228 shutdown_connections 2024-04-12T05:43:43.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.175+0000 7f399efe3700 1 --2- 172.21.15.32:0/2211859228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39981037b0 0x7f39980fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:43.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.175+0000 7f399efe3700 1 --2- 172.21.15.32:0/2211859228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3998102e10 0x7f39981031e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:43.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.175+0000 7f399efe3700 1 -- 172.21.15.32:0/2211859228 >> 172.21.15.32:0/2211859228 conn(0x7f39980f89a0 msgr2=0x7f39980fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:43.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.175+0000 7f399efe3700 1 -- 172.21.15.32:0/2211859228 shutdown_connections 2024-04-12T05:43:43.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.175+0000 7f399efe3700 1 -- 172.21.15.32:0/2211859228 wait complete. 2024-04-12T05:43:43.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.176+0000 7f399efe3700 1 Processor -- start 2024-04-12T05:43:43.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.176+0000 7f399efe3700 1 -- start start 2024-04-12T05:43:43.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.176+0000 7f399efe3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3998102e10 0x7f3998113b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:43.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.176+0000 7f399efe3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39981037b0 0x7f3998114050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:43.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.176+0000 7f399efe3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39981146a0 con 0x7f39981037b0 2024-04-12T05:43:43.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.176+0000 7f399efe3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f399810da90 con 0x7f3998102e10 2024-04-12T05:43:43.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.176+0000 7f3997fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39981037b0 0x7f3998114050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:43.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.176+0000 7f3997fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39981037b0 0x7f3998114050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33448/0 (socket says 172.21.15.32:33448) 2024-04-12T05:43:43.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.176+0000 7f3997fff700 1 -- 172.21.15.32:0/2912377588 learned_addr learned my addr 172.21.15.32:0/2912377588 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:43:43.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.177+0000 7f399cd7f700 1 --2- 172.21.15.32:0/2912377588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3998102e10 0x7f3998113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:43.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.177+0000 7f3997fff700 1 -- 172.21.15.32:0/2912377588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3998102e10 msgr2=0x7f3998113b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:43.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.177+0000 7f3997fff700 1 --2- 172.21.15.32:0/2912377588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3998102e10 0x7f3998113b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:43.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.177+0000 7f3997fff700 1 -- 172.21.15.32:0/2912377588 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f398c005040 con 0x7f39981037b0 2024-04-12T05:43:43.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.177+0000 7f399cd7f700 1 --2- 172.21.15.32:0/2912377588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3998102e10 0x7f3998113b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:43:43.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.177+0000 7f3997fff700 1 --2- 172.21.15.32:0/2912377588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39981037b0 0x7f3998114050 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3984002a80 tx=0x7f3984004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:43.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.177+0000 7f39957fa700 1 -- 172.21.15.32:0/2912377588 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3984007cc0 con 0x7f39981037b0 2024-04-12T05:43:43.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.177+0000 7f399efe3700 1 -- 172.21.15.32:0/2912377588 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f399810dd70 con 0x7f39981037b0 2024-04-12T05:43:43.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.177+0000 7f39957fa700 1 -- 172.21.15.32:0/2912377588 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f398400f040 con 0x7f39981037b0 2024-04-12T05:43:43.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.177+0000 7f39957fa700 1 -- 172.21.15.32:0/2912377588 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f398400da40 con 0x7f39981037b0 2024-04-12T05:43:43.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.178+0000 7f399efe3700 1 -- 172.21.15.32:0/2912377588 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f399810e330 con 0x7f39981037b0 2024-04-12T05:43:43.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.179+0000 7f39957fa700 1 -- 172.21.15.32:0/2912377588 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f398400dba0 con 0x7f39981037b0 2024-04-12T05:43:43.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.179+0000 7f399efe3700 1 -- 172.21.15.32:0/2912377588 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39980fe560 con 0x7f39981037b0 2024-04-12T05:43:43.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.179+0000 7f39957fa700 1 --2- 172.21.15.32:0/2912377588 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f398806cc30 0x7f398806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:43.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.179+0000 7f39957fa700 1 -- 172.21.15.32:0/2912377588 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f398408b8b0 con 0x7f39981037b0 2024-04-12T05:43:43.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.185+0000 7f399cd7f700 1 --2- 172.21.15.32:0/2912377588 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f398806cc30 0x7f398806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:43.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.185+0000 7f399cd7f700 1 --2- 172.21.15.32:0/2912377588 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f398806cc30 0x7f398806f0e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f398c00b3b0 tx=0x7f398c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:43.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.187+0000 7f39957fa700 1 -- 172.21.15.32:0/2912377588 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3984059350 con 0x7f39981037b0 2024-04-12T05:43:43.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.415+0000 7f399efe3700 1 -- 172.21.15.32:0/2912377588 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3998109e60 con 0x7f398806cc30 2024-04-12T05:43:43.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.416+0000 7f39957fa700 1 -- 172.21.15.32:0/2912377588 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f3998109e60 con 0x7f398806cc30 2024-04-12T05:43:43.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.418+0000 7f399efe3700 1 -- 172.21.15.32:0/2912377588 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f398806cc30 msgr2=0x7f398806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:43.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.418+0000 7f399efe3700 1 --2- 172.21.15.32:0/2912377588 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f398806cc30 0x7f398806f0e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f398c00b3b0 tx=0x7f398c009e30 comp rx=0 tx=0).stop 2024-04-12T05:43:43.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.418+0000 7f399efe3700 1 -- 172.21.15.32:0/2912377588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39981037b0 msgr2=0x7f3998114050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:43.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.418+0000 7f399efe3700 1 --2- 172.21.15.32:0/2912377588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39981037b0 0x7f3998114050 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3984002a80 tx=0x7f3984004630 comp rx=0 tx=0).stop 2024-04-12T05:43:43.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.419+0000 7f399efe3700 1 -- 172.21.15.32:0/2912377588 shutdown_connections 2024-04-12T05:43:43.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.419+0000 7f399efe3700 1 --2- 172.21.15.32:0/2912377588 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f398806cc30 0x7f398806f0e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:43.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.419+0000 7f399efe3700 1 --2- 172.21.15.32:0/2912377588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3998102e10 0x7f3998113b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:43.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.419+0000 7f399efe3700 1 --2- 172.21.15.32:0/2912377588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39981037b0 0x7f3998114050 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:43.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.419+0000 7f399efe3700 1 -- 172.21.15.32:0/2912377588 >> 172.21.15.32:0/2912377588 conn(0x7f39980f89a0 msgr2=0x7f39980f9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:43.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.419+0000 7f399efe3700 1 -- 172.21.15.32:0/2912377588 shutdown_connections 2024-04-12T05:43:43.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.419+0000 7f399efe3700 1 -- 172.21.15.32:0/2912377588 wait complete. 2024-04-12T05:43:43.426 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:43:43.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.531+0000 7f3b149f4700 1 -- 172.21.15.32:0/4238583468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b10102e00 msgr2=0x7f3b101031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:43.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.531+0000 7f3b149f4700 1 --2- 172.21.15.32:0/4238583468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b10102e00 0x7f3b101031d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3b04004940 tx=0x7f3b04009e30 comp rx=0 tx=0).stop 2024-04-12T05:43:43.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.531+0000 7f3b149f4700 1 -- 172.21.15.32:0/4238583468 shutdown_connections 2024-04-12T05:43:43.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.531+0000 7f3b149f4700 1 --2- 172.21.15.32:0/4238583468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b101037a0 0x7f3b100fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:43.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.531+0000 7f3b149f4700 1 --2- 172.21.15.32:0/4238583468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b10102e00 0x7f3b101031d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:43.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.531+0000 7f3b149f4700 1 -- 172.21.15.32:0/4238583468 >> 172.21.15.32:0/4238583468 conn(0x7f3b100f89b0 msgr2=0x7f3b100fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:43.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.531+0000 7f3b149f4700 1 -- 172.21.15.32:0/4238583468 shutdown_connections 2024-04-12T05:43:43.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.531+0000 7f3b149f4700 1 -- 172.21.15.32:0/4238583468 wait complete. 2024-04-12T05:43:43.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.531+0000 7f3b149f4700 1 Processor -- start 2024-04-12T05:43:43.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.532+0000 7f3b149f4700 1 -- start start 2024-04-12T05:43:43.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.532+0000 7f3b149f4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b10102e00 0x7f3b10113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:43.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.532+0000 7f3b149f4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b101037a0 0x7f3b10114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:43.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.532+0000 7f3b149f4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b101146b0 con 0x7f3b101037a0 2024-04-12T05:43:43.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.532+0000 7f3b149f4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1010daa0 con 0x7f3b10102e00 2024-04-12T05:43:43.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.532+0000 7f3b0dd9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b101037a0 0x7f3b10114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:43.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.532+0000 7f3b0dd9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b101037a0 0x7f3b10114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33472/0 (socket says 172.21.15.32:33472) 2024-04-12T05:43:43.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.532+0000 7f3b0dd9b700 1 -- 172.21.15.32:0/1990313506 learned_addr learned my addr 172.21.15.32:0/1990313506 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:43:43.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.532+0000 7f3b0e59c700 1 --2- 172.21.15.32:0/1990313506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b10102e00 0x7f3b10113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:43.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.532+0000 7f3b0dd9b700 1 -- 172.21.15.32:0/1990313506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b10102e00 msgr2=0x7f3b10113b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:43.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.532+0000 7f3b0dd9b700 1 --2- 172.21.15.32:0/1990313506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b10102e00 0x7f3b10113b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:43.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.532+0000 7f3b0dd9b700 1 -- 172.21.15.32:0/1990313506 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b04005040 con 0x7f3b101037a0 2024-04-12T05:43:43.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.532+0000 7f3b0dd9b700 1 --2- 172.21.15.32:0/1990313506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b101037a0 0x7f3b10114060 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3b0000c890 tx=0x7f3b000044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:43.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.533+0000 7f3afeffd700 1 -- 172.21.15.32:0/1990313506 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b0000a700 con 0x7f3b101037a0 2024-04-12T05:43:43.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.533+0000 7f3b149f4700 1 -- 172.21.15.32:0/1990313506 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b1010dd80 con 0x7f3b101037a0 2024-04-12T05:43:43.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.533+0000 7f3afeffd700 1 -- 172.21.15.32:0/1990313506 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3b0000ad10 con 0x7f3b101037a0 2024-04-12T05:43:43.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.533+0000 7f3afeffd700 1 -- 172.21.15.32:0/1990313506 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b00012e70 con 0x7f3b101037a0 2024-04-12T05:43:43.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.533+0000 7f3b149f4700 1 -- 172.21.15.32:0/1990313506 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b1010e340 con 0x7f3b101037a0 2024-04-12T05:43:43.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.534+0000 7f3b149f4700 1 -- 172.21.15.32:0/1990313506 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b100fe550 con 0x7f3b101037a0 2024-04-12T05:43:43.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.534+0000 7f3afeffd700 1 -- 172.21.15.32:0/1990313506 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3b0000a860 con 0x7f3b101037a0 2024-04-12T05:43:43.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.535+0000 7f3afeffd700 1 --2- 172.21.15.32:0/1990313506 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3af806cc20 0x7f3af806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:43.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.535+0000 7f3b0e59c700 1 --2- 172.21.15.32:0/1990313506 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3af806cc20 0x7f3af806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:43.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.535+0000 7f3afeffd700 1 -- 172.21.15.32:0/1990313506 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f3b0008c060 con 0x7f3b101037a0 2024-04-12T05:43:43.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.535+0000 7f3b0e59c700 1 --2- 172.21.15.32:0/1990313506 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3af806cc20 0x7f3af806f0d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3b04004f90 tx=0x7f3b04009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:43.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.540+0000 7f3afeffd700 1 -- 172.21.15.32:0/1990313506 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3b00059b80 con 0x7f3b101037a0 2024-04-12T05:43:43.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:43 smithi032 ceph-mon[146817]: pgmap v67: 65 pgs: 34 peering, 31 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:43:43.589 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:43:43 smithi032 ceph-osd[167975]: -- 172.21.15.32:0/1666127380 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56489422d1e0 con 0x5648941fcc00 2024-04-12T05:43:43.589 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:43:43 smithi032 bash[167914]: 2e7e5c6470fde7c6e40eaf208957da6dbdabd7ab8abf1c915d12b4f153cf7f69 2024-04-12T05:43:43.589 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:43:43 smithi032 systemd[1]: Started Ceph osd.1 for 37b58ce2-f88d-11ee-bc8c-c7b262605968. 2024-04-12T05:43:43.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:43 smithi067 ceph-mon[112979]: pgmap v67: 65 pgs: 34 peering, 31 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:43:43.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.771+0000 7f3b149f4700 1 -- 172.21.15.32:0/1990313506 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b10109dc0 con 0x7f3af806cc20 2024-04-12T05:43:43.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.772+0000 7f3afeffd700 1 -- 172.21.15.32:0/1990313506 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f3b10109dc0 con 0x7f3af806cc20 2024-04-12T05:43:43.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.776+0000 7f3b149f4700 1 -- 172.21.15.32:0/1990313506 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3af806cc20 msgr2=0x7f3af806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:43.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.776+0000 7f3b149f4700 1 --2- 172.21.15.32:0/1990313506 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3af806cc20 0x7f3af806f0d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3b04004f90 tx=0x7f3b04009e30 comp rx=0 tx=0).stop 2024-04-12T05:43:43.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.776+0000 7f3b149f4700 1 -- 172.21.15.32:0/1990313506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b101037a0 msgr2=0x7f3b10114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:43.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.776+0000 7f3b149f4700 1 --2- 172.21.15.32:0/1990313506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b101037a0 0x7f3b10114060 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3b0000c890 tx=0x7f3b000044c0 comp rx=0 tx=0).stop 2024-04-12T05:43:43.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.776+0000 7f3b149f4700 1 -- 172.21.15.32:0/1990313506 shutdown_connections 2024-04-12T05:43:43.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.776+0000 7f3b149f4700 1 --2- 172.21.15.32:0/1990313506 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3af806cc20 0x7f3af806f0d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:43.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.776+0000 7f3b149f4700 1 --2- 172.21.15.32:0/1990313506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b10102e00 0x7f3b10113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:43.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.776+0000 7f3b149f4700 1 --2- 172.21.15.32:0/1990313506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b101037a0 0x7f3b10114060 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:43.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.776+0000 7f3b149f4700 1 -- 172.21.15.32:0/1990313506 >> 172.21.15.32:0/1990313506 conn(0x7f3b100f89b0 msgr2=0x7f3b100f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:43.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.776+0000 7f3b149f4700 1 -- 172.21.15.32:0/1990313506 shutdown_connections 2024-04-12T05:43:43.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.776+0000 7f3b149f4700 1 -- 172.21.15.32:0/1990313506 wait complete. 2024-04-12T05:43:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.899+0000 7f655bae3700 1 -- 172.21.15.32:0/3601160677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65540fbbd0 msgr2=0x7f65540fbfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.899+0000 7f655bae3700 1 --2- 172.21.15.32:0/3601160677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65540fbbd0 0x7f65540fbfa0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6550004940 tx=0x7f6550009e30 comp rx=0 tx=0).stop 2024-04-12T05:43:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.900+0000 7f655bae3700 1 -- 172.21.15.32:0/3601160677 shutdown_connections 2024-04-12T05:43:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.900+0000 7f655bae3700 1 --2- 172.21.15.32:0/3601160677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65540fc4e0 0x7f6554109240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.900+0000 7f655bae3700 1 --2- 172.21.15.32:0/3601160677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65540fbbd0 0x7f65540fbfa0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.900+0000 7f655bae3700 1 -- 172.21.15.32:0/3601160677 >> 172.21.15.32:0/3601160677 conn(0x7f65540f7880 msgr2=0x7f65540f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.900+0000 7f655bae3700 1 -- 172.21.15.32:0/3601160677 shutdown_connections 2024-04-12T05:43:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.900+0000 7f655bae3700 1 -- 172.21.15.32:0/3601160677 wait complete. 2024-04-12T05:43:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.900+0000 7f655bae3700 1 Processor -- start 2024-04-12T05:43:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.900+0000 7f655bae3700 1 -- start start 2024-04-12T05:43:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.901+0000 7f655bae3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65540fbbd0 0x7f6554115f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.901+0000 7f655bae3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65540fc4e0 0x7f6554116440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.901+0000 7f655bae3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6554116b20 con 0x7f65540fbbd0 2024-04-12T05:43:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.901+0000 7f655bae3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655410fe60 con 0x7f65540fc4e0 2024-04-12T05:43:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.901+0000 7f655987f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65540fbbd0 0x7f6554115f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.901+0000 7f655987f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65540fbbd0 0x7f6554115f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33496/0 (socket says 172.21.15.32:33496) 2024-04-12T05:43:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.901+0000 7f655987f700 1 -- 172.21.15.32:0/2762929174 learned_addr learned my addr 172.21.15.32:0/2762929174 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:43:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.901+0000 7f655907e700 1 --2- 172.21.15.32:0/2762929174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65540fc4e0 0x7f6554116440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.901+0000 7f655987f700 1 -- 172.21.15.32:0/2762929174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65540fc4e0 msgr2=0x7f6554116440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.901+0000 7f655987f700 1 --2- 172.21.15.32:0/2762929174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65540fc4e0 0x7f6554116440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.901+0000 7f655987f700 1 -- 172.21.15.32:0/2762929174 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6550005040 con 0x7f65540fbbd0 2024-04-12T05:43:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.901+0000 7f655907e700 1 --2- 172.21.15.32:0/2762929174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65540fc4e0 0x7f6554116440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:43:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.901+0000 7f655987f700 1 --2- 172.21.15.32:0/2762929174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65540fbbd0 0x7f6554115f00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6550004890 tx=0x7f6550009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.902+0000 7f654a7fc700 1 -- 172.21.15.32:0/2762929174 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f655000e040 con 0x7f65540fbbd0 2024-04-12T05:43:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.902+0000 7f654a7fc700 1 -- 172.21.15.32:0/2762929174 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6550009af0 con 0x7f65540fbbd0 2024-04-12T05:43:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.902+0000 7f654a7fc700 1 -- 172.21.15.32:0/2762929174 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f655001c3f0 con 0x7f65540fbbd0 2024-04-12T05:43:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.902+0000 7f655bae3700 1 -- 172.21.15.32:0/2762929174 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65541100e0 con 0x7f65540fbbd0 2024-04-12T05:43:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.902+0000 7f655bae3700 1 -- 172.21.15.32:0/2762929174 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65541106a0 con 0x7f65540fbbd0 2024-04-12T05:43:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.903+0000 7f655bae3700 1 -- 172.21.15.32:0/2762929174 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6554107080 con 0x7f65540fbbd0 2024-04-12T05:43:43.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.904+0000 7f654a7fc700 1 -- 172.21.15.32:0/2762929174 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f6550013420 con 0x7f65540fbbd0 2024-04-12T05:43:43.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.905+0000 7f654a7fc700 1 --2- 172.21.15.32:0/2762929174 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f654006cc20 0x7f654006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:43.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.905+0000 7f654a7fc700 1 -- 172.21.15.32:0/2762929174 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f655008ddc0 con 0x7f65540fbbd0 2024-04-12T05:43:43.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.905+0000 7f655907e700 1 --2- 172.21.15.32:0/2762929174 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f654006cc20 0x7f654006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:43.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.906+0000 7f655907e700 1 --2- 172.21.15.32:0/2762929174 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f654006cc20 0x7f654006f0d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f65541117a0 tx=0x7f6544008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:43.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:43.910+0000 7f654a7fc700 1 -- 172.21.15.32:0/2762929174 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f655005b980 con 0x7f65540fbbd0 2024-04-12T05:43:44.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.142+0000 7f655bae3700 1 -- 172.21.15.32:0/2762929174 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f655410bfd0 con 0x7f654006cc20 2024-04-12T05:43:44.150 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:43:44.150 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (3m) 30s ago 14m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:43:44.150 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (62s) 30s ago 15m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:43:44.150 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (57s) 53s ago 14m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:43:44.150 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (3m) 30s ago 14m 77.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:43:44.150 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (9m) 30s ago 9m 64.7M - 17.2.7-827-gd3b5060a 21de9b7369ad e7ada8e5259c 2024-04-12T05:43:44.150 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (9m) 30s ago 9m 115M - 17.2.7-827-gd3b5060a 21de9b7369ad 55124da9afdb 2024-04-12T05:43:44.151 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (9m) 53s ago 9m 62.6M - 17.2.7-827-gd3b5060a 21de9b7369ad 0c295d31a4a6 2024-04-12T05:43:44.151 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (9m) 53s ago 9m 120M - 17.2.7-827-gd3b5060a 21de9b7369ad 6eecd7deaac2 2024-04-12T05:43:44.151 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (5m) 30s ago 16m 495M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:43:44.151 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (4m) 53s ago 13m 447M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:43:44.151 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (2m) 30s ago 16m 42.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:43:44.151 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (100s) 53s ago 13m 33.0M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:43:44.151 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (4m) 30s ago 15m 16.8M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:43:44.151 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (4m) 53s ago 14m 18.0M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:43:44.151 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (36s) 30s ago 13m 11.5M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T05:43:44.151 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 starting - - - 1570M 2024-04-12T05:43:44.151 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (12m) 30s ago 12m 190M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad f055cf1ca665 2024-04-12T05:43:44.151 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (11m) 53s ago 11m 258M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad 61b851235351 2024-04-12T05:43:44.151 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (11m) 53s ago 11m 230M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad 2c84d9047e9c 2024-04-12T05:43:44.151 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (10m) 53s ago 10m 220M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad e54839bb6055 2024-04-12T05:43:44.151 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (3m) 30s ago 14m 63.4M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:43:44.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.154+0000 7f654a7fc700 1 -- 172.21.15.32:0/2762929174 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f655410bfd0 con 0x7f654006cc20 2024-04-12T05:43:44.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.156+0000 7f655bae3700 1 -- 172.21.15.32:0/2762929174 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f654006cc20 msgr2=0x7f654006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:44.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.156+0000 7f655bae3700 1 --2- 172.21.15.32:0/2762929174 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f654006cc20 0x7f654006f0d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f65541117a0 tx=0x7f6544008040 comp rx=0 tx=0).stop 2024-04-12T05:43:44.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.156+0000 7f655bae3700 1 -- 172.21.15.32:0/2762929174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65540fbbd0 msgr2=0x7f6554115f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:44.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.156+0000 7f655bae3700 1 --2- 172.21.15.32:0/2762929174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65540fbbd0 0x7f6554115f00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6550004890 tx=0x7f6550009e30 comp rx=0 tx=0).stop 2024-04-12T05:43:44.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.156+0000 7f655bae3700 1 -- 172.21.15.32:0/2762929174 shutdown_connections 2024-04-12T05:43:44.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.156+0000 7f655bae3700 1 --2- 172.21.15.32:0/2762929174 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f654006cc20 0x7f654006f0d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:44.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.156+0000 7f655bae3700 1 --2- 172.21.15.32:0/2762929174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65540fbbd0 0x7f6554115f00 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:44.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.156+0000 7f655bae3700 1 --2- 172.21.15.32:0/2762929174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65540fc4e0 0x7f6554116440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:44.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.156+0000 7f655bae3700 1 -- 172.21.15.32:0/2762929174 >> 172.21.15.32:0/2762929174 conn(0x7f65540f7880 msgr2=0x7f6554104200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:44.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.156+0000 7f655bae3700 1 -- 172.21.15.32:0/2762929174 shutdown_connections 2024-04-12T05:43:44.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.156+0000 7f655bae3700 1 -- 172.21.15.32:0/2762929174 wait complete. 2024-04-12T05:43:44.244 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:43:43 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-1[167952]: 2024-04-12T05:43:43.904+0000 7f5ec9b317c0 -1 Falling back to public interface 2024-04-12T05:43:44.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.264+0000 7fcd19794700 1 -- 172.21.15.32:0/2650147686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd141037c0 msgr2=0x7fcd140fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:44.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.264+0000 7fcd19794700 1 --2- 172.21.15.32:0/2650147686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd141037c0 0x7fcd140fce30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcd08004940 tx=0x7fcd08009e30 comp rx=0 tx=0).stop 2024-04-12T05:43:44.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.264+0000 7fcd19794700 1 -- 172.21.15.32:0/2650147686 shutdown_connections 2024-04-12T05:43:44.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.264+0000 7fcd19794700 1 --2- 172.21.15.32:0/2650147686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd141037c0 0x7fcd140fce30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:44.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.264+0000 7fcd19794700 1 --2- 172.21.15.32:0/2650147686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd14102e20 0x7fcd141031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:44.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.264+0000 7fcd19794700 1 -- 172.21.15.32:0/2650147686 >> 172.21.15.32:0/2650147686 conn(0x7fcd140f89d0 msgr2=0x7fcd140fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:44.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.264+0000 7fcd19794700 1 -- 172.21.15.32:0/2650147686 shutdown_connections 2024-04-12T05:43:44.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.264+0000 7fcd19794700 1 -- 172.21.15.32:0/2650147686 wait complete. 2024-04-12T05:43:44.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.265+0000 7fcd19794700 1 Processor -- start 2024-04-12T05:43:44.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.265+0000 7fcd19794700 1 -- start start 2024-04-12T05:43:44.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.265+0000 7fcd19794700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd14102e20 0x7fcd14115c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:44.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.265+0000 7fcd19794700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd141037c0 0x7fcd14116180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:44.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.265+0000 7fcd19794700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd141167d0 con 0x7fcd14102e20 2024-04-12T05:43:44.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.265+0000 7fcd19794700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1410fba0 con 0x7fcd141037c0 2024-04-12T05:43:44.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.265+0000 7fcd12ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd14102e20 0x7fcd14115c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:44.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.265+0000 7fcd12ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd14102e20 0x7fcd14115c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33526/0 (socket says 172.21.15.32:33526) 2024-04-12T05:43:44.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.265+0000 7fcd12ffd700 1 -- 172.21.15.32:0/4032192141 learned_addr learned my addr 172.21.15.32:0/4032192141 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:43:44.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.265+0000 7fcd127fc700 1 --2- 172.21.15.32:0/4032192141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd141037c0 0x7fcd14116180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:44.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.266+0000 7fcd12ffd700 1 -- 172.21.15.32:0/4032192141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd141037c0 msgr2=0x7fcd14116180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:44.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.266+0000 7fcd12ffd700 1 --2- 172.21.15.32:0/4032192141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd141037c0 0x7fcd14116180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:44.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.266+0000 7fcd12ffd700 1 -- 172.21.15.32:0/4032192141 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd08005040 con 0x7fcd14102e20 2024-04-12T05:43:44.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.266+0000 7fcd12ffd700 1 --2- 172.21.15.32:0/4032192141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd14102e20 0x7fcd14115c40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcd04002a80 tx=0x7fcd040046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:44.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.266+0000 7fccfb7fe700 1 -- 172.21.15.32:0/4032192141 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd04007cc0 con 0x7fcd14102e20 2024-04-12T05:43:44.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.266+0000 7fccfb7fe700 1 -- 172.21.15.32:0/4032192141 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd0400f040 con 0x7fcd14102e20 2024-04-12T05:43:44.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.266+0000 7fccfb7fe700 1 -- 172.21.15.32:0/4032192141 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd0400da60 con 0x7fcd14102e20 2024-04-12T05:43:44.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.266+0000 7fcd19794700 1 -- 172.21.15.32:0/4032192141 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd1410fe80 con 0x7fcd14102e20 2024-04-12T05:43:44.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.266+0000 7fcd19794700 1 -- 172.21.15.32:0/4032192141 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd14110440 con 0x7fcd14102e20 2024-04-12T05:43:44.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.267+0000 7fcd19794700 1 -- 172.21.15.32:0/4032192141 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd140fe570 con 0x7fcd14102e20 2024-04-12T05:43:44.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.268+0000 7fccfb7fe700 1 -- 172.21.15.32:0/4032192141 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fcd0400dbc0 con 0x7fcd14102e20 2024-04-12T05:43:44.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.268+0000 7fccfb7fe700 1 --2- 172.21.15.32:0/4032192141 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fccfc06cc20 0x7fccfc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.269+0000 7fccfb7fe700 1 -- 172.21.15.32:0/4032192141 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fcd0408c1a0 con 0x7fcd14102e20 2024-04-12T05:43:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.269+0000 7fcd127fc700 1 --2- 172.21.15.32:0/4032192141 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fccfc06cc20 0x7fccfc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.269+0000 7fcd127fc700 1 --2- 172.21.15.32:0/4032192141 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fccfc06cc20 0x7fccfc06f0d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcd141114e0 tx=0x7fcd08009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:44.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.274+0000 7fccfb7fe700 1 -- 172.21.15.32:0/4032192141 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fcd04059ca0 con 0x7fcd14102e20 2024-04-12T05:43:44.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.562+0000 7fcd19794700 1 -- 172.21.15.32:0/4032192141 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fcd1404bed0 con 0x7fcd14102e20 2024-04-12T05:43:44.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.562+0000 7fccfb7fe700 1 -- 172.21.15.32:0/4032192141 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fcd04059830 con 0x7fcd14102e20 2024-04-12T05:43:44.558 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:43:44.558 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:43:44.559 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:43:44.559 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:43:44.559 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:43:44.559 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:43:44.559 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:43:44.559 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:43:44.559 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 4, 2024-04-12T05:43:44.559 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 1 2024-04-12T05:43:44.559 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:43:44.559 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:43:44.559 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 4 2024-04-12T05:43:44.559 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:43:44.559 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:43:44.559 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 8, 2024-04-12T05:43:44.560 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 5 2024-04-12T05:43:44.560 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:43:44.560 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:43:44.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.564+0000 7fcd19794700 1 -- 172.21.15.32:0/4032192141 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fccfc06cc20 msgr2=0x7fccfc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:44.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.564+0000 7fcd19794700 1 --2- 172.21.15.32:0/4032192141 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fccfc06cc20 0x7fccfc06f0d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcd141114e0 tx=0x7fcd08009e30 comp rx=0 tx=0).stop 2024-04-12T05:43:44.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.564+0000 7fcd19794700 1 -- 172.21.15.32:0/4032192141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd14102e20 msgr2=0x7fcd14115c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:44.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.564+0000 7fcd19794700 1 --2- 172.21.15.32:0/4032192141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd14102e20 0x7fcd14115c40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcd04002a80 tx=0x7fcd040046b0 comp rx=0 tx=0).stop 2024-04-12T05:43:44.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.564+0000 7fcd19794700 1 -- 172.21.15.32:0/4032192141 shutdown_connections 2024-04-12T05:43:44.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.564+0000 7fcd19794700 1 --2- 172.21.15.32:0/4032192141 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fccfc06cc20 0x7fccfc06f0d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:44.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.564+0000 7fcd19794700 1 --2- 172.21.15.32:0/4032192141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd14102e20 0x7fcd14115c40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:44.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.564+0000 7fcd19794700 1 --2- 172.21.15.32:0/4032192141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd141037c0 0x7fcd14116180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:44.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.564+0000 7fcd19794700 1 -- 172.21.15.32:0/4032192141 >> 172.21.15.32:0/4032192141 conn(0x7fcd140f89d0 msgr2=0x7fcd140f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:44.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.564+0000 7fcd19794700 1 -- 172.21.15.32:0/4032192141 shutdown_connections 2024-04-12T05:43:44.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.564+0000 7fcd19794700 1 -- 172.21.15.32:0/4032192141 wait complete. 2024-04-12T05:43:44.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.668+0000 7fde1e2c9700 1 -- 172.21.15.32:0/3713459538 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde18102e70 msgr2=0x7fde18103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:44.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.668+0000 7fde1e2c9700 1 --2- 172.21.15.32:0/3713459538 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde18102e70 0x7fde18103240 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fde08004940 tx=0x7fde08009e30 comp rx=0 tx=0).stop 2024-04-12T05:43:44.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.668+0000 7fde1e2c9700 1 -- 172.21.15.32:0/3713459538 shutdown_connections 2024-04-12T05:43:44.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.668+0000 7fde1e2c9700 1 --2- 172.21.15.32:0/3713459538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde180fb4a0 0x7fde180fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:44.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.668+0000 7fde1e2c9700 1 --2- 172.21.15.32:0/3713459538 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde18102e70 0x7fde18103240 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:44.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.668+0000 7fde1e2c9700 1 -- 172.21.15.32:0/3713459538 >> 172.21.15.32:0/3713459538 conn(0x7fde180714c0 msgr2=0x7fde180718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:44.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.668+0000 7fde1e2c9700 1 -- 172.21.15.32:0/3713459538 shutdown_connections 2024-04-12T05:43:44.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.668+0000 7fde1e2c9700 1 -- 172.21.15.32:0/3713459538 wait complete. 2024-04-12T05:43:44.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.669+0000 7fde1e2c9700 1 Processor -- start 2024-04-12T05:43:44.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.669+0000 7fde1e2c9700 1 -- start start 2024-04-12T05:43:44.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.669+0000 7fde1e2c9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde180fb4a0 0x7fde18113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:44.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.669+0000 7fde1e2c9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde18102e70 0x7fde18113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:44.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.669+0000 7fde1e2c9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde18114600 con 0x7fde180fb4a0 2024-04-12T05:43:44.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.669+0000 7fde1e2c9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1810d9f0 con 0x7fde18102e70 2024-04-12T05:43:44.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.669+0000 7fde17fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde180fb4a0 0x7fde18113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:44.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.669+0000 7fde17fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde180fb4a0 0x7fde18113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33558/0 (socket says 172.21.15.32:33558) 2024-04-12T05:43:44.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.669+0000 7fde17fff700 1 -- 172.21.15.32:0/718611167 learned_addr learned my addr 172.21.15.32:0/718611167 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:43:44.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.669+0000 7fde177fe700 1 --2- 172.21.15.32:0/718611167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde18102e70 0x7fde18113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:44.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.670+0000 7fde17fff700 1 -- 172.21.15.32:0/718611167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde18102e70 msgr2=0x7fde18113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:44.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.670+0000 7fde17fff700 1 --2- 172.21.15.32:0/718611167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde18102e70 0x7fde18113fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:44.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.670+0000 7fde17fff700 1 -- 172.21.15.32:0/718611167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde08005040 con 0x7fde180fb4a0 2024-04-12T05:43:44.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.670+0000 7fde17fff700 1 --2- 172.21.15.32:0/718611167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde180fb4a0 0x7fde18113a70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fde0800b3b0 tx=0x7fde08009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:44.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.670+0000 7fde14ff9700 1 -- 172.21.15.32:0/718611167 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde0800e040 con 0x7fde180fb4a0 2024-04-12T05:43:44.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.670+0000 7fde1e2c9700 1 -- 172.21.15.32:0/718611167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde1810dc70 con 0x7fde180fb4a0 2024-04-12T05:43:44.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.670+0000 7fde14ff9700 1 -- 172.21.15.32:0/718611167 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fde08013420 con 0x7fde180fb4a0 2024-04-12T05:43:44.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.670+0000 7fde14ff9700 1 -- 172.21.15.32:0/718611167 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde0801c610 con 0x7fde180fb4a0 2024-04-12T05:43:44.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.670+0000 7fde1e2c9700 1 -- 172.21.15.32:0/718611167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde1810e230 con 0x7fde180fb4a0 2024-04-12T05:43:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.671+0000 7fde1e2c9700 1 -- 172.21.15.32:0/718611167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde040052b0 con 0x7fde180fb4a0 2024-04-12T05:43:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.672+0000 7fde14ff9700 1 -- 172.21.15.32:0/718611167 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fde0801c8d0 con 0x7fde180fb4a0 2024-04-12T05:43:44.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.672+0000 7fde14ff9700 1 --2- 172.21.15.32:0/718611167 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fde0006cc20 0x7fde0006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:44.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.672+0000 7fde14ff9700 1 -- 172.21.15.32:0/718611167 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fde0808d0e0 con 0x7fde180fb4a0 2024-04-12T05:43:44.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.672+0000 7fde177fe700 1 --2- 172.21.15.32:0/718611167 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fde0006cc20 0x7fde0006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:44.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.673+0000 7fde177fe700 1 --2- 172.21.15.32:0/718611167 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fde0006cc20 0x7fde0006f0d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fde1810ebb0 tx=0x7fde0c0043c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:44.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.677+0000 7fde14ff9700 1 -- 172.21.15.32:0/718611167 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fde0805ac00 con 0x7fde180fb4a0 2024-04-12T05:43:44.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:44 smithi067 ceph-mon[112979]: from='client.34216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:44.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:44 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:44.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:44 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:44.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:44 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:43:44.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:44 smithi032 ceph-mon[146817]: from='client.34216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:44.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:44 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:44.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:44 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:44.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:44 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:43:44.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.958+0000 7fde1e2c9700 1 -- 172.21.15.32:0/718611167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fde04005d70 con 0x7fde180fb4a0 2024-04-12T05:43:44.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.958+0000 7fde14ff9700 1 -- 172.21.15.32:0/718611167 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1864 (secure 0 0 0) 0x7fde0805a790 con 0x7fde180fb4a0 2024-04-12T05:43:44.955 INFO:teuthology.orchestra.run.smithi032.stdout:e22 2024-04-12T05:43:44.955 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:43:44.955 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:43:44.955 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:43:44.955 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:43:44.955 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:43:44.955 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:43:44.955 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 22 2024-04-12T05:43:44.955 INFO:teuthology.orchestra.run.smithi032.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-12T05:43:44.955 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:43:44.955 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:41:05.358509+0000 2024-04-12T05:43:44.955 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:43:44.955 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:43:44.955 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:43:44.956 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:43:44.956 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:43:44.956 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:43:44.956 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:43:44.956 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:43:44.956 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 46 2024-04-12T05:43:44.956 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:43:44.956 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:43:44.956 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:43:44.956 INFO:teuthology.orchestra.run.smithi032.stdout:up {0=14480,1=24281} 2024-04-12T05:43:44.956 INFO:teuthology.orchestra.run.smithi032.stdout:failed 2024-04-12T05:43:44.956 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:43:44.956 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:43:44.956 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:43:44.956 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:43:44.957 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:43:44.957 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:43:44.957 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:43:44.957 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:43:44.957 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{0:14480} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.32:6828/387754149,v1:172.21.15.32:6829/387754149] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:43:44.957 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{0:14490} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3678031004,v1:172.21.15.32:6827/3678031004] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:43:44.957 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{1:24281} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.67:6824/1291914599,v1:172.21.15.67:6825/1291914599] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:43:44.957 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{1:24295} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.67:6826/873395842,v1:172.21.15.67:6827/873395842] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:43:44.957 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:43:44.957 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:43:44.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.961+0000 7fde1e2c9700 1 -- 172.21.15.32:0/718611167 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fde0006cc20 msgr2=0x7fde0006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:44.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.961+0000 7fde1e2c9700 1 --2- 172.21.15.32:0/718611167 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fde0006cc20 0x7fde0006f0d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fde1810ebb0 tx=0x7fde0c0043c0 comp rx=0 tx=0).stop 2024-04-12T05:43:44.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.961+0000 7fde1e2c9700 1 -- 172.21.15.32:0/718611167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde180fb4a0 msgr2=0x7fde18113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:44.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.961+0000 7fde1e2c9700 1 --2- 172.21.15.32:0/718611167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde180fb4a0 0x7fde18113a70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fde0800b3b0 tx=0x7fde08009e30 comp rx=0 tx=0).stop 2024-04-12T05:43:44.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.961+0000 7fde1e2c9700 1 -- 172.21.15.32:0/718611167 shutdown_connections 2024-04-12T05:43:44.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.961+0000 7fde1e2c9700 1 --2- 172.21.15.32:0/718611167 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fde0006cc20 0x7fde0006f0d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:44.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.961+0000 7fde1e2c9700 1 --2- 172.21.15.32:0/718611167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde180fb4a0 0x7fde18113a70 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:44.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.961+0000 7fde1e2c9700 1 --2- 172.21.15.32:0/718611167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde18102e70 0x7fde18113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:44.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.961+0000 7fde1e2c9700 1 -- 172.21.15.32:0/718611167 >> 172.21.15.32:0/718611167 conn(0x7fde180714c0 msgr2=0x7fde180fa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:44.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.961+0000 7fde1e2c9700 1 -- 172.21.15.32:0/718611167 shutdown_connections 2024-04-12T05:43:44.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:44.961+0000 7fde1e2c9700 1 -- 172.21.15.32:0/718611167 wait complete. 2024-04-12T05:43:44.959 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 22 2024-04-12T05:43:45.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.066+0000 7f9473479700 1 -- 172.21.15.32:0/1594831397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f946c102e70 msgr2=0x7f946c103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:45.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.066+0000 7f9473479700 1 --2- 172.21.15.32:0/1594831397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f946c102e70 0x7f946c103240 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9468004940 tx=0x7f9468009e30 comp rx=0 tx=0).stop 2024-04-12T05:43:45.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.067+0000 7f9473479700 1 -- 172.21.15.32:0/1594831397 shutdown_connections 2024-04-12T05:43:45.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.067+0000 7f9473479700 1 --2- 172.21.15.32:0/1594831397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f946c0fb4a0 0x7f946c0fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:45.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.067+0000 7f9473479700 1 --2- 172.21.15.32:0/1594831397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f946c102e70 0x7f946c103240 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:45.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.067+0000 7f9473479700 1 -- 172.21.15.32:0/1594831397 >> 172.21.15.32:0/1594831397 conn(0x7f946c0714c0 msgr2=0x7f946c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:45.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.067+0000 7f9473479700 1 -- 172.21.15.32:0/1594831397 shutdown_connections 2024-04-12T05:43:45.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.067+0000 7f9473479700 1 -- 172.21.15.32:0/1594831397 wait complete. 2024-04-12T05:43:45.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.067+0000 7f9473479700 1 Processor -- start 2024-04-12T05:43:45.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.067+0000 7f9473479700 1 -- start start 2024-04-12T05:43:45.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.068+0000 7f9473479700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f946c0fb4a0 0x7f946c1139c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:45.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.068+0000 7f9473479700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f946c102e70 0x7f946c113f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:45.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.068+0000 7f9473479700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946c114550 con 0x7f946c0fb4a0 2024-04-12T05:43:45.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.068+0000 7f9473479700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946c10d990 con 0x7f946c102e70 2024-04-12T05:43:45.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.068+0000 7f9471215700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f946c0fb4a0 0x7f946c1139c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:45.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.068+0000 7f9471215700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f946c0fb4a0 0x7f946c1139c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33576/0 (socket says 172.21.15.32:33576) 2024-04-12T05:43:45.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.068+0000 7f9471215700 1 -- 172.21.15.32:0/569687478 learned_addr learned my addr 172.21.15.32:0/569687478 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:43:45.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.068+0000 7f9470a14700 1 --2- 172.21.15.32:0/569687478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f946c102e70 0x7f946c113f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:45.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.068+0000 7f9471215700 1 -- 172.21.15.32:0/569687478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f946c102e70 msgr2=0x7f946c113f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:45.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.068+0000 7f9471215700 1 --2- 172.21.15.32:0/569687478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f946c102e70 0x7f946c113f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:45.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.068+0000 7f9471215700 1 -- 172.21.15.32:0/569687478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9468005040 con 0x7f946c0fb4a0 2024-04-12T05:43:45.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.068+0000 7f9470a14700 1 --2- 172.21.15.32:0/569687478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f946c102e70 0x7f946c113f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:43:45.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.068+0000 7f9471215700 1 --2- 172.21.15.32:0/569687478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f946c0fb4a0 0x7f946c1139c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9468009180 tx=0x7f9468009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:45.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.068+0000 7f9461ffb700 1 -- 172.21.15.32:0/569687478 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f946800e040 con 0x7f946c0fb4a0 2024-04-12T05:43:45.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.069+0000 7f9461ffb700 1 -- 172.21.15.32:0/569687478 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9468004290 con 0x7f946c0fb4a0 2024-04-12T05:43:45.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.069+0000 7f9473479700 1 -- 172.21.15.32:0/569687478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f946c10dc10 con 0x7f946c0fb4a0 2024-04-12T05:43:45.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.069+0000 7f9461ffb700 1 -- 172.21.15.32:0/569687478 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f946801c3f0 con 0x7f946c0fb4a0 2024-04-12T05:43:45.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.069+0000 7f9473479700 1 -- 172.21.15.32:0/569687478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f946c10e1d0 con 0x7f946c0fb4a0 2024-04-12T05:43:45.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.070+0000 7f9473479700 1 -- 172.21.15.32:0/569687478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f946c100110 con 0x7f946c0fb4a0 2024-04-12T05:43:45.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.070+0000 7f9461ffb700 1 -- 172.21.15.32:0/569687478 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9468013790 con 0x7f946c0fb4a0 2024-04-12T05:43:45.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.070+0000 7f9461ffb700 1 --2- 172.21.15.32:0/569687478 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f945806cc20 0x7f945806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:45.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.071+0000 7f9461ffb700 1 -- 172.21.15.32:0/569687478 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f946808d2b0 con 0x7f946c0fb4a0 2024-04-12T05:43:45.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.071+0000 7f9470a14700 1 --2- 172.21.15.32:0/569687478 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f945806cc20 0x7f945806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:45.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.071+0000 7f9470a14700 1 --2- 172.21.15.32:0/569687478 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f945806cc20 0x7f945806f0d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f945c009b80 tx=0x7f945c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:45.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.075+0000 7f9461ffb700 1 -- 172.21.15.32:0/569687478 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f946805add0 con 0x7f946c0fb4a0 2024-04-12T05:43:45.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.303+0000 7f9473479700 1 -- 172.21.15.32:0/569687478 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f946c10f010 con 0x7f945806cc20 2024-04-12T05:43:45.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.305+0000 7f9461ffb700 1 -- 172.21.15.32:0/569687478 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f946c10f010 con 0x7f945806cc20 2024-04-12T05:43:45.301 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:43:45.301 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:43:45.301 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:43:45.301 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:43:45.301 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:43:45.301 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:43:45.301 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:43:45.301 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-12T05:43:45.301 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:43:45.301 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "7/21 daemons upgraded", 2024-04-12T05:43:45.301 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading osd daemons", 2024-04-12T05:43:45.301 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:43:45.301 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:43:45.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.307+0000 7f9473479700 1 -- 172.21.15.32:0/569687478 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f945806cc20 msgr2=0x7f945806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:45.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.307+0000 7f9473479700 1 --2- 172.21.15.32:0/569687478 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f945806cc20 0x7f945806f0d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f945c009b80 tx=0x7f945c00b040 comp rx=0 tx=0).stop 2024-04-12T05:43:45.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.307+0000 7f9473479700 1 -- 172.21.15.32:0/569687478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f946c0fb4a0 msgr2=0x7f946c1139c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:45.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.307+0000 7f9473479700 1 --2- 172.21.15.32:0/569687478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f946c0fb4a0 0x7f946c1139c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9468009180 tx=0x7f9468009e30 comp rx=0 tx=0).stop 2024-04-12T05:43:45.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.307+0000 7f9473479700 1 -- 172.21.15.32:0/569687478 shutdown_connections 2024-04-12T05:43:45.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.307+0000 7f9473479700 1 --2- 172.21.15.32:0/569687478 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f945806cc20 0x7f945806f0d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:45.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.307+0000 7f9473479700 1 --2- 172.21.15.32:0/569687478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f946c0fb4a0 0x7f946c1139c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:45.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.307+0000 7f9473479700 1 --2- 172.21.15.32:0/569687478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f946c102e70 0x7f946c113f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:45.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.307+0000 7f9473479700 1 -- 172.21.15.32:0/569687478 >> 172.21.15.32:0/569687478 conn(0x7f946c0714c0 msgr2=0x7f946c0fa780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:45.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.307+0000 7f9473479700 1 -- 172.21.15.32:0/569687478 shutdown_connections 2024-04-12T05:43:45.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.307+0000 7f9473479700 1 -- 172.21.15.32:0/569687478 wait complete. 2024-04-12T05:43:45.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.410+0000 7f4180e04700 1 -- 172.21.15.32:0/3054829909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f417c101d70 msgr2=0x7f417c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:45.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.410+0000 7f4180e04700 1 --2- 172.21.15.32:0/3054829909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f417c101d70 0x7f417c102140 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f416c004940 tx=0x7f416c009e30 comp rx=0 tx=0).stop 2024-04-12T05:43:45.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.410+0000 7f4180e04700 1 -- 172.21.15.32:0/3054829909 shutdown_connections 2024-04-12T05:43:45.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.410+0000 7f4180e04700 1 --2- 172.21.15.32:0/3054829909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f417c102710 0x7f417c0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:45.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.410+0000 7f4180e04700 1 --2- 172.21.15.32:0/3054829909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f417c101d70 0x7f417c102140 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:45.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.410+0000 7f4180e04700 1 -- 172.21.15.32:0/3054829909 >> 172.21.15.32:0/3054829909 conn(0x7f417c072bc0 msgr2=0x7f417c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:45.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.410+0000 7f4180e04700 1 -- 172.21.15.32:0/3054829909 shutdown_connections 2024-04-12T05:43:45.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.410+0000 7f4180e04700 1 -- 172.21.15.32:0/3054829909 wait complete. 2024-04-12T05:43:45.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.411+0000 7f4180e04700 1 Processor -- start 2024-04-12T05:43:45.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.411+0000 7f4180e04700 1 -- start start 2024-04-12T05:43:45.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.411+0000 7f4180e04700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f417c101d70 0x7f417c10eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:45.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.411+0000 7f4180e04700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f417c102710 0x7f417c10f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:45.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.411+0000 7f4180e04700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f417c10f5e0 con 0x7f417c102710 2024-04-12T05:43:45.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.411+0000 7f4180e04700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f417c10f720 con 0x7f417c101d70 2024-04-12T05:43:45.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.412+0000 7f4179d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f417c102710 0x7f417c10f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:45.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.412+0000 7f4179d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f417c102710 0x7f417c10f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33598/0 (socket says 172.21.15.32:33598) 2024-04-12T05:43:45.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.412+0000 7f4179d9b700 1 -- 172.21.15.32:0/755338244 learned_addr learned my addr 172.21.15.32:0/755338244 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:43:45.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.412+0000 7f417a59c700 1 --2- 172.21.15.32:0/755338244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f417c101d70 0x7f417c10eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:45.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.412+0000 7f4179d9b700 1 -- 172.21.15.32:0/755338244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f417c101d70 msgr2=0x7f417c10eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:45.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.412+0000 7f4179d9b700 1 --2- 172.21.15.32:0/755338244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f417c101d70 0x7f417c10eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:45.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.412+0000 7f4179d9b700 1 -- 172.21.15.32:0/755338244 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f416c005040 con 0x7f417c102710 2024-04-12T05:43:45.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.412+0000 7f4179d9b700 1 --2- 172.21.15.32:0/755338244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f417c102710 0x7f417c10f0a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4164002a50 tx=0x7f41640046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:45.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.412+0000 7f4172ffd700 1 -- 172.21.15.32:0/755338244 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4164007cc0 con 0x7f417c102710 2024-04-12T05:43:45.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.412+0000 7f4172ffd700 1 -- 172.21.15.32:0/755338244 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f416400f040 con 0x7f417c102710 2024-04-12T05:43:45.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.412+0000 7f4180e04700 1 -- 172.21.15.32:0/755338244 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f417c117a10 con 0x7f417c102710 2024-04-12T05:43:45.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.412+0000 7f4172ffd700 1 -- 172.21.15.32:0/755338244 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f416400da30 con 0x7f417c102710 2024-04-12T05:43:45.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.412+0000 7f4180e04700 1 -- 172.21.15.32:0/755338244 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f417c117fa0 con 0x7f417c102710 2024-04-12T05:43:45.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.413+0000 7f4180e04700 1 -- 172.21.15.32:0/755338244 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f417c0ff080 con 0x7f417c102710 2024-04-12T05:43:45.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.414+0000 7f4172ffd700 1 -- 172.21.15.32:0/755338244 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f416400dbd0 con 0x7f417c102710 2024-04-12T05:43:45.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.414+0000 7f4172ffd700 1 --2- 172.21.15.32:0/755338244 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f416806cc30 0x7f416806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:43:45.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.414+0000 7f4172ffd700 1 -- 172.21.15.32:0/755338244 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f416408bc80 con 0x7f417c102710 2024-04-12T05:43:45.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.414+0000 7f417a59c700 1 --2- 172.21.15.32:0/755338244 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f416806cc30 0x7f416806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:45.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.415+0000 7f417a59c700 1 --2- 172.21.15.32:0/755338244 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f416806cc30 0x7f416806f0e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f416c009fd0 tx=0x7f416c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:45.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.419+0000 7f4172ffd700 1 -- 172.21.15.32:0/755338244 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4164055c90 con 0x7f417c102710 2024-04-12T05:43:45.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.726+0000 7f4180e04700 1 -- 172.21.15.32:0/755338244 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f417c065dd0 con 0x7f417c102710 2024-04-12T05:43:45.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.726+0000 7f4172ffd700 1 -- 172.21.15.32:0/755338244 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1387 (secure 0 0 0) 0x7f41640592b0 con 0x7f417c102710 2024-04-12T05:43:45.723 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_WARN 1 osds down; Reduced data availability: 3 pgs peering; Degraded data redundancy: 46/333 objects degraded (13.814%), 14 pgs degraded 2024-04-12T05:43:45.723 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-12T05:43:45.723 INFO:teuthology.orchestra.run.smithi032.stdout: osd.1 (root=default,host=smithi032) is down 2024-04-12T05:43:45.724 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] PG_AVAILABILITY: Reduced data availability: 3 pgs peering 2024-04-12T05:43:45.724 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.4 is stuck peering for 5m, current state peering, last acting [0,4] 2024-04-12T05:43:45.724 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.1b is stuck peering for 7m, current state peering, last acting [0,5] 2024-04-12T05:43:45.724 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.b is stuck peering for 4m, current state peering, last acting [0,4] 2024-04-12T05:43:45.724 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 46/333 objects degraded (13.814%), 14 pgs degraded 2024-04-12T05:43:45.724 INFO:teuthology.orchestra.run.smithi032.stdout: pg 1.0 is active+undersized+degraded, acting [3,0] 2024-04-12T05:43:45.725 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.0 is active+undersized+degraded, acting [3,0] 2024-04-12T05:43:45.725 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.1 is active+undersized+degraded, acting [2,0] 2024-04-12T05:43:45.725 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.2 is active+undersized+degraded, acting [5,0] 2024-04-12T05:43:45.725 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.3 is active+undersized+degraded, acting [5,2] 2024-04-12T05:43:45.725 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.6 is active+undersized+degraded, acting [3,4] 2024-04-12T05:43:45.725 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.9 is active+undersized+degraded, acting [4,0] 2024-04-12T05:43:45.726 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.a is active+undersized+degraded, acting [4,3] 2024-04-12T05:43:45.726 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.d is active+undersized+degraded, acting [3,2] 2024-04-12T05:43:45.726 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.10 is active+undersized+degraded, acting [2,0] 2024-04-12T05:43:45.726 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.11 is active+undersized+degraded, acting [4,3] 2024-04-12T05:43:45.726 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.12 is active+undersized+degraded, acting [3,0] 2024-04-12T05:43:45.727 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.15 is active+undersized+degraded, acting [3,0] 2024-04-12T05:43:45.727 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.17 is active+undersized+degraded, acting [5,2] 2024-04-12T05:43:45.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.728+0000 7f4180e04700 1 -- 172.21.15.32:0/755338244 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f416806cc30 msgr2=0x7f416806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:45.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.728+0000 7f4180e04700 1 --2- 172.21.15.32:0/755338244 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f416806cc30 0x7f416806f0e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f416c009fd0 tx=0x7f416c009e30 comp rx=0 tx=0).stop 2024-04-12T05:43:45.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.728+0000 7f4180e04700 1 -- 172.21.15.32:0/755338244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f417c102710 msgr2=0x7f417c10f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:45.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.728+0000 7f4180e04700 1 --2- 172.21.15.32:0/755338244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f417c102710 0x7f417c10f0a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4164002a50 tx=0x7f41640046b0 comp rx=0 tx=0).stop 2024-04-12T05:43:45.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.728+0000 7f4180e04700 1 -- 172.21.15.32:0/755338244 shutdown_connections 2024-04-12T05:43:45.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.728+0000 7f4180e04700 1 --2- 172.21.15.32:0/755338244 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f416806cc30 0x7f416806f0e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:45.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.728+0000 7f4180e04700 1 --2- 172.21.15.32:0/755338244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f417c101d70 0x7f417c10eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:45.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.728+0000 7f4180e04700 1 --2- 172.21.15.32:0/755338244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f417c102710 0x7f417c10f0a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:43:45.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.728+0000 7f4180e04700 1 -- 172.21.15.32:0/755338244 >> 172.21.15.32:0/755338244 conn(0x7f417c072bc0 msgr2=0x7f417c0faf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:45.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.728+0000 7f4180e04700 1 -- 172.21.15.32:0/755338244 shutdown_connections 2024-04-12T05:43:45.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:43:45.728+0000 7f4180e04700 1 -- 172.21.15.32:0/755338244 wait complete. 2024-04-12T05:43:45.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:45 smithi067 ceph-mon[112979]: from='client.34222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:45.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:45 smithi067 ceph-mon[112979]: pgmap v68: 65 pgs: 14 active+undersized+degraded, 5 peering, 15 active+undersized, 31 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 46/333 objects degraded (13.814%) 2024-04-12T05:43:45.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:45 smithi067 ceph-mon[112979]: from='client.34226 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:45.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:45 smithi067 ceph-mon[112979]: Health check failed: Degraded data redundancy: 46/333 objects degraded (13.814%), 14 pgs degraded (PG_DEGRADED) 2024-04-12T05:43:45.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:45 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/4032192141' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:43:45.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:45 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/718611167' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:43:45.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:45 smithi032 ceph-mon[146817]: from='client.34222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:45.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:45 smithi032 ceph-mon[146817]: pgmap v68: 65 pgs: 14 active+undersized+degraded, 5 peering, 15 active+undersized, 31 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 46/333 objects degraded (13.814%) 2024-04-12T05:43:45.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:45 smithi032 ceph-mon[146817]: from='client.34226 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:45.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:45 smithi032 ceph-mon[146817]: Health check failed: Degraded data redundancy: 46/333 objects degraded (13.814%), 14 pgs degraded (PG_DEGRADED) 2024-04-12T05:43:45.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:45 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/4032192141' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:43:45.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:45 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/718611167' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:43:46.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:46 smithi067 ceph-mon[112979]: from='client.34238 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:46.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:46 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/755338244' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:43:46.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:46 smithi032 ceph-mon[146817]: from='client.34238 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:46.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:46 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/755338244' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:43:47.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:47 smithi067 ceph-mon[112979]: pgmap v69: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 52/333 objects degraded (15.616%) 2024-04-12T05:43:47.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:47 smithi067 ceph-mon[112979]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-04-12T05:43:47.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:47 smithi032 ceph-mon[146817]: pgmap v69: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 52/333 objects degraded (15.616%) 2024-04-12T05:43:47.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:47 smithi032 ceph-mon[146817]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-04-12T05:43:49.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:48 smithi067 ceph-mon[112979]: pgmap v70: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 52/333 objects degraded (15.616%) 2024-04-12T05:43:49.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:48 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:49.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:48 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:43:49.338 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:43:48 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-1[167952]: 2024-04-12T05:43:48.982+0000 7f5ec9b317c0 -1 osd.1 54 log_to_monitors true 2024-04-12T05:43:49.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:48 smithi032 ceph-mon[146817]: pgmap v70: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 52/333 objects degraded (15.616%) 2024-04-12T05:43:49.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:48 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:49.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:48 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:43:50.194 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 05:43:49 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-1[167952]: 2024-04-12T05:43:49.873+0000 7f5ebaf8b700 -1 osd.1 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T05:43:50.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:49 smithi032 ceph-mon[146817]: from='osd.1 [v2:172.21.15.32:6810/1060060861,v1:172.21.15.32:6811/1060060861]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-12T05:43:50.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:49 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:50.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:49 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:50.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:49 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:50.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:49 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:50.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:49 smithi067 ceph-mon[112979]: from='osd.1 [v2:172.21.15.32:6810/1060060861,v1:172.21.15.32:6811/1060060861]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-12T05:43:50.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:49 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:50.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:49 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:50.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:49 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:50.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:49 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:51.248 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:50 smithi032 ceph-mon[146817]: pgmap v71: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s; 52/333 objects degraded (15.616%) 2024-04-12T05:43:51.248 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:50 smithi032 ceph-mon[146817]: from='osd.1 [v2:172.21.15.32:6810/1060060861,v1:172.21.15.32:6811/1060060861]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-12T05:43:51.248 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:50 smithi032 ceph-mon[146817]: osdmap e57: 6 total, 5 up, 6 in 2024-04-12T05:43:51.248 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:50 smithi032 ceph-mon[146817]: from='osd.1 [v2:172.21.15.32:6810/1060060861,v1:172.21.15.32:6811/1060060861]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-12T05:43:51.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:50 smithi067 ceph-mon[112979]: pgmap v71: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s; 52/333 objects degraded (15.616%) 2024-04-12T05:43:51.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:50 smithi067 ceph-mon[112979]: from='osd.1 [v2:172.21.15.32:6810/1060060861,v1:172.21.15.32:6811/1060060861]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-12T05:43:51.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:50 smithi067 ceph-mon[112979]: osdmap e57: 6 total, 5 up, 6 in 2024-04-12T05:43:51.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:50 smithi067 ceph-mon[112979]: from='osd.1 [v2:172.21.15.32:6810/1060060861,v1:172.21.15.32:6811/1060060861]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-12T05:43:52.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:51 smithi067 ceph-mon[112979]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T05:43:52.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:51 smithi067 ceph-mon[112979]: osd.1 [v2:172.21.15.32:6810/1060060861,v1:172.21.15.32:6811/1060060861] boot 2024-04-12T05:43:52.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:51 smithi067 ceph-mon[112979]: osdmap e58: 6 total, 6 up, 6 in 2024-04-12T05:43:52.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:51 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:43:52.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:51 smithi067 ceph-mon[112979]: osdmap e59: 6 total, 6 up, 6 in 2024-04-12T05:43:52.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:51 smithi032 ceph-mon[146817]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T05:43:52.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:51 smithi032 ceph-mon[146817]: osd.1 [v2:172.21.15.32:6810/1060060861,v1:172.21.15.32:6811/1060060861] boot 2024-04-12T05:43:52.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:51 smithi032 ceph-mon[146817]: osdmap e58: 6 total, 6 up, 6 in 2024-04-12T05:43:52.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:51 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:43:52.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:51 smithi032 ceph-mon[146817]: osdmap e59: 6 total, 6 up, 6 in 2024-04-12T05:43:53.007 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:52 smithi032 ceph-mon[146817]: pgmap v74: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s; 52/333 objects degraded (15.616%) 2024-04-12T05:43:53.007 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:52 smithi032 ceph-mon[146817]: Health check update: Degraded data redundancy: 52/333 objects degraded (15.616%), 16 pgs degraded (PG_DEGRADED) 2024-04-12T05:43:53.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:52 smithi067 ceph-mon[112979]: pgmap v74: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s; 52/333 objects degraded (15.616%) 2024-04-12T05:43:53.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:52 smithi067 ceph-mon[112979]: Health check update: Degraded data redundancy: 52/333 objects degraded (15.616%), 16 pgs degraded (PG_DEGRADED) 2024-04-12T05:43:54.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:54 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:54.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:54 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:54.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:54 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:43:54.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:54 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:43:54.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:54 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:54.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:54 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:43:54.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:54 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:43:54.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:54 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:43:54.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:54 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:43:54.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:54 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-12T05:43:54.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:54 smithi067 ceph-mon[112979]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-12T05:43:54.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:54 smithi067 ceph-mon[112979]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-12T05:43:54.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:54 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:54.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:54 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:54.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:54 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:43:54.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:54 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:43:54.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:54 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:43:54.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:54 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:43:54.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:54 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:43:54.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:54 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:43:54.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:54 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:43:54.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:54 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-12T05:43:54.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:54 smithi032 ceph-mon[146817]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-12T05:43:54.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:54 smithi032 ceph-mon[146817]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-12T05:43:55.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:55 smithi067 ceph-mon[112979]: pgmap v76: 65 pgs: 7 active+undersized+degraded, 7 active+undersized, 51 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.8 KiB/s rd, 5 op/s; 21/333 objects degraded (6.306%) 2024-04-12T05:43:55.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:55 smithi032 ceph-mon[146817]: pgmap v76: 65 pgs: 7 active+undersized+degraded, 7 active+undersized, 51 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.8 KiB/s rd, 5 op/s; 21/333 objects degraded (6.306%) 2024-04-12T05:43:56.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:56 smithi067 ceph-mon[112979]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 21/333 objects degraded (6.306%), 7 pgs degraded) 2024-04-12T05:43:56.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:56 smithi067 ceph-mon[112979]: Cluster is now healthy 2024-04-12T05:43:56.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:56 smithi032 ceph-mon[146817]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 21/333 objects degraded (6.306%), 7 pgs degraded) 2024-04-12T05:43:56.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:56 smithi032 ceph-mon[146817]: Cluster is now healthy 2024-04-12T05:43:57.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:57 smithi067 ceph-mon[112979]: pgmap v77: 65 pgs: 65 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T05:43:57.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:57 smithi032 ceph-mon[146817]: pgmap v77: 65 pgs: 65 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T05:43:59.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:43:59 smithi067 ceph-mon[112979]: pgmap v78: 65 pgs: 65 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 901 B/s rd, 1 op/s 2024-04-12T05:43:59.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:43:59 smithi032 ceph-mon[146817]: pgmap v78: 65 pgs: 65 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 901 B/s rd, 1 op/s 2024-04-12T05:44:01.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:01 smithi067 ceph-mon[112979]: pgmap v79: 65 pgs: 65 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-12T05:44:01.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:01 smithi032 ceph-mon[146817]: pgmap v79: 65 pgs: 65 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-12T05:44:03.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:03 smithi067 ceph-mon[112979]: pgmap v80: 65 pgs: 65 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:44:03.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:03.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:44:03.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:03.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:03 smithi032 ceph-mon[146817]: pgmap v80: 65 pgs: 65 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:44:03.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:03.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:44:03.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:05.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:05 smithi067 ceph-mon[112979]: pgmap v81: 65 pgs: 65 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-12T05:44:05.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:05 smithi032 ceph-mon[146817]: pgmap v81: 65 pgs: 65 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-12T05:44:07.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:07 smithi067 ceph-mon[112979]: pgmap v82: 65 pgs: 65 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:44:07.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:07 smithi032 ceph-mon[146817]: pgmap v82: 65 pgs: 65 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:44:09.738 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:09 smithi032 ceph-mon[146817]: pgmap v83: 65 pgs: 65 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:44:09.739 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-12T05:44:09.739 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:09 smithi032 ceph-mon[146817]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-12T05:44:09.739 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:09 smithi032 ceph-mon[146817]: Upgrade: osd.2 is safe to restart 2024-04-12T05:44:09.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:09 smithi067 ceph-mon[112979]: pgmap v83: 65 pgs: 65 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:44:09.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-12T05:44:09.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:09 smithi067 ceph-mon[112979]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-12T05:44:09.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:09 smithi067 ceph-mon[112979]: Upgrade: osd.2 is safe to restart 2024-04-12T05:44:11.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:11 smithi067 ceph-mon[112979]: pgmap v84: 65 pgs: 65 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:44:11.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:11 smithi067 ceph-mon[112979]: Upgrade: Updating osd.2 2024-04-12T05:44:11.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:11 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:11.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:11 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-12T05:44:11.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:11 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:44:11.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:11 smithi067 ceph-mon[112979]: Deploying daemon osd.2 on smithi032 2024-04-12T05:44:11.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:11 smithi032 ceph-mon[146817]: pgmap v84: 65 pgs: 65 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:44:11.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:11 smithi032 ceph-mon[146817]: Upgrade: Updating osd.2 2024-04-12T05:44:11.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:11 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:11.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:11 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-12T05:44:11.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:11 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:44:11.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:11 smithi032 ceph-mon[146817]: Deploying daemon osd.2 on smithi032 2024-04-12T05:44:13.723 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:13 smithi032 ceph-mon[146817]: pgmap v85: 65 pgs: 65 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:44:13.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:13 smithi067 ceph-mon[112979]: pgmap v85: 65 pgs: 65 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:44:14.089 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:44:13 smithi032 systemd[1]: Stopping Ceph osd.2 for 37b58ce2-f88d-11ee-bc8c-c7b262605968... 2024-04-12T05:44:14.089 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:44:13 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-2[82754]: 2024-04-12T05:44:13.965+0000 7fe635adb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T05:44:14.089 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:44:13 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-2[82754]: 2024-04-12T05:44:13.965+0000 7fe635adb700 -1 osd.2 59 *** Got signal Terminated *** 2024-04-12T05:44:14.089 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:44:13 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-2[82754]: 2024-04-12T05:44:13.965+0000 7fe635adb700 -1 osd.2 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T05:44:14.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:14 smithi067 ceph-mon[112979]: osd.2 marked itself down and dead 2024-04-12T05:44:14.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:14 smithi032 ceph-mon[146817]: osd.2 marked itself down and dead 2024-04-12T05:44:15.588 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:44:15 smithi032 bash[174157]: ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-2 2024-04-12T05:44:15.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:15 smithi032 ceph-mon[146817]: pgmap v86: 65 pgs: 65 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:44:15.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:15 smithi032 ceph-mon[146817]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T05:44:15.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:15 smithi032 ceph-mon[146817]: osdmap e60: 6 total, 5 up, 6 in 2024-04-12T05:44:15.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:15 smithi067 ceph-mon[112979]: pgmap v86: 65 pgs: 65 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:44:15.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:15 smithi067 ceph-mon[112979]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T05:44:15.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:15 smithi067 ceph-mon[112979]: osdmap e60: 6 total, 5 up, 6 in 2024-04-12T05:44:15.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.855+0000 7fbd9f59d700 1 -- 172.21.15.32:0/4093697724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd981037b0 msgr2=0x7fbd980fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:15.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.855+0000 7fbd9f59d700 1 --2- 172.21.15.32:0/4093697724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd981037b0 0x7fbd980fce20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbd88004940 tx=0x7fbd88009e30 comp rx=0 tx=0).stop 2024-04-12T05:44:15.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.855+0000 7fbd9f59d700 1 -- 172.21.15.32:0/4093697724 shutdown_connections 2024-04-12T05:44:15.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.855+0000 7fbd9f59d700 1 --2- 172.21.15.32:0/4093697724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd981037b0 0x7fbd980fce20 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:15.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.855+0000 7fbd9f59d700 1 --2- 172.21.15.32:0/4093697724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd98102e10 0x7fbd981031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:15.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.855+0000 7fbd9f59d700 1 -- 172.21.15.32:0/4093697724 >> 172.21.15.32:0/4093697724 conn(0x7fbd980f89a0 msgr2=0x7fbd980fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:15.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.855+0000 7fbd9f59d700 1 -- 172.21.15.32:0/4093697724 shutdown_connections 2024-04-12T05:44:15.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.855+0000 7fbd9f59d700 1 -- 172.21.15.32:0/4093697724 wait complete. 2024-04-12T05:44:15.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.856+0000 7fbd9f59d700 1 Processor -- start 2024-04-12T05:44:15.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.856+0000 7fbd9f59d700 1 -- start start 2024-04-12T05:44:15.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.856+0000 7fbd9f59d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd98102e10 0x7fbd98113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:15.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.856+0000 7fbd9f59d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd981037b0 0x7fbd98114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:15.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.856+0000 7fbd9f59d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd98114660 con 0x7fbd98102e10 2024-04-12T05:44:15.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.856+0000 7fbd9f59d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9810da50 con 0x7fbd981037b0 2024-04-12T05:44:15.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.856+0000 7fbd9d339700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd98102e10 0x7fbd98113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:15.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.856+0000 7fbd9d339700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd98102e10 0x7fbd98113ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40382/0 (socket says 172.21.15.32:40382) 2024-04-12T05:44:15.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.856+0000 7fbd9d339700 1 -- 172.21.15.32:0/3103711396 learned_addr learned my addr 172.21.15.32:0/3103711396 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:44:15.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.856+0000 7fbd9cb38700 1 --2- 172.21.15.32:0/3103711396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd981037b0 0x7fbd98114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:15.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.856+0000 7fbd9d339700 1 -- 172.21.15.32:0/3103711396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd981037b0 msgr2=0x7fbd98114010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:15.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.856+0000 7fbd9d339700 1 --2- 172.21.15.32:0/3103711396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd981037b0 0x7fbd98114010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:15.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.856+0000 7fbd9d339700 1 -- 172.21.15.32:0/3103711396 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd88005040 con 0x7fbd98102e10 2024-04-12T05:44:15.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.856+0000 7fbd9cb38700 1 --2- 172.21.15.32:0/3103711396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd981037b0 0x7fbd98114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:44:15.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.857+0000 7fbd9d339700 1 --2- 172.21.15.32:0/3103711396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd98102e10 0x7fbd98113ad0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbd94002a50 tx=0x7fbd940046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:15.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.857+0000 7fbd8dffb700 1 -- 172.21.15.32:0/3103711396 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd94007cc0 con 0x7fbd98102e10 2024-04-12T05:44:15.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.857+0000 7fbd8dffb700 1 -- 172.21.15.32:0/3103711396 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd9400f040 con 0x7fbd98102e10 2024-04-12T05:44:15.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.857+0000 7fbd9f59d700 1 -- 172.21.15.32:0/3103711396 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd9810dd30 con 0x7fbd98102e10 2024-04-12T05:44:15.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.857+0000 7fbd8dffb700 1 -- 172.21.15.32:0/3103711396 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd94009540 con 0x7fbd98102e10 2024-04-12T05:44:15.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.857+0000 7fbd9f59d700 1 -- 172.21.15.32:0/3103711396 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd9810e2f0 con 0x7fbd98102e10 2024-04-12T05:44:15.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.858+0000 7fbd9f59d700 1 -- 172.21.15.32:0/3103711396 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd980fe560 con 0x7fbd98102e10 2024-04-12T05:44:15.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.859+0000 7fbd8dffb700 1 -- 172.21.15.32:0/3103711396 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fbd940096a0 con 0x7fbd98102e10 2024-04-12T05:44:15.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.859+0000 7fbd8dffb700 1 --2- 172.21.15.32:0/3103711396 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbd8406cc20 0x7fbd8406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:15.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.859+0000 7fbd8dffb700 1 -- 172.21.15.32:0/3103711396 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fbd9408be20 con 0x7fbd98102e10 2024-04-12T05:44:15.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.859+0000 7fbd9cb38700 1 --2- 172.21.15.32:0/3103711396 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbd8406cc20 0x7fbd8406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:15.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.860+0000 7fbd9cb38700 1 --2- 172.21.15.32:0/3103711396 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbd8406cc20 0x7fbd8406f0d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbd9810ec10 tx=0x7fbd88009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:15.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:15.864+0000 7fbd8dffb700 1 -- 172.21.15.32:0/3103711396 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbd940598c0 con 0x7fbd98102e10 2024-04-12T05:44:16.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.094+0000 7fbd9f59d700 1 -- 172.21.15.32:0/3103711396 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbd98109e50 con 0x7fbd8406cc20 2024-04-12T05:44:16.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.096+0000 7fbd8dffb700 1 -- 172.21.15.32:0/3103711396 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fbd88007cc0 con 0x7fbd8406cc20 2024-04-12T05:44:16.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.098+0000 7fbd9f59d700 1 -- 172.21.15.32:0/3103711396 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbd8406cc20 msgr2=0x7fbd8406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:16.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.098+0000 7fbd9f59d700 1 --2- 172.21.15.32:0/3103711396 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbd8406cc20 0x7fbd8406f0d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbd9810ec10 tx=0x7fbd88009e30 comp rx=0 tx=0).stop 2024-04-12T05:44:16.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.098+0000 7fbd9f59d700 1 -- 172.21.15.32:0/3103711396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd98102e10 msgr2=0x7fbd98113ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:16.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.098+0000 7fbd9f59d700 1 --2- 172.21.15.32:0/3103711396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd98102e10 0x7fbd98113ad0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbd94002a50 tx=0x7fbd940046b0 comp rx=0 tx=0).stop 2024-04-12T05:44:16.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.098+0000 7fbd9f59d700 1 -- 172.21.15.32:0/3103711396 shutdown_connections 2024-04-12T05:44:16.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.098+0000 7fbd9f59d700 1 --2- 172.21.15.32:0/3103711396 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbd8406cc20 0x7fbd8406f0d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:16.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.098+0000 7fbd9f59d700 1 --2- 172.21.15.32:0/3103711396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd98102e10 0x7fbd98113ad0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:16.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.098+0000 7fbd9f59d700 1 --2- 172.21.15.32:0/3103711396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd981037b0 0x7fbd98114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:16.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.098+0000 7fbd9f59d700 1 -- 172.21.15.32:0/3103711396 >> 172.21.15.32:0/3103711396 conn(0x7fbd980f89a0 msgr2=0x7fbd980f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:16.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.098+0000 7fbd9f59d700 1 -- 172.21.15.32:0/3103711396 shutdown_connections 2024-04-12T05:44:16.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.098+0000 7fbd9f59d700 1 -- 172.21.15.32:0/3103711396 wait complete. 2024-04-12T05:44:16.105 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:44:16.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.209+0000 7f8b38b2e700 1 -- 172.21.15.32:0/484115723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b340fde90 msgr2=0x7f8b340fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:16.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.209+0000 7f8b38b2e700 1 --2- 172.21.15.32:0/484115723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b340fde90 0x7f8b340fe260 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8b24009ce0 tx=0x7f8b24004d10 comp rx=0 tx=0).stop 2024-04-12T05:44:16.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.209+0000 7f8b38b2e700 1 -- 172.21.15.32:0/484115723 shutdown_connections 2024-04-12T05:44:16.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.209+0000 7f8b38b2e700 1 --2- 172.21.15.32:0/484115723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b340fe830 0x7f8b34106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:16.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.209+0000 7f8b38b2e700 1 --2- 172.21.15.32:0/484115723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b340fde90 0x7f8b340fe260 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:16.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.209+0000 7f8b38b2e700 1 -- 172.21.15.32:0/484115723 >> 172.21.15.32:0/484115723 conn(0x7f8b34072bc0 msgr2=0x7f8b34072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:16.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.209+0000 7f8b38b2e700 1 -- 172.21.15.32:0/484115723 shutdown_connections 2024-04-12T05:44:16.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.209+0000 7f8b38b2e700 1 -- 172.21.15.32:0/484115723 wait complete. 2024-04-12T05:44:16.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.210+0000 7f8b38b2e700 1 Processor -- start 2024-04-12T05:44:16.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.210+0000 7f8b38b2e700 1 -- start start 2024-04-12T05:44:16.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.210+0000 7f8b38b2e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b340fde90 0x7f8b34113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:16.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.210+0000 7f8b38b2e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b340fe830 0x7f8b34113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:16.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.210+0000 7f8b38b2e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b341145f0 con 0x7f8b340fde90 2024-04-12T05:44:16.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.210+0000 7f8b38b2e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3410da30 con 0x7f8b340fe830 2024-04-12T05:44:16.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.210+0000 7f8b3259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b340fde90 0x7f8b34113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:16.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.210+0000 7f8b3259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b340fde90 0x7f8b34113a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40408/0 (socket says 172.21.15.32:40408) 2024-04-12T05:44:16.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.210+0000 7f8b3259c700 1 -- 172.21.15.32:0/742784134 learned_addr learned my addr 172.21.15.32:0/742784134 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:44:16.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.210+0000 7f8b31d9b700 1 --2- 172.21.15.32:0/742784134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b340fe830 0x7f8b34113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:16.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.210+0000 7f8b3259c700 1 -- 172.21.15.32:0/742784134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b340fe830 msgr2=0x7f8b34113fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:16.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.210+0000 7f8b3259c700 1 --2- 172.21.15.32:0/742784134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b340fe830 0x7f8b34113fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:16.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.210+0000 7f8b3259c700 1 -- 172.21.15.32:0/742784134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b24009940 con 0x7f8b340fde90 2024-04-12T05:44:16.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.211+0000 7f8b3259c700 1 --2- 172.21.15.32:0/742784134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b340fde90 0x7f8b34113a60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8b24009380 tx=0x7f8b24004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:16.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.211+0000 7f8b22ffd700 1 -- 172.21.15.32:0/742784134 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b2400e040 con 0x7f8b340fde90 2024-04-12T05:44:16.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.211+0000 7f8b38b2e700 1 -- 172.21.15.32:0/742784134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b3410dcb0 con 0x7f8b340fde90 2024-04-12T05:44:16.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.211+0000 7f8b22ffd700 1 -- 172.21.15.32:0/742784134 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b24004610 con 0x7f8b340fde90 2024-04-12T05:44:16.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.211+0000 7f8b22ffd700 1 -- 172.21.15.32:0/742784134 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b240129d0 con 0x7f8b340fde90 2024-04-12T05:44:16.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.211+0000 7f8b38b2e700 1 -- 172.21.15.32:0/742784134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b3410e1f0 con 0x7f8b340fde90 2024-04-12T05:44:16.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.212+0000 7f8b38b2e700 1 -- 172.21.15.32:0/742784134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b34065e30 con 0x7f8b340fde90 2024-04-12T05:44:16.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.212+0000 7f8b22ffd700 1 -- 172.21.15.32:0/742784134 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8b2401c650 con 0x7f8b340fde90 2024-04-12T05:44:16.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.213+0000 7f8b22ffd700 1 --2- 172.21.15.32:0/742784134 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f8b1c06cc20 0x7f8b1c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:16.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.213+0000 7f8b22ffd700 1 -- 172.21.15.32:0/742784134 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f8b2408e290 con 0x7f8b340fde90 2024-04-12T05:44:16.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.213+0000 7f8b31d9b700 1 --2- 172.21.15.32:0/742784134 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f8b1c06cc20 0x7f8b1c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:16.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.213+0000 7f8b31d9b700 1 --2- 172.21.15.32:0/742784134 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f8b1c06cc20 0x7f8b1c06f0d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8b28004600 tx=0x7f8b2800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:16.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.218+0000 7f8b22ffd700 1 -- 172.21.15.32:0/742784134 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8b2405bd30 con 0x7f8b340fde90 2024-04-12T05:44:16.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.444+0000 7f8b38b2e700 1 -- 172.21.15.32:0/742784134 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b34109e10 con 0x7f8b1c06cc20 2024-04-12T05:44:16.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.445+0000 7f8b22ffd700 1 -- 172.21.15.32:0/742784134 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f8b34109e10 con 0x7f8b1c06cc20 2024-04-12T05:44:16.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.447+0000 7f8b38b2e700 1 -- 172.21.15.32:0/742784134 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f8b1c06cc20 msgr2=0x7f8b1c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:16.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.447+0000 7f8b38b2e700 1 --2- 172.21.15.32:0/742784134 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f8b1c06cc20 0x7f8b1c06f0d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8b28004600 tx=0x7f8b2800b040 comp rx=0 tx=0).stop 2024-04-12T05:44:16.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.447+0000 7f8b38b2e700 1 -- 172.21.15.32:0/742784134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b340fde90 msgr2=0x7f8b34113a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:16.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.447+0000 7f8b38b2e700 1 --2- 172.21.15.32:0/742784134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b340fde90 0x7f8b34113a60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8b24009380 tx=0x7f8b24004d10 comp rx=0 tx=0).stop 2024-04-12T05:44:16.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.447+0000 7f8b38b2e700 1 -- 172.21.15.32:0/742784134 shutdown_connections 2024-04-12T05:44:16.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.447+0000 7f8b38b2e700 1 --2- 172.21.15.32:0/742784134 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f8b1c06cc20 0x7f8b1c06f0d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:16.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.447+0000 7f8b38b2e700 1 --2- 172.21.15.32:0/742784134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b340fde90 0x7f8b34113a60 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:16.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.447+0000 7f8b38b2e700 1 --2- 172.21.15.32:0/742784134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b340fe830 0x7f8b34113fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:16.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.447+0000 7f8b38b2e700 1 -- 172.21.15.32:0/742784134 >> 172.21.15.32:0/742784134 conn(0x7f8b34072bc0 msgr2=0x7f8b34103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:16.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.447+0000 7f8b38b2e700 1 -- 172.21.15.32:0/742784134 shutdown_connections 2024-04-12T05:44:16.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.447+0000 7f8b38b2e700 1 -- 172.21.15.32:0/742784134 wait complete. 2024-04-12T05:44:16.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.553+0000 7f4f6bc6f700 1 -- 172.21.15.32:0/3037408902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f640fde10 msgr2=0x7f4f640fe1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:16.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.553+0000 7f4f6bc6f700 1 --2- 172.21.15.32:0/3037408902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f640fde10 0x7f4f640fe1e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4f60004940 tx=0x7f4f60009e30 comp rx=0 tx=0).stop 2024-04-12T05:44:16.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.553+0000 7f4f6bc6f700 1 -- 172.21.15.32:0/3037408902 shutdown_connections 2024-04-12T05:44:16.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.553+0000 7f4f6bc6f700 1 --2- 172.21.15.32:0/3037408902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f640fe7b0 0x7f4f64106df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:16.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.553+0000 7f4f6bc6f700 1 --2- 172.21.15.32:0/3037408902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f640fde10 0x7f4f640fe1e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:16.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.553+0000 7f4f6bc6f700 1 -- 172.21.15.32:0/3037408902 >> 172.21.15.32:0/3037408902 conn(0x7f4f640f7880 msgr2=0x7f4f640f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:16.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.554+0000 7f4f6bc6f700 1 -- 172.21.15.32:0/3037408902 shutdown_connections 2024-04-12T05:44:16.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.554+0000 7f4f6bc6f700 1 -- 172.21.15.32:0/3037408902 wait complete. 2024-04-12T05:44:16.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.554+0000 7f4f6bc6f700 1 Processor -- start 2024-04-12T05:44:16.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.554+0000 7f4f6bc6f700 1 -- start start 2024-04-12T05:44:16.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.554+0000 7f4f6bc6f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f640fde10 0x7f4f64113d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:16.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.554+0000 7f4f6bc6f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f640fe7b0 0x7f4f64114260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:16.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.554+0000 7f4f6bc6f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f641148b0 con 0x7f4f640fe7b0 2024-04-12T05:44:16.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.554+0000 7f4f6bc6f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6410dcd0 con 0x7f4f640fde10 2024-04-12T05:44:16.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.554+0000 7f4f6920a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f640fe7b0 0x7f4f64114260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:16.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.554+0000 7f4f6920a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f640fe7b0 0x7f4f64114260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40414/0 (socket says 172.21.15.32:40414) 2024-04-12T05:44:16.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.555+0000 7f4f6920a700 1 -- 172.21.15.32:0/1598876427 learned_addr learned my addr 172.21.15.32:0/1598876427 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:44:16.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.555+0000 7f4f69a0b700 1 --2- 172.21.15.32:0/1598876427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f640fde10 0x7f4f64113d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:16.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.555+0000 7f4f6920a700 1 -- 172.21.15.32:0/1598876427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f640fde10 msgr2=0x7f4f64113d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:16.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.555+0000 7f4f6920a700 1 --2- 172.21.15.32:0/1598876427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f640fde10 0x7f4f64113d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:16.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.555+0000 7f4f6920a700 1 -- 172.21.15.32:0/1598876427 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f60005040 con 0x7f4f640fe7b0 2024-04-12T05:44:16.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.555+0000 7f4f69a0b700 1 --2- 172.21.15.32:0/1598876427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f640fde10 0x7f4f64113d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:44:16.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.555+0000 7f4f6920a700 1 --2- 172.21.15.32:0/1598876427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f640fe7b0 0x7f4f64114260 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4f5400eb40 tx=0x7f4f5400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:16.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.555+0000 7f4f5a7fc700 1 -- 172.21.15.32:0/1598876427 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f54007cc0 con 0x7f4f640fe7b0 2024-04-12T05:44:16.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.555+0000 7f4f5a7fc700 1 -- 172.21.15.32:0/1598876427 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f5400f040 con 0x7f4f640fe7b0 2024-04-12T05:44:16.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.555+0000 7f4f6bc6f700 1 -- 172.21.15.32:0/1598876427 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f6410df50 con 0x7f4f640fe7b0 2024-04-12T05:44:16.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.555+0000 7f4f5a7fc700 1 -- 172.21.15.32:0/1598876427 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f54009570 con 0x7f4f640fe7b0 2024-04-12T05:44:16.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.555+0000 7f4f6bc6f700 1 -- 172.21.15.32:0/1598876427 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f6410e490 con 0x7f4f640fe7b0 2024-04-12T05:44:16.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.556+0000 7f4f6bc6f700 1 -- 172.21.15.32:0/1598876427 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f64066770 con 0x7f4f640fe7b0 2024-04-12T05:44:16.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.558+0000 7f4f5a7fc700 1 -- 172.21.15.32:0/1598876427 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4f540096d0 con 0x7f4f640fe7b0 2024-04-12T05:44:16.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.558+0000 7f4f5a7fc700 1 --2- 172.21.15.32:0/1598876427 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f4f5006cc30 0x7f4f5006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:16.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.558+0000 7f4f5a7fc700 1 -- 172.21.15.32:0/1598876427 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f4f5408bf20 con 0x7f4f640fe7b0 2024-04-12T05:44:16.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.558+0000 7f4f69a0b700 1 --2- 172.21.15.32:0/1598876427 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f4f5006cc30 0x7f4f5006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:16.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.559+0000 7f4f69a0b700 1 --2- 172.21.15.32:0/1598876427 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f4f5006cc30 0x7f4f5006f0e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4f6000b0d0 tx=0x7f4f60009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:16.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.562+0000 7f4f5a7fc700 1 -- 172.21.15.32:0/1598876427 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4f54059af0 con 0x7f4f640fe7b0 2024-04-12T05:44:16.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:16 smithi067 ceph-mon[112979]: osdmap e61: 6 total, 5 up, 6 in 2024-04-12T05:44:16.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.777+0000 7f4f6bc6f700 1 -- 172.21.15.32:0/1598876427 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f64109dd0 con 0x7f4f5006cc30 2024-04-12T05:44:16.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.787+0000 7f4f5a7fc700 1 -- 172.21.15.32:0/1598876427 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f4f64109dd0 con 0x7f4f5006cc30 2024-04-12T05:44:16.785 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:44:16.785 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (4m) 27s ago 15m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:44:16.785 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (94s) 27s ago 15m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:44:16.785 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (89s) 86s ago 14m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:44:16.785 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (3m) 27s ago 15m 79.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:44:16.785 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (9m) 27s ago 9m 64.7M - 17.2.7-827-gd3b5060a 21de9b7369ad e7ada8e5259c 2024-04-12T05:44:16.785 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (9m) 27s ago 9m 115M - 17.2.7-827-gd3b5060a 21de9b7369ad 55124da9afdb 2024-04-12T05:44:16.785 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (9m) 86s ago 9m 62.6M - 17.2.7-827-gd3b5060a 21de9b7369ad 0c295d31a4a6 2024-04-12T05:44:16.785 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (9m) 86s ago 9m 120M - 17.2.7-827-gd3b5060a 21de9b7369ad 6eecd7deaac2 2024-04-12T05:44:16.785 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (5m) 27s ago 16m 497M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:44:16.785 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (4m) 86s ago 14m 447M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:44:16.785 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (2m) 27s ago 17m 46.0M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:44:16.785 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (2m) 86s ago 14m 33.0M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:44:16.785 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (4m) 27s ago 15m 16.6M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:44:16.786 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (4m) 86s ago 14m 18.0M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:44:16.786 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (69s) 27s ago 13m 85.7M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T05:44:16.786 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (33s) 27s ago 13m 13.9M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T05:44:16.786 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (12m) 27s ago 12m 191M 1570M 17.2.7-827-gd3b5060a 21de9b7369ad f055cf1ca665 2024-04-12T05:44:16.786 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (12m) 86s ago 12m 258M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad 61b851235351 2024-04-12T05:44:16.786 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (11m) 86s ago 11m 230M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad 2c84d9047e9c 2024-04-12T05:44:16.786 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (11m) 86s ago 11m 220M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad e54839bb6055 2024-04-12T05:44:16.786 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (4m) 27s ago 14m 63.5M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:44:16.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.790+0000 7f4f6bc6f700 1 -- 172.21.15.32:0/1598876427 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f4f5006cc30 msgr2=0x7f4f5006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:16.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.790+0000 7f4f6bc6f700 1 --2- 172.21.15.32:0/1598876427 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f4f5006cc30 0x7f4f5006f0e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4f6000b0d0 tx=0x7f4f60009e30 comp rx=0 tx=0).stop 2024-04-12T05:44:16.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.790+0000 7f4f6bc6f700 1 -- 172.21.15.32:0/1598876427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f640fe7b0 msgr2=0x7f4f64114260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:16.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.790+0000 7f4f6bc6f700 1 --2- 172.21.15.32:0/1598876427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f640fe7b0 0x7f4f64114260 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4f5400eb40 tx=0x7f4f5400b040 comp rx=0 tx=0).stop 2024-04-12T05:44:16.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.790+0000 7f4f6bc6f700 1 -- 172.21.15.32:0/1598876427 shutdown_connections 2024-04-12T05:44:16.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.790+0000 7f4f6bc6f700 1 --2- 172.21.15.32:0/1598876427 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f4f5006cc30 0x7f4f5006f0e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:16.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.790+0000 7f4f6bc6f700 1 --2- 172.21.15.32:0/1598876427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f640fde10 0x7f4f64113d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:16.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.790+0000 7f4f6bc6f700 1 --2- 172.21.15.32:0/1598876427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f640fe7b0 0x7f4f64114260 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.790+0000 7f4f6bc6f700 1 -- 172.21.15.32:0/1598876427 >> 172.21.15.32:0/1598876427 conn(0x7f4f640f7880 msgr2=0x7f4f640f8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.790+0000 7f4f6bc6f700 1 -- 172.21.15.32:0/1598876427 shutdown_connections 2024-04-12T05:44:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.790+0000 7f4f6bc6f700 1 -- 172.21.15.32:0/1598876427 wait complete. 2024-04-12T05:44:16.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:16 smithi032 ceph-mon[146817]: osdmap e61: 6 total, 5 up, 6 in 2024-04-12T05:44:16.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.894+0000 7f6759bdf700 1 -- 172.21.15.32:0/225113541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6754102710 msgr2=0x7f67540fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:16.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.894+0000 7f6759bdf700 1 --2- 172.21.15.32:0/225113541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6754102710 0x7f67540fbd80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6748004940 tx=0x7f6748009e30 comp rx=0 tx=0).stop 2024-04-12T05:44:16.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.894+0000 7f6759bdf700 1 -- 172.21.15.32:0/225113541 shutdown_connections 2024-04-12T05:44:16.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.894+0000 7f6759bdf700 1 --2- 172.21.15.32:0/225113541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6754102710 0x7f67540fbd80 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:16.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.894+0000 7f6759bdf700 1 --2- 172.21.15.32:0/225113541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6754101d70 0x7f6754102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:16.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.894+0000 7f6759bdf700 1 -- 172.21.15.32:0/225113541 >> 172.21.15.32:0/225113541 conn(0x7f6754072bc0 msgr2=0x7f6754072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:16.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.894+0000 7f6759bdf700 1 -- 172.21.15.32:0/225113541 shutdown_connections 2024-04-12T05:44:16.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.894+0000 7f6759bdf700 1 -- 172.21.15.32:0/225113541 wait complete. 2024-04-12T05:44:16.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.895+0000 7f6759bdf700 1 Processor -- start 2024-04-12T05:44:16.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.895+0000 7f6759bdf700 1 -- start start 2024-04-12T05:44:16.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.895+0000 7f6759bdf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6754101d70 0x7f67541032d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:16.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.895+0000 7f6759bdf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6754102710 0x7f6754103810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:16.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.895+0000 7f6759bdf700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6754103da0 con 0x7f6754102710 2024-04-12T05:44:16.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.895+0000 7f6759bdf700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6754103ee0 con 0x7f6754101d70 2024-04-12T05:44:16.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.895+0000 7f6752ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6754102710 0x7f6754103810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:16.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.895+0000 7f6752ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6754102710 0x7f6754103810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40434/0 (socket says 172.21.15.32:40434) 2024-04-12T05:44:16.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.895+0000 7f6752ffd700 1 -- 172.21.15.32:0/4031518141 learned_addr learned my addr 172.21.15.32:0/4031518141 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:44:16.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.896+0000 7f67537fe700 1 --2- 172.21.15.32:0/4031518141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6754101d70 0x7f67541032d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:16.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.896+0000 7f6752ffd700 1 -- 172.21.15.32:0/4031518141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6754101d70 msgr2=0x7f67541032d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:16.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.896+0000 7f6752ffd700 1 --2- 172.21.15.32:0/4031518141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6754101d70 0x7f67541032d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:16.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.896+0000 7f6752ffd700 1 -- 172.21.15.32:0/4031518141 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6748005040 con 0x7f6754102710 2024-04-12T05:44:16.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.896+0000 7f6752ffd700 1 --2- 172.21.15.32:0/4031518141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6754102710 0x7f6754103810 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6748008d50 tx=0x7f6748009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:16.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.896+0000 7f673bfff700 1 -- 172.21.15.32:0/4031518141 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f674800e030 con 0x7f6754102710 2024-04-12T05:44:16.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.896+0000 7f673bfff700 1 -- 172.21.15.32:0/4031518141 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6748013de0 con 0x7f6754102710 2024-04-12T05:44:16.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.896+0000 7f6759bdf700 1 -- 172.21.15.32:0/4031518141 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6754104130 con 0x7f6754102710 2024-04-12T05:44:16.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.896+0000 7f673bfff700 1 -- 172.21.15.32:0/4031518141 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f674801cd90 con 0x7f6754102710 2024-04-12T05:44:16.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.896+0000 7f6759bdf700 1 -- 172.21.15.32:0/4031518141 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f675411a140 con 0x7f6754102710 2024-04-12T05:44:16.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.897+0000 7f6759bdf700 1 -- 172.21.15.32:0/4031518141 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67540fd4c0 con 0x7f6754102710 2024-04-12T05:44:16.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.898+0000 7f673bfff700 1 -- 172.21.15.32:0/4031518141 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f6748013930 con 0x7f6754102710 2024-04-12T05:44:16.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.898+0000 7f673bfff700 1 --2- 172.21.15.32:0/4031518141 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f673c06cc30 0x7f673c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:16.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.898+0000 7f673bfff700 1 -- 172.21.15.32:0/4031518141 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f674808d890 con 0x7f6754102710 2024-04-12T05:44:16.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.898+0000 7f67537fe700 1 --2- 172.21.15.32:0/4031518141 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f673c06cc30 0x7f673c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:16.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.899+0000 7f67537fe700 1 --2- 172.21.15.32:0/4031518141 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f673c06cc30 0x7f673c06f0e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6744004600 tx=0x7f674400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:16.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:16.903+0000 7f673bfff700 1 -- 172.21.15.32:0/4031518141 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6748057920 con 0x7f6754102710 2024-04-12T05:44:17.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.193+0000 7f6759bdf700 1 -- 172.21.15.32:0/4031518141 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6754108d40 con 0x7f6754102710 2024-04-12T05:44:17.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.193+0000 7f673bfff700 1 -- 172.21.15.32:0/4031518141 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f674805af40 con 0x7f6754102710 2024-04-12T05:44:17.190 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:44:17.191 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:44:17.191 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:44:17.191 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:44:17.191 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:44:17.191 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:44:17.191 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:44:17.191 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:44:17.191 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 3, 2024-04-12T05:44:17.191 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:44:17.191 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:44:17.191 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:44:17.191 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 4 2024-04-12T05:44:17.191 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:44:17.191 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:44:17.192 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 7, 2024-04-12T05:44:17.192 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:44:17.192 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:44:17.192 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:44:17.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.195+0000 7f6759bdf700 1 -- 172.21.15.32:0/4031518141 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f673c06cc30 msgr2=0x7f673c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:17.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.195+0000 7f6759bdf700 1 --2- 172.21.15.32:0/4031518141 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f673c06cc30 0x7f673c06f0e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6744004600 tx=0x7f674400b040 comp rx=0 tx=0).stop 2024-04-12T05:44:17.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.195+0000 7f6759bdf700 1 -- 172.21.15.32:0/4031518141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6754102710 msgr2=0x7f6754103810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:17.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.195+0000 7f6759bdf700 1 --2- 172.21.15.32:0/4031518141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6754102710 0x7f6754103810 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6748008d50 tx=0x7f6748009e30 comp rx=0 tx=0).stop 2024-04-12T05:44:17.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.195+0000 7f6759bdf700 1 -- 172.21.15.32:0/4031518141 shutdown_connections 2024-04-12T05:44:17.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.195+0000 7f6759bdf700 1 --2- 172.21.15.32:0/4031518141 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f673c06cc30 0x7f673c06f0e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:17.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.195+0000 7f6759bdf700 1 --2- 172.21.15.32:0/4031518141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6754101d70 0x7f67541032d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:17.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.195+0000 7f6759bdf700 1 --2- 172.21.15.32:0/4031518141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6754102710 0x7f6754103810 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:17.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.195+0000 7f6759bdf700 1 -- 172.21.15.32:0/4031518141 >> 172.21.15.32:0/4031518141 conn(0x7f6754072bc0 msgr2=0x7f67541056f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:17.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.196+0000 7f6759bdf700 1 -- 172.21.15.32:0/4031518141 shutdown_connections 2024-04-12T05:44:17.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.196+0000 7f6759bdf700 1 -- 172.21.15.32:0/4031518141 wait complete. 2024-04-12T05:44:17.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.308+0000 7f528b14c700 1 -- 172.21.15.32:0/3882324480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52840fdeb0 msgr2=0x7f52840fe280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:17.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.308+0000 7f528b14c700 1 --2- 172.21.15.32:0/3882324480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52840fdeb0 0x7f52840fe280 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5278004970 tx=0x7f5278009e30 comp rx=0 tx=0).stop 2024-04-12T05:44:17.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.308+0000 7f528b14c700 1 -- 172.21.15.32:0/3882324480 shutdown_connections 2024-04-12T05:44:17.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.308+0000 7f528b14c700 1 --2- 172.21.15.32:0/3882324480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52840fe850 0x7f5284106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:17.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.308+0000 7f528b14c700 1 --2- 172.21.15.32:0/3882324480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52840fdeb0 0x7f52840fe280 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:17.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.308+0000 7f528b14c700 1 -- 172.21.15.32:0/3882324480 >> 172.21.15.32:0/3882324480 conn(0x7f5284072bc0 msgr2=0x7f5284072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:17.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.308+0000 7f528b14c700 1 -- 172.21.15.32:0/3882324480 shutdown_connections 2024-04-12T05:44:17.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.308+0000 7f528b14c700 1 -- 172.21.15.32:0/3882324480 wait complete. 2024-04-12T05:44:17.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.309+0000 7f528b14c700 1 Processor -- start 2024-04-12T05:44:17.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.309+0000 7f528b14c700 1 -- start start 2024-04-12T05:44:17.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.309+0000 7f528b14c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52840fdeb0 0x7f5284113a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:17.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.309+0000 7f528b14c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52840fe850 0x7f5284113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:17.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.309+0000 7f528b14c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52841145b0 con 0x7f52840fe850 2024-04-12T05:44:17.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.309+0000 7f528b14c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528410d9f0 con 0x7f52840fdeb0 2024-04-12T05:44:17.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.309+0000 7f5283fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52840fe850 0x7f5284113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:17.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.309+0000 7f5283fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52840fe850 0x7f5284113f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40456/0 (socket says 172.21.15.32:40456) 2024-04-12T05:44:17.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.309+0000 7f5283fff700 1 -- 172.21.15.32:0/1188940458 learned_addr learned my addr 172.21.15.32:0/1188940458 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:44:17.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.309+0000 7f5288ee8700 1 --2- 172.21.15.32:0/1188940458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52840fdeb0 0x7f5284113a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:17.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.310+0000 7f5283fff700 1 -- 172.21.15.32:0/1188940458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52840fdeb0 msgr2=0x7f5284113a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:17.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.310+0000 7f5283fff700 1 --2- 172.21.15.32:0/1188940458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52840fdeb0 0x7f5284113a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:17.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.310+0000 7f5283fff700 1 -- 172.21.15.32:0/1188940458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5278005040 con 0x7f52840fe850 2024-04-12T05:44:17.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.310+0000 7f5283fff700 1 --2- 172.21.15.32:0/1188940458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52840fe850 0x7f5284113f60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5270002a50 tx=0x7f52700046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:17.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.310+0000 7f52817fa700 1 -- 172.21.15.32:0/1188940458 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5270007cc0 con 0x7f52840fe850 2024-04-12T05:44:17.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.310+0000 7f52817fa700 1 -- 172.21.15.32:0/1188940458 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f527000f040 con 0x7f52840fe850 2024-04-12T05:44:17.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.310+0000 7f528b14c700 1 -- 172.21.15.32:0/1188940458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f528410dcd0 con 0x7f52840fe850 2024-04-12T05:44:17.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.310+0000 7f52817fa700 1 -- 172.21.15.32:0/1188940458 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f527000d9c0 con 0x7f52840fe850 2024-04-12T05:44:17.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.310+0000 7f528b14c700 1 -- 172.21.15.32:0/1188940458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f528410e210 con 0x7f52840fe850 2024-04-12T05:44:17.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.311+0000 7f528b14c700 1 -- 172.21.15.32:0/1188940458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5284065e10 con 0x7f52840fe850 2024-04-12T05:44:17.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.312+0000 7f52817fa700 1 -- 172.21.15.32:0/1188940458 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f527000db20 con 0x7f52840fe850 2024-04-12T05:44:17.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.312+0000 7f52817fa700 1 --2- 172.21.15.32:0/1188940458 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f527406cc30 0x7f527406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:17.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.312+0000 7f52817fa700 1 -- 172.21.15.32:0/1188940458 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f527008bc60 con 0x7f52840fe850 2024-04-12T05:44:17.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.312+0000 7f5288ee8700 1 --2- 172.21.15.32:0/1188940458 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f527406cc30 0x7f527406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:17.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.313+0000 7f5288ee8700 1 --2- 172.21.15.32:0/1188940458 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f527406cc30 0x7f527406f0e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5278009fd0 tx=0x7f5278009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:17.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.317+0000 7f52817fa700 1 -- 172.21.15.32:0/1188940458 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5270059780 con 0x7f52840fe850 2024-04-12T05:44:17.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.584+0000 7f528b14c700 1 -- 172.21.15.32:0/1188940458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f528405f5f0 con 0x7f52840fe850 2024-04-12T05:44:17.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.585+0000 7f52817fa700 1 -- 172.21.15.32:0/1188940458 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1864 (secure 0 0 0) 0x7f5270059310 con 0x7f52840fe850 2024-04-12T05:44:17.583 INFO:teuthology.orchestra.run.smithi032.stdout:e22 2024-04-12T05:44:17.583 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:44:17.583 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:44:17.583 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:44:17.583 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:44:17.583 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:44:17.583 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:44:17.583 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 22 2024-04-12T05:44:17.583 INFO:teuthology.orchestra.run.smithi032.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-12T05:44:17.583 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:44:17.583 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:41:05.358509+0000 2024-04-12T05:44:17.583 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:44:17.583 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:44:17.583 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:44:17.584 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:44:17.584 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:44:17.584 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:44:17.584 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:44:17.584 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:44:17.584 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 46 2024-04-12T05:44:17.584 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:44:17.584 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:44:17.584 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:44:17.584 INFO:teuthology.orchestra.run.smithi032.stdout:up {0=14480,1=24281} 2024-04-12T05:44:17.584 INFO:teuthology.orchestra.run.smithi032.stdout:failed 2024-04-12T05:44:17.584 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:44:17.584 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:44:17.584 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:44:17.585 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:44:17.585 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:44:17.585 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:44:17.585 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:44:17.585 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:44:17.585 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{0:14480} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.32:6828/387754149,v1:172.21.15.32:6829/387754149] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:44:17.585 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{0:14490} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3678031004,v1:172.21.15.32:6827/3678031004] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:44:17.585 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{1:24281} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.67:6824/1291914599,v1:172.21.15.67:6825/1291914599] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:44:17.585 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{1:24295} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.67:6826/873395842,v1:172.21.15.67:6827/873395842] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:44:17.585 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:44:17.585 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:44:17.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.587+0000 7f528b14c700 1 -- 172.21.15.32:0/1188940458 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f527406cc30 msgr2=0x7f527406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:17.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.587+0000 7f528b14c700 1 --2- 172.21.15.32:0/1188940458 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f527406cc30 0x7f527406f0e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5278009fd0 tx=0x7f5278009e30 comp rx=0 tx=0).stop 2024-04-12T05:44:17.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.587+0000 7f528b14c700 1 -- 172.21.15.32:0/1188940458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52840fe850 msgr2=0x7f5284113f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:17.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.587+0000 7f528b14c700 1 --2- 172.21.15.32:0/1188940458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52840fe850 0x7f5284113f60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5270002a50 tx=0x7f52700046b0 comp rx=0 tx=0).stop 2024-04-12T05:44:17.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.588+0000 7f528b14c700 1 -- 172.21.15.32:0/1188940458 shutdown_connections 2024-04-12T05:44:17.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.588+0000 7f528b14c700 1 --2- 172.21.15.32:0/1188940458 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f527406cc30 0x7f527406f0e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:17.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.588+0000 7f528b14c700 1 --2- 172.21.15.32:0/1188940458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52840fdeb0 0x7f5284113a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:17.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.588+0000 7f528b14c700 1 --2- 172.21.15.32:0/1188940458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52840fe850 0x7f5284113f60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:17.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.588+0000 7f528b14c700 1 -- 172.21.15.32:0/1188940458 >> 172.21.15.32:0/1188940458 conn(0x7f5284072bc0 msgr2=0x7f52840ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:17.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.588+0000 7f528b14c700 1 -- 172.21.15.32:0/1188940458 shutdown_connections 2024-04-12T05:44:17.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.588+0000 7f528b14c700 1 -- 172.21.15.32:0/1188940458 wait complete. 2024-04-12T05:44:17.587 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 22 2024-04-12T05:44:17.588 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:44:17 smithi032 systemd[1]: ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@osd.2.service: Succeeded. 2024-04-12T05:44:17.589 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:44:17 smithi032 systemd[1]: Stopped Ceph osd.2 for 37b58ce2-f88d-11ee-bc8c-c7b262605968. 2024-04-12T05:44:17.589 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:44:17 smithi032 systemd[1]: Starting Ceph osd.2 for 37b58ce2-f88d-11ee-bc8c-c7b262605968... 2024-04-12T05:44:17.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:17 smithi032 ceph-mon[146817]: pgmap v89: 65 pgs: 7 peering, 5 stale+active+clean, 53 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:44:17.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:17 smithi032 ceph-mon[146817]: from='client.34248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:17.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:17 smithi032 ceph-mon[146817]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-04-12T05:44:17.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:17 smithi032 ceph-mon[146817]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:17.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:17 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/4031518141' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:44:17.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.694+0000 7f9e7c3e9700 1 -- 172.21.15.32:0/4055843213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e740656a0 msgr2=0x7f9e74065af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:17.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.694+0000 7f9e7c3e9700 1 --2- 172.21.15.32:0/4055843213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e740656a0 0x7f9e74065af0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9e64004940 tx=0x7f9e64009e30 comp rx=0 tx=0).stop 2024-04-12T05:44:17.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.695+0000 7f9e7c3e9700 1 -- 172.21.15.32:0/4055843213 shutdown_connections 2024-04-12T05:44:17.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.695+0000 7f9e7c3e9700 1 --2- 172.21.15.32:0/4055843213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e740656a0 0x7f9e74065af0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:17.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.695+0000 7f9e7c3e9700 1 --2- 172.21.15.32:0/4055843213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e74102de0 0x7f9e741031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:17.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.695+0000 7f9e7c3e9700 1 -- 172.21.15.32:0/4055843213 >> 172.21.15.32:0/4055843213 conn(0x7f9e74072bc0 msgr2=0x7f9e74072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:17.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.695+0000 7f9e7c3e9700 1 -- 172.21.15.32:0/4055843213 shutdown_connections 2024-04-12T05:44:17.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.695+0000 7f9e7c3e9700 1 -- 172.21.15.32:0/4055843213 wait complete. 2024-04-12T05:44:17.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.695+0000 7f9e7c3e9700 1 Processor -- start 2024-04-12T05:44:17.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.695+0000 7f9e7c3e9700 1 -- start start 2024-04-12T05:44:17.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.695+0000 7f9e7c3e9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e740656a0 0x7f9e740fbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:17.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.695+0000 7f9e7c3e9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e74102de0 0x7f9e740fc210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:17.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.695+0000 7f9e7c3e9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e740fc7a0 con 0x7f9e74102de0 2024-04-12T05:44:17.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.695+0000 7f9e7c3e9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e740fc8e0 con 0x7f9e740656a0 2024-04-12T05:44:17.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.696+0000 7f9e79984700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e74102de0 0x7f9e740fc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:17.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.696+0000 7f9e79984700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e74102de0 0x7f9e740fc210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40474/0 (socket says 172.21.15.32:40474) 2024-04-12T05:44:17.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.696+0000 7f9e79984700 1 -- 172.21.15.32:0/1474378958 learned_addr learned my addr 172.21.15.32:0/1474378958 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:44:17.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.696+0000 7f9e7a185700 1 --2- 172.21.15.32:0/1474378958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e740656a0 0x7f9e740fbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:17.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.696+0000 7f9e79984700 1 -- 172.21.15.32:0/1474378958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e740656a0 msgr2=0x7f9e740fbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:17.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.696+0000 7f9e79984700 1 --2- 172.21.15.32:0/1474378958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e740656a0 0x7f9e740fbcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:17.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.696+0000 7f9e79984700 1 -- 172.21.15.32:0/1474378958 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e64005040 con 0x7f9e74102de0 2024-04-12T05:44:17.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.696+0000 7f9e79984700 1 --2- 172.21.15.32:0/1474378958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e74102de0 0x7f9e740fc210 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9e64009180 tx=0x7f9e64009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:17.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.696+0000 7f9e6affd700 1 -- 172.21.15.32:0/1474378958 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e6400e040 con 0x7f9e74102de0 2024-04-12T05:44:17.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.696+0000 7f9e6affd700 1 -- 172.21.15.32:0/1474378958 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e64004290 con 0x7f9e74102de0 2024-04-12T05:44:17.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.696+0000 7f9e7c3e9700 1 -- 172.21.15.32:0/1474378958 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e740fcb30 con 0x7f9e74102de0 2024-04-12T05:44:17.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.696+0000 7f9e6affd700 1 -- 172.21.15.32:0/1474378958 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e64013e60 con 0x7f9e74102de0 2024-04-12T05:44:17.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.696+0000 7f9e7c3e9700 1 -- 172.21.15.32:0/1474378958 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e7411a0d0 con 0x7f9e74102de0 2024-04-12T05:44:17.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.697+0000 7f9e7c3e9700 1 -- 172.21.15.32:0/1474378958 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e740fe4c0 con 0x7f9e74102de0 2024-04-12T05:44:17.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.698+0000 7f9e6affd700 1 -- 172.21.15.32:0/1474378958 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9e640136b0 con 0x7f9e74102de0 2024-04-12T05:44:17.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.698+0000 7f9e6affd700 1 --2- 172.21.15.32:0/1474378958 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9e6006cc20 0x7f9e6006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:17.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.698+0000 7f9e6affd700 1 -- 172.21.15.32:0/1474378958 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f9e6408d120 con 0x7f9e74102de0 2024-04-12T05:44:17.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.698+0000 7f9e7a185700 1 --2- 172.21.15.32:0/1474378958 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9e6006cc20 0x7f9e6006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:17.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.699+0000 7f9e7a185700 1 --2- 172.21.15.32:0/1474378958 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9e6006cc20 0x7f9e6006f0d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9e70009fd0 tx=0x7f9e7000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:17.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.703+0000 7f9e6affd700 1 -- 172.21.15.32:0/1474378958 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9e64057250 con 0x7f9e74102de0 2024-04-12T05:44:17.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:17 smithi067 ceph-mon[112979]: pgmap v89: 65 pgs: 7 peering, 5 stale+active+clean, 53 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:44:17.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:17 smithi067 ceph-mon[112979]: from='client.34248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:17.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:17 smithi067 ceph-mon[112979]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-04-12T05:44:17.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:17 smithi067 ceph-mon[112979]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:17.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:17 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/4031518141' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:44:17.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.936+0000 7f9e7c3e9700 1 -- 172.21.15.32:0/1474378958 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e74109d80 con 0x7f9e6006cc20 2024-04-12T05:44:17.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.938+0000 7f9e6affd700 1 -- 172.21.15.32:0/1474378958 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f9e7000f040 con 0x7f9e6006cc20 2024-04-12T05:44:17.937 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:44:17.937 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:44:17.937 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:44:17.937 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:44:17.937 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:44:17.938 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:44:17.938 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:44:17.938 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-12T05:44:17.938 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:44:17.938 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "8/21 daemons upgraded", 2024-04-12T05:44:17.938 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading osd daemons", 2024-04-12T05:44:17.939 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:44:17.939 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:44:17.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.940+0000 7f9e7c3e9700 1 -- 172.21.15.32:0/1474378958 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9e6006cc20 msgr2=0x7f9e6006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:17.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.940+0000 7f9e7c3e9700 1 --2- 172.21.15.32:0/1474378958 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9e6006cc20 0x7f9e6006f0d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9e70009fd0 tx=0x7f9e7000b040 comp rx=0 tx=0).stop 2024-04-12T05:44:17.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.940+0000 7f9e7c3e9700 1 -- 172.21.15.32:0/1474378958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e74102de0 msgr2=0x7f9e740fc210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:17.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.940+0000 7f9e7c3e9700 1 --2- 172.21.15.32:0/1474378958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e74102de0 0x7f9e740fc210 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9e64009180 tx=0x7f9e64009e30 comp rx=0 tx=0).stop 2024-04-12T05:44:17.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.940+0000 7f9e7c3e9700 1 -- 172.21.15.32:0/1474378958 shutdown_connections 2024-04-12T05:44:17.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.940+0000 7f9e7c3e9700 1 --2- 172.21.15.32:0/1474378958 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9e6006cc20 0x7f9e6006f0d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:17.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.940+0000 7f9e7c3e9700 1 --2- 172.21.15.32:0/1474378958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e740656a0 0x7f9e740fbcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:17.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.940+0000 7f9e7c3e9700 1 --2- 172.21.15.32:0/1474378958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e74102de0 0x7f9e740fc210 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:17.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.940+0000 7f9e7c3e9700 1 -- 172.21.15.32:0/1474378958 >> 172.21.15.32:0/1474378958 conn(0x7f9e74072bc0 msgr2=0x7f9e74105b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:17.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.940+0000 7f9e7c3e9700 1 -- 172.21.15.32:0/1474378958 shutdown_connections 2024-04-12T05:44:17.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:17.940+0000 7f9e7c3e9700 1 -- 172.21.15.32:0/1474378958 wait complete. 2024-04-12T05:44:18.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.050+0000 7fe9a60a3700 1 -- 172.21.15.32:0/448255505 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9a0102e00 msgr2=0x7fe9a01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:18.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.050+0000 7fe9a60a3700 1 --2- 172.21.15.32:0/448255505 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9a0102e00 0x7fe9a01031d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe994004940 tx=0x7fe994009e30 comp rx=0 tx=0).stop 2024-04-12T05:44:18.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.050+0000 7fe9a60a3700 1 -- 172.21.15.32:0/448255505 shutdown_connections 2024-04-12T05:44:18.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.050+0000 7fe9a60a3700 1 --2- 172.21.15.32:0/448255505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9a01037a0 0x7fe9a00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:18.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.050+0000 7fe9a60a3700 1 --2- 172.21.15.32:0/448255505 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9a0102e00 0x7fe9a01031d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:18.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.050+0000 7fe9a60a3700 1 -- 172.21.15.32:0/448255505 >> 172.21.15.32:0/448255505 conn(0x7fe9a00f89b0 msgr2=0x7fe9a00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:18.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.050+0000 7fe9a60a3700 1 -- 172.21.15.32:0/448255505 shutdown_connections 2024-04-12T05:44:18.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.050+0000 7fe9a60a3700 1 -- 172.21.15.32:0/448255505 wait complete. 2024-04-12T05:44:18.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.051+0000 7fe9a60a3700 1 Processor -- start 2024-04-12T05:44:18.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.051+0000 7fe9a60a3700 1 -- start start 2024-04-12T05:44:18.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.051+0000 7fe9a60a3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9a0102e00 0x7fe9a0113820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:18.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.051+0000 7fe9a60a3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9a01037a0 0x7fe9a0113d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:18.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.051+0000 7fe9a60a3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a01143b0 con 0x7fe9a0102e00 2024-04-12T05:44:18.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.051+0000 7fe9a60a3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a010d9b0 con 0x7fe9a01037a0 2024-04-12T05:44:18.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.051+0000 7fe99f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9a0102e00 0x7fe9a0113820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:18.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.051+0000 7fe99f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9a0102e00 0x7fe9a0113820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40496/0 (socket says 172.21.15.32:40496) 2024-04-12T05:44:18.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.051+0000 7fe99f7fe700 1 -- 172.21.15.32:0/3131789533 learned_addr learned my addr 172.21.15.32:0/3131789533 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:44:18.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.051+0000 7fe99effd700 1 --2- 172.21.15.32:0/3131789533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9a01037a0 0x7fe9a0113d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:18.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.052+0000 7fe99f7fe700 1 -- 172.21.15.32:0/3131789533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9a01037a0 msgr2=0x7fe9a0113d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:18.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.052+0000 7fe99f7fe700 1 --2- 172.21.15.32:0/3131789533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9a01037a0 0x7fe9a0113d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:18.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.052+0000 7fe99f7fe700 1 -- 172.21.15.32:0/3131789533 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe994005040 con 0x7fe9a0102e00 2024-04-12T05:44:18.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.052+0000 7fe99f7fe700 1 --2- 172.21.15.32:0/3131789533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9a0102e00 0x7fe9a0113820 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe994009390 tx=0x7fe994009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:18.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.052+0000 7fe987fff700 1 -- 172.21.15.32:0/3131789533 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe99400e040 con 0x7fe9a0102e00 2024-04-12T05:44:18.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.052+0000 7fe9a60a3700 1 -- 172.21.15.32:0/3131789533 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9a010dc30 con 0x7fe9a0102e00 2024-04-12T05:44:18.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.052+0000 7fe987fff700 1 -- 172.21.15.32:0/3131789533 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe994013420 con 0x7fe9a0102e00 2024-04-12T05:44:18.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.052+0000 7fe987fff700 1 -- 172.21.15.32:0/3131789533 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe99401c630 con 0x7fe9a0102e00 2024-04-12T05:44:18.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.052+0000 7fe9a60a3700 1 -- 172.21.15.32:0/3131789533 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9a010e1f0 con 0x7fe9a0102e00 2024-04-12T05:44:18.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.053+0000 7fe9a60a3700 1 -- 172.21.15.32:0/3131789533 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9a00fe550 con 0x7fe9a0102e00 2024-04-12T05:44:18.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.054+0000 7fe987fff700 1 -- 172.21.15.32:0/3131789533 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe99400bb50 con 0x7fe9a0102e00 2024-04-12T05:44:18.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.054+0000 7fe987fff700 1 --2- 172.21.15.32:0/3131789533 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe98806cc20 0x7fe98806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:18.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.055+0000 7fe987fff700 1 -- 172.21.15.32:0/3131789533 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fe99408d340 con 0x7fe9a0102e00 2024-04-12T05:44:18.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.055+0000 7fe99effd700 1 --2- 172.21.15.32:0/3131789533 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe98806cc20 0x7fe98806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:18.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.055+0000 7fe99effd700 1 --2- 172.21.15.32:0/3131789533 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe98806cc20 0x7fe98806f0d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe9a010eb70 tx=0x7fe99000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:18.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.059+0000 7fe987fff700 1 -- 172.21.15.32:0/3131789533 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe99405ae60 con 0x7fe9a0102e00 2024-04-12T05:44:18.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.347+0000 7fe9a60a3700 1 -- 172.21.15.32:0/3131789533 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe9a010eed0 con 0x7fe9a0102e00 2024-04-12T05:44:18.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.347+0000 7fe987fff700 1 -- 172.21.15.32:0/3131789533 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+437 (secure 0 0 0) 0x7fe99405a9f0 con 0x7fe9a0102e00 2024-04-12T05:44:18.345 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_WARN 1 osds down; Reduced data availability: 3 pgs peering 2024-04-12T05:44:18.345 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-12T05:44:18.345 INFO:teuthology.orchestra.run.smithi032.stdout: osd.2 (root=default,host=smithi032) is down 2024-04-12T05:44:18.345 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] PG_AVAILABILITY: Reduced data availability: 3 pgs peering 2024-04-12T05:44:18.346 INFO:teuthology.orchestra.run.smithi032.stdout: pg 2.1 is stuck peering for 5m, current state peering, last acting [1,0] 2024-04-12T05:44:18.346 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.7 is stuck peering for 5m, current state peering, last acting [1,4] 2024-04-12T05:44:18.346 INFO:teuthology.orchestra.run.smithi032.stdout: pg 3.8 is stuck peering for 5m, current state peering, last acting [1,5] 2024-04-12T05:44:18.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.349+0000 7fe9a60a3700 1 -- 172.21.15.32:0/3131789533 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe98806cc20 msgr2=0x7fe98806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:18.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.349+0000 7fe9a60a3700 1 --2- 172.21.15.32:0/3131789533 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe98806cc20 0x7fe98806f0d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe9a010eb70 tx=0x7fe99000b040 comp rx=0 tx=0).stop 2024-04-12T05:44:18.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.349+0000 7fe9a60a3700 1 -- 172.21.15.32:0/3131789533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9a0102e00 msgr2=0x7fe9a0113820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:18.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.349+0000 7fe9a60a3700 1 --2- 172.21.15.32:0/3131789533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9a0102e00 0x7fe9a0113820 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe994009390 tx=0x7fe994009e30 comp rx=0 tx=0).stop 2024-04-12T05:44:18.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.349+0000 7fe9a60a3700 1 -- 172.21.15.32:0/3131789533 shutdown_connections 2024-04-12T05:44:18.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.349+0000 7fe9a60a3700 1 --2- 172.21.15.32:0/3131789533 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe98806cc20 0x7fe98806f0d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:18.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.349+0000 7fe9a60a3700 1 --2- 172.21.15.32:0/3131789533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9a0102e00 0x7fe9a0113820 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:18.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.349+0000 7fe9a60a3700 1 --2- 172.21.15.32:0/3131789533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9a01037a0 0x7fe9a0113d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:18.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.349+0000 7fe9a60a3700 1 -- 172.21.15.32:0/3131789533 >> 172.21.15.32:0/3131789533 conn(0x7fe9a00f89b0 msgr2=0x7fe9a00f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:18.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.350+0000 7fe9a60a3700 1 -- 172.21.15.32:0/3131789533 shutdown_connections 2024-04-12T05:44:18.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:18.350+0000 7fe9a60a3700 1 -- 172.21.15.32:0/3131789533 wait complete. 2024-04-12T05:44:18.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:18 smithi067 ceph-mon[112979]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:18.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:18 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1188940458' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:44:18.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:18.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:44:18.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:18 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3131789533' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:44:18.793 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:18 smithi032 ceph-mon[146817]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:18.793 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:18 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1188940458' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:44:18.793 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:18.793 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:44:18.793 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:18 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3131789533' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:44:18.794 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:44:18 smithi032 bash[174537]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-12T05:44:18.794 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:44:18 smithi032 bash[174537]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-12T05:44:18.794 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:44:18 smithi032 bash[174537]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-12T05:44:18.794 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:44:18 smithi032 bash[174537]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-12T05:44:18.794 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:44:18 smithi032 bash[174537]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-12T05:44:18.794 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:44:18 smithi032 bash[174537]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-12T05:44:18.794 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:44:18 smithi032 bash[174537]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-12T05:44:19.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:19 smithi067 ceph-mon[112979]: pgmap v90: 65 pgs: 7 peering, 5 stale+active+clean, 53 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:44:19.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:19 smithi067 ceph-mon[112979]: from='client.34268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:19.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:19 smithi032 ceph-mon[146817]: pgmap v90: 65 pgs: 7 peering, 5 stale+active+clean, 53 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:44:19.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:19 smithi032 ceph-mon[146817]: from='client.34268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:19.839 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:44:19 smithi032 ceph-osd[174886]: -- 172.21.15.32:0/3057719349 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d74ce751e0 con 0x55d74ce45000 2024-04-12T05:44:19.839 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:44:19 smithi032 bash[174847]: 7d9052dbda4f9a12478c2486f1bd2b64e25a7ac703575451e86eeab819973344 2024-04-12T05:44:19.839 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:44:19 smithi032 systemd[1]: Started Ceph osd.2 for 37b58ce2-f88d-11ee-bc8c-c7b262605968. 2024-04-12T05:44:20.338 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:44:20 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-2[174863]: 2024-04-12T05:44:20.054+0000 7fbcbb2cc7c0 -1 Falling back to public interface 2024-04-12T05:44:21.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:20 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:21.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:20 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:21.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:20 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:44:21.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:20 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:21.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:20 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:21.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:20 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:44:21.882 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:21 smithi032 ceph-mon[146817]: pgmap v91: 65 pgs: 9 active+undersized+degraded, 7 peering, 11 active+undersized, 38 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 28/333 objects degraded (8.408%) 2024-04-12T05:44:21.883 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:21 smithi032 ceph-mon[146817]: Health check failed: Degraded data redundancy: 28/333 objects degraded (8.408%), 9 pgs degraded (PG_DEGRADED) 2024-04-12T05:44:22.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:21 smithi067 ceph-mon[112979]: pgmap v91: 65 pgs: 9 active+undersized+degraded, 7 peering, 11 active+undersized, 38 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 28/333 objects degraded (8.408%) 2024-04-12T05:44:22.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:21 smithi067 ceph-mon[112979]: Health check failed: Degraded data redundancy: 28/333 objects degraded (8.408%), 9 pgs degraded (PG_DEGRADED) 2024-04-12T05:44:23.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:22 smithi067 ceph-mon[112979]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2024-04-12T05:44:23.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:22 smithi032 ceph-mon[146817]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2024-04-12T05:44:24.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:23 smithi067 ceph-mon[112979]: pgmap v92: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 41/333 objects degraded (12.312%) 2024-04-12T05:44:24.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:23 smithi032 ceph-mon[146817]: pgmap v92: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 41/333 objects degraded (12.312%) 2024-04-12T05:44:24.665 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:44:24 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-2[174863]: 2024-04-12T05:44:24.387+0000 7fbcbb2cc7c0 -1 osd.2 59 log_to_monitors true 2024-04-12T05:44:25.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:24 smithi067 ceph-mon[112979]: from='osd.2 [v2:172.21.15.32:6818/1610966654,v1:172.21.15.32:6819/1610966654]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-12T05:44:25.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:24 smithi032 ceph-mon[146817]: from='osd.2 [v2:172.21.15.32:6818/1610966654,v1:172.21.15.32:6819/1610966654]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-12T05:44:25.089 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 05:44:24 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-2[174863]: 2024-04-12T05:44:24.692+0000 7fbcac726700 -1 osd.2 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T05:44:26.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:25 smithi067 ceph-mon[112979]: pgmap v93: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 41/333 objects degraded (12.312%) 2024-04-12T05:44:26.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:25 smithi067 ceph-mon[112979]: from='osd.2 [v2:172.21.15.32:6818/1610966654,v1:172.21.15.32:6819/1610966654]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-12T05:44:26.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:25 smithi067 ceph-mon[112979]: osdmap e62: 6 total, 5 up, 6 in 2024-04-12T05:44:26.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:25 smithi067 ceph-mon[112979]: from='osd.2 [v2:172.21.15.32:6818/1610966654,v1:172.21.15.32:6819/1610966654]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-12T05:44:26.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:25 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:26.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:25 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:26.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:25 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:26.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:25 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:26.039 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:25 smithi032 ceph-mon[146817]: pgmap v93: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 270 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 41/333 objects degraded (12.312%) 2024-04-12T05:44:26.039 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:25 smithi032 ceph-mon[146817]: from='osd.2 [v2:172.21.15.32:6818/1610966654,v1:172.21.15.32:6819/1610966654]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-12T05:44:26.039 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:25 smithi032 ceph-mon[146817]: osdmap e62: 6 total, 5 up, 6 in 2024-04-12T05:44:26.039 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:25 smithi032 ceph-mon[146817]: from='osd.2 [v2:172.21.15.32:6818/1610966654,v1:172.21.15.32:6819/1610966654]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-12T05:44:26.039 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:25 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:26.039 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:25 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:26.040 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:25 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:26.040 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:25 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:26.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:26 smithi032 ceph-mon[146817]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T05:44:26.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:26 smithi032 ceph-mon[146817]: osd.2 [v2:172.21.15.32:6818/1610966654,v1:172.21.15.32:6819/1610966654] boot 2024-04-12T05:44:26.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:26 smithi032 ceph-mon[146817]: osdmap e63: 6 total, 6 up, 6 in 2024-04-12T05:44:26.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:26 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:44:26.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:26 smithi032 ceph-mon[146817]: osdmap e64: 6 total, 6 up, 6 in 2024-04-12T05:44:27.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:26 smithi067 ceph-mon[112979]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T05:44:27.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:26 smithi067 ceph-mon[112979]: osd.2 [v2:172.21.15.32:6818/1610966654,v1:172.21.15.32:6819/1610966654] boot 2024-04-12T05:44:27.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:26 smithi067 ceph-mon[112979]: osdmap e63: 6 total, 6 up, 6 in 2024-04-12T05:44:27.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:26 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:44:27.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:26 smithi067 ceph-mon[112979]: osdmap e64: 6 total, 6 up, 6 in 2024-04-12T05:44:28.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:27 smithi067 ceph-mon[112979]: pgmap v96: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 41/333 objects degraded (12.312%) 2024-04-12T05:44:28.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:27 smithi067 ceph-mon[112979]: Health check update: Degraded data redundancy: 41/333 objects degraded (12.312%), 12 pgs degraded (PG_DEGRADED) 2024-04-12T05:44:28.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:27 smithi032 ceph-mon[146817]: pgmap v96: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 41/333 objects degraded (12.312%) 2024-04-12T05:44:28.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:27 smithi032 ceph-mon[146817]: Health check update: Degraded data redundancy: 41/333 objects degraded (12.312%), 12 pgs degraded (PG_DEGRADED) 2024-04-12T05:44:29.035 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:28 smithi032 ceph-mon[146817]: pgmap v98: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 41/333 objects degraded (12.312%) 2024-04-12T05:44:29.035 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:28 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:29.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:28 smithi067 ceph-mon[112979]: pgmap v98: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 41/333 objects degraded (12.312%) 2024-04-12T05:44:29.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:28 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:30.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:30 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:30.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:30 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:30.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:30 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:44:30.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:30 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:44:30.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:30 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:30.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:30 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:44:30.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:30 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:44:30.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:30 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:44:30.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:30 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:44:30.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:30 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-12T05:44:30.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:30 smithi032 ceph-mon[146817]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-12T05:44:30.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:30 smithi032 ceph-mon[146817]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-12T05:44:30.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:30 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:30.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:30 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:30.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:30 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:44:30.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:30 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:44:30.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:30 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:30.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:30 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:44:30.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:30 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:44:30.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:30 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:44:30.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:30 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:44:30.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:30 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-12T05:44:30.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:30 smithi067 ceph-mon[112979]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-12T05:44:30.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:30 smithi067 ceph-mon[112979]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-12T05:44:31.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:31 smithi067 ceph-mon[112979]: pgmap v99: 65 pgs: 1 active+undersized+degraded, 2 active+undersized, 62 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s; 3/333 objects degraded (0.901%) 2024-04-12T05:44:31.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:31 smithi032 ceph-mon[146817]: pgmap v99: 65 pgs: 1 active+undersized+degraded, 2 active+undersized, 62 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s; 3/333 objects degraded (0.901%) 2024-04-12T05:44:32.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:32 smithi067 ceph-mon[112979]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/333 objects degraded (0.901%), 1 pg degraded) 2024-04-12T05:44:32.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:32 smithi067 ceph-mon[112979]: Cluster is now healthy 2024-04-12T05:44:32.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:32 smithi032 ceph-mon[146817]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/333 objects degraded (0.901%), 1 pg degraded) 2024-04-12T05:44:32.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:32 smithi032 ceph-mon[146817]: Cluster is now healthy 2024-04-12T05:44:33.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:33 smithi067 ceph-mon[112979]: pgmap v100: 65 pgs: 65 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-12T05:44:33.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:33 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:33.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:33 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:44:33.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:33 smithi032 ceph-mon[146817]: pgmap v100: 65 pgs: 65 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-12T05:44:33.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:33 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:33.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:33 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:44:35.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:35 smithi067 ceph-mon[112979]: pgmap v101: 65 pgs: 65 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 2 op/s 2024-04-12T05:44:35.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:35 smithi032 ceph-mon[146817]: pgmap v101: 65 pgs: 65 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 2 op/s 2024-04-12T05:44:37.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:37 smithi067 ceph-mon[112979]: pgmap v102: 65 pgs: 65 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 2 op/s 2024-04-12T05:44:37.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:37 smithi032 ceph-mon[146817]: pgmap v102: 65 pgs: 65 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 2 op/s 2024-04-12T05:44:39.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:38 smithi067 ceph-mon[112979]: pgmap v103: 65 pgs: 65 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-04-12T05:44:39.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:38 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:39.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:38 smithi032 ceph-mon[146817]: pgmap v103: 65 pgs: 65 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-04-12T05:44:39.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:38 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:41.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:40 smithi067 ceph-mon[112979]: pgmap v104: 65 pgs: 65 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 3 op/s 2024-04-12T05:44:41.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:40 smithi032 ceph-mon[146817]: pgmap v104: 65 pgs: 65 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 3 op/s 2024-04-12T05:44:43.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:42 smithi067 ceph-mon[112979]: pgmap v105: 65 pgs: 65 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:44:43.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:42 smithi032 ceph-mon[146817]: pgmap v105: 65 pgs: 65 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:44:45.140 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:44 smithi067 ceph-mon[112979]: pgmap v106: 65 pgs: 65 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:44:45.140 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:44 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-12T05:44:45.140 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:44 smithi067 ceph-mon[112979]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-12T05:44:45.140 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:44 smithi067 ceph-mon[112979]: Upgrade: osd.3 is safe to restart 2024-04-12T05:44:45.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:44 smithi032 ceph-mon[146817]: pgmap v106: 65 pgs: 65 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:44:45.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:44 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-12T05:44:45.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:44 smithi032 ceph-mon[146817]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-12T05:44:45.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:44 smithi032 ceph-mon[146817]: Upgrade: osd.3 is safe to restart 2024-04-12T05:44:47.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:46 smithi032 ceph-mon[146817]: Upgrade: Updating osd.3 2024-04-12T05:44:47.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:46 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:47.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:46 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-12T05:44:47.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:46 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:44:47.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:46 smithi032 ceph-mon[146817]: Deploying daemon osd.3 on smithi067 2024-04-12T05:44:47.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:46 smithi032 ceph-mon[146817]: pgmap v107: 65 pgs: 65 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:44:47.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:46 smithi067 ceph-mon[112979]: Upgrade: Updating osd.3 2024-04-12T05:44:47.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:46 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:47.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:46 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-12T05:44:47.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:46 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:44:47.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:46 smithi067 ceph-mon[112979]: Deploying daemon osd.3 on smithi067 2024-04-12T05:44:47.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:46 smithi067 ceph-mon[112979]: pgmap v107: 65 pgs: 65 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:44:48.153 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:47 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:44:48.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:47 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:44:48.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.470+0000 7f7ab6add700 1 -- 172.21.15.32:0/569608888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ab0102e10 msgr2=0x7f7ab01031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:48.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.470+0000 7f7ab6add700 1 --2- 172.21.15.32:0/569608888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ab0102e10 0x7f7ab01031e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4004940 tx=0x7f7aa4009e30 comp rx=0 tx=0).stop 2024-04-12T05:44:48.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.470+0000 7f7ab6add700 1 -- 172.21.15.32:0/569608888 shutdown_connections 2024-04-12T05:44:48.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.470+0000 7f7ab6add700 1 --2- 172.21.15.32:0/569608888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ab01037b0 0x7f7ab00fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:48.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.470+0000 7f7ab6add700 1 --2- 172.21.15.32:0/569608888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ab0102e10 0x7f7ab01031e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:48.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.470+0000 7f7ab6add700 1 -- 172.21.15.32:0/569608888 >> 172.21.15.32:0/569608888 conn(0x7f7ab00f89a0 msgr2=0x7f7ab00fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:48.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.470+0000 7f7ab6add700 1 -- 172.21.15.32:0/569608888 shutdown_connections 2024-04-12T05:44:48.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.470+0000 7f7ab6add700 1 -- 172.21.15.32:0/569608888 wait complete. 2024-04-12T05:44:48.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.471+0000 7f7ab6add700 1 Processor -- start 2024-04-12T05:44:48.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.471+0000 7f7ab6add700 1 -- start start 2024-04-12T05:44:48.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.472+0000 7f7ab6add700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ab0102e10 0x7f7ab0113a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:48.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.472+0000 7f7ab6add700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ab01037b0 0x7f7ab0113f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:48.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.472+0000 7f7ab6add700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab01145e0 con 0x7f7ab0102e10 2024-04-12T05:44:48.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.472+0000 7f7ab6add700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab010d9d0 con 0x7f7ab01037b0 2024-04-12T05:44:48.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.472+0000 7f7ab4879700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ab0102e10 0x7f7ab0113a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:48.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.472+0000 7f7ab4879700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ab0102e10 0x7f7ab0113a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38596/0 (socket says 172.21.15.32:38596) 2024-04-12T05:44:48.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.472+0000 7f7ab4879700 1 -- 172.21.15.32:0/2309787520 learned_addr learned my addr 172.21.15.32:0/2309787520 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:44:48.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.472+0000 7f7aaffff700 1 --2- 172.21.15.32:0/2309787520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ab01037b0 0x7f7ab0113f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:48.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.472+0000 7f7ab4879700 1 -- 172.21.15.32:0/2309787520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ab01037b0 msgr2=0x7f7ab0113f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:48.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.472+0000 7f7ab4879700 1 --2- 172.21.15.32:0/2309787520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ab01037b0 0x7f7ab0113f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:48.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.472+0000 7f7ab4879700 1 -- 172.21.15.32:0/2309787520 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa4005040 con 0x7f7ab0102e10 2024-04-12T05:44:48.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.472+0000 7f7ab4879700 1 --2- 172.21.15.32:0/2309787520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ab0102e10 0x7f7ab0113a50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4000c00 tx=0x7f7aa4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:48.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.472+0000 7f7aad7fa700 1 -- 172.21.15.32:0/2309787520 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7aa4021070 con 0x7f7ab0102e10 2024-04-12T05:44:48.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.472+0000 7f7ab6add700 1 -- 172.21.15.32:0/2309787520 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ab010dc50 con 0x7f7ab0102e10 2024-04-12T05:44:48.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.473+0000 7f7aad7fa700 1 -- 172.21.15.32:0/2309787520 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7aa40093b0 con 0x7f7ab0102e10 2024-04-12T05:44:48.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.473+0000 7f7aad7fa700 1 -- 172.21.15.32:0/2309787520 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7aa400e070 con 0x7f7ab0102e10 2024-04-12T05:44:48.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.473+0000 7f7ab6add700 1 -- 172.21.15.32:0/2309787520 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ab010e210 con 0x7f7ab0102e10 2024-04-12T05:44:48.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.473+0000 7f7ab6add700 1 -- 172.21.15.32:0/2309787520 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ab00fe560 con 0x7f7ab0102e10 2024-04-12T05:44:48.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.474+0000 7f7aad7fa700 1 -- 172.21.15.32:0/2309787520 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f7aa401c490 con 0x7f7ab0102e10 2024-04-12T05:44:48.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.474+0000 7f7aad7fa700 1 --2- 172.21.15.32:0/2309787520 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7aa006cc20 0x7f7aa006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:48.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.475+0000 7f7aad7fa700 1 -- 172.21.15.32:0/2309787520 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f7aa408d0c0 con 0x7f7ab0102e10 2024-04-12T05:44:48.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.475+0000 7f7aaffff700 1 --2- 172.21.15.32:0/2309787520 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7aa006cc20 0x7f7aa006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:48.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.475+0000 7f7aaffff700 1 --2- 172.21.15.32:0/2309787520 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7aa006cc20 0x7f7aa006f0d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7ab010eb90 tx=0x7f7a9c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:48.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.479+0000 7f7aad7fa700 1 -- 172.21.15.32:0/2309787520 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7aa405abe0 con 0x7f7ab0102e10 2024-04-12T05:44:48.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.701+0000 7f7ab6add700 1 -- 172.21.15.32:0/2309787520 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ab0109dd0 con 0x7f7aa006cc20 2024-04-12T05:44:48.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.702+0000 7f7aad7fa700 1 -- 172.21.15.32:0/2309787520 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f7ab0109dd0 con 0x7f7aa006cc20 2024-04-12T05:44:48.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.705+0000 7f7ab6add700 1 -- 172.21.15.32:0/2309787520 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7aa006cc20 msgr2=0x7f7aa006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:48.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.705+0000 7f7ab6add700 1 --2- 172.21.15.32:0/2309787520 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7aa006cc20 0x7f7aa006f0d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7ab010eb90 tx=0x7f7a9c00b040 comp rx=0 tx=0).stop 2024-04-12T05:44:48.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.705+0000 7f7ab6add700 1 -- 172.21.15.32:0/2309787520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ab0102e10 msgr2=0x7f7ab0113a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:48.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.705+0000 7f7ab6add700 1 --2- 172.21.15.32:0/2309787520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ab0102e10 0x7f7ab0113a50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4000c00 tx=0x7f7aa4009e30 comp rx=0 tx=0).stop 2024-04-12T05:44:48.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.705+0000 7f7ab6add700 1 -- 172.21.15.32:0/2309787520 shutdown_connections 2024-04-12T05:44:48.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.705+0000 7f7ab6add700 1 --2- 172.21.15.32:0/2309787520 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7aa006cc20 0x7f7aa006f0d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:48.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.705+0000 7f7ab6add700 1 --2- 172.21.15.32:0/2309787520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ab0102e10 0x7f7ab0113a50 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:48.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.705+0000 7f7ab6add700 1 --2- 172.21.15.32:0/2309787520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ab01037b0 0x7f7ab0113f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:48.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.705+0000 7f7ab6add700 1 -- 172.21.15.32:0/2309787520 >> 172.21.15.32:0/2309787520 conn(0x7f7ab00f89a0 msgr2=0x7f7ab00f9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:48.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.705+0000 7f7ab6add700 1 -- 172.21.15.32:0/2309787520 shutdown_connections 2024-04-12T05:44:48.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.705+0000 7f7ab6add700 1 -- 172.21.15.32:0/2309787520 wait complete. 2024-04-12T05:44:48.712 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:44:48.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.812+0000 7f2368c3e700 1 -- 172.21.15.32:0/3311867670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23640fb4d0 msgr2=0x7f23640fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:48.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.812+0000 7f2368c3e700 1 --2- 172.21.15.32:0/3311867670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23640fb4d0 0x7f23640fb940 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2358004940 tx=0x7f2358009e30 comp rx=0 tx=0).stop 2024-04-12T05:44:48.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.813+0000 7f2368c3e700 1 -- 172.21.15.32:0/3311867670 shutdown_connections 2024-04-12T05:44:48.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.813+0000 7f2368c3e700 1 --2- 172.21.15.32:0/3311867670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23640fb4d0 0x7f23640fb940 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:48.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.813+0000 7f2368c3e700 1 --2- 172.21.15.32:0/3311867670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2364102f50 0x7f2364103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:48.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.813+0000 7f2368c3e700 1 -- 172.21.15.32:0/3311867670 >> 172.21.15.32:0/3311867670 conn(0x7f23640714c0 msgr2=0x7f23640718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:48.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.813+0000 7f2368c3e700 1 -- 172.21.15.32:0/3311867670 shutdown_connections 2024-04-12T05:44:48.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.813+0000 7f2368c3e700 1 -- 172.21.15.32:0/3311867670 wait complete. 2024-04-12T05:44:48.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.813+0000 7f2368c3e700 1 Processor -- start 2024-04-12T05:44:48.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.813+0000 7f2368c3e700 1 -- start start 2024-04-12T05:44:48.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.814+0000 7f2368c3e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23640fb4d0 0x7f2364113920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:48.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.814+0000 7f2368c3e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2364102f50 0x7f2364113e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:48.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.814+0000 7f2368c3e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23641144b0 con 0x7f2364102f50 2024-04-12T05:44:48.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.814+0000 7f2368c3e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236410dab0 con 0x7f23640fb4d0 2024-04-12T05:44:48.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.814+0000 7f2361d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2364102f50 0x7f2364113e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:48.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.814+0000 7f2361d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2364102f50 0x7f2364113e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38612/0 (socket says 172.21.15.32:38612) 2024-04-12T05:44:48.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.814+0000 7f2361d9b700 1 -- 172.21.15.32:0/647704175 learned_addr learned my addr 172.21.15.32:0/647704175 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:44:48.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.814+0000 7f236259c700 1 --2- 172.21.15.32:0/647704175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23640fb4d0 0x7f2364113920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:48.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.814+0000 7f2361d9b700 1 -- 172.21.15.32:0/647704175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23640fb4d0 msgr2=0x7f2364113920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:48.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.814+0000 7f2361d9b700 1 --2- 172.21.15.32:0/647704175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23640fb4d0 0x7f2364113920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:48.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.814+0000 7f2361d9b700 1 -- 172.21.15.32:0/647704175 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2358005040 con 0x7f2364102f50 2024-04-12T05:44:48.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.814+0000 7f2361d9b700 1 --2- 172.21.15.32:0/647704175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2364102f50 0x7f2364113e60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2358000c00 tx=0x7f2358009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:48.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.815+0000 7f2352ffd700 1 -- 172.21.15.32:0/647704175 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2358021070 con 0x7f2364102f50 2024-04-12T05:44:48.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.815+0000 7f2368c3e700 1 -- 172.21.15.32:0/647704175 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f236410dd30 con 0x7f2364102f50 2024-04-12T05:44:48.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.815+0000 7f2352ffd700 1 -- 172.21.15.32:0/647704175 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f23580093b0 con 0x7f2364102f50 2024-04-12T05:44:48.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.815+0000 7f2352ffd700 1 -- 172.21.15.32:0/647704175 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f235800e070 con 0x7f2364102f50 2024-04-12T05:44:48.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.815+0000 7f2368c3e700 1 -- 172.21.15.32:0/647704175 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f236410e270 con 0x7f2364102f50 2024-04-12T05:44:48.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.816+0000 7f2368c3e700 1 -- 172.21.15.32:0/647704175 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2364105330 con 0x7f2364102f50 2024-04-12T05:44:48.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.816+0000 7f2352ffd700 1 -- 172.21.15.32:0/647704175 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f235801c490 con 0x7f2364102f50 2024-04-12T05:44:48.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.817+0000 7f2352ffd700 1 --2- 172.21.15.32:0/647704175 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f234c06cc20 0x7f234c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:48.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.817+0000 7f2352ffd700 1 -- 172.21.15.32:0/647704175 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f235808d0c0 con 0x7f2364102f50 2024-04-12T05:44:48.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.817+0000 7f236259c700 1 --2- 172.21.15.32:0/647704175 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f234c06cc20 0x7f234c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:48.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.817+0000 7f236259c700 1 --2- 172.21.15.32:0/647704175 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f234c06cc20 0x7f234c06f0d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2354009b80 tx=0x7f235400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:48.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:48.821+0000 7f2352ffd700 1 -- 172.21.15.32:0/647704175 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f235805abe0 con 0x7f2364102f50 2024-04-12T05:44:49.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.044+0000 7f2368c3e700 1 -- 172.21.15.32:0/647704175 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f236410ef70 con 0x7f234c06cc20 2024-04-12T05:44:49.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.045+0000 7f2352ffd700 1 -- 172.21.15.32:0/647704175 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f236410ef70 con 0x7f234c06cc20 2024-04-12T05:44:49.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.047+0000 7f2368c3e700 1 -- 172.21.15.32:0/647704175 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f234c06cc20 msgr2=0x7f234c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:49.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.047+0000 7f2368c3e700 1 --2- 172.21.15.32:0/647704175 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f234c06cc20 0x7f234c06f0d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2354009b80 tx=0x7f235400b040 comp rx=0 tx=0).stop 2024-04-12T05:44:49.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.047+0000 7f2368c3e700 1 -- 172.21.15.32:0/647704175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2364102f50 msgr2=0x7f2364113e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:49.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.047+0000 7f2368c3e700 1 --2- 172.21.15.32:0/647704175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2364102f50 0x7f2364113e60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2358000c00 tx=0x7f2358009e30 comp rx=0 tx=0).stop 2024-04-12T05:44:49.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.048+0000 7f2368c3e700 1 -- 172.21.15.32:0/647704175 shutdown_connections 2024-04-12T05:44:49.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.048+0000 7f2368c3e700 1 --2- 172.21.15.32:0/647704175 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f234c06cc20 0x7f234c06f0d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:49.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.048+0000 7f2368c3e700 1 --2- 172.21.15.32:0/647704175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23640fb4d0 0x7f2364113920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:49.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.048+0000 7f2368c3e700 1 --2- 172.21.15.32:0/647704175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2364102f50 0x7f2364113e60 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:49.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.048+0000 7f2368c3e700 1 -- 172.21.15.32:0/647704175 >> 172.21.15.32:0/647704175 conn(0x7f23640714c0 msgr2=0x7f23640fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:49.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.048+0000 7f2368c3e700 1 -- 172.21.15.32:0/647704175 shutdown_connections 2024-04-12T05:44:49.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.048+0000 7f2368c3e700 1 -- 172.21.15.32:0/647704175 wait complete. 2024-04-12T05:44:49.089 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:48 smithi067 ceph-mon[112979]: pgmap v108: 65 pgs: 65 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:44:49.089 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:48 smithi067 ceph-mon[112979]: from='client.34280 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:49.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.151+0000 7faf3e567700 1 -- 172.21.15.32:0/3107988721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf380fc5b0 msgr2=0x7faf380fca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:49.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.151+0000 7faf3e567700 1 --2- 172.21.15.32:0/3107988721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf380fc5b0 0x7faf380fca20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7faf28004940 tx=0x7faf28009e30 comp rx=0 tx=0).stop 2024-04-12T05:44:49.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.151+0000 7faf3e567700 1 -- 172.21.15.32:0/3107988721 shutdown_connections 2024-04-12T05:44:49.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.151+0000 7faf3e567700 1 --2- 172.21.15.32:0/3107988721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf380fc5b0 0x7faf380fca20 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:49.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.151+0000 7faf3e567700 1 --2- 172.21.15.32:0/3107988721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf38101d70 0x7faf38102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:49.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.151+0000 7faf3e567700 1 -- 172.21.15.32:0/3107988721 >> 172.21.15.32:0/3107988721 conn(0x7faf380714c0 msgr2=0x7faf380718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:49.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.151+0000 7faf3e567700 1 -- 172.21.15.32:0/3107988721 shutdown_connections 2024-04-12T05:44:49.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.151+0000 7faf3e567700 1 -- 172.21.15.32:0/3107988721 wait complete. 2024-04-12T05:44:49.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.151+0000 7faf3e567700 1 Processor -- start 2024-04-12T05:44:49.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.152+0000 7faf3e567700 1 -- start start 2024-04-12T05:44:49.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.152+0000 7faf3e567700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf380fc5b0 0x7faf38102b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:49.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.152+0000 7faf3e567700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf38101d70 0x7faf381030a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:49.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.152+0000 7faf3e567700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf38103630 con 0x7faf38101d70 2024-04-12T05:44:49.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.152+0000 7faf3e567700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf38103770 con 0x7faf380fc5b0 2024-04-12T05:44:49.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.152+0000 7faf377fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf38101d70 0x7faf381030a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:49.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.152+0000 7faf377fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf38101d70 0x7faf381030a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38626/0 (socket says 172.21.15.32:38626) 2024-04-12T05:44:49.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.152+0000 7faf377fe700 1 -- 172.21.15.32:0/1426535207 learned_addr learned my addr 172.21.15.32:0/1426535207 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:44:49.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.152+0000 7faf37fff700 1 --2- 172.21.15.32:0/1426535207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf380fc5b0 0x7faf38102b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:49.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.152+0000 7faf377fe700 1 -- 172.21.15.32:0/1426535207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf380fc5b0 msgr2=0x7faf38102b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:49.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.152+0000 7faf377fe700 1 --2- 172.21.15.32:0/1426535207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf380fc5b0 0x7faf38102b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:49.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.152+0000 7faf377fe700 1 -- 172.21.15.32:0/1426535207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf28005040 con 0x7faf38101d70 2024-04-12T05:44:49.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.152+0000 7faf377fe700 1 --2- 172.21.15.32:0/1426535207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf38101d70 0x7faf381030a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7faf2800b3b0 tx=0x7faf28009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:49.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.152+0000 7faf34ff9700 1 -- 172.21.15.32:0/1426535207 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf2800e040 con 0x7faf38101d70 2024-04-12T05:44:49.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.153+0000 7faf34ff9700 1 -- 172.21.15.32:0/1426535207 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faf28013420 con 0x7faf38101d70 2024-04-12T05:44:49.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.153+0000 7faf3e567700 1 -- 172.21.15.32:0/1426535207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf381039c0 con 0x7faf38101d70 2024-04-12T05:44:49.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.153+0000 7faf34ff9700 1 -- 172.21.15.32:0/1426535207 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf2801c610 con 0x7faf38101d70 2024-04-12T05:44:49.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.153+0000 7faf3e567700 1 -- 172.21.15.32:0/1426535207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf3811a1c0 con 0x7faf38101d70 2024-04-12T05:44:49.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.154+0000 7faf3e567700 1 -- 172.21.15.32:0/1426535207 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf380fd5c0 con 0x7faf38101d70 2024-04-12T05:44:49.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.154+0000 7faf34ff9700 1 -- 172.21.15.32:0/1426535207 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7faf28013a00 con 0x7faf38101d70 2024-04-12T05:44:49.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.154+0000 7faf34ff9700 1 --2- 172.21.15.32:0/1426535207 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7faf2406cc20 0x7faf2406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:49.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.155+0000 7faf34ff9700 1 -- 172.21.15.32:0/1426535207 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6410+0+0 (secure 0 0 0) 0x7faf2808d160 con 0x7faf38101d70 2024-04-12T05:44:49.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.155+0000 7faf37fff700 1 --2- 172.21.15.32:0/1426535207 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7faf2406cc20 0x7faf2406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:49.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.155+0000 7faf37fff700 1 --2- 172.21.15.32:0/1426535207 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7faf2406cc20 0x7faf2406f0d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7faf38064f70 tx=0x7faf2000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:49.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.159+0000 7faf34ff9700 1 -- 172.21.15.32:0/1426535207 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7faf280571f0 con 0x7faf38101d70 2024-04-12T05:44:49.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:48 smithi032 ceph-mon[146817]: pgmap v108: 65 pgs: 65 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:44:49.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:48 smithi032 ceph-mon[146817]: from='client.34280 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:49.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.371+0000 7faf3e567700 1 -- 172.21.15.32:0/1426535207 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7faf3804bed0 con 0x7faf2406cc20 2024-04-12T05:44:49.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.380+0000 7faf34ff9700 1 -- 172.21.15.32:0/1426535207 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7faf3804bed0 con 0x7faf2406cc20 2024-04-12T05:44:49.378 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:44:49.378 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (4m) 24s ago 16m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:44:49.378 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (2m) 24s ago 16m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:44:49.378 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (2m) 118s ago 15m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:44:49.378 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (4m) 24s ago 15m 84.7M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:44:49.378 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (10m) 24s ago 10m 64.8M - 17.2.7-827-gd3b5060a 21de9b7369ad e7ada8e5259c 2024-04-12T05:44:49.378 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (10m) 24s ago 10m 115M - 17.2.7-827-gd3b5060a 21de9b7369ad 55124da9afdb 2024-04-12T05:44:49.378 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (10m) 118s ago 10m 62.6M - 17.2.7-827-gd3b5060a 21de9b7369ad 0c295d31a4a6 2024-04-12T05:44:49.378 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (10m) 118s ago 10m 120M - 17.2.7-827-gd3b5060a 21de9b7369ad 6eecd7deaac2 2024-04-12T05:44:49.378 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (6m) 24s ago 17m 499M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:44:49.379 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (5m) 118s ago 15m 447M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:44:49.379 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (3m) 24s ago 17m 46.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:44:49.379 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (2m) 118s ago 15m 33.0M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:44:49.379 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (5m) 24s ago 16m 16.6M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:44:49.379 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (5m) 118s ago 15m 18.0M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:44:49.379 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (101s) 24s ago 14m 86.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T05:44:49.379 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (66s) 24s ago 13m 94.5M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T05:44:49.379 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (30s) 24s ago 13m 11.2M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T05:44:49.379 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (12m) 118s ago 12m 258M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad 61b851235351 2024-04-12T05:44:49.379 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (12m) 118s ago 12m 230M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad 2c84d9047e9c 2024-04-12T05:44:49.379 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (11m) 118s ago 11m 220M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad e54839bb6055 2024-04-12T05:44:49.379 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (4m) 24s ago 15m 63.5M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:44:49.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.383+0000 7faf3e567700 1 -- 172.21.15.32:0/1426535207 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7faf2406cc20 msgr2=0x7faf2406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:49.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.383+0000 7faf3e567700 1 --2- 172.21.15.32:0/1426535207 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7faf2406cc20 0x7faf2406f0d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7faf38064f70 tx=0x7faf2000b040 comp rx=0 tx=0).stop 2024-04-12T05:44:49.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.383+0000 7faf3e567700 1 -- 172.21.15.32:0/1426535207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf38101d70 msgr2=0x7faf381030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:49.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.383+0000 7faf3e567700 1 --2- 172.21.15.32:0/1426535207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf38101d70 0x7faf381030a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7faf2800b3b0 tx=0x7faf28009e30 comp rx=0 tx=0).stop 2024-04-12T05:44:49.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.383+0000 7faf3e567700 1 -- 172.21.15.32:0/1426535207 shutdown_connections 2024-04-12T05:44:49.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.383+0000 7faf3e567700 1 --2- 172.21.15.32:0/1426535207 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7faf2406cc20 0x7faf2406f0d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:49.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.383+0000 7faf3e567700 1 --2- 172.21.15.32:0/1426535207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf380fc5b0 0x7faf38102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:49.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.383+0000 7faf3e567700 1 --2- 172.21.15.32:0/1426535207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf38101d70 0x7faf381030a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:49.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.383+0000 7faf3e567700 1 -- 172.21.15.32:0/1426535207 >> 172.21.15.32:0/1426535207 conn(0x7faf380714c0 msgr2=0x7faf38072840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:49.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.383+0000 7faf3e567700 1 -- 172.21.15.32:0/1426535207 shutdown_connections 2024-04-12T05:44:49.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.383+0000 7faf3e567700 1 -- 172.21.15.32:0/1426535207 wait complete. 2024-04-12T05:44:49.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.485+0000 7f3c5ff35700 1 -- 172.21.15.32:0/647390668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c580fb4a0 msgr2=0x7f3c580fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:49.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.485+0000 7f3c5ff35700 1 --2- 172.21.15.32:0/647390668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c580fb4a0 0x7f3c580fb910 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3c54004940 tx=0x7f3c54009e30 comp rx=0 tx=0).stop 2024-04-12T05:44:49.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.485+0000 7f3c5ff35700 1 -- 172.21.15.32:0/647390668 shutdown_connections 2024-04-12T05:44:49.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.485+0000 7f3c5ff35700 1 --2- 172.21.15.32:0/647390668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c580fb4a0 0x7f3c580fb910 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:49.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.485+0000 7f3c5ff35700 1 --2- 172.21.15.32:0/647390668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c58102e70 0x7f3c58103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:49.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.485+0000 7f3c5ff35700 1 -- 172.21.15.32:0/647390668 >> 172.21.15.32:0/647390668 conn(0x7f3c580714c0 msgr2=0x7f3c580718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:49.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.485+0000 7f3c5ff35700 1 -- 172.21.15.32:0/647390668 shutdown_connections 2024-04-12T05:44:49.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.485+0000 7f3c5ff35700 1 -- 172.21.15.32:0/647390668 wait complete. 2024-04-12T05:44:49.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.486+0000 7f3c5ff35700 1 Processor -- start 2024-04-12T05:44:49.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.486+0000 7f3c5ff35700 1 -- start start 2024-04-12T05:44:49.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.486+0000 7f3c5ff35700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c580fb4a0 0x7f3c580fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:49.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.486+0000 7f3c5ff35700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c58102e70 0x7f3c580fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:49.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.486+0000 7f3c5ff35700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c580fd170 con 0x7f3c58102e70 2024-04-12T05:44:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.486+0000 7f3c5ff35700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c580fd2e0 con 0x7f3c580fb4a0 2024-04-12T05:44:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.486+0000 7f3c5d4d0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c58102e70 0x7f3c580fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.486+0000 7f3c5d4d0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c58102e70 0x7f3c580fcbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38644/0 (socket says 172.21.15.32:38644) 2024-04-12T05:44:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.486+0000 7f3c5d4d0700 1 -- 172.21.15.32:0/3798870972 learned_addr learned my addr 172.21.15.32:0/3798870972 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:44:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.486+0000 7f3c5dcd1700 1 --2- 172.21.15.32:0/3798870972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c580fb4a0 0x7f3c580fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.486+0000 7f3c5d4d0700 1 -- 172.21.15.32:0/3798870972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c580fb4a0 msgr2=0x7f3c580fc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.486+0000 7f3c5d4d0700 1 --2- 172.21.15.32:0/3798870972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c580fb4a0 0x7f3c580fc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.486+0000 7f3c5d4d0700 1 -- 172.21.15.32:0/3798870972 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c54005040 con 0x7f3c58102e70 2024-04-12T05:44:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.486+0000 7f3c5d4d0700 1 --2- 172.21.15.32:0/3798870972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c58102e70 0x7f3c580fcbc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3c5400b3b0 tx=0x7f3c54009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.487+0000 7f3c4a7fc700 1 -- 172.21.15.32:0/3798870972 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c5400e040 con 0x7f3c58102e70 2024-04-12T05:44:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.487+0000 7f3c5ff35700 1 -- 172.21.15.32:0/3798870972 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c58119d70 con 0x7f3c58102e70 2024-04-12T05:44:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.487+0000 7f3c4a7fc700 1 -- 172.21.15.32:0/3798870972 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c54013420 con 0x7f3c58102e70 2024-04-12T05:44:49.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.487+0000 7f3c4a7fc700 1 -- 172.21.15.32:0/3798870972 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c5401c610 con 0x7f3c58102e70 2024-04-12T05:44:49.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.487+0000 7f3c5ff35700 1 -- 172.21.15.32:0/3798870972 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c5811a180 con 0x7f3c58102e70 2024-04-12T05:44:49.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.488+0000 7f3c5ff35700 1 -- 172.21.15.32:0/3798870972 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c58100110 con 0x7f3c58102e70 2024-04-12T05:44:49.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.488+0000 7f3c4a7fc700 1 -- 172.21.15.32:0/3798870972 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3c54013a00 con 0x7f3c58102e70 2024-04-12T05:44:49.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.488+0000 7f3c4a7fc700 1 --2- 172.21.15.32:0/3798870972 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3c4406cc20 0x7f3c4406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:49.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.489+0000 7f3c4a7fc700 1 -- 172.21.15.32:0/3798870972 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f3c5408d190 con 0x7f3c58102e70 2024-04-12T05:44:49.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.489+0000 7f3c5dcd1700 1 --2- 172.21.15.32:0/3798870972 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3c4406cc20 0x7f3c4406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:49.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.489+0000 7f3c5dcd1700 1 --2- 172.21.15.32:0/3798870972 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3c4406cc20 0x7f3c4406f0d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c004f40 tx=0x7f3c4c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:49.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.493+0000 7f3c4a7fc700 1 -- 172.21.15.32:0/3798870972 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3c54057220 con 0x7f3c58102e70 2024-04-12T05:44:49.518 INFO:journalctl@ceph.osd.3.smithi067.stdout:Apr 12 05:44:49 smithi067 systemd[1]: Stopping Ceph osd.3 for 37b58ce2-f88d-11ee-bc8c-c7b262605968... 2024-04-12T05:44:49.518 INFO:journalctl@ceph.osd.3.smithi067.stdout:Apr 12 05:44:49 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-3[55347]: 2024-04-12T05:44:49.307+0000 7f4f93de3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T05:44:49.518 INFO:journalctl@ceph.osd.3.smithi067.stdout:Apr 12 05:44:49 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-3[55347]: 2024-04-12T05:44:49.307+0000 7f4f93de3700 -1 osd.3 64 *** Got signal Terminated *** 2024-04-12T05:44:49.519 INFO:journalctl@ceph.osd.3.smithi067.stdout:Apr 12 05:44:49 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-3[55347]: 2024-04-12T05:44:49.307+0000 7f4f93de3700 -1 osd.3 64 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T05:44:49.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.785+0000 7f3c5ff35700 1 -- 172.21.15.32:0/3798870972 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3c58109e00 con 0x7f3c58102e70 2024-04-12T05:44:49.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.785+0000 7f3c4a7fc700 1 -- 172.21.15.32:0/3798870972 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f3c5405a840 con 0x7f3c58102e70 2024-04-12T05:44:49.782 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:44:49.782 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:44:49.782 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:44:49.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:44:49.782 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:44:49.782 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:44:49.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:44:49.783 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:44:49.783 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 3, 2024-04-12T05:44:49.783 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:44:49.783 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:44:49.783 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:44:49.783 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 4 2024-04-12T05:44:49.783 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:44:49.783 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:44:49.783 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 7, 2024-04-12T05:44:49.783 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 7 2024-04-12T05:44:49.783 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:44:49.783 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:44:49.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.787+0000 7f3c5ff35700 1 -- 172.21.15.32:0/3798870972 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3c4406cc20 msgr2=0x7f3c4406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:49.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.787+0000 7f3c5ff35700 1 --2- 172.21.15.32:0/3798870972 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3c4406cc20 0x7f3c4406f0d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c004f40 tx=0x7f3c4c00b040 comp rx=0 tx=0).stop 2024-04-12T05:44:49.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.787+0000 7f3c5ff35700 1 -- 172.21.15.32:0/3798870972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c58102e70 msgr2=0x7f3c580fcbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:49.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.787+0000 7f3c5ff35700 1 --2- 172.21.15.32:0/3798870972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c58102e70 0x7f3c580fcbc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3c5400b3b0 tx=0x7f3c54009e30 comp rx=0 tx=0).stop 2024-04-12T05:44:49.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.787+0000 7f3c5ff35700 1 -- 172.21.15.32:0/3798870972 shutdown_connections 2024-04-12T05:44:49.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.787+0000 7f3c5ff35700 1 --2- 172.21.15.32:0/3798870972 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3c4406cc20 0x7f3c4406f0d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:49.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.787+0000 7f3c5ff35700 1 --2- 172.21.15.32:0/3798870972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c580fb4a0 0x7f3c580fc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:49.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.787+0000 7f3c5ff35700 1 --2- 172.21.15.32:0/3798870972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c58102e70 0x7f3c580fcbc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:49.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.787+0000 7f3c5ff35700 1 -- 172.21.15.32:0/3798870972 >> 172.21.15.32:0/3798870972 conn(0x7f3c580714c0 msgr2=0x7f3c580fa910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:49.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.787+0000 7f3c5ff35700 1 -- 172.21.15.32:0/3798870972 shutdown_connections 2024-04-12T05:44:49.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.787+0000 7f3c5ff35700 1 -- 172.21.15.32:0/3798870972 wait complete. 2024-04-12T05:44:49.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.891+0000 7f84ca11d700 1 -- 172.21.15.32:0/1058980264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84c4101d70 msgr2=0x7f84c4102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:49.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.891+0000 7f84ca11d700 1 --2- 172.21.15.32:0/1058980264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84c4101d70 0x7f84c4102140 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f84bc009ce0 tx=0x7f84bc004d10 comp rx=0 tx=0).stop 2024-04-12T05:44:49.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.891+0000 7f84ca11d700 1 -- 172.21.15.32:0/1058980264 shutdown_connections 2024-04-12T05:44:49.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.891+0000 7f84ca11d700 1 --2- 172.21.15.32:0/1058980264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84c4102710 0x7f84c40fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:49.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.891+0000 7f84ca11d700 1 --2- 172.21.15.32:0/1058980264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84c4101d70 0x7f84c4102140 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:49.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.891+0000 7f84ca11d700 1 -- 172.21.15.32:0/1058980264 >> 172.21.15.32:0/1058980264 conn(0x7f84c4072bc0 msgr2=0x7f84c4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:49.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.891+0000 7f84ca11d700 1 -- 172.21.15.32:0/1058980264 shutdown_connections 2024-04-12T05:44:49.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.891+0000 7f84ca11d700 1 -- 172.21.15.32:0/1058980264 wait complete. 2024-04-12T05:44:49.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.892+0000 7f84ca11d700 1 Processor -- start 2024-04-12T05:44:49.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.892+0000 7f84ca11d700 1 -- start start 2024-04-12T05:44:49.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.892+0000 7f84ca11d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84c4101d70 0x7f84c4112a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:49.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.892+0000 7f84ca11d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84c4102710 0x7f84c4112f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:49.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.892+0000 7f84ca11d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c41135b0 con 0x7f84c4102710 2024-04-12T05:44:49.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.892+0000 7f84ca11d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c410c9a0 con 0x7f84c4101d70 2024-04-12T05:44:49.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.893+0000 7f84c2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84c4102710 0x7f84c4112f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:49.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.893+0000 7f84c2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84c4102710 0x7f84c4112f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38658/0 (socket says 172.21.15.32:38658) 2024-04-12T05:44:49.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.893+0000 7f84c2ffd700 1 -- 172.21.15.32:0/668050775 learned_addr learned my addr 172.21.15.32:0/668050775 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:44:49.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.893+0000 7f84c37fe700 1 --2- 172.21.15.32:0/668050775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84c4101d70 0x7f84c4112a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:49.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.893+0000 7f84c2ffd700 1 -- 172.21.15.32:0/668050775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84c4101d70 msgr2=0x7f84c4112a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:49.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.893+0000 7f84c2ffd700 1 --2- 172.21.15.32:0/668050775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84c4101d70 0x7f84c4112a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:49.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.893+0000 7f84c2ffd700 1 -- 172.21.15.32:0/668050775 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84bc009940 con 0x7f84c4102710 2024-04-12T05:44:49.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.893+0000 7f84c2ffd700 1 --2- 172.21.15.32:0/668050775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84c4102710 0x7f84c4112f60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f84ac00eb40 tx=0x7f84ac00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:49.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.893+0000 7f84bbfff700 1 -- 172.21.15.32:0/668050775 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84ac007cc0 con 0x7f84c4102710 2024-04-12T05:44:49.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.893+0000 7f84ca11d700 1 -- 172.21.15.32:0/668050775 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c410cc20 con 0x7f84c4102710 2024-04-12T05:44:49.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.893+0000 7f84bbfff700 1 -- 172.21.15.32:0/668050775 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f84ac00f040 con 0x7f84c4102710 2024-04-12T05:44:49.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.893+0000 7f84bbfff700 1 -- 172.21.15.32:0/668050775 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84ac00c8f0 con 0x7f84c4102710 2024-04-12T05:44:49.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.893+0000 7f84ca11d700 1 -- 172.21.15.32:0/668050775 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c410d1e0 con 0x7f84c4102710 2024-04-12T05:44:49.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.894+0000 7f84ca11d700 1 -- 172.21.15.32:0/668050775 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84c40ff080 con 0x7f84c4102710 2024-04-12T05:44:49.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.895+0000 7f84bbfff700 1 -- 172.21.15.32:0/668050775 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f84ac00cb60 con 0x7f84c4102710 2024-04-12T05:44:49.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.895+0000 7f84bbfff700 1 --2- 172.21.15.32:0/668050775 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f84b006cc30 0x7f84b006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:49.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.895+0000 7f84bbfff700 1 -- 172.21.15.32:0/668050775 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f84ac08ba90 con 0x7f84c4102710 2024-04-12T05:44:49.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.895+0000 7f84c37fe700 1 --2- 172.21.15.32:0/668050775 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f84b006cc30 0x7f84b006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:49.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.896+0000 7f84c37fe700 1 --2- 172.21.15.32:0/668050775 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f84b006cc30 0x7f84b006f0e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f84bc00b460 tx=0x7f84bc004d10 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:49.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:49.900+0000 7f84bbfff700 1 -- 172.21.15.32:0/668050775 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f84ac0595e0 con 0x7f84c4102710 2024-04-12T05:44:50.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.157+0000 7f84ca11d700 1 -- 172.21.15.32:0/668050775 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f84c40fd4c0 con 0x7f84c4102710 2024-04-12T05:44:50.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.157+0000 7f84bbfff700 1 -- 172.21.15.32:0/668050775 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1864 (secure 0 0 0) 0x7f84ac059170 con 0x7f84c4102710 2024-04-12T05:44:50.159 INFO:teuthology.orchestra.run.smithi032.stdout:e22 2024-04-12T05:44:50.159 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:44:50.159 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:44:50.159 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:44:50.159 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:44:50.159 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:44:50.159 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:44:50.159 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 22 2024-04-12T05:44:50.159 INFO:teuthology.orchestra.run.smithi032.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-12T05:44:50.159 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:44:50.160 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:41:05.358509+0000 2024-04-12T05:44:50.160 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:44:50.160 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:44:50.160 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:44:50.160 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:44:50.160 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:44:50.160 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:44:50.160 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:44:50.160 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:44:50.160 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 46 2024-04-12T05:44:50.160 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:44:50.160 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:44:50.160 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:44:50.160 INFO:teuthology.orchestra.run.smithi032.stdout:up {0=14480,1=24281} 2024-04-12T05:44:50.160 INFO:teuthology.orchestra.run.smithi032.stdout:failed 2024-04-12T05:44:50.161 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:44:50.161 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:44:50.161 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:44:50.161 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:44:50.161 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:44:50.161 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:44:50.161 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:44:50.161 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:44:50.161 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{0:14480} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.32:6828/387754149,v1:172.21.15.32:6829/387754149] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:44:50.161 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{0:14490} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3678031004,v1:172.21.15.32:6827/3678031004] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:44:50.161 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{1:24281} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.67:6824/1291914599,v1:172.21.15.67:6825/1291914599] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:44:50.161 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{1:24295} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.67:6826/873395842,v1:172.21.15.67:6827/873395842] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:44:50.161 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:44:50.161 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:44:50.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.160+0000 7f84ca11d700 1 -- 172.21.15.32:0/668050775 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f84b006cc30 msgr2=0x7f84b006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:50.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.160+0000 7f84ca11d700 1 --2- 172.21.15.32:0/668050775 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f84b006cc30 0x7f84b006f0e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f84bc00b460 tx=0x7f84bc004d10 comp rx=0 tx=0).stop 2024-04-12T05:44:50.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.160+0000 7f84ca11d700 1 -- 172.21.15.32:0/668050775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84c4102710 msgr2=0x7f84c4112f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:50.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.160+0000 7f84ca11d700 1 --2- 172.21.15.32:0/668050775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84c4102710 0x7f84c4112f60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f84ac00eb40 tx=0x7f84ac00b040 comp rx=0 tx=0).stop 2024-04-12T05:44:50.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.160+0000 7f84ca11d700 1 -- 172.21.15.32:0/668050775 shutdown_connections 2024-04-12T05:44:50.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.160+0000 7f84ca11d700 1 --2- 172.21.15.32:0/668050775 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f84b006cc30 0x7f84b006f0e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:50.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.160+0000 7f84ca11d700 1 --2- 172.21.15.32:0/668050775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84c4101d70 0x7f84c4112a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:50.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.160+0000 7f84ca11d700 1 --2- 172.21.15.32:0/668050775 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84c4102710 0x7f84c4112f60 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:50.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.160+0000 7f84ca11d700 1 -- 172.21.15.32:0/668050775 >> 172.21.15.32:0/668050775 conn(0x7f84c4072bc0 msgr2=0x7f84c40fb100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:50.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.160+0000 7f84ca11d700 1 -- 172.21.15.32:0/668050775 shutdown_connections 2024-04-12T05:44:50.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.160+0000 7f84ca11d700 1 -- 172.21.15.32:0/668050775 wait complete. 2024-04-12T05:44:50.163 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 22 2024-04-12T05:44:50.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.264+0000 7fa687143700 1 -- 172.21.15.32:0/3850500877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa680102e00 msgr2=0x7fa6801031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:50.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.264+0000 7fa687143700 1 --2- 172.21.15.32:0/3850500877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa680102e00 0x7fa6801031d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa674004940 tx=0x7fa674009e30 comp rx=0 tx=0).stop 2024-04-12T05:44:50.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.264+0000 7fa687143700 1 -- 172.21.15.32:0/3850500877 shutdown_connections 2024-04-12T05:44:50.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.264+0000 7fa687143700 1 --2- 172.21.15.32:0/3850500877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6801037a0 0x7fa6800fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:50.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.264+0000 7fa687143700 1 --2- 172.21.15.32:0/3850500877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa680102e00 0x7fa6801031d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:50.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.264+0000 7fa687143700 1 -- 172.21.15.32:0/3850500877 >> 172.21.15.32:0/3850500877 conn(0x7fa6800f89b0 msgr2=0x7fa6800fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:50.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.264+0000 7fa687143700 1 -- 172.21.15.32:0/3850500877 shutdown_connections 2024-04-12T05:44:50.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.264+0000 7fa687143700 1 -- 172.21.15.32:0/3850500877 wait complete. 2024-04-12T05:44:50.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.265+0000 7fa687143700 1 Processor -- start 2024-04-12T05:44:50.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.265+0000 7fa687143700 1 -- start start 2024-04-12T05:44:50.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.265+0000 7fa687143700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa680102e00 0x7fa680113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:50.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.265+0000 7fa687143700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6801037a0 0x7fa680114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:50.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.265+0000 7fa687143700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6801146b0 con 0x7fa6801037a0 2024-04-12T05:44:50.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.265+0000 7fa687143700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa68010daa0 con 0x7fa680102e00 2024-04-12T05:44:50.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.265+0000 7fa67ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6801037a0 0x7fa680114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:50.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.265+0000 7fa67ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6801037a0 0x7fa680114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38674/0 (socket says 172.21.15.32:38674) 2024-04-12T05:44:50.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.265+0000 7fa67ffff700 1 -- 172.21.15.32:0/524439287 learned_addr learned my addr 172.21.15.32:0/524439287 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:44:50.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.265+0000 7fa684edf700 1 --2- 172.21.15.32:0/524439287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa680102e00 0x7fa680113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:50.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.266+0000 7fa67ffff700 1 -- 172.21.15.32:0/524439287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa680102e00 msgr2=0x7fa680113b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:50.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.266+0000 7fa67ffff700 1 --2- 172.21.15.32:0/524439287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa680102e00 0x7fa680113b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:50.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.266+0000 7fa67ffff700 1 -- 172.21.15.32:0/524439287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa674005040 con 0x7fa6801037a0 2024-04-12T05:44:50.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.266+0000 7fa684edf700 1 --2- 172.21.15.32:0/524439287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa680102e00 0x7fa680113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:44:50.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.266+0000 7fa67ffff700 1 --2- 172.21.15.32:0/524439287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6801037a0 0x7fa680114060 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa66c00c800 tx=0x7fa66c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:50.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.266+0000 7fa67d7fa700 1 -- 172.21.15.32:0/524439287 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa66c00a760 con 0x7fa6801037a0 2024-04-12T05:44:50.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.266+0000 7fa67d7fa700 1 -- 172.21.15.32:0/524439287 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa66c00ad70 con 0x7fa6801037a0 2024-04-12T05:44:50.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.266+0000 7fa67d7fa700 1 -- 172.21.15.32:0/524439287 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa66c0113a0 con 0x7fa6801037a0 2024-04-12T05:44:50.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.266+0000 7fa687143700 1 -- 172.21.15.32:0/524439287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa68010dd80 con 0x7fa6801037a0 2024-04-12T05:44:50.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.266+0000 7fa687143700 1 -- 172.21.15.32:0/524439287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa68010e340 con 0x7fa6801037a0 2024-04-12T05:44:50.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.267+0000 7fa687143700 1 -- 172.21.15.32:0/524439287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6800fe550 con 0x7fa6801037a0 2024-04-12T05:44:50.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.267+0000 7fa67d7fa700 1 -- 172.21.15.32:0/524439287 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa66c012400 con 0x7fa6801037a0 2024-04-12T05:44:50.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.268+0000 7fa67d7fa700 1 --2- 172.21.15.32:0/524439287 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa67006cc30 0x7fa67006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:50.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.268+0000 7fa67d7fa700 1 -- 172.21.15.32:0/524439287 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fa66c08bf30 con 0x7fa6801037a0 2024-04-12T05:44:50.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.268+0000 7fa684edf700 1 --2- 172.21.15.32:0/524439287 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa67006cc30 0x7fa67006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:50.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.268+0000 7fa684edf700 1 --2- 172.21.15.32:0/524439287 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa67006cc30 0x7fa67006f0e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa67400b840 tx=0x7fa674009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:50.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:49 smithi067 ceph-mon[112979]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:50.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:49 smithi067 ceph-mon[112979]: osd.3 marked itself down and dead 2024-04-12T05:44:50.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:49 smithi067 ceph-mon[112979]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:50.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:49 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3798870972' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:44:50.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.272+0000 7fa67d7fa700 1 -- 172.21.15.32:0/524439287 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa66c0bfab0 con 0x7fa6801037a0 2024-04-12T05:44:50.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:49 smithi032 ceph-mon[146817]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:50.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:49 smithi032 ceph-mon[146817]: osd.3 marked itself down and dead 2024-04-12T05:44:50.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:49 smithi032 ceph-mon[146817]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:50.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:49 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3798870972' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:44:50.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.493+0000 7fa687143700 1 -- 172.21.15.32:0/524439287 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa680109dc0 con 0x7fa67006cc30 2024-04-12T05:44:50.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.494+0000 7fa67d7fa700 1 -- 172.21.15.32:0/524439287 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fa680109dc0 con 0x7fa67006cc30 2024-04-12T05:44:50.491 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:44:50.491 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:44:50.491 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:44:50.491 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:44:50.491 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:44:50.491 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:44:50.491 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:44:50.492 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-12T05:44:50.492 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:44:50.492 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "9/21 daemons upgraded", 2024-04-12T05:44:50.492 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading osd daemons", 2024-04-12T05:44:50.492 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:44:50.492 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:44:50.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.496+0000 7fa687143700 1 -- 172.21.15.32:0/524439287 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa67006cc30 msgr2=0x7fa67006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:50.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.496+0000 7fa687143700 1 --2- 172.21.15.32:0/524439287 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa67006cc30 0x7fa67006f0e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa67400b840 tx=0x7fa674009e30 comp rx=0 tx=0).stop 2024-04-12T05:44:50.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.496+0000 7fa687143700 1 -- 172.21.15.32:0/524439287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6801037a0 msgr2=0x7fa680114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:50.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.496+0000 7fa687143700 1 --2- 172.21.15.32:0/524439287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6801037a0 0x7fa680114060 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa66c00c800 tx=0x7fa66c004470 comp rx=0 tx=0).stop 2024-04-12T05:44:50.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.496+0000 7fa687143700 1 -- 172.21.15.32:0/524439287 shutdown_connections 2024-04-12T05:44:50.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.496+0000 7fa687143700 1 --2- 172.21.15.32:0/524439287 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa67006cc30 0x7fa67006f0e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:50.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.496+0000 7fa687143700 1 --2- 172.21.15.32:0/524439287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa680102e00 0x7fa680113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:50.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.496+0000 7fa687143700 1 --2- 172.21.15.32:0/524439287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6801037a0 0x7fa680114060 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:50.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.496+0000 7fa687143700 1 -- 172.21.15.32:0/524439287 >> 172.21.15.32:0/524439287 conn(0x7fa6800f89b0 msgr2=0x7fa6800f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:50.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.496+0000 7fa687143700 1 -- 172.21.15.32:0/524439287 shutdown_connections 2024-04-12T05:44:50.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.496+0000 7fa687143700 1 -- 172.21.15.32:0/524439287 wait complete. 2024-04-12T05:44:50.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.599+0000 7f1e49484700 1 -- 172.21.15.32:0/2451273712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e44102f70 msgr2=0x7f1e44103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:50.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.599+0000 7f1e49484700 1 --2- 172.21.15.32:0/2451273712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e44102f70 0x7f1e44103340 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1e34004940 tx=0x7f1e34009e30 comp rx=0 tx=0).stop 2024-04-12T05:44:50.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.599+0000 7f1e49484700 1 -- 172.21.15.32:0/2451273712 shutdown_connections 2024-04-12T05:44:50.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.599+0000 7f1e49484700 1 --2- 172.21.15.32:0/2451273712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e440fb3b0 0x7f1e440fb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:50.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.599+0000 7f1e49484700 1 --2- 172.21.15.32:0/2451273712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e44102f70 0x7f1e44103340 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:50.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.599+0000 7f1e49484700 1 -- 172.21.15.32:0/2451273712 >> 172.21.15.32:0/2451273712 conn(0x7f1e44072310 msgr2=0x7f1e44072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:50.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.599+0000 7f1e49484700 1 -- 172.21.15.32:0/2451273712 shutdown_connections 2024-04-12T05:44:50.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.599+0000 7f1e49484700 1 -- 172.21.15.32:0/2451273712 wait complete. 2024-04-12T05:44:50.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.600+0000 7f1e49484700 1 Processor -- start 2024-04-12T05:44:50.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.600+0000 7f1e49484700 1 -- start start 2024-04-12T05:44:50.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.600+0000 7f1e49484700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e440fb3b0 0x7f1e44113820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:50.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.600+0000 7f1e49484700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e44102f70 0x7f1e44113d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:50.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.600+0000 7f1e49484700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e441143b0 con 0x7f1e44102f70 2024-04-12T05:44:50.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.600+0000 7f1e49484700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4410d9b0 con 0x7f1e440fb3b0 2024-04-12T05:44:50.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.601+0000 7f1e427fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e44102f70 0x7f1e44113d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:50.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.601+0000 7f1e427fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e44102f70 0x7f1e44113d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38680/0 (socket says 172.21.15.32:38680) 2024-04-12T05:44:50.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.601+0000 7f1e427fc700 1 -- 172.21.15.32:0/833055862 learned_addr learned my addr 172.21.15.32:0/833055862 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:44:50.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.601+0000 7f1e42ffd700 1 --2- 172.21.15.32:0/833055862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e440fb3b0 0x7f1e44113820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:50.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.601+0000 7f1e427fc700 1 -- 172.21.15.32:0/833055862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e440fb3b0 msgr2=0x7f1e44113820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:50.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.601+0000 7f1e427fc700 1 --2- 172.21.15.32:0/833055862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e440fb3b0 0x7f1e44113820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:50.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.601+0000 7f1e427fc700 1 -- 172.21.15.32:0/833055862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e34005040 con 0x7f1e44102f70 2024-04-12T05:44:50.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.601+0000 7f1e427fc700 1 --2- 172.21.15.32:0/833055862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e44102f70 0x7f1e44113d60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1e38002a50 tx=0x7f1e380046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:50.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.601+0000 7f1e2b7fe700 1 -- 172.21.15.32:0/833055862 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e38007cc0 con 0x7f1e44102f70 2024-04-12T05:44:50.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.601+0000 7f1e2b7fe700 1 -- 172.21.15.32:0/833055862 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e3800f040 con 0x7f1e44102f70 2024-04-12T05:44:50.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.601+0000 7f1e49484700 1 -- 172.21.15.32:0/833055862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e4410dc90 con 0x7f1e44102f70 2024-04-12T05:44:50.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.601+0000 7f1e2b7fe700 1 -- 172.21.15.32:0/833055862 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e3800d9c0 con 0x7f1e44102f70 2024-04-12T05:44:50.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.601+0000 7f1e49484700 1 -- 172.21.15.32:0/833055862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e4410e250 con 0x7f1e44102f70 2024-04-12T05:44:50.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.602+0000 7f1e49484700 1 -- 172.21.15.32:0/833055862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e440fe6d0 con 0x7f1e44102f70 2024-04-12T05:44:50.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.603+0000 7f1e2b7fe700 1 -- 172.21.15.32:0/833055862 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1e3800db20 con 0x7f1e44102f70 2024-04-12T05:44:50.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.603+0000 7f1e2b7fe700 1 --2- 172.21.15.32:0/833055862 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1e2c06cc20 0x7f1e2c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:44:50.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.603+0000 7f1e2b7fe700 1 -- 172.21.15.32:0/833055862 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f1e3808bdc0 con 0x7f1e44102f70 2024-04-12T05:44:50.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.603+0000 7f1e42ffd700 1 --2- 172.21.15.32:0/833055862 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1e2c06cc20 0x7f1e2c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:50.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.604+0000 7f1e42ffd700 1 --2- 172.21.15.32:0/833055862 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1e2c06cc20 0x7f1e2c06f0d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1e34004560 tx=0x7f1e34009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:50.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.608+0000 7f1e2b7fe700 1 -- 172.21.15.32:0/833055862 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1e3808f030 con 0x7f1e44102f70 2024-04-12T05:44:50.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.896+0000 7f1e49484700 1 -- 172.21.15.32:0/833055862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1e44105390 con 0x7f1e44102f70 2024-04-12T05:44:50.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.896+0000 7f1e2b7fe700 1 -- 172.21.15.32:0/833055862 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+100 (secure 0 0 0) 0x7f1e380597c0 con 0x7f1e44102f70 2024-04-12T05:44:50.893 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_WARN 1 osds down 2024-04-12T05:44:50.894 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-12T05:44:50.894 INFO:teuthology.orchestra.run.smithi032.stdout: osd.3 (root=default,host=smithi067) is down 2024-04-12T05:44:50.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.898+0000 7f1e49484700 1 -- 172.21.15.32:0/833055862 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1e2c06cc20 msgr2=0x7f1e2c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:50.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.898+0000 7f1e49484700 1 --2- 172.21.15.32:0/833055862 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1e2c06cc20 0x7f1e2c06f0d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1e34004560 tx=0x7f1e34009e30 comp rx=0 tx=0).stop 2024-04-12T05:44:50.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.898+0000 7f1e49484700 1 -- 172.21.15.32:0/833055862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e44102f70 msgr2=0x7f1e44113d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:50.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.898+0000 7f1e49484700 1 --2- 172.21.15.32:0/833055862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e44102f70 0x7f1e44113d60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1e38002a50 tx=0x7f1e380046b0 comp rx=0 tx=0).stop 2024-04-12T05:44:50.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.898+0000 7f1e49484700 1 -- 172.21.15.32:0/833055862 shutdown_connections 2024-04-12T05:44:50.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.898+0000 7f1e49484700 1 --2- 172.21.15.32:0/833055862 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1e2c06cc20 0x7f1e2c06f0d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:50.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.898+0000 7f1e49484700 1 --2- 172.21.15.32:0/833055862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e440fb3b0 0x7f1e44113820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:50.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.898+0000 7f1e49484700 1 --2- 172.21.15.32:0/833055862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e44102f70 0x7f1e44113d60 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:44:50.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.899+0000 7f1e49484700 1 -- 172.21.15.32:0/833055862 >> 172.21.15.32:0/833055862 conn(0x7f1e44072310 msgr2=0x7f1e440f9600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:50.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.899+0000 7f1e49484700 1 -- 172.21.15.32:0/833055862 shutdown_connections 2024-04-12T05:44:50.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:44:50.899+0000 7f1e49484700 1 -- 172.21.15.32:0/833055862 wait complete. 2024-04-12T05:44:50.899 INFO:journalctl@ceph.osd.3.smithi067.stdout:Apr 12 05:44:50 smithi067 bash[124664]: ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-3 2024-04-12T05:44:51.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:50 smithi067 ceph-mon[112979]: pgmap v109: 65 pgs: 65 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:44:51.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:50 smithi067 ceph-mon[112979]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T05:44:51.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:50 smithi067 ceph-mon[112979]: osdmap e65: 6 total, 5 up, 6 in 2024-04-12T05:44:51.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:50 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/668050775' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:44:51.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:50 smithi067 ceph-mon[112979]: from='client.34300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:51.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:50 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/833055862' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:44:51.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:50 smithi032 ceph-mon[146817]: pgmap v109: 65 pgs: 65 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T05:44:51.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:50 smithi032 ceph-mon[146817]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T05:44:51.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:50 smithi032 ceph-mon[146817]: osdmap e65: 6 total, 5 up, 6 in 2024-04-12T05:44:51.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:50 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/668050775' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:44:51.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:50 smithi032 ceph-mon[146817]: from='client.34300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:51.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:50 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/833055862' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:44:52.233 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:51 smithi067 ceph-mon[112979]: osdmap e66: 6 total, 5 up, 6 in 2024-04-12T05:44:52.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:51 smithi032 ceph-mon[146817]: osdmap e66: 6 total, 5 up, 6 in 2024-04-12T05:44:52.909 INFO:journalctl@ceph.osd.3.smithi067.stdout:Apr 12 05:44:52 smithi067 systemd[1]: ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@osd.3.service: Succeeded. 2024-04-12T05:44:52.909 INFO:journalctl@ceph.osd.3.smithi067.stdout:Apr 12 05:44:52 smithi067 systemd[1]: Stopped Ceph osd.3 for 37b58ce2-f88d-11ee-bc8c-c7b262605968. 2024-04-12T05:44:52.909 INFO:journalctl@ceph.osd.3.smithi067.stdout:Apr 12 05:44:52 smithi067 systemd[1]: Starting Ceph osd.3 for 37b58ce2-f88d-11ee-bc8c-c7b262605968... 2024-04-12T05:44:53.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:52 smithi067 ceph-mon[112979]: pgmap v112: 65 pgs: 16 stale+active+clean, 49 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 3 op/s 2024-04-12T05:44:53.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:52 smithi032 ceph-mon[146817]: pgmap v112: 65 pgs: 16 stale+active+clean, 49 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 3 op/s 2024-04-12T05:44:53.961 INFO:journalctl@ceph.osd.3.smithi067.stdout:Apr 12 05:44:53 smithi067 bash[124939]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-12T05:44:53.961 INFO:journalctl@ceph.osd.3.smithi067.stdout:Apr 12 05:44:53 smithi067 bash[124939]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-12T05:44:53.961 INFO:journalctl@ceph.osd.3.smithi067.stdout:Apr 12 05:44:53 smithi067 bash[124939]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-12T05:44:53.961 INFO:journalctl@ceph.osd.3.smithi067.stdout:Apr 12 05:44:53 smithi067 bash[124939]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-12T05:44:53.961 INFO:journalctl@ceph.osd.3.smithi067.stdout:Apr 12 05:44:53 smithi067 bash[124939]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-04-12T05:44:53.961 INFO:journalctl@ceph.osd.3.smithi067.stdout:Apr 12 05:44:53 smithi067 bash[124939]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-12T05:44:53.961 INFO:journalctl@ceph.osd.3.smithi067.stdout:Apr 12 05:44:53 smithi067 bash[124939]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-12T05:44:54.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:53 smithi067 ceph-mon[112979]: Health check failed: Degraded data redundancy: 40/333 objects degraded (12.012%), 9 pgs degraded (PG_DEGRADED) 2024-04-12T05:44:54.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:53 smithi032 ceph-mon[146817]: Health check failed: Degraded data redundancy: 40/333 objects degraded (12.012%), 9 pgs degraded (PG_DEGRADED) 2024-04-12T05:44:55.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:54 smithi067 ceph-mon[112979]: pgmap v113: 65 pgs: 9 active+undersized+degraded, 6 stale+active+clean, 9 active+undersized, 41 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 40/333 objects degraded (12.012%) 2024-04-12T05:44:55.268 INFO:journalctl@ceph.osd.3.smithi067.stdout:Apr 12 05:44:54 smithi067 ceph-osd[125268]: -- 172.21.15.67:0/2539393243 <== mon.1 v2:172.21.15.67:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c16a17d1e0 con 0x55c16a14cc00 2024-04-12T05:44:55.268 INFO:journalctl@ceph.osd.3.smithi067.stdout:Apr 12 05:44:55 smithi067 bash[125223]: b30df94d2d14340867bf11bee92fecbd5cbae0f13ec4bc7ede1177d9a932cb1b 2024-04-12T05:44:55.268 INFO:journalctl@ceph.osd.3.smithi067.stdout:Apr 12 05:44:55 smithi067 systemd[1]: Started Ceph osd.3 for 37b58ce2-f88d-11ee-bc8c-c7b262605968. 2024-04-12T05:44:55.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:54 smithi032 ceph-mon[146817]: pgmap v113: 65 pgs: 9 active+undersized+degraded, 6 stale+active+clean, 9 active+undersized, 41 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 40/333 objects degraded (12.012%) 2024-04-12T05:44:55.768 INFO:journalctl@ceph.osd.3.smithi067.stdout:Apr 12 05:44:55 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-3[125245]: 2024-04-12T05:44:55.420+0000 7ff35117a7c0 -1 Falling back to public interface 2024-04-12T05:44:56.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:56 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:56.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:56 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:56.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:56 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:44:56.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:56 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:56.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:56 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:56.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:56 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:44:57.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:57 smithi032 ceph-mon[146817]: pgmap v114: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 69/333 objects degraded (20.721%) 2024-04-12T05:44:57.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:57 smithi067 ceph-mon[112979]: pgmap v114: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 69/333 objects degraded (20.721%) 2024-04-12T05:44:59.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:58 smithi067 ceph-mon[112979]: pgmap v115: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 69/333 objects degraded (20.721%) 2024-04-12T05:44:59.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:58 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:59.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:58 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:59.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:58 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:59.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:44:58 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:59.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:58 smithi032 ceph-mon[146817]: pgmap v115: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 69/333 objects degraded (20.721%) 2024-04-12T05:44:59.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:58 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:59.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:58 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:59.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:58 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:44:59.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:44:58 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:00.995 INFO:journalctl@ceph.osd.3.smithi067.stdout:Apr 12 05:45:00 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-3[125245]: 2024-04-12T05:45:00.620+0000 7ff35117a7c0 -1 osd.3 64 log_to_monitors true 2024-04-12T05:45:01.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:00 smithi067 ceph-mon[112979]: pgmap v116: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 69/333 objects degraded (20.721%) 2024-04-12T05:45:01.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:00 smithi067 ceph-mon[112979]: from='osd.3 [v2:172.21.15.67:6800/2625997716,v1:172.21.15.67:6801/2625997716]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T05:45:01.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:00 smithi067 ceph-mon[112979]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T05:45:01.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:00 smithi032 ceph-mon[146817]: pgmap v116: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 270 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 69/333 objects degraded (20.721%) 2024-04-12T05:45:01.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:00 smithi032 ceph-mon[146817]: from='osd.3 [v2:172.21.15.67:6800/2625997716,v1:172.21.15.67:6801/2625997716]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T05:45:01.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:00 smithi032 ceph-mon[146817]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T05:45:02.150 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:02 smithi067 ceph-mon[112979]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-12T05:45:02.151 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:02 smithi067 ceph-mon[112979]: osdmap e67: 6 total, 5 up, 6 in 2024-04-12T05:45:02.151 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:02 smithi067 ceph-mon[112979]: from='osd.3 [v2:172.21.15.67:6800/2625997716,v1:172.21.15.67:6801/2625997716]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-12T05:45:02.151 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:02 smithi067 ceph-mon[112979]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-12T05:45:02.151 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:02 smithi067 ceph-mon[112979]: from='osd.3 ' entity='osd.3' 2024-04-12T05:45:02.151 INFO:journalctl@ceph.osd.3.smithi067.stdout:Apr 12 05:45:01 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-3[125245]: 2024-04-12T05:45:01.963+0000 7ff33e5cc700 -1 osd.3 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T05:45:02.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:02 smithi032 ceph-mon[146817]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-12T05:45:02.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:02 smithi032 ceph-mon[146817]: osdmap e67: 6 total, 5 up, 6 in 2024-04-12T05:45:02.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:02 smithi032 ceph-mon[146817]: from='osd.3 [v2:172.21.15.67:6800/2625997716,v1:172.21.15.67:6801/2625997716]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-12T05:45:02.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:02 smithi032 ceph-mon[146817]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-12T05:45:02.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:02 smithi032 ceph-mon[146817]: from='osd.3 ' entity='osd.3' 2024-04-12T05:45:03.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:03 smithi067 ceph-mon[112979]: pgmap v118: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 69/333 objects degraded (20.721%) 2024-04-12T05:45:03.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:03.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:03.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:45:03.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:45:03.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:03.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:45:03.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:45:03.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:45:03.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:03 smithi067 ceph-mon[112979]: Health check update: Degraded data redundancy: 69/333 objects degraded (20.721%), 19 pgs degraded (PG_DEGRADED) 2024-04-12T05:45:03.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:45:03.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-12T05:45:03.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:03 smithi067 ceph-mon[112979]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-12T05:45:03.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:03 smithi067 ceph-mon[112979]: Upgrade: unsafe to stop osd(s) at this time (18 PGs are or would become offline) 2024-04-12T05:45:03.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:03.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:45:03.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:03 smithi067 ceph-mon[112979]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T05:45:03.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:03 smithi067 ceph-mon[112979]: osd.3 [v2:172.21.15.67:6800/2625997716,v1:172.21.15.67:6801/2625997716] boot 2024-04-12T05:45:03.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:03 smithi067 ceph-mon[112979]: osdmap e68: 6 total, 6 up, 6 in 2024-04-12T05:45:03.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:45:03.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:03 smithi032 ceph-mon[146817]: pgmap v118: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 69/333 objects degraded (20.721%) 2024-04-12T05:45:03.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:03.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:03.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:45:03.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:45:03.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:03.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:45:03.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:45:03.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:45:03.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:03 smithi032 ceph-mon[146817]: Health check update: Degraded data redundancy: 69/333 objects degraded (20.721%), 19 pgs degraded (PG_DEGRADED) 2024-04-12T05:45:03.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:45:03.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-12T05:45:03.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:03 smithi032 ceph-mon[146817]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-12T05:45:03.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:03 smithi032 ceph-mon[146817]: Upgrade: unsafe to stop osd(s) at this time (18 PGs are or would become offline) 2024-04-12T05:45:03.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:03.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:45:03.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:03 smithi032 ceph-mon[146817]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T05:45:03.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:03 smithi032 ceph-mon[146817]: osd.3 [v2:172.21.15.67:6800/2625997716,v1:172.21.15.67:6801/2625997716] boot 2024-04-12T05:45:03.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:03 smithi032 ceph-mon[146817]: osdmap e68: 6 total, 6 up, 6 in 2024-04-12T05:45:03.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:45:05.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:04 smithi067 ceph-mon[112979]: pgmap v120: 65 pgs: 8 peering, 16 active+undersized+degraded, 14 active+undersized, 27 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s; 59/333 objects degraded (17.718%) 2024-04-12T05:45:05.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:04 smithi067 ceph-mon[112979]: osdmap e69: 6 total, 6 up, 6 in 2024-04-12T05:45:05.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:04 smithi032 ceph-mon[146817]: pgmap v120: 65 pgs: 8 peering, 16 active+undersized+degraded, 14 active+undersized, 27 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s; 59/333 objects degraded (17.718%) 2024-04-12T05:45:05.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:04 smithi032 ceph-mon[146817]: osdmap e69: 6 total, 6 up, 6 in 2024-04-12T05:45:07.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:06 smithi067 ceph-mon[112979]: pgmap v122: 65 pgs: 12 peering, 10 active+undersized+degraded, 6 active+undersized, 37 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 2 op/s; 41/333 objects degraded (12.312%) 2024-04-12T05:45:07.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:06 smithi032 ceph-mon[146817]: pgmap v122: 65 pgs: 12 peering, 10 active+undersized+degraded, 6 active+undersized, 37 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 2 op/s; 41/333 objects degraded (12.312%) 2024-04-12T05:45:08.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:07 smithi067 ceph-mon[112979]: Health check update: Degraded data redundancy: 41/333 objects degraded (12.312%), 10 pgs degraded (PG_DEGRADED) 2024-04-12T05:45:08.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:07 smithi032 ceph-mon[146817]: Health check update: Degraded data redundancy: 41/333 objects degraded (12.312%), 10 pgs degraded (PG_DEGRADED) 2024-04-12T05:45:09.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:08 smithi067 ceph-mon[112979]: pgmap v123: 65 pgs: 12 peering, 53 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-12T05:45:09.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:08 smithi067 ceph-mon[112979]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 41/333 objects degraded (12.312%), 10 pgs degraded) 2024-04-12T05:45:09.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:08 smithi067 ceph-mon[112979]: Cluster is now healthy 2024-04-12T05:45:09.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:08 smithi032 ceph-mon[146817]: pgmap v123: 65 pgs: 12 peering, 53 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-12T05:45:09.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:08 smithi032 ceph-mon[146817]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 41/333 objects degraded (12.312%), 10 pgs degraded) 2024-04-12T05:45:09.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:08 smithi032 ceph-mon[146817]: Cluster is now healthy 2024-04-12T05:45:11.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:10 smithi067 ceph-mon[112979]: pgmap v124: 65 pgs: 65 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-12T05:45:11.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:10 smithi032 ceph-mon[146817]: pgmap v124: 65 pgs: 65 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-12T05:45:13.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:12 smithi067 ceph-mon[112979]: pgmap v125: 65 pgs: 65 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-12T05:45:13.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:12 smithi032 ceph-mon[146817]: pgmap v125: 65 pgs: 65 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-12T05:45:15.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:14 smithi067 ceph-mon[112979]: pgmap v126: 65 pgs: 65 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T05:45:15.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:14 smithi032 ceph-mon[146817]: pgmap v126: 65 pgs: 65 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T05:45:17.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:16 smithi067 ceph-mon[112979]: pgmap v127: 65 pgs: 65 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-12T05:45:17.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:16 smithi032 ceph-mon[146817]: pgmap v127: 65 pgs: 65 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-12T05:45:18.230 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:17 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-12T05:45:18.230 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:17 smithi067 ceph-mon[112979]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-12T05:45:18.230 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:17 smithi067 ceph-mon[112979]: Upgrade: osd.4 is safe to restart 2024-04-12T05:45:18.230 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:17 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:18.230 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:17 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:45:18.230 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:17 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:18.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:17 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-12T05:45:18.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:17 smithi032 ceph-mon[146817]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-12T05:45:18.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:17 smithi032 ceph-mon[146817]: Upgrade: osd.4 is safe to restart 2024-04-12T05:45:18.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:17 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:18.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:17 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:45:18.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:17 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:19.112 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:18 smithi067 ceph-mon[112979]: pgmap v128: 65 pgs: 65 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 3 op/s 2024-04-12T05:45:19.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:18 smithi032 ceph-mon[146817]: pgmap v128: 65 pgs: 65 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 3 op/s 2024-04-12T05:45:20.511 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:20 smithi067 ceph-mon[112979]: Upgrade: Updating osd.4 2024-04-12T05:45:20.511 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:20 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:20.511 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:20 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-12T05:45:20.511 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:20 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:45:20.511 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:20 smithi067 ceph-mon[112979]: Deploying daemon osd.4 on smithi067 2024-04-12T05:45:20.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:20 smithi032 ceph-mon[146817]: Upgrade: Updating osd.4 2024-04-12T05:45:20.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:20 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:20.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:20 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-12T05:45:20.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:20 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:45:20.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:20 smithi032 ceph-mon[146817]: Deploying daemon osd.4 on smithi067 2024-04-12T05:45:21.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.020+0000 7fe221d4e700 1 -- 172.21.15.32:0/3426754479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe21c102e10 msgr2=0x7fe21c1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:21.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.020+0000 7fe221d4e700 1 --2- 172.21.15.32:0/3426754479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe21c102e10 0x7fe21c1031e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe20c004940 tx=0x7fe20c009e30 comp rx=0 tx=0).stop 2024-04-12T05:45:21.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.020+0000 7fe221d4e700 1 -- 172.21.15.32:0/3426754479 shutdown_connections 2024-04-12T05:45:21.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.020+0000 7fe221d4e700 1 --2- 172.21.15.32:0/3426754479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe21c1037b0 0x7fe21c0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:21.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.020+0000 7fe221d4e700 1 --2- 172.21.15.32:0/3426754479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe21c102e10 0x7fe21c1031e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:21.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.020+0000 7fe221d4e700 1 -- 172.21.15.32:0/3426754479 >> 172.21.15.32:0/3426754479 conn(0x7fe21c0f89a0 msgr2=0x7fe21c0fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:21.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.020+0000 7fe221d4e700 1 -- 172.21.15.32:0/3426754479 shutdown_connections 2024-04-12T05:45:21.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.021+0000 7fe221d4e700 1 -- 172.21.15.32:0/3426754479 wait complete. 2024-04-12T05:45:21.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.021+0000 7fe221d4e700 1 Processor -- start 2024-04-12T05:45:21.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.021+0000 7fe221d4e700 1 -- start start 2024-04-12T05:45:21.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.022+0000 7fe221d4e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe21c102e10 0x7fe21c113a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:21.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.022+0000 7fe221d4e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe21c1037b0 0x7fe21c113f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:21.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.022+0000 7fe221d4e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c1145e0 con 0x7fe21c102e10 2024-04-12T05:45:21.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.022+0000 7fe221d4e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c10d9d0 con 0x7fe21c1037b0 2024-04-12T05:45:21.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.022+0000 7fe21b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe21c102e10 0x7fe21c113a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:21.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.022+0000 7fe21b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe21c102e10 0x7fe21c113a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50798/0 (socket says 172.21.15.32:50798) 2024-04-12T05:45:21.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.022+0000 7fe21b7fe700 1 -- 172.21.15.32:0/1882075463 learned_addr learned my addr 172.21.15.32:0/1882075463 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:45:21.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.022+0000 7fe21affd700 1 --2- 172.21.15.32:0/1882075463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe21c1037b0 0x7fe21c113f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:21.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.022+0000 7fe21b7fe700 1 -- 172.21.15.32:0/1882075463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe21c1037b0 msgr2=0x7fe21c113f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:21.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.022+0000 7fe21b7fe700 1 --2- 172.21.15.32:0/1882075463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe21c1037b0 0x7fe21c113f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:21.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.022+0000 7fe21b7fe700 1 -- 172.21.15.32:0/1882075463 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20c005040 con 0x7fe21c102e10 2024-04-12T05:45:21.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.022+0000 7fe21b7fe700 1 --2- 172.21.15.32:0/1882075463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe21c102e10 0x7fe21c113a50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe20c00b3b0 tx=0x7fe20c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:21.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.022+0000 7fe203fff700 1 -- 172.21.15.32:0/1882075463 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe20c00e040 con 0x7fe21c102e10 2024-04-12T05:45:21.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.023+0000 7fe221d4e700 1 -- 172.21.15.32:0/1882075463 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe21c10dc50 con 0x7fe21c102e10 2024-04-12T05:45:21.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.023+0000 7fe203fff700 1 -- 172.21.15.32:0/1882075463 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe20c013420 con 0x7fe21c102e10 2024-04-12T05:45:21.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.023+0000 7fe203fff700 1 -- 172.21.15.32:0/1882075463 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe20c01c610 con 0x7fe21c102e10 2024-04-12T05:45:21.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.023+0000 7fe221d4e700 1 -- 172.21.15.32:0/1882075463 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe21c10e210 con 0x7fe21c102e10 2024-04-12T05:45:21.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.024+0000 7fe221d4e700 1 -- 172.21.15.32:0/1882075463 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe21c0fe560 con 0x7fe21c102e10 2024-04-12T05:45:21.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.024+0000 7fe203fff700 1 -- 172.21.15.32:0/1882075463 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe20c013a00 con 0x7fe21c102e10 2024-04-12T05:45:21.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.025+0000 7fe203fff700 1 --2- 172.21.15.32:0/1882075463 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe20406cc20 0x7fe20406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:21.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.025+0000 7fe203fff700 1 -- 172.21.15.32:0/1882075463 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fe20c08d140 con 0x7fe21c102e10 2024-04-12T05:45:21.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.025+0000 7fe21affd700 1 --2- 172.21.15.32:0/1882075463 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe20406cc20 0x7fe20406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:21.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.025+0000 7fe21affd700 1 --2- 172.21.15.32:0/1882075463 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe20406cc20 0x7fe20406f0d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe21c10eb90 tx=0x7fe21000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:21.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.030+0000 7fe203fff700 1 -- 172.21.15.32:0/1882075463 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe20c05ac60 con 0x7fe21c102e10 2024-04-12T05:45:21.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.252+0000 7fe221d4e700 1 -- 172.21.15.32:0/1882075463 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe21c109dd0 con 0x7fe20406cc20 2024-04-12T05:45:21.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.253+0000 7fe203fff700 1 -- 172.21.15.32:0/1882075463 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fe21c109dd0 con 0x7fe20406cc20 2024-04-12T05:45:21.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.255+0000 7fe221d4e700 1 -- 172.21.15.32:0/1882075463 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe20406cc20 msgr2=0x7fe20406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:21.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.255+0000 7fe221d4e700 1 --2- 172.21.15.32:0/1882075463 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe20406cc20 0x7fe20406f0d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe21c10eb90 tx=0x7fe21000b040 comp rx=0 tx=0).stop 2024-04-12T05:45:21.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.255+0000 7fe221d4e700 1 -- 172.21.15.32:0/1882075463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe21c102e10 msgr2=0x7fe21c113a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:21.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.255+0000 7fe221d4e700 1 --2- 172.21.15.32:0/1882075463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe21c102e10 0x7fe21c113a50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe20c00b3b0 tx=0x7fe20c009e30 comp rx=0 tx=0).stop 2024-04-12T05:45:21.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.255+0000 7fe221d4e700 1 -- 172.21.15.32:0/1882075463 shutdown_connections 2024-04-12T05:45:21.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.255+0000 7fe221d4e700 1 --2- 172.21.15.32:0/1882075463 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe20406cc20 0x7fe20406f0d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:21.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.255+0000 7fe221d4e700 1 --2- 172.21.15.32:0/1882075463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe21c102e10 0x7fe21c113a50 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:21.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.255+0000 7fe221d4e700 1 --2- 172.21.15.32:0/1882075463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe21c1037b0 0x7fe21c113f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:21.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.255+0000 7fe221d4e700 1 -- 172.21.15.32:0/1882075463 >> 172.21.15.32:0/1882075463 conn(0x7fe21c0f89a0 msgr2=0x7fe21c0f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:21.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.256+0000 7fe221d4e700 1 -- 172.21.15.32:0/1882075463 shutdown_connections 2024-04-12T05:45:21.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.256+0000 7fe221d4e700 1 -- 172.21.15.32:0/1882075463 wait complete. 2024-04-12T05:45:21.263 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:45:21.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.360+0000 7f7375739700 1 -- 172.21.15.32:0/1969343174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7370102710 msgr2=0x7f73700fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:21.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.360+0000 7f7375739700 1 --2- 172.21.15.32:0/1969343174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7370102710 0x7f73700fbd80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7358004940 tx=0x7f7358009e30 comp rx=0 tx=0).stop 2024-04-12T05:45:21.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.360+0000 7f7375739700 1 -- 172.21.15.32:0/1969343174 shutdown_connections 2024-04-12T05:45:21.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.360+0000 7f7375739700 1 --2- 172.21.15.32:0/1969343174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7370102710 0x7f73700fbd80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:21.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.360+0000 7f7375739700 1 --2- 172.21.15.32:0/1969343174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7370101d70 0x7f7370102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:21.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.360+0000 7f7375739700 1 -- 172.21.15.32:0/1969343174 >> 172.21.15.32:0/1969343174 conn(0x7f7370072bc0 msgr2=0x7f7370072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:21.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.360+0000 7f7375739700 1 -- 172.21.15.32:0/1969343174 shutdown_connections 2024-04-12T05:45:21.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.360+0000 7f7375739700 1 -- 172.21.15.32:0/1969343174 wait complete. 2024-04-12T05:45:21.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.361+0000 7f7375739700 1 Processor -- start 2024-04-12T05:45:21.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.361+0000 7f7375739700 1 -- start start 2024-04-12T05:45:21.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.361+0000 7f7375739700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7370101d70 0x7f7370112900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:21.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.361+0000 7f7375739700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7370102710 0x7f7370112e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:21.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.361+0000 7f7375739700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7370113490 con 0x7f7370102710 2024-04-12T05:45:21.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.361+0000 7f7375739700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f737010c880 con 0x7f7370101d70 2024-04-12T05:45:21.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.361+0000 7f736e7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7370102710 0x7f7370112e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:21.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.361+0000 7f736e7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7370102710 0x7f7370112e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50812/0 (socket says 172.21.15.32:50812) 2024-04-12T05:45:21.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.361+0000 7f736e7fc700 1 -- 172.21.15.32:0/2558084335 learned_addr learned my addr 172.21.15.32:0/2558084335 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:45:21.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.362+0000 7f736effd700 1 --2- 172.21.15.32:0/2558084335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7370101d70 0x7f7370112900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:21.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.362+0000 7f736e7fc700 1 -- 172.21.15.32:0/2558084335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7370101d70 msgr2=0x7f7370112900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:21.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.362+0000 7f736e7fc700 1 --2- 172.21.15.32:0/2558084335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7370101d70 0x7f7370112900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:21.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.362+0000 7f736e7fc700 1 -- 172.21.15.32:0/2558084335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7358005040 con 0x7f7370102710 2024-04-12T05:45:21.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.362+0000 7f736effd700 1 --2- 172.21.15.32:0/2558084335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7370101d70 0x7f7370112900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:45:21.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.362+0000 7f736e7fc700 1 --2- 172.21.15.32:0/2558084335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7370102710 0x7f7370112e40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f73580093b0 tx=0x7f7358009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:21.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.362+0000 7f73677fe700 1 -- 172.21.15.32:0/2558084335 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f735800e040 con 0x7f7370102710 2024-04-12T05:45:21.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.362+0000 7f73677fe700 1 -- 172.21.15.32:0/2558084335 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7358013420 con 0x7f7370102710 2024-04-12T05:45:21.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.362+0000 7f7375739700 1 -- 172.21.15.32:0/2558084335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f737010cb00 con 0x7f7370102710 2024-04-12T05:45:21.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.362+0000 7f73677fe700 1 -- 172.21.15.32:0/2558084335 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f735801c630 con 0x7f7370102710 2024-04-12T05:45:21.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.362+0000 7f7375739700 1 -- 172.21.15.32:0/2558084335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f737010d0c0 con 0x7f7370102710 2024-04-12T05:45:21.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.363+0000 7f7375739700 1 -- 172.21.15.32:0/2558084335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73700ff080 con 0x7f7370102710 2024-04-12T05:45:21.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.364+0000 7f73677fe700 1 -- 172.21.15.32:0/2558084335 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f73580139e0 con 0x7f7370102710 2024-04-12T05:45:21.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.364+0000 7f73677fe700 1 --2- 172.21.15.32:0/2558084335 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f735c06cc20 0x7f735c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:21.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.364+0000 7f73677fe700 1 -- 172.21.15.32:0/2558084335 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f735808d240 con 0x7f7370102710 2024-04-12T05:45:21.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.364+0000 7f736effd700 1 --2- 172.21.15.32:0/2558084335 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f735c06cc20 0x7f735c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:21.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.364+0000 7f736effd700 1 --2- 172.21.15.32:0/2558084335 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f735c06cc20 0x7f735c06f0d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f73600099f0 tx=0x7f736000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:21.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.368+0000 7f73677fe700 1 -- 172.21.15.32:0/2558084335 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f735805ae10 con 0x7f7370102710 2024-04-12T05:45:21.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:21 smithi067 ceph-mon[112979]: pgmap v129: 65 pgs: 65 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T05:45:21.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:21 smithi032 ceph-mon[146817]: pgmap v129: 65 pgs: 65 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T05:45:21.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.592+0000 7f7375739700 1 -- 172.21.15.32:0/2558084335 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f737005f690 con 0x7f735c06cc20 2024-04-12T05:45:21.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.593+0000 7f73677fe700 1 -- 172.21.15.32:0/2558084335 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f737005f690 con 0x7f735c06cc20 2024-04-12T05:45:21.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.595+0000 7f7375739700 1 -- 172.21.15.32:0/2558084335 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f735c06cc20 msgr2=0x7f735c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:21.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.595+0000 7f7375739700 1 --2- 172.21.15.32:0/2558084335 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f735c06cc20 0x7f735c06f0d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f73600099f0 tx=0x7f736000b040 comp rx=0 tx=0).stop 2024-04-12T05:45:21.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.595+0000 7f7375739700 1 -- 172.21.15.32:0/2558084335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7370102710 msgr2=0x7f7370112e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:21.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.595+0000 7f7375739700 1 --2- 172.21.15.32:0/2558084335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7370102710 0x7f7370112e40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f73580093b0 tx=0x7f7358009e30 comp rx=0 tx=0).stop 2024-04-12T05:45:21.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.595+0000 7f7375739700 1 -- 172.21.15.32:0/2558084335 shutdown_connections 2024-04-12T05:45:21.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.595+0000 7f7375739700 1 --2- 172.21.15.32:0/2558084335 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f735c06cc20 0x7f735c06f0d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:21.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.595+0000 7f7375739700 1 --2- 172.21.15.32:0/2558084335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7370101d70 0x7f7370112900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:21.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.595+0000 7f7375739700 1 --2- 172.21.15.32:0/2558084335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7370102710 0x7f7370112e40 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:21.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.595+0000 7f7375739700 1 -- 172.21.15.32:0/2558084335 >> 172.21.15.32:0/2558084335 conn(0x7f7370072bc0 msgr2=0x7f73700faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:21.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.595+0000 7f7375739700 1 -- 172.21.15.32:0/2558084335 shutdown_connections 2024-04-12T05:45:21.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.595+0000 7f7375739700 1 -- 172.21.15.32:0/2558084335 wait complete. 2024-04-12T05:45:21.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.698+0000 7f93bc9a9700 1 -- 172.21.15.32:0/1694644226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93b81026a0 msgr2=0x7f93b80fbd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:21.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.699+0000 7f93bc9a9700 1 --2- 172.21.15.32:0/1694644226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93b81026a0 0x7f93b80fbd10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f93ac004940 tx=0x7f93ac009e30 comp rx=0 tx=0).stop 2024-04-12T05:45:21.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.699+0000 7f93bc9a9700 1 -- 172.21.15.32:0/1694644226 shutdown_connections 2024-04-12T05:45:21.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.699+0000 7f93bc9a9700 1 --2- 172.21.15.32:0/1694644226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93b81026a0 0x7f93b80fbd10 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:21.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.699+0000 7f93bc9a9700 1 --2- 172.21.15.32:0/1694644226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93b8101d00 0x7f93b81020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:21.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.699+0000 7f93bc9a9700 1 -- 172.21.15.32:0/1694644226 >> 172.21.15.32:0/1694644226 conn(0x7f93b80f78d0 msgr2=0x7f93b80f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:21.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.699+0000 7f93bc9a9700 1 -- 172.21.15.32:0/1694644226 shutdown_connections 2024-04-12T05:45:21.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.699+0000 7f93bc9a9700 1 -- 172.21.15.32:0/1694644226 wait complete. 2024-04-12T05:45:21.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.699+0000 7f93bc9a9700 1 Processor -- start 2024-04-12T05:45:21.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.699+0000 7f93bc9a9700 1 -- start start 2024-04-12T05:45:21.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.699+0000 7f93bc9a9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93b8101d00 0x7f93b810eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:21.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.699+0000 7f93bc9a9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93b81026a0 0x7f93b810f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:21.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.699+0000 7f93bc9a9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b810f5f0 con 0x7f93b8101d00 2024-04-12T05:45:21.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.699+0000 7f93bc9a9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b810f730 con 0x7f93b81026a0 2024-04-12T05:45:21.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.700+0000 7f93b659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93b8101d00 0x7f93b810eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:21.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.700+0000 7f93b659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93b8101d00 0x7f93b810eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51112/0 (socket says 172.21.15.32:51112) 2024-04-12T05:45:21.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.700+0000 7f93b659c700 1 -- 172.21.15.32:0/4040430488 learned_addr learned my addr 172.21.15.32:0/4040430488 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:45:21.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.700+0000 7f93b5d9b700 1 --2- 172.21.15.32:0/4040430488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93b81026a0 0x7f93b810f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:21.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.700+0000 7f93b659c700 1 -- 172.21.15.32:0/4040430488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93b81026a0 msgr2=0x7f93b810f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:21.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.700+0000 7f93b659c700 1 --2- 172.21.15.32:0/4040430488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93b81026a0 0x7f93b810f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:21.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.700+0000 7f93b659c700 1 -- 172.21.15.32:0/4040430488 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ac005040 con 0x7f93b8101d00 2024-04-12T05:45:21.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.700+0000 7f93b5d9b700 1 --2- 172.21.15.32:0/4040430488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93b81026a0 0x7f93b810f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:45:21.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.700+0000 7f93b659c700 1 --2- 172.21.15.32:0/4040430488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93b8101d00 0x7f93b810eb70 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f93a800eb40 tx=0x7f93a800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:21.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.700+0000 7f93a6ffd700 1 -- 172.21.15.32:0/4040430488 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93a8007cc0 con 0x7f93b8101d00 2024-04-12T05:45:21.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.700+0000 7f93a6ffd700 1 -- 172.21.15.32:0/4040430488 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93a800f040 con 0x7f93b8101d00 2024-04-12T05:45:21.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.700+0000 7f93bc9a9700 1 -- 172.21.15.32:0/4040430488 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93b8117a20 con 0x7f93b8101d00 2024-04-12T05:45:21.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.700+0000 7f93a6ffd700 1 -- 172.21.15.32:0/4040430488 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93a8009570 con 0x7f93b8101d00 2024-04-12T05:45:21.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.700+0000 7f93bc9a9700 1 -- 172.21.15.32:0/4040430488 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93b8117f80 con 0x7f93b8101d00 2024-04-12T05:45:21.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.701+0000 7f93bc9a9700 1 -- 172.21.15.32:0/4040430488 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93b80fd450 con 0x7f93b8101d00 2024-04-12T05:45:21.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.702+0000 7f93a6ffd700 1 -- 172.21.15.32:0/4040430488 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f93a80096d0 con 0x7f93b8101d00 2024-04-12T05:45:21.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.702+0000 7f93a6ffd700 1 --2- 172.21.15.32:0/4040430488 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f93a006cc20 0x7f93a006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:21.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.702+0000 7f93a6ffd700 1 -- 172.21.15.32:0/4040430488 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f93a808bce0 con 0x7f93b8101d00 2024-04-12T05:45:21.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.702+0000 7f93b5d9b700 1 --2- 172.21.15.32:0/4040430488 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f93a006cc20 0x7f93a006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:21.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.703+0000 7f93b5d9b700 1 --2- 172.21.15.32:0/4040430488 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f93a006cc20 0x7f93a006f0d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f93ac00b0d0 tx=0x7f93ac009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:21.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.707+0000 7f93a6ffd700 1 -- 172.21.15.32:0/4040430488 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f93a8055da0 con 0x7f93b8101d00 2024-04-12T05:45:21.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.918+0000 7f93bc9a9700 1 -- 172.21.15.32:0/4040430488 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f93b804bf70 con 0x7f93a006cc20 2024-04-12T05:45:21.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.928+0000 7f93a6ffd700 1 -- 172.21.15.32:0/4040430488 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f93ac007cc0 con 0x7f93a006cc20 2024-04-12T05:45:21.924 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:45:21.925 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (5m) 57s ago 16m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:45:21.925 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (2m) 57s ago 16m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:45:21.925 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (2m) 23s ago 15m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:45:21.925 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (4m) 57s ago 16m 84.7M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:45:21.925 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (10m) 57s ago 10m 64.8M - 17.2.7-827-gd3b5060a 21de9b7369ad e7ada8e5259c 2024-04-12T05:45:21.925 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (10m) 57s ago 10m 115M - 17.2.7-827-gd3b5060a 21de9b7369ad 55124da9afdb 2024-04-12T05:45:21.925 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (10m) 23s ago 10m 62.6M - 17.2.7-827-gd3b5060a 21de9b7369ad 0c295d31a4a6 2024-04-12T05:45:21.925 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (10m) 23s ago 10m 120M - 17.2.7-827-gd3b5060a 21de9b7369ad 6eecd7deaac2 2024-04-12T05:45:21.925 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (6m) 57s ago 18m 499M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:45:21.925 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (6m) 23s ago 15m 448M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:45:21.925 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (3m) 57s ago 18m 46.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:45:21.925 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (3m) 23s ago 15m 37.4M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:45:21.925 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (5m) 57s ago 16m 16.6M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:45:21.925 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (5m) 23s ago 15m 18.3M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:45:21.925 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (2m) 57s ago 14m 86.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T05:45:21.926 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (98s) 57s ago 14m 94.5M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T05:45:21.926 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (62s) 57s ago 13m 11.2M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T05:45:21.926 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (27s) 23s ago 13m 11.5M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T05:45:21.926 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (12m) 23s ago 12m 230M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad 2c84d9047e9c 2024-04-12T05:45:21.926 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (12m) 23s ago 12m 219M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad e54839bb6055 2024-04-12T05:45:21.926 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (5m) 57s ago 16m 63.5M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:45:21.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.930+0000 7f93bc9a9700 1 -- 172.21.15.32:0/4040430488 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f93a006cc20 msgr2=0x7f93a006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:21.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.930+0000 7f93bc9a9700 1 --2- 172.21.15.32:0/4040430488 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f93a006cc20 0x7f93a006f0d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f93ac00b0d0 tx=0x7f93ac009e30 comp rx=0 tx=0).stop 2024-04-12T05:45:21.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.930+0000 7f93bc9a9700 1 -- 172.21.15.32:0/4040430488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93b8101d00 msgr2=0x7f93b810eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:21.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.930+0000 7f93bc9a9700 1 --2- 172.21.15.32:0/4040430488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93b8101d00 0x7f93b810eb70 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f93a800eb40 tx=0x7f93a800b040 comp rx=0 tx=0).stop 2024-04-12T05:45:21.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.930+0000 7f93bc9a9700 1 -- 172.21.15.32:0/4040430488 shutdown_connections 2024-04-12T05:45:21.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.930+0000 7f93bc9a9700 1 --2- 172.21.15.32:0/4040430488 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f93a006cc20 0x7f93a006f0d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:21.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.930+0000 7f93bc9a9700 1 --2- 172.21.15.32:0/4040430488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93b8101d00 0x7f93b810eb70 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:21.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.930+0000 7f93bc9a9700 1 --2- 172.21.15.32:0/4040430488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93b81026a0 0x7f93b810f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:21.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.930+0000 7f93bc9a9700 1 -- 172.21.15.32:0/4040430488 >> 172.21.15.32:0/4040430488 conn(0x7f93b80f78d0 msgr2=0x7f93b80f8930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:21.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.930+0000 7f93bc9a9700 1 -- 172.21.15.32:0/4040430488 shutdown_connections 2024-04-12T05:45:21.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:21.930+0000 7f93bc9a9700 1 -- 172.21.15.32:0/4040430488 wait complete. 2024-04-12T05:45:22.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.032+0000 7fc3aadfc700 1 -- 172.21.15.32:0/2054432406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3a4101d70 msgr2=0x7fc3a4102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:22.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.032+0000 7fc3aadfc700 1 --2- 172.21.15.32:0/2054432406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3a4101d70 0x7fc3a4102140 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc398004940 tx=0x7fc398009e30 comp rx=0 tx=0).stop 2024-04-12T05:45:22.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.032+0000 7fc3aadfc700 1 -- 172.21.15.32:0/2054432406 shutdown_connections 2024-04-12T05:45:22.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.032+0000 7fc3aadfc700 1 --2- 172.21.15.32:0/2054432406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a4102710 0x7fc3a40fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:22.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.032+0000 7fc3aadfc700 1 --2- 172.21.15.32:0/2054432406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3a4101d70 0x7fc3a4102140 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:22.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.032+0000 7fc3aadfc700 1 -- 172.21.15.32:0/2054432406 >> 172.21.15.32:0/2054432406 conn(0x7fc3a4072bc0 msgr2=0x7fc3a4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:22.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.032+0000 7fc3aadfc700 1 -- 172.21.15.32:0/2054432406 shutdown_connections 2024-04-12T05:45:22.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.032+0000 7fc3aadfc700 1 -- 172.21.15.32:0/2054432406 wait complete. 2024-04-12T05:45:22.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.033+0000 7fc3aadfc700 1 Processor -- start 2024-04-12T05:45:22.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.033+0000 7fc3aadfc700 1 -- start start 2024-04-12T05:45:22.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.033+0000 7fc3aadfc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3a4101d70 0x7fc3a4113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:22.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.033+0000 7fc3aadfc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a4102710 0x7fc3a4113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:22.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.033+0000 7fc3aadfc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a41145f0 con 0x7fc3a4101d70 2024-04-12T05:45:22.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.033+0000 7fc3aadfc700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a410d9e0 con 0x7fc3a4102710 2024-04-12T05:45:22.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.033+0000 7fc3a8b98700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3a4101d70 0x7fc3a4113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:22.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.033+0000 7fc3a8b98700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3a4101d70 0x7fc3a4113a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51132/0 (socket says 172.21.15.32:51132) 2024-04-12T05:45:22.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.033+0000 7fc3a8b98700 1 -- 172.21.15.32:0/1205559889 learned_addr learned my addr 172.21.15.32:0/1205559889 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:45:22.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.033+0000 7fc3a3fff700 1 --2- 172.21.15.32:0/1205559889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a4102710 0x7fc3a4113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:22.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.033+0000 7fc3a8b98700 1 -- 172.21.15.32:0/1205559889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a4102710 msgr2=0x7fc3a4113fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:22.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.033+0000 7fc3a8b98700 1 --2- 172.21.15.32:0/1205559889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a4102710 0x7fc3a4113fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:22.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.033+0000 7fc3a8b98700 1 -- 172.21.15.32:0/1205559889 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc398005040 con 0x7fc3a4101d70 2024-04-12T05:45:22.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.034+0000 7fc3a8b98700 1 --2- 172.21.15.32:0/1205559889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3a4101d70 0x7fc3a4113a60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc398009180 tx=0x7fc398009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:22.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.034+0000 7fc3a17fa700 1 -- 172.21.15.32:0/1205559889 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc39800e040 con 0x7fc3a4101d70 2024-04-12T05:45:22.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.034+0000 7fc3a17fa700 1 -- 172.21.15.32:0/1205559889 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc398004290 con 0x7fc3a4101d70 2024-04-12T05:45:22.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.034+0000 7fc3aadfc700 1 -- 172.21.15.32:0/1205559889 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a410dc60 con 0x7fc3a4101d70 2024-04-12T05:45:22.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.034+0000 7fc3a17fa700 1 -- 172.21.15.32:0/1205559889 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc398013e60 con 0x7fc3a4101d70 2024-04-12T05:45:22.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.034+0000 7fc3aadfc700 1 -- 172.21.15.32:0/1205559889 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a410e220 con 0x7fc3a4101d70 2024-04-12T05:45:22.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.035+0000 7fc3aadfc700 1 -- 172.21.15.32:0/1205559889 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a40fd4c0 con 0x7fc3a4101d70 2024-04-12T05:45:22.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.035+0000 7fc3a17fa700 1 -- 172.21.15.32:0/1205559889 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc3980136b0 con 0x7fc3a4101d70 2024-04-12T05:45:22.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.036+0000 7fc3a17fa700 1 --2- 172.21.15.32:0/1205559889 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc39406cc20 0x7fc39406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:22.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.036+0000 7fc3a17fa700 1 -- 172.21.15.32:0/1205559889 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fc39808d170 con 0x7fc3a4101d70 2024-04-12T05:45:22.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.036+0000 7fc3a3fff700 1 --2- 172.21.15.32:0/1205559889 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc39406cc20 0x7fc39406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:22.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.036+0000 7fc3a3fff700 1 --2- 172.21.15.32:0/1205559889 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc39406cc20 0x7fc39406f0d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc3a410eba0 tx=0x7fc390009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:22.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.040+0000 7fc3a17fa700 1 -- 172.21.15.32:0/1205559889 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc39805ad30 con 0x7fc3a4101d70 2024-04-12T05:45:22.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.329+0000 7fc3aadfc700 1 -- 172.21.15.32:0/1205559889 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc3a4108d30 con 0x7fc3a4101d70 2024-04-12T05:45:22.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.330+0000 7fc3a17fa700 1 -- 172.21.15.32:0/1205559889 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fc39805a8c0 con 0x7fc3a4101d70 2024-04-12T05:45:22.327 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:45:22.327 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:45:22.327 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:45:22.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:45:22.327 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:45:22.327 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:45:22.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:45:22.327 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:45:22.327 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 2, 2024-04-12T05:45:22.327 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:45:22.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:45:22.327 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:45:22.328 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 4 2024-04-12T05:45:22.328 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:45:22.328 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:45:22.328 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 6, 2024-04-12T05:45:22.328 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 8 2024-04-12T05:45:22.328 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:45:22.328 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:45:22.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.332+0000 7fc3aadfc700 1 -- 172.21.15.32:0/1205559889 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc39406cc20 msgr2=0x7fc39406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:22.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.332+0000 7fc3aadfc700 1 --2- 172.21.15.32:0/1205559889 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc39406cc20 0x7fc39406f0d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc3a410eba0 tx=0x7fc390009040 comp rx=0 tx=0).stop 2024-04-12T05:45:22.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.332+0000 7fc3aadfc700 1 -- 172.21.15.32:0/1205559889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3a4101d70 msgr2=0x7fc3a4113a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:22.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.332+0000 7fc3aadfc700 1 --2- 172.21.15.32:0/1205559889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3a4101d70 0x7fc3a4113a60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc398009180 tx=0x7fc398009e30 comp rx=0 tx=0).stop 2024-04-12T05:45:22.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.332+0000 7fc3aadfc700 1 -- 172.21.15.32:0/1205559889 shutdown_connections 2024-04-12T05:45:22.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.332+0000 7fc3aadfc700 1 --2- 172.21.15.32:0/1205559889 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc39406cc20 0x7fc39406f0d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:22.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.332+0000 7fc3aadfc700 1 --2- 172.21.15.32:0/1205559889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3a4101d70 0x7fc3a4113a60 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:22.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.332+0000 7fc3aadfc700 1 --2- 172.21.15.32:0/1205559889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a4102710 0x7fc3a4113fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:22.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.332+0000 7fc3aadfc700 1 -- 172.21.15.32:0/1205559889 >> 172.21.15.32:0/1205559889 conn(0x7fc3a4072bc0 msgr2=0x7fc3a40659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:22.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.332+0000 7fc3aadfc700 1 -- 172.21.15.32:0/1205559889 shutdown_connections 2024-04-12T05:45:22.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.332+0000 7fc3aadfc700 1 -- 172.21.15.32:0/1205559889 wait complete. 2024-04-12T05:45:22.414 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 05:45:22 smithi067 systemd[1]: Stopping Ceph osd.4 for 37b58ce2-f88d-11ee-bc8c-c7b262605968... 2024-04-12T05:45:22.414 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:22 smithi067 ceph-mon[112979]: from='client.34310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:45:22.414 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:22 smithi067 ceph-mon[112979]: from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:45:22.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.435+0000 7f00293a7700 1 -- 172.21.15.32:0/1086071483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00241037a0 msgr2=0x7f00240fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:22.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.435+0000 7f00293a7700 1 --2- 172.21.15.32:0/1086071483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00241037a0 0x7f00240fce10 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0018004940 tx=0x7f0018009e30 comp rx=0 tx=0).stop 2024-04-12T05:45:22.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.435+0000 7f00293a7700 1 -- 172.21.15.32:0/1086071483 shutdown_connections 2024-04-12T05:45:22.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.435+0000 7f00293a7700 1 --2- 172.21.15.32:0/1086071483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00241037a0 0x7f00240fce10 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:22.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.435+0000 7f00293a7700 1 --2- 172.21.15.32:0/1086071483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0024102e00 0x7f00241031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:22.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.435+0000 7f00293a7700 1 -- 172.21.15.32:0/1086071483 >> 172.21.15.32:0/1086071483 conn(0x7f00240f89b0 msgr2=0x7f00240fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:22.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.435+0000 7f00293a7700 1 -- 172.21.15.32:0/1086071483 shutdown_connections 2024-04-12T05:45:22.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.435+0000 7f00293a7700 1 -- 172.21.15.32:0/1086071483 wait complete. 2024-04-12T05:45:22.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.436+0000 7f00293a7700 1 Processor -- start 2024-04-12T05:45:22.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.436+0000 7f00293a7700 1 -- start start 2024-04-12T05:45:22.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.436+0000 7f00293a7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0024102e00 0x7f00240702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:22.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.436+0000 7f00293a7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00241037a0 0x7f0024070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:22.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.436+0000 7f00293a7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0024072210 con 0x7f00241037a0 2024-04-12T05:45:22.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.436+0000 7f00293a7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0024070d40 con 0x7f0024102e00 2024-04-12T05:45:22.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.436+0000 7f00227fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00241037a0 0x7f0024070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:22.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.436+0000 7f00227fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00241037a0 0x7f0024070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51150/0 (socket says 172.21.15.32:51150) 2024-04-12T05:45:22.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.436+0000 7f00227fc700 1 -- 172.21.15.32:0/1132570022 learned_addr learned my addr 172.21.15.32:0/1132570022 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:45:22.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.436+0000 7f0022ffd700 1 --2- 172.21.15.32:0/1132570022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0024102e00 0x7f00240702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:22.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.437+0000 7f00227fc700 1 -- 172.21.15.32:0/1132570022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0024102e00 msgr2=0x7f00240702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:22.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.437+0000 7f00227fc700 1 --2- 172.21.15.32:0/1132570022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0024102e00 0x7f00240702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:22.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.437+0000 7f00227fc700 1 -- 172.21.15.32:0/1132570022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0018005040 con 0x7f00241037a0 2024-04-12T05:45:22.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.437+0000 7f00227fc700 1 --2- 172.21.15.32:0/1132570022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00241037a0 0x7f0024070800 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0018009390 tx=0x7f0018009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:22.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.437+0000 7f000b7fe700 1 -- 172.21.15.32:0/1132570022 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f001800e040 con 0x7f00241037a0 2024-04-12T05:45:22.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.437+0000 7f00293a7700 1 -- 172.21.15.32:0/1132570022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0024070fc0 con 0x7f00241037a0 2024-04-12T05:45:22.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.437+0000 7f000b7fe700 1 -- 172.21.15.32:0/1132570022 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0018004290 con 0x7f00241037a0 2024-04-12T05:45:22.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.437+0000 7f000b7fe700 1 -- 172.21.15.32:0/1132570022 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0018013e70 con 0x7f00241037a0 2024-04-12T05:45:22.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.437+0000 7f00293a7700 1 -- 172.21.15.32:0/1132570022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f002411a010 con 0x7f00241037a0 2024-04-12T05:45:22.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.438+0000 7f00293a7700 1 -- 172.21.15.32:0/1132570022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00240fe550 con 0x7f00241037a0 2024-04-12T05:45:22.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.439+0000 7f000b7fe700 1 -- 172.21.15.32:0/1132570022 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f001800ba80 con 0x7f00241037a0 2024-04-12T05:45:22.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.439+0000 7f000b7fe700 1 --2- 172.21.15.32:0/1132570022 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f000c06cbc0 0x7f000c06f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:22.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.439+0000 7f000b7fe700 1 -- 172.21.15.32:0/1132570022 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f001808d1d0 con 0x7f00241037a0 2024-04-12T05:45:22.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.439+0000 7f0022ffd700 1 --2- 172.21.15.32:0/1132570022 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f000c06cbc0 0x7f000c06f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:22.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.439+0000 7f0022ffd700 1 --2- 172.21.15.32:0/1132570022 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f000c06cbc0 0x7f000c06f070 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0014004f40 tx=0x7f001400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:22.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.444+0000 7f000b7fe700 1 -- 172.21.15.32:0/1132570022 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0018057260 con 0x7f00241037a0 2024-04-12T05:45:22.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:22 smithi032 ceph-mon[146817]: from='client.34310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:45:22.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:22 smithi032 ceph-mon[146817]: from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:45:22.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.695+0000 7f00293a7700 1 -- 172.21.15.32:0/1132570022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f002411a250 con 0x7f00241037a0 2024-04-12T05:45:22.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.695+0000 7f000b7fe700 1 -- 172.21.15.32:0/1132570022 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1864 (secure 0 0 0) 0x7f001805a880 con 0x7f00241037a0 2024-04-12T05:45:22.692 INFO:teuthology.orchestra.run.smithi032.stdout:e22 2024-04-12T05:45:22.692 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:45:22.692 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:45:22.692 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:45:22.692 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:45:22.692 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:45:22.692 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:45:22.692 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 22 2024-04-12T05:45:22.693 INFO:teuthology.orchestra.run.smithi032.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-12T05:45:22.693 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:45:22.693 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:41:05.358509+0000 2024-04-12T05:45:22.693 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:45:22.693 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:45:22.693 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:45:22.693 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:45:22.693 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:45:22.693 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:45:22.693 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:45:22.693 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:45:22.693 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 46 2024-04-12T05:45:22.693 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:45:22.693 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:45:22.694 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:45:22.694 INFO:teuthology.orchestra.run.smithi032.stdout:up {0=14480,1=24281} 2024-04-12T05:45:22.694 INFO:teuthology.orchestra.run.smithi032.stdout:failed 2024-04-12T05:45:22.694 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:45:22.694 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:45:22.694 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:45:22.694 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:45:22.694 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:45:22.694 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:45:22.694 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:45:22.694 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:45:22.694 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{0:14480} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.32:6828/387754149,v1:172.21.15.32:6829/387754149] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:45:22.694 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{0:14490} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3678031004,v1:172.21.15.32:6827/3678031004] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:45:22.694 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{1:24281} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.67:6824/1291914599,v1:172.21.15.67:6825/1291914599] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:45:22.695 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{1:24295} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.67:6826/873395842,v1:172.21.15.67:6827/873395842] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:45:22.695 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:45:22.695 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:45:22.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.697+0000 7f00293a7700 1 -- 172.21.15.32:0/1132570022 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f000c06cbc0 msgr2=0x7f000c06f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:22.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.697+0000 7f00293a7700 1 --2- 172.21.15.32:0/1132570022 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f000c06cbc0 0x7f000c06f070 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0014004f40 tx=0x7f001400b040 comp rx=0 tx=0).stop 2024-04-12T05:45:22.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.697+0000 7f00293a7700 1 -- 172.21.15.32:0/1132570022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00241037a0 msgr2=0x7f0024070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:22.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.697+0000 7f00293a7700 1 --2- 172.21.15.32:0/1132570022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00241037a0 0x7f0024070800 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0018009390 tx=0x7f0018009e30 comp rx=0 tx=0).stop 2024-04-12T05:45:22.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.698+0000 7f00293a7700 1 -- 172.21.15.32:0/1132570022 shutdown_connections 2024-04-12T05:45:22.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.698+0000 7f00293a7700 1 --2- 172.21.15.32:0/1132570022 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f000c06cbc0 0x7f000c06f070 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:22.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.698+0000 7f00293a7700 1 --2- 172.21.15.32:0/1132570022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0024102e00 0x7f00240702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:22.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.698+0000 7f00293a7700 1 --2- 172.21.15.32:0/1132570022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00241037a0 0x7f0024070800 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:22.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.698+0000 7f00293a7700 1 -- 172.21.15.32:0/1132570022 >> 172.21.15.32:0/1132570022 conn(0x7f00240f89b0 msgr2=0x7f00240f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:22.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.698+0000 7f00293a7700 1 -- 172.21.15.32:0/1132570022 shutdown_connections 2024-04-12T05:45:22.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.698+0000 7f00293a7700 1 -- 172.21.15.32:0/1132570022 wait complete. 2024-04-12T05:45:22.696 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 22 2024-04-12T05:45:22.767 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 05:45:22 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-4[63297]: 2024-04-12T05:45:22.416+0000 7f8b1d2f1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T05:45:22.767 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 05:45:22 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-4[63297]: 2024-04-12T05:45:22.416+0000 7f8b1d2f1700 -1 osd.4 69 *** Got signal Terminated *** 2024-04-12T05:45:22.768 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 05:45:22 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-4[63297]: 2024-04-12T05:45:22.416+0000 7f8b1d2f1700 -1 osd.4 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T05:45:22.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.803+0000 7fe6816ab700 1 -- 172.21.15.32:0/933095100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe67c101d70 msgr2=0x7fe67c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:22.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.803+0000 7fe6816ab700 1 --2- 172.21.15.32:0/933095100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe67c101d70 0x7fe67c102140 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe670004940 tx=0x7fe670009e30 comp rx=0 tx=0).stop 2024-04-12T05:45:22.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.803+0000 7fe6816ab700 1 -- 172.21.15.32:0/933095100 shutdown_connections 2024-04-12T05:45:22.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.803+0000 7fe6816ab700 1 --2- 172.21.15.32:0/933095100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67c102710 0x7fe67c0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:22.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.803+0000 7fe6816ab700 1 --2- 172.21.15.32:0/933095100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe67c101d70 0x7fe67c102140 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:22.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.803+0000 7fe6816ab700 1 -- 172.21.15.32:0/933095100 >> 172.21.15.32:0/933095100 conn(0x7fe67c072bc0 msgr2=0x7fe67c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:22.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.803+0000 7fe6816ab700 1 -- 172.21.15.32:0/933095100 shutdown_connections 2024-04-12T05:45:22.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.803+0000 7fe6816ab700 1 -- 172.21.15.32:0/933095100 wait complete. 2024-04-12T05:45:22.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.803+0000 7fe6816ab700 1 Processor -- start 2024-04-12T05:45:22.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.803+0000 7fe6816ab700 1 -- start start 2024-04-12T05:45:22.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.804+0000 7fe6816ab700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67c101d70 0x7fe67c10ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:22.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.804+0000 7fe6816ab700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe67c102710 0x7fe67c10f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:22.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.804+0000 7fe6816ab700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c10f670 con 0x7fe67c102710 2024-04-12T05:45:22.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.804+0000 7fe6816ab700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c10f7b0 con 0x7fe67c101d70 2024-04-12T05:45:22.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.804+0000 7fe67a7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe67c102710 0x7fe67c10f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:22.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.804+0000 7fe67a7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe67c102710 0x7fe67c10f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51168/0 (socket says 172.21.15.32:51168) 2024-04-12T05:45:22.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.804+0000 7fe67a7fc700 1 -- 172.21.15.32:0/1828018754 learned_addr learned my addr 172.21.15.32:0/1828018754 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:45:22.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.804+0000 7fe67affd700 1 --2- 172.21.15.32:0/1828018754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67c101d70 0x7fe67c10ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:22.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.804+0000 7fe67a7fc700 1 -- 172.21.15.32:0/1828018754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67c101d70 msgr2=0x7fe67c10ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:22.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.804+0000 7fe67a7fc700 1 --2- 172.21.15.32:0/1828018754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67c101d70 0x7fe67c10ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:22.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.804+0000 7fe67a7fc700 1 -- 172.21.15.32:0/1828018754 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe670005040 con 0x7fe67c102710 2024-04-12T05:45:22.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.804+0000 7fe67affd700 1 --2- 172.21.15.32:0/1828018754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67c101d70 0x7fe67c10ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:45:22.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.804+0000 7fe67a7fc700 1 --2- 172.21.15.32:0/1828018754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe67c102710 0x7fe67c10f130 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe66c00b790 tx=0x7fe66c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:22.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.804+0000 7fe6637fe700 1 -- 172.21.15.32:0/1828018754 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe66c00f6f0 con 0x7fe67c102710 2024-04-12T05:45:22.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.805+0000 7fe6816ab700 1 -- 172.21.15.32:0/1828018754 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe67c117b00 con 0x7fe67c102710 2024-04-12T05:45:22.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.805+0000 7fe6637fe700 1 -- 172.21.15.32:0/1828018754 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe66c00fd00 con 0x7fe67c102710 2024-04-12T05:45:22.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.805+0000 7fe6637fe700 1 -- 172.21.15.32:0/1828018754 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe66c0103a0 con 0x7fe67c102710 2024-04-12T05:45:22.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.805+0000 7fe6816ab700 1 -- 172.21.15.32:0/1828018754 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe67c1180c0 con 0x7fe67c102710 2024-04-12T05:45:22.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.806+0000 7fe6816ab700 1 -- 172.21.15.32:0/1828018754 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe67c0ff080 con 0x7fe67c102710 2024-04-12T05:45:22.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.806+0000 7fe6637fe700 1 -- 172.21.15.32:0/1828018754 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe66c00f850 con 0x7fe67c102710 2024-04-12T05:45:22.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.806+0000 7fe6637fe700 1 --2- 172.21.15.32:0/1828018754 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe66406cc20 0x7fe66406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:22.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.807+0000 7fe6637fe700 1 -- 172.21.15.32:0/1828018754 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fe66c08bef0 con 0x7fe67c102710 2024-04-12T05:45:22.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.807+0000 7fe67affd700 1 --2- 172.21.15.32:0/1828018754 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe66406cc20 0x7fe66406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:22.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.807+0000 7fe67affd700 1 --2- 172.21.15.32:0/1828018754 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe66406cc20 0x7fe66406f0d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe670004560 tx=0x7fe670009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:22.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:22.811+0000 7fe6637fe700 1 -- 172.21.15.32:0/1828018754 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe66c055f00 con 0x7fe67c102710 2024-04-12T05:45:23.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.033+0000 7fe6816ab700 1 -- 172.21.15.32:0/1828018754 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe67c065e40 con 0x7fe66406cc20 2024-04-12T05:45:23.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.034+0000 7fe6637fe700 1 -- 172.21.15.32:0/1828018754 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fe67c065e40 con 0x7fe66406cc20 2024-04-12T05:45:23.031 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:45:23.031 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:45:23.031 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:45:23.031 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:45:23.031 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:45:23.031 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:45:23.032 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:45:23.032 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-12T05:45:23.032 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:45:23.032 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "10/21 daemons upgraded", 2024-04-12T05:45:23.032 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading osd daemons", 2024-04-12T05:45:23.032 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:45:23.032 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:45:23.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.036+0000 7fe6816ab700 1 -- 172.21.15.32:0/1828018754 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe66406cc20 msgr2=0x7fe66406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:23.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.036+0000 7fe6816ab700 1 --2- 172.21.15.32:0/1828018754 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe66406cc20 0x7fe66406f0d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe670004560 tx=0x7fe670009e30 comp rx=0 tx=0).stop 2024-04-12T05:45:23.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.036+0000 7fe6816ab700 1 -- 172.21.15.32:0/1828018754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe67c102710 msgr2=0x7fe67c10f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:23.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.036+0000 7fe6816ab700 1 --2- 172.21.15.32:0/1828018754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe67c102710 0x7fe67c10f130 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe66c00b790 tx=0x7fe66c009220 comp rx=0 tx=0).stop 2024-04-12T05:45:23.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.036+0000 7fe6816ab700 1 -- 172.21.15.32:0/1828018754 shutdown_connections 2024-04-12T05:45:23.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.036+0000 7fe6816ab700 1 --2- 172.21.15.32:0/1828018754 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe66406cc20 0x7fe66406f0d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:23.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.036+0000 7fe6816ab700 1 --2- 172.21.15.32:0/1828018754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67c101d70 0x7fe67c10ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:23.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.036+0000 7fe6816ab700 1 --2- 172.21.15.32:0/1828018754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe67c102710 0x7fe67c10f130 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:23.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.036+0000 7fe6816ab700 1 -- 172.21.15.32:0/1828018754 >> 172.21.15.32:0/1828018754 conn(0x7fe67c072bc0 msgr2=0x7fe67c0faf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:23.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.036+0000 7fe6816ab700 1 -- 172.21.15.32:0/1828018754 shutdown_connections 2024-04-12T05:45:23.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.036+0000 7fe6816ab700 1 -- 172.21.15.32:0/1828018754 wait complete. 2024-04-12T05:45:23.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.140+0000 7fbe55ad2700 1 -- 172.21.15.32:0/2251607733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe500fe830 msgr2=0x7fbe50106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:23.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.140+0000 7fbe55ad2700 1 --2- 172.21.15.32:0/2251607733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe500fe830 0x7fbe50106f00 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbe44004940 tx=0x7fbe44009e30 comp rx=0 tx=0).stop 2024-04-12T05:45:23.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.140+0000 7fbe55ad2700 1 -- 172.21.15.32:0/2251607733 shutdown_connections 2024-04-12T05:45:23.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.140+0000 7fbe55ad2700 1 --2- 172.21.15.32:0/2251607733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe500fe830 0x7fbe50106f00 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:23.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.140+0000 7fbe55ad2700 1 --2- 172.21.15.32:0/2251607733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe500fde90 0x7fbe500fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:23.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.140+0000 7fbe55ad2700 1 -- 172.21.15.32:0/2251607733 >> 172.21.15.32:0/2251607733 conn(0x7fbe50072bc0 msgr2=0x7fbe50072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:23.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.140+0000 7fbe55ad2700 1 -- 172.21.15.32:0/2251607733 shutdown_connections 2024-04-12T05:45:23.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.140+0000 7fbe55ad2700 1 -- 172.21.15.32:0/2251607733 wait complete. 2024-04-12T05:45:23.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.140+0000 7fbe55ad2700 1 Processor -- start 2024-04-12T05:45:23.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.141+0000 7fbe55ad2700 1 -- start start 2024-04-12T05:45:23.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.141+0000 7fbe55ad2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe500fde90 0x7fbe50113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:23.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.141+0000 7fbe55ad2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe500fe830 0x7fbe50114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:23.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.141+0000 7fbe55ad2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe50114670 con 0x7fbe500fde90 2024-04-12T05:45:23.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.141+0000 7fbe55ad2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5010da60 con 0x7fbe500fe830 2024-04-12T05:45:23.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.141+0000 7fbe4f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe500fde90 0x7fbe50113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:23.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.141+0000 7fbe4f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe500fde90 0x7fbe50113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51176/0 (socket says 172.21.15.32:51176) 2024-04-12T05:45:23.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.141+0000 7fbe4f7fe700 1 -- 172.21.15.32:0/4282598063 learned_addr learned my addr 172.21.15.32:0/4282598063 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:45:23.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.141+0000 7fbe4effd700 1 --2- 172.21.15.32:0/4282598063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe500fe830 0x7fbe50114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:23.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.141+0000 7fbe4f7fe700 1 -- 172.21.15.32:0/4282598063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe500fe830 msgr2=0x7fbe50114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:23.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.141+0000 7fbe4f7fe700 1 --2- 172.21.15.32:0/4282598063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe500fe830 0x7fbe50114020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:23.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.141+0000 7fbe4f7fe700 1 -- 172.21.15.32:0/4282598063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe44005040 con 0x7fbe500fde90 2024-04-12T05:45:23.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.141+0000 7fbe4f7fe700 1 --2- 172.21.15.32:0/4282598063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe500fde90 0x7fbe50113ae0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbe40002a50 tx=0x7fbe400046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:23.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.142+0000 7fbe37fff700 1 -- 172.21.15.32:0/4282598063 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe40007cc0 con 0x7fbe500fde90 2024-04-12T05:45:23.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.142+0000 7fbe37fff700 1 -- 172.21.15.32:0/4282598063 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe4000f040 con 0x7fbe500fde90 2024-04-12T05:45:23.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.142+0000 7fbe37fff700 1 -- 172.21.15.32:0/4282598063 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe4000d9c0 con 0x7fbe500fde90 2024-04-12T05:45:23.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.142+0000 7fbe55ad2700 1 -- 172.21.15.32:0/4282598063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe5010dd40 con 0x7fbe500fde90 2024-04-12T05:45:23.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.142+0000 7fbe55ad2700 1 -- 172.21.15.32:0/4282598063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe5010e230 con 0x7fbe500fde90 2024-04-12T05:45:23.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.143+0000 7fbe55ad2700 1 -- 172.21.15.32:0/4282598063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe50065e30 con 0x7fbe500fde90 2024-04-12T05:45:23.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.143+0000 7fbe37fff700 1 -- 172.21.15.32:0/4282598063 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fbe4000db20 con 0x7fbe500fde90 2024-04-12T05:45:23.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.144+0000 7fbe37fff700 1 --2- 172.21.15.32:0/4282598063 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbe3806cc20 0x7fbe3806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:23.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.144+0000 7fbe37fff700 1 -- 172.21.15.32:0/4282598063 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6410+0+0 (secure 0 0 0) 0x7fbe4008bd20 con 0x7fbe500fde90 2024-04-12T05:45:23.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.144+0000 7fbe4effd700 1 --2- 172.21.15.32:0/4282598063 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbe3806cc20 0x7fbe3806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:23.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.144+0000 7fbe4effd700 1 --2- 172.21.15.32:0/4282598063 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbe3806cc20 0x7fbe3806f0d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbe5010ec20 tx=0x7fbe44009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:23.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.148+0000 7fbe37fff700 1 -- 172.21.15.32:0/4282598063 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbe400597c0 con 0x7fbe500fde90 2024-04-12T05:45:23.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.439+0000 7fbe55ad2700 1 -- 172.21.15.32:0/4282598063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fbe50109dd0 con 0x7fbe500fde90 2024-04-12T05:45:23.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.440+0000 7fbe37fff700 1 -- 172.21.15.32:0/4282598063 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+100 (secure 0 0 0) 0x7fbe40059350 con 0x7fbe500fde90 2024-04-12T05:45:23.437 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_WARN 1 osds down 2024-04-12T05:45:23.437 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-12T05:45:23.437 INFO:teuthology.orchestra.run.smithi032.stdout: osd.4 (root=default,host=smithi067) is down 2024-04-12T05:45:23.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.442+0000 7fbe55ad2700 1 -- 172.21.15.32:0/4282598063 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbe3806cc20 msgr2=0x7fbe3806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:23.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.442+0000 7fbe55ad2700 1 --2- 172.21.15.32:0/4282598063 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbe3806cc20 0x7fbe3806f0d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbe5010ec20 tx=0x7fbe44009e30 comp rx=0 tx=0).stop 2024-04-12T05:45:23.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.442+0000 7fbe55ad2700 1 -- 172.21.15.32:0/4282598063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe500fde90 msgr2=0x7fbe50113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:23.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.442+0000 7fbe55ad2700 1 --2- 172.21.15.32:0/4282598063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe500fde90 0x7fbe50113ae0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbe40002a50 tx=0x7fbe400046b0 comp rx=0 tx=0).stop 2024-04-12T05:45:23.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.442+0000 7fbe55ad2700 1 -- 172.21.15.32:0/4282598063 shutdown_connections 2024-04-12T05:45:23.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.442+0000 7fbe55ad2700 1 --2- 172.21.15.32:0/4282598063 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbe3806cc20 0x7fbe3806f0d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:23.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.442+0000 7fbe55ad2700 1 --2- 172.21.15.32:0/4282598063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe500fde90 0x7fbe50113ae0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:23.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.442+0000 7fbe55ad2700 1 --2- 172.21.15.32:0/4282598063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe500fe830 0x7fbe50114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:23.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.442+0000 7fbe55ad2700 1 -- 172.21.15.32:0/4282598063 >> 172.21.15.32:0/4282598063 conn(0x7fbe50072bc0 msgr2=0x7fbe50103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:23.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.442+0000 7fbe55ad2700 1 -- 172.21.15.32:0/4282598063 shutdown_connections 2024-04-12T05:45:23.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:23.442+0000 7fbe55ad2700 1 -- 172.21.15.32:0/4282598063 wait complete. 2024-04-12T05:45:23.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:23 smithi067 ceph-mon[112979]: pgmap v130: 65 pgs: 65 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T05:45:23.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:23 smithi067 ceph-mon[112979]: from='client.34318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:45:23.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:23 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1205559889' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:45:23.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:23 smithi067 ceph-mon[112979]: osd.4 marked itself down and dead 2024-04-12T05:45:23.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:23 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1132570022' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:45:23.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:23 smithi032 ceph-mon[146817]: pgmap v130: 65 pgs: 65 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T05:45:23.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:23 smithi032 ceph-mon[146817]: from='client.34318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:45:23.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:23 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1205559889' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:45:23.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:23 smithi032 ceph-mon[146817]: osd.4 marked itself down and dead 2024-04-12T05:45:23.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:23 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1132570022' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:45:24.166 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 05:45:23 smithi067 bash[130811]: ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-4 2024-04-12T05:45:24.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:24 smithi067 ceph-mon[112979]: from='client.34330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:45:24.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:24 smithi067 ceph-mon[112979]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T05:45:24.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:24 smithi067 ceph-mon[112979]: osdmap e70: 6 total, 5 up, 6 in 2024-04-12T05:45:24.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:24 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/4282598063' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:45:24.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:24 smithi032 ceph-mon[146817]: from='client.34330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:45:24.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:24 smithi032 ceph-mon[146817]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T05:45:24.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:24 smithi032 ceph-mon[146817]: osdmap e70: 6 total, 5 up, 6 in 2024-04-12T05:45:24.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:24 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/4282598063' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:45:25.372 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:25 smithi067 ceph-mon[112979]: pgmap v132: 65 pgs: 5 peering, 5 stale+active+clean, 55 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 3 op/s 2024-04-12T05:45:25.372 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:25 smithi067 ceph-mon[112979]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-12T05:45:25.372 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:25 smithi067 ceph-mon[112979]: osdmap e71: 6 total, 5 up, 6 in 2024-04-12T05:45:25.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:25 smithi032 ceph-mon[146817]: pgmap v132: 65 pgs: 5 peering, 5 stale+active+clean, 55 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 3 op/s 2024-04-12T05:45:25.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:25 smithi032 ceph-mon[146817]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-12T05:45:25.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:25 smithi032 ceph-mon[146817]: osdmap e71: 6 total, 5 up, 6 in 2024-04-12T05:45:26.018 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 05:45:25 smithi067 systemd[1]: ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@osd.4.service: Succeeded. 2024-04-12T05:45:26.018 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 05:45:25 smithi067 systemd[1]: Stopped Ceph osd.4 for 37b58ce2-f88d-11ee-bc8c-c7b262605968. 2024-04-12T05:45:26.018 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 05:45:25 smithi067 systemd[1]: Starting Ceph osd.4 for 37b58ce2-f88d-11ee-bc8c-c7b262605968... 2024-04-12T05:45:26.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:26 smithi067 ceph-mon[112979]: Health check failed: Degraded data redundancy: 10/333 objects degraded (3.003%), 4 pgs degraded (PG_DEGRADED) 2024-04-12T05:45:26.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:26 smithi032 ceph-mon[146817]: Health check failed: Degraded data redundancy: 10/333 objects degraded (3.003%), 4 pgs degraded (PG_DEGRADED) 2024-04-12T05:45:27.162 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 05:45:26 smithi067 bash[131085]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-12T05:45:27.162 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 05:45:26 smithi067 bash[131085]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-12T05:45:27.162 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 05:45:26 smithi067 bash[131085]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-12T05:45:27.163 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 05:45:26 smithi067 bash[131085]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-12T05:45:27.163 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 05:45:26 smithi067 bash[131085]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-04-12T05:45:27.163 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 05:45:26 smithi067 bash[131085]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-12T05:45:27.163 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 05:45:26 smithi067 bash[131085]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-12T05:45:27.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:27 smithi067 ceph-mon[112979]: pgmap v134: 65 pgs: 4 active+undersized+degraded, 13 peering, 1 stale+active+clean, 6 active+undersized, 41 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 10/333 objects degraded (3.003%) 2024-04-12T05:45:27.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:27 smithi032 ceph-mon[146817]: pgmap v134: 65 pgs: 4 active+undersized+degraded, 13 peering, 1 stale+active+clean, 6 active+undersized, 41 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 10/333 objects degraded (3.003%) 2024-04-12T05:45:28.267 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 05:45:27 smithi067 ceph-osd[131407]: -- 172.21.15.67:0/3634344707 <== mon.1 v2:172.21.15.67:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ca606f91e0 con 0x55ca606c8c00 2024-04-12T05:45:28.268 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 05:45:27 smithi067 bash[131369]: 30bb5e8d2751b4b706ed9cc70b17a6334427d0147aebdbf581fb1574d6a15bfd 2024-04-12T05:45:28.268 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 05:45:27 smithi067 systemd[1]: Started Ceph osd.4 for 37b58ce2-f88d-11ee-bc8c-c7b262605968. 2024-04-12T05:45:28.710 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 05:45:28 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-4[131384]: 2024-04-12T05:45:28.407+0000 7f154657b7c0 -1 Falling back to public interface 2024-04-12T05:45:29.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:28 smithi067 ceph-mon[112979]: pgmap v135: 65 pgs: 10 active+undersized+degraded, 13 peering, 11 active+undersized, 31 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 36/333 objects degraded (10.811%) 2024-04-12T05:45:29.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:28 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:29.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:28 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:29.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:28 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:45:29.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:28 smithi032 ceph-mon[146817]: pgmap v135: 65 pgs: 10 active+undersized+degraded, 13 peering, 11 active+undersized, 31 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 36/333 objects degraded (10.811%) 2024-04-12T05:45:29.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:28 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:29.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:28 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:29.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:28 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:45:30.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:30 smithi032 ceph-mon[146817]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-04-12T05:45:30.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:30 smithi067 ceph-mon[112979]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-04-12T05:45:31.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:31 smithi032 ceph-mon[146817]: pgmap v136: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 54/333 objects degraded (16.216%) 2024-04-12T05:45:31.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:31 smithi067 ceph-mon[112979]: pgmap v136: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 54/333 objects degraded (16.216%) 2024-04-12T05:45:32.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:32 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:32.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:32 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:32.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:32 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:32.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:32 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:32.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:32 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:32.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:32 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:32.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:32 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:32.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:32 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:33.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:33 smithi067 ceph-mon[112979]: pgmap v137: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 54/333 objects degraded (16.216%) 2024-04-12T05:45:33.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:33 smithi067 ceph-mon[112979]: Health check update: Degraded data redundancy: 54/333 objects degraded (16.216%), 16 pgs degraded (PG_DEGRADED) 2024-04-12T05:45:33.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:33 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:33.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:33 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:45:33.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:33 smithi032 ceph-mon[146817]: pgmap v137: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 54/333 objects degraded (16.216%) 2024-04-12T05:45:33.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:33 smithi032 ceph-mon[146817]: Health check update: Degraded data redundancy: 54/333 objects degraded (16.216%), 16 pgs degraded (PG_DEGRADED) 2024-04-12T05:45:33.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:33 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:33.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:33 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:45:33.638 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 05:45:33 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-4[131384]: 2024-04-12T05:45:33.460+0000 7f154657b7c0 -1 osd.4 69 log_to_monitors true 2024-04-12T05:45:34.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:34 smithi032 ceph-mon[146817]: from='osd.4 [v2:172.21.15.67:6808/452825880,v1:172.21.15.67:6809/452825880]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T05:45:34.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:34 smithi032 ceph-mon[146817]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T05:45:34.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:34 smithi067 ceph-mon[112979]: from='osd.4 [v2:172.21.15.67:6808/452825880,v1:172.21.15.67:6809/452825880]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T05:45:34.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:34 smithi067 ceph-mon[112979]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T05:45:34.518 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 05:45:34 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-4[131384]: 2024-04-12T05:45:34.101+0000 7f15379d5700 -1 osd.4 69 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T05:45:35.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:35 smithi067 ceph-mon[112979]: pgmap v138: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 54/333 objects degraded (16.216%) 2024-04-12T05:45:35.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:35 smithi067 ceph-mon[112979]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-12T05:45:35.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:35 smithi067 ceph-mon[112979]: osdmap e72: 6 total, 5 up, 6 in 2024-04-12T05:45:35.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:35 smithi067 ceph-mon[112979]: from='osd.4 [v2:172.21.15.67:6808/452825880,v1:172.21.15.67:6809/452825880]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-12T05:45:35.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:35 smithi067 ceph-mon[112979]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-12T05:45:35.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:35 smithi032 ceph-mon[146817]: pgmap v138: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 54/333 objects degraded (16.216%) 2024-04-12T05:45:35.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:35 smithi032 ceph-mon[146817]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-12T05:45:35.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:35 smithi032 ceph-mon[146817]: osdmap e72: 6 total, 5 up, 6 in 2024-04-12T05:45:35.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:35 smithi032 ceph-mon[146817]: from='osd.4 [v2:172.21.15.67:6808/452825880,v1:172.21.15.67:6809/452825880]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-12T05:45:35.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:35 smithi032 ceph-mon[146817]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-12T05:45:36.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:36 smithi067 ceph-mon[112979]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T05:45:36.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:36 smithi067 ceph-mon[112979]: osd.4 [v2:172.21.15.67:6808/452825880,v1:172.21.15.67:6809/452825880] boot 2024-04-12T05:45:36.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:36 smithi067 ceph-mon[112979]: osdmap e73: 6 total, 6 up, 6 in 2024-04-12T05:45:36.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:36 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:45:36.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:36 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:36.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:36 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:36.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:36 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:45:36.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:36 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:45:36.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:36 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:36.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:36 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:45:36.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:36 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:45:36.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:36 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:45:36.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:36 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:45:36.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:36 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T05:45:36.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:36 smithi067 ceph-mon[112979]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T05:45:36.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:36 smithi067 ceph-mon[112979]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-12T05:45:36.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:36 smithi067 ceph-mon[112979]: osdmap e74: 6 total, 6 up, 6 in 2024-04-12T05:45:36.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:36 smithi032 ceph-mon[146817]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T05:45:36.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:36 smithi032 ceph-mon[146817]: osd.4 [v2:172.21.15.67:6808/452825880,v1:172.21.15.67:6809/452825880] boot 2024-04-12T05:45:36.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:36 smithi032 ceph-mon[146817]: osdmap e73: 6 total, 6 up, 6 in 2024-04-12T05:45:36.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:36 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:45:36.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:36 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:36.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:36 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:36.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:36 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:45:36.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:36 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:45:36.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:36 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:36.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:36 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:45:36.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:36 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:45:36.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:36 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:45:36.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:36 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:45:36.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:36 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T05:45:36.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:36 smithi032 ceph-mon[146817]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T05:45:36.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:36 smithi032 ceph-mon[146817]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-12T05:45:36.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:36 smithi032 ceph-mon[146817]: osdmap e74: 6 total, 6 up, 6 in 2024-04-12T05:45:37.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:37 smithi067 ceph-mon[112979]: pgmap v141: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 4 op/s; 54/333 objects degraded (16.216%) 2024-04-12T05:45:37.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:37 smithi032 ceph-mon[146817]: pgmap v141: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 4 op/s; 54/333 objects degraded (16.216%) 2024-04-12T05:45:38.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:38 smithi067 ceph-mon[112979]: Health check update: Degraded data redundancy: 31/333 objects degraded (9.309%), 11 pgs degraded (PG_DEGRADED) 2024-04-12T05:45:38.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:38 smithi032 ceph-mon[146817]: Health check update: Degraded data redundancy: 31/333 objects degraded (9.309%), 11 pgs degraded (PG_DEGRADED) 2024-04-12T05:45:39.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:39 smithi067 ceph-mon[112979]: pgmap v143: 65 pgs: 10 peering, 11 active+undersized+degraded, 13 active+undersized, 31 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s; 31/333 objects degraded (9.309%) 2024-04-12T05:45:39.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:39 smithi032 ceph-mon[146817]: pgmap v143: 65 pgs: 10 peering, 11 active+undersized+degraded, 13 active+undersized, 31 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s; 31/333 objects degraded (9.309%) 2024-04-12T05:45:41.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:41 smithi067 ceph-mon[112979]: pgmap v144: 65 pgs: 10 peering, 4 active+undersized+degraded, 3 active+undersized, 48 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 3 op/s; 10/333 objects degraded (3.003%) 2024-04-12T05:45:41.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:41 smithi032 ceph-mon[146817]: pgmap v144: 65 pgs: 10 peering, 4 active+undersized+degraded, 3 active+undersized, 48 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 3 op/s; 10/333 objects degraded (3.003%) 2024-04-12T05:45:42.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:42 smithi032 ceph-mon[146817]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/333 objects degraded (3.003%), 4 pgs degraded) 2024-04-12T05:45:42.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:42 smithi032 ceph-mon[146817]: Cluster is now healthy 2024-04-12T05:45:43.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:42 smithi067 ceph-mon[112979]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/333 objects degraded (3.003%), 4 pgs degraded) 2024-04-12T05:45:43.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:42 smithi067 ceph-mon[112979]: Cluster is now healthy 2024-04-12T05:45:43.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:43 smithi032 ceph-mon[146817]: pgmap v145: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-12T05:45:44.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:43 smithi067 ceph-mon[112979]: pgmap v145: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-12T05:45:46.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:45 smithi067 ceph-mon[112979]: pgmap v146: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-12T05:45:46.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:45 smithi032 ceph-mon[146817]: pgmap v146: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-12T05:45:48.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:47 smithi067 ceph-mon[112979]: pgmap v147: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:45:48.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:47 smithi032 ceph-mon[146817]: pgmap v147: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:45:49.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:48 smithi067 ceph-mon[112979]: pgmap v148: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-12T05:45:49.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:48 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:49.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:48 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:45:49.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:48 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:49.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:48 smithi032 ceph-mon[146817]: pgmap v148: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-12T05:45:49.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:48 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:49.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:48 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:45:49.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:48 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:51.101 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:50 smithi067 ceph-mon[112979]: pgmap v149: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-12T05:45:51.101 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:50 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T05:45:51.101 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:50 smithi067 ceph-mon[112979]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T05:45:51.101 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:50 smithi067 ceph-mon[112979]: Upgrade: osd.5 is safe to restart 2024-04-12T05:45:51.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:50 smithi032 ceph-mon[146817]: pgmap v149: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-12T05:45:51.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:50 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T05:45:51.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:50 smithi032 ceph-mon[146817]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T05:45:51.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:50 smithi032 ceph-mon[146817]: Upgrade: osd.5 is safe to restart 2024-04-12T05:45:53.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:53 smithi032 ceph-mon[146817]: pgmap v150: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T05:45:53.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:53 smithi032 ceph-mon[146817]: Upgrade: Updating osd.5 2024-04-12T05:45:53.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:53 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:53.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:53 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-12T05:45:53.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:53 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:45:53.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:53 smithi032 ceph-mon[146817]: Deploying daemon osd.5 on smithi067 2024-04-12T05:45:53.390 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:53 smithi067 ceph-mon[112979]: pgmap v150: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T05:45:53.390 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:53 smithi067 ceph-mon[112979]: Upgrade: Updating osd.5 2024-04-12T05:45:53.390 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:53 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:45:53.390 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:53 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-12T05:45:53.390 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:53 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:45:53.390 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:53 smithi067 ceph-mon[112979]: Deploying daemon osd.5 on smithi067 2024-04-12T05:45:53.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.562+0000 7f70eb1e6700 1 -- 172.21.15.32:0/1677433108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70e41037c0 msgr2=0x7f70e40fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:53.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.562+0000 7f70eb1e6700 1 --2- 172.21.15.32:0/1677433108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70e41037c0 0x7f70e40fce30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f70d0004940 tx=0x7f70d0009e30 comp rx=0 tx=0).stop 2024-04-12T05:45:53.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.562+0000 7f70eb1e6700 1 -- 172.21.15.32:0/1677433108 shutdown_connections 2024-04-12T05:45:53.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.562+0000 7f70eb1e6700 1 --2- 172.21.15.32:0/1677433108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70e41037c0 0x7f70e40fce30 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:53.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.562+0000 7f70eb1e6700 1 --2- 172.21.15.32:0/1677433108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70e4102e20 0x7f70e41031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:53.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.562+0000 7f70eb1e6700 1 -- 172.21.15.32:0/1677433108 >> 172.21.15.32:0/1677433108 conn(0x7f70e40f89d0 msgr2=0x7f70e40fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:53.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.562+0000 7f70eb1e6700 1 -- 172.21.15.32:0/1677433108 shutdown_connections 2024-04-12T05:45:53.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.562+0000 7f70eb1e6700 1 -- 172.21.15.32:0/1677433108 wait complete. 2024-04-12T05:45:53.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.562+0000 7f70eb1e6700 1 Processor -- start 2024-04-12T05:45:53.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.563+0000 7f70eb1e6700 1 -- start start 2024-04-12T05:45:53.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.563+0000 7f70eb1e6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70e4102e20 0x7f70e4113a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:53.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.563+0000 7f70eb1e6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70e41037c0 0x7f70e4113f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:53.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.563+0000 7f70eb1e6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e41145c0 con 0x7f70e41037c0 2024-04-12T05:45:53.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.563+0000 7f70eb1e6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e410d9b0 con 0x7f70e4102e20 2024-04-12T05:45:53.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.563+0000 7f70e3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70e41037c0 0x7f70e4113f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:53.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.563+0000 7f70e3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70e41037c0 0x7f70e4113f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56174/0 (socket says 172.21.15.32:56174) 2024-04-12T05:45:53.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.563+0000 7f70e3fff700 1 -- 172.21.15.32:0/1544679458 learned_addr learned my addr 172.21.15.32:0/1544679458 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:45:53.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.563+0000 7f70e8f82700 1 --2- 172.21.15.32:0/1544679458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70e4102e20 0x7f70e4113a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:53.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.563+0000 7f70e3fff700 1 -- 172.21.15.32:0/1544679458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70e4102e20 msgr2=0x7f70e4113a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:53.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.564+0000 7f70e3fff700 1 --2- 172.21.15.32:0/1544679458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70e4102e20 0x7f70e4113a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:53.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.564+0000 7f70e3fff700 1 -- 172.21.15.32:0/1544679458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d8005040 con 0x7f70e41037c0 2024-04-12T05:45:53.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.564+0000 7f70e8f82700 1 --2- 172.21.15.32:0/1544679458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70e4102e20 0x7f70e4113a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T05:45:53.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.564+0000 7f70e3fff700 1 --2- 172.21.15.32:0/1544679458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70e41037c0 0x7f70e4113f70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f70d0000c00 tx=0x7f70d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:53.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.564+0000 7f70e17fa700 1 -- 172.21.15.32:0/1544679458 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70d0021070 con 0x7f70e41037c0 2024-04-12T05:45:53.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.564+0000 7f70eb1e6700 1 -- 172.21.15.32:0/1544679458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70d0005040 con 0x7f70e41037c0 2024-04-12T05:45:53.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.564+0000 7f70e17fa700 1 -- 172.21.15.32:0/1544679458 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f70d0013de0 con 0x7f70e41037c0 2024-04-12T05:45:53.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.564+0000 7f70e17fa700 1 -- 172.21.15.32:0/1544679458 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70d000e070 con 0x7f70e41037c0 2024-04-12T05:45:53.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.564+0000 7f70eb1e6700 1 -- 172.21.15.32:0/1544679458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70e410e000 con 0x7f70e41037c0 2024-04-12T05:45:53.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.565+0000 7f70eb1e6700 1 -- 172.21.15.32:0/1544679458 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70e40fe570 con 0x7f70e41037c0 2024-04-12T05:45:53.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.566+0000 7f70e17fa700 1 -- 172.21.15.32:0/1544679458 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f70d001c600 con 0x7f70e41037c0 2024-04-12T05:45:53.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.566+0000 7f70e17fa700 1 --2- 172.21.15.32:0/1544679458 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f70d406cc20 0x7f70d406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:53.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.566+0000 7f70e17fa700 1 -- 172.21.15.32:0/1544679458 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f70d008d040 con 0x7f70e41037c0 2024-04-12T05:45:53.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.566+0000 7f70e8f82700 1 --2- 172.21.15.32:0/1544679458 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f70d406cc20 0x7f70d406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:53.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.567+0000 7f70e8f82700 1 --2- 172.21.15.32:0/1544679458 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f70d406cc20 0x7f70d406f0d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f70d8009fd0 tx=0x7f70d800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:53.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.571+0000 7f70e17fa700 1 -- 172.21.15.32:0/1544679458 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f70d005ab60 con 0x7f70e41037c0 2024-04-12T05:45:53.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.791+0000 7f70eb1e6700 1 -- 172.21.15.32:0/1544679458 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f70e4109dc0 con 0x7f70d406cc20 2024-04-12T05:45:53.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.792+0000 7f70e17fa700 1 -- 172.21.15.32:0/1544679458 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f70e4109dc0 con 0x7f70d406cc20 2024-04-12T05:45:53.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.795+0000 7f70eb1e6700 1 -- 172.21.15.32:0/1544679458 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f70d406cc20 msgr2=0x7f70d406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:53.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.795+0000 7f70eb1e6700 1 --2- 172.21.15.32:0/1544679458 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f70d406cc20 0x7f70d406f0d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f70d8009fd0 tx=0x7f70d800b040 comp rx=0 tx=0).stop 2024-04-12T05:45:53.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.795+0000 7f70eb1e6700 1 -- 172.21.15.32:0/1544679458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70e41037c0 msgr2=0x7f70e4113f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:53.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.795+0000 7f70eb1e6700 1 --2- 172.21.15.32:0/1544679458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70e41037c0 0x7f70e4113f70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f70d0000c00 tx=0x7f70d0009e30 comp rx=0 tx=0).stop 2024-04-12T05:45:53.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.795+0000 7f70eb1e6700 1 -- 172.21.15.32:0/1544679458 shutdown_connections 2024-04-12T05:45:53.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.795+0000 7f70eb1e6700 1 --2- 172.21.15.32:0/1544679458 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f70d406cc20 0x7f70d406f0d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:53.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.795+0000 7f70eb1e6700 1 --2- 172.21.15.32:0/1544679458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70e4102e20 0x7f70e4113a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:53.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.795+0000 7f70eb1e6700 1 --2- 172.21.15.32:0/1544679458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70e41037c0 0x7f70e4113f70 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:53.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.795+0000 7f70eb1e6700 1 -- 172.21.15.32:0/1544679458 >> 172.21.15.32:0/1544679458 conn(0x7f70e40f89d0 msgr2=0x7f70e40f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:53.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.795+0000 7f70eb1e6700 1 -- 172.21.15.32:0/1544679458 shutdown_connections 2024-04-12T05:45:53.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.795+0000 7f70eb1e6700 1 -- 172.21.15.32:0/1544679458 wait complete. 2024-04-12T05:45:53.802 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:45:53.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.902+0000 7f1e0c4fd700 1 -- 172.21.15.32:0/1430116670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e04102e10 msgr2=0x7f1e041031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:53.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.902+0000 7f1e0c4fd700 1 --2- 172.21.15.32:0/1430116670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e04102e10 0x7f1e041031e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1e00004940 tx=0x7f1e00009e30 comp rx=0 tx=0).stop 2024-04-12T05:45:53.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.902+0000 7f1e0c4fd700 1 -- 172.21.15.32:0/1430116670 shutdown_connections 2024-04-12T05:45:53.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.902+0000 7f1e0c4fd700 1 --2- 172.21.15.32:0/1430116670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e041037b0 0x7f1e040fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:53.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.902+0000 7f1e0c4fd700 1 --2- 172.21.15.32:0/1430116670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e04102e10 0x7f1e041031e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:53.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.902+0000 7f1e0c4fd700 1 -- 172.21.15.32:0/1430116670 >> 172.21.15.32:0/1430116670 conn(0x7f1e040f89a0 msgr2=0x7f1e040fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:53.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.902+0000 7f1e0c4fd700 1 -- 172.21.15.32:0/1430116670 shutdown_connections 2024-04-12T05:45:53.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.902+0000 7f1e0c4fd700 1 -- 172.21.15.32:0/1430116670 wait complete. 2024-04-12T05:45:53.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.903+0000 7f1e0c4fd700 1 Processor -- start 2024-04-12T05:45:53.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.903+0000 7f1e0c4fd700 1 -- start start 2024-04-12T05:45:53.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.903+0000 7f1e0c4fd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e04102e10 0x7f1e040702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:53.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.903+0000 7f1e0c4fd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e041037b0 0x7f1e04070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:53.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.903+0000 7f1e0c4fd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e04070d90 con 0x7f1e041037b0 2024-04-12T05:45:53.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.903+0000 7f1e0c4fd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e04070ed0 con 0x7f1e04102e10 2024-04-12T05:45:53.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.903+0000 7f1e09a98700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e041037b0 0x7f1e04070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:53.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.903+0000 7f1e0a299700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e04102e10 0x7f1e040702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:53.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.903+0000 7f1e09a98700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e041037b0 0x7f1e04070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56194/0 (socket says 172.21.15.32:56194) 2024-04-12T05:45:53.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.903+0000 7f1e0a299700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e04102e10 0x7f1e040702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.32:36248/0 (socket says 172.21.15.32:36248) 2024-04-12T05:45:53.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.903+0000 7f1e09a98700 1 -- 172.21.15.32:0/1408287112 learned_addr learned my addr 172.21.15.32:0/1408287112 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:45:53.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.904+0000 7f1e09a98700 1 -- 172.21.15.32:0/1408287112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e04102e10 msgr2=0x7f1e040702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:53.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.904+0000 7f1e09a98700 1 --2- 172.21.15.32:0/1408287112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e04102e10 0x7f1e040702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:53.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.904+0000 7f1e09a98700 1 -- 172.21.15.32:0/1408287112 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e00005040 con 0x7f1e041037b0 2024-04-12T05:45:53.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.904+0000 7f1e09a98700 1 --2- 172.21.15.32:0/1408287112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e041037b0 0x7f1e04070800 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1df800eb40 tx=0x7f1df800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:53.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.904+0000 7f1df6ffd700 1 -- 172.21.15.32:0/1408287112 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1df8007cc0 con 0x7f1e041037b0 2024-04-12T05:45:53.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.904+0000 7f1df6ffd700 1 -- 172.21.15.32:0/1408287112 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1df800f040 con 0x7f1e041037b0 2024-04-12T05:45:53.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.904+0000 7f1e0c4fd700 1 -- 172.21.15.32:0/1408287112 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e04071120 con 0x7f1e041037b0 2024-04-12T05:45:53.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.904+0000 7f1df6ffd700 1 -- 172.21.15.32:0/1408287112 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1df800c8f0 con 0x7f1e041037b0 2024-04-12T05:45:53.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.904+0000 7f1e0c4fd700 1 -- 172.21.15.32:0/1408287112 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e0411a180 con 0x7f1e041037b0 2024-04-12T05:45:53.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.905+0000 7f1e0c4fd700 1 -- 172.21.15.32:0/1408287112 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e040fe560 con 0x7f1e041037b0 2024-04-12T05:45:53.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.906+0000 7f1df6ffd700 1 -- 172.21.15.32:0/1408287112 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1df800cb60 con 0x7f1e041037b0 2024-04-12T05:45:53.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.906+0000 7f1df6ffd700 1 --2- 172.21.15.32:0/1408287112 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1df006cc20 0x7f1df006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:53.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.906+0000 7f1df6ffd700 1 -- 172.21.15.32:0/1408287112 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f1df808ba80 con 0x7f1e041037b0 2024-04-12T05:45:53.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.906+0000 7f1e0a299700 1 --2- 172.21.15.32:0/1408287112 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1df006cc20 0x7f1df006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:53.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.906+0000 7f1e0a299700 1 --2- 172.21.15.32:0/1408287112 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1df006cc20 0x7f1df006f0d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1e0000b0d0 tx=0x7f1e00009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:53.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:53.911+0000 7f1df6ffd700 1 -- 172.21.15.32:0/1408287112 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1df8055b40 con 0x7f1e041037b0 2024-04-12T05:45:54.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.134+0000 7f1e0c4fd700 1 -- 172.21.15.32:0/1408287112 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e0404bed0 con 0x7f1df006cc20 2024-04-12T05:45:54.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.135+0000 7f1df6ffd700 1 -- 172.21.15.32:0/1408287112 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f1e0404bed0 con 0x7f1df006cc20 2024-04-12T05:45:54.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.137+0000 7f1e0c4fd700 1 -- 172.21.15.32:0/1408287112 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1df006cc20 msgr2=0x7f1df006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:54.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.137+0000 7f1e0c4fd700 1 --2- 172.21.15.32:0/1408287112 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1df006cc20 0x7f1df006f0d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1e0000b0d0 tx=0x7f1e00009e30 comp rx=0 tx=0).stop 2024-04-12T05:45:54.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.137+0000 7f1e0c4fd700 1 -- 172.21.15.32:0/1408287112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e041037b0 msgr2=0x7f1e04070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:54.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.137+0000 7f1e0c4fd700 1 --2- 172.21.15.32:0/1408287112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e041037b0 0x7f1e04070800 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1df800eb40 tx=0x7f1df800b040 comp rx=0 tx=0).stop 2024-04-12T05:45:54.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.137+0000 7f1e0c4fd700 1 -- 172.21.15.32:0/1408287112 shutdown_connections 2024-04-12T05:45:54.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.137+0000 7f1e0c4fd700 1 --2- 172.21.15.32:0/1408287112 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1df006cc20 0x7f1df006f0d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:54.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.137+0000 7f1e0c4fd700 1 --2- 172.21.15.32:0/1408287112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e04102e10 0x7f1e040702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:54.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.137+0000 7f1e0c4fd700 1 --2- 172.21.15.32:0/1408287112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e041037b0 0x7f1e04070800 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:54.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.137+0000 7f1e0c4fd700 1 -- 172.21.15.32:0/1408287112 >> 172.21.15.32:0/1408287112 conn(0x7f1e040f89a0 msgr2=0x7f1e040f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:54.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.137+0000 7f1e0c4fd700 1 -- 172.21.15.32:0/1408287112 shutdown_connections 2024-04-12T05:45:54.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.137+0000 7f1e0c4fd700 1 -- 172.21.15.32:0/1408287112 wait complete. 2024-04-12T05:45:54.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.241+0000 7f763accf700 1 -- 172.21.15.32:0/3979250854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7634101d70 msgr2=0x7f7634102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:54.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.241+0000 7f763accf700 1 --2- 172.21.15.32:0/3979250854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7634101d70 0x7f7634102140 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7628004940 tx=0x7f7628009e30 comp rx=0 tx=0).stop 2024-04-12T05:45:54.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.241+0000 7f763accf700 1 -- 172.21.15.32:0/3979250854 shutdown_connections 2024-04-12T05:45:54.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.241+0000 7f763accf700 1 --2- 172.21.15.32:0/3979250854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7634102710 0x7f76340fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:54.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.241+0000 7f763accf700 1 --2- 172.21.15.32:0/3979250854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7634101d70 0x7f7634102140 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:54.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.241+0000 7f763accf700 1 -- 172.21.15.32:0/3979250854 >> 172.21.15.32:0/3979250854 conn(0x7f7634072bc0 msgr2=0x7f7634072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:54.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.241+0000 7f763accf700 1 -- 172.21.15.32:0/3979250854 shutdown_connections 2024-04-12T05:45:54.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.241+0000 7f763accf700 1 -- 172.21.15.32:0/3979250854 wait complete. 2024-04-12T05:45:54.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.241+0000 7f763accf700 1 Processor -- start 2024-04-12T05:45:54.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.242+0000 7f763accf700 1 -- start start 2024-04-12T05:45:54.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.242+0000 7f763accf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7634101d70 0x7f7634113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:54.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.242+0000 7f763accf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7634102710 0x7f7634114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:54.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.242+0000 7f763accf700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7634114720 con 0x7f7634101d70 2024-04-12T05:45:54.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.242+0000 7f763accf700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f763410da80 con 0x7f7634102710 2024-04-12T05:45:54.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.242+0000 7f7638a6b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7634101d70 0x7f7634113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:54.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.242+0000 7f7638a6b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7634101d70 0x7f7634113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56212/0 (socket says 172.21.15.32:56212) 2024-04-12T05:45:54.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.242+0000 7f7638a6b700 1 -- 172.21.15.32:0/1259904060 learned_addr learned my addr 172.21.15.32:0/1259904060 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:45:54.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.242+0000 7f7633fff700 1 --2- 172.21.15.32:0/1259904060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7634102710 0x7f7634114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:54.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.242+0000 7f7638a6b700 1 -- 172.21.15.32:0/1259904060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7634102710 msgr2=0x7f7634114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:54.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.242+0000 7f7638a6b700 1 --2- 172.21.15.32:0/1259904060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7634102710 0x7f7634114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:54.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.242+0000 7f7638a6b700 1 -- 172.21.15.32:0/1259904060 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7628005040 con 0x7f7634101d70 2024-04-12T05:45:54.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.242+0000 7f7633fff700 1 --2- 172.21.15.32:0/1259904060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7634102710 0x7f7634114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:45:54.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.242+0000 7f7638a6b700 1 --2- 172.21.15.32:0/1259904060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7634101d70 0x7f7634113b00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f762800b3b0 tx=0x7f7628009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:54.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.243+0000 7f76317fa700 1 -- 172.21.15.32:0/1259904060 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f762800e040 con 0x7f7634101d70 2024-04-12T05:45:54.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.243+0000 7f76317fa700 1 -- 172.21.15.32:0/1259904060 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7628013420 con 0x7f7634101d70 2024-04-12T05:45:54.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.243+0000 7f76317fa700 1 -- 172.21.15.32:0/1259904060 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f762801c630 con 0x7f7634101d70 2024-04-12T05:45:54.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.243+0000 7f763accf700 1 -- 172.21.15.32:0/1259904060 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f763410dd00 con 0x7f7634101d70 2024-04-12T05:45:54.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.243+0000 7f763accf700 1 -- 172.21.15.32:0/1259904060 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f763410e240 con 0x7f7634101d70 2024-04-12T05:45:54.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.244+0000 7f763accf700 1 -- 172.21.15.32:0/1259904060 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76340fd4c0 con 0x7f7634101d70 2024-04-12T05:45:54.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.244+0000 7f76317fa700 1 -- 172.21.15.32:0/1259904060 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f7628013970 con 0x7f7634101d70 2024-04-12T05:45:54.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.245+0000 7f76317fa700 1 --2- 172.21.15.32:0/1259904060 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f762406cc20 0x7f762406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:54.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.245+0000 7f76317fa700 1 -- 172.21.15.32:0/1259904060 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f762808d150 con 0x7f7634101d70 2024-04-12T05:45:54.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.245+0000 7f7633fff700 1 --2- 172.21.15.32:0/1259904060 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f762406cc20 0x7f762406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:54.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.245+0000 7f7633fff700 1 --2- 172.21.15.32:0/1259904060 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f762406cc20 0x7f762406f0d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f763410ec40 tx=0x7f7620008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:54.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.249+0000 7f76317fa700 1 -- 172.21.15.32:0/1259904060 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f762805ad10 con 0x7f7634101d70 2024-04-12T05:45:54.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.460+0000 7f763accf700 1 -- 172.21.15.32:0/1259904060 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7634108e10 con 0x7f762406cc20 2024-04-12T05:45:54.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.470+0000 7f76317fa700 1 -- 172.21.15.32:0/1259904060 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f7634108e10 con 0x7f762406cc20 2024-04-12T05:45:54.467 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:45:54.467 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (5m) 89s ago 17m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:45:54.467 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (3m) 89s ago 17m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:45:54.467 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (3m) 23s ago 16m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:45:54.467 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (5m) 89s ago 16m 84.7M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:45:54.467 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (11m) 89s ago 11m 64.8M - 17.2.7-827-gd3b5060a 21de9b7369ad e7ada8e5259c 2024-04-12T05:45:54.467 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (11m) 89s ago 11m 115M - 17.2.7-827-gd3b5060a 21de9b7369ad 55124da9afdb 2024-04-12T05:45:54.468 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (11m) 23s ago 11m 62.6M - 17.2.7-827-gd3b5060a 21de9b7369ad 0c295d31a4a6 2024-04-12T05:45:54.468 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (11m) 23s ago 11m 120M - 17.2.7-827-gd3b5060a 21de9b7369ad 6eecd7deaac2 2024-04-12T05:45:54.468 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (7m) 89s ago 18m 499M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:45:54.468 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (6m) 23s ago 16m 449M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:45:54.468 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (4m) 89s ago 18m 46.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:45:54.468 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (3m) 23s ago 16m 37.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:45:54.468 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (6m) 89s ago 17m 16.6M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:45:54.468 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (6m) 23s ago 16m 18.1M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:45:54.468 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (2m) 89s ago 15m 86.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T05:45:54.468 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (2m) 89s ago 14m 94.5M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T05:45:54.468 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (95s) 89s ago 14m 11.2M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T05:45:54.468 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (59s) 23s ago 13m 110M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T05:45:54.468 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (26s) 23s ago 13m 10.6M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T05:45:54.468 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (13m) 23s ago 13m 220M 2592M 17.2.7-827-gd3b5060a 21de9b7369ad e54839bb6055 2024-04-12T05:45:54.469 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (5m) 89s ago 16m 63.5M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:45:54.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.472+0000 7f763accf700 1 -- 172.21.15.32:0/1259904060 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f762406cc20 msgr2=0x7f762406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:54.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.472+0000 7f763accf700 1 --2- 172.21.15.32:0/1259904060 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f762406cc20 0x7f762406f0d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f763410ec40 tx=0x7f7620008040 comp rx=0 tx=0).stop 2024-04-12T05:45:54.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.472+0000 7f763accf700 1 -- 172.21.15.32:0/1259904060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7634101d70 msgr2=0x7f7634113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:54.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.472+0000 7f763accf700 1 --2- 172.21.15.32:0/1259904060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7634101d70 0x7f7634113b00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f762800b3b0 tx=0x7f7628009e30 comp rx=0 tx=0).stop 2024-04-12T05:45:54.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.472+0000 7f763accf700 1 -- 172.21.15.32:0/1259904060 shutdown_connections 2024-04-12T05:45:54.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.472+0000 7f763accf700 1 --2- 172.21.15.32:0/1259904060 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f762406cc20 0x7f762406f0d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:54.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.472+0000 7f763accf700 1 --2- 172.21.15.32:0/1259904060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7634101d70 0x7f7634113b00 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:54.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.472+0000 7f763accf700 1 --2- 172.21.15.32:0/1259904060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7634102710 0x7f7634114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:54.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.472+0000 7f763accf700 1 -- 172.21.15.32:0/1259904060 >> 172.21.15.32:0/1259904060 conn(0x7f7634072bc0 msgr2=0x7f76340658e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:54.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.472+0000 7f763accf700 1 -- 172.21.15.32:0/1259904060 shutdown_connections 2024-04-12T05:45:54.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.472+0000 7f763accf700 1 -- 172.21.15.32:0/1259904060 wait complete. 2024-04-12T05:45:54.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.574+0000 7f32fac75700 1 -- 172.21.15.32:0/3325866233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32f4102da0 msgr2=0x7f32f4103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:54.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.574+0000 7f32fac75700 1 --2- 172.21.15.32:0/3325866233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32f4102da0 0x7f32f4103170 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f32e8004940 tx=0x7f32e8009e30 comp rx=0 tx=0).stop 2024-04-12T05:45:54.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.574+0000 7f32fac75700 1 -- 172.21.15.32:0/3325866233 shutdown_connections 2024-04-12T05:45:54.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.574+0000 7f32fac75700 1 --2- 172.21.15.32:0/3325866233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32f4103740 0x7f32f40fcdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:54.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.574+0000 7f32fac75700 1 --2- 172.21.15.32:0/3325866233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32f4102da0 0x7f32f4103170 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:54.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.574+0000 7f32fac75700 1 -- 172.21.15.32:0/3325866233 >> 172.21.15.32:0/3325866233 conn(0x7f32f40f8970 msgr2=0x7f32f40fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:54.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.574+0000 7f32fac75700 1 -- 172.21.15.32:0/3325866233 shutdown_connections 2024-04-12T05:45:54.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.574+0000 7f32fac75700 1 -- 172.21.15.32:0/3325866233 wait complete. 2024-04-12T05:45:54.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.575+0000 7f32fac75700 1 Processor -- start 2024-04-12T05:45:54.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.575+0000 7f32fac75700 1 -- start start 2024-04-12T05:45:54.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.575+0000 7f32fac75700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32f4102da0 0x7f32f4113a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:54.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.575+0000 7f32fac75700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32f4103740 0x7f32f4113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:54.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.575+0000 7f32fac75700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f41145b0 con 0x7f32f4102da0 2024-04-12T05:45:54.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.575+0000 7f32fac75700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f410d9a0 con 0x7f32f4103740 2024-04-12T05:45:54.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.575+0000 7f32f8a11700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32f4102da0 0x7f32f4113a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:54.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.575+0000 7f32f8a11700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32f4102da0 0x7f32f4113a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56226/0 (socket says 172.21.15.32:56226) 2024-04-12T05:45:54.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.575+0000 7f32f8a11700 1 -- 172.21.15.32:0/64240943 learned_addr learned my addr 172.21.15.32:0/64240943 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:45:54.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.575+0000 7f32f3fff700 1 --2- 172.21.15.32:0/64240943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32f4103740 0x7f32f4113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:54.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.576+0000 7f32f8a11700 1 -- 172.21.15.32:0/64240943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32f4103740 msgr2=0x7f32f4113f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:54.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.576+0000 7f32f8a11700 1 --2- 172.21.15.32:0/64240943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32f4103740 0x7f32f4113f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:54.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.576+0000 7f32f8a11700 1 -- 172.21.15.32:0/64240943 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e8005040 con 0x7f32f4102da0 2024-04-12T05:45:54.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.576+0000 7f32f3fff700 1 --2- 172.21.15.32:0/64240943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32f4103740 0x7f32f4113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T05:45:54.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.576+0000 7f32f8a11700 1 --2- 172.21.15.32:0/64240943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32f4102da0 0x7f32f4113a20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f32e8008d80 tx=0x7f32e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:54.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.576+0000 7f32f17fa700 1 -- 172.21.15.32:0/64240943 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32e800e030 con 0x7f32f4102da0 2024-04-12T05:45:54.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.576+0000 7f32f17fa700 1 -- 172.21.15.32:0/64240943 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f32e8013de0 con 0x7f32f4102da0 2024-04-12T05:45:54.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.576+0000 7f32fac75700 1 -- 172.21.15.32:0/64240943 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f410dc20 con 0x7f32f4102da0 2024-04-12T05:45:54.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.576+0000 7f32f17fa700 1 -- 172.21.15.32:0/64240943 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32e801cd00 con 0x7f32f4102da0 2024-04-12T05:45:54.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.576+0000 7f32fac75700 1 -- 172.21.15.32:0/64240943 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f410e1e0 con 0x7f32f4102da0 2024-04-12T05:45:54.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.577+0000 7f32fac75700 1 -- 172.21.15.32:0/64240943 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32f40fe4f0 con 0x7f32f4102da0 2024-04-12T05:45:54.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.578+0000 7f32f17fa700 1 -- 172.21.15.32:0/64240943 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f32e8013930 con 0x7f32f4102da0 2024-04-12T05:45:54.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.578+0000 7f32f17fa700 1 --2- 172.21.15.32:0/64240943 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f32e406cc30 0x7f32e406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:54.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.578+0000 7f32f17fa700 1 -- 172.21.15.32:0/64240943 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f32e808d790 con 0x7f32f4102da0 2024-04-12T05:45:54.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.578+0000 7f32f3fff700 1 --2- 172.21.15.32:0/64240943 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f32e406cc30 0x7f32e406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:54.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.578+0000 7f32f3fff700 1 --2- 172.21.15.32:0/64240943 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f32e406cc30 0x7f32e406f0e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f32f410eb60 tx=0x7f32e000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:54.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.582+0000 7f32f17fa700 1 -- 172.21.15.32:0/64240943 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f32e805b350 con 0x7f32f4102da0 2024-04-12T05:45:54.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.872+0000 7f32fac75700 1 -- 172.21.15.32:0/64240943 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f32f404bed0 con 0x7f32f4102da0 2024-04-12T05:45:54.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.872+0000 7f32f17fa700 1 -- 172.21.15.32:0/64240943 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f32e801ce60 con 0x7f32f4102da0 2024-04-12T05:45:54.869 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:45:54.869 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:45:54.869 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:45:54.869 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:45:54.869 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:45:54.869 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:45:54.869 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:45:54.870 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:45:54.870 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 1, 2024-04-12T05:45:54.870 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 5 2024-04-12T05:45:54.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:45:54.870 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:45:54.870 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 4 2024-04-12T05:45:54.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:45:54.870 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:45:54.870 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 5, 2024-04-12T05:45:54.870 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 9 2024-04-12T05:45:54.870 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:45:54.870 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:45:54.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.875+0000 7f32fac75700 1 -- 172.21.15.32:0/64240943 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f32e406cc30 msgr2=0x7f32e406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:54.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.875+0000 7f32fac75700 1 --2- 172.21.15.32:0/64240943 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f32e406cc30 0x7f32e406f0e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f32f410eb60 tx=0x7f32e000b040 comp rx=0 tx=0).stop 2024-04-12T05:45:54.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.875+0000 7f32fac75700 1 -- 172.21.15.32:0/64240943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32f4102da0 msgr2=0x7f32f4113a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:54.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.875+0000 7f32fac75700 1 --2- 172.21.15.32:0/64240943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32f4102da0 0x7f32f4113a20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f32e8008d80 tx=0x7f32e8009e30 comp rx=0 tx=0).stop 2024-04-12T05:45:54.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.875+0000 7f32fac75700 1 -- 172.21.15.32:0/64240943 shutdown_connections 2024-04-12T05:45:54.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.875+0000 7f32fac75700 1 --2- 172.21.15.32:0/64240943 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f32e406cc30 0x7f32e406f0e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:54.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.875+0000 7f32fac75700 1 --2- 172.21.15.32:0/64240943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32f4102da0 0x7f32f4113a20 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:54.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.875+0000 7f32fac75700 1 --2- 172.21.15.32:0/64240943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32f4103740 0x7f32f4113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:54.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.875+0000 7f32fac75700 1 -- 172.21.15.32:0/64240943 >> 172.21.15.32:0/64240943 conn(0x7f32f40f8970 msgr2=0x7f32f40f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:54.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.875+0000 7f32fac75700 1 -- 172.21.15.32:0/64240943 shutdown_connections 2024-04-12T05:45:54.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.875+0000 7f32fac75700 1 -- 172.21.15.32:0/64240943 wait complete. 2024-04-12T05:45:54.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.979+0000 7f69b293d700 1 -- 172.21.15.32:0/2639654025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69ac102710 msgr2=0x7f69ac0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:54.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.979+0000 7f69b293d700 1 --2- 172.21.15.32:0/2639654025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69ac102710 0x7f69ac0fbd80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6994004940 tx=0x7f6994009e30 comp rx=0 tx=0).stop 2024-04-12T05:45:54.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.979+0000 7f69b293d700 1 -- 172.21.15.32:0/2639654025 shutdown_connections 2024-04-12T05:45:54.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.979+0000 7f69b293d700 1 --2- 172.21.15.32:0/2639654025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69ac102710 0x7f69ac0fbd80 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:54.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.979+0000 7f69b293d700 1 --2- 172.21.15.32:0/2639654025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69ac101d70 0x7f69ac102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:54.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.979+0000 7f69b293d700 1 -- 172.21.15.32:0/2639654025 >> 172.21.15.32:0/2639654025 conn(0x7f69ac072bc0 msgr2=0x7f69ac072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:54.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.979+0000 7f69b293d700 1 -- 172.21.15.32:0/2639654025 shutdown_connections 2024-04-12T05:45:54.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.979+0000 7f69b293d700 1 -- 172.21.15.32:0/2639654025 wait complete. 2024-04-12T05:45:54.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.979+0000 7f69b293d700 1 Processor -- start 2024-04-12T05:45:54.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.979+0000 7f69b293d700 1 -- start start 2024-04-12T05:45:54.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.980+0000 7f69b293d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69ac101d70 0x7f69ac1129c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:54.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.980+0000 7f69b293d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69ac102710 0x7f69ac112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:54.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.980+0000 7f69b293d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac113550 con 0x7f69ac102710 2024-04-12T05:45:54.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.980+0000 7f69b293d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac10c940 con 0x7f69ac101d70 2024-04-12T05:45:54.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.980+0000 7f69ab7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69ac102710 0x7f69ac112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:54.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.980+0000 7f69ab7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69ac102710 0x7f69ac112f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56252/0 (socket says 172.21.15.32:56252) 2024-04-12T05:45:54.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.980+0000 7f69ab7fe700 1 -- 172.21.15.32:0/1619855587 learned_addr learned my addr 172.21.15.32:0/1619855587 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:45:54.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.980+0000 7f69abfff700 1 --2- 172.21.15.32:0/1619855587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69ac101d70 0x7f69ac1129c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:54.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.980+0000 7f69ab7fe700 1 -- 172.21.15.32:0/1619855587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69ac101d70 msgr2=0x7f69ac1129c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:54.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.980+0000 7f69ab7fe700 1 --2- 172.21.15.32:0/1619855587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69ac101d70 0x7f69ac1129c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:54.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.980+0000 7f69ab7fe700 1 -- 172.21.15.32:0/1619855587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6994005040 con 0x7f69ac102710 2024-04-12T05:45:54.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.980+0000 7f69ab7fe700 1 --2- 172.21.15.32:0/1619855587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69ac102710 0x7f69ac112f00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f699400b3b0 tx=0x7f6994009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:54.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.980+0000 7f69a8ff9700 1 -- 172.21.15.32:0/1619855587 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f699400e040 con 0x7f69ac102710 2024-04-12T05:45:54.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.981+0000 7f69a8ff9700 1 -- 172.21.15.32:0/1619855587 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f699400bd60 con 0x7f69ac102710 2024-04-12T05:45:54.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.981+0000 7f69b293d700 1 -- 172.21.15.32:0/1619855587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69ac10cbc0 con 0x7f69ac102710 2024-04-12T05:45:54.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.981+0000 7f69a8ff9700 1 -- 172.21.15.32:0/1619855587 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6994013660 con 0x7f69ac102710 2024-04-12T05:45:54.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.981+0000 7f69b293d700 1 -- 172.21.15.32:0/1619855587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69ac10d120 con 0x7f69ac102710 2024-04-12T05:45:54.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.981+0000 7f69b293d700 1 -- 172.21.15.32:0/1619855587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69ac0fd4c0 con 0x7f69ac102710 2024-04-12T05:45:54.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.982+0000 7f69a8ff9700 1 -- 172.21.15.32:0/1619855587 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f6994013920 con 0x7f69ac102710 2024-04-12T05:45:54.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.982+0000 7f69a8ff9700 1 --2- 172.21.15.32:0/1619855587 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f699806cc20 0x7f699806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:54.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.983+0000 7f69a8ff9700 1 -- 172.21.15.32:0/1619855587 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f699408fff0 con 0x7f69ac102710 2024-04-12T05:45:54.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.983+0000 7f69abfff700 1 --2- 172.21.15.32:0/1619855587 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f699806cc20 0x7f699806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:54.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.983+0000 7f69abfff700 1 --2- 172.21.15.32:0/1619855587 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f699806cc20 0x7f699806f0d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f699c004990 tx=0x7f699c009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:54.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:54.987+0000 7f69a8ff9700 1 -- 172.21.15.32:0/1619855587 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f699405da90 con 0x7f69ac102710 2024-04-12T05:45:55.159 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:55 smithi067 ceph-mon[112979]: from='client.34340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:45:55.159 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:55 smithi067 ceph-mon[112979]: pgmap v151: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T05:45:55.159 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:55 smithi067 ceph-mon[112979]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:45:55.159 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:55 smithi067 ceph-mon[112979]: from='client.34348 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:45:55.159 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:55 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/64240943' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:45:55.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.232+0000 7f69b293d700 1 -- 172.21.15.32:0/1619855587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f69ac05f5f0 con 0x7f69ac102710 2024-04-12T05:45:55.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.233+0000 7f69a8ff9700 1 -- 172.21.15.32:0/1619855587 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1864 (secure 0 0 0) 0x7f699405d620 con 0x7f69ac102710 2024-04-12T05:45:55.230 INFO:teuthology.orchestra.run.smithi032.stdout:e22 2024-04-12T05:45:55.230 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:45:55.230 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:45:55.230 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:45:55.230 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:45:55.230 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:45:55.230 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:45:55.230 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 22 2024-04-12T05:45:55.230 INFO:teuthology.orchestra.run.smithi032.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-12T05:45:55.230 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:45:55.230 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:41:05.358509+0000 2024-04-12T05:45:55.230 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:45:55.230 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:45:55.231 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:45:55.231 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:45:55.231 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:45:55.231 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:45:55.231 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:45:55.231 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:45:55.231 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 46 2024-04-12T05:45:55.231 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:45:55.231 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:45:55.231 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:45:55.231 INFO:teuthology.orchestra.run.smithi032.stdout:up {0=14480,1=24281} 2024-04-12T05:45:55.231 INFO:teuthology.orchestra.run.smithi032.stdout:failed 2024-04-12T05:45:55.231 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:45:55.231 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:45:55.231 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:45:55.232 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:45:55.232 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:45:55.232 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:45:55.232 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:45:55.232 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:45:55.232 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{0:14480} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.32:6828/387754149,v1:172.21.15.32:6829/387754149] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:45:55.232 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{0:14490} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/3678031004,v1:172.21.15.32:6827/3678031004] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:45:55.232 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{1:24281} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.67:6824/1291914599,v1:172.21.15.67:6825/1291914599] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:45:55.232 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{1:24295} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.67:6826/873395842,v1:172.21.15.67:6827/873395842] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:45:55.232 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:45:55.232 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:45:55.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.235+0000 7f69b293d700 1 -- 172.21.15.32:0/1619855587 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f699806cc20 msgr2=0x7f699806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:55.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.235+0000 7f69b293d700 1 --2- 172.21.15.32:0/1619855587 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f699806cc20 0x7f699806f0d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f699c004990 tx=0x7f699c009ee0 comp rx=0 tx=0).stop 2024-04-12T05:45:55.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.235+0000 7f69b293d700 1 -- 172.21.15.32:0/1619855587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69ac102710 msgr2=0x7f69ac112f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:55.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.235+0000 7f69b293d700 1 --2- 172.21.15.32:0/1619855587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69ac102710 0x7f69ac112f00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f699400b3b0 tx=0x7f6994009e30 comp rx=0 tx=0).stop 2024-04-12T05:45:55.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.235+0000 7f69b293d700 1 -- 172.21.15.32:0/1619855587 shutdown_connections 2024-04-12T05:45:55.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.235+0000 7f69b293d700 1 --2- 172.21.15.32:0/1619855587 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f699806cc20 0x7f699806f0d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:55.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.235+0000 7f69b293d700 1 --2- 172.21.15.32:0/1619855587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69ac101d70 0x7f69ac1129c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:55.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.235+0000 7f69b293d700 1 --2- 172.21.15.32:0/1619855587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69ac102710 0x7f69ac112f00 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:55.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.235+0000 7f69b293d700 1 -- 172.21.15.32:0/1619855587 >> 172.21.15.32:0/1619855587 conn(0x7f69ac072bc0 msgr2=0x7f69ac0fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:55.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.235+0000 7f69b293d700 1 -- 172.21.15.32:0/1619855587 shutdown_connections 2024-04-12T05:45:55.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.235+0000 7f69b293d700 1 -- 172.21.15.32:0/1619855587 wait complete. 2024-04-12T05:45:55.234 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 22 2024-04-12T05:45:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.339+0000 7ff450110700 1 -- 172.21.15.32:0/4172999826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff448101d70 msgr2=0x7ff448102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.339+0000 7ff450110700 1 --2- 172.21.15.32:0/4172999826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff448101d70 0x7ff448102140 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff444004940 tx=0x7ff444009e30 comp rx=0 tx=0).stop 2024-04-12T05:45:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.339+0000 7ff450110700 1 -- 172.21.15.32:0/4172999826 shutdown_connections 2024-04-12T05:45:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.339+0000 7ff450110700 1 --2- 172.21.15.32:0/4172999826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff448102710 0x7ff4480fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.339+0000 7ff450110700 1 --2- 172.21.15.32:0/4172999826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff448101d70 0x7ff448102140 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.339+0000 7ff450110700 1 -- 172.21.15.32:0/4172999826 >> 172.21.15.32:0/4172999826 conn(0x7ff448072bc0 msgr2=0x7ff448072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.339+0000 7ff450110700 1 -- 172.21.15.32:0/4172999826 shutdown_connections 2024-04-12T05:45:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.339+0000 7ff450110700 1 -- 172.21.15.32:0/4172999826 wait complete. 2024-04-12T05:45:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.340+0000 7ff450110700 1 Processor -- start 2024-04-12T05:45:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.340+0000 7ff450110700 1 -- start start 2024-04-12T05:45:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.340+0000 7ff450110700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff448101d70 0x7ff448115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.340+0000 7ff450110700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff448102710 0x7ff448116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.340+0000 7ff450110700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4481167b0 con 0x7ff448101d70 2024-04-12T05:45:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.340+0000 7ff450110700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44810fba0 con 0x7ff448102710 2024-04-12T05:45:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.340+0000 7ff44deac700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff448101d70 0x7ff448115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.340+0000 7ff44deac700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff448101d70 0x7ff448115c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56276/0 (socket says 172.21.15.32:56276) 2024-04-12T05:45:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.340+0000 7ff44deac700 1 -- 172.21.15.32:0/3721350906 learned_addr learned my addr 172.21.15.32:0/3721350906 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:45:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.340+0000 7ff44d6ab700 1 --2- 172.21.15.32:0/3721350906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff448102710 0x7ff448116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.340+0000 7ff44deac700 1 -- 172.21.15.32:0/3721350906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff448102710 msgr2=0x7ff448116160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.340+0000 7ff44deac700 1 --2- 172.21.15.32:0/3721350906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff448102710 0x7ff448116160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.340+0000 7ff44deac700 1 -- 172.21.15.32:0/3721350906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff444005040 con 0x7ff448101d70 2024-04-12T05:45:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.340+0000 7ff44d6ab700 1 --2- 172.21.15.32:0/3721350906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff448102710 0x7ff448116160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:45:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.341+0000 7ff44deac700 1 --2- 172.21.15.32:0/3721350906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff448101d70 0x7ff448115c20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff444009180 tx=0x7ff444009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.341+0000 7ff43a7fc700 1 -- 172.21.15.32:0/3721350906 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff44400e040 con 0x7ff448101d70 2024-04-12T05:45:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.341+0000 7ff43a7fc700 1 -- 172.21.15.32:0/3721350906 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff444004290 con 0x7ff448101d70 2024-04-12T05:45:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.341+0000 7ff450110700 1 -- 172.21.15.32:0/3721350906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff44810fe20 con 0x7ff448101d70 2024-04-12T05:45:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.341+0000 7ff43a7fc700 1 -- 172.21.15.32:0/3721350906 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff44401c3f0 con 0x7ff448101d70 2024-04-12T05:45:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.341+0000 7ff450110700 1 -- 172.21.15.32:0/3721350906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4481103e0 con 0x7ff448101d70 2024-04-12T05:45:55.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:55 smithi032 ceph-mon[146817]: from='client.34340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:45:55.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:55 smithi032 ceph-mon[146817]: pgmap v151: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T05:45:55.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:55 smithi032 ceph-mon[146817]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:45:55.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:55 smithi032 ceph-mon[146817]: from='client.34348 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:45:55.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:55 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/64240943' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:45:55.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.342+0000 7ff450110700 1 -- 172.21.15.32:0/3721350906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4480fd4c0 con 0x7ff448101d70 2024-04-12T05:45:55.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.342+0000 7ff43a7fc700 1 -- 172.21.15.32:0/3721350906 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ff444013790 con 0x7ff448101d70 2024-04-12T05:45:55.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.343+0000 7ff43a7fc700 1 --2- 172.21.15.32:0/3721350906 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff43406cc20 0x7ff43406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:55.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.343+0000 7ff43a7fc700 1 -- 172.21.15.32:0/3721350906 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6410+0+0 (secure 0 0 0) 0x7ff44408d1d0 con 0x7ff448101d70 2024-04-12T05:45:55.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.343+0000 7ff44d6ab700 1 --2- 172.21.15.32:0/3721350906 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff43406cc20 0x7ff43406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:55.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.343+0000 7ff44d6ab700 1 --2- 172.21.15.32:0/3721350906 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff43406cc20 0x7ff43406f0d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff43c009b80 tx=0x7ff43c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:55.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.347+0000 7ff43a7fc700 1 -- 172.21.15.32:0/3721350906 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff44405ad90 con 0x7ff448101d70 2024-04-12T05:45:55.428 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 05:45:55 smithi067 systemd[1]: Stopping Ceph osd.5 for 37b58ce2-f88d-11ee-bc8c-c7b262605968... 2024-04-12T05:45:55.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.567+0000 7ff450110700 1 -- 172.21.15.32:0/3721350906 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff44804bf40 con 0x7ff43406cc20 2024-04-12T05:45:55.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.568+0000 7ff43a7fc700 1 -- 172.21.15.32:0/3721350906 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7ff44804bf40 con 0x7ff43406cc20 2024-04-12T05:45:55.565 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:45:55.565 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:45:55.565 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:45:55.565 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:45:55.565 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:45:55.566 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:45:55.566 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:45:55.566 INFO:teuthology.orchestra.run.smithi032.stdout: "crash" 2024-04-12T05:45:55.566 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:45:55.566 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "11/21 daemons upgraded", 2024-04-12T05:45:55.566 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading osd daemons", 2024-04-12T05:45:55.566 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:45:55.566 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:45:55.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.571+0000 7ff450110700 1 -- 172.21.15.32:0/3721350906 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff43406cc20 msgr2=0x7ff43406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:55.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.571+0000 7ff450110700 1 --2- 172.21.15.32:0/3721350906 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff43406cc20 0x7ff43406f0d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff43c009b80 tx=0x7ff43c00b040 comp rx=0 tx=0).stop 2024-04-12T05:45:55.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.571+0000 7ff450110700 1 -- 172.21.15.32:0/3721350906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff448101d70 msgr2=0x7ff448115c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:55.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.571+0000 7ff450110700 1 --2- 172.21.15.32:0/3721350906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff448101d70 0x7ff448115c20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff444009180 tx=0x7ff444009e30 comp rx=0 tx=0).stop 2024-04-12T05:45:55.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.571+0000 7ff450110700 1 -- 172.21.15.32:0/3721350906 shutdown_connections 2024-04-12T05:45:55.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.571+0000 7ff450110700 1 --2- 172.21.15.32:0/3721350906 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff43406cc20 0x7ff43406f0d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:55.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.571+0000 7ff450110700 1 --2- 172.21.15.32:0/3721350906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff448101d70 0x7ff448115c20 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:55.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.571+0000 7ff450110700 1 --2- 172.21.15.32:0/3721350906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff448102710 0x7ff448116160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:55.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.571+0000 7ff450110700 1 -- 172.21.15.32:0/3721350906 >> 172.21.15.32:0/3721350906 conn(0x7ff448072bc0 msgr2=0x7ff448065970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:55.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.571+0000 7ff450110700 1 -- 172.21.15.32:0/3721350906 shutdown_connections 2024-04-12T05:45:55.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.571+0000 7ff450110700 1 -- 172.21.15.32:0/3721350906 wait complete. 2024-04-12T05:45:55.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.674+0000 7f46d4efe700 1 -- 172.21.15.32:0/3946394254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46d0102710 msgr2=0x7f46d00fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:55.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.674+0000 7f46d4efe700 1 --2- 172.21.15.32:0/3946394254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46d0102710 0x7f46d00fbd80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f46c4004940 tx=0x7f46c4009e30 comp rx=0 tx=0).stop 2024-04-12T05:45:55.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.674+0000 7f46d4efe700 1 -- 172.21.15.32:0/3946394254 shutdown_connections 2024-04-12T05:45:55.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.674+0000 7f46d4efe700 1 --2- 172.21.15.32:0/3946394254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46d0102710 0x7f46d00fbd80 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:55.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.674+0000 7f46d4efe700 1 --2- 172.21.15.32:0/3946394254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46d0101d70 0x7f46d0102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:55.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.674+0000 7f46d4efe700 1 -- 172.21.15.32:0/3946394254 >> 172.21.15.32:0/3946394254 conn(0x7f46d0072bc0 msgr2=0x7f46d0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:55.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.674+0000 7f46d4efe700 1 -- 172.21.15.32:0/3946394254 shutdown_connections 2024-04-12T05:45:55.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.674+0000 7f46d4efe700 1 -- 172.21.15.32:0/3946394254 wait complete. 2024-04-12T05:45:55.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.675+0000 7f46d4efe700 1 Processor -- start 2024-04-12T05:45:55.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.675+0000 7f46d4efe700 1 -- start start 2024-04-12T05:45:55.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.675+0000 7f46d4efe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46d0101d70 0x7f46d010ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:55.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.675+0000 7f46d4efe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46d0102710 0x7f46d010efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:55.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.675+0000 7f46d4efe700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d010f5f0 con 0x7f46d0102710 2024-04-12T05:45:55.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.675+0000 7f46d4efe700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d010f730 con 0x7f46d0101d70 2024-04-12T05:45:55.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.675+0000 7f46cdd9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46d0102710 0x7f46d010efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:55.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.675+0000 7f46cdd9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46d0102710 0x7f46d010efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56294/0 (socket says 172.21.15.32:56294) 2024-04-12T05:45:55.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.675+0000 7f46cdd9b700 1 -- 172.21.15.32:0/3677807641 learned_addr learned my addr 172.21.15.32:0/3677807641 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:45:55.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.675+0000 7f46ce59c700 1 --2- 172.21.15.32:0/3677807641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46d0101d70 0x7f46d010ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:55.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.675+0000 7f46cdd9b700 1 -- 172.21.15.32:0/3677807641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46d0101d70 msgr2=0x7f46d010ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:55.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.675+0000 7f46cdd9b700 1 --2- 172.21.15.32:0/3677807641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46d0101d70 0x7f46d010ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:55.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.675+0000 7f46cdd9b700 1 -- 172.21.15.32:0/3677807641 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c4005040 con 0x7f46d0102710 2024-04-12T05:45:55.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.675+0000 7f46cdd9b700 1 --2- 172.21.15.32:0/3677807641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46d0102710 0x7f46d010efa0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f46c4000c00 tx=0x7f46c4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:55.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.676+0000 7f46beffd700 1 -- 172.21.15.32:0/3677807641 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46c400e040 con 0x7f46d0102710 2024-04-12T05:45:55.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.676+0000 7f46beffd700 1 -- 172.21.15.32:0/3677807641 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f46c4013420 con 0x7f46d0102710 2024-04-12T05:45:55.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.676+0000 7f46d4efe700 1 -- 172.21.15.32:0/3677807641 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d0117ad0 con 0x7f46d0102710 2024-04-12T05:45:55.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.676+0000 7f46beffd700 1 -- 172.21.15.32:0/3677807641 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46c401c610 con 0x7f46d0102710 2024-04-12T05:45:55.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.676+0000 7f46d4efe700 1 -- 172.21.15.32:0/3677807641 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d0117fe0 con 0x7f46d0102710 2024-04-12T05:45:55.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.677+0000 7f46d4efe700 1 -- 172.21.15.32:0/3677807641 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d00ff080 con 0x7f46d0102710 2024-04-12T05:45:55.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.677+0000 7f46beffd700 1 -- 172.21.15.32:0/3677807641 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f46c4013590 con 0x7f46d0102710 2024-04-12T05:45:55.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.678+0000 7f46beffd700 1 --2- 172.21.15.32:0/3677807641 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f46b806cc20 0x7f46b806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:45:55.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.678+0000 7f46beffd700 1 -- 172.21.15.32:0/3677807641 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6410+0+0 (secure 0 0 0) 0x7f46c408d220 con 0x7f46d0102710 2024-04-12T05:45:55.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.678+0000 7f46ce59c700 1 --2- 172.21.15.32:0/3677807641 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f46b806cc20 0x7f46b806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:55.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.678+0000 7f46ce59c700 1 --2- 172.21.15.32:0/3677807641 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f46b806cc20 0x7f46b806f0d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f46c0007fd0 tx=0x7f46c0009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:55.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.683+0000 7f46beffd700 1 -- 172.21.15.32:0/3677807641 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f46c4057400 con 0x7f46d0102710 2024-04-12T05:45:55.767 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 05:45:55 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[71935]: 2024-04-12T05:45:55.431+0000 7f52b3b01700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T05:45:55.767 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 05:45:55 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[71935]: 2024-04-12T05:45:55.431+0000 7f52b3b01700 -1 osd.5 74 *** Got signal Terminated *** 2024-04-12T05:45:55.768 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 05:45:55 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[71935]: 2024-04-12T05:45:55.431+0000 7f52b3b01700 -1 osd.5 74 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T05:45:55.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.970+0000 7f46d4efe700 1 -- 172.21.15.32:0/3677807641 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f46d0065db0 con 0x7f46d0102710 2024-04-12T05:45:55.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.971+0000 7f46beffd700 1 -- 172.21.15.32:0/3677807641 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f46c405aa20 con 0x7f46d0102710 2024-04-12T05:45:55.967 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_OK 2024-04-12T05:45:55.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.973+0000 7f46d4efe700 1 -- 172.21.15.32:0/3677807641 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f46b806cc20 msgr2=0x7f46b806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:55.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.973+0000 7f46d4efe700 1 --2- 172.21.15.32:0/3677807641 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f46b806cc20 0x7f46b806f0d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f46c0007fd0 tx=0x7f46c0009040 comp rx=0 tx=0).stop 2024-04-12T05:45:55.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.973+0000 7f46d4efe700 1 -- 172.21.15.32:0/3677807641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46d0102710 msgr2=0x7f46d010efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:55.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.973+0000 7f46d4efe700 1 --2- 172.21.15.32:0/3677807641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46d0102710 0x7f46d010efa0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f46c4000c00 tx=0x7f46c4009e30 comp rx=0 tx=0).stop 2024-04-12T05:45:55.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.973+0000 7f46d4efe700 1 -- 172.21.15.32:0/3677807641 shutdown_connections 2024-04-12T05:45:55.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.973+0000 7f46d4efe700 1 --2- 172.21.15.32:0/3677807641 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f46b806cc20 0x7f46b806f0d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:55.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.973+0000 7f46d4efe700 1 --2- 172.21.15.32:0/3677807641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46d0101d70 0x7f46d010ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:55.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.973+0000 7f46d4efe700 1 --2- 172.21.15.32:0/3677807641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46d0102710 0x7f46d010efa0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:45:55.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.973+0000 7f46d4efe700 1 -- 172.21.15.32:0/3677807641 >> 172.21.15.32:0/3677807641 conn(0x7f46d0072bc0 msgr2=0x7f46d00fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:55.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.973+0000 7f46d4efe700 1 -- 172.21.15.32:0/3677807641 shutdown_connections 2024-04-12T05:45:55.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:45:55.973+0000 7f46d4efe700 1 -- 172.21.15.32:0/3677807641 wait complete. 2024-04-12T05:45:56.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:56 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1619855587' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:45:56.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:56 smithi032 ceph-mon[146817]: osd.5 marked itself down and dead 2024-04-12T05:45:56.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:56 smithi032 ceph-mon[146817]: from='client.34360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:45:56.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:56 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3677807641' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:45:56.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:56 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1619855587' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:45:56.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:56 smithi067 ceph-mon[112979]: osd.5 marked itself down and dead 2024-04-12T05:45:56.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:56 smithi067 ceph-mon[112979]: from='client.34360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:45:56.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:56 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3677807641' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:45:57.267 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 05:45:56 smithi067 bash[136941]: ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5 2024-04-12T05:45:57.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:57 smithi067 ceph-mon[112979]: pgmap v152: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:45:57.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:57 smithi067 ceph-mon[112979]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T05:45:57.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:57 smithi067 ceph-mon[112979]: osdmap e75: 6 total, 5 up, 6 in 2024-04-12T05:45:57.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:57 smithi032 ceph-mon[146817]: pgmap v152: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T05:45:57.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:57 smithi032 ceph-mon[146817]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T05:45:57.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:57 smithi032 ceph-mon[146817]: osdmap e75: 6 total, 5 up, 6 in 2024-04-12T05:45:58.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:58 smithi032 ceph-mon[146817]: osdmap e76: 6 total, 5 up, 6 in 2024-04-12T05:45:58.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:58 smithi067 ceph-mon[112979]: osdmap e76: 6 total, 5 up, 6 in 2024-04-12T05:45:59.267 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 05:45:58 smithi067 systemd[1]: ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@osd.5.service: Succeeded. 2024-04-12T05:45:59.268 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 05:45:58 smithi067 systemd[1]: Stopped Ceph osd.5 for 37b58ce2-f88d-11ee-bc8c-c7b262605968. 2024-04-12T05:45:59.268 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 05:45:58 smithi067 systemd[1]: Starting Ceph osd.5 for 37b58ce2-f88d-11ee-bc8c-c7b262605968... 2024-04-12T05:45:59.929 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:45:59 smithi067 ceph-mon[112979]: pgmap v155: 65 pgs: 13 stale+active+clean, 52 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-12T05:45:59.930 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 05:45:59 smithi067 bash[137212]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-12T05:45:59.930 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 05:45:59 smithi067 bash[137212]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-12T05:45:59.930 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 05:45:59 smithi067 bash[137212]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-12T05:45:59.930 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 05:45:59 smithi067 bash[137212]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-12T05:45:59.930 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 05:45:59 smithi067 bash[137212]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-04-12T05:45:59.930 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 05:45:59 smithi067 bash[137212]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-12T05:45:59.930 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 05:45:59 smithi067 bash[137212]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-12T05:46:00.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:45:59 smithi032 ceph-mon[146817]: pgmap v155: 65 pgs: 13 stale+active+clean, 52 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-12T05:46:01.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:00 smithi067 ceph-mon[112979]: Health check failed: Degraded data redundancy: 15/333 objects degraded (4.505%), 6 pgs degraded (PG_DEGRADED) 2024-04-12T05:46:01.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:00 smithi032 ceph-mon[146817]: Health check failed: Degraded data redundancy: 15/333 objects degraded (4.505%), 6 pgs degraded (PG_DEGRADED) 2024-04-12T05:46:01.767 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 05:46:01 smithi067 ceph-osd[137505]: -- 172.21.15.67:0/3552367046 <== mon.1 v2:172.21.15.67:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bcd9ca51e0 con 0x55bcd9c74c00 2024-04-12T05:46:01.767 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 05:46:01 smithi067 bash[137468]: 775899bf72bf8806a453102363f316ded68c3d02f2cd1edcd7b208b6132ade95 2024-04-12T05:46:01.767 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 05:46:01 smithi067 systemd[1]: Started Ceph osd.5 for 37b58ce2-f88d-11ee-bc8c-c7b262605968. 2024-04-12T05:46:01.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:01 smithi067 ceph-mon[112979]: pgmap v156: 65 pgs: 6 active+undersized+degraded, 4 stale+active+clean, 9 active+undersized, 46 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 15/333 objects degraded (4.505%) 2024-04-12T05:46:01.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:01 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:01.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:01 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:01.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:01 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:46:02.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:01 smithi032 ceph-mon[146817]: pgmap v156: 65 pgs: 6 active+undersized+degraded, 4 stale+active+clean, 9 active+undersized, 46 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 15/333 objects degraded (4.505%) 2024-04-12T05:46:02.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:01 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:02.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:01 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:02.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:01 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:46:02.366 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 05:46:02 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T05:46:02.025+0000 7f597074c7c0 -1 Falling back to public interface 2024-04-12T05:46:04.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:03 smithi067 ceph-mon[112979]: pgmap v157: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 53/333 objects degraded (15.916%) 2024-04-12T05:46:04.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:04.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:46:04.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:03 smithi032 ceph-mon[146817]: pgmap v157: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 53/333 objects degraded (15.916%) 2024-04-12T05:46:04.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:04.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:46:05.740 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:05 smithi067 ceph-mon[112979]: pgmap v158: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 53/333 objects degraded (15.916%) 2024-04-12T05:46:05.740 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:05 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:05.740 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:05 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:05.740 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:05 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:05.740 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:05 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:05.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:05 smithi032 ceph-mon[146817]: pgmap v158: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 53/333 objects degraded (15.916%) 2024-04-12T05:46:05.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:05 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:05.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:05 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:05.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:05 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:05.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:05 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:07.206 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 05:46:06 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T05:46:06.985+0000 7f597074c7c0 -1 osd.5 74 log_to_monitors true 2024-04-12T05:46:07.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:07 smithi067 ceph-mon[112979]: pgmap v159: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 53/333 objects degraded (15.916%) 2024-04-12T05:46:07.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:07 smithi067 ceph-mon[112979]: from='osd.5 [v2:172.21.15.67:6816/2273350169,v1:172.21.15.67:6817/2273350169]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T05:46:07.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:07 smithi067 ceph-mon[112979]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T05:46:07.768 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 05:46:07 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T05:46:07.496+0000 7f5961ba6700 -1 osd.5 74 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T05:46:07.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:07 smithi032 ceph-mon[146817]: pgmap v159: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 53/333 objects degraded (15.916%) 2024-04-12T05:46:07.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:07 smithi032 ceph-mon[146817]: from='osd.5 [v2:172.21.15.67:6816/2273350169,v1:172.21.15.67:6817/2273350169]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T05:46:07.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:07 smithi032 ceph-mon[146817]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T05:46:08.642 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:08 smithi067 ceph-mon[112979]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-12T05:46:08.642 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:08 smithi067 ceph-mon[112979]: osdmap e77: 6 total, 5 up, 6 in 2024-04-12T05:46:08.642 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:08 smithi067 ceph-mon[112979]: from='osd.5 [v2:172.21.15.67:6816/2273350169,v1:172.21.15.67:6817/2273350169]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-12T05:46:08.642 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:08 smithi067 ceph-mon[112979]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-12T05:46:08.642 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:08 smithi067 ceph-mon[112979]: Health check update: Degraded data redundancy: 53/333 objects degraded (15.916%), 14 pgs degraded (PG_DEGRADED) 2024-04-12T05:46:08.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:08 smithi032 ceph-mon[146817]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-12T05:46:08.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:08 smithi032 ceph-mon[146817]: osdmap e77: 6 total, 5 up, 6 in 2024-04-12T05:46:08.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:08 smithi032 ceph-mon[146817]: from='osd.5 [v2:172.21.15.67:6816/2273350169,v1:172.21.15.67:6817/2273350169]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-12T05:46:08.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:08 smithi032 ceph-mon[146817]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-12T05:46:08.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:08 smithi032 ceph-mon[146817]: Health check update: Degraded data redundancy: 53/333 objects degraded (15.916%), 14 pgs degraded (PG_DEGRADED) 2024-04-12T05:46:09.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: pgmap v161: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 270 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail; 53/333 objects degraded (15.916%) 2024-04-12T05:46:09.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T05:46:09.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: osd.5 [v2:172.21.15.67:6816/2273350169,v1:172.21.15.67:6817/2273350169] boot 2024-04-12T05:46:09.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: osdmap e78: 6 total, 6 up, 6 in 2024-04-12T05:46:09.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:46:09.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:09.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:09.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:46:09.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:46:09.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:09.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:46:09.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:09.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:09.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:09.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:09.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:09.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-12T05:46:09.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-12T05:46:09.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-12T05:46:09.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-12T05:46:09.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-12T05:46:09.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-12T05:46:09.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-12T05:46:09.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-12T05:46:09.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-12T05:46:09.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-12T05:46:09.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-12T05:46:09.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-12T05:46:09.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd require-osd-release", "release": "reef"}]: dispatch 2024-04-12T05:46:09.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "osd require-osd-release", "release": "reef"}]': finished 2024-04-12T05:46:09.770 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:09 smithi067 ceph-mon[112979]: osdmap e79: 6 total, 6 up, 6 in 2024-04-12T05:46:09.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: pgmap v161: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 270 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail; 53/333 objects degraded (15.916%) 2024-04-12T05:46:09.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T05:46:09.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: osd.5 [v2:172.21.15.67:6816/2273350169,v1:172.21.15.67:6817/2273350169] boot 2024-04-12T05:46:09.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: osdmap e78: 6 total, 6 up, 6 in 2024-04-12T05:46:09.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:46:09.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:09.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:09.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:46:09.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:46:09.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:09.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:46:09.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:09.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:09.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:09.841 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:09.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:09.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-12T05:46:09.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-12T05:46:09.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-12T05:46:09.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-12T05:46:09.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-12T05:46:09.842 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-12T05:46:09.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-12T05:46:09.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-12T05:46:09.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-12T05:46:09.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-12T05:46:09.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-12T05:46:09.843 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-12T05:46:09.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd require-osd-release", "release": "reef"}]: dispatch 2024-04-12T05:46:09.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "osd require-osd-release", "release": "reef"}]': finished 2024-04-12T05:46:09.844 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:09 smithi032 ceph-mon[146817]: osdmap e79: 6 total, 6 up, 6 in 2024-04-12T05:46:10.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:10 smithi067 ceph-mon[112979]: Upgrade: Setting container_image for all osd 2024-04-12T05:46:10.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:10 smithi067 ceph-mon[112979]: Upgrade: Setting require_osd_release to 18 reef 2024-04-12T05:46:10.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:10 smithi067 ceph-mon[112979]: Upgrade: Disabling standby-replay for filesystem cephfs 2024-04-12T05:46:10.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:10 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:10.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:10 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]: dispatch 2024-04-12T05:46:10.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:10 smithi067 ceph-mon[112979]: osdmap e80: 6 total, 6 up, 6 in 2024-04-12T05:46:10.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:10 smithi067 ceph-mon[112979]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-12T05:46:10.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:10 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]': finished 2024-04-12T05:46:10.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:10 smithi067 ceph-mon[112979]: fsmap cephfs:2 {0=cephfs.smithi032.pndtry=up:active,1=cephfs.smithi067.zesmgy=up:active} 2024-04-12T05:46:10.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:10 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:46:10.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:10 smithi032 ceph-mon[146817]: Upgrade: Setting container_image for all osd 2024-04-12T05:46:10.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:10 smithi032 ceph-mon[146817]: Upgrade: Setting require_osd_release to 18 reef 2024-04-12T05:46:10.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:10 smithi032 ceph-mon[146817]: Upgrade: Disabling standby-replay for filesystem cephfs 2024-04-12T05:46:10.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:10 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:10.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:10 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]: dispatch 2024-04-12T05:46:10.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:10 smithi032 ceph-mon[146817]: osdmap e80: 6 total, 6 up, 6 in 2024-04-12T05:46:10.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:10 smithi032 ceph-mon[146817]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-12T05:46:10.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:10 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]': finished 2024-04-12T05:46:10.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:10 smithi032 ceph-mon[146817]: fsmap cephfs:2 {0=cephfs.smithi032.pndtry=up:active,1=cephfs.smithi067.zesmgy=up:active} 2024-04-12T05:46:10.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:10 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:46:10.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:10 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-mon-smithi032[146794]: 2024-04-12T05:46:10.667+0000 7f3eb013c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T05:46:11.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:11 smithi067 ceph-mon[112979]: pgmap v165: 65 pgs: 4 peering, 11 active+undersized+degraded, 11 active+undersized, 39 active+clean; 270 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s; 47/333 objects degraded (14.114%) 2024-04-12T05:46:11.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:11 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:46:11.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:11 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:46:11.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:11 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:11.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:11 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:46:11.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:11 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:11.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:11 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:11.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:11 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:11.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:11 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:11.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:11 smithi067 ceph-mon[112979]: Upgrade: failing fs cephfs for rapid multi-rank mds upgrade 2024-04-12T05:46:11.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:11 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-12T05:46:11.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:11 smithi067 ceph-mon[112979]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T05:46:11.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:11 smithi067 ceph-mon[112979]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T05:46:11.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:11 smithi067 ceph-mon[112979]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-12T05:46:11.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:11 smithi067 ceph-mon[112979]: osdmap e81: 6 total, 6 up, 6 in 2024-04-12T05:46:11.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:11 smithi067 ceph-mon[112979]: mds.? [v2:172.21.15.67:6826/3573641538,v1:172.21.15.67:6827/3573641538] up:boot 2024-04-12T05:46:11.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:11 smithi067 ceph-mon[112979]: mds.? [v2:172.21.15.32:6826/268850045,v1:172.21.15.32:6827/268850045] up:boot 2024-04-12T05:46:11.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:11 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-12T05:46:11.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:11 smithi067 ceph-mon[112979]: fsmap cephfs:0/2 2 up:standby, 2 failed 2024-04-12T05:46:11.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:11 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.jehaaj"}]: dispatch 2024-04-12T05:46:11.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:11 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.ymvqih"}]: dispatch 2024-04-12T05:46:11.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:11 smithi032 ceph-mon[146817]: pgmap v165: 65 pgs: 4 peering, 11 active+undersized+degraded, 11 active+undersized, 39 active+clean; 270 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s; 47/333 objects degraded (14.114%) 2024-04-12T05:46:11.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:11 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:46:11.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:11 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:46:11.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:11 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:11.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:11 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:46:11.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:11 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:11.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:11 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:11.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:11 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:11.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:11 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:11.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:11 smithi032 ceph-mon[146817]: Upgrade: failing fs cephfs for rapid multi-rank mds upgrade 2024-04-12T05:46:11.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:11 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-12T05:46:11.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:11 smithi032 ceph-mon[146817]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T05:46:11.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:11 smithi032 ceph-mon[146817]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T05:46:11.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:11 smithi032 ceph-mon[146817]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-12T05:46:11.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:11 smithi032 ceph-mon[146817]: osdmap e81: 6 total, 6 up, 6 in 2024-04-12T05:46:11.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:11 smithi032 ceph-mon[146817]: mds.? [v2:172.21.15.67:6826/3573641538,v1:172.21.15.67:6827/3573641538] up:boot 2024-04-12T05:46:11.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:11 smithi032 ceph-mon[146817]: mds.? [v2:172.21.15.32:6826/268850045,v1:172.21.15.32:6827/268850045] up:boot 2024-04-12T05:46:11.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:11 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-12T05:46:11.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:11 smithi032 ceph-mon[146817]: fsmap cephfs:0/2 2 up:standby, 2 failed 2024-04-12T05:46:11.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:11 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.jehaaj"}]: dispatch 2024-04-12T05:46:11.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:11 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.ymvqih"}]: dispatch 2024-04-12T05:46:12.983 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:12 smithi032 ceph-mon[146817]: mds.? [v2:172.21.15.67:6824/973024107,v1:172.21.15.67:6825/973024107] up:boot 2024-04-12T05:46:12.983 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:12 smithi032 ceph-mon[146817]: mds.? [v2:172.21.15.32:6828/2545444205,v1:172.21.15.32:6829/2545444205] up:boot 2024-04-12T05:46:12.983 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:12 smithi032 ceph-mon[146817]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-12T05:46:12.983 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:12 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.pndtry"}]: dispatch 2024-04-12T05:46:12.983 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:12 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.zesmgy"}]: dispatch 2024-04-12T05:46:12.983 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:12 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:12.983 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:12 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.jehaaj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:46:12.983 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:12 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:46:12.983 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:12 smithi032 ceph-mon[146817]: Health check update: Degraded data redundancy: 47/333 objects degraded (14.114%), 11 pgs degraded (PG_DEGRADED) 2024-04-12T05:46:13.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:12 smithi067 ceph-mon[112979]: mds.? [v2:172.21.15.67:6824/973024107,v1:172.21.15.67:6825/973024107] up:boot 2024-04-12T05:46:13.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:12 smithi067 ceph-mon[112979]: mds.? [v2:172.21.15.32:6828/2545444205,v1:172.21.15.32:6829/2545444205] up:boot 2024-04-12T05:46:13.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:12 smithi067 ceph-mon[112979]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-12T05:46:13.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:12 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.pndtry"}]: dispatch 2024-04-12T05:46:13.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:12 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.zesmgy"}]: dispatch 2024-04-12T05:46:13.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:12 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:13.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:12 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.jehaaj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:46:13.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:12 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:46:13.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:12 smithi067 ceph-mon[112979]: Health check update: Degraded data redundancy: 47/333 objects degraded (14.114%), 11 pgs degraded (PG_DEGRADED) 2024-04-12T05:46:13.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:13 smithi032 ceph-mon[146817]: pgmap v167: 65 pgs: 4 peering, 5 active+undersized+degraded, 8 active+undersized, 48 active+clean; 270 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail; 767 B/s rd, 1 op/s; 16/333 objects degraded (4.805%) 2024-04-12T05:46:13.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:13 smithi032 ceph-mon[146817]: Upgrade: Updating mds.cephfs.smithi032.jehaaj 2024-04-12T05:46:13.973 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:13 smithi032 ceph-mon[146817]: Deploying daemon mds.cephfs.smithi032.jehaaj on smithi032 2024-04-12T05:46:14.020 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:13 smithi067 ceph-mon[112979]: pgmap v167: 65 pgs: 4 peering, 5 active+undersized+degraded, 8 active+undersized, 48 active+clean; 270 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail; 767 B/s rd, 1 op/s; 16/333 objects degraded (4.805%) 2024-04-12T05:46:14.020 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:13 smithi067 ceph-mon[112979]: Upgrade: Updating mds.cephfs.smithi032.jehaaj 2024-04-12T05:46:14.020 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:13 smithi067 ceph-mon[112979]: Deploying daemon mds.cephfs.smithi032.jehaaj on smithi032 2024-04-12T05:46:15.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:15 smithi067 ceph-mon[112979]: pgmap v168: 65 pgs: 4 peering, 61 active+clean; 270 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:15.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:15 smithi067 ceph-mon[112979]: osdmap e82: 6 total, 6 up, 6 in 2024-04-12T05:46:15.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:15 smithi067 ceph-mon[112979]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 16/333 objects degraded (4.805%), 5 pgs degraded) 2024-04-12T05:46:15.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:15 smithi067 ceph-mon[112979]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-12T05:46:15.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:15 smithi032 ceph-mon[146817]: pgmap v168: 65 pgs: 4 peering, 61 active+clean; 270 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:15.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:15 smithi032 ceph-mon[146817]: osdmap e82: 6 total, 6 up, 6 in 2024-04-12T05:46:15.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:15 smithi032 ceph-mon[146817]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 16/333 objects degraded (4.805%), 5 pgs degraded) 2024-04-12T05:46:15.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:15 smithi032 ceph-mon[146817]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-12T05:46:16.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:16 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.jehaaj"}]: dispatch 2024-04-12T05:46:16.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:16 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.jehaaj"}]: dispatch 2024-04-12T05:46:17.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:17 smithi067 ceph-mon[112979]: pgmap v170: 65 pgs: 65 active+clean; 270 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:17.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:17 smithi032 ceph-mon[146817]: pgmap v170: 65 pgs: 65 active+clean; 270 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:19.177 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:18 smithi032 ceph-mon[146817]: pgmap v171: 65 pgs: 65 active+clean; 270 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:19.177 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:19.177 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:46:19.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:18 smithi067 ceph-mon[112979]: pgmap v171: 65 pgs: 65 active+clean; 270 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:19.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:19.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:46:21.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:21 smithi032 ceph-mon[146817]: pgmap v172: 65 pgs: 65 active+clean; 270 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:21.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:21 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:21.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:21 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:21.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:21 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:46:21.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:21 smithi032 ceph-mon[146817]: mds.? [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] up:boot 2024-04-12T05:46:21.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:21 smithi032 ceph-mon[146817]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-12T05:46:21.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:21 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.jehaaj"}]: dispatch 2024-04-12T05:46:21.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:21 smithi067 ceph-mon[112979]: pgmap v172: 65 pgs: 65 active+clean; 270 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:21.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:21 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:21.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:21 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:21.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:21 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:46:21.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:21 smithi067 ceph-mon[112979]: mds.? [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] up:boot 2024-04-12T05:46:21.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:21 smithi067 ceph-mon[112979]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-12T05:46:21.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:21 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.jehaaj"}]: dispatch 2024-04-12T05:46:23.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:23 smithi032 ceph-mon[146817]: pgmap v173: 65 pgs: 65 active+clean; 270 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:23.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:23 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:23.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:23 smithi067 ceph-mon[112979]: pgmap v173: 65 pgs: 65 active+clean; 270 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:23.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:23 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:25.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:25 smithi032 ceph-mon[146817]: pgmap v174: 65 pgs: 65 active+clean; 270 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:25.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:25 smithi067 ceph-mon[112979]: pgmap v174: 65 pgs: 65 active+clean; 270 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:26.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.101+0000 7fb6c0ac9700 1 -- 172.21.15.32:0/2291665834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6bc0fe850 msgr2=0x7fb6bc106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:26.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.101+0000 7fb6c0ac9700 1 --2- 172.21.15.32:0/2291665834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6bc0fe850 0x7fb6bc106f20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb6b0004970 tx=0x7fb6b0009e30 comp rx=0 tx=0).stop 2024-04-12T05:46:26.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.101+0000 7fb6c0ac9700 1 -- 172.21.15.32:0/2291665834 shutdown_connections 2024-04-12T05:46:26.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.101+0000 7fb6c0ac9700 1 --2- 172.21.15.32:0/2291665834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6bc0fe850 0x7fb6bc106f20 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:26.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.101+0000 7fb6c0ac9700 1 --2- 172.21.15.32:0/2291665834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6bc0fdeb0 0x7fb6bc0fe280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:26.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.101+0000 7fb6c0ac9700 1 -- 172.21.15.32:0/2291665834 >> 172.21.15.32:0/2291665834 conn(0x7fb6bc072bc0 msgr2=0x7fb6bc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:26.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.101+0000 7fb6c0ac9700 1 -- 172.21.15.32:0/2291665834 shutdown_connections 2024-04-12T05:46:26.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.101+0000 7fb6c0ac9700 1 -- 172.21.15.32:0/2291665834 wait complete. 2024-04-12T05:46:26.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.102+0000 7fb6c0ac9700 1 Processor -- start 2024-04-12T05:46:26.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.102+0000 7fb6c0ac9700 1 -- start start 2024-04-12T05:46:26.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.102+0000 7fb6c0ac9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6bc0fdeb0 0x7fb6bc113a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:26.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.102+0000 7fb6c0ac9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6bc0fe850 0x7fb6bc113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:26.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.102+0000 7fb6c0ac9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc1145a0 con 0x7fb6bc0fe850 2024-04-12T05:46:26.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.102+0000 7fb6c0ac9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc10d990 con 0x7fb6bc0fdeb0 2024-04-12T05:46:26.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.102+0000 7fb6b9d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6bc0fe850 0x7fb6bc113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:26.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.103+0000 7fb6b9d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6bc0fe850 0x7fb6bc113f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41286/0 (socket says 172.21.15.32:41286) 2024-04-12T05:46:26.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.103+0000 7fb6b9d9b700 1 -- 172.21.15.32:0/1135674617 learned_addr learned my addr 172.21.15.32:0/1135674617 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:46:26.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.103+0000 7fb6ba59c700 1 --2- 172.21.15.32:0/1135674617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6bc0fdeb0 0x7fb6bc113a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:26.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.103+0000 7fb6b9d9b700 1 -- 172.21.15.32:0/1135674617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6bc0fdeb0 msgr2=0x7fb6bc113a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:26.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.103+0000 7fb6b9d9b700 1 --2- 172.21.15.32:0/1135674617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6bc0fdeb0 0x7fb6bc113a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:26.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.103+0000 7fb6b9d9b700 1 -- 172.21.15.32:0/1135674617 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b0005040 con 0x7fb6bc0fe850 2024-04-12T05:46:26.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.103+0000 7fb6b9d9b700 1 --2- 172.21.15.32:0/1135674617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6bc0fe850 0x7fb6bc113f50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb6b0005010 tx=0x7fb6b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:26.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.103+0000 7fb6aaffd700 1 -- 172.21.15.32:0/1135674617 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6b000e040 con 0x7fb6bc0fe850 2024-04-12T05:46:26.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.103+0000 7fb6aaffd700 1 -- 172.21.15.32:0/1135674617 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb6b00098b0 con 0x7fb6bc0fe850 2024-04-12T05:46:26.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.103+0000 7fb6c0ac9700 1 -- 172.21.15.32:0/1135674617 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6bc10dc10 con 0x7fb6bc0fe850 2024-04-12T05:46:26.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.103+0000 7fb6aaffd700 1 -- 172.21.15.32:0/1135674617 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6b0013e80 con 0x7fb6bc0fe850 2024-04-12T05:46:26.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.103+0000 7fb6c0ac9700 1 -- 172.21.15.32:0/1135674617 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6bc10e150 con 0x7fb6bc0fe850 2024-04-12T05:46:26.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.104+0000 7fb6c0ac9700 1 -- 172.21.15.32:0/1135674617 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6bc065e10 con 0x7fb6bc0fe850 2024-04-12T05:46:26.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.105+0000 7fb6aaffd700 1 -- 172.21.15.32:0/1135674617 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb6b0009400 con 0x7fb6bc0fe850 2024-04-12T05:46:26.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.105+0000 7fb6aaffd700 1 --2- 172.21.15.32:0/1135674617 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb6a406cc30 0x7fb6a406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:26.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.105+0000 7fb6aaffd700 1 -- 172.21.15.32:0/1135674617 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fb6b008d110 con 0x7fb6bc0fe850 2024-04-12T05:46:26.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.105+0000 7fb6ba59c700 1 --2- 172.21.15.32:0/1135674617 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb6a406cc30 0x7fb6a406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:26.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.106+0000 7fb6ba59c700 1 --2- 172.21.15.32:0/1135674617 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb6a406cc30 0x7fb6a406f0e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac009b80 tx=0x7fb6ac00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:26.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.110+0000 7fb6aaffd700 1 -- 172.21.15.32:0/1135674617 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb6b005ac20 con 0x7fb6bc0fe850 2024-04-12T05:46:26.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.341+0000 7fb6c0ac9700 1 -- 172.21.15.32:0/1135674617 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb6bc109e20 con 0x7fb6a406cc30 2024-04-12T05:46:26.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.342+0000 7fb6aaffd700 1 -- 172.21.15.32:0/1135674617 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb6bc109e20 con 0x7fb6a406cc30 2024-04-12T05:46:26.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.344+0000 7fb6c0ac9700 1 -- 172.21.15.32:0/1135674617 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb6a406cc30 msgr2=0x7fb6a406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:26.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.344+0000 7fb6c0ac9700 1 --2- 172.21.15.32:0/1135674617 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb6a406cc30 0x7fb6a406f0e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac009b80 tx=0x7fb6ac00b040 comp rx=0 tx=0).stop 2024-04-12T05:46:26.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.344+0000 7fb6c0ac9700 1 -- 172.21.15.32:0/1135674617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6bc0fe850 msgr2=0x7fb6bc113f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:26.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.344+0000 7fb6c0ac9700 1 --2- 172.21.15.32:0/1135674617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6bc0fe850 0x7fb6bc113f50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb6b0005010 tx=0x7fb6b0009e30 comp rx=0 tx=0).stop 2024-04-12T05:46:26.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.344+0000 7fb6c0ac9700 1 -- 172.21.15.32:0/1135674617 shutdown_connections 2024-04-12T05:46:26.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.344+0000 7fb6c0ac9700 1 --2- 172.21.15.32:0/1135674617 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb6a406cc30 0x7fb6a406f0e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:26.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.344+0000 7fb6c0ac9700 1 --2- 172.21.15.32:0/1135674617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6bc0fdeb0 0x7fb6bc113a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:26.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.344+0000 7fb6c0ac9700 1 --2- 172.21.15.32:0/1135674617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6bc0fe850 0x7fb6bc113f50 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:26.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.344+0000 7fb6c0ac9700 1 -- 172.21.15.32:0/1135674617 >> 172.21.15.32:0/1135674617 conn(0x7fb6bc072bc0 msgr2=0x7fb6bc103780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:26.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.345+0000 7fb6c0ac9700 1 -- 172.21.15.32:0/1135674617 shutdown_connections 2024-04-12T05:46:26.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.345+0000 7fb6c0ac9700 1 -- 172.21.15.32:0/1135674617 wait complete. 2024-04-12T05:46:26.352 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:46:26.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.455+0000 7f1479cf8700 1 -- 172.21.15.32:0/2985587348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14741037b0 msgr2=0x7f14740fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:26.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.455+0000 7f1479cf8700 1 --2- 172.21.15.32:0/2985587348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14741037b0 0x7f14740fce20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1468004940 tx=0x7f1468009e30 comp rx=0 tx=0).stop 2024-04-12T05:46:26.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.455+0000 7f1479cf8700 1 -- 172.21.15.32:0/2985587348 shutdown_connections 2024-04-12T05:46:26.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.455+0000 7f1479cf8700 1 --2- 172.21.15.32:0/2985587348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14741037b0 0x7f14740fce20 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:26.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.455+0000 7f1479cf8700 1 --2- 172.21.15.32:0/2985587348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1474102e10 0x7f14741031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:26.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.455+0000 7f1479cf8700 1 -- 172.21.15.32:0/2985587348 >> 172.21.15.32:0/2985587348 conn(0x7f14740f8980 msgr2=0x7f14740fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:26.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.455+0000 7f1479cf8700 1 -- 172.21.15.32:0/2985587348 shutdown_connections 2024-04-12T05:46:26.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.455+0000 7f1479cf8700 1 -- 172.21.15.32:0/2985587348 wait complete. 2024-04-12T05:46:26.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.456+0000 7f1479cf8700 1 Processor -- start 2024-04-12T05:46:26.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.456+0000 7f1479cf8700 1 -- start start 2024-04-12T05:46:26.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.456+0000 7f1479cf8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1474102e10 0x7f1474113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:26.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.456+0000 7f1479cf8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14741037b0 0x7f1474114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:26.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.456+0000 7f1479cf8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1474114670 con 0x7f14741037b0 2024-04-12T05:46:26.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.456+0000 7f1479cf8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147410da60 con 0x7f1474102e10 2024-04-12T05:46:26.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.456+0000 7f1472ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14741037b0 0x7f1474114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:26.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.456+0000 7f1472ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14741037b0 0x7f1474114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41302/0 (socket says 172.21.15.32:41302) 2024-04-12T05:46:26.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.456+0000 7f1472ffd700 1 -- 172.21.15.32:0/3683570653 learned_addr learned my addr 172.21.15.32:0/3683570653 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:46:26.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.456+0000 7f14737fe700 1 --2- 172.21.15.32:0/3683570653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1474102e10 0x7f1474113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:26.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.456+0000 7f1472ffd700 1 -- 172.21.15.32:0/3683570653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1474102e10 msgr2=0x7f1474113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:26.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.456+0000 7f1472ffd700 1 --2- 172.21.15.32:0/3683570653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1474102e10 0x7f1474113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:26.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.457+0000 7f1472ffd700 1 -- 172.21.15.32:0/3683570653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1468005040 con 0x7f14741037b0 2024-04-12T05:46:26.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.457+0000 7f1472ffd700 1 --2- 172.21.15.32:0/3683570653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14741037b0 0x7f1474114020 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1468009380 tx=0x7f1468009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:26.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.457+0000 7f145bfff700 1 -- 172.21.15.32:0/3683570653 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f146800e040 con 0x7f14741037b0 2024-04-12T05:46:26.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.457+0000 7f1479cf8700 1 -- 172.21.15.32:0/3683570653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f147410dce0 con 0x7f14741037b0 2024-04-12T05:46:26.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.457+0000 7f145bfff700 1 -- 172.21.15.32:0/3683570653 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1468013420 con 0x7f14741037b0 2024-04-12T05:46:26.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.457+0000 7f145bfff700 1 -- 172.21.15.32:0/3683570653 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f146801c630 con 0x7f14741037b0 2024-04-12T05:46:26.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.457+0000 7f1479cf8700 1 -- 172.21.15.32:0/3683570653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f147410e2a0 con 0x7f14741037b0 2024-04-12T05:46:26.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.458+0000 7f1479cf8700 1 -- 172.21.15.32:0/3683570653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14740fe560 con 0x7f14741037b0 2024-04-12T05:46:26.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.458+0000 7f145bfff700 1 -- 172.21.15.32:0/3683570653 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f146801c8f0 con 0x7f14741037b0 2024-04-12T05:46:26.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.459+0000 7f145bfff700 1 --2- 172.21.15.32:0/3683570653 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f145c06cc20 0x7f145c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:26.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.459+0000 7f145bfff700 1 -- 172.21.15.32:0/3683570653 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f146808d230 con 0x7f14741037b0 2024-04-12T05:46:26.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.459+0000 7f14737fe700 1 --2- 172.21.15.32:0/3683570653 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f145c06cc20 0x7f145c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:26.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.460+0000 7f14737fe700 1 --2- 172.21.15.32:0/3683570653 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f145c06cc20 0x7f145c06f0d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1464004f40 tx=0x7f146400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:26.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.464+0000 7f145bfff700 1 -- 172.21.15.32:0/3683570653 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f146805ad80 con 0x7f14741037b0 2024-04-12T05:46:26.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.703+0000 7f1479cf8700 1 -- 172.21.15.32:0/3683570653 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1474109db0 con 0x7f145c06cc20 2024-04-12T05:46:26.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.704+0000 7f145bfff700 1 -- 172.21.15.32:0/3683570653 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1474109db0 con 0x7f145c06cc20 2024-04-12T05:46:26.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.706+0000 7f1479cf8700 1 -- 172.21.15.32:0/3683570653 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f145c06cc20 msgr2=0x7f145c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:26.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.706+0000 7f1479cf8700 1 --2- 172.21.15.32:0/3683570653 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f145c06cc20 0x7f145c06f0d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1464004f40 tx=0x7f146400b040 comp rx=0 tx=0).stop 2024-04-12T05:46:26.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.707+0000 7f1479cf8700 1 -- 172.21.15.32:0/3683570653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14741037b0 msgr2=0x7f1474114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:26.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.707+0000 7f1479cf8700 1 --2- 172.21.15.32:0/3683570653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14741037b0 0x7f1474114020 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1468009380 tx=0x7f1468009e30 comp rx=0 tx=0).stop 2024-04-12T05:46:26.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.707+0000 7f1479cf8700 1 -- 172.21.15.32:0/3683570653 shutdown_connections 2024-04-12T05:46:26.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.707+0000 7f1479cf8700 1 --2- 172.21.15.32:0/3683570653 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f145c06cc20 0x7f145c06f0d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:26.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.707+0000 7f1479cf8700 1 --2- 172.21.15.32:0/3683570653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1474102e10 0x7f1474113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:26.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.707+0000 7f1479cf8700 1 --2- 172.21.15.32:0/3683570653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14741037b0 0x7f1474114020 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:26.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.707+0000 7f1479cf8700 1 -- 172.21.15.32:0/3683570653 >> 172.21.15.32:0/3683570653 conn(0x7f14740f8980 msgr2=0x7f14740f9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:26.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.707+0000 7f1479cf8700 1 -- 172.21.15.32:0/3683570653 shutdown_connections 2024-04-12T05:46:26.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.707+0000 7f1479cf8700 1 -- 172.21.15.32:0/3683570653 wait complete. 2024-04-12T05:46:26.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.816+0000 7f6e57574700 1 -- 172.21.15.32:0/974227639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e500fde10 msgr2=0x7f6e500fe1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:26.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.816+0000 7f6e57574700 1 --2- 172.21.15.32:0/974227639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e500fde10 0x7f6e500fe1e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c004940 tx=0x7f6e4c009e30 comp rx=0 tx=0).stop 2024-04-12T05:46:26.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.816+0000 7f6e57574700 1 -- 172.21.15.32:0/974227639 shutdown_connections 2024-04-12T05:46:26.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.816+0000 7f6e57574700 1 --2- 172.21.15.32:0/974227639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e500fe7b0 0x7f6e50106df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:26.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.816+0000 7f6e57574700 1 --2- 172.21.15.32:0/974227639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e500fde10 0x7f6e500fe1e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:26.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.816+0000 7f6e57574700 1 -- 172.21.15.32:0/974227639 >> 172.21.15.32:0/974227639 conn(0x7f6e500f7880 msgr2=0x7f6e500f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:26.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.817+0000 7f6e57574700 1 -- 172.21.15.32:0/974227639 shutdown_connections 2024-04-12T05:46:26.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.817+0000 7f6e57574700 1 -- 172.21.15.32:0/974227639 wait complete. 2024-04-12T05:46:26.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.817+0000 7f6e57574700 1 Processor -- start 2024-04-12T05:46:26.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.817+0000 7f6e57574700 1 -- start start 2024-04-12T05:46:26.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.818+0000 7f6e57574700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e500fde10 0x7f6e50113de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:26.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.818+0000 7f6e57574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e500fe7b0 0x7f6e50114320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:26.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.818+0000 7f6e57574700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e50114970 con 0x7f6e500fe7b0 2024-04-12T05:46:26.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.818+0000 7f6e57574700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5010dd40 con 0x7f6e500fde10 2024-04-12T05:46:26.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.818+0000 7f6e54b0f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e500fe7b0 0x7f6e50114320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:26.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.818+0000 7f6e54b0f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e500fe7b0 0x7f6e50114320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41314/0 (socket says 172.21.15.32:41314) 2024-04-12T05:46:26.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.818+0000 7f6e54b0f700 1 -- 172.21.15.32:0/3273344022 learned_addr learned my addr 172.21.15.32:0/3273344022 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:46:26.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.818+0000 7f6e55310700 1 --2- 172.21.15.32:0/3273344022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e500fde10 0x7f6e50113de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:26.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.818+0000 7f6e54b0f700 1 -- 172.21.15.32:0/3273344022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e500fde10 msgr2=0x7f6e50113de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:26.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.818+0000 7f6e54b0f700 1 --2- 172.21.15.32:0/3273344022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e500fde10 0x7f6e50113de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:26.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.818+0000 7f6e54b0f700 1 -- 172.21.15.32:0/3273344022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4c005040 con 0x7f6e500fe7b0 2024-04-12T05:46:26.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.818+0000 7f6e55310700 1 --2- 172.21.15.32:0/3273344022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e500fde10 0x7f6e50113de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:46:26.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.818+0000 7f6e54b0f700 1 --2- 172.21.15.32:0/3273344022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e500fe7b0 0x7f6e50114320 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6e4000c8c0 tx=0x7f6e400044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:26.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.818+0000 7f6e45ffb700 1 -- 172.21.15.32:0/3273344022 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e4000a980 con 0x7f6e500fe7b0 2024-04-12T05:46:26.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.818+0000 7f6e57574700 1 -- 172.21.15.32:0/3273344022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e5010e020 con 0x7f6e500fe7b0 2024-04-12T05:46:26.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.818+0000 7f6e45ffb700 1 -- 172.21.15.32:0/3273344022 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6e4000ce60 con 0x7f6e500fe7b0 2024-04-12T05:46:26.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.818+0000 7f6e45ffb700 1 -- 172.21.15.32:0/3273344022 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e400113a0 con 0x7f6e500fe7b0 2024-04-12T05:46:26.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.819+0000 7f6e57574700 1 -- 172.21.15.32:0/3273344022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e5010e560 con 0x7f6e500fe7b0 2024-04-12T05:46:26.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.820+0000 7f6e57574700 1 -- 172.21.15.32:0/3273344022 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e50066770 con 0x7f6e500fe7b0 2024-04-12T05:46:26.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.820+0000 7f6e45ffb700 1 -- 172.21.15.32:0/3273344022 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f6e40012400 con 0x7f6e500fe7b0 2024-04-12T05:46:26.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.820+0000 7f6e45ffb700 1 --2- 172.21.15.32:0/3273344022 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6e3c06cc20 0x7f6e3c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:26.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.820+0000 7f6e45ffb700 1 -- 172.21.15.32:0/3273344022 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f6e4008c1c0 con 0x7f6e500fe7b0 2024-04-12T05:46:26.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.821+0000 7f6e55310700 1 --2- 172.21.15.32:0/3273344022 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6e3c06cc20 0x7f6e3c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:26.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.821+0000 7f6e55310700 1 --2- 172.21.15.32:0/3273344022 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6e3c06cc20 0x7f6e3c06f0d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c00b3b0 tx=0x7f6e4c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:26.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:26.825+0000 7f6e45ffb700 1 -- 172.21.15.32:0/3273344022 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6e40059c30 con 0x7f6e500fe7b0 2024-04-12T05:46:26.959 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:26 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:26.959 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:26 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:26.959 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:26 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:26.959 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:26 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:27.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:26 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:27.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:26 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:27.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:26 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:27.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:26 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:27.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.045+0000 7f6e57574700 1 -- 172.21.15.32:0/3273344022 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e50109d60 con 0x7f6e3c06cc20 2024-04-12T05:46:27.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.055+0000 7f6e45ffb700 1 -- 172.21.15.32:0/3273344022 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f6e50109d60 con 0x7f6e3c06cc20 2024-04-12T05:46:27.052 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:46:27.052 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (6m) 1s ago 17m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:46:27.052 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (3m) 1s ago 17m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:46:27.052 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (3m) 22s ago 16m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:46:27.052 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (5m) 1s ago 17m 88.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:46:27.052 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (7s) 1s ago 12m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T05:46:27.052 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (11m) 1s ago 11m 10.5M - 17.2.7-827-gd3b5060a 21de9b7369ad 55124da9afdb 2024-04-12T05:46:27.052 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (11m) 22s ago 11m 62.6M - 17.2.7-827-gd3b5060a 21de9b7369ad 0c295d31a4a6 2024-04-12T05:46:27.052 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (11m) 22s ago 11m 120M - 17.2.7-827-gd3b5060a 21de9b7369ad 6eecd7deaac2 2024-04-12T05:46:27.053 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (7m) 1s ago 19m 506M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:46:27.053 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (7m) 22s ago 16m 449M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:46:27.053 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (4m) 1s ago 19m 53.5M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:46:27.053 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (4m) 22s ago 16m 39.5M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:46:27.053 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (6m) 1s ago 17m 16.8M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:46:27.053 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (6m) 22s ago 16m 18.3M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:46:27.053 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (3m) 1s ago 15m 90.7M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T05:46:27.053 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (2m) 1s ago 15m 97.7M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T05:46:27.053 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (2m) 1s ago 14m 81.7M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T05:46:27.053 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (92s) 22s ago 14m 112M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T05:46:27.053 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (59s) 22s ago 14m 90.4M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T05:46:27.053 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (25s) 22s ago 13m 10.9M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T05:46:27.053 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (6m) 1s ago 17m 64.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:46:27.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.057+0000 7f6e57574700 1 -- 172.21.15.32:0/3273344022 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6e3c06cc20 msgr2=0x7f6e3c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:27.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.057+0000 7f6e57574700 1 --2- 172.21.15.32:0/3273344022 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6e3c06cc20 0x7f6e3c06f0d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c00b3b0 tx=0x7f6e4c009e30 comp rx=0 tx=0).stop 2024-04-12T05:46:27.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.057+0000 7f6e57574700 1 -- 172.21.15.32:0/3273344022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e500fe7b0 msgr2=0x7f6e50114320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:27.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.057+0000 7f6e57574700 1 --2- 172.21.15.32:0/3273344022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e500fe7b0 0x7f6e50114320 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6e4000c8c0 tx=0x7f6e400044c0 comp rx=0 tx=0).stop 2024-04-12T05:46:27.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.057+0000 7f6e57574700 1 -- 172.21.15.32:0/3273344022 shutdown_connections 2024-04-12T05:46:27.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.057+0000 7f6e57574700 1 --2- 172.21.15.32:0/3273344022 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6e3c06cc20 0x7f6e3c06f0d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:27.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.057+0000 7f6e57574700 1 --2- 172.21.15.32:0/3273344022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e500fde10 0x7f6e50113de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:27.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.057+0000 7f6e57574700 1 --2- 172.21.15.32:0/3273344022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e500fe7b0 0x7f6e50114320 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:27.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.057+0000 7f6e57574700 1 -- 172.21.15.32:0/3273344022 >> 172.21.15.32:0/3273344022 conn(0x7f6e500f7880 msgr2=0x7f6e501030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:27.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.058+0000 7f6e57574700 1 -- 172.21.15.32:0/3273344022 shutdown_connections 2024-04-12T05:46:27.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.058+0000 7f6e57574700 1 -- 172.21.15.32:0/3273344022 wait complete. 2024-04-12T05:46:27.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.163+0000 7f5d75f1d700 1 -- 172.21.15.32:0/1042811726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d70102710 msgr2=0x7f5d700fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:27.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.163+0000 7f5d75f1d700 1 --2- 172.21.15.32:0/1042811726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d70102710 0x7f5d700fbd80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5d64004940 tx=0x7f5d64009e30 comp rx=0 tx=0).stop 2024-04-12T05:46:27.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.163+0000 7f5d75f1d700 1 -- 172.21.15.32:0/1042811726 shutdown_connections 2024-04-12T05:46:27.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.163+0000 7f5d75f1d700 1 --2- 172.21.15.32:0/1042811726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d70102710 0x7f5d700fbd80 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:27.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.163+0000 7f5d75f1d700 1 --2- 172.21.15.32:0/1042811726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d70101d70 0x7f5d70102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:27.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.163+0000 7f5d75f1d700 1 -- 172.21.15.32:0/1042811726 >> 172.21.15.32:0/1042811726 conn(0x7f5d70072bc0 msgr2=0x7f5d70072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:27.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.164+0000 7f5d75f1d700 1 -- 172.21.15.32:0/1042811726 shutdown_connections 2024-04-12T05:46:27.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.164+0000 7f5d75f1d700 1 -- 172.21.15.32:0/1042811726 wait complete. 2024-04-12T05:46:27.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.164+0000 7f5d75f1d700 1 Processor -- start 2024-04-12T05:46:27.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.164+0000 7f5d75f1d700 1 -- start start 2024-04-12T05:46:27.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.164+0000 7f5d75f1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d70101d70 0x7f5d70102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:27.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.164+0000 7f5d75f1d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d70102710 0x7f5d70103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:27.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.164+0000 7f5d75f1d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d70103650 con 0x7f5d70101d70 2024-04-12T05:46:27.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.165+0000 7f5d75f1d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d701037c0 con 0x7f5d70102710 2024-04-12T05:46:27.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.165+0000 7f5d6f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d70101d70 0x7f5d70102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:27.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.165+0000 7f5d6f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d70101d70 0x7f5d70102bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41326/0 (socket says 172.21.15.32:41326) 2024-04-12T05:46:27.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.165+0000 7f5d6f7fe700 1 -- 172.21.15.32:0/3606391219 learned_addr learned my addr 172.21.15.32:0/3606391219 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:46:27.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.165+0000 7f5d6effd700 1 --2- 172.21.15.32:0/3606391219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d70102710 0x7f5d70103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:27.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.165+0000 7f5d6f7fe700 1 -- 172.21.15.32:0/3606391219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d70102710 msgr2=0x7f5d70103110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:27.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.165+0000 7f5d6f7fe700 1 --2- 172.21.15.32:0/3606391219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d70102710 0x7f5d70103110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:27.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.165+0000 7f5d6f7fe700 1 -- 172.21.15.32:0/3606391219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d64005040 con 0x7f5d70101d70 2024-04-12T05:46:27.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.165+0000 7f5d6effd700 1 --2- 172.21.15.32:0/3606391219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d70102710 0x7f5d70103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:46:27.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.165+0000 7f5d6f7fe700 1 --2- 172.21.15.32:0/3606391219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d70101d70 0x7f5d70102bd0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5d6000eb40 tx=0x7f5d6000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:27.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.165+0000 7f5d57fff700 1 -- 172.21.15.32:0/3606391219 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d60007cc0 con 0x7f5d70101d70 2024-04-12T05:46:27.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.165+0000 7f5d57fff700 1 -- 172.21.15.32:0/3606391219 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d6000f040 con 0x7f5d70101d70 2024-04-12T05:46:27.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.165+0000 7f5d75f1d700 1 -- 172.21.15.32:0/3606391219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d70103a40 con 0x7f5d70101d70 2024-04-12T05:46:27.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.165+0000 7f5d57fff700 1 -- 172.21.15.32:0/3606391219 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d60009570 con 0x7f5d70101d70 2024-04-12T05:46:27.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.165+0000 7f5d75f1d700 1 -- 172.21.15.32:0/3606391219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d7011a0e0 con 0x7f5d70101d70 2024-04-12T05:46:27.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.166+0000 7f5d75f1d700 1 -- 172.21.15.32:0/3606391219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d700fd4c0 con 0x7f5d70101d70 2024-04-12T05:46:27.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.167+0000 7f5d57fff700 1 -- 172.21.15.32:0/3606391219 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f5d600096d0 con 0x7f5d70101d70 2024-04-12T05:46:27.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.167+0000 7f5d57fff700 1 --2- 172.21.15.32:0/3606391219 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5d5806cc20 0x7f5d5806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:27.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.167+0000 7f5d57fff700 1 -- 172.21.15.32:0/3606391219 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f5d6008bc70 con 0x7f5d70101d70 2024-04-12T05:46:27.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.167+0000 7f5d6effd700 1 --2- 172.21.15.32:0/3606391219 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5d5806cc20 0x7f5d5806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:27.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.168+0000 7f5d6effd700 1 --2- 172.21.15.32:0/3606391219 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5d5806cc20 0x7f5d5806f0d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5d6400b0d0 tx=0x7f5d64009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:27.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.172+0000 7f5d57fff700 1 -- 172.21.15.32:0/3606391219 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5d60092080 con 0x7f5d70101d70 2024-04-12T05:46:27.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.460+0000 7f5d75f1d700 1 -- 172.21.15.32:0/3606391219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5d701039e0 con 0x7f5d70101d70 2024-04-12T05:46:27.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.461+0000 7f5d57fff700 1 -- 172.21.15.32:0/3606391219 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f5d60055be0 con 0x7f5d70101d70 2024-04-12T05:46:27.458 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:46:27.458 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:46:27.458 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:46:27.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:46:27.458 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:46:27.458 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:46:27.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:46:27.458 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:46:27.458 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:46:27.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:46:27.458 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:46:27.458 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 3, 2024-04-12T05:46:27.458 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 1 2024-04-12T05:46:27.459 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:46:27.459 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:46:27.459 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 3, 2024-04-12T05:46:27.459 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 11 2024-04-12T05:46:27.459 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:46:27.459 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:46:27.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.463+0000 7f5d75f1d700 1 -- 172.21.15.32:0/3606391219 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5d5806cc20 msgr2=0x7f5d5806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:27.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.463+0000 7f5d75f1d700 1 --2- 172.21.15.32:0/3606391219 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5d5806cc20 0x7f5d5806f0d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5d6400b0d0 tx=0x7f5d64009e30 comp rx=0 tx=0).stop 2024-04-12T05:46:27.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.463+0000 7f5d75f1d700 1 -- 172.21.15.32:0/3606391219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d70101d70 msgr2=0x7f5d70102bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:27.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.463+0000 7f5d75f1d700 1 --2- 172.21.15.32:0/3606391219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d70101d70 0x7f5d70102bd0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5d6000eb40 tx=0x7f5d6000b040 comp rx=0 tx=0).stop 2024-04-12T05:46:27.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.463+0000 7f5d75f1d700 1 -- 172.21.15.32:0/3606391219 shutdown_connections 2024-04-12T05:46:27.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.463+0000 7f5d75f1d700 1 --2- 172.21.15.32:0/3606391219 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5d5806cc20 0x7f5d5806f0d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:27.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.463+0000 7f5d75f1d700 1 --2- 172.21.15.32:0/3606391219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d70101d70 0x7f5d70102bd0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:27.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.463+0000 7f5d75f1d700 1 --2- 172.21.15.32:0/3606391219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d70102710 0x7f5d70103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:27.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.463+0000 7f5d75f1d700 1 -- 172.21.15.32:0/3606391219 >> 172.21.15.32:0/3606391219 conn(0x7f5d70072bc0 msgr2=0x7f5d70065940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:27.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.463+0000 7f5d75f1d700 1 -- 172.21.15.32:0/3606391219 shutdown_connections 2024-04-12T05:46:27.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.463+0000 7f5d75f1d700 1 -- 172.21.15.32:0/3606391219 wait complete. 2024-04-12T05:46:27.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.569+0000 7eff1d295700 1 -- 172.21.15.32:0/2620891613 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff18102e10 msgr2=0x7eff181031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:27.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.569+0000 7eff1d295700 1 --2- 172.21.15.32:0/2620891613 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff18102e10 0x7eff181031e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7eff08004940 tx=0x7eff08009e30 comp rx=0 tx=0).stop 2024-04-12T05:46:27.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.569+0000 7eff1d295700 1 -- 172.21.15.32:0/2620891613 shutdown_connections 2024-04-12T05:46:27.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.569+0000 7eff1d295700 1 --2- 172.21.15.32:0/2620891613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff181037b0 0x7eff180fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:27.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.569+0000 7eff1d295700 1 --2- 172.21.15.32:0/2620891613 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff18102e10 0x7eff181031e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:27.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.569+0000 7eff1d295700 1 -- 172.21.15.32:0/2620891613 >> 172.21.15.32:0/2620891613 conn(0x7eff180f8980 msgr2=0x7eff180fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:27.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.569+0000 7eff1d295700 1 -- 172.21.15.32:0/2620891613 shutdown_connections 2024-04-12T05:46:27.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.569+0000 7eff1d295700 1 -- 172.21.15.32:0/2620891613 wait complete. 2024-04-12T05:46:27.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.570+0000 7eff1d295700 1 Processor -- start 2024-04-12T05:46:27.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.570+0000 7eff1d295700 1 -- start start 2024-04-12T05:46:27.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.570+0000 7eff1d295700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff18102e10 0x7eff18113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:27.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.570+0000 7eff1d295700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff181037b0 0x7eff18113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:27.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.570+0000 7eff1d295700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff181145f0 con 0x7eff18102e10 2024-04-12T05:46:27.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.570+0000 7eff1d295700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1810d9e0 con 0x7eff181037b0 2024-04-12T05:46:27.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.570+0000 7eff16ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff18102e10 0x7eff18113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:27.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.570+0000 7eff16ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff18102e10 0x7eff18113a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41340/0 (socket says 172.21.15.32:41340) 2024-04-12T05:46:27.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.570+0000 7eff16ffd700 1 -- 172.21.15.32:0/2936612583 learned_addr learned my addr 172.21.15.32:0/2936612583 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:46:27.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.570+0000 7eff167fc700 1 --2- 172.21.15.32:0/2936612583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff181037b0 0x7eff18113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:27.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.571+0000 7eff16ffd700 1 -- 172.21.15.32:0/2936612583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff181037b0 msgr2=0x7eff18113fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:27.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.571+0000 7eff16ffd700 1 --2- 172.21.15.32:0/2936612583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff181037b0 0x7eff18113fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:27.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.571+0000 7eff16ffd700 1 -- 172.21.15.32:0/2936612583 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff08005040 con 0x7eff18102e10 2024-04-12T05:46:27.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.571+0000 7eff16ffd700 1 --2- 172.21.15.32:0/2936612583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff18102e10 0x7eff18113a60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7eff08008d80 tx=0x7eff08009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:27.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.571+0000 7efeff7fe700 1 -- 172.21.15.32:0/2936612583 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff0800e030 con 0x7eff18102e10 2024-04-12T05:46:27.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.571+0000 7efeff7fe700 1 -- 172.21.15.32:0/2936612583 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff08013de0 con 0x7eff18102e10 2024-04-12T05:46:27.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.571+0000 7eff1d295700 1 -- 172.21.15.32:0/2936612583 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff1810dc60 con 0x7eff18102e10 2024-04-12T05:46:27.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.571+0000 7efeff7fe700 1 -- 172.21.15.32:0/2936612583 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff0801cd60 con 0x7eff18102e10 2024-04-12T05:46:27.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.571+0000 7eff1d295700 1 -- 172.21.15.32:0/2936612583 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff1810e220 con 0x7eff18102e10 2024-04-12T05:46:27.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.572+0000 7eff1d295700 1 -- 172.21.15.32:0/2936612583 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff180fe560 con 0x7eff18102e10 2024-04-12T05:46:27.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.572+0000 7efeff7fe700 1 -- 172.21.15.32:0/2936612583 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7eff0801c590 con 0x7eff18102e10 2024-04-12T05:46:27.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.573+0000 7efeff7fe700 1 --2- 172.21.15.32:0/2936612583 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7eff0006cc30 0x7eff0006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:27.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.573+0000 7efeff7fe700 1 -- 172.21.15.32:0/2936612583 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6582+0+0 (secure 0 0 0) 0x7eff0808d990 con 0x7eff18102e10 2024-04-12T05:46:27.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.573+0000 7eff167fc700 1 --2- 172.21.15.32:0/2936612583 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7eff0006cc30 0x7eff0006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:27.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.573+0000 7eff167fc700 1 --2- 172.21.15.32:0/2936612583 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7eff0006cc30 0x7eff0006f0e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7eff0c004600 tx=0x7eff0c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:27.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.577+0000 7efeff7fe700 1 -- 172.21.15.32:0/2936612583 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7eff0805b400 con 0x7eff18102e10 2024-04-12T05:46:27.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.828+0000 7eff1d295700 1 -- 172.21.15.32:0/2936612583 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7eff1804bed0 con 0x7eff18102e10 2024-04-12T05:46:27.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.828+0000 7efeff7fe700 1 -- 172.21.15.32:0/2936612583 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1835 (secure 0 0 0) 0x7eff0805af90 con 0x7eff18102e10 2024-04-12T05:46:27.825 INFO:teuthology.orchestra.run.smithi032.stdout:e27 2024-04-12T05:46:27.825 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:46:27.825 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:46:27.825 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:46:27.825 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:46:27.826 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:46:27.826 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:46:27.826 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 24 2024-04-12T05:46:27.826 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:46:27.826 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:46:27.826 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:46:10.668569+0000 2024-04-12T05:46:27.826 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:46:27.826 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:46:27.826 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:46:27.826 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:46:27.826 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:46:27.826 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:46:27.826 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:46:27.826 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:46:27.827 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T05:46:27.827 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:46:27.827 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:46:27.827 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:46:27.827 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T05:46:27.827 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-12T05:46:27.827 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:46:27.827 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:46:27.827 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:46:27.827 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:46:27.827 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:46:27.827 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:46:27.827 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:46:27.827 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:46:27.828 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:46:27.828 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:46:27.828 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-12T05:46:27.828 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:46:27.828 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/2545444205,v1:172.21.15.32:6829/2545444205] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:46:27.828 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:46:27.828 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{-1:44275} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/3573641538,v1:172.21.15.67:6827/3573641538] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:46:27.828 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/973024107,v1:172.21.15.67:6825/973024107] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:46:27.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.831+0000 7eff1d295700 1 -- 172.21.15.32:0/2936612583 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7eff0006cc30 msgr2=0x7eff0006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:27.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.831+0000 7eff1d295700 1 --2- 172.21.15.32:0/2936612583 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7eff0006cc30 0x7eff0006f0e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7eff0c004600 tx=0x7eff0c00b040 comp rx=0 tx=0).stop 2024-04-12T05:46:27.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.831+0000 7eff1d295700 1 -- 172.21.15.32:0/2936612583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff18102e10 msgr2=0x7eff18113a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:27.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.831+0000 7eff1d295700 1 --2- 172.21.15.32:0/2936612583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff18102e10 0x7eff18113a60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7eff08008d80 tx=0x7eff08009e30 comp rx=0 tx=0).stop 2024-04-12T05:46:27.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.831+0000 7eff1d295700 1 -- 172.21.15.32:0/2936612583 shutdown_connections 2024-04-12T05:46:27.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.831+0000 7eff1d295700 1 --2- 172.21.15.32:0/2936612583 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7eff0006cc30 0x7eff0006f0e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:27.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.831+0000 7eff1d295700 1 --2- 172.21.15.32:0/2936612583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff18102e10 0x7eff18113a60 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:27.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.831+0000 7eff1d295700 1 --2- 172.21.15.32:0/2936612583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff181037b0 0x7eff18113fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:27.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.831+0000 7eff1d295700 1 -- 172.21.15.32:0/2936612583 >> 172.21.15.32:0/2936612583 conn(0x7eff180f8980 msgr2=0x7eff180f9970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:27.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.831+0000 7eff1d295700 1 -- 172.21.15.32:0/2936612583 shutdown_connections 2024-04-12T05:46:27.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.831+0000 7eff1d295700 1 -- 172.21.15.32:0/2936612583 wait complete. 2024-04-12T05:46:27.829 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 27 2024-04-12T05:46:27.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:27 smithi032 ceph-mon[146817]: pgmap v175: 65 pgs: 65 active+clean; 270 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:27.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:27 smithi032 ceph-mon[146817]: from='client.34388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:46:27.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:27 smithi032 ceph-mon[146817]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:46:27.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:27 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3606391219' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:27.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.936+0000 7f3603f66700 1 -- 172.21.15.32:0/770288968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35fc0fded0 msgr2=0x7f35fc0fe2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:27.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.936+0000 7f3603f66700 1 --2- 172.21.15.32:0/770288968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35fc0fded0 0x7f35fc0fe2a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f35f8004940 tx=0x7f35f8009e30 comp rx=0 tx=0).stop 2024-04-12T05:46:27.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.936+0000 7f3603f66700 1 -- 172.21.15.32:0/770288968 shutdown_connections 2024-04-12T05:46:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.936+0000 7f3603f66700 1 --2- 172.21.15.32:0/770288968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35fc0fe870 0x7f35fc106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.936+0000 7f3603f66700 1 --2- 172.21.15.32:0/770288968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35fc0fded0 0x7f35fc0fe2a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.936+0000 7f3603f66700 1 -- 172.21.15.32:0/770288968 >> 172.21.15.32:0/770288968 conn(0x7f35fc072bc0 msgr2=0x7f35fc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.936+0000 7f3603f66700 1 -- 172.21.15.32:0/770288968 shutdown_connections 2024-04-12T05:46:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.936+0000 7f3603f66700 1 -- 172.21.15.32:0/770288968 wait complete. 2024-04-12T05:46:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.937+0000 7f3603f66700 1 Processor -- start 2024-04-12T05:46:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.937+0000 7f3603f66700 1 -- start start 2024-04-12T05:46:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.937+0000 7f3603f66700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35fc0fded0 0x7f35fc0fbd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.937+0000 7f3603f66700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35fc0fe870 0x7f35fc0fc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.937+0000 7f3603f66700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35fc0fc860 con 0x7f35fc0fe870 2024-04-12T05:46:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.937+0000 7f3603f66700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35fc0fc9a0 con 0x7f35fc0fded0 2024-04-12T05:46:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.937+0000 7f3601501700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35fc0fe870 0x7f35fc0fc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.937+0000 7f3601501700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35fc0fe870 0x7f35fc0fc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41372/0 (socket says 172.21.15.32:41372) 2024-04-12T05:46:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.937+0000 7f3601501700 1 -- 172.21.15.32:0/1586339222 learned_addr learned my addr 172.21.15.32:0/1586339222 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:46:27.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.937+0000 7f3601d02700 1 --2- 172.21.15.32:0/1586339222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35fc0fded0 0x7f35fc0fbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:27.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.937+0000 7f3601501700 1 -- 172.21.15.32:0/1586339222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35fc0fded0 msgr2=0x7f35fc0fbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:27.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.937+0000 7f3601501700 1 --2- 172.21.15.32:0/1586339222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35fc0fded0 0x7f35fc0fbd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:27.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.937+0000 7f3601501700 1 -- 172.21.15.32:0/1586339222 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f8005040 con 0x7f35fc0fe870 2024-04-12T05:46:27.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.937+0000 7f3601501700 1 --2- 172.21.15.32:0/1586339222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35fc0fe870 0x7f35fc0fc2d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f35f000c830 tx=0x7f35f0004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:27.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.938+0000 7f35ee7fc700 1 -- 172.21.15.32:0/1586339222 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35f000a760 con 0x7f35fc0fe870 2024-04-12T05:46:27.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.938+0000 7f35ee7fc700 1 -- 172.21.15.32:0/1586339222 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f35f000ad70 con 0x7f35fc0fe870 2024-04-12T05:46:27.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.938+0000 7f3603f66700 1 -- 172.21.15.32:0/1586339222 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35fc119d20 con 0x7f35fc0fe870 2024-04-12T05:46:27.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.938+0000 7f35ee7fc700 1 -- 172.21.15.32:0/1586339222 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35f00113a0 con 0x7f35fc0fe870 2024-04-12T05:46:27.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.938+0000 7f3603f66700 1 -- 172.21.15.32:0/1586339222 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35fc11a120 con 0x7f35fc0fe870 2024-04-12T05:46:27.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.939+0000 7f3603f66700 1 -- 172.21.15.32:0/1586339222 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35fc065df0 con 0x7f35fc0fe870 2024-04-12T05:46:27.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.940+0000 7f35ee7fc700 1 -- 172.21.15.32:0/1586339222 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f35f0012400 con 0x7f35fc0fe870 2024-04-12T05:46:27.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.940+0000 7f35ee7fc700 1 --2- 172.21.15.32:0/1586339222 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f35e806cc20 0x7f35e806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:27.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.940+0000 7f35ee7fc700 1 -- 172.21.15.32:0/1586339222 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f35f008cbe0 con 0x7f35fc0fe870 2024-04-12T05:46:27.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.940+0000 7f3601d02700 1 --2- 172.21.15.32:0/1586339222 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f35e806cc20 0x7f35e806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:27.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.941+0000 7f3601d02700 1 --2- 172.21.15.32:0/1586339222 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f35e806cc20 0x7f35e806f0d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f35f8004560 tx=0x7f35f8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:27.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:27.944+0000 7f35ee7fc700 1 -- 172.21.15.32:0/1586339222 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f35f0056c60 con 0x7f35fc0fe870 2024-04-12T05:46:28.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:27 smithi067 ceph-mon[112979]: pgmap v175: 65 pgs: 65 active+clean; 270 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:28.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:27 smithi067 ceph-mon[112979]: from='client.34388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:46:28.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:27 smithi067 ceph-mon[112979]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:46:28.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:27 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3606391219' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:28.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.168+0000 7f3603f66700 1 -- 172.21.15.32:0/1586339222 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f35fc04bf60 con 0x7f35e806cc20 2024-04-12T05:46:28.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.170+0000 7f35ee7fc700 1 -- 172.21.15.32:0/1586339222 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f35fc04bf60 con 0x7f35e806cc20 2024-04-12T05:46:28.166 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:46:28.166 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:46:28.166 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:46:28.167 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:46:28.167 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:46:28.167 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:46:28.167 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:46:28.167 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-12T05:46:28.167 INFO:teuthology.orchestra.run.smithi032.stdout: "osd" 2024-04-12T05:46:28.167 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:46:28.167 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "13/21 daemons upgraded", 2024-04-12T05:46:28.167 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:46:28.167 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:46:28.167 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:46:28.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.172+0000 7f3603f66700 1 -- 172.21.15.32:0/1586339222 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f35e806cc20 msgr2=0x7f35e806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:28.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.172+0000 7f3603f66700 1 --2- 172.21.15.32:0/1586339222 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f35e806cc20 0x7f35e806f0d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f35f8004560 tx=0x7f35f8009e30 comp rx=0 tx=0).stop 2024-04-12T05:46:28.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.172+0000 7f3603f66700 1 -- 172.21.15.32:0/1586339222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35fc0fe870 msgr2=0x7f35fc0fc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:28.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.172+0000 7f3603f66700 1 --2- 172.21.15.32:0/1586339222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35fc0fe870 0x7f35fc0fc2d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f35f000c830 tx=0x7f35f0004470 comp rx=0 tx=0).stop 2024-04-12T05:46:28.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.172+0000 7f3603f66700 1 -- 172.21.15.32:0/1586339222 shutdown_connections 2024-04-12T05:46:28.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.172+0000 7f3603f66700 1 --2- 172.21.15.32:0/1586339222 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f35e806cc20 0x7f35e806f0d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:28.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.172+0000 7f3603f66700 1 --2- 172.21.15.32:0/1586339222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35fc0fded0 0x7f35fc0fbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:28.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.172+0000 7f3603f66700 1 --2- 172.21.15.32:0/1586339222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35fc0fe870 0x7f35fc0fc2d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:28.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.172+0000 7f3603f66700 1 -- 172.21.15.32:0/1586339222 >> 172.21.15.32:0/1586339222 conn(0x7f35fc072bc0 msgr2=0x7f35fc0ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:28.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.172+0000 7f3603f66700 1 -- 172.21.15.32:0/1586339222 shutdown_connections 2024-04-12T05:46:28.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.172+0000 7f3603f66700 1 -- 172.21.15.32:0/1586339222 wait complete. 2024-04-12T05:46:28.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.281+0000 7f4581e85700 1 -- 172.21.15.32:0/3312012649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f457c1026b0 msgr2=0x7f457c0fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:28.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.281+0000 7f4581e85700 1 --2- 172.21.15.32:0/3312012649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f457c1026b0 0x7f457c0fbd20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4570004940 tx=0x7f4570009e30 comp rx=0 tx=0).stop 2024-04-12T05:46:28.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.281+0000 7f4581e85700 1 -- 172.21.15.32:0/3312012649 shutdown_connections 2024-04-12T05:46:28.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.281+0000 7f4581e85700 1 --2- 172.21.15.32:0/3312012649 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f457c1026b0 0x7f457c0fbd20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:28.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.281+0000 7f4581e85700 1 --2- 172.21.15.32:0/3312012649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f457c101d10 0x7f457c1020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:28.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.281+0000 7f4581e85700 1 -- 172.21.15.32:0/3312012649 >> 172.21.15.32:0/3312012649 conn(0x7f457c0f7880 msgr2=0x7f457c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:28.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.281+0000 7f4581e85700 1 -- 172.21.15.32:0/3312012649 shutdown_connections 2024-04-12T05:46:28.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.281+0000 7f4581e85700 1 -- 172.21.15.32:0/3312012649 wait complete. 2024-04-12T05:46:28.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.281+0000 7f4581e85700 1 Processor -- start 2024-04-12T05:46:28.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.282+0000 7f4581e85700 1 -- start start 2024-04-12T05:46:28.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.282+0000 7f4581e85700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f457c101d10 0x7f457c10ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:28.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.282+0000 7f4581e85700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f457c1026b0 0x7f457c10f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:28.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.282+0000 7f4581e85700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457c10f890 con 0x7f457c1026b0 2024-04-12T05:46:28.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.282+0000 7f4581e85700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457c10f9d0 con 0x7f457c101d10 2024-04-12T05:46:28.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.282+0000 7f457affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f457c1026b0 0x7f457c10f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:28.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.282+0000 7f457affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f457c1026b0 0x7f457c10f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41388/0 (socket says 172.21.15.32:41388) 2024-04-12T05:46:28.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.282+0000 7f457affd700 1 -- 172.21.15.32:0/918243957 learned_addr learned my addr 172.21.15.32:0/918243957 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:46:28.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.282+0000 7f457b7fe700 1 --2- 172.21.15.32:0/918243957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f457c101d10 0x7f457c10ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:28.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.282+0000 7f457affd700 1 -- 172.21.15.32:0/918243957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f457c101d10 msgr2=0x7f457c10ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:28.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.282+0000 7f457affd700 1 --2- 172.21.15.32:0/918243957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f457c101d10 0x7f457c10ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:28.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.282+0000 7f457affd700 1 -- 172.21.15.32:0/918243957 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4570005040 con 0x7f457c1026b0 2024-04-12T05:46:28.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.283+0000 7f457affd700 1 --2- 172.21.15.32:0/918243957 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f457c1026b0 0x7f457c10f240 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4570000c00 tx=0x7f4570009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:28.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.283+0000 7f4563fff700 1 -- 172.21.15.32:0/918243957 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f457000e070 con 0x7f457c1026b0 2024-04-12T05:46:28.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.283+0000 7f4581e85700 1 -- 172.21.15.32:0/918243957 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f457c117d70 con 0x7f457c1026b0 2024-04-12T05:46:28.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.283+0000 7f4563fff700 1 -- 172.21.15.32:0/918243957 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4570013de0 con 0x7f457c1026b0 2024-04-12T05:46:28.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.283+0000 7f4563fff700 1 -- 172.21.15.32:0/918243957 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f457001cde0 con 0x7f457c1026b0 2024-04-12T05:46:28.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.283+0000 7f4581e85700 1 -- 172.21.15.32:0/918243957 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f457c118250 con 0x7f457c1026b0 2024-04-12T05:46:28.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.284+0000 7f4581e85700 1 -- 172.21.15.32:0/918243957 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f457c0fd460 con 0x7f457c1026b0 2024-04-12T05:46:28.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.284+0000 7f4563fff700 1 -- 172.21.15.32:0/918243957 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f457001c600 con 0x7f457c1026b0 2024-04-12T05:46:28.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.285+0000 7f4563fff700 1 --2- 172.21.15.32:0/918243957 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f456406cc20 0x7f456406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:28.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.285+0000 7f4563fff700 1 -- 172.21.15.32:0/918243957 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f457008d550 con 0x7f457c1026b0 2024-04-12T05:46:28.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.285+0000 7f457b7fe700 1 --2- 172.21.15.32:0/918243957 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f456406cc20 0x7f456406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:28.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.285+0000 7f457b7fe700 1 --2- 172.21.15.32:0/918243957 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f456406cc20 0x7f456406f0d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f456c004600 tx=0x7f456c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:28.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.289+0000 7f4563fff700 1 -- 172.21.15.32:0/918243957 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4570057530 con 0x7f457c1026b0 2024-04-12T05:46:28.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.581+0000 7f4581e85700 1 -- 172.21.15.32:0/918243957 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f457c04bed0 con 0x7f457c1026b0 2024-04-12T05:46:28.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.582+0000 7f4563fff700 1 -- 172.21.15.32:0/918243957 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f457005ab50 con 0x7f457c1026b0 2024-04-12T05:46:28.578 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:46:28.578 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:46:28.578 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-12T05:46:28.579 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:46:28.579 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:46:28.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.584+0000 7f4581e85700 1 -- 172.21.15.32:0/918243957 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f456406cc20 msgr2=0x7f456406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:28.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.584+0000 7f4581e85700 1 --2- 172.21.15.32:0/918243957 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f456406cc20 0x7f456406f0d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f456c004600 tx=0x7f456c00b040 comp rx=0 tx=0).stop 2024-04-12T05:46:28.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.584+0000 7f4581e85700 1 -- 172.21.15.32:0/918243957 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f457c1026b0 msgr2=0x7f457c10f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:28.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.584+0000 7f4581e85700 1 --2- 172.21.15.32:0/918243957 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f457c1026b0 0x7f457c10f240 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4570000c00 tx=0x7f4570009e30 comp rx=0 tx=0).stop 2024-04-12T05:46:28.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.584+0000 7f4581e85700 1 -- 172.21.15.32:0/918243957 shutdown_connections 2024-04-12T05:46:28.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.584+0000 7f4581e85700 1 --2- 172.21.15.32:0/918243957 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f456406cc20 0x7f456406f0d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:28.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.584+0000 7f4581e85700 1 --2- 172.21.15.32:0/918243957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f457c101d10 0x7f457c10ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:28.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.584+0000 7f4581e85700 1 --2- 172.21.15.32:0/918243957 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f457c1026b0 0x7f457c10f240 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:28.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.584+0000 7f4581e85700 1 -- 172.21.15.32:0/918243957 >> 172.21.15.32:0/918243957 conn(0x7f457c0f7880 msgr2=0x7f457c0f88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:28.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.584+0000 7f4581e85700 1 -- 172.21.15.32:0/918243957 shutdown_connections 2024-04-12T05:46:28.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:28.584+0000 7f4581e85700 1 -- 172.21.15.32:0/918243957 wait complete. 2024-04-12T05:46:29.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:28 smithi067 ceph-mon[112979]: from='client.34396 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:46:29.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:28 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/2936612583' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:46:29.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:28 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/918243957' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:46:29.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:28 smithi032 ceph-mon[146817]: from='client.34396 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:46:29.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:28 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/2936612583' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:46:29.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:28 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/918243957' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:46:30.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:29 smithi067 ceph-mon[112979]: pgmap v176: 65 pgs: 65 active+clean; 270 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:30.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:29 smithi067 ceph-mon[112979]: from='client.34408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:46:30.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:29 smithi032 ceph-mon[146817]: pgmap v176: 65 pgs: 65 active+clean; 270 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:30.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:29 smithi032 ceph-mon[146817]: from='client.34408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:46:31.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:31 smithi067 ceph-mon[112979]: pgmap v177: 65 pgs: 65 active+clean; 270 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:31.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:31 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:31.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:31 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:31.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:31 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:46:31.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:31 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:46:31.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:31 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:31.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:31 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:46:31.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:31 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:31.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:31 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:31.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:31 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:31.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:31 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:31.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:31 smithi032 ceph-mon[146817]: pgmap v177: 65 pgs: 65 active+clean; 270 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:31.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:31 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:31.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:31 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:31.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:31 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:46:31.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:31 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:46:31.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:31 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:31.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:31 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:46:31.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:31 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:31.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:31 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:31.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:31 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:31.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:31 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:33.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:33 smithi032 ceph-mon[146817]: pgmap v178: 65 pgs: 65 active+clean; 270 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:33.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:33 smithi032 ceph-mon[146817]: Upgrade: Updating mds.cephfs.smithi032.pndtry 2024-04-12T05:46:33.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:33 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:33.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:33 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.pndtry", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:46:33.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:33 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:46:33.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:33 smithi032 ceph-mon[146817]: Deploying daemon mds.cephfs.smithi032.pndtry on smithi032 2024-04-12T05:46:33.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:33 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:46:33.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:33 smithi067 ceph-mon[112979]: pgmap v178: 65 pgs: 65 active+clean; 270 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:33.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:33 smithi067 ceph-mon[112979]: Upgrade: Updating mds.cephfs.smithi032.pndtry 2024-04-12T05:46:33.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:33 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:33.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:33 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi032.pndtry", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:46:33.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:33 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:46:33.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:33 smithi067 ceph-mon[112979]: Deploying daemon mds.cephfs.smithi032.pndtry on smithi032 2024-04-12T05:46:33.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:33 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:46:35.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:35 smithi067 ceph-mon[112979]: pgmap v179: 65 pgs: 65 active+clean; 270 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:35.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:35 smithi067 ceph-mon[112979]: osdmap e83: 6 total, 6 up, 6 in 2024-04-12T05:46:35.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:35 smithi067 ceph-mon[112979]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-12T05:46:35.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:35 smithi032 ceph-mon[146817]: pgmap v179: 65 pgs: 65 active+clean; 270 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:35.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:35 smithi032 ceph-mon[146817]: osdmap e83: 6 total, 6 up, 6 in 2024-04-12T05:46:35.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:35 smithi032 ceph-mon[146817]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-12T05:46:37.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:37 smithi067 ceph-mon[112979]: pgmap v181: 65 pgs: 65 active+clean; 270 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:37.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:37 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.pndtry"}]: dispatch 2024-04-12T05:46:37.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:37 smithi032 ceph-mon[146817]: pgmap v181: 65 pgs: 65 active+clean; 270 MiB data, 1023 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:37.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:37 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.pndtry"}]: dispatch 2024-04-12T05:46:39.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:39 smithi032 ceph-mon[146817]: pgmap v182: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:39.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:39 smithi067 ceph-mon[112979]: pgmap v182: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:41.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:40 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:41.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:40 smithi067 ceph-mon[112979]: pgmap v183: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:41.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:40 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:41.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:40 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:46:41.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:40 smithi067 ceph-mon[112979]: mds.? [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] up:boot 2024-04-12T05:46:41.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:40 smithi067 ceph-mon[112979]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-12T05:46:41.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:40 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.pndtry"}]: dispatch 2024-04-12T05:46:41.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:40 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:41.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:40 smithi032 ceph-mon[146817]: pgmap v183: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:41.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:40 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:41.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:40 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:46:41.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:40 smithi032 ceph-mon[146817]: mds.? [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] up:boot 2024-04-12T05:46:41.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:40 smithi032 ceph-mon[146817]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-12T05:46:41.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:40 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi032.pndtry"}]: dispatch 2024-04-12T05:46:43.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:42 smithi067 ceph-mon[112979]: pgmap v184: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:43.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:42 smithi032 ceph-mon[146817]: pgmap v184: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:45.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:45 smithi067 ceph-mon[112979]: pgmap v185: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:45.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:45 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:45.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:45 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:45.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:45 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:45.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:45 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:45.794 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:45 smithi032 ceph-mon[146817]: pgmap v185: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:45.794 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:45 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:45.794 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:45 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:45.794 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:45 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:45.794 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:45 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:47.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:47 smithi067 ceph-mon[112979]: pgmap v186: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:47.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:47 smithi032 ceph-mon[146817]: pgmap v186: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:48.419 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:48 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:46:48.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:48 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:46:49.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:49 smithi067 ceph-mon[112979]: pgmap v187: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:49.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:49 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:49.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:49 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:49.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:49 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:46:49.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:49 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:46:49.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:49 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:49.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:49 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:46:49.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:49 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:49.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:49 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:49.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:49 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:49.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:49 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:49.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:49 smithi032 ceph-mon[146817]: pgmap v187: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:49.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:49 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:49.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:49 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:49.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:49 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:46:49.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:49 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:46:49.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:49 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:49.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:49 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:46:49.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:49 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:49.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:49 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:49.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:49 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:49.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:49 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:51.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:51 smithi067 ceph-mon[112979]: pgmap v188: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:51.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:51 smithi067 ceph-mon[112979]: Upgrade: Updating mds.cephfs.smithi067.zesmgy 2024-04-12T05:46:51.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:51 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:51.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:51 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.zesmgy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:46:51.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:51 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:46:51.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:51 smithi067 ceph-mon[112979]: Deploying daemon mds.cephfs.smithi067.zesmgy on smithi067 2024-04-12T05:46:51.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:51 smithi032 ceph-mon[146817]: pgmap v188: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:51.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:51 smithi032 ceph-mon[146817]: Upgrade: Updating mds.cephfs.smithi067.zesmgy 2024-04-12T05:46:51.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:51 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:51.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:51 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.zesmgy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:46:51.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:51 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:46:51.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:51 smithi032 ceph-mon[146817]: Deploying daemon mds.cephfs.smithi067.zesmgy on smithi067 2024-04-12T05:46:53.397 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:53 smithi032 ceph-mon[146817]: pgmap v189: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:53.397 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:53 smithi032 ceph-mon[146817]: osdmap e84: 6 total, 6 up, 6 in 2024-04-12T05:46:53.397 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:53 smithi032 ceph-mon[146817]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-12T05:46:53.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:53 smithi067 ceph-mon[112979]: pgmap v189: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:53.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:53 smithi067 ceph-mon[112979]: osdmap e84: 6 total, 6 up, 6 in 2024-04-12T05:46:53.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:53 smithi067 ceph-mon[112979]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-12T05:46:55.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:55 smithi067 ceph-mon[112979]: pgmap v191: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:55.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:55 smithi032 ceph-mon[146817]: pgmap v191: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:57.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:57 smithi067 ceph-mon[112979]: pgmap v192: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:57.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:57 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:57.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:57 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:57.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:57 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:46:57.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:57 smithi067 ceph-mon[112979]: mds.? [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] up:boot 2024-04-12T05:46:57.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:57 smithi067 ceph-mon[112979]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-12T05:46:57.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:57 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.zesmgy"}]: dispatch 2024-04-12T05:46:57.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:57 smithi032 ceph-mon[146817]: pgmap v192: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:57.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:57 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:57.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:57 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:57.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:57 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:46:57.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:57 smithi032 ceph-mon[146817]: mds.? [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] up:boot 2024-04-12T05:46:57.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:57 smithi032 ceph-mon[146817]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-12T05:46:57.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:57 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.zesmgy"}]: dispatch 2024-04-12T05:46:58.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.703+0000 7fcfd667c700 1 -- 172.21.15.32:0/2886925926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfd00fde90 msgr2=0x7fcfd00fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:58.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.703+0000 7fcfd667c700 1 --2- 172.21.15.32:0/2886925926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfd00fde90 0x7fcfd00fe260 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0004940 tx=0x7fcfc0009e30 comp rx=0 tx=0).stop 2024-04-12T05:46:58.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.703+0000 7fcfd667c700 1 -- 172.21.15.32:0/2886925926 shutdown_connections 2024-04-12T05:46:58.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.703+0000 7fcfd667c700 1 --2- 172.21.15.32:0/2886925926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfd00fe830 0x7fcfd0106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:58.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.703+0000 7fcfd667c700 1 --2- 172.21.15.32:0/2886925926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfd00fde90 0x7fcfd00fe260 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:58.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.703+0000 7fcfd667c700 1 -- 172.21.15.32:0/2886925926 >> 172.21.15.32:0/2886925926 conn(0x7fcfd0072bc0 msgr2=0x7fcfd0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:58.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.703+0000 7fcfd667c700 1 -- 172.21.15.32:0/2886925926 shutdown_connections 2024-04-12T05:46:58.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.703+0000 7fcfd667c700 1 -- 172.21.15.32:0/2886925926 wait complete. 2024-04-12T05:46:58.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.704+0000 7fcfd667c700 1 Processor -- start 2024-04-12T05:46:58.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.704+0000 7fcfd667c700 1 -- start start 2024-04-12T05:46:58.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.704+0000 7fcfd667c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfd00fde90 0x7fcfd00fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.704+0000 7fcfd667c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfd00fe830 0x7fcfd00fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.704+0000 7fcfd667c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd00fc820 con 0x7fcfd00fe830 2024-04-12T05:46:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.704+0000 7fcfd667c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd00fc960 con 0x7fcfd00fde90 2024-04-12T05:46:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.705+0000 7fcfcf7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfd00fe830 0x7fcfd00fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.705+0000 7fcfcffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfd00fde90 0x7fcfd00fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.705+0000 7fcfcffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfd00fde90 0x7fcfd00fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.32:54266/0 (socket says 172.21.15.32:54266) 2024-04-12T05:46:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.705+0000 7fcfcffff700 1 -- 172.21.15.32:0/3176596058 learned_addr learned my addr 172.21.15.32:0/3176596058 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:46:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.705+0000 7fcfcf7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfd00fe830 0x7fcfd00fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48414/0 (socket says 172.21.15.32:48414) 2024-04-12T05:46:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.705+0000 7fcfcffff700 1 -- 172.21.15.32:0/3176596058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfd00fe830 msgr2=0x7fcfd00fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.705+0000 7fcfcffff700 1 --2- 172.21.15.32:0/3176596058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfd00fe830 0x7fcfd00fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.705+0000 7fcfcffff700 1 -- 172.21.15.32:0/3176596058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc0005040 con 0x7fcfd00fde90 2024-04-12T05:46:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.705+0000 7fcfcffff700 1 --2- 172.21.15.32:0/3176596058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfd00fde90 0x7fcfd00fbd50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0000c00 tx=0x7fcfc0009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.705+0000 7fcfccff9700 1 -- 172.21.15.32:0/3176596058 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfc0021070 con 0x7fcfd00fde90 2024-04-12T05:46:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.705+0000 7fcfccff9700 1 -- 172.21.15.32:0/3176596058 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcfc0013e00 con 0x7fcfd00fde90 2024-04-12T05:46:58.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.705+0000 7fcfd667c700 1 -- 172.21.15.32:0/3176596058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfd00fcbb0 con 0x7fcfd00fde90 2024-04-12T05:46:58.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.705+0000 7fcfccff9700 1 -- 172.21.15.32:0/3176596058 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfc000e070 con 0x7fcfd00fde90 2024-04-12T05:46:58.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.705+0000 7fcfd667c700 1 -- 172.21.15.32:0/3176596058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfd011a120 con 0x7fcfd00fde90 2024-04-12T05:46:58.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.706+0000 7fcfd667c700 1 -- 172.21.15.32:0/3176596058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfd0065e30 con 0x7fcfd00fde90 2024-04-12T05:46:58.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.707+0000 7fcfccff9700 1 -- 172.21.15.32:0/3176596058 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fcfc0013950 con 0x7fcfd00fde90 2024-04-12T05:46:58.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.707+0000 7fcfccff9700 1 --2- 172.21.15.32:0/3176596058 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fcfb806cc30 0x7fcfb806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:58.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.707+0000 7fcfccff9700 1 -- 172.21.15.32:0/3176596058 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fcfc008d130 con 0x7fcfd00fde90 2024-04-12T05:46:58.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.707+0000 7fcfcf7fe700 1 --2- 172.21.15.32:0/3176596058 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fcfb806cc30 0x7fcfb806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:58.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.708+0000 7fcfcf7fe700 1 --2- 172.21.15.32:0/3176596058 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fcfb806cc30 0x7fcfb806f0e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fcfd00fd690 tx=0x7fcfc400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:58.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.713+0000 7fcfccff9700 1 -- 172.21.15.32:0/3176596058 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fcfc0057070 con 0x7fcfd00fde90 2024-04-12T05:46:58.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.933+0000 7fcfd667c700 1 -- 172.21.15.32:0/3176596058 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcfd00fd440 con 0x7fcfb806cc30 2024-04-12T05:46:58.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.934+0000 7fcfccff9700 1 -- 172.21.15.32:0/3176596058 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fcfd00fd440 con 0x7fcfb806cc30 2024-04-12T05:46:58.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.936+0000 7fcfd667c700 1 -- 172.21.15.32:0/3176596058 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fcfb806cc30 msgr2=0x7fcfb806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:58.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.936+0000 7fcfd667c700 1 --2- 172.21.15.32:0/3176596058 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fcfb806cc30 0x7fcfb806f0e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fcfd00fd690 tx=0x7fcfc400b040 comp rx=0 tx=0).stop 2024-04-12T05:46:58.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.936+0000 7fcfd667c700 1 -- 172.21.15.32:0/3176596058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfd00fde90 msgr2=0x7fcfd00fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:58.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.936+0000 7fcfd667c700 1 --2- 172.21.15.32:0/3176596058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfd00fde90 0x7fcfd00fbd50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0000c00 tx=0x7fcfc0009e30 comp rx=0 tx=0).stop 2024-04-12T05:46:58.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.937+0000 7fcfd667c700 1 -- 172.21.15.32:0/3176596058 shutdown_connections 2024-04-12T05:46:58.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.937+0000 7fcfd667c700 1 --2- 172.21.15.32:0/3176596058 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fcfb806cc30 0x7fcfb806f0e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:58.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.937+0000 7fcfd667c700 1 --2- 172.21.15.32:0/3176596058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfd00fde90 0x7fcfd00fbd50 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:58.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.937+0000 7fcfd667c700 1 --2- 172.21.15.32:0/3176596058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfd00fe830 0x7fcfd00fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:58.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.937+0000 7fcfd667c700 1 -- 172.21.15.32:0/3176596058 >> 172.21.15.32:0/3176596058 conn(0x7fcfd0072bc0 msgr2=0x7fcfd01037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:58.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.937+0000 7fcfd667c700 1 -- 172.21.15.32:0/3176596058 shutdown_connections 2024-04-12T05:46:58.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:58.937+0000 7fcfd667c700 1 -- 172.21.15.32:0/3176596058 wait complete. 2024-04-12T05:46:58.943 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:46:59.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.042+0000 7f3d586c2700 1 -- 172.21.15.32:0/3119353601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d50102710 msgr2=0x7f3d500fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:59.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.042+0000 7f3d586c2700 1 --2- 172.21.15.32:0/3119353601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d50102710 0x7f3d500fbd80 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3d40004940 tx=0x7f3d40009e30 comp rx=0 tx=0).stop 2024-04-12T05:46:59.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.042+0000 7f3d586c2700 1 -- 172.21.15.32:0/3119353601 shutdown_connections 2024-04-12T05:46:59.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.042+0000 7f3d586c2700 1 --2- 172.21.15.32:0/3119353601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d50102710 0x7f3d500fbd80 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:59.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.042+0000 7f3d586c2700 1 --2- 172.21.15.32:0/3119353601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d50101d70 0x7f3d50102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:59.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.042+0000 7f3d586c2700 1 -- 172.21.15.32:0/3119353601 >> 172.21.15.32:0/3119353601 conn(0x7f3d50072bc0 msgr2=0x7f3d50072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:59.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.042+0000 7f3d586c2700 1 -- 172.21.15.32:0/3119353601 shutdown_connections 2024-04-12T05:46:59.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.042+0000 7f3d586c2700 1 -- 172.21.15.32:0/3119353601 wait complete. 2024-04-12T05:46:59.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.042+0000 7f3d586c2700 1 Processor -- start 2024-04-12T05:46:59.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.042+0000 7f3d586c2700 1 -- start start 2024-04-12T05:46:59.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.043+0000 7f3d586c2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d50101d70 0x7f3d5010ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:59.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.043+0000 7f3d586c2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d50102710 0x7f3d5010efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:59.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.043+0000 7f3d586c2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5010f6b0 con 0x7f3d50102710 2024-04-12T05:46:59.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.043+0000 7f3d586c2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5010f7f0 con 0x7f3d50101d70 2024-04-12T05:46:59.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.043+0000 7f3d55c5d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d50102710 0x7f3d5010efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:59.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.043+0000 7f3d5645e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d50101d70 0x7f3d5010ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:59.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.043+0000 7f3d55c5d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d50102710 0x7f3d5010efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48434/0 (socket says 172.21.15.32:48434) 2024-04-12T05:46:59.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.043+0000 7f3d55c5d700 1 -- 172.21.15.32:0/2507807576 learned_addr learned my addr 172.21.15.32:0/2507807576 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:46:59.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.043+0000 7f3d55c5d700 1 -- 172.21.15.32:0/2507807576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d50101d70 msgr2=0x7f3d5010ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:59.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.043+0000 7f3d55c5d700 1 --2- 172.21.15.32:0/2507807576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d50101d70 0x7f3d5010ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:59.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.043+0000 7f3d55c5d700 1 -- 172.21.15.32:0/2507807576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d40005040 con 0x7f3d50102710 2024-04-12T05:46:59.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.043+0000 7f3d55c5d700 1 --2- 172.21.15.32:0/2507807576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d50102710 0x7f3d5010efd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3d40009390 tx=0x7f3d40009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:59.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.043+0000 7f3d46ffd700 1 -- 172.21.15.32:0/2507807576 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d4000e040 con 0x7f3d50102710 2024-04-12T05:46:59.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.043+0000 7f3d46ffd700 1 -- 172.21.15.32:0/2507807576 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d40004290 con 0x7f3d50102710 2024-04-12T05:46:59.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.043+0000 7f3d586c2700 1 -- 172.21.15.32:0/2507807576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d50117b00 con 0x7f3d50102710 2024-04-12T05:46:59.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.043+0000 7f3d46ffd700 1 -- 172.21.15.32:0/2507807576 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d40013e70 con 0x7f3d50102710 2024-04-12T05:46:59.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.043+0000 7f3d586c2700 1 -- 172.21.15.32:0/2507807576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d50118090 con 0x7f3d50102710 2024-04-12T05:46:59.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.044+0000 7f3d586c2700 1 -- 172.21.15.32:0/2507807576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d500ff080 con 0x7f3d50102710 2024-04-12T05:46:59.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.045+0000 7f3d46ffd700 1 -- 172.21.15.32:0/2507807576 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3d40013690 con 0x7f3d50102710 2024-04-12T05:46:59.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.045+0000 7f3d46ffd700 1 --2- 172.21.15.32:0/2507807576 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3d3c06cc30 0x7f3d3c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:59.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.045+0000 7f3d46ffd700 1 -- 172.21.15.32:0/2507807576 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f3d4008d240 con 0x7f3d50102710 2024-04-12T05:46:59.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.045+0000 7f3d5645e700 1 --2- 172.21.15.32:0/2507807576 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3d3c06cc30 0x7f3d3c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:59.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.046+0000 7f3d5645e700 1 --2- 172.21.15.32:0/2507807576 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3d3c06cc30 0x7f3d3c06f0e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c009bf0 tx=0x7f3d4c0091a0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:59.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.050+0000 7f3d46ffd700 1 -- 172.21.15.32:0/2507807576 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3d400572b0 con 0x7f3d50102710 2024-04-12T05:46:59.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.274+0000 7f3d586c2700 1 -- 172.21.15.32:0/2507807576 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d50065e50 con 0x7f3d3c06cc30 2024-04-12T05:46:59.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.275+0000 7f3d46ffd700 1 -- 172.21.15.32:0/2507807576 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3d50065e50 con 0x7f3d3c06cc30 2024-04-12T05:46:59.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.278+0000 7f3d586c2700 1 -- 172.21.15.32:0/2507807576 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3d3c06cc30 msgr2=0x7f3d3c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:59.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.278+0000 7f3d586c2700 1 --2- 172.21.15.32:0/2507807576 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3d3c06cc30 0x7f3d3c06f0e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c009bf0 tx=0x7f3d4c0091a0 comp rx=0 tx=0).stop 2024-04-12T05:46:59.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.278+0000 7f3d586c2700 1 -- 172.21.15.32:0/2507807576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d50102710 msgr2=0x7f3d5010efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:59.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.278+0000 7f3d586c2700 1 --2- 172.21.15.32:0/2507807576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d50102710 0x7f3d5010efd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3d40009390 tx=0x7f3d40009e30 comp rx=0 tx=0).stop 2024-04-12T05:46:59.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.278+0000 7f3d586c2700 1 -- 172.21.15.32:0/2507807576 shutdown_connections 2024-04-12T05:46:59.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.278+0000 7f3d586c2700 1 --2- 172.21.15.32:0/2507807576 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3d3c06cc30 0x7f3d3c06f0e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:59.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.278+0000 7f3d586c2700 1 --2- 172.21.15.32:0/2507807576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d50101d70 0x7f3d5010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:59.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.278+0000 7f3d586c2700 1 --2- 172.21.15.32:0/2507807576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d50102710 0x7f3d5010efd0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:59.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.278+0000 7f3d586c2700 1 -- 172.21.15.32:0/2507807576 >> 172.21.15.32:0/2507807576 conn(0x7f3d50072bc0 msgr2=0x7f3d500fb060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:59.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.278+0000 7f3d586c2700 1 -- 172.21.15.32:0/2507807576 shutdown_connections 2024-04-12T05:46:59.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.278+0000 7f3d586c2700 1 -- 172.21.15.32:0/2507807576 wait complete. 2024-04-12T05:46:59.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.381+0000 7f41f973c700 1 -- 172.21.15.32:0/1637171676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f41026c0 msgr2=0x7f41f40fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:59.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.381+0000 7f41f973c700 1 --2- 172.21.15.32:0/1637171676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f41026c0 0x7f41f40fbd30 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f41e4004940 tx=0x7f41e4009e30 comp rx=0 tx=0).stop 2024-04-12T05:46:59.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.381+0000 7f41f973c700 1 -- 172.21.15.32:0/1637171676 shutdown_connections 2024-04-12T05:46:59.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.381+0000 7f41f973c700 1 --2- 172.21.15.32:0/1637171676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f41026c0 0x7f41f40fbd30 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:59.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.381+0000 7f41f973c700 1 --2- 172.21.15.32:0/1637171676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41f4101d20 0x7f41f41020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:59.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.381+0000 7f41f973c700 1 -- 172.21.15.32:0/1637171676 >> 172.21.15.32:0/1637171676 conn(0x7f41f40f7870 msgr2=0x7f41f40f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:59.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.381+0000 7f41f973c700 1 -- 172.21.15.32:0/1637171676 shutdown_connections 2024-04-12T05:46:59.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.381+0000 7f41f973c700 1 -- 172.21.15.32:0/1637171676 wait complete. 2024-04-12T05:46:59.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.382+0000 7f41f973c700 1 Processor -- start 2024-04-12T05:46:59.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.382+0000 7f41f973c700 1 -- start start 2024-04-12T05:46:59.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.382+0000 7f41f973c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f4101d20 0x7f41f4113890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:59.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.382+0000 7f41f973c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41f41026c0 0x7f41f4113dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:59.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.382+0000 7f41f973c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f4114420 con 0x7f41f4101d20 2024-04-12T05:46:59.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.382+0000 7f41f973c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f410da20 con 0x7f41f41026c0 2024-04-12T05:46:59.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.382+0000 7f41f2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f4101d20 0x7f41f4113890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:59.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.382+0000 7f41f2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f4101d20 0x7f41f4113890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48448/0 (socket says 172.21.15.32:48448) 2024-04-12T05:46:59.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.382+0000 7f41f2ffd700 1 -- 172.21.15.32:0/2570679338 learned_addr learned my addr 172.21.15.32:0/2570679338 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:46:59.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.382+0000 7f41f27fc700 1 --2- 172.21.15.32:0/2570679338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41f41026c0 0x7f41f4113dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:59.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.383+0000 7f41f2ffd700 1 -- 172.21.15.32:0/2570679338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41f41026c0 msgr2=0x7f41f4113dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:59.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.383+0000 7f41f2ffd700 1 --2- 172.21.15.32:0/2570679338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41f41026c0 0x7f41f4113dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:59.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.383+0000 7f41f2ffd700 1 -- 172.21.15.32:0/2570679338 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e4005040 con 0x7f41f4101d20 2024-04-12T05:46:59.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.383+0000 7f41f2ffd700 1 --2- 172.21.15.32:0/2570679338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f4101d20 0x7f41f4113890 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f41e8002a50 tx=0x7f41e80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:59.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.383+0000 7f41db7fe700 1 -- 172.21.15.32:0/2570679338 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41e8007cc0 con 0x7f41f4101d20 2024-04-12T05:46:59.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.383+0000 7f41db7fe700 1 -- 172.21.15.32:0/2570679338 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f41e800f040 con 0x7f41f4101d20 2024-04-12T05:46:59.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.383+0000 7f41f973c700 1 -- 172.21.15.32:0/2570679338 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41f410dd00 con 0x7f41f4101d20 2024-04-12T05:46:59.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.383+0000 7f41db7fe700 1 -- 172.21.15.32:0/2570679338 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41e800d9c0 con 0x7f41f4101d20 2024-04-12T05:46:59.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.383+0000 7f41f973c700 1 -- 172.21.15.32:0/2570679338 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41f410e290 con 0x7f41f4101d20 2024-04-12T05:46:59.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.384+0000 7f41f973c700 1 -- 172.21.15.32:0/2570679338 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41f40fd470 con 0x7f41f4101d20 2024-04-12T05:46:59.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.384+0000 7f41db7fe700 1 -- 172.21.15.32:0/2570679338 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f41e800db20 con 0x7f41f4101d20 2024-04-12T05:46:59.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.385+0000 7f41db7fe700 1 --2- 172.21.15.32:0/2570679338 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f41dc06ccf0 0x7f41dc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:59.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.385+0000 7f41db7fe700 1 -- 172.21.15.32:0/2570679338 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f41e808bdb0 con 0x7f41f4101d20 2024-04-12T05:46:59.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.385+0000 7f41f27fc700 1 --2- 172.21.15.32:0/2570679338 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f41dc06ccf0 0x7f41dc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:59.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.385+0000 7f41f27fc700 1 --2- 172.21.15.32:0/2570679338 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f41dc06ccf0 0x7f41dc06f1a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f41e4004890 tx=0x7f41e4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:59.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.389+0000 7f41db7fe700 1 -- 172.21.15.32:0/2570679338 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f41e8059780 con 0x7f41f4101d20 2024-04-12T05:46:59.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.605+0000 7f41f973c700 1 -- 172.21.15.32:0/2570679338 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f41f4108d10 con 0x7f41dc06ccf0 2024-04-12T05:46:59.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.615+0000 7f41db7fe700 1 -- 172.21.15.32:0/2570679338 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f41f4108d10 con 0x7f41dc06ccf0 2024-04-12T05:46:59.611 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:46:59.611 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (6m) 15s ago 18m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:46:59.612 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (4m) 15s ago 18m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:46:59.612 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (4m) 0s ago 17m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:46:59.612 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (6m) 15s ago 17m 87.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:46:59.612 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (39s) 15s ago 12m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T05:46:59.612 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (20s) 15s ago 12m 13.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T05:46:59.612 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (12m) 0s ago 12m 10.6M - 17.2.7-827-gd3b5060a 21de9b7369ad 0c295d31a4a6 2024-04-12T05:46:59.612 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (3s) 0s ago 12m 13.6M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T05:46:59.612 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (8m) 15s ago 19m 507M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:46:59.612 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (7m) 0s ago 17m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:46:59.612 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (5m) 15s ago 19m 53.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:46:59.612 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (4m) 0s ago 17m 43.0M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:46:59.612 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (7m) 15s ago 18m 16.9M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:46:59.612 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (7m) 0s ago 17m 18.2M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:46:59.612 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (3m) 15s ago 16m 91.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T05:46:59.613 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (3m) 15s ago 15m 97.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T05:46:59.613 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (2m) 15s ago 15m 81.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T05:46:59.613 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (2m) 0s ago 15m 112M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T05:46:59.613 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (92s) 0s ago 14m 90.2M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T05:46:59.613 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (58s) 0s ago 14m 84.7M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T05:46:59.613 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (7m) 15s ago 17m 64.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:46:59.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.617+0000 7f41f973c700 1 -- 172.21.15.32:0/2570679338 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f41dc06ccf0 msgr2=0x7f41dc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:59.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.617+0000 7f41f973c700 1 --2- 172.21.15.32:0/2570679338 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f41dc06ccf0 0x7f41dc06f1a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f41e4004890 tx=0x7f41e4009e30 comp rx=0 tx=0).stop 2024-04-12T05:46:59.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.617+0000 7f41f973c700 1 -- 172.21.15.32:0/2570679338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f4101d20 msgr2=0x7f41f4113890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:59.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.617+0000 7f41f973c700 1 --2- 172.21.15.32:0/2570679338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f4101d20 0x7f41f4113890 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f41e8002a50 tx=0x7f41e80046b0 comp rx=0 tx=0).stop 2024-04-12T05:46:59.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.617+0000 7f41f973c700 1 -- 172.21.15.32:0/2570679338 shutdown_connections 2024-04-12T05:46:59.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.617+0000 7f41f973c700 1 --2- 172.21.15.32:0/2570679338 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f41dc06ccf0 0x7f41dc06f1a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:59.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.617+0000 7f41f973c700 1 --2- 172.21.15.32:0/2570679338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f4101d20 0x7f41f4113890 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:59.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.617+0000 7f41f973c700 1 --2- 172.21.15.32:0/2570679338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41f41026c0 0x7f41f4113dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:59.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.617+0000 7f41f973c700 1 -- 172.21.15.32:0/2570679338 >> 172.21.15.32:0/2570679338 conn(0x7f41f40f7870 msgr2=0x7f41f40f90e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:59.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.617+0000 7f41f973c700 1 -- 172.21.15.32:0/2570679338 shutdown_connections 2024-04-12T05:46:59.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.617+0000 7f41f973c700 1 -- 172.21.15.32:0/2570679338 wait complete. 2024-04-12T05:46:59.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.720+0000 7fbb8301f700 1 -- 172.21.15.32:0/120443041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb7c102710 msgr2=0x7fbb7c0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:59.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.720+0000 7fbb8301f700 1 --2- 172.21.15.32:0/120443041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb7c102710 0x7fbb7c0fbd80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c004940 tx=0x7fbb6c009e30 comp rx=0 tx=0).stop 2024-04-12T05:46:59.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.720+0000 7fbb8301f700 1 -- 172.21.15.32:0/120443041 shutdown_connections 2024-04-12T05:46:59.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.720+0000 7fbb8301f700 1 --2- 172.21.15.32:0/120443041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb7c102710 0x7fbb7c0fbd80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:59.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.720+0000 7fbb8301f700 1 --2- 172.21.15.32:0/120443041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb7c101d70 0x7fbb7c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:59.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.720+0000 7fbb8301f700 1 -- 172.21.15.32:0/120443041 >> 172.21.15.32:0/120443041 conn(0x7fbb7c072bc0 msgr2=0x7fbb7c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:59.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.720+0000 7fbb8301f700 1 -- 172.21.15.32:0/120443041 shutdown_connections 2024-04-12T05:46:59.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.720+0000 7fbb8301f700 1 -- 172.21.15.32:0/120443041 wait complete. 2024-04-12T05:46:59.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.721+0000 7fbb8301f700 1 Processor -- start 2024-04-12T05:46:59.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.721+0000 7fbb8301f700 1 -- start start 2024-04-12T05:46:59.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.721+0000 7fbb8301f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb7c101d70 0x7fbb7c113a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:59.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.721+0000 7fbb8301f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb7c102710 0x7fbb7c113f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:59.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.721+0000 7fbb8301f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7c114590 con 0x7fbb7c101d70 2024-04-12T05:46:59.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.721+0000 7fbb8301f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7c10d980 con 0x7fbb7c102710 2024-04-12T05:46:59.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.721+0000 7fbb80dbb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb7c101d70 0x7fbb7c113a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:59.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.721+0000 7fbb80dbb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb7c101d70 0x7fbb7c113a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48464/0 (socket says 172.21.15.32:48464) 2024-04-12T05:46:59.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.721+0000 7fbb80dbb700 1 -- 172.21.15.32:0/49237477 learned_addr learned my addr 172.21.15.32:0/49237477 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:46:59.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.721+0000 7fbb73fff700 1 --2- 172.21.15.32:0/49237477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb7c102710 0x7fbb7c113f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:59.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.721+0000 7fbb80dbb700 1 -- 172.21.15.32:0/49237477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb7c102710 msgr2=0x7fbb7c113f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:59.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.721+0000 7fbb80dbb700 1 --2- 172.21.15.32:0/49237477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb7c102710 0x7fbb7c113f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:46:59.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.721+0000 7fbb80dbb700 1 -- 172.21.15.32:0/49237477 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb6c005040 con 0x7fbb7c101d70 2024-04-12T05:46:59.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.722+0000 7fbb80dbb700 1 --2- 172.21.15.32:0/49237477 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb7c101d70 0x7fbb7c113a00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbb78001db0 tx=0x7fbb78009ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:59.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.722+0000 7fbb717fa700 1 -- 172.21.15.32:0/49237477 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb7800ed50 con 0x7fbb7c101d70 2024-04-12T05:46:59.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.722+0000 7fbb717fa700 1 -- 172.21.15.32:0/49237477 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb78007dd0 con 0x7fbb7c101d70 2024-04-12T05:46:59.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.722+0000 7fbb8301f700 1 -- 172.21.15.32:0/49237477 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb7c10dc60 con 0x7fbb7c101d70 2024-04-12T05:46:59.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.722+0000 7fbb717fa700 1 -- 172.21.15.32:0/49237477 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb78014640 con 0x7fbb7c101d70 2024-04-12T05:46:59.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.722+0000 7fbb8301f700 1 -- 172.21.15.32:0/49237477 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb7c10e1f0 con 0x7fbb7c101d70 2024-04-12T05:46:59.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.723+0000 7fbb8301f700 1 -- 172.21.15.32:0/49237477 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb7c0fd4c0 con 0x7fbb7c101d70 2024-04-12T05:46:59.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.723+0000 7fbb717fa700 1 -- 172.21.15.32:0/49237477 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fbb78007920 con 0x7fbb7c101d70 2024-04-12T05:46:59.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.724+0000 7fbb717fa700 1 --2- 172.21.15.32:0/49237477 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbb6406cc20 0x7fbb6406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:46:59.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.724+0000 7fbb717fa700 1 -- 172.21.15.32:0/49237477 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fbb7808bdf0 con 0x7fbb7c101d70 2024-04-12T05:46:59.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.724+0000 7fbb73fff700 1 --2- 172.21.15.32:0/49237477 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbb6406cc20 0x7fbb6406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:59.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.724+0000 7fbb73fff700 1 --2- 172.21.15.32:0/49237477 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbb6406cc20 0x7fbb6406f0d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c10ec60 tx=0x7fbb6c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:59.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:46:59.728+0000 7fbb717fa700 1 -- 172.21.15.32:0/49237477 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbb78059910 con 0x7fbb7c101d70 2024-04-12T05:46:59.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:59 smithi067 ceph-mon[112979]: pgmap v193: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:59.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:59 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:59.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:46:59 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:59.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:59 smithi032 ceph-mon[146817]: pgmap v193: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:59.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:59 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:46:59.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:46:59 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:47:00.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.018+0000 7fbb8301f700 1 -- 172.21.15.32:0/49237477 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fbb7c108d60 con 0x7fbb7c101d70 2024-04-12T05:47:00.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.019+0000 7fbb717fa700 1 -- 172.21.15.32:0/49237477 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fbb780594a0 con 0x7fbb7c101d70 2024-04-12T05:47:00.015 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:47:00.015 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:47:00.015 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:47:00.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:47:00.016 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:47:00.016 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:47:00.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:47:00.016 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:47:00.016 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:47:00.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:47:00.016 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:47:00.016 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 1, 2024-04-12T05:47:00.016 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:47:00.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:47:00.016 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:47:00.016 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 17.2.7-827-gd3b5060a (d3b5060a17449edfae9385cdf792afec109fca5d) quincy (stable)": 1, 2024-04-12T05:47:00.016 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 13 2024-04-12T05:47:00.016 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:47:00.017 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:47:00.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.021+0000 7fbb8301f700 1 -- 172.21.15.32:0/49237477 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbb6406cc20 msgr2=0x7fbb6406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:00.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.021+0000 7fbb8301f700 1 --2- 172.21.15.32:0/49237477 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbb6406cc20 0x7fbb6406f0d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c10ec60 tx=0x7fbb6c009e30 comp rx=0 tx=0).stop 2024-04-12T05:47:00.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.021+0000 7fbb8301f700 1 -- 172.21.15.32:0/49237477 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb7c101d70 msgr2=0x7fbb7c113a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:00.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.021+0000 7fbb8301f700 1 --2- 172.21.15.32:0/49237477 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb7c101d70 0x7fbb7c113a00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbb78001db0 tx=0x7fbb78009ee0 comp rx=0 tx=0).stop 2024-04-12T05:47:00.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.021+0000 7fbb8301f700 1 -- 172.21.15.32:0/49237477 shutdown_connections 2024-04-12T05:47:00.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.021+0000 7fbb8301f700 1 --2- 172.21.15.32:0/49237477 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbb6406cc20 0x7fbb6406f0d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:00.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.021+0000 7fbb8301f700 1 --2- 172.21.15.32:0/49237477 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb7c101d70 0x7fbb7c113a00 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:00.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.021+0000 7fbb8301f700 1 --2- 172.21.15.32:0/49237477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb7c102710 0x7fbb7c113f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:00.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.021+0000 7fbb8301f700 1 -- 172.21.15.32:0/49237477 >> 172.21.15.32:0/49237477 conn(0x7fbb7c072bc0 msgr2=0x7fbb7c065970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:00.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.021+0000 7fbb8301f700 1 -- 172.21.15.32:0/49237477 shutdown_connections 2024-04-12T05:47:00.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.021+0000 7fbb8301f700 1 -- 172.21.15.32:0/49237477 wait complete. 2024-04-12T05:47:00.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.125+0000 7f001b6f2700 1 -- 172.21.15.32:0/3837803160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0014101d70 msgr2=0x7f0014102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:00.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.125+0000 7f001b6f2700 1 --2- 172.21.15.32:0/3837803160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0014101d70 0x7f0014102140 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0010004940 tx=0x7f0010009e30 comp rx=0 tx=0).stop 2024-04-12T05:47:00.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.125+0000 7f001b6f2700 1 -- 172.21.15.32:0/3837803160 shutdown_connections 2024-04-12T05:47:00.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.125+0000 7f001b6f2700 1 --2- 172.21.15.32:0/3837803160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0014102710 0x7f00140fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:00.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.125+0000 7f001b6f2700 1 --2- 172.21.15.32:0/3837803160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0014101d70 0x7f0014102140 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:00.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.125+0000 7f001b6f2700 1 -- 172.21.15.32:0/3837803160 >> 172.21.15.32:0/3837803160 conn(0x7f0014072bc0 msgr2=0x7f0014072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:00.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.125+0000 7f001b6f2700 1 -- 172.21.15.32:0/3837803160 shutdown_connections 2024-04-12T05:47:00.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.125+0000 7f001b6f2700 1 -- 172.21.15.32:0/3837803160 wait complete. 2024-04-12T05:47:00.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.125+0000 7f001b6f2700 1 Processor -- start 2024-04-12T05:47:00.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.126+0000 7f001b6f2700 1 -- start start 2024-04-12T05:47:00.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.126+0000 7f001b6f2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0014101d70 0x7f001410c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:47:00.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.126+0000 7f001b6f2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0014102710 0x7f001410cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:47:00.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.126+0000 7f001b6f2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f001410d250 con 0x7f0014102710 2024-04-12T05:47:00.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.126+0000 7f001b6f2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f001410d3c0 con 0x7f0014101d70 2024-04-12T05:47:00.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.126+0000 7f0018c8d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0014102710 0x7f001410cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:00.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.126+0000 7f0018c8d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0014102710 0x7f001410cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48486/0 (socket says 172.21.15.32:48486) 2024-04-12T05:47:00.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.126+0000 7f0018c8d700 1 -- 172.21.15.32:0/4174019730 learned_addr learned my addr 172.21.15.32:0/4174019730 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:47:00.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.126+0000 7f001948e700 1 --2- 172.21.15.32:0/4174019730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0014101d70 0x7f001410c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:00.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.126+0000 7f0018c8d700 1 -- 172.21.15.32:0/4174019730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0014101d70 msgr2=0x7f001410c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:00.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.126+0000 7f0018c8d700 1 --2- 172.21.15.32:0/4174019730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0014101d70 0x7f001410c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:00.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.126+0000 7f0018c8d700 1 -- 172.21.15.32:0/4174019730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0010005040 con 0x7f0014102710 2024-04-12T05:47:00.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.126+0000 7f0018c8d700 1 --2- 172.21.15.32:0/4174019730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0014102710 0x7f001410cd10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f000400eb40 tx=0x7f000400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:00.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.126+0000 7f0009ffb700 1 -- 172.21.15.32:0/4174019730 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0004007cc0 con 0x7f0014102710 2024-04-12T05:47:00.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.127+0000 7f0009ffb700 1 -- 172.21.15.32:0/4174019730 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f000400f040 con 0x7f0014102710 2024-04-12T05:47:00.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.127+0000 7f001b6f2700 1 -- 172.21.15.32:0/4174019730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f001410d640 con 0x7f0014102710 2024-04-12T05:47:00.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.127+0000 7f0009ffb700 1 -- 172.21.15.32:0/4174019730 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f000400ca30 con 0x7f0014102710 2024-04-12T05:47:00.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.127+0000 7f001b6f2700 1 -- 172.21.15.32:0/4174019730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f001411a1c0 con 0x7f0014102710 2024-04-12T05:47:00.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.128+0000 7f001b6f2700 1 -- 172.21.15.32:0/4174019730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00140ff080 con 0x7f0014102710 2024-04-12T05:47:00.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.128+0000 7f0009ffb700 1 -- 172.21.15.32:0/4174019730 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f000400cb90 con 0x7f0014102710 2024-04-12T05:47:00.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.129+0000 7f0009ffb700 1 --2- 172.21.15.32:0/4174019730 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f000006cc30 0x7f000006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:47:00.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.129+0000 7f0009ffb700 1 -- 172.21.15.32:0/4174019730 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f000408ba20 con 0x7f0014102710 2024-04-12T05:47:00.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.129+0000 7f001948e700 1 --2- 172.21.15.32:0/4174019730 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f000006cc30 0x7f000006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:00.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.129+0000 7f001948e700 1 --2- 172.21.15.32:0/4174019730 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f000006cc30 0x7f000006f0e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f001000b0d0 tx=0x7f0010009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:00.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.133+0000 7f0009ffb700 1 -- 172.21.15.32:0/4174019730 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0004059410 con 0x7f0014102710 2024-04-12T05:47:00.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.384+0000 7f001b6f2700 1 -- 172.21.15.32:0/4174019730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f00140fd4c0 con 0x7f0014102710 2024-04-12T05:47:00.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.385+0000 7f0009ffb700 1 -- 172.21.15.32:0/4174019730 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1835 (secure 0 0 0) 0x7f0004058fa0 con 0x7f0014102710 2024-04-12T05:47:00.381 INFO:teuthology.orchestra.run.smithi032.stdout:e31 2024-04-12T05:47:00.381 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:47:00.381 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:47:00.381 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:47:00.381 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:47:00.381 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:47:00.382 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:47:00.382 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 24 2024-04-12T05:47:00.382 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:47:00.382 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:47:00.382 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:46:10.668569+0000 2024-04-12T05:47:00.382 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:47:00.382 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:47:00.382 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:47:00.382 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:47:00.382 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:47:00.382 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:47:00.382 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:47:00.382 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:47:00.382 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T05:47:00.382 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:47:00.383 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:47:00.383 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:47:00.383 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T05:47:00.383 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-12T05:47:00.383 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:47:00.383 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:47:00.383 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:47:00.383 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:47:00.383 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:47:00.383 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:47:00.383 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:47:00.383 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:47:00.383 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:47:00.383 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:47:00.384 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-12T05:47:00.384 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:47:00.384 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:47:00.384 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:47:00.384 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{-1:44275} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/3573641538,v1:172.21.15.67:6827/3573641538] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:47:00.384 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{-1:44317} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:47:00.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.387+0000 7f001b6f2700 1 -- 172.21.15.32:0/4174019730 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f000006cc30 msgr2=0x7f000006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:00.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.387+0000 7f001b6f2700 1 --2- 172.21.15.32:0/4174019730 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f000006cc30 0x7f000006f0e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f001000b0d0 tx=0x7f0010009e30 comp rx=0 tx=0).stop 2024-04-12T05:47:00.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.387+0000 7f001b6f2700 1 -- 172.21.15.32:0/4174019730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0014102710 msgr2=0x7f001410cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:00.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.387+0000 7f001b6f2700 1 --2- 172.21.15.32:0/4174019730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0014102710 0x7f001410cd10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f000400eb40 tx=0x7f000400b040 comp rx=0 tx=0).stop 2024-04-12T05:47:00.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.387+0000 7f001b6f2700 1 -- 172.21.15.32:0/4174019730 shutdown_connections 2024-04-12T05:47:00.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.387+0000 7f001b6f2700 1 --2- 172.21.15.32:0/4174019730 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f000006cc30 0x7f000006f0e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:00.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.387+0000 7f001b6f2700 1 --2- 172.21.15.32:0/4174019730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0014101d70 0x7f001410c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:00.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.387+0000 7f001b6f2700 1 --2- 172.21.15.32:0/4174019730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0014102710 0x7f001410cd10 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:00.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.387+0000 7f001b6f2700 1 -- 172.21.15.32:0/4174019730 >> 172.21.15.32:0/4174019730 conn(0x7f0014072bc0 msgr2=0x7f00140faf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:00.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.387+0000 7f001b6f2700 1 -- 172.21.15.32:0/4174019730 shutdown_connections 2024-04-12T05:47:00.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.387+0000 7f001b6f2700 1 -- 172.21.15.32:0/4174019730 wait complete. 2024-04-12T05:47:00.385 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 31 2024-04-12T05:47:00.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.492+0000 7f81bba16700 1 -- 172.21.15.32:0/2581788993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81b4101d10 msgr2=0x7f81b41020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:00.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.492+0000 7f81bba16700 1 --2- 172.21.15.32:0/2581788993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81b4101d10 0x7f81b41020e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f81b0004940 tx=0x7f81b0009e30 comp rx=0 tx=0).stop 2024-04-12T05:47:00.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.492+0000 7f81bba16700 1 -- 172.21.15.32:0/2581788993 shutdown_connections 2024-04-12T05:47:00.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.492+0000 7f81bba16700 1 --2- 172.21.15.32:0/2581788993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81b41026b0 0x7f81b40fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:00.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.492+0000 7f81bba16700 1 --2- 172.21.15.32:0/2581788993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81b4101d10 0x7f81b41020e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:00.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.492+0000 7f81bba16700 1 -- 172.21.15.32:0/2581788993 >> 172.21.15.32:0/2581788993 conn(0x7f81b40f7880 msgr2=0x7f81b40f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:00.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.492+0000 7f81bba16700 1 -- 172.21.15.32:0/2581788993 shutdown_connections 2024-04-12T05:47:00.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.492+0000 7f81bba16700 1 -- 172.21.15.32:0/2581788993 wait complete. 2024-04-12T05:47:00.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.493+0000 7f81bba16700 1 Processor -- start 2024-04-12T05:47:00.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.493+0000 7f81bba16700 1 -- start start 2024-04-12T05:47:00.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.493+0000 7f81bba16700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81b4101d10 0x7f81b4112d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:47:00.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.493+0000 7f81bba16700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81b41026b0 0x7f81b4113260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:47:00.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.493+0000 7f81bba16700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b41138b0 con 0x7f81b41026b0 2024-04-12T05:47:00.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.493+0000 7f81bba16700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b410cc80 con 0x7f81b4101d10 2024-04-12T05:47:00.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.493+0000 7f81b8fb1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81b41026b0 0x7f81b4113260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:00.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.493+0000 7f81b8fb1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81b41026b0 0x7f81b4113260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48516/0 (socket says 172.21.15.32:48516) 2024-04-12T05:47:00.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.493+0000 7f81b97b2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81b4101d10 0x7f81b4112d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:00.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.493+0000 7f81b8fb1700 1 -- 172.21.15.32:0/1253479423 learned_addr learned my addr 172.21.15.32:0/1253479423 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:47:00.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.493+0000 7f81b8fb1700 1 -- 172.21.15.32:0/1253479423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81b4101d10 msgr2=0x7f81b4112d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:00.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.493+0000 7f81b8fb1700 1 --2- 172.21.15.32:0/1253479423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81b4101d10 0x7f81b4112d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:00.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.493+0000 7f81b8fb1700 1 -- 172.21.15.32:0/1253479423 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b0005040 con 0x7f81b41026b0 2024-04-12T05:47:00.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.493+0000 7f81b8fb1700 1 --2- 172.21.15.32:0/1253479423 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81b41026b0 0x7f81b4113260 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f81a800c890 tx=0x7f81a80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:00.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.494+0000 7f81a5ffb700 1 -- 172.21.15.32:0/1253479423 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81a800a700 con 0x7f81b41026b0 2024-04-12T05:47:00.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.494+0000 7f81a5ffb700 1 -- 172.21.15.32:0/1253479423 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f81a800ad10 con 0x7f81b41026b0 2024-04-12T05:47:00.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.494+0000 7f81bba16700 1 -- 172.21.15.32:0/1253479423 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81b410cf60 con 0x7f81b41026b0 2024-04-12T05:47:00.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.494+0000 7f81a5ffb700 1 -- 172.21.15.32:0/1253479423 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81a8012e70 con 0x7f81b41026b0 2024-04-12T05:47:00.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.494+0000 7f81bba16700 1 -- 172.21.15.32:0/1253479423 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81b410d520 con 0x7f81b41026b0 2024-04-12T05:47:00.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.495+0000 7f81bba16700 1 -- 172.21.15.32:0/1253479423 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81b40fd460 con 0x7f81b41026b0 2024-04-12T05:47:00.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.495+0000 7f81a5ffb700 1 -- 172.21.15.32:0/1253479423 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f81a800a860 con 0x7f81b41026b0 2024-04-12T05:47:00.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.496+0000 7f81a5ffb700 1 --2- 172.21.15.32:0/1253479423 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f81a006cc20 0x7f81a006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:47:00.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.496+0000 7f81a5ffb700 1 -- 172.21.15.32:0/1253479423 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f81a808c110 con 0x7f81b41026b0 2024-04-12T05:47:00.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.496+0000 7f81b97b2700 1 --2- 172.21.15.32:0/1253479423 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f81a006cc20 0x7f81a006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:00.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.496+0000 7f81b97b2700 1 --2- 172.21.15.32:0/1253479423 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f81a006cc20 0x7f81a006f0d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f81b0004f90 tx=0x7f81b0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:00.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.501+0000 7f81a5ffb700 1 -- 172.21.15.32:0/1253479423 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f81a8059b80 con 0x7f81b41026b0 2024-04-12T05:47:00.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.725+0000 7f81bba16700 1 -- 172.21.15.32:0/1253479423 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f81b4108d40 con 0x7f81a006cc20 2024-04-12T05:47:00.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.727+0000 7f81a5ffb700 1 -- 172.21.15.32:0/1253479423 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f81b4108d40 con 0x7f81a006cc20 2024-04-12T05:47:00.723 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:47:00.723 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:47:00.723 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:47:00.723 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:47:00.723 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:47:00.723 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:47:00.724 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:47:00.724 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-12T05:47:00.724 INFO:teuthology.orchestra.run.smithi032.stdout: "osd" 2024-04-12T05:47:00.724 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:47:00.724 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "15/21 daemons upgraded", 2024-04-12T05:47:00.724 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:47:00.724 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:47:00.724 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:47:00.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.729+0000 7f81bba16700 1 -- 172.21.15.32:0/1253479423 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f81a006cc20 msgr2=0x7f81a006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:00.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.729+0000 7f81bba16700 1 --2- 172.21.15.32:0/1253479423 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f81a006cc20 0x7f81a006f0d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f81b0004f90 tx=0x7f81b0009e30 comp rx=0 tx=0).stop 2024-04-12T05:47:00.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.729+0000 7f81bba16700 1 -- 172.21.15.32:0/1253479423 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81b41026b0 msgr2=0x7f81b4113260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:00.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.729+0000 7f81bba16700 1 --2- 172.21.15.32:0/1253479423 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81b41026b0 0x7f81b4113260 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f81a800c890 tx=0x7f81a80044c0 comp rx=0 tx=0).stop 2024-04-12T05:47:00.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.729+0000 7f81bba16700 1 -- 172.21.15.32:0/1253479423 shutdown_connections 2024-04-12T05:47:00.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.729+0000 7f81bba16700 1 --2- 172.21.15.32:0/1253479423 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f81a006cc20 0x7f81a006f0d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:00.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.729+0000 7f81bba16700 1 --2- 172.21.15.32:0/1253479423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81b4101d10 0x7f81b4112d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:00.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.729+0000 7f81bba16700 1 --2- 172.21.15.32:0/1253479423 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81b41026b0 0x7f81b4113260 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:00.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.729+0000 7f81bba16700 1 -- 172.21.15.32:0/1253479423 >> 172.21.15.32:0/1253479423 conn(0x7f81b40f7880 msgr2=0x7f81b40f8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:00.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.729+0000 7f81bba16700 1 -- 172.21.15.32:0/1253479423 shutdown_connections 2024-04-12T05:47:00.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.729+0000 7f81bba16700 1 -- 172.21.15.32:0/1253479423 wait complete. 2024-04-12T05:47:00.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:00 smithi067 ceph-mon[112979]: from='client.44321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:00.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:00 smithi067 ceph-mon[112979]: from='client.34426 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:00.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:00 smithi067 ceph-mon[112979]: from='client.34430 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:00.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:00 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:47:00.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:00 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:47:00.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:00 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/49237477' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:47:00.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:00 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/4174019730' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:47:00.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.835+0000 7fc7d5624700 1 -- 172.21.15.32:0/2271297061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7d0102e70 msgr2=0x7fc7d0103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:00.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.835+0000 7fc7d5624700 1 --2- 172.21.15.32:0/2271297061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7d0102e70 0x7fc7d0103240 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4004940 tx=0x7fc7c4009e30 comp rx=0 tx=0).stop 2024-04-12T05:47:00.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.835+0000 7fc7d5624700 1 -- 172.21.15.32:0/2271297061 shutdown_connections 2024-04-12T05:47:00.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.835+0000 7fc7d5624700 1 --2- 172.21.15.32:0/2271297061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7d00fb4a0 0x7fc7d00fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:00.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.835+0000 7fc7d5624700 1 --2- 172.21.15.32:0/2271297061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7d0102e70 0x7fc7d0103240 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:00.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.835+0000 7fc7d5624700 1 -- 172.21.15.32:0/2271297061 >> 172.21.15.32:0/2271297061 conn(0x7fc7d00714c0 msgr2=0x7fc7d00718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:00.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.835+0000 7fc7d5624700 1 -- 172.21.15.32:0/2271297061 shutdown_connections 2024-04-12T05:47:00.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.835+0000 7fc7d5624700 1 -- 172.21.15.32:0/2271297061 wait complete. 2024-04-12T05:47:00.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.836+0000 7fc7d5624700 1 Processor -- start 2024-04-12T05:47:00.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.836+0000 7fc7d5624700 1 -- start start 2024-04-12T05:47:00.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.836+0000 7fc7d5624700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7d00fb4a0 0x7fc7d00fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:47:00.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.836+0000 7fc7d5624700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7d0102e70 0x7fc7d00fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:47:00.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.836+0000 7fc7d5624700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d00fd1e0 con 0x7fc7d0102e70 2024-04-12T05:47:00.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.836+0000 7fc7d5624700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d00fd320 con 0x7fc7d00fb4a0 2024-04-12T05:47:00.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.836+0000 7fc7ce7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7d0102e70 0x7fc7d00fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:00.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.836+0000 7fc7ce7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7d0102e70 0x7fc7d00fcbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48534/0 (socket says 172.21.15.32:48534) 2024-04-12T05:47:00.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.836+0000 7fc7ce7fc700 1 -- 172.21.15.32:0/437392654 learned_addr learned my addr 172.21.15.32:0/437392654 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:47:00.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.836+0000 7fc7ceffd700 1 --2- 172.21.15.32:0/437392654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7d00fb4a0 0x7fc7d00fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:00.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.836+0000 7fc7ce7fc700 1 -- 172.21.15.32:0/437392654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7d00fb4a0 msgr2=0x7fc7d00fc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:00.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.836+0000 7fc7ce7fc700 1 --2- 172.21.15.32:0/437392654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7d00fb4a0 0x7fc7d00fc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:00.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.836+0000 7fc7ce7fc700 1 -- 172.21.15.32:0/437392654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c4005040 con 0x7fc7d0102e70 2024-04-12T05:47:00.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.836+0000 7fc7ce7fc700 1 --2- 172.21.15.32:0/437392654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7d0102e70 0x7fc7d00fcbc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc7c000b790 tx=0x7fc7c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:00.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.837+0000 7fc7b77fe700 1 -- 172.21.15.32:0/437392654 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7c000f6f0 con 0x7fc7d0102e70 2024-04-12T05:47:00.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.837+0000 7fc7b77fe700 1 -- 172.21.15.32:0/437392654 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc7c000fd00 con 0x7fc7d0102e70 2024-04-12T05:47:00.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.837+0000 7fc7d5624700 1 -- 172.21.15.32:0/437392654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7d0119d90 con 0x7fc7d0102e70 2024-04-12T05:47:00.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.837+0000 7fc7b77fe700 1 -- 172.21.15.32:0/437392654 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7c00103a0 con 0x7fc7d0102e70 2024-04-12T05:47:00.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.837+0000 7fc7d5624700 1 -- 172.21.15.32:0/437392654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7d011a2a0 con 0x7fc7d0102e70 2024-04-12T05:47:00.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.838+0000 7fc7d5624700 1 -- 172.21.15.32:0/437392654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7d0100110 con 0x7fc7d0102e70 2024-04-12T05:47:00.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.838+0000 7fc7b77fe700 1 -- 172.21.15.32:0/437392654 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc7c000f850 con 0x7fc7d0102e70 2024-04-12T05:47:00.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.838+0000 7fc7b77fe700 1 --2- 172.21.15.32:0/437392654 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc7b806cc20 0x7fc7b806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:47:00.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.839+0000 7fc7b77fe700 1 -- 172.21.15.32:0/437392654 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fc7c008be20 con 0x7fc7d0102e70 2024-04-12T05:47:00.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.839+0000 7fc7ceffd700 1 --2- 172.21.15.32:0/437392654 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc7b806cc20 0x7fc7b806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:00.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.839+0000 7fc7ceffd700 1 --2- 172.21.15.32:0/437392654 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc7b806cc20 0x7fc7b806f0d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4004f90 tx=0x7fc7c4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:00.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:00 smithi032 ceph-mon[146817]: from='client.44321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:00.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:00 smithi032 ceph-mon[146817]: from='client.34426 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:00.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:00 smithi032 ceph-mon[146817]: from='client.34430 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:00.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:00 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:47:00.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:00 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:47:00.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:00 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/49237477' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:47:00.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:00 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/4174019730' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:47:00.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:00.843+0000 7fc7b77fe700 1 -- 172.21.15.32:0/437392654 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc7c0055ea0 con 0x7fc7d0102e70 2024-04-12T05:47:01.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:01.130+0000 7fc7d5624700 1 -- 172.21.15.32:0/437392654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc7d0065380 con 0x7fc7d0102e70 2024-04-12T05:47:01.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:01.130+0000 7fc7b77fe700 1 -- 172.21.15.32:0/437392654 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fc7c00594c0 con 0x7fc7d0102e70 2024-04-12T05:47:01.127 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:47:01.127 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:47:01.127 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-12T05:47:01.127 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:47:01.127 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:47:01.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:01.132+0000 7fc7d5624700 1 -- 172.21.15.32:0/437392654 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc7b806cc20 msgr2=0x7fc7b806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:01.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:01.132+0000 7fc7d5624700 1 --2- 172.21.15.32:0/437392654 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc7b806cc20 0x7fc7b806f0d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4004f90 tx=0x7fc7c4009e30 comp rx=0 tx=0).stop 2024-04-12T05:47:01.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:01.132+0000 7fc7d5624700 1 -- 172.21.15.32:0/437392654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7d0102e70 msgr2=0x7fc7d00fcbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:01.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:01.132+0000 7fc7d5624700 1 --2- 172.21.15.32:0/437392654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7d0102e70 0x7fc7d00fcbc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc7c000b790 tx=0x7fc7c0009250 comp rx=0 tx=0).stop 2024-04-12T05:47:01.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:01.133+0000 7fc7d5624700 1 -- 172.21.15.32:0/437392654 shutdown_connections 2024-04-12T05:47:01.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:01.133+0000 7fc7d5624700 1 --2- 172.21.15.32:0/437392654 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc7b806cc20 0x7fc7b806f0d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:01.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:01.133+0000 7fc7d5624700 1 --2- 172.21.15.32:0/437392654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7d00fb4a0 0x7fc7d00fc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:01.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:01.133+0000 7fc7d5624700 1 --2- 172.21.15.32:0/437392654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7d0102e70 0x7fc7d00fcbc0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:01.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:01.133+0000 7fc7d5624700 1 -- 172.21.15.32:0/437392654 >> 172.21.15.32:0/437392654 conn(0x7fc7d00714c0 msgr2=0x7fc7d00fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:01.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:01.133+0000 7fc7d5624700 1 -- 172.21.15.32:0/437392654 shutdown_connections 2024-04-12T05:47:01.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:01.133+0000 7fc7d5624700 1 -- 172.21.15.32:0/437392654 wait complete. 2024-04-12T05:47:01.615 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:01 smithi067 ceph-mon[112979]: pgmap v194: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:01.615 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:01 smithi067 ceph-mon[112979]: from='client.34442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:01.615 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:01 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/437392654' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:47:01.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:01 smithi032 ceph-mon[146817]: pgmap v194: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:01.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:01 smithi032 ceph-mon[146817]: from='client.34442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:01.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:01 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/437392654' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:47:03.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:03 smithi067 ceph-mon[112979]: pgmap v195: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:03.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:47:03.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:47:03.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:47:03.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:47:03.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:47:03.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:47:03.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:03 smithi032 ceph-mon[146817]: pgmap v195: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:03.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:47:03.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:47:03.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:47:03.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:47:03.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:47:03.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:47:04.649 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:04 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:47:04.649 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:04 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:47:04.649 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:04 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:47:04.649 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:04 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:47:04.649 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:04 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:47:04.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:04 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:47:04.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:04 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:47:04.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:04 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:47:04.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:04 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:47:04.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:04 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:47:05.713 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:05 smithi067 ceph-mon[112979]: pgmap v196: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:05.714 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:05 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:47:05.714 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:05 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.ymvqih", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:47:05.714 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:05 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:47:05.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:05 smithi032 ceph-mon[146817]: pgmap v196: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:05.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:05 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:47:05.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:05 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.ymvqih", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:47:05.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:05 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:47:06.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:06 smithi067 ceph-mon[112979]: Upgrade: Updating mds.cephfs.smithi067.ymvqih 2024-04-12T05:47:06.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:06 smithi067 ceph-mon[112979]: Deploying daemon mds.cephfs.smithi067.ymvqih on smithi067 2024-04-12T05:47:06.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:06 smithi032 ceph-mon[146817]: Upgrade: Updating mds.cephfs.smithi067.ymvqih 2024-04-12T05:47:06.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:06 smithi032 ceph-mon[146817]: Deploying daemon mds.cephfs.smithi067.ymvqih on smithi067 2024-04-12T05:47:07.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:07 smithi067 ceph-mon[112979]: pgmap v197: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:07.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:07 smithi067 ceph-mon[112979]: osdmap e85: 6 total, 6 up, 6 in 2024-04-12T05:47:07.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:07 smithi032 ceph-mon[146817]: pgmap v197: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:07.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:07 smithi032 ceph-mon[146817]: osdmap e85: 6 total, 6 up, 6 in 2024-04-12T05:47:08.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:08 smithi067 ceph-mon[112979]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-12T05:47:08.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:08 smithi032 ceph-mon[146817]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-12T05:47:09.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:09 smithi067 ceph-mon[112979]: pgmap v199: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:09.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:09 smithi032 ceph-mon[146817]: pgmap v199: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:11.701 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:11 smithi067 ceph-mon[112979]: pgmap v200: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:11.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:11 smithi032 ceph-mon[146817]: pgmap v200: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:13.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:13 smithi067 ceph-mon[112979]: pgmap v201: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:13.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:13 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:47:13.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:13 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:47:13.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:13 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:47:13.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:13 smithi067 ceph-mon[112979]: mds.? [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] up:boot 2024-04-12T05:47:13.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:13 smithi067 ceph-mon[112979]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-12T05:47:13.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:13 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.ymvqih"}]: dispatch 2024-04-12T05:47:13.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:13 smithi032 ceph-mon[146817]: pgmap v201: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:13.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:13 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:47:13.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:13 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:47:13.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:13 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:47:13.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:13 smithi032 ceph-mon[146817]: mds.? [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] up:boot 2024-04-12T05:47:13.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:13 smithi032 ceph-mon[146817]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-12T05:47:13.839 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:13 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.ymvqih"}]: dispatch 2024-04-12T05:47:15.806 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:15 smithi067 ceph-mon[112979]: pgmap v202: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:15.807 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:15 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:47:15.807 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:15 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:47:15.807 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:15 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:47:15.807 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:15 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:47:15.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:15 smithi032 ceph-mon[146817]: pgmap v202: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:15.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:15 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:47:15.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:15 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:47:15.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:15 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:47:15.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:15 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T05:47:17.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:17 smithi032 ceph-mon[146817]: pgmap v203: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:18.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:17 smithi067 ceph-mon[112979]: pgmap v203: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:18.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:47:19.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:47:19.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:19 smithi032 ceph-mon[146817]: pgmap v204: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:20.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:19 smithi067 ceph-mon[112979]: pgmap v204: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:21.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:21 smithi032 ceph-mon[146817]: pgmap v205: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:22.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:21 smithi067 ceph-mon[112979]: pgmap v205: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:23.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:23 smithi032 ceph-mon[146817]: pgmap v206: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:24.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:23 smithi067 ceph-mon[112979]: pgmap v206: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:25.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:25 smithi032 ceph-mon[146817]: pgmap v207: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:26.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:25 smithi067 ceph-mon[112979]: pgmap v207: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:27.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:27 smithi032 ceph-mon[146817]: pgmap v208: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:28.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:27 smithi067 ceph-mon[112979]: pgmap v208: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:29.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:29 smithi032 ceph-mon[146817]: pgmap v209: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:30.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:29 smithi067 ceph-mon[112979]: pgmap v209: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:31.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.252+0000 7f5dcfe1a700 1 -- 172.21.15.32:0/295779820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dc8102710 msgr2=0x7f5dc80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:31.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.252+0000 7f5dcfe1a700 1 --2- 172.21.15.32:0/295779820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dc8102710 0x7f5dc80fbd80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5db8004940 tx=0x7f5db8009e30 comp rx=0 tx=0).stop 2024-04-12T05:47:31.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.252+0000 7f5dcfe1a700 1 -- 172.21.15.32:0/295779820 shutdown_connections 2024-04-12T05:47:31.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.252+0000 7f5dcfe1a700 1 --2- 172.21.15.32:0/295779820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dc8102710 0x7f5dc80fbd80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:31.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.252+0000 7f5dcfe1a700 1 --2- 172.21.15.32:0/295779820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dc8101d70 0x7f5dc8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:31.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.252+0000 7f5dcfe1a700 1 -- 172.21.15.32:0/295779820 >> 172.21.15.32:0/295779820 conn(0x7f5dc8072bc0 msgr2=0x7f5dc8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:31.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.252+0000 7f5dcfe1a700 1 -- 172.21.15.32:0/295779820 shutdown_connections 2024-04-12T05:47:31.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.252+0000 7f5dcfe1a700 1 -- 172.21.15.32:0/295779820 wait complete. 2024-04-12T05:47:31.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.252+0000 7f5dcfe1a700 1 Processor -- start 2024-04-12T05:47:31.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.253+0000 7f5dcfe1a700 1 -- start start 2024-04-12T05:47:31.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.253+0000 7f5dcfe1a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dc8101d70 0x7f5dc810ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:47:31.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.253+0000 7f5dcfe1a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dc8102710 0x7f5dc810efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:47:31.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.253+0000 7f5dcfe1a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc810f5f0 con 0x7f5dc8102710 2024-04-12T05:47:31.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.253+0000 7f5dcfe1a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc810f730 con 0x7f5dc8101d70 2024-04-12T05:47:31.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.253+0000 7f5dcd3b5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dc8102710 0x7f5dc810efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:31.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.253+0000 7f5dcd3b5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dc8102710 0x7f5dc810efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39338/0 (socket says 172.21.15.32:39338) 2024-04-12T05:47:31.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.253+0000 7f5dcd3b5700 1 -- 172.21.15.32:0/2935652553 learned_addr learned my addr 172.21.15.32:0/2935652553 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:47:31.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.253+0000 7f5dcdbb6700 1 --2- 172.21.15.32:0/2935652553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dc8101d70 0x7f5dc810ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:31.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.253+0000 7f5dcd3b5700 1 -- 172.21.15.32:0/2935652553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dc8101d70 msgr2=0x7f5dc810ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:31.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.253+0000 7f5dcd3b5700 1 --2- 172.21.15.32:0/2935652553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dc8101d70 0x7f5dc810ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:31.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.253+0000 7f5dcd3b5700 1 -- 172.21.15.32:0/2935652553 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db8005040 con 0x7f5dc8102710 2024-04-12T05:47:31.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.253+0000 7f5dcd3b5700 1 --2- 172.21.15.32:0/2935652553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dc8102710 0x7f5dc810efa0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5db800b3b0 tx=0x7f5db8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:31.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.254+0000 7f5dbe7fc700 1 -- 172.21.15.32:0/2935652553 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5db800e040 con 0x7f5dc8102710 2024-04-12T05:47:31.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.254+0000 7f5dbe7fc700 1 -- 172.21.15.32:0/2935652553 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5db8013420 con 0x7f5dc8102710 2024-04-12T05:47:31.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.254+0000 7f5dcfe1a700 1 -- 172.21.15.32:0/2935652553 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc8117ad0 con 0x7f5dc8102710 2024-04-12T05:47:31.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.254+0000 7f5dbe7fc700 1 -- 172.21.15.32:0/2935652553 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5db801c630 con 0x7f5dc8102710 2024-04-12T05:47:31.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.254+0000 7f5dcfe1a700 1 -- 172.21.15.32:0/2935652553 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc8117fe0 con 0x7f5dc8102710 2024-04-12T05:47:31.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.255+0000 7f5dcfe1a700 1 -- 172.21.15.32:0/2935652553 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc80ff080 con 0x7f5dc8102710 2024-04-12T05:47:31.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.255+0000 7f5dbe7fc700 1 -- 172.21.15.32:0/2935652553 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f5db8013590 con 0x7f5dc8102710 2024-04-12T05:47:31.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.256+0000 7f5dbe7fc700 1 --2- 172.21.15.32:0/2935652553 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5db406cc30 0x7f5db406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:47:31.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.256+0000 7f5dcdbb6700 1 --2- 172.21.15.32:0/2935652553 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5db406cc30 0x7f5db406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:31.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.256+0000 7f5dbe7fc700 1 -- 172.21.15.32:0/2935652553 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f5db808d220 con 0x7f5dc8102710 2024-04-12T05:47:31.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.256+0000 7f5dcdbb6700 1 --2- 172.21.15.32:0/2935652553 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5db406cc30 0x7f5db406f0e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4004d60 tx=0x7f5dc400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:31.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.261+0000 7f5dbe7fc700 1 -- 172.21.15.32:0/2935652553 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5db80572b0 con 0x7f5dc8102710 2024-04-12T05:47:31.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.482+0000 7f5dcfe1a700 1 -- 172.21.15.32:0/2935652553 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5dc8108d10 con 0x7f5db406cc30 2024-04-12T05:47:31.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.483+0000 7f5dbe7fc700 1 -- 172.21.15.32:0/2935652553 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f5dc8108d10 con 0x7f5db406cc30 2024-04-12T05:47:31.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.485+0000 7f5dcfe1a700 1 -- 172.21.15.32:0/2935652553 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5db406cc30 msgr2=0x7f5db406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:31.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.485+0000 7f5dcfe1a700 1 --2- 172.21.15.32:0/2935652553 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5db406cc30 0x7f5db406f0e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4004d60 tx=0x7f5dc400b040 comp rx=0 tx=0).stop 2024-04-12T05:47:31.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.485+0000 7f5dcfe1a700 1 -- 172.21.15.32:0/2935652553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dc8102710 msgr2=0x7f5dc810efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:31.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.485+0000 7f5dcfe1a700 1 --2- 172.21.15.32:0/2935652553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dc8102710 0x7f5dc810efa0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5db800b3b0 tx=0x7f5db8009e30 comp rx=0 tx=0).stop 2024-04-12T05:47:31.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.485+0000 7f5dcfe1a700 1 -- 172.21.15.32:0/2935652553 shutdown_connections 2024-04-12T05:47:31.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.485+0000 7f5dcfe1a700 1 --2- 172.21.15.32:0/2935652553 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5db406cc30 0x7f5db406f0e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:31.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.485+0000 7f5dcfe1a700 1 --2- 172.21.15.32:0/2935652553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dc8101d70 0x7f5dc810ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:31.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.485+0000 7f5dcfe1a700 1 --2- 172.21.15.32:0/2935652553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dc8102710 0x7f5dc810efa0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:31.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.485+0000 7f5dcfe1a700 1 -- 172.21.15.32:0/2935652553 >> 172.21.15.32:0/2935652553 conn(0x7f5dc8072bc0 msgr2=0x7f5dc80fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:31.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.485+0000 7f5dcfe1a700 1 -- 172.21.15.32:0/2935652553 shutdown_connections 2024-04-12T05:47:31.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.485+0000 7f5dcfe1a700 1 -- 172.21.15.32:0/2935652553 wait complete. 2024-04-12T05:47:31.492 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:47:31.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.590+0000 7fbd1c293700 1 -- 172.21.15.32:0/426706559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd14102f70 msgr2=0x7fbd14103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:31.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.590+0000 7fbd1c293700 1 --2- 172.21.15.32:0/426706559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd14102f70 0x7fbd14103340 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fbd10004940 tx=0x7fbd10009e30 comp rx=0 tx=0).stop 2024-04-12T05:47:31.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.591+0000 7fbd1c293700 1 -- 172.21.15.32:0/426706559 shutdown_connections 2024-04-12T05:47:31.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.591+0000 7fbd1c293700 1 --2- 172.21.15.32:0/426706559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd140fb3b0 0x7fbd140fb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:31.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.591+0000 7fbd1c293700 1 --2- 172.21.15.32:0/426706559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd14102f70 0x7fbd14103340 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:31.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.591+0000 7fbd1c293700 1 -- 172.21.15.32:0/426706559 >> 172.21.15.32:0/426706559 conn(0x7fbd14072310 msgr2=0x7fbd14072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:31.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.591+0000 7fbd1c293700 1 -- 172.21.15.32:0/426706559 shutdown_connections 2024-04-12T05:47:31.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.591+0000 7fbd1c293700 1 -- 172.21.15.32:0/426706559 wait complete. 2024-04-12T05:47:31.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.591+0000 7fbd1c293700 1 Processor -- start 2024-04-12T05:47:31.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.591+0000 7fbd1c293700 1 -- start start 2024-04-12T05:47:31.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.591+0000 7fbd1c293700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd140fb3b0 0x7fbd141138f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:47:31.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.591+0000 7fbd1c293700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd14102f70 0x7fbd14113e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:47:31.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.591+0000 7fbd1c293700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd14114480 con 0x7fbd140fb3b0 2024-04-12T05:47:31.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.591+0000 7fbd1c293700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1410dad0 con 0x7fbd14102f70 2024-04-12T05:47:31.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.592+0000 7fbd1a02f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd140fb3b0 0x7fbd141138f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:31.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.592+0000 7fbd1a02f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd140fb3b0 0x7fbd141138f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36264/0 (socket says 172.21.15.32:36264) 2024-04-12T05:47:31.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.592+0000 7fbd1a02f700 1 -- 172.21.15.32:0/2663574726 learned_addr learned my addr 172.21.15.32:0/2663574726 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:47:31.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.592+0000 7fbd1982e700 1 --2- 172.21.15.32:0/2663574726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd14102f70 0x7fbd14113e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:31.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.592+0000 7fbd1a02f700 1 -- 172.21.15.32:0/2663574726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd14102f70 msgr2=0x7fbd14113e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:31.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.592+0000 7fbd1a02f700 1 --2- 172.21.15.32:0/2663574726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd14102f70 0x7fbd14113e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:31.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.592+0000 7fbd1a02f700 1 -- 172.21.15.32:0/2663574726 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd10005040 con 0x7fbd140fb3b0 2024-04-12T05:47:31.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.592+0000 7fbd1a02f700 1 --2- 172.21.15.32:0/2663574726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd140fb3b0 0x7fbd141138f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbd10009150 tx=0x7fbd10009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:31.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.592+0000 7fbd0affd700 1 -- 172.21.15.32:0/2663574726 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd1000e040 con 0x7fbd140fb3b0 2024-04-12T05:47:31.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.592+0000 7fbd1c293700 1 -- 172.21.15.32:0/2663574726 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd1410dd50 con 0x7fbd140fb3b0 2024-04-12T05:47:31.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.592+0000 7fbd0affd700 1 -- 172.21.15.32:0/2663574726 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd10013420 con 0x7fbd140fb3b0 2024-04-12T05:47:31.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.592+0000 7fbd0affd700 1 -- 172.21.15.32:0/2663574726 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd1001c630 con 0x7fbd140fb3b0 2024-04-12T05:47:31.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.592+0000 7fbd1c293700 1 -- 172.21.15.32:0/2663574726 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd1410e290 con 0x7fbd140fb3b0 2024-04-12T05:47:31.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.593+0000 7fbd1c293700 1 -- 172.21.15.32:0/2663574726 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd141053b0 con 0x7fbd140fb3b0 2024-04-12T05:47:31.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.594+0000 7fbd0affd700 1 -- 172.21.15.32:0/2663574726 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fbd10013590 con 0x7fbd140fb3b0 2024-04-12T05:47:31.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.594+0000 7fbd0affd700 1 --2- 172.21.15.32:0/2663574726 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbd0006cc20 0x7fbd0006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:47:31.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.594+0000 7fbd1982e700 1 --2- 172.21.15.32:0/2663574726 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbd0006cc20 0x7fbd0006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:31.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.594+0000 7fbd0affd700 1 -- 172.21.15.32:0/2663574726 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fbd1008d2e0 con 0x7fbd140fb3b0 2024-04-12T05:47:31.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.595+0000 7fbd1982e700 1 --2- 172.21.15.32:0/2663574726 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbd0006cc20 0x7fbd0006f0d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbd1410ec40 tx=0x7fbd0400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:31.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.599+0000 7fbd0affd700 1 -- 172.21.15.32:0/2663574726 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbd1005ad50 con 0x7fbd140fb3b0 2024-04-12T05:47:31.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.821+0000 7fbd1c293700 1 -- 172.21.15.32:0/2663574726 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbd1410ef40 con 0x7fbd0006cc20 2024-04-12T05:47:31.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.822+0000 7fbd0affd700 1 -- 172.21.15.32:0/2663574726 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fbd1410ef40 con 0x7fbd0006cc20 2024-04-12T05:47:31.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.824+0000 7fbd1c293700 1 -- 172.21.15.32:0/2663574726 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbd0006cc20 msgr2=0x7fbd0006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:31.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.824+0000 7fbd1c293700 1 --2- 172.21.15.32:0/2663574726 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbd0006cc20 0x7fbd0006f0d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbd1410ec40 tx=0x7fbd0400b040 comp rx=0 tx=0).stop 2024-04-12T05:47:31.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.824+0000 7fbd1c293700 1 -- 172.21.15.32:0/2663574726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd140fb3b0 msgr2=0x7fbd141138f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:31.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.824+0000 7fbd1c293700 1 --2- 172.21.15.32:0/2663574726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd140fb3b0 0x7fbd141138f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbd10009150 tx=0x7fbd10009e30 comp rx=0 tx=0).stop 2024-04-12T05:47:31.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.824+0000 7fbd1c293700 1 -- 172.21.15.32:0/2663574726 shutdown_connections 2024-04-12T05:47:31.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.824+0000 7fbd1c293700 1 --2- 172.21.15.32:0/2663574726 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbd0006cc20 0x7fbd0006f0d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:31.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.824+0000 7fbd1c293700 1 --2- 172.21.15.32:0/2663574726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd140fb3b0 0x7fbd141138f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:31.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.824+0000 7fbd1c293700 1 --2- 172.21.15.32:0/2663574726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd14102f70 0x7fbd14113e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:31.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.824+0000 7fbd1c293700 1 -- 172.21.15.32:0/2663574726 >> 172.21.15.32:0/2663574726 conn(0x7fbd14072310 msgr2=0x7fbd140f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:31.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.824+0000 7fbd1c293700 1 -- 172.21.15.32:0/2663574726 shutdown_connections 2024-04-12T05:47:31.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.824+0000 7fbd1c293700 1 -- 172.21.15.32:0/2663574726 wait complete. 2024-04-12T05:47:31.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:31 smithi032 ceph-mon[146817]: pgmap v210: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:31.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.928+0000 7f2d54ae7700 1 -- 172.21.15.32:0/173841635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d50070360 msgr2=0x7f2d50070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:31.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.928+0000 7f2d54ae7700 1 --2- 172.21.15.32:0/173841635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d50070360 0x7f2d50070730 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2d40004970 tx=0x7f2d40009e30 comp rx=0 tx=0).stop 2024-04-12T05:47:31.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.928+0000 7f2d54ae7700 1 -- 172.21.15.32:0/173841635 shutdown_connections 2024-04-12T05:47:31.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.928+0000 7f2d54ae7700 1 --2- 172.21.15.32:0/173841635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d50070c70 0x7f2d50109120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:31.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.928+0000 7f2d54ae7700 1 --2- 172.21.15.32:0/173841635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d50070360 0x7f2d50070730 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:31.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.928+0000 7f2d54ae7700 1 -- 172.21.15.32:0/173841635 >> 172.21.15.32:0/173841635 conn(0x7f2d500f8980 msgr2=0x7f2d500fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:31.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.928+0000 7f2d54ae7700 1 -- 172.21.15.32:0/173841635 shutdown_connections 2024-04-12T05:47:31.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.928+0000 7f2d54ae7700 1 -- 172.21.15.32:0/173841635 wait complete. 2024-04-12T05:47:31.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.928+0000 7f2d54ae7700 1 Processor -- start 2024-04-12T05:47:31.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.928+0000 7f2d54ae7700 1 -- start start 2024-04-12T05:47:31.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.929+0000 7f2d54ae7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d50070360 0x7f2d50115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:47:31.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.929+0000 7f2d54ae7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d50070c70 0x7f2d501161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:47:31.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.929+0000 7f2d54ae7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d50116800 con 0x7f2d50070c70 2024-04-12T05:47:31.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.929+0000 7f2d54ae7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5010fbd0 con 0x7f2d50070360 2024-04-12T05:47:31.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.929+0000 7f2d4dd9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d50070c70 0x7f2d501161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:31.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.929+0000 7f2d4dd9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d50070c70 0x7f2d501161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36280/0 (socket says 172.21.15.32:36280) 2024-04-12T05:47:31.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.929+0000 7f2d4dd9b700 1 -- 172.21.15.32:0/2247581849 learned_addr learned my addr 172.21.15.32:0/2247581849 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:47:31.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.929+0000 7f2d4e59c700 1 --2- 172.21.15.32:0/2247581849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d50070360 0x7f2d50115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:31.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.929+0000 7f2d4dd9b700 1 -- 172.21.15.32:0/2247581849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d50070360 msgr2=0x7f2d50115c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:31.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.929+0000 7f2d4dd9b700 1 --2- 172.21.15.32:0/2247581849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d50070360 0x7f2d50115c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:31.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.929+0000 7f2d4dd9b700 1 -- 172.21.15.32:0/2247581849 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d40005040 con 0x7f2d50070c70 2024-04-12T05:47:31.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.929+0000 7f2d4dd9b700 1 --2- 172.21.15.32:0/2247581849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d50070c70 0x7f2d501161b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2d44002a50 tx=0x7f2d440046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:31.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.930+0000 7f2d3effd700 1 -- 172.21.15.32:0/2247581849 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d44007cc0 con 0x7f2d50070c70 2024-04-12T05:47:31.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.930+0000 7f2d3effd700 1 -- 172.21.15.32:0/2247581849 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d4400f040 con 0x7f2d50070c70 2024-04-12T05:47:31.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.930+0000 7f2d54ae7700 1 -- 172.21.15.32:0/2247581849 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d5010feb0 con 0x7f2d50070c70 2024-04-12T05:47:31.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.930+0000 7f2d3effd700 1 -- 172.21.15.32:0/2247581849 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d4400d9c0 con 0x7f2d50070c70 2024-04-12T05:47:31.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.930+0000 7f2d54ae7700 1 -- 172.21.15.32:0/2247581849 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d501103f0 con 0x7f2d50070c70 2024-04-12T05:47:31.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.931+0000 7f2d54ae7700 1 -- 172.21.15.32:0/2247581849 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d50071aa0 con 0x7f2d50070c70 2024-04-12T05:47:31.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.931+0000 7f2d3effd700 1 -- 172.21.15.32:0/2247581849 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f2d4400db20 con 0x7f2d50070c70 2024-04-12T05:47:31.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.932+0000 7f2d3effd700 1 --2- 172.21.15.32:0/2247581849 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f2d3806cc20 0x7f2d3806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:47:31.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.932+0000 7f2d4e59c700 1 --2- 172.21.15.32:0/2247581849 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f2d3806cc20 0x7f2d3806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:31.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.932+0000 7f2d3effd700 1 -- 172.21.15.32:0/2247581849 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f2d4408be70 con 0x7f2d50070c70 2024-04-12T05:47:31.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.932+0000 7f2d4e59c700 1 --2- 172.21.15.32:0/2247581849 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f2d3806cc20 0x7f2d3806f0d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2d40004f50 tx=0x7f2d40009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:31.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:31.936+0000 7f2d3effd700 1 -- 172.21.15.32:0/2247581849 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2d440597c0 con 0x7f2d50070c70 2024-04-12T05:47:32.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:31 smithi067 ceph-mon[112979]: pgmap v210: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:32.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.147+0000 7f2d54ae7700 1 -- 172.21.15.32:0/2247581849 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d5010c030 con 0x7f2d3806cc20 2024-04-12T05:47:32.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.157+0000 7f2d3effd700 1 -- 172.21.15.32:0/2247581849 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f2d5010c030 con 0x7f2d3806cc20 2024-04-12T05:47:32.154 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:47:32.154 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (7m) 47s ago 18m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:47:32.154 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (4m) 47s ago 18m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:47:32.154 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (4m) 17s ago 17m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:47:32.154 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (7m) 47s ago 18m 87.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:47:32.154 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (72s) 47s ago 13m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T05:47:32.154 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (52s) 47s ago 12m 13.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T05:47:32.154 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (20s) 17s ago 12m 11.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 6d4aca6f74a0 2024-04-12T05:47:32.154 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (36s) 17s ago 13m 14.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T05:47:32.154 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (8m) 47s ago 20m 507M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:47:32.154 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (8m) 17s ago 17m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:47:32.154 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (5m) 47s ago 20m 53.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:47:32.154 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (5m) 17s ago 17m 43.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:47:32.155 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (8m) 47s ago 18m 16.9M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:47:32.155 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (7m) 17s ago 17m 18.4M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:47:32.155 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (4m) 47s ago 16m 91.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T05:47:32.155 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (3m) 47s ago 16m 97.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T05:47:32.155 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (3m) 47s ago 16m 81.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T05:47:32.155 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (2m) 17s ago 15m 113M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T05:47:32.155 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (2m) 17s ago 15m 90.3M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T05:47:32.155 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (91s) 17s ago 14m 84.8M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T05:47:32.155 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (7m) 47s ago 18m 64.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:47:32.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.159+0000 7f2d54ae7700 1 -- 172.21.15.32:0/2247581849 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f2d3806cc20 msgr2=0x7f2d3806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:32.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.159+0000 7f2d54ae7700 1 --2- 172.21.15.32:0/2247581849 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f2d3806cc20 0x7f2d3806f0d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2d40004f50 tx=0x7f2d40009e30 comp rx=0 tx=0).stop 2024-04-12T05:47:32.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.159+0000 7f2d54ae7700 1 -- 172.21.15.32:0/2247581849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d50070c70 msgr2=0x7f2d501161b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:32.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.159+0000 7f2d54ae7700 1 --2- 172.21.15.32:0/2247581849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d50070c70 0x7f2d501161b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2d44002a50 tx=0x7f2d440046b0 comp rx=0 tx=0).stop 2024-04-12T05:47:32.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.159+0000 7f2d54ae7700 1 -- 172.21.15.32:0/2247581849 shutdown_connections 2024-04-12T05:47:32.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.159+0000 7f2d54ae7700 1 --2- 172.21.15.32:0/2247581849 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f2d3806cc20 0x7f2d3806f0d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:32.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.159+0000 7f2d54ae7700 1 --2- 172.21.15.32:0/2247581849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d50070360 0x7f2d50115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:32.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.159+0000 7f2d54ae7700 1 --2- 172.21.15.32:0/2247581849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d50070c70 0x7f2d501161b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:32.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.159+0000 7f2d54ae7700 1 -- 172.21.15.32:0/2247581849 >> 172.21.15.32:0/2247581849 conn(0x7f2d500f8980 msgr2=0x7f2d501059d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:32.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.160+0000 7f2d54ae7700 1 -- 172.21.15.32:0/2247581849 shutdown_connections 2024-04-12T05:47:32.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.160+0000 7f2d54ae7700 1 -- 172.21.15.32:0/2247581849 wait complete. 2024-04-12T05:47:32.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.262+0000 7f9b8eb0f700 1 -- 172.21.15.32:0/3053206601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b88102e10 msgr2=0x7f9b881031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:32.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.262+0000 7f9b8eb0f700 1 --2- 172.21.15.32:0/3053206601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b88102e10 0x7f9b881031e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c004940 tx=0x7f9b7c009e30 comp rx=0 tx=0).stop 2024-04-12T05:47:32.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.262+0000 7f9b8eb0f700 1 -- 172.21.15.32:0/3053206601 shutdown_connections 2024-04-12T05:47:32.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.262+0000 7f9b8eb0f700 1 --2- 172.21.15.32:0/3053206601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b881037b0 0x7f9b880fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:32.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.262+0000 7f9b8eb0f700 1 --2- 172.21.15.32:0/3053206601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b88102e10 0x7f9b881031e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:32.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.262+0000 7f9b8eb0f700 1 -- 172.21.15.32:0/3053206601 >> 172.21.15.32:0/3053206601 conn(0x7f9b880f8980 msgr2=0x7f9b880fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:32.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.262+0000 7f9b8eb0f700 1 -- 172.21.15.32:0/3053206601 shutdown_connections 2024-04-12T05:47:32.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.262+0000 7f9b8eb0f700 1 -- 172.21.15.32:0/3053206601 wait complete. 2024-04-12T05:47:32.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.263+0000 7f9b8eb0f700 1 Processor -- start 2024-04-12T05:47:32.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.263+0000 7f9b8eb0f700 1 -- start start 2024-04-12T05:47:32.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.263+0000 7f9b8eb0f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b88102e10 0x7f9b88113b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:47:32.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.263+0000 7f9b8eb0f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b881037b0 0x7f9b88114050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:47:32.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.263+0000 7f9b8eb0f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b881146a0 con 0x7f9b881037b0 2024-04-12T05:47:32.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.263+0000 7f9b8eb0f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8810da90 con 0x7f9b88102e10 2024-04-12T05:47:32.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.263+0000 7f9b87fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b881037b0 0x7f9b88114050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:32.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.263+0000 7f9b87fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b881037b0 0x7f9b88114050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36288/0 (socket says 172.21.15.32:36288) 2024-04-12T05:47:32.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.263+0000 7f9b87fff700 1 -- 172.21.15.32:0/306075317 learned_addr learned my addr 172.21.15.32:0/306075317 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:47:32.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.263+0000 7f9b8c8ab700 1 --2- 172.21.15.32:0/306075317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b88102e10 0x7f9b88113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:32.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.263+0000 7f9b87fff700 1 -- 172.21.15.32:0/306075317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b88102e10 msgr2=0x7f9b88113b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:32.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.263+0000 7f9b87fff700 1 --2- 172.21.15.32:0/306075317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b88102e10 0x7f9b88113b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:32.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.264+0000 7f9b87fff700 1 -- 172.21.15.32:0/306075317 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b7c005040 con 0x7f9b881037b0 2024-04-12T05:47:32.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.264+0000 7f9b87fff700 1 --2- 172.21.15.32:0/306075317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b881037b0 0x7f9b88114050 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9b7400c890 tx=0x7f9b740044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:32.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.264+0000 7f9b857fa700 1 -- 172.21.15.32:0/306075317 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b7400a7d0 con 0x7f9b881037b0 2024-04-12T05:47:32.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.264+0000 7f9b857fa700 1 -- 172.21.15.32:0/306075317 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9b7400ade0 con 0x7f9b881037b0 2024-04-12T05:47:32.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.264+0000 7f9b8eb0f700 1 -- 172.21.15.32:0/306075317 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b8810dd70 con 0x7f9b881037b0 2024-04-12T05:47:32.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.264+0000 7f9b857fa700 1 -- 172.21.15.32:0/306075317 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b740113a0 con 0x7f9b881037b0 2024-04-12T05:47:32.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.264+0000 7f9b8eb0f700 1 -- 172.21.15.32:0/306075317 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b8810e330 con 0x7f9b881037b0 2024-04-12T05:47:32.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.265+0000 7f9b8eb0f700 1 -- 172.21.15.32:0/306075317 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b880fe560 con 0x7f9b881037b0 2024-04-12T05:47:32.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.265+0000 7f9b857fa700 1 -- 172.21.15.32:0/306075317 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9b74012400 con 0x7f9b881037b0 2024-04-12T05:47:32.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.266+0000 7f9b857fa700 1 --2- 172.21.15.32:0/306075317 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9b7806cc30 0x7f9b7806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:47:32.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.266+0000 7f9b857fa700 1 -- 172.21.15.32:0/306075317 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f9b7408c280 con 0x7f9b881037b0 2024-04-12T05:47:32.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.266+0000 7f9b8c8ab700 1 --2- 172.21.15.32:0/306075317 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9b7806cc30 0x7f9b7806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:32.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.266+0000 7f9b8c8ab700 1 --2- 172.21.15.32:0/306075317 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9b7806cc30 0x7f9b7806f0e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c00b3b0 tx=0x7f9b7c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:32.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.270+0000 7f9b857fa700 1 -- 172.21.15.32:0/306075317 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9b74059c50 con 0x7f9b881037b0 2024-04-12T05:47:32.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.555+0000 7f9b8eb0f700 1 -- 172.21.15.32:0/306075317 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9b88109dd0 con 0x7f9b881037b0 2024-04-12T05:47:32.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.556+0000 7f9b857fa700 1 -- 172.21.15.32:0/306075317 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f9b740597e0 con 0x7f9b881037b0 2024-04-12T05:47:32.553 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:47:32.553 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:47:32.553 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:47:32.553 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:47:32.553 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:47:32.553 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:47:32.553 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:47:32.553 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:47:32.553 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:47:32.553 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:47:32.553 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:47:32.553 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:47:32.553 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:47:32.553 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:47:32.554 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:47:32.554 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:47:32.554 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:47:32.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.558+0000 7f9b8eb0f700 1 -- 172.21.15.32:0/306075317 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9b7806cc30 msgr2=0x7f9b7806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:32.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.558+0000 7f9b8eb0f700 1 --2- 172.21.15.32:0/306075317 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9b7806cc30 0x7f9b7806f0e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c00b3b0 tx=0x7f9b7c009e30 comp rx=0 tx=0).stop 2024-04-12T05:47:32.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.558+0000 7f9b8eb0f700 1 -- 172.21.15.32:0/306075317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b881037b0 msgr2=0x7f9b88114050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:32.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.558+0000 7f9b8eb0f700 1 --2- 172.21.15.32:0/306075317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b881037b0 0x7f9b88114050 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9b7400c890 tx=0x7f9b740044c0 comp rx=0 tx=0).stop 2024-04-12T05:47:32.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.558+0000 7f9b8eb0f700 1 -- 172.21.15.32:0/306075317 shutdown_connections 2024-04-12T05:47:32.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.558+0000 7f9b8eb0f700 1 --2- 172.21.15.32:0/306075317 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9b7806cc30 0x7f9b7806f0e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:32.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.558+0000 7f9b8eb0f700 1 --2- 172.21.15.32:0/306075317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b88102e10 0x7f9b88113b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:32.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.558+0000 7f9b8eb0f700 1 --2- 172.21.15.32:0/306075317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b881037b0 0x7f9b88114050 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:32.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.558+0000 7f9b8eb0f700 1 -- 172.21.15.32:0/306075317 >> 172.21.15.32:0/306075317 conn(0x7f9b880f8980 msgr2=0x7f9b880f9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:32.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.558+0000 7f9b8eb0f700 1 -- 172.21.15.32:0/306075317 shutdown_connections 2024-04-12T05:47:32.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.558+0000 7f9b8eb0f700 1 -- 172.21.15.32:0/306075317 wait complete. 2024-04-12T05:47:32.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.664+0000 7f8236728700 1 -- 172.21.15.32:0/3162207252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82300fb4a0 msgr2=0x7f82300fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:32.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.664+0000 7f8236728700 1 --2- 172.21.15.32:0/3162207252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82300fb4a0 0x7f82300fb910 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8220004940 tx=0x7f8220009e30 comp rx=0 tx=0).stop 2024-04-12T05:47:32.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.664+0000 7f8236728700 1 -- 172.21.15.32:0/3162207252 shutdown_connections 2024-04-12T05:47:32.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.664+0000 7f8236728700 1 --2- 172.21.15.32:0/3162207252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82300fb4a0 0x7f82300fb910 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:32.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.664+0000 7f8236728700 1 --2- 172.21.15.32:0/3162207252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8230102e70 0x7f8230103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:32.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.664+0000 7f8236728700 1 -- 172.21.15.32:0/3162207252 >> 172.21.15.32:0/3162207252 conn(0x7f82300714c0 msgr2=0x7f82300718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:32.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.664+0000 7f8236728700 1 -- 172.21.15.32:0/3162207252 shutdown_connections 2024-04-12T05:47:32.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.664+0000 7f8236728700 1 -- 172.21.15.32:0/3162207252 wait complete. 2024-04-12T05:47:32.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.664+0000 7f8236728700 1 Processor -- start 2024-04-12T05:47:32.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.665+0000 7f8236728700 1 -- start start 2024-04-12T05:47:32.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.665+0000 7f8236728700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82300fb4a0 0x7f8230115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:47:32.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.665+0000 7f8236728700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8230102e70 0x7f8230116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:47:32.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.665+0000 7f8236728700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8230116870 con 0x7f8230102e70 2024-04-12T05:47:32.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.665+0000 7f8236728700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f823010fc40 con 0x7f82300fb4a0 2024-04-12T05:47:32.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.665+0000 7f822f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8230102e70 0x7f8230116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:32.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.665+0000 7f822f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8230102e70 0x7f8230116220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36312/0 (socket says 172.21.15.32:36312) 2024-04-12T05:47:32.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.665+0000 7f822f7fe700 1 -- 172.21.15.32:0/2637356167 learned_addr learned my addr 172.21.15.32:0/2637356167 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:47:32.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.665+0000 7f822ffff700 1 --2- 172.21.15.32:0/2637356167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82300fb4a0 0x7f8230115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:32.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.665+0000 7f822f7fe700 1 -- 172.21.15.32:0/2637356167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82300fb4a0 msgr2=0x7f8230115ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:32.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.665+0000 7f822f7fe700 1 --2- 172.21.15.32:0/2637356167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82300fb4a0 0x7f8230115ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:32.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.665+0000 7f822f7fe700 1 -- 172.21.15.32:0/2637356167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8220005040 con 0x7f8230102e70 2024-04-12T05:47:32.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.665+0000 7f822f7fe700 1 --2- 172.21.15.32:0/2637356167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8230102e70 0x7f8230116220 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8220009180 tx=0x7f8220009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:32.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.666+0000 7f822cff9700 1 -- 172.21.15.32:0/2637356167 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f822000e040 con 0x7f8230102e70 2024-04-12T05:47:32.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.666+0000 7f822cff9700 1 -- 172.21.15.32:0/2637356167 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8220004290 con 0x7f8230102e70 2024-04-12T05:47:32.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.666+0000 7f8236728700 1 -- 172.21.15.32:0/2637356167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f823010fec0 con 0x7f8230102e70 2024-04-12T05:47:32.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.666+0000 7f822cff9700 1 -- 172.21.15.32:0/2637356167 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f822001c3f0 con 0x7f8230102e70 2024-04-12T05:47:32.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.666+0000 7f8236728700 1 -- 172.21.15.32:0/2637356167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8230110400 con 0x7f8230102e70 2024-04-12T05:47:32.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.666+0000 7f8236728700 1 -- 172.21.15.32:0/2637356167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8230107330 con 0x7f8230102e70 2024-04-12T05:47:32.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.667+0000 7f822cff9700 1 -- 172.21.15.32:0/2637356167 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8220013790 con 0x7f8230102e70 2024-04-12T05:47:32.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.667+0000 7f822cff9700 1 --2- 172.21.15.32:0/2637356167 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f821806cc20 0x7f821806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:47:32.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.668+0000 7f822cff9700 1 -- 172.21.15.32:0/2637356167 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f822008d280 con 0x7f8230102e70 2024-04-12T05:47:32.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.668+0000 7f822ffff700 1 --2- 172.21.15.32:0/2637356167 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f821806cc20 0x7f821806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:32.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.668+0000 7f822ffff700 1 --2- 172.21.15.32:0/2637356167 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f821806cc20 0x7f821806f0d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8224007fd0 tx=0x7f8224009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:32.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.672+0000 7f822cff9700 1 -- 172.21.15.32:0/2637356167 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f822005ad90 con 0x7f8230102e70 2024-04-12T05:47:32.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:32 smithi032 ceph-mon[146817]: from='client.34452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:32.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:32 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/306075317' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:47:32.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.923+0000 7f8236728700 1 -- 172.21.15.32:0/2637356167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f82300653a0 con 0x7f8230102e70 2024-04-12T05:47:32.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.923+0000 7f822cff9700 1 -- 172.21.15.32:0/2637356167 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1835 (secure 0 0 0) 0x7f8220013420 con 0x7f8230102e70 2024-04-12T05:47:32.920 INFO:teuthology.orchestra.run.smithi032.stdout:e33 2024-04-12T05:47:32.920 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:47:32.920 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:47:32.920 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:47:32.920 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:47:32.920 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:47:32.920 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:47:32.920 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 24 2024-04-12T05:47:32.920 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:47:32.920 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:47:32.920 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:46:10.668569+0000 2024-04-12T05:47:32.920 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:47:32.921 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:47:32.921 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:47:32.921 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:47:32.921 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:47:32.921 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:47:32.921 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:47:32.921 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:47:32.921 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T05:47:32.921 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:47:32.921 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:47:32.921 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:47:32.921 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T05:47:32.921 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-12T05:47:32.921 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:47:32.921 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:47:32.922 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:47:32.922 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:47:32.922 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:47:32.922 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:47:32.922 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:47:32.922 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:47:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:47:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:47:32.922 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-12T05:47:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:47:32.922 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:47:32.922 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:47:32.922 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{-1:44317} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:47:32.922 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:47:32.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.925+0000 7f8236728700 1 -- 172.21.15.32:0/2637356167 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f821806cc20 msgr2=0x7f821806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:32.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.925+0000 7f8236728700 1 --2- 172.21.15.32:0/2637356167 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f821806cc20 0x7f821806f0d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8224007fd0 tx=0x7f8224009040 comp rx=0 tx=0).stop 2024-04-12T05:47:32.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.925+0000 7f8236728700 1 -- 172.21.15.32:0/2637356167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8230102e70 msgr2=0x7f8230116220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:32.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.925+0000 7f8236728700 1 --2- 172.21.15.32:0/2637356167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8230102e70 0x7f8230116220 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8220009180 tx=0x7f8220009e30 comp rx=0 tx=0).stop 2024-04-12T05:47:32.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.925+0000 7f8236728700 1 -- 172.21.15.32:0/2637356167 shutdown_connections 2024-04-12T05:47:32.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.925+0000 7f8236728700 1 --2- 172.21.15.32:0/2637356167 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f821806cc20 0x7f821806f0d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:32.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.925+0000 7f8236728700 1 --2- 172.21.15.32:0/2637356167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82300fb4a0 0x7f8230115ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:32.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.925+0000 7f8236728700 1 --2- 172.21.15.32:0/2637356167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8230102e70 0x7f8230116220 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:32.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.925+0000 7f8236728700 1 -- 172.21.15.32:0/2637356167 >> 172.21.15.32:0/2637356167 conn(0x7f82300714c0 msgr2=0x7f82300fa660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:32.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.925+0000 7f8236728700 1 -- 172.21.15.32:0/2637356167 shutdown_connections 2024-04-12T05:47:32.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:32.925+0000 7f8236728700 1 -- 172.21.15.32:0/2637356167 wait complete. 2024-04-12T05:47:32.924 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-12T05:47:33.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:32 smithi067 ceph-mon[112979]: from='client.34452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:33.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:32 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/306075317' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:47:33.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.030+0000 7f833fc15700 1 -- 172.21.15.32:0/2061617744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8338101d20 msgr2=0x7f83381020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:33.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.030+0000 7f833fc15700 1 --2- 172.21.15.32:0/2061617744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8338101d20 0x7f83381020f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8330004940 tx=0x7f8330009e30 comp rx=0 tx=0).stop 2024-04-12T05:47:33.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.030+0000 7f833fc15700 1 -- 172.21.15.32:0/2061617744 shutdown_connections 2024-04-12T05:47:33.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.030+0000 7f833fc15700 1 --2- 172.21.15.32:0/2061617744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83381026c0 0x7f83380fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:33.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.030+0000 7f833fc15700 1 --2- 172.21.15.32:0/2061617744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8338101d20 0x7f83381020f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:33.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.030+0000 7f833fc15700 1 -- 172.21.15.32:0/2061617744 >> 172.21.15.32:0/2061617744 conn(0x7f83380f78d0 msgr2=0x7f83380f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:33.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.030+0000 7f833fc15700 1 -- 172.21.15.32:0/2061617744 shutdown_connections 2024-04-12T05:47:33.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.030+0000 7f833fc15700 1 -- 172.21.15.32:0/2061617744 wait complete. 2024-04-12T05:47:33.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.030+0000 7f833fc15700 1 Processor -- start 2024-04-12T05:47:33.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.031+0000 7f833fc15700 1 -- start start 2024-04-12T05:47:33.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.031+0000 7f833fc15700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8338101d20 0x7f833810ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:47:33.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.031+0000 7f833fc15700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83381026c0 0x7f833810efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:47:33.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.031+0000 7f833fc15700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833810f620 con 0x7f8338101d20 2024-04-12T05:47:33.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.031+0000 7f833fc15700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833810f760 con 0x7f83381026c0 2024-04-12T05:47:33.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.031+0000 7f833d9b1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8338101d20 0x7f833810ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:33.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.031+0000 7f833d9b1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8338101d20 0x7f833810ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36324/0 (socket says 172.21.15.32:36324) 2024-04-12T05:47:33.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.031+0000 7f833d9b1700 1 -- 172.21.15.32:0/2880673325 learned_addr learned my addr 172.21.15.32:0/2880673325 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:47:33.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.031+0000 7f833d1b0700 1 --2- 172.21.15.32:0/2880673325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83381026c0 0x7f833810efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:33.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.031+0000 7f833d9b1700 1 -- 172.21.15.32:0/2880673325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83381026c0 msgr2=0x7f833810efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:33.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.031+0000 7f833d9b1700 1 --2- 172.21.15.32:0/2880673325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83381026c0 0x7f833810efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:33.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.031+0000 7f833d9b1700 1 -- 172.21.15.32:0/2880673325 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8330005040 con 0x7f8338101d20 2024-04-12T05:47:33.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.031+0000 7f833d1b0700 1 --2- 172.21.15.32:0/2880673325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83381026c0 0x7f833810efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:47:33.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.031+0000 7f833d9b1700 1 --2- 172.21.15.32:0/2880673325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8338101d20 0x7f833810ea90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8330009360 tx=0x7f8330009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:33.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.031+0000 7f83367fc700 1 -- 172.21.15.32:0/2880673325 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f833000e040 con 0x7f8338101d20 2024-04-12T05:47:33.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.032+0000 7f83367fc700 1 -- 172.21.15.32:0/2880673325 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8330013420 con 0x7f8338101d20 2024-04-12T05:47:33.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.032+0000 7f833fc15700 1 -- 172.21.15.32:0/2880673325 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8338117b00 con 0x7f8338101d20 2024-04-12T05:47:33.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.032+0000 7f83367fc700 1 -- 172.21.15.32:0/2880673325 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f833001c5f0 con 0x7f8338101d20 2024-04-12T05:47:33.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.032+0000 7f833fc15700 1 -- 172.21.15.32:0/2880673325 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8338118010 con 0x7f8338101d20 2024-04-12T05:47:33.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.032+0000 7f833fc15700 1 -- 172.21.15.32:0/2880673325 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83380fd470 con 0x7f8338101d20 2024-04-12T05:47:33.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.033+0000 7f83367fc700 1 -- 172.21.15.32:0/2880673325 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8330013590 con 0x7f8338101d20 2024-04-12T05:47:33.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.033+0000 7f83367fc700 1 --2- 172.21.15.32:0/2880673325 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f832006cc30 0x7f832006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:47:33.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.033+0000 7f83367fc700 1 -- 172.21.15.32:0/2880673325 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f833008d210 con 0x7f8338101d20 2024-04-12T05:47:33.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.033+0000 7f833d1b0700 1 --2- 172.21.15.32:0/2880673325 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f832006cc30 0x7f832006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:33.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.034+0000 7f833d1b0700 1 --2- 172.21.15.32:0/2880673325 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f832006cc30 0x7f832006f0e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8338066890 tx=0x7f832c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:33.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.038+0000 7f83367fc700 1 -- 172.21.15.32:0/2880673325 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8330057290 con 0x7f8338101d20 2024-04-12T05:47:33.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.261+0000 7f833fc15700 1 -- 172.21.15.32:0/2880673325 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8338066640 con 0x7f832006cc30 2024-04-12T05:47:33.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.262+0000 7f83367fc700 1 -- 172.21.15.32:0/2880673325 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8338066640 con 0x7f832006cc30 2024-04-12T05:47:33.262 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:47:33.262 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:47:33.262 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:47:33.262 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:47:33.262 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:47:33.263 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-12T05:47:33.263 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:47:33.263 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-12T05:47:33.263 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:47:33.263 INFO:teuthology.orchestra.run.smithi032.stdout: "mds" 2024-04-12T05:47:33.263 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:47:33.263 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:47:33.263 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:47:33.263 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:47:33.263 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:47:33.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.264+0000 7f833fc15700 1 -- 172.21.15.32:0/2880673325 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f832006cc30 msgr2=0x7f832006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:33.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.264+0000 7f833fc15700 1 --2- 172.21.15.32:0/2880673325 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f832006cc30 0x7f832006f0e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8338066890 tx=0x7f832c009040 comp rx=0 tx=0).stop 2024-04-12T05:47:33.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.264+0000 7f833fc15700 1 -- 172.21.15.32:0/2880673325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8338101d20 msgr2=0x7f833810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:33.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.264+0000 7f833fc15700 1 --2- 172.21.15.32:0/2880673325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8338101d20 0x7f833810ea90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8330009360 tx=0x7f8330009e30 comp rx=0 tx=0).stop 2024-04-12T05:47:33.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.264+0000 7f833fc15700 1 -- 172.21.15.32:0/2880673325 shutdown_connections 2024-04-12T05:47:33.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.264+0000 7f833fc15700 1 --2- 172.21.15.32:0/2880673325 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f832006cc30 0x7f832006f0e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:33.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.264+0000 7f833fc15700 1 --2- 172.21.15.32:0/2880673325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8338101d20 0x7f833810ea90 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:33.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.264+0000 7f833fc15700 1 --2- 172.21.15.32:0/2880673325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83381026c0 0x7f833810efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:33.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.264+0000 7f833fc15700 1 -- 172.21.15.32:0/2880673325 >> 172.21.15.32:0/2880673325 conn(0x7f83380f78d0 msgr2=0x7f83380f89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:33.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.264+0000 7f833fc15700 1 -- 172.21.15.32:0/2880673325 shutdown_connections 2024-04-12T05:47:33.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.265+0000 7f833fc15700 1 -- 172.21.15.32:0/2880673325 wait complete. 2024-04-12T05:47:33.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.368+0000 7fc695301700 1 -- 172.21.15.32:0/1969786343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6900fe830 msgr2=0x7fc690106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:33.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.368+0000 7fc695301700 1 --2- 172.21.15.32:0/1969786343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6900fe830 0x7fc690106f00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc680004940 tx=0x7fc680009e30 comp rx=0 tx=0).stop 2024-04-12T05:47:33.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.368+0000 7fc695301700 1 -- 172.21.15.32:0/1969786343 shutdown_connections 2024-04-12T05:47:33.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.368+0000 7fc695301700 1 --2- 172.21.15.32:0/1969786343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6900fe830 0x7fc690106f00 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:33.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.368+0000 7fc695301700 1 --2- 172.21.15.32:0/1969786343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6900fde90 0x7fc6900fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:33.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.368+0000 7fc695301700 1 -- 172.21.15.32:0/1969786343 >> 172.21.15.32:0/1969786343 conn(0x7fc690072bc0 msgr2=0x7fc690072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:33.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.368+0000 7fc695301700 1 -- 172.21.15.32:0/1969786343 shutdown_connections 2024-04-12T05:47:33.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.368+0000 7fc695301700 1 -- 172.21.15.32:0/1969786343 wait complete. 2024-04-12T05:47:33.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.369+0000 7fc695301700 1 Processor -- start 2024-04-12T05:47:33.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.369+0000 7fc695301700 1 -- start start 2024-04-12T05:47:33.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.369+0000 7fc695301700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6900fde90 0x7fc6900fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:47:33.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.369+0000 7fc695301700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6900fe830 0x7fc6900fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:47:33.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.369+0000 7fc695301700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6900fc820 con 0x7fc6900fe830 2024-04-12T05:47:33.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.369+0000 7fc695301700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6900fc960 con 0x7fc6900fde90 2024-04-12T05:47:33.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.369+0000 7fc68e7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6900fe830 0x7fc6900fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:33.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.369+0000 7fc68e7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6900fe830 0x7fc6900fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36342/0 (socket says 172.21.15.32:36342) 2024-04-12T05:47:33.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.369+0000 7fc68e7fc700 1 -- 172.21.15.32:0/1770083382 learned_addr learned my addr 172.21.15.32:0/1770083382 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:47:33.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.369+0000 7fc68effd700 1 --2- 172.21.15.32:0/1770083382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6900fde90 0x7fc6900fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:33.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.369+0000 7fc68e7fc700 1 -- 172.21.15.32:0/1770083382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6900fde90 msgr2=0x7fc6900fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:33.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.369+0000 7fc68e7fc700 1 --2- 172.21.15.32:0/1770083382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6900fde90 0x7fc6900fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:33.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.369+0000 7fc68e7fc700 1 -- 172.21.15.32:0/1770083382 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc680005040 con 0x7fc6900fe830 2024-04-12T05:47:33.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.369+0000 7fc68e7fc700 1 --2- 172.21.15.32:0/1770083382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6900fe830 0x7fc6900fc290 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc680000c00 tx=0x7fc680009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:33.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.370+0000 7fc6777fe700 1 -- 172.21.15.32:0/1770083382 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc68000e040 con 0x7fc6900fe830 2024-04-12T05:47:33.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.370+0000 7fc6777fe700 1 -- 172.21.15.32:0/1770083382 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc680013420 con 0x7fc6900fe830 2024-04-12T05:47:33.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.370+0000 7fc695301700 1 -- 172.21.15.32:0/1770083382 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6900fcbb0 con 0x7fc6900fe830 2024-04-12T05:47:33.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.370+0000 7fc6777fe700 1 -- 172.21.15.32:0/1770083382 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc68001c630 con 0x7fc6900fe830 2024-04-12T05:47:33.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.370+0000 7fc695301700 1 -- 172.21.15.32:0/1770083382 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc69011a190 con 0x7fc6900fe830 2024-04-12T05:47:33.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.371+0000 7fc695301700 1 -- 172.21.15.32:0/1770083382 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc690065e30 con 0x7fc6900fe830 2024-04-12T05:47:33.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.371+0000 7fc6777fe700 1 -- 172.21.15.32:0/1770083382 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc680013970 con 0x7fc6900fe830 2024-04-12T05:47:33.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.371+0000 7fc6777fe700 1 --2- 172.21.15.32:0/1770083382 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc67806cc30 0x7fc67806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:47:33.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.372+0000 7fc6777fe700 1 -- 172.21.15.32:0/1770083382 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fc68008d2d0 con 0x7fc6900fe830 2024-04-12T05:47:33.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.372+0000 7fc68effd700 1 --2- 172.21.15.32:0/1770083382 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc67806cc30 0x7fc67806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:33.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.372+0000 7fc68effd700 1 --2- 172.21.15.32:0/1770083382 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc67806cc30 0x7fc67806f0e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc684004d60 tx=0x7fc68400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:33.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.376+0000 7fc6777fe700 1 -- 172.21.15.32:0/1770083382 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc6800572b0 con 0x7fc6900fe830 2024-04-12T05:47:33.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.662+0000 7fc695301700 1 -- 172.21.15.32:0/1770083382 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc690109e60 con 0x7fc6900fe830 2024-04-12T05:47:33.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.663+0000 7fc6777fe700 1 -- 172.21.15.32:0/1770083382 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fc68005a8d0 con 0x7fc6900fe830 2024-04-12T05:47:33.659 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:47:33.659 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:47:33.660 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-12T05:47:33.660 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:47:33.660 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:47:33.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.665+0000 7fc695301700 1 -- 172.21.15.32:0/1770083382 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc67806cc30 msgr2=0x7fc67806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:33.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.665+0000 7fc695301700 1 --2- 172.21.15.32:0/1770083382 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc67806cc30 0x7fc67806f0e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc684004d60 tx=0x7fc68400b040 comp rx=0 tx=0).stop 2024-04-12T05:47:33.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.665+0000 7fc695301700 1 -- 172.21.15.32:0/1770083382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6900fe830 msgr2=0x7fc6900fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:33.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.665+0000 7fc695301700 1 --2- 172.21.15.32:0/1770083382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6900fe830 0x7fc6900fc290 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc680000c00 tx=0x7fc680009e30 comp rx=0 tx=0).stop 2024-04-12T05:47:33.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.665+0000 7fc695301700 1 -- 172.21.15.32:0/1770083382 shutdown_connections 2024-04-12T05:47:33.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.665+0000 7fc695301700 1 --2- 172.21.15.32:0/1770083382 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc67806cc30 0x7fc67806f0e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:33.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.665+0000 7fc695301700 1 --2- 172.21.15.32:0/1770083382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6900fde90 0x7fc6900fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:33.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.665+0000 7fc695301700 1 --2- 172.21.15.32:0/1770083382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6900fe830 0x7fc6900fc290 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:47:33.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.665+0000 7fc695301700 1 -- 172.21.15.32:0/1770083382 >> 172.21.15.32:0/1770083382 conn(0x7fc690072bc0 msgr2=0x7fc690103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:33.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.665+0000 7fc695301700 1 -- 172.21.15.32:0/1770083382 shutdown_connections 2024-04-12T05:47:33.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:47:33.665+0000 7fc695301700 1 -- 172.21.15.32:0/1770083382 wait complete. 2024-04-12T05:47:33.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:33 smithi032 ceph-mon[146817]: from='client.34456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:33.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:33 smithi032 ceph-mon[146817]: pgmap v211: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:33.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:33 smithi032 ceph-mon[146817]: from='client.34460 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:33.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:33 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:47:33.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:33 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/2637356167' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:47:34.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:33 smithi067 ceph-mon[112979]: from='client.34456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:34.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:33 smithi067 ceph-mon[112979]: pgmap v211: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:34.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:33 smithi067 ceph-mon[112979]: from='client.34460 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:34.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:33 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:47:34.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:33 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/2637356167' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:47:34.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:34 smithi032 ceph-mon[146817]: from='client.34472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:34.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:34 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1770083382' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:47:35.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:34 smithi067 ceph-mon[112979]: from='client.34472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:35.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:34 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1770083382' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:47:35.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:35 smithi032 ceph-mon[146817]: pgmap v212: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:36.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:35 smithi067 ceph-mon[112979]: pgmap v212: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:38.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:37 smithi067 ceph-mon[112979]: pgmap v213: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:38.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:37 smithi032 ceph-mon[146817]: pgmap v213: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:40.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:39 smithi067 ceph-mon[112979]: pgmap v214: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:40.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:39 smithi032 ceph-mon[146817]: pgmap v214: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:42.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:41 smithi067 ceph-mon[112979]: pgmap v215: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:42.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:41 smithi032 ceph-mon[146817]: pgmap v215: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:43.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:43 smithi032 ceph-mon[146817]: pgmap v216: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:44.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:43 smithi067 ceph-mon[112979]: pgmap v216: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:46.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:45 smithi067 ceph-mon[112979]: pgmap v217: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:46.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:45 smithi032 ceph-mon[146817]: pgmap v217: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:48.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:47 smithi067 ceph-mon[112979]: pgmap v218: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:48.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:47 smithi032 ceph-mon[146817]: pgmap v218: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:49.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:48 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:47:49.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:48 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:47:50.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:49 smithi067 ceph-mon[112979]: pgmap v219: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:50.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:49 smithi032 ceph-mon[146817]: pgmap v219: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:52.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:51 smithi067 ceph-mon[112979]: pgmap v220: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:52.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:51 smithi032 ceph-mon[146817]: pgmap v220: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:53.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:53 smithi032 ceph-mon[146817]: pgmap v221: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:54.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:53 smithi067 ceph-mon[112979]: pgmap v221: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:56.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:55 smithi067 ceph-mon[112979]: pgmap v222: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:56.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:55 smithi032 ceph-mon[146817]: pgmap v222: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:58.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:57 smithi067 ceph-mon[112979]: pgmap v223: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:58.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:57 smithi032 ceph-mon[146817]: pgmap v223: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:00.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:47:59 smithi067 ceph-mon[112979]: pgmap v224: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:00.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:47:59 smithi032 ceph-mon[146817]: pgmap v224: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:02.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:01 smithi067 ceph-mon[112979]: pgmap v225: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:02.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:01 smithi032 ceph-mon[146817]: pgmap v225: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:03.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.785+0000 7f92cd33f700 1 -- 172.21.15.32:0/735375666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92c80fe820 msgr2=0x7f92c8106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:03.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.785+0000 7f92cd33f700 1 --2- 172.21.15.32:0/735375666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92c80fe820 0x7f92c8106e60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f92b0004940 tx=0x7f92b0009e30 comp rx=0 tx=0).stop 2024-04-12T05:48:03.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.785+0000 7f92cd33f700 1 -- 172.21.15.32:0/735375666 shutdown_connections 2024-04-12T05:48:03.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.785+0000 7f92cd33f700 1 --2- 172.21.15.32:0/735375666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92c80fe820 0x7f92c8106e60 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:03.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.785+0000 7f92cd33f700 1 --2- 172.21.15.32:0/735375666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92c80fde80 0x7f92c80fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:03.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.785+0000 7f92cd33f700 1 -- 172.21.15.32:0/735375666 >> 172.21.15.32:0/735375666 conn(0x7f92c80f7870 msgr2=0x7f92c80f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:03.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.785+0000 7f92cd33f700 1 -- 172.21.15.32:0/735375666 shutdown_connections 2024-04-12T05:48:03.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.785+0000 7f92cd33f700 1 -- 172.21.15.32:0/735375666 wait complete. 2024-04-12T05:48:03.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.786+0000 7f92cd33f700 1 Processor -- start 2024-04-12T05:48:03.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.786+0000 7f92cd33f700 1 -- start start 2024-04-12T05:48:03.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.787+0000 7f92cd33f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92c80fde80 0x7f92c8115970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:03.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.787+0000 7f92cd33f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92c80fe820 0x7f92c8115eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:03.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.787+0000 7f92cd33f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c8116500 con 0x7f92c80fe820 2024-04-12T05:48:03.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.787+0000 7f92cd33f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c810fb20 con 0x7f92c80fde80 2024-04-12T05:48:03.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.787+0000 7f92c67fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92c80fe820 0x7f92c8115eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:03.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.787+0000 7f92c67fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92c80fe820 0x7f92c8115eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49280/0 (socket says 172.21.15.32:49280) 2024-04-12T05:48:03.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.787+0000 7f92c67fc700 1 -- 172.21.15.32:0/2666893903 learned_addr learned my addr 172.21.15.32:0/2666893903 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:48:03.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.787+0000 7f92c6ffd700 1 --2- 172.21.15.32:0/2666893903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92c80fde80 0x7f92c8115970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:03.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.787+0000 7f92c67fc700 1 -- 172.21.15.32:0/2666893903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92c80fde80 msgr2=0x7f92c8115970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:03.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.787+0000 7f92c67fc700 1 --2- 172.21.15.32:0/2666893903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92c80fde80 0x7f92c8115970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:03.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.787+0000 7f92c67fc700 1 -- 172.21.15.32:0/2666893903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b0005040 con 0x7f92c80fe820 2024-04-12T05:48:03.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.787+0000 7f92c67fc700 1 --2- 172.21.15.32:0/2666893903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92c80fe820 0x7f92c8115eb0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f92b0000c00 tx=0x7f92b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:03.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.788+0000 7f92bf7fe700 1 -- 172.21.15.32:0/2666893903 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92b0021070 con 0x7f92c80fe820 2024-04-12T05:48:03.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.788+0000 7f92bf7fe700 1 -- 172.21.15.32:0/2666893903 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f92b0013e00 con 0x7f92c80fe820 2024-04-12T05:48:03.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.788+0000 7f92cd33f700 1 -- 172.21.15.32:0/2666893903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c810fda0 con 0x7f92c80fe820 2024-04-12T05:48:03.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.788+0000 7f92bf7fe700 1 -- 172.21.15.32:0/2666893903 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92b000e070 con 0x7f92c80fe820 2024-04-12T05:48:03.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.788+0000 7f92cd33f700 1 -- 172.21.15.32:0/2666893903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c81102e0 con 0x7f92c80fe820 2024-04-12T05:48:03.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.789+0000 7f92cd33f700 1 -- 172.21.15.32:0/2666893903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c8066b10 con 0x7f92c80fe820 2024-04-12T05:48:03.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.789+0000 7f92bf7fe700 1 -- 172.21.15.32:0/2666893903 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f92b0013950 con 0x7f92c80fe820 2024-04-12T05:48:03.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.790+0000 7f92bf7fe700 1 --2- 172.21.15.32:0/2666893903 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f92b406cc20 0x7f92b406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:03.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.790+0000 7f92bf7fe700 1 -- 172.21.15.32:0/2666893903 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f92b008d150 con 0x7f92c80fe820 2024-04-12T05:48:03.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.790+0000 7f92c6ffd700 1 --2- 172.21.15.32:0/2666893903 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f92b406cc20 0x7f92b406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:03.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.790+0000 7f92c6ffd700 1 --2- 172.21.15.32:0/2666893903 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f92b406cc20 0x7f92b406f0d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f92b8009b80 tx=0x7f92b800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:03.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:03.795+0000 7f92bf7fe700 1 -- 172.21.15.32:0/2666893903 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f92b0057130 con 0x7f92c80fe820 2024-04-12T05:48:03.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:03 smithi032 ceph-mon[146817]: pgmap v226: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:03.838 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:48:04.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.019+0000 7f92cd33f700 1 -- 172.21.15.32:0/2666893903 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f92c8109d00 con 0x7f92b406cc20 2024-04-12T05:48:04.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.020+0000 7f92bf7fe700 1 -- 172.21.15.32:0/2666893903 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f92c8109d00 con 0x7f92b406cc20 2024-04-12T05:48:04.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:03 smithi067 ceph-mon[112979]: pgmap v226: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:04.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:48:04.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.022+0000 7f92cd33f700 1 -- 172.21.15.32:0/2666893903 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f92b406cc20 msgr2=0x7f92b406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:04.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.022+0000 7f92cd33f700 1 --2- 172.21.15.32:0/2666893903 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f92b406cc20 0x7f92b406f0d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f92b8009b80 tx=0x7f92b800b040 comp rx=0 tx=0).stop 2024-04-12T05:48:04.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.022+0000 7f92cd33f700 1 -- 172.21.15.32:0/2666893903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92c80fe820 msgr2=0x7f92c8115eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:04.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.022+0000 7f92cd33f700 1 --2- 172.21.15.32:0/2666893903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92c80fe820 0x7f92c8115eb0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f92b0000c00 tx=0x7f92b0009e30 comp rx=0 tx=0).stop 2024-04-12T05:48:04.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.022+0000 7f92cd33f700 1 -- 172.21.15.32:0/2666893903 shutdown_connections 2024-04-12T05:48:04.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.022+0000 7f92cd33f700 1 --2- 172.21.15.32:0/2666893903 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f92b406cc20 0x7f92b406f0d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:04.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.022+0000 7f92cd33f700 1 --2- 172.21.15.32:0/2666893903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92c80fde80 0x7f92c8115970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:04.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.022+0000 7f92cd33f700 1 --2- 172.21.15.32:0/2666893903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92c80fe820 0x7f92c8115eb0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:04.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.022+0000 7f92cd33f700 1 -- 172.21.15.32:0/2666893903 >> 172.21.15.32:0/2666893903 conn(0x7f92c80f7870 msgr2=0x7f92c80fbee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:04.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.022+0000 7f92cd33f700 1 -- 172.21.15.32:0/2666893903 shutdown_connections 2024-04-12T05:48:04.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.022+0000 7f92cd33f700 1 -- 172.21.15.32:0/2666893903 wait complete. 2024-04-12T05:48:04.028 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:48:04.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.126+0000 7fc4188be700 1 -- 172.21.15.32:0/226491840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc414102710 msgr2=0x7fc4140fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:04.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.126+0000 7fc4188be700 1 --2- 172.21.15.32:0/226491840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc414102710 0x7fc4140fbd80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc408004940 tx=0x7fc408009e30 comp rx=0 tx=0).stop 2024-04-12T05:48:04.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.126+0000 7fc4188be700 1 -- 172.21.15.32:0/226491840 shutdown_connections 2024-04-12T05:48:04.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.126+0000 7fc4188be700 1 --2- 172.21.15.32:0/226491840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc414102710 0x7fc4140fbd80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:04.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.126+0000 7fc4188be700 1 --2- 172.21.15.32:0/226491840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc414101d70 0x7fc414102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:04.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.126+0000 7fc4188be700 1 -- 172.21.15.32:0/226491840 >> 172.21.15.32:0/226491840 conn(0x7fc414072bc0 msgr2=0x7fc414072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:04.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.126+0000 7fc4188be700 1 -- 172.21.15.32:0/226491840 shutdown_connections 2024-04-12T05:48:04.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.126+0000 7fc4188be700 1 -- 172.21.15.32:0/226491840 wait complete. 2024-04-12T05:48:04.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.127+0000 7fc4188be700 1 Processor -- start 2024-04-12T05:48:04.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.127+0000 7fc4188be700 1 -- start start 2024-04-12T05:48:04.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.127+0000 7fc4188be700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc414101d70 0x7fc414102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:04.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.127+0000 7fc4188be700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc414102710 0x7fc414103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:04.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.127+0000 7fc4188be700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4141036a0 con 0x7fc414101d70 2024-04-12T05:48:04.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.127+0000 7fc4188be700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4141037e0 con 0x7fc414102710 2024-04-12T05:48:04.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.127+0000 7fc41259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc414101d70 0x7fc414102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:04.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.127+0000 7fc41259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc414101d70 0x7fc414102bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49296/0 (socket says 172.21.15.32:49296) 2024-04-12T05:48:04.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.127+0000 7fc41259c700 1 -- 172.21.15.32:0/390182095 learned_addr learned my addr 172.21.15.32:0/390182095 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:48:04.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.127+0000 7fc411d9b700 1 --2- 172.21.15.32:0/390182095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc414102710 0x7fc414103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:04.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.128+0000 7fc41259c700 1 -- 172.21.15.32:0/390182095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc414102710 msgr2=0x7fc414103110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:04.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.128+0000 7fc41259c700 1 --2- 172.21.15.32:0/390182095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc414102710 0x7fc414103110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:04.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.128+0000 7fc41259c700 1 -- 172.21.15.32:0/390182095 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc404005040 con 0x7fc414101d70 2024-04-12T05:48:04.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.128+0000 7fc41259c700 1 --2- 172.21.15.32:0/390182095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc414101d70 0x7fc414102bd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc4040096d0 tx=0x7fc404009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:04.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.128+0000 7fc402ffd700 1 -- 172.21.15.32:0/390182095 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc40400e840 con 0x7fc414101d70 2024-04-12T05:48:04.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.128+0000 7fc402ffd700 1 -- 172.21.15.32:0/390182095 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc40400f040 con 0x7fc414101d70 2024-04-12T05:48:04.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.128+0000 7fc4188be700 1 -- 172.21.15.32:0/390182095 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc408005040 con 0x7fc414101d70 2024-04-12T05:48:04.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.128+0000 7fc402ffd700 1 -- 172.21.15.32:0/390182095 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc404013d50 con 0x7fc414101d70 2024-04-12T05:48:04.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.128+0000 7fc4188be700 1 -- 172.21.15.32:0/390182095 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc414119fd0 con 0x7fc414101d70 2024-04-12T05:48:04.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.129+0000 7fc4188be700 1 -- 172.21.15.32:0/390182095 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4140fd4c0 con 0x7fc414101d70 2024-04-12T05:48:04.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.129+0000 7fc402ffd700 1 -- 172.21.15.32:0/390182095 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc40400e9a0 con 0x7fc414101d70 2024-04-12T05:48:04.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.130+0000 7fc402ffd700 1 --2- 172.21.15.32:0/390182095 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc3fc06cc20 0x7fc3fc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:04.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.130+0000 7fc411d9b700 1 --2- 172.21.15.32:0/390182095 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc3fc06cc20 0x7fc3fc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:04.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.130+0000 7fc402ffd700 1 -- 172.21.15.32:0/390182095 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fc40408be90 con 0x7fc414101d70 2024-04-12T05:48:04.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.130+0000 7fc411d9b700 1 --2- 172.21.15.32:0/390182095 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc3fc06cc20 0x7fc3fc06f0d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc40800b0d0 tx=0x7fc408009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:04.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.135+0000 7fc402ffd700 1 -- 172.21.15.32:0/390182095 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc404055e70 con 0x7fc414101d70 2024-04-12T05:48:04.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.367+0000 7fc4188be700 1 -- 172.21.15.32:0/390182095 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc41404bf70 con 0x7fc3fc06cc20 2024-04-12T05:48:04.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.369+0000 7fc402ffd700 1 -- 172.21.15.32:0/390182095 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc408013580 con 0x7fc3fc06cc20 2024-04-12T05:48:04.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.371+0000 7fc4188be700 1 -- 172.21.15.32:0/390182095 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc3fc06cc20 msgr2=0x7fc3fc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:04.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.371+0000 7fc4188be700 1 --2- 172.21.15.32:0/390182095 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc3fc06cc20 0x7fc3fc06f0d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc40800b0d0 tx=0x7fc408009e30 comp rx=0 tx=0).stop 2024-04-12T05:48:04.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.371+0000 7fc4188be700 1 -- 172.21.15.32:0/390182095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc414101d70 msgr2=0x7fc414102bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:04.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.371+0000 7fc4188be700 1 --2- 172.21.15.32:0/390182095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc414101d70 0x7fc414102bd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc4040096d0 tx=0x7fc404009e30 comp rx=0 tx=0).stop 2024-04-12T05:48:04.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.371+0000 7fc4188be700 1 -- 172.21.15.32:0/390182095 shutdown_connections 2024-04-12T05:48:04.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.371+0000 7fc4188be700 1 --2- 172.21.15.32:0/390182095 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc3fc06cc20 0x7fc3fc06f0d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:04.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.371+0000 7fc4188be700 1 --2- 172.21.15.32:0/390182095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc414101d70 0x7fc414102bd0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:04.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.371+0000 7fc4188be700 1 --2- 172.21.15.32:0/390182095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc414102710 0x7fc414103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:04.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.371+0000 7fc4188be700 1 -- 172.21.15.32:0/390182095 >> 172.21.15.32:0/390182095 conn(0x7fc414072bc0 msgr2=0x7fc4140658e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:04.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.371+0000 7fc4188be700 1 -- 172.21.15.32:0/390182095 shutdown_connections 2024-04-12T05:48:04.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.371+0000 7fc4188be700 1 -- 172.21.15.32:0/390182095 wait complete. 2024-04-12T05:48:04.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.475+0000 7f62286c0700 1 -- 172.21.15.32:0/4225659511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6220102e10 msgr2=0x7f62201031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:04.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.475+0000 7f62286c0700 1 --2- 172.21.15.32:0/4225659511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6220102e10 0x7f62201031e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f621c004940 tx=0x7f621c009e30 comp rx=0 tx=0).stop 2024-04-12T05:48:04.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.475+0000 7f62286c0700 1 -- 172.21.15.32:0/4225659511 shutdown_connections 2024-04-12T05:48:04.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.475+0000 7f62286c0700 1 --2- 172.21.15.32:0/4225659511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62201037b0 0x7f62200fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:04.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.475+0000 7f62286c0700 1 --2- 172.21.15.32:0/4225659511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6220102e10 0x7f62201031e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:04.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.475+0000 7f62286c0700 1 -- 172.21.15.32:0/4225659511 >> 172.21.15.32:0/4225659511 conn(0x7f62200f89a0 msgr2=0x7f62200fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:04.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.475+0000 7f62286c0700 1 -- 172.21.15.32:0/4225659511 shutdown_connections 2024-04-12T05:48:04.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.475+0000 7f62286c0700 1 -- 172.21.15.32:0/4225659511 wait complete. 2024-04-12T05:48:04.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.475+0000 7f62286c0700 1 Processor -- start 2024-04-12T05:48:04.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.475+0000 7f62286c0700 1 -- start start 2024-04-12T05:48:04.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.476+0000 7f62286c0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6220102e10 0x7f6220113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:04.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.476+0000 7f62286c0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62201037b0 0x7f6220114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:04.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.476+0000 7f62286c0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6220114680 con 0x7f6220102e10 2024-04-12T05:48:04.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.476+0000 7f62286c0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f622010da70 con 0x7f62201037b0 2024-04-12T05:48:04.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.476+0000 7f622645c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6220102e10 0x7f6220113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:04.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.476+0000 7f622645c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6220102e10 0x7f6220113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49314/0 (socket says 172.21.15.32:49314) 2024-04-12T05:48:04.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.476+0000 7f622645c700 1 -- 172.21.15.32:0/2209429080 learned_addr learned my addr 172.21.15.32:0/2209429080 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:48:04.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.476+0000 7f622645c700 1 -- 172.21.15.32:0/2209429080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62201037b0 msgr2=0x7f6220114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:04.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.476+0000 7f6225c5b700 1 --2- 172.21.15.32:0/2209429080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62201037b0 0x7f6220114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:04.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.476+0000 7f622645c700 1 --2- 172.21.15.32:0/2209429080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62201037b0 0x7f6220114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:04.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.476+0000 7f622645c700 1 -- 172.21.15.32:0/2209429080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f621c005040 con 0x7f6220102e10 2024-04-12T05:48:04.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.476+0000 7f622645c700 1 --2- 172.21.15.32:0/2209429080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6220102e10 0x7f6220113af0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f621c009360 tx=0x7f621c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:04.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.476+0000 7f6216ffd700 1 -- 172.21.15.32:0/2209429080 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f621c00e040 con 0x7f6220102e10 2024-04-12T05:48:04.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.476+0000 7f6216ffd700 1 -- 172.21.15.32:0/2209429080 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f621c013420 con 0x7f6220102e10 2024-04-12T05:48:04.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.476+0000 7f62286c0700 1 -- 172.21.15.32:0/2209429080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f622010dcf0 con 0x7f6220102e10 2024-04-12T05:48:04.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.476+0000 7f6216ffd700 1 -- 172.21.15.32:0/2209429080 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f621c01c5f0 con 0x7f6220102e10 2024-04-12T05:48:04.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.476+0000 7f62286c0700 1 -- 172.21.15.32:0/2209429080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f622010e2b0 con 0x7f6220102e10 2024-04-12T05:48:04.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.477+0000 7f62286c0700 1 -- 172.21.15.32:0/2209429080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62200fe560 con 0x7f6220102e10 2024-04-12T05:48:04.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.478+0000 7f6216ffd700 1 -- 172.21.15.32:0/2209429080 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f621c013590 con 0x7f6220102e10 2024-04-12T05:48:04.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.478+0000 7f6216ffd700 1 --2- 172.21.15.32:0/2209429080 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f620c06cc30 0x7f620c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:04.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.478+0000 7f6216ffd700 1 -- 172.21.15.32:0/2209429080 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f621c08d330 con 0x7f6220102e10 2024-04-12T05:48:04.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.478+0000 7f6225c5b700 1 --2- 172.21.15.32:0/2209429080 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f620c06cc30 0x7f620c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:04.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.479+0000 7f6225c5b700 1 --2- 172.21.15.32:0/2209429080 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f620c06cc30 0x7f620c06f0e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f622010ec30 tx=0x7f6210009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:04.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.483+0000 7f6216ffd700 1 -- 172.21.15.32:0/2209429080 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f621c05ada0 con 0x7f6220102e10 2024-04-12T05:48:04.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.694+0000 7f62286c0700 1 -- 172.21.15.32:0/2209429080 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6220109dd0 con 0x7f620c06cc30 2024-04-12T05:48:04.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.704+0000 7f6216ffd700 1 -- 172.21.15.32:0/2209429080 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f6220109dd0 con 0x7f620c06cc30 2024-04-12T05:48:04.700 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:48:04.700 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (7m) 80s ago 19m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:48:04.701 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (5m) 80s ago 19m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:48:04.701 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (5m) 50s ago 18m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:48:04.701 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (7m) 80s ago 19m 87.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:48:04.701 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (104s) 80s ago 13m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T05:48:04.701 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (85s) 80s ago 13m 13.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T05:48:04.701 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (52s) 50s ago 13m 11.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 6d4aca6f74a0 2024-04-12T05:48:04.701 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (68s) 50s ago 13m 14.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T05:48:04.701 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (9m) 80s ago 20m 507M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:48:04.701 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (8m) 50s ago 18m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:48:04.701 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (6m) 80s ago 20m 53.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:48:04.701 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (6m) 50s ago 18m 43.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:48:04.701 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (8m) 80s ago 19m 16.9M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:48:04.701 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (8m) 50s ago 18m 18.4M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:48:04.701 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (4m) 80s ago 17m 91.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T05:48:04.701 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (4m) 80s ago 17m 97.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T05:48:04.702 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (3m) 80s ago 16m 81.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T05:48:04.702 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (3m) 50s ago 16m 113M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T05:48:04.702 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (2m) 50s ago 15m 90.3M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T05:48:04.702 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (2m) 50s ago 15m 84.8M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T05:48:04.702 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (8m) 80s ago 18m 64.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:48:04.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.706+0000 7f62286c0700 1 -- 172.21.15.32:0/2209429080 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f620c06cc30 msgr2=0x7f620c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:04.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.706+0000 7f62286c0700 1 --2- 172.21.15.32:0/2209429080 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f620c06cc30 0x7f620c06f0e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f622010ec30 tx=0x7f6210009040 comp rx=0 tx=0).stop 2024-04-12T05:48:04.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.706+0000 7f62286c0700 1 -- 172.21.15.32:0/2209429080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6220102e10 msgr2=0x7f6220113af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:04.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.706+0000 7f62286c0700 1 --2- 172.21.15.32:0/2209429080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6220102e10 0x7f6220113af0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f621c009360 tx=0x7f621c009e30 comp rx=0 tx=0).stop 2024-04-12T05:48:04.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.706+0000 7f62286c0700 1 -- 172.21.15.32:0/2209429080 shutdown_connections 2024-04-12T05:48:04.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.706+0000 7f62286c0700 1 --2- 172.21.15.32:0/2209429080 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f620c06cc30 0x7f620c06f0e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:04.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.706+0000 7f62286c0700 1 --2- 172.21.15.32:0/2209429080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6220102e10 0x7f6220113af0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:04.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.706+0000 7f62286c0700 1 --2- 172.21.15.32:0/2209429080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62201037b0 0x7f6220114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:04.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.706+0000 7f62286c0700 1 -- 172.21.15.32:0/2209429080 >> 172.21.15.32:0/2209429080 conn(0x7f62200f89a0 msgr2=0x7f62200f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:04.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.706+0000 7f62286c0700 1 -- 172.21.15.32:0/2209429080 shutdown_connections 2024-04-12T05:48:04.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.706+0000 7f62286c0700 1 -- 172.21.15.32:0/2209429080 wait complete. 2024-04-12T05:48:04.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.808+0000 7f9a9701b700 1 -- 172.21.15.32:0/3753641820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a901037a0 msgr2=0x7f9a900fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:04.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.808+0000 7f9a9701b700 1 --2- 172.21.15.32:0/3753641820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a901037a0 0x7f9a900fce10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c004940 tx=0x7f9a7c009e30 comp rx=0 tx=0).stop 2024-04-12T05:48:04.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.808+0000 7f9a9701b700 1 -- 172.21.15.32:0/3753641820 shutdown_connections 2024-04-12T05:48:04.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.808+0000 7f9a9701b700 1 --2- 172.21.15.32:0/3753641820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a901037a0 0x7f9a900fce10 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:04.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.808+0000 7f9a9701b700 1 --2- 172.21.15.32:0/3753641820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a90102e00 0x7f9a901031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:04.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.808+0000 7f9a9701b700 1 -- 172.21.15.32:0/3753641820 >> 172.21.15.32:0/3753641820 conn(0x7f9a900f89b0 msgr2=0x7f9a900fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:04.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.808+0000 7f9a9701b700 1 -- 172.21.15.32:0/3753641820 shutdown_connections 2024-04-12T05:48:04.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.808+0000 7f9a9701b700 1 -- 172.21.15.32:0/3753641820 wait complete. 2024-04-12T05:48:04.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.809+0000 7f9a9701b700 1 Processor -- start 2024-04-12T05:48:04.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.809+0000 7f9a9701b700 1 -- start start 2024-04-12T05:48:04.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.809+0000 7f9a9701b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a90102e00 0x7f9a900702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:04.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.809+0000 7f9a9701b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a901037a0 0x7f9a90070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:04.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.809+0000 7f9a9701b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a90072210 con 0x7f9a901037a0 2024-04-12T05:48:04.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.809+0000 7f9a9701b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a90070d40 con 0x7f9a90102e00 2024-04-12T05:48:04.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.809+0000 7f9a8ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a901037a0 0x7f9a90070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:04.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.809+0000 7f9a8ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a901037a0 0x7f9a90070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49344/0 (socket says 172.21.15.32:49344) 2024-04-12T05:48:04.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.809+0000 7f9a8ffff700 1 -- 172.21.15.32:0/2732069471 learned_addr learned my addr 172.21.15.32:0/2732069471 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:48:04.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.810+0000 7f9a94db7700 1 --2- 172.21.15.32:0/2732069471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a90102e00 0x7f9a900702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:04.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.810+0000 7f9a8ffff700 1 -- 172.21.15.32:0/2732069471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a90102e00 msgr2=0x7f9a900702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:04.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.810+0000 7f9a8ffff700 1 --2- 172.21.15.32:0/2732069471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a90102e00 0x7f9a900702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:04.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.810+0000 7f9a8ffff700 1 -- 172.21.15.32:0/2732069471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a7c005040 con 0x7f9a901037a0 2024-04-12T05:48:04.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.810+0000 7f9a8ffff700 1 --2- 172.21.15.32:0/2732069471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a901037a0 0x7f9a90070800 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c009380 tx=0x7f9a7c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:04.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.810+0000 7f9a8d7fa700 1 -- 172.21.15.32:0/2732069471 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a7c00e040 con 0x7f9a901037a0 2024-04-12T05:48:04.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.810+0000 7f9a8d7fa700 1 -- 172.21.15.32:0/2732069471 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a7c013420 con 0x7f9a901037a0 2024-04-12T05:48:04.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.810+0000 7f9a9701b700 1 -- 172.21.15.32:0/2732069471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a90070fc0 con 0x7f9a901037a0 2024-04-12T05:48:04.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.810+0000 7f9a8d7fa700 1 -- 172.21.15.32:0/2732069471 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a7c01c630 con 0x7f9a901037a0 2024-04-12T05:48:04.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.810+0000 7f9a9701b700 1 -- 172.21.15.32:0/2732069471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a90119fe0 con 0x7f9a901037a0 2024-04-12T05:48:04.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.811+0000 7f9a9701b700 1 -- 172.21.15.32:0/2732069471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a900fe550 con 0x7f9a901037a0 2024-04-12T05:48:04.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.812+0000 7f9a8d7fa700 1 -- 172.21.15.32:0/2732069471 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9a7c0139e0 con 0x7f9a901037a0 2024-04-12T05:48:04.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.812+0000 7f9a8d7fa700 1 --2- 172.21.15.32:0/2732069471 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9a8006cc20 0x7f9a8006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:04.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.812+0000 7f9a8d7fa700 1 -- 172.21.15.32:0/2732069471 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f9a7c08d270 con 0x7f9a901037a0 2024-04-12T05:48:04.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.812+0000 7f9a94db7700 1 --2- 172.21.15.32:0/2732069471 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9a8006cc20 0x7f9a8006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:04.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.813+0000 7f9a94db7700 1 --2- 172.21.15.32:0/2732069471 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9a8006cc20 0x7f9a8006f0d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9a84007fd0 tx=0x7f9a84009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:04.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:04.817+0000 7f9a8d7fa700 1 -- 172.21.15.32:0/2732069471 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9a7c057250 con 0x7f9a901037a0 2024-04-12T05:48:05.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.105+0000 7f9a9701b700 1 -- 172.21.15.32:0/2732069471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9a9011a1d0 con 0x7f9a901037a0 2024-04-12T05:48:05.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.105+0000 7f9a8d7fa700 1 -- 172.21.15.32:0/2732069471 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f9a7c05a870 con 0x7f9a901037a0 2024-04-12T05:48:05.102 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:48:05.102 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:48:05.102 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:48:05.102 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:48:05.102 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:48:05.102 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:48:05.102 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:48:05.102 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:48:05.102 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:48:05.102 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:48:05.102 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:48:05.102 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:48:05.102 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:48:05.103 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:48:05.103 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:48:05.103 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:48:05.103 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:48:05.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.107+0000 7f9a9701b700 1 -- 172.21.15.32:0/2732069471 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9a8006cc20 msgr2=0x7f9a8006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:05.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.107+0000 7f9a9701b700 1 --2- 172.21.15.32:0/2732069471 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9a8006cc20 0x7f9a8006f0d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9a84007fd0 tx=0x7f9a84009040 comp rx=0 tx=0).stop 2024-04-12T05:48:05.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.107+0000 7f9a9701b700 1 -- 172.21.15.32:0/2732069471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a901037a0 msgr2=0x7f9a90070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:05.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.107+0000 7f9a9701b700 1 --2- 172.21.15.32:0/2732069471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a901037a0 0x7f9a90070800 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c009380 tx=0x7f9a7c009e30 comp rx=0 tx=0).stop 2024-04-12T05:48:05.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.107+0000 7f9a9701b700 1 -- 172.21.15.32:0/2732069471 shutdown_connections 2024-04-12T05:48:05.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.107+0000 7f9a9701b700 1 --2- 172.21.15.32:0/2732069471 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9a8006cc20 0x7f9a8006f0d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:05.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.107+0000 7f9a9701b700 1 --2- 172.21.15.32:0/2732069471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a90102e00 0x7f9a900702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:05.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.107+0000 7f9a9701b700 1 --2- 172.21.15.32:0/2732069471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a901037a0 0x7f9a90070800 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:05.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.107+0000 7f9a9701b700 1 -- 172.21.15.32:0/2732069471 >> 172.21.15.32:0/2732069471 conn(0x7f9a900f89b0 msgr2=0x7f9a900f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:05.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.107+0000 7f9a9701b700 1 -- 172.21.15.32:0/2732069471 shutdown_connections 2024-04-12T05:48:05.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.107+0000 7f9a9701b700 1 -- 172.21.15.32:0/2732069471 wait complete. 2024-04-12T05:48:05.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.210+0000 7f83b3a34700 1 -- 172.21.15.32:0/1336529128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83ac070c10 msgr2=0x7f83ac109180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:05.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.210+0000 7f83b3a34700 1 --2- 172.21.15.32:0/1336529128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83ac070c10 0x7f83ac109180 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f839c004970 tx=0x7f839c009e30 comp rx=0 tx=0).stop 2024-04-12T05:48:05.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.211+0000 7f83b3a34700 1 -- 172.21.15.32:0/1336529128 shutdown_connections 2024-04-12T05:48:05.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.211+0000 7f83b3a34700 1 --2- 172.21.15.32:0/1336529128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83ac070c10 0x7f83ac109180 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:05.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.211+0000 7f83b3a34700 1 --2- 172.21.15.32:0/1336529128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83ac070300 0x7f83ac0706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:05.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.211+0000 7f83b3a34700 1 -- 172.21.15.32:0/1336529128 >> 172.21.15.32:0/1336529128 conn(0x7f83ac0f8980 msgr2=0x7f83ac0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:05.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.211+0000 7f83b3a34700 1 -- 172.21.15.32:0/1336529128 shutdown_connections 2024-04-12T05:48:05.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.211+0000 7f83b3a34700 1 -- 172.21.15.32:0/1336529128 wait complete. 2024-04-12T05:48:05.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.211+0000 7f83b3a34700 1 Processor -- start 2024-04-12T05:48:05.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.212+0000 7f83b3a34700 1 -- start start 2024-04-12T05:48:05.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.212+0000 7f83b3a34700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83ac070300 0x7f83ac115c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:05.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.212+0000 7f83b3a34700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83ac070c10 0x7f83ac116190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:05.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.212+0000 7f83b3a34700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac1167e0 con 0x7f83ac070300 2024-04-12T05:48:05.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.212+0000 7f83b3a34700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac10fbb0 con 0x7f83ac070c10 2024-04-12T05:48:05.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.212+0000 7f83b17d0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83ac070300 0x7f83ac115c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:05.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.212+0000 7f83b17d0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83ac070300 0x7f83ac115c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49372/0 (socket says 172.21.15.32:49372) 2024-04-12T05:48:05.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.212+0000 7f83b17d0700 1 -- 172.21.15.32:0/3115033202 learned_addr learned my addr 172.21.15.32:0/3115033202 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:48:05.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.212+0000 7f83b0fcf700 1 --2- 172.21.15.32:0/3115033202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83ac070c10 0x7f83ac116190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:05.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.212+0000 7f83b17d0700 1 -- 172.21.15.32:0/3115033202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83ac070c10 msgr2=0x7f83ac116190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:05.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.212+0000 7f83b17d0700 1 --2- 172.21.15.32:0/3115033202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83ac070c10 0x7f83ac116190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:05.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.212+0000 7f83b17d0700 1 -- 172.21.15.32:0/3115033202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f839c005040 con 0x7f83ac070300 2024-04-12T05:48:05.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.212+0000 7f83b17d0700 1 --2- 172.21.15.32:0/3115033202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83ac070300 0x7f83ac115c50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f83a8002a50 tx=0x7f83a80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:05.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.213+0000 7f83a1ffb700 1 -- 172.21.15.32:0/3115033202 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83a8007cc0 con 0x7f83ac070300 2024-04-12T05:48:05.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.213+0000 7f83b3a34700 1 -- 172.21.15.32:0/3115033202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83ac10fe90 con 0x7f83ac070300 2024-04-12T05:48:05.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.213+0000 7f83a1ffb700 1 -- 172.21.15.32:0/3115033202 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f83a800f040 con 0x7f83ac070300 2024-04-12T05:48:05.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.213+0000 7f83a1ffb700 1 -- 172.21.15.32:0/3115033202 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83a800d9c0 con 0x7f83ac070300 2024-04-12T05:48:05.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.213+0000 7f83b3a34700 1 -- 172.21.15.32:0/3115033202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83ac1103a0 con 0x7f83ac070300 2024-04-12T05:48:05.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.214+0000 7f83b3a34700 1 -- 172.21.15.32:0/3115033202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83ac071a40 con 0x7f83ac070300 2024-04-12T05:48:05.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.214+0000 7f83a1ffb700 1 -- 172.21.15.32:0/3115033202 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f83a800db20 con 0x7f83ac070300 2024-04-12T05:48:05.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.215+0000 7f83a1ffb700 1 --2- 172.21.15.32:0/3115033202 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f839806cc30 0x7f839806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:05.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.215+0000 7f83a1ffb700 1 -- 172.21.15.32:0/3115033202 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f83a808bdd0 con 0x7f83ac070300 2024-04-12T05:48:05.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.215+0000 7f83b0fcf700 1 --2- 172.21.15.32:0/3115033202 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f839806cc30 0x7f839806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:05.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.215+0000 7f83b0fcf700 1 --2- 172.21.15.32:0/3115033202 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f839806cc30 0x7f839806f0e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f839c009fd0 tx=0x7f839c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:05.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.220+0000 7f83a1ffb700 1 -- 172.21.15.32:0/3115033202 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f83a80597c0 con 0x7f83ac070300 2024-04-12T05:48:05.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.467+0000 7f83b3a34700 1 -- 172.21.15.32:0/3115033202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f83ac04bed0 con 0x7f83ac070300 2024-04-12T05:48:05.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.467+0000 7f83a1ffb700 1 -- 172.21.15.32:0/3115033202 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1835 (secure 0 0 0) 0x7f83a8059350 con 0x7f83ac070300 2024-04-12T05:48:05.464 INFO:teuthology.orchestra.run.smithi032.stdout:e33 2024-04-12T05:48:05.464 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:48:05.464 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:48:05.464 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:48:05.464 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:48:05.464 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:48:05.464 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:48:05.464 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 24 2024-04-12T05:48:05.464 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:48:05.464 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:48:05.464 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:46:10.668569+0000 2024-04-12T05:48:05.464 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:48:05.464 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:48:05.464 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:48:05.465 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:48:05.465 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:48:05.465 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:48:05.465 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:48:05.465 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:48:05.465 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T05:48:05.465 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:48:05.465 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:48:05.465 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:48:05.465 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T05:48:05.465 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-12T05:48:05.465 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:48:05.465 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:48:05.465 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:48:05.465 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:48:05.465 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:48:05.466 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:48:05.466 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:48:05.466 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:48:05.466 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:48:05.466 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:48:05.466 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-12T05:48:05.466 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:48:05.466 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:48:05.466 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:48:05.466 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{-1:44317} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:48:05.466 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:48:05.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.469+0000 7f83b3a34700 1 -- 172.21.15.32:0/3115033202 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f839806cc30 msgr2=0x7f839806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:05.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.469+0000 7f83b3a34700 1 --2- 172.21.15.32:0/3115033202 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f839806cc30 0x7f839806f0e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f839c009fd0 tx=0x7f839c009e30 comp rx=0 tx=0).stop 2024-04-12T05:48:05.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.469+0000 7f83b3a34700 1 -- 172.21.15.32:0/3115033202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83ac070300 msgr2=0x7f83ac115c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:05.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.469+0000 7f83b3a34700 1 --2- 172.21.15.32:0/3115033202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83ac070300 0x7f83ac115c50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f83a8002a50 tx=0x7f83a80046b0 comp rx=0 tx=0).stop 2024-04-12T05:48:05.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.469+0000 7f83b3a34700 1 -- 172.21.15.32:0/3115033202 shutdown_connections 2024-04-12T05:48:05.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.469+0000 7f83b3a34700 1 --2- 172.21.15.32:0/3115033202 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f839806cc30 0x7f839806f0e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:05.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.469+0000 7f83b3a34700 1 --2- 172.21.15.32:0/3115033202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83ac070300 0x7f83ac115c50 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:05.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.469+0000 7f83b3a34700 1 --2- 172.21.15.32:0/3115033202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83ac070c10 0x7f83ac116190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:05.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.469+0000 7f83b3a34700 1 -- 172.21.15.32:0/3115033202 >> 172.21.15.32:0/3115033202 conn(0x7f83ac0f8980 msgr2=0x7f83ac0f9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:05.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.469+0000 7f83b3a34700 1 -- 172.21.15.32:0/3115033202 shutdown_connections 2024-04-12T05:48:05.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.469+0000 7f83b3a34700 1 -- 172.21.15.32:0/3115033202 wait complete. 2024-04-12T05:48:05.468 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-12T05:48:05.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.573+0000 7f65020cd700 1 -- 172.21.15.32:0/3582602441 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64fc101d70 msgr2=0x7f64fc102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:05.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.573+0000 7f65020cd700 1 --2- 172.21.15.32:0/3582602441 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64fc101d70 0x7f64fc102140 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f64ec004940 tx=0x7f64ec009e30 comp rx=0 tx=0).stop 2024-04-12T05:48:05.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.574+0000 7f65020cd700 1 -- 172.21.15.32:0/3582602441 shutdown_connections 2024-04-12T05:48:05.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.574+0000 7f65020cd700 1 --2- 172.21.15.32:0/3582602441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64fc102710 0x7f64fc0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:05.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.574+0000 7f65020cd700 1 --2- 172.21.15.32:0/3582602441 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64fc101d70 0x7f64fc102140 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:05.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.574+0000 7f65020cd700 1 -- 172.21.15.32:0/3582602441 >> 172.21.15.32:0/3582602441 conn(0x7f64fc072bc0 msgr2=0x7f64fc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:05.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.574+0000 7f65020cd700 1 -- 172.21.15.32:0/3582602441 shutdown_connections 2024-04-12T05:48:05.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.574+0000 7f65020cd700 1 -- 172.21.15.32:0/3582602441 wait complete. 2024-04-12T05:48:05.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.574+0000 7f65020cd700 1 Processor -- start 2024-04-12T05:48:05.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.574+0000 7f65020cd700 1 -- start start 2024-04-12T05:48:05.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.575+0000 7f65020cd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64fc101d70 0x7f64fc1129a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:05.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.575+0000 7f65020cd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64fc102710 0x7f64fc112ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:05.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.575+0000 7f65020cd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64fc113530 con 0x7f64fc102710 2024-04-12T05:48:05.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.575+0000 7f65020cd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64fc10c920 con 0x7f64fc101d70 2024-04-12T05:48:05.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.575+0000 7f64faffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64fc102710 0x7f64fc112ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:05.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.575+0000 7f64faffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64fc102710 0x7f64fc112ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49388/0 (socket says 172.21.15.32:49388) 2024-04-12T05:48:05.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.575+0000 7f64faffd700 1 -- 172.21.15.32:0/4242409941 learned_addr learned my addr 172.21.15.32:0/4242409941 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:48:05.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.575+0000 7f64fb7fe700 1 --2- 172.21.15.32:0/4242409941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64fc101d70 0x7f64fc1129a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:05.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.575+0000 7f64faffd700 1 -- 172.21.15.32:0/4242409941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64fc101d70 msgr2=0x7f64fc1129a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:05.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.575+0000 7f64faffd700 1 --2- 172.21.15.32:0/4242409941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64fc101d70 0x7f64fc1129a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:05.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.575+0000 7f64faffd700 1 -- 172.21.15.32:0/4242409941 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64ec005040 con 0x7f64fc102710 2024-04-12T05:48:05.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.575+0000 7f64faffd700 1 --2- 172.21.15.32:0/4242409941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64fc102710 0x7f64fc112ee0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f64e400fce0 tx=0x7f64e4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:05.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.576+0000 7f64f3fff700 1 -- 172.21.15.32:0/4242409941 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64e400fe90 con 0x7f64fc102710 2024-04-12T05:48:05.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.576+0000 7f64f3fff700 1 -- 172.21.15.32:0/4242409941 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f64e40090d0 con 0x7f64fc102710 2024-04-12T05:48:05.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.576+0000 7f65020cd700 1 -- 172.21.15.32:0/4242409941 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64fc10cba0 con 0x7f64fc102710 2024-04-12T05:48:05.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.576+0000 7f64f3fff700 1 -- 172.21.15.32:0/4242409941 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64e40126a0 con 0x7f64fc102710 2024-04-12T05:48:05.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.576+0000 7f65020cd700 1 -- 172.21.15.32:0/4242409941 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64fc10d160 con 0x7f64fc102710 2024-04-12T05:48:05.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.576+0000 7f65020cd700 1 -- 172.21.15.32:0/4242409941 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64fc0fd4c0 con 0x7f64fc102710 2024-04-12T05:48:05.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.577+0000 7f64f3fff700 1 -- 172.21.15.32:0/4242409941 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f64e4009690 con 0x7f64fc102710 2024-04-12T05:48:05.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.577+0000 7f64f3fff700 1 --2- 172.21.15.32:0/4242409941 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f64e806cc30 0x7f64e806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:05.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.578+0000 7f64f3fff700 1 -- 172.21.15.32:0/4242409941 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f64e408e7d0 con 0x7f64fc102710 2024-04-12T05:48:05.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.578+0000 7f64fb7fe700 1 --2- 172.21.15.32:0/4242409941 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f64e806cc30 0x7f64e806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:05.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.578+0000 7f64fb7fe700 1 --2- 172.21.15.32:0/4242409941 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f64e806cc30 0x7f64e806f0e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f64ec00b020 tx=0x7f64ec009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:05.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.582+0000 7f64f3fff700 1 -- 172.21.15.32:0/4242409941 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f64e405c240 con 0x7f64fc102710 2024-04-12T05:48:05.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.807+0000 7f65020cd700 1 -- 172.21.15.32:0/4242409941 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f64fc108da0 con 0x7f64e806cc30 2024-04-12T05:48:05.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.808+0000 7f64f3fff700 1 -- 172.21.15.32:0/4242409941 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f64fc108da0 con 0x7f64e806cc30 2024-04-12T05:48:05.805 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:48:05.805 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:48:05.805 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:48:05.805 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:48:05.805 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:48:05.805 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-12T05:48:05.805 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:48:05.805 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-12T05:48:05.805 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:48:05.805 INFO:teuthology.orchestra.run.smithi032.stdout: "mds" 2024-04-12T05:48:05.805 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:48:05.805 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:48:05.805 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:48:05.805 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:48:05.806 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:48:05.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.810+0000 7f65020cd700 1 -- 172.21.15.32:0/4242409941 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f64e806cc30 msgr2=0x7f64e806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:05.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.810+0000 7f65020cd700 1 --2- 172.21.15.32:0/4242409941 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f64e806cc30 0x7f64e806f0e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f64ec00b020 tx=0x7f64ec009e30 comp rx=0 tx=0).stop 2024-04-12T05:48:05.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.810+0000 7f65020cd700 1 -- 172.21.15.32:0/4242409941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64fc102710 msgr2=0x7f64fc112ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:05.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.810+0000 7f65020cd700 1 --2- 172.21.15.32:0/4242409941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64fc102710 0x7f64fc112ee0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f64e400fce0 tx=0x7f64e4004600 comp rx=0 tx=0).stop 2024-04-12T05:48:05.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.810+0000 7f65020cd700 1 -- 172.21.15.32:0/4242409941 shutdown_connections 2024-04-12T05:48:05.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.810+0000 7f65020cd700 1 --2- 172.21.15.32:0/4242409941 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f64e806cc30 0x7f64e806f0e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:05.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.810+0000 7f65020cd700 1 --2- 172.21.15.32:0/4242409941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64fc101d70 0x7f64fc1129a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:05.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.810+0000 7f65020cd700 1 --2- 172.21.15.32:0/4242409941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64fc102710 0x7f64fc112ee0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:05.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.810+0000 7f65020cd700 1 -- 172.21.15.32:0/4242409941 >> 172.21.15.32:0/4242409941 conn(0x7f64fc072bc0 msgr2=0x7f64fc0fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:05.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.811+0000 7f65020cd700 1 -- 172.21.15.32:0/4242409941 shutdown_connections 2024-04-12T05:48:05.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.811+0000 7f65020cd700 1 -- 172.21.15.32:0/4242409941 wait complete. 2024-04-12T05:48:05.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.913+0000 7f4b5f8d2700 1 -- 172.21.15.32:0/3468525477 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b58102e10 msgr2=0x7f4b581031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:05.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.913+0000 7f4b5f8d2700 1 --2- 172.21.15.32:0/3468525477 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b58102e10 0x7f4b581031e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4b54004940 tx=0x7f4b54009e30 comp rx=0 tx=0).stop 2024-04-12T05:48:05.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.913+0000 7f4b5f8d2700 1 -- 172.21.15.32:0/3468525477 shutdown_connections 2024-04-12T05:48:05.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.913+0000 7f4b5f8d2700 1 --2- 172.21.15.32:0/3468525477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b581037b0 0x7f4b580fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:05.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.913+0000 7f4b5f8d2700 1 --2- 172.21.15.32:0/3468525477 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b58102e10 0x7f4b581031e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:05.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.913+0000 7f4b5f8d2700 1 -- 172.21.15.32:0/3468525477 >> 172.21.15.32:0/3468525477 conn(0x7f4b580f89a0 msgr2=0x7f4b580fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:05.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.914+0000 7f4b5f8d2700 1 -- 172.21.15.32:0/3468525477 shutdown_connections 2024-04-12T05:48:05.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.914+0000 7f4b5f8d2700 1 -- 172.21.15.32:0/3468525477 wait complete. 2024-04-12T05:48:05.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.914+0000 7f4b5f8d2700 1 Processor -- start 2024-04-12T05:48:05.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.914+0000 7f4b5f8d2700 1 -- start start 2024-04-12T05:48:05.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.915+0000 7f4b5f8d2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b58102e10 0x7f4b58113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:05.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.915+0000 7f4b5f8d2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b581037b0 0x7f4b58114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:05.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.915+0000 7f4b5f8d2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b58114680 con 0x7f4b58102e10 2024-04-12T05:48:05.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.915+0000 7f4b5f8d2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5810da70 con 0x7f4b581037b0 2024-04-12T05:48:05.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.915+0000 7f4b5d66e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b58102e10 0x7f4b58113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:05.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.915+0000 7f4b5d66e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b58102e10 0x7f4b58113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49410/0 (socket says 172.21.15.32:49410) 2024-04-12T05:48:05.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.915+0000 7f4b5d66e700 1 -- 172.21.15.32:0/3464674378 learned_addr learned my addr 172.21.15.32:0/3464674378 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:48:05.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.915+0000 7f4b5ce6d700 1 --2- 172.21.15.32:0/3464674378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b581037b0 0x7f4b58114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:05.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.915+0000 7f4b5d66e700 1 -- 172.21.15.32:0/3464674378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b581037b0 msgr2=0x7f4b58114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:05.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.915+0000 7f4b5d66e700 1 --2- 172.21.15.32:0/3464674378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b581037b0 0x7f4b58114030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:05.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.915+0000 7f4b5d66e700 1 -- 172.21.15.32:0/3464674378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b54005040 con 0x7f4b58102e10 2024-04-12T05:48:05.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.915+0000 7f4b5d66e700 1 --2- 172.21.15.32:0/3464674378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b58102e10 0x7f4b58113af0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4b54009390 tx=0x7f4b54009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:05.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.915+0000 7f4b49ffb700 1 -- 172.21.15.32:0/3464674378 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b5400e040 con 0x7f4b58102e10 2024-04-12T05:48:05.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.915+0000 7f4b49ffb700 1 -- 172.21.15.32:0/3464674378 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b54013420 con 0x7f4b58102e10 2024-04-12T05:48:05.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.915+0000 7f4b5f8d2700 1 -- 172.21.15.32:0/3464674378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b5810dcf0 con 0x7f4b58102e10 2024-04-12T05:48:05.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.915+0000 7f4b49ffb700 1 -- 172.21.15.32:0/3464674378 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b5401c630 con 0x7f4b58102e10 2024-04-12T05:48:05.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.916+0000 7f4b5f8d2700 1 -- 172.21.15.32:0/3464674378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b5810e2b0 con 0x7f4b58102e10 2024-04-12T05:48:05.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.917+0000 7f4b5f8d2700 1 -- 172.21.15.32:0/3464674378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b580fe560 con 0x7f4b58102e10 2024-04-12T05:48:05.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.917+0000 7f4b49ffb700 1 -- 172.21.15.32:0/3464674378 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4b5400bb50 con 0x7f4b58102e10 2024-04-12T05:48:05.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.917+0000 7f4b49ffb700 1 --2- 172.21.15.32:0/3464674378 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f4b4406cc30 0x7f4b4406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:05.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.917+0000 7f4b49ffb700 1 -- 172.21.15.32:0/3464674378 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f4b5408d470 con 0x7f4b58102e10 2024-04-12T05:48:05.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.917+0000 7f4b5ce6d700 1 --2- 172.21.15.32:0/3464674378 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f4b4406cc30 0x7f4b4406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:05.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.918+0000 7f4b5ce6d700 1 --2- 172.21.15.32:0/3464674378 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f4b4406cc30 0x7f4b4406f0e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4b5810ec30 tx=0x7f4b4c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:05.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:05.922+0000 7f4b49ffb700 1 -- 172.21.15.32:0/3464674378 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4b5405ae40 con 0x7f4b58102e10 2024-04-12T05:48:06.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:05 smithi067 ceph-mon[112979]: pgmap v227: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:06.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:05 smithi067 ceph-mon[112979]: from='client.34480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:06.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:05 smithi067 ceph-mon[112979]: from='client.34484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:06.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:05 smithi067 ceph-mon[112979]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:06.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:05 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/2732069471' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:48:06.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:05 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3115033202' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:48:06.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:05 smithi032 ceph-mon[146817]: pgmap v227: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:06.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:05 smithi032 ceph-mon[146817]: from='client.34480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:06.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:05 smithi032 ceph-mon[146817]: from='client.34484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:06.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:05 smithi032 ceph-mon[146817]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:06.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:05 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/2732069471' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:48:06.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:05 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3115033202' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:48:06.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:06.212+0000 7f4b5f8d2700 1 -- 172.21.15.32:0/3464674378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4b58109dd0 con 0x7f4b58102e10 2024-04-12T05:48:06.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:06.212+0000 7f4b49ffb700 1 -- 172.21.15.32:0/3464674378 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f4b5405a9d0 con 0x7f4b58102e10 2024-04-12T05:48:06.209 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:48:06.209 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:48:06.209 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-12T05:48:06.209 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:48:06.209 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:48:06.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:06.214+0000 7f4b5f8d2700 1 -- 172.21.15.32:0/3464674378 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f4b4406cc30 msgr2=0x7f4b4406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:06.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:06.214+0000 7f4b5f8d2700 1 --2- 172.21.15.32:0/3464674378 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f4b4406cc30 0x7f4b4406f0e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4b5810ec30 tx=0x7f4b4c009040 comp rx=0 tx=0).stop 2024-04-12T05:48:06.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:06.214+0000 7f4b5f8d2700 1 -- 172.21.15.32:0/3464674378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b58102e10 msgr2=0x7f4b58113af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:06.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:06.214+0000 7f4b5f8d2700 1 --2- 172.21.15.32:0/3464674378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b58102e10 0x7f4b58113af0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4b54009390 tx=0x7f4b54009e30 comp rx=0 tx=0).stop 2024-04-12T05:48:06.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:06.214+0000 7f4b5f8d2700 1 -- 172.21.15.32:0/3464674378 shutdown_connections 2024-04-12T05:48:06.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:06.214+0000 7f4b5f8d2700 1 --2- 172.21.15.32:0/3464674378 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f4b4406cc30 0x7f4b4406f0e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:06.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:06.214+0000 7f4b5f8d2700 1 --2- 172.21.15.32:0/3464674378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b58102e10 0x7f4b58113af0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:06.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:06.214+0000 7f4b5f8d2700 1 --2- 172.21.15.32:0/3464674378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b581037b0 0x7f4b58114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:06.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:06.214+0000 7f4b5f8d2700 1 -- 172.21.15.32:0/3464674378 >> 172.21.15.32:0/3464674378 conn(0x7f4b580f89a0 msgr2=0x7f4b580f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:06.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:06.215+0000 7f4b5f8d2700 1 -- 172.21.15.32:0/3464674378 shutdown_connections 2024-04-12T05:48:06.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:06.215+0000 7f4b5f8d2700 1 -- 172.21.15.32:0/3464674378 wait complete. 2024-04-12T05:48:07.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:06 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3464674378' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:48:07.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:06 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3464674378' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:48:08.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:07 smithi067 ceph-mon[112979]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:08.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:07 smithi067 ceph-mon[112979]: pgmap v228: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:08.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:07 smithi032 ceph-mon[146817]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:08.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:07 smithi032 ceph-mon[146817]: pgmap v228: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:10.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:09 smithi067 ceph-mon[112979]: pgmap v229: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:10.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:09 smithi032 ceph-mon[146817]: pgmap v229: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:12.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:11 smithi067 ceph-mon[112979]: pgmap v230: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:12.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:11 smithi032 ceph-mon[146817]: pgmap v230: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:14.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:13 smithi067 ceph-mon[112979]: pgmap v231: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:14.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:13 smithi032 ceph-mon[146817]: pgmap v231: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:16.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:15 smithi067 ceph-mon[112979]: pgmap v232: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:16.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:15 smithi032 ceph-mon[146817]: pgmap v232: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:18.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:17 smithi067 ceph-mon[112979]: pgmap v233: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:18.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:17 smithi032 ceph-mon[146817]: pgmap v233: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:19.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:48:19.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:48:20.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:19 smithi067 ceph-mon[112979]: pgmap v234: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:20.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:19 smithi032 ceph-mon[146817]: pgmap v234: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:22.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:21 smithi067 ceph-mon[112979]: pgmap v235: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:22.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:21 smithi032 ceph-mon[146817]: pgmap v235: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:24.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:23 smithi067 ceph-mon[112979]: pgmap v236: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:24.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:23 smithi032 ceph-mon[146817]: pgmap v236: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:26.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:25 smithi067 ceph-mon[112979]: pgmap v237: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:26.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:25 smithi032 ceph-mon[146817]: pgmap v237: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:28.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:27 smithi067 ceph-mon[112979]: pgmap v238: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:28.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:27 smithi032 ceph-mon[146817]: pgmap v238: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:30.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:29 smithi067 ceph-mon[112979]: pgmap v239: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:30.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:29 smithi032 ceph-mon[146817]: pgmap v239: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:32.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:31 smithi067 ceph-mon[112979]: pgmap v240: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:32.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:31 smithi032 ceph-mon[146817]: pgmap v240: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:34.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:33 smithi067 ceph-mon[112979]: pgmap v241: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:34.189 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:33 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:48:34.190 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:33 smithi032 ceph-mon[146817]: pgmap v241: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:34.191 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:33 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:48:36.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:35 smithi067 ceph-mon[112979]: pgmap v242: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:36.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:35 smithi032 ceph-mon[146817]: pgmap v242: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:36.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.332+0000 7feaa8059700 1 -- 172.21.15.32:0/3193589609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feaa0102e10 msgr2=0x7feaa01031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:36.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.332+0000 7feaa8059700 1 --2- 172.21.15.32:0/3193589609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feaa0102e10 0x7feaa01031e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fea9c004940 tx=0x7fea9c009e30 comp rx=0 tx=0).stop 2024-04-12T05:48:36.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.332+0000 7feaa8059700 1 -- 172.21.15.32:0/3193589609 shutdown_connections 2024-04-12T05:48:36.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.332+0000 7feaa8059700 1 --2- 172.21.15.32:0/3193589609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feaa01037b0 0x7feaa00fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:36.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.332+0000 7feaa8059700 1 --2- 172.21.15.32:0/3193589609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feaa0102e10 0x7feaa01031e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:36.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.332+0000 7feaa8059700 1 -- 172.21.15.32:0/3193589609 >> 172.21.15.32:0/3193589609 conn(0x7feaa00f8980 msgr2=0x7feaa00fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:36.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.332+0000 7feaa8059700 1 -- 172.21.15.32:0/3193589609 shutdown_connections 2024-04-12T05:48:36.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.333+0000 7feaa8059700 1 -- 172.21.15.32:0/3193589609 wait complete. 2024-04-12T05:48:36.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.333+0000 7feaa8059700 1 Processor -- start 2024-04-12T05:48:36.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.333+0000 7feaa8059700 1 -- start start 2024-04-12T05:48:36.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.333+0000 7feaa8059700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feaa0102e10 0x7feaa00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:36.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.333+0000 7feaa8059700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feaa01037b0 0x7feaa0070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:36.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.333+0000 7feaa8059700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa0070d90 con 0x7feaa01037b0 2024-04-12T05:48:36.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.333+0000 7feaa8059700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa0070ed0 con 0x7feaa0102e10 2024-04-12T05:48:36.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.333+0000 7feaa55f4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feaa01037b0 0x7feaa0070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:36.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.333+0000 7feaa55f4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feaa01037b0 0x7feaa0070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46928/0 (socket says 172.21.15.32:46928) 2024-04-12T05:48:36.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.333+0000 7feaa55f4700 1 -- 172.21.15.32:0/858760083 learned_addr learned my addr 172.21.15.32:0/858760083 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:48:36.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.334+0000 7feaa5df5700 1 --2- 172.21.15.32:0/858760083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feaa0102e10 0x7feaa00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:36.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.334+0000 7feaa55f4700 1 -- 172.21.15.32:0/858760083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feaa0102e10 msgr2=0x7feaa00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:36.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.334+0000 7feaa55f4700 1 --2- 172.21.15.32:0/858760083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feaa0102e10 0x7feaa00702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:36.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.334+0000 7feaa55f4700 1 -- 172.21.15.32:0/858760083 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea9c005040 con 0x7feaa01037b0 2024-04-12T05:48:36.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.334+0000 7feaa55f4700 1 --2- 172.21.15.32:0/858760083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feaa01037b0 0x7feaa0070800 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fea90002a50 tx=0x7fea90004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:36.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.334+0000 7fea967fc700 1 -- 172.21.15.32:0/858760083 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea90007cc0 con 0x7feaa01037b0 2024-04-12T05:48:36.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.334+0000 7feaa8059700 1 -- 172.21.15.32:0/858760083 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaa0119e10 con 0x7feaa01037b0 2024-04-12T05:48:36.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.334+0000 7fea967fc700 1 -- 172.21.15.32:0/858760083 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea9000f040 con 0x7feaa01037b0 2024-04-12T05:48:36.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.334+0000 7fea967fc700 1 -- 172.21.15.32:0/858760083 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea9000d920 con 0x7feaa01037b0 2024-04-12T05:48:36.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.334+0000 7feaa8059700 1 -- 172.21.15.32:0/858760083 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaa011a290 con 0x7feaa01037b0 2024-04-12T05:48:36.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.335+0000 7feaa8059700 1 -- 172.21.15.32:0/858760083 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaa00fe560 con 0x7feaa01037b0 2024-04-12T05:48:36.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.336+0000 7fea967fc700 1 -- 172.21.15.32:0/858760083 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fea90009680 con 0x7feaa01037b0 2024-04-12T05:48:36.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.336+0000 7fea967fc700 1 --2- 172.21.15.32:0/858760083 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fea8c06cc20 0x7fea8c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:36.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.336+0000 7fea967fc700 1 -- 172.21.15.32:0/858760083 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fea9008be10 con 0x7feaa01037b0 2024-04-12T05:48:36.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.336+0000 7feaa5df5700 1 --2- 172.21.15.32:0/858760083 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fea8c06cc20 0x7fea8c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:36.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.337+0000 7feaa5df5700 1 --2- 172.21.15.32:0/858760083 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fea8c06cc20 0x7fea8c06f0d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fea9c004f90 tx=0x7fea9c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:36.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.341+0000 7fea967fc700 1 -- 172.21.15.32:0/858760083 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fea90055d70 con 0x7feaa01037b0 2024-04-12T05:48:36.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.561+0000 7feaa8059700 1 -- 172.21.15.32:0/858760083 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7feaa004bed0 con 0x7fea8c06cc20 2024-04-12T05:48:36.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.563+0000 7fea967fc700 1 -- 172.21.15.32:0/858760083 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7feaa004bed0 con 0x7fea8c06cc20 2024-04-12T05:48:36.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.565+0000 7feaa8059700 1 -- 172.21.15.32:0/858760083 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fea8c06cc20 msgr2=0x7fea8c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:36.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.565+0000 7feaa8059700 1 --2- 172.21.15.32:0/858760083 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fea8c06cc20 0x7fea8c06f0d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fea9c004f90 tx=0x7fea9c009e30 comp rx=0 tx=0).stop 2024-04-12T05:48:36.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.565+0000 7feaa8059700 1 -- 172.21.15.32:0/858760083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feaa01037b0 msgr2=0x7feaa0070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:36.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.565+0000 7feaa8059700 1 --2- 172.21.15.32:0/858760083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feaa01037b0 0x7feaa0070800 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fea90002a50 tx=0x7fea90004600 comp rx=0 tx=0).stop 2024-04-12T05:48:36.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.565+0000 7feaa8059700 1 -- 172.21.15.32:0/858760083 shutdown_connections 2024-04-12T05:48:36.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.565+0000 7feaa8059700 1 --2- 172.21.15.32:0/858760083 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fea8c06cc20 0x7fea8c06f0d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:36.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.565+0000 7feaa8059700 1 --2- 172.21.15.32:0/858760083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feaa0102e10 0x7feaa00702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:36.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.565+0000 7feaa8059700 1 --2- 172.21.15.32:0/858760083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feaa01037b0 0x7feaa0070800 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:36.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.565+0000 7feaa8059700 1 -- 172.21.15.32:0/858760083 >> 172.21.15.32:0/858760083 conn(0x7feaa00f8980 msgr2=0x7feaa00f9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:36.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.565+0000 7feaa8059700 1 -- 172.21.15.32:0/858760083 shutdown_connections 2024-04-12T05:48:36.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.565+0000 7feaa8059700 1 -- 172.21.15.32:0/858760083 wait complete. 2024-04-12T05:48:36.571 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:48:36.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.669+0000 7f2f592cc700 1 -- 172.21.15.32:0/224151034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f540fe7d0 msgr2=0x7f2f54106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:36.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.669+0000 7f2f592cc700 1 --2- 172.21.15.32:0/224151034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f540fe7d0 0x7f2f54106ea0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c004970 tx=0x7f2f3c009e30 comp rx=0 tx=0).stop 2024-04-12T05:48:36.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.669+0000 7f2f592cc700 1 -- 172.21.15.32:0/224151034 shutdown_connections 2024-04-12T05:48:36.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.669+0000 7f2f592cc700 1 --2- 172.21.15.32:0/224151034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f540fe7d0 0x7f2f54106ea0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:36.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.669+0000 7f2f592cc700 1 --2- 172.21.15.32:0/224151034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f540fde30 0x7f2f540fe200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:36.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.669+0000 7f2f592cc700 1 -- 172.21.15.32:0/224151034 >> 172.21.15.32:0/224151034 conn(0x7f2f540f7870 msgr2=0x7f2f540f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:36.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.669+0000 7f2f592cc700 1 -- 172.21.15.32:0/224151034 shutdown_connections 2024-04-12T05:48:36.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.669+0000 7f2f592cc700 1 -- 172.21.15.32:0/224151034 wait complete. 2024-04-12T05:48:36.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.670+0000 7f2f592cc700 1 Processor -- start 2024-04-12T05:48:36.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.670+0000 7f2f592cc700 1 -- start start 2024-04-12T05:48:36.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.670+0000 7f2f592cc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f540fde30 0x7f2f540fd780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:36.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.670+0000 7f2f592cc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f540fe7d0 0x7f2f540fbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:36.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.670+0000 7f2f592cc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f540fc230 con 0x7f2f540fe7d0 2024-04-12T05:48:36.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.670+0000 7f2f592cc700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f540fc3a0 con 0x7f2f540fde30 2024-04-12T05:48:36.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.670+0000 7f2f527fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f540fe7d0 0x7f2f540fbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:36.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.670+0000 7f2f527fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f540fe7d0 0x7f2f540fbcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46950/0 (socket says 172.21.15.32:46950) 2024-04-12T05:48:36.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.670+0000 7f2f527fc700 1 -- 172.21.15.32:0/999593410 learned_addr learned my addr 172.21.15.32:0/999593410 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:48:36.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.670+0000 7f2f52ffd700 1 --2- 172.21.15.32:0/999593410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f540fde30 0x7f2f540fd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:36.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.671+0000 7f2f527fc700 1 -- 172.21.15.32:0/999593410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f540fde30 msgr2=0x7f2f540fd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:36.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.671+0000 7f2f527fc700 1 --2- 172.21.15.32:0/999593410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f540fde30 0x7f2f540fd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:36.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.671+0000 7f2f527fc700 1 -- 172.21.15.32:0/999593410 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f3c005040 con 0x7f2f540fe7d0 2024-04-12T05:48:36.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.671+0000 7f2f52ffd700 1 --2- 172.21.15.32:0/999593410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f540fde30 0x7f2f540fd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:48:36.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.671+0000 7f2f527fc700 1 --2- 172.21.15.32:0/999593410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f540fe7d0 0x7f2f540fbcf0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c0043d0 tx=0x7f2f3c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:36.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.671+0000 7f2f4b7fe700 1 -- 172.21.15.32:0/999593410 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f3c00e040 con 0x7f2f540fe7d0 2024-04-12T05:48:36.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.671+0000 7f2f4b7fe700 1 -- 172.21.15.32:0/999593410 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f3c013420 con 0x7f2f540fe7d0 2024-04-12T05:48:36.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.671+0000 7f2f592cc700 1 -- 172.21.15.32:0/999593410 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f540fc620 con 0x7f2f540fe7d0 2024-04-12T05:48:36.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.671+0000 7f2f4b7fe700 1 -- 172.21.15.32:0/999593410 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f3c01c630 con 0x7f2f540fe7d0 2024-04-12T05:48:36.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.671+0000 7f2f592cc700 1 -- 172.21.15.32:0/999593410 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f540fcb60 con 0x7f2f540fe7d0 2024-04-12T05:48:36.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.672+0000 7f2f592cc700 1 -- 172.21.15.32:0/999593410 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f54066ab0 con 0x7f2f540fe7d0 2024-04-12T05:48:36.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.672+0000 7f2f4b7fe700 1 -- 172.21.15.32:0/999593410 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f2f3c013970 con 0x7f2f540fe7d0 2024-04-12T05:48:36.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.673+0000 7f2f4b7fe700 1 --2- 172.21.15.32:0/999593410 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f2f4006cc20 0x7f2f4006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:36.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.673+0000 7f2f52ffd700 1 --2- 172.21.15.32:0/999593410 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f2f4006cc20 0x7f2f4006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:36.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.673+0000 7f2f4b7fe700 1 -- 172.21.15.32:0/999593410 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f2f3c08d290 con 0x7f2f540fe7d0 2024-04-12T05:48:36.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.673+0000 7f2f52ffd700 1 --2- 172.21.15.32:0/999593410 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f2f4006cc20 0x7f2f4006f0d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2f44009fd0 tx=0x7f2f4400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:36.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.677+0000 7f2f4b7fe700 1 -- 172.21.15.32:0/999593410 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2f3c05ac60 con 0x7f2f540fe7d0 2024-04-12T05:48:36.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.899+0000 7f2f592cc700 1 -- 172.21.15.32:0/999593410 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f540fcb60 con 0x7f2f4006cc20 2024-04-12T05:48:36.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.901+0000 7f2f4b7fe700 1 -- 172.21.15.32:0/999593410 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f2f540fcb60 con 0x7f2f4006cc20 2024-04-12T05:48:36.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.903+0000 7f2f592cc700 1 -- 172.21.15.32:0/999593410 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f2f4006cc20 msgr2=0x7f2f4006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:36.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.903+0000 7f2f592cc700 1 --2- 172.21.15.32:0/999593410 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f2f4006cc20 0x7f2f4006f0d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2f44009fd0 tx=0x7f2f4400b040 comp rx=0 tx=0).stop 2024-04-12T05:48:36.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.903+0000 7f2f592cc700 1 -- 172.21.15.32:0/999593410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f540fe7d0 msgr2=0x7f2f540fbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:36.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.903+0000 7f2f592cc700 1 --2- 172.21.15.32:0/999593410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f540fe7d0 0x7f2f540fbcf0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c0043d0 tx=0x7f2f3c009e30 comp rx=0 tx=0).stop 2024-04-12T05:48:36.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.903+0000 7f2f592cc700 1 -- 172.21.15.32:0/999593410 shutdown_connections 2024-04-12T05:48:36.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.903+0000 7f2f592cc700 1 --2- 172.21.15.32:0/999593410 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f2f4006cc20 0x7f2f4006f0d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:36.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.903+0000 7f2f592cc700 1 --2- 172.21.15.32:0/999593410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f540fde30 0x7f2f540fd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:36.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.903+0000 7f2f592cc700 1 --2- 172.21.15.32:0/999593410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f540fe7d0 0x7f2f540fbcf0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:36.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.903+0000 7f2f592cc700 1 -- 172.21.15.32:0/999593410 >> 172.21.15.32:0/999593410 conn(0x7f2f540f7870 msgr2=0x7f2f540f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:36.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.903+0000 7f2f592cc700 1 -- 172.21.15.32:0/999593410 shutdown_connections 2024-04-12T05:48:36.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:36.903+0000 7f2f592cc700 1 -- 172.21.15.32:0/999593410 wait complete. 2024-04-12T05:48:37.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.006+0000 7f48dbc0d700 1 -- 172.21.15.32:0/662924128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48d4102710 msgr2=0x7f48d40fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:37.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.006+0000 7f48dbc0d700 1 --2- 172.21.15.32:0/662924128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48d4102710 0x7f48d40fbd80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f48d0004940 tx=0x7f48d0009e30 comp rx=0 tx=0).stop 2024-04-12T05:48:37.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.006+0000 7f48dbc0d700 1 -- 172.21.15.32:0/662924128 shutdown_connections 2024-04-12T05:48:37.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.006+0000 7f48dbc0d700 1 --2- 172.21.15.32:0/662924128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48d4102710 0x7f48d40fbd80 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:37.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.006+0000 7f48dbc0d700 1 --2- 172.21.15.32:0/662924128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d4101d70 0x7f48d4102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:37.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.006+0000 7f48dbc0d700 1 -- 172.21.15.32:0/662924128 >> 172.21.15.32:0/662924128 conn(0x7f48d4072bc0 msgr2=0x7f48d4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:37.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.006+0000 7f48dbc0d700 1 -- 172.21.15.32:0/662924128 shutdown_connections 2024-04-12T05:48:37.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.006+0000 7f48dbc0d700 1 -- 172.21.15.32:0/662924128 wait complete. 2024-04-12T05:48:37.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.007+0000 7f48dbc0d700 1 Processor -- start 2024-04-12T05:48:37.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.007+0000 7f48dbc0d700 1 -- start start 2024-04-12T05:48:37.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.007+0000 7f48dbc0d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48d4101d70 0x7f48d4108690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:37.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.007+0000 7f48dbc0d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d4102710 0x7f48d4108bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:37.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.007+0000 7f48dbc0d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d4109110 con 0x7f48d4101d70 2024-04-12T05:48:37.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.007+0000 7f48dbc0d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d4109250 con 0x7f48d4102710 2024-04-12T05:48:37.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.007+0000 7f48d99a9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48d4101d70 0x7f48d4108690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:37.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.007+0000 7f48d99a9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48d4101d70 0x7f48d4108690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46980/0 (socket says 172.21.15.32:46980) 2024-04-12T05:48:37.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.007+0000 7f48d99a9700 1 -- 172.21.15.32:0/3377339227 learned_addr learned my addr 172.21.15.32:0/3377339227 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:48:37.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.007+0000 7f48d91a8700 1 --2- 172.21.15.32:0/3377339227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d4102710 0x7f48d4108bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:37.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.007+0000 7f48d99a9700 1 -- 172.21.15.32:0/3377339227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d4102710 msgr2=0x7f48d4108bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:37.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.007+0000 7f48d99a9700 1 --2- 172.21.15.32:0/3377339227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d4102710 0x7f48d4108bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:37.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.007+0000 7f48d99a9700 1 -- 172.21.15.32:0/3377339227 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d0005040 con 0x7f48d4101d70 2024-04-12T05:48:37.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.008+0000 7f48d99a9700 1 --2- 172.21.15.32:0/3377339227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48d4101d70 0x7f48d4108690 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f48c4002a50 tx=0x7f48c40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:37.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.008+0000 7f48ca7fc700 1 -- 172.21.15.32:0/3377339227 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48c4007cc0 con 0x7f48d4101d70 2024-04-12T05:48:37.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.008+0000 7f48ca7fc700 1 -- 172.21.15.32:0/3377339227 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f48c400f040 con 0x7f48d4101d70 2024-04-12T05:48:37.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.008+0000 7f48dbc0d700 1 -- 172.21.15.32:0/3377339227 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d4109500 con 0x7f48d4101d70 2024-04-12T05:48:37.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.008+0000 7f48ca7fc700 1 -- 172.21.15.32:0/3377339227 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48c400d9c0 con 0x7f48d4101d70 2024-04-12T05:48:37.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.008+0000 7f48dbc0d700 1 -- 172.21.15.32:0/3377339227 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48d411a250 con 0x7f48d4101d70 2024-04-12T05:48:37.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.009+0000 7f48dbc0d700 1 -- 172.21.15.32:0/3377339227 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48d40ff080 con 0x7f48d4101d70 2024-04-12T05:48:37.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.009+0000 7f48ca7fc700 1 -- 172.21.15.32:0/3377339227 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f48c400db20 con 0x7f48d4101d70 2024-04-12T05:48:37.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.010+0000 7f48ca7fc700 1 --2- 172.21.15.32:0/3377339227 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f48c006cc30 0x7f48c006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:37.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.010+0000 7f48ca7fc700 1 -- 172.21.15.32:0/3377339227 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f48c408bbd0 con 0x7f48d4101d70 2024-04-12T05:48:37.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.010+0000 7f48d91a8700 1 --2- 172.21.15.32:0/3377339227 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f48c006cc30 0x7f48c006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:37.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.010+0000 7f48d91a8700 1 --2- 172.21.15.32:0/3377339227 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f48c006cc30 0x7f48c006f0e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f48d4109f80 tx=0x7f48d0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:37.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.014+0000 7f48ca7fc700 1 -- 172.21.15.32:0/3377339227 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f48c4055b50 con 0x7f48d4101d70 2024-04-12T05:48:37.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.226+0000 7f48dbc0d700 1 -- 172.21.15.32:0/3377339227 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f48d4109500 con 0x7f48c006cc30 2024-04-12T05:48:37.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.236+0000 7f48ca7fc700 1 -- 172.21.15.32:0/3377339227 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f48d4109500 con 0x7f48c006cc30 2024-04-12T05:48:37.232 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:48:37.232 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (8m) 112s ago 19m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:48:37.233 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (5m) 112s ago 20m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:48:37.233 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (5m) 82s ago 19m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:48:37.233 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (8m) 112s ago 19m 87.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:48:37.233 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (2m) 112s ago 14m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T05:48:37.233 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (117s) 112s ago 14m 13.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T05:48:37.233 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (85s) 82s ago 14m 11.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 6d4aca6f74a0 2024-04-12T05:48:37.233 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (101s) 82s ago 14m 14.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T05:48:37.233 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (10m) 112s ago 21m 507M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:48:37.233 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (9m) 82s ago 18m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:48:37.233 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (7m) 112s ago 21m 53.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:48:37.233 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (6m) 82s ago 18m 43.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:48:37.233 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (9m) 112s ago 20m 16.9M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:48:37.233 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (9m) 82s ago 18m 18.4M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:48:37.233 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (5m) 112s ago 18m 91.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T05:48:37.233 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (4m) 112s ago 17m 97.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T05:48:37.234 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (4m) 112s ago 17m 81.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T05:48:37.234 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (3m) 82s ago 16m 113M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T05:48:37.234 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (3m) 82s ago 16m 90.3M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T05:48:37.234 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (2m) 82s ago 15m 84.8M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T05:48:37.234 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (8m) 112s ago 19m 64.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:48:37.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.238+0000 7f48dbc0d700 1 -- 172.21.15.32:0/3377339227 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f48c006cc30 msgr2=0x7f48c006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:37.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.238+0000 7f48dbc0d700 1 --2- 172.21.15.32:0/3377339227 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f48c006cc30 0x7f48c006f0e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f48d4109f80 tx=0x7f48d0009e30 comp rx=0 tx=0).stop 2024-04-12T05:48:37.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.238+0000 7f48dbc0d700 1 -- 172.21.15.32:0/3377339227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48d4101d70 msgr2=0x7f48d4108690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:37.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.238+0000 7f48dbc0d700 1 --2- 172.21.15.32:0/3377339227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48d4101d70 0x7f48d4108690 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f48c4002a50 tx=0x7f48c40046b0 comp rx=0 tx=0).stop 2024-04-12T05:48:37.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.238+0000 7f48dbc0d700 1 -- 172.21.15.32:0/3377339227 shutdown_connections 2024-04-12T05:48:37.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.238+0000 7f48dbc0d700 1 --2- 172.21.15.32:0/3377339227 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f48c006cc30 0x7f48c006f0e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:37.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.238+0000 7f48dbc0d700 1 --2- 172.21.15.32:0/3377339227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48d4101d70 0x7f48d4108690 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:37.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.238+0000 7f48dbc0d700 1 --2- 172.21.15.32:0/3377339227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d4102710 0x7f48d4108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:37.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.238+0000 7f48dbc0d700 1 -- 172.21.15.32:0/3377339227 >> 172.21.15.32:0/3377339227 conn(0x7f48d4072bc0 msgr2=0x7f48d40fac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:37.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.238+0000 7f48dbc0d700 1 -- 172.21.15.32:0/3377339227 shutdown_connections 2024-04-12T05:48:37.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.238+0000 7f48dbc0d700 1 -- 172.21.15.32:0/3377339227 wait complete. 2024-04-12T05:48:37.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.340+0000 7f44b5bc8700 1 -- 172.21.15.32:0/618935409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44b0101d10 msgr2=0x7f44b01020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:37.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.340+0000 7f44b5bc8700 1 --2- 172.21.15.32:0/618935409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44b0101d10 0x7f44b01020e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f44a0004940 tx=0x7f44a0009e30 comp rx=0 tx=0).stop 2024-04-12T05:48:37.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.340+0000 7f44b5bc8700 1 -- 172.21.15.32:0/618935409 shutdown_connections 2024-04-12T05:48:37.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.340+0000 7f44b5bc8700 1 --2- 172.21.15.32:0/618935409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44b01026b0 0x7f44b00fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:37.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.340+0000 7f44b5bc8700 1 --2- 172.21.15.32:0/618935409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44b0101d10 0x7f44b01020e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:37.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.340+0000 7f44b5bc8700 1 -- 172.21.15.32:0/618935409 >> 172.21.15.32:0/618935409 conn(0x7f44b00f7880 msgr2=0x7f44b00f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:37.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.340+0000 7f44b5bc8700 1 -- 172.21.15.32:0/618935409 shutdown_connections 2024-04-12T05:48:37.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.340+0000 7f44b5bc8700 1 -- 172.21.15.32:0/618935409 wait complete. 2024-04-12T05:48:37.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.341+0000 7f44b5bc8700 1 Processor -- start 2024-04-12T05:48:37.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.341+0000 7f44b5bc8700 1 -- start start 2024-04-12T05:48:37.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.341+0000 7f44b5bc8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44b0101d10 0x7f44b0113df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:37.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.341+0000 7f44b5bc8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44b01026b0 0x7f44b0114330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:37.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.341+0000 7f44b5bc8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b0114980 con 0x7f44b0101d10 2024-04-12T05:48:37.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.341+0000 7f44b5bc8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b010dd50 con 0x7f44b01026b0 2024-04-12T05:48:37.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.341+0000 7f44af7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44b0101d10 0x7f44b0113df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:37.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.341+0000 7f44af7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44b0101d10 0x7f44b0113df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46990/0 (socket says 172.21.15.32:46990) 2024-04-12T05:48:37.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.341+0000 7f44af7fe700 1 -- 172.21.15.32:0/3916368344 learned_addr learned my addr 172.21.15.32:0/3916368344 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:48:37.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.342+0000 7f44aeffd700 1 --2- 172.21.15.32:0/3916368344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44b01026b0 0x7f44b0114330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:37.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.342+0000 7f44af7fe700 1 -- 172.21.15.32:0/3916368344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44b01026b0 msgr2=0x7f44b0114330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:37.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.342+0000 7f44af7fe700 1 --2- 172.21.15.32:0/3916368344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44b01026b0 0x7f44b0114330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:37.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.342+0000 7f44af7fe700 1 -- 172.21.15.32:0/3916368344 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a0005040 con 0x7f44b0101d10 2024-04-12T05:48:37.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.342+0000 7f44af7fe700 1 --2- 172.21.15.32:0/3916368344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44b0101d10 0x7f44b0113df0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f44a0009390 tx=0x7f44a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:37.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.342+0000 7f4497fff700 1 -- 172.21.15.32:0/3916368344 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44a000e040 con 0x7f44b0101d10 2024-04-12T05:48:37.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.342+0000 7f44b5bc8700 1 -- 172.21.15.32:0/3916368344 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44b010dfd0 con 0x7f44b0101d10 2024-04-12T05:48:37.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.342+0000 7f4497fff700 1 -- 172.21.15.32:0/3916368344 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f44a0013420 con 0x7f44b0101d10 2024-04-12T05:48:37.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.342+0000 7f4497fff700 1 -- 172.21.15.32:0/3916368344 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44a001c630 con 0x7f44b0101d10 2024-04-12T05:48:37.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.342+0000 7f44b5bc8700 1 -- 172.21.15.32:0/3916368344 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44b010e590 con 0x7f44b0101d10 2024-04-12T05:48:37.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.343+0000 7f44b5bc8700 1 -- 172.21.15.32:0/3916368344 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44b00fd460 con 0x7f44b0101d10 2024-04-12T05:48:37.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.344+0000 7f4497fff700 1 -- 172.21.15.32:0/3916368344 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f44a00139e0 con 0x7f44b0101d10 2024-04-12T05:48:37.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.344+0000 7f4497fff700 1 --2- 172.21.15.32:0/3916368344 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f449806cc30 0x7f449806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:37.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.344+0000 7f4497fff700 1 -- 172.21.15.32:0/3916368344 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f44a008d310 con 0x7f44b0101d10 2024-04-12T05:48:37.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.344+0000 7f44aeffd700 1 --2- 172.21.15.32:0/3916368344 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f449806cc30 0x7f449806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:37.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.345+0000 7f44aeffd700 1 --2- 172.21.15.32:0/3916368344 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f449806cc30 0x7f449806f0e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f44b010f690 tx=0x7f44a4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:37.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.349+0000 7f4497fff700 1 -- 172.21.15.32:0/3916368344 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f44a005ae30 con 0x7f44b0101d10 2024-04-12T05:48:37.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.636+0000 7f44b5bc8700 1 -- 172.21.15.32:0/3916368344 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f44b004bed0 con 0x7f44b0101d10 2024-04-12T05:48:37.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.636+0000 7f4497fff700 1 -- 172.21.15.32:0/3916368344 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f44a005a9c0 con 0x7f44b0101d10 2024-04-12T05:48:37.632 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:48:37.632 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:48:37.633 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:48:37.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:48:37.633 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:48:37.633 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:48:37.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:48:37.633 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:48:37.633 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:48:37.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:48:37.633 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:48:37.633 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:48:37.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:48:37.633 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:48:37.633 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:48:37.633 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:48:37.634 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:48:37.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.638+0000 7f44b5bc8700 1 -- 172.21.15.32:0/3916368344 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f449806cc30 msgr2=0x7f449806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:37.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.638+0000 7f44b5bc8700 1 --2- 172.21.15.32:0/3916368344 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f449806cc30 0x7f449806f0e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f44b010f690 tx=0x7f44a4009040 comp rx=0 tx=0).stop 2024-04-12T05:48:37.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.638+0000 7f44b5bc8700 1 -- 172.21.15.32:0/3916368344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44b0101d10 msgr2=0x7f44b0113df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:37.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.638+0000 7f44b5bc8700 1 --2- 172.21.15.32:0/3916368344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44b0101d10 0x7f44b0113df0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f44a0009390 tx=0x7f44a0009e30 comp rx=0 tx=0).stop 2024-04-12T05:48:37.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.638+0000 7f44b5bc8700 1 -- 172.21.15.32:0/3916368344 shutdown_connections 2024-04-12T05:48:37.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.638+0000 7f44b5bc8700 1 --2- 172.21.15.32:0/3916368344 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f449806cc30 0x7f449806f0e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:37.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.638+0000 7f44b5bc8700 1 --2- 172.21.15.32:0/3916368344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44b0101d10 0x7f44b0113df0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:37.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.638+0000 7f44b5bc8700 1 --2- 172.21.15.32:0/3916368344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44b01026b0 0x7f44b0114330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:37.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.638+0000 7f44b5bc8700 1 -- 172.21.15.32:0/3916368344 >> 172.21.15.32:0/3916368344 conn(0x7f44b00f7880 msgr2=0x7f44b00f8920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:37.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.638+0000 7f44b5bc8700 1 -- 172.21.15.32:0/3916368344 shutdown_connections 2024-04-12T05:48:37.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.638+0000 7f44b5bc8700 1 -- 172.21.15.32:0/3916368344 wait complete. 2024-04-12T05:48:37.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.742+0000 7f4e0bf7e700 1 -- 172.21.15.32:0/1425764605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e041014f0 msgr2=0x7f4e041018c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:37.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.742+0000 7f4e0bf7e700 1 --2- 172.21.15.32:0/1425764605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e041014f0 0x7f4e041018c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4df8004940 tx=0x7f4df8009e30 comp rx=0 tx=0).stop 2024-04-12T05:48:37.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.742+0000 7f4e0bf7e700 1 -- 172.21.15.32:0/1425764605 shutdown_connections 2024-04-12T05:48:37.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.742+0000 7f4e0bf7e700 1 --2- 172.21.15.32:0/1425764605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e040fb4a0 0x7f4e040fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:37.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.742+0000 7f4e0bf7e700 1 --2- 172.21.15.32:0/1425764605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e041014f0 0x7f4e041018c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:37.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.742+0000 7f4e0bf7e700 1 -- 172.21.15.32:0/1425764605 >> 172.21.15.32:0/1425764605 conn(0x7f4e040714c0 msgr2=0x7f4e040718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:37.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.742+0000 7f4e0bf7e700 1 -- 172.21.15.32:0/1425764605 shutdown_connections 2024-04-12T05:48:37.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.742+0000 7f4e0bf7e700 1 -- 172.21.15.32:0/1425764605 wait complete. 2024-04-12T05:48:37.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.743+0000 7f4e0bf7e700 1 Processor -- start 2024-04-12T05:48:37.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.743+0000 7f4e0bf7e700 1 -- start start 2024-04-12T05:48:37.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.743+0000 7f4e0bf7e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e040fb4a0 0x7f4e0410e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:37.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.743+0000 7f4e0bf7e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e041014f0 0x7f4e0410eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:37.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.743+0000 7f4e0bf7e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0410f540 con 0x7f4e040fb4a0 2024-04-12T05:48:37.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.743+0000 7f4e0bf7e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0410f680 con 0x7f4e041014f0 2024-04-12T05:48:37.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.743+0000 7f4e09d1a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e040fb4a0 0x7f4e0410e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:37.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.743+0000 7f4e09d1a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e040fb4a0 0x7f4e0410e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47012/0 (socket says 172.21.15.32:47012) 2024-04-12T05:48:37.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.743+0000 7f4e09d1a700 1 -- 172.21.15.32:0/2484262285 learned_addr learned my addr 172.21.15.32:0/2484262285 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:48:37.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.743+0000 7f4e09519700 1 --2- 172.21.15.32:0/2484262285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e041014f0 0x7f4e0410eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:37.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.743+0000 7f4e09d1a700 1 -- 172.21.15.32:0/2484262285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e041014f0 msgr2=0x7f4e0410eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:37.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.743+0000 7f4e09d1a700 1 --2- 172.21.15.32:0/2484262285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e041014f0 0x7f4e0410eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:37.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.743+0000 7f4e09d1a700 1 -- 172.21.15.32:0/2484262285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df8005040 con 0x7f4e040fb4a0 2024-04-12T05:48:37.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.743+0000 7f4e09519700 1 --2- 172.21.15.32:0/2484262285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e041014f0 0x7f4e0410eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:48:37.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.744+0000 7f4e09d1a700 1 --2- 172.21.15.32:0/2484262285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e040fb4a0 0x7f4e0410e9b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4df800b3b0 tx=0x7f4df8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:37.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.744+0000 7f4df67fc700 1 -- 172.21.15.32:0/2484262285 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4df800e040 con 0x7f4e040fb4a0 2024-04-12T05:48:37.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.744+0000 7f4df67fc700 1 -- 172.21.15.32:0/2484262285 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4df8013420 con 0x7f4e040fb4a0 2024-04-12T05:48:37.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.744+0000 7f4e0bf7e700 1 -- 172.21.15.32:0/2484262285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e04117a20 con 0x7f4e040fb4a0 2024-04-12T05:48:37.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.744+0000 7f4df67fc700 1 -- 172.21.15.32:0/2484262285 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4df801c630 con 0x7f4e040fb4a0 2024-04-12T05:48:37.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.744+0000 7f4e0bf7e700 1 -- 172.21.15.32:0/2484262285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e04117f30 con 0x7f4e040fb4a0 2024-04-12T05:48:37.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.745+0000 7f4e0bf7e700 1 -- 172.21.15.32:0/2484262285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e040fcbd0 con 0x7f4e040fb4a0 2024-04-12T05:48:37.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.745+0000 7f4df67fc700 1 -- 172.21.15.32:0/2484262285 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4df8013970 con 0x7f4e040fb4a0 2024-04-12T05:48:37.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.746+0000 7f4df67fc700 1 --2- 172.21.15.32:0/2484262285 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f4df006cc20 0x7f4df006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:37.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.746+0000 7f4df67fc700 1 -- 172.21.15.32:0/2484262285 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f4df808d2d0 con 0x7f4e040fb4a0 2024-04-12T05:48:37.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.746+0000 7f4e09519700 1 --2- 172.21.15.32:0/2484262285 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f4df006cc20 0x7f4df006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:37.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.746+0000 7f4e09519700 1 --2- 172.21.15.32:0/2484262285 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f4df006cc20 0x7f4df006f0d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4e00007fd0 tx=0x7f4e00009f60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:37.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.750+0000 7f4df67fc700 1 -- 172.21.15.32:0/2484262285 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4df80572b0 con 0x7f4e040fb4a0 2024-04-12T05:48:37.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:37.999+0000 7f4e0bf7e700 1 -- 172.21.15.32:0/2484262285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4e0404bf00 con 0x7f4e040fb4a0 2024-04-12T05:48:37.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.000+0000 7f4df67fc700 1 -- 172.21.15.32:0/2484262285 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1835 (secure 0 0 0) 0x7f4df805a8d0 con 0x7f4e040fb4a0 2024-04-12T05:48:37.996 INFO:teuthology.orchestra.run.smithi032.stdout:e33 2024-04-12T05:48:37.996 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:48:37.996 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:48:37.997 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:48:37.997 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:48:37.997 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:48:37.997 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:48:37.997 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 24 2024-04-12T05:48:37.997 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:48:37.997 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:48:37.997 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:46:10.668569+0000 2024-04-12T05:48:37.997 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:48:37.997 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:48:37.997 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:48:37.997 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:48:37.997 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:48:37.997 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:48:37.997 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:48:37.998 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:48:37.998 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T05:48:37.998 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:48:37.998 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:48:37.998 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:48:37.998 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T05:48:37.998 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-12T05:48:37.998 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:48:37.998 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:48:37.998 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:48:37.998 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:48:37.998 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:48:37.998 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:48:37.999 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:48:37.999 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:48:37.999 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:48:37.999 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:48:37.999 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-12T05:48:37.999 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:48:37.999 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:48:37.999 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:48:37.999 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{-1:44317} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:48:37.999 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:48:38.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.002+0000 7f4e0bf7e700 1 -- 172.21.15.32:0/2484262285 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f4df006cc20 msgr2=0x7f4df006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:38.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.002+0000 7f4e0bf7e700 1 --2- 172.21.15.32:0/2484262285 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f4df006cc20 0x7f4df006f0d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4e00007fd0 tx=0x7f4e00009f60 comp rx=0 tx=0).stop 2024-04-12T05:48:38.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.002+0000 7f4e0bf7e700 1 -- 172.21.15.32:0/2484262285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e040fb4a0 msgr2=0x7f4e0410e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:38.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.002+0000 7f4e0bf7e700 1 --2- 172.21.15.32:0/2484262285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e040fb4a0 0x7f4e0410e9b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4df800b3b0 tx=0x7f4df8009e30 comp rx=0 tx=0).stop 2024-04-12T05:48:38.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.002+0000 7f4e0bf7e700 1 -- 172.21.15.32:0/2484262285 shutdown_connections 2024-04-12T05:48:38.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.002+0000 7f4e0bf7e700 1 --2- 172.21.15.32:0/2484262285 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f4df006cc20 0x7f4df006f0d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:38.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.002+0000 7f4e0bf7e700 1 --2- 172.21.15.32:0/2484262285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e040fb4a0 0x7f4e0410e9b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:38.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.002+0000 7f4e0bf7e700 1 --2- 172.21.15.32:0/2484262285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e041014f0 0x7f4e0410eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:38.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.002+0000 7f4e0bf7e700 1 -- 172.21.15.32:0/2484262285 >> 172.21.15.32:0/2484262285 conn(0x7f4e040714c0 msgr2=0x7f4e040f9490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:38.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.002+0000 7f4e0bf7e700 1 -- 172.21.15.32:0/2484262285 shutdown_connections 2024-04-12T05:48:38.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.002+0000 7f4e0bf7e700 1 -- 172.21.15.32:0/2484262285 wait complete. 2024-04-12T05:48:38.001 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-12T05:48:38.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:37 smithi067 ceph-mon[112979]: pgmap v243: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:38.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:37 smithi067 ceph-mon[112979]: from='client.34508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:38.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:37 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3916368344' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:48:38.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:37 smithi032 ceph-mon[146817]: pgmap v243: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:38.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:37 smithi032 ceph-mon[146817]: from='client.34508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:38.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:37 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3916368344' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:48:38.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.107+0000 7f488ae47700 1 -- 172.21.15.32:0/3177560381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48840fe7a0 msgr2=0x7f4884106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:38.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.107+0000 7f488ae47700 1 --2- 172.21.15.32:0/3177560381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48840fe7a0 0x7f4884106e70 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4870004940 tx=0x7f4870009e30 comp rx=0 tx=0).stop 2024-04-12T05:48:38.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.107+0000 7f488ae47700 1 -- 172.21.15.32:0/3177560381 shutdown_connections 2024-04-12T05:48:38.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.107+0000 7f488ae47700 1 --2- 172.21.15.32:0/3177560381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48840fe7a0 0x7f4884106e70 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:38.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.107+0000 7f488ae47700 1 --2- 172.21.15.32:0/3177560381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48840fde00 0x7f48840fe1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:38.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.107+0000 7f488ae47700 1 -- 172.21.15.32:0/3177560381 >> 172.21.15.32:0/3177560381 conn(0x7f48840f7880 msgr2=0x7f48840f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:38.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.107+0000 7f488ae47700 1 -- 172.21.15.32:0/3177560381 shutdown_connections 2024-04-12T05:48:38.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.107+0000 7f488ae47700 1 -- 172.21.15.32:0/3177560381 wait complete. 2024-04-12T05:48:38.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.107+0000 7f488ae47700 1 Processor -- start 2024-04-12T05:48:38.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.107+0000 7f488ae47700 1 -- start start 2024-04-12T05:48:38.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.108+0000 7f488ae47700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48840fde00 0x7f4884113db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:38.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.108+0000 7f488ae47700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48840fe7a0 0x7f48841142f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:38.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.108+0000 7f488ae47700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4884114940 con 0x7f48840fe7a0 2024-04-12T05:48:38.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.108+0000 7f488ae47700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488410dd10 con 0x7f48840fde00 2024-04-12T05:48:38.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.108+0000 7f4883fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48840fe7a0 0x7f48841142f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:38.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.108+0000 7f4883fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48840fe7a0 0x7f48841142f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47040/0 (socket says 172.21.15.32:47040) 2024-04-12T05:48:38.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.108+0000 7f4883fff700 1 -- 172.21.15.32:0/360818042 learned_addr learned my addr 172.21.15.32:0/360818042 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:48:38.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.108+0000 7f4888be3700 1 --2- 172.21.15.32:0/360818042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48840fde00 0x7f4884113db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:38.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.108+0000 7f4883fff700 1 -- 172.21.15.32:0/360818042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48840fde00 msgr2=0x7f4884113db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:38.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.108+0000 7f4883fff700 1 --2- 172.21.15.32:0/360818042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48840fde00 0x7f4884113db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:38.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.108+0000 7f4883fff700 1 -- 172.21.15.32:0/360818042 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4870005040 con 0x7f48840fe7a0 2024-04-12T05:48:38.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.108+0000 7f4883fff700 1 --2- 172.21.15.32:0/360818042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48840fe7a0 0x7f48841142f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4870009390 tx=0x7f4870009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:38.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.109+0000 7f48817fa700 1 -- 172.21.15.32:0/360818042 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f487000e040 con 0x7f48840fe7a0 2024-04-12T05:48:38.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.109+0000 7f488ae47700 1 -- 172.21.15.32:0/360818042 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f488410df90 con 0x7f48840fe7a0 2024-04-12T05:48:38.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.109+0000 7f48817fa700 1 -- 172.21.15.32:0/360818042 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4870013420 con 0x7f48840fe7a0 2024-04-12T05:48:38.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.109+0000 7f48817fa700 1 -- 172.21.15.32:0/360818042 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f487001c630 con 0x7f48840fe7a0 2024-04-12T05:48:38.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.109+0000 7f488ae47700 1 -- 172.21.15.32:0/360818042 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f488410e4d0 con 0x7f48840fe7a0 2024-04-12T05:48:38.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.110+0000 7f488ae47700 1 -- 172.21.15.32:0/360818042 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48840666f0 con 0x7f48840fe7a0 2024-04-12T05:48:38.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.110+0000 7f48817fa700 1 -- 172.21.15.32:0/360818042 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4870013a20 con 0x7f48840fe7a0 2024-04-12T05:48:38.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.110+0000 7f48817fa700 1 --2- 172.21.15.32:0/360818042 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f487406cc30 0x7f487406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:38.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.110+0000 7f48817fa700 1 -- 172.21.15.32:0/360818042 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f487008d260 con 0x7f48840fe7a0 2024-04-12T05:48:38.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.110+0000 7f4888be3700 1 --2- 172.21.15.32:0/360818042 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f487406cc30 0x7f487406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:38.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.111+0000 7f4888be3700 1 --2- 172.21.15.32:0/360818042 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f487406cc30 0x7f487406f0e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4878004f40 tx=0x7f487800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:38.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.115+0000 7f48817fa700 1 -- 172.21.15.32:0/360818042 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f487005acd0 con 0x7f48840fe7a0 2024-04-12T05:48:38.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.333+0000 7f488ae47700 1 -- 172.21.15.32:0/360818042 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4884109dc0 con 0x7f487406cc30 2024-04-12T05:48:38.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.334+0000 7f48817fa700 1 -- 172.21.15.32:0/360818042 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4884109dc0 con 0x7f487406cc30 2024-04-12T05:48:38.330 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:48:38.330 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:48:38.331 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:48:38.331 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:48:38.331 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:48:38.331 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-12T05:48:38.331 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:48:38.331 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-12T05:48:38.331 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:48:38.331 INFO:teuthology.orchestra.run.smithi032.stdout: "mds" 2024-04-12T05:48:38.331 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:48:38.331 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:48:38.331 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:48:38.331 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:48:38.331 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:48:38.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.336+0000 7f488ae47700 1 -- 172.21.15.32:0/360818042 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f487406cc30 msgr2=0x7f487406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:38.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.336+0000 7f488ae47700 1 --2- 172.21.15.32:0/360818042 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f487406cc30 0x7f487406f0e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4878004f40 tx=0x7f487800b040 comp rx=0 tx=0).stop 2024-04-12T05:48:38.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.336+0000 7f488ae47700 1 -- 172.21.15.32:0/360818042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48840fe7a0 msgr2=0x7f48841142f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:38.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.336+0000 7f488ae47700 1 --2- 172.21.15.32:0/360818042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48840fe7a0 0x7f48841142f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4870009390 tx=0x7f4870009e30 comp rx=0 tx=0).stop 2024-04-12T05:48:38.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.336+0000 7f488ae47700 1 -- 172.21.15.32:0/360818042 shutdown_connections 2024-04-12T05:48:38.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.336+0000 7f488ae47700 1 --2- 172.21.15.32:0/360818042 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f487406cc30 0x7f487406f0e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:38.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.336+0000 7f488ae47700 1 --2- 172.21.15.32:0/360818042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48840fde00 0x7f4884113db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:38.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.336+0000 7f488ae47700 1 --2- 172.21.15.32:0/360818042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48840fe7a0 0x7f48841142f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:38.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.336+0000 7f488ae47700 1 -- 172.21.15.32:0/360818042 >> 172.21.15.32:0/360818042 conn(0x7f48840f7880 msgr2=0x7f48840f8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:38.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.336+0000 7f488ae47700 1 -- 172.21.15.32:0/360818042 shutdown_connections 2024-04-12T05:48:38.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.336+0000 7f488ae47700 1 -- 172.21.15.32:0/360818042 wait complete. 2024-04-12T05:48:38.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.440+0000 7f434b0f4700 1 -- 172.21.15.32:0/482425037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43441037b0 msgr2=0x7f43440fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:38.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.440+0000 7f434b0f4700 1 --2- 172.21.15.32:0/482425037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43441037b0 0x7f43440fce20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4330004940 tx=0x7f4330009e30 comp rx=0 tx=0).stop 2024-04-12T05:48:38.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.440+0000 7f434b0f4700 1 -- 172.21.15.32:0/482425037 shutdown_connections 2024-04-12T05:48:38.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.440+0000 7f434b0f4700 1 --2- 172.21.15.32:0/482425037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43441037b0 0x7f43440fce20 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:38.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.440+0000 7f434b0f4700 1 --2- 172.21.15.32:0/482425037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4344102e10 0x7f43441031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:38.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.440+0000 7f434b0f4700 1 -- 172.21.15.32:0/482425037 >> 172.21.15.32:0/482425037 conn(0x7f43440f89a0 msgr2=0x7f43440fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:38.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.440+0000 7f434b0f4700 1 -- 172.21.15.32:0/482425037 shutdown_connections 2024-04-12T05:48:38.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.440+0000 7f434b0f4700 1 -- 172.21.15.32:0/482425037 wait complete. 2024-04-12T05:48:38.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.440+0000 7f434b0f4700 1 Processor -- start 2024-04-12T05:48:38.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.440+0000 7f434b0f4700 1 -- start start 2024-04-12T05:48:38.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.441+0000 7f434b0f4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4344102e10 0x7f4344113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:38.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.441+0000 7f434b0f4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43441037b0 0x7f4344114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:38.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.441+0000 7f434b0f4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4344114660 con 0x7f43441037b0 2024-04-12T05:48:38.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.441+0000 7f434b0f4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434410da50 con 0x7f4344102e10 2024-04-12T05:48:38.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.441+0000 7f4343fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43441037b0 0x7f4344114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:38.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.441+0000 7f4343fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43441037b0 0x7f4344114010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47064/0 (socket says 172.21.15.32:47064) 2024-04-12T05:48:38.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.441+0000 7f4343fff700 1 -- 172.21.15.32:0/13153728 learned_addr learned my addr 172.21.15.32:0/13153728 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:48:38.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.441+0000 7f4348e90700 1 --2- 172.21.15.32:0/13153728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4344102e10 0x7f4344113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:38.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.441+0000 7f4343fff700 1 -- 172.21.15.32:0/13153728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4344102e10 msgr2=0x7f4344113ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:38.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.441+0000 7f4343fff700 1 --2- 172.21.15.32:0/13153728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4344102e10 0x7f4344113ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:38.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.441+0000 7f4343fff700 1 -- 172.21.15.32:0/13153728 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4330005040 con 0x7f43441037b0 2024-04-12T05:48:38.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.441+0000 7f4343fff700 1 --2- 172.21.15.32:0/13153728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43441037b0 0x7f4344114010 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4330009390 tx=0x7f4330009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:38.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.441+0000 7f43417fa700 1 -- 172.21.15.32:0/13153728 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f433000e040 con 0x7f43441037b0 2024-04-12T05:48:38.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.441+0000 7f43417fa700 1 -- 172.21.15.32:0/13153728 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4330013420 con 0x7f43441037b0 2024-04-12T05:48:38.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.441+0000 7f434b0f4700 1 -- 172.21.15.32:0/13153728 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f434410dcd0 con 0x7f43441037b0 2024-04-12T05:48:38.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.441+0000 7f43417fa700 1 -- 172.21.15.32:0/13153728 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f433001c630 con 0x7f43441037b0 2024-04-12T05:48:38.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.442+0000 7f434b0f4700 1 -- 172.21.15.32:0/13153728 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f434410e290 con 0x7f43441037b0 2024-04-12T05:48:38.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.443+0000 7f434b0f4700 1 -- 172.21.15.32:0/13153728 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43440fe560 con 0x7f43441037b0 2024-04-12T05:48:38.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.443+0000 7f43417fa700 1 -- 172.21.15.32:0/13153728 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f43300139e0 con 0x7f43441037b0 2024-04-12T05:48:38.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.444+0000 7f43417fa700 1 --2- 172.21.15.32:0/13153728 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f433406cc30 0x7f433406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:48:38.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.444+0000 7f4348e90700 1 --2- 172.21.15.32:0/13153728 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f433406cc30 0x7f433406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:38.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.444+0000 7f43417fa700 1 -- 172.21.15.32:0/13153728 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f433008d350 con 0x7f43441037b0 2024-04-12T05:48:38.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.444+0000 7f4348e90700 1 --2- 172.21.15.32:0/13153728 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f433406cc30 0x7f433406f0e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4338004990 tx=0x7f4338009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:38.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.448+0000 7f43417fa700 1 -- 172.21.15.32:0/13153728 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f433005adc0 con 0x7f43441037b0 2024-04-12T05:48:38.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.737+0000 7f434b0f4700 1 -- 172.21.15.32:0/13153728 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4344109db0 con 0x7f43441037b0 2024-04-12T05:48:38.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.737+0000 7f43417fa700 1 -- 172.21.15.32:0/13153728 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f433005a950 con 0x7f43441037b0 2024-04-12T05:48:38.733 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:48:38.733 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:48:38.734 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-12T05:48:38.734 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:48:38.734 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:48:38.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.739+0000 7f434b0f4700 1 -- 172.21.15.32:0/13153728 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f433406cc30 msgr2=0x7f433406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:38.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.739+0000 7f434b0f4700 1 --2- 172.21.15.32:0/13153728 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f433406cc30 0x7f433406f0e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4338004990 tx=0x7f4338009ee0 comp rx=0 tx=0).stop 2024-04-12T05:48:38.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.739+0000 7f434b0f4700 1 -- 172.21.15.32:0/13153728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43441037b0 msgr2=0x7f4344114010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:38.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.739+0000 7f434b0f4700 1 --2- 172.21.15.32:0/13153728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43441037b0 0x7f4344114010 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4330009390 tx=0x7f4330009e30 comp rx=0 tx=0).stop 2024-04-12T05:48:38.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.739+0000 7f434b0f4700 1 -- 172.21.15.32:0/13153728 shutdown_connections 2024-04-12T05:48:38.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.739+0000 7f434b0f4700 1 --2- 172.21.15.32:0/13153728 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f433406cc30 0x7f433406f0e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:38.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.739+0000 7f434b0f4700 1 --2- 172.21.15.32:0/13153728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4344102e10 0x7f4344113ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:38.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.739+0000 7f434b0f4700 1 --2- 172.21.15.32:0/13153728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43441037b0 0x7f4344114010 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:48:38.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.739+0000 7f434b0f4700 1 -- 172.21.15.32:0/13153728 >> 172.21.15.32:0/13153728 conn(0x7f43440f89a0 msgr2=0x7f43440f9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:38.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.739+0000 7f434b0f4700 1 -- 172.21.15.32:0/13153728 shutdown_connections 2024-04-12T05:48:38.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:48:38.739+0000 7f434b0f4700 1 -- 172.21.15.32:0/13153728 wait complete. 2024-04-12T05:48:39.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:38 smithi067 ceph-mon[112979]: from='client.34512 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:39.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:38 smithi067 ceph-mon[112979]: from='client.34516 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:39.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:38 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/2484262285' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:48:39.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:38 smithi032 ceph-mon[146817]: from='client.34512 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:39.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:38 smithi032 ceph-mon[146817]: from='client.34516 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:39.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:38 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/2484262285' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:48:40.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:39 smithi067 ceph-mon[112979]: pgmap v244: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:40.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:39 smithi067 ceph-mon[112979]: from='client.34528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:40.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:39 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/13153728' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:48:40.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:39 smithi032 ceph-mon[146817]: pgmap v244: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:40.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:39 smithi032 ceph-mon[146817]: from='client.34528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:40.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:39 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/13153728' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:48:42.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:41 smithi067 ceph-mon[112979]: pgmap v245: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:42.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:41 smithi032 ceph-mon[146817]: pgmap v245: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:44.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:43 smithi067 ceph-mon[112979]: pgmap v246: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:44.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:43 smithi032 ceph-mon[146817]: pgmap v246: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:46.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:45 smithi067 ceph-mon[112979]: pgmap v247: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:46.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:45 smithi032 ceph-mon[146817]: pgmap v247: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:48.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:47 smithi067 ceph-mon[112979]: pgmap v248: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:48.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:47 smithi032 ceph-mon[146817]: pgmap v248: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:49.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:48 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:48:49.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:48 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:48:50.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:49 smithi067 ceph-mon[112979]: pgmap v249: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:50.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:49 smithi032 ceph-mon[146817]: pgmap v249: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:52.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:51 smithi067 ceph-mon[112979]: pgmap v250: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:52.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:51 smithi032 ceph-mon[146817]: pgmap v250: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:54.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:53 smithi067 ceph-mon[112979]: pgmap v251: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:54.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:53 smithi032 ceph-mon[146817]: pgmap v251: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:56.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:55 smithi067 ceph-mon[112979]: pgmap v252: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:56.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:55 smithi032 ceph-mon[146817]: pgmap v252: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:58.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:57 smithi067 ceph-mon[112979]: pgmap v253: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:58.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:57 smithi032 ceph-mon[146817]: pgmap v253: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:00.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:48:59 smithi067 ceph-mon[112979]: pgmap v254: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:00.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:48:59 smithi032 ceph-mon[146817]: pgmap v254: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:02.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:01 smithi067 ceph-mon[112979]: pgmap v255: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:02.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:01 smithi032 ceph-mon[146817]: pgmap v255: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:04.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:03 smithi067 ceph-mon[112979]: pgmap v256: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:04.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:49:04.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:03 smithi032 ceph-mon[146817]: pgmap v256: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:04.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:49:06.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:05 smithi067 ceph-mon[112979]: pgmap v257: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:06.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:05 smithi032 ceph-mon[146817]: pgmap v257: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:08.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:07 smithi067 ceph-mon[112979]: pgmap v258: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:08.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:07 smithi032 ceph-mon[146817]: pgmap v258: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:08.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.857+0000 7fab97396700 1 -- 172.21.15.32:0/2857670380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab90102e10 msgr2=0x7fab901031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:08.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.857+0000 7fab97396700 1 --2- 172.21.15.32:0/2857670380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab90102e10 0x7fab901031e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fab80004940 tx=0x7fab80009e30 comp rx=0 tx=0).stop 2024-04-12T05:49:08.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.857+0000 7fab97396700 1 -- 172.21.15.32:0/2857670380 shutdown_connections 2024-04-12T05:49:08.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.857+0000 7fab97396700 1 --2- 172.21.15.32:0/2857670380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab901037b0 0x7fab900fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:08.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.857+0000 7fab97396700 1 --2- 172.21.15.32:0/2857670380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab90102e10 0x7fab901031e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:08.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.857+0000 7fab97396700 1 -- 172.21.15.32:0/2857670380 >> 172.21.15.32:0/2857670380 conn(0x7fab900f8980 msgr2=0x7fab900fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:08.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.858+0000 7fab97396700 1 -- 172.21.15.32:0/2857670380 shutdown_connections 2024-04-12T05:49:08.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.858+0000 7fab97396700 1 -- 172.21.15.32:0/2857670380 wait complete. 2024-04-12T05:49:08.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.858+0000 7fab97396700 1 Processor -- start 2024-04-12T05:49:08.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.858+0000 7fab97396700 1 -- start start 2024-04-12T05:49:08.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.858+0000 7fab97396700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab90102e10 0x7fab900702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:08.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.858+0000 7fab97396700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab901037b0 0x7fab90070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:08.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.858+0000 7fab97396700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab90070d90 con 0x7fab901037b0 2024-04-12T05:49:08.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.858+0000 7fab97396700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab90070ed0 con 0x7fab90102e10 2024-04-12T05:49:08.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.859+0000 7fab94931700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab901037b0 0x7fab90070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:08.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.859+0000 7fab94931700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab901037b0 0x7fab90070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39464/0 (socket says 172.21.15.32:39464) 2024-04-12T05:49:08.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.859+0000 7fab94931700 1 -- 172.21.15.32:0/3415745419 learned_addr learned my addr 172.21.15.32:0/3415745419 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:49:08.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.859+0000 7fab95132700 1 --2- 172.21.15.32:0/3415745419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab90102e10 0x7fab900702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:08.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.859+0000 7fab94931700 1 -- 172.21.15.32:0/3415745419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab90102e10 msgr2=0x7fab900702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:08.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.859+0000 7fab94931700 1 --2- 172.21.15.32:0/3415745419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab90102e10 0x7fab900702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:08.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.859+0000 7fab94931700 1 -- 172.21.15.32:0/3415745419 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab80005040 con 0x7fab901037b0 2024-04-12T05:49:08.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.859+0000 7fab94931700 1 --2- 172.21.15.32:0/3415745419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab901037b0 0x7fab90070800 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fab8c00c800 tx=0x7fab8c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:08.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.859+0000 7fab85ffb700 1 -- 172.21.15.32:0/3415745419 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab8c0106f0 con 0x7fab901037b0 2024-04-12T05:49:08.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.859+0000 7fab85ffb700 1 -- 172.21.15.32:0/3415745419 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab8c010d00 con 0x7fab901037b0 2024-04-12T05:49:08.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.859+0000 7fab97396700 1 -- 172.21.15.32:0/3415745419 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab90119dd0 con 0x7fab901037b0 2024-04-12T05:49:08.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.859+0000 7fab85ffb700 1 -- 172.21.15.32:0/3415745419 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab8c0113a0 con 0x7fab901037b0 2024-04-12T05:49:08.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.860+0000 7fab97396700 1 -- 172.21.15.32:0/3415745419 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab9011a250 con 0x7fab901037b0 2024-04-12T05:49:08.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.861+0000 7fab97396700 1 -- 172.21.15.32:0/3415745419 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab900fe560 con 0x7fab901037b0 2024-04-12T05:49:08.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.861+0000 7fab85ffb700 1 -- 172.21.15.32:0/3415745419 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fab8c012400 con 0x7fab901037b0 2024-04-12T05:49:08.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.861+0000 7fab85ffb700 1 --2- 172.21.15.32:0/3415745419 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fab7c06cc20 0x7fab7c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:08.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.862+0000 7fab85ffb700 1 -- 172.21.15.32:0/3415745419 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fab8c08c010 con 0x7fab901037b0 2024-04-12T05:49:08.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.862+0000 7fab95132700 1 --2- 172.21.15.32:0/3415745419 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fab7c06cc20 0x7fab7c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:08.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.862+0000 7fab95132700 1 --2- 172.21.15.32:0/3415745419 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fab7c06cc20 0x7fab7c06f0d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fab80004f90 tx=0x7fab80009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:08.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:08.866+0000 7fab85ffb700 1 -- 172.21.15.32:0/3415745419 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fab8c055ff0 con 0x7fab901037b0 2024-04-12T05:49:09.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.088+0000 7fab97396700 1 -- 172.21.15.32:0/3415745419 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fab9004bed0 con 0x7fab7c06cc20 2024-04-12T05:49:09.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.089+0000 7fab85ffb700 1 -- 172.21.15.32:0/3415745419 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fab9004bed0 con 0x7fab7c06cc20 2024-04-12T05:49:09.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.091+0000 7fab97396700 1 -- 172.21.15.32:0/3415745419 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fab7c06cc20 msgr2=0x7fab7c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:09.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.091+0000 7fab97396700 1 --2- 172.21.15.32:0/3415745419 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fab7c06cc20 0x7fab7c06f0d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fab80004f90 tx=0x7fab80009e30 comp rx=0 tx=0).stop 2024-04-12T05:49:09.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.091+0000 7fab97396700 1 -- 172.21.15.32:0/3415745419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab901037b0 msgr2=0x7fab90070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:09.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.091+0000 7fab97396700 1 --2- 172.21.15.32:0/3415745419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab901037b0 0x7fab90070800 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fab8c00c800 tx=0x7fab8c004470 comp rx=0 tx=0).stop 2024-04-12T05:49:09.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.091+0000 7fab97396700 1 -- 172.21.15.32:0/3415745419 shutdown_connections 2024-04-12T05:49:09.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.091+0000 7fab97396700 1 --2- 172.21.15.32:0/3415745419 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fab7c06cc20 0x7fab7c06f0d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:09.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.091+0000 7fab97396700 1 --2- 172.21.15.32:0/3415745419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab90102e10 0x7fab900702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:09.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.091+0000 7fab97396700 1 --2- 172.21.15.32:0/3415745419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab901037b0 0x7fab90070800 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:09.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.091+0000 7fab97396700 1 -- 172.21.15.32:0/3415745419 >> 172.21.15.32:0/3415745419 conn(0x7fab900f8980 msgr2=0x7fab900f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:09.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.092+0000 7fab97396700 1 -- 172.21.15.32:0/3415745419 shutdown_connections 2024-04-12T05:49:09.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.092+0000 7fab97396700 1 -- 172.21.15.32:0/3415745419 wait complete. 2024-04-12T05:49:09.098 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:49:09.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.196+0000 7fe0e96a1700 1 -- 172.21.15.32:0/3929997890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0e40fe830 msgr2=0x7fe0e4106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:09.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.196+0000 7fe0e96a1700 1 --2- 172.21.15.32:0/3929997890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0e40fe830 0x7fe0e4106f00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4004940 tx=0x7fe0d4009e30 comp rx=0 tx=0).stop 2024-04-12T05:49:09.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.196+0000 7fe0e96a1700 1 -- 172.21.15.32:0/3929997890 shutdown_connections 2024-04-12T05:49:09.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.196+0000 7fe0e96a1700 1 --2- 172.21.15.32:0/3929997890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0e40fe830 0x7fe0e4106f00 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:09.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.196+0000 7fe0e96a1700 1 --2- 172.21.15.32:0/3929997890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0e40fde90 0x7fe0e40fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:09.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.196+0000 7fe0e96a1700 1 -- 172.21.15.32:0/3929997890 >> 172.21.15.32:0/3929997890 conn(0x7fe0e4072bc0 msgr2=0x7fe0e4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:09.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.196+0000 7fe0e96a1700 1 -- 172.21.15.32:0/3929997890 shutdown_connections 2024-04-12T05:49:09.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.196+0000 7fe0e96a1700 1 -- 172.21.15.32:0/3929997890 wait complete. 2024-04-12T05:49:09.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.197+0000 7fe0e96a1700 1 Processor -- start 2024-04-12T05:49:09.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.197+0000 7fe0e96a1700 1 -- start start 2024-04-12T05:49:09.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.197+0000 7fe0e96a1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0e40fde90 0x7fe0e4113a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:09.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.197+0000 7fe0e96a1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0e40fe830 0x7fe0e4113fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:09.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.197+0000 7fe0e96a1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e4114610 con 0x7fe0e40fe830 2024-04-12T05:49:09.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.197+0000 7fe0e96a1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e410da00 con 0x7fe0e40fde90 2024-04-12T05:49:09.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.197+0000 7fe0e27fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0e40fe830 0x7fe0e4113fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:09.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.197+0000 7fe0e27fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0e40fe830 0x7fe0e4113fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39474/0 (socket says 172.21.15.32:39474) 2024-04-12T05:49:09.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.197+0000 7fe0e27fc700 1 -- 172.21.15.32:0/1786678232 learned_addr learned my addr 172.21.15.32:0/1786678232 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:49:09.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.197+0000 7fe0e2ffd700 1 --2- 172.21.15.32:0/1786678232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0e40fde90 0x7fe0e4113a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:09.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.197+0000 7fe0e27fc700 1 -- 172.21.15.32:0/1786678232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0e40fde90 msgr2=0x7fe0e4113a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:09.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.197+0000 7fe0e27fc700 1 --2- 172.21.15.32:0/1786678232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0e40fde90 0x7fe0e4113a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:09.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.197+0000 7fe0e27fc700 1 -- 172.21.15.32:0/1786678232 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d4005040 con 0x7fe0e40fe830 2024-04-12T05:49:09.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.197+0000 7fe0e2ffd700 1 --2- 172.21.15.32:0/1786678232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0e40fde90 0x7fe0e4113a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:49:09.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.198+0000 7fe0e27fc700 1 --2- 172.21.15.32:0/1786678232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0e40fe830 0x7fe0e4113fc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4009360 tx=0x7fe0d4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:09.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.198+0000 7fe0cb7fe700 1 -- 172.21.15.32:0/1786678232 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0d400e040 con 0x7fe0e40fe830 2024-04-12T05:49:09.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.198+0000 7fe0cb7fe700 1 -- 172.21.15.32:0/1786678232 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe0d4013420 con 0x7fe0e40fe830 2024-04-12T05:49:09.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.198+0000 7fe0e96a1700 1 -- 172.21.15.32:0/1786678232 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0e410dc80 con 0x7fe0e40fe830 2024-04-12T05:49:09.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.198+0000 7fe0cb7fe700 1 -- 172.21.15.32:0/1786678232 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0d401c630 con 0x7fe0e40fe830 2024-04-12T05:49:09.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.198+0000 7fe0e96a1700 1 -- 172.21.15.32:0/1786678232 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0e410e1c0 con 0x7fe0e40fe830 2024-04-12T05:49:09.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.199+0000 7fe0e96a1700 1 -- 172.21.15.32:0/1786678232 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0e4101980 con 0x7fe0e40fe830 2024-04-12T05:49:09.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.199+0000 7fe0cb7fe700 1 -- 172.21.15.32:0/1786678232 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe0d40139e0 con 0x7fe0e40fe830 2024-04-12T05:49:09.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.200+0000 7fe0cb7fe700 1 --2- 172.21.15.32:0/1786678232 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe0cc06cc20 0x7fe0cc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:09.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.200+0000 7fe0cb7fe700 1 -- 172.21.15.32:0/1786678232 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fe0d408d210 con 0x7fe0e40fe830 2024-04-12T05:49:09.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.200+0000 7fe0e2ffd700 1 --2- 172.21.15.32:0/1786678232 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe0cc06cc20 0x7fe0cc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:09.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.200+0000 7fe0e2ffd700 1 --2- 172.21.15.32:0/1786678232 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe0cc06cc20 0x7fe0cc06f0d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe0d8009b80 tx=0x7fe0d800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:09.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.204+0000 7fe0cb7fe700 1 -- 172.21.15.32:0/1786678232 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe0d405ad20 con 0x7fe0e40fe830 2024-04-12T05:49:09.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.428+0000 7fe0e96a1700 1 -- 172.21.15.32:0/1786678232 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe0e410f0d0 con 0x7fe0cc06cc20 2024-04-12T05:49:09.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.430+0000 7fe0cb7fe700 1 -- 172.21.15.32:0/1786678232 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe0e410f0d0 con 0x7fe0cc06cc20 2024-04-12T05:49:09.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.432+0000 7fe0e96a1700 1 -- 172.21.15.32:0/1786678232 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe0cc06cc20 msgr2=0x7fe0cc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:09.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.432+0000 7fe0e96a1700 1 --2- 172.21.15.32:0/1786678232 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe0cc06cc20 0x7fe0cc06f0d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe0d8009b80 tx=0x7fe0d800b040 comp rx=0 tx=0).stop 2024-04-12T05:49:09.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.432+0000 7fe0e96a1700 1 -- 172.21.15.32:0/1786678232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0e40fe830 msgr2=0x7fe0e4113fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:09.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.432+0000 7fe0e96a1700 1 --2- 172.21.15.32:0/1786678232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0e40fe830 0x7fe0e4113fc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4009360 tx=0x7fe0d4009e30 comp rx=0 tx=0).stop 2024-04-12T05:49:09.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.432+0000 7fe0e96a1700 1 -- 172.21.15.32:0/1786678232 shutdown_connections 2024-04-12T05:49:09.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.432+0000 7fe0e96a1700 1 --2- 172.21.15.32:0/1786678232 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe0cc06cc20 0x7fe0cc06f0d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:09.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.432+0000 7fe0e96a1700 1 --2- 172.21.15.32:0/1786678232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0e40fde90 0x7fe0e4113a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:09.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.432+0000 7fe0e96a1700 1 --2- 172.21.15.32:0/1786678232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0e40fe830 0x7fe0e4113fc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:09.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.432+0000 7fe0e96a1700 1 -- 172.21.15.32:0/1786678232 >> 172.21.15.32:0/1786678232 conn(0x7fe0e4072bc0 msgr2=0x7fe0e40ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:09.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.432+0000 7fe0e96a1700 1 -- 172.21.15.32:0/1786678232 shutdown_connections 2024-04-12T05:49:09.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.432+0000 7fe0e96a1700 1 -- 172.21.15.32:0/1786678232 wait complete. 2024-04-12T05:49:09.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.535+0000 7fe8f9f17700 1 -- 172.21.15.32:0/2811190798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8f40fe7a0 msgr2=0x7fe8f4106de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:09.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.535+0000 7fe8f9f17700 1 --2- 172.21.15.32:0/2811190798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8f40fe7a0 0x7fe8f4106de0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc004940 tx=0x7fe8dc009e30 comp rx=0 tx=0).stop 2024-04-12T05:49:09.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.535+0000 7fe8f9f17700 1 -- 172.21.15.32:0/2811190798 shutdown_connections 2024-04-12T05:49:09.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.535+0000 7fe8f9f17700 1 --2- 172.21.15.32:0/2811190798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8f40fe7a0 0x7fe8f4106de0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:09.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.535+0000 7fe8f9f17700 1 --2- 172.21.15.32:0/2811190798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f40fde00 0x7fe8f40fe1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:09.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.535+0000 7fe8f9f17700 1 -- 172.21.15.32:0/2811190798 >> 172.21.15.32:0/2811190798 conn(0x7fe8f40f78d0 msgr2=0x7fe8f40f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:09.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.535+0000 7fe8f9f17700 1 -- 172.21.15.32:0/2811190798 shutdown_connections 2024-04-12T05:49:09.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.535+0000 7fe8f9f17700 1 -- 172.21.15.32:0/2811190798 wait complete. 2024-04-12T05:49:09.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.536+0000 7fe8f9f17700 1 Processor -- start 2024-04-12T05:49:09.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.536+0000 7fe8f9f17700 1 -- start start 2024-04-12T05:49:09.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.536+0000 7fe8f9f17700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8f40fde00 0x7fe8f40fbc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:09.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.536+0000 7fe8f9f17700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f40fe7a0 0x7fe8f40fc170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:09.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.536+0000 7fe8f9f17700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f40fc700 con 0x7fe8f40fde00 2024-04-12T05:49:09.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.536+0000 7fe8f9f17700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f40fc840 con 0x7fe8f40fe7a0 2024-04-12T05:49:09.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.536+0000 7fe8f37fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8f40fde00 0x7fe8f40fbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:09.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.536+0000 7fe8f37fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8f40fde00 0x7fe8f40fbc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39502/0 (socket says 172.21.15.32:39502) 2024-04-12T05:49:09.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.536+0000 7fe8f37fe700 1 -- 172.21.15.32:0/875500315 learned_addr learned my addr 172.21.15.32:0/875500315 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:49:09.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.537+0000 7fe8f2ffd700 1 --2- 172.21.15.32:0/875500315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f40fe7a0 0x7fe8f40fc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:09.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.537+0000 7fe8f37fe700 1 -- 172.21.15.32:0/875500315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f40fe7a0 msgr2=0x7fe8f40fc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:09.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.537+0000 7fe8f37fe700 1 --2- 172.21.15.32:0/875500315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f40fe7a0 0x7fe8f40fc170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:09.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.537+0000 7fe8f37fe700 1 -- 172.21.15.32:0/875500315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8dc005040 con 0x7fe8f40fde00 2024-04-12T05:49:09.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.537+0000 7fe8f37fe700 1 --2- 172.21.15.32:0/875500315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8f40fde00 0x7fe8f40fbc30 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe8e400ec90 tx=0x7fe8e400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:09.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.537+0000 7fe8ebfff700 1 -- 172.21.15.32:0/875500315 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8e400eea0 con 0x7fe8f40fde00 2024-04-12T05:49:09.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.537+0000 7fe8f9f17700 1 -- 172.21.15.32:0/875500315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8f4119cf0 con 0x7fe8f40fde00 2024-04-12T05:49:09.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.537+0000 7fe8ebfff700 1 -- 172.21.15.32:0/875500315 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe8e400f040 con 0x7fe8f40fde00 2024-04-12T05:49:09.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.537+0000 7fe8ebfff700 1 -- 172.21.15.32:0/875500315 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8e40095e0 con 0x7fe8f40fde00 2024-04-12T05:49:09.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.537+0000 7fe8f9f17700 1 -- 172.21.15.32:0/875500315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f411a0f0 con 0x7fe8f40fde00 2024-04-12T05:49:09.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.538+0000 7fe8f9f17700 1 -- 172.21.15.32:0/875500315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f4066770 con 0x7fe8f40fde00 2024-04-12T05:49:09.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.539+0000 7fe8ebfff700 1 -- 172.21.15.32:0/875500315 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe8e400c940 con 0x7fe8f40fde00 2024-04-12T05:49:09.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.539+0000 7fe8ebfff700 1 --2- 172.21.15.32:0/875500315 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe8e006cc20 0x7fe8e006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:09.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.539+0000 7fe8f2ffd700 1 --2- 172.21.15.32:0/875500315 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe8e006cc20 0x7fe8e006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:09.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.539+0000 7fe8ebfff700 1 -- 172.21.15.32:0/875500315 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fe8e408c610 con 0x7fe8f40fde00 2024-04-12T05:49:09.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.540+0000 7fe8f2ffd700 1 --2- 172.21.15.32:0/875500315 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe8e006cc20 0x7fe8e006f0d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc00b0d0 tx=0x7fe8dc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:09.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.544+0000 7fe8ebfff700 1 -- 172.21.15.32:0/875500315 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe8e4056570 con 0x7fe8f40fde00 2024-04-12T05:49:09.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.756+0000 7fe8f9f17700 1 -- 172.21.15.32:0/875500315 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe8f40fd320 con 0x7fe8e006cc20 2024-04-12T05:49:09.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.766+0000 7fe8ebfff700 1 -- 172.21.15.32:0/875500315 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fe8f40fd320 con 0x7fe8e006cc20 2024-04-12T05:49:09.762 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:49:09.762 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (8m) 2m ago 20m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:49:09.762 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (6m) 2m ago 20m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:49:09.762 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (6m) 115s ago 19m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:49:09.762 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (8m) 2m ago 20m 87.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:49:09.762 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (2m) 2m ago 14m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T05:49:09.762 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (2m) 2m ago 14m 13.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T05:49:09.762 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (117s) 115s ago 14m 11.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 6d4aca6f74a0 2024-04-12T05:49:09.762 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (2m) 115s ago 14m 14.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T05:49:09.763 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (10m) 2m ago 21m 507M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:49:09.763 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (9m) 115s ago 19m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:49:09.763 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (7m) 2m ago 22m 53.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:49:09.763 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (7m) 115s ago 19m 43.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:49:09.763 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (9m) 2m ago 20m 16.9M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:49:09.763 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (9m) 115s ago 19m 18.4M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:49:09.763 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (6m) 2m ago 18m 91.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T05:49:09.763 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (5m) 2m ago 18m 97.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T05:49:09.763 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (4m) 2m ago 17m 81.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T05:49:09.763 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (4m) 115s ago 17m 113M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T05:49:09.763 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (3m) 115s ago 16m 90.3M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T05:49:09.763 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (3m) 115s ago 16m 84.8M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T05:49:09.763 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (9m) 2m ago 19m 64.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:49:09.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.768+0000 7fe8f9f17700 1 -- 172.21.15.32:0/875500315 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe8e006cc20 msgr2=0x7fe8e006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:09.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.768+0000 7fe8f9f17700 1 --2- 172.21.15.32:0/875500315 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe8e006cc20 0x7fe8e006f0d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc00b0d0 tx=0x7fe8dc009e30 comp rx=0 tx=0).stop 2024-04-12T05:49:09.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.768+0000 7fe8f9f17700 1 -- 172.21.15.32:0/875500315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8f40fde00 msgr2=0x7fe8f40fbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:09.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.768+0000 7fe8f9f17700 1 --2- 172.21.15.32:0/875500315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8f40fde00 0x7fe8f40fbc30 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe8e400ec90 tx=0x7fe8e400b040 comp rx=0 tx=0).stop 2024-04-12T05:49:09.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.768+0000 7fe8f9f17700 1 -- 172.21.15.32:0/875500315 shutdown_connections 2024-04-12T05:49:09.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.768+0000 7fe8f9f17700 1 --2- 172.21.15.32:0/875500315 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe8e006cc20 0x7fe8e006f0d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:09.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.768+0000 7fe8f9f17700 1 --2- 172.21.15.32:0/875500315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8f40fde00 0x7fe8f40fbc30 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:09.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.768+0000 7fe8f9f17700 1 --2- 172.21.15.32:0/875500315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f40fe7a0 0x7fe8f40fc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:09.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.768+0000 7fe8f9f17700 1 -- 172.21.15.32:0/875500315 >> 172.21.15.32:0/875500315 conn(0x7fe8f40f78d0 msgr2=0x7fe8f40f8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:09.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.768+0000 7fe8f9f17700 1 -- 172.21.15.32:0/875500315 shutdown_connections 2024-04-12T05:49:09.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.768+0000 7fe8f9f17700 1 -- 172.21.15.32:0/875500315 wait complete. 2024-04-12T05:49:09.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.872+0000 7fa523c57700 1 -- 172.21.15.32:0/1251456253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa51c1026b0 msgr2=0x7fa51c0fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:09.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.872+0000 7fa523c57700 1 --2- 172.21.15.32:0/1251456253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa51c1026b0 0x7fa51c0fbd20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa510004940 tx=0x7fa510009e30 comp rx=0 tx=0).stop 2024-04-12T05:49:09.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.872+0000 7fa523c57700 1 -- 172.21.15.32:0/1251456253 shutdown_connections 2024-04-12T05:49:09.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.872+0000 7fa523c57700 1 --2- 172.21.15.32:0/1251456253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa51c1026b0 0x7fa51c0fbd20 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:09.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.872+0000 7fa523c57700 1 --2- 172.21.15.32:0/1251456253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa51c101d10 0x7fa51c1020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:09.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.872+0000 7fa523c57700 1 -- 172.21.15.32:0/1251456253 >> 172.21.15.32:0/1251456253 conn(0x7fa51c0f7880 msgr2=0x7fa51c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:09.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.873+0000 7fa523c57700 1 -- 172.21.15.32:0/1251456253 shutdown_connections 2024-04-12T05:49:09.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.873+0000 7fa523c57700 1 -- 172.21.15.32:0/1251456253 wait complete. 2024-04-12T05:49:09.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.873+0000 7fa523c57700 1 Processor -- start 2024-04-12T05:49:09.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.873+0000 7fa523c57700 1 -- start start 2024-04-12T05:49:09.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.873+0000 7fa523c57700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa51c101d10 0x7fa51c115ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:09.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.873+0000 7fa523c57700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa51c1026b0 0x7fa51c116530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:09.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.873+0000 7fa523c57700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51c116b80 con 0x7fa51c1026b0 2024-04-12T05:49:09.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.873+0000 7fa523c57700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51c10ff50 con 0x7fa51c101d10 2024-04-12T05:49:09.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.873+0000 7fa5211f2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa51c1026b0 0x7fa51c116530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:09.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.873+0000 7fa5211f2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa51c1026b0 0x7fa51c116530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39524/0 (socket says 172.21.15.32:39524) 2024-04-12T05:49:09.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.873+0000 7fa5211f2700 1 -- 172.21.15.32:0/1562127034 learned_addr learned my addr 172.21.15.32:0/1562127034 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:49:09.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.874+0000 7fa5219f3700 1 --2- 172.21.15.32:0/1562127034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa51c101d10 0x7fa51c115ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:09.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.874+0000 7fa5211f2700 1 -- 172.21.15.32:0/1562127034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa51c101d10 msgr2=0x7fa51c115ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:09.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.874+0000 7fa5211f2700 1 --2- 172.21.15.32:0/1562127034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa51c101d10 0x7fa51c115ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:09.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.874+0000 7fa5211f2700 1 -- 172.21.15.32:0/1562127034 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa510005040 con 0x7fa51c1026b0 2024-04-12T05:49:09.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.874+0000 7fa5219f3700 1 --2- 172.21.15.32:0/1562127034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa51c101d10 0x7fa51c115ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:49:09.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.874+0000 7fa5211f2700 1 --2- 172.21.15.32:0/1562127034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa51c1026b0 0x7fa51c116530 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa510009390 tx=0x7fa510009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:09.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.874+0000 7fa50e7fc700 1 -- 172.21.15.32:0/1562127034 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa51000e040 con 0x7fa51c1026b0 2024-04-12T05:49:09.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.874+0000 7fa523c57700 1 -- 172.21.15.32:0/1562127034 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa51c1101d0 con 0x7fa51c1026b0 2024-04-12T05:49:09.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.874+0000 7fa50e7fc700 1 -- 172.21.15.32:0/1562127034 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa510013420 con 0x7fa51c1026b0 2024-04-12T05:49:09.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.874+0000 7fa50e7fc700 1 -- 172.21.15.32:0/1562127034 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa51001c630 con 0x7fa51c1026b0 2024-04-12T05:49:09.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.874+0000 7fa523c57700 1 -- 172.21.15.32:0/1562127034 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa51c110790 con 0x7fa51c1026b0 2024-04-12T05:49:09.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.875+0000 7fa523c57700 1 -- 172.21.15.32:0/1562127034 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa51c0fd460 con 0x7fa51c1026b0 2024-04-12T05:49:09.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.876+0000 7fa50e7fc700 1 -- 172.21.15.32:0/1562127034 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa5100139e0 con 0x7fa51c1026b0 2024-04-12T05:49:09.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.876+0000 7fa50e7fc700 1 --2- 172.21.15.32:0/1562127034 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa50806cc30 0x7fa50806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:09.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.876+0000 7fa5219f3700 1 --2- 172.21.15.32:0/1562127034 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa50806cc30 0x7fa50806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:09.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.876+0000 7fa50e7fc700 1 -- 172.21.15.32:0/1562127034 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fa51008d350 con 0x7fa51c1026b0 2024-04-12T05:49:09.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.877+0000 7fa5219f3700 1 --2- 172.21.15.32:0/1562127034 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa50806cc30 0x7fa50806f0e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa518007fd0 tx=0x7fa518009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:09.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:09.881+0000 7fa50e7fc700 1 -- 172.21.15.32:0/1562127034 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa51005adc0 con 0x7fa51c1026b0 2024-04-12T05:49:10.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:09 smithi067 ceph-mon[112979]: pgmap v259: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:10.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:09 smithi032 ceph-mon[146817]: pgmap v259: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:10.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.166+0000 7fa523c57700 1 -- 172.21.15.32:0/1562127034 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa51c04bed0 con 0x7fa51c1026b0 2024-04-12T05:49:10.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.167+0000 7fa50e7fc700 1 -- 172.21.15.32:0/1562127034 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fa51005a950 con 0x7fa51c1026b0 2024-04-12T05:49:10.163 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:49:10.163 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:49:10.163 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:49:10.163 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:49:10.163 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:49:10.164 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:49:10.164 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:49:10.164 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:49:10.164 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:49:10.164 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:49:10.164 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:49:10.164 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:49:10.164 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:49:10.164 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:49:10.164 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:49:10.164 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:49:10.164 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:49:10.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.169+0000 7fa523c57700 1 -- 172.21.15.32:0/1562127034 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa50806cc30 msgr2=0x7fa50806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:10.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.169+0000 7fa523c57700 1 --2- 172.21.15.32:0/1562127034 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa50806cc30 0x7fa50806f0e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa518007fd0 tx=0x7fa518009040 comp rx=0 tx=0).stop 2024-04-12T05:49:10.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.169+0000 7fa523c57700 1 -- 172.21.15.32:0/1562127034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa51c1026b0 msgr2=0x7fa51c116530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:10.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.169+0000 7fa523c57700 1 --2- 172.21.15.32:0/1562127034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa51c1026b0 0x7fa51c116530 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa510009390 tx=0x7fa510009e30 comp rx=0 tx=0).stop 2024-04-12T05:49:10.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.169+0000 7fa523c57700 1 -- 172.21.15.32:0/1562127034 shutdown_connections 2024-04-12T05:49:10.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.169+0000 7fa523c57700 1 --2- 172.21.15.32:0/1562127034 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa50806cc30 0x7fa50806f0e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:10.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.169+0000 7fa523c57700 1 --2- 172.21.15.32:0/1562127034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa51c101d10 0x7fa51c115ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:10.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.169+0000 7fa523c57700 1 --2- 172.21.15.32:0/1562127034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa51c1026b0 0x7fa51c116530 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:10.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.169+0000 7fa523c57700 1 -- 172.21.15.32:0/1562127034 >> 172.21.15.32:0/1562127034 conn(0x7fa51c0f7880 msgr2=0x7fa51c0f8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:10.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.169+0000 7fa523c57700 1 -- 172.21.15.32:0/1562127034 shutdown_connections 2024-04-12T05:49:10.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.169+0000 7fa523c57700 1 -- 172.21.15.32:0/1562127034 wait complete. 2024-04-12T05:49:10.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.272+0000 7f5e6b16d700 1 -- 172.21.15.32:0/432240476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e641037a0 msgr2=0x7f5e640fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:10.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.272+0000 7f5e6b16d700 1 --2- 172.21.15.32:0/432240476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e641037a0 0x7f5e640fce10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5e50004940 tx=0x7f5e50009e30 comp rx=0 tx=0).stop 2024-04-12T05:49:10.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.272+0000 7f5e6b16d700 1 -- 172.21.15.32:0/432240476 shutdown_connections 2024-04-12T05:49:10.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.272+0000 7f5e6b16d700 1 --2- 172.21.15.32:0/432240476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e641037a0 0x7f5e640fce10 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:10.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.272+0000 7f5e6b16d700 1 --2- 172.21.15.32:0/432240476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e64102e00 0x7f5e641031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:10.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.272+0000 7f5e6b16d700 1 -- 172.21.15.32:0/432240476 >> 172.21.15.32:0/432240476 conn(0x7f5e640f89b0 msgr2=0x7f5e640fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:10.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.272+0000 7f5e6b16d700 1 -- 172.21.15.32:0/432240476 shutdown_connections 2024-04-12T05:49:10.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.272+0000 7f5e6b16d700 1 -- 172.21.15.32:0/432240476 wait complete. 2024-04-12T05:49:10.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.272+0000 7f5e6b16d700 1 Processor -- start 2024-04-12T05:49:10.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.273+0000 7f5e6b16d700 1 -- start start 2024-04-12T05:49:10.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.273+0000 7f5e6b16d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e64102e00 0x7f5e641138c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:10.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.273+0000 7f5e6b16d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e641037a0 0x7f5e64113e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:10.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.273+0000 7f5e6b16d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e64114450 con 0x7f5e64102e00 2024-04-12T05:49:10.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.273+0000 7f5e6b16d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6410da50 con 0x7f5e641037a0 2024-04-12T05:49:10.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.273+0000 7f5e68f09700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e64102e00 0x7f5e641138c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:10.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.273+0000 7f5e68f09700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e64102e00 0x7f5e641138c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39552/0 (socket says 172.21.15.32:39552) 2024-04-12T05:49:10.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.273+0000 7f5e68f09700 1 -- 172.21.15.32:0/1336816914 learned_addr learned my addr 172.21.15.32:0/1336816914 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:49:10.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.273+0000 7f5e63fff700 1 --2- 172.21.15.32:0/1336816914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e641037a0 0x7f5e64113e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:10.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.273+0000 7f5e68f09700 1 -- 172.21.15.32:0/1336816914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e641037a0 msgr2=0x7f5e64113e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:10.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.273+0000 7f5e68f09700 1 --2- 172.21.15.32:0/1336816914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e641037a0 0x7f5e64113e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:10.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.273+0000 7f5e68f09700 1 -- 172.21.15.32:0/1336816914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e50005040 con 0x7f5e64102e00 2024-04-12T05:49:10.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.274+0000 7f5e68f09700 1 --2- 172.21.15.32:0/1336816914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e64102e00 0x7f5e641138c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5e58002a50 tx=0x7f5e580046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:10.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.274+0000 7f5e617fa700 1 -- 172.21.15.32:0/1336816914 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e58007cc0 con 0x7f5e64102e00 2024-04-12T05:49:10.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.274+0000 7f5e6b16d700 1 -- 172.21.15.32:0/1336816914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e6410dd30 con 0x7f5e64102e00 2024-04-12T05:49:10.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.274+0000 7f5e617fa700 1 -- 172.21.15.32:0/1336816914 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e5800f040 con 0x7f5e64102e00 2024-04-12T05:49:10.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.274+0000 7f5e617fa700 1 -- 172.21.15.32:0/1336816914 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e58009540 con 0x7f5e64102e00 2024-04-12T05:49:10.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.274+0000 7f5e6b16d700 1 -- 172.21.15.32:0/1336816914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e6410e2c0 con 0x7f5e64102e00 2024-04-12T05:49:10.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.275+0000 7f5e6b16d700 1 -- 172.21.15.32:0/1336816914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e640fe550 con 0x7f5e64102e00 2024-04-12T05:49:10.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.276+0000 7f5e617fa700 1 -- 172.21.15.32:0/1336816914 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f5e580096a0 con 0x7f5e64102e00 2024-04-12T05:49:10.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.276+0000 7f5e617fa700 1 --2- 172.21.15.32:0/1336816914 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5e5406cc20 0x7f5e5406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:10.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.276+0000 7f5e617fa700 1 -- 172.21.15.32:0/1336816914 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f5e5808bed0 con 0x7f5e64102e00 2024-04-12T05:49:10.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.276+0000 7f5e63fff700 1 --2- 172.21.15.32:0/1336816914 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5e5406cc20 0x7f5e5406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:10.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.276+0000 7f5e63fff700 1 --2- 172.21.15.32:0/1336816914 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5e5406cc20 0x7f5e5406f0d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5e6410ed30 tx=0x7f5e50009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:10.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.281+0000 7f5e617fa700 1 -- 172.21.15.32:0/1336816914 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5e580598c0 con 0x7f5e64102e00 2024-04-12T05:49:10.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.526+0000 7f5e6b16d700 1 -- 172.21.15.32:0/1336816914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5e64109dd0 con 0x7f5e64102e00 2024-04-12T05:49:10.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.527+0000 7f5e617fa700 1 -- 172.21.15.32:0/1336816914 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1835 (secure 0 0 0) 0x7f5e58059450 con 0x7f5e64102e00 2024-04-12T05:49:10.524 INFO:teuthology.orchestra.run.smithi032.stdout:e33 2024-04-12T05:49:10.524 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:49:10.524 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:49:10.524 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:49:10.524 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:49:10.524 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:49:10.524 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:49:10.524 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 24 2024-04-12T05:49:10.524 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:49:10.524 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:49:10.525 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:46:10.668569+0000 2024-04-12T05:49:10.525 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:49:10.525 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:49:10.525 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:49:10.525 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:49:10.525 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:49:10.525 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:49:10.525 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:49:10.525 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:49:10.525 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T05:49:10.525 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:49:10.526 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:49:10.526 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:49:10.526 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T05:49:10.526 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-12T05:49:10.526 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:49:10.526 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:49:10.526 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:49:10.526 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:49:10.526 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:49:10.526 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:49:10.526 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:49:10.527 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:49:10.527 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:49:10.527 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:49:10.527 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-12T05:49:10.527 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:49:10.527 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:49:10.527 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:49:10.527 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{-1:44317} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:49:10.527 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:49:10.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.529+0000 7f5e6b16d700 1 -- 172.21.15.32:0/1336816914 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5e5406cc20 msgr2=0x7f5e5406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:10.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.529+0000 7f5e6b16d700 1 --2- 172.21.15.32:0/1336816914 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5e5406cc20 0x7f5e5406f0d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5e6410ed30 tx=0x7f5e50009e30 comp rx=0 tx=0).stop 2024-04-12T05:49:10.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.529+0000 7f5e6b16d700 1 -- 172.21.15.32:0/1336816914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e64102e00 msgr2=0x7f5e641138c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:10.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.529+0000 7f5e6b16d700 1 --2- 172.21.15.32:0/1336816914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e64102e00 0x7f5e641138c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5e58002a50 tx=0x7f5e580046b0 comp rx=0 tx=0).stop 2024-04-12T05:49:10.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.529+0000 7f5e6b16d700 1 -- 172.21.15.32:0/1336816914 shutdown_connections 2024-04-12T05:49:10.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.529+0000 7f5e6b16d700 1 --2- 172.21.15.32:0/1336816914 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5e5406cc20 0x7f5e5406f0d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:10.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.529+0000 7f5e6b16d700 1 --2- 172.21.15.32:0/1336816914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e64102e00 0x7f5e641138c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:10.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.529+0000 7f5e6b16d700 1 --2- 172.21.15.32:0/1336816914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e641037a0 0x7f5e64113e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:10.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.529+0000 7f5e6b16d700 1 -- 172.21.15.32:0/1336816914 >> 172.21.15.32:0/1336816914 conn(0x7f5e640f89b0 msgr2=0x7f5e640f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:10.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.529+0000 7f5e6b16d700 1 -- 172.21.15.32:0/1336816914 shutdown_connections 2024-04-12T05:49:10.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.529+0000 7f5e6b16d700 1 -- 172.21.15.32:0/1336816914 wait complete. 2024-04-12T05:49:10.529 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-12T05:49:10.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.633+0000 7f3f2f544700 1 -- 172.21.15.32:0/3429049353 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f280fde00 msgr2=0x7f3f280fe1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:10.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.633+0000 7f3f2f544700 1 --2- 172.21.15.32:0/3429049353 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f280fde00 0x7f3f280fe1d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3f24004940 tx=0x7f3f24009e30 comp rx=0 tx=0).stop 2024-04-12T05:49:10.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.633+0000 7f3f2f544700 1 -- 172.21.15.32:0/3429049353 shutdown_connections 2024-04-12T05:49:10.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.633+0000 7f3f2f544700 1 --2- 172.21.15.32:0/3429049353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f280fe7a0 0x7f3f28106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:10.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.633+0000 7f3f2f544700 1 --2- 172.21.15.32:0/3429049353 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f280fde00 0x7f3f280fe1d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:10.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.633+0000 7f3f2f544700 1 -- 172.21.15.32:0/3429049353 >> 172.21.15.32:0/3429049353 conn(0x7f3f280f7880 msgr2=0x7f3f280f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:10.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.633+0000 7f3f2f544700 1 -- 172.21.15.32:0/3429049353 shutdown_connections 2024-04-12T05:49:10.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.633+0000 7f3f2f544700 1 -- 172.21.15.32:0/3429049353 wait complete. 2024-04-12T05:49:10.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.634+0000 7f3f2f544700 1 Processor -- start 2024-04-12T05:49:10.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.634+0000 7f3f2f544700 1 -- start start 2024-04-12T05:49:10.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.634+0000 7f3f2f544700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f280fde00 0x7f3f28113de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:10.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.634+0000 7f3f2f544700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f280fe7a0 0x7f3f28114320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:10.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.634+0000 7f3f2f544700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f28114a00 con 0x7f3f280fe7a0 2024-04-12T05:49:10.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.634+0000 7f3f2f544700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2810dd40 con 0x7f3f280fde00 2024-04-12T05:49:10.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.634+0000 7f3f2cadf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f280fe7a0 0x7f3f28114320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:10.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.634+0000 7f3f2cadf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f280fe7a0 0x7f3f28114320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39564/0 (socket says 172.21.15.32:39564) 2024-04-12T05:49:10.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.634+0000 7f3f2cadf700 1 -- 172.21.15.32:0/4143396314 learned_addr learned my addr 172.21.15.32:0/4143396314 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:49:10.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.634+0000 7f3f2d2e0700 1 --2- 172.21.15.32:0/4143396314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f280fde00 0x7f3f28113de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:10.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.635+0000 7f3f2cadf700 1 -- 172.21.15.32:0/4143396314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f280fde00 msgr2=0x7f3f28113de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:10.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.635+0000 7f3f2cadf700 1 --2- 172.21.15.32:0/4143396314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f280fde00 0x7f3f28113de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:10.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.635+0000 7f3f2cadf700 1 -- 172.21.15.32:0/4143396314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f24005040 con 0x7f3f280fe7a0 2024-04-12T05:49:10.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.635+0000 7f3f2cadf700 1 --2- 172.21.15.32:0/4143396314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f280fe7a0 0x7f3f28114320 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c00b790 tx=0x7f3f1c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:10.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.635+0000 7f3f19ffb700 1 -- 172.21.15.32:0/4143396314 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f1c00f6f0 con 0x7f3f280fe7a0 2024-04-12T05:49:10.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.635+0000 7f3f2f544700 1 -- 172.21.15.32:0/4143396314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f2810dfc0 con 0x7f3f280fe7a0 2024-04-12T05:49:10.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.635+0000 7f3f19ffb700 1 -- 172.21.15.32:0/4143396314 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f1c00fd00 con 0x7f3f280fe7a0 2024-04-12T05:49:10.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.635+0000 7f3f19ffb700 1 -- 172.21.15.32:0/4143396314 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f1c0103a0 con 0x7f3f280fe7a0 2024-04-12T05:49:10.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.635+0000 7f3f2f544700 1 -- 172.21.15.32:0/4143396314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f2810e500 con 0x7f3f280fe7a0 2024-04-12T05:49:10.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.636+0000 7f3f2f544700 1 -- 172.21.15.32:0/4143396314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f280666f0 con 0x7f3f280fe7a0 2024-04-12T05:49:10.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.636+0000 7f3f19ffb700 1 -- 172.21.15.32:0/4143396314 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3f1c00f850 con 0x7f3f280fe7a0 2024-04-12T05:49:10.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.637+0000 7f3f19ffb700 1 --2- 172.21.15.32:0/4143396314 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3f1406cc20 0x7f3f1406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:10.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.637+0000 7f3f19ffb700 1 -- 172.21.15.32:0/4143396314 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f3f1c08bf20 con 0x7f3f280fe7a0 2024-04-12T05:49:10.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.637+0000 7f3f2d2e0700 1 --2- 172.21.15.32:0/4143396314 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3f1406cc20 0x7f3f1406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:10.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.637+0000 7f3f2d2e0700 1 --2- 172.21.15.32:0/4143396314 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3f1406cc20 0x7f3f1406f0d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3f24004f90 tx=0x7f3f24009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:10.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.641+0000 7f3f19ffb700 1 -- 172.21.15.32:0/4143396314 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3f1c0bfab0 con 0x7f3f280fe7a0 2024-04-12T05:49:10.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.868+0000 7f3f2f544700 1 -- 172.21.15.32:0/4143396314 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f28109e40 con 0x7f3f1406cc20 2024-04-12T05:49:10.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.869+0000 7f3f19ffb700 1 -- 172.21.15.32:0/4143396314 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3f28109e40 con 0x7f3f1406cc20 2024-04-12T05:49:10.865 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:49:10.866 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:49:10.866 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:49:10.866 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:49:10.866 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:49:10.866 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-12T05:49:10.866 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:49:10.866 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-12T05:49:10.866 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:49:10.866 INFO:teuthology.orchestra.run.smithi032.stdout: "mds" 2024-04-12T05:49:10.866 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:49:10.866 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:49:10.866 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:49:10.867 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:49:10.867 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:49:10.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.871+0000 7f3f2f544700 1 -- 172.21.15.32:0/4143396314 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3f1406cc20 msgr2=0x7f3f1406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:10.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.871+0000 7f3f2f544700 1 --2- 172.21.15.32:0/4143396314 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3f1406cc20 0x7f3f1406f0d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3f24004f90 tx=0x7f3f24009e30 comp rx=0 tx=0).stop 2024-04-12T05:49:10.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.871+0000 7f3f2f544700 1 -- 172.21.15.32:0/4143396314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f280fe7a0 msgr2=0x7f3f28114320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:10.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.871+0000 7f3f2f544700 1 --2- 172.21.15.32:0/4143396314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f280fe7a0 0x7f3f28114320 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c00b790 tx=0x7f3f1c009250 comp rx=0 tx=0).stop 2024-04-12T05:49:10.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.871+0000 7f3f2f544700 1 -- 172.21.15.32:0/4143396314 shutdown_connections 2024-04-12T05:49:10.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.871+0000 7f3f2f544700 1 --2- 172.21.15.32:0/4143396314 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3f1406cc20 0x7f3f1406f0d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:10.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.871+0000 7f3f2f544700 1 --2- 172.21.15.32:0/4143396314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f280fde00 0x7f3f28113de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:10.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.871+0000 7f3f2f544700 1 --2- 172.21.15.32:0/4143396314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f280fe7a0 0x7f3f28114320 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:10.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.871+0000 7f3f2f544700 1 -- 172.21.15.32:0/4143396314 >> 172.21.15.32:0/4143396314 conn(0x7f3f280f7880 msgr2=0x7f3f280f8df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:10.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.871+0000 7f3f2f544700 1 -- 172.21.15.32:0/4143396314 shutdown_connections 2024-04-12T05:49:10.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.871+0000 7f3f2f544700 1 -- 172.21.15.32:0/4143396314 wait complete. 2024-04-12T05:49:10.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.974+0000 7fc6feadb700 1 -- 172.21.15.32:0/3849100784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6f8102710 msgr2=0x7fc6f80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:10.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.974+0000 7fc6feadb700 1 --2- 172.21.15.32:0/3849100784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6f8102710 0x7fc6f80fbd80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4004940 tx=0x7fc6e4009e30 comp rx=0 tx=0).stop 2024-04-12T05:49:10.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.974+0000 7fc6feadb700 1 -- 172.21.15.32:0/3849100784 shutdown_connections 2024-04-12T05:49:10.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.974+0000 7fc6feadb700 1 --2- 172.21.15.32:0/3849100784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6f8102710 0x7fc6f80fbd80 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:10.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.974+0000 7fc6feadb700 1 --2- 172.21.15.32:0/3849100784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6f8101d70 0x7fc6f8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:10.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.974+0000 7fc6feadb700 1 -- 172.21.15.32:0/3849100784 >> 172.21.15.32:0/3849100784 conn(0x7fc6f8072bc0 msgr2=0x7fc6f8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:10.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.975+0000 7fc6feadb700 1 -- 172.21.15.32:0/3849100784 shutdown_connections 2024-04-12T05:49:10.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.975+0000 7fc6feadb700 1 -- 172.21.15.32:0/3849100784 wait complete. 2024-04-12T05:49:10.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.975+0000 7fc6feadb700 1 Processor -- start 2024-04-12T05:49:10.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.975+0000 7fc6feadb700 1 -- start start 2024-04-12T05:49:10.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.976+0000 7fc6feadb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6f8101d70 0x7fc6f8112960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:10.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.976+0000 7fc6feadb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6f8102710 0x7fc6f8112ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:10.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.976+0000 7fc6feadb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f81134f0 con 0x7fc6f8102710 2024-04-12T05:49:10.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.976+0000 7fc6feadb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f810c920 con 0x7fc6f8101d70 2024-04-12T05:49:10.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.976+0000 7fc6f7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6f8102710 0x7fc6f8112ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:10.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.976+0000 7fc6f7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6f8102710 0x7fc6f8112ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39588/0 (socket says 172.21.15.32:39588) 2024-04-12T05:49:10.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.976+0000 7fc6f7fff700 1 -- 172.21.15.32:0/4221501552 learned_addr learned my addr 172.21.15.32:0/4221501552 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:49:10.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.976+0000 7fc6fc877700 1 --2- 172.21.15.32:0/4221501552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6f8101d70 0x7fc6f8112960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:10.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.976+0000 7fc6f7fff700 1 -- 172.21.15.32:0/4221501552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6f8101d70 msgr2=0x7fc6f8112960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:10.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.976+0000 7fc6f7fff700 1 --2- 172.21.15.32:0/4221501552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6f8101d70 0x7fc6f8112960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:10.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.976+0000 7fc6f7fff700 1 -- 172.21.15.32:0/4221501552 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e4005040 con 0x7fc6f8102710 2024-04-12T05:49:10.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.976+0000 7fc6f7fff700 1 --2- 172.21.15.32:0/4221501552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6f8102710 0x7fc6f8112ea0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4009380 tx=0x7fc6e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:10.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.977+0000 7fc6f57fa700 1 -- 172.21.15.32:0/4221501552 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6e400e040 con 0x7fc6f8102710 2024-04-12T05:49:10.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.977+0000 7fc6feadb700 1 -- 172.21.15.32:0/4221501552 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6f810cba0 con 0x7fc6f8102710 2024-04-12T05:49:10.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.977+0000 7fc6f57fa700 1 -- 172.21.15.32:0/4221501552 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc6e4013420 con 0x7fc6f8102710 2024-04-12T05:49:10.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.977+0000 7fc6f57fa700 1 -- 172.21.15.32:0/4221501552 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6e401c630 con 0x7fc6f8102710 2024-04-12T05:49:10.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.977+0000 7fc6feadb700 1 -- 172.21.15.32:0/4221501552 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6f810d160 con 0x7fc6f8102710 2024-04-12T05:49:10.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.978+0000 7fc6feadb700 1 -- 172.21.15.32:0/4221501552 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6f80ff080 con 0x7fc6f8102710 2024-04-12T05:49:10.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.978+0000 7fc6f57fa700 1 -- 172.21.15.32:0/4221501552 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc6e400ba80 con 0x7fc6f8102710 2024-04-12T05:49:10.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.978+0000 7fc6f57fa700 1 --2- 172.21.15.32:0/4221501552 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc6e806cc20 0x7fc6e806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:10.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.979+0000 7fc6f57fa700 1 -- 172.21.15.32:0/4221501552 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fc6e408d280 con 0x7fc6f8102710 2024-04-12T05:49:10.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.979+0000 7fc6fc877700 1 --2- 172.21.15.32:0/4221501552 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc6e806cc20 0x7fc6e806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:10.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.979+0000 7fc6fc877700 1 --2- 172.21.15.32:0/4221501552 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc6e806cc20 0x7fc6e806f0d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec004f40 tx=0x7fc6ec00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:10.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:10.983+0000 7fc6f57fa700 1 -- 172.21.15.32:0/4221501552 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc6e405acf0 con 0x7fc6f8102710 2024-04-12T05:49:11.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:10 smithi067 ceph-mon[112979]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:49:11.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:10 smithi067 ceph-mon[112979]: from='client.34540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:49:11.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:10 smithi067 ceph-mon[112979]: from='client.34544 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:49:11.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:10 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1562127034' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:11.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:10 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1336816914' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:49:11.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:10 smithi032 ceph-mon[146817]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:49:11.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:10 smithi032 ceph-mon[146817]: from='client.34540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:49:11.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:10 smithi032 ceph-mon[146817]: from='client.34544 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:49:11.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:10 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1562127034' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:11.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:10 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1336816914' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:49:11.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:11.268+0000 7fc6feadb700 1 -- 172.21.15.32:0/4221501552 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc6f810e010 con 0x7fc6f8102710 2024-04-12T05:49:11.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:11.268+0000 7fc6f57fa700 1 -- 172.21.15.32:0/4221501552 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fc6e405a880 con 0x7fc6f8102710 2024-04-12T05:49:11.265 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:49:11.265 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:49:11.265 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-12T05:49:11.265 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:49:11.265 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:49:11.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:11.270+0000 7fc6feadb700 1 -- 172.21.15.32:0/4221501552 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc6e806cc20 msgr2=0x7fc6e806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:11.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:11.270+0000 7fc6feadb700 1 --2- 172.21.15.32:0/4221501552 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc6e806cc20 0x7fc6e806f0d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec004f40 tx=0x7fc6ec00b040 comp rx=0 tx=0).stop 2024-04-12T05:49:11.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:11.270+0000 7fc6feadb700 1 -- 172.21.15.32:0/4221501552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6f8102710 msgr2=0x7fc6f8112ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:11.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:11.270+0000 7fc6feadb700 1 --2- 172.21.15.32:0/4221501552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6f8102710 0x7fc6f8112ea0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4009380 tx=0x7fc6e4009e30 comp rx=0 tx=0).stop 2024-04-12T05:49:11.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:11.271+0000 7fc6feadb700 1 -- 172.21.15.32:0/4221501552 shutdown_connections 2024-04-12T05:49:11.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:11.271+0000 7fc6feadb700 1 --2- 172.21.15.32:0/4221501552 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc6e806cc20 0x7fc6e806f0d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:11.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:11.271+0000 7fc6feadb700 1 --2- 172.21.15.32:0/4221501552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6f8101d70 0x7fc6f8112960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:11.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:11.271+0000 7fc6feadb700 1 --2- 172.21.15.32:0/4221501552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6f8102710 0x7fc6f8112ea0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:11.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:11.271+0000 7fc6feadb700 1 -- 172.21.15.32:0/4221501552 >> 172.21.15.32:0/4221501552 conn(0x7fc6f8072bc0 msgr2=0x7fc6f80fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:11.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:11.271+0000 7fc6feadb700 1 -- 172.21.15.32:0/4221501552 shutdown_connections 2024-04-12T05:49:11.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:11.271+0000 7fc6feadb700 1 -- 172.21.15.32:0/4221501552 wait complete. 2024-04-12T05:49:12.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:11 smithi067 ceph-mon[112979]: pgmap v260: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:12.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:11 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/4221501552' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:49:12.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:11 smithi032 ceph-mon[146817]: pgmap v260: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:12.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:11 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/4221501552' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:49:13.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:12 smithi067 ceph-mon[112979]: from='client.34556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:49:13.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:12 smithi032 ceph-mon[146817]: from='client.34556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:49:14.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:13 smithi067 ceph-mon[112979]: pgmap v261: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:14.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:13 smithi032 ceph-mon[146817]: pgmap v261: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:16.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:15 smithi067 ceph-mon[112979]: pgmap v262: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:16.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:15 smithi032 ceph-mon[146817]: pgmap v262: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:18.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:17 smithi067 ceph-mon[112979]: pgmap v263: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:18.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:17 smithi032 ceph-mon[146817]: pgmap v263: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:19.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:49:19.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:49:20.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:19 smithi067 ceph-mon[112979]: pgmap v264: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:20.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:19 smithi032 ceph-mon[146817]: pgmap v264: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:22.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:21 smithi032 ceph-mon[146817]: pgmap v265: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:22.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:21 smithi067 ceph-mon[112979]: pgmap v265: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:24.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:23 smithi032 ceph-mon[146817]: pgmap v266: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:24.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:23 smithi067 ceph-mon[112979]: pgmap v266: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:26.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:25 smithi032 ceph-mon[146817]: pgmap v267: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:26.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:25 smithi067 ceph-mon[112979]: pgmap v267: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:27.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:26 smithi032 ceph-mon[146817]: pgmap v268: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:27.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:26 smithi067 ceph-mon[112979]: pgmap v268: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:29.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:28 smithi067 ceph-mon[112979]: pgmap v269: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:29.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:28 smithi032 ceph-mon[146817]: pgmap v269: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:31.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:30 smithi067 ceph-mon[112979]: pgmap v270: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:31.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:30 smithi032 ceph-mon[146817]: pgmap v270: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:33.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:32 smithi067 ceph-mon[112979]: pgmap v271: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:33.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:32 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:49:33.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:32 smithi032 ceph-mon[146817]: pgmap v271: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:33.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:32 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:49:35.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:34 smithi067 ceph-mon[112979]: pgmap v272: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:35.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:34 smithi032 ceph-mon[146817]: pgmap v272: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:37.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:36 smithi067 ceph-mon[112979]: pgmap v273: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:37.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:36 smithi032 ceph-mon[146817]: pgmap v273: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:39.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:38 smithi067 ceph-mon[112979]: pgmap v274: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:39.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:38 smithi032 ceph-mon[146817]: pgmap v274: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:41.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:40 smithi067 ceph-mon[112979]: pgmap v275: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:41.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:40 smithi032 ceph-mon[146817]: pgmap v275: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:41.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.390+0000 7f55ecffd700 1 -- 172.21.15.32:0/2871983302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55e8102710 msgr2=0x7f55e80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.390+0000 7f55ecffd700 1 --2- 172.21.15.32:0/2871983302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55e8102710 0x7f55e80fbd80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f55d0004940 tx=0x7f55d0009e30 comp rx=0 tx=0).stop 2024-04-12T05:49:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.390+0000 7f55ecffd700 1 -- 172.21.15.32:0/2871983302 shutdown_connections 2024-04-12T05:49:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.390+0000 7f55ecffd700 1 --2- 172.21.15.32:0/2871983302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55e8102710 0x7f55e80fbd80 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.390+0000 7f55ecffd700 1 --2- 172.21.15.32:0/2871983302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55e8101d70 0x7f55e8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.390+0000 7f55ecffd700 1 -- 172.21.15.32:0/2871983302 >> 172.21.15.32:0/2871983302 conn(0x7f55e8072bc0 msgr2=0x7f55e8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.390+0000 7f55ecffd700 1 -- 172.21.15.32:0/2871983302 shutdown_connections 2024-04-12T05:49:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.390+0000 7f55ecffd700 1 -- 172.21.15.32:0/2871983302 wait complete. 2024-04-12T05:49:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.390+0000 7f55ecffd700 1 Processor -- start 2024-04-12T05:49:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.390+0000 7f55ecffd700 1 -- start start 2024-04-12T05:49:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.391+0000 7f55ecffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55e8101d70 0x7f55e810ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.391+0000 7f55ecffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55e8102710 0x7f55e810f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.391+0000 7f55ecffd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e810f680 con 0x7f55e8101d70 2024-04-12T05:49:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.391+0000 7f55ecffd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e810f7c0 con 0x7f55e8102710 2024-04-12T05:49:41.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.391+0000 7f55e659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55e8101d70 0x7f55e810ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:41.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.391+0000 7f55e659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55e8101d70 0x7f55e810ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52256/0 (socket says 172.21.15.32:52256) 2024-04-12T05:49:41.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.391+0000 7f55e659c700 1 -- 172.21.15.32:0/2007482485 learned_addr learned my addr 172.21.15.32:0/2007482485 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:49:41.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.391+0000 7f55e5d9b700 1 --2- 172.21.15.32:0/2007482485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55e8102710 0x7f55e810f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:41.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.391+0000 7f55e659c700 1 -- 172.21.15.32:0/2007482485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55e8102710 msgr2=0x7f55e810f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:41.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.391+0000 7f55e659c700 1 --2- 172.21.15.32:0/2007482485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55e8102710 0x7f55e810f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:41.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.391+0000 7f55e659c700 1 -- 172.21.15.32:0/2007482485 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d0005040 con 0x7f55e8101d70 2024-04-12T05:49:41.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.391+0000 7f55e659c700 1 --2- 172.21.15.32:0/2007482485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55e8101d70 0x7f55e810ec00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f55d8002a50 tx=0x7f55d80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:41.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.391+0000 7f55deffd700 1 -- 172.21.15.32:0/2007482485 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55d8007cc0 con 0x7f55e8101d70 2024-04-12T05:49:41.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.392+0000 7f55deffd700 1 -- 172.21.15.32:0/2007482485 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f55d800f040 con 0x7f55e8101d70 2024-04-12T05:49:41.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.392+0000 7f55deffd700 1 -- 172.21.15.32:0/2007482485 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55d800d9c0 con 0x7f55e8101d70 2024-04-12T05:49:41.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.392+0000 7f55ecffd700 1 -- 172.21.15.32:0/2007482485 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55e8117ab0 con 0x7f55e8101d70 2024-04-12T05:49:41.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.392+0000 7f55ecffd700 1 -- 172.21.15.32:0/2007482485 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55e8118040 con 0x7f55e8101d70 2024-04-12T05:49:41.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.393+0000 7f55ecffd700 1 -- 172.21.15.32:0/2007482485 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55e80ff080 con 0x7f55e8101d70 2024-04-12T05:49:41.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.393+0000 7f55deffd700 1 -- 172.21.15.32:0/2007482485 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f55d800db80 con 0x7f55e8101d70 2024-04-12T05:49:41.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.394+0000 7f55deffd700 1 --2- 172.21.15.32:0/2007482485 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f55d406cc30 0x7f55d406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:41.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.394+0000 7f55deffd700 1 -- 172.21.15.32:0/2007482485 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f55d808be00 con 0x7f55e8101d70 2024-04-12T05:49:41.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.394+0000 7f55e5d9b700 1 --2- 172.21.15.32:0/2007482485 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f55d406cc30 0x7f55d406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:41.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.394+0000 7f55e5d9b700 1 --2- 172.21.15.32:0/2007482485 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f55d406cc30 0x7f55d406f0e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f55e8066020 tx=0x7f55d0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:41.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.398+0000 7f55deffd700 1 -- 172.21.15.32:0/2007482485 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f55d8055d60 con 0x7f55e8101d70 2024-04-12T05:49:41.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.620+0000 7f55ecffd700 1 -- 172.21.15.32:0/2007482485 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f55e804bed0 con 0x7f55d406cc30 2024-04-12T05:49:41.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.621+0000 7f55deffd700 1 -- 172.21.15.32:0/2007482485 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f55e804bed0 con 0x7f55d406cc30 2024-04-12T05:49:41.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.623+0000 7f55ecffd700 1 -- 172.21.15.32:0/2007482485 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f55d406cc30 msgr2=0x7f55d406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:41.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.623+0000 7f55ecffd700 1 --2- 172.21.15.32:0/2007482485 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f55d406cc30 0x7f55d406f0e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f55e8066020 tx=0x7f55d0009e30 comp rx=0 tx=0).stop 2024-04-12T05:49:41.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.623+0000 7f55ecffd700 1 -- 172.21.15.32:0/2007482485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55e8101d70 msgr2=0x7f55e810ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:41.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.623+0000 7f55ecffd700 1 --2- 172.21.15.32:0/2007482485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55e8101d70 0x7f55e810ec00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f55d8002a50 tx=0x7f55d80046b0 comp rx=0 tx=0).stop 2024-04-12T05:49:41.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.623+0000 7f55ecffd700 1 -- 172.21.15.32:0/2007482485 shutdown_connections 2024-04-12T05:49:41.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.623+0000 7f55ecffd700 1 --2- 172.21.15.32:0/2007482485 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f55d406cc30 0x7f55d406f0e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:41.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.623+0000 7f55ecffd700 1 --2- 172.21.15.32:0/2007482485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55e8101d70 0x7f55e810ec00 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:41.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.623+0000 7f55ecffd700 1 --2- 172.21.15.32:0/2007482485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55e8102710 0x7f55e810f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:41.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.623+0000 7f55ecffd700 1 -- 172.21.15.32:0/2007482485 >> 172.21.15.32:0/2007482485 conn(0x7f55e8072bc0 msgr2=0x7f55e80fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:41.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.623+0000 7f55ecffd700 1 -- 172.21.15.32:0/2007482485 shutdown_connections 2024-04-12T05:49:41.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.623+0000 7f55ecffd700 1 -- 172.21.15.32:0/2007482485 wait complete. 2024-04-12T05:49:41.628 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:49:41.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.728+0000 7f31c7c0c700 1 -- 172.21.15.32:0/3352210718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31c0102710 msgr2=0x7f31c00fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:41.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.728+0000 7f31c7c0c700 1 --2- 172.21.15.32:0/3352210718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31c0102710 0x7f31c00fbd80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f31b4004940 tx=0x7f31b4009e30 comp rx=0 tx=0).stop 2024-04-12T05:49:41.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.728+0000 7f31c7c0c700 1 -- 172.21.15.32:0/3352210718 shutdown_connections 2024-04-12T05:49:41.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.728+0000 7f31c7c0c700 1 --2- 172.21.15.32:0/3352210718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31c0102710 0x7f31c00fbd80 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:41.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.728+0000 7f31c7c0c700 1 --2- 172.21.15.32:0/3352210718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31c0101d70 0x7f31c0102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:41.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.728+0000 7f31c7c0c700 1 -- 172.21.15.32:0/3352210718 >> 172.21.15.32:0/3352210718 conn(0x7f31c0072bc0 msgr2=0x7f31c0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:41.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.728+0000 7f31c7c0c700 1 -- 172.21.15.32:0/3352210718 shutdown_connections 2024-04-12T05:49:41.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.728+0000 7f31c7c0c700 1 -- 172.21.15.32:0/3352210718 wait complete. 2024-04-12T05:49:41.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.729+0000 7f31c7c0c700 1 Processor -- start 2024-04-12T05:49:41.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.729+0000 7f31c7c0c700 1 -- start start 2024-04-12T05:49:41.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.729+0000 7f31c7c0c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31c0101d70 0x7f31c010eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:41.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.729+0000 7f31c7c0c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31c0102710 0x7f31c010f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:41.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.729+0000 7f31c7c0c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c010f580 con 0x7f31c0102710 2024-04-12T05:49:41.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.729+0000 7f31c7c0c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c010f6c0 con 0x7f31c0101d70 2024-04-12T05:49:41.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.729+0000 7f31c51a7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31c0102710 0x7f31c010f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:41.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.729+0000 7f31c51a7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31c0102710 0x7f31c010f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36096/0 (socket says 172.21.15.32:36096) 2024-04-12T05:49:41.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.729+0000 7f31c51a7700 1 -- 172.21.15.32:0/276645343 learned_addr learned my addr 172.21.15.32:0/276645343 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:49:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.729+0000 7f31c59a8700 1 --2- 172.21.15.32:0/276645343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31c0101d70 0x7f31c010eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.729+0000 7f31c51a7700 1 -- 172.21.15.32:0/276645343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31c0101d70 msgr2=0x7f31c010eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.729+0000 7f31c51a7700 1 --2- 172.21.15.32:0/276645343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31c0101d70 0x7f31c010eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.729+0000 7f31c51a7700 1 -- 172.21.15.32:0/276645343 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b4005040 con 0x7f31c0102710 2024-04-12T05:49:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.730+0000 7f31c51a7700 1 --2- 172.21.15.32:0/276645343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31c0102710 0x7f31c010f040 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f31b4008d50 tx=0x7f31b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.730+0000 7f31b27fc700 1 -- 172.21.15.32:0/276645343 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31b400e030 con 0x7f31c0102710 2024-04-12T05:49:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.730+0000 7f31b27fc700 1 -- 172.21.15.32:0/276645343 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f31b40093b0 con 0x7f31c0102710 2024-04-12T05:49:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.730+0000 7f31c7c0c700 1 -- 172.21.15.32:0/276645343 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31c01179b0 con 0x7f31c0102710 2024-04-12T05:49:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.730+0000 7f31b27fc700 1 -- 172.21.15.32:0/276645343 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31b401cbb0 con 0x7f31c0102710 2024-04-12T05:49:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.730+0000 7f31c7c0c700 1 -- 172.21.15.32:0/276645343 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31c0117f10 con 0x7f31c0102710 2024-04-12T05:49:41.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.731+0000 7f31c7c0c700 1 -- 172.21.15.32:0/276645343 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31c00ff080 con 0x7f31c0102710 2024-04-12T05:49:41.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.731+0000 7f31b27fc700 1 -- 172.21.15.32:0/276645343 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f31b4013930 con 0x7f31c0102710 2024-04-12T05:49:41.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.732+0000 7f31b27fc700 1 --2- 172.21.15.32:0/276645343 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f31ac06cc30 0x7f31ac06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:41.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.732+0000 7f31b27fc700 1 -- 172.21.15.32:0/276645343 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f31b408d7e0 con 0x7f31c0102710 2024-04-12T05:49:41.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.732+0000 7f31c59a8700 1 --2- 172.21.15.32:0/276645343 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f31ac06cc30 0x7f31ac06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:41.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.732+0000 7f31c59a8700 1 --2- 172.21.15.32:0/276645343 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f31ac06cc30 0x7f31ac06f0e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f31bc009b80 tx=0x7f31bc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:41.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.736+0000 7f31b27fc700 1 -- 172.21.15.32:0/276645343 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f31b40577c0 con 0x7f31c0102710 2024-04-12T05:49:41.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.959+0000 7f31c7c0c700 1 -- 172.21.15.32:0/276645343 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f31c0065e10 con 0x7f31ac06cc30 2024-04-12T05:49:41.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.961+0000 7f31b27fc700 1 -- 172.21.15.32:0/276645343 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f31c0065e10 con 0x7f31ac06cc30 2024-04-12T05:49:41.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.963+0000 7f31c7c0c700 1 -- 172.21.15.32:0/276645343 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f31ac06cc30 msgr2=0x7f31ac06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:41.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.963+0000 7f31c7c0c700 1 --2- 172.21.15.32:0/276645343 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f31ac06cc30 0x7f31ac06f0e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f31bc009b80 tx=0x7f31bc00b040 comp rx=0 tx=0).stop 2024-04-12T05:49:41.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.963+0000 7f31c7c0c700 1 -- 172.21.15.32:0/276645343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31c0102710 msgr2=0x7f31c010f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:41.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.963+0000 7f31c7c0c700 1 --2- 172.21.15.32:0/276645343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31c0102710 0x7f31c010f040 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f31b4008d50 tx=0x7f31b4009e30 comp rx=0 tx=0).stop 2024-04-12T05:49:41.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.963+0000 7f31c7c0c700 1 -- 172.21.15.32:0/276645343 shutdown_connections 2024-04-12T05:49:41.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.963+0000 7f31c7c0c700 1 --2- 172.21.15.32:0/276645343 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f31ac06cc30 0x7f31ac06f0e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:41.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.963+0000 7f31c7c0c700 1 --2- 172.21.15.32:0/276645343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31c0101d70 0x7f31c010eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:41.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.963+0000 7f31c7c0c700 1 --2- 172.21.15.32:0/276645343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31c0102710 0x7f31c010f040 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:41.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.963+0000 7f31c7c0c700 1 -- 172.21.15.32:0/276645343 >> 172.21.15.32:0/276645343 conn(0x7f31c0072bc0 msgr2=0x7f31c00faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:41.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.963+0000 7f31c7c0c700 1 -- 172.21.15.32:0/276645343 shutdown_connections 2024-04-12T05:49:41.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:41.963+0000 7f31c7c0c700 1 -- 172.21.15.32:0/276645343 wait complete. 2024-04-12T05:49:42.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.067+0000 7efe423f3700 1 -- 172.21.15.32:0/428828150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe3c102710 msgr2=0x7efe3c0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:42.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.067+0000 7efe423f3700 1 --2- 172.21.15.32:0/428828150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe3c102710 0x7efe3c0fbd80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7efe30004940 tx=0x7efe30009e30 comp rx=0 tx=0).stop 2024-04-12T05:49:42.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.067+0000 7efe423f3700 1 -- 172.21.15.32:0/428828150 shutdown_connections 2024-04-12T05:49:42.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.067+0000 7efe423f3700 1 --2- 172.21.15.32:0/428828150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe3c102710 0x7efe3c0fbd80 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:42.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.067+0000 7efe423f3700 1 --2- 172.21.15.32:0/428828150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe3c101d70 0x7efe3c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:42.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.067+0000 7efe423f3700 1 -- 172.21.15.32:0/428828150 >> 172.21.15.32:0/428828150 conn(0x7efe3c072bc0 msgr2=0x7efe3c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:42.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.067+0000 7efe423f3700 1 -- 172.21.15.32:0/428828150 shutdown_connections 2024-04-12T05:49:42.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.067+0000 7efe423f3700 1 -- 172.21.15.32:0/428828150 wait complete. 2024-04-12T05:49:42.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.068+0000 7efe423f3700 1 Processor -- start 2024-04-12T05:49:42.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.068+0000 7efe423f3700 1 -- start start 2024-04-12T05:49:42.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.068+0000 7efe423f3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe3c101d70 0x7efe3c1129f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:42.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.068+0000 7efe423f3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe3c102710 0x7efe3c112f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:42.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.068+0000 7efe423f3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3c113610 con 0x7efe3c102710 2024-04-12T05:49:42.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.068+0000 7efe423f3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3c10c970 con 0x7efe3c101d70 2024-04-12T05:49:42.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.068+0000 7efe3b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe3c102710 0x7efe3c112f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:42.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.068+0000 7efe3b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe3c102710 0x7efe3c112f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36116/0 (socket says 172.21.15.32:36116) 2024-04-12T05:49:42.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.069+0000 7efe3b7fe700 1 -- 172.21.15.32:0/1484866234 learned_addr learned my addr 172.21.15.32:0/1484866234 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:49:42.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.069+0000 7efe3bfff700 1 --2- 172.21.15.32:0/1484866234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe3c101d70 0x7efe3c1129f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:42.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.069+0000 7efe3b7fe700 1 -- 172.21.15.32:0/1484866234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe3c101d70 msgr2=0x7efe3c1129f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:42.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.069+0000 7efe3b7fe700 1 --2- 172.21.15.32:0/1484866234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe3c101d70 0x7efe3c1129f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:42.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.069+0000 7efe3b7fe700 1 -- 172.21.15.32:0/1484866234 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe30005040 con 0x7efe3c102710 2024-04-12T05:49:42.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.069+0000 7efe3bfff700 1 --2- 172.21.15.32:0/1484866234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe3c101d70 0x7efe3c1129f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:49:42.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.069+0000 7efe3b7fe700 1 --2- 172.21.15.32:0/1484866234 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe3c102710 0x7efe3c112f30 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7efe300093b0 tx=0x7efe30009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:42.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.069+0000 7efe38ff9700 1 -- 172.21.15.32:0/1484866234 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe3000e040 con 0x7efe3c102710 2024-04-12T05:49:42.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.069+0000 7efe38ff9700 1 -- 172.21.15.32:0/1484866234 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efe30013420 con 0x7efe3c102710 2024-04-12T05:49:42.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.069+0000 7efe38ff9700 1 -- 172.21.15.32:0/1484866234 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe3001c630 con 0x7efe3c102710 2024-04-12T05:49:42.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.069+0000 7efe423f3700 1 -- 172.21.15.32:0/1484866234 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe3c10cbf0 con 0x7efe3c102710 2024-04-12T05:49:42.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.069+0000 7efe423f3700 1 -- 172.21.15.32:0/1484866234 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe3c10d1b0 con 0x7efe3c102710 2024-04-12T05:49:42.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.070+0000 7efe423f3700 1 -- 172.21.15.32:0/1484866234 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe3c0ff080 con 0x7efe3c102710 2024-04-12T05:49:42.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.071+0000 7efe38ff9700 1 -- 172.21.15.32:0/1484866234 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7efe300139e0 con 0x7efe3c102710 2024-04-12T05:49:42.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.071+0000 7efe38ff9700 1 --2- 172.21.15.32:0/1484866234 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7efe2406cc20 0x7efe2406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:42.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.071+0000 7efe38ff9700 1 -- 172.21.15.32:0/1484866234 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7efe3008d190 con 0x7efe3c102710 2024-04-12T05:49:42.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.071+0000 7efe3bfff700 1 --2- 172.21.15.32:0/1484866234 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7efe2406cc20 0x7efe2406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:42.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.072+0000 7efe3bfff700 1 --2- 172.21.15.32:0/1484866234 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7efe2406cc20 0x7efe2406f0d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7efe2c009b10 tx=0x7efe2c008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:42.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.076+0000 7efe38ff9700 1 -- 172.21.15.32:0/1484866234 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7efe3005ad50 con 0x7efe3c102710 2024-04-12T05:49:42.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:41 smithi067 ceph-mon[112979]: from='client.34564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:49:42.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.291+0000 7efe423f3700 1 -- 172.21.15.32:0/1484866234 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7efe3c10e040 con 0x7efe2406cc20 2024-04-12T05:49:42.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.301+0000 7efe38ff9700 1 -- 172.21.15.32:0/1484866234 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7efe3c10e040 con 0x7efe2406cc20 2024-04-12T05:49:42.296 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:49:42.296 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (9m) 2m ago 20m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:49:42.296 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (7m) 2m ago 21m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:49:42.296 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (6m) 2m ago 20m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:49:42.296 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (9m) 2m ago 20m 87.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:49:42.296 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (3m) 2m ago 15m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T05:49:42.297 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (3m) 2m ago 15m 13.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T05:49:42.297 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (2m) 2m ago 15m 11.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 6d4aca6f74a0 2024-04-12T05:49:42.297 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (2m) 2m ago 15m 14.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T05:49:42.297 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (11m) 2m ago 22m 507M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:49:42.297 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (10m) 2m ago 19m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:49:42.297 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (8m) 2m ago 22m 53.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:49:42.297 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (7m) 2m ago 19m 43.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:49:42.297 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (10m) 2m ago 21m 16.9M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:49:42.297 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (10m) 2m ago 20m 18.4M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:49:42.298 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (6m) 2m ago 19m 91.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T05:49:42.298 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (5m) 2m ago 18m 97.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T05:49:42.298 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (5m) 2m ago 18m 81.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T05:49:42.298 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (4m) 2m ago 17m 113M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T05:49:42.298 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (4m) 2m ago 17m 90.3M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T05:49:42.298 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (3m) 2m ago 16m 84.8M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T05:49:42.298 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (9m) 2m ago 20m 64.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:49:42.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.303+0000 7efe423f3700 1 -- 172.21.15.32:0/1484866234 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7efe2406cc20 msgr2=0x7efe2406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:42.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.303+0000 7efe423f3700 1 --2- 172.21.15.32:0/1484866234 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7efe2406cc20 0x7efe2406f0d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7efe2c009b10 tx=0x7efe2c008040 comp rx=0 tx=0).stop 2024-04-12T05:49:42.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.303+0000 7efe423f3700 1 -- 172.21.15.32:0/1484866234 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe3c102710 msgr2=0x7efe3c112f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:42.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.303+0000 7efe423f3700 1 --2- 172.21.15.32:0/1484866234 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe3c102710 0x7efe3c112f30 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7efe300093b0 tx=0x7efe30009e30 comp rx=0 tx=0).stop 2024-04-12T05:49:42.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.303+0000 7efe423f3700 1 -- 172.21.15.32:0/1484866234 shutdown_connections 2024-04-12T05:49:42.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.303+0000 7efe423f3700 1 --2- 172.21.15.32:0/1484866234 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7efe2406cc20 0x7efe2406f0d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:42.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.303+0000 7efe423f3700 1 --2- 172.21.15.32:0/1484866234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe3c101d70 0x7efe3c1129f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:42.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.303+0000 7efe423f3700 1 --2- 172.21.15.32:0/1484866234 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe3c102710 0x7efe3c112f30 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:42.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.303+0000 7efe423f3700 1 -- 172.21.15.32:0/1484866234 >> 172.21.15.32:0/1484866234 conn(0x7efe3c072bc0 msgr2=0x7efe3c0fb060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:42.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.303+0000 7efe423f3700 1 -- 172.21.15.32:0/1484866234 shutdown_connections 2024-04-12T05:49:42.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.303+0000 7efe423f3700 1 -- 172.21.15.32:0/1484866234 wait complete. 2024-04-12T05:49:42.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:41 smithi032 ceph-mon[146817]: from='client.34564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:49:42.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.405+0000 7f43220a8700 1 -- 172.21.15.32:0/160264474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f431c1037b0 msgr2=0x7f431c0fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:42.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.405+0000 7f43220a8700 1 --2- 172.21.15.32:0/160264474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f431c1037b0 0x7f431c0fce20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4310004940 tx=0x7f4310009e30 comp rx=0 tx=0).stop 2024-04-12T05:49:42.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.405+0000 7f43220a8700 1 -- 172.21.15.32:0/160264474 shutdown_connections 2024-04-12T05:49:42.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.405+0000 7f43220a8700 1 --2- 172.21.15.32:0/160264474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f431c1037b0 0x7f431c0fce20 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:42.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.405+0000 7f43220a8700 1 --2- 172.21.15.32:0/160264474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f431c102e10 0x7f431c1031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:42.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.405+0000 7f43220a8700 1 -- 172.21.15.32:0/160264474 >> 172.21.15.32:0/160264474 conn(0x7f431c0f8980 msgr2=0x7f431c0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:42.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.405+0000 7f43220a8700 1 -- 172.21.15.32:0/160264474 shutdown_connections 2024-04-12T05:49:42.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.405+0000 7f43220a8700 1 -- 172.21.15.32:0/160264474 wait complete. 2024-04-12T05:49:42.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.406+0000 7f43220a8700 1 Processor -- start 2024-04-12T05:49:42.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.406+0000 7f43220a8700 1 -- start start 2024-04-12T05:49:42.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.406+0000 7f43220a8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f431c102e10 0x7f431c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:42.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.406+0000 7f43220a8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f431c1037b0 0x7f431c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:42.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.406+0000 7f43220a8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431c070d90 con 0x7f431c1037b0 2024-04-12T05:49:42.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.406+0000 7f43220a8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431c070ed0 con 0x7f431c102e10 2024-04-12T05:49:42.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.406+0000 7f431affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f431c1037b0 0x7f431c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:42.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.406+0000 7f431affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f431c1037b0 0x7f431c070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36146/0 (socket says 172.21.15.32:36146) 2024-04-12T05:49:42.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.406+0000 7f431affd700 1 -- 172.21.15.32:0/4251063278 learned_addr learned my addr 172.21.15.32:0/4251063278 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:49:42.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.406+0000 7f431b7fe700 1 --2- 172.21.15.32:0/4251063278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f431c102e10 0x7f431c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:42.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.407+0000 7f431affd700 1 -- 172.21.15.32:0/4251063278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f431c102e10 msgr2=0x7f431c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:42.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.407+0000 7f431affd700 1 --2- 172.21.15.32:0/4251063278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f431c102e10 0x7f431c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:42.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.407+0000 7f431affd700 1 -- 172.21.15.32:0/4251063278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4310005040 con 0x7f431c1037b0 2024-04-12T05:49:42.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.407+0000 7f431affd700 1 --2- 172.21.15.32:0/4251063278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f431c1037b0 0x7f431c070800 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4310009390 tx=0x7f4310009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:42.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.407+0000 7f4303fff700 1 -- 172.21.15.32:0/4251063278 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f431000e040 con 0x7f431c1037b0 2024-04-12T05:49:42.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.407+0000 7f4303fff700 1 -- 172.21.15.32:0/4251063278 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4310004290 con 0x7f431c1037b0 2024-04-12T05:49:42.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.407+0000 7f43220a8700 1 -- 172.21.15.32:0/4251063278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f431c071120 con 0x7f431c1037b0 2024-04-12T05:49:42.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.407+0000 7f4303fff700 1 -- 172.21.15.32:0/4251063278 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4310013e70 con 0x7f431c1037b0 2024-04-12T05:49:42.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.407+0000 7f43220a8700 1 -- 172.21.15.32:0/4251063278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f431c11a210 con 0x7f431c1037b0 2024-04-12T05:49:42.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.408+0000 7f43220a8700 1 -- 172.21.15.32:0/4251063278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f431c0fe560 con 0x7f431c1037b0 2024-04-12T05:49:42.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.408+0000 7f4303fff700 1 -- 172.21.15.32:0/4251063278 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4310013690 con 0x7f431c1037b0 2024-04-12T05:49:42.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.409+0000 7f4303fff700 1 --2- 172.21.15.32:0/4251063278 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f430406cbc0 0x7f430406f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:42.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.409+0000 7f4303fff700 1 -- 172.21.15.32:0/4251063278 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f431008d330 con 0x7f431c1037b0 2024-04-12T05:49:42.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.409+0000 7f431b7fe700 1 --2- 172.21.15.32:0/4251063278 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f430406cbc0 0x7f430406f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:42.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.409+0000 7f431b7fe700 1 --2- 172.21.15.32:0/4251063278 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f430406cbc0 0x7f430406f070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f430c004f40 tx=0x7f430c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:42.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.413+0000 7f4303fff700 1 -- 172.21.15.32:0/4251063278 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4310057320 con 0x7f431c1037b0 2024-04-12T05:49:42.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.701+0000 7f43220a8700 1 -- 172.21.15.32:0/4251063278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f431c109de0 con 0x7f431c1037b0 2024-04-12T05:49:42.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.702+0000 7f4303fff700 1 -- 172.21.15.32:0/4251063278 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f4310013420 con 0x7f431c1037b0 2024-04-12T05:49:42.697 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:49:42.697 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:49:42.697 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:49:42.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:49:42.697 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:49:42.697 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:49:42.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:49:42.698 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:49:42.698 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:49:42.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:49:42.698 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:49:42.698 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:49:42.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:49:42.698 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:49:42.698 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:49:42.698 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:49:42.698 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:49:42.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.704+0000 7f43220a8700 1 -- 172.21.15.32:0/4251063278 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f430406cbc0 msgr2=0x7f430406f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:42.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.704+0000 7f43220a8700 1 --2- 172.21.15.32:0/4251063278 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f430406cbc0 0x7f430406f070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f430c004f40 tx=0x7f430c00b040 comp rx=0 tx=0).stop 2024-04-12T05:49:42.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.704+0000 7f43220a8700 1 -- 172.21.15.32:0/4251063278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f431c1037b0 msgr2=0x7f431c070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:42.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.704+0000 7f43220a8700 1 --2- 172.21.15.32:0/4251063278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f431c1037b0 0x7f431c070800 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4310009390 tx=0x7f4310009e30 comp rx=0 tx=0).stop 2024-04-12T05:49:42.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.704+0000 7f43220a8700 1 -- 172.21.15.32:0/4251063278 shutdown_connections 2024-04-12T05:49:42.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.704+0000 7f43220a8700 1 --2- 172.21.15.32:0/4251063278 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f430406cbc0 0x7f430406f070 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:42.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.704+0000 7f43220a8700 1 --2- 172.21.15.32:0/4251063278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f431c102e10 0x7f431c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:42.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.704+0000 7f43220a8700 1 --2- 172.21.15.32:0/4251063278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f431c1037b0 0x7f431c070800 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:42.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.704+0000 7f43220a8700 1 -- 172.21.15.32:0/4251063278 >> 172.21.15.32:0/4251063278 conn(0x7f431c0f8980 msgr2=0x7f431c0f9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:42.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.704+0000 7f43220a8700 1 -- 172.21.15.32:0/4251063278 shutdown_connections 2024-04-12T05:49:42.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.704+0000 7f43220a8700 1 -- 172.21.15.32:0/4251063278 wait complete. 2024-04-12T05:49:42.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.808+0000 7f1ae0cbf700 1 -- 172.21.15.32:0/1072517611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1adc1014f0 msgr2=0x7f1adc1018c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:42.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.808+0000 7f1ae0cbf700 1 --2- 172.21.15.32:0/1072517611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1adc1014f0 0x7f1adc1018c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1acc004a50 tx=0x7f1acc00af40 comp rx=0 tx=0).stop 2024-04-12T05:49:42.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.808+0000 7f1ae0cbf700 1 -- 172.21.15.32:0/1072517611 shutdown_connections 2024-04-12T05:49:42.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.808+0000 7f1ae0cbf700 1 --2- 172.21.15.32:0/1072517611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1adc0fb4a0 0x7f1adc0fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:42.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.808+0000 7f1ae0cbf700 1 --2- 172.21.15.32:0/1072517611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1adc1014f0 0x7f1adc1018c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:42.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.808+0000 7f1ae0cbf700 1 -- 172.21.15.32:0/1072517611 >> 172.21.15.32:0/1072517611 conn(0x7f1adc0714c0 msgr2=0x7f1adc0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:42.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.808+0000 7f1ae0cbf700 1 -- 172.21.15.32:0/1072517611 shutdown_connections 2024-04-12T05:49:42.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.808+0000 7f1ae0cbf700 1 -- 172.21.15.32:0/1072517611 wait complete. 2024-04-12T05:49:42.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.808+0000 7f1ae0cbf700 1 Processor -- start 2024-04-12T05:49:42.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.808+0000 7f1ae0cbf700 1 -- start start 2024-04-12T05:49:42.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.809+0000 7f1ae0cbf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1adc0fb4a0 0x7f1adc115bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:42.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.809+0000 7f1ae0cbf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1adc1014f0 0x7f1adc1160f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:42.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.809+0000 7f1ae0cbf700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1adc116740 con 0x7f1adc0fb4a0 2024-04-12T05:49:42.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.809+0000 7f1ae0cbf700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1adc10fb50 con 0x7f1adc1014f0 2024-04-12T05:49:42.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.809+0000 7f1ada59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1adc0fb4a0 0x7f1adc115bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:42.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.809+0000 7f1ada59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1adc0fb4a0 0x7f1adc115bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36172/0 (socket says 172.21.15.32:36172) 2024-04-12T05:49:42.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.809+0000 7f1ada59c700 1 -- 172.21.15.32:0/3715715932 learned_addr learned my addr 172.21.15.32:0/3715715932 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:49:42.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.809+0000 7f1ad9d9b700 1 --2- 172.21.15.32:0/3715715932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1adc1014f0 0x7f1adc1160f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:42.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.809+0000 7f1ada59c700 1 -- 172.21.15.32:0/3715715932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1adc1014f0 msgr2=0x7f1adc1160f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:42.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.809+0000 7f1ada59c700 1 --2- 172.21.15.32:0/3715715932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1adc1014f0 0x7f1adc1160f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:42.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.809+0000 7f1ada59c700 1 -- 172.21.15.32:0/3715715932 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1acc005040 con 0x7f1adc0fb4a0 2024-04-12T05:49:42.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.809+0000 7f1ada59c700 1 --2- 172.21.15.32:0/3715715932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1adc0fb4a0 0x7f1adc115bb0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1acc00c870 tx=0x7f1acc00af40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:42.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.809+0000 7f1acaffd700 1 -- 172.21.15.32:0/3715715932 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1acc00f070 con 0x7f1adc0fb4a0 2024-04-12T05:49:42.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.809+0000 7f1acaffd700 1 -- 172.21.15.32:0/3715715932 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1acc00cdc0 con 0x7f1adc0fb4a0 2024-04-12T05:49:42.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.809+0000 7f1ae0cbf700 1 -- 172.21.15.32:0/3715715932 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1adc10fdd0 con 0x7f1adc0fb4a0 2024-04-12T05:49:42.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.809+0000 7f1acaffd700 1 -- 172.21.15.32:0/3715715932 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1acc01d7e0 con 0x7f1adc0fb4a0 2024-04-12T05:49:42.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.810+0000 7f1ae0cbf700 1 -- 172.21.15.32:0/3715715932 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1adc110390 con 0x7f1adc0fb4a0 2024-04-12T05:49:42.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.810+0000 7f1ae0cbf700 1 -- 172.21.15.32:0/3715715932 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1adc0fcbd0 con 0x7f1adc0fb4a0 2024-04-12T05:49:42.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.811+0000 7f1acaffd700 1 -- 172.21.15.32:0/3715715932 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1acc014630 con 0x7f1adc0fb4a0 2024-04-12T05:49:42.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.811+0000 7f1acaffd700 1 --2- 172.21.15.32:0/3715715932 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1ac406cc20 0x7f1ac406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:42.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.812+0000 7f1acaffd700 1 -- 172.21.15.32:0/3715715932 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f1acc08d420 con 0x7f1adc0fb4a0 2024-04-12T05:49:42.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.812+0000 7f1ad9d9b700 1 --2- 172.21.15.32:0/3715715932 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1ac406cc20 0x7f1ac406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:42.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.812+0000 7f1ad9d9b700 1 --2- 172.21.15.32:0/3715715932 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1ac406cc20 0x7f1ac406f0d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1adc111450 tx=0x7f1ad000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:42.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:42.816+0000 7f1acaffd700 1 -- 172.21.15.32:0/3715715932 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1acc057400 con 0x7f1adc0fb4a0 2024-04-12T05:49:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.062+0000 7f1ae0cbf700 1 -- 172.21.15.32:0/3715715932 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1adc04bed0 con 0x7f1adc0fb4a0 2024-04-12T05:49:43.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.062+0000 7f1acaffd700 1 -- 172.21.15.32:0/3715715932 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1835 (secure 0 0 0) 0x7f1acc05aa20 con 0x7f1adc0fb4a0 2024-04-12T05:49:43.058 INFO:teuthology.orchestra.run.smithi032.stdout:e33 2024-04-12T05:49:43.058 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:49:43.058 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:49:43.058 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:49:43.058 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:49:43.058 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:49:43.058 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:49:43.058 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 24 2024-04-12T05:49:43.058 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:49:43.058 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:49:43.058 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:46:10.668569+0000 2024-04-12T05:49:43.059 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:49:43.059 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:49:43.059 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:49:43.059 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:49:43.059 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:49:43.059 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:49:43.059 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:49:43.059 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:49:43.059 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T05:49:43.059 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:49:43.059 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:49:43.059 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:49:43.060 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T05:49:43.060 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-12T05:49:43.060 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:49:43.060 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:49:43.060 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:49:43.060 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:49:43.060 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:49:43.060 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:49:43.060 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:49:43.060 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:49:43.060 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:49:43.060 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:49:43.060 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-12T05:49:43.060 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:49:43.061 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:49:43.061 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:49:43.061 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{-1:44317} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:49:43.061 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:49:43.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.065+0000 7f1ae0cbf700 1 -- 172.21.15.32:0/3715715932 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1ac406cc20 msgr2=0x7f1ac406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:43.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.065+0000 7f1ae0cbf700 1 --2- 172.21.15.32:0/3715715932 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1ac406cc20 0x7f1ac406f0d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1adc111450 tx=0x7f1ad000b040 comp rx=0 tx=0).stop 2024-04-12T05:49:43.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.065+0000 7f1ae0cbf700 1 -- 172.21.15.32:0/3715715932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1adc0fb4a0 msgr2=0x7f1adc115bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:43.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.065+0000 7f1ae0cbf700 1 --2- 172.21.15.32:0/3715715932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1adc0fb4a0 0x7f1adc115bb0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1acc00c870 tx=0x7f1acc00af40 comp rx=0 tx=0).stop 2024-04-12T05:49:43.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.065+0000 7f1ae0cbf700 1 -- 172.21.15.32:0/3715715932 shutdown_connections 2024-04-12T05:49:43.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.065+0000 7f1ae0cbf700 1 --2- 172.21.15.32:0/3715715932 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1ac406cc20 0x7f1ac406f0d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:43.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.065+0000 7f1ae0cbf700 1 --2- 172.21.15.32:0/3715715932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1adc0fb4a0 0x7f1adc115bb0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:43.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.065+0000 7f1ae0cbf700 1 --2- 172.21.15.32:0/3715715932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1adc1014f0 0x7f1adc1160f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:43.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.065+0000 7f1ae0cbf700 1 -- 172.21.15.32:0/3715715932 >> 172.21.15.32:0/3715715932 conn(0x7f1adc0714c0 msgr2=0x7f1adc0f9490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:43.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.065+0000 7f1ae0cbf700 1 -- 172.21.15.32:0/3715715932 shutdown_connections 2024-04-12T05:49:43.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.065+0000 7f1ae0cbf700 1 -- 172.21.15.32:0/3715715932 wait complete. 2024-04-12T05:49:43.062 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-12T05:49:43.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.169+0000 7fea1350d700 1 -- 172.21.15.32:0/2705622128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea0c102710 msgr2=0x7fea0c0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:43.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.169+0000 7fea1350d700 1 --2- 172.21.15.32:0/2705622128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea0c102710 0x7fea0c0fbd80 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fea00004940 tx=0x7fea00009e30 comp rx=0 tx=0).stop 2024-04-12T05:49:43.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.169+0000 7fea1350d700 1 -- 172.21.15.32:0/2705622128 shutdown_connections 2024-04-12T05:49:43.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.169+0000 7fea1350d700 1 --2- 172.21.15.32:0/2705622128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea0c102710 0x7fea0c0fbd80 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:43.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.169+0000 7fea1350d700 1 --2- 172.21.15.32:0/2705622128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea0c101d70 0x7fea0c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:43.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.169+0000 7fea1350d700 1 -- 172.21.15.32:0/2705622128 >> 172.21.15.32:0/2705622128 conn(0x7fea0c072bc0 msgr2=0x7fea0c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:43.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.169+0000 7fea1350d700 1 -- 172.21.15.32:0/2705622128 shutdown_connections 2024-04-12T05:49:43.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.169+0000 7fea1350d700 1 -- 172.21.15.32:0/2705622128 wait complete. 2024-04-12T05:49:43.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.169+0000 7fea1350d700 1 Processor -- start 2024-04-12T05:49:43.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.170+0000 7fea1350d700 1 -- start start 2024-04-12T05:49:43.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.170+0000 7fea1350d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea0c101d70 0x7fea0c112900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:43.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.170+0000 7fea1350d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea0c102710 0x7fea0c112e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:43.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.170+0000 7fea1350d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea0c113490 con 0x7fea0c102710 2024-04-12T05:49:43.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.170+0000 7fea1350d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea0c10c880 con 0x7fea0c101d70 2024-04-12T05:49:43.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.170+0000 7fea10aa8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea0c102710 0x7fea0c112e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:43.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.170+0000 7fea10aa8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea0c102710 0x7fea0c112e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36198/0 (socket says 172.21.15.32:36198) 2024-04-12T05:49:43.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.170+0000 7fea10aa8700 1 -- 172.21.15.32:0/401529881 learned_addr learned my addr 172.21.15.32:0/401529881 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:49:43.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.170+0000 7fea112a9700 1 --2- 172.21.15.32:0/401529881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea0c101d70 0x7fea0c112900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:43.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.170+0000 7fea10aa8700 1 -- 172.21.15.32:0/401529881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea0c101d70 msgr2=0x7fea0c112900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:43.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.170+0000 7fea10aa8700 1 --2- 172.21.15.32:0/401529881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea0c101d70 0x7fea0c112900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:43.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.170+0000 7fea10aa8700 1 -- 172.21.15.32:0/401529881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea00005040 con 0x7fea0c102710 2024-04-12T05:49:43.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.170+0000 7fea10aa8700 1 --2- 172.21.15.32:0/401529881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea0c102710 0x7fea0c112e40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fea00000c00 tx=0x7fea00009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:43.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.171+0000 7fe9fdffb700 1 -- 172.21.15.32:0/401529881 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea00021070 con 0x7fea0c102710 2024-04-12T05:49:43.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.171+0000 7fea1350d700 1 -- 172.21.15.32:0/401529881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea0c10cb00 con 0x7fea0c102710 2024-04-12T05:49:43.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.171+0000 7fe9fdffb700 1 -- 172.21.15.32:0/401529881 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea00013de0 con 0x7fea0c102710 2024-04-12T05:49:43.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.171+0000 7fe9fdffb700 1 -- 172.21.15.32:0/401529881 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea0000e070 con 0x7fea0c102710 2024-04-12T05:49:43.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.171+0000 7fea1350d700 1 -- 172.21.15.32:0/401529881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea0c10d060 con 0x7fea0c102710 2024-04-12T05:49:43.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.172+0000 7fea1350d700 1 -- 172.21.15.32:0/401529881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea0c0ff080 con 0x7fea0c102710 2024-04-12T05:49:43.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.172+0000 7fe9fdffb700 1 -- 172.21.15.32:0/401529881 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fea0001c600 con 0x7fea0c102710 2024-04-12T05:49:43.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.173+0000 7fe9fdffb700 1 --2- 172.21.15.32:0/401529881 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe9f806cc20 0x7fe9f806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:43.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.173+0000 7fe9fdffb700 1 -- 172.21.15.32:0/401529881 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fea0008d190 con 0x7fea0c102710 2024-04-12T05:49:43.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.173+0000 7fea112a9700 1 --2- 172.21.15.32:0/401529881 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe9f806cc20 0x7fe9f806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:43.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.173+0000 7fea112a9700 1 --2- 172.21.15.32:0/401529881 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe9f806cc20 0x7fe9f806f0d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fea08009b80 tx=0x7fea0800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:43.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.177+0000 7fe9fdffb700 1 -- 172.21.15.32:0/401529881 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fea0005ab60 con 0x7fea0c102710 2024-04-12T05:49:43.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:42 smithi067 ceph-mon[112979]: pgmap v276: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:43.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:42 smithi067 ceph-mon[112979]: from='client.34568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:49:43.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:42 smithi067 ceph-mon[112979]: from='client.34572 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:49:43.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:42 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/4251063278' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:43.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:42 smithi032 ceph-mon[146817]: pgmap v276: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:43.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:42 smithi032 ceph-mon[146817]: from='client.34568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:49:43.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:42 smithi032 ceph-mon[146817]: from='client.34572 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:49:43.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:42 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/4251063278' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:43.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.396+0000 7fea1350d700 1 -- 172.21.15.32:0/401529881 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fea0c10df50 con 0x7fe9f806cc20 2024-04-12T05:49:43.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.397+0000 7fe9fdffb700 1 -- 172.21.15.32:0/401529881 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fea0c10df50 con 0x7fe9f806cc20 2024-04-12T05:49:43.392 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:49:43.392 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:49:43.392 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:49:43.392 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:49:43.392 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:49:43.392 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-12T05:49:43.393 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:49:43.393 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-12T05:49:43.393 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:49:43.393 INFO:teuthology.orchestra.run.smithi032.stdout: "mds" 2024-04-12T05:49:43.393 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:49:43.393 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:49:43.393 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:49:43.393 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:49:43.393 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:49:43.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.399+0000 7fea1350d700 1 -- 172.21.15.32:0/401529881 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe9f806cc20 msgr2=0x7fe9f806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:43.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.399+0000 7fea1350d700 1 --2- 172.21.15.32:0/401529881 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe9f806cc20 0x7fe9f806f0d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fea08009b80 tx=0x7fea0800b040 comp rx=0 tx=0).stop 2024-04-12T05:49:43.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.399+0000 7fea1350d700 1 -- 172.21.15.32:0/401529881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea0c102710 msgr2=0x7fea0c112e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:43.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.399+0000 7fea1350d700 1 --2- 172.21.15.32:0/401529881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea0c102710 0x7fea0c112e40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fea00000c00 tx=0x7fea00009e30 comp rx=0 tx=0).stop 2024-04-12T05:49:43.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.399+0000 7fea1350d700 1 -- 172.21.15.32:0/401529881 shutdown_connections 2024-04-12T05:49:43.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.399+0000 7fea1350d700 1 --2- 172.21.15.32:0/401529881 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe9f806cc20 0x7fe9f806f0d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:43.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.399+0000 7fea1350d700 1 --2- 172.21.15.32:0/401529881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea0c101d70 0x7fea0c112900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:43.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.399+0000 7fea1350d700 1 --2- 172.21.15.32:0/401529881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea0c102710 0x7fea0c112e40 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:43.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.399+0000 7fea1350d700 1 -- 172.21.15.32:0/401529881 >> 172.21.15.32:0/401529881 conn(0x7fea0c072bc0 msgr2=0x7fea0c0faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:43.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.399+0000 7fea1350d700 1 -- 172.21.15.32:0/401529881 shutdown_connections 2024-04-12T05:49:43.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.399+0000 7fea1350d700 1 -- 172.21.15.32:0/401529881 wait complete. 2024-04-12T05:49:43.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.503+0000 7f09ab902700 1 -- 172.21.15.32:0/2691819338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09a4101d70 msgr2=0x7f09a4102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:43.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.503+0000 7f09ab902700 1 --2- 172.21.15.32:0/2691819338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09a4101d70 0x7f09a4102140 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f09a0004940 tx=0x7f09a0009e30 comp rx=0 tx=0).stop 2024-04-12T05:49:43.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.503+0000 7f09ab902700 1 -- 172.21.15.32:0/2691819338 shutdown_connections 2024-04-12T05:49:43.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.503+0000 7f09ab902700 1 --2- 172.21.15.32:0/2691819338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09a4102710 0x7f09a40fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:43.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.503+0000 7f09ab902700 1 --2- 172.21.15.32:0/2691819338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09a4101d70 0x7f09a4102140 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:43.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.503+0000 7f09ab902700 1 -- 172.21.15.32:0/2691819338 >> 172.21.15.32:0/2691819338 conn(0x7f09a4072bc0 msgr2=0x7f09a4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:43.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.503+0000 7f09ab902700 1 -- 172.21.15.32:0/2691819338 shutdown_connections 2024-04-12T05:49:43.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.503+0000 7f09ab902700 1 -- 172.21.15.32:0/2691819338 wait complete. 2024-04-12T05:49:43.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.504+0000 7f09ab902700 1 Processor -- start 2024-04-12T05:49:43.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.504+0000 7f09ab902700 1 -- start start 2024-04-12T05:49:43.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.504+0000 7f09ab902700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09a4101d70 0x7f09a410eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:43.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.504+0000 7f09ab902700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09a4102710 0x7f09a410f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:43.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.504+0000 7f09ab902700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a410f5e0 con 0x7f09a4101d70 2024-04-12T05:49:43.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.504+0000 7f09ab902700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a410f720 con 0x7f09a4102710 2024-04-12T05:49:43.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.504+0000 7f09a969e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09a4101d70 0x7f09a410eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:43.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.504+0000 7f09a969e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09a4101d70 0x7f09a410eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36220/0 (socket says 172.21.15.32:36220) 2024-04-12T05:49:43.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.504+0000 7f09a969e700 1 -- 172.21.15.32:0/194617634 learned_addr learned my addr 172.21.15.32:0/194617634 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:49:43.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.504+0000 7f09a8e9d700 1 --2- 172.21.15.32:0/194617634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09a4102710 0x7f09a410f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:43.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.504+0000 7f09a969e700 1 -- 172.21.15.32:0/194617634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09a4102710 msgr2=0x7f09a410f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:43.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.504+0000 7f09a969e700 1 --2- 172.21.15.32:0/194617634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09a4102710 0x7f09a410f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:43.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.505+0000 7f09a969e700 1 -- 172.21.15.32:0/194617634 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a0005040 con 0x7f09a4101d70 2024-04-12T05:49:43.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.505+0000 7f09a8e9d700 1 --2- 172.21.15.32:0/194617634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09a4102710 0x7f09a410f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:49:43.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.505+0000 7f09a969e700 1 --2- 172.21.15.32:0/194617634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09a4101d70 0x7f09a410eb60 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f09a0009390 tx=0x7f09a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:43.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.505+0000 7f0999ffb700 1 -- 172.21.15.32:0/194617634 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09a000e040 con 0x7f09a4101d70 2024-04-12T05:49:43.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.505+0000 7f09ab902700 1 -- 172.21.15.32:0/194617634 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a4117a10 con 0x7f09a4101d70 2024-04-12T05:49:43.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.505+0000 7f0999ffb700 1 -- 172.21.15.32:0/194617634 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f09a0013420 con 0x7f09a4101d70 2024-04-12T05:49:43.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.505+0000 7f0999ffb700 1 -- 172.21.15.32:0/194617634 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09a001c630 con 0x7f09a4101d70 2024-04-12T05:49:43.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.505+0000 7f09ab902700 1 -- 172.21.15.32:0/194617634 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a4117fa0 con 0x7f09a4101d70 2024-04-12T05:49:43.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.506+0000 7f09ab902700 1 -- 172.21.15.32:0/194617634 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09a40ff080 con 0x7f09a4101d70 2024-04-12T05:49:43.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.507+0000 7f0999ffb700 1 -- 172.21.15.32:0/194617634 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f09a00139e0 con 0x7f09a4101d70 2024-04-12T05:49:43.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.507+0000 7f0999ffb700 1 --2- 172.21.15.32:0/194617634 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f099006cc30 0x7f099006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:49:43.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.507+0000 7f0999ffb700 1 -- 172.21.15.32:0/194617634 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f09a008d220 con 0x7f09a4101d70 2024-04-12T05:49:43.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.507+0000 7f09a8e9d700 1 --2- 172.21.15.32:0/194617634 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f099006cc30 0x7f099006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:43.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.508+0000 7f09a8e9d700 1 --2- 172.21.15.32:0/194617634 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f099006cc30 0x7f099006f0e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0994007fd0 tx=0x7f0994009f60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:43.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.512+0000 7f0999ffb700 1 -- 172.21.15.32:0/194617634 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f09a0057350 con 0x7f09a4101d70 2024-04-12T05:49:43.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.800+0000 7f09ab902700 1 -- 172.21.15.32:0/194617634 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f09a40fd4c0 con 0x7f09a4101d70 2024-04-12T05:49:43.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.800+0000 7f0999ffb700 1 -- 172.21.15.32:0/194617634 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f09a005a970 con 0x7f09a4101d70 2024-04-12T05:49:43.795 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:49:43.795 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:49:43.795 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-12T05:49:43.795 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:49:43.795 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:49:43.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.802+0000 7f09ab902700 1 -- 172.21.15.32:0/194617634 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f099006cc30 msgr2=0x7f099006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:43.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.802+0000 7f09ab902700 1 --2- 172.21.15.32:0/194617634 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f099006cc30 0x7f099006f0e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0994007fd0 tx=0x7f0994009f60 comp rx=0 tx=0).stop 2024-04-12T05:49:43.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.802+0000 7f09ab902700 1 -- 172.21.15.32:0/194617634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09a4101d70 msgr2=0x7f09a410eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:43.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.802+0000 7f09ab902700 1 --2- 172.21.15.32:0/194617634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09a4101d70 0x7f09a410eb60 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f09a0009390 tx=0x7f09a0009e30 comp rx=0 tx=0).stop 2024-04-12T05:49:43.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.802+0000 7f09ab902700 1 -- 172.21.15.32:0/194617634 shutdown_connections 2024-04-12T05:49:43.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.802+0000 7f09ab902700 1 --2- 172.21.15.32:0/194617634 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f099006cc30 0x7f099006f0e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:43.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.802+0000 7f09ab902700 1 --2- 172.21.15.32:0/194617634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09a4101d70 0x7f09a410eb60 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:43.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.802+0000 7f09ab902700 1 --2- 172.21.15.32:0/194617634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09a4102710 0x7f09a410f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:49:43.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.802+0000 7f09ab902700 1 -- 172.21.15.32:0/194617634 >> 172.21.15.32:0/194617634 conn(0x7f09a4072bc0 msgr2=0x7f09a40faf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:43.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.802+0000 7f09ab902700 1 -- 172.21.15.32:0/194617634 shutdown_connections 2024-04-12T05:49:43.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:49:43.803+0000 7f09ab902700 1 -- 172.21.15.32:0/194617634 wait complete. 2024-04-12T05:49:44.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:43 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3715715932' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:49:44.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:43 smithi067 ceph-mon[112979]: from='client.34584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:49:44.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:43 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/194617634' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:49:44.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:43 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3715715932' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:49:44.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:43 smithi032 ceph-mon[146817]: from='client.34584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:49:44.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:43 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/194617634' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:49:45.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:44 smithi067 ceph-mon[112979]: pgmap v277: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:45.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:44 smithi032 ceph-mon[146817]: pgmap v277: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:47.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:46 smithi067 ceph-mon[112979]: pgmap v278: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:47.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:46 smithi032 ceph-mon[146817]: pgmap v278: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:48.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:47 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:49:48.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:47 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:49:49.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:48 smithi067 ceph-mon[112979]: pgmap v279: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:49.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:48 smithi032 ceph-mon[146817]: pgmap v279: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:51.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:50 smithi067 ceph-mon[112979]: pgmap v280: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:51.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:50 smithi032 ceph-mon[146817]: pgmap v280: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:53.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:52 smithi067 ceph-mon[112979]: pgmap v281: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:53.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:52 smithi032 ceph-mon[146817]: pgmap v281: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:55.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:54 smithi067 ceph-mon[112979]: pgmap v282: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:55.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:54 smithi032 ceph-mon[146817]: pgmap v282: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:57.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:56 smithi067 ceph-mon[112979]: pgmap v283: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:57.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:56 smithi032 ceph-mon[146817]: pgmap v283: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:59.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:49:58 smithi067 ceph-mon[112979]: pgmap v284: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:59.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:49:58 smithi032 ceph-mon[146817]: pgmap v284: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:00.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:00 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-mon-smithi032[146794]: 2024-04-12T05:50:00.001+0000 7f3eb2941700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:50:00.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:00 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-mon-smithi032[146794]: 2024-04-12T05:50:00.001+0000 7f3eb2941700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:50:00.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:00 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-mon-smithi032[146794]: 2024-04-12T05:50:00.001+0000 7f3eb2941700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2024-04-12T05:50:00.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:00 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-mon-smithi032[146794]: 2024-04-12T05:50:00.001+0000 7f3eb2941700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:50:00.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:00 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-mon-smithi032[146794]: 2024-04-12T05:50:00.001+0000 7f3eb2941700 -1 log_channel(cluster) log [ERR] : fs cephfs is offline because no MDS is active for it. 2024-04-12T05:50:01.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:01 smithi067 ceph-mon[112979]: pgmap v285: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:01.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:01 smithi067 ceph-mon[112979]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:50:01.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:01 smithi067 ceph-mon[112979]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:50:01.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:01 smithi067 ceph-mon[112979]: fs cephfs is degraded 2024-04-12T05:50:01.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:01 smithi067 ceph-mon[112979]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:50:01.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:01 smithi067 ceph-mon[112979]: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:50:01.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:01 smithi032 ceph-mon[146817]: pgmap v285: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:01.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:01 smithi032 ceph-mon[146817]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:50:01.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:01 smithi032 ceph-mon[146817]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:50:01.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:01 smithi032 ceph-mon[146817]: fs cephfs is degraded 2024-04-12T05:50:01.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:01 smithi032 ceph-mon[146817]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:50:01.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:01 smithi032 ceph-mon[146817]: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:50:03.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:03 smithi067 ceph-mon[112979]: pgmap v286: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:03.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:50:03.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:03 smithi032 ceph-mon[146817]: pgmap v286: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:03.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:50:05.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:05 smithi067 ceph-mon[112979]: pgmap v287: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:05.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:05 smithi032 ceph-mon[146817]: pgmap v287: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:07.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:07 smithi067 ceph-mon[112979]: pgmap v288: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:07.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:07 smithi032 ceph-mon[146817]: pgmap v288: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:09.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:09 smithi067 ceph-mon[112979]: pgmap v289: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:09.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:09 smithi032 ceph-mon[146817]: pgmap v289: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:11.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:11 smithi067 ceph-mon[112979]: pgmap v290: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:11.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:11 smithi032 ceph-mon[146817]: pgmap v290: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:13.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:13 smithi067 ceph-mon[112979]: pgmap v291: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:13.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:13 smithi032 ceph-mon[146817]: pgmap v291: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:13.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.922+0000 7f60c7777700 1 -- 172.21.15.32:0/4161466108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c00fb4a0 msgr2=0x7f60c00fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:13.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.922+0000 7f60c7777700 1 --2- 172.21.15.32:0/4161466108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c00fb4a0 0x7f60c00fb910 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f60bc004940 tx=0x7f60bc009e30 comp rx=0 tx=0).stop 2024-04-12T05:50:13.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.922+0000 7f60c7777700 1 -- 172.21.15.32:0/4161466108 shutdown_connections 2024-04-12T05:50:13.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.922+0000 7f60c7777700 1 --2- 172.21.15.32:0/4161466108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c00fb4a0 0x7f60c00fb910 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:13.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.922+0000 7f60c7777700 1 --2- 172.21.15.32:0/4161466108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60c01014f0 0x7f60c01018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:13.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.922+0000 7f60c7777700 1 -- 172.21.15.32:0/4161466108 >> 172.21.15.32:0/4161466108 conn(0x7f60c00714c0 msgr2=0x7f60c00718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:50:13.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.922+0000 7f60c7777700 1 -- 172.21.15.32:0/4161466108 shutdown_connections 2024-04-12T05:50:13.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.922+0000 7f60c7777700 1 -- 172.21.15.32:0/4161466108 wait complete. 2024-04-12T05:50:13.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.923+0000 7f60c7777700 1 Processor -- start 2024-04-12T05:50:13.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.923+0000 7f60c7777700 1 -- start start 2024-04-12T05:50:13.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.923+0000 7f60c7777700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c00fb4a0 0x7f60c010ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:13.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.923+0000 7f60c7777700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60c01014f0 0x7f60c010efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:13.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.923+0000 7f60c7777700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c010f610 con 0x7f60c00fb4a0 2024-04-12T05:50:13.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.923+0000 7f60c7777700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c010f750 con 0x7f60c01014f0 2024-04-12T05:50:13.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.923+0000 7f60c5513700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c00fb4a0 0x7f60c010ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:13.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.923+0000 7f60c5513700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c00fb4a0 0x7f60c010ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33316/0 (socket says 172.21.15.32:33316) 2024-04-12T05:50:13.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.923+0000 7f60c5513700 1 -- 172.21.15.32:0/2827440658 learned_addr learned my addr 172.21.15.32:0/2827440658 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:50:13.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.923+0000 7f60c4d12700 1 --2- 172.21.15.32:0/2827440658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60c01014f0 0x7f60c010efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:13.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.923+0000 7f60c5513700 1 -- 172.21.15.32:0/2827440658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60c01014f0 msgr2=0x7f60c010efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:13.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.923+0000 7f60c5513700 1 --2- 172.21.15.32:0/2827440658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60c01014f0 0x7f60c010efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:13.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.923+0000 7f60c5513700 1 -- 172.21.15.32:0/2827440658 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60bc005040 con 0x7f60c00fb4a0 2024-04-12T05:50:13.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.923+0000 7f60c4d12700 1 --2- 172.21.15.32:0/2827440658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60c01014f0 0x7f60c010efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:50:13.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.924+0000 7f60c5513700 1 --2- 172.21.15.32:0/2827440658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c00fb4a0 0x7f60c010ea80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f60b0002a50 tx=0x7f60b00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:50:13.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.924+0000 7f60b5ffb700 1 -- 172.21.15.32:0/2827440658 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60b0007cc0 con 0x7f60c00fb4a0 2024-04-12T05:50:13.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.924+0000 7f60b5ffb700 1 -- 172.21.15.32:0/2827440658 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f60b000f040 con 0x7f60c00fb4a0 2024-04-12T05:50:13.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.924+0000 7f60c7777700 1 -- 172.21.15.32:0/2827440658 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60c0117af0 con 0x7f60c00fb4a0 2024-04-12T05:50:13.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.924+0000 7f60b5ffb700 1 -- 172.21.15.32:0/2827440658 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60b0009540 con 0x7f60c00fb4a0 2024-04-12T05:50:13.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.924+0000 7f60c7777700 1 -- 172.21.15.32:0/2827440658 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60c0118050 con 0x7f60c00fb4a0 2024-04-12T05:50:13.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.925+0000 7f60c7777700 1 -- 172.21.15.32:0/2827440658 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60c00fcbd0 con 0x7f60c00fb4a0 2024-04-12T05:50:13.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.926+0000 7f60b5ffb700 1 -- 172.21.15.32:0/2827440658 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f60b00096a0 con 0x7f60c00fb4a0 2024-04-12T05:50:13.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.926+0000 7f60b5ffb700 1 --2- 172.21.15.32:0/2827440658 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f60ac06cc30 0x7f60ac06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:13.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.926+0000 7f60b5ffb700 1 -- 172.21.15.32:0/2827440658 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f60b008bed0 con 0x7f60c00fb4a0 2024-04-12T05:50:13.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.926+0000 7f60c4d12700 1 --2- 172.21.15.32:0/2827440658 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f60ac06cc30 0x7f60ac06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:13.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.927+0000 7f60c4d12700 1 --2- 172.21.15.32:0/2827440658 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f60ac06cc30 0x7f60ac06f0e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f60bc004f90 tx=0x7f60bc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:50:13.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:13.931+0000 7f60b5ffb700 1 -- 172.21.15.32:0/2827440658 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f60b0055e30 con 0x7f60c00fb4a0 2024-04-12T05:50:14.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.155+0000 7f60c7777700 1 -- 172.21.15.32:0/2827440658 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f60c004bf70 con 0x7f60ac06cc30 2024-04-12T05:50:14.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.156+0000 7f60b5ffb700 1 -- 172.21.15.32:0/2827440658 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f60bc008e50 con 0x7f60ac06cc30 2024-04-12T05:50:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.159+0000 7f60c7777700 1 -- 172.21.15.32:0/2827440658 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f60ac06cc30 msgr2=0x7f60ac06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.159+0000 7f60c7777700 1 --2- 172.21.15.32:0/2827440658 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f60ac06cc30 0x7f60ac06f0e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f60bc004f90 tx=0x7f60bc009e30 comp rx=0 tx=0).stop 2024-04-12T05:50:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.159+0000 7f60c7777700 1 -- 172.21.15.32:0/2827440658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c00fb4a0 msgr2=0x7f60c010ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.159+0000 7f60c7777700 1 --2- 172.21.15.32:0/2827440658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c00fb4a0 0x7f60c010ea80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f60b0002a50 tx=0x7f60b00046b0 comp rx=0 tx=0).stop 2024-04-12T05:50:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.159+0000 7f60c7777700 1 -- 172.21.15.32:0/2827440658 shutdown_connections 2024-04-12T05:50:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.159+0000 7f60c7777700 1 --2- 172.21.15.32:0/2827440658 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f60ac06cc30 0x7f60ac06f0e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.159+0000 7f60c7777700 1 --2- 172.21.15.32:0/2827440658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c00fb4a0 0x7f60c010ea80 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.159+0000 7f60c7777700 1 --2- 172.21.15.32:0/2827440658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60c01014f0 0x7f60c010efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.159+0000 7f60c7777700 1 -- 172.21.15.32:0/2827440658 >> 172.21.15.32:0/2827440658 conn(0x7f60c00714c0 msgr2=0x7f60c00f9490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:50:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.159+0000 7f60c7777700 1 -- 172.21.15.32:0/2827440658 shutdown_connections 2024-04-12T05:50:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.159+0000 7f60c7777700 1 -- 172.21.15.32:0/2827440658 wait complete. 2024-04-12T05:50:14.165 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:50:14.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.263+0000 7fb504b9a700 1 -- 172.21.15.32:0/3780492245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb500101d70 msgr2=0x7fb500102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:14.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.263+0000 7fb504b9a700 1 --2- 172.21.15.32:0/3780492245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb500101d70 0x7fb500102140 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0004940 tx=0x7fb4f0009e30 comp rx=0 tx=0).stop 2024-04-12T05:50:14.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.263+0000 7fb504b9a700 1 -- 172.21.15.32:0/3780492245 shutdown_connections 2024-04-12T05:50:14.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.263+0000 7fb504b9a700 1 --2- 172.21.15.32:0/3780492245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb500102710 0x7fb5000fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:14.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.263+0000 7fb504b9a700 1 --2- 172.21.15.32:0/3780492245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb500101d70 0x7fb500102140 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:14.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.263+0000 7fb504b9a700 1 -- 172.21.15.32:0/3780492245 >> 172.21.15.32:0/3780492245 conn(0x7fb500072bc0 msgr2=0x7fb500072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:50:14.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.263+0000 7fb504b9a700 1 -- 172.21.15.32:0/3780492245 shutdown_connections 2024-04-12T05:50:14.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.264+0000 7fb504b9a700 1 -- 172.21.15.32:0/3780492245 wait complete. 2024-04-12T05:50:14.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.264+0000 7fb504b9a700 1 Processor -- start 2024-04-12T05:50:14.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.264+0000 7fb504b9a700 1 -- start start 2024-04-12T05:50:14.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.264+0000 7fb504b9a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb500101d70 0x7fb500113a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:14.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.264+0000 7fb504b9a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb500102710 0x7fb500113fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:14.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.264+0000 7fb504b9a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb500114620 con 0x7fb500101d70 2024-04-12T05:50:14.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.264+0000 7fb504b9a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb50010da60 con 0x7fb500102710 2024-04-12T05:50:14.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.264+0000 7fb4fe59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb500101d70 0x7fb500113a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:14.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.264+0000 7fb4fe59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb500101d70 0x7fb500113a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33338/0 (socket says 172.21.15.32:33338) 2024-04-12T05:50:14.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.264+0000 7fb4fe59c700 1 -- 172.21.15.32:0/353565389 learned_addr learned my addr 172.21.15.32:0/353565389 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:50:14.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.265+0000 7fb4fdd9b700 1 --2- 172.21.15.32:0/353565389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb500102710 0x7fb500113fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:14.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.265+0000 7fb4fe59c700 1 -- 172.21.15.32:0/353565389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb500102710 msgr2=0x7fb500113fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:14.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.265+0000 7fb4fe59c700 1 --2- 172.21.15.32:0/353565389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb500102710 0x7fb500113fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:14.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.265+0000 7fb4fe59c700 1 -- 172.21.15.32:0/353565389 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f0005040 con 0x7fb500101d70 2024-04-12T05:50:14.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.265+0000 7fb4fdd9b700 1 --2- 172.21.15.32:0/353565389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb500102710 0x7fb500113fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:50:14.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.265+0000 7fb4fe59c700 1 --2- 172.21.15.32:0/353565389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb500101d70 0x7fb500113a90 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb4f000b3b0 tx=0x7fb4f0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:50:14.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.265+0000 7fb4eeffd700 1 -- 172.21.15.32:0/353565389 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4f000e040 con 0x7fb500101d70 2024-04-12T05:50:14.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.265+0000 7fb504b9a700 1 -- 172.21.15.32:0/353565389 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb50010dce0 con 0x7fb500101d70 2024-04-12T05:50:14.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.265+0000 7fb4eeffd700 1 -- 172.21.15.32:0/353565389 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb4f0013420 con 0x7fb500101d70 2024-04-12T05:50:14.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.265+0000 7fb4eeffd700 1 -- 172.21.15.32:0/353565389 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4f001c630 con 0x7fb500101d70 2024-04-12T05:50:14.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.265+0000 7fb504b9a700 1 -- 172.21.15.32:0/353565389 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb50010e2a0 con 0x7fb500101d70 2024-04-12T05:50:14.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.266+0000 7fb504b9a700 1 -- 172.21.15.32:0/353565389 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5000fd4c0 con 0x7fb500101d70 2024-04-12T05:50:14.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.267+0000 7fb4eeffd700 1 -- 172.21.15.32:0/353565389 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb4f0013970 con 0x7fb500101d70 2024-04-12T05:50:14.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.267+0000 7fb4eeffd700 1 --2- 172.21.15.32:0/353565389 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb4e806cc20 0x7fb4e806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:14.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.267+0000 7fb4eeffd700 1 -- 172.21.15.32:0/353565389 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fb4f008d310 con 0x7fb500101d70 2024-04-12T05:50:14.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.267+0000 7fb4fdd9b700 1 --2- 172.21.15.32:0/353565389 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb4e806cc20 0x7fb4e806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:14.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.268+0000 7fb4fdd9b700 1 --2- 172.21.15.32:0/353565389 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb4e806cc20 0x7fb4e806f0d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4007fd0 tx=0x7fb4f4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:50:14.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.272+0000 7fb4eeffd700 1 -- 172.21.15.32:0/353565389 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb4f005ad80 con 0x7fb500101d70 2024-04-12T05:50:14.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.495+0000 7fb504b9a700 1 -- 172.21.15.32:0/353565389 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb500108da0 con 0x7fb4e806cc20 2024-04-12T05:50:14.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.496+0000 7fb4eeffd700 1 -- 172.21.15.32:0/353565389 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb500108da0 con 0x7fb4e806cc20 2024-04-12T05:50:14.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.498+0000 7fb504b9a700 1 -- 172.21.15.32:0/353565389 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb4e806cc20 msgr2=0x7fb4e806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:14.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.498+0000 7fb504b9a700 1 --2- 172.21.15.32:0/353565389 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb4e806cc20 0x7fb4e806f0d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4007fd0 tx=0x7fb4f4009040 comp rx=0 tx=0).stop 2024-04-12T05:50:14.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.498+0000 7fb504b9a700 1 -- 172.21.15.32:0/353565389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb500101d70 msgr2=0x7fb500113a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:14.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.498+0000 7fb504b9a700 1 --2- 172.21.15.32:0/353565389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb500101d70 0x7fb500113a90 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb4f000b3b0 tx=0x7fb4f0009e30 comp rx=0 tx=0).stop 2024-04-12T05:50:14.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.499+0000 7fb504b9a700 1 -- 172.21.15.32:0/353565389 shutdown_connections 2024-04-12T05:50:14.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.499+0000 7fb504b9a700 1 --2- 172.21.15.32:0/353565389 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb4e806cc20 0x7fb4e806f0d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:14.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.499+0000 7fb504b9a700 1 --2- 172.21.15.32:0/353565389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb500101d70 0x7fb500113a90 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:14.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.499+0000 7fb504b9a700 1 --2- 172.21.15.32:0/353565389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb500102710 0x7fb500113fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:14.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.499+0000 7fb504b9a700 1 -- 172.21.15.32:0/353565389 >> 172.21.15.32:0/353565389 conn(0x7fb500072bc0 msgr2=0x7fb500065a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:50:14.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.499+0000 7fb504b9a700 1 -- 172.21.15.32:0/353565389 shutdown_connections 2024-04-12T05:50:14.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.499+0000 7fb504b9a700 1 -- 172.21.15.32:0/353565389 wait complete. 2024-04-12T05:50:14.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.602+0000 7f6674e2d700 1 -- 172.21.15.32:0/3355011304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6670102e10 msgr2=0x7f66701031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:14.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.602+0000 7f6674e2d700 1 --2- 172.21.15.32:0/3355011304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6670102e10 0x7f66701031e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6664004940 tx=0x7f6664009e30 comp rx=0 tx=0).stop 2024-04-12T05:50:14.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.602+0000 7f6674e2d700 1 -- 172.21.15.32:0/3355011304 shutdown_connections 2024-04-12T05:50:14.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.602+0000 7f6674e2d700 1 --2- 172.21.15.32:0/3355011304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66701037b0 0x7f66700fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:14.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.602+0000 7f6674e2d700 1 --2- 172.21.15.32:0/3355011304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6670102e10 0x7f66701031e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:14.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.602+0000 7f6674e2d700 1 -- 172.21.15.32:0/3355011304 >> 172.21.15.32:0/3355011304 conn(0x7f66700f89a0 msgr2=0x7f66700fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:50:14.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.602+0000 7f6674e2d700 1 -- 172.21.15.32:0/3355011304 shutdown_connections 2024-04-12T05:50:14.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.602+0000 7f6674e2d700 1 -- 172.21.15.32:0/3355011304 wait complete. 2024-04-12T05:50:14.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.602+0000 7f6674e2d700 1 Processor -- start 2024-04-12T05:50:14.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.602+0000 7f6674e2d700 1 -- start start 2024-04-12T05:50:14.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.603+0000 7f6674e2d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6670102e10 0x7f66700702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:14.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.603+0000 7f6674e2d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66701037b0 0x7f6670070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:14.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.603+0000 7f6674e2d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6670070d90 con 0x7f6670102e10 2024-04-12T05:50:14.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.603+0000 7f6674e2d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6670070ed0 con 0x7f66701037b0 2024-04-12T05:50:14.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.603+0000 7f666e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6670102e10 0x7f66700702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:14.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.603+0000 7f666e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6670102e10 0x7f66700702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33358/0 (socket says 172.21.15.32:33358) 2024-04-12T05:50:14.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.603+0000 7f666e59c700 1 -- 172.21.15.32:0/2801343635 learned_addr learned my addr 172.21.15.32:0/2801343635 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:50:14.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.603+0000 7f666dd9b700 1 --2- 172.21.15.32:0/2801343635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66701037b0 0x7f6670070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:14.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.603+0000 7f666e59c700 1 -- 172.21.15.32:0/2801343635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66701037b0 msgr2=0x7f6670070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:14.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.603+0000 7f666e59c700 1 --2- 172.21.15.32:0/2801343635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66701037b0 0x7f6670070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:14.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.603+0000 7f666e59c700 1 -- 172.21.15.32:0/2801343635 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6664005040 con 0x7f6670102e10 2024-04-12T05:50:14.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.603+0000 7f666dd9b700 1 --2- 172.21.15.32:0/2801343635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66701037b0 0x7f6670070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:50:14.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.603+0000 7f666e59c700 1 --2- 172.21.15.32:0/2801343635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6670102e10 0x7f66700702c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6664000c00 tx=0x7f6664009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:50:14.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.603+0000 7f665effd700 1 -- 172.21.15.32:0/2801343635 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6664021070 con 0x7f6670102e10 2024-04-12T05:50:14.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.603+0000 7f665effd700 1 -- 172.21.15.32:0/2801343635 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6664013de0 con 0x7f6670102e10 2024-04-12T05:50:14.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.603+0000 7f6674e2d700 1 -- 172.21.15.32:0/2801343635 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6670071120 con 0x7f6670102e10 2024-04-12T05:50:14.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.603+0000 7f665effd700 1 -- 172.21.15.32:0/2801343635 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f666400e070 con 0x7f6670102e10 2024-04-12T05:50:14.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.603+0000 7f6674e2d700 1 -- 172.21.15.32:0/2801343635 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f667011a120 con 0x7f6670102e10 2024-04-12T05:50:14.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.604+0000 7f6674e2d700 1 -- 172.21.15.32:0/2801343635 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66700fe560 con 0x7f6670102e10 2024-04-12T05:50:14.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.605+0000 7f665effd700 1 -- 172.21.15.32:0/2801343635 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f666401c600 con 0x7f6670102e10 2024-04-12T05:50:14.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.605+0000 7f665effd700 1 --2- 172.21.15.32:0/2801343635 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f665806cc20 0x7f665806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:14.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.605+0000 7f665effd700 1 -- 172.21.15.32:0/2801343635 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f666408d090 con 0x7f6670102e10 2024-04-12T05:50:14.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.605+0000 7f666dd9b700 1 --2- 172.21.15.32:0/2801343635 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f665806cc20 0x7f665806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:14.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.606+0000 7f666dd9b700 1 --2- 172.21.15.32:0/2801343635 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f665806cc20 0x7f665806f0d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6660004600 tx=0x7f666000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:50:14.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.610+0000 7f665effd700 1 -- 172.21.15.32:0/2801343635 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6664057110 con 0x7f6670102e10 2024-04-12T05:50:14.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.821+0000 7f6674e2d700 1 -- 172.21.15.32:0/2801343635 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f667004bf70 con 0x7f665806cc20 2024-04-12T05:50:14.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.831+0000 7f665effd700 1 -- 172.21.15.32:0/2801343635 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f666000f040 con 0x7f665806cc20 2024-04-12T05:50:14.827 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:50:14.827 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (10m) 3m ago 21m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:50:14.827 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (7m) 3m ago 21m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:50:14.827 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (7m) 3m ago 20m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:50:14.827 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (9m) 3m ago 21m 87.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:50:14.827 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (3m) 3m ago 15m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T05:50:14.827 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (3m) 3m ago 15m 13.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T05:50:14.828 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (3m) 3m ago 15m 11.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 6d4aca6f74a0 2024-04-12T05:50:14.828 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (3m) 3m ago 15m 14.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T05:50:14.828 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (11m) 3m ago 22m 507M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:50:14.828 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (10m) 3m ago 20m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:50:14.828 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (8m) 3m ago 23m 53.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:50:14.828 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (8m) 3m ago 20m 43.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:50:14.828 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (10m) 3m ago 21m 16.9M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:50:14.828 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (10m) 3m ago 20m 18.4M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:50:14.828 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (7m) 3m ago 19m 91.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T05:50:14.828 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (6m) 3m ago 19m 97.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T05:50:14.828 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (5m) 3m ago 18m 81.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T05:50:14.828 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (5m) 3m ago 18m 113M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T05:50:14.828 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (4m) 3m ago 17m 90.3M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T05:50:14.828 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (4m) 3m ago 17m 84.8M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T05:50:14.829 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (10m) 3m ago 20m 64.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:50:14.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.833+0000 7f6674e2d700 1 -- 172.21.15.32:0/2801343635 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f665806cc20 msgr2=0x7f665806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:14.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.833+0000 7f6674e2d700 1 --2- 172.21.15.32:0/2801343635 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f665806cc20 0x7f665806f0d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6660004600 tx=0x7f666000b040 comp rx=0 tx=0).stop 2024-04-12T05:50:14.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.833+0000 7f6674e2d700 1 -- 172.21.15.32:0/2801343635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6670102e10 msgr2=0x7f66700702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:14.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.833+0000 7f6674e2d700 1 --2- 172.21.15.32:0/2801343635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6670102e10 0x7f66700702c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6664000c00 tx=0x7f6664009e30 comp rx=0 tx=0).stop 2024-04-12T05:50:14.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.833+0000 7f6674e2d700 1 -- 172.21.15.32:0/2801343635 shutdown_connections 2024-04-12T05:50:14.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.833+0000 7f6674e2d700 1 --2- 172.21.15.32:0/2801343635 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f665806cc20 0x7f665806f0d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:14.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.833+0000 7f6674e2d700 1 --2- 172.21.15.32:0/2801343635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6670102e10 0x7f66700702c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:14.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.833+0000 7f6674e2d700 1 --2- 172.21.15.32:0/2801343635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66701037b0 0x7f6670070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:14.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.833+0000 7f6674e2d700 1 -- 172.21.15.32:0/2801343635 >> 172.21.15.32:0/2801343635 conn(0x7f66700f89a0 msgr2=0x7f66700f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:50:14.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.833+0000 7f6674e2d700 1 -- 172.21.15.32:0/2801343635 shutdown_connections 2024-04-12T05:50:14.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.833+0000 7f6674e2d700 1 -- 172.21.15.32:0/2801343635 wait complete. 2024-04-12T05:50:14.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.936+0000 7f386dc9d700 1 -- 172.21.15.32:0/610950804 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38680fe830 msgr2=0x7f3868106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:14.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.936+0000 7f386dc9d700 1 --2- 172.21.15.32:0/610950804 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38680fe830 0x7f3868106e70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f385c004940 tx=0x7f385c009e30 comp rx=0 tx=0).stop 2024-04-12T05:50:14.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.936+0000 7f386dc9d700 1 -- 172.21.15.32:0/610950804 shutdown_connections 2024-04-12T05:50:14.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.936+0000 7f386dc9d700 1 --2- 172.21.15.32:0/610950804 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38680fe830 0x7f3868106e70 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:14.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.936+0000 7f386dc9d700 1 --2- 172.21.15.32:0/610950804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38680fde90 0x7f38680fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:14.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.936+0000 7f386dc9d700 1 -- 172.21.15.32:0/610950804 >> 172.21.15.32:0/610950804 conn(0x7f38680f78e0 msgr2=0x7f38680f9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:50:14.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.936+0000 7f386dc9d700 1 -- 172.21.15.32:0/610950804 shutdown_connections 2024-04-12T05:50:14.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.936+0000 7f386dc9d700 1 -- 172.21.15.32:0/610950804 wait complete. 2024-04-12T05:50:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.937+0000 7f386dc9d700 1 Processor -- start 2024-04-12T05:50:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.937+0000 7f386dc9d700 1 -- start start 2024-04-12T05:50:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.937+0000 7f386dc9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38680fde90 0x7f3868113840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.937+0000 7f386dc9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38680fe830 0x7f3868113d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.937+0000 7f386dc9d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38681143d0 con 0x7f38680fe830 2024-04-12T05:50:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.937+0000 7f386dc9d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386810d9d0 con 0x7f38680fde90 2024-04-12T05:50:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.938+0000 7f3866ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38680fe830 0x7f3868113d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.938+0000 7f3866ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38680fe830 0x7f3868113d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33372/0 (socket says 172.21.15.32:33372) 2024-04-12T05:50:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.938+0000 7f3866ffd700 1 -- 172.21.15.32:0/1815096659 learned_addr learned my addr 172.21.15.32:0/1815096659 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:50:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.938+0000 7f38677fe700 1 --2- 172.21.15.32:0/1815096659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38680fde90 0x7f3868113840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.938+0000 7f3866ffd700 1 -- 172.21.15.32:0/1815096659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38680fde90 msgr2=0x7f3868113840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.938+0000 7f3866ffd700 1 --2- 172.21.15.32:0/1815096659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38680fde90 0x7f3868113840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.938+0000 7f3866ffd700 1 -- 172.21.15.32:0/1815096659 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f385c005040 con 0x7f38680fe830 2024-04-12T05:50:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.938+0000 7f38677fe700 1 --2- 172.21.15.32:0/1815096659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38680fde90 0x7f3868113840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:50:14.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.938+0000 7f3866ffd700 1 --2- 172.21.15.32:0/1815096659 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38680fe830 0x7f3868113d80 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f385c004890 tx=0x7f385c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:50:14.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.938+0000 7f384ffff700 1 -- 172.21.15.32:0/1815096659 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f385c00e040 con 0x7f38680fe830 2024-04-12T05:50:14.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.938+0000 7f386dc9d700 1 -- 172.21.15.32:0/1815096659 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f386810dcb0 con 0x7f38680fe830 2024-04-12T05:50:14.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.938+0000 7f384ffff700 1 -- 172.21.15.32:0/1815096659 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f385c009af0 con 0x7f38680fe830 2024-04-12T05:50:14.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.938+0000 7f384ffff700 1 -- 172.21.15.32:0/1815096659 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f385c01c3f0 con 0x7f38680fe830 2024-04-12T05:50:14.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.938+0000 7f386dc9d700 1 -- 172.21.15.32:0/1815096659 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f386810e1f0 con 0x7f38680fe830 2024-04-12T05:50:14.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.939+0000 7f386dc9d700 1 -- 172.21.15.32:0/1815096659 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3868066af0 con 0x7f38680fe830 2024-04-12T05:50:14.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.940+0000 7f384ffff700 1 -- 172.21.15.32:0/1815096659 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f385c013420 con 0x7f38680fe830 2024-04-12T05:50:14.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.940+0000 7f384ffff700 1 --2- 172.21.15.32:0/1815096659 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f385006cc20 0x7f385006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:14.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.940+0000 7f384ffff700 1 -- 172.21.15.32:0/1815096659 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f385c08d540 con 0x7f38680fe830 2024-04-12T05:50:14.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.940+0000 7f38677fe700 1 --2- 172.21.15.32:0/1815096659 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f385006cc20 0x7f385006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:14.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.941+0000 7f38677fe700 1 --2- 172.21.15.32:0/1815096659 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f385006cc20 0x7f385006f0d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3858009fd0 tx=0x7f3858009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:50:14.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:14.945+0000 7f384ffff700 1 -- 172.21.15.32:0/1815096659 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f385c05af10 con 0x7f38680fe830 2024-04-12T05:50:15.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.233+0000 7f386dc9d700 1 -- 172.21.15.32:0/1815096659 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3868109d10 con 0x7f38680fe830 2024-04-12T05:50:15.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.233+0000 7f384ffff700 1 -- 172.21.15.32:0/1815096659 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f385c05aaa0 con 0x7f38680fe830 2024-04-12T05:50:15.229 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:50:15.229 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:50:15.229 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:50:15.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:50:15.230 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:50:15.230 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:50:15.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:50:15.230 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:50:15.230 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:50:15.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:50:15.230 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:50:15.230 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:50:15.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:50:15.230 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:50:15.230 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:50:15.230 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:50:15.230 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:50:15.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.235+0000 7f386dc9d700 1 -- 172.21.15.32:0/1815096659 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f385006cc20 msgr2=0x7f385006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:15.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.235+0000 7f386dc9d700 1 --2- 172.21.15.32:0/1815096659 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f385006cc20 0x7f385006f0d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3858009fd0 tx=0x7f3858009ee0 comp rx=0 tx=0).stop 2024-04-12T05:50:15.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.235+0000 7f386dc9d700 1 -- 172.21.15.32:0/1815096659 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38680fe830 msgr2=0x7f3868113d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:15.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.235+0000 7f386dc9d700 1 --2- 172.21.15.32:0/1815096659 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38680fe830 0x7f3868113d80 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f385c004890 tx=0x7f385c009e30 comp rx=0 tx=0).stop 2024-04-12T05:50:15.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.235+0000 7f386dc9d700 1 -- 172.21.15.32:0/1815096659 shutdown_connections 2024-04-12T05:50:15.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.235+0000 7f386dc9d700 1 --2- 172.21.15.32:0/1815096659 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f385006cc20 0x7f385006f0d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:15.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.235+0000 7f386dc9d700 1 --2- 172.21.15.32:0/1815096659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38680fde90 0x7f3868113840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:15.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.235+0000 7f386dc9d700 1 --2- 172.21.15.32:0/1815096659 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38680fe830 0x7f3868113d80 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:15.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.235+0000 7f386dc9d700 1 -- 172.21.15.32:0/1815096659 >> 172.21.15.32:0/1815096659 conn(0x7f38680f78e0 msgr2=0x7f38680f8e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:50:15.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.235+0000 7f386dc9d700 1 -- 172.21.15.32:0/1815096659 shutdown_connections 2024-04-12T05:50:15.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.235+0000 7f386dc9d700 1 -- 172.21.15.32:0/1815096659 wait complete. 2024-04-12T05:50:15.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:15 smithi067 ceph-mon[112979]: pgmap v292: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:15.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:15 smithi067 ceph-mon[112979]: from='client.34592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:50:15.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:15 smithi067 ceph-mon[112979]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:50:15.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.338+0000 7f563ffc1700 1 -- 172.21.15.32:0/3657313207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56380fde90 msgr2=0x7f56380fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:15.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.338+0000 7f563ffc1700 1 --2- 172.21.15.32:0/3657313207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56380fde90 0x7f56380fe260 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5634004940 tx=0x7f5634009e30 comp rx=0 tx=0).stop 2024-04-12T05:50:15.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.338+0000 7f563ffc1700 1 -- 172.21.15.32:0/3657313207 shutdown_connections 2024-04-12T05:50:15.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.338+0000 7f563ffc1700 1 --2- 172.21.15.32:0/3657313207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56380fe830 0x7f5638106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:15.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.338+0000 7f563ffc1700 1 --2- 172.21.15.32:0/3657313207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56380fde90 0x7f56380fe260 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:15.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.338+0000 7f563ffc1700 1 -- 172.21.15.32:0/3657313207 >> 172.21.15.32:0/3657313207 conn(0x7f5638072bc0 msgr2=0x7f5638072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:50:15.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.338+0000 7f563ffc1700 1 -- 172.21.15.32:0/3657313207 shutdown_connections 2024-04-12T05:50:15.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.338+0000 7f563ffc1700 1 -- 172.21.15.32:0/3657313207 wait complete. 2024-04-12T05:50:15.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.339+0000 7f563ffc1700 1 Processor -- start 2024-04-12T05:50:15.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.339+0000 7f563ffc1700 1 -- start start 2024-04-12T05:50:15.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.340+0000 7f563ffc1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56380fde90 0x7f5638113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:15.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.340+0000 7f563ffc1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56380fe830 0x7f5638113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:15.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.340+0000 7f563ffc1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5638114600 con 0x7f56380fe830 2024-04-12T05:50:15.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.340+0000 7f563ffc1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563810d9f0 con 0x7f56380fde90 2024-04-12T05:50:15.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.340+0000 7f563d55c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56380fe830 0x7f5638113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:15.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.340+0000 7f563d55c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56380fe830 0x7f5638113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33394/0 (socket says 172.21.15.32:33394) 2024-04-12T05:50:15.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.340+0000 7f563d55c700 1 -- 172.21.15.32:0/2686983394 learned_addr learned my addr 172.21.15.32:0/2686983394 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:50:15.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.340+0000 7f563dd5d700 1 --2- 172.21.15.32:0/2686983394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56380fde90 0x7f5638113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:15.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.340+0000 7f563d55c700 1 -- 172.21.15.32:0/2686983394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56380fde90 msgr2=0x7f5638113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:15.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.340+0000 7f563d55c700 1 --2- 172.21.15.32:0/2686983394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56380fde90 0x7f5638113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:15.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.340+0000 7f563d55c700 1 -- 172.21.15.32:0/2686983394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5634005040 con 0x7f56380fe830 2024-04-12T05:50:15.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.340+0000 7f563d55c700 1 --2- 172.21.15.32:0/2686983394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56380fe830 0x7f5638113fb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f562800eb40 tx=0x7f562800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:50:15.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.340+0000 7f562e7fc700 1 -- 172.21.15.32:0/2686983394 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5628007cc0 con 0x7f56380fe830 2024-04-12T05:50:15.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.340+0000 7f563ffc1700 1 -- 172.21.15.32:0/2686983394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f563810dc70 con 0x7f56380fe830 2024-04-12T05:50:15.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.340+0000 7f562e7fc700 1 -- 172.21.15.32:0/2686983394 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f562800f040 con 0x7f56380fe830 2024-04-12T05:50:15.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.340+0000 7f562e7fc700 1 -- 172.21.15.32:0/2686983394 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f562800c990 con 0x7f56380fe830 2024-04-12T05:50:15.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.340+0000 7f563ffc1700 1 -- 172.21.15.32:0/2686983394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f563810e1b0 con 0x7f56380fe830 2024-04-12T05:50:15.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.341+0000 7f563ffc1700 1 -- 172.21.15.32:0/2686983394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5638065e30 con 0x7f56380fe830 2024-04-12T05:50:15.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:15 smithi032 ceph-mon[146817]: pgmap v292: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:15.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:15 smithi032 ceph-mon[146817]: from='client.34592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:50:15.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:15 smithi032 ceph-mon[146817]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:50:15.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.342+0000 7f562e7fc700 1 -- 172.21.15.32:0/2686983394 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f562800cc00 con 0x7f56380fe830 2024-04-12T05:50:15.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.342+0000 7f562e7fc700 1 --2- 172.21.15.32:0/2686983394 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f562406cc20 0x7f562406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:15.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.342+0000 7f562e7fc700 1 -- 172.21.15.32:0/2686983394 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f562808bcd0 con 0x7f56380fe830 2024-04-12T05:50:15.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.342+0000 7f563dd5d700 1 --2- 172.21.15.32:0/2686983394 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f562406cc20 0x7f562406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:15.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.343+0000 7f563dd5d700 1 --2- 172.21.15.32:0/2686983394 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f562406cc20 0x7f562406f0d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f563400b0d0 tx=0x7f5634009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:50:15.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.347+0000 7f562e7fc700 1 -- 172.21.15.32:0/2686983394 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5628059770 con 0x7f56380fe830 2024-04-12T05:50:15.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.592+0000 7f563ffc1700 1 -- 172.21.15.32:0/2686983394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5638109dc0 con 0x7f56380fe830 2024-04-12T05:50:15.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.592+0000 7f562e7fc700 1 -- 172.21.15.32:0/2686983394 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1835 (secure 0 0 0) 0x7f5628059300 con 0x7f56380fe830 2024-04-12T05:50:15.588 INFO:teuthology.orchestra.run.smithi032.stdout:e33 2024-04-12T05:50:15.588 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:50:15.588 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:50:15.589 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:50:15.589 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:50:15.589 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:50:15.589 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:50:15.589 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 24 2024-04-12T05:50:15.589 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:50:15.589 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:50:15.589 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:46:10.668569+0000 2024-04-12T05:50:15.590 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:50:15.590 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:50:15.590 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:50:15.590 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:50:15.590 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:50:15.590 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:50:15.590 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:50:15.590 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:50:15.591 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T05:50:15.591 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:50:15.591 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:50:15.591 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:50:15.591 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T05:50:15.591 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-12T05:50:15.591 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:50:15.591 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:50:15.592 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:50:15.592 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:50:15.592 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:50:15.592 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:50:15.592 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:50:15.592 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:50:15.592 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:50:15.592 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:50:15.592 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-12T05:50:15.593 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:50:15.593 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:50:15.593 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:50:15.593 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{-1:44317} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:50:15.593 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:50:15.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.594+0000 7f563ffc1700 1 -- 172.21.15.32:0/2686983394 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f562406cc20 msgr2=0x7f562406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:15.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.594+0000 7f563ffc1700 1 --2- 172.21.15.32:0/2686983394 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f562406cc20 0x7f562406f0d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f563400b0d0 tx=0x7f5634009e30 comp rx=0 tx=0).stop 2024-04-12T05:50:15.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.594+0000 7f563ffc1700 1 -- 172.21.15.32:0/2686983394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56380fe830 msgr2=0x7f5638113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:15.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.594+0000 7f563ffc1700 1 --2- 172.21.15.32:0/2686983394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56380fe830 0x7f5638113fb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f562800eb40 tx=0x7f562800b040 comp rx=0 tx=0).stop 2024-04-12T05:50:15.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.594+0000 7f563ffc1700 1 -- 172.21.15.32:0/2686983394 shutdown_connections 2024-04-12T05:50:15.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.594+0000 7f563ffc1700 1 --2- 172.21.15.32:0/2686983394 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f562406cc20 0x7f562406f0d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:15.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.594+0000 7f563ffc1700 1 --2- 172.21.15.32:0/2686983394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56380fde90 0x7f5638113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:15.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.594+0000 7f563ffc1700 1 --2- 172.21.15.32:0/2686983394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56380fe830 0x7f5638113fb0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:15.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.594+0000 7f563ffc1700 1 -- 172.21.15.32:0/2686983394 >> 172.21.15.32:0/2686983394 conn(0x7f5638072bc0 msgr2=0x7f56381037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:50:15.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.594+0000 7f563ffc1700 1 -- 172.21.15.32:0/2686983394 shutdown_connections 2024-04-12T05:50:15.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.594+0000 7f563ffc1700 1 -- 172.21.15.32:0/2686983394 wait complete. 2024-04-12T05:50:15.595 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-12T05:50:15.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.698+0000 7fa2c0b34700 1 -- 172.21.15.32:0/1785907888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2bc101d70 msgr2=0x7fa2bc102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:15.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.698+0000 7fa2c0b34700 1 --2- 172.21.15.32:0/1785907888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2bc101d70 0x7fa2bc102140 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac004940 tx=0x7fa2ac009e30 comp rx=0 tx=0).stop 2024-04-12T05:50:15.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.698+0000 7fa2c0b34700 1 -- 172.21.15.32:0/1785907888 shutdown_connections 2024-04-12T05:50:15.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.698+0000 7fa2c0b34700 1 --2- 172.21.15.32:0/1785907888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2bc102710 0x7fa2bc0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:15.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.698+0000 7fa2c0b34700 1 --2- 172.21.15.32:0/1785907888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2bc101d70 0x7fa2bc102140 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:15.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.698+0000 7fa2c0b34700 1 -- 172.21.15.32:0/1785907888 >> 172.21.15.32:0/1785907888 conn(0x7fa2bc072bc0 msgr2=0x7fa2bc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:50:15.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.698+0000 7fa2c0b34700 1 -- 172.21.15.32:0/1785907888 shutdown_connections 2024-04-12T05:50:15.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.698+0000 7fa2c0b34700 1 -- 172.21.15.32:0/1785907888 wait complete. 2024-04-12T05:50:15.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.699+0000 7fa2c0b34700 1 Processor -- start 2024-04-12T05:50:15.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.699+0000 7fa2c0b34700 1 -- start start 2024-04-12T05:50:15.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.699+0000 7fa2c0b34700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2bc101d70 0x7fa2bc102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:15.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.699+0000 7fa2c0b34700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2bc102710 0x7fa2bc103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:15.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.699+0000 7fa2c0b34700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc1036a0 con 0x7fa2bc101d70 2024-04-12T05:50:15.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.699+0000 7fa2c0b34700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc1037e0 con 0x7fa2bc102710 2024-04-12T05:50:15.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.699+0000 7fa2ba59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2bc101d70 0x7fa2bc102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:15.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.699+0000 7fa2ba59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2bc101d70 0x7fa2bc102bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33400/0 (socket says 172.21.15.32:33400) 2024-04-12T05:50:15.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.699+0000 7fa2ba59c700 1 -- 172.21.15.32:0/1693959773 learned_addr learned my addr 172.21.15.32:0/1693959773 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:50:15.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.699+0000 7fa2b9d9b700 1 --2- 172.21.15.32:0/1693959773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2bc102710 0x7fa2bc103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:15.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.699+0000 7fa2ba59c700 1 -- 172.21.15.32:0/1693959773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2bc102710 msgr2=0x7fa2bc103110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:15.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.699+0000 7fa2ba59c700 1 --2- 172.21.15.32:0/1693959773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2bc102710 0x7fa2bc103110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:15.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.699+0000 7fa2ba59c700 1 -- 172.21.15.32:0/1693959773 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ac005040 con 0x7fa2bc101d70 2024-04-12T05:50:15.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.699+0000 7fa2b9d9b700 1 --2- 172.21.15.32:0/1693959773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2bc102710 0x7fa2bc103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:50:15.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.700+0000 7fa2ba59c700 1 --2- 172.21.15.32:0/1693959773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2bc101d70 0x7fa2bc102bd0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac009360 tx=0x7fa2ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:50:15.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.700+0000 7fa2b2ffd700 1 -- 172.21.15.32:0/1693959773 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2ac00e040 con 0x7fa2bc101d70 2024-04-12T05:50:15.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.700+0000 7fa2c0b34700 1 -- 172.21.15.32:0/1693959773 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2bc103a30 con 0x7fa2bc101d70 2024-04-12T05:50:15.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.700+0000 7fa2b2ffd700 1 -- 172.21.15.32:0/1693959773 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa2ac013420 con 0x7fa2bc101d70 2024-04-12T05:50:15.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.700+0000 7fa2b2ffd700 1 -- 172.21.15.32:0/1693959773 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2ac01c5f0 con 0x7fa2bc101d70 2024-04-12T05:50:15.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.700+0000 7fa2c0b34700 1 -- 172.21.15.32:0/1693959773 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2bc11a190 con 0x7fa2bc101d70 2024-04-12T05:50:15.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.701+0000 7fa2c0b34700 1 -- 172.21.15.32:0/1693959773 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2bc0fd4c0 con 0x7fa2bc101d70 2024-04-12T05:50:15.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.701+0000 7fa2b2ffd700 1 -- 172.21.15.32:0/1693959773 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa2ac013590 con 0x7fa2bc101d70 2024-04-12T05:50:15.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.702+0000 7fa2b2ffd700 1 --2- 172.21.15.32:0/1693959773 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa2a806cc30 0x7fa2a806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:15.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.702+0000 7fa2b2ffd700 1 -- 172.21.15.32:0/1693959773 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fa2ac08d330 con 0x7fa2bc101d70 2024-04-12T05:50:15.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.702+0000 7fa2b9d9b700 1 --2- 172.21.15.32:0/1693959773 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa2a806cc30 0x7fa2a806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:15.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.702+0000 7fa2b9d9b700 1 --2- 172.21.15.32:0/1693959773 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa2a806cc30 0x7fa2a806f0e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4007fd0 tx=0x7fa2a4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:50:15.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.706+0000 7fa2b2ffd700 1 -- 172.21.15.32:0/1693959773 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa2ac057310 con 0x7fa2bc101d70 2024-04-12T05:50:15.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.937+0000 7fa2c0b34700 1 -- 172.21.15.32:0/1693959773 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa2bc1042c0 con 0x7fa2a806cc30 2024-04-12T05:50:15.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.938+0000 7fa2b2ffd700 1 -- 172.21.15.32:0/1693959773 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa2bc1042c0 con 0x7fa2a806cc30 2024-04-12T05:50:15.934 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:50:15.935 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:50:15.935 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:50:15.935 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:50:15.935 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:50:15.935 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-12T05:50:15.935 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:50:15.935 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-12T05:50:15.935 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:50:15.935 INFO:teuthology.orchestra.run.smithi032.stdout: "mds" 2024-04-12T05:50:15.935 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:50:15.935 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:50:15.935 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:50:15.935 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:50:15.935 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:50:15.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.941+0000 7fa2c0b34700 1 -- 172.21.15.32:0/1693959773 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa2a806cc30 msgr2=0x7fa2a806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:15.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.941+0000 7fa2c0b34700 1 --2- 172.21.15.32:0/1693959773 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa2a806cc30 0x7fa2a806f0e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4007fd0 tx=0x7fa2a4009040 comp rx=0 tx=0).stop 2024-04-12T05:50:15.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.941+0000 7fa2c0b34700 1 -- 172.21.15.32:0/1693959773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2bc101d70 msgr2=0x7fa2bc102bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:15.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.941+0000 7fa2c0b34700 1 --2- 172.21.15.32:0/1693959773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2bc101d70 0x7fa2bc102bd0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac009360 tx=0x7fa2ac009e30 comp rx=0 tx=0).stop 2024-04-12T05:50:15.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.941+0000 7fa2c0b34700 1 -- 172.21.15.32:0/1693959773 shutdown_connections 2024-04-12T05:50:15.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.941+0000 7fa2c0b34700 1 --2- 172.21.15.32:0/1693959773 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa2a806cc30 0x7fa2a806f0e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:15.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.941+0000 7fa2c0b34700 1 --2- 172.21.15.32:0/1693959773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2bc101d70 0x7fa2bc102bd0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:15.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.941+0000 7fa2c0b34700 1 --2- 172.21.15.32:0/1693959773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2bc102710 0x7fa2bc103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:15.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.941+0000 7fa2c0b34700 1 -- 172.21.15.32:0/1693959773 >> 172.21.15.32:0/1693959773 conn(0x7fa2bc072bc0 msgr2=0x7fa2bc0659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:50:15.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.941+0000 7fa2c0b34700 1 -- 172.21.15.32:0/1693959773 shutdown_connections 2024-04-12T05:50:15.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:15.941+0000 7fa2c0b34700 1 -- 172.21.15.32:0/1693959773 wait complete. 2024-04-12T05:50:16.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.045+0000 7f155f5ca700 1 -- 172.21.15.32:0/2420528592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1558101d70 msgr2=0x7f1558102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:16.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.045+0000 7f155f5ca700 1 --2- 172.21.15.32:0/2420528592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1558101d70 0x7f1558102140 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1554004940 tx=0x7f1554009e30 comp rx=0 tx=0).stop 2024-04-12T05:50:16.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.045+0000 7f155f5ca700 1 -- 172.21.15.32:0/2420528592 shutdown_connections 2024-04-12T05:50:16.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.045+0000 7f155f5ca700 1 --2- 172.21.15.32:0/2420528592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1558102710 0x7f15580fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:16.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.045+0000 7f155f5ca700 1 --2- 172.21.15.32:0/2420528592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1558101d70 0x7f1558102140 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:16.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.045+0000 7f155f5ca700 1 -- 172.21.15.32:0/2420528592 >> 172.21.15.32:0/2420528592 conn(0x7f1558072bc0 msgr2=0x7f1558072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:50:16.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.045+0000 7f155f5ca700 1 -- 172.21.15.32:0/2420528592 shutdown_connections 2024-04-12T05:50:16.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.045+0000 7f155f5ca700 1 -- 172.21.15.32:0/2420528592 wait complete. 2024-04-12T05:50:16.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.046+0000 7f155f5ca700 1 Processor -- start 2024-04-12T05:50:16.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.046+0000 7f155f5ca700 1 -- start start 2024-04-12T05:50:16.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.046+0000 7f155f5ca700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1558101d70 0x7f1558113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:16.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.046+0000 7f155f5ca700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1558102710 0x7f1558114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:16.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.046+0000 7f155f5ca700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1558114650 con 0x7f1558101d70 2024-04-12T05:50:16.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.046+0000 7f155f5ca700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155810da40 con 0x7f1558102710 2024-04-12T05:50:16.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.046+0000 7f155d366700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1558101d70 0x7f1558113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:16.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.046+0000 7f155d366700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1558101d70 0x7f1558113ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33420/0 (socket says 172.21.15.32:33420) 2024-04-12T05:50:16.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.046+0000 7f155d366700 1 -- 172.21.15.32:0/2805408851 learned_addr learned my addr 172.21.15.32:0/2805408851 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:50:16.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.047+0000 7f155cb65700 1 --2- 172.21.15.32:0/2805408851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1558102710 0x7f1558114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:16.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.047+0000 7f155d366700 1 -- 172.21.15.32:0/2805408851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1558102710 msgr2=0x7f1558114000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:16.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.047+0000 7f155d366700 1 --2- 172.21.15.32:0/2805408851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1558102710 0x7f1558114000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:16.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.047+0000 7f155d366700 1 -- 172.21.15.32:0/2805408851 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1554005040 con 0x7f1558101d70 2024-04-12T05:50:16.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.047+0000 7f155d366700 1 --2- 172.21.15.32:0/2805408851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1558101d70 0x7f1558113ac0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f155400b3b0 tx=0x7f1554009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:50:16.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.047+0000 7f1549ffb700 1 -- 172.21.15.32:0/2805408851 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f155400e040 con 0x7f1558101d70 2024-04-12T05:50:16.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.047+0000 7f1549ffb700 1 -- 172.21.15.32:0/2805408851 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1554013420 con 0x7f1558101d70 2024-04-12T05:50:16.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.047+0000 7f155f5ca700 1 -- 172.21.15.32:0/2805408851 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f155810dcc0 con 0x7f1558101d70 2024-04-12T05:50:16.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.047+0000 7f1549ffb700 1 -- 172.21.15.32:0/2805408851 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f155401c630 con 0x7f1558101d70 2024-04-12T05:50:16.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.047+0000 7f155f5ca700 1 -- 172.21.15.32:0/2805408851 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f155810e280 con 0x7f1558101d70 2024-04-12T05:50:16.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.048+0000 7f155f5ca700 1 -- 172.21.15.32:0/2805408851 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15580fd4c0 con 0x7f1558101d70 2024-04-12T05:50:16.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.049+0000 7f1549ffb700 1 -- 172.21.15.32:0/2805408851 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1554013970 con 0x7f1558101d70 2024-04-12T05:50:16.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.049+0000 7f1549ffb700 1 --2- 172.21.15.32:0/2805408851 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f154406cc20 0x7f154406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:16.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.049+0000 7f1549ffb700 1 -- 172.21.15.32:0/2805408851 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f155408d220 con 0x7f1558101d70 2024-04-12T05:50:16.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.049+0000 7f155cb65700 1 --2- 172.21.15.32:0/2805408851 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f154406cc20 0x7f154406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:16.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.050+0000 7f155cb65700 1 --2- 172.21.15.32:0/2805408851 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f154406cc20 0x7f154406f0d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f155810ec00 tx=0x7f154c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:50:16.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.053+0000 7f1549ffb700 1 -- 172.21.15.32:0/2805408851 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f155405ad30 con 0x7f1558101d70 2024-04-12T05:50:16.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.341+0000 7f155f5ca700 1 -- 172.21.15.32:0/2805408851 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1558108d30 con 0x7f1558101d70 2024-04-12T05:50:16.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:16 smithi032 ceph-mon[146817]: from='client.34600 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:50:16.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:16 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1815096659' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:50:16.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:16 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/2686983394' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:50:16.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.342+0000 7f1549ffb700 1 -- 172.21.15.32:0/2805408851 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f155405a8c0 con 0x7f1558101d70 2024-04-12T05:50:16.339 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:50:16.339 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:50:16.339 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-12T05:50:16.339 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:50:16.339 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:50:16.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.344+0000 7f155f5ca700 1 -- 172.21.15.32:0/2805408851 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f154406cc20 msgr2=0x7f154406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:16.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.344+0000 7f155f5ca700 1 --2- 172.21.15.32:0/2805408851 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f154406cc20 0x7f154406f0d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f155810ec00 tx=0x7f154c00b040 comp rx=0 tx=0).stop 2024-04-12T05:50:16.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.344+0000 7f155f5ca700 1 -- 172.21.15.32:0/2805408851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1558101d70 msgr2=0x7f1558113ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:16.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.344+0000 7f155f5ca700 1 --2- 172.21.15.32:0/2805408851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1558101d70 0x7f1558113ac0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f155400b3b0 tx=0x7f1554009e30 comp rx=0 tx=0).stop 2024-04-12T05:50:16.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.344+0000 7f155f5ca700 1 -- 172.21.15.32:0/2805408851 shutdown_connections 2024-04-12T05:50:16.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.344+0000 7f155f5ca700 1 --2- 172.21.15.32:0/2805408851 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f154406cc20 0x7f154406f0d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:16.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.344+0000 7f155f5ca700 1 --2- 172.21.15.32:0/2805408851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1558101d70 0x7f1558113ac0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:16.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.344+0000 7f155f5ca700 1 --2- 172.21.15.32:0/2805408851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1558102710 0x7f1558114000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:16.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.344+0000 7f155f5ca700 1 -- 172.21.15.32:0/2805408851 >> 172.21.15.32:0/2805408851 conn(0x7f1558072bc0 msgr2=0x7f15580659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:50:16.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.344+0000 7f155f5ca700 1 -- 172.21.15.32:0/2805408851 shutdown_connections 2024-04-12T05:50:16.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:16.344+0000 7f155f5ca700 1 -- 172.21.15.32:0/2805408851 wait complete. 2024-04-12T05:50:16.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:16 smithi067 ceph-mon[112979]: from='client.34600 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:50:16.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:16 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1815096659' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:50:16.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:16 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/2686983394' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:50:17.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:17 smithi032 ceph-mon[146817]: pgmap v293: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:17.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:17 smithi032 ceph-mon[146817]: from='client.34612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:50:17.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:17 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/2805408851' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:50:17.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:17 smithi067 ceph-mon[112979]: pgmap v293: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:17.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:17 smithi067 ceph-mon[112979]: from='client.34612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:50:17.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:17 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/2805408851' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:50:18.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:50:18.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:50:19.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:19 smithi032 ceph-mon[146817]: pgmap v294: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:19.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:19 smithi067 ceph-mon[112979]: pgmap v294: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:21.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:21 smithi032 ceph-mon[146817]: pgmap v295: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:21.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:21 smithi067 ceph-mon[112979]: pgmap v295: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:23.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:23 smithi032 ceph-mon[146817]: pgmap v296: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:23.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:23 smithi067 ceph-mon[112979]: pgmap v296: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:25.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:25 smithi032 ceph-mon[146817]: pgmap v297: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:25.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:25 smithi067 ceph-mon[112979]: pgmap v297: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:27.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:27 smithi032 ceph-mon[146817]: pgmap v298: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:27.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:27 smithi067 ceph-mon[112979]: pgmap v298: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:29.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:29 smithi032 ceph-mon[146817]: pgmap v299: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:29.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:29 smithi067 ceph-mon[112979]: pgmap v299: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:31.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:31 smithi032 ceph-mon[146817]: pgmap v300: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:31.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:31 smithi067 ceph-mon[112979]: pgmap v300: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:33.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:33 smithi032 ceph-mon[146817]: pgmap v301: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:33.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:33 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:50:33.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:33 smithi067 ceph-mon[112979]: pgmap v301: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:33.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:33 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:50:35.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:35 smithi032 ceph-mon[146817]: pgmap v302: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:35.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:35 smithi067 ceph-mon[112979]: pgmap v302: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:37.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:37 smithi032 ceph-mon[146817]: pgmap v303: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:37.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:37 smithi067 ceph-mon[112979]: pgmap v303: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:39.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:39 smithi032 ceph-mon[146817]: pgmap v304: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:39.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:39 smithi067 ceph-mon[112979]: pgmap v304: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:41.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:41 smithi032 ceph-mon[146817]: pgmap v305: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:41.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:41 smithi067 ceph-mon[112979]: pgmap v305: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:43.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:43 smithi032 ceph-mon[146817]: pgmap v306: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:43.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:43 smithi067 ceph-mon[112979]: pgmap v306: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:45.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:45 smithi032 ceph-mon[146817]: pgmap v307: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:45.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:45 smithi067 ceph-mon[112979]: pgmap v307: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:46.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.463+0000 7fa3ebe2a700 1 -- 172.21.15.32:0/752551556 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3e4102710 msgr2=0x7fa3e40fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:46.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.463+0000 7fa3ebe2a700 1 --2- 172.21.15.32:0/752551556 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3e4102710 0x7fa3e40fbd80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0004940 tx=0x7fa3e0009e30 comp rx=0 tx=0).stop 2024-04-12T05:50:46.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.463+0000 7fa3ebe2a700 1 -- 172.21.15.32:0/752551556 shutdown_connections 2024-04-12T05:50:46.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.463+0000 7fa3ebe2a700 1 --2- 172.21.15.32:0/752551556 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3e4102710 0x7fa3e40fbd80 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:46.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.463+0000 7fa3ebe2a700 1 --2- 172.21.15.32:0/752551556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3e4101d70 0x7fa3e4102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:46.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.463+0000 7fa3ebe2a700 1 -- 172.21.15.32:0/752551556 >> 172.21.15.32:0/752551556 conn(0x7fa3e4072bc0 msgr2=0x7fa3e4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:50:46.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.463+0000 7fa3ebe2a700 1 -- 172.21.15.32:0/752551556 shutdown_connections 2024-04-12T05:50:46.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.463+0000 7fa3ebe2a700 1 -- 172.21.15.32:0/752551556 wait complete. 2024-04-12T05:50:46.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.463+0000 7fa3ebe2a700 1 Processor -- start 2024-04-12T05:50:46.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.463+0000 7fa3ebe2a700 1 -- start start 2024-04-12T05:50:46.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.464+0000 7fa3ebe2a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3e4101d70 0x7fa3e4113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:46.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.464+0000 7fa3ebe2a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3e4102710 0x7fa3e4114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:46.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.464+0000 7fa3ebe2a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e4114650 con 0x7fa3e4101d70 2024-04-12T05:50:46.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.464+0000 7fa3ebe2a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e410da40 con 0x7fa3e4102710 2024-04-12T05:50:46.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.464+0000 7fa3e9bc6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3e4101d70 0x7fa3e4113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:46.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.464+0000 7fa3e9bc6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3e4101d70 0x7fa3e4113ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55706/0 (socket says 172.21.15.32:55706) 2024-04-12T05:50:46.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.464+0000 7fa3e9bc6700 1 -- 172.21.15.32:0/1514929949 learned_addr learned my addr 172.21.15.32:0/1514929949 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:50:46.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.464+0000 7fa3e93c5700 1 --2- 172.21.15.32:0/1514929949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3e4102710 0x7fa3e4114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:46.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.464+0000 7fa3e9bc6700 1 -- 172.21.15.32:0/1514929949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3e4102710 msgr2=0x7fa3e4114000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:46.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.464+0000 7fa3e9bc6700 1 --2- 172.21.15.32:0/1514929949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3e4102710 0x7fa3e4114000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:46.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.464+0000 7fa3e9bc6700 1 -- 172.21.15.32:0/1514929949 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e0005040 con 0x7fa3e4101d70 2024-04-12T05:50:46.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.464+0000 7fa3e9bc6700 1 --2- 172.21.15.32:0/1514929949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3e4101d70 0x7fa3e4113ac0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa3d4002a50 tx=0x7fa3d40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:50:46.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.464+0000 7fa3da7fc700 1 -- 172.21.15.32:0/1514929949 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3d4007cc0 con 0x7fa3e4101d70 2024-04-12T05:50:46.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.464+0000 7fa3da7fc700 1 -- 172.21.15.32:0/1514929949 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa3d400f040 con 0x7fa3e4101d70 2024-04-12T05:50:46.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.465+0000 7fa3ebe2a700 1 -- 172.21.15.32:0/1514929949 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3e410dd20 con 0x7fa3e4101d70 2024-04-12T05:50:46.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.465+0000 7fa3da7fc700 1 -- 172.21.15.32:0/1514929949 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3d400d9c0 con 0x7fa3e4101d70 2024-04-12T05:50:46.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.465+0000 7fa3ebe2a700 1 -- 172.21.15.32:0/1514929949 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3e410e2b0 con 0x7fa3e4101d70 2024-04-12T05:50:46.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.466+0000 7fa3ebe2a700 1 -- 172.21.15.32:0/1514929949 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3e40fd4c0 con 0x7fa3e4101d70 2024-04-12T05:50:46.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.466+0000 7fa3da7fc700 1 -- 172.21.15.32:0/1514929949 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa3d400db20 con 0x7fa3e4101d70 2024-04-12T05:50:46.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.466+0000 7fa3da7fc700 1 --2- 172.21.15.32:0/1514929949 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa3d006cc20 0x7fa3d006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:46.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.467+0000 7fa3da7fc700 1 -- 172.21.15.32:0/1514929949 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fa3d408bd10 con 0x7fa3e4101d70 2024-04-12T05:50:46.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.467+0000 7fa3e93c5700 1 --2- 172.21.15.32:0/1514929949 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa3d006cc20 0x7fa3d006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:46.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.467+0000 7fa3e93c5700 1 --2- 172.21.15.32:0/1514929949 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa3d006cc20 0x7fa3d006f0d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa3e410ed20 tx=0x7fa3e0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:50:46.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.471+0000 7fa3da7fc700 1 -- 172.21.15.32:0/1514929949 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa3d4059780 con 0x7fa3e4101d70 2024-04-12T05:50:46.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.693+0000 7fa3ebe2a700 1 -- 172.21.15.32:0/1514929949 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3e4108d60 con 0x7fa3d006cc20 2024-04-12T05:50:46.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.694+0000 7fa3da7fc700 1 -- 172.21.15.32:0/1514929949 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa3e4108d60 con 0x7fa3d006cc20 2024-04-12T05:50:46.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.696+0000 7fa3ebe2a700 1 -- 172.21.15.32:0/1514929949 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa3d006cc20 msgr2=0x7fa3d006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:46.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.696+0000 7fa3ebe2a700 1 --2- 172.21.15.32:0/1514929949 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa3d006cc20 0x7fa3d006f0d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa3e410ed20 tx=0x7fa3e0009e30 comp rx=0 tx=0).stop 2024-04-12T05:50:46.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.696+0000 7fa3ebe2a700 1 -- 172.21.15.32:0/1514929949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3e4101d70 msgr2=0x7fa3e4113ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:46.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.696+0000 7fa3ebe2a700 1 --2- 172.21.15.32:0/1514929949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3e4101d70 0x7fa3e4113ac0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa3d4002a50 tx=0x7fa3d40046b0 comp rx=0 tx=0).stop 2024-04-12T05:50:46.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.696+0000 7fa3ebe2a700 1 -- 172.21.15.32:0/1514929949 shutdown_connections 2024-04-12T05:50:46.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.696+0000 7fa3ebe2a700 1 --2- 172.21.15.32:0/1514929949 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa3d006cc20 0x7fa3d006f0d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:46.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.696+0000 7fa3ebe2a700 1 --2- 172.21.15.32:0/1514929949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3e4101d70 0x7fa3e4113ac0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:46.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.696+0000 7fa3ebe2a700 1 --2- 172.21.15.32:0/1514929949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3e4102710 0x7fa3e4114000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:46.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.696+0000 7fa3ebe2a700 1 -- 172.21.15.32:0/1514929949 >> 172.21.15.32:0/1514929949 conn(0x7fa3e4072bc0 msgr2=0x7fa3e4065a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:50:46.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.697+0000 7fa3ebe2a700 1 -- 172.21.15.32:0/1514929949 shutdown_connections 2024-04-12T05:50:46.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.697+0000 7fa3ebe2a700 1 -- 172.21.15.32:0/1514929949 wait complete. 2024-04-12T05:50:46.703 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:50:46.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.800+0000 7fc085b37700 1 -- 172.21.15.32:0/1692111493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0801037b0 msgr2=0x7fc0800fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:46.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.800+0000 7fc085b37700 1 --2- 172.21.15.32:0/1692111493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0801037b0 0x7fc0800fce20 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc074004940 tx=0x7fc074009e30 comp rx=0 tx=0).stop 2024-04-12T05:50:46.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.800+0000 7fc085b37700 1 -- 172.21.15.32:0/1692111493 shutdown_connections 2024-04-12T05:50:46.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.800+0000 7fc085b37700 1 --2- 172.21.15.32:0/1692111493 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0801037b0 0x7fc0800fce20 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:46.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.800+0000 7fc085b37700 1 --2- 172.21.15.32:0/1692111493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc080102e10 0x7fc0801031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:46.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.800+0000 7fc085b37700 1 -- 172.21.15.32:0/1692111493 >> 172.21.15.32:0/1692111493 conn(0x7fc0800f8980 msgr2=0x7fc0800fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:50:46.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.801+0000 7fc085b37700 1 -- 172.21.15.32:0/1692111493 shutdown_connections 2024-04-12T05:50:46.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.801+0000 7fc085b37700 1 -- 172.21.15.32:0/1692111493 wait complete. 2024-04-12T05:50:46.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.801+0000 7fc085b37700 1 Processor -- start 2024-04-12T05:50:46.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.801+0000 7fc085b37700 1 -- start start 2024-04-12T05:50:46.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.801+0000 7fc085b37700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc080102e10 0x7fc080113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:46.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.801+0000 7fc085b37700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0801037b0 0x7fc080114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:46.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.801+0000 7fc085b37700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc080114670 con 0x7fc0801037b0 2024-04-12T05:50:46.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.801+0000 7fc085b37700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08010da60 con 0x7fc080102e10 2024-04-12T05:50:46.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.802+0000 7fc07effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0801037b0 0x7fc080114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:46.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.802+0000 7fc07effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0801037b0 0x7fc080114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55714/0 (socket says 172.21.15.32:55714) 2024-04-12T05:50:46.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.802+0000 7fc07effd700 1 -- 172.21.15.32:0/1221803212 learned_addr learned my addr 172.21.15.32:0/1221803212 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:50:46.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.802+0000 7fc07f7fe700 1 --2- 172.21.15.32:0/1221803212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc080102e10 0x7fc080113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:46.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.802+0000 7fc07effd700 1 -- 172.21.15.32:0/1221803212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc080102e10 msgr2=0x7fc080113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:46.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.802+0000 7fc07effd700 1 --2- 172.21.15.32:0/1221803212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc080102e10 0x7fc080113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:46.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.802+0000 7fc07effd700 1 -- 172.21.15.32:0/1221803212 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc074005040 con 0x7fc0801037b0 2024-04-12T05:50:46.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.802+0000 7fc07effd700 1 --2- 172.21.15.32:0/1221803212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0801037b0 0x7fc080114020 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc074009390 tx=0x7fc074009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:50:46.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.802+0000 7fc067fff700 1 -- 172.21.15.32:0/1221803212 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc07400e040 con 0x7fc0801037b0 2024-04-12T05:50:46.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.802+0000 7fc067fff700 1 -- 172.21.15.32:0/1221803212 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc074004290 con 0x7fc0801037b0 2024-04-12T05:50:46.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.802+0000 7fc085b37700 1 -- 172.21.15.32:0/1221803212 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc08010dce0 con 0x7fc0801037b0 2024-04-12T05:50:46.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.802+0000 7fc067fff700 1 -- 172.21.15.32:0/1221803212 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc074013e70 con 0x7fc0801037b0 2024-04-12T05:50:46.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.802+0000 7fc085b37700 1 -- 172.21.15.32:0/1221803212 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc08010e2a0 con 0x7fc0801037b0 2024-04-12T05:50:46.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.803+0000 7fc085b37700 1 -- 172.21.15.32:0/1221803212 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0800fe560 con 0x7fc0801037b0 2024-04-12T05:50:46.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.804+0000 7fc067fff700 1 -- 172.21.15.32:0/1221803212 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc074013690 con 0x7fc0801037b0 2024-04-12T05:50:46.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.804+0000 7fc067fff700 1 --2- 172.21.15.32:0/1221803212 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc06806cbc0 0x7fc06806f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:46.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.804+0000 7fc067fff700 1 -- 172.21.15.32:0/1221803212 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fc07408d2a0 con 0x7fc0801037b0 2024-04-12T05:50:46.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.804+0000 7fc07f7fe700 1 --2- 172.21.15.32:0/1221803212 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc06806cbc0 0x7fc06806f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:46.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.805+0000 7fc07f7fe700 1 --2- 172.21.15.32:0/1221803212 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc06806cbc0 0x7fc06806f070 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc070004f40 tx=0x7fc07000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:50:46.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:46.809+0000 7fc067fff700 1 -- 172.21.15.32:0/1221803212 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc07405adc0 con 0x7fc0801037b0 2024-04-12T05:50:47.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.030+0000 7fc085b37700 1 -- 172.21.15.32:0/1221803212 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc080109db0 con 0x7fc06806cbc0 2024-04-12T05:50:47.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.032+0000 7fc067fff700 1 -- 172.21.15.32:0/1221803212 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc080109db0 con 0x7fc06806cbc0 2024-04-12T05:50:47.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.034+0000 7fc085b37700 1 -- 172.21.15.32:0/1221803212 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc06806cbc0 msgr2=0x7fc06806f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:47.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.034+0000 7fc085b37700 1 --2- 172.21.15.32:0/1221803212 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc06806cbc0 0x7fc06806f070 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc070004f40 tx=0x7fc07000b040 comp rx=0 tx=0).stop 2024-04-12T05:50:47.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.034+0000 7fc085b37700 1 -- 172.21.15.32:0/1221803212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0801037b0 msgr2=0x7fc080114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:47.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.034+0000 7fc085b37700 1 --2- 172.21.15.32:0/1221803212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0801037b0 0x7fc080114020 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc074009390 tx=0x7fc074009e30 comp rx=0 tx=0).stop 2024-04-12T05:50:47.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.034+0000 7fc085b37700 1 -- 172.21.15.32:0/1221803212 shutdown_connections 2024-04-12T05:50:47.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.034+0000 7fc085b37700 1 --2- 172.21.15.32:0/1221803212 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc06806cbc0 0x7fc06806f070 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:47.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.034+0000 7fc085b37700 1 --2- 172.21.15.32:0/1221803212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc080102e10 0x7fc080113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:47.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.034+0000 7fc085b37700 1 --2- 172.21.15.32:0/1221803212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0801037b0 0x7fc080114020 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:47.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.034+0000 7fc085b37700 1 -- 172.21.15.32:0/1221803212 >> 172.21.15.32:0/1221803212 conn(0x7fc0800f8980 msgr2=0x7fc0800f9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:50:47.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.034+0000 7fc085b37700 1 -- 172.21.15.32:0/1221803212 shutdown_connections 2024-04-12T05:50:47.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.034+0000 7fc085b37700 1 -- 172.21.15.32:0/1221803212 wait complete. 2024-04-12T05:50:47.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.138+0000 7f1c67b2d700 1 -- 172.21.15.32:0/2401345499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c60102710 msgr2=0x7f1c600fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:47.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.138+0000 7f1c67b2d700 1 --2- 172.21.15.32:0/2401345499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c60102710 0x7f1c600fbd80 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c004940 tx=0x7f1c5c009e30 comp rx=0 tx=0).stop 2024-04-12T05:50:47.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.138+0000 7f1c67b2d700 1 -- 172.21.15.32:0/2401345499 shutdown_connections 2024-04-12T05:50:47.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.138+0000 7f1c67b2d700 1 --2- 172.21.15.32:0/2401345499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c60102710 0x7f1c600fbd80 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:47.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.138+0000 7f1c67b2d700 1 --2- 172.21.15.32:0/2401345499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c60101d70 0x7f1c60102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:47.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.138+0000 7f1c67b2d700 1 -- 172.21.15.32:0/2401345499 >> 172.21.15.32:0/2401345499 conn(0x7f1c60072bc0 msgr2=0x7f1c60072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:50:47.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.138+0000 7f1c67b2d700 1 -- 172.21.15.32:0/2401345499 shutdown_connections 2024-04-12T05:50:47.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.138+0000 7f1c67b2d700 1 -- 172.21.15.32:0/2401345499 wait complete. 2024-04-12T05:50:47.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.138+0000 7f1c67b2d700 1 Processor -- start 2024-04-12T05:50:47.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.138+0000 7f1c67b2d700 1 -- start start 2024-04-12T05:50:47.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.139+0000 7f1c67b2d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c60101d70 0x7f1c6010ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:47.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.139+0000 7f1c67b2d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c60102710 0x7f1c6010f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:47.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.139+0000 7f1c67b2d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6010f640 con 0x7f1c60102710 2024-04-12T05:50:47.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.139+0000 7f1c67b2d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6010f780 con 0x7f1c60101d70 2024-04-12T05:50:47.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.139+0000 7f1c650c8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c60102710 0x7f1c6010f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:47.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.139+0000 7f1c650c8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c60102710 0x7f1c6010f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55736/0 (socket says 172.21.15.32:55736) 2024-04-12T05:50:47.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.139+0000 7f1c650c8700 1 -- 172.21.15.32:0/3962120870 learned_addr learned my addr 172.21.15.32:0/3962120870 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:50:47.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.139+0000 7f1c658c9700 1 --2- 172.21.15.32:0/3962120870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c60101d70 0x7f1c6010ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:47.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.139+0000 7f1c650c8700 1 -- 172.21.15.32:0/3962120870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c60101d70 msgr2=0x7f1c6010ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:47.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.139+0000 7f1c650c8700 1 --2- 172.21.15.32:0/3962120870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c60101d70 0x7f1c6010ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:47.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.139+0000 7f1c650c8700 1 -- 172.21.15.32:0/3962120870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c5c005040 con 0x7f1c60102710 2024-04-12T05:50:47.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.139+0000 7f1c658c9700 1 --2- 172.21.15.32:0/3962120870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c60101d70 0x7f1c6010ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:50:47.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.139+0000 7f1c650c8700 1 --2- 172.21.15.32:0/3962120870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c60102710 0x7f1c6010f100 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c009390 tx=0x7f1c5c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:50:47.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.139+0000 7f1c567fc700 1 -- 172.21.15.32:0/3962120870 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c5c00e040 con 0x7f1c60102710 2024-04-12T05:50:47.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.139+0000 7f1c67b2d700 1 -- 172.21.15.32:0/3962120870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c60117a70 con 0x7f1c60102710 2024-04-12T05:50:47.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.139+0000 7f1c567fc700 1 -- 172.21.15.32:0/3962120870 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1c5c013420 con 0x7f1c60102710 2024-04-12T05:50:47.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.139+0000 7f1c567fc700 1 -- 172.21.15.32:0/3962120870 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c5c01c630 con 0x7f1c60102710 2024-04-12T05:50:47.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.140+0000 7f1c67b2d700 1 -- 172.21.15.32:0/3962120870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c60117fd0 con 0x7f1c60102710 2024-04-12T05:50:47.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.141+0000 7f1c67b2d700 1 -- 172.21.15.32:0/3962120870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c600ff080 con 0x7f1c60102710 2024-04-12T05:50:47.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.141+0000 7f1c567fc700 1 -- 172.21.15.32:0/3962120870 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1c5c013a00 con 0x7f1c60102710 2024-04-12T05:50:47.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.141+0000 7f1c567fc700 1 --2- 172.21.15.32:0/3962120870 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1c4c06cc30 0x7f1c4c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:47.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.141+0000 7f1c567fc700 1 -- 172.21.15.32:0/3962120870 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f1c5c08d380 con 0x7f1c60102710 2024-04-12T05:50:47.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.142+0000 7f1c658c9700 1 --2- 172.21.15.32:0/3962120870 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1c4c06cc30 0x7f1c4c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:47.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.142+0000 7f1c658c9700 1 --2- 172.21.15.32:0/3962120870 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1c4c06cc30 0x7f1c4c06f0e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1c50007fd0 tx=0x7f1c50009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:50:47.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.146+0000 7f1c567fc700 1 -- 172.21.15.32:0/3962120870 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1c5c057360 con 0x7f1c60102710 2024-04-12T05:50:47.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:47 smithi032 ceph-mon[146817]: pgmap v308: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:47.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:47 smithi032 ceph-mon[146817]: from='client.34620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:50:47.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.357+0000 7f1c67b2d700 1 -- 172.21.15.32:0/3962120870 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c60065e80 con 0x7f1c4c06cc30 2024-04-12T05:50:47.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.366+0000 7f1c567fc700 1 -- 172.21.15.32:0/3962120870 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f1c60065e80 con 0x7f1c4c06cc30 2024-04-12T05:50:47.362 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:50:47.362 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (10m) 4m ago 22m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:50:47.362 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (8m) 4m ago 22m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:50:47.363 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (8m) 3m ago 21m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:50:47.363 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (10m) 4m ago 21m 87.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:50:47.363 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (4m) 4m ago 16m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T05:50:47.363 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (4m) 4m ago 16m 13.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T05:50:47.363 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (3m) 3m ago 16m 11.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 6d4aca6f74a0 2024-04-12T05:50:47.363 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (3m) 3m ago 16m 14.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T05:50:47.363 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (12m) 4m ago 23m 507M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:50:47.363 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (11m) 3m ago 21m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:50:47.363 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (9m) 4m ago 23m 53.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:50:47.363 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (8m) 3m ago 20m 43.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:50:47.363 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (11m) 4m ago 22m 16.9M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:50:47.363 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (11m) 3m ago 21m 18.4M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:50:47.363 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (7m) 4m ago 20m 91.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T05:50:47.363 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (7m) 4m ago 19m 97.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T05:50:47.363 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (6m) 4m ago 19m 81.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T05:50:47.363 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (5m) 3m ago 18m 113M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T05:50:47.364 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (5m) 3m ago 18m 90.3M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T05:50:47.364 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (4m) 3m ago 17m 84.8M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T05:50:47.364 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (10m) 4m ago 21m 64.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:50:47.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.369+0000 7f1c67b2d700 1 -- 172.21.15.32:0/3962120870 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1c4c06cc30 msgr2=0x7f1c4c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:47.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.369+0000 7f1c67b2d700 1 --2- 172.21.15.32:0/3962120870 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1c4c06cc30 0x7f1c4c06f0e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1c50007fd0 tx=0x7f1c50009040 comp rx=0 tx=0).stop 2024-04-12T05:50:47.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.369+0000 7f1c67b2d700 1 -- 172.21.15.32:0/3962120870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c60102710 msgr2=0x7f1c6010f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:47.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.369+0000 7f1c67b2d700 1 --2- 172.21.15.32:0/3962120870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c60102710 0x7f1c6010f100 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c009390 tx=0x7f1c5c009e30 comp rx=0 tx=0).stop 2024-04-12T05:50:47.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.369+0000 7f1c67b2d700 1 -- 172.21.15.32:0/3962120870 shutdown_connections 2024-04-12T05:50:47.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.369+0000 7f1c67b2d700 1 --2- 172.21.15.32:0/3962120870 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1c4c06cc30 0x7f1c4c06f0e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:47.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.369+0000 7f1c67b2d700 1 --2- 172.21.15.32:0/3962120870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c60101d70 0x7f1c6010ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:47.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.369+0000 7f1c67b2d700 1 --2- 172.21.15.32:0/3962120870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c60102710 0x7f1c6010f100 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:47.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.369+0000 7f1c67b2d700 1 -- 172.21.15.32:0/3962120870 >> 172.21.15.32:0/3962120870 conn(0x7f1c60072bc0 msgr2=0x7f1c600fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:50:47.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.369+0000 7f1c67b2d700 1 -- 172.21.15.32:0/3962120870 shutdown_connections 2024-04-12T05:50:47.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.369+0000 7f1c67b2d700 1 -- 172.21.15.32:0/3962120870 wait complete. 2024-04-12T05:50:47.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.471+0000 7fae37386700 1 -- 172.21.15.32:0/1527643638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae301037b0 msgr2=0x7fae300fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:47.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.471+0000 7fae37386700 1 --2- 172.21.15.32:0/1527643638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae301037b0 0x7fae300fce20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fae24004940 tx=0x7fae24009e30 comp rx=0 tx=0).stop 2024-04-12T05:50:47.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.471+0000 7fae37386700 1 -- 172.21.15.32:0/1527643638 shutdown_connections 2024-04-12T05:50:47.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.471+0000 7fae37386700 1 --2- 172.21.15.32:0/1527643638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae301037b0 0x7fae300fce20 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:47.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.471+0000 7fae37386700 1 --2- 172.21.15.32:0/1527643638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae30102e10 0x7fae301031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:47.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.471+0000 7fae37386700 1 -- 172.21.15.32:0/1527643638 >> 172.21.15.32:0/1527643638 conn(0x7fae300f8980 msgr2=0x7fae300fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:50:47.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.471+0000 7fae37386700 1 -- 172.21.15.32:0/1527643638 shutdown_connections 2024-04-12T05:50:47.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.471+0000 7fae37386700 1 -- 172.21.15.32:0/1527643638 wait complete. 2024-04-12T05:50:47.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.472+0000 7fae37386700 1 Processor -- start 2024-04-12T05:50:47.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.472+0000 7fae37386700 1 -- start start 2024-04-12T05:50:47.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.472+0000 7fae37386700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae30102e10 0x7fae30113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:47.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.472+0000 7fae37386700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae301037b0 0x7fae30113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:47.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.472+0000 7fae37386700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae301145f0 con 0x7fae301037b0 2024-04-12T05:50:47.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.472+0000 7fae37386700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3010d9e0 con 0x7fae30102e10 2024-04-12T05:50:47.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.472+0000 7fae34921700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae301037b0 0x7fae30113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:47.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.472+0000 7fae34921700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae301037b0 0x7fae30113fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55764/0 (socket says 172.21.15.32:55764) 2024-04-12T05:50:47.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.472+0000 7fae34921700 1 -- 172.21.15.32:0/1576531072 learned_addr learned my addr 172.21.15.32:0/1576531072 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:50:47.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.472+0000 7fae35122700 1 --2- 172.21.15.32:0/1576531072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae30102e10 0x7fae30113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:47.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.472+0000 7fae34921700 1 -- 172.21.15.32:0/1576531072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae30102e10 msgr2=0x7fae30113a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:47.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.472+0000 7fae34921700 1 --2- 172.21.15.32:0/1576531072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae30102e10 0x7fae30113a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:47.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.472+0000 7fae34921700 1 -- 172.21.15.32:0/1576531072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae24005040 con 0x7fae301037b0 2024-04-12T05:50:47.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.473+0000 7fae34921700 1 --2- 172.21.15.32:0/1576531072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae301037b0 0x7fae30113fa0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fae24000c00 tx=0x7fae24009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:50:47.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.473+0000 7fae21ffb700 1 -- 172.21.15.32:0/1576531072 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae2400e070 con 0x7fae301037b0 2024-04-12T05:50:47.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.473+0000 7fae21ffb700 1 -- 172.21.15.32:0/1576531072 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fae240093b0 con 0x7fae301037b0 2024-04-12T05:50:47.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.473+0000 7fae37386700 1 -- 172.21.15.32:0/1576531072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae3010dc60 con 0x7fae301037b0 2024-04-12T05:50:47.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.473+0000 7fae21ffb700 1 -- 172.21.15.32:0/1576531072 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae2401ccb0 con 0x7fae301037b0 2024-04-12T05:50:47.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.473+0000 7fae37386700 1 -- 172.21.15.32:0/1576531072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae3010e1c0 con 0x7fae301037b0 2024-04-12T05:50:47.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.474+0000 7fae37386700 1 -- 172.21.15.32:0/1576531072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae300fe560 con 0x7fae301037b0 2024-04-12T05:50:47.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.475+0000 7fae21ffb700 1 -- 172.21.15.32:0/1576531072 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fae2401c490 con 0x7fae301037b0 2024-04-12T05:50:47.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.475+0000 7fae21ffb700 1 --2- 172.21.15.32:0/1576531072 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fae1c06cc20 0x7fae1c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:47.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.475+0000 7fae21ffb700 1 -- 172.21.15.32:0/1576531072 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fae2408d5a0 con 0x7fae301037b0 2024-04-12T05:50:47.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.475+0000 7fae35122700 1 --2- 172.21.15.32:0/1576531072 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fae1c06cc20 0x7fae1c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:47.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.475+0000 7fae35122700 1 --2- 172.21.15.32:0/1576531072 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fae1c06cc20 0x7fae1c06f0d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fae2c009b80 tx=0x7fae2c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:50:47.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.479+0000 7fae21ffb700 1 -- 172.21.15.32:0/1576531072 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fae2405af70 con 0x7fae301037b0 2024-04-12T05:50:47.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:47 smithi067 ceph-mon[112979]: pgmap v308: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:47.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:47 smithi067 ceph-mon[112979]: from='client.34620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:50:47.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.766+0000 7fae37386700 1 -- 172.21.15.32:0/1576531072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fae30109db0 con 0x7fae301037b0 2024-04-12T05:50:47.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.767+0000 7fae21ffb700 1 -- 172.21.15.32:0/1576531072 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fae2405ab00 con 0x7fae301037b0 2024-04-12T05:50:47.763 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:50:47.763 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:50:47.763 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:50:47.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:50:47.763 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:50:47.763 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:50:47.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:50:47.764 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:50:47.764 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:50:47.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:50:47.764 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:50:47.764 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:50:47.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:50:47.764 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:50:47.764 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:50:47.764 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:50:47.764 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:50:47.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.769+0000 7fae37386700 1 -- 172.21.15.32:0/1576531072 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fae1c06cc20 msgr2=0x7fae1c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:47.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.769+0000 7fae37386700 1 --2- 172.21.15.32:0/1576531072 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fae1c06cc20 0x7fae1c06f0d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fae2c009b80 tx=0x7fae2c00b040 comp rx=0 tx=0).stop 2024-04-12T05:50:47.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.769+0000 7fae37386700 1 -- 172.21.15.32:0/1576531072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae301037b0 msgr2=0x7fae30113fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:47.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.769+0000 7fae37386700 1 --2- 172.21.15.32:0/1576531072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae301037b0 0x7fae30113fa0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fae24000c00 tx=0x7fae24009e30 comp rx=0 tx=0).stop 2024-04-12T05:50:47.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.769+0000 7fae37386700 1 -- 172.21.15.32:0/1576531072 shutdown_connections 2024-04-12T05:50:47.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.769+0000 7fae37386700 1 --2- 172.21.15.32:0/1576531072 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fae1c06cc20 0x7fae1c06f0d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:47.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.769+0000 7fae37386700 1 --2- 172.21.15.32:0/1576531072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae30102e10 0x7fae30113a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:47.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.769+0000 7fae37386700 1 --2- 172.21.15.32:0/1576531072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae301037b0 0x7fae30113fa0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:47.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.769+0000 7fae37386700 1 -- 172.21.15.32:0/1576531072 >> 172.21.15.32:0/1576531072 conn(0x7fae300f8980 msgr2=0x7fae300f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:50:47.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.769+0000 7fae37386700 1 -- 172.21.15.32:0/1576531072 shutdown_connections 2024-04-12T05:50:47.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.769+0000 7fae37386700 1 -- 172.21.15.32:0/1576531072 wait complete. 2024-04-12T05:50:47.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.874+0000 7f36ca590700 1 -- 172.21.15.32:0/3799480387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36c4102710 msgr2=0x7f36c40fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:47.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.874+0000 7f36ca590700 1 --2- 172.21.15.32:0/3799480387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36c4102710 0x7f36c40fbd80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f36b8004940 tx=0x7f36b8009e30 comp rx=0 tx=0).stop 2024-04-12T05:50:47.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.874+0000 7f36ca590700 1 -- 172.21.15.32:0/3799480387 shutdown_connections 2024-04-12T05:50:47.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.874+0000 7f36ca590700 1 --2- 172.21.15.32:0/3799480387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36c4102710 0x7f36c40fbd80 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:47.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.874+0000 7f36ca590700 1 --2- 172.21.15.32:0/3799480387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36c4101d70 0x7f36c4102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:47.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.874+0000 7f36ca590700 1 -- 172.21.15.32:0/3799480387 >> 172.21.15.32:0/3799480387 conn(0x7f36c4072bc0 msgr2=0x7f36c4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:50:47.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.874+0000 7f36ca590700 1 -- 172.21.15.32:0/3799480387 shutdown_connections 2024-04-12T05:50:47.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.874+0000 7f36ca590700 1 -- 172.21.15.32:0/3799480387 wait complete. 2024-04-12T05:50:47.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.874+0000 7f36ca590700 1 Processor -- start 2024-04-12T05:50:47.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.874+0000 7f36ca590700 1 -- start start 2024-04-12T05:50:47.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.875+0000 7f36ca590700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36c4101d70 0x7f36c410ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:47.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.875+0000 7f36ca590700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36c4102710 0x7f36c410f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:47.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.875+0000 7f36ca590700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c410f680 con 0x7f36c4102710 2024-04-12T05:50:47.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.875+0000 7f36ca590700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c410f7c0 con 0x7f36c4101d70 2024-04-12T05:50:47.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.875+0000 7f36c37fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36c4102710 0x7f36c410f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:47.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.875+0000 7f36c37fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36c4102710 0x7f36c410f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55788/0 (socket says 172.21.15.32:55788) 2024-04-12T05:50:47.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.875+0000 7f36c37fe700 1 -- 172.21.15.32:0/3658468778 learned_addr learned my addr 172.21.15.32:0/3658468778 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:50:47.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.875+0000 7f36c3fff700 1 --2- 172.21.15.32:0/3658468778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36c4101d70 0x7f36c410ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:47.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.875+0000 7f36c37fe700 1 -- 172.21.15.32:0/3658468778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36c4101d70 msgr2=0x7f36c410ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:47.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.875+0000 7f36c37fe700 1 --2- 172.21.15.32:0/3658468778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36c4101d70 0x7f36c410ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:47.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.875+0000 7f36c37fe700 1 -- 172.21.15.32:0/3658468778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b8005040 con 0x7f36c4102710 2024-04-12T05:50:47.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.875+0000 7f36c37fe700 1 --2- 172.21.15.32:0/3658468778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36c4102710 0x7f36c410f140 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f36b8009390 tx=0x7f36b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:50:47.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.875+0000 7f36c0ff9700 1 -- 172.21.15.32:0/3658468778 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36b800e040 con 0x7f36c4102710 2024-04-12T05:50:47.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.875+0000 7f36c0ff9700 1 -- 172.21.15.32:0/3658468778 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f36b8004290 con 0x7f36c4102710 2024-04-12T05:50:47.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.875+0000 7f36ca590700 1 -- 172.21.15.32:0/3658468778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c4117ab0 con 0x7f36c4102710 2024-04-12T05:50:47.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.875+0000 7f36c0ff9700 1 -- 172.21.15.32:0/3658468778 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36b8013e70 con 0x7f36c4102710 2024-04-12T05:50:47.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.876+0000 7f36ca590700 1 -- 172.21.15.32:0/3658468778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36c4118040 con 0x7f36c4102710 2024-04-12T05:50:47.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.876+0000 7f36ca590700 1 -- 172.21.15.32:0/3658468778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c40ff080 con 0x7f36c4102710 2024-04-12T05:50:47.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.877+0000 7f36c0ff9700 1 -- 172.21.15.32:0/3658468778 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f36b8013690 con 0x7f36c4102710 2024-04-12T05:50:47.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.877+0000 7f36c0ff9700 1 --2- 172.21.15.32:0/3658468778 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f36ac06cbc0 0x7f36ac06f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:47.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.878+0000 7f36c0ff9700 1 -- 172.21.15.32:0/3658468778 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f36b808d2a0 con 0x7f36c4102710 2024-04-12T05:50:47.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.878+0000 7f36c3fff700 1 --2- 172.21.15.32:0/3658468778 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f36ac06cbc0 0x7f36ac06f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:47.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.878+0000 7f36c3fff700 1 --2- 172.21.15.32:0/3658468778 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f36ac06cbc0 0x7f36ac06f070 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f36b4004f40 tx=0x7f36b400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:50:47.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:47.882+0000 7f36c0ff9700 1 -- 172.21.15.32:0/3658468778 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f36b8057330 con 0x7f36c4102710 2024-04-12T05:50:48.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.131+0000 7f36ca590700 1 -- 172.21.15.32:0/3658468778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f36c40fd4c0 con 0x7f36c4102710 2024-04-12T05:50:48.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.131+0000 7f36c0ff9700 1 -- 172.21.15.32:0/3658468778 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1835 (secure 0 0 0) 0x7f36b8013420 con 0x7f36c4102710 2024-04-12T05:50:48.127 INFO:teuthology.orchestra.run.smithi032.stdout:e33 2024-04-12T05:50:48.127 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:50:48.127 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:50:48.128 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:50:48.128 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:50:48.128 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:50:48.128 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:50:48.128 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 24 2024-04-12T05:50:48.128 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:50:48.128 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:50:48.128 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:46:10.668569+0000 2024-04-12T05:50:48.128 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:50:48.128 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:50:48.128 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:50:48.128 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:50:48.128 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:50:48.128 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:50:48.129 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:50:48.129 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:50:48.129 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T05:50:48.129 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:50:48.129 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:50:48.129 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:50:48.129 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T05:50:48.129 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-12T05:50:48.129 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:50:48.129 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:50:48.129 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:50:48.129 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:50:48.129 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:50:48.129 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:50:48.130 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:50:48.130 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:50:48.130 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:50:48.130 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:50:48.130 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-12T05:50:48.130 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:50:48.130 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:50:48.130 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:50:48.130 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{-1:44317} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:50:48.130 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:50:48.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.133+0000 7f36ca590700 1 -- 172.21.15.32:0/3658468778 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f36ac06cbc0 msgr2=0x7f36ac06f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:48.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.133+0000 7f36ca590700 1 --2- 172.21.15.32:0/3658468778 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f36ac06cbc0 0x7f36ac06f070 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f36b4004f40 tx=0x7f36b400b040 comp rx=0 tx=0).stop 2024-04-12T05:50:48.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.133+0000 7f36ca590700 1 -- 172.21.15.32:0/3658468778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36c4102710 msgr2=0x7f36c410f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:48.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.133+0000 7f36ca590700 1 --2- 172.21.15.32:0/3658468778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36c4102710 0x7f36c410f140 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f36b8009390 tx=0x7f36b8009e30 comp rx=0 tx=0).stop 2024-04-12T05:50:48.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.133+0000 7f36ca590700 1 -- 172.21.15.32:0/3658468778 shutdown_connections 2024-04-12T05:50:48.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.133+0000 7f36ca590700 1 --2- 172.21.15.32:0/3658468778 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f36ac06cbc0 0x7f36ac06f070 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:48.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.133+0000 7f36ca590700 1 --2- 172.21.15.32:0/3658468778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36c4101d70 0x7f36c410ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:48.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.133+0000 7f36ca590700 1 --2- 172.21.15.32:0/3658468778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36c4102710 0x7f36c410f140 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:48.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.133+0000 7f36ca590700 1 -- 172.21.15.32:0/3658468778 >> 172.21.15.32:0/3658468778 conn(0x7f36c4072bc0 msgr2=0x7f36c40fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:50:48.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.133+0000 7f36ca590700 1 -- 172.21.15.32:0/3658468778 shutdown_connections 2024-04-12T05:50:48.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.133+0000 7f36ca590700 1 -- 172.21.15.32:0/3658468778 wait complete. 2024-04-12T05:50:48.132 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-12T05:50:48.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.237+0000 7f9d4ed7a700 1 -- 172.21.15.32:0/577880764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d480fde90 msgr2=0x7f9d480fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:48.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.237+0000 7f9d4ed7a700 1 --2- 172.21.15.32:0/577880764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d480fde90 0x7f9d480fe260 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c004940 tx=0x7f9d3c009e30 comp rx=0 tx=0).stop 2024-04-12T05:50:48.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.237+0000 7f9d4ed7a700 1 -- 172.21.15.32:0/577880764 shutdown_connections 2024-04-12T05:50:48.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.237+0000 7f9d4ed7a700 1 --2- 172.21.15.32:0/577880764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d480fe830 0x7f9d48106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:48.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.237+0000 7f9d4ed7a700 1 --2- 172.21.15.32:0/577880764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d480fde90 0x7f9d480fe260 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:48.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.237+0000 7f9d4ed7a700 1 -- 172.21.15.32:0/577880764 >> 172.21.15.32:0/577880764 conn(0x7f9d48072bc0 msgr2=0x7f9d48072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:50:48.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.238+0000 7f9d4ed7a700 1 -- 172.21.15.32:0/577880764 shutdown_connections 2024-04-12T05:50:48.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.238+0000 7f9d4ed7a700 1 -- 172.21.15.32:0/577880764 wait complete. 2024-04-12T05:50:48.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.238+0000 7f9d4ed7a700 1 Processor -- start 2024-04-12T05:50:48.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.238+0000 7f9d4ed7a700 1 -- start start 2024-04-12T05:50:48.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.238+0000 7f9d4ed7a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d480fde90 0x7f9d48113a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:48.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.238+0000 7f9d4ed7a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d480fe830 0x7f9d48113fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:48.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.238+0000 7f9d4ed7a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d48114620 con 0x7f9d480fe830 2024-04-12T05:50:48.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.238+0000 7f9d4ed7a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4810da50 con 0x7f9d480fde90 2024-04-12T05:50:48.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.239+0000 7f9d47fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d480fe830 0x7f9d48113fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:48.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.239+0000 7f9d47fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d480fe830 0x7f9d48113fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55798/0 (socket says 172.21.15.32:55798) 2024-04-12T05:50:48.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.239+0000 7f9d47fff700 1 -- 172.21.15.32:0/848274504 learned_addr learned my addr 172.21.15.32:0/848274504 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:50:48.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.239+0000 7f9d4cb16700 1 --2- 172.21.15.32:0/848274504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d480fde90 0x7f9d48113a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:48.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.239+0000 7f9d47fff700 1 -- 172.21.15.32:0/848274504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d480fde90 msgr2=0x7f9d48113a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:48.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.239+0000 7f9d47fff700 1 --2- 172.21.15.32:0/848274504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d480fde90 0x7f9d48113a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:48.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.239+0000 7f9d47fff700 1 -- 172.21.15.32:0/848274504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d3c005040 con 0x7f9d480fe830 2024-04-12T05:50:48.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.239+0000 7f9d47fff700 1 --2- 172.21.15.32:0/848274504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d480fe830 0x7f9d48113fd0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9d34002a50 tx=0x7f9d340046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:50:48.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.239+0000 7f9d457fa700 1 -- 172.21.15.32:0/848274504 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d34007cc0 con 0x7f9d480fe830 2024-04-12T05:50:48.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.239+0000 7f9d457fa700 1 -- 172.21.15.32:0/848274504 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d3400f040 con 0x7f9d480fe830 2024-04-12T05:50:48.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.239+0000 7f9d4ed7a700 1 -- 172.21.15.32:0/848274504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d4810dd30 con 0x7f9d480fe830 2024-04-12T05:50:48.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.239+0000 7f9d457fa700 1 -- 172.21.15.32:0/848274504 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d3400d9c0 con 0x7f9d480fe830 2024-04-12T05:50:48.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.239+0000 7f9d4ed7a700 1 -- 172.21.15.32:0/848274504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d4810e270 con 0x7f9d480fe830 2024-04-12T05:50:48.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.240+0000 7f9d4ed7a700 1 -- 172.21.15.32:0/848274504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d481019b0 con 0x7f9d480fe830 2024-04-12T05:50:48.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.241+0000 7f9d457fa700 1 -- 172.21.15.32:0/848274504 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9d3400db20 con 0x7f9d480fe830 2024-04-12T05:50:48.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.241+0000 7f9d457fa700 1 --2- 172.21.15.32:0/848274504 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9d3806cc20 0x7f9d3806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:48.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.241+0000 7f9d457fa700 1 -- 172.21.15.32:0/848274504 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f9d3408bd10 con 0x7f9d480fe830 2024-04-12T05:50:48.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.241+0000 7f9d4cb16700 1 --2- 172.21.15.32:0/848274504 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9d3806cc20 0x7f9d3806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:48.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.242+0000 7f9d4cb16700 1 --2- 172.21.15.32:0/848274504 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9d3806cc20 0x7f9d3806f0d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c00b7e0 tx=0x7f9d3c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:50:48.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.246+0000 7f9d457fa700 1 -- 172.21.15.32:0/848274504 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9d34059780 con 0x7f9d480fe830 2024-04-12T05:50:48.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:48 smithi032 ceph-mon[146817]: from='client.34624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:50:48.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:48 smithi032 ceph-mon[146817]: from='client.34628 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:50:48.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:48 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1576531072' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:50:48.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:48 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:50:48.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.466+0000 7f9d4ed7a700 1 -- 172.21.15.32:0/848274504 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d4810f1b0 con 0x7f9d3806cc20 2024-04-12T05:50:48.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.467+0000 7f9d457fa700 1 -- 172.21.15.32:0/848274504 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9d4810f1b0 con 0x7f9d3806cc20 2024-04-12T05:50:48.463 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:50:48.463 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:50:48.463 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:50:48.463 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:50:48.463 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:50:48.463 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-12T05:50:48.463 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:50:48.463 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-12T05:50:48.463 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:50:48.463 INFO:teuthology.orchestra.run.smithi032.stdout: "mds" 2024-04-12T05:50:48.464 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:50:48.464 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:50:48.464 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:50:48.464 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:50:48.464 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:50:48.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.469+0000 7f9d4ed7a700 1 -- 172.21.15.32:0/848274504 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9d3806cc20 msgr2=0x7f9d3806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:48.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.469+0000 7f9d4ed7a700 1 --2- 172.21.15.32:0/848274504 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9d3806cc20 0x7f9d3806f0d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c00b7e0 tx=0x7f9d3c009e30 comp rx=0 tx=0).stop 2024-04-12T05:50:48.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.469+0000 7f9d4ed7a700 1 -- 172.21.15.32:0/848274504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d480fe830 msgr2=0x7f9d48113fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:48.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.469+0000 7f9d4ed7a700 1 --2- 172.21.15.32:0/848274504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d480fe830 0x7f9d48113fd0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9d34002a50 tx=0x7f9d340046b0 comp rx=0 tx=0).stop 2024-04-12T05:50:48.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.469+0000 7f9d4ed7a700 1 -- 172.21.15.32:0/848274504 shutdown_connections 2024-04-12T05:50:48.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.469+0000 7f9d4ed7a700 1 --2- 172.21.15.32:0/848274504 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9d3806cc20 0x7f9d3806f0d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:48.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.469+0000 7f9d4ed7a700 1 --2- 172.21.15.32:0/848274504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d480fde90 0x7f9d48113a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:48.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.469+0000 7f9d4ed7a700 1 --2- 172.21.15.32:0/848274504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d480fe830 0x7f9d48113fd0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:48.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.469+0000 7f9d4ed7a700 1 -- 172.21.15.32:0/848274504 >> 172.21.15.32:0/848274504 conn(0x7f9d48072bc0 msgr2=0x7f9d480ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:50:48.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.469+0000 7f9d4ed7a700 1 -- 172.21.15.32:0/848274504 shutdown_connections 2024-04-12T05:50:48.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.469+0000 7f9d4ed7a700 1 -- 172.21.15.32:0/848274504 wait complete. 2024-04-12T05:50:48.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:48 smithi067 ceph-mon[112979]: from='client.34624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:50:48.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:48 smithi067 ceph-mon[112979]: from='client.34628 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:50:48.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:48 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1576531072' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:50:48.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:48 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:50:48.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.572+0000 7f0ca0c95700 1 -- 172.21.15.32:0/724525825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c9c0fde00 msgr2=0x7f0c9c0fe1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:48.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.572+0000 7f0ca0c95700 1 --2- 172.21.15.32:0/724525825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c9c0fde00 0x7f0c9c0fe1d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c004940 tx=0x7f0c8c009e30 comp rx=0 tx=0).stop 2024-04-12T05:50:48.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.572+0000 7f0ca0c95700 1 -- 172.21.15.32:0/724525825 shutdown_connections 2024-04-12T05:50:48.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.572+0000 7f0ca0c95700 1 --2- 172.21.15.32:0/724525825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c9c0fe7a0 0x7f0c9c106de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:48.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.572+0000 7f0ca0c95700 1 --2- 172.21.15.32:0/724525825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c9c0fde00 0x7f0c9c0fe1d0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:48.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.572+0000 7f0ca0c95700 1 -- 172.21.15.32:0/724525825 >> 172.21.15.32:0/724525825 conn(0x7f0c9c0f78d0 msgr2=0x7f0c9c0f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:50:48.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.572+0000 7f0ca0c95700 1 -- 172.21.15.32:0/724525825 shutdown_connections 2024-04-12T05:50:48.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.572+0000 7f0ca0c95700 1 -- 172.21.15.32:0/724525825 wait complete. 2024-04-12T05:50:48.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.573+0000 7f0ca0c95700 1 Processor -- start 2024-04-12T05:50:48.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.573+0000 7f0ca0c95700 1 -- start start 2024-04-12T05:50:48.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.573+0000 7f0ca0c95700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c9c0fde00 0x7f0c9c113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:48.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.573+0000 7f0ca0c95700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c9c0fe7a0 0x7f0c9c114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:48.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.573+0000 7f0ca0c95700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9c114660 con 0x7f0c9c0fe7a0 2024-04-12T05:50:48.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.573+0000 7f0ca0c95700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9c10da50 con 0x7f0c9c0fde00 2024-04-12T05:50:48.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.573+0000 7f0c93fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c9c0fe7a0 0x7f0c9c114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:48.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.573+0000 7f0c93fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c9c0fe7a0 0x7f0c9c114010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55808/0 (socket says 172.21.15.32:55808) 2024-04-12T05:50:48.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.573+0000 7f0c93fff700 1 -- 172.21.15.32:0/3339737761 learned_addr learned my addr 172.21.15.32:0/3339737761 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:50:48.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.573+0000 7f0c9a59c700 1 --2- 172.21.15.32:0/3339737761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c9c0fde00 0x7f0c9c113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:48.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.574+0000 7f0c93fff700 1 -- 172.21.15.32:0/3339737761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c9c0fde00 msgr2=0x7f0c9c113ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:48.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.574+0000 7f0c93fff700 1 --2- 172.21.15.32:0/3339737761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c9c0fde00 0x7f0c9c113ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:48.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.574+0000 7f0c93fff700 1 -- 172.21.15.32:0/3339737761 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c8c005040 con 0x7f0c9c0fe7a0 2024-04-12T05:50:48.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.574+0000 7f0c93fff700 1 --2- 172.21.15.32:0/3339737761 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c9c0fe7a0 0x7f0c9c114010 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0c84002a50 tx=0x7f0c840046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:50:48.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.574+0000 7f0c92ffd700 1 -- 172.21.15.32:0/3339737761 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c84007cc0 con 0x7f0c9c0fe7a0 2024-04-12T05:50:48.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.574+0000 7f0c92ffd700 1 -- 172.21.15.32:0/3339737761 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c8400f040 con 0x7f0c9c0fe7a0 2024-04-12T05:50:48.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.574+0000 7f0c92ffd700 1 -- 172.21.15.32:0/3339737761 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c8400d9c0 con 0x7f0c9c0fe7a0 2024-04-12T05:50:48.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.574+0000 7f0ca0c95700 1 -- 172.21.15.32:0/3339737761 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c9c10dd30 con 0x7f0c9c0fe7a0 2024-04-12T05:50:48.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.574+0000 7f0ca0c95700 1 -- 172.21.15.32:0/3339737761 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c9c10e270 con 0x7f0c9c0fe7a0 2024-04-12T05:50:48.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.575+0000 7f0ca0c95700 1 -- 172.21.15.32:0/3339737761 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c9c066770 con 0x7f0c9c0fe7a0 2024-04-12T05:50:48.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.576+0000 7f0c92ffd700 1 -- 172.21.15.32:0/3339737761 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f0c8400db20 con 0x7f0c9c0fe7a0 2024-04-12T05:50:48.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.576+0000 7f0c92ffd700 1 --2- 172.21.15.32:0/3339737761 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0c8806cc30 0x7f0c8806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:50:48.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.576+0000 7f0c92ffd700 1 -- 172.21.15.32:0/3339737761 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f0c8408bd10 con 0x7f0c9c0fe7a0 2024-04-12T05:50:48.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.576+0000 7f0c9a59c700 1 --2- 172.21.15.32:0/3339737761 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0c8806cc30 0x7f0c8806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:48.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.577+0000 7f0c9a59c700 1 --2- 172.21.15.32:0/3339737761 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0c8806cc30 0x7f0c8806f0e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c009fd0 tx=0x7f0c8c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:50:48.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.581+0000 7f0c92ffd700 1 -- 172.21.15.32:0/3339737761 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0c84059780 con 0x7f0c9c0fe7a0 2024-04-12T05:50:48.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.870+0000 7f0ca0c95700 1 -- 172.21.15.32:0/3339737761 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0c9c109d50 con 0x7f0c9c0fe7a0 2024-04-12T05:50:48.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.870+0000 7f0c92ffd700 1 -- 172.21.15.32:0/3339737761 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f0c84059310 con 0x7f0c9c0fe7a0 2024-04-12T05:50:48.867 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:50:48.867 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:50:48.867 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-12T05:50:48.867 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:50:48.867 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:50:48.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.872+0000 7f0ca0c95700 1 -- 172.21.15.32:0/3339737761 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0c8806cc30 msgr2=0x7f0c8806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:48.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.872+0000 7f0ca0c95700 1 --2- 172.21.15.32:0/3339737761 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0c8806cc30 0x7f0c8806f0e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c009fd0 tx=0x7f0c8c009e30 comp rx=0 tx=0).stop 2024-04-12T05:50:48.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.872+0000 7f0ca0c95700 1 -- 172.21.15.32:0/3339737761 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c9c0fe7a0 msgr2=0x7f0c9c114010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:48.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.872+0000 7f0ca0c95700 1 --2- 172.21.15.32:0/3339737761 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c9c0fe7a0 0x7f0c9c114010 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0c84002a50 tx=0x7f0c840046b0 comp rx=0 tx=0).stop 2024-04-12T05:50:48.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.872+0000 7f0ca0c95700 1 -- 172.21.15.32:0/3339737761 shutdown_connections 2024-04-12T05:50:48.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.872+0000 7f0ca0c95700 1 --2- 172.21.15.32:0/3339737761 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0c8806cc30 0x7f0c8806f0e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:48.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.872+0000 7f0ca0c95700 1 --2- 172.21.15.32:0/3339737761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c9c0fde00 0x7f0c9c113ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:48.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.872+0000 7f0ca0c95700 1 --2- 172.21.15.32:0/3339737761 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c9c0fe7a0 0x7f0c9c114010 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:50:48.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.872+0000 7f0ca0c95700 1 -- 172.21.15.32:0/3339737761 >> 172.21.15.32:0/3339737761 conn(0x7f0c9c0f78d0 msgr2=0x7f0c9c0f8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:50:48.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.872+0000 7f0ca0c95700 1 -- 172.21.15.32:0/3339737761 shutdown_connections 2024-04-12T05:50:48.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:50:48.872+0000 7f0ca0c95700 1 -- 172.21.15.32:0/3339737761 wait complete. 2024-04-12T05:50:49.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:49 smithi032 ceph-mon[146817]: pgmap v309: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:49.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:49 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3658468778' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:50:49.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:49 smithi032 ceph-mon[146817]: from='client.34640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:50:49.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:49 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3339737761' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:50:49.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:49 smithi067 ceph-mon[112979]: pgmap v309: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:49.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:49 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3658468778' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:50:49.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:49 smithi067 ceph-mon[112979]: from='client.34640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:50:49.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:49 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3339737761' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:50:51.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:51 smithi032 ceph-mon[146817]: pgmap v310: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:51.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:51 smithi067 ceph-mon[112979]: pgmap v310: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:53.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:53 smithi032 ceph-mon[146817]: pgmap v311: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:53.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:53 smithi067 ceph-mon[112979]: pgmap v311: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:55.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:55 smithi032 ceph-mon[146817]: pgmap v312: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:55.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:55 smithi067 ceph-mon[112979]: pgmap v312: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:57.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:57 smithi067 ceph-mon[112979]: pgmap v313: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:57.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:57 smithi032 ceph-mon[146817]: pgmap v313: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:59.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:50:59 smithi067 ceph-mon[112979]: pgmap v314: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:59.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:50:59 smithi032 ceph-mon[146817]: pgmap v314: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:01.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:01 smithi067 ceph-mon[112979]: pgmap v315: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:01.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:01 smithi032 ceph-mon[146817]: pgmap v315: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:03.394 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:03 smithi032 ceph-mon[146817]: pgmap v316: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:03.394 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:51:03.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:03 smithi067 ceph-mon[112979]: pgmap v316: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:03.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:51:05.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:05 smithi067 ceph-mon[112979]: pgmap v317: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:05.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:05 smithi032 ceph-mon[146817]: pgmap v317: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:07.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:07 smithi067 ceph-mon[112979]: pgmap v318: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:07.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:07 smithi032 ceph-mon[146817]: pgmap v318: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:09.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:09 smithi067 ceph-mon[112979]: pgmap v319: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:09.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:09 smithi032 ceph-mon[146817]: pgmap v319: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:11.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:11 smithi067 ceph-mon[112979]: pgmap v320: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:11.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:11 smithi032 ceph-mon[146817]: pgmap v320: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:14.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:13 smithi067 ceph-mon[112979]: pgmap v321: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:14.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:13 smithi032 ceph-mon[146817]: pgmap v321: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:16.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:15 smithi067 ceph-mon[112979]: pgmap v322: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:16.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:15 smithi032 ceph-mon[146817]: pgmap v322: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:18.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:17 smithi067 ceph-mon[112979]: pgmap v323: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:18.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:17 smithi032 ceph-mon[146817]: pgmap v323: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:18.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.991+0000 7fd850769700 1 -- 172.21.15.32:0/1912143938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8480fe7b0 msgr2=0x7fd848106df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:18.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.991+0000 7fd850769700 1 --2- 172.21.15.32:0/1912143938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8480fe7b0 0x7fd848106df0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd838004940 tx=0x7fd838009e30 comp rx=0 tx=0).stop 2024-04-12T05:51:18.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.991+0000 7fd850769700 1 -- 172.21.15.32:0/1912143938 shutdown_connections 2024-04-12T05:51:18.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.991+0000 7fd850769700 1 --2- 172.21.15.32:0/1912143938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8480fe7b0 0x7fd848106df0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:18.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.991+0000 7fd850769700 1 --2- 172.21.15.32:0/1912143938 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8480fde10 0x7fd8480fe1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:18.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.991+0000 7fd850769700 1 -- 172.21.15.32:0/1912143938 >> 172.21.15.32:0/1912143938 conn(0x7fd8480f7880 msgr2=0x7fd8480f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:18.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.991+0000 7fd850769700 1 -- 172.21.15.32:0/1912143938 shutdown_connections 2024-04-12T05:51:18.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.991+0000 7fd850769700 1 -- 172.21.15.32:0/1912143938 wait complete. 2024-04-12T05:51:18.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.992+0000 7fd850769700 1 Processor -- start 2024-04-12T05:51:18.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.992+0000 7fd850769700 1 -- start start 2024-04-12T05:51:18.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.992+0000 7fd850769700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8480fde10 0x7fd848113d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:18.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.992+0000 7fd850769700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8480fe7b0 0x7fd8481142c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:18.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.992+0000 7fd850769700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd848114910 con 0x7fd8480fe7b0 2024-04-12T05:51:18.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.992+0000 7fd850769700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84810dd20 con 0x7fd8480fde10 2024-04-12T05:51:18.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.992+0000 7fd84dd04700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8480fe7b0 0x7fd8481142c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:18.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.992+0000 7fd84dd04700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8480fe7b0 0x7fd8481142c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42268/0 (socket says 172.21.15.32:42268) 2024-04-12T05:51:18.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.992+0000 7fd84dd04700 1 -- 172.21.15.32:0/2242211508 learned_addr learned my addr 172.21.15.32:0/2242211508 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:51:18.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.993+0000 7fd84e505700 1 --2- 172.21.15.32:0/2242211508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8480fde10 0x7fd848113d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:18.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.993+0000 7fd84dd04700 1 -- 172.21.15.32:0/2242211508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8480fde10 msgr2=0x7fd848113d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:18.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.993+0000 7fd84dd04700 1 --2- 172.21.15.32:0/2242211508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8480fde10 0x7fd848113d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:18.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.993+0000 7fd84dd04700 1 -- 172.21.15.32:0/2242211508 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd838005040 con 0x7fd8480fe7b0 2024-04-12T05:51:18.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.993+0000 7fd84e505700 1 --2- 172.21.15.32:0/2242211508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8480fde10 0x7fd848113d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:51:18.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.993+0000 7fd84dd04700 1 --2- 172.21.15.32:0/2242211508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8480fe7b0 0x7fd8481142c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd838009380 tx=0x7fd838009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:18.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.993+0000 7fd83effd700 1 -- 172.21.15.32:0/2242211508 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd83800e040 con 0x7fd8480fe7b0 2024-04-12T05:51:18.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.993+0000 7fd83effd700 1 -- 172.21.15.32:0/2242211508 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd838004290 con 0x7fd8480fe7b0 2024-04-12T05:51:18.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.993+0000 7fd850769700 1 -- 172.21.15.32:0/2242211508 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd84810dfa0 con 0x7fd8480fe7b0 2024-04-12T05:51:18.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.993+0000 7fd83effd700 1 -- 172.21.15.32:0/2242211508 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd83801c3f0 con 0x7fd8480fe7b0 2024-04-12T05:51:18.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.993+0000 7fd850769700 1 -- 172.21.15.32:0/2242211508 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd84810e4e0 con 0x7fd8480fe7b0 2024-04-12T05:51:18.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.994+0000 7fd850769700 1 -- 172.21.15.32:0/2242211508 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd848066770 con 0x7fd8480fe7b0 2024-04-12T05:51:18.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.995+0000 7fd83effd700 1 -- 172.21.15.32:0/2242211508 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd838013790 con 0x7fd8480fe7b0 2024-04-12T05:51:18.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.995+0000 7fd83effd700 1 --2- 172.21.15.32:0/2242211508 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd83406cc20 0x7fd83406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:18.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.995+0000 7fd83effd700 1 -- 172.21.15.32:0/2242211508 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fd83808d400 con 0x7fd8480fe7b0 2024-04-12T05:51:18.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.995+0000 7fd84e505700 1 --2- 172.21.15.32:0/2242211508 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd83406cc20 0x7fd83406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:18.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:18.996+0000 7fd84e505700 1 --2- 172.21.15.32:0/2242211508 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd83406cc20 0x7fd83406f0d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd844001250 tx=0x7fd84400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:18.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.000+0000 7fd83effd700 1 -- 172.21.15.32:0/2242211508 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd8380573e0 con 0x7fd8480fe7b0 2024-04-12T05:51:19.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:51:19.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:51:19.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.222+0000 7fd850769700 1 -- 172.21.15.32:0/2242211508 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd848109d60 con 0x7fd83406cc20 2024-04-12T05:51:19.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.223+0000 7fd83effd700 1 -- 172.21.15.32:0/2242211508 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd848109d60 con 0x7fd83406cc20 2024-04-12T05:51:19.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.225+0000 7fd850769700 1 -- 172.21.15.32:0/2242211508 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd83406cc20 msgr2=0x7fd83406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:19.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.225+0000 7fd850769700 1 --2- 172.21.15.32:0/2242211508 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd83406cc20 0x7fd83406f0d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd844001250 tx=0x7fd84400b040 comp rx=0 tx=0).stop 2024-04-12T05:51:19.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.225+0000 7fd850769700 1 -- 172.21.15.32:0/2242211508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8480fe7b0 msgr2=0x7fd8481142c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:19.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.225+0000 7fd850769700 1 --2- 172.21.15.32:0/2242211508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8480fe7b0 0x7fd8481142c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd838009380 tx=0x7fd838009e30 comp rx=0 tx=0).stop 2024-04-12T05:51:19.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.225+0000 7fd850769700 1 -- 172.21.15.32:0/2242211508 shutdown_connections 2024-04-12T05:51:19.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.225+0000 7fd850769700 1 --2- 172.21.15.32:0/2242211508 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd83406cc20 0x7fd83406f0d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:19.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.225+0000 7fd850769700 1 --2- 172.21.15.32:0/2242211508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8480fde10 0x7fd848113d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:19.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.225+0000 7fd850769700 1 --2- 172.21.15.32:0/2242211508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8480fe7b0 0x7fd8481142c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:19.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.225+0000 7fd850769700 1 -- 172.21.15.32:0/2242211508 >> 172.21.15.32:0/2242211508 conn(0x7fd8480f7880 msgr2=0x7fd8480f8ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:19.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.225+0000 7fd850769700 1 -- 172.21.15.32:0/2242211508 shutdown_connections 2024-04-12T05:51:19.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.225+0000 7fd850769700 1 -- 172.21.15.32:0/2242211508 wait complete. 2024-04-12T05:51:19.230 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:51:19.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.329+0000 7f6cd0766700 1 -- 172.21.15.32:0/2671531323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cc8102e00 msgr2=0x7f6cc81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:19.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.329+0000 7f6cd0766700 1 --2- 172.21.15.32:0/2671531323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cc8102e00 0x7f6cc81031d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4004940 tx=0x7f6cc4009e30 comp rx=0 tx=0).stop 2024-04-12T05:51:19.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.329+0000 7f6cd0766700 1 -- 172.21.15.32:0/2671531323 shutdown_connections 2024-04-12T05:51:19.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.329+0000 7f6cd0766700 1 --2- 172.21.15.32:0/2671531323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cc81037a0 0x7f6cc80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:19.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.329+0000 7f6cd0766700 1 --2- 172.21.15.32:0/2671531323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cc8102e00 0x7f6cc81031d0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:19.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.329+0000 7f6cd0766700 1 -- 172.21.15.32:0/2671531323 >> 172.21.15.32:0/2671531323 conn(0x7f6cc80f89b0 msgr2=0x7f6cc80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:19.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.329+0000 7f6cd0766700 1 -- 172.21.15.32:0/2671531323 shutdown_connections 2024-04-12T05:51:19.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.329+0000 7f6cd0766700 1 -- 172.21.15.32:0/2671531323 wait complete. 2024-04-12T05:51:19.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.330+0000 7f6cd0766700 1 Processor -- start 2024-04-12T05:51:19.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.330+0000 7f6cd0766700 1 -- start start 2024-04-12T05:51:19.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.330+0000 7f6cd0766700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cc8102e00 0x7f6cc80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:19.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.330+0000 7f6cd0766700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cc81037a0 0x7f6cc8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:19.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.330+0000 7f6cd0766700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc8072210 con 0x7f6cc81037a0 2024-04-12T05:51:19.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.330+0000 7f6cd0766700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc8070d40 con 0x7f6cc8102e00 2024-04-12T05:51:19.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.330+0000 7f6ccdd01700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cc81037a0 0x7f6cc8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:19.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.330+0000 7f6ccdd01700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cc81037a0 0x7f6cc8070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42284/0 (socket says 172.21.15.32:42284) 2024-04-12T05:51:19.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.330+0000 7f6cce502700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cc8102e00 0x7f6cc80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:19.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.330+0000 7f6ccdd01700 1 -- 172.21.15.32:0/4251702438 learned_addr learned my addr 172.21.15.32:0/4251702438 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:51:19.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.331+0000 7f6ccdd01700 1 -- 172.21.15.32:0/4251702438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cc8102e00 msgr2=0x7f6cc80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:19.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.331+0000 7f6ccdd01700 1 --2- 172.21.15.32:0/4251702438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cc8102e00 0x7f6cc80702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:19.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.331+0000 7f6ccdd01700 1 -- 172.21.15.32:0/4251702438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc4005040 con 0x7f6cc81037a0 2024-04-12T05:51:19.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.331+0000 7f6ccdd01700 1 --2- 172.21.15.32:0/4251702438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cc81037a0 0x7f6cc8070800 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6cb8002a50 tx=0x7f6cb8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:19.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.331+0000 7f6cbeffd700 1 -- 172.21.15.32:0/4251702438 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6cb8007cc0 con 0x7f6cc81037a0 2024-04-12T05:51:19.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.331+0000 7f6cd0766700 1 -- 172.21.15.32:0/4251702438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cc8071020 con 0x7f6cc81037a0 2024-04-12T05:51:19.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.331+0000 7f6cbeffd700 1 -- 172.21.15.32:0/4251702438 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6cb800f040 con 0x7f6cc81037a0 2024-04-12T05:51:19.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.331+0000 7f6cbeffd700 1 -- 172.21.15.32:0/4251702438 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6cb800d920 con 0x7f6cc81037a0 2024-04-12T05:51:19.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.331+0000 7f6cd0766700 1 -- 172.21.15.32:0/4251702438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cc811a060 con 0x7f6cc81037a0 2024-04-12T05:51:19.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.332+0000 7f6cd0766700 1 -- 172.21.15.32:0/4251702438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cc80fe550 con 0x7f6cc81037a0 2024-04-12T05:51:19.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.333+0000 7f6cbeffd700 1 -- 172.21.15.32:0/4251702438 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f6cb8009680 con 0x7f6cc81037a0 2024-04-12T05:51:19.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.333+0000 7f6cbeffd700 1 --2- 172.21.15.32:0/4251702438 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6cb406cc20 0x7f6cb406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:19.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.334+0000 7f6cbeffd700 1 -- 172.21.15.32:0/4251702438 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f6cb808c410 con 0x7f6cc81037a0 2024-04-12T05:51:19.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.334+0000 7f6cce502700 1 --2- 172.21.15.32:0/4251702438 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6cb406cc20 0x7f6cb406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:19.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.334+0000 7f6cce502700 1 --2- 172.21.15.32:0/4251702438 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6cb406cc20 0x7f6cb406f0d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4004f90 tx=0x7f6cc4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:19.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.337+0000 7f6cbeffd700 1 -- 172.21.15.32:0/4251702438 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6cb8056410 con 0x7f6cc81037a0 2024-04-12T05:51:19.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.557+0000 7f6cd0766700 1 -- 172.21.15.32:0/4251702438 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6cc80fa180 con 0x7f6cb406cc20 2024-04-12T05:51:19.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.558+0000 7f6cbeffd700 1 -- 172.21.15.32:0/4251702438 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6cc80fa180 con 0x7f6cb406cc20 2024-04-12T05:51:19.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.560+0000 7f6cd0766700 1 -- 172.21.15.32:0/4251702438 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6cb406cc20 msgr2=0x7f6cb406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:19.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.560+0000 7f6cd0766700 1 --2- 172.21.15.32:0/4251702438 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6cb406cc20 0x7f6cb406f0d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4004f90 tx=0x7f6cc4009e30 comp rx=0 tx=0).stop 2024-04-12T05:51:19.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.560+0000 7f6cd0766700 1 -- 172.21.15.32:0/4251702438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cc81037a0 msgr2=0x7f6cc8070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:19.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.560+0000 7f6cd0766700 1 --2- 172.21.15.32:0/4251702438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cc81037a0 0x7f6cc8070800 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6cb8002a50 tx=0x7f6cb8004600 comp rx=0 tx=0).stop 2024-04-12T05:51:19.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.560+0000 7f6cd0766700 1 -- 172.21.15.32:0/4251702438 shutdown_connections 2024-04-12T05:51:19.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.560+0000 7f6cd0766700 1 --2- 172.21.15.32:0/4251702438 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6cb406cc20 0x7f6cb406f0d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:19.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.560+0000 7f6cd0766700 1 --2- 172.21.15.32:0/4251702438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cc8102e00 0x7f6cc80702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:19.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.560+0000 7f6cd0766700 1 --2- 172.21.15.32:0/4251702438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cc81037a0 0x7f6cc8070800 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:19.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.560+0000 7f6cd0766700 1 -- 172.21.15.32:0/4251702438 >> 172.21.15.32:0/4251702438 conn(0x7f6cc80f89b0 msgr2=0x7f6cc80f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:19.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.560+0000 7f6cd0766700 1 -- 172.21.15.32:0/4251702438 shutdown_connections 2024-04-12T05:51:19.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.560+0000 7f6cd0766700 1 -- 172.21.15.32:0/4251702438 wait complete. 2024-04-12T05:51:19.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.663+0000 7f05695f3700 1 -- 172.21.15.32:0/3055026614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0564102710 msgr2=0x7f05640fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:19.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.663+0000 7f05695f3700 1 --2- 172.21.15.32:0/3055026614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0564102710 0x7f05640fbd80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f054c004940 tx=0x7f054c009e30 comp rx=0 tx=0).stop 2024-04-12T05:51:19.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.663+0000 7f05695f3700 1 -- 172.21.15.32:0/3055026614 shutdown_connections 2024-04-12T05:51:19.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.663+0000 7f05695f3700 1 --2- 172.21.15.32:0/3055026614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0564102710 0x7f05640fbd80 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:19.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.663+0000 7f05695f3700 1 --2- 172.21.15.32:0/3055026614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0564101d70 0x7f0564102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:19.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.663+0000 7f05695f3700 1 -- 172.21.15.32:0/3055026614 >> 172.21.15.32:0/3055026614 conn(0x7f0564072bc0 msgr2=0x7f0564072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:19.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.663+0000 7f05695f3700 1 -- 172.21.15.32:0/3055026614 shutdown_connections 2024-04-12T05:51:19.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.663+0000 7f05695f3700 1 -- 172.21.15.32:0/3055026614 wait complete. 2024-04-12T05:51:19.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.664+0000 7f05695f3700 1 Processor -- start 2024-04-12T05:51:19.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.664+0000 7f05695f3700 1 -- start start 2024-04-12T05:51:19.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.664+0000 7f05695f3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0564101d70 0x7f0564113aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:19.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.664+0000 7f05695f3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0564102710 0x7f0564113fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:19.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.664+0000 7f05695f3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05641146c0 con 0x7f0564102710 2024-04-12T05:51:19.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.664+0000 7f05695f3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056410da20 con 0x7f0564101d70 2024-04-12T05:51:19.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.664+0000 7f05627fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0564102710 0x7f0564113fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:19.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.664+0000 7f05627fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0564102710 0x7f0564113fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42298/0 (socket says 172.21.15.32:42298) 2024-04-12T05:51:19.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.664+0000 7f05627fc700 1 -- 172.21.15.32:0/3547195402 learned_addr learned my addr 172.21.15.32:0/3547195402 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:51:19.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.665+0000 7f0562ffd700 1 --2- 172.21.15.32:0/3547195402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0564101d70 0x7f0564113aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:19.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.665+0000 7f05627fc700 1 -- 172.21.15.32:0/3547195402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0564101d70 msgr2=0x7f0564113aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:19.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.665+0000 7f05627fc700 1 --2- 172.21.15.32:0/3547195402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0564101d70 0x7f0564113aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:19.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.665+0000 7f05627fc700 1 -- 172.21.15.32:0/3547195402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054c005040 con 0x7f0564102710 2024-04-12T05:51:19.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.665+0000 7f05627fc700 1 --2- 172.21.15.32:0/3547195402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0564102710 0x7f0564113fe0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f054c009390 tx=0x7f054c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:19.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.665+0000 7f055b7fe700 1 -- 172.21.15.32:0/3547195402 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f054c00e040 con 0x7f0564102710 2024-04-12T05:51:19.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.665+0000 7f05695f3700 1 -- 172.21.15.32:0/3547195402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f056410dca0 con 0x7f0564102710 2024-04-12T05:51:19.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.665+0000 7f055b7fe700 1 -- 172.21.15.32:0/3547195402 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f054c013420 con 0x7f0564102710 2024-04-12T05:51:19.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.665+0000 7f055b7fe700 1 -- 172.21.15.32:0/3547195402 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f054c01c630 con 0x7f0564102710 2024-04-12T05:51:19.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.665+0000 7f05695f3700 1 -- 172.21.15.32:0/3547195402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f056410e260 con 0x7f0564102710 2024-04-12T05:51:19.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.666+0000 7f05695f3700 1 -- 172.21.15.32:0/3547195402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05640fd4c0 con 0x7f0564102710 2024-04-12T05:51:19.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.666+0000 7f055b7fe700 1 -- 172.21.15.32:0/3547195402 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f054c0139e0 con 0x7f0564102710 2024-04-12T05:51:19.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.667+0000 7f055b7fe700 1 --2- 172.21.15.32:0/3547195402 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f055006cc30 0x7f055006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:19.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.667+0000 7f055b7fe700 1 -- 172.21.15.32:0/3547195402 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f054c08d350 con 0x7f0564102710 2024-04-12T05:51:19.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.667+0000 7f0562ffd700 1 --2- 172.21.15.32:0/3547195402 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f055006cc30 0x7f055006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:19.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.667+0000 7f0562ffd700 1 --2- 172.21.15.32:0/3547195402 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f055006cc30 0x7f055006f0e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0554006fd0 tx=0x7f0554008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:19.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.671+0000 7f055b7fe700 1 -- 172.21.15.32:0/3547195402 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f054c05adc0 con 0x7f0564102710 2024-04-12T05:51:19.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.883+0000 7f05695f3700 1 -- 172.21.15.32:0/3547195402 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f05640666a0 con 0x7f055006cc30 2024-04-12T05:51:19.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.892+0000 7f055b7fe700 1 -- 172.21.15.32:0/3547195402 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f05640666a0 con 0x7f055006cc30 2024-04-12T05:51:19.887 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:51:19.888 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (11m) 4m ago 22m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:51:19.888 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (8m) 4m ago 22m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:51:19.888 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (8m) 4m ago 21m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:51:19.888 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (10m) 4m ago 22m 87.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:51:19.888 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (5m) 4m ago 16m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T05:51:19.888 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (4m) 4m ago 16m 13.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T05:51:19.888 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (4m) 4m ago 16m 11.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 6d4aca6f74a0 2024-04-12T05:51:19.888 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (4m) 4m ago 16m 14.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T05:51:19.888 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (12m) 4m ago 24m 507M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:51:19.888 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (12m) 4m ago 21m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:51:19.888 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (9m) 4m ago 24m 53.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:51:19.888 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (9m) 4m ago 21m 43.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:51:19.888 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (11m) 4m ago 22m 16.9M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:51:19.888 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (11m) 4m ago 21m 18.4M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:51:19.889 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (8m) 4m ago 20m 91.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T05:51:19.889 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (7m) 4m ago 20m 97.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T05:51:19.889 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (7m) 4m ago 19m 81.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T05:51:19.889 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (6m) 4m ago 19m 113M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T05:51:19.889 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (5m) 4m ago 18m 90.3M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T05:51:19.889 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (5m) 4m ago 18m 84.8M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T05:51:19.889 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (11m) 4m ago 22m 64.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:51:19.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.895+0000 7f05695f3700 1 -- 172.21.15.32:0/3547195402 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f055006cc30 msgr2=0x7f055006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:19.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.895+0000 7f05695f3700 1 --2- 172.21.15.32:0/3547195402 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f055006cc30 0x7f055006f0e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0554006fd0 tx=0x7f0554008040 comp rx=0 tx=0).stop 2024-04-12T05:51:19.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.895+0000 7f05695f3700 1 -- 172.21.15.32:0/3547195402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0564102710 msgr2=0x7f0564113fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:19.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.895+0000 7f05695f3700 1 --2- 172.21.15.32:0/3547195402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0564102710 0x7f0564113fe0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f054c009390 tx=0x7f054c009e30 comp rx=0 tx=0).stop 2024-04-12T05:51:19.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.895+0000 7f05695f3700 1 -- 172.21.15.32:0/3547195402 shutdown_connections 2024-04-12T05:51:19.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.895+0000 7f05695f3700 1 --2- 172.21.15.32:0/3547195402 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f055006cc30 0x7f055006f0e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:19.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.895+0000 7f05695f3700 1 --2- 172.21.15.32:0/3547195402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0564101d70 0x7f0564113aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:19.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.895+0000 7f05695f3700 1 --2- 172.21.15.32:0/3547195402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0564102710 0x7f0564113fe0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:19.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.895+0000 7f05695f3700 1 -- 172.21.15.32:0/3547195402 >> 172.21.15.32:0/3547195402 conn(0x7f0564072bc0 msgr2=0x7f05641057d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:19.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.895+0000 7f05695f3700 1 -- 172.21.15.32:0/3547195402 shutdown_connections 2024-04-12T05:51:19.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.895+0000 7f05695f3700 1 -- 172.21.15.32:0/3547195402 wait complete. 2024-04-12T05:51:19.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.998+0000 7fe99c0cf700 1 -- 172.21.15.32:0/1273548522 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe994102e10 msgr2=0x7fe9941031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:19.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.998+0000 7fe99c0cf700 1 --2- 172.21.15.32:0/1273548522 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe994102e10 0x7fe9941031e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe990009ce0 tx=0x7fe990004d10 comp rx=0 tx=0).stop 2024-04-12T05:51:19.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.998+0000 7fe99c0cf700 1 -- 172.21.15.32:0/1273548522 shutdown_connections 2024-04-12T05:51:19.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.998+0000 7fe99c0cf700 1 --2- 172.21.15.32:0/1273548522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9941037b0 0x7fe9940fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:19.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.998+0000 7fe99c0cf700 1 --2- 172.21.15.32:0/1273548522 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe994102e10 0x7fe9941031e0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:19.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.998+0000 7fe99c0cf700 1 -- 172.21.15.32:0/1273548522 >> 172.21.15.32:0/1273548522 conn(0x7fe9940f8980 msgr2=0x7fe9940fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:19.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.998+0000 7fe99c0cf700 1 -- 172.21.15.32:0/1273548522 shutdown_connections 2024-04-12T05:51:19.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.998+0000 7fe99c0cf700 1 -- 172.21.15.32:0/1273548522 wait complete. 2024-04-12T05:51:19.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.998+0000 7fe99c0cf700 1 Processor -- start 2024-04-12T05:51:19.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.998+0000 7fe99c0cf700 1 -- start start 2024-04-12T05:51:19.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.999+0000 7fe99c0cf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe994102e10 0x7fe994113cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:19.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.999+0000 7fe99c0cf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9941037b0 0x7fe9941141f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:19.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.999+0000 7fe99c0cf700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe994114840 con 0x7fe9941037b0 2024-04-12T05:51:19.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.999+0000 7fe99c0cf700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe99410dc10 con 0x7fe994102e10 2024-04-12T05:51:19.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.999+0000 7fe999e6b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe994102e10 0x7fe994113cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:19.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.999+0000 7fe99966a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9941037b0 0x7fe9941141f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:19.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.999+0000 7fe999e6b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe994102e10 0x7fe994113cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.32:38826/0 (socket says 172.21.15.32:38826) 2024-04-12T05:51:19.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.999+0000 7fe999e6b700 1 -- 172.21.15.32:0/1687079734 learned_addr learned my addr 172.21.15.32:0/1687079734 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:51:19.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.999+0000 7fe99966a700 1 -- 172.21.15.32:0/1687079734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe994102e10 msgr2=0x7fe994113cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:19.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.999+0000 7fe99966a700 1 --2- 172.21.15.32:0/1687079734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe994102e10 0x7fe994113cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:19.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:19.999+0000 7fe99966a700 1 -- 172.21.15.32:0/1687079734 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe990009940 con 0x7fe9941037b0 2024-04-12T05:51:19.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.000+0000 7fe99966a700 1 --2- 172.21.15.32:0/1687079734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9941037b0 0x7fe9941141f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe984004fd0 tx=0x7fe984004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:19.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.000+0000 7fe98a7fc700 1 -- 172.21.15.32:0/1687079734 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe984007cc0 con 0x7fe9941037b0 2024-04-12T05:51:19.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.000+0000 7fe98a7fc700 1 -- 172.21.15.32:0/1687079734 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe98400f040 con 0x7fe9941037b0 2024-04-12T05:51:19.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.000+0000 7fe99c0cf700 1 -- 172.21.15.32:0/1687079734 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe99410def0 con 0x7fe9941037b0 2024-04-12T05:51:19.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.000+0000 7fe98a7fc700 1 -- 172.21.15.32:0/1687079734 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9840137b0 con 0x7fe9941037b0 2024-04-12T05:51:19.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.000+0000 7fe99c0cf700 1 -- 172.21.15.32:0/1687079734 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe99410e4b0 con 0x7fe9941037b0 2024-04-12T05:51:19.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.001+0000 7fe99c0cf700 1 -- 172.21.15.32:0/1687079734 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9940fe560 con 0x7fe9941037b0 2024-04-12T05:51:19.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.001+0000 7fe98a7fc700 1 -- 172.21.15.32:0/1687079734 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe98401c400 con 0x7fe9941037b0 2024-04-12T05:51:19.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.002+0000 7fe98a7fc700 1 --2- 172.21.15.32:0/1687079734 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe98006cc30 0x7fe98006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:19.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.002+0000 7fe98a7fc700 1 -- 172.21.15.32:0/1687079734 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fe98408be00 con 0x7fe9941037b0 2024-04-12T05:51:19.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.002+0000 7fe999e6b700 1 --2- 172.21.15.32:0/1687079734 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe98006cc30 0x7fe98006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:19.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.002+0000 7fe999e6b700 1 --2- 172.21.15.32:0/1687079734 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe98006cc30 0x7fe98006f0e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe9900048e0 tx=0x7fe990004d10 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:20.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.006+0000 7fe98a7fc700 1 -- 172.21.15.32:0/1687079734 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe984059750 con 0x7fe9941037b0 2024-04-12T05:51:20.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:19 smithi067 ceph-mon[112979]: pgmap v324: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:20.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:19 smithi032 ceph-mon[146817]: pgmap v324: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:20.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.294+0000 7fe99c0cf700 1 -- 172.21.15.32:0/1687079734 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe99404bed0 con 0x7fe9941037b0 2024-04-12T05:51:20.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.294+0000 7fe98a7fc700 1 -- 172.21.15.32:0/1687079734 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fe9840592e0 con 0x7fe9941037b0 2024-04-12T05:51:20.289 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:51:20.290 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:51:20.290 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:51:20.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:51:20.290 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:51:20.290 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:51:20.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:51:20.290 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:51:20.290 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:51:20.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:51:20.290 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:51:20.290 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:51:20.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:51:20.290 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:51:20.291 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:51:20.291 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:51:20.291 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:51:20.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.296+0000 7fe99c0cf700 1 -- 172.21.15.32:0/1687079734 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe98006cc30 msgr2=0x7fe98006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:20.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.296+0000 7fe99c0cf700 1 --2- 172.21.15.32:0/1687079734 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe98006cc30 0x7fe98006f0e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe9900048e0 tx=0x7fe990004d10 comp rx=0 tx=0).stop 2024-04-12T05:51:20.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.296+0000 7fe99c0cf700 1 -- 172.21.15.32:0/1687079734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9941037b0 msgr2=0x7fe9941141f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:20.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.296+0000 7fe99c0cf700 1 --2- 172.21.15.32:0/1687079734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9941037b0 0x7fe9941141f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe984004fd0 tx=0x7fe984004570 comp rx=0 tx=0).stop 2024-04-12T05:51:20.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.297+0000 7fe99c0cf700 1 -- 172.21.15.32:0/1687079734 shutdown_connections 2024-04-12T05:51:20.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.297+0000 7fe99c0cf700 1 --2- 172.21.15.32:0/1687079734 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe98006cc30 0x7fe98006f0e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:20.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.297+0000 7fe99c0cf700 1 --2- 172.21.15.32:0/1687079734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe994102e10 0x7fe994113cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:20.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.297+0000 7fe99c0cf700 1 --2- 172.21.15.32:0/1687079734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9941037b0 0x7fe9941141f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:20.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.297+0000 7fe99c0cf700 1 -- 172.21.15.32:0/1687079734 >> 172.21.15.32:0/1687079734 conn(0x7fe9940f8980 msgr2=0x7fe9940f9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:20.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.297+0000 7fe99c0cf700 1 -- 172.21.15.32:0/1687079734 shutdown_connections 2024-04-12T05:51:20.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.297+0000 7fe99c0cf700 1 -- 172.21.15.32:0/1687079734 wait complete. 2024-04-12T05:51:20.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.399+0000 7f50a9243700 1 -- 172.21.15.32:0/2519234886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50a4070c10 msgr2=0x7f50a4109180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:20.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.399+0000 7f50a9243700 1 --2- 172.21.15.32:0/2519234886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50a4070c10 0x7f50a4109180 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5094004970 tx=0x7f5094009e30 comp rx=0 tx=0).stop 2024-04-12T05:51:20.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.400+0000 7f50a9243700 1 -- 172.21.15.32:0/2519234886 shutdown_connections 2024-04-12T05:51:20.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.400+0000 7f50a9243700 1 --2- 172.21.15.32:0/2519234886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50a4070c10 0x7f50a4109180 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:20.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.400+0000 7f50a9243700 1 --2- 172.21.15.32:0/2519234886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a4070300 0x7f50a40706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:20.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.400+0000 7f50a9243700 1 -- 172.21.15.32:0/2519234886 >> 172.21.15.32:0/2519234886 conn(0x7f50a40f89a0 msgr2=0x7f50a40fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:20.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.400+0000 7f50a9243700 1 -- 172.21.15.32:0/2519234886 shutdown_connections 2024-04-12T05:51:20.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.400+0000 7f50a9243700 1 -- 172.21.15.32:0/2519234886 wait complete. 2024-04-12T05:51:20.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.400+0000 7f50a9243700 1 Processor -- start 2024-04-12T05:51:20.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.400+0000 7f50a9243700 1 -- start start 2024-04-12T05:51:20.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.401+0000 7f50a9243700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50a4070300 0x7f50a4115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:20.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.401+0000 7f50a9243700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a4070c10 0x7f50a41161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:20.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.401+0000 7f50a9243700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a41167f0 con 0x7f50a4070300 2024-04-12T05:51:20.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.401+0000 7f50a9243700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a410fbc0 con 0x7f50a4070c10 2024-04-12T05:51:20.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.401+0000 7f50a2d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50a4070300 0x7f50a4115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:20.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.401+0000 7f50a2d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50a4070300 0x7f50a4115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42332/0 (socket says 172.21.15.32:42332) 2024-04-12T05:51:20.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.401+0000 7f50a2d9d700 1 -- 172.21.15.32:0/879049371 learned_addr learned my addr 172.21.15.32:0/879049371 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:51:20.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.401+0000 7f50a259c700 1 --2- 172.21.15.32:0/879049371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a4070c10 0x7f50a41161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:20.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.401+0000 7f50a2d9d700 1 -- 172.21.15.32:0/879049371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a4070c10 msgr2=0x7f50a41161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:20.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.401+0000 7f50a2d9d700 1 --2- 172.21.15.32:0/879049371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a4070c10 0x7f50a41161a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:20.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.401+0000 7f50a2d9d700 1 -- 172.21.15.32:0/879049371 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5094005040 con 0x7f50a4070300 2024-04-12T05:51:20.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.401+0000 7f50a2d9d700 1 --2- 172.21.15.32:0/879049371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50a4070300 0x7f50a4115c60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5098002a50 tx=0x7f50980046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:20.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.401+0000 7f508b7fe700 1 -- 172.21.15.32:0/879049371 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5098007cc0 con 0x7f50a4070300 2024-04-12T05:51:20.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.401+0000 7f508b7fe700 1 -- 172.21.15.32:0/879049371 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f509800f040 con 0x7f50a4070300 2024-04-12T05:51:20.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.401+0000 7f50a9243700 1 -- 172.21.15.32:0/879049371 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50a410fea0 con 0x7f50a4070300 2024-04-12T05:51:20.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.401+0000 7f508b7fe700 1 -- 172.21.15.32:0/879049371 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f509800d9c0 con 0x7f50a4070300 2024-04-12T05:51:20.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.401+0000 7f50a9243700 1 -- 172.21.15.32:0/879049371 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50a41103b0 con 0x7f50a4070300 2024-04-12T05:51:20.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.402+0000 7f50a9243700 1 -- 172.21.15.32:0/879049371 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50a4071a40 con 0x7f50a4070300 2024-04-12T05:51:20.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.403+0000 7f508b7fe700 1 -- 172.21.15.32:0/879049371 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f509800db60 con 0x7f50a4070300 2024-04-12T05:51:20.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.403+0000 7f508b7fe700 1 --2- 172.21.15.32:0/879049371 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f508c06cc30 0x7f508c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:20.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.403+0000 7f50a259c700 1 --2- 172.21.15.32:0/879049371 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f508c06cc30 0x7f508c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:20.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.403+0000 7f508b7fe700 1 -- 172.21.15.32:0/879049371 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f509808b750 con 0x7f50a4070300 2024-04-12T05:51:20.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.404+0000 7f50a259c700 1 --2- 172.21.15.32:0/879049371 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f508c06cc30 0x7f508c06f0e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5094009fd0 tx=0x7f5094009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:20.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.408+0000 7f508b7fe700 1 -- 172.21.15.32:0/879049371 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5098059140 con 0x7f50a4070300 2024-04-12T05:51:20.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.653+0000 7f50a9243700 1 -- 172.21.15.32:0/879049371 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f50a404bf20 con 0x7f50a4070300 2024-04-12T05:51:20.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.654+0000 7f508b7fe700 1 -- 172.21.15.32:0/879049371 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1835 (secure 0 0 0) 0x7f5098058cd0 con 0x7f50a4070300 2024-04-12T05:51:20.649 INFO:teuthology.orchestra.run.smithi032.stdout:e33 2024-04-12T05:51:20.649 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:51:20.649 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:51:20.649 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:51:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:51:20.649 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:51:20.649 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:51:20.649 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 24 2024-04-12T05:51:20.649 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:51:20.649 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:51:20.649 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:46:10.668569+0000 2024-04-12T05:51:20.649 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:51:20.650 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:51:20.650 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:51:20.650 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:51:20.650 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:51:20.650 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:51:20.650 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:51:20.650 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:51:20.650 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T05:51:20.650 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:51:20.650 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:51:20.650 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:51:20.650 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T05:51:20.650 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-12T05:51:20.651 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:51:20.651 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:51:20.651 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:51:20.651 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:51:20.651 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:51:20.651 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:51:20.651 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:51:20.651 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:51:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:51:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:51:20.651 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-12T05:51:20.651 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:51:20.651 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:51:20.651 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:51:20.651 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{-1:44317} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:51:20.652 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:51:20.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.656+0000 7f50a9243700 1 -- 172.21.15.32:0/879049371 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f508c06cc30 msgr2=0x7f508c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:20.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.656+0000 7f50a9243700 1 --2- 172.21.15.32:0/879049371 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f508c06cc30 0x7f508c06f0e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5094009fd0 tx=0x7f5094009e30 comp rx=0 tx=0).stop 2024-04-12T05:51:20.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.656+0000 7f50a9243700 1 -- 172.21.15.32:0/879049371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50a4070300 msgr2=0x7f50a4115c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:20.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.656+0000 7f50a9243700 1 --2- 172.21.15.32:0/879049371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50a4070300 0x7f50a4115c60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5098002a50 tx=0x7f50980046b0 comp rx=0 tx=0).stop 2024-04-12T05:51:20.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.656+0000 7f50a9243700 1 -- 172.21.15.32:0/879049371 shutdown_connections 2024-04-12T05:51:20.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.656+0000 7f50a9243700 1 --2- 172.21.15.32:0/879049371 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f508c06cc30 0x7f508c06f0e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:20.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.656+0000 7f50a9243700 1 --2- 172.21.15.32:0/879049371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50a4070300 0x7f50a4115c60 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:20.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.656+0000 7f50a9243700 1 --2- 172.21.15.32:0/879049371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a4070c10 0x7f50a41161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:20.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.656+0000 7f50a9243700 1 -- 172.21.15.32:0/879049371 >> 172.21.15.32:0/879049371 conn(0x7f50a40f89a0 msgr2=0x7f50a40f9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:20.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.656+0000 7f50a9243700 1 -- 172.21.15.32:0/879049371 shutdown_connections 2024-04-12T05:51:20.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.656+0000 7f50a9243700 1 -- 172.21.15.32:0/879049371 wait complete. 2024-04-12T05:51:20.653 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-12T05:51:20.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.761+0000 7f985dbc6700 1 -- 172.21.15.32:0/408474146 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98580fe830 msgr2=0x7f9858106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:20.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.761+0000 7f985dbc6700 1 --2- 172.21.15.32:0/408474146 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98580fe830 0x7f9858106f00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f984c004940 tx=0x7f984c009e30 comp rx=0 tx=0).stop 2024-04-12T05:51:20.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.761+0000 7f985dbc6700 1 -- 172.21.15.32:0/408474146 shutdown_connections 2024-04-12T05:51:20.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.761+0000 7f985dbc6700 1 --2- 172.21.15.32:0/408474146 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98580fe830 0x7f9858106f00 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:20.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.761+0000 7f985dbc6700 1 --2- 172.21.15.32:0/408474146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98580fde90 0x7f98580fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:20.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.761+0000 7f985dbc6700 1 -- 172.21.15.32:0/408474146 >> 172.21.15.32:0/408474146 conn(0x7f9858072bc0 msgr2=0x7f9858072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:20.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.761+0000 7f985dbc6700 1 -- 172.21.15.32:0/408474146 shutdown_connections 2024-04-12T05:51:20.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.761+0000 7f985dbc6700 1 -- 172.21.15.32:0/408474146 wait complete. 2024-04-12T05:51:20.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.762+0000 7f985dbc6700 1 Processor -- start 2024-04-12T05:51:20.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.762+0000 7f985dbc6700 1 -- start start 2024-04-12T05:51:20.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.762+0000 7f985dbc6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98580fde90 0x7f98580fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:20.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.762+0000 7f985dbc6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98580fe830 0x7f98580fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:20.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.762+0000 7f985dbc6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98580fc820 con 0x7f98580fe830 2024-04-12T05:51:20.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.762+0000 7f985dbc6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98580fc960 con 0x7f98580fde90 2024-04-12T05:51:20.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.762+0000 7f9856ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98580fe830 0x7f98580fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:20.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.762+0000 7f9856ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98580fe830 0x7f98580fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42346/0 (socket says 172.21.15.32:42346) 2024-04-12T05:51:20.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.762+0000 7f9856ffd700 1 -- 172.21.15.32:0/1924624554 learned_addr learned my addr 172.21.15.32:0/1924624554 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:51:20.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.762+0000 7f98577fe700 1 --2- 172.21.15.32:0/1924624554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98580fde90 0x7f98580fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:20.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.763+0000 7f9856ffd700 1 -- 172.21.15.32:0/1924624554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98580fde90 msgr2=0x7f98580fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:20.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.763+0000 7f9856ffd700 1 --2- 172.21.15.32:0/1924624554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98580fde90 0x7f98580fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:20.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.763+0000 7f9856ffd700 1 -- 172.21.15.32:0/1924624554 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f984c005040 con 0x7f98580fe830 2024-04-12T05:51:20.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.763+0000 7f9856ffd700 1 --2- 172.21.15.32:0/1924624554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98580fe830 0x7f98580fc290 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f984c00b3b0 tx=0x7f984c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:20.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.763+0000 7f983ffff700 1 -- 172.21.15.32:0/1924624554 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f984c00e040 con 0x7f98580fe830 2024-04-12T05:51:20.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.763+0000 7f985dbc6700 1 -- 172.21.15.32:0/1924624554 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98580fcbb0 con 0x7f98580fe830 2024-04-12T05:51:20.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.763+0000 7f983ffff700 1 -- 172.21.15.32:0/1924624554 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f984c013420 con 0x7f98580fe830 2024-04-12T05:51:20.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.763+0000 7f983ffff700 1 -- 172.21.15.32:0/1924624554 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f984c01c5f0 con 0x7f98580fe830 2024-04-12T05:51:20.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.763+0000 7f985dbc6700 1 -- 172.21.15.32:0/1924624554 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f985811a190 con 0x7f98580fe830 2024-04-12T05:51:20.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.764+0000 7f985dbc6700 1 -- 172.21.15.32:0/1924624554 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9858065e30 con 0x7f98580fe830 2024-04-12T05:51:20.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.764+0000 7f983ffff700 1 -- 172.21.15.32:0/1924624554 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f984c013590 con 0x7f98580fe830 2024-04-12T05:51:20.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.765+0000 7f983ffff700 1 --2- 172.21.15.32:0/1924624554 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f984006cc20 0x7f984006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:20.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.765+0000 7f983ffff700 1 -- 172.21.15.32:0/1924624554 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f984c08d2e0 con 0x7f98580fe830 2024-04-12T05:51:20.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.765+0000 7f98577fe700 1 --2- 172.21.15.32:0/1924624554 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f984006cc20 0x7f984006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:20.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.766+0000 7f98577fe700 1 --2- 172.21.15.32:0/1924624554 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f984006cc20 0x7f984006f0d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9848004f40 tx=0x7f984800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:20.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.770+0000 7f983ffff700 1 -- 172.21.15.32:0/1924624554 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f984c057360 con 0x7f98580fe830 2024-04-12T05:51:20.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.991+0000 7f985dbc6700 1 -- 172.21.15.32:0/1924624554 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9858109e70 con 0x7f984006cc20 2024-04-12T05:51:20.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.993+0000 7f983ffff700 1 -- 172.21.15.32:0/1924624554 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f984800f040 con 0x7f984006cc20 2024-04-12T05:51:20.988 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:51:20.988 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:51:20.988 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:51:20.988 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:51:20.988 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:51:20.988 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-12T05:51:20.988 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:51:20.988 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-12T05:51:20.988 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:51:20.988 INFO:teuthology.orchestra.run.smithi032.stdout: "mds" 2024-04-12T05:51:20.988 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:51:20.988 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:51:20.988 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:51:20.988 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:51:20.989 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:51:20.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.995+0000 7f985dbc6700 1 -- 172.21.15.32:0/1924624554 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f984006cc20 msgr2=0x7f984006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:20.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.995+0000 7f985dbc6700 1 --2- 172.21.15.32:0/1924624554 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f984006cc20 0x7f984006f0d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9848004f40 tx=0x7f984800b040 comp rx=0 tx=0).stop 2024-04-12T05:51:20.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.995+0000 7f985dbc6700 1 -- 172.21.15.32:0/1924624554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98580fe830 msgr2=0x7f98580fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:20.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.995+0000 7f985dbc6700 1 --2- 172.21.15.32:0/1924624554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98580fe830 0x7f98580fc290 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f984c00b3b0 tx=0x7f984c009e30 comp rx=0 tx=0).stop 2024-04-12T05:51:20.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.995+0000 7f985dbc6700 1 -- 172.21.15.32:0/1924624554 shutdown_connections 2024-04-12T05:51:20.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.995+0000 7f985dbc6700 1 --2- 172.21.15.32:0/1924624554 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f984006cc20 0x7f984006f0d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:20.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.995+0000 7f985dbc6700 1 --2- 172.21.15.32:0/1924624554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98580fde90 0x7f98580fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:20.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.995+0000 7f985dbc6700 1 --2- 172.21.15.32:0/1924624554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98580fe830 0x7f98580fc290 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:20.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.995+0000 7f985dbc6700 1 -- 172.21.15.32:0/1924624554 >> 172.21.15.32:0/1924624554 conn(0x7f9858072bc0 msgr2=0x7f9858103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:20.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.995+0000 7f985dbc6700 1 -- 172.21.15.32:0/1924624554 shutdown_connections 2024-04-12T05:51:20.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:20.995+0000 7f985dbc6700 1 -- 172.21.15.32:0/1924624554 wait complete. 2024-04-12T05:51:21.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:20 smithi067 ceph-mon[112979]: from='client.34648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:51:21.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:20 smithi067 ceph-mon[112979]: from='client.34652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:51:21.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:20 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1687079734' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:51:21.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:20 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/879049371' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:51:21.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:20 smithi032 ceph-mon[146817]: from='client.34648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:51:21.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:20 smithi032 ceph-mon[146817]: from='client.34652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:51:21.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:20 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1687079734' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:51:21.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:20 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/879049371' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:51:21.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.098+0000 7faa4d970700 1 -- 172.21.15.32:0/83484881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa480fde90 msgr2=0x7faa480fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:21.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.098+0000 7faa4d970700 1 --2- 172.21.15.32:0/83484881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa480fde90 0x7faa480fe260 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7faa3c004940 tx=0x7faa3c009e30 comp rx=0 tx=0).stop 2024-04-12T05:51:21.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.098+0000 7faa4d970700 1 -- 172.21.15.32:0/83484881 shutdown_connections 2024-04-12T05:51:21.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.098+0000 7faa4d970700 1 --2- 172.21.15.32:0/83484881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa480fe830 0x7faa48106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:21.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.098+0000 7faa4d970700 1 --2- 172.21.15.32:0/83484881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa480fde90 0x7faa480fe260 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:21.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.098+0000 7faa4d970700 1 -- 172.21.15.32:0/83484881 >> 172.21.15.32:0/83484881 conn(0x7faa48072bc0 msgr2=0x7faa48072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:21.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.098+0000 7faa4d970700 1 -- 172.21.15.32:0/83484881 shutdown_connections 2024-04-12T05:51:21.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.098+0000 7faa4d970700 1 -- 172.21.15.32:0/83484881 wait complete. 2024-04-12T05:51:21.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.099+0000 7faa4d970700 1 Processor -- start 2024-04-12T05:51:21.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.099+0000 7faa4d970700 1 -- start start 2024-04-12T05:51:21.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.099+0000 7faa4d970700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa480fde90 0x7faa48115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:21.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.099+0000 7faa4d970700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa480fe830 0x7faa48116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:21.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.099+0000 7faa4d970700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa481168b0 con 0x7faa480fde90 2024-04-12T05:51:21.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.099+0000 7faa4d970700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa4810fc80 con 0x7faa480fe830 2024-04-12T05:51:21.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.099+0000 7faa46ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa480fde90 0x7faa48115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:21.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.099+0000 7faa46ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa480fde90 0x7faa48115d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42366/0 (socket says 172.21.15.32:42366) 2024-04-12T05:51:21.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.099+0000 7faa46ffd700 1 -- 172.21.15.32:0/3925480930 learned_addr learned my addr 172.21.15.32:0/3925480930 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:51:21.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.099+0000 7faa46ffd700 1 -- 172.21.15.32:0/3925480930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa480fe830 msgr2=0x7faa48116260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:21.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.099+0000 7faa46ffd700 1 --2- 172.21.15.32:0/3925480930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa480fe830 0x7faa48116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:21.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.099+0000 7faa46ffd700 1 -- 172.21.15.32:0/3925480930 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa3c005040 con 0x7faa480fde90 2024-04-12T05:51:21.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.099+0000 7faa46ffd700 1 --2- 172.21.15.32:0/3925480930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa480fde90 0x7faa48115d20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7faa3c009360 tx=0x7faa3c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:21.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.100+0000 7faa2f7fe700 1 -- 172.21.15.32:0/3925480930 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa3c00e040 con 0x7faa480fde90 2024-04-12T05:51:21.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.100+0000 7faa4d970700 1 -- 172.21.15.32:0/3925480930 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa4810ff00 con 0x7faa480fde90 2024-04-12T05:51:21.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.100+0000 7faa2f7fe700 1 -- 172.21.15.32:0/3925480930 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa3c013420 con 0x7faa480fde90 2024-04-12T05:51:21.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.100+0000 7faa2f7fe700 1 -- 172.21.15.32:0/3925480930 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa3c01c630 con 0x7faa480fde90 2024-04-12T05:51:21.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.100+0000 7faa4d970700 1 -- 172.21.15.32:0/3925480930 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa48110440 con 0x7faa480fde90 2024-04-12T05:51:21.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.101+0000 7faa4d970700 1 -- 172.21.15.32:0/3925480930 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa48106a20 con 0x7faa480fde90 2024-04-12T05:51:21.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.101+0000 7faa2f7fe700 1 -- 172.21.15.32:0/3925480930 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7faa3c0139e0 con 0x7faa480fde90 2024-04-12T05:51:21.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.102+0000 7faa2f7fe700 1 --2- 172.21.15.32:0/3925480930 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7faa3006cc20 0x7faa3006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:21.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.102+0000 7faa2f7fe700 1 -- 172.21.15.32:0/3925480930 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7faa3c08d320 con 0x7faa480fde90 2024-04-12T05:51:21.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.102+0000 7faa467fc700 1 --2- 172.21.15.32:0/3925480930 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7faa3006cc20 0x7faa3006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:21.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.102+0000 7faa467fc700 1 --2- 172.21.15.32:0/3925480930 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7faa3006cc20 0x7faa3006f0d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7faa481115c0 tx=0x7faa38007cc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:21.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.106+0000 7faa2f7fe700 1 -- 172.21.15.32:0/3925480930 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7faa3c05ad90 con 0x7faa480fde90 2024-04-12T05:51:21.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.391+0000 7faa4d970700 1 -- 172.21.15.32:0/3925480930 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7faa48109dc0 con 0x7faa480fde90 2024-04-12T05:51:21.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.391+0000 7faa2f7fe700 1 -- 172.21.15.32:0/3925480930 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7faa3c05a920 con 0x7faa480fde90 2024-04-12T05:51:21.387 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:51:21.387 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:51:21.387 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-12T05:51:21.387 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:51:21.387 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:51:21.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.393+0000 7faa4d970700 1 -- 172.21.15.32:0/3925480930 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7faa3006cc20 msgr2=0x7faa3006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:21.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.393+0000 7faa4d970700 1 --2- 172.21.15.32:0/3925480930 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7faa3006cc20 0x7faa3006f0d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7faa481115c0 tx=0x7faa38007cc0 comp rx=0 tx=0).stop 2024-04-12T05:51:21.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.393+0000 7faa4d970700 1 -- 172.21.15.32:0/3925480930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa480fde90 msgr2=0x7faa48115d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:21.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.393+0000 7faa4d970700 1 --2- 172.21.15.32:0/3925480930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa480fde90 0x7faa48115d20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7faa3c009360 tx=0x7faa3c009e30 comp rx=0 tx=0).stop 2024-04-12T05:51:21.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.394+0000 7faa4d970700 1 -- 172.21.15.32:0/3925480930 shutdown_connections 2024-04-12T05:51:21.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.394+0000 7faa4d970700 1 --2- 172.21.15.32:0/3925480930 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7faa3006cc20 0x7faa3006f0d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:21.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.394+0000 7faa4d970700 1 --2- 172.21.15.32:0/3925480930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa480fde90 0x7faa48115d20 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:21.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.394+0000 7faa4d970700 1 --2- 172.21.15.32:0/3925480930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa480fe830 0x7faa48116260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:21.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.394+0000 7faa4d970700 1 -- 172.21.15.32:0/3925480930 >> 172.21.15.32:0/3925480930 conn(0x7faa48072bc0 msgr2=0x7faa480fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:21.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.394+0000 7faa4d970700 1 -- 172.21.15.32:0/3925480930 shutdown_connections 2024-04-12T05:51:21.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:21.394+0000 7faa4d970700 1 -- 172.21.15.32:0/3925480930 wait complete. 2024-04-12T05:51:22.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:21 smithi067 ceph-mon[112979]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:51:22.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:21 smithi067 ceph-mon[112979]: pgmap v325: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:22.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:21 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3925480930' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:51:22.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:21 smithi032 ceph-mon[146817]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:51:22.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:21 smithi032 ceph-mon[146817]: pgmap v325: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:22.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:21 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3925480930' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:51:23.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:22 smithi067 ceph-mon[112979]: from='client.34668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:51:23.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:22 smithi032 ceph-mon[146817]: from='client.34668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:51:24.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:23 smithi067 ceph-mon[112979]: pgmap v326: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:24.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:23 smithi032 ceph-mon[146817]: pgmap v326: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:26.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:25 smithi067 ceph-mon[112979]: pgmap v327: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:26.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:25 smithi032 ceph-mon[146817]: pgmap v327: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:28.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:27 smithi067 ceph-mon[112979]: pgmap v328: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:28.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:27 smithi032 ceph-mon[146817]: pgmap v328: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:30.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:29 smithi067 ceph-mon[112979]: pgmap v329: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:30.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:29 smithi032 ceph-mon[146817]: pgmap v329: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:32.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:31 smithi067 ceph-mon[112979]: pgmap v330: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:32.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:31 smithi032 ceph-mon[146817]: pgmap v330: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:34.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:33 smithi067 ceph-mon[112979]: pgmap v331: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:34.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:33 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:51:34.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:33 smithi032 ceph-mon[146817]: pgmap v331: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:34.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:33 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:51:36.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:35 smithi067 ceph-mon[112979]: pgmap v332: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:36.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:35 smithi032 ceph-mon[146817]: pgmap v332: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:38.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:37 smithi067 ceph-mon[112979]: pgmap v333: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:38.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:37 smithi032 ceph-mon[146817]: pgmap v333: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:40.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:39 smithi067 ceph-mon[112979]: pgmap v334: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:40.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:39 smithi032 ceph-mon[146817]: pgmap v334: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:42.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:41 smithi067 ceph-mon[112979]: pgmap v335: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:42.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:41 smithi032 ceph-mon[146817]: pgmap v335: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:44.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:43 smithi067 ceph-mon[112979]: pgmap v336: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:44.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:43 smithi032 ceph-mon[146817]: pgmap v336: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:46.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:45 smithi067 ceph-mon[112979]: pgmap v337: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:46.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:45 smithi032 ceph-mon[146817]: pgmap v337: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:48.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:47 smithi067 ceph-mon[112979]: pgmap v338: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:48.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:47 smithi032 ceph-mon[146817]: pgmap v338: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:49.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:48 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:51:49.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:48 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:51:50.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:49 smithi067 ceph-mon[112979]: pgmap v339: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:50.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:49 smithi032 ceph-mon[146817]: pgmap v339: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:51.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.511+0000 7faad1e42700 1 -- 172.21.15.32:0/2737211430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faacc102710 msgr2=0x7faacc0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:51.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.511+0000 7faad1e42700 1 --2- 172.21.15.32:0/2737211430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faacc102710 0x7faacc0fbd80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7faabc004940 tx=0x7faabc009e30 comp rx=0 tx=0).stop 2024-04-12T05:51:51.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.511+0000 7faad1e42700 1 -- 172.21.15.32:0/2737211430 shutdown_connections 2024-04-12T05:51:51.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.511+0000 7faad1e42700 1 --2- 172.21.15.32:0/2737211430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faacc102710 0x7faacc0fbd80 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:51.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.511+0000 7faad1e42700 1 --2- 172.21.15.32:0/2737211430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faacc101d70 0x7faacc102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:51.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.511+0000 7faad1e42700 1 -- 172.21.15.32:0/2737211430 >> 172.21.15.32:0/2737211430 conn(0x7faacc072bc0 msgr2=0x7faacc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:51.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.511+0000 7faad1e42700 1 -- 172.21.15.32:0/2737211430 shutdown_connections 2024-04-12T05:51:51.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.511+0000 7faad1e42700 1 -- 172.21.15.32:0/2737211430 wait complete. 2024-04-12T05:51:51.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.511+0000 7faad1e42700 1 Processor -- start 2024-04-12T05:51:51.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.512+0000 7faad1e42700 1 -- start start 2024-04-12T05:51:51.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.512+0000 7faad1e42700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faacc101d70 0x7faacc10ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:51.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.512+0000 7faad1e42700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faacc102710 0x7faacc10ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:51.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.512+0000 7faad1e42700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faacc10f5b0 con 0x7faacc101d70 2024-04-12T05:51:51.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.512+0000 7faad1e42700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faacc10f6f0 con 0x7faacc102710 2024-04-12T05:51:51.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.512+0000 7faacb7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faacc101d70 0x7faacc10ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:51.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.512+0000 7faacb7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faacc101d70 0x7faacc10ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51342/0 (socket says 172.21.15.32:51342) 2024-04-12T05:51:51.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.512+0000 7faacb7fe700 1 -- 172.21.15.32:0/460123450 learned_addr learned my addr 172.21.15.32:0/460123450 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:51:51.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.512+0000 7faacaffd700 1 --2- 172.21.15.32:0/460123450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faacc102710 0x7faacc10ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:51.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.512+0000 7faacb7fe700 1 -- 172.21.15.32:0/460123450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faacc102710 msgr2=0x7faacc10ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:51.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.512+0000 7faacb7fe700 1 --2- 172.21.15.32:0/460123450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faacc102710 0x7faacc10ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:51.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.512+0000 7faacb7fe700 1 -- 172.21.15.32:0/460123450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faabc005040 con 0x7faacc101d70 2024-04-12T05:51:51.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.512+0000 7faacb7fe700 1 --2- 172.21.15.32:0/460123450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faacc101d70 0x7faacc10ea20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7faac000eb70 tx=0x7faac000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:51.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.513+0000 7faab3fff700 1 -- 172.21.15.32:0/460123450 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faac0007cc0 con 0x7faacc101d70 2024-04-12T05:51:51.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.513+0000 7faab3fff700 1 -- 172.21.15.32:0/460123450 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faac000f040 con 0x7faacc101d70 2024-04-12T05:51:51.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.513+0000 7faad1e42700 1 -- 172.21.15.32:0/460123450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faacc117a90 con 0x7faacc101d70 2024-04-12T05:51:51.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.513+0000 7faab3fff700 1 -- 172.21.15.32:0/460123450 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faac000c9f0 con 0x7faacc101d70 2024-04-12T05:51:51.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.513+0000 7faad1e42700 1 -- 172.21.15.32:0/460123450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faacc117f70 con 0x7faacc101d70 2024-04-12T05:51:51.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.514+0000 7faad1e42700 1 -- 172.21.15.32:0/460123450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faacc104240 con 0x7faacc101d70 2024-04-12T05:51:51.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.514+0000 7faab3fff700 1 -- 172.21.15.32:0/460123450 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7faac000cb50 con 0x7faacc101d70 2024-04-12T05:51:51.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.515+0000 7faab3fff700 1 --2- 172.21.15.32:0/460123450 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7faab406cc20 0x7faab406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:51.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.515+0000 7faab3fff700 1 -- 172.21.15.32:0/460123450 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7faac008bd40 con 0x7faacc101d70 2024-04-12T05:51:51.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.515+0000 7faacaffd700 1 --2- 172.21.15.32:0/460123450 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7faab406cc20 0x7faab406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:51.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.515+0000 7faacaffd700 1 --2- 172.21.15.32:0/460123450 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7faab406cc20 0x7faab406f0d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7faabc00b0d0 tx=0x7faabc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:51.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.520+0000 7faab3fff700 1 -- 172.21.15.32:0/460123450 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7faac0055c00 con 0x7faacc101d70 2024-04-12T05:51:51.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.743+0000 7faad1e42700 1 -- 172.21.15.32:0/460123450 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7faacc04bed0 con 0x7faab406cc20 2024-04-12T05:51:51.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.745+0000 7faab3fff700 1 -- 172.21.15.32:0/460123450 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7faacc04bed0 con 0x7faab406cc20 2024-04-12T05:51:51.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.747+0000 7faad1e42700 1 -- 172.21.15.32:0/460123450 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7faab406cc20 msgr2=0x7faab406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:51.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.747+0000 7faad1e42700 1 --2- 172.21.15.32:0/460123450 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7faab406cc20 0x7faab406f0d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7faabc00b0d0 tx=0x7faabc009e30 comp rx=0 tx=0).stop 2024-04-12T05:51:51.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.747+0000 7faad1e42700 1 -- 172.21.15.32:0/460123450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faacc101d70 msgr2=0x7faacc10ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:51.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.747+0000 7faad1e42700 1 --2- 172.21.15.32:0/460123450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faacc101d70 0x7faacc10ea20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7faac000eb70 tx=0x7faac000b040 comp rx=0 tx=0).stop 2024-04-12T05:51:51.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.747+0000 7faad1e42700 1 -- 172.21.15.32:0/460123450 shutdown_connections 2024-04-12T05:51:51.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.747+0000 7faad1e42700 1 --2- 172.21.15.32:0/460123450 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7faab406cc20 0x7faab406f0d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:51.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.747+0000 7faad1e42700 1 --2- 172.21.15.32:0/460123450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faacc101d70 0x7faacc10ea20 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:51.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.747+0000 7faad1e42700 1 --2- 172.21.15.32:0/460123450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faacc102710 0x7faacc10ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:51.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.747+0000 7faad1e42700 1 -- 172.21.15.32:0/460123450 >> 172.21.15.32:0/460123450 conn(0x7faacc072bc0 msgr2=0x7faacc0fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:51.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.747+0000 7faad1e42700 1 -- 172.21.15.32:0/460123450 shutdown_connections 2024-04-12T05:51:51.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.747+0000 7faad1e42700 1 -- 172.21.15.32:0/460123450 wait complete. 2024-04-12T05:51:51.753 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:51:51.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.857+0000 7fa43fc39700 1 -- 172.21.15.32:0/1052902053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4380fe830 msgr2=0x7fa438106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:51.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.857+0000 7fa43fc39700 1 --2- 172.21.15.32:0/1052902053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4380fe830 0x7fa438106f00 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa434004940 tx=0x7fa434009e30 comp rx=0 tx=0).stop 2024-04-12T05:51:51.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.857+0000 7fa43fc39700 1 -- 172.21.15.32:0/1052902053 shutdown_connections 2024-04-12T05:51:51.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.857+0000 7fa43fc39700 1 --2- 172.21.15.32:0/1052902053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4380fe830 0x7fa438106f00 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:51.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.857+0000 7fa43fc39700 1 --2- 172.21.15.32:0/1052902053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4380fde90 0x7fa4380fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:51.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.857+0000 7fa43fc39700 1 -- 172.21.15.32:0/1052902053 >> 172.21.15.32:0/1052902053 conn(0x7fa438072bc0 msgr2=0x7fa438072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:51.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.857+0000 7fa43fc39700 1 -- 172.21.15.32:0/1052902053 shutdown_connections 2024-04-12T05:51:51.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.857+0000 7fa43fc39700 1 -- 172.21.15.32:0/1052902053 wait complete. 2024-04-12T05:51:51.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.858+0000 7fa43fc39700 1 Processor -- start 2024-04-12T05:51:51.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.858+0000 7fa43fc39700 1 -- start start 2024-04-12T05:51:51.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.858+0000 7fa43fc39700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4380fde90 0x7fa43811a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:51.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.858+0000 7fa43fc39700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4380fe830 0x7fa43811a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:51.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.858+0000 7fa43fc39700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa43811aca0 con 0x7fa4380fde90 2024-04-12T05:51:51.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.858+0000 7fa43fc39700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa438114070 con 0x7fa4380fe830 2024-04-12T05:51:51.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.858+0000 7fa43d9d5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4380fde90 0x7fa43811a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:51.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.858+0000 7fa43d9d5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4380fde90 0x7fa43811a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51356/0 (socket says 172.21.15.32:51356) 2024-04-12T05:51:51.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.858+0000 7fa43d9d5700 1 -- 172.21.15.32:0/3618978986 learned_addr learned my addr 172.21.15.32:0/3618978986 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:51:51.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.859+0000 7fa43d1d4700 1 --2- 172.21.15.32:0/3618978986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4380fe830 0x7fa43811a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:51.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.859+0000 7fa43d9d5700 1 -- 172.21.15.32:0/3618978986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4380fe830 msgr2=0x7fa43811a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:51.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.859+0000 7fa43d9d5700 1 --2- 172.21.15.32:0/3618978986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4380fe830 0x7fa43811a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:51.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.859+0000 7fa43d9d5700 1 -- 172.21.15.32:0/3618978986 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa434005040 con 0x7fa4380fde90 2024-04-12T05:51:51.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.859+0000 7fa43d1d4700 1 --2- 172.21.15.32:0/3618978986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4380fe830 0x7fa43811a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:51:51.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.859+0000 7fa43d9d5700 1 --2- 172.21.15.32:0/3618978986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4380fde90 0x7fa43811a110 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa428002a50 tx=0x7fa4280046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:51.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.859+0000 7fa42e7fc700 1 -- 172.21.15.32:0/3618978986 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa428007cc0 con 0x7fa4380fde90 2024-04-12T05:51:51.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.859+0000 7fa42e7fc700 1 -- 172.21.15.32:0/3618978986 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa42800f040 con 0x7fa4380fde90 2024-04-12T05:51:51.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.859+0000 7fa43fc39700 1 -- 172.21.15.32:0/3618978986 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa438114350 con 0x7fa4380fde90 2024-04-12T05:51:51.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.859+0000 7fa42e7fc700 1 -- 172.21.15.32:0/3618978986 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa428009540 con 0x7fa4380fde90 2024-04-12T05:51:51.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.859+0000 7fa43fc39700 1 -- 172.21.15.32:0/3618978986 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa438114890 con 0x7fa4380fde90 2024-04-12T05:51:51.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.860+0000 7fa43fc39700 1 -- 172.21.15.32:0/3618978986 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa438102610 con 0x7fa4380fde90 2024-04-12T05:51:51.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.861+0000 7fa42e7fc700 1 -- 172.21.15.32:0/3618978986 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa4280096a0 con 0x7fa4380fde90 2024-04-12T05:51:51.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.861+0000 7fa42e7fc700 1 --2- 172.21.15.32:0/3618978986 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa42406cc20 0x7fa42406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:51.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.861+0000 7fa42e7fc700 1 -- 172.21.15.32:0/3618978986 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fa42808bdd0 con 0x7fa4380fde90 2024-04-12T05:51:51.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.861+0000 7fa43d1d4700 1 --2- 172.21.15.32:0/3618978986 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa42406cc20 0x7fa42406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:51.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.862+0000 7fa43d1d4700 1 --2- 172.21.15.32:0/3618978986 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa42406cc20 0x7fa42406f0d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa43400b3b0 tx=0x7fa434009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:51.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:51.866+0000 7fa42e7fc700 1 -- 172.21.15.32:0/3618978986 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa428059840 con 0x7fa4380fde90 2024-04-12T05:51:52.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:51 smithi067 ceph-mon[112979]: pgmap v340: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:52.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.090+0000 7fa43fc39700 1 -- 172.21.15.32:0/3618978986 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa438109e40 con 0x7fa42406cc20 2024-04-12T05:51:52.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.091+0000 7fa42e7fc700 1 -- 172.21.15.32:0/3618978986 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa438109e40 con 0x7fa42406cc20 2024-04-12T05:51:52.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:51 smithi032 ceph-mon[146817]: pgmap v340: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:52.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.093+0000 7fa43fc39700 1 -- 172.21.15.32:0/3618978986 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa42406cc20 msgr2=0x7fa42406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:52.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.093+0000 7fa43fc39700 1 --2- 172.21.15.32:0/3618978986 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa42406cc20 0x7fa42406f0d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa43400b3b0 tx=0x7fa434009e30 comp rx=0 tx=0).stop 2024-04-12T05:51:52.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.093+0000 7fa43fc39700 1 -- 172.21.15.32:0/3618978986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4380fde90 msgr2=0x7fa43811a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:52.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.093+0000 7fa43fc39700 1 --2- 172.21.15.32:0/3618978986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4380fde90 0x7fa43811a110 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa428002a50 tx=0x7fa4280046b0 comp rx=0 tx=0).stop 2024-04-12T05:51:52.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.093+0000 7fa43fc39700 1 -- 172.21.15.32:0/3618978986 shutdown_connections 2024-04-12T05:51:52.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.093+0000 7fa43fc39700 1 --2- 172.21.15.32:0/3618978986 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa42406cc20 0x7fa42406f0d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:52.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.093+0000 7fa43fc39700 1 --2- 172.21.15.32:0/3618978986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4380fde90 0x7fa43811a110 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:52.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.093+0000 7fa43fc39700 1 --2- 172.21.15.32:0/3618978986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4380fe830 0x7fa43811a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:52.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.094+0000 7fa43fc39700 1 -- 172.21.15.32:0/3618978986 >> 172.21.15.32:0/3618978986 conn(0x7fa438072bc0 msgr2=0x7fa4380fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:52.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.094+0000 7fa43fc39700 1 -- 172.21.15.32:0/3618978986 shutdown_connections 2024-04-12T05:51:52.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.094+0000 7fa43fc39700 1 -- 172.21.15.32:0/3618978986 wait complete. 2024-04-12T05:51:52.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.198+0000 7f91f6fb2700 1 -- 172.21.15.32:0/3121347026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f0102e10 msgr2=0x7f91f01031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:52.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.198+0000 7f91f6fb2700 1 --2- 172.21.15.32:0/3121347026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f0102e10 0x7f91f01031e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f91e4004940 tx=0x7f91e4009e30 comp rx=0 tx=0).stop 2024-04-12T05:51:52.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.198+0000 7f91f6fb2700 1 -- 172.21.15.32:0/3121347026 shutdown_connections 2024-04-12T05:51:52.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.198+0000 7f91f6fb2700 1 --2- 172.21.15.32:0/3121347026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91f01037b0 0x7f91f00fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:52.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.198+0000 7f91f6fb2700 1 --2- 172.21.15.32:0/3121347026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f0102e10 0x7f91f01031e0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:52.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.198+0000 7f91f6fb2700 1 -- 172.21.15.32:0/3121347026 >> 172.21.15.32:0/3121347026 conn(0x7f91f00f89a0 msgr2=0x7f91f00fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:52.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.198+0000 7f91f6fb2700 1 -- 172.21.15.32:0/3121347026 shutdown_connections 2024-04-12T05:51:52.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.198+0000 7f91f6fb2700 1 -- 172.21.15.32:0/3121347026 wait complete. 2024-04-12T05:51:52.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.199+0000 7f91f6fb2700 1 Processor -- start 2024-04-12T05:51:52.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.199+0000 7f91f6fb2700 1 -- start start 2024-04-12T05:51:52.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.199+0000 7f91f6fb2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f0102e10 0x7f91f0115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:52.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.199+0000 7f91f6fb2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91f01037b0 0x7f91f0116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:52.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.199+0000 7f91f6fb2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f01168a0 con 0x7f91f0102e10 2024-04-12T05:51:52.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.199+0000 7f91f6fb2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f010fc70 con 0x7f91f01037b0 2024-04-12T05:51:52.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.199+0000 7f91f4d4e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f0102e10 0x7f91f0115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:52.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.199+0000 7f91f4d4e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f0102e10 0x7f91f0115d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51374/0 (socket says 172.21.15.32:51374) 2024-04-12T05:51:52.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.199+0000 7f91f4d4e700 1 -- 172.21.15.32:0/4287285303 learned_addr learned my addr 172.21.15.32:0/4287285303 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:51:52.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.200+0000 7f91effff700 1 --2- 172.21.15.32:0/4287285303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91f01037b0 0x7f91f0116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:52.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.200+0000 7f91f4d4e700 1 -- 172.21.15.32:0/4287285303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91f01037b0 msgr2=0x7f91f0116250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:52.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.200+0000 7f91f4d4e700 1 --2- 172.21.15.32:0/4287285303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91f01037b0 0x7f91f0116250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:52.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.200+0000 7f91f4d4e700 1 -- 172.21.15.32:0/4287285303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e4005040 con 0x7f91f0102e10 2024-04-12T05:51:52.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.200+0000 7f91f4d4e700 1 --2- 172.21.15.32:0/4287285303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f0102e10 0x7f91f0115d10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f91e4009180 tx=0x7f91e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:52.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.200+0000 7f91ed7fa700 1 -- 172.21.15.32:0/4287285303 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91e400e040 con 0x7f91f0102e10 2024-04-12T05:51:52.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.200+0000 7f91f6fb2700 1 -- 172.21.15.32:0/4287285303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f010fef0 con 0x7f91f0102e10 2024-04-12T05:51:52.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.200+0000 7f91ed7fa700 1 -- 172.21.15.32:0/4287285303 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f91e4004290 con 0x7f91f0102e10 2024-04-12T05:51:52.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.200+0000 7f91ed7fa700 1 -- 172.21.15.32:0/4287285303 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91e401c3f0 con 0x7f91f0102e10 2024-04-12T05:51:52.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.200+0000 7f91f6fb2700 1 -- 172.21.15.32:0/4287285303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91f01104b0 con 0x7f91f0102e10 2024-04-12T05:51:52.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.201+0000 7f91f6fb2700 1 -- 172.21.15.32:0/4287285303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91f00fe560 con 0x7f91f0102e10 2024-04-12T05:51:52.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.201+0000 7f91ed7fa700 1 -- 172.21.15.32:0/4287285303 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f91e4013790 con 0x7f91f0102e10 2024-04-12T05:51:52.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.202+0000 7f91ed7fa700 1 --2- 172.21.15.32:0/4287285303 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f91e006cc20 0x7f91e006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:52.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.202+0000 7f91ed7fa700 1 -- 172.21.15.32:0/4287285303 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f91e408d360 con 0x7f91f0102e10 2024-04-12T05:51:52.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.202+0000 7f91effff700 1 --2- 172.21.15.32:0/4287285303 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f91e006cc20 0x7f91e006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:52.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.202+0000 7f91effff700 1 --2- 172.21.15.32:0/4287285303 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f91e006cc20 0x7f91e006f0d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f91dc007fd0 tx=0x7f91dc009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:52.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.206+0000 7f91ed7fa700 1 -- 172.21.15.32:0/4287285303 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f91e405add0 con 0x7f91f0102e10 2024-04-12T05:51:52.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.422+0000 7f91f6fb2700 1 -- 172.21.15.32:0/4287285303 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f91f004bf40 con 0x7f91e006cc20 2024-04-12T05:51:52.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.432+0000 7f91ed7fa700 1 -- 172.21.15.32:0/4287285303 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f91f004bf40 con 0x7f91e006cc20 2024-04-12T05:51:52.428 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:51:52.428 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (11m) 5m ago 23m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:51:52.428 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (9m) 5m ago 23m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:51:52.428 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (9m) 4m ago 22m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:51:52.428 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (11m) 5m ago 22m 87.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:51:52.428 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (5m) 5m ago 17m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T05:51:52.428 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (5m) 5m ago 17m 13.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T05:51:52.428 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (4m) 4m ago 17m 11.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 6d4aca6f74a0 2024-04-12T05:51:52.428 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (4m) 4m ago 17m 14.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T05:51:52.429 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (13m) 5m ago 24m 507M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:51:52.429 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (12m) 4m ago 22m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:51:52.429 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (10m) 5m ago 24m 53.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:51:52.429 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (9m) 4m ago 22m 43.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:51:52.429 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (12m) 5m ago 23m 16.9M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:51:52.429 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (12m) 4m ago 22m 18.4M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:51:52.429 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (8m) 5m ago 21m 91.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T05:51:52.429 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (8m) 5m ago 20m 97.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T05:51:52.429 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (7m) 5m ago 20m 81.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T05:51:52.429 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (6m) 4m ago 19m 113M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T05:51:52.429 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (6m) 4m ago 19m 90.3M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T05:51:52.429 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (5m) 4m ago 18m 84.8M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T05:51:52.429 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (11m) 5m ago 22m 64.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:51:52.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.434+0000 7f91f6fb2700 1 -- 172.21.15.32:0/4287285303 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f91e006cc20 msgr2=0x7f91e006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:52.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.434+0000 7f91f6fb2700 1 --2- 172.21.15.32:0/4287285303 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f91e006cc20 0x7f91e006f0d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f91dc007fd0 tx=0x7f91dc009040 comp rx=0 tx=0).stop 2024-04-12T05:51:52.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.434+0000 7f91f6fb2700 1 -- 172.21.15.32:0/4287285303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f0102e10 msgr2=0x7f91f0115d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:52.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.434+0000 7f91f6fb2700 1 --2- 172.21.15.32:0/4287285303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f0102e10 0x7f91f0115d10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f91e4009180 tx=0x7f91e4009e30 comp rx=0 tx=0).stop 2024-04-12T05:51:52.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.434+0000 7f91f6fb2700 1 -- 172.21.15.32:0/4287285303 shutdown_connections 2024-04-12T05:51:52.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.434+0000 7f91f6fb2700 1 --2- 172.21.15.32:0/4287285303 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f91e006cc20 0x7f91e006f0d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:52.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.434+0000 7f91f6fb2700 1 --2- 172.21.15.32:0/4287285303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f0102e10 0x7f91f0115d10 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:52.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.434+0000 7f91f6fb2700 1 --2- 172.21.15.32:0/4287285303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91f01037b0 0x7f91f0116250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:52.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.434+0000 7f91f6fb2700 1 -- 172.21.15.32:0/4287285303 >> 172.21.15.32:0/4287285303 conn(0x7f91f00f89a0 msgr2=0x7f91f00f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:52.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.434+0000 7f91f6fb2700 1 -- 172.21.15.32:0/4287285303 shutdown_connections 2024-04-12T05:51:52.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.434+0000 7f91f6fb2700 1 -- 172.21.15.32:0/4287285303 wait complete. 2024-04-12T05:51:52.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.536+0000 7fe28f4b1700 1 -- 172.21.15.32:0/1909936542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe288102e10 msgr2=0x7fe2881031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:52.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.536+0000 7fe28f4b1700 1 --2- 172.21.15.32:0/1909936542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe288102e10 0x7fe2881031e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe27c004940 tx=0x7fe27c009e30 comp rx=0 tx=0).stop 2024-04-12T05:51:52.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.536+0000 7fe28f4b1700 1 -- 172.21.15.32:0/1909936542 shutdown_connections 2024-04-12T05:51:52.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.536+0000 7fe28f4b1700 1 --2- 172.21.15.32:0/1909936542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2881037b0 0x7fe2880fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:52.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.536+0000 7fe28f4b1700 1 --2- 172.21.15.32:0/1909936542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe288102e10 0x7fe2881031e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:52.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.536+0000 7fe28f4b1700 1 -- 172.21.15.32:0/1909936542 >> 172.21.15.32:0/1909936542 conn(0x7fe2880f89a0 msgr2=0x7fe2880fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:52.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.536+0000 7fe28f4b1700 1 -- 172.21.15.32:0/1909936542 shutdown_connections 2024-04-12T05:51:52.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.536+0000 7fe28f4b1700 1 -- 172.21.15.32:0/1909936542 wait complete. 2024-04-12T05:51:52.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.537+0000 7fe28f4b1700 1 Processor -- start 2024-04-12T05:51:52.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.537+0000 7fe28f4b1700 1 -- start start 2024-04-12T05:51:52.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.537+0000 7fe28f4b1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe288102e10 0x7fe2880702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:52.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.537+0000 7fe28f4b1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2881037b0 0x7fe288070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:52.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.537+0000 7fe28f4b1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe288070d90 con 0x7fe2881037b0 2024-04-12T05:51:52.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.537+0000 7fe28f4b1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe288070ed0 con 0x7fe288102e10 2024-04-12T05:51:52.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.537+0000 7fe28ca4c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2881037b0 0x7fe288070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:52.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.537+0000 7fe28ca4c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2881037b0 0x7fe288070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51396/0 (socket says 172.21.15.32:51396) 2024-04-12T05:51:52.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.537+0000 7fe28ca4c700 1 -- 172.21.15.32:0/1886175840 learned_addr learned my addr 172.21.15.32:0/1886175840 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:51:52.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.537+0000 7fe28d24d700 1 --2- 172.21.15.32:0/1886175840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe288102e10 0x7fe2880702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:52.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.537+0000 7fe28ca4c700 1 -- 172.21.15.32:0/1886175840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe288102e10 msgr2=0x7fe2880702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:52.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.537+0000 7fe28ca4c700 1 --2- 172.21.15.32:0/1886175840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe288102e10 0x7fe2880702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:52.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.537+0000 7fe28ca4c700 1 -- 172.21.15.32:0/1886175840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe27c005040 con 0x7fe2881037b0 2024-04-12T05:51:52.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.538+0000 7fe28ca4c700 1 --2- 172.21.15.32:0/1886175840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2881037b0 0x7fe288070800 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe284002a50 tx=0x7fe284004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:52.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.538+0000 7fe279ffb700 1 -- 172.21.15.32:0/1886175840 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe284007cc0 con 0x7fe2881037b0 2024-04-12T05:51:52.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.538+0000 7fe279ffb700 1 -- 172.21.15.32:0/1886175840 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe28400f040 con 0x7fe2881037b0 2024-04-12T05:51:52.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.538+0000 7fe28f4b1700 1 -- 172.21.15.32:0/1886175840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe288119df0 con 0x7fe2881037b0 2024-04-12T05:51:52.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.538+0000 7fe279ffb700 1 -- 172.21.15.32:0/1886175840 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe28400d990 con 0x7fe2881037b0 2024-04-12T05:51:52.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.538+0000 7fe28f4b1700 1 -- 172.21.15.32:0/1886175840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe28811a270 con 0x7fe2881037b0 2024-04-12T05:51:52.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.539+0000 7fe28f4b1700 1 -- 172.21.15.32:0/1886175840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2880fe560 con 0x7fe2881037b0 2024-04-12T05:51:52.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.539+0000 7fe279ffb700 1 -- 172.21.15.32:0/1886175840 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe284009710 con 0x7fe2881037b0 2024-04-12T05:51:52.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.540+0000 7fe279ffb700 1 --2- 172.21.15.32:0/1886175840 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe27406cc20 0x7fe27406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:52.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.540+0000 7fe279ffb700 1 -- 172.21.15.32:0/1886175840 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fe28408bec0 con 0x7fe2881037b0 2024-04-12T05:51:52.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.540+0000 7fe28d24d700 1 --2- 172.21.15.32:0/1886175840 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe27406cc20 0x7fe27406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:52.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.540+0000 7fe28d24d700 1 --2- 172.21.15.32:0/1886175840 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe27406cc20 0x7fe27406f0d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe27c004f90 tx=0x7fe27c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:52.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.544+0000 7fe279ffb700 1 -- 172.21.15.32:0/1886175840 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe2840598b0 con 0x7fe2881037b0 2024-04-12T05:51:52.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.834+0000 7fe28f4b1700 1 -- 172.21.15.32:0/1886175840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe28804bed0 con 0x7fe2881037b0 2024-04-12T05:51:52.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.834+0000 7fe279ffb700 1 -- 172.21.15.32:0/1886175840 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fe284059440 con 0x7fe2881037b0 2024-04-12T05:51:52.830 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:51:52.830 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:51:52.831 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:51:52.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:51:52.831 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:51:52.831 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:51:52.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:51:52.831 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:51:52.831 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:51:52.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:51:52.831 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:51:52.831 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:51:52.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:51:52.831 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:51:52.832 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:51:52.832 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:51:52.832 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:51:52.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.837+0000 7fe28f4b1700 1 -- 172.21.15.32:0/1886175840 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe27406cc20 msgr2=0x7fe27406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:52.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.837+0000 7fe28f4b1700 1 --2- 172.21.15.32:0/1886175840 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe27406cc20 0x7fe27406f0d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe27c004f90 tx=0x7fe27c009e30 comp rx=0 tx=0).stop 2024-04-12T05:51:52.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.837+0000 7fe28f4b1700 1 -- 172.21.15.32:0/1886175840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2881037b0 msgr2=0x7fe288070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:52.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.837+0000 7fe28f4b1700 1 --2- 172.21.15.32:0/1886175840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2881037b0 0x7fe288070800 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe284002a50 tx=0x7fe284004600 comp rx=0 tx=0).stop 2024-04-12T05:51:52.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.837+0000 7fe28f4b1700 1 -- 172.21.15.32:0/1886175840 shutdown_connections 2024-04-12T05:51:52.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.837+0000 7fe28f4b1700 1 --2- 172.21.15.32:0/1886175840 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe27406cc20 0x7fe27406f0d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:52.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.837+0000 7fe28f4b1700 1 --2- 172.21.15.32:0/1886175840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe288102e10 0x7fe2880702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:52.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.837+0000 7fe28f4b1700 1 --2- 172.21.15.32:0/1886175840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2881037b0 0x7fe288070800 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:52.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.837+0000 7fe28f4b1700 1 -- 172.21.15.32:0/1886175840 >> 172.21.15.32:0/1886175840 conn(0x7fe2880f89a0 msgr2=0x7fe2880f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:52.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.837+0000 7fe28f4b1700 1 -- 172.21.15.32:0/1886175840 shutdown_connections 2024-04-12T05:51:52.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.837+0000 7fe28f4b1700 1 -- 172.21.15.32:0/1886175840 wait complete. 2024-04-12T05:51:52.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.940+0000 7fda8c787700 1 -- 172.21.15.32:0/3222145948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda840fb4a0 msgr2=0x7fda840fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:52.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.940+0000 7fda8c787700 1 --2- 172.21.15.32:0/3222145948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda840fb4a0 0x7fda840fb910 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fda74004940 tx=0x7fda74009e30 comp rx=0 tx=0).stop 2024-04-12T05:51:52.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.940+0000 7fda8c787700 1 -- 172.21.15.32:0/3222145948 shutdown_connections 2024-04-12T05:51:52.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.940+0000 7fda8c787700 1 --2- 172.21.15.32:0/3222145948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda840fb4a0 0x7fda840fb910 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:52.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.940+0000 7fda8c787700 1 --2- 172.21.15.32:0/3222145948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda84102e70 0x7fda84103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:52.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.940+0000 7fda8c787700 1 -- 172.21.15.32:0/3222145948 >> 172.21.15.32:0/3222145948 conn(0x7fda840714c0 msgr2=0x7fda840718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:52.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.940+0000 7fda8c787700 1 -- 172.21.15.32:0/3222145948 shutdown_connections 2024-04-12T05:51:52.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.940+0000 7fda8c787700 1 -- 172.21.15.32:0/3222145948 wait complete. 2024-04-12T05:51:52.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.940+0000 7fda8c787700 1 Processor -- start 2024-04-12T05:51:52.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.941+0000 7fda8c787700 1 -- start start 2024-04-12T05:51:52.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.941+0000 7fda8c787700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda840fb4a0 0x7fda84113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:52.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.941+0000 7fda8c787700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda84102e70 0x7fda84113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:52.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.941+0000 7fda8c787700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda84114600 con 0x7fda84102e70 2024-04-12T05:51:52.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.941+0000 7fda8c787700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8410d9f0 con 0x7fda840fb4a0 2024-04-12T05:51:52.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.941+0000 7fda89d22700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda84102e70 0x7fda84113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:52.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.941+0000 7fda89d22700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda84102e70 0x7fda84113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51420/0 (socket says 172.21.15.32:51420) 2024-04-12T05:51:52.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.941+0000 7fda89d22700 1 -- 172.21.15.32:0/846828928 learned_addr learned my addr 172.21.15.32:0/846828928 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:51:52.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.941+0000 7fda8a523700 1 --2- 172.21.15.32:0/846828928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda840fb4a0 0x7fda84113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:52.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.941+0000 7fda89d22700 1 -- 172.21.15.32:0/846828928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda840fb4a0 msgr2=0x7fda84113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:52.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.941+0000 7fda89d22700 1 --2- 172.21.15.32:0/846828928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda840fb4a0 0x7fda84113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:52.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.941+0000 7fda89d22700 1 -- 172.21.15.32:0/846828928 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda74005040 con 0x7fda84102e70 2024-04-12T05:51:52.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.941+0000 7fda89d22700 1 --2- 172.21.15.32:0/846828928 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda84102e70 0x7fda84113fb0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fda7400b3b0 tx=0x7fda74009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:52.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.942+0000 7fda7affd700 1 -- 172.21.15.32:0/846828928 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda7400e040 con 0x7fda84102e70 2024-04-12T05:51:52.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.942+0000 7fda7affd700 1 -- 172.21.15.32:0/846828928 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda74013420 con 0x7fda84102e70 2024-04-12T05:51:52.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.942+0000 7fda8c787700 1 -- 172.21.15.32:0/846828928 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda8410dc70 con 0x7fda84102e70 2024-04-12T05:51:52.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.942+0000 7fda7affd700 1 -- 172.21.15.32:0/846828928 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda7401c610 con 0x7fda84102e70 2024-04-12T05:51:52.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.942+0000 7fda8c787700 1 -- 172.21.15.32:0/846828928 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda8410e230 con 0x7fda84102e70 2024-04-12T05:51:52.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.943+0000 7fda8c787700 1 -- 172.21.15.32:0/846828928 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda84100110 con 0x7fda84102e70 2024-04-12T05:51:52.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.943+0000 7fda7affd700 1 -- 172.21.15.32:0/846828928 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fda74013a00 con 0x7fda84102e70 2024-04-12T05:51:52.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.943+0000 7fda7affd700 1 --2- 172.21.15.32:0/846828928 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fda7006cc20 0x7fda7006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:52.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.944+0000 7fda7affd700 1 -- 172.21.15.32:0/846828928 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fda7408d220 con 0x7fda84102e70 2024-04-12T05:51:52.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.944+0000 7fda8a523700 1 --2- 172.21.15.32:0/846828928 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fda7006cc20 0x7fda7006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:52.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.944+0000 7fda8a523700 1 --2- 172.21.15.32:0/846828928 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fda7006cc20 0x7fda7006f0d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fda80004f40 tx=0x7fda8000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:52.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:52.948+0000 7fda7affd700 1 -- 172.21.15.32:0/846828928 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fda7405ac90 con 0x7fda84102e70 2024-04-12T05:51:53.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:52 smithi067 ceph-mon[112979]: from='client.34676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:51:53.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:52 smithi032 ceph-mon[146817]: from='client.34676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:51:53.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.193+0000 7fda8c787700 1 -- 172.21.15.32:0/846828928 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fda8410f0c0 con 0x7fda84102e70 2024-04-12T05:51:53.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.194+0000 7fda7affd700 1 -- 172.21.15.32:0/846828928 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1835 (secure 0 0 0) 0x7fda7405a820 con 0x7fda84102e70 2024-04-12T05:51:53.191 INFO:teuthology.orchestra.run.smithi032.stdout:e33 2024-04-12T05:51:53.191 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:51:53.191 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:51:53.191 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:51:53.191 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:51:53.192 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:51:53.192 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:51:53.192 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 24 2024-04-12T05:51:53.192 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:51:53.192 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:51:53.192 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:46:10.668569+0000 2024-04-12T05:51:53.192 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:51:53.192 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:51:53.192 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:51:53.192 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:51:53.192 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:51:53.192 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:51:53.192 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:51:53.192 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:51:53.192 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T05:51:53.193 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:51:53.193 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:51:53.193 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:51:53.193 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T05:51:53.193 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-12T05:51:53.193 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:51:53.193 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:51:53.193 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:51:53.193 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:51:53.193 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:51:53.193 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:51:53.193 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:51:53.193 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:51:53.193 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:51:53.193 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:51:53.194 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-12T05:51:53.194 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:51:53.194 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:51:53.194 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:51:53.194 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{-1:44317} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:51:53.195 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:51:53.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.196+0000 7fda8c787700 1 -- 172.21.15.32:0/846828928 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fda7006cc20 msgr2=0x7fda7006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:53.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.196+0000 7fda8c787700 1 --2- 172.21.15.32:0/846828928 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fda7006cc20 0x7fda7006f0d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fda80004f40 tx=0x7fda8000b040 comp rx=0 tx=0).stop 2024-04-12T05:51:53.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.196+0000 7fda8c787700 1 -- 172.21.15.32:0/846828928 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda84102e70 msgr2=0x7fda84113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:53.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.196+0000 7fda8c787700 1 --2- 172.21.15.32:0/846828928 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda84102e70 0x7fda84113fb0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fda7400b3b0 tx=0x7fda74009e30 comp rx=0 tx=0).stop 2024-04-12T05:51:53.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.196+0000 7fda8c787700 1 -- 172.21.15.32:0/846828928 shutdown_connections 2024-04-12T05:51:53.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.196+0000 7fda8c787700 1 --2- 172.21.15.32:0/846828928 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fda7006cc20 0x7fda7006f0d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:53.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.196+0000 7fda8c787700 1 --2- 172.21.15.32:0/846828928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda840fb4a0 0x7fda84113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:53.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.196+0000 7fda8c787700 1 --2- 172.21.15.32:0/846828928 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda84102e70 0x7fda84113fb0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:53.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.196+0000 7fda8c787700 1 -- 172.21.15.32:0/846828928 >> 172.21.15.32:0/846828928 conn(0x7fda840714c0 msgr2=0x7fda840fa910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:53.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.196+0000 7fda8c787700 1 -- 172.21.15.32:0/846828928 shutdown_connections 2024-04-12T05:51:53.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.196+0000 7fda8c787700 1 -- 172.21.15.32:0/846828928 wait complete. 2024-04-12T05:51:53.196 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-12T05:51:53.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.300+0000 7fd5e540f700 1 -- 172.21.15.32:0/3901120824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5e0102710 msgr2=0x7fd5e00fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:53.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.300+0000 7fd5e540f700 1 --2- 172.21.15.32:0/3901120824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5e0102710 0x7fd5e00fbd80 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0004940 tx=0x7fd5d0009e30 comp rx=0 tx=0).stop 2024-04-12T05:51:53.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.300+0000 7fd5e540f700 1 -- 172.21.15.32:0/3901120824 shutdown_connections 2024-04-12T05:51:53.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.300+0000 7fd5e540f700 1 --2- 172.21.15.32:0/3901120824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5e0102710 0x7fd5e00fbd80 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:53.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.300+0000 7fd5e540f700 1 --2- 172.21.15.32:0/3901120824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5e0101d70 0x7fd5e0102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:53.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.300+0000 7fd5e540f700 1 -- 172.21.15.32:0/3901120824 >> 172.21.15.32:0/3901120824 conn(0x7fd5e0072bc0 msgr2=0x7fd5e0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:53.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.300+0000 7fd5e540f700 1 -- 172.21.15.32:0/3901120824 shutdown_connections 2024-04-12T05:51:53.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.300+0000 7fd5e540f700 1 -- 172.21.15.32:0/3901120824 wait complete. 2024-04-12T05:51:53.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.301+0000 7fd5e540f700 1 Processor -- start 2024-04-12T05:51:53.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.301+0000 7fd5e540f700 1 -- start start 2024-04-12T05:51:53.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.301+0000 7fd5e540f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5e0101d70 0x7fd5e0115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:53.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.301+0000 7fd5e540f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5e0102710 0x7fd5e01161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:53.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.301+0000 7fd5e540f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e0116830 con 0x7fd5e0101d70 2024-04-12T05:51:53.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.301+0000 7fd5e540f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e010fc00 con 0x7fd5e0102710 2024-04-12T05:51:53.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.301+0000 7fd5deffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5e0101d70 0x7fd5e0115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:53.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.301+0000 7fd5deffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5e0101d70 0x7fd5e0115ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51438/0 (socket says 172.21.15.32:51438) 2024-04-12T05:51:53.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.301+0000 7fd5deffd700 1 -- 172.21.15.32:0/1875735809 learned_addr learned my addr 172.21.15.32:0/1875735809 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:51:53.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.301+0000 7fd5de7fc700 1 --2- 172.21.15.32:0/1875735809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5e0102710 0x7fd5e01161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:53.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.301+0000 7fd5deffd700 1 -- 172.21.15.32:0/1875735809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5e0102710 msgr2=0x7fd5e01161e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:53.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.301+0000 7fd5deffd700 1 --2- 172.21.15.32:0/1875735809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5e0102710 0x7fd5e01161e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:53.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.301+0000 7fd5deffd700 1 -- 172.21.15.32:0/1875735809 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d0005040 con 0x7fd5e0101d70 2024-04-12T05:51:53.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.301+0000 7fd5deffd700 1 --2- 172.21.15.32:0/1875735809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5e0101d70 0x7fd5e0115ca0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4002a50 tx=0x7fd5d40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:53.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.301+0000 7fd5c77fe700 1 -- 172.21.15.32:0/1875735809 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5d4007cc0 con 0x7fd5e0101d70 2024-04-12T05:51:53.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.302+0000 7fd5e540f700 1 -- 172.21.15.32:0/1875735809 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5e010fee0 con 0x7fd5e0101d70 2024-04-12T05:51:53.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.302+0000 7fd5c77fe700 1 -- 172.21.15.32:0/1875735809 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd5d400f040 con 0x7fd5e0101d70 2024-04-12T05:51:53.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.302+0000 7fd5c77fe700 1 -- 172.21.15.32:0/1875735809 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5d4009540 con 0x7fd5e0101d70 2024-04-12T05:51:53.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.302+0000 7fd5e540f700 1 -- 172.21.15.32:0/1875735809 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5e01104a0 con 0x7fd5e0101d70 2024-04-12T05:51:53.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.302+0000 7fd5e540f700 1 -- 172.21.15.32:0/1875735809 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5e00fd4c0 con 0x7fd5e0101d70 2024-04-12T05:51:53.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.303+0000 7fd5c77fe700 1 -- 172.21.15.32:0/1875735809 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd5d40096a0 con 0x7fd5e0101d70 2024-04-12T05:51:53.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.303+0000 7fd5c77fe700 1 --2- 172.21.15.32:0/1875735809 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd5c806cc30 0x7fd5c806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:53.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.303+0000 7fd5c77fe700 1 -- 172.21.15.32:0/1875735809 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fd5d408bed0 con 0x7fd5e0101d70 2024-04-12T05:51:53.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.303+0000 7fd5de7fc700 1 --2- 172.21.15.32:0/1875735809 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd5c806cc30 0x7fd5c806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:53.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.304+0000 7fd5de7fc700 1 --2- 172.21.15.32:0/1875735809 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd5c806cc30 0x7fd5c806f0e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0111540 tx=0x7fd5d0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:53.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.308+0000 7fd5c77fe700 1 -- 172.21.15.32:0/1875735809 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd5d40598c0 con 0x7fd5e0101d70 2024-04-12T05:51:53.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.533+0000 7fd5e540f700 1 -- 172.21.15.32:0/1875735809 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5e004bf40 con 0x7fd5c806cc30 2024-04-12T05:51:53.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.535+0000 7fd5c77fe700 1 -- 172.21.15.32:0/1875735809 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd5e004bf40 con 0x7fd5c806cc30 2024-04-12T05:51:53.531 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:51:53.531 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:51:53.531 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:51:53.531 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:51:53.531 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:51:53.531 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-12T05:51:53.531 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:51:53.531 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-12T05:51:53.531 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:51:53.531 INFO:teuthology.orchestra.run.smithi032.stdout: "mds" 2024-04-12T05:51:53.531 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:51:53.531 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:51:53.531 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:51:53.531 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:51:53.531 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:51:53.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.537+0000 7fd5e540f700 1 -- 172.21.15.32:0/1875735809 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd5c806cc30 msgr2=0x7fd5c806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:53.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.537+0000 7fd5e540f700 1 --2- 172.21.15.32:0/1875735809 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd5c806cc30 0x7fd5c806f0e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0111540 tx=0x7fd5d0009e30 comp rx=0 tx=0).stop 2024-04-12T05:51:53.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.537+0000 7fd5e540f700 1 -- 172.21.15.32:0/1875735809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5e0101d70 msgr2=0x7fd5e0115ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:53.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.537+0000 7fd5e540f700 1 --2- 172.21.15.32:0/1875735809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5e0101d70 0x7fd5e0115ca0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4002a50 tx=0x7fd5d40046b0 comp rx=0 tx=0).stop 2024-04-12T05:51:53.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.537+0000 7fd5e540f700 1 -- 172.21.15.32:0/1875735809 shutdown_connections 2024-04-12T05:51:53.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.537+0000 7fd5e540f700 1 --2- 172.21.15.32:0/1875735809 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd5c806cc30 0x7fd5c806f0e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:53.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.537+0000 7fd5e540f700 1 --2- 172.21.15.32:0/1875735809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5e0101d70 0x7fd5e0115ca0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:53.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.537+0000 7fd5e540f700 1 --2- 172.21.15.32:0/1875735809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5e0102710 0x7fd5e01161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:53.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.537+0000 7fd5e540f700 1 -- 172.21.15.32:0/1875735809 >> 172.21.15.32:0/1875735809 conn(0x7fd5e0072bc0 msgr2=0x7fd5e00659e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:53.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.537+0000 7fd5e540f700 1 -- 172.21.15.32:0/1875735809 shutdown_connections 2024-04-12T05:51:53.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.537+0000 7fd5e540f700 1 -- 172.21.15.32:0/1875735809 wait complete. 2024-04-12T05:51:53.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.641+0000 7f7c1aa3f700 1 -- 172.21.15.32:0/3510086233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c14101d80 msgr2=0x7f7c14102150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:53.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.641+0000 7f7c1aa3f700 1 --2- 172.21.15.32:0/3510086233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c14101d80 0x7f7c14102150 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7c04004940 tx=0x7f7c04009e30 comp rx=0 tx=0).stop 2024-04-12T05:51:53.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.641+0000 7f7c1aa3f700 1 -- 172.21.15.32:0/3510086233 shutdown_connections 2024-04-12T05:51:53.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.641+0000 7f7c1aa3f700 1 --2- 172.21.15.32:0/3510086233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c14102720 0x7f7c140fbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:53.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.641+0000 7f7c1aa3f700 1 --2- 172.21.15.32:0/3510086233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c14101d80 0x7f7c14102150 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:53.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.641+0000 7f7c1aa3f700 1 -- 172.21.15.32:0/3510086233 >> 172.21.15.32:0/3510086233 conn(0x7f7c140f78f0 msgr2=0x7f7c140f9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:53.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.641+0000 7f7c1aa3f700 1 -- 172.21.15.32:0/3510086233 shutdown_connections 2024-04-12T05:51:53.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.641+0000 7f7c1aa3f700 1 -- 172.21.15.32:0/3510086233 wait complete. 2024-04-12T05:51:53.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.641+0000 7f7c1aa3f700 1 Processor -- start 2024-04-12T05:51:53.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.642+0000 7f7c1aa3f700 1 -- start start 2024-04-12T05:51:53.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.642+0000 7f7c1aa3f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c14101d80 0x7f7c141127c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:53.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.642+0000 7f7c1aa3f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c14102720 0x7f7c14112d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:53.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.642+0000 7f7c1aa3f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c14113350 con 0x7f7c14102720 2024-04-12T05:51:53.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.642+0000 7f7c1aa3f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1410c950 con 0x7f7c14101d80 2024-04-12T05:51:53.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.642+0000 7f7c137fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c14102720 0x7f7c14112d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:53.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.642+0000 7f7c137fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c14102720 0x7f7c14112d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51458/0 (socket says 172.21.15.32:51458) 2024-04-12T05:51:53.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.642+0000 7f7c137fe700 1 -- 172.21.15.32:0/1040849986 learned_addr learned my addr 172.21.15.32:0/1040849986 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:51:53.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.642+0000 7f7c13fff700 1 --2- 172.21.15.32:0/1040849986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c14101d80 0x7f7c141127c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:53.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.642+0000 7f7c137fe700 1 -- 172.21.15.32:0/1040849986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c14101d80 msgr2=0x7f7c141127c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:53.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.642+0000 7f7c137fe700 1 --2- 172.21.15.32:0/1040849986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c14101d80 0x7f7c141127c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:53.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.642+0000 7f7c137fe700 1 -- 172.21.15.32:0/1040849986 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c04005040 con 0x7f7c14102720 2024-04-12T05:51:53.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.643+0000 7f7c137fe700 1 --2- 172.21.15.32:0/1040849986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c14102720 0x7f7c14112d00 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7c0800c800 tx=0x7f7c08004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:53.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.643+0000 7f7c10ff9700 1 -- 172.21.15.32:0/1040849986 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c0800a760 con 0x7f7c14102720 2024-04-12T05:51:53.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.643+0000 7f7c10ff9700 1 -- 172.21.15.32:0/1040849986 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c0800ad70 con 0x7f7c14102720 2024-04-12T05:51:53.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.643+0000 7f7c1aa3f700 1 -- 172.21.15.32:0/1040849986 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c1410cc30 con 0x7f7c14102720 2024-04-12T05:51:53.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.643+0000 7f7c10ff9700 1 -- 172.21.15.32:0/1040849986 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c080113a0 con 0x7f7c14102720 2024-04-12T05:51:53.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.643+0000 7f7c1aa3f700 1 -- 172.21.15.32:0/1040849986 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c1410d1f0 con 0x7f7c14102720 2024-04-12T05:51:53.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.644+0000 7f7c1aa3f700 1 -- 172.21.15.32:0/1040849986 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c140ff090 con 0x7f7c14102720 2024-04-12T05:51:53.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.644+0000 7f7c10ff9700 1 -- 172.21.15.32:0/1040849986 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f7c08012400 con 0x7f7c14102720 2024-04-12T05:51:53.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.645+0000 7f7c10ff9700 1 --2- 172.21.15.32:0/1040849986 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7bfc06cc20 0x7f7bfc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:51:53.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.645+0000 7f7c13fff700 1 --2- 172.21.15.32:0/1040849986 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7bfc06cc20 0x7f7bfc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:53.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.645+0000 7f7c10ff9700 1 -- 172.21.15.32:0/1040849986 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f7c0808c140 con 0x7f7c14102720 2024-04-12T05:51:53.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.645+0000 7f7c13fff700 1 --2- 172.21.15.32:0/1040849986 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7bfc06cc20 0x7f7bfc06f0d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7c0400b3b0 tx=0x7f7c04009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:53.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.650+0000 7f7c10ff9700 1 -- 172.21.15.32:0/1040849986 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7c08059bb0 con 0x7f7c14102720 2024-04-12T05:51:53.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.943+0000 7f7c1aa3f700 1 -- 172.21.15.32:0/1040849986 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7c1410de10 con 0x7f7c14102720 2024-04-12T05:51:53.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.943+0000 7f7c10ff9700 1 -- 172.21.15.32:0/1040849986 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f7c08059740 con 0x7f7c14102720 2024-04-12T05:51:53.939 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:51:53.939 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:51:53.939 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-12T05:51:53.939 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:51:53.939 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:51:53.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.945+0000 7f7c1aa3f700 1 -- 172.21.15.32:0/1040849986 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7bfc06cc20 msgr2=0x7f7bfc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:53.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.945+0000 7f7c1aa3f700 1 --2- 172.21.15.32:0/1040849986 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7bfc06cc20 0x7f7bfc06f0d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7c0400b3b0 tx=0x7f7c04009e30 comp rx=0 tx=0).stop 2024-04-12T05:51:53.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.945+0000 7f7c1aa3f700 1 -- 172.21.15.32:0/1040849986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c14102720 msgr2=0x7f7c14112d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:53.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.945+0000 7f7c1aa3f700 1 --2- 172.21.15.32:0/1040849986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c14102720 0x7f7c14112d00 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7c0800c800 tx=0x7f7c08004470 comp rx=0 tx=0).stop 2024-04-12T05:51:53.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.945+0000 7f7c1aa3f700 1 -- 172.21.15.32:0/1040849986 shutdown_connections 2024-04-12T05:51:53.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.945+0000 7f7c1aa3f700 1 --2- 172.21.15.32:0/1040849986 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7bfc06cc20 0x7f7bfc06f0d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:53.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.945+0000 7f7c1aa3f700 1 --2- 172.21.15.32:0/1040849986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c14101d80 0x7f7c141127c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:53.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.945+0000 7f7c1aa3f700 1 --2- 172.21.15.32:0/1040849986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c14102720 0x7f7c14112d00 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:51:53.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.945+0000 7f7c1aa3f700 1 -- 172.21.15.32:0/1040849986 >> 172.21.15.32:0/1040849986 conn(0x7f7c140f78f0 msgr2=0x7f7c140f9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:53.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.945+0000 7f7c1aa3f700 1 -- 172.21.15.32:0/1040849986 shutdown_connections 2024-04-12T05:51:53.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:51:53.945+0000 7f7c1aa3f700 1 -- 172.21.15.32:0/1040849986 wait complete. 2024-04-12T05:51:54.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:53 smithi032 ceph-mon[146817]: pgmap v341: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:54.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:53 smithi032 ceph-mon[146817]: from='client.34680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:51:54.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:53 smithi032 ceph-mon[146817]: from='client.34684 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:51:54.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:53 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1886175840' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:51:54.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:53 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/846828928' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:51:54.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:53 smithi067 ceph-mon[112979]: pgmap v341: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:54.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:53 smithi067 ceph-mon[112979]: from='client.34680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:51:54.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:53 smithi067 ceph-mon[112979]: from='client.34684 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:51:54.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:53 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1886175840' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:51:54.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:53 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/846828928' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:51:55.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:54 smithi032 ceph-mon[146817]: from='client.34696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:51:55.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:54 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1040849986' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:51:55.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:54 smithi067 ceph-mon[112979]: from='client.34696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:51:55.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:54 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1040849986' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:51:56.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:55 smithi032 ceph-mon[146817]: pgmap v342: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:56.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:55 smithi067 ceph-mon[112979]: pgmap v342: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:58.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:57 smithi032 ceph-mon[146817]: pgmap v343: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:58.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:57 smithi067 ceph-mon[112979]: pgmap v343: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:00.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:51:59 smithi032 ceph-mon[146817]: pgmap v344: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:00.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:51:59 smithi067 ceph-mon[112979]: pgmap v344: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:02.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:01 smithi032 ceph-mon[146817]: pgmap v345: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:02.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:01 smithi067 ceph-mon[112979]: pgmap v345: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:04.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:03 smithi032 ceph-mon[146817]: pgmap v346: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:04.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:52:04.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:03 smithi067 ceph-mon[112979]: pgmap v346: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:04.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:52:06.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:05 smithi032 ceph-mon[146817]: pgmap v347: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:06.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:05 smithi067 ceph-mon[112979]: pgmap v347: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:08.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:07 smithi032 ceph-mon[146817]: pgmap v348: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:08.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:07 smithi067 ceph-mon[112979]: pgmap v348: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:10.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:09 smithi032 ceph-mon[146817]: pgmap v349: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:10.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:09 smithi067 ceph-mon[112979]: pgmap v349: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:12.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:11 smithi032 ceph-mon[146817]: pgmap v350: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:12.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:11 smithi067 ceph-mon[112979]: pgmap v350: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:14.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:13 smithi032 ceph-mon[146817]: pgmap v351: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:14.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:13 smithi067 ceph-mon[112979]: pgmap v351: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:16.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:15 smithi032 ceph-mon[146817]: pgmap v352: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:16.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:15 smithi067 ceph-mon[112979]: pgmap v352: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:18.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:17 smithi032 ceph-mon[146817]: pgmap v353: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:18.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:17 smithi067 ceph-mon[112979]: pgmap v353: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:19.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:52:19.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:52:20.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:19 smithi032 ceph-mon[146817]: pgmap v354: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:20.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:19 smithi067 ceph-mon[112979]: pgmap v354: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:21.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:20 smithi032 ceph-mon[146817]: pgmap v355: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:21.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:20 smithi067 ceph-mon[112979]: pgmap v355: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:23.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:22 smithi067 ceph-mon[112979]: pgmap v356: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:23.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:22 smithi032 ceph-mon[146817]: pgmap v356: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:24.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.064+0000 7ff6f4d5f700 1 -- 172.21.15.32:0/3417418715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6f01037b0 msgr2=0x7ff6f00fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:24.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.065+0000 7ff6f4d5f700 1 --2- 172.21.15.32:0/3417418715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6f01037b0 0x7ff6f00fce20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4004940 tx=0x7ff6e4009e30 comp rx=0 tx=0).stop 2024-04-12T05:52:24.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.065+0000 7ff6f4d5f700 1 -- 172.21.15.32:0/3417418715 shutdown_connections 2024-04-12T05:52:24.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.065+0000 7ff6f4d5f700 1 --2- 172.21.15.32:0/3417418715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6f01037b0 0x7ff6f00fce20 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:24.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.065+0000 7ff6f4d5f700 1 --2- 172.21.15.32:0/3417418715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6f0102e10 0x7ff6f01031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:24.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.065+0000 7ff6f4d5f700 1 -- 172.21.15.32:0/3417418715 >> 172.21.15.32:0/3417418715 conn(0x7ff6f00f8980 msgr2=0x7ff6f00fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:52:24.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.065+0000 7ff6f4d5f700 1 -- 172.21.15.32:0/3417418715 shutdown_connections 2024-04-12T05:52:24.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.065+0000 7ff6f4d5f700 1 -- 172.21.15.32:0/3417418715 wait complete. 2024-04-12T05:52:24.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.065+0000 7ff6f4d5f700 1 Processor -- start 2024-04-12T05:52:24.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.065+0000 7ff6f4d5f700 1 -- start start 2024-04-12T05:52:24.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.066+0000 7ff6f4d5f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6f0102e10 0x7ff6f0113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.066+0000 7ff6f4d5f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6f01037b0 0x7ff6f0114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.066+0000 7ff6f4d5f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f0114670 con 0x7ff6f01037b0 2024-04-12T05:52:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.066+0000 7ff6f4d5f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f010da60 con 0x7ff6f0102e10 2024-04-12T05:52:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.066+0000 7ff6edd9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6f01037b0 0x7ff6f0114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.066+0000 7ff6edd9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6f01037b0 0x7ff6f0114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43502/0 (socket says 172.21.15.32:43502) 2024-04-12T05:52:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.066+0000 7ff6edd9b700 1 -- 172.21.15.32:0/2433584897 learned_addr learned my addr 172.21.15.32:0/2433584897 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:52:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.066+0000 7ff6ee59c700 1 --2- 172.21.15.32:0/2433584897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6f0102e10 0x7ff6f0113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:24.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.066+0000 7ff6edd9b700 1 -- 172.21.15.32:0/2433584897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6f0102e10 msgr2=0x7ff6f0113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:24.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.066+0000 7ff6edd9b700 1 --2- 172.21.15.32:0/2433584897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6f0102e10 0x7ff6f0113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:24.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.066+0000 7ff6edd9b700 1 -- 172.21.15.32:0/2433584897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e4005040 con 0x7ff6f01037b0 2024-04-12T05:52:24.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.066+0000 7ff6edd9b700 1 --2- 172.21.15.32:0/2433584897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6f01037b0 0x7ff6f0114020 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff6e40093b0 tx=0x7ff6e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:52:24.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.066+0000 7ff6deffd700 1 -- 172.21.15.32:0/2433584897 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff6e400e040 con 0x7ff6f01037b0 2024-04-12T05:52:24.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.067+0000 7ff6f4d5f700 1 -- 172.21.15.32:0/2433584897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f010dce0 con 0x7ff6f01037b0 2024-04-12T05:52:24.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.067+0000 7ff6deffd700 1 -- 172.21.15.32:0/2433584897 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff6e4013420 con 0x7ff6f01037b0 2024-04-12T05:52:24.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.067+0000 7ff6deffd700 1 -- 172.21.15.32:0/2433584897 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff6e401c630 con 0x7ff6f01037b0 2024-04-12T05:52:24.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.067+0000 7ff6f4d5f700 1 -- 172.21.15.32:0/2433584897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f010e2a0 con 0x7ff6f01037b0 2024-04-12T05:52:24.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.068+0000 7ff6f4d5f700 1 -- 172.21.15.32:0/2433584897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f00fe560 con 0x7ff6f01037b0 2024-04-12T05:52:24.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.068+0000 7ff6deffd700 1 -- 172.21.15.32:0/2433584897 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ff6e40139e0 con 0x7ff6f01037b0 2024-04-12T05:52:24.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.069+0000 7ff6deffd700 1 --2- 172.21.15.32:0/2433584897 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff6d806cc20 0x7ff6d806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:24.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.069+0000 7ff6ee59c700 1 --2- 172.21.15.32:0/2433584897 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff6d806cc20 0x7ff6d806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:24.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.069+0000 7ff6deffd700 1 -- 172.21.15.32:0/2433584897 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7ff6e408d1a0 con 0x7ff6f01037b0 2024-04-12T05:52:24.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.069+0000 7ff6ee59c700 1 --2- 172.21.15.32:0/2433584897 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff6d806cc20 0x7ff6d806f0d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff6e0007fd0 tx=0x7ff6e0009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:52:24.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.074+0000 7ff6deffd700 1 -- 172.21.15.32:0/2433584897 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff6e405ac10 con 0x7ff6f01037b0 2024-04-12T05:52:24.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.296+0000 7ff6f4d5f700 1 -- 172.21.15.32:0/2433584897 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6f0109db0 con 0x7ff6d806cc20 2024-04-12T05:52:24.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.297+0000 7ff6deffd700 1 -- 172.21.15.32:0/2433584897 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff6f0109db0 con 0x7ff6d806cc20 2024-04-12T05:52:24.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.299+0000 7ff6f4d5f700 1 -- 172.21.15.32:0/2433584897 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff6d806cc20 msgr2=0x7ff6d806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:24.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.299+0000 7ff6f4d5f700 1 --2- 172.21.15.32:0/2433584897 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff6d806cc20 0x7ff6d806f0d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff6e0007fd0 tx=0x7ff6e0009040 comp rx=0 tx=0).stop 2024-04-12T05:52:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.299+0000 7ff6f4d5f700 1 -- 172.21.15.32:0/2433584897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6f01037b0 msgr2=0x7ff6f0114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.299+0000 7ff6f4d5f700 1 --2- 172.21.15.32:0/2433584897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6f01037b0 0x7ff6f0114020 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff6e40093b0 tx=0x7ff6e4009e30 comp rx=0 tx=0).stop 2024-04-12T05:52:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.300+0000 7ff6f4d5f700 1 -- 172.21.15.32:0/2433584897 shutdown_connections 2024-04-12T05:52:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.300+0000 7ff6f4d5f700 1 --2- 172.21.15.32:0/2433584897 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff6d806cc20 0x7ff6d806f0d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.300+0000 7ff6f4d5f700 1 --2- 172.21.15.32:0/2433584897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6f0102e10 0x7ff6f0113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.300+0000 7ff6f4d5f700 1 --2- 172.21.15.32:0/2433584897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6f01037b0 0x7ff6f0114020 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.300+0000 7ff6f4d5f700 1 -- 172.21.15.32:0/2433584897 >> 172.21.15.32:0/2433584897 conn(0x7ff6f00f8980 msgr2=0x7ff6f00f9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:52:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.300+0000 7ff6f4d5f700 1 -- 172.21.15.32:0/2433584897 shutdown_connections 2024-04-12T05:52:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.300+0000 7ff6f4d5f700 1 -- 172.21.15.32:0/2433584897 wait complete. 2024-04-12T05:52:24.306 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:52:24.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.404+0000 7f0d56ca9700 1 -- 172.21.15.32:0/1743354034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d50101d70 msgr2=0x7f0d50102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:24.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.404+0000 7f0d56ca9700 1 --2- 172.21.15.32:0/1743354034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d50101d70 0x7f0d50102140 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0d44004940 tx=0x7f0d44009e30 comp rx=0 tx=0).stop 2024-04-12T05:52:24.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.404+0000 7f0d56ca9700 1 -- 172.21.15.32:0/1743354034 shutdown_connections 2024-04-12T05:52:24.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.404+0000 7f0d56ca9700 1 --2- 172.21.15.32:0/1743354034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d500fc5b0 0x7f0d500fca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:24.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.404+0000 7f0d56ca9700 1 --2- 172.21.15.32:0/1743354034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d50101d70 0x7f0d50102140 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:24.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.404+0000 7f0d56ca9700 1 -- 172.21.15.32:0/1743354034 >> 172.21.15.32:0/1743354034 conn(0x7f0d500714c0 msgr2=0x7f0d500718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:52:24.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.404+0000 7f0d56ca9700 1 -- 172.21.15.32:0/1743354034 shutdown_connections 2024-04-12T05:52:24.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.404+0000 7f0d56ca9700 1 -- 172.21.15.32:0/1743354034 wait complete. 2024-04-12T05:52:24.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.405+0000 7f0d56ca9700 1 Processor -- start 2024-04-12T05:52:24.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.405+0000 7f0d56ca9700 1 -- start start 2024-04-12T05:52:24.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.405+0000 7f0d56ca9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d500fc5b0 0x7f0d5010e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:24.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.405+0000 7f0d56ca9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d50101d70 0x7f0d5010ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:24.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.405+0000 7f0d56ca9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5010f580 con 0x7f0d50101d70 2024-04-12T05:52:24.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.405+0000 7f0d56ca9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5010f6c0 con 0x7f0d500fc5b0 2024-04-12T05:52:24.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.405+0000 7f0d4ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d50101d70 0x7f0d5010ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:24.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.405+0000 7f0d4ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d50101d70 0x7f0d5010ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43526/0 (socket says 172.21.15.32:43526) 2024-04-12T05:52:24.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.406+0000 7f0d4ffff700 1 -- 172.21.15.32:0/486407364 learned_addr learned my addr 172.21.15.32:0/486407364 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:52:24.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.406+0000 7f0d54a45700 1 --2- 172.21.15.32:0/486407364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d500fc5b0 0x7f0d5010e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:24.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.406+0000 7f0d4ffff700 1 -- 172.21.15.32:0/486407364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d500fc5b0 msgr2=0x7f0d5010e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:24.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.406+0000 7f0d4ffff700 1 --2- 172.21.15.32:0/486407364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d500fc5b0 0x7f0d5010e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:24.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.406+0000 7f0d4ffff700 1 -- 172.21.15.32:0/486407364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d44005040 con 0x7f0d50101d70 2024-04-12T05:52:24.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.406+0000 7f0d54a45700 1 --2- 172.21.15.32:0/486407364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d500fc5b0 0x7f0d5010e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:52:24.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.406+0000 7f0d4ffff700 1 --2- 172.21.15.32:0/486407364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d50101d70 0x7f0d5010ef30 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c002a80 tx=0x7f0d3c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:52:24.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.406+0000 7f0d4d7fa700 1 -- 172.21.15.32:0/486407364 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d3c007cc0 con 0x7f0d50101d70 2024-04-12T05:52:24.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.406+0000 7f0d4d7fa700 1 -- 172.21.15.32:0/486407364 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d3c00f040 con 0x7f0d50101d70 2024-04-12T05:52:24.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.406+0000 7f0d56ca9700 1 -- 172.21.15.32:0/486407364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d50117a60 con 0x7f0d50101d70 2024-04-12T05:52:24.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.406+0000 7f0d4d7fa700 1 -- 172.21.15.32:0/486407364 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d3c009640 con 0x7f0d50101d70 2024-04-12T05:52:24.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.406+0000 7f0d56ca9700 1 -- 172.21.15.32:0/486407364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d50117fc0 con 0x7f0d50101d70 2024-04-12T05:52:24.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.407+0000 7f0d56ca9700 1 -- 172.21.15.32:0/486407364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d500fd5c0 con 0x7f0d50101d70 2024-04-12T05:52:24.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.408+0000 7f0d4d7fa700 1 -- 172.21.15.32:0/486407364 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f0d3c0097a0 con 0x7f0d50101d70 2024-04-12T05:52:24.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.408+0000 7f0d4d7fa700 1 --2- 172.21.15.32:0/486407364 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0d4006cc20 0x7f0d4006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:24.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.409+0000 7f0d4d7fa700 1 -- 172.21.15.32:0/486407364 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f0d3c08bb80 con 0x7f0d50101d70 2024-04-12T05:52:24.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.409+0000 7f0d54a45700 1 --2- 172.21.15.32:0/486407364 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0d4006cc20 0x7f0d4006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:24.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.409+0000 7f0d54a45700 1 --2- 172.21.15.32:0/486407364 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0d4006cc20 0x7f0d4006f0d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0d44004f90 tx=0x7f0d44009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:52:24.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.412+0000 7f0d4d7fa700 1 -- 172.21.15.32:0/486407364 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0d3c055ac0 con 0x7f0d50101d70 2024-04-12T05:52:24.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.635+0000 7f0d56ca9700 1 -- 172.21.15.32:0/486407364 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d50108d50 con 0x7f0d4006cc20 2024-04-12T05:52:24.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.636+0000 7f0d4d7fa700 1 -- 172.21.15.32:0/486407364 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0d44007cc0 con 0x7f0d4006cc20 2024-04-12T05:52:24.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.638+0000 7f0d56ca9700 1 -- 172.21.15.32:0/486407364 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0d4006cc20 msgr2=0x7f0d4006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:24.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.638+0000 7f0d56ca9700 1 --2- 172.21.15.32:0/486407364 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0d4006cc20 0x7f0d4006f0d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0d44004f90 tx=0x7f0d44009e30 comp rx=0 tx=0).stop 2024-04-12T05:52:24.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.638+0000 7f0d56ca9700 1 -- 172.21.15.32:0/486407364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d50101d70 msgr2=0x7f0d5010ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:24.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.638+0000 7f0d56ca9700 1 --2- 172.21.15.32:0/486407364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d50101d70 0x7f0d5010ef30 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c002a80 tx=0x7f0d3c0046b0 comp rx=0 tx=0).stop 2024-04-12T05:52:24.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.638+0000 7f0d56ca9700 1 -- 172.21.15.32:0/486407364 shutdown_connections 2024-04-12T05:52:24.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.638+0000 7f0d56ca9700 1 --2- 172.21.15.32:0/486407364 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0d4006cc20 0x7f0d4006f0d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:24.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.638+0000 7f0d56ca9700 1 --2- 172.21.15.32:0/486407364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d500fc5b0 0x7f0d5010e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:24.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.638+0000 7f0d56ca9700 1 --2- 172.21.15.32:0/486407364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d50101d70 0x7f0d5010ef30 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:24.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.638+0000 7f0d56ca9700 1 -- 172.21.15.32:0/486407364 >> 172.21.15.32:0/486407364 conn(0x7f0d500714c0 msgr2=0x7f0d50104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:52:24.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.638+0000 7f0d56ca9700 1 -- 172.21.15.32:0/486407364 shutdown_connections 2024-04-12T05:52:24.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.638+0000 7f0d56ca9700 1 -- 172.21.15.32:0/486407364 wait complete. 2024-04-12T05:52:24.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.741+0000 7ff984dcc700 1 -- 172.21.15.32:0/3631788468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9801037b0 msgr2=0x7ff9800fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:24.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.741+0000 7ff984dcc700 1 --2- 172.21.15.32:0/3631788468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9801037b0 0x7ff9800fce20 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff974004940 tx=0x7ff974009e30 comp rx=0 tx=0).stop 2024-04-12T05:52:24.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.742+0000 7ff984dcc700 1 -- 172.21.15.32:0/3631788468 shutdown_connections 2024-04-12T05:52:24.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.742+0000 7ff984dcc700 1 --2- 172.21.15.32:0/3631788468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9801037b0 0x7ff9800fce20 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:24.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.742+0000 7ff984dcc700 1 --2- 172.21.15.32:0/3631788468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff980102e10 0x7ff9801031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:24.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.742+0000 7ff984dcc700 1 -- 172.21.15.32:0/3631788468 >> 172.21.15.32:0/3631788468 conn(0x7ff9800f89a0 msgr2=0x7ff9800fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:52:24.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.742+0000 7ff984dcc700 1 -- 172.21.15.32:0/3631788468 shutdown_connections 2024-04-12T05:52:24.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.742+0000 7ff984dcc700 1 -- 172.21.15.32:0/3631788468 wait complete. 2024-04-12T05:52:24.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.742+0000 7ff984dcc700 1 Processor -- start 2024-04-12T05:52:24.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.742+0000 7ff984dcc700 1 -- start start 2024-04-12T05:52:24.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.742+0000 7ff984dcc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff980102e10 0x7ff980113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:24.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.742+0000 7ff984dcc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9801037b0 0x7ff980114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:24.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.742+0000 7ff984dcc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980114660 con 0x7ff980102e10 2024-04-12T05:52:24.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.742+0000 7ff984dcc700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98010da50 con 0x7ff9801037b0 2024-04-12T05:52:24.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.743+0000 7ff97e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff980102e10 0x7ff980113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:24.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.743+0000 7ff97e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff980102e10 0x7ff980113ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43550/0 (socket says 172.21.15.32:43550) 2024-04-12T05:52:24.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.743+0000 7ff97e59c700 1 -- 172.21.15.32:0/2083544004 learned_addr learned my addr 172.21.15.32:0/2083544004 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:52:24.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.743+0000 7ff97dd9b700 1 --2- 172.21.15.32:0/2083544004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9801037b0 0x7ff980114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:24.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.743+0000 7ff97e59c700 1 -- 172.21.15.32:0/2083544004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9801037b0 msgr2=0x7ff980114010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:24.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.743+0000 7ff97e59c700 1 --2- 172.21.15.32:0/2083544004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9801037b0 0x7ff980114010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:24.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.743+0000 7ff97e59c700 1 -- 172.21.15.32:0/2083544004 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff974005040 con 0x7ff980102e10 2024-04-12T05:52:24.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.743+0000 7ff97e59c700 1 --2- 172.21.15.32:0/2083544004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff980102e10 0x7ff980113ad0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff97000c830 tx=0x7ff970004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:52:24.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.743+0000 7ff96effd700 1 -- 172.21.15.32:0/2083544004 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9700106f0 con 0x7ff980102e10 2024-04-12T05:52:24.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.743+0000 7ff96effd700 1 -- 172.21.15.32:0/2083544004 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff970010d00 con 0x7ff980102e10 2024-04-12T05:52:24.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.743+0000 7ff984dcc700 1 -- 172.21.15.32:0/2083544004 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff98010dd30 con 0x7ff980102e10 2024-04-12T05:52:24.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.743+0000 7ff96effd700 1 -- 172.21.15.32:0/2083544004 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff970012e90 con 0x7ff980102e10 2024-04-12T05:52:24.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.743+0000 7ff984dcc700 1 -- 172.21.15.32:0/2083544004 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff98010e2c0 con 0x7ff980102e10 2024-04-12T05:52:24.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.744+0000 7ff984dcc700 1 -- 172.21.15.32:0/2083544004 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9800fe560 con 0x7ff980102e10 2024-04-12T05:52:24.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.745+0000 7ff96effd700 1 -- 172.21.15.32:0/2083544004 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ff970010850 con 0x7ff980102e10 2024-04-12T05:52:24.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.745+0000 7ff96effd700 1 --2- 172.21.15.32:0/2083544004 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff96806cc20 0x7ff96806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:24.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.745+0000 7ff96effd700 1 -- 172.21.15.32:0/2083544004 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7ff97008c070 con 0x7ff980102e10 2024-04-12T05:52:24.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.745+0000 7ff97dd9b700 1 --2- 172.21.15.32:0/2083544004 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff96806cc20 0x7ff96806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:24.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.746+0000 7ff97dd9b700 1 --2- 172.21.15.32:0/2083544004 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff96806cc20 0x7ff96806f0d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff974004890 tx=0x7ff974009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:52:24.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.750+0000 7ff96effd700 1 -- 172.21.15.32:0/2083544004 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff970059ae0 con 0x7ff980102e10 2024-04-12T05:52:24.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.963+0000 7ff984dcc700 1 -- 172.21.15.32:0/2083544004 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff980109e00 con 0x7ff96806cc20 2024-04-12T05:52:24.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.973+0000 7ff96effd700 1 -- 172.21.15.32:0/2083544004 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7ff980109e00 con 0x7ff96806cc20 2024-04-12T05:52:24.969 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:52:24.969 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (12m) 5m ago 23m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:52:24.969 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (9m) 5m ago 23m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:52:24.969 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (9m) 5m ago 22m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:52:24.969 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (11m) 5m ago 23m 87.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:52:24.969 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (6m) 5m ago 17m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T05:52:24.969 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (5m) 5m ago 17m 13.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T05:52:24.969 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (5m) 5m ago 17m 11.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 6d4aca6f74a0 2024-04-12T05:52:24.969 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (5m) 5m ago 17m 14.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T05:52:24.970 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (13m) 5m ago 25m 507M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:52:24.970 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (13m) 5m ago 22m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:52:24.970 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (10m) 5m ago 25m 53.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:52:24.970 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (10m) 5m ago 22m 43.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:52:24.970 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (12m) 5m ago 23m 16.9M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:52:24.970 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (12m) 5m ago 22m 18.4M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:52:24.970 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (9m) 5m ago 21m 91.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T05:52:24.970 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (8m) 5m ago 21m 97.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T05:52:24.970 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (8m) 5m ago 20m 81.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T05:52:24.970 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (7m) 5m ago 20m 113M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T05:52:24.970 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (6m) 5m ago 19m 90.3M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T05:52:24.970 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (6m) 5m ago 19m 84.8M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T05:52:24.970 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (12m) 5m ago 23m 64.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:52:24.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.975+0000 7ff984dcc700 1 -- 172.21.15.32:0/2083544004 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff96806cc20 msgr2=0x7ff96806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:24.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.975+0000 7ff984dcc700 1 --2- 172.21.15.32:0/2083544004 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff96806cc20 0x7ff96806f0d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff974004890 tx=0x7ff974009e30 comp rx=0 tx=0).stop 2024-04-12T05:52:24.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.975+0000 7ff984dcc700 1 -- 172.21.15.32:0/2083544004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff980102e10 msgr2=0x7ff980113ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:24.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.975+0000 7ff984dcc700 1 --2- 172.21.15.32:0/2083544004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff980102e10 0x7ff980113ad0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff97000c830 tx=0x7ff970004470 comp rx=0 tx=0).stop 2024-04-12T05:52:24.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.975+0000 7ff984dcc700 1 -- 172.21.15.32:0/2083544004 shutdown_connections 2024-04-12T05:52:24.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.975+0000 7ff984dcc700 1 --2- 172.21.15.32:0/2083544004 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff96806cc20 0x7ff96806f0d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:24.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.975+0000 7ff984dcc700 1 --2- 172.21.15.32:0/2083544004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff980102e10 0x7ff980113ad0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:24.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.975+0000 7ff984dcc700 1 --2- 172.21.15.32:0/2083544004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9801037b0 0x7ff980114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:24.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.975+0000 7ff984dcc700 1 -- 172.21.15.32:0/2083544004 >> 172.21.15.32:0/2083544004 conn(0x7ff9800f89a0 msgr2=0x7ff9800f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:52:24.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.975+0000 7ff984dcc700 1 -- 172.21.15.32:0/2083544004 shutdown_connections 2024-04-12T05:52:24.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:24.975+0000 7ff984dcc700 1 -- 172.21.15.32:0/2083544004 wait complete. 2024-04-12T05:52:25.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.078+0000 7f526b174700 1 -- 172.21.15.32:0/2366146930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5264102e70 msgr2=0x7f5264103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:25.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.078+0000 7f526b174700 1 --2- 172.21.15.32:0/2366146930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5264102e70 0x7f5264103240 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5258004940 tx=0x7f5258009e30 comp rx=0 tx=0).stop 2024-04-12T05:52:25.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.079+0000 7f526b174700 1 -- 172.21.15.32:0/2366146930 shutdown_connections 2024-04-12T05:52:25.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.079+0000 7f526b174700 1 --2- 172.21.15.32:0/2366146930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52640fb4a0 0x7f52640fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:25.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.079+0000 7f526b174700 1 --2- 172.21.15.32:0/2366146930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5264102e70 0x7f5264103240 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:25.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.079+0000 7f526b174700 1 -- 172.21.15.32:0/2366146930 >> 172.21.15.32:0/2366146930 conn(0x7f52640714c0 msgr2=0x7f52640718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:52:25.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.079+0000 7f526b174700 1 -- 172.21.15.32:0/2366146930 shutdown_connections 2024-04-12T05:52:25.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.079+0000 7f526b174700 1 -- 172.21.15.32:0/2366146930 wait complete. 2024-04-12T05:52:25.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.079+0000 7f526b174700 1 Processor -- start 2024-04-12T05:52:25.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.079+0000 7f526b174700 1 -- start start 2024-04-12T05:52:25.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.080+0000 7f526b174700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52640fb4a0 0x7f5264113a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:25.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.080+0000 7f526b174700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5264102e70 0x7f5264113fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:25.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.080+0000 7f526b174700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5264114620 con 0x7f5264102e70 2024-04-12T05:52:25.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.080+0000 7f526b174700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f526410da10 con 0x7f52640fb4a0 2024-04-12T05:52:25.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.080+0000 7f5263fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5264102e70 0x7f5264113fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:25.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.080+0000 7f5263fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5264102e70 0x7f5264113fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43566/0 (socket says 172.21.15.32:43566) 2024-04-12T05:52:25.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.080+0000 7f5263fff700 1 -- 172.21.15.32:0/293395160 learned_addr learned my addr 172.21.15.32:0/293395160 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:52:25.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.080+0000 7f5268f10700 1 --2- 172.21.15.32:0/293395160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52640fb4a0 0x7f5264113a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:25.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.080+0000 7f5263fff700 1 -- 172.21.15.32:0/293395160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52640fb4a0 msgr2=0x7f5264113a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:25.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.080+0000 7f5263fff700 1 --2- 172.21.15.32:0/293395160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52640fb4a0 0x7f5264113a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:25.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.080+0000 7f5263fff700 1 -- 172.21.15.32:0/293395160 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5258005040 con 0x7f5264102e70 2024-04-12T05:52:25.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.080+0000 7f5263fff700 1 --2- 172.21.15.32:0/293395160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5264102e70 0x7f5264113fd0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f525000eb40 tx=0x7f525000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:52:25.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.080+0000 7f52617fa700 1 -- 172.21.15.32:0/293395160 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5250007cc0 con 0x7f5264102e70 2024-04-12T05:52:25.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.080+0000 7f52617fa700 1 -- 172.21.15.32:0/293395160 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f525000f040 con 0x7f5264102e70 2024-04-12T05:52:25.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.080+0000 7f526b174700 1 -- 172.21.15.32:0/293395160 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f526410dc90 con 0x7f5264102e70 2024-04-12T05:52:25.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.080+0000 7f52617fa700 1 -- 172.21.15.32:0/293395160 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f525000c990 con 0x7f5264102e70 2024-04-12T05:52:25.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.080+0000 7f526b174700 1 -- 172.21.15.32:0/293395160 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f526410e1d0 con 0x7f5264102e70 2024-04-12T05:52:25.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.081+0000 7f526b174700 1 -- 172.21.15.32:0/293395160 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5264100170 con 0x7f5264102e70 2024-04-12T05:52:25.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.082+0000 7f52617fa700 1 -- 172.21.15.32:0/293395160 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f525000cc00 con 0x7f5264102e70 2024-04-12T05:52:25.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.082+0000 7f52617fa700 1 --2- 172.21.15.32:0/293395160 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f525406cc30 0x7f525406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:25.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.082+0000 7f52617fa700 1 -- 172.21.15.32:0/293395160 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f525008bcc0 con 0x7f5264102e70 2024-04-12T05:52:25.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.083+0000 7f5268f10700 1 --2- 172.21.15.32:0/293395160 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f525406cc30 0x7f525406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:25.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.083+0000 7f5268f10700 1 --2- 172.21.15.32:0/293395160 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f525406cc30 0x7f525406f0e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f525800b0d0 tx=0x7f5258009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:52:25.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.087+0000 7f52617fa700 1 -- 172.21.15.32:0/293395160 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5250059760 con 0x7f5264102e70 2024-04-12T05:52:25.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:24 smithi067 ceph-mon[112979]: pgmap v357: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:25.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:24 smithi067 ceph-mon[112979]: from='client.34704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:52:25.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:24 smithi067 ceph-mon[112979]: from='client.34708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:52:25.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:24 smithi032 ceph-mon[146817]: pgmap v357: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:25.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:24 smithi032 ceph-mon[146817]: from='client.34704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:52:25.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:24 smithi032 ceph-mon[146817]: from='client.34708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:52:25.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.376+0000 7f526b174700 1 -- 172.21.15.32:0/293395160 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f526410f0e0 con 0x7f5264102e70 2024-04-12T05:52:25.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.377+0000 7f52617fa700 1 -- 172.21.15.32:0/293395160 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f52500592f0 con 0x7f5264102e70 2024-04-12T05:52:25.372 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:52:25.372 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:52:25.373 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:52:25.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:52:25.373 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:52:25.373 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:52:25.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:52:25.373 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:52:25.373 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:52:25.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:52:25.373 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:52:25.373 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:52:25.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:52:25.373 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:52:25.373 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:52:25.373 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:52:25.374 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:52:25.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.379+0000 7f526b174700 1 -- 172.21.15.32:0/293395160 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f525406cc30 msgr2=0x7f525406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:25.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.379+0000 7f526b174700 1 --2- 172.21.15.32:0/293395160 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f525406cc30 0x7f525406f0e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f525800b0d0 tx=0x7f5258009e30 comp rx=0 tx=0).stop 2024-04-12T05:52:25.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.379+0000 7f526b174700 1 -- 172.21.15.32:0/293395160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5264102e70 msgr2=0x7f5264113fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:25.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.379+0000 7f526b174700 1 --2- 172.21.15.32:0/293395160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5264102e70 0x7f5264113fd0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f525000eb40 tx=0x7f525000b040 comp rx=0 tx=0).stop 2024-04-12T05:52:25.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.379+0000 7f526b174700 1 -- 172.21.15.32:0/293395160 shutdown_connections 2024-04-12T05:52:25.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.379+0000 7f526b174700 1 --2- 172.21.15.32:0/293395160 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f525406cc30 0x7f525406f0e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:25.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.379+0000 7f526b174700 1 --2- 172.21.15.32:0/293395160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52640fb4a0 0x7f5264113a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:25.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.379+0000 7f526b174700 1 --2- 172.21.15.32:0/293395160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5264102e70 0x7f5264113fd0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:25.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.379+0000 7f526b174700 1 -- 172.21.15.32:0/293395160 >> 172.21.15.32:0/293395160 conn(0x7f52640714c0 msgr2=0x7f52640fa950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:52:25.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.379+0000 7f526b174700 1 -- 172.21.15.32:0/293395160 shutdown_connections 2024-04-12T05:52:25.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.379+0000 7f526b174700 1 -- 172.21.15.32:0/293395160 wait complete. 2024-04-12T05:52:25.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.482+0000 7f17ba04a700 1 -- 172.21.15.32:0/607491976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17b41037b0 msgr2=0x7f17b40fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:25.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.482+0000 7f17ba04a700 1 --2- 172.21.15.32:0/607491976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17b41037b0 0x7f17b40fce20 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f179c004940 tx=0x7f179c009e30 comp rx=0 tx=0).stop 2024-04-12T05:52:25.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.482+0000 7f17ba04a700 1 -- 172.21.15.32:0/607491976 shutdown_connections 2024-04-12T05:52:25.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.482+0000 7f17ba04a700 1 --2- 172.21.15.32:0/607491976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17b41037b0 0x7f17b40fce20 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:25.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.482+0000 7f17ba04a700 1 --2- 172.21.15.32:0/607491976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17b4102e10 0x7f17b41031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:25.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.482+0000 7f17ba04a700 1 -- 172.21.15.32:0/607491976 >> 172.21.15.32:0/607491976 conn(0x7f17b40f89a0 msgr2=0x7f17b40fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:52:25.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.482+0000 7f17ba04a700 1 -- 172.21.15.32:0/607491976 shutdown_connections 2024-04-12T05:52:25.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.482+0000 7f17ba04a700 1 -- 172.21.15.32:0/607491976 wait complete. 2024-04-12T05:52:25.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.483+0000 7f17ba04a700 1 Processor -- start 2024-04-12T05:52:25.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.483+0000 7f17ba04a700 1 -- start start 2024-04-12T05:52:25.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.483+0000 7f17ba04a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17b4102e10 0x7f17b40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:25.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.483+0000 7f17ba04a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17b41037b0 0x7f17b4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:25.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.483+0000 7f17ba04a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b4070d90 con 0x7f17b41037b0 2024-04-12T05:52:25.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.483+0000 7f17ba04a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b4070ed0 con 0x7f17b4102e10 2024-04-12T05:52:25.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.483+0000 7f17b2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17b41037b0 0x7f17b4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:25.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.483+0000 7f17b2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17b41037b0 0x7f17b4070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43578/0 (socket says 172.21.15.32:43578) 2024-04-12T05:52:25.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.483+0000 7f17b2ffd700 1 -- 172.21.15.32:0/3450955516 learned_addr learned my addr 172.21.15.32:0/3450955516 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:52:25.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.483+0000 7f17b37fe700 1 --2- 172.21.15.32:0/3450955516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17b4102e10 0x7f17b40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:25.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.484+0000 7f17b2ffd700 1 -- 172.21.15.32:0/3450955516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17b4102e10 msgr2=0x7f17b40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:25.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.484+0000 7f17b2ffd700 1 --2- 172.21.15.32:0/3450955516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17b4102e10 0x7f17b40702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:25.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.484+0000 7f17b2ffd700 1 -- 172.21.15.32:0/3450955516 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f179c005040 con 0x7f17b41037b0 2024-04-12T05:52:25.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.484+0000 7f17b2ffd700 1 --2- 172.21.15.32:0/3450955516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17b41037b0 0x7f17b4070800 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f179c009390 tx=0x7f179c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:52:25.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.484+0000 7f17abfff700 1 -- 172.21.15.32:0/3450955516 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f179c00e040 con 0x7f17b41037b0 2024-04-12T05:52:25.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.484+0000 7f17abfff700 1 -- 172.21.15.32:0/3450955516 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f179c004290 con 0x7f17b41037b0 2024-04-12T05:52:25.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.484+0000 7f17ba04a700 1 -- 172.21.15.32:0/3450955516 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17b4071120 con 0x7f17b41037b0 2024-04-12T05:52:25.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.484+0000 7f17abfff700 1 -- 172.21.15.32:0/3450955516 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f179c013e70 con 0x7f17b41037b0 2024-04-12T05:52:25.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.484+0000 7f17ba04a700 1 -- 172.21.15.32:0/3450955516 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17b411a200 con 0x7f17b41037b0 2024-04-12T05:52:25.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.485+0000 7f17ba04a700 1 -- 172.21.15.32:0/3450955516 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17b40fe560 con 0x7f17b41037b0 2024-04-12T05:52:25.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.486+0000 7f17abfff700 1 -- 172.21.15.32:0/3450955516 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f179c013690 con 0x7f17b41037b0 2024-04-12T05:52:25.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.486+0000 7f17abfff700 1 --2- 172.21.15.32:0/3450955516 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f17a006cbc0 0x7f17a006f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:25.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.486+0000 7f17abfff700 1 -- 172.21.15.32:0/3450955516 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f179c08d290 con 0x7f17b41037b0 2024-04-12T05:52:25.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.486+0000 7f17b37fe700 1 --2- 172.21.15.32:0/3450955516 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f17a006cbc0 0x7f17a006f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:25.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.486+0000 7f17b37fe700 1 --2- 172.21.15.32:0/3450955516 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f17a006cbc0 0x7f17a006f070 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f17a4004f40 tx=0x7f17a400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:52:25.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.490+0000 7f17abfff700 1 -- 172.21.15.32:0/3450955516 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f179c057320 con 0x7f17b41037b0 2024-04-12T05:52:25.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.738+0000 7f17ba04a700 1 -- 172.21.15.32:0/3450955516 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f17b4109de0 con 0x7f17b41037b0 2024-04-12T05:52:25.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.738+0000 7f17abfff700 1 -- 172.21.15.32:0/3450955516 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1835 (secure 0 0 0) 0x7f179c013420 con 0x7f17b41037b0 2024-04-12T05:52:25.734 INFO:teuthology.orchestra.run.smithi032.stdout:e33 2024-04-12T05:52:25.734 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:52:25.734 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:52:25.734 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:52:25.734 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:52:25.734 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:52:25.735 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:52:25.735 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 24 2024-04-12T05:52:25.735 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:52:25.735 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:52:25.735 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:46:10.668569+0000 2024-04-12T05:52:25.735 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:52:25.735 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:52:25.735 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:52:25.735 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:52:25.735 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:52:25.735 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:52:25.735 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:52:25.735 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:52:25.735 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T05:52:25.735 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:52:25.736 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:52:25.736 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:52:25.736 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T05:52:25.736 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-12T05:52:25.736 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:52:25.736 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:52:25.736 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:52:25.736 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:52:25.736 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:52:25.736 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:52:25.736 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:52:25.736 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:52:25.736 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:52:25.737 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:52:25.737 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-12T05:52:25.737 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:52:25.737 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:52:25.737 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:52:25.737 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{-1:44317} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:52:25.737 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:52:25.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.740+0000 7f17ba04a700 1 -- 172.21.15.32:0/3450955516 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f17a006cbc0 msgr2=0x7f17a006f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:25.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.740+0000 7f17ba04a700 1 --2- 172.21.15.32:0/3450955516 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f17a006cbc0 0x7f17a006f070 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f17a4004f40 tx=0x7f17a400b040 comp rx=0 tx=0).stop 2024-04-12T05:52:25.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.740+0000 7f17ba04a700 1 -- 172.21.15.32:0/3450955516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17b41037b0 msgr2=0x7f17b4070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:25.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.740+0000 7f17ba04a700 1 --2- 172.21.15.32:0/3450955516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17b41037b0 0x7f17b4070800 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f179c009390 tx=0x7f179c009e30 comp rx=0 tx=0).stop 2024-04-12T05:52:25.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.740+0000 7f17ba04a700 1 -- 172.21.15.32:0/3450955516 shutdown_connections 2024-04-12T05:52:25.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.740+0000 7f17ba04a700 1 --2- 172.21.15.32:0/3450955516 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f17a006cbc0 0x7f17a006f070 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:25.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.740+0000 7f17ba04a700 1 --2- 172.21.15.32:0/3450955516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17b4102e10 0x7f17b40702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:25.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.740+0000 7f17ba04a700 1 --2- 172.21.15.32:0/3450955516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17b41037b0 0x7f17b4070800 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:25.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.740+0000 7f17ba04a700 1 -- 172.21.15.32:0/3450955516 >> 172.21.15.32:0/3450955516 conn(0x7f17b40f89a0 msgr2=0x7f17b40f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:52:25.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.740+0000 7f17ba04a700 1 -- 172.21.15.32:0/3450955516 shutdown_connections 2024-04-12T05:52:25.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.741+0000 7f17ba04a700 1 -- 172.21.15.32:0/3450955516 wait complete. 2024-04-12T05:52:25.739 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-12T05:52:25.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.844+0000 7f6b4741d700 1 -- 172.21.15.32:0/15063125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b40102710 msgr2=0x7f6b400fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:25.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.844+0000 7f6b4741d700 1 --2- 172.21.15.32:0/15063125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b40102710 0x7f6b400fbd80 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6b34004940 tx=0x7f6b34009e30 comp rx=0 tx=0).stop 2024-04-12T05:52:25.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.844+0000 7f6b4741d700 1 -- 172.21.15.32:0/15063125 shutdown_connections 2024-04-12T05:52:25.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.844+0000 7f6b4741d700 1 --2- 172.21.15.32:0/15063125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b40102710 0x7f6b400fbd80 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:25.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.844+0000 7f6b4741d700 1 --2- 172.21.15.32:0/15063125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b40101d70 0x7f6b40102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:25.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.844+0000 7f6b4741d700 1 -- 172.21.15.32:0/15063125 >> 172.21.15.32:0/15063125 conn(0x7f6b40072bc0 msgr2=0x7f6b40072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:52:25.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.844+0000 7f6b4741d700 1 -- 172.21.15.32:0/15063125 shutdown_connections 2024-04-12T05:52:25.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.844+0000 7f6b4741d700 1 -- 172.21.15.32:0/15063125 wait complete. 2024-04-12T05:52:25.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.845+0000 7f6b4741d700 1 Processor -- start 2024-04-12T05:52:25.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.845+0000 7f6b4741d700 1 -- start start 2024-04-12T05:52:25.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.845+0000 7f6b4741d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b40101d70 0x7f6b40113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:25.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.845+0000 7f6b4741d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b40102710 0x7f6b40114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:25.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.845+0000 7f6b4741d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b40114670 con 0x7f6b40101d70 2024-04-12T05:52:25.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.845+0000 7f6b4741d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4010da60 con 0x7f6b40102710 2024-04-12T05:52:25.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.845+0000 7f6b451b9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b40101d70 0x7f6b40113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:25.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.845+0000 7f6b451b9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b40101d70 0x7f6b40113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43606/0 (socket says 172.21.15.32:43606) 2024-04-12T05:52:25.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.845+0000 7f6b451b9700 1 -- 172.21.15.32:0/73664839 learned_addr learned my addr 172.21.15.32:0/73664839 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:52:25.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.845+0000 7f6b449b8700 1 --2- 172.21.15.32:0/73664839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b40102710 0x7f6b40114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:25.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.845+0000 7f6b451b9700 1 -- 172.21.15.32:0/73664839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b40102710 msgr2=0x7f6b40114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:25.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.845+0000 7f6b451b9700 1 --2- 172.21.15.32:0/73664839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b40102710 0x7f6b40114020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:25.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.845+0000 7f6b451b9700 1 -- 172.21.15.32:0/73664839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b34005040 con 0x7f6b40101d70 2024-04-12T05:52:25.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.845+0000 7f6b449b8700 1 --2- 172.21.15.32:0/73664839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b40102710 0x7f6b40114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:52:25.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.846+0000 7f6b451b9700 1 --2- 172.21.15.32:0/73664839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b40101d70 0x7f6b40113ae0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c00c830 tx=0x7f6b3c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:52:25.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.846+0000 7f6b31ffb700 1 -- 172.21.15.32:0/73664839 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b3c00a760 con 0x7f6b40101d70 2024-04-12T05:52:25.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.846+0000 7f6b31ffb700 1 -- 172.21.15.32:0/73664839 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6b3c00ad70 con 0x7f6b40101d70 2024-04-12T05:52:25.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.846+0000 7f6b4741d700 1 -- 172.21.15.32:0/73664839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b4010dd40 con 0x7f6b40101d70 2024-04-12T05:52:25.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.846+0000 7f6b31ffb700 1 -- 172.21.15.32:0/73664839 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b3c0113a0 con 0x7f6b40101d70 2024-04-12T05:52:25.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.846+0000 7f6b4741d700 1 -- 172.21.15.32:0/73664839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b4010e300 con 0x7f6b40101d70 2024-04-12T05:52:25.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.847+0000 7f6b4741d700 1 -- 172.21.15.32:0/73664839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b400fd4c0 con 0x7f6b40101d70 2024-04-12T05:52:25.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.847+0000 7f6b31ffb700 1 -- 172.21.15.32:0/73664839 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f6b3c012400 con 0x7f6b40101d70 2024-04-12T05:52:25.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.848+0000 7f6b31ffb700 1 --2- 172.21.15.32:0/73664839 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6b2c06cc20 0x7f6b2c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:25.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.848+0000 7f6b31ffb700 1 -- 172.21.15.32:0/73664839 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f6b3c08c110 con 0x7f6b40101d70 2024-04-12T05:52:25.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.848+0000 7f6b449b8700 1 --2- 172.21.15.32:0/73664839 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6b2c06cc20 0x7f6b2c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:25.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.848+0000 7f6b449b8700 1 --2- 172.21.15.32:0/73664839 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6b2c06cc20 0x7f6b2c06f0d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6b4010ec20 tx=0x7f6b34009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:52:25.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:25.852+0000 7f6b31ffb700 1 -- 172.21.15.32:0/73664839 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6b3c059b80 con 0x7f6b40101d70 2024-04-12T05:52:26.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.074+0000 7f6b4741d700 1 -- 172.21.15.32:0/73664839 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b40108d10 con 0x7f6b2c06cc20 2024-04-12T05:52:26.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.076+0000 7f6b31ffb700 1 -- 172.21.15.32:0/73664839 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6b40108d10 con 0x7f6b2c06cc20 2024-04-12T05:52:26.071 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:52:26.072 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:52:26.072 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:52:26.072 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:52:26.072 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:52:26.072 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-12T05:52:26.072 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:52:26.072 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-12T05:52:26.072 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:52:26.072 INFO:teuthology.orchestra.run.smithi032.stdout: "mds" 2024-04-12T05:52:26.072 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:52:26.072 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:52:26.072 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:52:26.072 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:52:26.072 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:52:26.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.078+0000 7f6b4741d700 1 -- 172.21.15.32:0/73664839 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6b2c06cc20 msgr2=0x7f6b2c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:26.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.078+0000 7f6b4741d700 1 --2- 172.21.15.32:0/73664839 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6b2c06cc20 0x7f6b2c06f0d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6b4010ec20 tx=0x7f6b34009e30 comp rx=0 tx=0).stop 2024-04-12T05:52:26.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.078+0000 7f6b4741d700 1 -- 172.21.15.32:0/73664839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b40101d70 msgr2=0x7f6b40113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:26.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.078+0000 7f6b4741d700 1 --2- 172.21.15.32:0/73664839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b40101d70 0x7f6b40113ae0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c00c830 tx=0x7f6b3c004470 comp rx=0 tx=0).stop 2024-04-12T05:52:26.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.078+0000 7f6b4741d700 1 -- 172.21.15.32:0/73664839 shutdown_connections 2024-04-12T05:52:26.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.078+0000 7f6b4741d700 1 --2- 172.21.15.32:0/73664839 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6b2c06cc20 0x7f6b2c06f0d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:26.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.078+0000 7f6b4741d700 1 --2- 172.21.15.32:0/73664839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b40101d70 0x7f6b40113ae0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:26.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.078+0000 7f6b4741d700 1 --2- 172.21.15.32:0/73664839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b40102710 0x7f6b40114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:26.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.078+0000 7f6b4741d700 1 -- 172.21.15.32:0/73664839 >> 172.21.15.32:0/73664839 conn(0x7f6b40072bc0 msgr2=0x7f6b40065a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:52:26.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.078+0000 7f6b4741d700 1 -- 172.21.15.32:0/73664839 shutdown_connections 2024-04-12T05:52:26.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.078+0000 7f6b4741d700 1 -- 172.21.15.32:0/73664839 wait complete. 2024-04-12T05:52:26.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.182+0000 7f4df4f1f700 1 -- 172.21.15.32:0/2169816245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4df0102e00 msgr2=0x7f4df01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:26.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.182+0000 7f4df4f1f700 1 --2- 172.21.15.32:0/2169816245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4df0102e00 0x7f4df01031d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4de0004940 tx=0x7f4de0009e30 comp rx=0 tx=0).stop 2024-04-12T05:52:26.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.182+0000 7f4df4f1f700 1 -- 172.21.15.32:0/2169816245 shutdown_connections 2024-04-12T05:52:26.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.182+0000 7f4df4f1f700 1 --2- 172.21.15.32:0/2169816245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4df01037a0 0x7f4df00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:26.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.182+0000 7f4df4f1f700 1 --2- 172.21.15.32:0/2169816245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4df0102e00 0x7f4df01031d0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:26.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.182+0000 7f4df4f1f700 1 -- 172.21.15.32:0/2169816245 >> 172.21.15.32:0/2169816245 conn(0x7f4df00f89b0 msgr2=0x7f4df00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:52:26.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.182+0000 7f4df4f1f700 1 -- 172.21.15.32:0/2169816245 shutdown_connections 2024-04-12T05:52:26.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.182+0000 7f4df4f1f700 1 -- 172.21.15.32:0/2169816245 wait complete. 2024-04-12T05:52:26.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.182+0000 7f4df4f1f700 1 Processor -- start 2024-04-12T05:52:26.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.183+0000 7f4df4f1f700 1 -- start start 2024-04-12T05:52:26.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.183+0000 7f4df4f1f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4df0102e00 0x7f4df0113820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:26.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.183+0000 7f4df4f1f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4df01037a0 0x7f4df0113d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:26.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.183+0000 7f4df4f1f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df01143b0 con 0x7f4df0102e00 2024-04-12T05:52:26.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.183+0000 7f4df4f1f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df010d9b0 con 0x7f4df01037a0 2024-04-12T05:52:26.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.183+0000 7f4dee59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4df0102e00 0x7f4df0113820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:26.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.183+0000 7f4dee59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4df0102e00 0x7f4df0113820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43614/0 (socket says 172.21.15.32:43614) 2024-04-12T05:52:26.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.183+0000 7f4dee59c700 1 -- 172.21.15.32:0/850471198 learned_addr learned my addr 172.21.15.32:0/850471198 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:52:26.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.183+0000 7f4dedd9b700 1 --2- 172.21.15.32:0/850471198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4df01037a0 0x7f4df0113d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:26.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.183+0000 7f4dee59c700 1 -- 172.21.15.32:0/850471198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4df01037a0 msgr2=0x7f4df0113d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:26.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.183+0000 7f4dee59c700 1 --2- 172.21.15.32:0/850471198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4df01037a0 0x7f4df0113d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:26.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.183+0000 7f4dee59c700 1 -- 172.21.15.32:0/850471198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de0005040 con 0x7f4df0102e00 2024-04-12T05:52:26.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.183+0000 7f4dee59c700 1 --2- 172.21.15.32:0/850471198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4df0102e00 0x7f4df0113820 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4de0000c00 tx=0x7f4de0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:52:26.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.184+0000 7f4ddeffd700 1 -- 172.21.15.32:0/850471198 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4de0021070 con 0x7f4df0102e00 2024-04-12T05:52:26.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.184+0000 7f4df4f1f700 1 -- 172.21.15.32:0/850471198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df010dc30 con 0x7f4df0102e00 2024-04-12T05:52:26.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.184+0000 7f4ddeffd700 1 -- 172.21.15.32:0/850471198 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4de0013de0 con 0x7f4df0102e00 2024-04-12T05:52:26.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.184+0000 7f4ddeffd700 1 -- 172.21.15.32:0/850471198 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4de000e070 con 0x7f4df0102e00 2024-04-12T05:52:26.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.184+0000 7f4df4f1f700 1 -- 172.21.15.32:0/850471198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4df010e190 con 0x7f4df0102e00 2024-04-12T05:52:26.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.185+0000 7f4df4f1f700 1 -- 172.21.15.32:0/850471198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df00fe550 con 0x7f4df0102e00 2024-04-12T05:52:26.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.186+0000 7f4ddeffd700 1 -- 172.21.15.32:0/850471198 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4de001c600 con 0x7f4df0102e00 2024-04-12T05:52:26.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.186+0000 7f4ddeffd700 1 --2- 172.21.15.32:0/850471198 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f4dd806cc20 0x7f4dd806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:26.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.186+0000 7f4ddeffd700 1 -- 172.21.15.32:0/850471198 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f4de008d970 con 0x7f4df0102e00 2024-04-12T05:52:26.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.186+0000 7f4dedd9b700 1 --2- 172.21.15.32:0/850471198 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f4dd806cc20 0x7f4dd806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:26.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.187+0000 7f4dedd9b700 1 --2- 172.21.15.32:0/850471198 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f4dd806cc20 0x7f4dd806f0d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4de4004600 tx=0x7f4de400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:52:26.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.190+0000 7f4ddeffd700 1 -- 172.21.15.32:0/850471198 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4de005b480 con 0x7f4df0102e00 2024-04-12T05:52:26.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:25 smithi067 ceph-mon[112979]: from='client.34712 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:52:26.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:25 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/293395160' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:52:26.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:25 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3450955516' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:52:26.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:25 smithi032 ceph-mon[146817]: from='client.34712 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:52:26.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:25 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/293395160' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:52:26.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:25 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3450955516' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:52:26.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.476+0000 7f4df4f1f700 1 -- 172.21.15.32:0/850471198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4df0109e20 con 0x7f4df0102e00 2024-04-12T05:52:26.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.476+0000 7f4ddeffd700 1 -- 172.21.15.32:0/850471198 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f4de005b010 con 0x7f4df0102e00 2024-04-12T05:52:26.472 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:52:26.472 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:52:26.472 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-12T05:52:26.472 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:52:26.472 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:52:26.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.478+0000 7f4df4f1f700 1 -- 172.21.15.32:0/850471198 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f4dd806cc20 msgr2=0x7f4dd806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:26.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.478+0000 7f4df4f1f700 1 --2- 172.21.15.32:0/850471198 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f4dd806cc20 0x7f4dd806f0d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4de4004600 tx=0x7f4de400b040 comp rx=0 tx=0).stop 2024-04-12T05:52:26.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.478+0000 7f4df4f1f700 1 -- 172.21.15.32:0/850471198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4df0102e00 msgr2=0x7f4df0113820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:26.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.478+0000 7f4df4f1f700 1 --2- 172.21.15.32:0/850471198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4df0102e00 0x7f4df0113820 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4de0000c00 tx=0x7f4de0009e30 comp rx=0 tx=0).stop 2024-04-12T05:52:26.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.478+0000 7f4df4f1f700 1 -- 172.21.15.32:0/850471198 shutdown_connections 2024-04-12T05:52:26.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.478+0000 7f4df4f1f700 1 --2- 172.21.15.32:0/850471198 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f4dd806cc20 0x7f4dd806f0d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:26.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.478+0000 7f4df4f1f700 1 --2- 172.21.15.32:0/850471198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4df0102e00 0x7f4df0113820 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:26.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.478+0000 7f4df4f1f700 1 --2- 172.21.15.32:0/850471198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4df01037a0 0x7f4df0113d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:26.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.478+0000 7f4df4f1f700 1 -- 172.21.15.32:0/850471198 >> 172.21.15.32:0/850471198 conn(0x7f4df00f89b0 msgr2=0x7f4df00f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:52:26.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.478+0000 7f4df4f1f700 1 -- 172.21.15.32:0/850471198 shutdown_connections 2024-04-12T05:52:26.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:26.478+0000 7f4df4f1f700 1 -- 172.21.15.32:0/850471198 wait complete. 2024-04-12T05:52:27.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:26 smithi067 ceph-mon[112979]: pgmap v358: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:27.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:26 smithi067 ceph-mon[112979]: from='client.34724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:52:27.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:26 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/850471198' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:52:27.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:26 smithi032 ceph-mon[146817]: pgmap v358: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:27.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:26 smithi032 ceph-mon[146817]: from='client.34724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:52:27.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:26 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/850471198' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:52:29.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:29 smithi067 ceph-mon[112979]: pgmap v359: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:29.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:29 smithi032 ceph-mon[146817]: pgmap v359: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:31.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:31 smithi067 ceph-mon[112979]: pgmap v360: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:31.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:31 smithi032 ceph-mon[146817]: pgmap v360: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:33.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:33 smithi067 ceph-mon[112979]: pgmap v361: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:33.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:33 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:52:33.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:33 smithi032 ceph-mon[146817]: pgmap v361: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:33.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:33 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:52:35.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:35 smithi067 ceph-mon[112979]: pgmap v362: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:35.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:35 smithi032 ceph-mon[146817]: pgmap v362: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:37.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:37 smithi067 ceph-mon[112979]: pgmap v363: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:37.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:37 smithi032 ceph-mon[146817]: pgmap v363: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:39.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:39 smithi067 ceph-mon[112979]: pgmap v364: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:39.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:39 smithi032 ceph-mon[146817]: pgmap v364: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:41.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:41 smithi067 ceph-mon[112979]: pgmap v365: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:41.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:41 smithi032 ceph-mon[146817]: pgmap v365: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:43.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:43 smithi032 ceph-mon[146817]: pgmap v366: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:43.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:43 smithi067 ceph-mon[112979]: pgmap v366: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:45.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:45 smithi032 ceph-mon[146817]: pgmap v367: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:45.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:45 smithi067 ceph-mon[112979]: pgmap v367: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:47.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:47 smithi032 ceph-mon[146817]: pgmap v368: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:47.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:47 smithi067 ceph-mon[112979]: pgmap v368: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:49.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:48 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:52:49.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:48 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:52:50.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:49 smithi067 ceph-mon[112979]: pgmap v369: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:50.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:49 smithi032 ceph-mon[146817]: pgmap v369: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:52.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:51 smithi067 ceph-mon[112979]: pgmap v370: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:52.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:51 smithi032 ceph-mon[146817]: pgmap v370: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:54.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:53 smithi067 ceph-mon[112979]: pgmap v371: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:54.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:53 smithi032 ceph-mon[146817]: pgmap v371: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:56.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:55 smithi067 ceph-mon[112979]: pgmap v372: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:56.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:55 smithi032 ceph-mon[146817]: pgmap v372: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:56.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.596+0000 7f7f7a85c700 1 -- 172.21.15.32:0/1360841829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f741037a0 msgr2=0x7f7f740fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:56.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.596+0000 7f7f7a85c700 1 --2- 172.21.15.32:0/1360841829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f741037a0 0x7f7f740fce10 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c004940 tx=0x7f7f5c009e30 comp rx=0 tx=0).stop 2024-04-12T05:52:56.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.596+0000 7f7f7a85c700 1 -- 172.21.15.32:0/1360841829 shutdown_connections 2024-04-12T05:52:56.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.596+0000 7f7f7a85c700 1 --2- 172.21.15.32:0/1360841829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f741037a0 0x7f7f740fce10 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:56.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.596+0000 7f7f7a85c700 1 --2- 172.21.15.32:0/1360841829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f74102e00 0x7f7f741031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:56.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.596+0000 7f7f7a85c700 1 -- 172.21.15.32:0/1360841829 >> 172.21.15.32:0/1360841829 conn(0x7f7f740f89b0 msgr2=0x7f7f740fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:52:56.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.596+0000 7f7f7a85c700 1 -- 172.21.15.32:0/1360841829 shutdown_connections 2024-04-12T05:52:56.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.596+0000 7f7f7a85c700 1 -- 172.21.15.32:0/1360841829 wait complete. 2024-04-12T05:52:56.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.597+0000 7f7f7a85c700 1 Processor -- start 2024-04-12T05:52:56.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.597+0000 7f7f7a85c700 1 -- start start 2024-04-12T05:52:56.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.597+0000 7f7f7a85c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f74102e00 0x7f7f74113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:56.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.597+0000 7f7f7a85c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f741037a0 0x7f7f74114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:56.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.597+0000 7f7f7a85c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f74114650 con 0x7f7f741037a0 2024-04-12T05:52:56.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.597+0000 7f7f7a85c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7410da40 con 0x7f7f74102e00 2024-04-12T05:52:56.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.598+0000 7f7f737fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f741037a0 0x7f7f74114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:56.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.598+0000 7f7f737fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f741037a0 0x7f7f74114000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50028/0 (socket says 172.21.15.32:50028) 2024-04-12T05:52:56.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.598+0000 7f7f737fe700 1 -- 172.21.15.32:0/3989112823 learned_addr learned my addr 172.21.15.32:0/3989112823 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:52:56.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.598+0000 7f7f73fff700 1 --2- 172.21.15.32:0/3989112823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f74102e00 0x7f7f74113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:56.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.598+0000 7f7f737fe700 1 -- 172.21.15.32:0/3989112823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f74102e00 msgr2=0x7f7f74113ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:56.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.598+0000 7f7f737fe700 1 --2- 172.21.15.32:0/3989112823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f74102e00 0x7f7f74113ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:56.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.598+0000 7f7f737fe700 1 -- 172.21.15.32:0/3989112823 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5c005040 con 0x7f7f741037a0 2024-04-12T05:52:56.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.598+0000 7f7f73fff700 1 --2- 172.21.15.32:0/3989112823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f74102e00 0x7f7f74113ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:52:56.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.598+0000 7f7f737fe700 1 --2- 172.21.15.32:0/3989112823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f741037a0 0x7f7f74114000 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c009390 tx=0x7f7f5c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:52:56.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.598+0000 7f7f70ff9700 1 -- 172.21.15.32:0/3989112823 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f5c00e040 con 0x7f7f741037a0 2024-04-12T05:52:56.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.598+0000 7f7f70ff9700 1 -- 172.21.15.32:0/3989112823 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f5c013420 con 0x7f7f741037a0 2024-04-12T05:52:56.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.598+0000 7f7f7a85c700 1 -- 172.21.15.32:0/3989112823 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f7410dcc0 con 0x7f7f741037a0 2024-04-12T05:52:56.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.598+0000 7f7f70ff9700 1 -- 172.21.15.32:0/3989112823 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f5c01c630 con 0x7f7f741037a0 2024-04-12T05:52:56.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.598+0000 7f7f7a85c700 1 -- 172.21.15.32:0/3989112823 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f7410e280 con 0x7f7f741037a0 2024-04-12T05:52:56.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.599+0000 7f7f7a85c700 1 -- 172.21.15.32:0/3989112823 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f740fe550 con 0x7f7f741037a0 2024-04-12T05:52:56.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.600+0000 7f7f70ff9700 1 -- 172.21.15.32:0/3989112823 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f7f5c0139e0 con 0x7f7f741037a0 2024-04-12T05:52:56.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.600+0000 7f7f70ff9700 1 --2- 172.21.15.32:0/3989112823 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7f6006cc30 0x7f7f6006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:56.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.600+0000 7f7f70ff9700 1 -- 172.21.15.32:0/3989112823 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f7f5c08d350 con 0x7f7f741037a0 2024-04-12T05:52:56.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.600+0000 7f7f73fff700 1 --2- 172.21.15.32:0/3989112823 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7f6006cc30 0x7f7f6006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:56.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.601+0000 7f7f73fff700 1 --2- 172.21.15.32:0/3989112823 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7f6006cc30 0x7f7f6006f0e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7f64007fd0 tx=0x7f7f64009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:52:56.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.605+0000 7f7f70ff9700 1 -- 172.21.15.32:0/3989112823 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7f5c05adc0 con 0x7f7f741037a0 2024-04-12T05:52:56.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.826+0000 7f7f7a85c700 1 -- 172.21.15.32:0/3989112823 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f74109da0 con 0x7f7f6006cc30 2024-04-12T05:52:56.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.827+0000 7f7f70ff9700 1 -- 172.21.15.32:0/3989112823 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7f74109da0 con 0x7f7f6006cc30 2024-04-12T05:52:56.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.829+0000 7f7f7a85c700 1 -- 172.21.15.32:0/3989112823 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7f6006cc30 msgr2=0x7f7f6006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:56.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.829+0000 7f7f7a85c700 1 --2- 172.21.15.32:0/3989112823 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7f6006cc30 0x7f7f6006f0e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7f64007fd0 tx=0x7f7f64009040 comp rx=0 tx=0).stop 2024-04-12T05:52:56.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.829+0000 7f7f7a85c700 1 -- 172.21.15.32:0/3989112823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f741037a0 msgr2=0x7f7f74114000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:56.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.829+0000 7f7f7a85c700 1 --2- 172.21.15.32:0/3989112823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f741037a0 0x7f7f74114000 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c009390 tx=0x7f7f5c009e30 comp rx=0 tx=0).stop 2024-04-12T05:52:56.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.830+0000 7f7f7a85c700 1 -- 172.21.15.32:0/3989112823 shutdown_connections 2024-04-12T05:52:56.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.830+0000 7f7f7a85c700 1 --2- 172.21.15.32:0/3989112823 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7f6006cc30 0x7f7f6006f0e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:56.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.830+0000 7f7f7a85c700 1 --2- 172.21.15.32:0/3989112823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f74102e00 0x7f7f74113ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:56.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.830+0000 7f7f7a85c700 1 --2- 172.21.15.32:0/3989112823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f741037a0 0x7f7f74114000 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:56.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.830+0000 7f7f7a85c700 1 -- 172.21.15.32:0/3989112823 >> 172.21.15.32:0/3989112823 conn(0x7f7f740f89b0 msgr2=0x7f7f740f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:52:56.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.830+0000 7f7f7a85c700 1 -- 172.21.15.32:0/3989112823 shutdown_connections 2024-04-12T05:52:56.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.830+0000 7f7f7a85c700 1 -- 172.21.15.32:0/3989112823 wait complete. 2024-04-12T05:52:56.835 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:52:56.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.934+0000 7fbec14d0700 1 -- 172.21.15.32:0/257479009 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbebc070300 msgr2=0x7fbebc0706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:56.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.934+0000 7fbec14d0700 1 --2- 172.21.15.32:0/257479009 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbebc070300 0x7fbebc0706d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fbeac004970 tx=0x7fbeac009e30 comp rx=0 tx=0).stop 2024-04-12T05:52:56.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.934+0000 7fbec14d0700 1 -- 172.21.15.32:0/257479009 shutdown_connections 2024-04-12T05:52:56.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.934+0000 7fbec14d0700 1 --2- 172.21.15.32:0/257479009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbebc070c10 0x7fbebc109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:56.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.934+0000 7fbec14d0700 1 --2- 172.21.15.32:0/257479009 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbebc070300 0x7fbebc0706d0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:56.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.934+0000 7fbec14d0700 1 -- 172.21.15.32:0/257479009 >> 172.21.15.32:0/257479009 conn(0x7fbebc0f89b0 msgr2=0x7fbebc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:52:56.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.934+0000 7fbec14d0700 1 -- 172.21.15.32:0/257479009 shutdown_connections 2024-04-12T05:52:56.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.934+0000 7fbec14d0700 1 -- 172.21.15.32:0/257479009 wait complete. 2024-04-12T05:52:56.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.935+0000 7fbec14d0700 1 Processor -- start 2024-04-12T05:52:56.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.935+0000 7fbec14d0700 1 -- start start 2024-04-12T05:52:56.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.935+0000 7fbec14d0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbebc070300 0x7fbebc115a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:56.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.935+0000 7fbec14d0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbebc070c10 0x7fbebc115fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:56.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.935+0000 7fbec14d0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc1166a0 con 0x7fbebc070c10 2024-04-12T05:52:56.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.935+0000 7fbec14d0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc10fc30 con 0x7fbebc070300 2024-04-12T05:52:56.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.935+0000 7fbeba7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbebc070c10 0x7fbebc115fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:56.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.935+0000 7fbeba7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbebc070c10 0x7fbebc115fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50044/0 (socket says 172.21.15.32:50044) 2024-04-12T05:52:56.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.936+0000 7fbeba7fc700 1 -- 172.21.15.32:0/1842136680 learned_addr learned my addr 172.21.15.32:0/1842136680 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:52:56.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.936+0000 7fbebaffd700 1 --2- 172.21.15.32:0/1842136680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbebc070300 0x7fbebc115a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:56.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.936+0000 7fbeba7fc700 1 -- 172.21.15.32:0/1842136680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbebc070300 msgr2=0x7fbebc115a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:56.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.936+0000 7fbeba7fc700 1 --2- 172.21.15.32:0/1842136680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbebc070300 0x7fbebc115a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:56.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.936+0000 7fbeba7fc700 1 -- 172.21.15.32:0/1842136680 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeac005040 con 0x7fbebc070c10 2024-04-12T05:52:56.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.936+0000 7fbeba7fc700 1 --2- 172.21.15.32:0/1842136680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbebc070c10 0x7fbebc115fc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fbeb000b790 tx=0x7fbeb0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:52:56.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.936+0000 7fbea37fe700 1 -- 172.21.15.32:0/1842136680 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbeb000f770 con 0x7fbebc070c10 2024-04-12T05:52:56.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.936+0000 7fbec14d0700 1 -- 172.21.15.32:0/1842136680 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbebc10ff10 con 0x7fbebc070c10 2024-04-12T05:52:56.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.936+0000 7fbea37fe700 1 -- 172.21.15.32:0/1842136680 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbeb000fd80 con 0x7fbebc070c10 2024-04-12T05:52:56.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.936+0000 7fbea37fe700 1 -- 172.21.15.32:0/1842136680 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbeb000d310 con 0x7fbebc070c10 2024-04-12T05:52:56.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.936+0000 7fbec14d0700 1 -- 172.21.15.32:0/1842136680 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbebc110450 con 0x7fbebc070c10 2024-04-12T05:52:56.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.937+0000 7fbec14d0700 1 -- 172.21.15.32:0/1842136680 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbebc071a40 con 0x7fbebc070c10 2024-04-12T05:52:56.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.938+0000 7fbea37fe700 1 -- 172.21.15.32:0/1842136680 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fbeb0010420 con 0x7fbebc070c10 2024-04-12T05:52:56.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.938+0000 7fbea37fe700 1 --2- 172.21.15.32:0/1842136680 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbea406cc20 0x7fbea406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:56.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.938+0000 7fbea37fe700 1 -- 172.21.15.32:0/1842136680 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fbeb008c0b0 con 0x7fbebc070c10 2024-04-12T05:52:56.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.938+0000 7fbebaffd700 1 --2- 172.21.15.32:0/1842136680 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbea406cc20 0x7fbea406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:56.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.939+0000 7fbebaffd700 1 --2- 172.21.15.32:0/1842136680 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbea406cc20 0x7fbea406f0d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbeac004540 tx=0x7fbeac009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:52:56.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:56.943+0000 7fbea37fe700 1 -- 172.21.15.32:0/1842136680 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbeb0056090 con 0x7fbebc070c10 2024-04-12T05:52:57.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.163+0000 7fbec14d0700 1 -- 172.21.15.32:0/1842136680 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbebc10c030 con 0x7fbea406cc20 2024-04-12T05:52:57.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.164+0000 7fbea37fe700 1 -- 172.21.15.32:0/1842136680 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fbebc10c030 con 0x7fbea406cc20 2024-04-12T05:52:57.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.166+0000 7fbec14d0700 1 -- 172.21.15.32:0/1842136680 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbea406cc20 msgr2=0x7fbea406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:57.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.166+0000 7fbec14d0700 1 --2- 172.21.15.32:0/1842136680 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbea406cc20 0x7fbea406f0d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbeac004540 tx=0x7fbeac009e30 comp rx=0 tx=0).stop 2024-04-12T05:52:57.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.166+0000 7fbec14d0700 1 -- 172.21.15.32:0/1842136680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbebc070c10 msgr2=0x7fbebc115fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:57.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.166+0000 7fbec14d0700 1 --2- 172.21.15.32:0/1842136680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbebc070c10 0x7fbebc115fc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fbeb000b790 tx=0x7fbeb0009250 comp rx=0 tx=0).stop 2024-04-12T05:52:57.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.166+0000 7fbec14d0700 1 -- 172.21.15.32:0/1842136680 shutdown_connections 2024-04-12T05:52:57.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.166+0000 7fbec14d0700 1 --2- 172.21.15.32:0/1842136680 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbea406cc20 0x7fbea406f0d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:57.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.166+0000 7fbec14d0700 1 --2- 172.21.15.32:0/1842136680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbebc070300 0x7fbebc115a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:57.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.166+0000 7fbec14d0700 1 --2- 172.21.15.32:0/1842136680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbebc070c10 0x7fbebc115fc0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:57.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.166+0000 7fbec14d0700 1 -- 172.21.15.32:0/1842136680 >> 172.21.15.32:0/1842136680 conn(0x7fbebc0f89b0 msgr2=0x7fbebc0f9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:52:57.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.166+0000 7fbec14d0700 1 -- 172.21.15.32:0/1842136680 shutdown_connections 2024-04-12T05:52:57.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.166+0000 7fbec14d0700 1 -- 172.21.15.32:0/1842136680 wait complete. 2024-04-12T05:52:57.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.269+0000 7f40f2014700 1 -- 172.21.15.32:0/1084523914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40ec102e70 msgr2=0x7f40ec103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:57.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.269+0000 7f40f2014700 1 --2- 172.21.15.32:0/1084523914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40ec102e70 0x7f40ec103240 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f40d4004940 tx=0x7f40d4009e30 comp rx=0 tx=0).stop 2024-04-12T05:52:57.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.270+0000 7f40f2014700 1 -- 172.21.15.32:0/1084523914 shutdown_connections 2024-04-12T05:52:57.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.270+0000 7f40f2014700 1 --2- 172.21.15.32:0/1084523914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ec0fb4a0 0x7f40ec0fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:57.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.270+0000 7f40f2014700 1 --2- 172.21.15.32:0/1084523914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40ec102e70 0x7f40ec103240 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:57.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.270+0000 7f40f2014700 1 -- 172.21.15.32:0/1084523914 >> 172.21.15.32:0/1084523914 conn(0x7f40ec0714c0 msgr2=0x7f40ec0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:52:57.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.270+0000 7f40f2014700 1 -- 172.21.15.32:0/1084523914 shutdown_connections 2024-04-12T05:52:57.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.270+0000 7f40f2014700 1 -- 172.21.15.32:0/1084523914 wait complete. 2024-04-12T05:52:57.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.270+0000 7f40f2014700 1 Processor -- start 2024-04-12T05:52:57.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.270+0000 7f40f2014700 1 -- start start 2024-04-12T05:52:57.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.271+0000 7f40f2014700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ec0fb4a0 0x7f40ec113b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:57.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.271+0000 7f40f2014700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40ec102e70 0x7f40ec114090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:57.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.271+0000 7f40f2014700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ec1146e0 con 0x7f40ec102e70 2024-04-12T05:52:57.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.271+0000 7f40f2014700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ec10dad0 con 0x7f40ec0fb4a0 2024-04-12T05:52:57.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.271+0000 7f40eaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40ec102e70 0x7f40ec114090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:57.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.271+0000 7f40eaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40ec102e70 0x7f40ec114090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50062/0 (socket says 172.21.15.32:50062) 2024-04-12T05:52:57.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.271+0000 7f40eaffd700 1 -- 172.21.15.32:0/3229125037 learned_addr learned my addr 172.21.15.32:0/3229125037 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:52:57.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.271+0000 7f40eb7fe700 1 --2- 172.21.15.32:0/3229125037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ec0fb4a0 0x7f40ec113b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:57.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.271+0000 7f40eaffd700 1 -- 172.21.15.32:0/3229125037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ec0fb4a0 msgr2=0x7f40ec113b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:57.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.271+0000 7f40eaffd700 1 --2- 172.21.15.32:0/3229125037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ec0fb4a0 0x7f40ec113b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:57.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.271+0000 7f40eaffd700 1 -- 172.21.15.32:0/3229125037 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d4005040 con 0x7f40ec102e70 2024-04-12T05:52:57.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.271+0000 7f40eaffd700 1 --2- 172.21.15.32:0/3229125037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40ec102e70 0x7f40ec114090 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f40dc00c830 tx=0x7f40dc004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:52:57.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.271+0000 7f40e3fff700 1 -- 172.21.15.32:0/3229125037 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40dc00a760 con 0x7f40ec102e70 2024-04-12T05:52:57.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.271+0000 7f40e3fff700 1 -- 172.21.15.32:0/3229125037 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f40dc00ad70 con 0x7f40ec102e70 2024-04-12T05:52:57.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.271+0000 7f40f2014700 1 -- 172.21.15.32:0/3229125037 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40ec10ddb0 con 0x7f40ec102e70 2024-04-12T05:52:57.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.271+0000 7f40e3fff700 1 -- 172.21.15.32:0/3229125037 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40dc0113a0 con 0x7f40ec102e70 2024-04-12T05:52:57.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.271+0000 7f40f2014700 1 -- 172.21.15.32:0/3229125037 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40ec10e2f0 con 0x7f40ec102e70 2024-04-12T05:52:57.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.273+0000 7f40f2014700 1 -- 172.21.15.32:0/3229125037 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40ec0653a0 con 0x7f40ec102e70 2024-04-12T05:52:57.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.273+0000 7f40e3fff700 1 -- 172.21.15.32:0/3229125037 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f40dc012400 con 0x7f40ec102e70 2024-04-12T05:52:57.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.273+0000 7f40e3fff700 1 --2- 172.21.15.32:0/3229125037 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f40d806cc30 0x7f40d806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:57.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.273+0000 7f40e3fff700 1 -- 172.21.15.32:0/3229125037 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f40dc08c070 con 0x7f40ec102e70 2024-04-12T05:52:57.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.273+0000 7f40eb7fe700 1 --2- 172.21.15.32:0/3229125037 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f40d806cc30 0x7f40d806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:57.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.274+0000 7f40eb7fe700 1 --2- 172.21.15.32:0/3229125037 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f40d806cc30 0x7f40d806f0e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f40d400b3b0 tx=0x7f40d4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:52:57.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.278+0000 7f40e3fff700 1 -- 172.21.15.32:0/3229125037 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f40dc059b80 con 0x7f40ec102e70 2024-04-12T05:52:57.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.488+0000 7f40f2014700 1 -- 172.21.15.32:0/3229125037 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f40ec113820 con 0x7f40d806cc30 2024-04-12T05:52:57.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.498+0000 7f40e3fff700 1 -- 172.21.15.32:0/3229125037 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f40ec113820 con 0x7f40d806cc30 2024-04-12T05:52:57.493 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:52:57.493 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (12m) 6m ago 24m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:52:57.493 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (10m) 6m ago 24m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:52:57.493 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (10m) 5m ago 23m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:52:57.493 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (12m) 6m ago 23m 87.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:52:57.493 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (6m) 6m ago 18m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T05:52:57.493 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (6m) 6m ago 18m 13.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T05:52:57.493 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (5m) 5m ago 18m 11.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 6d4aca6f74a0 2024-04-12T05:52:57.494 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (6m) 5m ago 18m 14.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T05:52:57.494 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (14m) 6m ago 25m 507M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:52:57.494 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (13m) 5m ago 23m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:52:57.494 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (11m) 6m ago 25m 53.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:52:57.494 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (10m) 5m ago 23m 43.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:52:57.494 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (13m) 6m ago 24m 16.9M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:52:57.494 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (13m) 5m ago 23m 18.4M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:52:57.494 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (9m) 6m ago 22m 91.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T05:52:57.494 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (9m) 6m ago 21m 97.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T05:52:57.494 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (8m) 6m ago 21m 81.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T05:52:57.494 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (8m) 5m ago 20m 113M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T05:52:57.494 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (7m) 5m ago 20m 90.3M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T05:52:57.494 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (6m) 5m ago 20m 84.8M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T05:52:57.494 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (13m) 6m ago 23m 64.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:52:57.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.500+0000 7f40f2014700 1 -- 172.21.15.32:0/3229125037 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f40d806cc30 msgr2=0x7f40d806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:57.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.500+0000 7f40f2014700 1 --2- 172.21.15.32:0/3229125037 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f40d806cc30 0x7f40d806f0e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f40d400b3b0 tx=0x7f40d4009e30 comp rx=0 tx=0).stop 2024-04-12T05:52:57.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.500+0000 7f40f2014700 1 -- 172.21.15.32:0/3229125037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40ec102e70 msgr2=0x7f40ec114090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:57.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.500+0000 7f40f2014700 1 --2- 172.21.15.32:0/3229125037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40ec102e70 0x7f40ec114090 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f40dc00c830 tx=0x7f40dc004470 comp rx=0 tx=0).stop 2024-04-12T05:52:57.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.500+0000 7f40f2014700 1 -- 172.21.15.32:0/3229125037 shutdown_connections 2024-04-12T05:52:57.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.500+0000 7f40f2014700 1 --2- 172.21.15.32:0/3229125037 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f40d806cc30 0x7f40d806f0e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:57.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.500+0000 7f40f2014700 1 --2- 172.21.15.32:0/3229125037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ec0fb4a0 0x7f40ec113b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:57.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.500+0000 7f40f2014700 1 --2- 172.21.15.32:0/3229125037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40ec102e70 0x7f40ec114090 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:57.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.500+0000 7f40f2014700 1 -- 172.21.15.32:0/3229125037 >> 172.21.15.32:0/3229125037 conn(0x7f40ec0714c0 msgr2=0x7f40ec0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:52:57.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.501+0000 7f40f2014700 1 -- 172.21.15.32:0/3229125037 shutdown_connections 2024-04-12T05:52:57.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.501+0000 7f40f2014700 1 -- 172.21.15.32:0/3229125037 wait complete. 2024-04-12T05:52:57.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.604+0000 7f1f3da0b700 1 -- 172.21.15.32:0/3572779907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f380fe830 msgr2=0x7f1f38106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:57.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.604+0000 7f1f3da0b700 1 --2- 172.21.15.32:0/3572779907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f380fe830 0x7f1f38106f00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f1f28004940 tx=0x7f1f28009e30 comp rx=0 tx=0).stop 2024-04-12T05:52:57.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.604+0000 7f1f3da0b700 1 -- 172.21.15.32:0/3572779907 shutdown_connections 2024-04-12T05:52:57.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.604+0000 7f1f3da0b700 1 --2- 172.21.15.32:0/3572779907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f380fe830 0x7f1f38106f00 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:57.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.604+0000 7f1f3da0b700 1 --2- 172.21.15.32:0/3572779907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f380fde90 0x7f1f380fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:57.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.604+0000 7f1f3da0b700 1 -- 172.21.15.32:0/3572779907 >> 172.21.15.32:0/3572779907 conn(0x7f1f38072bc0 msgr2=0x7f1f38072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:52:57.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.604+0000 7f1f3da0b700 1 -- 172.21.15.32:0/3572779907 shutdown_connections 2024-04-12T05:52:57.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.604+0000 7f1f3da0b700 1 -- 172.21.15.32:0/3572779907 wait complete. 2024-04-12T05:52:57.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.605+0000 7f1f3da0b700 1 Processor -- start 2024-04-12T05:52:57.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.605+0000 7f1f3da0b700 1 -- start start 2024-04-12T05:52:57.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.606+0000 7f1f3da0b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f380fde90 0x7f1f38113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:57.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.606+0000 7f1f3da0b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f380fe830 0x7f1f38114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:57.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.606+0000 7f1f3da0b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f38114670 con 0x7f1f380fe830 2024-04-12T05:52:57.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.606+0000 7f1f3da0b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3810da60 con 0x7f1f380fde90 2024-04-12T05:52:57.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.606+0000 7f1f367fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f380fe830 0x7f1f38114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:57.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.606+0000 7f1f367fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f380fe830 0x7f1f38114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50074/0 (socket says 172.21.15.32:50074) 2024-04-12T05:52:57.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.606+0000 7f1f367fc700 1 -- 172.21.15.32:0/1012406681 learned_addr learned my addr 172.21.15.32:0/1012406681 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:52:57.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.606+0000 7f1f36ffd700 1 --2- 172.21.15.32:0/1012406681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f380fde90 0x7f1f38113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:57.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.606+0000 7f1f367fc700 1 -- 172.21.15.32:0/1012406681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f380fde90 msgr2=0x7f1f38113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:57.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.606+0000 7f1f367fc700 1 --2- 172.21.15.32:0/1012406681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f380fde90 0x7f1f38113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:57.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.606+0000 7f1f367fc700 1 -- 172.21.15.32:0/1012406681 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f28005040 con 0x7f1f380fe830 2024-04-12T05:52:57.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.606+0000 7f1f367fc700 1 --2- 172.21.15.32:0/1012406681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f380fe830 0x7f1f38114020 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1f280093b0 tx=0x7f1f28009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:52:57.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.606+0000 7f1f1f7fe700 1 -- 172.21.15.32:0/1012406681 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f2800e040 con 0x7f1f380fe830 2024-04-12T05:52:57.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.606+0000 7f1f1f7fe700 1 -- 172.21.15.32:0/1012406681 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f2800bed0 con 0x7f1f380fe830 2024-04-12T05:52:57.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.606+0000 7f1f3da0b700 1 -- 172.21.15.32:0/1012406681 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f3810dce0 con 0x7f1f380fe830 2024-04-12T05:52:57.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.606+0000 7f1f1f7fe700 1 -- 172.21.15.32:0/1012406681 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f2801c3f0 con 0x7f1f380fe830 2024-04-12T05:52:57.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.606+0000 7f1f3da0b700 1 -- 172.21.15.32:0/1012406681 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f3810e220 con 0x7f1f380fe830 2024-04-12T05:52:57.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.607+0000 7f1f3da0b700 1 -- 172.21.15.32:0/1012406681 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f38065e30 con 0x7f1f380fe830 2024-04-12T05:52:57.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.608+0000 7f1f1f7fe700 1 -- 172.21.15.32:0/1012406681 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1f28013790 con 0x7f1f380fe830 2024-04-12T05:52:57.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.608+0000 7f1f1f7fe700 1 --2- 172.21.15.32:0/1012406681 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1f2006cc30 0x7f1f2006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:57.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.608+0000 7f1f1f7fe700 1 -- 172.21.15.32:0/1012406681 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f1f2808d2c0 con 0x7f1f380fe830 2024-04-12T05:52:57.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.608+0000 7f1f36ffd700 1 --2- 172.21.15.32:0/1012406681 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1f2006cc30 0x7f1f2006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:57.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.609+0000 7f1f36ffd700 1 --2- 172.21.15.32:0/1012406681 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1f2006cc30 0x7f1f2006f0e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c004f40 tx=0x7f1f2c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:52:57.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.613+0000 7f1f1f7fe700 1 -- 172.21.15.32:0/1012406681 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1f2805add0 con 0x7f1f380fe830 2024-04-12T05:52:57.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.899+0000 7f1f3da0b700 1 -- 172.21.15.32:0/1012406681 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1f38109dd0 con 0x7f1f380fe830 2024-04-12T05:52:57.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.900+0000 7f1f1f7fe700 1 -- 172.21.15.32:0/1012406681 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f1f28013420 con 0x7f1f380fe830 2024-04-12T05:52:57.895 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:52:57.895 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:52:57.895 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:52:57.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:52:57.895 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:52:57.895 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:52:57.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:52:57.895 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:52:57.895 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:52:57.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:52:57.895 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:52:57.895 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:52:57.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:52:57.896 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:52:57.896 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:52:57.896 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:52:57.896 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:52:57.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.902+0000 7f1f3da0b700 1 -- 172.21.15.32:0/1012406681 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1f2006cc30 msgr2=0x7f1f2006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:57.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.902+0000 7f1f3da0b700 1 --2- 172.21.15.32:0/1012406681 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1f2006cc30 0x7f1f2006f0e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c004f40 tx=0x7f1f2c00b040 comp rx=0 tx=0).stop 2024-04-12T05:52:57.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.902+0000 7f1f3da0b700 1 -- 172.21.15.32:0/1012406681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f380fe830 msgr2=0x7f1f38114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:57.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.902+0000 7f1f3da0b700 1 --2- 172.21.15.32:0/1012406681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f380fe830 0x7f1f38114020 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1f280093b0 tx=0x7f1f28009e30 comp rx=0 tx=0).stop 2024-04-12T05:52:57.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.902+0000 7f1f3da0b700 1 -- 172.21.15.32:0/1012406681 shutdown_connections 2024-04-12T05:52:57.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.902+0000 7f1f3da0b700 1 --2- 172.21.15.32:0/1012406681 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1f2006cc30 0x7f1f2006f0e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:57.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.902+0000 7f1f3da0b700 1 --2- 172.21.15.32:0/1012406681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f380fde90 0x7f1f38113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:57.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.902+0000 7f1f3da0b700 1 --2- 172.21.15.32:0/1012406681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f380fe830 0x7f1f38114020 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:57.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.902+0000 7f1f3da0b700 1 -- 172.21.15.32:0/1012406681 >> 172.21.15.32:0/1012406681 conn(0x7f1f38072bc0 msgr2=0x7f1f38103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:52:57.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.902+0000 7f1f3da0b700 1 -- 172.21.15.32:0/1012406681 shutdown_connections 2024-04-12T05:52:57.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:57.902+0000 7f1f3da0b700 1 -- 172.21.15.32:0/1012406681 wait complete. 2024-04-12T05:52:58.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.005+0000 7fac1328f700 1 -- 172.21.15.32:0/2120903064 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac0c102e70 msgr2=0x7fac0c103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:58.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.005+0000 7fac1328f700 1 --2- 172.21.15.32:0/2120903064 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac0c102e70 0x7fac0c103240 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fac08004940 tx=0x7fac08009e30 comp rx=0 tx=0).stop 2024-04-12T05:52:58.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.005+0000 7fac1328f700 1 -- 172.21.15.32:0/2120903064 shutdown_connections 2024-04-12T05:52:58.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.005+0000 7fac1328f700 1 --2- 172.21.15.32:0/2120903064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac0c0fb4a0 0x7fac0c0fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:58.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.005+0000 7fac1328f700 1 --2- 172.21.15.32:0/2120903064 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac0c102e70 0x7fac0c103240 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:58.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.005+0000 7fac1328f700 1 -- 172.21.15.32:0/2120903064 >> 172.21.15.32:0/2120903064 conn(0x7fac0c0714c0 msgr2=0x7fac0c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:52:58.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.006+0000 7fac1328f700 1 -- 172.21.15.32:0/2120903064 shutdown_connections 2024-04-12T05:52:58.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.006+0000 7fac1328f700 1 -- 172.21.15.32:0/2120903064 wait complete. 2024-04-12T05:52:58.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.006+0000 7fac1328f700 1 Processor -- start 2024-04-12T05:52:58.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.006+0000 7fac1328f700 1 -- start start 2024-04-12T05:52:58.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.006+0000 7fac1328f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac0c0fb4a0 0x7fac0c0fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:58.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.006+0000 7fac1328f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac0c102e70 0x7fac0c0fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:58.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.006+0000 7fac1328f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0c0fd100 con 0x7fac0c102e70 2024-04-12T05:52:58.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.006+0000 7fac1328f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0c0fd240 con 0x7fac0c0fb4a0 2024-04-12T05:52:58.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.006+0000 7fac1082a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac0c102e70 0x7fac0c0fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:58.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.006+0000 7fac1082a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac0c102e70 0x7fac0c0fcbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50078/0 (socket says 172.21.15.32:50078) 2024-04-12T05:52:58.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.006+0000 7fac1082a700 1 -- 172.21.15.32:0/3091686395 learned_addr learned my addr 172.21.15.32:0/3091686395 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:52:58.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.007+0000 7fac1102b700 1 --2- 172.21.15.32:0/3091686395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac0c0fb4a0 0x7fac0c0fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:58.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.007+0000 7fac1082a700 1 -- 172.21.15.32:0/3091686395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac0c0fb4a0 msgr2=0x7fac0c0fc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:58.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.007+0000 7fac1082a700 1 --2- 172.21.15.32:0/3091686395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac0c0fb4a0 0x7fac0c0fc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:58.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.007+0000 7fac1082a700 1 -- 172.21.15.32:0/3091686395 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac08005040 con 0x7fac0c102e70 2024-04-12T05:52:58.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.007+0000 7fac1082a700 1 --2- 172.21.15.32:0/3091686395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac0c102e70 0x7fac0c0fcbc0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fac0000eb40 tx=0x7fac0000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:52:58.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.007+0000 7fabfdffb700 1 -- 172.21.15.32:0/3091686395 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac00007cc0 con 0x7fac0c102e70 2024-04-12T05:52:58.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.007+0000 7fabfdffb700 1 -- 172.21.15.32:0/3091686395 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac0000f040 con 0x7fac0c102e70 2024-04-12T05:52:58.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.007+0000 7fac1328f700 1 -- 172.21.15.32:0/3091686395 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac0c0fd490 con 0x7fac0c102e70 2024-04-12T05:52:58.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.007+0000 7fabfdffb700 1 -- 172.21.15.32:0/3091686395 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac00009570 con 0x7fac0c102e70 2024-04-12T05:52:58.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.007+0000 7fac1328f700 1 -- 172.21.15.32:0/3091686395 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac0c11a0e0 con 0x7fac0c102e70 2024-04-12T05:52:58.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.008+0000 7fac1328f700 1 -- 172.21.15.32:0/3091686395 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac0c100110 con 0x7fac0c102e70 2024-04-12T05:52:58.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.009+0000 7fabfdffb700 1 -- 172.21.15.32:0/3091686395 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fac000096d0 con 0x7fac0c102e70 2024-04-12T05:52:58.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.009+0000 7fabfdffb700 1 --2- 172.21.15.32:0/3091686395 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fabf806cc20 0x7fabf806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:58.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.009+0000 7fabfdffb700 1 -- 172.21.15.32:0/3091686395 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fac0008bcd0 con 0x7fac0c102e70 2024-04-12T05:52:58.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.009+0000 7fac1102b700 1 --2- 172.21.15.32:0/3091686395 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fabf806cc20 0x7fabf806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:58.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.010+0000 7fac1102b700 1 --2- 172.21.15.32:0/3091686395 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fabf806cc20 0x7fabf806f0d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fac08004540 tx=0x7fac08009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:52:58.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.014+0000 7fabfdffb700 1 -- 172.21.15.32:0/3091686395 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fac00055ce0 con 0x7fac0c102e70 2024-04-12T05:52:58.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:57 smithi067 ceph-mon[112979]: pgmap v373: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:58.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:57 smithi032 ceph-mon[146817]: pgmap v373: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:58.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.259+0000 7fac1328f700 1 -- 172.21.15.32:0/3091686395 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fac0c109e00 con 0x7fac0c102e70 2024-04-12T05:52:58.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.260+0000 7fabfdffb700 1 -- 172.21.15.32:0/3091686395 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1835 (secure 0 0 0) 0x7fac00059300 con 0x7fac0c102e70 2024-04-12T05:52:58.255 INFO:teuthology.orchestra.run.smithi032.stdout:e33 2024-04-12T05:52:58.255 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:52:58.255 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:52:58.255 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:52:58.255 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:52:58.255 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:52:58.255 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:52:58.255 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 24 2024-04-12T05:52:58.255 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:52:58.255 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:52:58.255 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:46:10.668569+0000 2024-04-12T05:52:58.255 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:52:58.256 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:52:58.256 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:52:58.256 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:52:58.256 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:52:58.256 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:52:58.256 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:52:58.256 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:52:58.256 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T05:52:58.256 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:52:58.256 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:52:58.256 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:52:58.256 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T05:52:58.256 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-12T05:52:58.256 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:52:58.256 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:52:58.256 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:52:58.257 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:52:58.257 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:52:58.257 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:52:58.257 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:52:58.257 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:52:58.257 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:52:58.257 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:52:58.257 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-12T05:52:58.257 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:52:58.257 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:52:58.257 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:52:58.257 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{-1:44317} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:52:58.257 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:52:58.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.262+0000 7fac1328f700 1 -- 172.21.15.32:0/3091686395 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fabf806cc20 msgr2=0x7fabf806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:58.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.262+0000 7fac1328f700 1 --2- 172.21.15.32:0/3091686395 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fabf806cc20 0x7fabf806f0d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fac08004540 tx=0x7fac08009e30 comp rx=0 tx=0).stop 2024-04-12T05:52:58.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.262+0000 7fac1328f700 1 -- 172.21.15.32:0/3091686395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac0c102e70 msgr2=0x7fac0c0fcbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:58.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.262+0000 7fac1328f700 1 --2- 172.21.15.32:0/3091686395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac0c102e70 0x7fac0c0fcbc0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fac0000eb40 tx=0x7fac0000b040 comp rx=0 tx=0).stop 2024-04-12T05:52:58.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.262+0000 7fac1328f700 1 -- 172.21.15.32:0/3091686395 shutdown_connections 2024-04-12T05:52:58.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.262+0000 7fac1328f700 1 --2- 172.21.15.32:0/3091686395 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fabf806cc20 0x7fabf806f0d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:58.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.262+0000 7fac1328f700 1 --2- 172.21.15.32:0/3091686395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac0c0fb4a0 0x7fac0c0fc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:58.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.262+0000 7fac1328f700 1 --2- 172.21.15.32:0/3091686395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac0c102e70 0x7fac0c0fcbc0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:58.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.262+0000 7fac1328f700 1 -- 172.21.15.32:0/3091686395 >> 172.21.15.32:0/3091686395 conn(0x7fac0c0714c0 msgr2=0x7fac0c0fa780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:52:58.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.262+0000 7fac1328f700 1 -- 172.21.15.32:0/3091686395 shutdown_connections 2024-04-12T05:52:58.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.262+0000 7fac1328f700 1 -- 172.21.15.32:0/3091686395 wait complete. 2024-04-12T05:52:58.259 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-12T05:52:58.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.366+0000 7f5edc10e700 1 -- 172.21.15.32:0/3717376579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ed4101d70 msgr2=0x7f5ed4102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:58.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.366+0000 7f5edc10e700 1 --2- 172.21.15.32:0/3717376579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ed4101d70 0x7f5ed4102140 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8004940 tx=0x7f5ec8009e30 comp rx=0 tx=0).stop 2024-04-12T05:52:58.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.366+0000 7f5edc10e700 1 -- 172.21.15.32:0/3717376579 shutdown_connections 2024-04-12T05:52:58.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.366+0000 7f5edc10e700 1 --2- 172.21.15.32:0/3717376579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ed4102710 0x7f5ed40fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:58.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.366+0000 7f5edc10e700 1 --2- 172.21.15.32:0/3717376579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ed4101d70 0x7f5ed4102140 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:58.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.366+0000 7f5edc10e700 1 -- 172.21.15.32:0/3717376579 >> 172.21.15.32:0/3717376579 conn(0x7f5ed4072bc0 msgr2=0x7f5ed4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:52:58.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.366+0000 7f5edc10e700 1 -- 172.21.15.32:0/3717376579 shutdown_connections 2024-04-12T05:52:58.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.366+0000 7f5edc10e700 1 -- 172.21.15.32:0/3717376579 wait complete. 2024-04-12T05:52:58.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.366+0000 7f5edc10e700 1 Processor -- start 2024-04-12T05:52:58.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.367+0000 7f5edc10e700 1 -- start start 2024-04-12T05:52:58.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.367+0000 7f5edc10e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ed4101d70 0x7f5ed4102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:58.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.367+0000 7f5edc10e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ed4102710 0x7f5ed4103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:58.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.367+0000 7f5edc10e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed41036a0 con 0x7f5ed4101d70 2024-04-12T05:52:58.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.367+0000 7f5edc10e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed41037e0 con 0x7f5ed4102710 2024-04-12T05:52:58.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.367+0000 7f5ed9eaa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ed4101d70 0x7f5ed4102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:58.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.367+0000 7f5ed9eaa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ed4101d70 0x7f5ed4102bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50102/0 (socket says 172.21.15.32:50102) 2024-04-12T05:52:58.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.367+0000 7f5ed9eaa700 1 -- 172.21.15.32:0/920732388 learned_addr learned my addr 172.21.15.32:0/920732388 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:52:58.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.367+0000 7f5ed96a9700 1 --2- 172.21.15.32:0/920732388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ed4102710 0x7f5ed4103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:58.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.367+0000 7f5ed9eaa700 1 -- 172.21.15.32:0/920732388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ed4102710 msgr2=0x7f5ed4103110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:58.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.367+0000 7f5ed9eaa700 1 --2- 172.21.15.32:0/920732388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ed4102710 0x7f5ed4103110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:58.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.367+0000 7f5ed9eaa700 1 -- 172.21.15.32:0/920732388 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec8005040 con 0x7f5ed4101d70 2024-04-12T05:52:58.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.367+0000 7f5ed9eaa700 1 --2- 172.21.15.32:0/920732388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ed4101d70 0x7f5ed4102bd0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8009390 tx=0x7f5ec8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:52:58.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.367+0000 7f5ec67fc700 1 -- 172.21.15.32:0/920732388 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ec800e040 con 0x7f5ed4101d70 2024-04-12T05:52:58.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.368+0000 7f5edc10e700 1 -- 172.21.15.32:0/920732388 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ed4103a30 con 0x7f5ed4101d70 2024-04-12T05:52:58.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.368+0000 7f5ec67fc700 1 -- 172.21.15.32:0/920732388 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ec8013420 con 0x7f5ed4101d70 2024-04-12T05:52:58.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.368+0000 7f5ec67fc700 1 -- 172.21.15.32:0/920732388 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ec801c630 con 0x7f5ed4101d70 2024-04-12T05:52:58.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.368+0000 7f5edc10e700 1 -- 172.21.15.32:0/920732388 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ed411a190 con 0x7f5ed4101d70 2024-04-12T05:52:58.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.368+0000 7f5edc10e700 1 -- 172.21.15.32:0/920732388 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ed40fd4c0 con 0x7f5ed4101d70 2024-04-12T05:52:58.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.370+0000 7f5ec67fc700 1 -- 172.21.15.32:0/920732388 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f5ec80139e0 con 0x7f5ed4101d70 2024-04-12T05:52:58.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.370+0000 7f5ec67fc700 1 --2- 172.21.15.32:0/920732388 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5ec006cc30 0x7f5ec006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:58.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.370+0000 7f5ec67fc700 1 -- 172.21.15.32:0/920732388 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f5ec808dd10 con 0x7f5ed4101d70 2024-04-12T05:52:58.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.370+0000 7f5ed96a9700 1 --2- 172.21.15.32:0/920732388 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5ec006cc30 0x7f5ec006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:58.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.371+0000 7f5ed96a9700 1 --2- 172.21.15.32:0/920732388 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5ec006cc30 0x7f5ec006f0e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4104510 tx=0x7f5ed000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:52:58.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.374+0000 7f5ec67fc700 1 -- 172.21.15.32:0/920732388 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5ec8057d90 con 0x7f5ed4101d70 2024-04-12T05:52:58.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.595+0000 7f5edc10e700 1 -- 172.21.15.32:0/920732388 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ed41042c0 con 0x7f5ec006cc30 2024-04-12T05:52:58.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.596+0000 7f5ec67fc700 1 -- 172.21.15.32:0/920732388 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f5ed41042c0 con 0x7f5ec006cc30 2024-04-12T05:52:58.591 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:52:58.591 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:52:58.591 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:52:58.591 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:52:58.591 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:52:58.591 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-12T05:52:58.591 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:52:58.591 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-12T05:52:58.592 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:52:58.592 INFO:teuthology.orchestra.run.smithi032.stdout: "mds" 2024-04-12T05:52:58.592 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:52:58.592 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:52:58.592 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:52:58.592 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:52:58.592 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:52:58.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.598+0000 7f5edc10e700 1 -- 172.21.15.32:0/920732388 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5ec006cc30 msgr2=0x7f5ec006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:58.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.598+0000 7f5edc10e700 1 --2- 172.21.15.32:0/920732388 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5ec006cc30 0x7f5ec006f0e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4104510 tx=0x7f5ed000b040 comp rx=0 tx=0).stop 2024-04-12T05:52:58.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.598+0000 7f5edc10e700 1 -- 172.21.15.32:0/920732388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ed4101d70 msgr2=0x7f5ed4102bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:58.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.598+0000 7f5edc10e700 1 --2- 172.21.15.32:0/920732388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ed4101d70 0x7f5ed4102bd0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8009390 tx=0x7f5ec8009e30 comp rx=0 tx=0).stop 2024-04-12T05:52:58.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.598+0000 7f5edc10e700 1 -- 172.21.15.32:0/920732388 shutdown_connections 2024-04-12T05:52:58.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.598+0000 7f5edc10e700 1 --2- 172.21.15.32:0/920732388 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5ec006cc30 0x7f5ec006f0e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:58.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.598+0000 7f5edc10e700 1 --2- 172.21.15.32:0/920732388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ed4101d70 0x7f5ed4102bd0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:58.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.598+0000 7f5edc10e700 1 --2- 172.21.15.32:0/920732388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ed4102710 0x7f5ed4103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:58.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.598+0000 7f5edc10e700 1 -- 172.21.15.32:0/920732388 >> 172.21.15.32:0/920732388 conn(0x7f5ed4072bc0 msgr2=0x7f5ed40659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:52:58.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.598+0000 7f5edc10e700 1 -- 172.21.15.32:0/920732388 shutdown_connections 2024-04-12T05:52:58.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.599+0000 7f5edc10e700 1 -- 172.21.15.32:0/920732388 wait complete. 2024-04-12T05:52:58.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.702+0000 7ffbd7edf700 1 -- 172.21.15.32:0/2254800844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbd0101d10 msgr2=0x7ffbd01020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:58.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.702+0000 7ffbd7edf700 1 --2- 172.21.15.32:0/2254800844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbd0101d10 0x7ffbd01020e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc004940 tx=0x7ffbcc009e30 comp rx=0 tx=0).stop 2024-04-12T05:52:58.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.702+0000 7ffbd7edf700 1 -- 172.21.15.32:0/2254800844 shutdown_connections 2024-04-12T05:52:58.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.702+0000 7ffbd7edf700 1 --2- 172.21.15.32:0/2254800844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd01026b0 0x7ffbd00fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:58.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.702+0000 7ffbd7edf700 1 --2- 172.21.15.32:0/2254800844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbd0101d10 0x7ffbd01020e0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:58.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.702+0000 7ffbd7edf700 1 -- 172.21.15.32:0/2254800844 >> 172.21.15.32:0/2254800844 conn(0x7ffbd00f7880 msgr2=0x7ffbd00f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:52:58.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.702+0000 7ffbd7edf700 1 -- 172.21.15.32:0/2254800844 shutdown_connections 2024-04-12T05:52:58.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.702+0000 7ffbd7edf700 1 -- 172.21.15.32:0/2254800844 wait complete. 2024-04-12T05:52:58.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.703+0000 7ffbd7edf700 1 Processor -- start 2024-04-12T05:52:58.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.703+0000 7ffbd7edf700 1 -- start start 2024-04-12T05:52:58.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.703+0000 7ffbd7edf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd0101d10 0x7ffbd0112d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:58.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.703+0000 7ffbd7edf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbd01026b0 0x7ffbd0113260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:58.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.703+0000 7ffbd7edf700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd01138b0 con 0x7ffbd01026b0 2024-04-12T05:52:58.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.703+0000 7ffbd7edf700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd010cc80 con 0x7ffbd0101d10 2024-04-12T05:52:58.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.703+0000 7ffbd547a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbd01026b0 0x7ffbd0113260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:58.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.703+0000 7ffbd547a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbd01026b0 0x7ffbd0113260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50134/0 (socket says 172.21.15.32:50134) 2024-04-12T05:52:58.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.703+0000 7ffbd547a700 1 -- 172.21.15.32:0/3565251570 learned_addr learned my addr 172.21.15.32:0/3565251570 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:52:58.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.703+0000 7ffbd5c7b700 1 --2- 172.21.15.32:0/3565251570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd0101d10 0x7ffbd0112d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:58.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.704+0000 7ffbd547a700 1 -- 172.21.15.32:0/3565251570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd0101d10 msgr2=0x7ffbd0112d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:58.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.704+0000 7ffbd547a700 1 --2- 172.21.15.32:0/3565251570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd0101d10 0x7ffbd0112d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:58.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.704+0000 7ffbd547a700 1 -- 172.21.15.32:0/3565251570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbcc005040 con 0x7ffbd01026b0 2024-04-12T05:52:58.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.704+0000 7ffbd547a700 1 --2- 172.21.15.32:0/3565251570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbd01026b0 0x7ffbd0113260 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ffbc400c890 tx=0x7ffbc40044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:52:58.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.704+0000 7ffbc27fc700 1 -- 172.21.15.32:0/3565251570 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffbc400a7d0 con 0x7ffbd01026b0 2024-04-12T05:52:58.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.704+0000 7ffbc27fc700 1 -- 172.21.15.32:0/3565251570 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffbc400ade0 con 0x7ffbd01026b0 2024-04-12T05:52:58.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.704+0000 7ffbd7edf700 1 -- 172.21.15.32:0/3565251570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbd010cf60 con 0x7ffbd01026b0 2024-04-12T05:52:58.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.704+0000 7ffbc27fc700 1 -- 172.21.15.32:0/3565251570 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffbc40113a0 con 0x7ffbd01026b0 2024-04-12T05:52:58.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.704+0000 7ffbd7edf700 1 -- 172.21.15.32:0/3565251570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbd010d520 con 0x7ffbd01026b0 2024-04-12T05:52:58.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.705+0000 7ffbd7edf700 1 -- 172.21.15.32:0/3565251570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbd00fd460 con 0x7ffbd01026b0 2024-04-12T05:52:58.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.705+0000 7ffbc27fc700 1 -- 172.21.15.32:0/3565251570 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ffbc4012400 con 0x7ffbd01026b0 2024-04-12T05:52:58.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.706+0000 7ffbc27fc700 1 --2- 172.21.15.32:0/3565251570 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ffbbc06cc20 0x7ffbbc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:52:58.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.706+0000 7ffbc27fc700 1 -- 172.21.15.32:0/3565251570 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7ffbc408c1e0 con 0x7ffbd01026b0 2024-04-12T05:52:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.706+0000 7ffbd5c7b700 1 --2- 172.21.15.32:0/3565251570 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ffbbc06cc20 0x7ffbbc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:52:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.706+0000 7ffbd5c7b700 1 --2- 172.21.15.32:0/3565251570 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ffbbc06cc20 0x7ffbbc06f0d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc004560 tx=0x7ffbcc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:52:58.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:58.710+0000 7ffbc27fc700 1 -- 172.21.15.32:0/3565251570 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ffbc4059c50 con 0x7ffbd01026b0 2024-04-12T05:52:58.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:59.002+0000 7ffbd7edf700 1 -- 172.21.15.32:0/3565251570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ffbd0108cd0 con 0x7ffbd01026b0 2024-04-12T05:52:58.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:59.002+0000 7ffbc27fc700 1 -- 172.21.15.32:0/3565251570 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7ffbc40597e0 con 0x7ffbd01026b0 2024-04-12T05:52:58.997 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:52:58.997 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:52:58.997 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-12T05:52:58.998 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:52:58.998 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:52:58.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:59.004+0000 7ffbd7edf700 1 -- 172.21.15.32:0/3565251570 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ffbbc06cc20 msgr2=0x7ffbbc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:58.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:59.004+0000 7ffbd7edf700 1 --2- 172.21.15.32:0/3565251570 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ffbbc06cc20 0x7ffbbc06f0d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc004560 tx=0x7ffbcc009e30 comp rx=0 tx=0).stop 2024-04-12T05:52:58.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:59.004+0000 7ffbd7edf700 1 -- 172.21.15.32:0/3565251570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbd01026b0 msgr2=0x7ffbd0113260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:52:58.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:59.004+0000 7ffbd7edf700 1 --2- 172.21.15.32:0/3565251570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbd01026b0 0x7ffbd0113260 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ffbc400c890 tx=0x7ffbc40044c0 comp rx=0 tx=0).stop 2024-04-12T05:52:58.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:59.005+0000 7ffbd7edf700 1 -- 172.21.15.32:0/3565251570 shutdown_connections 2024-04-12T05:52:58.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:59.005+0000 7ffbd7edf700 1 --2- 172.21.15.32:0/3565251570 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ffbbc06cc20 0x7ffbbc06f0d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:59.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:59.005+0000 7ffbd7edf700 1 --2- 172.21.15.32:0/3565251570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd0101d10 0x7ffbd0112d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:59.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:59.005+0000 7ffbd7edf700 1 --2- 172.21.15.32:0/3565251570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbd01026b0 0x7ffbd0113260 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:52:59.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:59.005+0000 7ffbd7edf700 1 -- 172.21.15.32:0/3565251570 >> 172.21.15.32:0/3565251570 conn(0x7ffbd00f7880 msgr2=0x7ffbd00f8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:52:59.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:59.005+0000 7ffbd7edf700 1 -- 172.21.15.32:0/3565251570 shutdown_connections 2024-04-12T05:52:59.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:52:59.005+0000 7ffbd7edf700 1 -- 172.21.15.32:0/3565251570 wait complete. 2024-04-12T05:52:59.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:58 smithi067 ceph-mon[112979]: from='client.34732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:52:59.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:58 smithi067 ceph-mon[112979]: from='client.34736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:52:59.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:58 smithi067 ceph-mon[112979]: from='client.34740 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:52:59.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:58 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1012406681' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:52:59.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:58 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3091686395' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:52:59.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:58 smithi032 ceph-mon[146817]: from='client.34732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:52:59.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:58 smithi032 ceph-mon[146817]: from='client.34736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:52:59.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:58 smithi032 ceph-mon[146817]: from='client.34740 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:52:59.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:58 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1012406681' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:52:59.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:58 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3091686395' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:53:00.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:59 smithi067 ceph-mon[112979]: pgmap v374: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:00.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:59 smithi067 ceph-mon[112979]: from='client.34752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:53:00.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:52:59 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3565251570' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:53:00.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:59 smithi032 ceph-mon[146817]: pgmap v374: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:00.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:59 smithi032 ceph-mon[146817]: from='client.34752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:53:00.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:52:59 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3565251570' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:53:02.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:01 smithi067 ceph-mon[112979]: pgmap v375: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:02.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:01 smithi032 ceph-mon[146817]: pgmap v375: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:04.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:03 smithi067 ceph-mon[112979]: pgmap v376: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:04.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:53:04.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:03 smithi032 ceph-mon[146817]: pgmap v376: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:04.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:53:06.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:05 smithi067 ceph-mon[112979]: pgmap v377: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:06.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:05 smithi032 ceph-mon[146817]: pgmap v377: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:08.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:07 smithi067 ceph-mon[112979]: pgmap v378: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:08.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:07 smithi032 ceph-mon[146817]: pgmap v378: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:10.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:09 smithi067 ceph-mon[112979]: pgmap v379: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:10.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:09 smithi032 ceph-mon[146817]: pgmap v379: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:12.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:11 smithi067 ceph-mon[112979]: pgmap v380: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:12.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:11 smithi032 ceph-mon[146817]: pgmap v380: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:14.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:13 smithi067 ceph-mon[112979]: pgmap v381: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:14.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:13 smithi032 ceph-mon[146817]: pgmap v381: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:16.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:15 smithi067 ceph-mon[112979]: pgmap v382: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:16.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:15 smithi032 ceph-mon[146817]: pgmap v382: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:18.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:17 smithi067 ceph-mon[112979]: pgmap v383: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:18.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:17 smithi032 ceph-mon[146817]: pgmap v383: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:19.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:53:19.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:53:20.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:19 smithi032 ceph-mon[146817]: pgmap v384: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:20.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:19 smithi067 ceph-mon[112979]: pgmap v384: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:22.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:21 smithi032 ceph-mon[146817]: pgmap v385: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:22.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:21 smithi067 ceph-mon[112979]: pgmap v385: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:24.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:23 smithi032 ceph-mon[146817]: pgmap v386: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:24.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:23 smithi067 ceph-mon[112979]: pgmap v386: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:26.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:25 smithi032 ceph-mon[146817]: pgmap v387: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:26.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:25 smithi067 ceph-mon[112979]: pgmap v387: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:28.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:27 smithi032 ceph-mon[146817]: pgmap v388: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:28.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:27 smithi067 ceph-mon[112979]: pgmap v388: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:29.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.121+0000 7f56d1f97700 1 -- 172.21.15.32:0/1341059098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56cc0fb4a0 msgr2=0x7f56cc0fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:53:29.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.121+0000 7f56d1f97700 1 --2- 172.21.15.32:0/1341059098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56cc0fb4a0 0x7f56cc0fb910 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f56b4004940 tx=0x7f56b4009e30 comp rx=0 tx=0).stop 2024-04-12T05:53:29.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.121+0000 7f56d1f97700 1 -- 172.21.15.32:0/1341059098 shutdown_connections 2024-04-12T05:53:29.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.121+0000 7f56d1f97700 1 --2- 172.21.15.32:0/1341059098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56cc0fb4a0 0x7f56cc0fb910 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:29.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.121+0000 7f56d1f97700 1 --2- 172.21.15.32:0/1341059098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56cc102e70 0x7f56cc103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:29.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.121+0000 7f56d1f97700 1 -- 172.21.15.32:0/1341059098 >> 172.21.15.32:0/1341059098 conn(0x7f56cc0714c0 msgr2=0x7f56cc0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:53:29.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.121+0000 7f56d1f97700 1 -- 172.21.15.32:0/1341059098 shutdown_connections 2024-04-12T05:53:29.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.121+0000 7f56d1f97700 1 -- 172.21.15.32:0/1341059098 wait complete. 2024-04-12T05:53:29.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.122+0000 7f56d1f97700 1 Processor -- start 2024-04-12T05:53:29.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.122+0000 7f56d1f97700 1 -- start start 2024-04-12T05:53:29.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.122+0000 7f56d1f97700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56cc0fb4a0 0x7f56cc113a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:53:29.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.122+0000 7f56d1f97700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56cc102e70 0x7f56cc113f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:53:29.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.122+0000 7f56d1f97700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc1145c0 con 0x7f56cc102e70 2024-04-12T05:53:29.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.122+0000 7f56d1f97700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc10d9b0 con 0x7f56cc0fb4a0 2024-04-12T05:53:29.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.122+0000 7f56c2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56cc102e70 0x7f56cc113f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:53:29.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.122+0000 7f56c2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56cc102e70 0x7f56cc113f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34686/0 (socket says 172.21.15.32:34686) 2024-04-12T05:53:29.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.122+0000 7f56c2ffd700 1 -- 172.21.15.32:0/581575985 learned_addr learned my addr 172.21.15.32:0/581575985 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:53:29.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.123+0000 7f56cb7fe700 1 --2- 172.21.15.32:0/581575985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56cc0fb4a0 0x7f56cc113a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:53:29.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.123+0000 7f56c2ffd700 1 -- 172.21.15.32:0/581575985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56cc0fb4a0 msgr2=0x7f56cc113a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:53:29.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.123+0000 7f56c2ffd700 1 --2- 172.21.15.32:0/581575985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56cc0fb4a0 0x7f56cc113a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:29.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.123+0000 7f56c2ffd700 1 -- 172.21.15.32:0/581575985 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b4005040 con 0x7f56cc102e70 2024-04-12T05:53:29.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.123+0000 7f56c2ffd700 1 --2- 172.21.15.32:0/581575985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56cc102e70 0x7f56cc113f70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f56b4000c00 tx=0x7f56b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:53:29.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.123+0000 7f56c8ff9700 1 -- 172.21.15.32:0/581575985 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56b4021070 con 0x7f56cc102e70 2024-04-12T05:53:29.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.123+0000 7f56c8ff9700 1 -- 172.21.15.32:0/581575985 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f56b40093b0 con 0x7f56cc102e70 2024-04-12T05:53:29.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.123+0000 7f56c8ff9700 1 -- 172.21.15.32:0/581575985 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56b400e070 con 0x7f56cc102e70 2024-04-12T05:53:29.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.123+0000 7f56d1f97700 1 -- 172.21.15.32:0/581575985 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56cc10dc30 con 0x7f56cc102e70 2024-04-12T05:53:29.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.123+0000 7f56d1f97700 1 -- 172.21.15.32:0/581575985 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56cc10e1f0 con 0x7f56cc102e70 2024-04-12T05:53:29.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.124+0000 7f56d1f97700 1 -- 172.21.15.32:0/581575985 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56cc100110 con 0x7f56cc102e70 2024-04-12T05:53:29.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.125+0000 7f56c8ff9700 1 -- 172.21.15.32:0/581575985 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f56b401c490 con 0x7f56cc102e70 2024-04-12T05:53:29.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.125+0000 7f56c8ff9700 1 --2- 172.21.15.32:0/581575985 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f56b806cc20 0x7f56b806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:53:29.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.125+0000 7f56c8ff9700 1 -- 172.21.15.32:0/581575985 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f56b408d170 con 0x7f56cc102e70 2024-04-12T05:53:29.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.125+0000 7f56cb7fe700 1 --2- 172.21.15.32:0/581575985 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f56b806cc20 0x7f56b806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:53:29.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.126+0000 7f56cb7fe700 1 --2- 172.21.15.32:0/581575985 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f56b806cc20 0x7f56b806f0d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f56bc009b80 tx=0x7f56bc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:53:29.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.130+0000 7f56c8ff9700 1 -- 172.21.15.32:0/581575985 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f56b405abe0 con 0x7f56cc102e70 2024-04-12T05:53:29.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.352+0000 7f56d1f97700 1 -- 172.21.15.32:0/581575985 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f56cc113700 con 0x7f56b806cc20 2024-04-12T05:53:29.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.354+0000 7f56c8ff9700 1 -- 172.21.15.32:0/581575985 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f56cc113700 con 0x7f56b806cc20 2024-04-12T05:53:29.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.356+0000 7f56d1f97700 1 -- 172.21.15.32:0/581575985 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f56b806cc20 msgr2=0x7f56b806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:53:29.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.356+0000 7f56d1f97700 1 --2- 172.21.15.32:0/581575985 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f56b806cc20 0x7f56b806f0d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f56bc009b80 tx=0x7f56bc00b040 comp rx=0 tx=0).stop 2024-04-12T05:53:29.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.356+0000 7f56d1f97700 1 -- 172.21.15.32:0/581575985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56cc102e70 msgr2=0x7f56cc113f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:53:29.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.356+0000 7f56d1f97700 1 --2- 172.21.15.32:0/581575985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56cc102e70 0x7f56cc113f70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f56b4000c00 tx=0x7f56b4009e30 comp rx=0 tx=0).stop 2024-04-12T05:53:29.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.356+0000 7f56d1f97700 1 -- 172.21.15.32:0/581575985 shutdown_connections 2024-04-12T05:53:29.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.356+0000 7f56d1f97700 1 --2- 172.21.15.32:0/581575985 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f56b806cc20 0x7f56b806f0d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:29.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.356+0000 7f56d1f97700 1 --2- 172.21.15.32:0/581575985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56cc0fb4a0 0x7f56cc113a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:29.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.356+0000 7f56d1f97700 1 --2- 172.21.15.32:0/581575985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56cc102e70 0x7f56cc113f70 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:29.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.356+0000 7f56d1f97700 1 -- 172.21.15.32:0/581575985 >> 172.21.15.32:0/581575985 conn(0x7f56cc0714c0 msgr2=0x7f56cc0fa830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:53:29.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.356+0000 7f56d1f97700 1 -- 172.21.15.32:0/581575985 shutdown_connections 2024-04-12T05:53:29.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.356+0000 7f56d1f97700 1 -- 172.21.15.32:0/581575985 wait complete. 2024-04-12T05:53:29.362 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:53:29.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.461+0000 7fbca2ebd700 1 -- 172.21.15.32:0/2700577208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc9c101d70 msgr2=0x7fbc9c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:53:29.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.461+0000 7fbca2ebd700 1 --2- 172.21.15.32:0/2700577208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc9c101d70 0x7fbc9c102140 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c004940 tx=0x7fbc8c009e30 comp rx=0 tx=0).stop 2024-04-12T05:53:29.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.461+0000 7fbca2ebd700 1 -- 172.21.15.32:0/2700577208 shutdown_connections 2024-04-12T05:53:29.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.461+0000 7fbca2ebd700 1 --2- 172.21.15.32:0/2700577208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc9c102710 0x7fbc9c0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:29.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.461+0000 7fbca2ebd700 1 --2- 172.21.15.32:0/2700577208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc9c101d70 0x7fbc9c102140 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:29.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.461+0000 7fbca2ebd700 1 -- 172.21.15.32:0/2700577208 >> 172.21.15.32:0/2700577208 conn(0x7fbc9c072bc0 msgr2=0x7fbc9c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:53:29.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.461+0000 7fbca2ebd700 1 -- 172.21.15.32:0/2700577208 shutdown_connections 2024-04-12T05:53:29.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.461+0000 7fbca2ebd700 1 -- 172.21.15.32:0/2700577208 wait complete. 2024-04-12T05:53:29.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.462+0000 7fbca2ebd700 1 Processor -- start 2024-04-12T05:53:29.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.462+0000 7fbca2ebd700 1 -- start start 2024-04-12T05:53:29.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.462+0000 7fbca2ebd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc9c101d70 0x7fbc9c102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:53:29.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.462+0000 7fbca2ebd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc9c102710 0x7fbc9c103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:53:29.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.462+0000 7fbca2ebd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c1036a0 con 0x7fbc9c101d70 2024-04-12T05:53:29.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.462+0000 7fbca2ebd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c1037e0 con 0x7fbc9c102710 2024-04-12T05:53:29.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.462+0000 7fbca0c59700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc9c101d70 0x7fbc9c102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:53:29.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.462+0000 7fbca0c59700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc9c101d70 0x7fbc9c102bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34714/0 (socket says 172.21.15.32:34714) 2024-04-12T05:53:29.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.462+0000 7fbca0c59700 1 -- 172.21.15.32:0/1733771023 learned_addr learned my addr 172.21.15.32:0/1733771023 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:53:29.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.462+0000 7fbc9bfff700 1 --2- 172.21.15.32:0/1733771023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc9c102710 0x7fbc9c103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:53:29.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.463+0000 7fbca0c59700 1 -- 172.21.15.32:0/1733771023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc9c102710 msgr2=0x7fbc9c103110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:53:29.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.463+0000 7fbca0c59700 1 --2- 172.21.15.32:0/1733771023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc9c102710 0x7fbc9c103110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:29.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.463+0000 7fbca0c59700 1 -- 172.21.15.32:0/1733771023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc8c005040 con 0x7fbc9c101d70 2024-04-12T05:53:29.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.463+0000 7fbca0c59700 1 --2- 172.21.15.32:0/1733771023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc9c101d70 0x7fbc9c102bd0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c00b3b0 tx=0x7fbc8c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:53:29.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.463+0000 7fbc997fa700 1 -- 172.21.15.32:0/1733771023 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc8c00e040 con 0x7fbc9c101d70 2024-04-12T05:53:29.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.463+0000 7fbca2ebd700 1 -- 172.21.15.32:0/1733771023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc9c103a30 con 0x7fbc9c101d70 2024-04-12T05:53:29.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.463+0000 7fbc997fa700 1 -- 172.21.15.32:0/1733771023 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc8c013420 con 0x7fbc9c101d70 2024-04-12T05:53:29.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.463+0000 7fbc997fa700 1 -- 172.21.15.32:0/1733771023 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc8c01c610 con 0x7fbc9c101d70 2024-04-12T05:53:29.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.463+0000 7fbca2ebd700 1 -- 172.21.15.32:0/1733771023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc9c11a1f0 con 0x7fbc9c101d70 2024-04-12T05:53:29.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.464+0000 7fbca2ebd700 1 -- 172.21.15.32:0/1733771023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc9c0fd4c0 con 0x7fbc9c101d70 2024-04-12T05:53:29.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.464+0000 7fbc997fa700 1 -- 172.21.15.32:0/1733771023 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fbc8c00ba80 con 0x7fbc9c101d70 2024-04-12T05:53:29.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.465+0000 7fbc997fa700 1 --2- 172.21.15.32:0/1733771023 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbc8406cc20 0x7fbc8406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:53:29.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.465+0000 7fbc997fa700 1 -- 172.21.15.32:0/1733771023 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fbc8c08d250 con 0x7fbc9c101d70 2024-04-12T05:53:29.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.465+0000 7fbc9bfff700 1 --2- 172.21.15.32:0/1733771023 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbc8406cc20 0x7fbc8406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:53:29.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.465+0000 7fbc9bfff700 1 --2- 172.21.15.32:0/1733771023 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbc8406cc20 0x7fbc8406f0d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c104510 tx=0x7fbc9000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:53:29.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.469+0000 7fbc997fa700 1 -- 172.21.15.32:0/1733771023 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbc8c057230 con 0x7fbc9c101d70 2024-04-12T05:53:29.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.691+0000 7fbca2ebd700 1 -- 172.21.15.32:0/1733771023 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbc9c1042c0 con 0x7fbc8406cc20 2024-04-12T05:53:29.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.692+0000 7fbc997fa700 1 -- 172.21.15.32:0/1733771023 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fbc9c1042c0 con 0x7fbc8406cc20 2024-04-12T05:53:29.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.694+0000 7fbca2ebd700 1 -- 172.21.15.32:0/1733771023 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbc8406cc20 msgr2=0x7fbc8406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:53:29.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.694+0000 7fbca2ebd700 1 --2- 172.21.15.32:0/1733771023 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbc8406cc20 0x7fbc8406f0d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c104510 tx=0x7fbc9000b040 comp rx=0 tx=0).stop 2024-04-12T05:53:29.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.694+0000 7fbca2ebd700 1 -- 172.21.15.32:0/1733771023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc9c101d70 msgr2=0x7fbc9c102bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:53:29.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.694+0000 7fbca2ebd700 1 --2- 172.21.15.32:0/1733771023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc9c101d70 0x7fbc9c102bd0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c00b3b0 tx=0x7fbc8c009e30 comp rx=0 tx=0).stop 2024-04-12T05:53:29.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.694+0000 7fbca2ebd700 1 -- 172.21.15.32:0/1733771023 shutdown_connections 2024-04-12T05:53:29.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.694+0000 7fbca2ebd700 1 --2- 172.21.15.32:0/1733771023 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbc8406cc20 0x7fbc8406f0d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:29.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.694+0000 7fbca2ebd700 1 --2- 172.21.15.32:0/1733771023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc9c101d70 0x7fbc9c102bd0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:29.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.694+0000 7fbca2ebd700 1 --2- 172.21.15.32:0/1733771023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc9c102710 0x7fbc9c103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:29.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.694+0000 7fbca2ebd700 1 -- 172.21.15.32:0/1733771023 >> 172.21.15.32:0/1733771023 conn(0x7fbc9c072bc0 msgr2=0x7fbc9c0659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:53:29.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.694+0000 7fbca2ebd700 1 -- 172.21.15.32:0/1733771023 shutdown_connections 2024-04-12T05:53:29.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.694+0000 7fbca2ebd700 1 -- 172.21.15.32:0/1733771023 wait complete. 2024-04-12T05:53:29.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.798+0000 7f3042fba700 1 -- 172.21.15.32:0/1013445518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f303c0fe830 msgr2=0x7f303c106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:53:29.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.798+0000 7f3042fba700 1 --2- 172.21.15.32:0/1013445518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f303c0fe830 0x7f303c106f00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f3028004940 tx=0x7f3028009e30 comp rx=0 tx=0).stop 2024-04-12T05:53:29.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.798+0000 7f3042fba700 1 -- 172.21.15.32:0/1013445518 shutdown_connections 2024-04-12T05:53:29.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.798+0000 7f3042fba700 1 --2- 172.21.15.32:0/1013445518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f303c0fe830 0x7f303c106f00 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:29.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.798+0000 7f3042fba700 1 --2- 172.21.15.32:0/1013445518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f303c0fde90 0x7f303c0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:29.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.798+0000 7f3042fba700 1 -- 172.21.15.32:0/1013445518 >> 172.21.15.32:0/1013445518 conn(0x7f303c072bc0 msgr2=0x7f303c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:53:29.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.798+0000 7f3042fba700 1 -- 172.21.15.32:0/1013445518 shutdown_connections 2024-04-12T05:53:29.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.798+0000 7f3042fba700 1 -- 172.21.15.32:0/1013445518 wait complete. 2024-04-12T05:53:29.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.798+0000 7f3042fba700 1 Processor -- start 2024-04-12T05:53:29.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.798+0000 7f3042fba700 1 -- start start 2024-04-12T05:53:29.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.799+0000 7f3042fba700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f303c0fde90 0x7f303c0fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:53:29.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.799+0000 7f3042fba700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f303c0fe830 0x7f303c0fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:53:29.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.799+0000 7f3042fba700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303c0fc820 con 0x7f303c0fde90 2024-04-12T05:53:29.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.799+0000 7f3042fba700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303c0fc960 con 0x7f303c0fe830 2024-04-12T05:53:29.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.799+0000 7f3040d56700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f303c0fde90 0x7f303c0fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:53:29.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.799+0000 7f3040d56700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f303c0fde90 0x7f303c0fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34740/0 (socket says 172.21.15.32:34740) 2024-04-12T05:53:29.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.799+0000 7f3040d56700 1 -- 172.21.15.32:0/3911475068 learned_addr learned my addr 172.21.15.32:0/3911475068 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:53:29.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.799+0000 7f303bfff700 1 --2- 172.21.15.32:0/3911475068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f303c0fe830 0x7f303c0fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:53:29.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.799+0000 7f3040d56700 1 -- 172.21.15.32:0/3911475068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f303c0fe830 msgr2=0x7f303c0fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:53:29.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.799+0000 7f3040d56700 1 --2- 172.21.15.32:0/3911475068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f303c0fe830 0x7f303c0fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:29.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.799+0000 7f3040d56700 1 -- 172.21.15.32:0/3911475068 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3028005040 con 0x7f303c0fde90 2024-04-12T05:53:29.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.799+0000 7f3040d56700 1 --2- 172.21.15.32:0/3911475068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f303c0fde90 0x7f303c0fbd50 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f3030002a50 tx=0x7f30300046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:53:29.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.800+0000 7f30397fa700 1 -- 172.21.15.32:0/3911475068 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3030007cc0 con 0x7f303c0fde90 2024-04-12T05:53:29.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.800+0000 7f30397fa700 1 -- 172.21.15.32:0/3911475068 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f303000f040 con 0x7f303c0fde90 2024-04-12T05:53:29.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.800+0000 7f3042fba700 1 -- 172.21.15.32:0/3911475068 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f303c119de0 con 0x7f303c0fde90 2024-04-12T05:53:29.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.800+0000 7f30397fa700 1 -- 172.21.15.32:0/3911475068 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f303000d9c0 con 0x7f303c0fde90 2024-04-12T05:53:29.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.800+0000 7f3042fba700 1 -- 172.21.15.32:0/3911475068 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f303c11a1e0 con 0x7f303c0fde90 2024-04-12T05:53:29.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.801+0000 7f3042fba700 1 -- 172.21.15.32:0/3911475068 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f303c065e30 con 0x7f303c0fde90 2024-04-12T05:53:29.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.801+0000 7f30397fa700 1 -- 172.21.15.32:0/3911475068 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f303000db20 con 0x7f303c0fde90 2024-04-12T05:53:29.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.801+0000 7f30397fa700 1 --2- 172.21.15.32:0/3911475068 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f302c06cc30 0x7f302c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:53:29.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.802+0000 7f30397fa700 1 -- 172.21.15.32:0/3911475068 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f303008bcf0 con 0x7f303c0fde90 2024-04-12T05:53:29.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.802+0000 7f303bfff700 1 --2- 172.21.15.32:0/3911475068 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f302c06cc30 0x7f302c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:53:29.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.802+0000 7f303bfff700 1 --2- 172.21.15.32:0/3911475068 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f302c06cc30 0x7f302c06f0e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f303c0fd690 tx=0x7f3028009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:53:29.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:29.806+0000 7f30397fa700 1 -- 172.21.15.32:0/3911475068 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3030055cd0 con 0x7f303c0fde90 2024-04-12T05:53:30.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.020+0000 7f3042fba700 1 -- 172.21.15.32:0/3911475068 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f303c0fd440 con 0x7f302c06cc30 2024-04-12T05:53:30.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.030+0000 7f30397fa700 1 -- 172.21.15.32:0/3911475068 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f303c0fd440 con 0x7f302c06cc30 2024-04-12T05:53:30.026 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:53:30.026 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (13m) 6m ago 24m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:53:30.026 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (10m) 6m ago 24m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:53:30.026 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (10m) 6m ago 23m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:53:30.026 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (13m) 6m ago 24m 87.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:53:30.026 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (7m) 6m ago 19m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T05:53:30.026 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (6m) 6m ago 18m 13.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T05:53:30.026 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (6m) 6m ago 18m 11.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 6d4aca6f74a0 2024-04-12T05:53:30.026 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (6m) 6m ago 19m 14.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T05:53:30.026 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (14m) 6m ago 26m 507M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:53:30.026 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (14m) 6m ago 23m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:53:30.027 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (11m) 6m ago 26m 53.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:53:30.027 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (11m) 6m ago 23m 43.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:53:30.027 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (13m) 6m ago 24m 16.9M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:53:30.027 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (13m) 6m ago 23m 18.4M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:53:30.027 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (10m) 6m ago 22m 91.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T05:53:30.027 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (9m) 6m ago 22m 97.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T05:53:30.027 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (9m) 6m ago 21m 81.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T05:53:30.027 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (8m) 6m ago 21m 113M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T05:53:30.027 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (8m) 6m ago 21m 90.3M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T05:53:30.027 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (7m) 6m ago 20m 84.8M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T05:53:30.027 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (13m) 6m ago 24m 64.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:53:30.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.032+0000 7f3042fba700 1 -- 172.21.15.32:0/3911475068 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f302c06cc30 msgr2=0x7f302c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:53:30.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.032+0000 7f3042fba700 1 --2- 172.21.15.32:0/3911475068 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f302c06cc30 0x7f302c06f0e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f303c0fd690 tx=0x7f3028009e30 comp rx=0 tx=0).stop 2024-04-12T05:53:30.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.032+0000 7f3042fba700 1 -- 172.21.15.32:0/3911475068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f303c0fde90 msgr2=0x7f303c0fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:53:30.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.032+0000 7f3042fba700 1 --2- 172.21.15.32:0/3911475068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f303c0fde90 0x7f303c0fbd50 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f3030002a50 tx=0x7f30300046b0 comp rx=0 tx=0).stop 2024-04-12T05:53:30.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.032+0000 7f3042fba700 1 -- 172.21.15.32:0/3911475068 shutdown_connections 2024-04-12T05:53:30.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.032+0000 7f3042fba700 1 --2- 172.21.15.32:0/3911475068 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f302c06cc30 0x7f302c06f0e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:30.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.032+0000 7f3042fba700 1 --2- 172.21.15.32:0/3911475068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f303c0fde90 0x7f303c0fbd50 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:30.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.032+0000 7f3042fba700 1 --2- 172.21.15.32:0/3911475068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f303c0fe830 0x7f303c0fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:30.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.032+0000 7f3042fba700 1 -- 172.21.15.32:0/3911475068 >> 172.21.15.32:0/3911475068 conn(0x7f303c072bc0 msgr2=0x7f303c103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:53:30.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.033+0000 7f3042fba700 1 -- 172.21.15.32:0/3911475068 shutdown_connections 2024-04-12T05:53:30.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.033+0000 7f3042fba700 1 -- 172.21.15.32:0/3911475068 wait complete. 2024-04-12T05:53:30.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:29 smithi032 ceph-mon[146817]: pgmap v389: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:30.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.135+0000 7fb758975700 1 -- 172.21.15.32:0/541497195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb754101cf0 msgr2=0x7fb7541020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:53:30.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.135+0000 7fb758975700 1 --2- 172.21.15.32:0/541497195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb754101cf0 0x7fb7541020c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb744004940 tx=0x7fb744009e30 comp rx=0 tx=0).stop 2024-04-12T05:53:30.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.135+0000 7fb758975700 1 -- 172.21.15.32:0/541497195 shutdown_connections 2024-04-12T05:53:30.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.135+0000 7fb758975700 1 --2- 172.21.15.32:0/541497195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7540fbca0 0x7fb7540fc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:30.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.135+0000 7fb758975700 1 --2- 172.21.15.32:0/541497195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb754101cf0 0x7fb7541020c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:30.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.135+0000 7fb758975700 1 -- 172.21.15.32:0/541497195 >> 172.21.15.32:0/541497195 conn(0x7fb754072bc0 msgr2=0x7fb754072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:53:30.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.135+0000 7fb758975700 1 -- 172.21.15.32:0/541497195 shutdown_connections 2024-04-12T05:53:30.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.135+0000 7fb758975700 1 -- 172.21.15.32:0/541497195 wait complete. 2024-04-12T05:53:30.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.136+0000 7fb758975700 1 Processor -- start 2024-04-12T05:53:30.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.136+0000 7fb758975700 1 -- start start 2024-04-12T05:53:30.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.136+0000 7fb758975700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7540fbca0 0x7fb75410e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:53:30.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.136+0000 7fb758975700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb754101cf0 0x7fb75410ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:53:30.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.136+0000 7fb758975700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75410f4c0 con 0x7fb7540fbca0 2024-04-12T05:53:30.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.136+0000 7fb758975700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75410f600 con 0x7fb754101cf0 2024-04-12T05:53:30.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.136+0000 7fb75259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7540fbca0 0x7fb75410e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:53:30.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.136+0000 7fb75259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7540fbca0 0x7fb75410e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34766/0 (socket says 172.21.15.32:34766) 2024-04-12T05:53:30.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.136+0000 7fb75259c700 1 -- 172.21.15.32:0/443336147 learned_addr learned my addr 172.21.15.32:0/443336147 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:53:30.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.136+0000 7fb751d9b700 1 --2- 172.21.15.32:0/443336147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb754101cf0 0x7fb75410ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:53:30.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.137+0000 7fb75259c700 1 -- 172.21.15.32:0/443336147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb754101cf0 msgr2=0x7fb75410ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:53:30.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.137+0000 7fb75259c700 1 --2- 172.21.15.32:0/443336147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb754101cf0 0x7fb75410ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:30.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.137+0000 7fb75259c700 1 -- 172.21.15.32:0/443336147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb744005040 con 0x7fb7540fbca0 2024-04-12T05:53:30.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.137+0000 7fb75259c700 1 --2- 172.21.15.32:0/443336147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7540fbca0 0x7fb75410e930 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb744008d80 tx=0x7fb744009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:53:30.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.137+0000 7fb74affd700 1 -- 172.21.15.32:0/443336147 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb74400e030 con 0x7fb7540fbca0 2024-04-12T05:53:30.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.137+0000 7fb74affd700 1 -- 172.21.15.32:0/443336147 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb744013de0 con 0x7fb7540fbca0 2024-04-12T05:53:30.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.137+0000 7fb758975700 1 -- 172.21.15.32:0/443336147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7541179a0 con 0x7fb7540fbca0 2024-04-12T05:53:30.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.137+0000 7fb74affd700 1 -- 172.21.15.32:0/443336147 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb74401cd90 con 0x7fb7540fbca0 2024-04-12T05:53:30.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.137+0000 7fb758975700 1 -- 172.21.15.32:0/443336147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb754117eb0 con 0x7fb7540fbca0 2024-04-12T05:53:30.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.138+0000 7fb758975700 1 -- 172.21.15.32:0/443336147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7540fd390 con 0x7fb7540fbca0 2024-04-12T05:53:30.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.138+0000 7fb74affd700 1 -- 172.21.15.32:0/443336147 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb744013930 con 0x7fb7540fbca0 2024-04-12T05:53:30.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.139+0000 7fb74affd700 1 --2- 172.21.15.32:0/443336147 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb74006cc30 0x7fb74006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:53:30.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.139+0000 7fb751d9b700 1 --2- 172.21.15.32:0/443336147 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb74006cc30 0x7fb74006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:53:30.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.139+0000 7fb74affd700 1 -- 172.21.15.32:0/443336147 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fb74408d880 con 0x7fb7540fbca0 2024-04-12T05:53:30.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.140+0000 7fb751d9b700 1 --2- 172.21.15.32:0/443336147 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb74006cc30 0x7fb74006f0e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb754066000 tx=0x7fb73c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:53:30.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.144+0000 7fb74affd700 1 -- 172.21.15.32:0/443336147 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb744057900 con 0x7fb7540fbca0 2024-04-12T05:53:30.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:29 smithi067 ceph-mon[112979]: pgmap v389: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:30.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.432+0000 7fb758975700 1 -- 172.21.15.32:0/443336147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb75404bf00 con 0x7fb7540fbca0 2024-04-12T05:53:30.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.433+0000 7fb74affd700 1 -- 172.21.15.32:0/443336147 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fb74405af20 con 0x7fb7540fbca0 2024-04-12T05:53:30.429 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:53:30.429 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:53:30.429 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:53:30.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:53:30.429 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:53:30.429 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:53:30.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:53:30.429 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:53:30.429 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:53:30.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:53:30.430 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:53:30.430 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:53:30.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:53:30.430 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:53:30.430 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:53:30.430 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:53:30.430 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:53:30.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.435+0000 7fb758975700 1 -- 172.21.15.32:0/443336147 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb74006cc30 msgr2=0x7fb74006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:53:30.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.435+0000 7fb758975700 1 --2- 172.21.15.32:0/443336147 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb74006cc30 0x7fb74006f0e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb754066000 tx=0x7fb73c00b040 comp rx=0 tx=0).stop 2024-04-12T05:53:30.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.435+0000 7fb758975700 1 -- 172.21.15.32:0/443336147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7540fbca0 msgr2=0x7fb75410e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:53:30.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.435+0000 7fb758975700 1 --2- 172.21.15.32:0/443336147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7540fbca0 0x7fb75410e930 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb744008d80 tx=0x7fb744009e30 comp rx=0 tx=0).stop 2024-04-12T05:53:30.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.435+0000 7fb758975700 1 -- 172.21.15.32:0/443336147 shutdown_connections 2024-04-12T05:53:30.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.435+0000 7fb758975700 1 --2- 172.21.15.32:0/443336147 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb74006cc30 0x7fb74006f0e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:30.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.435+0000 7fb758975700 1 --2- 172.21.15.32:0/443336147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7540fbca0 0x7fb75410e930 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:30.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.435+0000 7fb758975700 1 --2- 172.21.15.32:0/443336147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb754101cf0 0x7fb75410ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:30.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.435+0000 7fb758975700 1 -- 172.21.15.32:0/443336147 >> 172.21.15.32:0/443336147 conn(0x7fb754072bc0 msgr2=0x7fb7541049c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:53:30.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.435+0000 7fb758975700 1 -- 172.21.15.32:0/443336147 shutdown_connections 2024-04-12T05:53:30.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.435+0000 7fb758975700 1 -- 172.21.15.32:0/443336147 wait complete. 2024-04-12T05:53:30.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.539+0000 7ff6c7f5e700 1 -- 172.21.15.32:0/1197974307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6c0102710 msgr2=0x7ff6c00fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:53:30.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.539+0000 7ff6c7f5e700 1 --2- 172.21.15.32:0/1197974307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6c0102710 0x7ff6c00fbd80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0004940 tx=0x7ff6b0009e30 comp rx=0 tx=0).stop 2024-04-12T05:53:30.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.539+0000 7ff6c7f5e700 1 -- 172.21.15.32:0/1197974307 shutdown_connections 2024-04-12T05:53:30.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.539+0000 7ff6c7f5e700 1 --2- 172.21.15.32:0/1197974307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6c0102710 0x7ff6c00fbd80 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:30.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.539+0000 7ff6c7f5e700 1 --2- 172.21.15.32:0/1197974307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6c0101d70 0x7ff6c0102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:30.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.539+0000 7ff6c7f5e700 1 -- 172.21.15.32:0/1197974307 >> 172.21.15.32:0/1197974307 conn(0x7ff6c0072bc0 msgr2=0x7ff6c0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:53:30.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.539+0000 7ff6c7f5e700 1 -- 172.21.15.32:0/1197974307 shutdown_connections 2024-04-12T05:53:30.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.539+0000 7ff6c7f5e700 1 -- 172.21.15.32:0/1197974307 wait complete. 2024-04-12T05:53:30.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.540+0000 7ff6c7f5e700 1 Processor -- start 2024-04-12T05:53:30.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.540+0000 7ff6c7f5e700 1 -- start start 2024-04-12T05:53:30.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.540+0000 7ff6c7f5e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6c0101d70 0x7ff6c010ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:53:30.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.540+0000 7ff6c7f5e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6c0102710 0x7ff6c010efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:53:30.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.540+0000 7ff6c7f5e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c010f5f0 con 0x7ff6c0101d70 2024-04-12T05:53:30.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.540+0000 7ff6c7f5e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c010f730 con 0x7ff6c0102710 2024-04-12T05:53:30.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.540+0000 7ff6c5cfa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6c0101d70 0x7ff6c010ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:53:30.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.540+0000 7ff6c5cfa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6c0101d70 0x7ff6c010ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34778/0 (socket says 172.21.15.32:34778) 2024-04-12T05:53:30.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.540+0000 7ff6c5cfa700 1 -- 172.21.15.32:0/2003321771 learned_addr learned my addr 172.21.15.32:0/2003321771 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:53:30.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.540+0000 7ff6c54f9700 1 --2- 172.21.15.32:0/2003321771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6c0102710 0x7ff6c010efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:53:30.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.540+0000 7ff6c5cfa700 1 -- 172.21.15.32:0/2003321771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6c0102710 msgr2=0x7ff6c010efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:53:30.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.540+0000 7ff6c5cfa700 1 --2- 172.21.15.32:0/2003321771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6c0102710 0x7ff6c010efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:30.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.540+0000 7ff6c5cfa700 1 -- 172.21.15.32:0/2003321771 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b0005040 con 0x7ff6c0101d70 2024-04-12T05:53:30.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.541+0000 7ff6c5cfa700 1 --2- 172.21.15.32:0/2003321771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6c0101d70 0x7ff6c010ea60 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc002a50 tx=0x7ff6bc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:53:30.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.541+0000 7ff6b67fc700 1 -- 172.21.15.32:0/2003321771 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff6bc007cc0 con 0x7ff6c0101d70 2024-04-12T05:53:30.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.541+0000 7ff6b67fc700 1 -- 172.21.15.32:0/2003321771 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff6bc00f040 con 0x7ff6c0101d70 2024-04-12T05:53:30.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.541+0000 7ff6c7f5e700 1 -- 172.21.15.32:0/2003321771 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6c0117ad0 con 0x7ff6c0101d70 2024-04-12T05:53:30.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.541+0000 7ff6b67fc700 1 -- 172.21.15.32:0/2003321771 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff6bc00d9c0 con 0x7ff6c0101d70 2024-04-12T05:53:30.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.541+0000 7ff6c7f5e700 1 -- 172.21.15.32:0/2003321771 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6c0118000 con 0x7ff6c0101d70 2024-04-12T05:53:30.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.542+0000 7ff6c7f5e700 1 -- 172.21.15.32:0/2003321771 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6c00ff080 con 0x7ff6c0101d70 2024-04-12T05:53:30.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.542+0000 7ff6b67fc700 1 -- 172.21.15.32:0/2003321771 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ff6bc00db20 con 0x7ff6c0101d70 2024-04-12T05:53:30.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.543+0000 7ff6b67fc700 1 --2- 172.21.15.32:0/2003321771 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff6ac06cc20 0x7ff6ac06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:53:30.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.543+0000 7ff6b67fc700 1 -- 172.21.15.32:0/2003321771 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7ff6bc08bdd0 con 0x7ff6c0101d70 2024-04-12T05:53:30.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.543+0000 7ff6c54f9700 1 --2- 172.21.15.32:0/2003321771 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff6ac06cc20 0x7ff6ac06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:53:30.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.543+0000 7ff6c54f9700 1 --2- 172.21.15.32:0/2003321771 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff6ac06cc20 0x7ff6ac06f0d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0004890 tx=0x7ff6b0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:53:30.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.547+0000 7ff6b67fc700 1 -- 172.21.15.32:0/2003321771 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff6bc055d30 con 0x7ff6c0101d70 2024-04-12T05:53:30.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.795+0000 7ff6c7f5e700 1 -- 172.21.15.32:0/2003321771 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff6c0108da0 con 0x7ff6c0101d70 2024-04-12T05:53:30.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.795+0000 7ff6b67fc700 1 -- 172.21.15.32:0/2003321771 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1835 (secure 0 0 0) 0x7ff6bc059350 con 0x7ff6c0101d70 2024-04-12T05:53:30.791 INFO:teuthology.orchestra.run.smithi032.stdout:e33 2024-04-12T05:53:30.791 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:53:30.791 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:53:30.791 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:53:30.792 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:53:30.792 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:53:30.792 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:53:30.792 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 24 2024-04-12T05:53:30.792 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:53:30.792 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:53:30.792 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:46:10.668569+0000 2024-04-12T05:53:30.792 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:53:30.792 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:53:30.792 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:53:30.792 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:53:30.792 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:53:30.792 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:53:30.792 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:53:30.792 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:53:30.792 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T05:53:30.792 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:53:30.793 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:53:30.793 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:53:30.793 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T05:53:30.793 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-12T05:53:30.793 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:53:30.793 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:53:30.793 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:53:30.793 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:53:30.793 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:53:30.793 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:53:30.793 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:53:30.793 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:53:30.793 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:53:30.794 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:53:30.794 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-12T05:53:30.794 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:53:30.794 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:53:30.794 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:53:30.794 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{-1:44317} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:53:30.794 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:53:30.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.798+0000 7ff6c7f5e700 1 -- 172.21.15.32:0/2003321771 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff6ac06cc20 msgr2=0x7ff6ac06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:53:30.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.798+0000 7ff6c7f5e700 1 --2- 172.21.15.32:0/2003321771 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff6ac06cc20 0x7ff6ac06f0d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0004890 tx=0x7ff6b0009e30 comp rx=0 tx=0).stop 2024-04-12T05:53:30.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.798+0000 7ff6c7f5e700 1 -- 172.21.15.32:0/2003321771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6c0101d70 msgr2=0x7ff6c010ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:53:30.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.798+0000 7ff6c7f5e700 1 --2- 172.21.15.32:0/2003321771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6c0101d70 0x7ff6c010ea60 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc002a50 tx=0x7ff6bc0046b0 comp rx=0 tx=0).stop 2024-04-12T05:53:30.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.798+0000 7ff6c7f5e700 1 -- 172.21.15.32:0/2003321771 shutdown_connections 2024-04-12T05:53:30.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.798+0000 7ff6c7f5e700 1 --2- 172.21.15.32:0/2003321771 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff6ac06cc20 0x7ff6ac06f0d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:30.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.798+0000 7ff6c7f5e700 1 --2- 172.21.15.32:0/2003321771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6c0101d70 0x7ff6c010ea60 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:30.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.798+0000 7ff6c7f5e700 1 --2- 172.21.15.32:0/2003321771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6c0102710 0x7ff6c010efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:30.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.798+0000 7ff6c7f5e700 1 -- 172.21.15.32:0/2003321771 >> 172.21.15.32:0/2003321771 conn(0x7ff6c0072bc0 msgr2=0x7ff6c00fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:53:30.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.798+0000 7ff6c7f5e700 1 -- 172.21.15.32:0/2003321771 shutdown_connections 2024-04-12T05:53:30.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.798+0000 7ff6c7f5e700 1 -- 172.21.15.32:0/2003321771 wait complete. 2024-04-12T05:53:30.795 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-12T05:53:30.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.902+0000 7f78171da700 1 -- 172.21.15.32:0/1483701987 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78100fde90 msgr2=0x7f78100fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:53:30.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.902+0000 7f78171da700 1 --2- 172.21.15.32:0/1483701987 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78100fde90 0x7f78100fe260 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7804004940 tx=0x7f7804009e30 comp rx=0 tx=0).stop 2024-04-12T05:53:30.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.902+0000 7f78171da700 1 -- 172.21.15.32:0/1483701987 shutdown_connections 2024-04-12T05:53:30.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.902+0000 7f78171da700 1 --2- 172.21.15.32:0/1483701987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78100fe830 0x7f7810106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:30.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.902+0000 7f78171da700 1 --2- 172.21.15.32:0/1483701987 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78100fde90 0x7f78100fe260 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:30.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.902+0000 7f78171da700 1 -- 172.21.15.32:0/1483701987 >> 172.21.15.32:0/1483701987 conn(0x7f7810072bc0 msgr2=0x7f7810072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:53:30.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.902+0000 7f78171da700 1 -- 172.21.15.32:0/1483701987 shutdown_connections 2024-04-12T05:53:30.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.902+0000 7f78171da700 1 -- 172.21.15.32:0/1483701987 wait complete. 2024-04-12T05:53:30.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.903+0000 7f78171da700 1 Processor -- start 2024-04-12T05:53:30.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.903+0000 7f78171da700 1 -- start start 2024-04-12T05:53:30.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.903+0000 7f78171da700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78100fde90 0x7f7810113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:53:30.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.903+0000 7f78171da700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78100fe830 0x7f7810114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:53:30.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.903+0000 7f78171da700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7810114680 con 0x7f78100fe830 2024-04-12T05:53:30.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.903+0000 7f78171da700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f781010da70 con 0x7f78100fde90 2024-04-12T05:53:30.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.903+0000 7f780ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78100fe830 0x7f7810114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:53:30.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.903+0000 7f780ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78100fe830 0x7f7810114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34798/0 (socket says 172.21.15.32:34798) 2024-04-12T05:53:30.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.903+0000 7f780ffff700 1 -- 172.21.15.32:0/481399214 learned_addr learned my addr 172.21.15.32:0/481399214 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:53:30.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.903+0000 7f7814f76700 1 --2- 172.21.15.32:0/481399214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78100fde90 0x7f7810113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:53:30.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.904+0000 7f780ffff700 1 -- 172.21.15.32:0/481399214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78100fde90 msgr2=0x7f7810113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:53:30.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.904+0000 7f780ffff700 1 --2- 172.21.15.32:0/481399214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78100fde90 0x7f7810113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:30.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.904+0000 7f780ffff700 1 -- 172.21.15.32:0/481399214 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7804005040 con 0x7f78100fe830 2024-04-12T05:53:30.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.904+0000 7f780ffff700 1 --2- 172.21.15.32:0/481399214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78100fe830 0x7f7810114030 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f77fc002a50 tx=0x7f77fc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:53:30.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.904+0000 7f780d7fa700 1 -- 172.21.15.32:0/481399214 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77fc007cc0 con 0x7f78100fe830 2024-04-12T05:53:30.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.904+0000 7f780d7fa700 1 -- 172.21.15.32:0/481399214 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f77fc00f040 con 0x7f78100fe830 2024-04-12T05:53:30.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.904+0000 7f78171da700 1 -- 172.21.15.32:0/481399214 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f781010dd50 con 0x7f78100fe830 2024-04-12T05:53:30.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.904+0000 7f780d7fa700 1 -- 172.21.15.32:0/481399214 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77fc00d9c0 con 0x7f78100fe830 2024-04-12T05:53:30.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.904+0000 7f78171da700 1 -- 172.21.15.32:0/481399214 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f781010e290 con 0x7f78100fe830 2024-04-12T05:53:30.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.905+0000 7f78171da700 1 -- 172.21.15.32:0/481399214 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78101019b0 con 0x7f78100fe830 2024-04-12T05:53:30.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.906+0000 7f780d7fa700 1 -- 172.21.15.32:0/481399214 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f77fc00db20 con 0x7f78100fe830 2024-04-12T05:53:30.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.906+0000 7f780d7fa700 1 --2- 172.21.15.32:0/481399214 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f780006cc30 0x7f780006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:53:30.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.906+0000 7f780d7fa700 1 -- 172.21.15.32:0/481399214 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f77fc08be70 con 0x7f78100fe830 2024-04-12T05:53:30.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.906+0000 7f7814f76700 1 --2- 172.21.15.32:0/481399214 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f780006cc30 0x7f780006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:53:30.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.906+0000 7f7814f76700 1 --2- 172.21.15.32:0/481399214 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f780006cc30 0x7f780006f0e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f780400b3b0 tx=0x7f7804009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:53:30.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:30.910+0000 7f780d7fa700 1 -- 172.21.15.32:0/481399214 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f77fc0597c0 con 0x7f78100fe830 2024-04-12T05:53:31.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:30 smithi032 ceph-mon[146817]: from='client.34760 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:53:31.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:30 smithi032 ceph-mon[146817]: from='client.34764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:53:31.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:30 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/443336147' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:53:31.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.132+0000 7f78171da700 1 -- 172.21.15.32:0/481399214 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f78101137c0 con 0x7f780006cc30 2024-04-12T05:53:31.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.133+0000 7f780d7fa700 1 -- 172.21.15.32:0/481399214 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f78101137c0 con 0x7f780006cc30 2024-04-12T05:53:31.129 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:53:31.129 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:53:31.129 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:53:31.129 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:53:31.129 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:53:31.129 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-12T05:53:31.129 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:53:31.129 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-12T05:53:31.129 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:53:31.129 INFO:teuthology.orchestra.run.smithi032.stdout: "mds" 2024-04-12T05:53:31.129 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:53:31.129 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:53:31.129 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:53:31.130 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:53:31.130 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:53:31.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.135+0000 7f78171da700 1 -- 172.21.15.32:0/481399214 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f780006cc30 msgr2=0x7f780006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:53:31.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.135+0000 7f78171da700 1 --2- 172.21.15.32:0/481399214 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f780006cc30 0x7f780006f0e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f780400b3b0 tx=0x7f7804009e30 comp rx=0 tx=0).stop 2024-04-12T05:53:31.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.135+0000 7f78171da700 1 -- 172.21.15.32:0/481399214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78100fe830 msgr2=0x7f7810114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:53:31.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.135+0000 7f78171da700 1 --2- 172.21.15.32:0/481399214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78100fe830 0x7f7810114030 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f77fc002a50 tx=0x7f77fc0046b0 comp rx=0 tx=0).stop 2024-04-12T05:53:31.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.135+0000 7f78171da700 1 -- 172.21.15.32:0/481399214 shutdown_connections 2024-04-12T05:53:31.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.135+0000 7f78171da700 1 --2- 172.21.15.32:0/481399214 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f780006cc30 0x7f780006f0e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:31.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.135+0000 7f78171da700 1 --2- 172.21.15.32:0/481399214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78100fde90 0x7f7810113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:31.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.135+0000 7f78171da700 1 --2- 172.21.15.32:0/481399214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78100fe830 0x7f7810114030 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:31.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.135+0000 7f78171da700 1 -- 172.21.15.32:0/481399214 >> 172.21.15.32:0/481399214 conn(0x7f7810072bc0 msgr2=0x7f78100ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:53:31.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.135+0000 7f78171da700 1 -- 172.21.15.32:0/481399214 shutdown_connections 2024-04-12T05:53:31.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.135+0000 7f78171da700 1 -- 172.21.15.32:0/481399214 wait complete. 2024-04-12T05:53:31.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.238+0000 7ff5b1f2e700 1 -- 172.21.15.32:0/3591520147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5ac0fde90 msgr2=0x7ff5ac0fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:53:31.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.238+0000 7ff5b1f2e700 1 --2- 172.21.15.32:0/3591520147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5ac0fde90 0x7ff5ac0fe260 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff59c004940 tx=0x7ff59c009e30 comp rx=0 tx=0).stop 2024-04-12T05:53:31.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.238+0000 7ff5b1f2e700 1 -- 172.21.15.32:0/3591520147 shutdown_connections 2024-04-12T05:53:31.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.238+0000 7ff5b1f2e700 1 --2- 172.21.15.32:0/3591520147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5ac0fe830 0x7ff5ac106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:31.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.238+0000 7ff5b1f2e700 1 --2- 172.21.15.32:0/3591520147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5ac0fde90 0x7ff5ac0fe260 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:31.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.238+0000 7ff5b1f2e700 1 -- 172.21.15.32:0/3591520147 >> 172.21.15.32:0/3591520147 conn(0x7ff5ac072bc0 msgr2=0x7ff5ac072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:53:31.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.238+0000 7ff5b1f2e700 1 -- 172.21.15.32:0/3591520147 shutdown_connections 2024-04-12T05:53:31.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.238+0000 7ff5b1f2e700 1 -- 172.21.15.32:0/3591520147 wait complete. 2024-04-12T05:53:31.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.238+0000 7ff5b1f2e700 1 Processor -- start 2024-04-12T05:53:31.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.239+0000 7ff5b1f2e700 1 -- start start 2024-04-12T05:53:31.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.239+0000 7ff5b1f2e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5ac0fde90 0x7ff5ac0fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:53:31.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.239+0000 7ff5b1f2e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5ac0fe830 0x7ff5ac0fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:53:31.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.239+0000 7ff5b1f2e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ac0fc820 con 0x7ff5ac0fe830 2024-04-12T05:53:31.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.239+0000 7ff5b1f2e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ac0fc960 con 0x7ff5ac0fde90 2024-04-12T05:53:31.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.239+0000 7ff5aaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5ac0fe830 0x7ff5ac0fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:53:31.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.239+0000 7ff5aaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5ac0fe830 0x7ff5ac0fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34818/0 (socket says 172.21.15.32:34818) 2024-04-12T05:53:31.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.239+0000 7ff5aaffd700 1 -- 172.21.15.32:0/1213757766 learned_addr learned my addr 172.21.15.32:0/1213757766 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:53:31.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.239+0000 7ff5ab7fe700 1 --2- 172.21.15.32:0/1213757766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5ac0fde90 0x7ff5ac0fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:53:31.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.239+0000 7ff5aaffd700 1 -- 172.21.15.32:0/1213757766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5ac0fde90 msgr2=0x7ff5ac0fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:53:31.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.239+0000 7ff5aaffd700 1 --2- 172.21.15.32:0/1213757766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5ac0fde90 0x7ff5ac0fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:31.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.239+0000 7ff5aaffd700 1 -- 172.21.15.32:0/1213757766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff59c005040 con 0x7ff5ac0fe830 2024-04-12T05:53:31.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.239+0000 7ff5aaffd700 1 --2- 172.21.15.32:0/1213757766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5ac0fe830 0x7ff5ac0fc290 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0002a50 tx=0x7ff5a00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:53:31.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.240+0000 7ff593fff700 1 -- 172.21.15.32:0/1213757766 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5a0007cc0 con 0x7ff5ac0fe830 2024-04-12T05:53:31.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.240+0000 7ff5b1f2e700 1 -- 172.21.15.32:0/1213757766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5ac119df0 con 0x7ff5ac0fe830 2024-04-12T05:53:31.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.240+0000 7ff593fff700 1 -- 172.21.15.32:0/1213757766 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff5a000f040 con 0x7ff5ac0fe830 2024-04-12T05:53:31.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.240+0000 7ff593fff700 1 -- 172.21.15.32:0/1213757766 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5a000d9c0 con 0x7ff5ac0fe830 2024-04-12T05:53:31.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.240+0000 7ff5b1f2e700 1 -- 172.21.15.32:0/1213757766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5ac11a1f0 con 0x7ff5ac0fe830 2024-04-12T05:53:31.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.240+0000 7ff5b1f2e700 1 -- 172.21.15.32:0/1213757766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5ac065e30 con 0x7ff5ac0fe830 2024-04-12T05:53:31.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.241+0000 7ff593fff700 1 -- 172.21.15.32:0/1213757766 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ff5a000db20 con 0x7ff5ac0fe830 2024-04-12T05:53:31.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.241+0000 7ff593fff700 1 --2- 172.21.15.32:0/1213757766 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff59406cc20 0x7ff59406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:53:31.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.242+0000 7ff593fff700 1 -- 172.21.15.32:0/1213757766 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7ff5a008be10 con 0x7ff5ac0fe830 2024-04-12T05:53:31.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.242+0000 7ff5ab7fe700 1 --2- 172.21.15.32:0/1213757766 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff59406cc20 0x7ff59406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:53:31.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.242+0000 7ff5ab7fe700 1 --2- 172.21.15.32:0/1213757766 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff59406cc20 0x7ff59406f0d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff59c00b7e0 tx=0x7ff59c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:53:31.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.246+0000 7ff593fff700 1 -- 172.21.15.32:0/1213757766 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff5a0055d70 con 0x7ff5ac0fe830 2024-04-12T05:53:31.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:30 smithi067 ceph-mon[112979]: from='client.34760 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:53:31.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:30 smithi067 ceph-mon[112979]: from='client.34764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:53:31.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:30 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/443336147' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:53:31.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.535+0000 7ff5b1f2e700 1 -- 172.21.15.32:0/1213757766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff5ac0fd440 con 0x7ff5ac0fe830 2024-04-12T05:53:31.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.535+0000 7ff593fff700 1 -- 172.21.15.32:0/1213757766 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7ff5a0059390 con 0x7ff5ac0fe830 2024-04-12T05:53:31.531 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:53:31.531 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:53:31.531 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-12T05:53:31.531 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:53:31.531 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:53:31.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.537+0000 7ff5b1f2e700 1 -- 172.21.15.32:0/1213757766 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff59406cc20 msgr2=0x7ff59406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:53:31.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.537+0000 7ff5b1f2e700 1 --2- 172.21.15.32:0/1213757766 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff59406cc20 0x7ff59406f0d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff59c00b7e0 tx=0x7ff59c009e30 comp rx=0 tx=0).stop 2024-04-12T05:53:31.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.537+0000 7ff5b1f2e700 1 -- 172.21.15.32:0/1213757766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5ac0fe830 msgr2=0x7ff5ac0fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:53:31.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.537+0000 7ff5b1f2e700 1 --2- 172.21.15.32:0/1213757766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5ac0fe830 0x7ff5ac0fc290 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0002a50 tx=0x7ff5a00046b0 comp rx=0 tx=0).stop 2024-04-12T05:53:31.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.537+0000 7ff5b1f2e700 1 -- 172.21.15.32:0/1213757766 shutdown_connections 2024-04-12T05:53:31.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.537+0000 7ff5b1f2e700 1 --2- 172.21.15.32:0/1213757766 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff59406cc20 0x7ff59406f0d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:31.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.537+0000 7ff5b1f2e700 1 --2- 172.21.15.32:0/1213757766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5ac0fde90 0x7ff5ac0fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:31.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.537+0000 7ff5b1f2e700 1 --2- 172.21.15.32:0/1213757766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5ac0fe830 0x7ff5ac0fc290 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:53:31.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.537+0000 7ff5b1f2e700 1 -- 172.21.15.32:0/1213757766 >> 172.21.15.32:0/1213757766 conn(0x7ff5ac072bc0 msgr2=0x7ff5ac0ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:53:31.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.537+0000 7ff5b1f2e700 1 -- 172.21.15.32:0/1213757766 shutdown_connections 2024-04-12T05:53:31.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:53:31.537+0000 7ff5b1f2e700 1 -- 172.21.15.32:0/1213757766 wait complete. 2024-04-12T05:53:32.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:31 smithi032 ceph-mon[146817]: pgmap v390: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:32.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:31 smithi032 ceph-mon[146817]: from='client.34768 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:53:32.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:31 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/2003321771' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:53:32.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:31 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1213757766' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:53:32.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:31 smithi067 ceph-mon[112979]: pgmap v390: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:32.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:31 smithi067 ceph-mon[112979]: from='client.34768 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:53:32.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:31 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/2003321771' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:53:32.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:31 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1213757766' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:53:33.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:32 smithi032 ceph-mon[146817]: from='client.34780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:53:33.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:32 smithi067 ceph-mon[112979]: from='client.34780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:53:34.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:33 smithi032 ceph-mon[146817]: pgmap v391: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:34.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:33 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:53:34.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:33 smithi067 ceph-mon[112979]: pgmap v391: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:34.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:33 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:53:36.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:35 smithi032 ceph-mon[146817]: pgmap v392: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:36.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:35 smithi067 ceph-mon[112979]: pgmap v392: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:38.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:37 smithi032 ceph-mon[146817]: pgmap v393: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:38.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:37 smithi067 ceph-mon[112979]: pgmap v393: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:40.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:39 smithi032 ceph-mon[146817]: pgmap v394: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:40.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:39 smithi067 ceph-mon[112979]: pgmap v394: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:42.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:41 smithi032 ceph-mon[146817]: pgmap v395: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:42.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:41 smithi067 ceph-mon[112979]: pgmap v395: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:44.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:43 smithi032 ceph-mon[146817]: pgmap v396: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:44.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:43 smithi067 ceph-mon[112979]: pgmap v396: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:46.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:45 smithi032 ceph-mon[146817]: pgmap v397: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:46.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:45 smithi067 ceph-mon[112979]: pgmap v397: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:48.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:47 smithi032 ceph-mon[146817]: pgmap v398: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:48.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:47 smithi067 ceph-mon[112979]: pgmap v398: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:49.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:48 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:53:49.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:48 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:53:50.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:49 smithi032 ceph-mon[146817]: pgmap v399: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:50.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:49 smithi067 ceph-mon[112979]: pgmap v399: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:52.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:51 smithi032 ceph-mon[146817]: pgmap v400: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:52.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:51 smithi067 ceph-mon[112979]: pgmap v400: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:54.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:53 smithi032 ceph-mon[146817]: pgmap v401: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:54.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:53 smithi067 ceph-mon[112979]: pgmap v401: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:55.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:54 smithi032 ceph-mon[146817]: pgmap v402: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:55.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:54 smithi067 ceph-mon[112979]: pgmap v402: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:57.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:57 smithi067 ceph-mon[112979]: pgmap v403: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:57.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:57 smithi032 ceph-mon[146817]: pgmap v403: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:59.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:53:59 smithi067 ceph-mon[112979]: pgmap v404: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:59.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:53:59 smithi032 ceph-mon[146817]: pgmap v404: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:01.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:01 smithi067 ceph-mon[112979]: pgmap v405: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:01.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:01 smithi032 ceph-mon[146817]: pgmap v405: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:01.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.654+0000 7fbfbaa6d700 1 -- 172.21.15.32:0/429499107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfb4101d70 msgr2=0x7fbfb4102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:01.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.654+0000 7fbfbaa6d700 1 --2- 172.21.15.32:0/429499107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfb4101d70 0x7fbfb4102140 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8004940 tx=0x7fbfa8009e30 comp rx=0 tx=0).stop 2024-04-12T05:54:01.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.654+0000 7fbfbaa6d700 1 -- 172.21.15.32:0/429499107 shutdown_connections 2024-04-12T05:54:01.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.654+0000 7fbfbaa6d700 1 --2- 172.21.15.32:0/429499107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfb4102710 0x7fbfb40fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:01.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.654+0000 7fbfbaa6d700 1 --2- 172.21.15.32:0/429499107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfb4101d70 0x7fbfb4102140 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:01.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.654+0000 7fbfbaa6d700 1 -- 172.21.15.32:0/429499107 >> 172.21.15.32:0/429499107 conn(0x7fbfb4072bc0 msgr2=0x7fbfb4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:54:01.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.654+0000 7fbfbaa6d700 1 -- 172.21.15.32:0/429499107 shutdown_connections 2024-04-12T05:54:01.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.654+0000 7fbfbaa6d700 1 -- 172.21.15.32:0/429499107 wait complete. 2024-04-12T05:54:01.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.655+0000 7fbfbaa6d700 1 Processor -- start 2024-04-12T05:54:01.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.655+0000 7fbfbaa6d700 1 -- start start 2024-04-12T05:54:01.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.655+0000 7fbfbaa6d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfb4101d70 0x7fbfb4113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:01.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.655+0000 7fbfbaa6d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfb4102710 0x7fbfb4113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:01.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.655+0000 7fbfbaa6d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb41145f0 con 0x7fbfb4101d70 2024-04-12T05:54:01.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.655+0000 7fbfbaa6d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb410d9e0 con 0x7fbfb4102710 2024-04-12T05:54:01.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.655+0000 7fbfb8809700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfb4101d70 0x7fbfb4113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:01.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.656+0000 7fbfb8809700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfb4101d70 0x7fbfb4113a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45012/0 (socket says 172.21.15.32:45012) 2024-04-12T05:54:01.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.656+0000 7fbfb8809700 1 -- 172.21.15.32:0/3825547960 learned_addr learned my addr 172.21.15.32:0/3825547960 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:54:01.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.656+0000 7fbfb3fff700 1 --2- 172.21.15.32:0/3825547960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfb4102710 0x7fbfb4113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:01.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.656+0000 7fbfb8809700 1 -- 172.21.15.32:0/3825547960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfb4102710 msgr2=0x7fbfb4113fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:01.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.656+0000 7fbfb8809700 1 --2- 172.21.15.32:0/3825547960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfb4102710 0x7fbfb4113fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:01.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.656+0000 7fbfb8809700 1 -- 172.21.15.32:0/3825547960 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa8005040 con 0x7fbfb4101d70 2024-04-12T05:54:01.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.656+0000 7fbfb8809700 1 --2- 172.21.15.32:0/3825547960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfb4101d70 0x7fbfb4113a60 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8009360 tx=0x7fbfa8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:54:01.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.656+0000 7fbfb17fa700 1 -- 172.21.15.32:0/3825547960 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfa800e040 con 0x7fbfb4101d70 2024-04-12T05:54:01.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.656+0000 7fbfb17fa700 1 -- 172.21.15.32:0/3825547960 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbfa8013420 con 0x7fbfb4101d70 2024-04-12T05:54:01.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.656+0000 7fbfbaa6d700 1 -- 172.21.15.32:0/3825547960 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfb410dc60 con 0x7fbfb4101d70 2024-04-12T05:54:01.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.656+0000 7fbfb17fa700 1 -- 172.21.15.32:0/3825547960 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfa801c5f0 con 0x7fbfb4101d70 2024-04-12T05:54:01.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.656+0000 7fbfbaa6d700 1 -- 172.21.15.32:0/3825547960 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfb410e220 con 0x7fbfb4101d70 2024-04-12T05:54:01.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.657+0000 7fbfbaa6d700 1 -- 172.21.15.32:0/3825547960 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfb40fd4c0 con 0x7fbfb4101d70 2024-04-12T05:54:01.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.658+0000 7fbfb17fa700 1 -- 172.21.15.32:0/3825547960 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fbfa8013590 con 0x7fbfb4101d70 2024-04-12T05:54:01.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.658+0000 7fbfb17fa700 1 --2- 172.21.15.32:0/3825547960 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbfa406cc30 0x7fbfa406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:01.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.658+0000 7fbfb17fa700 1 -- 172.21.15.32:0/3825547960 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fbfa808d330 con 0x7fbfb4101d70 2024-04-12T05:54:01.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.658+0000 7fbfb3fff700 1 --2- 172.21.15.32:0/3825547960 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbfa406cc30 0x7fbfa406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:01.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.659+0000 7fbfb3fff700 1 --2- 172.21.15.32:0/3825547960 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbfa406cc30 0x7fbfa406f0e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbfb410eba0 tx=0x7fbfa0009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:54:01.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.663+0000 7fbfb17fa700 1 -- 172.21.15.32:0/3825547960 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbfa805ada0 con 0x7fbfb4101d70 2024-04-12T05:54:01.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.885+0000 7fbfbaa6d700 1 -- 172.21.15.32:0/3825547960 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbfb4108da0 con 0x7fbfa406cc30 2024-04-12T05:54:01.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.886+0000 7fbfb17fa700 1 -- 172.21.15.32:0/3825547960 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fbfb4108da0 con 0x7fbfa406cc30 2024-04-12T05:54:01.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.888+0000 7fbfbaa6d700 1 -- 172.21.15.32:0/3825547960 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbfa406cc30 msgr2=0x7fbfa406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:01.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.888+0000 7fbfbaa6d700 1 --2- 172.21.15.32:0/3825547960 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbfa406cc30 0x7fbfa406f0e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbfb410eba0 tx=0x7fbfa0009ee0 comp rx=0 tx=0).stop 2024-04-12T05:54:01.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.888+0000 7fbfbaa6d700 1 -- 172.21.15.32:0/3825547960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfb4101d70 msgr2=0x7fbfb4113a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:01.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.888+0000 7fbfbaa6d700 1 --2- 172.21.15.32:0/3825547960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfb4101d70 0x7fbfb4113a60 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8009360 tx=0x7fbfa8009e30 comp rx=0 tx=0).stop 2024-04-12T05:54:01.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.888+0000 7fbfbaa6d700 1 -- 172.21.15.32:0/3825547960 shutdown_connections 2024-04-12T05:54:01.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.888+0000 7fbfbaa6d700 1 --2- 172.21.15.32:0/3825547960 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbfa406cc30 0x7fbfa406f0e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:01.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.888+0000 7fbfbaa6d700 1 --2- 172.21.15.32:0/3825547960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfb4101d70 0x7fbfb4113a60 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:01.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.888+0000 7fbfbaa6d700 1 --2- 172.21.15.32:0/3825547960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfb4102710 0x7fbfb4113fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:01.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.888+0000 7fbfbaa6d700 1 -- 172.21.15.32:0/3825547960 >> 172.21.15.32:0/3825547960 conn(0x7fbfb4072bc0 msgr2=0x7fbfb40659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:54:01.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.888+0000 7fbfbaa6d700 1 -- 172.21.15.32:0/3825547960 shutdown_connections 2024-04-12T05:54:01.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.888+0000 7fbfbaa6d700 1 -- 172.21.15.32:0/3825547960 wait complete. 2024-04-12T05:54:01.894 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:54:01.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.993+0000 7f917e5fd700 1 -- 172.21.15.32:0/2715881869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9178101d70 msgr2=0x7f9178102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:01.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.993+0000 7f917e5fd700 1 --2- 172.21.15.32:0/2715881869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9178101d70 0x7f9178102140 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9168004940 tx=0x7f9168009e30 comp rx=0 tx=0).stop 2024-04-12T05:54:01.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.993+0000 7f917e5fd700 1 -- 172.21.15.32:0/2715881869 shutdown_connections 2024-04-12T05:54:01.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.993+0000 7f917e5fd700 1 --2- 172.21.15.32:0/2715881869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9178102710 0x7f91780fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:01.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.993+0000 7f917e5fd700 1 --2- 172.21.15.32:0/2715881869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9178101d70 0x7f9178102140 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:01.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.993+0000 7f917e5fd700 1 -- 172.21.15.32:0/2715881869 >> 172.21.15.32:0/2715881869 conn(0x7f9178072bc0 msgr2=0x7f9178072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:54:01.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.993+0000 7f917e5fd700 1 -- 172.21.15.32:0/2715881869 shutdown_connections 2024-04-12T05:54:01.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.993+0000 7f917e5fd700 1 -- 172.21.15.32:0/2715881869 wait complete. 2024-04-12T05:54:01.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.993+0000 7f917e5fd700 1 Processor -- start 2024-04-12T05:54:01.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.993+0000 7f917e5fd700 1 -- start start 2024-04-12T05:54:01.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.994+0000 7f917e5fd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9178101d70 0x7f917810c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:01.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.994+0000 7f917e5fd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9178102710 0x7f917810cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:01.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.994+0000 7f917e5fd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917810d2b0 con 0x7f9178101d70 2024-04-12T05:54:01.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.994+0000 7f917e5fd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917810d420 con 0x7f9178102710 2024-04-12T05:54:01.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.994+0000 7f9177fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9178101d70 0x7f917810c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:01.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.994+0000 7f9177fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9178101d70 0x7f917810c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45026/0 (socket says 172.21.15.32:45026) 2024-04-12T05:54:01.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.994+0000 7f9177fff700 1 -- 172.21.15.32:0/2439030531 learned_addr learned my addr 172.21.15.32:0/2439030531 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:54:01.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.994+0000 7f91777fe700 1 --2- 172.21.15.32:0/2439030531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9178102710 0x7f917810cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:01.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.994+0000 7f9177fff700 1 -- 172.21.15.32:0/2439030531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9178102710 msgr2=0x7f917810cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:01.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.994+0000 7f9177fff700 1 --2- 172.21.15.32:0/2439030531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9178102710 0x7f917810cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:01.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.994+0000 7f9177fff700 1 -- 172.21.15.32:0/2439030531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9168005040 con 0x7f9178101d70 2024-04-12T05:54:01.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.994+0000 7f9177fff700 1 --2- 172.21.15.32:0/2439030531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9178101d70 0x7f917810c830 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f9168009180 tx=0x7f9168009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:54:01.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.994+0000 7f9174ff9700 1 -- 172.21.15.32:0/2439030531 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f916800e040 con 0x7f9178101d70 2024-04-12T05:54:01.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.994+0000 7f917e5fd700 1 -- 172.21.15.32:0/2439030531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f917810d6a0 con 0x7f9178101d70 2024-04-12T05:54:01.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.994+0000 7f9174ff9700 1 -- 172.21.15.32:0/2439030531 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9168004290 con 0x7f9178101d70 2024-04-12T05:54:01.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.994+0000 7f9174ff9700 1 -- 172.21.15.32:0/2439030531 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9168013e60 con 0x7f9178101d70 2024-04-12T05:54:01.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.995+0000 7f917e5fd700 1 -- 172.21.15.32:0/2439030531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f917811a220 con 0x7f9178101d70 2024-04-12T05:54:01.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.995+0000 7f917e5fd700 1 -- 172.21.15.32:0/2439030531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91780ff080 con 0x7f9178101d70 2024-04-12T05:54:01.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.996+0000 7f9174ff9700 1 -- 172.21.15.32:0/2439030531 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f91680136b0 con 0x7f9178101d70 2024-04-12T05:54:01.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.996+0000 7f9174ff9700 1 --2- 172.21.15.32:0/2439030531 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f916406cbb0 0x7f916406f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:01.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.996+0000 7f9174ff9700 1 -- 172.21.15.32:0/2439030531 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f916808d0c0 con 0x7f9178101d70 2024-04-12T05:54:01.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.996+0000 7f91777fe700 1 --2- 172.21.15.32:0/2439030531 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f916406cbb0 0x7f916406f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:01.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:01.997+0000 7f91777fe700 1 --2- 172.21.15.32:0/2439030531 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f916406cbb0 0x7f916406f060 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9178066000 tx=0x7f91600043c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:54:01.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.001+0000 7f9174ff9700 1 -- 172.21.15.32:0/2439030531 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f916805ab30 con 0x7f9178101d70 2024-04-12T05:54:02.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.224+0000 7f917e5fd700 1 -- 172.21.15.32:0/2439030531 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f917810d6a0 con 0x7f916406cbb0 2024-04-12T05:54:02.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.225+0000 7f9174ff9700 1 -- 172.21.15.32:0/2439030531 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f917810d6a0 con 0x7f916406cbb0 2024-04-12T05:54:02.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.227+0000 7f917e5fd700 1 -- 172.21.15.32:0/2439030531 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f916406cbb0 msgr2=0x7f916406f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:02.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.227+0000 7f917e5fd700 1 --2- 172.21.15.32:0/2439030531 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f916406cbb0 0x7f916406f060 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9178066000 tx=0x7f91600043c0 comp rx=0 tx=0).stop 2024-04-12T05:54:02.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.227+0000 7f917e5fd700 1 -- 172.21.15.32:0/2439030531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9178101d70 msgr2=0x7f917810c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:02.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.227+0000 7f917e5fd700 1 --2- 172.21.15.32:0/2439030531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9178101d70 0x7f917810c830 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f9168009180 tx=0x7f9168009e30 comp rx=0 tx=0).stop 2024-04-12T05:54:02.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.227+0000 7f917e5fd700 1 -- 172.21.15.32:0/2439030531 shutdown_connections 2024-04-12T05:54:02.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.227+0000 7f917e5fd700 1 --2- 172.21.15.32:0/2439030531 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f916406cbb0 0x7f916406f060 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:02.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.227+0000 7f917e5fd700 1 --2- 172.21.15.32:0/2439030531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9178101d70 0x7f917810c830 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:02.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.227+0000 7f917e5fd700 1 --2- 172.21.15.32:0/2439030531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9178102710 0x7f917810cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:02.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.227+0000 7f917e5fd700 1 -- 172.21.15.32:0/2439030531 >> 172.21.15.32:0/2439030531 conn(0x7f9178072bc0 msgr2=0x7f91780fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:54:02.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.227+0000 7f917e5fd700 1 -- 172.21.15.32:0/2439030531 shutdown_connections 2024-04-12T05:54:02.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.227+0000 7f917e5fd700 1 -- 172.21.15.32:0/2439030531 wait complete. 2024-04-12T05:54:02.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.330+0000 7f949408c700 1 -- 172.21.15.32:0/4092445915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f948c0702e0 msgr2=0x7f948c0706b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:02.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.330+0000 7f949408c700 1 --2- 172.21.15.32:0/4092445915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f948c0702e0 0x7f948c0706b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f9488009ce0 tx=0x7f9488004d10 comp rx=0 tx=0).stop 2024-04-12T05:54:02.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.330+0000 7f949408c700 1 -- 172.21.15.32:0/4092445915 shutdown_connections 2024-04-12T05:54:02.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.330+0000 7f949408c700 1 --2- 172.21.15.32:0/4092445915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f948c070bf0 0x7f948c1091a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:02.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.330+0000 7f949408c700 1 --2- 172.21.15.32:0/4092445915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f948c0702e0 0x7f948c0706b0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:02.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.330+0000 7f949408c700 1 -- 172.21.15.32:0/4092445915 >> 172.21.15.32:0/4092445915 conn(0x7f948c0f89a0 msgr2=0x7f948c0fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:54:02.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.331+0000 7f949408c700 1 -- 172.21.15.32:0/4092445915 shutdown_connections 2024-04-12T05:54:02.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.331+0000 7f949408c700 1 -- 172.21.15.32:0/4092445915 wait complete. 2024-04-12T05:54:02.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.331+0000 7f949408c700 1 Processor -- start 2024-04-12T05:54:02.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.331+0000 7f949408c700 1 -- start start 2024-04-12T05:54:02.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.331+0000 7f949408c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f948c0702e0 0x7f948c115eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:02.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.331+0000 7f949408c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f948c070bf0 0x7f948c1163f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:02.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.331+0000 7f949408c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f948c116a40 con 0x7f948c070bf0 2024-04-12T05:54:02.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.331+0000 7f949408c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f948c10fe10 con 0x7f948c0702e0 2024-04-12T05:54:02.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.331+0000 7f9491627700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f948c070bf0 0x7f948c1163f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:02.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.331+0000 7f9491627700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f948c070bf0 0x7f948c1163f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45042/0 (socket says 172.21.15.32:45042) 2024-04-12T05:54:02.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.332+0000 7f9491627700 1 -- 172.21.15.32:0/1889968822 learned_addr learned my addr 172.21.15.32:0/1889968822 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:54:02.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.332+0000 7f9491e28700 1 --2- 172.21.15.32:0/1889968822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f948c0702e0 0x7f948c115eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:02.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.332+0000 7f9491627700 1 -- 172.21.15.32:0/1889968822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f948c0702e0 msgr2=0x7f948c115eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:02.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.332+0000 7f9491627700 1 --2- 172.21.15.32:0/1889968822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f948c0702e0 0x7f948c115eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:02.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.332+0000 7f9491627700 1 -- 172.21.15.32:0/1889968822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9488009940 con 0x7f948c070bf0 2024-04-12T05:54:02.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.332+0000 7f9491627700 1 --2- 172.21.15.32:0/1889968822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f948c070bf0 0x7f948c1163f0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f947c002a50 tx=0x7f947c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:54:02.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.332+0000 7f94827fc700 1 -- 172.21.15.32:0/1889968822 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f947c007cc0 con 0x7f948c070bf0 2024-04-12T05:54:02.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.332+0000 7f94827fc700 1 -- 172.21.15.32:0/1889968822 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f947c00f040 con 0x7f948c070bf0 2024-04-12T05:54:02.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.332+0000 7f949408c700 1 -- 172.21.15.32:0/1889968822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f948c1100f0 con 0x7f948c070bf0 2024-04-12T05:54:02.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.332+0000 7f94827fc700 1 -- 172.21.15.32:0/1889968822 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f947c009540 con 0x7f948c070bf0 2024-04-12T05:54:02.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.332+0000 7f949408c700 1 -- 172.21.15.32:0/1889968822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f948c110630 con 0x7f948c070bf0 2024-04-12T05:54:02.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.333+0000 7f949408c700 1 -- 172.21.15.32:0/1889968822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f948c071a20 con 0x7f948c070bf0 2024-04-12T05:54:02.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.334+0000 7f94827fc700 1 -- 172.21.15.32:0/1889968822 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f947c0096a0 con 0x7f948c070bf0 2024-04-12T05:54:02.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.334+0000 7f94827fc700 1 --2- 172.21.15.32:0/1889968822 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f947806cc30 0x7f947806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:02.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.334+0000 7f94827fc700 1 -- 172.21.15.32:0/1889968822 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f947c08bed0 con 0x7f948c070bf0 2024-04-12T05:54:02.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.334+0000 7f9491e28700 1 --2- 172.21.15.32:0/1889968822 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f947806cc30 0x7f947806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:02.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.335+0000 7f9491e28700 1 --2- 172.21.15.32:0/1889968822 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f947806cc30 0x7f947806f0e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9488005010 tx=0x7f9488004d10 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:54:02.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.339+0000 7f94827fc700 1 -- 172.21.15.32:0/1889968822 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f947c0598c0 con 0x7f948c070bf0 2024-04-12T05:54:02.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.550+0000 7f949408c700 1 -- 172.21.15.32:0/1889968822 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f948c04bed0 con 0x7f947806cc30 2024-04-12T05:54:02.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.560+0000 7f94827fc700 1 -- 172.21.15.32:0/1889968822 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f948c04bed0 con 0x7f947806cc30 2024-04-12T05:54:02.556 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:54:02.556 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (13m) 7m ago 25m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:54:02.556 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (11m) 7m ago 25m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:54:02.556 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (11m) 6m ago 24m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:54:02.556 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (13m) 7m ago 25m 87.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:54:02.556 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (7m) 7m ago 19m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T05:54:02.556 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (7m) 7m ago 19m 13.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T05:54:02.556 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (6m) 6m ago 19m 11.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 6d4aca6f74a0 2024-04-12T05:54:02.556 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (7m) 6m ago 19m 14.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T05:54:02.556 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (15m) 7m ago 26m 507M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:54:02.556 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (14m) 6m ago 24m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:54:02.556 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (12m) 7m ago 26m 53.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:54:02.556 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (11m) 6m ago 24m 43.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:54:02.556 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (14m) 7m ago 25m 16.9M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:54:02.556 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (14m) 6m ago 24m 18.4M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:54:02.557 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (10m) 7m ago 23m 91.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T05:54:02.557 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (10m) 7m ago 22m 97.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T05:54:02.557 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (9m) 7m ago 22m 81.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T05:54:02.557 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (9m) 6m ago 22m 113M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T05:54:02.557 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (8m) 6m ago 21m 90.3M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T05:54:02.557 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (8m) 6m ago 21m 84.8M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T05:54:02.557 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (14m) 7m ago 24m 64.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:54:02.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.562+0000 7f949408c700 1 -- 172.21.15.32:0/1889968822 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f947806cc30 msgr2=0x7f947806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:02.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.562+0000 7f949408c700 1 --2- 172.21.15.32:0/1889968822 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f947806cc30 0x7f947806f0e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9488005010 tx=0x7f9488004d10 comp rx=0 tx=0).stop 2024-04-12T05:54:02.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.562+0000 7f949408c700 1 -- 172.21.15.32:0/1889968822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f948c070bf0 msgr2=0x7f948c1163f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:02.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.562+0000 7f949408c700 1 --2- 172.21.15.32:0/1889968822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f948c070bf0 0x7f948c1163f0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f947c002a50 tx=0x7f947c0046b0 comp rx=0 tx=0).stop 2024-04-12T05:54:02.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.562+0000 7f949408c700 1 -- 172.21.15.32:0/1889968822 shutdown_connections 2024-04-12T05:54:02.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.562+0000 7f949408c700 1 --2- 172.21.15.32:0/1889968822 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f947806cc30 0x7f947806f0e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:02.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.562+0000 7f949408c700 1 --2- 172.21.15.32:0/1889968822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f948c0702e0 0x7f948c115eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:02.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.562+0000 7f949408c700 1 --2- 172.21.15.32:0/1889968822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f948c070bf0 0x7f948c1163f0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:02.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.562+0000 7f949408c700 1 -- 172.21.15.32:0/1889968822 >> 172.21.15.32:0/1889968822 conn(0x7f948c0f89a0 msgr2=0x7f948c105b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:54:02.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.562+0000 7f949408c700 1 -- 172.21.15.32:0/1889968822 shutdown_connections 2024-04-12T05:54:02.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.562+0000 7f949408c700 1 -- 172.21.15.32:0/1889968822 wait complete. 2024-04-12T05:54:02.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.665+0000 7f5dfaf7b700 1 -- 172.21.15.32:0/3202981178 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5df4070300 msgr2=0x7f5df40706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:02.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.665+0000 7f5dfaf7b700 1 --2- 172.21.15.32:0/3202981178 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5df4070300 0x7f5df40706d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5de8004970 tx=0x7f5de8009e30 comp rx=0 tx=0).stop 2024-04-12T05:54:02.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.665+0000 7f5dfaf7b700 1 -- 172.21.15.32:0/3202981178 shutdown_connections 2024-04-12T05:54:02.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.665+0000 7f5dfaf7b700 1 --2- 172.21.15.32:0/3202981178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5df4070c10 0x7f5df4109180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:02.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.665+0000 7f5dfaf7b700 1 --2- 172.21.15.32:0/3202981178 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5df4070300 0x7f5df40706d0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:02.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.665+0000 7f5dfaf7b700 1 -- 172.21.15.32:0/3202981178 >> 172.21.15.32:0/3202981178 conn(0x7f5df40f89a0 msgr2=0x7f5df40fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:54:02.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.665+0000 7f5dfaf7b700 1 -- 172.21.15.32:0/3202981178 shutdown_connections 2024-04-12T05:54:02.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.665+0000 7f5dfaf7b700 1 -- 172.21.15.32:0/3202981178 wait complete. 2024-04-12T05:54:02.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.665+0000 7f5dfaf7b700 1 Processor -- start 2024-04-12T05:54:02.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.665+0000 7f5dfaf7b700 1 -- start start 2024-04-12T05:54:02.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.666+0000 7f5dfaf7b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5df4070300 0x7f5df4115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:02.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.666+0000 7f5dfaf7b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5df4070c10 0x7f5df4116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:02.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.666+0000 7f5dfaf7b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df4116790 con 0x7f5df4070300 2024-04-12T05:54:02.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.666+0000 7f5dfaf7b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df410fc10 con 0x7f5df4070c10 2024-04-12T05:54:02.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.666+0000 7f5df8d17700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5df4070300 0x7f5df4115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:02.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.666+0000 7f5df8d17700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5df4070300 0x7f5df4115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45062/0 (socket says 172.21.15.32:45062) 2024-04-12T05:54:02.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.666+0000 7f5df8d17700 1 -- 172.21.15.32:0/4123282875 learned_addr learned my addr 172.21.15.32:0/4123282875 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:54:02.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.666+0000 7f5df3fff700 1 --2- 172.21.15.32:0/4123282875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5df4070c10 0x7f5df4116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:02.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.666+0000 7f5df8d17700 1 -- 172.21.15.32:0/4123282875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5df4070c10 msgr2=0x7f5df4116140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:02.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.666+0000 7f5df8d17700 1 --2- 172.21.15.32:0/4123282875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5df4070c10 0x7f5df4116140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:02.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.666+0000 7f5df8d17700 1 -- 172.21.15.32:0/4123282875 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de8005040 con 0x7f5df4070300 2024-04-12T05:54:02.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.666+0000 7f5df8d17700 1 --2- 172.21.15.32:0/4123282875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5df4070300 0x7f5df4115c00 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f5de80043d0 tx=0x7f5de8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:54:02.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.666+0000 7f5df17fa700 1 -- 172.21.15.32:0/4123282875 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5de800e040 con 0x7f5df4070300 2024-04-12T05:54:02.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.666+0000 7f5df17fa700 1 -- 172.21.15.32:0/4123282875 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5de8013420 con 0x7f5df4070300 2024-04-12T05:54:02.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.666+0000 7f5dfaf7b700 1 -- 172.21.15.32:0/4123282875 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5df410fe90 con 0x7f5df4070300 2024-04-12T05:54:02.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.666+0000 7f5df17fa700 1 -- 172.21.15.32:0/4123282875 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5de801c630 con 0x7f5df4070300 2024-04-12T05:54:02.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.666+0000 7f5dfaf7b700 1 -- 172.21.15.32:0/4123282875 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5df41103d0 con 0x7f5df4070300 2024-04-12T05:54:02.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.667+0000 7f5dfaf7b700 1 -- 172.21.15.32:0/4123282875 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5df4103b90 con 0x7f5df4070300 2024-04-12T05:54:02.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.668+0000 7f5df17fa700 1 -- 172.21.15.32:0/4123282875 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f5de8013970 con 0x7f5df4070300 2024-04-12T05:54:02.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.669+0000 7f5df17fa700 1 --2- 172.21.15.32:0/4123282875 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5de406cc20 0x7f5de406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:02.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.669+0000 7f5df17fa700 1 -- 172.21.15.32:0/4123282875 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f5de808deb0 con 0x7f5df4070300 2024-04-12T05:54:02.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.669+0000 7f5df3fff700 1 --2- 172.21.15.32:0/4123282875 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5de406cc20 0x7f5de406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:02.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.669+0000 7f5df3fff700 1 --2- 172.21.15.32:0/4123282875 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5de406cc20 0x7f5de406f0d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5de0007fd0 tx=0x7f5de0009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:54:02.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.673+0000 7f5df17fa700 1 -- 172.21.15.32:0/4123282875 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5de8057e70 con 0x7f5df4070300 2024-04-12T05:54:02.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.963+0000 7f5dfaf7b700 1 -- 172.21.15.32:0/4123282875 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5df4111300 con 0x7f5df4070300 2024-04-12T05:54:02.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.964+0000 7f5df17fa700 1 -- 172.21.15.32:0/4123282875 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f5de8004de0 con 0x7f5df4070300 2024-04-12T05:54:02.960 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:54:02.960 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:54:02.960 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:54:02.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:54:02.960 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:54:02.960 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:54:02.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:54:02.960 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:54:02.960 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:54:02.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:54:02.960 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:54:02.960 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:54:02.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:54:02.961 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:54:02.961 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:54:02.961 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:54:02.961 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:54:02.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.966+0000 7f5dfaf7b700 1 -- 172.21.15.32:0/4123282875 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5de406cc20 msgr2=0x7f5de406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:02.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.966+0000 7f5dfaf7b700 1 --2- 172.21.15.32:0/4123282875 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5de406cc20 0x7f5de406f0d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5de0007fd0 tx=0x7f5de0009040 comp rx=0 tx=0).stop 2024-04-12T05:54:02.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.966+0000 7f5dfaf7b700 1 -- 172.21.15.32:0/4123282875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5df4070300 msgr2=0x7f5df4115c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:02.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.966+0000 7f5dfaf7b700 1 --2- 172.21.15.32:0/4123282875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5df4070300 0x7f5df4115c00 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f5de80043d0 tx=0x7f5de8009e30 comp rx=0 tx=0).stop 2024-04-12T05:54:02.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.966+0000 7f5dfaf7b700 1 -- 172.21.15.32:0/4123282875 shutdown_connections 2024-04-12T05:54:02.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.966+0000 7f5dfaf7b700 1 --2- 172.21.15.32:0/4123282875 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5de406cc20 0x7f5de406f0d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:02.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.966+0000 7f5dfaf7b700 1 --2- 172.21.15.32:0/4123282875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5df4070300 0x7f5df4115c00 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:02.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.966+0000 7f5dfaf7b700 1 --2- 172.21.15.32:0/4123282875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5df4070c10 0x7f5df4116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:02.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.966+0000 7f5dfaf7b700 1 -- 172.21.15.32:0/4123282875 >> 172.21.15.32:0/4123282875 conn(0x7f5df40f89a0 msgr2=0x7f5df4101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:54:02.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.966+0000 7f5dfaf7b700 1 -- 172.21.15.32:0/4123282875 shutdown_connections 2024-04-12T05:54:02.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:02.966+0000 7f5dfaf7b700 1 -- 172.21.15.32:0/4123282875 wait complete. 2024-04-12T05:54:03.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.069+0000 7f1c4c6ca700 1 -- 172.21.15.32:0/2314463249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c44102e10 msgr2=0x7f1c441031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:03.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.069+0000 7f1c4c6ca700 1 --2- 172.21.15.32:0/2314463249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c44102e10 0x7f1c441031e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1c40004940 tx=0x7f1c40009e30 comp rx=0 tx=0).stop 2024-04-12T05:54:03.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.069+0000 7f1c4c6ca700 1 -- 172.21.15.32:0/2314463249 shutdown_connections 2024-04-12T05:54:03.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.069+0000 7f1c4c6ca700 1 --2- 172.21.15.32:0/2314463249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c441037b0 0x7f1c440fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:03.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.069+0000 7f1c4c6ca700 1 --2- 172.21.15.32:0/2314463249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c44102e10 0x7f1c441031e0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:03.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.069+0000 7f1c4c6ca700 1 -- 172.21.15.32:0/2314463249 >> 172.21.15.32:0/2314463249 conn(0x7f1c440f89a0 msgr2=0x7f1c440fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:54:03.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.069+0000 7f1c4c6ca700 1 -- 172.21.15.32:0/2314463249 shutdown_connections 2024-04-12T05:54:03.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.069+0000 7f1c4c6ca700 1 -- 172.21.15.32:0/2314463249 wait complete. 2024-04-12T05:54:03.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.070+0000 7f1c4c6ca700 1 Processor -- start 2024-04-12T05:54:03.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.070+0000 7f1c4c6ca700 1 -- start start 2024-04-12T05:54:03.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.070+0000 7f1c4c6ca700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c44102e10 0x7f1c440702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:03.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.070+0000 7f1c4c6ca700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c441037b0 0x7f1c44070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:03.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.070+0000 7f1c4c6ca700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c44070d90 con 0x7f1c441037b0 2024-04-12T05:54:03.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.070+0000 7f1c4c6ca700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c44070ed0 con 0x7f1c44102e10 2024-04-12T05:54:03.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.070+0000 7f1c49c65700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c441037b0 0x7f1c44070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:03.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.070+0000 7f1c49c65700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c441037b0 0x7f1c44070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45092/0 (socket says 172.21.15.32:45092) 2024-04-12T05:54:03.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.070+0000 7f1c49c65700 1 -- 172.21.15.32:0/983542732 learned_addr learned my addr 172.21.15.32:0/983542732 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:54:03.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.070+0000 7f1c4a466700 1 --2- 172.21.15.32:0/983542732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c44102e10 0x7f1c440702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:03.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.070+0000 7f1c49c65700 1 -- 172.21.15.32:0/983542732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c44102e10 msgr2=0x7f1c440702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:03.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.070+0000 7f1c49c65700 1 --2- 172.21.15.32:0/983542732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c44102e10 0x7f1c440702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:03.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.070+0000 7f1c49c65700 1 -- 172.21.15.32:0/983542732 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c40005040 con 0x7f1c441037b0 2024-04-12T05:54:03.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.071+0000 7f1c49c65700 1 --2- 172.21.15.32:0/983542732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c441037b0 0x7f1c44070800 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1c34002a50 tx=0x7f1c340046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:54:03.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.071+0000 7f1c3affd700 1 -- 172.21.15.32:0/983542732 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c34007cc0 con 0x7f1c441037b0 2024-04-12T05:54:03.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.071+0000 7f1c3affd700 1 -- 172.21.15.32:0/983542732 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1c3400f040 con 0x7f1c441037b0 2024-04-12T05:54:03.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.071+0000 7f1c4c6ca700 1 -- 172.21.15.32:0/983542732 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c44119d50 con 0x7f1c441037b0 2024-04-12T05:54:03.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.071+0000 7f1c3affd700 1 -- 172.21.15.32:0/983542732 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c3400da30 con 0x7f1c441037b0 2024-04-12T05:54:03.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.071+0000 7f1c4c6ca700 1 -- 172.21.15.32:0/983542732 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c4411a1d0 con 0x7f1c441037b0 2024-04-12T05:54:03.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.072+0000 7f1c4c6ca700 1 -- 172.21.15.32:0/983542732 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c440fe560 con 0x7f1c441037b0 2024-04-12T05:54:03.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.072+0000 7f1c3affd700 1 -- 172.21.15.32:0/983542732 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1c3400db90 con 0x7f1c441037b0 2024-04-12T05:54:03.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.073+0000 7f1c3affd700 1 --2- 172.21.15.32:0/983542732 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1c3006cc30 0x7f1c3006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:03.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.073+0000 7f1c4a466700 1 --2- 172.21.15.32:0/983542732 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1c3006cc30 0x7f1c3006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:03.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.073+0000 7f1c3affd700 1 -- 172.21.15.32:0/983542732 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f1c3408bce0 con 0x7f1c441037b0 2024-04-12T05:54:03.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.073+0000 7f1c4a466700 1 --2- 172.21.15.32:0/983542732 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1c3006cc30 0x7f1c3006f0e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1c40009fd0 tx=0x7f1c40009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:54:03.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.078+0000 7f1c3affd700 1 -- 172.21.15.32:0/983542732 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1c34055c40 con 0x7f1c441037b0 2024-04-12T05:54:03.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.322+0000 7f1c4c6ca700 1 -- 172.21.15.32:0/983542732 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1c4404bed0 con 0x7f1c441037b0 2024-04-12T05:54:03.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.322+0000 7f1c3affd700 1 -- 172.21.15.32:0/983542732 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1835 (secure 0 0 0) 0x7f1c34077e60 con 0x7f1c441037b0 2024-04-12T05:54:03.318 INFO:teuthology.orchestra.run.smithi032.stdout:e33 2024-04-12T05:54:03.318 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:54:03.318 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:54:03.318 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:54:03.318 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:54:03.318 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:54:03.318 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:54:03.318 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 24 2024-04-12T05:54:03.318 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:54:03.319 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:54:03.319 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:46:10.668569+0000 2024-04-12T05:54:03.319 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:54:03.319 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:54:03.319 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:54:03.319 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:54:03.319 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:54:03.319 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:54:03.319 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:54:03.319 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:54:03.319 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T05:54:03.319 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:54:03.319 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:54:03.319 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:54:03.319 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T05:54:03.320 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-12T05:54:03.320 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:54:03.320 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:54:03.320 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:54:03.320 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:54:03.320 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:54:03.320 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:54:03.320 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:54:03.320 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:54:03.320 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:54:03.320 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:54:03.320 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-12T05:54:03.320 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:54:03.320 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:54:03.320 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:54:03.320 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{-1:44317} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:54:03.321 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:54:03.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.324+0000 7f1c4c6ca700 1 -- 172.21.15.32:0/983542732 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1c3006cc30 msgr2=0x7f1c3006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:03.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.324+0000 7f1c4c6ca700 1 --2- 172.21.15.32:0/983542732 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1c3006cc30 0x7f1c3006f0e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1c40009fd0 tx=0x7f1c40009e30 comp rx=0 tx=0).stop 2024-04-12T05:54:03.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.324+0000 7f1c4c6ca700 1 -- 172.21.15.32:0/983542732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c441037b0 msgr2=0x7f1c44070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:03.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.324+0000 7f1c4c6ca700 1 --2- 172.21.15.32:0/983542732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c441037b0 0x7f1c44070800 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1c34002a50 tx=0x7f1c340046b0 comp rx=0 tx=0).stop 2024-04-12T05:54:03.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.324+0000 7f1c4c6ca700 1 -- 172.21.15.32:0/983542732 shutdown_connections 2024-04-12T05:54:03.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.324+0000 7f1c4c6ca700 1 --2- 172.21.15.32:0/983542732 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1c3006cc30 0x7f1c3006f0e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:03.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.324+0000 7f1c4c6ca700 1 --2- 172.21.15.32:0/983542732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c44102e10 0x7f1c440702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:03.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.324+0000 7f1c4c6ca700 1 --2- 172.21.15.32:0/983542732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c441037b0 0x7f1c44070800 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:03.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.324+0000 7f1c4c6ca700 1 -- 172.21.15.32:0/983542732 >> 172.21.15.32:0/983542732 conn(0x7f1c440f89a0 msgr2=0x7f1c440f9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:54:03.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.324+0000 7f1c4c6ca700 1 -- 172.21.15.32:0/983542732 shutdown_connections 2024-04-12T05:54:03.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.324+0000 7f1c4c6ca700 1 -- 172.21.15.32:0/983542732 wait complete. 2024-04-12T05:54:03.322 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-12T05:54:03.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:03 smithi032 ceph-mon[146817]: from='client.34788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:54:03.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:03 smithi032 ceph-mon[146817]: pgmap v406: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:03.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:03 smithi032 ceph-mon[146817]: from='client.34792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:54:03.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:03 smithi032 ceph-mon[146817]: from='client.34796 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:54:03.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:54:03.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:03 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/4123282875' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:54:03.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.428+0000 7f863ad5e700 1 -- 172.21.15.32:0/3934052250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8634102710 msgr2=0x7f86340fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:03.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.428+0000 7f863ad5e700 1 --2- 172.21.15.32:0/3934052250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8634102710 0x7f86340fbd80 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f8620004940 tx=0x7f8620009e30 comp rx=0 tx=0).stop 2024-04-12T05:54:03.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.429+0000 7f863ad5e700 1 -- 172.21.15.32:0/3934052250 shutdown_connections 2024-04-12T05:54:03.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.429+0000 7f863ad5e700 1 --2- 172.21.15.32:0/3934052250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8634102710 0x7f86340fbd80 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:03.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.429+0000 7f863ad5e700 1 --2- 172.21.15.32:0/3934052250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8634101d70 0x7f8634102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:03.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.429+0000 7f863ad5e700 1 -- 172.21.15.32:0/3934052250 >> 172.21.15.32:0/3934052250 conn(0x7f8634072bc0 msgr2=0x7f8634072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:54:03.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.429+0000 7f863ad5e700 1 -- 172.21.15.32:0/3934052250 shutdown_connections 2024-04-12T05:54:03.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.429+0000 7f863ad5e700 1 -- 172.21.15.32:0/3934052250 wait complete. 2024-04-12T05:54:03.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.429+0000 7f863ad5e700 1 Processor -- start 2024-04-12T05:54:03.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.429+0000 7f863ad5e700 1 -- start start 2024-04-12T05:54:03.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.430+0000 7f863ad5e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8634101d70 0x7f86341129c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:03.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.430+0000 7f863ad5e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8634102710 0x7f8634112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:03.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.430+0000 7f863ad5e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8634113550 con 0x7f8634101d70 2024-04-12T05:54:03.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.430+0000 7f863ad5e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f863410c940 con 0x7f8634102710 2024-04-12T05:54:03.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.430+0000 7f8638afa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8634101d70 0x7f86341129c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:03.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.430+0000 7f8638afa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8634101d70 0x7f86341129c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45104/0 (socket says 172.21.15.32:45104) 2024-04-12T05:54:03.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.430+0000 7f8638afa700 1 -- 172.21.15.32:0/1201013675 learned_addr learned my addr 172.21.15.32:0/1201013675 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:54:03.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.430+0000 7f8633fff700 1 --2- 172.21.15.32:0/1201013675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8634102710 0x7f8634112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:03.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.430+0000 7f8638afa700 1 -- 172.21.15.32:0/1201013675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8634102710 msgr2=0x7f8634112f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:03.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.430+0000 7f8638afa700 1 --2- 172.21.15.32:0/1201013675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8634102710 0x7f8634112f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:03.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.430+0000 7f8638afa700 1 -- 172.21.15.32:0/1201013675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8620005040 con 0x7f8634101d70 2024-04-12T05:54:03.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.430+0000 7f8633fff700 1 --2- 172.21.15.32:0/1201013675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8634102710 0x7f8634112f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:54:03.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.430+0000 7f8638afa700 1 --2- 172.21.15.32:0/1201013675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8634101d70 0x7f86341129c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f862800c8c0 tx=0x7f86280044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:54:03.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.431+0000 7f86317fa700 1 -- 172.21.15.32:0/1201013675 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f862800a980 con 0x7f8634101d70 2024-04-12T05:54:03.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.431+0000 7f86317fa700 1 -- 172.21.15.32:0/1201013675 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f862800ce60 con 0x7f8634101d70 2024-04-12T05:54:03.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.431+0000 7f863ad5e700 1 -- 172.21.15.32:0/1201013675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f863410cc20 con 0x7f8634101d70 2024-04-12T05:54:03.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.431+0000 7f86317fa700 1 -- 172.21.15.32:0/1201013675 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86280113a0 con 0x7f8634101d70 2024-04-12T05:54:03.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.431+0000 7f863ad5e700 1 -- 172.21.15.32:0/1201013675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f863410d1e0 con 0x7f8634101d70 2024-04-12T05:54:03.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.432+0000 7f863ad5e700 1 -- 172.21.15.32:0/1201013675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86340ff080 con 0x7f8634101d70 2024-04-12T05:54:03.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.432+0000 7f86317fa700 1 -- 172.21.15.32:0/1201013675 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8628012400 con 0x7f8634101d70 2024-04-12T05:54:03.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.433+0000 7f86317fa700 1 --2- 172.21.15.32:0/1201013675 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f862406cc30 0x7f862406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:03.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.433+0000 7f86317fa700 1 -- 172.21.15.32:0/1201013675 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f862808c1c0 con 0x7f8634101d70 2024-04-12T05:54:03.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.433+0000 7f8633fff700 1 --2- 172.21.15.32:0/1201013675 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f862406cc30 0x7f862406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:03.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.433+0000 7f8633fff700 1 --2- 172.21.15.32:0/1201013675 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f862406cc30 0x7f862406f0e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f863410db00 tx=0x7f8620009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:54:03.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.437+0000 7f86317fa700 1 -- 172.21.15.32:0/1201013675 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8628059c30 con 0x7f8634101d70 2024-04-12T05:54:03.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:03 smithi067 ceph-mon[112979]: from='client.34788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:54:03.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:03 smithi067 ceph-mon[112979]: pgmap v406: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:03.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:03 smithi067 ceph-mon[112979]: from='client.34792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:54:03.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:03 smithi067 ceph-mon[112979]: from='client.34796 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:54:03.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:54:03.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:03 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/4123282875' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:54:03.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.656+0000 7f863ad5e700 1 -- 172.21.15.32:0/1201013675 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f863410e010 con 0x7f862406cc30 2024-04-12T05:54:03.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.657+0000 7f86317fa700 1 -- 172.21.15.32:0/1201013675 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f863410e010 con 0x7f862406cc30 2024-04-12T05:54:03.653 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:54:03.653 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:54:03.653 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:54:03.653 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:54:03.653 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:54:03.653 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-12T05:54:03.653 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:54:03.653 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-12T05:54:03.653 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:54:03.653 INFO:teuthology.orchestra.run.smithi032.stdout: "mds" 2024-04-12T05:54:03.653 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:54:03.654 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:54:03.654 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:54:03.654 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:54:03.654 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:54:03.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.659+0000 7f863ad5e700 1 -- 172.21.15.32:0/1201013675 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f862406cc30 msgr2=0x7f862406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:03.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.659+0000 7f863ad5e700 1 --2- 172.21.15.32:0/1201013675 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f862406cc30 0x7f862406f0e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f863410db00 tx=0x7f8620009e30 comp rx=0 tx=0).stop 2024-04-12T05:54:03.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.659+0000 7f863ad5e700 1 -- 172.21.15.32:0/1201013675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8634101d70 msgr2=0x7f86341129c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:03.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.659+0000 7f863ad5e700 1 --2- 172.21.15.32:0/1201013675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8634101d70 0x7f86341129c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f862800c8c0 tx=0x7f86280044c0 comp rx=0 tx=0).stop 2024-04-12T05:54:03.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.659+0000 7f863ad5e700 1 -- 172.21.15.32:0/1201013675 shutdown_connections 2024-04-12T05:54:03.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.659+0000 7f863ad5e700 1 --2- 172.21.15.32:0/1201013675 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f862406cc30 0x7f862406f0e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:03.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.659+0000 7f863ad5e700 1 --2- 172.21.15.32:0/1201013675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8634101d70 0x7f86341129c0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:03.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.659+0000 7f863ad5e700 1 --2- 172.21.15.32:0/1201013675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8634102710 0x7f8634112f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:03.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.659+0000 7f863ad5e700 1 -- 172.21.15.32:0/1201013675 >> 172.21.15.32:0/1201013675 conn(0x7f8634072bc0 msgr2=0x7f86340fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:54:03.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.659+0000 7f863ad5e700 1 -- 172.21.15.32:0/1201013675 shutdown_connections 2024-04-12T05:54:03.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.659+0000 7f863ad5e700 1 -- 172.21.15.32:0/1201013675 wait complete. 2024-04-12T05:54:03.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.762+0000 7f7ec82ca700 1 -- 172.21.15.32:0/2975676359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ec00fe7a0 msgr2=0x7f7ec0106de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:03.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.762+0000 7f7ec82ca700 1 --2- 172.21.15.32:0/2975676359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ec00fe7a0 0x7f7ec0106de0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7eb0004940 tx=0x7f7eb0009e30 comp rx=0 tx=0).stop 2024-04-12T05:54:03.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.763+0000 7f7ec82ca700 1 -- 172.21.15.32:0/2975676359 shutdown_connections 2024-04-12T05:54:03.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.763+0000 7f7ec82ca700 1 --2- 172.21.15.32:0/2975676359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ec00fe7a0 0x7f7ec0106de0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:03.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.763+0000 7f7ec82ca700 1 --2- 172.21.15.32:0/2975676359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ec00fde00 0x7f7ec00fe1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:03.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.763+0000 7f7ec82ca700 1 -- 172.21.15.32:0/2975676359 >> 172.21.15.32:0/2975676359 conn(0x7f7ec00f78d0 msgr2=0x7f7ec00f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:54:03.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.763+0000 7f7ec82ca700 1 -- 172.21.15.32:0/2975676359 shutdown_connections 2024-04-12T05:54:03.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.763+0000 7f7ec82ca700 1 -- 172.21.15.32:0/2975676359 wait complete. 2024-04-12T05:54:03.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.763+0000 7f7ec82ca700 1 Processor -- start 2024-04-12T05:54:03.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.763+0000 7f7ec82ca700 1 -- start start 2024-04-12T05:54:03.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.763+0000 7f7ec82ca700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ec00fde00 0x7f7ec00fbc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:03.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.763+0000 7f7ec82ca700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ec00fe7a0 0x7f7ec00fc170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:03.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.763+0000 7f7ec82ca700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec00fc700 con 0x7f7ec00fe7a0 2024-04-12T05:54:03.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.763+0000 7f7ec82ca700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec00fc840 con 0x7f7ec00fde00 2024-04-12T05:54:03.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.764+0000 7f7ec5865700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ec00fe7a0 0x7f7ec00fc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:03.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.764+0000 7f7ec5865700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ec00fe7a0 0x7f7ec00fc170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45118/0 (socket says 172.21.15.32:45118) 2024-04-12T05:54:03.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.764+0000 7f7ec5865700 1 -- 172.21.15.32:0/1047460124 learned_addr learned my addr 172.21.15.32:0/1047460124 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:54:03.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.764+0000 7f7ec6066700 1 --2- 172.21.15.32:0/1047460124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ec00fde00 0x7f7ec00fbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:03.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.764+0000 7f7ec5865700 1 -- 172.21.15.32:0/1047460124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ec00fde00 msgr2=0x7f7ec00fbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:03.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.764+0000 7f7ec5865700 1 --2- 172.21.15.32:0/1047460124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ec00fde00 0x7f7ec00fbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:03.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.764+0000 7f7ec5865700 1 -- 172.21.15.32:0/1047460124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb0005040 con 0x7f7ec00fe7a0 2024-04-12T05:54:03.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.764+0000 7f7ec5865700 1 --2- 172.21.15.32:0/1047460124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ec00fe7a0 0x7f7ec00fc170 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7eb0009380 tx=0x7f7eb0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:54:03.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.764+0000 7f7eb6ffd700 1 -- 172.21.15.32:0/1047460124 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7eb000e040 con 0x7f7ec00fe7a0 2024-04-12T05:54:03.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.764+0000 7f7eb6ffd700 1 -- 172.21.15.32:0/1047460124 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7eb0004290 con 0x7f7ec00fe7a0 2024-04-12T05:54:03.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.764+0000 7f7ec82ca700 1 -- 172.21.15.32:0/1047460124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec00fca90 con 0x7f7ec00fe7a0 2024-04-12T05:54:03.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.764+0000 7f7eb6ffd700 1 -- 172.21.15.32:0/1047460124 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7eb0013e90 con 0x7f7ec00fe7a0 2024-04-12T05:54:03.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.764+0000 7f7ec82ca700 1 -- 172.21.15.32:0/1047460124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec011a100 con 0x7f7ec00fe7a0 2024-04-12T05:54:03.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.765+0000 7f7ec82ca700 1 -- 172.21.15.32:0/1047460124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ec0066770 con 0x7f7ec00fe7a0 2024-04-12T05:54:03.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.766+0000 7f7eb6ffd700 1 -- 172.21.15.32:0/1047460124 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f7eb00136b0 con 0x7f7ec00fe7a0 2024-04-12T05:54:03.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.766+0000 7f7eb6ffd700 1 --2- 172.21.15.32:0/1047460124 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7eac06cc20 0x7f7eac06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:03.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.766+0000 7f7eb6ffd700 1 -- 172.21.15.32:0/1047460124 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f7eb008d3a0 con 0x7f7ec00fe7a0 2024-04-12T05:54:03.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.766+0000 7f7ec6066700 1 --2- 172.21.15.32:0/1047460124 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7eac06cc20 0x7f7eac06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:03.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.767+0000 7f7ec6066700 1 --2- 172.21.15.32:0/1047460124 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7eac06cc20 0x7f7eac06f0d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc004f40 tx=0x7f7ebc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:54:03.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:03.771+0000 7f7eb6ffd700 1 -- 172.21.15.32:0/1047460124 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7eb0057380 con 0x7f7ec00fe7a0 2024-04-12T05:54:04.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:04.068+0000 7f7ec82ca700 1 -- 172.21.15.32:0/1047460124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7ec0109d80 con 0x7f7ec00fe7a0 2024-04-12T05:54:04.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:04.068+0000 7f7eb6ffd700 1 -- 172.21.15.32:0/1047460124 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f7eb005a9a0 con 0x7f7ec00fe7a0 2024-04-12T05:54:04.064 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:54:04.064 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:54:04.064 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-12T05:54:04.064 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:54:04.064 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:54:04.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:04.070+0000 7f7ec82ca700 1 -- 172.21.15.32:0/1047460124 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7eac06cc20 msgr2=0x7f7eac06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:04.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:04.070+0000 7f7ec82ca700 1 --2- 172.21.15.32:0/1047460124 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7eac06cc20 0x7f7eac06f0d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc004f40 tx=0x7f7ebc00b040 comp rx=0 tx=0).stop 2024-04-12T05:54:04.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:04.070+0000 7f7ec82ca700 1 -- 172.21.15.32:0/1047460124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ec00fe7a0 msgr2=0x7f7ec00fc170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:04.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:04.070+0000 7f7ec82ca700 1 --2- 172.21.15.32:0/1047460124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ec00fe7a0 0x7f7ec00fc170 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7eb0009380 tx=0x7f7eb0009e30 comp rx=0 tx=0).stop 2024-04-12T05:54:04.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:04.070+0000 7f7ec82ca700 1 -- 172.21.15.32:0/1047460124 shutdown_connections 2024-04-12T05:54:04.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:04.070+0000 7f7ec82ca700 1 --2- 172.21.15.32:0/1047460124 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7eac06cc20 0x7f7eac06f0d0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:04.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:04.070+0000 7f7ec82ca700 1 --2- 172.21.15.32:0/1047460124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ec00fde00 0x7f7ec00fbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:04.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:04.070+0000 7f7ec82ca700 1 --2- 172.21.15.32:0/1047460124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ec00fe7a0 0x7f7ec00fc170 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:04.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:04.070+0000 7f7ec82ca700 1 -- 172.21.15.32:0/1047460124 >> 172.21.15.32:0/1047460124 conn(0x7f7ec00f78d0 msgr2=0x7f7ec00f8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:54:04.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:04.070+0000 7f7ec82ca700 1 -- 172.21.15.32:0/1047460124 shutdown_connections 2024-04-12T05:54:04.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:04.070+0000 7f7ec82ca700 1 -- 172.21.15.32:0/1047460124 wait complete. 2024-04-12T05:54:04.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:04 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/983542732' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:54:04.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:04 smithi032 ceph-mon[146817]: from='client.34808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:54:04.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:04 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/983542732' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:54:04.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:04 smithi067 ceph-mon[112979]: from='client.34808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:54:05.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:05 smithi032 ceph-mon[146817]: pgmap v407: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:05.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:05 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1047460124' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:54:05.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:05 smithi067 ceph-mon[112979]: pgmap v407: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:05.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:05 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1047460124' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:54:07.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:07 smithi032 ceph-mon[146817]: pgmap v408: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:07.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:07 smithi067 ceph-mon[112979]: pgmap v408: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:09.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:09 smithi032 ceph-mon[146817]: pgmap v409: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:09.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:09 smithi067 ceph-mon[112979]: pgmap v409: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:11.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:11 smithi032 ceph-mon[146817]: pgmap v410: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:11.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:11 smithi067 ceph-mon[112979]: pgmap v410: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:13.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:13 smithi032 ceph-mon[146817]: pgmap v411: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:13.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:13 smithi067 ceph-mon[112979]: pgmap v411: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:15.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:15 smithi032 ceph-mon[146817]: pgmap v412: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:15.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:15 smithi067 ceph-mon[112979]: pgmap v412: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:17.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:17 smithi032 ceph-mon[146817]: pgmap v413: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:17.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:17 smithi067 ceph-mon[112979]: pgmap v413: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:19.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:54:19.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:54:20.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:19 smithi067 ceph-mon[112979]: pgmap v414: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:20.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:19 smithi032 ceph-mon[146817]: pgmap v414: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:22.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:21 smithi067 ceph-mon[112979]: pgmap v415: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:22.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:21 smithi032 ceph-mon[146817]: pgmap v415: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:24.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:23 smithi067 ceph-mon[112979]: pgmap v416: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:24.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:23 smithi032 ceph-mon[146817]: pgmap v416: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:26.017 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:25 smithi067 ceph-mon[112979]: pgmap v417: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:26.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:25 smithi032 ceph-mon[146817]: pgmap v417: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:28.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:27 smithi067 ceph-mon[112979]: pgmap v418: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:28.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:27 smithi032 ceph-mon[146817]: pgmap v418: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:30.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:29 smithi067 ceph-mon[112979]: pgmap v419: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:30.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:29 smithi032 ceph-mon[146817]: pgmap v419: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:32.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:31 smithi067 ceph-mon[112979]: pgmap v420: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:32.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:31 smithi032 ceph-mon[146817]: pgmap v420: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:34.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:33 smithi067 ceph-mon[112979]: pgmap v421: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:34.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:33 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:54:34.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:33 smithi032 ceph-mon[146817]: pgmap v421: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:34.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:33 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:54:34.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.189+0000 7f85e5d81700 1 -- 172.21.15.32:0/448934948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85e00fde90 msgr2=0x7f85e00fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:34.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.189+0000 7f85e5d81700 1 --2- 172.21.15.32:0/448934948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85e00fde90 0x7f85e00fe260 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f85d0004940 tx=0x7f85d0009e30 comp rx=0 tx=0).stop 2024-04-12T05:54:34.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.189+0000 7f85e5d81700 1 -- 172.21.15.32:0/448934948 shutdown_connections 2024-04-12T05:54:34.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.189+0000 7f85e5d81700 1 --2- 172.21.15.32:0/448934948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e00fe830 0x7f85e0106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:34.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.189+0000 7f85e5d81700 1 --2- 172.21.15.32:0/448934948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85e00fde90 0x7f85e00fe260 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:34.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.189+0000 7f85e5d81700 1 -- 172.21.15.32:0/448934948 >> 172.21.15.32:0/448934948 conn(0x7f85e0072bc0 msgr2=0x7f85e0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:54:34.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.189+0000 7f85e5d81700 1 -- 172.21.15.32:0/448934948 shutdown_connections 2024-04-12T05:54:34.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.189+0000 7f85e5d81700 1 -- 172.21.15.32:0/448934948 wait complete. 2024-04-12T05:54:34.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.189+0000 7f85e5d81700 1 Processor -- start 2024-04-12T05:54:34.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.190+0000 7f85e5d81700 1 -- start start 2024-04-12T05:54:34.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.190+0000 7f85e5d81700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e00fde90 0x7f85e0113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:34.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.190+0000 7f85e5d81700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85e00fe830 0x7f85e0114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:34.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.190+0000 7f85e5d81700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e0114680 con 0x7f85e00fe830 2024-04-12T05:54:34.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.190+0000 7f85e5d81700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e010da70 con 0x7f85e00fde90 2024-04-12T05:54:34.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.190+0000 7f85deffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85e00fe830 0x7f85e0114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:34.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.190+0000 7f85deffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85e00fe830 0x7f85e0114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60514/0 (socket says 172.21.15.32:60514) 2024-04-12T05:54:34.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.190+0000 7f85deffd700 1 -- 172.21.15.32:0/1110119863 learned_addr learned my addr 172.21.15.32:0/1110119863 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:54:34.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.191+0000 7f85df7fe700 1 --2- 172.21.15.32:0/1110119863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e00fde90 0x7f85e0113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:34.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.191+0000 7f85deffd700 1 -- 172.21.15.32:0/1110119863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e00fde90 msgr2=0x7f85e0113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:34.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.191+0000 7f85deffd700 1 --2- 172.21.15.32:0/1110119863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e00fde90 0x7f85e0113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:34.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.191+0000 7f85deffd700 1 -- 172.21.15.32:0/1110119863 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d0005040 con 0x7f85e00fe830 2024-04-12T05:54:34.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.191+0000 7f85deffd700 1 --2- 172.21.15.32:0/1110119863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85e00fe830 0x7f85e0114030 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f85d4002a50 tx=0x7f85d40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:54:34.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.191+0000 7f85c7fff700 1 -- 172.21.15.32:0/1110119863 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85d4007cc0 con 0x7f85e00fe830 2024-04-12T05:54:34.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.191+0000 7f85c7fff700 1 -- 172.21.15.32:0/1110119863 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f85d400f040 con 0x7f85e00fe830 2024-04-12T05:54:34.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.191+0000 7f85e5d81700 1 -- 172.21.15.32:0/1110119863 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85e010dd50 con 0x7f85e00fe830 2024-04-12T05:54:34.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.191+0000 7f85c7fff700 1 -- 172.21.15.32:0/1110119863 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85d4009540 con 0x7f85e00fe830 2024-04-12T05:54:34.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.191+0000 7f85e5d81700 1 -- 172.21.15.32:0/1110119863 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85e010e290 con 0x7f85e00fe830 2024-04-12T05:54:34.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.192+0000 7f85e5d81700 1 -- 172.21.15.32:0/1110119863 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85e01019b0 con 0x7f85e00fe830 2024-04-12T05:54:34.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.193+0000 7f85c7fff700 1 -- 172.21.15.32:0/1110119863 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f85d40096c0 con 0x7f85e00fe830 2024-04-12T05:54:34.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.193+0000 7f85c7fff700 1 --2- 172.21.15.32:0/1110119863 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f85c806cc20 0x7f85c806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:34.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.193+0000 7f85c7fff700 1 -- 172.21.15.32:0/1110119863 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f85d408bed0 con 0x7f85e00fe830 2024-04-12T05:54:34.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.193+0000 7f85df7fe700 1 --2- 172.21.15.32:0/1110119863 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f85c806cc20 0x7f85c806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:34.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.194+0000 7f85df7fe700 1 --2- 172.21.15.32:0/1110119863 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f85c806cc20 0x7f85c806f0d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f85d000b3b0 tx=0x7f85d0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:54:34.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.198+0000 7f85c7fff700 1 -- 172.21.15.32:0/1110119863 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f85d40598c0 con 0x7f85e00fe830 2024-04-12T05:54:34.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.422+0000 7f85e5d81700 1 -- 172.21.15.32:0/1110119863 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f85e010f140 con 0x7f85c806cc20 2024-04-12T05:54:34.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.423+0000 7f85c7fff700 1 -- 172.21.15.32:0/1110119863 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f85e010f140 con 0x7f85c806cc20 2024-04-12T05:54:34.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.425+0000 7f85e5d81700 1 -- 172.21.15.32:0/1110119863 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f85c806cc20 msgr2=0x7f85c806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:34.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.425+0000 7f85e5d81700 1 --2- 172.21.15.32:0/1110119863 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f85c806cc20 0x7f85c806f0d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f85d000b3b0 tx=0x7f85d0009e30 comp rx=0 tx=0).stop 2024-04-12T05:54:34.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.425+0000 7f85e5d81700 1 -- 172.21.15.32:0/1110119863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85e00fe830 msgr2=0x7f85e0114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:34.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.425+0000 7f85e5d81700 1 --2- 172.21.15.32:0/1110119863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85e00fe830 0x7f85e0114030 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f85d4002a50 tx=0x7f85d40046b0 comp rx=0 tx=0).stop 2024-04-12T05:54:34.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.425+0000 7f85e5d81700 1 -- 172.21.15.32:0/1110119863 shutdown_connections 2024-04-12T05:54:34.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.425+0000 7f85e5d81700 1 --2- 172.21.15.32:0/1110119863 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f85c806cc20 0x7f85c806f0d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:34.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.425+0000 7f85e5d81700 1 --2- 172.21.15.32:0/1110119863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e00fde90 0x7f85e0113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:34.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.425+0000 7f85e5d81700 1 --2- 172.21.15.32:0/1110119863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85e00fe830 0x7f85e0114030 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:34.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.425+0000 7f85e5d81700 1 -- 172.21.15.32:0/1110119863 >> 172.21.15.32:0/1110119863 conn(0x7f85e0072bc0 msgr2=0x7f85e00ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:54:34.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.425+0000 7f85e5d81700 1 -- 172.21.15.32:0/1110119863 shutdown_connections 2024-04-12T05:54:34.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.425+0000 7f85e5d81700 1 -- 172.21.15.32:0/1110119863 wait complete. 2024-04-12T05:54:34.430 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:54:34.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.529+0000 7f689f27c700 1 -- 172.21.15.32:0/4062768274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6898102e10 msgr2=0x7f68981031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:34.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.529+0000 7f689f27c700 1 --2- 172.21.15.32:0/4062768274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6898102e10 0x7f68981031e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f6894004940 tx=0x7f6894009e30 comp rx=0 tx=0).stop 2024-04-12T05:54:34.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.529+0000 7f689f27c700 1 -- 172.21.15.32:0/4062768274 shutdown_connections 2024-04-12T05:54:34.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.529+0000 7f689f27c700 1 --2- 172.21.15.32:0/4062768274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68981037b0 0x7f68980fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:34.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.529+0000 7f689f27c700 1 --2- 172.21.15.32:0/4062768274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6898102e10 0x7f68981031e0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:34.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.529+0000 7f689f27c700 1 -- 172.21.15.32:0/4062768274 >> 172.21.15.32:0/4062768274 conn(0x7f68980f89a0 msgr2=0x7f68980fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:54:34.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.530+0000 7f689f27c700 1 -- 172.21.15.32:0/4062768274 shutdown_connections 2024-04-12T05:54:34.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.530+0000 7f689f27c700 1 -- 172.21.15.32:0/4062768274 wait complete. 2024-04-12T05:54:34.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.530+0000 7f689f27c700 1 Processor -- start 2024-04-12T05:54:34.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.530+0000 7f689f27c700 1 -- start start 2024-04-12T05:54:34.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.530+0000 7f689f27c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6898102e10 0x7f6898113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:34.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.530+0000 7f689f27c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68981037b0 0x7f6898114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:34.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.530+0000 7f689f27c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6898114680 con 0x7f6898102e10 2024-04-12T05:54:34.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.530+0000 7f689f27c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f689810da70 con 0x7f68981037b0 2024-04-12T05:54:34.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.530+0000 7f689d018700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6898102e10 0x7f6898113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:34.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.530+0000 7f689d018700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6898102e10 0x7f6898113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60536/0 (socket says 172.21.15.32:60536) 2024-04-12T05:54:34.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.530+0000 7f689d018700 1 -- 172.21.15.32:0/1362978800 learned_addr learned my addr 172.21.15.32:0/1362978800 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:54:34.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.531+0000 7f689c817700 1 --2- 172.21.15.32:0/1362978800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68981037b0 0x7f6898114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:34.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.531+0000 7f689d018700 1 -- 172.21.15.32:0/1362978800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68981037b0 msgr2=0x7f6898114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:34.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.531+0000 7f689d018700 1 --2- 172.21.15.32:0/1362978800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68981037b0 0x7f6898114030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:34.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.531+0000 7f689d018700 1 -- 172.21.15.32:0/1362978800 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6894005040 con 0x7f6898102e10 2024-04-12T05:54:34.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.531+0000 7f689c817700 1 --2- 172.21.15.32:0/1362978800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68981037b0 0x7f6898114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:54:34.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.531+0000 7f689d018700 1 --2- 172.21.15.32:0/1362978800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6898102e10 0x7f6898113af0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6894009180 tx=0x7f6894009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:54:34.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.531+0000 7f688dffb700 1 -- 172.21.15.32:0/1362978800 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f689400e040 con 0x7f6898102e10 2024-04-12T05:54:34.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.531+0000 7f689f27c700 1 -- 172.21.15.32:0/1362978800 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f689810dcf0 con 0x7f6898102e10 2024-04-12T05:54:34.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.531+0000 7f688dffb700 1 -- 172.21.15.32:0/1362978800 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6894004290 con 0x7f6898102e10 2024-04-12T05:54:34.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.531+0000 7f688dffb700 1 -- 172.21.15.32:0/1362978800 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f689401c3f0 con 0x7f6898102e10 2024-04-12T05:54:34.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.531+0000 7f689f27c700 1 -- 172.21.15.32:0/1362978800 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f689810e2b0 con 0x7f6898102e10 2024-04-12T05:54:34.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.532+0000 7f689f27c700 1 -- 172.21.15.32:0/1362978800 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68980fe560 con 0x7f6898102e10 2024-04-12T05:54:34.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.533+0000 7f688dffb700 1 -- 172.21.15.32:0/1362978800 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f6894013790 con 0x7f6898102e10 2024-04-12T05:54:34.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.533+0000 7f688dffb700 1 --2- 172.21.15.32:0/1362978800 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f688406cc20 0x7f688406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:34.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.533+0000 7f688dffb700 1 -- 172.21.15.32:0/1362978800 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f689408d400 con 0x7f6898102e10 2024-04-12T05:54:34.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.533+0000 7f689c817700 1 --2- 172.21.15.32:0/1362978800 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f688406cc20 0x7f688406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:34.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.534+0000 7f689c817700 1 --2- 172.21.15.32:0/1362978800 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f688406cc20 0x7f688406f0d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6888007fd0 tx=0x7f6888009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:54:34.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.538+0000 7f688dffb700 1 -- 172.21.15.32:0/1362978800 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f689405add0 con 0x7f6898102e10 2024-04-12T05:54:34.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.760+0000 7f689f27c700 1 -- 172.21.15.32:0/1362978800 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6898109e40 con 0x7f688406cc20 2024-04-12T05:54:34.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.762+0000 7f688dffb700 1 -- 172.21.15.32:0/1362978800 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6898109e40 con 0x7f688406cc20 2024-04-12T05:54:34.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.764+0000 7f689f27c700 1 -- 172.21.15.32:0/1362978800 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f688406cc20 msgr2=0x7f688406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:34.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.764+0000 7f689f27c700 1 --2- 172.21.15.32:0/1362978800 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f688406cc20 0x7f688406f0d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6888007fd0 tx=0x7f6888009040 comp rx=0 tx=0).stop 2024-04-12T05:54:34.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.764+0000 7f689f27c700 1 -- 172.21.15.32:0/1362978800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6898102e10 msgr2=0x7f6898113af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:34.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.764+0000 7f689f27c700 1 --2- 172.21.15.32:0/1362978800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6898102e10 0x7f6898113af0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6894009180 tx=0x7f6894009e30 comp rx=0 tx=0).stop 2024-04-12T05:54:34.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.764+0000 7f689f27c700 1 -- 172.21.15.32:0/1362978800 shutdown_connections 2024-04-12T05:54:34.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.764+0000 7f689f27c700 1 --2- 172.21.15.32:0/1362978800 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f688406cc20 0x7f688406f0d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:34.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.764+0000 7f689f27c700 1 --2- 172.21.15.32:0/1362978800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6898102e10 0x7f6898113af0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:34.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.764+0000 7f689f27c700 1 --2- 172.21.15.32:0/1362978800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68981037b0 0x7f6898114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:34.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.764+0000 7f689f27c700 1 -- 172.21.15.32:0/1362978800 >> 172.21.15.32:0/1362978800 conn(0x7f68980f89a0 msgr2=0x7f68980f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:54:34.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.764+0000 7f689f27c700 1 -- 172.21.15.32:0/1362978800 shutdown_connections 2024-04-12T05:54:34.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.764+0000 7f689f27c700 1 -- 172.21.15.32:0/1362978800 wait complete. 2024-04-12T05:54:34.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.867+0000 7fc58af9f700 1 -- 172.21.15.32:0/788518526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5841037a0 msgr2=0x7fc5840fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:34.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.867+0000 7fc58af9f700 1 --2- 172.21.15.32:0/788518526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5841037a0 0x7fc5840fce10 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fc570004940 tx=0x7fc570009e30 comp rx=0 tx=0).stop 2024-04-12T05:54:34.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.867+0000 7fc58af9f700 1 -- 172.21.15.32:0/788518526 shutdown_connections 2024-04-12T05:54:34.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.867+0000 7fc58af9f700 1 --2- 172.21.15.32:0/788518526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5841037a0 0x7fc5840fce10 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:34.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.867+0000 7fc58af9f700 1 --2- 172.21.15.32:0/788518526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc584102e00 0x7fc5841031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:34.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.867+0000 7fc58af9f700 1 -- 172.21.15.32:0/788518526 >> 172.21.15.32:0/788518526 conn(0x7fc5840f89b0 msgr2=0x7fc5840fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:54:34.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.867+0000 7fc58af9f700 1 -- 172.21.15.32:0/788518526 shutdown_connections 2024-04-12T05:54:34.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.867+0000 7fc58af9f700 1 -- 172.21.15.32:0/788518526 wait complete. 2024-04-12T05:54:34.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.868+0000 7fc58af9f700 1 Processor -- start 2024-04-12T05:54:34.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.868+0000 7fc58af9f700 1 -- start start 2024-04-12T05:54:34.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.868+0000 7fc58af9f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc584102e00 0x7fc5840702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:34.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.868+0000 7fc58af9f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5841037a0 0x7fc584070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:34.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.868+0000 7fc58af9f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc584072210 con 0x7fc584102e00 2024-04-12T05:54:34.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.868+0000 7fc58af9f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc584070d40 con 0x7fc5841037a0 2024-04-12T05:54:34.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.868+0000 7fc588d3b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc584102e00 0x7fc5840702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:34.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.869+0000 7fc588d3b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc584102e00 0x7fc5840702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60554/0 (socket says 172.21.15.32:60554) 2024-04-12T05:54:34.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.869+0000 7fc588d3b700 1 -- 172.21.15.32:0/2971627839 learned_addr learned my addr 172.21.15.32:0/2971627839 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:54:34.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.869+0000 7fc583fff700 1 --2- 172.21.15.32:0/2971627839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5841037a0 0x7fc584070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:34.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.869+0000 7fc588d3b700 1 -- 172.21.15.32:0/2971627839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5841037a0 msgr2=0x7fc584070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:34.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.869+0000 7fc588d3b700 1 --2- 172.21.15.32:0/2971627839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5841037a0 0x7fc584070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:34.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.869+0000 7fc588d3b700 1 -- 172.21.15.32:0/2971627839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc570005040 con 0x7fc584102e00 2024-04-12T05:54:34.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.869+0000 7fc588d3b700 1 --2- 172.21.15.32:0/2971627839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc584102e00 0x7fc5840702c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc57800c830 tx=0x7fc578004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:54:34.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.869+0000 7fc5817fa700 1 -- 172.21.15.32:0/2971627839 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc57800a760 con 0x7fc584102e00 2024-04-12T05:54:34.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.869+0000 7fc58af9f700 1 -- 172.21.15.32:0/2971627839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc584071020 con 0x7fc584102e00 2024-04-12T05:54:34.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.869+0000 7fc58af9f700 1 -- 172.21.15.32:0/2971627839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc58411a040 con 0x7fc584102e00 2024-04-12T05:54:34.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.869+0000 7fc5817fa700 1 -- 172.21.15.32:0/2971627839 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc57800ad70 con 0x7fc584102e00 2024-04-12T05:54:34.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.869+0000 7fc5817fa700 1 -- 172.21.15.32:0/2971627839 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc578012400 con 0x7fc584102e00 2024-04-12T05:54:34.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.870+0000 7fc58af9f700 1 -- 172.21.15.32:0/2971627839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5680052b0 con 0x7fc584102e00 2024-04-12T05:54:34.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.871+0000 7fc5817fa700 1 -- 172.21.15.32:0/2971627839 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc578012560 con 0x7fc584102e00 2024-04-12T05:54:34.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.871+0000 7fc5817fa700 1 --2- 172.21.15.32:0/2971627839 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc57406cc20 0x7fc57406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:34.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.871+0000 7fc5817fa700 1 -- 172.21.15.32:0/2971627839 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fc57808c100 con 0x7fc584102e00 2024-04-12T05:54:34.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.871+0000 7fc583fff700 1 --2- 172.21.15.32:0/2971627839 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc57406cc20 0x7fc57406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:34.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.871+0000 7fc583fff700 1 --2- 172.21.15.32:0/2971627839 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc57406cc20 0x7fc57406f0d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc5840719f0 tx=0x7fc570009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:54:34.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:34.876+0000 7fc5817fa700 1 -- 172.21.15.32:0/2971627839 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc5780560e0 con 0x7fc584102e00 2024-04-12T05:54:35.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.086+0000 7fc58af9f700 1 -- 172.21.15.32:0/2971627839 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc568000bc0 con 0x7fc57406cc20 2024-04-12T05:54:35.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.096+0000 7fc5817fa700 1 -- 172.21.15.32:0/2971627839 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fc568000bc0 con 0x7fc57406cc20 2024-04-12T05:54:35.091 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:54:35.091 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (14m) 7m ago 25m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:54:35.091 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (11m) 7m ago 26m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:54:35.091 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (11m) 7m ago 24m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:54:35.091 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (14m) 7m ago 25m 87.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:54:35.091 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (8m) 7m ago 20m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T05:54:35.091 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (7m) 7m ago 20m 13.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T05:54:35.091 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (7m) 7m ago 19m 11.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 6d4aca6f74a0 2024-04-12T05:54:35.091 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (7m) 7m ago 20m 14.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T05:54:35.091 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (16m) 7m ago 27m 507M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:54:35.092 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (15m) 7m ago 24m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:54:35.092 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (12m) 7m ago 27m 53.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:54:35.092 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (12m) 7m ago 24m 43.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:54:35.092 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (15m) 7m ago 25m 16.9M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:54:35.092 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (14m) 7m ago 24m 18.4M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:54:35.092 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (11m) 7m ago 24m 91.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T05:54:35.092 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (10m) 7m ago 23m 97.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T05:54:35.092 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (10m) 7m ago 23m 81.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T05:54:35.092 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (9m) 7m ago 22m 113M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T05:54:35.092 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (9m) 7m ago 22m 90.3M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T05:54:35.092 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (8m) 7m ago 21m 84.8M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T05:54:35.092 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (14m) 7m ago 25m 64.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:54:35.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.098+0000 7fc58af9f700 1 -- 172.21.15.32:0/2971627839 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc57406cc20 msgr2=0x7fc57406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:35.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.098+0000 7fc58af9f700 1 --2- 172.21.15.32:0/2971627839 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc57406cc20 0x7fc57406f0d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc5840719f0 tx=0x7fc570009e30 comp rx=0 tx=0).stop 2024-04-12T05:54:35.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.098+0000 7fc58af9f700 1 -- 172.21.15.32:0/2971627839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc584102e00 msgr2=0x7fc5840702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:35.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.098+0000 7fc58af9f700 1 --2- 172.21.15.32:0/2971627839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc584102e00 0x7fc5840702c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc57800c830 tx=0x7fc578004470 comp rx=0 tx=0).stop 2024-04-12T05:54:35.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.098+0000 7fc58af9f700 1 -- 172.21.15.32:0/2971627839 shutdown_connections 2024-04-12T05:54:35.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.098+0000 7fc58af9f700 1 --2- 172.21.15.32:0/2971627839 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc57406cc20 0x7fc57406f0d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:35.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.098+0000 7fc58af9f700 1 --2- 172.21.15.32:0/2971627839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc584102e00 0x7fc5840702c0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:35.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.098+0000 7fc58af9f700 1 --2- 172.21.15.32:0/2971627839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5841037a0 0x7fc584070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:35.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.098+0000 7fc58af9f700 1 -- 172.21.15.32:0/2971627839 >> 172.21.15.32:0/2971627839 conn(0x7fc5840f89b0 msgr2=0x7fc5840f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:54:35.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.098+0000 7fc58af9f700 1 -- 172.21.15.32:0/2971627839 shutdown_connections 2024-04-12T05:54:35.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.098+0000 7fc58af9f700 1 -- 172.21.15.32:0/2971627839 wait complete. 2024-04-12T05:54:35.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.201+0000 7f9dc20a7700 1 -- 172.21.15.32:0/2986077825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dbc102e10 msgr2=0x7f9dbc1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:35.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.201+0000 7f9dc20a7700 1 --2- 172.21.15.32:0/2986077825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dbc102e10 0x7f9dbc1031e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9db0004940 tx=0x7f9db0009e30 comp rx=0 tx=0).stop 2024-04-12T05:54:35.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.201+0000 7f9dc20a7700 1 -- 172.21.15.32:0/2986077825 shutdown_connections 2024-04-12T05:54:35.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.201+0000 7f9dc20a7700 1 --2- 172.21.15.32:0/2986077825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dbc1037b0 0x7f9dbc0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:35.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.201+0000 7f9dc20a7700 1 --2- 172.21.15.32:0/2986077825 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dbc102e10 0x7f9dbc1031e0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:35.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.201+0000 7f9dc20a7700 1 -- 172.21.15.32:0/2986077825 >> 172.21.15.32:0/2986077825 conn(0x7f9dbc0f8980 msgr2=0x7f9dbc0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:54:35.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.201+0000 7f9dc20a7700 1 -- 172.21.15.32:0/2986077825 shutdown_connections 2024-04-12T05:54:35.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.201+0000 7f9dc20a7700 1 -- 172.21.15.32:0/2986077825 wait complete. 2024-04-12T05:54:35.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.201+0000 7f9dc20a7700 1 Processor -- start 2024-04-12T05:54:35.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.202+0000 7f9dc20a7700 1 -- start start 2024-04-12T05:54:35.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.202+0000 7f9dc20a7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dbc102e10 0x7f9dbc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:35.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.202+0000 7f9dc20a7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dbc1037b0 0x7f9dbc070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:35.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.202+0000 7f9dc20a7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dbc070d90 con 0x7f9dbc1037b0 2024-04-12T05:54:35.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.202+0000 7f9dc20a7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dbc070ed0 con 0x7f9dbc102e10 2024-04-12T05:54:35.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.202+0000 7f9dbaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dbc1037b0 0x7f9dbc070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:35.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.202+0000 7f9dbaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dbc1037b0 0x7f9dbc070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60570/0 (socket says 172.21.15.32:60570) 2024-04-12T05:54:35.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.202+0000 7f9dbaffd700 1 -- 172.21.15.32:0/1905581510 learned_addr learned my addr 172.21.15.32:0/1905581510 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:54:35.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.202+0000 7f9dbb7fe700 1 --2- 172.21.15.32:0/1905581510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dbc102e10 0x7f9dbc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:35.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.202+0000 7f9dbaffd700 1 -- 172.21.15.32:0/1905581510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dbc102e10 msgr2=0x7f9dbc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:35.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.202+0000 7f9dbaffd700 1 --2- 172.21.15.32:0/1905581510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dbc102e10 0x7f9dbc0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:35.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.202+0000 7f9dbaffd700 1 -- 172.21.15.32:0/1905581510 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db0005040 con 0x7f9dbc1037b0 2024-04-12T05:54:35.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.202+0000 7f9dbb7fe700 1 --2- 172.21.15.32:0/1905581510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dbc102e10 0x7f9dbc0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:54:35.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.202+0000 7f9dbaffd700 1 --2- 172.21.15.32:0/1905581510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dbc1037b0 0x7f9dbc070800 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f9dac00c800 tx=0x7f9dac004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:54:35.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.203+0000 7f9da3fff700 1 -- 172.21.15.32:0/1905581510 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9dac00a760 con 0x7f9dbc1037b0 2024-04-12T05:54:35.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.203+0000 7f9dc20a7700 1 -- 172.21.15.32:0/1905581510 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dbc119dd0 con 0x7f9dbc1037b0 2024-04-12T05:54:35.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.203+0000 7f9da3fff700 1 -- 172.21.15.32:0/1905581510 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9dac00ad70 con 0x7f9dbc1037b0 2024-04-12T05:54:35.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.203+0000 7f9da3fff700 1 -- 172.21.15.32:0/1905581510 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9dac0113a0 con 0x7f9dbc1037b0 2024-04-12T05:54:35.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.203+0000 7f9dc20a7700 1 -- 172.21.15.32:0/1905581510 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dbc11a250 con 0x7f9dbc1037b0 2024-04-12T05:54:35.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.204+0000 7f9dc20a7700 1 -- 172.21.15.32:0/1905581510 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dbc0fe560 con 0x7f9dbc1037b0 2024-04-12T05:54:35.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.204+0000 7f9da3fff700 1 -- 172.21.15.32:0/1905581510 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9dac012400 con 0x7f9dbc1037b0 2024-04-12T05:54:35.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.205+0000 7f9da3fff700 1 --2- 172.21.15.32:0/1905581510 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9da406cc20 0x7f9da406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:35.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.205+0000 7f9da3fff700 1 -- 172.21.15.32:0/1905581510 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f9dac08c0d0 con 0x7f9dbc1037b0 2024-04-12T05:54:35.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.205+0000 7f9dbb7fe700 1 --2- 172.21.15.32:0/1905581510 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9da406cc20 0x7f9da406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:35.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.205+0000 7f9dbb7fe700 1 --2- 172.21.15.32:0/1905581510 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9da406cc20 0x7f9da406f0d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9db0004f90 tx=0x7f9db0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:54:35.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.209+0000 7f9da3fff700 1 -- 172.21.15.32:0/1905581510 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9dac0560b0 con 0x7f9dbc1037b0 2024-04-12T05:54:35.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.496+0000 7f9dc20a7700 1 -- 172.21.15.32:0/1905581510 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9dbc04bed0 con 0x7f9dbc1037b0 2024-04-12T05:54:35.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.496+0000 7f9da3fff700 1 -- 172.21.15.32:0/1905581510 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f9dac0596d0 con 0x7f9dbc1037b0 2024-04-12T05:54:35.491 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:54:35.491 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:54:35.491 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:54:35.491 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:54:35.491 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:54:35.491 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:54:35.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:54:35.492 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:54:35.492 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:54:35.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:54:35.492 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:54:35.492 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:54:35.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:54:35.492 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:54:35.492 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:54:35.492 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:54:35.492 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:54:35.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.499+0000 7f9dc20a7700 1 -- 172.21.15.32:0/1905581510 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9da406cc20 msgr2=0x7f9da406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:35.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.499+0000 7f9dc20a7700 1 --2- 172.21.15.32:0/1905581510 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9da406cc20 0x7f9da406f0d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9db0004f90 tx=0x7f9db0009e30 comp rx=0 tx=0).stop 2024-04-12T05:54:35.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.499+0000 7f9dc20a7700 1 -- 172.21.15.32:0/1905581510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dbc1037b0 msgr2=0x7f9dbc070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:35.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.499+0000 7f9dc20a7700 1 --2- 172.21.15.32:0/1905581510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dbc1037b0 0x7f9dbc070800 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f9dac00c800 tx=0x7f9dac004470 comp rx=0 tx=0).stop 2024-04-12T05:54:35.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.499+0000 7f9dc20a7700 1 -- 172.21.15.32:0/1905581510 shutdown_connections 2024-04-12T05:54:35.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.499+0000 7f9dc20a7700 1 --2- 172.21.15.32:0/1905581510 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9da406cc20 0x7f9da406f0d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:35.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.499+0000 7f9dc20a7700 1 --2- 172.21.15.32:0/1905581510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dbc102e10 0x7f9dbc0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:35.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.499+0000 7f9dc20a7700 1 --2- 172.21.15.32:0/1905581510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dbc1037b0 0x7f9dbc070800 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:35.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.499+0000 7f9dc20a7700 1 -- 172.21.15.32:0/1905581510 >> 172.21.15.32:0/1905581510 conn(0x7f9dbc0f8980 msgr2=0x7f9dbc0f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:54:35.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.499+0000 7f9dc20a7700 1 -- 172.21.15.32:0/1905581510 shutdown_connections 2024-04-12T05:54:35.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.499+0000 7f9dc20a7700 1 -- 172.21.15.32:0/1905581510 wait complete. 2024-04-12T05:54:35.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.602+0000 7fe36c817700 1 -- 172.21.15.32:0/561083707 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3681037b0 msgr2=0x7fe3680fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:35.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.602+0000 7fe36c817700 1 --2- 172.21.15.32:0/561083707 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3681037b0 0x7fe3680fce20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe358004940 tx=0x7fe358009e30 comp rx=0 tx=0).stop 2024-04-12T05:54:35.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.602+0000 7fe36c817700 1 -- 172.21.15.32:0/561083707 shutdown_connections 2024-04-12T05:54:35.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.602+0000 7fe36c817700 1 --2- 172.21.15.32:0/561083707 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3681037b0 0x7fe3680fce20 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:35.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.602+0000 7fe36c817700 1 --2- 172.21.15.32:0/561083707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe368102e10 0x7fe3681031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:35.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.602+0000 7fe36c817700 1 -- 172.21.15.32:0/561083707 >> 172.21.15.32:0/561083707 conn(0x7fe3680f89a0 msgr2=0x7fe3680fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:54:35.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.603+0000 7fe36c817700 1 -- 172.21.15.32:0/561083707 shutdown_connections 2024-04-12T05:54:35.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.603+0000 7fe36c817700 1 -- 172.21.15.32:0/561083707 wait complete. 2024-04-12T05:54:35.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.603+0000 7fe36c817700 1 Processor -- start 2024-04-12T05:54:35.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.603+0000 7fe36c817700 1 -- start start 2024-04-12T05:54:35.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.603+0000 7fe36c817700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe368102e10 0x7fe3680702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:35.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.603+0000 7fe36c817700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3681037b0 0x7fe368070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:35.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.603+0000 7fe36c817700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe368070d90 con 0x7fe368102e10 2024-04-12T05:54:35.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.603+0000 7fe36c817700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe368070ed0 con 0x7fe3681037b0 2024-04-12T05:54:35.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.603+0000 7fe36659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe368102e10 0x7fe3680702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:35.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.603+0000 7fe36659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe368102e10 0x7fe3680702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60584/0 (socket says 172.21.15.32:60584) 2024-04-12T05:54:35.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.603+0000 7fe36659c700 1 -- 172.21.15.32:0/1444154165 learned_addr learned my addr 172.21.15.32:0/1444154165 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:54:35.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.604+0000 7fe365d9b700 1 --2- 172.21.15.32:0/1444154165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3681037b0 0x7fe368070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:35.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.604+0000 7fe36659c700 1 -- 172.21.15.32:0/1444154165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3681037b0 msgr2=0x7fe368070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:35.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.604+0000 7fe36659c700 1 --2- 172.21.15.32:0/1444154165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3681037b0 0x7fe368070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:35.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.604+0000 7fe36659c700 1 -- 172.21.15.32:0/1444154165 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe358005040 con 0x7fe368102e10 2024-04-12T05:54:35.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.604+0000 7fe365d9b700 1 --2- 172.21.15.32:0/1444154165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3681037b0 0x7fe368070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:54:35.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.604+0000 7fe36659c700 1 --2- 172.21.15.32:0/1444154165 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe368102e10 0x7fe3680702c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fe35c002a50 tx=0x7fe35c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:54:35.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.604+0000 7fe356ffd700 1 -- 172.21.15.32:0/1444154165 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe35c007cc0 con 0x7fe368102e10 2024-04-12T05:54:35.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.604+0000 7fe356ffd700 1 -- 172.21.15.32:0/1444154165 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe35c00f040 con 0x7fe368102e10 2024-04-12T05:54:35.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.604+0000 7fe36c817700 1 -- 172.21.15.32:0/1444154165 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe368119dd0 con 0x7fe368102e10 2024-04-12T05:54:35.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.604+0000 7fe356ffd700 1 -- 172.21.15.32:0/1444154165 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe35c009540 con 0x7fe368102e10 2024-04-12T05:54:35.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.604+0000 7fe36c817700 1 -- 172.21.15.32:0/1444154165 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe36811a250 con 0x7fe368102e10 2024-04-12T05:54:35.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.605+0000 7fe36c817700 1 -- 172.21.15.32:0/1444154165 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3680fe560 con 0x7fe368102e10 2024-04-12T05:54:35.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.606+0000 7fe356ffd700 1 -- 172.21.15.32:0/1444154165 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe35c0096a0 con 0x7fe368102e10 2024-04-12T05:54:35.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.606+0000 7fe356ffd700 1 --2- 172.21.15.32:0/1444154165 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe35006cc30 0x7fe35006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:35.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.606+0000 7fe356ffd700 1 -- 172.21.15.32:0/1444154165 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fe35c08bdf0 con 0x7fe368102e10 2024-04-12T05:54:35.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.606+0000 7fe365d9b700 1 --2- 172.21.15.32:0/1444154165 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe35006cc30 0x7fe35006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:35.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.606+0000 7fe365d9b700 1 --2- 172.21.15.32:0/1444154165 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe35006cc30 0x7fe35006f0e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe358009fd0 tx=0x7fe358009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:54:35.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.610+0000 7fe356ffd700 1 -- 172.21.15.32:0/1444154165 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe35c055dd0 con 0x7fe368102e10 2024-04-12T05:54:35.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.857+0000 7fe36c817700 1 -- 172.21.15.32:0/1444154165 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe3680719b0 con 0x7fe368102e10 2024-04-12T05:54:35.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.857+0000 7fe356ffd700 1 -- 172.21.15.32:0/1444154165 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1835 (secure 0 0 0) 0x7fe35c0593f0 con 0x7fe368102e10 2024-04-12T05:54:35.852 INFO:teuthology.orchestra.run.smithi032.stdout:e33 2024-04-12T05:54:35.852 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:54:35.852 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:54:35.852 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:54:35.852 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:54:35.852 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:54:35.853 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:54:35.853 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 24 2024-04-12T05:54:35.853 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:54:35.853 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:54:35.853 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:46:10.668569+0000 2024-04-12T05:54:35.853 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:54:35.853 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:54:35.853 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:54:35.853 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:54:35.853 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:54:35.853 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:54:35.853 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:54:35.853 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:54:35.854 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T05:54:35.854 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:54:35.854 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:54:35.854 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:54:35.854 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T05:54:35.854 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-12T05:54:35.854 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:54:35.854 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:54:35.854 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:54:35.854 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:54:35.854 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:54:35.854 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:54:35.854 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:54:35.855 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:54:35.855 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:54:35.855 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:54:35.855 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-12T05:54:35.855 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:54:35.855 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:54:35.855 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:54:35.855 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{-1:44317} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:54:35.855 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:54:35.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.859+0000 7fe36c817700 1 -- 172.21.15.32:0/1444154165 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe35006cc30 msgr2=0x7fe35006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:35.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.859+0000 7fe36c817700 1 --2- 172.21.15.32:0/1444154165 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe35006cc30 0x7fe35006f0e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe358009fd0 tx=0x7fe358009e30 comp rx=0 tx=0).stop 2024-04-12T05:54:35.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.859+0000 7fe36c817700 1 -- 172.21.15.32:0/1444154165 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe368102e10 msgr2=0x7fe3680702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:35.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.859+0000 7fe36c817700 1 --2- 172.21.15.32:0/1444154165 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe368102e10 0x7fe3680702c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fe35c002a50 tx=0x7fe35c0046b0 comp rx=0 tx=0).stop 2024-04-12T05:54:35.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.859+0000 7fe36c817700 1 -- 172.21.15.32:0/1444154165 shutdown_connections 2024-04-12T05:54:35.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.859+0000 7fe36c817700 1 --2- 172.21.15.32:0/1444154165 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe35006cc30 0x7fe35006f0e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:35.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.859+0000 7fe36c817700 1 --2- 172.21.15.32:0/1444154165 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe368102e10 0x7fe3680702c0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:35.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.859+0000 7fe36c817700 1 --2- 172.21.15.32:0/1444154165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3681037b0 0x7fe368070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:35.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.859+0000 7fe36c817700 1 -- 172.21.15.32:0/1444154165 >> 172.21.15.32:0/1444154165 conn(0x7fe3680f89a0 msgr2=0x7fe3680f9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:54:35.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.859+0000 7fe36c817700 1 -- 172.21.15.32:0/1444154165 shutdown_connections 2024-04-12T05:54:35.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.859+0000 7fe36c817700 1 -- 172.21.15.32:0/1444154165 wait complete. 2024-04-12T05:54:35.857 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-12T05:54:35.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.964+0000 7feb9e5b9700 1 -- 172.21.15.32:0/499391549 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb981037a0 msgr2=0x7feb980fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:35.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.964+0000 7feb9e5b9700 1 --2- 172.21.15.32:0/499391549 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb981037a0 0x7feb980fce10 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7feb8c004940 tx=0x7feb8c009e30 comp rx=0 tx=0).stop 2024-04-12T05:54:35.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.964+0000 7feb9e5b9700 1 -- 172.21.15.32:0/499391549 shutdown_connections 2024-04-12T05:54:35.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.964+0000 7feb9e5b9700 1 --2- 172.21.15.32:0/499391549 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb981037a0 0x7feb980fce10 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:35.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.964+0000 7feb9e5b9700 1 --2- 172.21.15.32:0/499391549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb98102e00 0x7feb981031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:35.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.964+0000 7feb9e5b9700 1 -- 172.21.15.32:0/499391549 >> 172.21.15.32:0/499391549 conn(0x7feb980f89b0 msgr2=0x7feb980fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:54:35.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.964+0000 7feb9e5b9700 1 -- 172.21.15.32:0/499391549 shutdown_connections 2024-04-12T05:54:35.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.964+0000 7feb9e5b9700 1 -- 172.21.15.32:0/499391549 wait complete. 2024-04-12T05:54:35.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.965+0000 7feb9e5b9700 1 Processor -- start 2024-04-12T05:54:35.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.965+0000 7feb9e5b9700 1 -- start start 2024-04-12T05:54:35.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.965+0000 7feb9e5b9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb98102e00 0x7feb980702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:35.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.965+0000 7feb9e5b9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb981037a0 0x7feb98070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:35.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.965+0000 7feb9e5b9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb98072210 con 0x7feb981037a0 2024-04-12T05:54:35.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.965+0000 7feb9e5b9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb98070d40 con 0x7feb98102e00 2024-04-12T05:54:35.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.965+0000 7feb977fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb981037a0 0x7feb98070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:35.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.966+0000 7feb977fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb981037a0 0x7feb98070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60602/0 (socket says 172.21.15.32:60602) 2024-04-12T05:54:35.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.966+0000 7feb977fe700 1 -- 172.21.15.32:0/2475638056 learned_addr learned my addr 172.21.15.32:0/2475638056 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:54:35.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.966+0000 7feb97fff700 1 --2- 172.21.15.32:0/2475638056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb98102e00 0x7feb980702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:35.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.966+0000 7feb977fe700 1 -- 172.21.15.32:0/2475638056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb98102e00 msgr2=0x7feb980702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:35.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.966+0000 7feb977fe700 1 --2- 172.21.15.32:0/2475638056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb98102e00 0x7feb980702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:35.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.966+0000 7feb977fe700 1 -- 172.21.15.32:0/2475638056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8c005040 con 0x7feb981037a0 2024-04-12T05:54:35.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.966+0000 7feb977fe700 1 --2- 172.21.15.32:0/2475638056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb981037a0 0x7feb98070800 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7feb8c009380 tx=0x7feb8c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:54:35.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.966+0000 7feb94ff9700 1 -- 172.21.15.32:0/2475638056 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb8c00e040 con 0x7feb981037a0 2024-04-12T05:54:35.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.966+0000 7feb94ff9700 1 -- 172.21.15.32:0/2475638056 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb8c013420 con 0x7feb981037a0 2024-04-12T05:54:35.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.966+0000 7feb94ff9700 1 -- 172.21.15.32:0/2475638056 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb8c01c630 con 0x7feb981037a0 2024-04-12T05:54:35.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.966+0000 7feb9e5b9700 1 -- 172.21.15.32:0/2475638056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb98070fc0 con 0x7feb981037a0 2024-04-12T05:54:35.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.966+0000 7feb9e5b9700 1 -- 172.21.15.32:0/2475638056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb9811a010 con 0x7feb981037a0 2024-04-12T05:54:35.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.967+0000 7feb9e5b9700 1 -- 172.21.15.32:0/2475638056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb980fe550 con 0x7feb981037a0 2024-04-12T05:54:35.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.968+0000 7feb94ff9700 1 -- 172.21.15.32:0/2475638056 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7feb8c0139e0 con 0x7feb981037a0 2024-04-12T05:54:35.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.968+0000 7feb94ff9700 1 --2- 172.21.15.32:0/2475638056 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7feb8006cc20 0x7feb8006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:35.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.968+0000 7feb94ff9700 1 -- 172.21.15.32:0/2475638056 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7feb8c08d170 con 0x7feb981037a0 2024-04-12T05:54:35.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.968+0000 7feb97fff700 1 --2- 172.21.15.32:0/2475638056 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7feb8006cc20 0x7feb8006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:35.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.969+0000 7feb97fff700 1 --2- 172.21.15.32:0/2475638056 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7feb8006cc20 0x7feb8006f0d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7feb88004990 tx=0x7feb88009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:54:35.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:35.973+0000 7feb94ff9700 1 -- 172.21.15.32:0/2475638056 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7feb8c0571f0 con 0x7feb981037a0 2024-04-12T05:54:36.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:35 smithi067 ceph-mon[112979]: pgmap v422: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:36.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:35 smithi067 ceph-mon[112979]: from='client.34816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:54:36.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:35 smithi067 ceph-mon[112979]: from='client.34820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:54:36.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:35 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1905581510' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:54:36.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:35 smithi032 ceph-mon[146817]: pgmap v422: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:36.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:35 smithi032 ceph-mon[146817]: from='client.34816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:54:36.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:35 smithi032 ceph-mon[146817]: from='client.34820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:54:36.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:35 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1905581510' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:54:36.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.196+0000 7feb9e5b9700 1 -- 172.21.15.32:0/2475638056 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7feb980fa140 con 0x7feb8006cc20 2024-04-12T05:54:36.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.198+0000 7feb94ff9700 1 -- 172.21.15.32:0/2475638056 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7feb980fa140 con 0x7feb8006cc20 2024-04-12T05:54:36.192 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:54:36.192 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:54:36.192 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:54:36.192 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:54:36.193 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:54:36.193 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-12T05:54:36.193 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:54:36.193 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-12T05:54:36.193 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:54:36.193 INFO:teuthology.orchestra.run.smithi032.stdout: "mds" 2024-04-12T05:54:36.193 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:54:36.193 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:54:36.193 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:54:36.193 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:54:36.193 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:54:36.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.200+0000 7feb9e5b9700 1 -- 172.21.15.32:0/2475638056 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7feb8006cc20 msgr2=0x7feb8006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:36.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.200+0000 7feb9e5b9700 1 --2- 172.21.15.32:0/2475638056 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7feb8006cc20 0x7feb8006f0d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7feb88004990 tx=0x7feb88009ee0 comp rx=0 tx=0).stop 2024-04-12T05:54:36.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.200+0000 7feb9e5b9700 1 -- 172.21.15.32:0/2475638056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb981037a0 msgr2=0x7feb98070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:36.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.200+0000 7feb9e5b9700 1 --2- 172.21.15.32:0/2475638056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb981037a0 0x7feb98070800 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7feb8c009380 tx=0x7feb8c009e30 comp rx=0 tx=0).stop 2024-04-12T05:54:36.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.200+0000 7feb9e5b9700 1 -- 172.21.15.32:0/2475638056 shutdown_connections 2024-04-12T05:54:36.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.200+0000 7feb9e5b9700 1 --2- 172.21.15.32:0/2475638056 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7feb8006cc20 0x7feb8006f0d0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:36.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.200+0000 7feb9e5b9700 1 --2- 172.21.15.32:0/2475638056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb98102e00 0x7feb980702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:36.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.200+0000 7feb9e5b9700 1 --2- 172.21.15.32:0/2475638056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb981037a0 0x7feb98070800 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:36.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.200+0000 7feb9e5b9700 1 -- 172.21.15.32:0/2475638056 >> 172.21.15.32:0/2475638056 conn(0x7feb980f89b0 msgr2=0x7feb980f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:54:36.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.200+0000 7feb9e5b9700 1 -- 172.21.15.32:0/2475638056 shutdown_connections 2024-04-12T05:54:36.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.200+0000 7feb9e5b9700 1 -- 172.21.15.32:0/2475638056 wait complete. 2024-04-12T05:54:36.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.303+0000 7fb811d01700 1 -- 172.21.15.32:0/1460805374 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb80c102710 msgr2=0x7fb80c0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:36.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.303+0000 7fb811d01700 1 --2- 172.21.15.32:0/1460805374 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb80c102710 0x7fb80c0fbd80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb7f4004940 tx=0x7fb7f4009e30 comp rx=0 tx=0).stop 2024-04-12T05:54:36.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.303+0000 7fb811d01700 1 -- 172.21.15.32:0/1460805374 shutdown_connections 2024-04-12T05:54:36.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.303+0000 7fb811d01700 1 --2- 172.21.15.32:0/1460805374 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb80c102710 0x7fb80c0fbd80 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:36.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.303+0000 7fb811d01700 1 --2- 172.21.15.32:0/1460805374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb80c101d70 0x7fb80c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:36.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.303+0000 7fb811d01700 1 -- 172.21.15.32:0/1460805374 >> 172.21.15.32:0/1460805374 conn(0x7fb80c072bc0 msgr2=0x7fb80c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:54:36.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.303+0000 7fb811d01700 1 -- 172.21.15.32:0/1460805374 shutdown_connections 2024-04-12T05:54:36.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.303+0000 7fb811d01700 1 -- 172.21.15.32:0/1460805374 wait complete. 2024-04-12T05:54:36.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.303+0000 7fb811d01700 1 Processor -- start 2024-04-12T05:54:36.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.304+0000 7fb811d01700 1 -- start start 2024-04-12T05:54:36.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.304+0000 7fb811d01700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb80c101d70 0x7fb80c10e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:36.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.304+0000 7fb811d01700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb80c102710 0x7fb80c10eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:36.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.304+0000 7fb811d01700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80c10f530 con 0x7fb80c101d70 2024-04-12T05:54:36.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.304+0000 7fb811d01700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80c10f670 con 0x7fb80c102710 2024-04-12T05:54:36.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.304+0000 7fb80b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb80c101d70 0x7fb80c10e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:36.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.304+0000 7fb80b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb80c101d70 0x7fb80c10e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60622/0 (socket says 172.21.15.32:60622) 2024-04-12T05:54:36.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.304+0000 7fb80b7fe700 1 -- 172.21.15.32:0/1566487945 learned_addr learned my addr 172.21.15.32:0/1566487945 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:54:36.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.304+0000 7fb80affd700 1 --2- 172.21.15.32:0/1566487945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb80c102710 0x7fb80c10eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:36.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.304+0000 7fb80b7fe700 1 -- 172.21.15.32:0/1566487945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb80c102710 msgr2=0x7fb80c10eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:36.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.304+0000 7fb80b7fe700 1 --2- 172.21.15.32:0/1566487945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb80c102710 0x7fb80c10eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:36.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.304+0000 7fb80b7fe700 1 -- 172.21.15.32:0/1566487945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f4005040 con 0x7fb80c101d70 2024-04-12T05:54:36.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.304+0000 7fb80b7fe700 1 --2- 172.21.15.32:0/1566487945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb80c101d70 0x7fb80c10e9a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc00eb40 tx=0x7fb7fc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:54:36.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.304+0000 7fb803fff700 1 -- 172.21.15.32:0/1566487945 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7fc007cc0 con 0x7fb80c101d70 2024-04-12T05:54:36.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.305+0000 7fb811d01700 1 -- 172.21.15.32:0/1566487945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb80c117a10 con 0x7fb80c101d70 2024-04-12T05:54:36.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.305+0000 7fb803fff700 1 -- 172.21.15.32:0/1566487945 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb7fc00f040 con 0x7fb80c101d70 2024-04-12T05:54:36.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.305+0000 7fb803fff700 1 -- 172.21.15.32:0/1566487945 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7fc009570 con 0x7fb80c101d70 2024-04-12T05:54:36.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.305+0000 7fb811d01700 1 -- 172.21.15.32:0/1566487945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb80c117f20 con 0x7fb80c101d70 2024-04-12T05:54:36.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.305+0000 7fb811d01700 1 -- 172.21.15.32:0/1566487945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb80c0ff080 con 0x7fb80c101d70 2024-04-12T05:54:36.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.306+0000 7fb803fff700 1 -- 172.21.15.32:0/1566487945 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb7fc0096d0 con 0x7fb80c101d70 2024-04-12T05:54:36.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.306+0000 7fb803fff700 1 --2- 172.21.15.32:0/1566487945 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb7f806cc30 0x7fb7f806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:54:36.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.307+0000 7fb803fff700 1 -- 172.21.15.32:0/1566487945 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fb7fc08bd60 con 0x7fb80c101d70 2024-04-12T05:54:36.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.307+0000 7fb80affd700 1 --2- 172.21.15.32:0/1566487945 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb7f806cc30 0x7fb7f806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:54:36.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.307+0000 7fb80affd700 1 --2- 172.21.15.32:0/1566487945 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb7f806cc30 0x7fb7f806f0e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb7f400b0d0 tx=0x7fb7f4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:54:36.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.311+0000 7fb803fff700 1 -- 172.21.15.32:0/1566487945 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb7fc055d70 con 0x7fb80c101d70 2024-04-12T05:54:36.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.597+0000 7fb811d01700 1 -- 172.21.15.32:0/1566487945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb80c0fd4c0 con 0x7fb80c101d70 2024-04-12T05:54:36.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.597+0000 7fb803fff700 1 -- 172.21.15.32:0/1566487945 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fb7fc059390 con 0x7fb80c101d70 2024-04-12T05:54:36.592 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:54:36.592 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:54:36.592 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-12T05:54:36.592 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:54:36.592 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:54:36.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.599+0000 7fb811d01700 1 -- 172.21.15.32:0/1566487945 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb7f806cc30 msgr2=0x7fb7f806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:36.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.599+0000 7fb811d01700 1 --2- 172.21.15.32:0/1566487945 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb7f806cc30 0x7fb7f806f0e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb7f400b0d0 tx=0x7fb7f4009e30 comp rx=0 tx=0).stop 2024-04-12T05:54:36.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.599+0000 7fb811d01700 1 -- 172.21.15.32:0/1566487945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb80c101d70 msgr2=0x7fb80c10e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:54:36.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.599+0000 7fb811d01700 1 --2- 172.21.15.32:0/1566487945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb80c101d70 0x7fb80c10e9a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc00eb40 tx=0x7fb7fc00b040 comp rx=0 tx=0).stop 2024-04-12T05:54:36.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.599+0000 7fb811d01700 1 -- 172.21.15.32:0/1566487945 shutdown_connections 2024-04-12T05:54:36.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.599+0000 7fb811d01700 1 --2- 172.21.15.32:0/1566487945 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb7f806cc30 0x7fb7f806f0e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:36.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.599+0000 7fb811d01700 1 --2- 172.21.15.32:0/1566487945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb80c101d70 0x7fb80c10e9a0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:36.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.599+0000 7fb811d01700 1 --2- 172.21.15.32:0/1566487945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb80c102710 0x7fb80c10eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:54:36.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.599+0000 7fb811d01700 1 -- 172.21.15.32:0/1566487945 >> 172.21.15.32:0/1566487945 conn(0x7fb80c072bc0 msgr2=0x7fb80c0faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:54:36.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.600+0000 7fb811d01700 1 -- 172.21.15.32:0/1566487945 shutdown_connections 2024-04-12T05:54:36.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:54:36.600+0000 7fb811d01700 1 -- 172.21.15.32:0/1566487945 wait complete. 2024-04-12T05:54:37.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:36 smithi067 ceph-mon[112979]: from='client.34824 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:54:37.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:36 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1444154165' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:54:37.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:36 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1566487945' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:54:37.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:36 smithi032 ceph-mon[146817]: from='client.34824 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:54:37.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:36 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1444154165' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:54:37.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:36 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1566487945' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:54:38.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:37 smithi067 ceph-mon[112979]: pgmap v423: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:38.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:37 smithi067 ceph-mon[112979]: from='client.34836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:54:38.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:37 smithi032 ceph-mon[146817]: pgmap v423: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:38.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:37 smithi032 ceph-mon[146817]: from='client.34836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:54:40.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:39 smithi067 ceph-mon[112979]: pgmap v424: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:40.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:39 smithi032 ceph-mon[146817]: pgmap v424: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:42.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:41 smithi067 ceph-mon[112979]: pgmap v425: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:42.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:41 smithi032 ceph-mon[146817]: pgmap v425: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:44.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:43 smithi032 ceph-mon[146817]: pgmap v426: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:44.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:43 smithi067 ceph-mon[112979]: pgmap v426: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:46.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:45 smithi032 ceph-mon[146817]: pgmap v427: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:46.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:45 smithi067 ceph-mon[112979]: pgmap v427: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:48.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:47 smithi032 ceph-mon[146817]: pgmap v428: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:48.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:47 smithi067 ceph-mon[112979]: pgmap v428: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:49.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:48 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:54:49.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:48 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:54:50.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:49 smithi032 ceph-mon[146817]: pgmap v429: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:50.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:49 smithi067 ceph-mon[112979]: pgmap v429: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:52.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:51 smithi032 ceph-mon[146817]: pgmap v430: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:52.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:51 smithi067 ceph-mon[112979]: pgmap v430: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:54.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:53 smithi032 ceph-mon[146817]: pgmap v431: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:54.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:53 smithi067 ceph-mon[112979]: pgmap v431: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:56.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:55 smithi032 ceph-mon[146817]: pgmap v432: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:56.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:55 smithi067 ceph-mon[112979]: pgmap v432: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:58.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:57 smithi032 ceph-mon[146817]: pgmap v433: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:58.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:57 smithi067 ceph-mon[112979]: pgmap v433: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:00.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:54:59 smithi032 ceph-mon[146817]: pgmap v434: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:00.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:54:59 smithi067 ceph-mon[112979]: pgmap v434: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:02.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:01 smithi032 ceph-mon[146817]: pgmap v435: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:02.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:01 smithi067 ceph-mon[112979]: pgmap v435: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:04.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:03 smithi032 ceph-mon[146817]: pgmap v436: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:04.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:55:04.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:03 smithi067 ceph-mon[112979]: pgmap v436: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:04.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:55:06.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:05 smithi032 ceph-mon[146817]: pgmap v437: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:06.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:05 smithi067 ceph-mon[112979]: pgmap v437: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:06.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.718+0000 7fd8eba64700 1 -- 172.21.15.32:0/89296362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8e4102e10 msgr2=0x7fd8e41031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:06.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.718+0000 7fd8eba64700 1 --2- 172.21.15.32:0/89296362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8e4102e10 0x7fd8e41031e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd8e0004940 tx=0x7fd8e0009e30 comp rx=0 tx=0).stop 2024-04-12T05:55:06.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.718+0000 7fd8eba64700 1 -- 172.21.15.32:0/89296362 shutdown_connections 2024-04-12T05:55:06.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.718+0000 7fd8eba64700 1 --2- 172.21.15.32:0/89296362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8e41037b0 0x7fd8e40fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:06.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.718+0000 7fd8eba64700 1 --2- 172.21.15.32:0/89296362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8e4102e10 0x7fd8e41031e0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:06.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.718+0000 7fd8eba64700 1 -- 172.21.15.32:0/89296362 >> 172.21.15.32:0/89296362 conn(0x7fd8e40f89a0 msgr2=0x7fd8e40fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:55:06.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.718+0000 7fd8eba64700 1 -- 172.21.15.32:0/89296362 shutdown_connections 2024-04-12T05:55:06.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.718+0000 7fd8eba64700 1 -- 172.21.15.32:0/89296362 wait complete. 2024-04-12T05:55:06.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.718+0000 7fd8eba64700 1 Processor -- start 2024-04-12T05:55:06.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.718+0000 7fd8eba64700 1 -- start start 2024-04-12T05:55:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.719+0000 7fd8eba64700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8e4102e10 0x7fd8e4113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.719+0000 7fd8eba64700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8e41037b0 0x7fd8e4114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.719+0000 7fd8eba64700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e4114680 con 0x7fd8e4102e10 2024-04-12T05:55:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.719+0000 7fd8eba64700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e410da70 con 0x7fd8e41037b0 2024-04-12T05:55:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.719+0000 7fd8e9800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8e4102e10 0x7fd8e4113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.719+0000 7fd8e9800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8e4102e10 0x7fd8e4113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50504/0 (socket says 172.21.15.32:50504) 2024-04-12T05:55:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.719+0000 7fd8e9800700 1 -- 172.21.15.32:0/1769099404 learned_addr learned my addr 172.21.15.32:0/1769099404 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:55:06.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.719+0000 7fd8e8fff700 1 --2- 172.21.15.32:0/1769099404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8e41037b0 0x7fd8e4114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:06.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.719+0000 7fd8e9800700 1 -- 172.21.15.32:0/1769099404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8e41037b0 msgr2=0x7fd8e4114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:06.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.719+0000 7fd8e9800700 1 --2- 172.21.15.32:0/1769099404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8e41037b0 0x7fd8e4114030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:06.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.719+0000 7fd8e9800700 1 -- 172.21.15.32:0/1769099404 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e0005040 con 0x7fd8e4102e10 2024-04-12T05:55:06.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.719+0000 7fd8e8fff700 1 --2- 172.21.15.32:0/1769099404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8e41037b0 0x7fd8e4114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:55:06.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.719+0000 7fd8e9800700 1 --2- 172.21.15.32:0/1769099404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8e4102e10 0x7fd8e4113af0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fd8e0009180 tx=0x7fd8e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:55:06.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.719+0000 7fd8d9ffb700 1 -- 172.21.15.32:0/1769099404 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8e000e040 con 0x7fd8e4102e10 2024-04-12T05:55:06.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.719+0000 7fd8eba64700 1 -- 172.21.15.32:0/1769099404 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e410dcf0 con 0x7fd8e4102e10 2024-04-12T05:55:06.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.719+0000 7fd8d9ffb700 1 -- 172.21.15.32:0/1769099404 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd8e0004290 con 0x7fd8e4102e10 2024-04-12T05:55:06.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.719+0000 7fd8d9ffb700 1 -- 172.21.15.32:0/1769099404 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8e001c3f0 con 0x7fd8e4102e10 2024-04-12T05:55:06.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.720+0000 7fd8eba64700 1 -- 172.21.15.32:0/1769099404 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8e410e2b0 con 0x7fd8e4102e10 2024-04-12T05:55:06.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.721+0000 7fd8eba64700 1 -- 172.21.15.32:0/1769099404 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e40fe560 con 0x7fd8e4102e10 2024-04-12T05:55:06.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.721+0000 7fd8d9ffb700 1 -- 172.21.15.32:0/1769099404 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd8e0013790 con 0x7fd8e4102e10 2024-04-12T05:55:06.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.721+0000 7fd8d9ffb700 1 --2- 172.21.15.32:0/1769099404 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd8d006cc20 0x7fd8d006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:06.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.722+0000 7fd8d9ffb700 1 -- 172.21.15.32:0/1769099404 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fd8e008d360 con 0x7fd8e4102e10 2024-04-12T05:55:06.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.722+0000 7fd8e8fff700 1 --2- 172.21.15.32:0/1769099404 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd8d006cc20 0x7fd8d006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:06.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.722+0000 7fd8e8fff700 1 --2- 172.21.15.32:0/1769099404 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd8d006cc20 0x7fd8d006f0d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd8e410ec30 tx=0x7fd8d4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:55:06.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.726+0000 7fd8d9ffb700 1 -- 172.21.15.32:0/1769099404 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd8e005add0 con 0x7fd8e4102e10 2024-04-12T05:55:06.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.949+0000 7fd8eba64700 1 -- 172.21.15.32:0/1769099404 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8e4109e40 con 0x7fd8d006cc20 2024-04-12T05:55:06.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.950+0000 7fd8d9ffb700 1 -- 172.21.15.32:0/1769099404 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd8e4109e40 con 0x7fd8d006cc20 2024-04-12T05:55:06.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.952+0000 7fd8eba64700 1 -- 172.21.15.32:0/1769099404 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd8d006cc20 msgr2=0x7fd8d006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:06.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.952+0000 7fd8eba64700 1 --2- 172.21.15.32:0/1769099404 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd8d006cc20 0x7fd8d006f0d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd8e410ec30 tx=0x7fd8d4009040 comp rx=0 tx=0).stop 2024-04-12T05:55:06.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.952+0000 7fd8eba64700 1 -- 172.21.15.32:0/1769099404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8e4102e10 msgr2=0x7fd8e4113af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:06.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.952+0000 7fd8eba64700 1 --2- 172.21.15.32:0/1769099404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8e4102e10 0x7fd8e4113af0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fd8e0009180 tx=0x7fd8e0009e30 comp rx=0 tx=0).stop 2024-04-12T05:55:06.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.952+0000 7fd8eba64700 1 -- 172.21.15.32:0/1769099404 shutdown_connections 2024-04-12T05:55:06.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.952+0000 7fd8eba64700 1 --2- 172.21.15.32:0/1769099404 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd8d006cc20 0x7fd8d006f0d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:06.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.952+0000 7fd8eba64700 1 --2- 172.21.15.32:0/1769099404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8e4102e10 0x7fd8e4113af0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:06.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.952+0000 7fd8eba64700 1 --2- 172.21.15.32:0/1769099404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8e41037b0 0x7fd8e4114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:06.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.952+0000 7fd8eba64700 1 -- 172.21.15.32:0/1769099404 >> 172.21.15.32:0/1769099404 conn(0x7fd8e40f89a0 msgr2=0x7fd8e40f9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:55:06.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.953+0000 7fd8eba64700 1 -- 172.21.15.32:0/1769099404 shutdown_connections 2024-04-12T05:55:06.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:06.953+0000 7fd8eba64700 1 -- 172.21.15.32:0/1769099404 wait complete. 2024-04-12T05:55:06.958 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:55:07.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.057+0000 7f3ed1ed9700 1 -- 172.21.15.32:0/3403257567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ecc102e10 msgr2=0x7f3ecc1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:07.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.057+0000 7f3ed1ed9700 1 --2- 172.21.15.32:0/3403257567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ecc102e10 0x7f3ecc1031e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc004940 tx=0x7f3ebc009e30 comp rx=0 tx=0).stop 2024-04-12T05:55:07.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.057+0000 7f3ed1ed9700 1 -- 172.21.15.32:0/3403257567 shutdown_connections 2024-04-12T05:55:07.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.057+0000 7f3ed1ed9700 1 --2- 172.21.15.32:0/3403257567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ecc1037b0 0x7f3ecc0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:07.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.057+0000 7f3ed1ed9700 1 --2- 172.21.15.32:0/3403257567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ecc102e10 0x7f3ecc1031e0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:07.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.057+0000 7f3ed1ed9700 1 -- 172.21.15.32:0/3403257567 >> 172.21.15.32:0/3403257567 conn(0x7f3ecc0f8980 msgr2=0x7f3ecc0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:55:07.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.057+0000 7f3ed1ed9700 1 -- 172.21.15.32:0/3403257567 shutdown_connections 2024-04-12T05:55:07.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.057+0000 7f3ed1ed9700 1 -- 172.21.15.32:0/3403257567 wait complete. 2024-04-12T05:55:07.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.057+0000 7f3ed1ed9700 1 Processor -- start 2024-04-12T05:55:07.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.058+0000 7f3ed1ed9700 1 -- start start 2024-04-12T05:55:07.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.058+0000 7f3ed1ed9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ecc102e10 0x7f3ecc113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:07.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.058+0000 7f3ed1ed9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ecc1037b0 0x7f3ecc113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:07.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.058+0000 7f3ed1ed9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ecc1145d0 con 0x7f3ecc1037b0 2024-04-12T05:55:07.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.058+0000 7f3ed1ed9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ecc10d9c0 con 0x7f3ecc102e10 2024-04-12T05:55:07.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.058+0000 7f3ecaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ecc1037b0 0x7f3ecc113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:07.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.058+0000 7f3ecaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ecc1037b0 0x7f3ecc113f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50526/0 (socket says 172.21.15.32:50526) 2024-04-12T05:55:07.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.058+0000 7f3ecaffd700 1 -- 172.21.15.32:0/244944868 learned_addr learned my addr 172.21.15.32:0/244944868 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:55:07.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.058+0000 7f3ecb7fe700 1 --2- 172.21.15.32:0/244944868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ecc102e10 0x7f3ecc113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:07.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.058+0000 7f3ecaffd700 1 -- 172.21.15.32:0/244944868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ecc102e10 msgr2=0x7f3ecc113a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:07.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.058+0000 7f3ecaffd700 1 --2- 172.21.15.32:0/244944868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ecc102e10 0x7f3ecc113a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:07.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.058+0000 7f3ecaffd700 1 -- 172.21.15.32:0/244944868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ebc005040 con 0x7f3ecc1037b0 2024-04-12T05:55:07.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.059+0000 7f3ecaffd700 1 --2- 172.21.15.32:0/244944868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ecc1037b0 0x7f3ecc113f80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3ec000eb40 tx=0x7f3ec000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:55:07.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.059+0000 7f3eb3fff700 1 -- 172.21.15.32:0/244944868 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ec0007cc0 con 0x7f3ecc1037b0 2024-04-12T05:55:07.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.059+0000 7f3eb3fff700 1 -- 172.21.15.32:0/244944868 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ec000f040 con 0x7f3ecc1037b0 2024-04-12T05:55:07.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.059+0000 7f3ed1ed9700 1 -- 172.21.15.32:0/244944868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ecc10dc40 con 0x7f3ecc1037b0 2024-04-12T05:55:07.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.059+0000 7f3eb3fff700 1 -- 172.21.15.32:0/244944868 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ec000c8f0 con 0x7f3ecc1037b0 2024-04-12T05:55:07.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.059+0000 7f3ed1ed9700 1 -- 172.21.15.32:0/244944868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ecc10e200 con 0x7f3ecc1037b0 2024-04-12T05:55:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.060+0000 7f3ed1ed9700 1 -- 172.21.15.32:0/244944868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ecc0fe560 con 0x7f3ecc1037b0 2024-04-12T05:55:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.060+0000 7f3eb3fff700 1 -- 172.21.15.32:0/244944868 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3ec000cb60 con 0x7f3ecc1037b0 2024-04-12T05:55:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.061+0000 7f3eb3fff700 1 --2- 172.21.15.32:0/244944868 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3eb406cc20 0x7f3eb406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.061+0000 7f3eb3fff700 1 -- 172.21.15.32:0/244944868 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f3ec008bb40 con 0x7f3ecc1037b0 2024-04-12T05:55:07.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.061+0000 7f3ecb7fe700 1 --2- 172.21.15.32:0/244944868 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3eb406cc20 0x7f3eb406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:07.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.061+0000 7f3ecb7fe700 1 --2- 172.21.15.32:0/244944868 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3eb406cc20 0x7f3eb406f0d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc00b0d0 tx=0x7f3ebc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:55:07.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.065+0000 7f3eb3fff700 1 -- 172.21.15.32:0/244944868 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3ec00595e0 con 0x7f3ecc1037b0 2024-04-12T05:55:07.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.286+0000 7f3ed1ed9700 1 -- 172.21.15.32:0/244944868 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ecc109dd0 con 0x7f3eb406cc20 2024-04-12T05:55:07.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.287+0000 7f3eb3fff700 1 -- 172.21.15.32:0/244944868 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3ecc109dd0 con 0x7f3eb406cc20 2024-04-12T05:55:07.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.289+0000 7f3ed1ed9700 1 -- 172.21.15.32:0/244944868 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3eb406cc20 msgr2=0x7f3eb406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:07.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.289+0000 7f3ed1ed9700 1 --2- 172.21.15.32:0/244944868 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3eb406cc20 0x7f3eb406f0d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc00b0d0 tx=0x7f3ebc009e30 comp rx=0 tx=0).stop 2024-04-12T05:55:07.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.289+0000 7f3ed1ed9700 1 -- 172.21.15.32:0/244944868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ecc1037b0 msgr2=0x7f3ecc113f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:07.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.289+0000 7f3ed1ed9700 1 --2- 172.21.15.32:0/244944868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ecc1037b0 0x7f3ecc113f80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3ec000eb40 tx=0x7f3ec000b040 comp rx=0 tx=0).stop 2024-04-12T05:55:07.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.289+0000 7f3ed1ed9700 1 -- 172.21.15.32:0/244944868 shutdown_connections 2024-04-12T05:55:07.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.289+0000 7f3ed1ed9700 1 --2- 172.21.15.32:0/244944868 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3eb406cc20 0x7f3eb406f0d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:07.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.289+0000 7f3ed1ed9700 1 --2- 172.21.15.32:0/244944868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ecc102e10 0x7f3ecc113a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:07.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.289+0000 7f3ed1ed9700 1 --2- 172.21.15.32:0/244944868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ecc1037b0 0x7f3ecc113f80 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:07.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.289+0000 7f3ed1ed9700 1 -- 172.21.15.32:0/244944868 >> 172.21.15.32:0/244944868 conn(0x7f3ecc0f8980 msgr2=0x7f3ecc0f99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:55:07.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.289+0000 7f3ed1ed9700 1 -- 172.21.15.32:0/244944868 shutdown_connections 2024-04-12T05:55:07.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.289+0000 7f3ed1ed9700 1 -- 172.21.15.32:0/244944868 wait complete. 2024-04-12T05:55:07.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.392+0000 7fab8d416700 1 -- 172.21.15.32:0/2752953767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab880fb3b0 msgr2=0x7fab880fb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:07.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.392+0000 7fab8d416700 1 --2- 172.21.15.32:0/2752953767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab880fb3b0 0x7fab880fb820 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fab70004940 tx=0x7fab70009e30 comp rx=0 tx=0).stop 2024-04-12T05:55:07.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.392+0000 7fab8d416700 1 -- 172.21.15.32:0/2752953767 shutdown_connections 2024-04-12T05:55:07.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.392+0000 7fab8d416700 1 --2- 172.21.15.32:0/2752953767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab880fb3b0 0x7fab880fb820 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:07.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.392+0000 7fab8d416700 1 --2- 172.21.15.32:0/2752953767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab88102f70 0x7fab88103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:07.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.392+0000 7fab8d416700 1 -- 172.21.15.32:0/2752953767 >> 172.21.15.32:0/2752953767 conn(0x7fab88072310 msgr2=0x7fab88072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:55:07.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.392+0000 7fab8d416700 1 -- 172.21.15.32:0/2752953767 shutdown_connections 2024-04-12T05:55:07.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.392+0000 7fab8d416700 1 -- 172.21.15.32:0/2752953767 wait complete. 2024-04-12T05:55:07.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.393+0000 7fab8d416700 1 Processor -- start 2024-04-12T05:55:07.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.393+0000 7fab8d416700 1 -- start start 2024-04-12T05:55:07.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.393+0000 7fab8d416700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab880fb3b0 0x7fab880fc590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:07.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.393+0000 7fab8d416700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab88102f70 0x7fab880fcad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:07.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.393+0000 7fab8d416700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab880fe4e0 con 0x7fab880fb3b0 2024-04-12T05:55:07.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.393+0000 7fab8d416700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab880fd010 con 0x7fab88102f70 2024-04-12T05:55:07.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.393+0000 7fab86ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab880fb3b0 0x7fab880fc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:07.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.393+0000 7fab86ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab880fb3b0 0x7fab880fc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50546/0 (socket says 172.21.15.32:50546) 2024-04-12T05:55:07.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.393+0000 7fab86ffd700 1 -- 172.21.15.32:0/1663886959 learned_addr learned my addr 172.21.15.32:0/1663886959 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:55:07.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.393+0000 7fab867fc700 1 --2- 172.21.15.32:0/1663886959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab88102f70 0x7fab880fcad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:07.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.394+0000 7fab86ffd700 1 -- 172.21.15.32:0/1663886959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab88102f70 msgr2=0x7fab880fcad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:07.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.394+0000 7fab86ffd700 1 --2- 172.21.15.32:0/1663886959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab88102f70 0x7fab880fcad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:07.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.394+0000 7fab86ffd700 1 -- 172.21.15.32:0/1663886959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab70005040 con 0x7fab880fb3b0 2024-04-12T05:55:07.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.394+0000 7fab86ffd700 1 --2- 172.21.15.32:0/1663886959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab880fb3b0 0x7fab880fc590 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fab7800c8c0 tx=0x7fab780044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:55:07.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.394+0000 7fab7f7fe700 1 -- 172.21.15.32:0/1663886959 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab7800a760 con 0x7fab880fb3b0 2024-04-12T05:55:07.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.394+0000 7fab7f7fe700 1 -- 172.21.15.32:0/1663886959 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab7800ad70 con 0x7fab880fb3b0 2024-04-12T05:55:07.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.394+0000 7fab8d416700 1 -- 172.21.15.32:0/1663886959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab880fd2f0 con 0x7fab880fb3b0 2024-04-12T05:55:07.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.394+0000 7fab7f7fe700 1 -- 172.21.15.32:0/1663886959 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab780113a0 con 0x7fab880fb3b0 2024-04-12T05:55:07.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.394+0000 7fab8d416700 1 -- 172.21.15.32:0/1663886959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab8811a010 con 0x7fab880fb3b0 2024-04-12T05:55:07.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.395+0000 7fab8d416700 1 -- 172.21.15.32:0/1663886959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab881053b0 con 0x7fab880fb3b0 2024-04-12T05:55:07.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.395+0000 7fab7f7fe700 1 -- 172.21.15.32:0/1663886959 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fab78012400 con 0x7fab880fb3b0 2024-04-12T05:55:07.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.396+0000 7fab7f7fe700 1 --2- 172.21.15.32:0/1663886959 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fab7406cc20 0x7fab7406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:07.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.396+0000 7fab867fc700 1 --2- 172.21.15.32:0/1663886959 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fab7406cc20 0x7fab7406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:07.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.396+0000 7fab7f7fe700 1 -- 172.21.15.32:0/1663886959 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fab7808c040 con 0x7fab880fb3b0 2024-04-12T05:55:07.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.396+0000 7fab867fc700 1 --2- 172.21.15.32:0/1663886959 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fab7406cc20 0x7fab7406f0d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fab70005010 tx=0x7fab70009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:55:07.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.401+0000 7fab7f7fe700 1 -- 172.21.15.32:0/1663886959 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fab780560c0 con 0x7fab880fb3b0 2024-04-12T05:55:07.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.612+0000 7fab8d416700 1 -- 172.21.15.32:0/1663886959 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fab880fda70 con 0x7fab7406cc20 2024-04-12T05:55:07.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.622+0000 7fab7f7fe700 1 -- 172.21.15.32:0/1663886959 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fab880fda70 con 0x7fab7406cc20 2024-04-12T05:55:07.617 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:55:07.618 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (14m) 8m ago 26m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:55:07.618 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (12m) 8m ago 26m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:55:07.618 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (12m) 7m ago 25m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:55:07.618 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (14m) 8m ago 26m 87.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:55:07.618 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (8m) 8m ago 20m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T05:55:07.618 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (8m) 8m ago 20m 13.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T05:55:07.618 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (7m) 7m ago 20m 11.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 6d4aca6f74a0 2024-04-12T05:55:07.618 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (8m) 7m ago 20m 14.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T05:55:07.618 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (16m) 8m ago 27m 507M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:55:07.618 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (15m) 7m ago 25m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:55:07.618 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (13m) 8m ago 27m 53.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:55:07.618 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (13m) 7m ago 25m 43.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:55:07.619 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (15m) 8m ago 26m 16.9M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:55:07.619 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (15m) 7m ago 25m 18.4M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:55:07.619 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (12m) 8m ago 24m 91.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T05:55:07.619 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (11m) 8m ago 24m 97.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T05:55:07.619 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (10m) 8m ago 23m 81.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T05:55:07.619 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (10m) 7m ago 23m 113M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T05:55:07.619 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (9m) 7m ago 22m 90.3M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T05:55:07.619 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (9m) 7m ago 22m 84.8M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T05:55:07.619 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (15m) 8m ago 25m 64.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:55:07.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.624+0000 7fab8d416700 1 -- 172.21.15.32:0/1663886959 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fab7406cc20 msgr2=0x7fab7406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:07.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.624+0000 7fab8d416700 1 --2- 172.21.15.32:0/1663886959 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fab7406cc20 0x7fab7406f0d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fab70005010 tx=0x7fab70009e30 comp rx=0 tx=0).stop 2024-04-12T05:55:07.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.624+0000 7fab8d416700 1 -- 172.21.15.32:0/1663886959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab880fb3b0 msgr2=0x7fab880fc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:07.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.624+0000 7fab8d416700 1 --2- 172.21.15.32:0/1663886959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab880fb3b0 0x7fab880fc590 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fab7800c8c0 tx=0x7fab780044c0 comp rx=0 tx=0).stop 2024-04-12T05:55:07.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.624+0000 7fab8d416700 1 -- 172.21.15.32:0/1663886959 shutdown_connections 2024-04-12T05:55:07.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.624+0000 7fab8d416700 1 --2- 172.21.15.32:0/1663886959 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fab7406cc20 0x7fab7406f0d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:07.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.624+0000 7fab8d416700 1 --2- 172.21.15.32:0/1663886959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab880fb3b0 0x7fab880fc590 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:07.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.624+0000 7fab8d416700 1 --2- 172.21.15.32:0/1663886959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab88102f70 0x7fab880fcad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:07.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.624+0000 7fab8d416700 1 -- 172.21.15.32:0/1663886959 >> 172.21.15.32:0/1663886959 conn(0x7fab88072310 msgr2=0x7fab880f9600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:55:07.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.624+0000 7fab8d416700 1 -- 172.21.15.32:0/1663886959 shutdown_connections 2024-04-12T05:55:07.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.624+0000 7fab8d416700 1 -- 172.21.15.32:0/1663886959 wait complete. 2024-04-12T05:55:07.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.726+0000 7f107422d700 1 -- 172.21.15.32:0/1836582033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f106c102710 msgr2=0x7f106c0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:07.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.726+0000 7f107422d700 1 --2- 172.21.15.32:0/1836582033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f106c102710 0x7f106c0fbd80 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1060004940 tx=0x7f1060009e30 comp rx=0 tx=0).stop 2024-04-12T05:55:07.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.726+0000 7f107422d700 1 -- 172.21.15.32:0/1836582033 shutdown_connections 2024-04-12T05:55:07.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.726+0000 7f107422d700 1 --2- 172.21.15.32:0/1836582033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f106c102710 0x7f106c0fbd80 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:07.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.726+0000 7f107422d700 1 --2- 172.21.15.32:0/1836582033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f106c101d70 0x7f106c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:07.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.726+0000 7f107422d700 1 -- 172.21.15.32:0/1836582033 >> 172.21.15.32:0/1836582033 conn(0x7f106c072bc0 msgr2=0x7f106c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:55:07.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.726+0000 7f107422d700 1 -- 172.21.15.32:0/1836582033 shutdown_connections 2024-04-12T05:55:07.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.726+0000 7f107422d700 1 -- 172.21.15.32:0/1836582033 wait complete. 2024-04-12T05:55:07.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.727+0000 7f107422d700 1 Processor -- start 2024-04-12T05:55:07.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.727+0000 7f107422d700 1 -- start start 2024-04-12T05:55:07.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.727+0000 7f107422d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f106c101d70 0x7f106c10e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:07.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.727+0000 7f107422d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f106c102710 0x7f106c10eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:07.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.727+0000 7f107422d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c10f530 con 0x7f106c101d70 2024-04-12T05:55:07.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.727+0000 7f107422d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c10f670 con 0x7f106c102710 2024-04-12T05:55:07.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.727+0000 7f1071fc9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f106c101d70 0x7f106c10e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:07.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.727+0000 7f1071fc9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f106c101d70 0x7f106c10e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50560/0 (socket says 172.21.15.32:50560) 2024-04-12T05:55:07.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.727+0000 7f1071fc9700 1 -- 172.21.15.32:0/1107582264 learned_addr learned my addr 172.21.15.32:0/1107582264 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:55:07.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.727+0000 7f10717c8700 1 --2- 172.21.15.32:0/1107582264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f106c102710 0x7f106c10eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:07.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.728+0000 7f1071fc9700 1 -- 172.21.15.32:0/1107582264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f106c102710 msgr2=0x7f106c10eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:07.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.728+0000 7f1071fc9700 1 --2- 172.21.15.32:0/1107582264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f106c102710 0x7f106c10eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:07.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.728+0000 7f1071fc9700 1 -- 172.21.15.32:0/1107582264 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1060005040 con 0x7f106c101d70 2024-04-12T05:55:07.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.728+0000 7f10717c8700 1 --2- 172.21.15.32:0/1107582264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f106c102710 0x7f106c10eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:55:07.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.728+0000 7f1071fc9700 1 --2- 172.21.15.32:0/1107582264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f106c101d70 0x7f106c10e9a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f106800eb40 tx=0x7f106800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:55:07.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.728+0000 7f105e7fc700 1 -- 172.21.15.32:0/1107582264 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1068007cc0 con 0x7f106c101d70 2024-04-12T05:55:07.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.728+0000 7f105e7fc700 1 -- 172.21.15.32:0/1107582264 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f106800f040 con 0x7f106c101d70 2024-04-12T05:55:07.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.728+0000 7f107422d700 1 -- 172.21.15.32:0/1107582264 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f106c117a10 con 0x7f106c101d70 2024-04-12T05:55:07.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.728+0000 7f105e7fc700 1 -- 172.21.15.32:0/1107582264 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1068009570 con 0x7f106c101d70 2024-04-12T05:55:07.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.728+0000 7f107422d700 1 -- 172.21.15.32:0/1107582264 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f106c117f20 con 0x7f106c101d70 2024-04-12T05:55:07.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.729+0000 7f107422d700 1 -- 172.21.15.32:0/1107582264 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f106c0ff080 con 0x7f106c101d70 2024-04-12T05:55:07.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.730+0000 7f105e7fc700 1 -- 172.21.15.32:0/1107582264 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f10680096d0 con 0x7f106c101d70 2024-04-12T05:55:07.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.730+0000 7f105e7fc700 1 --2- 172.21.15.32:0/1107582264 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f105806cc20 0x7f105806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:07.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.730+0000 7f105e7fc700 1 -- 172.21.15.32:0/1107582264 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f106808bfa0 con 0x7f106c101d70 2024-04-12T05:55:07.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.730+0000 7f10717c8700 1 --2- 172.21.15.32:0/1107582264 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f105806cc20 0x7f105806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:07.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.731+0000 7f10717c8700 1 --2- 172.21.15.32:0/1107582264 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f105806cc20 0x7f105806f0d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f106000b0d0 tx=0x7f1060009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:55:07.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:07.734+0000 7f105e7fc700 1 -- 172.21.15.32:0/1107582264 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1068056030 con 0x7f106c101d70 2024-04-12T05:55:08.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.021+0000 7f107422d700 1 -- 172.21.15.32:0/1107582264 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f106c04bed0 con 0x7f106c101d70 2024-04-12T05:55:08.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.021+0000 7f105e7fc700 1 -- 172.21.15.32:0/1107582264 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f1068059650 con 0x7f106c101d70 2024-04-12T05:55:08.017 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:55:08.017 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:55:08.017 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:55:08.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:55:08.017 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:55:08.017 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:55:08.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:55:08.017 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:55:08.017 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:55:08.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:55:08.018 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:55:08.018 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:55:08.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:55:08.018 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:55:08.018 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:55:08.018 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:55:08.018 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:55:08.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.023+0000 7f107422d700 1 -- 172.21.15.32:0/1107582264 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f105806cc20 msgr2=0x7f105806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:08.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.023+0000 7f107422d700 1 --2- 172.21.15.32:0/1107582264 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f105806cc20 0x7f105806f0d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f106000b0d0 tx=0x7f1060009e30 comp rx=0 tx=0).stop 2024-04-12T05:55:08.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.023+0000 7f107422d700 1 -- 172.21.15.32:0/1107582264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f106c101d70 msgr2=0x7f106c10e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:08.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.023+0000 7f107422d700 1 --2- 172.21.15.32:0/1107582264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f106c101d70 0x7f106c10e9a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f106800eb40 tx=0x7f106800b040 comp rx=0 tx=0).stop 2024-04-12T05:55:08.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.023+0000 7f107422d700 1 -- 172.21.15.32:0/1107582264 shutdown_connections 2024-04-12T05:55:08.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.023+0000 7f107422d700 1 --2- 172.21.15.32:0/1107582264 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f105806cc20 0x7f105806f0d0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:08.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.023+0000 7f107422d700 1 --2- 172.21.15.32:0/1107582264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f106c101d70 0x7f106c10e9a0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:08.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.023+0000 7f107422d700 1 --2- 172.21.15.32:0/1107582264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f106c102710 0x7f106c10eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:08.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.023+0000 7f107422d700 1 -- 172.21.15.32:0/1107582264 >> 172.21.15.32:0/1107582264 conn(0x7f106c072bc0 msgr2=0x7f106c0faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:55:08.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.024+0000 7f107422d700 1 -- 172.21.15.32:0/1107582264 shutdown_connections 2024-04-12T05:55:08.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.024+0000 7f107422d700 1 -- 172.21.15.32:0/1107582264 wait complete. 2024-04-12T05:55:08.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:07 smithi032 ceph-mon[146817]: pgmap v438: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:08.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.127+0000 7f87362fc700 1 -- 172.21.15.32:0/1170546580 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8730101d70 msgr2=0x7f8730102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:08.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.127+0000 7f87362fc700 1 --2- 172.21.15.32:0/1170546580 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8730101d70 0x7f8730102140 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f8720004940 tx=0x7f8720009e30 comp rx=0 tx=0).stop 2024-04-12T05:55:08.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.127+0000 7f87362fc700 1 -- 172.21.15.32:0/1170546580 shutdown_connections 2024-04-12T05:55:08.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.127+0000 7f87362fc700 1 --2- 172.21.15.32:0/1170546580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8730102710 0x7f87300fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:08.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.127+0000 7f87362fc700 1 --2- 172.21.15.32:0/1170546580 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8730101d70 0x7f8730102140 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:08.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.127+0000 7f87362fc700 1 -- 172.21.15.32:0/1170546580 >> 172.21.15.32:0/1170546580 conn(0x7f8730072bc0 msgr2=0x7f8730072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:55:08.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.127+0000 7f87362fc700 1 -- 172.21.15.32:0/1170546580 shutdown_connections 2024-04-12T05:55:08.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.127+0000 7f87362fc700 1 -- 172.21.15.32:0/1170546580 wait complete. 2024-04-12T05:55:08.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.128+0000 7f87362fc700 1 Processor -- start 2024-04-12T05:55:08.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.128+0000 7f87362fc700 1 -- start start 2024-04-12T05:55:08.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.128+0000 7f87362fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8730101d70 0x7f873010ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:08.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.128+0000 7f87362fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8730102710 0x7f873010ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:08.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.128+0000 7f87362fc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873010f5d0 con 0x7f8730101d70 2024-04-12T05:55:08.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.128+0000 7f87362fc700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873010f710 con 0x7f8730102710 2024-04-12T05:55:08.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.128+0000 7f872ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8730101d70 0x7f873010ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:08.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.128+0000 7f872ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8730101d70 0x7f873010ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50574/0 (socket says 172.21.15.32:50574) 2024-04-12T05:55:08.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.128+0000 7f872ffff700 1 -- 172.21.15.32:0/1937728293 learned_addr learned my addr 172.21.15.32:0/1937728293 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:55:08.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.128+0000 7f872ffff700 1 -- 172.21.15.32:0/1937728293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8730102710 msgr2=0x7f873010ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:08.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.128+0000 7f872ffff700 1 --2- 172.21.15.32:0/1937728293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8730102710 0x7f873010ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:08.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.129+0000 7f872ffff700 1 -- 172.21.15.32:0/1937728293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8720005040 con 0x7f8730101d70 2024-04-12T05:55:08.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.129+0000 7f872ffff700 1 --2- 172.21.15.32:0/1937728293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8730101d70 0x7f873010ea40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f872000b3b0 tx=0x7f8720009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:55:08.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.129+0000 7f872cff9700 1 -- 172.21.15.32:0/1937728293 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f872000e040 con 0x7f8730101d70 2024-04-12T05:55:08.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.129+0000 7f872cff9700 1 -- 172.21.15.32:0/1937728293 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8720013420 con 0x7f8730101d70 2024-04-12T05:55:08.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.129+0000 7f87362fc700 1 -- 172.21.15.32:0/1937728293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8730117ab0 con 0x7f8730101d70 2024-04-12T05:55:08.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.129+0000 7f872cff9700 1 -- 172.21.15.32:0/1937728293 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f872001c630 con 0x7f8730101d70 2024-04-12T05:55:08.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.129+0000 7f87362fc700 1 -- 172.21.15.32:0/1937728293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8730117fc0 con 0x7f8730101d70 2024-04-12T05:55:08.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.130+0000 7f87362fc700 1 -- 172.21.15.32:0/1937728293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87300ff080 con 0x7f8730101d70 2024-04-12T05:55:08.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.130+0000 7f872cff9700 1 -- 172.21.15.32:0/1937728293 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8720013970 con 0x7f8730101d70 2024-04-12T05:55:08.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.131+0000 7f872cff9700 1 --2- 172.21.15.32:0/1937728293 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f871806cc20 0x7f871806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:08.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.131+0000 7f872cff9700 1 -- 172.21.15.32:0/1937728293 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f872008d2d0 con 0x7f8730101d70 2024-04-12T05:55:08.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.131+0000 7f872f7fe700 1 --2- 172.21.15.32:0/1937728293 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f871806cc20 0x7f871806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:08.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.131+0000 7f872f7fe700 1 --2- 172.21.15.32:0/1937728293 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f871806cc20 0x7f871806f0d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8730066000 tx=0x7f8724007d40 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:55:08.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.135+0000 7f872cff9700 1 -- 172.21.15.32:0/1937728293 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f87200572b0 con 0x7f8730101d70 2024-04-12T05:55:08.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:07 smithi067 ceph-mon[112979]: pgmap v438: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:08.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.386+0000 7f87362fc700 1 -- 172.21.15.32:0/1937728293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8730108d30 con 0x7f8730101d70 2024-04-12T05:55:08.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.386+0000 7f872cff9700 1 -- 172.21.15.32:0/1937728293 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1835 (secure 0 0 0) 0x7f872005a8d0 con 0x7f8730101d70 2024-04-12T05:55:08.382 INFO:teuthology.orchestra.run.smithi032.stdout:e33 2024-04-12T05:55:08.382 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:55:08.382 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:55:08.382 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:55:08.382 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:55:08.382 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:55:08.382 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:55:08.382 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 24 2024-04-12T05:55:08.382 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:55:08.382 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:55:08.382 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:46:10.668569+0000 2024-04-12T05:55:08.383 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:55:08.383 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:55:08.383 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:55:08.383 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:55:08.383 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:55:08.383 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:55:08.383 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:55:08.383 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:55:08.383 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T05:55:08.383 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:55:08.383 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:55:08.383 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:55:08.383 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T05:55:08.384 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-12T05:55:08.384 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:55:08.384 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:55:08.384 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:55:08.384 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:55:08.384 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:55:08.384 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:55:08.384 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:55:08.384 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:55:08.384 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:55:08.384 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:55:08.384 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-12T05:55:08.384 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:55:08.384 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:55:08.384 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:55:08.385 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{-1:44317} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:55:08.385 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:55:08.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.388+0000 7f87362fc700 1 -- 172.21.15.32:0/1937728293 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f871806cc20 msgr2=0x7f871806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:08.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.388+0000 7f87362fc700 1 --2- 172.21.15.32:0/1937728293 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f871806cc20 0x7f871806f0d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8730066000 tx=0x7f8724007d40 comp rx=0 tx=0).stop 2024-04-12T05:55:08.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.388+0000 7f87362fc700 1 -- 172.21.15.32:0/1937728293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8730101d70 msgr2=0x7f873010ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:08.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.388+0000 7f87362fc700 1 --2- 172.21.15.32:0/1937728293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8730101d70 0x7f873010ea40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f872000b3b0 tx=0x7f8720009e30 comp rx=0 tx=0).stop 2024-04-12T05:55:08.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.388+0000 7f87362fc700 1 -- 172.21.15.32:0/1937728293 shutdown_connections 2024-04-12T05:55:08.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.388+0000 7f87362fc700 1 --2- 172.21.15.32:0/1937728293 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f871806cc20 0x7f871806f0d0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:08.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.388+0000 7f87362fc700 1 --2- 172.21.15.32:0/1937728293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8730101d70 0x7f873010ea40 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:08.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.388+0000 7f87362fc700 1 --2- 172.21.15.32:0/1937728293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8730102710 0x7f873010ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:08.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.388+0000 7f87362fc700 1 -- 172.21.15.32:0/1937728293 >> 172.21.15.32:0/1937728293 conn(0x7f8730072bc0 msgr2=0x7f87300fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:55:08.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.388+0000 7f87362fc700 1 -- 172.21.15.32:0/1937728293 shutdown_connections 2024-04-12T05:55:08.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.388+0000 7f87362fc700 1 -- 172.21.15.32:0/1937728293 wait complete. 2024-04-12T05:55:08.386 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-12T05:55:08.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.492+0000 7f23fe0c1700 1 -- 172.21.15.32:0/102732479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23f8102710 msgr2=0x7f23f80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:08.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.492+0000 7f23fe0c1700 1 --2- 172.21.15.32:0/102732479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23f8102710 0x7f23f80fbd80 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f23ec004940 tx=0x7f23ec009e30 comp rx=0 tx=0).stop 2024-04-12T05:55:08.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.493+0000 7f23fe0c1700 1 -- 172.21.15.32:0/102732479 shutdown_connections 2024-04-12T05:55:08.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.493+0000 7f23fe0c1700 1 --2- 172.21.15.32:0/102732479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23f8102710 0x7f23f80fbd80 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:08.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.493+0000 7f23fe0c1700 1 --2- 172.21.15.32:0/102732479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23f8101d70 0x7f23f8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:08.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.493+0000 7f23fe0c1700 1 -- 172.21.15.32:0/102732479 >> 172.21.15.32:0/102732479 conn(0x7f23f8072bc0 msgr2=0x7f23f8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:55:08.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.493+0000 7f23fe0c1700 1 -- 172.21.15.32:0/102732479 shutdown_connections 2024-04-12T05:55:08.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.493+0000 7f23fe0c1700 1 -- 172.21.15.32:0/102732479 wait complete. 2024-04-12T05:55:08.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.493+0000 7f23fe0c1700 1 Processor -- start 2024-04-12T05:55:08.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.493+0000 7f23fe0c1700 1 -- start start 2024-04-12T05:55:08.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.493+0000 7f23fe0c1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23f8101d70 0x7f23f810ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:08.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.493+0000 7f23fe0c1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23f8102710 0x7f23f810efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:08.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.493+0000 7f23fe0c1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f810f5f0 con 0x7f23f8101d70 2024-04-12T05:55:08.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.493+0000 7f23fe0c1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f810f730 con 0x7f23f8102710 2024-04-12T05:55:08.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.494+0000 7f23f77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23f8101d70 0x7f23f810ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:08.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.494+0000 7f23f77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23f8101d70 0x7f23f810ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50594/0 (socket says 172.21.15.32:50594) 2024-04-12T05:55:08.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.494+0000 7f23f6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23f8102710 0x7f23f810efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:08.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.494+0000 7f23f77fe700 1 -- 172.21.15.32:0/775365198 learned_addr learned my addr 172.21.15.32:0/775365198 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:55:08.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.494+0000 7f23f77fe700 1 -- 172.21.15.32:0/775365198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23f8102710 msgr2=0x7f23f810efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:08.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.494+0000 7f23f77fe700 1 --2- 172.21.15.32:0/775365198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23f8102710 0x7f23f810efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:08.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.494+0000 7f23f77fe700 1 -- 172.21.15.32:0/775365198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23ec005040 con 0x7f23f8101d70 2024-04-12T05:55:08.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.494+0000 7f23f77fe700 1 --2- 172.21.15.32:0/775365198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23f8101d70 0x7f23f810ea60 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f23e8002a50 tx=0x7f23e80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:55:08.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.494+0000 7f23dffff700 1 -- 172.21.15.32:0/775365198 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23e8007cc0 con 0x7f23f8101d70 2024-04-12T05:55:08.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.494+0000 7f23dffff700 1 -- 172.21.15.32:0/775365198 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f23e800f040 con 0x7f23f8101d70 2024-04-12T05:55:08.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.494+0000 7f23fe0c1700 1 -- 172.21.15.32:0/775365198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23f8117ad0 con 0x7f23f8101d70 2024-04-12T05:55:08.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.494+0000 7f23dffff700 1 -- 172.21.15.32:0/775365198 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23e800d9c0 con 0x7f23f8101d70 2024-04-12T05:55:08.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.495+0000 7f23fe0c1700 1 -- 172.21.15.32:0/775365198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23f8118000 con 0x7f23f8101d70 2024-04-12T05:55:08.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.495+0000 7f23fe0c1700 1 -- 172.21.15.32:0/775365198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23f80ff080 con 0x7f23f8101d70 2024-04-12T05:55:08.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.496+0000 7f23dffff700 1 -- 172.21.15.32:0/775365198 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f23e800db40 con 0x7f23f8101d70 2024-04-12T05:55:08.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.496+0000 7f23dffff700 1 --2- 172.21.15.32:0/775365198 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f23e006cc20 0x7f23e006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:08.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.496+0000 7f23dffff700 1 -- 172.21.15.32:0/775365198 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f23e808bd00 con 0x7f23f8101d70 2024-04-12T05:55:08.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.497+0000 7f23f6ffd700 1 --2- 172.21.15.32:0/775365198 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f23e006cc20 0x7f23e006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:08.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.497+0000 7f23f6ffd700 1 --2- 172.21.15.32:0/775365198 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f23e006cc20 0x7f23e006f0d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f23ec004890 tx=0x7f23ec009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:55:08.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.501+0000 7f23dffff700 1 -- 172.21.15.32:0/775365198 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f23e808f030 con 0x7f23f8101d70 2024-04-12T05:55:08.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.723+0000 7f23fe0c1700 1 -- 172.21.15.32:0/775365198 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f23f8108d10 con 0x7f23e006cc20 2024-04-12T05:55:08.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.724+0000 7f23dffff700 1 -- 172.21.15.32:0/775365198 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f23f8108d10 con 0x7f23e006cc20 2024-04-12T05:55:08.720 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:55:08.720 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:55:08.720 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:55:08.720 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:55:08.720 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:55:08.720 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-12T05:55:08.720 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:55:08.720 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-12T05:55:08.720 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:55:08.720 INFO:teuthology.orchestra.run.smithi032.stdout: "mds" 2024-04-12T05:55:08.720 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:55:08.720 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:55:08.720 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:55:08.721 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:55:08.721 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:55:08.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.726+0000 7f23fe0c1700 1 -- 172.21.15.32:0/775365198 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f23e006cc20 msgr2=0x7f23e006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:08.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.726+0000 7f23fe0c1700 1 --2- 172.21.15.32:0/775365198 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f23e006cc20 0x7f23e006f0d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f23ec004890 tx=0x7f23ec009e30 comp rx=0 tx=0).stop 2024-04-12T05:55:08.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.726+0000 7f23fe0c1700 1 -- 172.21.15.32:0/775365198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23f8101d70 msgr2=0x7f23f810ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:08.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.726+0000 7f23fe0c1700 1 --2- 172.21.15.32:0/775365198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23f8101d70 0x7f23f810ea60 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f23e8002a50 tx=0x7f23e80046b0 comp rx=0 tx=0).stop 2024-04-12T05:55:08.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.726+0000 7f23fe0c1700 1 -- 172.21.15.32:0/775365198 shutdown_connections 2024-04-12T05:55:08.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.726+0000 7f23fe0c1700 1 --2- 172.21.15.32:0/775365198 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f23e006cc20 0x7f23e006f0d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:08.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.726+0000 7f23fe0c1700 1 --2- 172.21.15.32:0/775365198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23f8101d70 0x7f23f810ea60 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:08.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.726+0000 7f23fe0c1700 1 --2- 172.21.15.32:0/775365198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23f8102710 0x7f23f810efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:08.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.726+0000 7f23fe0c1700 1 -- 172.21.15.32:0/775365198 >> 172.21.15.32:0/775365198 conn(0x7f23f8072bc0 msgr2=0x7f23f80fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:55:08.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.726+0000 7f23fe0c1700 1 -- 172.21.15.32:0/775365198 shutdown_connections 2024-04-12T05:55:08.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.726+0000 7f23fe0c1700 1 -- 172.21.15.32:0/775365198 wait complete. 2024-04-12T05:55:08.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.831+0000 7f3f169ce700 1 -- 172.21.15.32:0/1424039963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f10101d70 msgr2=0x7f3f10102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:08.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.831+0000 7f3f169ce700 1 --2- 172.21.15.32:0/1424039963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f10101d70 0x7f3f10102140 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8004940 tx=0x7f3ef8009e30 comp rx=0 tx=0).stop 2024-04-12T05:55:08.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.831+0000 7f3f169ce700 1 -- 172.21.15.32:0/1424039963 shutdown_connections 2024-04-12T05:55:08.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.831+0000 7f3f169ce700 1 --2- 172.21.15.32:0/1424039963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f10102710 0x7f3f100fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:08.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.831+0000 7f3f169ce700 1 --2- 172.21.15.32:0/1424039963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f10101d70 0x7f3f10102140 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:08.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.831+0000 7f3f169ce700 1 -- 172.21.15.32:0/1424039963 >> 172.21.15.32:0/1424039963 conn(0x7f3f10072bc0 msgr2=0x7f3f10072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:55:08.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.831+0000 7f3f169ce700 1 -- 172.21.15.32:0/1424039963 shutdown_connections 2024-04-12T05:55:08.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.831+0000 7f3f169ce700 1 -- 172.21.15.32:0/1424039963 wait complete. 2024-04-12T05:55:08.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.831+0000 7f3f169ce700 1 Processor -- start 2024-04-12T05:55:08.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.832+0000 7f3f169ce700 1 -- start start 2024-04-12T05:55:08.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.832+0000 7f3f169ce700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f10101d70 0x7f3f1010ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:08.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.832+0000 7f3f169ce700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f10102710 0x7f3f1010f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:08.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.832+0000 7f3f169ce700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1010f660 con 0x7f3f10102710 2024-04-12T05:55:08.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.832+0000 7f3f169ce700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1010f7a0 con 0x7f3f10101d70 2024-04-12T05:55:08.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.832+0000 7f3f0f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f10102710 0x7f3f1010f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:08.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.832+0000 7f3f0f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f10102710 0x7f3f1010f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50614/0 (socket says 172.21.15.32:50614) 2024-04-12T05:55:08.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.832+0000 7f3f0f7fe700 1 -- 172.21.15.32:0/3184167828 learned_addr learned my addr 172.21.15.32:0/3184167828 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:55:08.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.832+0000 7f3f0ffff700 1 --2- 172.21.15.32:0/3184167828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f10101d70 0x7f3f1010ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:08.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.832+0000 7f3f0f7fe700 1 -- 172.21.15.32:0/3184167828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f10101d70 msgr2=0x7f3f1010ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:08.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.832+0000 7f3f0f7fe700 1 --2- 172.21.15.32:0/3184167828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f10101d70 0x7f3f1010ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:08.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.832+0000 7f3f0f7fe700 1 -- 172.21.15.32:0/3184167828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ef8005040 con 0x7f3f10102710 2024-04-12T05:55:08.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.833+0000 7f3f0f7fe700 1 --2- 172.21.15.32:0/3184167828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f10102710 0x7f3f1010f120 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f3f0000c800 tx=0x7f3f00004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:55:08.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.833+0000 7f3f0cff9700 1 -- 172.21.15.32:0/3184167828 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f000106f0 con 0x7f3f10102710 2024-04-12T05:55:08.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.833+0000 7f3f0cff9700 1 -- 172.21.15.32:0/3184167828 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f00010d00 con 0x7f3f10102710 2024-04-12T05:55:08.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.833+0000 7f3f0cff9700 1 -- 172.21.15.32:0/3184167828 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f000113a0 con 0x7f3f10102710 2024-04-12T05:55:08.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.833+0000 7f3f169ce700 1 -- 172.21.15.32:0/3184167828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f10117a90 con 0x7f3f10102710 2024-04-12T05:55:08.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.833+0000 7f3f169ce700 1 -- 172.21.15.32:0/3184167828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f10118020 con 0x7f3f10102710 2024-04-12T05:55:08.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.834+0000 7f3f169ce700 1 -- 172.21.15.32:0/3184167828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f100ff080 con 0x7f3f10102710 2024-04-12T05:55:08.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.834+0000 7f3f0cff9700 1 -- 172.21.15.32:0/3184167828 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3f00012400 con 0x7f3f10102710 2024-04-12T05:55:08.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.835+0000 7f3f0cff9700 1 --2- 172.21.15.32:0/3184167828 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3efc06cc20 0x7f3efc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:08.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.835+0000 7f3f0cff9700 1 -- 172.21.15.32:0/3184167828 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f3f0008c090 con 0x7f3f10102710 2024-04-12T05:55:08.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.835+0000 7f3f0ffff700 1 --2- 172.21.15.32:0/3184167828 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3efc06cc20 0x7f3efc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:08.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.835+0000 7f3f0ffff700 1 --2- 172.21.15.32:0/3184167828 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3efc06cc20 0x7f3efc06f0d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8004f90 tx=0x7f3ef8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:55:08.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:08.839+0000 7f3f0cff9700 1 -- 172.21.15.32:0/3184167828 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3f00056050 con 0x7f3f10102710 2024-04-12T05:55:09.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:08 smithi032 ceph-mon[146817]: from='client.34844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:55:09.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:08 smithi032 ceph-mon[146817]: from='client.34848 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:55:09.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:08 smithi032 ceph-mon[146817]: from='client.34852 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:55:09.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:08 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1107582264' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:55:09.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:08 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1937728293' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:55:09.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:09.128+0000 7f3f169ce700 1 -- 172.21.15.32:0/3184167828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f3f10065e00 con 0x7f3f10102710 2024-04-12T05:55:09.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:09.128+0000 7f3f0cff9700 1 -- 172.21.15.32:0/3184167828 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f3f00059670 con 0x7f3f10102710 2024-04-12T05:55:09.124 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:55:09.124 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:55:09.124 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-12T05:55:09.124 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:55:09.124 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:55:09.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:09.130+0000 7f3f169ce700 1 -- 172.21.15.32:0/3184167828 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3efc06cc20 msgr2=0x7f3efc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:09.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:09.130+0000 7f3f169ce700 1 --2- 172.21.15.32:0/3184167828 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3efc06cc20 0x7f3efc06f0d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8004f90 tx=0x7f3ef8009e30 comp rx=0 tx=0).stop 2024-04-12T05:55:09.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:09.130+0000 7f3f169ce700 1 -- 172.21.15.32:0/3184167828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f10102710 msgr2=0x7f3f1010f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:09.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:09.130+0000 7f3f169ce700 1 --2- 172.21.15.32:0/3184167828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f10102710 0x7f3f1010f120 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f3f0000c800 tx=0x7f3f00004470 comp rx=0 tx=0).stop 2024-04-12T05:55:09.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:09.130+0000 7f3f169ce700 1 -- 172.21.15.32:0/3184167828 shutdown_connections 2024-04-12T05:55:09.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:09.130+0000 7f3f169ce700 1 --2- 172.21.15.32:0/3184167828 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3efc06cc20 0x7f3efc06f0d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:09.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:09.130+0000 7f3f169ce700 1 --2- 172.21.15.32:0/3184167828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f10101d70 0x7f3f1010ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:09.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:09.130+0000 7f3f169ce700 1 --2- 172.21.15.32:0/3184167828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f10102710 0x7f3f1010f120 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:09.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:09.130+0000 7f3f169ce700 1 -- 172.21.15.32:0/3184167828 >> 172.21.15.32:0/3184167828 conn(0x7f3f10072bc0 msgr2=0x7f3f100fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:55:09.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:09.130+0000 7f3f169ce700 1 -- 172.21.15.32:0/3184167828 shutdown_connections 2024-04-12T05:55:09.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:09.130+0000 7f3f169ce700 1 -- 172.21.15.32:0/3184167828 wait complete. 2024-04-12T05:55:09.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:08 smithi067 ceph-mon[112979]: from='client.34844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:55:09.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:08 smithi067 ceph-mon[112979]: from='client.34848 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:55:09.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:08 smithi067 ceph-mon[112979]: from='client.34852 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:55:09.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:08 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1107582264' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:55:09.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:08 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1937728293' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:55:10.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:09 smithi032 ceph-mon[146817]: pgmap v439: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:10.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:09 smithi032 ceph-mon[146817]: from='client.34864 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:55:10.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:09 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3184167828' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:55:10.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:09 smithi067 ceph-mon[112979]: pgmap v439: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:10.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:09 smithi067 ceph-mon[112979]: from='client.34864 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:55:10.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:09 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3184167828' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:55:12.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:11 smithi032 ceph-mon[146817]: pgmap v440: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:12.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:11 smithi067 ceph-mon[112979]: pgmap v440: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:14.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:13 smithi032 ceph-mon[146817]: pgmap v441: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:14.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:13 smithi067 ceph-mon[112979]: pgmap v441: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:16.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:15 smithi032 ceph-mon[146817]: pgmap v442: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:16.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:15 smithi067 ceph-mon[112979]: pgmap v442: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:18.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:17 smithi032 ceph-mon[146817]: pgmap v443: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:18.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:17 smithi067 ceph-mon[112979]: pgmap v443: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:19.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:55:19.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:55:20.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:19 smithi032 ceph-mon[146817]: pgmap v444: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:20.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:19 smithi067 ceph-mon[112979]: pgmap v444: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:22.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:21 smithi032 ceph-mon[146817]: pgmap v445: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:22.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:21 smithi067 ceph-mon[112979]: pgmap v445: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:24.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:23 smithi032 ceph-mon[146817]: pgmap v446: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:24.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:23 smithi067 ceph-mon[112979]: pgmap v446: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:26.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:25 smithi067 ceph-mon[112979]: pgmap v447: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:26.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:25 smithi032 ceph-mon[146817]: pgmap v447: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:27.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:26 smithi067 ceph-mon[112979]: pgmap v448: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:27.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:26 smithi032 ceph-mon[146817]: pgmap v448: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:29.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:29 smithi032 ceph-mon[146817]: pgmap v449: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:29.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:29 smithi067 ceph-mon[112979]: pgmap v449: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:31.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:31 smithi032 ceph-mon[146817]: pgmap v450: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:31.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:31 smithi067 ceph-mon[112979]: pgmap v450: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:33.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:33 smithi032 ceph-mon[146817]: pgmap v451: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:33.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:33 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:55:33.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:33 smithi067 ceph-mon[112979]: pgmap v451: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:33.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:33 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:55:35.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:35 smithi032 ceph-mon[146817]: pgmap v452: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:35.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:35 smithi067 ceph-mon[112979]: pgmap v452: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:37.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:37 smithi032 ceph-mon[146817]: pgmap v453: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:37.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:37 smithi067 ceph-mon[112979]: pgmap v453: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:39.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.249+0000 7fbf8fd3c700 1 -- 172.21.15.32:0/1023976581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf88102e10 msgr2=0x7fbf881031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:39.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.249+0000 7fbf8fd3c700 1 --2- 172.21.15.32:0/1023976581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf88102e10 0x7fbf881031e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fbf84004940 tx=0x7fbf84009e30 comp rx=0 tx=0).stop 2024-04-12T05:55:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.249+0000 7fbf8fd3c700 1 -- 172.21.15.32:0/1023976581 shutdown_connections 2024-04-12T05:55:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.249+0000 7fbf8fd3c700 1 --2- 172.21.15.32:0/1023976581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf881037b0 0x7fbf880fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.249+0000 7fbf8fd3c700 1 --2- 172.21.15.32:0/1023976581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf88102e10 0x7fbf881031e0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.249+0000 7fbf8fd3c700 1 -- 172.21.15.32:0/1023976581 >> 172.21.15.32:0/1023976581 conn(0x7fbf880f89a0 msgr2=0x7fbf880fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:55:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.249+0000 7fbf8fd3c700 1 -- 172.21.15.32:0/1023976581 shutdown_connections 2024-04-12T05:55:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.249+0000 7fbf8fd3c700 1 -- 172.21.15.32:0/1023976581 wait complete. 2024-04-12T05:55:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.249+0000 7fbf8fd3c700 1 Processor -- start 2024-04-12T05:55:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.250+0000 7fbf8fd3c700 1 -- start start 2024-04-12T05:55:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.250+0000 7fbf8fd3c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf88102e10 0x7fbf880702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.250+0000 7fbf8fd3c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf881037b0 0x7fbf88070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.250+0000 7fbf8fd3c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf88070d90 con 0x7fbf88102e10 2024-04-12T05:55:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.250+0000 7fbf8fd3c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf88070ed0 con 0x7fbf881037b0 2024-04-12T05:55:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.250+0000 7fbf8dad8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf88102e10 0x7fbf880702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:39.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.250+0000 7fbf8dad8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf88102e10 0x7fbf880702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37886/0 (socket says 172.21.15.32:37886) 2024-04-12T05:55:39.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.250+0000 7fbf8dad8700 1 -- 172.21.15.32:0/1506161479 learned_addr learned my addr 172.21.15.32:0/1506161479 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:55:39.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.250+0000 7fbf8d2d7700 1 --2- 172.21.15.32:0/1506161479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf881037b0 0x7fbf88070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:39.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.250+0000 7fbf8dad8700 1 -- 172.21.15.32:0/1506161479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf881037b0 msgr2=0x7fbf88070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:39.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.250+0000 7fbf8dad8700 1 --2- 172.21.15.32:0/1506161479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf881037b0 0x7fbf88070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:39.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.250+0000 7fbf8dad8700 1 -- 172.21.15.32:0/1506161479 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf84005040 con 0x7fbf88102e10 2024-04-12T05:55:39.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.250+0000 7fbf8dad8700 1 --2- 172.21.15.32:0/1506161479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf88102e10 0x7fbf880702c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fbf84009180 tx=0x7fbf84009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:55:39.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.251+0000 7fbf7e7fc700 1 -- 172.21.15.32:0/1506161479 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf8400e040 con 0x7fbf88102e10 2024-04-12T05:55:39.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.251+0000 7fbf7e7fc700 1 -- 172.21.15.32:0/1506161479 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbf84004290 con 0x7fbf88102e10 2024-04-12T05:55:39.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.251+0000 7fbf8fd3c700 1 -- 172.21.15.32:0/1506161479 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf88071120 con 0x7fbf88102e10 2024-04-12T05:55:39.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.251+0000 7fbf7e7fc700 1 -- 172.21.15.32:0/1506161479 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf84013e60 con 0x7fbf88102e10 2024-04-12T05:55:39.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.251+0000 7fbf8fd3c700 1 -- 172.21.15.32:0/1506161479 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf8811a220 con 0x7fbf88102e10 2024-04-12T05:55:39.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.252+0000 7fbf8fd3c700 1 -- 172.21.15.32:0/1506161479 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf880fe560 con 0x7fbf88102e10 2024-04-12T05:55:39.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.252+0000 7fbf7e7fc700 1 -- 172.21.15.32:0/1506161479 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fbf840136b0 con 0x7fbf88102e10 2024-04-12T05:55:39.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.253+0000 7fbf7e7fc700 1 --2- 172.21.15.32:0/1506161479 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbf7406cbb0 0x7fbf7406f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:39.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.253+0000 7fbf7e7fc700 1 -- 172.21.15.32:0/1506161479 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fbf8408d300 con 0x7fbf88102e10 2024-04-12T05:55:39.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.253+0000 7fbf8d2d7700 1 --2- 172.21.15.32:0/1506161479 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbf7406cbb0 0x7fbf7406f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:39.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.253+0000 7fbf8d2d7700 1 --2- 172.21.15.32:0/1506161479 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbf7406cbb0 0x7fbf7406f060 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbf88071c00 tx=0x7fbf7800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:55:39.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.257+0000 7fbf7e7fc700 1 -- 172.21.15.32:0/1506161479 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbf840572e0 con 0x7fbf88102e10 2024-04-12T05:55:39.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:39 smithi032 ceph-mon[146817]: pgmap v454: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:39.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.480+0000 7fbf8fd3c700 1 -- 172.21.15.32:0/1506161479 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf880719b0 con 0x7fbf7406cbb0 2024-04-12T05:55:39.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.481+0000 7fbf7e7fc700 1 -- 172.21.15.32:0/1506161479 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fbf880719b0 con 0x7fbf7406cbb0 2024-04-12T05:55:39.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.483+0000 7fbf8fd3c700 1 -- 172.21.15.32:0/1506161479 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbf7406cbb0 msgr2=0x7fbf7406f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:39.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.483+0000 7fbf8fd3c700 1 --2- 172.21.15.32:0/1506161479 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbf7406cbb0 0x7fbf7406f060 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbf88071c00 tx=0x7fbf7800b040 comp rx=0 tx=0).stop 2024-04-12T05:55:39.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.483+0000 7fbf8fd3c700 1 -- 172.21.15.32:0/1506161479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf88102e10 msgr2=0x7fbf880702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:39.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.483+0000 7fbf8fd3c700 1 --2- 172.21.15.32:0/1506161479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf88102e10 0x7fbf880702c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fbf84009180 tx=0x7fbf84009e30 comp rx=0 tx=0).stop 2024-04-12T05:55:39.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.483+0000 7fbf8fd3c700 1 -- 172.21.15.32:0/1506161479 shutdown_connections 2024-04-12T05:55:39.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.483+0000 7fbf8fd3c700 1 --2- 172.21.15.32:0/1506161479 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fbf7406cbb0 0x7fbf7406f060 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:39.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.483+0000 7fbf8fd3c700 1 --2- 172.21.15.32:0/1506161479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf88102e10 0x7fbf880702c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:39.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.483+0000 7fbf8fd3c700 1 --2- 172.21.15.32:0/1506161479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf881037b0 0x7fbf88070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:39.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.483+0000 7fbf8fd3c700 1 -- 172.21.15.32:0/1506161479 >> 172.21.15.32:0/1506161479 conn(0x7fbf880f89a0 msgr2=0x7fbf880f9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:55:39.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.483+0000 7fbf8fd3c700 1 -- 172.21.15.32:0/1506161479 shutdown_connections 2024-04-12T05:55:39.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.483+0000 7fbf8fd3c700 1 -- 172.21.15.32:0/1506161479 wait complete. 2024-04-12T05:55:39.489 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:55:39.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:39 smithi067 ceph-mon[112979]: pgmap v454: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:39.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.588+0000 7f495c803700 1 -- 172.21.15.32:0/523875844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49580fb090 msgr2=0x7f49580fb500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:39.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.588+0000 7f495c803700 1 --2- 172.21.15.32:0/523875844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49580fb090 0x7f49580fb500 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f4948004940 tx=0x7f4948009e30 comp rx=0 tx=0).stop 2024-04-12T05:55:39.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.588+0000 7f495c803700 1 -- 172.21.15.32:0/523875844 shutdown_connections 2024-04-12T05:55:39.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.588+0000 7f495c803700 1 --2- 172.21.15.32:0/523875844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49580fb090 0x7f49580fb500 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:39.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.588+0000 7f495c803700 1 --2- 172.21.15.32:0/523875844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4958102c90 0x7f4958103060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:39.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.588+0000 7f495c803700 1 -- 172.21.15.32:0/523875844 >> 172.21.15.32:0/523875844 conn(0x7f4958075090 msgr2=0x7f49580f92d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:55:39.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.588+0000 7f495c803700 1 -- 172.21.15.32:0/523875844 shutdown_connections 2024-04-12T05:55:39.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.588+0000 7f495c803700 1 -- 172.21.15.32:0/523875844 wait complete. 2024-04-12T05:55:39.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.589+0000 7f495c803700 1 Processor -- start 2024-04-12T05:55:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.589+0000 7f495c803700 1 -- start start 2024-04-12T05:55:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.589+0000 7f495c803700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49580fb090 0x7f4958113900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.589+0000 7f495c803700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4958102c90 0x7f4958113e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.589+0000 7f495c803700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4958114490 con 0x7f4958102c90 2024-04-12T05:55:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.589+0000 7f495c803700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495810d880 con 0x7f49580fb090 2024-04-12T05:55:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.589+0000 7f495659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4958102c90 0x7f4958113e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.589+0000 7f495659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4958102c90 0x7f4958113e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37912/0 (socket says 172.21.15.32:37912) 2024-04-12T05:55:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.589+0000 7f495659c700 1 -- 172.21.15.32:0/2304133522 learned_addr learned my addr 172.21.15.32:0/2304133522 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:55:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.590+0000 7f4956d9d700 1 --2- 172.21.15.32:0/2304133522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49580fb090 0x7f4958113900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.590+0000 7f495659c700 1 -- 172.21.15.32:0/2304133522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49580fb090 msgr2=0x7f4958113900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.590+0000 7f495659c700 1 --2- 172.21.15.32:0/2304133522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49580fb090 0x7f4958113900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.590+0000 7f495659c700 1 -- 172.21.15.32:0/2304133522 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4948005040 con 0x7f4958102c90 2024-04-12T05:55:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.590+0000 7f495659c700 1 --2- 172.21.15.32:0/2304133522 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4958102c90 0x7f4958113e40 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f494800b3b0 tx=0x7f4948009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:55:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.590+0000 7f494f7fe700 1 -- 172.21.15.32:0/2304133522 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f494800e040 con 0x7f4958102c90 2024-04-12T05:55:39.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.590+0000 7f494f7fe700 1 -- 172.21.15.32:0/2304133522 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4948013420 con 0x7f4958102c90 2024-04-12T05:55:39.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.590+0000 7f495c803700 1 -- 172.21.15.32:0/2304133522 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f495810db00 con 0x7f4958102c90 2024-04-12T05:55:39.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.590+0000 7f494f7fe700 1 -- 172.21.15.32:0/2304133522 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f494801c630 con 0x7f4958102c90 2024-04-12T05:55:39.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.590+0000 7f495c803700 1 -- 172.21.15.32:0/2304133522 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f495810e040 con 0x7f4958102c90 2024-04-12T05:55:39.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.591+0000 7f495c803700 1 -- 172.21.15.32:0/2304133522 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49580fff70 con 0x7f4958102c90 2024-04-12T05:55:39.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.592+0000 7f494f7fe700 1 -- 172.21.15.32:0/2304133522 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4948013970 con 0x7f4958102c90 2024-04-12T05:55:39.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.592+0000 7f494f7fe700 1 --2- 172.21.15.32:0/2304133522 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f494406cc20 0x7f494406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:39.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.592+0000 7f494f7fe700 1 -- 172.21.15.32:0/2304133522 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f494808d310 con 0x7f4958102c90 2024-04-12T05:55:39.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.592+0000 7f4956d9d700 1 --2- 172.21.15.32:0/2304133522 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f494406cc20 0x7f494406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:39.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.593+0000 7f4956d9d700 1 --2- 172.21.15.32:0/2304133522 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f494406cc20 0x7f494406f0d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4940004d60 tx=0x7f494000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:55:39.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.597+0000 7f494f7fe700 1 -- 172.21.15.32:0/2304133522 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f494805ad80 con 0x7f4958102c90 2024-04-12T05:55:39.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.822+0000 7f495c803700 1 -- 172.21.15.32:0/2304133522 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f49581135d0 con 0x7f494406cc20 2024-04-12T05:55:39.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.823+0000 7f494f7fe700 1 -- 172.21.15.32:0/2304133522 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f49581135d0 con 0x7f494406cc20 2024-04-12T05:55:39.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.825+0000 7f495c803700 1 -- 172.21.15.32:0/2304133522 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f494406cc20 msgr2=0x7f494406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:39.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.825+0000 7f495c803700 1 --2- 172.21.15.32:0/2304133522 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f494406cc20 0x7f494406f0d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4940004d60 tx=0x7f494000b040 comp rx=0 tx=0).stop 2024-04-12T05:55:39.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.825+0000 7f495c803700 1 -- 172.21.15.32:0/2304133522 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4958102c90 msgr2=0x7f4958113e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:39.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.825+0000 7f495c803700 1 --2- 172.21.15.32:0/2304133522 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4958102c90 0x7f4958113e40 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f494800b3b0 tx=0x7f4948009e30 comp rx=0 tx=0).stop 2024-04-12T05:55:39.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.825+0000 7f495c803700 1 -- 172.21.15.32:0/2304133522 shutdown_connections 2024-04-12T05:55:39.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.825+0000 7f495c803700 1 --2- 172.21.15.32:0/2304133522 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f494406cc20 0x7f494406f0d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:39.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.825+0000 7f495c803700 1 --2- 172.21.15.32:0/2304133522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49580fb090 0x7f4958113900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:39.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.825+0000 7f495c803700 1 --2- 172.21.15.32:0/2304133522 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4958102c90 0x7f4958113e40 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:39.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.825+0000 7f495c803700 1 -- 172.21.15.32:0/2304133522 >> 172.21.15.32:0/2304133522 conn(0x7f4958075090 msgr2=0x7f49580fa1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:55:39.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.825+0000 7f495c803700 1 -- 172.21.15.32:0/2304133522 shutdown_connections 2024-04-12T05:55:39.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.825+0000 7f495c803700 1 -- 172.21.15.32:0/2304133522 wait complete. 2024-04-12T05:55:39.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.929+0000 7fb148bb9700 1 -- 172.21.15.32:0/3438129533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb144102e10 msgr2=0x7fb1441031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:39.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.929+0000 7fb148bb9700 1 --2- 172.21.15.32:0/3438129533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb144102e10 0x7fb1441031e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fb134004940 tx=0x7fb134009e30 comp rx=0 tx=0).stop 2024-04-12T05:55:39.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.929+0000 7fb148bb9700 1 -- 172.21.15.32:0/3438129533 shutdown_connections 2024-04-12T05:55:39.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.929+0000 7fb148bb9700 1 --2- 172.21.15.32:0/3438129533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1441037b0 0x7fb1440fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:39.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.929+0000 7fb148bb9700 1 --2- 172.21.15.32:0/3438129533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb144102e10 0x7fb1441031e0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:39.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.929+0000 7fb148bb9700 1 -- 172.21.15.32:0/3438129533 >> 172.21.15.32:0/3438129533 conn(0x7fb1440f8980 msgr2=0x7fb1440fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:55:39.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.929+0000 7fb148bb9700 1 -- 172.21.15.32:0/3438129533 shutdown_connections 2024-04-12T05:55:39.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.929+0000 7fb148bb9700 1 -- 172.21.15.32:0/3438129533 wait complete. 2024-04-12T05:55:39.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.930+0000 7fb148bb9700 1 Processor -- start 2024-04-12T05:55:39.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.930+0000 7fb148bb9700 1 -- start start 2024-04-12T05:55:39.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.930+0000 7fb148bb9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb144102e10 0x7fb1440702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:39.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.930+0000 7fb148bb9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1441037b0 0x7fb144070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:39.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.930+0000 7fb148bb9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb144070d90 con 0x7fb144102e10 2024-04-12T05:55:39.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.930+0000 7fb148bb9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb144070ed0 con 0x7fb1441037b0 2024-04-12T05:55:39.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.930+0000 7fb14259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb144102e10 0x7fb1440702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:39.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.930+0000 7fb14259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb144102e10 0x7fb1440702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37934/0 (socket says 172.21.15.32:37934) 2024-04-12T05:55:39.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.930+0000 7fb14259c700 1 -- 172.21.15.32:0/3190906331 learned_addr learned my addr 172.21.15.32:0/3190906331 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:55:39.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.931+0000 7fb14259c700 1 -- 172.21.15.32:0/3190906331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1441037b0 msgr2=0x7fb144070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:39.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.931+0000 7fb141d9b700 1 --2- 172.21.15.32:0/3190906331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1441037b0 0x7fb144070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:39.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.931+0000 7fb14259c700 1 --2- 172.21.15.32:0/3190906331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1441037b0 0x7fb144070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:39.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.931+0000 7fb14259c700 1 -- 172.21.15.32:0/3190906331 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb134005040 con 0x7fb144102e10 2024-04-12T05:55:39.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.931+0000 7fb14259c700 1 --2- 172.21.15.32:0/3190906331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb144102e10 0x7fb1440702c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb13400b3b0 tx=0x7fb134009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:55:39.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.931+0000 7fb13affd700 1 -- 172.21.15.32:0/3190906331 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb13400e040 con 0x7fb144102e10 2024-04-12T05:55:39.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.931+0000 7fb13affd700 1 -- 172.21.15.32:0/3190906331 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb134013420 con 0x7fb144102e10 2024-04-12T05:55:39.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.931+0000 7fb148bb9700 1 -- 172.21.15.32:0/3190906331 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb144071120 con 0x7fb144102e10 2024-04-12T05:55:39.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.931+0000 7fb13affd700 1 -- 172.21.15.32:0/3190906331 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb13401c630 con 0x7fb144102e10 2024-04-12T05:55:39.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.931+0000 7fb148bb9700 1 -- 172.21.15.32:0/3190906331 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb14411a240 con 0x7fb144102e10 2024-04-12T05:55:39.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.932+0000 7fb148bb9700 1 -- 172.21.15.32:0/3190906331 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1440fe560 con 0x7fb144102e10 2024-04-12T05:55:39.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.933+0000 7fb13affd700 1 -- 172.21.15.32:0/3190906331 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb134013970 con 0x7fb144102e10 2024-04-12T05:55:39.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.933+0000 7fb13affd700 1 --2- 172.21.15.32:0/3190906331 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb13006cc20 0x7fb13006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:39.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.933+0000 7fb13affd700 1 -- 172.21.15.32:0/3190906331 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fb13408d230 con 0x7fb144102e10 2024-04-12T05:55:39.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.933+0000 7fb141d9b700 1 --2- 172.21.15.32:0/3190906331 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb13006cc20 0x7fb13006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:39.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.934+0000 7fb141d9b700 1 --2- 172.21.15.32:0/3190906331 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb13006cc20 0x7fb13006f0d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb144071c00 tx=0x7fb12c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:55:39.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:39.938+0000 7fb13affd700 1 -- 172.21.15.32:0/3190906331 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb1340572b0 con 0x7fb144102e10 2024-04-12T05:55:40.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.150+0000 7fb148bb9700 1 -- 172.21.15.32:0/3190906331 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1440719b0 con 0x7fb13006cc20 2024-04-12T05:55:40.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.160+0000 7fb13affd700 1 -- 172.21.15.32:0/3190906331 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fb1440719b0 con 0x7fb13006cc20 2024-04-12T05:55:40.156 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:55:40.156 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (15m) 8m ago 26m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:55:40.156 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (12m) 8m ago 27m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:55:40.156 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (12m) 8m ago 26m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:55:40.156 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (15m) 8m ago 26m 87.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:55:40.156 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (9m) 8m ago 21m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T05:55:40.156 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (9m) 8m ago 21m 13.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T05:55:40.156 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (8m) 8m ago 21m 11.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 6d4aca6f74a0 2024-04-12T05:55:40.156 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (8m) 8m ago 21m 14.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T05:55:40.156 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (17m) 8m ago 28m 507M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:55:40.157 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (16m) 8m ago 25m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:55:40.157 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (14m) 8m ago 28m 53.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:55:40.157 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (13m) 8m ago 25m 43.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:55:40.157 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (16m) 8m ago 27m 16.9M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:55:40.157 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (16m) 8m ago 25m 18.4M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:55:40.157 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (12m) 8m ago 25m 91.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T05:55:40.157 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (11m) 8m ago 24m 97.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T05:55:40.157 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (11m) 8m ago 24m 81.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T05:55:40.157 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (10m) 8m ago 23m 113M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T05:55:40.157 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (10m) 8m ago 23m 90.3M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T05:55:40.157 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (9m) 8m ago 22m 84.8M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T05:55:40.157 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (15m) 8m ago 26m 64.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:55:40.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.162+0000 7fb148bb9700 1 -- 172.21.15.32:0/3190906331 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb13006cc20 msgr2=0x7fb13006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:40.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.162+0000 7fb148bb9700 1 --2- 172.21.15.32:0/3190906331 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb13006cc20 0x7fb13006f0d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb144071c00 tx=0x7fb12c00b040 comp rx=0 tx=0).stop 2024-04-12T05:55:40.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.162+0000 7fb148bb9700 1 -- 172.21.15.32:0/3190906331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb144102e10 msgr2=0x7fb1440702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:40.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.162+0000 7fb148bb9700 1 --2- 172.21.15.32:0/3190906331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb144102e10 0x7fb1440702c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb13400b3b0 tx=0x7fb134009e30 comp rx=0 tx=0).stop 2024-04-12T05:55:40.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.162+0000 7fb148bb9700 1 -- 172.21.15.32:0/3190906331 shutdown_connections 2024-04-12T05:55:40.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.163+0000 7fb148bb9700 1 --2- 172.21.15.32:0/3190906331 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb13006cc20 0x7fb13006f0d0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:40.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.163+0000 7fb148bb9700 1 --2- 172.21.15.32:0/3190906331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb144102e10 0x7fb1440702c0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:40.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.163+0000 7fb148bb9700 1 --2- 172.21.15.32:0/3190906331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1441037b0 0x7fb144070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:40.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.163+0000 7fb148bb9700 1 -- 172.21.15.32:0/3190906331 >> 172.21.15.32:0/3190906331 conn(0x7fb1440f8980 msgr2=0x7fb1440f9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:55:40.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.163+0000 7fb148bb9700 1 -- 172.21.15.32:0/3190906331 shutdown_connections 2024-04-12T05:55:40.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.163+0000 7fb148bb9700 1 -- 172.21.15.32:0/3190906331 wait complete. 2024-04-12T05:55:40.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.265+0000 7f40993d0700 1 -- 172.21.15.32:0/3087419373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40941026c0 msgr2=0x7f40940fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:40.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.265+0000 7f40993d0700 1 --2- 172.21.15.32:0/3087419373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40941026c0 0x7f40940fbd30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4088004940 tx=0x7f4088009e30 comp rx=0 tx=0).stop 2024-04-12T05:55:40.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.265+0000 7f40993d0700 1 -- 172.21.15.32:0/3087419373 shutdown_connections 2024-04-12T05:55:40.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.265+0000 7f40993d0700 1 --2- 172.21.15.32:0/3087419373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40941026c0 0x7f40940fbd30 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:40.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.265+0000 7f40993d0700 1 --2- 172.21.15.32:0/3087419373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4094101d20 0x7f40941020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:40.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.265+0000 7f40993d0700 1 -- 172.21.15.32:0/3087419373 >> 172.21.15.32:0/3087419373 conn(0x7f40940f7870 msgr2=0x7f40940f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:55:40.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.266+0000 7f40993d0700 1 -- 172.21.15.32:0/3087419373 shutdown_connections 2024-04-12T05:55:40.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.266+0000 7f40993d0700 1 -- 172.21.15.32:0/3087419373 wait complete. 2024-04-12T05:55:40.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.266+0000 7f40993d0700 1 Processor -- start 2024-04-12T05:55:40.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.266+0000 7f40993d0700 1 -- start start 2024-04-12T05:55:40.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.266+0000 7f40993d0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4094101d20 0x7f40941138e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:40.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.267+0000 7f40993d0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40941026c0 0x7f4094113e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:40.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.267+0000 7f40993d0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4094114470 con 0x7f4094101d20 2024-04-12T05:55:40.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.267+0000 7f40993d0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409410da70 con 0x7f40941026c0 2024-04-12T05:55:40.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.267+0000 7f4092ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4094101d20 0x7f40941138e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:40.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.267+0000 7f4092ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4094101d20 0x7f40941138e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37944/0 (socket says 172.21.15.32:37944) 2024-04-12T05:55:40.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.267+0000 7f4092ffd700 1 -- 172.21.15.32:0/3335515719 learned_addr learned my addr 172.21.15.32:0/3335515719 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:55:40.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.267+0000 7f40927fc700 1 --2- 172.21.15.32:0/3335515719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40941026c0 0x7f4094113e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:40.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.267+0000 7f4092ffd700 1 -- 172.21.15.32:0/3335515719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40941026c0 msgr2=0x7f4094113e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:40.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.267+0000 7f4092ffd700 1 --2- 172.21.15.32:0/3335515719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40941026c0 0x7f4094113e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:40.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.267+0000 7f4092ffd700 1 -- 172.21.15.32:0/3335515719 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4088005040 con 0x7f4094101d20 2024-04-12T05:55:40.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.267+0000 7f4092ffd700 1 --2- 172.21.15.32:0/3335515719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4094101d20 0x7f40941138e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f408400c8c0 tx=0x7f40840044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:55:40.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.267+0000 7f407b7fe700 1 -- 172.21.15.32:0/3335515719 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f408400a980 con 0x7f4094101d20 2024-04-12T05:55:40.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.267+0000 7f40993d0700 1 -- 172.21.15.32:0/3335515719 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f409410dd50 con 0x7f4094101d20 2024-04-12T05:55:40.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.267+0000 7f407b7fe700 1 -- 172.21.15.32:0/3335515719 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f408400ce60 con 0x7f4094101d20 2024-04-12T05:55:40.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.267+0000 7f407b7fe700 1 -- 172.21.15.32:0/3335515719 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40840113a0 con 0x7f4094101d20 2024-04-12T05:55:40.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.267+0000 7f40993d0700 1 -- 172.21.15.32:0/3335515719 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f409410e310 con 0x7f4094101d20 2024-04-12T05:55:40.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.268+0000 7f40993d0700 1 -- 172.21.15.32:0/3335515719 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40940fd470 con 0x7f4094101d20 2024-04-12T05:55:40.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.269+0000 7f407b7fe700 1 -- 172.21.15.32:0/3335515719 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4084012400 con 0x7f4094101d20 2024-04-12T05:55:40.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.269+0000 7f407b7fe700 1 --2- 172.21.15.32:0/3335515719 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f407c06cc20 0x7f407c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:40.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.269+0000 7f407b7fe700 1 -- 172.21.15.32:0/3335515719 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f408408c1d0 con 0x7f4094101d20 2024-04-12T05:55:40.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.269+0000 7f40927fc700 1 --2- 172.21.15.32:0/3335515719 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f407c06cc20 0x7f407c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:40.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.270+0000 7f40927fc700 1 --2- 172.21.15.32:0/3335515719 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f407c06cc20 0x7f407c06f0d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f408800b3b0 tx=0x7f4088009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:55:40.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.274+0000 7f407b7fe700 1 -- 172.21.15.32:0/3335515719 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4084059c40 con 0x7f4094101d20 2024-04-12T05:55:40.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:40 smithi032 ceph-mon[146817]: from='client.34872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:55:40.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:40 smithi032 ceph-mon[146817]: from='client.34876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:55:40.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:40 smithi067 ceph-mon[112979]: from='client.34872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:55:40.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:40 smithi067 ceph-mon[112979]: from='client.34876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:55:40.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.560+0000 7f40993d0700 1 -- 172.21.15.32:0/3335515719 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4094108cc0 con 0x7f4094101d20 2024-04-12T05:55:40.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.561+0000 7f407b7fe700 1 -- 172.21.15.32:0/3335515719 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f40840597d0 con 0x7f4094101d20 2024-04-12T05:55:40.556 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:55:40.556 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:55:40.556 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:55:40.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:55:40.557 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:55:40.557 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:55:40.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:55:40.557 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:55:40.557 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:55:40.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:55:40.557 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:55:40.557 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:55:40.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:55:40.557 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:55:40.557 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:55:40.557 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:55:40.557 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:55:40.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.563+0000 7f40993d0700 1 -- 172.21.15.32:0/3335515719 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f407c06cc20 msgr2=0x7f407c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:40.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.563+0000 7f40993d0700 1 --2- 172.21.15.32:0/3335515719 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f407c06cc20 0x7f407c06f0d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f408800b3b0 tx=0x7f4088009e30 comp rx=0 tx=0).stop 2024-04-12T05:55:40.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.563+0000 7f40993d0700 1 -- 172.21.15.32:0/3335515719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4094101d20 msgr2=0x7f40941138e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:40.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.563+0000 7f40993d0700 1 --2- 172.21.15.32:0/3335515719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4094101d20 0x7f40941138e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f408400c8c0 tx=0x7f40840044c0 comp rx=0 tx=0).stop 2024-04-12T05:55:40.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.563+0000 7f40993d0700 1 -- 172.21.15.32:0/3335515719 shutdown_connections 2024-04-12T05:55:40.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.563+0000 7f40993d0700 1 --2- 172.21.15.32:0/3335515719 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f407c06cc20 0x7f407c06f0d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:40.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.563+0000 7f40993d0700 1 --2- 172.21.15.32:0/3335515719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4094101d20 0x7f40941138e0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:40.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.563+0000 7f40993d0700 1 --2- 172.21.15.32:0/3335515719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40941026c0 0x7f4094113e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:40.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.563+0000 7f40993d0700 1 -- 172.21.15.32:0/3335515719 >> 172.21.15.32:0/3335515719 conn(0x7f40940f7870 msgr2=0x7f40940f90a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:55:40.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.563+0000 7f40993d0700 1 -- 172.21.15.32:0/3335515719 shutdown_connections 2024-04-12T05:55:40.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.563+0000 7f40993d0700 1 -- 172.21.15.32:0/3335515719 wait complete. 2024-04-12T05:55:40.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.666+0000 7ff5615da700 1 -- 172.21.15.32:0/1606256081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff55c102710 msgr2=0x7ff55c0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:40.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.666+0000 7ff5615da700 1 --2- 172.21.15.32:0/1606256081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff55c102710 0x7ff55c0fbd80 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff544004940 tx=0x7ff544009e30 comp rx=0 tx=0).stop 2024-04-12T05:55:40.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.666+0000 7ff5615da700 1 -- 172.21.15.32:0/1606256081 shutdown_connections 2024-04-12T05:55:40.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.666+0000 7ff5615da700 1 --2- 172.21.15.32:0/1606256081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff55c102710 0x7ff55c0fbd80 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:40.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.666+0000 7ff5615da700 1 --2- 172.21.15.32:0/1606256081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff55c101d70 0x7ff55c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:40.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.666+0000 7ff5615da700 1 -- 172.21.15.32:0/1606256081 >> 172.21.15.32:0/1606256081 conn(0x7ff55c072bc0 msgr2=0x7ff55c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:55:40.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.666+0000 7ff5615da700 1 -- 172.21.15.32:0/1606256081 shutdown_connections 2024-04-12T05:55:40.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.666+0000 7ff5615da700 1 -- 172.21.15.32:0/1606256081 wait complete. 2024-04-12T05:55:40.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.667+0000 7ff5615da700 1 Processor -- start 2024-04-12T05:55:40.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.667+0000 7ff5615da700 1 -- start start 2024-04-12T05:55:40.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.667+0000 7ff5615da700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff55c101d70 0x7ff55c113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:40.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.667+0000 7ff5615da700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff55c102710 0x7ff55c114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:40.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.667+0000 7ff5615da700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55c114670 con 0x7ff55c101d70 2024-04-12T05:55:40.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.667+0000 7ff5615da700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55c10da60 con 0x7ff55c102710 2024-04-12T05:55:40.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.667+0000 7ff55affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff55c101d70 0x7ff55c113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:40.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.667+0000 7ff55affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff55c101d70 0x7ff55c113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37962/0 (socket says 172.21.15.32:37962) 2024-04-12T05:55:40.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.667+0000 7ff55affd700 1 -- 172.21.15.32:0/596262920 learned_addr learned my addr 172.21.15.32:0/596262920 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:55:40.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.668+0000 7ff55affd700 1 -- 172.21.15.32:0/596262920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff55c102710 msgr2=0x7ff55c114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:40.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.668+0000 7ff55affd700 1 --2- 172.21.15.32:0/596262920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff55c102710 0x7ff55c114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:40.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.668+0000 7ff55affd700 1 -- 172.21.15.32:0/596262920 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff544005040 con 0x7ff55c101d70 2024-04-12T05:55:40.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.668+0000 7ff55affd700 1 --2- 172.21.15.32:0/596262920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff55c101d70 0x7ff55c113ae0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff54c002a50 tx=0x7ff54c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:55:40.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.668+0000 7ff5537fe700 1 -- 172.21.15.32:0/596262920 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff54c007cc0 con 0x7ff55c101d70 2024-04-12T05:55:40.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.668+0000 7ff5537fe700 1 -- 172.21.15.32:0/596262920 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff54c00f040 con 0x7ff55c101d70 2024-04-12T05:55:40.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.668+0000 7ff5615da700 1 -- 172.21.15.32:0/596262920 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff55c10dd40 con 0x7ff55c101d70 2024-04-12T05:55:40.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.668+0000 7ff5537fe700 1 -- 172.21.15.32:0/596262920 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff54c009540 con 0x7ff55c101d70 2024-04-12T05:55:40.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.668+0000 7ff5615da700 1 -- 172.21.15.32:0/596262920 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff55c10e300 con 0x7ff55c101d70 2024-04-12T05:55:40.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.669+0000 7ff5615da700 1 -- 172.21.15.32:0/596262920 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff55c0fd4c0 con 0x7ff55c101d70 2024-04-12T05:55:40.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.670+0000 7ff5537fe700 1 -- 172.21.15.32:0/596262920 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ff54c0096a0 con 0x7ff55c101d70 2024-04-12T05:55:40.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.670+0000 7ff5537fe700 1 --2- 172.21.15.32:0/596262920 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff54806cc20 0x7ff54806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:40.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.670+0000 7ff5537fe700 1 -- 172.21.15.32:0/596262920 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7ff54c08bed0 con 0x7ff55c101d70 2024-04-12T05:55:40.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.670+0000 7ff55a7fc700 1 --2- 172.21.15.32:0/596262920 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff54806cc20 0x7ff54806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:40.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.671+0000 7ff55a7fc700 1 --2- 172.21.15.32:0/596262920 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff54806cc20 0x7ff54806f0d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff54400b3b0 tx=0x7ff544009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:55:40.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.675+0000 7ff5537fe700 1 -- 172.21.15.32:0/596262920 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff54c0598c0 con 0x7ff55c101d70 2024-04-12T05:55:40.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.920+0000 7ff5615da700 1 -- 172.21.15.32:0/596262920 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff55c04bed0 con 0x7ff55c101d70 2024-04-12T05:55:40.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.920+0000 7ff5537fe700 1 -- 172.21.15.32:0/596262920 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1835 (secure 0 0 0) 0x7ff54c059450 con 0x7ff55c101d70 2024-04-12T05:55:40.916 INFO:teuthology.orchestra.run.smithi032.stdout:e33 2024-04-12T05:55:40.916 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:55:40.916 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:55:40.916 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:55:40.916 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:55:40.916 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:55:40.916 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:55:40.916 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 24 2024-04-12T05:55:40.917 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:55:40.917 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:55:40.917 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:46:10.668569+0000 2024-04-12T05:55:40.917 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:55:40.917 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:55:40.917 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:55:40.917 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:55:40.917 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:55:40.917 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:55:40.917 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:55:40.917 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:55:40.917 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T05:55:40.917 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:55:40.917 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:55:40.918 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:55:40.918 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T05:55:40.918 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-12T05:55:40.918 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:55:40.918 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:55:40.918 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:55:40.918 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:55:40.918 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:55:40.918 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:55:40.918 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:55:40.918 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:55:40.918 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:55:40.918 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:55:40.918 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-12T05:55:40.919 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:55:40.919 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:55:40.919 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:55:40.919 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{-1:44317} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:55:40.919 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:55:40.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.922+0000 7ff5615da700 1 -- 172.21.15.32:0/596262920 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff54806cc20 msgr2=0x7ff54806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:40.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.922+0000 7ff5615da700 1 --2- 172.21.15.32:0/596262920 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff54806cc20 0x7ff54806f0d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff54400b3b0 tx=0x7ff544009e30 comp rx=0 tx=0).stop 2024-04-12T05:55:40.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.922+0000 7ff5615da700 1 -- 172.21.15.32:0/596262920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff55c101d70 msgr2=0x7ff55c113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:40.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.922+0000 7ff5615da700 1 --2- 172.21.15.32:0/596262920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff55c101d70 0x7ff55c113ae0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff54c002a50 tx=0x7ff54c0046b0 comp rx=0 tx=0).stop 2024-04-12T05:55:40.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.922+0000 7ff5615da700 1 -- 172.21.15.32:0/596262920 shutdown_connections 2024-04-12T05:55:40.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.922+0000 7ff5615da700 1 --2- 172.21.15.32:0/596262920 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff54806cc20 0x7ff54806f0d0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:40.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.922+0000 7ff5615da700 1 --2- 172.21.15.32:0/596262920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff55c101d70 0x7ff55c113ae0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:40.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.922+0000 7ff5615da700 1 --2- 172.21.15.32:0/596262920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff55c102710 0x7ff55c114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:40.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.922+0000 7ff5615da700 1 -- 172.21.15.32:0/596262920 >> 172.21.15.32:0/596262920 conn(0x7ff55c072bc0 msgr2=0x7ff55c065a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:55:40.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.923+0000 7ff5615da700 1 -- 172.21.15.32:0/596262920 shutdown_connections 2024-04-12T05:55:40.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:40.923+0000 7ff5615da700 1 -- 172.21.15.32:0/596262920 wait complete. 2024-04-12T05:55:40.923 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-12T05:55:41.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.027+0000 7f72a2169700 1 -- 172.21.15.32:0/1175767624 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f729c0fb4a0 msgr2=0x7f729c0fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:41.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.027+0000 7f72a2169700 1 --2- 172.21.15.32:0/1175767624 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f729c0fb4a0 0x7f729c0fb910 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f7290004940 tx=0x7f7290009e30 comp rx=0 tx=0).stop 2024-04-12T05:55:41.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.027+0000 7f72a2169700 1 -- 172.21.15.32:0/1175767624 shutdown_connections 2024-04-12T05:55:41.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.027+0000 7f72a2169700 1 --2- 172.21.15.32:0/1175767624 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f729c0fb4a0 0x7f729c0fb910 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:41.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.027+0000 7f72a2169700 1 --2- 172.21.15.32:0/1175767624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f729c102e70 0x7f729c103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:41.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.027+0000 7f72a2169700 1 -- 172.21.15.32:0/1175767624 >> 172.21.15.32:0/1175767624 conn(0x7f729c0714c0 msgr2=0x7f729c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:55:41.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.027+0000 7f72a2169700 1 -- 172.21.15.32:0/1175767624 shutdown_connections 2024-04-12T05:55:41.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.027+0000 7f72a2169700 1 -- 172.21.15.32:0/1175767624 wait complete. 2024-04-12T05:55:41.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.027+0000 7f72a2169700 1 Processor -- start 2024-04-12T05:55:41.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.028+0000 7f72a2169700 1 -- start start 2024-04-12T05:55:41.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.028+0000 7f72a2169700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f729c0fb4a0 0x7f729c115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:41.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.028+0000 7f72a2169700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f729c102e70 0x7f729c1161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:41.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.028+0000 7f72a2169700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f729c116830 con 0x7f729c0fb4a0 2024-04-12T05:55:41.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.028+0000 7f72a2169700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f729c10fc00 con 0x7f729c102e70 2024-04-12T05:55:41.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.028+0000 7f729b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f729c0fb4a0 0x7f729c115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:41.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.028+0000 7f729b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f729c0fb4a0 0x7f729c115ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37994/0 (socket says 172.21.15.32:37994) 2024-04-12T05:55:41.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.028+0000 7f729b7fe700 1 -- 172.21.15.32:0/458088178 learned_addr learned my addr 172.21.15.32:0/458088178 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:55:41.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.028+0000 7f729affd700 1 --2- 172.21.15.32:0/458088178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f729c102e70 0x7f729c1161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:41.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.028+0000 7f729b7fe700 1 -- 172.21.15.32:0/458088178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f729c102e70 msgr2=0x7f729c1161e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:41.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.028+0000 7f729b7fe700 1 --2- 172.21.15.32:0/458088178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f729c102e70 0x7f729c1161e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:41.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.028+0000 7f729b7fe700 1 -- 172.21.15.32:0/458088178 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7290005040 con 0x7f729c0fb4a0 2024-04-12T05:55:41.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.029+0000 7f729b7fe700 1 --2- 172.21.15.32:0/458088178 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f729c0fb4a0 0x7f729c115ca0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f728c00c830 tx=0x7f728c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:55:41.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.029+0000 7f7283fff700 1 -- 172.21.15.32:0/458088178 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f728c00a760 con 0x7f729c0fb4a0 2024-04-12T05:55:41.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.029+0000 7f7283fff700 1 -- 172.21.15.32:0/458088178 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f728c00ad70 con 0x7f729c0fb4a0 2024-04-12T05:55:41.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.029+0000 7f72a2169700 1 -- 172.21.15.32:0/458088178 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f729c10fee0 con 0x7f729c0fb4a0 2024-04-12T05:55:41.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.029+0000 7f7283fff700 1 -- 172.21.15.32:0/458088178 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f728c0113a0 con 0x7f729c0fb4a0 2024-04-12T05:55:41.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.029+0000 7f72a2169700 1 -- 172.21.15.32:0/458088178 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f729c1104a0 con 0x7f729c0fb4a0 2024-04-12T05:55:41.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.030+0000 7f72a2169700 1 -- 172.21.15.32:0/458088178 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f729c107290 con 0x7f729c0fb4a0 2024-04-12T05:55:41.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.030+0000 7f7283fff700 1 -- 172.21.15.32:0/458088178 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f728c012400 con 0x7f729c0fb4a0 2024-04-12T05:55:41.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.031+0000 7f7283fff700 1 --2- 172.21.15.32:0/458088178 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f728406cc30 0x7f728406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:41.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.031+0000 7f7283fff700 1 -- 172.21.15.32:0/458088178 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f728c08c070 con 0x7f729c0fb4a0 2024-04-12T05:55:41.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.031+0000 7f729affd700 1 --2- 172.21.15.32:0/458088178 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f728406cc30 0x7f728406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:41.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.031+0000 7f729affd700 1 --2- 172.21.15.32:0/458088178 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f728406cc30 0x7f728406f0e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f729c111540 tx=0x7f7290009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:55:41.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.035+0000 7f7283fff700 1 -- 172.21.15.32:0/458088178 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f728c059b80 con 0x7f729c0fb4a0 2024-04-12T05:55:41.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.255+0000 7f72a2169700 1 -- 172.21.15.32:0/458088178 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f729c109ea0 con 0x7f728406cc30 2024-04-12T05:55:41.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.257+0000 7f7283fff700 1 -- 172.21.15.32:0/458088178 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7290008e30 con 0x7f728406cc30 2024-04-12T05:55:41.252 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:55:41.252 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:55:41.252 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:55:41.252 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:55:41.253 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:55:41.253 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-12T05:55:41.253 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:55:41.253 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-12T05:55:41.253 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:55:41.253 INFO:teuthology.orchestra.run.smithi032.stdout: "mds" 2024-04-12T05:55:41.253 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:55:41.253 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:55:41.253 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:55:41.253 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:55:41.253 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:55:41.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.259+0000 7f72a2169700 1 -- 172.21.15.32:0/458088178 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f728406cc30 msgr2=0x7f728406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:41.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.259+0000 7f72a2169700 1 --2- 172.21.15.32:0/458088178 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f728406cc30 0x7f728406f0e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f729c111540 tx=0x7f7290009e30 comp rx=0 tx=0).stop 2024-04-12T05:55:41.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.259+0000 7f72a2169700 1 -- 172.21.15.32:0/458088178 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f729c0fb4a0 msgr2=0x7f729c115ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:41.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.259+0000 7f72a2169700 1 --2- 172.21.15.32:0/458088178 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f729c0fb4a0 0x7f729c115ca0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f728c00c830 tx=0x7f728c004470 comp rx=0 tx=0).stop 2024-04-12T05:55:41.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.259+0000 7f72a2169700 1 -- 172.21.15.32:0/458088178 shutdown_connections 2024-04-12T05:55:41.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.259+0000 7f72a2169700 1 --2- 172.21.15.32:0/458088178 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f728406cc30 0x7f728406f0e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:41.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.259+0000 7f72a2169700 1 --2- 172.21.15.32:0/458088178 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f729c0fb4a0 0x7f729c115ca0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:41.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.259+0000 7f72a2169700 1 --2- 172.21.15.32:0/458088178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f729c102e70 0x7f729c1161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:41.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.259+0000 7f72a2169700 1 -- 172.21.15.32:0/458088178 >> 172.21.15.32:0/458088178 conn(0x7f729c0714c0 msgr2=0x7f729c0fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:55:41.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.259+0000 7f72a2169700 1 -- 172.21.15.32:0/458088178 shutdown_connections 2024-04-12T05:55:41.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.259+0000 7f72a2169700 1 -- 172.21.15.32:0/458088178 wait complete. 2024-04-12T05:55:41.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:41 smithi032 ceph-mon[146817]: pgmap v455: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:41.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:41 smithi032 ceph-mon[146817]: from='client.34880 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:55:41.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:41 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3335515719' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:55:41.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:41 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/596262920' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:55:41.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.362+0000 7f9256b08700 1 -- 172.21.15.32:0/984775884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92500fe7a0 msgr2=0x7f9250106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:41.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.362+0000 7f9256b08700 1 --2- 172.21.15.32:0/984775884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92500fe7a0 0x7f9250106e70 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f923c004940 tx=0x7f923c009e30 comp rx=0 tx=0).stop 2024-04-12T05:55:41.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.362+0000 7f9256b08700 1 -- 172.21.15.32:0/984775884 shutdown_connections 2024-04-12T05:55:41.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.362+0000 7f9256b08700 1 --2- 172.21.15.32:0/984775884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92500fe7a0 0x7f9250106e70 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:41.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.362+0000 7f9256b08700 1 --2- 172.21.15.32:0/984775884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92500fde00 0x7f92500fe1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:41.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.362+0000 7f9256b08700 1 -- 172.21.15.32:0/984775884 >> 172.21.15.32:0/984775884 conn(0x7f92500f7880 msgr2=0x7f92500f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:55:41.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.363+0000 7f9256b08700 1 -- 172.21.15.32:0/984775884 shutdown_connections 2024-04-12T05:55:41.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.363+0000 7f9256b08700 1 -- 172.21.15.32:0/984775884 wait complete. 2024-04-12T05:55:41.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.363+0000 7f9256b08700 1 Processor -- start 2024-04-12T05:55:41.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.363+0000 7f9256b08700 1 -- start start 2024-04-12T05:55:41.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.363+0000 7f9256b08700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92500fde00 0x7f92500fbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:41.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.363+0000 7f9256b08700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92500fe7a0 0x7f92500fc200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:41.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.363+0000 7f9256b08700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92500fc790 con 0x7f92500fde00 2024-04-12T05:55:41.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.363+0000 7f9256b08700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92500fc8d0 con 0x7f92500fe7a0 2024-04-12T05:55:41.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.364+0000 7f92548a4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92500fde00 0x7f92500fbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:41.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.364+0000 7f92548a4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92500fde00 0x7f92500fbcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:38024/0 (socket says 172.21.15.32:38024) 2024-04-12T05:55:41.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.364+0000 7f92548a4700 1 -- 172.21.15.32:0/781927455 learned_addr learned my addr 172.21.15.32:0/781927455 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:55:41.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.364+0000 7f924ffff700 1 --2- 172.21.15.32:0/781927455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92500fe7a0 0x7f92500fc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:41.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.364+0000 7f92548a4700 1 -- 172.21.15.32:0/781927455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92500fe7a0 msgr2=0x7f92500fc200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:41.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.364+0000 7f92548a4700 1 --2- 172.21.15.32:0/781927455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92500fe7a0 0x7f92500fc200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:41.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.364+0000 7f92548a4700 1 -- 172.21.15.32:0/781927455 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f923c005040 con 0x7f92500fde00 2024-04-12T05:55:41.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.364+0000 7f92548a4700 1 --2- 172.21.15.32:0/781927455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92500fde00 0x7f92500fbcc0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f924400c830 tx=0x7f9244004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:55:41.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.364+0000 7f924d7fa700 1 -- 172.21.15.32:0/781927455 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92440106f0 con 0x7f92500fde00 2024-04-12T05:55:41.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.364+0000 7f924d7fa700 1 -- 172.21.15.32:0/781927455 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9244010d00 con 0x7f92500fde00 2024-04-12T05:55:41.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.364+0000 7f9256b08700 1 -- 172.21.15.32:0/781927455 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f925011a090 con 0x7f92500fde00 2024-04-12T05:55:41.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.364+0000 7f924d7fa700 1 -- 172.21.15.32:0/781927455 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9244012e90 con 0x7f92500fde00 2024-04-12T05:55:41.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.364+0000 7f9256b08700 1 -- 172.21.15.32:0/781927455 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f925011a490 con 0x7f92500fde00 2024-04-12T05:55:41.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.365+0000 7f9256b08700 1 -- 172.21.15.32:0/781927455 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92500666f0 con 0x7f92500fde00 2024-04-12T05:55:41.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.366+0000 7f924d7fa700 1 -- 172.21.15.32:0/781927455 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9244010850 con 0x7f92500fde00 2024-04-12T05:55:41.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.366+0000 7f924d7fa700 1 --2- 172.21.15.32:0/781927455 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f924006cc30 0x7f924006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:55:41.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.366+0000 7f924d7fa700 1 -- 172.21.15.32:0/781927455 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f924408bfe0 con 0x7f92500fde00 2024-04-12T05:55:41.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.366+0000 7f924ffff700 1 --2- 172.21.15.32:0/781927455 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f924006cc30 0x7f924006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:55:41.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.367+0000 7f924ffff700 1 --2- 172.21.15.32:0/781927455 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f924006cc30 0x7f924006f0e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f92500fd600 tx=0x7f923c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:55:41.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.371+0000 7f924d7fa700 1 -- 172.21.15.32:0/781927455 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9244056060 con 0x7f92500fde00 2024-04-12T05:55:41.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:41 smithi067 ceph-mon[112979]: pgmap v455: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:41.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:41 smithi067 ceph-mon[112979]: from='client.34880 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:55:41.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:41 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3335515719' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:55:41.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:41 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/596262920' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:55:41.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.661+0000 7f9256b08700 1 -- 172.21.15.32:0/781927455 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f925005f5f0 con 0x7f92500fde00 2024-04-12T05:55:41.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.661+0000 7f924d7fa700 1 -- 172.21.15.32:0/781927455 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f9244059680 con 0x7f92500fde00 2024-04-12T05:55:41.657 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:55:41.657 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:55:41.657 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-12T05:55:41.657 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:55:41.657 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:55:41.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.663+0000 7f9256b08700 1 -- 172.21.15.32:0/781927455 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f924006cc30 msgr2=0x7f924006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:41.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.663+0000 7f9256b08700 1 --2- 172.21.15.32:0/781927455 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f924006cc30 0x7f924006f0e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f92500fd600 tx=0x7f923c009e30 comp rx=0 tx=0).stop 2024-04-12T05:55:41.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.663+0000 7f9256b08700 1 -- 172.21.15.32:0/781927455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92500fde00 msgr2=0x7f92500fbcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:55:41.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.663+0000 7f9256b08700 1 --2- 172.21.15.32:0/781927455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92500fde00 0x7f92500fbcc0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f924400c830 tx=0x7f9244004470 comp rx=0 tx=0).stop 2024-04-12T05:55:41.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.663+0000 7f9256b08700 1 -- 172.21.15.32:0/781927455 shutdown_connections 2024-04-12T05:55:41.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.663+0000 7f9256b08700 1 --2- 172.21.15.32:0/781927455 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f924006cc30 0x7f924006f0e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:41.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.663+0000 7f9256b08700 1 --2- 172.21.15.32:0/781927455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92500fde00 0x7f92500fbcc0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:41.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.663+0000 7f9256b08700 1 --2- 172.21.15.32:0/781927455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92500fe7a0 0x7f92500fc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:55:41.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.663+0000 7f9256b08700 1 -- 172.21.15.32:0/781927455 >> 172.21.15.32:0/781927455 conn(0x7f92500f7880 msgr2=0x7f92500f8ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:55:41.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.663+0000 7f9256b08700 1 -- 172.21.15.32:0/781927455 shutdown_connections 2024-04-12T05:55:41.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:55:41.663+0000 7f9256b08700 1 -- 172.21.15.32:0/781927455 wait complete. 2024-04-12T05:55:42.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:42 smithi032 ceph-mon[146817]: from='client.34892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:55:42.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:42 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/781927455' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:55:42.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:42 smithi067 ceph-mon[112979]: from='client.34892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:55:42.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:42 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/781927455' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:55:43.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:43 smithi032 ceph-mon[146817]: pgmap v456: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:43.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:43 smithi067 ceph-mon[112979]: pgmap v456: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:45.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:45 smithi032 ceph-mon[146817]: pgmap v457: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:45.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:45 smithi067 ceph-mon[112979]: pgmap v457: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:47.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:47 smithi032 ceph-mon[146817]: pgmap v458: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:47.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:47 smithi067 ceph-mon[112979]: pgmap v458: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:48.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:48 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:55:48.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:48 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:55:49.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:49 smithi032 ceph-mon[146817]: pgmap v459: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:49.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:49 smithi067 ceph-mon[112979]: pgmap v459: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:51.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:51 smithi032 ceph-mon[146817]: pgmap v460: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:51.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:51 smithi067 ceph-mon[112979]: pgmap v460: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:53.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:53 smithi032 ceph-mon[146817]: pgmap v461: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:53.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:53 smithi067 ceph-mon[112979]: pgmap v461: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:55.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:55 smithi032 ceph-mon[146817]: pgmap v462: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:55.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:55 smithi067 ceph-mon[112979]: pgmap v462: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:57.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:57 smithi032 ceph-mon[146817]: pgmap v463: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:57.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:57 smithi067 ceph-mon[112979]: pgmap v463: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:59.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:55:59 smithi032 ceph-mon[146817]: pgmap v464: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:59.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:55:59 smithi067 ceph-mon[112979]: pgmap v464: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:01.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:01 smithi032 ceph-mon[146817]: pgmap v465: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:01.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:01 smithi067 ceph-mon[112979]: pgmap v465: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:03.394 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:03 smithi032 ceph-mon[146817]: pgmap v466: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:03.394 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:56:03.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:03 smithi067 ceph-mon[112979]: pgmap v466: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:03.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:56:05.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:05 smithi067 ceph-mon[112979]: pgmap v467: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:05.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:05 smithi032 ceph-mon[146817]: pgmap v467: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:07.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:07 smithi067 ceph-mon[112979]: pgmap v468: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:07.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:07 smithi032 ceph-mon[146817]: pgmap v468: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:09.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:09 smithi067 ceph-mon[112979]: pgmap v469: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:09.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:09 smithi032 ceph-mon[146817]: pgmap v469: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:11.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:11 smithi067 ceph-mon[112979]: pgmap v470: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:11.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:11 smithi032 ceph-mon[146817]: pgmap v470: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:11.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.782+0000 7ff8b0bda700 1 -- 172.21.15.32:0/3061371071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8ac102710 msgr2=0x7ff8ac0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:11.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.782+0000 7ff8b0bda700 1 --2- 172.21.15.32:0/3061371071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8ac102710 0x7ff8ac0fbd80 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff894004940 tx=0x7ff894009e30 comp rx=0 tx=0).stop 2024-04-12T05:56:11.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.783+0000 7ff8b0bda700 1 -- 172.21.15.32:0/3061371071 shutdown_connections 2024-04-12T05:56:11.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.783+0000 7ff8b0bda700 1 --2- 172.21.15.32:0/3061371071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8ac102710 0x7ff8ac0fbd80 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:11.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.783+0000 7ff8b0bda700 1 --2- 172.21.15.32:0/3061371071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8ac101d70 0x7ff8ac102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:11.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.783+0000 7ff8b0bda700 1 -- 172.21.15.32:0/3061371071 >> 172.21.15.32:0/3061371071 conn(0x7ff8ac072bc0 msgr2=0x7ff8ac072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:56:11.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.783+0000 7ff8b0bda700 1 -- 172.21.15.32:0/3061371071 shutdown_connections 2024-04-12T05:56:11.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.783+0000 7ff8b0bda700 1 -- 172.21.15.32:0/3061371071 wait complete. 2024-04-12T05:56:11.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.783+0000 7ff8b0bda700 1 Processor -- start 2024-04-12T05:56:11.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.783+0000 7ff8b0bda700 1 -- start start 2024-04-12T05:56:11.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.783+0000 7ff8b0bda700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8ac101d70 0x7ff8ac10e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:11.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.783+0000 7ff8b0bda700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8ac102710 0x7ff8ac10eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:11.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.783+0000 7ff8b0bda700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac10f530 con 0x7ff8ac102710 2024-04-12T05:56:11.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.783+0000 7ff8b0bda700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac10f670 con 0x7ff8ac101d70 2024-04-12T05:56:11.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.784+0000 7ff8a9d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8ac102710 0x7ff8ac10eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:11.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.784+0000 7ff8a9d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8ac102710 0x7ff8ac10eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57722/0 (socket says 172.21.15.32:57722) 2024-04-12T05:56:11.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.784+0000 7ff8a9d9b700 1 -- 172.21.15.32:0/2855646768 learned_addr learned my addr 172.21.15.32:0/2855646768 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:56:11.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.784+0000 7ff8aa59c700 1 --2- 172.21.15.32:0/2855646768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8ac101d70 0x7ff8ac10e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:11.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.784+0000 7ff8a9d9b700 1 -- 172.21.15.32:0/2855646768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8ac101d70 msgr2=0x7ff8ac10e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:11.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.784+0000 7ff8a9d9b700 1 --2- 172.21.15.32:0/2855646768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8ac101d70 0x7ff8ac10e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:11.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.784+0000 7ff8a9d9b700 1 -- 172.21.15.32:0/2855646768 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff894005040 con 0x7ff8ac102710 2024-04-12T05:56:11.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.784+0000 7ff8a9d9b700 1 --2- 172.21.15.32:0/2855646768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8ac102710 0x7ff8ac10eee0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ff894000c00 tx=0x7ff894009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:56:11.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.784+0000 7ff8a2ffd700 1 -- 172.21.15.32:0/2855646768 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff89400e070 con 0x7ff8ac102710 2024-04-12T05:56:11.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.784+0000 7ff8a2ffd700 1 -- 172.21.15.32:0/2855646768 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff894013de0 con 0x7ff8ac102710 2024-04-12T05:56:11.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.784+0000 7ff8b0bda700 1 -- 172.21.15.32:0/2855646768 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8ac117a10 con 0x7ff8ac102710 2024-04-12T05:56:11.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.784+0000 7ff8a2ffd700 1 -- 172.21.15.32:0/2855646768 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff89401cde0 con 0x7ff8ac102710 2024-04-12T05:56:11.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.784+0000 7ff8b0bda700 1 -- 172.21.15.32:0/2855646768 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8ac117ef0 con 0x7ff8ac102710 2024-04-12T05:56:11.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.785+0000 7ff8b0bda700 1 -- 172.21.15.32:0/2855646768 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8ac0ff080 con 0x7ff8ac102710 2024-04-12T05:56:11.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.786+0000 7ff8a2ffd700 1 -- 172.21.15.32:0/2855646768 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ff89401c600 con 0x7ff8ac102710 2024-04-12T05:56:11.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.786+0000 7ff8a2ffd700 1 --2- 172.21.15.32:0/2855646768 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff89806cc20 0x7ff89806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:11.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.786+0000 7ff8a2ffd700 1 -- 172.21.15.32:0/2855646768 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7ff89408d500 con 0x7ff8ac102710 2024-04-12T05:56:11.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.786+0000 7ff8aa59c700 1 --2- 172.21.15.32:0/2855646768 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff89806cc20 0x7ff89806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:11.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.787+0000 7ff8aa59c700 1 --2- 172.21.15.32:0/2855646768 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff89806cc20 0x7ff89806f0d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff89c004600 tx=0x7ff89c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:56:11.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:11.791+0000 7ff8a2ffd700 1 -- 172.21.15.32:0/2855646768 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff894057580 con 0x7ff8ac102710 2024-04-12T05:56:12.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.014+0000 7ff8b0bda700 1 -- 172.21.15.32:0/2855646768 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff8ac065e20 con 0x7ff89806cc20 2024-04-12T05:56:12.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.016+0000 7ff8a2ffd700 1 -- 172.21.15.32:0/2855646768 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff8ac065e20 con 0x7ff89806cc20 2024-04-12T05:56:12.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.018+0000 7ff8b0bda700 1 -- 172.21.15.32:0/2855646768 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff89806cc20 msgr2=0x7ff89806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:12.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.018+0000 7ff8b0bda700 1 --2- 172.21.15.32:0/2855646768 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff89806cc20 0x7ff89806f0d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff89c004600 tx=0x7ff89c00b040 comp rx=0 tx=0).stop 2024-04-12T05:56:12.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.018+0000 7ff8b0bda700 1 -- 172.21.15.32:0/2855646768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8ac102710 msgr2=0x7ff8ac10eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:12.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.018+0000 7ff8b0bda700 1 --2- 172.21.15.32:0/2855646768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8ac102710 0x7ff8ac10eee0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ff894000c00 tx=0x7ff894009e30 comp rx=0 tx=0).stop 2024-04-12T05:56:12.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.018+0000 7ff8b0bda700 1 -- 172.21.15.32:0/2855646768 shutdown_connections 2024-04-12T05:56:12.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.018+0000 7ff8b0bda700 1 --2- 172.21.15.32:0/2855646768 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff89806cc20 0x7ff89806f0d0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:12.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.018+0000 7ff8b0bda700 1 --2- 172.21.15.32:0/2855646768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8ac101d70 0x7ff8ac10e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:12.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.018+0000 7ff8b0bda700 1 --2- 172.21.15.32:0/2855646768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8ac102710 0x7ff8ac10eee0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:12.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.018+0000 7ff8b0bda700 1 -- 172.21.15.32:0/2855646768 >> 172.21.15.32:0/2855646768 conn(0x7ff8ac072bc0 msgr2=0x7ff8ac0faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:56:12.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.018+0000 7ff8b0bda700 1 -- 172.21.15.32:0/2855646768 shutdown_connections 2024-04-12T05:56:12.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.018+0000 7ff8b0bda700 1 -- 172.21.15.32:0/2855646768 wait complete. 2024-04-12T05:56:12.022 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:56:12.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.124+0000 7facf64e1700 1 -- 172.21.15.32:0/2912806780 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facf0101d70 msgr2=0x7facf0102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:12.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.124+0000 7facf64e1700 1 --2- 172.21.15.32:0/2912806780 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facf0101d70 0x7facf0102140 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7face0004940 tx=0x7face0009e30 comp rx=0 tx=0).stop 2024-04-12T05:56:12.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.124+0000 7facf64e1700 1 -- 172.21.15.32:0/2912806780 shutdown_connections 2024-04-12T05:56:12.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.124+0000 7facf64e1700 1 --2- 172.21.15.32:0/2912806780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facf0102710 0x7facf00fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:12.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.124+0000 7facf64e1700 1 --2- 172.21.15.32:0/2912806780 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facf0101d70 0x7facf0102140 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:12.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.124+0000 7facf64e1700 1 -- 172.21.15.32:0/2912806780 >> 172.21.15.32:0/2912806780 conn(0x7facf0072bc0 msgr2=0x7facf0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:56:12.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.124+0000 7facf64e1700 1 -- 172.21.15.32:0/2912806780 shutdown_connections 2024-04-12T05:56:12.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.124+0000 7facf64e1700 1 -- 172.21.15.32:0/2912806780 wait complete. 2024-04-12T05:56:12.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.125+0000 7facf64e1700 1 Processor -- start 2024-04-12T05:56:12.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.125+0000 7facf64e1700 1 -- start start 2024-04-12T05:56:12.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.125+0000 7facf64e1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facf0101d70 0x7facf01129a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:12.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.125+0000 7facf64e1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facf0102710 0x7facf0112ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:12.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.125+0000 7facf64e1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf0113530 con 0x7facf0101d70 2024-04-12T05:56:12.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.125+0000 7facf64e1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf010c920 con 0x7facf0102710 2024-04-12T05:56:12.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.125+0000 7faceffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facf0101d70 0x7facf01129a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:12.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.125+0000 7faceffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facf0101d70 0x7facf01129a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57738/0 (socket says 172.21.15.32:57738) 2024-04-12T05:56:12.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.125+0000 7faceffff700 1 -- 172.21.15.32:0/3159712915 learned_addr learned my addr 172.21.15.32:0/3159712915 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:56:12.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.126+0000 7faceffff700 1 -- 172.21.15.32:0/3159712915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facf0102710 msgr2=0x7facf0112ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:12.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.126+0000 7faceffff700 1 --2- 172.21.15.32:0/3159712915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facf0102710 0x7facf0112ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:12.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.126+0000 7faceffff700 1 -- 172.21.15.32:0/3159712915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face0005040 con 0x7facf0101d70 2024-04-12T05:56:12.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.126+0000 7faceffff700 1 --2- 172.21.15.32:0/3159712915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facf0101d70 0x7facf01129a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7face0009360 tx=0x7face0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:56:12.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.126+0000 7facecff9700 1 -- 172.21.15.32:0/3159712915 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7face000e040 con 0x7facf0101d70 2024-04-12T05:56:12.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.126+0000 7facecff9700 1 -- 172.21.15.32:0/3159712915 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7face000bd60 con 0x7facf0101d70 2024-04-12T05:56:12.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.126+0000 7facf64e1700 1 -- 172.21.15.32:0/3159712915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facf010cba0 con 0x7facf0101d70 2024-04-12T05:56:12.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.126+0000 7facecff9700 1 -- 172.21.15.32:0/3159712915 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7face0013660 con 0x7facf0101d70 2024-04-12T05:56:12.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.126+0000 7facf64e1700 1 -- 172.21.15.32:0/3159712915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facf010d100 con 0x7facf0101d70 2024-04-12T05:56:12.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.127+0000 7facf64e1700 1 -- 172.21.15.32:0/3159712915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facf00fd4c0 con 0x7facf0101d70 2024-04-12T05:56:12.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.128+0000 7facecff9700 1 -- 172.21.15.32:0/3159712915 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7face001e420 con 0x7facf0101d70 2024-04-12T05:56:12.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.128+0000 7facecff9700 1 --2- 172.21.15.32:0/3159712915 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7facd806cc30 0x7facd806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:12.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.128+0000 7facecff9700 1 -- 172.21.15.32:0/3159712915 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7face008ff40 con 0x7facf0101d70 2024-04-12T05:56:12.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.128+0000 7facef7fe700 1 --2- 172.21.15.32:0/3159712915 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7facd806cc30 0x7facd806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:12.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.128+0000 7facef7fe700 1 --2- 172.21.15.32:0/3159712915 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7facd806cc30 0x7facd806f0e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7facf010dae0 tx=0x7face4009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:56:12.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.133+0000 7facecff9700 1 -- 172.21.15.32:0/3159712915 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7face00c3ab0 con 0x7facf0101d70 2024-04-12T05:56:12.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.355+0000 7facf64e1700 1 -- 172.21.15.32:0/3159712915 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7facf004bed0 con 0x7facd806cc30 2024-04-12T05:56:12.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.357+0000 7facecff9700 1 -- 172.21.15.32:0/3159712915 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7facf004bed0 con 0x7facd806cc30 2024-04-12T05:56:12.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.359+0000 7facf64e1700 1 -- 172.21.15.32:0/3159712915 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7facd806cc30 msgr2=0x7facd806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:12.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.359+0000 7facf64e1700 1 --2- 172.21.15.32:0/3159712915 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7facd806cc30 0x7facd806f0e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7facf010dae0 tx=0x7face4009ce0 comp rx=0 tx=0).stop 2024-04-12T05:56:12.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.359+0000 7facf64e1700 1 -- 172.21.15.32:0/3159712915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facf0101d70 msgr2=0x7facf01129a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:12.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.359+0000 7facf64e1700 1 --2- 172.21.15.32:0/3159712915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facf0101d70 0x7facf01129a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7face0009360 tx=0x7face0009e30 comp rx=0 tx=0).stop 2024-04-12T05:56:12.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.359+0000 7facf64e1700 1 -- 172.21.15.32:0/3159712915 shutdown_connections 2024-04-12T05:56:12.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.359+0000 7facf64e1700 1 --2- 172.21.15.32:0/3159712915 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7facd806cc30 0x7facd806f0e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:12.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.359+0000 7facf64e1700 1 --2- 172.21.15.32:0/3159712915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facf0101d70 0x7facf01129a0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:12.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.359+0000 7facf64e1700 1 --2- 172.21.15.32:0/3159712915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facf0102710 0x7facf0112ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:12.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.359+0000 7facf64e1700 1 -- 172.21.15.32:0/3159712915 >> 172.21.15.32:0/3159712915 conn(0x7facf0072bc0 msgr2=0x7facf00fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:56:12.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.359+0000 7facf64e1700 1 -- 172.21.15.32:0/3159712915 shutdown_connections 2024-04-12T05:56:12.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.359+0000 7facf64e1700 1 -- 172.21.15.32:0/3159712915 wait complete. 2024-04-12T05:56:12.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.462+0000 7f664700d700 1 -- 172.21.15.32:0/4268523107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6640102710 msgr2=0x7f66400fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:12.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.462+0000 7f664700d700 1 --2- 172.21.15.32:0/4268523107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6640102710 0x7f66400fbd80 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f662c004940 tx=0x7f662c009e30 comp rx=0 tx=0).stop 2024-04-12T05:56:12.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.462+0000 7f664700d700 1 -- 172.21.15.32:0/4268523107 shutdown_connections 2024-04-12T05:56:12.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.462+0000 7f664700d700 1 --2- 172.21.15.32:0/4268523107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6640102710 0x7f66400fbd80 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:12.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.462+0000 7f664700d700 1 --2- 172.21.15.32:0/4268523107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6640101d70 0x7f6640102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:12.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.462+0000 7f664700d700 1 -- 172.21.15.32:0/4268523107 >> 172.21.15.32:0/4268523107 conn(0x7f6640072bc0 msgr2=0x7f6640072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:56:12.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.462+0000 7f664700d700 1 -- 172.21.15.32:0/4268523107 shutdown_connections 2024-04-12T05:56:12.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.462+0000 7f664700d700 1 -- 172.21.15.32:0/4268523107 wait complete. 2024-04-12T05:56:12.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.463+0000 7f664700d700 1 Processor -- start 2024-04-12T05:56:12.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.463+0000 7f664700d700 1 -- start start 2024-04-12T05:56:12.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.463+0000 7f664700d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6640101d70 0x7f664010eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:12.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.463+0000 7f664700d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6640102710 0x7f664010f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:12.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.463+0000 7f664700d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664010f5c0 con 0x7f6640102710 2024-04-12T05:56:12.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.463+0000 7f664700d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664010f700 con 0x7f6640101d70 2024-04-12T05:56:12.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.463+0000 7f663ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6640102710 0x7f664010f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:12.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.463+0000 7f663ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6640102710 0x7f664010f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57756/0 (socket says 172.21.15.32:57756) 2024-04-12T05:56:12.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.464+0000 7f663ffff700 1 -- 172.21.15.32:0/3305531399 learned_addr learned my addr 172.21.15.32:0/3305531399 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:56:12.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.464+0000 7f6644da9700 1 --2- 172.21.15.32:0/3305531399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6640101d70 0x7f664010eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:12.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.464+0000 7f663ffff700 1 -- 172.21.15.32:0/3305531399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6640101d70 msgr2=0x7f664010eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:12.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.464+0000 7f663ffff700 1 --2- 172.21.15.32:0/3305531399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6640101d70 0x7f664010eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:12.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.464+0000 7f663ffff700 1 -- 172.21.15.32:0/3305531399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f662c005040 con 0x7f6640102710 2024-04-12T05:56:12.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.464+0000 7f663ffff700 1 --2- 172.21.15.32:0/3305531399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6640102710 0x7f664010f080 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f662c0093b0 tx=0x7f662c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:56:12.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.464+0000 7f663d7fa700 1 -- 172.21.15.32:0/3305531399 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f662c00e040 con 0x7f6640102710 2024-04-12T05:56:12.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.464+0000 7f663d7fa700 1 -- 172.21.15.32:0/3305531399 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f662c013420 con 0x7f6640102710 2024-04-12T05:56:12.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.464+0000 7f664700d700 1 -- 172.21.15.32:0/3305531399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66401179f0 con 0x7f6640102710 2024-04-12T05:56:12.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.464+0000 7f663d7fa700 1 -- 172.21.15.32:0/3305531399 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f662c01c630 con 0x7f6640102710 2024-04-12T05:56:12.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.464+0000 7f664700d700 1 -- 172.21.15.32:0/3305531399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6640117f80 con 0x7f6640102710 2024-04-12T05:56:12.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.465+0000 7f664700d700 1 -- 172.21.15.32:0/3305531399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66400ff080 con 0x7f6640102710 2024-04-12T05:56:12.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.466+0000 7f663d7fa700 1 -- 172.21.15.32:0/3305531399 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f662c00bb80 con 0x7f6640102710 2024-04-12T05:56:12.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.466+0000 7f663d7fa700 1 --2- 172.21.15.32:0/3305531399 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f663006cc20 0x7f663006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:12.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.466+0000 7f663d7fa700 1 -- 172.21.15.32:0/3305531399 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f662c08d280 con 0x7f6640102710 2024-04-12T05:56:12.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.466+0000 7f6644da9700 1 --2- 172.21.15.32:0/3305531399 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f663006cc20 0x7f663006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:12.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.466+0000 7f6644da9700 1 --2- 172.21.15.32:0/3305531399 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f663006cc20 0x7f663006f0d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6634009fd0 tx=0x7f6634009f60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:56:12.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.471+0000 7f663d7fa700 1 -- 172.21.15.32:0/3305531399 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f662c057260 con 0x7f6640102710 2024-04-12T05:56:12.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.682+0000 7f664700d700 1 -- 172.21.15.32:0/3305531399 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6640065e40 con 0x7f663006cc20 2024-04-12T05:56:12.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.692+0000 7f663d7fa700 1 -- 172.21.15.32:0/3305531399 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f6640065e40 con 0x7f663006cc20 2024-04-12T05:56:12.686 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:56:12.686 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (16m) 9m ago 27m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:56:12.686 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (13m) 9m ago 27m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:56:12.686 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (13m) 8m ago 26m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:56:12.686 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (15m) 9m ago 27m 87.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:56:12.686 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (9m) 9m ago 21m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T05:56:12.687 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (9m) 9m ago 21m 13.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T05:56:12.687 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (9m) 8m ago 21m 11.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 6d4aca6f74a0 2024-04-12T05:56:12.687 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (9m) 8m ago 21m 14.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T05:56:12.687 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (17m) 9m ago 28m 507M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:56:12.687 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (16m) 8m ago 26m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:56:12.687 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (14m) 9m ago 29m 53.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:56:12.687 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (14m) 8m ago 26m 43.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:56:12.687 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (16m) 9m ago 27m 16.9M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:56:12.687 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (16m) 8m ago 26m 18.4M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:56:12.687 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (13m) 9m ago 25m 91.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T05:56:12.687 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (12m) 9m ago 25m 97.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T05:56:12.687 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (11m) 9m ago 24m 81.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T05:56:12.687 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (11m) 8m ago 24m 113M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T05:56:12.687 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (10m) 8m ago 23m 90.3M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T05:56:12.687 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (10m) 8m ago 23m 84.8M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T05:56:12.688 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (16m) 9m ago 26m 64.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:56:12.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.694+0000 7f664700d700 1 -- 172.21.15.32:0/3305531399 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f663006cc20 msgr2=0x7f663006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:12.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.694+0000 7f664700d700 1 --2- 172.21.15.32:0/3305531399 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f663006cc20 0x7f663006f0d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6634009fd0 tx=0x7f6634009f60 comp rx=0 tx=0).stop 2024-04-12T05:56:12.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.694+0000 7f664700d700 1 -- 172.21.15.32:0/3305531399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6640102710 msgr2=0x7f664010f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:12.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.694+0000 7f664700d700 1 --2- 172.21.15.32:0/3305531399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6640102710 0x7f664010f080 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f662c0093b0 tx=0x7f662c009e30 comp rx=0 tx=0).stop 2024-04-12T05:56:12.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.694+0000 7f664700d700 1 -- 172.21.15.32:0/3305531399 shutdown_connections 2024-04-12T05:56:12.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.694+0000 7f664700d700 1 --2- 172.21.15.32:0/3305531399 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f663006cc20 0x7f663006f0d0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:12.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.694+0000 7f664700d700 1 --2- 172.21.15.32:0/3305531399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6640101d70 0x7f664010eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:12.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.694+0000 7f664700d700 1 --2- 172.21.15.32:0/3305531399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6640102710 0x7f664010f080 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:12.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.694+0000 7f664700d700 1 -- 172.21.15.32:0/3305531399 >> 172.21.15.32:0/3305531399 conn(0x7f6640072bc0 msgr2=0x7f66400faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:56:12.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.694+0000 7f664700d700 1 -- 172.21.15.32:0/3305531399 shutdown_connections 2024-04-12T05:56:12.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.694+0000 7f664700d700 1 -- 172.21.15.32:0/3305531399 wait complete. 2024-04-12T05:56:12.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.797+0000 7f33cda77700 1 -- 172.21.15.32:0/3613857441 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33c80fb4a0 msgr2=0x7f33c80fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:12.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.797+0000 7f33cda77700 1 --2- 172.21.15.32:0/3613857441 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33c80fb4a0 0x7f33c80fb910 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f33bc004940 tx=0x7f33bc009e30 comp rx=0 tx=0).stop 2024-04-12T05:56:12.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.797+0000 7f33cda77700 1 -- 172.21.15.32:0/3613857441 shutdown_connections 2024-04-12T05:56:12.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.797+0000 7f33cda77700 1 --2- 172.21.15.32:0/3613857441 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33c80fb4a0 0x7f33c80fb910 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:12.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.797+0000 7f33cda77700 1 --2- 172.21.15.32:0/3613857441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33c8102e70 0x7f33c8103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:12.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.797+0000 7f33cda77700 1 -- 172.21.15.32:0/3613857441 >> 172.21.15.32:0/3613857441 conn(0x7f33c80714c0 msgr2=0x7f33c80718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:56:12.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.797+0000 7f33cda77700 1 -- 172.21.15.32:0/3613857441 shutdown_connections 2024-04-12T05:56:12.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.797+0000 7f33cda77700 1 -- 172.21.15.32:0/3613857441 wait complete. 2024-04-12T05:56:12.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.797+0000 7f33cda77700 1 Processor -- start 2024-04-12T05:56:12.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.797+0000 7f33cda77700 1 -- start start 2024-04-12T05:56:12.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.798+0000 7f33cda77700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33c80fb4a0 0x7f33c8113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:12.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.798+0000 7f33cda77700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33c8102e70 0x7f33c8113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:12.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.798+0000 7f33cda77700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c8114600 con 0x7f33c80fb4a0 2024-04-12T05:56:12.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.798+0000 7f33cda77700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c810d9f0 con 0x7f33c8102e70 2024-04-12T05:56:12.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.798+0000 7f33c77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33c80fb4a0 0x7f33c8113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:12.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.798+0000 7f33c77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33c80fb4a0 0x7f33c8113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57774/0 (socket says 172.21.15.32:57774) 2024-04-12T05:56:12.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.798+0000 7f33c77fe700 1 -- 172.21.15.32:0/2209504501 learned_addr learned my addr 172.21.15.32:0/2209504501 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:56:12.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.798+0000 7f33c6ffd700 1 --2- 172.21.15.32:0/2209504501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33c8102e70 0x7f33c8113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:12.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.798+0000 7f33c77fe700 1 -- 172.21.15.32:0/2209504501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33c8102e70 msgr2=0x7f33c8113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:12.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.798+0000 7f33c77fe700 1 --2- 172.21.15.32:0/2209504501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33c8102e70 0x7f33c8113fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:12.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.798+0000 7f33c77fe700 1 -- 172.21.15.32:0/2209504501 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33bc005040 con 0x7f33c80fb4a0 2024-04-12T05:56:12.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.798+0000 7f33c77fe700 1 --2- 172.21.15.32:0/2209504501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33c80fb4a0 0x7f33c8113a70 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f33b800c830 tx=0x7f33b8004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:56:12.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.798+0000 7f33affff700 1 -- 172.21.15.32:0/2209504501 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33b80106f0 con 0x7f33c80fb4a0 2024-04-12T05:56:12.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.798+0000 7f33cda77700 1 -- 172.21.15.32:0/2209504501 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33c810dcd0 con 0x7f33c80fb4a0 2024-04-12T05:56:12.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.798+0000 7f33affff700 1 -- 172.21.15.32:0/2209504501 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f33b8010d00 con 0x7f33c80fb4a0 2024-04-12T05:56:12.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.798+0000 7f33affff700 1 -- 172.21.15.32:0/2209504501 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33b8012e90 con 0x7f33c80fb4a0 2024-04-12T05:56:12.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.799+0000 7f33cda77700 1 -- 172.21.15.32:0/2209504501 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33c810e260 con 0x7f33c80fb4a0 2024-04-12T05:56:12.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.799+0000 7f33cda77700 1 -- 172.21.15.32:0/2209504501 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33c8100110 con 0x7f33c80fb4a0 2024-04-12T05:56:12.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.800+0000 7f33affff700 1 -- 172.21.15.32:0/2209504501 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f33b8010850 con 0x7f33c80fb4a0 2024-04-12T05:56:12.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.800+0000 7f33affff700 1 --2- 172.21.15.32:0/2209504501 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f33b006cc20 0x7f33b006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:12.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.801+0000 7f33affff700 1 -- 172.21.15.32:0/2209504501 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f33b808bfe0 con 0x7f33c80fb4a0 2024-04-12T05:56:12.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.801+0000 7f33c6ffd700 1 --2- 172.21.15.32:0/2209504501 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f33b006cc20 0x7f33b006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:12.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.801+0000 7f33c6ffd700 1 --2- 172.21.15.32:0/2209504501 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f33b006cc20 0x7f33b006f0d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f33c810f3a0 tx=0x7f33bc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:56:12.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:12.805+0000 7f33affff700 1 -- 172.21.15.32:0/2209504501 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f33b8059af0 con 0x7f33c80fb4a0 2024-04-12T05:56:13.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.091+0000 7f33cda77700 1 -- 172.21.15.32:0/2209504501 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f33c810f0c0 con 0x7f33c80fb4a0 2024-04-12T05:56:13.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.092+0000 7f33affff700 1 -- 172.21.15.32:0/2209504501 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f33b8059680 con 0x7f33c80fb4a0 2024-04-12T05:56:13.087 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:56:13.087 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:56:13.087 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:56:13.087 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:56:13.087 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:56:13.087 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:56:13.087 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:56:13.087 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:56:13.087 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:56:13.087 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:56:13.087 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:56:13.087 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:56:13.087 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:56:13.088 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:56:13.088 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:56:13.088 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:56:13.088 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:56:13.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.094+0000 7f33cda77700 1 -- 172.21.15.32:0/2209504501 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f33b006cc20 msgr2=0x7f33b006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:13.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.094+0000 7f33cda77700 1 --2- 172.21.15.32:0/2209504501 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f33b006cc20 0x7f33b006f0d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f33c810f3a0 tx=0x7f33bc009e30 comp rx=0 tx=0).stop 2024-04-12T05:56:13.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.094+0000 7f33cda77700 1 -- 172.21.15.32:0/2209504501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33c80fb4a0 msgr2=0x7f33c8113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:13.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.094+0000 7f33cda77700 1 --2- 172.21.15.32:0/2209504501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33c80fb4a0 0x7f33c8113a70 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f33b800c830 tx=0x7f33b8004470 comp rx=0 tx=0).stop 2024-04-12T05:56:13.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.094+0000 7f33cda77700 1 -- 172.21.15.32:0/2209504501 shutdown_connections 2024-04-12T05:56:13.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.094+0000 7f33cda77700 1 --2- 172.21.15.32:0/2209504501 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f33b006cc20 0x7f33b006f0d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:13.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.094+0000 7f33cda77700 1 --2- 172.21.15.32:0/2209504501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33c80fb4a0 0x7f33c8113a70 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:13.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.094+0000 7f33cda77700 1 --2- 172.21.15.32:0/2209504501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33c8102e70 0x7f33c8113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:13.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.094+0000 7f33cda77700 1 -- 172.21.15.32:0/2209504501 >> 172.21.15.32:0/2209504501 conn(0x7f33c80714c0 msgr2=0x7f33c80fa910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:56:13.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.094+0000 7f33cda77700 1 -- 172.21.15.32:0/2209504501 shutdown_connections 2024-04-12T05:56:13.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.094+0000 7f33cda77700 1 -- 172.21.15.32:0/2209504501 wait complete. 2024-04-12T05:56:13.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.200+0000 7f98d734d700 1 -- 172.21.15.32:0/3094256083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98d0101d70 msgr2=0x7f98d0102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:13.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.200+0000 7f98d734d700 1 --2- 172.21.15.32:0/3094256083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98d0101d70 0x7f98d0102140 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f98cc004940 tx=0x7f98cc009e30 comp rx=0 tx=0).stop 2024-04-12T05:56:13.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.200+0000 7f98d734d700 1 -- 172.21.15.32:0/3094256083 shutdown_connections 2024-04-12T05:56:13.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.200+0000 7f98d734d700 1 --2- 172.21.15.32:0/3094256083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98d0102710 0x7f98d00fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:13.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.200+0000 7f98d734d700 1 --2- 172.21.15.32:0/3094256083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98d0101d70 0x7f98d0102140 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:13.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.200+0000 7f98d734d700 1 -- 172.21.15.32:0/3094256083 >> 172.21.15.32:0/3094256083 conn(0x7f98d0072bc0 msgr2=0x7f98d0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:56:13.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.200+0000 7f98d734d700 1 -- 172.21.15.32:0/3094256083 shutdown_connections 2024-04-12T05:56:13.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.200+0000 7f98d734d700 1 -- 172.21.15.32:0/3094256083 wait complete. 2024-04-12T05:56:13.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.200+0000 7f98d734d700 1 Processor -- start 2024-04-12T05:56:13.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.200+0000 7f98d734d700 1 -- start start 2024-04-12T05:56:13.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.200+0000 7f98d734d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98d0101d70 0x7f98d0102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:13.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.201+0000 7f98d734d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98d0102710 0x7f98d0103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:13.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.201+0000 7f98d734d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d01036a0 con 0x7f98d0102710 2024-04-12T05:56:13.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.201+0000 7f98d734d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d01037e0 con 0x7f98d0101d70 2024-04-12T05:56:13.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.201+0000 7f98d48e8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98d0102710 0x7f98d0103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:13.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.201+0000 7f98d48e8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98d0102710 0x7f98d0103110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57784/0 (socket says 172.21.15.32:57784) 2024-04-12T05:56:13.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.201+0000 7f98d48e8700 1 -- 172.21.15.32:0/3182340152 learned_addr learned my addr 172.21.15.32:0/3182340152 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:56:13.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.201+0000 7f98d50e9700 1 --2- 172.21.15.32:0/3182340152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98d0101d70 0x7f98d0102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:13.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.201+0000 7f98d48e8700 1 -- 172.21.15.32:0/3182340152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98d0101d70 msgr2=0x7f98d0102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:13.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.201+0000 7f98d48e8700 1 --2- 172.21.15.32:0/3182340152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98d0101d70 0x7f98d0102bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:13.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.201+0000 7f98d48e8700 1 -- 172.21.15.32:0/3182340152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98cc005040 con 0x7f98d0102710 2024-04-12T05:56:13.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.201+0000 7f98d48e8700 1 --2- 172.21.15.32:0/3182340152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98d0102710 0x7f98d0103110 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f98c400c890 tx=0x7f98c40044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:56:13.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.201+0000 7f98c1ffb700 1 -- 172.21.15.32:0/3182340152 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98c400a7d0 con 0x7f98d0102710 2024-04-12T05:56:13.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.201+0000 7f98c1ffb700 1 -- 172.21.15.32:0/3182340152 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f98c400ade0 con 0x7f98d0102710 2024-04-12T05:56:13.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.201+0000 7f98d734d700 1 -- 172.21.15.32:0/3182340152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d0119d60 con 0x7f98d0102710 2024-04-12T05:56:13.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.201+0000 7f98c1ffb700 1 -- 172.21.15.32:0/3182340152 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98c40113a0 con 0x7f98d0102710 2024-04-12T05:56:13.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.202+0000 7f98d734d700 1 -- 172.21.15.32:0/3182340152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d011a1e0 con 0x7f98d0102710 2024-04-12T05:56:13.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.202+0000 7f98d734d700 1 -- 172.21.15.32:0/3182340152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d00fd4c0 con 0x7f98d0102710 2024-04-12T05:56:13.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.203+0000 7f98c1ffb700 1 -- 172.21.15.32:0/3182340152 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f98c4012400 con 0x7f98d0102710 2024-04-12T05:56:13.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.203+0000 7f98c1ffb700 1 --2- 172.21.15.32:0/3182340152 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f98bc06cc30 0x7f98bc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:13.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.203+0000 7f98c1ffb700 1 -- 172.21.15.32:0/3182340152 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f98c408c060 con 0x7f98d0102710 2024-04-12T05:56:13.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.203+0000 7f98d50e9700 1 --2- 172.21.15.32:0/3182340152 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f98bc06cc30 0x7f98bc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:13.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.204+0000 7f98d50e9700 1 --2- 172.21.15.32:0/3182340152 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f98bc06cc30 0x7f98bc06f0e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f98cc00b020 tx=0x7f98cc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:56:13.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.208+0000 7f98c1ffb700 1 -- 172.21.15.32:0/3182340152 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f98c40560e0 con 0x7f98d0102710 2024-04-12T05:56:13.394 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:13 smithi032 ceph-mon[146817]: pgmap v471: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:13.394 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:13 smithi032 ceph-mon[146817]: from='client.34900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:56:13.394 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:13 smithi032 ceph-mon[146817]: from='client.34904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:56:13.394 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:13 smithi032 ceph-mon[146817]: from='client.34908 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:56:13.394 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:13 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/2209504501' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:56:13.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.457+0000 7f98d734d700 1 -- 172.21.15.32:0/3182340152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f98d004bed0 con 0x7f98d0102710 2024-04-12T05:56:13.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.457+0000 7f98c1ffb700 1 -- 172.21.15.32:0/3182340152 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1835 (secure 0 0 0) 0x7f98c4059700 con 0x7f98d0102710 2024-04-12T05:56:13.452 INFO:teuthology.orchestra.run.smithi032.stdout:e33 2024-04-12T05:56:13.452 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:56:13.452 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:56:13.452 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:56:13.452 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:56:13.452 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:56:13.452 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:56:13.452 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 24 2024-04-12T05:56:13.452 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:56:13.452 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:56:13.453 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:46:10.668569+0000 2024-04-12T05:56:13.453 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:56:13.453 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:56:13.453 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:56:13.453 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:56:13.453 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:56:13.453 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:56:13.453 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:56:13.453 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:56:13.453 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T05:56:13.453 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:56:13.453 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:56:13.453 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:56:13.453 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T05:56:13.453 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-12T05:56:13.454 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:56:13.454 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:56:13.454 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:56:13.454 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:56:13.454 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:56:13.454 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:56:13.454 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:56:13.454 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:56:13.454 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:56:13.454 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:56:13.454 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-12T05:56:13.454 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:56:13.455 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:56:13.455 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:56:13.455 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{-1:44317} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:56:13.455 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:56:13.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.459+0000 7f98d734d700 1 -- 172.21.15.32:0/3182340152 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f98bc06cc30 msgr2=0x7f98bc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:13.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.459+0000 7f98d734d700 1 --2- 172.21.15.32:0/3182340152 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f98bc06cc30 0x7f98bc06f0e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f98cc00b020 tx=0x7f98cc009e30 comp rx=0 tx=0).stop 2024-04-12T05:56:13.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.459+0000 7f98d734d700 1 -- 172.21.15.32:0/3182340152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98d0102710 msgr2=0x7f98d0103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:13.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.459+0000 7f98d734d700 1 --2- 172.21.15.32:0/3182340152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98d0102710 0x7f98d0103110 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f98c400c890 tx=0x7f98c40044c0 comp rx=0 tx=0).stop 2024-04-12T05:56:13.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.459+0000 7f98d734d700 1 -- 172.21.15.32:0/3182340152 shutdown_connections 2024-04-12T05:56:13.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.459+0000 7f98d734d700 1 --2- 172.21.15.32:0/3182340152 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f98bc06cc30 0x7f98bc06f0e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:13.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.459+0000 7f98d734d700 1 --2- 172.21.15.32:0/3182340152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98d0101d70 0x7f98d0102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:13.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.459+0000 7f98d734d700 1 --2- 172.21.15.32:0/3182340152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98d0102710 0x7f98d0103110 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:13.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.459+0000 7f98d734d700 1 -- 172.21.15.32:0/3182340152 >> 172.21.15.32:0/3182340152 conn(0x7f98d0072bc0 msgr2=0x7f98d00659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:56:13.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.459+0000 7f98d734d700 1 -- 172.21.15.32:0/3182340152 shutdown_connections 2024-04-12T05:56:13.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.459+0000 7f98d734d700 1 -- 172.21.15.32:0/3182340152 wait complete. 2024-04-12T05:56:13.457 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-12T05:56:13.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:13 smithi067 ceph-mon[112979]: pgmap v471: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:13.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:13 smithi067 ceph-mon[112979]: from='client.34900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:56:13.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:13 smithi067 ceph-mon[112979]: from='client.34904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:56:13.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:13 smithi067 ceph-mon[112979]: from='client.34908 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:56:13.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:13 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/2209504501' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:56:13.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.566+0000 7fb96dd16700 1 -- 172.21.15.32:0/4238691447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9681026b0 msgr2=0x7fb9680fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:13.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.566+0000 7fb96dd16700 1 --2- 172.21.15.32:0/4238691447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9681026b0 0x7fb9680fbd20 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb958004940 tx=0x7fb958009e30 comp rx=0 tx=0).stop 2024-04-12T05:56:13.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.566+0000 7fb96dd16700 1 -- 172.21.15.32:0/4238691447 shutdown_connections 2024-04-12T05:56:13.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.566+0000 7fb96dd16700 1 --2- 172.21.15.32:0/4238691447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9681026b0 0x7fb9680fbd20 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:13.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.566+0000 7fb96dd16700 1 --2- 172.21.15.32:0/4238691447 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb968101d10 0x7fb9681020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:13.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.566+0000 7fb96dd16700 1 -- 172.21.15.32:0/4238691447 >> 172.21.15.32:0/4238691447 conn(0x7fb9680f7880 msgr2=0x7fb9680f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:56:13.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.566+0000 7fb96dd16700 1 -- 172.21.15.32:0/4238691447 shutdown_connections 2024-04-12T05:56:13.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.566+0000 7fb96dd16700 1 -- 172.21.15.32:0/4238691447 wait complete. 2024-04-12T05:56:13.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.566+0000 7fb96dd16700 1 Processor -- start 2024-04-12T05:56:13.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.567+0000 7fb96dd16700 1 -- start start 2024-04-12T05:56:13.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.567+0000 7fb96dd16700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb968101d10 0x7fb968115ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:13.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.567+0000 7fb96dd16700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9681026b0 0x7fb968116530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:13.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.567+0000 7fb96dd16700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb968116b80 con 0x7fb9681026b0 2024-04-12T05:56:13.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.567+0000 7fb96dd16700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb96810ff50 con 0x7fb968101d10 2024-04-12T05:56:13.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.567+0000 7fb966ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9681026b0 0x7fb968116530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:13.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.567+0000 7fb966ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9681026b0 0x7fb968116530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57798/0 (socket says 172.21.15.32:57798) 2024-04-12T05:56:13.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.567+0000 7fb966ffd700 1 -- 172.21.15.32:0/2011127651 learned_addr learned my addr 172.21.15.32:0/2011127651 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:56:13.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.567+0000 7fb9677fe700 1 --2- 172.21.15.32:0/2011127651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb968101d10 0x7fb968115ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:13.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.567+0000 7fb966ffd700 1 -- 172.21.15.32:0/2011127651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb968101d10 msgr2=0x7fb968115ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:13.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.567+0000 7fb966ffd700 1 --2- 172.21.15.32:0/2011127651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb968101d10 0x7fb968115ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:13.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.567+0000 7fb966ffd700 1 -- 172.21.15.32:0/2011127651 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb958005040 con 0x7fb9681026b0 2024-04-12T05:56:13.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.567+0000 7fb966ffd700 1 --2- 172.21.15.32:0/2011127651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9681026b0 0x7fb968116530 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb958009380 tx=0x7fb958009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:56:13.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.568+0000 7fb94ffff700 1 -- 172.21.15.32:0/2011127651 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb95800e040 con 0x7fb9681026b0 2024-04-12T05:56:13.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.568+0000 7fb94ffff700 1 -- 172.21.15.32:0/2011127651 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb958013420 con 0x7fb9681026b0 2024-04-12T05:56:13.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.568+0000 7fb96dd16700 1 -- 172.21.15.32:0/2011127651 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9681101d0 con 0x7fb9681026b0 2024-04-12T05:56:13.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.568+0000 7fb94ffff700 1 -- 172.21.15.32:0/2011127651 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb95801c630 con 0x7fb9681026b0 2024-04-12T05:56:13.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.568+0000 7fb96dd16700 1 -- 172.21.15.32:0/2011127651 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb968110790 con 0x7fb9681026b0 2024-04-12T05:56:13.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.569+0000 7fb96dd16700 1 -- 172.21.15.32:0/2011127651 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9680fd460 con 0x7fb9681026b0 2024-04-12T05:56:13.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.569+0000 7fb94ffff700 1 -- 172.21.15.32:0/2011127651 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb9580139b0 con 0x7fb9681026b0 2024-04-12T05:56:13.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.570+0000 7fb94ffff700 1 --2- 172.21.15.32:0/2011127651 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb95006cc20 0x7fb95006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:13.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.570+0000 7fb94ffff700 1 -- 172.21.15.32:0/2011127651 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fb95808d140 con 0x7fb9681026b0 2024-04-12T05:56:13.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.570+0000 7fb9677fe700 1 --2- 172.21.15.32:0/2011127651 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb95006cc20 0x7fb95006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:13.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.570+0000 7fb9677fe700 1 --2- 172.21.15.32:0/2011127651 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb95006cc20 0x7fb95006f0d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb95c004f40 tx=0x7fb95c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:56:13.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.574+0000 7fb94ffff700 1 -- 172.21.15.32:0/2011127651 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb95805ac50 con 0x7fb9681026b0 2024-04-12T05:56:13.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.797+0000 7fb96dd16700 1 -- 172.21.15.32:0/2011127651 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb968108d20 con 0x7fb95006cc20 2024-04-12T05:56:13.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.799+0000 7fb94ffff700 1 -- 172.21.15.32:0/2011127651 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb968108d20 con 0x7fb95006cc20 2024-04-12T05:56:13.793 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:56:13.793 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:56:13.793 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:56:13.793 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:56:13.793 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:56:13.793 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-12T05:56:13.794 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:56:13.794 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-12T05:56:13.794 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:56:13.794 INFO:teuthology.orchestra.run.smithi032.stdout: "mds" 2024-04-12T05:56:13.794 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:56:13.794 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:56:13.794 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:56:13.794 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:56:13.794 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:56:13.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.801+0000 7fb96dd16700 1 -- 172.21.15.32:0/2011127651 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb95006cc20 msgr2=0x7fb95006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:13.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.801+0000 7fb96dd16700 1 --2- 172.21.15.32:0/2011127651 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb95006cc20 0x7fb95006f0d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb95c004f40 tx=0x7fb95c00b040 comp rx=0 tx=0).stop 2024-04-12T05:56:13.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.801+0000 7fb96dd16700 1 -- 172.21.15.32:0/2011127651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9681026b0 msgr2=0x7fb968116530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:13.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.801+0000 7fb96dd16700 1 --2- 172.21.15.32:0/2011127651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9681026b0 0x7fb968116530 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb958009380 tx=0x7fb958009e30 comp rx=0 tx=0).stop 2024-04-12T05:56:13.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.801+0000 7fb96dd16700 1 -- 172.21.15.32:0/2011127651 shutdown_connections 2024-04-12T05:56:13.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.801+0000 7fb96dd16700 1 --2- 172.21.15.32:0/2011127651 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb95006cc20 0x7fb95006f0d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:13.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.801+0000 7fb96dd16700 1 --2- 172.21.15.32:0/2011127651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb968101d10 0x7fb968115ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:13.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.801+0000 7fb96dd16700 1 --2- 172.21.15.32:0/2011127651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9681026b0 0x7fb968116530 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:13.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.801+0000 7fb96dd16700 1 -- 172.21.15.32:0/2011127651 >> 172.21.15.32:0/2011127651 conn(0x7fb9680f7880 msgr2=0x7fb9680f9060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:56:13.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.801+0000 7fb96dd16700 1 -- 172.21.15.32:0/2011127651 shutdown_connections 2024-04-12T05:56:13.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.801+0000 7fb96dd16700 1 -- 172.21.15.32:0/2011127651 wait complete. 2024-04-12T05:56:13.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.915+0000 7fda9bd33700 1 -- 172.21.15.32:0/3098273171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda94102e10 msgr2=0x7fda941031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:13.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.915+0000 7fda9bd33700 1 --2- 172.21.15.32:0/3098273171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda94102e10 0x7fda941031e0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fda90004940 tx=0x7fda90009e30 comp rx=0 tx=0).stop 2024-04-12T05:56:13.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.915+0000 7fda9bd33700 1 -- 172.21.15.32:0/3098273171 shutdown_connections 2024-04-12T05:56:13.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.915+0000 7fda9bd33700 1 --2- 172.21.15.32:0/3098273171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda941037b0 0x7fda940fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:13.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.915+0000 7fda9bd33700 1 --2- 172.21.15.32:0/3098273171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda94102e10 0x7fda941031e0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:13.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.915+0000 7fda9bd33700 1 -- 172.21.15.32:0/3098273171 >> 172.21.15.32:0/3098273171 conn(0x7fda940f89a0 msgr2=0x7fda940fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:56:13.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.915+0000 7fda9bd33700 1 -- 172.21.15.32:0/3098273171 shutdown_connections 2024-04-12T05:56:13.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.915+0000 7fda9bd33700 1 -- 172.21.15.32:0/3098273171 wait complete. 2024-04-12T05:56:13.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.916+0000 7fda9bd33700 1 Processor -- start 2024-04-12T05:56:13.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.916+0000 7fda9bd33700 1 -- start start 2024-04-12T05:56:13.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.916+0000 7fda9bd33700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda94102e10 0x7fda94113b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:13.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.916+0000 7fda9bd33700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda941037b0 0x7fda94114050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:13.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.916+0000 7fda9bd33700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda941146a0 con 0x7fda94102e10 2024-04-12T05:56:13.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.916+0000 7fda9bd33700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda9410da90 con 0x7fda941037b0 2024-04-12T05:56:13.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.916+0000 7fda99acf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda94102e10 0x7fda94113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:13.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.916+0000 7fda99acf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda94102e10 0x7fda94113b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57828/0 (socket says 172.21.15.32:57828) 2024-04-12T05:56:13.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.916+0000 7fda99acf700 1 -- 172.21.15.32:0/189086680 learned_addr learned my addr 172.21.15.32:0/189086680 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:56:13.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.916+0000 7fda99acf700 1 -- 172.21.15.32:0/189086680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda941037b0 msgr2=0x7fda94114050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:13.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.916+0000 7fda99acf700 1 --2- 172.21.15.32:0/189086680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda941037b0 0x7fda94114050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:13.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.916+0000 7fda99acf700 1 -- 172.21.15.32:0/189086680 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda90005040 con 0x7fda94102e10 2024-04-12T05:56:13.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.916+0000 7fda99acf700 1 --2- 172.21.15.32:0/189086680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda94102e10 0x7fda94113b10 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fda9000b3b0 tx=0x7fda90009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:56:13.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.917+0000 7fda867fc700 1 -- 172.21.15.32:0/189086680 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda9000e040 con 0x7fda94102e10 2024-04-12T05:56:13.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.917+0000 7fda867fc700 1 -- 172.21.15.32:0/189086680 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda90013420 con 0x7fda94102e10 2024-04-12T05:56:13.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.917+0000 7fda9bd33700 1 -- 172.21.15.32:0/189086680 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda9410dd10 con 0x7fda94102e10 2024-04-12T05:56:13.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.917+0000 7fda867fc700 1 -- 172.21.15.32:0/189086680 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda9001c630 con 0x7fda94102e10 2024-04-12T05:56:13.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.917+0000 7fda9bd33700 1 -- 172.21.15.32:0/189086680 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda9410e2d0 con 0x7fda94102e10 2024-04-12T05:56:13.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.918+0000 7fda9bd33700 1 -- 172.21.15.32:0/189086680 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda940fe560 con 0x7fda94102e10 2024-04-12T05:56:13.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.918+0000 7fda867fc700 1 -- 172.21.15.32:0/189086680 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fda90013970 con 0x7fda94102e10 2024-04-12T05:56:13.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.919+0000 7fda867fc700 1 --2- 172.21.15.32:0/189086680 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fda8006cc20 0x7fda8006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:13.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.919+0000 7fda867fc700 1 -- 172.21.15.32:0/189086680 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fda9008d200 con 0x7fda94102e10 2024-04-12T05:56:13.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.919+0000 7fda992ce700 1 --2- 172.21.15.32:0/189086680 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fda8006cc20 0x7fda8006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:13.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.919+0000 7fda992ce700 1 --2- 172.21.15.32:0/189086680 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fda8006cc20 0x7fda8006f0d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fda9410ec50 tx=0x7fda88009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:56:13.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:13.923+0000 7fda867fc700 1 -- 172.21.15.32:0/189086680 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fda9005ad10 con 0x7fda94102e10 2024-04-12T05:56:14.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:14.210+0000 7fda9bd33700 1 -- 172.21.15.32:0/189086680 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fda9404bed0 con 0x7fda94102e10 2024-04-12T05:56:14.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:14.210+0000 7fda867fc700 1 -- 172.21.15.32:0/189086680 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fda9005a8a0 con 0x7fda94102e10 2024-04-12T05:56:14.205 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:56:14.205 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:56:14.205 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-12T05:56:14.205 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:56:14.205 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:56:14.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:14.212+0000 7fda9bd33700 1 -- 172.21.15.32:0/189086680 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fda8006cc20 msgr2=0x7fda8006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:14.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:14.212+0000 7fda9bd33700 1 --2- 172.21.15.32:0/189086680 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fda8006cc20 0x7fda8006f0d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fda9410ec50 tx=0x7fda88009ce0 comp rx=0 tx=0).stop 2024-04-12T05:56:14.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:14.212+0000 7fda9bd33700 1 -- 172.21.15.32:0/189086680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda94102e10 msgr2=0x7fda94113b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:14.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:14.212+0000 7fda9bd33700 1 --2- 172.21.15.32:0/189086680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda94102e10 0x7fda94113b10 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fda9000b3b0 tx=0x7fda90009e30 comp rx=0 tx=0).stop 2024-04-12T05:56:14.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:14.212+0000 7fda9bd33700 1 -- 172.21.15.32:0/189086680 shutdown_connections 2024-04-12T05:56:14.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:14.212+0000 7fda9bd33700 1 --2- 172.21.15.32:0/189086680 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fda8006cc20 0x7fda8006f0d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:14.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:14.212+0000 7fda9bd33700 1 --2- 172.21.15.32:0/189086680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda94102e10 0x7fda94113b10 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:14.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:14.212+0000 7fda9bd33700 1 --2- 172.21.15.32:0/189086680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda941037b0 0x7fda94114050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:14.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:14.212+0000 7fda9bd33700 1 -- 172.21.15.32:0/189086680 >> 172.21.15.32:0/189086680 conn(0x7fda940f89a0 msgr2=0x7fda940f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:56:14.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:14.212+0000 7fda9bd33700 1 -- 172.21.15.32:0/189086680 shutdown_connections 2024-04-12T05:56:14.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:14.212+0000 7fda9bd33700 1 -- 172.21.15.32:0/189086680 wait complete. 2024-04-12T05:56:14.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:14 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3182340152' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:56:14.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:14 smithi067 ceph-mon[112979]: from='client.34920 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:56:14.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:14 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3182340152' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:56:14.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:14 smithi032 ceph-mon[146817]: from='client.34920 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:56:15.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:15 smithi067 ceph-mon[112979]: pgmap v472: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:15.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:15 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/189086680' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:56:15.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:15 smithi032 ceph-mon[146817]: pgmap v472: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:15.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:15 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/189086680' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:56:17.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:17 smithi067 ceph-mon[112979]: pgmap v473: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:17.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:17 smithi032 ceph-mon[146817]: pgmap v473: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:18.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:56:18.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:56:19.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:19 smithi067 ceph-mon[112979]: pgmap v474: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:19.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:19 smithi032 ceph-mon[146817]: pgmap v474: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:21.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:21 smithi067 ceph-mon[112979]: pgmap v475: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:21.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:21 smithi032 ceph-mon[146817]: pgmap v475: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:23.394 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:23 smithi032 ceph-mon[146817]: pgmap v476: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:23.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:23 smithi067 ceph-mon[112979]: pgmap v476: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:25.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:25 smithi067 ceph-mon[112979]: pgmap v477: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:25.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:25 smithi032 ceph-mon[146817]: pgmap v477: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:27.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:27 smithi067 ceph-mon[112979]: pgmap v478: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:27.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:27 smithi032 ceph-mon[146817]: pgmap v478: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:29.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:29 smithi067 ceph-mon[112979]: pgmap v479: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:29.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:29 smithi032 ceph-mon[146817]: pgmap v479: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:31.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:31 smithi067 ceph-mon[112979]: pgmap v480: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:31.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:31 smithi032 ceph-mon[146817]: pgmap v480: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:33.393 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:33 smithi032 ceph-mon[146817]: pgmap v481: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:33.394 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:33 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:56:33.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:33 smithi067 ceph-mon[112979]: pgmap v481: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:33.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:33 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:56:35.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:35 smithi067 ceph-mon[112979]: pgmap v482: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:35.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:35 smithi032 ceph-mon[146817]: pgmap v482: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:37.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:37 smithi067 ceph-mon[112979]: pgmap v483: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:37.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:37 smithi032 ceph-mon[146817]: pgmap v483: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:39.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:39 smithi067 ceph-mon[112979]: pgmap v484: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:39.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:39 smithi032 ceph-mon[146817]: pgmap v484: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:41.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:41 smithi067 ceph-mon[112979]: pgmap v485: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:41.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:41 smithi032 ceph-mon[146817]: pgmap v485: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:43.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:43 smithi067 ceph-mon[112979]: pgmap v486: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:43.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:43 smithi032 ceph-mon[146817]: pgmap v486: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:44.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.330+0000 7f6f9415a700 1 -- 172.21.15.32:0/863897896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f8c0fe830 msgr2=0x7f6f8c106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:44.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.330+0000 7f6f9415a700 1 --2- 172.21.15.32:0/863897896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f8c0fe830 0x7f6f8c106f00 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f6f88004940 tx=0x7f6f88009e30 comp rx=0 tx=0).stop 2024-04-12T05:56:44.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.330+0000 7f6f9415a700 1 -- 172.21.15.32:0/863897896 shutdown_connections 2024-04-12T05:56:44.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.330+0000 7f6f9415a700 1 --2- 172.21.15.32:0/863897896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f8c0fe830 0x7f6f8c106f00 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:44.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.330+0000 7f6f9415a700 1 --2- 172.21.15.32:0/863897896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f8c0fde90 0x7f6f8c0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:44.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.330+0000 7f6f9415a700 1 -- 172.21.15.32:0/863897896 >> 172.21.15.32:0/863897896 conn(0x7f6f8c072bc0 msgr2=0x7f6f8c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:56:44.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.330+0000 7f6f9415a700 1 -- 172.21.15.32:0/863897896 shutdown_connections 2024-04-12T05:56:44.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.330+0000 7f6f9415a700 1 -- 172.21.15.32:0/863897896 wait complete. 2024-04-12T05:56:44.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.331+0000 7f6f9415a700 1 Processor -- start 2024-04-12T05:56:44.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.331+0000 7f6f9415a700 1 -- start start 2024-04-12T05:56:44.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.331+0000 7f6f9415a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f8c0fde90 0x7f6f8c115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:44.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.331+0000 7f6f9415a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f8c0fe830 0x7f6f8c1161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:44.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.331+0000 7f6f9415a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8c116800 con 0x7f6f8c0fe830 2024-04-12T05:56:44.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.331+0000 7f6f9415a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8c10fbd0 con 0x7f6f8c0fde90 2024-04-12T05:56:44.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.331+0000 7f6f916f5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f8c0fe830 0x7f6f8c1161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:44.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.331+0000 7f6f916f5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f8c0fe830 0x7f6f8c1161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43098/0 (socket says 172.21.15.32:43098) 2024-04-12T05:56:44.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.331+0000 7f6f916f5700 1 -- 172.21.15.32:0/2636727923 learned_addr learned my addr 172.21.15.32:0/2636727923 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:56:44.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.332+0000 7f6f91ef6700 1 --2- 172.21.15.32:0/2636727923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f8c0fde90 0x7f6f8c115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:44.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.332+0000 7f6f916f5700 1 -- 172.21.15.32:0/2636727923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f8c0fde90 msgr2=0x7f6f8c115c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:44.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.332+0000 7f6f916f5700 1 --2- 172.21.15.32:0/2636727923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f8c0fde90 0x7f6f8c115c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:44.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.332+0000 7f6f916f5700 1 -- 172.21.15.32:0/2636727923 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f88005040 con 0x7f6f8c0fe830 2024-04-12T05:56:44.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.332+0000 7f6f916f5700 1 --2- 172.21.15.32:0/2636727923 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f8c0fe830 0x7f6f8c1161b0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f6f88000c00 tx=0x7f6f88009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:56:44.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.332+0000 7f6f827fc700 1 -- 172.21.15.32:0/2636727923 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f88021070 con 0x7f6f8c0fe830 2024-04-12T05:56:44.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.332+0000 7f6f827fc700 1 -- 172.21.15.32:0/2636727923 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f88013e00 con 0x7f6f8c0fe830 2024-04-12T05:56:44.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.332+0000 7f6f9415a700 1 -- 172.21.15.32:0/2636727923 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f8c10fe50 con 0x7f6f8c0fe830 2024-04-12T05:56:44.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.332+0000 7f6f827fc700 1 -- 172.21.15.32:0/2636727923 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f8800e070 con 0x7f6f8c0fe830 2024-04-12T05:56:44.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.332+0000 7f6f9415a700 1 -- 172.21.15.32:0/2636727923 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f8c110390 con 0x7f6f8c0fe830 2024-04-12T05:56:44.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.333+0000 7f6f9415a700 1 -- 172.21.15.32:0/2636727923 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f8c106a20 con 0x7f6f8c0fe830 2024-04-12T05:56:44.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.334+0000 7f6f827fc700 1 -- 172.21.15.32:0/2636727923 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f6f88013950 con 0x7f6f8c0fe830 2024-04-12T05:56:44.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.334+0000 7f6f827fc700 1 --2- 172.21.15.32:0/2636727923 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6f7806cc30 0x7f6f7806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:44.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.334+0000 7f6f827fc700 1 -- 172.21.15.32:0/2636727923 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f6f8808d110 con 0x7f6f8c0fe830 2024-04-12T05:56:44.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.334+0000 7f6f91ef6700 1 --2- 172.21.15.32:0/2636727923 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6f7806cc30 0x7f6f7806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:44.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.335+0000 7f6f91ef6700 1 --2- 172.21.15.32:0/2636727923 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6f7806cc30 0x7f6f7806f0e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c004600 tx=0x7f6f7c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:56:44.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.339+0000 7f6f827fc700 1 -- 172.21.15.32:0/2636727923 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6f8805ab80 con 0x7f6f8c0fe830 2024-04-12T05:56:44.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.563+0000 7f6f9415a700 1 -- 172.21.15.32:0/2636727923 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f8c111310 con 0x7f6f7806cc30 2024-04-12T05:56:44.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.564+0000 7f6f827fc700 1 -- 172.21.15.32:0/2636727923 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6f8c111310 con 0x7f6f7806cc30 2024-04-12T05:56:44.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.566+0000 7f6f9415a700 1 -- 172.21.15.32:0/2636727923 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6f7806cc30 msgr2=0x7f6f7806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:44.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.566+0000 7f6f9415a700 1 --2- 172.21.15.32:0/2636727923 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6f7806cc30 0x7f6f7806f0e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c004600 tx=0x7f6f7c00b040 comp rx=0 tx=0).stop 2024-04-12T05:56:44.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.566+0000 7f6f9415a700 1 -- 172.21.15.32:0/2636727923 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f8c0fe830 msgr2=0x7f6f8c1161b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:44.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.566+0000 7f6f9415a700 1 --2- 172.21.15.32:0/2636727923 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f8c0fe830 0x7f6f8c1161b0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f6f88000c00 tx=0x7f6f88009e30 comp rx=0 tx=0).stop 2024-04-12T05:56:44.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.566+0000 7f6f9415a700 1 -- 172.21.15.32:0/2636727923 shutdown_connections 2024-04-12T05:56:44.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.566+0000 7f6f9415a700 1 --2- 172.21.15.32:0/2636727923 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6f7806cc30 0x7f6f7806f0e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:44.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.566+0000 7f6f9415a700 1 --2- 172.21.15.32:0/2636727923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f8c0fde90 0x7f6f8c115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:44.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.566+0000 7f6f9415a700 1 --2- 172.21.15.32:0/2636727923 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f8c0fe830 0x7f6f8c1161b0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:44.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.566+0000 7f6f9415a700 1 -- 172.21.15.32:0/2636727923 >> 172.21.15.32:0/2636727923 conn(0x7f6f8c072bc0 msgr2=0x7f6f8c0fb6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:56:44.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.566+0000 7f6f9415a700 1 -- 172.21.15.32:0/2636727923 shutdown_connections 2024-04-12T05:56:44.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.566+0000 7f6f9415a700 1 -- 172.21.15.32:0/2636727923 wait complete. 2024-04-12T05:56:44.571 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:56:44.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.670+0000 7f200387c700 1 -- 172.21.15.32:0/4015570639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ffc102e10 msgr2=0x7f1ffc1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:44.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.670+0000 7f200387c700 1 --2- 172.21.15.32:0/4015570639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ffc102e10 0x7f1ffc1031e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8004940 tx=0x7f1ff8009e30 comp rx=0 tx=0).stop 2024-04-12T05:56:44.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.670+0000 7f200387c700 1 -- 172.21.15.32:0/4015570639 shutdown_connections 2024-04-12T05:56:44.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.670+0000 7f200387c700 1 --2- 172.21.15.32:0/4015570639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ffc1037b0 0x7f1ffc0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:44.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.670+0000 7f200387c700 1 --2- 172.21.15.32:0/4015570639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ffc102e10 0x7f1ffc1031e0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:44.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.670+0000 7f200387c700 1 -- 172.21.15.32:0/4015570639 >> 172.21.15.32:0/4015570639 conn(0x7f1ffc0f89a0 msgr2=0x7f1ffc0fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:56:44.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.671+0000 7f200387c700 1 -- 172.21.15.32:0/4015570639 shutdown_connections 2024-04-12T05:56:44.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.671+0000 7f200387c700 1 -- 172.21.15.32:0/4015570639 wait complete. 2024-04-12T05:56:44.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.671+0000 7f200387c700 1 Processor -- start 2024-04-12T05:56:44.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.671+0000 7f200387c700 1 -- start start 2024-04-12T05:56:44.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.671+0000 7f200387c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ffc102e10 0x7f1ffc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:44.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.671+0000 7f200387c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ffc1037b0 0x7f1ffc070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:44.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.671+0000 7f200387c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc070d90 con 0x7f1ffc1037b0 2024-04-12T05:56:44.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.671+0000 7f200387c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc070ed0 con 0x7f1ffc102e10 2024-04-12T05:56:44.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.672+0000 7f2000e17700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ffc1037b0 0x7f1ffc070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.672+0000 7f2000e17700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ffc1037b0 0x7f1ffc070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43116/0 (socket says 172.21.15.32:43116) 2024-04-12T05:56:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.672+0000 7f2000e17700 1 -- 172.21.15.32:0/3078778345 learned_addr learned my addr 172.21.15.32:0/3078778345 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:56:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.672+0000 7f2001618700 1 --2- 172.21.15.32:0/3078778345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ffc102e10 0x7f1ffc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.672+0000 7f2000e17700 1 -- 172.21.15.32:0/3078778345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ffc102e10 msgr2=0x7f1ffc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.672+0000 7f2000e17700 1 --2- 172.21.15.32:0/3078778345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ffc102e10 0x7f1ffc0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.672+0000 7f2000e17700 1 -- 172.21.15.32:0/3078778345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff8005040 con 0x7f1ffc1037b0 2024-04-12T05:56:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.672+0000 7f2000e17700 1 --2- 172.21.15.32:0/3078778345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ffc1037b0 0x7f1ffc070800 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0002a50 tx=0x7f1ff0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:56:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.672+0000 7f1fedffb700 1 -- 172.21.15.32:0/3078778345 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ff0007cc0 con 0x7f1ffc1037b0 2024-04-12T05:56:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.672+0000 7f1fedffb700 1 -- 172.21.15.32:0/3078778345 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ff000f040 con 0x7f1ffc1037b0 2024-04-12T05:56:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.672+0000 7f1fedffb700 1 -- 172.21.15.32:0/3078778345 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ff000d920 con 0x7f1ffc1037b0 2024-04-12T05:56:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.672+0000 7f200387c700 1 -- 172.21.15.32:0/3078778345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ffc119df0 con 0x7f1ffc1037b0 2024-04-12T05:56:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.672+0000 7f200387c700 1 -- 172.21.15.32:0/3078778345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ffc11a270 con 0x7f1ffc1037b0 2024-04-12T05:56:44.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.673+0000 7f200387c700 1 -- 172.21.15.32:0/3078778345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ffc0fe560 con 0x7f1ffc1037b0 2024-04-12T05:56:44.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.674+0000 7f1fedffb700 1 -- 172.21.15.32:0/3078778345 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1ff0009680 con 0x7f1ffc1037b0 2024-04-12T05:56:44.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.674+0000 7f1fedffb700 1 --2- 172.21.15.32:0/3078778345 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1fe806cc20 0x7f1fe806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:44.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.674+0000 7f1fedffb700 1 -- 172.21.15.32:0/3078778345 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f1ff008bd50 con 0x7f1ffc1037b0 2024-04-12T05:56:44.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.674+0000 7f2001618700 1 --2- 172.21.15.32:0/3078778345 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1fe806cc20 0x7f1fe806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:44.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.675+0000 7f2001618700 1 --2- 172.21.15.32:0/3078778345 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1fe806cc20 0x7f1fe806f0d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8004f90 tx=0x7f1ff8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:56:44.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.679+0000 7f1fedffb700 1 -- 172.21.15.32:0/3078778345 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1ff0055d30 con 0x7f1ffc1037b0 2024-04-12T05:56:44.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.899+0000 7f200387c700 1 -- 172.21.15.32:0/3078778345 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ffc04bed0 con 0x7f1fe806cc20 2024-04-12T05:56:44.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.900+0000 7f1fedffb700 1 -- 172.21.15.32:0/3078778345 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f1ffc04bed0 con 0x7f1fe806cc20 2024-04-12T05:56:44.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.902+0000 7f200387c700 1 -- 172.21.15.32:0/3078778345 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1fe806cc20 msgr2=0x7f1fe806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:44.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.902+0000 7f200387c700 1 --2- 172.21.15.32:0/3078778345 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1fe806cc20 0x7f1fe806f0d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8004f90 tx=0x7f1ff8009e30 comp rx=0 tx=0).stop 2024-04-12T05:56:44.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.902+0000 7f200387c700 1 -- 172.21.15.32:0/3078778345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ffc1037b0 msgr2=0x7f1ffc070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:44.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.902+0000 7f200387c700 1 --2- 172.21.15.32:0/3078778345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ffc1037b0 0x7f1ffc070800 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0002a50 tx=0x7f1ff0004600 comp rx=0 tx=0).stop 2024-04-12T05:56:44.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.902+0000 7f200387c700 1 -- 172.21.15.32:0/3078778345 shutdown_connections 2024-04-12T05:56:44.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.902+0000 7f200387c700 1 --2- 172.21.15.32:0/3078778345 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1fe806cc20 0x7f1fe806f0d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:44.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.902+0000 7f200387c700 1 --2- 172.21.15.32:0/3078778345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ffc102e10 0x7f1ffc0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:44.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.902+0000 7f200387c700 1 --2- 172.21.15.32:0/3078778345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ffc1037b0 0x7f1ffc070800 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:44.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.902+0000 7f200387c700 1 -- 172.21.15.32:0/3078778345 >> 172.21.15.32:0/3078778345 conn(0x7f1ffc0f89a0 msgr2=0x7f1ffc0f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:56:44.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.902+0000 7f200387c700 1 -- 172.21.15.32:0/3078778345 shutdown_connections 2024-04-12T05:56:44.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:44.902+0000 7f200387c700 1 -- 172.21.15.32:0/3078778345 wait complete. 2024-04-12T05:56:45.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.005+0000 7fda75299700 1 -- 172.21.15.32:0/3002610495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda701037a0 msgr2=0x7fda700fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:45.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.005+0000 7fda75299700 1 --2- 172.21.15.32:0/3002610495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda701037a0 0x7fda700fce10 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fda64004940 tx=0x7fda64009e30 comp rx=0 tx=0).stop 2024-04-12T05:56:45.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.005+0000 7fda75299700 1 -- 172.21.15.32:0/3002610495 shutdown_connections 2024-04-12T05:56:45.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.005+0000 7fda75299700 1 --2- 172.21.15.32:0/3002610495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda701037a0 0x7fda700fce10 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:45.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.005+0000 7fda75299700 1 --2- 172.21.15.32:0/3002610495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda70102e00 0x7fda701031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:45.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.005+0000 7fda75299700 1 -- 172.21.15.32:0/3002610495 >> 172.21.15.32:0/3002610495 conn(0x7fda700f89b0 msgr2=0x7fda700fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:56:45.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.006+0000 7fda75299700 1 -- 172.21.15.32:0/3002610495 shutdown_connections 2024-04-12T05:56:45.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.006+0000 7fda75299700 1 -- 172.21.15.32:0/3002610495 wait complete. 2024-04-12T05:56:45.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.006+0000 7fda75299700 1 Processor -- start 2024-04-12T05:56:45.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.006+0000 7fda75299700 1 -- start start 2024-04-12T05:56:45.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.006+0000 7fda75299700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda70102e00 0x7fda70115ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:45.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.006+0000 7fda75299700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda701037a0 0x7fda70116000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:45.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.007+0000 7fda75299700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda70116650 con 0x7fda701037a0 2024-04-12T05:56:45.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.007+0000 7fda75299700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7010fc30 con 0x7fda70102e00 2024-04-12T05:56:45.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.007+0000 7fda6e7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda701037a0 0x7fda70116000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:45.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.007+0000 7fda6e7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda701037a0 0x7fda70116000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43132/0 (socket says 172.21.15.32:43132) 2024-04-12T05:56:45.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.007+0000 7fda6e7fc700 1 -- 172.21.15.32:0/4127868798 learned_addr learned my addr 172.21.15.32:0/4127868798 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:56:45.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.007+0000 7fda6effd700 1 --2- 172.21.15.32:0/4127868798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda70102e00 0x7fda70115ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:45.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.007+0000 7fda6e7fc700 1 -- 172.21.15.32:0/4127868798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda70102e00 msgr2=0x7fda70115ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:45.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.007+0000 7fda6e7fc700 1 --2- 172.21.15.32:0/4127868798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda70102e00 0x7fda70115ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:45.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.007+0000 7fda6e7fc700 1 -- 172.21.15.32:0/4127868798 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda64005040 con 0x7fda701037a0 2024-04-12T05:56:45.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.007+0000 7fda6effd700 1 --2- 172.21.15.32:0/4127868798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda70102e00 0x7fda70115ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:56:45.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.007+0000 7fda6e7fc700 1 --2- 172.21.15.32:0/4127868798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda701037a0 0x7fda70116000 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fda640093b0 tx=0x7fda64009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:56:45.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.007+0000 7fda577fe700 1 -- 172.21.15.32:0/4127868798 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda6400e040 con 0x7fda701037a0 2024-04-12T05:56:45.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.007+0000 7fda75299700 1 -- 172.21.15.32:0/4127868798 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda7010feb0 con 0x7fda701037a0 2024-04-12T05:56:45.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.007+0000 7fda577fe700 1 -- 172.21.15.32:0/4127868798 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda64013420 con 0x7fda701037a0 2024-04-12T05:56:45.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.007+0000 7fda577fe700 1 -- 172.21.15.32:0/4127868798 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda6401c630 con 0x7fda701037a0 2024-04-12T05:56:45.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.007+0000 7fda75299700 1 -- 172.21.15.32:0/4127868798 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda70110470 con 0x7fda701037a0 2024-04-12T05:56:45.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.008+0000 7fda75299700 1 -- 172.21.15.32:0/4127868798 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda700fe550 con 0x7fda701037a0 2024-04-12T05:56:45.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.009+0000 7fda577fe700 1 -- 172.21.15.32:0/4127868798 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fda640139e0 con 0x7fda701037a0 2024-04-12T05:56:45.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.009+0000 7fda577fe700 1 --2- 172.21.15.32:0/4127868798 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fda5806cc20 0x7fda5806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:45.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.009+0000 7fda577fe700 1 -- 172.21.15.32:0/4127868798 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fda6408d2f0 con 0x7fda701037a0 2024-04-12T05:56:45.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.009+0000 7fda6effd700 1 --2- 172.21.15.32:0/4127868798 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fda5806cc20 0x7fda5806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:45.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.010+0000 7fda6effd700 1 --2- 172.21.15.32:0/4127868798 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fda5806cc20 0x7fda5806f0d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fda60009fd0 tx=0x7fda6000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:56:45.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.014+0000 7fda577fe700 1 -- 172.21.15.32:0/4127868798 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fda6405ad60 con 0x7fda701037a0 2024-04-12T05:56:45.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.229+0000 7fda75299700 1 -- 172.21.15.32:0/4127868798 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fda70109e10 con 0x7fda5806cc20 2024-04-12T05:56:45.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.238+0000 7fda577fe700 1 -- 172.21.15.32:0/4127868798 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fda70109e10 con 0x7fda5806cc20 2024-04-12T05:56:45.234 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:56:45.234 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (16m) 10m ago 27m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:56:45.234 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (14m) 10m ago 28m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:56:45.234 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (13m) 9m ago 27m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:56:45.235 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (16m) 10m ago 27m 87.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:56:45.235 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (10m) 10m ago 22m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T05:56:45.235 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (10m) 10m ago 22m 13.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T05:56:45.235 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (9m) 9m ago 22m 11.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 6d4aca6f74a0 2024-04-12T05:56:45.235 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (9m) 9m ago 22m 14.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T05:56:45.235 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (18m) 10m ago 29m 507M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:56:45.235 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (17m) 9m ago 27m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:56:45.236 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (15m) 10m ago 29m 53.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:56:45.236 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (14m) 9m ago 26m 43.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:56:45.236 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (17m) 10m ago 28m 16.9M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:56:45.236 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (17m) 9m ago 27m 18.4M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:56:45.236 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (13m) 10m ago 26m 91.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T05:56:45.236 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (13m) 10m ago 25m 97.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T05:56:45.236 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (12m) 10m ago 25m 81.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T05:56:45.237 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (11m) 9m ago 24m 113M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T05:56:45.237 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (11m) 9m ago 24m 90.3M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T05:56:45.237 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (10m) 9m ago 23m 84.8M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T05:56:45.237 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (16m) 10m ago 27m 64.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:56:45.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.241+0000 7fda75299700 1 -- 172.21.15.32:0/4127868798 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fda5806cc20 msgr2=0x7fda5806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:45.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.241+0000 7fda75299700 1 --2- 172.21.15.32:0/4127868798 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fda5806cc20 0x7fda5806f0d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fda60009fd0 tx=0x7fda6000b040 comp rx=0 tx=0).stop 2024-04-12T05:56:45.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.241+0000 7fda75299700 1 -- 172.21.15.32:0/4127868798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda701037a0 msgr2=0x7fda70116000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:45.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.241+0000 7fda75299700 1 --2- 172.21.15.32:0/4127868798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda701037a0 0x7fda70116000 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fda640093b0 tx=0x7fda64009e30 comp rx=0 tx=0).stop 2024-04-12T05:56:45.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.241+0000 7fda75299700 1 -- 172.21.15.32:0/4127868798 shutdown_connections 2024-04-12T05:56:45.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.241+0000 7fda75299700 1 --2- 172.21.15.32:0/4127868798 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fda5806cc20 0x7fda5806f0d0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:45.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.241+0000 7fda75299700 1 --2- 172.21.15.32:0/4127868798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda70102e00 0x7fda70115ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:45.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.241+0000 7fda75299700 1 --2- 172.21.15.32:0/4127868798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda701037a0 0x7fda70116000 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:45.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.241+0000 7fda75299700 1 -- 172.21.15.32:0/4127868798 >> 172.21.15.32:0/4127868798 conn(0x7fda700f89b0 msgr2=0x7fda700f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:56:45.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.241+0000 7fda75299700 1 -- 172.21.15.32:0/4127868798 shutdown_connections 2024-04-12T05:56:45.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.241+0000 7fda75299700 1 -- 172.21.15.32:0/4127868798 wait complete. 2024-04-12T05:56:45.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.343+0000 7fe84ec81700 1 -- 172.21.15.32:0/3874240022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe848102710 msgr2=0x7fe8480fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:45.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.343+0000 7fe84ec81700 1 --2- 172.21.15.32:0/3874240022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe848102710 0x7fe8480fbd80 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe834004940 tx=0x7fe834009e30 comp rx=0 tx=0).stop 2024-04-12T05:56:45.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.343+0000 7fe84ec81700 1 -- 172.21.15.32:0/3874240022 shutdown_connections 2024-04-12T05:56:45.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.343+0000 7fe84ec81700 1 --2- 172.21.15.32:0/3874240022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe848102710 0x7fe8480fbd80 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:45.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.343+0000 7fe84ec81700 1 --2- 172.21.15.32:0/3874240022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe848101d70 0x7fe848102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:45.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.343+0000 7fe84ec81700 1 -- 172.21.15.32:0/3874240022 >> 172.21.15.32:0/3874240022 conn(0x7fe848072bc0 msgr2=0x7fe848072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:56:45.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.343+0000 7fe84ec81700 1 -- 172.21.15.32:0/3874240022 shutdown_connections 2024-04-12T05:56:45.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.343+0000 7fe84ec81700 1 -- 172.21.15.32:0/3874240022 wait complete. 2024-04-12T05:56:45.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.343+0000 7fe84ec81700 1 Processor -- start 2024-04-12T05:56:45.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.343+0000 7fe84ec81700 1 -- start start 2024-04-12T05:56:45.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.344+0000 7fe84ec81700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe848101d70 0x7fe8481032d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:45.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.344+0000 7fe84ec81700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe848102710 0x7fe848103810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:45.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.344+0000 7fe84ec81700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe848103da0 con 0x7fe848101d70 2024-04-12T05:56:45.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.344+0000 7fe84ec81700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe848103ee0 con 0x7fe848102710 2024-04-12T05:56:45.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.344+0000 7fe84ca1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe848101d70 0x7fe8481032d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:45.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.344+0000 7fe84ca1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe848101d70 0x7fe8481032d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43156/0 (socket says 172.21.15.32:43156) 2024-04-12T05:56:45.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.344+0000 7fe84ca1d700 1 -- 172.21.15.32:0/1707175127 learned_addr learned my addr 172.21.15.32:0/1707175127 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:56:45.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.344+0000 7fe847fff700 1 --2- 172.21.15.32:0/1707175127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe848102710 0x7fe848103810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:45.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.344+0000 7fe84ca1d700 1 -- 172.21.15.32:0/1707175127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe848102710 msgr2=0x7fe848103810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:45.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.344+0000 7fe84ca1d700 1 --2- 172.21.15.32:0/1707175127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe848102710 0x7fe848103810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:45.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.344+0000 7fe84ca1d700 1 -- 172.21.15.32:0/1707175127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe834005040 con 0x7fe848101d70 2024-04-12T05:56:45.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.344+0000 7fe847fff700 1 --2- 172.21.15.32:0/1707175127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe848102710 0x7fe848103810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:56:45.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.344+0000 7fe84ca1d700 1 --2- 172.21.15.32:0/1707175127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe848101d70 0x7fe8481032d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe83c00eb40 tx=0x7fe83c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:56:45.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.344+0000 7fe8457fa700 1 -- 172.21.15.32:0/1707175127 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe83c007cc0 con 0x7fe848101d70 2024-04-12T05:56:45.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.344+0000 7fe84ec81700 1 -- 172.21.15.32:0/1707175127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe848104130 con 0x7fe848101d70 2024-04-12T05:56:45.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.344+0000 7fe8457fa700 1 -- 172.21.15.32:0/1707175127 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe83c00f040 con 0x7fe848101d70 2024-04-12T05:56:45.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.344+0000 7fe8457fa700 1 -- 172.21.15.32:0/1707175127 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe83c009570 con 0x7fe848101d70 2024-04-12T05:56:45.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.344+0000 7fe84ec81700 1 -- 172.21.15.32:0/1707175127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe84811a140 con 0x7fe848101d70 2024-04-12T05:56:45.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.345+0000 7fe84ec81700 1 -- 172.21.15.32:0/1707175127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8480fd4c0 con 0x7fe848101d70 2024-04-12T05:56:45.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.346+0000 7fe8457fa700 1 -- 172.21.15.32:0/1707175127 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe83c0096d0 con 0x7fe848101d70 2024-04-12T05:56:45.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.346+0000 7fe8457fa700 1 --2- 172.21.15.32:0/1707175127 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe83806cc30 0x7fe83806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:45.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.346+0000 7fe8457fa700 1 -- 172.21.15.32:0/1707175127 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fe83c08bd70 con 0x7fe848101d70 2024-04-12T05:56:45.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.346+0000 7fe847fff700 1 --2- 172.21.15.32:0/1707175127 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe83806cc30 0x7fe83806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:45.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.347+0000 7fe847fff700 1 --2- 172.21.15.32:0/1707175127 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe83806cc30 0x7fe83806f0e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe83400b0d0 tx=0x7fe834009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:56:45.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.351+0000 7fe8457fa700 1 -- 172.21.15.32:0/1707175127 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe83c055d80 con 0x7fe848101d70 2024-04-12T05:56:45.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:45 smithi067 ceph-mon[112979]: pgmap v487: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:45.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:45 smithi067 ceph-mon[112979]: from='client.34928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:56:45.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:45 smithi032 ceph-mon[146817]: pgmap v487: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:45.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:45 smithi032 ceph-mon[146817]: from='client.34928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:56:45.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.636+0000 7fe84ec81700 1 -- 172.21.15.32:0/1707175127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe8481049c0 con 0x7fe848101d70 2024-04-12T05:56:45.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.636+0000 7fe8457fa700 1 -- 172.21.15.32:0/1707175127 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fe83c0593a0 con 0x7fe848101d70 2024-04-12T05:56:45.632 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:56:45.632 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:56:45.632 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:56:45.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:56:45.632 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:56:45.633 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:56:45.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:56:45.633 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:56:45.633 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:56:45.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:56:45.633 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:56:45.633 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:56:45.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:56:45.634 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:56:45.634 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:56:45.634 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:56:45.634 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:56:45.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.638+0000 7fe84ec81700 1 -- 172.21.15.32:0/1707175127 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe83806cc30 msgr2=0x7fe83806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:45.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.638+0000 7fe84ec81700 1 --2- 172.21.15.32:0/1707175127 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe83806cc30 0x7fe83806f0e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe83400b0d0 tx=0x7fe834009e30 comp rx=0 tx=0).stop 2024-04-12T05:56:45.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.638+0000 7fe84ec81700 1 -- 172.21.15.32:0/1707175127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe848101d70 msgr2=0x7fe8481032d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:45.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.638+0000 7fe84ec81700 1 --2- 172.21.15.32:0/1707175127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe848101d70 0x7fe8481032d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe83c00eb40 tx=0x7fe83c00b040 comp rx=0 tx=0).stop 2024-04-12T05:56:45.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.638+0000 7fe84ec81700 1 -- 172.21.15.32:0/1707175127 shutdown_connections 2024-04-12T05:56:45.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.638+0000 7fe84ec81700 1 --2- 172.21.15.32:0/1707175127 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe83806cc30 0x7fe83806f0e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:45.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.638+0000 7fe84ec81700 1 --2- 172.21.15.32:0/1707175127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe848101d70 0x7fe8481032d0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:45.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.638+0000 7fe84ec81700 1 --2- 172.21.15.32:0/1707175127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe848102710 0x7fe848103810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:45.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.638+0000 7fe84ec81700 1 -- 172.21.15.32:0/1707175127 >> 172.21.15.32:0/1707175127 conn(0x7fe848072bc0 msgr2=0x7fe8481056f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:56:45.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.638+0000 7fe84ec81700 1 -- 172.21.15.32:0/1707175127 shutdown_connections 2024-04-12T05:56:45.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.638+0000 7fe84ec81700 1 -- 172.21.15.32:0/1707175127 wait complete. 2024-04-12T05:56:45.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.742+0000 7f2a83ffc700 1 -- 172.21.15.32:0/264028381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a7c102e10 msgr2=0x7f2a7c1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:45.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.742+0000 7f2a83ffc700 1 --2- 172.21.15.32:0/264028381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a7c102e10 0x7f2a7c1031e0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f2a78004940 tx=0x7f2a78009e30 comp rx=0 tx=0).stop 2024-04-12T05:56:45.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.742+0000 7f2a83ffc700 1 -- 172.21.15.32:0/264028381 shutdown_connections 2024-04-12T05:56:45.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.742+0000 7f2a83ffc700 1 --2- 172.21.15.32:0/264028381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a7c1037b0 0x7f2a7c0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:45.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.742+0000 7f2a83ffc700 1 --2- 172.21.15.32:0/264028381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a7c102e10 0x7f2a7c1031e0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:45.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.742+0000 7f2a83ffc700 1 -- 172.21.15.32:0/264028381 >> 172.21.15.32:0/264028381 conn(0x7f2a7c0f8980 msgr2=0x7f2a7c0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:56:45.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.742+0000 7f2a83ffc700 1 -- 172.21.15.32:0/264028381 shutdown_connections 2024-04-12T05:56:45.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.742+0000 7f2a83ffc700 1 -- 172.21.15.32:0/264028381 wait complete. 2024-04-12T05:56:45.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.742+0000 7f2a83ffc700 1 Processor -- start 2024-04-12T05:56:45.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.742+0000 7f2a83ffc700 1 -- start start 2024-04-12T05:56:45.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.743+0000 7f2a83ffc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a7c102e10 0x7f2a7c113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:45.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.743+0000 7f2a83ffc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a7c1037b0 0x7f2a7c113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:45.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.743+0000 7f2a83ffc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7c1145f0 con 0x7f2a7c1037b0 2024-04-12T05:56:45.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.743+0000 7f2a83ffc700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7c10d9e0 con 0x7f2a7c102e10 2024-04-12T05:56:45.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.743+0000 7f2a81597700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a7c1037b0 0x7f2a7c113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:45.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.743+0000 7f2a81597700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a7c1037b0 0x7f2a7c113fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43174/0 (socket says 172.21.15.32:43174) 2024-04-12T05:56:45.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.743+0000 7f2a81597700 1 -- 172.21.15.32:0/3900056432 learned_addr learned my addr 172.21.15.32:0/3900056432 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:56:45.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.743+0000 7f2a81d98700 1 --2- 172.21.15.32:0/3900056432 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a7c102e10 0x7f2a7c113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:45.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.743+0000 7f2a81597700 1 -- 172.21.15.32:0/3900056432 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a7c102e10 msgr2=0x7f2a7c113a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:45.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.743+0000 7f2a81597700 1 --2- 172.21.15.32:0/3900056432 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a7c102e10 0x7f2a7c113a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:45.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.743+0000 7f2a81597700 1 -- 172.21.15.32:0/3900056432 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a78005040 con 0x7f2a7c1037b0 2024-04-12T05:56:45.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.743+0000 7f2a81d98700 1 --2- 172.21.15.32:0/3900056432 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a7c102e10 0x7f2a7c113a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:56:45.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.743+0000 7f2a81597700 1 --2- 172.21.15.32:0/3900056432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a7c1037b0 0x7f2a7c113fa0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c00eb40 tx=0x7f2a6c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:56:45.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.743+0000 7f2a727fc700 1 -- 172.21.15.32:0/3900056432 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a6c007cc0 con 0x7f2a7c1037b0 2024-04-12T05:56:45.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.744+0000 7f2a83ffc700 1 -- 172.21.15.32:0/3900056432 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a7c10dc60 con 0x7f2a7c1037b0 2024-04-12T05:56:45.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.744+0000 7f2a727fc700 1 -- 172.21.15.32:0/3900056432 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2a6c00f040 con 0x7f2a7c1037b0 2024-04-12T05:56:45.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.744+0000 7f2a727fc700 1 -- 172.21.15.32:0/3900056432 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a6c009570 con 0x7f2a7c1037b0 2024-04-12T05:56:45.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.744+0000 7f2a83ffc700 1 -- 172.21.15.32:0/3900056432 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a7c10e220 con 0x7f2a7c1037b0 2024-04-12T05:56:45.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.745+0000 7f2a83ffc700 1 -- 172.21.15.32:0/3900056432 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a7c0fe560 con 0x7f2a7c1037b0 2024-04-12T05:56:45.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.745+0000 7f2a727fc700 1 -- 172.21.15.32:0/3900056432 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f2a6c0096d0 con 0x7f2a7c1037b0 2024-04-12T05:56:45.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.746+0000 7f2a727fc700 1 --2- 172.21.15.32:0/3900056432 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f2a6806cc20 0x7f2a6806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:45.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.746+0000 7f2a81d98700 1 --2- 172.21.15.32:0/3900056432 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f2a6806cc20 0x7f2a6806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:45.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.746+0000 7f2a727fc700 1 -- 172.21.15.32:0/3900056432 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f2a6c08bc00 con 0x7f2a7c1037b0 2024-04-12T05:56:45.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.746+0000 7f2a81d98700 1 --2- 172.21.15.32:0/3900056432 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f2a6806cc20 0x7f2a6806f0d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2a78004560 tx=0x7f2a78009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:56:45.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.750+0000 7f2a727fc700 1 -- 172.21.15.32:0/3900056432 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2a6c091020 con 0x7f2a7c1037b0 2024-04-12T05:56:45.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.996+0000 7f2a83ffc700 1 -- 172.21.15.32:0/3900056432 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2a7c109dd0 con 0x7f2a7c1037b0 2024-04-12T05:56:45.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.997+0000 7f2a727fc700 1 -- 172.21.15.32:0/3900056432 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1835 (secure 0 0 0) 0x7f2a6c0596a0 con 0x7f2a7c1037b0 2024-04-12T05:56:45.992 INFO:teuthology.orchestra.run.smithi032.stdout:e33 2024-04-12T05:56:45.992 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:56:45.993 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:56:45.993 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:56:45.993 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:56:45.993 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:56:45.993 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:56:45.993 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 24 2024-04-12T05:56:45.993 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:56:45.993 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:56:45.993 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:46:10.668569+0000 2024-04-12T05:56:45.993 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:56:45.993 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:56:45.993 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:56:45.993 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:56:45.994 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:56:45.994 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:56:45.994 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:56:45.994 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:56:45.994 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T05:56:45.994 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:56:45.994 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:56:45.994 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:56:45.994 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T05:56:45.994 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-12T05:56:45.994 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:56:45.994 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:56:45.994 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:56:45.994 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:56:45.995 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:56:45.995 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:56:45.995 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:56:45.995 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:56:45.995 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:56:45.995 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:56:45.995 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-12T05:56:45.995 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:56:45.995 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:56:45.995 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:56:45.995 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{-1:44317} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:56:45.995 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:56:45.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.999+0000 7f2a83ffc700 1 -- 172.21.15.32:0/3900056432 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f2a6806cc20 msgr2=0x7f2a6806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:45.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.999+0000 7f2a83ffc700 1 --2- 172.21.15.32:0/3900056432 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f2a6806cc20 0x7f2a6806f0d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2a78004560 tx=0x7f2a78009e30 comp rx=0 tx=0).stop 2024-04-12T05:56:45.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.999+0000 7f2a83ffc700 1 -- 172.21.15.32:0/3900056432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a7c1037b0 msgr2=0x7f2a7c113fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:45.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.999+0000 7f2a83ffc700 1 --2- 172.21.15.32:0/3900056432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a7c1037b0 0x7f2a7c113fa0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c00eb40 tx=0x7f2a6c00b040 comp rx=0 tx=0).stop 2024-04-12T05:56:45.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.999+0000 7f2a83ffc700 1 -- 172.21.15.32:0/3900056432 shutdown_connections 2024-04-12T05:56:45.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.999+0000 7f2a83ffc700 1 --2- 172.21.15.32:0/3900056432 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f2a6806cc20 0x7f2a6806f0d0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:45.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.999+0000 7f2a83ffc700 1 --2- 172.21.15.32:0/3900056432 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a7c102e10 0x7f2a7c113a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:45.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.999+0000 7f2a83ffc700 1 --2- 172.21.15.32:0/3900056432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a7c1037b0 0x7f2a7c113fa0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:45.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.999+0000 7f2a83ffc700 1 -- 172.21.15.32:0/3900056432 >> 172.21.15.32:0/3900056432 conn(0x7f2a7c0f8980 msgr2=0x7f2a7c0f9970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:56:45.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.999+0000 7f2a83ffc700 1 -- 172.21.15.32:0/3900056432 shutdown_connections 2024-04-12T05:56:45.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:45.999+0000 7f2a83ffc700 1 -- 172.21.15.32:0/3900056432 wait complete. 2024-04-12T05:56:45.997 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-12T05:56:46.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.103+0000 7f0c829be700 1 -- 172.21.15.32:0/2531132152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c7c0fded0 msgr2=0x7f0c7c0fe2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:46.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.103+0000 7f0c829be700 1 --2- 172.21.15.32:0/2531132152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c7c0fded0 0x7f0c7c0fe2a0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c004940 tx=0x7f0c6c009e30 comp rx=0 tx=0).stop 2024-04-12T05:56:46.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.103+0000 7f0c829be700 1 -- 172.21.15.32:0/2531132152 shutdown_connections 2024-04-12T05:56:46.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.103+0000 7f0c829be700 1 --2- 172.21.15.32:0/2531132152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c7c0fe870 0x7f0c7c106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:46.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.103+0000 7f0c829be700 1 --2- 172.21.15.32:0/2531132152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c7c0fded0 0x7f0c7c0fe2a0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:46.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.103+0000 7f0c829be700 1 -- 172.21.15.32:0/2531132152 >> 172.21.15.32:0/2531132152 conn(0x7f0c7c072bc0 msgr2=0x7f0c7c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:56:46.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.103+0000 7f0c829be700 1 -- 172.21.15.32:0/2531132152 shutdown_connections 2024-04-12T05:56:46.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.103+0000 7f0c829be700 1 -- 172.21.15.32:0/2531132152 wait complete. 2024-04-12T05:56:46.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.104+0000 7f0c829be700 1 Processor -- start 2024-04-12T05:56:46.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.104+0000 7f0c829be700 1 -- start start 2024-04-12T05:56:46.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.104+0000 7f0c829be700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c7c0fded0 0x7f0c7c113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:46.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.104+0000 7f0c829be700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c7c0fe870 0x7f0c7c113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:46.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.104+0000 7f0c829be700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7c114520 con 0x7f0c7c0fe870 2024-04-12T05:56:46.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.104+0000 7f0c829be700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7c10d960 con 0x7f0c7c0fded0 2024-04-12T05:56:46.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.104+0000 7f0c7b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c7c0fe870 0x7f0c7c113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:46.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.104+0000 7f0c7b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c7c0fe870 0x7f0c7c113ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43188/0 (socket says 172.21.15.32:43188) 2024-04-12T05:56:46.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.104+0000 7f0c7b7fe700 1 -- 172.21.15.32:0/4168116646 learned_addr learned my addr 172.21.15.32:0/4168116646 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:56:46.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.105+0000 7f0c7bfff700 1 --2- 172.21.15.32:0/4168116646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c7c0fded0 0x7f0c7c113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:46.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.105+0000 7f0c7b7fe700 1 -- 172.21.15.32:0/4168116646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c7c0fded0 msgr2=0x7f0c7c113990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:46.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.105+0000 7f0c7b7fe700 1 --2- 172.21.15.32:0/4168116646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c7c0fded0 0x7f0c7c113990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:46.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.105+0000 7f0c7b7fe700 1 -- 172.21.15.32:0/4168116646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c6c005040 con 0x7f0c7c0fe870 2024-04-12T05:56:46.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.105+0000 7f0c7b7fe700 1 --2- 172.21.15.32:0/4168116646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c7c0fe870 0x7f0c7c113ed0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f0c70002a80 tx=0x7f0c700046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:56:46.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.105+0000 7f0c78ff9700 1 -- 172.21.15.32:0/4168116646 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c70007cc0 con 0x7f0c7c0fe870 2024-04-12T05:56:46.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.105+0000 7f0c78ff9700 1 -- 172.21.15.32:0/4168116646 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c7000f040 con 0x7f0c7c0fe870 2024-04-12T05:56:46.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.105+0000 7f0c78ff9700 1 -- 172.21.15.32:0/4168116646 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c70009640 con 0x7f0c7c0fe870 2024-04-12T05:56:46.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.105+0000 7f0c829be700 1 -- 172.21.15.32:0/4168116646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c7c10dc40 con 0x7f0c7c0fe870 2024-04-12T05:56:46.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.105+0000 7f0c829be700 1 -- 172.21.15.32:0/4168116646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c7c10e180 con 0x7f0c7c0fe870 2024-04-12T05:56:46.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.106+0000 7f0c829be700 1 -- 172.21.15.32:0/4168116646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c7c065df0 con 0x7f0c7c0fe870 2024-04-12T05:56:46.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.107+0000 7f0c78ff9700 1 -- 172.21.15.32:0/4168116646 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f0c700097a0 con 0x7f0c7c0fe870 2024-04-12T05:56:46.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.107+0000 7f0c78ff9700 1 --2- 172.21.15.32:0/4168116646 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0c6406cc20 0x7f0c6406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:46.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.107+0000 7f0c78ff9700 1 -- 172.21.15.32:0/4168116646 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f0c7008b8e0 con 0x7f0c7c0fe870 2024-04-12T05:56:46.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.107+0000 7f0c7bfff700 1 --2- 172.21.15.32:0/4168116646 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0c6406cc20 0x7f0c6406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:46.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.107+0000 7f0c7bfff700 1 --2- 172.21.15.32:0/4168116646 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0c6406cc20 0x7f0c6406f0d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c004f90 tx=0x7f0c6c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:56:46.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.112+0000 7f0c78ff9700 1 -- 172.21.15.32:0/4168116646 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0c700592d0 con 0x7f0c7c0fe870 2024-04-12T05:56:46.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.340+0000 7f0c829be700 1 -- 172.21.15.32:0/4168116646 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c7c109d50 con 0x7f0c6406cc20 2024-04-12T05:56:46.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.341+0000 7f0c78ff9700 1 -- 172.21.15.32:0/4168116646 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0c7c109d50 con 0x7f0c6406cc20 2024-04-12T05:56:46.336 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:56:46.336 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:56:46.336 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:56:46.337 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:56:46.337 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:56:46.337 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-12T05:56:46.337 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:56:46.337 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-12T05:56:46.337 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:56:46.337 INFO:teuthology.orchestra.run.smithi032.stdout: "mds" 2024-04-12T05:56:46.337 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:56:46.337 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:56:46.337 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:56:46.337 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:56:46.337 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:56:46.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.343+0000 7f0c829be700 1 -- 172.21.15.32:0/4168116646 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0c6406cc20 msgr2=0x7f0c6406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:46.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.343+0000 7f0c829be700 1 --2- 172.21.15.32:0/4168116646 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0c6406cc20 0x7f0c6406f0d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c004f90 tx=0x7f0c6c009e30 comp rx=0 tx=0).stop 2024-04-12T05:56:46.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.343+0000 7f0c829be700 1 -- 172.21.15.32:0/4168116646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c7c0fe870 msgr2=0x7f0c7c113ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:46.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.343+0000 7f0c829be700 1 --2- 172.21.15.32:0/4168116646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c7c0fe870 0x7f0c7c113ed0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f0c70002a80 tx=0x7f0c700046b0 comp rx=0 tx=0).stop 2024-04-12T05:56:46.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.343+0000 7f0c829be700 1 -- 172.21.15.32:0/4168116646 shutdown_connections 2024-04-12T05:56:46.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.343+0000 7f0c829be700 1 --2- 172.21.15.32:0/4168116646 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0c6406cc20 0x7f0c6406f0d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:46.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.343+0000 7f0c829be700 1 --2- 172.21.15.32:0/4168116646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c7c0fded0 0x7f0c7c113990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:46.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.343+0000 7f0c829be700 1 --2- 172.21.15.32:0/4168116646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c7c0fe870 0x7f0c7c113ed0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:46.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.343+0000 7f0c829be700 1 -- 172.21.15.32:0/4168116646 >> 172.21.15.32:0/4168116646 conn(0x7f0c7c072bc0 msgr2=0x7f0c7c103800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:56:46.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.343+0000 7f0c829be700 1 -- 172.21.15.32:0/4168116646 shutdown_connections 2024-04-12T05:56:46.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.343+0000 7f0c829be700 1 -- 172.21.15.32:0/4168116646 wait complete. 2024-04-12T05:56:46.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.446+0000 7f52ee0ad700 1 -- 172.21.15.32:0/1420352718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52e8101d70 msgr2=0x7f52e8102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:46.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.446+0000 7f52ee0ad700 1 --2- 172.21.15.32:0/1420352718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52e8101d70 0x7f52e8102140 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f52d8004940 tx=0x7f52d8009e30 comp rx=0 tx=0).stop 2024-04-12T05:56:46.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.446+0000 7f52ee0ad700 1 -- 172.21.15.32:0/1420352718 shutdown_connections 2024-04-12T05:56:46.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.446+0000 7f52ee0ad700 1 --2- 172.21.15.32:0/1420352718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52e8102710 0x7f52e80fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:46.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.446+0000 7f52ee0ad700 1 --2- 172.21.15.32:0/1420352718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52e8101d70 0x7f52e8102140 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:46.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.446+0000 7f52ee0ad700 1 -- 172.21.15.32:0/1420352718 >> 172.21.15.32:0/1420352718 conn(0x7f52e8072bc0 msgr2=0x7f52e8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:56:46.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.446+0000 7f52ee0ad700 1 -- 172.21.15.32:0/1420352718 shutdown_connections 2024-04-12T05:56:46.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.446+0000 7f52ee0ad700 1 -- 172.21.15.32:0/1420352718 wait complete. 2024-04-12T05:56:46.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.447+0000 7f52ee0ad700 1 Processor -- start 2024-04-12T05:56:46.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.447+0000 7f52ee0ad700 1 -- start start 2024-04-12T05:56:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.447+0000 7f52ee0ad700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52e8101d70 0x7f52e810ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.447+0000 7f52ee0ad700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52e8102710 0x7f52e810ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.447+0000 7f52ee0ad700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e810f670 con 0x7f52e8101d70 2024-04-12T05:56:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.447+0000 7f52ee0ad700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e810f7b0 con 0x7f52e8102710 2024-04-12T05:56:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.447+0000 7f52e77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52e8101d70 0x7f52e810ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.447+0000 7f52e77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52e8101d70 0x7f52e810ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43202/0 (socket says 172.21.15.32:43202) 2024-04-12T05:56:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.447+0000 7f52e77fe700 1 -- 172.21.15.32:0/1152139292 learned_addr learned my addr 172.21.15.32:0/1152139292 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:56:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.447+0000 7f52e6ffd700 1 --2- 172.21.15.32:0/1152139292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52e8102710 0x7f52e810ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.448+0000 7f52e77fe700 1 -- 172.21.15.32:0/1152139292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52e8102710 msgr2=0x7f52e810ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:46.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.448+0000 7f52e77fe700 1 --2- 172.21.15.32:0/1152139292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52e8102710 0x7f52e810ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:46.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.448+0000 7f52e77fe700 1 -- 172.21.15.32:0/1152139292 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d8005040 con 0x7f52e8101d70 2024-04-12T05:56:46.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.448+0000 7f52e6ffd700 1 --2- 172.21.15.32:0/1152139292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52e8102710 0x7f52e810ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:56:46.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.448+0000 7f52e77fe700 1 --2- 172.21.15.32:0/1152139292 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52e8101d70 0x7f52e810ea50 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f52d8009390 tx=0x7f52d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:56:46.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.448+0000 7f52cffff700 1 -- 172.21.15.32:0/1152139292 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52d800e040 con 0x7f52e8101d70 2024-04-12T05:56:46.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.448+0000 7f52ee0ad700 1 -- 172.21.15.32:0/1152139292 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52e8117ac0 con 0x7f52e8101d70 2024-04-12T05:56:46.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.448+0000 7f52cffff700 1 -- 172.21.15.32:0/1152139292 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f52d8013420 con 0x7f52e8101d70 2024-04-12T05:56:46.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.448+0000 7f52cffff700 1 -- 172.21.15.32:0/1152139292 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52d801c630 con 0x7f52e8101d70 2024-04-12T05:56:46.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.448+0000 7f52ee0ad700 1 -- 172.21.15.32:0/1152139292 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52e8118050 con 0x7f52e8101d70 2024-04-12T05:56:46.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.449+0000 7f52ee0ad700 1 -- 172.21.15.32:0/1152139292 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52e80ff080 con 0x7f52e8101d70 2024-04-12T05:56:46.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.449+0000 7f52cffff700 1 -- 172.21.15.32:0/1152139292 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f52d80139e0 con 0x7f52e8101d70 2024-04-12T05:56:46.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.450+0000 7f52cffff700 1 --2- 172.21.15.32:0/1152139292 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f52d006cc30 0x7f52d006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:56:46.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.450+0000 7f52cffff700 1 -- 172.21.15.32:0/1152139292 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f52d808d310 con 0x7f52e8101d70 2024-04-12T05:56:46.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.450+0000 7f52e6ffd700 1 --2- 172.21.15.32:0/1152139292 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f52d006cc30 0x7f52d006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:56:46.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.450+0000 7f52e6ffd700 1 --2- 172.21.15.32:0/1152139292 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f52d006cc30 0x7f52d006f0e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f52e8066000 tx=0x7f52dc008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:56:46.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.454+0000 7f52cffff700 1 -- 172.21.15.32:0/1152139292 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f52d80573a0 con 0x7f52e8101d70 2024-04-12T05:56:46.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:46 smithi067 ceph-mon[112979]: from='client.34932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:56:46.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:46 smithi067 ceph-mon[112979]: from='client.34936 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:56:46.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:46 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1707175127' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:56:46.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:46 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3900056432' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:56:46.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:46 smithi032 ceph-mon[146817]: from='client.34932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:56:46.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:46 smithi032 ceph-mon[146817]: from='client.34936 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:56:46.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:46 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1707175127' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:56:46.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:46 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3900056432' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:56:46.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.740+0000 7f52ee0ad700 1 -- 172.21.15.32:0/1152139292 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f52e804bed0 con 0x7f52e8101d70 2024-04-12T05:56:46.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.740+0000 7f52cffff700 1 -- 172.21.15.32:0/1152139292 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f52d805a9c0 con 0x7f52e8101d70 2024-04-12T05:56:46.735 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:56:46.736 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:56:46.736 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-12T05:56:46.736 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:56:46.736 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:56:46.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.742+0000 7f52ee0ad700 1 -- 172.21.15.32:0/1152139292 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f52d006cc30 msgr2=0x7f52d006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:46.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.742+0000 7f52ee0ad700 1 --2- 172.21.15.32:0/1152139292 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f52d006cc30 0x7f52d006f0e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f52e8066000 tx=0x7f52dc008040 comp rx=0 tx=0).stop 2024-04-12T05:56:46.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.742+0000 7f52ee0ad700 1 -- 172.21.15.32:0/1152139292 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52e8101d70 msgr2=0x7f52e810ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:56:46.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.742+0000 7f52ee0ad700 1 --2- 172.21.15.32:0/1152139292 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52e8101d70 0x7f52e810ea50 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f52d8009390 tx=0x7f52d8009e30 comp rx=0 tx=0).stop 2024-04-12T05:56:46.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.742+0000 7f52ee0ad700 1 -- 172.21.15.32:0/1152139292 shutdown_connections 2024-04-12T05:56:46.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.742+0000 7f52ee0ad700 1 --2- 172.21.15.32:0/1152139292 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f52d006cc30 0x7f52d006f0e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:46.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.742+0000 7f52ee0ad700 1 --2- 172.21.15.32:0/1152139292 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52e8101d70 0x7f52e810ea50 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:46.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.742+0000 7f52ee0ad700 1 --2- 172.21.15.32:0/1152139292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52e8102710 0x7f52e810ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:56:46.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.742+0000 7f52ee0ad700 1 -- 172.21.15.32:0/1152139292 >> 172.21.15.32:0/1152139292 conn(0x7f52e8072bc0 msgr2=0x7f52e80faf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:56:46.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.742+0000 7f52ee0ad700 1 -- 172.21.15.32:0/1152139292 shutdown_connections 2024-04-12T05:56:46.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:56:46.742+0000 7f52ee0ad700 1 -- 172.21.15.32:0/1152139292 wait complete. 2024-04-12T05:56:47.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:47 smithi067 ceph-mon[112979]: pgmap v488: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:47.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:47 smithi067 ceph-mon[112979]: from='client.34948 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:56:47.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:47 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1152139292' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:56:47.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:47 smithi032 ceph-mon[146817]: pgmap v488: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:47.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:47 smithi032 ceph-mon[146817]: from='client.34948 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:56:47.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:47 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1152139292' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:56:48.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:48 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:56:48.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:48 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:56:49.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:49 smithi067 ceph-mon[112979]: pgmap v489: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:49.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:49 smithi032 ceph-mon[146817]: pgmap v489: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:51.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:51 smithi067 ceph-mon[112979]: pgmap v490: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:51.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:51 smithi032 ceph-mon[146817]: pgmap v490: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:53.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:53 smithi067 ceph-mon[112979]: pgmap v491: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:53.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:53 smithi032 ceph-mon[146817]: pgmap v491: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:55.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:55 smithi067 ceph-mon[112979]: pgmap v492: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:55.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:55 smithi032 ceph-mon[146817]: pgmap v492: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:57.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:57 smithi067 ceph-mon[112979]: pgmap v493: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:57.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:57 smithi032 ceph-mon[146817]: pgmap v493: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:59.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:56:59 smithi067 ceph-mon[112979]: pgmap v494: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:59.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:56:59 smithi032 ceph-mon[146817]: pgmap v494: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:01.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:01 smithi067 ceph-mon[112979]: pgmap v495: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:01.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:01 smithi032 ceph-mon[146817]: pgmap v495: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:03.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:03 smithi067 ceph-mon[112979]: pgmap v496: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:03.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:57:03.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:03 smithi032 ceph-mon[146817]: pgmap v496: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:03.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:57:05.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:05 smithi067 ceph-mon[112979]: pgmap v497: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:05.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:05 smithi032 ceph-mon[146817]: pgmap v497: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:07.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:07 smithi067 ceph-mon[112979]: pgmap v498: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:07.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:07 smithi032 ceph-mon[146817]: pgmap v498: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:09.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:09 smithi067 ceph-mon[112979]: pgmap v499: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:09.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:09 smithi032 ceph-mon[146817]: pgmap v499: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:11.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:11 smithi067 ceph-mon[112979]: pgmap v500: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:11.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:11 smithi032 ceph-mon[146817]: pgmap v500: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:13.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:13 smithi067 ceph-mon[112979]: pgmap v501: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:13.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:13 smithi032 ceph-mon[146817]: pgmap v501: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:15.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:15 smithi067 ceph-mon[112979]: pgmap v502: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:15.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:15 smithi032 ceph-mon[146817]: pgmap v502: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:16.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.860+0000 7f1ed1919700 1 -- 172.21.15.32:0/616876421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ecc070c10 msgr2=0x7f1ecc109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:16.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.860+0000 7f1ed1919700 1 --2- 172.21.15.32:0/616876421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ecc070c10 0x7f1ecc109170 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f1ec0004970 tx=0x7f1ec0009e30 comp rx=0 tx=0).stop 2024-04-12T05:57:16.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.860+0000 7f1ed1919700 1 -- 172.21.15.32:0/616876421 shutdown_connections 2024-04-12T05:57:16.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.860+0000 7f1ed1919700 1 --2- 172.21.15.32:0/616876421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ecc070c10 0x7f1ecc109170 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:16.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.860+0000 7f1ed1919700 1 --2- 172.21.15.32:0/616876421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ecc070300 0x7f1ecc0706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:16.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.860+0000 7f1ed1919700 1 -- 172.21.15.32:0/616876421 >> 172.21.15.32:0/616876421 conn(0x7f1ecc0f89b0 msgr2=0x7f1ecc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:57:16.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.861+0000 7f1ed1919700 1 -- 172.21.15.32:0/616876421 shutdown_connections 2024-04-12T05:57:16.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.861+0000 7f1ed1919700 1 -- 172.21.15.32:0/616876421 wait complete. 2024-04-12T05:57:16.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.861+0000 7f1ed1919700 1 Processor -- start 2024-04-12T05:57:16.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.861+0000 7f1ed1919700 1 -- start start 2024-04-12T05:57:16.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.862+0000 7f1ed1919700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ecc070300 0x7f1ecc115a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:16.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.862+0000 7f1ed1919700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ecc070c10 0x7f1ecc115fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:16.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.862+0000 7f1ed1919700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ecc116610 con 0x7f1ecc070c10 2024-04-12T05:57:16.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.862+0000 7f1ed1919700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ecc10fbf0 con 0x7f1ecc070300 2024-04-12T05:57:16.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.862+0000 7f1eca7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ecc070c10 0x7f1ecc115fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:16.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.862+0000 7f1eca7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ecc070c10 0x7f1ecc115fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44668/0 (socket says 172.21.15.32:44668) 2024-04-12T05:57:16.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.862+0000 7f1eca7fc700 1 -- 172.21.15.32:0/2814474959 learned_addr learned my addr 172.21.15.32:0/2814474959 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:57:16.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.862+0000 7f1ecaffd700 1 --2- 172.21.15.32:0/2814474959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ecc070300 0x7f1ecc115a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:16.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.862+0000 7f1eca7fc700 1 -- 172.21.15.32:0/2814474959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ecc070300 msgr2=0x7f1ecc115a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:16.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.862+0000 7f1eca7fc700 1 --2- 172.21.15.32:0/2814474959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ecc070300 0x7f1ecc115a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:16.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.862+0000 7f1eca7fc700 1 -- 172.21.15.32:0/2814474959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec0005040 con 0x7f1ecc070c10 2024-04-12T05:57:16.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.862+0000 7f1eca7fc700 1 --2- 172.21.15.32:0/2814474959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ecc070c10 0x7f1ecc115fc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1ec00043d0 tx=0x7f1ec0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:57:16.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.863+0000 7f1eb37fe700 1 -- 172.21.15.32:0/2814474959 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ec000e040 con 0x7f1ecc070c10 2024-04-12T05:57:16.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.863+0000 7f1ed1919700 1 -- 172.21.15.32:0/2814474959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ecc10fe70 con 0x7f1ecc070c10 2024-04-12T05:57:16.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.863+0000 7f1eb37fe700 1 -- 172.21.15.32:0/2814474959 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ec0013420 con 0x7f1ecc070c10 2024-04-12T05:57:16.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.863+0000 7f1eb37fe700 1 -- 172.21.15.32:0/2814474959 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ec001c630 con 0x7f1ecc070c10 2024-04-12T05:57:16.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.863+0000 7f1ed1919700 1 -- 172.21.15.32:0/2814474959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ecc1103b0 con 0x7f1ecc070c10 2024-04-12T05:57:16.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.864+0000 7f1ed1919700 1 -- 172.21.15.32:0/2814474959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ecc071a40 con 0x7f1ecc070c10 2024-04-12T05:57:16.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.864+0000 7f1eb37fe700 1 -- 172.21.15.32:0/2814474959 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1ec0013970 con 0x7f1ecc070c10 2024-04-12T05:57:16.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.864+0000 7f1eb37fe700 1 --2- 172.21.15.32:0/2814474959 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1eb406cc20 0x7f1eb406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:16.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.865+0000 7f1eb37fe700 1 -- 172.21.15.32:0/2814474959 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f1ec008d330 con 0x7f1ecc070c10 2024-04-12T05:57:16.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.865+0000 7f1ecaffd700 1 --2- 172.21.15.32:0/2814474959 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1eb406cc20 0x7f1eb406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:16.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.865+0000 7f1ecaffd700 1 --2- 172.21.15.32:0/2814474959 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1eb406cc20 0x7f1eb406f0d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc007fd0 tx=0x7f1ebc009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:57:16.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:16.869+0000 7f1eb37fe700 1 -- 172.21.15.32:0/2814474959 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1ec005ada0 con 0x7f1ecc070c10 2024-04-12T05:57:17.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.092+0000 7f1ed1919700 1 -- 172.21.15.32:0/2814474959 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ecc10bfc0 con 0x7f1eb406cc20 2024-04-12T05:57:17.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.093+0000 7f1eb37fe700 1 -- 172.21.15.32:0/2814474959 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f1ecc10bfc0 con 0x7f1eb406cc20 2024-04-12T05:57:17.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.095+0000 7f1ed1919700 1 -- 172.21.15.32:0/2814474959 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1eb406cc20 msgr2=0x7f1eb406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:17.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.095+0000 7f1ed1919700 1 --2- 172.21.15.32:0/2814474959 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1eb406cc20 0x7f1eb406f0d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc007fd0 tx=0x7f1ebc009040 comp rx=0 tx=0).stop 2024-04-12T05:57:17.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.095+0000 7f1ed1919700 1 -- 172.21.15.32:0/2814474959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ecc070c10 msgr2=0x7f1ecc115fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:17.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.095+0000 7f1ed1919700 1 --2- 172.21.15.32:0/2814474959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ecc070c10 0x7f1ecc115fc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1ec00043d0 tx=0x7f1ec0009e30 comp rx=0 tx=0).stop 2024-04-12T05:57:17.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.095+0000 7f1ed1919700 1 -- 172.21.15.32:0/2814474959 shutdown_connections 2024-04-12T05:57:17.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.095+0000 7f1ed1919700 1 --2- 172.21.15.32:0/2814474959 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1eb406cc20 0x7f1eb406f0d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:17.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.095+0000 7f1ed1919700 1 --2- 172.21.15.32:0/2814474959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ecc070300 0x7f1ecc115a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:17.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.095+0000 7f1ed1919700 1 --2- 172.21.15.32:0/2814474959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ecc070c10 0x7f1ecc115fc0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:17.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.095+0000 7f1ed1919700 1 -- 172.21.15.32:0/2814474959 >> 172.21.15.32:0/2814474959 conn(0x7f1ecc0f89b0 msgr2=0x7f1ecc105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:57:17.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.095+0000 7f1ed1919700 1 -- 172.21.15.32:0/2814474959 shutdown_connections 2024-04-12T05:57:17.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.095+0000 7f1ed1919700 1 -- 172.21.15.32:0/2814474959 wait complete. 2024-04-12T05:57:17.101 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:57:17.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.200+0000 7fe3b56b6700 1 -- 172.21.15.32:0/73742252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3b0102e00 msgr2=0x7fe3b01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:17.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.200+0000 7fe3b56b6700 1 --2- 172.21.15.32:0/73742252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3b0102e00 0x7fe3b01031d0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0004940 tx=0x7fe3a0009e30 comp rx=0 tx=0).stop 2024-04-12T05:57:17.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.200+0000 7fe3b56b6700 1 -- 172.21.15.32:0/73742252 shutdown_connections 2024-04-12T05:57:17.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.200+0000 7fe3b56b6700 1 --2- 172.21.15.32:0/73742252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3b01037a0 0x7fe3b00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:17.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.200+0000 7fe3b56b6700 1 --2- 172.21.15.32:0/73742252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3b0102e00 0x7fe3b01031d0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:17.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.200+0000 7fe3b56b6700 1 -- 172.21.15.32:0/73742252 >> 172.21.15.32:0/73742252 conn(0x7fe3b00f89b0 msgr2=0x7fe3b00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:57:17.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.200+0000 7fe3b56b6700 1 -- 172.21.15.32:0/73742252 shutdown_connections 2024-04-12T05:57:17.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.200+0000 7fe3b56b6700 1 -- 172.21.15.32:0/73742252 wait complete. 2024-04-12T05:57:17.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.201+0000 7fe3b56b6700 1 Processor -- start 2024-04-12T05:57:17.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.201+0000 7fe3b56b6700 1 -- start start 2024-04-12T05:57:17.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.201+0000 7fe3b56b6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3b0102e00 0x7fe3b0113820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:17.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.201+0000 7fe3b56b6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3b01037a0 0x7fe3b0113d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:17.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.201+0000 7fe3b56b6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b01143b0 con 0x7fe3b01037a0 2024-04-12T05:57:17.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.201+0000 7fe3b56b6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b010d9b0 con 0x7fe3b0102e00 2024-04-12T05:57:17.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.201+0000 7fe3ae7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3b01037a0 0x7fe3b0113d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:17.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.201+0000 7fe3ae7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3b01037a0 0x7fe3b0113d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44684/0 (socket says 172.21.15.32:44684) 2024-04-12T05:57:17.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.201+0000 7fe3ae7fc700 1 -- 172.21.15.32:0/3276617906 learned_addr learned my addr 172.21.15.32:0/3276617906 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:57:17.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.201+0000 7fe3aeffd700 1 --2- 172.21.15.32:0/3276617906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3b0102e00 0x7fe3b0113820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:17.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.202+0000 7fe3ae7fc700 1 -- 172.21.15.32:0/3276617906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3b0102e00 msgr2=0x7fe3b0113820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:17.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.202+0000 7fe3ae7fc700 1 --2- 172.21.15.32:0/3276617906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3b0102e00 0x7fe3b0113820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:17.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.202+0000 7fe3ae7fc700 1 -- 172.21.15.32:0/3276617906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a0005040 con 0x7fe3b01037a0 2024-04-12T05:57:17.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.202+0000 7fe3ae7fc700 1 --2- 172.21.15.32:0/3276617906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3b01037a0 0x7fe3b0113d60 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fe398002a50 tx=0x7fe3980046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:57:17.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.202+0000 7fe3a77fe700 1 -- 172.21.15.32:0/3276617906 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe398007cc0 con 0x7fe3b01037a0 2024-04-12T05:57:17.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.202+0000 7fe3a77fe700 1 -- 172.21.15.32:0/3276617906 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe39800f040 con 0x7fe3b01037a0 2024-04-12T05:57:17.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.202+0000 7fe3b56b6700 1 -- 172.21.15.32:0/3276617906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3b010dc90 con 0x7fe3b01037a0 2024-04-12T05:57:17.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.202+0000 7fe3a77fe700 1 -- 172.21.15.32:0/3276617906 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe39800da30 con 0x7fe3b01037a0 2024-04-12T05:57:17.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.202+0000 7fe3b56b6700 1 -- 172.21.15.32:0/3276617906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3b010e250 con 0x7fe3b01037a0 2024-04-12T05:57:17.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.203+0000 7fe3b56b6700 1 -- 172.21.15.32:0/3276617906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3b00fe550 con 0x7fe3b01037a0 2024-04-12T05:57:17.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.204+0000 7fe3a77fe700 1 -- 172.21.15.32:0/3276617906 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe39800db90 con 0x7fe3b01037a0 2024-04-12T05:57:17.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.205+0000 7fe3a77fe700 1 --2- 172.21.15.32:0/3276617906 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe39c06cc20 0x7fe39c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:17.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.205+0000 7fe3a77fe700 1 -- 172.21.15.32:0/3276617906 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fe39808c3a0 con 0x7fe3b01037a0 2024-04-12T05:57:17.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.205+0000 7fe3aeffd700 1 --2- 172.21.15.32:0/3276617906 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe39c06cc20 0x7fe39c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:17.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.205+0000 7fe3aeffd700 1 --2- 172.21.15.32:0/3276617906 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe39c06cc20 0x7fe39c06f0d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0004f90 tx=0x7fe3a0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:57:17.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.209+0000 7fe3a77fe700 1 -- 172.21.15.32:0/3276617906 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe398059d70 con 0x7fe3b01037a0 2024-04-12T05:57:17.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.431+0000 7fe3b56b6700 1 -- 172.21.15.32:0/3276617906 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3b0109e60 con 0x7fe39c06cc20 2024-04-12T05:57:17.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.432+0000 7fe3a77fe700 1 -- 172.21.15.32:0/3276617906 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe3a0007d50 con 0x7fe39c06cc20 2024-04-12T05:57:17.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.435+0000 7fe3b56b6700 1 -- 172.21.15.32:0/3276617906 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe39c06cc20 msgr2=0x7fe39c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:17.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.435+0000 7fe3b56b6700 1 --2- 172.21.15.32:0/3276617906 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe39c06cc20 0x7fe39c06f0d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0004f90 tx=0x7fe3a0009e30 comp rx=0 tx=0).stop 2024-04-12T05:57:17.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.435+0000 7fe3b56b6700 1 -- 172.21.15.32:0/3276617906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3b01037a0 msgr2=0x7fe3b0113d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:17.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.435+0000 7fe3b56b6700 1 --2- 172.21.15.32:0/3276617906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3b01037a0 0x7fe3b0113d60 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fe398002a50 tx=0x7fe3980046b0 comp rx=0 tx=0).stop 2024-04-12T05:57:17.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.435+0000 7fe3b56b6700 1 -- 172.21.15.32:0/3276617906 shutdown_connections 2024-04-12T05:57:17.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.435+0000 7fe3b56b6700 1 --2- 172.21.15.32:0/3276617906 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe39c06cc20 0x7fe39c06f0d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:17.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.435+0000 7fe3b56b6700 1 --2- 172.21.15.32:0/3276617906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3b0102e00 0x7fe3b0113820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:17.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.435+0000 7fe3b56b6700 1 --2- 172.21.15.32:0/3276617906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3b01037a0 0x7fe3b0113d60 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:17.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.435+0000 7fe3b56b6700 1 -- 172.21.15.32:0/3276617906 >> 172.21.15.32:0/3276617906 conn(0x7fe3b00f89b0 msgr2=0x7fe3b00f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:57:17.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.435+0000 7fe3b56b6700 1 -- 172.21.15.32:0/3276617906 shutdown_connections 2024-04-12T05:57:17.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.435+0000 7fe3b56b6700 1 -- 172.21.15.32:0/3276617906 wait complete. 2024-04-12T05:57:17.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:17 smithi067 ceph-mon[112979]: pgmap v503: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:17.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.538+0000 7f91dea5e700 1 -- 172.21.15.32:0/2717159512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d81037a0 msgr2=0x7f91d80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:17.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.538+0000 7f91dea5e700 1 --2- 172.21.15.32:0/2717159512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d81037a0 0x7f91d80fce10 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f91cc004940 tx=0x7f91cc009e30 comp rx=0 tx=0).stop 2024-04-12T05:57:17.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.538+0000 7f91dea5e700 1 -- 172.21.15.32:0/2717159512 shutdown_connections 2024-04-12T05:57:17.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.538+0000 7f91dea5e700 1 --2- 172.21.15.32:0/2717159512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d81037a0 0x7f91d80fce10 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:17.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.538+0000 7f91dea5e700 1 --2- 172.21.15.32:0/2717159512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91d8102e00 0x7f91d81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:17.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.538+0000 7f91dea5e700 1 -- 172.21.15.32:0/2717159512 >> 172.21.15.32:0/2717159512 conn(0x7f91d80f89b0 msgr2=0x7f91d80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:57:17.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.539+0000 7f91dea5e700 1 -- 172.21.15.32:0/2717159512 shutdown_connections 2024-04-12T05:57:17.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.539+0000 7f91dea5e700 1 -- 172.21.15.32:0/2717159512 wait complete. 2024-04-12T05:57:17.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.539+0000 7f91dea5e700 1 Processor -- start 2024-04-12T05:57:17.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.539+0000 7f91dea5e700 1 -- start start 2024-04-12T05:57:17.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.539+0000 7f91dea5e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d8102e00 0x7f91d8113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:17.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.539+0000 7f91dea5e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91d81037a0 0x7f91d8114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:17.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.539+0000 7f91dea5e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d8114650 con 0x7f91d8102e00 2024-04-12T05:57:17.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.539+0000 7f91dea5e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d810da40 con 0x7f91d81037a0 2024-04-12T05:57:17.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.539+0000 7f91d7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d8102e00 0x7f91d8113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:17.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.540+0000 7f91d7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d8102e00 0x7f91d8113ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44702/0 (socket says 172.21.15.32:44702) 2024-04-12T05:57:17.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.540+0000 7f91d7fff700 1 -- 172.21.15.32:0/19002250 learned_addr learned my addr 172.21.15.32:0/19002250 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:57:17.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.540+0000 7f91d77fe700 1 --2- 172.21.15.32:0/19002250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91d81037a0 0x7f91d8114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:17.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.540+0000 7f91d7fff700 1 -- 172.21.15.32:0/19002250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91d81037a0 msgr2=0x7f91d8114000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:17.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.540+0000 7f91d7fff700 1 --2- 172.21.15.32:0/19002250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91d81037a0 0x7f91d8114000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:17.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.540+0000 7f91d7fff700 1 -- 172.21.15.32:0/19002250 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91cc005040 con 0x7f91d8102e00 2024-04-12T05:57:17.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.540+0000 7f91d7fff700 1 --2- 172.21.15.32:0/19002250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d8102e00 0x7f91d8113ac0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f91c800c830 tx=0x7f91c8004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:57:17.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.540+0000 7f91d4ff9700 1 -- 172.21.15.32:0/19002250 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91c800a760 con 0x7f91d8102e00 2024-04-12T05:57:17.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.540+0000 7f91dea5e700 1 -- 172.21.15.32:0/19002250 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91d810dd20 con 0x7f91d8102e00 2024-04-12T05:57:17.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.540+0000 7f91d4ff9700 1 -- 172.21.15.32:0/19002250 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f91c800ad70 con 0x7f91d8102e00 2024-04-12T05:57:17.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.540+0000 7f91d4ff9700 1 -- 172.21.15.32:0/19002250 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91c80113a0 con 0x7f91d8102e00 2024-04-12T05:57:17.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.540+0000 7f91dea5e700 1 -- 172.21.15.32:0/19002250 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91d810e2e0 con 0x7f91d8102e00 2024-04-12T05:57:17.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.541+0000 7f91dea5e700 1 -- 172.21.15.32:0/19002250 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91d80fe550 con 0x7f91d8102e00 2024-04-12T05:57:17.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.542+0000 7f91d4ff9700 1 -- 172.21.15.32:0/19002250 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f91c8012400 con 0x7f91d8102e00 2024-04-12T05:57:17.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.542+0000 7f91d4ff9700 1 --2- 172.21.15.32:0/19002250 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f91c006cc20 0x7f91c006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:17.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.542+0000 7f91d4ff9700 1 -- 172.21.15.32:0/19002250 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f91c808c110 con 0x7f91d8102e00 2024-04-12T05:57:17.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.542+0000 7f91d77fe700 1 --2- 172.21.15.32:0/19002250 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f91c006cc20 0x7f91c006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:17.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.543+0000 7f91d77fe700 1 --2- 172.21.15.32:0/19002250 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f91c006cc20 0x7f91c006f0d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f91d810ec00 tx=0x7f91cc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:57:17.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.547+0000 7f91d4ff9700 1 -- 172.21.15.32:0/19002250 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f91c8059b80 con 0x7f91d8102e00 2024-04-12T05:57:17.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:17 smithi032 ceph-mon[146817]: pgmap v503: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:17.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.757+0000 7f91dea5e700 1 -- 172.21.15.32:0/19002250 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f91d8109da0 con 0x7f91c006cc20 2024-04-12T05:57:17.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.766+0000 7f91d4ff9700 1 -- 172.21.15.32:0/19002250 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f91d8109da0 con 0x7f91c006cc20 2024-04-12T05:57:17.762 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:57:17.762 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (17m) 10m ago 28m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:57:17.762 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (14m) 10m ago 28m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:57:17.762 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (14m) 10m ago 27m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:57:17.762 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (16m) 10m ago 28m 87.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:57:17.762 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (10m) 10m ago 22m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T05:57:17.762 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (10m) 10m ago 22m 13.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T05:57:17.762 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (10m) 10m ago 22m 11.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 6d4aca6f74a0 2024-04-12T05:57:17.762 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (10m) 10m ago 22m 14.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T05:57:17.762 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (18m) 10m ago 29m 507M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:57:17.763 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (17m) 10m ago 27m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:57:17.763 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (15m) 10m ago 30m 53.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:57:17.763 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (15m) 10m ago 27m 43.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:57:17.763 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (17m) 10m ago 28m 16.9M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:57:17.763 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (17m) 10m ago 27m 18.4M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:57:17.763 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (14m) 10m ago 26m 91.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T05:57:17.763 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (13m) 10m ago 26m 97.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T05:57:17.763 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (12m) 10m ago 25m 81.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T05:57:17.763 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (12m) 10m ago 25m 113M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T05:57:17.763 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (11m) 10m ago 24m 90.3M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T05:57:17.763 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (11m) 10m ago 24m 84.8M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T05:57:17.763 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (17m) 10m ago 28m 64.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:57:17.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.769+0000 7f91dea5e700 1 -- 172.21.15.32:0/19002250 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f91c006cc20 msgr2=0x7f91c006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:17.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.769+0000 7f91dea5e700 1 --2- 172.21.15.32:0/19002250 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f91c006cc20 0x7f91c006f0d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f91d810ec00 tx=0x7f91cc009e30 comp rx=0 tx=0).stop 2024-04-12T05:57:17.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.769+0000 7f91dea5e700 1 -- 172.21.15.32:0/19002250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d8102e00 msgr2=0x7f91d8113ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:17.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.769+0000 7f91dea5e700 1 --2- 172.21.15.32:0/19002250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d8102e00 0x7f91d8113ac0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f91c800c830 tx=0x7f91c8004470 comp rx=0 tx=0).stop 2024-04-12T05:57:17.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.769+0000 7f91dea5e700 1 -- 172.21.15.32:0/19002250 shutdown_connections 2024-04-12T05:57:17.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.769+0000 7f91dea5e700 1 --2- 172.21.15.32:0/19002250 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f91c006cc20 0x7f91c006f0d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:17.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.769+0000 7f91dea5e700 1 --2- 172.21.15.32:0/19002250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d8102e00 0x7f91d8113ac0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:17.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.769+0000 7f91dea5e700 1 --2- 172.21.15.32:0/19002250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91d81037a0 0x7f91d8114000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:17.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.769+0000 7f91dea5e700 1 -- 172.21.15.32:0/19002250 >> 172.21.15.32:0/19002250 conn(0x7f91d80f89b0 msgr2=0x7f91d80f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:57:17.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.769+0000 7f91dea5e700 1 -- 172.21.15.32:0/19002250 shutdown_connections 2024-04-12T05:57:17.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.769+0000 7f91dea5e700 1 -- 172.21.15.32:0/19002250 wait complete. 2024-04-12T05:57:17.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.874+0000 7fd1757a2700 1 -- 172.21.15.32:0/2351553231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1700fde90 msgr2=0x7fd1700fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:17.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.874+0000 7fd1757a2700 1 --2- 172.21.15.32:0/2351553231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1700fde90 0x7fd1700fe260 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fd160004940 tx=0x7fd160009e30 comp rx=0 tx=0).stop 2024-04-12T05:57:17.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.874+0000 7fd1757a2700 1 -- 172.21.15.32:0/2351553231 shutdown_connections 2024-04-12T05:57:17.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.874+0000 7fd1757a2700 1 --2- 172.21.15.32:0/2351553231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1700fe830 0x7fd170106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:17.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.874+0000 7fd1757a2700 1 --2- 172.21.15.32:0/2351553231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1700fde90 0x7fd1700fe260 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:17.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.874+0000 7fd1757a2700 1 -- 172.21.15.32:0/2351553231 >> 172.21.15.32:0/2351553231 conn(0x7fd170072bc0 msgr2=0x7fd170072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:57:17.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.874+0000 7fd1757a2700 1 -- 172.21.15.32:0/2351553231 shutdown_connections 2024-04-12T05:57:17.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.874+0000 7fd1757a2700 1 -- 172.21.15.32:0/2351553231 wait complete. 2024-04-12T05:57:17.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.875+0000 7fd1757a2700 1 Processor -- start 2024-04-12T05:57:17.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.875+0000 7fd1757a2700 1 -- start start 2024-04-12T05:57:17.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.875+0000 7fd1757a2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1700fde90 0x7fd170113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:17.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.875+0000 7fd1757a2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1700fe830 0x7fd170114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:17.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.875+0000 7fd1757a2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd170114680 con 0x7fd1700fe830 2024-04-12T05:57:17.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.875+0000 7fd1757a2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17010da70 con 0x7fd1700fde90 2024-04-12T05:57:17.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.875+0000 7fd16e7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1700fe830 0x7fd170114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:17.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.875+0000 7fd16e7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1700fe830 0x7fd170114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44724/0 (socket says 172.21.15.32:44724) 2024-04-12T05:57:17.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.875+0000 7fd16e7fc700 1 -- 172.21.15.32:0/408133329 learned_addr learned my addr 172.21.15.32:0/408133329 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:57:17.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.875+0000 7fd16effd700 1 --2- 172.21.15.32:0/408133329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1700fde90 0x7fd170113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:17.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.875+0000 7fd16e7fc700 1 -- 172.21.15.32:0/408133329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1700fde90 msgr2=0x7fd170113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:17.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.875+0000 7fd16e7fc700 1 --2- 172.21.15.32:0/408133329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1700fde90 0x7fd170113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:17.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.875+0000 7fd16e7fc700 1 -- 172.21.15.32:0/408133329 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd160005040 con 0x7fd1700fe830 2024-04-12T05:57:17.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.876+0000 7fd16e7fc700 1 --2- 172.21.15.32:0/408133329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1700fe830 0x7fd170114030 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd164002a50 tx=0x7fd1640046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:57:17.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.876+0000 7fd1577fe700 1 -- 172.21.15.32:0/408133329 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd164007cc0 con 0x7fd1700fe830 2024-04-12T05:57:17.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.876+0000 7fd1757a2700 1 -- 172.21.15.32:0/408133329 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd17010dd50 con 0x7fd1700fe830 2024-04-12T05:57:17.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.876+0000 7fd1577fe700 1 -- 172.21.15.32:0/408133329 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd16400f040 con 0x7fd1700fe830 2024-04-12T05:57:17.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.876+0000 7fd1577fe700 1 -- 172.21.15.32:0/408133329 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd16400d9c0 con 0x7fd1700fe830 2024-04-12T05:57:17.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.876+0000 7fd1757a2700 1 -- 172.21.15.32:0/408133329 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd17010e290 con 0x7fd1700fe830 2024-04-12T05:57:17.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.877+0000 7fd1757a2700 1 -- 172.21.15.32:0/408133329 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1701019b0 con 0x7fd1700fe830 2024-04-12T05:57:17.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.877+0000 7fd1577fe700 1 -- 172.21.15.32:0/408133329 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd16400db20 con 0x7fd1700fe830 2024-04-12T05:57:17.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.878+0000 7fd1577fe700 1 --2- 172.21.15.32:0/408133329 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd15806cc30 0x7fd15806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:17.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.878+0000 7fd1577fe700 1 -- 172.21.15.32:0/408133329 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fd16408bd70 con 0x7fd1700fe830 2024-04-12T05:57:17.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.878+0000 7fd16effd700 1 --2- 172.21.15.32:0/408133329 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd15806cc30 0x7fd15806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:17.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.878+0000 7fd16effd700 1 --2- 172.21.15.32:0/408133329 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd15806cc30 0x7fd15806f0e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd16000b3b0 tx=0x7fd160009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:57:17.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:17.883+0000 7fd1577fe700 1 -- 172.21.15.32:0/408133329 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd1640597e0 con 0x7fd1700fe830 2024-04-12T05:57:18.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.168+0000 7fd1757a2700 1 -- 172.21.15.32:0/408133329 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd17010f140 con 0x7fd1700fe830 2024-04-12T05:57:18.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.169+0000 7fd1577fe700 1 -- 172.21.15.32:0/408133329 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fd164059370 con 0x7fd1700fe830 2024-04-12T05:57:18.164 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:57:18.164 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:57:18.165 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:57:18.165 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:57:18.165 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:57:18.165 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:57:18.165 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:57:18.165 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:57:18.165 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:57:18.165 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:57:18.165 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:57:18.165 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:57:18.165 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:57:18.165 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:57:18.165 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:57:18.166 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:57:18.166 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:57:18.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.171+0000 7fd1757a2700 1 -- 172.21.15.32:0/408133329 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd15806cc30 msgr2=0x7fd15806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:18.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.171+0000 7fd1757a2700 1 --2- 172.21.15.32:0/408133329 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd15806cc30 0x7fd15806f0e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd16000b3b0 tx=0x7fd160009e30 comp rx=0 tx=0).stop 2024-04-12T05:57:18.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.171+0000 7fd1757a2700 1 -- 172.21.15.32:0/408133329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1700fe830 msgr2=0x7fd170114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:18.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.171+0000 7fd1757a2700 1 --2- 172.21.15.32:0/408133329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1700fe830 0x7fd170114030 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd164002a50 tx=0x7fd1640046b0 comp rx=0 tx=0).stop 2024-04-12T05:57:18.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.171+0000 7fd1757a2700 1 -- 172.21.15.32:0/408133329 shutdown_connections 2024-04-12T05:57:18.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.171+0000 7fd1757a2700 1 --2- 172.21.15.32:0/408133329 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd15806cc30 0x7fd15806f0e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:18.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.171+0000 7fd1757a2700 1 --2- 172.21.15.32:0/408133329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1700fde90 0x7fd170113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:18.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.171+0000 7fd1757a2700 1 --2- 172.21.15.32:0/408133329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1700fe830 0x7fd170114030 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:18.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.171+0000 7fd1757a2700 1 -- 172.21.15.32:0/408133329 >> 172.21.15.32:0/408133329 conn(0x7fd170072bc0 msgr2=0x7fd1700ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:57:18.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.171+0000 7fd1757a2700 1 -- 172.21.15.32:0/408133329 shutdown_connections 2024-04-12T05:57:18.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.171+0000 7fd1757a2700 1 -- 172.21.15.32:0/408133329 wait complete. 2024-04-12T05:57:18.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.275+0000 7f13f524d700 1 -- 172.21.15.32:0/987788431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13f0101d70 msgr2=0x7f13f0102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:18.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.275+0000 7f13f524d700 1 --2- 172.21.15.32:0/987788431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13f0101d70 0x7f13f0102140 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f13e0004940 tx=0x7f13e0009e30 comp rx=0 tx=0).stop 2024-04-12T05:57:18.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.275+0000 7f13f524d700 1 -- 172.21.15.32:0/987788431 shutdown_connections 2024-04-12T05:57:18.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.275+0000 7f13f524d700 1 --2- 172.21.15.32:0/987788431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13f0102710 0x7f13f00fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:18.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.275+0000 7f13f524d700 1 --2- 172.21.15.32:0/987788431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13f0101d70 0x7f13f0102140 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:18.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.275+0000 7f13f524d700 1 -- 172.21.15.32:0/987788431 >> 172.21.15.32:0/987788431 conn(0x7f13f0072bc0 msgr2=0x7f13f0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:57:18.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.275+0000 7f13f524d700 1 -- 172.21.15.32:0/987788431 shutdown_connections 2024-04-12T05:57:18.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.275+0000 7f13f524d700 1 -- 172.21.15.32:0/987788431 wait complete. 2024-04-12T05:57:18.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.276+0000 7f13f524d700 1 Processor -- start 2024-04-12T05:57:18.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.276+0000 7f13f524d700 1 -- start start 2024-04-12T05:57:18.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.276+0000 7f13f524d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13f0101d70 0x7f13f0112920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:18.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.276+0000 7f13f524d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13f0102710 0x7f13f0112e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:18.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.276+0000 7f13f524d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f01134b0 con 0x7f13f0102710 2024-04-12T05:57:18.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.276+0000 7f13f524d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f010c8a0 con 0x7f13f0101d70 2024-04-12T05:57:18.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.276+0000 7f13ee59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13f0102710 0x7f13f0112e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:18.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.276+0000 7f13ee59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13f0102710 0x7f13f0112e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44738/0 (socket says 172.21.15.32:44738) 2024-04-12T05:57:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.276+0000 7f13ee59c700 1 -- 172.21.15.32:0/3307259195 learned_addr learned my addr 172.21.15.32:0/3307259195 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:57:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.277+0000 7f13eed9d700 1 --2- 172.21.15.32:0/3307259195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13f0101d70 0x7f13f0112920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.277+0000 7f13ee59c700 1 -- 172.21.15.32:0/3307259195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13f0101d70 msgr2=0x7f13f0112920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.277+0000 7f13ee59c700 1 --2- 172.21.15.32:0/3307259195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13f0101d70 0x7f13f0112920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.277+0000 7f13ee59c700 1 -- 172.21.15.32:0/3307259195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e0005040 con 0x7f13f0102710 2024-04-12T05:57:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.277+0000 7f13ee59c700 1 --2- 172.21.15.32:0/3307259195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13f0102710 0x7f13f0112e60 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f13e4002a50 tx=0x7f13e40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:57:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.277+0000 7f13d77fe700 1 -- 172.21.15.32:0/3307259195 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13e4007cc0 con 0x7f13f0102710 2024-04-12T05:57:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.277+0000 7f13d77fe700 1 -- 172.21.15.32:0/3307259195 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f13e400f040 con 0x7f13f0102710 2024-04-12T05:57:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.277+0000 7f13f524d700 1 -- 172.21.15.32:0/3307259195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13f010cb80 con 0x7f13f0102710 2024-04-12T05:57:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.277+0000 7f13d77fe700 1 -- 172.21.15.32:0/3307259195 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13e400da30 con 0x7f13f0102710 2024-04-12T05:57:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.277+0000 7f13f524d700 1 -- 172.21.15.32:0/3307259195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13f010d140 con 0x7f13f0102710 2024-04-12T05:57:18.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.278+0000 7f13f524d700 1 -- 172.21.15.32:0/3307259195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13f00ff080 con 0x7f13f0102710 2024-04-12T05:57:18.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.279+0000 7f13d77fe700 1 -- 172.21.15.32:0/3307259195 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f13e400db90 con 0x7f13f0102710 2024-04-12T05:57:18.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.279+0000 7f13d77fe700 1 --2- 172.21.15.32:0/3307259195 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f13d806cc20 0x7f13d806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:18.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.279+0000 7f13d77fe700 1 -- 172.21.15.32:0/3307259195 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f13e408bcf0 con 0x7f13f0102710 2024-04-12T05:57:18.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.279+0000 7f13eed9d700 1 --2- 172.21.15.32:0/3307259195 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f13d806cc20 0x7f13d806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:18.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.279+0000 7f13eed9d700 1 --2- 172.21.15.32:0/3307259195 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f13d806cc20 0x7f13d806f0d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f13e0004f90 tx=0x7f13e0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:57:18.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.284+0000 7f13d77fe700 1 -- 172.21.15.32:0/3307259195 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f13e40596e0 con 0x7f13f0102710 2024-04-12T05:57:18.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:18 smithi067 ceph-mon[112979]: from='client.34956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:57:18.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:18 smithi067 ceph-mon[112979]: from='client.34960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:57:18.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:18 smithi067 ceph-mon[112979]: from='client.34964 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:57:18.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:57:18.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:18 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/408133329' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:57:18.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.529+0000 7f13f524d700 1 -- 172.21.15.32:0/3307259195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f13f00fd4c0 con 0x7f13f0102710 2024-04-12T05:57:18.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.529+0000 7f13d77fe700 1 -- 172.21.15.32:0/3307259195 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1835 (secure 0 0 0) 0x7f13e4059270 con 0x7f13f0102710 2024-04-12T05:57:18.525 INFO:teuthology.orchestra.run.smithi032.stdout:e33 2024-04-12T05:57:18.525 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:57:18.525 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:57:18.525 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:57:18.525 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:57:18.525 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:57:18.525 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:57:18.525 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 24 2024-04-12T05:57:18.525 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:57:18.525 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:57:18.525 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:46:10.668569+0000 2024-04-12T05:57:18.525 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:57:18.525 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:57:18.525 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:57:18.526 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:57:18.526 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:57:18.526 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:57:18.526 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:57:18.526 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:57:18.526 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T05:57:18.526 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:57:18.526 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:57:18.526 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:57:18.526 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T05:57:18.526 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-12T05:57:18.526 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:57:18.526 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:57:18.527 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:57:18.527 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:57:18.527 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:57:18.527 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:57:18.527 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:57:18.527 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:57:18.527 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:57:18.527 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:57:18.527 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-12T05:57:18.527 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:57:18.527 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:57:18.527 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:57:18.527 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{-1:44317} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:57:18.527 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:57:18.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.531+0000 7f13f524d700 1 -- 172.21.15.32:0/3307259195 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f13d806cc20 msgr2=0x7f13d806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:18.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.531+0000 7f13f524d700 1 --2- 172.21.15.32:0/3307259195 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f13d806cc20 0x7f13d806f0d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f13e0004f90 tx=0x7f13e0009e30 comp rx=0 tx=0).stop 2024-04-12T05:57:18.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.531+0000 7f13f524d700 1 -- 172.21.15.32:0/3307259195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13f0102710 msgr2=0x7f13f0112e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:18.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.531+0000 7f13f524d700 1 --2- 172.21.15.32:0/3307259195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13f0102710 0x7f13f0112e60 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f13e4002a50 tx=0x7f13e40046b0 comp rx=0 tx=0).stop 2024-04-12T05:57:18.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.531+0000 7f13f524d700 1 -- 172.21.15.32:0/3307259195 shutdown_connections 2024-04-12T05:57:18.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.531+0000 7f13f524d700 1 --2- 172.21.15.32:0/3307259195 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f13d806cc20 0x7f13d806f0d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:18.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.531+0000 7f13f524d700 1 --2- 172.21.15.32:0/3307259195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13f0101d70 0x7f13f0112920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:18.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.531+0000 7f13f524d700 1 --2- 172.21.15.32:0/3307259195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13f0102710 0x7f13f0112e60 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:18.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.531+0000 7f13f524d700 1 -- 172.21.15.32:0/3307259195 >> 172.21.15.32:0/3307259195 conn(0x7f13f0072bc0 msgr2=0x7f13f00faf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:57:18.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.532+0000 7f13f524d700 1 -- 172.21.15.32:0/3307259195 shutdown_connections 2024-04-12T05:57:18.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.532+0000 7f13f524d700 1 -- 172.21.15.32:0/3307259195 wait complete. 2024-04-12T05:57:18.529 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-12T05:57:18.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:18 smithi032 ceph-mon[146817]: from='client.34956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:57:18.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:18 smithi032 ceph-mon[146817]: from='client.34960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:57:18.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:18 smithi032 ceph-mon[146817]: from='client.34964 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:57:18.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:57:18.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:18 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/408133329' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:57:18.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.636+0000 7f13ffa20700 1 -- 172.21.15.32:0/125136818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13f8102710 msgr2=0x7f13f80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:18.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.636+0000 7f13ffa20700 1 --2- 172.21.15.32:0/125136818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13f8102710 0x7f13f80fbd80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f13e8004940 tx=0x7f13e8009e30 comp rx=0 tx=0).stop 2024-04-12T05:57:18.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.636+0000 7f13ffa20700 1 -- 172.21.15.32:0/125136818 shutdown_connections 2024-04-12T05:57:18.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.636+0000 7f13ffa20700 1 --2- 172.21.15.32:0/125136818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13f8102710 0x7f13f80fbd80 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:18.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.636+0000 7f13ffa20700 1 --2- 172.21.15.32:0/125136818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13f8101d70 0x7f13f8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:18.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.636+0000 7f13ffa20700 1 -- 172.21.15.32:0/125136818 >> 172.21.15.32:0/125136818 conn(0x7f13f8072bc0 msgr2=0x7f13f8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:57:18.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.636+0000 7f13ffa20700 1 -- 172.21.15.32:0/125136818 shutdown_connections 2024-04-12T05:57:18.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.636+0000 7f13ffa20700 1 -- 172.21.15.32:0/125136818 wait complete. 2024-04-12T05:57:18.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.637+0000 7f13ffa20700 1 Processor -- start 2024-04-12T05:57:18.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.637+0000 7f13ffa20700 1 -- start start 2024-04-12T05:57:18.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.637+0000 7f13ffa20700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13f8101d70 0x7f13f8113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:18.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.637+0000 7f13ffa20700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13f8102710 0x7f13f8114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:18.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.637+0000 7f13ffa20700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f8114650 con 0x7f13f8102710 2024-04-12T05:57:18.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.637+0000 7f13ffa20700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f810da40 con 0x7f13f8101d70 2024-04-12T05:57:18.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.637+0000 7f13fcfbb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13f8102710 0x7f13f8114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:18.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.637+0000 7f13fcfbb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13f8102710 0x7f13f8114000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44768/0 (socket says 172.21.15.32:44768) 2024-04-12T05:57:18.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.637+0000 7f13fcfbb700 1 -- 172.21.15.32:0/369669106 learned_addr learned my addr 172.21.15.32:0/369669106 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:57:18.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.637+0000 7f13fd7bc700 1 --2- 172.21.15.32:0/369669106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13f8101d70 0x7f13f8113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:18.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.637+0000 7f13fcfbb700 1 -- 172.21.15.32:0/369669106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13f8101d70 msgr2=0x7f13f8113ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:18.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.637+0000 7f13fcfbb700 1 --2- 172.21.15.32:0/369669106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13f8101d70 0x7f13f8113ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:18.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.637+0000 7f13fcfbb700 1 -- 172.21.15.32:0/369669106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e8005040 con 0x7f13f8102710 2024-04-12T05:57:18.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.637+0000 7f13fcfbb700 1 --2- 172.21.15.32:0/369669106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13f8102710 0x7f13f8114000 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f13e8009390 tx=0x7f13e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:57:18.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.638+0000 7f13edffb700 1 -- 172.21.15.32:0/369669106 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13e800e040 con 0x7f13f8102710 2024-04-12T05:57:18.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.638+0000 7f13edffb700 1 -- 172.21.15.32:0/369669106 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f13e8013420 con 0x7f13f8102710 2024-04-12T05:57:18.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.638+0000 7f13edffb700 1 -- 172.21.15.32:0/369669106 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13e801c630 con 0x7f13f8102710 2024-04-12T05:57:18.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.638+0000 7f13ffa20700 1 -- 172.21.15.32:0/369669106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13f810dcc0 con 0x7f13f8102710 2024-04-12T05:57:18.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.638+0000 7f13ffa20700 1 -- 172.21.15.32:0/369669106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13f810e280 con 0x7f13f8102710 2024-04-12T05:57:18.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.639+0000 7f13ffa20700 1 -- 172.21.15.32:0/369669106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13f80fd4c0 con 0x7f13f8102710 2024-04-12T05:57:18.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.639+0000 7f13edffb700 1 -- 172.21.15.32:0/369669106 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f13e8013a00 con 0x7f13f8102710 2024-04-12T05:57:18.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.640+0000 7f13edffb700 1 --2- 172.21.15.32:0/369669106 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f13e406cc30 0x7f13e406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:18.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.640+0000 7f13edffb700 1 -- 172.21.15.32:0/369669106 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f13e808d290 con 0x7f13f8102710 2024-04-12T05:57:18.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.640+0000 7f13fd7bc700 1 --2- 172.21.15.32:0/369669106 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f13e406cc30 0x7f13e406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:18.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.640+0000 7f13fd7bc700 1 --2- 172.21.15.32:0/369669106 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f13e406cc30 0x7f13e406f0e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f13f4007fd0 tx=0x7f13f4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:57:18.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.644+0000 7f13edffb700 1 -- 172.21.15.32:0/369669106 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f13e805ada0 con 0x7f13f8102710 2024-04-12T05:57:18.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.866+0000 7f13ffa20700 1 -- 172.21.15.32:0/369669106 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f13f8108d10 con 0x7f13e406cc30 2024-04-12T05:57:18.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.867+0000 7f13edffb700 1 -- 172.21.15.32:0/369669106 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f13f8108d10 con 0x7f13e406cc30 2024-04-12T05:57:18.862 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:57:18.862 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:57:18.862 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:57:18.862 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:57:18.862 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:57:18.863 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-12T05:57:18.863 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:57:18.863 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-12T05:57:18.863 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:57:18.863 INFO:teuthology.orchestra.run.smithi032.stdout: "mds" 2024-04-12T05:57:18.863 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:57:18.863 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:57:18.863 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:57:18.863 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:57:18.863 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:57:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.869+0000 7f13ffa20700 1 -- 172.21.15.32:0/369669106 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f13e406cc30 msgr2=0x7f13e406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.869+0000 7f13ffa20700 1 --2- 172.21.15.32:0/369669106 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f13e406cc30 0x7f13e406f0e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f13f4007fd0 tx=0x7f13f4009040 comp rx=0 tx=0).stop 2024-04-12T05:57:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.869+0000 7f13ffa20700 1 -- 172.21.15.32:0/369669106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13f8102710 msgr2=0x7f13f8114000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.869+0000 7f13ffa20700 1 --2- 172.21.15.32:0/369669106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13f8102710 0x7f13f8114000 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f13e8009390 tx=0x7f13e8009e30 comp rx=0 tx=0).stop 2024-04-12T05:57:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.869+0000 7f13ffa20700 1 -- 172.21.15.32:0/369669106 shutdown_connections 2024-04-12T05:57:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.869+0000 7f13ffa20700 1 --2- 172.21.15.32:0/369669106 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f13e406cc30 0x7f13e406f0e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.869+0000 7f13ffa20700 1 --2- 172.21.15.32:0/369669106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13f8101d70 0x7f13f8113ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.869+0000 7f13ffa20700 1 --2- 172.21.15.32:0/369669106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13f8102710 0x7f13f8114000 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.869+0000 7f13ffa20700 1 -- 172.21.15.32:0/369669106 >> 172.21.15.32:0/369669106 conn(0x7f13f8072bc0 msgr2=0x7f13f8065a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:57:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.869+0000 7f13ffa20700 1 -- 172.21.15.32:0/369669106 shutdown_connections 2024-04-12T05:57:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.869+0000 7f13ffa20700 1 -- 172.21.15.32:0/369669106 wait complete. 2024-04-12T05:57:18.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.973+0000 7f26ec3f6700 1 -- 172.21.15.32:0/2315001769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26e4102e10 msgr2=0x7f26e41031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:18.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.973+0000 7f26ec3f6700 1 --2- 172.21.15.32:0/2315001769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26e4102e10 0x7f26e41031e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f26e0004940 tx=0x7f26e0009e30 comp rx=0 tx=0).stop 2024-04-12T05:57:18.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.973+0000 7f26ec3f6700 1 -- 172.21.15.32:0/2315001769 shutdown_connections 2024-04-12T05:57:18.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.973+0000 7f26ec3f6700 1 --2- 172.21.15.32:0/2315001769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e41037b0 0x7f26e40fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:18.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.973+0000 7f26ec3f6700 1 --2- 172.21.15.32:0/2315001769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26e4102e10 0x7f26e41031e0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:18.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.973+0000 7f26ec3f6700 1 -- 172.21.15.32:0/2315001769 >> 172.21.15.32:0/2315001769 conn(0x7f26e40f8980 msgr2=0x7f26e40fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:57:18.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.973+0000 7f26ec3f6700 1 -- 172.21.15.32:0/2315001769 shutdown_connections 2024-04-12T05:57:18.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.973+0000 7f26ec3f6700 1 -- 172.21.15.32:0/2315001769 wait complete. 2024-04-12T05:57:18.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.973+0000 7f26ec3f6700 1 Processor -- start 2024-04-12T05:57:18.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.973+0000 7f26ec3f6700 1 -- start start 2024-04-12T05:57:18.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.973+0000 7f26ec3f6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26e4102e10 0x7f26e4113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:18.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.973+0000 7f26ec3f6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e41037b0 0x7f26e4114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:18.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.973+0000 7f26ec3f6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e4114660 con 0x7f26e4102e10 2024-04-12T05:57:18.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.974+0000 7f26ec3f6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e410da50 con 0x7f26e41037b0 2024-04-12T05:57:18.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.974+0000 7f26ea192700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26e4102e10 0x7f26e4113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:18.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.974+0000 7f26ea192700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26e4102e10 0x7f26e4113ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44782/0 (socket says 172.21.15.32:44782) 2024-04-12T05:57:18.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.974+0000 7f26ea192700 1 -- 172.21.15.32:0/29388210 learned_addr learned my addr 172.21.15.32:0/29388210 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:57:18.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.974+0000 7f26e9991700 1 --2- 172.21.15.32:0/29388210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e41037b0 0x7f26e4114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:18.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.974+0000 7f26ea192700 1 -- 172.21.15.32:0/29388210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e41037b0 msgr2=0x7f26e4114010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:18.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.974+0000 7f26ea192700 1 --2- 172.21.15.32:0/29388210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e41037b0 0x7f26e4114010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:18.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.974+0000 7f26ea192700 1 -- 172.21.15.32:0/29388210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e0005040 con 0x7f26e4102e10 2024-04-12T05:57:18.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.974+0000 7f26ea192700 1 --2- 172.21.15.32:0/29388210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26e4102e10 0x7f26e4113ad0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f26e0009360 tx=0x7f26e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:57:18.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.974+0000 7f26daffd700 1 -- 172.21.15.32:0/29388210 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26e000e040 con 0x7f26e4102e10 2024-04-12T05:57:18.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.974+0000 7f26daffd700 1 -- 172.21.15.32:0/29388210 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f26e0013420 con 0x7f26e4102e10 2024-04-12T05:57:18.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.974+0000 7f26ec3f6700 1 -- 172.21.15.32:0/29388210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e410dcd0 con 0x7f26e4102e10 2024-04-12T05:57:18.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.974+0000 7f26daffd700 1 -- 172.21.15.32:0/29388210 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26e001c5d0 con 0x7f26e4102e10 2024-04-12T05:57:18.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.974+0000 7f26ec3f6700 1 -- 172.21.15.32:0/29388210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e410e290 con 0x7f26e4102e10 2024-04-12T05:57:18.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.975+0000 7f26ec3f6700 1 -- 172.21.15.32:0/29388210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e40fe560 con 0x7f26e4102e10 2024-04-12T05:57:18.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.976+0000 7f26daffd700 1 -- 172.21.15.32:0/29388210 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f26e0013590 con 0x7f26e4102e10 2024-04-12T05:57:18.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.976+0000 7f26daffd700 1 --2- 172.21.15.32:0/29388210 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f26d006cc30 0x7f26d006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:18.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.976+0000 7f26e9991700 1 --2- 172.21.15.32:0/29388210 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f26d006cc30 0x7f26d006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:18.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.976+0000 7f26daffd700 1 -- 172.21.15.32:0/29388210 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f26e008d1a0 con 0x7f26e4102e10 2024-04-12T05:57:18.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.977+0000 7f26e9991700 1 --2- 172.21.15.32:0/29388210 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f26d006cc30 0x7f26d006f0e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f26e410ec10 tx=0x7f26d40043c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:57:18.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:18.981+0000 7f26daffd700 1 -- 172.21.15.32:0/29388210 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f26e005ac10 con 0x7f26e4102e10 2024-04-12T05:57:19.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:19.268+0000 7f26ec3f6700 1 -- 172.21.15.32:0/29388210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f26e4109dd0 con 0x7f26e4102e10 2024-04-12T05:57:19.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:19.268+0000 7f26daffd700 1 -- 172.21.15.32:0/29388210 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f26e005a7a0 con 0x7f26e4102e10 2024-04-12T05:57:19.263 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:57:19.263 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:57:19.264 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-12T05:57:19.264 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:57:19.264 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:57:19.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:19.270+0000 7f26ec3f6700 1 -- 172.21.15.32:0/29388210 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f26d006cc30 msgr2=0x7f26d006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:19.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:19.270+0000 7f26ec3f6700 1 --2- 172.21.15.32:0/29388210 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f26d006cc30 0x7f26d006f0e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f26e410ec10 tx=0x7f26d40043c0 comp rx=0 tx=0).stop 2024-04-12T05:57:19.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:19.270+0000 7f26ec3f6700 1 -- 172.21.15.32:0/29388210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26e4102e10 msgr2=0x7f26e4113ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:19.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:19.270+0000 7f26ec3f6700 1 --2- 172.21.15.32:0/29388210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26e4102e10 0x7f26e4113ad0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f26e0009360 tx=0x7f26e0009e30 comp rx=0 tx=0).stop 2024-04-12T05:57:19.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:19.270+0000 7f26ec3f6700 1 -- 172.21.15.32:0/29388210 shutdown_connections 2024-04-12T05:57:19.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:19.270+0000 7f26ec3f6700 1 --2- 172.21.15.32:0/29388210 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f26d006cc30 0x7f26d006f0e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:19.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:19.270+0000 7f26ec3f6700 1 --2- 172.21.15.32:0/29388210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26e4102e10 0x7f26e4113ad0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:19.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:19.270+0000 7f26ec3f6700 1 --2- 172.21.15.32:0/29388210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e41037b0 0x7f26e4114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:19.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:19.270+0000 7f26ec3f6700 1 -- 172.21.15.32:0/29388210 >> 172.21.15.32:0/29388210 conn(0x7f26e40f8980 msgr2=0x7f26e40f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:57:19.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:19.270+0000 7f26ec3f6700 1 -- 172.21.15.32:0/29388210 shutdown_connections 2024-04-12T05:57:19.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:19.270+0000 7f26ec3f6700 1 -- 172.21.15.32:0/29388210 wait complete. 2024-04-12T05:57:19.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:19 smithi067 ceph-mon[112979]: pgmap v504: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:19.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:19 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3307259195' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:57:19.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:19 smithi032 ceph-mon[146817]: pgmap v504: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:19.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:19 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3307259195' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:57:20.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:20 smithi067 ceph-mon[112979]: from='client.34976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:57:20.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:20 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/29388210' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:57:20.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:20 smithi032 ceph-mon[146817]: from='client.34976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:57:20.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:20 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/29388210' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:57:21.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:21 smithi067 ceph-mon[112979]: pgmap v505: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:21.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:21 smithi032 ceph-mon[146817]: pgmap v505: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:23.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:23 smithi067 ceph-mon[112979]: pgmap v506: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:23.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:23 smithi032 ceph-mon[146817]: pgmap v506: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:25.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:25 smithi067 ceph-mon[112979]: pgmap v507: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:25.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:25 smithi032 ceph-mon[146817]: pgmap v507: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:27.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:27 smithi067 ceph-mon[112979]: pgmap v508: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:27.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:27 smithi032 ceph-mon[146817]: pgmap v508: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:29.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:29 smithi067 ceph-mon[112979]: pgmap v509: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:29.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:29 smithi032 ceph-mon[146817]: pgmap v509: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:31.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:31 smithi067 ceph-mon[112979]: pgmap v510: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:31.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:31 smithi032 ceph-mon[146817]: pgmap v510: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:33.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:33 smithi067 ceph-mon[112979]: pgmap v511: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:33.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:33 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:57:33.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:33 smithi032 ceph-mon[146817]: pgmap v511: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:33.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:33 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:57:35.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:35 smithi067 ceph-mon[112979]: pgmap v512: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:35.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:35 smithi032 ceph-mon[146817]: pgmap v512: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:37.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:37 smithi067 ceph-mon[112979]: pgmap v513: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:37.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:37 smithi032 ceph-mon[146817]: pgmap v513: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:39.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:39 smithi067 ceph-mon[112979]: pgmap v514: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:39.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:39 smithi032 ceph-mon[146817]: pgmap v514: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:41.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:41 smithi067 ceph-mon[112979]: pgmap v515: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:41.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:41 smithi032 ceph-mon[146817]: pgmap v515: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:43.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:43 smithi067 ceph-mon[112979]: pgmap v516: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:43.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:43 smithi032 ceph-mon[146817]: pgmap v516: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:45.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:45 smithi067 ceph-mon[112979]: pgmap v517: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:45.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:45 smithi032 ceph-mon[146817]: pgmap v517: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:47.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:47 smithi067 ceph-mon[112979]: pgmap v518: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:47.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:47 smithi032 ceph-mon[146817]: pgmap v518: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:48.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:48 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:57:48.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:48 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:57:49.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.388+0000 7f34aa17b700 1 -- 172.21.15.32:0/3928932862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34a40fe830 msgr2=0x7f34a4106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:49.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.388+0000 7f34aa17b700 1 --2- 172.21.15.32:0/3928932862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34a40fe830 0x7f34a4106f00 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f3498004940 tx=0x7f3498009e30 comp rx=0 tx=0).stop 2024-04-12T05:57:49.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.388+0000 7f34aa17b700 1 -- 172.21.15.32:0/3928932862 shutdown_connections 2024-04-12T05:57:49.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.388+0000 7f34aa17b700 1 --2- 172.21.15.32:0/3928932862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34a40fe830 0x7f34a4106f00 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:49.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.388+0000 7f34aa17b700 1 --2- 172.21.15.32:0/3928932862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34a40fde90 0x7f34a40fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:49.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.388+0000 7f34aa17b700 1 -- 172.21.15.32:0/3928932862 >> 172.21.15.32:0/3928932862 conn(0x7f34a4072bc0 msgr2=0x7f34a4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:57:49.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.388+0000 7f34aa17b700 1 -- 172.21.15.32:0/3928932862 shutdown_connections 2024-04-12T05:57:49.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.388+0000 7f34aa17b700 1 -- 172.21.15.32:0/3928932862 wait complete. 2024-04-12T05:57:49.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.389+0000 7f34aa17b700 1 Processor -- start 2024-04-12T05:57:49.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.389+0000 7f34aa17b700 1 -- start start 2024-04-12T05:57:49.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.389+0000 7f34aa17b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34a40fde90 0x7f34a40fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:49.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.389+0000 7f34aa17b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34a40fe830 0x7f34a40fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:49.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.389+0000 7f34aa17b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a40fc820 con 0x7f34a40fde90 2024-04-12T05:57:49.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.389+0000 7f34aa17b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a40fc960 con 0x7f34a40fe830 2024-04-12T05:57:49.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.389+0000 7f34a37fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34a40fde90 0x7f34a40fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:49.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.389+0000 7f34a37fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34a40fde90 0x7f34a40fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34936/0 (socket says 172.21.15.32:34936) 2024-04-12T05:57:49.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.389+0000 7f34a37fe700 1 -- 172.21.15.32:0/14817971 learned_addr learned my addr 172.21.15.32:0/14817971 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:57:49.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.390+0000 7f34a2ffd700 1 --2- 172.21.15.32:0/14817971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34a40fe830 0x7f34a40fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:49.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.390+0000 7f34a37fe700 1 -- 172.21.15.32:0/14817971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34a40fe830 msgr2=0x7f34a40fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:49.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.390+0000 7f34a37fe700 1 --2- 172.21.15.32:0/14817971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34a40fe830 0x7f34a40fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:49.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.390+0000 7f34a37fe700 1 -- 172.21.15.32:0/14817971 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3498005040 con 0x7f34a40fde90 2024-04-12T05:57:49.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.390+0000 7f34a37fe700 1 --2- 172.21.15.32:0/14817971 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34a40fde90 0x7f34a40fbd50 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f349400eb40 tx=0x7f349400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:57:49.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.390+0000 7f348bfff700 1 -- 172.21.15.32:0/14817971 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3494007cc0 con 0x7f34a40fde90 2024-04-12T05:57:49.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.390+0000 7f348bfff700 1 -- 172.21.15.32:0/14817971 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f349400f040 con 0x7f34a40fde90 2024-04-12T05:57:49.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.390+0000 7f34aa17b700 1 -- 172.21.15.32:0/14817971 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a40fcbb0 con 0x7f34a40fde90 2024-04-12T05:57:49.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.390+0000 7f348bfff700 1 -- 172.21.15.32:0/14817971 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f349400c990 con 0x7f34a40fde90 2024-04-12T05:57:49.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.390+0000 7f34aa17b700 1 -- 172.21.15.32:0/14817971 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34a411a130 con 0x7f34a40fde90 2024-04-12T05:57:49.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.391+0000 7f34aa17b700 1 -- 172.21.15.32:0/14817971 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34a4065e30 con 0x7f34a40fde90 2024-04-12T05:57:49.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.392+0000 7f348bfff700 1 -- 172.21.15.32:0/14817971 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f349400cc00 con 0x7f34a40fde90 2024-04-12T05:57:49.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.392+0000 7f348bfff700 1 --2- 172.21.15.32:0/14817971 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f348c06cc20 0x7f348c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:49.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.392+0000 7f348bfff700 1 -- 172.21.15.32:0/14817971 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f349408bca0 con 0x7f34a40fde90 2024-04-12T05:57:49.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.392+0000 7f34a2ffd700 1 --2- 172.21.15.32:0/14817971 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f348c06cc20 0x7f348c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:49.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.393+0000 7f34a2ffd700 1 --2- 172.21.15.32:0/14817971 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f348c06cc20 0x7f348c06f0d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f34a40fd690 tx=0x7f3498009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:57:49.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.397+0000 7f348bfff700 1 -- 172.21.15.32:0/14817971 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3494090070 con 0x7f34a40fde90 2024-04-12T05:57:49.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:49 smithi032 ceph-mon[146817]: pgmap v519: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:49.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.619+0000 7f34aa17b700 1 -- 172.21.15.32:0/14817971 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f34a40fd440 con 0x7f348c06cc20 2024-04-12T05:57:49.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.620+0000 7f348bfff700 1 -- 172.21.15.32:0/14817971 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f34a40fd440 con 0x7f348c06cc20 2024-04-12T05:57:49.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.622+0000 7f34aa17b700 1 -- 172.21.15.32:0/14817971 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f348c06cc20 msgr2=0x7f348c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:49.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.622+0000 7f34aa17b700 1 --2- 172.21.15.32:0/14817971 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f348c06cc20 0x7f348c06f0d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f34a40fd690 tx=0x7f3498009e30 comp rx=0 tx=0).stop 2024-04-12T05:57:49.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.622+0000 7f34aa17b700 1 -- 172.21.15.32:0/14817971 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34a40fde90 msgr2=0x7f34a40fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:49.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.622+0000 7f34aa17b700 1 --2- 172.21.15.32:0/14817971 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34a40fde90 0x7f34a40fbd50 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f349400eb40 tx=0x7f349400b040 comp rx=0 tx=0).stop 2024-04-12T05:57:49.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.623+0000 7f34aa17b700 1 -- 172.21.15.32:0/14817971 shutdown_connections 2024-04-12T05:57:49.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.623+0000 7f34aa17b700 1 --2- 172.21.15.32:0/14817971 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f348c06cc20 0x7f348c06f0d0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:49.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.623+0000 7f34aa17b700 1 --2- 172.21.15.32:0/14817971 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34a40fde90 0x7f34a40fbd50 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:49.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.623+0000 7f34aa17b700 1 --2- 172.21.15.32:0/14817971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34a40fe830 0x7f34a40fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:49.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.623+0000 7f34aa17b700 1 -- 172.21.15.32:0/14817971 >> 172.21.15.32:0/14817971 conn(0x7f34a4072bc0 msgr2=0x7f34a40ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:57:49.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.623+0000 7f34aa17b700 1 -- 172.21.15.32:0/14817971 shutdown_connections 2024-04-12T05:57:49.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.623+0000 7f34aa17b700 1 -- 172.21.15.32:0/14817971 wait complete. 2024-04-12T05:57:49.627 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:57:49.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.727+0000 7f3f21505700 1 -- 172.21.15.32:0/654309017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f1c102710 msgr2=0x7f3f1c0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:49.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.727+0000 7f3f21505700 1 --2- 172.21.15.32:0/654309017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f1c102710 0x7f3f1c0fbd80 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f3f04004940 tx=0x7f3f04009e30 comp rx=0 tx=0).stop 2024-04-12T05:57:49.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.727+0000 7f3f21505700 1 -- 172.21.15.32:0/654309017 shutdown_connections 2024-04-12T05:57:49.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.727+0000 7f3f21505700 1 --2- 172.21.15.32:0/654309017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f1c102710 0x7f3f1c0fbd80 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:49.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.727+0000 7f3f21505700 1 --2- 172.21.15.32:0/654309017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f1c101d70 0x7f3f1c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:49.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.727+0000 7f3f21505700 1 -- 172.21.15.32:0/654309017 >> 172.21.15.32:0/654309017 conn(0x7f3f1c072bc0 msgr2=0x7f3f1c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:57:49.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.727+0000 7f3f21505700 1 -- 172.21.15.32:0/654309017 shutdown_connections 2024-04-12T05:57:49.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.727+0000 7f3f21505700 1 -- 172.21.15.32:0/654309017 wait complete. 2024-04-12T05:57:49.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.727+0000 7f3f21505700 1 Processor -- start 2024-04-12T05:57:49.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.728+0000 7f3f21505700 1 -- start start 2024-04-12T05:57:49.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.728+0000 7f3f21505700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f1c101d70 0x7f3f1c113aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:49.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.728+0000 7f3f21505700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f1c102710 0x7f3f1c113fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:49.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.728+0000 7f3f21505700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1c114630 con 0x7f3f1c102710 2024-04-12T05:57:49.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.728+0000 7f3f21505700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1c10da20 con 0x7f3f1c101d70 2024-04-12T05:57:49.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.728+0000 7f3f1a7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f1c102710 0x7f3f1c113fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:49.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.728+0000 7f3f1a7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f1c102710 0x7f3f1c113fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34948/0 (socket says 172.21.15.32:34948) 2024-04-12T05:57:49.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.728+0000 7f3f1a7fc700 1 -- 172.21.15.32:0/1020028931 learned_addr learned my addr 172.21.15.32:0/1020028931 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:57:49.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.728+0000 7f3f1affd700 1 --2- 172.21.15.32:0/1020028931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f1c101d70 0x7f3f1c113aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:49.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.728+0000 7f3f1a7fc700 1 -- 172.21.15.32:0/1020028931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f1c101d70 msgr2=0x7f3f1c113aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:49.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.728+0000 7f3f1a7fc700 1 --2- 172.21.15.32:0/1020028931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f1c101d70 0x7f3f1c113aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:49.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.728+0000 7f3f1a7fc700 1 -- 172.21.15.32:0/1020028931 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f04005040 con 0x7f3f1c102710 2024-04-12T05:57:49.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.728+0000 7f3f1affd700 1 --2- 172.21.15.32:0/1020028931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f1c101d70 0x7f3f1c113aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:57:49.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.728+0000 7f3f1a7fc700 1 --2- 172.21.15.32:0/1020028931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f1c102710 0x7f3f1c113fe0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f3f04009390 tx=0x7f3f04009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:57:49.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.728+0000 7f3f137fe700 1 -- 172.21.15.32:0/1020028931 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f0400e040 con 0x7f3f1c102710 2024-04-12T05:57:49.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.729+0000 7f3f137fe700 1 -- 172.21.15.32:0/1020028931 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f04013420 con 0x7f3f1c102710 2024-04-12T05:57:49.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.729+0000 7f3f137fe700 1 -- 172.21.15.32:0/1020028931 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f0401c630 con 0x7f3f1c102710 2024-04-12T05:57:49.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.729+0000 7f3f21505700 1 -- 172.21.15.32:0/1020028931 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f1c10dca0 con 0x7f3f1c102710 2024-04-12T05:57:49.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.729+0000 7f3f21505700 1 -- 172.21.15.32:0/1020028931 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f1c10e260 con 0x7f3f1c102710 2024-04-12T05:57:49.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.730+0000 7f3f21505700 1 -- 172.21.15.32:0/1020028931 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f1c0fd4c0 con 0x7f3f1c102710 2024-04-12T05:57:49.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.730+0000 7f3f137fe700 1 -- 172.21.15.32:0/1020028931 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3f040139e0 con 0x7f3f1c102710 2024-04-12T05:57:49.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.731+0000 7f3f137fe700 1 --2- 172.21.15.32:0/1020028931 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3f0806cc30 0x7f3f0806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:49.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.731+0000 7f3f137fe700 1 -- 172.21.15.32:0/1020028931 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f3f0408d280 con 0x7f3f1c102710 2024-04-12T05:57:49.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.731+0000 7f3f1affd700 1 --2- 172.21.15.32:0/1020028931 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3f0806cc30 0x7f3f0806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:49.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.731+0000 7f3f1affd700 1 --2- 172.21.15.32:0/1020028931 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3f0806cc30 0x7f3f0806f0e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c001250 tx=0x7f3f0c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:57:49.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.735+0000 7f3f137fe700 1 -- 172.21.15.32:0/1020028931 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3f0405ad90 con 0x7f3f1c102710 2024-04-12T05:57:49.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:49 smithi067 ceph-mon[112979]: pgmap v519: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:49.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.956+0000 7f3f21505700 1 -- 172.21.15.32:0/1020028931 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f1c0666a0 con 0x7f3f0806cc30 2024-04-12T05:57:49.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.957+0000 7f3f137fe700 1 -- 172.21.15.32:0/1020028931 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3f1c0666a0 con 0x7f3f0806cc30 2024-04-12T05:57:49.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.959+0000 7f3f21505700 1 -- 172.21.15.32:0/1020028931 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3f0806cc30 msgr2=0x7f3f0806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:49.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.959+0000 7f3f21505700 1 --2- 172.21.15.32:0/1020028931 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3f0806cc30 0x7f3f0806f0e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c001250 tx=0x7f3f0c00b040 comp rx=0 tx=0).stop 2024-04-12T05:57:49.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.959+0000 7f3f21505700 1 -- 172.21.15.32:0/1020028931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f1c102710 msgr2=0x7f3f1c113fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:49.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.959+0000 7f3f21505700 1 --2- 172.21.15.32:0/1020028931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f1c102710 0x7f3f1c113fe0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f3f04009390 tx=0x7f3f04009e30 comp rx=0 tx=0).stop 2024-04-12T05:57:49.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.960+0000 7f3f21505700 1 -- 172.21.15.32:0/1020028931 shutdown_connections 2024-04-12T05:57:49.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.960+0000 7f3f21505700 1 --2- 172.21.15.32:0/1020028931 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3f0806cc30 0x7f3f0806f0e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:49.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.960+0000 7f3f21505700 1 --2- 172.21.15.32:0/1020028931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f1c101d70 0x7f3f1c113aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:49.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.960+0000 7f3f21505700 1 --2- 172.21.15.32:0/1020028931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f1c102710 0x7f3f1c113fe0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:49.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.960+0000 7f3f21505700 1 -- 172.21.15.32:0/1020028931 >> 172.21.15.32:0/1020028931 conn(0x7f3f1c072bc0 msgr2=0x7f3f1c1058c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:57:49.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.960+0000 7f3f21505700 1 -- 172.21.15.32:0/1020028931 shutdown_connections 2024-04-12T05:57:49.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:49.960+0000 7f3f21505700 1 -- 172.21.15.32:0/1020028931 wait complete. 2024-04-12T05:57:50.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.063+0000 7f5870f0c700 1 -- 172.21.15.32:0/1604262508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f586c101d70 msgr2=0x7f586c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:50.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.063+0000 7f5870f0c700 1 --2- 172.21.15.32:0/1604262508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f586c101d70 0x7f586c102140 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f585c004940 tx=0x7f585c009e30 comp rx=0 tx=0).stop 2024-04-12T05:57:50.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.063+0000 7f5870f0c700 1 -- 172.21.15.32:0/1604262508 shutdown_connections 2024-04-12T05:57:50.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.063+0000 7f5870f0c700 1 --2- 172.21.15.32:0/1604262508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f586c102710 0x7f586c0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:50.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.063+0000 7f5870f0c700 1 --2- 172.21.15.32:0/1604262508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f586c101d70 0x7f586c102140 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:50.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.063+0000 7f5870f0c700 1 -- 172.21.15.32:0/1604262508 >> 172.21.15.32:0/1604262508 conn(0x7f586c072bc0 msgr2=0x7f586c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:57:50.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.063+0000 7f5870f0c700 1 -- 172.21.15.32:0/1604262508 shutdown_connections 2024-04-12T05:57:50.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.063+0000 7f5870f0c700 1 -- 172.21.15.32:0/1604262508 wait complete. 2024-04-12T05:57:50.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.064+0000 7f5870f0c700 1 Processor -- start 2024-04-12T05:57:50.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.064+0000 7f5870f0c700 1 -- start start 2024-04-12T05:57:50.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.064+0000 7f5870f0c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f586c101d70 0x7f586c113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:50.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.064+0000 7f5870f0c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f586c102710 0x7f586c114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:50.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.064+0000 7f5870f0c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f586c114670 con 0x7f586c102710 2024-04-12T05:57:50.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.064+0000 7f5870f0c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f586c10da60 con 0x7f586c101d70 2024-04-12T05:57:50.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.064+0000 7f5869d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f586c102710 0x7f586c114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:50.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.064+0000 7f5869d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f586c102710 0x7f586c114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34966/0 (socket says 172.21.15.32:34966) 2024-04-12T05:57:50.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.064+0000 7f5869d9b700 1 -- 172.21.15.32:0/3670307548 learned_addr learned my addr 172.21.15.32:0/3670307548 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:57:50.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.064+0000 7f586a59c700 1 --2- 172.21.15.32:0/3670307548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f586c101d70 0x7f586c113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:50.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.064+0000 7f5869d9b700 1 -- 172.21.15.32:0/3670307548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f586c101d70 msgr2=0x7f586c113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:50.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.064+0000 7f5869d9b700 1 --2- 172.21.15.32:0/3670307548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f586c101d70 0x7f586c113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:50.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.064+0000 7f5869d9b700 1 -- 172.21.15.32:0/3670307548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f585c005040 con 0x7f586c102710 2024-04-12T05:57:50.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.064+0000 7f586a59c700 1 --2- 172.21.15.32:0/3670307548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f586c101d70 0x7f586c113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:57:50.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.065+0000 7f5869d9b700 1 --2- 172.21.15.32:0/3670307548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f586c102710 0x7f586c114020 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f585400c800 tx=0x7f5854004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:57:50.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.065+0000 7f5862ffd700 1 -- 172.21.15.32:0/3670307548 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f585400a760 con 0x7f586c102710 2024-04-12T05:57:50.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.065+0000 7f5862ffd700 1 -- 172.21.15.32:0/3670307548 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f585400ad70 con 0x7f586c102710 2024-04-12T05:57:50.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.065+0000 7f5862ffd700 1 -- 172.21.15.32:0/3670307548 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58540113a0 con 0x7f586c102710 2024-04-12T05:57:50.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.065+0000 7f5870f0c700 1 -- 172.21.15.32:0/3670307548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f586c10dd40 con 0x7f586c102710 2024-04-12T05:57:50.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.065+0000 7f5870f0c700 1 -- 172.21.15.32:0/3670307548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f586c10e300 con 0x7f586c102710 2024-04-12T05:57:50.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.066+0000 7f5870f0c700 1 -- 172.21.15.32:0/3670307548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f586c0fd4c0 con 0x7f586c102710 2024-04-12T05:57:50.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.066+0000 7f5862ffd700 1 -- 172.21.15.32:0/3670307548 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f5854012400 con 0x7f586c102710 2024-04-12T05:57:50.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.067+0000 7f5862ffd700 1 --2- 172.21.15.32:0/3670307548 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f585806cc30 0x7f585806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:50.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.067+0000 7f5862ffd700 1 -- 172.21.15.32:0/3670307548 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f585408c0e0 con 0x7f586c102710 2024-04-12T05:57:50.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.067+0000 7f586a59c700 1 --2- 172.21.15.32:0/3670307548 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f585806cc30 0x7f585806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:50.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.067+0000 7f586a59c700 1 --2- 172.21.15.32:0/3670307548 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f585806cc30 0x7f585806f0e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f585c000c00 tx=0x7f585c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:57:50.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.071+0000 7f5862ffd700 1 -- 172.21.15.32:0/3670307548 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5854059b30 con 0x7f586c102710 2024-04-12T05:57:50.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.285+0000 7f5870f0c700 1 -- 172.21.15.32:0/3670307548 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f586c108d30 con 0x7f585806cc30 2024-04-12T05:57:50.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.295+0000 7f5862ffd700 1 -- 172.21.15.32:0/3670307548 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f586c108d30 con 0x7f585806cc30 2024-04-12T05:57:50.289 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:57:50.289 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (17m) 11m ago 29m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:57:50.290 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (15m) 11m ago 29m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:57:50.290 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (15m) 10m ago 28m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:57:50.290 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (17m) 11m ago 28m 87.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:57:50.290 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (11m) 11m ago 23m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T05:57:50.290 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (11m) 11m ago 23m 13.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T05:57:50.290 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (10m) 10m ago 23m 11.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 6d4aca6f74a0 2024-04-12T05:57:50.290 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (10m) 10m ago 23m 14.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T05:57:50.290 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (19m) 11m ago 30m 507M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:57:50.290 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (18m) 10m ago 28m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:57:50.290 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (16m) 11m ago 30m 53.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:57:50.290 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (15m) 10m ago 28m 43.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:57:50.290 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (18m) 11m ago 29m 16.9M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:57:50.290 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (18m) 10m ago 28m 18.4M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:57:50.290 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (14m) 11m ago 27m 91.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T05:57:50.291 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (14m) 11m ago 26m 97.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T05:57:50.291 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (13m) 11m ago 26m 81.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T05:57:50.291 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (12m) 10m ago 25m 113M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T05:57:50.291 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (12m) 10m ago 25m 90.3M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T05:57:50.291 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (11m) 10m ago 24m 84.8M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T05:57:50.291 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (17m) 11m ago 28m 64.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:57:50.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.297+0000 7f5870f0c700 1 -- 172.21.15.32:0/3670307548 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f585806cc30 msgr2=0x7f585806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:50.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.297+0000 7f5870f0c700 1 --2- 172.21.15.32:0/3670307548 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f585806cc30 0x7f585806f0e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f585c000c00 tx=0x7f585c009e30 comp rx=0 tx=0).stop 2024-04-12T05:57:50.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.297+0000 7f5870f0c700 1 -- 172.21.15.32:0/3670307548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f586c102710 msgr2=0x7f586c114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:50.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.297+0000 7f5870f0c700 1 --2- 172.21.15.32:0/3670307548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f586c102710 0x7f586c114020 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f585400c800 tx=0x7f5854004470 comp rx=0 tx=0).stop 2024-04-12T05:57:50.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.297+0000 7f5870f0c700 1 -- 172.21.15.32:0/3670307548 shutdown_connections 2024-04-12T05:57:50.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.297+0000 7f5870f0c700 1 --2- 172.21.15.32:0/3670307548 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f585806cc30 0x7f585806f0e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:50.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.297+0000 7f5870f0c700 1 --2- 172.21.15.32:0/3670307548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f586c101d70 0x7f586c113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:50.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.297+0000 7f5870f0c700 1 --2- 172.21.15.32:0/3670307548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f586c102710 0x7f586c114020 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:50.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.297+0000 7f5870f0c700 1 -- 172.21.15.32:0/3670307548 >> 172.21.15.32:0/3670307548 conn(0x7f586c072bc0 msgr2=0x7f586c0659e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:57:50.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.297+0000 7f5870f0c700 1 -- 172.21.15.32:0/3670307548 shutdown_connections 2024-04-12T05:57:50.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.297+0000 7f5870f0c700 1 -- 172.21.15.32:0/3670307548 wait complete. 2024-04-12T05:57:50.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.399+0000 7f62e8fc0700 1 -- 172.21.15.32:0/2308735631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62e4101d70 msgr2=0x7f62e4102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:50.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.399+0000 7f62e8fc0700 1 --2- 172.21.15.32:0/2308735631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62e4101d70 0x7f62e4102140 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f62d4004940 tx=0x7f62d4009e30 comp rx=0 tx=0).stop 2024-04-12T05:57:50.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.399+0000 7f62e8fc0700 1 -- 172.21.15.32:0/2308735631 shutdown_connections 2024-04-12T05:57:50.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.399+0000 7f62e8fc0700 1 --2- 172.21.15.32:0/2308735631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62e4102710 0x7f62e40fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:50.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.399+0000 7f62e8fc0700 1 --2- 172.21.15.32:0/2308735631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62e4101d70 0x7f62e4102140 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:50.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.399+0000 7f62e8fc0700 1 -- 172.21.15.32:0/2308735631 >> 172.21.15.32:0/2308735631 conn(0x7f62e4072bc0 msgr2=0x7f62e4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:57:50.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.400+0000 7f62e8fc0700 1 -- 172.21.15.32:0/2308735631 shutdown_connections 2024-04-12T05:57:50.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.400+0000 7f62e8fc0700 1 -- 172.21.15.32:0/2308735631 wait complete. 2024-04-12T05:57:50.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.400+0000 7f62e8fc0700 1 Processor -- start 2024-04-12T05:57:50.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.400+0000 7f62e8fc0700 1 -- start start 2024-04-12T05:57:50.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.400+0000 7f62e8fc0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62e4101d70 0x7f62e410ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:50.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.400+0000 7f62e8fc0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62e4102710 0x7f62e410ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:50.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.400+0000 7f62e8fc0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e410f5d0 con 0x7f62e4102710 2024-04-12T05:57:50.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.400+0000 7f62e8fc0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e410f710 con 0x7f62e4101d70 2024-04-12T05:57:50.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.400+0000 7f62e1d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62e4102710 0x7f62e410ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:50.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.400+0000 7f62e1d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62e4102710 0x7f62e410ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34982/0 (socket says 172.21.15.32:34982) 2024-04-12T05:57:50.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.400+0000 7f62e1d9b700 1 -- 172.21.15.32:0/3846736811 learned_addr learned my addr 172.21.15.32:0/3846736811 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:57:50.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.401+0000 7f62e259c700 1 --2- 172.21.15.32:0/3846736811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62e4101d70 0x7f62e410ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:50.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.401+0000 7f62e1d9b700 1 -- 172.21.15.32:0/3846736811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62e4101d70 msgr2=0x7f62e410ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:50.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.401+0000 7f62e1d9b700 1 --2- 172.21.15.32:0/3846736811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62e4101d70 0x7f62e410ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:50.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.401+0000 7f62e1d9b700 1 -- 172.21.15.32:0/3846736811 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d4005040 con 0x7f62e4102710 2024-04-12T05:57:50.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.401+0000 7f62e1d9b700 1 --2- 172.21.15.32:0/3846736811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62e4102710 0x7f62e410ef80 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f62d8002a50 tx=0x7f62d8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:57:50.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.401+0000 7f62d2ffd700 1 -- 172.21.15.32:0/3846736811 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62d8007cc0 con 0x7f62e4102710 2024-04-12T05:57:50.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.401+0000 7f62d2ffd700 1 -- 172.21.15.32:0/3846736811 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f62d800f040 con 0x7f62e4102710 2024-04-12T05:57:50.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.401+0000 7f62d2ffd700 1 -- 172.21.15.32:0/3846736811 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62d800d920 con 0x7f62e4102710 2024-04-12T05:57:50.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.401+0000 7f62e8fc0700 1 -- 172.21.15.32:0/3846736811 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62e4117ab0 con 0x7f62e4102710 2024-04-12T05:57:50.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.401+0000 7f62e8fc0700 1 -- 172.21.15.32:0/3846736811 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62e4118010 con 0x7f62e4102710 2024-04-12T05:57:50.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.402+0000 7f62e8fc0700 1 -- 172.21.15.32:0/3846736811 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62e40ff080 con 0x7f62e4102710 2024-04-12T05:57:50.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.403+0000 7f62d2ffd700 1 -- 172.21.15.32:0/3846736811 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f62d8009680 con 0x7f62e4102710 2024-04-12T05:57:50.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.403+0000 7f62d2ffd700 1 --2- 172.21.15.32:0/3846736811 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f62cc06cc30 0x7f62cc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:50.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.403+0000 7f62e259c700 1 --2- 172.21.15.32:0/3846736811 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f62cc06cc30 0x7f62cc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:50.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.403+0000 7f62d2ffd700 1 -- 172.21.15.32:0/3846736811 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f62d808bd70 con 0x7f62e4102710 2024-04-12T05:57:50.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.404+0000 7f62e259c700 1 --2- 172.21.15.32:0/3846736811 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f62cc06cc30 0x7f62cc06f0e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f62d400b840 tx=0x7f62d4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:57:50.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.408+0000 7f62d2ffd700 1 -- 172.21.15.32:0/3846736811 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f62d8055d50 con 0x7f62e4102710 2024-04-12T05:57:50.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:50 smithi032 ceph-mon[146817]: from='client.34984 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:57:50.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.696+0000 7f62e8fc0700 1 -- 172.21.15.32:0/3846736811 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f62e40fd4c0 con 0x7f62e4102710 2024-04-12T05:57:50.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.697+0000 7f62d2ffd700 1 -- 172.21.15.32:0/3846736811 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f62d8059370 con 0x7f62e4102710 2024-04-12T05:57:50.691 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:57:50.691 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:57:50.691 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:57:50.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:57:50.691 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:57:50.691 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:57:50.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:57:50.692 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:57:50.692 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:57:50.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:57:50.692 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:57:50.692 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:57:50.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:57:50.692 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:57:50.692 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:57:50.692 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:57:50.692 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:57:50.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.699+0000 7f62e8fc0700 1 -- 172.21.15.32:0/3846736811 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f62cc06cc30 msgr2=0x7f62cc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:50.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.699+0000 7f62e8fc0700 1 --2- 172.21.15.32:0/3846736811 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f62cc06cc30 0x7f62cc06f0e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f62d400b840 tx=0x7f62d4009e30 comp rx=0 tx=0).stop 2024-04-12T05:57:50.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.699+0000 7f62e8fc0700 1 -- 172.21.15.32:0/3846736811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62e4102710 msgr2=0x7f62e410ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:50.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.699+0000 7f62e8fc0700 1 --2- 172.21.15.32:0/3846736811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62e4102710 0x7f62e410ef80 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f62d8002a50 tx=0x7f62d8004600 comp rx=0 tx=0).stop 2024-04-12T05:57:50.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.699+0000 7f62e8fc0700 1 -- 172.21.15.32:0/3846736811 shutdown_connections 2024-04-12T05:57:50.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.699+0000 7f62e8fc0700 1 --2- 172.21.15.32:0/3846736811 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f62cc06cc30 0x7f62cc06f0e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:50.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.699+0000 7f62e8fc0700 1 --2- 172.21.15.32:0/3846736811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62e4101d70 0x7f62e410ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:50.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.699+0000 7f62e8fc0700 1 --2- 172.21.15.32:0/3846736811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62e4102710 0x7f62e410ef80 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:50.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.699+0000 7f62e8fc0700 1 -- 172.21.15.32:0/3846736811 >> 172.21.15.32:0/3846736811 conn(0x7f62e4072bc0 msgr2=0x7f62e40fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:57:50.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.699+0000 7f62e8fc0700 1 -- 172.21.15.32:0/3846736811 shutdown_connections 2024-04-12T05:57:50.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.699+0000 7f62e8fc0700 1 -- 172.21.15.32:0/3846736811 wait complete. 2024-04-12T05:57:50.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:50 smithi067 ceph-mon[112979]: from='client.34984 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:57:50.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.803+0000 7fc54a50b700 1 -- 172.21.15.32:0/3833209749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc544102710 msgr2=0x7fc5440fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:50.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.803+0000 7fc54a50b700 1 --2- 172.21.15.32:0/3833209749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc544102710 0x7fc5440fbd80 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fc538004940 tx=0x7fc538009e30 comp rx=0 tx=0).stop 2024-04-12T05:57:50.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.803+0000 7fc54a50b700 1 -- 172.21.15.32:0/3833209749 shutdown_connections 2024-04-12T05:57:50.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.803+0000 7fc54a50b700 1 --2- 172.21.15.32:0/3833209749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc544102710 0x7fc5440fbd80 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:50.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.803+0000 7fc54a50b700 1 --2- 172.21.15.32:0/3833209749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc544101d70 0x7fc544102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:50.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.803+0000 7fc54a50b700 1 -- 172.21.15.32:0/3833209749 >> 172.21.15.32:0/3833209749 conn(0x7fc544072bc0 msgr2=0x7fc544072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:57:50.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.803+0000 7fc54a50b700 1 -- 172.21.15.32:0/3833209749 shutdown_connections 2024-04-12T05:57:50.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.803+0000 7fc54a50b700 1 -- 172.21.15.32:0/3833209749 wait complete. 2024-04-12T05:57:50.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.804+0000 7fc54a50b700 1 Processor -- start 2024-04-12T05:57:50.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.804+0000 7fc54a50b700 1 -- start start 2024-04-12T05:57:50.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.804+0000 7fc54a50b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc544101d70 0x7fc544102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:50.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.804+0000 7fc54a50b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc544102710 0x7fc544103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:50.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.804+0000 7fc54a50b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5441036a0 con 0x7fc544101d70 2024-04-12T05:57:50.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.804+0000 7fc54a50b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5441037e0 con 0x7fc544102710 2024-04-12T05:57:50.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.804+0000 7fc543fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc544101d70 0x7fc544102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:50.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.804+0000 7fc543fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc544101d70 0x7fc544102bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35008/0 (socket says 172.21.15.32:35008) 2024-04-12T05:57:50.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.804+0000 7fc543fff700 1 -- 172.21.15.32:0/359642270 learned_addr learned my addr 172.21.15.32:0/359642270 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:57:50.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.804+0000 7fc5437fe700 1 --2- 172.21.15.32:0/359642270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc544102710 0x7fc544103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:50.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.804+0000 7fc543fff700 1 -- 172.21.15.32:0/359642270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc544102710 msgr2=0x7fc544103110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:50.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.804+0000 7fc543fff700 1 --2- 172.21.15.32:0/359642270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc544102710 0x7fc544103110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:50.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.804+0000 7fc543fff700 1 -- 172.21.15.32:0/359642270 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc538005040 con 0x7fc544101d70 2024-04-12T05:57:50.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.805+0000 7fc543fff700 1 --2- 172.21.15.32:0/359642270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc544101d70 0x7fc544102bd0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fc534002a50 tx=0x7fc5340046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:57:50.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.805+0000 7fc540ff9700 1 -- 172.21.15.32:0/359642270 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc534007cc0 con 0x7fc544101d70 2024-04-12T05:57:50.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.805+0000 7fc540ff9700 1 -- 172.21.15.32:0/359642270 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc53400f040 con 0x7fc544101d70 2024-04-12T05:57:50.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.805+0000 7fc54a50b700 1 -- 172.21.15.32:0/359642270 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc544119dc0 con 0x7fc544101d70 2024-04-12T05:57:50.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.805+0000 7fc540ff9700 1 -- 172.21.15.32:0/359642270 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc534009540 con 0x7fc544101d70 2024-04-12T05:57:50.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.805+0000 7fc54a50b700 1 -- 172.21.15.32:0/359642270 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc54411a240 con 0x7fc544101d70 2024-04-12T05:57:50.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.806+0000 7fc54a50b700 1 -- 172.21.15.32:0/359642270 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5440fd4c0 con 0x7fc544101d70 2024-04-12T05:57:50.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.806+0000 7fc540ff9700 1 -- 172.21.15.32:0/359642270 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc5340096a0 con 0x7fc544101d70 2024-04-12T05:57:50.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.807+0000 7fc540ff9700 1 --2- 172.21.15.32:0/359642270 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc52c06cc20 0x7fc52c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:50.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.807+0000 7fc540ff9700 1 -- 172.21.15.32:0/359642270 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fc53408bdf0 con 0x7fc544101d70 2024-04-12T05:57:50.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.807+0000 7fc5437fe700 1 --2- 172.21.15.32:0/359642270 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc52c06cc20 0x7fc52c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:50.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.807+0000 7fc5437fe700 1 --2- 172.21.15.32:0/359642270 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc52c06cc20 0x7fc52c06f0d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc53800b3b0 tx=0x7fc538009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:57:50.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:50.811+0000 7fc540ff9700 1 -- 172.21.15.32:0/359642270 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc534055dd0 con 0x7fc544101d70 2024-04-12T05:57:51.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.063+0000 7fc54a50b700 1 -- 172.21.15.32:0/359642270 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc5441042c0 con 0x7fc544101d70 2024-04-12T05:57:51.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.064+0000 7fc540ff9700 1 -- 172.21.15.32:0/359642270 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1835 (secure 0 0 0) 0x7fc5340593f0 con 0x7fc544101d70 2024-04-12T05:57:51.058 INFO:teuthology.orchestra.run.smithi032.stdout:e33 2024-04-12T05:57:51.058 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:57:51.058 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:57:51.058 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:57:51.058 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:57:51.058 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:57:51.058 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:57:51.059 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 24 2024-04-12T05:57:51.059 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:57:51.059 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:57:51.059 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:46:10.668569+0000 2024-04-12T05:57:51.059 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:57:51.059 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:57:51.059 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:57:51.059 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:57:51.059 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:57:51.059 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:57:51.059 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:57:51.059 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:57:51.059 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T05:57:51.059 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:57:51.059 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:57:51.060 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:57:51.060 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T05:57:51.060 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-12T05:57:51.060 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:57:51.060 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:57:51.060 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:57:51.060 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:57:51.060 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:57:51.060 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:57:51.060 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:57:51.060 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:57:51.060 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:57:51.060 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:57:51.060 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-12T05:57:51.060 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:57:51.061 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:57:51.061 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:57:51.061 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{-1:44317} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:57:51.061 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:57:51.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.066+0000 7fc54a50b700 1 -- 172.21.15.32:0/359642270 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc52c06cc20 msgr2=0x7fc52c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:51.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.066+0000 7fc54a50b700 1 --2- 172.21.15.32:0/359642270 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc52c06cc20 0x7fc52c06f0d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc53800b3b0 tx=0x7fc538009e30 comp rx=0 tx=0).stop 2024-04-12T05:57:51.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.066+0000 7fc54a50b700 1 -- 172.21.15.32:0/359642270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc544101d70 msgr2=0x7fc544102bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:51.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.066+0000 7fc54a50b700 1 --2- 172.21.15.32:0/359642270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc544101d70 0x7fc544102bd0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fc534002a50 tx=0x7fc5340046b0 comp rx=0 tx=0).stop 2024-04-12T05:57:51.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.066+0000 7fc54a50b700 1 -- 172.21.15.32:0/359642270 shutdown_connections 2024-04-12T05:57:51.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.066+0000 7fc54a50b700 1 --2- 172.21.15.32:0/359642270 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fc52c06cc20 0x7fc52c06f0d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:51.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.066+0000 7fc54a50b700 1 --2- 172.21.15.32:0/359642270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc544101d70 0x7fc544102bd0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:51.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.066+0000 7fc54a50b700 1 --2- 172.21.15.32:0/359642270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc544102710 0x7fc544103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:51.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.066+0000 7fc54a50b700 1 -- 172.21.15.32:0/359642270 >> 172.21.15.32:0/359642270 conn(0x7fc544072bc0 msgr2=0x7fc544065a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:57:51.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.066+0000 7fc54a50b700 1 -- 172.21.15.32:0/359642270 shutdown_connections 2024-04-12T05:57:51.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.066+0000 7fc54a50b700 1 -- 172.21.15.32:0/359642270 wait complete. 2024-04-12T05:57:51.062 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-12T05:57:51.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.171+0000 7f9ed401d700 1 -- 172.21.15.32:0/1416461741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ecc102710 msgr2=0x7f9ecc0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:51.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.171+0000 7f9ed401d700 1 --2- 172.21.15.32:0/1416461741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ecc102710 0x7f9ecc0fbd80 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc004940 tx=0x7f9ebc009e30 comp rx=0 tx=0).stop 2024-04-12T05:57:51.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.171+0000 7f9ed401d700 1 -- 172.21.15.32:0/1416461741 shutdown_connections 2024-04-12T05:57:51.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.171+0000 7f9ed401d700 1 --2- 172.21.15.32:0/1416461741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ecc102710 0x7f9ecc0fbd80 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:51.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.171+0000 7f9ed401d700 1 --2- 172.21.15.32:0/1416461741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ecc101d70 0x7f9ecc102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:51.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.171+0000 7f9ed401d700 1 -- 172.21.15.32:0/1416461741 >> 172.21.15.32:0/1416461741 conn(0x7f9ecc072bc0 msgr2=0x7f9ecc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:57:51.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.171+0000 7f9ed401d700 1 -- 172.21.15.32:0/1416461741 shutdown_connections 2024-04-12T05:57:51.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.171+0000 7f9ed401d700 1 -- 172.21.15.32:0/1416461741 wait complete. 2024-04-12T05:57:51.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.171+0000 7f9ed401d700 1 Processor -- start 2024-04-12T05:57:51.166 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.171+0000 7f9ed401d700 1 -- start start 2024-04-12T05:57:51.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.172+0000 7f9ed401d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ecc101d70 0x7f9ecc10ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:51.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.172+0000 7f9ed401d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ecc102710 0x7f9ecc10f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:51.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.172+0000 7f9ed401d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc10f640 con 0x7f9ecc101d70 2024-04-12T05:57:51.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.172+0000 7f9ed401d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc10f780 con 0x7f9ecc102710 2024-04-12T05:57:51.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.172+0000 7f9ed1db9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ecc101d70 0x7f9ecc10ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:51.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.172+0000 7f9ed1db9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ecc101d70 0x7f9ecc10ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35028/0 (socket says 172.21.15.32:35028) 2024-04-12T05:57:51.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.172+0000 7f9ed1db9700 1 -- 172.21.15.32:0/3567170043 learned_addr learned my addr 172.21.15.32:0/3567170043 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:57:51.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.172+0000 7f9ed15b8700 1 --2- 172.21.15.32:0/3567170043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ecc102710 0x7f9ecc10f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:51.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.172+0000 7f9ed1db9700 1 -- 172.21.15.32:0/3567170043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ecc102710 msgr2=0x7f9ecc10f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:51.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.172+0000 7f9ed1db9700 1 --2- 172.21.15.32:0/3567170043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ecc102710 0x7f9ecc10f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:51.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.172+0000 7f9ed1db9700 1 -- 172.21.15.32:0/3567170043 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ebc005040 con 0x7f9ecc101d70 2024-04-12T05:57:51.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.172+0000 7f9ed1db9700 1 --2- 172.21.15.32:0/3567170043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ecc101d70 0x7f9ecc10ebc0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8002a50 tx=0x7f9ec80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:57:51.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.172+0000 7f9ec27fc700 1 -- 172.21.15.32:0/3567170043 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ec8007cc0 con 0x7f9ecc101d70 2024-04-12T05:57:51.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.172+0000 7f9ec27fc700 1 -- 172.21.15.32:0/3567170043 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ec800f040 con 0x7f9ecc101d70 2024-04-12T05:57:51.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.172+0000 7f9ed401d700 1 -- 172.21.15.32:0/3567170043 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ecc117a70 con 0x7f9ecc101d70 2024-04-12T05:57:51.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.172+0000 7f9ec27fc700 1 -- 172.21.15.32:0/3567170043 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ec8009540 con 0x7f9ecc101d70 2024-04-12T05:57:51.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.172+0000 7f9ed401d700 1 -- 172.21.15.32:0/3567170043 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ecc118030 con 0x7f9ecc101d70 2024-04-12T05:57:51.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.173+0000 7f9ed401d700 1 -- 172.21.15.32:0/3567170043 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ecc0ff080 con 0x7f9ecc101d70 2024-04-12T05:57:51.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.174+0000 7f9ec27fc700 1 -- 172.21.15.32:0/3567170043 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9ec80096a0 con 0x7f9ecc101d70 2024-04-12T05:57:51.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.174+0000 7f9ec27fc700 1 --2- 172.21.15.32:0/3567170043 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9eb806cc20 0x7f9eb806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:51.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.175+0000 7f9ed15b8700 1 --2- 172.21.15.32:0/3567170043 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9eb806cc20 0x7f9eb806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:51.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.175+0000 7f9ec27fc700 1 -- 172.21.15.32:0/3567170043 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f9ec808bf70 con 0x7f9ecc101d70 2024-04-12T05:57:51.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.175+0000 7f9ed15b8700 1 --2- 172.21.15.32:0/3567170043 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9eb806cc20 0x7f9eb806f0d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc00b3b0 tx=0x7f9ebc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:57:51.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.179+0000 7f9ec27fc700 1 -- 172.21.15.32:0/3567170043 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9ec8055e30 con 0x7f9ecc101d70 2024-04-12T05:57:51.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.400+0000 7f9ed401d700 1 -- 172.21.15.32:0/3567170043 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ecc04bed0 con 0x7f9eb806cc20 2024-04-12T05:57:51.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.401+0000 7f9ec27fc700 1 -- 172.21.15.32:0/3567170043 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9ecc04bed0 con 0x7f9eb806cc20 2024-04-12T05:57:51.396 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:57:51.396 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:57:51.396 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:57:51.396 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:57:51.396 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:57:51.396 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-12T05:57:51.396 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:57:51.396 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-12T05:57:51.396 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:57:51.396 INFO:teuthology.orchestra.run.smithi032.stdout: "mds" 2024-04-12T05:57:51.396 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:57:51.396 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:57:51.397 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:57:51.397 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:57:51.397 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:57:51.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.403+0000 7f9ed401d700 1 -- 172.21.15.32:0/3567170043 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9eb806cc20 msgr2=0x7f9eb806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:51.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.403+0000 7f9ed401d700 1 --2- 172.21.15.32:0/3567170043 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9eb806cc20 0x7f9eb806f0d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc00b3b0 tx=0x7f9ebc009e30 comp rx=0 tx=0).stop 2024-04-12T05:57:51.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.403+0000 7f9ed401d700 1 -- 172.21.15.32:0/3567170043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ecc101d70 msgr2=0x7f9ecc10ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:51.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.403+0000 7f9ed401d700 1 --2- 172.21.15.32:0/3567170043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ecc101d70 0x7f9ecc10ebc0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8002a50 tx=0x7f9ec80046b0 comp rx=0 tx=0).stop 2024-04-12T05:57:51.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.404+0000 7f9ed401d700 1 -- 172.21.15.32:0/3567170043 shutdown_connections 2024-04-12T05:57:51.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.404+0000 7f9ed401d700 1 --2- 172.21.15.32:0/3567170043 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9eb806cc20 0x7f9eb806f0d0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:51.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.404+0000 7f9ed401d700 1 --2- 172.21.15.32:0/3567170043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ecc101d70 0x7f9ecc10ebc0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:51.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.404+0000 7f9ed401d700 1 --2- 172.21.15.32:0/3567170043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ecc102710 0x7f9ecc10f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:51.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.404+0000 7f9ed401d700 1 -- 172.21.15.32:0/3567170043 >> 172.21.15.32:0/3567170043 conn(0x7f9ecc072bc0 msgr2=0x7f9ecc0fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:57:51.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.404+0000 7f9ed401d700 1 -- 172.21.15.32:0/3567170043 shutdown_connections 2024-04-12T05:57:51.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.404+0000 7f9ed401d700 1 -- 172.21.15.32:0/3567170043 wait complete. 2024-04-12T05:57:51.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.507+0000 7f080dfd4700 1 -- 172.21.15.32:0/1696896810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0808102e70 msgr2=0x7f0808103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:51.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.507+0000 7f080dfd4700 1 --2- 172.21.15.32:0/1696896810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0808102e70 0x7f0808103240 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f07f8004940 tx=0x7f07f8009e30 comp rx=0 tx=0).stop 2024-04-12T05:57:51.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.507+0000 7f080dfd4700 1 -- 172.21.15.32:0/1696896810 shutdown_connections 2024-04-12T05:57:51.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.507+0000 7f080dfd4700 1 --2- 172.21.15.32:0/1696896810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08080fb4a0 0x7f08080fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:51.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.507+0000 7f080dfd4700 1 --2- 172.21.15.32:0/1696896810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0808102e70 0x7f0808103240 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:51.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.507+0000 7f080dfd4700 1 -- 172.21.15.32:0/1696896810 >> 172.21.15.32:0/1696896810 conn(0x7f08080714c0 msgr2=0x7f08080718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:57:51.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.507+0000 7f080dfd4700 1 -- 172.21.15.32:0/1696896810 shutdown_connections 2024-04-12T05:57:51.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.507+0000 7f080dfd4700 1 -- 172.21.15.32:0/1696896810 wait complete. 2024-04-12T05:57:51.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.508+0000 7f080dfd4700 1 Processor -- start 2024-04-12T05:57:51.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.508+0000 7f080dfd4700 1 -- start start 2024-04-12T05:57:51.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.508+0000 7f080dfd4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08080fb4a0 0x7f0808113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:51.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.508+0000 7f080dfd4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0808102e70 0x7f0808113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:51.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.508+0000 7f080dfd4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0808114600 con 0x7f08080fb4a0 2024-04-12T05:57:51.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.508+0000 7f080dfd4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080810d9f0 con 0x7f0808102e70 2024-04-12T05:57:51.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.508+0000 7f08077fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08080fb4a0 0x7f0808113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:51.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.508+0000 7f08077fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08080fb4a0 0x7f0808113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48418/0 (socket says 172.21.15.32:48418) 2024-04-12T05:57:51.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.509+0000 7f08077fe700 1 -- 172.21.15.32:0/635055712 learned_addr learned my addr 172.21.15.32:0/635055712 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:57:51.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.509+0000 7f07feffd700 1 --2- 172.21.15.32:0/635055712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0808102e70 0x7f0808113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:51.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.509+0000 7f08077fe700 1 -- 172.21.15.32:0/635055712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0808102e70 msgr2=0x7f0808113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:51.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.509+0000 7f08077fe700 1 --2- 172.21.15.32:0/635055712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0808102e70 0x7f0808113fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:51.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.509+0000 7f08077fe700 1 -- 172.21.15.32:0/635055712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f8005040 con 0x7f08080fb4a0 2024-04-12T05:57:51.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.509+0000 7f08077fe700 1 --2- 172.21.15.32:0/635055712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08080fb4a0 0x7f0808113a70 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f07f800b3b0 tx=0x7f07f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:57:51.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.509+0000 7f0804ff9700 1 -- 172.21.15.32:0/635055712 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07f800e040 con 0x7f08080fb4a0 2024-04-12T05:57:51.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.509+0000 7f0804ff9700 1 -- 172.21.15.32:0/635055712 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07f8013420 con 0x7f08080fb4a0 2024-04-12T05:57:51.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.509+0000 7f080dfd4700 1 -- 172.21.15.32:0/635055712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f080810dc70 con 0x7f08080fb4a0 2024-04-12T05:57:51.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.509+0000 7f0804ff9700 1 -- 172.21.15.32:0/635055712 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07f801c610 con 0x7f08080fb4a0 2024-04-12T05:57:51.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.509+0000 7f080dfd4700 1 -- 172.21.15.32:0/635055712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f080810e230 con 0x7f08080fb4a0 2024-04-12T05:57:51.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.510+0000 7f080dfd4700 1 -- 172.21.15.32:0/635055712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0808100110 con 0x7f08080fb4a0 2024-04-12T05:57:51.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.511+0000 7f0804ff9700 1 -- 172.21.15.32:0/635055712 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f07f8013a00 con 0x7f08080fb4a0 2024-04-12T05:57:51.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.511+0000 7f0804ff9700 1 --2- 172.21.15.32:0/635055712 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f07f406cc20 0x7f07f406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:57:51.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.511+0000 7f0804ff9700 1 -- 172.21.15.32:0/635055712 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f07f808d100 con 0x7f08080fb4a0 2024-04-12T05:57:51.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.511+0000 7f07feffd700 1 --2- 172.21.15.32:0/635055712 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f07f406cc20 0x7f07f406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:57:51.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.512+0000 7f07feffd700 1 --2- 172.21.15.32:0/635055712 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f07f406cc20 0x7f07f406f0d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f080810ebb0 tx=0x7f07f00043c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:57:51.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.516+0000 7f0804ff9700 1 -- 172.21.15.32:0/635055712 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f07f805ac10 con 0x7f08080fb4a0 2024-04-12T05:57:51.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:51 smithi032 ceph-mon[146817]: from='client.34988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:57:51.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:51 smithi032 ceph-mon[146817]: pgmap v520: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:51.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:51 smithi032 ceph-mon[146817]: from='client.34992 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:57:51.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:51 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3846736811' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:57:51.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:51 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/359642270' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:57:51.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:51 smithi067 ceph-mon[112979]: from='client.34988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:57:51.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:51 smithi067 ceph-mon[112979]: pgmap v520: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:51.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:51 smithi067 ceph-mon[112979]: from='client.34992 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:57:51.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:51 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3846736811' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:57:51.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:51 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/359642270' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:57:51.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.805+0000 7f080dfd4700 1 -- 172.21.15.32:0/635055712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f080810f0c0 con 0x7f08080fb4a0 2024-04-12T05:57:51.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.805+0000 7f0804ff9700 1 -- 172.21.15.32:0/635055712 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f07f805a7a0 con 0x7f08080fb4a0 2024-04-12T05:57:51.799 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:57:51.799 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:57:51.799 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-12T05:57:51.800 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:57:51.800 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:57:51.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.807+0000 7f080dfd4700 1 -- 172.21.15.32:0/635055712 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f07f406cc20 msgr2=0x7f07f406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:51.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.807+0000 7f080dfd4700 1 --2- 172.21.15.32:0/635055712 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f07f406cc20 0x7f07f406f0d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f080810ebb0 tx=0x7f07f00043c0 comp rx=0 tx=0).stop 2024-04-12T05:57:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.807+0000 7f080dfd4700 1 -- 172.21.15.32:0/635055712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08080fb4a0 msgr2=0x7f0808113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:57:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.807+0000 7f080dfd4700 1 --2- 172.21.15.32:0/635055712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08080fb4a0 0x7f0808113a70 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f07f800b3b0 tx=0x7f07f8009e30 comp rx=0 tx=0).stop 2024-04-12T05:57:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.807+0000 7f080dfd4700 1 -- 172.21.15.32:0/635055712 shutdown_connections 2024-04-12T05:57:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.807+0000 7f080dfd4700 1 --2- 172.21.15.32:0/635055712 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f07f406cc20 0x7f07f406f0d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.807+0000 7f080dfd4700 1 --2- 172.21.15.32:0/635055712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08080fb4a0 0x7f0808113a70 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.807+0000 7f080dfd4700 1 --2- 172.21.15.32:0/635055712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0808102e70 0x7f0808113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:57:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.807+0000 7f080dfd4700 1 -- 172.21.15.32:0/635055712 >> 172.21.15.32:0/635055712 conn(0x7f08080714c0 msgr2=0x7f08080fa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:57:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.807+0000 7f080dfd4700 1 -- 172.21.15.32:0/635055712 shutdown_connections 2024-04-12T05:57:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:57:51.807+0000 7f080dfd4700 1 -- 172.21.15.32:0/635055712 wait complete. 2024-04-12T05:57:52.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:52 smithi032 ceph-mon[146817]: from='client.35004 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:57:52.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:52 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/635055712' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:57:52.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:52 smithi067 ceph-mon[112979]: from='client.35004 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:57:52.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:52 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/635055712' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:57:53.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:53 smithi032 ceph-mon[146817]: pgmap v521: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:53.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:53 smithi067 ceph-mon[112979]: pgmap v521: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:55.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:55 smithi032 ceph-mon[146817]: pgmap v522: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:55.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:55 smithi067 ceph-mon[112979]: pgmap v522: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:57.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:57 smithi032 ceph-mon[146817]: pgmap v523: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:57.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:57 smithi067 ceph-mon[112979]: pgmap v523: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:59.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:57:59 smithi032 ceph-mon[146817]: pgmap v524: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:57:59.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:57:59 smithi067 ceph-mon[112979]: pgmap v524: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:01.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:01 smithi032 ceph-mon[146817]: pgmap v525: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:01.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:01 smithi067 ceph-mon[112979]: pgmap v525: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:03.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:03 smithi032 ceph-mon[146817]: pgmap v526: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:03.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:58:03.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:03 smithi067 ceph-mon[112979]: pgmap v526: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:03.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:58:05.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:05 smithi032 ceph-mon[146817]: pgmap v527: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:05.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:05 smithi067 ceph-mon[112979]: pgmap v527: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:07.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:07 smithi032 ceph-mon[146817]: pgmap v528: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:07.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:07 smithi067 ceph-mon[112979]: pgmap v528: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:09.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:09 smithi032 ceph-mon[146817]: pgmap v529: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:09.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:09 smithi067 ceph-mon[112979]: pgmap v529: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:11.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:11 smithi032 ceph-mon[146817]: pgmap v530: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:11.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:11 smithi067 ceph-mon[112979]: pgmap v530: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:13.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:13 smithi032 ceph-mon[146817]: pgmap v531: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:13.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:13 smithi067 ceph-mon[112979]: pgmap v531: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:15.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:15 smithi032 ceph-mon[146817]: pgmap v532: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:15.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:15 smithi067 ceph-mon[112979]: pgmap v532: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:17.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:17 smithi032 ceph-mon[146817]: pgmap v533: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:17.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:17 smithi067 ceph-mon[112979]: pgmap v533: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:18.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:58:18.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:58:19.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:19 smithi032 ceph-mon[146817]: pgmap v534: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:19.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:19 smithi067 ceph-mon[112979]: pgmap v534: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:21.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:21 smithi032 ceph-mon[146817]: pgmap v535: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:21.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:21 smithi067 ceph-mon[112979]: pgmap v535: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:21.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.924+0000 7f452e835700 1 -- 172.21.15.32:0/3354472863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45280fc4f0 msgr2=0x7f4528109250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:21.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.924+0000 7f452e835700 1 --2- 172.21.15.32:0/3354472863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45280fc4f0 0x7f4528109250 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f4510004940 tx=0x7f4510009e30 comp rx=0 tx=0).stop 2024-04-12T05:58:21.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.924+0000 7f452e835700 1 -- 172.21.15.32:0/3354472863 shutdown_connections 2024-04-12T05:58:21.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.924+0000 7f452e835700 1 --2- 172.21.15.32:0/3354472863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45280fc4f0 0x7f4528109250 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:21.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.924+0000 7f452e835700 1 --2- 172.21.15.32:0/3354472863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45280fbbe0 0x7f45280fbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:21.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.924+0000 7f452e835700 1 -- 172.21.15.32:0/3354472863 >> 172.21.15.32:0/3354472863 conn(0x7f45280f78d0 msgr2=0x7f45280f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:58:21.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.924+0000 7f452e835700 1 -- 172.21.15.32:0/3354472863 shutdown_connections 2024-04-12T05:58:21.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.924+0000 7f452e835700 1 -- 172.21.15.32:0/3354472863 wait complete. 2024-04-12T05:58:21.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.925+0000 7f452e835700 1 Processor -- start 2024-04-12T05:58:21.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.925+0000 7f452e835700 1 -- start start 2024-04-12T05:58:21.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.925+0000 7f452e835700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45280fbbe0 0x7f4528115bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:21.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.925+0000 7f452e835700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45280fc4f0 0x7f45281160f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:21.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.925+0000 7f452e835700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4528116740 con 0x7f45280fc4f0 2024-04-12T05:58:21.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.925+0000 7f452e835700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452810fb30 con 0x7f45280fbbe0 2024-04-12T05:58:21.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.925+0000 7f45277fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45280fc4f0 0x7f45281160f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:21.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.925+0000 7f45277fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45280fc4f0 0x7f45281160f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32768/0 (socket says 172.21.15.32:32768) 2024-04-12T05:58:21.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.925+0000 7f45277fe700 1 -- 172.21.15.32:0/1998018601 learned_addr learned my addr 172.21.15.32:0/1998018601 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:58:21.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.926+0000 7f4527fff700 1 --2- 172.21.15.32:0/1998018601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45280fbbe0 0x7f4528115bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:21.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.926+0000 7f45277fe700 1 -- 172.21.15.32:0/1998018601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45280fbbe0 msgr2=0x7f4528115bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:21.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.926+0000 7f45277fe700 1 --2- 172.21.15.32:0/1998018601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45280fbbe0 0x7f4528115bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:21.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.926+0000 7f45277fe700 1 -- 172.21.15.32:0/1998018601 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4510005040 con 0x7f45280fc4f0 2024-04-12T05:58:21.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.926+0000 7f4527fff700 1 --2- 172.21.15.32:0/1998018601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45280fbbe0 0x7f4528115bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:58:21.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.926+0000 7f45277fe700 1 --2- 172.21.15.32:0/1998018601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45280fc4f0 0x7f45281160f0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f4510000c00 tx=0x7f4510009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:58:21.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.926+0000 7f4524ff9700 1 -- 172.21.15.32:0/1998018601 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4510021070 con 0x7f45280fc4f0 2024-04-12T05:58:21.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.926+0000 7f452e835700 1 -- 172.21.15.32:0/1998018601 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f452810fdb0 con 0x7f45280fc4f0 2024-04-12T05:58:21.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.926+0000 7f4524ff9700 1 -- 172.21.15.32:0/1998018601 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4510013e00 con 0x7f45280fc4f0 2024-04-12T05:58:21.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.926+0000 7f4524ff9700 1 -- 172.21.15.32:0/1998018601 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f451000e070 con 0x7f45280fc4f0 2024-04-12T05:58:21.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.926+0000 7f452e835700 1 -- 172.21.15.32:0/1998018601 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4528110370 con 0x7f45280fc4f0 2024-04-12T05:58:21.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.927+0000 7f452e835700 1 -- 172.21.15.32:0/1998018601 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4528066710 con 0x7f45280fc4f0 2024-04-12T05:58:21.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.928+0000 7f4524ff9700 1 -- 172.21.15.32:0/1998018601 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f451001c690 con 0x7f45280fc4f0 2024-04-12T05:58:21.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.928+0000 7f4524ff9700 1 --2- 172.21.15.32:0/1998018601 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f451406cc20 0x7f451406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:21.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.928+0000 7f4524ff9700 1 -- 172.21.15.32:0/1998018601 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f451008d2a0 con 0x7f45280fc4f0 2024-04-12T05:58:21.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.928+0000 7f4527fff700 1 --2- 172.21.15.32:0/1998018601 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f451406cc20 0x7f451406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:21.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.929+0000 7f4527fff700 1 --2- 172.21.15.32:0/1998018601 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f451406cc20 0x7f451406f0d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4518004600 tx=0x7f451800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:58:21.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:21.933+0000 7f4524ff9700 1 -- 172.21.15.32:0/1998018601 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f451005ad10 con 0x7f45280fc4f0 2024-04-12T05:58:22.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.152+0000 7f452e835700 1 -- 172.21.15.32:0/1998018601 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f452810bef0 con 0x7f451406cc20 2024-04-12T05:58:22.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.154+0000 7f4524ff9700 1 -- 172.21.15.32:0/1998018601 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f452810bef0 con 0x7f451406cc20 2024-04-12T05:58:22.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.156+0000 7f452e835700 1 -- 172.21.15.32:0/1998018601 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f451406cc20 msgr2=0x7f451406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:22.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.156+0000 7f452e835700 1 --2- 172.21.15.32:0/1998018601 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f451406cc20 0x7f451406f0d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4518004600 tx=0x7f451800b040 comp rx=0 tx=0).stop 2024-04-12T05:58:22.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.156+0000 7f452e835700 1 -- 172.21.15.32:0/1998018601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45280fc4f0 msgr2=0x7f45281160f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:22.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.156+0000 7f452e835700 1 --2- 172.21.15.32:0/1998018601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45280fc4f0 0x7f45281160f0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f4510000c00 tx=0x7f4510009e30 comp rx=0 tx=0).stop 2024-04-12T05:58:22.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.156+0000 7f452e835700 1 -- 172.21.15.32:0/1998018601 shutdown_connections 2024-04-12T05:58:22.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.156+0000 7f452e835700 1 --2- 172.21.15.32:0/1998018601 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f451406cc20 0x7f451406f0d0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:22.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.156+0000 7f452e835700 1 --2- 172.21.15.32:0/1998018601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45280fbbe0 0x7f4528115bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:22.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.156+0000 7f452e835700 1 --2- 172.21.15.32:0/1998018601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45280fc4f0 0x7f45281160f0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:22.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.156+0000 7f452e835700 1 -- 172.21.15.32:0/1998018601 >> 172.21.15.32:0/1998018601 conn(0x7f45280f78d0 msgr2=0x7f45280f7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:58:22.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.156+0000 7f452e835700 1 -- 172.21.15.32:0/1998018601 shutdown_connections 2024-04-12T05:58:22.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.156+0000 7f452e835700 1 -- 172.21.15.32:0/1998018601 wait complete. 2024-04-12T05:58:22.161 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:58:22.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.260+0000 7f118bccb700 1 -- 172.21.15.32:0/3950719705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11841037b0 msgr2=0x7f11840fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:22.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.260+0000 7f118bccb700 1 --2- 172.21.15.32:0/3950719705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11841037b0 0x7f11840fce20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1180004940 tx=0x7f1180009e30 comp rx=0 tx=0).stop 2024-04-12T05:58:22.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.260+0000 7f118bccb700 1 -- 172.21.15.32:0/3950719705 shutdown_connections 2024-04-12T05:58:22.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.260+0000 7f118bccb700 1 --2- 172.21.15.32:0/3950719705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11841037b0 0x7f11840fce20 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:22.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.260+0000 7f118bccb700 1 --2- 172.21.15.32:0/3950719705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1184102e10 0x7f11841031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:22.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.260+0000 7f118bccb700 1 -- 172.21.15.32:0/3950719705 >> 172.21.15.32:0/3950719705 conn(0x7f11840f89a0 msgr2=0x7f11840fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:58:22.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.260+0000 7f118bccb700 1 -- 172.21.15.32:0/3950719705 shutdown_connections 2024-04-12T05:58:22.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.260+0000 7f118bccb700 1 -- 172.21.15.32:0/3950719705 wait complete. 2024-04-12T05:58:22.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.261+0000 7f118bccb700 1 Processor -- start 2024-04-12T05:58:22.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.261+0000 7f118bccb700 1 -- start start 2024-04-12T05:58:22.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.261+0000 7f118bccb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1184102e10 0x7f1184113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:22.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.261+0000 7f118bccb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11841037b0 0x7f1184114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:22.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.261+0000 7f118bccb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1184114660 con 0x7f11841037b0 2024-04-12T05:58:22.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.261+0000 7f118bccb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118410da50 con 0x7f1184102e10 2024-04-12T05:58:22.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.261+0000 7f1189266700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11841037b0 0x7f1184114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:22.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.261+0000 7f1189266700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11841037b0 0x7f1184114010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32776/0 (socket says 172.21.15.32:32776) 2024-04-12T05:58:22.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.261+0000 7f1189266700 1 -- 172.21.15.32:0/3240781467 learned_addr learned my addr 172.21.15.32:0/3240781467 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:58:22.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.261+0000 7f1189a67700 1 --2- 172.21.15.32:0/3240781467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1184102e10 0x7f1184113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:22.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.261+0000 7f1189266700 1 -- 172.21.15.32:0/3240781467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1184102e10 msgr2=0x7f1184113ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:22.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.261+0000 7f1189266700 1 --2- 172.21.15.32:0/3240781467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1184102e10 0x7f1184113ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:22.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.261+0000 7f1189266700 1 -- 172.21.15.32:0/3240781467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1180005040 con 0x7f11841037b0 2024-04-12T05:58:22.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.261+0000 7f1189a67700 1 --2- 172.21.15.32:0/3240781467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1184102e10 0x7f1184113ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:58:22.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.261+0000 7f1189266700 1 --2- 172.21.15.32:0/3240781467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11841037b0 0x7f1184114010 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f118000b3b0 tx=0x7f1180009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:58:22.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.262+0000 7f117a7fc700 1 -- 172.21.15.32:0/3240781467 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f118000e040 con 0x7f11841037b0 2024-04-12T05:58:22.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.262+0000 7f118bccb700 1 -- 172.21.15.32:0/3240781467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f118410dcd0 con 0x7f11841037b0 2024-04-12T05:58:22.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.262+0000 7f117a7fc700 1 -- 172.21.15.32:0/3240781467 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1180009b10 con 0x7f11841037b0 2024-04-12T05:58:22.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.262+0000 7f117a7fc700 1 -- 172.21.15.32:0/3240781467 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1180013420 con 0x7f11841037b0 2024-04-12T05:58:22.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.262+0000 7f118bccb700 1 -- 172.21.15.32:0/3240781467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f118410e290 con 0x7f11841037b0 2024-04-12T05:58:22.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.263+0000 7f118bccb700 1 -- 172.21.15.32:0/3240781467 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11840fe560 con 0x7f11841037b0 2024-04-12T05:58:22.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.263+0000 7f117a7fc700 1 -- 172.21.15.32:0/3240781467 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f118001c430 con 0x7f11841037b0 2024-04-12T05:58:22.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.264+0000 7f117a7fc700 1 --2- 172.21.15.32:0/3240781467 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f117006cc20 0x7f117006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:22.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.264+0000 7f117a7fc700 1 -- 172.21.15.32:0/3240781467 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f118008cc90 con 0x7f11841037b0 2024-04-12T05:58:22.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.264+0000 7f1189a67700 1 --2- 172.21.15.32:0/3240781467 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f117006cc20 0x7f117006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:22.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.264+0000 7f1189a67700 1 --2- 172.21.15.32:0/3240781467 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f117006cc20 0x7f117006f0d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f11740041d0 tx=0x7f1174009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:58:22.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.268+0000 7f117a7fc700 1 -- 172.21.15.32:0/3240781467 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f118005a7b0 con 0x7f11841037b0 2024-04-12T05:58:22.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.492+0000 7f118bccb700 1 -- 172.21.15.32:0/3240781467 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1184109db0 con 0x7f117006cc20 2024-04-12T05:58:22.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.493+0000 7f117a7fc700 1 -- 172.21.15.32:0/3240781467 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f1184109db0 con 0x7f117006cc20 2024-04-12T05:58:22.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.495+0000 7f118bccb700 1 -- 172.21.15.32:0/3240781467 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f117006cc20 msgr2=0x7f117006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:22.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.495+0000 7f118bccb700 1 --2- 172.21.15.32:0/3240781467 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f117006cc20 0x7f117006f0d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f11740041d0 tx=0x7f1174009040 comp rx=0 tx=0).stop 2024-04-12T05:58:22.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.495+0000 7f118bccb700 1 -- 172.21.15.32:0/3240781467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11841037b0 msgr2=0x7f1184114010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:22.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.495+0000 7f118bccb700 1 --2- 172.21.15.32:0/3240781467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11841037b0 0x7f1184114010 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f118000b3b0 tx=0x7f1180009e30 comp rx=0 tx=0).stop 2024-04-12T05:58:22.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.495+0000 7f118bccb700 1 -- 172.21.15.32:0/3240781467 shutdown_connections 2024-04-12T05:58:22.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.495+0000 7f118bccb700 1 --2- 172.21.15.32:0/3240781467 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f117006cc20 0x7f117006f0d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:22.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.495+0000 7f118bccb700 1 --2- 172.21.15.32:0/3240781467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1184102e10 0x7f1184113ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:22.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.495+0000 7f118bccb700 1 --2- 172.21.15.32:0/3240781467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11841037b0 0x7f1184114010 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:22.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.495+0000 7f118bccb700 1 -- 172.21.15.32:0/3240781467 >> 172.21.15.32:0/3240781467 conn(0x7f11840f89a0 msgr2=0x7f11840f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:58:22.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.495+0000 7f118bccb700 1 -- 172.21.15.32:0/3240781467 shutdown_connections 2024-04-12T05:58:22.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.495+0000 7f118bccb700 1 -- 172.21.15.32:0/3240781467 wait complete. 2024-04-12T05:58:22.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.598+0000 7f67c6b84700 1 -- 172.21.15.32:0/4043254281 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67c00fe810 msgr2=0x7f67c0106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:22.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.598+0000 7f67c6b84700 1 --2- 172.21.15.32:0/4043254281 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67c00fe810 0x7f67c0106e50 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f67ac004940 tx=0x7f67ac009e30 comp rx=0 tx=0).stop 2024-04-12T05:58:22.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.598+0000 7f67c6b84700 1 -- 172.21.15.32:0/4043254281 shutdown_connections 2024-04-12T05:58:22.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.598+0000 7f67c6b84700 1 --2- 172.21.15.32:0/4043254281 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67c00fe810 0x7f67c0106e50 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:22.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.598+0000 7f67c6b84700 1 --2- 172.21.15.32:0/4043254281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67c00fde70 0x7f67c00fe240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:22.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.598+0000 7f67c6b84700 1 -- 172.21.15.32:0/4043254281 >> 172.21.15.32:0/4043254281 conn(0x7f67c00f7880 msgr2=0x7f67c00f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:58:22.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.598+0000 7f67c6b84700 1 -- 172.21.15.32:0/4043254281 shutdown_connections 2024-04-12T05:58:22.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.598+0000 7f67c6b84700 1 -- 172.21.15.32:0/4043254281 wait complete. 2024-04-12T05:58:22.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.598+0000 7f67c6b84700 1 Processor -- start 2024-04-12T05:58:22.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.598+0000 7f67c6b84700 1 -- start start 2024-04-12T05:58:22.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.599+0000 7f67c6b84700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67c00fde70 0x7f67c00fbca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:22.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.599+0000 7f67c6b84700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67c00fe810 0x7f67c00fc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:22.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.599+0000 7f67c6b84700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c00fc770 con 0x7f67c00fe810 2024-04-12T05:58:22.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.599+0000 7f67c6b84700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c00fc8b0 con 0x7f67c00fde70 2024-04-12T05:58:22.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.599+0000 7f67bffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67c00fe810 0x7f67c00fc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:22.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.599+0000 7f67bffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67c00fe810 0x7f67c00fc1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32792/0 (socket says 172.21.15.32:32792) 2024-04-12T05:58:22.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.599+0000 7f67bffff700 1 -- 172.21.15.32:0/1335235027 learned_addr learned my addr 172.21.15.32:0/1335235027 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:58:22.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.599+0000 7f67c4920700 1 --2- 172.21.15.32:0/1335235027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67c00fde70 0x7f67c00fbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:22.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.599+0000 7f67bffff700 1 -- 172.21.15.32:0/1335235027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67c00fde70 msgr2=0x7f67c00fbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:22.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.599+0000 7f67bffff700 1 --2- 172.21.15.32:0/1335235027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67c00fde70 0x7f67c00fbca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:22.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.599+0000 7f67bffff700 1 -- 172.21.15.32:0/1335235027 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67ac005040 con 0x7f67c00fe810 2024-04-12T05:58:22.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.599+0000 7f67c4920700 1 --2- 172.21.15.32:0/1335235027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67c00fde70 0x7f67c00fbca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:58:22.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.599+0000 7f67bffff700 1 --2- 172.21.15.32:0/1335235027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67c00fe810 0x7f67c00fc1e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f67ac009380 tx=0x7f67ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:58:22.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.600+0000 7f67bd7fa700 1 -- 172.21.15.32:0/1335235027 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67ac00e040 con 0x7f67c00fe810 2024-04-12T05:58:22.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.600+0000 7f67c6b84700 1 -- 172.21.15.32:0/1335235027 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67c011a030 con 0x7f67c00fe810 2024-04-12T05:58:22.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.600+0000 7f67bd7fa700 1 -- 172.21.15.32:0/1335235027 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f67ac004290 con 0x7f67c00fe810 2024-04-12T05:58:22.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.600+0000 7f67bd7fa700 1 -- 172.21.15.32:0/1335235027 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67ac01c3f0 con 0x7f67c00fe810 2024-04-12T05:58:22.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.600+0000 7f67c6b84700 1 -- 172.21.15.32:0/1335235027 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67c011a430 con 0x7f67c00fe810 2024-04-12T05:58:22.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.601+0000 7f67c6b84700 1 -- 172.21.15.32:0/1335235027 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67c0066710 con 0x7f67c00fe810 2024-04-12T05:58:22.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.601+0000 7f67bd7fa700 1 -- 172.21.15.32:0/1335235027 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f67ac013790 con 0x7f67c00fe810 2024-04-12T05:58:22.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.602+0000 7f67bd7fa700 1 --2- 172.21.15.32:0/1335235027 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f67b006cc20 0x7f67b006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:22.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.602+0000 7f67bd7fa700 1 -- 172.21.15.32:0/1335235027 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f67ac08d3c0 con 0x7f67c00fe810 2024-04-12T05:58:22.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.602+0000 7f67c4920700 1 --2- 172.21.15.32:0/1335235027 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f67b006cc20 0x7f67b006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:22.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.602+0000 7f67c4920700 1 --2- 172.21.15.32:0/1335235027 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f67b006cc20 0x7f67b006f0d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f67b4009fd0 tx=0x7f67b4009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:58:22.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.606+0000 7f67bd7fa700 1 -- 172.21.15.32:0/1335235027 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f67ac0573a0 con 0x7f67c00fe810 2024-04-12T05:58:22.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.819+0000 7f67c6b84700 1 -- 172.21.15.32:0/1335235027 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f67c0109d90 con 0x7f67b006cc20 2024-04-12T05:58:22.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.829+0000 7f67bd7fa700 1 -- 172.21.15.32:0/1335235027 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f67b4009970 con 0x7f67b006cc20 2024-04-12T05:58:22.824 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:58:22.824 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (18m) 11m ago 29m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:58:22.824 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (15m) 11m ago 29m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:58:22.824 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (15m) 11m ago 28m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:58:22.825 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (17m) 11m ago 29m 87.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:58:22.825 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (12m) 11m ago 23m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T05:58:22.825 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (11m) 11m ago 23m 13.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T05:58:22.825 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (11m) 11m ago 23m 11.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 6d4aca6f74a0 2024-04-12T05:58:22.825 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (11m) 11m ago 23m 14.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T05:58:22.825 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (19m) 11m ago 31m 507M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:58:22.825 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (19m) 11m ago 28m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:58:22.825 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (16m) 11m ago 31m 53.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:58:22.825 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (16m) 11m ago 28m 43.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:58:22.825 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (18m) 11m ago 29m 16.9M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:58:22.825 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (18m) 11m ago 28m 18.4M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:58:22.825 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (15m) 11m ago 27m 91.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T05:58:22.825 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (14m) 11m ago 27m 97.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T05:58:22.825 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (14m) 11m ago 26m 81.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T05:58:22.825 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (13m) 11m ago 26m 113M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T05:58:22.826 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (12m) 11m ago 25m 90.3M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T05:58:22.826 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (12m) 11m ago 25m 84.8M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T05:58:22.826 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (18m) 11m ago 29m 64.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:58:22.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.831+0000 7f67c6b84700 1 -- 172.21.15.32:0/1335235027 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f67b006cc20 msgr2=0x7f67b006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:22.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.831+0000 7f67c6b84700 1 --2- 172.21.15.32:0/1335235027 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f67b006cc20 0x7f67b006f0d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f67b4009fd0 tx=0x7f67b4009ee0 comp rx=0 tx=0).stop 2024-04-12T05:58:22.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.831+0000 7f67c6b84700 1 -- 172.21.15.32:0/1335235027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67c00fe810 msgr2=0x7f67c00fc1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:22.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.831+0000 7f67c6b84700 1 --2- 172.21.15.32:0/1335235027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67c00fe810 0x7f67c00fc1e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f67ac009380 tx=0x7f67ac009e30 comp rx=0 tx=0).stop 2024-04-12T05:58:22.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.832+0000 7f67c6b84700 1 -- 172.21.15.32:0/1335235027 shutdown_connections 2024-04-12T05:58:22.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.832+0000 7f67c6b84700 1 --2- 172.21.15.32:0/1335235027 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f67b006cc20 0x7f67b006f0d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:22.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.832+0000 7f67c6b84700 1 --2- 172.21.15.32:0/1335235027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67c00fde70 0x7f67c00fbca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:22.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.832+0000 7f67c6b84700 1 --2- 172.21.15.32:0/1335235027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67c00fe810 0x7f67c00fc1e0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:22.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.832+0000 7f67c6b84700 1 -- 172.21.15.32:0/1335235027 >> 172.21.15.32:0/1335235027 conn(0x7f67c00f7880 msgr2=0x7f67c00f8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:58:22.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.832+0000 7f67c6b84700 1 -- 172.21.15.32:0/1335235027 shutdown_connections 2024-04-12T05:58:22.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.832+0000 7f67c6b84700 1 -- 172.21.15.32:0/1335235027 wait complete. 2024-04-12T05:58:22.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.933+0000 7f50a23f6700 1 -- 172.21.15.32:0/2695589408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f509c0fde80 msgr2=0x7f509c0fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:22.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.933+0000 7f50a23f6700 1 --2- 172.21.15.32:0/2695589408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f509c0fde80 0x7f509c0fe250 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f508c004940 tx=0x7f508c009e30 comp rx=0 tx=0).stop 2024-04-12T05:58:22.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.934+0000 7f50a23f6700 1 -- 172.21.15.32:0/2695589408 shutdown_connections 2024-04-12T05:58:22.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.934+0000 7f50a23f6700 1 --2- 172.21.15.32:0/2695589408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f509c0fe820 0x7f509c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:22.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.934+0000 7f50a23f6700 1 --2- 172.21.15.32:0/2695589408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f509c0fde80 0x7f509c0fe250 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:22.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.934+0000 7f50a23f6700 1 -- 172.21.15.32:0/2695589408 >> 172.21.15.32:0/2695589408 conn(0x7f509c0f78f0 msgr2=0x7f509c0f9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:58:22.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.934+0000 7f50a23f6700 1 -- 172.21.15.32:0/2695589408 shutdown_connections 2024-04-12T05:58:22.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.934+0000 7f50a23f6700 1 -- 172.21.15.32:0/2695589408 wait complete. 2024-04-12T05:58:22.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.934+0000 7f50a23f6700 1 Processor -- start 2024-04-12T05:58:22.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.934+0000 7f50a23f6700 1 -- start start 2024-04-12T05:58:22.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.935+0000 7f50a23f6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f509c0fde80 0x7f509c113890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:22.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.935+0000 7f50a23f6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f509c0fe820 0x7f509c113dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:22.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.935+0000 7f50a23f6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509c114420 con 0x7f509c0fe820 2024-04-12T05:58:22.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.935+0000 7f50a23f6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509c10da20 con 0x7f509c0fde80 2024-04-12T05:58:22.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.935+0000 7f509b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f509c0fe820 0x7f509c113dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:22.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.935+0000 7f509b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f509c0fe820 0x7f509c113dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32804/0 (socket says 172.21.15.32:32804) 2024-04-12T05:58:22.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.935+0000 7f509b7fe700 1 -- 172.21.15.32:0/1869476672 learned_addr learned my addr 172.21.15.32:0/1869476672 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:58:22.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.935+0000 7f509bfff700 1 --2- 172.21.15.32:0/1869476672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f509c0fde80 0x7f509c113890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:22.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.935+0000 7f509b7fe700 1 -- 172.21.15.32:0/1869476672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f509c0fde80 msgr2=0x7f509c113890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:22.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.935+0000 7f509b7fe700 1 --2- 172.21.15.32:0/1869476672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f509c0fde80 0x7f509c113890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:22.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.935+0000 7f509b7fe700 1 -- 172.21.15.32:0/1869476672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f508c005040 con 0x7f509c0fe820 2024-04-12T05:58:22.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.935+0000 7f509b7fe700 1 --2- 172.21.15.32:0/1869476672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f509c0fe820 0x7f509c113dd0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f509000c8c0 tx=0x7f50900044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:58:22.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.935+0000 7f5098ff9700 1 -- 172.21.15.32:0/1869476672 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f509000a760 con 0x7f509c0fe820 2024-04-12T05:58:22.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.935+0000 7f5098ff9700 1 -- 172.21.15.32:0/1869476672 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f509000ad70 con 0x7f509c0fe820 2024-04-12T05:58:22.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.935+0000 7f50a23f6700 1 -- 172.21.15.32:0/1869476672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f509c10dd00 con 0x7f509c0fe820 2024-04-12T05:58:22.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.935+0000 7f5098ff9700 1 -- 172.21.15.32:0/1869476672 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50900113a0 con 0x7f509c0fe820 2024-04-12T05:58:22.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.935+0000 7f50a23f6700 1 -- 172.21.15.32:0/1869476672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f509c10e240 con 0x7f509c0fe820 2024-04-12T05:58:22.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.936+0000 7f50a23f6700 1 -- 172.21.15.32:0/1869476672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f509c066af0 con 0x7f509c0fe820 2024-04-12T05:58:22.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.937+0000 7f5098ff9700 1 -- 172.21.15.32:0/1869476672 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f5090012400 con 0x7f509c0fe820 2024-04-12T05:58:22.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.937+0000 7f5098ff9700 1 --2- 172.21.15.32:0/1869476672 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f508406cc20 0x7f508406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:22.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.937+0000 7f5098ff9700 1 -- 172.21.15.32:0/1869476672 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f509008c190 con 0x7f509c0fe820 2024-04-12T05:58:22.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.937+0000 7f509bfff700 1 --2- 172.21.15.32:0/1869476672 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f508406cc20 0x7f508406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:22.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.938+0000 7f509bfff700 1 --2- 172.21.15.32:0/1869476672 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f508406cc20 0x7f508406f0d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f508c004f90 tx=0x7f508c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:58:22.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:22.942+0000 7f5098ff9700 1 -- 172.21.15.32:0/1869476672 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5090059b60 con 0x7f509c0fe820 2024-04-12T05:58:23.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.229+0000 7f50a23f6700 1 -- 172.21.15.32:0/1869476672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f509c109dd0 con 0x7f509c0fe820 2024-04-12T05:58:23.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.229+0000 7f5098ff9700 1 -- 172.21.15.32:0/1869476672 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f50900596f0 con 0x7f509c0fe820 2024-04-12T05:58:23.224 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:58:23.224 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:58:23.224 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:58:23.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:58:23.225 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:58:23.225 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:58:23.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:58:23.225 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:58:23.225 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:58:23.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:58:23.225 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:58:23.225 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:58:23.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:58:23.225 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:58:23.225 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:58:23.225 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:58:23.225 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:58:23.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.231+0000 7f50a23f6700 1 -- 172.21.15.32:0/1869476672 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f508406cc20 msgr2=0x7f508406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:23.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.231+0000 7f50a23f6700 1 --2- 172.21.15.32:0/1869476672 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f508406cc20 0x7f508406f0d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f508c004f90 tx=0x7f508c009e30 comp rx=0 tx=0).stop 2024-04-12T05:58:23.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.231+0000 7f50a23f6700 1 -- 172.21.15.32:0/1869476672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f509c0fe820 msgr2=0x7f509c113dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:23.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.231+0000 7f50a23f6700 1 --2- 172.21.15.32:0/1869476672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f509c0fe820 0x7f509c113dd0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f509000c8c0 tx=0x7f50900044c0 comp rx=0 tx=0).stop 2024-04-12T05:58:23.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.231+0000 7f50a23f6700 1 -- 172.21.15.32:0/1869476672 shutdown_connections 2024-04-12T05:58:23.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.231+0000 7f50a23f6700 1 --2- 172.21.15.32:0/1869476672 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f508406cc20 0x7f508406f0d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:23.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.231+0000 7f50a23f6700 1 --2- 172.21.15.32:0/1869476672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f509c0fde80 0x7f509c113890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:23.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.231+0000 7f50a23f6700 1 --2- 172.21.15.32:0/1869476672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f509c0fe820 0x7f509c113dd0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:23.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.231+0000 7f50a23f6700 1 -- 172.21.15.32:0/1869476672 >> 172.21.15.32:0/1869476672 conn(0x7f509c0f78f0 msgr2=0x7f509c103740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:58:23.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.232+0000 7f50a23f6700 1 -- 172.21.15.32:0/1869476672 shutdown_connections 2024-04-12T05:58:23.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.232+0000 7f50a23f6700 1 -- 172.21.15.32:0/1869476672 wait complete. 2024-04-12T05:58:23.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.335+0000 7ff42dff2700 1 -- 172.21.15.32:0/2120330275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4280fbc80 msgr2=0x7ff4280fc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:23.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.335+0000 7ff42dff2700 1 --2- 172.21.15.32:0/2120330275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4280fbc80 0x7ff4280fc050 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff418004940 tx=0x7ff418009e30 comp rx=0 tx=0).stop 2024-04-12T05:58:23.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.335+0000 7ff42dff2700 1 -- 172.21.15.32:0/2120330275 shutdown_connections 2024-04-12T05:58:23.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.335+0000 7ff42dff2700 1 --2- 172.21.15.32:0/2120330275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4280fc590 0x7ff4281092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:23.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.335+0000 7ff42dff2700 1 --2- 172.21.15.32:0/2120330275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4280fbc80 0x7ff4280fc050 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:23.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.335+0000 7ff42dff2700 1 -- 172.21.15.32:0/2120330275 >> 172.21.15.32:0/2120330275 conn(0x7ff428072bc0 msgr2=0x7ff428072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:58:23.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.335+0000 7ff42dff2700 1 -- 172.21.15.32:0/2120330275 shutdown_connections 2024-04-12T05:58:23.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.335+0000 7ff42dff2700 1 -- 172.21.15.32:0/2120330275 wait complete. 2024-04-12T05:58:23.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.336+0000 7ff42dff2700 1 Processor -- start 2024-04-12T05:58:23.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.336+0000 7ff42dff2700 1 -- start start 2024-04-12T05:58:23.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.336+0000 7ff42dff2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4280fbc80 0x7ff428115bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:23.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.336+0000 7ff42dff2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4280fc590 0x7ff4281160f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:23.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.336+0000 7ff42dff2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff428116740 con 0x7ff4280fc590 2024-04-12T05:58:23.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.336+0000 7ff42dff2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42810fb50 con 0x7ff4280fbc80 2024-04-12T05:58:23.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.336+0000 7ff426ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4280fc590 0x7ff4281160f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:23.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.336+0000 7ff426ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4280fc590 0x7ff4281160f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32818/0 (socket says 172.21.15.32:32818) 2024-04-12T05:58:23.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.336+0000 7ff426ffd700 1 -- 172.21.15.32:0/2769394622 learned_addr learned my addr 172.21.15.32:0/2769394622 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:58:23.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.336+0000 7ff4277fe700 1 --2- 172.21.15.32:0/2769394622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4280fbc80 0x7ff428115bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:23.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.336+0000 7ff426ffd700 1 -- 172.21.15.32:0/2769394622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4280fbc80 msgr2=0x7ff428115bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:23.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.336+0000 7ff426ffd700 1 --2- 172.21.15.32:0/2769394622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4280fbc80 0x7ff428115bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:23.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.336+0000 7ff426ffd700 1 -- 172.21.15.32:0/2769394622 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff418005040 con 0x7ff4280fc590 2024-04-12T05:58:23.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.337+0000 7ff426ffd700 1 --2- 172.21.15.32:0/2769394622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4280fc590 0x7ff4281160f0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff41c00eb40 tx=0x7ff41c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:58:23.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.337+0000 7ff40ffff700 1 -- 172.21.15.32:0/2769394622 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff41c007cc0 con 0x7ff4280fc590 2024-04-12T05:58:23.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.337+0000 7ff40ffff700 1 -- 172.21.15.32:0/2769394622 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff41c00f040 con 0x7ff4280fc590 2024-04-12T05:58:23.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.337+0000 7ff42dff2700 1 -- 172.21.15.32:0/2769394622 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff42810fdd0 con 0x7ff4280fc590 2024-04-12T05:58:23.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.337+0000 7ff40ffff700 1 -- 172.21.15.32:0/2769394622 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff41c00c990 con 0x7ff4280fc590 2024-04-12T05:58:23.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.337+0000 7ff42dff2700 1 -- 172.21.15.32:0/2769394622 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff428110390 con 0x7ff4280fc590 2024-04-12T05:58:23.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.338+0000 7ff42dff2700 1 -- 172.21.15.32:0/2769394622 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff428065e30 con 0x7ff4280fc590 2024-04-12T05:58:23.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.338+0000 7ff40ffff700 1 -- 172.21.15.32:0/2769394622 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ff41c00caf0 con 0x7ff4280fc590 2024-04-12T05:58:23.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.339+0000 7ff40ffff700 1 --2- 172.21.15.32:0/2769394622 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff41006cc20 0x7ff41006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:23.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.339+0000 7ff40ffff700 1 -- 172.21.15.32:0/2769394622 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7ff41c08bc50 con 0x7ff4280fc590 2024-04-12T05:58:23.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.339+0000 7ff4277fe700 1 --2- 172.21.15.32:0/2769394622 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff41006cc20 0x7ff41006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:23.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.339+0000 7ff4277fe700 1 --2- 172.21.15.32:0/2769394622 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff41006cc20 0x7ff41006f0d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff418009fd0 tx=0x7ff418009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:58:23.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.343+0000 7ff40ffff700 1 -- 172.21.15.32:0/2769394622 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff41c055c80 con 0x7ff4280fc590 2024-04-12T05:58:23.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.590+0000 7ff42dff2700 1 -- 172.21.15.32:0/2769394622 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff42810c020 con 0x7ff4280fc590 2024-04-12T05:58:23.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.590+0000 7ff40ffff700 1 -- 172.21.15.32:0/2769394622 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1835 (secure 0 0 0) 0x7ff41c019740 con 0x7ff4280fc590 2024-04-12T05:58:23.586 INFO:teuthology.orchestra.run.smithi032.stdout:e33 2024-04-12T05:58:23.586 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:58:23.586 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:58:23.586 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:58:23.586 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:58:23.586 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:58:23.586 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:58:23.586 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 24 2024-04-12T05:58:23.586 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:58:23.586 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:58:23.586 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:46:10.668569+0000 2024-04-12T05:58:23.587 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:58:23.587 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:58:23.587 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:58:23.587 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:58:23.587 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:58:23.587 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:58:23.587 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:58:23.587 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:58:23.587 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T05:58:23.587 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:58:23.587 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:58:23.587 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:58:23.587 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T05:58:23.587 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-12T05:58:23.587 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:58:23.588 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:58:23.588 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:58:23.588 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:58:23.588 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:58:23.588 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:58:23.588 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:58:23.588 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:58:23.588 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:58:23.588 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:58:23.588 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-12T05:58:23.588 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:58:23.588 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:58:23.588 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:58:23.588 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{-1:44317} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:58:23.588 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:58:23.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:23 smithi032 ceph-mon[146817]: pgmap v536: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:23.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:23 smithi032 ceph-mon[146817]: from='client.35012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:58:23.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:23 smithi032 ceph-mon[146817]: from='client.35016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:58:23.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:23 smithi032 ceph-mon[146817]: from='client.35020 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:58:23.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:23 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1869476672' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:58:23.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.592+0000 7ff42dff2700 1 -- 172.21.15.32:0/2769394622 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff41006cc20 msgr2=0x7ff41006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:23.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.592+0000 7ff42dff2700 1 --2- 172.21.15.32:0/2769394622 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff41006cc20 0x7ff41006f0d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff418009fd0 tx=0x7ff418009e30 comp rx=0 tx=0).stop 2024-04-12T05:58:23.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.592+0000 7ff42dff2700 1 -- 172.21.15.32:0/2769394622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4280fc590 msgr2=0x7ff4281160f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:23.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.592+0000 7ff42dff2700 1 --2- 172.21.15.32:0/2769394622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4280fc590 0x7ff4281160f0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff41c00eb40 tx=0x7ff41c00b040 comp rx=0 tx=0).stop 2024-04-12T05:58:23.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.592+0000 7ff42dff2700 1 -- 172.21.15.32:0/2769394622 shutdown_connections 2024-04-12T05:58:23.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.592+0000 7ff42dff2700 1 --2- 172.21.15.32:0/2769394622 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff41006cc20 0x7ff41006f0d0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:23.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.592+0000 7ff42dff2700 1 --2- 172.21.15.32:0/2769394622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4280fbc80 0x7ff428115bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:23.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.592+0000 7ff42dff2700 1 --2- 172.21.15.32:0/2769394622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4280fc590 0x7ff4281160f0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:23.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.592+0000 7ff42dff2700 1 -- 172.21.15.32:0/2769394622 >> 172.21.15.32:0/2769394622 conn(0x7ff428072bc0 msgr2=0x7ff428105b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:58:23.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.593+0000 7ff42dff2700 1 -- 172.21.15.32:0/2769394622 shutdown_connections 2024-04-12T05:58:23.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.593+0000 7ff42dff2700 1 -- 172.21.15.32:0/2769394622 wait complete. 2024-04-12T05:58:23.591 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-12T05:58:23.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.697+0000 7f1488dbe700 1 -- 172.21.15.32:0/700579607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1484101d70 msgr2=0x7f1484102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:23.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.697+0000 7f1488dbe700 1 --2- 172.21.15.32:0/700579607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1484101d70 0x7f1484102140 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f1474004940 tx=0x7f1474009e30 comp rx=0 tx=0).stop 2024-04-12T05:58:23.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.697+0000 7f1488dbe700 1 -- 172.21.15.32:0/700579607 shutdown_connections 2024-04-12T05:58:23.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.697+0000 7f1488dbe700 1 --2- 172.21.15.32:0/700579607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1484102710 0x7f14840fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:23.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.697+0000 7f1488dbe700 1 --2- 172.21.15.32:0/700579607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1484101d70 0x7f1484102140 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:23.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.697+0000 7f1488dbe700 1 -- 172.21.15.32:0/700579607 >> 172.21.15.32:0/700579607 conn(0x7f1484072bc0 msgr2=0x7f1484072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:58:23.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.697+0000 7f1488dbe700 1 -- 172.21.15.32:0/700579607 shutdown_connections 2024-04-12T05:58:23.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.697+0000 7f1488dbe700 1 -- 172.21.15.32:0/700579607 wait complete. 2024-04-12T05:58:23.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.698+0000 7f1488dbe700 1 Processor -- start 2024-04-12T05:58:23.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.698+0000 7f1488dbe700 1 -- start start 2024-04-12T05:58:23.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.698+0000 7f1488dbe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1484101d70 0x7f1484113aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:23.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.698+0000 7f1488dbe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1484102710 0x7f1484113fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:23.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.698+0000 7f1488dbe700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1484114630 con 0x7f1484101d70 2024-04-12T05:58:23.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.698+0000 7f1488dbe700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148410da20 con 0x7f1484102710 2024-04-12T05:58:23.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.698+0000 7f148259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1484101d70 0x7f1484113aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:23.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.698+0000 7f148259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1484101d70 0x7f1484113aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32828/0 (socket says 172.21.15.32:32828) 2024-04-12T05:58:23.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.698+0000 7f148259c700 1 -- 172.21.15.32:0/4112215234 learned_addr learned my addr 172.21.15.32:0/4112215234 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:58:23.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.698+0000 7f1481d9b700 1 --2- 172.21.15.32:0/4112215234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1484102710 0x7f1484113fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:23.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.698+0000 7f148259c700 1 -- 172.21.15.32:0/4112215234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1484102710 msgr2=0x7f1484113fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:23.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.698+0000 7f148259c700 1 --2- 172.21.15.32:0/4112215234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1484102710 0x7f1484113fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:23.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.698+0000 7f148259c700 1 -- 172.21.15.32:0/4112215234 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1474005040 con 0x7f1484101d70 2024-04-12T05:58:23.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.698+0000 7f1481d9b700 1 --2- 172.21.15.32:0/4112215234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1484102710 0x7f1484113fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:58:23.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.699+0000 7f148259c700 1 --2- 172.21.15.32:0/4112215234 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1484101d70 0x7f1484113aa0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f1474009180 tx=0x7f1474009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:58:23.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.699+0000 7f147affd700 1 -- 172.21.15.32:0/4112215234 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f147400e040 con 0x7f1484101d70 2024-04-12T05:58:23.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.699+0000 7f147affd700 1 -- 172.21.15.32:0/4112215234 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1474004290 con 0x7f1484101d70 2024-04-12T05:58:23.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.699+0000 7f1488dbe700 1 -- 172.21.15.32:0/4112215234 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f148410dca0 con 0x7f1484101d70 2024-04-12T05:58:23.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.699+0000 7f147affd700 1 -- 172.21.15.32:0/4112215234 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f147401c3f0 con 0x7f1484101d70 2024-04-12T05:58:23.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.699+0000 7f1488dbe700 1 -- 172.21.15.32:0/4112215234 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f148410e260 con 0x7f1484101d70 2024-04-12T05:58:23.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.700+0000 7f1488dbe700 1 -- 172.21.15.32:0/4112215234 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14840ff080 con 0x7f1484101d70 2024-04-12T05:58:23.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.700+0000 7f147affd700 1 -- 172.21.15.32:0/4112215234 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1474013790 con 0x7f1484101d70 2024-04-12T05:58:23.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.701+0000 7f147affd700 1 --2- 172.21.15.32:0/4112215234 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f147006cc20 0x7f147006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:23.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.701+0000 7f147affd700 1 -- 172.21.15.32:0/4112215234 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f147408d2b0 con 0x7f1484101d70 2024-04-12T05:58:23.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.701+0000 7f1481d9b700 1 --2- 172.21.15.32:0/4112215234 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f147006cc20 0x7f147006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:23.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.701+0000 7f1481d9b700 1 --2- 172.21.15.32:0/4112215234 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f147006cc20 0x7f147006f0d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f146c007fd0 tx=0x7f146c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:58:23.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.705+0000 7f147affd700 1 -- 172.21.15.32:0/4112215234 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f147405adc0 con 0x7f1484101d70 2024-04-12T05:58:23.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:23 smithi067 ceph-mon[112979]: pgmap v536: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:23.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:23 smithi067 ceph-mon[112979]: from='client.35012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:58:23.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:23 smithi067 ceph-mon[112979]: from='client.35016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:58:23.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:23 smithi067 ceph-mon[112979]: from='client.35020 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:58:23.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:23 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1869476672' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:58:23.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.927+0000 7f1488dbe700 1 -- 172.21.15.32:0/4112215234 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f148410f0f0 con 0x7f147006cc20 2024-04-12T05:58:23.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.928+0000 7f147affd700 1 -- 172.21.15.32:0/4112215234 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f148410f0f0 con 0x7f147006cc20 2024-04-12T05:58:23.923 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:58:23.923 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:58:23.923 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:58:23.923 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:58:23.923 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:58:23.923 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-12T05:58:23.923 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:58:23.923 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-12T05:58:23.923 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:58:23.924 INFO:teuthology.orchestra.run.smithi032.stdout: "mds" 2024-04-12T05:58:23.924 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:58:23.924 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:58:23.924 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:58:23.924 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:58:23.924 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:58:23.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.930+0000 7f1488dbe700 1 -- 172.21.15.32:0/4112215234 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f147006cc20 msgr2=0x7f147006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:23.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.930+0000 7f1488dbe700 1 --2- 172.21.15.32:0/4112215234 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f147006cc20 0x7f147006f0d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f146c007fd0 tx=0x7f146c009040 comp rx=0 tx=0).stop 2024-04-12T05:58:23.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.930+0000 7f1488dbe700 1 -- 172.21.15.32:0/4112215234 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1484101d70 msgr2=0x7f1484113aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:23.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.930+0000 7f1488dbe700 1 --2- 172.21.15.32:0/4112215234 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1484101d70 0x7f1484113aa0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f1474009180 tx=0x7f1474009e30 comp rx=0 tx=0).stop 2024-04-12T05:58:23.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.930+0000 7f1488dbe700 1 -- 172.21.15.32:0/4112215234 shutdown_connections 2024-04-12T05:58:23.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.930+0000 7f1488dbe700 1 --2- 172.21.15.32:0/4112215234 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f147006cc20 0x7f147006f0d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:23.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.930+0000 7f1488dbe700 1 --2- 172.21.15.32:0/4112215234 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1484101d70 0x7f1484113aa0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:23.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.930+0000 7f1488dbe700 1 --2- 172.21.15.32:0/4112215234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1484102710 0x7f1484113fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:23.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.930+0000 7f1488dbe700 1 -- 172.21.15.32:0/4112215234 >> 172.21.15.32:0/4112215234 conn(0x7f1484072bc0 msgr2=0x7f14840fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:58:23.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.930+0000 7f1488dbe700 1 -- 172.21.15.32:0/4112215234 shutdown_connections 2024-04-12T05:58:23.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:23.930+0000 7f1488dbe700 1 -- 172.21.15.32:0/4112215234 wait complete. 2024-04-12T05:58:24.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.034+0000 7f37b2e11700 1 -- 172.21.15.32:0/2598401468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37ac102e10 msgr2=0x7f37ac1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:24.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.034+0000 7f37b2e11700 1 --2- 172.21.15.32:0/2598401468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37ac102e10 0x7f37ac1031e0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f37a0004940 tx=0x7f37a0009e30 comp rx=0 tx=0).stop 2024-04-12T05:58:24.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.034+0000 7f37b2e11700 1 -- 172.21.15.32:0/2598401468 shutdown_connections 2024-04-12T05:58:24.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.034+0000 7f37b2e11700 1 --2- 172.21.15.32:0/2598401468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37ac1037b0 0x7f37ac0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:24.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.034+0000 7f37b2e11700 1 --2- 172.21.15.32:0/2598401468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37ac102e10 0x7f37ac1031e0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:24.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.034+0000 7f37b2e11700 1 -- 172.21.15.32:0/2598401468 >> 172.21.15.32:0/2598401468 conn(0x7f37ac0f89a0 msgr2=0x7f37ac0fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:58:24.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.034+0000 7f37b2e11700 1 -- 172.21.15.32:0/2598401468 shutdown_connections 2024-04-12T05:58:24.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.034+0000 7f37b2e11700 1 -- 172.21.15.32:0/2598401468 wait complete. 2024-04-12T05:58:24.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.034+0000 7f37b2e11700 1 Processor -- start 2024-04-12T05:58:24.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.034+0000 7f37b2e11700 1 -- start start 2024-04-12T05:58:24.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.035+0000 7f37b2e11700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37ac102e10 0x7f37ac0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:24.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.035+0000 7f37b2e11700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37ac1037b0 0x7f37ac070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:24.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.035+0000 7f37b2e11700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ac070d90 con 0x7f37ac102e10 2024-04-12T05:58:24.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.035+0000 7f37b2e11700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ac070ed0 con 0x7f37ac1037b0 2024-04-12T05:58:24.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.035+0000 7f37b0bad700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37ac102e10 0x7f37ac0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:24.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.035+0000 7f37b0bad700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37ac102e10 0x7f37ac0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32850/0 (socket says 172.21.15.32:32850) 2024-04-12T05:58:24.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.035+0000 7f37b0bad700 1 -- 172.21.15.32:0/2380547434 learned_addr learned my addr 172.21.15.32:0/2380547434 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:58:24.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.035+0000 7f37abfff700 1 --2- 172.21.15.32:0/2380547434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37ac1037b0 0x7f37ac070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:24.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.035+0000 7f37b0bad700 1 -- 172.21.15.32:0/2380547434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37ac1037b0 msgr2=0x7f37ac070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:24.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.035+0000 7f37b0bad700 1 --2- 172.21.15.32:0/2380547434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37ac1037b0 0x7f37ac070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:24.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.035+0000 7f37b0bad700 1 -- 172.21.15.32:0/2380547434 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a0005040 con 0x7f37ac102e10 2024-04-12T05:58:24.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.035+0000 7f37b0bad700 1 --2- 172.21.15.32:0/2380547434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37ac102e10 0x7f37ac0702c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f37a0008d80 tx=0x7f37a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:58:24.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.036+0000 7f37a97fa700 1 -- 172.21.15.32:0/2380547434 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37a000e030 con 0x7f37ac102e10 2024-04-12T05:58:24.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.036+0000 7f37a97fa700 1 -- 172.21.15.32:0/2380547434 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f37a0013de0 con 0x7f37ac102e10 2024-04-12T05:58:24.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.036+0000 7f37b2e11700 1 -- 172.21.15.32:0/2380547434 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37ac071120 con 0x7f37ac102e10 2024-04-12T05:58:24.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.036+0000 7f37a97fa700 1 -- 172.21.15.32:0/2380547434 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37a001cd00 con 0x7f37ac102e10 2024-04-12T05:58:24.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.036+0000 7f37b2e11700 1 -- 172.21.15.32:0/2380547434 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37ac11a180 con 0x7f37ac102e10 2024-04-12T05:58:24.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.037+0000 7f37b2e11700 1 -- 172.21.15.32:0/2380547434 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37ac0fe560 con 0x7f37ac102e10 2024-04-12T05:58:24.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.037+0000 7f37a97fa700 1 -- 172.21.15.32:0/2380547434 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f37a0025400 con 0x7f37ac102e10 2024-04-12T05:58:24.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.037+0000 7f37a97fa700 1 --2- 172.21.15.32:0/2380547434 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f379c06cc30 0x7f379c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:24.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.038+0000 7f37abfff700 1 --2- 172.21.15.32:0/2380547434 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f379c06cc30 0x7f379c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:24.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.038+0000 7f37a97fa700 1 -- 172.21.15.32:0/2380547434 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f37a008d800 con 0x7f37ac102e10 2024-04-12T05:58:24.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.038+0000 7f37abfff700 1 --2- 172.21.15.32:0/2380547434 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f379c06cc30 0x7f379c06f0e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f37ac071c00 tx=0x7f379800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:58:24.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.042+0000 7f37a97fa700 1 -- 172.21.15.32:0/2380547434 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f37a0057880 con 0x7f37ac102e10 2024-04-12T05:58:24.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.327+0000 7f37b2e11700 1 -- 172.21.15.32:0/2380547434 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f37ac05f5f0 con 0x7f37ac102e10 2024-04-12T05:58:24.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.328+0000 7f37a97fa700 1 -- 172.21.15.32:0/2380547434 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f37a001ce60 con 0x7f37ac102e10 2024-04-12T05:58:24.323 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:58:24.323 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:58:24.323 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-12T05:58:24.324 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:58:24.324 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:58:24.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.330+0000 7f37b2e11700 1 -- 172.21.15.32:0/2380547434 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f379c06cc30 msgr2=0x7f379c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:24.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.330+0000 7f37b2e11700 1 --2- 172.21.15.32:0/2380547434 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f379c06cc30 0x7f379c06f0e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f37ac071c00 tx=0x7f379800b040 comp rx=0 tx=0).stop 2024-04-12T05:58:24.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.330+0000 7f37b2e11700 1 -- 172.21.15.32:0/2380547434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37ac102e10 msgr2=0x7f37ac0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:24.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.330+0000 7f37b2e11700 1 --2- 172.21.15.32:0/2380547434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37ac102e10 0x7f37ac0702c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f37a0008d80 tx=0x7f37a0009e30 comp rx=0 tx=0).stop 2024-04-12T05:58:24.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.330+0000 7f37b2e11700 1 -- 172.21.15.32:0/2380547434 shutdown_connections 2024-04-12T05:58:24.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.330+0000 7f37b2e11700 1 --2- 172.21.15.32:0/2380547434 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f379c06cc30 0x7f379c06f0e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:24.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.330+0000 7f37b2e11700 1 --2- 172.21.15.32:0/2380547434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37ac102e10 0x7f37ac0702c0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:24.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.330+0000 7f37b2e11700 1 --2- 172.21.15.32:0/2380547434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37ac1037b0 0x7f37ac070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:24.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.330+0000 7f37b2e11700 1 -- 172.21.15.32:0/2380547434 >> 172.21.15.32:0/2380547434 conn(0x7f37ac0f89a0 msgr2=0x7f37ac0f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:58:24.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.330+0000 7f37b2e11700 1 -- 172.21.15.32:0/2380547434 shutdown_connections 2024-04-12T05:58:24.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:24.330+0000 7f37b2e11700 1 -- 172.21.15.32:0/2380547434 wait complete. 2024-04-12T05:58:24.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:24 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/2769394622' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:58:24.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:24 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/2380547434' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:58:24.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:24 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/2769394622' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:58:24.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:24 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/2380547434' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:58:25.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:25 smithi032 ceph-mon[146817]: from='client.35032 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:58:25.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:25 smithi032 ceph-mon[146817]: pgmap v537: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:25.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:25 smithi067 ceph-mon[112979]: from='client.35032 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:58:25.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:25 smithi067 ceph-mon[112979]: pgmap v537: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:27.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:27 smithi032 ceph-mon[146817]: pgmap v538: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:27.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:27 smithi067 ceph-mon[112979]: pgmap v538: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:29.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:29 smithi067 ceph-mon[112979]: pgmap v539: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:29.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:29 smithi032 ceph-mon[146817]: pgmap v539: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:31.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:31 smithi067 ceph-mon[112979]: pgmap v540: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:31.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:31 smithi032 ceph-mon[146817]: pgmap v540: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:33.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:33 smithi067 ceph-mon[112979]: pgmap v541: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:33.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:33 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:58:33.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:33 smithi032 ceph-mon[146817]: pgmap v541: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:33.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:33 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:58:35.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:35 smithi067 ceph-mon[112979]: pgmap v542: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:35.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:35 smithi032 ceph-mon[146817]: pgmap v542: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:37.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:37 smithi067 ceph-mon[112979]: pgmap v543: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:37.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:37 smithi032 ceph-mon[146817]: pgmap v543: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:39.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:39 smithi067 ceph-mon[112979]: pgmap v544: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:39.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:39 smithi032 ceph-mon[146817]: pgmap v544: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:41.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:41 smithi067 ceph-mon[112979]: pgmap v545: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:41.836 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:41 smithi032 ceph-mon[146817]: pgmap v545: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:43.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:43 smithi067 ceph-mon[112979]: pgmap v546: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:43.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:43 smithi032 ceph-mon[146817]: pgmap v546: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:45.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:45 smithi067 ceph-mon[112979]: pgmap v547: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:45.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:45 smithi032 ceph-mon[146817]: pgmap v547: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:47.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:47 smithi067 ceph-mon[112979]: pgmap v548: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:47.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:47 smithi032 ceph-mon[146817]: pgmap v548: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:48.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:48 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:58:48.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:48 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:58:49.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:49 smithi067 ceph-mon[112979]: pgmap v549: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:49.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:49 smithi032 ceph-mon[146817]: pgmap v549: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:51.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:51 smithi067 ceph-mon[112979]: pgmap v550: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:51.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:51 smithi032 ceph-mon[146817]: pgmap v550: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:53.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:53 smithi067 ceph-mon[112979]: pgmap v551: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:53.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:53 smithi032 ceph-mon[146817]: pgmap v551: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:54.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.450+0000 7f3f401d4700 1 -- 172.21.15.32:0/682801304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f38101d70 msgr2=0x7f3f38102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:54.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.450+0000 7f3f401d4700 1 --2- 172.21.15.32:0/682801304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f38101d70 0x7f3f38102140 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f3f34004940 tx=0x7f3f34009e30 comp rx=0 tx=0).stop 2024-04-12T05:58:54.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.450+0000 7f3f401d4700 1 -- 172.21.15.32:0/682801304 shutdown_connections 2024-04-12T05:58:54.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.450+0000 7f3f401d4700 1 --2- 172.21.15.32:0/682801304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f38102710 0x7f3f380fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:54.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.450+0000 7f3f401d4700 1 --2- 172.21.15.32:0/682801304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f38101d70 0x7f3f38102140 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:54.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.450+0000 7f3f401d4700 1 -- 172.21.15.32:0/682801304 >> 172.21.15.32:0/682801304 conn(0x7f3f38072bc0 msgr2=0x7f3f38072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:58:54.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.450+0000 7f3f401d4700 1 -- 172.21.15.32:0/682801304 shutdown_connections 2024-04-12T05:58:54.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.450+0000 7f3f401d4700 1 -- 172.21.15.32:0/682801304 wait complete. 2024-04-12T05:58:54.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.450+0000 7f3f401d4700 1 Processor -- start 2024-04-12T05:58:54.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.450+0000 7f3f401d4700 1 -- start start 2024-04-12T05:58:54.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.451+0000 7f3f401d4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f38101d70 0x7f3f38102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:54.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.451+0000 7f3f401d4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f38102710 0x7f3f38103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:54.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.451+0000 7f3f401d4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f381036a0 con 0x7f3f38101d70 2024-04-12T05:58:54.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.451+0000 7f3f401d4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f381037e0 con 0x7f3f38102710 2024-04-12T05:58:54.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.451+0000 7f3f3df70700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f38101d70 0x7f3f38102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:54.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.451+0000 7f3f3df70700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f38101d70 0x7f3f38102bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34494/0 (socket says 172.21.15.32:34494) 2024-04-12T05:58:54.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.451+0000 7f3f3df70700 1 -- 172.21.15.32:0/3802070194 learned_addr learned my addr 172.21.15.32:0/3802070194 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:58:54.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.451+0000 7f3f3d76f700 1 --2- 172.21.15.32:0/3802070194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f38102710 0x7f3f38103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:54.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.451+0000 7f3f3df70700 1 -- 172.21.15.32:0/3802070194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f38102710 msgr2=0x7f3f38103110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:54.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.451+0000 7f3f3df70700 1 --2- 172.21.15.32:0/3802070194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f38102710 0x7f3f38103110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:54.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.451+0000 7f3f3df70700 1 -- 172.21.15.32:0/3802070194 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f34005040 con 0x7f3f38101d70 2024-04-12T05:58:54.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.451+0000 7f3f3df70700 1 --2- 172.21.15.32:0/3802070194 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f38101d70 0x7f3f38102bd0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f3f3400b3b0 tx=0x7f3f34009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:58:54.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.451+0000 7f3f2e7fc700 1 -- 172.21.15.32:0/3802070194 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f3400e040 con 0x7f3f38101d70 2024-04-12T05:58:54.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.451+0000 7f3f2e7fc700 1 -- 172.21.15.32:0/3802070194 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f34013420 con 0x7f3f38101d70 2024-04-12T05:58:54.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.452+0000 7f3f401d4700 1 -- 172.21.15.32:0/3802070194 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f38103a30 con 0x7f3f38101d70 2024-04-12T05:58:54.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.452+0000 7f3f2e7fc700 1 -- 172.21.15.32:0/3802070194 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f3401c610 con 0x7f3f38101d70 2024-04-12T05:58:54.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.452+0000 7f3f401d4700 1 -- 172.21.15.32:0/3802070194 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f3811a1b0 con 0x7f3f38101d70 2024-04-12T05:58:54.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.452+0000 7f3f401d4700 1 -- 172.21.15.32:0/3802070194 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f380fd4c0 con 0x7f3f38101d70 2024-04-12T05:58:54.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.453+0000 7f3f2e7fc700 1 -- 172.21.15.32:0/3802070194 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3f34013a00 con 0x7f3f38101d70 2024-04-12T05:58:54.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.454+0000 7f3f2e7fc700 1 --2- 172.21.15.32:0/3802070194 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3f2406cc20 0x7f3f2406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:54.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.454+0000 7f3f2e7fc700 1 -- 172.21.15.32:0/3802070194 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f3f3408d240 con 0x7f3f38101d70 2024-04-12T05:58:54.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.454+0000 7f3f3d76f700 1 --2- 172.21.15.32:0/3802070194 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3f2406cc20 0x7f3f2406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:54.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.454+0000 7f3f3d76f700 1 --2- 172.21.15.32:0/3802070194 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3f2406cc20 0x7f3f2406f0d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3f38104510 tx=0x7f3f2800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:58:54.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.458+0000 7f3f2e7fc700 1 -- 172.21.15.32:0/3802070194 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3f34057220 con 0x7f3f38101d70 2024-04-12T05:58:54.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.681+0000 7f3f401d4700 1 -- 172.21.15.32:0/3802070194 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f381042c0 con 0x7f3f2406cc20 2024-04-12T05:58:54.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.682+0000 7f3f2e7fc700 1 -- 172.21.15.32:0/3802070194 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3f381042c0 con 0x7f3f2406cc20 2024-04-12T05:58:54.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.684+0000 7f3f401d4700 1 -- 172.21.15.32:0/3802070194 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3f2406cc20 msgr2=0x7f3f2406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:54.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.684+0000 7f3f401d4700 1 --2- 172.21.15.32:0/3802070194 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3f2406cc20 0x7f3f2406f0d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3f38104510 tx=0x7f3f2800b040 comp rx=0 tx=0).stop 2024-04-12T05:58:54.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.684+0000 7f3f401d4700 1 -- 172.21.15.32:0/3802070194 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f38101d70 msgr2=0x7f3f38102bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:54.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.684+0000 7f3f401d4700 1 --2- 172.21.15.32:0/3802070194 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f38101d70 0x7f3f38102bd0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f3f3400b3b0 tx=0x7f3f34009e30 comp rx=0 tx=0).stop 2024-04-12T05:58:54.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.684+0000 7f3f401d4700 1 -- 172.21.15.32:0/3802070194 shutdown_connections 2024-04-12T05:58:54.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.684+0000 7f3f401d4700 1 --2- 172.21.15.32:0/3802070194 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3f2406cc20 0x7f3f2406f0d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:54.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.684+0000 7f3f401d4700 1 --2- 172.21.15.32:0/3802070194 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f38101d70 0x7f3f38102bd0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:54.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.684+0000 7f3f401d4700 1 --2- 172.21.15.32:0/3802070194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f38102710 0x7f3f38103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:54.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.684+0000 7f3f401d4700 1 -- 172.21.15.32:0/3802070194 >> 172.21.15.32:0/3802070194 conn(0x7f3f38072bc0 msgr2=0x7f3f380659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:58:54.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.684+0000 7f3f401d4700 1 -- 172.21.15.32:0/3802070194 shutdown_connections 2024-04-12T05:58:54.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.684+0000 7f3f401d4700 1 -- 172.21.15.32:0/3802070194 wait complete. 2024-04-12T05:58:54.689 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:58:54.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.791+0000 7f90bf4cf700 1 -- 172.21.15.32:0/3897793142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90b80fe7c0 msgr2=0x7f90b8106e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:54.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.791+0000 7f90bf4cf700 1 --2- 172.21.15.32:0/3897793142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90b80fe7c0 0x7f90b8106e00 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f90a8004940 tx=0x7f90a8009e30 comp rx=0 tx=0).stop 2024-04-12T05:58:54.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.792+0000 7f90bf4cf700 1 -- 172.21.15.32:0/3897793142 shutdown_connections 2024-04-12T05:58:54.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.792+0000 7f90bf4cf700 1 --2- 172.21.15.32:0/3897793142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90b80fe7c0 0x7f90b8106e00 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:54.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.792+0000 7f90bf4cf700 1 --2- 172.21.15.32:0/3897793142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90b80fde20 0x7f90b80fe1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:54.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.792+0000 7f90bf4cf700 1 -- 172.21.15.32:0/3897793142 >> 172.21.15.32:0/3897793142 conn(0x7f90b80f7870 msgr2=0x7f90b80f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:58:54.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.792+0000 7f90bf4cf700 1 -- 172.21.15.32:0/3897793142 shutdown_connections 2024-04-12T05:58:54.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.792+0000 7f90bf4cf700 1 -- 172.21.15.32:0/3897793142 wait complete. 2024-04-12T05:58:54.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.792+0000 7f90bf4cf700 1 Processor -- start 2024-04-12T05:58:54.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.792+0000 7f90bf4cf700 1 -- start start 2024-04-12T05:58:54.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.793+0000 7f90bf4cf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90b80fde20 0x7f90b8113820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:54.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.793+0000 7f90bf4cf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90b80fe7c0 0x7f90b8113d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:54.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.793+0000 7f90bf4cf700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b81143b0 con 0x7f90b80fde20 2024-04-12T05:58:54.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.793+0000 7f90bf4cf700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b810d9b0 con 0x7f90b80fe7c0 2024-04-12T05:58:54.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.793+0000 7f90bd26b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90b80fde20 0x7f90b8113820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:54.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.793+0000 7f90bd26b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90b80fde20 0x7f90b8113820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34512/0 (socket says 172.21.15.32:34512) 2024-04-12T05:58:54.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.793+0000 7f90bd26b700 1 -- 172.21.15.32:0/3481637170 learned_addr learned my addr 172.21.15.32:0/3481637170 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:58:54.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.793+0000 7f90bca6a700 1 --2- 172.21.15.32:0/3481637170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90b80fe7c0 0x7f90b8113d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:54.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.793+0000 7f90bd26b700 1 -- 172.21.15.32:0/3481637170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90b80fe7c0 msgr2=0x7f90b8113d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:54.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.793+0000 7f90bd26b700 1 --2- 172.21.15.32:0/3481637170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90b80fe7c0 0x7f90b8113d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:54.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.793+0000 7f90bd26b700 1 -- 172.21.15.32:0/3481637170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a8005040 con 0x7f90b80fde20 2024-04-12T05:58:54.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.793+0000 7f90bd26b700 1 --2- 172.21.15.32:0/3481637170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90b80fde20 0x7f90b8113820 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f90b4002a50 tx=0x7f90b40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:58:54.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.794+0000 7f90adffb700 1 -- 172.21.15.32:0/3481637170 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90b4007cc0 con 0x7f90b80fde20 2024-04-12T05:58:54.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.794+0000 7f90bf4cf700 1 -- 172.21.15.32:0/3481637170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90b810dc90 con 0x7f90b80fde20 2024-04-12T05:58:54.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.794+0000 7f90adffb700 1 -- 172.21.15.32:0/3481637170 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f90b400f040 con 0x7f90b80fde20 2024-04-12T05:58:54.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.794+0000 7f90adffb700 1 -- 172.21.15.32:0/3481637170 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90b4009540 con 0x7f90b80fde20 2024-04-12T05:58:54.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.794+0000 7f90bf4cf700 1 -- 172.21.15.32:0/3481637170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90b810e1d0 con 0x7f90b80fde20 2024-04-12T05:58:54.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.795+0000 7f90bf4cf700 1 -- 172.21.15.32:0/3481637170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90b8066b50 con 0x7f90b80fde20 2024-04-12T05:58:54.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.795+0000 7f90adffb700 1 -- 172.21.15.32:0/3481637170 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f90b40096a0 con 0x7f90b80fde20 2024-04-12T05:58:54.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.796+0000 7f90adffb700 1 --2- 172.21.15.32:0/3481637170 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f90a406cc20 0x7f90a406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:54.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.796+0000 7f90adffb700 1 -- 172.21.15.32:0/3481637170 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f90b408bf30 con 0x7f90b80fde20 2024-04-12T05:58:54.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.796+0000 7f90bca6a700 1 --2- 172.21.15.32:0/3481637170 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f90a406cc20 0x7f90a406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:54.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.796+0000 7f90bca6a700 1 --2- 172.21.15.32:0/3481637170 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f90a406cc20 0x7f90a406f0d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f90a800b3b0 tx=0x7f90a8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:58:54.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:54.801+0000 7f90adffb700 1 -- 172.21.15.32:0/3481637170 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f90b40598c0 con 0x7f90b80fde20 2024-04-12T05:58:55.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.021+0000 7f90bf4cf700 1 -- 172.21.15.32:0/3481637170 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f90b8109d70 con 0x7f90a406cc20 2024-04-12T05:58:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.022+0000 7f90adffb700 1 -- 172.21.15.32:0/3481637170 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f90b8109d70 con 0x7f90a406cc20 2024-04-12T05:58:55.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.025+0000 7f90bf4cf700 1 -- 172.21.15.32:0/3481637170 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f90a406cc20 msgr2=0x7f90a406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:55.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.025+0000 7f90bf4cf700 1 --2- 172.21.15.32:0/3481637170 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f90a406cc20 0x7f90a406f0d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f90a800b3b0 tx=0x7f90a8009e30 comp rx=0 tx=0).stop 2024-04-12T05:58:55.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.025+0000 7f90bf4cf700 1 -- 172.21.15.32:0/3481637170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90b80fde20 msgr2=0x7f90b8113820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:55.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.025+0000 7f90bf4cf700 1 --2- 172.21.15.32:0/3481637170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90b80fde20 0x7f90b8113820 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f90b4002a50 tx=0x7f90b40046b0 comp rx=0 tx=0).stop 2024-04-12T05:58:55.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.025+0000 7f90bf4cf700 1 -- 172.21.15.32:0/3481637170 shutdown_connections 2024-04-12T05:58:55.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.025+0000 7f90bf4cf700 1 --2- 172.21.15.32:0/3481637170 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f90a406cc20 0x7f90a406f0d0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:55.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.025+0000 7f90bf4cf700 1 --2- 172.21.15.32:0/3481637170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90b80fde20 0x7f90b8113820 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:55.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.025+0000 7f90bf4cf700 1 --2- 172.21.15.32:0/3481637170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90b80fe7c0 0x7f90b8113d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:55.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.025+0000 7f90bf4cf700 1 -- 172.21.15.32:0/3481637170 >> 172.21.15.32:0/3481637170 conn(0x7f90b80f7870 msgr2=0x7f90b80f8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:58:55.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.025+0000 7f90bf4cf700 1 -- 172.21.15.32:0/3481637170 shutdown_connections 2024-04-12T05:58:55.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.025+0000 7f90bf4cf700 1 -- 172.21.15.32:0/3481637170 wait complete. 2024-04-12T05:58:55.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.128+0000 7fdc2887c700 1 -- 172.21.15.32:0/2952858064 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc241037a0 msgr2=0x7fdc240fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:55.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.128+0000 7fdc2887c700 1 --2- 172.21.15.32:0/2952858064 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc241037a0 0x7fdc240fce10 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fdc18004940 tx=0x7fdc18009e30 comp rx=0 tx=0).stop 2024-04-12T05:58:55.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.128+0000 7fdc2887c700 1 -- 172.21.15.32:0/2952858064 shutdown_connections 2024-04-12T05:58:55.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.128+0000 7fdc2887c700 1 --2- 172.21.15.32:0/2952858064 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc241037a0 0x7fdc240fce10 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:55.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.128+0000 7fdc2887c700 1 --2- 172.21.15.32:0/2952858064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc24102e00 0x7fdc241031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:55.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.128+0000 7fdc2887c700 1 -- 172.21.15.32:0/2952858064 >> 172.21.15.32:0/2952858064 conn(0x7fdc240f89b0 msgr2=0x7fdc240fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:58:55.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.128+0000 7fdc2887c700 1 -- 172.21.15.32:0/2952858064 shutdown_connections 2024-04-12T05:58:55.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.128+0000 7fdc2887c700 1 -- 172.21.15.32:0/2952858064 wait complete. 2024-04-12T05:58:55.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.128+0000 7fdc2887c700 1 Processor -- start 2024-04-12T05:58:55.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.128+0000 7fdc2887c700 1 -- start start 2024-04-12T05:58:55.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.129+0000 7fdc2887c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc24102e00 0x7fdc24113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:55.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.129+0000 7fdc2887c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc241037a0 0x7fdc24114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:55.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.129+0000 7fdc2887c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc24114690 con 0x7fdc241037a0 2024-04-12T05:58:55.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.129+0000 7fdc2887c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2410da80 con 0x7fdc24102e00 2024-04-12T05:58:55.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.129+0000 7fdc21d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc241037a0 0x7fdc24114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:55.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.129+0000 7fdc21d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc241037a0 0x7fdc24114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34540/0 (socket says 172.21.15.32:34540) 2024-04-12T05:58:55.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.129+0000 7fdc21d9b700 1 -- 172.21.15.32:0/3340807651 learned_addr learned my addr 172.21.15.32:0/3340807651 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:58:55.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.129+0000 7fdc2259c700 1 --2- 172.21.15.32:0/3340807651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc24102e00 0x7fdc24113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:55.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.129+0000 7fdc21d9b700 1 -- 172.21.15.32:0/3340807651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc24102e00 msgr2=0x7fdc24113b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:55.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.129+0000 7fdc21d9b700 1 --2- 172.21.15.32:0/3340807651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc24102e00 0x7fdc24113b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:55.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.129+0000 7fdc21d9b700 1 -- 172.21.15.32:0/3340807651 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc18005040 con 0x7fdc241037a0 2024-04-12T05:58:55.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.129+0000 7fdc21d9b700 1 --2- 172.21.15.32:0/3340807651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc241037a0 0x7fdc24114040 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fdc18009390 tx=0x7fdc18009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:58:55.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.129+0000 7fdc12ffd700 1 -- 172.21.15.32:0/3340807651 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc1800e040 con 0x7fdc241037a0 2024-04-12T05:58:55.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.129+0000 7fdc12ffd700 1 -- 172.21.15.32:0/3340807651 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc18013420 con 0x7fdc241037a0 2024-04-12T05:58:55.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.130+0000 7fdc2887c700 1 -- 172.21.15.32:0/3340807651 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc2410dd00 con 0x7fdc241037a0 2024-04-12T05:58:55.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.130+0000 7fdc12ffd700 1 -- 172.21.15.32:0/3340807651 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc1801c630 con 0x7fdc241037a0 2024-04-12T05:58:55.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.130+0000 7fdc2887c700 1 -- 172.21.15.32:0/3340807651 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc2410e2c0 con 0x7fdc241037a0 2024-04-12T05:58:55.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.131+0000 7fdc2887c700 1 -- 172.21.15.32:0/3340807651 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc240fe550 con 0x7fdc241037a0 2024-04-12T05:58:55.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.131+0000 7fdc12ffd700 1 -- 172.21.15.32:0/3340807651 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fdc180139e0 con 0x7fdc241037a0 2024-04-12T05:58:55.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.132+0000 7fdc12ffd700 1 --2- 172.21.15.32:0/3340807651 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fdc0c06cc30 0x7fdc0c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:55.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.132+0000 7fdc12ffd700 1 -- 172.21.15.32:0/3340807651 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fdc1808d260 con 0x7fdc241037a0 2024-04-12T05:58:55.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.132+0000 7fdc2259c700 1 --2- 172.21.15.32:0/3340807651 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fdc0c06cc30 0x7fdc0c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:55.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.132+0000 7fdc2259c700 1 --2- 172.21.15.32:0/3340807651 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fdc0c06cc30 0x7fdc0c06f0e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fdc14004990 tx=0x7fdc14009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:58:55.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.136+0000 7fdc12ffd700 1 -- 172.21.15.32:0/3340807651 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fdc1805ad70 con 0x7fdc241037a0 2024-04-12T05:58:55.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.346+0000 7fdc2887c700 1 -- 172.21.15.32:0/3340807651 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc241137d0 con 0x7fdc0c06cc30 2024-04-12T05:58:55.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.356+0000 7fdc12ffd700 1 -- 172.21.15.32:0/3340807651 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fdc241137d0 con 0x7fdc0c06cc30 2024-04-12T05:58:55.351 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:58:55.351 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (18m) 12m ago 30m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:58:55.352 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (16m) 12m ago 30m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:58:55.352 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (16m) 11m ago 29m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:58:55.352 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (18m) 12m ago 29m 87.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:58:55.352 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (12m) 12m ago 24m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T05:58:55.352 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (12m) 12m ago 24m 13.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T05:58:55.352 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (11m) 11m ago 24m 11.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 6d4aca6f74a0 2024-04-12T05:58:55.352 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (11m) 11m ago 24m 14.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T05:58:55.352 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (20m) 12m ago 31m 507M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:58:55.352 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (19m) 11m ago 29m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:58:55.352 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (17m) 12m ago 31m 53.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:58:55.352 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (16m) 11m ago 29m 43.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:58:55.352 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (19m) 12m ago 30m 16.9M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:58:55.352 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (19m) 11m ago 29m 18.4M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:58:55.352 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (15m) 12m ago 28m 91.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T05:58:55.352 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (15m) 12m ago 27m 97.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T05:58:55.353 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (14m) 12m ago 27m 81.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T05:58:55.353 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (14m) 11m ago 26m 113M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T05:58:55.353 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (13m) 11m ago 26m 90.3M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T05:58:55.353 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (12m) 11m ago 26m 84.8M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T05:58:55.353 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (19m) 12m ago 29m 64.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:58:55.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.358+0000 7fdc2887c700 1 -- 172.21.15.32:0/3340807651 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fdc0c06cc30 msgr2=0x7fdc0c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:55.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.358+0000 7fdc2887c700 1 --2- 172.21.15.32:0/3340807651 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fdc0c06cc30 0x7fdc0c06f0e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fdc14004990 tx=0x7fdc14009ee0 comp rx=0 tx=0).stop 2024-04-12T05:58:55.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.358+0000 7fdc2887c700 1 -- 172.21.15.32:0/3340807651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc241037a0 msgr2=0x7fdc24114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:55.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.358+0000 7fdc2887c700 1 --2- 172.21.15.32:0/3340807651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc241037a0 0x7fdc24114040 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fdc18009390 tx=0x7fdc18009e30 comp rx=0 tx=0).stop 2024-04-12T05:58:55.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.358+0000 7fdc2887c700 1 -- 172.21.15.32:0/3340807651 shutdown_connections 2024-04-12T05:58:55.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.358+0000 7fdc2887c700 1 --2- 172.21.15.32:0/3340807651 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fdc0c06cc30 0x7fdc0c06f0e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:55.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.358+0000 7fdc2887c700 1 --2- 172.21.15.32:0/3340807651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc24102e00 0x7fdc24113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:55.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.358+0000 7fdc2887c700 1 --2- 172.21.15.32:0/3340807651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc241037a0 0x7fdc24114040 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:55.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.358+0000 7fdc2887c700 1 -- 172.21.15.32:0/3340807651 >> 172.21.15.32:0/3340807651 conn(0x7fdc240f89b0 msgr2=0x7fdc240f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:58:55.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.359+0000 7fdc2887c700 1 -- 172.21.15.32:0/3340807651 shutdown_connections 2024-04-12T05:58:55.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.359+0000 7fdc2887c700 1 -- 172.21.15.32:0/3340807651 wait complete. 2024-04-12T05:58:55.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.462+0000 7ff1f058e700 1 -- 172.21.15.32:0/673584414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1e80fe830 msgr2=0x7ff1e8106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:55.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.462+0000 7ff1f058e700 1 --2- 172.21.15.32:0/673584414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1e80fe830 0x7ff1e8106f00 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8004940 tx=0x7ff1d8009e30 comp rx=0 tx=0).stop 2024-04-12T05:58:55.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.462+0000 7ff1f058e700 1 -- 172.21.15.32:0/673584414 shutdown_connections 2024-04-12T05:58:55.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.462+0000 7ff1f058e700 1 --2- 172.21.15.32:0/673584414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1e80fe830 0x7ff1e8106f00 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:55.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.462+0000 7ff1f058e700 1 --2- 172.21.15.32:0/673584414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1e80fde90 0x7ff1e80fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:55.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.462+0000 7ff1f058e700 1 -- 172.21.15.32:0/673584414 >> 172.21.15.32:0/673584414 conn(0x7ff1e8072bc0 msgr2=0x7ff1e8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:58:55.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.462+0000 7ff1f058e700 1 -- 172.21.15.32:0/673584414 shutdown_connections 2024-04-12T05:58:55.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.462+0000 7ff1f058e700 1 -- 172.21.15.32:0/673584414 wait complete. 2024-04-12T05:58:55.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.463+0000 7ff1f058e700 1 Processor -- start 2024-04-12T05:58:55.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.463+0000 7ff1f058e700 1 -- start start 2024-04-12T05:58:55.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.463+0000 7ff1f058e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1e80fde90 0x7ff1e80fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:55.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.463+0000 7ff1f058e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1e80fe830 0x7ff1e80fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:55.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.463+0000 7ff1f058e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e80fc820 con 0x7ff1e80fde90 2024-04-12T05:58:55.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.463+0000 7ff1f058e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e80fc960 con 0x7ff1e80fe830 2024-04-12T05:58:55.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.463+0000 7ff1ee32a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1e80fde90 0x7ff1e80fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:55.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.463+0000 7ff1ee32a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1e80fde90 0x7ff1e80fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34550/0 (socket says 172.21.15.32:34550) 2024-04-12T05:58:55.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.463+0000 7ff1ee32a700 1 -- 172.21.15.32:0/1786583602 learned_addr learned my addr 172.21.15.32:0/1786583602 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:58:55.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.463+0000 7ff1ee32a700 1 -- 172.21.15.32:0/1786583602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1e80fe830 msgr2=0x7ff1e80fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:55.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.463+0000 7ff1ee32a700 1 --2- 172.21.15.32:0/1786583602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1e80fe830 0x7ff1e80fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:55.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.463+0000 7ff1ee32a700 1 -- 172.21.15.32:0/1786583602 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d8005040 con 0x7ff1e80fde90 2024-04-12T05:58:55.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.463+0000 7ff1ee32a700 1 --2- 172.21.15.32:0/1786583602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1e80fde90 0x7ff1e80fbd50 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff1e4002a50 tx=0x7ff1e40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:58:55.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.464+0000 7ff1deffd700 1 -- 172.21.15.32:0/1786583602 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1e4007cc0 con 0x7ff1e80fde90 2024-04-12T05:58:55.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.464+0000 7ff1f058e700 1 -- 172.21.15.32:0/1786583602 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1e8119de0 con 0x7ff1e80fde90 2024-04-12T05:58:55.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.464+0000 7ff1deffd700 1 -- 172.21.15.32:0/1786583602 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff1e400f040 con 0x7ff1e80fde90 2024-04-12T05:58:55.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.464+0000 7ff1deffd700 1 -- 172.21.15.32:0/1786583602 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1e4009540 con 0x7ff1e80fde90 2024-04-12T05:58:55.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.464+0000 7ff1f058e700 1 -- 172.21.15.32:0/1786583602 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1e811a1e0 con 0x7ff1e80fde90 2024-04-12T05:58:55.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.465+0000 7ff1f058e700 1 -- 172.21.15.32:0/1786583602 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1e8065e30 con 0x7ff1e80fde90 2024-04-12T05:58:55.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.465+0000 7ff1deffd700 1 -- 172.21.15.32:0/1786583602 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ff1e40096a0 con 0x7ff1e80fde90 2024-04-12T05:58:55.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.465+0000 7ff1deffd700 1 --2- 172.21.15.32:0/1786583602 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff1d406cc30 0x7ff1d406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:55.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.466+0000 7ff1deffd700 1 -- 172.21.15.32:0/1786583602 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7ff1e408bec0 con 0x7ff1e80fde90 2024-04-12T05:58:55.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.466+0000 7ff1edb29700 1 --2- 172.21.15.32:0/1786583602 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff1d406cc30 0x7ff1d406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:55.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.466+0000 7ff1edb29700 1 --2- 172.21.15.32:0/1786583602 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff1d406cc30 0x7ff1d406f0e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8004f90 tx=0x7ff1d8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:58:55.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.470+0000 7ff1deffd700 1 -- 172.21.15.32:0/1786583602 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff1e4055e20 con 0x7ff1e80fde90 2024-04-12T05:58:55.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.761+0000 7ff1f058e700 1 -- 172.21.15.32:0/1786583602 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff1e805f5f0 con 0x7ff1e80fde90 2024-04-12T05:58:55.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.761+0000 7ff1deffd700 1 -- 172.21.15.32:0/1786583602 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7ff1e4059440 con 0x7ff1e80fde90 2024-04-12T05:58:55.756 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:58:55.757 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:58:55.757 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:58:55.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:58:55.757 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:58:55.757 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:58:55.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:58:55.757 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:58:55.757 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:58:55.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:58:55.757 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:58:55.757 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:58:55.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:58:55.757 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:58:55.757 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:58:55.757 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:58:55.758 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:58:55.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.763+0000 7ff1f058e700 1 -- 172.21.15.32:0/1786583602 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff1d406cc30 msgr2=0x7ff1d406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:55.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.763+0000 7ff1f058e700 1 --2- 172.21.15.32:0/1786583602 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff1d406cc30 0x7ff1d406f0e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8004f90 tx=0x7ff1d8009e30 comp rx=0 tx=0).stop 2024-04-12T05:58:55.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.763+0000 7ff1f058e700 1 -- 172.21.15.32:0/1786583602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1e80fde90 msgr2=0x7ff1e80fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:55.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.763+0000 7ff1f058e700 1 --2- 172.21.15.32:0/1786583602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1e80fde90 0x7ff1e80fbd50 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff1e4002a50 tx=0x7ff1e40046b0 comp rx=0 tx=0).stop 2024-04-12T05:58:55.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.763+0000 7ff1f058e700 1 -- 172.21.15.32:0/1786583602 shutdown_connections 2024-04-12T05:58:55.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.763+0000 7ff1f058e700 1 --2- 172.21.15.32:0/1786583602 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff1d406cc30 0x7ff1d406f0e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:55.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.763+0000 7ff1f058e700 1 --2- 172.21.15.32:0/1786583602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1e80fde90 0x7ff1e80fbd50 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:55.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.763+0000 7ff1f058e700 1 --2- 172.21.15.32:0/1786583602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1e80fe830 0x7ff1e80fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:55.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.763+0000 7ff1f058e700 1 -- 172.21.15.32:0/1786583602 >> 172.21.15.32:0/1786583602 conn(0x7ff1e8072bc0 msgr2=0x7ff1e8103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:58:55.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.764+0000 7ff1f058e700 1 -- 172.21.15.32:0/1786583602 shutdown_connections 2024-04-12T05:58:55.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.764+0000 7ff1f058e700 1 -- 172.21.15.32:0/1786583602 wait complete. 2024-04-12T05:58:55.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:55 smithi067 ceph-mon[112979]: pgmap v552: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:55.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:55 smithi067 ceph-mon[112979]: from='client.35040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:58:55.836 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:55 smithi032 ceph-mon[146817]: pgmap v552: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:55.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:55 smithi032 ceph-mon[146817]: from='client.35040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:58:55.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.867+0000 7f66b4272700 1 -- 172.21.15.32:0/1537340295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66ac0fe830 msgr2=0x7f66ac106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:55.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.867+0000 7f66b4272700 1 --2- 172.21.15.32:0/1537340295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66ac0fe830 0x7f66ac106f00 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f66a8004940 tx=0x7f66a8009e30 comp rx=0 tx=0).stop 2024-04-12T05:58:55.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.867+0000 7f66b4272700 1 -- 172.21.15.32:0/1537340295 shutdown_connections 2024-04-12T05:58:55.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.867+0000 7f66b4272700 1 --2- 172.21.15.32:0/1537340295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66ac0fe830 0x7f66ac106f00 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:55.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.867+0000 7f66b4272700 1 --2- 172.21.15.32:0/1537340295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66ac0fde90 0x7f66ac0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:55.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.867+0000 7f66b4272700 1 -- 172.21.15.32:0/1537340295 >> 172.21.15.32:0/1537340295 conn(0x7f66ac072bc0 msgr2=0x7f66ac072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:58:55.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.867+0000 7f66b4272700 1 -- 172.21.15.32:0/1537340295 shutdown_connections 2024-04-12T05:58:55.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.867+0000 7f66b4272700 1 -- 172.21.15.32:0/1537340295 wait complete. 2024-04-12T05:58:55.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.868+0000 7f66b4272700 1 Processor -- start 2024-04-12T05:58:55.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.868+0000 7f66b4272700 1 -- start start 2024-04-12T05:58:55.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.868+0000 7f66b4272700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66ac0fde90 0x7f66ac115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:55.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.868+0000 7f66b4272700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66ac0fe830 0x7f66ac1161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:55.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.868+0000 7f66b4272700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac116810 con 0x7f66ac0fe830 2024-04-12T05:58:55.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.868+0000 7f66b4272700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac10fc30 con 0x7f66ac0fde90 2024-04-12T05:58:55.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.868+0000 7f66b180d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66ac0fe830 0x7f66ac1161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:55.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.868+0000 7f66b180d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66ac0fe830 0x7f66ac1161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34554/0 (socket says 172.21.15.32:34554) 2024-04-12T05:58:55.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.868+0000 7f66b180d700 1 -- 172.21.15.32:0/3040132014 learned_addr learned my addr 172.21.15.32:0/3040132014 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:58:55.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.868+0000 7f66b200e700 1 --2- 172.21.15.32:0/3040132014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66ac0fde90 0x7f66ac115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:55.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.868+0000 7f66b180d700 1 -- 172.21.15.32:0/3040132014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66ac0fde90 msgr2=0x7f66ac115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:55.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.868+0000 7f66b180d700 1 --2- 172.21.15.32:0/3040132014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66ac0fde90 0x7f66ac115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:55.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.868+0000 7f66b180d700 1 -- 172.21.15.32:0/3040132014 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a8005040 con 0x7f66ac0fe830 2024-04-12T05:58:55.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.868+0000 7f66b180d700 1 --2- 172.21.15.32:0/3040132014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66ac0fe830 0x7f66ac1161c0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009360 tx=0x7f66a8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:58:55.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.869+0000 7f66a2ffd700 1 -- 172.21.15.32:0/3040132014 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66a800e040 con 0x7f66ac0fe830 2024-04-12T05:58:55.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.869+0000 7f66a2ffd700 1 -- 172.21.15.32:0/3040132014 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f66a800bed0 con 0x7f66ac0fe830 2024-04-12T05:58:55.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.869+0000 7f66b4272700 1 -- 172.21.15.32:0/3040132014 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66ac10feb0 con 0x7f66ac0fe830 2024-04-12T05:58:55.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.869+0000 7f66a2ffd700 1 -- 172.21.15.32:0/3040132014 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66a801c3f0 con 0x7f66ac0fe830 2024-04-12T05:58:55.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.869+0000 7f66b4272700 1 -- 172.21.15.32:0/3040132014 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66ac1103f0 con 0x7f66ac0fe830 2024-04-12T05:58:55.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.870+0000 7f66b4272700 1 -- 172.21.15.32:0/3040132014 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66ac106a20 con 0x7f66ac0fe830 2024-04-12T05:58:55.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.870+0000 7f66a2ffd700 1 -- 172.21.15.32:0/3040132014 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f66a8013790 con 0x7f66ac0fe830 2024-04-12T05:58:55.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.871+0000 7f66a2ffd700 1 --2- 172.21.15.32:0/3040132014 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f669806cc20 0x7f669806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:55.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.871+0000 7f66a2ffd700 1 -- 172.21.15.32:0/3040132014 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f66a808d470 con 0x7f66ac0fe830 2024-04-12T05:58:55.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.871+0000 7f66b200e700 1 --2- 172.21.15.32:0/3040132014 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f669806cc20 0x7f669806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:55.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.871+0000 7f66b200e700 1 --2- 172.21.15.32:0/3040132014 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f669806cc20 0x7f669806f0d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f669c009fd0 tx=0x7f669c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:58:55.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:55.875+0000 7f66a2ffd700 1 -- 172.21.15.32:0/3040132014 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f66a80573d0 con 0x7f66ac0fe830 2024-04-12T05:58:56.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.122+0000 7f66b4272700 1 -- 172.21.15.32:0/3040132014 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f66ac1112d0 con 0x7f66ac0fe830 2024-04-12T05:58:56.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.122+0000 7f66a2ffd700 1 -- 172.21.15.32:0/3040132014 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1835 (secure 0 0 0) 0x7f66a8013420 con 0x7f66ac0fe830 2024-04-12T05:58:56.117 INFO:teuthology.orchestra.run.smithi032.stdout:e33 2024-04-12T05:58:56.117 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:58:56.117 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:58:56.118 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:58:56.118 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:58:56.118 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:58:56.118 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:58:56.118 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 24 2024-04-12T05:58:56.118 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:58:56.118 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:58:56.118 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:46:10.668569+0000 2024-04-12T05:58:56.118 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:58:56.118 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:58:56.118 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:58:56.118 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:58:56.118 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:58:56.118 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:58:56.119 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:58:56.119 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:58:56.119 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T05:58:56.119 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:58:56.119 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:58:56.119 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:58:56.119 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T05:58:56.119 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-12T05:58:56.119 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:58:56.119 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:58:56.119 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:58:56.119 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:58:56.119 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:58:56.119 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:58:56.120 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:58:56.120 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:58:56.120 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:58:56.120 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:58:56.120 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-12T05:58:56.120 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:58:56.120 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:58:56.120 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:58:56.120 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{-1:44317} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:58:56.120 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:58:56.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.124+0000 7f66b4272700 1 -- 172.21.15.32:0/3040132014 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f669806cc20 msgr2=0x7f669806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:56.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.124+0000 7f66b4272700 1 --2- 172.21.15.32:0/3040132014 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f669806cc20 0x7f669806f0d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f669c009fd0 tx=0x7f669c00b040 comp rx=0 tx=0).stop 2024-04-12T05:58:56.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.124+0000 7f66b4272700 1 -- 172.21.15.32:0/3040132014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66ac0fe830 msgr2=0x7f66ac1161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:56.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.124+0000 7f66b4272700 1 --2- 172.21.15.32:0/3040132014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66ac0fe830 0x7f66ac1161c0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009360 tx=0x7f66a8009e30 comp rx=0 tx=0).stop 2024-04-12T05:58:56.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.124+0000 7f66b4272700 1 -- 172.21.15.32:0/3040132014 shutdown_connections 2024-04-12T05:58:56.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.124+0000 7f66b4272700 1 --2- 172.21.15.32:0/3040132014 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f669806cc20 0x7f669806f0d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:56.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.124+0000 7f66b4272700 1 --2- 172.21.15.32:0/3040132014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66ac0fde90 0x7f66ac115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:56.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.124+0000 7f66b4272700 1 --2- 172.21.15.32:0/3040132014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66ac0fe830 0x7f66ac1161c0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:56.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.124+0000 7f66b4272700 1 -- 172.21.15.32:0/3040132014 >> 172.21.15.32:0/3040132014 conn(0x7f66ac072bc0 msgr2=0x7f66ac0fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:58:56.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.124+0000 7f66b4272700 1 -- 172.21.15.32:0/3040132014 shutdown_connections 2024-04-12T05:58:56.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.124+0000 7f66b4272700 1 -- 172.21.15.32:0/3040132014 wait complete. 2024-04-12T05:58:56.122 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-12T05:58:56.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.229+0000 7feea51bb700 1 -- 172.21.15.32:0/567966961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feea0102710 msgr2=0x7feea00fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:56.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.229+0000 7feea51bb700 1 --2- 172.21.15.32:0/567966961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feea0102710 0x7feea00fbd80 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fee88004940 tx=0x7fee88009e30 comp rx=0 tx=0).stop 2024-04-12T05:58:56.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.229+0000 7feea51bb700 1 -- 172.21.15.32:0/567966961 shutdown_connections 2024-04-12T05:58:56.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.229+0000 7feea51bb700 1 --2- 172.21.15.32:0/567966961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feea0102710 0x7feea00fbd80 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:56.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.229+0000 7feea51bb700 1 --2- 172.21.15.32:0/567966961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feea0101d70 0x7feea0102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:56.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.229+0000 7feea51bb700 1 -- 172.21.15.32:0/567966961 >> 172.21.15.32:0/567966961 conn(0x7feea0072bc0 msgr2=0x7feea0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:58:56.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.229+0000 7feea51bb700 1 -- 172.21.15.32:0/567966961 shutdown_connections 2024-04-12T05:58:56.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.229+0000 7feea51bb700 1 -- 172.21.15.32:0/567966961 wait complete. 2024-04-12T05:58:56.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.230+0000 7feea51bb700 1 Processor -- start 2024-04-12T05:58:56.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.230+0000 7feea51bb700 1 -- start start 2024-04-12T05:58:56.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.230+0000 7feea51bb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feea0101d70 0x7feea010eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:56.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.230+0000 7feea51bb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feea0102710 0x7feea010f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:56.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.230+0000 7feea51bb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea010f5c0 con 0x7feea0102710 2024-04-12T05:58:56.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.230+0000 7feea51bb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea010f700 con 0x7feea0101d70 2024-04-12T05:58:56.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.230+0000 7fee9e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feea0102710 0x7feea010f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:56.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.230+0000 7fee9e59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feea0102710 0x7feea010f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34574/0 (socket says 172.21.15.32:34574) 2024-04-12T05:58:56.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.230+0000 7fee9e59c700 1 -- 172.21.15.32:0/2782543759 learned_addr learned my addr 172.21.15.32:0/2782543759 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:58:56.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.231+0000 7fee9ed9d700 1 --2- 172.21.15.32:0/2782543759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feea0101d70 0x7feea010eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:56.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.231+0000 7fee9e59c700 1 -- 172.21.15.32:0/2782543759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feea0101d70 msgr2=0x7feea010eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:56.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.231+0000 7fee9e59c700 1 --2- 172.21.15.32:0/2782543759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feea0101d70 0x7feea010eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:56.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.231+0000 7fee9e59c700 1 -- 172.21.15.32:0/2782543759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee88005040 con 0x7feea0102710 2024-04-12T05:58:56.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.231+0000 7fee9e59c700 1 --2- 172.21.15.32:0/2782543759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feea0102710 0x7feea010f080 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fee88000c00 tx=0x7fee88009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:58:56.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.231+0000 7fee977fe700 1 -- 172.21.15.32:0/2782543759 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee8800e070 con 0x7feea0102710 2024-04-12T05:58:56.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.231+0000 7fee977fe700 1 -- 172.21.15.32:0/2782543759 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fee88013de0 con 0x7feea0102710 2024-04-12T05:58:56.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.231+0000 7feea51bb700 1 -- 172.21.15.32:0/2782543759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feea01179f0 con 0x7feea0102710 2024-04-12T05:58:56.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.231+0000 7fee977fe700 1 -- 172.21.15.32:0/2782543759 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee8801cde0 con 0x7feea0102710 2024-04-12T05:58:56.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.231+0000 7feea51bb700 1 -- 172.21.15.32:0/2782543759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feea0117f20 con 0x7feea0102710 2024-04-12T05:58:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.232+0000 7feea51bb700 1 -- 172.21.15.32:0/2782543759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feea00ff080 con 0x7feea0102710 2024-04-12T05:58:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.233+0000 7fee977fe700 1 -- 172.21.15.32:0/2782543759 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fee8801c600 con 0x7feea0102710 2024-04-12T05:58:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.233+0000 7fee977fe700 1 --2- 172.21.15.32:0/2782543759 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fee8c06cc20 0x7fee8c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.233+0000 7fee977fe700 1 -- 172.21.15.32:0/2782543759 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fee8808d580 con 0x7feea0102710 2024-04-12T05:58:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.233+0000 7fee9ed9d700 1 --2- 172.21.15.32:0/2782543759 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fee8c06cc20 0x7fee8c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.233+0000 7fee9ed9d700 1 --2- 172.21.15.32:0/2782543759 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fee8c06cc20 0x7fee8c06f0d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fee90004600 tx=0x7fee9000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:58:56.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.238+0000 7fee977fe700 1 -- 172.21.15.32:0/2782543759 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fee88057560 con 0x7feea0102710 2024-04-12T05:58:56.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.460+0000 7feea51bb700 1 -- 172.21.15.32:0/2782543759 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7feea0065e70 con 0x7fee8c06cc20 2024-04-12T05:58:56.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.461+0000 7fee977fe700 1 -- 172.21.15.32:0/2782543759 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7feea0065e70 con 0x7fee8c06cc20 2024-04-12T05:58:56.457 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:58:56.457 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:58:56.457 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:58:56.457 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:58:56.457 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:58:56.457 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-12T05:58:56.457 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:58:56.457 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-12T05:58:56.457 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:58:56.457 INFO:teuthology.orchestra.run.smithi032.stdout: "mds" 2024-04-12T05:58:56.457 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:58:56.457 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:58:56.457 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:58:56.458 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:58:56.458 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:58:56.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.463+0000 7feea51bb700 1 -- 172.21.15.32:0/2782543759 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fee8c06cc20 msgr2=0x7fee8c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:56.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.463+0000 7feea51bb700 1 --2- 172.21.15.32:0/2782543759 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fee8c06cc20 0x7fee8c06f0d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fee90004600 tx=0x7fee9000b040 comp rx=0 tx=0).stop 2024-04-12T05:58:56.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.463+0000 7feea51bb700 1 -- 172.21.15.32:0/2782543759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feea0102710 msgr2=0x7feea010f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:56.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.463+0000 7feea51bb700 1 --2- 172.21.15.32:0/2782543759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feea0102710 0x7feea010f080 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fee88000c00 tx=0x7fee88009e30 comp rx=0 tx=0).stop 2024-04-12T05:58:56.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.463+0000 7feea51bb700 1 -- 172.21.15.32:0/2782543759 shutdown_connections 2024-04-12T05:58:56.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.463+0000 7feea51bb700 1 --2- 172.21.15.32:0/2782543759 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fee8c06cc20 0x7fee8c06f0d0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:56.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.463+0000 7feea51bb700 1 --2- 172.21.15.32:0/2782543759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feea0101d70 0x7feea010eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:56.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.463+0000 7feea51bb700 1 --2- 172.21.15.32:0/2782543759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feea0102710 0x7feea010f080 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:56.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.463+0000 7feea51bb700 1 -- 172.21.15.32:0/2782543759 >> 172.21.15.32:0/2782543759 conn(0x7feea0072bc0 msgr2=0x7feea00faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:58:56.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.463+0000 7feea51bb700 1 -- 172.21.15.32:0/2782543759 shutdown_connections 2024-04-12T05:58:56.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.464+0000 7feea51bb700 1 -- 172.21.15.32:0/2782543759 wait complete. 2024-04-12T05:58:56.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.567+0000 7f6df1b9b700 1 -- 172.21.15.32:0/3441886848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6dec0fde90 msgr2=0x7f6dec0fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:56.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.567+0000 7f6df1b9b700 1 --2- 172.21.15.32:0/3441886848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6dec0fde90 0x7f6dec0fe260 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f6de0004940 tx=0x7f6de0009e30 comp rx=0 tx=0).stop 2024-04-12T05:58:56.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.567+0000 7f6df1b9b700 1 -- 172.21.15.32:0/3441886848 shutdown_connections 2024-04-12T05:58:56.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.567+0000 7f6df1b9b700 1 --2- 172.21.15.32:0/3441886848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dec0fe830 0x7f6dec106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:56.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.567+0000 7f6df1b9b700 1 --2- 172.21.15.32:0/3441886848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6dec0fde90 0x7f6dec0fe260 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:56.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.567+0000 7f6df1b9b700 1 -- 172.21.15.32:0/3441886848 >> 172.21.15.32:0/3441886848 conn(0x7f6dec072bc0 msgr2=0x7f6dec072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:58:56.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.567+0000 7f6df1b9b700 1 -- 172.21.15.32:0/3441886848 shutdown_connections 2024-04-12T05:58:56.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.567+0000 7f6df1b9b700 1 -- 172.21.15.32:0/3441886848 wait complete. 2024-04-12T05:58:56.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.567+0000 7f6df1b9b700 1 Processor -- start 2024-04-12T05:58:56.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.568+0000 7f6df1b9b700 1 -- start start 2024-04-12T05:58:56.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.568+0000 7f6df1b9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dec0fde90 0x7f6dec113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:56.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.568+0000 7f6df1b9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6dec0fe830 0x7f6dec114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:56.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.568+0000 7f6df1b9b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec114680 con 0x7f6dec0fe830 2024-04-12T05:58:56.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.568+0000 7f6df1b9b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec10da70 con 0x7f6dec0fde90 2024-04-12T05:58:56.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.568+0000 7f6deaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6dec0fe830 0x7f6dec114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:56.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.568+0000 7f6deaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6dec0fe830 0x7f6dec114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34602/0 (socket says 172.21.15.32:34602) 2024-04-12T05:58:56.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.568+0000 7f6deaffd700 1 -- 172.21.15.32:0/3548077221 learned_addr learned my addr 172.21.15.32:0/3548077221 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:58:56.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.568+0000 7f6deb7fe700 1 --2- 172.21.15.32:0/3548077221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dec0fde90 0x7f6dec113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:56.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.568+0000 7f6deaffd700 1 -- 172.21.15.32:0/3548077221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dec0fde90 msgr2=0x7f6dec113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:56.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.568+0000 7f6deaffd700 1 --2- 172.21.15.32:0/3548077221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dec0fde90 0x7f6dec113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:56.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.568+0000 7f6deaffd700 1 -- 172.21.15.32:0/3548077221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de0005040 con 0x7f6dec0fe830 2024-04-12T05:58:56.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.568+0000 7f6deaffd700 1 --2- 172.21.15.32:0/3548077221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6dec0fe830 0x7f6dec114030 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc002a50 tx=0x7f6ddc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:58:56.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.569+0000 7f6dd3fff700 1 -- 172.21.15.32:0/3548077221 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ddc007cc0 con 0x7f6dec0fe830 2024-04-12T05:58:56.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.569+0000 7f6dd3fff700 1 -- 172.21.15.32:0/3548077221 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ddc00f040 con 0x7f6dec0fe830 2024-04-12T05:58:56.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.569+0000 7f6df1b9b700 1 -- 172.21.15.32:0/3548077221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dec10dd50 con 0x7f6dec0fe830 2024-04-12T05:58:56.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.569+0000 7f6dd3fff700 1 -- 172.21.15.32:0/3548077221 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ddc009540 con 0x7f6dec0fe830 2024-04-12T05:58:56.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.569+0000 7f6df1b9b700 1 -- 172.21.15.32:0/3548077221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dec10e290 con 0x7f6dec0fe830 2024-04-12T05:58:56.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.570+0000 7f6df1b9b700 1 -- 172.21.15.32:0/3548077221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dec1019b0 con 0x7f6dec0fe830 2024-04-12T05:58:56.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.570+0000 7f6dd3fff700 1 -- 172.21.15.32:0/3548077221 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f6ddc0096a0 con 0x7f6dec0fe830 2024-04-12T05:58:56.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.571+0000 7f6dd3fff700 1 --2- 172.21.15.32:0/3548077221 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6dd406cc20 0x7f6dd406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:58:56.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.571+0000 7f6dd3fff700 1 -- 172.21.15.32:0/3548077221 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f6ddc08bed0 con 0x7f6dec0fe830 2024-04-12T05:58:56.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.571+0000 7f6deb7fe700 1 --2- 172.21.15.32:0/3548077221 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6dd406cc20 0x7f6dd406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:58:56.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.571+0000 7f6deb7fe700 1 --2- 172.21.15.32:0/3548077221 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6dd406cc20 0x7f6dd406f0d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6de000b3b0 tx=0x7f6de0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:58:56.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.575+0000 7f6dd3fff700 1 -- 172.21.15.32:0/3548077221 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6ddc0598c0 con 0x7f6dec0fe830 2024-04-12T05:58:56.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:56 smithi067 ceph-mon[112979]: from='client.35044 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:58:56.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:56 smithi067 ceph-mon[112979]: from='client.35048 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:58:56.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:56 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1786583602' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:58:56.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:56 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3040132014' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:58:56.836 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:56 smithi032 ceph-mon[146817]: from='client.35044 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:58:56.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:56 smithi032 ceph-mon[146817]: from='client.35048 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:58:56.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:56 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1786583602' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:58:56.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:56 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3040132014' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:58:56.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.861+0000 7f6df1b9b700 1 -- 172.21.15.32:0/3548077221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6dec10f140 con 0x7f6dec0fe830 2024-04-12T05:58:56.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.862+0000 7f6dd3fff700 1 -- 172.21.15.32:0/3548077221 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f6ddc059450 con 0x7f6dec0fe830 2024-04-12T05:58:56.857 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:58:56.857 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:58:56.857 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-12T05:58:56.857 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:58:56.857 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:58:56.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.864+0000 7f6df1b9b700 1 -- 172.21.15.32:0/3548077221 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6dd406cc20 msgr2=0x7f6dd406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:56.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.864+0000 7f6df1b9b700 1 --2- 172.21.15.32:0/3548077221 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6dd406cc20 0x7f6dd406f0d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6de000b3b0 tx=0x7f6de0009e30 comp rx=0 tx=0).stop 2024-04-12T05:58:56.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.864+0000 7f6df1b9b700 1 -- 172.21.15.32:0/3548077221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6dec0fe830 msgr2=0x7f6dec114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:58:56.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.864+0000 7f6df1b9b700 1 --2- 172.21.15.32:0/3548077221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6dec0fe830 0x7f6dec114030 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc002a50 tx=0x7f6ddc0046b0 comp rx=0 tx=0).stop 2024-04-12T05:58:56.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.864+0000 7f6df1b9b700 1 -- 172.21.15.32:0/3548077221 shutdown_connections 2024-04-12T05:58:56.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.864+0000 7f6df1b9b700 1 --2- 172.21.15.32:0/3548077221 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6dd406cc20 0x7f6dd406f0d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:56.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.864+0000 7f6df1b9b700 1 --2- 172.21.15.32:0/3548077221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dec0fde90 0x7f6dec113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:56.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.864+0000 7f6df1b9b700 1 --2- 172.21.15.32:0/3548077221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6dec0fe830 0x7f6dec114030 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:58:56.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.864+0000 7f6df1b9b700 1 -- 172.21.15.32:0/3548077221 >> 172.21.15.32:0/3548077221 conn(0x7f6dec072bc0 msgr2=0x7f6dec0ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:58:56.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.864+0000 7f6df1b9b700 1 -- 172.21.15.32:0/3548077221 shutdown_connections 2024-04-12T05:58:56.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:58:56.864+0000 7f6df1b9b700 1 -- 172.21.15.32:0/3548077221 wait complete. 2024-04-12T05:58:57.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:57 smithi067 ceph-mon[112979]: pgmap v553: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:57.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:57 smithi067 ceph-mon[112979]: from='client.35060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:58:57.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:57 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3548077221' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:58:57.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:57 smithi032 ceph-mon[146817]: pgmap v553: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:57.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:57 smithi032 ceph-mon[146817]: from='client.35060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:58:57.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:57 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3548077221' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:58:59.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:58:59 smithi067 ceph-mon[112979]: pgmap v554: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:58:59.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:58:59 smithi032 ceph-mon[146817]: pgmap v554: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:01.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:01 smithi067 ceph-mon[112979]: pgmap v555: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:01.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:01 smithi032 ceph-mon[146817]: pgmap v555: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:03.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:03 smithi067 ceph-mon[112979]: pgmap v556: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:03.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:59:03.836 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:03 smithi032 ceph-mon[146817]: pgmap v556: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:03.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:59:05.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:05 smithi067 ceph-mon[112979]: pgmap v557: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:05.836 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:05 smithi032 ceph-mon[146817]: pgmap v557: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:07.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:07 smithi067 ceph-mon[112979]: pgmap v558: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:07.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:07 smithi032 ceph-mon[146817]: pgmap v558: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:09.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:09 smithi067 ceph-mon[112979]: pgmap v559: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:09.837 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:09 smithi032 ceph-mon[146817]: pgmap v559: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:11.765 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:11 smithi067 ceph-mon[112979]: pgmap v560: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:11.836 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:11 smithi032 ceph-mon[146817]: pgmap v560: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:14.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:13 smithi032 ceph-mon[146817]: pgmap v561: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:14.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:13 smithi067 ceph-mon[112979]: pgmap v561: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:16.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:15 smithi032 ceph-mon[146817]: pgmap v562: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:16.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:15 smithi067 ceph-mon[112979]: pgmap v562: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:18.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:17 smithi032 ceph-mon[146817]: pgmap v563: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:18.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:17 smithi067 ceph-mon[112979]: pgmap v563: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:19.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:59:19.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:59:20.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:19 smithi032 ceph-mon[146817]: pgmap v564: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:20.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:19 smithi067 ceph-mon[112979]: pgmap v564: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:22.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:21 smithi032 ceph-mon[146817]: pgmap v565: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:22.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:21 smithi067 ceph-mon[112979]: pgmap v565: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:24.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:23 smithi032 ceph-mon[146817]: pgmap v566: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:24.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:23 smithi067 ceph-mon[112979]: pgmap v566: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:26.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:25 smithi032 ceph-mon[146817]: pgmap v567: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:26.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:25 smithi067 ceph-mon[112979]: pgmap v567: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:26.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.983+0000 7f29e7c95700 1 -- 172.21.15.32:0/1455412207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29e0070c70 msgr2=0x7f29e0109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:26.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.983+0000 7f29e7c95700 1 --2- 172.21.15.32:0/1455412207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29e0070c70 0x7f29e0109110 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f29dc004970 tx=0x7f29dc009e30 comp rx=0 tx=0).stop 2024-04-12T05:59:26.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.983+0000 7f29e7c95700 1 -- 172.21.15.32:0/1455412207 shutdown_connections 2024-04-12T05:59:26.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.983+0000 7f29e7c95700 1 --2- 172.21.15.32:0/1455412207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29e0070c70 0x7f29e0109110 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:26.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.983+0000 7f29e7c95700 1 --2- 172.21.15.32:0/1455412207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29e0070360 0x7f29e0070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:26.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.983+0000 7f29e7c95700 1 -- 172.21.15.32:0/1455412207 >> 172.21.15.32:0/1455412207 conn(0x7f29e00f89b0 msgr2=0x7f29e00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:59:26.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.983+0000 7f29e7c95700 1 -- 172.21.15.32:0/1455412207 shutdown_connections 2024-04-12T05:59:26.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.983+0000 7f29e7c95700 1 -- 172.21.15.32:0/1455412207 wait complete. 2024-04-12T05:59:26.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.984+0000 7f29e7c95700 1 Processor -- start 2024-04-12T05:59:26.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.984+0000 7f29e7c95700 1 -- start start 2024-04-12T05:59:26.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.984+0000 7f29e7c95700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29e0070360 0x7f29e0115a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:59:26.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.984+0000 7f29e7c95700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29e0070c70 0x7f29e0115fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:59:26.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.984+0000 7f29e7c95700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e0116600 con 0x7f29e0070360 2024-04-12T05:59:26.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.984+0000 7f29e7c95700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e010fbe0 con 0x7f29e0070c70 2024-04-12T05:59:26.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.984+0000 7f29e5a31700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29e0070360 0x7f29e0115a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:59:26.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.984+0000 7f29e5a31700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29e0070360 0x7f29e0115a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59826/0 (socket says 172.21.15.32:59826) 2024-04-12T05:59:26.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.984+0000 7f29e5a31700 1 -- 172.21.15.32:0/183843403 learned_addr learned my addr 172.21.15.32:0/183843403 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:59:26.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.984+0000 7f29e5a31700 1 -- 172.21.15.32:0/183843403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29e0070c70 msgr2=0x7f29e0115fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:26.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.985+0000 7f29e5230700 1 --2- 172.21.15.32:0/183843403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29e0070c70 0x7f29e0115fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:59:26.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.985+0000 7f29e5a31700 1 --2- 172.21.15.32:0/183843403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29e0070c70 0x7f29e0115fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:26.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.985+0000 7f29e5a31700 1 -- 172.21.15.32:0/183843403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29dc005040 con 0x7f29e0070360 2024-04-12T05:59:26.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.985+0000 7f29e5a31700 1 --2- 172.21.15.32:0/183843403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29e0070360 0x7f29e0115a70 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f29d0002a50 tx=0x7f29d00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:59:26.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.985+0000 7f29d67fc700 1 -- 172.21.15.32:0/183843403 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29d0007cc0 con 0x7f29e0070360 2024-04-12T05:59:26.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.985+0000 7f29d67fc700 1 -- 172.21.15.32:0/183843403 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f29d000f040 con 0x7f29e0070360 2024-04-12T05:59:26.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.985+0000 7f29e7c95700 1 -- 172.21.15.32:0/183843403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e010fec0 con 0x7f29e0070360 2024-04-12T05:59:26.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.985+0000 7f29d67fc700 1 -- 172.21.15.32:0/183843403 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29d0009540 con 0x7f29e0070360 2024-04-12T05:59:26.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.985+0000 7f29e7c95700 1 -- 172.21.15.32:0/183843403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e0110400 con 0x7f29e0070360 2024-04-12T05:59:26.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.986+0000 7f29e7c95700 1 -- 172.21.15.32:0/183843403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e0071aa0 con 0x7f29e0070360 2024-04-12T05:59:26.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.986+0000 7f29d67fc700 1 -- 172.21.15.32:0/183843403 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f29d00096a0 con 0x7f29e0070360 2024-04-12T05:59:26.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.987+0000 7f29d67fc700 1 --2- 172.21.15.32:0/183843403 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f29cc06cc20 0x7f29cc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:59:26.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.987+0000 7f29d67fc700 1 -- 172.21.15.32:0/183843403 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f29d008bed0 con 0x7f29e0070360 2024-04-12T05:59:26.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.987+0000 7f29e5230700 1 --2- 172.21.15.32:0/183843403 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f29cc06cc20 0x7f29cc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:59:26.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.987+0000 7f29e5230700 1 --2- 172.21.15.32:0/183843403 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f29cc06cc20 0x7f29cc06f0d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f29dc00b370 tx=0x7f29dc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:59:26.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:26.992+0000 7f29d67fc700 1 -- 172.21.15.32:0/183843403 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f29d00598c0 con 0x7f29e0070360 2024-04-12T05:59:27.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.212+0000 7f29e7c95700 1 -- 172.21.15.32:0/183843403 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f29e0111130 con 0x7f29cc06cc20 2024-04-12T05:59:27.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.214+0000 7f29d67fc700 1 -- 172.21.15.32:0/183843403 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f29e0111130 con 0x7f29cc06cc20 2024-04-12T05:59:27.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.216+0000 7f29e7c95700 1 -- 172.21.15.32:0/183843403 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f29cc06cc20 msgr2=0x7f29cc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:27.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.216+0000 7f29e7c95700 1 --2- 172.21.15.32:0/183843403 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f29cc06cc20 0x7f29cc06f0d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f29dc00b370 tx=0x7f29dc009e30 comp rx=0 tx=0).stop 2024-04-12T05:59:27.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.216+0000 7f29e7c95700 1 -- 172.21.15.32:0/183843403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29e0070360 msgr2=0x7f29e0115a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:27.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.216+0000 7f29e7c95700 1 --2- 172.21.15.32:0/183843403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29e0070360 0x7f29e0115a70 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f29d0002a50 tx=0x7f29d00046b0 comp rx=0 tx=0).stop 2024-04-12T05:59:27.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.216+0000 7f29e7c95700 1 -- 172.21.15.32:0/183843403 shutdown_connections 2024-04-12T05:59:27.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.216+0000 7f29e7c95700 1 --2- 172.21.15.32:0/183843403 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f29cc06cc20 0x7f29cc06f0d0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:27.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.216+0000 7f29e7c95700 1 --2- 172.21.15.32:0/183843403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29e0070360 0x7f29e0115a70 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:27.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.216+0000 7f29e7c95700 1 --2- 172.21.15.32:0/183843403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29e0070c70 0x7f29e0115fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:27.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.216+0000 7f29e7c95700 1 -- 172.21.15.32:0/183843403 >> 172.21.15.32:0/183843403 conn(0x7f29e00f89b0 msgr2=0x7f29e0105a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:59:27.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.216+0000 7f29e7c95700 1 -- 172.21.15.32:0/183843403 shutdown_connections 2024-04-12T05:59:27.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.216+0000 7f29e7c95700 1 -- 172.21.15.32:0/183843403 wait complete. 2024-04-12T05:59:27.220 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:59:27.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.320+0000 7f0db5f98700 1 -- 172.21.15.32:0/3185840128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0db0102e70 msgr2=0x7f0db0103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:27.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.320+0000 7f0db5f98700 1 --2- 172.21.15.32:0/3185840128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0db0102e70 0x7f0db0103240 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f0da0004940 tx=0x7f0da0009e30 comp rx=0 tx=0).stop 2024-04-12T05:59:27.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.320+0000 7f0db5f98700 1 -- 172.21.15.32:0/3185840128 shutdown_connections 2024-04-12T05:59:27.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.320+0000 7f0db5f98700 1 --2- 172.21.15.32:0/3185840128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0db00fb4a0 0x7f0db00fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:27.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.320+0000 7f0db5f98700 1 --2- 172.21.15.32:0/3185840128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0db0102e70 0x7f0db0103240 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:27.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.320+0000 7f0db5f98700 1 -- 172.21.15.32:0/3185840128 >> 172.21.15.32:0/3185840128 conn(0x7f0db00714c0 msgr2=0x7f0db00718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:59:27.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.320+0000 7f0db5f98700 1 -- 172.21.15.32:0/3185840128 shutdown_connections 2024-04-12T05:59:27.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.320+0000 7f0db5f98700 1 -- 172.21.15.32:0/3185840128 wait complete. 2024-04-12T05:59:27.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.320+0000 7f0db5f98700 1 Processor -- start 2024-04-12T05:59:27.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.321+0000 7f0db5f98700 1 -- start start 2024-04-12T05:59:27.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.321+0000 7f0db5f98700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0db00fb4a0 0x7f0db0113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:59:27.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.321+0000 7f0db5f98700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0db0102e70 0x7f0db0113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:59:27.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.321+0000 7f0db5f98700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db0114600 con 0x7f0db0102e70 2024-04-12T05:59:27.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.321+0000 7f0db5f98700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db010d9f0 con 0x7f0db00fb4a0 2024-04-12T05:59:27.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.321+0000 7f0daeffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0db0102e70 0x7f0db0113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:59:27.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.321+0000 7f0daeffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0db0102e70 0x7f0db0113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59854/0 (socket says 172.21.15.32:59854) 2024-04-12T05:59:27.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.321+0000 7f0daeffd700 1 -- 172.21.15.32:0/3951118703 learned_addr learned my addr 172.21.15.32:0/3951118703 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:59:27.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.321+0000 7f0daf7fe700 1 --2- 172.21.15.32:0/3951118703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0db00fb4a0 0x7f0db0113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:59:27.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.321+0000 7f0daeffd700 1 -- 172.21.15.32:0/3951118703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0db00fb4a0 msgr2=0x7f0db0113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:27.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.321+0000 7f0daeffd700 1 --2- 172.21.15.32:0/3951118703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0db00fb4a0 0x7f0db0113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:27.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.321+0000 7f0daeffd700 1 -- 172.21.15.32:0/3951118703 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da0005040 con 0x7f0db0102e70 2024-04-12T05:59:27.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.321+0000 7f0daeffd700 1 --2- 172.21.15.32:0/3951118703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0db0102e70 0x7f0db0113fb0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f0da400c830 tx=0x7f0da4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:59:27.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.322+0000 7f0d97fff700 1 -- 172.21.15.32:0/3951118703 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0da40106f0 con 0x7f0db0102e70 2024-04-12T05:59:27.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.322+0000 7f0db5f98700 1 -- 172.21.15.32:0/3951118703 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db010dcd0 con 0x7f0db0102e70 2024-04-12T05:59:27.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.322+0000 7f0d97fff700 1 -- 172.21.15.32:0/3951118703 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0da4010d00 con 0x7f0db0102e70 2024-04-12T05:59:27.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.322+0000 7f0d97fff700 1 -- 172.21.15.32:0/3951118703 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0da4012e90 con 0x7f0db0102e70 2024-04-12T05:59:27.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.322+0000 7f0db5f98700 1 -- 172.21.15.32:0/3951118703 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db010e290 con 0x7f0db0102e70 2024-04-12T05:59:27.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.323+0000 7f0db5f98700 1 -- 172.21.15.32:0/3951118703 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db0100110 con 0x7f0db0102e70 2024-04-12T05:59:27.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.323+0000 7f0d97fff700 1 -- 172.21.15.32:0/3951118703 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f0da4010850 con 0x7f0db0102e70 2024-04-12T05:59:27.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.324+0000 7f0d97fff700 1 --2- 172.21.15.32:0/3951118703 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0d9806cc20 0x7f0d9806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:59:27.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.324+0000 7f0daf7fe700 1 --2- 172.21.15.32:0/3951118703 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0d9806cc20 0x7f0d9806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:59:27.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.324+0000 7f0d97fff700 1 -- 172.21.15.32:0/3951118703 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f0da408c070 con 0x7f0db0102e70 2024-04-12T05:59:27.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.324+0000 7f0daf7fe700 1 --2- 172.21.15.32:0/3951118703 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0d9806cc20 0x7f0d9806f0d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0da0004f90 tx=0x7f0da0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:59:27.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.328+0000 7f0d97fff700 1 -- 172.21.15.32:0/3951118703 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0da4059ae0 con 0x7f0db0102e70 2024-04-12T05:59:27.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.550+0000 7f0db5f98700 1 -- 172.21.15.32:0/3951118703 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0db0113740 con 0x7f0d9806cc20 2024-04-12T05:59:27.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.551+0000 7f0d97fff700 1 -- 172.21.15.32:0/3951118703 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0db0113740 con 0x7f0d9806cc20 2024-04-12T05:59:27.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.553+0000 7f0db5f98700 1 -- 172.21.15.32:0/3951118703 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0d9806cc20 msgr2=0x7f0d9806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:27.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.553+0000 7f0db5f98700 1 --2- 172.21.15.32:0/3951118703 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0d9806cc20 0x7f0d9806f0d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0da0004f90 tx=0x7f0da0009e30 comp rx=0 tx=0).stop 2024-04-12T05:59:27.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.553+0000 7f0db5f98700 1 -- 172.21.15.32:0/3951118703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0db0102e70 msgr2=0x7f0db0113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:27.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.553+0000 7f0db5f98700 1 --2- 172.21.15.32:0/3951118703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0db0102e70 0x7f0db0113fb0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f0da400c830 tx=0x7f0da4004470 comp rx=0 tx=0).stop 2024-04-12T05:59:27.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.553+0000 7f0db5f98700 1 -- 172.21.15.32:0/3951118703 shutdown_connections 2024-04-12T05:59:27.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.553+0000 7f0db5f98700 1 --2- 172.21.15.32:0/3951118703 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0d9806cc20 0x7f0d9806f0d0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:27.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.553+0000 7f0db5f98700 1 --2- 172.21.15.32:0/3951118703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0db00fb4a0 0x7f0db0113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:27.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.553+0000 7f0db5f98700 1 --2- 172.21.15.32:0/3951118703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0db0102e70 0x7f0db0113fb0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:27.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.553+0000 7f0db5f98700 1 -- 172.21.15.32:0/3951118703 >> 172.21.15.32:0/3951118703 conn(0x7f0db00714c0 msgr2=0x7f0db00fa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:59:27.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.554+0000 7f0db5f98700 1 -- 172.21.15.32:0/3951118703 shutdown_connections 2024-04-12T05:59:27.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.554+0000 7f0db5f98700 1 -- 172.21.15.32:0/3951118703 wait complete. 2024-04-12T05:59:27.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.656+0000 7ff2c391f700 1 -- 172.21.15.32:0/3841742478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2bc0fe830 msgr2=0x7ff2bc106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:27.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.656+0000 7ff2c391f700 1 --2- 172.21.15.32:0/3841742478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2bc0fe830 0x7ff2bc106f00 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7ff2b0004940 tx=0x7ff2b0009e30 comp rx=0 tx=0).stop 2024-04-12T05:59:27.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.656+0000 7ff2c391f700 1 -- 172.21.15.32:0/3841742478 shutdown_connections 2024-04-12T05:59:27.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.656+0000 7ff2c391f700 1 --2- 172.21.15.32:0/3841742478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2bc0fe830 0x7ff2bc106f00 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:27.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.656+0000 7ff2c391f700 1 --2- 172.21.15.32:0/3841742478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2bc0fde90 0x7ff2bc0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:27.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.656+0000 7ff2c391f700 1 -- 172.21.15.32:0/3841742478 >> 172.21.15.32:0/3841742478 conn(0x7ff2bc072bc0 msgr2=0x7ff2bc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:59:27.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.656+0000 7ff2c391f700 1 -- 172.21.15.32:0/3841742478 shutdown_connections 2024-04-12T05:59:27.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.656+0000 7ff2c391f700 1 -- 172.21.15.32:0/3841742478 wait complete. 2024-04-12T05:59:27.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.657+0000 7ff2c391f700 1 Processor -- start 2024-04-12T05:59:27.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.657+0000 7ff2c391f700 1 -- start start 2024-04-12T05:59:27.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.657+0000 7ff2c391f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2bc0fde90 0x7ff2bc113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:59:27.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.657+0000 7ff2c391f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2bc0fe830 0x7ff2bc114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:59:27.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.657+0000 7ff2c391f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2bc114670 con 0x7ff2bc0fde90 2024-04-12T05:59:27.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.657+0000 7ff2c391f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2bc10da60 con 0x7ff2bc0fe830 2024-04-12T05:59:27.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.657+0000 7ff2c16bb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2bc0fde90 0x7ff2bc113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:59:27.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.657+0000 7ff2c16bb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2bc0fde90 0x7ff2bc113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59878/0 (socket says 172.21.15.32:59878) 2024-04-12T05:59:27.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.657+0000 7ff2c16bb700 1 -- 172.21.15.32:0/610358758 learned_addr learned my addr 172.21.15.32:0/610358758 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:59:27.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.658+0000 7ff2c0eba700 1 --2- 172.21.15.32:0/610358758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2bc0fe830 0x7ff2bc114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:59:27.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.658+0000 7ff2c16bb700 1 -- 172.21.15.32:0/610358758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2bc0fe830 msgr2=0x7ff2bc114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:27.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.658+0000 7ff2c16bb700 1 --2- 172.21.15.32:0/610358758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2bc0fe830 0x7ff2bc114020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:27.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.658+0000 7ff2c16bb700 1 -- 172.21.15.32:0/610358758 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b0005040 con 0x7ff2bc0fde90 2024-04-12T05:59:27.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.658+0000 7ff2c0eba700 1 --2- 172.21.15.32:0/610358758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2bc0fe830 0x7ff2bc114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:59:27.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.658+0000 7ff2c16bb700 1 --2- 172.21.15.32:0/610358758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2bc0fde90 0x7ff2bc113ae0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac002a50 tx=0x7ff2ac0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:59:27.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.658+0000 7ff2b9ffb700 1 -- 172.21.15.32:0/610358758 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2ac007cc0 con 0x7ff2bc0fde90 2024-04-12T05:59:27.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.658+0000 7ff2b9ffb700 1 -- 172.21.15.32:0/610358758 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff2ac00f040 con 0x7ff2bc0fde90 2024-04-12T05:59:27.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.658+0000 7ff2c391f700 1 -- 172.21.15.32:0/610358758 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2bc10dd40 con 0x7ff2bc0fde90 2024-04-12T05:59:27.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.658+0000 7ff2b9ffb700 1 -- 172.21.15.32:0/610358758 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2ac009540 con 0x7ff2bc0fde90 2024-04-12T05:59:27.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.658+0000 7ff2c391f700 1 -- 172.21.15.32:0/610358758 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2bc10e280 con 0x7ff2bc0fde90 2024-04-12T05:59:27.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.659+0000 7ff2c391f700 1 -- 172.21.15.32:0/610358758 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2bc065e30 con 0x7ff2bc0fde90 2024-04-12T05:59:27.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.659+0000 7ff2b9ffb700 1 -- 172.21.15.32:0/610358758 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ff2ac0096a0 con 0x7ff2bc0fde90 2024-04-12T05:59:27.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.660+0000 7ff2b9ffb700 1 --2- 172.21.15.32:0/610358758 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff2a406cc20 0x7ff2a406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:59:27.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.660+0000 7ff2b9ffb700 1 -- 172.21.15.32:0/610358758 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7ff2ac08be30 con 0x7ff2bc0fde90 2024-04-12T05:59:27.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.660+0000 7ff2c0eba700 1 --2- 172.21.15.32:0/610358758 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff2a406cc20 0x7ff2a406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:59:27.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.660+0000 7ff2c0eba700 1 --2- 172.21.15.32:0/610358758 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff2a406cc20 0x7ff2a406f0d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff2b000b3b0 tx=0x7ff2b0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:59:27.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.664+0000 7ff2b9ffb700 1 -- 172.21.15.32:0/610358758 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff2ac059840 con 0x7ff2bc0fde90 2024-04-12T05:59:27.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.883+0000 7ff2c391f700 1 -- 172.21.15.32:0/610358758 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff2bc109dd0 con 0x7ff2a406cc20 2024-04-12T05:59:27.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.893+0000 7ff2b9ffb700 1 -- 172.21.15.32:0/610358758 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7ff2bc109dd0 con 0x7ff2a406cc20 2024-04-12T05:59:27.888 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:59:27.888 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (19m) 12m ago 30m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T05:59:27.888 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (16m) 12m ago 30m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T05:59:27.888 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (16m) 12m ago 29m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T05:59:27.888 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (18m) 12m ago 30m 87.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T05:59:27.888 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (13m) 12m ago 25m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T05:59:27.888 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (12m) 12m ago 24m 13.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T05:59:27.888 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (12m) 12m ago 24m 11.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 6d4aca6f74a0 2024-04-12T05:59:27.888 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (12m) 12m ago 24m 14.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T05:59:27.888 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (20m) 12m ago 32m 507M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T05:59:27.888 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (20m) 12m ago 29m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T05:59:27.888 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (17m) 12m ago 32m 53.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T05:59:27.888 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (17m) 12m ago 29m 43.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T05:59:27.888 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (19m) 12m ago 30m 16.9M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T05:59:27.889 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (19m) 12m ago 29m 18.4M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T05:59:27.889 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (16m) 12m ago 28m 91.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T05:59:27.889 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (15m) 12m ago 28m 97.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T05:59:27.889 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (15m) 12m ago 27m 81.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T05:59:27.889 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (14m) 12m ago 27m 113M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T05:59:27.889 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (14m) 12m ago 27m 90.3M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T05:59:27.889 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (13m) 12m ago 26m 84.8M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T05:59:27.889 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (19m) 12m ago 30m 64.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T05:59:27.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.896+0000 7ff2c391f700 1 -- 172.21.15.32:0/610358758 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff2a406cc20 msgr2=0x7ff2a406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:27.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.896+0000 7ff2c391f700 1 --2- 172.21.15.32:0/610358758 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff2a406cc20 0x7ff2a406f0d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff2b000b3b0 tx=0x7ff2b0009e30 comp rx=0 tx=0).stop 2024-04-12T05:59:27.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.896+0000 7ff2c391f700 1 -- 172.21.15.32:0/610358758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2bc0fde90 msgr2=0x7ff2bc113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:27.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.896+0000 7ff2c391f700 1 --2- 172.21.15.32:0/610358758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2bc0fde90 0x7ff2bc113ae0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac002a50 tx=0x7ff2ac0046b0 comp rx=0 tx=0).stop 2024-04-12T05:59:27.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.896+0000 7ff2c391f700 1 -- 172.21.15.32:0/610358758 shutdown_connections 2024-04-12T05:59:27.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.896+0000 7ff2c391f700 1 --2- 172.21.15.32:0/610358758 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff2a406cc20 0x7ff2a406f0d0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:27.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.896+0000 7ff2c391f700 1 --2- 172.21.15.32:0/610358758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2bc0fde90 0x7ff2bc113ae0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:27.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.896+0000 7ff2c391f700 1 --2- 172.21.15.32:0/610358758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2bc0fe830 0x7ff2bc114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:27.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.896+0000 7ff2c391f700 1 -- 172.21.15.32:0/610358758 >> 172.21.15.32:0/610358758 conn(0x7ff2bc072bc0 msgr2=0x7ff2bc103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:59:27.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.896+0000 7ff2c391f700 1 -- 172.21.15.32:0/610358758 shutdown_connections 2024-04-12T05:59:27.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.896+0000 7ff2c391f700 1 -- 172.21.15.32:0/610358758 wait complete. 2024-04-12T05:59:27.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.997+0000 7f627dd5b700 1 -- 172.21.15.32:0/2733683023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62780fbc80 msgr2=0x7f62780fc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:27.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.997+0000 7f627dd5b700 1 --2- 172.21.15.32:0/2733683023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62780fbc80 0x7f62780fc050 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f6268004940 tx=0x7f6268009e30 comp rx=0 tx=0).stop 2024-04-12T05:59:27.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.997+0000 7f627dd5b700 1 -- 172.21.15.32:0/2733683023 shutdown_connections 2024-04-12T05:59:27.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.997+0000 7f627dd5b700 1 --2- 172.21.15.32:0/2733683023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62780fc590 0x7f62781092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:27.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.997+0000 7f627dd5b700 1 --2- 172.21.15.32:0/2733683023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62780fbc80 0x7f62780fc050 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:27.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.997+0000 7f627dd5b700 1 -- 172.21.15.32:0/2733683023 >> 172.21.15.32:0/2733683023 conn(0x7f6278072bc0 msgr2=0x7f6278072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:59:27.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.997+0000 7f627dd5b700 1 -- 172.21.15.32:0/2733683023 shutdown_connections 2024-04-12T05:59:27.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.997+0000 7f627dd5b700 1 -- 172.21.15.32:0/2733683023 wait complete. 2024-04-12T05:59:27.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.998+0000 7f627dd5b700 1 Processor -- start 2024-04-12T05:59:27.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.998+0000 7f627dd5b700 1 -- start start 2024-04-12T05:59:27.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.998+0000 7f627dd5b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62780fbc80 0x7f6278115d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:59:27.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.998+0000 7f627dd5b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62780fc590 0x7f6278116270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:59:27.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.998+0000 7f627dd5b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62781168c0 con 0x7f62780fbc80 2024-04-12T05:59:27.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.998+0000 7f627dd5b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627810fc90 con 0x7f62780fc590 2024-04-12T05:59:27.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.999+0000 7f62777fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62780fbc80 0x7f6278115d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:59:27.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.999+0000 7f62777fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62780fbc80 0x7f6278115d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59910/0 (socket says 172.21.15.32:59910) 2024-04-12T05:59:27.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.999+0000 7f62777fe700 1 -- 172.21.15.32:0/1706246450 learned_addr learned my addr 172.21.15.32:0/1706246450 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:59:27.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.999+0000 7f6276ffd700 1 --2- 172.21.15.32:0/1706246450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62780fc590 0x7f6278116270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:59:27.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.999+0000 7f62777fe700 1 -- 172.21.15.32:0/1706246450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62780fc590 msgr2=0x7f6278116270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:27.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.999+0000 7f62777fe700 1 --2- 172.21.15.32:0/1706246450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62780fc590 0x7f6278116270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:27.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.999+0000 7f62777fe700 1 -- 172.21.15.32:0/1706246450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6268005040 con 0x7f62780fbc80 2024-04-12T05:59:27.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.999+0000 7f62777fe700 1 --2- 172.21.15.32:0/1706246450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62780fbc80 0x7f6278115d30 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6268009360 tx=0x7f6268009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:59:27.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.999+0000 7f625ffff700 1 -- 172.21.15.32:0/1706246450 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f626800e040 con 0x7f62780fbc80 2024-04-12T05:59:27.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.999+0000 7f625ffff700 1 -- 172.21.15.32:0/1706246450 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6268013420 con 0x7f62780fbc80 2024-04-12T05:59:27.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.999+0000 7f627dd5b700 1 -- 172.21.15.32:0/1706246450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f627810ff10 con 0x7f62780fbc80 2024-04-12T05:59:27.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.999+0000 7f625ffff700 1 -- 172.21.15.32:0/1706246450 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f626801c5f0 con 0x7f62780fbc80 2024-04-12T05:59:27.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:27.999+0000 7f627dd5b700 1 -- 172.21.15.32:0/1706246450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6278110450 con 0x7f62780fbc80 2024-04-12T05:59:27.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.000+0000 7f627dd5b700 1 -- 172.21.15.32:0/1706246450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6278065e30 con 0x7f62780fbc80 2024-04-12T05:59:27.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.001+0000 7f625ffff700 1 -- 172.21.15.32:0/1706246450 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f6268013590 con 0x7f62780fbc80 2024-04-12T05:59:27.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.001+0000 7f625ffff700 1 --2- 172.21.15.32:0/1706246450 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f626006cc30 0x7f626006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:59:27.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.001+0000 7f625ffff700 1 -- 172.21.15.32:0/1706246450 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f626808d330 con 0x7f62780fbc80 2024-04-12T05:59:27.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.001+0000 7f6276ffd700 1 --2- 172.21.15.32:0/1706246450 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f626006cc30 0x7f626006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:59:27.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.002+0000 7f6276ffd700 1 --2- 172.21.15.32:0/1706246450 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f626006cc30 0x7f626006f0e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f626c007fd0 tx=0x7f626c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:59:28.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.006+0000 7f625ffff700 1 -- 172.21.15.32:0/1706246450 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f626805ada0 con 0x7f62780fbc80 2024-04-12T05:59:28.086 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:27 smithi032 ceph-mon[146817]: pgmap v568: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:28.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:27 smithi067 ceph-mon[112979]: pgmap v568: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:28.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.296+0000 7f627dd5b700 1 -- 172.21.15.32:0/1706246450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f627804bed0 con 0x7f62780fbc80 2024-04-12T05:59:28.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.297+0000 7f625ffff700 1 -- 172.21.15.32:0/1706246450 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f626805a930 con 0x7f62780fbc80 2024-04-12T05:59:28.291 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:59:28.291 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T05:59:28.291 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:59:28.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:59:28.291 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T05:59:28.292 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:59:28.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:59:28.292 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T05:59:28.292 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:59:28.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:59:28.292 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T05:59:28.292 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:59:28.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T05:59:28.292 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T05:59:28.292 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:59:28.292 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T05:59:28.293 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:59:28.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.298+0000 7f627dd5b700 1 -- 172.21.15.32:0/1706246450 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f626006cc30 msgr2=0x7f626006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:28.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.299+0000 7f627dd5b700 1 --2- 172.21.15.32:0/1706246450 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f626006cc30 0x7f626006f0e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f626c007fd0 tx=0x7f626c009040 comp rx=0 tx=0).stop 2024-04-12T05:59:28.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.299+0000 7f627dd5b700 1 -- 172.21.15.32:0/1706246450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62780fbc80 msgr2=0x7f6278115d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:28.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.299+0000 7f627dd5b700 1 --2- 172.21.15.32:0/1706246450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62780fbc80 0x7f6278115d30 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6268009360 tx=0x7f6268009e30 comp rx=0 tx=0).stop 2024-04-12T05:59:28.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.299+0000 7f627dd5b700 1 -- 172.21.15.32:0/1706246450 shutdown_connections 2024-04-12T05:59:28.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.299+0000 7f627dd5b700 1 --2- 172.21.15.32:0/1706246450 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f626006cc30 0x7f626006f0e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:28.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.299+0000 7f627dd5b700 1 --2- 172.21.15.32:0/1706246450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62780fbc80 0x7f6278115d30 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:28.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.299+0000 7f627dd5b700 1 --2- 172.21.15.32:0/1706246450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62780fc590 0x7f6278116270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:28.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.299+0000 7f627dd5b700 1 -- 172.21.15.32:0/1706246450 >> 172.21.15.32:0/1706246450 conn(0x7f6278072bc0 msgr2=0x7f6278105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:59:28.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.299+0000 7f627dd5b700 1 -- 172.21.15.32:0/1706246450 shutdown_connections 2024-04-12T05:59:28.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.299+0000 7f627dd5b700 1 -- 172.21.15.32:0/1706246450 wait complete. 2024-04-12T05:59:28.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.402+0000 7f12e021f700 1 -- 172.21.15.32:0/4194201736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12d81026b0 msgr2=0x7f12d80fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:28.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.402+0000 7f12e021f700 1 --2- 172.21.15.32:0/4194201736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12d81026b0 0x7f12d80fbd20 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f12c8004940 tx=0x7f12c8009e30 comp rx=0 tx=0).stop 2024-04-12T05:59:28.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.402+0000 7f12e021f700 1 -- 172.21.15.32:0/4194201736 shutdown_connections 2024-04-12T05:59:28.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.402+0000 7f12e021f700 1 --2- 172.21.15.32:0/4194201736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12d81026b0 0x7f12d80fbd20 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:28.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.402+0000 7f12e021f700 1 --2- 172.21.15.32:0/4194201736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12d8101d10 0x7f12d81020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:28.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.402+0000 7f12e021f700 1 -- 172.21.15.32:0/4194201736 >> 172.21.15.32:0/4194201736 conn(0x7f12d80f7880 msgr2=0x7f12d80f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:59:28.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.402+0000 7f12e021f700 1 -- 172.21.15.32:0/4194201736 shutdown_connections 2024-04-12T05:59:28.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.402+0000 7f12e021f700 1 -- 172.21.15.32:0/4194201736 wait complete. 2024-04-12T05:59:28.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.402+0000 7f12e021f700 1 Processor -- start 2024-04-12T05:59:28.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.402+0000 7f12e021f700 1 -- start start 2024-04-12T05:59:28.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.403+0000 7f12e021f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12d8101d10 0x7f12d810ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:59:28.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.403+0000 7f12e021f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12d81026b0 0x7f12d810f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:59:28.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.403+0000 7f12e021f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d810f8a0 con 0x7f12d8101d10 2024-04-12T05:59:28.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.403+0000 7f12e021f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d810f9e0 con 0x7f12d81026b0 2024-04-12T05:59:28.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.403+0000 7f12ddfbb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12d8101d10 0x7f12d810ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:59:28.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.403+0000 7f12ddfbb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12d8101d10 0x7f12d810ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59932/0 (socket says 172.21.15.32:59932) 2024-04-12T05:59:28.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.403+0000 7f12ddfbb700 1 -- 172.21.15.32:0/3752014410 learned_addr learned my addr 172.21.15.32:0/3752014410 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:59:28.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.403+0000 7f12dd7ba700 1 --2- 172.21.15.32:0/3752014410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12d81026b0 0x7f12d810f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:59:28.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.403+0000 7f12ddfbb700 1 -- 172.21.15.32:0/3752014410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12d81026b0 msgr2=0x7f12d810f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:28.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.403+0000 7f12ddfbb700 1 --2- 172.21.15.32:0/3752014410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12d81026b0 0x7f12d810f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:28.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.403+0000 7f12ddfbb700 1 -- 172.21.15.32:0/3752014410 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c8005040 con 0x7f12d8101d10 2024-04-12T05:59:28.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.403+0000 7f12ddfbb700 1 --2- 172.21.15.32:0/3752014410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12d8101d10 0x7f12d810ed10 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f12d4002a80 tx=0x7f12d40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:59:28.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.403+0000 7f12ce7fc700 1 -- 172.21.15.32:0/3752014410 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12d4007cc0 con 0x7f12d8101d10 2024-04-12T05:59:28.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.403+0000 7f12ce7fc700 1 -- 172.21.15.32:0/3752014410 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f12d400f040 con 0x7f12d8101d10 2024-04-12T05:59:28.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.403+0000 7f12e021f700 1 -- 172.21.15.32:0/3752014410 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12d8117d80 con 0x7f12d8101d10 2024-04-12T05:59:28.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.403+0000 7f12ce7fc700 1 -- 172.21.15.32:0/3752014410 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12d400da60 con 0x7f12d8101d10 2024-04-12T05:59:28.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.404+0000 7f12e021f700 1 -- 172.21.15.32:0/3752014410 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12d81182e0 con 0x7f12d8101d10 2024-04-12T05:59:28.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.404+0000 7f12e021f700 1 -- 172.21.15.32:0/3752014410 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12d80fd460 con 0x7f12d8101d10 2024-04-12T05:59:28.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.405+0000 7f12ce7fc700 1 -- 172.21.15.32:0/3752014410 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f12d400dbc0 con 0x7f12d8101d10 2024-04-12T05:59:28.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.405+0000 7f12ce7fc700 1 --2- 172.21.15.32:0/3752014410 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f12c406cc30 0x7f12c406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:59:28.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.406+0000 7f12ce7fc700 1 -- 172.21.15.32:0/3752014410 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f12d408bdf0 con 0x7f12d8101d10 2024-04-12T05:59:28.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.406+0000 7f12dd7ba700 1 --2- 172.21.15.32:0/3752014410 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f12c406cc30 0x7f12c406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:59:28.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.406+0000 7f12dd7ba700 1 --2- 172.21.15.32:0/3752014410 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f12c406cc30 0x7f12c406f0e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f12d8066890 tx=0x7f12c8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:59:28.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.410+0000 7f12ce7fc700 1 -- 172.21.15.32:0/3752014410 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f12d4055cb0 con 0x7f12d8101d10 2024-04-12T05:59:28.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.662+0000 7f12e021f700 1 -- 172.21.15.32:0/3752014410 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f12d8066640 con 0x7f12d8101d10 2024-04-12T05:59:28.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.663+0000 7f12ce7fc700 1 -- 172.21.15.32:0/3752014410 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1835 (secure 0 0 0) 0x7f12d40592d0 con 0x7f12d8101d10 2024-04-12T05:59:28.657 INFO:teuthology.orchestra.run.smithi032.stdout:e33 2024-04-12T05:59:28.657 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:59:28.657 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:59:28.657 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T05:59:28.657 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:59:28.657 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:59:28.657 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T05:59:28.658 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 24 2024-04-12T05:59:28.658 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:59:28.658 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T05:59:28.658 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:46:10.668569+0000 2024-04-12T05:59:28.658 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T05:59:28.658 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T05:59:28.658 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T05:59:28.658 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T05:59:28.658 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T05:59:28.658 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T05:59:28.658 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T05:59:28.658 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T05:59:28.658 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T05:59:28.658 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:59:28.659 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T05:59:28.659 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T05:59:28.659 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T05:59:28.659 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-12T05:59:28.659 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T05:59:28.659 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T05:59:28.659 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T05:59:28.659 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T05:59:28.659 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T05:59:28.659 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T05:59:28.659 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T05:59:28.659 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T05:59:28.659 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:59:28.659 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:59:28.660 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-12T05:59:28.660 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T05:59:28.660 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:59:28.660 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:59:28.660 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{-1:44317} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:59:28.660 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:59:28.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.665+0000 7f12e021f700 1 -- 172.21.15.32:0/3752014410 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f12c406cc30 msgr2=0x7f12c406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:28.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.665+0000 7f12e021f700 1 --2- 172.21.15.32:0/3752014410 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f12c406cc30 0x7f12c406f0e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f12d8066890 tx=0x7f12c8009e30 comp rx=0 tx=0).stop 2024-04-12T05:59:28.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.665+0000 7f12e021f700 1 -- 172.21.15.32:0/3752014410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12d8101d10 msgr2=0x7f12d810ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:28.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.665+0000 7f12e021f700 1 --2- 172.21.15.32:0/3752014410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12d8101d10 0x7f12d810ed10 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f12d4002a80 tx=0x7f12d40046b0 comp rx=0 tx=0).stop 2024-04-12T05:59:28.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.665+0000 7f12e021f700 1 -- 172.21.15.32:0/3752014410 shutdown_connections 2024-04-12T05:59:28.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.665+0000 7f12e021f700 1 --2- 172.21.15.32:0/3752014410 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f12c406cc30 0x7f12c406f0e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:28.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.665+0000 7f12e021f700 1 --2- 172.21.15.32:0/3752014410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12d8101d10 0x7f12d810ed10 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:28.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.665+0000 7f12e021f700 1 --2- 172.21.15.32:0/3752014410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12d81026b0 0x7f12d810f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:28.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.665+0000 7f12e021f700 1 -- 172.21.15.32:0/3752014410 >> 172.21.15.32:0/3752014410 conn(0x7f12d80f7880 msgr2=0x7f12d80f88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:59:28.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.665+0000 7f12e021f700 1 -- 172.21.15.32:0/3752014410 shutdown_connections 2024-04-12T05:59:28.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.665+0000 7f12e021f700 1 -- 172.21.15.32:0/3752014410 wait complete. 2024-04-12T05:59:28.661 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-12T05:59:28.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.768+0000 7fa53c53e700 1 -- 172.21.15.32:0/2857406085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa534102e00 msgr2=0x7fa5341031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:28.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.768+0000 7fa53c53e700 1 --2- 172.21.15.32:0/2857406085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa534102e00 0x7fa5341031d0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fa530004940 tx=0x7fa530009e30 comp rx=0 tx=0).stop 2024-04-12T05:59:28.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.768+0000 7fa53c53e700 1 -- 172.21.15.32:0/2857406085 shutdown_connections 2024-04-12T05:59:28.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.768+0000 7fa53c53e700 1 --2- 172.21.15.32:0/2857406085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5341037a0 0x7fa5340fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:28.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.768+0000 7fa53c53e700 1 --2- 172.21.15.32:0/2857406085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa534102e00 0x7fa5341031d0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:28.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.768+0000 7fa53c53e700 1 -- 172.21.15.32:0/2857406085 >> 172.21.15.32:0/2857406085 conn(0x7fa5340f89b0 msgr2=0x7fa5340fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:59:28.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.768+0000 7fa53c53e700 1 -- 172.21.15.32:0/2857406085 shutdown_connections 2024-04-12T05:59:28.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.769+0000 7fa53c53e700 1 -- 172.21.15.32:0/2857406085 wait complete. 2024-04-12T05:59:28.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.769+0000 7fa53c53e700 1 Processor -- start 2024-04-12T05:59:28.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.769+0000 7fa53c53e700 1 -- start start 2024-04-12T05:59:28.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.769+0000 7fa53c53e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa534102e00 0x7fa534113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:59:28.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.769+0000 7fa53c53e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5341037a0 0x7fa534114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:59:28.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.769+0000 7fa53c53e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5341146b0 con 0x7fa5341037a0 2024-04-12T05:59:28.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.769+0000 7fa53c53e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53410daa0 con 0x7fa534102e00 2024-04-12T05:59:28.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.769+0000 7fa539ad9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5341037a0 0x7fa534114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:59:28.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.769+0000 7fa539ad9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5341037a0 0x7fa534114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59956/0 (socket says 172.21.15.32:59956) 2024-04-12T05:59:28.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.769+0000 7fa539ad9700 1 -- 172.21.15.32:0/926015647 learned_addr learned my addr 172.21.15.32:0/926015647 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:59:28.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.770+0000 7fa53a2da700 1 --2- 172.21.15.32:0/926015647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa534102e00 0x7fa534113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:59:28.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.770+0000 7fa539ad9700 1 -- 172.21.15.32:0/926015647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa534102e00 msgr2=0x7fa534113b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:28.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.770+0000 7fa539ad9700 1 --2- 172.21.15.32:0/926015647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa534102e00 0x7fa534113b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:28.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.770+0000 7fa539ad9700 1 -- 172.21.15.32:0/926015647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa530005040 con 0x7fa5341037a0 2024-04-12T05:59:28.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.770+0000 7fa539ad9700 1 --2- 172.21.15.32:0/926015647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5341037a0 0x7fa534114060 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fa52800c890 tx=0x7fa5280044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:59:28.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.770+0000 7fa526ffd700 1 -- 172.21.15.32:0/926015647 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa52800a7d0 con 0x7fa5341037a0 2024-04-12T05:59:28.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.770+0000 7fa526ffd700 1 -- 172.21.15.32:0/926015647 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa52800ade0 con 0x7fa5341037a0 2024-04-12T05:59:28.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.770+0000 7fa53c53e700 1 -- 172.21.15.32:0/926015647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa53410dd80 con 0x7fa5341037a0 2024-04-12T05:59:28.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.770+0000 7fa526ffd700 1 -- 172.21.15.32:0/926015647 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5280113a0 con 0x7fa5341037a0 2024-04-12T05:59:28.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.770+0000 7fa53c53e700 1 -- 172.21.15.32:0/926015647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa53410e340 con 0x7fa5341037a0 2024-04-12T05:59:28.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.771+0000 7fa53c53e700 1 -- 172.21.15.32:0/926015647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5340fe550 con 0x7fa5341037a0 2024-04-12T05:59:28.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.772+0000 7fa526ffd700 1 -- 172.21.15.32:0/926015647 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa528012400 con 0x7fa5341037a0 2024-04-12T05:59:28.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.772+0000 7fa526ffd700 1 --2- 172.21.15.32:0/926015647 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa52006cc30 0x7fa52006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:59:28.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.772+0000 7fa53a2da700 1 --2- 172.21.15.32:0/926015647 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa52006cc30 0x7fa52006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:59:28.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.772+0000 7fa526ffd700 1 -- 172.21.15.32:0/926015647 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fa52808c0d0 con 0x7fa5341037a0 2024-04-12T05:59:28.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.773+0000 7fa53a2da700 1 --2- 172.21.15.32:0/926015647 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa52006cc30 0x7fa52006f0e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa53000b020 tx=0x7fa530009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:59:28.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:28.777+0000 7fa526ffd700 1 -- 172.21.15.32:0/926015647 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa528059be0 con 0x7fa5341037a0 2024-04-12T05:59:28.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.000+0000 7fa53c53e700 1 -- 172.21.15.32:0/926015647 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa534109dc0 con 0x7fa52006cc30 2024-04-12T05:59:28.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.001+0000 7fa526ffd700 1 -- 172.21.15.32:0/926015647 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa534109dc0 con 0x7fa52006cc30 2024-04-12T05:59:28.995 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T05:59:28.995 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:59:28.996 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T05:59:28.996 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:59:28.996 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T05:59:28.996 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-12T05:59:28.996 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T05:59:28.996 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-12T05:59:28.996 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T05:59:28.996 INFO:teuthology.orchestra.run.smithi032.stdout: "mds" 2024-04-12T05:59:28.996 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T05:59:28.996 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:59:28.996 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:59:28.996 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T05:59:28.996 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T05:59:28.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.003+0000 7fa53c53e700 1 -- 172.21.15.32:0/926015647 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa52006cc30 msgr2=0x7fa52006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:28.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.003+0000 7fa53c53e700 1 --2- 172.21.15.32:0/926015647 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa52006cc30 0x7fa52006f0e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa53000b020 tx=0x7fa530009e30 comp rx=0 tx=0).stop 2024-04-12T05:59:28.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.003+0000 7fa53c53e700 1 -- 172.21.15.32:0/926015647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5341037a0 msgr2=0x7fa534114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:28.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.003+0000 7fa53c53e700 1 --2- 172.21.15.32:0/926015647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5341037a0 0x7fa534114060 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fa52800c890 tx=0x7fa5280044c0 comp rx=0 tx=0).stop 2024-04-12T05:59:28.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.003+0000 7fa53c53e700 1 -- 172.21.15.32:0/926015647 shutdown_connections 2024-04-12T05:59:28.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.003+0000 7fa53c53e700 1 --2- 172.21.15.32:0/926015647 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa52006cc30 0x7fa52006f0e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:28.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.003+0000 7fa53c53e700 1 --2- 172.21.15.32:0/926015647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa534102e00 0x7fa534113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:28.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.003+0000 7fa53c53e700 1 --2- 172.21.15.32:0/926015647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5341037a0 0x7fa534114060 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:28.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.003+0000 7fa53c53e700 1 -- 172.21.15.32:0/926015647 >> 172.21.15.32:0/926015647 conn(0x7fa5340f89b0 msgr2=0x7fa5340f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:59:28.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.004+0000 7fa53c53e700 1 -- 172.21.15.32:0/926015647 shutdown_connections 2024-04-12T05:59:28.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.004+0000 7fa53c53e700 1 -- 172.21.15.32:0/926015647 wait complete. 2024-04-12T05:59:29.086 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:28 smithi032 ceph-mon[146817]: from='client.35068 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:59:29.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:28 smithi032 ceph-mon[146817]: from='client.35072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:59:29.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:28 smithi032 ceph-mon[146817]: from='client.35076 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:59:29.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:28 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1706246450' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:59:29.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:28 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3752014410' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:59:29.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.106+0000 7fe9526e6700 1 -- 172.21.15.32:0/3525094187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe94c0fb4a0 msgr2=0x7fe94c0fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:29.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.106+0000 7fe9526e6700 1 --2- 172.21.15.32:0/3525094187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe94c0fb4a0 0x7fe94c0fb910 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fe940004940 tx=0x7fe940009e30 comp rx=0 tx=0).stop 2024-04-12T05:59:29.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.106+0000 7fe9526e6700 1 -- 172.21.15.32:0/3525094187 shutdown_connections 2024-04-12T05:59:29.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.106+0000 7fe9526e6700 1 --2- 172.21.15.32:0/3525094187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe94c0fb4a0 0x7fe94c0fb910 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:29.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.106+0000 7fe9526e6700 1 --2- 172.21.15.32:0/3525094187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe94c102e70 0x7fe94c103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:29.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.106+0000 7fe9526e6700 1 -- 172.21.15.32:0/3525094187 >> 172.21.15.32:0/3525094187 conn(0x7fe94c0714c0 msgr2=0x7fe94c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:59:29.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.106+0000 7fe9526e6700 1 -- 172.21.15.32:0/3525094187 shutdown_connections 2024-04-12T05:59:29.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.106+0000 7fe9526e6700 1 -- 172.21.15.32:0/3525094187 wait complete. 2024-04-12T05:59:29.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.107+0000 7fe9526e6700 1 Processor -- start 2024-04-12T05:59:29.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.107+0000 7fe9526e6700 1 -- start start 2024-04-12T05:59:29.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.107+0000 7fe9526e6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe94c0fb4a0 0x7fe94c113a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:59:29.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.107+0000 7fe9526e6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe94c102e70 0x7fe94c113fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:59:29.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.107+0000 7fe9526e6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe94c1146b0 con 0x7fe94c102e70 2024-04-12T05:59:29.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.107+0000 7fe9526e6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe94c10da10 con 0x7fe94c0fb4a0 2024-04-12T05:59:29.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.107+0000 7fe94b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe94c102e70 0x7fe94c113fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:59:29.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.107+0000 7fe94b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe94c102e70 0x7fe94c113fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59974/0 (socket says 172.21.15.32:59974) 2024-04-12T05:59:29.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.107+0000 7fe94b7fe700 1 -- 172.21.15.32:0/4061173188 learned_addr learned my addr 172.21.15.32:0/4061173188 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:59:29.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.108+0000 7fe94bfff700 1 --2- 172.21.15.32:0/4061173188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe94c0fb4a0 0x7fe94c113a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:59:29.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.108+0000 7fe94b7fe700 1 -- 172.21.15.32:0/4061173188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe94c0fb4a0 msgr2=0x7fe94c113a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:29.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.108+0000 7fe94b7fe700 1 --2- 172.21.15.32:0/4061173188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe94c0fb4a0 0x7fe94c113a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:29.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.108+0000 7fe94b7fe700 1 -- 172.21.15.32:0/4061173188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe940005040 con 0x7fe94c102e70 2024-04-12T05:59:29.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.108+0000 7fe94bfff700 1 --2- 172.21.15.32:0/4061173188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe94c0fb4a0 0x7fe94c113a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:59:29.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.108+0000 7fe94b7fe700 1 --2- 172.21.15.32:0/4061173188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe94c102e70 0x7fe94c113fd0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fe94000b3b0 tx=0x7fe940009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:59:29.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.108+0000 7fe948ff9700 1 -- 172.21.15.32:0/4061173188 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe94000e040 con 0x7fe94c102e70 2024-04-12T05:59:29.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.108+0000 7fe948ff9700 1 -- 172.21.15.32:0/4061173188 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe940013420 con 0x7fe94c102e70 2024-04-12T05:59:29.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.108+0000 7fe9526e6700 1 -- 172.21.15.32:0/4061173188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe94c10dc90 con 0x7fe94c102e70 2024-04-12T05:59:29.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.108+0000 7fe948ff9700 1 -- 172.21.15.32:0/4061173188 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe94001c630 con 0x7fe94c102e70 2024-04-12T05:59:29.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.108+0000 7fe9526e6700 1 -- 172.21.15.32:0/4061173188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe94c10e250 con 0x7fe94c102e70 2024-04-12T05:59:29.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.109+0000 7fe9526e6700 1 -- 172.21.15.32:0/4061173188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe94c100110 con 0x7fe94c102e70 2024-04-12T05:59:29.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.110+0000 7fe948ff9700 1 -- 172.21.15.32:0/4061173188 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe940013970 con 0x7fe94c102e70 2024-04-12T05:59:29.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.110+0000 7fe948ff9700 1 --2- 172.21.15.32:0/4061173188 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe93406cc20 0x7fe93406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:59:29.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.110+0000 7fe948ff9700 1 -- 172.21.15.32:0/4061173188 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fe94008d3b0 con 0x7fe94c102e70 2024-04-12T05:59:29.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.110+0000 7fe94bfff700 1 --2- 172.21.15.32:0/4061173188 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe93406cc20 0x7fe93406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:59:29.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.111+0000 7fe94bfff700 1 --2- 172.21.15.32:0/4061173188 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe93406cc20 0x7fe93406f0d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe93c006fd0 tx=0x7fe93c008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:59:29.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.115+0000 7fe948ff9700 1 -- 172.21.15.32:0/4061173188 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe94005ad80 con 0x7fe94c102e70 2024-04-12T05:59:29.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:28 smithi067 ceph-mon[112979]: from='client.35068 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:59:29.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:28 smithi067 ceph-mon[112979]: from='client.35072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:59:29.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:28 smithi067 ceph-mon[112979]: from='client.35076 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:59:29.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:28 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1706246450' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:59:29.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:28 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3752014410' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:59:29.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.400+0000 7fe9526e6700 1 -- 172.21.15.32:0/4061173188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe94c10f0e0 con 0x7fe94c102e70 2024-04-12T05:59:29.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.400+0000 7fe948ff9700 1 -- 172.21.15.32:0/4061173188 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fe94005a910 con 0x7fe94c102e70 2024-04-12T05:59:29.394 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:59:29.394 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:59:29.395 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-12T05:59:29.395 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:59:29.395 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:59:29.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.402+0000 7fe9526e6700 1 -- 172.21.15.32:0/4061173188 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe93406cc20 msgr2=0x7fe93406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:29.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.402+0000 7fe9526e6700 1 --2- 172.21.15.32:0/4061173188 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe93406cc20 0x7fe93406f0d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe93c006fd0 tx=0x7fe93c008040 comp rx=0 tx=0).stop 2024-04-12T05:59:29.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.402+0000 7fe9526e6700 1 -- 172.21.15.32:0/4061173188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe94c102e70 msgr2=0x7fe94c113fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:29.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.402+0000 7fe9526e6700 1 --2- 172.21.15.32:0/4061173188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe94c102e70 0x7fe94c113fd0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fe94000b3b0 tx=0x7fe940009e30 comp rx=0 tx=0).stop 2024-04-12T05:59:29.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.402+0000 7fe9526e6700 1 -- 172.21.15.32:0/4061173188 shutdown_connections 2024-04-12T05:59:29.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.402+0000 7fe9526e6700 1 --2- 172.21.15.32:0/4061173188 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe93406cc20 0x7fe93406f0d0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:29.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.402+0000 7fe9526e6700 1 --2- 172.21.15.32:0/4061173188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe94c0fb4a0 0x7fe94c113a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:29.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.402+0000 7fe9526e6700 1 --2- 172.21.15.32:0/4061173188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe94c102e70 0x7fe94c113fd0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:29.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.402+0000 7fe9526e6700 1 -- 172.21.15.32:0/4061173188 >> 172.21.15.32:0/4061173188 conn(0x7fe94c0714c0 msgr2=0x7fe94c0fa7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:59:29.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.402+0000 7fe9526e6700 1 -- 172.21.15.32:0/4061173188 shutdown_connections 2024-04-12T05:59:29.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:29.402+0000 7fe9526e6700 1 -- 172.21.15.32:0/4061173188 wait complete. 2024-04-12T05:59:30.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:29 smithi032 ceph-mon[146817]: pgmap v569: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:30.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:29 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/4061173188' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:59:30.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:29 smithi067 ceph-mon[112979]: pgmap v569: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:30.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:29 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/4061173188' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:59:31.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:30 smithi032 ceph-mon[146817]: from='client.35088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:59:31.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:30 smithi067 ceph-mon[112979]: from='client.35088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:59:32.086 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:31 smithi032 ceph-mon[146817]: pgmap v570: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:32.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:31 smithi067 ceph-mon[112979]: pgmap v570: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:34.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:33 smithi032 ceph-mon[146817]: pgmap v571: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:34.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:33 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:59:34.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:33 smithi067 ceph-mon[112979]: pgmap v571: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:34.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:33 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:59:36.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:35 smithi032 ceph-mon[146817]: pgmap v572: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:36.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:35 smithi067 ceph-mon[112979]: pgmap v572: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:38.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:37 smithi032 ceph-mon[146817]: pgmap v573: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:38.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:37 smithi067 ceph-mon[112979]: pgmap v573: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:40.086 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:39 smithi032 ceph-mon[146817]: pgmap v574: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:40.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:39 smithi067 ceph-mon[112979]: pgmap v574: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:42.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:41 smithi032 ceph-mon[146817]: pgmap v575: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:42.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:41 smithi067 ceph-mon[112979]: pgmap v575: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:44.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:43 smithi032 ceph-mon[146817]: pgmap v576: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:44.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:43 smithi067 ceph-mon[112979]: pgmap v576: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:46.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:45 smithi032 ceph-mon[146817]: pgmap v577: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:46.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:45 smithi067 ceph-mon[112979]: pgmap v577: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:48.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:47 smithi032 ceph-mon[146817]: pgmap v578: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:48.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:47 smithi067 ceph-mon[112979]: pgmap v578: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:49.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:48 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:59:49.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:48 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:59:50.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:49 smithi067 ceph-mon[112979]: pgmap v579: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:50.336 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:49 smithi032 ceph-mon[146817]: pgmap v579: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:52.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:51 smithi067 ceph-mon[112979]: pgmap v580: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:52.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:51 smithi032 ceph-mon[146817]: pgmap v580: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:54.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:53 smithi067 ceph-mon[112979]: pgmap v581: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:54.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:53 smithi032 ceph-mon[146817]: pgmap v581: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:56.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:55 smithi067 ceph-mon[112979]: pgmap v582: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:56.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:55 smithi032 ceph-mon[146817]: pgmap v582: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:58.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:57 smithi067 ceph-mon[112979]: pgmap v583: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:58.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:57 smithi032 ceph-mon[146817]: pgmap v583: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:59:59.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.520+0000 7f618bbb8700 1 -- 172.21.15.32:0/4114584812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61841037c0 msgr2=0x7f61840fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:59.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.520+0000 7f618bbb8700 1 --2- 172.21.15.32:0/4114584812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61841037c0 0x7f61840fce30 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f6178004940 tx=0x7f6178009e30 comp rx=0 tx=0).stop 2024-04-12T05:59:59.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.520+0000 7f618bbb8700 1 -- 172.21.15.32:0/4114584812 shutdown_connections 2024-04-12T05:59:59.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.520+0000 7f618bbb8700 1 --2- 172.21.15.32:0/4114584812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61841037c0 0x7f61840fce30 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:59.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.520+0000 7f618bbb8700 1 --2- 172.21.15.32:0/4114584812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6184102e20 0x7f61841031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:59.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.520+0000 7f618bbb8700 1 -- 172.21.15.32:0/4114584812 >> 172.21.15.32:0/4114584812 conn(0x7f61840f89d0 msgr2=0x7f61840fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:59:59.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.520+0000 7f618bbb8700 1 -- 172.21.15.32:0/4114584812 shutdown_connections 2024-04-12T05:59:59.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.520+0000 7f618bbb8700 1 -- 172.21.15.32:0/4114584812 wait complete. 2024-04-12T05:59:59.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.520+0000 7f618bbb8700 1 Processor -- start 2024-04-12T05:59:59.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.520+0000 7f618bbb8700 1 -- start start 2024-04-12T05:59:59.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.521+0000 7f618bbb8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6184102e20 0x7f6184113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:59:59.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.521+0000 7f618bbb8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61841037c0 0x7f6184114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:59:59.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.521+0000 7f618bbb8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6184114660 con 0x7f6184102e20 2024-04-12T05:59:59.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.521+0000 7f618bbb8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f618410da50 con 0x7f61841037c0 2024-04-12T05:59:59.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.521+0000 7f6189954700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6184102e20 0x7f6184113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:59:59.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.521+0000 7f6189954700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6184102e20 0x7f6184113ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54816/0 (socket says 172.21.15.32:54816) 2024-04-12T05:59:59.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.521+0000 7f6189954700 1 -- 172.21.15.32:0/2654226206 learned_addr learned my addr 172.21.15.32:0/2654226206 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:59:59.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.521+0000 7f6189153700 1 --2- 172.21.15.32:0/2654226206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61841037c0 0x7f6184114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:59:59.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.521+0000 7f6189954700 1 -- 172.21.15.32:0/2654226206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61841037c0 msgr2=0x7f6184114010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:59.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.521+0000 7f6189954700 1 --2- 172.21.15.32:0/2654226206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61841037c0 0x7f6184114010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:59.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.521+0000 7f6189954700 1 -- 172.21.15.32:0/2654226206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6178005040 con 0x7f6184102e20 2024-04-12T05:59:59.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.521+0000 7f6189954700 1 --2- 172.21.15.32:0/2654226206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6184102e20 0x7f6184113ad0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f6180002a50 tx=0x7f61800046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:59:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.521+0000 7f61767fc700 1 -- 172.21.15.32:0/2654226206 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6180007cc0 con 0x7f6184102e20 2024-04-12T05:59:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.522+0000 7f61767fc700 1 -- 172.21.15.32:0/2654226206 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f618000f040 con 0x7f6184102e20 2024-04-12T05:59:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.522+0000 7f618bbb8700 1 -- 172.21.15.32:0/2654226206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f618410dd30 con 0x7f6184102e20 2024-04-12T05:59:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.522+0000 7f61767fc700 1 -- 172.21.15.32:0/2654226206 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6180009540 con 0x7f6184102e20 2024-04-12T05:59:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.522+0000 7f618bbb8700 1 -- 172.21.15.32:0/2654226206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f618410e2c0 con 0x7f6184102e20 2024-04-12T05:59:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.523+0000 7f618bbb8700 1 -- 172.21.15.32:0/2654226206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61840fe570 con 0x7f6184102e20 2024-04-12T05:59:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.523+0000 7f61767fc700 1 -- 172.21.15.32:0/2654226206 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f61800096a0 con 0x7f6184102e20 2024-04-12T05:59:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.524+0000 7f61767fc700 1 --2- 172.21.15.32:0/2654226206 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f617006cc30 0x7f617006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:59:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.524+0000 7f61767fc700 1 -- 172.21.15.32:0/2654226206 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f618008bf70 con 0x7f6184102e20 2024-04-12T05:59:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.524+0000 7f6189153700 1 --2- 172.21.15.32:0/2654226206 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f617006cc30 0x7f617006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:59:59.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.524+0000 7f6189153700 1 --2- 172.21.15.32:0/2654226206 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f617006cc30 0x7f617006f0e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f618410ed30 tx=0x7f6178009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:59:59.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.528+0000 7f61767fc700 1 -- 172.21.15.32:0/2654226206 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f61800598c0 con 0x7f6184102e20 2024-04-12T05:59:59.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.749+0000 7f618bbb8700 1 -- 172.21.15.32:0/2654226206 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6184109e10 con 0x7f617006cc30 2024-04-12T05:59:59.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.750+0000 7f61767fc700 1 -- 172.21.15.32:0/2654226206 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6184109e10 con 0x7f617006cc30 2024-04-12T05:59:59.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.752+0000 7f618bbb8700 1 -- 172.21.15.32:0/2654226206 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f617006cc30 msgr2=0x7f617006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:59.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.752+0000 7f618bbb8700 1 --2- 172.21.15.32:0/2654226206 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f617006cc30 0x7f617006f0e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f618410ed30 tx=0x7f6178009e30 comp rx=0 tx=0).stop 2024-04-12T05:59:59.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.752+0000 7f618bbb8700 1 -- 172.21.15.32:0/2654226206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6184102e20 msgr2=0x7f6184113ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:59.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.752+0000 7f618bbb8700 1 --2- 172.21.15.32:0/2654226206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6184102e20 0x7f6184113ad0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f6180002a50 tx=0x7f61800046b0 comp rx=0 tx=0).stop 2024-04-12T05:59:59.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.752+0000 7f618bbb8700 1 -- 172.21.15.32:0/2654226206 shutdown_connections 2024-04-12T05:59:59.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.752+0000 7f618bbb8700 1 --2- 172.21.15.32:0/2654226206 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f617006cc30 0x7f617006f0e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:59.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.752+0000 7f618bbb8700 1 --2- 172.21.15.32:0/2654226206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6184102e20 0x7f6184113ad0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:59.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.752+0000 7f618bbb8700 1 --2- 172.21.15.32:0/2654226206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61841037c0 0x7f6184114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:59.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.752+0000 7f618bbb8700 1 -- 172.21.15.32:0/2654226206 >> 172.21.15.32:0/2654226206 conn(0x7f61840f89d0 msgr2=0x7f61840f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:59:59.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.753+0000 7f618bbb8700 1 -- 172.21.15.32:0/2654226206 shutdown_connections 2024-04-12T05:59:59.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.753+0000 7f618bbb8700 1 -- 172.21.15.32:0/2654226206 wait complete. 2024-04-12T05:59:59.757 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T05:59:59.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.857+0000 7f102b398700 1 -- 172.21.15.32:0/709017621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10240fb4a0 msgr2=0x7f10240fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:59.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.857+0000 7f102b398700 1 --2- 172.21.15.32:0/709017621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10240fb4a0 0x7f10240fb910 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f1014004940 tx=0x7f1014009e30 comp rx=0 tx=0).stop 2024-04-12T05:59:59.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.857+0000 7f102b398700 1 -- 172.21.15.32:0/709017621 shutdown_connections 2024-04-12T05:59:59.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.857+0000 7f102b398700 1 --2- 172.21.15.32:0/709017621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10240fb4a0 0x7f10240fb910 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:59.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.857+0000 7f102b398700 1 --2- 172.21.15.32:0/709017621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1024102e70 0x7f1024103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:59.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.857+0000 7f102b398700 1 -- 172.21.15.32:0/709017621 >> 172.21.15.32:0/709017621 conn(0x7f10240714c0 msgr2=0x7f10240718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:59:59.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.857+0000 7f102b398700 1 -- 172.21.15.32:0/709017621 shutdown_connections 2024-04-12T05:59:59.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.857+0000 7f102b398700 1 -- 172.21.15.32:0/709017621 wait complete. 2024-04-12T05:59:59.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.858+0000 7f102b398700 1 Processor -- start 2024-04-12T05:59:59.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.858+0000 7f102b398700 1 -- start start 2024-04-12T05:59:59.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.858+0000 7f102b398700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10240fb4a0 0x7f1024113a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:59:59.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.858+0000 7f102b398700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1024102e70 0x7f1024113f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:59:59.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.858+0000 7f102b398700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10241145c0 con 0x7f10240fb4a0 2024-04-12T05:59:59.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.858+0000 7f102b398700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f102410d9b0 con 0x7f1024102e70 2024-04-12T05:59:59.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.858+0000 7f1029134700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10240fb4a0 0x7f1024113a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:59:59.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.858+0000 7f1029134700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10240fb4a0 0x7f1024113a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54836/0 (socket says 172.21.15.32:54836) 2024-04-12T05:59:59.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.858+0000 7f1029134700 1 -- 172.21.15.32:0/385761821 learned_addr learned my addr 172.21.15.32:0/385761821 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T05:59:59.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.858+0000 7f1028933700 1 --2- 172.21.15.32:0/385761821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1024102e70 0x7f1024113f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:59:59.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.859+0000 7f1029134700 1 -- 172.21.15.32:0/385761821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1024102e70 msgr2=0x7f1024113f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:59:59.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.859+0000 7f1029134700 1 --2- 172.21.15.32:0/385761821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1024102e70 0x7f1024113f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T05:59:59.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.859+0000 7f1029134700 1 -- 172.21.15.32:0/385761821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1014005040 con 0x7f10240fb4a0 2024-04-12T05:59:59.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.859+0000 7f1028933700 1 --2- 172.21.15.32:0/385761821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1024102e70 0x7f1024113f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:59:59.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.859+0000 7f1029134700 1 --2- 172.21.15.32:0/385761821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10240fb4a0 0x7f1024113a30 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f1020002a80 tx=0x7f10200046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:59:59.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.859+0000 7f1019ffb700 1 -- 172.21.15.32:0/385761821 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1020007cc0 con 0x7f10240fb4a0 2024-04-12T05:59:59.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.859+0000 7f102b398700 1 -- 172.21.15.32:0/385761821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f102410dc90 con 0x7f10240fb4a0 2024-04-12T05:59:59.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.859+0000 7f1019ffb700 1 -- 172.21.15.32:0/385761821 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f102000f040 con 0x7f10240fb4a0 2024-04-12T05:59:59.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.859+0000 7f1019ffb700 1 -- 172.21.15.32:0/385761821 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1020009640 con 0x7f10240fb4a0 2024-04-12T05:59:59.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.859+0000 7f102b398700 1 -- 172.21.15.32:0/385761821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f102410e250 con 0x7f10240fb4a0 2024-04-12T05:59:59.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.860+0000 7f102b398700 1 -- 172.21.15.32:0/385761821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1024100110 con 0x7f10240fb4a0 2024-04-12T05:59:59.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.861+0000 7f1019ffb700 1 -- 172.21.15.32:0/385761821 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f10200097a0 con 0x7f10240fb4a0 2024-04-12T05:59:59.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.861+0000 7f1019ffb700 1 --2- 172.21.15.32:0/385761821 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f101006cc30 0x7f101006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T05:59:59.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.861+0000 7f1019ffb700 1 -- 172.21.15.32:0/385761821 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f102008b8e0 con 0x7f10240fb4a0 2024-04-12T05:59:59.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.861+0000 7f1028933700 1 --2- 172.21.15.32:0/385761821 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f101006cc30 0x7f101006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:59:59.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.861+0000 7f1028933700 1 --2- 172.21.15.32:0/385761821 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f101006cc30 0x7f101006f0e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f102410eb70 tx=0x7f1014009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:59:59.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T05:59:59.866+0000 7f1019ffb700 1 -- 172.21.15.32:0/385761821 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f10200592d0 con 0x7f10240fb4a0 2024-04-12T06:00:00.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.088+0000 7f102b398700 1 -- 172.21.15.32:0/385761821 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f102410f080 con 0x7f101006cc30 2024-04-12T06:00:00.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.089+0000 7f1019ffb700 1 -- 172.21.15.32:0/385761821 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f102410f080 con 0x7f101006cc30 2024-04-12T06:00:00.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.091+0000 7f102b398700 1 -- 172.21.15.32:0/385761821 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f101006cc30 msgr2=0x7f101006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:00.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.091+0000 7f102b398700 1 --2- 172.21.15.32:0/385761821 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f101006cc30 0x7f101006f0e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f102410eb70 tx=0x7f1014009e30 comp rx=0 tx=0).stop 2024-04-12T06:00:00.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.091+0000 7f102b398700 1 -- 172.21.15.32:0/385761821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10240fb4a0 msgr2=0x7f1024113a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:00.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.091+0000 7f102b398700 1 --2- 172.21.15.32:0/385761821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10240fb4a0 0x7f1024113a30 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f1020002a80 tx=0x7f10200046b0 comp rx=0 tx=0).stop 2024-04-12T06:00:00.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.092+0000 7f102b398700 1 -- 172.21.15.32:0/385761821 shutdown_connections 2024-04-12T06:00:00.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.092+0000 7f102b398700 1 --2- 172.21.15.32:0/385761821 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f101006cc30 0x7f101006f0e0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:00.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.092+0000 7f102b398700 1 --2- 172.21.15.32:0/385761821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10240fb4a0 0x7f1024113a30 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:00.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.092+0000 7f102b398700 1 --2- 172.21.15.32:0/385761821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1024102e70 0x7f1024113f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:00.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.092+0000 7f102b398700 1 -- 172.21.15.32:0/385761821 >> 172.21.15.32:0/385761821 conn(0x7f10240714c0 msgr2=0x7f10240fa830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:00:00.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.092+0000 7f102b398700 1 -- 172.21.15.32:0/385761821 shutdown_connections 2024-04-12T06:00:00.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.092+0000 7f102b398700 1 -- 172.21.15.32:0/385761821 wait complete. 2024-04-12T06:00:00.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.197+0000 7f7a2ba8b700 1 -- 172.21.15.32:0/289793363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a24102710 msgr2=0x7f7a240fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:00.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.197+0000 7f7a2ba8b700 1 --2- 172.21.15.32:0/289793363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a24102710 0x7f7a240fbd80 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f7a18004940 tx=0x7f7a18009e30 comp rx=0 tx=0).stop 2024-04-12T06:00:00.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.197+0000 7f7a2ba8b700 1 -- 172.21.15.32:0/289793363 shutdown_connections 2024-04-12T06:00:00.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.197+0000 7f7a2ba8b700 1 --2- 172.21.15.32:0/289793363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a24102710 0x7f7a240fbd80 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:00.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.197+0000 7f7a2ba8b700 1 --2- 172.21.15.32:0/289793363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a24101d70 0x7f7a24102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:00.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.197+0000 7f7a2ba8b700 1 -- 172.21.15.32:0/289793363 >> 172.21.15.32:0/289793363 conn(0x7f7a24072bc0 msgr2=0x7f7a24072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:00:00.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.198+0000 7f7a2ba8b700 1 -- 172.21.15.32:0/289793363 shutdown_connections 2024-04-12T06:00:00.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.198+0000 7f7a2ba8b700 1 -- 172.21.15.32:0/289793363 wait complete. 2024-04-12T06:00:00.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.198+0000 7f7a2ba8b700 1 Processor -- start 2024-04-12T06:00:00.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.198+0000 7f7a2ba8b700 1 -- start start 2024-04-12T06:00:00.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.198+0000 7f7a2ba8b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a24101d70 0x7f7a2410e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:00.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.198+0000 7f7a2ba8b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a24102710 0x7f7a2410eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:00.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.198+0000 7f7a2ba8b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2410f530 con 0x7f7a24102710 2024-04-12T06:00:00.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.198+0000 7f7a2ba8b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2410f670 con 0x7f7a24101d70 2024-04-12T06:00:00.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.198+0000 7f7a29026700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a24102710 0x7f7a2410eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:00.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.198+0000 7f7a29026700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a24102710 0x7f7a2410eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54856/0 (socket says 172.21.15.32:54856) 2024-04-12T06:00:00.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.198+0000 7f7a29026700 1 -- 172.21.15.32:0/1950745332 learned_addr learned my addr 172.21.15.32:0/1950745332 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:00:00.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.199+0000 7f7a29827700 1 --2- 172.21.15.32:0/1950745332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a24101d70 0x7f7a2410e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:00.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.199+0000 7f7a29026700 1 -- 172.21.15.32:0/1950745332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a24101d70 msgr2=0x7f7a2410e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:00.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.199+0000 7f7a29026700 1 --2- 172.21.15.32:0/1950745332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a24101d70 0x7f7a2410e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:00.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.199+0000 7f7a29026700 1 -- 172.21.15.32:0/1950745332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a18005040 con 0x7f7a24102710 2024-04-12T06:00:00.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.199+0000 7f7a29026700 1 --2- 172.21.15.32:0/1950745332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a24102710 0x7f7a2410eee0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f7a18008d50 tx=0x7f7a18009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:00:00.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.199+0000 7f7a167fc700 1 -- 172.21.15.32:0/1950745332 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a1800e030 con 0x7f7a24102710 2024-04-12T06:00:00.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.199+0000 7f7a167fc700 1 -- 172.21.15.32:0/1950745332 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a18013de0 con 0x7f7a24102710 2024-04-12T06:00:00.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.199+0000 7f7a2ba8b700 1 -- 172.21.15.32:0/1950745332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a24117a10 con 0x7f7a24102710 2024-04-12T06:00:00.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.199+0000 7f7a167fc700 1 -- 172.21.15.32:0/1950745332 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a1801cd90 con 0x7f7a24102710 2024-04-12T06:00:00.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.199+0000 7f7a2ba8b700 1 -- 172.21.15.32:0/1950745332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a24117f20 con 0x7f7a24102710 2024-04-12T06:00:00.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.200+0000 7f7a2ba8b700 1 -- 172.21.15.32:0/1950745332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a240ff080 con 0x7f7a24102710 2024-04-12T06:00:00.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.201+0000 7f7a167fc700 1 -- 172.21.15.32:0/1950745332 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f7a18013930 con 0x7f7a24102710 2024-04-12T06:00:00.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.201+0000 7f7a167fc700 1 --2- 172.21.15.32:0/1950745332 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7a1006cc30 0x7f7a1006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:00.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.201+0000 7f7a167fc700 1 -- 172.21.15.32:0/1950745332 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f7a1808d970 con 0x7f7a24102710 2024-04-12T06:00:00.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.201+0000 7f7a29827700 1 --2- 172.21.15.32:0/1950745332 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7a1006cc30 0x7f7a1006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:00.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.202+0000 7f7a29827700 1 --2- 172.21.15.32:0/1950745332 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7a1006cc30 0x7f7a1006f0e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7a20004600 tx=0x7f7a2000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:00:00.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.206+0000 7f7a167fc700 1 -- 172.21.15.32:0/1950745332 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7a18057950 con 0x7f7a24102710 2024-04-12T06:00:00.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 05:59:59 smithi067 ceph-mon[112979]: pgmap v584: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:00.336 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 05:59:59 smithi032 ceph-mon[146817]: pgmap v584: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:00.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:00 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-mon-smithi032[146794]: 2024-04-12T05:59:59.998+0000 7f3eb2941700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T06:00:00.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.418+0000 7f7a2ba8b700 1 -- 172.21.15.32:0/1950745332 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a24065e50 con 0x7f7a1006cc30 2024-04-12T06:00:00.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.427+0000 7f7a167fc700 1 -- 172.21.15.32:0/1950745332 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f7a24065e50 con 0x7f7a1006cc30 2024-04-12T06:00:00.423 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T06:00:00.423 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (19m) 13m ago 31m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T06:00:00.423 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (17m) 13m ago 31m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T06:00:00.423 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (17m) 12m ago 30m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T06:00:00.423 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (19m) 13m ago 31m 87.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T06:00:00.423 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (13m) 13m ago 25m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T06:00:00.424 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (13m) 13m ago 25m 13.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T06:00:00.424 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (12m) 12m ago 25m 11.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 6d4aca6f74a0 2024-04-12T06:00:00.424 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (13m) 12m ago 25m 14.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T06:00:00.424 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (21m) 13m ago 32m 507M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T06:00:00.424 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (20m) 12m ago 30m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T06:00:00.424 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (18m) 13m ago 32m 53.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T06:00:00.424 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (17m) 12m ago 30m 43.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T06:00:00.424 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (20m) 13m ago 31m 16.9M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T06:00:00.424 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (20m) 12m ago 30m 18.4M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T06:00:00.424 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (16m) 13m ago 29m 91.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T06:00:00.424 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (16m) 13m ago 28m 97.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T06:00:00.424 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (15m) 13m ago 28m 81.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T06:00:00.424 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (15m) 12m ago 28m 113M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T06:00:00.424 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (14m) 12m ago 27m 90.3M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T06:00:00.425 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (13m) 12m ago 27m 84.8M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T06:00:00.425 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (20m) 13m ago 30m 64.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T06:00:00.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.430+0000 7f7a2ba8b700 1 -- 172.21.15.32:0/1950745332 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7a1006cc30 msgr2=0x7f7a1006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:00.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.430+0000 7f7a2ba8b700 1 --2- 172.21.15.32:0/1950745332 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7a1006cc30 0x7f7a1006f0e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7a20004600 tx=0x7f7a2000b040 comp rx=0 tx=0).stop 2024-04-12T06:00:00.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.430+0000 7f7a2ba8b700 1 -- 172.21.15.32:0/1950745332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a24102710 msgr2=0x7f7a2410eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:00.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.430+0000 7f7a2ba8b700 1 --2- 172.21.15.32:0/1950745332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a24102710 0x7f7a2410eee0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f7a18008d50 tx=0x7f7a18009e30 comp rx=0 tx=0).stop 2024-04-12T06:00:00.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.430+0000 7f7a2ba8b700 1 -- 172.21.15.32:0/1950745332 shutdown_connections 2024-04-12T06:00:00.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.430+0000 7f7a2ba8b700 1 --2- 172.21.15.32:0/1950745332 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7a1006cc30 0x7f7a1006f0e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:00.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.430+0000 7f7a2ba8b700 1 --2- 172.21.15.32:0/1950745332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a24101d70 0x7f7a2410e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:00.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.430+0000 7f7a2ba8b700 1 --2- 172.21.15.32:0/1950745332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a24102710 0x7f7a2410eee0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:00.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.430+0000 7f7a2ba8b700 1 -- 172.21.15.32:0/1950745332 >> 172.21.15.32:0/1950745332 conn(0x7f7a24072bc0 msgr2=0x7f7a240faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:00:00.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.430+0000 7f7a2ba8b700 1 -- 172.21.15.32:0/1950745332 shutdown_connections 2024-04-12T06:00:00.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.430+0000 7f7a2ba8b700 1 -- 172.21.15.32:0/1950745332 wait complete. 2024-04-12T06:00:00.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.532+0000 7f9570256700 1 -- 172.21.15.32:0/2507219106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95680fe810 msgr2=0x7f9568106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:00.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.532+0000 7f9570256700 1 --2- 172.21.15.32:0/2507219106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95680fe810 0x7f9568106e50 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f9564004940 tx=0x7f9564009e30 comp rx=0 tx=0).stop 2024-04-12T06:00:00.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.532+0000 7f9570256700 1 -- 172.21.15.32:0/2507219106 shutdown_connections 2024-04-12T06:00:00.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.532+0000 7f9570256700 1 --2- 172.21.15.32:0/2507219106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95680fe810 0x7f9568106e50 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:00.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.532+0000 7f9570256700 1 --2- 172.21.15.32:0/2507219106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95680fde70 0x7f95680fe240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:00.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.532+0000 7f9570256700 1 -- 172.21.15.32:0/2507219106 >> 172.21.15.32:0/2507219106 conn(0x7f95680f7880 msgr2=0x7f95680f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:00:00.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.532+0000 7f9570256700 1 -- 172.21.15.32:0/2507219106 shutdown_connections 2024-04-12T06:00:00.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.532+0000 7f9570256700 1 -- 172.21.15.32:0/2507219106 wait complete. 2024-04-12T06:00:00.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.533+0000 7f9570256700 1 Processor -- start 2024-04-12T06:00:00.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.533+0000 7f9570256700 1 -- start start 2024-04-12T06:00:00.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.533+0000 7f9570256700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95680fde70 0x7f9568113d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:00.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.533+0000 7f9570256700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95680fe810 0x7f9568114260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:00.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.533+0000 7f9570256700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95681148b0 con 0x7f95680fe810 2024-04-12T06:00:00.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.533+0000 7f9570256700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956810dc80 con 0x7f95680fde70 2024-04-12T06:00:00.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.533+0000 7f956d7f1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95680fe810 0x7f9568114260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:00.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.533+0000 7f956d7f1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95680fe810 0x7f9568114260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54884/0 (socket says 172.21.15.32:54884) 2024-04-12T06:00:00.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.533+0000 7f956d7f1700 1 -- 172.21.15.32:0/243601772 learned_addr learned my addr 172.21.15.32:0/243601772 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:00:00.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.533+0000 7f956dff2700 1 --2- 172.21.15.32:0/243601772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95680fde70 0x7f9568113d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:00.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.533+0000 7f956d7f1700 1 -- 172.21.15.32:0/243601772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95680fde70 msgr2=0x7f9568113d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:00.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.533+0000 7f956d7f1700 1 --2- 172.21.15.32:0/243601772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95680fde70 0x7f9568113d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:00.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.533+0000 7f956d7f1700 1 -- 172.21.15.32:0/243601772 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9564005040 con 0x7f95680fe810 2024-04-12T06:00:00.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.534+0000 7f956d7f1700 1 --2- 172.21.15.32:0/243601772 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95680fe810 0x7f9568114260 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f956400b3b0 tx=0x7f9564009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:00:00.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.534+0000 7f955e7fc700 1 -- 172.21.15.32:0/243601772 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f956400e040 con 0x7f95680fe810 2024-04-12T06:00:00.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.534+0000 7f9570256700 1 -- 172.21.15.32:0/243601772 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f956810df60 con 0x7f95680fe810 2024-04-12T06:00:00.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.534+0000 7f955e7fc700 1 -- 172.21.15.32:0/243601772 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9564013420 con 0x7f95680fe810 2024-04-12T06:00:00.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.534+0000 7f955e7fc700 1 -- 172.21.15.32:0/243601772 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f956401c630 con 0x7f95680fe810 2024-04-12T06:00:00.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.534+0000 7f9570256700 1 -- 172.21.15.32:0/243601772 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f956810e4a0 con 0x7f95680fe810 2024-04-12T06:00:00.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.535+0000 7f9570256700 1 -- 172.21.15.32:0/243601772 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9568066710 con 0x7f95680fe810 2024-04-12T06:00:00.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.535+0000 7f955e7fc700 1 -- 172.21.15.32:0/243601772 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9564013590 con 0x7f95680fe810 2024-04-12T06:00:00.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.536+0000 7f955e7fc700 1 --2- 172.21.15.32:0/243601772 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f955406cc30 0x7f955406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:00.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.536+0000 7f955e7fc700 1 -- 172.21.15.32:0/243601772 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f956408d360 con 0x7f95680fe810 2024-04-12T06:00:00.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.536+0000 7f956dff2700 1 --2- 172.21.15.32:0/243601772 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f955406cc30 0x7f955406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:00.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.536+0000 7f956dff2700 1 --2- 172.21.15.32:0/243601772 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f955406cc30 0x7f955406f0e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9558004f40 tx=0x7f955800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:00:00.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.540+0000 7f955e7fc700 1 -- 172.21.15.32:0/243601772 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f956405ad30 con 0x7f95680fe810 2024-04-12T06:00:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.826+0000 7f9570256700 1 -- 172.21.15.32:0/243601772 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f956804bed0 con 0x7f95680fe810 2024-04-12T06:00:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.827+0000 7f955e7fc700 1 -- 172.21.15.32:0/243601772 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f956405a8c0 con 0x7f95680fe810 2024-04-12T06:00:00.822 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T06:00:00.822 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T06:00:00.823 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T06:00:00.823 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T06:00:00.823 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T06:00:00.823 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T06:00:00.823 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T06:00:00.823 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T06:00:00.823 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T06:00:00.823 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T06:00:00.824 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T06:00:00.824 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T06:00:00.824 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T06:00:00.824 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T06:00:00.824 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T06:00:00.824 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T06:00:00.824 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T06:00:00.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.829+0000 7f9570256700 1 -- 172.21.15.32:0/243601772 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f955406cc30 msgr2=0x7f955406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:00.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.829+0000 7f9570256700 1 --2- 172.21.15.32:0/243601772 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f955406cc30 0x7f955406f0e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9558004f40 tx=0x7f955800b040 comp rx=0 tx=0).stop 2024-04-12T06:00:00.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.829+0000 7f9570256700 1 -- 172.21.15.32:0/243601772 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95680fe810 msgr2=0x7f9568114260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:00.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.829+0000 7f9570256700 1 --2- 172.21.15.32:0/243601772 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95680fe810 0x7f9568114260 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f956400b3b0 tx=0x7f9564009e30 comp rx=0 tx=0).stop 2024-04-12T06:00:00.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.829+0000 7f9570256700 1 -- 172.21.15.32:0/243601772 shutdown_connections 2024-04-12T06:00:00.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.829+0000 7f9570256700 1 --2- 172.21.15.32:0/243601772 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f955406cc30 0x7f955406f0e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:00.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.829+0000 7f9570256700 1 --2- 172.21.15.32:0/243601772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95680fde70 0x7f9568113d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:00.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.829+0000 7f9570256700 1 --2- 172.21.15.32:0/243601772 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95680fe810 0x7f9568114260 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:00.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.829+0000 7f9570256700 1 -- 172.21.15.32:0/243601772 >> 172.21.15.32:0/243601772 conn(0x7f95680f7880 msgr2=0x7f95680f8ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:00:00.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.829+0000 7f9570256700 1 -- 172.21.15.32:0/243601772 shutdown_connections 2024-04-12T06:00:00.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.829+0000 7f9570256700 1 -- 172.21.15.32:0/243601772 wait complete. 2024-04-12T06:00:00.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.933+0000 7f5f0c0d2700 1 -- 172.21.15.32:0/1699904914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f04102710 msgr2=0x7f5f040fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:00.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.933+0000 7f5f0c0d2700 1 --2- 172.21.15.32:0/1699904914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f04102710 0x7f5f040fbd80 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4004940 tx=0x7f5ef4009e30 comp rx=0 tx=0).stop 2024-04-12T06:00:00.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.933+0000 7f5f0c0d2700 1 -- 172.21.15.32:0/1699904914 shutdown_connections 2024-04-12T06:00:00.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.933+0000 7f5f0c0d2700 1 --2- 172.21.15.32:0/1699904914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f04102710 0x7f5f040fbd80 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:00.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.933+0000 7f5f0c0d2700 1 --2- 172.21.15.32:0/1699904914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f04101d70 0x7f5f04102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:00.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.933+0000 7f5f0c0d2700 1 -- 172.21.15.32:0/1699904914 >> 172.21.15.32:0/1699904914 conn(0x7f5f04072bc0 msgr2=0x7f5f04072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:00:00.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.933+0000 7f5f0c0d2700 1 -- 172.21.15.32:0/1699904914 shutdown_connections 2024-04-12T06:00:00.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.933+0000 7f5f0c0d2700 1 -- 172.21.15.32:0/1699904914 wait complete. 2024-04-12T06:00:00.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.933+0000 7f5f0c0d2700 1 Processor -- start 2024-04-12T06:00:00.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.934+0000 7f5f0c0d2700 1 -- start start 2024-04-12T06:00:00.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.934+0000 7f5f0c0d2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f04101d70 0x7f5f0410ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:00.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.934+0000 7f5f0c0d2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f04102710 0x7f5f0410f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:00.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.934+0000 7f5f0c0d2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0410f6a0 con 0x7f5f04101d70 2024-04-12T06:00:00.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.934+0000 7f5f0c0d2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0410f7e0 con 0x7f5f04102710 2024-04-12T06:00:00.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.934+0000 7f5f09e6e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f04101d70 0x7f5f0410ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:00.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.934+0000 7f5f09e6e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f04101d70 0x7f5f0410ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54908/0 (socket says 172.21.15.32:54908) 2024-04-12T06:00:00.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.934+0000 7f5f09e6e700 1 -- 172.21.15.32:0/1218651292 learned_addr learned my addr 172.21.15.32:0/1218651292 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:00:00.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.934+0000 7f5f0966d700 1 --2- 172.21.15.32:0/1218651292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f04102710 0x7f5f0410f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:00.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.934+0000 7f5f09e6e700 1 -- 172.21.15.32:0/1218651292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f04102710 msgr2=0x7f5f0410f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:00.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.934+0000 7f5f09e6e700 1 --2- 172.21.15.32:0/1218651292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f04102710 0x7f5f0410f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:00.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.934+0000 7f5f09e6e700 1 -- 172.21.15.32:0/1218651292 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef4005040 con 0x7f5f04101d70 2024-04-12T06:00:00.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.934+0000 7f5f09e6e700 1 --2- 172.21.15.32:0/1218651292 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f04101d70 0x7f5f0410ec20 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5f0000c810 tx=0x7f5f00004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:00:00.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.934+0000 7f5efa7fc700 1 -- 172.21.15.32:0/1218651292 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f00012810 con 0x7f5f04101d70 2024-04-12T06:00:00.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.935+0000 7f5efa7fc700 1 -- 172.21.15.32:0/1218651292 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f00012e20 con 0x7f5f04101d70 2024-04-12T06:00:00.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.935+0000 7f5f0c0d2700 1 -- 172.21.15.32:0/1218651292 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f04117ad0 con 0x7f5f04101d70 2024-04-12T06:00:00.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.935+0000 7f5efa7fc700 1 -- 172.21.15.32:0/1218651292 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f000102e0 con 0x7f5f04101d70 2024-04-12T06:00:00.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.935+0000 7f5f0c0d2700 1 -- 172.21.15.32:0/1218651292 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f04118060 con 0x7f5f04101d70 2024-04-12T06:00:00.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.936+0000 7f5f0c0d2700 1 -- 172.21.15.32:0/1218651292 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f04104240 con 0x7f5f04101d70 2024-04-12T06:00:00.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.936+0000 7f5efa7fc700 1 -- 172.21.15.32:0/1218651292 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f5f00011380 con 0x7f5f04101d70 2024-04-12T06:00:00.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.936+0000 7f5efa7fc700 1 --2- 172.21.15.32:0/1218651292 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5ef006cc20 0x7f5ef006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:00.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.937+0000 7f5efa7fc700 1 -- 172.21.15.32:0/1218651292 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f5f0008c3e0 con 0x7f5f04101d70 2024-04-12T06:00:00.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.937+0000 7f5f0966d700 1 --2- 172.21.15.32:0/1218651292 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5ef006cc20 0x7f5ef006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:00.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.937+0000 7f5f0966d700 1 --2- 172.21.15.32:0/1218651292 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5ef006cc20 0x7f5ef006f0d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4004890 tx=0x7f5ef4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:00:00.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:00.941+0000 7f5efa7fc700 1 -- 172.21.15.32:0/1218651292 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5f000563c0 con 0x7f5f04101d70 2024-04-12T06:00:01.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.189+0000 7f5f0c0d2700 1 -- 172.21.15.32:0/1218651292 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5f04108dc0 con 0x7f5f04101d70 2024-04-12T06:00:01.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.189+0000 7f5efa7fc700 1 -- 172.21.15.32:0/1218651292 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1835 (secure 0 0 0) 0x7f5f000599e0 con 0x7f5f04101d70 2024-04-12T06:00:01.185 INFO:teuthology.orchestra.run.smithi032.stdout:e33 2024-04-12T06:00:01.185 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T06:00:01.185 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T06:00:01.185 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T06:00:01.185 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:00:01.185 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T06:00:01.185 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T06:00:01.185 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 24 2024-04-12T06:00:01.185 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T06:00:01.185 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T06:00:01.185 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:46:10.668569+0000 2024-04-12T06:00:01.185 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T06:00:01.185 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T06:00:01.185 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T06:00:01.186 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T06:00:01.186 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T06:00:01.186 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T06:00:01.186 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T06:00:01.186 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T06:00:01.186 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T06:00:01.186 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T06:00:01.186 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T06:00:01.186 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T06:00:01.186 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T06:00:01.186 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-12T06:00:01.186 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T06:00:01.187 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T06:00:01.187 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T06:00:01.187 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T06:00:01.187 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T06:00:01.187 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T06:00:01.187 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T06:00:01.187 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T06:00:01.187 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:00:01.187 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:00:01.187 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-12T06:00:01.187 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:00:01.187 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T06:00:01.187 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T06:00:01.187 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{-1:44317} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T06:00:01.187 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T06:00:01.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.192+0000 7f5f0c0d2700 1 -- 172.21.15.32:0/1218651292 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5ef006cc20 msgr2=0x7f5ef006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:01.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.192+0000 7f5f0c0d2700 1 --2- 172.21.15.32:0/1218651292 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5ef006cc20 0x7f5ef006f0d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4004890 tx=0x7f5ef4009e30 comp rx=0 tx=0).stop 2024-04-12T06:00:01.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.192+0000 7f5f0c0d2700 1 -- 172.21.15.32:0/1218651292 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f04101d70 msgr2=0x7f5f0410ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:01.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.192+0000 7f5f0c0d2700 1 --2- 172.21.15.32:0/1218651292 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f04101d70 0x7f5f0410ec20 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5f0000c810 tx=0x7f5f00004470 comp rx=0 tx=0).stop 2024-04-12T06:00:01.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.192+0000 7f5f0c0d2700 1 -- 172.21.15.32:0/1218651292 shutdown_connections 2024-04-12T06:00:01.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.192+0000 7f5f0c0d2700 1 --2- 172.21.15.32:0/1218651292 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5ef006cc20 0x7f5ef006f0d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:01.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.192+0000 7f5f0c0d2700 1 --2- 172.21.15.32:0/1218651292 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f04101d70 0x7f5f0410ec20 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:01.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.192+0000 7f5f0c0d2700 1 --2- 172.21.15.32:0/1218651292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f04102710 0x7f5f0410f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:01.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.192+0000 7f5f0c0d2700 1 -- 172.21.15.32:0/1218651292 >> 172.21.15.32:0/1218651292 conn(0x7f5f04072bc0 msgr2=0x7f5f040fb1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:00:01.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.192+0000 7f5f0c0d2700 1 -- 172.21.15.32:0/1218651292 shutdown_connections 2024-04-12T06:00:01.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.192+0000 7f5f0c0d2700 1 -- 172.21.15.32:0/1218651292 wait complete. 2024-04-12T06:00:01.189 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-12T06:00:01.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:00 smithi067 ceph-mon[112979]: from='client.35096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:00:01.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:00 smithi067 ceph-mon[112979]: overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T06:00:01.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:00 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/243601772' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:00:01.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.295+0000 7f2447655700 1 -- 172.21.15.32:0/983931638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2440070320 msgr2=0x7f24400706f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:01.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.295+0000 7f2447655700 1 --2- 172.21.15.32:0/983931638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2440070320 0x7f24400706f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f2434004940 tx=0x7f2434009e30 comp rx=0 tx=0).stop 2024-04-12T06:00:01.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.296+0000 7f2447655700 1 -- 172.21.15.32:0/983931638 shutdown_connections 2024-04-12T06:00:01.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.296+0000 7f2447655700 1 --2- 172.21.15.32:0/983931638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2440070c30 0x7f2440109160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:01.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.296+0000 7f2447655700 1 --2- 172.21.15.32:0/983931638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2440070320 0x7f24400706f0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:01.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.296+0000 7f2447655700 1 -- 172.21.15.32:0/983931638 >> 172.21.15.32:0/983931638 conn(0x7f24400f8980 msgr2=0x7f24400fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:00:01.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.296+0000 7f2447655700 1 -- 172.21.15.32:0/983931638 shutdown_connections 2024-04-12T06:00:01.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.296+0000 7f2447655700 1 -- 172.21.15.32:0/983931638 wait complete. 2024-04-12T06:00:01.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.296+0000 7f2447655700 1 Processor -- start 2024-04-12T06:00:01.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.296+0000 7f2447655700 1 -- start start 2024-04-12T06:00:01.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.296+0000 7f2447655700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2440070320 0x7f2440115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:01.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.296+0000 7f2447655700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2440070c30 0x7f24401161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:01.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.296+0000 7f2447655700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24401168a0 con 0x7f2440070320 2024-04-12T06:00:01.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.296+0000 7f2447655700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f244010fbe0 con 0x7f2440070c30 2024-04-12T06:00:01.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.296+0000 7f24453f1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2440070320 0x7f2440115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:01.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.297+0000 7f24453f1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2440070320 0x7f2440115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54914/0 (socket says 172.21.15.32:54914) 2024-04-12T06:00:01.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.297+0000 7f24453f1700 1 -- 172.21.15.32:0/2407220126 learned_addr learned my addr 172.21.15.32:0/2407220126 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:00:01.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.297+0000 7f24453f1700 1 -- 172.21.15.32:0/2407220126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2440070c30 msgr2=0x7f24401161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:01.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.297+0000 7f2444bf0700 1 --2- 172.21.15.32:0/2407220126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2440070c30 0x7f24401161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:01.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.297+0000 7f24453f1700 1 --2- 172.21.15.32:0/2407220126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2440070c30 0x7f24401161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:01.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.297+0000 7f24453f1700 1 -- 172.21.15.32:0/2407220126 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2434005040 con 0x7f2440070320 2024-04-12T06:00:01.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.297+0000 7f2444bf0700 1 --2- 172.21.15.32:0/2407220126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2440070c30 0x7f24401161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T06:00:01.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.297+0000 7f24453f1700 1 --2- 172.21.15.32:0/2407220126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2440070320 0x7f2440115c80 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f243400b3b0 tx=0x7f2434009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:00:01.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.297+0000 7f2431ffb700 1 -- 172.21.15.32:0/2407220126 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f243400e040 con 0x7f2440070320 2024-04-12T06:00:01.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.297+0000 7f2447655700 1 -- 172.21.15.32:0/2407220126 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f244010fe00 con 0x7f2440070320 2024-04-12T06:00:01.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.297+0000 7f2431ffb700 1 -- 172.21.15.32:0/2407220126 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2434013420 con 0x7f2440070320 2024-04-12T06:00:01.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.297+0000 7f2431ffb700 1 -- 172.21.15.32:0/2407220126 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f243401c630 con 0x7f2440070320 2024-04-12T06:00:01.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.297+0000 7f2447655700 1 -- 172.21.15.32:0/2407220126 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2440110340 con 0x7f2440070320 2024-04-12T06:00:01.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.298+0000 7f2447655700 1 -- 172.21.15.32:0/2407220126 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2440071a60 con 0x7f2440070320 2024-04-12T06:00:01.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.299+0000 7f2431ffb700 1 -- 172.21.15.32:0/2407220126 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f2434013590 con 0x7f2440070320 2024-04-12T06:00:01.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.299+0000 7f2431ffb700 1 --2- 172.21.15.32:0/2407220126 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f242c06cc30 0x7f242c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:01.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.299+0000 7f2431ffb700 1 -- 172.21.15.32:0/2407220126 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f243408d310 con 0x7f2440070320 2024-04-12T06:00:01.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.299+0000 7f2444bf0700 1 --2- 172.21.15.32:0/2407220126 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f242c06cc30 0x7f242c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:01.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.300+0000 7f2444bf0700 1 --2- 172.21.15.32:0/2407220126 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f242c06cc30 0x7f242c06f0e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f243c006fd0 tx=0x7f243c008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:00:01.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.304+0000 7f2431ffb700 1 -- 172.21.15.32:0/2407220126 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f243405ace0 con 0x7f2440070320 2024-04-12T06:00:01.336 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:00 smithi032 ceph-mon[146817]: from='client.35096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:00:01.336 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:00 smithi032 ceph-mon[146817]: overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T06:00:01.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:00 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/243601772' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:00:01.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.527+0000 7f2447655700 1 -- 172.21.15.32:0/2407220126 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f244004bf70 con 0x7f242c06cc30 2024-04-12T06:00:01.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.528+0000 7f2431ffb700 1 -- 172.21.15.32:0/2407220126 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f243c00c660 con 0x7f242c06cc30 2024-04-12T06:00:01.523 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T06:00:01.523 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T06:00:01.523 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T06:00:01.523 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T06:00:01.523 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T06:00:01.523 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-12T06:00:01.523 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T06:00:01.523 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-12T06:00:01.523 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T06:00:01.523 INFO:teuthology.orchestra.run.smithi032.stdout: "mds" 2024-04-12T06:00:01.523 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T06:00:01.523 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T06:00:01.523 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T06:00:01.524 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T06:00:01.524 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T06:00:01.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.530+0000 7f2447655700 1 -- 172.21.15.32:0/2407220126 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f242c06cc30 msgr2=0x7f242c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:01.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.530+0000 7f2447655700 1 --2- 172.21.15.32:0/2407220126 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f242c06cc30 0x7f242c06f0e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f243c006fd0 tx=0x7f243c008040 comp rx=0 tx=0).stop 2024-04-12T06:00:01.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.530+0000 7f2447655700 1 -- 172.21.15.32:0/2407220126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2440070320 msgr2=0x7f2440115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:01.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.530+0000 7f2447655700 1 --2- 172.21.15.32:0/2407220126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2440070320 0x7f2440115c80 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f243400b3b0 tx=0x7f2434009e30 comp rx=0 tx=0).stop 2024-04-12T06:00:01.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.530+0000 7f2447655700 1 -- 172.21.15.32:0/2407220126 shutdown_connections 2024-04-12T06:00:01.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.530+0000 7f2447655700 1 --2- 172.21.15.32:0/2407220126 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f242c06cc30 0x7f242c06f0e0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:01.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.530+0000 7f2447655700 1 --2- 172.21.15.32:0/2407220126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2440070320 0x7f2440115c80 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:01.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.530+0000 7f2447655700 1 --2- 172.21.15.32:0/2407220126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2440070c30 0x7f24401161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:01.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.530+0000 7f2447655700 1 -- 172.21.15.32:0/2407220126 >> 172.21.15.32:0/2407220126 conn(0x7f24400f8980 msgr2=0x7f24400f9ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:00:01.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.530+0000 7f2447655700 1 -- 172.21.15.32:0/2407220126 shutdown_connections 2024-04-12T06:00:01.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.530+0000 7f2447655700 1 -- 172.21.15.32:0/2407220126 wait complete. 2024-04-12T06:00:01.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.635+0000 7fac69006700 1 -- 172.21.15.32:0/2774672676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac640fde90 msgr2=0x7fac640fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:01.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.635+0000 7fac69006700 1 --2- 172.21.15.32:0/2774672676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac640fde90 0x7fac640fe260 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fac54004940 tx=0x7fac54009e30 comp rx=0 tx=0).stop 2024-04-12T06:00:01.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.635+0000 7fac69006700 1 -- 172.21.15.32:0/2774672676 shutdown_connections 2024-04-12T06:00:01.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.635+0000 7fac69006700 1 --2- 172.21.15.32:0/2774672676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac640fe830 0x7fac64106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:01.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.635+0000 7fac69006700 1 --2- 172.21.15.32:0/2774672676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac640fde90 0x7fac640fe260 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:01.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.635+0000 7fac69006700 1 -- 172.21.15.32:0/2774672676 >> 172.21.15.32:0/2774672676 conn(0x7fac64072bc0 msgr2=0x7fac64072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:00:01.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.635+0000 7fac69006700 1 -- 172.21.15.32:0/2774672676 shutdown_connections 2024-04-12T06:00:01.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.635+0000 7fac69006700 1 -- 172.21.15.32:0/2774672676 wait complete. 2024-04-12T06:00:01.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.636+0000 7fac69006700 1 Processor -- start 2024-04-12T06:00:01.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.636+0000 7fac69006700 1 -- start start 2024-04-12T06:00:01.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.636+0000 7fac69006700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac640fde90 0x7fac640fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:01.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.636+0000 7fac69006700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac640fe830 0x7fac640fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:01.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.636+0000 7fac69006700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac640fc820 con 0x7fac640fe830 2024-04-12T06:00:01.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.636+0000 7fac69006700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac640fc960 con 0x7fac640fde90 2024-04-12T06:00:01.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.636+0000 7fac6259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac640fe830 0x7fac640fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:01.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.636+0000 7fac6259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac640fe830 0x7fac640fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51334/0 (socket says 172.21.15.32:51334) 2024-04-12T06:00:01.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.636+0000 7fac6259c700 1 -- 172.21.15.32:0/1672206481 learned_addr learned my addr 172.21.15.32:0/1672206481 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:00:01.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.636+0000 7fac62d9d700 1 --2- 172.21.15.32:0/1672206481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac640fde90 0x7fac640fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:01.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.636+0000 7fac6259c700 1 -- 172.21.15.32:0/1672206481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac640fde90 msgr2=0x7fac640fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:01.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.636+0000 7fac6259c700 1 --2- 172.21.15.32:0/1672206481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac640fde90 0x7fac640fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:01.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.636+0000 7fac6259c700 1 -- 172.21.15.32:0/1672206481 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac54005040 con 0x7fac640fe830 2024-04-12T06:00:01.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.636+0000 7fac62d9d700 1 --2- 172.21.15.32:0/1672206481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac640fde90 0x7fac640fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T06:00:01.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.637+0000 7fac6259c700 1 --2- 172.21.15.32:0/1672206481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac640fe830 0x7fac640fc290 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fac4c002a50 tx=0x7fac4c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:00:01.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.637+0000 7fac5b7fe700 1 -- 172.21.15.32:0/1672206481 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac4c007cc0 con 0x7fac640fe830 2024-04-12T06:00:01.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.637+0000 7fac5b7fe700 1 -- 172.21.15.32:0/1672206481 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac4c00f040 con 0x7fac640fe830 2024-04-12T06:00:01.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.637+0000 7fac69006700 1 -- 172.21.15.32:0/1672206481 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac64119df0 con 0x7fac640fe830 2024-04-12T06:00:01.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.637+0000 7fac5b7fe700 1 -- 172.21.15.32:0/1672206481 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac4c009540 con 0x7fac640fe830 2024-04-12T06:00:01.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.637+0000 7fac69006700 1 -- 172.21.15.32:0/1672206481 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac6411a1f0 con 0x7fac640fe830 2024-04-12T06:00:01.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.638+0000 7fac69006700 1 -- 172.21.15.32:0/1672206481 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac64065e30 con 0x7fac640fe830 2024-04-12T06:00:01.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.638+0000 7fac5b7fe700 1 -- 172.21.15.32:0/1672206481 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fac4c0096a0 con 0x7fac640fe830 2024-04-12T06:00:01.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.639+0000 7fac5b7fe700 1 --2- 172.21.15.32:0/1672206481 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fac5006cc20 0x7fac5006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:01.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.639+0000 7fac5b7fe700 1 -- 172.21.15.32:0/1672206481 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fac4c08bf60 con 0x7fac640fe830 2024-04-12T06:00:01.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.639+0000 7fac62d9d700 1 --2- 172.21.15.32:0/1672206481 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fac5006cc20 0x7fac5006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:01.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.639+0000 7fac62d9d700 1 --2- 172.21.15.32:0/1672206481 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fac5006cc20 0x7fac5006f0d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fac5400b3b0 tx=0x7fac54009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:00:01.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.643+0000 7fac5b7fe700 1 -- 172.21.15.32:0/1672206481 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fac4c055e20 con 0x7fac640fe830 2024-04-12T06:00:01.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.935+0000 7fac69006700 1 -- 172.21.15.32:0/1672206481 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fac6405f5f0 con 0x7fac640fe830 2024-04-12T06:00:01.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.935+0000 7fac5b7fe700 1 -- 172.21.15.32:0/1672206481 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fac4c059440 con 0x7fac640fe830 2024-04-12T06:00:01.930 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T06:00:01.930 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T06:00:01.930 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-12T06:00:01.930 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T06:00:01.931 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T06:00:01.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.937+0000 7fac69006700 1 -- 172.21.15.32:0/1672206481 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fac5006cc20 msgr2=0x7fac5006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:01.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.937+0000 7fac69006700 1 --2- 172.21.15.32:0/1672206481 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fac5006cc20 0x7fac5006f0d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fac5400b3b0 tx=0x7fac54009e30 comp rx=0 tx=0).stop 2024-04-12T06:00:01.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.937+0000 7fac69006700 1 -- 172.21.15.32:0/1672206481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac640fe830 msgr2=0x7fac640fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:01.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.937+0000 7fac69006700 1 --2- 172.21.15.32:0/1672206481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac640fe830 0x7fac640fc290 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fac4c002a50 tx=0x7fac4c0046b0 comp rx=0 tx=0).stop 2024-04-12T06:00:01.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.937+0000 7fac69006700 1 -- 172.21.15.32:0/1672206481 shutdown_connections 2024-04-12T06:00:01.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.937+0000 7fac69006700 1 --2- 172.21.15.32:0/1672206481 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fac5006cc20 0x7fac5006f0d0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:01.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.937+0000 7fac69006700 1 --2- 172.21.15.32:0/1672206481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac640fde90 0x7fac640fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:01.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.937+0000 7fac69006700 1 --2- 172.21.15.32:0/1672206481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac640fe830 0x7fac640fc290 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:01.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.937+0000 7fac69006700 1 -- 172.21.15.32:0/1672206481 >> 172.21.15.32:0/1672206481 conn(0x7fac64072bc0 msgr2=0x7fac640ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:00:01.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.937+0000 7fac69006700 1 -- 172.21.15.32:0/1672206481 shutdown_connections 2024-04-12T06:00:01.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:01.937+0000 7fac69006700 1 -- 172.21.15.32:0/1672206481 wait complete. 2024-04-12T06:00:02.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:01 smithi067 ceph-mon[112979]: pgmap v585: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:02.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:01 smithi067 ceph-mon[112979]: from='client.35100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:00:02.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:01 smithi067 ceph-mon[112979]: from='client.35104 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:00:02.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:01 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1218651292' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T06:00:02.336 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:01 smithi032 ceph-mon[146817]: pgmap v585: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:02.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:01 smithi032 ceph-mon[146817]: from='client.35100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:00:02.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:01 smithi032 ceph-mon[146817]: from='client.35104 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:00:02.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:01 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1218651292' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T06:00:03.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:02 smithi067 ceph-mon[112979]: from='client.35116 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:00:03.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:02 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1672206481' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T06:00:03.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:02 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:00:03.336 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:02 smithi032 ceph-mon[146817]: from='client.35116 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:00:03.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:02 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1672206481' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T06:00:03.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:02 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:00:04.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:03 smithi067 ceph-mon[112979]: pgmap v586: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:04.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:03 smithi032 ceph-mon[146817]: pgmap v586: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:06.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:05 smithi067 ceph-mon[112979]: pgmap v587: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:06.336 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:05 smithi032 ceph-mon[146817]: pgmap v587: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:08.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:07 smithi067 ceph-mon[112979]: pgmap v588: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:08.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:07 smithi032 ceph-mon[146817]: pgmap v588: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:10.265 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:09 smithi067 ceph-mon[112979]: pgmap v589: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:10.336 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:09 smithi032 ceph-mon[146817]: pgmap v589: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:12.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:11 smithi067 ceph-mon[112979]: pgmap v590: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:12.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:11 smithi032 ceph-mon[146817]: pgmap v590: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:14.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:13 smithi067 ceph-mon[112979]: pgmap v591: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:14.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:13 smithi032 ceph-mon[146817]: pgmap v591: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:15.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:14 smithi067 ceph-mon[112979]: pgmap v592: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:15.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:14 smithi032 ceph-mon[146817]: pgmap v592: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:17.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:17 smithi067 ceph-mon[112979]: pgmap v593: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:17.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:17 smithi032 ceph-mon[146817]: pgmap v593: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:18.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:00:18.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:00:19.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:19 smithi067 ceph-mon[112979]: pgmap v594: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:19.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:19 smithi032 ceph-mon[146817]: pgmap v594: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:21.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:21 smithi067 ceph-mon[112979]: pgmap v595: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:21.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:21 smithi032 ceph-mon[146817]: pgmap v595: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:23.393 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:23 smithi032 ceph-mon[146817]: pgmap v596: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:23.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:23 smithi067 ceph-mon[112979]: pgmap v596: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:25.515 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:25 smithi067 ceph-mon[112979]: pgmap v597: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:25.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:25 smithi032 ceph-mon[146817]: pgmap v597: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:27.515 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:27 smithi067 ceph-mon[112979]: pgmap v598: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:27.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:27 smithi032 ceph-mon[146817]: pgmap v598: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:29.515 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:29 smithi067 ceph-mon[112979]: pgmap v599: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:29.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:29 smithi032 ceph-mon[146817]: pgmap v599: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:31.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:31 smithi067 ceph-mon[112979]: pgmap v600: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:31.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:31 smithi032 ceph-mon[146817]: pgmap v600: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:32.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.054+0000 7f07c0553700 1 -- 172.21.15.32:0/1008662084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b8102710 msgr2=0x7f07b80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:32.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.054+0000 7f07c0553700 1 --2- 172.21.15.32:0/1008662084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b8102710 0x7f07b80fbd80 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f07ac004940 tx=0x7f07ac009e30 comp rx=0 tx=0).stop 2024-04-12T06:00:32.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.055+0000 7f07c0553700 1 -- 172.21.15.32:0/1008662084 shutdown_connections 2024-04-12T06:00:32.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.055+0000 7f07c0553700 1 --2- 172.21.15.32:0/1008662084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b8102710 0x7f07b80fbd80 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:32.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.055+0000 7f07c0553700 1 --2- 172.21.15.32:0/1008662084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07b8101d70 0x7f07b8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:32.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.055+0000 7f07c0553700 1 -- 172.21.15.32:0/1008662084 >> 172.21.15.32:0/1008662084 conn(0x7f07b8072bc0 msgr2=0x7f07b8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:00:32.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.055+0000 7f07c0553700 1 -- 172.21.15.32:0/1008662084 shutdown_connections 2024-04-12T06:00:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.055+0000 7f07c0553700 1 -- 172.21.15.32:0/1008662084 wait complete. 2024-04-12T06:00:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.055+0000 7f07c0553700 1 Processor -- start 2024-04-12T06:00:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.055+0000 7f07c0553700 1 -- start start 2024-04-12T06:00:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.055+0000 7f07c0553700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b8101d70 0x7f07b810c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.056+0000 7f07c0553700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07b8102710 0x7f07b810cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.056+0000 7f07c0553700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b810d330 con 0x7f07b8101d70 2024-04-12T06:00:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.056+0000 7f07c0553700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b810d4a0 con 0x7f07b8102710 2024-04-12T06:00:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.056+0000 7f07be2ef700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b8101d70 0x7f07b810c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.056+0000 7f07be2ef700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b8101d70 0x7f07b810c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55568/0 (socket says 172.21.15.32:55568) 2024-04-12T06:00:32.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.056+0000 7f07be2ef700 1 -- 172.21.15.32:0/750794802 learned_addr learned my addr 172.21.15.32:0/750794802 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:00:32.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.056+0000 7f07bdaee700 1 --2- 172.21.15.32:0/750794802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07b8102710 0x7f07b810cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:32.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.056+0000 7f07be2ef700 1 -- 172.21.15.32:0/750794802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07b8102710 msgr2=0x7f07b810cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:32.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.056+0000 7f07be2ef700 1 --2- 172.21.15.32:0/750794802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07b8102710 0x7f07b810cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:32.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.056+0000 7f07be2ef700 1 -- 172.21.15.32:0/750794802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ac005040 con 0x7f07b8101d70 2024-04-12T06:00:32.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.056+0000 7f07be2ef700 1 --2- 172.21.15.32:0/750794802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b8101d70 0x7f07b810c8b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f07b400c810 tx=0x7f07b4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:00:32.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.056+0000 7f07aaffd700 1 -- 172.21.15.32:0/750794802 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07b4012810 con 0x7f07b8101d70 2024-04-12T06:00:32.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.056+0000 7f07aaffd700 1 -- 172.21.15.32:0/750794802 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07b4012e20 con 0x7f07b8101d70 2024-04-12T06:00:32.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.056+0000 7f07c0553700 1 -- 172.21.15.32:0/750794802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b8119f20 con 0x7f07b8101d70 2024-04-12T06:00:32.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.056+0000 7f07aaffd700 1 -- 172.21.15.32:0/750794802 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07b40102e0 con 0x7f07b8101d70 2024-04-12T06:00:32.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.056+0000 7f07c0553700 1 -- 172.21.15.32:0/750794802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b811a270 con 0x7f07b8101d70 2024-04-12T06:00:32.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.057+0000 7f07c0553700 1 -- 172.21.15.32:0/750794802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b8104240 con 0x7f07b8101d70 2024-04-12T06:00:32.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.058+0000 7f07aaffd700 1 -- 172.21.15.32:0/750794802 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f07b4011380 con 0x7f07b8101d70 2024-04-12T06:00:32.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.058+0000 7f07aaffd700 1 --2- 172.21.15.32:0/750794802 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f07a406cc20 0x7f07a406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:32.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.058+0000 7f07aaffd700 1 -- 172.21.15.32:0/750794802 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f07b408c420 con 0x7f07b8101d70 2024-04-12T06:00:32.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.058+0000 7f07bdaee700 1 --2- 172.21.15.32:0/750794802 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f07a406cc20 0x7f07a406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:32.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.059+0000 7f07bdaee700 1 --2- 172.21.15.32:0/750794802 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f07a406cc20 0x7f07a406f0d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f07b80660b0 tx=0x7f07ac009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:00:32.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.063+0000 7f07aaffd700 1 -- 172.21.15.32:0/750794802 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f07b4059e10 con 0x7f07b8101d70 2024-04-12T06:00:32.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.288+0000 7f07c0553700 1 -- 172.21.15.32:0/750794802 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f07b804bed0 con 0x7f07a406cc20 2024-04-12T06:00:32.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.289+0000 7f07aaffd700 1 -- 172.21.15.32:0/750794802 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f07b804bed0 con 0x7f07a406cc20 2024-04-12T06:00:32.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.292+0000 7f07c0553700 1 -- 172.21.15.32:0/750794802 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f07a406cc20 msgr2=0x7f07a406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:32.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.292+0000 7f07c0553700 1 --2- 172.21.15.32:0/750794802 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f07a406cc20 0x7f07a406f0d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f07b80660b0 tx=0x7f07ac009e30 comp rx=0 tx=0).stop 2024-04-12T06:00:32.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.292+0000 7f07c0553700 1 -- 172.21.15.32:0/750794802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b8101d70 msgr2=0x7f07b810c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:32.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.292+0000 7f07c0553700 1 --2- 172.21.15.32:0/750794802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b8101d70 0x7f07b810c8b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f07b400c810 tx=0x7f07b4004470 comp rx=0 tx=0).stop 2024-04-12T06:00:32.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.292+0000 7f07c0553700 1 -- 172.21.15.32:0/750794802 shutdown_connections 2024-04-12T06:00:32.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.292+0000 7f07c0553700 1 --2- 172.21.15.32:0/750794802 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f07a406cc20 0x7f07a406f0d0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:32.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.292+0000 7f07c0553700 1 --2- 172.21.15.32:0/750794802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b8101d70 0x7f07b810c8b0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:32.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.292+0000 7f07c0553700 1 --2- 172.21.15.32:0/750794802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07b8102710 0x7f07b810cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:32.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.292+0000 7f07c0553700 1 -- 172.21.15.32:0/750794802 >> 172.21.15.32:0/750794802 conn(0x7f07b8072bc0 msgr2=0x7f07b80fb1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:00:32.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.292+0000 7f07c0553700 1 -- 172.21.15.32:0/750794802 shutdown_connections 2024-04-12T06:00:32.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.292+0000 7f07c0553700 1 -- 172.21.15.32:0/750794802 wait complete. 2024-04-12T06:00:32.297 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T06:00:32.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.396+0000 7f9aaffaf700 1 -- 172.21.15.32:0/2742847867 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9aa8101d00 msgr2=0x7f9aa81020d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:32.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.396+0000 7f9aaffaf700 1 --2- 172.21.15.32:0/2742847867 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9aa8101d00 0x7f9aa81020d0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f9a98004940 tx=0x7f9a98009e30 comp rx=0 tx=0).stop 2024-04-12T06:00:32.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.396+0000 7f9aaffaf700 1 -- 172.21.15.32:0/2742847867 shutdown_connections 2024-04-12T06:00:32.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.396+0000 7f9aaffaf700 1 --2- 172.21.15.32:0/2742847867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9aa81026a0 0x7f9aa80fbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:32.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.396+0000 7f9aaffaf700 1 --2- 172.21.15.32:0/2742847867 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9aa8101d00 0x7f9aa81020d0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:32.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.396+0000 7f9aaffaf700 1 -- 172.21.15.32:0/2742847867 >> 172.21.15.32:0/2742847867 conn(0x7f9aa80f78d0 msgr2=0x7f9aa80f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:00:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.396+0000 7f9aaffaf700 1 -- 172.21.15.32:0/2742847867 shutdown_connections 2024-04-12T06:00:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.396+0000 7f9aaffaf700 1 -- 172.21.15.32:0/2742847867 wait complete. 2024-04-12T06:00:32.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.397+0000 7f9aaffaf700 1 Processor -- start 2024-04-12T06:00:32.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.397+0000 7f9aaffaf700 1 -- start start 2024-04-12T06:00:32.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.397+0000 7f9aaffaf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9aa8101d00 0x7f9aa8102b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:32.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.397+0000 7f9aaffaf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9aa81026a0 0x7f9aa81030a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:32.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.397+0000 7f9aaffaf700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa8103630 con 0x7f9aa81026a0 2024-04-12T06:00:32.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.397+0000 7f9aaffaf700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa8103770 con 0x7f9aa8101d00 2024-04-12T06:00:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.397+0000 7f9aad54a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9aa81026a0 0x7f9aa81030a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.397+0000 7f9aad54a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9aa81026a0 0x7f9aa81030a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55594/0 (socket says 172.21.15.32:55594) 2024-04-12T06:00:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.397+0000 7f9aad54a700 1 -- 172.21.15.32:0/1595285529 learned_addr learned my addr 172.21.15.32:0/1595285529 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:00:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.397+0000 7f9aadd4b700 1 --2- 172.21.15.32:0/1595285529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9aa8101d00 0x7f9aa8102b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.397+0000 7f9aad54a700 1 -- 172.21.15.32:0/1595285529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9aa8101d00 msgr2=0x7f9aa8102b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.397+0000 7f9aad54a700 1 --2- 172.21.15.32:0/1595285529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9aa8101d00 0x7f9aa8102b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.397+0000 7f9aad54a700 1 -- 172.21.15.32:0/1595285529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a98005040 con 0x7f9aa81026a0 2024-04-12T06:00:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.397+0000 7f9aadd4b700 1 --2- 172.21.15.32:0/1595285529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9aa8101d00 0x7f9aa8102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T06:00:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.398+0000 7f9aad54a700 1 --2- 172.21.15.32:0/1595285529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9aa81026a0 0x7f9aa81030a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f9aa400c890 tx=0x7f9aa40044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:00:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.398+0000 7f9a9e7fc700 1 -- 172.21.15.32:0/1595285529 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9aa400a7d0 con 0x7f9aa81026a0 2024-04-12T06:00:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.398+0000 7f9a9e7fc700 1 -- 172.21.15.32:0/1595285529 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9aa400ade0 con 0x7f9aa81026a0 2024-04-12T06:00:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.398+0000 7f9aaffaf700 1 -- 172.21.15.32:0/1595285529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aa8119df0 con 0x7f9aa81026a0 2024-04-12T06:00:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.398+0000 7f9a9e7fc700 1 -- 172.21.15.32:0/1595285529 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9aa40113a0 con 0x7f9aa81026a0 2024-04-12T06:00:32.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.398+0000 7f9aaffaf700 1 -- 172.21.15.32:0/1595285529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aa811a270 con 0x7f9aa81026a0 2024-04-12T06:00:32.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.399+0000 7f9aaffaf700 1 -- 172.21.15.32:0/1595285529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aa80fd450 con 0x7f9aa81026a0 2024-04-12T06:00:32.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.399+0000 7f9a9e7fc700 1 -- 172.21.15.32:0/1595285529 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9aa4012400 con 0x7f9aa81026a0 2024-04-12T06:00:32.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.400+0000 7f9a9e7fc700 1 --2- 172.21.15.32:0/1595285529 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9a9406cc30 0x7f9a9406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:32.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.400+0000 7f9a9e7fc700 1 -- 172.21.15.32:0/1595285529 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f9aa408c0d0 con 0x7f9aa81026a0 2024-04-12T06:00:32.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.400+0000 7f9aadd4b700 1 --2- 172.21.15.32:0/1595285529 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9a9406cc30 0x7f9a9406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:32.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.400+0000 7f9aadd4b700 1 --2- 172.21.15.32:0/1595285529 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9a9406cc30 0x7f9a9406f0e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9a9800b840 tx=0x7f9a98009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:00:32.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.404+0000 7f9a9e7fc700 1 -- 172.21.15.32:0/1595285529 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9aa4056150 con 0x7f9aa81026a0 2024-04-12T06:00:32.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.626+0000 7f9aaffaf700 1 -- 172.21.15.32:0/1595285529 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9aa804bed0 con 0x7f9a9406cc30 2024-04-12T06:00:32.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.628+0000 7f9a9e7fc700 1 -- 172.21.15.32:0/1595285529 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9aa804bed0 con 0x7f9a9406cc30 2024-04-12T06:00:32.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.629+0000 7f9aaffaf700 1 -- 172.21.15.32:0/1595285529 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9a9406cc30 msgr2=0x7f9a9406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:32.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.629+0000 7f9aaffaf700 1 --2- 172.21.15.32:0/1595285529 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9a9406cc30 0x7f9a9406f0e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9a9800b840 tx=0x7f9a98009e30 comp rx=0 tx=0).stop 2024-04-12T06:00:32.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.630+0000 7f9aaffaf700 1 -- 172.21.15.32:0/1595285529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9aa81026a0 msgr2=0x7f9aa81030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:32.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.630+0000 7f9aaffaf700 1 --2- 172.21.15.32:0/1595285529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9aa81026a0 0x7f9aa81030a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f9aa400c890 tx=0x7f9aa40044c0 comp rx=0 tx=0).stop 2024-04-12T06:00:32.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.630+0000 7f9aaffaf700 1 -- 172.21.15.32:0/1595285529 shutdown_connections 2024-04-12T06:00:32.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.630+0000 7f9aaffaf700 1 --2- 172.21.15.32:0/1595285529 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9a9406cc30 0x7f9a9406f0e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:32.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.630+0000 7f9aaffaf700 1 --2- 172.21.15.32:0/1595285529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9aa8101d00 0x7f9aa8102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:32.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.630+0000 7f9aaffaf700 1 --2- 172.21.15.32:0/1595285529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9aa81026a0 0x7f9aa81030a0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:32.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.630+0000 7f9aaffaf700 1 -- 172.21.15.32:0/1595285529 >> 172.21.15.32:0/1595285529 conn(0x7f9aa80f78d0 msgr2=0x7f9aa80f9070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:00:32.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.630+0000 7f9aaffaf700 1 -- 172.21.15.32:0/1595285529 shutdown_connections 2024-04-12T06:00:32.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.630+0000 7f9aaffaf700 1 -- 172.21.15.32:0/1595285529 wait complete. 2024-04-12T06:00:32.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.732+0000 7f089dacc700 1 -- 172.21.15.32:0/1337312340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0898101d70 msgr2=0x7f0898102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:32.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.733+0000 7f089dacc700 1 --2- 172.21.15.32:0/1337312340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0898101d70 0x7f0898102140 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f088c004940 tx=0x7f088c009e30 comp rx=0 tx=0).stop 2024-04-12T06:00:32.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.733+0000 7f089dacc700 1 -- 172.21.15.32:0/1337312340 shutdown_connections 2024-04-12T06:00:32.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.733+0000 7f089dacc700 1 --2- 172.21.15.32:0/1337312340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0898102710 0x7f08980fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:32.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.733+0000 7f089dacc700 1 --2- 172.21.15.32:0/1337312340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0898101d70 0x7f0898102140 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:32.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.733+0000 7f089dacc700 1 -- 172.21.15.32:0/1337312340 >> 172.21.15.32:0/1337312340 conn(0x7f0898072bc0 msgr2=0x7f0898072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:00:32.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.733+0000 7f089dacc700 1 -- 172.21.15.32:0/1337312340 shutdown_connections 2024-04-12T06:00:32.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.733+0000 7f089dacc700 1 -- 172.21.15.32:0/1337312340 wait complete. 2024-04-12T06:00:32.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.733+0000 7f089dacc700 1 Processor -- start 2024-04-12T06:00:32.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.733+0000 7f089dacc700 1 -- start start 2024-04-12T06:00:32.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.733+0000 7f089dacc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0898101d70 0x7f0898102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:32.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.734+0000 7f089dacc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0898102710 0x7f0898103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:32.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.734+0000 7f089dacc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08981036a0 con 0x7f0898101d70 2024-04-12T06:00:32.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.734+0000 7f089dacc700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08981037e0 con 0x7f0898102710 2024-04-12T06:00:32.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.734+0000 7f08977fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0898101d70 0x7f0898102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:32.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.734+0000 7f08977fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0898101d70 0x7f0898102bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55602/0 (socket says 172.21.15.32:55602) 2024-04-12T06:00:32.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.734+0000 7f08977fe700 1 -- 172.21.15.32:0/2688341514 learned_addr learned my addr 172.21.15.32:0/2688341514 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:00:32.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.734+0000 7f0896ffd700 1 --2- 172.21.15.32:0/2688341514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0898102710 0x7f0898103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:32.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.734+0000 7f08977fe700 1 -- 172.21.15.32:0/2688341514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0898102710 msgr2=0x7f0898103110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:32.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.734+0000 7f08977fe700 1 --2- 172.21.15.32:0/2688341514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0898102710 0x7f0898103110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:32.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.734+0000 7f08977fe700 1 -- 172.21.15.32:0/2688341514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f088c005040 con 0x7f0898101d70 2024-04-12T06:00:32.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.734+0000 7f08977fe700 1 --2- 172.21.15.32:0/2688341514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0898101d70 0x7f0898102bd0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f088c000c00 tx=0x7f088c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:00:32.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.734+0000 7f087ffff700 1 -- 172.21.15.32:0/2688341514 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f088c021070 con 0x7f0898101d70 2024-04-12T06:00:32.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.734+0000 7f087ffff700 1 -- 172.21.15.32:0/2688341514 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f088c0093b0 con 0x7f0898101d70 2024-04-12T06:00:32.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.734+0000 7f089dacc700 1 -- 172.21.15.32:0/2688341514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0898103a30 con 0x7f0898101d70 2024-04-12T06:00:32.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.734+0000 7f087ffff700 1 -- 172.21.15.32:0/2688341514 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f088c00e070 con 0x7f0898101d70 2024-04-12T06:00:32.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.734+0000 7f089dacc700 1 -- 172.21.15.32:0/2688341514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f089811a140 con 0x7f0898101d70 2024-04-12T06:00:32.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.735+0000 7f089dacc700 1 -- 172.21.15.32:0/2688341514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08980fd4c0 con 0x7f0898101d70 2024-04-12T06:00:32.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.736+0000 7f087ffff700 1 -- 172.21.15.32:0/2688341514 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f088c01c490 con 0x7f0898101d70 2024-04-12T06:00:32.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.737+0000 7f087ffff700 1 --2- 172.21.15.32:0/2688341514 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f088006cc20 0x7f088006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:32.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.737+0000 7f087ffff700 1 -- 172.21.15.32:0/2688341514 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f088c08c910 con 0x7f0898101d70 2024-04-12T06:00:32.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.737+0000 7f0896ffd700 1 --2- 172.21.15.32:0/2688341514 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f088006cc20 0x7f088006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:32.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.737+0000 7f0896ffd700 1 --2- 172.21.15.32:0/2688341514 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f088006cc20 0x7f088006f0d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0898104510 tx=0x7f088800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:00:32.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.741+0000 7f087ffff700 1 -- 172.21.15.32:0/2688341514 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f088c0568e0 con 0x7f0898101d70 2024-04-12T06:00:32.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.956+0000 7f089dacc700 1 -- 172.21.15.32:0/2688341514 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f08981042c0 con 0x7f088006cc20 2024-04-12T06:00:32.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.966+0000 7f087ffff700 1 -- 172.21.15.32:0/2688341514 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f08981042c0 con 0x7f088006cc20 2024-04-12T06:00:32.961 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T06:00:32.961 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (20m) 13m ago 31m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T06:00:32.961 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (17m) 13m ago 32m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T06:00:32.961 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (17m) 13m ago 30m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T06:00:32.961 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (20m) 13m ago 31m 87.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T06:00:32.961 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (14m) 13m ago 26m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T06:00:32.961 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (13m) 13m ago 25m 13.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T06:00:32.961 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (13m) 13m ago 25m 11.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 6d4aca6f74a0 2024-04-12T06:00:32.961 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (13m) 13m ago 26m 14.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T06:00:32.961 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (22m) 13m ago 33m 507M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T06:00:32.961 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (21m) 13m ago 30m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T06:00:32.961 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (18m) 13m ago 33m 53.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T06:00:32.962 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (18m) 13m ago 30m 43.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T06:00:32.962 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (21m) 13m ago 31m 16.9M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T06:00:32.962 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (20m) 13m ago 30m 18.4M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T06:00:32.962 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (17m) 13m ago 29m 91.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T06:00:32.962 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (16m) 13m ago 29m 97.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T06:00:32.962 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (16m) 13m ago 29m 81.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T06:00:32.962 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (15m) 13m ago 28m 113M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T06:00:32.962 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (15m) 13m ago 28m 90.3M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T06:00:32.962 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (14m) 13m ago 27m 84.8M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T06:00:32.962 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (20m) 13m ago 31m 64.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T06:00:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.968+0000 7f089dacc700 1 -- 172.21.15.32:0/2688341514 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f088006cc20 msgr2=0x7f088006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.968+0000 7f089dacc700 1 --2- 172.21.15.32:0/2688341514 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f088006cc20 0x7f088006f0d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0898104510 tx=0x7f088800b040 comp rx=0 tx=0).stop 2024-04-12T06:00:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.968+0000 7f089dacc700 1 -- 172.21.15.32:0/2688341514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0898101d70 msgr2=0x7f0898102bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.968+0000 7f089dacc700 1 --2- 172.21.15.32:0/2688341514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0898101d70 0x7f0898102bd0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f088c000c00 tx=0x7f088c009e30 comp rx=0 tx=0).stop 2024-04-12T06:00:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.968+0000 7f089dacc700 1 -- 172.21.15.32:0/2688341514 shutdown_connections 2024-04-12T06:00:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.968+0000 7f089dacc700 1 --2- 172.21.15.32:0/2688341514 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f088006cc20 0x7f088006f0d0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.968+0000 7f089dacc700 1 --2- 172.21.15.32:0/2688341514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0898101d70 0x7f0898102bd0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.968+0000 7f089dacc700 1 --2- 172.21.15.32:0/2688341514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0898102710 0x7f0898103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.968+0000 7f089dacc700 1 -- 172.21.15.32:0/2688341514 >> 172.21.15.32:0/2688341514 conn(0x7f0898072bc0 msgr2=0x7f0898065970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:00:32.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.968+0000 7f089dacc700 1 -- 172.21.15.32:0/2688341514 shutdown_connections 2024-04-12T06:00:32.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:32.968+0000 7f089dacc700 1 -- 172.21.15.32:0/2688341514 wait complete. 2024-04-12T06:00:33.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.070+0000 7f416d527700 1 -- 172.21.15.32:0/4222647582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4168102e00 msgr2=0x7f41681031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:33.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.070+0000 7f416d527700 1 --2- 172.21.15.32:0/4222647582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4168102e00 0x7f41681031d0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f4158009ce0 tx=0x7f4158004d10 comp rx=0 tx=0).stop 2024-04-12T06:00:33.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.070+0000 7f416d527700 1 -- 172.21.15.32:0/4222647582 shutdown_connections 2024-04-12T06:00:33.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.070+0000 7f416d527700 1 --2- 172.21.15.32:0/4222647582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41681037a0 0x7f41680fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:33.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.070+0000 7f416d527700 1 --2- 172.21.15.32:0/4222647582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4168102e00 0x7f41681031d0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:33.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.070+0000 7f416d527700 1 -- 172.21.15.32:0/4222647582 >> 172.21.15.32:0/4222647582 conn(0x7f41680f89b0 msgr2=0x7f41680fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:00:33.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.070+0000 7f416d527700 1 -- 172.21.15.32:0/4222647582 shutdown_connections 2024-04-12T06:00:33.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.070+0000 7f416d527700 1 -- 172.21.15.32:0/4222647582 wait complete. 2024-04-12T06:00:33.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.071+0000 7f416d527700 1 Processor -- start 2024-04-12T06:00:33.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.071+0000 7f416d527700 1 -- start start 2024-04-12T06:00:33.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.071+0000 7f416d527700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4168102e00 0x7f4168113a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:33.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.071+0000 7f416d527700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41681037a0 0x7f4168113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:33.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.071+0000 7f416d527700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41681145a0 con 0x7f4168102e00 2024-04-12T06:00:33.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.071+0000 7f416d527700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416810dba0 con 0x7f41681037a0 2024-04-12T06:00:33.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.071+0000 7f4166ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4168102e00 0x7f4168113a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:33.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.072+0000 7f4166ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4168102e00 0x7f4168113a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55610/0 (socket says 172.21.15.32:55610) 2024-04-12T06:00:33.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.072+0000 7f4166ffd700 1 -- 172.21.15.32:0/3582744504 learned_addr learned my addr 172.21.15.32:0/3582744504 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:00:33.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.072+0000 7f41667fc700 1 --2- 172.21.15.32:0/3582744504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41681037a0 0x7f4168113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:33.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.072+0000 7f4166ffd700 1 -- 172.21.15.32:0/3582744504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41681037a0 msgr2=0x7f4168113f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:33.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.072+0000 7f4166ffd700 1 --2- 172.21.15.32:0/3582744504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41681037a0 0x7f4168113f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:33.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.072+0000 7f4166ffd700 1 -- 172.21.15.32:0/3582744504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4158009940 con 0x7f4168102e00 2024-04-12T06:00:33.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.072+0000 7f4166ffd700 1 --2- 172.21.15.32:0/3582744504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4168102e00 0x7f4168113a10 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f4158003400 tx=0x7f4158004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:00:33.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.072+0000 7f414f7fe700 1 -- 172.21.15.32:0/3582744504 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f415800e040 con 0x7f4168102e00 2024-04-12T06:00:33.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.072+0000 7f414f7fe700 1 -- 172.21.15.32:0/3582744504 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4158004610 con 0x7f4168102e00 2024-04-12T06:00:33.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.072+0000 7f414f7fe700 1 -- 172.21.15.32:0/3582744504 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f415801c640 con 0x7f4168102e00 2024-04-12T06:00:33.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.072+0000 7f416d527700 1 -- 172.21.15.32:0/3582744504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f416810de20 con 0x7f4168102e00 2024-04-12T06:00:33.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.072+0000 7f416d527700 1 -- 172.21.15.32:0/3582744504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f416810e3e0 con 0x7f4168102e00 2024-04-12T06:00:33.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.073+0000 7f416d527700 1 -- 172.21.15.32:0/3582744504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41680fe550 con 0x7f4168102e00 2024-04-12T06:00:33.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.074+0000 7f414f7fe700 1 -- 172.21.15.32:0/3582744504 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4158012410 con 0x7f4168102e00 2024-04-12T06:00:33.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.074+0000 7f414f7fe700 1 --2- 172.21.15.32:0/3582744504 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f415006cc20 0x7f415006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:33.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.074+0000 7f414f7fe700 1 -- 172.21.15.32:0/3582744504 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f415808da90 con 0x7f4168102e00 2024-04-12T06:00:33.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.074+0000 7f41667fc700 1 --2- 172.21.15.32:0/3582744504 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f415006cc20 0x7f415006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:33.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.075+0000 7f41667fc700 1 --2- 172.21.15.32:0/3582744504 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f415006cc20 0x7f415006f0d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f416810f240 tx=0x7f415c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:00:33.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.079+0000 7f414f7fe700 1 -- 172.21.15.32:0/3582744504 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f415805b500 con 0x7f4168102e00 2024-04-12T06:00:33.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.365+0000 7f416d527700 1 -- 172.21.15.32:0/3582744504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f416810eff0 con 0x7f4168102e00 2024-04-12T06:00:33.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.366+0000 7f414f7fe700 1 -- 172.21.15.32:0/3582744504 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f4158004e10 con 0x7f4168102e00 2024-04-12T06:00:33.361 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T06:00:33.361 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T06:00:33.361 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T06:00:33.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T06:00:33.361 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T06:00:33.361 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T06:00:33.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T06:00:33.361 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T06:00:33.361 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T06:00:33.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T06:00:33.362 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T06:00:33.362 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T06:00:33.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T06:00:33.362 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T06:00:33.362 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T06:00:33.362 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T06:00:33.362 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T06:00:33.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.368+0000 7f416d527700 1 -- 172.21.15.32:0/3582744504 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f415006cc20 msgr2=0x7f415006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:33.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.368+0000 7f416d527700 1 --2- 172.21.15.32:0/3582744504 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f415006cc20 0x7f415006f0d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f416810f240 tx=0x7f415c00b040 comp rx=0 tx=0).stop 2024-04-12T06:00:33.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.368+0000 7f416d527700 1 -- 172.21.15.32:0/3582744504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4168102e00 msgr2=0x7f4168113a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:33.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.368+0000 7f416d527700 1 --2- 172.21.15.32:0/3582744504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4168102e00 0x7f4168113a10 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f4158003400 tx=0x7f4158004d10 comp rx=0 tx=0).stop 2024-04-12T06:00:33.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.368+0000 7f416d527700 1 -- 172.21.15.32:0/3582744504 shutdown_connections 2024-04-12T06:00:33.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.368+0000 7f416d527700 1 --2- 172.21.15.32:0/3582744504 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f415006cc20 0x7f415006f0d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:33.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.368+0000 7f416d527700 1 --2- 172.21.15.32:0/3582744504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4168102e00 0x7f4168113a10 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:33.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.368+0000 7f416d527700 1 --2- 172.21.15.32:0/3582744504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41681037a0 0x7f4168113f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:33.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.368+0000 7f416d527700 1 -- 172.21.15.32:0/3582744504 >> 172.21.15.32:0/3582744504 conn(0x7f41680f89b0 msgr2=0x7f41680f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:00:33.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.368+0000 7f416d527700 1 -- 172.21.15.32:0/3582744504 shutdown_connections 2024-04-12T06:00:33.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.368+0000 7f416d527700 1 -- 172.21.15.32:0/3582744504 wait complete. 2024-04-12T06:00:33.393 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:33 smithi032 ceph-mon[146817]: pgmap v601: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:33.393 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:33 smithi032 ceph-mon[146817]: from='client.35124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:00:33.393 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:33 smithi032 ceph-mon[146817]: from='client.35128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:00:33.393 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:33 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:00:33.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.472+0000 7f35a495c700 1 -- 172.21.15.32:0/2890718589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35a01037a0 msgr2=0x7f35a00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:33.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.472+0000 7f35a495c700 1 --2- 172.21.15.32:0/2890718589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35a01037a0 0x7f35a00fce10 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f3594004940 tx=0x7f3594009e30 comp rx=0 tx=0).stop 2024-04-12T06:00:33.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.472+0000 7f35a495c700 1 -- 172.21.15.32:0/2890718589 shutdown_connections 2024-04-12T06:00:33.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.472+0000 7f35a495c700 1 --2- 172.21.15.32:0/2890718589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35a01037a0 0x7f35a00fce10 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:33.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.472+0000 7f35a495c700 1 --2- 172.21.15.32:0/2890718589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35a0102e00 0x7f35a01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:33.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.472+0000 7f35a495c700 1 -- 172.21.15.32:0/2890718589 >> 172.21.15.32:0/2890718589 conn(0x7f35a00f89b0 msgr2=0x7f35a00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:00:33.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.472+0000 7f35a495c700 1 -- 172.21.15.32:0/2890718589 shutdown_connections 2024-04-12T06:00:33.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.472+0000 7f35a495c700 1 -- 172.21.15.32:0/2890718589 wait complete. 2024-04-12T06:00:33.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.472+0000 7f35a495c700 1 Processor -- start 2024-04-12T06:00:33.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.472+0000 7f35a495c700 1 -- start start 2024-04-12T06:00:33.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.473+0000 7f35a495c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35a0102e00 0x7f35a0113810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:33.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.473+0000 7f35a495c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35a01037a0 0x7f35a0113d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:33.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.473+0000 7f35a495c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a01143a0 con 0x7f35a01037a0 2024-04-12T06:00:33.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.473+0000 7f35a495c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a010d9a0 con 0x7f35a0102e00 2024-04-12T06:00:33.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.473+0000 7f359dd9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35a01037a0 0x7f35a0113d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:33.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.473+0000 7f359dd9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35a01037a0 0x7f35a0113d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55626/0 (socket says 172.21.15.32:55626) 2024-04-12T06:00:33.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.473+0000 7f359dd9b700 1 -- 172.21.15.32:0/3327957276 learned_addr learned my addr 172.21.15.32:0/3327957276 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:00:33.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.473+0000 7f359e59c700 1 --2- 172.21.15.32:0/3327957276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35a0102e00 0x7f35a0113810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:33.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.473+0000 7f359dd9b700 1 -- 172.21.15.32:0/3327957276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35a0102e00 msgr2=0x7f35a0113810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:33.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.473+0000 7f359dd9b700 1 --2- 172.21.15.32:0/3327957276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35a0102e00 0x7f35a0113810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:33.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.473+0000 7f359dd9b700 1 -- 172.21.15.32:0/3327957276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3594005040 con 0x7f35a01037a0 2024-04-12T06:00:33.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.473+0000 7f359dd9b700 1 --2- 172.21.15.32:0/3327957276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35a01037a0 0x7f35a0113d50 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f3594000c00 tx=0x7f3594009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:00:33.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.473+0000 7f358effd700 1 -- 172.21.15.32:0/3327957276 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f359400e070 con 0x7f35a01037a0 2024-04-12T06:00:33.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.473+0000 7f358effd700 1 -- 172.21.15.32:0/3327957276 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f35940093b0 con 0x7f35a01037a0 2024-04-12T06:00:33.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.473+0000 7f358effd700 1 -- 172.21.15.32:0/3327957276 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f359401ccb0 con 0x7f35a01037a0 2024-04-12T06:00:33.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.473+0000 7f35a495c700 1 -- 172.21.15.32:0/3327957276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a010dc20 con 0x7f35a01037a0 2024-04-12T06:00:33.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.473+0000 7f35a495c700 1 -- 172.21.15.32:0/3327957276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a010e130 con 0x7f35a01037a0 2024-04-12T06:00:33.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.475+0000 7f35a495c700 1 -- 172.21.15.32:0/3327957276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a00fe550 con 0x7f35a01037a0 2024-04-12T06:00:33.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.475+0000 7f358effd700 1 -- 172.21.15.32:0/3327957276 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f359401c490 con 0x7f35a01037a0 2024-04-12T06:00:33.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.476+0000 7f358effd700 1 --2- 172.21.15.32:0/3327957276 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f358806cc20 0x7f358806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:33.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.476+0000 7f359e59c700 1 --2- 172.21.15.32:0/3327957276 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f358806cc20 0x7f358806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:33.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.476+0000 7f358effd700 1 -- 172.21.15.32:0/3327957276 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f359408d550 con 0x7f35a01037a0 2024-04-12T06:00:33.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.476+0000 7f359e59c700 1 --2- 172.21.15.32:0/3327957276 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f358806cc20 0x7f358806f0d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3590009b80 tx=0x7f359000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:00:33.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.480+0000 7f358effd700 1 -- 172.21.15.32:0/3327957276 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f359405afc0 con 0x7f35a01037a0 2024-04-12T06:00:33.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:33 smithi067 ceph-mon[112979]: pgmap v601: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:33.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:33 smithi067 ceph-mon[112979]: from='client.35124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:00:33.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:33 smithi067 ceph-mon[112979]: from='client.35128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:00:33.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:33 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:00:33.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.733+0000 7f35a495c700 1 -- 172.21.15.32:0/3327957276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f35a0109da0 con 0x7f35a01037a0 2024-04-12T06:00:33.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.733+0000 7f358effd700 1 -- 172.21.15.32:0/3327957276 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1835 (secure 0 0 0) 0x7f359405ab50 con 0x7f35a01037a0 2024-04-12T06:00:33.729 INFO:teuthology.orchestra.run.smithi032.stdout:e33 2024-04-12T06:00:33.729 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T06:00:33.729 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T06:00:33.729 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T06:00:33.729 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:00:33.729 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T06:00:33.729 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T06:00:33.729 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 24 2024-04-12T06:00:33.729 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T06:00:33.729 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T06:00:33.729 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:46:10.668569+0000 2024-04-12T06:00:33.729 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T06:00:33.729 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T06:00:33.729 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T06:00:33.730 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T06:00:33.730 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T06:00:33.730 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T06:00:33.730 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T06:00:33.730 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T06:00:33.730 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T06:00:33.730 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T06:00:33.730 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T06:00:33.730 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T06:00:33.730 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T06:00:33.730 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-12T06:00:33.730 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T06:00:33.730 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T06:00:33.730 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T06:00:33.731 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T06:00:33.731 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T06:00:33.731 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T06:00:33.731 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T06:00:33.731 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T06:00:33.731 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:00:33.731 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:00:33.731 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-12T06:00:33.731 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:00:33.731 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T06:00:33.731 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T06:00:33.731 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{-1:44317} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T06:00:33.731 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T06:00:33.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.735+0000 7f35a495c700 1 -- 172.21.15.32:0/3327957276 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f358806cc20 msgr2=0x7f358806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:33.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.735+0000 7f35a495c700 1 --2- 172.21.15.32:0/3327957276 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f358806cc20 0x7f358806f0d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3590009b80 tx=0x7f359000b040 comp rx=0 tx=0).stop 2024-04-12T06:00:33.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.735+0000 7f35a495c700 1 -- 172.21.15.32:0/3327957276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35a01037a0 msgr2=0x7f35a0113d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:33.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.735+0000 7f35a495c700 1 --2- 172.21.15.32:0/3327957276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35a01037a0 0x7f35a0113d50 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f3594000c00 tx=0x7f3594009e30 comp rx=0 tx=0).stop 2024-04-12T06:00:33.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.735+0000 7f35a495c700 1 -- 172.21.15.32:0/3327957276 shutdown_connections 2024-04-12T06:00:33.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.735+0000 7f35a495c700 1 --2- 172.21.15.32:0/3327957276 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f358806cc20 0x7f358806f0d0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:33.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.735+0000 7f35a495c700 1 --2- 172.21.15.32:0/3327957276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35a0102e00 0x7f35a0113810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:33.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.736+0000 7f35a495c700 1 --2- 172.21.15.32:0/3327957276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35a01037a0 0x7f35a0113d50 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:33.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.736+0000 7f35a495c700 1 -- 172.21.15.32:0/3327957276 >> 172.21.15.32:0/3327957276 conn(0x7f35a00f89b0 msgr2=0x7f35a00f9990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:00:33.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.736+0000 7f35a495c700 1 -- 172.21.15.32:0/3327957276 shutdown_connections 2024-04-12T06:00:33.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.736+0000 7f35a495c700 1 -- 172.21.15.32:0/3327957276 wait complete. 2024-04-12T06:00:33.733 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-12T06:00:33.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.843+0000 7f25df444700 1 -- 172.21.15.32:0/217352341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25d8102e10 msgr2=0x7f25d81031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:33.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.843+0000 7f25df444700 1 --2- 172.21.15.32:0/217352341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25d8102e10 0x7f25d81031e0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f25d4004940 tx=0x7f25d4009e30 comp rx=0 tx=0).stop 2024-04-12T06:00:33.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.843+0000 7f25df444700 1 -- 172.21.15.32:0/217352341 shutdown_connections 2024-04-12T06:00:33.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.843+0000 7f25df444700 1 --2- 172.21.15.32:0/217352341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25d81037b0 0x7f25d80fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:33.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.843+0000 7f25df444700 1 --2- 172.21.15.32:0/217352341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25d8102e10 0x7f25d81031e0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:33.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.843+0000 7f25df444700 1 -- 172.21.15.32:0/217352341 >> 172.21.15.32:0/217352341 conn(0x7f25d80f8980 msgr2=0x7f25d80fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:00:33.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.843+0000 7f25df444700 1 -- 172.21.15.32:0/217352341 shutdown_connections 2024-04-12T06:00:33.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.843+0000 7f25df444700 1 -- 172.21.15.32:0/217352341 wait complete. 2024-04-12T06:00:33.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.844+0000 7f25df444700 1 Processor -- start 2024-04-12T06:00:33.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.844+0000 7f25df444700 1 -- start start 2024-04-12T06:00:33.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.844+0000 7f25df444700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25d8102e10 0x7f25d80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:33.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.844+0000 7f25df444700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25d81037b0 0x7f25d8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:33.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.844+0000 7f25df444700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d8070d90 con 0x7f25d8102e10 2024-04-12T06:00:33.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.844+0000 7f25df444700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d8070ed0 con 0x7f25d81037b0 2024-04-12T06:00:33.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.844+0000 7f25dd1e0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25d8102e10 0x7f25d80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:33.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.844+0000 7f25dd1e0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25d8102e10 0x7f25d80702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55640/0 (socket says 172.21.15.32:55640) 2024-04-12T06:00:33.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.844+0000 7f25dd1e0700 1 -- 172.21.15.32:0/3974351927 learned_addr learned my addr 172.21.15.32:0/3974351927 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:00:33.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.844+0000 7f25dc9df700 1 --2- 172.21.15.32:0/3974351927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25d81037b0 0x7f25d8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:33.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.844+0000 7f25dd1e0700 1 -- 172.21.15.32:0/3974351927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25d81037b0 msgr2=0x7f25d8070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:33.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.844+0000 7f25dd1e0700 1 --2- 172.21.15.32:0/3974351927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25d81037b0 0x7f25d8070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:33.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.844+0000 7f25dd1e0700 1 -- 172.21.15.32:0/3974351927 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d4005040 con 0x7f25d8102e10 2024-04-12T06:00:33.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.844+0000 7f25dc9df700 1 --2- 172.21.15.32:0/3974351927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25d81037b0 0x7f25d8070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T06:00:33.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.845+0000 7f25dd1e0700 1 --2- 172.21.15.32:0/3974351927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25d8102e10 0x7f25d80702c0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f25d4009360 tx=0x7f25d4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:00:33.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.845+0000 7f25cdffb700 1 -- 172.21.15.32:0/3974351927 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25d400e040 con 0x7f25d8102e10 2024-04-12T06:00:33.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.845+0000 7f25cdffb700 1 -- 172.21.15.32:0/3974351927 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f25d4013420 con 0x7f25d8102e10 2024-04-12T06:00:33.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.845+0000 7f25df444700 1 -- 172.21.15.32:0/3974351927 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25d8071120 con 0x7f25d8102e10 2024-04-12T06:00:33.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.845+0000 7f25cdffb700 1 -- 172.21.15.32:0/3974351927 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25d401c5f0 con 0x7f25d8102e10 2024-04-12T06:00:33.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.845+0000 7f25df444700 1 -- 172.21.15.32:0/3974351927 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25d811a200 con 0x7f25d8102e10 2024-04-12T06:00:33.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.846+0000 7f25df444700 1 -- 172.21.15.32:0/3974351927 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25d80fe560 con 0x7f25d8102e10 2024-04-12T06:00:33.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.846+0000 7f25cdffb700 1 -- 172.21.15.32:0/3974351927 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f25d4013590 con 0x7f25d8102e10 2024-04-12T06:00:33.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.847+0000 7f25cdffb700 1 --2- 172.21.15.32:0/3974351927 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f25c406cc30 0x7f25c406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:33.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.847+0000 7f25cdffb700 1 -- 172.21.15.32:0/3974351927 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f25d408d330 con 0x7f25d8102e10 2024-04-12T06:00:33.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.847+0000 7f25dc9df700 1 --2- 172.21.15.32:0/3974351927 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f25c406cc30 0x7f25c406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:33.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.847+0000 7f25dc9df700 1 --2- 172.21.15.32:0/3974351927 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f25c406cc30 0x7f25c406f0e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f25c8009fd0 tx=0x7f25c800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:00:33.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:33.851+0000 7f25cdffb700 1 -- 172.21.15.32:0/3974351927 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f25d4057310 con 0x7f25d8102e10 2024-04-12T06:00:34.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.075+0000 7f25df444700 1 -- 172.21.15.32:0/3974351927 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f25d80719b0 con 0x7f25c406cc30 2024-04-12T06:00:34.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.076+0000 7f25cdffb700 1 -- 172.21.15.32:0/3974351927 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f25d80719b0 con 0x7f25c406cc30 2024-04-12T06:00:34.071 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T06:00:34.071 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T06:00:34.071 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T06:00:34.072 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T06:00:34.072 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T06:00:34.072 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-12T06:00:34.072 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T06:00:34.072 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-12T06:00:34.072 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T06:00:34.072 INFO:teuthology.orchestra.run.smithi032.stdout: "mds" 2024-04-12T06:00:34.072 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T06:00:34.072 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T06:00:34.072 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T06:00:34.072 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T06:00:34.072 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T06:00:34.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.078+0000 7f25df444700 1 -- 172.21.15.32:0/3974351927 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f25c406cc30 msgr2=0x7f25c406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:34.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.078+0000 7f25df444700 1 --2- 172.21.15.32:0/3974351927 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f25c406cc30 0x7f25c406f0e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f25c8009fd0 tx=0x7f25c800b040 comp rx=0 tx=0).stop 2024-04-12T06:00:34.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.078+0000 7f25df444700 1 -- 172.21.15.32:0/3974351927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25d8102e10 msgr2=0x7f25d80702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:34.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.078+0000 7f25df444700 1 --2- 172.21.15.32:0/3974351927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25d8102e10 0x7f25d80702c0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f25d4009360 tx=0x7f25d4009e30 comp rx=0 tx=0).stop 2024-04-12T06:00:34.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.079+0000 7f25df444700 1 -- 172.21.15.32:0/3974351927 shutdown_connections 2024-04-12T06:00:34.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.079+0000 7f25df444700 1 --2- 172.21.15.32:0/3974351927 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f25c406cc30 0x7f25c406f0e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:34.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.079+0000 7f25df444700 1 --2- 172.21.15.32:0/3974351927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25d8102e10 0x7f25d80702c0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:34.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.079+0000 7f25df444700 1 --2- 172.21.15.32:0/3974351927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25d81037b0 0x7f25d8070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:34.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.079+0000 7f25df444700 1 -- 172.21.15.32:0/3974351927 >> 172.21.15.32:0/3974351927 conn(0x7f25d80f8980 msgr2=0x7f25d80f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:00:34.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.079+0000 7f25df444700 1 -- 172.21.15.32:0/3974351927 shutdown_connections 2024-04-12T06:00:34.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.079+0000 7f25df444700 1 -- 172.21.15.32:0/3974351927 wait complete. 2024-04-12T06:00:34.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.182+0000 7f58c21f5700 1 -- 172.21.15.32:0/299314620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58bc101d10 msgr2=0x7f58bc1020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:34.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.182+0000 7f58c21f5700 1 --2- 172.21.15.32:0/299314620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58bc101d10 0x7f58bc1020e0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f58ac004940 tx=0x7f58ac009e30 comp rx=0 tx=0).stop 2024-04-12T06:00:34.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.183+0000 7f58c21f5700 1 -- 172.21.15.32:0/299314620 shutdown_connections 2024-04-12T06:00:34.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.183+0000 7f58c21f5700 1 --2- 172.21.15.32:0/299314620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58bc1026b0 0x7f58bc0fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:34.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.183+0000 7f58c21f5700 1 --2- 172.21.15.32:0/299314620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58bc101d10 0x7f58bc1020e0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:34.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.183+0000 7f58c21f5700 1 -- 172.21.15.32:0/299314620 >> 172.21.15.32:0/299314620 conn(0x7f58bc0f7880 msgr2=0x7f58bc0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:00:34.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.183+0000 7f58c21f5700 1 -- 172.21.15.32:0/299314620 shutdown_connections 2024-04-12T06:00:34.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.183+0000 7f58c21f5700 1 -- 172.21.15.32:0/299314620 wait complete. 2024-04-12T06:00:34.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.183+0000 7f58c21f5700 1 Processor -- start 2024-04-12T06:00:34.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.183+0000 7f58c21f5700 1 -- start start 2024-04-12T06:00:34.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.184+0000 7f58c21f5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58bc101d10 0x7f58bc113d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:34.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.184+0000 7f58c21f5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58bc1026b0 0x7f58bc114270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:34.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.184+0000 7f58c21f5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc1148c0 con 0x7f58bc101d10 2024-04-12T06:00:34.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.184+0000 7f58c21f5700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc10dc90 con 0x7f58bc1026b0 2024-04-12T06:00:34.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.184+0000 7f58bb7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58bc101d10 0x7f58bc113d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:34.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.184+0000 7f58bb7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58bc101d10 0x7f58bc113d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55648/0 (socket says 172.21.15.32:55648) 2024-04-12T06:00:34.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.184+0000 7f58bb7fe700 1 -- 172.21.15.32:0/3373966101 learned_addr learned my addr 172.21.15.32:0/3373966101 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:00:34.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.184+0000 7f58baffd700 1 --2- 172.21.15.32:0/3373966101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58bc1026b0 0x7f58bc114270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:34.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.184+0000 7f58bb7fe700 1 -- 172.21.15.32:0/3373966101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58bc1026b0 msgr2=0x7f58bc114270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:34.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.184+0000 7f58bb7fe700 1 --2- 172.21.15.32:0/3373966101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58bc1026b0 0x7f58bc114270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:34.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.184+0000 7f58bb7fe700 1 -- 172.21.15.32:0/3373966101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ac005040 con 0x7f58bc101d10 2024-04-12T06:00:34.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.184+0000 7f58bb7fe700 1 --2- 172.21.15.32:0/3373966101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58bc101d10 0x7f58bc113d30 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f58ac000c00 tx=0x7f58ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:00:34.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.184+0000 7f58a3fff700 1 -- 172.21.15.32:0/3373966101 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58ac021070 con 0x7f58bc101d10 2024-04-12T06:00:34.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.184+0000 7f58a3fff700 1 -- 172.21.15.32:0/3373966101 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f58ac013de0 con 0x7f58bc101d10 2024-04-12T06:00:34.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.184+0000 7f58c21f5700 1 -- 172.21.15.32:0/3373966101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58bc10df10 con 0x7f58bc101d10 2024-04-12T06:00:34.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.184+0000 7f58a3fff700 1 -- 172.21.15.32:0/3373966101 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58ac00e070 con 0x7f58bc101d10 2024-04-12T06:00:34.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.184+0000 7f58c21f5700 1 -- 172.21.15.32:0/3373966101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58bc10e470 con 0x7f58bc101d10 2024-04-12T06:00:34.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.185+0000 7f58c21f5700 1 -- 172.21.15.32:0/3373966101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58bc0fd460 con 0x7f58bc101d10 2024-04-12T06:00:34.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.186+0000 7f58a3fff700 1 -- 172.21.15.32:0/3373966101 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f58ac01c600 con 0x7f58bc101d10 2024-04-12T06:00:34.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.186+0000 7f58a3fff700 1 --2- 172.21.15.32:0/3373966101 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f58a406cc20 0x7f58a406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:00:34.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.186+0000 7f58a3fff700 1 -- 172.21.15.32:0/3373966101 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f58ac08d190 con 0x7f58bc101d10 2024-04-12T06:00:34.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.187+0000 7f58baffd700 1 --2- 172.21.15.32:0/3373966101 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f58a406cc20 0x7f58a406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:00:34.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.187+0000 7f58baffd700 1 --2- 172.21.15.32:0/3373966101 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f58a406cc20 0x7f58a406f0d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f58b0004600 tx=0x7f58b000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:00:34.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.191+0000 7f58a3fff700 1 -- 172.21.15.32:0/3373966101 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f58ac05ac00 con 0x7f58bc101d10 2024-04-12T06:00:34.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.476+0000 7f58c21f5700 1 -- 172.21.15.32:0/3373966101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f58bc05f5f0 con 0x7f58bc101d10 2024-04-12T06:00:34.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.476+0000 7f58a3fff700 1 -- 172.21.15.32:0/3373966101 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f58ac05a790 con 0x7f58bc101d10 2024-04-12T06:00:34.471 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T06:00:34.472 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T06:00:34.472 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-12T06:00:34.472 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T06:00:34.472 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T06:00:34.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.479+0000 7f58c21f5700 1 -- 172.21.15.32:0/3373966101 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f58a406cc20 msgr2=0x7f58a406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:34.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.479+0000 7f58c21f5700 1 --2- 172.21.15.32:0/3373966101 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f58a406cc20 0x7f58a406f0d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f58b0004600 tx=0x7f58b000b040 comp rx=0 tx=0).stop 2024-04-12T06:00:34.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.479+0000 7f58c21f5700 1 -- 172.21.15.32:0/3373966101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58bc101d10 msgr2=0x7f58bc113d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:00:34.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.479+0000 7f58c21f5700 1 --2- 172.21.15.32:0/3373966101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58bc101d10 0x7f58bc113d30 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f58ac000c00 tx=0x7f58ac009e30 comp rx=0 tx=0).stop 2024-04-12T06:00:34.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.479+0000 7f58c21f5700 1 -- 172.21.15.32:0/3373966101 shutdown_connections 2024-04-12T06:00:34.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.479+0000 7f58c21f5700 1 --2- 172.21.15.32:0/3373966101 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f58a406cc20 0x7f58a406f0d0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:34.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.479+0000 7f58c21f5700 1 --2- 172.21.15.32:0/3373966101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58bc101d10 0x7f58bc113d30 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:34.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.479+0000 7f58c21f5700 1 --2- 172.21.15.32:0/3373966101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58bc1026b0 0x7f58bc114270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:00:34.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.479+0000 7f58c21f5700 1 -- 172.21.15.32:0/3373966101 >> 172.21.15.32:0/3373966101 conn(0x7f58bc0f7880 msgr2=0x7f58bc0f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:00:34.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.479+0000 7f58c21f5700 1 -- 172.21.15.32:0/3373966101 shutdown_connections 2024-04-12T06:00:34.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:00:34.479+0000 7f58c21f5700 1 -- 172.21.15.32:0/3373966101 wait complete. 2024-04-12T06:00:34.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:34 smithi067 ceph-mon[112979]: from='client.35132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:00:34.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:34 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3582744504' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:00:34.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:34 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3327957276' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T06:00:34.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:34 smithi032 ceph-mon[146817]: from='client.35132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:00:34.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:34 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3582744504' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:00:34.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:34 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3327957276' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T06:00:35.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:35 smithi067 ceph-mon[112979]: pgmap v602: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:35.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:35 smithi067 ceph-mon[112979]: from='client.35144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:00:35.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:35 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3373966101' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T06:00:35.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:35 smithi032 ceph-mon[146817]: pgmap v602: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:35.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:35 smithi032 ceph-mon[146817]: from='client.35144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:00:35.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:35 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3373966101' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T06:00:37.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:37 smithi067 ceph-mon[112979]: pgmap v603: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:37.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:37 smithi032 ceph-mon[146817]: pgmap v603: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:39.515 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:39 smithi067 ceph-mon[112979]: pgmap v604: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:39.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:39 smithi032 ceph-mon[146817]: pgmap v604: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:41.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:41 smithi067 ceph-mon[112979]: pgmap v605: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:41.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:41 smithi032 ceph-mon[146817]: pgmap v605: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:43.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:43 smithi067 ceph-mon[112979]: pgmap v606: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:43.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:43 smithi032 ceph-mon[146817]: pgmap v606: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:45.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:45 smithi067 ceph-mon[112979]: pgmap v607: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:45.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:45 smithi032 ceph-mon[146817]: pgmap v607: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:47.517 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:47 smithi067 ceph-mon[112979]: pgmap v608: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:47.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:47 smithi032 ceph-mon[146817]: pgmap v608: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:48.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:48 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:00:48.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:48 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:00:49.515 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:49 smithi067 ceph-mon[112979]: pgmap v609: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:49.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:49 smithi032 ceph-mon[146817]: pgmap v609: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:51.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:51 smithi067 ceph-mon[112979]: pgmap v610: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:51.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:51 smithi032 ceph-mon[146817]: pgmap v610: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:53.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:53 smithi067 ceph-mon[112979]: pgmap v611: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:53.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:53 smithi032 ceph-mon[146817]: pgmap v611: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:55.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:55 smithi067 ceph-mon[112979]: pgmap v612: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:55.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:55 smithi032 ceph-mon[146817]: pgmap v612: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:57.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:57 smithi067 ceph-mon[112979]: pgmap v613: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:57.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:57 smithi032 ceph-mon[146817]: pgmap v613: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:59.515 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:00:59 smithi067 ceph-mon[112979]: pgmap v614: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:00:59.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:00:59 smithi032 ceph-mon[146817]: pgmap v614: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:01.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:01 smithi067 ceph-mon[112979]: pgmap v615: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:01.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:01 smithi032 ceph-mon[146817]: pgmap v615: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:03.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:03 smithi067 ceph-mon[112979]: pgmap v616: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:03.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:01:03.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:03 smithi032 ceph-mon[146817]: pgmap v616: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:03.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:01:04.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.598+0000 7f25f0091700 1 -- 172.21.15.32:0/99595793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25e80fded0 msgr2=0x7f25e80fe2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:04.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.598+0000 7f25f0091700 1 --2- 172.21.15.32:0/99595793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25e80fded0 0x7f25e80fe2a0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f25e4004940 tx=0x7f25e4009e30 comp rx=0 tx=0).stop 2024-04-12T06:01:04.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.598+0000 7f25f0091700 1 -- 172.21.15.32:0/99595793 shutdown_connections 2024-04-12T06:01:04.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.598+0000 7f25f0091700 1 --2- 172.21.15.32:0/99595793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25e80fe870 0x7f25e8106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:04.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.598+0000 7f25f0091700 1 --2- 172.21.15.32:0/99595793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25e80fded0 0x7f25e80fe2a0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:04.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.598+0000 7f25f0091700 1 -- 172.21.15.32:0/99595793 >> 172.21.15.32:0/99595793 conn(0x7f25e8072bc0 msgr2=0x7f25e8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:01:04.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.598+0000 7f25f0091700 1 -- 172.21.15.32:0/99595793 shutdown_connections 2024-04-12T06:01:04.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.598+0000 7f25f0091700 1 -- 172.21.15.32:0/99595793 wait complete. 2024-04-12T06:01:04.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.598+0000 7f25f0091700 1 Processor -- start 2024-04-12T06:01:04.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.599+0000 7f25f0091700 1 -- start start 2024-04-12T06:01:04.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.599+0000 7f25f0091700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25e80fded0 0x7f25e80fbd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:04.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.599+0000 7f25f0091700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25e80fe870 0x7f25e80fc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:04.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.599+0000 7f25f0091700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e80fc860 con 0x7f25e80fe870 2024-04-12T06:01:04.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.599+0000 7f25f0091700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e80fc9a0 con 0x7f25e80fded0 2024-04-12T06:01:04.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.599+0000 7f25ed62c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25e80fe870 0x7f25e80fc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:04.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.599+0000 7f25ed62c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25e80fe870 0x7f25e80fc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56896/0 (socket says 172.21.15.32:56896) 2024-04-12T06:01:04.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.599+0000 7f25ed62c700 1 -- 172.21.15.32:0/3705274523 learned_addr learned my addr 172.21.15.32:0/3705274523 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:01:04.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.599+0000 7f25ede2d700 1 --2- 172.21.15.32:0/3705274523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25e80fded0 0x7f25e80fbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:04.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.599+0000 7f25ed62c700 1 -- 172.21.15.32:0/3705274523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25e80fded0 msgr2=0x7f25e80fbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:04.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.599+0000 7f25ed62c700 1 --2- 172.21.15.32:0/3705274523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25e80fded0 0x7f25e80fbd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:04.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.599+0000 7f25ed62c700 1 -- 172.21.15.32:0/3705274523 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e4005040 con 0x7f25e80fe870 2024-04-12T06:01:04.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.599+0000 7f25ed62c700 1 --2- 172.21.15.32:0/3705274523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25e80fe870 0x7f25e80fc2d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f25d800c830 tx=0x7f25d8004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:01:04.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.600+0000 7f25de7fc700 1 -- 172.21.15.32:0/3705274523 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25d80106f0 con 0x7f25e80fe870 2024-04-12T06:01:04.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.600+0000 7f25de7fc700 1 -- 172.21.15.32:0/3705274523 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f25d8010d00 con 0x7f25e80fe870 2024-04-12T06:01:04.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.600+0000 7f25f0091700 1 -- 172.21.15.32:0/3705274523 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25e8119d20 con 0x7f25e80fe870 2024-04-12T06:01:04.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.600+0000 7f25de7fc700 1 -- 172.21.15.32:0/3705274523 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25d8012e90 con 0x7f25e80fe870 2024-04-12T06:01:04.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.600+0000 7f25f0091700 1 -- 172.21.15.32:0/3705274523 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25e811a120 con 0x7f25e80fe870 2024-04-12T06:01:04.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.601+0000 7f25f0091700 1 -- 172.21.15.32:0/3705274523 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25e8065df0 con 0x7f25e80fe870 2024-04-12T06:01:04.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.601+0000 7f25de7fc700 1 -- 172.21.15.32:0/3705274523 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f25d8010850 con 0x7f25e80fe870 2024-04-12T06:01:04.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.602+0000 7f25de7fc700 1 --2- 172.21.15.32:0/3705274523 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f25d406cc30 0x7f25d406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:04.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.602+0000 7f25de7fc700 1 -- 172.21.15.32:0/3705274523 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f25d808c050 con 0x7f25e80fe870 2024-04-12T06:01:04.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.602+0000 7f25ede2d700 1 --2- 172.21.15.32:0/3705274523 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f25d406cc30 0x7f25d406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:04.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.602+0000 7f25ede2d700 1 --2- 172.21.15.32:0/3705274523 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f25d406cc30 0x7f25d406f0e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f25e4009fd0 tx=0x7f25e4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:01:04.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.607+0000 7f25de7fc700 1 -- 172.21.15.32:0/3705274523 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f25d8056030 con 0x7f25e80fe870 2024-04-12T06:01:04.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.831+0000 7f25f0091700 1 -- 172.21.15.32:0/3705274523 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f25e804bed0 con 0x7f25d406cc30 2024-04-12T06:01:04.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.833+0000 7f25de7fc700 1 -- 172.21.15.32:0/3705274523 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f25e804bed0 con 0x7f25d406cc30 2024-04-12T06:01:04.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.835+0000 7f25f0091700 1 -- 172.21.15.32:0/3705274523 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f25d406cc30 msgr2=0x7f25d406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:04.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.835+0000 7f25f0091700 1 --2- 172.21.15.32:0/3705274523 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f25d406cc30 0x7f25d406f0e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f25e4009fd0 tx=0x7f25e4009e30 comp rx=0 tx=0).stop 2024-04-12T06:01:04.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.835+0000 7f25f0091700 1 -- 172.21.15.32:0/3705274523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25e80fe870 msgr2=0x7f25e80fc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:04.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.835+0000 7f25f0091700 1 --2- 172.21.15.32:0/3705274523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25e80fe870 0x7f25e80fc2d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f25d800c830 tx=0x7f25d8004470 comp rx=0 tx=0).stop 2024-04-12T06:01:04.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.835+0000 7f25f0091700 1 -- 172.21.15.32:0/3705274523 shutdown_connections 2024-04-12T06:01:04.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.835+0000 7f25f0091700 1 --2- 172.21.15.32:0/3705274523 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f25d406cc30 0x7f25d406f0e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:04.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.835+0000 7f25f0091700 1 --2- 172.21.15.32:0/3705274523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25e80fded0 0x7f25e80fbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:04.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.835+0000 7f25f0091700 1 --2- 172.21.15.32:0/3705274523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25e80fe870 0x7f25e80fc2d0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:04.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.835+0000 7f25f0091700 1 -- 172.21.15.32:0/3705274523 >> 172.21.15.32:0/3705274523 conn(0x7f25e8072bc0 msgr2=0x7f25e80ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:01:04.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.835+0000 7f25f0091700 1 -- 172.21.15.32:0/3705274523 shutdown_connections 2024-04-12T06:01:04.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.835+0000 7f25f0091700 1 -- 172.21.15.32:0/3705274523 wait complete. 2024-04-12T06:01:04.839 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T06:01:04.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.939+0000 7f8c3629a700 1 -- 172.21.15.32:0/1919897330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c30102e00 msgr2=0x7f8c301031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:04.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.939+0000 7f8c3629a700 1 --2- 172.21.15.32:0/1919897330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c30102e00 0x7f8c301031d0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f8c20004940 tx=0x7f8c20009e30 comp rx=0 tx=0).stop 2024-04-12T06:01:04.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.939+0000 7f8c3629a700 1 -- 172.21.15.32:0/1919897330 shutdown_connections 2024-04-12T06:01:04.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.939+0000 7f8c3629a700 1 --2- 172.21.15.32:0/1919897330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c301037a0 0x7f8c300fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:04.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.939+0000 7f8c3629a700 1 --2- 172.21.15.32:0/1919897330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c30102e00 0x7f8c301031d0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:04.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.939+0000 7f8c3629a700 1 -- 172.21.15.32:0/1919897330 >> 172.21.15.32:0/1919897330 conn(0x7f8c300f89b0 msgr2=0x7f8c300fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:01:04.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.939+0000 7f8c3629a700 1 -- 172.21.15.32:0/1919897330 shutdown_connections 2024-04-12T06:01:04.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.939+0000 7f8c3629a700 1 -- 172.21.15.32:0/1919897330 wait complete. 2024-04-12T06:01:04.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.939+0000 7f8c3629a700 1 Processor -- start 2024-04-12T06:01:04.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.940+0000 7f8c3629a700 1 -- start start 2024-04-12T06:01:04.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.940+0000 7f8c3629a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c30102e00 0x7f8c301138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:04.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.940+0000 7f8c3629a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c301037a0 0x7f8c30113df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:04.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.940+0000 7f8c3629a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c30114440 con 0x7f8c301037a0 2024-04-12T06:01:04.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.940+0000 7f8c3629a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3010da40 con 0x7f8c30102e00 2024-04-12T06:01:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.940+0000 7f8c2f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c301037a0 0x7f8c30113df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.940+0000 7f8c2f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c301037a0 0x7f8c30113df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56924/0 (socket says 172.21.15.32:56924) 2024-04-12T06:01:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.940+0000 7f8c2f7fe700 1 -- 172.21.15.32:0/225106517 learned_addr learned my addr 172.21.15.32:0/225106517 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:01:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.940+0000 7f8c2ffff700 1 --2- 172.21.15.32:0/225106517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c30102e00 0x7f8c301138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.940+0000 7f8c2f7fe700 1 -- 172.21.15.32:0/225106517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c30102e00 msgr2=0x7f8c301138b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.940+0000 7f8c2f7fe700 1 --2- 172.21.15.32:0/225106517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c30102e00 0x7f8c301138b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.940+0000 7f8c2f7fe700 1 -- 172.21.15.32:0/225106517 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c20005040 con 0x7f8c301037a0 2024-04-12T06:01:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.940+0000 7f8c2f7fe700 1 --2- 172.21.15.32:0/225106517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c301037a0 0x7f8c30113df0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f8c1800c800 tx=0x7f8c18004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:01:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.941+0000 7f8c2cff9700 1 -- 172.21.15.32:0/225106517 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c180106f0 con 0x7f8c301037a0 2024-04-12T06:01:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.941+0000 7f8c2cff9700 1 -- 172.21.15.32:0/225106517 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8c18010d00 con 0x7f8c301037a0 2024-04-12T06:01:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.941+0000 7f8c3629a700 1 -- 172.21.15.32:0/225106517 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c3010dd20 con 0x7f8c301037a0 2024-04-12T06:01:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.941+0000 7f8c2cff9700 1 -- 172.21.15.32:0/225106517 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c180113a0 con 0x7f8c301037a0 2024-04-12T06:01:04.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.941+0000 7f8c3629a700 1 -- 172.21.15.32:0/225106517 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c3010e2e0 con 0x7f8c301037a0 2024-04-12T06:01:04.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.942+0000 7f8c3629a700 1 -- 172.21.15.32:0/225106517 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c300fe550 con 0x7f8c301037a0 2024-04-12T06:01:04.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.942+0000 7f8c2cff9700 1 -- 172.21.15.32:0/225106517 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8c18012400 con 0x7f8c301037a0 2024-04-12T06:01:04.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.943+0000 7f8c2cff9700 1 --2- 172.21.15.32:0/225106517 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f8c1c06cc20 0x7f8c1c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:04.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.943+0000 7f8c2ffff700 1 --2- 172.21.15.32:0/225106517 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f8c1c06cc20 0x7f8c1c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:04.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.943+0000 7f8c2cff9700 1 -- 172.21.15.32:0/225106517 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f8c1808bf50 con 0x7f8c301037a0 2024-04-12T06:01:04.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.943+0000 7f8c2ffff700 1 --2- 172.21.15.32:0/225106517 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f8c1c06cc20 0x7f8c1c06f0d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8c2000b840 tx=0x7f8c20009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:01:04.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:04.947+0000 7f8c2cff9700 1 -- 172.21.15.32:0/225106517 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8c180bfab0 con 0x7f8c301037a0 2024-04-12T06:01:05.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.167+0000 7f8c3629a700 1 -- 172.21.15.32:0/225106517 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8c30109dc0 con 0x7f8c1c06cc20 2024-04-12T06:01:05.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.168+0000 7f8c2cff9700 1 -- 172.21.15.32:0/225106517 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8c30109dc0 con 0x7f8c1c06cc20 2024-04-12T06:01:05.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.170+0000 7f8c3629a700 1 -- 172.21.15.32:0/225106517 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f8c1c06cc20 msgr2=0x7f8c1c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:05.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.170+0000 7f8c3629a700 1 --2- 172.21.15.32:0/225106517 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f8c1c06cc20 0x7f8c1c06f0d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8c2000b840 tx=0x7f8c20009e30 comp rx=0 tx=0).stop 2024-04-12T06:01:05.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.170+0000 7f8c3629a700 1 -- 172.21.15.32:0/225106517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c301037a0 msgr2=0x7f8c30113df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:05.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.170+0000 7f8c3629a700 1 --2- 172.21.15.32:0/225106517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c301037a0 0x7f8c30113df0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f8c1800c800 tx=0x7f8c18004470 comp rx=0 tx=0).stop 2024-04-12T06:01:05.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.170+0000 7f8c3629a700 1 -- 172.21.15.32:0/225106517 shutdown_connections 2024-04-12T06:01:05.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.170+0000 7f8c3629a700 1 --2- 172.21.15.32:0/225106517 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f8c1c06cc20 0x7f8c1c06f0d0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:05.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.170+0000 7f8c3629a700 1 --2- 172.21.15.32:0/225106517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c30102e00 0x7f8c301138b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:05.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.170+0000 7f8c3629a700 1 --2- 172.21.15.32:0/225106517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c301037a0 0x7f8c30113df0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:05.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.170+0000 7f8c3629a700 1 -- 172.21.15.32:0/225106517 >> 172.21.15.32:0/225106517 conn(0x7f8c300f89b0 msgr2=0x7f8c300f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:01:05.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.171+0000 7f8c3629a700 1 -- 172.21.15.32:0/225106517 shutdown_connections 2024-04-12T06:01:05.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.171+0000 7f8c3629a700 1 -- 172.21.15.32:0/225106517 wait complete. 2024-04-12T06:01:05.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.274+0000 7f529b323700 1 -- 172.21.15.32:0/2329491591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52941037a0 msgr2=0x7f52940fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:05.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.274+0000 7f529b323700 1 --2- 172.21.15.32:0/2329491591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52941037a0 0x7f52940fce10 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f5290004940 tx=0x7f5290009e30 comp rx=0 tx=0).stop 2024-04-12T06:01:05.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.274+0000 7f529b323700 1 -- 172.21.15.32:0/2329491591 shutdown_connections 2024-04-12T06:01:05.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.274+0000 7f529b323700 1 --2- 172.21.15.32:0/2329491591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52941037a0 0x7f52940fce10 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:05.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.274+0000 7f529b323700 1 --2- 172.21.15.32:0/2329491591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5294102e00 0x7f52941031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:05.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.274+0000 7f529b323700 1 -- 172.21.15.32:0/2329491591 >> 172.21.15.32:0/2329491591 conn(0x7f52940f89b0 msgr2=0x7f52940fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:01:05.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.274+0000 7f529b323700 1 -- 172.21.15.32:0/2329491591 shutdown_connections 2024-04-12T06:01:05.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.274+0000 7f529b323700 1 -- 172.21.15.32:0/2329491591 wait complete. 2024-04-12T06:01:05.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.274+0000 7f529b323700 1 Processor -- start 2024-04-12T06:01:05.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.275+0000 7f529b323700 1 -- start start 2024-04-12T06:01:05.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.275+0000 7f529b323700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5294102e00 0x7f5294113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:05.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.275+0000 7f529b323700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52941037a0 0x7f5294114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:05.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.275+0000 7f529b323700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5294114700 con 0x7f52941037a0 2024-04-12T06:01:05.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.275+0000 7f529b323700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529410da60 con 0x7f5294102e00 2024-04-12T06:01:05.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.275+0000 7f52988be700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52941037a0 0x7f5294114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:05.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.275+0000 7f52988be700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52941037a0 0x7f5294114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56928/0 (socket says 172.21.15.32:56928) 2024-04-12T06:01:05.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.275+0000 7f52988be700 1 -- 172.21.15.32:0/3413857028 learned_addr learned my addr 172.21.15.32:0/3413857028 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:01:05.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.275+0000 7f52990bf700 1 --2- 172.21.15.32:0/3413857028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5294102e00 0x7f5294113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:05.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.275+0000 7f52988be700 1 -- 172.21.15.32:0/3413857028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5294102e00 msgr2=0x7f5294113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:05.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.275+0000 7f52988be700 1 --2- 172.21.15.32:0/3413857028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5294102e00 0x7f5294113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:05.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.275+0000 7f52988be700 1 -- 172.21.15.32:0/3413857028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5290005040 con 0x7f52941037a0 2024-04-12T06:01:05.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.275+0000 7f52990bf700 1 --2- 172.21.15.32:0/3413857028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5294102e00 0x7f5294113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T06:01:05.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.275+0000 7f52988be700 1 --2- 172.21.15.32:0/3413857028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52941037a0 0x7f5294114020 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f5290009380 tx=0x7f5290009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:01:05.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.276+0000 7f5289ffb700 1 -- 172.21.15.32:0/3413857028 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f529000e040 con 0x7f52941037a0 2024-04-12T06:01:05.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.276+0000 7f5289ffb700 1 -- 172.21.15.32:0/3413857028 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5290013420 con 0x7f52941037a0 2024-04-12T06:01:05.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.276+0000 7f529b323700 1 -- 172.21.15.32:0/3413857028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f529410dce0 con 0x7f52941037a0 2024-04-12T06:01:05.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.276+0000 7f5289ffb700 1 -- 172.21.15.32:0/3413857028 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f529001c630 con 0x7f52941037a0 2024-04-12T06:01:05.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.276+0000 7f529b323700 1 -- 172.21.15.32:0/3413857028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f529410e2a0 con 0x7f52941037a0 2024-04-12T06:01:05.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.277+0000 7f529b323700 1 -- 172.21.15.32:0/3413857028 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52940fe550 con 0x7f52941037a0 2024-04-12T06:01:05.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.277+0000 7f5289ffb700 1 -- 172.21.15.32:0/3413857028 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f52900139e0 con 0x7f52941037a0 2024-04-12T06:01:05.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.278+0000 7f5289ffb700 1 --2- 172.21.15.32:0/3413857028 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f528006cc20 0x7f528006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:05.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.278+0000 7f5289ffb700 1 -- 172.21.15.32:0/3413857028 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f529008d1c0 con 0x7f52941037a0 2024-04-12T06:01:05.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.278+0000 7f52990bf700 1 --2- 172.21.15.32:0/3413857028 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f528006cc20 0x7f528006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:05.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.278+0000 7f52990bf700 1 --2- 172.21.15.32:0/3413857028 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f528006cc20 0x7f528006f0d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f5284006fd0 tx=0x7f5284008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:01:05.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.282+0000 7f5289ffb700 1 -- 172.21.15.32:0/3413857028 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f529005acd0 con 0x7f52941037a0 2024-04-12T06:01:05.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.493+0000 7f529b323700 1 -- 172.21.15.32:0/3413857028 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f52941137b0 con 0x7f528006cc20 2024-04-12T06:01:05.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.503+0000 7f5289ffb700 1 -- 172.21.15.32:0/3413857028 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f52941137b0 con 0x7f528006cc20 2024-04-12T06:01:05.497 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T06:01:05.497 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (20m) 14m ago 32m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T06:01:05.497 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (18m) 14m ago 32m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T06:01:05.497 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (18m) 13m ago 31m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T06:01:05.497 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (20m) 14m ago 32m 87.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T06:01:05.497 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (14m) 14m ago 26m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T06:01:05.497 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (14m) 14m ago 26m 13.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T06:01:05.497 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (13m) 13m ago 26m 11.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 6d4aca6f74a0 2024-04-12T06:01:05.497 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (14m) 13m ago 26m 14.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T06:01:05.497 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (22m) 14m ago 33m 507M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T06:01:05.497 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (21m) 13m ago 31m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T06:01:05.497 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (19m) 14m ago 33m 53.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T06:01:05.497 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (19m) 13m ago 31m 43.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T06:01:05.497 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (21m) 14m ago 32m 16.9M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T06:01:05.498 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (21m) 13m ago 31m 18.4M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T06:01:05.498 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (17m) 14m ago 30m 91.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T06:01:05.498 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (17m) 14m ago 30m 97.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T06:01:05.498 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (16m) 14m ago 29m 81.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T06:01:05.498 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (16m) 13m ago 29m 113M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T06:01:05.498 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (15m) 13m ago 28m 90.3M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T06:01:05.498 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (15m) 13m ago 28m 84.8M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T06:01:05.498 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (21m) 14m ago 31m 64.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T06:01:05.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.505+0000 7f529b323700 1 -- 172.21.15.32:0/3413857028 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f528006cc20 msgr2=0x7f528006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:05.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.505+0000 7f529b323700 1 --2- 172.21.15.32:0/3413857028 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f528006cc20 0x7f528006f0d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f5284006fd0 tx=0x7f5284008040 comp rx=0 tx=0).stop 2024-04-12T06:01:05.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.505+0000 7f529b323700 1 -- 172.21.15.32:0/3413857028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52941037a0 msgr2=0x7f5294114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:05.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.505+0000 7f529b323700 1 --2- 172.21.15.32:0/3413857028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52941037a0 0x7f5294114020 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f5290009380 tx=0x7f5290009e30 comp rx=0 tx=0).stop 2024-04-12T06:01:05.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.505+0000 7f529b323700 1 -- 172.21.15.32:0/3413857028 shutdown_connections 2024-04-12T06:01:05.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.505+0000 7f529b323700 1 --2- 172.21.15.32:0/3413857028 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f528006cc20 0x7f528006f0d0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:05.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.505+0000 7f529b323700 1 --2- 172.21.15.32:0/3413857028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5294102e00 0x7f5294113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:05.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.505+0000 7f529b323700 1 --2- 172.21.15.32:0/3413857028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52941037a0 0x7f5294114020 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:05.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.505+0000 7f529b323700 1 -- 172.21.15.32:0/3413857028 >> 172.21.15.32:0/3413857028 conn(0x7f52940f89b0 msgr2=0x7f52940f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:01:05.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.505+0000 7f529b323700 1 -- 172.21.15.32:0/3413857028 shutdown_connections 2024-04-12T06:01:05.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.505+0000 7f529b323700 1 -- 172.21.15.32:0/3413857028 wait complete. 2024-04-12T06:01:05.515 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:05 smithi067 ceph-mon[112979]: pgmap v617: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:05.515 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:05 smithi067 ceph-mon[112979]: from='client.35152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:01:05.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:05 smithi032 ceph-mon[146817]: pgmap v617: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:05.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:05 smithi032 ceph-mon[146817]: from='client.35152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:01:05.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.609+0000 7f81721f8700 1 -- 172.21.15.32:0/567677171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f816c1037b0 msgr2=0x7f816c0fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:05.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.609+0000 7f81721f8700 1 --2- 172.21.15.32:0/567677171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f816c1037b0 0x7f816c0fce20 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f8154004940 tx=0x7f8154009e30 comp rx=0 tx=0).stop 2024-04-12T06:01:05.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.609+0000 7f81721f8700 1 -- 172.21.15.32:0/567677171 shutdown_connections 2024-04-12T06:01:05.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.609+0000 7f81721f8700 1 --2- 172.21.15.32:0/567677171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f816c1037b0 0x7f816c0fce20 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:05.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.609+0000 7f81721f8700 1 --2- 172.21.15.32:0/567677171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f816c102e10 0x7f816c1031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:05.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.609+0000 7f81721f8700 1 -- 172.21.15.32:0/567677171 >> 172.21.15.32:0/567677171 conn(0x7f816c0f8980 msgr2=0x7f816c0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:01:05.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.609+0000 7f81721f8700 1 -- 172.21.15.32:0/567677171 shutdown_connections 2024-04-12T06:01:05.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.609+0000 7f81721f8700 1 -- 172.21.15.32:0/567677171 wait complete. 2024-04-12T06:01:05.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.610+0000 7f81721f8700 1 Processor -- start 2024-04-12T06:01:05.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.610+0000 7f81721f8700 1 -- start start 2024-04-12T06:01:05.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.610+0000 7f81721f8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f816c102e10 0x7f816c113ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:05.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.610+0000 7f81721f8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f816c1037b0 0x7f816c113ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:05.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.610+0000 7f81721f8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f816c1146d0 con 0x7f816c1037b0 2024-04-12T06:01:05.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.610+0000 7f81721f8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f816c10da30 con 0x7f816c102e10 2024-04-12T06:01:05.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.610+0000 7f816affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f816c1037b0 0x7f816c113ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:05.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.610+0000 7f816affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f816c1037b0 0x7f816c113ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56944/0 (socket says 172.21.15.32:56944) 2024-04-12T06:01:05.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.610+0000 7f816affd700 1 -- 172.21.15.32:0/2544788166 learned_addr learned my addr 172.21.15.32:0/2544788166 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:01:05.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.610+0000 7f816b7fe700 1 --2- 172.21.15.32:0/2544788166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f816c102e10 0x7f816c113ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:05.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.611+0000 7f816affd700 1 -- 172.21.15.32:0/2544788166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f816c102e10 msgr2=0x7f816c113ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:05.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.611+0000 7f816affd700 1 --2- 172.21.15.32:0/2544788166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f816c102e10 0x7f816c113ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:05.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.611+0000 7f816affd700 1 -- 172.21.15.32:0/2544788166 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8154005040 con 0x7f816c1037b0 2024-04-12T06:01:05.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.611+0000 7f816affd700 1 --2- 172.21.15.32:0/2544788166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f816c1037b0 0x7f816c113ff0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f8154009390 tx=0x7f8154009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:01:05.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.611+0000 7f8163fff700 1 -- 172.21.15.32:0/2544788166 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f815400e040 con 0x7f816c1037b0 2024-04-12T06:01:05.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.611+0000 7f8163fff700 1 -- 172.21.15.32:0/2544788166 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8154013420 con 0x7f816c1037b0 2024-04-12T06:01:05.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.611+0000 7f81721f8700 1 -- 172.21.15.32:0/2544788166 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f816c10dcb0 con 0x7f816c1037b0 2024-04-12T06:01:05.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.611+0000 7f8163fff700 1 -- 172.21.15.32:0/2544788166 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f815401c630 con 0x7f816c1037b0 2024-04-12T06:01:05.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.611+0000 7f81721f8700 1 -- 172.21.15.32:0/2544788166 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f816c10e270 con 0x7f816c1037b0 2024-04-12T06:01:05.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.613+0000 7f81721f8700 1 -- 172.21.15.32:0/2544788166 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f816c0fe560 con 0x7f816c1037b0 2024-04-12T06:01:05.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.613+0000 7f8163fff700 1 -- 172.21.15.32:0/2544788166 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8154013a00 con 0x7f816c1037b0 2024-04-12T06:01:05.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.613+0000 7f8163fff700 1 --2- 172.21.15.32:0/2544788166 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f815806cc30 0x7f815806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:05.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.613+0000 7f8163fff700 1 -- 172.21.15.32:0/2544788166 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f815408d0b0 con 0x7f816c1037b0 2024-04-12T06:01:05.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.613+0000 7f816b7fe700 1 --2- 172.21.15.32:0/2544788166 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f815806cc30 0x7f815806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:05.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.614+0000 7f816b7fe700 1 --2- 172.21.15.32:0/2544788166 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f815806cc30 0x7f815806f0e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f815c009bf0 tx=0x7f815c0091a0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:01:05.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.618+0000 7f8163fff700 1 -- 172.21.15.32:0/2544788166 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f815405abc0 con 0x7f816c1037b0 2024-04-12T06:01:05.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.904+0000 7f81721f8700 1 -- 172.21.15.32:0/2544788166 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f816c05f5f0 con 0x7f816c1037b0 2024-04-12T06:01:05.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.904+0000 7f8163fff700 1 -- 172.21.15.32:0/2544788166 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f815405a750 con 0x7f816c1037b0 2024-04-12T06:01:05.898 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T06:01:05.898 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T06:01:05.898 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T06:01:05.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T06:01:05.898 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T06:01:05.899 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T06:01:05.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T06:01:05.899 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T06:01:05.899 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T06:01:05.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T06:01:05.899 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T06:01:05.899 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T06:01:05.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T06:01:05.899 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T06:01:05.899 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T06:01:05.899 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T06:01:05.899 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T06:01:05.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.906+0000 7f81721f8700 1 -- 172.21.15.32:0/2544788166 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f815806cc30 msgr2=0x7f815806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:05.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.906+0000 7f81721f8700 1 --2- 172.21.15.32:0/2544788166 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f815806cc30 0x7f815806f0e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f815c009bf0 tx=0x7f815c0091a0 comp rx=0 tx=0).stop 2024-04-12T06:01:05.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.906+0000 7f81721f8700 1 -- 172.21.15.32:0/2544788166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f816c1037b0 msgr2=0x7f816c113ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:05.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.906+0000 7f81721f8700 1 --2- 172.21.15.32:0/2544788166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f816c1037b0 0x7f816c113ff0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f8154009390 tx=0x7f8154009e30 comp rx=0 tx=0).stop 2024-04-12T06:01:05.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.906+0000 7f81721f8700 1 -- 172.21.15.32:0/2544788166 shutdown_connections 2024-04-12T06:01:05.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.906+0000 7f81721f8700 1 --2- 172.21.15.32:0/2544788166 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f815806cc30 0x7f815806f0e0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:05.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.906+0000 7f81721f8700 1 --2- 172.21.15.32:0/2544788166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f816c102e10 0x7f816c113ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:05.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.906+0000 7f81721f8700 1 --2- 172.21.15.32:0/2544788166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f816c1037b0 0x7f816c113ff0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:05.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.906+0000 7f81721f8700 1 -- 172.21.15.32:0/2544788166 >> 172.21.15.32:0/2544788166 conn(0x7f816c0f8980 msgr2=0x7f816c0f99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:01:05.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.906+0000 7f81721f8700 1 -- 172.21.15.32:0/2544788166 shutdown_connections 2024-04-12T06:01:05.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:05.906+0000 7f81721f8700 1 -- 172.21.15.32:0/2544788166 wait complete. 2024-04-12T06:01:06.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.009+0000 7f1f2235c700 1 -- 172.21.15.32:0/1677359463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f1c101d70 msgr2=0x7f1f1c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:06.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.009+0000 7f1f2235c700 1 --2- 172.21.15.32:0/1677359463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f1c101d70 0x7f1f1c102140 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f1f04004940 tx=0x7f1f04009e30 comp rx=0 tx=0).stop 2024-04-12T06:01:06.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.009+0000 7f1f2235c700 1 -- 172.21.15.32:0/1677359463 shutdown_connections 2024-04-12T06:01:06.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.009+0000 7f1f2235c700 1 --2- 172.21.15.32:0/1677359463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f1c102710 0x7f1f1c0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:06.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.009+0000 7f1f2235c700 1 --2- 172.21.15.32:0/1677359463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f1c101d70 0x7f1f1c102140 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:06.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.009+0000 7f1f2235c700 1 -- 172.21.15.32:0/1677359463 >> 172.21.15.32:0/1677359463 conn(0x7f1f1c072bc0 msgr2=0x7f1f1c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:01:06.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.009+0000 7f1f2235c700 1 -- 172.21.15.32:0/1677359463 shutdown_connections 2024-04-12T06:01:06.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.009+0000 7f1f2235c700 1 -- 172.21.15.32:0/1677359463 wait complete. 2024-04-12T06:01:06.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.010+0000 7f1f2235c700 1 Processor -- start 2024-04-12T06:01:06.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.010+0000 7f1f2235c700 1 -- start start 2024-04-12T06:01:06.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.011+0000 7f1f2235c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f1c101d70 0x7f1f1c1129a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:06.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.011+0000 7f1f2235c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f1c102710 0x7f1f1c112ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:06.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.011+0000 7f1f2235c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c113530 con 0x7f1f1c102710 2024-04-12T06:01:06.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.011+0000 7f1f2235c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c10c920 con 0x7f1f1c101d70 2024-04-12T06:01:06.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.011+0000 7f1f1b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f1c102710 0x7f1f1c112ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:06.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.011+0000 7f1f1b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f1c102710 0x7f1f1c112ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56964/0 (socket says 172.21.15.32:56964) 2024-04-12T06:01:06.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.011+0000 7f1f1b7fe700 1 -- 172.21.15.32:0/113958911 learned_addr learned my addr 172.21.15.32:0/113958911 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:01:06.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.011+0000 7f1f1bfff700 1 --2- 172.21.15.32:0/113958911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f1c101d70 0x7f1f1c1129a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:06.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.011+0000 7f1f1b7fe700 1 -- 172.21.15.32:0/113958911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f1c101d70 msgr2=0x7f1f1c1129a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:06.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.011+0000 7f1f1b7fe700 1 --2- 172.21.15.32:0/113958911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f1c101d70 0x7f1f1c1129a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:06.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.011+0000 7f1f1b7fe700 1 -- 172.21.15.32:0/113958911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f04005040 con 0x7f1f1c102710 2024-04-12T06:01:06.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.011+0000 7f1f1b7fe700 1 --2- 172.21.15.32:0/113958911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f1c102710 0x7f1f1c112ee0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c002a50 tx=0x7f1f0c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:01:06.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.011+0000 7f1f18ff9700 1 -- 172.21.15.32:0/113958911 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f0c007cc0 con 0x7f1f1c102710 2024-04-12T06:01:06.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.011+0000 7f1f2235c700 1 -- 172.21.15.32:0/113958911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f1c10cc00 con 0x7f1f1c102710 2024-04-12T06:01:06.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.011+0000 7f1f18ff9700 1 -- 172.21.15.32:0/113958911 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f0c00f040 con 0x7f1f1c102710 2024-04-12T06:01:06.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.012+0000 7f1f18ff9700 1 -- 172.21.15.32:0/113958911 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f0c00d920 con 0x7f1f1c102710 2024-04-12T06:01:06.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.012+0000 7f1f2235c700 1 -- 172.21.15.32:0/113958911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f1c10d1c0 con 0x7f1f1c102710 2024-04-12T06:01:06.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.012+0000 7f1f2235c700 1 -- 172.21.15.32:0/113958911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f1c0ff080 con 0x7f1f1c102710 2024-04-12T06:01:06.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.013+0000 7f1f18ff9700 1 -- 172.21.15.32:0/113958911 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1f0c009680 con 0x7f1f1c102710 2024-04-12T06:01:06.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.013+0000 7f1f18ff9700 1 --2- 172.21.15.32:0/113958911 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1f0806cc30 0x7f1f0806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:06.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.014+0000 7f1f18ff9700 1 -- 172.21.15.32:0/113958911 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f1f0c08be40 con 0x7f1f1c102710 2024-04-12T06:01:06.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.014+0000 7f1f1bfff700 1 --2- 172.21.15.32:0/113958911 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1f0806cc30 0x7f1f0806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:06.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.014+0000 7f1f1bfff700 1 --2- 172.21.15.32:0/113958911 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1f0806cc30 0x7f1f0806f0e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1f0400b840 tx=0x7f1f04009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:01:06.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.018+0000 7f1f18ff9700 1 -- 172.21.15.32:0/113958911 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1f0c059830 con 0x7f1f1c102710 2024-04-12T06:01:06.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.269+0000 7f1f2235c700 1 -- 172.21.15.32:0/113958911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1f1c10dff0 con 0x7f1f1c102710 2024-04-12T06:01:06.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.270+0000 7f1f18ff9700 1 -- 172.21.15.32:0/113958911 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1835 (secure 0 0 0) 0x7f1f0c0593c0 con 0x7f1f1c102710 2024-04-12T06:01:06.264 INFO:teuthology.orchestra.run.smithi032.stdout:e33 2024-04-12T06:01:06.264 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T06:01:06.264 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T06:01:06.264 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T06:01:06.264 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:01:06.264 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T06:01:06.264 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T06:01:06.264 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 24 2024-04-12T06:01:06.264 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T06:01:06.264 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T06:01:06.264 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:46:10.668569+0000 2024-04-12T06:01:06.265 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T06:01:06.265 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T06:01:06.265 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T06:01:06.265 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T06:01:06.265 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T06:01:06.265 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T06:01:06.265 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T06:01:06.265 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T06:01:06.265 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T06:01:06.265 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T06:01:06.265 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T06:01:06.265 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T06:01:06.265 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T06:01:06.265 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-12T06:01:06.266 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T06:01:06.266 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T06:01:06.266 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T06:01:06.266 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T06:01:06.266 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T06:01:06.266 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T06:01:06.266 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T06:01:06.266 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T06:01:06.266 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:01:06.266 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:01:06.266 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-12T06:01:06.266 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:01:06.266 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T06:01:06.267 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T06:01:06.267 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{-1:44317} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T06:01:06.267 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T06:01:06.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.272+0000 7f1f2235c700 1 -- 172.21.15.32:0/113958911 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1f0806cc30 msgr2=0x7f1f0806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:06.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.272+0000 7f1f2235c700 1 --2- 172.21.15.32:0/113958911 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1f0806cc30 0x7f1f0806f0e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1f0400b840 tx=0x7f1f04009e30 comp rx=0 tx=0).stop 2024-04-12T06:01:06.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.272+0000 7f1f2235c700 1 -- 172.21.15.32:0/113958911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f1c102710 msgr2=0x7f1f1c112ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:06.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.272+0000 7f1f2235c700 1 --2- 172.21.15.32:0/113958911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f1c102710 0x7f1f1c112ee0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c002a50 tx=0x7f1f0c004600 comp rx=0 tx=0).stop 2024-04-12T06:01:06.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.272+0000 7f1f2235c700 1 -- 172.21.15.32:0/113958911 shutdown_connections 2024-04-12T06:01:06.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.272+0000 7f1f2235c700 1 --2- 172.21.15.32:0/113958911 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f1f0806cc30 0x7f1f0806f0e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:06.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.272+0000 7f1f2235c700 1 --2- 172.21.15.32:0/113958911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f1c101d70 0x7f1f1c1129a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:06.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.272+0000 7f1f2235c700 1 --2- 172.21.15.32:0/113958911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f1c102710 0x7f1f1c112ee0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:06.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.272+0000 7f1f2235c700 1 -- 172.21.15.32:0/113958911 >> 172.21.15.32:0/113958911 conn(0x7f1f1c072bc0 msgr2=0x7f1f1c0fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:01:06.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.272+0000 7f1f2235c700 1 -- 172.21.15.32:0/113958911 shutdown_connections 2024-04-12T06:01:06.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.272+0000 7f1f2235c700 1 -- 172.21.15.32:0/113958911 wait complete. 2024-04-12T06:01:06.268 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-12T06:01:06.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.376+0000 7f6d8e5b0700 1 -- 172.21.15.32:0/3897674075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d88102e10 msgr2=0x7f6d881031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:06.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.376+0000 7f6d8e5b0700 1 --2- 172.21.15.32:0/3897674075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d88102e10 0x7f6d881031e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f6d78004940 tx=0x7f6d78009e30 comp rx=0 tx=0).stop 2024-04-12T06:01:06.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.377+0000 7f6d8e5b0700 1 -- 172.21.15.32:0/3897674075 shutdown_connections 2024-04-12T06:01:06.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.377+0000 7f6d8e5b0700 1 --2- 172.21.15.32:0/3897674075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d881037b0 0x7f6d880fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:06.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.377+0000 7f6d8e5b0700 1 --2- 172.21.15.32:0/3897674075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d88102e10 0x7f6d881031e0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:06.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.377+0000 7f6d8e5b0700 1 -- 172.21.15.32:0/3897674075 >> 172.21.15.32:0/3897674075 conn(0x7f6d880f89a0 msgr2=0x7f6d880fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:01:06.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.377+0000 7f6d8e5b0700 1 -- 172.21.15.32:0/3897674075 shutdown_connections 2024-04-12T06:01:06.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.377+0000 7f6d8e5b0700 1 -- 172.21.15.32:0/3897674075 wait complete. 2024-04-12T06:01:06.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.377+0000 7f6d8e5b0700 1 Processor -- start 2024-04-12T06:01:06.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.378+0000 7f6d8e5b0700 1 -- start start 2024-04-12T06:01:06.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.378+0000 7f6d8e5b0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d88102e10 0x7f6d88113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:06.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.378+0000 7f6d8e5b0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d881037b0 0x7f6d88114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:06.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.378+0000 7f6d8e5b0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d88114680 con 0x7f6d881037b0 2024-04-12T06:01:06.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.378+0000 7f6d8e5b0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8810da70 con 0x7f6d88102e10 2024-04-12T06:01:06.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.378+0000 7f6d877fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d881037b0 0x7f6d88114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:06.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.378+0000 7f6d877fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d881037b0 0x7f6d88114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56982/0 (socket says 172.21.15.32:56982) 2024-04-12T06:01:06.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.378+0000 7f6d877fe700 1 -- 172.21.15.32:0/4262297927 learned_addr learned my addr 172.21.15.32:0/4262297927 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:01:06.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.378+0000 7f6d87fff700 1 --2- 172.21.15.32:0/4262297927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d88102e10 0x7f6d88113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:06.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.378+0000 7f6d877fe700 1 -- 172.21.15.32:0/4262297927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d88102e10 msgr2=0x7f6d88113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:06.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.378+0000 7f6d877fe700 1 --2- 172.21.15.32:0/4262297927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d88102e10 0x7f6d88113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:06.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.378+0000 7f6d877fe700 1 -- 172.21.15.32:0/4262297927 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d78005040 con 0x7f6d881037b0 2024-04-12T06:01:06.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.378+0000 7f6d877fe700 1 --2- 172.21.15.32:0/4262297927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d881037b0 0x7f6d88114030 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c002a50 tx=0x7f6d7c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:01:06.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.379+0000 7f6d84ff9700 1 -- 172.21.15.32:0/4262297927 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d7c007cc0 con 0x7f6d881037b0 2024-04-12T06:01:06.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.379+0000 7f6d84ff9700 1 -- 172.21.15.32:0/4262297927 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6d7c00f040 con 0x7f6d881037b0 2024-04-12T06:01:06.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.379+0000 7f6d8e5b0700 1 -- 172.21.15.32:0/4262297927 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d8810dd50 con 0x7f6d881037b0 2024-04-12T06:01:06.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.379+0000 7f6d84ff9700 1 -- 172.21.15.32:0/4262297927 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d7c00d990 con 0x7f6d881037b0 2024-04-12T06:01:06.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.379+0000 7f6d8e5b0700 1 -- 172.21.15.32:0/4262297927 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d8810e310 con 0x7f6d881037b0 2024-04-12T06:01:06.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.380+0000 7f6d8e5b0700 1 -- 172.21.15.32:0/4262297927 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d880fe560 con 0x7f6d881037b0 2024-04-12T06:01:06.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.380+0000 7f6d84ff9700 1 -- 172.21.15.32:0/4262297927 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f6d7c009710 con 0x7f6d881037b0 2024-04-12T06:01:06.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.380+0000 7f6d84ff9700 1 --2- 172.21.15.32:0/4262297927 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6d7006cc20 0x7f6d7006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:06.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.380+0000 7f6d84ff9700 1 -- 172.21.15.32:0/4262297927 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f6d7c08bec0 con 0x7f6d881037b0 2024-04-12T06:01:06.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.381+0000 7f6d87fff700 1 --2- 172.21.15.32:0/4262297927 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6d7006cc20 0x7f6d7006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:06.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.381+0000 7f6d87fff700 1 --2- 172.21.15.32:0/4262297927 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6d7006cc20 0x7f6d7006f0d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6d78004f90 tx=0x7f6d78009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:01:06.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.385+0000 7f6d84ff9700 1 -- 172.21.15.32:0/4262297927 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6d7c059620 con 0x7f6d881037b0 2024-04-12T06:01:06.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:06 smithi067 ceph-mon[112979]: from='client.35156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:01:06.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:06 smithi067 ceph-mon[112979]: from='client.35160 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:01:06.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:06 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/2544788166' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:01:06.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:06 smithi032 ceph-mon[146817]: from='client.35156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:01:06.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:06 smithi032 ceph-mon[146817]: from='client.35160 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:01:06.590 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:06 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/2544788166' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:01:06.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.607+0000 7f6d8e5b0700 1 -- 172.21.15.32:0/4262297927 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d88109dd0 con 0x7f6d7006cc20 2024-04-12T06:01:06.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.609+0000 7f6d84ff9700 1 -- 172.21.15.32:0/4262297927 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6d88109dd0 con 0x7f6d7006cc20 2024-04-12T06:01:06.603 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T06:01:06.603 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T06:01:06.603 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T06:01:06.603 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T06:01:06.603 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T06:01:06.603 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-12T06:01:06.603 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T06:01:06.603 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-12T06:01:06.603 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T06:01:06.603 INFO:teuthology.orchestra.run.smithi032.stdout: "mds" 2024-04-12T06:01:06.603 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T06:01:06.603 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T06:01:06.603 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T06:01:06.603 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T06:01:06.604 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T06:01:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.611+0000 7f6d8e5b0700 1 -- 172.21.15.32:0/4262297927 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6d7006cc20 msgr2=0x7f6d7006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.611+0000 7f6d8e5b0700 1 --2- 172.21.15.32:0/4262297927 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6d7006cc20 0x7f6d7006f0d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6d78004f90 tx=0x7f6d78009e30 comp rx=0 tx=0).stop 2024-04-12T06:01:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.611+0000 7f6d8e5b0700 1 -- 172.21.15.32:0/4262297927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d881037b0 msgr2=0x7f6d88114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.611+0000 7f6d8e5b0700 1 --2- 172.21.15.32:0/4262297927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d881037b0 0x7f6d88114030 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c002a50 tx=0x7f6d7c004600 comp rx=0 tx=0).stop 2024-04-12T06:01:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.611+0000 7f6d8e5b0700 1 -- 172.21.15.32:0/4262297927 shutdown_connections 2024-04-12T06:01:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.611+0000 7f6d8e5b0700 1 --2- 172.21.15.32:0/4262297927 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f6d7006cc20 0x7f6d7006f0d0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.611+0000 7f6d8e5b0700 1 --2- 172.21.15.32:0/4262297927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d88102e10 0x7f6d88113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.611+0000 7f6d8e5b0700 1 --2- 172.21.15.32:0/4262297927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d881037b0 0x7f6d88114030 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.611+0000 7f6d8e5b0700 1 -- 172.21.15.32:0/4262297927 >> 172.21.15.32:0/4262297927 conn(0x7f6d880f89a0 msgr2=0x7f6d880f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:01:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.611+0000 7f6d8e5b0700 1 -- 172.21.15.32:0/4262297927 shutdown_connections 2024-04-12T06:01:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.611+0000 7f6d8e5b0700 1 -- 172.21.15.32:0/4262297927 wait complete. 2024-04-12T06:01:06.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.714+0000 7fa6e6422700 1 -- 172.21.15.32:0/1536458100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6e0101d70 msgr2=0x7fa6e0102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:06.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.714+0000 7fa6e6422700 1 --2- 172.21.15.32:0/1536458100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6e0101d70 0x7fa6e0102140 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0004940 tx=0x7fa6d0009e30 comp rx=0 tx=0).stop 2024-04-12T06:01:06.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.714+0000 7fa6e6422700 1 -- 172.21.15.32:0/1536458100 shutdown_connections 2024-04-12T06:01:06.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.714+0000 7fa6e6422700 1 --2- 172.21.15.32:0/1536458100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6e0102710 0x7fa6e00fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:06.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.714+0000 7fa6e6422700 1 --2- 172.21.15.32:0/1536458100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6e0101d70 0x7fa6e0102140 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:06.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.714+0000 7fa6e6422700 1 -- 172.21.15.32:0/1536458100 >> 172.21.15.32:0/1536458100 conn(0x7fa6e0072bc0 msgr2=0x7fa6e0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:01:06.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.714+0000 7fa6e6422700 1 -- 172.21.15.32:0/1536458100 shutdown_connections 2024-04-12T06:01:06.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.714+0000 7fa6e6422700 1 -- 172.21.15.32:0/1536458100 wait complete. 2024-04-12T06:01:06.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.714+0000 7fa6e6422700 1 Processor -- start 2024-04-12T06:01:06.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.714+0000 7fa6e6422700 1 -- start start 2024-04-12T06:01:06.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.715+0000 7fa6e6422700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6e0101d70 0x7fa6e010ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:06.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.715+0000 7fa6e6422700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6e0102710 0x7fa6e010ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:06.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.715+0000 7fa6e6422700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e010f5d0 con 0x7fa6e0101d70 2024-04-12T06:01:06.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.715+0000 7fa6e6422700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e010f710 con 0x7fa6e0102710 2024-04-12T06:01:06.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.715+0000 7fa6dffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6e0101d70 0x7fa6e010ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:06.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.715+0000 7fa6dffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6e0101d70 0x7fa6e010ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57006/0 (socket says 172.21.15.32:57006) 2024-04-12T06:01:06.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.715+0000 7fa6dffff700 1 -- 172.21.15.32:0/3007403387 learned_addr learned my addr 172.21.15.32:0/3007403387 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:01:06.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.715+0000 7fa6df7fe700 1 --2- 172.21.15.32:0/3007403387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6e0102710 0x7fa6e010ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:06.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.715+0000 7fa6dffff700 1 -- 172.21.15.32:0/3007403387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6e0102710 msgr2=0x7fa6e010ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:06.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.715+0000 7fa6dffff700 1 --2- 172.21.15.32:0/3007403387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6e0102710 0x7fa6e010ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:06.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.715+0000 7fa6dffff700 1 -- 172.21.15.32:0/3007403387 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d0005040 con 0x7fa6e0101d70 2024-04-12T06:01:06.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.715+0000 7fa6dffff700 1 --2- 172.21.15.32:0/3007403387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6e0101d70 0x7fa6e010ea40 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0000c00 tx=0x7fa6d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:01:06.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.715+0000 7fa6dcff9700 1 -- 172.21.15.32:0/3007403387 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6d000e040 con 0x7fa6e0101d70 2024-04-12T06:01:06.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.715+0000 7fa6dcff9700 1 -- 172.21.15.32:0/3007403387 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa6d0013420 con 0x7fa6e0101d70 2024-04-12T06:01:06.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.716+0000 7fa6e6422700 1 -- 172.21.15.32:0/3007403387 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6e0117ab0 con 0x7fa6e0101d70 2024-04-12T06:01:06.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.716+0000 7fa6dcff9700 1 -- 172.21.15.32:0/3007403387 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6d001c630 con 0x7fa6e0101d70 2024-04-12T06:01:06.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.716+0000 7fa6e6422700 1 -- 172.21.15.32:0/3007403387 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e0117fc0 con 0x7fa6e0101d70 2024-04-12T06:01:06.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.717+0000 7fa6e6422700 1 -- 172.21.15.32:0/3007403387 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6e00ff080 con 0x7fa6e0101d70 2024-04-12T06:01:06.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.717+0000 7fa6dcff9700 1 -- 172.21.15.32:0/3007403387 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa6d0013590 con 0x7fa6e0101d70 2024-04-12T06:01:06.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.718+0000 7fa6dcff9700 1 --2- 172.21.15.32:0/3007403387 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa6c806cc20 0x7fa6c806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:06.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.718+0000 7fa6dcff9700 1 -- 172.21.15.32:0/3007403387 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fa6d008d150 con 0x7fa6e0101d70 2024-04-12T06:01:06.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.718+0000 7fa6df7fe700 1 --2- 172.21.15.32:0/3007403387 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa6c806cc20 0x7fa6c806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:06.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.718+0000 7fa6df7fe700 1 --2- 172.21.15.32:0/3007403387 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa6c806cc20 0x7fa6c806f0d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0066000 tx=0x7fa6d400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:01:06.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:06.722+0000 7fa6dcff9700 1 -- 172.21.15.32:0/3007403387 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa6d0057280 con 0x7fa6e0101d70 2024-04-12T06:01:07.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:07.009+0000 7fa6e6422700 1 -- 172.21.15.32:0/3007403387 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa6e00fd4c0 con 0x7fa6e0101d70 2024-04-12T06:01:07.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:07.010+0000 7fa6dcff9700 1 -- 172.21.15.32:0/3007403387 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fa6d005a8a0 con 0x7fa6e0101d70 2024-04-12T06:01:07.004 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T06:01:07.004 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T06:01:07.004 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-12T06:01:07.004 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T06:01:07.004 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T06:01:07.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:07.012+0000 7fa6e6422700 1 -- 172.21.15.32:0/3007403387 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa6c806cc20 msgr2=0x7fa6c806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:07.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:07.012+0000 7fa6e6422700 1 --2- 172.21.15.32:0/3007403387 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa6c806cc20 0x7fa6c806f0d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0066000 tx=0x7fa6d400b040 comp rx=0 tx=0).stop 2024-04-12T06:01:07.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:07.012+0000 7fa6e6422700 1 -- 172.21.15.32:0/3007403387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6e0101d70 msgr2=0x7fa6e010ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:07.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:07.012+0000 7fa6e6422700 1 --2- 172.21.15.32:0/3007403387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6e0101d70 0x7fa6e010ea40 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0000c00 tx=0x7fa6d0009e30 comp rx=0 tx=0).stop 2024-04-12T06:01:07.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:07.012+0000 7fa6e6422700 1 -- 172.21.15.32:0/3007403387 shutdown_connections 2024-04-12T06:01:07.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:07.012+0000 7fa6e6422700 1 --2- 172.21.15.32:0/3007403387 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa6c806cc20 0x7fa6c806f0d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:07.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:07.012+0000 7fa6e6422700 1 --2- 172.21.15.32:0/3007403387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6e0101d70 0x7fa6e010ea40 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:07.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:07.012+0000 7fa6e6422700 1 --2- 172.21.15.32:0/3007403387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6e0102710 0x7fa6e010ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:07.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:07.012+0000 7fa6e6422700 1 -- 172.21.15.32:0/3007403387 >> 172.21.15.32:0/3007403387 conn(0x7fa6e0072bc0 msgr2=0x7fa6e00fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:01:07.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:07.012+0000 7fa6e6422700 1 -- 172.21.15.32:0/3007403387 shutdown_connections 2024-04-12T06:01:07.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:07.012+0000 7fa6e6422700 1 -- 172.21.15.32:0/3007403387 wait complete. 2024-04-12T06:01:07.515 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:07 smithi067 ceph-mon[112979]: pgmap v618: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:07.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:07 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/113958911' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T06:01:07.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:07 smithi067 ceph-mon[112979]: from='client.35172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:01:07.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:07 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3007403387' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T06:01:07.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:07 smithi032 ceph-mon[146817]: pgmap v618: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:07.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:07 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/113958911' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T06:01:07.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:07 smithi032 ceph-mon[146817]: from='client.35172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:01:07.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:07 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3007403387' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T06:01:09.515 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:09 smithi067 ceph-mon[112979]: pgmap v619: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:09.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:09 smithi032 ceph-mon[146817]: pgmap v619: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:11.515 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:11 smithi067 ceph-mon[112979]: pgmap v620: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:11.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:11 smithi032 ceph-mon[146817]: pgmap v620: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:13.515 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:13 smithi067 ceph-mon[112979]: pgmap v621: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:13.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:13 smithi032 ceph-mon[146817]: pgmap v621: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:15.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:15 smithi067 ceph-mon[112979]: pgmap v622: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:15.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:15 smithi032 ceph-mon[146817]: pgmap v622: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:17.515 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:17 smithi067 ceph-mon[112979]: pgmap v623: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:17.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:17 smithi032 ceph-mon[146817]: pgmap v623: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:18.515 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:01:18.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:01:19.515 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:19 smithi067 ceph-mon[112979]: pgmap v624: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:19.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:19 smithi032 ceph-mon[146817]: pgmap v624: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:21.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:21 smithi067 ceph-mon[112979]: pgmap v625: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:21.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:21 smithi032 ceph-mon[146817]: pgmap v625: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:23.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:23 smithi067 ceph-mon[112979]: pgmap v626: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:23.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:23 smithi032 ceph-mon[146817]: pgmap v626: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:25.515 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:25 smithi067 ceph-mon[112979]: pgmap v627: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:25.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:25 smithi032 ceph-mon[146817]: pgmap v627: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:27.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:27 smithi067 ceph-mon[112979]: pgmap v628: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:27.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:27 smithi032 ceph-mon[146817]: pgmap v628: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:29.515 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:29 smithi067 ceph-mon[112979]: pgmap v629: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:29.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:29 smithi032 ceph-mon[146817]: pgmap v629: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:31.515 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:31 smithi067 ceph-mon[112979]: pgmap v630: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:31.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:31 smithi032 ceph-mon[146817]: pgmap v630: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:33.515 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:33 smithi067 ceph-mon[112979]: pgmap v631: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:33.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:33 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:01:33.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:33 smithi032 ceph-mon[146817]: pgmap v631: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:33.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:33 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:01:35.515 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:35 smithi067 ceph-mon[112979]: pgmap v632: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:35.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:35 smithi032 ceph-mon[146817]: pgmap v632: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:37.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.128+0000 7ff3dd569700 1 -- 172.21.15.32:0/258755950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3d81037b0 msgr2=0x7ff3d80fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:37.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.128+0000 7ff3dd569700 1 --2- 172.21.15.32:0/258755950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3d81037b0 0x7ff3d80fce20 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0004940 tx=0x7ff3c0009e30 comp rx=0 tx=0).stop 2024-04-12T06:01:37.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.128+0000 7ff3dd569700 1 -- 172.21.15.32:0/258755950 shutdown_connections 2024-04-12T06:01:37.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.128+0000 7ff3dd569700 1 --2- 172.21.15.32:0/258755950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3d81037b0 0x7ff3d80fce20 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:37.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.128+0000 7ff3dd569700 1 --2- 172.21.15.32:0/258755950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3d8102e10 0x7ff3d81031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:37.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.128+0000 7ff3dd569700 1 -- 172.21.15.32:0/258755950 >> 172.21.15.32:0/258755950 conn(0x7ff3d80f8980 msgr2=0x7ff3d80fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:01:37.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.128+0000 7ff3dd569700 1 -- 172.21.15.32:0/258755950 shutdown_connections 2024-04-12T06:01:37.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.128+0000 7ff3dd569700 1 -- 172.21.15.32:0/258755950 wait complete. 2024-04-12T06:01:37.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.129+0000 7ff3dd569700 1 Processor -- start 2024-04-12T06:01:37.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.129+0000 7ff3dd569700 1 -- start start 2024-04-12T06:01:37.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.129+0000 7ff3dd569700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3d8102e10 0x7ff3d8115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:37.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.129+0000 7ff3dd569700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3d81037b0 0x7ff3d8116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:37.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.129+0000 7ff3dd569700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d8116880 con 0x7ff3d8102e10 2024-04-12T06:01:37.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.129+0000 7ff3dd569700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d810fc50 con 0x7ff3d81037b0 2024-04-12T06:01:37.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.129+0000 7ff3d6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3d8102e10 0x7ff3d8115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:37.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.129+0000 7ff3d6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3d8102e10 0x7ff3d8115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60484/0 (socket says 172.21.15.32:60484) 2024-04-12T06:01:37.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.129+0000 7ff3d6ffd700 1 -- 172.21.15.32:0/2987487777 learned_addr learned my addr 172.21.15.32:0/2987487777 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:01:37.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.129+0000 7ff3d67fc700 1 --2- 172.21.15.32:0/2987487777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3d81037b0 0x7ff3d8116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:37.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.130+0000 7ff3d6ffd700 1 -- 172.21.15.32:0/2987487777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3d81037b0 msgr2=0x7ff3d8116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:37.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.130+0000 7ff3d6ffd700 1 --2- 172.21.15.32:0/2987487777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3d81037b0 0x7ff3d8116230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:37.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.130+0000 7ff3d6ffd700 1 -- 172.21.15.32:0/2987487777 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c0005040 con 0x7ff3d8102e10 2024-04-12T06:01:37.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.130+0000 7ff3d6ffd700 1 --2- 172.21.15.32:0/2987487777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3d8102e10 0x7ff3d8115cf0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7ff3c800cac0 tx=0x7ff3c8007ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:01:37.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.130+0000 7ff3cf7fe700 1 -- 172.21.15.32:0/2987487777 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3c800a980 con 0x7ff3d8102e10 2024-04-12T06:01:37.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.130+0000 7ff3cf7fe700 1 -- 172.21.15.32:0/2987487777 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff3c8012440 con 0x7ff3d8102e10 2024-04-12T06:01:37.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.130+0000 7ff3dd569700 1 -- 172.21.15.32:0/2987487777 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3d810ff30 con 0x7ff3d8102e10 2024-04-12T06:01:37.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.130+0000 7ff3cf7fe700 1 -- 172.21.15.32:0/2987487777 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3c80115e0 con 0x7ff3d8102e10 2024-04-12T06:01:37.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.130+0000 7ff3dd569700 1 -- 172.21.15.32:0/2987487777 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d81104f0 con 0x7ff3d8102e10 2024-04-12T06:01:37.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.131+0000 7ff3dd569700 1 -- 172.21.15.32:0/2987487777 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3d80fe560 con 0x7ff3d8102e10 2024-04-12T06:01:37.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.132+0000 7ff3cf7fe700 1 -- 172.21.15.32:0/2987487777 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ff3c80125b0 con 0x7ff3d8102e10 2024-04-12T06:01:37.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.132+0000 7ff3cf7fe700 1 --2- 172.21.15.32:0/2987487777 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff3c406cc20 0x7ff3c406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:37.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.132+0000 7ff3cf7fe700 1 -- 172.21.15.32:0/2987487777 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7ff3c808c6c0 con 0x7ff3d8102e10 2024-04-12T06:01:37.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.132+0000 7ff3d67fc700 1 --2- 172.21.15.32:0/2987487777 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff3c406cc20 0x7ff3c406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:37.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.133+0000 7ff3d67fc700 1 --2- 172.21.15.32:0/2987487777 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff3c406cc20 0x7ff3c406f0d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8111590 tx=0x7ff3c0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:01:37.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.137+0000 7ff3cf7fe700 1 -- 172.21.15.32:0/2987487777 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff3c805a130 con 0x7ff3d8102e10 2024-04-12T06:01:37.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.361+0000 7ff3dd569700 1 -- 172.21.15.32:0/2987487777 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3d804bed0 con 0x7ff3c406cc20 2024-04-12T06:01:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.362+0000 7ff3cf7fe700 1 -- 172.21.15.32:0/2987487777 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff3d804bed0 con 0x7ff3c406cc20 2024-04-12T06:01:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.364+0000 7ff3dd569700 1 -- 172.21.15.32:0/2987487777 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff3c406cc20 msgr2=0x7ff3c406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.364+0000 7ff3dd569700 1 --2- 172.21.15.32:0/2987487777 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff3c406cc20 0x7ff3c406f0d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8111590 tx=0x7ff3c0009e30 comp rx=0 tx=0).stop 2024-04-12T06:01:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.364+0000 7ff3dd569700 1 -- 172.21.15.32:0/2987487777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3d8102e10 msgr2=0x7ff3d8115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.364+0000 7ff3dd569700 1 --2- 172.21.15.32:0/2987487777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3d8102e10 0x7ff3d8115cf0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7ff3c800cac0 tx=0x7ff3c8007ee0 comp rx=0 tx=0).stop 2024-04-12T06:01:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.364+0000 7ff3dd569700 1 -- 172.21.15.32:0/2987487777 shutdown_connections 2024-04-12T06:01:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.364+0000 7ff3dd569700 1 --2- 172.21.15.32:0/2987487777 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff3c406cc20 0x7ff3c406f0d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:37.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.364+0000 7ff3dd569700 1 --2- 172.21.15.32:0/2987487777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3d8102e10 0x7ff3d8115cf0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:37.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.364+0000 7ff3dd569700 1 --2- 172.21.15.32:0/2987487777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3d81037b0 0x7ff3d8116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:37.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.364+0000 7ff3dd569700 1 -- 172.21.15.32:0/2987487777 >> 172.21.15.32:0/2987487777 conn(0x7ff3d80f8980 msgr2=0x7ff3d8070560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:01:37.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.364+0000 7ff3dd569700 1 -- 172.21.15.32:0/2987487777 shutdown_connections 2024-04-12T06:01:37.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.364+0000 7ff3dd569700 1 -- 172.21.15.32:0/2987487777 wait complete. 2024-04-12T06:01:37.369 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T06:01:37.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.468+0000 7fac630c8700 1 -- 172.21.15.32:0/1008282337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac5c102650 msgr2=0x7fac5c0fbcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:37.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.468+0000 7fac630c8700 1 --2- 172.21.15.32:0/1008282337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac5c102650 0x7fac5c0fbcc0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fac48004940 tx=0x7fac48009e30 comp rx=0 tx=0).stop 2024-04-12T06:01:37.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.468+0000 7fac630c8700 1 -- 172.21.15.32:0/1008282337 shutdown_connections 2024-04-12T06:01:37.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.468+0000 7fac630c8700 1 --2- 172.21.15.32:0/1008282337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac5c102650 0x7fac5c0fbcc0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:37.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.468+0000 7fac630c8700 1 --2- 172.21.15.32:0/1008282337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac5c101cb0 0x7fac5c102080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:37.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.468+0000 7fac630c8700 1 -- 172.21.15.32:0/1008282337 >> 172.21.15.32:0/1008282337 conn(0x7fac5c0f7860 msgr2=0x7fac5c0f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:01:37.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.468+0000 7fac630c8700 1 -- 172.21.15.32:0/1008282337 shutdown_connections 2024-04-12T06:01:37.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.468+0000 7fac630c8700 1 -- 172.21.15.32:0/1008282337 wait complete. 2024-04-12T06:01:37.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.469+0000 7fac630c8700 1 Processor -- start 2024-04-12T06:01:37.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.469+0000 7fac630c8700 1 -- start start 2024-04-12T06:01:37.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.469+0000 7fac630c8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac5c101cb0 0x7fac5c113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:37.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.469+0000 7fac630c8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac5c102650 0x7fac5c113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:37.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.469+0000 7fac630c8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c1145d0 con 0x7fac5c101cb0 2024-04-12T06:01:37.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.469+0000 7fac630c8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c10d9c0 con 0x7fac5c102650 2024-04-12T06:01:37.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.469+0000 7fac60e64700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac5c101cb0 0x7fac5c113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:37.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.469+0000 7fac60e64700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac5c101cb0 0x7fac5c113a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60494/0 (socket says 172.21.15.32:60494) 2024-04-12T06:01:37.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.469+0000 7fac60e64700 1 -- 172.21.15.32:0/3390477556 learned_addr learned my addr 172.21.15.32:0/3390477556 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:01:37.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.469+0000 7fac5bfff700 1 --2- 172.21.15.32:0/3390477556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac5c102650 0x7fac5c113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:37.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.470+0000 7fac60e64700 1 -- 172.21.15.32:0/3390477556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac5c102650 msgr2=0x7fac5c113f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:37.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.470+0000 7fac60e64700 1 --2- 172.21.15.32:0/3390477556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac5c102650 0x7fac5c113f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:37.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.470+0000 7fac60e64700 1 -- 172.21.15.32:0/3390477556 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac48005040 con 0x7fac5c101cb0 2024-04-12T06:01:37.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.470+0000 7fac60e64700 1 --2- 172.21.15.32:0/3390477556 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac5c101cb0 0x7fac5c113a40 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fac50002a50 tx=0x7fac500046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:01:37.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.470+0000 7fac597fa700 1 -- 172.21.15.32:0/3390477556 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac50007cc0 con 0x7fac5c101cb0 2024-04-12T06:01:37.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.470+0000 7fac630c8700 1 -- 172.21.15.32:0/3390477556 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac5c10dca0 con 0x7fac5c101cb0 2024-04-12T06:01:37.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.470+0000 7fac597fa700 1 -- 172.21.15.32:0/3390477556 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac5000f040 con 0x7fac5c101cb0 2024-04-12T06:01:37.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.470+0000 7fac597fa700 1 -- 172.21.15.32:0/3390477556 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac5000d9c0 con 0x7fac5c101cb0 2024-04-12T06:01:37.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.470+0000 7fac630c8700 1 -- 172.21.15.32:0/3390477556 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac5c10e230 con 0x7fac5c101cb0 2024-04-12T06:01:37.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.471+0000 7fac630c8700 1 -- 172.21.15.32:0/3390477556 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac5c0fd400 con 0x7fac5c101cb0 2024-04-12T06:01:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.471+0000 7fac597fa700 1 -- 172.21.15.32:0/3390477556 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fac5000db20 con 0x7fac5c101cb0 2024-04-12T06:01:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.472+0000 7fac597fa700 1 --2- 172.21.15.32:0/3390477556 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fac4c06cc20 0x7fac4c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.472+0000 7fac597fa700 1 -- 172.21.15.32:0/3390477556 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fac5008bdd0 con 0x7fac5c101cb0 2024-04-12T06:01:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.472+0000 7fac5bfff700 1 --2- 172.21.15.32:0/3390477556 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fac4c06cc20 0x7fac4c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.472+0000 7fac5bfff700 1 --2- 172.21.15.32:0/3390477556 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fac4c06cc20 0x7fac4c06f0d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fac48004890 tx=0x7fac48009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:01:37.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.477+0000 7fac597fa700 1 -- 172.21.15.32:0/3390477556 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fac500597c0 con 0x7fac5c101cb0 2024-04-12T06:01:37.515 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:37 smithi067 ceph-mon[112979]: pgmap v633: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:37.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:37 smithi032 ceph-mon[146817]: pgmap v633: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:37.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.699+0000 7fac630c8700 1 -- 172.21.15.32:0/3390477556 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fac5c108c40 con 0x7fac4c06cc20 2024-04-12T06:01:37.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.700+0000 7fac597fa700 1 -- 172.21.15.32:0/3390477556 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fac5c108c40 con 0x7fac4c06cc20 2024-04-12T06:01:37.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.702+0000 7fac630c8700 1 -- 172.21.15.32:0/3390477556 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fac4c06cc20 msgr2=0x7fac4c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:37.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.702+0000 7fac630c8700 1 --2- 172.21.15.32:0/3390477556 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fac4c06cc20 0x7fac4c06f0d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fac48004890 tx=0x7fac48009e30 comp rx=0 tx=0).stop 2024-04-12T06:01:37.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.702+0000 7fac630c8700 1 -- 172.21.15.32:0/3390477556 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac5c101cb0 msgr2=0x7fac5c113a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:37.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.702+0000 7fac630c8700 1 --2- 172.21.15.32:0/3390477556 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac5c101cb0 0x7fac5c113a40 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fac50002a50 tx=0x7fac500046b0 comp rx=0 tx=0).stop 2024-04-12T06:01:37.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.702+0000 7fac630c8700 1 -- 172.21.15.32:0/3390477556 shutdown_connections 2024-04-12T06:01:37.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.702+0000 7fac630c8700 1 --2- 172.21.15.32:0/3390477556 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fac4c06cc20 0x7fac4c06f0d0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:37.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.702+0000 7fac630c8700 1 --2- 172.21.15.32:0/3390477556 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac5c101cb0 0x7fac5c113a40 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:37.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.702+0000 7fac630c8700 1 --2- 172.21.15.32:0/3390477556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac5c102650 0x7fac5c113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:37.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.702+0000 7fac630c8700 1 -- 172.21.15.32:0/3390477556 >> 172.21.15.32:0/3390477556 conn(0x7fac5c0f7860 msgr2=0x7fac5c065ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:01:37.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.702+0000 7fac630c8700 1 -- 172.21.15.32:0/3390477556 shutdown_connections 2024-04-12T06:01:37.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.702+0000 7fac630c8700 1 -- 172.21.15.32:0/3390477556 wait complete. 2024-04-12T06:01:37.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.805+0000 7fa993374700 1 -- 172.21.15.32:0/1687752044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa98c0fbc00 msgr2=0x7fa98c0fbfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:37.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.805+0000 7fa993374700 1 --2- 172.21.15.32:0/1687752044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa98c0fbc00 0x7fa98c0fbfd0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fa988004940 tx=0x7fa988009e30 comp rx=0 tx=0).stop 2024-04-12T06:01:37.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.805+0000 7fa993374700 1 -- 172.21.15.32:0/1687752044 shutdown_connections 2024-04-12T06:01:37.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.805+0000 7fa993374700 1 --2- 172.21.15.32:0/1687752044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa98c0fc510 0x7fa98c109270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:37.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.805+0000 7fa993374700 1 --2- 172.21.15.32:0/1687752044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa98c0fbc00 0x7fa98c0fbfd0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:37.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.806+0000 7fa993374700 1 -- 172.21.15.32:0/1687752044 >> 172.21.15.32:0/1687752044 conn(0x7fa98c0f78d0 msgr2=0x7fa98c0f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:01:37.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.806+0000 7fa993374700 1 -- 172.21.15.32:0/1687752044 shutdown_connections 2024-04-12T06:01:37.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.806+0000 7fa993374700 1 -- 172.21.15.32:0/1687752044 wait complete. 2024-04-12T06:01:37.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.806+0000 7fa993374700 1 Processor -- start 2024-04-12T06:01:37.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.806+0000 7fa993374700 1 -- start start 2024-04-12T06:01:37.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.806+0000 7fa993374700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa98c0fbc00 0x7fa98c115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:37.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.806+0000 7fa993374700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa98c0fc510 0x7fa98c116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:37.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.806+0000 7fa993374700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c116850 con 0x7fa98c0fbc00 2024-04-12T06:01:37.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.806+0000 7fa993374700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c10fc20 con 0x7fa98c0fc510 2024-04-12T06:01:37.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.807+0000 7fa991110700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa98c0fbc00 0x7fa98c115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:37.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.807+0000 7fa991110700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa98c0fbc00 0x7fa98c115cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60504/0 (socket says 172.21.15.32:60504) 2024-04-12T06:01:37.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.807+0000 7fa991110700 1 -- 172.21.15.32:0/1557204672 learned_addr learned my addr 172.21.15.32:0/1557204672 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:01:37.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.807+0000 7fa99090f700 1 --2- 172.21.15.32:0/1557204672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa98c0fc510 0x7fa98c116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:37.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.807+0000 7fa991110700 1 -- 172.21.15.32:0/1557204672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa98c0fc510 msgr2=0x7fa98c116200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:37.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.807+0000 7fa991110700 1 --2- 172.21.15.32:0/1557204672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa98c0fc510 0x7fa98c116200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:37.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.807+0000 7fa991110700 1 -- 172.21.15.32:0/1557204672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa988005040 con 0x7fa98c0fbc00 2024-04-12T06:01:37.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.807+0000 7fa991110700 1 --2- 172.21.15.32:0/1557204672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa98c0fbc00 0x7fa98c115cc0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fa98800b3b0 tx=0x7fa988009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:01:37.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.807+0000 7fa981ffb700 1 -- 172.21.15.32:0/1557204672 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa98800e040 con 0x7fa98c0fbc00 2024-04-12T06:01:37.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.807+0000 7fa993374700 1 -- 172.21.15.32:0/1557204672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa98c10fea0 con 0x7fa98c0fbc00 2024-04-12T06:01:37.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.807+0000 7fa981ffb700 1 -- 172.21.15.32:0/1557204672 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa98800bed0 con 0x7fa98c0fbc00 2024-04-12T06:01:37.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.807+0000 7fa981ffb700 1 -- 172.21.15.32:0/1557204672 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa988013db0 con 0x7fa98c0fbc00 2024-04-12T06:01:37.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.807+0000 7fa993374700 1 -- 172.21.15.32:0/1557204672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa98c110460 con 0x7fa98c0fbc00 2024-04-12T06:01:37.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.808+0000 7fa993374700 1 -- 172.21.15.32:0/1557204672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa98c108f10 con 0x7fa98c0fbc00 2024-04-12T06:01:37.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.809+0000 7fa981ffb700 1 -- 172.21.15.32:0/1557204672 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa98800ba20 con 0x7fa98c0fbc00 2024-04-12T06:01:37.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.809+0000 7fa981ffb700 1 --2- 172.21.15.32:0/1557204672 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa97806cc20 0x7fa97806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:37.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.809+0000 7fa981ffb700 1 -- 172.21.15.32:0/1557204672 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fa98808d2a0 con 0x7fa98c0fbc00 2024-04-12T06:01:37.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.809+0000 7fa99090f700 1 --2- 172.21.15.32:0/1557204672 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa97806cc20 0x7fa97806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:37.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.810+0000 7fa99090f700 1 --2- 172.21.15.32:0/1557204672 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa97806cc20 0x7fa97806f0d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa98c111560 tx=0x7fa97c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:01:37.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:37.814+0000 7fa981ffb700 1 -- 172.21.15.32:0/1557204672 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa98805ad10 con 0x7fa98c0fbc00 2024-04-12T06:01:38.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.026+0000 7fa993374700 1 -- 172.21.15.32:0/1557204672 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa98c10bfe0 con 0x7fa97806cc20 2024-04-12T06:01:38.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.036+0000 7fa981ffb700 1 -- 172.21.15.32:0/1557204672 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fa98c10bfe0 con 0x7fa97806cc20 2024-04-12T06:01:38.031 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T06:01:38.031 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (21m) 14m ago 32m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T06:01:38.031 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (18m) 14m ago 33m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T06:01:38.031 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (18m) 14m ago 32m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T06:01:38.031 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (21m) 14m ago 32m 87.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T06:01:38.031 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (15m) 14m ago 27m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T06:01:38.031 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (14m) 14m ago 27m 13.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T06:01:38.031 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (14m) 14m ago 27m 11.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 6d4aca6f74a0 2024-04-12T06:01:38.031 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (14m) 14m ago 27m 14.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T06:01:38.031 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (23m) 14m ago 34m 507M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T06:01:38.031 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (22m) 14m ago 31m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T06:01:38.031 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (20m) 14m ago 34m 53.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T06:01:38.032 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (19m) 14m ago 31m 43.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T06:01:38.032 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (22m) 14m ago 33m 16.9M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T06:01:38.032 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (22m) 14m ago 31m 18.4M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T06:01:38.032 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (18m) 14m ago 31m 91.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T06:01:38.032 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (17m) 14m ago 30m 97.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T06:01:38.032 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (17m) 14m ago 30m 81.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T06:01:38.032 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (16m) 14m ago 29m 113M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T06:01:38.032 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (16m) 14m ago 29m 90.3M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T06:01:38.032 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (15m) 14m ago 28m 84.8M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T06:01:38.032 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (21m) 14m ago 32m 64.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T06:01:38.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.038+0000 7fa993374700 1 -- 172.21.15.32:0/1557204672 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa97806cc20 msgr2=0x7fa97806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:38.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.038+0000 7fa993374700 1 --2- 172.21.15.32:0/1557204672 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa97806cc20 0x7fa97806f0d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa98c111560 tx=0x7fa97c00b040 comp rx=0 tx=0).stop 2024-04-12T06:01:38.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.038+0000 7fa993374700 1 -- 172.21.15.32:0/1557204672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa98c0fbc00 msgr2=0x7fa98c115cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:38.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.038+0000 7fa993374700 1 --2- 172.21.15.32:0/1557204672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa98c0fbc00 0x7fa98c115cc0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fa98800b3b0 tx=0x7fa988009e30 comp rx=0 tx=0).stop 2024-04-12T06:01:38.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.038+0000 7fa993374700 1 -- 172.21.15.32:0/1557204672 shutdown_connections 2024-04-12T06:01:38.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.038+0000 7fa993374700 1 --2- 172.21.15.32:0/1557204672 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa97806cc20 0x7fa97806f0d0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:38.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.038+0000 7fa993374700 1 --2- 172.21.15.32:0/1557204672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa98c0fbc00 0x7fa98c115cc0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:38.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.038+0000 7fa993374700 1 --2- 172.21.15.32:0/1557204672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa98c0fc510 0x7fa98c116200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:38.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.038+0000 7fa993374700 1 -- 172.21.15.32:0/1557204672 >> 172.21.15.32:0/1557204672 conn(0x7fa98c0f78d0 msgr2=0x7fa98c0f8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:01:38.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.038+0000 7fa993374700 1 -- 172.21.15.32:0/1557204672 shutdown_connections 2024-04-12T06:01:38.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.038+0000 7fa993374700 1 -- 172.21.15.32:0/1557204672 wait complete. 2024-04-12T06:01:38.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.141+0000 7f17fa32c700 1 -- 172.21.15.32:0/933714476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f40fde90 msgr2=0x7f17f40fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:38.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.141+0000 7f17fa32c700 1 --2- 172.21.15.32:0/933714476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f40fde90 0x7f17f40fe260 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f17e4004940 tx=0x7f17e4009e30 comp rx=0 tx=0).stop 2024-04-12T06:01:38.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.141+0000 7f17fa32c700 1 -- 172.21.15.32:0/933714476 shutdown_connections 2024-04-12T06:01:38.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.141+0000 7f17fa32c700 1 --2- 172.21.15.32:0/933714476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17f40fe830 0x7f17f4106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:38.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.141+0000 7f17fa32c700 1 --2- 172.21.15.32:0/933714476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f40fde90 0x7f17f40fe260 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:38.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.141+0000 7f17fa32c700 1 -- 172.21.15.32:0/933714476 >> 172.21.15.32:0/933714476 conn(0x7f17f4072bc0 msgr2=0x7f17f4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:01:38.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.141+0000 7f17fa32c700 1 -- 172.21.15.32:0/933714476 shutdown_connections 2024-04-12T06:01:38.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.141+0000 7f17fa32c700 1 -- 172.21.15.32:0/933714476 wait complete. 2024-04-12T06:01:38.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.141+0000 7f17fa32c700 1 Processor -- start 2024-04-12T06:01:38.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.141+0000 7f17fa32c700 1 -- start start 2024-04-12T06:01:38.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.142+0000 7f17fa32c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f40fde90 0x7f17f4113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:38.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.142+0000 7f17fa32c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17f40fe830 0x7f17f4114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:38.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.142+0000 7f17fa32c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f4114680 con 0x7f17f40fde90 2024-04-12T06:01:38.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.142+0000 7f17fa32c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f410da70 con 0x7f17f40fe830 2024-04-12T06:01:38.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.142+0000 7f17f3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f40fde90 0x7f17f4113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:38.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.142+0000 7f17f3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f40fde90 0x7f17f4113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60518/0 (socket says 172.21.15.32:60518) 2024-04-12T06:01:38.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.142+0000 7f17f3fff700 1 -- 172.21.15.32:0/3598535602 learned_addr learned my addr 172.21.15.32:0/3598535602 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:01:38.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.142+0000 7f17f37fe700 1 --2- 172.21.15.32:0/3598535602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17f40fe830 0x7f17f4114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:38.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.142+0000 7f17f3fff700 1 -- 172.21.15.32:0/3598535602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17f40fe830 msgr2=0x7f17f4114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:38.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.142+0000 7f17f3fff700 1 --2- 172.21.15.32:0/3598535602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17f40fe830 0x7f17f4114030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:38.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.142+0000 7f17f3fff700 1 -- 172.21.15.32:0/3598535602 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e4005040 con 0x7f17f40fde90 2024-04-12T06:01:38.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.142+0000 7f17f3fff700 1 --2- 172.21.15.32:0/3598535602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f40fde90 0x7f17f4113af0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f17e40093b0 tx=0x7f17e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:01:38.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.142+0000 7f17f0ff9700 1 -- 172.21.15.32:0/3598535602 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17e400e040 con 0x7f17f40fde90 2024-04-12T06:01:38.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.143+0000 7f17fa32c700 1 -- 172.21.15.32:0/3598535602 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f410dcf0 con 0x7f17f40fde90 2024-04-12T06:01:38.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.143+0000 7f17f0ff9700 1 -- 172.21.15.32:0/3598535602 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f17e400bed0 con 0x7f17f40fde90 2024-04-12T06:01:38.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.143+0000 7f17f0ff9700 1 -- 172.21.15.32:0/3598535602 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17e401c3f0 con 0x7f17f40fde90 2024-04-12T06:01:38.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.143+0000 7f17fa32c700 1 -- 172.21.15.32:0/3598535602 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f410e230 con 0x7f17f40fde90 2024-04-12T06:01:38.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.144+0000 7f17fa32c700 1 -- 172.21.15.32:0/3598535602 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f41019b0 con 0x7f17f40fde90 2024-04-12T06:01:38.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.144+0000 7f17f0ff9700 1 -- 172.21.15.32:0/3598535602 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f17e4013790 con 0x7f17f40fde90 2024-04-12T06:01:38.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.145+0000 7f17f0ff9700 1 --2- 172.21.15.32:0/3598535602 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f17dc06cc30 0x7f17dc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:38.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.145+0000 7f17f0ff9700 1 -- 172.21.15.32:0/3598535602 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f17e408d480 con 0x7f17f40fde90 2024-04-12T06:01:38.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.145+0000 7f17f37fe700 1 --2- 172.21.15.32:0/3598535602 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f17dc06cc30 0x7f17dc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:38.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.145+0000 7f17f37fe700 1 --2- 172.21.15.32:0/3598535602 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f17dc06cc30 0x7f17dc06f0e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f17f410ec30 tx=0x7f17e800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:01:38.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.149+0000 7f17f0ff9700 1 -- 172.21.15.32:0/3598535602 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f17e405af00 con 0x7f17f40fde90 2024-04-12T06:01:38.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.435+0000 7f17fa32c700 1 -- 172.21.15.32:0/3598535602 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f17f410f140 con 0x7f17f40fde90 2024-04-12T06:01:38.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.436+0000 7f17f0ff9700 1 -- 172.21.15.32:0/3598535602 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f17e4013420 con 0x7f17f40fde90 2024-04-12T06:01:38.431 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T06:01:38.431 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T06:01:38.431 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T06:01:38.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T06:01:38.431 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T06:01:38.431 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T06:01:38.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T06:01:38.431 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T06:01:38.432 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T06:01:38.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T06:01:38.432 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T06:01:38.432 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T06:01:38.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T06:01:38.432 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T06:01:38.432 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T06:01:38.432 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T06:01:38.432 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T06:01:38.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.438+0000 7f17fa32c700 1 -- 172.21.15.32:0/3598535602 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f17dc06cc30 msgr2=0x7f17dc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:38.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.438+0000 7f17fa32c700 1 --2- 172.21.15.32:0/3598535602 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f17dc06cc30 0x7f17dc06f0e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f17f410ec30 tx=0x7f17e800b040 comp rx=0 tx=0).stop 2024-04-12T06:01:38.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.438+0000 7f17fa32c700 1 -- 172.21.15.32:0/3598535602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f40fde90 msgr2=0x7f17f4113af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:38.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.438+0000 7f17fa32c700 1 --2- 172.21.15.32:0/3598535602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f40fde90 0x7f17f4113af0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f17e40093b0 tx=0x7f17e4009e30 comp rx=0 tx=0).stop 2024-04-12T06:01:38.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.438+0000 7f17fa32c700 1 -- 172.21.15.32:0/3598535602 shutdown_connections 2024-04-12T06:01:38.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.438+0000 7f17fa32c700 1 --2- 172.21.15.32:0/3598535602 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f17dc06cc30 0x7f17dc06f0e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:38.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.438+0000 7f17fa32c700 1 --2- 172.21.15.32:0/3598535602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f40fde90 0x7f17f4113af0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:38.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.438+0000 7f17fa32c700 1 --2- 172.21.15.32:0/3598535602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17f40fe830 0x7f17f4114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:38.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.438+0000 7f17fa32c700 1 -- 172.21.15.32:0/3598535602 >> 172.21.15.32:0/3598535602 conn(0x7f17f4072bc0 msgr2=0x7f17f40ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:01:38.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.438+0000 7f17fa32c700 1 -- 172.21.15.32:0/3598535602 shutdown_connections 2024-04-12T06:01:38.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.438+0000 7f17fa32c700 1 -- 172.21.15.32:0/3598535602 wait complete. 2024-04-12T06:01:38.515 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:38 smithi067 ceph-mon[112979]: from='client.35180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:01:38.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:38 smithi067 ceph-mon[112979]: from='client.35184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:01:38.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.541+0000 7f4134f8e700 1 -- 172.21.15.32:0/1473964645 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4130101d70 msgr2=0x7f4130102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:38.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.541+0000 7f4134f8e700 1 --2- 172.21.15.32:0/1473964645 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4130101d70 0x7f4130102140 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f4124004940 tx=0x7f4124009e30 comp rx=0 tx=0).stop 2024-04-12T06:01:38.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.541+0000 7f4134f8e700 1 -- 172.21.15.32:0/1473964645 shutdown_connections 2024-04-12T06:01:38.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.541+0000 7f4134f8e700 1 --2- 172.21.15.32:0/1473964645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4130102710 0x7f41300fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:38.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.541+0000 7f4134f8e700 1 --2- 172.21.15.32:0/1473964645 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4130101d70 0x7f4130102140 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:38.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.541+0000 7f4134f8e700 1 -- 172.21.15.32:0/1473964645 >> 172.21.15.32:0/1473964645 conn(0x7f4130072bc0 msgr2=0x7f4130072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:01:38.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.541+0000 7f4134f8e700 1 -- 172.21.15.32:0/1473964645 shutdown_connections 2024-04-12T06:01:38.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.541+0000 7f4134f8e700 1 -- 172.21.15.32:0/1473964645 wait complete. 2024-04-12T06:01:38.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.541+0000 7f4134f8e700 1 Processor -- start 2024-04-12T06:01:38.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.541+0000 7f4134f8e700 1 -- start start 2024-04-12T06:01:38.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.542+0000 7f4134f8e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4130101d70 0x7f4130113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:38.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.542+0000 7f4134f8e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4130102710 0x7f4130114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:38.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.542+0000 7f4134f8e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4130114650 con 0x7f4130102710 2024-04-12T06:01:38.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.542+0000 7f4134f8e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413010da40 con 0x7f4130101d70 2024-04-12T06:01:38.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.542+0000 7f412dd9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4130102710 0x7f4130114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:38.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.542+0000 7f412dd9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4130102710 0x7f4130114000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60542/0 (socket says 172.21.15.32:60542) 2024-04-12T06:01:38.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.542+0000 7f412dd9b700 1 -- 172.21.15.32:0/2100458325 learned_addr learned my addr 172.21.15.32:0/2100458325 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:01:38.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.542+0000 7f412e59c700 1 --2- 172.21.15.32:0/2100458325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4130101d70 0x7f4130113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:38.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.542+0000 7f412dd9b700 1 -- 172.21.15.32:0/2100458325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4130101d70 msgr2=0x7f4130113ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:38.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.542+0000 7f412dd9b700 1 --2- 172.21.15.32:0/2100458325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4130101d70 0x7f4130113ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:38.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.542+0000 7f412dd9b700 1 -- 172.21.15.32:0/2100458325 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4124005040 con 0x7f4130102710 2024-04-12T06:01:38.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.542+0000 7f412e59c700 1 --2- 172.21.15.32:0/2100458325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4130101d70 0x7f4130113ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T06:01:38.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.542+0000 7f412dd9b700 1 --2- 172.21.15.32:0/2100458325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4130102710 0x7f4130114000 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f4120002a50 tx=0x7f4120004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:01:38.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.542+0000 7f411effd700 1 -- 172.21.15.32:0/2100458325 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4120007cc0 con 0x7f4130102710 2024-04-12T06:01:38.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.542+0000 7f411effd700 1 -- 172.21.15.32:0/2100458325 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f412000f040 con 0x7f4130102710 2024-04-12T06:01:38.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.542+0000 7f4134f8e700 1 -- 172.21.15.32:0/2100458325 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f413010dd20 con 0x7f4130102710 2024-04-12T06:01:38.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.542+0000 7f411effd700 1 -- 172.21.15.32:0/2100458325 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f412000d990 con 0x7f4130102710 2024-04-12T06:01:38.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.542+0000 7f4134f8e700 1 -- 172.21.15.32:0/2100458325 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f413010e2e0 con 0x7f4130102710 2024-04-12T06:01:38.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.543+0000 7f4134f8e700 1 -- 172.21.15.32:0/2100458325 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41300fd4c0 con 0x7f4130102710 2024-04-12T06:01:38.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.544+0000 7f411effd700 1 -- 172.21.15.32:0/2100458325 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4120009710 con 0x7f4130102710 2024-04-12T06:01:38.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.544+0000 7f411effd700 1 --2- 172.21.15.32:0/2100458325 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f411806cc30 0x7f411806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:38.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.544+0000 7f411effd700 1 -- 172.21.15.32:0/2100458325 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f412008bec0 con 0x7f4130102710 2024-04-12T06:01:38.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.544+0000 7f412e59c700 1 --2- 172.21.15.32:0/2100458325 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f411806cc30 0x7f411806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:38.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.545+0000 7f412e59c700 1 --2- 172.21.15.32:0/2100458325 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f411806cc30 0x7f411806f0e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f412400b3b0 tx=0x7f4124009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:01:38.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.549+0000 7f411effd700 1 -- 172.21.15.32:0/2100458325 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4120059620 con 0x7f4130102710 2024-04-12T06:01:38.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:38 smithi032 ceph-mon[146817]: from='client.35180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:01:38.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:38 smithi032 ceph-mon[146817]: from='client.35184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:01:38.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.796+0000 7f4134f8e700 1 -- 172.21.15.32:0/2100458325 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4130108d30 con 0x7f4130102710 2024-04-12T06:01:38.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.796+0000 7f411effd700 1 -- 172.21.15.32:0/2100458325 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1835 (secure 0 0 0) 0x7f4120059440 con 0x7f4130102710 2024-04-12T06:01:38.792 INFO:teuthology.orchestra.run.smithi032.stdout:e33 2024-04-12T06:01:38.792 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T06:01:38.792 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T06:01:38.792 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T06:01:38.792 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:01:38.792 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T06:01:38.792 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T06:01:38.792 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 24 2024-04-12T06:01:38.792 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T06:01:38.792 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T06:01:38.792 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:46:10.668569+0000 2024-04-12T06:01:38.792 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T06:01:38.792 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T06:01:38.793 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T06:01:38.793 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T06:01:38.793 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T06:01:38.793 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T06:01:38.793 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T06:01:38.793 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T06:01:38.793 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T06:01:38.793 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T06:01:38.793 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T06:01:38.793 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T06:01:38.793 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T06:01:38.793 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-12T06:01:38.793 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T06:01:38.793 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T06:01:38.794 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T06:01:38.794 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T06:01:38.794 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T06:01:38.794 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T06:01:38.794 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T06:01:38.794 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T06:01:38.794 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:01:38.794 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:01:38.794 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-12T06:01:38.794 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:01:38.794 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T06:01:38.794 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T06:01:38.794 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{-1:44317} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T06:01:38.794 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T06:01:38.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.799+0000 7f4134f8e700 1 -- 172.21.15.32:0/2100458325 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f411806cc30 msgr2=0x7f411806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:38.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.799+0000 7f4134f8e700 1 --2- 172.21.15.32:0/2100458325 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f411806cc30 0x7f411806f0e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f412400b3b0 tx=0x7f4124009e30 comp rx=0 tx=0).stop 2024-04-12T06:01:38.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.799+0000 7f4134f8e700 1 -- 172.21.15.32:0/2100458325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4130102710 msgr2=0x7f4130114000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:38.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.799+0000 7f4134f8e700 1 --2- 172.21.15.32:0/2100458325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4130102710 0x7f4130114000 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f4120002a50 tx=0x7f4120004600 comp rx=0 tx=0).stop 2024-04-12T06:01:38.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.799+0000 7f4134f8e700 1 -- 172.21.15.32:0/2100458325 shutdown_connections 2024-04-12T06:01:38.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.799+0000 7f4134f8e700 1 --2- 172.21.15.32:0/2100458325 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f411806cc30 0x7f411806f0e0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:38.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.799+0000 7f4134f8e700 1 --2- 172.21.15.32:0/2100458325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4130101d70 0x7f4130113ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:38.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.799+0000 7f4134f8e700 1 --2- 172.21.15.32:0/2100458325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4130102710 0x7f4130114000 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:38.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.799+0000 7f4134f8e700 1 -- 172.21.15.32:0/2100458325 >> 172.21.15.32:0/2100458325 conn(0x7f4130072bc0 msgr2=0x7f41300659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:01:38.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.799+0000 7f4134f8e700 1 -- 172.21.15.32:0/2100458325 shutdown_connections 2024-04-12T06:01:38.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.799+0000 7f4134f8e700 1 -- 172.21.15.32:0/2100458325 wait complete. 2024-04-12T06:01:38.796 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-12T06:01:38.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.903+0000 7f13d989f700 1 -- 172.21.15.32:0/3519169964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13d4101d70 msgr2=0x7f13d4102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:38.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.903+0000 7f13d989f700 1 --2- 172.21.15.32:0/3519169964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13d4101d70 0x7f13d4102140 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f13c4004940 tx=0x7f13c4009e30 comp rx=0 tx=0).stop 2024-04-12T06:01:38.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.903+0000 7f13d989f700 1 -- 172.21.15.32:0/3519169964 shutdown_connections 2024-04-12T06:01:38.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.903+0000 7f13d989f700 1 --2- 172.21.15.32:0/3519169964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13d4102710 0x7f13d40fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:38.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.903+0000 7f13d989f700 1 --2- 172.21.15.32:0/3519169964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13d4101d70 0x7f13d4102140 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:38.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.903+0000 7f13d989f700 1 -- 172.21.15.32:0/3519169964 >> 172.21.15.32:0/3519169964 conn(0x7f13d4072bc0 msgr2=0x7f13d4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:01:38.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.903+0000 7f13d989f700 1 -- 172.21.15.32:0/3519169964 shutdown_connections 2024-04-12T06:01:38.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.903+0000 7f13d989f700 1 -- 172.21.15.32:0/3519169964 wait complete. 2024-04-12T06:01:38.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.904+0000 7f13d989f700 1 Processor -- start 2024-04-12T06:01:38.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.904+0000 7f13d989f700 1 -- start start 2024-04-12T06:01:38.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.904+0000 7f13d989f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13d4101d70 0x7f13d4113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:38.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.904+0000 7f13d989f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13d4102710 0x7f13d4114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:38.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.904+0000 7f13d989f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d4114670 con 0x7f13d4101d70 2024-04-12T06:01:38.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.904+0000 7f13d989f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d410da60 con 0x7f13d4102710 2024-04-12T06:01:38.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.904+0000 7f13d2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13d4101d70 0x7f13d4113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:38.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.904+0000 7f13d2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13d4101d70 0x7f13d4113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60556/0 (socket says 172.21.15.32:60556) 2024-04-12T06:01:38.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.904+0000 7f13d2ffd700 1 -- 172.21.15.32:0/4238427382 learned_addr learned my addr 172.21.15.32:0/4238427382 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:01:38.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.904+0000 7f13d2ffd700 1 -- 172.21.15.32:0/4238427382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13d4102710 msgr2=0x7f13d4114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:38.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.904+0000 7f13d2ffd700 1 --2- 172.21.15.32:0/4238427382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13d4102710 0x7f13d4114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:38.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.904+0000 7f13d2ffd700 1 -- 172.21.15.32:0/4238427382 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c4005040 con 0x7f13d4101d70 2024-04-12T06:01:38.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.905+0000 7f13d2ffd700 1 --2- 172.21.15.32:0/4238427382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13d4101d70 0x7f13d4113ae0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f13c4009360 tx=0x7f13c4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:01:38.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.905+0000 7f13cb7fe700 1 -- 172.21.15.32:0/4238427382 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13c400e040 con 0x7f13d4101d70 2024-04-12T06:01:38.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.905+0000 7f13d989f700 1 -- 172.21.15.32:0/4238427382 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13d410dce0 con 0x7f13d4101d70 2024-04-12T06:01:38.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.905+0000 7f13cb7fe700 1 -- 172.21.15.32:0/4238427382 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f13c4013420 con 0x7f13d4101d70 2024-04-12T06:01:38.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.905+0000 7f13cb7fe700 1 -- 172.21.15.32:0/4238427382 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13c401c5f0 con 0x7f13d4101d70 2024-04-12T06:01:38.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.905+0000 7f13d989f700 1 -- 172.21.15.32:0/4238427382 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13d410e2a0 con 0x7f13d4101d70 2024-04-12T06:01:38.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.906+0000 7f13d989f700 1 -- 172.21.15.32:0/4238427382 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13d40fd4c0 con 0x7f13d4101d70 2024-04-12T06:01:38.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.906+0000 7f13cb7fe700 1 -- 172.21.15.32:0/4238427382 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f13c4013590 con 0x7f13d4101d70 2024-04-12T06:01:38.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.907+0000 7f13cb7fe700 1 --2- 172.21.15.32:0/4238427382 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f13c006cc30 0x7f13c006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:38.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.907+0000 7f13cb7fe700 1 -- 172.21.15.32:0/4238427382 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f13c408d3d0 con 0x7f13d4101d70 2024-04-12T06:01:38.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.907+0000 7f13d27fc700 1 --2- 172.21.15.32:0/4238427382 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f13c006cc30 0x7f13c006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:38.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.907+0000 7f13d27fc700 1 --2- 172.21.15.32:0/4238427382 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f13c006cc30 0x7f13c006f0e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f13d410ec20 tx=0x7f13bc007d40 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:01:38.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:38.911+0000 7f13cb7fe700 1 -- 172.21.15.32:0/4238427382 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f13c405ada0 con 0x7f13d4101d70 2024-04-12T06:01:39.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.135+0000 7f13d989f700 1 -- 172.21.15.32:0/4238427382 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f13d4108dd0 con 0x7f13c006cc30 2024-04-12T06:01:39.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.136+0000 7f13cb7fe700 1 -- 172.21.15.32:0/4238427382 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f13bc00c5f0 con 0x7f13c006cc30 2024-04-12T06:01:39.131 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T06:01:39.131 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T06:01:39.132 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T06:01:39.132 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T06:01:39.132 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T06:01:39.132 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-12T06:01:39.132 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T06:01:39.132 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-12T06:01:39.132 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T06:01:39.132 INFO:teuthology.orchestra.run.smithi032.stdout: "mds" 2024-04-12T06:01:39.132 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T06:01:39.132 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T06:01:39.132 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T06:01:39.132 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T06:01:39.132 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T06:01:39.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.139+0000 7f13d989f700 1 -- 172.21.15.32:0/4238427382 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f13c006cc30 msgr2=0x7f13c006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:39.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.139+0000 7f13d989f700 1 --2- 172.21.15.32:0/4238427382 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f13c006cc30 0x7f13c006f0e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f13d410ec20 tx=0x7f13bc007d40 comp rx=0 tx=0).stop 2024-04-12T06:01:39.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.139+0000 7f13d989f700 1 -- 172.21.15.32:0/4238427382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13d4101d70 msgr2=0x7f13d4113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:39.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.139+0000 7f13d989f700 1 --2- 172.21.15.32:0/4238427382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13d4101d70 0x7f13d4113ae0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f13c4009360 tx=0x7f13c4009e30 comp rx=0 tx=0).stop 2024-04-12T06:01:39.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.139+0000 7f13d989f700 1 -- 172.21.15.32:0/4238427382 shutdown_connections 2024-04-12T06:01:39.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.139+0000 7f13d989f700 1 --2- 172.21.15.32:0/4238427382 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f13c006cc30 0x7f13c006f0e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:39.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.139+0000 7f13d989f700 1 --2- 172.21.15.32:0/4238427382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13d4101d70 0x7f13d4113ae0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:39.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.139+0000 7f13d989f700 1 --2- 172.21.15.32:0/4238427382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13d4102710 0x7f13d4114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:39.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.139+0000 7f13d989f700 1 -- 172.21.15.32:0/4238427382 >> 172.21.15.32:0/4238427382 conn(0x7f13d4072bc0 msgr2=0x7f13d40659e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:01:39.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.139+0000 7f13d989f700 1 -- 172.21.15.32:0/4238427382 shutdown_connections 2024-04-12T06:01:39.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.139+0000 7f13d989f700 1 -- 172.21.15.32:0/4238427382 wait complete. 2024-04-12T06:01:39.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.242+0000 7f64d190e700 1 -- 172.21.15.32:0/694323751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64cc102f70 msgr2=0x7f64cc103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:39.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.242+0000 7f64d190e700 1 --2- 172.21.15.32:0/694323751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64cc102f70 0x7f64cc103340 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f64c0004940 tx=0x7f64c0009e30 comp rx=0 tx=0).stop 2024-04-12T06:01:39.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.242+0000 7f64d190e700 1 -- 172.21.15.32:0/694323751 shutdown_connections 2024-04-12T06:01:39.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.242+0000 7f64d190e700 1 --2- 172.21.15.32:0/694323751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64cc0fb3b0 0x7f64cc0fb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:39.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.242+0000 7f64d190e700 1 --2- 172.21.15.32:0/694323751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64cc102f70 0x7f64cc103340 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:39.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.242+0000 7f64d190e700 1 -- 172.21.15.32:0/694323751 >> 172.21.15.32:0/694323751 conn(0x7f64cc072310 msgr2=0x7f64cc072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:01:39.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.242+0000 7f64d190e700 1 -- 172.21.15.32:0/694323751 shutdown_connections 2024-04-12T06:01:39.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.242+0000 7f64d190e700 1 -- 172.21.15.32:0/694323751 wait complete. 2024-04-12T06:01:39.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.242+0000 7f64d190e700 1 Processor -- start 2024-04-12T06:01:39.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.243+0000 7f64d190e700 1 -- start start 2024-04-12T06:01:39.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.243+0000 7f64d190e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64cc0fb3b0 0x7f64cc1137c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:39.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.243+0000 7f64d190e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64cc102f70 0x7f64cc113d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:39.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.243+0000 7f64d190e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc114350 con 0x7f64cc102f70 2024-04-12T06:01:39.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.243+0000 7f64d190e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc10d990 con 0x7f64cc0fb3b0 2024-04-12T06:01:39.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.243+0000 7f64ca7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64cc102f70 0x7f64cc113d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:39.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.243+0000 7f64ca7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64cc102f70 0x7f64cc113d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60570/0 (socket says 172.21.15.32:60570) 2024-04-12T06:01:39.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.243+0000 7f64ca7fc700 1 -- 172.21.15.32:0/2251394144 learned_addr learned my addr 172.21.15.32:0/2251394144 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:01:39.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.243+0000 7f64caffd700 1 --2- 172.21.15.32:0/2251394144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64cc0fb3b0 0x7f64cc1137c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:39.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.243+0000 7f64ca7fc700 1 -- 172.21.15.32:0/2251394144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64cc0fb3b0 msgr2=0x7f64cc1137c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:39.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.243+0000 7f64ca7fc700 1 --2- 172.21.15.32:0/2251394144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64cc0fb3b0 0x7f64cc1137c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:39.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.243+0000 7f64ca7fc700 1 -- 172.21.15.32:0/2251394144 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c0005040 con 0x7f64cc102f70 2024-04-12T06:01:39.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.243+0000 7f64ca7fc700 1 --2- 172.21.15.32:0/2251394144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64cc102f70 0x7f64cc113d00 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f64bc002a50 tx=0x7f64bc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:01:39.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.244+0000 7f64b37fe700 1 -- 172.21.15.32:0/2251394144 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64bc007cc0 con 0x7f64cc102f70 2024-04-12T06:01:39.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.244+0000 7f64d190e700 1 -- 172.21.15.32:0/2251394144 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64cc10dc70 con 0x7f64cc102f70 2024-04-12T06:01:39.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.244+0000 7f64b37fe700 1 -- 172.21.15.32:0/2251394144 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f64bc00f040 con 0x7f64cc102f70 2024-04-12T06:01:39.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.244+0000 7f64b37fe700 1 -- 172.21.15.32:0/2251394144 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64bc00d9c0 con 0x7f64cc102f70 2024-04-12T06:01:39.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.244+0000 7f64d190e700 1 -- 172.21.15.32:0/2251394144 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64cc10e230 con 0x7f64cc102f70 2024-04-12T06:01:39.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.245+0000 7f64d190e700 1 -- 172.21.15.32:0/2251394144 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64cc0fe6d0 con 0x7f64cc102f70 2024-04-12T06:01:39.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.245+0000 7f64b37fe700 1 -- 172.21.15.32:0/2251394144 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f64bc00db20 con 0x7f64cc102f70 2024-04-12T06:01:39.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.246+0000 7f64b37fe700 1 --2- 172.21.15.32:0/2251394144 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f64b406cc20 0x7f64b406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:01:39.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.246+0000 7f64b37fe700 1 -- 172.21.15.32:0/2251394144 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f64bc08bdd0 con 0x7f64cc102f70 2024-04-12T06:01:39.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.246+0000 7f64caffd700 1 --2- 172.21.15.32:0/2251394144 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f64b406cc20 0x7f64b406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:01:39.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.246+0000 7f64caffd700 1 --2- 172.21.15.32:0/2251394144 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f64b406cc20 0x7f64b406f0d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f64c0004560 tx=0x7f64c0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:01:39.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.250+0000 7f64b37fe700 1 -- 172.21.15.32:0/2251394144 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f64bc0597c0 con 0x7f64cc102f70 2024-04-12T06:01:39.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:39 smithi067 ceph-mon[112979]: from='client.35188 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:01:39.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:39 smithi067 ceph-mon[112979]: pgmap v634: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:39.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:39 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3598535602' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:01:39.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:39 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/2100458325' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T06:01:39.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.537+0000 7f64d190e700 1 -- 172.21.15.32:0/2251394144 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f64cc105390 con 0x7f64cc102f70 2024-04-12T06:01:39.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.537+0000 7f64b37fe700 1 -- 172.21.15.32:0/2251394144 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f64bc059350 con 0x7f64cc102f70 2024-04-12T06:01:39.532 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T06:01:39.532 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T06:01:39.532 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-12T06:01:39.532 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T06:01:39.532 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T06:01:39.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.539+0000 7f64d190e700 1 -- 172.21.15.32:0/2251394144 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f64b406cc20 msgr2=0x7f64b406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:39.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.539+0000 7f64d190e700 1 --2- 172.21.15.32:0/2251394144 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f64b406cc20 0x7f64b406f0d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f64c0004560 tx=0x7f64c0009e30 comp rx=0 tx=0).stop 2024-04-12T06:01:39.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.539+0000 7f64d190e700 1 -- 172.21.15.32:0/2251394144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64cc102f70 msgr2=0x7f64cc113d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:01:39.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.539+0000 7f64d190e700 1 --2- 172.21.15.32:0/2251394144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64cc102f70 0x7f64cc113d00 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f64bc002a50 tx=0x7f64bc0046b0 comp rx=0 tx=0).stop 2024-04-12T06:01:39.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.539+0000 7f64d190e700 1 -- 172.21.15.32:0/2251394144 shutdown_connections 2024-04-12T06:01:39.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.539+0000 7f64d190e700 1 --2- 172.21.15.32:0/2251394144 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f64b406cc20 0x7f64b406f0d0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:39.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.539+0000 7f64d190e700 1 --2- 172.21.15.32:0/2251394144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64cc0fb3b0 0x7f64cc1137c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:39.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.539+0000 7f64d190e700 1 --2- 172.21.15.32:0/2251394144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64cc102f70 0x7f64cc113d00 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:01:39.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.539+0000 7f64d190e700 1 -- 172.21.15.32:0/2251394144 >> 172.21.15.32:0/2251394144 conn(0x7f64cc072310 msgr2=0x7f64cc0f9600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:01:39.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.539+0000 7f64d190e700 1 -- 172.21.15.32:0/2251394144 shutdown_connections 2024-04-12T06:01:39.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:01:39.539+0000 7f64d190e700 1 -- 172.21.15.32:0/2251394144 wait complete. 2024-04-12T06:01:39.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:39 smithi032 ceph-mon[146817]: from='client.35188 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:01:39.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:39 smithi032 ceph-mon[146817]: pgmap v634: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:39.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:39 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3598535602' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:01:39.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:39 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/2100458325' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T06:01:40.515 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:40 smithi067 ceph-mon[112979]: from='client.35200 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:01:40.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:40 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/2251394144' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T06:01:40.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:40 smithi032 ceph-mon[146817]: from='client.35200 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:01:40.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:40 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/2251394144' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T06:01:41.515 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:41 smithi067 ceph-mon[112979]: pgmap v635: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:41.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:41 smithi032 ceph-mon[146817]: pgmap v635: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:43.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:43 smithi067 ceph-mon[112979]: pgmap v636: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:43.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:43 smithi032 ceph-mon[146817]: pgmap v636: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:45.515 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:45 smithi067 ceph-mon[112979]: pgmap v637: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:45.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:45 smithi032 ceph-mon[146817]: pgmap v637: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:47.516 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:47 smithi067 ceph-mon[112979]: pgmap v638: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:47.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:47 smithi032 ceph-mon[146817]: pgmap v638: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:48.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:48 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:01:48.765 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:48 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:01:49.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:49 smithi032 ceph-mon[146817]: pgmap v639: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:49.765 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:49 smithi067 ceph-mon[112979]: pgmap v639: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:51.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:51 smithi032 ceph-mon[146817]: pgmap v640: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:51.765 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:51 smithi067 ceph-mon[112979]: pgmap v640: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:53.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:53 smithi032 ceph-mon[146817]: pgmap v641: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:53.765 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:53 smithi067 ceph-mon[112979]: pgmap v641: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:55.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:55 smithi032 ceph-mon[146817]: pgmap v642: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:55.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:55 smithi067 ceph-mon[112979]: pgmap v642: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:57.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:57 smithi032 ceph-mon[146817]: pgmap v643: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:57.765 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:57 smithi067 ceph-mon[112979]: pgmap v643: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:59.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:01:59 smithi032 ceph-mon[146817]: pgmap v644: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:01:59.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:01:59 smithi067 ceph-mon[112979]: pgmap v644: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:02:01.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:01 smithi032 ceph-mon[146817]: pgmap v645: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:02:01.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:01 smithi067 ceph-mon[112979]: pgmap v645: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:02:03.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:03 smithi032 ceph-mon[146817]: pgmap v646: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:02:03.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:03 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:02:03.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:03 smithi067 ceph-mon[112979]: pgmap v646: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:02:03.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:03 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:02:05.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:05 smithi032 ceph-mon[146817]: pgmap v647: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:02:05.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:05 smithi067 ceph-mon[112979]: pgmap v647: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:02:07.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:07 smithi032 ceph-mon[146817]: pgmap v648: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:02:07.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:07 smithi067 ceph-mon[112979]: pgmap v648: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:02:09.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:09 smithi032 ceph-mon[146817]: pgmap v649: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:02:09.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.657+0000 7fd231d74700 1 -- 172.21.15.32:0/861133936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd22c1037b0 msgr2=0x7fd22c0fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:09.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.657+0000 7fd231d74700 1 --2- 172.21.15.32:0/861133936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd22c1037b0 0x7fd22c0fce20 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fd220004940 tx=0x7fd220009e30 comp rx=0 tx=0).stop 2024-04-12T06:02:09.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.657+0000 7fd231d74700 1 -- 172.21.15.32:0/861133936 shutdown_connections 2024-04-12T06:02:09.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.657+0000 7fd231d74700 1 --2- 172.21.15.32:0/861133936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd22c1037b0 0x7fd22c0fce20 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:09.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.657+0000 7fd231d74700 1 --2- 172.21.15.32:0/861133936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd22c102e10 0x7fd22c1031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:09.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.657+0000 7fd231d74700 1 -- 172.21.15.32:0/861133936 >> 172.21.15.32:0/861133936 conn(0x7fd22c0f8980 msgr2=0x7fd22c0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:02:09.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.657+0000 7fd231d74700 1 -- 172.21.15.32:0/861133936 shutdown_connections 2024-04-12T06:02:09.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.657+0000 7fd231d74700 1 -- 172.21.15.32:0/861133936 wait complete. 2024-04-12T06:02:09.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.657+0000 7fd231d74700 1 Processor -- start 2024-04-12T06:02:09.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.658+0000 7fd231d74700 1 -- start start 2024-04-12T06:02:09.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.658+0000 7fd231d74700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd22c102e10 0x7fd22c113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:09.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.658+0000 7fd231d74700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd22c1037b0 0x7fd22c114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:09.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.658+0000 7fd231d74700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c114680 con 0x7fd22c1037b0 2024-04-12T06:02:09.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.658+0000 7fd231d74700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c10da70 con 0x7fd22c102e10 2024-04-12T06:02:09.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.658+0000 7fd22affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd22c1037b0 0x7fd22c114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:09.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.658+0000 7fd22affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd22c1037b0 0x7fd22c114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54832/0 (socket says 172.21.15.32:54832) 2024-04-12T06:02:09.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.658+0000 7fd22affd700 1 -- 172.21.15.32:0/1783006917 learned_addr learned my addr 172.21.15.32:0/1783006917 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:02:09.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.658+0000 7fd22b7fe700 1 --2- 172.21.15.32:0/1783006917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd22c102e10 0x7fd22c113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:09.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.658+0000 7fd22affd700 1 -- 172.21.15.32:0/1783006917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd22c102e10 msgr2=0x7fd22c113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:09.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.658+0000 7fd22affd700 1 --2- 172.21.15.32:0/1783006917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd22c102e10 0x7fd22c113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:09.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.658+0000 7fd22affd700 1 -- 172.21.15.32:0/1783006917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd220005040 con 0x7fd22c1037b0 2024-04-12T06:02:09.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.659+0000 7fd22affd700 1 --2- 172.21.15.32:0/1783006917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd22c1037b0 0x7fd22c114030 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fd2200093b0 tx=0x7fd220009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:02:09.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.659+0000 7fd213fff700 1 -- 172.21.15.32:0/1783006917 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd22000e040 con 0x7fd22c1037b0 2024-04-12T06:02:09.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.659+0000 7fd213fff700 1 -- 172.21.15.32:0/1783006917 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd220013420 con 0x7fd22c1037b0 2024-04-12T06:02:09.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.659+0000 7fd231d74700 1 -- 172.21.15.32:0/1783006917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd22c10dcf0 con 0x7fd22c1037b0 2024-04-12T06:02:09.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.659+0000 7fd213fff700 1 -- 172.21.15.32:0/1783006917 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd22001c630 con 0x7fd22c1037b0 2024-04-12T06:02:09.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.659+0000 7fd231d74700 1 -- 172.21.15.32:0/1783006917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd22c10e2b0 con 0x7fd22c1037b0 2024-04-12T06:02:09.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.660+0000 7fd231d74700 1 -- 172.21.15.32:0/1783006917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd22c0fe560 con 0x7fd22c1037b0 2024-04-12T06:02:09.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.660+0000 7fd213fff700 1 -- 172.21.15.32:0/1783006917 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd2200139e0 con 0x7fd22c1037b0 2024-04-12T06:02:09.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.661+0000 7fd213fff700 1 --2- 172.21.15.32:0/1783006917 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd21406cc20 0x7fd21406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:09.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.661+0000 7fd213fff700 1 -- 172.21.15.32:0/1783006917 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fd22008d0e0 con 0x7fd22c1037b0 2024-04-12T06:02:09.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.661+0000 7fd22b7fe700 1 --2- 172.21.15.32:0/1783006917 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd21406cc20 0x7fd21406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:09.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.661+0000 7fd22b7fe700 1 --2- 172.21.15.32:0/1783006917 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd21406cc20 0x7fd21406f0d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fd21c004f40 tx=0x7fd21c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:02:09.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.666+0000 7fd213fff700 1 -- 172.21.15.32:0/1783006917 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd22005aca0 con 0x7fd22c1037b0 2024-04-12T06:02:09.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:09 smithi067 ceph-mon[112979]: pgmap v649: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:02:09.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.887+0000 7fd231d74700 1 -- 172.21.15.32:0/1783006917 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd22c109db0 con 0x7fd21406cc20 2024-04-12T06:02:09.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.888+0000 7fd213fff700 1 -- 172.21.15.32:0/1783006917 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd22c109db0 con 0x7fd21406cc20 2024-04-12T06:02:09.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.890+0000 7fd231d74700 1 -- 172.21.15.32:0/1783006917 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd21406cc20 msgr2=0x7fd21406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:09.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.890+0000 7fd231d74700 1 --2- 172.21.15.32:0/1783006917 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd21406cc20 0x7fd21406f0d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fd21c004f40 tx=0x7fd21c00b040 comp rx=0 tx=0).stop 2024-04-12T06:02:09.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.890+0000 7fd231d74700 1 -- 172.21.15.32:0/1783006917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd22c1037b0 msgr2=0x7fd22c114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:09.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.890+0000 7fd231d74700 1 --2- 172.21.15.32:0/1783006917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd22c1037b0 0x7fd22c114030 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fd2200093b0 tx=0x7fd220009e30 comp rx=0 tx=0).stop 2024-04-12T06:02:09.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.890+0000 7fd231d74700 1 -- 172.21.15.32:0/1783006917 shutdown_connections 2024-04-12T06:02:09.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.890+0000 7fd231d74700 1 --2- 172.21.15.32:0/1783006917 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd21406cc20 0x7fd21406f0d0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:09.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.890+0000 7fd231d74700 1 --2- 172.21.15.32:0/1783006917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd22c102e10 0x7fd22c113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:09.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.890+0000 7fd231d74700 1 --2- 172.21.15.32:0/1783006917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd22c1037b0 0x7fd22c114030 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:09.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.890+0000 7fd231d74700 1 -- 172.21.15.32:0/1783006917 >> 172.21.15.32:0/1783006917 conn(0x7fd22c0f8980 msgr2=0x7fd22c0f9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:02:09.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.890+0000 7fd231d74700 1 -- 172.21.15.32:0/1783006917 shutdown_connections 2024-04-12T06:02:09.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.890+0000 7fd231d74700 1 -- 172.21.15.32:0/1783006917 wait complete. 2024-04-12T06:02:09.896 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T06:02:09.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.995+0000 7f9a5f2c9700 1 -- 172.21.15.32:0/3322947579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a58102e10 msgr2=0x7f9a581031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:09.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.995+0000 7f9a5f2c9700 1 --2- 172.21.15.32:0/3322947579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a58102e10 0x7f9a581031e0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f9a54004940 tx=0x7f9a54009e30 comp rx=0 tx=0).stop 2024-04-12T06:02:09.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.995+0000 7f9a5f2c9700 1 -- 172.21.15.32:0/3322947579 shutdown_connections 2024-04-12T06:02:09.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.995+0000 7f9a5f2c9700 1 --2- 172.21.15.32:0/3322947579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a581037b0 0x7f9a580fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:09.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.995+0000 7f9a5f2c9700 1 --2- 172.21.15.32:0/3322947579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a58102e10 0x7f9a581031e0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:09.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.995+0000 7f9a5f2c9700 1 -- 172.21.15.32:0/3322947579 >> 172.21.15.32:0/3322947579 conn(0x7f9a580f8980 msgr2=0x7f9a580fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:02:09.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.995+0000 7f9a5f2c9700 1 -- 172.21.15.32:0/3322947579 shutdown_connections 2024-04-12T06:02:09.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.995+0000 7f9a5f2c9700 1 -- 172.21.15.32:0/3322947579 wait complete. 2024-04-12T06:02:09.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.995+0000 7f9a5f2c9700 1 Processor -- start 2024-04-12T06:02:09.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.995+0000 7f9a5f2c9700 1 -- start start 2024-04-12T06:02:09.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.995+0000 7f9a5f2c9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a58102e10 0x7f9a58113b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:09.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.996+0000 7f9a5f2c9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a581037b0 0x7f9a58114050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:09.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.996+0000 7f9a5f2c9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a581146a0 con 0x7f9a581037b0 2024-04-12T06:02:09.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.996+0000 7f9a5f2c9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5810da90 con 0x7f9a58102e10 2024-04-12T06:02:09.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.996+0000 7f9a5c864700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a581037b0 0x7f9a58114050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:09.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.996+0000 7f9a5c864700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a581037b0 0x7f9a58114050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54846/0 (socket says 172.21.15.32:54846) 2024-04-12T06:02:09.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.996+0000 7f9a5c864700 1 -- 172.21.15.32:0/3770642165 learned_addr learned my addr 172.21.15.32:0/3770642165 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:02:09.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.996+0000 7f9a5d065700 1 --2- 172.21.15.32:0/3770642165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a58102e10 0x7f9a58113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:09.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.996+0000 7f9a5c864700 1 -- 172.21.15.32:0/3770642165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a58102e10 msgr2=0x7f9a58113b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:09.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.996+0000 7f9a5c864700 1 --2- 172.21.15.32:0/3770642165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a58102e10 0x7f9a58113b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:09.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.996+0000 7f9a5c864700 1 -- 172.21.15.32:0/3770642165 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a54005040 con 0x7f9a581037b0 2024-04-12T06:02:09.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.996+0000 7f9a5c864700 1 --2- 172.21.15.32:0/3770642165 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a581037b0 0x7f9a58114050 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9a4800c890 tx=0x7f9a480044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:02:09.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.996+0000 7f9a4dffb700 1 -- 172.21.15.32:0/3770642165 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a4800a700 con 0x7f9a581037b0 2024-04-12T06:02:09.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.996+0000 7f9a4dffb700 1 -- 172.21.15.32:0/3770642165 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a4800ad10 con 0x7f9a581037b0 2024-04-12T06:02:09.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.996+0000 7f9a5f2c9700 1 -- 172.21.15.32:0/3770642165 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a5810dd70 con 0x7f9a581037b0 2024-04-12T06:02:09.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.996+0000 7f9a4dffb700 1 -- 172.21.15.32:0/3770642165 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a48012e70 con 0x7f9a581037b0 2024-04-12T06:02:09.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.997+0000 7f9a5f2c9700 1 -- 172.21.15.32:0/3770642165 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a5810e330 con 0x7f9a581037b0 2024-04-12T06:02:09.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.997+0000 7f9a5f2c9700 1 -- 172.21.15.32:0/3770642165 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a580fe560 con 0x7f9a581037b0 2024-04-12T06:02:09.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.998+0000 7f9a4dffb700 1 -- 172.21.15.32:0/3770642165 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9a4800a860 con 0x7f9a581037b0 2024-04-12T06:02:09.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.998+0000 7f9a4dffb700 1 --2- 172.21.15.32:0/3770642165 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9a4406cc30 0x7f9a4406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:09.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.998+0000 7f9a4dffb700 1 -- 172.21.15.32:0/3770642165 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f9a4808c020 con 0x7f9a581037b0 2024-04-12T06:02:09.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.998+0000 7f9a5d065700 1 --2- 172.21.15.32:0/3770642165 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9a4406cc30 0x7f9a4406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:09.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:09.999+0000 7f9a5d065700 1 --2- 172.21.15.32:0/3770642165 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9a4406cc30 0x7f9a4406f0e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f9a54009fd0 tx=0x7f9a54009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:02:09.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.003+0000 7f9a4dffb700 1 -- 172.21.15.32:0/3770642165 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9a48059b30 con 0x7f9a581037b0 2024-04-12T06:02:10.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.225+0000 7f9a5f2c9700 1 -- 172.21.15.32:0/3770642165 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a58109dd0 con 0x7f9a4406cc30 2024-04-12T06:02:10.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.226+0000 7f9a4dffb700 1 -- 172.21.15.32:0/3770642165 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9a58109dd0 con 0x7f9a4406cc30 2024-04-12T06:02:10.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.228+0000 7f9a5f2c9700 1 -- 172.21.15.32:0/3770642165 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9a4406cc30 msgr2=0x7f9a4406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:10.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.228+0000 7f9a5f2c9700 1 --2- 172.21.15.32:0/3770642165 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9a4406cc30 0x7f9a4406f0e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f9a54009fd0 tx=0x7f9a54009e30 comp rx=0 tx=0).stop 2024-04-12T06:02:10.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.228+0000 7f9a5f2c9700 1 -- 172.21.15.32:0/3770642165 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a581037b0 msgr2=0x7f9a58114050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:10.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.228+0000 7f9a5f2c9700 1 --2- 172.21.15.32:0/3770642165 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a581037b0 0x7f9a58114050 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9a4800c890 tx=0x7f9a480044c0 comp rx=0 tx=0).stop 2024-04-12T06:02:10.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.228+0000 7f9a5f2c9700 1 -- 172.21.15.32:0/3770642165 shutdown_connections 2024-04-12T06:02:10.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.228+0000 7f9a5f2c9700 1 --2- 172.21.15.32:0/3770642165 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f9a4406cc30 0x7f9a4406f0e0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:10.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.228+0000 7f9a5f2c9700 1 --2- 172.21.15.32:0/3770642165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a58102e10 0x7f9a58113b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:10.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.228+0000 7f9a5f2c9700 1 --2- 172.21.15.32:0/3770642165 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a581037b0 0x7f9a58114050 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:10.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.228+0000 7f9a5f2c9700 1 -- 172.21.15.32:0/3770642165 >> 172.21.15.32:0/3770642165 conn(0x7f9a580f8980 msgr2=0x7f9a580f9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:02:10.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.228+0000 7f9a5f2c9700 1 -- 172.21.15.32:0/3770642165 shutdown_connections 2024-04-12T06:02:10.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.228+0000 7f9a5f2c9700 1 -- 172.21.15.32:0/3770642165 wait complete. 2024-04-12T06:02:10.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.332+0000 7f91a281a700 1 -- 172.21.15.32:0/3530447917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f919c101d70 msgr2=0x7f919c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:10.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.332+0000 7f91a281a700 1 --2- 172.21.15.32:0/3530447917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f919c101d70 0x7f919c102140 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f9184004940 tx=0x7f9184009e30 comp rx=0 tx=0).stop 2024-04-12T06:02:10.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.332+0000 7f91a281a700 1 -- 172.21.15.32:0/3530447917 shutdown_connections 2024-04-12T06:02:10.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.332+0000 7f91a281a700 1 --2- 172.21.15.32:0/3530447917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f919c102710 0x7f919c0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:10.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.332+0000 7f91a281a700 1 --2- 172.21.15.32:0/3530447917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f919c101d70 0x7f919c102140 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:10.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.332+0000 7f91a281a700 1 -- 172.21.15.32:0/3530447917 >> 172.21.15.32:0/3530447917 conn(0x7f919c072bc0 msgr2=0x7f919c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:02:10.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.332+0000 7f91a281a700 1 -- 172.21.15.32:0/3530447917 shutdown_connections 2024-04-12T06:02:10.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.332+0000 7f91a281a700 1 -- 172.21.15.32:0/3530447917 wait complete. 2024-04-12T06:02:10.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.333+0000 7f91a281a700 1 Processor -- start 2024-04-12T06:02:10.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.333+0000 7f91a281a700 1 -- start start 2024-04-12T06:02:10.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.333+0000 7f91a281a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f919c101d70 0x7f919c10e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:10.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.333+0000 7f91a281a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f919c102710 0x7f919c10ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:10.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.333+0000 7f91a281a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919c10f550 con 0x7f919c101d70 2024-04-12T06:02:10.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.333+0000 7f91a281a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919c10f690 con 0x7f919c102710 2024-04-12T06:02:10.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.333+0000 7f919bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f919c101d70 0x7f919c10e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:10.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.333+0000 7f919bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f919c101d70 0x7f919c10e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54868/0 (socket says 172.21.15.32:54868) 2024-04-12T06:02:10.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.333+0000 7f919bfff700 1 -- 172.21.15.32:0/3330216630 learned_addr learned my addr 172.21.15.32:0/3330216630 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:02:10.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.333+0000 7f919b7fe700 1 --2- 172.21.15.32:0/3330216630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f919c102710 0x7f919c10ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:10.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.333+0000 7f919bfff700 1 -- 172.21.15.32:0/3330216630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f919c102710 msgr2=0x7f919c10ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:10.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.333+0000 7f919bfff700 1 --2- 172.21.15.32:0/3330216630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f919c102710 0x7f919c10ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:10.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.333+0000 7f919bfff700 1 -- 172.21.15.32:0/3330216630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9184005040 con 0x7f919c101d70 2024-04-12T06:02:10.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.334+0000 7f919bfff700 1 --2- 172.21.15.32:0/3330216630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f919c101d70 0x7f919c10e9c0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f9184000c00 tx=0x7f9184009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:02:10.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.334+0000 7f9198ff9700 1 -- 172.21.15.32:0/3330216630 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9184021070 con 0x7f919c101d70 2024-04-12T06:02:10.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.334+0000 7f9198ff9700 1 -- 172.21.15.32:0/3330216630 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f91840093b0 con 0x7f919c101d70 2024-04-12T06:02:10.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.334+0000 7f91a281a700 1 -- 172.21.15.32:0/3330216630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f919c117a30 con 0x7f919c101d70 2024-04-12T06:02:10.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.334+0000 7f9198ff9700 1 -- 172.21.15.32:0/3330216630 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f918400e070 con 0x7f919c101d70 2024-04-12T06:02:10.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.334+0000 7f91a281a700 1 -- 172.21.15.32:0/3330216630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f919c117f40 con 0x7f919c101d70 2024-04-12T06:02:10.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.335+0000 7f91a281a700 1 -- 172.21.15.32:0/3330216630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f919c0ff080 con 0x7f919c101d70 2024-04-12T06:02:10.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.335+0000 7f9198ff9700 1 -- 172.21.15.32:0/3330216630 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f918401c490 con 0x7f919c101d70 2024-04-12T06:02:10.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.336+0000 7f9198ff9700 1 --2- 172.21.15.32:0/3330216630 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f918806cc20 0x7f918806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:10.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.336+0000 7f9198ff9700 1 -- 172.21.15.32:0/3330216630 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f918408d160 con 0x7f919c101d70 2024-04-12T06:02:10.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.336+0000 7f919b7fe700 1 --2- 172.21.15.32:0/3330216630 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f918806cc20 0x7f918806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:10.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.336+0000 7f919b7fe700 1 --2- 172.21.15.32:0/3330216630 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f918806cc20 0x7f918806f0d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f919c066000 tx=0x7f918c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:02:10.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.341+0000 7f9198ff9700 1 -- 172.21.15.32:0/3330216630 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9184057140 con 0x7f919c101d70 2024-04-12T06:02:10.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.551+0000 7f91a281a700 1 -- 172.21.15.32:0/3330216630 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f919c04bed0 con 0x7f918806cc20 2024-04-12T06:02:10.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.561+0000 7f9198ff9700 1 -- 172.21.15.32:0/3330216630 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f919c04bed0 con 0x7f918806cc20 2024-04-12T06:02:10.556 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T06:02:10.556 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (21m) 15m ago 33m 28.2M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T06:02:10.557 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (19m) 15m ago 33m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T06:02:10.557 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (19m) 14m ago 32m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T06:02:10.557 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (21m) 15m ago 33m 87.2M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T06:02:10.557 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (15m) 15m ago 27m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T06:02:10.557 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (15m) 15m ago 27m 13.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T06:02:10.557 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (14m) 14m ago 27m 11.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 6d4aca6f74a0 2024-04-12T06:02:10.557 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (15m) 14m ago 27m 14.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T06:02:10.557 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (23m) 15m ago 34m 507M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T06:02:10.557 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (22m) 14m ago 32m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T06:02:10.557 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (20m) 15m ago 35m 53.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T06:02:10.557 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (20m) 14m ago 32m 43.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T06:02:10.557 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (22m) 15m ago 33m 16.9M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T06:02:10.557 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (22m) 14m ago 32m 18.4M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T06:02:10.557 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (19m) 15m ago 31m 91.1M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T06:02:10.557 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (18m) 15m ago 31m 97.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T06:02:10.558 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (17m) 15m ago 30m 81.8M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T06:02:10.558 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (17m) 14m ago 30m 113M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T06:02:10.558 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (16m) 14m ago 29m 90.3M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T06:02:10.558 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (16m) 14m ago 29m 84.8M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T06:02:10.558 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (22m) 15m ago 32m 64.2M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T06:02:10.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.563+0000 7f91a281a700 1 -- 172.21.15.32:0/3330216630 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f918806cc20 msgr2=0x7f918806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:10.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.563+0000 7f91a281a700 1 --2- 172.21.15.32:0/3330216630 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f918806cc20 0x7f918806f0d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f919c066000 tx=0x7f918c00b040 comp rx=0 tx=0).stop 2024-04-12T06:02:10.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.563+0000 7f91a281a700 1 -- 172.21.15.32:0/3330216630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f919c101d70 msgr2=0x7f919c10e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:10.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.563+0000 7f91a281a700 1 --2- 172.21.15.32:0/3330216630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f919c101d70 0x7f919c10e9c0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f9184000c00 tx=0x7f9184009e30 comp rx=0 tx=0).stop 2024-04-12T06:02:10.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.564+0000 7f91a281a700 1 -- 172.21.15.32:0/3330216630 shutdown_connections 2024-04-12T06:02:10.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.564+0000 7f91a281a700 1 --2- 172.21.15.32:0/3330216630 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f918806cc20 0x7f918806f0d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:10.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.564+0000 7f91a281a700 1 --2- 172.21.15.32:0/3330216630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f919c101d70 0x7f919c10e9c0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:10.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.564+0000 7f91a281a700 1 --2- 172.21.15.32:0/3330216630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f919c102710 0x7f919c10ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:10.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.564+0000 7f91a281a700 1 -- 172.21.15.32:0/3330216630 >> 172.21.15.32:0/3330216630 conn(0x7f919c072bc0 msgr2=0x7f919c0faf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:02:10.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.564+0000 7f91a281a700 1 -- 172.21.15.32:0/3330216630 shutdown_connections 2024-04-12T06:02:10.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.564+0000 7f91a281a700 1 -- 172.21.15.32:0/3330216630 wait complete. 2024-04-12T06:02:10.586 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:10 smithi032 ceph-mon[146817]: from='client.35208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:02:10.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.665+0000 7f51535a6700 1 -- 172.21.15.32:0/1457374640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f514c102e10 msgr2=0x7f514c1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:10.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.665+0000 7f51535a6700 1 --2- 172.21.15.32:0/1457374640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f514c102e10 0x7f514c1031e0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f5148004940 tx=0x7f5148009e30 comp rx=0 tx=0).stop 2024-04-12T06:02:10.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.665+0000 7f51535a6700 1 -- 172.21.15.32:0/1457374640 shutdown_connections 2024-04-12T06:02:10.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.665+0000 7f51535a6700 1 --2- 172.21.15.32:0/1457374640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f514c1037b0 0x7f514c0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:10.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.665+0000 7f51535a6700 1 --2- 172.21.15.32:0/1457374640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f514c102e10 0x7f514c1031e0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:10.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.665+0000 7f51535a6700 1 -- 172.21.15.32:0/1457374640 >> 172.21.15.32:0/1457374640 conn(0x7f514c0f8980 msgr2=0x7f514c0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:02:10.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.666+0000 7f51535a6700 1 -- 172.21.15.32:0/1457374640 shutdown_connections 2024-04-12T06:02:10.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.666+0000 7f51535a6700 1 -- 172.21.15.32:0/1457374640 wait complete. 2024-04-12T06:02:10.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.666+0000 7f51535a6700 1 Processor -- start 2024-04-12T06:02:10.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.666+0000 7f51535a6700 1 -- start start 2024-04-12T06:02:10.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.666+0000 7f51535a6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f514c102e10 0x7f514c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:10.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.666+0000 7f51535a6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f514c1037b0 0x7f514c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:10.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.666+0000 7f51535a6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c070d90 con 0x7f514c102e10 2024-04-12T06:02:10.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.666+0000 7f51535a6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c070ed0 con 0x7f514c1037b0 2024-04-12T06:02:10.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.666+0000 7f5151342700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f514c102e10 0x7f514c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:10.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.666+0000 7f5151342700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f514c102e10 0x7f514c0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54878/0 (socket says 172.21.15.32:54878) 2024-04-12T06:02:10.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.666+0000 7f5151342700 1 -- 172.21.15.32:0/1895022521 learned_addr learned my addr 172.21.15.32:0/1895022521 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:02:10.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.667+0000 7f5150b41700 1 --2- 172.21.15.32:0/1895022521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f514c1037b0 0x7f514c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:10.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.667+0000 7f5151342700 1 -- 172.21.15.32:0/1895022521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f514c1037b0 msgr2=0x7f514c070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:10.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.667+0000 7f5151342700 1 --2- 172.21.15.32:0/1895022521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f514c1037b0 0x7f514c070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:10.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.667+0000 7f5151342700 1 -- 172.21.15.32:0/1895022521 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5148005040 con 0x7f514c102e10 2024-04-12T06:02:10.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.667+0000 7f5151342700 1 --2- 172.21.15.32:0/1895022521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f514c102e10 0x7f514c0702c0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f5148009360 tx=0x7f5148009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:02:10.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.667+0000 7f5141ffb700 1 -- 172.21.15.32:0/1895022521 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f514800e040 con 0x7f514c102e10 2024-04-12T06:02:10.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.667+0000 7f5141ffb700 1 -- 172.21.15.32:0/1895022521 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5148013420 con 0x7f514c102e10 2024-04-12T06:02:10.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.667+0000 7f51535a6700 1 -- 172.21.15.32:0/1895022521 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f514c071120 con 0x7f514c102e10 2024-04-12T06:02:10.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.667+0000 7f5141ffb700 1 -- 172.21.15.32:0/1895022521 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f514801c5f0 con 0x7f514c102e10 2024-04-12T06:02:10.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.667+0000 7f51535a6700 1 -- 172.21.15.32:0/1895022521 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f514c11a240 con 0x7f514c102e10 2024-04-12T06:02:10.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.668+0000 7f51535a6700 1 -- 172.21.15.32:0/1895022521 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51300052b0 con 0x7f514c102e10 2024-04-12T06:02:10.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.669+0000 7f5141ffb700 1 -- 172.21.15.32:0/1895022521 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f514801c8b0 con 0x7f514c102e10 2024-04-12T06:02:10.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.669+0000 7f5141ffb700 1 --2- 172.21.15.32:0/1895022521 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f513806cc30 0x7f513806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:10.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.669+0000 7f5141ffb700 1 -- 172.21.15.32:0/1895022521 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f514808d2d0 con 0x7f514c102e10 2024-04-12T06:02:10.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.669+0000 7f5150b41700 1 --2- 172.21.15.32:0/1895022521 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f513806cc30 0x7f513806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:10.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.670+0000 7f5150b41700 1 --2- 172.21.15.32:0/1895022521 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f513806cc30 0x7f513806f0e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f513c007fd0 tx=0x7f513c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:02:10.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.674+0000 7f5141ffb700 1 -- 172.21.15.32:0/1895022521 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f51480572b0 con 0x7f514c102e10 2024-04-12T06:02:10.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:10 smithi067 ceph-mon[112979]: from='client.35208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:02:10.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.963+0000 7f51535a6700 1 -- 172.21.15.32:0/1895022521 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5130005d70 con 0x7f514c102e10 2024-04-12T06:02:10.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.964+0000 7f5141ffb700 1 -- 172.21.15.32:0/1895022521 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f514805a8d0 con 0x7f514c102e10 2024-04-12T06:02:10.959 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T06:02:10.959 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T06:02:10.959 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T06:02:10.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T06:02:10.960 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T06:02:10.960 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T06:02:10.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T06:02:10.960 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T06:02:10.960 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T06:02:10.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T06:02:10.960 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T06:02:10.960 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T06:02:10.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T06:02:10.960 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T06:02:10.960 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T06:02:10.960 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T06:02:10.960 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T06:02:10.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.966+0000 7f51535a6700 1 -- 172.21.15.32:0/1895022521 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f513806cc30 msgr2=0x7f513806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:10.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.966+0000 7f51535a6700 1 --2- 172.21.15.32:0/1895022521 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f513806cc30 0x7f513806f0e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f513c007fd0 tx=0x7f513c009040 comp rx=0 tx=0).stop 2024-04-12T06:02:10.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.966+0000 7f51535a6700 1 -- 172.21.15.32:0/1895022521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f514c102e10 msgr2=0x7f514c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:10.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.966+0000 7f51535a6700 1 --2- 172.21.15.32:0/1895022521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f514c102e10 0x7f514c0702c0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f5148009360 tx=0x7f5148009e30 comp rx=0 tx=0).stop 2024-04-12T06:02:10.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.966+0000 7f51535a6700 1 -- 172.21.15.32:0/1895022521 shutdown_connections 2024-04-12T06:02:10.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.966+0000 7f51535a6700 1 --2- 172.21.15.32:0/1895022521 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f513806cc30 0x7f513806f0e0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:10.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.966+0000 7f51535a6700 1 --2- 172.21.15.32:0/1895022521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f514c102e10 0x7f514c0702c0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:10.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.966+0000 7f51535a6700 1 --2- 172.21.15.32:0/1895022521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f514c1037b0 0x7f514c070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:10.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.966+0000 7f51535a6700 1 -- 172.21.15.32:0/1895022521 >> 172.21.15.32:0/1895022521 conn(0x7f514c0f8980 msgr2=0x7f514c0f9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:02:10.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.966+0000 7f51535a6700 1 -- 172.21.15.32:0/1895022521 shutdown_connections 2024-04-12T06:02:10.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:10.966+0000 7f51535a6700 1 -- 172.21.15.32:0/1895022521 wait complete. 2024-04-12T06:02:11.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.069+0000 7f16769fa700 1 -- 172.21.15.32:0/3357196339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16700fe830 msgr2=0x7f1670106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:11.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.069+0000 7f16769fa700 1 --2- 172.21.15.32:0/3357196339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16700fe830 0x7f1670106f00 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f1664004940 tx=0x7f1664009e30 comp rx=0 tx=0).stop 2024-04-12T06:02:11.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.069+0000 7f16769fa700 1 -- 172.21.15.32:0/3357196339 shutdown_connections 2024-04-12T06:02:11.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.069+0000 7f16769fa700 1 --2- 172.21.15.32:0/3357196339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16700fe830 0x7f1670106f00 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:11.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.069+0000 7f16769fa700 1 --2- 172.21.15.32:0/3357196339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16700fde90 0x7f16700fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:11.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.069+0000 7f16769fa700 1 -- 172.21.15.32:0/3357196339 >> 172.21.15.32:0/3357196339 conn(0x7f1670072bc0 msgr2=0x7f1670072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:02:11.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.069+0000 7f16769fa700 1 -- 172.21.15.32:0/3357196339 shutdown_connections 2024-04-12T06:02:11.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.069+0000 7f16769fa700 1 -- 172.21.15.32:0/3357196339 wait complete. 2024-04-12T06:02:11.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.070+0000 7f16769fa700 1 Processor -- start 2024-04-12T06:02:11.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.070+0000 7f16769fa700 1 -- start start 2024-04-12T06:02:11.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.070+0000 7f16769fa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16700fde90 0x7f1670113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:11.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.070+0000 7f16769fa700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16700fe830 0x7f1670114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:11.066 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.070+0000 7f16769fa700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1670114670 con 0x7f16700fde90 2024-04-12T06:02:11.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.070+0000 7f16769fa700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167010da60 con 0x7f16700fe830 2024-04-12T06:02:11.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.071+0000 7f166f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16700fe830 0x7f1670114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:11.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.071+0000 7f166ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16700fde90 0x7f1670113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:11.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.071+0000 7f166f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16700fe830 0x7f1670114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.32:54820/0 (socket says 172.21.15.32:54820) 2024-04-12T06:02:11.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.071+0000 7f166f7fe700 1 -- 172.21.15.32:0/290479444 learned_addr learned my addr 172.21.15.32:0/290479444 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:02:11.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.071+0000 7f166ffff700 1 -- 172.21.15.32:0/290479444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16700fe830 msgr2=0x7f1670114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:11.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.071+0000 7f166ffff700 1 --2- 172.21.15.32:0/290479444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16700fe830 0x7f1670114020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:11.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.071+0000 7f166ffff700 1 -- 172.21.15.32:0/290479444 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1664005040 con 0x7f16700fde90 2024-04-12T06:02:11.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.071+0000 7f166ffff700 1 --2- 172.21.15.32:0/290479444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16700fde90 0x7f1670113ae0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f1660002820 tx=0x7f1660004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:02:11.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.071+0000 7f166cff9700 1 -- 172.21.15.32:0/290479444 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1660007cc0 con 0x7f16700fde90 2024-04-12T06:02:11.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.071+0000 7f16769fa700 1 -- 172.21.15.32:0/290479444 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f167010dd40 con 0x7f16700fde90 2024-04-12T06:02:11.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.071+0000 7f166cff9700 1 -- 172.21.15.32:0/290479444 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f166000f040 con 0x7f16700fde90 2024-04-12T06:02:11.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.071+0000 7f16769fa700 1 -- 172.21.15.32:0/290479444 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f167010e280 con 0x7f16700fde90 2024-04-12T06:02:11.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.071+0000 7f166cff9700 1 -- 172.21.15.32:0/290479444 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16600137d0 con 0x7f16700fde90 2024-04-12T06:02:11.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.072+0000 7f16769fa700 1 -- 172.21.15.32:0/290479444 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1670065e30 con 0x7f16700fde90 2024-04-12T06:02:11.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.073+0000 7f166cff9700 1 -- 172.21.15.32:0/290479444 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1660013ad0 con 0x7f16700fde90 2024-04-12T06:02:11.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.073+0000 7f166cff9700 1 --2- 172.21.15.32:0/290479444 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f165806cc20 0x7f165806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:11.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.073+0000 7f166cff9700 1 -- 172.21.15.32:0/290479444 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f166008bdd0 con 0x7f16700fde90 2024-04-12T06:02:11.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.073+0000 7f166f7fe700 1 --2- 172.21.15.32:0/290479444 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f165806cc20 0x7f165806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:11.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.074+0000 7f166f7fe700 1 --2- 172.21.15.32:0/290479444 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f165806cc20 0x7f165806f0d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f167010ec20 tx=0x7f1664009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:02:11.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.078+0000 7f166cff9700 1 -- 172.21.15.32:0/290479444 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f16600597c0 con 0x7f16700fde90 2024-04-12T06:02:11.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.324+0000 7f16769fa700 1 -- 172.21.15.32:0/290479444 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f167010f160 con 0x7f16700fde90 2024-04-12T06:02:11.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.324+0000 7f166cff9700 1 -- 172.21.15.32:0/290479444 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1835 (secure 0 0 0) 0x7f1660059350 con 0x7f16700fde90 2024-04-12T06:02:11.320 INFO:teuthology.orchestra.run.smithi032.stdout:e33 2024-04-12T06:02:11.320 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T06:02:11.320 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T06:02:11.320 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T06:02:11.320 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:02:11.320 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T06:02:11.320 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T06:02:11.320 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 24 2024-04-12T06:02:11.320 INFO:teuthology.orchestra.run.smithi032.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T06:02:11.320 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T06:02:11.320 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T05:46:10.668569+0000 2024-04-12T06:02:11.320 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T06:02:11.320 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T06:02:11.320 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T06:02:11.321 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T06:02:11.321 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T06:02:11.321 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T06:02:11.321 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T06:02:11.321 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T06:02:11.321 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T06:02:11.321 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T06:02:11.321 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T06:02:11.321 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T06:02:11.321 INFO:teuthology.orchestra.run.smithi032.stdout:up {} 2024-04-12T06:02:11.321 INFO:teuthology.orchestra.run.smithi032.stdout:failed 0,1 2024-04-12T06:02:11.321 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T06:02:11.321 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T06:02:11.321 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T06:02:11.321 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T06:02:11.321 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T06:02:11.322 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T06:02:11.322 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T06:02:11.322 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T06:02:11.322 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:02:11.322 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:02:11.322 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-12T06:02:11.322 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:02:11.322 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T06:02:11.322 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{-1:34416} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T06:02:11.322 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{-1:44317} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T06:02:11.322 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T06:02:11.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.326+0000 7f16769fa700 1 -- 172.21.15.32:0/290479444 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f165806cc20 msgr2=0x7f165806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:11.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.326+0000 7f16769fa700 1 --2- 172.21.15.32:0/290479444 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f165806cc20 0x7f165806f0d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f167010ec20 tx=0x7f1664009e30 comp rx=0 tx=0).stop 2024-04-12T06:02:11.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.326+0000 7f16769fa700 1 -- 172.21.15.32:0/290479444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16700fde90 msgr2=0x7f1670113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:11.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.326+0000 7f16769fa700 1 --2- 172.21.15.32:0/290479444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16700fde90 0x7f1670113ae0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f1660002820 tx=0x7f1660004570 comp rx=0 tx=0).stop 2024-04-12T06:02:11.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.326+0000 7f16769fa700 1 -- 172.21.15.32:0/290479444 shutdown_connections 2024-04-12T06:02:11.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.326+0000 7f16769fa700 1 --2- 172.21.15.32:0/290479444 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f165806cc20 0x7f165806f0d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:11.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.326+0000 7f16769fa700 1 --2- 172.21.15.32:0/290479444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16700fde90 0x7f1670113ae0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:11.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.326+0000 7f16769fa700 1 --2- 172.21.15.32:0/290479444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16700fe830 0x7f1670114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:11.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.326+0000 7f16769fa700 1 -- 172.21.15.32:0/290479444 >> 172.21.15.32:0/290479444 conn(0x7f1670072bc0 msgr2=0x7f1670103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:02:11.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.327+0000 7f16769fa700 1 -- 172.21.15.32:0/290479444 shutdown_connections 2024-04-12T06:02:11.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.327+0000 7f16769fa700 1 -- 172.21.15.32:0/290479444 wait complete. 2024-04-12T06:02:11.324 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-12T06:02:11.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.430+0000 7f89b63af700 1 -- 172.21.15.32:0/4198429301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89b0102710 msgr2=0x7f89b00fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:11.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.430+0000 7f89b63af700 1 --2- 172.21.15.32:0/4198429301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89b0102710 0x7f89b00fbd80 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f89a4004940 tx=0x7f89a4009e30 comp rx=0 tx=0).stop 2024-04-12T06:02:11.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.431+0000 7f89b63af700 1 -- 172.21.15.32:0/4198429301 shutdown_connections 2024-04-12T06:02:11.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.431+0000 7f89b63af700 1 --2- 172.21.15.32:0/4198429301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89b0102710 0x7f89b00fbd80 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:11.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.431+0000 7f89b63af700 1 --2- 172.21.15.32:0/4198429301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b0101d70 0x7f89b0102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:11.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.431+0000 7f89b63af700 1 -- 172.21.15.32:0/4198429301 >> 172.21.15.32:0/4198429301 conn(0x7f89b0072bc0 msgr2=0x7f89b0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:02:11.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.431+0000 7f89b63af700 1 -- 172.21.15.32:0/4198429301 shutdown_connections 2024-04-12T06:02:11.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.431+0000 7f89b63af700 1 -- 172.21.15.32:0/4198429301 wait complete. 2024-04-12T06:02:11.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.431+0000 7f89b63af700 1 Processor -- start 2024-04-12T06:02:11.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.432+0000 7f89b63af700 1 -- start start 2024-04-12T06:02:11.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.432+0000 7f89b63af700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b0101d70 0x7f89b0112900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:11.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.432+0000 7f89b63af700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89b0102710 0x7f89b0112e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:11.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.432+0000 7f89b63af700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b0113490 con 0x7f89b0102710 2024-04-12T06:02:11.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.432+0000 7f89b63af700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b010c880 con 0x7f89b0101d70 2024-04-12T06:02:11.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.432+0000 7f89af7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89b0102710 0x7f89b0112e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:11.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.432+0000 7f89af7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89b0102710 0x7f89b0112e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51962/0 (socket says 172.21.15.32:51962) 2024-04-12T06:02:11.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.432+0000 7f89af7fe700 1 -- 172.21.15.32:0/3495908542 learned_addr learned my addr 172.21.15.32:0/3495908542 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:02:11.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.432+0000 7f89affff700 1 --2- 172.21.15.32:0/3495908542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b0101d70 0x7f89b0112900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:11.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.432+0000 7f89af7fe700 1 -- 172.21.15.32:0/3495908542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b0101d70 msgr2=0x7f89b0112900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:11.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.432+0000 7f89af7fe700 1 --2- 172.21.15.32:0/3495908542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b0101d70 0x7f89b0112900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:11.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.432+0000 7f89af7fe700 1 -- 172.21.15.32:0/3495908542 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a4005040 con 0x7f89b0102710 2024-04-12T06:02:11.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.433+0000 7f89af7fe700 1 --2- 172.21.15.32:0/3495908542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89b0102710 0x7f89b0112e40 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f89a4000c00 tx=0x7f89a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:02:11.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.433+0000 7f89acff9700 1 -- 172.21.15.32:0/3495908542 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f89a400e070 con 0x7f89b0102710 2024-04-12T06:02:11.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.433+0000 7f89acff9700 1 -- 172.21.15.32:0/3495908542 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f89a40093b0 con 0x7f89b0102710 2024-04-12T06:02:11.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.433+0000 7f89acff9700 1 -- 172.21.15.32:0/3495908542 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f89a401ccb0 con 0x7f89b0102710 2024-04-12T06:02:11.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.433+0000 7f89b63af700 1 -- 172.21.15.32:0/3495908542 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89b010cb00 con 0x7f89b0102710 2024-04-12T06:02:11.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.433+0000 7f89b63af700 1 -- 172.21.15.32:0/3495908542 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89b010d060 con 0x7f89b0102710 2024-04-12T06:02:11.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.434+0000 7f89b63af700 1 -- 172.21.15.32:0/3495908542 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89b00ff080 con 0x7f89b0102710 2024-04-12T06:02:11.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.434+0000 7f89acff9700 1 -- 172.21.15.32:0/3495908542 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f89a401c490 con 0x7f89b0102710 2024-04-12T06:02:11.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.435+0000 7f89acff9700 1 --2- 172.21.15.32:0/3495908542 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f899806cc20 0x7f899806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:11.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.435+0000 7f89acff9700 1 -- 172.21.15.32:0/3495908542 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f89a408d3f0 con 0x7f89b0102710 2024-04-12T06:02:11.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.435+0000 7f89affff700 1 --2- 172.21.15.32:0/3495908542 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f899806cc20 0x7f899806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:11.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.435+0000 7f89affff700 1 --2- 172.21.15.32:0/3495908542 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f899806cc20 0x7f899806f0d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f89a0009b80 tx=0x7f89a000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:02:11.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.440+0000 7f89acff9700 1 -- 172.21.15.32:0/3495908542 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f89a405af00 con 0x7f89b0102710 2024-04-12T06:02:11.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:11 smithi032 ceph-mon[146817]: pgmap v650: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:02:11.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:11 smithi032 ceph-mon[146817]: from='client.35212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:02:11.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:11 smithi032 ceph-mon[146817]: from='client.35216 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:02:11.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:11 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1895022521' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:11.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.662+0000 7f89b63af700 1 -- 172.21.15.32:0/3495908542 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f89b010df50 con 0x7f899806cc20 2024-04-12T06:02:11.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.663+0000 7f89acff9700 1 -- 172.21.15.32:0/3495908542 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f89b010df50 con 0x7f899806cc20 2024-04-12T06:02:11.658 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T06:02:11.658 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T06:02:11.658 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true, 2024-04-12T06:02:11.658 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T06:02:11.658 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [ 2024-04-12T06:02:11.658 INFO:teuthology.orchestra.run.smithi032.stdout: "osd", 2024-04-12T06:02:11.658 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr", 2024-04-12T06:02:11.659 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-12T06:02:11.659 INFO:teuthology.orchestra.run.smithi032.stdout: "mon", 2024-04-12T06:02:11.659 INFO:teuthology.orchestra.run.smithi032.stdout: "mds" 2024-04-12T06:02:11.659 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-12T06:02:11.659 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T06:02:11.659 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T06:02:11.659 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T06:02:11.659 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T06:02:11.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.665+0000 7f89b63af700 1 -- 172.21.15.32:0/3495908542 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f899806cc20 msgr2=0x7f899806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:11.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.665+0000 7f89b63af700 1 --2- 172.21.15.32:0/3495908542 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f899806cc20 0x7f899806f0d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f89a0009b80 tx=0x7f89a000b040 comp rx=0 tx=0).stop 2024-04-12T06:02:11.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.665+0000 7f89b63af700 1 -- 172.21.15.32:0/3495908542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89b0102710 msgr2=0x7f89b0112e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:11.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.665+0000 7f89b63af700 1 --2- 172.21.15.32:0/3495908542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89b0102710 0x7f89b0112e40 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f89a4000c00 tx=0x7f89a4009e30 comp rx=0 tx=0).stop 2024-04-12T06:02:11.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.665+0000 7f89b63af700 1 -- 172.21.15.32:0/3495908542 shutdown_connections 2024-04-12T06:02:11.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.665+0000 7f89b63af700 1 --2- 172.21.15.32:0/3495908542 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f899806cc20 0x7f899806f0d0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:11.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.665+0000 7f89b63af700 1 --2- 172.21.15.32:0/3495908542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b0101d70 0x7f89b0112900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:11.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.665+0000 7f89b63af700 1 --2- 172.21.15.32:0/3495908542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89b0102710 0x7f89b0112e40 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:11.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.665+0000 7f89b63af700 1 -- 172.21.15.32:0/3495908542 >> 172.21.15.32:0/3495908542 conn(0x7f89b0072bc0 msgr2=0x7f89b00faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:02:11.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.665+0000 7f89b63af700 1 -- 172.21.15.32:0/3495908542 shutdown_connections 2024-04-12T06:02:11.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.665+0000 7f89b63af700 1 -- 172.21.15.32:0/3495908542 wait complete. 2024-04-12T06:02:11.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.768+0000 7f5121cae700 1 -- 172.21.15.32:0/5408038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f511c102710 msgr2=0x7f511c0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:11.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.768+0000 7f5121cae700 1 --2- 172.21.15.32:0/5408038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f511c102710 0x7f511c0fbd80 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f5104004940 tx=0x7f5104009e30 comp rx=0 tx=0).stop 2024-04-12T06:02:11.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.768+0000 7f5121cae700 1 -- 172.21.15.32:0/5408038 shutdown_connections 2024-04-12T06:02:11.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.768+0000 7f5121cae700 1 --2- 172.21.15.32:0/5408038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f511c102710 0x7f511c0fbd80 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:11.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.768+0000 7f5121cae700 1 --2- 172.21.15.32:0/5408038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f511c101d70 0x7f511c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:11.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.768+0000 7f5121cae700 1 -- 172.21.15.32:0/5408038 >> 172.21.15.32:0/5408038 conn(0x7f511c072bc0 msgr2=0x7f511c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:02:11.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.768+0000 7f5121cae700 1 -- 172.21.15.32:0/5408038 shutdown_connections 2024-04-12T06:02:11.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.768+0000 7f5121cae700 1 -- 172.21.15.32:0/5408038 wait complete. 2024-04-12T06:02:11.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.769+0000 7f5121cae700 1 Processor -- start 2024-04-12T06:02:11.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.769+0000 7f5121cae700 1 -- start start 2024-04-12T06:02:11.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.769+0000 7f5121cae700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f511c101d70 0x7f511c112900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:11.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.769+0000 7f5121cae700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f511c102710 0x7f511c112e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:11.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.769+0000 7f5121cae700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511c113490 con 0x7f511c102710 2024-04-12T06:02:11.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.769+0000 7f5121cae700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511c10c880 con 0x7f511c101d70 2024-04-12T06:02:11.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.769+0000 7f511affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f511c102710 0x7f511c112e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:11.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.769+0000 7f511affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f511c102710 0x7f511c112e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51988/0 (socket says 172.21.15.32:51988) 2024-04-12T06:02:11.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.769+0000 7f511affd700 1 -- 172.21.15.32:0/1539656605 learned_addr learned my addr 172.21.15.32:0/1539656605 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:02:11.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.770+0000 7f511b7fe700 1 --2- 172.21.15.32:0/1539656605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f511c101d70 0x7f511c112900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:11.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.770+0000 7f511affd700 1 -- 172.21.15.32:0/1539656605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f511c101d70 msgr2=0x7f511c112900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:11.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.770+0000 7f511affd700 1 --2- 172.21.15.32:0/1539656605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f511c101d70 0x7f511c112900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:11.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.770+0000 7f511affd700 1 -- 172.21.15.32:0/1539656605 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5104005040 con 0x7f511c102710 2024-04-12T06:02:11.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.770+0000 7f511affd700 1 --2- 172.21.15.32:0/1539656605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f511c102710 0x7f511c112e40 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f5104009380 tx=0x7f5104009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:02:11.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.770+0000 7f5113fff700 1 -- 172.21.15.32:0/1539656605 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f510400e040 con 0x7f511c102710 2024-04-12T06:02:11.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.770+0000 7f5113fff700 1 -- 172.21.15.32:0/1539656605 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5104013420 con 0x7f511c102710 2024-04-12T06:02:11.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.770+0000 7f5121cae700 1 -- 172.21.15.32:0/1539656605 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f511c10cb00 con 0x7f511c102710 2024-04-12T06:02:11.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.770+0000 7f5113fff700 1 -- 172.21.15.32:0/1539656605 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f510401c630 con 0x7f511c102710 2024-04-12T06:02:11.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.770+0000 7f5121cae700 1 -- 172.21.15.32:0/1539656605 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f511c10d0c0 con 0x7f511c102710 2024-04-12T06:02:11.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:11 smithi067 ceph-mon[112979]: pgmap v650: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:02:11.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:11 smithi067 ceph-mon[112979]: from='client.35212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:02:11.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:11 smithi067 ceph-mon[112979]: from='client.35216 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:02:11.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:11 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1895022521' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:11.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.771+0000 7f5121cae700 1 -- 172.21.15.32:0/1539656605 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f511c0ff080 con 0x7f511c102710 2024-04-12T06:02:11.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.772+0000 7f5113fff700 1 -- 172.21.15.32:0/1539656605 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f51040139e0 con 0x7f511c102710 2024-04-12T06:02:11.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.772+0000 7f5113fff700 1 --2- 172.21.15.32:0/1539656605 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f510806cc20 0x7f510806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:11.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.772+0000 7f5113fff700 1 -- 172.21.15.32:0/1539656605 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f510408d2f0 con 0x7f511c102710 2024-04-12T06:02:11.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.772+0000 7f511b7fe700 1 --2- 172.21.15.32:0/1539656605 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f510806cc20 0x7f510806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:11.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.772+0000 7f511b7fe700 1 --2- 172.21.15.32:0/1539656605 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f510806cc20 0x7f510806f0d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f510c009b80 tx=0x7f510c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:02:11.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:11.777+0000 7f5113fff700 1 -- 172.21.15.32:0/1539656605 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f510405ad00 con 0x7f511c102710 2024-04-12T06:02:12.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:12.073+0000 7f5121cae700 1 -- 172.21.15.32:0/1539656605 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f511c10df50 con 0x7f511c102710 2024-04-12T06:02:12.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:12.074+0000 7f5113fff700 1 -- 172.21.15.32:0/1539656605 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f510405a890 con 0x7f511c102710 2024-04-12T06:02:12.069 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T06:02:12.069 INFO:teuthology.orchestra.run.smithi032.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T06:02:12.069 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is degraded 2024-04-12T06:02:12.069 INFO:teuthology.orchestra.run.smithi032.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T06:02:12.069 INFO:teuthology.orchestra.run.smithi032.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T06:02:12.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:12.076+0000 7f5121cae700 1 -- 172.21.15.32:0/1539656605 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f510806cc20 msgr2=0x7f510806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:12.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:12.076+0000 7f5121cae700 1 --2- 172.21.15.32:0/1539656605 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f510806cc20 0x7f510806f0d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f510c009b80 tx=0x7f510c00b040 comp rx=0 tx=0).stop 2024-04-12T06:02:12.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:12.076+0000 7f5121cae700 1 -- 172.21.15.32:0/1539656605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f511c102710 msgr2=0x7f511c112e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:12.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:12.076+0000 7f5121cae700 1 --2- 172.21.15.32:0/1539656605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f511c102710 0x7f511c112e40 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f5104009380 tx=0x7f5104009e30 comp rx=0 tx=0).stop 2024-04-12T06:02:12.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:12.076+0000 7f5121cae700 1 -- 172.21.15.32:0/1539656605 shutdown_connections 2024-04-12T06:02:12.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:12.076+0000 7f5121cae700 1 --2- 172.21.15.32:0/1539656605 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f510806cc20 0x7f510806f0d0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:12.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:12.076+0000 7f5121cae700 1 --2- 172.21.15.32:0/1539656605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f511c101d70 0x7f511c112900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:12.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:12.076+0000 7f5121cae700 1 --2- 172.21.15.32:0/1539656605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f511c102710 0x7f511c112e40 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:12.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:12.076+0000 7f5121cae700 1 -- 172.21.15.32:0/1539656605 >> 172.21.15.32:0/1539656605 conn(0x7f511c072bc0 msgr2=0x7f511c0faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:02:12.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:12.076+0000 7f5121cae700 1 -- 172.21.15.32:0/1539656605 shutdown_connections 2024-04-12T06:02:12.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:12.076+0000 7f5121cae700 1 -- 172.21.15.32:0/1539656605 wait complete. 2024-04-12T06:02:12.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:12 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/290479444' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T06:02:12.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:12 smithi032 ceph-mon[146817]: from='client.35228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:02:12.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:12 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T06:02:12.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:12 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T06:02:12.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:12 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:12.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:12 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1539656605' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T06:02:12.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:12 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T06:02:12.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:12 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:12.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:12 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:12.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:12 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:12.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:12 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:12.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:12 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:12.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:12 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:12.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:12 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi032.jehaaj"}]: dispatch 2024-04-12T06:02:12.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:12 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi032.jehaaj"}]': finished 2024-04-12T06:02:12.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:12 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi032.pndtry"}]: dispatch 2024-04-12T06:02:12.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:12 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi032.pndtry"}]': finished 2024-04-12T06:02:12.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:12 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi067.ymvqih"}]: dispatch 2024-04-12T06:02:12.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:12 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi067.ymvqih"}]': finished 2024-04-12T06:02:12.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:12 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi067.zesmgy"}]: dispatch 2024-04-12T06:02:12.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:12 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi067.zesmgy"}]': finished 2024-04-12T06:02:12.589 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:12 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-12T06:02:12.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:12 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/290479444' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T06:02:12.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:12 smithi067 ceph-mon[112979]: from='client.35228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:02:12.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:12 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T06:02:12.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:12 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T06:02:12.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:12 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:12.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:12 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1539656605' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T06:02:12.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:12 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T06:02:12.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:12 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:12.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:12 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:12.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:12 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:12.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:12 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:12.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:12 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:12.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:12 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:12.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:12 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi032.jehaaj"}]: dispatch 2024-04-12T06:02:12.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:12 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi032.jehaaj"}]': finished 2024-04-12T06:02:12.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:12 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi032.pndtry"}]: dispatch 2024-04-12T06:02:12.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:12 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi032.pndtry"}]': finished 2024-04-12T06:02:12.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:12 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi067.ymvqih"}]: dispatch 2024-04-12T06:02:12.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:12 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi067.ymvqih"}]': finished 2024-04-12T06:02:12.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:12 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi067.zesmgy"}]: dispatch 2024-04-12T06:02:12.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:12 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi067.zesmgy"}]': finished 2024-04-12T06:02:12.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:12 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-12T06:02:13.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:13 smithi032 ceph-mon[146817]: pgmap v651: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:02:13.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:13 smithi032 ceph-mon[146817]: Upgrade: Setting container_image for all mds 2024-04-12T06:02:13.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:13 smithi032 ceph-mon[146817]: Upgrade: Setting filesystem cephfs Joinable 2024-04-12T06:02:13.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:13 smithi032 ceph-mon[146817]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-04-12T06:02:13.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:13 smithi032 ceph-mon[146817]: Standby daemon mds.cephfs.smithi032.jehaaj assigned to filesystem cephfs as rank 0 2024-04-12T06:02:13.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:13 smithi032 ceph-mon[146817]: Standby daemon mds.cephfs.smithi032.pndtry assigned to filesystem cephfs as rank 1 2024-04-12T06:02:13.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:13 smithi032 ceph-mon[146817]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T06:02:13.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:13 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-12T06:02:13.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:13 smithi032 ceph-mon[146817]: fsmap cephfs:2/2 {0=cephfs.smithi032.jehaaj=up:replay,1=cephfs.smithi032.pndtry=up:replay} 2 up:standby 2024-04-12T06:02:13.587 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:13 smithi032 ceph-mon[146817]: Upgrade: Enabling allow_standby_replay on filesystem cephfs 2024-04-12T06:02:13.588 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:13 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]: dispatch 2024-04-12T06:02:13.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:13 smithi067 ceph-mon[112979]: pgmap v651: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:02:13.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:13 smithi067 ceph-mon[112979]: Upgrade: Setting container_image for all mds 2024-04-12T06:02:13.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:13 smithi067 ceph-mon[112979]: Upgrade: Setting filesystem cephfs Joinable 2024-04-12T06:02:13.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:13 smithi067 ceph-mon[112979]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-04-12T06:02:13.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:13 smithi067 ceph-mon[112979]: Standby daemon mds.cephfs.smithi032.jehaaj assigned to filesystem cephfs as rank 0 2024-04-12T06:02:13.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:13 smithi067 ceph-mon[112979]: Standby daemon mds.cephfs.smithi032.pndtry assigned to filesystem cephfs as rank 1 2024-04-12T06:02:13.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:13 smithi067 ceph-mon[112979]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T06:02:13.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:13 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-12T06:02:13.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:13 smithi067 ceph-mon[112979]: fsmap cephfs:2/2 {0=cephfs.smithi032.jehaaj=up:replay,1=cephfs.smithi032.pndtry=up:replay} 2 up:standby 2024-04-12T06:02:13.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:13 smithi067 ceph-mon[112979]: Upgrade: Enabling allow_standby_replay on filesystem cephfs 2024-04-12T06:02:13.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:13 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]: dispatch 2024-04-12T06:02:15.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]': finished 2024-04-12T06:02:15.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: fsmap cephfs:2/2 {0=cephfs.smithi032.jehaaj=up:replay,1=cephfs.smithi032.pndtry=up:replay} 2 up:standby 2024-04-12T06:02:15.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:15.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:15.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: Upgrade: Setting container_image for all rgw 2024-04-12T06:02:15.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:15.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:15.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: Upgrade: Setting container_image for all rbd-mirror 2024-04-12T06:02:15.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:15.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:15.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:15.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: Upgrade: Setting container_image for all ceph-exporter 2024-04-12T06:02:15.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:15.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:15.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: Upgrade: Setting container_image for all iscsi 2024-04-12T06:02:15.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:15.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:15.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: Upgrade: Setting container_image for all nfs 2024-04-12T06:02:15.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:15.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:15.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: Upgrade: Setting container_image for all nvmeof 2024-04-12T06:02:15.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:15.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:15.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:15.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:15.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:15.089 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:15.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:15.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: Upgrade: Finalizing container_image settings 2024-04-12T06:02:15.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-12T06:02:15.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-12T06:02:15.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T06:02:15.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-12T06:02:15.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-12T06:02:15.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-12T06:02:15.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-12T06:02:15.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-12T06:02:15.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-12T06:02:15.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-12T06:02:15.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-12T06:02:15.090 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-12T06:02:15.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-12T06:02:15.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-12T06:02:15.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T06:02:15.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-12T06:02:15.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-12T06:02:15.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-12T06:02:15.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-12T06:02:15.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-12T06:02:15.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-12T06:02:15.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-12T06:02:15.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-12T06:02:15.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T06:02:15.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T06:02:15.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T06:02:15.091 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T06:02:15.092 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T06:02:15.092 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T06:02:15.092 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-04-12T06:02:15.092 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-04-12T06:02:15.092 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-12T06:02:15.092 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-12T06:02:15.092 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T06:02:15.092 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:15.092 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:15.092 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:15.092 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:14 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:15.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]': finished 2024-04-12T06:02:15.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: fsmap cephfs:2/2 {0=cephfs.smithi032.jehaaj=up:replay,1=cephfs.smithi032.pndtry=up:replay} 2 up:standby 2024-04-12T06:02:15.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:15.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:15.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: Upgrade: Setting container_image for all rgw 2024-04-12T06:02:15.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:15.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:15.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: Upgrade: Setting container_image for all rbd-mirror 2024-04-12T06:02:15.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:15.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:15.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:15.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: Upgrade: Setting container_image for all ceph-exporter 2024-04-12T06:02:15.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:15.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:15.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: Upgrade: Setting container_image for all iscsi 2024-04-12T06:02:15.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:15.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:15.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: Upgrade: Setting container_image for all nfs 2024-04-12T06:02:15.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:15.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:15.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: Upgrade: Setting container_image for all nvmeof 2024-04-12T06:02:15.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:15.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:15.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:15.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:15.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:15.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:15.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:15.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: Upgrade: Finalizing container_image settings 2024-04-12T06:02:15.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-12T06:02:15.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-12T06:02:15.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T06:02:15.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-12T06:02:15.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-12T06:02:15.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-12T06:02:15.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-12T06:02:15.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-12T06:02:15.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-12T06:02:15.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-12T06:02:15.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-12T06:02:15.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-12T06:02:15.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-12T06:02:15.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-12T06:02:15.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T06:02:15.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-12T06:02:15.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-12T06:02:15.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-12T06:02:15.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-12T06:02:15.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-12T06:02:15.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-12T06:02:15.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-12T06:02:15.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-12T06:02:15.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T06:02:15.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T06:02:15.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T06:02:15.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T06:02:15.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T06:02:15.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T06:02:15.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-04-12T06:02:15.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-04-12T06:02:15.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-12T06:02:15.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-12T06:02:15.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T06:02:15.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:15.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:15.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:15.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:14 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:16.086 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:15 smithi032 ceph-mon[146817]: Upgrade: Complete! 2024-04-12T06:02:16.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:15 smithi032 ceph-mon[146817]: pgmap v652: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:02:16.265 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:15 smithi067 ceph-mon[112979]: Upgrade: Complete! 2024-04-12T06:02:16.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:15 smithi067 ceph-mon[112979]: pgmap v652: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail 2024-04-12T06:02:18.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:17 smithi067 ceph-mon[112979]: pgmap v653: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 5.4 MiB/s rd, 2 op/s 2024-04-12T06:02:18.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:17 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:18.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:17 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:18.086 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:17 smithi032 ceph-mon[146817]: pgmap v653: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 5.4 MiB/s rd, 2 op/s 2024-04-12T06:02:18.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:17 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:18.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:17 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:19.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:19.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:02:19.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:18 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:19.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:19.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:02:19.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:18 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:20.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:19 smithi067 ceph-mon[112979]: pgmap v654: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 16 MiB/s rd, 6 op/s 2024-04-12T06:02:20.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:19 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:20.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:19 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:20.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:19 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T06:02:20.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:19 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T06:02:20.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:19 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:20.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:19 smithi032 ceph-mon[146817]: pgmap v654: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 16 MiB/s rd, 6 op/s 2024-04-12T06:02:20.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:19 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:20.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:19 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:20.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:19 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T06:02:20.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:19 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T06:02:20.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:19 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:21.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:20 smithi067 ceph-mon[112979]: pgmap v655: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 6 op/s 2024-04-12T06:02:21.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:20 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T06:02:21.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:20 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T06:02:21.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:20 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T06:02:21.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:20 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:21.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:20 smithi067 ceph-mon[112979]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-04-12T06:02:21.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:20 smithi032 ceph-mon[146817]: pgmap v655: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 6 op/s 2024-04-12T06:02:21.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:20 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T06:02:21.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:20 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T06:02:21.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:20 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T06:02:21.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:20 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:21.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:20 smithi032 ceph-mon[146817]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-04-12T06:02:22.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:21 smithi067 ceph-mon[112979]: mds.? [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] up:resolve 2024-04-12T06:02:22.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:21 smithi067 ceph-mon[112979]: mds.? [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] up:resolve 2024-04-12T06:02:22.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:21 smithi067 ceph-mon[112979]: fsmap cephfs:2/2 {0=cephfs.smithi032.jehaaj=up:resolve,1=cephfs.smithi032.pndtry=up:resolve} 2 up:standby 2024-04-12T06:02:22.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:21 smithi067 ceph-mon[112979]: pgmap v656: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 23 MiB/s rd, 7 op/s 2024-04-12T06:02:22.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:21 smithi032 ceph-mon[146817]: mds.? [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] up:resolve 2024-04-12T06:02:22.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:21 smithi032 ceph-mon[146817]: mds.? [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] up:resolve 2024-04-12T06:02:22.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:21 smithi032 ceph-mon[146817]: fsmap cephfs:2/2 {0=cephfs.smithi032.jehaaj=up:resolve,1=cephfs.smithi032.pndtry=up:resolve} 2 up:standby 2024-04-12T06:02:22.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:21 smithi032 ceph-mon[146817]: pgmap v656: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 23 MiB/s rd, 7 op/s 2024-04-12T06:02:23.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:22 smithi067 ceph-mon[112979]: reconnect by client.24339 192.168.0.1:0/304126001 after 0.00199996 2024-04-12T06:02:23.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:22 smithi067 ceph-mon[112979]: reconnect by client.14534 192.168.0.1:0/2053061720 after 0.00199996 2024-04-12T06:02:23.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:22 smithi067 ceph-mon[112979]: reconnect by client.24339 192.168.0.1:0/304126001 after 0.00199996 2024-04-12T06:02:23.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:22 smithi067 ceph-mon[112979]: reconnect by client.14534 192.168.0.1:0/2053061720 after 0.00199996 2024-04-12T06:02:23.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:22 smithi067 ceph-mon[112979]: mds.? [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] up:reconnect 2024-04-12T06:02:23.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:22 smithi067 ceph-mon[112979]: mds.? [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] up:reconnect 2024-04-12T06:02:23.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:22 smithi067 ceph-mon[112979]: fsmap cephfs:2/2 {0=cephfs.smithi032.jehaaj=up:reconnect,1=cephfs.smithi032.pndtry=up:reconnect} 2 up:standby 2024-04-12T06:02:23.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:22 smithi032 ceph-mon[146817]: reconnect by client.24339 192.168.0.1:0/304126001 after 0.00199996 2024-04-12T06:02:23.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:22 smithi032 ceph-mon[146817]: reconnect by client.14534 192.168.0.1:0/2053061720 after 0.00199996 2024-04-12T06:02:23.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:22 smithi032 ceph-mon[146817]: reconnect by client.24339 192.168.0.1:0/304126001 after 0.00199996 2024-04-12T06:02:23.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:22 smithi032 ceph-mon[146817]: reconnect by client.14534 192.168.0.1:0/2053061720 after 0.00199996 2024-04-12T06:02:23.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:22 smithi032 ceph-mon[146817]: mds.? [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] up:reconnect 2024-04-12T06:02:23.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:22 smithi032 ceph-mon[146817]: mds.? [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] up:reconnect 2024-04-12T06:02:23.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:22 smithi032 ceph-mon[146817]: fsmap cephfs:2/2 {0=cephfs.smithi032.jehaaj=up:reconnect,1=cephfs.smithi032.pndtry=up:reconnect} 2 up:standby 2024-04-12T06:02:24.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:23 smithi067 ceph-mon[112979]: mds.? [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] up:rejoin 2024-04-12T06:02:24.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:23 smithi067 ceph-mon[112979]: mds.? [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] up:rejoin 2024-04-12T06:02:24.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:23 smithi067 ceph-mon[112979]: fsmap cephfs:2/2 {0=cephfs.smithi032.jehaaj=up:rejoin,1=cephfs.smithi032.pndtry=up:rejoin} 2 up:standby 2024-04-12T06:02:24.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:23 smithi067 ceph-mon[112979]: daemon mds.cephfs.smithi032.jehaaj is now active in filesystem cephfs as rank 0 2024-04-12T06:02:24.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:23 smithi067 ceph-mon[112979]: daemon mds.cephfs.smithi032.pndtry is now active in filesystem cephfs as rank 1 2024-04-12T06:02:24.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:23 smithi067 ceph-mon[112979]: pgmap v657: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 23 MiB/s rd, 7 op/s 2024-04-12T06:02:24.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:23 smithi032 ceph-mon[146817]: mds.? [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] up:rejoin 2024-04-12T06:02:24.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:23 smithi032 ceph-mon[146817]: mds.? [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] up:rejoin 2024-04-12T06:02:24.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:23 smithi032 ceph-mon[146817]: fsmap cephfs:2/2 {0=cephfs.smithi032.jehaaj=up:rejoin,1=cephfs.smithi032.pndtry=up:rejoin} 2 up:standby 2024-04-12T06:02:24.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:23 smithi032 ceph-mon[146817]: daemon mds.cephfs.smithi032.jehaaj is now active in filesystem cephfs as rank 0 2024-04-12T06:02:24.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:23 smithi032 ceph-mon[146817]: daemon mds.cephfs.smithi032.pndtry is now active in filesystem cephfs as rank 1 2024-04-12T06:02:24.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:23 smithi032 ceph-mon[146817]: pgmap v657: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 23 MiB/s rd, 7 op/s 2024-04-12T06:02:25.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:24 smithi067 ceph-mon[112979]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T06:02:25.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:24 smithi067 ceph-mon[112979]: Cluster is now healthy 2024-04-12T06:02:25.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:24 smithi067 ceph-mon[112979]: mds.? [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] up:active 2024-04-12T06:02:25.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:24 smithi067 ceph-mon[112979]: mds.? [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] up:active 2024-04-12T06:02:25.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:24 smithi067 ceph-mon[112979]: fsmap cephfs:2 {0=cephfs.smithi032.jehaaj=up:active,1=cephfs.smithi032.pndtry=up:active} 2 up:standby 2024-04-12T06:02:25.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:24 smithi067 ceph-mon[112979]: fsmap cephfs:2 {0=cephfs.smithi032.jehaaj=up:active,1=cephfs.smithi032.pndtry=up:active} 2 up:standby-replay 2024-04-12T06:02:25.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:24 smithi032 ceph-mon[146817]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T06:02:25.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:24 smithi032 ceph-mon[146817]: Cluster is now healthy 2024-04-12T06:02:25.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:24 smithi032 ceph-mon[146817]: mds.? [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] up:active 2024-04-12T06:02:25.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:24 smithi032 ceph-mon[146817]: mds.? [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] up:active 2024-04-12T06:02:25.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:24 smithi032 ceph-mon[146817]: fsmap cephfs:2 {0=cephfs.smithi032.jehaaj=up:active,1=cephfs.smithi032.pndtry=up:active} 2 up:standby 2024-04-12T06:02:25.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:24 smithi032 ceph-mon[146817]: fsmap cephfs:2 {0=cephfs.smithi032.jehaaj=up:active,1=cephfs.smithi032.pndtry=up:active} 2 up:standby-replay 2024-04-12T06:02:26.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:25 smithi067 ceph-mon[112979]: pgmap v658: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 27 MiB/s rd, 521 B/s wr, 13 op/s 2024-04-12T06:02:26.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:25 smithi032 ceph-mon[146817]: pgmap v658: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 27 MiB/s rd, 521 B/s wr, 13 op/s 2024-04-12T06:02:29.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:28 smithi067 ceph-mon[112979]: pgmap v659: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 31 MiB/s rd, 4.7 KiB/s wr, 18 op/s 2024-04-12T06:02:29.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:28 smithi032 ceph-mon[146817]: pgmap v659: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 31 MiB/s rd, 4.7 KiB/s wr, 18 op/s 2024-04-12T06:02:31.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:30 smithi067 ceph-mon[112979]: pgmap v660: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 24 MiB/s rd, 9.3 KiB/s wr, 18 op/s 2024-04-12T06:02:31.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:30 smithi032 ceph-mon[146817]: pgmap v660: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 24 MiB/s rd, 9.3 KiB/s wr, 18 op/s 2024-04-12T06:02:33.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:32 smithi067 ceph-mon[112979]: pgmap v661: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 25 MiB/s rd, 9.2 KiB/s wr, 18 op/s 2024-04-12T06:02:33.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:32 smithi032 ceph-mon[146817]: pgmap v661: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 25 MiB/s rd, 9.2 KiB/s wr, 18 op/s 2024-04-12T06:02:34.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:33 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:34.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:33 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:02:34.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:33 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:02:34.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:33 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:02:35.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:34 smithi067 ceph-mon[112979]: pgmap v662: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 22 MiB/s rd, 9.2 KiB/s wr, 18 op/s 2024-04-12T06:02:35.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:34 smithi032 ceph-mon[146817]: pgmap v662: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 22 MiB/s rd, 9.2 KiB/s wr, 18 op/s 2024-04-12T06:02:37.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:36 smithi067 ceph-mon[112979]: pgmap v663: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 22 MiB/s rd, 9.2 KiB/s wr, 18 op/s 2024-04-12T06:02:37.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:36 smithi032 ceph-mon[146817]: pgmap v663: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 22 MiB/s rd, 9.2 KiB/s wr, 18 op/s 2024-04-12T06:02:39.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:38 smithi067 ceph-mon[112979]: pgmap v664: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 18 MiB/s rd, 8.7 KiB/s wr, 14 op/s 2024-04-12T06:02:39.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:38 smithi032 ceph-mon[146817]: pgmap v664: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 18 MiB/s rd, 8.7 KiB/s wr, 14 op/s 2024-04-12T06:02:41.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:40 smithi067 ceph-mon[112979]: pgmap v665: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 8.9 MiB/s rd, 4.6 KiB/s wr, 6 op/s 2024-04-12T06:02:41.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:40 smithi032 ceph-mon[146817]: pgmap v665: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 8.9 MiB/s rd, 4.6 KiB/s wr, 6 op/s 2024-04-12T06:02:42.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.198+0000 7f0b77a62700 1 -- 172.21.15.32:0/2156422591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b70102de0 msgr2=0x7f0b701031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:42.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.198+0000 7f0b77a62700 1 --2- 172.21.15.32:0/2156422591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b70102de0 0x7f0b701031b0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f0b60004940 tx=0x7f0b60009e30 comp rx=0 tx=0).stop 2024-04-12T06:02:42.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.198+0000 7f0b77a62700 1 -- 172.21.15.32:0/2156422591 shutdown_connections 2024-04-12T06:02:42.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.198+0000 7f0b77a62700 1 --2- 172.21.15.32:0/2156422591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b700656a0 0x7f0b70065af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:42.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.198+0000 7f0b77a62700 1 --2- 172.21.15.32:0/2156422591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b70102de0 0x7f0b701031b0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:42.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.198+0000 7f0b77a62700 1 -- 172.21.15.32:0/2156422591 >> 172.21.15.32:0/2156422591 conn(0x7f0b70072bc0 msgr2=0x7f0b70072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:02:42.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.198+0000 7f0b77a62700 1 -- 172.21.15.32:0/2156422591 shutdown_connections 2024-04-12T06:02:42.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.198+0000 7f0b77a62700 1 -- 172.21.15.32:0/2156422591 wait complete. 2024-04-12T06:02:42.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.199+0000 7f0b77a62700 1 Processor -- start 2024-04-12T06:02:42.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.199+0000 7f0b77a62700 1 -- start start 2024-04-12T06:02:42.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.199+0000 7f0b77a62700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b700656a0 0x7f0b70113a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:42.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.199+0000 7f0b77a62700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b70102de0 0x7f0b70113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:42.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.199+0000 7f0b77a62700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b701145a0 con 0x7f0b700656a0 2024-04-12T06:02:42.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.199+0000 7f0b77a62700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7010d990 con 0x7f0b70102de0 2024-04-12T06:02:42.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.199+0000 7f0b757fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b700656a0 0x7f0b70113a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:42.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.199+0000 7f0b757fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b700656a0 0x7f0b70113a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33384/0 (socket says 172.21.15.32:33384) 2024-04-12T06:02:42.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.199+0000 7f0b757fe700 1 -- 172.21.15.32:0/2946274779 learned_addr learned my addr 172.21.15.32:0/2946274779 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:02:42.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.199+0000 7f0b74ffd700 1 --2- 172.21.15.32:0/2946274779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b70102de0 0x7f0b70113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:42.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.199+0000 7f0b757fe700 1 -- 172.21.15.32:0/2946274779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b70102de0 msgr2=0x7f0b70113f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:42.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.199+0000 7f0b757fe700 1 --2- 172.21.15.32:0/2946274779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b70102de0 0x7f0b70113f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:42.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.200+0000 7f0b757fe700 1 -- 172.21.15.32:0/2946274779 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b60005040 con 0x7f0b700656a0 2024-04-12T06:02:42.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.200+0000 7f0b757fe700 1 --2- 172.21.15.32:0/2946274779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b700656a0 0x7f0b70113a10 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f0b6000b3b0 tx=0x7f0b60009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:02:42.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.200+0000 7f0b65ffb700 1 -- 172.21.15.32:0/2946274779 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b6000e040 con 0x7f0b700656a0 2024-04-12T06:02:42.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.200+0000 7f0b65ffb700 1 -- 172.21.15.32:0/2946274779 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b60013420 con 0x7f0b700656a0 2024-04-12T06:02:42.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.200+0000 7f0b77a62700 1 -- 172.21.15.32:0/2946274779 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b7010dc10 con 0x7f0b700656a0 2024-04-12T06:02:42.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.200+0000 7f0b65ffb700 1 -- 172.21.15.32:0/2946274779 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b6001c630 con 0x7f0b700656a0 2024-04-12T06:02:42.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.200+0000 7f0b77a62700 1 -- 172.21.15.32:0/2946274779 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b7010e1d0 con 0x7f0b700656a0 2024-04-12T06:02:42.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.201+0000 7f0b77a62700 1 -- 172.21.15.32:0/2946274779 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b700fe4c0 con 0x7f0b700656a0 2024-04-12T06:02:42.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.202+0000 7f0b65ffb700 1 -- 172.21.15.32:0/2946274779 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f0b60013970 con 0x7f0b700656a0 2024-04-12T06:02:42.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.202+0000 7f0b65ffb700 1 --2- 172.21.15.32:0/2946274779 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0b5c06cc20 0x7f0b5c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:42.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.202+0000 7f0b65ffb700 1 -- 172.21.15.32:0/2946274779 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f0b6008d3b0 con 0x7f0b700656a0 2024-04-12T06:02:42.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.202+0000 7f0b74ffd700 1 --2- 172.21.15.32:0/2946274779 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0b5c06cc20 0x7f0b5c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:42.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.203+0000 7f0b74ffd700 1 --2- 172.21.15.32:0/2946274779 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0b5c06cc20 0x7f0b5c06f0d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c009fd0 tx=0x7f0b6c009f20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:02:42.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.207+0000 7f0b65ffb700 1 -- 172.21.15.32:0/2946274779 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0b6005ad80 con 0x7f0b700656a0 2024-04-12T06:02:42.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.429+0000 7f0b77a62700 1 -- 172.21.15.32:0/2946274779 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b70109d00 con 0x7f0b5c06cc20 2024-04-12T06:02:42.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.430+0000 7f0b65ffb700 1 -- 172.21.15.32:0/2946274779 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f0b70109d00 con 0x7f0b5c06cc20 2024-04-12T06:02:42.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.432+0000 7f0b77a62700 1 -- 172.21.15.32:0/2946274779 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0b5c06cc20 msgr2=0x7f0b5c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:42.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.432+0000 7f0b77a62700 1 --2- 172.21.15.32:0/2946274779 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0b5c06cc20 0x7f0b5c06f0d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c009fd0 tx=0x7f0b6c009f20 comp rx=0 tx=0).stop 2024-04-12T06:02:42.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.432+0000 7f0b77a62700 1 -- 172.21.15.32:0/2946274779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b700656a0 msgr2=0x7f0b70113a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:42.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.432+0000 7f0b77a62700 1 --2- 172.21.15.32:0/2946274779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b700656a0 0x7f0b70113a10 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f0b6000b3b0 tx=0x7f0b60009e30 comp rx=0 tx=0).stop 2024-04-12T06:02:42.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.432+0000 7f0b77a62700 1 -- 172.21.15.32:0/2946274779 shutdown_connections 2024-04-12T06:02:42.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.432+0000 7f0b77a62700 1 --2- 172.21.15.32:0/2946274779 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0b5c06cc20 0x7f0b5c06f0d0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:42.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.432+0000 7f0b77a62700 1 --2- 172.21.15.32:0/2946274779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b700656a0 0x7f0b70113a10 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:42.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.432+0000 7f0b77a62700 1 --2- 172.21.15.32:0/2946274779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b70102de0 0x7f0b70113f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:42.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.432+0000 7f0b77a62700 1 -- 172.21.15.32:0/2946274779 >> 172.21.15.32:0/2946274779 conn(0x7f0b70072bc0 msgr2=0x7f0b70105b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:02:42.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.432+0000 7f0b77a62700 1 -- 172.21.15.32:0/2946274779 shutdown_connections 2024-04-12T06:02:42.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:42.432+0000 7f0b77a62700 1 -- 172.21.15.32:0/2946274779 wait complete. 2024-04-12T06:02:43.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:42 smithi032 ceph-mon[146817]: pgmap v666: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 5.1 MiB/s rd, 0 B/s wr, 4 op/s 2024-04-12T06:02:43.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:42 smithi067 ceph-mon[112979]: pgmap v666: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 5.1 MiB/s rd, 0 B/s wr, 4 op/s 2024-04-12T06:02:43.441 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph orch ps' 2024-04-12T06:02:43.728 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T06:02:44.178 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:43 smithi032 ceph-mon[146817]: from='client.35236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:02:44.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:43 smithi067 ceph-mon[112979]: from='client.35236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:02:45.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:44 smithi067 ceph-mon[112979]: pgmap v667: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 94 KiB/s rd, 3 op/s 2024-04-12T06:02:45.292 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:44 smithi032 ceph-mon[146817]: pgmap v667: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 94 KiB/s rd, 3 op/s 2024-04-12T06:02:45.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.595+0000 7f94d61f9700 1 -- 172.21.15.32:0/3605606737 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94d0102710 msgr2=0x7f94d00fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:45.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.595+0000 7f94d61f9700 1 --2- 172.21.15.32:0/3605606737 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94d0102710 0x7f94d00fbd80 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f94c4004940 tx=0x7f94c4009e30 comp rx=0 tx=0).stop 2024-04-12T06:02:45.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.595+0000 7f94d61f9700 1 -- 172.21.15.32:0/3605606737 shutdown_connections 2024-04-12T06:02:45.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.595+0000 7f94d61f9700 1 --2- 172.21.15.32:0/3605606737 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94d0102710 0x7f94d00fbd80 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:45.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.595+0000 7f94d61f9700 1 --2- 172.21.15.32:0/3605606737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94d0101d70 0x7f94d0102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:45.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.596+0000 7f94d61f9700 1 -- 172.21.15.32:0/3605606737 >> 172.21.15.32:0/3605606737 conn(0x7f94d0072bc0 msgr2=0x7f94d0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:02:45.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.596+0000 7f94d61f9700 1 -- 172.21.15.32:0/3605606737 shutdown_connections 2024-04-12T06:02:45.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.596+0000 7f94d61f9700 1 -- 172.21.15.32:0/3605606737 wait complete. 2024-04-12T06:02:45.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.596+0000 7f94d61f9700 1 Processor -- start 2024-04-12T06:02:45.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.597+0000 7f94d61f9700 1 -- start start 2024-04-12T06:02:45.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.597+0000 7f94d61f9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94d0101d70 0x7f94d010ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:45.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.597+0000 7f94d61f9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94d0102710 0x7f94d010f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:45.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.597+0000 7f94d61f9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d010f680 con 0x7f94d0101d70 2024-04-12T06:02:45.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.597+0000 7f94d61f9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d010f7c0 con 0x7f94d0102710 2024-04-12T06:02:45.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.597+0000 7f94cf7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94d0101d70 0x7f94d010ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:45.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.597+0000 7f94cf7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94d0101d70 0x7f94d010ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33408/0 (socket says 172.21.15.32:33408) 2024-04-12T06:02:45.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.597+0000 7f94cf7fe700 1 -- 172.21.15.32:0/1997654107 learned_addr learned my addr 172.21.15.32:0/1997654107 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:02:45.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.597+0000 7f94ceffd700 1 --2- 172.21.15.32:0/1997654107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94d0102710 0x7f94d010f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:45.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.597+0000 7f94cf7fe700 1 -- 172.21.15.32:0/1997654107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94d0102710 msgr2=0x7f94d010f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:45.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.597+0000 7f94cf7fe700 1 --2- 172.21.15.32:0/1997654107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94d0102710 0x7f94d010f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:45.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.597+0000 7f94cf7fe700 1 -- 172.21.15.32:0/1997654107 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c4005040 con 0x7f94d0101d70 2024-04-12T06:02:45.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.597+0000 7f94cf7fe700 1 --2- 172.21.15.32:0/1997654107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94d0101d70 0x7f94d010ec00 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f94c0002a50 tx=0x7f94c00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:02:45.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.598+0000 7f94b7fff700 1 -- 172.21.15.32:0/1997654107 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94c0007cc0 con 0x7f94d0101d70 2024-04-12T06:02:45.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.598+0000 7f94b7fff700 1 -- 172.21.15.32:0/1997654107 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f94c000f040 con 0x7f94d0101d70 2024-04-12T06:02:45.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.598+0000 7f94d61f9700 1 -- 172.21.15.32:0/1997654107 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94d0117ab0 con 0x7f94d0101d70 2024-04-12T06:02:45.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.598+0000 7f94b7fff700 1 -- 172.21.15.32:0/1997654107 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94c0009540 con 0x7f94d0101d70 2024-04-12T06:02:45.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.598+0000 7f94d61f9700 1 -- 172.21.15.32:0/1997654107 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94d0118040 con 0x7f94d0101d70 2024-04-12T06:02:45.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.599+0000 7f94d61f9700 1 -- 172.21.15.32:0/1997654107 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94d00ff080 con 0x7f94d0101d70 2024-04-12T06:02:45.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.599+0000 7f94b7fff700 1 -- 172.21.15.32:0/1997654107 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f94c0009700 con 0x7f94d0101d70 2024-04-12T06:02:45.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.600+0000 7f94b7fff700 1 --2- 172.21.15.32:0/1997654107 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f94b806cc20 0x7f94b806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:45.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.600+0000 7f94b7fff700 1 -- 172.21.15.32:0/1997654107 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f94c008bed0 con 0x7f94d0101d70 2024-04-12T06:02:45.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.600+0000 7f94ceffd700 1 --2- 172.21.15.32:0/1997654107 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f94b806cc20 0x7f94b806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:45.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.600+0000 7f94ceffd700 1 --2- 172.21.15.32:0/1997654107 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f94b806cc20 0x7f94b806f0d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f94c400b3b0 tx=0x7f94c4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:02:45.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.604+0000 7f94b7fff700 1 -- 172.21.15.32:0/1997654107 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f94c0055e30 con 0x7f94d0101d70 2024-04-12T06:02:45.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.815+0000 7f94d61f9700 1 -- 172.21.15.32:0/1997654107 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f94d004bed0 con 0x7f94b806cc20 2024-04-12T06:02:45.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.825+0000 7f94b7fff700 1 -- 172.21.15.32:0/1997654107 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f94d004bed0 con 0x7f94b806cc20 2024-04-12T06:02:45.820 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T06:02:45.820 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (22m) 26s ago 33m 28.4M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T06:02:45.820 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (20m) 26s ago 34m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T06:02:45.820 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (19m) 29s ago 33m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T06:02:45.820 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (22m) 26s ago 33m 88.0M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T06:02:45.820 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (16m) 26s ago 28m 95.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T06:02:45.820 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (16m) 26s ago 28m 119M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T06:02:45.820 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (15m) 29s ago 28m 17.1M - 18.2.2-1187-gd540ebac bb2c5eac26a4 6d4aca6f74a0 2024-04-12T06:02:45.820 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (15m) 29s ago 28m 20.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T06:02:45.820 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (24m) 26s ago 35m 511M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T06:02:45.820 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (23m) 29s ago 33m 456M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T06:02:45.821 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (21m) 26s ago 35m 70.2M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T06:02:45.821 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (20m) 29s ago 32m 50.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T06:02:45.821 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (23m) 26s ago 34m 19.0M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T06:02:45.821 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (23m) 29s ago 33m 18.7M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T06:02:45.821 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (19m) 26s ago 32m 96.9M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T06:02:45.821 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (19m) 26s ago 31m 110M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T06:02:45.821 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (18m) 26s ago 31m 89.6M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T06:02:45.821 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (17m) 29s ago 30m 122M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T06:02:45.821 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (17m) 29s ago 30m 95.7M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T06:02:45.821 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (16m) 29s ago 29m 95.3M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T06:02:45.821 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (22m) 26s ago 33m 76.6M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T06:02:45.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.827+0000 7f94d61f9700 1 -- 172.21.15.32:0/1997654107 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f94b806cc20 msgr2=0x7f94b806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:45.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.827+0000 7f94d61f9700 1 --2- 172.21.15.32:0/1997654107 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f94b806cc20 0x7f94b806f0d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f94c400b3b0 tx=0x7f94c4009e30 comp rx=0 tx=0).stop 2024-04-12T06:02:45.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.827+0000 7f94d61f9700 1 -- 172.21.15.32:0/1997654107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94d0101d70 msgr2=0x7f94d010ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:45.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.827+0000 7f94d61f9700 1 --2- 172.21.15.32:0/1997654107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94d0101d70 0x7f94d010ec00 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f94c0002a50 tx=0x7f94c00046b0 comp rx=0 tx=0).stop 2024-04-12T06:02:45.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.827+0000 7f94d61f9700 1 -- 172.21.15.32:0/1997654107 shutdown_connections 2024-04-12T06:02:45.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.827+0000 7f94d61f9700 1 --2- 172.21.15.32:0/1997654107 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f94b806cc20 0x7f94b806f0d0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:45.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.827+0000 7f94d61f9700 1 --2- 172.21.15.32:0/1997654107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94d0101d70 0x7f94d010ec00 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:45.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.827+0000 7f94d61f9700 1 --2- 172.21.15.32:0/1997654107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94d0102710 0x7f94d010f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:45.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.827+0000 7f94d61f9700 1 -- 172.21.15.32:0/1997654107 >> 172.21.15.32:0/1997654107 conn(0x7f94d0072bc0 msgr2=0x7f94d00fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:02:45.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.827+0000 7f94d61f9700 1 -- 172.21.15.32:0/1997654107 shutdown_connections 2024-04-12T06:02:45.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:45.827+0000 7f94d61f9700 1 -- 172.21.15.32:0/1997654107 wait complete. 2024-04-12T06:02:46.545 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph orch upgrade status' 2024-04-12T06:02:46.828 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T06:02:47.195 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:46 smithi032 ceph-mon[146817]: from='client.35240 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:02:47.196 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:46 smithi032 ceph-mon[146817]: pgmap v668: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 5.1 KiB/s rd, 3 op/s 2024-04-12T06:02:47.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:46 smithi067 ceph-mon[112979]: from='client.35240 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:02:47.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:46 smithi067 ceph-mon[112979]: pgmap v668: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 5.1 KiB/s rd, 3 op/s 2024-04-12T06:02:48.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:47 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:02:48.323 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:47 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:02:48.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.854+0000 7f838f736700 1 -- 172.21.15.32:0/3490633002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8388102710 msgr2=0x7f83880fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:48.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.854+0000 7f838f736700 1 --2- 172.21.15.32:0/3490633002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8388102710 0x7f83880fbd80 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f8378004940 tx=0x7f8378009e30 comp rx=0 tx=0).stop 2024-04-12T06:02:48.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.854+0000 7f838f736700 1 -- 172.21.15.32:0/3490633002 shutdown_connections 2024-04-12T06:02:48.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.854+0000 7f838f736700 1 --2- 172.21.15.32:0/3490633002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8388102710 0x7f83880fbd80 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:48.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.854+0000 7f838f736700 1 --2- 172.21.15.32:0/3490633002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8388101d70 0x7f8388102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:48.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.854+0000 7f838f736700 1 -- 172.21.15.32:0/3490633002 >> 172.21.15.32:0/3490633002 conn(0x7f8388072bc0 msgr2=0x7f8388072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:02:48.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.854+0000 7f838f736700 1 -- 172.21.15.32:0/3490633002 shutdown_connections 2024-04-12T06:02:48.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.854+0000 7f838f736700 1 -- 172.21.15.32:0/3490633002 wait complete. 2024-04-12T06:02:48.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.854+0000 7f838f736700 1 Processor -- start 2024-04-12T06:02:48.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.855+0000 7f838f736700 1 -- start start 2024-04-12T06:02:48.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.855+0000 7f838f736700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8388101d70 0x7f8388102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:48.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.855+0000 7f838f736700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8388102710 0x7f8388103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:48.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.855+0000 7f838f736700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83881036a0 con 0x7f8388102710 2024-04-12T06:02:48.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.855+0000 7f838f736700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83881037e0 con 0x7f8388101d70 2024-04-12T06:02:48.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.855+0000 7f838ccd1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8388102710 0x7f8388103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:48.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.855+0000 7f838ccd1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8388102710 0x7f8388103110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33414/0 (socket says 172.21.15.32:33414) 2024-04-12T06:02:48.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.855+0000 7f838ccd1700 1 -- 172.21.15.32:0/3499572475 learned_addr learned my addr 172.21.15.32:0/3499572475 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:02:48.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.855+0000 7f838d4d2700 1 --2- 172.21.15.32:0/3499572475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8388101d70 0x7f8388102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:48.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.855+0000 7f838ccd1700 1 -- 172.21.15.32:0/3499572475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8388101d70 msgr2=0x7f8388102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:48.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.855+0000 7f838ccd1700 1 --2- 172.21.15.32:0/3499572475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8388101d70 0x7f8388102bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:48.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.855+0000 7f838ccd1700 1 -- 172.21.15.32:0/3499572475 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8378005040 con 0x7f8388102710 2024-04-12T06:02:48.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.855+0000 7f838ccd1700 1 --2- 172.21.15.32:0/3499572475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8388102710 0x7f8388103110 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f8378009380 tx=0x7f8378009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:02:48.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.855+0000 7f837dffb700 1 -- 172.21.15.32:0/3499572475 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f837800e040 con 0x7f8388102710 2024-04-12T06:02:48.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.855+0000 7f838f736700 1 -- 172.21.15.32:0/3499572475 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8388103a30 con 0x7f8388102710 2024-04-12T06:02:48.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.856+0000 7f837dffb700 1 -- 172.21.15.32:0/3499572475 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8378013420 con 0x7f8388102710 2024-04-12T06:02:48.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.856+0000 7f837dffb700 1 -- 172.21.15.32:0/3499572475 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f837801c630 con 0x7f8388102710 2024-04-12T06:02:48.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.856+0000 7f838f736700 1 -- 172.21.15.32:0/3499572475 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f838811a1f0 con 0x7f8388102710 2024-04-12T06:02:48.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.857+0000 7f838f736700 1 -- 172.21.15.32:0/3499572475 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83880fd4c0 con 0x7f8388102710 2024-04-12T06:02:48.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.857+0000 7f837dffb700 1 -- 172.21.15.32:0/3499572475 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f83780139e0 con 0x7f8388102710 2024-04-12T06:02:48.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.857+0000 7f837dffb700 1 --2- 172.21.15.32:0/3499572475 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f837406cc20 0x7f837406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:48.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.858+0000 7f837dffb700 1 -- 172.21.15.32:0/3499572475 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f837808d310 con 0x7f8388102710 2024-04-12T06:02:48.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.858+0000 7f838d4d2700 1 --2- 172.21.15.32:0/3499572475 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f837406cc20 0x7f837406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:48.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.858+0000 7f838d4d2700 1 --2- 172.21.15.32:0/3499572475 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f837406cc20 0x7f837406f0d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f8384004990 tx=0x7f8384009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:02:48.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:48.862+0000 7f837dffb700 1 -- 172.21.15.32:0/3499572475 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8378057250 con 0x7f8388102710 2024-04-12T06:02:49.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:49.086+0000 7f838f736700 1 -- 172.21.15.32:0/3499572475 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8388108db0 con 0x7f837406cc20 2024-04-12T06:02:49.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:49.087+0000 7f837dffb700 1 -- 172.21.15.32:0/3499572475 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f8384004e60 con 0x7f837406cc20 2024-04-12T06:02:49.083 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T06:02:49.083 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": null, 2024-04-12T06:02:49.083 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": false, 2024-04-12T06:02:49.083 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "", 2024-04-12T06:02:49.083 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [], 2024-04-12T06:02:49.083 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": null, 2024-04-12T06:02:49.083 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "", 2024-04-12T06:02:49.083 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false 2024-04-12T06:02:49.083 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T06:02:49.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:49.089+0000 7f838f736700 1 -- 172.21.15.32:0/3499572475 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f837406cc20 msgr2=0x7f837406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:49.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:49.089+0000 7f838f736700 1 --2- 172.21.15.32:0/3499572475 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f837406cc20 0x7f837406f0d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f8384004990 tx=0x7f8384009ee0 comp rx=0 tx=0).stop 2024-04-12T06:02:49.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:49.089+0000 7f838f736700 1 -- 172.21.15.32:0/3499572475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8388102710 msgr2=0x7f8388103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:49.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:49.089+0000 7f838f736700 1 --2- 172.21.15.32:0/3499572475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8388102710 0x7f8388103110 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f8378009380 tx=0x7f8378009e30 comp rx=0 tx=0).stop 2024-04-12T06:02:49.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:49.089+0000 7f838f736700 1 -- 172.21.15.32:0/3499572475 shutdown_connections 2024-04-12T06:02:49.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:49.089+0000 7f838f736700 1 --2- 172.21.15.32:0/3499572475 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f837406cc20 0x7f837406f0d0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:49.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:49.089+0000 7f838f736700 1 --2- 172.21.15.32:0/3499572475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8388101d70 0x7f8388102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:49.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:49.089+0000 7f838f736700 1 --2- 172.21.15.32:0/3499572475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8388102710 0x7f8388103110 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:49.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:49.089+0000 7f838f736700 1 -- 172.21.15.32:0/3499572475 >> 172.21.15.32:0/3499572475 conn(0x7f8388072bc0 msgr2=0x7f83880659e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:02:49.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:49.089+0000 7f838f736700 1 -- 172.21.15.32:0/3499572475 shutdown_connections 2024-04-12T06:02:49.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:49.089+0000 7f838f736700 1 -- 172.21.15.32:0/3499572475 wait complete. 2024-04-12T06:02:49.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:48 smithi067 ceph-mon[112979]: pgmap v669: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:02:49.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:48 smithi032 ceph-mon[146817]: pgmap v669: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:02:49.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph health detail' 2024-04-12T06:02:50.098 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T06:02:51.224 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:50 smithi032 ceph-mon[146817]: from='client.35244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:02:51.224 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:50 smithi032 ceph-mon[146817]: pgmap v670: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:02:51.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:50 smithi067 ceph-mon[112979]: from='client.35244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:02:51.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:50 smithi067 ceph-mon[112979]: pgmap v670: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:02:52.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.292+0000 7f3951fa0700 1 -- 172.21.15.32:0/787633747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f394c102710 msgr2=0x7f394c0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:52.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.292+0000 7f3951fa0700 1 --2- 172.21.15.32:0/787633747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f394c102710 0x7f394c0fbd80 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f393c004940 tx=0x7f393c009e30 comp rx=0 tx=0).stop 2024-04-12T06:02:52.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.292+0000 7f3951fa0700 1 -- 172.21.15.32:0/787633747 shutdown_connections 2024-04-12T06:02:52.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.292+0000 7f3951fa0700 1 --2- 172.21.15.32:0/787633747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f394c102710 0x7f394c0fbd80 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:52.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.292+0000 7f3951fa0700 1 --2- 172.21.15.32:0/787633747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f394c101d70 0x7f394c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:52.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.292+0000 7f3951fa0700 1 -- 172.21.15.32:0/787633747 >> 172.21.15.32:0/787633747 conn(0x7f394c072bc0 msgr2=0x7f394c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:02:52.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.292+0000 7f3951fa0700 1 -- 172.21.15.32:0/787633747 shutdown_connections 2024-04-12T06:02:52.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.292+0000 7f3951fa0700 1 -- 172.21.15.32:0/787633747 wait complete. 2024-04-12T06:02:52.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.293+0000 7f3951fa0700 1 Processor -- start 2024-04-12T06:02:52.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.293+0000 7f3951fa0700 1 -- start start 2024-04-12T06:02:52.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.293+0000 7f3951fa0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f394c101d70 0x7f394c102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:52.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.293+0000 7f3951fa0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f394c102710 0x7f394c103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:52.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.293+0000 7f3951fa0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394c1036a0 con 0x7f394c102710 2024-04-12T06:02:52.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.293+0000 7f3951fa0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394c1037e0 con 0x7f394c101d70 2024-04-12T06:02:52.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.293+0000 7f394affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f394c102710 0x7f394c103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:52.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.293+0000 7f394affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f394c102710 0x7f394c103110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39656/0 (socket says 172.21.15.32:39656) 2024-04-12T06:02:52.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.293+0000 7f394affd700 1 -- 172.21.15.32:0/2900707305 learned_addr learned my addr 172.21.15.32:0/2900707305 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:02:52.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.294+0000 7f394b7fe700 1 --2- 172.21.15.32:0/2900707305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f394c101d70 0x7f394c102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:52.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.294+0000 7f394affd700 1 -- 172.21.15.32:0/2900707305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f394c101d70 msgr2=0x7f394c102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:52.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.294+0000 7f394affd700 1 --2- 172.21.15.32:0/2900707305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f394c101d70 0x7f394c102bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:52.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.294+0000 7f394affd700 1 -- 172.21.15.32:0/2900707305 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393c005040 con 0x7f394c102710 2024-04-12T06:02:52.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.294+0000 7f394b7fe700 1 --2- 172.21.15.32:0/2900707305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f394c101d70 0x7f394c102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T06:02:52.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.294+0000 7f394affd700 1 --2- 172.21.15.32:0/2900707305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f394c102710 0x7f394c103110 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f393c009380 tx=0x7f393c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:02:52.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.294+0000 7f3933fff700 1 -- 172.21.15.32:0/2900707305 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f393c00e040 con 0x7f394c102710 2024-04-12T06:02:52.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.294+0000 7f3933fff700 1 -- 172.21.15.32:0/2900707305 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f393c013420 con 0x7f394c102710 2024-04-12T06:02:52.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.294+0000 7f3951fa0700 1 -- 172.21.15.32:0/2900707305 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f394c103a30 con 0x7f394c102710 2024-04-12T06:02:52.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.294+0000 7f3933fff700 1 -- 172.21.15.32:0/2900707305 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f393c01c630 con 0x7f394c102710 2024-04-12T06:02:52.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.294+0000 7f3951fa0700 1 -- 172.21.15.32:0/2900707305 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f394c11a1f0 con 0x7f394c102710 2024-04-12T06:02:52.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.295+0000 7f3951fa0700 1 -- 172.21.15.32:0/2900707305 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f394c0fd4c0 con 0x7f394c102710 2024-04-12T06:02:52.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.296+0000 7f3933fff700 1 -- 172.21.15.32:0/2900707305 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f393c0139e0 con 0x7f394c102710 2024-04-12T06:02:52.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.296+0000 7f3933fff700 1 --2- 172.21.15.32:0/2900707305 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f393406cc20 0x7f393406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:52.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.296+0000 7f3933fff700 1 -- 172.21.15.32:0/2900707305 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f393c08d270 con 0x7f394c102710 2024-04-12T06:02:52.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.296+0000 7f394b7fe700 1 --2- 172.21.15.32:0/2900707305 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f393406cc20 0x7f393406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:52.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.297+0000 7f394b7fe700 1 --2- 172.21.15.32:0/2900707305 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f393406cc20 0x7f393406f0d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3940009fd0 tx=0x7f3940009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:02:52.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.301+0000 7f3933fff700 1 -- 172.21.15.32:0/2900707305 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f393c057250 con 0x7f394c102710 2024-04-12T06:02:52.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.591+0000 7f3951fa0700 1 -- 172.21.15.32:0/2900707305 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f394c108d40 con 0x7f394c102710 2024-04-12T06:02:52.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.591+0000 7f3933fff700 1 -- 172.21.15.32:0/2900707305 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f393c05a870 con 0x7f394c102710 2024-04-12T06:02:52.587 INFO:teuthology.orchestra.run.smithi032.stdout:HEALTH_OK 2024-04-12T06:02:52.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.593+0000 7f3951fa0700 1 -- 172.21.15.32:0/2900707305 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f393406cc20 msgr2=0x7f393406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:52.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.593+0000 7f3951fa0700 1 --2- 172.21.15.32:0/2900707305 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f393406cc20 0x7f393406f0d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3940009fd0 tx=0x7f3940009ee0 comp rx=0 tx=0).stop 2024-04-12T06:02:52.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.593+0000 7f3951fa0700 1 -- 172.21.15.32:0/2900707305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f394c102710 msgr2=0x7f394c103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:52.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.593+0000 7f3951fa0700 1 --2- 172.21.15.32:0/2900707305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f394c102710 0x7f394c103110 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f393c009380 tx=0x7f393c009e30 comp rx=0 tx=0).stop 2024-04-12T06:02:52.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.593+0000 7f3951fa0700 1 -- 172.21.15.32:0/2900707305 shutdown_connections 2024-04-12T06:02:52.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.593+0000 7f3951fa0700 1 --2- 172.21.15.32:0/2900707305 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f393406cc20 0x7f393406f0d0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:52.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.593+0000 7f3951fa0700 1 --2- 172.21.15.32:0/2900707305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f394c101d70 0x7f394c102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:52.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.593+0000 7f3951fa0700 1 --2- 172.21.15.32:0/2900707305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f394c102710 0x7f394c103110 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:52.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.593+0000 7f3951fa0700 1 -- 172.21.15.32:0/2900707305 >> 172.21.15.32:0/2900707305 conn(0x7f394c072bc0 msgr2=0x7f394c0659e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:02:52.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.593+0000 7f3951fa0700 1 -- 172.21.15.32:0/2900707305 shutdown_connections 2024-04-12T06:02:52.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:52.593+0000 7f3951fa0700 1 -- 172.21.15.32:0/2900707305 wait complete. 2024-04-12T06:02:53.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:52 smithi067 ceph-mon[112979]: pgmap v671: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:02:53.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:52 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/2900707305' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T06:02:53.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:52 smithi032 ceph-mon[146817]: pgmap v671: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:02:53.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:52 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/2900707305' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T06:02:53.452 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph versions' 2024-04-12T06:02:53.736 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T06:02:55.047 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:54 smithi032 ceph-mon[146817]: pgmap v672: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:02:55.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:54 smithi067 ceph-mon[112979]: pgmap v672: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:02:55.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.616+0000 7f9885d9e700 1 -- 172.21.15.32:0/3207924187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9880101d70 msgr2=0x7f9880102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:55.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.616+0000 7f9885d9e700 1 --2- 172.21.15.32:0/3207924187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9880101d70 0x7f9880102140 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f9870004940 tx=0x7f9870009e30 comp rx=0 tx=0).stop 2024-04-12T06:02:55.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.616+0000 7f9885d9e700 1 -- 172.21.15.32:0/3207924187 shutdown_connections 2024-04-12T06:02:55.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.616+0000 7f9885d9e700 1 --2- 172.21.15.32:0/3207924187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9880102710 0x7f98800fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:55.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.616+0000 7f9885d9e700 1 --2- 172.21.15.32:0/3207924187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9880101d70 0x7f9880102140 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:55.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.616+0000 7f9885d9e700 1 -- 172.21.15.32:0/3207924187 >> 172.21.15.32:0/3207924187 conn(0x7f9880072bc0 msgr2=0x7f9880072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:02:55.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.616+0000 7f9885d9e700 1 -- 172.21.15.32:0/3207924187 shutdown_connections 2024-04-12T06:02:55.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.616+0000 7f9885d9e700 1 -- 172.21.15.32:0/3207924187 wait complete. 2024-04-12T06:02:55.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.617+0000 7f9885d9e700 1 Processor -- start 2024-04-12T06:02:55.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.617+0000 7f9885d9e700 1 -- start start 2024-04-12T06:02:55.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.617+0000 7f9885d9e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9880101d70 0x7f988010eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:55.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.617+0000 7f9885d9e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9880102710 0x7f988010f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:55.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.617+0000 7f9885d9e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988010f5e0 con 0x7f9880101d70 2024-04-12T06:02:55.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.617+0000 7f9885d9e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988010f720 con 0x7f9880102710 2024-04-12T06:02:55.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.617+0000 7f987f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9880101d70 0x7f988010eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:55.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.617+0000 7f987f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9880101d70 0x7f988010eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39674/0 (socket says 172.21.15.32:39674) 2024-04-12T06:02:55.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.617+0000 7f987f7fe700 1 -- 172.21.15.32:0/2876171667 learned_addr learned my addr 172.21.15.32:0/2876171667 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:02:55.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.617+0000 7f987effd700 1 --2- 172.21.15.32:0/2876171667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9880102710 0x7f988010f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:55.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.617+0000 7f987f7fe700 1 -- 172.21.15.32:0/2876171667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9880102710 msgr2=0x7f988010f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:55.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.617+0000 7f987f7fe700 1 --2- 172.21.15.32:0/2876171667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9880102710 0x7f988010f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:55.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.617+0000 7f987f7fe700 1 -- 172.21.15.32:0/2876171667 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9870005040 con 0x7f9880101d70 2024-04-12T06:02:55.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.618+0000 7f987f7fe700 1 --2- 172.21.15.32:0/2876171667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9880101d70 0x7f988010eb60 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f9870009180 tx=0x7f9870009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:02:55.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.618+0000 7f9877fff700 1 -- 172.21.15.32:0/2876171667 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f987000e040 con 0x7f9880101d70 2024-04-12T06:02:55.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.618+0000 7f9877fff700 1 -- 172.21.15.32:0/2876171667 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9870004290 con 0x7f9880101d70 2024-04-12T06:02:55.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.618+0000 7f9885d9e700 1 -- 172.21.15.32:0/2876171667 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9880117a10 con 0x7f9880101d70 2024-04-12T06:02:55.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.618+0000 7f9877fff700 1 -- 172.21.15.32:0/2876171667 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9870013e60 con 0x7f9880101d70 2024-04-12T06:02:55.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.618+0000 7f9885d9e700 1 -- 172.21.15.32:0/2876171667 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9880117fa0 con 0x7f9880101d70 2024-04-12T06:02:55.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.619+0000 7f9885d9e700 1 -- 172.21.15.32:0/2876171667 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98800ff080 con 0x7f9880101d70 2024-04-12T06:02:55.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.619+0000 7f9877fff700 1 -- 172.21.15.32:0/2876171667 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f98700136b0 con 0x7f9880101d70 2024-04-12T06:02:55.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.620+0000 7f9877fff700 1 --2- 172.21.15.32:0/2876171667 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f986c06cbb0 0x7f986c06f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:02:55.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.620+0000 7f9877fff700 1 -- 172.21.15.32:0/2876171667 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f987008d300 con 0x7f9880101d70 2024-04-12T06:02:55.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.620+0000 7f987effd700 1 --2- 172.21.15.32:0/2876171667 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f986c06cbb0 0x7f986c06f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:02:55.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.620+0000 7f987effd700 1 --2- 172.21.15.32:0/2876171667 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f986c06cbb0 0x7f986c06f060 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9880066020 tx=0x7f986800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:02:55.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.624+0000 7f9877fff700 1 -- 172.21.15.32:0/2876171667 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9870057390 con 0x7f9880101d70 2024-04-12T06:02:55.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.910+0000 7f9885d9e700 1 -- 172.21.15.32:0/2876171667 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9880108dc0 con 0x7f9880101d70 2024-04-12T06:02:55.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.911+0000 7f9877fff700 1 -- 172.21.15.32:0/2876171667 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f987005a9b0 con 0x7f9880101d70 2024-04-12T06:02:55.907 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T06:02:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T06:02:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T06:02:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T06:02:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T06:02:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T06:02:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T06:02:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T06:02:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T06:02:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T06:02:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T06:02:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T06:02:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T06:02:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T06:02:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T06:02:55.908 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T06:02:55.908 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T06:02:55.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.913+0000 7f9885d9e700 1 -- 172.21.15.32:0/2876171667 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f986c06cbb0 msgr2=0x7f986c06f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:55.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.913+0000 7f9885d9e700 1 --2- 172.21.15.32:0/2876171667 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f986c06cbb0 0x7f986c06f060 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9880066020 tx=0x7f986800b040 comp rx=0 tx=0).stop 2024-04-12T06:02:55.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.913+0000 7f9885d9e700 1 -- 172.21.15.32:0/2876171667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9880101d70 msgr2=0x7f988010eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:02:55.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.913+0000 7f9885d9e700 1 --2- 172.21.15.32:0/2876171667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9880101d70 0x7f988010eb60 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f9870009180 tx=0x7f9870009e30 comp rx=0 tx=0).stop 2024-04-12T06:02:55.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.913+0000 7f9885d9e700 1 -- 172.21.15.32:0/2876171667 shutdown_connections 2024-04-12T06:02:55.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.913+0000 7f9885d9e700 1 --2- 172.21.15.32:0/2876171667 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f986c06cbb0 0x7f986c06f060 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:55.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.913+0000 7f9885d9e700 1 --2- 172.21.15.32:0/2876171667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9880101d70 0x7f988010eb60 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:55.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.913+0000 7f9885d9e700 1 --2- 172.21.15.32:0/2876171667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9880102710 0x7f988010f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:02:55.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.913+0000 7f9885d9e700 1 -- 172.21.15.32:0/2876171667 >> 172.21.15.32:0/2876171667 conn(0x7f9880072bc0 msgr2=0x7f98800faf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:02:55.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.913+0000 7f9885d9e700 1 -- 172.21.15.32:0/2876171667 shutdown_connections 2024-04-12T06:02:55.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:02:55.913+0000 7f9885d9e700 1 -- 172.21.15.32:0/2876171667 wait complete. 2024-04-12T06:02:56.256 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:55 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/2876171667' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:56.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:55 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/2876171667' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:02:56.633 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-12T06:02:56.916 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T06:02:57.175 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:56 smithi032 ceph-mon[146817]: pgmap v673: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:02:57.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:56 smithi067 ceph-mon[112979]: pgmap v673: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:02:58.245 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:02:57 smithi032 ceph-mon[146817]: pgmap v674: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:02:58.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:02:57 smithi067 ceph-mon[112979]: pgmap v674: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:02:58.833 INFO:teuthology.orchestra.run.smithi032.stdout:wait for servicemap items w/ changing names to refresh 2024-04-12T06:02:59.434 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'sleep 60' 2024-04-12T06:02:59.718 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T06:03:01.176 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:00 smithi032 ceph-mon[146817]: pgmap v675: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:03:01.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:00 smithi067 ceph-mon[112979]: pgmap v675: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:03:03.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:02 smithi067 ceph-mon[112979]: pgmap v676: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:03:03.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:02 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:03:03.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:02 smithi032 ceph-mon[146817]: pgmap v676: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:03:03.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:02 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:03:05.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:04 smithi067 ceph-mon[112979]: pgmap v677: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:03:05.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:04 smithi032 ceph-mon[146817]: pgmap v677: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:03:07.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:06 smithi067 ceph-mon[112979]: pgmap v678: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:03:07.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:06 smithi032 ceph-mon[146817]: pgmap v678: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:03:09.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:08 smithi067 ceph-mon[112979]: pgmap v679: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:03:09.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:08 smithi032 ceph-mon[146817]: pgmap v679: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:03:11.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:10 smithi067 ceph-mon[112979]: pgmap v680: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:03:11.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:10 smithi032 ceph-mon[146817]: pgmap v680: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:03:13.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:12 smithi067 ceph-mon[112979]: pgmap v681: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:03:13.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:12 smithi032 ceph-mon[146817]: pgmap v681: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:03:15.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:14 smithi067 ceph-mon[112979]: pgmap v682: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:03:15.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:14 smithi032 ceph-mon[146817]: pgmap v682: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:03:17.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:16 smithi067 ceph-mon[112979]: pgmap v683: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:03:17.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:16 smithi032 ceph-mon[146817]: pgmap v683: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:03:18.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:17 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:03:18.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:17 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:03:19.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:18 smithi067 ceph-mon[112979]: pgmap v684: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:03:19.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:18 smithi032 ceph-mon[146817]: pgmap v684: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:03:21.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:20 smithi067 ceph-mon[112979]: pgmap v685: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:03:21.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:20 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T06:03:21.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:20 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T06:03:21.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:20 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T06:03:21.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:20 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:03:21.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:20 smithi032 ceph-mon[146817]: pgmap v685: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:03:21.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:20 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T06:03:21.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:20 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T06:03:21.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:20 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T06:03:21.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:20 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:03:23.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:22 smithi067 ceph-mon[112979]: pgmap v686: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:03:23.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:22 smithi032 ceph-mon[146817]: pgmap v686: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:03:25.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:24 smithi067 ceph-mon[112979]: pgmap v687: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:03:25.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:24 smithi032 ceph-mon[146817]: pgmap v687: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:03:27.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:26 smithi067 ceph-mon[112979]: pgmap v688: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:03:27.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:26 smithi032 ceph-mon[146817]: pgmap v688: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:03:29.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:28 smithi067 ceph-mon[112979]: pgmap v689: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:03:29.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:28 smithi032 ceph-mon[146817]: pgmap v689: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:03:31.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:30 smithi067 ceph-mon[112979]: pgmap v690: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:03:31.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:30 smithi032 ceph-mon[146817]: pgmap v690: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:03:33.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:32 smithi067 ceph-mon[112979]: pgmap v691: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:03:33.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:32 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:03:33.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:32 smithi032 ceph-mon[146817]: pgmap v691: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:03:33.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:32 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:03:34.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:33 smithi067 ceph-mon[112979]: pgmap v692: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:03:34.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:33 smithi032 ceph-mon[146817]: pgmap v692: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:03:37.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:36 smithi067 ceph-mon[112979]: pgmap v693: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:03:37.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:36 smithi032 ceph-mon[146817]: pgmap v693: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:03:39.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:38 smithi067 ceph-mon[112979]: pgmap v694: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:03:39.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:38 smithi032 ceph-mon[146817]: pgmap v694: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:03:41.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:40 smithi067 ceph-mon[112979]: pgmap v695: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:03:41.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:40 smithi032 ceph-mon[146817]: pgmap v695: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:03:43.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:42 smithi067 ceph-mon[112979]: pgmap v696: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:03:43.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:42 smithi032 ceph-mon[146817]: pgmap v696: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:03:45.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:44 smithi067 ceph-mon[112979]: pgmap v697: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:03:45.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:44 smithi032 ceph-mon[146817]: pgmap v697: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:03:47.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:46 smithi067 ceph-mon[112979]: pgmap v698: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:03:47.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:46 smithi032 ceph-mon[146817]: pgmap v698: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:03:48.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:47 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:03:48.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:47 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:03:49.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:48 smithi067 ceph-mon[112979]: pgmap v699: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:03:49.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:48 smithi032 ceph-mon[146817]: pgmap v699: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:03:51.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:50 smithi067 ceph-mon[112979]: pgmap v700: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:03:51.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:50 smithi032 ceph-mon[146817]: pgmap v700: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:03:53.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:52 smithi067 ceph-mon[112979]: pgmap v701: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:03:53.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:52 smithi032 ceph-mon[146817]: pgmap v701: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:03:55.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:54 smithi067 ceph-mon[112979]: pgmap v702: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:03:55.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:54 smithi032 ceph-mon[146817]: pgmap v702: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:03:57.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:56 smithi067 ceph-mon[112979]: pgmap v703: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:03:57.337 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:56 smithi032 ceph-mon[146817]: pgmap v703: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:03:59.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:03:58 smithi067 ceph-mon[112979]: pgmap v704: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:03:59.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:03:58 smithi032 ceph-mon[146817]: pgmap v704: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:04:01.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:00 smithi067 ceph-mon[112979]: pgmap v705: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:04:01.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:00 smithi032 ceph-mon[146817]: pgmap v705: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:04:02.368 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph orch ps' 2024-04-12T06:04:02.653 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T06:04:03.252 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:02 smithi032 ceph-mon[146817]: pgmap v706: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:04:03.252 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:02 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:04:03.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:02 smithi067 ceph-mon[112979]: pgmap v706: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:04:03.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:02 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:04:04.251 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:03 smithi032 ceph-mon[146817]: pgmap v707: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:04:04.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:03 smithi067 ceph-mon[112979]: pgmap v707: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:04:04.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.536+0000 7fe75da4e700 1 -- 172.21.15.32:0/2762069168 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7581037a0 msgr2=0x7fe7580fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:04.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.536+0000 7fe75da4e700 1 --2- 172.21.15.32:0/2762069168 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7581037a0 0x7fe7580fce10 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fe740004940 tx=0x7fe740009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:04.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.536+0000 7fe75da4e700 1 -- 172.21.15.32:0/2762069168 shutdown_connections 2024-04-12T06:04:04.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.536+0000 7fe75da4e700 1 --2- 172.21.15.32:0/2762069168 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7581037a0 0x7fe7580fce10 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:04.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.536+0000 7fe75da4e700 1 --2- 172.21.15.32:0/2762069168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe758102e00 0x7fe7581031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:04.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.536+0000 7fe75da4e700 1 -- 172.21.15.32:0/2762069168 >> 172.21.15.32:0/2762069168 conn(0x7fe7580f89b0 msgr2=0x7fe7580fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:04.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.536+0000 7fe75da4e700 1 -- 172.21.15.32:0/2762069168 shutdown_connections 2024-04-12T06:04:04.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.536+0000 7fe75da4e700 1 -- 172.21.15.32:0/2762069168 wait complete. 2024-04-12T06:04:04.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.536+0000 7fe75da4e700 1 Processor -- start 2024-04-12T06:04:04.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.537+0000 7fe75da4e700 1 -- start start 2024-04-12T06:04:04.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.537+0000 7fe75da4e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe758102e00 0x7fe7580702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:04.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.537+0000 7fe75da4e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7581037a0 0x7fe758070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:04.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.537+0000 7fe75da4e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe758072210 con 0x7fe7581037a0 2024-04-12T06:04:04.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.537+0000 7fe75da4e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe758070d40 con 0x7fe758102e00 2024-04-12T06:04:04.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.537+0000 7fe7567fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7581037a0 0x7fe758070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:04.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.537+0000 7fe7567fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7581037a0 0x7fe758070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58686/0 (socket says 172.21.15.32:58686) 2024-04-12T06:04:04.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.537+0000 7fe7567fc700 1 -- 172.21.15.32:0/740433891 learned_addr learned my addr 172.21.15.32:0/740433891 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:04.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.537+0000 7fe756ffd700 1 --2- 172.21.15.32:0/740433891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe758102e00 0x7fe7580702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:04.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.537+0000 7fe7567fc700 1 -- 172.21.15.32:0/740433891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe758102e00 msgr2=0x7fe7580702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:04.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.537+0000 7fe7567fc700 1 --2- 172.21.15.32:0/740433891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe758102e00 0x7fe7580702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:04.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.537+0000 7fe7567fc700 1 -- 172.21.15.32:0/740433891 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe740005040 con 0x7fe7581037a0 2024-04-12T06:04:04.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.537+0000 7fe7567fc700 1 --2- 172.21.15.32:0/740433891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7581037a0 0x7fe758070800 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fe740009390 tx=0x7fe740009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:04.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.538+0000 7fe74f7fe700 1 -- 172.21.15.32:0/740433891 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe74000e040 con 0x7fe7581037a0 2024-04-12T06:04:04.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.538+0000 7fe75da4e700 1 -- 172.21.15.32:0/740433891 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe758070fc0 con 0x7fe7581037a0 2024-04-12T06:04:04.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.538+0000 7fe74f7fe700 1 -- 172.21.15.32:0/740433891 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe740004290 con 0x7fe7581037a0 2024-04-12T06:04:04.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.538+0000 7fe74f7fe700 1 -- 172.21.15.32:0/740433891 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe740013e70 con 0x7fe7581037a0 2024-04-12T06:04:04.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.538+0000 7fe75da4e700 1 -- 172.21.15.32:0/740433891 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe75811a010 con 0x7fe7581037a0 2024-04-12T06:04:04.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.539+0000 7fe75da4e700 1 -- 172.21.15.32:0/740433891 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7580fe550 con 0x7fe7581037a0 2024-04-12T06:04:04.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.539+0000 7fe74f7fe700 1 -- 172.21.15.32:0/740433891 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe740013690 con 0x7fe7581037a0 2024-04-12T06:04:04.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.540+0000 7fe74f7fe700 1 --2- 172.21.15.32:0/740433891 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe74406cbc0 0x7fe74406f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:04.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.540+0000 7fe756ffd700 1 --2- 172.21.15.32:0/740433891 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe74406cbc0 0x7fe74406f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:04.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.540+0000 7fe74f7fe700 1 -- 172.21.15.32:0/740433891 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fe74008d290 con 0x7fe7581037a0 2024-04-12T06:04:04.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.540+0000 7fe756ffd700 1 --2- 172.21.15.32:0/740433891 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe74406cbc0 0x7fe74406f070 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe748004f40 tx=0x7fe74800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:04.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.545+0000 7fe74f7fe700 1 -- 172.21.15.32:0/740433891 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe740057320 con 0x7fe7581037a0 2024-04-12T06:04:04.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.757+0000 7fe75da4e700 1 -- 172.21.15.32:0/740433891 --> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7580fa140 con 0x7fe74406cbc0 2024-04-12T06:04:04.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.768+0000 7fe74f7fe700 1 -- 172.21.15.32:0/740433891 <== mgr.34104 v2:172.21.15.32:6800/3171834473 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fe7580fa140 con 0x7fe74406cbc0 2024-04-12T06:04:04.764 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T06:04:04.764 INFO:teuthology.orchestra.run.smithi032.stdout:alertmanager.smithi032 smithi032 *:9093,9094 running (23m) 105s ago 35m 28.4M - 0.25.0 c8568f914cd2 9de9bcce2386 2024-04-12T06:04:04.764 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi032 smithi032 running (21m) 105s ago 35m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bf4b87cb0901 2024-04-12T06:04:04.764 INFO:teuthology.orchestra.run.smithi032.stdout:crash.smithi067 smithi067 running (21m) 108s ago 34m 6651k - 18.2.2-1187-gd540ebac bb2c5eac26a4 6995687efeca 2024-04-12T06:04:04.764 INFO:teuthology.orchestra.run.smithi032.stdout:grafana.smithi032 smithi032 *:3000 running (23m) 105s ago 35m 88.0M - 9.4.7 954c08fa6188 0114a0412ef3 2024-04-12T06:04:04.764 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.jehaaj smithi032 running (17m) 105s ago 29m 95.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 2df9f52cc254 2024-04-12T06:04:04.764 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi032.pndtry smithi032 running (17m) 105s ago 29m 119M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd099fea7c61 2024-04-12T06:04:04.764 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.ymvqih smithi067 running (16m) 108s ago 29m 17.1M - 18.2.2-1187-gd540ebac bb2c5eac26a4 6d4aca6f74a0 2024-04-12T06:04:04.765 INFO:teuthology.orchestra.run.smithi032.stdout:mds.cephfs.smithi067.zesmgy smithi067 running (17m) 108s ago 29m 20.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 c61668a9e650 2024-04-12T06:04:04.765 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi032.hrwxrh smithi032 *:8443,9283,8765 running (25m) 105s ago 36m 511M - 18.2.2-1187-gd540ebac bb2c5eac26a4 bd45a204935e 2024-04-12T06:04:04.765 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.smithi067.dwichg smithi067 *:8443,9283,8765 running (24m) 108s ago 34m 456M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0783ab768199 2024-04-12T06:04:04.765 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi032 smithi032 running (22m) 105s ago 36m 70.2M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 ad789df6d5c7 2024-04-12T06:04:04.765 INFO:teuthology.orchestra.run.smithi032.stdout:mon.smithi067 smithi067 running (22m) 108s ago 34m 50.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 b4591addf064 2024-04-12T06:04:04.765 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi032 smithi032 *:9100 running (24m) 105s ago 35m 19.0M - 1.5.0 0da6a335fe13 5a5d04d11e6d 2024-04-12T06:04:04.765 INFO:teuthology.orchestra.run.smithi032.stdout:node-exporter.smithi067 smithi067 *:9100 running (24m) 108s ago 34m 18.7M - 1.5.0 0da6a335fe13 7b101883fd19 2024-04-12T06:04:04.765 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (20m) 105s ago 33m 96.9M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 ab15e07d77a2 2024-04-12T06:04:04.765 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi032 running (20m) 105s ago 33m 110M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 2e7e5c6470fd 2024-04-12T06:04:04.765 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi032 running (19m) 105s ago 32m 89.6M 1570M 18.2.2-1187-gd540ebac bb2c5eac26a4 7d9052dbda4f 2024-04-12T06:04:04.765 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi067 running (19m) 108s ago 32m 122M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 b30df94d2d14 2024-04-12T06:04:04.765 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi067 running (18m) 108s ago 31m 95.7M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 30bb5e8d2751 2024-04-12T06:04:04.765 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi067 running (18m) 108s ago 31m 95.3M 2592M 18.2.2-1187-gd540ebac bb2c5eac26a4 775899bf72bf 2024-04-12T06:04:04.765 INFO:teuthology.orchestra.run.smithi032.stdout:prometheus.smithi032 smithi032 *:9095 running (24m) 105s ago 34m 76.6M - 2.43.0 a07b618ecd1d c276326fbff2 2024-04-12T06:04:04.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.770+0000 7fe75da4e700 1 -- 172.21.15.32:0/740433891 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe74406cbc0 msgr2=0x7fe74406f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:04.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.770+0000 7fe75da4e700 1 --2- 172.21.15.32:0/740433891 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe74406cbc0 0x7fe74406f070 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe748004f40 tx=0x7fe74800b040 comp rx=0 tx=0).stop 2024-04-12T06:04:04.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.770+0000 7fe75da4e700 1 -- 172.21.15.32:0/740433891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7581037a0 msgr2=0x7fe758070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:04.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.770+0000 7fe75da4e700 1 --2- 172.21.15.32:0/740433891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7581037a0 0x7fe758070800 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fe740009390 tx=0x7fe740009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:04.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.770+0000 7fe75da4e700 1 -- 172.21.15.32:0/740433891 shutdown_connections 2024-04-12T06:04:04.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.770+0000 7fe75da4e700 1 --2- 172.21.15.32:0/740433891 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fe74406cbc0 0x7fe74406f070 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:04.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.770+0000 7fe75da4e700 1 --2- 172.21.15.32:0/740433891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe758102e00 0x7fe7580702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:04.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.770+0000 7fe75da4e700 1 --2- 172.21.15.32:0/740433891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7581037a0 0x7fe758070800 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:04.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.770+0000 7fe75da4e700 1 -- 172.21.15.32:0/740433891 >> 172.21.15.32:0/740433891 conn(0x7fe7580f89b0 msgr2=0x7fe7580f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:04.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.770+0000 7fe75da4e700 1 -- 172.21.15.32:0/740433891 shutdown_connections 2024-04-12T06:04:04.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:04.770+0000 7fe75da4e700 1 -- 172.21.15.32:0/740433891 wait complete. 2024-04-12T06:04:05.087 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:04 smithi032 ceph-mon[146817]: from='client.35256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:04:05.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:04 smithi067 ceph-mon[112979]: from='client.35256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T06:04:05.485 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph versions' 2024-04-12T06:04:05.772 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T06:04:06.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:05 smithi067 ceph-mon[112979]: pgmap v708: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:04:06.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:05 smithi032 ceph-mon[146817]: pgmap v708: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:04:08.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.031+0000 7fb693da5700 1 -- 172.21.15.32:0/2598173660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb68c1037b0 msgr2=0x7fb68c0fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:08.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.031+0000 7fb693da5700 1 --2- 172.21.15.32:0/2598173660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb68c1037b0 0x7fb68c0fce20 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fb680004940 tx=0x7fb680009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:08.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.032+0000 7fb693da5700 1 -- 172.21.15.32:0/2598173660 shutdown_connections 2024-04-12T06:04:08.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.032+0000 7fb693da5700 1 --2- 172.21.15.32:0/2598173660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb68c1037b0 0x7fb68c0fce20 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:08.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.032+0000 7fb693da5700 1 --2- 172.21.15.32:0/2598173660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb68c102e10 0x7fb68c1031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:08.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.032+0000 7fb693da5700 1 -- 172.21.15.32:0/2598173660 >> 172.21.15.32:0/2598173660 conn(0x7fb68c0f8980 msgr2=0x7fb68c0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:08.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.032+0000 7fb693da5700 1 -- 172.21.15.32:0/2598173660 shutdown_connections 2024-04-12T06:04:08.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.032+0000 7fb693da5700 1 -- 172.21.15.32:0/2598173660 wait complete. 2024-04-12T06:04:08.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.032+0000 7fb693da5700 1 Processor -- start 2024-04-12T06:04:08.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.032+0000 7fb693da5700 1 -- start start 2024-04-12T06:04:08.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.033+0000 7fb693da5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb68c102e10 0x7fb68c113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:08.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.033+0000 7fb693da5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb68c1037b0 0x7fb68c113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:08.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.033+0000 7fb693da5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c1145f0 con 0x7fb68c102e10 2024-04-12T06:04:08.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.033+0000 7fb693da5700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c10d9e0 con 0x7fb68c1037b0 2024-04-12T06:04:08.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.033+0000 7fb691b41700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb68c102e10 0x7fb68c113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:08.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.033+0000 7fb691b41700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb68c102e10 0x7fb68c113a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58704/0 (socket says 172.21.15.32:58704) 2024-04-12T06:04:08.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.033+0000 7fb691b41700 1 -- 172.21.15.32:0/395978723 learned_addr learned my addr 172.21.15.32:0/395978723 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:08.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.033+0000 7fb691340700 1 --2- 172.21.15.32:0/395978723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb68c1037b0 0x7fb68c113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:08.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.033+0000 7fb691b41700 1 -- 172.21.15.32:0/395978723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb68c1037b0 msgr2=0x7fb68c113fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:08.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.033+0000 7fb691b41700 1 --2- 172.21.15.32:0/395978723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb68c1037b0 0x7fb68c113fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:08.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.033+0000 7fb691b41700 1 -- 172.21.15.32:0/395978723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb680005040 con 0x7fb68c102e10 2024-04-12T06:04:08.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.033+0000 7fb691340700 1 --2- 172.21.15.32:0/395978723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb68c1037b0 0x7fb68c113fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T06:04:08.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.033+0000 7fb691b41700 1 --2- 172.21.15.32:0/395978723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb68c102e10 0x7fb68c113a60 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fb68800eb40 tx=0x7fb68800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:08.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.034+0000 7fb67e7fc700 1 -- 172.21.15.32:0/395978723 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb688007cc0 con 0x7fb68c102e10 2024-04-12T06:04:08.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.034+0000 7fb693da5700 1 -- 172.21.15.32:0/395978723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb68c10dc60 con 0x7fb68c102e10 2024-04-12T06:04:08.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.034+0000 7fb67e7fc700 1 -- 172.21.15.32:0/395978723 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb68800f040 con 0x7fb68c102e10 2024-04-12T06:04:08.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.034+0000 7fb67e7fc700 1 -- 172.21.15.32:0/395978723 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb688009570 con 0x7fb68c102e10 2024-04-12T06:04:08.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.034+0000 7fb693da5700 1 -- 172.21.15.32:0/395978723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb68c10e1f0 con 0x7fb68c102e10 2024-04-12T06:04:08.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.035+0000 7fb693da5700 1 -- 172.21.15.32:0/395978723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb68c0fe560 con 0x7fb68c102e10 2024-04-12T06:04:08.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.035+0000 7fb67e7fc700 1 -- 172.21.15.32:0/395978723 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb6880096d0 con 0x7fb68c102e10 2024-04-12T06:04:08.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.036+0000 7fb67e7fc700 1 --2- 172.21.15.32:0/395978723 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb67806cc30 0x7fb67806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:08.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.036+0000 7fb67e7fc700 1 -- 172.21.15.32:0/395978723 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fb68808bd90 con 0x7fb68c102e10 2024-04-12T06:04:08.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.036+0000 7fb691340700 1 --2- 172.21.15.32:0/395978723 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb67806cc30 0x7fb67806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:08.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.036+0000 7fb691340700 1 --2- 172.21.15.32:0/395978723 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb67806cc30 0x7fb67806f0e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb68c10ecc0 tx=0x7fb680009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:08.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.041+0000 7fb67e7fc700 1 -- 172.21.15.32:0/395978723 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb688059830 con 0x7fb68c102e10 2024-04-12T06:04:08.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.328+0000 7fb693da5700 1 -- 172.21.15.32:0/395978723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb68c109db0 con 0x7fb68c102e10 2024-04-12T06:04:08.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.329+0000 7fb67e7fc700 1 -- 172.21.15.32:0/395978723 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fb6880593c0 con 0x7fb68c102e10 2024-04-12T06:04:08.326 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-12T06:04:08.326 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": { 2024-04-12T06:04:08.326 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T06:04:08.326 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T06:04:08.326 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": { 2024-04-12T06:04:08.326 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T06:04:08.326 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T06:04:08.327 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": { 2024-04-12T06:04:08.327 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T06:04:08.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T06:04:08.327 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-12T06:04:08.327 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T06:04:08.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-12T06:04:08.327 INFO:teuthology.orchestra.run.smithi032.stdout: "overall": { 2024-04-12T06:04:08.327 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T06:04:08.327 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-12T06:04:08.327 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-12T06:04:08.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.331+0000 7fb693da5700 1 -- 172.21.15.32:0/395978723 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb67806cc30 msgr2=0x7fb67806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:08.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.331+0000 7fb693da5700 1 --2- 172.21.15.32:0/395978723 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb67806cc30 0x7fb67806f0e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb68c10ecc0 tx=0x7fb680009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:08.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.331+0000 7fb693da5700 1 -- 172.21.15.32:0/395978723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb68c102e10 msgr2=0x7fb68c113a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:08.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.331+0000 7fb693da5700 1 --2- 172.21.15.32:0/395978723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb68c102e10 0x7fb68c113a60 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fb68800eb40 tx=0x7fb68800b040 comp rx=0 tx=0).stop 2024-04-12T06:04:08.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.331+0000 7fb693da5700 1 -- 172.21.15.32:0/395978723 shutdown_connections 2024-04-12T06:04:08.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.331+0000 7fb693da5700 1 --2- 172.21.15.32:0/395978723 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb67806cc30 0x7fb67806f0e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:08.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.331+0000 7fb693da5700 1 --2- 172.21.15.32:0/395978723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb68c102e10 0x7fb68c113a60 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:08.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.331+0000 7fb693da5700 1 --2- 172.21.15.32:0/395978723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb68c1037b0 0x7fb68c113fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:08.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.331+0000 7fb693da5700 1 -- 172.21.15.32:0/395978723 >> 172.21.15.32:0/395978723 conn(0x7fb68c0f8980 msgr2=0x7fb68c0f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:08.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.331+0000 7fb693da5700 1 -- 172.21.15.32:0/395978723 shutdown_connections 2024-04-12T06:04:08.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:08.331+0000 7fb693da5700 1 -- 172.21.15.32:0/395978723 wait complete. 2024-04-12T06:04:08.991 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:08 smithi032 ceph-mon[146817]: pgmap v709: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:04:08.992 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:08 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/395978723' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:04:09.017 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-12T06:04:09.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:08 smithi067 ceph-mon[112979]: pgmap v709: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:04:09.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:08 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/395978723' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:04:09.318 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T06:04:10.963 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:10 smithi032 ceph-mon[146817]: pgmap v710: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:04:11.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:10 smithi067 ceph-mon[112979]: pgmap v710: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:04:11.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.308+0000 7f65f6d98700 1 -- 172.21.15.32:0/3123468892 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65f01026b0 msgr2=0x7f65f00fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:11.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.308+0000 7f65f6d98700 1 --2- 172.21.15.32:0/3123468892 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65f01026b0 0x7f65f00fbd20 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f65dc004940 tx=0x7f65dc009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:11.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.308+0000 7f65f6d98700 1 -- 172.21.15.32:0/3123468892 shutdown_connections 2024-04-12T06:04:11.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.308+0000 7f65f6d98700 1 --2- 172.21.15.32:0/3123468892 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65f01026b0 0x7f65f00fbd20 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:11.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.308+0000 7f65f6d98700 1 --2- 172.21.15.32:0/3123468892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65f0101d10 0x7f65f01020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:11.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.308+0000 7f65f6d98700 1 -- 172.21.15.32:0/3123468892 >> 172.21.15.32:0/3123468892 conn(0x7f65f00f7880 msgr2=0x7f65f00f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:11.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.308+0000 7f65f6d98700 1 -- 172.21.15.32:0/3123468892 shutdown_connections 2024-04-12T06:04:11.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.308+0000 7f65f6d98700 1 -- 172.21.15.32:0/3123468892 wait complete. 2024-04-12T06:04:11.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.309+0000 7f65f6d98700 1 Processor -- start 2024-04-12T06:04:11.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.309+0000 7f65f6d98700 1 -- start start 2024-04-12T06:04:11.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.309+0000 7f65f6d98700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65f0101d10 0x7f65f0112c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:11.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.309+0000 7f65f6d98700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65f01026b0 0x7f65f01131a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:11.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.309+0000 7f65f6d98700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f01137f0 con 0x7f65f0101d10 2024-04-12T06:04:11.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.309+0000 7f65f6d98700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f010cbc0 con 0x7f65f01026b0 2024-04-12T06:04:11.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.310+0000 7f65f4b34700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65f0101d10 0x7f65f0112c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:11.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.310+0000 7f65f4b34700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65f0101d10 0x7f65f0112c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58710/0 (socket says 172.21.15.32:58710) 2024-04-12T06:04:11.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.310+0000 7f65f4b34700 1 -- 172.21.15.32:0/1875289103 learned_addr learned my addr 172.21.15.32:0/1875289103 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:11.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.310+0000 7f65effff700 1 --2- 172.21.15.32:0/1875289103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65f01026b0 0x7f65f01131a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:11.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.310+0000 7f65f4b34700 1 -- 172.21.15.32:0/1875289103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65f01026b0 msgr2=0x7f65f01131a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:11.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.310+0000 7f65f4b34700 1 --2- 172.21.15.32:0/1875289103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65f01026b0 0x7f65f01131a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:11.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.310+0000 7f65f4b34700 1 -- 172.21.15.32:0/1875289103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65dc005040 con 0x7f65f0101d10 2024-04-12T06:04:11.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.310+0000 7f65effff700 1 --2- 172.21.15.32:0/1875289103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65f01026b0 0x7f65f01131a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T06:04:11.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.310+0000 7f65f4b34700 1 --2- 172.21.15.32:0/1875289103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65f0101d10 0x7f65f0112c60 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f65e400eb40 tx=0x7f65e400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:11.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.310+0000 7f65ed7fa700 1 -- 172.21.15.32:0/1875289103 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65e4007cc0 con 0x7f65f0101d10 2024-04-12T06:04:11.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.310+0000 7f65f6d98700 1 -- 172.21.15.32:0/1875289103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f010ce40 con 0x7f65f0101d10 2024-04-12T06:04:11.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.310+0000 7f65ed7fa700 1 -- 172.21.15.32:0/1875289103 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f65e400f040 con 0x7f65f0101d10 2024-04-12T06:04:11.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.310+0000 7f65ed7fa700 1 -- 172.21.15.32:0/1875289103 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65e4009570 con 0x7f65f0101d10 2024-04-12T06:04:11.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.310+0000 7f65f6d98700 1 -- 172.21.15.32:0/1875289103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f010d400 con 0x7f65f0101d10 2024-04-12T06:04:11.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.311+0000 7f65f6d98700 1 -- 172.21.15.32:0/1875289103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65f00fd460 con 0x7f65f0101d10 2024-04-12T06:04:11.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.312+0000 7f65ed7fa700 1 -- 172.21.15.32:0/1875289103 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f65e40096d0 con 0x7f65f0101d10 2024-04-12T06:04:11.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.312+0000 7f65ed7fa700 1 --2- 172.21.15.32:0/1875289103 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f65e006cc20 0x7f65e006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:11.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.312+0000 7f65ed7fa700 1 -- 172.21.15.32:0/1875289103 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f65e408bd90 con 0x7f65f0101d10 2024-04-12T06:04:11.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.313+0000 7f65effff700 1 --2- 172.21.15.32:0/1875289103 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f65e006cc20 0x7f65e006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:11.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.313+0000 7f65effff700 1 --2- 172.21.15.32:0/1875289103 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f65e006cc20 0x7f65e006f0d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f65f010e500 tx=0x7f65dc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:11.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.317+0000 7f65ed7fa700 1 -- 172.21.15.32:0/1875289103 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f65e4059830 con 0x7f65f0101d10 2024-04-12T06:04:11.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.608+0000 7f65f6d98700 1 -- 172.21.15.32:0/1875289103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f65f004bed0 con 0x7f65f0101d10 2024-04-12T06:04:11.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.608+0000 7f65ed7fa700 1 -- 172.21.15.32:0/1875289103 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f65e40593c0 con 0x7f65f0101d10 2024-04-12T06:04:11.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.611+0000 7f65f6d98700 1 -- 172.21.15.32:0/1875289103 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f65e006cc20 msgr2=0x7f65e006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:11.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.611+0000 7f65f6d98700 1 --2- 172.21.15.32:0/1875289103 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f65e006cc20 0x7f65e006f0d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f65f010e500 tx=0x7f65dc009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:11.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.611+0000 7f65f6d98700 1 -- 172.21.15.32:0/1875289103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65f0101d10 msgr2=0x7f65f0112c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:11.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.611+0000 7f65f6d98700 1 --2- 172.21.15.32:0/1875289103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65f0101d10 0x7f65f0112c60 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f65e400eb40 tx=0x7f65e400b040 comp rx=0 tx=0).stop 2024-04-12T06:04:11.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.611+0000 7f65f6d98700 1 -- 172.21.15.32:0/1875289103 shutdown_connections 2024-04-12T06:04:11.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.611+0000 7f65f6d98700 1 --2- 172.21.15.32:0/1875289103 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f65e006cc20 0x7f65e006f0d0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:11.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.611+0000 7f65f6d98700 1 --2- 172.21.15.32:0/1875289103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65f0101d10 0x7f65f0112c60 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:11.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.611+0000 7f65f6d98700 1 --2- 172.21.15.32:0/1875289103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65f01026b0 0x7f65f01131a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:11.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.611+0000 7f65f6d98700 1 -- 172.21.15.32:0/1875289103 >> 172.21.15.32:0/1875289103 conn(0x7f65f00f7880 msgr2=0x7f65f00f88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:11.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.611+0000 7f65f6d98700 1 -- 172.21.15.32:0/1875289103 shutdown_connections 2024-04-12T06:04:11.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:11.611+0000 7f65f6d98700 1 -- 172.21.15.32:0/1875289103 wait complete. 2024-04-12T06:04:11.617 INFO:teuthology.orchestra.run.smithi032.stdout:true 2024-04-12T06:04:12.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:11 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1875289103' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:04:12.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:11 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1875289103' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:04:12.380 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-12T06:04:12.665 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T06:04:13.173 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:12 smithi032 ceph-mon[146817]: pgmap v711: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:04:13.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:12 smithi067 ceph-mon[112979]: pgmap v711: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:04:14.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.438+0000 7fdc3bc52700 1 -- 172.21.15.32:0/3741111596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc34102710 msgr2=0x7fdc340fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:14.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.438+0000 7fdc3bc52700 1 --2- 172.21.15.32:0/3741111596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc34102710 0x7fdc340fbd80 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fdc24004940 tx=0x7fdc24009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:14.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.438+0000 7fdc3bc52700 1 -- 172.21.15.32:0/3741111596 shutdown_connections 2024-04-12T06:04:14.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.438+0000 7fdc3bc52700 1 --2- 172.21.15.32:0/3741111596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc34102710 0x7fdc340fbd80 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:14.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.438+0000 7fdc3bc52700 1 --2- 172.21.15.32:0/3741111596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc34101d70 0x7fdc34102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:14.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.438+0000 7fdc3bc52700 1 -- 172.21.15.32:0/3741111596 >> 172.21.15.32:0/3741111596 conn(0x7fdc34072bc0 msgr2=0x7fdc34072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:14.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.438+0000 7fdc3bc52700 1 -- 172.21.15.32:0/3741111596 shutdown_connections 2024-04-12T06:04:14.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.438+0000 7fdc3bc52700 1 -- 172.21.15.32:0/3741111596 wait complete. 2024-04-12T06:04:14.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.439+0000 7fdc3bc52700 1 Processor -- start 2024-04-12T06:04:14.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.439+0000 7fdc3bc52700 1 -- start start 2024-04-12T06:04:14.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.439+0000 7fdc3bc52700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc34101d70 0x7fdc340fd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:14.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.439+0000 7fdc3bc52700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc34102710 0x7fdc340fdf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:14.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.439+0000 7fdc3bc52700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc340fe4c0 con 0x7fdc34101d70 2024-04-12T06:04:14.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.439+0000 7fdc3bc52700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc340fe600 con 0x7fdc34102710 2024-04-12T06:04:14.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.439+0000 7fdc399ee700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc34101d70 0x7fdc340fd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:14.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.439+0000 7fdc399ee700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc34101d70 0x7fdc340fd9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53460/0 (socket says 172.21.15.32:53460) 2024-04-12T06:04:14.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.439+0000 7fdc399ee700 1 -- 172.21.15.32:0/3952945508 learned_addr learned my addr 172.21.15.32:0/3952945508 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:14.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.439+0000 7fdc391ed700 1 --2- 172.21.15.32:0/3952945508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc34102710 0x7fdc340fdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:14.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.439+0000 7fdc399ee700 1 -- 172.21.15.32:0/3952945508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc34102710 msgr2=0x7fdc340fdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:14.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.439+0000 7fdc399ee700 1 --2- 172.21.15.32:0/3952945508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc34102710 0x7fdc340fdf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:14.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.439+0000 7fdc399ee700 1 -- 172.21.15.32:0/3952945508 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc24005040 con 0x7fdc34101d70 2024-04-12T06:04:14.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.440+0000 7fdc399ee700 1 --2- 172.21.15.32:0/3952945508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc34101d70 0x7fdc340fd9f0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fdc3000c810 tx=0x7fdc30004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:14.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.440+0000 7fdc2a7fc700 1 -- 172.21.15.32:0/3952945508 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc30012810 con 0x7fdc34101d70 2024-04-12T06:04:14.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.440+0000 7fdc2a7fc700 1 -- 172.21.15.32:0/3952945508 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc30012e20 con 0x7fdc34101d70 2024-04-12T06:04:14.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.440+0000 7fdc3bc52700 1 -- 172.21.15.32:0/3952945508 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc34119e00 con 0x7fdc34101d70 2024-04-12T06:04:14.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.440+0000 7fdc2a7fc700 1 -- 172.21.15.32:0/3952945508 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc300102e0 con 0x7fdc34101d70 2024-04-12T06:04:14.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.440+0000 7fdc3bc52700 1 -- 172.21.15.32:0/3952945508 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc3411a280 con 0x7fdc34101d70 2024-04-12T06:04:14.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.441+0000 7fdc3bc52700 1 -- 172.21.15.32:0/3952945508 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc340fd4c0 con 0x7fdc34101d70 2024-04-12T06:04:14.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.441+0000 7fdc2a7fc700 1 -- 172.21.15.32:0/3952945508 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fdc30011380 con 0x7fdc34101d70 2024-04-12T06:04:14.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.442+0000 7fdc2a7fc700 1 --2- 172.21.15.32:0/3952945508 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fdc2006cc30 0x7fdc2006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:14.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.442+0000 7fdc2a7fc700 1 -- 172.21.15.32:0/3952945508 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fdc3008c4c0 con 0x7fdc34101d70 2024-04-12T06:04:14.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.442+0000 7fdc391ed700 1 --2- 172.21.15.32:0/3952945508 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fdc2006cc30 0x7fdc2006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:14.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.442+0000 7fdc391ed700 1 --2- 172.21.15.32:0/3952945508 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fdc2006cc30 0x7fdc2006f0e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fdc340ff330 tx=0x7fdc24009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:14.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.446+0000 7fdc2a7fc700 1 -- 172.21.15.32:0/3952945508 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fdc30056420 con 0x7fdc34101d70 2024-04-12T06:04:14.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.732+0000 7fdc3bc52700 1 -- 172.21.15.32:0/3952945508 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fdc340ff0e0 con 0x7fdc34101d70 2024-04-12T06:04:14.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.733+0000 7fdc2a7fc700 1 -- 172.21.15.32:0/3952945508 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fdc30059a40 con 0x7fdc34101d70 2024-04-12T06:04:14.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.735+0000 7fdc3bc52700 1 -- 172.21.15.32:0/3952945508 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fdc2006cc30 msgr2=0x7fdc2006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:14.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.735+0000 7fdc3bc52700 1 --2- 172.21.15.32:0/3952945508 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fdc2006cc30 0x7fdc2006f0e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fdc340ff330 tx=0x7fdc24009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:14.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.735+0000 7fdc3bc52700 1 -- 172.21.15.32:0/3952945508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc34101d70 msgr2=0x7fdc340fd9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:14.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.735+0000 7fdc3bc52700 1 --2- 172.21.15.32:0/3952945508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc34101d70 0x7fdc340fd9f0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fdc3000c810 tx=0x7fdc30004470 comp rx=0 tx=0).stop 2024-04-12T06:04:14.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.735+0000 7fdc3bc52700 1 -- 172.21.15.32:0/3952945508 shutdown_connections 2024-04-12T06:04:14.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.735+0000 7fdc3bc52700 1 --2- 172.21.15.32:0/3952945508 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fdc2006cc30 0x7fdc2006f0e0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:14.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.735+0000 7fdc3bc52700 1 --2- 172.21.15.32:0/3952945508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc34101d70 0x7fdc340fd9f0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:14.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.735+0000 7fdc3bc52700 1 --2- 172.21.15.32:0/3952945508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc34102710 0x7fdc340fdf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:14.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.735+0000 7fdc3bc52700 1 -- 172.21.15.32:0/3952945508 >> 172.21.15.32:0/3952945508 conn(0x7fdc34072bc0 msgr2=0x7fdc34065a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:14.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.735+0000 7fdc3bc52700 1 -- 172.21.15.32:0/3952945508 shutdown_connections 2024-04-12T06:04:14.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:14.735+0000 7fdc3bc52700 1 -- 172.21.15.32:0/3952945508 wait complete. 2024-04-12T06:04:14.741 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)" 2024-04-12T06:04:15.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:14 smithi032 ceph-mon[146817]: pgmap v712: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:04:15.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:14 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3952945508' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:04:15.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:14 smithi067 ceph-mon[112979]: pgmap v712: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:04:15.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:14 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3952945508' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T06:04:15.458 DEBUG:teuthology.parallel:result is None 2024-04-12T06:04:15.459 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-12T06:04:15.471 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-12T06:04:15.471 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-12T06:04:15.757 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/mon.smithi032/config 2024-04-12T06:04:16.935 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:16 smithi032 ceph-mon[146817]: pgmap v713: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:04:17.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:16 smithi067 ceph-mon[112979]: pgmap v713: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:04:17.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.550+0000 7feb9a54c700 1 -- 172.21.15.32:0/2918717560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb941037c0 msgr2=0x7feb940fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:17.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.550+0000 7feb9a54c700 1 --2- 172.21.15.32:0/2918717560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb941037c0 0x7feb940fce30 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7feb88004940 tx=0x7feb88009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:17.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.550+0000 7feb9a54c700 1 -- 172.21.15.32:0/2918717560 shutdown_connections 2024-04-12T06:04:17.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.550+0000 7feb9a54c700 1 --2- 172.21.15.32:0/2918717560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb941037c0 0x7feb940fce30 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:17.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.550+0000 7feb9a54c700 1 --2- 172.21.15.32:0/2918717560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb94102e20 0x7feb941031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:17.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.550+0000 7feb9a54c700 1 -- 172.21.15.32:0/2918717560 >> 172.21.15.32:0/2918717560 conn(0x7feb940f89d0 msgr2=0x7feb940fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:17.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.550+0000 7feb9a54c700 1 -- 172.21.15.32:0/2918717560 shutdown_connections 2024-04-12T06:04:17.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.550+0000 7feb9a54c700 1 -- 172.21.15.32:0/2918717560 wait complete. 2024-04-12T06:04:17.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.550+0000 7feb9a54c700 1 Processor -- start 2024-04-12T06:04:17.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.551+0000 7feb9a54c700 1 -- start start 2024-04-12T06:04:17.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.551+0000 7feb9a54c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb94102e20 0x7feb940702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:17.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.551+0000 7feb9a54c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb941037c0 0x7feb94070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:17.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.551+0000 7feb9a54c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb94070d90 con 0x7feb941037c0 2024-04-12T06:04:17.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.551+0000 7feb9a54c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb94070ed0 con 0x7feb94102e20 2024-04-12T06:04:17.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.551+0000 7feb937fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb941037c0 0x7feb94070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:17.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.551+0000 7feb937fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb941037c0 0x7feb94070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53490/0 (socket says 172.21.15.32:53490) 2024-04-12T06:04:17.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.551+0000 7feb937fe700 1 -- 172.21.15.32:0/3788264253 learned_addr learned my addr 172.21.15.32:0/3788264253 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:17.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.551+0000 7feb93fff700 1 --2- 172.21.15.32:0/3788264253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb94102e20 0x7feb940702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:17.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.551+0000 7feb937fe700 1 -- 172.21.15.32:0/3788264253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb94102e20 msgr2=0x7feb940702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:17.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.551+0000 7feb937fe700 1 --2- 172.21.15.32:0/3788264253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb94102e20 0x7feb940702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:17.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.551+0000 7feb937fe700 1 -- 172.21.15.32:0/3788264253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb88005040 con 0x7feb941037c0 2024-04-12T06:04:17.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.552+0000 7feb937fe700 1 --2- 172.21.15.32:0/3788264253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb941037c0 0x7feb94070800 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7feb88008d50 tx=0x7feb88009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:17.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.552+0000 7feb90ff9700 1 -- 172.21.15.32:0/3788264253 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb8800e030 con 0x7feb941037c0 2024-04-12T06:04:17.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.552+0000 7feb90ff9700 1 -- 172.21.15.32:0/3788264253 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb880093b0 con 0x7feb941037c0 2024-04-12T06:04:17.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.552+0000 7feb9a54c700 1 -- 172.21.15.32:0/3788264253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb94071120 con 0x7feb941037c0 2024-04-12T06:04:17.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.552+0000 7feb90ff9700 1 -- 172.21.15.32:0/3788264253 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb8801cbb0 con 0x7feb941037c0 2024-04-12T06:04:17.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.552+0000 7feb9a54c700 1 -- 172.21.15.32:0/3788264253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb9411a170 con 0x7feb941037c0 2024-04-12T06:04:17.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.553+0000 7feb9a54c700 1 -- 172.21.15.32:0/3788264253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb940fe570 con 0x7feb941037c0 2024-04-12T06:04:17.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.553+0000 7feb90ff9700 1 -- 172.21.15.32:0/3788264253 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7feb88013930 con 0x7feb941037c0 2024-04-12T06:04:17.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.554+0000 7feb90ff9700 1 --2- 172.21.15.32:0/3788264253 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7feb7c06cc30 0x7feb7c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:17.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.554+0000 7feb90ff9700 1 -- 172.21.15.32:0/3788264253 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7feb8808d800 con 0x7feb941037c0 2024-04-12T06:04:17.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.554+0000 7feb93fff700 1 --2- 172.21.15.32:0/3788264253 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7feb7c06cc30 0x7feb7c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:17.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.554+0000 7feb93fff700 1 --2- 172.21.15.32:0/3788264253 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7feb7c06cc30 0x7feb7c06f0e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7feb84009b80 tx=0x7feb8400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:17.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.558+0000 7feb90ff9700 1 -- 172.21.15.32:0/3788264253 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7feb880577e0 con 0x7feb941037c0 2024-04-12T06:04:17.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.808+0000 7feb9a54c700 1 -- 172.21.15.32:0/3788264253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7feb94109df0 con 0x7feb941037c0 2024-04-12T06:04:17.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.809+0000 7feb90ff9700 1 -- 172.21.15.32:0/3788264253 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 40 v40) v1 ==== 76+0+1870 (secure 0 0 0) 0x7feb8805ae00 con 0x7feb941037c0 2024-04-12T06:04:17.804 INFO:teuthology.orchestra.run.smithi032.stdout:e40 2024-04-12T06:04:17.804 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T06:04:17.804 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T06:04:17.804 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-12T06:04:17.804 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:04:17.804 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-12T06:04:17.805 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-12T06:04:17.805 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 40 2024-04-12T06:04:17.805 INFO:teuthology.orchestra.run.smithi032.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-12T06:04:17.805 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-12T05:34:22.641718+0000 2024-04-12T06:04:17.805 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-12T06:02:23.988915+0000 2024-04-12T06:04:17.805 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-12T06:04:17.805 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-12T06:04:17.805 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 60 2024-04-12T06:04:17.805 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-12T06:04:17.805 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-12T06:04:17.805 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-12T06:04:17.805 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-12T06:04:17.805 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-12T06:04:17.805 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 81 2024-04-12T06:04:17.805 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T06:04:17.806 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 2 2024-04-12T06:04:17.806 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1 2024-04-12T06:04:17.806 INFO:teuthology.orchestra.run.smithi032.stdout:up {0=34384,1=34416} 2024-04-12T06:04:17.806 INFO:teuthology.orchestra.run.smithi032.stdout:failed 2024-04-12T06:04:17.806 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-12T06:04:17.806 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-12T06:04:17.806 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3] 2024-04-12T06:04:17.806 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-12T06:04:17.806 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-12T06:04:17.806 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-12T06:04:17.806 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-12T06:04:17.806 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 1 2024-04-12T06:04:17.806 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.jehaaj{0:34384} state up:active seq 244 join_fscid=1 addr [v2:172.21.15.32:6826/1884332935,v1:172.21.15.32:6827/1884332935] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T06:04:17.806 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.zesmgy{0:44317} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.67:6824/142107794,v1:172.21.15.67:6825/142107794] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T06:04:17.807 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi032.pndtry{1:34416} state up:active seq 240 join_fscid=1 addr [v2:172.21.15.32:6828/1671696137,v1:172.21.15.32:6829/1671696137] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T06:04:17.807 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.cephfs.smithi067.ymvqih{1:44343} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/2631637348,v1:172.21.15.67:6827/2631637348] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T06:04:17.807 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:04:17.807 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:04:17.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.811+0000 7feb9a54c700 1 -- 172.21.15.32:0/3788264253 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7feb7c06cc30 msgr2=0x7feb7c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:17.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.811+0000 7feb9a54c700 1 --2- 172.21.15.32:0/3788264253 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7feb7c06cc30 0x7feb7c06f0e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7feb84009b80 tx=0x7feb8400b040 comp rx=0 tx=0).stop 2024-04-12T06:04:17.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.811+0000 7feb9a54c700 1 -- 172.21.15.32:0/3788264253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb941037c0 msgr2=0x7feb94070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:17.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.811+0000 7feb9a54c700 1 --2- 172.21.15.32:0/3788264253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb941037c0 0x7feb94070800 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7feb88008d50 tx=0x7feb88009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:17.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.811+0000 7feb9a54c700 1 -- 172.21.15.32:0/3788264253 shutdown_connections 2024-04-12T06:04:17.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.811+0000 7feb9a54c700 1 --2- 172.21.15.32:0/3788264253 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7feb7c06cc30 0x7feb7c06f0e0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:17.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.811+0000 7feb9a54c700 1 --2- 172.21.15.32:0/3788264253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb94102e20 0x7feb940702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:17.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.811+0000 7feb9a54c700 1 --2- 172.21.15.32:0/3788264253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb941037c0 0x7feb94070800 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:17.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.811+0000 7feb9a54c700 1 -- 172.21.15.32:0/3788264253 >> 172.21.15.32:0/3788264253 conn(0x7feb940f89d0 msgr2=0x7feb940f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:17.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.811+0000 7feb9a54c700 1 -- 172.21.15.32:0/3788264253 shutdown_connections 2024-04-12T06:04:17.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:17.811+0000 7feb9a54c700 1 -- 172.21.15.32:0/3788264253 wait complete. 2024-04-12T06:04:17.808 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 40 2024-04-12T06:04:18.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:17 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3788264253' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T06:04:18.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:17 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:04:18.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:17 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3788264253' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T06:04:18.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:17 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:04:18.529 INFO:teuthology.run_tasks:Running task fs.post_upgrade_checks... 2024-04-12T06:04:18.542 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-12T06:04:18.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.672+0000 7f338e77a700 1 -- 172.21.15.32:0/2899858217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3388064e40 msgr2=0x7f33880652b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:18.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.672+0000 7f338e77a700 1 --2- 172.21.15.32:0/2899858217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3388064e40 0x7f33880652b0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f337c004940 tx=0x7f337c009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:18.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.672+0000 7f338e77a700 1 -- 172.21.15.32:0/2899858217 shutdown_connections 2024-04-12T06:04:18.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.672+0000 7f338e77a700 1 --2- 172.21.15.32:0/2899858217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3388064e40 0x7f33880652b0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:18.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.672+0000 7f338e77a700 1 --2- 172.21.15.32:0/2899858217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3388102d80 0x7f3388103150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:18.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.672+0000 7f338e77a700 1 -- 172.21.15.32:0/2899858217 >> 172.21.15.32:0/2899858217 conn(0x7f3388071c80 msgr2=0x7f3388072080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:18.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.672+0000 7f338e77a700 1 -- 172.21.15.32:0/2899858217 shutdown_connections 2024-04-12T06:04:18.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.673+0000 7f338e77a700 1 -- 172.21.15.32:0/2899858217 wait complete. 2024-04-12T06:04:18.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.673+0000 7f338e77a700 1 Processor -- start 2024-04-12T06:04:18.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.674+0000 7f338e77a700 1 -- start start 2024-04-12T06:04:18.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.674+0000 7f338e77a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3388064e40 0x7f33881137f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:18.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.674+0000 7f338e77a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3388102d80 0x7f3388113d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:18.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.674+0000 7f338e77a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3388114380 con 0x7f3388064e40 2024-04-12T06:04:18.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.674+0000 7f338e77a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338810d980 con 0x7f3388102d80 2024-04-12T06:04:18.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.674+0000 7f3387fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3388064e40 0x7f33881137f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:18.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.674+0000 7f3387fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3388064e40 0x7f33881137f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53498/0 (socket says 172.21.15.32:53498) 2024-04-12T06:04:18.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.674+0000 7f3387fff700 1 -- 172.21.15.32:0/711583758 learned_addr learned my addr 172.21.15.32:0/711583758 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:18.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.674+0000 7f33877fe700 1 --2- 172.21.15.32:0/711583758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3388102d80 0x7f3388113d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:18.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.674+0000 7f3387fff700 1 -- 172.21.15.32:0/711583758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3388102d80 msgr2=0x7f3388113d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:18.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.674+0000 7f3387fff700 1 --2- 172.21.15.32:0/711583758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3388102d80 0x7f3388113d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:18.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.674+0000 7f3387fff700 1 -- 172.21.15.32:0/711583758 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f337c005040 con 0x7f3388064e40 2024-04-12T06:04:18.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.674+0000 7f3387fff700 1 --2- 172.21.15.32:0/711583758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3388064e40 0x7f33881137f0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f3378002a50 tx=0x7f33780046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:18.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.675+0000 7f3384ff9700 1 -- 172.21.15.32:0/711583758 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3378007cc0 con 0x7f3388064e40 2024-04-12T06:04:18.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.675+0000 7f3384ff9700 1 -- 172.21.15.32:0/711583758 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f337800f040 con 0x7f3388064e40 2024-04-12T06:04:18.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.675+0000 7f3384ff9700 1 -- 172.21.15.32:0/711583758 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f337800d9c0 con 0x7f3388064e40 2024-04-12T06:04:18.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.675+0000 7f338e77a700 1 -- 172.21.15.32:0/711583758 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f338810dc60 con 0x7f3388064e40 2024-04-12T06:04:18.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.675+0000 7f338e77a700 1 -- 172.21.15.32:0/711583758 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f338810e1a0 con 0x7f3388064e40 2024-04-12T06:04:18.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.676+0000 7f338e77a700 1 -- 172.21.15.32:0/711583758 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33880fe4e0 con 0x7f3388064e40 2024-04-12T06:04:18.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.676+0000 7f3384ff9700 1 -- 172.21.15.32:0/711583758 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f337800db20 con 0x7f3388064e40 2024-04-12T06:04:18.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.677+0000 7f3384ff9700 1 --2- 172.21.15.32:0/711583758 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f337006cc20 0x7f337006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:18.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.677+0000 7f33877fe700 1 --2- 172.21.15.32:0/711583758 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f337006cc20 0x7f337006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:18.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.677+0000 7f3384ff9700 1 -- 172.21.15.32:0/711583758 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f337808bdd0 con 0x7f3388064e40 2024-04-12T06:04:18.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.677+0000 7f33877fe700 1 --2- 172.21.15.32:0/711583758 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f337006cc20 0x7f337006f0d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f338810eb40 tx=0x7f337c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:18.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.681+0000 7f3384ff9700 1 -- 172.21.15.32:0/711583758 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f33780597c0 con 0x7f3388064e40 2024-04-12T06:04:18.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.888+0000 7f338e77a700 1 -- 172.21.15.32:0/711583758 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3388109d30 con 0x7f3388064e40 2024-04-12T06:04:18.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.889+0000 7f3384ff9700 1 -- 172.21.15.32:0/711583758 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16631 (secure 0 0 0) 0x7f3378059350 con 0x7f3388064e40 2024-04-12T06:04:18.885 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:04:18.885 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":85,"fsid":"37b58ce2-f88d-11ee-bc8c-c7b262605968","created":"2024-04-12T05:27:07.779407+0000","modified":"2024-04-12T05:47:06.905923+0000","last_up_change":"2024-04-12T05:46:08.474847+0000","last_in_change":"2024-04-12T05:32:39.814568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T05:31:39.568876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-12T05:34:21.574210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-12T05:34:22.056623+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c1f69d2a-5e37-47b8-b68f-0568af894b91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":49,"up_from":53,"up_thru":78,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1783741888},{"type":"v1","addr":"172.21.15.32:6803","nonce":1783741888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1783741888},{"type":"v1","addr":"172.21.15.32:6805","nonce":1783741888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1783741888},{"type":"v1","addr":"172.21.15.32:6809","nonce":1783741888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1783741888},{"type":"v1","addr":"172.21.15.32:6807","nonce":1783741888}]},"public_addr":"172.21.15.32:6803/1783741888","cluster_addr":"172.21.15.32:6805/1783741888","heartbeat_back_addr":"172.21.15.32:6809/1783741888","heartbeat_front_addr":"172.21.15.32:6807/1783741888","state":["exists","up"]},{"osd":1,"uuid":"86b93557-6e6f-4b84-a5f5-5bce4aa65a32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":54,"up_from":58,"up_thru":78,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1060060861},{"type":"v1","addr":"172.21.15.32:6811","nonce":1060060861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1060060861},{"type":"v1","addr":"172.21.15.32:6813","nonce":1060060861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1060060861},{"type":"v1","addr":"172.21.15.32:6817","nonce":1060060861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1060060861},{"type":"v1","addr":"172.21.15.32:6815","nonce":1060060861}]},"public_addr":"172.21.15.32:6811/1060060861","cluster_addr":"172.21.15.32:6813/1060060861","heartbeat_back_addr":"172.21.15.32:6817/1060060861","heartbeat_front_addr":"172.21.15.32:6815/1060060861","state":["exists","up"]},{"osd":2,"uuid":"d1556613-e0de-4b0e-bb7d-e90361ec9007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":59,"up_from":63,"up_thru":78,"down_at":60,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1610966654},{"type":"v1","addr":"172.21.15.32:6819","nonce":1610966654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1610966654},{"type":"v1","addr":"172.21.15.32:6821","nonce":1610966654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1610966654},{"type":"v1","addr":"172.21.15.32:6825","nonce":1610966654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1610966654},{"type":"v1","addr":"172.21.15.32:6823","nonce":1610966654}]},"public_addr":"172.21.15.32:6819/1610966654","cluster_addr":"172.21.15.32:6821/1610966654","heartbeat_back_addr":"172.21.15.32:6825/1610966654","heartbeat_front_addr":"172.21.15.32:6823/1610966654","state":["exists","up"]},{"osd":3,"uuid":"213dc62e-a894-4bc7-a2dd-ca98d1313d6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":64,"up_from":68,"up_thru":78,"down_at":65,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2625997716},{"type":"v1","addr":"172.21.15.67:6801","nonce":2625997716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2625997716},{"type":"v1","addr":"172.21.15.67:6803","nonce":2625997716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":2625997716},{"type":"v1","addr":"172.21.15.67:6807","nonce":2625997716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":2625997716},{"type":"v1","addr":"172.21.15.67:6805","nonce":2625997716}]},"public_addr":"172.21.15.67:6801/2625997716","cluster_addr":"172.21.15.67:6803/2625997716","heartbeat_back_addr":"172.21.15.67:6807/2625997716","heartbeat_front_addr":"172.21.15.67:6805/2625997716","state":["exists","up"]},{"osd":4,"uuid":"8a0ff860-fff7-434a-9e78-f2020bc84966","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":69,"up_from":73,"up_thru":78,"down_at":70,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":452825880},{"type":"v1","addr":"172.21.15.67:6809","nonce":452825880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":452825880},{"type":"v1","addr":"172.21.15.67:6811","nonce":452825880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":452825880},{"type":"v1","addr":"172.21.15.67:6815","nonce":452825880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":452825880},{"type":"v1","addr":"172.21.15.67:6813","nonce":452825880}]},"public_addr":"172.21.15.67:6809/452825880","cluster_addr":"172.21.15.67:6811/452825880","heartbeat_back_addr":"172.21.15.67:6815/452825880","heartbeat_front_addr":"172.21.15.67:6813/452825880","state":["exists","up"]},{"osd":5,"uuid":"3486e5c9-9f44-4f32-a186-fc6198f09549","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":74,"up_from":78,"up_thru":78,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2273350169},{"type":"v1","addr":"172.21.15.67:6817","nonce":2273350169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2273350169},{"type":"v1","addr":"172.21.15.67:6819","nonce":2273350169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2273350169},{"type":"v1","addr":"172.21.15.67:6823","nonce":2273350169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2273350169},{"type":"v1","addr":"172.21.15.67:6821","nonce":2273350169}]},"public_addr":"172.21.15.67:6817/2273350169","cluster_addr":"172.21.15.67:6819/2273350169","heartbeat_back_addr":"172.21.15.67:6823/2273350169","heartbeat_front_addr":"172.21.15.67:6821/2273350169","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:30:38.748993+0000","dead_epoch":49},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:31:08.677208+0000","dead_epoch":54},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:31:36.979481+0000","dead_epoch":59},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:32:02.668398+0000","dead_epoch":64},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:32:30.143788+0000","dead_epoch":69},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:32:57.613087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6829/387754149":"2024-04-13T05:46:10.668153+0000","172.21.15.32:6828/387754149":"2024-04-13T05:46:10.668153+0000","172.21.15.67:6825/1291914599":"2024-04-13T05:46:10.668178+0000","172.21.15.67:6824/1291914599":"2024-04-13T05:46:10.668178+0000","172.21.15.32:0/138657961":"2024-04-13T05:27:40.952118+0000","172.21.15.67:0/1683579802":"2024-04-13T05:38:47.052969+0000","172.21.15.32:0/368340628":"2024-04-13T05:37:39.471472+0000","172.21.15.32:6801/3971961003":"2024-04-13T05:29:25.054120+0000","172.21.15.32:0/652449107":"2024-04-13T05:41:47.767117+0000","172.21.15.32:0/4112976740":"2024-04-13T05:29:25.054120+0000","172.21.15.32:0/2247439109":"2024-04-13T05:37:39.471472+0000","172.21.15.32:0/1643708423":"2024-04-13T05:27:40.952118+0000","172.21.15.32:6800/2564765508":"2024-04-13T05:41:47.767117+0000","172.21.15.67:6828/3925433122":"2024-04-13T05:38:47.052969+0000","172.21.15.32:6800/1962278696":"2024-04-13T05:28:14.470413+0000","172.21.15.32:0/2162993076":"2024-04-13T05:28:14.470413+0000","172.21.15.32:6827/2962187015":"2024-04-13T05:34:37.113413+0000","172.21.15.32:6800/3971961003":"2024-04-13T05:29:25.054120+0000","172.21.15.32:0/2105642006":"2024-04-13T05:37:39.471472+0000","172.21.15.67:0/31773612":"2024-04-13T05:38:47.052969+0000","172.21.15.67:6829/3925433122":"2024-04-13T05:38:47.052969+0000","172.21.15.67:0/2846171310":"2024-04-13T05:38:47.052969+0000","172.21.15.32:0/3370471392":"2024-04-13T05:41:47.767117+0000","172.21.15.32:0/3082743475":"2024-04-13T05:28:14.470413+0000","172.21.15.32:6801/656201914":"2024-04-13T05:27:40.952118+0000","172.21.15.32:6801/1962278696":"2024-04-13T05:28:14.470413+0000","172.21.15.32:0/260214199":"2024-04-13T05:37:39.471472+0000","172.21.15.32:6800/656201914":"2024-04-13T05:27:40.952118+0000","172.21.15.67:0/3462505447":"2024-04-13T05:38:47.052969+0000","172.21.15.32:0/674029871":"2024-04-13T05:29:25.054120+0000","172.21.15.67:0/125735613":"2024-04-13T05:38:47.052969+0000","172.21.15.67:0/3697060296":"2024-04-13T05:38:47.052969+0000","172.21.15.32:0/4031244002":"2024-04-13T05:27:40.952118+0000","172.21.15.32:6801/2564765508":"2024-04-13T05:41:47.767117+0000","172.21.15.32:0/795906028":"2024-04-13T05:29:25.054120+0000","172.21.15.32:6826/2962187015":"2024-04-13T05:34:37.113413+0000","172.21.15.32:6801/595990262":"2024-04-13T05:37:39.471472+0000","172.21.15.32:0/146872823":"2024-04-13T05:28:14.470413+0000","172.21.15.32:0/138775819":"2024-04-13T05:41:47.767117+0000","172.21.15.32:6800/595990262":"2024-04-13T05:37:39.471472+0000","172.21.15.32:0/2743601944":"2024-04-13T05:41:47.767117+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T06:04:18.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.892+0000 7f338e77a700 1 -- 172.21.15.32:0/711583758 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f337006cc20 msgr2=0x7f337006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:18.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.892+0000 7f338e77a700 1 --2- 172.21.15.32:0/711583758 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f337006cc20 0x7f337006f0d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f338810eb40 tx=0x7f337c009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:18.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.892+0000 7f338e77a700 1 -- 172.21.15.32:0/711583758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3388064e40 msgr2=0x7f33881137f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:18.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.892+0000 7f338e77a700 1 --2- 172.21.15.32:0/711583758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3388064e40 0x7f33881137f0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f3378002a50 tx=0x7f33780046b0 comp rx=0 tx=0).stop 2024-04-12T06:04:18.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.892+0000 7f338e77a700 1 -- 172.21.15.32:0/711583758 shutdown_connections 2024-04-12T06:04:18.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.892+0000 7f338e77a700 1 --2- 172.21.15.32:0/711583758 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f337006cc20 0x7f337006f0d0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:18.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.892+0000 7f338e77a700 1 --2- 172.21.15.32:0/711583758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3388064e40 0x7f33881137f0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:18.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.892+0000 7f338e77a700 1 --2- 172.21.15.32:0/711583758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3388102d80 0x7f3388113d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:18.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.892+0000 7f338e77a700 1 -- 172.21.15.32:0/711583758 >> 172.21.15.32:0/711583758 conn(0x7f3388071c80 msgr2=0x7f33880fab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:18.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.892+0000 7f338e77a700 1 -- 172.21.15.32:0/711583758 shutdown_connections 2024-04-12T06:04:18.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:18.892+0000 7f338e77a700 1 -- 172.21.15.32:0/711583758 wait complete. 2024-04-12T06:04:18.900 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-12T05:31:39.568876+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-12T05:34:21.574210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-12T05:34:22.056623+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-12T06:04:18.900 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-12T06:04:19.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.031+0000 7fd2674fe700 1 -- 172.21.15.32:0/4036512707 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd260102580 msgr2=0x7fd2600fbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:19.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.031+0000 7fd2674fe700 1 --2- 172.21.15.32:0/4036512707 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd260102580 0x7fd2600fbbf0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fd25c004940 tx=0x7fd25c009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:19.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.031+0000 7fd2674fe700 1 -- 172.21.15.32:0/4036512707 shutdown_connections 2024-04-12T06:04:19.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.031+0000 7fd2674fe700 1 --2- 172.21.15.32:0/4036512707 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd260102580 0x7fd2600fbbf0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:19.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.031+0000 7fd2674fe700 1 --2- 172.21.15.32:0/4036512707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd260101be0 0x7fd260101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:19.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.031+0000 7fd2674fe700 1 -- 172.21.15.32:0/4036512707 >> 172.21.15.32:0/4036512707 conn(0x7fd260072ad0 msgr2=0x7fd260072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.031+0000 7fd2674fe700 1 -- 172.21.15.32:0/4036512707 shutdown_connections 2024-04-12T06:04:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.031+0000 7fd2674fe700 1 -- 172.21.15.32:0/4036512707 wait complete. 2024-04-12T06:04:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.031+0000 7fd2674fe700 1 Processor -- start 2024-04-12T06:04:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.032+0000 7fd2674fe700 1 -- start start 2024-04-12T06:04:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.032+0000 7fd2674fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd260101be0 0x7fd260102a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.032+0000 7fd2674fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd260102580 0x7fd260102f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.032+0000 7fd2674fe700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd260104940 con 0x7fd260101be0 2024-04-12T06:04:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.032+0000 7fd2674fe700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2601034c0 con 0x7fd260102580 2024-04-12T06:04:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.032+0000 7fd26529a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd260101be0 0x7fd260102a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.032+0000 7fd26529a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd260101be0 0x7fd260102a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53506/0 (socket says 172.21.15.32:53506) 2024-04-12T06:04:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.032+0000 7fd26529a700 1 -- 172.21.15.32:0/2588624498 learned_addr learned my addr 172.21.15.32:0/2588624498 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.032+0000 7fd264a99700 1 --2- 172.21.15.32:0/2588624498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd260102580 0x7fd260102f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.032+0000 7fd26529a700 1 -- 172.21.15.32:0/2588624498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd260102580 msgr2=0x7fd260102f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.032+0000 7fd26529a700 1 --2- 172.21.15.32:0/2588624498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd260102580 0x7fd260102f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:19.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.032+0000 7fd26529a700 1 -- 172.21.15.32:0/2588624498 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd25c005040 con 0x7fd260101be0 2024-04-12T06:04:19.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.032+0000 7fd26529a700 1 --2- 172.21.15.32:0/2588624498 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd260101be0 0x7fd260102a40 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fd250002a50 tx=0x7fd2500046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:19.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.032+0000 7fd255ffb700 1 -- 172.21.15.32:0/2588624498 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd250007cc0 con 0x7fd260101be0 2024-04-12T06:04:19.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.032+0000 7fd2674fe700 1 -- 172.21.15.32:0/2588624498 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2601037a0 con 0x7fd260101be0 2024-04-12T06:04:19.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.032+0000 7fd255ffb700 1 -- 172.21.15.32:0/2588624498 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd25000f040 con 0x7fd260101be0 2024-04-12T06:04:19.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.032+0000 7fd255ffb700 1 -- 172.21.15.32:0/2588624498 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd250009540 con 0x7fd260101be0 2024-04-12T06:04:19.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.033+0000 7fd2674fe700 1 -- 172.21.15.32:0/2588624498 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd260119f70 con 0x7fd260101be0 2024-04-12T06:04:19.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.034+0000 7fd2674fe700 1 -- 172.21.15.32:0/2588624498 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2600fd330 con 0x7fd260101be0 2024-04-12T06:04:19.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.034+0000 7fd255ffb700 1 -- 172.21.15.32:0/2588624498 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd2500096a0 con 0x7fd260101be0 2024-04-12T06:04:19.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.034+0000 7fd255ffb700 1 --2- 172.21.15.32:0/2588624498 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd24c06cc20 0x7fd24c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:19.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.035+0000 7fd255ffb700 1 -- 172.21.15.32:0/2588624498 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fd25008bf60 con 0x7fd260101be0 2024-04-12T06:04:19.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.035+0000 7fd264a99700 1 --2- 172.21.15.32:0/2588624498 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd24c06cc20 0x7fd24c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:19.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.035+0000 7fd264a99700 1 --2- 172.21.15.32:0/2588624498 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd24c06cc20 0x7fd24c06f0d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd25c00b3b0 tx=0x7fd25c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:19.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.039+0000 7fd255ffb700 1 -- 172.21.15.32:0/2588624498 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd250055e20 con 0x7fd260101be0 2024-04-12T06:04:19.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.245+0000 7fd2674fe700 1 -- 172.21.15.32:0/2588624498 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd260103f00 con 0x7fd260101be0 2024-04-12T06:04:19.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.245+0000 7fd255ffb700 1 -- 172.21.15.32:0/2588624498 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd250059440 con 0x7fd260101be0 2024-04-12T06:04:19.241 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-12T06:04:19.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.247+0000 7fd2674fe700 1 -- 172.21.15.32:0/2588624498 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd24c06cc20 msgr2=0x7fd24c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:19.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.247+0000 7fd2674fe700 1 --2- 172.21.15.32:0/2588624498 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd24c06cc20 0x7fd24c06f0d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd25c00b3b0 tx=0x7fd25c009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:19.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.247+0000 7fd2674fe700 1 -- 172.21.15.32:0/2588624498 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd260101be0 msgr2=0x7fd260102a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:19.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.247+0000 7fd2674fe700 1 --2- 172.21.15.32:0/2588624498 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd260101be0 0x7fd260102a40 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fd250002a50 tx=0x7fd2500046b0 comp rx=0 tx=0).stop 2024-04-12T06:04:19.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.247+0000 7fd2674fe700 1 -- 172.21.15.32:0/2588624498 shutdown_connections 2024-04-12T06:04:19.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.247+0000 7fd2674fe700 1 --2- 172.21.15.32:0/2588624498 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd24c06cc20 0x7fd24c06f0d0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:19.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.247+0000 7fd2674fe700 1 --2- 172.21.15.32:0/2588624498 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd260101be0 0x7fd260102a40 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:19.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.247+0000 7fd2674fe700 1 --2- 172.21.15.32:0/2588624498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd260102580 0x7fd260102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:19.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.247+0000 7fd2674fe700 1 -- 172.21.15.32:0/2588624498 >> 172.21.15.32:0/2588624498 conn(0x7fd260072ad0 msgr2=0x7fd260064dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:19.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.248+0000 7fd2674fe700 1 -- 172.21.15.32:0/2588624498 shutdown_connections 2024-04-12T06:04:19.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.248+0000 7fd2674fe700 1 -- 172.21.15.32:0/2588624498 wait complete. 2024-04-12T06:04:19.253 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:18 smithi032 ceph-mon[146817]: pgmap v714: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:04:19.254 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:18 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/711583758' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T06:04:19.255 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-12T06:04:19.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:18 smithi067 ceph-mon[112979]: pgmap v714: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:04:19.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:18 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/711583758' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T06:04:19.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.386+0000 7fd81a18c700 1 -- 172.21.15.32:0/3112759969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd814102580 msgr2=0x7fd8140fbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:19.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.386+0000 7fd81a18c700 1 --2- 172.21.15.32:0/3112759969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd814102580 0x7fd8140fbbf0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd804004940 tx=0x7fd804009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:19.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.386+0000 7fd81a18c700 1 -- 172.21.15.32:0/3112759969 shutdown_connections 2024-04-12T06:04:19.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.386+0000 7fd81a18c700 1 --2- 172.21.15.32:0/3112759969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd814102580 0x7fd8140fbbf0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:19.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.386+0000 7fd81a18c700 1 --2- 172.21.15.32:0/3112759969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd814101be0 0x7fd814101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:19.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.386+0000 7fd81a18c700 1 -- 172.21.15.32:0/3112759969 >> 172.21.15.32:0/3112759969 conn(0x7fd814072ad0 msgr2=0x7fd814072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:19.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.386+0000 7fd81a18c700 1 -- 172.21.15.32:0/3112759969 shutdown_connections 2024-04-12T06:04:19.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.386+0000 7fd81a18c700 1 -- 172.21.15.32:0/3112759969 wait complete. 2024-04-12T06:04:19.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.386+0000 7fd81a18c700 1 Processor -- start 2024-04-12T06:04:19.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.386+0000 7fd81a18c700 1 -- start start 2024-04-12T06:04:19.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.387+0000 7fd81a18c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd814101be0 0x7fd8141126a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:19.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.387+0000 7fd81a18c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd814102580 0x7fd814112be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:19.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.387+0000 7fd81a18c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd814113230 con 0x7fd814102580 2024-04-12T06:04:19.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.387+0000 7fd81a18c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81410c7f0 con 0x7fd814101be0 2024-04-12T06:04:19.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.387+0000 7fd812ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd814102580 0x7fd814112be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:19.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.387+0000 7fd812ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd814102580 0x7fd814112be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53526/0 (socket says 172.21.15.32:53526) 2024-04-12T06:04:19.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.387+0000 7fd812ffd700 1 -- 172.21.15.32:0/3611957648 learned_addr learned my addr 172.21.15.32:0/3611957648 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:19.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.387+0000 7fd8137fe700 1 --2- 172.21.15.32:0/3611957648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd814101be0 0x7fd8141126a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:19.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.387+0000 7fd812ffd700 1 -- 172.21.15.32:0/3611957648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd814101be0 msgr2=0x7fd8141126a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:19.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.387+0000 7fd812ffd700 1 --2- 172.21.15.32:0/3611957648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd814101be0 0x7fd8141126a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:19.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.387+0000 7fd812ffd700 1 -- 172.21.15.32:0/3611957648 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd804005040 con 0x7fd814102580 2024-04-12T06:04:19.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.387+0000 7fd812ffd700 1 --2- 172.21.15.32:0/3611957648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd814102580 0x7fd814112be0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fd80400b3b0 tx=0x7fd804009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:19.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.387+0000 7fd7fbfff700 1 -- 172.21.15.32:0/3611957648 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd80400e040 con 0x7fd814102580 2024-04-12T06:04:19.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.387+0000 7fd7fbfff700 1 -- 172.21.15.32:0/3611957648 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd80400bc90 con 0x7fd814102580 2024-04-12T06:04:19.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.387+0000 7fd81a18c700 1 -- 172.21.15.32:0/3611957648 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd81410ca70 con 0x7fd814102580 2024-04-12T06:04:19.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.387+0000 7fd7fbfff700 1 -- 172.21.15.32:0/3611957648 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd804013660 con 0x7fd814102580 2024-04-12T06:04:19.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.388+0000 7fd81a18c700 1 -- 172.21.15.32:0/3611957648 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd81410cf50 con 0x7fd814102580 2024-04-12T06:04:19.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.388+0000 7fd81a18c700 1 -- 172.21.15.32:0/3611957648 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd814112330 con 0x7fd814102580 2024-04-12T06:04:19.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.389+0000 7fd7fbfff700 1 -- 172.21.15.32:0/3611957648 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd804013920 con 0x7fd814102580 2024-04-12T06:04:19.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.389+0000 7fd7fbfff700 1 --2- 172.21.15.32:0/3611957648 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd7fc06cc20 0x7fd7fc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:19.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.389+0000 7fd7fbfff700 1 -- 172.21.15.32:0/3611957648 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fd80408ff60 con 0x7fd814102580 2024-04-12T06:04:19.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.389+0000 7fd8137fe700 1 --2- 172.21.15.32:0/3611957648 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd7fc06cc20 0x7fd7fc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:19.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.390+0000 7fd8137fe700 1 --2- 172.21.15.32:0/3611957648 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd7fc06cc20 0x7fd7fc06f0d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd808004f40 tx=0x7fd80800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:19.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.394+0000 7fd7fbfff700 1 -- 172.21.15.32:0/3611957648 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd8040c3ab0 con 0x7fd814102580 2024-04-12T06:04:19.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.595+0000 7fd81a18c700 1 -- 172.21.15.32:0/3611957648 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fd8140fd330 con 0x7fd814102580 2024-04-12T06:04:19.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.595+0000 7fd7fbfff700 1 -- 172.21.15.32:0/3611957648 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v85) v1 ==== 107+0+11 (secure 0 0 0) 0x7fd80405d950 con 0x7fd814102580 2024-04-12T06:04:19.591 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-12T06:04:19.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.597+0000 7fd81a18c700 1 -- 172.21.15.32:0/3611957648 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd7fc06cc20 msgr2=0x7fd7fc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:19.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.597+0000 7fd81a18c700 1 --2- 172.21.15.32:0/3611957648 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd7fc06cc20 0x7fd7fc06f0d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd808004f40 tx=0x7fd80800b040 comp rx=0 tx=0).stop 2024-04-12T06:04:19.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.597+0000 7fd81a18c700 1 -- 172.21.15.32:0/3611957648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd814102580 msgr2=0x7fd814112be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:19.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.597+0000 7fd81a18c700 1 --2- 172.21.15.32:0/3611957648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd814102580 0x7fd814112be0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fd80400b3b0 tx=0x7fd804009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:19.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.597+0000 7fd81a18c700 1 -- 172.21.15.32:0/3611957648 shutdown_connections 2024-04-12T06:04:19.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.597+0000 7fd81a18c700 1 --2- 172.21.15.32:0/3611957648 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fd7fc06cc20 0x7fd7fc06f0d0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:19.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.597+0000 7fd81a18c700 1 --2- 172.21.15.32:0/3611957648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd814101be0 0x7fd8141126a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:19.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.598+0000 7fd81a18c700 1 --2- 172.21.15.32:0/3611957648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd814102580 0x7fd814112be0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:19.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.598+0000 7fd81a18c700 1 -- 172.21.15.32:0/3611957648 >> 172.21.15.32:0/3611957648 conn(0x7fd814072ad0 msgr2=0x7fd814104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:19.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.598+0000 7fd81a18c700 1 -- 172.21.15.32:0/3611957648 shutdown_connections 2024-04-12T06:04:19.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.598+0000 7fd81a18c700 1 -- 172.21.15.32:0/3611957648 wait complete. 2024-04-12T06:04:19.605 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-12T06:04:19.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.735+0000 7f0069fea700 1 -- 172.21.15.32:0/2870870126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0064102580 msgr2=0x7f00640fbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:19.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.735+0000 7f0069fea700 1 --2- 172.21.15.32:0/2870870126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0064102580 0x7f00640fbbf0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f004c004940 tx=0x7f004c009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:19.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.735+0000 7f0069fea700 1 -- 172.21.15.32:0/2870870126 shutdown_connections 2024-04-12T06:04:19.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.735+0000 7f0069fea700 1 --2- 172.21.15.32:0/2870870126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0064102580 0x7f00640fbbf0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:19.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.735+0000 7f0069fea700 1 --2- 172.21.15.32:0/2870870126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0064101be0 0x7f0064101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:19.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.735+0000 7f0069fea700 1 -- 172.21.15.32:0/2870870126 >> 172.21.15.32:0/2870870126 conn(0x7f0064072ad0 msgr2=0x7f0064072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:19.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.735+0000 7f0069fea700 1 -- 172.21.15.32:0/2870870126 shutdown_connections 2024-04-12T06:04:19.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.735+0000 7f0069fea700 1 -- 172.21.15.32:0/2870870126 wait complete. 2024-04-12T06:04:19.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.735+0000 7f0069fea700 1 Processor -- start 2024-04-12T06:04:19.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.736+0000 7f0069fea700 1 -- start start 2024-04-12T06:04:19.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.736+0000 7f0069fea700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0064101be0 0x7f006410e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:19.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.736+0000 7f0069fea700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0064102580 0x7f006410ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:19.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.736+0000 7f0069fea700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006410f370 con 0x7f0064102580 2024-04-12T06:04:19.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.736+0000 7f0069fea700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006410f4e0 con 0x7f0064101be0 2024-04-12T06:04:19.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.736+0000 7f0062ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0064102580 0x7f006410ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:19.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.736+0000 7f0062ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0064102580 0x7f006410ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53540/0 (socket says 172.21.15.32:53540) 2024-04-12T06:04:19.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.736+0000 7f0062ffd700 1 -- 172.21.15.32:0/959153134 learned_addr learned my addr 172.21.15.32:0/959153134 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:19.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.736+0000 7f00637fe700 1 --2- 172.21.15.32:0/959153134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0064101be0 0x7f006410e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:19.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.736+0000 7f0062ffd700 1 -- 172.21.15.32:0/959153134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0064101be0 msgr2=0x7f006410e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:19.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.736+0000 7f0062ffd700 1 --2- 172.21.15.32:0/959153134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0064101be0 0x7f006410e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:19.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.736+0000 7f0062ffd700 1 -- 172.21.15.32:0/959153134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f004c005040 con 0x7f0064102580 2024-04-12T06:04:19.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.737+0000 7f0062ffd700 1 --2- 172.21.15.32:0/959153134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0064102580 0x7f006410ee30 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f004c009380 tx=0x7f004c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:19.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.737+0000 7f005bfff700 1 -- 172.21.15.32:0/959153134 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f004c00e040 con 0x7f0064102580 2024-04-12T06:04:19.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.737+0000 7f005bfff700 1 -- 172.21.15.32:0/959153134 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f004c013420 con 0x7f0064102580 2024-04-12T06:04:19.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.737+0000 7f0069fea700 1 -- 172.21.15.32:0/959153134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f006410f760 con 0x7f0064102580 2024-04-12T06:04:19.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.737+0000 7f005bfff700 1 -- 172.21.15.32:0/959153134 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f004c01c630 con 0x7f0064102580 2024-04-12T06:04:19.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.737+0000 7f0069fea700 1 -- 172.21.15.32:0/959153134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0064117c80 con 0x7f0064102580 2024-04-12T06:04:19.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.738+0000 7f0069fea700 1 -- 172.21.15.32:0/959153134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00640fd330 con 0x7f0064102580 2024-04-12T06:04:19.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.738+0000 7f005bfff700 1 -- 172.21.15.32:0/959153134 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f004c0139b0 con 0x7f0064102580 2024-04-12T06:04:19.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.739+0000 7f005bfff700 1 --2- 172.21.15.32:0/959153134 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f005006cc20 0x7f005006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:19.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.739+0000 7f005bfff700 1 -- 172.21.15.32:0/959153134 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f004c08d140 con 0x7f0064102580 2024-04-12T06:04:19.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.739+0000 7f00637fe700 1 --2- 172.21.15.32:0/959153134 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f005006cc20 0x7f005006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:19.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.739+0000 7f00637fe700 1 --2- 172.21.15.32:0/959153134 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f005006cc20 0x7f005006f0d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0054004f40 tx=0x7f005400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:19.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.744+0000 7f005bfff700 1 -- 172.21.15.32:0/959153134 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f004c057080 con 0x7f0064102580 2024-04-12T06:04:19.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.947+0000 7f0069fea700 1 -- 172.21.15.32:0/959153134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f006410f760 con 0x7f0064102580 2024-04-12T06:04:19.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.947+0000 7f005bfff700 1 -- 172.21.15.32:0/959153134 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v85) v1 ==== 107+0+11 (secure 0 0 0) 0x7f006410f760 con 0x7f0064102580 2024-04-12T06:04:19.942 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-12T06:04:19.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.949+0000 7f0069fea700 1 -- 172.21.15.32:0/959153134 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f005006cc20 msgr2=0x7f005006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:19.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.949+0000 7f0069fea700 1 --2- 172.21.15.32:0/959153134 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f005006cc20 0x7f005006f0d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0054004f40 tx=0x7f005400b040 comp rx=0 tx=0).stop 2024-04-12T06:04:19.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.949+0000 7f0069fea700 1 -- 172.21.15.32:0/959153134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0064102580 msgr2=0x7f006410ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:19.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.949+0000 7f0069fea700 1 --2- 172.21.15.32:0/959153134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0064102580 0x7f006410ee30 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f004c009380 tx=0x7f004c009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:19.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.949+0000 7f0069fea700 1 -- 172.21.15.32:0/959153134 shutdown_connections 2024-04-12T06:04:19.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.949+0000 7f0069fea700 1 --2- 172.21.15.32:0/959153134 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f005006cc20 0x7f005006f0d0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:19.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.949+0000 7f0069fea700 1 --2- 172.21.15.32:0/959153134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0064101be0 0x7f006410e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:19.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.949+0000 7f0069fea700 1 --2- 172.21.15.32:0/959153134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0064102580 0x7f006410ee30 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:19.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.949+0000 7f0069fea700 1 -- 172.21.15.32:0/959153134 >> 172.21.15.32:0/959153134 conn(0x7f0064072ad0 msgr2=0x7f0064104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:19.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.949+0000 7f0069fea700 1 -- 172.21.15.32:0/959153134 shutdown_connections 2024-04-12T06:04:19.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:19.949+0000 7f0069fea700 1 -- 172.21.15.32:0/959153134 wait complete. 2024-04-12T06:04:19.957 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-12T06:04:20.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.084+0000 7f90d7242700 1 -- 172.21.15.32:0/1929016562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90d0102610 msgr2=0x7f90d00fbc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:20.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.084+0000 7f90d7242700 1 --2- 172.21.15.32:0/1929016562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90d0102610 0x7f90d00fbc80 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f90bc004940 tx=0x7f90bc009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:20.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.084+0000 7f90d7242700 1 -- 172.21.15.32:0/1929016562 shutdown_connections 2024-04-12T06:04:20.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.084+0000 7f90d7242700 1 --2- 172.21.15.32:0/1929016562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90d0102610 0x7f90d00fbc80 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:20.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.084+0000 7f90d7242700 1 --2- 172.21.15.32:0/1929016562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90d0101c70 0x7f90d0102040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:20.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.084+0000 7f90d7242700 1 -- 172.21.15.32:0/1929016562 >> 172.21.15.32:0/1929016562 conn(0x7f90d0071c80 msgr2=0x7f90d0072080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:20.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.084+0000 7f90d7242700 1 -- 172.21.15.32:0/1929016562 shutdown_connections 2024-04-12T06:04:20.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.084+0000 7f90d7242700 1 -- 172.21.15.32:0/1929016562 wait complete. 2024-04-12T06:04:20.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.085+0000 7f90d7242700 1 Processor -- start 2024-04-12T06:04:20.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.085+0000 7f90d7242700 1 -- start start 2024-04-12T06:04:20.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.085+0000 7f90d7242700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90d0101c70 0x7f90d01137e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:20.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.085+0000 7f90d7242700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90d0102610 0x7f90d0113d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:20.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.085+0000 7f90d7242700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d0114370 con 0x7f90d0101c70 2024-04-12T06:04:20.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.085+0000 7f90d7242700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d010d970 con 0x7f90d0102610 2024-04-12T06:04:20.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.085+0000 7f90d4fde700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90d0101c70 0x7f90d01137e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:20.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.085+0000 7f90d4fde700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90d0101c70 0x7f90d01137e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53560/0 (socket says 172.21.15.32:53560) 2024-04-12T06:04:20.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.085+0000 7f90d4fde700 1 -- 172.21.15.32:0/2729646663 learned_addr learned my addr 172.21.15.32:0/2729646663 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:20.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.085+0000 7f90cffff700 1 --2- 172.21.15.32:0/2729646663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90d0102610 0x7f90d0113d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:20.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.085+0000 7f90d4fde700 1 -- 172.21.15.32:0/2729646663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90d0102610 msgr2=0x7f90d0113d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:20.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.085+0000 7f90d4fde700 1 --2- 172.21.15.32:0/2729646663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90d0102610 0x7f90d0113d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:20.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.085+0000 7f90d4fde700 1 -- 172.21.15.32:0/2729646663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90bc005040 con 0x7f90d0101c70 2024-04-12T06:04:20.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.086+0000 7f90d4fde700 1 --2- 172.21.15.32:0/2729646663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90d0101c70 0x7f90d01137e0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f90c4002a50 tx=0x7f90c40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:20.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.086+0000 7f90cd7fa700 1 -- 172.21.15.32:0/2729646663 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90c4007cc0 con 0x7f90d0101c70 2024-04-12T06:04:20.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.086+0000 7f90cd7fa700 1 -- 172.21.15.32:0/2729646663 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f90c400f040 con 0x7f90d0101c70 2024-04-12T06:04:20.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.086+0000 7f90d7242700 1 -- 172.21.15.32:0/2729646663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d010dc50 con 0x7f90d0101c70 2024-04-12T06:04:20.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.086+0000 7f90cd7fa700 1 -- 172.21.15.32:0/2729646663 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90c400d9c0 con 0x7f90d0101c70 2024-04-12T06:04:20.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.086+0000 7f90d7242700 1 -- 172.21.15.32:0/2729646663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d010e1e0 con 0x7f90d0101c70 2024-04-12T06:04:20.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.087+0000 7f90d7242700 1 -- 172.21.15.32:0/2729646663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90d00fd3c0 con 0x7f90d0101c70 2024-04-12T06:04:20.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.087+0000 7f90cd7fa700 1 -- 172.21.15.32:0/2729646663 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f90c400db60 con 0x7f90d0101c70 2024-04-12T06:04:20.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.088+0000 7f90cd7fa700 1 --2- 172.21.15.32:0/2729646663 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f90c006cc20 0x7f90c006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:20.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.088+0000 7f90cd7fa700 1 -- 172.21.15.32:0/2729646663 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f90c408b750 con 0x7f90d0101c70 2024-04-12T06:04:20.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.088+0000 7f90cffff700 1 --2- 172.21.15.32:0/2729646663 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f90c006cc20 0x7f90c006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:20.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.088+0000 7f90cffff700 1 --2- 172.21.15.32:0/2729646663 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f90c006cc20 0x7f90c006f0d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f90d010ec50 tx=0x7f90bc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:20.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.092+0000 7f90cd7fa700 1 -- 172.21.15.32:0/2729646663 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f90c4059140 con 0x7f90d0101c70 2024-04-12T06:04:20.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:19 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/2588624498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T06:04:20.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:19 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3611957648' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-12T06:04:20.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:19 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/2588624498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T06:04:20.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:19 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3611957648' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-12T06:04:20.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.345+0000 7f90d7242700 1 -- 172.21.15.32:0/2729646663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90d0065130 con 0x7f90d0101c70 2024-04-12T06:04:20.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.345+0000 7f90cd7fa700 1 -- 172.21.15.32:0/2729646663 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4858 (secure 0 0 0) 0x7f90c4058cd0 con 0x7f90d0101c70 2024-04-12T06:04:20.341 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:04:20.341 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":40,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-12T05:34:22.641718+0000","modified":"2024-04-12T06:02:23.988915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34384,"mds_1":34416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34384":{"gid":34384,"name":"cephfs.smithi032.jehaaj","rank":0,"incarnation":34,"state":"up:active","state_seq":244,"addr":"172.21.15.32:6827/1884332935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1884332935},{"type":"v1","addr":"172.21.15.32:6827","nonce":1884332935}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34416":{"gid":34416,"name":"cephfs.smithi032.pndtry","rank":1,"incarnation":34,"state":"up:active","state_seq":240,"addr":"172.21.15.32:6829/1671696137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1671696137},{"type":"v1","addr":"172.21.15.32:6829","nonce":1671696137}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_44317":{"gid":44317,"name":"cephfs.smithi067.zesmgy","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6825/142107794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":142107794},{"type":"v1","addr":"172.21.15.67:6825","nonce":142107794}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_44343":{"gid":44343,"name":"cephfs.smithi067.ymvqih","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6827/2631637348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2631637348},{"type":"v1","addr":"172.21.15.67:6827","nonce":2631637348}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T06:04:20.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.348+0000 7f90d7242700 1 -- 172.21.15.32:0/2729646663 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f90c006cc20 msgr2=0x7f90c006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:20.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.348+0000 7f90d7242700 1 --2- 172.21.15.32:0/2729646663 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f90c006cc20 0x7f90c006f0d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f90d010ec50 tx=0x7f90bc009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:20.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.348+0000 7f90d7242700 1 -- 172.21.15.32:0/2729646663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90d0101c70 msgr2=0x7f90d01137e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:20.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.348+0000 7f90d7242700 1 --2- 172.21.15.32:0/2729646663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90d0101c70 0x7f90d01137e0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f90c4002a50 tx=0x7f90c40046b0 comp rx=0 tx=0).stop 2024-04-12T06:04:20.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.348+0000 7f90d7242700 1 -- 172.21.15.32:0/2729646663 shutdown_connections 2024-04-12T06:04:20.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.348+0000 7f90d7242700 1 --2- 172.21.15.32:0/2729646663 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f90c006cc20 0x7f90c006f0d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:20.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.348+0000 7f90d7242700 1 --2- 172.21.15.32:0/2729646663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90d0101c70 0x7f90d01137e0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:20.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.348+0000 7f90d7242700 1 --2- 172.21.15.32:0/2729646663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90d0102610 0x7f90d0113d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:20.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.348+0000 7f90d7242700 1 -- 172.21.15.32:0/2729646663 >> 172.21.15.32:0/2729646663 conn(0x7f90d0071c80 msgr2=0x7f90d00fa3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:20.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.348+0000 7f90d7242700 1 -- 172.21.15.32:0/2729646663 shutdown_connections 2024-04-12T06:04:20.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.348+0000 7f90d7242700 1 -- 172.21.15.32:0/2729646663 wait complete. 2024-04-12T06:04:20.344 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 40 2024-04-12T06:04:20.355 DEBUG:tasks.fs:checking fs fscid=1,name=cephfs state = {'epoch': 18, 'max_mds': 2, 'flags': 50} 2024-04-12T06:04:20.356 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 19 2024-04-12T06:04:20.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.484+0000 7f83963a0700 1 -- 172.21.15.32:0/3552129677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8390102ce0 msgr2=0x7f83901030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:20.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.484+0000 7f83963a0700 1 --2- 172.21.15.32:0/3552129677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8390102ce0 0x7f83901030b0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f8378004940 tx=0x7f8378009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:20.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.484+0000 7f83963a0700 1 -- 172.21.15.32:0/3552129677 shutdown_connections 2024-04-12T06:04:20.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.484+0000 7f83963a0700 1 --2- 172.21.15.32:0/3552129677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8390103680 0x7f8390064e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:20.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.484+0000 7f83963a0700 1 --2- 172.21.15.32:0/3552129677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8390102ce0 0x7f83901030b0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:20.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.484+0000 7f83963a0700 1 -- 172.21.15.32:0/3552129677 >> 172.21.15.32:0/3552129677 conn(0x7f8390072ad0 msgr2=0x7f8390072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:20.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.484+0000 7f83963a0700 1 -- 172.21.15.32:0/3552129677 shutdown_connections 2024-04-12T06:04:20.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.484+0000 7f83963a0700 1 -- 172.21.15.32:0/3552129677 wait complete. 2024-04-12T06:04:20.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.484+0000 7f83963a0700 1 Processor -- start 2024-04-12T06:04:20.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.485+0000 7f83963a0700 1 -- start start 2024-04-12T06:04:20.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.485+0000 7f83963a0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8390102ce0 0x7f83901137b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:20.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.485+0000 7f83963a0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8390103680 0x7f8390113cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:20.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.485+0000 7f83963a0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8390114340 con 0x7f8390103680 2024-04-12T06:04:20.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.485+0000 7f83963a0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839010d940 con 0x7f8390102ce0 2024-04-12T06:04:20.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.485+0000 7f838f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8390103680 0x7f8390113cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:20.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.485+0000 7f838f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8390103680 0x7f8390113cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53576/0 (socket says 172.21.15.32:53576) 2024-04-12T06:04:20.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.485+0000 7f838f7fe700 1 -- 172.21.15.32:0/3198998595 learned_addr learned my addr 172.21.15.32:0/3198998595 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:20.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.485+0000 7f838ffff700 1 --2- 172.21.15.32:0/3198998595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8390102ce0 0x7f83901137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:20.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.485+0000 7f838f7fe700 1 -- 172.21.15.32:0/3198998595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8390102ce0 msgr2=0x7f83901137b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:20.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.485+0000 7f838f7fe700 1 --2- 172.21.15.32:0/3198998595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8390102ce0 0x7f83901137b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:20.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.485+0000 7f838f7fe700 1 -- 172.21.15.32:0/3198998595 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8378005040 con 0x7f8390103680 2024-04-12T06:04:20.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.485+0000 7f838f7fe700 1 --2- 172.21.15.32:0/3198998595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8390103680 0x7f8390113cf0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f838000c800 tx=0x7f8380004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:20.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.486+0000 7f838cff9700 1 -- 172.21.15.32:0/3198998595 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83800106f0 con 0x7f8390103680 2024-04-12T06:04:20.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.486+0000 7f83963a0700 1 -- 172.21.15.32:0/3198998595 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f839010dc20 con 0x7f8390103680 2024-04-12T06:04:20.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.486+0000 7f838cff9700 1 -- 172.21.15.32:0/3198998595 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8380010d00 con 0x7f8390103680 2024-04-12T06:04:20.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.486+0000 7f838cff9700 1 -- 172.21.15.32:0/3198998595 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83800113a0 con 0x7f8390103680 2024-04-12T06:04:20.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.486+0000 7f83963a0700 1 -- 172.21.15.32:0/3198998595 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f839010e1e0 con 0x7f8390103680 2024-04-12T06:04:20.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.487+0000 7f83963a0700 1 -- 172.21.15.32:0/3198998595 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83900fe430 con 0x7f8390103680 2024-04-12T06:04:20.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.487+0000 7f838cff9700 1 -- 172.21.15.32:0/3198998595 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8380012400 con 0x7f8390103680 2024-04-12T06:04:20.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.488+0000 7f838cff9700 1 --2- 172.21.15.32:0/3198998595 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f837c06cc20 0x7f837c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:20.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.488+0000 7f838cff9700 1 -- 172.21.15.32:0/3198998595 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f838008c080 con 0x7f8390103680 2024-04-12T06:04:20.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.488+0000 7f838ffff700 1 --2- 172.21.15.32:0/3198998595 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f837c06cc20 0x7f837c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:20.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.488+0000 7f838ffff700 1 --2- 172.21.15.32:0/3198998595 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f837c06cc20 0x7f837c06f0d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8378004f90 tx=0x7f8378009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:20.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.492+0000 7f838cff9700 1 -- 172.21.15.32:0/3198998595 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8380059af0 con 0x7f8390103680 2024-04-12T06:04:20.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.744+0000 7f83963a0700 1 -- 172.21.15.32:0/3198998595 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 19, "format": "json"} v 0) v1 -- 0x7f8390109ca0 con 0x7f8390103680 2024-04-12T06:04:20.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.745+0000 7f838cff9700 1 -- 172.21.15.32:0/3198998595 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 19, "format": "json"}]=0 dumped fsmap epoch 19 v40) v1 ==== 107+0+4851 (secure 0 0 0) 0x7f8380059680 con 0x7f8390103680 2024-04-12T06:04:20.740 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:04:20.740 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-12T05:34:22.641718+0000","modified":"2024-04-12T05:36:08.246955+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14480,"mds_1":24281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14480":{"gid":14480,"name":"cephfs.smithi032.pndtry","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6829/387754149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":387754149},{"type":"v1","addr":"172.21.15.32:6829","nonce":387754149}]},"join_fscid":1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14490":{"gid":14490,"name":"cephfs.smithi032.jehaaj","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6827/3678031004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3678031004},{"type":"v1","addr":"172.21.15.32:6827","nonce":3678031004}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24281":{"gid":24281,"name":"cephfs.smithi067.zesmgy","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6825/1291914599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1291914599},{"type":"v1","addr":"172.21.15.67:6825","nonce":1291914599}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24295":{"gid":24295,"name":"cephfs.smithi067.ymvqih","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.67:6827/873395842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873395842},{"type":"v1","addr":"172.21.15.67:6827","nonce":873395842}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T06:04:20.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.747+0000 7f83963a0700 1 -- 172.21.15.32:0/3198998595 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f837c06cc20 msgr2=0x7f837c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:20.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.747+0000 7f83963a0700 1 --2- 172.21.15.32:0/3198998595 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f837c06cc20 0x7f837c06f0d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8378004f90 tx=0x7f8378009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:20.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.747+0000 7f83963a0700 1 -- 172.21.15.32:0/3198998595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8390103680 msgr2=0x7f8390113cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:20.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.747+0000 7f83963a0700 1 --2- 172.21.15.32:0/3198998595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8390103680 0x7f8390113cf0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f838000c800 tx=0x7f8380004470 comp rx=0 tx=0).stop 2024-04-12T06:04:20.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.747+0000 7f83963a0700 1 -- 172.21.15.32:0/3198998595 shutdown_connections 2024-04-12T06:04:20.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.747+0000 7f83963a0700 1 --2- 172.21.15.32:0/3198998595 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f837c06cc20 0x7f837c06f0d0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:20.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.747+0000 7f83963a0700 1 --2- 172.21.15.32:0/3198998595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8390102ce0 0x7f83901137b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:20.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.747+0000 7f83963a0700 1 --2- 172.21.15.32:0/3198998595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8390103680 0x7f8390113cf0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:20.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.747+0000 7f83963a0700 1 -- 172.21.15.32:0/3198998595 >> 172.21.15.32:0/3198998595 conn(0x7f8390072ad0 msgr2=0x7f8390105a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:20.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.747+0000 7f83963a0700 1 -- 172.21.15.32:0/3198998595 shutdown_connections 2024-04-12T06:04:20.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.747+0000 7f83963a0700 1 -- 172.21.15.32:0/3198998595 wait complete. 2024-04-12T06:04:20.743 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 19 2024-04-12T06:04:20.756 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 20 2024-04-12T06:04:20.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.885+0000 7ff92b496700 1 -- 172.21.15.32:0/2400544834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9240fdce0 msgr2=0x7ff9240fe0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:20.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.885+0000 7ff92b496700 1 --2- 172.21.15.32:0/2400544834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9240fdce0 0x7ff9240fe0b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff914004940 tx=0x7ff914009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:20.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.885+0000 7ff92b496700 1 -- 172.21.15.32:0/2400544834 shutdown_connections 2024-04-12T06:04:20.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.885+0000 7ff92b496700 1 --2- 172.21.15.32:0/2400544834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9240fe680 0x7ff924106cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:20.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.885+0000 7ff92b496700 1 --2- 172.21.15.32:0/2400544834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9240fdce0 0x7ff9240fe0b0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:20.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.885+0000 7ff92b496700 1 -- 172.21.15.32:0/2400544834 >> 172.21.15.32:0/2400544834 conn(0x7ff9240f7710 msgr2=0x7ff9240f9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:20.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.885+0000 7ff92b496700 1 -- 172.21.15.32:0/2400544834 shutdown_connections 2024-04-12T06:04:20.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.885+0000 7ff92b496700 1 -- 172.21.15.32:0/2400544834 wait complete. 2024-04-12T06:04:20.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.886+0000 7ff92b496700 1 Processor -- start 2024-04-12T06:04:20.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.886+0000 7ff92b496700 1 -- start start 2024-04-12T06:04:20.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.886+0000 7ff92b496700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9240fdce0 0x7ff924113710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:20.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.886+0000 7ff92b496700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9240fe680 0x7ff924113c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:20.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.886+0000 7ff92b496700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9241142a0 con 0x7ff9240fdce0 2024-04-12T06:04:20.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.886+0000 7ff92b496700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92410d870 con 0x7ff9240fe680 2024-04-12T06:04:20.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.886+0000 7ff929232700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9240fdce0 0x7ff924113710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:20.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.886+0000 7ff929232700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9240fdce0 0x7ff924113710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53598/0 (socket says 172.21.15.32:53598) 2024-04-12T06:04:20.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.886+0000 7ff929232700 1 -- 172.21.15.32:0/298349068 learned_addr learned my addr 172.21.15.32:0/298349068 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:20.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.886+0000 7ff928a31700 1 --2- 172.21.15.32:0/298349068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9240fe680 0x7ff924113c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:20.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.886+0000 7ff929232700 1 -- 172.21.15.32:0/298349068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9240fe680 msgr2=0x7ff924113c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:20.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.886+0000 7ff929232700 1 --2- 172.21.15.32:0/298349068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9240fe680 0x7ff924113c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:20.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.886+0000 7ff929232700 1 -- 172.21.15.32:0/298349068 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff914005040 con 0x7ff9240fdce0 2024-04-12T06:04:20.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.886+0000 7ff929232700 1 --2- 172.21.15.32:0/298349068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9240fdce0 0x7ff924113710 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff914004890 tx=0x7ff914009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:20.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.887+0000 7ff919ffb700 1 -- 172.21.15.32:0/298349068 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff91400e040 con 0x7ff9240fdce0 2024-04-12T06:04:20.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.887+0000 7ff92b496700 1 -- 172.21.15.32:0/298349068 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff92410daf0 con 0x7ff9240fdce0 2024-04-12T06:04:20.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.887+0000 7ff919ffb700 1 -- 172.21.15.32:0/298349068 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff914009af0 con 0x7ff9240fdce0 2024-04-12T06:04:20.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.887+0000 7ff919ffb700 1 -- 172.21.15.32:0/298349068 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff91401c3f0 con 0x7ff9240fdce0 2024-04-12T06:04:20.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.887+0000 7ff92b496700 1 -- 172.21.15.32:0/298349068 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff92410e030 con 0x7ff9240fdce0 2024-04-12T06:04:20.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.888+0000 7ff92b496700 1 -- 172.21.15.32:0/298349068 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff924066510 con 0x7ff9240fdce0 2024-04-12T06:04:20.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.888+0000 7ff919ffb700 1 -- 172.21.15.32:0/298349068 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ff914013420 con 0x7ff9240fdce0 2024-04-12T06:04:20.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.889+0000 7ff919ffb700 1 --2- 172.21.15.32:0/298349068 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff91006cc20 0x7ff91006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:20.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.889+0000 7ff919ffb700 1 -- 172.21.15.32:0/298349068 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7ff91408d3c0 con 0x7ff9240fdce0 2024-04-12T06:04:20.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.889+0000 7ff928a31700 1 --2- 172.21.15.32:0/298349068 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff91006cc20 0x7ff91006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:20.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.889+0000 7ff928a31700 1 --2- 172.21.15.32:0/298349068 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff91006cc20 0x7ff91006f0d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff9240ff3c0 tx=0x7ff92000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:20.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:20.893+0000 7ff919ffb700 1 -- 172.21.15.32:0/298349068 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff91405ad90 con 0x7ff9240fdce0 2024-04-12T06:04:21.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:20 smithi032 ceph-mon[146817]: pgmap v715: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:04:21.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:20 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/959153134' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-12T06:04:21.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:20 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/2729646663' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T06:04:21.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:20 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T06:04:21.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:20 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3198998595' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-04-12T06:04:21.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.149+0000 7ff92b496700 1 -- 172.21.15.32:0/298349068 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 20, "format": "json"} v 0) v1 -- 0x7ff92410ed50 con 0x7ff9240fdce0 2024-04-12T06:04:21.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.150+0000 7ff919ffb700 1 -- 172.21.15.32:0/298349068 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 20, "format": "json"}]=0 dumped fsmap epoch 20 v40) v1 ==== 107+0+4852 (secure 0 0 0) 0x7ff91405a920 con 0x7ff9240fdce0 2024-04-12T06:04:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:04:21.146 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-12T05:34:22.641718+0000","modified":"2024-04-12T05:37:10.261768+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14480,"mds_1":24281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14480":{"gid":14480,"name":"cephfs.smithi032.pndtry","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6829/387754149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":387754149},{"type":"v1","addr":"172.21.15.32:6829","nonce":387754149}]},"join_fscid":1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14490":{"gid":14490,"name":"cephfs.smithi032.jehaaj","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6827/3678031004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3678031004},{"type":"v1","addr":"172.21.15.32:6827","nonce":3678031004}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24281":{"gid":24281,"name":"cephfs.smithi067.zesmgy","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6825/1291914599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1291914599},{"type":"v1","addr":"172.21.15.67:6825","nonce":1291914599}]},"join_fscid":1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24295":{"gid":24295,"name":"cephfs.smithi067.ymvqih","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.67:6827/873395842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873395842},{"type":"v1","addr":"172.21.15.67:6827","nonce":873395842}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T06:04:21.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.152+0000 7ff92b496700 1 -- 172.21.15.32:0/298349068 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff91006cc20 msgr2=0x7ff91006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:21.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.152+0000 7ff92b496700 1 --2- 172.21.15.32:0/298349068 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff91006cc20 0x7ff91006f0d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff9240ff3c0 tx=0x7ff92000b040 comp rx=0 tx=0).stop 2024-04-12T06:04:21.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.152+0000 7ff92b496700 1 -- 172.21.15.32:0/298349068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9240fdce0 msgr2=0x7ff924113710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:21.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.152+0000 7ff92b496700 1 --2- 172.21.15.32:0/298349068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9240fdce0 0x7ff924113710 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff914004890 tx=0x7ff914009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:21.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.152+0000 7ff92b496700 1 -- 172.21.15.32:0/298349068 shutdown_connections 2024-04-12T06:04:21.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.152+0000 7ff92b496700 1 --2- 172.21.15.32:0/298349068 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff91006cc20 0x7ff91006f0d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:21.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.152+0000 7ff92b496700 1 --2- 172.21.15.32:0/298349068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9240fdce0 0x7ff924113710 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:21.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.152+0000 7ff92b496700 1 --2- 172.21.15.32:0/298349068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9240fe680 0x7ff924113c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:21.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.152+0000 7ff92b496700 1 -- 172.21.15.32:0/298349068 >> 172.21.15.32:0/298349068 conn(0x7ff9240f7710 msgr2=0x7ff9240f8d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:21.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.152+0000 7ff92b496700 1 -- 172.21.15.32:0/298349068 shutdown_connections 2024-04-12T06:04:21.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.152+0000 7ff92b496700 1 -- 172.21.15.32:0/298349068 wait complete. 2024-04-12T06:04:21.149 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 20 2024-04-12T06:04:21.160 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 21 2024-04-12T06:04:21.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:20 smithi067 ceph-mon[112979]: pgmap v715: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:04:21.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:20 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/959153134' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-12T06:04:21.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:20 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/2729646663' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T06:04:21.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:20 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T06:04:21.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:20 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3198998595' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-04-12T06:04:21.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.291+0000 7f6491950700 1 -- 172.21.15.32:0/142777673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f648c102590 msgr2=0x7f648c0fbc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:21.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.291+0000 7f6491950700 1 --2- 172.21.15.32:0/142777673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f648c102590 0x7f648c0fbc00 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f647c004940 tx=0x7f647c009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:21.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.291+0000 7f6491950700 1 -- 172.21.15.32:0/142777673 shutdown_connections 2024-04-12T06:04:21.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.291+0000 7f6491950700 1 --2- 172.21.15.32:0/142777673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f648c102590 0x7f648c0fbc00 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:21.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.291+0000 7f6491950700 1 --2- 172.21.15.32:0/142777673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f648c101bf0 0x7f648c101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:21.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.291+0000 7f6491950700 1 -- 172.21.15.32:0/142777673 >> 172.21.15.32:0/142777673 conn(0x7f648c0f77a0 msgr2=0x7f648c0f9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:21.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.291+0000 7f6491950700 1 -- 172.21.15.32:0/142777673 shutdown_connections 2024-04-12T06:04:21.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.291+0000 7f6491950700 1 -- 172.21.15.32:0/142777673 wait complete. 2024-04-12T06:04:21.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.291+0000 7f6491950700 1 Processor -- start 2024-04-12T06:04:21.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.292+0000 7f6491950700 1 -- start start 2024-04-12T06:04:21.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.292+0000 7f6491950700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f648c101bf0 0x7f648c10e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:21.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.292+0000 7f6491950700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f648c102590 0x7f648c10edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:21.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.292+0000 7f6491950700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648c10f330 con 0x7f648c102590 2024-04-12T06:04:21.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.292+0000 7f6491950700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648c10f470 con 0x7f648c101bf0 2024-04-12T06:04:21.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.292+0000 7f648a7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f648c102590 0x7f648c10edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:21.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.292+0000 7f648a7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f648c102590 0x7f648c10edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53608/0 (socket says 172.21.15.32:53608) 2024-04-12T06:04:21.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.292+0000 7f648a7fc700 1 -- 172.21.15.32:0/2738776665 learned_addr learned my addr 172.21.15.32:0/2738776665 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:21.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.292+0000 7f648affd700 1 --2- 172.21.15.32:0/2738776665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f648c101bf0 0x7f648c10e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:21.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.292+0000 7f648a7fc700 1 -- 172.21.15.32:0/2738776665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f648c101bf0 msgr2=0x7f648c10e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:21.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.292+0000 7f648a7fc700 1 --2- 172.21.15.32:0/2738776665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f648c101bf0 0x7f648c10e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:21.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.292+0000 7f648a7fc700 1 -- 172.21.15.32:0/2738776665 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647c005040 con 0x7f648c102590 2024-04-12T06:04:21.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.292+0000 7f648a7fc700 1 --2- 172.21.15.32:0/2738776665 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f648c102590 0x7f648c10edf0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f647c000c00 tx=0x7f647c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:21.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.292+0000 7f64737fe700 1 -- 172.21.15.32:0/2738776665 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f647c00e070 con 0x7f648c102590 2024-04-12T06:04:21.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.293+0000 7f6491950700 1 -- 172.21.15.32:0/2738776665 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f648c10f6f0 con 0x7f648c102590 2024-04-12T06:04:21.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.293+0000 7f64737fe700 1 -- 172.21.15.32:0/2738776665 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f647c0093b0 con 0x7f648c102590 2024-04-12T06:04:21.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.293+0000 7f64737fe700 1 -- 172.21.15.32:0/2738776665 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f647c01ccb0 con 0x7f648c102590 2024-04-12T06:04:21.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.293+0000 7f6491950700 1 -- 172.21.15.32:0/2738776665 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f648c117c10 con 0x7f648c102590 2024-04-12T06:04:21.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.293+0000 7f6491950700 1 -- 172.21.15.32:0/2738776665 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f648c0fd340 con 0x7f648c102590 2024-04-12T06:04:21.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.294+0000 7f64737fe700 1 -- 172.21.15.32:0/2738776665 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f647c01c490 con 0x7f648c102590 2024-04-12T06:04:21.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.294+0000 7f64737fe700 1 --2- 172.21.15.32:0/2738776665 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f647406cc20 0x7f647406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:21.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.294+0000 7f64737fe700 1 -- 172.21.15.32:0/2738776665 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f647c08d500 con 0x7f648c102590 2024-04-12T06:04:21.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.294+0000 7f648affd700 1 --2- 172.21.15.32:0/2738776665 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f647406cc20 0x7f647406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:21.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.295+0000 7f648affd700 1 --2- 172.21.15.32:0/2738776665 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f647406cc20 0x7f647406f0d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6480009b80 tx=0x7f648000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:21.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.299+0000 7f64737fe700 1 -- 172.21.15.32:0/2738776665 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f647c0574e0 con 0x7f648c102590 2024-04-12T06:04:21.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.549+0000 7f6491950700 1 -- 172.21.15.32:0/2738776665 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 21, "format": "json"} v 0) v1 -- 0x7f648c117e00 con 0x7f648c102590 2024-04-12T06:04:21.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.550+0000 7f64737fe700 1 -- 172.21.15.32:0/2738776665 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 21, "format": "json"}]=0 dumped fsmap epoch 21 v40) v1 ==== 107+0+4851 (secure 0 0 0) 0x7f647c05ab00 con 0x7f648c102590 2024-04-12T06:04:21.545 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:04:21.545 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-12T05:34:22.641718+0000","modified":"2024-04-12T05:41:00.341546+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14480,"mds_1":24281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14480":{"gid":14480,"name":"cephfs.smithi032.pndtry","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6829/387754149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":387754149},{"type":"v1","addr":"172.21.15.32:6829","nonce":387754149}]},"join_fscid":1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14490":{"gid":14490,"name":"cephfs.smithi032.jehaaj","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6827/3678031004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3678031004},{"type":"v1","addr":"172.21.15.32:6827","nonce":3678031004}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24281":{"gid":24281,"name":"cephfs.smithi067.zesmgy","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6825/1291914599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1291914599},{"type":"v1","addr":"172.21.15.67:6825","nonce":1291914599}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24295":{"gid":24295,"name":"cephfs.smithi067.ymvqih","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.67:6827/873395842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873395842},{"type":"v1","addr":"172.21.15.67:6827","nonce":873395842}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T06:04:21.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.552+0000 7f6491950700 1 -- 172.21.15.32:0/2738776665 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f647406cc20 msgr2=0x7f647406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:21.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.552+0000 7f6491950700 1 --2- 172.21.15.32:0/2738776665 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f647406cc20 0x7f647406f0d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6480009b80 tx=0x7f648000b040 comp rx=0 tx=0).stop 2024-04-12T06:04:21.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.552+0000 7f6491950700 1 -- 172.21.15.32:0/2738776665 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f648c102590 msgr2=0x7f648c10edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:21.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.552+0000 7f6491950700 1 --2- 172.21.15.32:0/2738776665 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f648c102590 0x7f648c10edf0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f647c000c00 tx=0x7f647c009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:21.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.552+0000 7f6491950700 1 -- 172.21.15.32:0/2738776665 shutdown_connections 2024-04-12T06:04:21.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.552+0000 7f6491950700 1 --2- 172.21.15.32:0/2738776665 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f647406cc20 0x7f647406f0d0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:21.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.552+0000 7f6491950700 1 --2- 172.21.15.32:0/2738776665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f648c101bf0 0x7f648c10e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:21.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.552+0000 7f6491950700 1 --2- 172.21.15.32:0/2738776665 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f648c102590 0x7f648c10edf0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:21.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.552+0000 7f6491950700 1 -- 172.21.15.32:0/2738776665 >> 172.21.15.32:0/2738776665 conn(0x7f648c0f77a0 msgr2=0x7f648c0f87f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:21.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.552+0000 7f6491950700 1 -- 172.21.15.32:0/2738776665 shutdown_connections 2024-04-12T06:04:21.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.552+0000 7f6491950700 1 -- 172.21.15.32:0/2738776665 wait complete. 2024-04-12T06:04:21.549 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 21 2024-04-12T06:04:21.560 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 22 2024-04-12T06:04:21.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.688+0000 7f3ed024c700 1 -- 172.21.15.32:0/1607107753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec8102d00 msgr2=0x7f3ec81030d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:21.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.688+0000 7f3ed024c700 1 --2- 172.21.15.32:0/1607107753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec8102d00 0x7f3ec81030d0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4004940 tx=0x7f3ec4009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:21.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.688+0000 7f3ed024c700 1 -- 172.21.15.32:0/1607107753 shutdown_connections 2024-04-12T06:04:21.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.688+0000 7f3ed024c700 1 --2- 172.21.15.32:0/1607107753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ec81036a0 0x7f3ec80fcd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:21.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.688+0000 7f3ed024c700 1 --2- 172.21.15.32:0/1607107753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec8102d00 0x7f3ec81030d0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:21.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.688+0000 7f3ed024c700 1 -- 172.21.15.32:0/1607107753 >> 172.21.15.32:0/1607107753 conn(0x7f3ec80f8810 msgr2=0x7f3ec80fac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:21.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.688+0000 7f3ed024c700 1 -- 172.21.15.32:0/1607107753 shutdown_connections 2024-04-12T06:04:21.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.688+0000 7f3ed024c700 1 -- 172.21.15.32:0/1607107753 wait complete. 2024-04-12T06:04:21.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.689+0000 7f3ed024c700 1 Processor -- start 2024-04-12T06:04:21.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.689+0000 7f3ed024c700 1 -- start start 2024-04-12T06:04:21.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.689+0000 7f3ed024c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ec8102d00 0x7f3ec8113790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:21.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.689+0000 7f3ed024c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec81036a0 0x7f3ec8113cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:21.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.689+0000 7f3ed024c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec8114320 con 0x7f3ec81036a0 2024-04-12T06:04:21.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.689+0000 7f3ed024c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec810d920 con 0x7f3ec8102d00 2024-04-12T06:04:21.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.689+0000 7f3ecd7e7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec81036a0 0x7f3ec8113cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:21.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.689+0000 7f3ecd7e7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec81036a0 0x7f3ec8113cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51602/0 (socket says 172.21.15.32:51602) 2024-04-12T06:04:21.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.689+0000 7f3ecdfe8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ec8102d00 0x7f3ec8113790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:21.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.689+0000 7f3ecd7e7700 1 -- 172.21.15.32:0/517296408 learned_addr learned my addr 172.21.15.32:0/517296408 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:21.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.690+0000 7f3ecd7e7700 1 -- 172.21.15.32:0/517296408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ec8102d00 msgr2=0x7f3ec8113790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:21.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.690+0000 7f3ecd7e7700 1 --2- 172.21.15.32:0/517296408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ec8102d00 0x7f3ec8113790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:21.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.690+0000 7f3ecd7e7700 1 -- 172.21.15.32:0/517296408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec4005040 con 0x7f3ec81036a0 2024-04-12T06:04:21.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.690+0000 7f3ecd7e7700 1 --2- 172.21.15.32:0/517296408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec81036a0 0x7f3ec8113cd0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc00c890 tx=0x7f3ebc0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:21.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.690+0000 7f3eba7fc700 1 -- 172.21.15.32:0/517296408 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ebc00a700 con 0x7f3ec81036a0 2024-04-12T06:04:21.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.690+0000 7f3ed024c700 1 -- 172.21.15.32:0/517296408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ec810dc00 con 0x7f3ec81036a0 2024-04-12T06:04:21.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.690+0000 7f3eba7fc700 1 -- 172.21.15.32:0/517296408 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ebc00ad10 con 0x7f3ec81036a0 2024-04-12T06:04:21.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.690+0000 7f3eba7fc700 1 -- 172.21.15.32:0/517296408 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ebc012e70 con 0x7f3ec81036a0 2024-04-12T06:04:21.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.690+0000 7f3ed024c700 1 -- 172.21.15.32:0/517296408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ec810e1c0 con 0x7f3ec81036a0 2024-04-12T06:04:21.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.691+0000 7f3ed024c700 1 -- 172.21.15.32:0/517296408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ec80fe450 con 0x7f3ec81036a0 2024-04-12T06:04:21.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.692+0000 7f3eba7fc700 1 -- 172.21.15.32:0/517296408 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3ebc00a860 con 0x7f3ec81036a0 2024-04-12T06:04:21.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.692+0000 7f3eba7fc700 1 --2- 172.21.15.32:0/517296408 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3eb406cc20 0x7f3eb406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:21.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.692+0000 7f3eba7fc700 1 -- 172.21.15.32:0/517296408 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f3ebc08c110 con 0x7f3ec81036a0 2024-04-12T06:04:21.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.692+0000 7f3ecdfe8700 1 --2- 172.21.15.32:0/517296408 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3eb406cc20 0x7f3eb406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:21.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.692+0000 7f3ecdfe8700 1 --2- 172.21.15.32:0/517296408 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3eb406cc20 0x7f3eb406f0d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4009fd0 tx=0x7f3ec4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:21.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.696+0000 7f3eba7fc700 1 -- 172.21.15.32:0/517296408 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3ebc059b80 con 0x7f3ec81036a0 2024-04-12T06:04:21.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.950+0000 7f3ed024c700 1 -- 172.21.15.32:0/517296408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 22, "format": "json"} v 0) v1 -- 0x7f3ec8109cc0 con 0x7f3ec81036a0 2024-04-12T06:04:21.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.951+0000 7f3eba7fc700 1 -- 172.21.15.32:0/517296408 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 22, "format": "json"}]=0 dumped fsmap epoch 22 v40) v1 ==== 107+0+4850 (secure 0 0 0) 0x7f3ebc059710 con 0x7f3ec81036a0 2024-04-12T06:04:21.946 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:04:21.946 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-12T05:34:22.641718+0000","modified":"2024-04-12T05:41:05.358509+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14480,"mds_1":24281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14480":{"gid":14480,"name":"cephfs.smithi032.pndtry","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6829/387754149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":387754149},{"type":"v1","addr":"172.21.15.32:6829","nonce":387754149}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14490":{"gid":14490,"name":"cephfs.smithi032.jehaaj","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6827/3678031004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3678031004},{"type":"v1","addr":"172.21.15.32:6827","nonce":3678031004}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24281":{"gid":24281,"name":"cephfs.smithi067.zesmgy","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6825/1291914599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1291914599},{"type":"v1","addr":"172.21.15.67:6825","nonce":1291914599}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24295":{"gid":24295,"name":"cephfs.smithi067.ymvqih","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.67:6827/873395842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873395842},{"type":"v1","addr":"172.21.15.67:6827","nonce":873395842}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T06:04:21.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.953+0000 7f3ed024c700 1 -- 172.21.15.32:0/517296408 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3eb406cc20 msgr2=0x7f3eb406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:21.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.953+0000 7f3ed024c700 1 --2- 172.21.15.32:0/517296408 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3eb406cc20 0x7f3eb406f0d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4009fd0 tx=0x7f3ec4009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:21.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.953+0000 7f3ed024c700 1 -- 172.21.15.32:0/517296408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec81036a0 msgr2=0x7f3ec8113cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:21.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.953+0000 7f3ed024c700 1 --2- 172.21.15.32:0/517296408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec81036a0 0x7f3ec8113cd0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc00c890 tx=0x7f3ebc0044c0 comp rx=0 tx=0).stop 2024-04-12T06:04:21.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.953+0000 7f3ed024c700 1 -- 172.21.15.32:0/517296408 shutdown_connections 2024-04-12T06:04:21.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.953+0000 7f3ed024c700 1 --2- 172.21.15.32:0/517296408 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3eb406cc20 0x7f3eb406f0d0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:21.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.953+0000 7f3ed024c700 1 --2- 172.21.15.32:0/517296408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ec8102d00 0x7f3ec8113790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:21.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.953+0000 7f3ed024c700 1 --2- 172.21.15.32:0/517296408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec81036a0 0x7f3ec8113cd0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:21.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.953+0000 7f3ed024c700 1 -- 172.21.15.32:0/517296408 >> 172.21.15.32:0/517296408 conn(0x7f3ec80f8810 msgr2=0x7f3ec80f9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:21.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.953+0000 7f3ed024c700 1 -- 172.21.15.32:0/517296408 shutdown_connections 2024-04-12T06:04:21.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:21.953+0000 7f3ed024c700 1 -- 172.21.15.32:0/517296408 wait complete. 2024-04-12T06:04:21.950 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 22 2024-04-12T06:04:21.961 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 23 2024-04-12T06:04:22.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.089+0000 7f0804d76700 1 -- 172.21.15.32:0/2007920324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0800102d80 msgr2=0x7f0800103150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:22.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.089+0000 7f0804d76700 1 --2- 172.21.15.32:0/2007920324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0800102d80 0x7f0800103150 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f07f4004940 tx=0x7f07f4009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:22.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.089+0000 7f0804d76700 1 -- 172.21.15.32:0/2007920324 shutdown_connections 2024-04-12T06:04:22.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.089+0000 7f0804d76700 1 --2- 172.21.15.32:0/2007920324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0800103720 0x7f08000fb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:22.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.089+0000 7f0804d76700 1 --2- 172.21.15.32:0/2007920324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0800102d80 0x7f0800103150 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:22.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.089+0000 7f0804d76700 1 -- 172.21.15.32:0/2007920324 >> 172.21.15.32:0/2007920324 conn(0x7f0800072230 msgr2=0x7f0800072630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:22.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.089+0000 7f0804d76700 1 -- 172.21.15.32:0/2007920324 shutdown_connections 2024-04-12T06:04:22.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.089+0000 7f0804d76700 1 -- 172.21.15.32:0/2007920324 wait complete. 2024-04-12T06:04:22.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.089+0000 7f0804d76700 1 Processor -- start 2024-04-12T06:04:22.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.089+0000 7f0804d76700 1 -- start start 2024-04-12T06:04:22.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.090+0000 7f0804d76700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0800102d80 0x7f0800113680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:22.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.090+0000 7f0804d76700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0800103720 0x7f0800113c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:22.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.090+0000 7f0804d76700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0800114250 con 0x7f0800103720 2024-04-12T06:04:22.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.090+0000 7f0804d76700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080010d810 con 0x7f0800102d80 2024-04-12T06:04:22.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.090+0000 7f07fdd9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0800103720 0x7f0800113c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:22.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.090+0000 7f07fdd9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0800103720 0x7f0800113c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51624/0 (socket says 172.21.15.32:51624) 2024-04-12T06:04:22.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.090+0000 7f07fdd9b700 1 -- 172.21.15.32:0/4229939736 learned_addr learned my addr 172.21.15.32:0/4229939736 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:22.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.090+0000 7f07fe59c700 1 --2- 172.21.15.32:0/4229939736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0800102d80 0x7f0800113680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:22.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.090+0000 7f07fdd9b700 1 -- 172.21.15.32:0/4229939736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0800102d80 msgr2=0x7f0800113680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:22.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.090+0000 7f07fdd9b700 1 --2- 172.21.15.32:0/4229939736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0800102d80 0x7f0800113680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:22.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.090+0000 7f07fdd9b700 1 -- 172.21.15.32:0/4229939736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f4005040 con 0x7f0800103720 2024-04-12T06:04:22.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.090+0000 7f07fdd9b700 1 --2- 172.21.15.32:0/4229939736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0800103720 0x7f0800113c00 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f07f000eb40 tx=0x7f07f000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:22.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.090+0000 7f07eeffd700 1 -- 172.21.15.32:0/4229939736 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07f0007cc0 con 0x7f0800103720 2024-04-12T06:04:22.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.090+0000 7f07eeffd700 1 -- 172.21.15.32:0/4229939736 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07f000f040 con 0x7f0800103720 2024-04-12T06:04:22.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.090+0000 7f0804d76700 1 -- 172.21.15.32:0/4229939736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f080010da40 con 0x7f0800103720 2024-04-12T06:04:22.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.090+0000 7f07eeffd700 1 -- 172.21.15.32:0/4229939736 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07f000c8f0 con 0x7f0800103720 2024-04-12T06:04:22.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.090+0000 7f0804d76700 1 -- 172.21.15.32:0/4229939736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f080010df80 con 0x7f0800103720 2024-04-12T06:04:22.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.091+0000 7f0804d76700 1 -- 172.21.15.32:0/4229939736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08001000e0 con 0x7f0800103720 2024-04-12T06:04:22.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.092+0000 7f07eeffd700 1 -- 172.21.15.32:0/4229939736 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f07f000cb60 con 0x7f0800103720 2024-04-12T06:04:22.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.092+0000 7f07eeffd700 1 --2- 172.21.15.32:0/4229939736 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f07e806cc20 0x7f07e806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:22.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.092+0000 7f07eeffd700 1 -- 172.21.15.32:0/4229939736 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f07f008bbe0 con 0x7f0800103720 2024-04-12T06:04:22.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.092+0000 7f07fe59c700 1 --2- 172.21.15.32:0/4229939736 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f07e806cc20 0x7f07e806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:22.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.093+0000 7f07fe59c700 1 --2- 172.21.15.32:0/4229939736 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f07e806cc20 0x7f07e806f0d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f07f400b0d0 tx=0x7f07f4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:22.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.097+0000 7f07eeffd700 1 -- 172.21.15.32:0/4229939736 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f07f00595e0 con 0x7f0800103720 2024-04-12T06:04:22.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:21 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/298349068' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-04-12T06:04:22.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:21 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T06:04:22.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:21 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T06:04:22.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:21 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:04:22.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:21 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/2738776665' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-04-12T06:04:22.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:21 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/298349068' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-04-12T06:04:22.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:21 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T06:04:22.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:21 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T06:04:22.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:21 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:04:22.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:21 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/2738776665' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-04-12T06:04:22.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.351+0000 7f0804d76700 1 -- 172.21.15.32:0/4229939736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 23, "format": "json"} v 0) v1 -- 0x7f0800109d40 con 0x7f0800103720 2024-04-12T06:04:22.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.352+0000 7f07eeffd700 1 -- 172.21.15.32:0/4229939736 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 23, "format": "json"}]=0 dumped fsmap epoch 23 v40) v1 ==== 107+0+3281 (secure 0 0 0) 0x7f07f0019cf0 con 0x7f0800103720 2024-04-12T06:04:22.347 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:04:22.347 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-12T05:34:22.641718+0000","modified":"2024-04-12T05:46:10.494829+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14480,"mds_1":24281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14480":{"gid":14480,"name":"cephfs.smithi032.pndtry","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6829/387754149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":387754149},{"type":"v1","addr":"172.21.15.32:6829","nonce":387754149}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24281":{"gid":24281,"name":"cephfs.smithi067.zesmgy","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6825/1291914599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1291914599},{"type":"v1","addr":"172.21.15.67:6825","nonce":1291914599}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T06:04:22.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.354+0000 7f0804d76700 1 -- 172.21.15.32:0/4229939736 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f07e806cc20 msgr2=0x7f07e806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:22.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.354+0000 7f0804d76700 1 --2- 172.21.15.32:0/4229939736 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f07e806cc20 0x7f07e806f0d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f07f400b0d0 tx=0x7f07f4009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:22.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.354+0000 7f0804d76700 1 -- 172.21.15.32:0/4229939736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0800103720 msgr2=0x7f0800113c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:22.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.354+0000 7f0804d76700 1 --2- 172.21.15.32:0/4229939736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0800103720 0x7f0800113c00 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f07f000eb40 tx=0x7f07f000b040 comp rx=0 tx=0).stop 2024-04-12T06:04:22.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.354+0000 7f0804d76700 1 -- 172.21.15.32:0/4229939736 shutdown_connections 2024-04-12T06:04:22.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.354+0000 7f0804d76700 1 --2- 172.21.15.32:0/4229939736 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f07e806cc20 0x7f07e806f0d0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:22.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.354+0000 7f0804d76700 1 --2- 172.21.15.32:0/4229939736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0800102d80 0x7f0800113680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:22.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.354+0000 7f0804d76700 1 --2- 172.21.15.32:0/4229939736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0800103720 0x7f0800113c00 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:22.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.354+0000 7f0804d76700 1 -- 172.21.15.32:0/4229939736 >> 172.21.15.32:0/4229939736 conn(0x7f0800072230 msgr2=0x7f08000faac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:22.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.354+0000 7f0804d76700 1 -- 172.21.15.32:0/4229939736 shutdown_connections 2024-04-12T06:04:22.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.354+0000 7f0804d76700 1 -- 172.21.15.32:0/4229939736 wait complete. 2024-04-12T06:04:22.350 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 23 2024-04-12T06:04:22.362 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 23 2024-04-12T06:04:22.362 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 24 2024-04-12T06:04:22.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.490+0000 7f5e6a375700 1 -- 172.21.15.32:0/2968360557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e64102cc0 msgr2=0x7f5e64103090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:22.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.490+0000 7f5e6a375700 1 --2- 172.21.15.32:0/2968360557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e64102cc0 0x7f5e64103090 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c004940 tx=0x7f5e4c009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:22.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.490+0000 7f5e6a375700 1 -- 172.21.15.32:0/2968360557 shutdown_connections 2024-04-12T06:04:22.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.490+0000 7f5e6a375700 1 --2- 172.21.15.32:0/2968360557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e64103660 0x7f5e640fccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:22.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.490+0000 7f5e6a375700 1 --2- 172.21.15.32:0/2968360557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e64102cc0 0x7f5e64103090 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:22.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.490+0000 7f5e6a375700 1 -- 172.21.15.32:0/2968360557 >> 172.21.15.32:0/2968360557 conn(0x7f5e640f8830 msgr2=0x7f5e640fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:22.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.490+0000 7f5e6a375700 1 -- 172.21.15.32:0/2968360557 shutdown_connections 2024-04-12T06:04:22.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.490+0000 7f5e6a375700 1 -- 172.21.15.32:0/2968360557 wait complete. 2024-04-12T06:04:22.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.490+0000 7f5e6a375700 1 Processor -- start 2024-04-12T06:04:22.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.491+0000 7f5e6a375700 1 -- start start 2024-04-12T06:04:22.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.491+0000 7f5e6a375700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e64102cc0 0x7f5e64113760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:22.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.491+0000 7f5e6a375700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e64103660 0x7f5e64113ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:22.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.491+0000 7f5e6a375700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e64114380 con 0x7f5e64102cc0 2024-04-12T06:04:22.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.491+0000 7f5e6a375700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6410d8f0 con 0x7f5e64103660 2024-04-12T06:04:22.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.491+0000 7f5e63fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e64102cc0 0x7f5e64113760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:22.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.491+0000 7f5e63fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e64102cc0 0x7f5e64113760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51652/0 (socket says 172.21.15.32:51652) 2024-04-12T06:04:22.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.491+0000 7f5e63fff700 1 -- 172.21.15.32:0/3761005694 learned_addr learned my addr 172.21.15.32:0/3761005694 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:22.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.491+0000 7f5e637fe700 1 --2- 172.21.15.32:0/3761005694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e64103660 0x7f5e64113ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:22.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.491+0000 7f5e63fff700 1 -- 172.21.15.32:0/3761005694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e64103660 msgr2=0x7f5e64113ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:22.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.491+0000 7f5e63fff700 1 --2- 172.21.15.32:0/3761005694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e64103660 0x7f5e64113ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:22.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.491+0000 7f5e63fff700 1 -- 172.21.15.32:0/3761005694 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e4c005040 con 0x7f5e64102cc0 2024-04-12T06:04:22.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.491+0000 7f5e637fe700 1 --2- 172.21.15.32:0/3761005694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e64103660 0x7f5e64113ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T06:04:22.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.491+0000 7f5e63fff700 1 --2- 172.21.15.32:0/3761005694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e64102cc0 0x7f5e64113760 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009180 tx=0x7f5e4c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:22.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.491+0000 7f5e60ff9700 1 -- 172.21.15.32:0/3761005694 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e4c00e040 con 0x7f5e64102cc0 2024-04-12T06:04:22.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.492+0000 7f5e60ff9700 1 -- 172.21.15.32:0/3761005694 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e4c004290 con 0x7f5e64102cc0 2024-04-12T06:04:22.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.492+0000 7f5e6a375700 1 -- 172.21.15.32:0/3761005694 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e6410db70 con 0x7f5e64102cc0 2024-04-12T06:04:22.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.492+0000 7f5e60ff9700 1 -- 172.21.15.32:0/3761005694 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e4c01c3f0 con 0x7f5e64102cc0 2024-04-12T06:04:22.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.492+0000 7f5e6a375700 1 -- 172.21.15.32:0/3761005694 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e6410e130 con 0x7f5e64102cc0 2024-04-12T06:04:22.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.492+0000 7f5e6a375700 1 -- 172.21.15.32:0/3761005694 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e640fe410 con 0x7f5e64102cc0 2024-04-12T06:04:22.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.493+0000 7f5e60ff9700 1 -- 172.21.15.32:0/3761005694 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f5e4c013790 con 0x7f5e64102cc0 2024-04-12T06:04:22.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.493+0000 7f5e60ff9700 1 --2- 172.21.15.32:0/3761005694 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5e5006cc20 0x7f5e5006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:22.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.494+0000 7f5e60ff9700 1 -- 172.21.15.32:0/3761005694 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f5e4c08d400 con 0x7f5e64102cc0 2024-04-12T06:04:22.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.494+0000 7f5e637fe700 1 --2- 172.21.15.32:0/3761005694 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5e5006cc20 0x7f5e5006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:22.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.494+0000 7f5e637fe700 1 --2- 172.21.15.32:0/3761005694 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5e5006cc20 0x7f5e5006f0d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f5e54006fd0 tx=0x7f5e54008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:22.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.498+0000 7f5e60ff9700 1 -- 172.21.15.32:0/3761005694 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5e4c05add0 con 0x7f5e64102cc0 2024-04-12T06:04:22.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.749+0000 7f5e6a375700 1 -- 172.21.15.32:0/3761005694 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 24, "format": "json"} v 0) v1 -- 0x7f5e640640e0 con 0x7f5e64102cc0 2024-04-12T06:04:22.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.750+0000 7f5e60ff9700 1 -- 172.21.15.32:0/3761005694 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 24, "format": "json"}]=0 dumped fsmap epoch 24 v40) v1 ==== 107+0+3260 (secure 0 0 0) 0x7f5e4c013420 con 0x7f5e64102cc0 2024-04-12T06:04:22.745 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:04:22.745 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34372,"name":"cephfs.smithi032.jehaaj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6827/268850045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":268850045},{"type":"v1","addr":"172.21.15.32:6827","nonce":268850045}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44275,"name":"cephfs.smithi067.ymvqih","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6827/3573641538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3573641538},{"type":"v1","addr":"172.21.15.67:6827","nonce":3573641538}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-12T05:34:22.641718+0000","modified":"2024-04-12T05:46:10.668569+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T06:04:22.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.752+0000 7f5e6a375700 1 -- 172.21.15.32:0/3761005694 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5e5006cc20 msgr2=0x7f5e5006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:22.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.752+0000 7f5e6a375700 1 --2- 172.21.15.32:0/3761005694 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5e5006cc20 0x7f5e5006f0d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f5e54006fd0 tx=0x7f5e54008040 comp rx=0 tx=0).stop 2024-04-12T06:04:22.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.752+0000 7f5e6a375700 1 -- 172.21.15.32:0/3761005694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e64102cc0 msgr2=0x7f5e64113760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:22.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.752+0000 7f5e6a375700 1 --2- 172.21.15.32:0/3761005694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e64102cc0 0x7f5e64113760 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009180 tx=0x7f5e4c009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:22.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.752+0000 7f5e6a375700 1 -- 172.21.15.32:0/3761005694 shutdown_connections 2024-04-12T06:04:22.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.752+0000 7f5e6a375700 1 --2- 172.21.15.32:0/3761005694 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f5e5006cc20 0x7f5e5006f0d0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:22.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.752+0000 7f5e6a375700 1 --2- 172.21.15.32:0/3761005694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e64102cc0 0x7f5e64113760 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:22.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.752+0000 7f5e6a375700 1 --2- 172.21.15.32:0/3761005694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e64103660 0x7f5e64113ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:22.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.752+0000 7f5e6a375700 1 -- 172.21.15.32:0/3761005694 >> 172.21.15.32:0/3761005694 conn(0x7f5e640f8830 msgr2=0x7f5e640f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:22.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.752+0000 7f5e6a375700 1 -- 172.21.15.32:0/3761005694 shutdown_connections 2024-04-12T06:04:22.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.752+0000 7f5e6a375700 1 -- 172.21.15.32:0/3761005694 wait complete. 2024-04-12T06:04:22.749 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 24 2024-04-12T06:04:22.760 DEBUG:tasks.fs:max_mds not reduced in epoch 24 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-12T06:04:22.760 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 24 2024-04-12T06:04:22.760 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 25 2024-04-12T06:04:22.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.888+0000 7f774d6aa700 1 -- 172.21.15.32:0/2432242880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7748101bf0 msgr2=0x7f7748101fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:22.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.888+0000 7f774d6aa700 1 --2- 172.21.15.32:0/2432242880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7748101bf0 0x7f7748101fc0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f7738004940 tx=0x7f7738009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:22.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.888+0000 7f774d6aa700 1 -- 172.21.15.32:0/2432242880 shutdown_connections 2024-04-12T06:04:22.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.888+0000 7f774d6aa700 1 --2- 172.21.15.32:0/2432242880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7748102590 0x7f77480fbc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:22.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.888+0000 7f774d6aa700 1 --2- 172.21.15.32:0/2432242880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7748101bf0 0x7f7748101fc0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:22.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.888+0000 7f774d6aa700 1 -- 172.21.15.32:0/2432242880 >> 172.21.15.32:0/2432242880 conn(0x7f77480f7760 msgr2=0x7f77480f9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:22.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.888+0000 7f774d6aa700 1 -- 172.21.15.32:0/2432242880 shutdown_connections 2024-04-12T06:04:22.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.888+0000 7f774d6aa700 1 -- 172.21.15.32:0/2432242880 wait complete. 2024-04-12T06:04:22.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.888+0000 7f774d6aa700 1 Processor -- start 2024-04-12T06:04:22.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.888+0000 7f774d6aa700 1 -- start start 2024-04-12T06:04:22.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.889+0000 7f774d6aa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7748101bf0 0x7f7748115970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:22.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.889+0000 7f774d6aa700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7748102590 0x7f7748115eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:22.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.889+0000 7f774d6aa700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7748116500 con 0x7f7748101bf0 2024-04-12T06:04:22.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.889+0000 7f774d6aa700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f774810fb00 con 0x7f7748102590 2024-04-12T06:04:22.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.889+0000 7f7746ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7748101bf0 0x7f7748115970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:22.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.889+0000 7f7746ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7748101bf0 0x7f7748115970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51680/0 (socket says 172.21.15.32:51680) 2024-04-12T06:04:22.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.889+0000 7f7746ffd700 1 -- 172.21.15.32:0/1427852664 learned_addr learned my addr 172.21.15.32:0/1427852664 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:22.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.889+0000 7f77467fc700 1 --2- 172.21.15.32:0/1427852664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7748102590 0x7f7748115eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:22.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.889+0000 7f7746ffd700 1 -- 172.21.15.32:0/1427852664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7748102590 msgr2=0x7f7748115eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:22.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.889+0000 7f7746ffd700 1 --2- 172.21.15.32:0/1427852664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7748102590 0x7f7748115eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:22.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.889+0000 7f7746ffd700 1 -- 172.21.15.32:0/1427852664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7738005040 con 0x7f7748101bf0 2024-04-12T06:04:22.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.889+0000 7f77467fc700 1 --2- 172.21.15.32:0/1427852664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7748102590 0x7f7748115eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T06:04:22.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.889+0000 7f7746ffd700 1 --2- 172.21.15.32:0/1427852664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7748101bf0 0x7f7748115970 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f7738009360 tx=0x7f7738009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:22.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.889+0000 7f772f7fe700 1 -- 172.21.15.32:0/1427852664 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f773800e040 con 0x7f7748101bf0 2024-04-12T06:04:22.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.889+0000 7f772f7fe700 1 -- 172.21.15.32:0/1427852664 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7738013420 con 0x7f7748101bf0 2024-04-12T06:04:22.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.889+0000 7f774d6aa700 1 -- 172.21.15.32:0/1427852664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f774810fd80 con 0x7f7748101bf0 2024-04-12T06:04:22.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.889+0000 7f772f7fe700 1 -- 172.21.15.32:0/1427852664 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f773801c5f0 con 0x7f7748101bf0 2024-04-12T06:04:22.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.889+0000 7f774d6aa700 1 -- 172.21.15.32:0/1427852664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7748110340 con 0x7f7748101bf0 2024-04-12T06:04:22.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.890+0000 7f774d6aa700 1 -- 172.21.15.32:0/1427852664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77480fd340 con 0x7f7748101bf0 2024-04-12T06:04:22.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.891+0000 7f772f7fe700 1 -- 172.21.15.32:0/1427852664 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f7738013590 con 0x7f7748101bf0 2024-04-12T06:04:22.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.891+0000 7f772f7fe700 1 --2- 172.21.15.32:0/1427852664 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f773006cc20 0x7f773006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:22.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.891+0000 7f772f7fe700 1 -- 172.21.15.32:0/1427852664 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f773808d210 con 0x7f7748101bf0 2024-04-12T06:04:22.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.891+0000 7f77467fc700 1 --2- 172.21.15.32:0/1427852664 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f773006cc20 0x7f773006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:22.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.892+0000 7f77467fc700 1 --2- 172.21.15.32:0/1427852664 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f773006cc20 0x7f773006f0d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f773c009fd0 tx=0x7f773c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:22.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:22.896+0000 7f772f7fe700 1 -- 172.21.15.32:0/1427852664 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f773805ad20 con 0x7f7748101bf0 2024-04-12T06:04:23.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:22 smithi032 ceph-mon[146817]: pgmap v716: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:04:23.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:22 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/517296408' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-04-12T06:04:23.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:22 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/4229939736' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-04-12T06:04:23.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:22 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3761005694' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-04-12T06:04:23.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.150+0000 7f774d6aa700 1 -- 172.21.15.32:0/1427852664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 25, "format": "json"} v 0) v1 -- 0x7f7748108ba0 con 0x7f7748101bf0 2024-04-12T06:04:23.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.150+0000 7f772f7fe700 1 -- 172.21.15.32:0/1427852664 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 25, "format": "json"}]=0 dumped fsmap epoch 25 v40) v1 ==== 107+0+4815 (secure 0 0 0) 0x7f773805a8b0 con 0x7f7748101bf0 2024-04-12T06:04:23.147 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:04:23.147 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34372,"name":"cephfs.smithi032.jehaaj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6827/268850045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":268850045},{"type":"v1","addr":"172.21.15.32:6827","nonce":268850045}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":34380,"name":"cephfs.smithi032.pndtry","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6829/2545444205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":2545444205},{"type":"v1","addr":"172.21.15.32:6829","nonce":2545444205}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":44275,"name":"cephfs.smithi067.ymvqih","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6827/3573641538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3573641538},{"type":"v1","addr":"172.21.15.67:6827","nonce":3573641538}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44285,"name":"cephfs.smithi067.zesmgy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6825/973024107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":973024107},{"type":"v1","addr":"172.21.15.67:6825","nonce":973024107}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-12T05:34:22.641718+0000","modified":"2024-04-12T05:46:10.668569+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T06:04:23.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.152+0000 7f774d6aa700 1 -- 172.21.15.32:0/1427852664 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f773006cc20 msgr2=0x7f773006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:23.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.152+0000 7f774d6aa700 1 --2- 172.21.15.32:0/1427852664 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f773006cc20 0x7f773006f0d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f773c009fd0 tx=0x7f773c00b040 comp rx=0 tx=0).stop 2024-04-12T06:04:23.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.152+0000 7f774d6aa700 1 -- 172.21.15.32:0/1427852664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7748101bf0 msgr2=0x7f7748115970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:23.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.152+0000 7f774d6aa700 1 --2- 172.21.15.32:0/1427852664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7748101bf0 0x7f7748115970 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f7738009360 tx=0x7f7738009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:23.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.152+0000 7f774d6aa700 1 -- 172.21.15.32:0/1427852664 shutdown_connections 2024-04-12T06:04:23.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.152+0000 7f774d6aa700 1 --2- 172.21.15.32:0/1427852664 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f773006cc20 0x7f773006f0d0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:23.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.152+0000 7f774d6aa700 1 --2- 172.21.15.32:0/1427852664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7748101bf0 0x7f7748115970 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:23.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.152+0000 7f774d6aa700 1 --2- 172.21.15.32:0/1427852664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7748102590 0x7f7748115eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:23.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.153+0000 7f774d6aa700 1 -- 172.21.15.32:0/1427852664 >> 172.21.15.32:0/1427852664 conn(0x7f77480f7760 msgr2=0x7f77480f8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:23.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.153+0000 7f774d6aa700 1 -- 172.21.15.32:0/1427852664 shutdown_connections 2024-04-12T06:04:23.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.153+0000 7f774d6aa700 1 -- 172.21.15.32:0/1427852664 wait complete. 2024-04-12T06:04:23.149 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 25 2024-04-12T06:04:23.160 DEBUG:tasks.fs:max_mds not reduced in epoch 25 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-12T06:04:23.160 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 25 2024-04-12T06:04:23.160 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 26 2024-04-12T06:04:23.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:22 smithi067 ceph-mon[112979]: pgmap v716: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:04:23.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:22 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/517296408' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-04-12T06:04:23.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:22 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/4229939736' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-04-12T06:04:23.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:22 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3761005694' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-04-12T06:04:23.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.287+0000 7f4134748700 1 -- 172.21.15.32:0/415587130 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f412c102570 msgr2=0x7f412c0fbbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:23.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.287+0000 7f4134748700 1 --2- 172.21.15.32:0/415587130 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f412c102570 0x7f412c0fbbe0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f411c004940 tx=0x7f411c009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:23.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.288+0000 7f4134748700 1 -- 172.21.15.32:0/415587130 shutdown_connections 2024-04-12T06:04:23.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.288+0000 7f4134748700 1 --2- 172.21.15.32:0/415587130 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f412c102570 0x7f412c0fbbe0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:23.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.288+0000 7f4134748700 1 --2- 172.21.15.32:0/415587130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f412c101bd0 0x7f412c101fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:23.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.288+0000 7f4134748700 1 -- 172.21.15.32:0/415587130 >> 172.21.15.32:0/415587130 conn(0x7f412c0f7700 msgr2=0x7f412c0f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:23.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.288+0000 7f4134748700 1 -- 172.21.15.32:0/415587130 shutdown_connections 2024-04-12T06:04:23.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.288+0000 7f4134748700 1 -- 172.21.15.32:0/415587130 wait complete. 2024-04-12T06:04:23.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.288+0000 7f4134748700 1 Processor -- start 2024-04-12T06:04:23.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.288+0000 7f4134748700 1 -- start start 2024-04-12T06:04:23.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.288+0000 7f4134748700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f412c101bd0 0x7f412c112690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:23.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.288+0000 7f4134748700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f412c102570 0x7f412c112c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:23.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.288+0000 7f4134748700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412c113260 con 0x7f412c101bd0 2024-04-12T06:04:23.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.288+0000 7f4134748700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412c10c820 con 0x7f412c102570 2024-04-12T06:04:23.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.289+0000 7f41324e4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f412c101bd0 0x7f412c112690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:23.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.289+0000 7f41324e4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f412c101bd0 0x7f412c112690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51690/0 (socket says 172.21.15.32:51690) 2024-04-12T06:04:23.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.289+0000 7f41324e4700 1 -- 172.21.15.32:0/2918960495 learned_addr learned my addr 172.21.15.32:0/2918960495 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:23.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.289+0000 7f4131ce3700 1 --2- 172.21.15.32:0/2918960495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f412c102570 0x7f412c112c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:23.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.289+0000 7f41324e4700 1 -- 172.21.15.32:0/2918960495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f412c102570 msgr2=0x7f412c112c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:23.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.289+0000 7f41324e4700 1 --2- 172.21.15.32:0/2918960495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f412c102570 0x7f412c112c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:23.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.289+0000 7f41324e4700 1 -- 172.21.15.32:0/2918960495 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f411c005040 con 0x7f412c101bd0 2024-04-12T06:04:23.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.289+0000 7f41324e4700 1 --2- 172.21.15.32:0/2918960495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f412c101bd0 0x7f412c112690 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f412800c830 tx=0x7f4128004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:23.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.289+0000 7f4122ffd700 1 -- 172.21.15.32:0/2918960495 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f412800a760 con 0x7f412c101bd0 2024-04-12T06:04:23.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.289+0000 7f4122ffd700 1 -- 172.21.15.32:0/2918960495 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f412800ad70 con 0x7f412c101bd0 2024-04-12T06:04:23.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.289+0000 7f4134748700 1 -- 172.21.15.32:0/2918960495 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f412c10cab0 con 0x7f412c101bd0 2024-04-12T06:04:23.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.289+0000 7f4122ffd700 1 -- 172.21.15.32:0/2918960495 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41280113a0 con 0x7f412c101bd0 2024-04-12T06:04:23.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.289+0000 7f4134748700 1 -- 172.21.15.32:0/2918960495 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f412c10cff0 con 0x7f412c101bd0 2024-04-12T06:04:23.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.290+0000 7f4134748700 1 -- 172.21.15.32:0/2918960495 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f412c0fd320 con 0x7f412c101bd0 2024-04-12T06:04:23.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.290+0000 7f4122ffd700 1 -- 172.21.15.32:0/2918960495 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4128012400 con 0x7f412c101bd0 2024-04-12T06:04:23.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.291+0000 7f4122ffd700 1 --2- 172.21.15.32:0/2918960495 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f411806cc20 0x7f411806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:23.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.291+0000 7f4122ffd700 1 -- 172.21.15.32:0/2918960495 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f412808c070 con 0x7f412c101bd0 2024-04-12T06:04:23.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.291+0000 7f4131ce3700 1 --2- 172.21.15.32:0/2918960495 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f411806cc20 0x7f411806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:23.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.291+0000 7f4131ce3700 1 --2- 172.21.15.32:0/2918960495 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f411806cc20 0x7f411806f0d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f411c00b3b0 tx=0x7f411c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:23.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.296+0000 7f4122ffd700 1 -- 172.21.15.32:0/2918960495 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4128059b80 con 0x7f412c101bd0 2024-04-12T06:04:23.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.551+0000 7f4134748700 1 -- 172.21.15.32:0/2918960495 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 26, "format": "json"} v 0) v1 -- 0x7f412c10d2c0 con 0x7f412c101bd0 2024-04-12T06:04:23.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.552+0000 7f4122ffd700 1 -- 172.21.15.32:0/2918960495 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 26, "format": "json"}]=0 dumped fsmap epoch 26 v40) v1 ==== 107+0+4039 (secure 0 0 0) 0x7f4128059710 con 0x7f412c101bd0 2024-04-12T06:04:23.547 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:04:23.547 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34380,"name":"cephfs.smithi032.pndtry","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6829/2545444205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":2545444205},{"type":"v1","addr":"172.21.15.32:6829","nonce":2545444205}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":44275,"name":"cephfs.smithi067.ymvqih","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6827/3573641538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3573641538},{"type":"v1","addr":"172.21.15.67:6827","nonce":3573641538}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44285,"name":"cephfs.smithi067.zesmgy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6825/973024107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":973024107},{"type":"v1","addr":"172.21.15.67:6825","nonce":973024107}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-12T05:34:22.641718+0000","modified":"2024-04-12T05:46:10.668569+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T06:04:23.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.554+0000 7f4134748700 1 -- 172.21.15.32:0/2918960495 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f411806cc20 msgr2=0x7f411806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:23.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.554+0000 7f4134748700 1 --2- 172.21.15.32:0/2918960495 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f411806cc20 0x7f411806f0d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f411c00b3b0 tx=0x7f411c009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:23.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.554+0000 7f4134748700 1 -- 172.21.15.32:0/2918960495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f412c101bd0 msgr2=0x7f412c112690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:23.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.554+0000 7f4134748700 1 --2- 172.21.15.32:0/2918960495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f412c101bd0 0x7f412c112690 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f412800c830 tx=0x7f4128004470 comp rx=0 tx=0).stop 2024-04-12T06:04:23.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.554+0000 7f4134748700 1 -- 172.21.15.32:0/2918960495 shutdown_connections 2024-04-12T06:04:23.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.554+0000 7f4134748700 1 --2- 172.21.15.32:0/2918960495 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f411806cc20 0x7f411806f0d0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:23.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.554+0000 7f4134748700 1 --2- 172.21.15.32:0/2918960495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f412c101bd0 0x7f412c112690 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:23.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.554+0000 7f4134748700 1 --2- 172.21.15.32:0/2918960495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f412c102570 0x7f412c112c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:23.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.554+0000 7f4134748700 1 -- 172.21.15.32:0/2918960495 >> 172.21.15.32:0/2918960495 conn(0x7f412c0f7700 msgr2=0x7f412c0f8860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:23.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.554+0000 7f4134748700 1 -- 172.21.15.32:0/2918960495 shutdown_connections 2024-04-12T06:04:23.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.554+0000 7f4134748700 1 -- 172.21.15.32:0/2918960495 wait complete. 2024-04-12T06:04:23.550 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 26 2024-04-12T06:04:23.561 DEBUG:tasks.fs:max_mds not reduced in epoch 26 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-12T06:04:23.561 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 26 2024-04-12T06:04:23.561 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 27 2024-04-12T06:04:23.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.689+0000 7f71ae8f4700 1 -- 172.21.15.32:0/3200766840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71a8101c00 msgr2=0x7f71a8101fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:23.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.689+0000 7f71ae8f4700 1 --2- 172.21.15.32:0/3200766840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71a8101c00 0x7f71a8101fd0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f7198004940 tx=0x7f7198009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:23.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.689+0000 7f71ae8f4700 1 -- 172.21.15.32:0/3200766840 shutdown_connections 2024-04-12T06:04:23.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.689+0000 7f71ae8f4700 1 --2- 172.21.15.32:0/3200766840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71a81025a0 0x7f71a80fbc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:23.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.689+0000 7f71ae8f4700 1 --2- 172.21.15.32:0/3200766840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71a8101c00 0x7f71a8101fd0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:23.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.689+0000 7f71ae8f4700 1 -- 172.21.15.32:0/3200766840 >> 172.21.15.32:0/3200766840 conn(0x7f71a80f7750 msgr2=0x7f71a80f9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:23.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.689+0000 7f71ae8f4700 1 -- 172.21.15.32:0/3200766840 shutdown_connections 2024-04-12T06:04:23.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.689+0000 7f71ae8f4700 1 -- 172.21.15.32:0/3200766840 wait complete. 2024-04-12T06:04:23.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.689+0000 7f71ae8f4700 1 Processor -- start 2024-04-12T06:04:23.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.690+0000 7f71ae8f4700 1 -- start start 2024-04-12T06:04:23.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.690+0000 7f71ae8f4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71a8101c00 0x7f71a810e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:23.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.690+0000 7f71ae8f4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71a81025a0 0x7f71a810ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:23.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.690+0000 7f71ae8f4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a810f350 con 0x7f71a81025a0 2024-04-12T06:04:23.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.690+0000 7f71ae8f4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a810f490 con 0x7f71a8101c00 2024-04-12T06:04:23.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.690+0000 7f71a77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71a81025a0 0x7f71a810ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:23.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.690+0000 7f71a77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71a81025a0 0x7f71a810ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51696/0 (socket says 172.21.15.32:51696) 2024-04-12T06:04:23.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.690+0000 7f71a77fe700 1 -- 172.21.15.32:0/186485425 learned_addr learned my addr 172.21.15.32:0/186485425 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:23.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.690+0000 7f71a7fff700 1 --2- 172.21.15.32:0/186485425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71a8101c00 0x7f71a810e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:23.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.690+0000 7f71a77fe700 1 -- 172.21.15.32:0/186485425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71a8101c00 msgr2=0x7f71a810e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:23.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.690+0000 7f71a77fe700 1 --2- 172.21.15.32:0/186485425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71a8101c00 0x7f71a810e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:23.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.690+0000 7f71a77fe700 1 -- 172.21.15.32:0/186485425 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7198005040 con 0x7f71a81025a0 2024-04-12T06:04:23.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.690+0000 7f71a77fe700 1 --2- 172.21.15.32:0/186485425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71a81025a0 0x7f71a810ee10 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f719c00eb40 tx=0x7f719c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:23.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.690+0000 7f71a4ff9700 1 -- 172.21.15.32:0/186485425 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f719c007cc0 con 0x7f71a81025a0 2024-04-12T06:04:23.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.691+0000 7f71a4ff9700 1 -- 172.21.15.32:0/186485425 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f719c00f040 con 0x7f71a81025a0 2024-04-12T06:04:23.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.691+0000 7f71ae8f4700 1 -- 172.21.15.32:0/186485425 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71a810f710 con 0x7f71a81025a0 2024-04-12T06:04:23.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.691+0000 7f71a4ff9700 1 -- 172.21.15.32:0/186485425 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f719c00c8f0 con 0x7f71a81025a0 2024-04-12T06:04:23.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.691+0000 7f71ae8f4700 1 -- 172.21.15.32:0/186485425 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71a8117c40 con 0x7f71a81025a0 2024-04-12T06:04:23.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.691+0000 7f71ae8f4700 1 -- 172.21.15.32:0/186485425 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71a80fd350 con 0x7f71a81025a0 2024-04-12T06:04:23.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.692+0000 7f71a4ff9700 1 -- 172.21.15.32:0/186485425 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f719c00cb60 con 0x7f71a81025a0 2024-04-12T06:04:23.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.692+0000 7f71a4ff9700 1 --2- 172.21.15.32:0/186485425 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f719006cc20 0x7f719006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:23.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.692+0000 7f71a4ff9700 1 -- 172.21.15.32:0/186485425 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f719c08bb30 con 0x7f71a81025a0 2024-04-12T06:04:23.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.692+0000 7f71a7fff700 1 --2- 172.21.15.32:0/186485425 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f719006cc20 0x7f719006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:23.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.693+0000 7f71a7fff700 1 --2- 172.21.15.32:0/186485425 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f719006cc20 0x7f719006f0d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f719800b0d0 tx=0x7f7198009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.697+0000 7f71a4ff9700 1 -- 172.21.15.32:0/186485425 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f719c055b40 con 0x7f71a81025a0 2024-04-12T06:04:23.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.949+0000 7f71ae8f4700 1 -- 172.21.15.32:0/186485425 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 27, "format": "json"} v 0) v1 -- 0x7f71a8108be0 con 0x7f71a81025a0 2024-04-12T06:04:23.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.949+0000 7f71a4ff9700 1 -- 172.21.15.32:0/186485425 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 27, "format": "json"}]=0 dumped fsmap epoch 27 v40) v1 ==== 107+0+4818 (secure 0 0 0) 0x7f719c059160 con 0x7f71a81025a0 2024-04-12T06:04:23.945 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:04:23.945 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34380,"name":"cephfs.smithi032.pndtry","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6829/2545444205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":2545444205},{"type":"v1","addr":"172.21.15.32:6829","nonce":2545444205}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":34384,"name":"cephfs.smithi032.jehaaj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6827/1884332935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1884332935},{"type":"v1","addr":"172.21.15.32:6827","nonce":1884332935}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":44275,"name":"cephfs.smithi067.ymvqih","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6827/3573641538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3573641538},{"type":"v1","addr":"172.21.15.67:6827","nonce":3573641538}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44285,"name":"cephfs.smithi067.zesmgy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6825/973024107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":973024107},{"type":"v1","addr":"172.21.15.67:6825","nonce":973024107}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-12T05:34:22.641718+0000","modified":"2024-04-12T05:46:10.668569+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T06:04:23.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.952+0000 7f71ae8f4700 1 -- 172.21.15.32:0/186485425 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f719006cc20 msgr2=0x7f719006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:23.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.952+0000 7f71ae8f4700 1 --2- 172.21.15.32:0/186485425 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f719006cc20 0x7f719006f0d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f719800b0d0 tx=0x7f7198009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:23.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.952+0000 7f71ae8f4700 1 -- 172.21.15.32:0/186485425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71a81025a0 msgr2=0x7f71a810ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:23.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.952+0000 7f71ae8f4700 1 --2- 172.21.15.32:0/186485425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71a81025a0 0x7f71a810ee10 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f719c00eb40 tx=0x7f719c00b040 comp rx=0 tx=0).stop 2024-04-12T06:04:23.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.952+0000 7f71ae8f4700 1 -- 172.21.15.32:0/186485425 shutdown_connections 2024-04-12T06:04:23.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.952+0000 7f71ae8f4700 1 --2- 172.21.15.32:0/186485425 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f719006cc20 0x7f719006f0d0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:23.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.952+0000 7f71ae8f4700 1 --2- 172.21.15.32:0/186485425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71a8101c00 0x7f71a810e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:23.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.952+0000 7f71ae8f4700 1 --2- 172.21.15.32:0/186485425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71a81025a0 0x7f71a810ee10 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:23.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.952+0000 7f71ae8f4700 1 -- 172.21.15.32:0/186485425 >> 172.21.15.32:0/186485425 conn(0x7f71a80f7750 msgr2=0x7f71a80f87a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:23.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.952+0000 7f71ae8f4700 1 -- 172.21.15.32:0/186485425 shutdown_connections 2024-04-12T06:04:23.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:23.952+0000 7f71ae8f4700 1 -- 172.21.15.32:0/186485425 wait complete. 2024-04-12T06:04:23.948 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 27 2024-04-12T06:04:23.959 DEBUG:tasks.fs:max_mds not reduced in epoch 27 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-12T06:04:23.959 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 27 2024-04-12T06:04:23.959 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 28 2024-04-12T06:04:24.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.088+0000 7f8d779d5700 1 -- 172.21.15.32:0/2655922100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d70101be0 msgr2=0x7f8d70101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:24.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.088+0000 7f8d779d5700 1 --2- 172.21.15.32:0/2655922100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d70101be0 0x7f8d70101fb0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c004940 tx=0x7f8d6c009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:24.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.088+0000 7f8d779d5700 1 -- 172.21.15.32:0/2655922100 shutdown_connections 2024-04-12T06:04:24.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.088+0000 7f8d779d5700 1 --2- 172.21.15.32:0/2655922100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d70102580 0x7f8d700fbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:24.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.088+0000 7f8d779d5700 1 --2- 172.21.15.32:0/2655922100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d70101be0 0x7f8d70101fb0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:24.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.088+0000 7f8d779d5700 1 -- 172.21.15.32:0/2655922100 >> 172.21.15.32:0/2655922100 conn(0x7f8d70072ad0 msgr2=0x7f8d70072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:24.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.088+0000 7f8d779d5700 1 -- 172.21.15.32:0/2655922100 shutdown_connections 2024-04-12T06:04:24.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.088+0000 7f8d779d5700 1 -- 172.21.15.32:0/2655922100 wait complete. 2024-04-12T06:04:24.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.089+0000 7f8d779d5700 1 Processor -- start 2024-04-12T06:04:24.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.089+0000 7f8d779d5700 1 -- start start 2024-04-12T06:04:24.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.089+0000 7f8d779d5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d70101be0 0x7f8d701126c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:24.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.089+0000 7f8d779d5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d70102580 0x7f8d70112c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:24.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.089+0000 7f8d779d5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d70113290 con 0x7f8d70101be0 2024-04-12T06:04:24.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.089+0000 7f8d779d5700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7010c850 con 0x7f8d70102580 2024-04-12T06:04:24.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.089+0000 7f8d75771700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d70101be0 0x7f8d701126c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:24.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.089+0000 7f8d75771700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d70101be0 0x7f8d701126c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51714/0 (socket says 172.21.15.32:51714) 2024-04-12T06:04:24.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.089+0000 7f8d75771700 1 -- 172.21.15.32:0/2947595834 learned_addr learned my addr 172.21.15.32:0/2947595834 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:24.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.089+0000 7f8d74f70700 1 --2- 172.21.15.32:0/2947595834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d70102580 0x7f8d70112c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:24.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.090+0000 7f8d75771700 1 -- 172.21.15.32:0/2947595834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d70102580 msgr2=0x7f8d70112c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:24.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.090+0000 7f8d75771700 1 --2- 172.21.15.32:0/2947595834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d70102580 0x7f8d70112c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:24.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.090+0000 7f8d75771700 1 -- 172.21.15.32:0/2947595834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d6c005040 con 0x7f8d70101be0 2024-04-12T06:04:24.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.090+0000 7f8d75771700 1 --2- 172.21.15.32:0/2947595834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d70101be0 0x7f8d701126c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c00b3b0 tx=0x7f8d6c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:24.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.090+0000 7f8d65ffb700 1 -- 172.21.15.32:0/2947595834 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d6c00e040 con 0x7f8d70101be0 2024-04-12T06:04:24.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.090+0000 7f8d779d5700 1 -- 172.21.15.32:0/2947595834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d7010ca80 con 0x7f8d70101be0 2024-04-12T06:04:24.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.090+0000 7f8d65ffb700 1 -- 172.21.15.32:0/2947595834 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d6c013420 con 0x7f8d70101be0 2024-04-12T06:04:24.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.090+0000 7f8d65ffb700 1 -- 172.21.15.32:0/2947595834 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d6c01c610 con 0x7f8d70101be0 2024-04-12T06:04:24.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.090+0000 7f8d779d5700 1 -- 172.21.15.32:0/2947595834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d7010cfc0 con 0x7f8d70101be0 2024-04-12T06:04:24.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.091+0000 7f8d779d5700 1 -- 172.21.15.32:0/2947595834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d700fd330 con 0x7f8d70101be0 2024-04-12T06:04:24.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.092+0000 7f8d65ffb700 1 -- 172.21.15.32:0/2947595834 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8d6c013a00 con 0x7f8d70101be0 2024-04-12T06:04:24.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.092+0000 7f8d65ffb700 1 --2- 172.21.15.32:0/2947595834 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f8d5c06cc20 0x7f8d5c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:24.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.092+0000 7f8d65ffb700 1 -- 172.21.15.32:0/2947595834 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f8d6c08d1f0 con 0x7f8d70101be0 2024-04-12T06:04:24.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.092+0000 7f8d74f70700 1 --2- 172.21.15.32:0/2947595834 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f8d5c06cc20 0x7f8d5c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:24.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.093+0000 7f8d74f70700 1 --2- 172.21.15.32:0/2947595834 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f8d5c06cc20 0x7f8d5c06f0d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8d7010df60 tx=0x7f8d60002040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:24.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.097+0000 7f8d65ffb700 1 -- 172.21.15.32:0/2947595834 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8d6c05ac60 con 0x7f8d70101be0 2024-04-12T06:04:24.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:23 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1427852664' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-04-12T06:04:24.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:23 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/2918960495' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-04-12T06:04:24.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:23 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1427852664' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-04-12T06:04:24.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:23 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/2918960495' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-04-12T06:04:24.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.354+0000 7f8d779d5700 1 -- 172.21.15.32:0/2947595834 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 28, "format": "json"} v 0) v1 -- 0x7f8d7010dd10 con 0x7f8d70101be0 2024-04-12T06:04:24.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.354+0000 7f8d65ffb700 1 -- 172.21.15.32:0/2947595834 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 28, "format": "json"}]=0 dumped fsmap epoch 28 v40) v1 ==== 107+0+4039 (secure 0 0 0) 0x7f8d6c05a7f0 con 0x7f8d70101be0 2024-04-12T06:04:24.350 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:04:24.350 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34384,"name":"cephfs.smithi032.jehaaj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6827/1884332935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1884332935},{"type":"v1","addr":"172.21.15.32:6827","nonce":1884332935}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":44275,"name":"cephfs.smithi067.ymvqih","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6827/3573641538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3573641538},{"type":"v1","addr":"172.21.15.67:6827","nonce":3573641538}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44285,"name":"cephfs.smithi067.zesmgy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6825/973024107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":973024107},{"type":"v1","addr":"172.21.15.67:6825","nonce":973024107}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-12T05:34:22.641718+0000","modified":"2024-04-12T05:46:10.668569+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T06:04:24.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.357+0000 7f8d779d5700 1 -- 172.21.15.32:0/2947595834 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f8d5c06cc20 msgr2=0x7f8d5c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:24.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.357+0000 7f8d779d5700 1 --2- 172.21.15.32:0/2947595834 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f8d5c06cc20 0x7f8d5c06f0d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8d7010df60 tx=0x7f8d60002040 comp rx=0 tx=0).stop 2024-04-12T06:04:24.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.357+0000 7f8d779d5700 1 -- 172.21.15.32:0/2947595834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d70101be0 msgr2=0x7f8d701126c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:24.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.357+0000 7f8d779d5700 1 --2- 172.21.15.32:0/2947595834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d70101be0 0x7f8d701126c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c00b3b0 tx=0x7f8d6c009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:24.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.357+0000 7f8d779d5700 1 -- 172.21.15.32:0/2947595834 shutdown_connections 2024-04-12T06:04:24.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.357+0000 7f8d779d5700 1 --2- 172.21.15.32:0/2947595834 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f8d5c06cc20 0x7f8d5c06f0d0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:24.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.357+0000 7f8d779d5700 1 --2- 172.21.15.32:0/2947595834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d70101be0 0x7f8d701126c0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.357+0000 7f8d779d5700 1 --2- 172.21.15.32:0/2947595834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d70102580 0x7f8d70112c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.357+0000 7f8d779d5700 1 -- 172.21.15.32:0/2947595834 >> 172.21.15.32:0/2947595834 conn(0x7f8d70072ad0 msgr2=0x7f8d70104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.357+0000 7f8d779d5700 1 -- 172.21.15.32:0/2947595834 shutdown_connections 2024-04-12T06:04:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.357+0000 7f8d779d5700 1 -- 172.21.15.32:0/2947595834 wait complete. 2024-04-12T06:04:24.353 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-04-12T06:04:24.364 DEBUG:tasks.fs:max_mds not reduced in epoch 28 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-12T06:04:24.364 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 28 2024-04-12T06:04:24.364 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 29 2024-04-12T06:04:24.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.492+0000 7f4e25372700 1 -- 172.21.15.32:0/1275589509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e20103670 msgr2=0x7f4e200fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:24.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.492+0000 7f4e25372700 1 --2- 172.21.15.32:0/1275589509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e20103670 0x7f4e200fcce0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f4e08004940 tx=0x7f4e08009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:24.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.492+0000 7f4e25372700 1 -- 172.21.15.32:0/1275589509 shutdown_connections 2024-04-12T06:04:24.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.492+0000 7f4e25372700 1 --2- 172.21.15.32:0/1275589509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e20103670 0x7f4e200fcce0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:24.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.492+0000 7f4e25372700 1 --2- 172.21.15.32:0/1275589509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e20102cd0 0x7f4e201030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:24.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.492+0000 7f4e25372700 1 -- 172.21.15.32:0/1275589509 >> 172.21.15.32:0/1275589509 conn(0x7f4e200f8820 msgr2=0x7f4e200fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:24.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.492+0000 7f4e25372700 1 -- 172.21.15.32:0/1275589509 shutdown_connections 2024-04-12T06:04:24.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.492+0000 7f4e25372700 1 -- 172.21.15.32:0/1275589509 wait complete. 2024-04-12T06:04:24.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.493+0000 7f4e25372700 1 Processor -- start 2024-04-12T06:04:24.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.493+0000 7f4e25372700 1 -- start start 2024-04-12T06:04:24.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.493+0000 7f4e25372700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e20102cd0 0x7f4e200702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:24.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.493+0000 7f4e25372700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e20103670 0x7f4e20070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:24.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.493+0000 7f4e25372700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e20072210 con 0x7f4e20103670 2024-04-12T06:04:24.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.493+0000 7f4e25372700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e20070d40 con 0x7f4e20102cd0 2024-04-12T06:04:24.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.493+0000 7f4e17fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e20103670 0x7f4e20070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:24.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.493+0000 7f4e17fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e20103670 0x7f4e20070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51728/0 (socket says 172.21.15.32:51728) 2024-04-12T06:04:24.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.493+0000 7f4e17fff700 1 -- 172.21.15.32:0/112753009 learned_addr learned my addr 172.21.15.32:0/112753009 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:24.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.493+0000 7f4e1effd700 1 --2- 172.21.15.32:0/112753009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e20102cd0 0x7f4e200702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:24.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.493+0000 7f4e17fff700 1 -- 172.21.15.32:0/112753009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e20102cd0 msgr2=0x7f4e200702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:24.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.493+0000 7f4e17fff700 1 --2- 172.21.15.32:0/112753009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e20102cd0 0x7f4e200702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:24.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.493+0000 7f4e17fff700 1 -- 172.21.15.32:0/112753009 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e08005040 con 0x7f4e20103670 2024-04-12T06:04:24.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.494+0000 7f4e17fff700 1 --2- 172.21.15.32:0/112753009 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e20103670 0x7f4e20070800 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f4e08009390 tx=0x7f4e08009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:24.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.494+0000 7f4e177fe700 1 -- 172.21.15.32:0/112753009 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e0800e040 con 0x7f4e20103670 2024-04-12T06:04:24.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.494+0000 7f4e177fe700 1 -- 172.21.15.32:0/112753009 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4e08004290 con 0x7f4e20103670 2024-04-12T06:04:24.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.494+0000 7f4e25372700 1 -- 172.21.15.32:0/112753009 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e20070fc0 con 0x7f4e20103670 2024-04-12T06:04:24.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.494+0000 7f4e177fe700 1 -- 172.21.15.32:0/112753009 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e08013e70 con 0x7f4e20103670 2024-04-12T06:04:24.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.494+0000 7f4e25372700 1 -- 172.21.15.32:0/112753009 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e20119ef0 con 0x7f4e20103670 2024-04-12T06:04:24.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.495+0000 7f4e25372700 1 -- 172.21.15.32:0/112753009 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e200fe420 con 0x7f4e20103670 2024-04-12T06:04:24.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.495+0000 7f4e177fe700 1 -- 172.21.15.32:0/112753009 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4e08013690 con 0x7f4e20103670 2024-04-12T06:04:24.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.496+0000 7f4e177fe700 1 --2- 172.21.15.32:0/112753009 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f4e0c06cbc0 0x7f4e0c06f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:24.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.496+0000 7f4e177fe700 1 -- 172.21.15.32:0/112753009 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f4e0808d290 con 0x7f4e20103670 2024-04-12T06:04:24.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.496+0000 7f4e1effd700 1 --2- 172.21.15.32:0/112753009 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f4e0c06cbc0 0x7f4e0c06f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:24.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.496+0000 7f4e1effd700 1 --2- 172.21.15.32:0/112753009 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f4e0c06cbc0 0x7f4e0c06f070 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4e10004f60 tx=0x7f4e1000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:24.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.500+0000 7f4e177fe700 1 -- 172.21.15.32:0/112753009 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4e08057320 con 0x7f4e20103670 2024-04-12T06:04:24.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.756+0000 7f4e25372700 1 -- 172.21.15.32:0/112753009 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 29, "format": "json"} v 0) v1 -- 0x7f4e20109cc0 con 0x7f4e20103670 2024-04-12T06:04:24.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.757+0000 7f4e177fe700 1 -- 172.21.15.32:0/112753009 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 29, "format": "json"}]=0 dumped fsmap epoch 29 v40) v1 ==== 107+0+4818 (secure 0 0 0) 0x7f4e08013420 con 0x7f4e20103670 2024-04-12T06:04:24.752 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:04:24.752 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34384,"name":"cephfs.smithi032.jehaaj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6827/1884332935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1884332935},{"type":"v1","addr":"172.21.15.32:6827","nonce":1884332935}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":34416,"name":"cephfs.smithi032.pndtry","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6829/1671696137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1671696137},{"type":"v1","addr":"172.21.15.32:6829","nonce":1671696137}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":44275,"name":"cephfs.smithi067.ymvqih","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6827/3573641538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3573641538},{"type":"v1","addr":"172.21.15.67:6827","nonce":3573641538}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44285,"name":"cephfs.smithi067.zesmgy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6825/973024107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":973024107},{"type":"v1","addr":"172.21.15.67:6825","nonce":973024107}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-12T05:34:22.641718+0000","modified":"2024-04-12T05:46:10.668569+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T06:04:24.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.759+0000 7f4e25372700 1 -- 172.21.15.32:0/112753009 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f4e0c06cbc0 msgr2=0x7f4e0c06f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:24.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.759+0000 7f4e25372700 1 --2- 172.21.15.32:0/112753009 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f4e0c06cbc0 0x7f4e0c06f070 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4e10004f60 tx=0x7f4e1000b040 comp rx=0 tx=0).stop 2024-04-12T06:04:24.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.759+0000 7f4e25372700 1 -- 172.21.15.32:0/112753009 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e20103670 msgr2=0x7f4e20070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:24.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.759+0000 7f4e25372700 1 --2- 172.21.15.32:0/112753009 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e20103670 0x7f4e20070800 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f4e08009390 tx=0x7f4e08009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:24.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.759+0000 7f4e25372700 1 -- 172.21.15.32:0/112753009 shutdown_connections 2024-04-12T06:04:24.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.759+0000 7f4e25372700 1 --2- 172.21.15.32:0/112753009 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f4e0c06cbc0 0x7f4e0c06f070 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:24.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.759+0000 7f4e25372700 1 --2- 172.21.15.32:0/112753009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e20102cd0 0x7f4e200702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:24.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.759+0000 7f4e25372700 1 --2- 172.21.15.32:0/112753009 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e20103670 0x7f4e20070800 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:24.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.759+0000 7f4e25372700 1 -- 172.21.15.32:0/112753009 >> 172.21.15.32:0/112753009 conn(0x7f4e200f8820 msgr2=0x7f4e200f9910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:24.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.759+0000 7f4e25372700 1 -- 172.21.15.32:0/112753009 shutdown_connections 2024-04-12T06:04:24.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.759+0000 7f4e25372700 1 -- 172.21.15.32:0/112753009 wait complete. 2024-04-12T06:04:24.756 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-04-12T06:04:24.767 DEBUG:tasks.fs:max_mds not reduced in epoch 29 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-12T06:04:24.767 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 29 2024-04-12T06:04:24.767 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 30 2024-04-12T06:04:24.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.900+0000 7f94b51fd700 1 -- 172.21.15.32:0/1979695282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b0101be0 msgr2=0x7f94b0101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:24.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.900+0000 7f94b51fd700 1 --2- 172.21.15.32:0/1979695282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b0101be0 0x7f94b0101fb0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f94a4004940 tx=0x7f94a4009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:24.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.900+0000 7f94b51fd700 1 -- 172.21.15.32:0/1979695282 shutdown_connections 2024-04-12T06:04:24.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.900+0000 7f94b51fd700 1 --2- 172.21.15.32:0/1979695282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94b0102580 0x7f94b00fbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:24.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.900+0000 7f94b51fd700 1 --2- 172.21.15.32:0/1979695282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b0101be0 0x7f94b0101fb0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:24.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.900+0000 7f94b51fd700 1 -- 172.21.15.32:0/1979695282 >> 172.21.15.32:0/1979695282 conn(0x7f94b0072ad0 msgr2=0x7f94b0072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:24.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.900+0000 7f94b51fd700 1 -- 172.21.15.32:0/1979695282 shutdown_connections 2024-04-12T06:04:24.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.900+0000 7f94b51fd700 1 -- 172.21.15.32:0/1979695282 wait complete. 2024-04-12T06:04:24.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.901+0000 7f94b51fd700 1 Processor -- start 2024-04-12T06:04:24.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.901+0000 7f94b51fd700 1 -- start start 2024-04-12T06:04:24.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.901+0000 7f94b51fd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94b0101be0 0x7f94b010e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:24.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.901+0000 7f94b51fd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b0102580 0x7f94b010edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:24.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.901+0000 7f94b51fd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b010f310 con 0x7f94b0102580 2024-04-12T06:04:24.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.901+0000 7f94b51fd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b010f450 con 0x7f94b0101be0 2024-04-12T06:04:24.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.901+0000 7f94ae59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b0102580 0x7f94b010edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:24.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.901+0000 7f94ae59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b0102580 0x7f94b010edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51734/0 (socket says 172.21.15.32:51734) 2024-04-12T06:04:24.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.901+0000 7f94ae59c700 1 -- 172.21.15.32:0/197436688 learned_addr learned my addr 172.21.15.32:0/197436688 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:24.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.901+0000 7f94aed9d700 1 --2- 172.21.15.32:0/197436688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94b0101be0 0x7f94b010e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:24.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.901+0000 7f94ae59c700 1 -- 172.21.15.32:0/197436688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94b0101be0 msgr2=0x7f94b010e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:24.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.901+0000 7f94ae59c700 1 --2- 172.21.15.32:0/197436688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94b0101be0 0x7f94b010e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:24.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.901+0000 7f94ae59c700 1 -- 172.21.15.32:0/197436688 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a4005040 con 0x7f94b0102580 2024-04-12T06:04:24.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.901+0000 7f94ae59c700 1 --2- 172.21.15.32:0/197436688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b0102580 0x7f94b010edd0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f94a000eb40 tx=0x7f94a000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:24.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.902+0000 7f94977fe700 1 -- 172.21.15.32:0/197436688 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94a0007cc0 con 0x7f94b0102580 2024-04-12T06:04:24.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.902+0000 7f94b51fd700 1 -- 172.21.15.32:0/197436688 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94b010f6d0 con 0x7f94b0102580 2024-04-12T06:04:24.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.902+0000 7f94977fe700 1 -- 172.21.15.32:0/197436688 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f94a000f040 con 0x7f94b0102580 2024-04-12T06:04:24.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.902+0000 7f94977fe700 1 -- 172.21.15.32:0/197436688 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94a0009570 con 0x7f94b0102580 2024-04-12T06:04:24.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.902+0000 7f94b51fd700 1 -- 172.21.15.32:0/197436688 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94b0117c50 con 0x7f94b0102580 2024-04-12T06:04:24.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.903+0000 7f94b51fd700 1 -- 172.21.15.32:0/197436688 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94b00fd330 con 0x7f94b0102580 2024-04-12T06:04:24.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.903+0000 7f94977fe700 1 -- 172.21.15.32:0/197436688 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f94a00096d0 con 0x7f94b0102580 2024-04-12T06:04:24.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.903+0000 7f94977fe700 1 --2- 172.21.15.32:0/197436688 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f949806cc20 0x7f949806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:24.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.904+0000 7f94977fe700 1 -- 172.21.15.32:0/197436688 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f94a008bbb0 con 0x7f94b0102580 2024-04-12T06:04:24.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.904+0000 7f94aed9d700 1 --2- 172.21.15.32:0/197436688 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f949806cc20 0x7f949806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:24.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.904+0000 7f94aed9d700 1 --2- 172.21.15.32:0/197436688 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f949806cc20 0x7f949806f0d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f94a400b0d0 tx=0x7f94a4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:24.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:24.908+0000 7f94977fe700 1 -- 172.21.15.32:0/197436688 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f94a0092080 con 0x7f94b0102580 2024-04-12T06:04:25.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:24 smithi032 ceph-mon[146817]: pgmap v717: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:04:25.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:24 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/186485425' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-04-12T06:04:25.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:24 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/2947595834' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-04-12T06:04:25.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:24 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/112753009' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-04-12T06:04:25.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.161+0000 7f94b51fd700 1 -- 172.21.15.32:0/197436688 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 30, "format": "json"} v 0) v1 -- 0x7f94b0108bc0 con 0x7f94b0102580 2024-04-12T06:04:25.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.162+0000 7f94977fe700 1 -- 172.21.15.32:0/197436688 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 30, "format": "json"}]=0 dumped fsmap epoch 30 v40) v1 ==== 107+0+4042 (secure 0 0 0) 0x7f94a0055bc0 con 0x7f94b0102580 2024-04-12T06:04:25.157 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:04:25.157 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34384,"name":"cephfs.smithi032.jehaaj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6827/1884332935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1884332935},{"type":"v1","addr":"172.21.15.32:6827","nonce":1884332935}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":34416,"name":"cephfs.smithi032.pndtry","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6829/1671696137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1671696137},{"type":"v1","addr":"172.21.15.32:6829","nonce":1671696137}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":44275,"name":"cephfs.smithi067.ymvqih","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6827/3573641538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3573641538},{"type":"v1","addr":"172.21.15.67:6827","nonce":3573641538}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-12T05:34:22.641718+0000","modified":"2024-04-12T05:46:10.668569+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T06:04:25.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.164+0000 7f94b51fd700 1 -- 172.21.15.32:0/197436688 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f949806cc20 msgr2=0x7f949806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:25.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.164+0000 7f94b51fd700 1 --2- 172.21.15.32:0/197436688 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f949806cc20 0x7f949806f0d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f94a400b0d0 tx=0x7f94a4009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:25.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.164+0000 7f94b51fd700 1 -- 172.21.15.32:0/197436688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b0102580 msgr2=0x7f94b010edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:25.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.164+0000 7f94b51fd700 1 --2- 172.21.15.32:0/197436688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b0102580 0x7f94b010edd0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f94a000eb40 tx=0x7f94a000b040 comp rx=0 tx=0).stop 2024-04-12T06:04:25.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.164+0000 7f94b51fd700 1 -- 172.21.15.32:0/197436688 shutdown_connections 2024-04-12T06:04:25.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.164+0000 7f94b51fd700 1 --2- 172.21.15.32:0/197436688 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f949806cc20 0x7f949806f0d0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:25.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.164+0000 7f94b51fd700 1 --2- 172.21.15.32:0/197436688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94b0101be0 0x7f94b010e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:25.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.164+0000 7f94b51fd700 1 --2- 172.21.15.32:0/197436688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b0102580 0x7f94b010edd0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:25.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.164+0000 7f94b51fd700 1 -- 172.21.15.32:0/197436688 >> 172.21.15.32:0/197436688 conn(0x7f94b0072ad0 msgr2=0x7f94b0104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:25.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.164+0000 7f94b51fd700 1 -- 172.21.15.32:0/197436688 shutdown_connections 2024-04-12T06:04:25.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.164+0000 7f94b51fd700 1 -- 172.21.15.32:0/197436688 wait complete. 2024-04-12T06:04:25.160 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2024-04-12T06:04:25.171 DEBUG:tasks.fs:max_mds not reduced in epoch 30 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-12T06:04:25.172 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 30 2024-04-12T06:04:25.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 31 2024-04-12T06:04:25.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:24 smithi067 ceph-mon[112979]: pgmap v717: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:04:25.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:24 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/186485425' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-04-12T06:04:25.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:24 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/2947595834' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-04-12T06:04:25.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:24 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/112753009' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-04-12T06:04:25.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.299+0000 7f8dc8314700 1 -- 172.21.15.32:0/1407769072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dc0103660 msgr2=0x7f8dc00fccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:25.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.299+0000 7f8dc8314700 1 --2- 172.21.15.32:0/1407769072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dc0103660 0x7f8dc00fccd0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f8db4004940 tx=0x7f8db4009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:25.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.299+0000 7f8dc8314700 1 -- 172.21.15.32:0/1407769072 shutdown_connections 2024-04-12T06:04:25.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.299+0000 7f8dc8314700 1 --2- 172.21.15.32:0/1407769072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dc0103660 0x7f8dc00fccd0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:25.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.299+0000 7f8dc8314700 1 --2- 172.21.15.32:0/1407769072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc0102cc0 0x7f8dc0103090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:25.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.299+0000 7f8dc8314700 1 -- 172.21.15.32:0/1407769072 >> 172.21.15.32:0/1407769072 conn(0x7f8dc00f8830 msgr2=0x7f8dc00fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:25.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.299+0000 7f8dc8314700 1 -- 172.21.15.32:0/1407769072 shutdown_connections 2024-04-12T06:04:25.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.299+0000 7f8dc8314700 1 -- 172.21.15.32:0/1407769072 wait complete. 2024-04-12T06:04:25.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.300+0000 7f8dc8314700 1 Processor -- start 2024-04-12T06:04:25.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.300+0000 7f8dc8314700 1 -- start start 2024-04-12T06:04:25.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.300+0000 7f8dc8314700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dc0102cc0 0x7f8dc00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:25.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.300+0000 7f8dc8314700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc0103660 0x7f8dc0070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:25.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.300+0000 7f8dc8314700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc0072210 con 0x7f8dc0102cc0 2024-04-12T06:04:25.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.300+0000 7f8dc8314700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc0070d40 con 0x7f8dc0103660 2024-04-12T06:04:25.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.300+0000 7f8dc60b0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dc0102cc0 0x7f8dc00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:25.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.300+0000 7f8dc60b0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dc0102cc0 0x7f8dc00702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51750/0 (socket says 172.21.15.32:51750) 2024-04-12T06:04:25.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.300+0000 7f8dc60b0700 1 -- 172.21.15.32:0/3583526822 learned_addr learned my addr 172.21.15.32:0/3583526822 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:25.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.300+0000 7f8dc58af700 1 --2- 172.21.15.32:0/3583526822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc0103660 0x7f8dc0070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:25.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.300+0000 7f8dc60b0700 1 -- 172.21.15.32:0/3583526822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc0103660 msgr2=0x7f8dc0070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:25.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.300+0000 7f8dc60b0700 1 --2- 172.21.15.32:0/3583526822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc0103660 0x7f8dc0070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:25.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.300+0000 7f8dc60b0700 1 -- 172.21.15.32:0/3583526822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db4005040 con 0x7f8dc0102cc0 2024-04-12T06:04:25.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.300+0000 7f8dc60b0700 1 --2- 172.21.15.32:0/3583526822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dc0102cc0 0x7f8dc00702c0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc00c8c0 tx=0x7f8dbc0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:25.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.301+0000 7f8db2ffd700 1 -- 172.21.15.32:0/3583526822 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8dbc00a760 con 0x7f8dc0102cc0 2024-04-12T06:04:25.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.301+0000 7f8db2ffd700 1 -- 172.21.15.32:0/3583526822 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8dbc00ad70 con 0x7f8dc0102cc0 2024-04-12T06:04:25.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.301+0000 7f8dc8314700 1 -- 172.21.15.32:0/3583526822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc0071020 con 0x7f8dc0102cc0 2024-04-12T06:04:25.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.301+0000 7f8db2ffd700 1 -- 172.21.15.32:0/3583526822 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8dbc0113a0 con 0x7f8dc0102cc0 2024-04-12T06:04:25.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.301+0000 7f8dc8314700 1 -- 172.21.15.32:0/3583526822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc0119f10 con 0x7f8dc0102cc0 2024-04-12T06:04:25.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.302+0000 7f8dc8314700 1 -- 172.21.15.32:0/3583526822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dc00fe410 con 0x7f8dc0102cc0 2024-04-12T06:04:25.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.302+0000 7f8db2ffd700 1 -- 172.21.15.32:0/3583526822 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8dbc012400 con 0x7f8dc0102cc0 2024-04-12T06:04:25.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.302+0000 7f8db2ffd700 1 --2- 172.21.15.32:0/3583526822 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f8dac06cc20 0x7f8dac06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:25.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.302+0000 7f8db2ffd700 1 -- 172.21.15.32:0/3583526822 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f8dbc08c100 con 0x7f8dc0102cc0 2024-04-12T06:04:25.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.303+0000 7f8dc58af700 1 --2- 172.21.15.32:0/3583526822 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f8dac06cc20 0x7f8dac06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:25.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.303+0000 7f8dc58af700 1 --2- 172.21.15.32:0/3583526822 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f8dac06cc20 0x7f8dac06f0d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f8dc00719f0 tx=0x7f8db4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:25.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.307+0000 7f8db2ffd700 1 -- 172.21.15.32:0/3583526822 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8dbc0560c0 con 0x7f8dc0102cc0 2024-04-12T06:04:25.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.559+0000 7f8dc8314700 1 -- 172.21.15.32:0/3583526822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 31, "format": "json"} v 0) v1 -- 0x7f8dc00640e0 con 0x7f8dc0102cc0 2024-04-12T06:04:25.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.560+0000 7f8db2ffd700 1 -- 172.21.15.32:0/3583526822 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 31, "format": "json"}]=0 dumped fsmap epoch 31 v40) v1 ==== 107+0+4818 (secure 0 0 0) 0x7f8dbc0596e0 con 0x7f8dc0102cc0 2024-04-12T06:04:25.556 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:04:25.556 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34384,"name":"cephfs.smithi032.jehaaj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6827/1884332935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1884332935},{"type":"v1","addr":"172.21.15.32:6827","nonce":1884332935}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":34416,"name":"cephfs.smithi032.pndtry","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6829/1671696137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1671696137},{"type":"v1","addr":"172.21.15.32:6829","nonce":1671696137}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":44275,"name":"cephfs.smithi067.ymvqih","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6827/3573641538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3573641538},{"type":"v1","addr":"172.21.15.67:6827","nonce":3573641538}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44317,"name":"cephfs.smithi067.zesmgy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6825/142107794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":142107794},{"type":"v1","addr":"172.21.15.67:6825","nonce":142107794}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-12T05:34:22.641718+0000","modified":"2024-04-12T05:46:10.668569+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T06:04:25.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.562+0000 7f8dc8314700 1 -- 172.21.15.32:0/3583526822 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f8dac06cc20 msgr2=0x7f8dac06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:25.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.562+0000 7f8dc8314700 1 --2- 172.21.15.32:0/3583526822 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f8dac06cc20 0x7f8dac06f0d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f8dc00719f0 tx=0x7f8db4009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:25.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.562+0000 7f8dc8314700 1 -- 172.21.15.32:0/3583526822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dc0102cc0 msgr2=0x7f8dc00702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:25.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.562+0000 7f8dc8314700 1 --2- 172.21.15.32:0/3583526822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dc0102cc0 0x7f8dc00702c0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc00c8c0 tx=0x7f8dbc0044c0 comp rx=0 tx=0).stop 2024-04-12T06:04:25.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.562+0000 7f8dc8314700 1 -- 172.21.15.32:0/3583526822 shutdown_connections 2024-04-12T06:04:25.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.562+0000 7f8dc8314700 1 --2- 172.21.15.32:0/3583526822 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f8dac06cc20 0x7f8dac06f0d0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:25.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.562+0000 7f8dc8314700 1 --2- 172.21.15.32:0/3583526822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dc0102cc0 0x7f8dc00702c0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:25.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.562+0000 7f8dc8314700 1 --2- 172.21.15.32:0/3583526822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc0103660 0x7f8dc0070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:25.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.562+0000 7f8dc8314700 1 -- 172.21.15.32:0/3583526822 >> 172.21.15.32:0/3583526822 conn(0x7f8dc00f8830 msgr2=0x7f8dc00f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:25.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.562+0000 7f8dc8314700 1 -- 172.21.15.32:0/3583526822 shutdown_connections 2024-04-12T06:04:25.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.562+0000 7f8dc8314700 1 -- 172.21.15.32:0/3583526822 wait complete. 2024-04-12T06:04:25.560 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 31 2024-04-12T06:04:25.571 DEBUG:tasks.fs:max_mds not reduced in epoch 31 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-12T06:04:25.571 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 31 2024-04-12T06:04:25.571 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 32 2024-04-12T06:04:25.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.698+0000 7f16c5de2700 1 -- 172.21.15.32:0/2486860958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16c0102ce0 msgr2=0x7f16c01030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:25.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.698+0000 7f16c5de2700 1 --2- 172.21.15.32:0/2486860958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16c0102ce0 0x7f16c01030b0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f16b4004940 tx=0x7f16b4009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:25.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.698+0000 7f16c5de2700 1 -- 172.21.15.32:0/2486860958 shutdown_connections 2024-04-12T06:04:25.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.698+0000 7f16c5de2700 1 --2- 172.21.15.32:0/2486860958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c0103680 0x7f16c0064e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:25.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.698+0000 7f16c5de2700 1 --2- 172.21.15.32:0/2486860958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16c0102ce0 0x7f16c01030b0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:25.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.698+0000 7f16c5de2700 1 -- 172.21.15.32:0/2486860958 >> 172.21.15.32:0/2486860958 conn(0x7f16c0072ad0 msgr2=0x7f16c0072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:25.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.698+0000 7f16c5de2700 1 -- 172.21.15.32:0/2486860958 shutdown_connections 2024-04-12T06:04:25.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.698+0000 7f16c5de2700 1 -- 172.21.15.32:0/2486860958 wait complete. 2024-04-12T06:04:25.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.699+0000 7f16c5de2700 1 Processor -- start 2024-04-12T06:04:25.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.699+0000 7f16c5de2700 1 -- start start 2024-04-12T06:04:25.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.699+0000 7f16c5de2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c0102ce0 0x7f16c00fbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:25.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.699+0000 7f16c5de2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16c0103680 0x7f16c00fc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:25.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.699+0000 7f16c5de2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c00fdb90 con 0x7f16c0103680 2024-04-12T06:04:25.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.699+0000 7f16c5de2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c00fc6c0 con 0x7f16c0102ce0 2024-04-12T06:04:25.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.699+0000 7f16beffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16c0103680 0x7f16c00fc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:25.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.699+0000 7f16beffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16c0103680 0x7f16c00fc180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51758/0 (socket says 172.21.15.32:51758) 2024-04-12T06:04:25.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.699+0000 7f16beffd700 1 -- 172.21.15.32:0/2961372261 learned_addr learned my addr 172.21.15.32:0/2961372261 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:25.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.699+0000 7f16bf7fe700 1 --2- 172.21.15.32:0/2961372261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c0102ce0 0x7f16c00fbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:25.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.700+0000 7f16beffd700 1 -- 172.21.15.32:0/2961372261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c0102ce0 msgr2=0x7f16c00fbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:25.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.700+0000 7f16beffd700 1 --2- 172.21.15.32:0/2961372261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c0102ce0 0x7f16c00fbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:25.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.700+0000 7f16beffd700 1 -- 172.21.15.32:0/2961372261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b4005040 con 0x7f16c0103680 2024-04-12T06:04:25.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.700+0000 7f16bf7fe700 1 --2- 172.21.15.32:0/2961372261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c0102ce0 0x7f16c00fbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T06:04:25.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.700+0000 7f16beffd700 1 --2- 172.21.15.32:0/2961372261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16c0103680 0x7f16c00fc180 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f16b0002a50 tx=0x7f16b00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:25.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.700+0000 7f16a7fff700 1 -- 172.21.15.32:0/2961372261 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16b0007cc0 con 0x7f16c0103680 2024-04-12T06:04:25.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.700+0000 7f16a7fff700 1 -- 172.21.15.32:0/2961372261 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f16b000f040 con 0x7f16c0103680 2024-04-12T06:04:25.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.700+0000 7f16c5de2700 1 -- 172.21.15.32:0/2961372261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16c00fc9a0 con 0x7f16c0103680 2024-04-12T06:04:25.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.700+0000 7f16a7fff700 1 -- 172.21.15.32:0/2961372261 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16b00095d0 con 0x7f16c0103680 2024-04-12T06:04:25.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.700+0000 7f16c5de2700 1 -- 172.21.15.32:0/2961372261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c0119ec0 con 0x7f16c0103680 2024-04-12T06:04:25.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.701+0000 7f16c5de2700 1 -- 172.21.15.32:0/2961372261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16c00fe430 con 0x7f16c0103680 2024-04-12T06:04:25.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.701+0000 7f16a7fff700 1 -- 172.21.15.32:0/2961372261 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f16b0009730 con 0x7f16c0103680 2024-04-12T06:04:25.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.702+0000 7f16a7fff700 1 --2- 172.21.15.32:0/2961372261 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f16a806cc20 0x7f16a806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:25.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.702+0000 7f16a7fff700 1 -- 172.21.15.32:0/2961372261 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f16b008bda0 con 0x7f16c0103680 2024-04-12T06:04:25.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.702+0000 7f16bf7fe700 1 --2- 172.21.15.32:0/2961372261 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f16a806cc20 0x7f16a806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:25.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.702+0000 7f16bf7fe700 1 --2- 172.21.15.32:0/2961372261 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f16a806cc20 0x7f16a806f0d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f16b4004f90 tx=0x7f16b4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:25.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.707+0000 7f16a7fff700 1 -- 172.21.15.32:0/2961372261 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f16b0055d00 con 0x7f16c0103680 2024-04-12T06:04:25.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.958+0000 7f16c5de2700 1 -- 172.21.15.32:0/2961372261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 32, "format": "json"} v 0) v1 -- 0x7f16c011a0b0 con 0x7f16c0103680 2024-04-12T06:04:25.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.958+0000 7f16a7fff700 1 -- 172.21.15.32:0/2961372261 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 32, "format": "json"}]=0 dumped fsmap epoch 32 v40) v1 ==== 107+0+4039 (secure 0 0 0) 0x7f16b0059320 con 0x7f16c0103680 2024-04-12T06:04:25.955 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:04:25.955 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34384,"name":"cephfs.smithi032.jehaaj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6827/1884332935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1884332935},{"type":"v1","addr":"172.21.15.32:6827","nonce":1884332935}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":34416,"name":"cephfs.smithi032.pndtry","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6829/1671696137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1671696137},{"type":"v1","addr":"172.21.15.32:6829","nonce":1671696137}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":44317,"name":"cephfs.smithi067.zesmgy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6825/142107794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":142107794},{"type":"v1","addr":"172.21.15.67:6825","nonce":142107794}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-12T05:34:22.641718+0000","modified":"2024-04-12T05:46:10.668569+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T06:04:25.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.960+0000 7f16c5de2700 1 -- 172.21.15.32:0/2961372261 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f16a806cc20 msgr2=0x7f16a806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:25.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.960+0000 7f16c5de2700 1 --2- 172.21.15.32:0/2961372261 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f16a806cc20 0x7f16a806f0d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f16b4004f90 tx=0x7f16b4009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:25.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.960+0000 7f16c5de2700 1 -- 172.21.15.32:0/2961372261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16c0103680 msgr2=0x7f16c00fc180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:25.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.960+0000 7f16c5de2700 1 --2- 172.21.15.32:0/2961372261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16c0103680 0x7f16c00fc180 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f16b0002a50 tx=0x7f16b00046b0 comp rx=0 tx=0).stop 2024-04-12T06:04:25.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.961+0000 7f16c5de2700 1 -- 172.21.15.32:0/2961372261 shutdown_connections 2024-04-12T06:04:25.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.961+0000 7f16c5de2700 1 --2- 172.21.15.32:0/2961372261 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f16a806cc20 0x7f16a806f0d0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:25.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.961+0000 7f16c5de2700 1 --2- 172.21.15.32:0/2961372261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c0102ce0 0x7f16c00fbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:25.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.961+0000 7f16c5de2700 1 --2- 172.21.15.32:0/2961372261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16c0103680 0x7f16c00fc180 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:25.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.961+0000 7f16c5de2700 1 -- 172.21.15.32:0/2961372261 >> 172.21.15.32:0/2961372261 conn(0x7f16c0072ad0 msgr2=0x7f16c0105a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:25.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.961+0000 7f16c5de2700 1 -- 172.21.15.32:0/2961372261 shutdown_connections 2024-04-12T06:04:25.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:25.961+0000 7f16c5de2700 1 -- 172.21.15.32:0/2961372261 wait complete. 2024-04-12T06:04:25.958 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 32 2024-04-12T06:04:25.969 DEBUG:tasks.fs:max_mds not reduced in epoch 32 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-12T06:04:25.969 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 32 2024-04-12T06:04:25.969 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 33 2024-04-12T06:04:26.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.096+0000 7f0d436af700 1 -- 172.21.15.32:0/2511617696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d3c101ca0 msgr2=0x7f0d3c102070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:26.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.096+0000 7f0d436af700 1 --2- 172.21.15.32:0/2511617696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d3c101ca0 0x7f0d3c102070 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c004940 tx=0x7f0d2c009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:26.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.096+0000 7f0d436af700 1 -- 172.21.15.32:0/2511617696 shutdown_connections 2024-04-12T06:04:26.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.096+0000 7f0d436af700 1 --2- 172.21.15.32:0/2511617696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d3c102640 0x7f0d3c0fc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:26.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.096+0000 7f0d436af700 1 --2- 172.21.15.32:0/2511617696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d3c101ca0 0x7f0d3c102070 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:26.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.096+0000 7f0d436af700 1 -- 172.21.15.32:0/2511617696 >> 172.21.15.32:0/2511617696 conn(0x7f0d3c072230 msgr2=0x7f0d3c072630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:26.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.096+0000 7f0d436af700 1 -- 172.21.15.32:0/2511617696 shutdown_connections 2024-04-12T06:04:26.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.096+0000 7f0d436af700 1 -- 172.21.15.32:0/2511617696 wait complete. 2024-04-12T06:04:26.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.097+0000 7f0d436af700 1 Processor -- start 2024-04-12T06:04:26.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.097+0000 7f0d436af700 1 -- start start 2024-04-12T06:04:26.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.097+0000 7f0d436af700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d3c101ca0 0x7f0d3c10e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:26.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.097+0000 7f0d436af700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d3c102640 0x7f0d3c10eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:26.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.097+0000 7f0d436af700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3c10f410 con 0x7f0d3c102640 2024-04-12T06:04:26.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.097+0000 7f0d436af700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3c10f550 con 0x7f0d3c101ca0 2024-04-12T06:04:26.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.097+0000 7f0d40c4a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d3c102640 0x7f0d3c10eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:26.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.097+0000 7f0d40c4a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d3c102640 0x7f0d3c10eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51790/0 (socket says 172.21.15.32:51790) 2024-04-12T06:04:26.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.097+0000 7f0d40c4a700 1 -- 172.21.15.32:0/1800348312 learned_addr learned my addr 172.21.15.32:0/1800348312 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:26.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.097+0000 7f0d4144b700 1 --2- 172.21.15.32:0/1800348312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d3c101ca0 0x7f0d3c10e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:26.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.097+0000 7f0d40c4a700 1 -- 172.21.15.32:0/1800348312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d3c101ca0 msgr2=0x7f0d3c10e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:26.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.097+0000 7f0d40c4a700 1 --2- 172.21.15.32:0/1800348312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d3c101ca0 0x7f0d3c10e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:26.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.097+0000 7f0d40c4a700 1 -- 172.21.15.32:0/1800348312 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d2c005040 con 0x7f0d3c102640 2024-04-12T06:04:26.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.097+0000 7f0d40c4a700 1 --2- 172.21.15.32:0/1800348312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d3c102640 0x7f0d3c10eed0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f0d3800c800 tx=0x7f0d38004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:26.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.097+0000 7f0d31ffb700 1 -- 172.21.15.32:0/1800348312 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d380106f0 con 0x7f0d3c102640 2024-04-12T06:04:26.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.098+0000 7f0d31ffb700 1 -- 172.21.15.32:0/1800348312 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d38010d00 con 0x7f0d3c102640 2024-04-12T06:04:26.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.098+0000 7f0d436af700 1 -- 172.21.15.32:0/1800348312 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d3c117a00 con 0x7f0d3c102640 2024-04-12T06:04:26.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.098+0000 7f0d31ffb700 1 -- 172.21.15.32:0/1800348312 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d380113a0 con 0x7f0d3c102640 2024-04-12T06:04:26.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.098+0000 7f0d436af700 1 -- 172.21.15.32:0/1800348312 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d3c117d50 con 0x7f0d3c102640 2024-04-12T06:04:26.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.099+0000 7f0d436af700 1 -- 172.21.15.32:0/1800348312 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d3c0fefc0 con 0x7f0d3c102640 2024-04-12T06:04:26.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.099+0000 7f0d31ffb700 1 -- 172.21.15.32:0/1800348312 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f0d38012400 con 0x7f0d3c102640 2024-04-12T06:04:26.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.099+0000 7f0d31ffb700 1 --2- 172.21.15.32:0/1800348312 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0d2806cc20 0x7f0d2806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:26.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.100+0000 7f0d4144b700 1 --2- 172.21.15.32:0/1800348312 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0d2806cc20 0x7f0d2806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:26.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.100+0000 7f0d31ffb700 1 -- 172.21.15.32:0/1800348312 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f0d3808c040 con 0x7f0d3c102640 2024-04-12T06:04:26.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.100+0000 7f0d4144b700 1 --2- 172.21.15.32:0/1800348312 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0d2806cc20 0x7f0d2806f0d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c004f90 tx=0x7f0d2c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:26.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.104+0000 7f0d31ffb700 1 -- 172.21.15.32:0/1800348312 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0d38056020 con 0x7f0d3c102640 2024-04-12T06:04:26.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:25 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/197436688' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-04-12T06:04:26.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:25 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3583526822' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-04-12T06:04:26.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:25 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/197436688' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-04-12T06:04:26.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:25 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3583526822' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-04-12T06:04:26.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.355+0000 7f0d436af700 1 -- 172.21.15.32:0/1800348312 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 33, "format": "json"} v 0) v1 -- 0x7f0d3c0fd460 con 0x7f0d3c102640 2024-04-12T06:04:26.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.355+0000 7f0d31ffb700 1 -- 172.21.15.32:0/1800348312 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 33, "format": "json"}]=0 dumped fsmap epoch 33 v40) v1 ==== 107+0+4818 (secure 0 0 0) 0x7f0d38059640 con 0x7f0d3c102640 2024-04-12T06:04:26.352 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:04:26.352 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34384,"name":"cephfs.smithi032.jehaaj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6827/1884332935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1884332935},{"type":"v1","addr":"172.21.15.32:6827","nonce":1884332935}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":34416,"name":"cephfs.smithi032.pndtry","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6829/1671696137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1671696137},{"type":"v1","addr":"172.21.15.32:6829","nonce":1671696137}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":44317,"name":"cephfs.smithi067.zesmgy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6825/142107794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":142107794},{"type":"v1","addr":"172.21.15.67:6825","nonce":142107794}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":44343,"name":"cephfs.smithi067.ymvqih","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6827/2631637348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2631637348},{"type":"v1","addr":"172.21.15.67:6827","nonce":2631637348}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-12T05:34:22.641718+0000","modified":"2024-04-12T05:46:10.668569+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T06:04:26.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.358+0000 7f0d436af700 1 -- 172.21.15.32:0/1800348312 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0d2806cc20 msgr2=0x7f0d2806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:26.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.358+0000 7f0d436af700 1 --2- 172.21.15.32:0/1800348312 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0d2806cc20 0x7f0d2806f0d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c004f90 tx=0x7f0d2c009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:26.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.358+0000 7f0d436af700 1 -- 172.21.15.32:0/1800348312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d3c102640 msgr2=0x7f0d3c10eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:26.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.358+0000 7f0d436af700 1 --2- 172.21.15.32:0/1800348312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d3c102640 0x7f0d3c10eed0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f0d3800c800 tx=0x7f0d38004470 comp rx=0 tx=0).stop 2024-04-12T06:04:26.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.358+0000 7f0d436af700 1 -- 172.21.15.32:0/1800348312 shutdown_connections 2024-04-12T06:04:26.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.358+0000 7f0d436af700 1 --2- 172.21.15.32:0/1800348312 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0d2806cc20 0x7f0d2806f0d0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:26.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.358+0000 7f0d436af700 1 --2- 172.21.15.32:0/1800348312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d3c101ca0 0x7f0d3c10e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:26.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.358+0000 7f0d436af700 1 --2- 172.21.15.32:0/1800348312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d3c102640 0x7f0d3c10eed0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:26.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.358+0000 7f0d436af700 1 -- 172.21.15.32:0/1800348312 >> 172.21.15.32:0/1800348312 conn(0x7f0d3c072230 msgr2=0x7f0d3c106270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:26.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.358+0000 7f0d436af700 1 -- 172.21.15.32:0/1800348312 shutdown_connections 2024-04-12T06:04:26.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.358+0000 7f0d436af700 1 -- 172.21.15.32:0/1800348312 wait complete. 2024-04-12T06:04:26.355 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2024-04-12T06:04:26.366 DEBUG:tasks.fs:max_mds not reduced in epoch 33 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-12T06:04:26.366 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 33 2024-04-12T06:04:26.366 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 34 2024-04-12T06:04:26.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.493+0000 7fb4c4c37700 1 -- 172.21.15.32:0/3817236372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4c0102d80 msgr2=0x7fb4c0103150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:26.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.493+0000 7fb4c4c37700 1 --2- 172.21.15.32:0/3817236372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4c0102d80 0x7fb4c0103150 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0004940 tx=0x7fb4b0009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:26.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.493+0000 7fb4c4c37700 1 -- 172.21.15.32:0/3817236372 shutdown_connections 2024-04-12T06:04:26.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.493+0000 7fb4c4c37700 1 --2- 172.21.15.32:0/3817236372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4c0103720 0x7fb4c00fb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:26.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.493+0000 7fb4c4c37700 1 --2- 172.21.15.32:0/3817236372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4c0102d80 0x7fb4c0103150 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:26.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.493+0000 7fb4c4c37700 1 -- 172.21.15.32:0/3817236372 >> 172.21.15.32:0/3817236372 conn(0x7fb4c0072230 msgr2=0x7fb4c0072630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:26.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.493+0000 7fb4c4c37700 1 -- 172.21.15.32:0/3817236372 shutdown_connections 2024-04-12T06:04:26.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.493+0000 7fb4c4c37700 1 -- 172.21.15.32:0/3817236372 wait complete. 2024-04-12T06:04:26.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.494+0000 7fb4c4c37700 1 Processor -- start 2024-04-12T06:04:26.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.494+0000 7fb4c4c37700 1 -- start start 2024-04-12T06:04:26.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.494+0000 7fb4c4c37700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4c0102d80 0x7fb4c0113740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:26.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.494+0000 7fb4c4c37700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4c0103720 0x7fb4c0113c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:26.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.494+0000 7fb4c4c37700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c01142d0 con 0x7fb4c0102d80 2024-04-12T06:04:26.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.494+0000 7fb4c4c37700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c010d8d0 con 0x7fb4c0103720 2024-04-12T06:04:26.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.494+0000 7fb4be59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4c0102d80 0x7fb4c0113740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:26.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.494+0000 7fb4be59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4c0102d80 0x7fb4c0113740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51818/0 (socket says 172.21.15.32:51818) 2024-04-12T06:04:26.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.494+0000 7fb4be59c700 1 -- 172.21.15.32:0/3061409490 learned_addr learned my addr 172.21.15.32:0/3061409490 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:26.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.494+0000 7fb4bdd9b700 1 --2- 172.21.15.32:0/3061409490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4c0103720 0x7fb4c0113c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:26.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.495+0000 7fb4be59c700 1 -- 172.21.15.32:0/3061409490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4c0103720 msgr2=0x7fb4c0113c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:26.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.495+0000 7fb4be59c700 1 --2- 172.21.15.32:0/3061409490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4c0103720 0x7fb4c0113c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:26.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.495+0000 7fb4be59c700 1 -- 172.21.15.32:0/3061409490 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b0005040 con 0x7fb4c0102d80 2024-04-12T06:04:26.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.495+0000 7fb4bdd9b700 1 --2- 172.21.15.32:0/3061409490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4c0103720 0x7fb4c0113c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T06:04:26.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.495+0000 7fb4be59c700 1 --2- 172.21.15.32:0/3061409490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4c0102d80 0x7fb4c0113740 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fb4b000b3b0 tx=0x7fb4b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:26.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.495+0000 7fb4aeffd700 1 -- 172.21.15.32:0/3061409490 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4b000e040 con 0x7fb4c0102d80 2024-04-12T06:04:26.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.495+0000 7fb4c4c37700 1 -- 172.21.15.32:0/3061409490 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4c010db50 con 0x7fb4c0102d80 2024-04-12T06:04:26.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.495+0000 7fb4aeffd700 1 -- 172.21.15.32:0/3061409490 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb4b0013420 con 0x7fb4c0102d80 2024-04-12T06:04:26.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.495+0000 7fb4aeffd700 1 -- 172.21.15.32:0/3061409490 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4b001c630 con 0x7fb4c0102d80 2024-04-12T06:04:26.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.495+0000 7fb4c4c37700 1 -- 172.21.15.32:0/3061409490 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4c010e110 con 0x7fb4c0102d80 2024-04-12T06:04:26.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.496+0000 7fb4c4c37700 1 -- 172.21.15.32:0/3061409490 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4c01000e0 con 0x7fb4c0102d80 2024-04-12T06:04:26.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.496+0000 7fb4aeffd700 1 -- 172.21.15.32:0/3061409490 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb4b0013970 con 0x7fb4c0102d80 2024-04-12T06:04:26.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.497+0000 7fb4aeffd700 1 --2- 172.21.15.32:0/3061409490 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb4a806cc20 0x7fb4a806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:26.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.497+0000 7fb4aeffd700 1 -- 172.21.15.32:0/3061409490 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fb4b008d310 con 0x7fb4c0102d80 2024-04-12T06:04:26.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.497+0000 7fb4bdd9b700 1 --2- 172.21.15.32:0/3061409490 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb4a806cc20 0x7fb4a806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:26.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.497+0000 7fb4bdd9b700 1 --2- 172.21.15.32:0/3061409490 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb4a806cc20 0x7fb4a806f0d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4009fd0 tx=0x7fb4b4009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:26.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.501+0000 7fb4aeffd700 1 -- 172.21.15.32:0/3061409490 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb4b005ad80 con 0x7fb4c0102d80 2024-04-12T06:04:26.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.755+0000 7fb4c4c37700 1 -- 172.21.15.32:0/3061409490 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 34, "format": "json"} v 0) v1 -- 0x7fb4c010edc0 con 0x7fb4c0102d80 2024-04-12T06:04:26.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.756+0000 7fb4aeffd700 1 -- 172.21.15.32:0/3061409490 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 34, "format": "json"}]=0 dumped fsmap epoch 34 v40) v1 ==== 107+0+4840 (secure 0 0 0) 0x7fb4b005a910 con 0x7fb4c0102d80 2024-04-12T06:04:26.752 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:04:26.752 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44317,"name":"cephfs.smithi067.zesmgy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6825/142107794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":142107794},{"type":"v1","addr":"172.21.15.67:6825","nonce":142107794}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":44343,"name":"cephfs.smithi067.ymvqih","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6827/2631637348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2631637348},{"type":"v1","addr":"172.21.15.67:6827","nonce":2631637348}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-12T05:34:22.641718+0000","modified":"2024-04-12T06:02:12.810556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34384,"mds_1":34416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34384":{"gid":34384,"name":"cephfs.smithi032.jehaaj","rank":0,"incarnation":34,"state":"up:replay","state_seq":1,"addr":"172.21.15.32:6827/1884332935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1884332935},{"type":"v1","addr":"172.21.15.32:6827","nonce":1884332935}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34416":{"gid":34416,"name":"cephfs.smithi032.pndtry","rank":1,"incarnation":34,"state":"up:replay","state_seq":1,"addr":"172.21.15.32:6829/1671696137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1671696137},{"type":"v1","addr":"172.21.15.32:6829","nonce":1671696137}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T06:04:26.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.758+0000 7fb4c4c37700 1 -- 172.21.15.32:0/3061409490 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb4a806cc20 msgr2=0x7fb4a806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:26.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.758+0000 7fb4c4c37700 1 --2- 172.21.15.32:0/3061409490 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb4a806cc20 0x7fb4a806f0d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4009fd0 tx=0x7fb4b4009ee0 comp rx=0 tx=0).stop 2024-04-12T06:04:26.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.758+0000 7fb4c4c37700 1 -- 172.21.15.32:0/3061409490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4c0102d80 msgr2=0x7fb4c0113740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:26.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.758+0000 7fb4c4c37700 1 --2- 172.21.15.32:0/3061409490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4c0102d80 0x7fb4c0113740 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fb4b000b3b0 tx=0x7fb4b0009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:26.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.758+0000 7fb4c4c37700 1 -- 172.21.15.32:0/3061409490 shutdown_connections 2024-04-12T06:04:26.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.758+0000 7fb4c4c37700 1 --2- 172.21.15.32:0/3061409490 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fb4a806cc20 0x7fb4a806f0d0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:26.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.758+0000 7fb4c4c37700 1 --2- 172.21.15.32:0/3061409490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4c0102d80 0x7fb4c0113740 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:26.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.758+0000 7fb4c4c37700 1 --2- 172.21.15.32:0/3061409490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4c0103720 0x7fb4c0113c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:26.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.758+0000 7fb4c4c37700 1 -- 172.21.15.32:0/3061409490 >> 172.21.15.32:0/3061409490 conn(0x7fb4c0072230 msgr2=0x7fb4c00fac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:26.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.758+0000 7fb4c4c37700 1 -- 172.21.15.32:0/3061409490 shutdown_connections 2024-04-12T06:04:26.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.758+0000 7fb4c4c37700 1 -- 172.21.15.32:0/3061409490 wait complete. 2024-04-12T06:04:26.756 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 34 2024-04-12T06:04:26.767 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 34 2024-04-12T06:04:26.767 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 35 2024-04-12T06:04:26.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.893+0000 7f75ebbd8700 1 -- 172.21.15.32:0/2579650335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e4102580 msgr2=0x7f75e40fbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:26.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.893+0000 7f75ebbd8700 1 --2- 172.21.15.32:0/2579650335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e4102580 0x7f75e40fbbf0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f75d8004940 tx=0x7f75d8009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:26.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.893+0000 7f75ebbd8700 1 -- 172.21.15.32:0/2579650335 shutdown_connections 2024-04-12T06:04:26.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.893+0000 7f75ebbd8700 1 --2- 172.21.15.32:0/2579650335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e4102580 0x7f75e40fbbf0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:26.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.893+0000 7f75ebbd8700 1 --2- 172.21.15.32:0/2579650335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75e4101be0 0x7f75e4101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:26.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.893+0000 7f75ebbd8700 1 -- 172.21.15.32:0/2579650335 >> 172.21.15.32:0/2579650335 conn(0x7f75e40f7710 msgr2=0x7f75e40f9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:26.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.894+0000 7f75ebbd8700 1 -- 172.21.15.32:0/2579650335 shutdown_connections 2024-04-12T06:04:26.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.894+0000 7f75ebbd8700 1 -- 172.21.15.32:0/2579650335 wait complete. 2024-04-12T06:04:26.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.894+0000 7f75ebbd8700 1 Processor -- start 2024-04-12T06:04:26.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.894+0000 7f75ebbd8700 1 -- start start 2024-04-12T06:04:26.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.894+0000 7f75ebbd8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e4101be0 0x7f75e4113790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:26.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.894+0000 7f75ebbd8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75e4102580 0x7f75e4113cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:26.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.894+0000 7f75ebbd8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e4114320 con 0x7f75e4101be0 2024-04-12T06:04:26.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.894+0000 7f75ebbd8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e410d920 con 0x7f75e4102580 2024-04-12T06:04:26.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.894+0000 7f75e9974700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e4101be0 0x7f75e4113790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:26.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.894+0000 7f75e9974700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e4101be0 0x7f75e4113790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51834/0 (socket says 172.21.15.32:51834) 2024-04-12T06:04:26.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.894+0000 7f75e9974700 1 -- 172.21.15.32:0/3732232524 learned_addr learned my addr 172.21.15.32:0/3732232524 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:26.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.894+0000 7f75e9173700 1 --2- 172.21.15.32:0/3732232524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75e4102580 0x7f75e4113cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:26.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.895+0000 7f75e9974700 1 -- 172.21.15.32:0/3732232524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75e4102580 msgr2=0x7f75e4113cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:26.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.895+0000 7f75e9974700 1 --2- 172.21.15.32:0/3732232524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75e4102580 0x7f75e4113cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:26.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.895+0000 7f75e9974700 1 -- 172.21.15.32:0/3732232524 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d8005040 con 0x7f75e4101be0 2024-04-12T06:04:26.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.895+0000 7f75e9974700 1 --2- 172.21.15.32:0/3732232524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e4101be0 0x7f75e4113790 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f75e0002a50 tx=0x7f75e00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:26.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.895+0000 7f75d67fc700 1 -- 172.21.15.32:0/3732232524 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75e0007cc0 con 0x7f75e4101be0 2024-04-12T06:04:26.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.895+0000 7f75ebbd8700 1 -- 172.21.15.32:0/3732232524 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75e410dc00 con 0x7f75e4101be0 2024-04-12T06:04:26.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.895+0000 7f75d67fc700 1 -- 172.21.15.32:0/3732232524 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f75e000f040 con 0x7f75e4101be0 2024-04-12T06:04:26.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.895+0000 7f75d67fc700 1 -- 172.21.15.32:0/3732232524 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75e000d9c0 con 0x7f75e4101be0 2024-04-12T06:04:26.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.895+0000 7f75ebbd8700 1 -- 172.21.15.32:0/3732232524 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75e410e190 con 0x7f75e4101be0 2024-04-12T06:04:26.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.896+0000 7f75ebbd8700 1 -- 172.21.15.32:0/3732232524 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75e40fd330 con 0x7f75e4101be0 2024-04-12T06:04:26.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.897+0000 7f75d67fc700 1 -- 172.21.15.32:0/3732232524 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f75e000db20 con 0x7f75e4101be0 2024-04-12T06:04:26.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.897+0000 7f75d67fc700 1 --2- 172.21.15.32:0/3732232524 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f75d006cc20 0x7f75d006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:26.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.897+0000 7f75d67fc700 1 -- 172.21.15.32:0/3732232524 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f75e008bdd0 con 0x7f75e4101be0 2024-04-12T06:04:26.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.897+0000 7f75e9173700 1 --2- 172.21.15.32:0/3732232524 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f75d006cc20 0x7f75d006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:26.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.898+0000 7f75e9173700 1 --2- 172.21.15.32:0/3732232524 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f75d006cc20 0x7f75d006f0d0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f75d8004890 tx=0x7f75d8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:26.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:26.902+0000 7f75d67fc700 1 -- 172.21.15.32:0/3732232524 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f75e00597c0 con 0x7f75e4101be0 2024-04-12T06:04:27.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:26 smithi032 ceph-mon[146817]: pgmap v718: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:04:27.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:26 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/2961372261' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-04-12T06:04:27.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:26 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1800348312' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-04-12T06:04:27.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:26 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3061409490' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-04-12T06:04:27.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.155+0000 7f75ebbd8700 1 -- 172.21.15.32:0/3732232524 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 35, "format": "json"} v 0) v1 -- 0x7f75e4108b70 con 0x7f75e4101be0 2024-04-12T06:04:27.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.155+0000 7f75d67fc700 1 -- 172.21.15.32:0/3732232524 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 35, "format": "json"}]=0 dumped fsmap epoch 35 v40) v1 ==== 107+0+4839 (secure 0 0 0) 0x7f75e0059350 con 0x7f75e4101be0 2024-04-12T06:04:27.151 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:04:27.152 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44317,"name":"cephfs.smithi067.zesmgy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6825/142107794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":142107794},{"type":"v1","addr":"172.21.15.67:6825","nonce":142107794}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":44343,"name":"cephfs.smithi067.ymvqih","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6827/2631637348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2631637348},{"type":"v1","addr":"172.21.15.67:6827","nonce":2631637348}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":35,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-12T05:34:22.641718+0000","modified":"2024-04-12T06:02:13.811894+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34384,"mds_1":34416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34384":{"gid":34384,"name":"cephfs.smithi032.jehaaj","rank":0,"incarnation":34,"state":"up:replay","state_seq":1,"addr":"172.21.15.32:6827/1884332935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1884332935},{"type":"v1","addr":"172.21.15.32:6827","nonce":1884332935}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34416":{"gid":34416,"name":"cephfs.smithi032.pndtry","rank":1,"incarnation":34,"state":"up:replay","state_seq":1,"addr":"172.21.15.32:6829/1671696137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1671696137},{"type":"v1","addr":"172.21.15.32:6829","nonce":1671696137}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T06:04:27.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.157+0000 7f75ebbd8700 1 -- 172.21.15.32:0/3732232524 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f75d006cc20 msgr2=0x7f75d006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:27.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.157+0000 7f75ebbd8700 1 --2- 172.21.15.32:0/3732232524 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f75d006cc20 0x7f75d006f0d0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f75d8004890 tx=0x7f75d8009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:27.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.157+0000 7f75ebbd8700 1 -- 172.21.15.32:0/3732232524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e4101be0 msgr2=0x7f75e4113790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:27.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.157+0000 7f75ebbd8700 1 --2- 172.21.15.32:0/3732232524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e4101be0 0x7f75e4113790 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f75e0002a50 tx=0x7f75e00046b0 comp rx=0 tx=0).stop 2024-04-12T06:04:27.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.157+0000 7f75ebbd8700 1 -- 172.21.15.32:0/3732232524 shutdown_connections 2024-04-12T06:04:27.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.157+0000 7f75ebbd8700 1 --2- 172.21.15.32:0/3732232524 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f75d006cc20 0x7f75d006f0d0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:27.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.157+0000 7f75ebbd8700 1 --2- 172.21.15.32:0/3732232524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e4101be0 0x7f75e4113790 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:27.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.157+0000 7f75ebbd8700 1 --2- 172.21.15.32:0/3732232524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75e4102580 0x7f75e4113cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:27.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.157+0000 7f75ebbd8700 1 -- 172.21.15.32:0/3732232524 >> 172.21.15.32:0/3732232524 conn(0x7f75e40f7710 msgr2=0x7f75e40f8e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:27.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.157+0000 7f75ebbd8700 1 -- 172.21.15.32:0/3732232524 shutdown_connections 2024-04-12T06:04:27.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.157+0000 7f75ebbd8700 1 -- 172.21.15.32:0/3732232524 wait complete. 2024-04-12T06:04:27.155 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 35 2024-04-12T06:04:27.166 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 36 2024-04-12T06:04:27.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:26 smithi067 ceph-mon[112979]: pgmap v718: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:04:27.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:26 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/2961372261' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-04-12T06:04:27.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:26 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1800348312' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-04-12T06:04:27.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:26 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3061409490' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-04-12T06:04:27.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.293+0000 7ff71a63f700 1 -- 172.21.15.32:0/3785996733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff714101c10 msgr2=0x7ff714101fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:27.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.293+0000 7ff71a63f700 1 --2- 172.21.15.32:0/3785996733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff714101c10 0x7ff714101fe0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7ff704004940 tx=0x7ff704009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:27.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.293+0000 7ff71a63f700 1 -- 172.21.15.32:0/3785996733 shutdown_connections 2024-04-12T06:04:27.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.293+0000 7ff71a63f700 1 --2- 172.21.15.32:0/3785996733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7141025b0 0x7ff7140fbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:27.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.293+0000 7ff71a63f700 1 --2- 172.21.15.32:0/3785996733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff714101c10 0x7ff714101fe0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:27.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.293+0000 7ff71a63f700 1 -- 172.21.15.32:0/3785996733 >> 172.21.15.32:0/3785996733 conn(0x7ff7140f7720 msgr2=0x7ff7140f9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:27.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.293+0000 7ff71a63f700 1 -- 172.21.15.32:0/3785996733 shutdown_connections 2024-04-12T06:04:27.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.293+0000 7ff71a63f700 1 -- 172.21.15.32:0/3785996733 wait complete. 2024-04-12T06:04:27.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.293+0000 7ff71a63f700 1 Processor -- start 2024-04-12T06:04:27.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.294+0000 7ff71a63f700 1 -- start start 2024-04-12T06:04:27.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.294+0000 7ff71a63f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff714101c10 0x7ff714113790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:27.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.294+0000 7ff71a63f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7141025b0 0x7ff714113cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:27.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.294+0000 7ff71a63f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff714114320 con 0x7ff7141025b0 2024-04-12T06:04:27.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.294+0000 7ff71a63f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71410d920 con 0x7ff714101c10 2024-04-12T06:04:27.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.294+0000 7ff7137fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7141025b0 0x7ff714113cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:27.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.294+0000 7ff7137fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7141025b0 0x7ff714113cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51864/0 (socket says 172.21.15.32:51864) 2024-04-12T06:04:27.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.294+0000 7ff7137fe700 1 -- 172.21.15.32:0/2130790 learned_addr learned my addr 172.21.15.32:0/2130790 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:27.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.294+0000 7ff713fff700 1 --2- 172.21.15.32:0/2130790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff714101c10 0x7ff714113790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:27.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.294+0000 7ff7137fe700 1 -- 172.21.15.32:0/2130790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff714101c10 msgr2=0x7ff714113790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:27.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.294+0000 7ff7137fe700 1 --2- 172.21.15.32:0/2130790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff714101c10 0x7ff714113790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:27.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.294+0000 7ff7137fe700 1 -- 172.21.15.32:0/2130790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff704005040 con 0x7ff7141025b0 2024-04-12T06:04:27.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.294+0000 7ff7137fe700 1 --2- 172.21.15.32:0/2130790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7141025b0 0x7ff714113cd0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7ff708002a50 tx=0x7ff708004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:27.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.295+0000 7ff710ff9700 1 -- 172.21.15.32:0/2130790 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff708007cc0 con 0x7ff7141025b0 2024-04-12T06:04:27.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.295+0000 7ff71a63f700 1 -- 172.21.15.32:0/2130790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff71410dc00 con 0x7ff7141025b0 2024-04-12T06:04:27.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.295+0000 7ff710ff9700 1 -- 172.21.15.32:0/2130790 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff70800f040 con 0x7ff7141025b0 2024-04-12T06:04:27.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.295+0000 7ff710ff9700 1 -- 172.21.15.32:0/2130790 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff70800d920 con 0x7ff7141025b0 2024-04-12T06:04:27.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.295+0000 7ff71a63f700 1 -- 172.21.15.32:0/2130790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff71410e1c0 con 0x7ff7141025b0 2024-04-12T06:04:27.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.296+0000 7ff71a63f700 1 -- 172.21.15.32:0/2130790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7140fd360 con 0x7ff7141025b0 2024-04-12T06:04:27.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.296+0000 7ff710ff9700 1 -- 172.21.15.32:0/2130790 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ff708009680 con 0x7ff7141025b0 2024-04-12T06:04:27.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.297+0000 7ff710ff9700 1 --2- 172.21.15.32:0/2130790 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff6fc06cc20 0x7ff6fc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:27.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.297+0000 7ff710ff9700 1 -- 172.21.15.32:0/2130790 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7ff70808be40 con 0x7ff7141025b0 2024-04-12T06:04:27.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.297+0000 7ff713fff700 1 --2- 172.21.15.32:0/2130790 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff6fc06cc20 0x7ff6fc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:27.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.297+0000 7ff713fff700 1 --2- 172.21.15.32:0/2130790 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff6fc06cc20 0x7ff6fc06f0d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff70400b840 tx=0x7ff704009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:27.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.301+0000 7ff710ff9700 1 -- 172.21.15.32:0/2130790 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff708059830 con 0x7ff7141025b0 2024-04-12T06:04:27.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.552+0000 7ff71a63f700 1 -- 172.21.15.32:0/2130790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 36, "format": "json"} v 0) v1 -- 0x7ff7140640e0 con 0x7ff7141025b0 2024-04-12T06:04:27.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.553+0000 7ff710ff9700 1 -- 172.21.15.32:0/2130790 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 36, "format": "json"}]=0 dumped fsmap epoch 36 v40) v1 ==== 107+0+4845 (secure 0 0 0) 0x7ff7080593c0 con 0x7ff7141025b0 2024-04-12T06:04:27.549 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:04:27.549 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44317,"name":"cephfs.smithi067.zesmgy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6825/142107794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":142107794},{"type":"v1","addr":"172.21.15.67:6825","nonce":142107794}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":44343,"name":"cephfs.smithi067.ymvqih","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6827/2631637348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2631637348},{"type":"v1","addr":"172.21.15.67:6827","nonce":2631637348}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-12T05:34:22.641718+0000","modified":"2024-04-12T06:02:20.964979+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34384,"mds_1":34416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34384":{"gid":34384,"name":"cephfs.smithi032.jehaaj","rank":0,"incarnation":34,"state":"up:resolve","state_seq":241,"addr":"172.21.15.32:6827/1884332935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1884332935},{"type":"v1","addr":"172.21.15.32:6827","nonce":1884332935}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34416":{"gid":34416,"name":"cephfs.smithi032.pndtry","rank":1,"incarnation":34,"state":"up:resolve","state_seq":237,"addr":"172.21.15.32:6829/1671696137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1671696137},{"type":"v1","addr":"172.21.15.32:6829","nonce":1671696137}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T06:04:27.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.555+0000 7ff71a63f700 1 -- 172.21.15.32:0/2130790 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff6fc06cc20 msgr2=0x7ff6fc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:27.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.555+0000 7ff71a63f700 1 --2- 172.21.15.32:0/2130790 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff6fc06cc20 0x7ff6fc06f0d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff70400b840 tx=0x7ff704009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:27.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.555+0000 7ff71a63f700 1 -- 172.21.15.32:0/2130790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7141025b0 msgr2=0x7ff714113cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:27.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.555+0000 7ff71a63f700 1 --2- 172.21.15.32:0/2130790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7141025b0 0x7ff714113cd0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7ff708002a50 tx=0x7ff708004600 comp rx=0 tx=0).stop 2024-04-12T06:04:27.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.555+0000 7ff71a63f700 1 -- 172.21.15.32:0/2130790 shutdown_connections 2024-04-12T06:04:27.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.555+0000 7ff71a63f700 1 --2- 172.21.15.32:0/2130790 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7ff6fc06cc20 0x7ff6fc06f0d0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:27.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.555+0000 7ff71a63f700 1 --2- 172.21.15.32:0/2130790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff714101c10 0x7ff714113790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:27.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.555+0000 7ff71a63f700 1 --2- 172.21.15.32:0/2130790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7141025b0 0x7ff714113cd0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:27.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.555+0000 7ff71a63f700 1 -- 172.21.15.32:0/2130790 >> 172.21.15.32:0/2130790 conn(0x7ff7140f7720 msgr2=0x7ff714104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:27.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.555+0000 7ff71a63f700 1 -- 172.21.15.32:0/2130790 shutdown_connections 2024-04-12T06:04:27.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.555+0000 7ff71a63f700 1 -- 172.21.15.32:0/2130790 wait complete. 2024-04-12T06:04:27.552 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 36 2024-04-12T06:04:27.563 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 37 2024-04-12T06:04:27.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.690+0000 7f134e729700 1 -- 172.21.15.32:0/3457612677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1348103720 msgr2=0x7f13480fb340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:27.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.690+0000 7f134e729700 1 --2- 172.21.15.32:0/3457612677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1348103720 0x7f13480fb340 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f1338004940 tx=0x7f1338009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:27.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.690+0000 7f134e729700 1 -- 172.21.15.32:0/3457612677 shutdown_connections 2024-04-12T06:04:27.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.690+0000 7f134e729700 1 --2- 172.21.15.32:0/3457612677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1348103720 0x7f13480fb340 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:27.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.690+0000 7f134e729700 1 --2- 172.21.15.32:0/3457612677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1348102d80 0x7f1348103150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:27.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.690+0000 7f134e729700 1 -- 172.21.15.32:0/3457612677 >> 172.21.15.32:0/3457612677 conn(0x7f1348072230 msgr2=0x7f1348072630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:27.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.690+0000 7f134e729700 1 -- 172.21.15.32:0/3457612677 shutdown_connections 2024-04-12T06:04:27.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.690+0000 7f134e729700 1 -- 172.21.15.32:0/3457612677 wait complete. 2024-04-12T06:04:27.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.690+0000 7f134e729700 1 Processor -- start 2024-04-12T06:04:27.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.691+0000 7f134e729700 1 -- start start 2024-04-12T06:04:27.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.691+0000 7f134e729700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1348102d80 0x7f1348113780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:27.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.691+0000 7f134e729700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1348103720 0x7f1348113cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:27.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.691+0000 7f134e729700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1348114310 con 0x7f1348103720 2024-04-12T06:04:27.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.691+0000 7f134e729700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134810d910 con 0x7f1348102d80 2024-04-12T06:04:27.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.691+0000 7f13477fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1348103720 0x7f1348113cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:27.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.691+0000 7f13477fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1348103720 0x7f1348113cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51880/0 (socket says 172.21.15.32:51880) 2024-04-12T06:04:27.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.691+0000 7f13477fe700 1 -- 172.21.15.32:0/1065445076 learned_addr learned my addr 172.21.15.32:0/1065445076 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:27.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.691+0000 7f1347fff700 1 --2- 172.21.15.32:0/1065445076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1348102d80 0x7f1348113780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:27.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.691+0000 7f13477fe700 1 -- 172.21.15.32:0/1065445076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1348102d80 msgr2=0x7f1348113780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:27.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.691+0000 7f13477fe700 1 --2- 172.21.15.32:0/1065445076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1348102d80 0x7f1348113780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:27.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.691+0000 7f13477fe700 1 -- 172.21.15.32:0/1065445076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1338005040 con 0x7f1348103720 2024-04-12T06:04:27.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.691+0000 7f13477fe700 1 --2- 172.21.15.32:0/1065445076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1348103720 0x7f1348113cc0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f1338009390 tx=0x7f1338009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:27.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.692+0000 7f1344ff9700 1 -- 172.21.15.32:0/1065445076 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f133800e040 con 0x7f1348103720 2024-04-12T06:04:27.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.692+0000 7f134e729700 1 -- 172.21.15.32:0/1065445076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f134810db90 con 0x7f1348103720 2024-04-12T06:04:27.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.692+0000 7f1344ff9700 1 -- 172.21.15.32:0/1065445076 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1338004290 con 0x7f1348103720 2024-04-12T06:04:27.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.692+0000 7f1344ff9700 1 -- 172.21.15.32:0/1065445076 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1338013e70 con 0x7f1348103720 2024-04-12T06:04:27.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.692+0000 7f134e729700 1 -- 172.21.15.32:0/1065445076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f134810e150 con 0x7f1348103720 2024-04-12T06:04:27.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.693+0000 7f134e729700 1 -- 172.21.15.32:0/1065445076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13481000e0 con 0x7f1348103720 2024-04-12T06:04:27.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.693+0000 7f1344ff9700 1 -- 172.21.15.32:0/1065445076 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1338013690 con 0x7f1348103720 2024-04-12T06:04:27.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.694+0000 7f1344ff9700 1 --2- 172.21.15.32:0/1065445076 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f133406cbc0 0x7f133406f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:27.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.694+0000 7f1347fff700 1 --2- 172.21.15.32:0/1065445076 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f133406cbc0 0x7f133406f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:27.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.694+0000 7f1344ff9700 1 -- 172.21.15.32:0/1065445076 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f133808d2b0 con 0x7f1348103720 2024-04-12T06:04:27.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.694+0000 7f1347fff700 1 --2- 172.21.15.32:0/1065445076 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f133406cbc0 0x7f133406f070 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1330004f40 tx=0x7f133000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:27.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.698+0000 7f1344ff9700 1 -- 172.21.15.32:0/1065445076 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f133805add0 con 0x7f1348103720 2024-04-12T06:04:27.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.951+0000 7f134e729700 1 -- 172.21.15.32:0/1065445076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 37, "format": "json"} v 0) v1 -- 0x7f134810edd0 con 0x7f1348103720 2024-04-12T06:04:27.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.951+0000 7f1344ff9700 1 -- 172.21.15.32:0/1065445076 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 37, "format": "json"}]=0 dumped fsmap epoch 37 v40) v1 ==== 107+0+4849 (secure 0 0 0) 0x7f1338013420 con 0x7f1348103720 2024-04-12T06:04:27.948 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:04:27.948 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44317,"name":"cephfs.smithi067.zesmgy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6825/142107794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":142107794},{"type":"v1","addr":"172.21.15.67:6825","nonce":142107794}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":44343,"name":"cephfs.smithi067.ymvqih","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6827/2631637348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2631637348},{"type":"v1","addr":"172.21.15.67:6827","nonce":2631637348}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-12T05:34:22.641718+0000","modified":"2024-04-12T06:02:21.967263+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34384,"mds_1":34416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34384":{"gid":34384,"name":"cephfs.smithi032.jehaaj","rank":0,"incarnation":34,"state":"up:reconnect","state_seq":242,"addr":"172.21.15.32:6827/1884332935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1884332935},{"type":"v1","addr":"172.21.15.32:6827","nonce":1884332935}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34416":{"gid":34416,"name":"cephfs.smithi032.pndtry","rank":1,"incarnation":34,"state":"up:reconnect","state_seq":238,"addr":"172.21.15.32:6829/1671696137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1671696137},{"type":"v1","addr":"172.21.15.32:6829","nonce":1671696137}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T06:04:27.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.953+0000 7f134e729700 1 -- 172.21.15.32:0/1065445076 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f133406cbc0 msgr2=0x7f133406f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:27.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.953+0000 7f134e729700 1 --2- 172.21.15.32:0/1065445076 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f133406cbc0 0x7f133406f070 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1330004f40 tx=0x7f133000b040 comp rx=0 tx=0).stop 2024-04-12T06:04:27.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.953+0000 7f134e729700 1 -- 172.21.15.32:0/1065445076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1348103720 msgr2=0x7f1348113cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:27.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.953+0000 7f134e729700 1 --2- 172.21.15.32:0/1065445076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1348103720 0x7f1348113cc0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f1338009390 tx=0x7f1338009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:27.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.953+0000 7f134e729700 1 -- 172.21.15.32:0/1065445076 shutdown_connections 2024-04-12T06:04:27.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.953+0000 7f134e729700 1 --2- 172.21.15.32:0/1065445076 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f133406cbc0 0x7f133406f070 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:27.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.953+0000 7f134e729700 1 --2- 172.21.15.32:0/1065445076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1348102d80 0x7f1348113780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:27.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.953+0000 7f134e729700 1 --2- 172.21.15.32:0/1065445076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1348103720 0x7f1348113cc0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:27.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.953+0000 7f134e729700 1 -- 172.21.15.32:0/1065445076 >> 172.21.15.32:0/1065445076 conn(0x7f1348072230 msgr2=0x7f13480fac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:27.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.953+0000 7f134e729700 1 -- 172.21.15.32:0/1065445076 shutdown_connections 2024-04-12T06:04:27.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:27.953+0000 7f134e729700 1 -- 172.21.15.32:0/1065445076 wait complete. 2024-04-12T06:04:27.951 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 37 2024-04-12T06:04:27.962 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 38 2024-04-12T06:04:28.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.089+0000 7f86dc074700 1 -- 172.21.15.32:0/1838627100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d4103660 msgr2=0x7f86d40fccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:28.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.089+0000 7f86dc074700 1 --2- 172.21.15.32:0/1838627100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d4103660 0x7f86d40fccd0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f86c4004940 tx=0x7f86c4009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:28.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.089+0000 7f86dc074700 1 -- 172.21.15.32:0/1838627100 shutdown_connections 2024-04-12T06:04:28.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.089+0000 7f86dc074700 1 --2- 172.21.15.32:0/1838627100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d4103660 0x7f86d40fccd0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:28.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.089+0000 7f86dc074700 1 --2- 172.21.15.32:0/1838627100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86d4102cc0 0x7f86d4103090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:28.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.089+0000 7f86dc074700 1 -- 172.21.15.32:0/1838627100 >> 172.21.15.32:0/1838627100 conn(0x7f86d40f8830 msgr2=0x7f86d40fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:28.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.089+0000 7f86dc074700 1 -- 172.21.15.32:0/1838627100 shutdown_connections 2024-04-12T06:04:28.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.089+0000 7f86dc074700 1 -- 172.21.15.32:0/1838627100 wait complete. 2024-04-12T06:04:28.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.089+0000 7f86dc074700 1 Processor -- start 2024-04-12T06:04:28.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.089+0000 7f86dc074700 1 -- start start 2024-04-12T06:04:28.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.089+0000 7f86dc074700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86d4102cc0 0x7f86d4113760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:28.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.089+0000 7f86dc074700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d4103660 0x7f86d4113ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:28.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.089+0000 7f86dc074700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d41142f0 con 0x7f86d4103660 2024-04-12T06:04:28.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.089+0000 7f86dc074700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d410d8f0 con 0x7f86d4102cc0 2024-04-12T06:04:28.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.090+0000 7f86d960f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d4103660 0x7f86d4113ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:28.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.090+0000 7f86d960f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d4103660 0x7f86d4113ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51906/0 (socket says 172.21.15.32:51906) 2024-04-12T06:04:28.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.090+0000 7f86d960f700 1 -- 172.21.15.32:0/3679810020 learned_addr learned my addr 172.21.15.32:0/3679810020 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:28.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.090+0000 7f86d9e10700 1 --2- 172.21.15.32:0/3679810020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86d4102cc0 0x7f86d4113760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:28.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.090+0000 7f86d960f700 1 -- 172.21.15.32:0/3679810020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86d4102cc0 msgr2=0x7f86d4113760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:28.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.090+0000 7f86d960f700 1 --2- 172.21.15.32:0/3679810020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86d4102cc0 0x7f86d4113760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:28.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.090+0000 7f86d960f700 1 -- 172.21.15.32:0/3679810020 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c4005040 con 0x7f86d4103660 2024-04-12T06:04:28.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.090+0000 7f86d960f700 1 --2- 172.21.15.32:0/3679810020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d4103660 0x7f86d4113ca0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f86c4009380 tx=0x7f86c4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:28.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.090+0000 7f86ca7fc700 1 -- 172.21.15.32:0/3679810020 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86c400e040 con 0x7f86d4103660 2024-04-12T06:04:28.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.090+0000 7f86ca7fc700 1 -- 172.21.15.32:0/3679810020 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f86c4013420 con 0x7f86d4103660 2024-04-12T06:04:28.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.090+0000 7f86dc074700 1 -- 172.21.15.32:0/3679810020 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86d410db70 con 0x7f86d4103660 2024-04-12T06:04:28.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.090+0000 7f86ca7fc700 1 -- 172.21.15.32:0/3679810020 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86c401c630 con 0x7f86d4103660 2024-04-12T06:04:28.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.090+0000 7f86dc074700 1 -- 172.21.15.32:0/3679810020 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d410e130 con 0x7f86d4103660 2024-04-12T06:04:28.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.091+0000 7f86dc074700 1 -- 172.21.15.32:0/3679810020 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86d40fe410 con 0x7f86d4103660 2024-04-12T06:04:28.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.092+0000 7f86ca7fc700 1 -- 172.21.15.32:0/3679810020 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f86c40139b0 con 0x7f86d4103660 2024-04-12T06:04:28.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.092+0000 7f86ca7fc700 1 --2- 172.21.15.32:0/3679810020 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f86c006cc20 0x7f86c006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:28.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.092+0000 7f86ca7fc700 1 -- 172.21.15.32:0/3679810020 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f86c408d2b0 con 0x7f86d4103660 2024-04-12T06:04:28.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.093+0000 7f86d9e10700 1 --2- 172.21.15.32:0/3679810020 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f86c006cc20 0x7f86c006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:28.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.093+0000 7f86d9e10700 1 --2- 172.21.15.32:0/3679810020 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f86c006cc20 0x7f86c006f0d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f86d0004f40 tx=0x7f86d000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:28.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.097+0000 7f86ca7fc700 1 -- 172.21.15.32:0/3679810020 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f86c405ac80 con 0x7f86d4103660 2024-04-12T06:04:28.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:27 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3732232524' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 35, "format": "json"}]: dispatch 2024-04-12T06:04:28.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:27 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/2130790' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 36, "format": "json"}]: dispatch 2024-04-12T06:04:28.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:27 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3732232524' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 35, "format": "json"}]: dispatch 2024-04-12T06:04:28.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:27 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/2130790' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 36, "format": "json"}]: dispatch 2024-04-12T06:04:28.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.346+0000 7f86dc074700 1 -- 172.21.15.32:0/3679810020 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 38, "format": "json"} v 0) v1 -- 0x7f86d410edb0 con 0x7f86d4103660 2024-04-12T06:04:28.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.346+0000 7f86ca7fc700 1 -- 172.21.15.32:0/3679810020 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 38, "format": "json"}]=0 dumped fsmap epoch 38 v40) v1 ==== 107+0+4843 (secure 0 0 0) 0x7f86c405a810 con 0x7f86d4103660 2024-04-12T06:04:28.342 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:04:28.343 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44317,"name":"cephfs.smithi067.zesmgy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6825/142107794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":142107794},{"type":"v1","addr":"172.21.15.67:6825","nonce":142107794}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":44343,"name":"cephfs.smithi067.ymvqih","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6827/2631637348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2631637348},{"type":"v1","addr":"172.21.15.67:6827","nonce":2631637348}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-12T05:34:22.641718+0000","modified":"2024-04-12T06:02:22.971753+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34384,"mds_1":34416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34384":{"gid":34384,"name":"cephfs.smithi032.jehaaj","rank":0,"incarnation":34,"state":"up:rejoin","state_seq":243,"addr":"172.21.15.32:6827/1884332935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1884332935},{"type":"v1","addr":"172.21.15.32:6827","nonce":1884332935}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34416":{"gid":34416,"name":"cephfs.smithi032.pndtry","rank":1,"incarnation":34,"state":"up:rejoin","state_seq":239,"addr":"172.21.15.32:6829/1671696137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1671696137},{"type":"v1","addr":"172.21.15.32:6829","nonce":1671696137}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T06:04:28.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.348+0000 7f86dc074700 1 -- 172.21.15.32:0/3679810020 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f86c006cc20 msgr2=0x7f86c006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:28.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.348+0000 7f86dc074700 1 --2- 172.21.15.32:0/3679810020 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f86c006cc20 0x7f86c006f0d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f86d0004f40 tx=0x7f86d000b040 comp rx=0 tx=0).stop 2024-04-12T06:04:28.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.348+0000 7f86dc074700 1 -- 172.21.15.32:0/3679810020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d4103660 msgr2=0x7f86d4113ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:28.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.348+0000 7f86dc074700 1 --2- 172.21.15.32:0/3679810020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d4103660 0x7f86d4113ca0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f86c4009380 tx=0x7f86c4009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:28.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.348+0000 7f86dc074700 1 -- 172.21.15.32:0/3679810020 shutdown_connections 2024-04-12T06:04:28.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.348+0000 7f86dc074700 1 --2- 172.21.15.32:0/3679810020 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f86c006cc20 0x7f86c006f0d0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:28.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.348+0000 7f86dc074700 1 --2- 172.21.15.32:0/3679810020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86d4102cc0 0x7f86d4113760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:28.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.348+0000 7f86dc074700 1 --2- 172.21.15.32:0/3679810020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d4103660 0x7f86d4113ca0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:28.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.348+0000 7f86dc074700 1 -- 172.21.15.32:0/3679810020 >> 172.21.15.32:0/3679810020 conn(0x7f86d40f8830 msgr2=0x7f86d40f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:28.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.348+0000 7f86dc074700 1 -- 172.21.15.32:0/3679810020 shutdown_connections 2024-04-12T06:04:28.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.348+0000 7f86dc074700 1 -- 172.21.15.32:0/3679810020 wait complete. 2024-04-12T06:04:28.346 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 38 2024-04-12T06:04:28.357 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 39 2024-04-12T06:04:28.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.483+0000 7f7aaf9e9700 1 -- 172.21.15.32:0/352240975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa8102580 msgr2=0x7f7aa80fbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:28.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.483+0000 7f7aaf9e9700 1 --2- 172.21.15.32:0/352240975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa8102580 0x7f7aa80fbbf0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f7a98004940 tx=0x7f7a98009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:28.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.483+0000 7f7aaf9e9700 1 -- 172.21.15.32:0/352240975 shutdown_connections 2024-04-12T06:04:28.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.483+0000 7f7aaf9e9700 1 --2- 172.21.15.32:0/352240975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa8102580 0x7f7aa80fbbf0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:28.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.483+0000 7f7aaf9e9700 1 --2- 172.21.15.32:0/352240975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7aa8101be0 0x7f7aa8101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:28.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.483+0000 7f7aaf9e9700 1 -- 172.21.15.32:0/352240975 >> 172.21.15.32:0/352240975 conn(0x7f7aa80f7710 msgr2=0x7f7aa80f9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:28.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.483+0000 7f7aaf9e9700 1 -- 172.21.15.32:0/352240975 shutdown_connections 2024-04-12T06:04:28.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.483+0000 7f7aaf9e9700 1 -- 172.21.15.32:0/352240975 wait complete. 2024-04-12T06:04:28.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.483+0000 7f7aaf9e9700 1 Processor -- start 2024-04-12T06:04:28.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.483+0000 7f7aaf9e9700 1 -- start start 2024-04-12T06:04:28.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.484+0000 7f7aaf9e9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7aa8101be0 0x7f7aa8112670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:28.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.484+0000 7f7aaf9e9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa8102580 0x7f7aa8112bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:28.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.484+0000 7f7aaf9e9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa8113240 con 0x7f7aa8102580 2024-04-12T06:04:28.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.484+0000 7f7aaf9e9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa810c800 con 0x7f7aa8101be0 2024-04-12T06:04:28.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.484+0000 7f7aacf84700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa8102580 0x7f7aa8112bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:28.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.484+0000 7f7aacf84700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa8102580 0x7f7aa8112bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51928/0 (socket says 172.21.15.32:51928) 2024-04-12T06:04:28.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.484+0000 7f7aacf84700 1 -- 172.21.15.32:0/912140040 learned_addr learned my addr 172.21.15.32:0/912140040 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:28.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.484+0000 7f7aad785700 1 --2- 172.21.15.32:0/912140040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7aa8101be0 0x7f7aa8112670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:28.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.484+0000 7f7aacf84700 1 -- 172.21.15.32:0/912140040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7aa8101be0 msgr2=0x7f7aa8112670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:28.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.484+0000 7f7aacf84700 1 --2- 172.21.15.32:0/912140040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7aa8101be0 0x7f7aa8112670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:28.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.484+0000 7f7aacf84700 1 -- 172.21.15.32:0/912140040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a98005040 con 0x7f7aa8102580 2024-04-12T06:04:28.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.484+0000 7f7aacf84700 1 --2- 172.21.15.32:0/912140040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa8102580 0x7f7aa8112bf0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f7a980093b0 tx=0x7f7a98009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:28.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.484+0000 7f7a9dffb700 1 -- 172.21.15.32:0/912140040 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a9800e040 con 0x7f7aa8102580 2024-04-12T06:04:28.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.484+0000 7f7a9dffb700 1 -- 172.21.15.32:0/912140040 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a98013420 con 0x7f7aa8102580 2024-04-12T06:04:28.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.484+0000 7f7aaf9e9700 1 -- 172.21.15.32:0/912140040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aa810ca30 con 0x7f7aa8102580 2024-04-12T06:04:28.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.484+0000 7f7a9dffb700 1 -- 172.21.15.32:0/912140040 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a9801c630 con 0x7f7aa8102580 2024-04-12T06:04:28.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.485+0000 7f7aaf9e9700 1 -- 172.21.15.32:0/912140040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aa810cf70 con 0x7f7aa8102580 2024-04-12T06:04:28.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.485+0000 7f7aaf9e9700 1 -- 172.21.15.32:0/912140040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aa80fd330 con 0x7f7aa8102580 2024-04-12T06:04:28.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.486+0000 7f7a9dffb700 1 -- 172.21.15.32:0/912140040 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f7a980139e0 con 0x7f7aa8102580 2024-04-12T06:04:28.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.486+0000 7f7a9dffb700 1 --2- 172.21.15.32:0/912140040 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7a9406cc20 0x7f7a9406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:28.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.487+0000 7f7aad785700 1 --2- 172.21.15.32:0/912140040 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7a9406cc20 0x7f7a9406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:28.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.487+0000 7f7a9dffb700 1 -- 172.21.15.32:0/912140040 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f7a9808d190 con 0x7f7aa8102580 2024-04-12T06:04:28.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.487+0000 7f7aad785700 1 --2- 172.21.15.32:0/912140040 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7a9406cc20 0x7f7a9406f0d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8102ce0 tx=0x7f7aa4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:28.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.491+0000 7f7a9dffb700 1 -- 172.21.15.32:0/912140040 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7a9805aca0 con 0x7f7aa8102580 2024-04-12T06:04:28.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.739+0000 7f7aaf9e9700 1 -- 172.21.15.32:0/912140040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 39, "format": "json"} v 0) v1 -- 0x7f7aa8108b70 con 0x7f7aa8102580 2024-04-12T06:04:28.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.740+0000 7f7a9dffb700 1 -- 172.21.15.32:0/912140040 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 39, "format": "json"}]=0 dumped fsmap epoch 39 v40) v1 ==== 107+0+4843 (secure 0 0 0) 0x7f7a9805a830 con 0x7f7aa8102580 2024-04-12T06:04:28.736 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:04:28.736 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44317,"name":"cephfs.smithi067.zesmgy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6825/142107794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":142107794},{"type":"v1","addr":"172.21.15.67:6825","nonce":142107794}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":44343,"name":"cephfs.smithi067.ymvqih","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6827/2631637348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2631637348},{"type":"v1","addr":"172.21.15.67:6827","nonce":2631637348}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-12T05:34:22.641718+0000","modified":"2024-04-12T06:02:23.979161+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34384,"mds_1":34416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34384":{"gid":34384,"name":"cephfs.smithi032.jehaaj","rank":0,"incarnation":34,"state":"up:active","state_seq":244,"addr":"172.21.15.32:6827/1884332935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1884332935},{"type":"v1","addr":"172.21.15.32:6827","nonce":1884332935}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34416":{"gid":34416,"name":"cephfs.smithi032.pndtry","rank":1,"incarnation":34,"state":"up:active","state_seq":240,"addr":"172.21.15.32:6829/1671696137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1671696137},{"type":"v1","addr":"172.21.15.32:6829","nonce":1671696137}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T06:04:28.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.742+0000 7f7aaf9e9700 1 -- 172.21.15.32:0/912140040 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7a9406cc20 msgr2=0x7f7a9406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:28.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.742+0000 7f7aaf9e9700 1 --2- 172.21.15.32:0/912140040 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7a9406cc20 0x7f7a9406f0d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8102ce0 tx=0x7f7aa4009040 comp rx=0 tx=0).stop 2024-04-12T06:04:28.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.742+0000 7f7aaf9e9700 1 -- 172.21.15.32:0/912140040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa8102580 msgr2=0x7f7aa8112bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:28.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.742+0000 7f7aaf9e9700 1 --2- 172.21.15.32:0/912140040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa8102580 0x7f7aa8112bf0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f7a980093b0 tx=0x7f7a98009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:28.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.742+0000 7f7aaf9e9700 1 -- 172.21.15.32:0/912140040 shutdown_connections 2024-04-12T06:04:28.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.742+0000 7f7aaf9e9700 1 --2- 172.21.15.32:0/912140040 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f7a9406cc20 0x7f7a9406f0d0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:28.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.742+0000 7f7aaf9e9700 1 --2- 172.21.15.32:0/912140040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7aa8101be0 0x7f7aa8112670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:28.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.742+0000 7f7aaf9e9700 1 --2- 172.21.15.32:0/912140040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7aa8102580 0x7f7aa8112bf0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:28.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.742+0000 7f7aaf9e9700 1 -- 172.21.15.32:0/912140040 >> 172.21.15.32:0/912140040 conn(0x7f7aa80f7710 msgr2=0x7f7aa80f8790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:28.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.742+0000 7f7aaf9e9700 1 -- 172.21.15.32:0/912140040 shutdown_connections 2024-04-12T06:04:28.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:28.742+0000 7f7aaf9e9700 1 -- 172.21.15.32:0/912140040 wait complete. 2024-04-12T06:04:28.741 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 39 2024-04-12T06:04:28.751 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-04-12T06:04:28.763 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-04-12T06:04:28.763 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-12T06:04:28.764 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-12T06:04:28.784 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-12T06:04:28.785 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-12T06:04:28.844 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-12T06:04:29.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.010+0000 7fa1558e7700 1 -- 172.21.15.32:0/2724486849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1500fc3d0 msgr2=0x7fa1500fc840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:29.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.010+0000 7fa1558e7700 1 --2- 172.21.15.32:0/2724486849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1500fc3d0 0x7fa1500fc840 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fa144004940 tx=0x7fa144009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:29.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.010+0000 7fa1558e7700 1 -- 172.21.15.32:0/2724486849 shutdown_connections 2024-04-12T06:04:29.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.010+0000 7fa1558e7700 1 --2- 172.21.15.32:0/2724486849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1500fc3d0 0x7fa1500fc840 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:29.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.010+0000 7fa1558e7700 1 --2- 172.21.15.32:0/2724486849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa150101c70 0x7fa150102040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:29.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.010+0000 7fa1558e7700 1 -- 172.21.15.32:0/2724486849 >> 172.21.15.32:0/2724486849 conn(0x7fa150073290 msgr2=0x7fa150073690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:29.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.010+0000 7fa1558e7700 1 -- 172.21.15.32:0/2724486849 shutdown_connections 2024-04-12T06:04:29.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.010+0000 7fa1558e7700 1 -- 172.21.15.32:0/2724486849 wait complete. 2024-04-12T06:04:29.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.011+0000 7fa1558e7700 1 Processor -- start 2024-04-12T06:04:29.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.011+0000 7fa1558e7700 1 -- start start 2024-04-12T06:04:29.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.011+0000 7fa1558e7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1500fc3d0 0x7fa1500fd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:29.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.011+0000 7fa1558e7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa150101c70 0x7fa1500fddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:29.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.011+0000 7fa1558e7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1500ff800 con 0x7fa1500fc3d0 2024-04-12T06:04:29.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.011+0000 7fa1558e7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1500fe330 con 0x7fa150101c70 2024-04-12T06:04:29.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.011+0000 7fa14effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1500fc3d0 0x7fa1500fd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:29.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.011+0000 7fa14effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1500fc3d0 0x7fa1500fd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51940/0 (socket says 172.21.15.32:51940) 2024-04-12T06:04:29.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.011+0000 7fa14effd700 1 -- 172.21.15.32:0/140743054 learned_addr learned my addr 172.21.15.32:0/140743054 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:29.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.012+0000 7fa14e7fc700 1 --2- 172.21.15.32:0/140743054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa150101c70 0x7fa1500fddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:29.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.012+0000 7fa14effd700 1 -- 172.21.15.32:0/140743054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa150101c70 msgr2=0x7fa1500fddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:29.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.012+0000 7fa14effd700 1 --2- 172.21.15.32:0/140743054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa150101c70 0x7fa1500fddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:29.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.012+0000 7fa14effd700 1 -- 172.21.15.32:0/140743054 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa144005040 con 0x7fa1500fc3d0 2024-04-12T06:04:29.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.012+0000 7fa14effd700 1 --2- 172.21.15.32:0/140743054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1500fc3d0 0x7fa1500fd8b0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fa140002a80 tx=0x7fa1400046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:29.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.012+0000 7fa1377fe700 1 -- 172.21.15.32:0/140743054 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa140007cc0 con 0x7fa1500fc3d0 2024-04-12T06:04:29.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.012+0000 7fa1558e7700 1 -- 172.21.15.32:0/140743054 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1500fe610 con 0x7fa1500fc3d0 2024-04-12T06:04:29.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.012+0000 7fa1377fe700 1 -- 172.21.15.32:0/140743054 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa14000f040 con 0x7fa1500fc3d0 2024-04-12T06:04:29.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.012+0000 7fa1377fe700 1 -- 172.21.15.32:0/140743054 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa14000d9e0 con 0x7fa1500fc3d0 2024-04-12T06:04:29.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.012+0000 7fa1558e7700 1 -- 172.21.15.32:0/140743054 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa150119ee0 con 0x7fa1500fc3d0 2024-04-12T06:04:29.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.013+0000 7fa1558e7700 1 -- 172.21.15.32:0/140743054 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1500fd3e0 con 0x7fa1500fc3d0 2024-04-12T06:04:29.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.014+0000 7fa1377fe700 1 -- 172.21.15.32:0/140743054 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa14000dc50 con 0x7fa1500fc3d0 2024-04-12T06:04:29.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.014+0000 7fa1377fe700 1 --2- 172.21.15.32:0/140743054 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa13806cc20 0x7fa13806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:29.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.014+0000 7fa1377fe700 1 -- 172.21.15.32:0/140743054 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fa14008b8c0 con 0x7fa1500fc3d0 2024-04-12T06:04:29.012 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.014+0000 7fa14e7fc700 1 --2- 172.21.15.32:0/140743054 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa13806cc20 0x7fa13806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:29.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.014+0000 7fa14e7fc700 1 --2- 172.21.15.32:0/140743054 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa13806cc20 0x7fa13806f0d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa144004f90 tx=0x7fa144009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:29.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.018+0000 7fa1377fe700 1 -- 172.21.15.32:0/140743054 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa140055820 con 0x7fa1500fc3d0 2024-04-12T06:04:29.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:28 smithi032 ceph-mon[146817]: pgmap v719: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:04:29.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:28 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1065445076' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 37, "format": "json"}]: dispatch 2024-04-12T06:04:29.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:28 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3679810020' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 38, "format": "json"}]: dispatch 2024-04-12T06:04:29.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:28 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/912140040' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 39, "format": "json"}]: dispatch 2024-04-12T06:04:29.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.226+0000 7fa1558e7700 1 -- 172.21.15.32:0/140743054 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa150064dc0 con 0x7fa1500fc3d0 2024-04-12T06:04:29.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.227+0000 7fa1377fe700 1 -- 172.21.15.32:0/140743054 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16631 (secure 0 0 0) 0x7fa140058e40 con 0x7fa1500fc3d0 2024-04-12T06:04:29.224 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:04:29.225 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":85,"fsid":"37b58ce2-f88d-11ee-bc8c-c7b262605968","created":"2024-04-12T05:27:07.779407+0000","modified":"2024-04-12T05:47:06.905923+0000","last_up_change":"2024-04-12T05:46:08.474847+0000","last_in_change":"2024-04-12T05:32:39.814568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T05:31:39.568876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-12T05:34:21.574210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-12T05:34:22.056623+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c1f69d2a-5e37-47b8-b68f-0568af894b91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":49,"up_from":53,"up_thru":78,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1783741888},{"type":"v1","addr":"172.21.15.32:6803","nonce":1783741888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1783741888},{"type":"v1","addr":"172.21.15.32:6805","nonce":1783741888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1783741888},{"type":"v1","addr":"172.21.15.32:6809","nonce":1783741888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1783741888},{"type":"v1","addr":"172.21.15.32:6807","nonce":1783741888}]},"public_addr":"172.21.15.32:6803/1783741888","cluster_addr":"172.21.15.32:6805/1783741888","heartbeat_back_addr":"172.21.15.32:6809/1783741888","heartbeat_front_addr":"172.21.15.32:6807/1783741888","state":["exists","up"]},{"osd":1,"uuid":"86b93557-6e6f-4b84-a5f5-5bce4aa65a32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":54,"up_from":58,"up_thru":78,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1060060861},{"type":"v1","addr":"172.21.15.32:6811","nonce":1060060861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1060060861},{"type":"v1","addr":"172.21.15.32:6813","nonce":1060060861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1060060861},{"type":"v1","addr":"172.21.15.32:6817","nonce":1060060861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1060060861},{"type":"v1","addr":"172.21.15.32:6815","nonce":1060060861}]},"public_addr":"172.21.15.32:6811/1060060861","cluster_addr":"172.21.15.32:6813/1060060861","heartbeat_back_addr":"172.21.15.32:6817/1060060861","heartbeat_front_addr":"172.21.15.32:6815/1060060861","state":["exists","up"]},{"osd":2,"uuid":"d1556613-e0de-4b0e-bb7d-e90361ec9007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":59,"up_from":63,"up_thru":78,"down_at":60,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1610966654},{"type":"v1","addr":"172.21.15.32:6819","nonce":1610966654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1610966654},{"type":"v1","addr":"172.21.15.32:6821","nonce":1610966654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1610966654},{"type":"v1","addr":"172.21.15.32:6825","nonce":1610966654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1610966654},{"type":"v1","addr":"172.21.15.32:6823","nonce":1610966654}]},"public_addr":"172.21.15.32:6819/1610966654","cluster_addr":"172.21.15.32:6821/1610966654","heartbeat_back_addr":"172.21.15.32:6825/1610966654","heartbeat_front_addr":"172.21.15.32:6823/1610966654","state":["exists","up"]},{"osd":3,"uuid":"213dc62e-a894-4bc7-a2dd-ca98d1313d6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":64,"up_from":68,"up_thru":78,"down_at":65,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2625997716},{"type":"v1","addr":"172.21.15.67:6801","nonce":2625997716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2625997716},{"type":"v1","addr":"172.21.15.67:6803","nonce":2625997716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":2625997716},{"type":"v1","addr":"172.21.15.67:6807","nonce":2625997716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":2625997716},{"type":"v1","addr":"172.21.15.67:6805","nonce":2625997716}]},"public_addr":"172.21.15.67:6801/2625997716","cluster_addr":"172.21.15.67:6803/2625997716","heartbeat_back_addr":"172.21.15.67:6807/2625997716","heartbeat_front_addr":"172.21.15.67:6805/2625997716","state":["exists","up"]},{"osd":4,"uuid":"8a0ff860-fff7-434a-9e78-f2020bc84966","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":69,"up_from":73,"up_thru":78,"down_at":70,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":452825880},{"type":"v1","addr":"172.21.15.67:6809","nonce":452825880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":452825880},{"type":"v1","addr":"172.21.15.67:6811","nonce":452825880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":452825880},{"type":"v1","addr":"172.21.15.67:6815","nonce":452825880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":452825880},{"type":"v1","addr":"172.21.15.67:6813","nonce":452825880}]},"public_addr":"172.21.15.67:6809/452825880","cluster_addr":"172.21.15.67:6811/452825880","heartbeat_back_addr":"172.21.15.67:6815/452825880","heartbeat_front_addr":"172.21.15.67:6813/452825880","state":["exists","up"]},{"osd":5,"uuid":"3486e5c9-9f44-4f32-a186-fc6198f09549","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":74,"up_from":78,"up_thru":78,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2273350169},{"type":"v1","addr":"172.21.15.67:6817","nonce":2273350169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2273350169},{"type":"v1","addr":"172.21.15.67:6819","nonce":2273350169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2273350169},{"type":"v1","addr":"172.21.15.67:6823","nonce":2273350169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2273350169},{"type":"v1","addr":"172.21.15.67:6821","nonce":2273350169}]},"public_addr":"172.21.15.67:6817/2273350169","cluster_addr":"172.21.15.67:6819/2273350169","heartbeat_back_addr":"172.21.15.67:6823/2273350169","heartbeat_front_addr":"172.21.15.67:6821/2273350169","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:30:38.748993+0000","dead_epoch":49},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:31:08.677208+0000","dead_epoch":54},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:31:36.979481+0000","dead_epoch":59},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:32:02.668398+0000","dead_epoch":64},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:32:30.143788+0000","dead_epoch":69},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:32:57.613087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6829/387754149":"2024-04-13T05:46:10.668153+0000","172.21.15.32:6828/387754149":"2024-04-13T05:46:10.668153+0000","172.21.15.67:6825/1291914599":"2024-04-13T05:46:10.668178+0000","172.21.15.67:6824/1291914599":"2024-04-13T05:46:10.668178+0000","172.21.15.32:0/138657961":"2024-04-13T05:27:40.952118+0000","172.21.15.67:0/1683579802":"2024-04-13T05:38:47.052969+0000","172.21.15.32:0/368340628":"2024-04-13T05:37:39.471472+0000","172.21.15.32:6801/3971961003":"2024-04-13T05:29:25.054120+0000","172.21.15.32:0/652449107":"2024-04-13T05:41:47.767117+0000","172.21.15.32:0/4112976740":"2024-04-13T05:29:25.054120+0000","172.21.15.32:0/2247439109":"2024-04-13T05:37:39.471472+0000","172.21.15.32:0/1643708423":"2024-04-13T05:27:40.952118+0000","172.21.15.32:6800/2564765508":"2024-04-13T05:41:47.767117+0000","172.21.15.67:6828/3925433122":"2024-04-13T05:38:47.052969+0000","172.21.15.32:6800/1962278696":"2024-04-13T05:28:14.470413+0000","172.21.15.32:0/2162993076":"2024-04-13T05:28:14.470413+0000","172.21.15.32:6827/2962187015":"2024-04-13T05:34:37.113413+0000","172.21.15.32:6800/3971961003":"2024-04-13T05:29:25.054120+0000","172.21.15.32:0/2105642006":"2024-04-13T05:37:39.471472+0000","172.21.15.67:0/31773612":"2024-04-13T05:38:47.052969+0000","172.21.15.67:6829/3925433122":"2024-04-13T05:38:47.052969+0000","172.21.15.67:0/2846171310":"2024-04-13T05:38:47.052969+0000","172.21.15.32:0/3370471392":"2024-04-13T05:41:47.767117+0000","172.21.15.32:0/3082743475":"2024-04-13T05:28:14.470413+0000","172.21.15.32:6801/656201914":"2024-04-13T05:27:40.952118+0000","172.21.15.32:6801/1962278696":"2024-04-13T05:28:14.470413+0000","172.21.15.32:0/260214199":"2024-04-13T05:37:39.471472+0000","172.21.15.32:6800/656201914":"2024-04-13T05:27:40.952118+0000","172.21.15.67:0/3462505447":"2024-04-13T05:38:47.052969+0000","172.21.15.32:0/674029871":"2024-04-13T05:29:25.054120+0000","172.21.15.67:0/125735613":"2024-04-13T05:38:47.052969+0000","172.21.15.67:0/3697060296":"2024-04-13T05:38:47.052969+0000","172.21.15.32:0/4031244002":"2024-04-13T05:27:40.952118+0000","172.21.15.32:6801/2564765508":"2024-04-13T05:41:47.767117+0000","172.21.15.32:0/795906028":"2024-04-13T05:29:25.054120+0000","172.21.15.32:6826/2962187015":"2024-04-13T05:34:37.113413+0000","172.21.15.32:6801/595990262":"2024-04-13T05:37:39.471472+0000","172.21.15.32:0/146872823":"2024-04-13T05:28:14.470413+0000","172.21.15.32:0/138775819":"2024-04-13T05:41:47.767117+0000","172.21.15.32:6800/595990262":"2024-04-13T05:37:39.471472+0000","172.21.15.32:0/2743601944":"2024-04-13T05:41:47.767117+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T06:04:29.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.230+0000 7fa1558e7700 1 -- 172.21.15.32:0/140743054 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa13806cc20 msgr2=0x7fa13806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:29.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.230+0000 7fa1558e7700 1 --2- 172.21.15.32:0/140743054 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa13806cc20 0x7fa13806f0d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa144004f90 tx=0x7fa144009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:29.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.230+0000 7fa1558e7700 1 -- 172.21.15.32:0/140743054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1500fc3d0 msgr2=0x7fa1500fd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:29.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.230+0000 7fa1558e7700 1 --2- 172.21.15.32:0/140743054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1500fc3d0 0x7fa1500fd8b0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fa140002a80 tx=0x7fa1400046b0 comp rx=0 tx=0).stop 2024-04-12T06:04:29.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.230+0000 7fa1558e7700 1 -- 172.21.15.32:0/140743054 shutdown_connections 2024-04-12T06:04:29.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.230+0000 7fa1558e7700 1 --2- 172.21.15.32:0/140743054 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa13806cc20 0x7fa13806f0d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:29.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.230+0000 7fa1558e7700 1 --2- 172.21.15.32:0/140743054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1500fc3d0 0x7fa1500fd8b0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:29.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.230+0000 7fa1558e7700 1 --2- 172.21.15.32:0/140743054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa150101c70 0x7fa1500fddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:29.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.230+0000 7fa1558e7700 1 -- 172.21.15.32:0/140743054 >> 172.21.15.32:0/140743054 conn(0x7fa150073290 msgr2=0x7fa150100280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:29.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.230+0000 7fa1558e7700 1 -- 172.21.15.32:0/140743054 shutdown_connections 2024-04-12T06:04:29.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.230+0000 7fa1558e7700 1 -- 172.21.15.32:0/140743054 wait complete. 2024-04-12T06:04:29.239 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-12T05:31:39.568876+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-12T05:34:21.574210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-12T05:34:22.056623+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-12T06:04:29.239 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-12T06:04:29.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:28 smithi067 ceph-mon[112979]: pgmap v719: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:04:29.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:28 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1065445076' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 37, "format": "json"}]: dispatch 2024-04-12T06:04:29.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:28 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3679810020' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 38, "format": "json"}]: dispatch 2024-04-12T06:04:29.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:28 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/912140040' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 39, "format": "json"}]: dispatch 2024-04-12T06:04:29.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.368+0000 7f220d434700 1 -- 172.21.15.32:0/1594929285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22080fd7b0 msgr2=0x7f22080fdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:29.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.368+0000 7f220d434700 1 --2- 172.21.15.32:0/1594929285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22080fd7b0 0x7f22080fdc20 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f21f8004940 tx=0x7f21f8009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:29.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.368+0000 7f220d434700 1 -- 172.21.15.32:0/1594929285 shutdown_connections 2024-04-12T06:04:29.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.368+0000 7f220d434700 1 --2- 172.21.15.32:0/1594929285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22080fd7b0 0x7f22080fdc20 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:29.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.368+0000 7f220d434700 1 --2- 172.21.15.32:0/1594929285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22080fbfb0 0x7f22080fc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:29.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.368+0000 7f220d434700 1 -- 172.21.15.32:0/1594929285 >> 172.21.15.32:0/1594929285 conn(0x7f22080f7730 msgr2=0x7f22080f9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:29.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.368+0000 7f220d434700 1 -- 172.21.15.32:0/1594929285 shutdown_connections 2024-04-12T06:04:29.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.368+0000 7f220d434700 1 -- 172.21.15.32:0/1594929285 wait complete. 2024-04-12T06:04:29.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.369+0000 7f220d434700 1 Processor -- start 2024-04-12T06:04:29.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.369+0000 7f220d434700 1 -- start start 2024-04-12T06:04:29.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.369+0000 7f220d434700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22080fbfb0 0x7f22081136e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:29.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.369+0000 7f220d434700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22080fd7b0 0x7f2208113c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:29.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.369+0000 7f220d434700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2208114270 con 0x7f22080fd7b0 2024-04-12T06:04:29.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.369+0000 7f220d434700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220810d890 con 0x7f22080fbfb0 2024-04-12T06:04:29.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.369+0000 7f22067fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22080fd7b0 0x7f2208113c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:29.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.369+0000 7f22067fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22080fd7b0 0x7f2208113c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51968/0 (socket says 172.21.15.32:51968) 2024-04-12T06:04:29.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.369+0000 7f22067fc700 1 -- 172.21.15.32:0/8381654 learned_addr learned my addr 172.21.15.32:0/8381654 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:29.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.369+0000 7f2206ffd700 1 --2- 172.21.15.32:0/8381654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22080fbfb0 0x7f22081136e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:29.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.369+0000 7f22067fc700 1 -- 172.21.15.32:0/8381654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22080fbfb0 msgr2=0x7f22081136e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:29.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.369+0000 7f22067fc700 1 --2- 172.21.15.32:0/8381654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22080fbfb0 0x7f22081136e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:29.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.369+0000 7f22067fc700 1 -- 172.21.15.32:0/8381654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f8005040 con 0x7f22080fd7b0 2024-04-12T06:04:29.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.369+0000 7f22067fc700 1 --2- 172.21.15.32:0/8381654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22080fd7b0 0x7f2208113c20 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f21f8000c00 tx=0x7f21f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:29.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.370+0000 7f21ef7fe700 1 -- 172.21.15.32:0/8381654 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21f8021070 con 0x7f22080fd7b0 2024-04-12T06:04:29.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.370+0000 7f21ef7fe700 1 -- 172.21.15.32:0/8381654 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f21f80093b0 con 0x7f22080fd7b0 2024-04-12T06:04:29.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.370+0000 7f220d434700 1 -- 172.21.15.32:0/8381654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f220810db10 con 0x7f22080fd7b0 2024-04-12T06:04:29.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.370+0000 7f21ef7fe700 1 -- 172.21.15.32:0/8381654 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21f800e070 con 0x7f22080fd7b0 2024-04-12T06:04:29.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.370+0000 7f220d434700 1 -- 172.21.15.32:0/8381654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f220810e0d0 con 0x7f22080fd7b0 2024-04-12T06:04:29.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.371+0000 7f220d434700 1 -- 172.21.15.32:0/8381654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22080fd350 con 0x7f22080fd7b0 2024-04-12T06:04:29.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.371+0000 7f21ef7fe700 1 -- 172.21.15.32:0/8381654 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f21f801c490 con 0x7f22080fd7b0 2024-04-12T06:04:29.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.372+0000 7f21ef7fe700 1 --2- 172.21.15.32:0/8381654 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f21f006cc20 0x7f21f006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:29.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.372+0000 7f21ef7fe700 1 -- 172.21.15.32:0/8381654 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f21f808d170 con 0x7f22080fd7b0 2024-04-12T06:04:29.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.372+0000 7f2206ffd700 1 --2- 172.21.15.32:0/8381654 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f21f006cc20 0x7f21f006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:29.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.372+0000 7f2206ffd700 1 --2- 172.21.15.32:0/8381654 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f21f006cc20 0x7f21f006f0d0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f21fc009b80 tx=0x7f21fc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:29.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.376+0000 7f21ef7fe700 1 -- 172.21.15.32:0/8381654 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f21f805abe0 con 0x7f22080fd7b0 2024-04-12T06:04:29.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.583+0000 7f220d434700 1 -- 172.21.15.32:0/8381654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f220810ed70 con 0x7f22080fd7b0 2024-04-12T06:04:29.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.583+0000 7f21ef7fe700 1 -- 172.21.15.32:0/8381654 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f21f805a770 con 0x7f22080fd7b0 2024-04-12T06:04:29.580 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-12T06:04:29.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.585+0000 7f220d434700 1 -- 172.21.15.32:0/8381654 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f21f006cc20 msgr2=0x7f21f006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:29.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.585+0000 7f220d434700 1 --2- 172.21.15.32:0/8381654 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f21f006cc20 0x7f21f006f0d0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f21fc009b80 tx=0x7f21fc00b040 comp rx=0 tx=0).stop 2024-04-12T06:04:29.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.585+0000 7f220d434700 1 -- 172.21.15.32:0/8381654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22080fd7b0 msgr2=0x7f2208113c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:29.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.585+0000 7f220d434700 1 --2- 172.21.15.32:0/8381654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22080fd7b0 0x7f2208113c20 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f21f8000c00 tx=0x7f21f8009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:29.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.585+0000 7f220d434700 1 -- 172.21.15.32:0/8381654 shutdown_connections 2024-04-12T06:04:29.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.585+0000 7f220d434700 1 --2- 172.21.15.32:0/8381654 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f21f006cc20 0x7f21f006f0d0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:29.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.585+0000 7f220d434700 1 --2- 172.21.15.32:0/8381654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22080fbfb0 0x7f22081136e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:29.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.585+0000 7f220d434700 1 --2- 172.21.15.32:0/8381654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22080fd7b0 0x7f2208113c20 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:29.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.585+0000 7f220d434700 1 -- 172.21.15.32:0/8381654 >> 172.21.15.32:0/8381654 conn(0x7f22080f7730 msgr2=0x7f2208066130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:29.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.586+0000 7f220d434700 1 -- 172.21.15.32:0/8381654 shutdown_connections 2024-04-12T06:04:29.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.586+0000 7f220d434700 1 -- 172.21.15.32:0/8381654 wait complete. 2024-04-12T06:04:29.594 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-12T06:04:29.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.723+0000 7f3a2ef4d700 1 -- 172.21.15.32:0/553311590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a28102580 msgr2=0x7f3a280fbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:29.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.723+0000 7f3a2ef4d700 1 --2- 172.21.15.32:0/553311590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a28102580 0x7f3a280fbbf0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f3a14004940 tx=0x7f3a14009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:29.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.723+0000 7f3a2ef4d700 1 -- 172.21.15.32:0/553311590 shutdown_connections 2024-04-12T06:04:29.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.723+0000 7f3a2ef4d700 1 --2- 172.21.15.32:0/553311590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a28102580 0x7f3a280fbbf0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:29.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.723+0000 7f3a2ef4d700 1 --2- 172.21.15.32:0/553311590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a28101be0 0x7f3a28101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.723+0000 7f3a2ef4d700 1 -- 172.21.15.32:0/553311590 >> 172.21.15.32:0/553311590 conn(0x7f3a28072ad0 msgr2=0x7f3a28072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.723+0000 7f3a2ef4d700 1 -- 172.21.15.32:0/553311590 shutdown_connections 2024-04-12T06:04:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.723+0000 7f3a2ef4d700 1 -- 172.21.15.32:0/553311590 wait complete. 2024-04-12T06:04:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.724+0000 7f3a2ef4d700 1 Processor -- start 2024-04-12T06:04:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.724+0000 7f3a2ef4d700 1 -- start start 2024-04-12T06:04:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.724+0000 7f3a2ef4d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a28101be0 0x7f3a281126a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.724+0000 7f3a2ef4d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a28102580 0x7f3a28112be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.724+0000 7f3a2ef4d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a28113230 con 0x7f3a28102580 2024-04-12T06:04:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.724+0000 7f3a2ef4d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2810c7f0 con 0x7f3a28101be0 2024-04-12T06:04:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.724+0000 7f3a27fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a28102580 0x7f3a28112be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.724+0000 7f3a27fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a28102580 0x7f3a28112be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51998/0 (socket says 172.21.15.32:51998) 2024-04-12T06:04:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.724+0000 7f3a27fff700 1 -- 172.21.15.32:0/3083245839 learned_addr learned my addr 172.21.15.32:0/3083245839 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.724+0000 7f3a2cce9700 1 --2- 172.21.15.32:0/3083245839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a28101be0 0x7f3a281126a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:29.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.724+0000 7f3a27fff700 1 -- 172.21.15.32:0/3083245839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a28101be0 msgr2=0x7f3a281126a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:29.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.724+0000 7f3a27fff700 1 --2- 172.21.15.32:0/3083245839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a28101be0 0x7f3a281126a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:29.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.724+0000 7f3a27fff700 1 -- 172.21.15.32:0/3083245839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a14005040 con 0x7f3a28102580 2024-04-12T06:04:29.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.725+0000 7f3a27fff700 1 --2- 172.21.15.32:0/3083245839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a28102580 0x7f3a28112be0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f3a1400b3b0 tx=0x7f3a14009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:29.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.725+0000 7f3a257fa700 1 -- 172.21.15.32:0/3083245839 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a1400e040 con 0x7f3a28102580 2024-04-12T06:04:29.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.725+0000 7f3a257fa700 1 -- 172.21.15.32:0/3083245839 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3a1400bc90 con 0x7f3a28102580 2024-04-12T06:04:29.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.725+0000 7f3a257fa700 1 -- 172.21.15.32:0/3083245839 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a14013660 con 0x7f3a28102580 2024-04-12T06:04:29.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.725+0000 7f3a2ef4d700 1 -- 172.21.15.32:0/3083245839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a2810ca70 con 0x7f3a28102580 2024-04-12T06:04:29.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.725+0000 7f3a2ef4d700 1 -- 172.21.15.32:0/3083245839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a2810cf00 con 0x7f3a28102580 2024-04-12T06:04:29.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.726+0000 7f3a2ef4d700 1 -- 172.21.15.32:0/3083245839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a28112330 con 0x7f3a28102580 2024-04-12T06:04:29.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.726+0000 7f3a257fa700 1 -- 172.21.15.32:0/3083245839 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3a14013920 con 0x7f3a28102580 2024-04-12T06:04:29.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.727+0000 7f3a257fa700 1 --2- 172.21.15.32:0/3083245839 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3a1806cc20 0x7f3a1806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:29.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.727+0000 7f3a257fa700 1 -- 172.21.15.32:0/3083245839 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f3a1408ff60 con 0x7f3a28102580 2024-04-12T06:04:29.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.727+0000 7f3a2cce9700 1 --2- 172.21.15.32:0/3083245839 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3a1806cc20 0x7f3a1806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:29.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.727+0000 7f3a2cce9700 1 --2- 172.21.15.32:0/3083245839 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3a1806cc20 0x7f3a1806f0d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c0041d0 tx=0x7f3a1c0043c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:29.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.731+0000 7f3a257fa700 1 -- 172.21.15.32:0/3083245839 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3a140c3ab0 con 0x7f3a28102580 2024-04-12T06:04:29.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.934+0000 7f3a2ef4d700 1 -- 172.21.15.32:0/3083245839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f3a280fd330 con 0x7f3a28102580 2024-04-12T06:04:29.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.934+0000 7f3a257fa700 1 -- 172.21.15.32:0/3083245839 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v85) v1 ==== 107+0+11 (secure 0 0 0) 0x7f3a1405d950 con 0x7f3a28102580 2024-04-12T06:04:29.931 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-12T06:04:29.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.936+0000 7f3a2ef4d700 1 -- 172.21.15.32:0/3083245839 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3a1806cc20 msgr2=0x7f3a1806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:29.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.936+0000 7f3a2ef4d700 1 --2- 172.21.15.32:0/3083245839 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3a1806cc20 0x7f3a1806f0d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c0041d0 tx=0x7f3a1c0043c0 comp rx=0 tx=0).stop 2024-04-12T06:04:29.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.936+0000 7f3a2ef4d700 1 -- 172.21.15.32:0/3083245839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a28102580 msgr2=0x7f3a28112be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:29.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.936+0000 7f3a2ef4d700 1 --2- 172.21.15.32:0/3083245839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a28102580 0x7f3a28112be0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f3a1400b3b0 tx=0x7f3a14009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:29.933 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.937+0000 7f3a2ef4d700 1 -- 172.21.15.32:0/3083245839 shutdown_connections 2024-04-12T06:04:29.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.937+0000 7f3a2ef4d700 1 --2- 172.21.15.32:0/3083245839 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f3a1806cc20 0x7f3a1806f0d0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:29.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.937+0000 7f3a2ef4d700 1 --2- 172.21.15.32:0/3083245839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a28101be0 0x7f3a281126a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:29.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.937+0000 7f3a2ef4d700 1 --2- 172.21.15.32:0/3083245839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a28102580 0x7f3a28112be0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:29.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.937+0000 7f3a2ef4d700 1 -- 172.21.15.32:0/3083245839 >> 172.21.15.32:0/3083245839 conn(0x7f3a28072ad0 msgr2=0x7f3a28104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:29.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.937+0000 7f3a2ef4d700 1 -- 172.21.15.32:0/3083245839 shutdown_connections 2024-04-12T06:04:29.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:29.937+0000 7f3a2ef4d700 1 -- 172.21.15.32:0/3083245839 wait complete. 2024-04-12T06:04:29.946 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-12T06:04:30.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.075+0000 7f63ce435700 1 -- 172.21.15.32:0/2613842497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c8101be0 msgr2=0x7f63c8101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:30.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.075+0000 7f63ce435700 1 --2- 172.21.15.32:0/2613842497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c8101be0 0x7f63c8101fb0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f63b8004940 tx=0x7f63b8009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:30.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.075+0000 7f63ce435700 1 -- 172.21.15.32:0/2613842497 shutdown_connections 2024-04-12T06:04:30.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.075+0000 7f63ce435700 1 --2- 172.21.15.32:0/2613842497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c8102580 0x7f63c80fbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:30.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.075+0000 7f63ce435700 1 --2- 172.21.15.32:0/2613842497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c8101be0 0x7f63c8101fb0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:30.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.075+0000 7f63ce435700 1 -- 172.21.15.32:0/2613842497 >> 172.21.15.32:0/2613842497 conn(0x7f63c80f7710 msgr2=0x7f63c80f9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:30.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.075+0000 7f63ce435700 1 -- 172.21.15.32:0/2613842497 shutdown_connections 2024-04-12T06:04:30.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.075+0000 7f63ce435700 1 -- 172.21.15.32:0/2613842497 wait complete. 2024-04-12T06:04:30.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.075+0000 7f63ce435700 1 Processor -- start 2024-04-12T06:04:30.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.076+0000 7f63ce435700 1 -- start start 2024-04-12T06:04:30.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.076+0000 7f63ce435700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c8101be0 0x7f63c8113760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:30.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.076+0000 7f63ce435700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c8102580 0x7f63c8113ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:30.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.076+0000 7f63ce435700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c81142f0 con 0x7f63c8101be0 2024-04-12T06:04:30.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.076+0000 7f63ce435700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c810d8f0 con 0x7f63c8102580 2024-04-12T06:04:30.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.076+0000 7f63c7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c8101be0 0x7f63c8113760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:30.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.076+0000 7f63c7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c8101be0 0x7f63c8113760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52028/0 (socket says 172.21.15.32:52028) 2024-04-12T06:04:30.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.076+0000 7f63c7fff700 1 -- 172.21.15.32:0/1816640611 learned_addr learned my addr 172.21.15.32:0/1816640611 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:30.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.076+0000 7f63c77fe700 1 --2- 172.21.15.32:0/1816640611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c8102580 0x7f63c8113ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:30.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.076+0000 7f63c7fff700 1 -- 172.21.15.32:0/1816640611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c8102580 msgr2=0x7f63c8113ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:30.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.076+0000 7f63c7fff700 1 --2- 172.21.15.32:0/1816640611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c8102580 0x7f63c8113ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:30.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.076+0000 7f63c7fff700 1 -- 172.21.15.32:0/1816640611 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b8005040 con 0x7f63c8101be0 2024-04-12T06:04:30.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.076+0000 7f63c77fe700 1 --2- 172.21.15.32:0/1816640611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c8102580 0x7f63c8113ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T06:04:30.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.076+0000 7f63c7fff700 1 --2- 172.21.15.32:0/1816640611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c8101be0 0x7f63c8113760 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f63b800b3b0 tx=0x7f63b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:30.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.076+0000 7f63c4ff9700 1 -- 172.21.15.32:0/1816640611 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63b800e040 con 0x7f63c8101be0 2024-04-12T06:04:30.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.077+0000 7f63c4ff9700 1 -- 172.21.15.32:0/1816640611 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f63b8013420 con 0x7f63c8101be0 2024-04-12T06:04:30.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.077+0000 7f63ce435700 1 -- 172.21.15.32:0/1816640611 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c810db70 con 0x7f63c8101be0 2024-04-12T06:04:30.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.077+0000 7f63c4ff9700 1 -- 172.21.15.32:0/1816640611 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63b801c630 con 0x7f63c8101be0 2024-04-12T06:04:30.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.077+0000 7f63ce435700 1 -- 172.21.15.32:0/1816640611 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c810e130 con 0x7f63c8101be0 2024-04-12T06:04:30.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.078+0000 7f63ce435700 1 -- 172.21.15.32:0/1816640611 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c80fd330 con 0x7f63c8101be0 2024-04-12T06:04:30.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.078+0000 7f63c4ff9700 1 -- 172.21.15.32:0/1816640611 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f63b8013970 con 0x7f63c8101be0 2024-04-12T06:04:30.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.078+0000 7f63c4ff9700 1 --2- 172.21.15.32:0/1816640611 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f63b406cc20 0x7f63b406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:30.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.079+0000 7f63c4ff9700 1 -- 172.21.15.32:0/1816640611 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f63b808d310 con 0x7f63c8101be0 2024-04-12T06:04:30.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.079+0000 7f63c77fe700 1 --2- 172.21.15.32:0/1816640611 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f63b406cc20 0x7f63b406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:30.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.079+0000 7f63c77fe700 1 --2- 172.21.15.32:0/1816640611 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f63b406cc20 0x7f63b406f0d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f63b0009fd0 tx=0x7f63b0009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:30.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.083+0000 7f63c4ff9700 1 -- 172.21.15.32:0/1816640611 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f63b805ad80 con 0x7f63c8101be0 2024-04-12T06:04:30.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:29 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/140743054' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T06:04:30.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:29 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/8381654' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T06:04:30.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:29 smithi067 ceph-mon[112979]: pgmap v720: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:04:30.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:29 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/3083245839' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-12T06:04:30.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.286+0000 7f63ce435700 1 -- 172.21.15.32:0/1816640611 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f63c8108be0 con 0x7f63c8101be0 2024-04-12T06:04:30.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.287+0000 7f63c4ff9700 1 -- 172.21.15.32:0/1816640611 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v85) v1 ==== 107+0+11 (secure 0 0 0) 0x7f63b805a910 con 0x7f63c8101be0 2024-04-12T06:04:30.283 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-12T06:04:30.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.288+0000 7f63ce435700 1 -- 172.21.15.32:0/1816640611 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f63b406cc20 msgr2=0x7f63b406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:30.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.288+0000 7f63ce435700 1 --2- 172.21.15.32:0/1816640611 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f63b406cc20 0x7f63b406f0d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f63b0009fd0 tx=0x7f63b0009ee0 comp rx=0 tx=0).stop 2024-04-12T06:04:30.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.289+0000 7f63ce435700 1 -- 172.21.15.32:0/1816640611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c8101be0 msgr2=0x7f63c8113760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:30.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.289+0000 7f63ce435700 1 --2- 172.21.15.32:0/1816640611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c8101be0 0x7f63c8113760 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f63b800b3b0 tx=0x7f63b8009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:30.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.289+0000 7f63ce435700 1 -- 172.21.15.32:0/1816640611 shutdown_connections 2024-04-12T06:04:30.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.289+0000 7f63ce435700 1 --2- 172.21.15.32:0/1816640611 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f63b406cc20 0x7f63b406f0d0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:30.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.289+0000 7f63ce435700 1 --2- 172.21.15.32:0/1816640611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c8101be0 0x7f63c8113760 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:30.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.289+0000 7f63ce435700 1 --2- 172.21.15.32:0/1816640611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c8102580 0x7f63c8113ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:30.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.289+0000 7f63ce435700 1 -- 172.21.15.32:0/1816640611 >> 172.21.15.32:0/1816640611 conn(0x7f63c80f7710 msgr2=0x7f63c80f87a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:30.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.289+0000 7f63ce435700 1 -- 172.21.15.32:0/1816640611 shutdown_connections 2024-04-12T06:04:30.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.289+0000 7f63ce435700 1 -- 172.21.15.32:0/1816640611 wait complete. 2024-04-12T06:04:30.296 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:29 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/140743054' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T06:04:30.296 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:29 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/8381654' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T06:04:30.296 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:29 smithi032 ceph-mon[146817]: pgmap v720: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:04:30.296 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:29 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/3083245839' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-12T06:04:30.298 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-12T06:04:30.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.425+0000 7f0a75603700 1 -- 172.21.15.32:0/860449336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a700fdd00 msgr2=0x7f0a700fe0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:30.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.425+0000 7f0a75603700 1 --2- 172.21.15.32:0/860449336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a700fdd00 0x7f0a700fe0d0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f0a60004940 tx=0x7f0a60009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:30.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.425+0000 7f0a75603700 1 -- 172.21.15.32:0/860449336 shutdown_connections 2024-04-12T06:04:30.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.425+0000 7f0a75603700 1 --2- 172.21.15.32:0/860449336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a700fe6a0 0x7f0a70106ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:30.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.425+0000 7f0a75603700 1 --2- 172.21.15.32:0/860449336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a700fdd00 0x7f0a700fe0d0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:30.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.425+0000 7f0a75603700 1 -- 172.21.15.32:0/860449336 >> 172.21.15.32:0/860449336 conn(0x7f0a700f7770 msgr2=0x7f0a700f9b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:30.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.425+0000 7f0a75603700 1 -- 172.21.15.32:0/860449336 shutdown_connections 2024-04-12T06:04:30.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.425+0000 7f0a75603700 1 -- 172.21.15.32:0/860449336 wait complete. 2024-04-12T06:04:30.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.426+0000 7f0a75603700 1 Processor -- start 2024-04-12T06:04:30.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.426+0000 7f0a75603700 1 -- start start 2024-04-12T06:04:30.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.427+0000 7f0a75603700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a700fdd00 0x7f0a701137b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:30.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.427+0000 7f0a75603700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a700fe6a0 0x7f0a70113cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:30.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.427+0000 7f0a75603700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a70114340 con 0x7f0a700fe6a0 2024-04-12T06:04:30.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.427+0000 7f0a75603700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7010d940 con 0x7f0a700fdd00 2024-04-12T06:04:30.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.427+0000 7f0a6e7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a700fe6a0 0x7f0a70113cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:30.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.427+0000 7f0a6e7fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a700fe6a0 0x7f0a70113cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52060/0 (socket says 172.21.15.32:52060) 2024-04-12T06:04:30.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.427+0000 7f0a6e7fc700 1 -- 172.21.15.32:0/916336801 learned_addr learned my addr 172.21.15.32:0/916336801 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:30.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.427+0000 7f0a6effd700 1 --2- 172.21.15.32:0/916336801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a700fdd00 0x7f0a701137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:30.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.427+0000 7f0a6e7fc700 1 -- 172.21.15.32:0/916336801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a700fdd00 msgr2=0x7f0a701137b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:30.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.427+0000 7f0a6e7fc700 1 --2- 172.21.15.32:0/916336801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a700fdd00 0x7f0a701137b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:30.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.427+0000 7f0a6e7fc700 1 -- 172.21.15.32:0/916336801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a60005040 con 0x7f0a700fe6a0 2024-04-12T06:04:30.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.427+0000 7f0a6effd700 1 --2- 172.21.15.32:0/916336801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a700fdd00 0x7f0a701137b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T06:04:30.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.427+0000 7f0a6e7fc700 1 --2- 172.21.15.32:0/916336801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a700fe6a0 0x7f0a70113cf0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f0a6400c8c0 tx=0x7f0a640044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:30.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.427+0000 7f0a577fe700 1 -- 172.21.15.32:0/916336801 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a6400a980 con 0x7f0a700fe6a0 2024-04-12T06:04:30.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.427+0000 7f0a577fe700 1 -- 172.21.15.32:0/916336801 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0a6400ce60 con 0x7f0a700fe6a0 2024-04-12T06:04:30.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.427+0000 7f0a75603700 1 -- 172.21.15.32:0/916336801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a7010dc20 con 0x7f0a700fe6a0 2024-04-12T06:04:30.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.427+0000 7f0a577fe700 1 -- 172.21.15.32:0/916336801 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a640113a0 con 0x7f0a700fe6a0 2024-04-12T06:04:30.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.427+0000 7f0a75603700 1 -- 172.21.15.32:0/916336801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a7010e160 con 0x7f0a700fe6a0 2024-04-12T06:04:30.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.428+0000 7f0a75603700 1 -- 172.21.15.32:0/916336801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a70066ab0 con 0x7f0a700fe6a0 2024-04-12T06:04:30.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.429+0000 7f0a577fe700 1 -- 172.21.15.32:0/916336801 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f0a64012400 con 0x7f0a700fe6a0 2024-04-12T06:04:30.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.429+0000 7f0a577fe700 1 --2- 172.21.15.32:0/916336801 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0a5806cc20 0x7f0a5806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:30.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.430+0000 7f0a577fe700 1 -- 172.21.15.32:0/916336801 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f0a6408c1c0 con 0x7f0a700fe6a0 2024-04-12T06:04:30.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.430+0000 7f0a6effd700 1 --2- 172.21.15.32:0/916336801 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0a5806cc20 0x7f0a5806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:30.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.430+0000 7f0a6effd700 1 --2- 172.21.15.32:0/916336801 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0a5806cc20 0x7f0a5806f0d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0a6000b3b0 tx=0x7f0a60009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:30.430 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.434+0000 7f0a577fe700 1 -- 172.21.15.32:0/916336801 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0a64059c30 con 0x7f0a700fe6a0 2024-04-12T06:04:30.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.653+0000 7f0a75603700 1 -- 172.21.15.32:0/916336801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0a70109c60 con 0x7f0a700fe6a0 2024-04-12T06:04:30.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.653+0000 7f0a577fe700 1 -- 172.21.15.32:0/916336801 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 41 entries v85) v1 ==== 81+0+2415 (secure 0 0 0) 0x7f0a640597c0 con 0x7f0a700fe6a0 2024-04-12T06:04:30.649 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6829/387754149 2024-04-13T05:46:10.668153+0000 2024-04-12T06:04:30.650 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6828/387754149 2024-04-13T05:46:10.668153+0000 2024-04-12T06:04:30.650 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.67:6825/1291914599 2024-04-13T05:46:10.668178+0000 2024-04-12T06:04:30.650 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.67:6824/1291914599 2024-04-13T05:46:10.668178+0000 2024-04-12T06:04:30.650 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/138657961 2024-04-13T05:27:40.952118+0000 2024-04-12T06:04:30.650 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.67:0/1683579802 2024-04-13T05:38:47.052969+0000 2024-04-12T06:04:30.650 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/368340628 2024-04-13T05:37:39.471472+0000 2024-04-12T06:04:30.650 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6801/3971961003 2024-04-13T05:29:25.054120+0000 2024-04-12T06:04:30.650 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/652449107 2024-04-13T05:41:47.767117+0000 2024-04-12T06:04:30.650 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/4112976740 2024-04-13T05:29:25.054120+0000 2024-04-12T06:04:30.650 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/2247439109 2024-04-13T05:37:39.471472+0000 2024-04-12T06:04:30.650 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/1643708423 2024-04-13T05:27:40.952118+0000 2024-04-12T06:04:30.650 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6800/2564765508 2024-04-13T05:41:47.767117+0000 2024-04-12T06:04:30.650 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.67:6828/3925433122 2024-04-13T05:38:47.052969+0000 2024-04-12T06:04:30.651 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6800/1962278696 2024-04-13T05:28:14.470413+0000 2024-04-12T06:04:30.651 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/2162993076 2024-04-13T05:28:14.470413+0000 2024-04-12T06:04:30.651 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6827/2962187015 2024-04-13T05:34:37.113413+0000 2024-04-12T06:04:30.651 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6800/3971961003 2024-04-13T05:29:25.054120+0000 2024-04-12T06:04:30.651 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/2105642006 2024-04-13T05:37:39.471472+0000 2024-04-12T06:04:30.651 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.67:0/31773612 2024-04-13T05:38:47.052969+0000 2024-04-12T06:04:30.651 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.67:6829/3925433122 2024-04-13T05:38:47.052969+0000 2024-04-12T06:04:30.651 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.67:0/2846171310 2024-04-13T05:38:47.052969+0000 2024-04-12T06:04:30.651 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/3370471392 2024-04-13T05:41:47.767117+0000 2024-04-12T06:04:30.651 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/3082743475 2024-04-13T05:28:14.470413+0000 2024-04-12T06:04:30.651 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6801/656201914 2024-04-13T05:27:40.952118+0000 2024-04-12T06:04:30.651 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6801/1962278696 2024-04-13T05:28:14.470413+0000 2024-04-12T06:04:30.651 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/260214199 2024-04-13T05:37:39.471472+0000 2024-04-12T06:04:30.651 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6800/656201914 2024-04-13T05:27:40.952118+0000 2024-04-12T06:04:30.651 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.67:0/3462505447 2024-04-13T05:38:47.052969+0000 2024-04-12T06:04:30.652 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/674029871 2024-04-13T05:29:25.054120+0000 2024-04-12T06:04:30.652 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.67:0/125735613 2024-04-13T05:38:47.052969+0000 2024-04-12T06:04:30.652 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.67:0/3697060296 2024-04-13T05:38:47.052969+0000 2024-04-12T06:04:30.652 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/4031244002 2024-04-13T05:27:40.952118+0000 2024-04-12T06:04:30.652 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6801/2564765508 2024-04-13T05:41:47.767117+0000 2024-04-12T06:04:30.652 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/795906028 2024-04-13T05:29:25.054120+0000 2024-04-12T06:04:30.652 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6826/2962187015 2024-04-13T05:34:37.113413+0000 2024-04-12T06:04:30.652 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6801/595990262 2024-04-13T05:37:39.471472+0000 2024-04-12T06:04:30.652 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/146872823 2024-04-13T05:28:14.470413+0000 2024-04-12T06:04:30.652 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/138775819 2024-04-13T05:41:47.767117+0000 2024-04-12T06:04:30.652 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6800/595990262 2024-04-13T05:37:39.471472+0000 2024-04-12T06:04:30.652 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/2743601944 2024-04-13T05:41:47.767117+0000 2024-04-12T06:04:30.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.655+0000 7f0a75603700 1 -- 172.21.15.32:0/916336801 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0a5806cc20 msgr2=0x7f0a5806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:30.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.655+0000 7f0a75603700 1 --2- 172.21.15.32:0/916336801 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0a5806cc20 0x7f0a5806f0d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0a6000b3b0 tx=0x7f0a60009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:30.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.655+0000 7f0a75603700 1 -- 172.21.15.32:0/916336801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a700fe6a0 msgr2=0x7f0a70113cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:30.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.655+0000 7f0a75603700 1 --2- 172.21.15.32:0/916336801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a700fe6a0 0x7f0a70113cf0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f0a6400c8c0 tx=0x7f0a640044c0 comp rx=0 tx=0).stop 2024-04-12T06:04:30.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.655+0000 7f0a75603700 1 -- 172.21.15.32:0/916336801 shutdown_connections 2024-04-12T06:04:30.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.655+0000 7f0a75603700 1 --2- 172.21.15.32:0/916336801 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f0a5806cc20 0x7f0a5806f0d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:30.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.655+0000 7f0a75603700 1 --2- 172.21.15.32:0/916336801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a700fdd00 0x7f0a701137b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:30.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.655+0000 7f0a75603700 1 --2- 172.21.15.32:0/916336801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a700fe6a0 0x7f0a70113cf0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:30.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.655+0000 7f0a75603700 1 -- 172.21.15.32:0/916336801 >> 172.21.15.32:0/916336801 conn(0x7f0a700f7770 msgr2=0x7f0a700f8e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:30.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.655+0000 7f0a75603700 1 -- 172.21.15.32:0/916336801 shutdown_connections 2024-04-12T06:04:30.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.655+0000 7f0a75603700 1 -- 172.21.15.32:0/916336801 wait complete. 2024-04-12T06:04:30.654 INFO:teuthology.orchestra.run.smithi032.stderr:listed 41 entries 2024-04-12T06:04:30.664 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-12T06:04:30.664 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-12T06:04:30.684 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-12T06:04:30.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.854+0000 7f6190a71700 1 -- 172.21.15.32:0/1562649539 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f618c101c70 msgr2=0x7f618c102040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:30.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.854+0000 7f6190a71700 1 --2- 172.21.15.32:0/1562649539 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f618c101c70 0x7f618c102040 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f617c004940 tx=0x7f617c009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:30.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.855+0000 7f6190a71700 1 -- 172.21.15.32:0/1562649539 shutdown_connections 2024-04-12T06:04:30.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.855+0000 7f6190a71700 1 --2- 172.21.15.32:0/1562649539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f618c102610 0x7f618c0fbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:30.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.855+0000 7f6190a71700 1 --2- 172.21.15.32:0/1562649539 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f618c101c70 0x7f618c102040 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:30.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.855+0000 7f6190a71700 1 -- 172.21.15.32:0/1562649539 >> 172.21.15.32:0/1562649539 conn(0x7f618c071c80 msgr2=0x7f618c072080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:30.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.855+0000 7f6190a71700 1 -- 172.21.15.32:0/1562649539 shutdown_connections 2024-04-12T06:04:30.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.855+0000 7f6190a71700 1 -- 172.21.15.32:0/1562649539 wait complete. 2024-04-12T06:04:30.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.855+0000 7f6190a71700 1 Processor -- start 2024-04-12T06:04:30.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.855+0000 7f6190a71700 1 -- start start 2024-04-12T06:04:30.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.855+0000 7f6190a71700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f618c101c70 0x7f618c112660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:30.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.855+0000 7f6190a71700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f618c102610 0x7f618c112be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:30.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.855+0000 7f6190a71700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f618c113230 con 0x7f618c102610 2024-04-12T06:04:30.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.855+0000 7f6190a71700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f618c10c7f0 con 0x7f618c101c70 2024-04-12T06:04:30.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.855+0000 7f6183fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f618c102610 0x7f618c112be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:30.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.855+0000 7f6183fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f618c102610 0x7f618c112be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52082/0 (socket says 172.21.15.32:52082) 2024-04-12T06:04:30.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.855+0000 7f6183fff700 1 -- 172.21.15.32:0/1799509527 learned_addr learned my addr 172.21.15.32:0/1799509527 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:30.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.856+0000 7f6183fff700 1 -- 172.21.15.32:0/1799509527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f618c101c70 msgr2=0x7f618c112660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:30.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.856+0000 7f6183fff700 1 --2- 172.21.15.32:0/1799509527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f618c101c70 0x7f618c112660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:30.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.856+0000 7f6183fff700 1 -- 172.21.15.32:0/1799509527 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617c005040 con 0x7f618c102610 2024-04-12T06:04:30.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.856+0000 7f6183fff700 1 --2- 172.21.15.32:0/1799509527 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f618c102610 0x7f618c112be0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f6174002a50 tx=0x7f6174004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:30.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.856+0000 7f6182ffd700 1 -- 172.21.15.32:0/1799509527 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6174007cc0 con 0x7f618c102610 2024-04-12T06:04:30.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.856+0000 7f6190a71700 1 -- 172.21.15.32:0/1799509527 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f618c10ca80 con 0x7f618c102610 2024-04-12T06:04:30.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.856+0000 7f6182ffd700 1 -- 172.21.15.32:0/1799509527 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f617400f040 con 0x7f618c102610 2024-04-12T06:04:30.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.856+0000 7f6182ffd700 1 -- 172.21.15.32:0/1799509527 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f617400d9a0 con 0x7f618c102610 2024-04-12T06:04:30.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.856+0000 7f6190a71700 1 -- 172.21.15.32:0/1799509527 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f618c10cfc0 con 0x7f618c102610 2024-04-12T06:04:30.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.857+0000 7f6190a71700 1 -- 172.21.15.32:0/1799509527 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f618c0fef80 con 0x7f618c102610 2024-04-12T06:04:30.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.857+0000 7f6182ffd700 1 -- 172.21.15.32:0/1799509527 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f617400dc60 con 0x7f618c102610 2024-04-12T06:04:30.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.858+0000 7f6182ffd700 1 --2- 172.21.15.32:0/1799509527 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f617806cc20 0x7f617806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:30.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.858+0000 7f6182ffd700 1 -- 172.21.15.32:0/1799509527 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f617408bee0 con 0x7f618c102610 2024-04-12T06:04:30.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.858+0000 7f618a59c700 1 --2- 172.21.15.32:0/1799509527 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f617806cc20 0x7f617806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:30.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.858+0000 7f618a59c700 1 --2- 172.21.15.32:0/1799509527 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f617806cc20 0x7f617806f0d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f617c004f90 tx=0x7f617c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:30.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:30.862+0000 7f6182ffd700 1 -- 172.21.15.32:0/1799509527 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f61740598d0 con 0x7f618c102610 2024-04-12T06:04:31.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.068+0000 7f6190a71700 1 -- 172.21.15.32:0/1799509527 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f618c108c20 con 0x7f618c102610 2024-04-12T06:04:31.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.069+0000 7f6182ffd700 1 -- 172.21.15.32:0/1799509527 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16631 (secure 0 0 0) 0x7f6174059460 con 0x7f618c102610 2024-04-12T06:04:31.066 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-12T06:04:31.066 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":85,"fsid":"37b58ce2-f88d-11ee-bc8c-c7b262605968","created":"2024-04-12T05:27:07.779407+0000","modified":"2024-04-12T05:47:06.905923+0000","last_up_change":"2024-04-12T05:46:08.474847+0000","last_in_change":"2024-04-12T05:32:39.814568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T05:31:39.568876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-12T05:34:21.574210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-12T05:34:22.056623+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c1f69d2a-5e37-47b8-b68f-0568af894b91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":49,"up_from":53,"up_thru":78,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":1783741888},{"type":"v1","addr":"172.21.15.32:6803","nonce":1783741888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":1783741888},{"type":"v1","addr":"172.21.15.32:6805","nonce":1783741888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":1783741888},{"type":"v1","addr":"172.21.15.32:6809","nonce":1783741888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":1783741888},{"type":"v1","addr":"172.21.15.32:6807","nonce":1783741888}]},"public_addr":"172.21.15.32:6803/1783741888","cluster_addr":"172.21.15.32:6805/1783741888","heartbeat_back_addr":"172.21.15.32:6809/1783741888","heartbeat_front_addr":"172.21.15.32:6807/1783741888","state":["exists","up"]},{"osd":1,"uuid":"86b93557-6e6f-4b84-a5f5-5bce4aa65a32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":54,"up_from":58,"up_thru":78,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1060060861},{"type":"v1","addr":"172.21.15.32:6811","nonce":1060060861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1060060861},{"type":"v1","addr":"172.21.15.32:6813","nonce":1060060861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1060060861},{"type":"v1","addr":"172.21.15.32:6817","nonce":1060060861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1060060861},{"type":"v1","addr":"172.21.15.32:6815","nonce":1060060861}]},"public_addr":"172.21.15.32:6811/1060060861","cluster_addr":"172.21.15.32:6813/1060060861","heartbeat_back_addr":"172.21.15.32:6817/1060060861","heartbeat_front_addr":"172.21.15.32:6815/1060060861","state":["exists","up"]},{"osd":2,"uuid":"d1556613-e0de-4b0e-bb7d-e90361ec9007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":59,"up_from":63,"up_thru":78,"down_at":60,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1610966654},{"type":"v1","addr":"172.21.15.32:6819","nonce":1610966654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1610966654},{"type":"v1","addr":"172.21.15.32:6821","nonce":1610966654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1610966654},{"type":"v1","addr":"172.21.15.32:6825","nonce":1610966654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1610966654},{"type":"v1","addr":"172.21.15.32:6823","nonce":1610966654}]},"public_addr":"172.21.15.32:6819/1610966654","cluster_addr":"172.21.15.32:6821/1610966654","heartbeat_back_addr":"172.21.15.32:6825/1610966654","heartbeat_front_addr":"172.21.15.32:6823/1610966654","state":["exists","up"]},{"osd":3,"uuid":"213dc62e-a894-4bc7-a2dd-ca98d1313d6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":64,"up_from":68,"up_thru":78,"down_at":65,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2625997716},{"type":"v1","addr":"172.21.15.67:6801","nonce":2625997716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2625997716},{"type":"v1","addr":"172.21.15.67:6803","nonce":2625997716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":2625997716},{"type":"v1","addr":"172.21.15.67:6807","nonce":2625997716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":2625997716},{"type":"v1","addr":"172.21.15.67:6805","nonce":2625997716}]},"public_addr":"172.21.15.67:6801/2625997716","cluster_addr":"172.21.15.67:6803/2625997716","heartbeat_back_addr":"172.21.15.67:6807/2625997716","heartbeat_front_addr":"172.21.15.67:6805/2625997716","state":["exists","up"]},{"osd":4,"uuid":"8a0ff860-fff7-434a-9e78-f2020bc84966","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":69,"up_from":73,"up_thru":78,"down_at":70,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":452825880},{"type":"v1","addr":"172.21.15.67:6809","nonce":452825880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":452825880},{"type":"v1","addr":"172.21.15.67:6811","nonce":452825880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":452825880},{"type":"v1","addr":"172.21.15.67:6815","nonce":452825880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":452825880},{"type":"v1","addr":"172.21.15.67:6813","nonce":452825880}]},"public_addr":"172.21.15.67:6809/452825880","cluster_addr":"172.21.15.67:6811/452825880","heartbeat_back_addr":"172.21.15.67:6815/452825880","heartbeat_front_addr":"172.21.15.67:6813/452825880","state":["exists","up"]},{"osd":5,"uuid":"3486e5c9-9f44-4f32-a186-fc6198f09549","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":74,"up_from":78,"up_thru":78,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2273350169},{"type":"v1","addr":"172.21.15.67:6817","nonce":2273350169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2273350169},{"type":"v1","addr":"172.21.15.67:6819","nonce":2273350169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2273350169},{"type":"v1","addr":"172.21.15.67:6823","nonce":2273350169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2273350169},{"type":"v1","addr":"172.21.15.67:6821","nonce":2273350169}]},"public_addr":"172.21.15.67:6817/2273350169","cluster_addr":"172.21.15.67:6819/2273350169","heartbeat_back_addr":"172.21.15.67:6823/2273350169","heartbeat_front_addr":"172.21.15.67:6821/2273350169","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:30:38.748993+0000","dead_epoch":49},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:31:08.677208+0000","dead_epoch":54},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:31:36.979481+0000","dead_epoch":59},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:32:02.668398+0000","dead_epoch":64},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:32:30.143788+0000","dead_epoch":69},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:32:57.613087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:6829/387754149":"2024-04-13T05:46:10.668153+0000","172.21.15.32:6828/387754149":"2024-04-13T05:46:10.668153+0000","172.21.15.67:6825/1291914599":"2024-04-13T05:46:10.668178+0000","172.21.15.67:6824/1291914599":"2024-04-13T05:46:10.668178+0000","172.21.15.32:0/138657961":"2024-04-13T05:27:40.952118+0000","172.21.15.67:0/1683579802":"2024-04-13T05:38:47.052969+0000","172.21.15.32:0/368340628":"2024-04-13T05:37:39.471472+0000","172.21.15.32:6801/3971961003":"2024-04-13T05:29:25.054120+0000","172.21.15.32:0/652449107":"2024-04-13T05:41:47.767117+0000","172.21.15.32:0/4112976740":"2024-04-13T05:29:25.054120+0000","172.21.15.32:0/2247439109":"2024-04-13T05:37:39.471472+0000","172.21.15.32:0/1643708423":"2024-04-13T05:27:40.952118+0000","172.21.15.32:6800/2564765508":"2024-04-13T05:41:47.767117+0000","172.21.15.67:6828/3925433122":"2024-04-13T05:38:47.052969+0000","172.21.15.32:6800/1962278696":"2024-04-13T05:28:14.470413+0000","172.21.15.32:0/2162993076":"2024-04-13T05:28:14.470413+0000","172.21.15.32:6827/2962187015":"2024-04-13T05:34:37.113413+0000","172.21.15.32:6800/3971961003":"2024-04-13T05:29:25.054120+0000","172.21.15.32:0/2105642006":"2024-04-13T05:37:39.471472+0000","172.21.15.67:0/31773612":"2024-04-13T05:38:47.052969+0000","172.21.15.67:6829/3925433122":"2024-04-13T05:38:47.052969+0000","172.21.15.67:0/2846171310":"2024-04-13T05:38:47.052969+0000","172.21.15.32:0/3370471392":"2024-04-13T05:41:47.767117+0000","172.21.15.32:0/3082743475":"2024-04-13T05:28:14.470413+0000","172.21.15.32:6801/656201914":"2024-04-13T05:27:40.952118+0000","172.21.15.32:6801/1962278696":"2024-04-13T05:28:14.470413+0000","172.21.15.32:0/260214199":"2024-04-13T05:37:39.471472+0000","172.21.15.32:6800/656201914":"2024-04-13T05:27:40.952118+0000","172.21.15.67:0/3462505447":"2024-04-13T05:38:47.052969+0000","172.21.15.32:0/674029871":"2024-04-13T05:29:25.054120+0000","172.21.15.67:0/125735613":"2024-04-13T05:38:47.052969+0000","172.21.15.67:0/3697060296":"2024-04-13T05:38:47.052969+0000","172.21.15.32:0/4031244002":"2024-04-13T05:27:40.952118+0000","172.21.15.32:6801/2564765508":"2024-04-13T05:41:47.767117+0000","172.21.15.32:0/795906028":"2024-04-13T05:29:25.054120+0000","172.21.15.32:6826/2962187015":"2024-04-13T05:34:37.113413+0000","172.21.15.32:6801/595990262":"2024-04-13T05:37:39.471472+0000","172.21.15.32:0/146872823":"2024-04-13T05:28:14.470413+0000","172.21.15.32:0/138775819":"2024-04-13T05:41:47.767117+0000","172.21.15.32:6800/595990262":"2024-04-13T05:37:39.471472+0000","172.21.15.32:0/2743601944":"2024-04-13T05:41:47.767117+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T06:04:31.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.071+0000 7f6190a71700 1 -- 172.21.15.32:0/1799509527 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f617806cc20 msgr2=0x7f617806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:31.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.071+0000 7f6190a71700 1 --2- 172.21.15.32:0/1799509527 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f617806cc20 0x7f617806f0d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f617c004f90 tx=0x7f617c009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:31.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.071+0000 7f6190a71700 1 -- 172.21.15.32:0/1799509527 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f618c102610 msgr2=0x7f618c112be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:31.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.071+0000 7f6190a71700 1 --2- 172.21.15.32:0/1799509527 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f618c102610 0x7f618c112be0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f6174002a50 tx=0x7f6174004600 comp rx=0 tx=0).stop 2024-04-12T06:04:31.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.071+0000 7f6190a71700 1 -- 172.21.15.32:0/1799509527 shutdown_connections 2024-04-12T06:04:31.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.071+0000 7f6190a71700 1 --2- 172.21.15.32:0/1799509527 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f617806cc20 0x7f617806f0d0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:31.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.071+0000 7f6190a71700 1 --2- 172.21.15.32:0/1799509527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f618c101c70 0x7f618c112660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:31.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.071+0000 7f6190a71700 1 --2- 172.21.15.32:0/1799509527 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f618c102610 0x7f618c112be0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:31.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.071+0000 7f6190a71700 1 -- 172.21.15.32:0/1799509527 >> 172.21.15.32:0/1799509527 conn(0x7f618c071c80 msgr2=0x7f618c0faf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:31.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.072+0000 7f6190a71700 1 -- 172.21.15.32:0/1799509527 shutdown_connections 2024-04-12T06:04:31.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.072+0000 7f6190a71700 1 -- 172.21.15.32:0/1799509527 wait complete. 2024-04-12T06:04:31.080 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-12T05:31:39.568876+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-12T05:34:21.574210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-12T05:34:22.056623+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-12T06:04:31.080 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-12T06:04:31.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.210+0000 7fdfe1d4a700 1 -- 172.21.15.32:0/1240969127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfdc101be0 msgr2=0x7fdfdc101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:31.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.210+0000 7fdfe1d4a700 1 --2- 172.21.15.32:0/1240969127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfdc101be0 0x7fdfdc101fb0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4004940 tx=0x7fdfc4009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:31.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.210+0000 7fdfe1d4a700 1 -- 172.21.15.32:0/1240969127 shutdown_connections 2024-04-12T06:04:31.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.210+0000 7fdfe1d4a700 1 --2- 172.21.15.32:0/1240969127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfdc102580 0x7fdfdc0fbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:31.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.210+0000 7fdfe1d4a700 1 --2- 172.21.15.32:0/1240969127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfdc101be0 0x7fdfdc101fb0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:31.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.210+0000 7fdfe1d4a700 1 -- 172.21.15.32:0/1240969127 >> 172.21.15.32:0/1240969127 conn(0x7fdfdc072ad0 msgr2=0x7fdfdc072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:31.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.210+0000 7fdfe1d4a700 1 -- 172.21.15.32:0/1240969127 shutdown_connections 2024-04-12T06:04:31.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.210+0000 7fdfe1d4a700 1 -- 172.21.15.32:0/1240969127 wait complete. 2024-04-12T06:04:31.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.210+0000 7fdfe1d4a700 1 Processor -- start 2024-04-12T06:04:31.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.210+0000 7fdfe1d4a700 1 -- start start 2024-04-12T06:04:31.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.211+0000 7fdfe1d4a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfdc101be0 0x7fdfdc113790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:31.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.211+0000 7fdfe1d4a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfdc102580 0x7fdfdc113cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:31.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.211+0000 7fdfe1d4a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc114320 con 0x7fdfdc102580 2024-04-12T06:04:31.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.211+0000 7fdfe1d4a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc10d970 con 0x7fdfdc101be0 2024-04-12T06:04:31.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.211+0000 7fdfdaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfdc102580 0x7fdfdc113cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:31.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.211+0000 7fdfdaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfdc102580 0x7fdfdc113cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52104/0 (socket says 172.21.15.32:52104) 2024-04-12T06:04:31.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.211+0000 7fdfdaffd700 1 -- 172.21.15.32:0/2712051738 learned_addr learned my addr 172.21.15.32:0/2712051738 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:31.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.211+0000 7fdfdb7fe700 1 --2- 172.21.15.32:0/2712051738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfdc101be0 0x7fdfdc113790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:31.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.211+0000 7fdfdaffd700 1 -- 172.21.15.32:0/2712051738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfdc101be0 msgr2=0x7fdfdc113790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:31.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.211+0000 7fdfdaffd700 1 --2- 172.21.15.32:0/2712051738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfdc101be0 0x7fdfdc113790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:31.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.211+0000 7fdfdaffd700 1 -- 172.21.15.32:0/2712051738 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc4005040 con 0x7fdfdc102580 2024-04-12T06:04:31.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.211+0000 7fdfdaffd700 1 --2- 172.21.15.32:0/2712051738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfdc102580 0x7fdfdc113cd0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc002a50 tx=0x7fdfcc004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:31.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.211+0000 7fdfd3fff700 1 -- 172.21.15.32:0/2712051738 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfcc007cc0 con 0x7fdfdc102580 2024-04-12T06:04:31.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.211+0000 7fdfe1d4a700 1 -- 172.21.15.32:0/2712051738 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfdc10dc50 con 0x7fdfdc102580 2024-04-12T06:04:31.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.211+0000 7fdfd3fff700 1 -- 172.21.15.32:0/2712051738 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdfcc00f040 con 0x7fdfdc102580 2024-04-12T06:04:31.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.211+0000 7fdfd3fff700 1 -- 172.21.15.32:0/2712051738 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfcc00d920 con 0x7fdfdc102580 2024-04-12T06:04:31.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.211+0000 7fdfe1d4a700 1 -- 172.21.15.32:0/2712051738 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfdc10e210 con 0x7fdfdc102580 2024-04-12T06:04:31.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.212+0000 7fdfe1d4a700 1 -- 172.21.15.32:0/2712051738 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfdc0fd330 con 0x7fdfdc102580 2024-04-12T06:04:31.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.213+0000 7fdfd3fff700 1 -- 172.21.15.32:0/2712051738 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fdfcc009680 con 0x7fdfdc102580 2024-04-12T06:04:31.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.213+0000 7fdfd3fff700 1 --2- 172.21.15.32:0/2712051738 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fdfc806cc20 0x7fdfc806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:31.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.213+0000 7fdfd3fff700 1 -- 172.21.15.32:0/2712051738 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fdfcc08be20 con 0x7fdfdc102580 2024-04-12T06:04:31.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.213+0000 7fdfdb7fe700 1 --2- 172.21.15.32:0/2712051738 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fdfc806cc20 0x7fdfc806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:31.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.214+0000 7fdfdb7fe700 1 --2- 172.21.15.32:0/2712051738 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fdfc806cc20 0x7fdfc806f0d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4004f90 tx=0x7fdfc4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:31.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.218+0000 7fdfd3fff700 1 -- 172.21.15.32:0/2712051738 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fdfcc059810 con 0x7fdfdc102580 2024-04-12T06:04:31.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:30 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1816640611' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-12T06:04:31.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:30 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/916336801' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-12T06:04:31.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:30 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1816640611' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-12T06:04:31.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:30 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/916336801' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-12T06:04:31.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.423+0000 7fdfe1d4a700 1 -- 172.21.15.32:0/2712051738 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdfdc108b90 con 0x7fdfdc102580 2024-04-12T06:04:31.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.423+0000 7fdfd3fff700 1 -- 172.21.15.32:0/2712051738 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdfcc0593a0 con 0x7fdfdc102580 2024-04-12T06:04:31.420 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-12T06:04:31.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.426+0000 7fdfe1d4a700 1 -- 172.21.15.32:0/2712051738 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fdfc806cc20 msgr2=0x7fdfc806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:31.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.426+0000 7fdfe1d4a700 1 --2- 172.21.15.32:0/2712051738 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fdfc806cc20 0x7fdfc806f0d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4004f90 tx=0x7fdfc4009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:31.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.426+0000 7fdfe1d4a700 1 -- 172.21.15.32:0/2712051738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfdc102580 msgr2=0x7fdfdc113cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:31.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.426+0000 7fdfe1d4a700 1 --2- 172.21.15.32:0/2712051738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfdc102580 0x7fdfdc113cd0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc002a50 tx=0x7fdfcc004600 comp rx=0 tx=0).stop 2024-04-12T06:04:31.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.426+0000 7fdfe1d4a700 1 -- 172.21.15.32:0/2712051738 shutdown_connections 2024-04-12T06:04:31.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.426+0000 7fdfe1d4a700 1 --2- 172.21.15.32:0/2712051738 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fdfc806cc20 0x7fdfc806f0d0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:31.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.426+0000 7fdfe1d4a700 1 --2- 172.21.15.32:0/2712051738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfdc101be0 0x7fdfdc113790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:31.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.426+0000 7fdfe1d4a700 1 --2- 172.21.15.32:0/2712051738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfdc102580 0x7fdfdc113cd0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:31.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.426+0000 7fdfe1d4a700 1 -- 172.21.15.32:0/2712051738 >> 172.21.15.32:0/2712051738 conn(0x7fdfdc072ad0 msgr2=0x7fdfdc064dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:31.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.426+0000 7fdfe1d4a700 1 -- 172.21.15.32:0/2712051738 shutdown_connections 2024-04-12T06:04:31.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.426+0000 7fdfe1d4a700 1 -- 172.21.15.32:0/2712051738 wait complete. 2024-04-12T06:04:31.434 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-12T06:04:31.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.564+0000 7fba6ab5e700 1 -- 172.21.15.32:0/1820185331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba64103670 msgr2=0x7fba640fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:31.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.564+0000 7fba6ab5e700 1 --2- 172.21.15.32:0/1820185331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba64103670 0x7fba640fcce0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fba50004940 tx=0x7fba50009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:31.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.564+0000 7fba6ab5e700 1 -- 172.21.15.32:0/1820185331 shutdown_connections 2024-04-12T06:04:31.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.564+0000 7fba6ab5e700 1 --2- 172.21.15.32:0/1820185331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba64103670 0x7fba640fcce0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:31.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.564+0000 7fba6ab5e700 1 --2- 172.21.15.32:0/1820185331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba64102cd0 0x7fba641030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:31.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.564+0000 7fba6ab5e700 1 -- 172.21.15.32:0/1820185331 >> 172.21.15.32:0/1820185331 conn(0x7fba640f8820 msgr2=0x7fba640fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:31.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.564+0000 7fba6ab5e700 1 -- 172.21.15.32:0/1820185331 shutdown_connections 2024-04-12T06:04:31.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.564+0000 7fba6ab5e700 1 -- 172.21.15.32:0/1820185331 wait complete. 2024-04-12T06:04:31.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.564+0000 7fba6ab5e700 1 Processor -- start 2024-04-12T06:04:31.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.564+0000 7fba6ab5e700 1 -- start start 2024-04-12T06:04:31.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.565+0000 7fba6ab5e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba64102cd0 0x7fba64113780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:31.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.565+0000 7fba6ab5e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba64103670 0x7fba64113cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:31.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.565+0000 7fba6ab5e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba64114310 con 0x7fba64103670 2024-04-12T06:04:31.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.565+0000 7fba6ab5e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6410d910 con 0x7fba64102cd0 2024-04-12T06:04:31.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.565+0000 7fba63fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba64103670 0x7fba64113cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:31.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.565+0000 7fba63fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba64103670 0x7fba64113cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32852/0 (socket says 172.21.15.32:32852) 2024-04-12T06:04:31.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.565+0000 7fba63fff700 1 -- 172.21.15.32:0/1994743830 learned_addr learned my addr 172.21.15.32:0/1994743830 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:31.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.565+0000 7fba688fa700 1 --2- 172.21.15.32:0/1994743830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba64102cd0 0x7fba64113780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:31.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.565+0000 7fba63fff700 1 -- 172.21.15.32:0/1994743830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba64102cd0 msgr2=0x7fba64113780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:31.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.565+0000 7fba63fff700 1 --2- 172.21.15.32:0/1994743830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba64102cd0 0x7fba64113780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:31.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.565+0000 7fba63fff700 1 -- 172.21.15.32:0/1994743830 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba50005040 con 0x7fba64103670 2024-04-12T06:04:31.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.565+0000 7fba688fa700 1 --2- 172.21.15.32:0/1994743830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba64102cd0 0x7fba64113780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T06:04:31.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.565+0000 7fba63fff700 1 --2- 172.21.15.32:0/1994743830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba64103670 0x7fba64113cc0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fba50009380 tx=0x7fba50009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:31.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.565+0000 7fba617fa700 1 -- 172.21.15.32:0/1994743830 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba5000e040 con 0x7fba64103670 2024-04-12T06:04:31.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.566+0000 7fba6ab5e700 1 -- 172.21.15.32:0/1994743830 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba6410db90 con 0x7fba64103670 2024-04-12T06:04:31.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.566+0000 7fba617fa700 1 -- 172.21.15.32:0/1994743830 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba50013420 con 0x7fba64103670 2024-04-12T06:04:31.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.566+0000 7fba617fa700 1 -- 172.21.15.32:0/1994743830 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba5001c630 con 0x7fba64103670 2024-04-12T06:04:31.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.566+0000 7fba6ab5e700 1 -- 172.21.15.32:0/1994743830 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba6410e150 con 0x7fba64103670 2024-04-12T06:04:31.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.567+0000 7fba6ab5e700 1 -- 172.21.15.32:0/1994743830 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba640fe420 con 0x7fba64103670 2024-04-12T06:04:31.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.567+0000 7fba617fa700 1 -- 172.21.15.32:0/1994743830 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fba500139e0 con 0x7fba64103670 2024-04-12T06:04:31.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.567+0000 7fba617fa700 1 --2- 172.21.15.32:0/1994743830 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fba5406cc20 0x7fba5406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:31.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.568+0000 7fba617fa700 1 -- 172.21.15.32:0/1994743830 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fba5008d280 con 0x7fba64103670 2024-04-12T06:04:31.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.568+0000 7fba688fa700 1 --2- 172.21.15.32:0/1994743830 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fba5406cc20 0x7fba5406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:31.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.568+0000 7fba688fa700 1 --2- 172.21.15.32:0/1994743830 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fba5406cc20 0x7fba5406f0d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fba58007fd0 tx=0x7fba58009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:31.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.572+0000 7fba617fa700 1 -- 172.21.15.32:0/1994743830 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fba5005acf0 con 0x7fba64103670 2024-04-12T06:04:31.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.775+0000 7fba6ab5e700 1 -- 172.21.15.32:0/1994743830 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fba6410edd0 con 0x7fba64103670 2024-04-12T06:04:31.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.775+0000 7fba617fa700 1 -- 172.21.15.32:0/1994743830 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v85) v1 ==== 107+0+11 (secure 0 0 0) 0x7fba5005a880 con 0x7fba64103670 2024-04-12T06:04:31.772 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-12T06:04:31.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.777+0000 7fba6ab5e700 1 -- 172.21.15.32:0/1994743830 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fba5406cc20 msgr2=0x7fba5406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:31.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.777+0000 7fba6ab5e700 1 --2- 172.21.15.32:0/1994743830 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fba5406cc20 0x7fba5406f0d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fba58007fd0 tx=0x7fba58009040 comp rx=0 tx=0).stop 2024-04-12T06:04:31.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.777+0000 7fba6ab5e700 1 -- 172.21.15.32:0/1994743830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba64103670 msgr2=0x7fba64113cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:31.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.777+0000 7fba6ab5e700 1 --2- 172.21.15.32:0/1994743830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba64103670 0x7fba64113cc0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fba50009380 tx=0x7fba50009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:31.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.777+0000 7fba6ab5e700 1 -- 172.21.15.32:0/1994743830 shutdown_connections 2024-04-12T06:04:31.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.777+0000 7fba6ab5e700 1 --2- 172.21.15.32:0/1994743830 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fba5406cc20 0x7fba5406f0d0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:31.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.777+0000 7fba6ab5e700 1 --2- 172.21.15.32:0/1994743830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba64102cd0 0x7fba64113780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:31.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.777+0000 7fba6ab5e700 1 --2- 172.21.15.32:0/1994743830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba64103670 0x7fba64113cc0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:31.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.777+0000 7fba6ab5e700 1 -- 172.21.15.32:0/1994743830 >> 172.21.15.32:0/1994743830 conn(0x7fba640f8820 msgr2=0x7fba640f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:31.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.778+0000 7fba6ab5e700 1 -- 172.21.15.32:0/1994743830 shutdown_connections 2024-04-12T06:04:31.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.778+0000 7fba6ab5e700 1 -- 172.21.15.32:0/1994743830 wait complete. 2024-04-12T06:04:31.786 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-12T06:04:31.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.914+0000 7f6180bb5700 1 -- 172.21.15.32:0/2923016380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f617c103720 msgr2=0x7f617c0fb340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:31.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.915+0000 7f6180bb5700 1 --2- 172.21.15.32:0/2923016380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f617c103720 0x7f617c0fb340 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f6170004940 tx=0x7f6170009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:31.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.915+0000 7f6180bb5700 1 -- 172.21.15.32:0/2923016380 shutdown_connections 2024-04-12T06:04:31.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.915+0000 7f6180bb5700 1 --2- 172.21.15.32:0/2923016380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f617c103720 0x7f617c0fb340 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:31.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.915+0000 7f6180bb5700 1 --2- 172.21.15.32:0/2923016380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f617c102d80 0x7f617c103150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:31.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.915+0000 7f6180bb5700 1 -- 172.21.15.32:0/2923016380 >> 172.21.15.32:0/2923016380 conn(0x7f617c072230 msgr2=0x7f617c072630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:31.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.915+0000 7f6180bb5700 1 -- 172.21.15.32:0/2923016380 shutdown_connections 2024-04-12T06:04:31.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.915+0000 7f6180bb5700 1 -- 172.21.15.32:0/2923016380 wait complete. 2024-04-12T06:04:31.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.915+0000 7f6180bb5700 1 Processor -- start 2024-04-12T06:04:31.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.915+0000 7f6180bb5700 1 -- start start 2024-04-12T06:04:31.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.915+0000 7f6180bb5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f617c102d80 0x7f617c1136c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:31.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.915+0000 7f6180bb5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f617c103720 0x7f617c113c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:31.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.915+0000 7f6180bb5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617c114250 con 0x7f617c103720 2024-04-12T06:04:31.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.915+0000 7f6180bb5700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617c10d870 con 0x7f617c102d80 2024-04-12T06:04:31.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.916+0000 7f6179d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f617c103720 0x7f617c113c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:31.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.916+0000 7f6179d9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f617c103720 0x7f617c113c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32868/0 (socket says 172.21.15.32:32868) 2024-04-12T06:04:31.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.916+0000 7f6179d9b700 1 -- 172.21.15.32:0/1221571152 learned_addr learned my addr 172.21.15.32:0/1221571152 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:31.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.916+0000 7f6179d9b700 1 -- 172.21.15.32:0/1221571152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f617c102d80 msgr2=0x7f617c1136c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:31.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.916+0000 7f617a59c700 1 --2- 172.21.15.32:0/1221571152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f617c102d80 0x7f617c1136c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:31.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.916+0000 7f6179d9b700 1 --2- 172.21.15.32:0/1221571152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f617c102d80 0x7f617c1136c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:31.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.916+0000 7f6179d9b700 1 -- 172.21.15.32:0/1221571152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6170005040 con 0x7f617c103720 2024-04-12T06:04:31.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.916+0000 7f6179d9b700 1 --2- 172.21.15.32:0/1221571152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f617c103720 0x7f617c113c00 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f6170008d50 tx=0x7f6170009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:31.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.916+0000 7f616affd700 1 -- 172.21.15.32:0/1221571152 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f617000e030 con 0x7f617c103720 2024-04-12T06:04:31.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.916+0000 7f6180bb5700 1 -- 172.21.15.32:0/1221571152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f617c10daf0 con 0x7f617c103720 2024-04-12T06:04:31.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.916+0000 7f616affd700 1 -- 172.21.15.32:0/1221571152 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6170013de0 con 0x7f617c103720 2024-04-12T06:04:31.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.916+0000 7f616affd700 1 -- 172.21.15.32:0/1221571152 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f617001cd90 con 0x7f617c103720 2024-04-12T06:04:31.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.916+0000 7f6180bb5700 1 -- 172.21.15.32:0/1221571152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f617c10e0b0 con 0x7f617c103720 2024-04-12T06:04:31.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.917+0000 7f6180bb5700 1 -- 172.21.15.32:0/1221571152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f617c1000e0 con 0x7f617c103720 2024-04-12T06:04:31.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.918+0000 7f616affd700 1 -- 172.21.15.32:0/1221571152 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f6170013930 con 0x7f617c103720 2024-04-12T06:04:31.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.918+0000 7f616affd700 1 --2- 172.21.15.32:0/1221571152 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f616406cc20 0x7f616406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:31.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.918+0000 7f616affd700 1 -- 172.21.15.32:0/1221571152 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f617008d970 con 0x7f617c103720 2024-04-12T06:04:31.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.918+0000 7f617a59c700 1 --2- 172.21.15.32:0/1221571152 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f616406cc20 0x7f616406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:31.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.919+0000 7f617a59c700 1 --2- 172.21.15.32:0/1221571152 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f616406cc20 0x7f616406f0d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f616c004600 tx=0x7f616c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:31.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:31.923+0000 7f616affd700 1 -- 172.21.15.32:0/1221571152 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f617005b3e0 con 0x7f617c103720 2024-04-12T06:04:32.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:31 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1799509527' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T06:04:32.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:31 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/2712051738' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T06:04:32.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:31 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1994743830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-12T06:04:32.088 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:31 smithi032 ceph-mon[146817]: pgmap v721: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:04:32.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.126+0000 7f6180bb5700 1 -- 172.21.15.32:0/1221571152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f617c10ed50 con 0x7f617c103720 2024-04-12T06:04:32.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.126+0000 7f616affd700 1 -- 172.21.15.32:0/1221571152 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v85) v1 ==== 107+0+11 (secure 0 0 0) 0x7f617005af70 con 0x7f617c103720 2024-04-12T06:04:32.123 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-12T06:04:32.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.128+0000 7f6180bb5700 1 -- 172.21.15.32:0/1221571152 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f616406cc20 msgr2=0x7f616406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:32.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.128+0000 7f6180bb5700 1 --2- 172.21.15.32:0/1221571152 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f616406cc20 0x7f616406f0d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f616c004600 tx=0x7f616c00b040 comp rx=0 tx=0).stop 2024-04-12T06:04:32.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.128+0000 7f6180bb5700 1 -- 172.21.15.32:0/1221571152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f617c103720 msgr2=0x7f617c113c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:32.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.128+0000 7f6180bb5700 1 --2- 172.21.15.32:0/1221571152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f617c103720 0x7f617c113c00 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f6170008d50 tx=0x7f6170009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:32.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.129+0000 7f6180bb5700 1 -- 172.21.15.32:0/1221571152 shutdown_connections 2024-04-12T06:04:32.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.129+0000 7f6180bb5700 1 --2- 172.21.15.32:0/1221571152 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7f616406cc20 0x7f616406f0d0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:32.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.129+0000 7f6180bb5700 1 --2- 172.21.15.32:0/1221571152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f617c102d80 0x7f617c1136c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:32.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.129+0000 7f6180bb5700 1 --2- 172.21.15.32:0/1221571152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f617c103720 0x7f617c113c00 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:32.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.129+0000 7f6180bb5700 1 -- 172.21.15.32:0/1221571152 >> 172.21.15.32:0/1221571152 conn(0x7f617c072230 msgr2=0x7f617c0faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:32.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.129+0000 7f6180bb5700 1 -- 172.21.15.32:0/1221571152 shutdown_connections 2024-04-12T06:04:32.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.129+0000 7f6180bb5700 1 -- 172.21.15.32:0/1221571152 wait complete. 2024-04-12T06:04:32.137 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-12T06:04:32.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.265+0000 7fa5e2e99700 1 -- 172.21.15.32:0/1663857488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5dc101be0 msgr2=0x7fa5dc101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:32.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.265+0000 7fa5e2e99700 1 --2- 172.21.15.32:0/1663857488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5dc101be0 0x7fa5dc101fb0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0004940 tx=0x7fa5d0009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:32.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.265+0000 7fa5e2e99700 1 -- 172.21.15.32:0/1663857488 shutdown_connections 2024-04-12T06:04:32.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.265+0000 7fa5e2e99700 1 --2- 172.21.15.32:0/1663857488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5dc102580 0x7fa5dc0fbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:32.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.265+0000 7fa5e2e99700 1 --2- 172.21.15.32:0/1663857488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5dc101be0 0x7fa5dc101fb0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:32.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.265+0000 7fa5e2e99700 1 -- 172.21.15.32:0/1663857488 >> 172.21.15.32:0/1663857488 conn(0x7fa5dc072ad0 msgr2=0x7fa5dc072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:32.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.265+0000 7fa5e2e99700 1 -- 172.21.15.32:0/1663857488 shutdown_connections 2024-04-12T06:04:32.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.265+0000 7fa5e2e99700 1 -- 172.21.15.32:0/1663857488 wait complete. 2024-04-12T06:04:32.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.265+0000 7fa5e2e99700 1 Processor -- start 2024-04-12T06:04:32.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.265+0000 7fa5e2e99700 1 -- start start 2024-04-12T06:04:32.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.266+0000 7fa5e2e99700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5dc101be0 0x7fa5dc1126c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:32.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.266+0000 7fa5e2e99700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5dc102580 0x7fa5dc112c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:32.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.266+0000 7fa5e2e99700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5dc113290 con 0x7fa5dc101be0 2024-04-12T06:04:32.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.266+0000 7fa5e2e99700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5dc10c850 con 0x7fa5dc102580 2024-04-12T06:04:32.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.266+0000 7fa5e0c35700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5dc101be0 0x7fa5dc1126c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:32.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.266+0000 7fa5e0c35700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5dc101be0 0x7fa5dc1126c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32874/0 (socket says 172.21.15.32:32874) 2024-04-12T06:04:32.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.266+0000 7fa5e0c35700 1 -- 172.21.15.32:0/37841627 learned_addr learned my addr 172.21.15.32:0/37841627 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-12T06:04:32.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.266+0000 7fa5dbfff700 1 --2- 172.21.15.32:0/37841627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5dc102580 0x7fa5dc112c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:32.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.266+0000 7fa5e0c35700 1 -- 172.21.15.32:0/37841627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5dc102580 msgr2=0x7fa5dc112c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:32.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.266+0000 7fa5e0c35700 1 --2- 172.21.15.32:0/37841627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5dc102580 0x7fa5dc112c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:32.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.266+0000 7fa5e0c35700 1 -- 172.21.15.32:0/37841627 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d0005040 con 0x7fa5dc101be0 2024-04-12T06:04:32.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.266+0000 7fa5e0c35700 1 --2- 172.21.15.32:0/37841627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5dc101be0 0x7fa5dc1126c0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0009360 tx=0x7fa5d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:32.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.266+0000 7fa5d97fa700 1 -- 172.21.15.32:0/37841627 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5d000e040 con 0x7fa5dc101be0 2024-04-12T06:04:32.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.266+0000 7fa5d97fa700 1 -- 172.21.15.32:0/37841627 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa5d0013420 con 0x7fa5dc101be0 2024-04-12T06:04:32.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.266+0000 7fa5e2e99700 1 -- 172.21.15.32:0/37841627 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5dc10ca80 con 0x7fa5dc101be0 2024-04-12T06:04:32.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.266+0000 7fa5d97fa700 1 -- 172.21.15.32:0/37841627 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5d001c5f0 con 0x7fa5dc101be0 2024-04-12T06:04:32.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.267+0000 7fa5e2e99700 1 -- 172.21.15.32:0/37841627 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5dc10cfc0 con 0x7fa5dc101be0 2024-04-12T06:04:32.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.267+0000 7fa5e2e99700 1 -- 172.21.15.32:0/37841627 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5dc0fd330 con 0x7fa5dc101be0 2024-04-12T06:04:32.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.268+0000 7fa5d97fa700 1 -- 172.21.15.32:0/37841627 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 40) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa5d0013590 con 0x7fa5dc101be0 2024-04-12T06:04:32.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.268+0000 7fa5d97fa700 1 --2- 172.21.15.32:0/37841627 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa5cc06cc20 0x7fa5cc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T06:04:32.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.269+0000 7fa5d97fa700 1 -- 172.21.15.32:0/37841627 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fa5d008d330 con 0x7fa5dc101be0 2024-04-12T06:04:32.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.269+0000 7fa5dbfff700 1 --2- 172.21.15.32:0/37841627 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa5cc06cc20 0x7fa5cc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T06:04:32.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.269+0000 7fa5dbfff700 1 --2- 172.21.15.32:0/37841627 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa5cc06cc20 0x7fa5cc06f0d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8007fd0 tx=0x7fa5c8009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T06:04:32.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:31 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1799509527' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T06:04:32.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:31 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/2712051738' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T06:04:32.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:31 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1994743830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-12T06:04:32.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:31 smithi067 ceph-mon[112979]: pgmap v721: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:04:32.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.273+0000 7fa5d97fa700 1 -- 172.21.15.32:0/37841627 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa5d005ada0 con 0x7fa5dc101be0 2024-04-12T06:04:32.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.494+0000 7fa5e2e99700 1 -- 172.21.15.32:0/37841627 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa5dc10ddb0 con 0x7fa5dc101be0 2024-04-12T06:04:32.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.494+0000 7fa5d97fa700 1 -- 172.21.15.32:0/37841627 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 41 entries v85) v1 ==== 81+0+2415 (secure 0 0 0) 0x7fa5d005a930 con 0x7fa5dc101be0 2024-04-12T06:04:32.491 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6829/387754149 2024-04-13T05:46:10.668153+0000 2024-04-12T06:04:32.491 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6828/387754149 2024-04-13T05:46:10.668153+0000 2024-04-12T06:04:32.491 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.67:6825/1291914599 2024-04-13T05:46:10.668178+0000 2024-04-12T06:04:32.491 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.67:6824/1291914599 2024-04-13T05:46:10.668178+0000 2024-04-12T06:04:32.491 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/138657961 2024-04-13T05:27:40.952118+0000 2024-04-12T06:04:32.491 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.67:0/1683579802 2024-04-13T05:38:47.052969+0000 2024-04-12T06:04:32.491 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/368340628 2024-04-13T05:37:39.471472+0000 2024-04-12T06:04:32.491 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6801/3971961003 2024-04-13T05:29:25.054120+0000 2024-04-12T06:04:32.491 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/652449107 2024-04-13T05:41:47.767117+0000 2024-04-12T06:04:32.491 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/4112976740 2024-04-13T05:29:25.054120+0000 2024-04-12T06:04:32.491 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/2247439109 2024-04-13T05:37:39.471472+0000 2024-04-12T06:04:32.491 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/1643708423 2024-04-13T05:27:40.952118+0000 2024-04-12T06:04:32.492 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6800/2564765508 2024-04-13T05:41:47.767117+0000 2024-04-12T06:04:32.492 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.67:6828/3925433122 2024-04-13T05:38:47.052969+0000 2024-04-12T06:04:32.492 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6800/1962278696 2024-04-13T05:28:14.470413+0000 2024-04-12T06:04:32.492 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/2162993076 2024-04-13T05:28:14.470413+0000 2024-04-12T06:04:32.492 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6827/2962187015 2024-04-13T05:34:37.113413+0000 2024-04-12T06:04:32.492 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6800/3971961003 2024-04-13T05:29:25.054120+0000 2024-04-12T06:04:32.492 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/2105642006 2024-04-13T05:37:39.471472+0000 2024-04-12T06:04:32.492 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.67:0/31773612 2024-04-13T05:38:47.052969+0000 2024-04-12T06:04:32.492 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.67:6829/3925433122 2024-04-13T05:38:47.052969+0000 2024-04-12T06:04:32.492 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.67:0/2846171310 2024-04-13T05:38:47.052969+0000 2024-04-12T06:04:32.492 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/3370471392 2024-04-13T05:41:47.767117+0000 2024-04-12T06:04:32.492 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/3082743475 2024-04-13T05:28:14.470413+0000 2024-04-12T06:04:32.492 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6801/656201914 2024-04-13T05:27:40.952118+0000 2024-04-12T06:04:32.493 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6801/1962278696 2024-04-13T05:28:14.470413+0000 2024-04-12T06:04:32.493 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/260214199 2024-04-13T05:37:39.471472+0000 2024-04-12T06:04:32.493 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6800/656201914 2024-04-13T05:27:40.952118+0000 2024-04-12T06:04:32.493 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.67:0/3462505447 2024-04-13T05:38:47.052969+0000 2024-04-12T06:04:32.493 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/674029871 2024-04-13T05:29:25.054120+0000 2024-04-12T06:04:32.493 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.67:0/125735613 2024-04-13T05:38:47.052969+0000 2024-04-12T06:04:32.493 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.67:0/3697060296 2024-04-13T05:38:47.052969+0000 2024-04-12T06:04:32.493 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/4031244002 2024-04-13T05:27:40.952118+0000 2024-04-12T06:04:32.493 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6801/2564765508 2024-04-13T05:41:47.767117+0000 2024-04-12T06:04:32.493 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/795906028 2024-04-13T05:29:25.054120+0000 2024-04-12T06:04:32.493 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6826/2962187015 2024-04-13T05:34:37.113413+0000 2024-04-12T06:04:32.493 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6801/595990262 2024-04-13T05:37:39.471472+0000 2024-04-12T06:04:32.493 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/146872823 2024-04-13T05:28:14.470413+0000 2024-04-12T06:04:32.493 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/138775819 2024-04-13T05:41:47.767117+0000 2024-04-12T06:04:32.494 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6800/595990262 2024-04-13T05:37:39.471472+0000 2024-04-12T06:04:32.494 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/2743601944 2024-04-13T05:41:47.767117+0000 2024-04-12T06:04:32.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.496+0000 7fa5e2e99700 1 -- 172.21.15.32:0/37841627 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa5cc06cc20 msgr2=0x7fa5cc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:32.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.496+0000 7fa5e2e99700 1 --2- 172.21.15.32:0/37841627 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa5cc06cc20 0x7fa5cc06f0d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8007fd0 tx=0x7fa5c8009040 comp rx=0 tx=0).stop 2024-04-12T06:04:32.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.496+0000 7fa5e2e99700 1 -- 172.21.15.32:0/37841627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5dc101be0 msgr2=0x7fa5dc1126c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T06:04:32.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.496+0000 7fa5e2e99700 1 --2- 172.21.15.32:0/37841627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5dc101be0 0x7fa5dc1126c0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0009360 tx=0x7fa5d0009e30 comp rx=0 tx=0).stop 2024-04-12T06:04:32.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.496+0000 7fa5e2e99700 1 -- 172.21.15.32:0/37841627 shutdown_connections 2024-04-12T06:04:32.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.496+0000 7fa5e2e99700 1 --2- 172.21.15.32:0/37841627 >> [v2:172.21.15.32:6800/3171834473,v1:172.21.15.32:6801/3171834473] conn(0x7fa5cc06cc20 0x7fa5cc06f0d0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:32.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.496+0000 7fa5e2e99700 1 --2- 172.21.15.32:0/37841627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5dc101be0 0x7fa5dc1126c0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:32.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.496+0000 7fa5e2e99700 1 --2- 172.21.15.32:0/37841627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5dc102580 0x7fa5dc112c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T06:04:32.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.496+0000 7fa5e2e99700 1 -- 172.21.15.32:0/37841627 >> 172.21.15.32:0/37841627 conn(0x7fa5dc072ad0 msgr2=0x7fa5dc104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T06:04:32.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.496+0000 7fa5e2e99700 1 -- 172.21.15.32:0/37841627 shutdown_connections 2024-04-12T06:04:32.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-12T06:04:32.496+0000 7fa5e2e99700 1 -- 172.21.15.32:0/37841627 wait complete. 2024-04-12T06:04:32.495 INFO:teuthology.orchestra.run.smithi032.stderr:listed 41 entries 2024-04-12T06:04:32.505 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi032.front.sepia.ceph.com... 2024-04-12T06:04:32.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T06:04:32.505 DEBUG:teuthology.orchestra.run.smithi032:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-12T06:04:32.549 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T06:04:33.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:32 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/1221571152' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-12T06:04:33.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:32 smithi067 ceph-mon[112979]: from='client.? 172.21.15.32:0/37841627' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-12T06:04:33.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:32 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:04:33.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:32 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/1221571152' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-12T06:04:33.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:32 smithi032 ceph-mon[146817]: from='client.? 172.21.15.32:0/37841627' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-12T06:04:33.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:32 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:04:34.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:33 smithi067 ceph-mon[112979]: pgmap v722: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:04:34.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:33 smithi032 ceph-mon[146817]: pgmap v722: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:04:37.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:36 smithi067 ceph-mon[112979]: pgmap v723: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:04:37.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:36 smithi032 ceph-mon[146817]: pgmap v723: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:04:39.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:38 smithi067 ceph-mon[112979]: pgmap v724: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:04:39.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:38 smithi032 ceph-mon[146817]: pgmap v724: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:04:41.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:40 smithi067 ceph-mon[112979]: pgmap v725: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:04:41.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:40 smithi032 ceph-mon[146817]: pgmap v725: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:04:43.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:42 smithi067 ceph-mon[112979]: pgmap v726: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:04:43.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:42 smithi032 ceph-mon[146817]: pgmap v726: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:04:45.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:44 smithi067 ceph-mon[112979]: pgmap v727: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:04:45.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:44 smithi032 ceph-mon[146817]: pgmap v727: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:04:47.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:46 smithi067 ceph-mon[112979]: pgmap v728: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:04:47.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:46 smithi032 ceph-mon[146817]: pgmap v728: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:04:48.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:47 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:04:48.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:47 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:04:49.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:48 smithi067 ceph-mon[112979]: pgmap v729: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:04:49.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:48 smithi032 ceph-mon[146817]: pgmap v729: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:04:51.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:50 smithi067 ceph-mon[112979]: pgmap v730: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:04:51.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:50 smithi032 ceph-mon[146817]: pgmap v730: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:04:53.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:52 smithi067 ceph-mon[112979]: pgmap v731: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:04:53.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:52 smithi032 ceph-mon[146817]: pgmap v731: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:04:55.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:54 smithi067 ceph-mon[112979]: pgmap v732: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:04:55.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:54 smithi032 ceph-mon[146817]: pgmap v732: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:04:57.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:56 smithi067 ceph-mon[112979]: pgmap v733: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:04:57.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:56 smithi032 ceph-mon[146817]: pgmap v733: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:04:59.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:04:58 smithi067 ceph-mon[112979]: pgmap v734: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:04:59.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:04:58 smithi032 ceph-mon[146817]: pgmap v734: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:05:01.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:00 smithi067 ceph-mon[112979]: pgmap v735: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:05:01.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:00 smithi032 ceph-mon[146817]: pgmap v735: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:05:02.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:01 smithi067 ceph-mon[112979]: pgmap v736: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:05:02.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:01 smithi032 ceph-mon[146817]: pgmap v736: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:05:03.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:02 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:05:03.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:02 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:05:04.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:03 smithi067 ceph-mon[112979]: pgmap v737: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:05:04.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:03 smithi032 ceph-mon[146817]: pgmap v737: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:05:07.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:06 smithi067 ceph-mon[112979]: pgmap v738: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:05:07.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:06 smithi032 ceph-mon[146817]: pgmap v738: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:05:09.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:08 smithi067 ceph-mon[112979]: pgmap v739: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:05:09.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:08 smithi032 ceph-mon[146817]: pgmap v739: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:05:11.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:10 smithi067 ceph-mon[112979]: pgmap v740: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:05:11.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:10 smithi032 ceph-mon[146817]: pgmap v740: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:05:13.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:12 smithi067 ceph-mon[112979]: pgmap v741: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:05:13.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:12 smithi032 ceph-mon[146817]: pgmap v741: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:05:15.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:14 smithi067 ceph-mon[112979]: pgmap v742: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:05:15.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:14 smithi032 ceph-mon[146817]: pgmap v742: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:05:17.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:16 smithi067 ceph-mon[112979]: pgmap v743: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:05:17.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:16 smithi032 ceph-mon[146817]: pgmap v743: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:05:18.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:17 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:05:18.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:17 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:05:19.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:18 smithi067 ceph-mon[112979]: pgmap v744: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:05:19.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:18 smithi032 ceph-mon[146817]: pgmap v744: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:05:21.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:20 smithi067 ceph-mon[112979]: pgmap v745: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:05:21.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:20 smithi032 ceph-mon[146817]: pgmap v745: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:05:22.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:21 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T06:05:22.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:21 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T06:05:22.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:21 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T06:05:22.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:21 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:05:22.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:21 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T06:05:22.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:21 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T06:05:22.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:21 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T06:05:22.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:21 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:05:23.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:22 smithi067 ceph-mon[112979]: pgmap v746: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:05:23.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:22 smithi032 ceph-mon[146817]: pgmap v746: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:05:25.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:24 smithi067 ceph-mon[112979]: pgmap v747: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:05:25.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:24 smithi032 ceph-mon[146817]: pgmap v747: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:05:27.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:26 smithi067 ceph-mon[112979]: pgmap v748: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:05:27.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:26 smithi032 ceph-mon[146817]: pgmap v748: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:05:29.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:28 smithi067 ceph-mon[112979]: pgmap v749: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:05:29.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:28 smithi032 ceph-mon[146817]: pgmap v749: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:05:31.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:30 smithi067 ceph-mon[112979]: pgmap v750: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:05:31.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:30 smithi032 ceph-mon[146817]: pgmap v750: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:05:32.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:31 smithi067 ceph-mon[112979]: pgmap v751: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:05:32.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:31 smithi032 ceph-mon[146817]: pgmap v751: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:05:33.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:32 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:05:33.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:32 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:05:34.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:33 smithi067 ceph-mon[112979]: pgmap v752: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:05:34.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:33 smithi032 ceph-mon[146817]: pgmap v752: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:05:37.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:36 smithi067 ceph-mon[112979]: pgmap v753: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:05:37.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:36 smithi032 ceph-mon[146817]: pgmap v753: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:05:39.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:38 smithi067 ceph-mon[112979]: pgmap v754: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:05:39.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:38 smithi032 ceph-mon[146817]: pgmap v754: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:05:41.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:40 smithi067 ceph-mon[112979]: pgmap v755: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:05:41.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:40 smithi032 ceph-mon[146817]: pgmap v755: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:05:43.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:42 smithi067 ceph-mon[112979]: pgmap v756: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:05:43.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:42 smithi032 ceph-mon[146817]: pgmap v756: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:05:45.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:44 smithi067 ceph-mon[112979]: pgmap v757: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:05:45.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:44 smithi032 ceph-mon[146817]: pgmap v757: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:05:47.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:46 smithi067 ceph-mon[112979]: pgmap v758: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:05:47.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:46 smithi032 ceph-mon[146817]: pgmap v758: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:05:48.267 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:47 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:05:48.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:47 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:05:49.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:48 smithi067 ceph-mon[112979]: pgmap v759: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:05:49.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:48 smithi032 ceph-mon[146817]: pgmap v759: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:05:51.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:50 smithi067 ceph-mon[112979]: pgmap v760: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:05:51.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:50 smithi032 ceph-mon[146817]: pgmap v760: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:05:53.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:52 smithi067 ceph-mon[112979]: pgmap v761: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:05:53.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:52 smithi032 ceph-mon[146817]: pgmap v761: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:05:55.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:54 smithi067 ceph-mon[112979]: pgmap v762: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:05:55.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:54 smithi032 ceph-mon[146817]: pgmap v762: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:05:57.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:56 smithi067 ceph-mon[112979]: pgmap v763: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:05:57.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:56 smithi032 ceph-mon[146817]: pgmap v763: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:05:58.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:05:57 smithi067 ceph-mon[112979]: pgmap v764: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:05:58.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:05:57 smithi032 ceph-mon[146817]: pgmap v764: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:06:01.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:00 smithi067 ceph-mon[112979]: pgmap v765: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:06:01.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:00 smithi032 ceph-mon[146817]: pgmap v765: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:06:03.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:02 smithi067 ceph-mon[112979]: pgmap v766: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:06:03.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:02 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:06:03.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:02 smithi032 ceph-mon[146817]: pgmap v766: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:06:03.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:02 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:06:05.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:04 smithi067 ceph-mon[112979]: pgmap v767: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:06:05.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:04 smithi032 ceph-mon[146817]: pgmap v767: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:06:07.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:06 smithi067 ceph-mon[112979]: pgmap v768: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:06:07.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:06 smithi032 ceph-mon[146817]: pgmap v768: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:06:09.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:08 smithi067 ceph-mon[112979]: pgmap v769: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:06:09.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:08 smithi032 ceph-mon[146817]: pgmap v769: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:06:11.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:10 smithi067 ceph-mon[112979]: pgmap v770: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:06:11.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:10 smithi032 ceph-mon[146817]: pgmap v770: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:06:13.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:12 smithi067 ceph-mon[112979]: pgmap v771: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:06:13.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:12 smithi032 ceph-mon[146817]: pgmap v771: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:06:15.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:14 smithi067 ceph-mon[112979]: pgmap v772: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:06:15.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:14 smithi032 ceph-mon[146817]: pgmap v772: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:06:17.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:16 smithi067 ceph-mon[112979]: pgmap v773: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:06:17.338 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:16 smithi032 ceph-mon[146817]: pgmap v773: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:06:18.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:17 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:06:18.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:17 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:06:19.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:18 smithi067 ceph-mon[112979]: pgmap v774: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-12T06:06:19.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:18 smithi032 ceph-mon[146817]: pgmap v774: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-12T06:06:20.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:20 smithi067 ceph-mon[112979]: pgmap v775: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-12T06:06:20.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:20 smithi032 ceph-mon[146817]: pgmap v775: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-12T06:06:23.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:22 smithi067 ceph-mon[112979]: pgmap v776: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-12T06:06:23.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:22 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T06:06:23.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:22 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T06:06:23.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:22 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T06:06:23.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:22 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:06:23.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:22 smithi032 ceph-mon[146817]: pgmap v776: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-12T06:06:23.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:22 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T06:06:23.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:22 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T06:06:23.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:22 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T06:06:23.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:22 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:06:25.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:24 smithi067 ceph-mon[112979]: pgmap v777: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-12T06:06:25.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:24 smithi032 ceph-mon[146817]: pgmap v777: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-12T06:06:27.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:26 smithi067 ceph-mon[112979]: pgmap v778: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-12T06:06:27.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:26 smithi032 ceph-mon[146817]: pgmap v778: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-12T06:06:29.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:28 smithi067 ceph-mon[112979]: pgmap v779: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-12T06:06:29.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:28 smithi032 ceph-mon[146817]: pgmap v779: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-12T06:06:31.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:30 smithi067 ceph-mon[112979]: pgmap v780: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:06:31.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:30 smithi032 ceph-mon[146817]: pgmap v780: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:06:33.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:32 smithi067 ceph-mon[112979]: pgmap v781: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:06:33.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:32 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:06:33.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:32 smithi032 ceph-mon[146817]: pgmap v781: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:06:33.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:32 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:06:35.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:34 smithi067 ceph-mon[112979]: pgmap v782: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:06:35.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:34 smithi032 ceph-mon[146817]: pgmap v782: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:06:37.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:36 smithi067 ceph-mon[112979]: pgmap v783: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:06:37.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:36 smithi032 ceph-mon[146817]: pgmap v783: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:06:39.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:38 smithi067 ceph-mon[112979]: pgmap v784: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:06:39.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:38 smithi032 ceph-mon[146817]: pgmap v784: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:06:41.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:40 smithi067 ceph-mon[112979]: pgmap v785: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:06:41.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:40 smithi032 ceph-mon[146817]: pgmap v785: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:06:42.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:42 smithi067 ceph-mon[112979]: pgmap v786: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:06:42.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:42 smithi032 ceph-mon[146817]: pgmap v786: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:06:44.932 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:44 smithi067 ceph-mon[112979]: pgmap v787: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:06:45.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:44 smithi032 ceph-mon[146817]: pgmap v787: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:06:47.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:46 smithi067 ceph-mon[112979]: pgmap v788: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:06:47.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:46 smithi032 ceph-mon[146817]: pgmap v788: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:06:48.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:47 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:06:48.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:47 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:06:49.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:48 smithi067 ceph-mon[112979]: pgmap v789: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:06:49.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:48 smithi032 ceph-mon[146817]: pgmap v789: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:06:51.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:50 smithi067 ceph-mon[112979]: pgmap v790: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:06:51.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:50 smithi032 ceph-mon[146817]: pgmap v790: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:06:53.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:52 smithi067 ceph-mon[112979]: pgmap v791: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:06:53.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:52 smithi032 ceph-mon[146817]: pgmap v791: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:06:55.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:54 smithi067 ceph-mon[112979]: pgmap v792: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:06:55.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:54 smithi032 ceph-mon[146817]: pgmap v792: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:06:57.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:56 smithi067 ceph-mon[112979]: pgmap v793: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:06:57.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:56 smithi032 ceph-mon[146817]: pgmap v793: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:06:59.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:06:58 smithi067 ceph-mon[112979]: pgmap v794: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:06:59.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:06:58 smithi032 ceph-mon[146817]: pgmap v794: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:07:00.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:00 smithi067 ceph-mon[112979]: pgmap v795: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:07:00.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:00 smithi032 ceph-mon[146817]: pgmap v795: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:07:03.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:02 smithi067 ceph-mon[112979]: pgmap v796: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:07:03.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:02 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:07:03.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:02 smithi032 ceph-mon[146817]: pgmap v796: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:07:03.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:02 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:07:05.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:04 smithi067 ceph-mon[112979]: pgmap v797: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:07:05.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:04 smithi032 ceph-mon[146817]: pgmap v797: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:07:07.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:06 smithi067 ceph-mon[112979]: pgmap v798: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:07:07.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:06 smithi032 ceph-mon[146817]: pgmap v798: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:07:09.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:08 smithi067 ceph-mon[112979]: pgmap v799: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:07:09.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:08 smithi032 ceph-mon[146817]: pgmap v799: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:07:11.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:10 smithi067 ceph-mon[112979]: pgmap v800: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:07:11.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:10 smithi032 ceph-mon[146817]: pgmap v800: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:07:13.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:12 smithi067 ceph-mon[112979]: pgmap v801: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:07:13.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:12 smithi032 ceph-mon[146817]: pgmap v801: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:07:15.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:14 smithi067 ceph-mon[112979]: pgmap v802: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:07:15.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:14 smithi032 ceph-mon[146817]: pgmap v802: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:07:17.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:16 smithi067 ceph-mon[112979]: pgmap v803: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:07:17.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:16 smithi032 ceph-mon[146817]: pgmap v803: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:07:18.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:17 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:07:18.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:17 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:07:19.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:18 smithi067 ceph-mon[112979]: pgmap v804: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:07:19.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:18 smithi032 ceph-mon[146817]: pgmap v804: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:07:21.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:20 smithi067 ceph-mon[112979]: pgmap v805: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:07:21.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:20 smithi032 ceph-mon[146817]: pgmap v805: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:07:23.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:22 smithi067 ceph-mon[112979]: pgmap v806: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:07:23.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:22 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T06:07:23.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:22 smithi032 ceph-mon[146817]: pgmap v806: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:07:23.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:22 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T06:07:24.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:23 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T06:07:24.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:23 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T06:07:24.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:23 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:07:24.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:23 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T06:07:24.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:23 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T06:07:24.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:23 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:07:25.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:24 smithi067 ceph-mon[112979]: pgmap v807: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:07:25.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:24 smithi032 ceph-mon[146817]: pgmap v807: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:07:26.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:25 smithi067 ceph-mon[112979]: pgmap v808: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:07:26.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:25 smithi032 ceph-mon[146817]: pgmap v808: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:07:29.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:28 smithi067 ceph-mon[112979]: pgmap v809: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:07:29.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:28 smithi032 ceph-mon[146817]: pgmap v809: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:07:31.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:30 smithi067 ceph-mon[112979]: pgmap v810: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:07:31.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:30 smithi032 ceph-mon[146817]: pgmap v810: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:07:33.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:32 smithi067 ceph-mon[112979]: pgmap v811: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:07:33.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:32 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:07:33.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:32 smithi032 ceph-mon[146817]: pgmap v811: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:07:33.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:32 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:07:35.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:34 smithi067 ceph-mon[112979]: pgmap v812: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:07:35.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:34 smithi032 ceph-mon[146817]: pgmap v812: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:07:37.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:36 smithi067 ceph-mon[112979]: pgmap v813: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:07:37.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:36 smithi032 ceph-mon[146817]: pgmap v813: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:07:39.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:38 smithi067 ceph-mon[112979]: pgmap v814: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:07:39.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:38 smithi032 ceph-mon[146817]: pgmap v814: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:07:41.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:40 smithi067 ceph-mon[112979]: pgmap v815: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:07:41.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:40 smithi032 ceph-mon[146817]: pgmap v815: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:07:43.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:42 smithi067 ceph-mon[112979]: pgmap v816: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:07:43.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:42 smithi032 ceph-mon[146817]: pgmap v816: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:07:44.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:44 smithi067 ceph-mon[112979]: pgmap v817: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:07:44.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:44 smithi032 ceph-mon[146817]: pgmap v817: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:07:47.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:46 smithi067 ceph-mon[112979]: pgmap v818: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:07:47.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:46 smithi032 ceph-mon[146817]: pgmap v818: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:07:48.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:47 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:07:48.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:47 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:07:49.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:48 smithi067 ceph-mon[112979]: pgmap v819: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:07:49.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:48 smithi032 ceph-mon[146817]: pgmap v819: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:07:51.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:50 smithi067 ceph-mon[112979]: pgmap v820: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:07:51.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:50 smithi032 ceph-mon[146817]: pgmap v820: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:07:53.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:52 smithi067 ceph-mon[112979]: pgmap v821: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:07:53.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:52 smithi032 ceph-mon[146817]: pgmap v821: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:07:55.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:54 smithi067 ceph-mon[112979]: pgmap v822: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:07:55.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:54 smithi032 ceph-mon[146817]: pgmap v822: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:07:57.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:56 smithi067 ceph-mon[112979]: pgmap v823: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:07:57.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:56 smithi032 ceph-mon[146817]: pgmap v823: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:07:58.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:07:57 smithi067 ceph-mon[112979]: pgmap v824: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:07:58.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:07:57 smithi032 ceph-mon[146817]: pgmap v824: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:08:01.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:00 smithi067 ceph-mon[112979]: pgmap v825: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:08:01.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:00 smithi032 ceph-mon[146817]: pgmap v825: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:08:03.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:02 smithi067 ceph-mon[112979]: pgmap v826: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:08:03.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:02 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:08:03.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:02 smithi032 ceph-mon[146817]: pgmap v826: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:08:03.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:02 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:08:05.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:04 smithi067 ceph-mon[112979]: pgmap v827: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:08:05.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:04 smithi032 ceph-mon[146817]: pgmap v827: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:08:07.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:06 smithi067 ceph-mon[112979]: pgmap v828: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:08:07.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:06 smithi032 ceph-mon[146817]: pgmap v828: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:08:09.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:08 smithi067 ceph-mon[112979]: pgmap v829: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:08:09.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:08 smithi032 ceph-mon[146817]: pgmap v829: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:08:11.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:10 smithi067 ceph-mon[112979]: pgmap v830: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:08:11.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:10 smithi032 ceph-mon[146817]: pgmap v830: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:08:13.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:12 smithi067 ceph-mon[112979]: pgmap v831: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:08:13.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:12 smithi032 ceph-mon[146817]: pgmap v831: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:08:14.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:13 smithi067 ceph-mon[112979]: pgmap v832: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:08:14.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:13 smithi032 ceph-mon[146817]: pgmap v832: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:08:17.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:16 smithi067 ceph-mon[112979]: pgmap v833: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:08:17.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:16 smithi032 ceph-mon[146817]: pgmap v833: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:08:18.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:17 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:08:18.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:17 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:08:19.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:18 smithi067 ceph-mon[112979]: pgmap v834: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:08:19.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:18 smithi032 ceph-mon[146817]: pgmap v834: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:08:21.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:20 smithi067 ceph-mon[112979]: pgmap v835: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:08:21.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:20 smithi032 ceph-mon[146817]: pgmap v835: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:08:23.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:22 smithi067 ceph-mon[112979]: pgmap v836: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:08:23.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:22 smithi032 ceph-mon[146817]: pgmap v836: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:08:24.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:23 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T06:08:24.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:23 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T06:08:24.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:23 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T06:08:24.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:23 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:08:24.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:23 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T06:08:24.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:23 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T06:08:24.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:23 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T06:08:24.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:23 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:08:25.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:24 smithi067 ceph-mon[112979]: pgmap v837: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:08:25.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:24 smithi032 ceph-mon[146817]: pgmap v837: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:08:26.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:25 smithi067 ceph-mon[112979]: pgmap v838: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:08:26.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:25 smithi032 ceph-mon[146817]: pgmap v838: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:08:29.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:28 smithi067 ceph-mon[112979]: pgmap v839: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:08:29.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:28 smithi032 ceph-mon[146817]: pgmap v839: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:08:31.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:30 smithi067 ceph-mon[112979]: pgmap v840: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:08:31.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:30 smithi032 ceph-mon[146817]: pgmap v840: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:08:33.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:32 smithi067 ceph-mon[112979]: pgmap v841: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:08:33.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:32 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:08:33.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:32 smithi032 ceph-mon[146817]: pgmap v841: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:08:33.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:32 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:08:35.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:34 smithi067 ceph-mon[112979]: pgmap v842: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:08:35.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:34 smithi032 ceph-mon[146817]: pgmap v842: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:08:37.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:36 smithi067 ceph-mon[112979]: pgmap v843: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:08:37.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:36 smithi032 ceph-mon[146817]: pgmap v843: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:08:39.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:38 smithi067 ceph-mon[112979]: pgmap v844: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:08:39.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:38 smithi032 ceph-mon[146817]: pgmap v844: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:08:40.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:39 smithi067 ceph-mon[112979]: pgmap v845: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:08:40.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:39 smithi032 ceph-mon[146817]: pgmap v845: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:08:43.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:42 smithi067 ceph-mon[112979]: pgmap v846: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:08:43.342 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:42 smithi032 ceph-mon[146817]: pgmap v846: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:08:45.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:44 smithi067 ceph-mon[112979]: pgmap v847: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:08:45.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:44 smithi032 ceph-mon[146817]: pgmap v847: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:08:47.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:46 smithi067 ceph-mon[112979]: pgmap v848: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:08:47.339 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:46 smithi032 ceph-mon[146817]: pgmap v848: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:08:48.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:47 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:08:48.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:47 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:08:49.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:48 smithi067 ceph-mon[112979]: pgmap v849: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:08:49.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:48 smithi032 ceph-mon[146817]: pgmap v849: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:08:50.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:49 smithi067 ceph-mon[112979]: pgmap v850: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:08:50.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:49 smithi032 ceph-mon[146817]: pgmap v850: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:08:53.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:52 smithi067 ceph-mon[112979]: pgmap v851: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:08:53.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:52 smithi032 ceph-mon[146817]: pgmap v851: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:08:55.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:54 smithi067 ceph-mon[112979]: pgmap v852: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:08:55.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:54 smithi032 ceph-mon[146817]: pgmap v852: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:08:57.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:56 smithi067 ceph-mon[112979]: pgmap v853: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:08:57.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:56 smithi032 ceph-mon[146817]: pgmap v853: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:08:59.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:08:58 smithi067 ceph-mon[112979]: pgmap v854: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:08:59.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:08:58 smithi032 ceph-mon[146817]: pgmap v854: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:09:01.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:09:00 smithi067 ceph-mon[112979]: pgmap v855: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:09:01.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:09:00 smithi032 ceph-mon[146817]: pgmap v855: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:09:02.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:09:01 smithi067 ceph-mon[112979]: pgmap v856: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:09:02.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:09:01 smithi032 ceph-mon[146817]: pgmap v856: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:09:03.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:09:02 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:09:03.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:09:02 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:09:04.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:09:03 smithi067 ceph-mon[112979]: pgmap v857: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:09:04.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:09:03 smithi032 ceph-mon[146817]: pgmap v857: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:09:07.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:09:06 smithi067 ceph-mon[112979]: pgmap v858: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:09:07.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:09:06 smithi032 ceph-mon[146817]: pgmap v858: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:09:09.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:09:08 smithi067 ceph-mon[112979]: pgmap v859: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:09:09.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:09:08 smithi032 ceph-mon[146817]: pgmap v859: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:09:11.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:09:10 smithi067 ceph-mon[112979]: pgmap v860: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:09:11.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:09:10 smithi032 ceph-mon[146817]: pgmap v860: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:09:13.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:09:12 smithi067 ceph-mon[112979]: pgmap v861: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:09:13.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:09:12 smithi032 ceph-mon[146817]: pgmap v861: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:09:14.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:09:14 smithi032 ceph-mon[146817]: pgmap v862: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:09:14.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:09:14 smithi067 ceph-mon[112979]: pgmap v862: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:09:17.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:09:16 smithi067 ceph-mon[112979]: pgmap v863: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:09:17.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:09:16 smithi032 ceph-mon[146817]: pgmap v863: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:09:18.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:09:17 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:09:18.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:09:17 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T06:09:19.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:09:18 smithi067 ceph-mon[112979]: pgmap v864: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:09:19.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:09:18 smithi032 ceph-mon[146817]: pgmap v864: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:09:20.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:09:20 smithi032 ceph-mon[146817]: pgmap v865: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:09:20.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:09:20 smithi067 ceph-mon[112979]: pgmap v865: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:09:23.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:09:22 smithi067 ceph-mon[112979]: pgmap v866: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:09:23.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:09:22 smithi032 ceph-mon[146817]: pgmap v866: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-12T06:09:24.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:09:23 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T06:09:24.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:09:23 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T06:09:25.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:09:24 smithi067 ceph-mon[112979]: pgmap v867: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:09:25.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:09:24 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T06:09:25.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:09:24 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T06:09:25.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:09:24 smithi067 ceph-mon[112979]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:09:25.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:09:24 smithi032 ceph-mon[146817]: pgmap v867: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-12T06:09:25.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:09:24 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T06:09:25.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:09:24 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T06:09:25.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:09:24 smithi032 ceph-mon[146817]: from='mgr.34104 172.21.15.32:0/917068178' entity='mgr.smithi032.hrwxrh' 2024-04-12T06:09:26.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:09:26 smithi032 ceph-mon[146817]: pgmap v868: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:09:26.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:09:26 smithi067 ceph-mon[112979]: pgmap v868: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:09:29.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:09:28 smithi067 ceph-mon[112979]: pgmap v869: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:09:29.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:09:28 smithi032 ceph-mon[146817]: pgmap v869: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-12T06:09:31.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:09:30 smithi067 ceph-mon[112979]: pgmap v870: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:09:31.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:09:30 smithi032 ceph-mon[146817]: pgmap v870: 65 pgs: 65 active+clean; 270 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-12T06:09:31.619 ERROR:tasks.cephfs.fuse_mount:process failed to terminate after unmount. This probably indicates a bug within ceph-fuse. 2024-04-12T06:09:31.619 ERROR:teuthology.run_tasks:Manager failed: ceph-fuse Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/cephfs/fuse_mount.py", line 400, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-12T06:09:31.621 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-12T06:09:31.633 INFO:tasks.cephadm:Teardown begin 2024-04-12T06:09:31.633 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/cephadm.py", line 1750, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/cephfs/fuse_mount.py", line 400, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-12T06:09:31.634 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-12T06:09:31.672 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-12T06:09:31.711 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-12T06:09:31.711 DEBUG:teuthology.orchestra.run.smithi032:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-12T06:09:31.733 DEBUG:teuthology.orchestra.run.smithi067:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-12T06:09:31.773 INFO:tasks.cephadm:Stopping all daemons... 2024-04-12T06:09:31.773 INFO:tasks.cephadm.mon.smithi032:Stopping mon.smithi032... 2024-04-12T06:09:31.773 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@mon.smithi032 2024-04-12T06:09:32.077 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:09:31 smithi032 systemd[1]: Stopping Ceph mon.smithi032 for 37b58ce2-f88d-11ee-bc8c-c7b262605968... 2024-04-12T06:09:32.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:09:32 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-mon-smithi032[146794]: 2024-04-12T06:09:32.078+0000 7f3eb5947700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi032 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T06:09:32.340 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:09:32 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-mon-smithi032[146794]: 2024-04-12T06:09:32.078+0000 7f3eb5947700 -1 mon.smithi032@0(leader) e3 *** Got Signal Terminated *** 2024-04-12T06:09:32.840 INFO:journalctl@ceph.mon.smithi032.smithi032.stdout:Apr 12 06:09:32 smithi032 bash[203662]: ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-mon-smithi032 2024-04-12T06:09:33.116 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@mon.smithi032.service' 2024-04-12T06:09:33.191 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-12T06:09:33.191 INFO:tasks.cephadm.mon.smithi032:Stopped mon.smithi032 2024-04-12T06:09:33.191 INFO:tasks.cephadm.mon.smithi067:Stopping mon.smithi067... 2024-04-12T06:09:33.191 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@mon.smithi067 2024-04-12T06:09:33.500 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:09:33 smithi067 systemd[1]: Stopping Ceph mon.smithi067 for 37b58ce2-f88d-11ee-bc8c-c7b262605968... 2024-04-12T06:09:33.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:09:33 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-mon-smithi067[112956]: 2024-04-12T06:09:33.500+0000 7fe850839700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi067 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T06:09:33.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:09:33 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-mon-smithi067[112956]: 2024-04-12T06:09:33.500+0000 7fe850839700 -1 mon.smithi067@1(peon) e3 *** Got Signal Terminated *** 2024-04-12T06:09:34.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 12 06:09:34 smithi067 bash[154978]: ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-mon-smithi067 2024-04-12T06:09:34.650 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@mon.smithi067.service' 2024-04-12T06:09:34.700 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-12T06:09:34.700 INFO:tasks.cephadm.mon.smithi067:Stopped mon.smithi067 2024-04-12T06:09:34.700 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-12T06:09:34.700 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@osd.0 2024-04-12T06:09:35.015 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 06:09:34 smithi032 systemd[1]: Stopping Ceph osd.0 for 37b58ce2-f88d-11ee-bc8c-c7b262605968... 2024-04-12T06:09:35.340 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 06:09:35 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-0[161267]: 2024-04-12T06:09:35.016+0000 7f26c0f6d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T06:09:35.340 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 06:09:35 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-0[161267]: 2024-04-12T06:09:35.016+0000 7f26c0f6d700 -1 osd.0 85 *** Got signal Terminated *** 2024-04-12T06:09:35.340 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 06:09:35 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-0[161267]: 2024-04-12T06:09:35.016+0000 7f26c0f6d700 -1 osd.0 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T06:09:41.340 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 12 06:09:40 smithi032 bash[203823]: ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-0 2024-04-12T06:09:42.879 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@osd.0.service' 2024-04-12T06:09:42.926 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-12T06:09:42.926 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-12T06:09:42.926 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-12T06:09:42.926 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@osd.1 2024-04-12T06:09:43.185 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 06:09:42 smithi032 systemd[1]: Stopping Ceph osd.1 for 37b58ce2-f88d-11ee-bc8c-c7b262605968... 2024-04-12T06:09:43.590 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 06:09:43 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-1[167952]: 2024-04-12T06:09:43.186+0000 7f5ebf997700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T06:09:43.590 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 06:09:43 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-1[167952]: 2024-04-12T06:09:43.186+0000 7f5ebf997700 -1 osd.1 85 *** Got signal Terminated *** 2024-04-12T06:09:43.590 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 06:09:43 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-1[167952]: 2024-04-12T06:09:43.186+0000 7f5ebf997700 -1 osd.1 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T06:09:49.340 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 06:09:49 smithi032 bash[204093]: ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-1 2024-04-12T06:09:51.069 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 06:09:51 smithi032 systemd[1]: ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@osd.1.service: Succeeded. 2024-04-12T06:09:51.069 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 12 06:09:51 smithi032 systemd[1]: Stopped Ceph osd.1 for 37b58ce2-f88d-11ee-bc8c-c7b262605968. 2024-04-12T06:09:51.071 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@osd.1.service' 2024-04-12T06:09:51.114 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-12T06:09:51.114 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-12T06:09:51.115 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-12T06:09:51.115 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@osd.2 2024-04-12T06:09:51.340 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 06:09:51 smithi032 systemd[1]: Stopping Ceph osd.2 for 37b58ce2-f88d-11ee-bc8c-c7b262605968... 2024-04-12T06:09:51.840 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 06:09:51 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-2[174863]: 2024-04-12T06:09:51.497+0000 7fbcb1132700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T06:09:51.840 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 06:09:51 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-2[174863]: 2024-04-12T06:09:51.497+0000 7fbcb1132700 -1 osd.2 85 *** Got signal Terminated *** 2024-04-12T06:09:51.840 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 06:09:51 smithi032 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-2[174863]: 2024-04-12T06:09:51.497+0000 7fbcb1132700 -1 osd.2 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T06:09:57.590 INFO:journalctl@ceph.osd.2.smithi032.stdout:Apr 12 06:09:57 smithi032 bash[204363]: ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-2 2024-04-12T06:09:59.409 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@osd.2.service' 2024-04-12T06:09:59.458 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-12T06:09:59.459 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-12T06:09:59.459 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-12T06:09:59.459 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@osd.3 2024-04-12T06:09:59.770 INFO:journalctl@ceph.osd.3.smithi067.stdout:Apr 12 06:09:59 smithi067 systemd[1]: Stopping Ceph osd.3 for 37b58ce2-f88d-11ee-bc8c-c7b262605968... 2024-04-12T06:09:59.770 INFO:journalctl@ceph.osd.3.smithi067.stdout:Apr 12 06:09:59 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-3[125245]: 2024-04-12T06:09:59.726+0000 7ff346fe0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T06:09:59.770 INFO:journalctl@ceph.osd.3.smithi067.stdout:Apr 12 06:09:59 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-3[125245]: 2024-04-12T06:09:59.726+0000 7ff346fe0700 -1 osd.3 85 *** Got signal Terminated *** 2024-04-12T06:09:59.771 INFO:journalctl@ceph.osd.3.smithi067.stdout:Apr 12 06:09:59 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-3[125245]: 2024-04-12T06:09:59.726+0000 7ff346fe0700 -1 osd.3 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T06:10:02.019 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:01 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:01.719+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-12T06:09:39.186969+0000 front 2024-04-12T06:09:39.187036+0000 (oldest deadline 2024-04-12T06:10:01.486934+0000) 2024-04-12T06:10:03.019 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:02 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:02.754+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-12T06:09:39.186969+0000 front 2024-04-12T06:09:39.187036+0000 (oldest deadline 2024-04-12T06:10:01.486934+0000) 2024-04-12T06:10:04.019 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:03 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:03.765+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-12T06:09:39.186969+0000 front 2024-04-12T06:09:39.187036+0000 (oldest deadline 2024-04-12T06:10:01.486934+0000) 2024-04-12T06:10:05.064 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:04 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:04.785+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-12T06:09:39.186969+0000 front 2024-04-12T06:09:39.187036+0000 (oldest deadline 2024-04-12T06:10:01.486934+0000) 2024-04-12T06:10:05.347 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 06:10:05 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-4[131384]: 2024-04-12T06:10:05.066+0000 7f15391d8700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-12T06:09:38.959986+0000 front 2024-04-12T06:09:38.960024+0000 (oldest deadline 2024-04-12T06:10:04.859946+0000) 2024-04-12T06:10:05.769 INFO:journalctl@ceph.osd.3.smithi067.stdout:Apr 12 06:10:05 smithi067 bash[155137]: ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-3 2024-04-12T06:10:06.269 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 06:10:06 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-4[131384]: 2024-04-12T06:10:06.045+0000 7f15391d8700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-12T06:09:38.959986+0000 front 2024-04-12T06:09:38.960024+0000 (oldest deadline 2024-04-12T06:10:04.859946+0000) 2024-04-12T06:10:06.269 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:05 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:05.832+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-12T06:09:39.186969+0000 front 2024-04-12T06:09:39.187036+0000 (oldest deadline 2024-04-12T06:10:01.486934+0000) 2024-04-12T06:10:07.019 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:06 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:06.805+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-12T06:09:39.186969+0000 front 2024-04-12T06:09:39.187036+0000 (oldest deadline 2024-04-12T06:10:01.486934+0000) 2024-04-12T06:10:07.519 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 06:10:07 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-4[131384]: 2024-04-12T06:10:07.059+0000 7f15391d8700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-12T06:09:38.959986+0000 front 2024-04-12T06:09:38.960024+0000 (oldest deadline 2024-04-12T06:10:04.859946+0000) 2024-04-12T06:10:07.556 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@osd.3.service' 2024-04-12T06:10:07.602 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-12T06:10:07.603 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-12T06:10:07.603 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-12T06:10:07.603 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@osd.4 2024-04-12T06:10:07.847 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 06:10:07 smithi067 systemd[1]: Stopping Ceph osd.4 for 37b58ce2-f88d-11ee-bc8c-c7b262605968... 2024-04-12T06:10:08.269 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 06:10:07 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-4[131384]: 2024-04-12T06:10:07.878+0000 7f153c3e1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T06:10:08.269 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 06:10:07 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-4[131384]: 2024-04-12T06:10:07.878+0000 7f153c3e1700 -1 osd.4 85 *** Got signal Terminated *** 2024-04-12T06:10:08.270 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 06:10:07 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-4[131384]: 2024-04-12T06:10:07.878+0000 7f153c3e1700 -1 osd.4 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T06:10:08.270 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 06:10:08 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-4[131384]: 2024-04-12T06:10:08.020+0000 7f15391d8700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-12T06:09:38.959986+0000 front 2024-04-12T06:09:38.960024+0000 (oldest deadline 2024-04-12T06:10:04.859946+0000) 2024-04-12T06:10:08.270 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:07 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:07.848+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-12T06:09:39.186969+0000 front 2024-04-12T06:09:39.187036+0000 (oldest deadline 2024-04-12T06:10:01.486934+0000) 2024-04-12T06:10:09.269 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 06:10:09 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-4[131384]: 2024-04-12T06:10:09.050+0000 7f15391d8700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-12T06:09:38.959986+0000 front 2024-04-12T06:09:38.960024+0000 (oldest deadline 2024-04-12T06:10:04.859946+0000) 2024-04-12T06:10:09.270 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:08 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:08.815+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-12T06:09:39.186969+0000 front 2024-04-12T06:09:39.187036+0000 (oldest deadline 2024-04-12T06:10:01.486934+0000) 2024-04-12T06:10:10.019 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:09 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:09.768+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-12T06:09:39.186969+0000 front 2024-04-12T06:09:39.187036+0000 (oldest deadline 2024-04-12T06:10:01.486934+0000) 2024-04-12T06:10:10.019 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:09 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:09.768+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-04-12T06:09:47.288469+0000 front 2024-04-12T06:09:47.288520+0000 (oldest deadline 2024-04-12T06:10:08.988361+0000) 2024-04-12T06:10:10.519 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 06:10:10 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-4[131384]: 2024-04-12T06:10:10.038+0000 7f15391d8700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-12T06:09:38.959986+0000 front 2024-04-12T06:09:38.960024+0000 (oldest deadline 2024-04-12T06:10:04.859946+0000) 2024-04-12T06:10:11.019 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:10 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:10.764+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-12T06:09:39.186969+0000 front 2024-04-12T06:09:39.187036+0000 (oldest deadline 2024-04-12T06:10:01.486934+0000) 2024-04-12T06:10:11.019 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:10 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:10.764+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-04-12T06:09:47.288469+0000 front 2024-04-12T06:09:47.288520+0000 (oldest deadline 2024-04-12T06:10:08.988361+0000) 2024-04-12T06:10:11.519 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 06:10:11 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-4[131384]: 2024-04-12T06:10:11.035+0000 7f15391d8700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-12T06:09:38.959986+0000 front 2024-04-12T06:09:38.960024+0000 (oldest deadline 2024-04-12T06:10:04.859946+0000) 2024-04-12T06:10:12.019 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:11 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:11.770+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-12T06:09:39.186969+0000 front 2024-04-12T06:09:39.187036+0000 (oldest deadline 2024-04-12T06:10:01.486934+0000) 2024-04-12T06:10:12.019 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:11 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:11.770+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-04-12T06:09:47.288469+0000 front 2024-04-12T06:09:47.288520+0000 (oldest deadline 2024-04-12T06:10:08.988361+0000) 2024-04-12T06:10:12.519 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 06:10:12 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-4[131384]: 2024-04-12T06:10:12.077+0000 7f15391d8700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-12T06:09:38.959986+0000 front 2024-04-12T06:09:38.960024+0000 (oldest deadline 2024-04-12T06:10:04.859946+0000) 2024-04-12T06:10:13.269 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:12 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:12.804+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-12T06:09:39.186969+0000 front 2024-04-12T06:09:39.187036+0000 (oldest deadline 2024-04-12T06:10:01.486934+0000) 2024-04-12T06:10:13.269 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:12 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:12.804+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-04-12T06:09:47.288469+0000 front 2024-04-12T06:09:47.288520+0000 (oldest deadline 2024-04-12T06:10:08.988361+0000) 2024-04-12T06:10:14.019 INFO:journalctl@ceph.osd.4.smithi067.stdout:Apr 12 06:10:13 smithi067 bash[155411]: ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-4 2024-04-12T06:10:14.019 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:13 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:13.791+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-12T06:09:39.186969+0000 front 2024-04-12T06:09:39.187036+0000 (oldest deadline 2024-04-12T06:10:01.486934+0000) 2024-04-12T06:10:14.020 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:13 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:13.791+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-04-12T06:09:47.288469+0000 front 2024-04-12T06:09:47.288520+0000 (oldest deadline 2024-04-12T06:10:08.988361+0000) 2024-04-12T06:10:15.269 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:14 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:14.797+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-12T06:09:39.186969+0000 front 2024-04-12T06:09:39.187036+0000 (oldest deadline 2024-04-12T06:10:01.486934+0000) 2024-04-12T06:10:15.270 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:14 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:14.797+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-04-12T06:09:47.288469+0000 front 2024-04-12T06:09:47.288520+0000 (oldest deadline 2024-04-12T06:10:08.988361+0000) 2024-04-12T06:10:15.684 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@osd.4.service' 2024-04-12T06:10:15.730 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-12T06:10:15.731 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-12T06:10:15.731 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-12T06:10:15.731 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@osd.5 2024-04-12T06:10:16.016 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:15 smithi067 systemd[1]: Stopping Ceph osd.5 for 37b58ce2-f88d-11ee-bc8c-c7b262605968... 2024-04-12T06:10:16.016 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:15 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:15.819+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-12T06:09:39.186969+0000 front 2024-04-12T06:09:39.187036+0000 (oldest deadline 2024-04-12T06:10:01.486934+0000) 2024-04-12T06:10:16.016 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:15 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:15.819+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-04-12T06:09:47.288469+0000 front 2024-04-12T06:09:47.288520+0000 (oldest deadline 2024-04-12T06:10:08.988361+0000) 2024-04-12T06:10:16.017 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:16 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:16.017+0000 7f59665b2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T06:10:16.017 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:16 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:16.017+0000 7f59665b2700 -1 osd.5 85 *** Got signal Terminated *** 2024-04-12T06:10:16.269 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:16 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:16.017+0000 7f59665b2700 -1 osd.5 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T06:10:17.269 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:16 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:16.854+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-12T06:09:39.186969+0000 front 2024-04-12T06:09:39.187036+0000 (oldest deadline 2024-04-12T06:10:01.486934+0000) 2024-04-12T06:10:17.269 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:16 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:16.854+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-04-12T06:09:47.288469+0000 front 2024-04-12T06:09:47.288520+0000 (oldest deadline 2024-04-12T06:10:08.988361+0000) 2024-04-12T06:10:18.269 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:17 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:17.875+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-12T06:09:39.186969+0000 front 2024-04-12T06:09:39.187036+0000 (oldest deadline 2024-04-12T06:10:01.486934+0000) 2024-04-12T06:10:18.269 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:17 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:17.875+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-04-12T06:09:47.288469+0000 front 2024-04-12T06:09:47.288520+0000 (oldest deadline 2024-04-12T06:10:08.988361+0000) 2024-04-12T06:10:18.269 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:17 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:17.875+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-04-12T06:09:53.689282+0000 front 2024-04-12T06:09:53.689314+0000 (oldest deadline 2024-04-12T06:10:17.789278+0000) 2024-04-12T06:10:19.269 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:18 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:18.895+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-12T06:09:39.186969+0000 front 2024-04-12T06:09:39.187036+0000 (oldest deadline 2024-04-12T06:10:01.486934+0000) 2024-04-12T06:10:19.269 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:18 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:18.895+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-04-12T06:09:47.288469+0000 front 2024-04-12T06:09:47.288520+0000 (oldest deadline 2024-04-12T06:10:08.988361+0000) 2024-04-12T06:10:19.269 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:18 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:18.895+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-04-12T06:09:53.689282+0000 front 2024-04-12T06:09:53.689314+0000 (oldest deadline 2024-04-12T06:10:17.789278+0000) 2024-04-12T06:10:20.269 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:19 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:19.852+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-12T06:09:39.186969+0000 front 2024-04-12T06:09:39.187036+0000 (oldest deadline 2024-04-12T06:10:01.486934+0000) 2024-04-12T06:10:20.269 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:19 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:19.852+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-04-12T06:09:47.288469+0000 front 2024-04-12T06:09:47.288520+0000 (oldest deadline 2024-04-12T06:10:08.988361+0000) 2024-04-12T06:10:20.269 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:19 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:19.852+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-04-12T06:09:53.689282+0000 front 2024-04-12T06:09:53.689314+0000 (oldest deadline 2024-04-12T06:10:17.789278+0000) 2024-04-12T06:10:21.269 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:20 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:20.870+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-12T06:09:39.186969+0000 front 2024-04-12T06:09:39.187036+0000 (oldest deadline 2024-04-12T06:10:01.486934+0000) 2024-04-12T06:10:21.269 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:20 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:20.870+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-04-12T06:09:47.288469+0000 front 2024-04-12T06:09:47.288520+0000 (oldest deadline 2024-04-12T06:10:08.988361+0000) 2024-04-12T06:10:21.269 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:20 smithi067 ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5[137482]: 2024-04-12T06:10:20.870+0000 7f59633a9700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-04-12T06:09:53.689282+0000 front 2024-04-12T06:09:53.689314+0000 (oldest deadline 2024-04-12T06:10:17.789278+0000) 2024-04-12T06:10:22.020 INFO:journalctl@ceph.osd.5.smithi067.stdout:Apr 12 06:10:21 smithi067 bash[155673]: ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968-osd-5 2024-04-12T06:10:23.523 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-37b58ce2-f88d-11ee-bc8c-c7b262605968@osd.5.service' 2024-04-12T06:10:23.567 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-12T06:10:23.567 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-12T06:10:23.568 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 --force --keep-logs 2024-04-12T06:10:26.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[101577]: fuse finished with error 0 and tester_r 0 2024-04-12T06:10:45.303 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 --force --keep-logs 2024-04-12T06:10:57.798 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-12T06:10:57.836 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-12T06:10:57.868 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-12T06:10:57.870 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/crash to /home/teuthworker/archive/yuriw-2024-04-11_20:32:36-orch-reef-release-distro-default-smithi/7652637/remote/smithi032/crash 2024-04-12T06:10:57.871 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/crash -- . 2024-04-12T06:10:57.908 INFO:teuthology.orchestra.run.smithi032.stderr:tar: /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/crash: Cannot open: No such file or directory 2024-04-12T06:10:57.909 INFO:teuthology.orchestra.run.smithi032.stderr:tar: Error is not recoverable: exiting now 2024-04-12T06:10:57.910 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/crash to /home/teuthworker/archive/yuriw-2024-04-11_20:32:36-orch-reef-release-distro-default-smithi/7652637/remote/smithi067/crash 2024-04-12T06:10:57.911 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/crash -- . 2024-04-12T06:10:57.938 INFO:teuthology.orchestra.run.smithi067.stderr:tar: /var/lib/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/crash: Cannot open: No such file or directory 2024-04-12T06:10:57.939 INFO:teuthology.orchestra.run.smithi067.stderr:tar: Error is not recoverable: exiting now 2024-04-12T06:10:57.939 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-12T06:10:57.940 DEBUG:teuthology.orchestra.run.smithi032:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-04-12T06:10:57.985 INFO:teuthology.orchestra.run.smithi032.stderr:grep: /var/log/ceph/37b58ce2-f88d-11ee-bc8c-c7b262605968/ceph.log: No such file or directory 2024-04-12T06:10:57.988 INFO:tasks.cephadm:Compressing logs... 2024-04-12T06:10:57.988 DEBUG:teuthology.orchestra.run.smithi032:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-12T06:10:58.031 DEBUG:teuthology.orchestra.run.smithi067:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-12T06:10:58.058 INFO:teuthology.orchestra.run.smithi067.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-12T06:10:58.059 INFO:teuthology.orchestra.run.smithi032.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-12T06:12:46.339 INFO:teuthology.orchestra.run.smithi067.stderr:gzip: /var/log/ceph/ceph-client.1.log: file size changed while zipping 2024-04-12T06:12:46.871 INFO:tasks.cephadm:Archiving logs... 2024-04-12T06:12:46.872 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-11_20:32:36-orch-reef-release-distro-default-smithi/7652637/remote/smithi032/log 2024-04-12T06:12:46.873 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-12T06:12:59.137 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-11_20:32:36-orch-reef-release-distro-default-smithi/7652637/remote/smithi067/log 2024-04-12T06:12:59.138 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-12T06:13:10.971 INFO:tasks.cephadm:Removing cluster... 2024-04-12T06:13:10.971 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 --force 2024-04-12T06:13:11.535 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 37b58ce2-f88d-11ee-bc8c-c7b262605968 --force 2024-04-12T06:13:12.102 INFO:tasks.cephadm:Removing cephadm ... 2024-04-12T06:13:12.102 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-12T06:13:12.127 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-12T06:13:12.148 INFO:tasks.cephadm:Teardown complete 2024-04-12T06:13:12.149 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-12T06:13:12.241 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/cephfs/fuse_mount.py", line 400, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-12T06:13:12.242 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-12T06:13:12.242 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-12T06:13:12.245 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-12T06:13:12.329 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-12T06:13:12.387 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-12T06:13:12.387 DEBUG:teuthology.orchestra.run.smithi032:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-12T06:13:12.391 DEBUG:teuthology.orchestra.run.smithi067:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-12T06:13:12.407 INFO:teuthology.orchestra.run.smithi032.stderr:bash: ntpq: command not found 2024-04-12T06:13:12.409 INFO:teuthology.orchestra.run.smithi067.stderr:bash: ntpq: command not found 2024-04-12T06:13:12.413 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-12T06:13:12.413 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2024-04-12T06:13:12.413 INFO:teuthology.orchestra.run.smithi032.stdout:^* hv01.front.sepia.ceph.com 3 6 377 50 +756us[ +779us] +/- 43ms 2024-04-12T06:13:12.413 INFO:teuthology.orchestra.run.smithi032.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 53 -748us[ -726us] +/- 81ms 2024-04-12T06:13:12.413 INFO:teuthology.orchestra.run.smithi032.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 -577us[ -554us] +/- 76ms 2024-04-12T06:13:12.413 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T06:13:12.414 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-12T06:13:12.414 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-04-12T06:13:12.414 INFO:teuthology.orchestra.run.smithi067.stdout:^* hv01.front.sepia.ceph.com 3 6 377 50 +748us[ +766us] +/- 43ms 2024-04-12T06:13:12.414 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 52 -746us[ -727us] +/- 81ms 2024-04-12T06:13:12.414 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 -570us[ -552us] +/- 76ms 2024-04-12T06:13:12.414 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T06:13:12.416 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-12T06:13:12.438 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-12T06:13:12.438 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-12T06:13:12.451 DEBUG:teuthology.orchestra.run.smithi032:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-12T06:13:12.539 DEBUG:teuthology.orchestra.run.smithi067:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-12T06:13:12.615 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-12T06:13:12.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T06:13:12.649 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-12T06:13:12.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T06:13:12.680 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-12T06:13:12.692 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-12T06:13:12.703 INFO:teuthology.task.internal:Duration was 3501.584840 seconds 2024-04-12T06:13:12.704 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-12T06:13:12.715 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-12T06:13:12.715 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-12T06:13:12.717 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-12T06:13:12.765 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-12T06:13:12.767 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-12T06:13:13.104 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-12T06:13:13.104 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi032.front.sepia.ceph.com 2024-04-12T06:13:13.104 DEBUG:teuthology.orchestra.run.smithi032:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-12T06:13:13.129 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi067.front.sepia.ceph.com 2024-04-12T06:13:13.129 DEBUG:teuthology.orchestra.run.smithi067:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-12T06:13:13.152 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-12T06:13:13.153 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-12T06:13:13.172 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-12T06:13:13.250 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-12T06:13:13.250 DEBUG:teuthology.orchestra.run.smithi032:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-12T06:13:13.253 DEBUG:teuthology.orchestra.run.smithi067:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-12T06:13:13.387 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-12T06:13:13.399 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-12T06:13:13.399 DEBUG:teuthology.orchestra.run.smithi032:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-12T06:13:13.427 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-12T06:13:13.455 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-12T06:13:13.467 DEBUG:teuthology.orchestra.run.smithi032:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-12T06:13:13.470 DEBUG:teuthology.orchestra.run.smithi067:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-12T06:13:13.495 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = core 2024-04-12T06:13:13.522 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = core 2024-04-12T06:13:13.538 DEBUG:teuthology.orchestra.run.smithi032:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-12T06:13:13.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T06:13:13.590 DEBUG:teuthology.orchestra.run.smithi067:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-12T06:13:13.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T06:13:13.608 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-12T06:13:13.620 INFO:teuthology.task.internal:Transferring archived files... 2024-04-12T06:13:13.621 DEBUG:teuthology.misc:Transferring archived files from smithi032:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-11_20:32:36-orch-reef-release-distro-default-smithi/7652637/remote/smithi032 2024-04-12T06:13:13.621 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-12T06:13:13.678 DEBUG:teuthology.misc:Transferring archived files from smithi067:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-11_20:32:36-orch-reef-release-distro-default-smithi/7652637/remote/smithi067 2024-04-12T06:13:13.678 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-12T06:13:13.720 INFO:teuthology.task.internal:Removing archive directory... 2024-04-12T06:13:13.720 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-12T06:13:13.722 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-12T06:13:13.771 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-12T06:13:13.787 INFO:teuthology.task.internal:Not uploading archives. 2024-04-12T06:13:13.788 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-12T06:13:13.800 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-12T06:13:13.800 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-12T06:13:13.803 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-12T06:13:13.818 INFO:teuthology.orchestra.run.smithi032.stdout: 525357 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 12 06:13 /home/ubuntu/cephtest 2024-04-12T06:13:13.818 INFO:teuthology.orchestra.run.smithi032.stdout: 527596 4 d--------- 2 ubuntu ubuntu 4096 Apr 12 05:34 /home/ubuntu/cephtest/mnt.0 2024-04-12T06:13:13.818 INFO:teuthology.orchestra.run.smithi032.stderr:find: ‘/home/ubuntu/cephtest/mnt.0’: Permission denied 2024-04-12T06:13:13.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T06:13:13.819 INFO:teuthology.orchestra.run.smithi032.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-12T06:13:13.819 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/cephfs/fuse_mount.py", line 400, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi032 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-04-12T06:13:13.820 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-12T06:13:13.833 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-12T06:13:13.881 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-12T06:13:13.881 INFO:teuthology.nuke:Checking targets against current locks 2024-04-12T06:13:13.907 DEBUG:teuthology.nuke:shortname: smithi032 2024-04-12T06:13:13.907 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-12T06:13:13.927 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-11_20:32:36-orch-reef-release-distro-default-smithi/7652637', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-12 05:06:36.170556', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGwLR3pYMLw2TyxNMHAAobb3adjJnjw42qJCHwFMzm8aE4E0dQ2GbQUtEsEKQqZDklHhErCNA+7oG7UyJm7OX6E='} 2024-04-12T06:13:13.930 DEBUG:teuthology.nuke:shortname: smithi067 2024-04-12T06:13:13.930 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-12T06:13:13.947 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-11_20:32:36-orch-reef-release-distro-default-smithi/7652637', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-12 05:06:36.168617', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCm5sT0zJNfvCAmY690EA0xz0PW/cSYnhpKCrJCL1Lg+BAHvqx6BIRr6D1bIVyb7LtWm42y47x55+MBu0dYE0wE='} 2024-04-12T06:13:13.978 INFO:teuthology.orchestra.console.smithi032:Power off 2024-04-12T06:13:13.978 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-12T06:13:13.996 INFO:teuthology.orchestra.console.smithi067:Power off 2024-04-12T06:13:13.996 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-12T06:13:14.007 DEBUG:teuthology.orchestra.console.smithi032:power off output: Chassis Power Control: Down/Off 2024-04-12T06:13:14.007 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T06:13:14.021 DEBUG:teuthology.orchestra.console.smithi067:power off output: Chassis Power Control: Down/Off 2024-04-12T06:13:14.021 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T06:13:14.031 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is on 2024-04-12T06:13:14.044 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-04-12T06:13:18.032 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T06:13:18.045 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T06:13:18.142 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is on 2024-04-12T06:13:18.154 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-04-12T06:13:22.147 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T06:13:22.157 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T06:13:22.260 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is off 2024-04-12T06:13:22.266 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is off 2024-04-12T06:13:22.361 INFO:teuthology.orchestra.console.smithi032:Power off completed 2024-04-12T06:13:22.367 INFO:teuthology.orchestra.console.smithi067:Power off completed 2024-04-12T06:13:22.472 INFO:teuthology.run:Summary data: description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 3501.584839820862 failure_reason: reached maximum tries (51) after waiting for 300 seconds flavor: default owner: scheduled_yuriw@teuthology status: fail success: false 2024-04-12T06:13:22.472 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-12T06:13:22.537 INFO:teuthology.run:FAIL